Serving Layer
The serving layer is the pair of services that sit in front of your models:
- an API gateway — LiteLLM: one OpenAI-compatible endpoint in front of any number of models, with keys, quotas and request logs;
- a chat interface — Open WebUI, pointed at a gateway or at any OpenAI-compatible endpoint.
They can come up together with a model launch (see the Access step of the Launch Guide, or the credentials block in AI Launch), or be managed entirely on their own from LLM Launcher → Serving Layer.
The page
The Serving Layer page is a summary of your account, not of one device: one endpoint in front of the models, wherever they run. The subheader reports how many of your devices are reachable, and a notice appears when some are not:
{n} device(s) are offline, so anything running on them is not listed here.
This is said explicitly because a gateway on a device that is off cannot be seen, and reporting that as "no gateway" would invite deploying a second one.
Two panels: API gateway and Chat interface, each with a running count.
API gateway (LiteLLM)
LiteLLM: one OpenAI-compatible endpoint in front of any number of models, with keys, quotas and request logs. One per device — models are registered into it as they come and go.
Deploying one
Deploy gateway asks for:
- Deploy on — which device.
- Credentials — the master key and admin details, shown before you deploy.
- Public URL — optionally reachable from outside this network.
- Register a model now — optionally point it at a model that is already serving.
Copy the credentials before you deploy — Cordatus does not store them. If an existing gateway is reused instead, it keeps its own and these are ignored.
A deploy in progress appears as a row in the panel and narrates itself, because pulling an image looks identical to a hang from the outside. It stays until the container itself shows up.
If a gateway is already running on the chosen device, the dialog says so and reuses it rather than deploying a second one. Deploying an extra one is still possible, offered quietly as Deploy an additional gateway anyway.
A gateway row
- Its name, its device, and its port.
- Serving — every model registered into it, as badges, or nothing yet.
- Open — opens its UI in a new tab.
Two states are reported rather than hidden:
| Badge | Meaning |
|---|---|
| not answering | Recorded on the device but not responding — stopped or removed outside Cordatus. The record is kept, so it can be started again from the Containers page. |
| not manageable | Its master key is not in its environment, so Cordatus cannot register models or issue keys on it. Use the LiteLLM UI directly. |
Register a model
Register a model puts a model that is already serving behind the gateway. Nothing is started — the model has to be running already.
You describe where it is:
- Take the details from — a running Container, a Sparkrun workload, a port on this device, or a full URL.
- Name clients send — what clients ask the gateway for. Keeping it separate from the real weights lets you swap them later without breaking anything.
- Name the backend answers to — the engine's own
--served-model-name. Left empty, the name above is used for both.
Issue keys
Issue keys creates LiteLLM virtual keys. Each has a name, an optional budget, and a scope — Every model on the gateway or one model only.
Hand these to your clients instead of the master key: each is scoped as you configured it and can be revoked from the gateway.
A key row carries its name, its generated value (revealed with the eye, copied with the button — the value is generated here and Cordatus does not store it), an optional budget, and a this model only checkbox. + Add key adds another row; the bin removes one.
With this model only ticked, a model has to be chosen in Scope the keys to. If it is left on Every model on the gateway, the key is refused rather than silently widened: "Virtual key ... was NOT created: it is scoped to this model, and the model name could not be resolved. Creating it unscoped would have granted access to every model on the gateway."
Chat interface (Open WebUI)
Open WebUI, pointed at a gateway or at any OpenAI-compatible endpoint. It needs no model of its own, so one interface can front everything the gateway serves.
Deploying one
Deploy a chat interface asks for:
- Deploy on — which device.
- Point it at — a Gateway, a running Model container, a Sparkrun workload, or a
base URL you type. For a gateway you can also give an API key; left empty, the scoped virtual
key is used instead. The key is passed to the interface as
OPENAI_API_KEY. - Admin account — e-mail and password, shown once. Reset all passwords regenerates them.
- Public URL — optionally.
- Open WebUI accounts — extra users created once the interface has booted, each with a name, an
e-mail, a generated password and a role (
useroradmin). A row with no e-mail or no password is skipped, and the dialog says so.
If nothing is chosen to talk to, the dialog says "Pick something for it to talk to first."
Add accounts
Add accounts creates extra Open WebUI users so people sign in with their own e-mail instead of sharing the admin account.
Open WebUI has no way to add an account without an admin session, and Cordatus never stored the admin credentials — so this dialog asks for them. Reset all passwords is available if they were lost.
Credentials, everywhere
Whenever a deploy creates something with a password, Cordatus shows a Save your serving layer credentials dialog immediately after the environment starts:
- Open WebUI admin account — address, e-mail, UI username and password. Sign in with these; the first-run registration step is already done for you.
- LiteLLM gateway — address, master key, model name for clients, image and ports.
- Virtual keys — one row per key.
- Additional accounts — one row per extra Open WebUI account. A rejected one (duplicate address, or a password under 8 characters) is named in the deploy log.
Copy all takes the lot.
Shown once only. Cordatus does not store these values — copy them before closing the dialog. The addresses themselves remain visible afterwards in each container's Ports tab.
If a gateway was reused rather than created, the dialog says so: no new gateway and no new password were created, so sign in with the credentials that gateway was deployed with.
Where these containers show up
Everything the serving layer deploys is an ordinary container group on the Containers page: the gateway plus its PostgreSQL, Redis and Prometheus containers, and the Open WebUI container. Start, stop, delete, logs and ports all work there as usual.
A gateway's first boot runs database migrations and takes a few minutes. It will not answer until they finish.