AI Launch
AI Launch is the one-dialog path to a running model. You give it a Hugging Face repository id — or press AI Launch on a row in the model list, which fills it in — and Cordatus works out the rest: which of your devices can run it, which engine, which container image, how many GPUs, which quantization, what context length, and which flags.
Nothing starts until you have seen the plan.
Where to start it from
| Starting point | What happens |
|---|---|
| LLM Models → Hugging Face, the AI Launch button on a model row | The model is already chosen; the dialog only has to pick a device and plan. |
| LLM Models → User Models, the AI Launch button on a model row | The model and the device are already chosen — the plan is made for the device that stores it. |
| Opened without a model | The dialog asks for a Hugging Face model id first (for example Qwen/Qwen3-8B). Press Enter to start planning. |
Next to AI Launch there is always Manual → Run manually with…, which sends you into the launch wizard instead.
1. Gated models and tokens
If the model list already knows the repository is gated, the dialog asks for a Hugging Face token before planning — a gated model needs one both to read its configuration and to download its weights, so asking up front saves a guaranteed failed round trip.
You can either:
- Choose a saved token from your token vault, or
- paste one into the field below it (
hf_…).
The token field is an API token, not a login password. Cordatus deliberately stops the browser's password manager from auto-filling it.
2. Planning
Planning is six phases, and the dialog shows which one it is in rather than a single spinner:
- Checking your devices — which devices are online, and what they report.
- Reading the model —
config.jsonand the repository file list. If the model is already on the device, this is read from the device itself rather than from Hugging Face. - Choosing an inference engine
- Matching a container image
- Allocating GPUs and context
- Tuning engine arguments
When it finishes, the header says Plan ready.
If Cordatus has a model path registered for the device but cannot read the files there, a warning appears above the result: "Could not read the model's files on the device … The plan fell back to Hugging Face." This is the cause of any download that follows — fix the path rather than the repository id or the token.
3. Choosing the target device
Under the input, Cordatus lists the devices that can run this model, ranked, with the chosen one selected:
{n} of {m} device(s) can run this model- Devices that were not considered are hidden behind Show {n} device(s) that were not considered, with the reason on each row.
- When the model came from User Models, the row reads "planned for {device}, where this model is registered" and the device is pinned.
- If the signal server cannot be reached, the list is built from the last heartbeat and says so.
There is no free device picker on purpose: the point of AI Launch is that the ranking is the recommendation, and you approve or override it by picking a different row.
4. The plan card
The result is a single answer — Ready to launch on {device} — with the facts beside it. If the plan cannot run, that line becomes the blocker instead, and blockers and warnings stay above the summary where they cannot be collapsed away.
Blockers, warnings and the model author's own caveats
- Blockers (red) stop the launch. Example: the model does not fit on any GPU in this device.
- Warnings (orange) let it proceed but tell you what to expect.
- Guide warnings (brown) are quoted verbatim from the recipe's own guide, with the quote shown as evidence.
Details
Details expands the full row list. Depending on the model you will see:
| Row | Meaning |
|---|---|
| Model | The repository the plan was made for. |
| Engine | The inference engine chosen (vLLM, llama.cpp, TensorRT-LLM, Ollama…). |
| Image | The exact container image and tag. |
| GPUs | Which GPUs, with tp= when tensor parallelism is used, or CPU only. |
| VRAM | Free vs. total memory on each selected GPU. Marked as a warning when the GPU is busy. |
| Context | The context window the flags will ask for. |
| Quantization | Which quantization was chosen, and the approximate bits-per-weight. |
| Model cache | Where the weights come from — already on the device and mounted at a path, or not downloaded yet. change lets you pick a different host directory. |
| Concurrency | How many requests at once the settings allow. |
| Speculative decoding | On or off, and what it was measured against. |
| Fits now | Whether it fits with the memory currently free, not just in theory. |
| Device capacity | What it would look like when the GPU is idle. |
Why these settings
Collapsed by default. Inside it:
- Evidence — where each decision came from: named in the docs, engine source, documented minimum, newest release, found in the registry, chosen from the published file sizes, measured against this plan's spare memory, GPU plan, where the weights come from, engine choice.
- Observations — qualitative notes about the model or hardware.
- Adjustments — what was changed to make it fit.
- Suggestions — what could be changed to make it better.
- Chosen for you / Default setting — the last decision and its rationale, plus other options listing the alternatives that were verified but not picked.
- Recipe note — where model-specific flags came from, and which machine that advice was originally written for.
Show the docker command
The exact command the device will run. Nothing is hidden behind the plan.
5. Credentials, access and accounts
Below the plan, the same serving layer configuration as the wizard, in compact form:
- Chat interface — Open WebUI, with an admin account created for you.
- API gateway — LiteLLM: one OpenAI-compatible endpoint with keys, quotas and request logs.
Expanding Credentials, access and accounts gives you the admin credentials, the master key, the extra Open WebUI accounts and the LiteLLM virtual keys, each with Reveal, Copy and Regenerate, plus Public URLs for reaching either service from outside the network.
Copy the credentials before you launch. Cordatus does not store them, and the dialog closes as the launch starts.
6. Launch, or hand over to the wizard
| Button | Result |
|---|---|
| Launch | Starts the plan. Disabled while a blocker stands. |
| Edit in Advanced | Opens the launch wizard with the device, image, model, GPUs and flags the planner worked out already filled in. |
| Try again | Re-runs planning after an error. |
| Cancel | Closes without starting anything. |
After launching, the containers appear on the Containers page like any other deployment.
Troubleshooting
| What you see | What it means |
|---|---|
| "This plan cannot be launched" | A blocker is listed directly above. Fix that, or pick a different device row. |
| "does not fit" under Fits now | The weights alone would fill the selected GPUs. Choose a quantized variant or a bigger device. |
| "Could not read the model's files on the device" | The registered model path on that device is unreadable. The plan fell back to Hugging Face and will download. |
| Planning never leaves Checking your devices | No device is online, or the signal server is unreachable. Check the Devices page. |
| The dialog asks for a token you already saved | The saved token is not scoped to this repository. Paste a token with access to it. |