{ "pages": [ { "slug": "api/authentication", "title": "API authentication", "description": "The shipped native email+password and wallet (SIWE) sign-in flows, the slate_session cookie, the AuthContext, optional OIDC, and the pluggable provider registry.", "section": "api", "body": "# Authentication Per [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md), the shipped and default auth plane is **native email+password sign-in** plus **wallet (SIWE) sign-in** — these two are co-primary. A successful sign-in mints a amini session and sets it as an httpOnly cookie named `slate_session`. OIDC (Authelia, Dex, Keycloak, Auth0, …) is now an **optional / legacy** path for operators who want to federate against an external issuer; it is no longer the default and the Compose stack no longer requires it for sign-in. This page is the developer's reference for **how to authenticate, what to do with the session, and how Amini Cloud validates it**. The native endpoints live in `internal/admission/auth_routes.go` and `internal/admission/wallet_routes.go`; the optional-OIDC machinery and the architectural choices behind pluggable providers live in [ADR 0013](../../decisions/0013-pluggable-auth-providers.md) and the matching concept page [Auth providers](../concepts/auth-providers.md). ## Native sign-in (the default) The native endpoints, all unauthenticated POSTs unless noted: | Endpoint | What it does | |---|---| | `POST /v1/auth/signup` | Create an account; `201` + a signup magic link emailed | | `POST /v1/auth/login` | Email + password → `200` + `slate_session` cookie | | `POST /v1/auth/wallet/nonce` | Body `{wallet_address, chain}` → a nonce to sign | | `POST /v1/auth/wallet/verify` | Body `{wallet_address, chain, nonce, signature}` → `200` + `slate_session` cookie | | `POST /v1/auth/magic-link/request` | Always `200` (no email enumeration); emails a link | | `GET /v1/auth/magic-link/verify?token=…` | `302 → /dashboard` + `slate_session` cookie | | `POST /v1/auth/logout` | `200`, cookie cleared | | `POST /v1/auth/password/reset/request` | Always `200`; emails a reset link | | `POST /v1/auth/password/reset/confirm` | `200` | | `POST /v1/auth/exchange` | The browser-PKCE exchange the SPA uses for the **optional** OIDC path (only mounted when OIDC is configured) | ### Browser path (the dashboard) The web sign-in page is at **`/login`**. It POSTs email + password directly to `/v1/auth/login`, sets the `slate_session` cookie, and lands you on `/dashboard`. Sibling paths on the same page: - **`/login/wallet`** — sign a nonce with an EVM or Cosmos wallet (the `/v1/auth/wallet/*` flow). No email, no password. - **`/signup`** — create an account. - **`/forgot-password`** (and `/forgot-password?mode=magic-link`) — password reset / magic-link sign-in. The dashboard talks to admission with `credentials: 'include'`, so the `slate_session` cookie rides along automatically; you do not copy a token by hand. ### Script / curl path ```bash # Email + password → session cookie (saved to a cookie jar) curl -fsS -c amini.cookies -X POST http://localhost:18080/v1/auth/login \\ -H 'content-type: application/json' \\ -d '{\"email\":\"you@example.com\",\"password\":\"\"}' # Reuse the cookie on subsequent calls curl -fsS -b amini.cookies http://localhost:18080/v1/tenants/me | jq # (single-host Compose default — see deploy/compose/.env.example for # AMINI_ADMISSION_PORT override; older builds bound to :8080) ``` A `200 OK` from `/v1/tenants/me` means admission accepted your session and you are provisioned in `tenant_users`. For scripts, turn that session into an **API key** and send it as `Authorization: Bearer `. Keys are the credential lane meant for API calls: named, listed, and revoked one at a time. The three calls that mint one are in [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api). > **The `AminiDev` shim is off by default, including locally.** Admission > honours `Authorization: AminiDev ` only when **all three** of > `AMINI_ENABLE_DEV_BEARER=true`, `WALLET_DEV_AUTO_PROVISION=true` and > `WALLET_DEV_AUTO_TENANT=` are set (`devBearerEnabled()`, > `cmd/amini-admission/main.go`). `deploy/compose/.env.example` ships the > last two but not the first, so a fre" }, { "slug": "api/overview", "title": "API overview", "description": "Base URL, content types, auth header, error shape, versioning rules, and capability discovery for the Amini Cloud HTTP API.", "section": "api", "body": "# API overview amini-admission exposes one HTTP API, prefixed `/v1/`, served by the chi router in [`internal/admission/server.go`](../../../internal/admission/server.go) plus the per-surface `Register*Routes` registrars it mounts. Every load-bearing tenant or operator action goes through this surface: - **Auth** — native email+password + magic-link (`/v1/auth/*`), wallet sign-in (`/v1/auth/wallet/*`), and the legacy OIDC browser flow (`/v1/auth/oidc/login`, `/v1/auth/exchange`). - **Slices** — CRUD + lifecycle, plus exec / logs / shell / models sub-resources. - **Apps** — the curated app catalog, publish, install, and instances (`/v1/admin/apps*`, `/v1/apps/schema`). - **Models** — the curated model catalog + Hugging Face search (`/v1/admin/models*`, `/v1/slices/{id}/models*`). - **Hosts** — public self-join + install-token onboarding, the approval queue, the removed-hosts blocklist, and host-agent system calls. - **Grants** and **Volumes** — tenant-scoped sharing + storage (`/v1/grants*`, `/v1/volumes*`). - **Tenants** (admin), **audit** reads, **SLA + KPI** reads, **geo** policy, **clusters**, **chain** (AminiChain), and the public-ingress routes for custom domains. This page is the orientation. Endpoint-by-endpoint detail lives in [`v1-reference.md`](v1-reference.md). The auth flow lives in [`authentication.md`](authentication.md). ## Base URL Single-host Compose default: `http://localhost:18080` (Compose intentionally avoids `:8080` to dodge collisions with ULAP-ONE / Beszel-style HTTP surfaces that commonly squat 8080-8090; override via `AMINI_ADMISSION_PORT` in [`deploy/compose/.env.example`](../../../deploy/compose/.env.example)). Older builds bound to `:8080` — if a legacy doc / environment cites that port, set `AMINI_ADMISSION_URL=http://localhost:18080` to match. Production: whatever your operator configured Caddy to terminate TLS on — typically `https://api.amini.example`. Health endpoints live at the root, off the `/v1/` prefix: | Path | Auth | Purpose | |---|---|---| | `GET /healthz` | none | liveness — always `{\"status\":\"ok\"}` if the process is up | | `GET /readyz` | none | readiness — pings Postgres; 503 if DB is down | Every other route requires an authenticated session (or, for host-agent system calls, a host bearer token). ## Content types - **Request:** `Content-Type: application/json; charset=utf-8`. Bodies are JSON. The slice spec is additionally JSON-Schema-validated against [`api/slice.schema.json`](../../../api/slice.schema.json) and rejected if it contains unknown fields (`additionalProperties: false`). - **Request body limit:** 64 KiB (`http.MaxBytesReader` in `internal/admission/server.go`). The slice spec is small by design; large payloads (datasets, images) go through SeaweedFS or your runtime registry, not admission. - **Response:** `Content-Type: application/json; charset=utf-8`. One endpoint (`GET /v1/sla/report?format=csv`) optionally returns `text/csv; charset=utf-8`. ## Auth As of [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md) the shipped default is **amini-native** auth, so authenticated requests carry a **session** minted by one of the in-process sign-in paths: - **Native email+password / magic-link** — `POST /v1/auth/login` (or the magic-link verify) sets an httpOnly session cookie. The SvelteKit UI rides the cookie; programmatic clients send the session token as `Authorization: Bearer `. - **Wallet** — `POST /v1/auth/wallet/verify` returns a session of the same shape (see [Wallet sign-in](../concepts/wallet-signin.md)). - **Legacy OIDC** — still supported: a Bearer ID token is verified per-tenant against the issuer named in `amini.tenants.oidc_issuer` (verifiers cached per issuer in `internal/auth/oidc.go`), or the browser PKCE flow exchanges via `POST /v1/auth/exchange`. Whichever path mints it, the result is the same `AuthContext` (tenant id + subject + role + email), so every `/v1/*` route behaves identically regardless of how the caller signed in. The " }, { "slug": "api/v1-reference", "title": "v1 API reference", "description": "Every endpoint amini-admission serves — method, path, auth, request, response, curl, error shapes. Slice CRUD, SLA, KPIs, uptime, public ingress, custom domains.", "section": "api", "body": "# v1 reference Every route on `amini-admission` is listed below. The source of truth is the chi router plus the per-surface `Register*Routes` registrars: - [`internal/admission/server.go`](../../../internal/admission/server.go) — slice CRUD, tenant CRUD, audit, `tenants/me`, `/v1/apps/schema` - [`internal/admission/auth_routes.go`](../../../internal/admission/auth_routes.go) — native email+password + magic-link - [`internal/admission/wallet_routes.go`](../../../internal/admission/wallet_routes.go) + [`oidc_routes.go`](../../../internal/admission/oidc_routes.go) — wallet sign-in, legacy OIDC / exchange - [`internal/admission/admin_apps_routes.go`](../../../internal/admission/admin_apps_routes.go) — apps catalog, publish, install, instances - [`internal/admission/admin_models_routes.go`](../../../internal/admission/admin_models_routes.go) + [`slice_models_routes.go`](../../../internal/admission/slice_models_routes.go) — curated model catalog - [`internal/admission/host_join_routes.go`](../../../internal/admission/host_join_routes.go) + [`admin_host_install_routes.go`](../../../internal/admission/admin_host_install_routes.go) + [`admin_host_lifecycle_routes.go`](../../../internal/admission/admin_host_lifecycle_routes.go) — host self-join, install-token, approval, blocklist - [`internal/admission/admin_misc_routes.go`](../../../internal/admission/admin_misc_routes.go) — grants, volumes, hosts list, capabilities - [`internal/admission/sla_routes.go`](../../../internal/admission/sla_routes.go) — SLA, KPI, uptime - [`internal/admission/ingress_routes.go`](../../../internal/admission/ingress_routes.go) — domains, public endpoints If anything below contradicts those files, the file wins and this page is a doc bug (see the accuracy contract in [`README.md`](../README.md)). Conventions: - Every example assumes `http://localhost:18080` (the single-host Compose default; see [`deploy/compose/.env.example`](../../../deploy/compose/.env.example) for the `AMINI_ADMISSION_PORT` override) and `TOKEN=`. Older builds bound admission to `:8080`; if you hit that in legacy docs or an old environment, set `AMINI_ADMISSION_URL` to match. - `` placeholders are UUIDs unless noted. - Cross-tenant access returns `404 not found`, not `403`. The same rule applies inside a tenant when the caller has no `slice_grants` row on the target slice. This is intentional (see [Authentication](authentication.md)). ## Health ### `GET /healthz` Liveness. Returns 200 if the process is up. ```bash curl -fsS http://localhost:18080/healthz # → {\"status\":\"ok\"} ``` No auth. No body. Always 200 if the process answers. ### `GET /readyz` Readiness. Pings Postgres with a 2s timeout. ```bash curl -fsS http://localhost:18080/readyz # → {\"status\":\"ready\"} (200) # or # → {\"error\":\"pg not ready: ...\"} (503) ``` No auth. ## Auth Native sign-in (ADR 0069) is served in-process. Emails (signup verify, password reset, magic-link) deliver via a Cloudflare Email Worker the operator configures (`AMINI_EMAIL_WORKER_URL` + `AMINI_EMAIL_WORKER_HMAC_KEY`); without it, signup/login still succeed but no email is delivered. All routes below are **public** (no Bearer) except `password/change`, which requires an active session. | Method | Path | Notes | |---|---|---| | POST | `/v1/auth/signup` | 201; creates a `member` `tenant_users` row (`auth_method='email-password'`) + emails a verify link. No-enumeration: an existing email returns 201 without re-creating. | | POST | `/v1/auth/login` | 200 + session cookie on valid email+password. | | POST | `/v1/auth/magic-link/request` | always 200 (no enumeration); emails a one-time login link. | | GET | `/v1/auth/magic-link/verify?token=…` | 302 → `/dashboard` + session cookie. | | POST | `/v1/auth/logout` | 200; clears the session cookie. | | POST | `/v1/auth/password/reset/request` | always 200 (no enumeration); emails a reset link. | | POST | `/v1/auth/password/reset/confirm` | 200; sets a new password from a reset token. | | POST | `/v1/auth/pas" }, { "slug": "beta-release-notes", "title": "Beta release notes", "description": "What works, what is planned, and what's deliberately incomplete in the Amini Cloud beta release (current tag target v0.2.0-beta — see RELEASE-NOTES-v0.2.0-beta.md at repo root for the v0.2 delta).", "section": "root", "body": "# Beta release notes This is the canonical \"is this feature real yet?\" page for the Amini Cloud beta. Read it before you stake any production decision on Amini Cloud, and read it again before you file a bug. Every line below is cross-checked against the code, the schema (`db/migrations/`), the feature catalog (`slate_features` seeded in migration 0005/0006), and the ADRs in `docs/decisions/`. If a claim here contradicts what Amini Cloud actually does in your install, that is a documentation bug and we want to hear about it. ## What Amini Cloud is in this beta Amini Cloud is a proprietary, self-hosted, multi-tenant private-cloud platform — © 2026 Amini Infra Tech, all rights reserved. Point it at a fleet of two or more compute hosts (one cheap public VPS for the control plane, one or more compute nodes that can sit behind CGNAT) and get back a system that subdivides into orgs, subdivides each org into users, and serves a curated catalog of one-click workloads (JupyterLab, VS Code, MLflow, vLLM, Postgres, MinIO, and more) over a per-tenant Tailscale-compatible private network. The full design is [`docs/whitepaper.md`](../whitepaper.md); the operator ladder is [`docs/user-journeys.md`](../user-journeys.md). This beta targets **levels 0 through 4** in the user-journey ladder: single-host dev, small team in one tenant, multi-org, multi-node pool, and the first untrusted-tenant deployment. ## What works These features are marked `stable` or `beta` in the `slate_features` table and have real code behind them. \"beta\" means the feature is wired end-to-end but has not been hardened against adversarial load yet. - **Single-host bring-up via Docker Compose.** `make up` brings Postgres, Authelia, amini-admission, Caddy, one Headscale, and one derper online on a single machine. See [`installation/single-host.md`](installation/single-host.md). - **Per-tenant Tailscale-via-Headscale overlay.** Default overlay provider. One Headscale 0.28.0 process per tenant, each with its own Postgres schema. See [`concepts/overlays-and-providers.md`](concepts/overlays-and-providers.md) and [ADR 0001](../decisions/0001-headscale-fleet-not-netbird.md). - **Per-tenant MagicDNS.** Slices reach each other as `..amini.ts.net` (or the operator-chosen base domain). - **Default-deny per-user `slice_grants` inside a tenant.** Every slice carries `owner_subject`; cross-user reach within the tenant requires an explicit grant of `read`, `use`, or `admin`. See [`concepts/tenants-and-users.md`](concepts/tenants-and-users.md) and [ADR 0007](../decisions/0007-per-user-isolation-within-tenant.md). - **Native email + password auth** ([ADR 0069](../decisions/0069-native-auth-replace-authelia.md)). Identity is amini's own `tenant_users.password_hash` — the default sign-in is native email + password, with magic-link via a Cloudflare Email Worker. The bundled Authelia/Dex OIDC IdP is no longer the default. An operator can still wire an external OIDC issuer through the pluggable-auth provider interface ([ADR 0013](../decisions/0013-pluggable-auth-providers.md)); per-tenant external issuers are configured via `tenants.oidc_issuer`. See [`api/authentication.md`](api/authentication.md). - **Service catalog.** Eight original MVP templates (`jupyter`, `jupyter-gpu`, `code-server`, `mlflow`, `vllm-inference`, `postgres`, `minio`, `custom-oci`) plus runtime-expansion templates (`vm-ubuntu`, `vm-cuda-workstation`, `k3s-single-node`, `kata-podman`, `k8s-deploy`) plus data-plane templates (`tabular-duckdb`, `tensor-zarr-server`, `multimodal-lance-server`, `stream-redpanda`). See [`tenant/using-templates.md`](tenant/using-templates.md). - **Six slice runtimes:** `podman` (trusted default), `firecracker` (untrusted-mandatory), `qemu`, `kata`, `nspawn`, `kubernetes`. Marked `stable` for `podman`, `beta` for the rest. See [`concepts/runtimes.md`](concepts/runtimes.md) and [ADR 0009](../decisions/0009-vm-container-k8s-as-first-class-runtimes.md). - **NVIDIA GPU support.** Compute " }, { "slug": "changelog", "title": "Changelog", "description": "Per-release notes for Amini Cloud; oldest at the bottom.", "section": "root", "body": "# Changelog Newest at the top. Every dated entry corresponds to a git tag in the Amini Cloud repository. The current build of amini-admission writes its component + semver into `amini.server_versions` on every start; `amini admin upgrade --check` (planned) will surface this history per deployment. The compatibility policy each release operates under lives in [ADR 0010](../decisions/0010-versioning-and-backward-compatibility.md). ## Unreleased — 2026-09-14: docs pass — the `AminiDev` bearer is off by default now, and API keys are the documented path - **Docs now match the post-F-001 auth posture.** After the `AminiDev` dev-bearer was found reachable on a shared dev stack and disabled there (security finding F-001), admission gates it behind a third env var, `AMINI_ENABLE_DEV_BEARER`, that `deploy/compose/.env.example` does **not** set — so it is off by default even on a fresh local `make up`. The tutorials and operator docs previously said two flags were enough and that Compose set them by default; both were wrong as of this pass. Fixed in the README auth caution, `docs/operators/two-ways-to-run.md` (new \"Getting a token to call the API\" section), and every in-scope tutorial / template page that used the bearer. - **The rest of `docs/` swept in the same pass.** Reader-facing curl recipes now authenticate with an API key (`Authorization: Bearer …`, or a `$TOKEN` set once per page) and link to the canonical section instead of restating it: the remaining 11 template tutorials, `guides/api/ authentication.md`, `guides/operator/{install,cluster-operations, acceptance-test-plan}.md`, `guides/user/reaching-your-slice.md`, `guides/tenant/known-gaps.md`, `operators/{run-live-end-to-end, app-bundles,multi-host-smoke,aminichain-oracle-bootstrap}.md`, `product/amini-product-spec.md`, `acceptance/v0.1.0-acceptance-plan.md` and the verification block in ADR 0093. - **Pages that document the bypass now name all three flags and the blast radius.** `operators/credentials-and-access.md` (which listed the bearer as one of four ways in, gated on the wrong flag, and did not mention API keys at all), `operators/{zero-touch-bringup, agent-onboarding,production-readiness,PROD-BRINGUP}.md`, `guides/concepts/node-enrollment.md` and `architecture/what-is-tested.md`. The hardening checklists mattered most: they told an operator that unsetting `AMINI_DEV_MODE` + `WALLET_DEV_AUTO_PROVISION` + `AUTO_PROVISION_USERS` closed the bearer, which is not what `devBearerEnabled()` reads. ADR 0084 kept its original text and gained a dated correction: the gate it names, `AMINI_DEV_BEARER_TENANT`, never shipped under that name. - **API keys documented as the bearer to use everywhere else.** `POST /v1/account/api-keys` mints one (secret shown once), `GET` lists, `DELETE /v1/account/api-keys/{id}` revokes — minting and revoking need a signed-in session (an API key can't manage API keys). No console page for keys yet; the path is login → mint → `Authorization: Bearer `. - **`AMINI_EXTERNAL_URL` and `AMINI_SLICE_BIND_ADDR` documented as two separate reachability gaps**, each with its exact failure text, in \"Adding a host (any OS / arch)\": the install one-liner dials the wrong box without the first; a healthy-looking slice is unreachable from ingress without the second (read only at container-create time — an already-running slice needs to be redeployed to pick up a fix). - **README gained an \"Is it fully up?\" checklist** (`make up-postflight` + `make doctor`, a host that's both `online` and `approved`, and a real slice URL returning 200 — reachable is not the same as serving). ## Unreleased — 2026-09-08: the host registry reads the chain's `found` bool instead of guessing - **`HostRegistry.Get` now learns \"no such host\" from the chain instead of inferring it.** The `0x1908` precompile's `get(bytes32)` returns **8** fields; the Go decoder declared **7**, stopping at `owner` and never reading the trailing `found` bool. Nothing broke, which is why it lasted: `found` is t" }, { "slug": "concepts/aminichain-integration", "title": "AminiChain integration (operator concept)", "description": "How Amini Cloud registers itself as an Oracle on AminiChain — three modes, how to turn it on, runtime flow, monitoring, security model.", "section": "concepts", "body": "# AminiChain integration Amini Cloud can register itself as an **Oracle** on [AminiChain](https://github.com/aminitech/aminichain) — a Cosmos SDK + Tendermint blockchain with EVM precompiles at `0x1900`-`0x1904` that orchestrates compute / storage / identity across a network of Oracles. When the integration is enabled, amini-admission watches for `JobCreated` events, races to ACK eligible jobs, provisions an Amini Cloud slice to run them, and submits the results back on-chain. This is the **operator-facing** concept page. The protocol detail lives in [`docs/integrations/aminichain.md`](../../integrations/aminichain.md); the architectural decisions live in [ADR 0014](../../decisions/0014-aminichain-integration.md). The precompile layout the engine targets (codename amini): | Precompile | Registry | Used for | |------------|------------------|-------------------------------------------| | `0x1900` | Oracle Registry | `registerOracle`, `heartbeat`, `updateOracleStatus` | | `0x1901` | Job Manager | `JobCreated` events, `ackJob`, `submitResult` | | `0x1902` | Compute Registry | `getCompute(regID)` — the per-job template | | `0x1903` | Content Registry | `registerContent` — dataset_version anchoring | | `0x1904` | MDI Registry | NFT-AIM, the user-side identity bridge | ## Three modes 1. **Amini Cloud beside a remote AminiChain.** You don't run an AminiChain node. `AMINICHAIN_RPC_URL` points at someone else's endpoint. Default for homelab / dev setups. 2. **Amini Cloud beside a colocated AminiChain validator.** You run an Amini validator as an Amini Cloud slice (template `aminichain-validator`, Year-1 — see [ADR 0014](../../decisions/0014-aminichain-integration.md)). amini-admission reaches it on the overlay tailnet. 3. **Amini-Cloud-only, no AminiChain.** `AMINICHAIN_ENABLED=false` (default). The code compiles in (the slice template `aminichain-oracle` still exists) but never runs. Operators in regulated environments can compile Amini Cloud without the `runtime_aminichain` build tag altogether to remove the EVM JSON-RPC client from the binary. ## Turning it on 0. **Build the AminiChain daemon** (only needed for modes 2 and the `aminichain-validator` slice template — mode 1 / 3 operators can skip this step). ```bash make build-aminichaind # produces build/aminichaind for linux/ # bootstrap run on a host with upstream access: make build-aminichaind UPDATE_CHECKSUMS=1 # populates the sha256 in deploy/bundle/amini-bundle.yaml ``` The script (`scripts/build-external-binaries.sh`) clones `github.com/aminitech/aminichain` at the SHA pinned in `deploy/bundle/amini-bundle.yaml` (today: `faeb8864`), runs the upstream's `make build`, and drops the produced binary at `build/aminichaind`. It also copies it into `dist/aminichaind--` so amini-admission's `/dist/` route can serve it to slices that need the daemon. The script is idempotent: a populated sha256 turns subsequent runs into a verify-then-skip. If the upstream repo isn't reachable from your account (private / auth-required), the script exits with a clear error rather than silently shipping a half-stack. 1. **Bootstrap vault.** AminiChain wallet keys live in OpenBao at `amini/aminichain/oracles//wallet.key`. They are never in env, never in config, never on disk outside vault. `make vault-bootstrap` creates the `amini-aminichain-oracle` AppRole with read access to that prefix (see `scripts/bootstrap-vault.sh`). 2. **Build amini-admission with the tag.** ```bash go build -tags runtime_aminichain -o bin/amini-admission ./cmd/amini-admission ``` Without `-tags runtime_aminichain`, the AminiChain code is not compiled into the binary and the integration is inert regardless of env config. The plain single-host compose ships without the tag; the **sovereign reference deployment** builds **with** it, so on that deployment the chain is on by default — the admission oracle self-provisions on a fresh vault and autofund grants it (and joining hosts) dev-net gas bef" }, { "slug": "concepts/auth-providers", "title": "Auth providers", "description": "Amini Cloud's authentication plane is **pluggable**. As of [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md), the **shipped default** is amini-native authentication served directly inside `amini-admission`: **email + password** (with **magic-link** as the passwordl", "section": "concepts", "body": "# Auth providers Amini Cloud's authentication plane is **pluggable**. As of [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md), the **shipped default** is amini-native authentication served directly inside `amini-admission`: **email + password** (with **magic-link** as the passwordless variant) and **wallet sign-in** (SIWE / Keplr). OIDC (Authelia, Dex, Keycloak) is now an **optional, legacy** provider — it still registers and still works, but a fresh install authenticates real users without standing up an OIDC issuer. The AminiChain user-NFT proof (\"NFT-AIM\") remains a registered-but-stubbed Year-1 provider. The registry is still pluggable: operators (or vendors) can register additional providers at startup, and every provider yields the same `AuthContext` so endpoints don't care which one fired. This is the concept page. The architectural decisions live in [ADR 0013](../../decisions/0013-pluggable-auth-providers.md) (the pluggable seam) and [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md) (native auth replaces Authelia as the default). ## The Provider interface Every auth provider implements: ```go type Provider interface { ID() ProviderID DisplayName() string Status() string // stable | beta | planned Match(req AuthRequest) bool // first match wins Authenticate(ctx, req) (AuthContext, error) } ``` The registry is built at startup. The middleware delegates to the first provider whose `Match` returns true; if none match, it falls back to the configured fallback path. The `AuthContext` shape (tenant id + subject + role + email) is **unchanged** so every existing endpoint keeps working. ## Built-in providers | Provider | Status | When it matches | |---|---|---| | `email-password` | beta (default) | native signup/login mints a session cookie; the session token resolves to a tenant_user | | `magic-link` | beta | passwordless email-link login; same session shape as `email-password` | | `wallet-evm` / `wallet-cosmos` | stable | SIWE / Keplr `signArbitrary` over a server-minted nonce (see [Wallet sign-in](wallet-signin.md)) | | `oidc-authelia` | stable (legacy) | request has `Authorization: Bearer `; the JWT's `iss` resolves to a tenant | | `oidc-dex` | beta (legacy) | same as above; differs in group / role claim handling | | `nft-aim` | planned (Year-1) | request has `Amini-NFTAIM-Token` or an NFT-AIM proof envelope | The native methods are seeded in `amini.auth_methods` by migrations 0060/0061 (`email-password`, `magic-link`); the OIDC + wallet rows arrive in migrations 0008 + 0018. The full catalog is surfaced by `GET /v1/capabilities` so clients can discover what THIS binary supports. The native HTTP surface lives at `/v1/auth/{signup,login,magic-link/request,magic-link/verify,logout,password/*}` (see [`internal/admission/auth_routes.go`](../../../internal/admission/auth_routes.go)) and `/v1/auth/wallet/{nonce,verify}`; the legacy OIDC browser flow is `/v1/auth/{oidc/login,exchange}`. ## How a tenant_users row picks its provider Each `tenant_users` row carries an `auth_method` column (migration 0008) naming the provider that authenticates it. The native signup path writes `auth_method='email-password'` on the row it inserts; wallet rows carry `wallet-evm` / `wallet-cosmos`; OIDC rows carry `oidc-authelia`. The column DEFAULT in the schema is still the historical `'oidc-authelia'` literal (migration 0008 set it and no later migration alters it), but native and wallet handlers set `auth_method` **explicitly** on insert, so the column default no longer reflects the shipped default auth path. Treat the value on the row — not the column default — as the source of truth for how a given user signs in. To flip a row to NFT-AIM (Year-1): ```bash amini admin tenant-user link-nft \\ --tenant=acme --subject=alice --nft-token-id=0x1234... ``` This calls `LinkTenantUserToNFT` on the Amini Cloud DAL, which sets `auth_method='nft-aim'` and `nft_token_id=...` on the row. The referenced subject keeps its OIDC sub" }, { "slug": "concepts/capability-discovery", "title": "Capability discovery", "description": "The slate_features catalog. How clients discover what this amini-admission supports, and how they react to missing features.", "section": "concepts", "body": "# Capability discovery Amini Cloud is meant to be installed once and stay installed for years. The binary you run today and the one a tenant runs three years from now will not have the same feature set. The mechanism that bridges that gap is the **`slate_features` catalog**: every capability the platform exposes is a row in that table, and the API publishes the catalog so clients can discover what THIS Amini Cloud supports. This page is the developer's reference. The schema is migration 0005; the matching ADR is [ADR 0010 — Versioning and backward compatibility](../../decisions/0010-versioning-and-backward-compatibility.md). ## The shape ```sql CREATE TABLE amini.slate_features ( id TEXT PRIMARY KEY, -- e.g. 'runtime_kubernetes', 'overlay_netbird' display_name TEXT NOT NULL, category TEXT NOT NULL CHECK (category IN ( 'runtime','overlay','storage','identity','observability', 'compute','networking','policy','workflow','catalog')), status TEXT NOT NULL CHECK (status IN ( 'planned','beta','stable','deprecated','removed')), introduced_in TEXT NOT NULL, -- semver of amini-admission when this landed deprecated_in TEXT, removal_date TIMESTAMPTZ, -- when it WILL be removed; NULL = no schedule replaces TEXT, -- id of feature this supersedes replaced_by TEXT, -- id of the successor docs_url TEXT, description TEXT NOT NULL ); ``` ## Status semantics | Status | What it means | |---|---| | `stable` | wired end-to-end; covered by tests; safe to depend on | | `beta` | wired end-to-end; not yet hardened against adversarial load | | `planned` | row exists; binary returns \"not implemented in this build\" or refuses spec features that need it | | `deprecated` | still works; `removal_date` set; switch off it | | `removed` | row stays for archeology; binary no longer ships this | A client should treat `stable` and `beta` as \"this works.\" `planned` means \"do not depend on this.\" `deprecated` means \"the clock is ticking — read `removal_date`.\" ## Reading the catalog Today the canonical read path is `psql` against `amini.slate_features`: ```bash psql \"$POSTGRES_DSN\" -c \" SELECT id, status, introduced_in, deprecated_in, removal_date, replaced_by FROM amini.slate_features ORDER BY category, id; \" ``` A Year-1 HTTP endpoint (`GET /v1/capabilities`) will return the same catalog as JSON; the route name lives in `docs/guides/README.md` and is scaffolded but not wired in this beta. The capability flag `workflow_self_update` covers this scaffolding. ## How clients should react The point of the catalog is so clients don't hardcode \"this Amini Cloud supports k8s dispatch.\" Three patterns: ### 1. Refuse to use a missing feature ```python features = read_features() if features.get('runtime_kubernetes') not in ('stable', 'beta'): raise CapabilityMissing( \"this amini doesn't ship runtime_kubernetes; \" \"either pick a different runtime or upgrade amini\") ``` ### 2. Degrade to a fallback ```python features = read_features() overlay = 'netbird' if features.get('overlay_netbird') in ('stable','beta') else 'tailscale' ``` ### 3. Refuse to start For an SDK that depends on a planned feature, refuse to initialise. Better to fail at import-time than to surprise the operator later. ## The MVP catalog at a glance The full set lives in migration 0005 + extensions in 0006, 0007, 0008, 0011, 0012, 0014, 0016. Status counts (as of this beta): | Category | Stable | Beta | Planned | |---|---|---|---| | runtime | 1 (podman) | 5 | 0 | | overlay | 1 (tailscale) | 0 | 3 | | storage | 0 | 2 | 1 | | storage (data plane) | 0 | 4 | 1 (`archive`) | | identity | 2 | 1 | 0 (NFT-AIM is registered as planned via the auth path) | | compute | 0 | 1 (NVIDIA) | 1 (AMD) | | networking | 2 (SSH, MagicDNS) | 0 | 1 (peer relays / Headscale v0.29) | | observability | 1 (audit) | 2 (Prom, Loki) | 0 | | policy | 1 (default-deny) | 0 | 1 (grants) | | workflow | 1 (reviewer pipeline) | 1 (self-update) | 0 | | catalog | 6 | 7 | 0 | A long-tail of additive rows (GPU KPIs, ingress, SLA reporting) lands via late" }, { "slug": "concepts/data-plane", "title": "Data plane", "description": "Datasets, versions, components, lineage. The typed data abstraction that sits above storage classes — with a multi-modal worked example.", "section": "concepts", "body": "# Data plane Amini Cloud's data plane is what makes \"ML/AI-friendly private cloud\" more than a marketing line. It is the layer that says **datasets are typed, versioned, lineage-tracked, and tenant-scoped**. Storage classes are the *backends*; datasets are the *interface tenants and slices reach for*. This page is the conceptual map. The rationale lives in [ADR 0011](../../decisions/0011-data-plane-beyond-buckets.md). The schema is migration 0006. ## The shape Four tables, each tied to the next: ```mermaid classDiagram datasets \"1\" --> \"N\" dataset_versions : has dataset_versions \"1\" --> \"N\" dataset_components : has dataset_versions \"*\" --> \"*\" dataset_versions : data_lineage
(derived_from / sampled_from /
merged_with / annotated_from, by slice) ``` *Four tables: dataset → versions → components, with a self-referential lineage edge through `data_lineage`.* | Table | What it holds | |---|---| | `datasets` | named, typed, tenant-scoped collection; one row per logical dataset | | `dataset_versions` | immutable snapshots; one row per `(dataset, version_label)` | | `dataset_components` | typed pieces of a version; one row per `(version, name)` | | `data_lineage` | \"version X was derived_from / sampled_from / merged_with / annotated_from version Y\" | The schema is in `db/migrations/0006_data_plane.sql`. ## A dataset ```sql SELECT id, name, primary_class, owner_subject, tags FROM amini.datasets WHERE tenant_id = '' ORDER BY name; ``` | Column | Notes | |---|---| | `id` UUID | platform-unique | | `tenant_id` UUID | the owning tenant | | `owner_subject` TEXT NULLABLE | NULL = tenant-shared; non-NULL = per-user | | `name` TEXT | `^[a-z][a-z0-9_-]{0,62}[a-z0-9]$`; unique per `(tenant, owner, name)` | | `primary_class` TEXT | references `storage_classes.id` | | `description` TEXT | free-form | | `tags` TEXT[] | for catalog UI filters | `primary_class` is the dominant storage class. A multimodal dataset can still have tensor + tabular + object components; `primary_class` is `multimodal_dataset` because that's how the catalog and the slice templates classify it. ## A version ```sql SELECT id, version_label, is_latest, created_by, size_bytes, object_count, checksum FROM amini.dataset_versions WHERE dataset_id = '' ORDER BY created_at DESC; ``` | Column | Notes | |---|---| | `version_label` TEXT | tenant picks; `1.0.0`, `2026-05-13`, `alpha`, anything | | `is_latest` BOOLEAN | exactly one row per dataset has `is_latest=TRUE` (enforced by partial unique index) | | `created_by` TEXT | OIDC subject or `system` | | `size_bytes`, `object_count` | aggregate across components | | `checksum` TEXT | whole-version Merkle root (optional) | | `metadata` JSONB | free-form (training run id, anchoring flag, …) | Versions are **immutable**. To change a dataset, create a new version. Lance handles this natively (branch + commit); Zarr v3 atomic writes on top of `object` are the Amini-Cloud-side enforcement. The `is_latest` flag is the convenience pointer for slice mounts that declare `version: \"latest\"` — it's atomically flipped on new version-create. ## A component ```sql SELECT name, storage_class, backend_ref, schema, metadata FROM amini.dataset_components WHERE dataset_version_id = ''; ``` | Column | Notes | |---|---| | `name` TEXT | within-version unique (`features`, `labels`, `audio`, `thumbnails`) | | `storage_class` TEXT | references `storage_classes.id` — one of seven | | `backend_ref` TEXT | the actual ref — `s3://...`, `postgres://...`, `kafka://...` | | `schema` JSONB | per-class typed schema (see below) | | `metadata` JSONB | free-form | Per-class schema shapes: - `tensor` — `{ \"dims\": [\"time\",\"band\",\"lat\",\"lon\"], \"dtype\": \"float32\", \"chunks\": [1,8,256,256] }` - `tabular` — `{ \"columns\": [{\"name\":\"ts\",\"type\":\"timestamptz\"}, ...] }` - `stream` — `{ \"topic\": \"...\", \"partition_count\": 12 }` - `multimodal_dataset` — Lance schema as JSON - `object` — `{}` or arbitrary ## Lineage Every produced version records its inputs: ``" }, { "slug": "concepts/geofencing-and-multi-network", "title": "Geofencing + multi-network placement", "description": "How Amini Cloud composes overlay-agnostic networking with three-layer geo policy so hosts can sit on any ISP / country while tenants enforce data-residency.", "section": "concepts", "body": "# Geofencing + multi-network placement > See also: [ADR 0029](../../decisions/0029-geo-policy-and-multi-network.md), > [migration 0020](../../../db/migrations/0020_geo_and_multi_network.sql), > [Concept — Placement policies](placement.md), [ADR 0004 — overlay > abstraction](../../decisions/0004-overlay-provider-abstraction.md). Amini Cloud hosts can live anywhere. The overlay (Tailscale + DERP fallback in the MVP, WireGuard/Netbird/Nebula via the [overlay-provider abstraction](../../decisions/0004-overlay-provider-abstraction.md)) stitches them into one /48-per-tenant network regardless of NAT, ISP, or continent. A Linux server in Limuru, a Mac mini at home in Edinburgh, and an `m6i.large` in `us-east-1` can all serve one tenant's slices and the workload doesn't know. That's the WHERE-PACKETS-FLOW story. The WHERE-WORKLOADS-LIVE story needs its own primitives — which is what migration 0020 + ADR 0029 introduce: - **Explicit geo + network identity on every host** (country, zone, jurisdiction tags, lat/lon, ISP). - **Three-layer geo policy** (cluster default → tenant default → slice override) that pre-filters the host inventory before normal placement scoring. The two layers compose orthogonally. A slice can REQUIRE the EU AND allow ANY overlay provider — geo decides which hosts are eligible, the overlay decides how their traffic reaches the user. ## The mental model ```mermaid flowchart LR subgraph Network[\"overlay layer (ADR 0004)\"] direction TB T[\"Tailscale headscale\"] W[\"WireGuard userspace\"] DERP[\"DERP fallback\"] end subgraph Geo[\"geo policy layer (ADR 0029)\"] direction TB C[\"cluster floor\\n(forbidden_*)\"] Td[\"tenant default\"] So[\"slice override\"] end Slice[\"Slice spec\"] --> Geo Geo --> Filter[\"eligible hosts\"] Filter --> Pick[\"scheduler.Place\"] Pick --> Host[\"chosen host\"] Host --> Network Network --> Reach[\"reachable by overlay v6\"] classDef layer fill:#0f172a,color:#fff,stroke:#475569 class Network,Geo layer ``` The two layers are **independent**. You can: - Run with NO geo policy at all (the default install). Every host is eligible. The overlay still does its job. - Run with strict geofencing AND swap overlay providers without changing anything in the geo layer. The slice's `geo_policy_json` doesn't reference the provider. - Run with geofencing AND multi-network spread to address the \"all my replicas on one ISP\" failure mode flagged in [ADR 0016](../../decisions/0016-redundancy-and-ha.md). The two placement-time hints (spread + max distance) layer on top of the eligibility filter. ## Host identity (migration 0020) Every `amini.hosts` row gains seven optional columns: | column | type | purpose | | -------------- | -------------- | --------------------------------------------------------- | | `country_code` | TEXT (2-char) | ISO 3166-1 alpha-2 — `KE`, `US`, `DE`, `SG` | | `geo_zone` | TEXT | coarse region — `eu-central`, `af-east`, `ap-southeast`, … | | `jurisdiction` | TEXT[] | compliance tags — `{GDPR, SOC2}` | | `lat` / `lon` | NUMERIC(9,6) | optional precise coordinates | | `network_isp` | TEXT | coarse ISP/ASN tag — `hetzner-fsn1`, `kplc-fiber` | | `public_ipv4` | INET | last-known public egress (NULL when CGNAT) | All seven are nullable. A host that never reports them shows up as \"unknown\" and simply doesn't match any `allowed_*` clause — it remains eligible only under empty policies. Operators tag hosts via the CLI: ```bash amini admin hosts set-geo node-23 \\ --country=DE \\ --zone=eu-central \\ --jurisdiction=GDPR,SOC2 \\ --isp=hetzner-fsn1 ``` Or via the HTTP route: ```http PUT /v1/admin/hosts/node-23/geo Content-Type: application/json { \"country_code\": \"DE\", \"geo_zone\": \"eu-central\", \"jurisdiction\": [\"GDPR\", \"SOC2\"], \"network_isp\": \"hetzner-fsn1\" } ``` The host-agent may also fill these in on heartbeat — see `internal/hostagent/` for the optional autodiscovery path. ## The policy document The policy JSON shape lives on three tables: | layer | column | who can set it | | --------- | ---------------------------------" }, { "slug": "concepts/host-classes", "title": "Host classes", "description": "The eight Amini Cloud compute classes — rpi, workstation, gaming-pc, mac, server, gpu-node, dgx-spark, edge. Capabilities, caveats, and onboarding.", "section": "concepts", "body": "# Host classes Amini Cloud runs on a deliberately heterogeneous fleet. The design conviction is that a tenant should be able to span \"a Raspberry Pi plus a Mac Studio plus a rack server plus a DGX Spark\" as one cluster, honestly — not by pretending the Pi is a server or the Mac is Linux. Every host declares a `compute_class` at registration; the scheduler reads it; each class has its own onboarding script. This page is the operator's taxonomy. The architectural decision is [ADR 0017](../../decisions/0017-heterogeneous-compute-classes.md). The schema is migration 0010 + 0011 (the CHECK constraint widened from six to eight values). ## The eight classes | `compute_class` | Hardware | OS | Onboarding script | Runtimes | |---|---|---|---|---| | `rpi` | Pi 4 / Pi 5 / arm64 SBC | Linux arm64 | `scripts/onboard-rpi.sh` | podman, nspawn | | `workstation` | Desktop 16-64 GB RAM | Linux x86_64/arm64 | `scripts/onboard-workstation-linux.sh` | podman, qemu, kata, nspawn, firecracker (if KVM) | | `gaming-pc` | Consumer-GPU desktop (RTX 4080/90/5090) | Linux only | `scripts/onboard-gaming-pc.sh` | podman, qemu, kata, firecracker | | `mac` | Apple Silicon Mac mini / Studio / MBP | macOS | `scripts/onboard-workstation-mac.sh` | podman (via Apple Virtualization) | | `server` | 1U/2U rack server | Linux | `scripts/onboard-server.sh` | all six | | `gpu-node` | Server with enterprise GPUs (A100/H100/L40S/MI300) | Linux | `scripts/onboard-gpu-node.sh` | all six; firecracker + qemu with vfio | | `dgx-spark` | NVIDIA DGX Spark FE (Grace-Blackwell aarch64) | Linux (DGX OS) | `scripts/onboard-dgx-spark.sh` | podman, qemu, kata, firecracker (KVM-arm) | | `edge` | Industrial PC / Mini PC / Jetson | Linux | `scripts/onboard-edge.sh` | podman, firecracker (if kernel supports) | The CHECK constraint on `amini.hosts.compute_class` accepts exactly these eight values. Adding a class is an additive migration (drop + re-add the CHECK with the new value); removing follows the one-release deprecation cycle. ## Three load-bearing caveats ### macOS hosts refuse untrusted-tenant slices Apple Silicon has no cgroups, no nftables, no KVM, and none of Amini Cloud's three VM-isolation runtimes (Firecracker, QEMU, Kata) ship a usable Apple Silicon path in this beta. So untrusted-tenant slices on `mac` hosts are refused outright at `PrepareSlice` time with `FailedPrecondition`. The macOS host-agent still works for **trusted** tenants — Podman via Apple Virtualization Framework. It just declines to host untrusted workloads. See [ADR 0017](../../decisions/0017-heterogeneous-compute-classes.md). ### gaming-pc hosts refuse untrusted-tenant slices Consumer GPUs (RTX 4090, 5090, etc.) lack the hardware multi-tenancy partitioning that datacenter GPUs (A100, H100, L40S) provide. Sharing a 4090 across two untrusted tenants is unsafe. So: - `gaming-pc` hosts decline untrusted-tenant slices at admission time. - WSL2 is **not** supported — Linux only. The Windows kernel doesn't give us the isolation primitives Amini Cloud needs. A `gaming-pc` host is great for trusted-tenant ML notebooks and gaming streaming workloads; not for hostile multi-tenancy. ### DGX Spark has no BMC NVIDIA DGX Spark Founders Edition (Grace-Blackwell aarch64) has **no BMC and no out-of-band management** (confirmed on the NVIDIA developer forum). Remote power-cycle requires an external smart plug (Shelly Plus 1PM, Tasmota) or a KVM dongle (JetKVM, PiKVM, GL.iNet Comet). The Amini Cloud host-agent's lifeline goroutine still works — what's missing is **out-of-band recovery when the OS is wedged**. Operators with a DGX Spark in production must wire: - A smart plug for remote AC cycle (Shelly / Tasmota). - BIOS \"always-on AC power recovery\" enabled. - systemd `Restart=always` on amini-host-agent. - NTP + RTC battery healthy (Grace's RTC behavior is sensitive). For the full ops model see the `dgx-spark-remote-ops` skill at `~/.claude/skills/dgx-spark-remote-ops/` (operator playbook for the remote DGX Spark scenario). " }, { "slug": "concepts/lifeline", "title": "Lifeline", "description": "The per-host lifeline goroutine, health-state classification, auto-drain, slice restart with backoff, and orphaned-resource GC.", "section": "concepts", "body": "# Lifeline Lifeline is Amini Cloud's self-healing layer. Every host runs a lifeline goroutine that samples health signals every 30 seconds, classifies the host into one of five states, and emits transitions over the audit stream. Admission watches those transitions and acts — auto-drain a sustained-degraded host, restart a flapping slice, GC orphaned netns and pf anchors. This page is the developer's reference. The decision lives in [ADR 0018](../../decisions/0018-lifeline-self-healing.md). The matching SLA / KPI subsystem (ADR 0020) **reads** lifeline events but doesn't write them. ## The five host states | State | Counts as up? | Scheduling | Meaning | |---|:---:|:---:|---| | `healthy` | ✓ | enabled | normal operations | | `warning` | ✓ | enabled | operator alerted; no scheduling change | | `degraded` | — | disabled | new slices won't schedule here; running slices keep running | | `draining` | — | disabled | slices being moved off | | `offline` | — | disabled | heartbeats missed; admission has given up | `healthy` and `warning` both count as \"up\" for uptime math (`internal/sla/uptime.go`). The lifeline doesn't punish operators for being honest about pre-failure signals. The full transition stream lives in `amini.host_uptime_events` (one row per state change with `started_at`, `ended_at`, `source`). ## Where the classifier lives `internal/hostagent/lifeline.go::classify` is a **pure function** on `HealthSignals` + thresholds. That's deliberate — the unit tests are trivially exhaustive (each case sets a single signal and asserts the bucket). Sampled signals include: - Free disk percentage (`AMINI_LIFELINE_DISK_FREE_PCT`, default 5). - Memory PSI pressure percentage (`AMINI_LIFELINE_MEM_PSI_PCT`, default 50). - Last heartbeat age (degraded if > 60s). - Lifeline interval (`AMINI_LIFELINE_INTERVAL`, default 30s). - Pre-drain dwell time (`AMINI_LIFELINE_DEGRADED_TO_DRAIN`, default 15min — degraded for this long → auto-drain). The sample function is **injected on construction** so tests don't have to read `/proc` or `psi` files. ## What the lifeline does NOT do The lifeline only **classifies and emits**. It does not make scheduling decisions. That's the admission-side sweeper's job. ## The admission-side sweeper `internal/admission/healthsweep.go` consumes `host.health` audit rows on a 30-second ticker: | Transition | Action | |---|---| | `healthy` | `hosts.status = 'online'` | | `warning` | log only | | `degraded` / `draining` | `hosts.status = 'draining'` (placer refuses new slices); emit `host.drained` | | `offline` | `hosts.status = 'offline'`; emit `host.offline` | In single-replica mode (the MVP), the sweeper just runs. In HA mode it gates on `Leader.IsLeader()` from the Postgres advisory-lock leader-election. ## Slice restart with backoff `internal/admission/provision.go::provisionOne` runs inside an exponential-backoff loop: ``` attempt 1 wait 1s retry attempt 2 wait 2s retry attempt 3 wait 4s retry attempt 4 wait 8s retry attempt 5 wait 16s retry (capped at 5 min for very-bad clusters) ``` Cap: 5 attempts inside a 10-minute window. Errors are classified: - **Retryable** — timeouts, Nomad alloc failures, host-agent unreachable, transient Postgres errors. - **Hard** — bad runtime (404 in driver), unsupported placement, schema-rejected spec. Hard errors short-circuit to `slice.status = failed`. This is **Amini Cloud's** retry loop, not Nomad's. Nomad has its own restart-policy block for in-place restarts; lifeline operates at the admission layer for cross-host reschedules and end-to-end correctness. ## Orphaned resource GC `internal/hostagent/gc.go` provides the cross-platform skeleton. Per platform: - **Linux** — `LinuxNetnsSweep` walks `/run/netns/amini-*` and reaps entries whose slice id is not in the host-agent's store AND whose mtime is older than 10 minutes. - **macOS** — `DarwinPfAnchorSweep` runs `pfctl -s Anchors -a amini` and flushes anchors not known to the store. GC runs on a 5-minute ticker. The interval and mini" }, { "slug": "concepts/node-enrollment", "title": "Node enrollment + the capacity market", "description": "How a runner (someone with spare compute) joins an Amini Cloud cluster they don't operate — the two-phase flow, the AminiChain anchor, and the homelab-vs-capacity-market choice.", "section": "concepts", "body": "# Node enrollment + the capacity market Amini Cloud is designed to grow horizontally. Anyone with spare compute should be able to add it to an Amini Cloud cluster with one paste: > The installer is served by the cluster you are joining, at > `/install.sh`. There is no `get.amini.sh`: that hostname > has never had a DNS record, so the command these docs used to print > failed for everyone who tried it. Use the URL of the cluster you were > given — the same one you pass to `amini join`. ```bash curl -fsSL https://acme-amini.example/install.sh | sh amini join https://acme-amini.example ``` This page is the concept doc for that flow. The full integration spec lives in [`docs/integrations/node-enrollment.md`](../../integrations/node-enrollment.md); the implementing rationale is in [ADR 0024](../../decisions/0024-node-enrollment-and-capacity-market.md); the schema is migration [`0017_node_enrollment.sql`](../../../db/migrations/0017_node_enrollment.sql); the runner walkthrough is [Tutorial 08](../tutorials/08-contribute-compute.md). ## Three actors Amini Cloud splits the trust boundary along three roles that have always been implicit in the platform; node enrollment is what makes them first-class: 1. **Cluster operator.** Runs amini-cp. Owns the policy. Decides who can contribute and on what terms. Today, in Level 0–2, the operator is also the runner — the same person who installs Amini Cloud is the same person who owns the hosts. 2. **Runner.** A human (or org) with a computer they want to contribute. Might be the operator (homelab), a friend with a spare GPU rig, or a stranger responding to a public capacity-market signal. The runner doesn't have an Amini Cloud tenant — their identity is a wallet address + a secp256k1 signature. 3. **Tenant.** Uses the cluster to run workloads. Doesn't care which physical host their slice lands on, and shouldn't. The Level 0–2 onboarding scripts under [`scripts/`](../../../scripts/) assume the runner IS the operator. Level 3+ — the case node enrollment unlocks — needs a real two-party flow. ## The two-phase flow ### Phase 1 — Enrollment request (runner) ``` runner: amini join https://acme-amini.example → generates a secp256k1 keypair locally → stores the private key at ~/.amini/runner/key (mode 0600) → fetches GET /v1/cluster/info — public; carries policy + cluster_id → detects host capacity (CPU, RAM, GPU) → signs a canonical challenge (cluster_id + nonce + ts + pubkey) → POSTs to /v1/node-enrollments ``` The request body carries the runner's pubkey, the EVM address derived from it, the host envelope, and a 65-byte secp256k1 signature over the challenge under the EIP-191 personal_sign envelope (the same envelope wallet sign-in uses — see [ADR 0025](../../decisions/0025-wallet-signin-agnostic-with-aminichain-background.md)). amini-admission's `internal/enrollment.VerifyPoP` recovers the signer address from the signature and compares it against the advertised `runner_address`. Mismatches return 401. Once verification passes, the policy gate runs (see [Policy evaluation](#policy-evaluation) below). Approved-by-policy requests are auto-approved immediately; others land in the operator's pending queue. ### Phase 2 — Cluster approval (operator) ``` operator: amini admin nodes pending → lists queued enrollments with hostname, capacity, time-since-submitted operator: amini admin nodes approve [--tags ...] → amini-admission: • flips node_enrollments.status = 'approved' • inserts a row in node_owners (the durable identity) • optionally calls AminiChain.RegisterOracle (settlement_mode='aminichain') • issues a pre-auth key for the cluster overlay • emits an audit event → the runner's host-agent picks it up (or `amini runner status` reports it) • joins the tailnet • starts heartbeating into admission • appears in `amini admin hosts ls` with compute_class + capacity ``` Approval is idempotent and the AminiChain bridge call happens **outside** the database transaction, so a chain-si" }, { "slug": "concepts/overlays-and-providers", "title": "Overlays and providers", "description": "The overlay.Provider interface, the four implementations, and how a tenant picks the right overlay backend.", "section": "concepts", "body": "# Overlays and providers Amini Cloud's wire layer between tenants and slices is the **overlay**. By default it's Tailscale via Headscale — one Headscale process per tenant, Tailscale clients on every device + slice. The product thesis is \"feels like Tailscale, runs on your hardware.\" But Tailscale-only would be a hole. Operators ask for bare WireGuard, NetBird, Nebula. So Amini Cloud's overlay layer is **pluggable** via the `overlay.Provider` interface ([ADR 0004](../../decisions/0004-overlay-provider-abstraction.md)). This page is the developer's reference. Implementations live in [`internal/overlay/`](../../../internal/overlay/) with one subpackage per provider. ## The Provider interface `internal/overlay/provider.go` declares: ```go type Provider interface { ID() ProviderID DisplayName() string Status() string // stable | beta | planned Capabilities() Capabilities EnsureTenant(ctx, tenant) error DeleteTenant(ctx, tenant) error IssueEnrollment(ctx, tenant, slice) (Enrollment, error) RevokeEnrollment(ctx, tenant, enrollmentID) error GetNodeIdentity(ctx, tenant, node) (NodeIdentity, error) DeleteNode(ctx, tenant, nodeID) error } ``` The interface owns everything Amini Cloud needs to: - create / delete a tenant's coordination state - mint a one-shot enrollment credential (pre-auth key) - look up a node's identity - revoke a node `Capabilities()` lets admission refuse spec features a provider does not support — for example, a tenant asking for grant-style policy on a provider that only does ACLs. ## The four providers | Provider | Status | Capabilities | |---|---|---| | `tailscale` | stable (MVP default) | full — ACLs, grants (via Headscale v0.29 when it lands), MagicDNS, DERP | | `wireguard` | planned (seam-stub) | minimal — Amini-Cloud-managed key + endpoint table | | `netbird` | planned (seam-stub) | full — Year-1 backup if Headscale upstream stalls | | `nebula` | planned (seam-stub) | cert-authority — coordination-server-free | Today only `tailscale` is wired end-to-end. The other three compile and register but return `ErrUnsupported` on non-trivial calls. The catalog flag in `amini.slate_features` reflects this: ``` overlay_tailscale stable overlay_wireguard planned overlay_netbird planned overlay_nebula planned ``` A `amini-admission` that registers but cannot run `netbird` logs a clear warning at startup so operators don't pick a stub by mistake. ## How a tenant picks a provider `amini.tenants.overlay_provider` (migration 0002) defaults to `'tailscale'`. Operators choose at tenant creation. Today the only defensible non-default choice is \"Don't deploy this tenant until the provider you want is real\" — which means: it's `tailscale` for now. Year-1 brings at least one non-default provider to \"real\" status — the gating commitment in ADR 0004. A per-slice override exists (`slices.overlay_provider`) but at MVP must equal the tenant's. The column is forward-compatible for the Year-1 cross-provider service-share. ## Why one Headscale per tenant? Amini Cloud's tenancy boundary is the tailnet ([ADR 0003](../../decisions/0003-tenant-equals-tailnet.md)). Each tenant gets its own: - Headscale OS process (`headscale@.service`) - Postgres schema (`headscale_`) - machine table — tenant A's `tailscaled` has no knowledge of tenant B's nodes - DERP map — no DERP server is shared across tenants in logs - ACL — HuJSON policy lives per-tenant This makes cross-tenant access **physically impossible** by default, not \"a policy bug away from a leak.\" A buggy amini-admission cannot cross-tenant-leak via shared overlay state because there is no shared overlay state. Cost: N processes for N tenants. At 50 tenants on a single control plane this is ~2.5 GB of Headscale RAM total + 50 systemd units; Amini Cloud accepts it. At 500 tenants the operational pattern needs revisiting (NetBird-style single-control-plane, or whatever upstream multi-tenant Headscale ships). ## The data plane (how packets actually move) - **Tenant devices** run the o" }, { "slug": "concepts/placement", "title": "Placement", "description": "The three placement policies — single-host, replicated, sharded — and the constraint vocabulary the scheduler honours.", "section": "concepts", "body": "# Placement A slice's `placement` block tells the scheduler **what shape to spread it in** and **what kinds of hosts can run it**. Three policies; one shared constraint vocabulary. The implementation is `internal/scheduler/` — a pure-function placer that the provisioner calls before submitting jobs to Nomad. This page is the concept map. The decision lives in [ADR 0015](../../decisions/0015-placement-policies.md). The matching ADR for heterogeneous hosts is [ADR 0017](../../decisions/0017-heterogeneous-compute-classes.md). ## The three policies | Policy | Slice rows | Use case | |---|---|---| | `single-host` (default) | 1 | The classic \"one slice on one host\" model | | `replicated` | N copies on **distinct** hosts | HA for stateless workloads (HTTP API, MLflow read replicas) | | `sharded` | N partitions sharing a `shard_set_id` | Embarrassingly-parallel ML training, batch processing | The block in the spec: ```json \"placement\": { \"policy\": \"replicated\", \"replicas\": 3, \"constraints\": { \"compute_class\": [\"server\",\"gpu-node\"], \"region\": \"nairobi\", \"spread_across\": \"rack\", \"anti_affinity_with\": [\"other-slice-name\"], \"labels\": { \"tier\": \"hot\" } } } ``` The block is **optional**. Absent → `single-host` with `replicas=1` and no constraints. Existing specs continue to work unchanged (backward-compat per [ADR 0010](../../decisions/0010-versioning-and-backward-compatibility.md)). ## `single-host` (the default) One slice row. Admission picks the best-fit host that satisfies constraints; the provisioner submits one Nomad job. This is what every single-host tutorial uses. ## `replicated` with `replicas: N` Admission creates **N child slice rows** sharing a `group_id`: - Names: `-r0`, `-r1`, …, `-r{N-1}`. - Env: each child gets `AMINI_REPLICA_INDEX` and `AMINI_REPLICA_COUNT`. - The scheduler forces **distinct hosts** — each replica lands on a different machine. - If `spread_across` is set (typically `rack` or `region`), the scheduler also spreads across distinct values of that host label. Quota: replicated multiplies the resource ask. Admission charges quota for all N children — the schema validates the parent ask first, then the (N-1) additional ask. Schema constraint: `replicas` must be ≥ 2 for `replicated`. Admission rejects with 400. ## `sharded` with `shards: N` Same shape as replicated, with two differences: - Child names: `-s0`, `-s1`, … - Env: `SHARD_INDEX`, `SHARD_COUNT`. - Multiple shards **may** stack on one beefy host — N shards are about parallelism, not redundancy. - Optional `shard_size` overrides the per-child resource ask (the default is the top-level `resources` block applied per shard). Schema constraint: `shards` must be ≥ 2 for `sharded`. Admission rejects with 400. ## The constraint vocabulary `placement.constraints` is honored by the scheduler regardless of policy. Five fields: | Field | Meaning | Nomad mapping | |---|---|---| | `compute_class` | array of allowed host classes | `${meta.slate_compute_class}` `set_contains_any` | | `region` | host's region must equal | `${meta.slate_region}` `=` | | `spread_across` | spread replicas across distinct values of this label | Nomad `spread` block | | `anti_affinity_with` | array of slice names to avoid co-locating with | per-sibling `${node.unique.id}` affinity weight `-100` | | `labels` | arbitrary k=v host labels | one constraint per pair | Anti-affinity is **soft** (Nomad `Affinity` with negative weight), so the scheduler can still place if the cluster runs hot. The single-host hard pin is `${node.unique.id} = ` — that one cannot be relaxed without intervention. ## Untrusted tenants on `mac` Per [ADR 0017](../../decisions/0017-heterogeneous-compute-classes.md), untrusted tenants cannot land on `mac` hosts. The scheduler enforces this in `filterHosts`: any candidate with `compute_class == \"mac\"` is dropped when `spec.Untrusted` is true. The caller sees `ErrInsufficientHosts` as a 503 from admission. The same logic applies to `gami" }, { "slug": "concepts/public-endpoints", "title": "Public endpoints", "description": "How Amini Cloud publishes tenant slices to the public internet — auto-generated hostnames, custom domains, automatic TLS, and the trust boundary that makes it safe.", "section": "concepts", "body": "# Public endpoints A **public endpoint** is the publicly-reachable HTTPS URL a tenant gets when a slice declares `exposure[].scope = \"public\"`. The Amini Cloud operator owns one wildcard domain; tenants get stable hostnames under it for free, and may optionally bring their own custom domains. TLS is automatic. Replicated slices ([ADR 0015](../../decisions/0015-placement-policies.md)) load-balance across healthy children with no operator config. This is the concept page. The architectural decisions live in [ADR 0023](../../decisions/0023-public-cloud-ingress.md). The runtime code lives in [`internal/ingress/`](../../../internal/ingress/) and the slice-lifecycle wiring is in [`internal/admission/ingress_wire.go`](../../../internal/admission/ingress_wire.go). The schema is migration 0016. ## TL;DR Declare `scope: \"public\"` on a slice exposure and Amini Cloud publishes the slice on `..` with automatic Let's Encrypt TLS; bring your own domain via `POST /v1/tenants/me/domains` and the same exposure picks it up once ownership is verified. ## What it is Two hostname shapes, both first-class: 1. **Auto-generated under the operator wildcard.** The operator's `platform_config.public_domain` (e.g. `cloud.example`) carries a wildcard `*.` A/AAAA record. Tenants get slice hostnames at `..` with zero DNS work on their side. 2. **Tenant custom domains.** A tenant registers their own FQDN (`api.acme.com`) via `POST /v1/tenants/me/domains`, proves ownership through a `dns-txt`, `http-01`, or `acme-dns-01` challenge, and then references that hostname from a slice's `exposure[].public_hostname`. **One hostname per exposed port.** A slice that declares more than one public exposure gets a **distinct public hostname per port**, not one host with multiple paths. The reconciler emits one `ingress.Plan` — and one published hostname — per (hostname, port) pair, and the slice-URL layer (`internal/admission/slice_urls.go`) renders one clickable URL per route. A `minio` slice exposing the S3 API (9000) and the console (9001), for instance, surfaces **two** separate public URLs. This is why each exposure may carry its own `public_hostname`: ports that need different names get them. Three TLS modes on the exposure (`tls.mode`): | Mode | What it means | |---|---| | `auto` (default) | Caddy issues + renews via Let's Encrypt. DNS-01 for the wildcard, HTTP-01 (or DNS-01 if configured) for custom domains. | | `manual` | Operator supplies a cert in Caddy's store; Amini Cloud does no issuance. | | `off` | HTTP only. Refused unless the operator has explicitly set `platform_config.allow_plaintext_public=true`. | The trust boundary is load-bearing: the public-ingress Caddy runs as the `caddy-ingress` system user — a different user from the control-plane Caddy's `caddy` — with no read access to admission's Postgres credentials and no permission to issue itself a cert for `admission.`. A compromised tenant slice that escapes its sandbox lands behind the public-ingress Caddy, not the admission one. The Caddy admin API is bound to `127.0.0.1:2019` only — never exposed on the overlay or the public interface. ## How it works A tenant slice declares one or more public exposures in its spec: ```json \"exposure\": [ { \"port\": 8888, \"proto\": \"tcp\", \"scope\": \"public\", \"public_hostname\": \"research\", \"tls\": { \"mode\": \"auto\", \"min_version\": \"1.3\" } } ] ``` The lifecycle wiring in [`internal/admission/ingress_wire.go`](../../../internal/admission/ingress_wire.go) runs a 30-second reconciler that: 1. Lists all slices in `status='running'` whose tenant has `metadata.public_ingress_enabled=true` and whose exposure array contains a `scope='public'` entry. 2. Builds an `ingress.Plan` per (group_id, hostname) pair — the single helper `parseIngressExposures` decodes the JSON (renamed from `parseExposures` to avoid colliding with the prober's helper of the same name). 3. Calls `Controller.Ap" }, { "slug": "concepts/runtimes", "title": "Runtimes", "description": "The six Amini Cloud runtimes — podman, firecracker, qemu, kata, nspawn, kubernetes — what they isolate, when to pick which, and the untrusted-tenant rules.", "section": "concepts", "body": "# Runtimes A slice runs in one of six runtimes. Amini Cloud is **multi-runtime from day one** ([ADR 0009](../../decisions/0009-vm-container-k8s-as-first-class-runtimes.md)) because \"container OR microVM\" is too narrow a default for a platform that hosts ML notebooks, full-VM workstations, untrusted tenants, and external Kubernetes dispatch. This page is the picker's guide. Implementation seams live in [`internal/runtime/`](../../../internal/runtime/) with one subpackage per driver; Nomad job templates for the Nomad-backed drivers live in [`deploy/nomad/`](../../../deploy/nomad/). The schema enum is in migration 0004. ## The six | Runtime | Isolation | Scheduler | Trusted tenant? | Untrusted tenant? | |---|---|---|:---:|:---:| | `podman` | Process (cgroups + userns, rootless) | Nomad | yes (default) | **no** | | `firecracker` | microVM (KVM) | Nomad | yes | **yes** | | `qemu` | Full VM (KVM, real kernel) | Nomad | yes | **yes** | | `kata` | OCI image wrapped in a microVM | Nomad | yes | **yes** | | `nspawn` | systemd-nspawn system container | Nomad | yes | **no** | | `kubernetes` | k8s namespace + NetworkPolicy + ResourceQuota | external k8s API | yes | yes, IF the target cluster's `runtimeClassName` is VM-isolated | The schema's CHECK (migration 0004) constrains `slices.runtime` to those six values. Adding a runtime is an additive migration that widens the constraint. ## Isolation levels in plain language Stronger is **lower in the table**. Amini Cloud's policy is \"pick the lightest runtime that satisfies your isolation needs\": - **Process** (podman, nspawn). The slice shares the host kernel. Linux namespaces (mount, pid, net, ipc, uts, user) and cgroups prevent escape **as long as the kernel is trusted-by-the-tenant**. - **microVM** (firecracker, kata). The slice has its own kernel. A VMM (Firecracker or Cloud Hypervisor) sits between the slice and the host. Cold-start is hundreds of milliseconds. - **Full VM** (qemu). Like microVM but with vfio passthrough, custom kernels, real PCI devices, and a wider hardware compatibility surface. Cold-start is seconds. - **External k8s** (kubernetes). Amini Cloud dispatches to a kube-apiserver you registered in `amini.clusters`. Isolation depends on the cluster's `runtimeClassName` (kata, gvisor, plain runc). ## Why six? Each fills a real gap: - **podman** — the trusted-tenant default. Rootless. Cold-start in hundreds of milliseconds. The shape every notebook / API / database template targets. - **firecracker** — the untrusted-tenant default. AWS Firecracker reading Nomad firecracker-task-driver. Mandatory for the `amini-aminichain` system tenant ([ADR 0014](../../decisions/0014-aminichain-integration.md)). - **qemu** — when a tenant says \"I want a real Linux box.\" Custom kernel. vfio GPU passthrough. `vm-cuda-workstation` template. - **kata** — OCI ergonomics with VM isolation. Drop-in for podman when paranoia is justified but rebuilding to a Firecracker rootfs is too much. - **nspawn** — systemd-nspawn. Niche. Trusted-only. We ship it because it's free (the host has systemd) and a few Year-1 use cases (slice templates that need a full PID 1) want it. - **kubernetes** — when the tenant already runs k8s. Amini Cloud creates a per-tenant namespace with NetworkPolicy + ResourceQuota; the slice becomes a Deployment or Job in that namespace. ## The untrusted gate (SEC-004) `tenants.untrusted = TRUE` triggers a runtime allow-list. Admission returns HTTP 422 if the spec violates it ([`internal/admission/server.go`](../../../internal/admission/server.go)): ```go if tenant.Untrusted { switch spec.Runtime { case \"firecracker\", \"qemu\", \"kata\": // VM isolation; OK case \"kubernetes\": // OK only if target cluster's // runtimeClassName is VM-isolated default: return 422 \"untrusted tenant must use a VM-isolation runtime\" } } ``` - `podman` and `nspawn` are refused at admission time. - `kubernetes` is admitted at the API; the provisioner checks `clusters.capabilities.runtime_class` at dispatch time and " }, { "slug": "concepts/slices", "title": "Slices", "description": "The Amini Cloud product unit — what a slice is, its lifecycle states, its identity, its lease.", "section": "concepts", "body": "# Slices A **slice** is Amini Cloud's product unit. It is one running workload owned by one tenant user, reachable on the tenant's overlay tailnet, with a declared lifetime, a measurable availability, and a clear teardown. Every other concept (placement, runtimes, storage, grants, SLA) is expressed in terms of slices. This page is the conceptual deep-dive. Lifecycle code lives in [`internal/admission/server.go`](../../../internal/admission/server.go) and the matching provisioner in [`internal/admission/provision.go`](../../../internal/admission/provision.go). The wire shape is [`api/slice.schema.json`](../../../api/slice.schema.json). The schema is migration 0001 + 0007 (security) + 0011 (compute_class) + 0013 (placement). ## The product object A slice is a row in `amini.slices`. Required columns (migration 0001): | Column | What | |---|---| | `id` UUID | platform-issued | | `tenant_id` UUID | the owning tenant | | `name` TEXT | unique within tenant; `^[a-z][a-z0-9-]{0,38}[a-z0-9]$` | | `owner_subject` TEXT | the `tenant_users.subject` of the creating user (provider-agnostic — see [Tenants and users](tenants-and-users.md)) | | `runtime` TEXT | one of six (see [Runtimes](runtimes.md)) | | `image` TEXT | OCI ref / rootfs ref | | `cpu_milli`, `mem_bytes`, `disk_bytes` | resources | | `ula_v6` CIDR | per-slice `/64` carved from the tenant's `/48` | | `status` TEXT | lifecycle (below) | | `lease_expires_at` TIMESTAMPTZ | when the slice gets garbage-collected | | `exposure` JSONB | declared ports | | `env` JSONB | environment injected at start | Migration 0013 added placement columns: `placement_policy`, `group_id`, `shard_index`, `shard_count`, `placement_constraints`. See [Placement](placement.md). ## Lifecycle states `status` is one of six values (CHECK constraint in migration 0001): ``` pending → scheduled → running → suspended → running \\→ terminated \\→ failed ``` | State | Meaning | |---|---| | `pending` | row accepted under the reservation lock; provisioner has not yet enqueued | | `scheduled` | Nomad alloc placed; host-agent not yet finished `PrepareSlice` | | `running` | overlay-reachable; prober sampling | | `suspended` | Nomad alloc stopped; state retained; prober idle | | `terminated` | irreversible teardown; row kept for audit | | `failed` | provisioning hit a hard error; retry path exhausted (5 attempts) | Create itself runs under a **reservation lock** (ADR 0063, closing \\#127): `POST /v1/slices` takes a per-host reserved-row lock and re-checks every capacity dimension (cpu/mem/gpu + the per-tenant slice count) against what is actually free *at insert time*. A create that passed the up-front pre-validation can still be **refused on this recheck** if a concurrent slice grabbed the capacity first — the response is `422` with a per-dimension `{detail, hint, field}` body (e.g. \"requested N cpu_milli; no eligible host has that much free right now\"), not a `pending` row. The lock is what makes the `pending → scheduled → running` walk safe under concurrent creates. Transitions are driven by: - HTTP routes (`POST /v1/slices/{id}/{suspend,resume,delete}` and `POST /v1/slices` itself). - The provisioner state machine ([`provision.go::provisionOne`](../../../internal/admission/provision.go)) walking `pending → scheduled → running` over Headscale + Nomad + host-agent. - The lifeline sweeper (`internal/admission/healthsweep.go`) marking failed slices when a host goes offline. Every transition writes an `audit_events` row (`action ='slice.create'` / `.renew` / `.suspend` / `.resume` / `.delete`). ## Identity A slice has three identities: 1. **`name`** — tenant-unique, DNS-safe, what humans type. MagicDNS resolves `..amini.ts.net` from inside the tenant tailnet. 2. **`id` (UUID)** — platform-unique, what every API path uses. 3. **`ula_v6`** — a stable `/64` carved from the tenant's `/48` at provision time. Unique per tenant (`slices_tenant_ula_unique` from migration 0007). Identity is **stable across restarts**. I" }, { "slug": "concepts/storage-classes", "title": "Storage classes", "description": "The typed storage classes — object, block, tabular, tensor, stream, archive, multimodal_dataset. Backends, lifecycle, and which are shipped vs roadmapped.", "section": "concepts", "body": "# Storage classes Amini Cloud is **opinionated about storage**. The schema defines seven typed storage classes (migration 0003 + 0006) so tenants get typed primitives that fit ML/AI and research workloads — not just \"drop a file in a bucket and hope.\" The discipline matters even when only a subset is fully wired today: the spec stays portable across the swap from MVP backend to Year-1 backend. The honest accounting of what ships end-to-end today is **`object` + `block`**; the other five are roadmapped (see the table below). This page is the menu. The rationale lives in [ADR 0011 (data plane beyond buckets)](../../decisions/0011-data-plane-beyond-buckets.md). The dataset abstraction that sits above storage classes lives in [Data plane](data-plane.md). ## The classes | Class | Backend (MVP) | Today | Roadmap backend | |---|---|---|---| | `object` | SeaweedFS / MinIO (S3-compatible) | **shipped** | unchanged | | `block` | `local-zfs` (single-host) | **shipped** | Longhorn (distributed) | | `tabular` | Postgres | `roadmap` | Iceberg over `object` | | `tensor` | Zarr over SeaweedFS | `roadmap` | TileDB | | `stream` | Redpanda (Kafka-compatible) | `roadmap` as a data-plane primitive; available today via the `stream-redpanda` slice template | unchanged; scale-out | | `archive` | SeaweedFS cold tier | `roadmap` | full | | `multimodal_dataset` | Lance | `roadmap` | deltalake | > **What \"shipped\" means here:** `object` and `block` have end-to-end > backend wiring, credential mint, `slice_mounts` resolution, and > per-class janitor in the host-agent. The `roadmap` rows exist in > `amini.storage_classes` and the schema accepts component references > against them today, but the per-class wiring is interface-only — use > `object` (S3 client) or `block` (filesystem mount) for production > workloads. For streams + tensors + multimodal today, use the > matching **slice template** (`stream-redpanda`, `tensor-zarr-server`, > `multimodal-lance-server`) which stands up the backing service > in-slice; the per-class data-plane primitive lands in a later wave. The catalog is in `amini.storage_classes`. The first two land via migration 0003; the other five via migration 0006. Read with: ```sql SELECT id, backend, description FROM amini.storage_classes ORDER BY id; ``` ## When to pick which ### `object` — generic blobs S3-compatible buckets via SeaweedFS. Tenants and users each get a namespace; amini-admission mints per-bucket credentials and injects them into the slice's env at provision time. The right choice for: - model checkpoints - OCI artifacts a slice produces - arbitrary files - anything an S3 SDK already fits Replicated across compute hosts by default (RF=2 in three-host cluster). NOT replicated off-cluster — that is the operator's backup job. See [Backup and restore](../operator/backup-and-restore.md). ### `block` — persistent slice-attached volumes Single-host block volumes via `local-zfs` zvols. The right choice for: - Postgres data directories - MinIO data directories - anything that wants a Linux filesystem mount **Pinning caveat:** MVP `block` volumes pin to the host they were created on. A slice mounting a local-zfs volume cannot be drained off that host. Year-1 brings Longhorn for distributed block. The volume row interface stays unchanged; the backend swap is invisible to tenants. See [Volumes and storage](../tenant/volumes-and-storage.md). ### `tabular` — rows + columns at SQL scale Postgres-backed by default. The right choice for: - structured data a slice queries with SQL - experimental result tables - labels for an image / audio dataset Year-1 wires Apache Iceberg over `object` for warehouse-scale datasets. The MVP shape (Postgres) is fine through tens of GB. ### `tensor` — N-dim arrays Zarr over SeaweedFS. The right choice for: - hyperspectral imagery - RF data cubes - neural-net weights as tensors (not as opaque blobs) - anything `xarray` opens with `xr.open_zarr` Components on a tensor dataset carry chunk + dtype + dims in the" }, { "slug": "concepts/tenants-and-users", "title": "Tenants and users", "description": "One Headscale per tenant, the role hierarchy, ULA /48 carving, and the per-user default-deny grant model.", "section": "concepts", "body": "# Tenants and users Amini Cloud has two sandboxing levels: **between tenants** and **between users inside one tenant**. The first is enforced by separate processes with disjoint state; the second is enforced by application-layer default-deny with an explicit grant table. Both boundaries return `404 not found` to violators — never `403`. This page is the conceptual map. The decisions live in [ADR 0003 (tenant = tailnet)](../../decisions/0003-tenant-equals-tailnet.md) and [ADR 0007 (per-user isolation)](../../decisions/0007-per-user-isolation-within-tenant.md). The schema is migration 0001 + 0003 + 0007 + 0008. ## A tenant A tenant is a row in `amini.tenants` (migration 0001): | Column | What | |---|---| | `slug` TEXT UNIQUE | `^[a-z][a-z0-9-]{1,30}[a-z0-9]$` — used in URLs, MagicDNS, schema names | | `display_name` TEXT | for the UI | | `oidc_issuer` TEXT | the IdP that authenticates this tenant's users | | `status` | `active` / `suspended` / `terminated` | | `untrusted` BOOLEAN | when TRUE, admission refuses non-VM runtimes (SEC-004) | | `headscale_url` TEXT | the public hostname of THIS tenant's Headscale | | `headscale_schema` TEXT | the Postgres schema name (convention: `headscale_`) | | `ipv6_ula_prefix` CIDR | the `/48` slices' `/64`s are carved from | | `cpu_milli_quota`, `mem_bytes_quota`, `slice_count_quota` | quotas | | `metadata` JSONB | free-form (SLA target, custom claims, etc.) | | `overlay_provider` | from migration 0002; default `tailscale` | **Each tenant is its own process boundary.** Tenant A's `tailscaled` does not know tenant B's Headscale exists, because they are different OS processes with different Postgres schemas. An admission bug cannot cross-tenant-leak because there is no shared object to leak through. Per-tenant resources: - **One Headscale process** (`/etc/systemd/system/headscale@.service`). - **One Postgres schema** (`headscale_`). - **One IPv6 ULA /48** (`fd5a:7e2:...:/48`). - **One Nomad namespace** with CPU + memory quotas. - **One MagicDNS namespace** (`..amini.ts.net`). - **One audit feed** (rows in `amini.audit_events` scoped by `tenant_id`). ## A user A user is a row in `amini.tenant_users` (migration 0001 + 0008): | Column | What | |---|---| | `tenant_id` UUID | composite key part | | `subject` TEXT | the user's identity within the tenant — composite key part | | `email` TEXT | for display | | `role` TEXT | `owner` / `admin` / `member` | | `auth_method` TEXT | which provider authenticates this user (`email-password` for native signups, `wallet-evm`/`wallet-cosmos` for wallets, `oidc-authelia` for legacy OIDC) | | `nft_token_id` TEXT | NFT-AIM linkage (Year-1) | `subject` is a **generic per-tenant identity string**, not tied to any one provider. For native email+password users it is the lowercased email; for wallet users it is the recovered wallet address; for legacy OIDC users it is the JWT `sub` claim. Which provider minted it is recorded in `auth_method` (see [Auth providers](auth-providers.md)). The primary key is `(tenant_id, subject)`. A subject can exist in multiple tenants — they are different users with different identities (Amini Cloud doesn't try to unify identities across tenants). ### Roles Three roles, hierarchical: | Role | Conferred capabilities | |---|---| | `member` | own own slices; share via grants; consume tenant resources | | `admin` | everything `member` does, plus list/get/modify any slice in the tenant | | `owner` | currently identical to `admin` for Amini Cloud's own checks; reserved for \"tenant lifecycle\" actions in Year-1 | The role is not the same as \"tenant admin can reach all slices in network terms.\" The overlay tailnet has its own ACL; the role gates amini-API operations, not packet flow. Admin tenant note: a small set of routes (`POST /v1/tenants`, `/v1/kpi`, `/v1/uptime/hosts`, cross-tenant SLA reads) additionally require the caller to be on the **admin tenant** (default slug `admin`). See [Authentication](../api/authenti" }, { "slug": "concepts/uptime-and-sla", "title": "Uptime, KPIs, and SLA reporting", "description": "Amini Cloud measures and reports its own reliability. This document is the single place that explains, in plain language, **what** is measured, **how** it's measured, and **how operators and tenants consume it**.", "section": "concepts", "body": "# Uptime, KPIs, and SLA reporting Amini Cloud measures and reports its own reliability. This document is the single place that explains, in plain language, **what** is measured, **how** it's measured, and **how operators and tenants consume it**. The architectural decisions live in [ADR 0020](../../decisions/0020-uptime-kpi-sla.md). The spec is in [`docs/integrations/cluster-fabric.md`](../../integrations/cluster-fabric.md), \"Uptime, KPIs, and SLA reporting\". ## TL;DR - Every host-state transition lands in `amini.host_uptime_events`. - A reachability prober inside admission writes a row to `amini.slice_availability_samples` every 30s per running slice. - Operators see eight platform KPIs in `amini admin kpi list` and the cluster Grafana dashboard. - Tenants see their own SLA scorecard via `GET /v1/sla/report/me` and the tenant Grafana dashboard. - When a tenant's SLA is missed, Amini Cloud **reports** the miss. The operator decides whether to **pay** a credit. Amini Cloud never moves money. ## What gets measured ### Host uptime Every host emits or receives state transitions, recorded in `amini.host_uptime_events`. The host states (from [ADR 0018](../../decisions/0018-lifeline-self-healing.md) when WS-I lands; here for context): | State | Counts as up? | Meaning | |---|:---:|---| | `healthy` | ✓ | Normal operations | | `warning` | ✓ | Operator alerted; no scheduling change | | `degraded` | — | Scheduling paused on this host | | `draining` | — | Slices being moved off | | `offline` | — | Heartbeats missed; admission has given up | Uptime fraction is computed by `internal/sla.IntegrateUptime`: ``` upDuration = sum of clipped durations where state ∈ {healthy, warning} fraction = upDuration / windowDuration ``` Open events (the most recent state, `ended_at IS NULL`) are treated as ending at the moment of the integration call. ### Slice availability A reachability prober (`internal/admission/prober.go`) runs as a goroutine inside amini-admission. Every 30 seconds it: 1. Lists all slices in `status='running'`. 2. For each, looks up the overlay v4/v6 address. 3. For each declared exposure port, does a TCP connect plus (for likely-HTTP ports) an HTTP HEAD. 4. Writes one row to `amini.slice_availability_samples`: `reachable bool, latency_ms, error_kind`. What counts as **reachable**: - TCP connect succeeded within the timeout. - For HTTP probes: any 1xx-5xx response (the slice answered). 401/403 count as reachable too — auth is the tenant's posture. What counts as **unreachable**: dial-timeout, connection refused, no route, no DNS, HTTP client error other than timeout. What is NEVER logged: slice response bodies. The prober is HEAD-only. `error_kind` is a coarse category. ### Tenant SLA Composed by `internal/sla.Reporter.GenerateReport`: 1. List all slices the tenant had during the window. 2. For each, compute its availability fraction. 3. Aggregate: `observed = mean(per-slice availability)`. Slices with zero samples are excluded from the mean (but listed in the report). 4. `met = observed >= target`. 5. Persist the report; return the in-memory struct. The target comes from `tenants.metadata.sla.availability_target` if set, else `sla.DefaultTarget` (0.99 — the \"standard\" tier). ## The eight default KPIs | ID | Source | Default target | Unit | |---|---|---|---| | `cluster_uptime` | mean of per-host uptime | 99.5% / 30d | fraction | | `slice_availability_p95` | p95 across slices | 99% / 30d | fraction | | `provisioning_latency_p95` | admission timing | < 60s | seconds | | `mttr_host` | lifeline events | < 10 min | seconds | | `mtbf_host` | lifeline events | > 30d | seconds | | `audit_integrity` | audit-chain consistency | 100% | fraction | | `backup_success` | backup script exit codes | 100% / week | fraction | | `reviewer_pass_rate` | docs/audits/ | 100% | fraction | `amini admin kpi list` prints the catalog directly from the static array in `internal/sla/kpi.go`. `amini admin kpi show --since=30d` returns recent snapshots from `ami" }, { "slug": "concepts/vault", "title": "Vault (OpenBao)", "description": "Amini Cloud uses **OpenBao** — the LF-sponsored open-source fork of HashiCorp Vault — as the single source of truth for every long-lived secret in the platform: the Postgres password, the Headscale admin token, OIDC client secrets, mTLS certificates, and (when the AminiChain inte", "section": "concepts", "body": "# Vault (OpenBao) Amini Cloud uses **OpenBao** — the LF-sponsored open-source fork of HashiCorp Vault — as the single source of truth for every long-lived secret in the platform: the Postgres password, the Headscale admin token, OIDC client secrets, mTLS certificates, and (when the AminiChain integration is enabled) Oracle wallet private keys. This is the concept page. The architectural decisions live in [ADR 0012](../../decisions/0012-vault-manager.md). ## TL;DR ``` make up # brings up postgres + authelia + amini + ... + vault make vault-bootstrap # initializes vault, unseals, writes policies + roles # operator wires the printed role-id + secret-id into amini-admission env ``` After bootstrap, amini-admission authenticates to vault via AppRole at startup and pulls its secrets just-in-time. ## Why this matters Pre-OpenBao, every long-lived secret lived in `deploy/compose/secrets/` as a file or was passed through env. That works for one operator on one box. It doesn't work for: - Multi-host operation (every host had to bootstrap its own files). - Secret rotation without restarting admission. - Per-service least privilege (admission and host-agent had identical read access to the secrets directory). - The AminiChain integration's EVM wallet keys — putting those on disk is a regression on the SEC-001 audit fix. OpenBao removes all four problems at once. ## What Amini Cloud stores in vault Every Amini Cloud secret lives under a documented prefix. The full table is in ADR 0012; the load-bearing ones are: | Path | What | |---|---| | `amini/postgres/dsn` | full pgx DSN (DEPRECATED for `*_FILE` once vault is in place) | | `amini/headscale/admin-token` | Headscale admin API token | | `amini/oidc//client-secret` | per-OIDC-provider OAuth secret | | `amini/mtls/admission/*` | mTLS material for the admission HTTP server | | `amini/aminichain/oracles//wallet.key` | EVM private key (never on disk) | Adding new prefixes is fine and is a normal additive change per ADR 0010 (versioning + backward compatibility). ## Bootstrapping the vault `make vault-bootstrap` runs `scripts/bootstrap-vault.sh`, which: 1. Waits for vault to be reachable on `$VAULT_ADDR`. 2. Initializes if not yet initialized (`bao operator init`), persisting shamir keys + root token to `deploy/compose/secrets/vault-keys.json` (mode 0600). 3. Unseals if sealed. 4. Enables KV v2 at `secret/`. 5. Enables AppRole auth. 6. Writes the three Amini Cloud policies from `deploy/vault/policies/`. 7. Creates the three AppRoles (`amini-admission`, `amini-host-agent`, `amini-aminichain-oracle`). The script is idempotent — running it again only re-prints the role-id + secret-id pairs. ### Dev vs production unseal The dev bootstrap auto-persists the shamir keys + root token to `deploy/compose/secrets/vault-keys.json` for convenience. **Production operators must not use this path** — see ADR 0012 for the production unseal story (cloud KMS auto-unseal, or shamir shares distributed out-of-band). ## How Amini Cloud authenticates amini-admission and amini-host-agent **never** use the root token. They authenticate via AppRole: ```yaml environment: VAULT_ADDR: http://vault:8200 VAULT_ROLE_ID_FILE: /run/secrets/vault-role-id VAULT_SECRET_ID_FILE: /run/secrets/vault-secret-id ``` The role-id is treated as service identity; the secret-id is the short-lived credential. Both are emitted at the end of the bootstrap run; the operator wires them in via the same Docker-secrets pattern that SEC-001 introduced for Postgres credentials. ## Rotation | Secret | How | |---|---| | OIDC client secrets | rotate the value in vault + bounce admission | | Postgres DSN | rotate via the operator runbook (Year-1: `amini admin rotate-pg-credential`) | | Pre-auth keys | TTL 5m by design; no manual rotation | | AminiChain wallet | `amini admin aminichain rotate-oracle-wallet --oracle-id=` (Year-1 — the SQL + vault paths are already in place) | ## Failure modes - **Vault sealed.** amini-admission retur" }, { "slug": "concepts/wallet-signin", "title": "Wallet sign-in", "description": "Wallet sign-in is one of Amini Cloud's two **co-primary** auth paths. Since [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md) flipped the default to amini-native auth, the shipped surface is **native email+password (+ magic-link)** and **wallet sign-in** — both ser", "section": "concepts", "body": "# Wallet sign-in Wallet sign-in is one of Amini Cloud's two **co-primary** auth paths. Since [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md) flipped the default to amini-native auth, the shipped surface is **native email+password (+ magic-link)** and **wallet sign-in** — both served directly inside `amini-admission`. OIDC (Authelia, Dex, Keycloak) is the legacy/optional path; the NFT-AIM provider remains a Year-1 stub. Wallet sign-in exists for users whose primary identity lives in a wallet — MetaMask, Keplr, Rabby, Frame, hardware-wallets behind any of these — and who do not want a Amini-Cloud-specific identity ceremony to sign in. This is the concept page. The decision lives in [ADR 0025](../../decisions/0025-wallet-signin-agnostic-with-aminichain-background.md); the auth-default flip is [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md). ## When to use it Wallet sign-in is the right choice when: - Your tenants are wallet-native and already manage keys (capacity- market runners, oracle operators, on-chain public-tenant administrators, hackathon teams). - You don't want to stand up an OIDC issuer for a tenant — the wallet-signed nonce IS the identity proof. - You operate in an ecosystem where the wallet address is already the canonical identifier in audit logs and on-chain receipts; reusing it inside Amini Cloud keeps the cross-system audit story coherent. It is NOT the right choice when: - You want the simplest onboarding for ordinary human users. The native **email + password** path (the other co-primary) needs no wallet and no OIDC issuer. - Your tenant is an organisation that already has SSO with Authelia, Dex, Keycloak, or any other OIDC issuer. The legacy OIDC path reuses it. - You require email-bound recovery flows. A wallet has no `email` claim; if the user loses the private key, Amini Cloud has no recovery path on its own. (Native email+password / magic-link does carry email-bound recovery.) - You require a group/role claim from a centrally-managed directory. Wallet sign-in produces a `tenant_users.role` that the operator manages at the Amini Cloud level. ## The two signature schemes Amini Cloud accepts two open-standard signature schemes under the same user-facing surface. The tenant picks whichever their wallet supports; the operator picks which schemes to enable at the Amini Cloud level. ### EVM (`wallet-evm`) — EIP-4361 / Sign-In-with-Ethereum The dominant signing scheme for any EVM-compatible wallet (MetaMask, Rabby, Frame, hardware wallets via MetaMask). The client: 1. Calls `POST /v1/auth/wallet/nonce` with `{ wallet_address, chain: \"evm\" }`. 2. Receives a `nonce` (32-byte hex) and `expires_at` (5 minutes ahead). 3. Builds an EIP-4361 message that includes the nonce. The simplest form is the nonce itself; the long form is the canonical SIWE message (see `walletprovider.BuildSIWEMessage`). 4. Asks the wallet to `personal_sign` the message. MetaMask shows a readable confirmation prompt. 5. POSTs `{ wallet_address, chain: \"evm\", nonce, signature }` to `/v1/auth/wallet/verify`. Amini Cloud recovers the signer address from the signature and compares to the claimed `wallet_address`. The recovered address (never the claimed address) is what Amini Cloud trusts. ### Cosmos (`wallet-cosmos`) — ADR-036 / Keplr signArbitrary The Cosmos ecosystem's equivalent. Keplr, Leap, and Cosmostation all support it under the name `signArbitrary`. The wire shape is identical except: - The signature scheme is secp256k1 over SHA-256 of a canonical amino-JSON sign doc. - The Amini Cloud wire form glues the compressed pubkey to the (r||s) signature with a colon: ``` signature = base64(pubkey_compressed) + \":\" + base64(r||s) ``` - The recovered address is bech32-encoded under the `cosmos` HRP. If your chain uses a different HRP (`osmo`, `juno`, etc), the operator-side configuration accepts the same bech32 prefix. ## The flow ```mermaid sequenceDiagram autonumber participant T as Tenant (UI/CLI) participant A " }, { "slug": "h200-bringup", "title": "H200 bring-up — serve models on `bgi-ran-01` (no tailnet)", "description": "Box: Supermicro SYS-222HE-FTN, Rocky Linux 9.7, x86-64, **1× H200 NVL (141 GB)**, CUDA 13.1. Not on the tailnet → serve locally, reach it over an SSH tunnel (private, works today).", "section": "root", "body": "# H200 bring-up — serve models on `bgi-ran-01` (no tailnet) Box: Supermicro SYS-222HE-FTN, Rocky Linux 9.7, x86-64, **1× H200 NVL (141 GB)**, CUDA 13.1. Not on the tailnet → serve locally, reach it over an SSH tunnel (private, works today). ## A. Check the box ```bash nvidia-smi --query-gpu=name,memory.total --format=csv command -v docker || command -v podman docker info 2>/dev/null | grep -i runtime ``` ## B. Fastest path — ollama native ```bash curl -fsSL https://ollama.com/install.sh | sh # installs a systemd service, auto-uses the H200 ollama pull qwen2.5-coder:32b # coding (daily driver) ollama pull qwen2.5:72b # strong general/chat ollama pull qwen2.5vl:32b # vision curl -s http://localhost:11434/v1/chat/completions -H 'Content-Type: application/json' \\ -d '{\"model\":\"qwen2.5-coder:32b\",\"messages\":[{\"role\":\"user\",\"content\":\"say H200-OK\"}]}' ``` ## C. Throughput path — vLLM (docker, OpenAI endpoint on :8000) ```bash # one-time: NVIDIA container toolkit curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo sudo dnf install -y nvidia-container-toolkit && sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker docker run -d --name vllm-coder --gpus all -p 8000:8000 \\ -v ~/.cache/huggingface:/root/.cache/huggingface \\ vllm/vllm-openai:latest \\ --model Qwen/Qwen2.5-Coder-32B-Instruct --served-model-name qwen2.5-coder-32b \\ --gpu-memory-utilization 0.90 --max-model-len 32768 curl -s http://localhost:8000/v1/models ``` ## D. Reach it — access overlays (Spark = Tailscale, H200 = Twingate) The H200 is reached via **Twingate** (the Spark via Tailscale — two separate overlays; that's fine). With the Twingate client connected on your Mac, the H200's private IP is directly reachable, so bind the server to all interfaces and hit it over Twingate: ```bash # On the H200 — bind ollama to all interfaces so Twingate clients can reach it: sudo mkdir -p /etc/systemd/system/ollama.service.d printf '[Service]\\nEnvironment=\"OLLAMA_HOST=0.0.0.0:11434\"\\n' | sudo tee /etc/systemd/system/ollama.service.d/override.conf sudo systemctl daemon-reload && sudo systemctl restart ollama # (vLLM already binds 0.0.0.0 via -p 8000:8000) ``` ```bash # On the Mac (Twingate client connected) — use the H200's private IP: export OPENAI_BASE_URL=\"http://:11434/v1\" # or :8000/v1 for vLLM export OPENAI_API_KEY=\"ollama\" curl -s http://:11434/v1/models ``` - **Keep the Twingate Resource private** — only your enrolled devices should resolve/reach the H200. Do NOT also expose these ports on a public interface/firewall. - **Fallback if Twingate isn't up**: `ssh -N -L 11434:localhost:11434 USER@H200` then use `http://localhost:11434/v1` on the Mac. - **Two overlays note**: Twingate gives *you* access to the H200; amini's host-agent federation expects **Tailscale**. To pull the H200 into Amini Cloud, run Tailscale on it too (Tailscale + Twingate coexist) — see §E — or leave it standalone and let the keyed gateway (`deploy/inference-gateway/`) reach it over Twingate. ## E. Make it part of Amini Cloud (recommended, when ready) ```bash # join the tailnet so amini can federate + route to it: curl -fsSL https://tailscale.com/install.sh | sh sudo tailscale up # authenticate to your Headscale/tailnet # then register it as a amini host from the console: /admin/hosts/new ``` ## Right-sizing (1× H200, 141 GB) — honest - Fits well: qwen2.5-coder:32b, qwen2.5:72b (fp8), llama3.3:70b, deepseek-r1:70b, qwen2.5vl:32b. - Does **not** fit one H200: GLM-4.6 (needs 2× NVL), DeepSeek-V3 (needs 8×), **GLM-5.2** (needs ~8×). - Add the 2nd H200 NVL (→282 GB, NVLink) to serve GLM-4.6 (4-bit) / Qwen3-235B (fp8). ## Security Never publish raw ollama/vLLM to the internet (no auth). Keep it private via SSH tunnel or the tailnet; for a shared/public endpoint, put it behind the keyed gateway (`deploy/inference-gateway/`) + Cloudflare Tunnel. " }, { "slug": "host-agent-install", "title": "Installing `amini-host-agent` on Linux", "description": "`amini-host-agent` is the per-host gRPC daemon that advertises capacity to an Amini Cloud control-plane. It runs on **any flavour of Linux** — Ubuntu, Debian, RHEL/Fedora, Arch, Alpine, openSUSE, Void, and more — and on macOS via launchd. The installer at `scripts/install-host-ag", "section": "root", "body": "# Installing `amini-host-agent` on Linux `amini-host-agent` is the per-host gRPC daemon that advertises capacity to an Amini Cloud control-plane. It runs on **any flavour of Linux** — Ubuntu, Debian, RHEL/Fedora, Arch, Alpine, openSUSE, Void, and more — and on macOS via launchd. The installer at `scripts/install-host-agent.sh` auto-detects the distro, init system, and libc, then installs the matching service template. ## Or: use the dashboard wizard at `/admin/hosts/new` If you have admin access to the Amini Cloud dashboard, the 3-step host- onboarding wizard at **Admin → Compute hosts → Add host** ships the fast path: pick an OS + cluster, paste the generated one-liner, and the wizard polls for the host to come online. See [Adding hosts](operator/adding-hosts.md) for the full walkthrough. The wizard is a thin wrapper over the script-mode flow described below — under the hood it generates a 10-minute install token, calls `scripts/install-host-agent.sh` with the right env, and registers with the same `/v1/hosts/join` endpoint. If your network restricts outbound HTTPS from new hosts (or if you onboard via Ansible / a config-management tool), prefer the script flow below. ## Quick start ```bash make host-agent-static # build static linux binaries sudo bash scripts/install-host-agent.sh \\ --admission-addr https://amini.example.com ``` The installer detects everything else and starts the daemon. See [`deploy/host-agent/SUPPORTED.md`](../../deploy/host-agent/SUPPORTED.md) for the full distro/init/libc matrix. > [!info] **What URL goes after `--admission-addr` / in the wizard > one-liner?** It's whatever the control-plane operator set > `AMINI_INGRESS_BASE_DOMAIN` to in `deploy/compose/.env`. On a public > deploy behind a Cloudflare Tunnel (the recommended path — see > [`docs/operators/cloudflare-tunnel.md`](../operators/cloudflare-tunnel.md)) > that's a clean public hostname: > `https://amini.ai/install/`. On the laptop / loopback > dev path it's the alt-HTTPS-port form > `https://amini.localtest.me:8446/install/`. The install one- > liner emitted by `make host-link` and the Admin → Add host wizard > always uses whatever the control plane has configured — paste it > as-is. ### Who needs `sudo`? Only the **host-agent daemon installer** needs root — it writes to `/usr/local/bin`, `/etc/amini/`, and the init-system unit dirs. The **runner CLI** (`amini join`, `amini runner status`, `amini runner earnings`, …) does **not** need sudo. It writes its keypair and enrollment state under the invoking user's home: | env var | resulting state dir | | ---------------- | ---------------------------- | | `AMINI_RUNNER_HOME=/path` | `/path/` | | `XDG_STATE_HOME=/x` | `/x/amini/runner/` | | `HOME=/home/mike` (default) | `/home/mike/.amini/runner/` | If you accidentally ran `sudo amini join …` once and it wrote to `/root/.amini/runner/`, copy or move that directory to your real `$HOME/.amini/runner/` and re-run `amini runner status` as your normal user. ## What the installer does 1. **Detection** — distro from `/etc/os-release`, init system from `ps -p 1 -o comm=`, libc from `ldd --version`, escalation tool (`sudo` → `doas` → root). Override probes with `AMINI_FAKE_*` env vars for dry-runs. 2. **Binary install** — copies `dist/amini-host-agent-linux-[-static]` to `/usr/local/bin/amini-host-agent`. 3. **Config** — writes `/etc/amini/host-agent.env` (KEY=VALUE pairs). 4. **Service** — picks the right template per init system: - systemd: `deploy/host-agent/systemd-unit.service` - OpenRC: `deploy/host-agent/openrc.initd` - runit: `deploy/host-agent/runit.run` + `runit-log.run` - sysvinit: `deploy/host-agent/sysvinit.sh` 5. **Start** — enables + restarts the service via the matching tool (`systemctl`, `rc-update`/`rc-service`, runsvdir symlink, `update-rc.d`/`chkconfig`). ## Distro-specific notes ### Alpine (musl, OpenRC) ```bash apk add libgcc # only if you ship a non-static binary make host-agent-static # produces -static suffix linu" }, { "slug": "how-amini-works", "title": "How Amini Cloud works", "description": "A walkthrough of Amini Cloud's six planes, the lifecycle of a slice, the lifecycle of an AminiChain job, and the lifecycle of a tenant. The first page to read.", "section": "root", "body": "# How Amini Cloud works This is the page to read first. It walks the architecture top-down and then walks three end-to-end flows in detail: how a slice is born, how an AminiChain job becomes a slice, and how a tenant is provisioned. Every file path is real. Every column is in a migration in `db/migrations/`. Every HTTP route is in `internal/admission/server.go` or `internal/admission/sla_routes.go`. If you find a claim here that contradicts the code, that is a documentation bug; see the accuracy contract in [`README.md`](README.md). ## Mental model in one paragraph Amini Cloud is two Go binaries (`amini-admission` on the control plane, `amini-host-agent` on every compute host), one Postgres cluster, a Nomad cluster, one Caddy router, one Headscale process per tenant, and a SeaweedFS + ZFS storage layer. Tenants POST a slice spec to `/v1/slices`; admission validates, schedules, mints an overlay identity, and hands provisioning off to a host-agent; the host-agent prepares a network namespace, applies nftables, talks to a runtime (Podman, Firecracker, QEMU, Kata, nspawn, or an external Kubernetes), and the slice comes up reachable on a per-tenant Tailscale-compatible overlay. The slice's name resolves as `..amini.ts.net` from any device in that tenant's tailnet. SSH, HTTP, and TCP work over that overlay. Default-deny is enforced at three independent layers (overlay ACL + host nftables + runtime capability dropping). Sharing across users inside one tenant is one `slice_grants` row. ## How a compute host is reached — no VPN (ADR-0102) Two networking questions are separate and answered differently. *How a tenant's devices reach their slices* is the per-tenant overlay above (Tailscale-compatible; the data plane in §2). *How the control plane reaches a compute host to dispatch work to it* is the **sovereign transport** — and that path needs no VPN. This sovereign transport is proven end-to-end — a real service served over libp2p to a NAT'd host, Tailscale-free — but it is not the fleet default yet. It ships behind `AMINI_P2P_ENABLED`, off unless an operator turns it on; a stock install still runs the legacy Tailscale/DERP path. Flipping the default is a staged operator cutover ([`docs/adr0102/08-HOW-TO-FLIP-the-libp2p-cutover.md`](../adr0102/08-HOW-TO-FLIP-the-libp2p-cutover.md)), not a remaining code gap. How a host joins and is reached over it: 1. **Add the host** through the admin wizard — one-command install. 2. The host-agent **registers itself on the AminiChain HostRegistry precompile (`0x1908`), gasless** — a zero-balance key writes the record. The **chain is the directory**: the host's libp2p `peer_id` + endpoints live on-chain, not in a mutable `host_agent_addr` column. 3. The host **reserves a slot on a public circuit relay** (the rendezvous). 4. Admission **resolves the host from the chain** and dials it over libp2p — direct first, then a **DCUtR hole-punch through NAT** (proven: direct QUIC through Starlink carrier-grade NAT, 324 ms), falling back to the blind relay only if the punch can't land. No Tailscale in that path. 5. **Slices provision over that libp2p connection.** Proven end-to-end: a slice dispatched to a remote host behind its own home NAT, reached purely over the transport. Why it is built this way: a decentralised cloud runs on machines the operator does not own, on networks it does not control, behind NATs it cannot configure. A third-party VPN control plane that can de-authorise a node is the wrong dependency at the root. libp2p + the chain-as-directory removes it — no VPN, no inbound-port requirement (CGNAT hosts are first-class), no gas to join. The relay is blind (Noise ciphertext end-to-end), authorises nothing (identity is a cryptographic `peer_id`), and is deletable once any fleet host has a public IP. The full technical write-ups — the hole-punch guide and the end-to-end plan — are in `docs/adr0102/` and [ADR-0102](../decisions/0102-chain-as-network-substrate.md) in the repository. ## " }, { "slug": "install-tailscale-sidecar", "title": "Tailscale-sidecar compose override", "description": "Use this guide when you want to run the Amini Cloud control plane in Docker Compose on a **Mac** (or any host where the Docker daemon's network namespace is isolated from the host's Tailscale interface) AND have amini-admission reach a **remote compute host** over the tailnet.", "section": "root", "body": "# Tailscale-sidecar compose override Use this guide when you want to run the Amini Cloud control plane in Docker Compose on a **Mac** (or any host where the Docker daemon's network namespace is isolated from the host's Tailscale interface) AND have amini-admission reach a **remote compute host** over the tailnet. > If your CP runs on Linux and the host already has Tailscale on the > root netns, you almost certainly do not need this override — > compose containers can dial `100.x.x.x` directly via the host network. ## Why this exists Docker Desktop on macOS runs containers inside a hidden Linux VM. The Mac's Tailscale daemon (`tailscaled` on the macOS host) does NOT expose its tailnet interface to that VM. So even though the Mac itself can `curl http://100.113.57.52:7443/healthz` (the spark-5804 host-agent), a container running in compose on the same Mac gets `no route to host`. Three options to fix this: 1. **Run admission natively on the Mac.** Works, but loses \"everything in compose\" and the operator now juggles two startup surfaces. 2. **Move the CP to the spark.** Works, but eliminates the convenient \"developer iterates on a Mac\" loop, and forces every dev to ship code to the spark to test. 3. **Add a Tailscale sidecar container to compose.** (This guide.) The sidecar joins the tailnet from inside the Docker VM. Admission shares its network namespace via `network_mode: \"service:tailscale\"`, so admission's egress goes out over the sidecar's tailnet interface and `100.x.x.x` peers become reachable. We picked option 3. See [`ADR 0038`](../decisions/0038-tailscale-sidecar.md) for the trade-off analysis. ## When to use this override | Situation | Use this? | | -------------------------------------------------------------- | --------- | | Mac CP + a remote Linux compute host on the same tailnet | **Yes** | | Linux CP on a host that has Tailscale on the root netns | No | | Anything in production (use `deploy/ansible/` + Headscale) | No | | You want admission to reach tailnet IPs from inside compose | **Yes** | ## Step 1 — generate a Tailscale auth key The sidecar joins the tailnet using a **one-shot auth key** that the Tailscale admin console mints. The key is consumed on first boot; the resulting node identity lives in the `./state/tailscale/` named volume and survives restarts. 1. Open the admin console: . 2. Click **\"Generate auth key…\"**. Recommended dev settings: - **Reusable**: `yes` — so a sidecar restart with an empty state dir does not require a new key. - **Ephemeral**: `no` — keeps the `amini-cp` node visible in your tailnet between restarts. - **Tags**: optional, e.g. `tag:amini-cp` if your ACL is tag-aware. Production should always use tags. - **Expiry**: 90 days (Tailscale's max). Set a calendar reminder to rotate before expiry; see [Auth-key rotation](#auth-key-rotation) below. 3. Copy the resulting `tskey-auth-…` value. 4. Write it to `deploy/compose/secrets/tailscale-authkey` with mode `0600`: ```bash install -m 0600 /dev/null deploy/compose/secrets/tailscale-authkey printf 'tskey-auth-…' > deploy/compose/secrets/tailscale-authkey ``` The file is gitignored — the real key never gets committed. The shape is documented in `deploy/compose/secrets/tailscale-authkey.example`. ## Step 2 — verify with the bootstrap script ```bash bash scripts/bootstrap-tailscale-sidecar.sh ``` The script: - Confirms `deploy/compose/secrets/tailscale-authkey` exists + is non-empty + starts with `tskey-auth-`. - Tightens the file mode to `0600` if it was looser. - Runs `docker compose -f compose.yml -f compose.tailscale.yml config` to make sure the override merges cleanly. If any check fails, the script prints the remediation steps and exits non-zero. ## Step 3 — bring the stack up with the override ```bash docker compose \\ -f deploy/compose/compose.yml \\ -f deploy/compose/compose.tailscale.yml \\ up -d ``` Equivalent shortcut: ```bash make tailscale-sidecar-up ``` The first boot does three " }, { "slug": "installation/single-host", "title": "Single-host installation", "description": "Bring Amini Cloud up on one machine via Docker Compose. The fastest path to a running stack.", "section": "installation", "body": "# Single-host installation > ⚠️ **This guide (2026-05-13) predates the current `make up` flow and is kept for > history.** The current one-command path is `make quickstart`, the web console ships > in compose (`:5173`), and some secret names here are out of date. **Start at > [operators/agent-onboarding.md](../../operators/agent-onboarding.md)** for a current setup. This is the fastest path to a running amini. Docker Compose, no remote hosts, ephemeral state in `./state/`. Suitable for development, demos, and the level-0 single-user journey from [`docs/user-journeys.md`](../../user-journeys.md). For real multi-host operation use [`three-host-cluster.md`](three-host-cluster.md). The Compose file is [`deploy/compose/compose.yml`](../../../deploy/compose/compose.yml). Everything below is what `make up` actually does. ## Prerequisites On the host: - Linux x86_64 or arm64 (DGX Spark FE / Grace-Blackwell aarch64 verified). - Docker 27+ with Compose v2 plugin. - Go 1.22+ (only if you intend to build `amini-host-agent` locally). - 4 vCPU and 8 GiB of RAM minimum for the control-plane containers, plus whatever headroom your slices need. The Raspberry Pi 4 is too small. - Free TCP ports 5432 (or `AMINI_PG_PORT`, often 15432), 8089 (Caddy HTTP), 8210 (Vault), 8445 (Caddy HTTPS), 18080 (admission), 50001, 50051; UDP 3478; TCP 8443 (DERP). The non-stock literals above (`:18080`, `:8210`, `:8089`, `:8445`) are the single-host Compose defaults — see [`deploy/compose/.env.example`](../../../deploy/compose/.env.example) for the `AMINI_*_PORT` override surface. (Port 9091 is only needed if you opt into the legacy Authelia/OIDC profile — see [OIDC sign-in setup](../oidc-setup.md).) - Outbound HTTPS to pull container images. - Root access (or membership in the `docker` group). ## Step 1 — Clone and inspect ```bash git clone https://github.com/aminitech/amini-cloud cd amini-cloud ``` The Compose file you'll be running is `deploy/compose/compose.yml`. Read it before you run it; it is the canonical record of every service the single-host stack starts. ## Step 2 — Pre-create the Headscale admin token The amini-admission container mounts a file containing a Headscale admin token. Create it before `make up`: ```bash mkdir -p deploy/compose/secrets openssl rand -hex 32 > deploy/compose/secrets/headscale-admin-token chmod 600 deploy/compose/secrets/headscale-admin-token ``` ## Step 3 — Bring the stack up ```bash make up ``` > **Want the full sovereign bring-up instead?** `make up` gives you the plain > single-host stack (chain off, Tailscale data path). For the sovereign > reference deployment — on AminiChain, with a self-provisioning oracle, > autofund, and the libp2p transport — use the near-zero-touch flow: > `sudo make wipe` then `make bring-up EMAIL=… PASSWORD=…` (the AminiChain > localnet systemd service must be up). See > [`zero-touch-bringup`](../../operators/zero-touch-bringup.md). The target runs `docker compose -f deploy/compose/compose.yml up -d` and prints a short reminder of where each service listens. The full set: | Service | Listen | What it does | |---|---|---| | `postgres` | `127.0.0.1:5432` | Single Postgres 16-alpine cluster. The `amini` schema holds platform tables — **all** migrations under `db/migrations/` (0094+ at time of writing) auto-apply on first start via `docker-entrypoint-initdb.d`. Each tenant gets a `headscale_` schema for its Headscale process. | | `authelia` | `127.0.0.1:9091` | **Optional / legacy** OIDC issuer (only in the `full` profile). Native email+password + wallet sign-in is the default and needs no IdP. Reads `deploy/compose/authelia/`. See [OIDC sign-in setup](../oidc-setup.md). | | `headscale-demo` | `127.0.0.1:50001` (HTTP), `:50051` (gRPC admin) | One Headscale 0.28.0 process for the `demo` tenant; production runs one per tenant. | | `derper` | `127.0.0.1:8443` (HTTPS), `127.0.0.1:3478/udp` | Self-hosted DERP relay built from `build/Dockerfile.derper`. | | `amini-admission` | `127.0.0.1:18080` | The" }, { "slug": "installation/three-host-cluster", "title": "Three-host cluster installation", "description": "Planned wave-2 production bring-up via Ansible across amini-cp + two compute hosts.", "section": "installation", "body": "# Three-host cluster installation > **Status: planned wave-2 deliverable.** The `deploy/ansible/` > directory is scaffolded in the repo but the production playbooks > (`site.yml`, `tenant.yml`, the `amini-cp` / `compute-node` / > `compute-storage` roles) are not yet fully working in this beta. The > single-host Compose path in [`single-host.md`](single-host.md) is the > validated install path. This page captures the *target* shape so you > can plan ahead, audit the topology, or wire it manually until the > playbooks land. For the live, authoritative status of multi-host / > clustered capabilities on your build, check > `GET /v1/capabilities` (or `/admin/capabilities` in the dashboard) > rather than trusting this page's prose. ## What you'll have when this is done The shape per [`docs/whitepaper.md`](../../whitepaper.md) and [ADR 0006](../../decisions/0006-three-host-cluster-and-storage.md): ```mermaid flowchart TB CP[\"amini-cp (public VPS)
Caddy SNI router
amini-admission (native + wallet auth)
Headscale fleet (per-tenant)
Postgres 16
Nomad servers
derper-1\"] C1[\"amini-compute-1
Nomad client
amini-host-agent
Podman + FC
SeaweedFS vol
local-zfs
GPUs (optional)\"] C2[\"amini-compute-2
Nomad client
amini-host-agent
Podman + FC
SeaweedFS vol
local-zfs
GPUs (optional)\"] DR2[\"derper-2
(free-tier cloud, different network)\"] CP -- WireGuard mesh --- C1 CP -- WireGuard mesh --- C2 CP -. DERP fallback .- DR2 ``` *Three Amini Cloud hosts plus a fourth free-tier node for DERP fallback.* Three hosts in the Amini Cloud cluster plus a fourth free-tier node for DERP fallback. Per [`docs/build-plan.md`](../../build-plan.md) the canonical first compute host is `spark-5804` (DGX Spark FE in Limuru, Kenya, aarch64). ## Why three hosts and not two Two hosts give you \"one writer plus one backup,\" which is not really distributed storage. Three hosts (one control plane + two compute) let you run SeaweedFS with replication factor 2 across the two compute nodes and survive a single-host outage for object storage. Block storage is harder — `local-zfs` is single-host at MVP, and Year-1 Longhorn lifts it. See [ADR 0006](../../decisions/0006-three-host-cluster-and-storage.md). ## Prerequisites ### `amini-cp` (the control plane host) - A public VPS (Hetzner, Vultr, DigitalOcean, OVH — anything with a public IPv4 + IPv6 and no CGNAT). ~$20–40/month. - Ubuntu 24.04 LTS or Debian 12. - 4 vCPU, 8 GiB RAM, 80 GiB disk minimum. - Ports 80 and 443 reachable from the internet (Caddy will request Let's Encrypt certificates). - DNS: `*.amini.example` pointing to the VPS public IPv4. - SSH user with sudo, key-based access. ### Each compute host - Linux x86_64 or aarch64. Kernel 6.1+. - 4+ vCPU, 16+ GiB RAM, 200+ GiB free disk (for SeaweedFS data + slice rootfs). - Outbound HTTPS to the control-plane VPS and DERP relays. **CGNAT is fine** — compute hosts make outbound connections only. - Optional NVIDIA GPU with the proprietary driver (525+ for L40S/H100; 535+ for Grace-Blackwell / GB10). DCGM exporter will be installed. - Podman 5.x, cgroups v2 enabled, `/dev/kvm` available, Firecracker 1.15.x binary installed under `/usr/local/bin/firecracker`. - An empty disk or partition you are willing to give to ZFS for the `block` storage class. - SSH user with sudo, key-based access. ### `derper-2` (the second relay) - Any free-tier cloud node in a different network (Oracle ARM, Fly.io free, etc.). - Public IPv4 + IPv6, ports 443/TCP and 3478/UDP reachable. - Ubuntu / Debian. ### From the laptop running Ansible - Ansible 2.16+ (`apt install ansible` or `pip install ansible`). - SSH access to every host (keys loaded in `ssh-agent`). ## Target bring-up procedure (planned) When the playbooks land, the flow is: ```bash git clone https://github.com/aminitech/amini-cloud cd amini-cloud # Edit the inventory with your hostnames + SSH user $EDITOR deploy/ansible/inventory.example.ini # Syntax " }, { "slug": "installation/upgrading", "title": "Upgrading Amini Cloud", "description": "The safe path to a new Amini Cloud release — pre-flight, drain, migrate, restart, verify.", "section": "installation", "body": "# Upgrading Amini Cloud Amini Cloud is meant to be installed once and stay installed. The policy behind this — what we will and will not change between minor releases, how long deprecation cycles run — is [ADR 0010](../../decisions/0010-versioning-and-backward-compatibility.md). This page is the operator-facing procedure. > **Status of `amini update`:** the `workflow_self_update` capability is > marked `beta` in `slate_features`. The CLI-driven one-command upgrade > (`amini admin upgrade`) is scaffolded but not yet wired end-to-end. > Until it is, use the manual procedure below. ## Pre-flight Before any upgrade: 1. **Read the changelog.** Open [`changelog.md`](../changelog.md) and read every entry between your current version and the target. Schema migrations are forward-only and idempotent; that does not make them free. 2. **Verify current version.** Query the version log: ```bash psql \"$POSTGRES_DSN\" -c \" SELECT component, semver, commit_sha, started_at FROM amini.server_versions ORDER BY started_at DESC LIMIT 10; \" ``` The `server_versions` table is populated by every `amini-admission` start (migration 0005). 3. **Check the feature catalog.** Look for any capability whose `removal_date` is in the past: ```bash psql \"$POSTGRES_DSN\" -c \" SELECT id, status, removal_date FROM amini.slate_features WHERE removal_date IS NOT NULL AND removal_date < now(); \" ``` If a tenant is using a removed capability, you must migrate them before upgrading. 4. **Snapshot Postgres.** Either `pg_dump` or your provider's snapshot API: ```bash pg_dump \"$POSTGRES_DSN\" -F c -f amini-pre-upgrade-$(date +%F).dump ``` See [`backup-and-restore.md`](../operator/backup-and-restore.md). 5. **Snapshot SeaweedFS metadata.** Object data is replicated across compute hosts; the master's metadata is on `amini-compute-1`. 6. **Tell tenants.** If you are pinning slices to a `local-zfs` block volume, a host restart will stop their slices. Give them 24 hours. ## Single-host (Docker Compose) ```bash cd amini-cloud git fetch --tags git checkout v # Pull new images docker compose -f deploy/compose/compose.yml pull # Run migrations in-place (Compose mounts db/migrations as # docker-entrypoint-initdb.d; this only runs them on a fresh Postgres # volume — for an upgrade on an existing volume you must apply manually) make db-migrate POSTGRES_DSN=\"postgres://amini:amini@localhost:5432/amini?sslmode=disable\" # Note: 5432 is the Compose default; when AMINI_PG_PORT remaps Postgres # (e.g. to 15432 because the host already runs Postgres on 5432), swap # the port literal above to match. See deploy/compose/.env.example. # Restart the stack make down make up # Verify (admission binds to :18080 by default — Compose intentionally # avoids :8080 to dodge ULAP-ONE / Beszel collisions; see # deploy/compose/.env.example for AMINI_ADMISSION_PORT) curl -fsS http://localhost:18080/healthz curl -fsS http://localhost:18080/readyz ``` ## Multi-host (production) Order matters. Postgres first, then admission, then host-agents. 1. **Apply migrations.** From a workstation that can reach Postgres: ```bash POSTGRES_DSN=postgres://amini:...@amini-cp:5432/amini \\ make db-migrate ``` Schema migrations are idempotent (each ends with `INSERT INTO schema_migrations (version) VALUES (N)` guarded by the numeric primary key). Re-running them is safe. 2. **Drain compute hosts one at a time.** For each compute host: ```bash # Amini Cloud admin hosts drain # scaffold, prints \"not yet implemented\" nomad node drain -enable -yes -deadline 30m ``` This stops scheduling new slices on the host and waits for existing slices to terminate or move. Note that OSS Firecracker has no live migration; slices stop and restart on the destination host. See [`troubleshooting/known-issues.md`](../troubleshooting/known-issues.md). 3. **Restart `amini-admission`.** It writes a new row to `server_versions` on start: ```bash ssh amini-cp 'systemctl restart amini-admission' curl -fsS https://api.amini.examp" }, { "slug": "llm-model-gallery", "title": "LLM model gallery", "description": "Deploy a chat-completions endpoint on Amini Cloud in under five minutes. The model gallery turns a vLLM / Ollama / TGI image into a slice with an OpenAI-compatible API at a tenant-scoped URL, gated by your tenant's RBAC matrix.", "section": "root", "body": "# LLM model gallery Deploy a chat-completions endpoint on Amini Cloud in under five minutes. The model gallery turns a vLLM / Ollama / TGI image into a slice with an OpenAI-compatible API at a tenant-scoped URL, gated by your tenant's RBAC matrix. ## What you get - A slice running the model's container image, sized to fit your hardware floor. - An OpenAI-compatible HTTP endpoint at `https://..amini.localtest.me:8446/v1/chat/completions`. - Per-role access gating — `developer` rows are free for any tenant member; `admin` / `owner` rows are gated by `tenant_users.role`. - A sharable ingress URL (via `amini.slice_ingress_routes`) and the option to bind a custom domain. ## Path 1 — pick from the curated gallery 1. Sign in with your wallet or OIDC identity, then go to **Admin → Models** (`/admin/models`). 2. Filter by family (`vLLM`, `Ollama`, `TGI`), role gate, or GPU requirement. 3. Click a model card. The detail page shows the pinned `default_args`, `default_env`, the image, and the resource floor. 4. Press **Launch as slice**. Pick a slice name and confirm the exposure scope: - `tenant` — only signed-in tenant members can reach the endpoint. - `public` — anyone with the URL can reach it (owner-only toggle). 5. You'll land on the slice detail page. Wait for `status=running`; the OpenAI-compatible URL is shown in the **Access** card. ### Curated models, honestly sized Every row in the curated catalog has a `vram_gb_min` set against what actually fits on a single Grace-Blackwell DGX Spark (GB10). 70B-class models are gated to `admin` so they don't evict everything else off the box. Smaller models (Phi-3 Mini, Mistral 7B, Llama 3 8B, Qwen2.5 7B, Gemma 2 9B) are open to any developer. ### Gated models need an HF token \"Gated\" here means **gated on Hugging Face**, not gated by amini RBAC. A gated repo (Google's `google/gemma-4-*` family is the canonical example) is public to browse, but downloading the weights requires a Hugging Face account that has accepted the model's license plus an access token. amini launches a model on the serving backend its `runtime_compat` declares, in the preference order `ollama > vllm > hf-transformers > llama-cpp`. The gated Gemma 4 rows lead with `hf-transformers`, so amini **correctly routes them to a Jupyter GPU notebook (the `jupyter-gpu` template), not a vLLM/Ollama serving slice** — that fallback is deliberate, so a serving backend never crash-loops trying to fetch weights it can't authenticate for. - **Symptom without a token.** The notebook's first `from_pretrained(...)` against the gated repo fails with a 401 / \"gated repo\" / \"you must be authenticated\" error. The slice itself stays up — only the download is refused. - **Fix.** Make the download authenticated. The operator sets `AMINI_HF_TOKEN` for the platform (see `deploy/compose/.env.example`); inside the notebook you can also `huggingface-cli login` or export `HF_TOKEN` before downloading. After the token is present **and** you've accepted the model's license on Hugging Face, the download succeeds. Ungated models (most of the catalog) need none of this. ## Path 2 — register a custom model If the curated catalog doesn't cover your model, go to **Admin → Models → Register custom model** (`/admin/models/new`). Fields: - `display_name` — what the gallery shows. - `hf_repo` — Hugging Face repo id, e.g. `meta-llama/Meta-Llama-3-8B-Instruct`. - `family` — `vLLM`, `Ollama`, or `TGI`. - `default_image` — OCI reference, e.g. `vllm/vllm-openai:latest`. - `default_args` — one per line. The container listens on port 8000. - `default_env` — `KEY=VALUE` per line. - `required_role` — who in your tenant can launch this row. - `vram_gb_min` — used to size the slice request. Under-stating it will make launches fail. > **Status:** the tenant-scoped custom-model registry (`amini.custom_models`) is wired through the UI but the backend handler returns `501` until the next wave ships. The UI catches the 501 and tells you the draft is preserved. ## Ca" }, { "slug": "multi-tenant-serving", "title": "Multi-tenant model serving — tuning + how many users at once", "description": "Two layers give you multi-user / multi-tenant model access:", "section": "root", "body": "# Multi-tenant model serving — tuning + how many users at once Two layers give you multi-user / multi-tenant model access: 1. **Gateway + per-user keys (shared-model multi-tenancy)** — the LiteLLM gateway (`deploy/inference-gateway/`) fronts the fleet; each user gets a **revocable, rate-limited, usage-tracked API key**. Users share the model pool concurrently. This is live today. 2. **amini slices (isolated multi-tenancy)** — each tenant gets its own ollama/vLLM slice on a host (separate model instance, separate network). Heavier (per-tenant GPU/memory), but full isolation. Launch from the amini console (`/slices/new`). Most teams want #1 for day-to-day and #2 only for hard isolation needs. ## Concurrency tuning — ollama (both the Spark and the H200) ollama serializes per model unless you raise its parallelism. Set these on **every** ollama host (systemd), then restart: ```bash sudo mkdir -p /etc/systemd/system/ollama.service.d sudo tee /etc/systemd/system/ollama.service.d/amini-tuning.conf >/dev/null <<'EOF' [Service] Environment=\"OLLAMA_HOST=0.0.0.0:11434\" Environment=\"OLLAMA_NUM_PARALLEL=4\" # concurrent requests handled per loaded model Environment=\"OLLAMA_MAX_LOADED_MODELS=3\" # distinct models kept resident at once Environment=\"OLLAMA_KEEP_ALIVE=30m\" # keep models warm (avoid reload latency) Environment=\"OLLAMA_FLASH_ATTENTION=1\" # faster + less KV memory EOF sudo systemctl daemon-reload && sudo systemctl restart ollama ``` Sizing: - `NUM_PARALLEL × KV-per-request` must fit VRAM alongside the weights. On the H200 (141 GB) a 72B leaves plenty of room for `NUM_PARALLEL=4`. On the Spark (128 GB unified) keep an eye on total = (sum of loaded model weights) + (parallel × KV). - Raise `NUM_PARALLEL`/`MAX_LOADED_MODELS` for more users; lower if you hit OOM. ## Higher-throughput tier — vLLM (for real concurrency at scale) ollama is convenient; **vLLM** is the multi-tenant workhorse (continuous batching, paged KV, far higher concurrent throughput). For a heavily-shared model, serve it with vLLM and point the gateway at it: ```bash docker run -d --gpus all -p 8000:8000 vllm/vllm-openai:latest \\ --model Qwen/Qwen2.5-72B-Instruct --served-model-name qwen2.5-72b \\ --gpu-memory-utilization 0.90 --max-num-seqs 64 # then in litellm-config.yaml: api_base http://:8000/v1 ``` ## Per-key rate limits (fairness across tenants) Mint keys with limits so one user can't starve others: ```bash curl -s http://localhost:4000/key/generate \\ -H \"Authorization: Bearer $LITELLM_MASTER_KEY\" -H \"Content-Type: application/json\" \\ -d '{\"models\":[\"qwen2.5-coder\",\"qwen2.5-72b\"],\"rpm_limit\":60,\"tpm_limit\":200000,\"key_alias\":\"alice\"}' ``` Revoke: `POST /key/delete {\"keys\":[\"sk-...\"]}`. Usage/spend per key: `GET /key/info`. ## What \"multiple users at once\" looks like end-to-end user (key) → gateway `:4000` (auth, rate-limit, usage log) → router → ollama/vLLM on the host that owns the model, which serves `NUM_PARALLEL` requests concurrently per model, across `MAX_LOADED_MODELS`. Add users by minting keys; add capacity by raising parallelism, adding replicas, or moving hot models to vLLM. " }, { "slug": "oidc-setup", "title": "OIDC sign-in setup (optional / legacy — Authelia, compose stack)", "description": "> [!info] **You probably don't need this.** > The shipped, default sign-in for Amini Cloud is **native > email+password** plus **wallet (SIWE) sign-in** — neither touches > Authelia or any OIDC issuer (see > [ADR 0069](../decisions/0069-native-auth-replace-authelia.md) and > [API", "section": "root", "body": "# OIDC sign-in setup (optional / legacy — Authelia, compose stack) > [!info] **You probably don't need this.** > The shipped, default sign-in for Amini Cloud is **native > email+password** plus **wallet (SIWE) sign-in** — neither touches > Authelia or any OIDC issuer (see > [ADR 0069](../decisions/0069-native-auth-replace-authelia.md) and > [API authentication](api/authentication.md)). This guide is an > **opt-in advanced path** for operators who want to federate sign-in > against an external OIDC issuer. If you just want users to log in, > skip this page. This guide covers the operator-side wiring needed for the optional **Sign in with Authelia** path on `amini-web` to complete a full PKCE round-trip against the compose-stack Authelia. It only applies when you bring up the `full` compose profile (which includes Authelia); the default profile does not. The compose stack ships Authelia 4.39 with one public client (`amini-web`) and one confidential client (`amini-admission`). The default redirect-URI registry covers the common Vite ports (`:5173`, `:5174-:5176` for auto-bump, `:4173` for `vite preview`, `:3000` and `:8080` for alternate dev ports), plus `127.0.0.1` variants. If your dev server runs on something else, see \"Adding a new redirect URI\" below. > **Production:** these are dev-only defaults. Real deployments render > Authelia config via `scripts/render-authelia.sh` from the > `deploy/authelia/` template suite. --- ## What breaks without setup If you click **Sign in with Authelia** and you get any of: - Browser-level `DNS_PROBE_POSSIBLE` for `auth.amini.local` - `https://auth.amini.local/consent/completion?error=invalid_request&...redirect_uri parameter does not match...` - A blank page after the IdP redirect …you're hitting one of these issues: 1. **`/etc/hosts` not configured.** The compose Authelia is configured with `authelia_url: https://auth.amini.local` (this is the hostname the cookie domain is scoped to — see ADR 0011). Authelia uses this URL for its OWN consent / error pages. If `auth.amini.local` doesn't resolve on your machine, the browser can't follow Authelia's error redirects. 2. **Your dev server is on an unregistered port.** Authelia 4.39 validates the `redirect_uri` parameter byte-for-byte against the registered list. `localhost:5173` and `127.0.0.1:5173` are different strings; a port-bump from Vite (`:5174`) is a different string again. 3. **Authelia isn't reachable on `:9091`.** The compose stack publishes Authelia (via the `authelia-proxy` sidecar) on `127.0.0.1:9091`. The Vite dev server proxies `/api/oidc/*` to that port. If the port is already taken or the `full` compose profile isn't up, the discovery probe fails and the sign-in button self-disables. --- ## One-time operator setup ### 1. Add `/etc/hosts` entries ```bash sudo sh -c 'cat >> /etc/hosts <-acceptance--.md` and treat its \"Failure mode\" entry as the canonical bug ticket. --- ## L0 — Smoke (every integration round) Every API call below sends `-H \"Authorization: $TOKEN\"`. Set it once from an API key minted off a signed-in session: ```bash export TOKEN=\"Bearer $AMINI_API_KEY\" ``` The three calls that mint the key are in [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api). ### L0.1 — Compose stack comes up cleanly ```bash make up # or: docker compose -f deploy/compose/compose.yml up -d docker compose ps # all \"healthy\" or \"Up\" ``` Expected: 8/8 services healthy (postgres, vault, headscale-demo, authelia, derper, minio, caddy, amini-admission, ingress-caddy). Failure mode: port collisions (8080, 9000, 9091) — see operator port inventory. ### L0.2 — Admission responds ```bash curl -fsS -H \"Authorization: $TOKEN\" \\ http://127.0.0.1:18080/v1/templates | jq 'length' ``` Expected: 18 (or whatever current catalog count is). Failure mode: 401 if $TOKEN is unset, expired or revoked; 503 if Postgres is still starting. ### L0.3 — One slice from one template renders its URL ```bash SLICE=$(curl -sS -X POST -H \"Authorization: $TOKEN\" \\ -H \"Content-Type: application/json\" \\ -d '{\"name\":\"smoke\",\"runtime\":\"podman\",\"template_id\":\"nginx-web\",\"image\":\"nginx:1.27-alpine\",\"resources\":{\"cpu_milli\":250,\"mem_bytes\":134217728,\"disk_bytes\":536870912},\"exposure\":[{\"port\":80,\"proto\":\"tcp\",\"scope\":\"tenant\"}]}' \\ http://127.0.0.1:18080/v1/slices) ID=$(echo \"$SLICE\" | jq -r .id) SHORT=${ID:0:8} sleep 10 curl -fsS \"http://smoke-${SHORT}.demo.amini.localtest.me:8095/\" | head -1 ``` Expected: `` from nginx welcome. Failure mode: 404 catch-all means reconciler hasn't reached Caddy admin API yet (check ingress-caddy logs for admin port `:2019` binding). --- ### L0.4 — Reconcile-from-clean (required before any tag) The 2026-05-15 round surfaced a class of bug where code is correct but the *running stack was never reconciled to it* — the SEC-T2-001 isolation fix put slices on the `slate_slices` network in code, but the live stack had never created that network, so every new slice silently failed to reach `running`. Gates that ran against committed code/artefacts passed; the hand-mutated running stack diverged. (Devil's-advocate finding DA-02, `docs/audits/2026-05-15-devils-advocate-1eaceee.md`.) This gate makes the divergence impossible to miss: ```bash # From a CLEAN checkout of the tag candidate — not the dev box's # long-lived stack. docker compose -f deploy/compose/compose.yml down -v make up # rebuild + recreate from committed source # then one REAL end-to-end smoke per template newly flipped this # release (not an Engine-API-equivalence stand-in): for tpl in ; do # create a slice fr" }, { "slug": "operator/auth-e2e-runbook", "title": "Operator Runbook: Auth End-to-End Browser Test (L4.3 OIDC + L4.4 Wallet)", "description": "Status: READY for operator run Companion: code+probe wiring verification in `docs/audits/2026-05-15-acceptance-L4-auth.md` (read its \"Bugs surfaced\" section first — L4.3 has a known unimplemented server-side leg).", "section": "operator", "body": "# Operator Runbook: Auth End-to-End Browser Test (L4.3 OIDC + L4.4 Wallet) Status: READY for operator run Companion: code+probe wiring verification in `docs/audits/2026-05-15-acceptance-L4-auth.md` (read its \"Bugs surfaced\" section first — L4.3 has a known unimplemented server-side leg). This runbook is the **operator-run** verification for L4.3 (OIDC PKCE) and L4.4 (wallet sign-in). It is the only way to assert the browser legs that no agent could test (real IdP redirect, real wallet extension signing). > **Read this before you start (L4.3):** The audit confirmed by live probe > that `POST /v1/auth/exchange` returns **404** — amini-admission does > **not** implement the OIDC code→token exchange yet. The browser flow up > to the callback bounce page works; the final exchange step will fail > with an honest \"admission has not enabled OIDC exchange yet\" message > until that handler ships. Run L4.3 to confirm the front-end + IdP legs; > mark the exchange leg **blocked** unless the handler has since landed > (re-probe step 0). --- ## 0. Pre-flight re-probe (always do this first) ```bash # Authelia OIDC discovery reachable? curl -s http://localhost:9091/.well-known/openid-configuration | head -c 300; echo # expect HTTP 200 + JSON with authorization_endpoint + token_endpoint # Has the OIDC exchange handler shipped since 2026-05-15? curl -s -o /dev/null -w '%{http_code}\\n' -X POST \\ http://127.0.0.1:18080/v1/auth/exchange \\ -H 'Content-Type: application/json' \\ -d '{\"code\":\"x\",\"state\":\"y\",\"code_verifier\":\"z\"}' # 404 = exchange leg still NOT implemented → L4.3 final step is BLOCKED # 400/401 = handler now exists → L4.3 can go fully browser-green # Wallet nonce route live? curl -s -X POST http://127.0.0.1:18080/v1/auth/wallet/nonce \\ -H 'Content-Type: application/json' \\ -d '{\"wallet_address\":\"0x0000000000000000000000000000000000000000\",\"chain\":\"evm\"}' # expect 200 + {\"nonce\":\"amini sign-in: …\",\"expires_at\":\"…\"} ``` --- ## Pre-requisites ### Shared - amini-admission running and reachable at the URL the web app proxies to (`AMINI_ADMISSION_URL` on the Vite proxy; live dev = `http://127.0.0.1:18080`). - The Amini Cloud web app running (SvelteKit). Note the **origin** you load it from — it becomes the OIDC `redirect_uri` (`/login/callback`, `web/src/routes/login/+page.svelte:28`). - A modern browser with DevTools (Network + Application/Storage tabs). - Postgres `amini-postgres-1` up (tenant + tenant_users rows live here). ### L4.3 (OIDC) extra pre-reqs - A **real, resolvable issuer domain** for the tenant. Authelia's discovery returns `issuer=https://auth.amini.local` — that hostname must resolve from the browser (add to `/etc/hosts` or use the real DNS) **and** the tenant row's `tenants.oidc_issuer` must exactly equal the token `iss`. - **Authelia configured**: an OIDC client registered with `client_id=amini-web`, `redirect_uri=/login/callback`, PKCE `S256` allowed, `response_type=code`, scopes `openid profile email`. - **A test user in Authelia** that can complete Authelia's own login + 2FA. If you intend to exercise tenant-group scoping, give the user a group claim — **but note (SEC-L4-1) Amini Cloud does not currently enforce `oidc_group_claim`; tenant binding is by issuer only.** Record this. - The tenant must exist in Amini Cloud with `tenants.oidc_issuer` = the Authelia issuer and `status='active'`. ### L4.4 (Wallet) extra pre-reqs - **MetaMask** (EVM) and/or **Keplr** (Cosmos) installed with a test account/wallet. Never use a wallet holding real funds. - The test wallet **bound to a tenant_user** *before* sign-in: - run `amini admin tenant-user link-wallet` to set `tenant_users.wallet_address` (lowercased) for a tenant, **or** - enable the NFT resolver (`AMINI_WALLET_NFT_RESOLVER_ENABLED=true`) and have the wallet own a amini-tenant-NFT (JIT provisioning). - Confirm the binding in Postgres: ```bash docker exec -i amini-postgres-1 psql -U Amini Cloud -d Amini Cloud -c \\ \"select tenant_id, subject, role, wallet_a" }, { "slug": "operator/backup-and-restore", "title": "Backup and restore", "description": "Postgres dumps, SeaweedFS replication, OpenBao secret backup, per-tenant export, and the restore procedure.", "section": "operator", "body": "# Backup and restore Amini Cloud's state lives in four places. A good backup strategy covers all four. This page is the operator handbook. ## The four state stores | Store | Holds | Loss scenario | |---|---|---| | **Postgres** | `amini.*` tables (tenants, slices, audit, datasets, lineage, KPIs) plus `headscale_` per-tenant Headscale state | Operator-fatal. Tenants lose every slice spec, every audit row, every grant. | | **SeaweedFS** | Object data: tenant buckets, dataset components in `object` storage class, Zarr tensors | Tenants lose datasets + model checkpoints. Replicated by default but cross-host (not cross-cluster). | | **`local-zfs` zvols** | Block volumes (Postgres data, MinIO data) | Tenant loses one stateful slice's data. | | **OpenBao** | Postgres password, Headscale admin token, OIDC client secrets, AminiChain wallet keys | Operator-fatal. Amini Cloud cannot start without these. | ## What you need - Operator (root) shell on `amini-cp` and each compute host. - Encrypted offsite snapshot store (S3-compatible bucket, Backblaze B2, Wasabi, …). - A passphrase or KMS key for envelope encryption. - Cron access for nightly automation. ## Backup procedure ### 1. Postgres — daily, encrypted ```bash DATE=$(date -u +%F-%H%M%S) DUMP=\"/tmp/amini-$DATE.dump\" pg_dump \"$POSTGRES_DSN\" -F c -f \"$DUMP\" # Encrypt envelope gpg --symmetric --cipher-algo AES256 \\ --batch --passphrase-file /etc/amini/backup-passphrase \\ --output \"${DUMP}.gpg\" \"$DUMP\" shred -u \"$DUMP\" # Ship aws s3 cp \"${DUMP}.gpg\" \"s3://amini-backups/postgres/$DATE.dump.gpg\" shred -u \"${DUMP}.gpg\" ``` Recommend default retention 30 days; longer for SLA-relevant tenants who may need historical SLA reports rebuilt from raw uptime samples. Wire as a cron entry on `amini-cp`: ```cron 0 2 * * * /etc/amini/cron/backup-postgres.sh ``` ### 2. SeaweedFS — replication audit + cold copy SeaweedFS already replicates volume data across compute hosts (default RF=2 between the two compute hosts in a 3-host cluster). What it does NOT do is back up to an off-cluster store. Add a daily cold copy: ```bash # Bucket-level S3-style copy for bucket in $(seaweedfs filer.s3 list); do aws s3 sync \"s3://$bucket\" \"s3://amini-backups/seaweedfs-cold/$bucket/\" \\ --endpoint-url=http://localhost:8333 done ``` Cheaper alternative: snapshot the SeaweedFS master's metadata and ship that nightly; rebuild data from the surviving compute host if you ever lose master + filer. ```bash # Master metadata seaweedfs master.snapshot --filename=/tmp/master-$DATE.snap gpg --symmetric ... -o /tmp/master-$DATE.snap.gpg /tmp/master-$DATE.snap aws s3 cp /tmp/master-$DATE.snap.gpg \"s3://amini-backups/seaweedfs/\" ``` ### 3. ZFS — snapshot before host work Block volumes pin to a single host (MVP `local-zfs`; Year-1 Longhorn replicates). For each compute host: ```bash # Daily snapshot of every amini-owned dataset DATE=$(date -u +%F-%H%M%S) for ds in $(zfs list -H -o name amini); do zfs snapshot \"$ds@nightly-$DATE\" done # Prune older than 7 days zfs list -H -t snapshot -o name amini | grep '@nightly-' | \\ awk -v cutoff=\"$(date -u -d '7 days ago' +%F)\" \\ '{ split($1, a, \"@nightly-\"); if (a[2] < cutoff) print $1 }' | \\ xargs -r -n1 zfs destroy ``` For real off-host backup of stateful slices, `zfs send | aws s3 cp -` to a snapshot bucket. Stream-encrypt: ```bash zfs send amini/pgdata-acme-pg-prod-1@nightly-$DATE | \\ zstd | \\ gpg --symmetric --cipher-algo AES256 \\ --batch --passphrase-file /etc/amini/backup-passphrase | \\ aws s3 cp - \"s3://amini-backups/zvols/pgdata-acme-pg-prod-1/$DATE.zfs.zst.gpg\" ``` ### 4. OpenBao — secret backup The dev-mode bootstrap (`make vault-bootstrap`) auto-persists the shamir keys + root token to `deploy/compose/secrets/vault-keys.json`. **That is dev convenience; in production those keys must not be on the same machine as vault.** Production bootstrap shamir-splits the unseal keys across operators out-of-band and stores no root token on disk. Backup is: 1. Snapshot the vault data dir (`./state/v" }, { "slug": "operator/cloud-platform-truth-criteria", "title": "Cloud-platform truth criteria", "description": "This document is the **no-bias acceptance bar** an outside operator — a Series-A devops engineer evaluating Amini Cloud alongside Fly.io, Railway, Coolify, Coder, and NetBird+Nomad — would hold us to in May 2026. It is not Amini Cloud's wishlist. It is the union of:", "section": "operator", "body": "# Cloud-platform truth criteria — what Amini Cloud must do to be a \"real\" cloud This document is the **no-bias acceptance bar** an outside operator — a Series-A devops engineer evaluating Amini Cloud alongside Fly.io, Railway, Coolify, Coder, and NetBird+Nomad — would hold us to in May 2026. It is not Amini Cloud's wishlist. It is the union of: 1. What an external 2026-era multi-tenant cloud-platform evaluator expects, with citations to primary docs and named practitioners. 2. What [the whitepaper](../../whitepaper.md) already claims Amini Cloud will deliver, taken verbatim and tested against (1). Failure to meet **any** item below means Amini Cloud is at best a beta — there is no hand-waving. Each item is paired with a Amini-Cloud-side artifact that proves it, or a `[NOT YET — gap]` honest stub. The honest gap table in §16 is the most load-bearing part of this document. If we cannot point to a passing artifact, we cannot claim the capability. --- ## 0. Definitions We use these terms with no overlap: - **IaaS** — primitives that look like hardware: virtual machines, block volumes, virtual networks, IAM. OpenStack is the canonical OSS IaaS. The operator is responsible for \"what should this VM do.\" - **PaaS** — primitives that look like applications: deploy a repo, get a URL. Fly.io, Railway, Heroku, Coolify, Dokku, Kamal sit here. The operator is responsible for \"is the app healthy and reachable.\" - **Orchestrator** — a scheduler that places workloads onto hosts. Kubernetes, Nomad, Mesos. By itself, not a \"cloud\" — it has no identity, no tenancy boundary, no billing surface. - **Cloud platform** (the bar this document defines) — a *multi-tenant* product that bundles compute + network + identity + lifecycle + observability + billing-shape such that an outside tenant signs in, provisions a workload, and gets a working URL with isolation guarantees — without the operator touching a config file. Amini Cloud is positioned by [the whitepaper](../../whitepaper.md) §1 as the third one: \"a proprietary, self-hosted, multi-tenant platform — © 2026 Amini Infra Tech — that gives each tenant their own Tailscale-compatible private network and a fleet of isolated compute slices on shared hosts.\" --- ## 1. Tenant isolation The bar in 2026 is **hardware-level isolation between tenants on shared hosts, plus deny-by-default network isolation, plus per-tenant audit boundary**. Container-only isolation is acceptable for tenants that trust each other; for the general case, the industry has moved to microVMs. Fly.io's own architecture page is unambiguous: \"Application code runs in Firecracker microVMs. These are lightweight, secure virtual machines based on strong hardware virtualization\" and \"We make a best-effort attempt to dedicate hardware resources to only one microVM at a time.\"[^fly-arch] Firecracker itself documents the design as \"purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models.\"[^firecracker] Network isolation must be deny-by-default. Tailscale's policy file syntax doc states it directly: \"Tailscale access rules deny access by default. The only possible action is `accept`, which lets traffic flow from the source (src) to the destination (dst).\"[^tailscale-policy] Cilium, Calico, and Kubernetes NetworkPolicy all converge on the same posture in 2026 — global default-deny in the default tier is the documented best practice.[^calico-deny][^cilium-deny] Multi-tenancy is also an *organizational* primitive, not only a network one. Northflank's 2026 multi-tenancy guide and the CNCF Cloud Native Maturity Model 4.0 both treat Namespace-as-a-Service, per-tenant quotas, and per-tenant audit as Level-2 production-ready table stakes.[^northflank-mt][^cncf-mm4] **Amini Cloud artifact**: per-tenant Headscale instance (whitepaper §5.1), per-tenant Postgres schema (`headscale_`), `tenants.untrusted` flag forcing Firecracker (§10), nftables default-drop per slice" }, { "slug": "operator/cluster-operations", "title": "Cluster operations", "description": "Operator-facing runbook for the wave-6 real-cluster-formation surface (GitHub issue #77). Covers the day-2 operations every multi-host Amini Cloud deploy needs: capacity rollup, drain/resume, host moves, and slice placement validation.", "section": "operator", "body": "# Cluster operations Operator-facing runbook for the wave-6 real-cluster-formation surface (GitHub issue #77). Covers the day-2 operations every multi-host Amini Cloud deploy needs: capacity rollup, drain/resume, host moves, and slice placement validation. The endpoints below are admin-gated. Every call needs an admin bearer: either an API key minted from an admin session (`Authorization: Bearer $AMINI_API_KEY`) or, where a tenant is wired to OIDC, an admin JWT (`Authorization: Bearer `). Minting a key takes three curl calls. See [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api). --- ## TL;DR — common tasks | I want to … | CLI | HTTP | |---|---|---| | See every cluster + capacity | `amini admin clusters ls` | `GET /v1/admin/clusters` | | See one cluster's hosts | `amini admin clusters show ` | `GET /v1/admin/clusters/{id}` | | Block new placements on a cluster | `amini admin clusters drain ` | `POST /v1/admin/clusters/{id}/drain` | | Bring a cluster back online | `amini admin clusters resume ` | `POST /v1/admin/clusters/{id}/resume` | | Move a host to another cluster | `amini admin clusters move-host ` | `POST /v1/admin/clusters/{cid}/host-moves` | | Capacity rollup only | `amini admin clusters capacity ` | `GET /v1/admin/clusters/{id}/capacity` | The web dashboard mirrors all of the above under `/admin/clusters` and `/admin/clusters/` — the detail page renders the host list + audit timeline + drain/resume buttons. --- ## Capacity rollup `GET /v1/admin/clusters` returns every cluster row with the capacity rollup joined in. Each row has: - `host_count_total` — every host attached to the cluster (any approval status, any heartbeat freshness). The raw fleet inventory. - `host_count_healthy` — approved hosts heartbeat-ing within the last 90 seconds AND not marked offline. This is the number a scheduler can actually place onto right now. - `cpu_milli_total`, `mem_bytes_total`, `gpu_count_total` — sums across every **approved** attached host (pending + rejected hosts are not fleet capacity). `host_count_healthy < host_count_total` is the operator's signal that something is wrong with a specific host — a stale heartbeat, a host flipped offline, or an approval still pending. Use `amini admin clusters show ` to find which host. ### Example ``` $ amini admin clusters ls ID NAME KIND STATUS HOSTS CPU(milli) MEM(GiB) GPUs 9f3e2b1a-... nomad-prod nomad online 3/4 28000 88.0 1 e1c7d6f4-... k8s-research kubernetes draining 0/2 0 0.0 0 ``` Read: nomad-prod has 4 attached hosts but 1 is unhealthy (stale heartbeat or offline). k8s-research is draining and nobody is heartbeating — operator probably already cordoned for maintenance. --- ## Drain a cluster Two reasons operators drain a cluster: maintenance window, or moving load before deregistration. Both follow the same pattern. ``` amini admin clusters drain # or pass --no-cascade to leave per-host status untouched ``` What happens server-side: 1. `amini.clusters.status` flips to `draining` in a single transaction that locks the cluster row. 2. By default the change cascades: every approved attached host's `amini.hosts.status` flips from `online` to `draining` too — so the scheduler's next sweep skips them. Hosts already `offline` or `draining` are left alone. 3. An audit event `cluster.drain` is recorded with payload `{cluster_id, cascade, hosts_cascaded}`. While draining, slice placements that pin this cluster (top-level `cluster_id` or `placement.cluster_id` in the slice spec) are refused with HTTP 422 and a friendly reason. Slices without a pin land on other clusters per normal scheduling. ### Resume ``` amini admin clusters resume ``` Flips `amini.clusters.status` back to `online`. With cascade=true (default), any host that was flipped to `draining` by the cascade is flipped back to `online`. Hosts that healthsweep marked `offline` — a real heartbeat problem, not just" }, { "slug": "operator/headscale-fleet", "title": "Headscale fleet — operator guide", "description": "Day-to-day operation of the per-tenant Headscale fleet — port ranges, manual reap, schema layout, capacity planning.", "section": "operator", "body": "# Headscale fleet — operator guide This page is the operator handbook for the wave-10 H1 per-tenant Headscale fleet (see ADR 0056). The fleet spawns one `headscale/headscale:0.28.0` container per tenant on tenant-create and reaps it on tenant-terminate. The compose-baked `amini-headscale-demo` container remains the fallback for the `demo` tenant. ## When to use this - Investigating why a tenant's overlay isn't working. - Planning capacity on a CP host (port range, container count). - Manually reaping a stuck fleet member. - Cleaning up the per-tenant Postgres schema after a tenant is terminated. - Diagnosing a spawn failure. ## What you need - Operator (root) access on the CP host running amini-admission. - `psql` reachable at the Amini Cloud Postgres (`docker exec -it amini-postgres-1 psql -U Amini Cloud -d Amini Cloud` in the compose dev stack). - `docker ps`, `docker logs`, `docker inspect`. ## Mental model Every tenant gets exactly ONE fleet member. ``` ┌────────────────────────────────────────────┐ │ amini.tenants │ │ ────────────── │ │ id, slug, headscale_url, headscale_schema│ └────────────────────────────────────────────┘ │ tenant_id (PK) ▼ ┌────────────────────────────────────────────┐ │ amini.headscale_fleet (migration 0040) │ │ ────────────────────────────────────── │ │ tenant_id, container_id, port, │ │ schema_name, status, last_error │ └────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────┐ │ docker container │ │ name: amini-headscale- │ │ image: headscale/headscale:0.28.0 │ │ bind: 127.0.0.1::8080 │ │ network: slate_default │ │ label: amini.role=headscale-fleet │ └────────────────────────────────────────────┘ ``` The `demo` tenant is special: its container is owned by docker-compose, NOT by admission. The fleet records a synthetic row for `demo` pointing at `http://amini-headscale-demo:8080` but never spawns or reaps that container. ## Capacity planning ### How many fleet members can a single CP host run? **MVP cap: 200.** The fleet allocates loopback ports from the range `[51000, 51200)`. Each fleet member also holds an open Postgres connection, a Go-side HTTP client to the engine socket, and a container that itself opens ~10 file descriptors for Headscale's inner SQLite (or Postgres) backend. In practice: | Tenants | Memory (per host) | Loopback ports used | |---------|-------------------|---------------------| | 1 | ~80 MB | 50001 (demo) | | 10 | ~700 MB | 50001 + 51000..51008| | 100 | ~6.5 GB | 50001 + 51000..51098| | 200 | ~13 GB | 50001 + 51000..51198| Beyond ~200 the loopback port range is exhausted (`ErrNoCapacity` on `/v1/tenants` POST). Operators should scale horizontally — multiple CP hosts, each hosting up to ~200 fleet members. Cross-CP-host scheduling is wave-11 H2. ### Tuning the port range Override via env on amini-admission: ```bash AMINI_HEADSCALE_FLEET_PORT_BASE=51000 # default AMINI_HEADSCALE_FLEET_PORT_LIMIT=51200 # default ``` Note: these env vars are NOT yet read at admission boot — the MVP hard-codes the defaults inside `internal/headscale/fleet.go`. Wave-11 wires them through `internal/config`. Until then, edit the `DefaultPortBase` / `DefaultPortLimit` constants and rebuild. ### Where are the per-tenant Postgres schemas? Each fleet member uses a Postgres schema named `headscale_`, created by Headscale on its first boot via the auto-migration baked into the image. The schema name is stored in `amini.headscale_fleet.schema_name` for forensics. Inspect on the host: ```bash docker exec -it amini-postgres-1 psql -U amini -d amini \\ -c \"SELECT slug, container_id, port, schema_name, status FROM amini.headscale_fleet ORDER BY created_at\" ``` List the actual schemas Headscale has created: ```bash docker exec -it amini-postgres-1 psql -U amini -d amini \\ -c \"SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'headscale_%'\" ``` ## Day-to-day procedures ### Check fleet health ```bash # 1. What does the DB think? dock" }, { "slug": "operator/host-management", "title": "Host management", "description": "Add, drain, cordon, uncordon, and remove compute hosts. Pick the right onboarding script per compute class.", "section": "operator", "body": "# Host management Amini Cloud runs on a heterogeneous fleet — Raspberry Pis, workstations, Macs, datacenter servers, GPU nodes, DGX Spark, edge boxes, even gaming PCs. Each compute class has a tailored onboarding script under `scripts/`. This page is the operator handbook for adding capacity and taking it offline cleanly. ## When to use this - Adding a new compute host to an existing Amini Cloud cluster. - Performing maintenance on an existing host (drain, do work, un-drain). - Permanently removing a host. - Verifying the cluster's current capacity + class mix. ## What you need - Operator (root or sudoer) shell on the target compute host. - The Amini Cloud repo cloned on the target host (the onboarding scripts read templates from `deploy/`). - The control-plane URL the host should report to (`https://api.amini.example`). - Decision on the host's `compute_class` (eight options below). ## The eight compute classes The CHECK constraint on `amini.hosts.compute_class` (migration 0011) allows these eight values: | `compute_class` | Typical hardware | Onboarding script | Runtimes available | |---|---|---|---| | `rpi` | Raspberry Pi 4/5 / arm64 SBC | `scripts/onboard-rpi.sh` | podman, nspawn | | `workstation` | Desktop Linux 16-64 GB RAM | `scripts/onboard-workstation-linux.sh` | podman, qemu, kata, nspawn, firecracker (if KVM) | | `mac` | Apple Silicon Mac mini / Studio / MBP | `scripts/onboard-workstation-mac.sh` | podman (via Apple VM) | | `server` | 1U/2U rack server | `scripts/onboard-server.sh` | All runtimes | | `gpu-node` | Server with enterprise GPUs (A100/H100/L40S/MI300) | `scripts/onboard-gpu-node.sh` | All; firecracker + qemu with vfio | | `dgx-spark` | NVIDIA DGX Spark FE (Grace-Blackwell aarch64) | `scripts/onboard-dgx-spark.sh` | podman, qemu, kata, firecracker (KVM-arm) | | `gaming-pc` | Consumer-GPU desktop (RTX 4080/90, 5090 class) | `scripts/onboard-gaming-pc.sh` | podman, qemu, kata, firecracker (Linux only) | | `edge` | Industrial PC / Mini PC / Jetson | `scripts/onboard-edge.sh` | podman, firecracker (if kernel supports) | Two class-specific gotchas: - **`mac` hosts refuse untrusted-tenant slices.** macOS has no cgroups, no nftables, no KVM. The host-agent runs natively but can only host trusted-tenant workloads via Apple Virtualization Framework + `podman`. - **`gaming-pc` hosts refuse untrusted-tenant slices.** Consumer GPUs lack hardware multi-tenancy partitioning; sharing a 4090 across two untrusted tenants is unsafe. The host-agent enforces this at registration time. WSL2 is also refused — Linux only. - **`dgx-spark` has no BMC.** Remote power-cycle requires an external smart plug or KVM. See the [dgx-spark-remote-ops skill](https://github.com/aminitech/amini-cloud/tree/main/.claude/skills/dgx-spark-remote-ops) for the full ops model. The host-agent's lifeline still works; what's missing is out-of-band recovery when the OS is wedged. ## The procedure ### 1. Add a host Run the right script on the host. Linux server example: ```bash ssh amini-compute-new.example.com cd ~/amini # or wherever you cloned the repo sudo bash scripts/onboard-server.sh \\ --admission-addr=https://api.amini.example \\ --region=nairobi \\ --zfs-disk=/dev/nvme1n1 \\ --labels=rack=A1,owner=ulap ``` Each script: - Validates prerequisites (kernel features, free memory, free disk). - Installs the matching runtimes (Podman + Firecracker + optional QEMU/Kata for trusted-capable hosts). - Installs the host-agent binary + the systemd unit (`scripts/host-agent-systemd.service`). - Configures cgroups v2 + user namespaces (Linux). - Generates a host certificate, registers it with admission's `/v1/hosts/preflight` (Year-1 endpoint; current bring-up uses a shared bootstrap token). - Starts the host-agent service. - Verifies the first heartbeat reached admission before exiting. Scripts are idempotent — re-running is safe. For macOS workstations the equivalent is: ```bash # macOS-specific bash scripts/onboard-workstation-mac.sh \\ --admission-addr=https:/" }, { "slug": "operator/ingress-and-domains", "title": "Ingress and custom domains", "description": "Wave-9 ships a Railway-/Vercel-shape sharable-URL story for every running slice plus a tenant-admin BYO-domain flow with auto-TLS via Let's Encrypt. This guide covers the operator-side configuration.", "section": "operator", "body": "# Ingress and custom domains Wave-9 ships a Railway-/Vercel-shape sharable-URL story for every running slice plus a tenant-admin BYO-domain flow with auto-TLS via Let's Encrypt. This guide covers the operator-side configuration. ## What runs where | Component | Where it lives | Touches | |-----------------------|----------------------------------------------------|-----------------------| | ingress-caddy | `deploy/compose/compose.yml` service (Caddy 2.10) | `:80`, `:443`, `:2019`| | Caddyfile template | `deploy/ingress/caddy/Caddyfile.tmpl` | (rendered on boot) | | Admin-API bootstrap | `deploy/compose/ingress-caddy/admin-bootstrap.json`| `POST /load` | | Reconciler | `internal/admission/ingress_reconciler.go` | every 5 s | | Route table | `amini.slice_ingress_routes` (migration 0031) | one row per exposure | | Tenant domains table | `amini.tenant_domains` (migrations 0016 + 0031) | one row per BYOD | The reconciler is part of the admission process — there is **no separate \"ingress controller\" process**. When admission restarts the reconciler re-applies every active route on the next tick. Caddy keeps its in-memory config across the admission restart, so traffic flows without a blip. ### How Caddy reaches the slice (wave-9 K2) Every slice container joins the same docker network as `amini-ingress-caddy` — **`slate_default`** by default, configurable via `PodmanLocalRunner.NetworkName`. The runner registers a docker-DNS alias of the form `amini-<8hex-of-slice-id>` on that network, so the ingress sidecar can dial the slice by container name. The K1 hostname-target wire format is: ``` http://amini-: ``` No host port-publish is required. The previous `host.docker.internal:` fallback is now gated on `AMINI_RUNNER_HOST_INSIDE` and only used for non-compose deployments where the runner DOES publish ports. The architectural record is [ADR 0050](../../decisions/0050-slice-docker-dns-attach.md); the seam is `internal/ingress/types.go: Upstream.DialOverride`. This has two operator-visible consequences: - A slice that doesn't appear in `docker network inspect slate_default` is unreachable from ingress. If a slice was created before `slate_default` existed (rare; see the wave-9 K2 commit), recreate it; the runner will attach on the next start. - Two slices on the same `slate_default` network can dial each other at the docker layer. Tenant isolation across listening ports stays enforced at the amini-overlay layer (Headscale ACLs); per-tenant docker-network isolation is tracked under wave-10. ## Environment variables | Variable | Default | Purpose | |------------------------------|----------------------------------------------------|------------------------------------------------------------------| | `AMINI_INGRESS_BASE_DOMAIN` | `amini.localtest.me` | Wildcard suffix for auto-URLs. | | `AMINI_INGRESS_URL_PORT` | `8095` | Port advertised in the URL. Match the compose port mapping. | | `AMINI_INGRESS_ADMIN_URL` | `http://ingress-caddy:2019` | Reconciler target. In-compose service name. | | `AMINI_INGRESS_HTTP_PORT` | `8095` | Host port for Caddy's `:80`. | | `AMINI_INGRESS_HTTPS_PORT` | `8446` | Host port for Caddy's `:443`. | | `AMINI_INGRESS_ADMIN_PORT` | `2019` | Host port for Caddy's admin API (loopback only). | | `AMINI_INGRESS_ACME_EMAIL` | `dev@amini.localtest.me` | Required by Let's Encrypt for prod certs. | | `AMINI_INGRESS_ACME_CA` | `https://acme-v02.api.letsencrypt.org/directory` | LE prod by default — staging directory works for testing. | ## Dev quickstart (laptop, no DNS) The defaults Just Work. `*.localtest.me` resolves to `127.0.0.1` publicly, so: ```bash cd deploy/compose docker compose up -d amini-admission postgres ingress-caddy # Create a slice with an exposure: curl -X POST http://localhost:18080/v1/slices \\ -H 'authorization: Bearer …' \\ -d '{ \"name\":\"pgdemo\", \"image\":\"postgres:16-alpine\", \"exposure\":[{\"port\":5432,\"proto\":\"tcp\",\"scope\":\"tenant\"}] }' # Within ~10s: curl http://localhost:" }, { "slug": "operator/install", "title": "Install Amini Cloud (single-host quickstart)", "description": "> ⚠️ **This guide (2026-05-15) predates the current `make up` flow and is kept for > history.** The web console now ships *inside* compose (`amini-web` on `:5173`, > `deploy/compose/compose.yml`), `make quickstart` is the one-command path, and some > secret names below are out of", "section": "operator", "body": "# Install Amini Cloud > ⚠️ **This guide (2026-05-15) predates the current `make up` flow and is kept for > history.** The web console now ships *inside* compose (`amini-web` on `:5173`, > `deploy/compose/compose.yml`), `make quickstart` is the one-command path, and some > secret names below are out of date. **For a current setup start at > [operators/agent-onboarding.md](../../operators/agent-onboarding.md)** and > [operators/credentials-and-access.md](../../operators/credentials-and-access.md); > the README is the current single-host quickstart. This is the canonical install path for a fresh operator. It resolves prior doc disagreement between `README.md`, install/single-host.md, the `Makefile`, and tutorial 00 about whether secrets must be pre-created (they aren't), whether the web UI is part of `make up` (it isn't), and whether `make vault-bootstrap` runs by itself (it doesn't). ## TL;DR — 4 commands ```bash git clone git@github.com:aminitech/amini-cloud.git && cd amini-cloud make up # full compose stack + auto-bootstrap secrets make vault-bootstrap # initialize OpenBao (orphan from make up; see §\"Vault bootstrap\") cd web && pnpm install && pnpm dev # web UI on :5173 (separate shell) ``` Open `http://localhost:5173` and sign in as `admin@demo` (dev-bearer mode). ## Prerequisites | Component | Version | Notes | |---|---|---| | Linux or macOS | any recent | tested: Ubuntu 22.04, Debian 12, macOS 14 | | Docker + Compose | docker 24+, compose 2.20+ | `docker compose version` | | Go | 1.22+ | for the `amini` and `amini-admission` binaries | | Node | 20+ | for the web UI | | pnpm OR npm | pnpm 8+ preferred | npm works as a fallback | | RAM | ~16 GB | Postgres + Headscale + Caddy + ingress-caddy + Authelia + Vault + admission + Web | | Disk | ~50 GB | Postgres state + image cache (`docker images` grows fast) | Optional (for the 3-host production path, currently scaffolded only): - An additional VPS for the control-plane (amini-cp) - A working tailnet (Tailscale OAuth client OR Headscale auto-deploy) - A real domain for production ingress (Caddy will auto-issue Let's Encrypt certs) ## What `make up` actually does The hidden product win: `make up` is **fully self-bootstrapping** if your prerequisites are met. Specifically: 1. Runs `scripts/bootstrap-compose-secrets.sh` (idempotent). This generates: - `secrets/pg-password` — Postgres credential - `secrets/headscale-admin-token` — Headscale admin API token - `secrets/authelia-jwt-secret`, `authelia-session-secret`, etc. — Authelia secrets - `secrets/minio-root-user`, `secrets/minio-root-password` — MinIO admin creds - `secrets/cluster-id` — Amini Cloud's cluster ID 2. Runs `docker compose up -d` against `deploy/compose/compose.yml`, which brings up: - `amini-postgres-1` — Postgres 16 (the platform DB) - `amini-vault-1` — OpenBao (vault, see §\"Vault bootstrap\") - `amini-headscale-demo-1` — the single shared Headscale process for the `demo` tenant - `amini-authelia-1` + `amini-authelia-proxy-1` — OIDC provider - `amini-derper-1` — Tailscale DERP relay - `amini-minio-1` — MinIO (operator's S3 storage, NOT a slice) - `amini-caddy-1` — control-plane Caddy (admission + Authelia routes) - `amini-ingress-caddy-1` — per-slice ingress Caddy - `amini-amini-admission-1` — the Amini Cloud API server 3. Migrations apply automatically on Postgres startup via `docker-entrypoint-initdb.d`. New migrations land via `make db-migrate-pending`. You do NOT pre-create any secrets manually. You do NOT touch `secrets/` until `make up` has succeeded. ## Vault bootstrap `make vault-bootstrap` is **orphan from `make up`** — you must run it as a second step. It: 1. Calls `scripts/bootstrap-vault.sh`. 2. Initializes OpenBao if not already initialized; persists shamir keys + root token to `deploy/compose/secrets/vault-keys.json` (mode 0600). Production operators replace this with cloud KMS auto-unseal. 3. Unseals if sealed. 4. Enables KV v2 at `secret/`. 5. Enables AppRole auth. 6. Writes the three Amini Cloud polic" }, { "slug": "operator/monitoring", "title": "Monitoring", "description": "Prometheus + Grafana + Loki wiring, the 12 default KPIs, and the alert routing Amini Cloud ships out of the box.", "section": "operator", "body": "# Monitoring Amini Cloud is an honest measurement product. The platform measures its own reliability against the same definitions it advertises, persists those measurements in Postgres, and exports them to Prometheus / Loki so operators can dashboard, alert, and audit without writing their own glue. This page is the operator handbook for wiring the observability stack up. The decisions behind the metric and KPI shape live in [ADR 0020](../../decisions/0020-uptime-kpi-sla.md) and [ADR 0018](../../decisions/0018-lifeline-self-healing.md). The conceptual overview is in [Uptime and SLA](../concepts/uptime-and-sla.md). ## When to use this - Right after a fresh `make up` or `make ansible-up`, to attach dashboards. - After upgrading Amini Cloud to a release that added KPIs (GPU KPIs landed in `internal/sla/gpu_kpis.go`; the catalog grows additively). - When you need to debug \"why is admission slow\" and want a panel that isn't `journalctl`. ## What you need - Prometheus 2.45+ reachable from `amini-cp` and every compute host. - Grafana 10+ pointed at the Prometheus datasource. - Loki 2.9+ if you want log aggregation (optional). - Reachable HTTPS for the amini-admission scrape endpoint. ## The pieces | Producer | Endpoint | What it emits | |---|---|---| | `amini-admission` | `/metrics` on the admission port | `slate_*` metrics: provision counters, audit-write latency, prober samples, leader status, KPI snapshot ticks | | `amini-host-agent` | `/metrics` on the host-agent gRPC port (HTTP sidecar) | `slate_host_*`: heartbeat age, lifeline state gauge, netns/anchor counts, per-slice resource attribution | | `node_exporter` | port 9100 per host | `node_*`: cpu, mem, disk, fs, network | | `postgres_exporter` | port 9187 on `amini-cp` | `pg_*`: connections, replication lag, slow queries | | `nomad` agent | `/v1/metrics?format=prometheus` per host | `nomad_*`: scheduler queue, alloc state, client capacity | | `dcgm-exporter` | port 9400 on every `gpu-node` / `dgx-spark` | `DCGM_FI_*`: GPU utilization, VRAM, temperature, power | The exporters that ship in `deploy/compose/compose.yml` are `postgres_exporter` and the admission `/metrics` endpoint. The rest are opt-in via `scripts/onboard-*.sh` (the onboarding script installs node_exporter + dcgm-exporter alongside the host-agent). ## Wire Prometheus Point Prometheus at the Amini Cloud jobs. A minimal `prometheus.yml` snippet: ```yaml scrape_configs: - job_name: amini-admission static_configs: - targets: ['amini-cp:18080'] # Compose default; older builds used :8080. See deploy/compose/.env.example. metrics_path: /metrics - job_name: amini-host-agent static_configs: - targets: ['amini-compute-1:9101', 'amini-compute-2:9101'] - job_name: node-exporter static_configs: - targets: - 'amini-cp:9100' - 'amini-compute-1:9100' - 'amini-compute-2:9100' - job_name: postgres static_configs: - targets: ['amini-cp:9187'] - job_name: nomad metrics_path: /v1/metrics params: { format: ['prometheus'] } static_configs: - targets: ['amini-cp:4646', 'amini-compute-1:4646'] - job_name: dcgm static_configs: - targets: ['amini-compute-gpu-1:9400'] rule_files: - /etc/prometheus/amini-alerts.yml ``` `deploy/observability/alerts.yml` is the canonical alert rule set — copy it to `/etc/prometheus/amini-alerts.yml` and reload. ## The 12 default KPIs The catalog is a static array in `internal/sla/kpi.go` plus a GPU-specific extension in `internal/sla/gpu_kpis.go`. Eight base KPIs plus four GPU KPIs land in `DefaultKPIs`: | ID | Default target | Unit | Source | |---|---|---|---| | `cluster_uptime` | 0.995 | fraction | mean of per-host uptime, `IntegrateUptime` | | `slice_availability_p95` | 0.99 | fraction | p95 across slices, reachability prober | | `provisioning_latency_p95` | 60 | seconds | admission timing of `POST /v1/slices` → `running` | | `mttr_host` | 600 | seconds | lifeline `degraded → healthy` durations | | `mtbf_host` | 2592000 (30d) | seconds | lifeline `healthy → degraded` intervals | | `audit_integrity` | 1.0 | fr" }, { "slug": "operator/runbooks", "title": "Runbooks", "description": "Operator response playbooks for the common incident shapes — host hard-fail, pre-auth-key compromise, replication lag, tenant abuse, SLA breach, Headscale hang, vault sealed, DNS leak, oracle score crash, GPU failure, ingress TLS failure.", "section": "operator", "body": "# Runbooks Each section is a self-contained playbook. Format is: **Symptom → Triage → Verify → Mitigate → Resolve → Postmortem trigger.** Every shipped alert in `deploy/observability/alerts.yml` has a `runbook` annotation that links into one of these sections. The accuracy contract from [`README.md`](../README.md) applies here too — every Postgres query below was run against migration 0001-0008 + 0010-0016 schema; every file path is real; every CLI command marked **(scaffolded)** returns \"not yet implemented\" today and is documented so the procedure survives into Year-1. ## 1. Host hard-fails **Symptom.** `AminiHostOffline` page fires. `amini.hosts.last_heartbeat` is older than 60s. Slices on that host are flipping to `failed`. **Triage (60 seconds).** ```bash # What slices were on that host? psql \"$POSTGRES_DSN\" -c \" SELECT id, tenant_id, name, status, runtime FROM amini.slices WHERE host_id = '' AND status IN ('running','scheduled','pending'); \" # What does Nomad say? nomad node status # Anything in the host_uptime_events trail? psql \"$POSTGRES_DSN\" -c \" SELECT state, started_at, ended_at, source FROM amini.host_uptime_events WHERE host_id = '' ORDER BY started_at DESC LIMIT 10; \" ``` **Verify.** Is this network or hardware? - SSH the host. If SSH works but the agent is dead: agent crash — `journalctl -u amini-host-agent --since='10 min ago'`. - If SSH hangs and IPMI/console is reachable: kernel panic or disk full — check console log. - If nothing responds: power / network / hardware. For a DGX Spark FE with no BMC, fall through to runbook #11 (GPU/host recovery without BMC). **Mitigate.** 1. Cordon the host so the scheduler stops trying: ```bash nomad node eligibility -disable ``` 2. The lifeline sweeper (ADR 0018) will flip `hosts.status` to `offline` within 60 seconds and stop assigning to it. 3. Slices with placement policy `replicated` or `sharded` get rescheduled automatically on un-cordoned peers. `single-host` slices stay `failed` until the host returns or you delete them. **Resolve.** - Hardware: replace; re-onboard via `scripts/onboard-.sh`. - Process: `sudo systemctl reset-failed amini-host-agent && sudo systemctl start amini-host-agent`. - Disk full: extend or clear `/var/lib/amini-host-agent` and `/run/netns/amini-*`. **Postmortem trigger.** Any `page`-severity host outage that affected more than one tenant. ## 2. Pre-auth-key compromise **Symptom.** A device you don't recognise registered into a tenant's tailnet. Or: a stale audit row in `amini.preauth_key_issuances` matches an unknown 8-char prefix observed in a Headscale log. **Triage.** ```bash # Last 1 hour of pre-auth-key issuance for the tenant psql \"$POSTGRES_DSN\" -c \" SELECT id, tenant_id, slice_id, key_prefix, issued_at, used_at, ttl_seconds FROM amini.preauth_key_issuances WHERE tenant_id = '' AND issued_at > now() - interval '1 hour' ORDER BY issued_at DESC; \" ``` The full key value is NEVER stored — only the first 8 chars (`db/migrations/0001_initial.sql`). If you see a `key_prefix` you did not issue, the upstream provisioner is compromised. **Mitigate (immediately).** 1. **Revoke at Headscale.** For each tenant's Headscale process: ```bash ssh amini-cp 'sudo headscale -c /etc/headscale/.yaml \\ preauthkeys list --user ' ssh amini-cp 'sudo headscale -c /etc/headscale/.yaml \\ preauthkeys expire ' ``` 2. **Rotate the Headscale admin token.** It lives at the vault path `amini/headscale/admin-token`. Re-bootstrap via `scripts/bootstrap-vault.sh` and restart `amini-admission`. 3. **Audit the tailnet for unknown devices.** Per tenant: ```bash ssh amini-cp 'sudo headscale -c /etc/headscale/.yaml nodes list' ``` Any device whose registration time falls inside the compromise window: `headscale nodes delete `. **Resolve.** Investigate how the key leaked. Common causes: - SEC-002 (open audit finding): plaintext pre-auth key embedded in Nomad job spec via `Emb" }, { "slug": "operator/tailscale-fronting", "title": "Operator runbook — Tailscale-fronting Amini Cloud (control plane, slices, Funnel)", "description": "**Audience:** the operator running Amini Cloud on a CGNAT host (no public inbound). **Companion:** `deploy/tailscale/README.md` (bootstrap recipe), `docs/audits/2026-05-15-acceptance-L4-tailscale.md` (the L4.2 result).", "section": "operator", "body": "# Operator runbook — Tailscale-fronting Amini Cloud (control plane, slices, Funnel) **Audience:** the operator running Amini Cloud on a CGNAT host (no public inbound). **Companion:** `deploy/tailscale/README.md` (bootstrap recipe), `docs/audits/2026-05-15-acceptance-L4-tailscale.md` (the L4.2 result). This runbook is the exact, copy-pasteable procedure for putting Amini Cloud's control plane and slices behind Tailscale with a **real, browser-trusted TLS cert**, plus the **Funnel** path that gives a CGNAT box a genuine *public* HTTPS URL without a VPS or an owned domain. --- ## Mental model: serve vs funnel (read this first) | | `tailscale serve` | `tailscale funnel` | |---|---|---| | Reachable from | **Only your tailnet** (ACL-gated devices) | **The entire public internet** | | TLS cert | Real Let's Encrypt, `*.ts.net` | Real Let's Encrypt, `*.ts.net` (same) | | Needs public inbound? | No (works through CGNAT) | No (Tailscale relays — works through CGNAT) | | Needs ACL change? | No (uses existing tailnet ACL) | **Yes** — `nodeAttrs` `funnel` grant | | Use for | Control plane, `tenant`/`shared` slices | `scope=public` slices ONLY | | Security gate | Network (tailnet) + Amini Cloud authz | Amini Cloud app authz **only** | Rule of thumb: **default to serve. Funnel is a deliberate decision per public endpoint, never the control plane.** --- ## The sacrosanct rule (do not break the existing stack) This host already runs ~26 `tailscale serve` mappings for an unrelated production ops/monitoring stack (dgx-ops: Beszel, Kuma, ttyd, AminiChain, DCGM, etc.). **Never** remove, overwrite, or reconfigure an existing mapping. Only ever **ADD** a mapping on a fresh, verified-free port. Before any change: ```bash tailscale serve status > /tmp/serve-before.txt tailscale serve status | grep -c spark-5804 # record the count ``` After any change, confirm count is exactly +N and every prior line survives: ```bash diff <(grep -o 'spark-5804[^ ]*' /tmp/serve-before.txt | sort -u) \\ <(tailscale serve status | grep -o 'spark-5804[^ ]*' | sort -u) # should show ONLY the newly added port(s) as additions, zero deletions ``` --- ## Recorded baseline (2026-05-15, the state proven not-broken) - `tailscale version`: **1.96.4** (go1.26.1) - `tailscale status`: `spark-5804` = `100.113.57.52` / `spark-5804.tail4e3a73.ts.net`, peers `m2mkm`, `admin`, `ipad153`. (A benign \"can't reach configured DNS servers\" health note — outbound DNS flap; MagicDNS/serve unaffected.) - `tailscale serve status`: **26 mappings, all `tailnet only`.** Of note: ports **8461** and **18443** *already* proxy `localhost:18080` (amini-admission) — pre-existing, must not be touched. - `tailscale funnel status`: same entries, **all `(tailnet only)`** — nothing on this box is actually Funnel-public. Funnel is *capable* but *unexposed*. - This runbook ADDED exactly one mapping: **8480 → 127.0.0.1:18080** (tailnet-only). 26 → 27. Verified zero prior removals. --- ## 1. Front amini-admission (the control plane) — tailnet only amini-admission listens on `127.0.0.1:18080`. Pick a free high port. ```bash # 1. verify the port is free (must print \"free\") PORT=8480 ss -tlnp | grep -q \":$PORT \" && echo \"IN USE — pick another\" || echo \"$PORT free\" tailscale serve status | grep -q \":$PORT \" && echo \"IN USE in serve\" || echo \"$PORT free in serve\" # 2. add the mapping (additive; mutates no existing entry) tailscale serve --bg --https=$PORT --set-path=/ http://127.0.0.1:18080 # 3. prove it: real cert, no -k curl -fsS https://spark-5804.tail4e3a73.ts.net:$PORT/healthz # -> {\"status\":\"ok\"} curl -fsS -o /dev/null -w 'ssl_verify=%{ssl_verify_result}\\n' \\ https://spark-5804.tail4e3a73.ts.net:$PORT/healthz # -> ssl_verify=0 (0 == chain valid against the default trust store) ``` > **No sudo needed on this host** — the `ulap01` user holds Tailscale > operator rights, and `serve --bg` on a *new* port is purely additive. > If on another host `tailscale serve` refuses with a permission error, > run it as root or grant" }, { "slug": "operator/templates", "title": "Slice template catalog — operator reference", "description": "How the amini.slate_templates catalog works, how to add a row, the seed-fix lesson, the pre-flight smoke test, and the connect-hints registry.", "section": "operator", "body": "# Slice template catalog — operator reference The Amini Cloud slice catalog is a database table — `amini.slate_templates` — seeded from migrations and editable from `/admin/templates/` in the SPA. This page is the operator's view of what's actually wired in, how to add a new row safely, and why every column matters. The shipped wave-9 catalog is documented for tenants in [`../user/slice-templates.md`](../user/slice-templates.md). This page is the per-row authoring contract. ## What's in a template row ``` amini.slate_templates (defined in migrations 0003 + 0034 + 0035) ───────────────────────────────────────────────────────────────── id text PRIMARY KEY — e.g. 'jupyter', 'minio' tenant_id uuid NULL — NULL = built-in / shared display_name text NOT NULL description text NOT NULL category text NOT NULL — 'dev' | 'data' | 'ml' | … runtime text NOT NULL — 'podman' | 'firecracker' | … image text NOT NULL — pinned image reference default_cpu_milli int NOT NULL default_mem_bytes bigint NOT NULL default_disk_bytes bigint NOT NULL default_exposure jsonb NOT NULL DEFAULT '[]' default_env jsonb NOT NULL DEFAULT '{}' default_cmd text[] NOT NULL DEFAULT '{}' — wave-9 K6 default_args text[] NOT NULL DEFAULT '{}' — wave-9 K6 health_path text NULL — wave-9 K6 requires_gpu bool NOT NULL DEFAULT false requires_storage_classes text[] NOT NULL DEFAULT '{}' preflight_passed_at timestamptz NULL preflight_evidence jsonb NOT NULL DEFAULT '{}' created_at timestamptz NOT NULL DEFAULT now() updated_at timestamptz NOT NULL DEFAULT now() ``` The three wave-9 K6 columns (`default_cmd`, `default_args`, `health_path`) close the gap the K6 audit found: a catalog with no notion of `CMD`/`ENTRYPOINT` override couldn't ship images whose entrypoint expects a sub-command. See [ADR 0051](../../decisions/0051-template-cmd-args-health.md) for the architectural record. ## The seed-fix lesson — containers that exit immediately The K6 audit caught this in production: > The catalog row for MinIO had `default_args = '{}'`. The container > launched as just `minio` (no `server /data`), refused to start the > server because it had no `server` sub-command, and exited > immediately. The slice landed in `running` for ~2 seconds, the > container exited, the reconciler hadn't even ticked yet — the user > saw a \"running\" badge and clicked into an empty page. The fix is enforced at TWO points: 1. **At create time** (`internal/admission/server.go: handleCreateSlice`): if `spec.template_id` is set, admission calls `Q.GetTemplate(id)` and copies `default_cmd`/`default_args` onto the in-memory slice. `default_env` is merged (request env wins per-key). 2. **At provision dispatch** (`internal/admission/provision.go: provisionOne`): the provisioner re-reads the slice from the DB before handing to `runner.Start`. `db.Slice.Cmd`/`db.Slice.Args` carry `json:\"-\"` tags (not persisted) — they're transient. So the provisioner re-resolves from `slice.template_id` and re-populates `Cmd`/`Args` before dispatching. Without this step the K6.1 in-memory values are lost across the round-trip. If you're adding a template and your image is one of the \"needs an explicit sub-command\" cases (MinIO, MLflow, vLLM, etc.), populate `default_args` in the migration. If the image's `ENTRYPOINT` is itself the binary you want and `CMD` is the args (the common case), leave `default_cmd` empty and put the args in `default_args`. ## Pre-flight smoke test `health_path` gives the smoke tester a uniform liveness probe. The shipped pattern is `GET http://amini-:` — exact 200 within ~30s of the container reaching `State.Running = true`. `NULL = no HTTP probe` (Postgres only speaks tcp; the smoke tester falls back to a TCP-connect probe). Common values: | `health_path` | Used by | |-----------------------|----------------------| | `/` | jupyter, code-server | | `/health` | vllm-inference | | `/minio/health/live` | minio | | (NULL) | postgres, ssh-server | The smoke tester runs after `runner.Start` re" }, { "slug": "operator/tenant-administration", "title": "Tenant administration", "description": "Create, suspend, terminate tenants. Set quotas. Mark untrusted. Walk through the provisioning script.", "section": "operator", "body": "# Tenant administration This is the operator's reference for the full tenant lifecycle. For the guided first-time walkthrough see [Tutorial 05 — Onboard a tenant](../tutorials/05-onboard-a-tenant.md); this page is the day-to-day handbook. ## When to use this - Creating a new tenant. - Adjusting a tenant's quotas (CPU, memory, slice count). - Flipping a tenant to `untrusted=true` for hostile workloads. - Suspending a tenant (preserve data, refuse new admission). - Terminating a tenant (destructive). ## What you need - Platform admin role: a `tenant_users` row in the admin tenant (slug set via `ADMIN_TENANT_SLUG` env on amini-admission) with `role='admin'` or `role='owner'`. - Operator (root) access on `amini-cp`. - `psql`, `nomad`, `headscale`, `systemctl` on PATH. ## The procedure ### 1. Create a tenant via HTTP (preferred) `POST /v1/tenants` is the canonical path. It writes a `tenant.create` audit event and falls back to the same SQL the script uses. ```bash curl -fsS -X POST https://api.amini.example/v1/tenants \\ -H \"Authorization: Bearer $ADMIN_TOKEN\" \\ -H \"Content-Type: application/json\" \\ -d '{ \"slug\": \"acme\", \"display_name\": \"Acme Inc\", \"oidc_issuer\": \"https://auth.amini.example\", \"headscale_url\": \"https://acme.amini.example\", \"headscale_schema\": \"headscale_acme\", \"ipv6_ula_prefix\": \"fd5a:7e2:abcd::/48\", \"cpu_milli_quota\": 8000, \"mem_bytes_quota\": 17179869184, \"slice_count_quota\": 20, \"untrusted\": false }' | jq ``` `201 Created` returns the new tenant row. Caveat: this only writes the DB row. The full bring-up (Postgres schema, Headscale unit, Nomad namespace) is in `scripts/provision-tenant.sh`. For now run the script after the API call, or use the script alone (it's idempotent and ON CONFLICT DO UPDATE). ### 2. Create a tenant via the reference script ```bash sudo bash scripts/provision-tenant.sh \\ --slug acme \\ --display-name 'Acme Inc' \\ --oidc-issuer https://auth.amini.example ``` Optional env knobs: ```bash TENANT_CPU_MILLI=16000 \\ TENANT_MEM_BYTES=34359738368 \\ TENANT_SLICE_COUNT=50 \\ TENANT_IPV6_PREFIX=fd5a:7e2:abcd::/48 \\ sudo -E bash scripts/provision-tenant.sh \\ --slug acme \\ --display-name 'Acme Inc' \\ --oidc-issuer https://auth.amini.example ``` The script's full step list lives in its header (`sed -n '4,50p' scripts/provision-tenant.sh`). ### 3. Change a quota Quotas are columns on `amini.tenants`: ```bash psql \"$POSTGRES_DSN\" -c \" UPDATE amini.tenants SET cpu_milli_quota=16000, mem_bytes_quota=34359738368, slice_count_quota=50 WHERE slug='acme'; \" ``` The change takes effect immediately. Existing slices are unaffected; new `POST /v1/slices` calls use the updated quota. Recommend also updating the Nomad quota: ```bash nomad quota apply <` (scaffolded; returns \"not yet implemented\" in this beta). Effects: - The auth middleware refuses every request from the tenant's users (Year-1 — currently `status` is read a" }, { "slug": "operator/upgrade-procedure", "title": "Upgrade procedure", "description": "Channel selection, pre-flight check, rollout, rollback. The safe path to a new Amini Cloud release.", "section": "operator", "body": "# Upgrade procedure This is the operator-facing rollout handbook. The compatibility policy that makes upgrades safe lives in [ADR 0010](../../decisions/0010-versioning-and-backward-compatibility.md); the schema-discipline rules live in [`installation/upgrading.md`](../installation/upgrading.md). This page is the day-of-rollout checklist. ## When to use this - A new Amini Cloud release is out and you want to land it without surprising any tenants. - A critical security finding has been pushed to the `stable` channel and you need to upgrade today. - You're moving from `beta` to `stable` channels (or vice versa) on a staging cluster. ## What you need - Operator (root) shell on `amini-cp` and every compute host. - A recent Postgres snapshot (per [Backup and restore](backup-and-restore.md)). - A maintenance window agreed with tenants — at least 30 minutes. - Read the [changelog](../changelog.md) entry for every version between yours and the target. ## Release channels Amini Cloud ships three channels. Tags in git are the source of truth. | Channel | Example tag | When to use | |---|---|---| | `stable` | `v0.1.0`, `v0.2.0`, ... | Production. Has gone through reviewer pipeline + audit. | | `beta` | `v0.2.0-beta.1`, ... | Pre-release. Schema is stable; bugs likely. | | `nightly` | `v0.2.0-nightly.20260515`, ... | Tip of `main`. Anything goes. Useful for dev clusters. | You pick a channel by checking out the matching tag. There's no in-cluster channel switch — it's a git checkout. ## Pre-flight checks ### 1. Read the changelog ```bash cd /path/to/amini git log --oneline --pretty=format:'%h %s' v..v | head -50 ``` Open [`docs/guides/changelog.md`](../changelog.md) and read every entry between your current version and the target. Schema migrations are forward-only and idempotent — that doesn't mean free. ### 2. Verify the current version ```bash psql \"$POSTGRES_DSN\" -c \" SELECT component, semver, commit_sha, started_at FROM amini.server_versions ORDER BY started_at DESC LIMIT 10; \" ``` The `server_versions` table (migration 0005) records every amini-admission start. Match the most-recent `amini-admission` row to the git tag you have checked out. ### 3. Check the feature catalog for deprecations ```bash psql \"$POSTGRES_DSN\" -c \" SELECT id, status, deprecated_in, removal_date FROM amini.slate_features WHERE removal_date IS NOT NULL ORDER BY removal_date; \" ``` Any row with `removal_date` in the past against the target version means a tenant is using a capability that will disappear. Migrate that tenant before upgrading. ### 4. Snapshot Postgres ```bash pg_dump \"$POSTGRES_DSN\" -F c -f \"amini-pre-upgrade-$(date +%F).dump\" ``` Encrypt + ship to your snapshot store. See [Backup and restore](backup-and-restore.md). ### 5. Pre-flight check (Year-1) ```bash # Scaffolded; returns \"not yet implemented\" in this beta amini admin upgrade --check ``` The intent: this command queries `server_versions` + `slate_features` + `amini.hosts` + active slice runtimes and reports any compat issues before you actually upgrade. Until it's wired, do the manual checks above. ## Rollout Order matters: Postgres first, then admission, then host-agents. ### 1. Apply migrations From a workstation that can reach Postgres: ```bash git fetch --tags git checkout v POSTGRES_DSN=\"$POSTGRES_DSN\" make db-migrate ``` `make db-migrate` walks `db/migrations/*.sql` in alphabetical order and runs each through `psql`. Each migration ends with `INSERT INTO schema_migrations (version) VALUES (N) ON CONFLICT DO NOTHING` so re-running is safe. If a migration fails mid-flight, the per-migration `BEGIN; ... COMMIT;` envelope rolls back the partial work; fix the cause and re-run. ### 2. Suspend or warn tenants If the rollout will visibly degrade service (a compute host drain on a Postgres slice with `local-zfs`), give tenants notice via your operator channel. Suspend is also an option: ```bash psql \"$POSTGRES_DSN\" -c \" UPDATE amini.tenants SET status='suspended' WHERE slug" }, { "slug": "packaging/amini-as-a-cloud-platform", "title": "Amini Cloud as a cloud platform", "description": "The same Amini Cloud that runs on a single mini-PC in a homelab also runs as a public-facing cloud platform: a control plane in one region, compute fleets in several, a per-tenant overlay, billing that produces real invoices, a status page, the works. This document explains how t", "section": "packaging", "body": "# Amini Cloud as a cloud platform The same Amini Cloud that runs on a single mini-PC in a homelab also runs as a public-facing cloud platform: a control plane in one region, compute fleets in several, a per-tenant overlay, billing that produces real invoices, a status page, the works. This document explains how to package and operate Amini Cloud as a cloud platform you can charge money for. ## What changes when Amini Cloud becomes a cloud platform The codebase is the same. The shape of the deployment, the contracts with users, and the supporting fabric change: | Concern | Appliance | Cloud platform | | -------------------- | ------------------------------- | ------------------------------------------- | | Customer per host | usually 1 (the operator's org) | many — multi-tenant by design | | Control plane | co-located with compute | separated, HA, geographically distributed | | Public DNS | optional | mandatory (every tenant gets a sub-domain) | | Inbound traffic | over the tailnet | over a public ingress + ACME certs | | Billing | manual / chargeback CSV | automated monthly invoices | | Compliance | operator-bound | platform-bound (SOC 2, GDPR, ISO 27001…) | | Status / SLA | operator's internal Beszel | public status page + per-tenant SLA report | | Support | one operator | tiered support org | | Pricing | flat / unbilled | usage-metered + plans | Amini Cloud's design accommodates both shapes without forks. The cloud- platform mode is just \"appliance mode, multiplied, with public ingress turned on and the billing pipeline enabled\". ## Reference topology ```mermaid flowchart TB subgraph PublicEdge[Public edge - per region] LB[Anycast load balancer] CDN[Static asset CDN] end subgraph ControlPlaneEU[Control plane EU] AdmA[amini-admission #1] AdmB[amini-admission #2] AdmC[amini-admission #3] PG[(Patroni-Postgres, 3-node)] BAO[OpenBao - HA, 3-node] HS[Headscale fleet] end subgraph ComputeEU[Compute fleet EU] direction LR H1[host-agent #1] H2[host-agent #2] H3[host-agent #3] HN[host-agent #N] end subgraph ComputeUS[Compute fleet US] UH1[host-agent #1] UHN[host-agent #N] end CDN --> LB --> AdmA & AdmB & AdmC AdmA <--> PG AdmB <--> PG AdmC <--> PG AdmA <--> BAO AdmA --> H1 & H2 & H3 & HN AdmA --> UH1 & UHN Billing[amini-admission billing pipeline] --> PG Billing --> Invoicer[Invoice generator] Invoicer --> Stripe[Stripe / KES MoMo] ``` The five horizontal slabs are layered: 1. **Public edge** — anycast LB, CDN, the only public-IP surface. 2. **Control plane** — 3× amini-admission behind the LB, a 3-node Patroni-Postgres cluster, 3-node OpenBao HA, the per-tenant Headscale fleet. 3. **Compute** — N hosts per region running amini-host-agent. Hetero- geneous classes are fine (some `server`, some `gpu-node`, some `dgx-spark`). 4. **Billing pipeline** — runs out of amini-admission, scrapes Prometheus / cAdvisor for per-tenant usage, emits monthly invoices (Stripe in the US/EU, M-Pesa MoMo in Kenya / east-Africa). 5. **Public ingress** — Caddy + ACME, one wildcard cert per region. ## Stand it up ### 1. Build the control-plane Packer image once ```bash make appliance-image CLASS=server ARCH=amd64 # → dist/packer/qemu-server-amd64/amini-appliance.qcow2 ``` Upload to your IaaS as an AMI / GCE image / OpenStack image. Same template, different builder; the Packer files in `build/packer/` ship with `amazon-ebs`, `googlecompute`, and `qemu` sources. ### 2. Deploy a three-host control plane ```bash cd deploy/ansible cp inventory.example.ini inventory.cloud.ini $EDITOR inventory.cloud.ini # three CP hosts, one compute host ansible-playbook -i inventory.cloud.ini site.yml \\ -e \"vault_token=${VAULT_TOKEN}\" \\ -e \"tailscale_authkey=${TAILSCALE_AUTHKEY}\" ``` The playbook runs OpenBao bootstrap → Postgres clustering → admission rolling deploy → Tailscale expose → first compute host → smoke-test. ### 3. Wire up public ingress Amini Cloud ships a Caddy snippet in `deploy/caddy/` that: - terminates ACME on the LB, - proxies `*.` to amini-admissi" }, { "slug": "packaging/amini-as-an-appliance", "title": "Amini Cloud as an appliance", "description": "Amini Cloud runs on commodity hardware. That means every shape of \"appliance\" is on the table: a single mini-PC on a shelf in an office, a stack of beige-box workstations in a homelab, a 1U pizza box bolted into a rack, or a hardened arm64 board zip-tied to the side of a network ", "section": "packaging", "body": "# Amini Cloud as an appliance Amini Cloud runs on commodity hardware. That means every shape of \"appliance\" is on the table: a single mini-PC on a shelf in an office, a stack of beige-box workstations in a homelab, a 1U pizza box bolted into a rack, or a hardened arm64 board zip-tied to the side of a network cabinet at a customer site. This document explains how to package, ship, install, and support Amini Cloud as a hardware appliance — a single SKU your operator or your customer plugs into power + ethernet and walks away from. ## Why ship Amini Cloud as an appliance There are three classes of buyer: 1. **The hands-off operator** — has hardware budget, no platform-engineer budget. They want a box that does one thing well: gives their team a private cloud, with sane defaults, that boots from cold start. 2. **The compliance-bound buyer** — air-gapped office, sovereign-data regulation, \"the data can never leave this room\". An appliance is the only correct form factor. 3. **The integrator's customer** — a managed-service provider builds a service on top of Amini Cloud and resells it as a turnkey rig to an end-user who knows nothing about amini. In all three cases the appliance form factor is what closes the deal. ## Architecture overview ```mermaid flowchart LR subgraph Appliance K[Kernel + Ubuntu 24.04] S[amini binary] A[amini-admission] H[amini-host-agent] P[Postgres / OpenBao] T[Tailscale] end Operator -->|SSH over tailnet| Appliance Tenant -->|Tailscale Serve| A Appliance -->|reports| Beszel(beszel hub) Appliance -->|alerts| Ntfy(ntfy topic) ``` Everything inside the dashed boundary ships on the disk image. The two external dependencies — Beszel hub for monitoring and ntfy topic for alerts — are operator-owned and join the same tailnet. ## The two delivery formats ### Format A — pre-built disk image (preferred) You hand the customer a `.qcow2`, `.iso`, or `.img` file built from the Packer template in `build/packer/`. They flash it to the box's internal SSD, boot, and `amini init --non-interactive` runs from a systemd oneshot. ```bash # Operator build: packer init build/packer/amini-appliance-ubuntu-2404-amd64.pkr.hcl packer build \\ -var class=server \\ -var arch=amd64 \\ -var version=v0.4.0 \\ build/packer/amini-appliance-ubuntu-2404-amd64.pkr.hcl # Output: dist/packer/qemu-server-amd64/amini-appliance.qcow2 ``` The qcow2 can be flashed to bare metal via `qemu-img convert -O raw amini-appliance.qcow2 /dev/sdX` or shipped to a public cloud as an AMI / GCE image (the same template has builders for both). ### Format B — installable tarball You hand the customer a `amini-bundle--linux-amd64.tar.gz`. They unpack it onto a stock Ubuntu install and run the universal installer: ```bash tar -xzf amini-bundle-v0.4.0-linux-amd64.tar.gz cd amini-bundle-v0.4.0-linux-amd64 bash scripts/install.sh ``` `scripts/install.sh` detects the host class (rpi / workstation / server / gpu-node / dgx-spark / edge) and dispatches to the right `onboard-.sh`. It's idempotent — re-running on an already- installed box does nothing destructive. `AMINI_INSTALL_DRY_RUN=1` shows every step that would run without changing state. Bundle build: ```bash make bundle-archive # → dist/amini-bundle-v0.4.0-linux-amd64.tar.gz ``` ## Hardware classes Amini Cloud auto-detects | Class | When it fires | Onboarder used | | ------------ | ------------------------------------------------------------- | ----------------------------- | | `rpi` | `/proc/device-tree/model` contains \"Raspberry Pi\" | `onboard-rpi.sh` | | `dgx-spark` | Tegra/GB10 in `/proc/device-tree/compatible` + hostname match | `onboard-dgx-spark.sh` | | `gpu-node` | `nvidia-smi` or `rocm-smi` succeeds | `onboard-gpu-node.sh` | | `gaming-pc` | hostname matches `gaming|rig|workshop` | `onboard-gaming-pc.sh` | | `mac` | `uname -s == Darwin` | `onboard-workstation-mac.sh` | | `workstation`| `$DISPLAY` set on Linux | `onboard-workstation-linux.sh`| | `server` | headless Linux, no special markers | `onboar" }, { "slug": "packaging/amini-on-customer-hardware", "title": "Amini Cloud on customer hardware", "description": "The third packaging mode — neither \"we ship the box\" nor \"we run the cloud\" — is **install Amini Cloud on a customer's existing iron**. This is the bread-and-butter mode for integrators, managed-service providers, and consultancies. The customer already has a Dell tower in a clos", "section": "packaging", "body": "# Amini Cloud on customer hardware The third packaging mode — neither \"we ship the box\" nor \"we run the cloud\" — is **install Amini Cloud on a customer's existing iron**. This is the bread-and-butter mode for integrators, managed-service providers, and consultancies. The customer already has a Dell tower in a closet, a rack of refurbished Intel NUCs, or a Hetzner dedicated server they want repurposed. You bring amini. ## Why this mode exists Customers don't always want a new SKU. Reasons we hear constantly: - They already have hardware sitting idle (capex sunk). - They have specific compliance constraints around the OS image (golden image, BIOS hardening, FIPS-validated crypto) that they don't want you to override. - They want a single throat to choke for ongoing ops — yours — without reselling them somebody else's appliance. - They're a homelabber who wants to point Amini Cloud at the gaming rig they're not using overnight. For all of those, the install-on-customer-hardware mode is the right answer. ## What's in scope ```mermaid flowchart LR CustomerHW[Customer hardware - any OS we support] BundleA[amini-bundle-vX.tar.gz] CustomerHW -->|bash scripts/install.sh| Amini[Working amini install] BundleA --> CustomerHW Operator[You - the integrator] -.->|SSH over tailnet| Amini ``` You ship a `amini-bundle--.tar.gz` plus an operations runbook. The customer (or you, on their behalf) extracts the tarball and runs `bash scripts/install.sh`. The installer detects the host class, dispatches to the right onboard script, and brings Amini Cloud up. You join the customer's tailnet from your side and SSH in to verify. Steps: ```bash # 1. operator: build the bundle from a pinned commit make bundle-archive # → dist/amini-bundle-v0.4.0-linux-amd64.tar.gz # 2. operator: scp it to the customer scp dist/amini-bundle-*.tar.gz customer@customer-box:/tmp/ # 3. on the customer box cd /tmp tar -xzf amini-bundle-v0.4.0-linux-amd64.tar.gz -C ~/amini cd ~/amini AMINI_INSTALL_DRY_RUN=1 bash scripts/install.sh # show plan bash scripts/install.sh # actually do it # 4. operator: verify from the tailnet ssh ubuntu@customer-box..ts.net amini doctor ``` The installer is idempotent. Running it on an already-installed box with the same version is a no-op (modulo \"let me confirm everything still looks fine\" via `amini doctor`). Running it with a newer version performs a rolling in-place upgrade with config preserved. ## Supported host shapes Amini Cloud's installer auto-classifies the host into one of eight classes (see [amini-as-an-appliance.md](./amini-as-an-appliance.md) for the detection rules). Each class has an onboard script: | Class | Path | What it sets up | | ------------- | ----------------------------------- | ---------------------------------------------- | | `server` | `scripts/onboard-server.sh` | podman + nomad + amini-host-agent | | `gpu-node` | `scripts/onboard-gpu-node.sh` | NVIDIA + CUDA + nvidia-container-toolkit | | `workstation` | `scripts/onboard-workstation-*.sh` | minimal agent, no compute scheduling | | `mac` | `scripts/onboard-workstation-mac.sh`| brew + amini-host-agent via launchd | | `dgx-spark` | `scripts/onboard-dgx-spark.sh` | DGX-specific drivers + GB10 quirks | | `rpi` | `scripts/onboard-rpi.sh` | armv6/7/aarch64 quirks, lower memory budgets | | `edge` | `scripts/onboard-edge.sh` | minimal footprint, no Postgres / OpenBao | | `gaming-pc` | `scripts/onboard-gaming-pc.sh` | dual-boot-aware, only runs when host is idle | Per-distro overlays are picked automatically: `onboard-alpine.sh`, `onboard-arch.sh`, `onboard-fedora.sh`, `onboard-nixos.sh`, `onboard-rhel.sh` carry distro-specific package-manager dispatch. Windows is supported via `scripts/onboard-server-windows.ps1` for the host-agent only — the control plane is Linux/Mac-only at MVP. ## Integration with the customer's existing infra Amini Cloud is designed to slot into a homelab or office network that already has its own: - **Tailscale or Headscale tailn" }, { "slug": "packaging/pricing-and-licensing", "title": "pricing and licensing", "description": "This is the short reference for Amini Infra Tech personnel and authorized licensees packaging Amini Cloud — as an appliance, on a customer's box, or as a managed cloud. Read it once before any deployment or brochure goes out. Amini Cloud is a proprietary Amini Infra Tech product;", "section": "packaging", "body": "# pricing and licensing This is the short reference for Amini Infra Tech personnel and authorized licensees packaging Amini Cloud — as an appliance, on a customer's box, or as a managed cloud. Read it once before any deployment or brochure goes out. Amini Cloud is a proprietary Amini Infra Tech product; nothing in this document grants any right to use Amini Cloud absent a signed commercial license. ## License Amini Cloud is **proprietary software**, copyright © 2026 Amini Infra Tech, all rights reserved. It is licensed — never sold — exclusively under a written commercial license agreement with Amini Infra Tech. The full proprietary notice ships at `LICENSE` at the root of this repository. The key terms: - **Use is licensed, not granted**: you may run Amini Cloud only under the scope, host count, user count, and term defined in your signed Amini Infra Tech commercial license. There is no implied right to use the software absent such an agreement. - **No redistribution or resale without authorization**: you may not repackage, resell, sublicense, host as a service, or otherwise make Amini Cloud available to third parties except under an explicit reseller or OEM agreement signed by Amini Infra Tech. - **No modification without written permission**: Amini Cloud source and binaries may not be modified, reverse engineered, or used to create derivative works without prior written permission from Amini Infra Tech. Any binary distributed under an authorized agreement must remain traceable to the source tree it was built from; the build embeds the git commit via `-ldflags` (the Makefile already does this). - **Confidential**: Amini Cloud contains trade secrets and confidential information of Amini Infra Tech. It must not be disclosed outside the licensed organization. - **Trademark-restrictive**: the word *Amini Cloud*, \"Amini Infra Tech\", and the Amini Cloud logo are marks of Amini Infra Tech. Any use requires written authorization under the commercial license. Amini Cloud is developed solely by Amini Infra Tech; it does not accept external contributions and there is no public contribution process. ## Pricing — Amini Infra Tech commercial licensing Amini Cloud is licensed commercially by Amini Infra Tech. Pricing is set per engagement under a signed agreement; contact Amini Infra Tech for a quote. Typical license dimensions: - **Platform license** — priced per operator org, scaling by host count and/or managed user count, for a defined term with renewal. - **Long-term support** (LTS) — back-ported security fixes for a named release for 18 months past its retirement date. Bundled with or added to the platform license. - **Vendor certification** — Amini Infra Tech tests + certifies your specific hardware SKU. One-time fee per SKU. - **Operator on-call assistance** — 24/7 escalation channel staffed by Amini Infra Tech engineers. Monthly retainer. - **Reseller / OEM** — separate written agreement required before Amini Cloud may be embedded in or resold as part of another offering. No use of Amini Cloud in production is permitted without a current Amini Infra Tech commercial license covering the deployment. ## Pricing — what *you* charge Amini Cloud the platform doesn't dictate your pricing. Five common shapes operators run: 1. **Flat per-month per-org** (most common for managed services) 2. **Per-seat per-month** (familiar to SaaS buyers) 3. **Usage-metered** (Stripe-style, drives off Amini Cloud's billing pipeline) 4. **One-off install + retainer** (consultancy / integrator) 5. **Hybrid** — flat base + usage overage (matches most cloud-vendor shapes) The `amini billing` CLI supports all five. The Pricing rate card (`internal/billing/pricing.go`) governs flow #3 directly; flows #1 + #2 ignore it; flow #5 sets a non-zero `BasePerTenant` and overlays usage rates on top. A sample list-price rate card for a US-region cloud platform: | Resource | Rate | Unit | | -------------------- | ------------------- | ------------------- | | CPU | $0.00004" }, { "slug": "README", "title": "Amini Cloud guides", "description": "User-facing documentation. The pages here are designed to be served by the Amini Cloud UI at `/docs/` AND read directly from a GitHub clone. Markdown is the source of truth; the UI renders these files at runtime (or at build-time if you use the static-export mode).", "section": "root", "body": "# Amini Cloud guides User-facing documentation. The pages here are designed to be served by the Amini Cloud UI at `/docs/` AND read directly from a GitHub clone. Markdown is the source of truth; the UI renders these files at runtime (or at build-time if you use the static-export mode). Three audiences: | Audience | What they need | Where to look | |---|---|---| | **Tenant user** — has an account, wants to run workloads | how to log in, create slices, SSH, share, manage volumes | `tenant/` | | **Operator** — runs Amini Cloud; manages tenants and hosts | how to install, upgrade, back up, monitor, run runbooks | `operator/` + `installation/` | | **Developer** — calling the API or extending Amini Cloud | the HTTP API reference, concept docs, the runtime/overlay interfaces | `api/` + `concepts/` | ## Layout ``` docs/guides/ ├── README.md (this file) ├── beta-release-notes.md what's in this beta and what isn't ├── changelog.md per-release notes ├── installation/ │ ├── single-host.md make up demo-tenant │ ├── three-host-cluster.md make ansible-up │ └── upgrading.md amini update; the safe path ├── tenant/ │ ├── getting-started.md sign-in, install client, first slice │ ├── creating-slices.md templates, custom OCI, GPUs │ ├── sharing-via-grants.md slice_grants, scopes, revocation │ ├── ssh-into-slices.md Tailscale-SSH; from laptop, phone, CI │ ├── using-templates.md catalog walkthrough │ └── volumes-and-storage.md object + block; per-tenant + per-user ├── operator/ │ ├── tenant-administration.md create, suspend, terminate │ ├── host-management.md add, drain, decommission │ ├── upgrade-procedure.md amini admin upgrade --check; rollout │ ├── backup-and-restore.md Postgres, SeaweedFS, secrets │ ├── monitoring.md Prometheus + Loki + Grafana wiring │ └── runbooks.md common incident shapes + responses ├── api/ │ ├── overview.md auth model, error shape, versions │ ├── authentication.md OIDC flow, bearer tokens, refresh │ └── v1-reference.md endpoint-by-endpoint ├── concepts/ │ ├── slices.md the product unit │ ├── tenants-and-users.md sandboxing model │ ├── overlays-and-providers.md tailscale, wireguard, netbird, nebula │ ├── runtimes.md podman, firecracker, qemu, kata, nspawn, kubernetes │ ├── storage-classes.md object (SeaweedFS), block (Longhorn/local-zfs) │ └── capability-discovery.md /v1/capabilities; what THIS amini supports └── troubleshooting/ ├── known-issues.md current bugs + workarounds └── faq.md real questions, real answers ``` ## How the UI consumes these The Amini Cloud UI at `/docs` lists this tree and renders each markdown file client-side (markdown-it + a minimal toc plugin). The UI source is at `web/`; the doc-fetcher reads from `/docs.json` (an index generated at build time from this directory). If you fork Amini Cloud and replace these docs, the UI picks up your version automatically — there's no embedded copy. ## Accuracy contract Every page in this directory is treated as load-bearing in beta: - A page that claims a feature works, and the feature does not work in this release, is a **release-blocking bug** at the tech-lead reviewer agent's discretion. - A page that documents a known limitation is the canonical source for that limitation; it must be linked from the corresponding code path via a `// docs:` comment. - Pages are versioned with the rest of the repo; release tags freeze the doc tree alongside the binaries. Out-of-date docs get the `:warning: Out of date` banner at the top (handled by a UI front-matter check) until a maintainer either updates them or marks them deprecated. ## Contributing a new guide 1. Pick the right subdirectory (audience). 2. Use the existing pages as templates for tone and shape. 3. Run `make docs-check` before opening a PR — it verifies internal links resolve and front-matter is valid. 4. The tech-lead reviewer agent will flag pages that contradict the code; resolve those before merge. " }, { "slug": "sovereign-inference-quickstart", "title": "Sovereign Inference — Quickstart (use your own models via Amini Cloud)", "description": "> Goal: stop paying for frontier AI by using open models you host yourself, over an > OpenAI-compatible endpoint, from Amini Cloud (`amini`), Claude Code / Codex / opencode, > and day-to-day chat.", "section": "root", "body": "# Sovereign Inference — Quickstart (use your own models via Amini Cloud) > Goal: stop paying for frontier AI by using open models you host yourself, over an > OpenAI-compatible endpoint, from Amini Cloud (`amini`), Claude Code / Codex / opencode, > and day-to-day chat. This is the **honest, works-today** path on the current fleet. It does **not** require the H200 — it uses the DGX Spark (`spark-5804`) where Amini Cloud is already running. ## 0. What's there right now (verified 2026-07-09) On `spark-5804` (NVIDIA GB10, Ubuntu 24.04, Docker 29.2.1, tailnet `100.113.57.52`): - **Amini Cloud is live**: `amini-admission` (healthy), `amini-host-agent` (healthy), `amini-web`, public console at **https://amini.ai/**. - **An ollama slice** (`amini-6b7d8b23`) published on `:32770` → speaks OpenAI at `/v1`. Reachable across the tailnet at `http://100.113.57.52:32770/v1`. - **Host ollama** (`:11434`, localhost only) already has real models pulled: `qwen2.5-coder:14b`, `qwen2.5:7b`, `qwen2.5vl:32b` (vision), `qwen2.5vl:7b`, `qwen2.5:7b-instruct`, `nomic-embed-text`. Verified working: `qwen2.5-coder:14b` answers over `/v1/chat/completions` (host), and the amini slice serves chat via its own endpoint. **The one gap:** the tailnet-reachable slice only had a tiny `0.5b` model; the good models were host-only. The fix is to pull the good models **into the slice** (below). ## 1. Check the fleet (run anytime) ```bash # On the Spark (or any host): docker ps --format '{{.Names}}\\t{{.Image}}\\t{{.Ports}}' | grep -iE 'amini|ollama' curl -s http://localhost:11434/api/tags | python3 -m json.tool | grep '\"name\"' # host models curl -s http://localhost:32770/v1/models # slice models (OpenAI) nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv ``` ## 2. Put the good models into the amini slice (so every device can use them) The slice container is `amini-6b7d8b23` (find yours with the `docker ps` above). ```bash # Pull the models you want INTO the slice (reachable at :32770 over the tailnet): docker exec amini-6b7d8b23 ollama pull qwen2.5-coder:14b # coding / Claude Code docker exec amini-6b7d8b23 ollama pull qwen2.5vl:32b # vision + general docker exec amini-6b7d8b23 ollama pull nomic-embed-text # embeddings # confirm: curl -s http://localhost:32770/v1/models | python3 -m json.tool ``` > Tip: the host ollama already has these downloaded. Pulling into the slice re-downloads > them into the slice's own store (ollama doesn't share stores across instances). ## 3. Use it from Claude Code / Codex / opencode (over the tailnet) Point any OpenAI-compatible client at the slice endpoint. From a MacBook on the tailnet: ```bash # OpenAI-native tools (Codex / opencode / Cursor / Continue) point straight at the slice: export OPENAI_BASE_URL=\"http://100.113.57.52:32770/v1\" export OPENAI_API_KEY=\"ollama\" # slice ollama accepts any key today # then select model: qwen2.5-coder:14b # # Claude Code speaks the ANTHROPIC API, not OpenAI — it CANNOT point at the ollama /v1 # directly. Run the keyed gateway (deploy/inference-gateway/, LiteLLM exposes an # Anthropic-compatible route) and set ANTHROPIC_BASE_URL to it instead. # Quick smoke test from the Mac: curl -s http://100.113.57.52:32770/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{\"model\":\"qwen2.5-coder:14b\",\"messages\":[{\"role\":\"user\",\"content\":\"write a bash one-liner to count files\"}]}' \\ | python3 -c \"import sys,json;print(json.load(sys.stdin)['choices'][0]['message']['content'])\" ``` - **Codex / opencode**: same — set the OpenAI base URL to `http://100.113.57.52:32770/v1` and the model to `qwen2.5-coder:14b`. - **Reduce cost**: with this set, coding/chat runs on your hardware, not a paid API. ## 4. Use it from Amini Cloud (the console) 1. Open **https://amini.ai/** in your browser and sign in. 2. Go to **Slices** → the ollama slice → the **API / chat** tab. The console chats with the slice's own endpoint; the models you pulled in step 2 appear there. 3. To launch a *new* model slice from the " }, { "slug": "tenant/creating-slices", "title": "Creating slices", "description": "How to compose a slice spec, what each field does, and how admission validates it.", "section": "tenant", "body": "# Creating slices A slice is the unit you provision on amini. It is one isolated execution domain with a stable identity, a lease, an owner, a runtime, an exposure list, and an audit trail. The canonical concept is in [`concepts/slices.md`](../concepts/slices.md); the JSON contract is in [`api/slice.schema.json`](../../../api/slice.schema.json). This page is the workflow. ## The minimum viable spec The schema requires four fields: `name`, `runtime`, `image`, and `resources`. The smallest legal POST body: ```json { \"name\": \"hello\", \"runtime\": \"podman\", \"image\": \"docker.io/library/nginx:1.27-alpine\", \"resources\": { \"cpu_milli\": 500, \"mem_bytes\": 268435456, \"disk_bytes\": 1073741824 } } ``` Field-by-field: - **`name`** — DNS-safe identifier unique within your tenant. Pattern `^[a-z][a-z0-9-]{0,38}[a-z0-9]$`. The slice will be reachable as `..amini.ts.net`. - **`runtime`** — one of `podman` | `firecracker` | `qemu` | `kata` | `nspawn` | `kubernetes`. Untrusted tenants must pick a VM-isolation runtime; trusted tenants default to `podman`. See [`concepts/runtimes.md`](../concepts/runtimes.md). - **`image`** — OCI image reference (for `podman`, `nspawn`, `kata`), rootfs reference (for `firecracker`), cloud image (for `qemu`), or OCI image (for `kubernetes`). - **`resources.cpu_milli`** — CPU in millicores. 1000 = 1 vCPU. Minimum 50, maximum 64000. - **`resources.mem_bytes`** — memory in bytes. Minimum 32 MiB. - **`resources.disk_bytes`** — disk in bytes. Minimum 128 MiB. ## Optional fields The full schema is in `api/slice.schema.json`. The fields you'll commonly add: ### Exposure By default a slice can dial out but nothing can dial in. To accept connections, list the ports: ```json \"exposure\": [ {\"port\": 8888, \"proto\": \"tcp\", \"scope\": \"tenant\"}, {\"port\": 9000, \"proto\": \"tcp\", \"scope\": \"tenant\"} ] ``` The `scope` field controls reachability: - **`tenant`** — reachable from any device in this tenant's tailnet. This is the default and what you want 99% of the time. - **`shared`** — reachable via the Year-1 cross-tenant service-share bridge. **Not implemented in the beta.** Listing this scope today causes admission to write the row but no traffic flows. - **`public`** — reachable from outside the overlay via platform ingress. **Year-1 feature, not implemented in beta.** ### Environment variables ```json \"env\": { \"JUPYTER_TOKEN\": \"my-token-here\", \"POSTGRES_PASSWORD\": \"another-secret\" } ``` Values must be strings. Slice templates pre-fill some env vars; you can override them. ### Lease ```json \"lease_seconds\": 604800 ``` Amini Cloud auto-terminates slices after their lease expires; this defends the platform against orphaned workloads. Minimum 60 seconds, maximum one year. Default 86400 (24 hours) — applied by admission if you omit the field, per `internal/admission/server.go`. Renew with `POST /v1/slices/{id}/renew`. ### Node constraints Pin a slice to a specific architecture or region: ```json \"node_constraints\": { \"region\": \"nairobi\", \"arch\": \"arm64\", \"labels\": { \"gpu\": \"nvidia-h100\" } } ``` Admission refuses to schedule if no host matches. ### Cluster For runtime `kubernetes`, specify which registered cluster: ```json \"runtime\": \"kubernetes\", \"cluster_id\": \"\" ``` For Nomad runtimes, `cluster_id` is optional; NULL means \"the platform default Nomad cluster.\" ### Runtime-specific config The `runtime_config` jsonb carries per-runtime knobs: ```json \"runtime\": \"qemu\", \"runtime_config\": { \"kernel\": \"vmlinuz\", \"initrd\": \"initrd\", \"disk_format\": \"qcow2\", \"cloudinit\": true, \"gpu_passthrough\": \"vfio\" } ``` See [`concepts/runtimes.md`](../concepts/runtimes.md) for the shape per runtime. ## Using a template instead Templates pre-fill defaults. Instead of writing every field, reference a template id: ```json { \"name\": \"alice-notebook\", \"template_id\": \"jupyter-gpu\", \"template_version\": \"1.0.0\", \"gpu_count\": 1 } ``` > **Beta caveat:** the template-expansion path on `amini-admission` is > the design contract, but " }, { "slug": "tenant/getting-started", "title": "Getting started as a tenant", "description": "Sign in (native email+password or wallet), install a Tailscale client, and create your first slice.", "section": "tenant", "body": "# Getting started as a tenant This page walks a new tenant from \"an operator just gave me an account\" to \"I have a JupyterLab slice running and I can open a shell on it.\" It assumes zero ops knowledge. If you are the operator and have not yet stood up Amini Cloud, start with [`installation/single-host.md`](../installation/single-host.md). ## What you have Your operator gave you: - The Amini Cloud control URL (e.g. `https://api.amini.example` or `http://localhost:18080` for a local-dev install via Compose; older builds bound to `:8080`, so if your operator's docs cite that port, set `AMINI_ADMISSION_URL=http://localhost:18080` to match the current Compose default — see `deploy/compose/.env.example`). - The dashboard / web URL (e.g. `https://app.amini.example` or `http://localhost:5173` for a local-dev install). - Either an email + password for your account, or a wallet you'll sign in with. (If your operator runs the optional/legacy OIDC path instead, they'll give you an IdP login URL — see [OIDC sign-in setup](../oidc-setup.md).) - A tenant slug — the short name of your organisation in this Amini Cloud install (e.g. `acme`, `demo`, `ulap`). If you do not have all four, stop and ask your operator. ## Step 1 — Sign in Amini Cloud's shipped, default sign-in is **native email+password** plus **wallet (SIWE) sign-in** (per [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md)). Open the dashboard URL and go to **`/login`**: - **Email + password:** enter them and sign in. New here? Use **`/signup`** first (or your operator may have created the account). - **Wallet:** click **Sign in with a wallet** (`/login/wallet`) and sign the nonce with your EVM or Cosmos wallet — no email, no password. Either way you land on `/dashboard` with an httpOnly `slate_session` cookie set; the dashboard reuses it on every API call. If your operator has `AUTO_PROVISION_USERS=true` set (the default for the demo Compose stack — see `internal/config/config.go`), your first authenticated call also creates a row in `amini.tenant_users` for you. ## Step 2 — Confirm you can reach amini-admission From the dashboard this is automatic — the `/dashboard` page is exactly a `GET /v1/tenants/me` call, and the header shows your tenant + subject. To check from a shell, sign in natively to a cookie jar and reuse it: ```bash # Sign in → slate_session cookie curl -fsS -c amini.cookies -X POST https://api.amini.example/v1/auth/login \\ -H 'content-type: application/json' \\ -d '{\"email\":\"you@example.com\",\"password\":\"\"}' # Reuse the cookie curl -fsS -b amini.cookies https://api.amini.example/v1/tenants/me | jq ``` A `200 OK` response with a tenant object means amini-admission accepted your session and you are now provisioned as a user. If you get `401 Unauthorized`, your credentials or session are wrong or expired. If you get `403 Forbidden`, your operator has set `AUTO_PROVISION_USERS=false` and a row has not been created yet. ## Step 3 — Install a Tailscale client Amini Cloud's overlay layer is a per-tenant Tailscale-compatible tailnet served by a Headscale 0.28.0 process. To reach slices you need any official Tailscale client: - **macOS / iOS / Windows / Android:** install the official Tailscale app from the App Store / Google Play / `tailscale.com/download`. - **Linux:** `curl -fsSL https://tailscale.com/install.sh | sh`. You can use any device that has a Tailscale client — and you can install it on multiple devices at once (laptop + phone + work box). All of them will share your tenant identity. ## Step 4 — Join your tenant's tailnet Headscale uses pre-auth keys to admit new devices. Your operator either gave you a pre-auth key directly, or pointed you at the Headscale URL where you can interactively register a device. The Headscale URL for your tenant typically looks like `https://.amini.example` (production) or `http://localhost:50001` (single-host Compose default for the `demo` tenant). Run: ```bash # Linux sudo tailscale up --login-server=h" }, { "slug": "tenant/installing-apps", "title": "Installing apps", "description": "What the app catalog is, how to install an app into your tenant, and where to watch the instance and its component slices come up.", "section": "tenant", "body": "# Installing apps A **slice** is a single container. An **app** is a packaged bundle of one or more slices that belong together — a manifest pins each component's image, resources, exposure, and env so you get a working multi-service workload without composing every slice by hand. Installing an app creates one **app instance** plus one **component slice** per component declared in the app's manifest. The slices are ordinary amini slices: they show up in your slice list, get their own ingress URLs, and obey your tenant's quota. The instance is the grouping that ties them together so you can see and tear them down as a unit. ## The app catalog The catalog is the set of apps available to install. Browse it at **Catalog** (`/admin/apps`) in the dashboard. Each card is one app; the tabs let you filter: - **Curated** — apps maintained and verified by the amini distribution. - **Mine** — apps your tenant published. - **All** — everything you can see. Click a card to open the app detail page (`/admin/apps/{slug}`). It renders the parsed manifest: the components, each component's image and resource floor, the exposures it will publish, and the version history. > The same catalog is reachable over the API: `GET /v1/admin/apps` > lists every app; `GET /v1/admin/apps/{slug}` returns one app plus > its parsed manifest; `GET /v1/apps/schema` is the manifest schema > apps are validated against. ## Installing an app From the app detail page, click **Install**. The install dialog asks for two things: 1. **Name** — the instance name. Must be DNS-safe: lowercase letters, digits, and dashes, 3–64 characters, no leading or trailing dash. Leave it blank and amini generates one for you (`-`). The component slices are named `-` so re-installing the same app into the same tenant never collides on slice names. 2. **Tenant** — which tenant the instance lands in. The instance, its slices, and its quota usage all belong to this tenant. Confirm, and Amini Cloud runs the scheduler, creates the `app_instance` row, creates one slice per component, and returns the instance plus the new slice IDs. The dialog's **Advanced** panel is optional. It lets an operator override the scheduler defaults: pin a specific `host_id`, override each component's CPU / memory floor, or replace the env block. Omit all of it to keep the manifest's defaults. The same operation over the API is: ```bash curl -X POST https://api.amini.example/v1/admin/apps//install \\ -H 'Authorization: Bearer ' \\ -d '{\"name\": \"my-instance\", \"tenant_slug\": \"demo\"}' ``` ### If install is refused - **`409` insufficient compute.** No host has enough free capacity for the app's component floors. The response carries the deficit and how many hosts were considered. Free up capacity, or use the Advanced panel to under-commit resources for a test. - **`409` name already in use.** The instance name collides with an existing slice in the tenant. The response includes a suggested name; the dialog offers a one-click retry with it. ## Watching it come up After install you land on the instance page (`/admin/apps/instances/{id}`). It lists every component slice and its live status (`pending` → `running`, or `failed`). Each slice links to its own detail page, where the **Access** card shows the ingress URL once the slice is `running`. The instance list (`/admin/apps/instances`, `GET /v1/admin/apps/instances`) shows every running instance in your tenant if you want the fleet-wide view. ## Uninstalling From the instance page, choose **Uninstall** (or `DELETE /v1/admin/apps/instances/{id}`). This soft-deletes the instance and stops every linked component slice. The instance disappears from the running list; its audit history is preserved. ## Related tasks - [Creating slices](creating-slices.md) — the single-slice path that each app component is built on. - [Using slice templates](using-templates.md) — the catalog of slice shapes apps are composed from. - [Using the mod" }, { "slug": "tenant/known-gaps", "title": "Known gaps — what's planned vs shipped today", "description": "Honest list of capabilities the docs (whitepaper, tutorials, README) describe that are not fully wired yet. Each gap names the workaround that DOES work today.", "section": "tenant", "body": "# Known gaps > [!warning] **This is the honest list.** > Amini Cloud's [whitepaper](../../whitepaper.md) is the **design contract**, not the as-built > description. The user-facing tutorials and README reach forward > toward the contract in places where the code hasn't caught up yet. > This page is the diff. If anything else in `docs/` disagrees with > what's on this page, **this page wins**. The > [`/admin/capabilities`](http://localhost:5173/admin/capabilities) > dashboard reads `amini.slate_features` live; it is the ultimate > source of truth. > > Authoritative cross-refs: the three audits in > [`docs/audits/2026-05-14-*`](../../audits/) — chain-layer-audit, > docs-and-journey-audit, and slice-ux-audit. Anchor migration: > [`db/migrations/0029_capability_honesty_pass.sql`](../../../db/migrations/0029_capability_honesty_pass.sql). --- ## TL;DR matrix | Capability | What works today | What's planned | Workaround | |---|---|---|---| | Runtime | `podman` end-to-end | Firecracker / Kata / QEMU / Kubernetes / systemd-nspawn | Pick `runtime: \"podman\"` on every slice | | Storage | `local-zfs` (rough) | SeaweedFS, Longhorn, archive | Stick to ephemeral slices for tutorials; manual host-side bind-mount for persistence | | GPU | NVIDIA discrete | AMD ROCm, NVIDIA MIG, Grace UMA partitions | Use NVIDIA-only hosts; one-tenant-per-GPU | | Slice SSH | Dev WebSocket (`amini slices ssh --dev`) | Tailscale-SSH from your tailnet device | Use `--dev` flag, or use the Shell tab in the web UI | | HA control plane | Single Postgres in Compose | Patroni-managed Postgres cluster | Back up `state/postgres/` regularly | | AminiChain integration | SIWE signature verification (real); audit anchor writes (stub tx hash); NFT-AIM `ownerOf` client code-complete | Operator NFT-AIM contract-deploy; real chain talk, oracle attestation, jobs settlement | Default build is fine; don't gate prod features on `runtime_aminichain` tag | | Public ingress + auto-TLS | **(wave-9 beta)** Reconciler-driven Caddy, sharable URLs, BYO custom domains, HTTP-01 ACME | DNS-01 wildcard, per-tenant ACME ACL, Prometheus metrics | See `docs/guides/operator/ingress-and-domains.md` | | CLI commands | ~half of `amini slices/clusters/...` | The other half (31 `notYet` stubs) | Use `curl` against admission's REST API; web UI for everything else | | Per-tenant overlays | Single shared Headscale (one `demo` tenant) | Per-tenant Headscale spawner | Stick to a single tenant per Amini Cloud install for now | --- ## The `AminiDev` dev-bearer is off by default; use an API key **The trap.** `Authorization: AminiDev ` is a dev-only bypass, and admission honours it only when **all three** of these are set on the admission process (`devBearerEnabled()`, `cmd/amini-admission/main.go`): ```bash AMINI_ENABLE_DEV_BEARER=true WALLET_DEV_AUTO_PROVISION=true WALLET_DEV_AUTO_TENANT=demo ``` `deploy/compose/.env.example` ships the last two but deliberately leaves `AMINI_ENABLE_DEV_BEARER` unset, so the bearer is **off even on a fresh local `make up`**. It was found reachable on a shared dev stack and disabled there (security finding F-001). With it off you get `401 malformed Authorization header`, which reads like a bad credential but means the feature is disabled. **What to use instead.** Mint an API key from a signed-in session and send `Authorization: Bearer `. It works on every deployment, and it can be named, listed and revoked one key at a time. The three calls are in [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api). The `$TOKEN` in the curl examples further down this page is that key: ```bash export TOKEN=\"Bearer $AMINI_API_KEY\" ``` **If you do turn the bearer on**, do it only on a box nobody else can reach: it accepts any known `tenant_user` subject and grants that user's full role, so a guessable seeded subject like `admin` hands out owner-level access. Add the flag to your own `deploy/compose/.env` and recreate admission. **What's" }, { "slug": "tenant/sharing-via-grants", "title": "How to share a slice via grants", "description": "Use slice_grants to give another tenant user read, use, or admin scope on one of your slices.", "section": "tenant", "body": "# How to share a slice via grants Amini Cloud's per-user model inside a tenant is **default-deny**. Owning a slice is the only way to access it by default. Sharing is one row in `amini.slice_grants` per (slice, target user, scope) tuple. ## When to use this - You want to give a teammate read-only access to look at a slice (a shared Jupyter session, a Postgres for debugging). - You want to hand off operational control of a slice to a colleague (suspend, resume, renew) without giving them ownership. - You want CI in your tenant to reach a slice with a service-account subject. If you instead want the slice to be visible to **every** tenant user, make the requester a tenant `admin` (`tenant_users.role='admin'`) and they see every slice in the tenant by default. Don't use grants for this — it doesn't scale. ## What you need - Slice ownership (your OIDC subject is the slice's `owner_subject`), OR the tenant `admin` role. - The target user's OIDC `subject` — exactly the string in `tenant_users.subject` for them. Usually their username or email. - Decision on scope: | Scope | Grantee can … | |---|---| | `read` | GET the slice; see it in `GET /v1/slices` | | `use` | All of `read` + SSH + connect to exposed ports | | `admin` | All of `use` + suspend/resume/renew. Cannot delete (delete = owner + tenant admin only) | ## The procedure ### 1. Confirm you can manage the slice You must be the owner. Check: ```bash curl -fsS -H \"Authorization: Bearer $TOKEN\" \\ http://localhost:18080/v1/slices/$SLICE_ID | jq .owner_subject ``` If the response is `null` (404) or the subject doesn't match yours, you can't grant on this slice. ### 2. Add the grant row > **Beta caveat:** the CLI `amini grants add` and the matching HTTP > endpoint are scaffolded but not yet wired (the CLI prints \"not yet > implemented\"). The supported MVP workflow is a direct SQL insert. ```bash psql \"$POSTGRES_DSN\" < [!warning] **Tailscale-SSH into slices is not shipped yet.** > The `ssh you@..amini.ts.net` path described later on > this page is the *target* shape, not the current reality. The > `amini.slices.ssh_enabled` column is never read by the runner, and the > catalog images do not ship a `tailscaled` binary — so there is nothing > listening on port 22 over the tenant tailnet. Use the **web Shell tab** > (the working path, below). See > [known gaps § Tailscale-SSH](known-gaps.md#tailscale-ssh-to-slices). The working way to run commands inside a slice today is the **Shell** tab on the slice detail page in the dashboard, backed by `POST /v1/slices/{id}/exec`. For interactive sessions there is also the dev WebSocket shell (`amini slices ssh --dev `). ## When to use this - You want a shell on a slice you own. - You want a shell on a slice someone has granted you `use` or `admin` scope on. ## What you need - The dashboard URL and a signed-in session (native email+password or wallet — see [Getting started](getting-started.md#step-1--sign-in)). - You must be the slice owner, a tenant admin, or hold a `slice_grants` row with scope `use` or `admin`. See [Sharing via grants](sharing-via-grants.md). - The slice must be in `status='running'`. ## The working path — web Shell tab 1. Open the slice detail page (`/slices/`) in the dashboard. 2. Click the **Shell** tab. 3. Type a command and hit **Run**. Output streams back inline. > [!warning] **The Shell tab is argv-only — no `bash -c`.** > The Shell tab and the `/v1/slices/{id}/exec` API behind it do **not** > invoke a shell. Pass argv directly: `cat /etc/os-release` works, > `ls /workspace` works, but `bash -c \"echo $HOME\"` does not (no shell > to expand `$HOME`, no pipes, one command per call). For the long form > and the design reasoning see > [Tutorial 12 — Run a command in a slice](../tutorials/12-run-a-command-in-a-slice.md) > and [ADR-0031](../../decisions/0031-slice-exec-surface.md). The same call from a shell, reusing the `amini.cookies` jar from sign-in: ```bash curl -fsS -b amini.cookies -X POST \\ https://api.amini.example/v1/slices/$SLICE_ID/exec \\ -H 'content-type: application/json' \\ -d '{\"argv\":[\"cat\",\"/etc/os-release\"],\"timeout_seconds\":15}' | jq -r .stdout ``` Each exec writes a `slice.exec` (or `slice.shell.open`) row to `amini.audit_events`, scoped to your subject. ## The interactive path — dev WebSocket shell For a real interactive shell (pipes, `$HOME` expansion, a prompt), use the `--dev` WebSocket shell: ```bash amini slices ssh --dev ``` This is a true shell over a WebSocket to admission — distinct from the argv-only exec surface above, and distinct from the (planned) Tailscale-SSH path. Covered in [Tutorial 12](../tutorials/12-run-a-command-in-a-slice.md). ## Planned — Tailscale-SSH into slices The target shape, once first-party `tailscaled`-baked slice images ship (see [known gaps](known-gaps.md#tailscale-ssh-to-slices)): ```bash ssh you@..amini.ts.net ``` No `ssh-keygen`, no `authorized_keys` — Tailscale-SSH would authenticate against your tailnet identity, with authorization coming from your slice grant. The plan is a `jupyter`-class image with `tailscaled` baked in plus a one-shot enrolment so admission can flip `ssh_enabled`. Until that lands, **use the Shell tab or the `--dev` WebSocket shell above.** ## If it doesn't work - **Shell tab shows \"not implemented\" / empty state.** Your admission build doesn't have the exec/WebSocket surface wired in. Fall back to the `curl … /exec` call above, or to `amini slices ssh --dev`. - **`bash -c \"...\"` fails on the Shell tab.** Expected — it's argv-only. Pass the command and its args as separate argv entries. - **`403` on exec.** You don't own the slice and don't hold a `use` or `admin` grant. Confirm `amini.slice_grants` has a row with your subject, or ask the owner to grant you. - **Slice not in `running`.** Exec only works against a " }, { "slug": "tenant/team-management", "title": "Team management", "description": "How to invite members, change roles, set quotas, and audit your tenant's activity. Vercel-team-style controls.", "section": "tenant", "body": "# Team management > [!tip] **You'll have this open most often.** As a tenant admin, this is > the page you live in: who's on your team, what they're allowed to do, > what they've been doing, how much capacity you've consumed. `/admin/tenants/` is your tenant's control center. Five tabs: **Overview**, **Members**, **Resources**, **Activity**, **Settings**. ## Inviting a member 1. Open `/admin/tenants/` (or click your tenant from `/admin/tenants`). 2. Click the **Members** tab. 3. Click **\"Invite member\"**. 4. Fill the modal: - **Email** — required. The invitee's email address. - **Role** — `member` (default; can create slices), `admin` (also manages members + settings), `owner` (also can terminate the tenant). At least one `owner` per tenant; the demotion guard refuses the last-owner-remove. 5. Click **\"Send invite\"**. Amini Cloud creates the invite, logs the URL to the admission process's stdout, writes a `tenant.invite.sent` audit row, and shows the pending invite in the table with a 7-day TTL countdown. > [!info] **Email is stubbed today.** Wave-10 wires SMTP. For now, get the > invite URL from the admin process logs or from the audit feed (each row > includes the URL in `payload.invite_url`). Hand it to the invitee > out-of-band. ## Accepting an invite The invitee opens the URL. The page at `/invites/` shows: - The tenant they're being invited to (display name + slug) - The role they'll have on accept - Who invited them - An **Accept** button If the invitee is not signed in, the page redirects to `/login` first, then back. After accept, they get a `tenant_users` row + immediate access to the tenant's slices/templates/etc. — no admin re-approval needed. If the token is expired, revoked, or already accepted, the page shows the specific reason — never just a generic 404. ## Changing a member's role In the Members tab, each row has a role dropdown. Change it inline. The backend writes a `tenant.role_changed` audit row capturing both the before-role and after-role + who made the change. Demoting an owner triggers a confirm dialog (and refuses if it would leave zero owners). ## Removing a member Click **Remove** on a member row. Confirm. The `tenant_users` row is deleted; the member's slices stay (a tenant admin can claim or transfer them via the Resources tab in wave-10). An audit row of action `tenant.user.removed` records the action. A member can self-remove via their own `/account` page (wave-9 K5 follow-up on the account flow; today, removal is admin-driven). ## Resources tab — quota dashboard For each quota dimension (CPU milli, memory bytes, slice count): - Current usage as a sum of *running* slices' resources - Limit (from `amini.tenants.cpu_milli_quota`, etc.) - A visual progress bar: green ≤80%, amber 80-95%, red >95% - An **\"Edit quota\"** button (owner-only) opens a form Editing a quota writes a `tenant.quota_changed` audit row + immediately takes effect on the next slice-create call (admission re-reads quotas on every accept). > [!warning] **Quotas don't terminate already-running slices.** If you lower > the CPU quota below current usage, existing slices keep running; only > *new* slice creates are blocked. For hard-stop behavior, suspend or > terminate the offending slices via `/slices` first. ## Activity tab — tenant-scoped audit feed Last 50 audit rows for this tenant, latest first. Filters by action (`slice.create`, `tenant.invite.sent`, etc.) + by subject (member). Auto-refresh every 30s. For richer queries, hit the API directly: ```bash curl -fsS \\ -H \"Authorization: Bearer $TOKEN\" \\ \"http://localhost:18080/v1/admin/audit?tenant_id=&since=2026-05-01T00:00:00Z&action=slice.create\" ``` ## Settings tab Owner-only controls: - **Display name** — friendly name shown in the UI; doesn't affect URLs - **Slug** — DNS-safe identifier; **changing this breaks invite URLs**, Headscale tailnet routing, and existing slice access URLs. Don't change except as part of a planned migration. - **Untrusted fl" }, { "slug": "tenant/templates/code-server", "title": "code-server — browser-based VS Code", "description": "How to launch the code-server template; set a password; open the editor.", "section": "tenant", "body": "# `code-server` — browser-based VS Code ## What it is A full VS Code IDE that runs in your browser. Backed by [`codercom/code-server:latest`](https://hub.docker.com/r/codercom/code-server/tags). Listens on tcp/8080. Verified against the upstream registry on 2026-05-14 (`docker manifest inspect docker.io/codercom/code-server:latest`). ## How to launch From the SPA: `/slices/new` → **VS Code (code-server)**. Defaults: 1.5 vCPU / 4 GiB / 10 GiB / exposure 8080. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"ide\", \"runtime\": \"podman\", \"image\": \"codercom/code-server:latest\", \"resources\": {\"cpu_milli\":1500,\"mem_bytes\":4294967296,\"disk_bytes\":10737418240}, \"exposure\": [{\"port\":8080,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": {\"PASSWORD\":\"changeme-or-better\"}, \"lease_seconds\": 86400 }' | jq ``` ## How to access 1. Wait for `status: running`. 2. Click **Open VS Code** in the Connect panel at the top of the slice detail page. 3. Sign in with the `PASSWORD` env var you set on launch (surfaced as a click-to-reveal credential in the Connect panel). If you didn't set one, the random default is logged on container startup — the Connect panel's credentials_hint points you at `amini slices logs | grep -i password`. ## Connect panel — what you'll see The Connect panel for `code-server` shows: - **Open VS Code** button — opens the IDE in a new tab. - **PASSWORD** credential, click-to-reveal if you set it on launch. When empty (random password), a credentials_hint tells you where to grep. ## Run a smoke test In the browser editor: 1. New file → `hello.py`. 2. Paste: ```python import sys, platform print(\"python:\", sys.version) print(\"platform:\", platform.platform()) print(\"hello from amini\") ``` 3. Open a terminal (Terminal → New Terminal) and run `python hello.py`. You should see the python version + a \"hello from Amini Cloud\" line. ## What to do next - Install extensions via the marketplace tab (the image bundles the open-vsx mirror by default). - Mount a `block` volume at `/home/coder/project` so your workspace survives a slice suspend → resume. - Connect to a side-by-side [`postgres`](postgres.md) slice from a terminal — the editor and DB live on the same loopback. ## Related - [Using slice templates](../using-templates.md) - [code-server docs](https://coder.com/docs/code-server/latest) " }, { "slug": "tenant/templates/jupyter", "title": "jupyter — CPU notebooks (SciPy stack)", "description": "How to launch and use the CPU JupyterLab template; what env vars matter; how to find the token and access the notebook.", "section": "tenant", "body": "# `jupyter` — CPU notebooks (SciPy stack) ## What it is A CPU-only JupyterLab slice. Backed by [`quay.io/jupyter/scipy-notebook:lab-4.5.7`](https://quay.io/repository/jupyter/scipy-notebook?tab=tags) — NumPy, SciPy, scikit-learn, pandas, matplotlib pre-installed. Listens on tcp/8888 inside the container. Verified against the upstream registry on 2026-05-14 (`docker manifest inspect quay.io/jupyter/scipy-notebook:lab-4.5.7`). ## How to launch From the SPA: `/slices/new` → pick **JupyterLab** → set a name → continue. The defaults (2 vCPU / 4 GiB / 10 GiB / exposure 8888) come from the template. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"my-jupyter\", \"runtime\": \"podman\", \"image\": \"quay.io/jupyter/scipy-notebook:lab-4.5.7\", \"resources\": {\"cpu_milli\":2000,\"mem_bytes\":4294967296,\"disk_bytes\":10737418240}, \"exposure\": [{\"port\":8888,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": {\"JUPYTER_TOKEN\":\"pickme\"}, \"lease_seconds\": 86400 }' | jq ``` ## How to access 1. Wait for `status: running` on the slice (a few seconds for the first launch on a warm cache, several minutes on a cold cache — the SciPy image is ~3 GiB). 2. Open the slice detail page. The **Connect panel** at the top shows a green **Open Jupyter** button that already has your token substituted into the URL — click it, you're in the notebook. If the engine is on a remote host and the dev runner published on `127.0.0.1`, ssh-tunnel the port or use the Tailscale-overlay route (Year-1). ## Connect panel — what you'll see Once the slice is running, the slice detail page shows a Connect panel (wave-8 J3) at the top of the Overview tab: - **Open Jupyter** button — opens `http://127.0.0.1:/?token=` in a new tab. No copy-paste, no logs grep. - **JUPYTER_TOKEN** is surfaced as a click-to-reveal credential. Copy it directly to clipboard with the icon next to it. - **Smoke test**: `print('hello from amini')` — copy that into the first cell to confirm the kernel is responsive. - **What you can do next** links to the GPU variant and volume mounts so the obvious follow-ups are one tap away. If you didn't set `JUPYTER_TOKEN` at launch, the panel still shows the open URL but with an empty token query — the real token is in the container logs (Logs tab) until you re-launch with the token set. ## Run a smoke test Open a new notebook in the JupyterLab UI and run: ```python import numpy as np, pandas as pd print(\"numpy:\", np.__version__, \"| pandas:\", pd.__version__) print(pd.DataFrame({\"x\":[1,2,3]}).describe()) ``` You should see the version banner + a 3-row describe output. ## What to do next - GPU work? Switch to [`jupyter-gpu`](jupyter-gpu.md). - Need a Postgres backend? Spin up a [`postgres`](postgres.md) slice side-by-side and connect from the notebook via the slice's `..amini.ts.net` overlay hostname (Year-1) or the slice access URL today. - Mount a Lance dataset for multimodal training? Use [`multimodal-lance-server`](../using-templates.md#data--database--storage). ## Related - [Using slice templates](../using-templates.md) - [Tutorial 01 — Your first slice](../../tutorials/01-your-first-slice.md) - [Tutorial 03 — GPU ML training on Jupyter](../../tutorials/03-gpu-ml-training-on-jupyter.md) " }, { "slug": "tenant/templates/jupyter-gpu", "title": "jupyter-gpu — GPU notebooks (PyTorch + CUDA 12)", "description": "How to launch and use the GPU JupyterLab template; smoke-test cuda.is_available(); how to mount a dataset.", "section": "tenant", "body": "# `jupyter-gpu` — GPU notebooks (PyTorch + CUDA 12) ## What it is GPU-attached JupyterLab. Backed by [`quay.io/jupyter/pytorch-notebook:cuda12-latest`](https://quay.io/repository/jupyter/pytorch-notebook?tab=tags) — the upstream Jupyter pytorch-notebook image's CUDA 12 line. The slice requests `requires_gpu=true`, so the scheduler will only place it on a host with a free GPU; you must also include `gpu_count: 1` (or higher) in the spec's `resources` block. Verified against the upstream registry on 2026-05-14 (`docker manifest inspect quay.io/jupyter/pytorch-notebook:cuda12-latest`). ## How to launch From the SPA: `/slices/new` → **JupyterLab + CUDA (PyTorch)**. Defaults: 4 vCPU / 16 GiB / 30 GiB / exposure 8888 / `requires_gpu=true`. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"ml-train\", \"runtime\": \"podman\", \"image\": \"quay.io/jupyter/pytorch-notebook:cuda12-latest\", \"resources\": { \"cpu_milli\":4000, \"mem_bytes\":17179869184, \"disk_bytes\":32212254720, \"gpu_count\":1 }, \"exposure\": [{\"port\":8888,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": {\"JUPYTER_TOKEN\":\"ml-tutorial-token\"}, \"lease_seconds\": 86400 }' | jq ``` If the cluster has no free GPU the slice will sit in `status=pending` with reason `no_eligible_host`. Use `amini hosts ls` (or the admin **Hosts** page) to confirm GPU availability. ## How to access 1. Wait for `status: running`. The PyTorch + CUDA image is ~6 GiB on a cold cache — first pull takes 5–10 minutes on a typical home link. 2. Click **Open Jupyter** in the Connect panel at the top of the slice detail page. The URL already has your token substituted in. ## Connect panel — what you'll see The Connect panel for `jupyter-gpu` is identical to the CPU [`jupyter`](jupyter.md) variant, with one tweak: - The smoke test rendered next to the **Open Jupyter** button is `import torch; print(torch.cuda.is_available(), torch.cuda.device_count())` — copy it into the first cell and you'll see `True` + the number of GPUs the engine attached. - The \"What you can do next\" links point at the GPU ML training tutorial as the obvious first move. ## Run a smoke test In a new notebook cell: ```python import torch print(\"torch:\", torch.__version__) print(\"cuda available:\", torch.cuda.is_available()) print(\"device:\", torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"cpu\") x = torch.randn(2048, 2048, device=\"cuda\" if torch.cuda.is_available() else \"cpu\") print(\"matmul result norm:\", (x @ x.T).norm().item()) ``` You should see `cuda available: True` and a non-zero norm. ## What to do next - Mount a tensor / multimodal dataset — see [`tensor-zarr-server`](../using-templates.md#data--database--storage) for Zarr-backed N-d arrays or [`multimodal-lance-server`](../using-templates.md#data--database--storage) for Lance. - Track experiments — spin up an [`mlflow`](mlflow.md) slice and set `MLFLOW_TRACKING_URI=http://:` in your notebook's env. - Long-running training? Renew the lease via the **Renew** button on the slice detail page — default is 24 hours. ## Related - [Tutorial 03 — GPU ML training on Jupyter](../../tutorials/03-gpu-ml-training-on-jupyter.md) - [Using slice templates](../using-templates.md) " }, { "slug": "tenant/templates/k3s-single-node", "title": "k3s-single-node — your own single-node Kubernetes", "description": "How to launch a per-slice k3s cluster; extract the kubeconfig; deploy your first Pod.", "section": "tenant", "body": "# `k3s-single-node` — your own single-node Kubernetes ## What it is A k3s single-node Kubernetes cluster inside a Firecracker VM. Backed by [`rancher/k3s:v1.30.6-k3s1`](https://hub.docker.com/r/rancher/k3s/tags). Listens on tcp/6443 (kube-apiserver). Verified against the upstream registry on 2026-05-14 (`docker manifest inspect docker.io/rancher/k3s:v1.30.6-k3s1`). **Not the operator-cluster runtime.** This is a *per-slice* k3s: every tenant gets their own playground. For \"I want to deploy a Pod onto Amini Cloud's existing cluster\", use the [`k8s-deploy`](../using-templates.md#workflow--orchestration) template instead. **Untrusted tenant safe.** Because the template runs on `firecracker`, even SEC-004 untrusted tenants can use it — the kernel boundary isolates a malicious k3s from the host. ## How to launch From the SPA: `/slices/new` → **k3s (single-node)**. Defaults: 4 vCPU / 8 GiB / 30 GiB / exposure 6443 / runtime `firecracker`. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"my-k3s\", \"runtime\": \"firecracker\", \"image\": \"rancher/k3s:v1.30.6-k3s1\", \"resources\": {\"cpu_milli\":4000,\"mem_bytes\":8589934592,\"disk_bytes\":32212254720}, \"exposure\": [{\"port\":6443,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": {\"K3S_KUBECONFIG_MODE\":\"644\"}, \"lease_seconds\": 86400 }' | jq ``` ## How to access 1. Wait for `status: running` and for the **Logs** tab to print `k3s is up and running`. 2. The **Connect panel** on the slice detail page surfaces the apiserver URL (`https://127.0.0.1:`). Note: k3s isn't in the wave-8 J3 per-template hint set, so the panel falls through to the generic \"custom\" recipe — you'll see the URL + a hint to bring your own auth. Use the Shell tab to extract the kubeconfig as below. 3. Pull the generated kubeconfig from inside the VM (the **Shell** tab works for this): ```bash cat /etc/rancher/k3s/k3s.yaml ``` 4. Save it locally, replace `server: https://127.0.0.1:6443` with `server: https://127.0.0.1:`, and run `kubectl --kubeconfig ~/.kube/amini-k3s get nodes`. ## Run a smoke test ```bash kubectl --kubeconfig ~/.kube/amini-k3s create deployment hello \\ --image=nginx:1.27-alpine kubectl --kubeconfig ~/.kube/amini-k3s get pods ``` You should see a single `hello-…` Pod in `Running` after a few seconds. ## What to do next - Install Helm charts, build an internal PaaS, run your CI job artifacts in your own Amini-Cloud-hosted cluster. - For a *replicated* HA k3s, fan out via the wizard's **Replicated?** field — each replica is an independent k3s; you'd then layer a multi-cluster topology on top. - For a slice that deploys *into the operator's existing cluster*, use the `k8s-deploy` template instead. ## Related - [Using slice templates](../using-templates.md) - [k3s upstream docs](https://docs.k3s.io/) " }, { "slug": "tenant/templates/minio", "title": "minio — per-tenant S3-compatible storage", "description": "How to launch MinIO; access the web console; create a bucket; smoke-test with mc.", "section": "tenant", "body": "# `minio` — per-tenant S3-compatible storage ## What it is A single-instance MinIO server. Backed by [`quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z`](https://quay.io/repository/minio/minio?tab=tags). Exposes two ports: - **9000** — S3 API (used by `mc`, `aws s3`, any S3-compatible client) - **9001** — Web console (browser UI) Verified against the upstream registry on 2026-05-14 (`docker manifest inspect quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z`). ## How to launch From the SPA: `/slices/new` → **MinIO**. Defaults: 1 vCPU / 2 GiB / 50 GiB. **Important** — the MinIO image needs a `CMD` argument (`server /data`) to start the server. The compose / runner adds this automatically for the template id; if you supply a raw spec the launch will exit immediately with no published ports. Pre-wave-9 the workaround is to use the template id, not a raw image spec. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"team-blob\", \"runtime\": \"podman\", \"image\": \"quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z\", \"resources\": {\"cpu_milli\":1000,\"mem_bytes\":2147483648,\"disk_bytes\":53687091200}, \"exposure\": [ {\"port\":9000,\"proto\":\"tcp\",\"scope\":\"tenant\"}, {\"port\":9001,\"proto\":\"tcp\",\"scope\":\"tenant\"} ], \"env\": {\"MINIO_ROOT_USER\":\"minioadmin\",\"MINIO_ROOT_PASSWORD\":\"changeme123\"}, \"lease_seconds\": 86400 }' | jq ``` ## How to access 1. Wait for `status: running`. 2. Open the slice detail page. The **Connect panel** at the top shows an **Open MinIO console** button + both port URLs (9000 for S3 API, 9001 for the browser console). 3. Sign in to the console with `MINIO_ROOT_USER` + `MINIO_ROOT_PASSWORD` from the env block — both surfaced in the Credentials section of the Connect panel (password is click-to-reveal). ## Connect panel — what you'll see The Connect panel for `minio` shows: - **Open MinIO console** button — opens the S3 API port URL. (We surface the first port; the console is on the second port. Year-1 the panel will pick the higher-numbered port when the template declares both, but for today click the URL labelled `9001/tcp` in the URLs list.) - **URLs**: one row per exposed port, both clickable + copy-able. - **Credentials**: `MINIO_ROOT_USER` (plaintext) + `MINIO_ROOT_PASSWORD` (click-to-reveal). These are the S3 access/secret key pair as well. - **mc alias** CLI snippet: `mc alias set amini http://127.0.0.1: ` — paste into your shell. - **AWS CLI** snippet with `--endpoint-url` set. - **Smoke test**: `mc ls amini/` — confirms the alias works. ## Run a smoke test Install [`mc`](https://min.io/docs/minio/linux/reference/minio-mc.html) on your laptop, then: ```bash S3PORT=$(curl -fsS -H \"Authorization: Bearer $TOKEN\" \\ \"$ADMISSION/v1/slices/$SLICE_ID/access\" | \\ jq -r '.urls[] | select(.container_port==9000) | .host_port') mc alias set amini \"http://127.0.0.1:$S3PORT\" minioadmin changeme123 mc mb amini/hello echo \"hello amini\" | mc pipe amini/hello/greeting.txt mc ls amini/hello mc cat amini/hello/greeting.txt ``` You should see `greeting.txt` listed and its contents echoed back. ## What to do next - Mount a tenant-scoped S3 bucket from a Jupyter slice — set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL=http://:` in the notebook env. - Front MinIO with a Lance dataset for multimodal training — see [`multimodal-lance-server`](../using-templates.md#data--database--storage). - Production: pin a `block` volume so objects survive a slice suspend → resume; default 50 GiB is the throwaway tier. ## Related - [Volumes and storage](../volumes-and-storage.md) - [Using slice templates](../using-templates.md) " }, { "slug": "tenant/templates/mlflow", "title": "mlflow — experiment tracking", "description": "How to launch an MLflow tracking server; point your training notebook at it; browse runs.", "section": "tenant", "body": "# `mlflow` — experiment tracking ## What it is An MLflow tracking server. Backed by [`ghcr.io/mlflow/mlflow:latest`](https://github.com/mlflow/mlflow/pkgs/container/mlflow). Listens on tcp/5000. Browse runs, register models, compare metrics in the built-in UI. Verified against the upstream registry on 2026-05-14 (`docker manifest inspect ghcr.io/mlflow/mlflow:latest`). ## How to launch From the SPA: `/slices/new` → **MLflow Tracking**. Defaults: 1 vCPU / 2 GiB / 10 GiB / exposure 5000. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"tracking\", \"runtime\": \"podman\", \"image\": \"ghcr.io/mlflow/mlflow:latest\", \"resources\": {\"cpu_milli\":1000,\"mem_bytes\":2147483648,\"disk_bytes\":10737418240}, \"exposure\": [{\"port\":5000,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": {}, \"lease_seconds\": 86400 }' | jq ``` For a production-grade setup, pin a backend store (`MLFLOW_BACKEND_STORE_URI`) to an external Postgres + an object-store artifact root (`MLFLOW_ARTIFACT_ROOT=s3://...`). The default ephemeral SQLite store works for local exploration. ## How to access 1. Wait for `status: running` (the image is ~400 MiB; a few minutes on a cold cache). 2. Click **Open MLflow UI** in the Connect panel at the top of the slice detail page. The MLflow web UI loads directly — no auth in front of it by default. **Do not** expose this to the public internet without an auth proxy. ## Connect panel — what you'll see The Connect panel for `mlflow` shows: - **Open MLflow UI** button — opens the web UI in a new tab. - **Python tracking URI** snippet: `import mlflow; mlflow.set_tracking_uri('http://127.0.0.1:')` — paste into the first cell of your training notebook. - **Smoke test**: a one-shot `mlflow.start_run() / log_param / end_run` Python line. Copy into a notebook to verify the tracking server accepts writes. ## Run a smoke test From a Jupyter notebook in the same tenant: ```python import mlflow mlflow.set_tracking_uri(\"http://127.0.0.1:\") mlflow.set_experiment(\"hello-mlflow\") with mlflow.start_run(): mlflow.log_param(\"alpha\", 0.05) mlflow.log_metric(\"rmse\", 1.234) mlflow.log_metric(\"rmse\", 0.987, step=1) print(\"logged. browse the slice URL to see the run.\") ``` Refresh the MLflow UI — the `hello-mlflow` experiment should appear with one run, two `rmse` data points, and the `alpha` parameter. ## What to do next - Wire your [`jupyter-gpu`](jupyter-gpu.md) training loop to log every epoch — see the [MLflow autolog docs](https://mlflow.org/docs/latest/tracking.html#automatic-logging). - Persist runs across restarts — pin a `block` volume and a Postgres backend store. Default ephemeral storage loses everything on a container restart. ## Related - [Using slice templates](../using-templates.md) - [Tutorial 03 — GPU ML training on Jupyter](../../tutorials/03-gpu-ml-training-on-jupyter.md) " }, { "slug": "tenant/templates/postgres", "title": "postgres — single-node PostgreSQL", "description": "How to launch a single-node Postgres slice; how to connect with psql; auth + block-volume specifics.", "section": "tenant", "body": "# `postgres` — single-node PostgreSQL ## What it is A single-instance Postgres 16. Backed by [`postgres:16-alpine`](https://hub.docker.com/_/postgres/tags?name=16-alpine) on Docker Hub. Listens on tcp/5432 inside the container. Verified against the upstream registry on 2026-05-14 (`docker manifest inspect docker.io/postgres:16-alpine`). ## How to launch From the SPA: `/slices/new` → **Postgres** → set a name → continue. Defaults: 1 vCPU / 2 GiB / 20 GiB / exposure 5432 / requires a `block` storage class for persistence. From `curl`: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"demo-pg\", \"runtime\": \"podman\", \"image\": \"postgres:16-alpine\", \"resources\": {\"cpu_milli\":1000,\"mem_bytes\":1073741824,\"disk_bytes\":21474836480}, \"exposure\": [{\"port\":5432,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": {\"POSTGRES_PASSWORD\":\"changeme\",\"POSTGRES_DB\":\"demo\"}, \"lease_seconds\": 86400 }' | jq ``` **Always set `POSTGRES_PASSWORD`** (or `POSTGRES_HOST_AUTH_METHOD=trust` in dev) — the image refuses to start without one of them. ## How to access 1. Wait for `status: running`. Cold-cache pull is ~70 MiB, ~10 s. 2. Open the slice detail page. The **Connect panel** at the top shows the rendered `psql` command line with the host, port, password, user, and db already substituted in. Copy it into your shell and you're connected. For scripted access (no browser), the same data is in the access endpoint: ```bash curl -fsS -H \"Authorization: Bearer $TOKEN\" \\ \"$ADMISSION/v1/slices/$SLICE_ID/access\" \\ | jq -r '.connect.cli_commands[] | select(.label==\"psql\") | .cmd' \\ | sh ``` ## Connect panel — what you'll see The Connect panel for `postgres` shows: - **Connection string**: `postgres://:@127.0.0.1:/` — fully substituted, copy-able with one click. Paste it into any Postgres client. - **psql** CLI command: `PGPASSWORD= psql -h 127.0.0.1 -p -U -d ` — ready to paste into your terminal. - **Python (psycopg)** CLI command: `psycopg.connect('postgres://...')` — same substituted DSN, ready for a notebook. - **Credentials**: `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` — password is masked behind a click-to-reveal eye icon, the others are plaintext. - **Smoke test**: `SELECT version();` — copy and run after `\\c`-ing into the db. - The \"Postgres template guide\" + \"Attach a block volume\" links sit under \"What you can do next\". The Connect panel never re-derives the host or port — it renders the values the engine reported via the runner inspect, so what you see is what the slice is actually listening on right now. ## Run a smoke test ```sql SELECT version(); CREATE TABLE hello (msg text); INSERT INTO hello VALUES ('it works'), ('persisted to a block volume'); SELECT * FROM hello; ``` The rows survive a slice suspend → resume cycle if the slice has a `block` volume mounted. Without a volume the data is wiped on container restart — beta limitation. ## What to do next - Attach a `block` volume so the data survives — see [`volumes-and-storage.md`](../volumes-and-storage.md) and [Tutorial 04 — Postgres with a block volume](../../tutorials/04-deploy-postgres-with-block-volume.md). - Use as a backend for your Jupyter notebook — set `POSTGRES_HOST=127.0.0.1`, `POSTGRES_PORT=` env in the notebook slice. - Need HA? Year-1 ships a `postgres-replicated` template (3 hot replicas, sync replication). Track on the roadmap. ## Related - [Tutorial 04 — Deploy Postgres with a block volume](../../tutorials/04-deploy-postgres-with-block-volume.md) - [Volumes and storage](../volumes-and-storage.md) " }, { "slug": "tenant/templates/vllm-inference", "title": "vllm-inference — OpenAI-compatible LLM serving", "description": "How to launch vLLM with a chosen model; call /v1/chat/completions; smoke-test with curl.", "section": "tenant", "body": "# `vllm-inference` — OpenAI-compatible LLM serving ## What it is vLLM's OpenAI-compatible API server. Backed by [`vllm/vllm-openai:latest`](https://hub.docker.com/r/vllm/vllm-openai/tags). Listens on tcp/8000. Implements `/v1/chat/completions`, `/v1/completions`, `/v1/models` — drop-in for any OpenAI SDK with `base_url` rewritten. Verified against the upstream registry on 2026-05-14 (`docker manifest inspect docker.io/vllm/vllm-openai:latest`). **Requires a GPU.** The vLLM image won't start without `--gpus all` (or its podman equivalent); the catalog template sets `requires_gpu=true` so admission rejects launches on hosts without free GPU capacity. **Runtime: use `podman`.** vLLM runs as a normal GPU container — GPU passthrough goes through the Docker/Podman Engine API `DeviceRequests` path (ADR 0060), not a microVM. The only runtime that works end-to-end on the shipped stack is `podman` (see [known-gaps → Runtimes](../known-gaps.md#runtimes--only-podman-works-end-to-end)). > ⚠️ **Catalog seed mismatch (open).** The seeded `vllm-inference` > catalog row still declares `runtime = firecracker` > (`db/migrations/0030_slate_templates.sql`). firecracker is > *not* implemented and would break GPU passthrough anyway, so a > slice created with the seeded default 422s at admission > (*\"runtime not supported on any approved host\"*). Until the seed > is corrected, **always send `\"runtime\": \"podman\"` explicitly** (the > examples below do) and, in the `/slices/new` wizard, change the > Runtime dropdown from `firecracker` to `podman` before launching. ## How to launch From the SPA: `/slices/new` → **vLLM inference**. Defaults: 4 vCPU / 32 GiB / 50 GiB / exposure 8000 / `requires_gpu=true`. The wizard pre-fills the runtime from the seeded catalog row, which is currently `firecracker` — **switch the Runtime dropdown to `podman`** before you launch (see the runtime note above). From `curl` — pick a model from the [vLLM supported list](https://docs.vllm.ai/en/latest/models/supported_models.html) that fits your VRAM: ```bash curl -fsS -X POST \"$ADMISSION/v1/slices\" \\ -H \"Authorization: Bearer $TOKEN\" \\ -H 'content-type: application/json' \\ -d '{ \"name\": \"llama-serve\", \"runtime\": \"podman\", \"image\": \"vllm/vllm-openai:latest\", \"resources\": { \"cpu_milli\":4000, \"mem_bytes\":34359738368, \"disk_bytes\":53687091200, \"gpu_count\":1 }, \"exposure\": [{\"port\":8000,\"proto\":\"tcp\",\"scope\":\"tenant\"}], \"env\": { \"MODEL\":\"meta-llama/Llama-3.2-3B-Instruct\", \"HUGGING_FACE_HUB_TOKEN\":\"\" }, \"lease_seconds\": 86400 }' | jq ``` Smaller models (3B parameters) fit on a single 24 GiB GPU and load in ~2 minutes. Larger models need a host with more VRAM — the slice will start, then crashloop OOM in the logs. Watch **Logs** for `HF_HOME` download progress and a `Application startup complete.` line. ## How to access 1. Wait for `status: running` AND for the logs to print `Application startup complete.` (the container is \"running\" well before the model is loaded — the first request will 503 until vLLM finishes loading). 2. The **Connect panel** at the top of the slice detail page surfaces both a curl `chat/completions` snippet (with `MODEL` substituted from the slice env) and a Python OpenAI-SDK snippet that points `base_url` at your slice. Copy and run. ## Connect panel — what you'll see The Connect panel for `vllm-inference` shows: - **URL** — the OpenAI-compatible API root. Not a browser UI; the panel does not show an \"Open in browser\" button. - **curl chat completion** snippet — a complete POST against `/v1/chat/completions` with your `MODEL` env interpolated. Paste into your shell to test end-to-end. - **Python (OpenAI SDK)** snippet: `client = OpenAI(base_url='http://127.0.0.1:/v1', api_key='not-needed')` — same shape as a real OpenAI client. Plug into your existing client code. - **Smoke test**: `curl http://127.0.0.1:/v1/models` — confirms the model finished loading. ## Run a smoke test ```bash PORT=$(curl -fsS -H \"Authorization: Bearer $T" }, { "slug": "tenant/using-templates", "title": "Using slice templates", "description": "How to pick a slice template, how template defaults compose with your spec, and how to add your own — with the live catalog as the source of truth.", "section": "tenant", "body": "# Using slice templates Amini Cloud ships a curated catalog of pre-built slice shapes. Tenants pick a template by id when creating a slice instead of composing every field by hand. Each template has pinned defaults: image, resources, exposure, env, bootstrap command. You override what you need; the rest comes from the template. > **Wave-8 J1 catalog-truth pass (2026-05-14).** Every image tag below > has been verified against the upstream registry via > `docker manifest inspect`. The matching FALLBACK list in > `web/src/routes/slices/new/+page.svelte` was updated in lock-step. > Per-template \"what this gives you + how to use it\" runbooks live in > [`templates/`](templates/) (linked from the table). Once a slice is > running, the slice-detail page renders an **\"Open this slice →\"** > button (`GET /v1/slices/{id}/access`) that hands you the host-side > URL the engine assigned — no more SSHing in to find the port. ## When to use a template - You want a Jupyter or VS Code or Postgres slice without composing every field of the spec. - You want maintainer-pinned defaults (correct image, correct port, correct env vars). - You want a slice that opts into one of the typed storage classes (`tabular`, `tensor`, `stream`, `multimodal_dataset`). If your workload is none of the above, use `template_id='custom-oci'` (or `NULL`) and supply a raw OCI image plus your own resources. ## The catalog The live catalog is **`amini.slate_templates`** — the source of truth. Don't trust a hardcoded list in any doc (this one included); read the table: ```bash psql \"$POSTGRES_DSN\" -c \" SELECT id, display_name, category, runtime, image, health_path FROM amini.slate_templates ORDER BY category, id; \" ``` The 18 templates shipped today, by id: | Category | Template ids | |---|---| | Workspace | `jupyter`, `code-server` | | Compute / dev | `nginx-web`, `ssh-server` | | Database | `postgres` | | Data plane | `redis`, `clickhouse`, `nats`, `redpanda` | | Storage | `minio` | | ML / training | `jupyter-gpu`, `mlflow` | | ML / inference | `vllm-inference`, `ollama` | | Observability | `grafana`, `prometheus` | | Chain | `aminichain-oracle` | | Escape hatch | `custom-oci` | For the per-template image, port, defaults, status, and a copy-pasteable `amini slices create --template=` command for each one, see the full reference: **[Slice templates — user reference](../user/slice-templates.md)**. > The ids above are seeded by migrations `0030`, `0035`, and `0037` > into `amini.slate_templates`. Older drafts of this page listed an > aspirational catalog (`slice_templates` table, ids like `vm-ubuntu` > and `tabular-duckdb`) that was never wired end-to-end; those ids are > **not** in the live catalog. Use the table above. ## How template defaults compose with your spec Admission expands `template_id` server-side: when you POST a slice spec with a `template_id` (the `amini slices create --template=` path does this for you), `server.go` looks the row up via `GetTemplate` and fills in the image / resources / exposure / env you didn't override. You can still POST a raw spec with no `template_id` (see [`tenant/creating-slices.md`](creating-slices.md)) — then nothing is merged and you supply every field yourself. The merge semantics: 1. Start with the template's defaults (`default_cpu_milli`, etc.). 2. Layer your spec's fields on top — your values win. 3. Validate the merged spec against `api/slice.schema.json`. 4. Apply the runtime gate (untrusted tenants only on VM runtimes). 5. Apply the quota gate (your tenant's CPU + memory + slice-count caps). ## Picking a template A short flowchart: - **Notebook?** → `jupyter` (CPU) or `jupyter-gpu` (with GPU access). - **IDE in browser?** → `code-server`. - **LLM serving?** → `vllm-inference` (GPU) or `ollama` (CPU/GPU). - **Web server?** → `nginx-web`. - **Shell access?** → `ssh-server`. - **Database?** → `postgres` (with a `block` volume). - **Object storage per tenant?** → `minio`. - **Data plane:** - Cache / pub-sub → `redis`. " }, { "slug": "tenant/using-the-model-catalog", "title": "Using the model catalog", "description": "Browse the curated model catalog, launch a model as a slice, and understand which serving backend it picks — plus why gated models need a Hugging Face token.", "section": "tenant", "body": "# Using the model catalog The model catalog is a curated list of LLMs you can launch as a slice with one click. Picking a model creates a slice running the right serving backend for that model, sized to the model's recommended resource floor, with an endpoint at a tenant-scoped URL. ## Browse the catalog Go to **Admin → Models** (`/admin/models`). Each card is one model. Filter by family (Qwen, Llama, Gemma, Mistral, …), by GPU requirement, or by edge-capability. Click a card to open the detail page (`/admin/models/{id}`), which shows the model's parameter count, context window, recommended CPU / memory / GPU, and the runtimes it can run on (`runtime_compat`). ## Launch a model From the detail page, click **Launch**. You pick: - **Name** — lowercase a-z0-9- slug; becomes the slice name and its ingress host. - **Exposure scope** — `tenant` (only signed-in members reach it), `shared`, or `public` (owner-only). - Optional resource bumps. You can raise the CPU / memory / GPU above the recommended floor; you can never shrink below it. Confirm, and amini creates the slice. You land on the slice detail page; wait for `status=running`, then the endpoint URL appears in the **Access** card. ## Which backend it launches on amini does not run every model the same way. It reads the model's `runtime_compat` list and picks the first runtime in this preference order that the model supports: ``` ollama > vllm > hf-transformers > llama-cpp ``` What each picks, and what you do once it's up: | Runtime chosen | Slice template | What it is | How you reach it | |---|---|---|---| | `ollama` | `ollama` | Ollama serving the model. The model is pulled into the slice right after boot. | Ollama HTTP API on port 11434 (OpenAI-compatible shim at `/v1`). | | `vllm` | `vllm-inference` | vLLM-OpenAI loads the model at boot from its `--model` arg. | OpenAI-compatible API on port 8000. | | `hf-transformers` | `jupyter-gpu` | A GPU JupyterLab notebook with `HF_MODEL_ID` pre-set. No serving backend is started for you. | Open the notebook (port 8888) and load the model in a cell with `AutoModelForCausalLM.from_pretrained(os.environ['HF_MODEL_ID'])`. | | `llama-cpp` | `custom-oci` | A llama.cpp server that fetches the GGUF at boot. | llama.cpp HTTP server on port 8080. | The `ollama` and `vllm` paths give you a serving endpoint with nothing left to do. The `hf-transformers` path deliberately drops you into a **notebook** instead — that is the fallback when a model has no turnkey serving runtime, and it's also where gated models land (see below). ## Calling a serving endpoint For `ollama` and `vllm` models, the endpoint is OpenAI-compatible: ```bash curl -X POST https://..amini.example/v1/chat/completions \\ -H 'Authorization: Bearer ' \\ -d '{\"model\":\"\",\"messages\":[{\"role\":\"user\",\"content\":\"Hi\"}]}' ``` The `Authorization` header is forwarded to the container untouched, so vLLM / Ollama treat it as their own API key. ## Gated models need a Hugging Face token Some models are **gated** on Hugging Face: the repo is public to browse, but downloading the weights requires a Hugging Face account that has accepted the model's license, plus an access token. Google's `google/gemma-4-*` family is the common example in the catalog. What this means for you: - **The backend choice.** The gated Gemma 4 rows lead with `hf-transformers` in their `runtime_compat`, so amini routes them to a **Jupyter GPU notebook**, not a serving backend. This is by design — it avoids crash-looping a vLLM/Ollama slice that can't fetch the weights. - **The symptom without a token.** In the notebook, the first `from_pretrained(...)` call against a gated repo fails with a 401 / \"gated repo\" / \"you must be authenticated\" error. Nothing crash-loops — the slice stays up, the download is just refused. - **The fix.** Configure a Hugging Face token so the download is authenticated. The operator sets `AMINI_HF_TOKEN` for the platform (see `deploy/compose/.env.example`); inside the n" }, { "slug": "tenant/volumes-and-storage", "title": "Volumes and storage", "description": "The typed storage classes, per-tenant vs per-user scoping, how to mount, and how lineage works for datasets.", "section": "tenant", "body": "# Volumes and storage Amini Cloud's data plane is **typed** — the slice spec declares what *shape* of data it needs (an object bucket, a block volume, a tensor dataset…) and admission resolves the right backend. The schema defines seven storage classes (migration 0003 + 0006); the **two that ship wired end-to-end today** are `object` (SeaweedFS / S3-compatible) and `block` (local-zfs); the remaining five are roadmapped. This page is the tenant-facing how-to. ## The classes ```sql SELECT id, backend, description FROM amini.storage_classes ORDER BY id; ``` | `id` | Backend | What it stores | Today | |---|---|---|---| | `object` | SeaweedFS / MinIO (S3-compatible) | S3-compatible bucket storage | **shipped** | | `block` | local-zfs (single-host) → Longhorn (roadmap) | Block volumes (databases, MinIO data dirs) | **shipped** | | `tabular` | Postgres (default) → Iceberg (roadmap) | SQL-queryable tables | `roadmap` | | `tensor` | Zarr-over-SeaweedFS (default) → TileDB (roadmap) | N-dim arrays (xarray-friendly) | `roadmap` | | `stream` | Redpanda (Kafka-compatible) | Kafka-protocol append-only streams | `roadmap` (per-slice template exists; first-class data-plane wiring planned) | | `archive` | SeaweedFS cold tier | Slow, cheap object storage | `roadmap` | | `multimodal_dataset` | Lance (default) → deltalake (roadmap) | Mixed text/image/audio/tensor with branch/commit versioning | `roadmap` | > **Roadmap:** the five `roadmap` rows exist in `amini.storage_classes` > so admission accepts spec references today, but the per-class > backend wiring + janitor + credential mint paths are not yet shipped. > Use `object` (S3) or `block` (local-zfs) for production work; the > other classes are available via per-slice templates today (e.g. the > `stream-redpanda` slice template stands up a Redpanda broker you > can talk Kafka to) and will become first-class data-plane primitives > in a later wave. Pick by **what shape your data has**, not by what backend you want. The backend can change (Year-1 Longhorn replaces local-zfs for `block`) without your code knowing. ## Two scopes: per-tenant and per-user The `volumes.owner_subject` column (migration 0003) controls who can mount a volume: | `owner_subject` | Who can mount | |---|---| | `NULL` | Any tenant member | | your OIDC subject | Only you, unless you grant via `slice_grants` on the slice that mounts it | Per-user is the default for ML work — every researcher has their own private workspace. Per-tenant is the default for shared resources — the team's MLflow artefact store, a shared cache, a public read-only reference dataset. ## Volumes vs datasets Two related concepts: - **`volumes`** — a single blob of storage. One mountpoint per slice. Lifecycle = the volume row. Used for \"Postgres data lives here\", \"MinIO writes objects here\", \"Jupyter user home directory.\" - **`datasets`** + **`dataset_versions`** + **`dataset_components`** — named, typed, versioned, lineage-tracked collections. Used for ML training data, model artefacts, evaluation suites. See [Concept — Data plane](../concepts/data-plane.md). If you're not sure: start with a volume. Promote to a dataset later when you want immutable versions + lineage. ## Creating a volume > **Beta caveat:** the CLI `amini volumes create` and the matching HTTP > endpoint are scaffolded; in this beta, create volumes via SQL. ### Per-user, object class ```bash psql \"$POSTGRES_DSN\" < **The \"three\" is illustrative — Amini Cloud supports any N ≥ 1 > hosts.** This walkthrough uses three concrete machines because that > is the most common heterogeneous cluster (GPU + laptop + workstation), > but the install + join + approve flow is identical for N=2, N=10, > or N=50. See the **Scaling to N hosts** callout below for the audit > that backs the explicit \"tested up to N=50\" claim and the multi-host > onboarding shortcut. This is the canonical concrete walkthrough for the most common non-trivial Amini Cloud cluster: one GPU compute host (DGX Spark), one operator laptop (MacBook), one general-purpose workstation (Linux HP / Dell / etc). After 30 minutes you should be able to: - Sign in to `https://amini.localtest.me:8446` (or your real domain) - See three hosts under `/admin/hosts` - See one cluster under `/admin/clusters` with all three hosts attached - Schedule a slice that lands on the DGX (because it's the only host that advertises `compute_class=gpu`) - Open the slice's sharable URL from the MacBook *or* the workstation You do NOT need Tailscale for this — Amini Cloud manages a tenant tailnet itself once the hosts join. ## Host roles | Host | Role | Compute class | Notes | |---|---|---|---| | DGX Spark (ARM, Grace-Blackwell, Ubuntu) | **compute + control plane** | `gpu` | Runs `amini-admission` + Postgres + Authelia + Caddy ingress here. Has the GPU. | | MacBook (Apple Silicon, macOS) | **compute + operator workstation** | `cpu` | Joins the tailnet, gets host-agent under launchd. Operator does admin work from here. | | Linux HP workstation (x86_64, Ubuntu) | **compute** | `cpu` | Joins, attests capacity, accepts slice placements. | The DGX is the natural control-plane host because (a) it's always-on, (b) it's wired (vs the MacBook on Wi-Fi), and (c) it has the GPU we want to keep scheduler-visible. The two CPU hosts are scheduler-equivalent — Amini Cloud doesn't care which is which. ## Pre-flight (5 min) On **all three** hosts: ```bash # Linux + macOS: SSH-able, Docker / Podman available, /etc/hosts has: sudo bash -c 'cat >> /etc/hosts <

use # scaffolded # Or directly: psql \"" }, { "slug": "troubleshooting/known-issues", "title": "Known issues", "description": "Open audit findings, unimplemented CLI leaves, runtime quirks (no live migration, userspace tailscaled, ICMP), AminiChain placeholder ABI, vault dev-mode caveats.", "section": "troubleshooting", "body": "# Known issues Amini Cloud is in beta. This page is the **canonical list** of bugs, limitations, and deliberate scope cuts. The accuracy contract in [`README.md`](../README.md) says we ship docs that match the code; if a feature is broken, it gets a row below, not a marketing paragraph. The corresponding \"what's planned but not built\" view is in [Beta release notes](../beta-release-notes.md). Where they overlap, the release notes are the higher-level summary and this page is the debug-day detail. ## Open security audit findings The reviewer pipeline ran on `b772df0` ([`docs/audits/2026-05-13-security-b772df0.md`](../../audits/2026-05-13-security-b772df0.md)). Five were resolved in that commit; three remain open. The audit is re-read at the top of every wave. ### SEC-002 — pre-auth key in Nomad job spec (open) `internal/nomad/jobspec.go::BuildPodmanJob` embeds the full Headscale pre-auth key into the Nomad job as `EmbeddedTmpl`. Anyone with `nomad job inspect` access (the Nomad UI, any operator with `read` ACL on that namespace, the Nomad audit log) can read the literal key during its 5-minute TTL window. The intended fix — write the key to a Nomad Variable at `nomad/jobs/slice-/preauth_key` and consume via `{{ with nomadVar … }}` — is scoped at one focused agent dispatch. **Workaround:** restrict Nomad UI access to platform admins only. The Nomad ACL system handles this. Do not give tenants Nomad ACLs. ### SEC-003 — `-insecure` flag on amini-host-agent (open) `cmd/amini-host-agent/main.go` accepts a `-insecure` flag that drops mTLS entirely on the gRPC server. There is no DB-level cluster marker that says \"this host requires mTLS\"; admission has no way to know. Plus admission's own `hostclient.Client` defaults to `grpc.WithTransportCredentials(insecure.NewCredentials())` when callers don't pass dial opts — and `cmd/amini-admission/main.go` currently doesn't. The only working configuration is \"both sides insecure.\" **Workaround:** in beta, run admission + host-agent on the same physical network with no third-party tenants. The fix lives in the next round: hide `-insecure` behind a `//go:build dev` tag; load TLS material from env paths in `cmd/amini-admission/main.go`. ### SEC-007 — host-agent gRPC `ListSlices` / `DescribeSlice` (open) `internal/hostagent/describe.go` filters by an optional `tenant_id` **supplied by the caller** — it doesn't verify the caller is allowed to inspect that tenant. With SEC-003 still open, any process on the host's network can call `ListSlices(req.TenantId=\"\")` and enumerate every tenant's slice on the host (netns names, ULA prefixes, exposure ports, runtime, tenant slug). **Workaround:** same as SEC-003 — keep admission + host-agent on a private network until both fixes ship together (they share a code path). ## Unimplemented CLI leaves The `amini` CLI binary (`cmd/amini/cmd/`) ships with a complete command tree so docs and demos can reference real commands. Almost every leaf currently returns `\"not yet implemented\"` until `internal/slateclient/` lands. The shape is real; the body is missing. What works: - `amini version` — prints `amini (dev build) — see https://github.com/aminitech/amini-cloud` - `amini admin kpi list` — reads the static catalog from `internal/sla/kpi.go`; works without an HTTP client What doesn't (returns \"not yet implemented\"): - `amini login` / `logout` / `whoami` — the device-code OIDC flow - `amini slices ls/get/create/ssh/logs/suspend/resume/delete/renew` - `amini templates ls/show` - `amini volumes ls/create/delete` - `amini grants ls/add/remove` - `amini admin tenants ls/create/show/suspend` - `amini admin hosts ls/show/drain` - `amini admin cluster status/add-host/drain/cordon/uncordon/remove-host/info/check` - `amini admin policy show/apply` - `amini admin reviewer-dispatch` - `amini admin upgrade --check` - `amini update` **Workaround:** every tutorial uses `curl` against the HTTP API or `psql` against Postgres. See: - [Tutorial 01 — Your first slice](../tutorials/0" }, { "slug": "tutorials/00-your-first-compute-job", "title": "Tutorial 00 — Your first compute job", "description": "Clone the repo, bring up Compose, onboard the demo tenant, click the jupyter template, run `print(2+2)`, tear down. 10 minutes, no SSH, no Tailscale, no Ansible.", "section": "tutorials", "body": "# Tutorial 00 — Your first compute job > [!tip] **Start here.** Tutorial 01 (\"Your first slice\") covers the > richer Tailscale/SSH path. This is the shorter version: **make up → > demo-tenant → click jupyter → run a cell → delete**. Eight steps, ten > minutes. The goal is to prove Amini Cloud executes your code end-to-end on > a single host with no extra moving pieces. > > Read [`known-gaps.md`](../tenant/known-gaps.md) **first** if you want > the honest list of what isn't shipped yet. The short version: only > the `podman` runtime works end-to-end today; SSH-into-slices over > Tailscale is planned, not built; storage backends past `local-zfs` > are planned. This tutorial sidesteps all of that. ## What you'll have at the end A running `jupyter` slice in the `demo` tenant, reachable from your laptop's browser, that you've successfully run a Python cell inside and poked at via a shell. Plus a clean teardown. The audit log will carry one `slice.create` + one `slice.exec` (or `slice.shell.open`) + one `slice.terminate` row that you can verify. ## What you need - Docker Desktop or Docker Engine + Compose v2. - `curl` and `jq` on your shell PATH. - A clone of `amini` checked out somewhere. That's it. No real GPU, no second host, no Tailscale client, no SSH key. ## Step 1 — `make up` From the repo root: ```bash make up ``` This brings up the single-host Compose stack: Postgres, Vault (OpenBao), Caddy, one Headscale, the `amini-admission` control plane on `http://localhost:18080`. (Authelia is optional — it only comes up under the `full` profile, for the legacy OIDC sign-in path; native + wallet sign-in needs no IdP.) `make up` does **not** start the web UI. Open a second shell and run: ```bash cd web && pnpm install && pnpm dev # → web UI on http://localhost:5173 ``` (Or `make web-watch` from the repo root — runs Vite + Vitest concurrently.) Wait for the admission `/healthz` to go green: ```bash curl -fsS http://localhost:18080/healthz # → {\"status\":\"ok\"} ``` > [!info] Ports cheat-sheet > Compose intentionally avoids `:80` / `:443` / `:8080`. The admission > API is **`18080`**, the web UI is **`5173`**, and Vault is **`8210`**. > (Authelia on **`9091`** only runs under the optional `full` profile — > see [OIDC sign-in setup](../oidc-setup.md).) The full list is in > [`deploy/compose/.env.example`](../../../deploy/compose/.env.example). ## Step 2 — `make demo-tenant` Create the `demo` tenant + a `wallet-0xabc…` dev user that you can auth as: ```bash make demo-tenant ``` What this does: inserts a `tenants` row with slug `demo`, a `tenant_users` row for the dev wallet subject, and seeds the per-tenant Headscale namespace. ## Step 3 — Set up dev auth > [!warning] **`AminiDev` is a local-machine-only shortcut, and it's off > by default even here.** It needs **three** env vars, all true, on the > admission process: `AMINI_ENABLE_DEV_BEARER=true`, > `WALLET_DEV_AUTO_PROVISION=true`, `WALLET_DEV_AUTO_TENANT=demo`. > `deploy/compose/.env.example` ships the last two but leaves > `AMINI_ENABLE_DEV_BEARER` **unset** (= off) — a deliberate gap added > after this bearer was found reachable on a shared dev stack and > disabled there (security finding F-001). Add > `AMINI_ENABLE_DEV_BEARER=true` to your own `deploy/compose/.env` and > recreate admission before Step 3 works: > ```bash > echo 'AMINI_ENABLE_DEV_BEARER=true' >> deploy/compose/.env > docker compose --env-file deploy/compose/.env -f deploy/compose/compose.yml \\ > up -d --no-deps --force-recreate amini-admission > ``` > Without it you'll get `401 malformed Authorization header` here, which > reads like a broken credential rather than a disabled feature. Never > set this on a box anyone else can reach — see > [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api) > for the API-key path that works everywhere, including here. Export the dev token you'll attach to every request: ```bash export ADMISSION=http://localhost:18080 export TOKEN=\"AminiDe" }, { "slug": "tutorials/01-your-first-slice", "title": "Tutorial 01 — Your first slice", "description": "Spin up a single-host JupyterLab slice and open it in your browser.", "section": "tutorials", "body": "# Tutorial 01 — Your first slice > [!tip] **You can complete this tutorial via the dashboard OR the > CLI; both paths are shown side-by-side below.** Each `### N. Step` > heading carries a CLI block followed by a \"**Dashboard equivalent:**\" > note showing the same action in the Amini Cloud web UI. Pick whichever > matches your taste; the resulting `amini.slices` row is identical. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* ![Step 1 — Signed-in dashboard, wallet address in the header](./img/01/step-1-dashboard.png) > [!info] Where do I find my wallet address? > Your wallet address is shown in the **top-right of every dashboard > page** after you sign in. It looks like > `wallet-0x9937…f43419`. The same address appears under > **Account → Identities**. > [!info] Where do I find my tenant slug? > The tenant slug is shown in the dashboard header next to your wallet > address, on the **Account** page, and via `amini tenants list` from the > CLI. The Compose default is `demo`. This is the very first thing every operator does. By the end you will have a working JupyterLab slice on a single-host Amini Cloud install, reachable from your laptop's browser over the tenant tailnet. ## What you'll have at the end A running `jupyter` slice in the `demo` tenant. It will be reachable from your laptop at `http://my-first-jupyter.demo.amini.ts.net:8888` (or the equivalent MagicDNS name your operator has configured). You will be able to open a shell on it via the dashboard **Shell** tab and you will have created a slice row in `amini.slices` with audit trail. ## What you need - An Amini Cloud single-host stack already running per [Single-host installation](../installation/single-host.md). Verify with `curl -fsS http://localhost:18080/healthz`. - The `demo` tenant created (`make demo-tenant` already run). - An account on the platform — sign up natively (email+password) or use a wallet. The Compose default has `AUTO_PROVISION_USERS=true` so the first authenticated call creates your `tenant_users` row. - A Tailscale client installed on the machine you'll use to reach the slice. On Linux: `curl -fsSL https://tailscale.com/install.sh | sh`. - `curl` and `jq` on whatever machine you're driving the API from. ## Steps ### 1. Sign in Amini Cloud's default sign-in is **native email+password** plus **wallet sign-in** (per [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md)). Sign in from a shell to a cookie jar: ```bash # Email + password → slate_session cookie (saved to amini.cookies) curl -fsS -c amini.cookies -X POST http://localhost:18080/v1/auth/login \\ -H 'content-type: application/json' \\ -d '{\"email\":\"you@example.com\",\"password\":\"\"}' ``` New here? Create the account first at `http://localhost:5173/signup`. > [!info] **Dev shortcut, your own machine only.** On a local Compose stack > you own you can skip the cookie dance and use the `AminiDev` bearer shim > (`Authorization: AminiDev admin@demo`) instead of `-b amini.cookies` in > the curl blocks below — [Tutorial 00](00-your-first-compute-job.md) uses > it. It needs **three** flags on admission, all true: > `AMINI_ENABLE_DEV_BEARER=true` + `WALLET_DEV_AUTO_PROVISION=true` + > `WALLET_DEV_AUTO_TENANT=demo` — the Compose *example* only sets the last > two, so add the first to your own `deploy/compose/.env` and recreate > admission (see [Tutorial 00, Step 3](00-your-first-compute-job.md#step-3--set-up-dev-auth)). > Never enable it on a box anyone else can reach. On any shared or public > deployment, mint an API key instead — see > [Getting a token to call " }, { "slug": "tutorials/02-share-a-slice-with-a-teammate", "title": "Tutorial 02 — Share a slice with a teammate", "description": "Add a second user to your tenant and grant them access to one slice via slice_grants.", "section": "tutorials", "body": "# Tutorial 02 — Share a slice with a teammate > [!tip] **You can complete this tutorial via the dashboard OR the > CLI; both paths are shown side-by-side below.** The grant > management surface lives at **`/grants`** in the dashboard; the > CLI / SQL path produces the same `slice_grants` row. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find a teammate's OIDC subject? > The OIDC `sub` claim is whatever your IdP issues — usually the > username or email. The teammate can read it from their `/account` > page after they sign in, or from the JWT payload of any token they > hold (base64-decode the second segment). In Amini Cloud, every slice has exactly one owner. Other users in the same tenant cannot see, list, or reach your slices unless you give them an explicit grant. This tutorial walks the operator path of adding a second user and the tenant path of sharing one slice with them. ## What you'll have at the end A second tenant user (Bob) signed in to the `demo` tenant. The slice you created in [Tutorial 01](01-your-first-slice.md) shared with Bob under `use` scope. Bob can SSH into the slice and reach Jupyter; he cannot delete it. ## What you need - Tutorial 01 completed: a running `my-first-jupyter` slice you own. - A second machine (or a second browser profile + Tailscale account) to play the part of Bob. - Operator access to Authelia (or whichever IdP your Amini Cloud install uses) to create the second account. - The same `TOKEN` and `SLICE_ID` env vars from Tutorial 01. ## Steps ### 1. The operator creates Bob's IdP account This step depends on which IdP your operator wired up. For the Compose-default Authelia, edit `deploy/compose/authelia/users_database.yml` and add Bob's entry, then bounce Authelia: ```bash # placeholder: substitute the operator's real path cd /path/to/amini $EDITOR deploy/compose/authelia/users_database.yml docker compose -f deploy/compose/compose.yml restart authelia ``` A fresh Authelia entry looks like: ```yaml users: bob: displayname: \"Bob\" password: \"$argon2id$...\" # placeholder: generate via 'authelia hash-password' email: bob@example.com groups: - members ``` If the operator runs Dex, Keycloak, or any other OIDC issuer, the steps are equivalent: create the user, ensure the IdP issues tokens with the right `iss` claim for the tenant. **Dashboard equivalent:** user-creation in Authelia is out-of-band of Amini Cloud's UI (it lives in the IdP, not amini-admission). The dashboard **`/admin/tenants`** page lists tenants and lets an admin create new ones, but per-user provisioning is delegated to the upstream OIDC provider's own admin UI. ![Step 1 — Admin tenants list (user provisioning is upstream of amini)](./img/02/step-1-admin-tenants.png) ### 2. Bob signs in and his `tenant_users` row is created Bob opens the Authelia URL in a fresh browser, signs in, and grabs his ID token the same way you did in Tutorial 01. He runs: ```bash # On Bob's machine export BOB_TOKEN=\"\" # placeholder curl -fsS -H \"Authorization: Bearer $BOB_TOKEN\" \\ http://localhost:18080/v1/tenants/me | jq ``` With `AUTO_PROVISION_USERS=true` (the Compose default), the first call inserts a row into `amini.tenant_users` for Bob in the `demo` tenant with role `member`. He should see: **Dashboard equivalent:** Bob signs in to `/login` in his own browser. The same auto-provision path runs server-side; he lands on `/dashboard` showing his tenant + subject in the header. Bob's `/slices` page is empty (existence-disclosure rule, next step). ```json { \"tenant\": { \"slug\": \"demo\", \"" }, { "slug": "tutorials/03-gpu-ml-training-on-jupyter", "title": "Tutorial 03 — GPU ML training on Jupyter", "description": "Provision a jupyter-gpu slice on a GPU host, mount a tensor volume of training data, and run a PyTorch model.", "section": "tutorials", "body": "# Tutorial 03 — GPU ML training on Jupyter > [!tip] **You can complete this tutorial via the dashboard OR the > CLI.** The slice-creation step is dashboard-equivalent at > **`/slices/new`** (pick the **jupyter-gpu** template). The dataset > registration and volume creation steps stay CLI/SQL for now — > dataset CRUD UI is Year-1 work. Both paths land at the same > running slice. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find a host with a GPU? > Open **Admin → Hosts** (`/admin/hosts`) in the dashboard and filter > the table by **GPU count ≥ 1**. The `gpu_kind` column shows the > vendor. From the CLI run `amini admin hosts ls --gpu`, or query > `amini.hosts` directly as shown above. If your GPU host doesn't show > up, `amini-host-agent` either isn't running or isn't detecting the > device. ![Step 1 — Admin hosts page filtered to gpu_count ≥ 1](./img/03/step-1-admin-hosts-gpu.png) This tutorial walks an ML researcher's workflow: stand up a GPU-backed Jupyter slice, mount a tensor dataset, and train a small PyTorch model end-to-end. ## What you'll have at the end A running `jupyter-gpu` slice on a host with at least one NVIDIA GPU. The slice will have a `tensor` dataset mounted at `/data` (read-only) plus a per-user `object` volume mounted at `/workspace` (read-write). You will train a tiny PyTorch image classifier and save checkpoints to the workspace volume, exercising Amini Cloud's GPU scheduling, data plane, and lineage. ## What you need - A multi-host Amini Cloud install with at least one compute host whose heartbeat reports `gpu_count >= 1` and `gpu_kind` set (NVIDIA only at beta — AMD MI300 is `planned`). Verify: ```bash psql \"$POSTGRES_DSN\" -c \" SELECT id, hostname, compute_class, gpu_count, gpu_kind FROM amini.hosts WHERE gpu_count >= 1; \" ``` If you see zero rows on a single-host install with a GPU, `amini-host-agent` either isn't running or isn't detecting the device. See [Host management](../operator/host-management.md). - Your tenant has a GPU quota high enough for `gpu_count=1`. The tenant `quota` model in beta is CPU + memory + slice-count only; GPU is gated by host availability, not per-tenant quota (Year-1 work). - A signed-in session — sign in natively (email+password) or with a wallet and save the cookie to `amini.cookies` (see [Tutorial 01 — Step 1](01-your-first-slice.md#1-sign-in)), then mint an API key from it (see [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api)). On a local Compose stack **you own**, the `AminiDev admin@demo` bearer shim works too, but only with `AMINI_ENABLE_DEV_BEARER=true` set — see [Tutorial 00, Step 3](00-your-first-compute-job.md#step-3--set-up-dev-auth). - A small training dataset you've uploaded to your tenant's object bucket (any S3-compatible client works — see [Volumes and storage](../tenant/volumes-and-storage.md)). - `curl`, `jq`, and the official Tailscale client on the laptop you'll drive Jupyter from. ## Steps ### 1. Create the training-data tensor dataset You'll mount a Zarr-on-SeaweedFS tensor for read-only training data. The dataset is created at the platform level by inserting rows into `datasets` + `dataset_versions` + `dataset_components`. In beta this is a direct psql operation: ```bash psql \"$POSTGRES_DSN\" <<'SQL' WITH ds AS ( INSERT INTO amini.datasets (tenant_id, owner_subject, name, primary_class, description) VALUES ( (SELECT id FROM amini.tenants WHERE slug='demo'), 'you@example.com', 'cifar10-mini', 'tensor', 'CIFAR-10 mini training subset, Zarr-formatted' ) R" }, { "slug": "tutorials/04-deploy-postgres-with-block-volume", "title": "Tutorial 04 — Deploy Postgres with a block volume", "description": "Provision a postgres slice on a block volume; connect to it from a sibling slice over the overlay.", "section": "tutorials", "body": "# Tutorial 04 — Deploy Postgres with a block volume > [!tip] **You can complete this tutorial via the dashboard OR the > CLI; both paths are shown side-by-side below.** Volume creation > lives at **`/volumes`** in the dashboard; slice creation at > **`/slices/new`** (pick the **postgres** template). The mount > wiring still needs SQL until the slice-mount UI ships. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find the available storage classes? > Open **Volumes** (`/volumes`) and click **New volume** — the > **Storage class** dropdown lists every class the cluster has > configured (`block (local-zfs)`, `object (SeaweedFS)`, `tensor`). > The CLI prints them via `amini admin storage-classes ls`. This tutorial walks the canonical \"stateful service\" pattern: a Postgres slice mounted on a persistent block volume, with a sibling client slice that connects over the tenant tailnet. ## What you'll have at the end A `postgres` slice running on a `block` storage class volume (backed by `local-zfs` at MVP). A second slice (`postgres-client`) that connects to it by MagicDNS name over the overlay and issues a SQL query. ## What you need - A running Amini Cloud stack and the `demo` tenant (Tutorial 01 prereqs). - `TOKEN` exported in your shell. - A compute host with the `local-zfs` storage class configured (the default for the single-host Compose stack with a ZFS pool; multi- host needs each compute host onboarded via `scripts/onboard-server.sh`). - `curl`, `jq`, and `psql` for inspection. ## Steps ### 1. Create the block volume The `postgres` catalog template defaults to `requires_storage_classes ARRAY['block']`. In beta, volume creation is a direct insert: ```bash psql \"$POSTGRES_DSN\" <<'SQL' INSERT INTO amini.volumes (tenant_id, owner_subject, name, storage_class, size_bytes, backend_ref) VALUES ( (SELECT id FROM amini.tenants WHERE slug='demo'), 'you@example.com', 'pg-data', 'block', 21474836480, -- 20 GiB 'zfs:amini/pgdata-pg-prod-1' ); SQL ``` The `backend_ref` is the host-agent's contract for resolving the underlying zvol. The host-agent reads it at `PrepareSlice` time, creates the zvol if it doesn't exist (idempotent), and mounts it into the slice. **Dashboard equivalent:** open **`/volumes`**. In the **New volume** card: - **Name** → `pg-data` - **Storage class** → `block (local-zfs)` - **Size (GiB)** → `20` Click **Create volume**. The dashboard POSTs the same row; the new volume's UUID becomes your `$VOLUME_ID`. ![Step 1 — Volumes page with the New volume affordance](./img/04/step-1-volumes-new.png) Capture the new volume id: ```bash PG_VOLUME_ID=$(psql -At \"$POSTGRES_DSN\" -c \\ \"SELECT id FROM amini.volumes WHERE name='pg-data'\") echo $PG_VOLUME_ID ``` > **Beta caveat:** `local-zfs` block volumes are **single-host pinned**. > A slice mounting a `local-zfs` volume can only run on the host that > physically owns the zvol. Year-1 Longhorn moves the data with the > slice; for now, plan accordingly. See > [Beta release notes](../beta-release-notes.md). ### 2. Provision the Postgres slice Save as `pg-slice.json`: ```json { \"name\": \"pg-prod-1\", \"runtime\": \"podman\", \"image\": \"postgres:16-alpine\", \"resources\": { \"cpu_milli\": 1000, \"mem_bytes\": 1073741824, \"disk_bytes\": 21474836480 }, \"exposure\": [ {\"port\": 5432, \"proto\": \"tcp\", \"scope\": \"tenant\"} ], \"env\": { \"POSTGRES_PASSWORD\": \"change-me-please\", \"POSTGRES_DB\": \"appdb\", \"POSTGRES_USER\": \"appuser\", \"PGDATA\": \"/var/lib/postgresql/data/pgdata\" }, \"lease_seconds\": 604800 } ``` Submit: ```bash curl -fsS -X POST http://localhost:18080/v1/slices \\ -H" }, { "slug": "tutorials/05-onboard-a-tenant", "title": "Tutorial 05 — Onboard a tenant", "description": "Operator workflow for creating a new tenant, generating a one-time admin pre-auth key, and handing off to a tenant admin.", "section": "tutorials", "body": "# Tutorial 05 — Onboard a tenant > [!tip] **You can complete this tutorial via the dashboard OR the > CLI.** The dashboard at **`/admin/tenants`** can create the > `amini.tenants` row and read existing tenants. The systemd / > Headscale / Nomad-namespace side of provisioning still requires the > reference script (`scripts/provision-tenant.sh`) — those happen > off-box. Run the script for the full workflow; use the dashboard > for the row-level CRUD. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] How do the tenant's users sign in? > By default they sign in **natively** (email+password) or with a > **wallet** — no IdP to wire (see > [ADR 0069](../../decisions/0069-native-auth-replace-authelia.md)). > The `oidc_issuer_url` column on `amini.tenants` is only relevant if > you opt this tenant into the legacy OIDC path (Authelia / Dex / > Keycloak / the tenant's own IdP); see > [OIDC sign-in setup](../oidc-setup.md). Leave it unset for native + > wallet sign-in. > [!info] Where do I find the tenant slug? > The slug is whatever you choose at provisioning time — the URL-safe > identifier the operator picks for the tenant (e.g. `acme`, > `team-blue`). It's the value of the `slug` column on > `amini.tenants` and shows up in the dashboard's > **Admin → Tenants** list. ![Step 1 — Admin tenants list before onboarding `acme`](./img/05/step-1-admin-tenants.png) This is the operator's workflow when a new organization joins your Amini Cloud install. By the end you'll have a fully-provisioned tenant with its own Headscale process, its own ULA `/48`, its own Nomad namespace with quota, and a one-time admin pre-auth key ready to hand off. ## What you'll have at the end A new tenant called `acme` (substitute your real slug): - Row in `amini.tenants` with `status=active`. - Postgres schema `headscale_acme` for its Headscale state. - Systemd unit `headscale@acme.service` running on `amini-cp`. - Nomad namespace `acme` with the configured CPU + memory quota. - IPv6 ULA `/48` reserved for the tenant. - One-time admin pre-auth key delivered to the tenant admin out-of- band. ## What you need - Operator (root or sudoer) shell on `amini-cp`. - The Amini Cloud stack already running per [Three-host cluster installation](../installation/three-host-cluster.md) or [Single-host installation](../installation/single-host.md). - `psql`, `nomad`, `headscale`, `systemctl`, `install`, `openssl`, `awk`, `sed` on PATH — checked by the provisioning script. - Decision on how the tenant signs in: native email+password + wallet (the default — nothing to wire) or, optionally, an external OIDC issuer URL if you're federating against an IdP (see [OIDC sign-in setup](../oidc-setup.md)). - Decision on whether the tenant is `untrusted=true`. Untrusted tenants are restricted to VM-isolation runtimes (firecracker, qemu, kata, Kubernetes with a VM-isolated runtime_class). See [Concept — Tenants and users](../concepts/tenants-and-users.md). ## Steps ### 1. Decide the tenant inputs - **Slug.** Lowercase, DNS-safe, 3–32 chars (`^[a-z][a-z0-9-]{1,30}[a-z0-9]$`). Will appear in MagicDNS as `.acme.amini.ts.net`. - **Display name.** Human-friendly. - **OIDC issuer.** The reference `scripts/provision-tenant.sh` still **requires** an `--oidc-issuer` value and writes it to `amini.tenants.oidc_issuer`. It is only load-bearing if you opt this tenant into the legacy OIDC sign-in path (see [OIDC sign-in setup](../oidc-setup.md)); for the default native + wallet sign-in it is unused. Pass any reachable HTTPS URL serving `/.well-known/openid-configuration` (e." }, { "slug": "tutorials/06-register-as-an-aminichain-oracle", "title": "Tutorial 06 — Register as an AminiChain Oracle", "description": "Wire OpenBao with an Amini wallet key, register amini as an AminiChain Oracle, and flip audit anchors from stub to real on-chain tx hashes. Cross-references the operator runbook.", "section": "tutorials", "body": "# Tutorial 06 — Register as an AminiChain Oracle > [!tip] **The canonical runbook is at > [`docs/operators/aminichain-oracle-bootstrap.md`](../../operators/aminichain-oracle-bootstrap.md).** > That doc is the single source of truth for the bootstrap as of Wave 14; > it ships a `--preflight` gate, an idempotent one-shot script > (`scripts/aminichain-oracle-bootstrap.sh`), and 7 manual steps you > can walk by hand. **Use the runbook for the actual flow.** This > tutorial keeps the longer narrative + the dashboard-side observations > below the runbook hand-off. > [!info] **What changed in Wave 14** > - The compose `amini-admission` image is built with `BUILD_TAGS=runtime_aminichain` by default > via `compose.build.yml`, so the build-tag step the older flow > walked is now automatic. The Chain Inspector banner names the > actual blocker (`AMINICHAIN_ORACLE_ID` not yet set), not the > obsolete \"rebuild with -tags\" hint. > - The OpenBao path is **`amini/aminichain/oracles//wallet.key`** > (KV v2 under the `amini` mount), NOT `secret/amini/...`. > - The `amini aminichain register` CLI flow exists today (built with > the runtime tag); no need to hand-craft `cast send` calldata. > - Audit anchors start as `amini-stub-tx:...` until the operator runs > this runbook — the absence of a real Oracle ID is what keeps the > audit-anchor batcher on the stub chain client. Running this > runbook is what flips Chain Inspector from `mode:\"stub\"` to > `mode:\"live\"`. > [!tip] **This tutorial is CLI-first.** AminiChain Oracle > registration is a one-time bootstrap that requires OpenBao writes > and an on-chain RPC call. Once an Oracle is registered, the > dashboard's **`/dashboard`** page renders Oracle status pulled from > `/v1/capabilities`, and Oracle-provisioned slices show up in > **`/slices`** alongside tenant-created ones. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find my AminiChain wallet address + private key? > Generate a fresh secp256k1 keypair (`cast wallet new --json` from > Foundry prints both fields). **The private key never leaves the > operator's box — write it straight into OpenBao at > `amini/aminichain/oracles//wallet.key`.** The public address is > safe to share; it's how the chain identifies your Oracle. > [!info] Where do I find the AminiChain RPC + WebSocket URLs? > If you run the AminiChain dev node locally, the JSON-RPC is at > `http://:8466` and the EVM JSON-RPC at `:8467` over Tailscale > (chain-id `aminichain_1000000-1`). Production AminiChain endpoints > are published in the AminiChain docs; ask the chain operator for > the values that go into `AMINICHAIN_RPC_URL` and > `AMINICHAIN_WS_URL`. > [!info] Where do I find an Oracle's `oracle_id`? > Amini Cloud derives it deterministically from the wallet address, by > **hex-encoding the address text** — there is no hash involved: > > ``` > oracle_id = \"oracle_\" + hex(wallet_address_as_ascii) > ``` > > For `0x2c7536E3605D9C16a7a3D7b1898e529396a65c23` that is > `oracle_30783263...3633` — **84 hex characters**, two per character of > the address string. The source is `deriveOracleID` in > `internal/aminichain/oracle.go`. It also lands in the `id` column of > `amini.aminichain_oracles` after registration. > > This page previously said `sha256(wallet_address)` truncated to 32 > chars, and the snippet below computed exactly that. Both were wrong: > the id it produced pointed at a vault path the agent never reads, so > the wallet key was written where registration could not find it. > [!warning] A host registers its OWN wal" }, { "slug": "tutorials/07-three-host-cluster-bringup", "title": "Tutorial 07 — Three-host cluster bring-up", "description": "Full Ansible-driven 3-host bring-up — amini-cp + two compute hosts including SeaweedFS volume servers and first cross-host scheduling.", "section": "tutorials", "body": "# Tutorial 07 — Three-host cluster bring-up > [!tip] **This tutorial is CLI/Ansible-first.** Three-host bring-up > is one-time infrastructure work that lives below amini-admission: > systemd, Postgres, Headscale, Nomad. Once the cluster is up, the > **`/admin/hosts`** dashboard page renders the per-host heartbeat > rows live, and **`/admin/clusters`** lists the registered Nomad + > Kubernetes backends. Use the dashboard to **verify** the cluster > after Ansible finishes. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find each host's ID and hostname? > Open **Admin → Hosts** (`/admin/hosts`). The **Hostname** column > shows what the host registered itself as (matches `uname -n`); the > **ID** column is the UUID Amini Cloud assigns at first heartbeat. Both > values are in `amini.hosts`. CLI: `amini admin hosts ls`. > [!info] Where do I find the production control-plane domain? > Whatever you configure for Caddy on `amini-cp`. The Ansible > inventory's `slate_domain` variable is the source of truth; it > ends up in Caddyfile and as the `OIDC_REDIRECT_URI` of every > tenant's Authelia client. This tutorial walks the production bring-up: one control-plane VPS plus two compute hosts. By the end you'll have a multi-host Amini Cloud cluster that can schedule slices on either compute host, with SeaweedFS object storage distributed across both and a per-tenant Tailscale tailnet that spans all three. > **Status:** the Ansible roles for `amini-cp`, `compute-node`, and > `compute-storage` are scaffolded per > [Three-host cluster installation](../installation/three-host-cluster.md) > but not yet wired end-to-end in this beta. This tutorial documents > both the **target Ansible flow** and the **manual interim path** an > operator can use today. Pick one mode and stay consistent. ## What you'll have at the end ```mermaid flowchart TB CP[\"amini-cp (public VPS)
Caddy 2.10 SNI router
Authelia 4.39 (OIDC)
amini-admission
Headscale 0.28.0 (fleet)
Postgres 16
OpenBao 2 (vault)
Nomad 2.0.1 servers
derper-1\"] C1[\"amini-compute-1
Nomad 2.0.1 client
amini-host-agent
Podman 5 + FC
SeaweedFS vol + master + filer
local-zfs pool
GPUs (optional)\"] C2[\"amini-compute-2
Nomad 2.0.1 client
amini-host-agent
Podman 5 + FC
SeaweedFS vol
local-zfs pool
GPUs (optional)\"] DR2[\"derper-2
(free-tier cloud, different network)\"] CP --- C1 CP --- C2 CP -. DERP fallback .- DR2 ``` *End-state topology: one control plane, two compute hosts, one extra DERP relay.* A scheduling smoke test that creates one slice on each compute host will pass at the end. ## What you need ### `amini-cp` (the control plane) - Public VPS (Hetzner, Vultr, DO, OVH) with **no CGNAT**. Public IPv4 and IPv6. - Ubuntu 24.04 LTS or Debian 12. 4 vCPU, 8 GiB RAM, 80 GiB disk. - Ports 80, 443 reachable from the internet (Caddy needs Let's Encrypt — in production, control-plane Caddy fronts admission on the standard 80/443 even though the single-host Compose dev defaults are `:8089` / `:8445`; see [`deploy/compose/.env.example`](../../../deploy/compose/.env.example)). Ports 4646 (Nomad RPC), 8210 (vault — Compose default; older builds used 8200), 18080 (admission — Compose default; older builds used 8080) reachable from each compute host. Override the host-side literals via `AMINI_VAULT_PORT` / `AMINI_ADMISSION_PORT` if the defaults collide. - DNS: `*.amini.example` → the VPS public IPv4. - SSH user with sudo + key auth. ### Each compute host - Linux x86_64 or aa" }, { "slug": "tutorials/08-contribute-compute", "title": "Tutorial 08 — Contribute compute to an Amini Cloud cluster", "description": "End-to-end runner walkthrough — install Amini Cloud, generate a keypair, join a cluster, and watch the operator approve.", "section": "tutorials", "body": "# Tutorial 08 — Contribute compute to an Amini Cloud cluster > [!tip] **This tutorial is CLI-first.** Runner enrollment requires > running the `amini` CLI on the contributing host to generate a > keypair and prove ownership — that has to be local, not browser- > driven. Once your enrollment is **approved**, the operator's > dashboard at **`/admin/hosts`** will show your host with its > heartbeat status, and you can read enrollment status from > `/v1/runner/me` via the dashboard's JSON viewer. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find the cluster URL to join? > The cluster URL is whatever the operator publishes (e.g. > `https://acme-amini.example`). It's the same URL that the operator's > dashboard runs on; the operator can read it from the > `slate_domain` Ansible var or from `amini admin clusters ls`. > [!info] Where do I find my runner wallet address? > `amini runner keygen` generates a secp256k1 keypair at > `~/.amini/runner/key` and prints the public address (`0x…`). It is > also shown on the `amini runner status` output. The operator sees > it under **Admin → Hosts → Pending enrollments**. This is the runner-side walkthrough for the node enrollment flow. You — the runner — have a computer with spare capacity (an old workstation, a gaming PC, a spare GPU rig, a homelab box). An operator somewhere is running an Amini Cloud cluster that accepts contributed compute. You want to join. If you are also the cluster operator (homelab case), the operator- facing companion to this tutorial is the [Operator runbook](#operator-side-the-other-half-of-the-flow) section at the end. For background: - The concept doc: [`docs/guides/concepts/node-enrollment.md`](../concepts/node-enrollment.md) - The integration spec: [`docs/integrations/node-enrollment.md`](../../integrations/node-enrollment.md) - The implementing rationale: [ADR 0024](../../decisions/0024-node-enrollment-and-capacity-market.md) ## What you'll have at the end - A secp256k1 keypair at `~/.amini/runner/key` that proves you own this host. - An enrollment record at the cluster, either pending or approved. - (On approval) your host running the Amini Cloud host-agent and visible in `amini admin hosts ls` on the operator's side. - (When `settlement_mode='aminichain'`) an AminiChain Oracle id registered to your wallet address. Reputation accumulates as your host completes jobs. ## What you need - A computer running Linux, macOS, or Windows. The runner CLI itself is platform-agnostic; the host-agent that comes online after approval is platform-specific (see [`scripts/onboard-*.sh`](../../../scripts/)). - The cluster URL of an Amini Cloud operator who has enabled node enrollment. The operator sends this to you. - Network egress to the cluster URL on TCP 443. - (Optional) A few minutes to read the cluster's policy at `GET /v1/cluster/info` before you join. ## Steps ### 1. Install Amini Cloud > The installer is served by the cluster you are joining, at > `/install.sh`. There is no `get.amini.sh`: that hostname > has never had a DNS record, so the command these docs used to print > failed for everyone who tried it. Use the URL of the cluster you were > given — the same one you pass to `amini join`. ```bash curl -fsSL https://acme-amini.example/install.sh | sh amini version ``` The installer is the same one operators use; the runner subcommands are part of the default binary (no build tags required). ### 2. Inspect the cluster's enrollment policy Before you commit a keypair to a cluster, look at its policy: ``" }, { "slug": "tutorials/09-sign-in-with-wallet", "title": "Tutorial 09 — Sign in with wallet", "description": "End-to-end walkthrough of the Amini Cloud wallet sign-in flow. Derives an EVM keypair locally, issues a nonce, signs it, verifies, and calls /v1/tenants/me with the resulting session token.", "section": "tutorials", "body": "# Tutorial 09 — Sign in with wallet > [!tip] **You can complete this tutorial via the dashboard OR the CLI.** > The canonical dashboard tour is sketched in the box below; the CLI > path that follows reproduces the same flow with `curl` for operators > who want to inspect the wire shape. Both reach the same `Me` > response and the same persisted `tenant_users` row. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find my wallet address? > Open MetaMask (or any EIP-1193 wallet you have installed) — the > address is at the top of the popup. After signing in it also > appears in the Amini Cloud dashboard header on every page. > [!info] Where do I find a wallet keypair for the CLI walkthrough? > `cast wallet new --json` (Foundry) prints a fresh address + > private key. Keep the private key local. The CLI walkthrough > below uses a per-tutorial throwaway keypair so the private key > never lands in your real wallet. ## Dashboard tour (canonical path) The dashboard is the recommended way to sign in with a wallet — it wraps the same two HTTP calls below in a one-click flow. 1. **Open `/login`.** You'll see two sign-in options: OIDC and Wallet. Click **Wallet**. ![Step 1 — /login showing OIDC + Wallet buttons](./img/09/step-1-login.png) 2. **`/login/wallet` opens.** It detects MetaMask (or any EIP-1193 provider) in your browser. Click **Connect wallet**. ![Step 2 — /login/wallet with the Connect wallet button](./img/09/step-2-login-wallet.png) 3. **MetaMask pops up.** Approve the connection request. The dashboard shows your address and asks you to sign a one-time Amini-Cloud-issued nonce (step 3 of the CLI flow below). 4. **MetaMask pops up again** with a SIWE-style message. Click **Sign**. You never type a password. 5. **Redirected to `/dashboard`.** The header shows your wallet address and tenant slug. From here every link in the nav works: - **`/slices`** — your slices (empty for a brand-new account) - **`/slices/new`** — pick a template from the catalog and create - **`/templates`** — browse the full catalog - **`/account`** — view linked identities + sign out ![Step 3 — Dashboard immediately after wallet sign-in](./img/09/step-3-dashboard-after.png) ![Step 4 — Account page showing the linked wallet identity](./img/09/step-4-account.png) 6. **Click `/slices/new`.** Pick **Jupyter** from the template list, accept the defaults, click **Create**. The dashboard redirects you to `/slices/{id}` where you can watch the status walk `pending → scheduled → running`. That's the whole sign-in + first-slice flow without ever leaving the browser. The CLI walkthrough below produces the same effect for operators who need to script it or inspect each HTTP call. ```mermaid flowchart LR A[/login/wallet] -->|MetaMask connect| B[Nonce request] B --> C[/v1/auth/wallet/nonce/] C -->|nonce| D[MetaMask personal_sign] D -->|signature| E[/v1/auth/wallet/verify/] E -->|session cookie| F[/dashboard] F --> G[/slices/new] G --> H[/slices/{id}] ``` --- ## CLI walkthrough This walks through the wallet sign-in flow end-to-end against a local Amini Cloud stack. By the end you'll have: - A fresh EVM keypair (we use it for one tutorial run and discard it). - A Amini-Cloud-issued nonce. - A locally-computed SIWE-style signature over the nonce. - A verify response with a session token. - A `GET /v1/tenants/me` call that succeeds using the session token. The tutorial uses the EVM (EIP-4361) path because the verifier flow is shorter; the Cosmos (ADR-036) path uses the same two HTTP endpoints and differs only in the signing" }, { "slug": "tutorials/11-geo-policy", "title": "Tutorial 11 — Geofencing + multi-network placement", "description": "End-to-end walkthrough of Amini Cloud's three-layer geo policy. Sets a cluster floor, a tenant default, and a per-slice override, then verifies eligibility against a tagged host inventory.", "section": "tutorials", "body": "# Tutorial 11 — Geofencing + multi-network placement > [!tip] **This tutorial is CLI-first today.** The geo-policy editor > dashboard view is Year-1 work; today the `/admin/hosts` page > renders per-host country / zone / jurisdiction badges (read-only), > and per-slice / per-tenant / per-cluster policy management still > goes through the CLI or HTTP PUT. Where dashboard surfaces exist > they're called out inline; otherwise the CLI block is the only > path. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find my admin token + tenant token? > The admin token comes from > `amini admin tokens issue --role admin --ttl 1h` (run by an > operator). The tenant token is your normal user token from OIDC or > wallet sign-in. The dashboard surfaces active tokens under > **Account → Tokens** once that page ships; until then the CLI is > the source of truth. > [!info] Where do I find a host's country / zone / jurisdiction tags? > Open **Admin → Hosts** (`/admin/hosts`). Each host card renders > badges for **Country**, **Zone**, **Jurisdiction**, and **ISP**. > SQL fallback: > `SELECT country, zone, jurisdictions, isp FROM amini.hosts;`. ![Admin hosts page rendering geo badges per host (read-only today)](./img/11/step-1-admin-hosts-geo.png) This tutorial walks the three-layer geo policy end-to-end. By the end you will have: - A host tagged with country + zone + jurisdiction + ISP metadata. - A cluster-level operator floor (`forbidden_countries=[RU,CN,IR]`). - A tenant-level default (`required_jurisdictions=[GDPR]`). - A slice-level override (`allowed_zones=[eu-central]`). - A live eligibility preview confirming the merged policy admits the host. The tutorial uses the Amini Cloud CLI throughout; every command has the equivalent `curl`/HTTP call documented for scripting. ## What you need - A working Amini Cloud stack with migration 0020 applied. Verify: ```bash psql \"$POSTGRES_DSN\" -c \"SELECT version FROM amini.schema_migrations WHERE version=20;\" # → version: 20 ``` - `amini-admission` running with the geo routes wired (the default since 0.1.0). - An admin token in `$AMINI_ADMIN_TOKEN` for the operator steps, and a tenant token in `$AMINI_TOKEN` for the tenant steps. - At least one host in `amini.hosts` to tag. ## Step 1 — Operator tags one host ```bash amini admin hosts set-geo node-de-1 \\ --country=DE \\ --zone=eu-central \\ --jurisdiction=GDPR,SOC2 \\ --isp=hetzner-fsn1 \\ --lat=50.110924 --lon=8.682127 ``` HTTP equivalent: ```http PUT /v1/admin/hosts/node-de-1/geo Content-Type: application/json Authorization: Bearer $AMINI_ADMIN_TOKEN { \"country_code\": \"DE\", \"geo_zone\": \"eu-central\", \"jurisdiction\": [\"GDPR\", \"SOC2\"], \"network_isp\": \"hetzner-fsn1\", \"lat\": 50.110924, \"lon\": 8.682127 } ``` Verify the row: ```bash psql \"$POSTGRES_DSN\" -c \" SELECT id, country_code, geo_zone, jurisdiction, network_isp FROM amini.hosts WHERE id='node-de-1';\" ``` **Dashboard equivalent (read):** open **`/admin/hosts`** — the row for `node-de-1` now shows the country flag emoji, the zone, the jurisdiction badges, and (if you hover) lat/lon. The dashboard does not yet have a host-geo editor; the PUT call above is the only way to set the values for now. ## Step 2 — Operator sets a cluster floor ```bash amini admin cluster geo-policy set --forbidden-countries=RU,CN,IR amini admin cluster geo-policy show ``` The floor is UNION-merged with tenant + slice policies. A tenant who later sets `allowed_countries=[\"RU\"]` will be rejected at policy-set time — the floor is unforgeable. ## Step 3 — Tenant sets a default policy Switching to th" }, { "slug": "tutorials/12-run-a-command-in-a-slice", "title": "Tutorial 12 — Run a command in a slice", "description": "Use the one-shot exec surface to retrieve a Jupyter token, peek at /etc/hostname, and verify the audit trail. Five steps, no SSH client required.", "section": "tutorials", "body": "# Tutorial 12 — Run a command in a slice > [!tip] **You can complete this tutorial via the dashboard OR the > CLI; both paths are shown side-by-side below.** The slice detail > page's **Shell** tab wraps the same `POST /v1/slices/{id}/exec` > the curl examples drive. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* > [!info] Where do I find the admission URL? > The Compose default is `http://localhost:18080` (Compose > intentionally avoids `:8080` — see > [`deploy/compose/.env.example`](../../../deploy/compose/.env.example) > for `AMINI_ADMISSION_PORT`). A production install exposes it as > `https://api.` via Caddy. Set `$AMINI_ADMISSION_URL` to > override; legacy environments may still bind admission to `:8080`. > [!info] Where do I find my tenant token? > Sign in via the dashboard at `/login` (OIDC or wallet) — the > dashboard automatically attaches the session cookie to every call. > For curl, mint an API key from a signed-in session — see > [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api). > On a local Compose stack **you own**, `Authorization: AminiDev > ` also works, but only once `AMINI_ENABLE_DEV_BEARER=true` > is set on admission (off by default even locally — see > [Tutorial 00, Step 3](00-your-first-compute-job.md#step-3--set-up-dev-auth)). The Amini Cloud dashboard's Shell tab gives you a quick one-shot exec surface for any slice you own. It is **not** an interactive terminal — for that, use `amini slices ssh ` from the CLI. But for the 80% case (\"show me my Jupyter token\", \"cat this file\", \"tail the log\") the dashboard surface is the fastest path. By the end of this tutorial you will have: - Created a one-off `nginx` slice. - Verified it reached `running` status. - Run `cat /etc/hostname` inside it via the dashboard's `POST /v1/slices/{id}/exec`. - Asked `jupyter lab list` against a `jupyter` slice (skipping if none is handy). - Confirmed the exec showed up in the audit log as `slice.exec`. - Torn the slice down. ## What you need - A working Amini Cloud stack with admission on `:18080` (Compose default; legacy environments may still use `:8080` — override via `$AMINI_ADMISSION_URL`). Verify: ```bash curl -fsS \"${AMINI_ADMISSION_URL:-http://localhost:18080}/healthz\" # → {\"status\":\"ok\"} ``` - A tenant token: an API key (works everywhere — see [Getting a token to call the API](../../operators/two-ways-to-run.md#getting-a-token-to-call-the-api)) or, on a local Compose stack **you own** with `AMINI_ENABLE_DEV_BEARER=true` set, the `Authorization: AminiDev ` shim. - `curl` and `jq`. Export your bits: ```bash export ADMISSION=\"${AMINI_ADMISSION_URL:-http://localhost:18080}\" export TOKEN=\"${AMINI_TOKEN:-Bearer $AMINI_API_KEY}\" # or \"AminiDev wallet-0xYOURADDR\" locally ``` ## Step 1 — Create a slice The lightest possible slice is a static nginx; it has `cat`, `/etc/hostname`, and not much else. Perfect for an exec demo. ```bash SLICE_JSON=$(curl -fsS -X POST \"${ADMISSION}/v1/slices\" \\ -H \"Authorization: ${TOKEN}\" \\ -H 'content-type: application/json' \\ -d '{ \"name\":\"exec-demo\", \"runtime\":\"podman\", \"image\":\"docker.io/library/nginx:1.27-alpine\", \"resources\":{\"cpu_milli\":250,\"mem_bytes\":134217728,\"disk_bytes\":268435456}, \"lease_seconds\":1800 }') SLICE_ID=$(echo \"${SLICE_JSON}\" | jq -r .id) echo \"slice id: ${SLICE_ID}\" ``` **Dashboard equivalent:** open `/slices/new` in the web UI, pick the nginx template (or fill the form by hand), submit. ![Step 1 — New slice form with the nginx template](./img/12/step-1-slice-new-ng" }, { "slug": "tutorials/13-build-your-own-cloud", "title": "Tutorial 13 — Build your own cloud", "description": "Form a 3-host Amini Cloud cluster from this control-plane plus a Mac and a Windows workstation, deploy nginx, hit it from any device's browser, SSH into one replica, and watch failover when a host drops.", "section": "tutorials", "body": "# Tutorial 13 — Build your own cloud > [!tip] **You can complete this tutorial via the dashboard OR the > CLI; both paths are shown side-by-side below.** The dashboard's > **`/admin/clusters`**, **`/admin/hosts`**, and **`/slices`** pages > wrap the same APIs the CLI snippets drive. > [!warning] **Some sections marked Year-1.** Amini Cloud today implements > the single-host dispatch path end-to-end. Replicated slices with > cross-host load-balancing exist in the schema (see migration 0013) > but the runtime wiring is stubbed. Where this matters the step > shows a working stand-in (three single-host slices on three hosts) > and notes the Year-1 target. ## Watch the recording

*Don't have the recording locally? Follow the written steps below — they match the video frame-for-frame.* ## Where you'll end up A 3-host Amini Cloud cluster: one control plane (this amini-cp box) plus a Mac and a Windows workstation as compute. Three nginx slices addressed by MagicDNS hostnames. SSH into any replica. Drop one host; the others keep serving. ```text +-------------------------+ | amini-cp (this DGX) | admission, postgres, headscale +-----------+-------------+ | +----------+----------+ | | +--+--------+ +-----+--------+ | mac-pro | | win-work | amini-host-agent + podman +-----------+ +--------------+ ``` > [!info] Where do I find my admission URL? > Compose default is `http://localhost:18080` (Compose intentionally > avoids `:8080` — see > [`deploy/compose/.env.example`](../../../deploy/compose/.env.example) > for the `AMINI_ADMISSION_PORT` override). Production puts admission > behind Caddy at `https://api.`. This tutorial uses > `https://amini.example.com` — substitute your own. Set > `$AMINI_ADMISSION_URL` to override the CLI default; if your stack > still binds the legacy `:8080` port, point that env var there. > [!info] Where do I find a host_id? > Free-form `[a-z0-9-]`, max 63 chars. The install scripts default to > the machine's short hostname. Picking something human-readable > (`mac-pro`, `win-workstation`) makes **`/admin/hosts`** easier to scan. > [!info] Where do I find the cluster URL? > After `amini clusters create`, the `id` shows up on > **`/admin/clusters`** and in the CLI output. Export it as > `$CLUSTER_ID`. CLI: `amini clusters ls`. ## Prerequisites - This amini-cp host running and reachable from both other devices (open internet via Caddy/DNS, or same LAN). - A Mac (Intel or Apple Silicon) and a Windows 10/11 workstation, both with admin rights. - The Amini Cloud repo cloned, and `make host-agent-binaries` already run on amini-cp so `dist/` contains the five cross-platform binaries. A Tailscale account is **not** required for the smoke test — Amini Cloud's dev path uses direct host-agent dial. Production adds Tailscale (or another WireGuard overlay) for MagicDNS and NAT traversal, see [Tutorial 07](07-three-host-cluster-bringup.md). ## Step 1 — Confirm amini-cp is up From the amini-cp host: ```bash curl -fsS \"${AMINI_ADMISSION_URL:-http://localhost:18080}/healthz\" curl -fsS \"${AMINI_ADMISSION_URL:-http://localhost:18080}/readyz\" | jq . ``` `readyz` surfaces any subsystem (postgres, headscale, vault) that isn't green; fix it before proceeding or the host-agent registrations will fail. **Dashboard equivalent:** open **`/admin/status`** and confirm postgres, headscale, and admission are all green. ![Step 1 — admin status all green](./img/13/step-1-admin-status.png) ## Step 2 — Install the host-agent on the Mac Copy `dist/` and `scripts/install-host-agent.sh` from amini-cp to the Mac (`scp`, `rsync`, share folder — your pick). Then from a Mac Terminal: ```bash sudo bash " }, { "slug": "tutorials/14-clustered-compute", "title": "Tutorial 14 — Clustered compute, hands-on", "description": "Connect a second host, form a cluster, submit a slice that lands on that cluster, then run a replicated slice across every host. The short, opinionated version of Tutorial 13.", "section": "tutorials", "body": "# Tutorial 14 — Clustered compute, hands-on The working version of [Tutorial 13](13-build-your-own-cloud.md). Same end state — multi-host cluster, slices landing across it — but written as a 20-minute punch list, not a 45-minute walk. Copy-paste-runnable end-to-end. > [!info] **Prereq.** `amini-admission` is up (`make up` from this > repo) and **one** host already shows up under `/admin/hosts`. The > control-plane host you ran `make up` on counts; on the operator's > rig that's `spark-5804`. If `/admin/hosts` is empty, run > [Tutorial 13](13-build-your-own-cloud.md) Step 1 first. > [!info] **Env vars used below.** > `$AMINI_ADMISSION_URL` (default `http://localhost:18080`), > `$TOKEN` (an admin tenant token — `make demo-tenant` prints one), > `$CLUSTER_ID` (filled in Part 2). ## Part 1 — Add a second host amini-host-agent is one cross-built static binary; the installer scripts under `scripts/install-host-agent.{sh,ps1}` pick the right artefact for the OS + arch and register the agent with admission in one shot. On the second machine (laptop, workstation, another Spark, whatever): ```bash # From the amini checkout on the new host. Replace --admission-addr # with whatever $AMINI_ADMISSION_URL resolves to from THIS host. sudo bash scripts/install-host-agent.sh \\ --admission-addr \"$AMINI_ADMISSION_URL\" \\ --host-id \"$(hostname -s)\" \\ --region home-office ``` ```powershell # Windows equivalent — run PowerShell as Administrator. .\\scripts\\install-host-agent.ps1 ` -AdmissionAddr \"$env:AMINI_ADMISSION_URL\" ` -HostId \"$env:COMPUTERNAME\" ` -Region \"home-office\" ``` The installer drops the binary at `/usr/local/bin/amini-host-agent` (or `C:\\Program Files\\amini\\`), writes the host token, installs a service, and starts it. The agent self-POSTs to `/v1/hosts//register-endpoint` so its `host_agent_addr` lands in `amini.hosts`. **Verify.** On the control-plane box: ```bash curl -fsS \"$AMINI_ADMISSION_URL/v1/admin/hosts\" \\ -H \"Authorization: $TOKEN\" | jq '.[] | {id, status, host_agent_addr, last_heartbeat}' ``` You want the new host's row with `status: \"online\"` and a non-empty `host_agent_addr`. The dashboard equivalent is **`/admin/hosts`**. **Troubleshooting.** Three failure modes cover ~90% of the support queue: | Symptom | Cause | Fix | |---|---|---| | Agent up, host stuck `pending` | Firewall on the host blocks inbound from amini-cp on `:7443` | open the port (or set `--listen :0` on the agent to use an ephemeral port + let admission dial back) | | `register-endpoint` returns 401 | Stale `HOST_AGENT_TOKEN` after token rotation | Re-run the installer; it minted a fresh token | | Host row never appears | Wrong `--admission-addr` (loopback on the new host instead of the cp tailnet IP) | re-export `AMINI_ADMISSION_URL` to the cp's reachable URL and re-run | ## Part 2 — Form a cluster A *cluster* is the dispatch-target abstraction. Hosts get attached to a cluster; slices target a cluster (not a host) when you want the scheduler to pick the host for you. ```bash # Create the cluster row. kind=nomad is the default scheduler today. NEW_CLUSTER=$(curl -fsS -X POST \"$AMINI_ADMISSION_URL/v1/clusters\" \\ -H \"Authorization: $TOKEN\" -H 'content-type: application/json' \\ -d '{\"name\":\"home-cluster\",\"kind\":\"nomad\"}') export CLUSTER_ID=$(echo \"$NEW_CLUSTER\" | jq -r .id) echo \"CLUSTER_ID=$CLUSTER_ID\" ``` Attach the hosts. Today this is a direct UPDATE of `hosts.cluster_id` (see migration `0023_hosts_cluster_id.sql`); the admin CLI wrapper lands in the next sprint: ```bash # CLI path (when wired): # amini hosts attach --cluster home-cluster # # Direct SQL path that's known-good today: psql \"$(cat deploy/compose/secrets/pg-dsn)\" < [!info] **Where does the CLI form go?** The `amini clusters create` / > `amini hosts attach` commands are scaffolded in > `cmd/amini/cmd/admin_cluster.go` but currently `notYet`. The HTTP + > SQL path above is" }, { "slug": "tutorials/15-add-mac-as-compute-host", "title": "Tutorial 14 — Add a Mac (or Linux box) as a compute host", "description": "Join a Mac via OrbStack VM or a native Linux host to an existing Amini Cloud cluster with a single make target.", "section": "tutorials", "body": "# Tutorial 14 — Add a Mac (or Linux box) as a compute host > [!tip] **One command, two operating systems.** `make host-join` > auto-detects whether you're on macOS or Linux and runs the matching > path: on macOS it provisions an OrbStack Ubuntu VM and runs the > agent inside; on Linux it runs the agent natively. Both paths end > with the same cluster-admin approval step from the existing > [Tutorial 08](08-contribute-compute.md) flow. ## Goal By the end you'll have your Mac contributing CPU + RAM to an Amini Cloud cluster as a compute host (`compute_class=mac`), reachable over your tailnet from the control plane on another machine. The same target, `make host-join`, works unchanged on a native Linux box — it just skips the VM-provisioning phases. ## Why a VM on macOS? > [!info] **macOS doesn't have what `amini-host-agent` needs.** The > agent depends on cgroup v2, nftables, the WireGuard kernel module, > and podman — Linux-only primitives. We provision a small Ubuntu VM > via OrbStack (~5 minutes from zero, ~1 GiB disk), join it to your > tailnet, and run the agent inside. The Mac itself stays clean; if > you delete the VM later, nothing leaks onto the host. Native Linux hosts skip the VM and run the agent directly. ## Prerequisites - macOS 13+ (Apple Silicon or Intel) **or** Linux x86_64 / aarch64 with kernel 6.1+ and cgroups v2. - On macOS: OrbStack installed. ```bash brew install --cask orbstack open -a OrbStack # start the daemon once before running host-join ``` - Go 1.22+ (matches the repo's `go.mod` minimum — the macOS path cross-compiles the host-agent for `linux/$ARCH` from your Mac). - A Tailscale account and an auth key. Mint one at [https://login.tailscale.com/admin/settings/keys](https://login.tailscale.com/admin/settings/keys). Recommended settings: - **Reusable:** off (one key per host avoids accidental re-use) - **Ephemeral:** on (the device entry vanishes if the VM goes away) - **Tags:** `tag:amini-compute` (matches the operator's ACL) - An Amini Cloud cluster you have admin rights on. You need: - The cluster URL (e.g. `https://spark-5804.tail4e3a73.ts.net:18443`) - An admin wallet address on that cluster (the same wallet you'd use with `amini admin nodes pending`) ## One-command path From a checkout of the Amini Cloud repo on the host you're adding: ```bash make host-join CLUSTER=https://spark-5804.tail4e3a73.ts.net:18443 \\ HOST_ID=mac-orbstack \\ TAILSCALE_AUTHKEY=tskey-auth-... ``` The optional knobs: - `HOST_ID=` — friendly name the cluster will show. Defaults to the hostname. - `COMPUTE_CLASS=` — one of `mac`, `server`, `gpu-node`, `rpi`, `workstation`. Defaults to `mac` on darwin, `server` on Linux. - `TAILSCALE_AUTHKEY=` — pre-minted key. Omit to be prompted to run `tailscale up` interactively inside the VM. What the script walks through, in order (the same labels A–I print on its own stdout so you can match the phase to the line): - **A.** Detect OS (`darwin` → macOS path; `linux` → native path). - **B.** Verify prerequisites (OrbStack daemon, Go toolchain, network reachability to `$CLUSTER`). - **C.** *(macOS only)* Create an Ubuntu OrbStack VM if one named `amini-host-` doesn't already exist. - **D.** *(macOS only)* Install Tailscale inside the VM and bring it up (uses `TAILSCALE_AUTHKEY` if set; otherwise prompts). - **E.** *(macOS only)* Cross-compile `amini-host-agent` for `linux/$ARCH` from the Mac and copy it into the VM. - **F.** Run the kernel preflight (cgroups v2, nftables, wireguard kmod, podman). Fail fast with a clear error if missing. - **G.** Run `amini join $CLUSTER` to submit the enrollment record. - **H.** **Pause** and wait for the cluster admin to approve (see next section). The script prints a watch loop polling `amini runner status`. - **I.** Start `amini-host-agent` under systemd (preferred) or under a foreground supervisor if systemd is unavailable. If you've never read [Tutorial 08](08-contribute-compute.md), it covers the underlying `amini join` flow in detail. This t" }, { "slug": "tutorials/README", "title": "Tutorials index", "description": "End-to-end runnable scenarios for amini (the single-operator private cloud) and the aminicloud federation it joins when chain integration is wired.", "section": "tutorials", "body": "# Tutorials Each tutorial is end-to-end runnable: copy-paste-able commands, expected outputs called out, troubleshooting at each step. The tutorials assume you already have an Amini Cloud stack running. If you don't, start with [Single-host installation](../installation/single-host.md). > **Two layers in the architecture.** A **amini** is the single > private cloud one operator runs on hardware they own (control plane > + hosts + tenants + data plane — what tutorials 00–05 + 07–15 > exercise). **aminicloud** is the *federation of slates* — > independent amini clusters discovering each other and trading idle > compute via the **aminichain** Cosmos-SDK + Tendermint chain. > A single instance is a self-contained sovereign substrate; joining > **aminichain** is what makes it an aminicloud node. On the sovereign > reference deployment the oracle self-provisions and the chain is on by > default; Tutorial 06 walks the manual bootstrap for a legacy single-host > build, and the canonical runbook is at > [`docs/operators/aminichain-oracle-bootstrap.md`](../../operators/aminichain-oracle-bootstrap.md) > and the architecture story at > [`docs/integrations/aminicloud-architecture.md`](../../integrations/aminicloud-architecture.md). ## Which tutorial do you need? | You are … | You want to … | Start here | |---|---|---| | A new tenant | Spin up your first Jupyter slice and reach it from a browser | [Tutorial 01 — Your first slice](01-your-first-slice.md) | | A new tenant | Install a packaged multi-slice app from the catalog | [Installing apps](../tenant/installing-apps.md) | | A new tenant | Launch a curated LLM as a slice from the model catalog | [Using the model catalog](../tenant/using-the-model-catalog.md) | | A team admin | Share a slice with a teammate | [Tutorial 02 — Share a slice with a teammate](02-share-a-slice-with-a-teammate.md) | | An ML researcher | Run PyTorch training on a GPU slice with a tensor dataset | [Tutorial 03 — GPU ML training on Jupyter](03-gpu-ml-training-on-jupyter.md) | | A backend developer | Deploy Postgres on a block volume and connect from a sibling slice | [Tutorial 04 — Deploy Postgres with a block volume](04-deploy-postgres-with-block-volume.md) | | The platform operator | Onboard a brand-new tenant | [Tutorial 05 — Onboard a tenant](05-onboard-a-tenant.md) | | The platform operator | Register amini as an AminiChain Oracle (flips audit anchors stub→live; turns amini into an aminicloud node) | [Tutorial 06 — Register as an AminiChain Oracle](06-register-as-an-aminichain-oracle.md) — narrative; the [oracle bootstrap runbook](../../operators/aminichain-oracle-bootstrap.md) is the source-of-truth flow | | The platform operator | Bring up a real 3-host cluster | [Tutorial 07 — Three-host cluster bring-up](07-three-host-cluster-bringup.md) | | The platform operator | Build a multi-machine cloud (3-host capstone) | [Tutorial 13 — Build your own cloud](13-build-your-own-cloud.md) | | The platform operator | Connect a second host + submit clustered jobs | [Tutorial 14 — Clustered compute, hands-on](14-clustered-compute.md) | | A Mac operator | Add your Mac as a compute provider via an OrbStack Linux VM | [Tutorial 15 — Add a Mac as a compute host](15-add-mac-as-compute-host.md) | ## Conventions used in every tutorial - **Imperative voice.** \"Run X\", not \"you can run X\". - **Real commands.** Every fenced shell block is intended to run as-is. Placeholders are explicitly marked with a `# placeholder:` comment; substitute before running. - **Expected output.** Where the output is load-bearing it is shown in a separate fenced block immediately after. - **Troubleshooting.** When a command can fail in a known way, the expected error and the fix are inline. - **Linux bash assumed.** macOS-specific paths get a separate fenced block with a comment header. ## When a tutorial breaks If a command in a tutorial returns a different shape than documented, that is a documentation bug per the [accuracy contract](../README.md#accur" }, { "slug": "tutorials/SCREENSHOT_TASKS", "title": "Tutorial screenshot tasks", "description": "", "section": "tutorials", "body": "# Tutorial screenshot tasks This file is the source of truth for tutorial screenshots. Each row is one PNG that should live at the listed path. Capture them either by running `scripts/capture-tutorial-screenshots.sh` (Playwright + headless Chromium) or by recording manually and saving to the same path. The conventions: - **Resolution:** 1280x800 (matches Playwright `chromium-desktop` project). - **Format:** PNG, `fullPage: false` unless the row says otherwise. - **Auth context:** wallet sign-in as `wallet-0x9937584dc2404aa06ffc17174c6757aa4fa43419`, tenant `demo`. - **Path style:** relative to `docs/guides/tutorials/`. The Playwright spec writes to `../docs/guides/tutorials/img//...` from `web/`. - **Idempotency:** re-running the capture script overwrites in place. If a tutorial has no clean dashboard equivalent for a step (e.g. Tutorial 06 step 3 rebuilds the binary), the row is intentionally omitted and the written tutorial keeps a CLI-only flow there. ## Posters + recordings (one per tutorial) Every tutorial wants a poster PNG (used as the `

served from a amini slice

served from a amini slice