How to add a project (user) post deployment?

[lionello]

TL;DR — Add a tool and a UI that create a project (= linux user) after deployment. Today this needs a Nix edit plus nixos-rebuild switch. Build it on the native backend (#325), where agentbox apply already does the work. Gate it behind the deployment password, which is the admin credential.

This body is the current state of the discussion. The comments below hold the working detail.

The ask

During deployment we create a linux user with a password for HTTP basic auth. The user is the isolated env. One user can run many agents (see #321) which some customized settings, but for real isolation we'd need a new user.

Suggest to add a tool+UI to create a new user. This could be on the root (see #221) with a new route for scope access to allow adding linux user = project. If so, the root UI needs to change to remove the redirect and to explain user = project (without using too much slang).

Settled

1. The deployment password is the admin credential. It authenticates the box owner, never a project. So "add project" sits behind it, not behind a project user's basic auth. "Root" in the earlier comments means the root route, not uid 0. The scope is the admin scope: web login admin, its own hash and its own cookie, and admin is a reserved project name. No uid-0 daemon anywhere.
2. Native-first. No nixos-rebuild round-trip, because Portability: native Ubuntu/RHEL support (Nix-without-NixOS) — refactor tracker #154 Phase 4 removes that path. NixOS boxes get the same button at Phase 5, when the option tree renders the same spec.
3. The mechanism already exists. cmd_apply in bin/agentbox (refactor(native): Ubuntu/RHEL backend — no nixos-infect (#154 Phase 4) #325) creates the login, renders the per-user env, drop-ins, sudoers and Caddyfile, enables that user's four template instances, and disables the instances of a user that left the config. So "add a project" is one validated users: entry in /etc/agent-box/config.yaml, then agentbox apply. No second spec file.
4. Privilege, with no uid 0. Whoever can write config.yaml is root-equivalent, because apply renders sudoers.d fragments and unit drop-ins from it. So the admin daemon never writes that file and never runs as root. Instead a narrow wrapper — agent-box-project add|set-password|rm NAME — behind one sudoers line, the same shape as agent-box-session and agent-box-webhook.
5. / stops being served by a project's settings daemon. Today it proxies to rootUser's daemon with AGENT_BOX_HOME=1 (modules/agent-box.nix.in:2374). That daemon runs as an unprivileged project user, and that project's agent can reach its socket. A separate admin daemon owns /; the per-user daemons keep zero privileged actions.
6. The single-project box does not change. Admin may enter any project, so a sign-in at / mints the chosen project's cookie and lands in the workspace. No picker. The picker appears with two or more projects. So the redirect stays, but it means "admin opens the only project".
7. Each project gets its own password, set by admin at creation. A per-project password matters when you hand one project to a different person. Native first_boot currently hashes the same deployment password into every /etc/agent-box/<user>.hash (bin/agentbox:927); that loop must write the admin hash only.
8. Avoid sending WebPassword plaintext to CloudFormation/user-data #25 matters more now. The deployment password is the highest-value credential on the box, and it travels as plaintext through CloudFormation and user-data. So the admin page must also rotate it.

Blocker

#325 must gain an auth-secrets oneshot first. /run/agent-box-web/env holds WEB_PASSWORD_HASH_* and WEB_COOKIE_SECRET_*. Only apply --first-boot writes it (bin/agentbox:938), and caddy.service requires it with no - prefix (bin/agentbox:285). /run is tmpfs, and cloud-init runs first-boot once. So a native box loses its whole web front door on the first reboot, and a project added later has no hash and no cookie secret in that file. Fix: port webAuthSecretsService to the native renderer as a oneshot, RequiredBy=caddy.service. Reported on #325.

Plan

1. refactor(native): Ubuntu/RHEL backend — no nixos-infect (#154 Phase 4) #325 merges.
2. Auth-secrets oneshot in the native renderer.
3. The agent-box-project wrapper, plus its one sudoers line.
4. The admin route: its own hash and cookie, the project picker, "add project", and password rotation.
5. NixOS parity at Portability: native Ubuntu/RHEL support (Nix-without-NixOS) — refactor tracker #154 Phase 5.

Related: #321 (profiles: many agents in one project), #221 (the /user/session path hierarchy), #127 (users are the trust boundary, sessions are the unit of work), #154 / #295 / #325 (the native refactor), #25 (plaintext deployment password), #313 (the other privileged affordance on the same page).

[defangdevs]

Current state

Today, a new linux user (= project) can only be created by editing the host's Nix config and running sudo nixos-rebuild switch — add an entry under services.agent-box.users.<name> = { ... } (README "Quick start, multiple users"). The module turns that into a real users.users.<name> account, its own hardened agent-box-<name>.service, and a per-user Caddy basic_auth block. There's no in-band tool for this — agent-box-session add/rm (#321) only manages sessions within an already-existing user, which is a different isolation unit.

The "root" page from #221 (served by settings-daemon.py's render_users()) is a read-only picker: with >1 user it lists links into each user's space; with exactly 1 user it currently redirects straight into that user's TERM_HOME, skipping the picker entirely. It has zero create/manage-user affordance today — the docstring is explicit it "grants no access, and it says which user this browser is already authenticated as."

What adding this would take

Creating a linux user at runtime (not at rebuild) means a privileged helper — something that can call useradd, set a password hash, and register the new user with Caddy's basic-auth config and the session-supervisor unit, all without a full nixos-rebuild. That's a bigger surface than the current locked-down sudo allowlist (essentially caddy-reload + self-update, no general root) — see AGENTS.md's "sudo is a tight allowlist, not general root."

Before I build anything, want to confirm the shape:

• A new root-owned setuid-ish daemon/socket that the settings-daemon calls into (add-user, set-password), audited and restricted to that one action?
• Should "add user" be gated behind an existing user's basic-auth session (so any of today's users could add a sibling), or reserved for a separate root credential?
• Drop the single-user auto-redirect unconditionally (always show the picker + an "add project" affordance), or only once >1 user exists?

Happy to prototype the daemon + UI change once the access-control shape is settled — this is a new privileged-action surface so I'd rather get that right before writing code.

[defangdevs]

Decision recorded: the deployment password is the root password

The WebPassword CFN parameter authenticates the box owner. It does not authenticate a project. So "add project" sits behind the root credential, not behind any project user's session. That answers question 2.

Auth mechanisms in general stay out of scope here, as you say.

Today the two are the same thing by accident

The CFN parameter becomes /var/lib/agent-box-web/password-hash. The template assigns that same file to the single user agent as its web.passwordHashFile (aws/template.yaml:645, :689). The vhost root block then authenticates with that user's login name, hash and cookie (modules/src/caddyfile-root.caddy: basic_auth ... @USER@). rootUser is web.user, else the first user (modules/agent-box.nix.in:1967).

The root route is therefore a project user's route today. Two things follow.

1. The root scope needs its own identity. Login name root, which is not a linux account. Its hash comes from the deploy password file. It gets its own cookie, __Host-agent_box_auth_root. Each project then gets its own hash file, which root sets when it creates the project. The module must also refuse a project named root, the same way it already refuses reserved session names.

2. The root scope must stop being served by a project's settings daemon. Today / proxies to rootUser's settings daemon with AGENT_BOX_HOME=1 (agent-box.nix.in:2374). That daemon runs as an unprivileged project user, and that project's own agent can reach its socket. If that daemon could create linux users, every agent in the first project would hold root on the box. So / moves to a new root-owned daemon, agent-box-root.service, with a narrow action set: add project, set project password, rotate root password. The per-user daemons keep zero privileged actions.

That answers question 1: one root daemon, no setuid helper.

Question 3 answers itself

Root may enter any project, because it is root. So a root sign-in mints the chosen project's cookie. All cookie secrets already sit in one caddy env file, written by agent-web-auth-secrets, so the root daemon can do this.

A single-project box then behaves exactly as today. You give one password at /, and you land in the workspace. You see no picker. With two or more projects, / shows the picker. So the redirect stays, but it changes meaning: root opens the only project, instead of no picker existing. A per-project password matters only when you hand one project to a different person.

The one hard problem: a runtime project must survive a rebuild

useradd at runtime does persist, because users.mutableUsers defaults to true. The rest does not. The per-user unit and the per-user Caddy auth block are declarative (agent-box.nix.in:1638 and the Caddyfile generator), and /etc/systemd/system is store-managed on NixOS. A self-update runs nixos-rebuild switch. A project added purely at runtime would therefore lose its unit and its auth block at the next update, and its home would stay behind. That is the worst kind of bug, so the design must round-trip through a persistent spec file.

Two shapes. Both use the same spec file, /etc/agent-box/projects.json:

A. Declarative round-trip. The root daemon writes projects.json, then runs nixos-rebuild switch. The module reads that file at eval and folds it into services.agent-box.users. There is one source of truth. It survives self-update, and it needs no new unit-management code. It needs nixos-rebuild switch in the sudo allowlist. The box already allows agent-box-update.service, which does the same thing, so this adds no new privilege class. Cost: 30 to 60 seconds per project.

B. Runtime reconcile. The root daemon writes the same file, then calls agentbox apply from #325. That does the useradd, renders the units and the Caddyfile, and reloads. It takes seconds. It is the Phase 4/5 direction, where both backends render one spec. It needs #295 and #325 merged first, and it needs a config layer that NixOS will not clobber.

Recommendation: build A now, and converge on B at Phase 5. Both consume the same projects.json, so A is not throwaway work.

One consequence for #25

The deploy password is now the highest-value credential on the box. That raises the stakes of #25, where the plaintext parameter travels through CloudFormation and user-data. So the root page must also rotate the root password. The deploy-time secret must not be permanent.

[defangdevs]

Revision: "root" is a scope, and the target backend is native

Two corrections from Lio, and they change the plan.

1. "Root" means the root route, not uid 0. So there is no uid-0 daemon, and no linux account named root. I will call the scope the admin scope: the credential the deployment password sets, which owns / and may create projects. The web login name becomes admin, and the page says "Projects on this box". Reserve admin as a project name.

2. The move off NixOS is in flight (#325), so the plan is native-first. My plan A (write a spec file, then nixos-rebuild switch) built a NixOS-only privileged path that Phase 4 deletes. Dropping it.

The mechanism already exists in #325

cmd_apply in bin/agentbox is already the exact reconciler #352 asks for. It creates the login (ensure_user, useradd --create-home), renders the per-user env, drop-ins, tmpfiles, sudoers and Caddyfile, runs systemctl enable --now for that user's four template instances, and disables the instances of a user that left the config. So on the native backend, "add a project post deployment" is already:

1. add one entry under users: in /etc/agent-box/config.yaml
2. run agentbox apply

No new privileged mechanism is needed. I withdraw the projects.json file I proposed — config.yaml is the spec, and a second one would only drift.

The privilege model, with no uid 0

One caution, because it decides the shape. Whoever can write config.yaml is root-equivalent: apply renders sudoers.d fragments and unit drop-ins from that file. So the admin daemon must not hold write access to config.yaml, and must not run as root.

Shape instead:

• A wrapper, agent-box-project add NAME (and set-password, rm). It validates the name, mints an argon2id hash with caddy hash-password, inserts one users: entry, and calls apply. It never accepts arbitrary config.
• The admin daemon runs as its own unprivileged system account. It gets one sudoers line, for that wrapper only.
• The audited surface is therefore one small script. This mirrors agent-box-session and agent-box-webhook, which is the shape the box already uses.

One gap in #325 blocks this (and is a reboot bug on its own)

/run/agent-box-web/env holds WEB_PASSWORD_HASH_* and WEB_COOKIE_SECRET_*. On the native backend, only apply --first-boot writes it (bin/agentbox:938), plus the password helper on a password change. caddy.service reads it with EnvironmentFile=/run/agent-box-web/env — no - prefix, so systemd fails the unit when the file is missing (bin/agentbox:285).

/run is tmpfs, and the native template runs apply --first-boot once, from cloud-init runcmd (aws/lightsail-native-template.yaml:256). Two consequences:

• A native box loses its whole web front door on the first reboot. On the NixOS side agent-web-auth-secrets.service is requiredBy = caddy.service and rebuilds this file every boot. The native renderer has no equivalent. Verified on current HEAD eef3ee4.
• A project added later has no hash and no cookie secret in that file, so its rendered Caddyfile block references env vars that do not exist.

One fix serves both: port webAuthSecretsService to the native renderer, as a oneshot RequiredBy=caddy.service that rebuilds the env file from the inputs that already persist — /etc/agent-box/<user>.hash and /var/lib/agent-box-web/cookie-secret-<user>. I have posted this on #325 rather than pushing, because a session is working that branch right now.

Per-project passwords

first_boot currently hashes the same deployment password into every /etc/agent-box/<user>.hash (bin/agentbox:927). Under the new rule, the deployment password writes only the admin scope's hash. Each project gets its own hash when the admin creates it. The admin may still enter any project, because a sign-in at / can mint that project's cookie.

Sequence

1. refactor(native): Ubuntu/RHEL backend — no nixos-infect (#154 Phase 4) #325 merges.
2. Auth-secrets oneshot in the native renderer (fixes the reboot bug, and is a precondition here).
3. agent-box-project wrapper plus its one sudoers line.
4. Admin route: its own hash, its own cookie, the picker, and "add project".

NixOS boxes — including this one — get the same button at Phase 5, when the option tree renders the same spec. I will not build a NixOS-only daemon for it.

[defangdevs]

Consolidated the thread into the issue body: the ask (quoted verbatim), the eight settled points, the one blocker, and the plan. The three comments above stay as the working record — the body is now the thing to read first.