Introduction

Leia esta documentação em português

zombie-crab-project gives every user their own real, isolated AI agent, behind a single authenticated front door. This book explains how to run it, how to configure it, and how it is put together.

The problem it solves

A self-hosted AI assistant is usually built around one idea: one agent, one owner. That is fine on your own laptop. It stops being fine the moment a second person is involved.

An AI agent reads and writes files, runs tools, executes code and keeps long-lived memory — all of it steered by natural language nobody has verified. In a single shared process, one prompt injection, one path-traversal bug or one leaky tool is enough for one user to read another user’s conversations, files and secrets. Separating users by a key in a map looks like isolation. It is not one.

This project answers that with a boundary the kernel enforces rather than the application. Every user gets their own container and their own volume, started when they first speak and stopped when they go quiet. If one user’s agent is completely compromised, it still cannot reach another user’s data: different container, different volume, non-root, no shared surface.

In front of that sits one authenticated entrance. An API gateway verifies the caller and injects an account profile the caller cannot forge, so identity flows down from something trusted instead of up from a request body. Users are keyed on a stable account id rather than an e-mail address — change your e-mail and your agent, along with everything it remembers, stays yours.

Who this is for

If you want to run it, the quick start is a clone, a .env and one compose command away from a working chat. You need a terminal and Docker; you do not need to know Go, or how any of this is built.

If you are administering a deployment — inviting members, deciding which model people get, publishing shared skills and documents — the administration chapters are written for you, and assume no more than that you can use a web interface.

If you are extending it, the component chapters describe each moving part on its own, and the development chapter covers the build and the gates a change has to pass.

If you are evaluating it, read this page and then How the stack fits together. The short version: the stack is three layers, each with exactly one job, and each replaceable without touching the other two.

The words this book uses

Four terms appear everywhere and are worth fixing before you meet them in a command.

A harness is the program that actually is the agent: it holds the conversation, calls the model, runs the tools. This project writes its own, the ganglion, and it is the one this documentation teaches. An older harness, picoclaw, is still supported and on its way out. Which one answers is declared per agent.

An agent is a named configuration — a harness, a model, a lifecycle policy, a personality. alpha and beta are the two this repository ships. An agent is not a process: each member who uses one gets their own container running it.

A tenant is an organisation, and a subscription is an account inside it that members belong to. Together they decide who may reach which agent. A member reaches an agent when they hold a role named after it.

A workspace is one member’s directory for one agent: their conversations, their memory, their files. It is the thing the container gets and nothing else can see.

Reading this book

Start with the quick start. It is the chapter everything else assumes. It takes a fresh clone to a conversation with your own agent in eight numbered steps, and each one tells you how to know it worked. Read it even if you do not intend to follow it, because the later chapters are easier to place once you have seen the pieces come up.

From there the book is in groups, and you can pick the one that matches what you are doing.

Getting started is the quick start, then Installation for the longer version — detailed prerequisites, what the first run writes to disk, and how to reset — and Configuration for every file and variable you touched on the way.

Core concepts explains what the stack is doing. How the stack fits together is the architecture and the reasoning behind its shape. Harnesses covers the two agent runtimes and how one is chosen. Agents, workspaces and projects covers the layout on disk, skills and memory what an agent knows and remembers, and files and delivery how a document gets in and how a result gets back out.

Using it is written for a member rather than an operator: the chat client, projects, and scheduled tasks.

Administration is day-two work: the admin guide, creating a custom agent end to end, and models and providers.

Operations covers deployment modes, the database and its one manual migration step, observability, and troubleshooting — which collects the failures people actually hit, and is worth skimming before you need it.

The components are one chapter each for the four programs in the repository: the orchestrator, the ganglion harness, the chat client and the watcher.

Development covers working on the stack and contributing.

Each chapter owns its subject and links to the others rather than repeating them. If a page seems to stop short of a topic, the link at that point is where the topic lives.

A word about what this is not

This stack is tuned to be readable and easy to run locally, not hardened out of the box. The orchestrator holds the Docker socket and runs as root; it is the most privileged component in the stack and the one you isolate before exposing anything. Traffic between the gateway and its downstreams is not encrypted, because the gateway is expected to be the only thing facing a network. The secrets in the example configuration are placeholders and say so.

None of that is hidden in this book. Where a default is a development convenience, the chapter that owns it says so and says what to do instead.

Quick Start

This chapter takes you from a fresh clone to a conversation with your own agent container. Follow it top to bottom. Every command here is run from the root of the checkout, and nothing in it assumes you have used this stack before.

The path described is the shortest one that works: the development compose file, the alpha agent, and one user — you. Everything else the stack can do is a later chapter.

Before you start

You need four things:

• Docker, with the Compose v2 plugin (docker compose, not docker-compose). The development compose file uses depends_on: condition: service_completed_successfully, which is a Compose v2 feature.
• Git, because the product is assembled from four submodules and a clone without them builds nothing.
• An LLM API key. The two agents this repository ships are both configured for DeepSeek (provider: "deepseek", name: "deepseek-chat" in crab/crab-shell-proxy/config.yaml), so a DeepSeek key is the least work. Changing the provider is configuration, not a code change.
• Internet access during the build, and patience the first time. This compose file builds six images from source, including Mycelium’s gateway from an upstream git commit.

The image builds run with network: host because a BuildKit container gets one usable DNS resolver and no fallback, and parallel builds lose lookups. That is already written into the compose file; you do not pass any flag for it.

1. Clone the repository with its submodules

git clone --recurse-submodules \
  https://github.com/LepistaBioinformatics/zombie-crab-project.git
cd zombie-crab-project

How you know it worked: ls crab/crab-shell-proxy lists Go source. If the directory is empty you cloned without submodules; run git submodule update --init --recursive to fix it without re-cloning.

2. Create your .env

The repository ships an annotated example per deployment mode. Copy the standalone one, which is the local default:

cp deploy/standalone/.env.example .env

Now open .env and replace the placeholder values. Five of them matter for this walkthrough:

Generate the secrets rather than inventing them; the example file suggests openssl rand -hex 32 for each, and any long random string will do.

The picoclaw-shaped names on alpha are not a copy-paste error. alpha ran picoclaw once and was moved to the ganglion against the same per-user directories; its serviceName and its token were deliberately left alone, because the gateway routes by the first and injects the second, so renaming either would have been a new agent to every member rather than the same one on a new runtime.

Both agent tokens have to be set, even though you only use one agent. The beta agent runs the picoclaw harness, and for a picoclaw agent a token that resolves to nothing is fatal: the proxy refuses to start rather than quietly removing a member’s access. alpha runs the ganglion harness, where the same omission disables that one agent instead. See harnesses for what that difference is about.

PICOCLAW_BETA_API_KEY can stay as the example’s sk-your-beta-key. A picoclaw agent’s key is written into the user’s own configuration at provisioning time and a wrong one surfaces as an authentication error on the first message — it does not stop anything from booting. PICOCLAW_ALPHA_API_KEY is different: because alpha is a ganglion agent, an empty value makes the proxy disable alpha at boot, and its routes then answer 404.

3. Bring the stack up

docker compose up -d --build

How you know it worked: docker compose ps lists the services. Two of them are supposed to be gone:

picoclaw-image    Exited (0)
ganglion-image    Exited (0)

Those two are build-only services. They exist so that docker compose up produces the two harness images instead of leaving that as a manual step someone forgets; each runs /bin/true and exits. Everything else should be running, and crab-shell-proxy and mycelium-gateway should reach healthy.

The proxy publishes a loopback-only port in this mode, so you can check it directly:

curl http://127.0.0.1:18080/healthz

Then confirm that no agent disabled itself:

docker compose logs crab-shell-proxy | grep disabled

Silence is the good outcome. A line of the form agent "alpha" disabled: … — its routes will answer 404 names the environment variable you still have to set, and you should fix it before going on.

The ganglion image’s own build runs go vet and go test before it links the binary. That is deliberate — the tests are the image’s acceptance check — so a failing test stops the stack from coming up rather than shipping a broken harness.

4. Claim the Staff account (once per deployment)

The gateway starts with no accounts at all. The first one is claimed through a one-time web flow, gated by the bootstrap secret you set in step 2.

Open http://localhost:8080/_adm/instance/bootstrap, submit the bootstrap secret and your e-mail address. The standalone deployment does not send mail: its e-mail transport writes to the log instead. Read the six-digit code from there:

docker compose logs mycelium-gateway | grep -i bootstrap

Complete the claim with that code.

How you know it worked: the flow returns a Staff token, and the bootstrap URL stops answering — it is a one-time endpoint and becomes a 404 once claimed.

Use a real address you can recognise later. It is the account you will sign in as, and the same address has to be the one you invite in step 6.

5. Create a tenant and a subscription

An agent is reached inside a tenant (an organisation) through a subscription (an account under that tenant that members belong to). Your Staff account can create both, in Mycelium’s own admin interface at http://localhost:8081: sign in there as the account you just claimed, then follow the Staff → tenant → subscription chain.

Those screens belong to Mycelium and are built from upstream sources, so this book does not describe them click by click. What matters here is the outcome: one tenant exists, and one subscription account exists under it.

How you know it worked: the subscription appears in the tenant’s list in that same interface, and you can select it in step 6.

6. Grant yourself the alpha role

Reaching an agent is a permission, not a default. The gateway declares its routes as protectedByRoles, with one role named after each agent, so an account must hold the guest role alpha — at write, which is what sending a message needs — before anything else works. Until it does, every request is refused with a permission error.

The roles themselves already exist: declaring them in the gateway configuration is enough for Mycelium to create them at boot. What is manual is the grant. Do it from the same Staff → tenant → subscription → guest-invite flow in http://localhost:8081, inviting your own e-mail address to the alpha role with write access.

How you know it worked: the invitation appears against your account in that subscription. Once a subscription has members, the same grants can be made from chat-webapp’s own admin area — see the admin guide.

7. Sign in to the chat client

Open http://localhost:3000. This is chat-webapp, the member-facing client. Sign in with the same e-mail address: there is no password, only a magic link, and in this mode its code is logged rather than mailed:

docker compose logs mycelium-gateway | tail -50

On a first sign-in the app offers to create your account before it lets you into the chat; accept, and it creates it for you against the gateway.

How you know it worked: you land on the chat screen instead of the sign-in form.

8. Send your first message

With no workspace selected, the chat area is a picker: one row per tenant, a box per subscription, and a tile for each agent you can reach. Pick alpha and send a message.

How you know it worked: the agent answers — and a container that did not exist a moment ago is now running:

docker ps --filter name=crabshell

You will see something like crabshell-alpha-1a2b3c4d5e6f7890, running as a non-root user. The suffix is a hash of your tenant, subscription and account: a container name is limited to 63 characters, so the identity lives in the container’s labels rather than in its name.

That container is yours. Nobody else’s agent can read its files, its memory or its conversations, because it is a different container with a different volume — which is the whole point of the stack.

alpha is configured to scale to zero. Roughly half a minute after your last message its container stops, freeing the memory; your next message starts it again with everything intact. A stopped agent is normal, not a failure.

Where to go next

If a step did not fit your machine — a Docker version, a path, a port already in use — Installation is the longer version of steps 1 to 3, and explains what the first run writes to disk and how to wipe it. To change the model, add an agent, or understand what you just edited in .env, read Configuration. To understand what actually happened when you pressed send, read How the stack fits together.

Installation

This is the long version of the first three steps of the quick start. Read it when that page did not fit your machine, when you want to know what the stack writes to your disk, or when you need to put an environment back to zero.

Prerequisites

Docker Engine with the Compose v2 plugin. The command is docker compose, with a space. The development compose file waits on build-only services with depends_on: condition: service_completed_successfully, which the v1 docker-compose script does not understand. If you also intend to run the production overlay, you need Compose 2.24 or newer: docker-compose.prod.yaml uses the !reset tag to drop the base file’s build: and its development port publication, and older versions do not parse it.

Git. The product is four submodules — the orchestrator, the harness, the web client and the watcher — and the compose file builds from their working trees.

Enough room to build. A first up --build compiles seven images from source: the patched picoclaw, the ganglion harness, the proxy, the chat client, the watcher, the Mycelium gateway (built from an upstream git commit) and Mycelium’s admin interface. Only Postgres is pulled ready-made. Expect the first run to take a while and every later one to be fast.

Free host ports. In the default configuration the stack publishes 8080 for the gateway, 3000 for the chat client, 8081 for Mycelium’s admin interface, and 127.0.0.1:18080 for the proxy’s development-only port. The databases and the agent containers publish nothing at all: they are reachable only from inside the stack’s own network. Each published port is an environment variable (MYCELIUM_PORT, CHAT_WEBAPP_PORT, MYCELIUM_WEBAPP_PORT), so a collision is an edit to .env, not a problem. If you change MYCELIUM_WEBAPP_PORT, change allowedOrigins in the gateway configuration with it — the admin interface is a browser-side application that calls the gateway directly, and a mismatch is a CORS wall.

An LLM key. Both shipped agents are configured for DeepSeek. The provider is configuration, not code; see models and providers.

Cloning with submodules

git clone --recurse-submodules \
  https://github.com/LepistaBioinformatics/zombie-crab-project.git

If you already cloned without that flag, the submodule directories exist but are empty. Fill them in place:

git submodule update --init --recursive

--recursive matters: the submodules are named in .gitmodules at this level, but this repository is itself the middle of a chain and the habit is worth keeping.

What is in the repository

docker-compose.yaml               the whole stack, standalone/development default
docker-compose.prod.yaml          production overlay (published images, Postgres)
docker-compose.observability.yaml opt-in metrics backend (collector, Prometheus, Grafana)
deploy/                           per-mode configuration
  standalone/                       .env.example + the gateway config this mode mounts
  prod/                             the same pair for production
  observability/                    collector, Prometheus and Grafana configuration
  picoclaw-glob/                    the Dockerfile and patches for the picoclaw image
crab/                             the crab side, one submodule per component
  crab-shell-proxy/                 the orchestrator (Go) — holds the Docker socket
  crab-ganglion-harness/            this project's own agent runtime (Go)
  crab-exoskeleton-webapp/          the chat client; its compose service is chat-webapp
  harness-sphere/                   the watcher; observability only, never a Docker socket
fungi/                            the Mycelium side: gateway and admin UI Dockerfiles
docs/                             task guides and this book
data/                             everything the running stack writes (gitignored)

Two names are worth fixing in your head now, because they differ from the directory they live in. The chat client’s repository is crab-exoskeleton-webapp, but its compose service — and therefore the name in every docker compose command — is chat-webapp. And deploy/ holds two deployment modes, standalone and prod; observability/ and picoclaw-glob/ are configuration for an overlay and for an image build, not modes you can bring the stack up in.

What the first run creates

Nothing under data/ is in git, and the directory does not need to exist before you start. The proxy creates what it needs as it goes, as root, because it is the component holding the Docker socket.

• data/templates/<agent>/ — the per-agent seed cloned into each new user’s directory. For a picoclaw agent the proxy bootstraps this itself from a default template compiled into its own binary, so a fresh checkout works with no seeding step. A ganglion agent gets no template at all: the template is a picoclaw configuration file plus its security file, and writing one for the ganglion would leave two files nothing reads.
• data/tenants/<tenant>/subscriptions/<subscription>/agents/<agent>/users/<account>/ — one isolated workspace per member per agent. This is the tree that gets bind-mounted into the agent containers, and the only place a member’s conversations, memory and files live.
• data/user-secrets/ and data/effective-secrets/ — a member’s own secret store, and the merged view of it with whatever an administrator shared at the tenant or subscription level. The merged view is what gets mounted into a container, read-only.
• data/effective-skills/, data/effective-persona/, data/managed-skills/ — the same idea for skills and for the agent’s identity files. See skills and memory.
• data/restart/ — restart markers, deliberately outside the tenant tree so that an agent cannot read or write its own.
• data/model-registry.db — the proxy’s model registry, a single embedded database file beside the directories.

Two Docker named volumes are created as well, and they are not under data/: mycelium-data holds the gateway’s own SQLite database, which is where accounts, tenants and roles live, and chat-webapp-postgres-data holds the conversation list the chat client keeps. The chat client creates its own tables on first use, so there is no migration step for it.

The production deployment is different in exactly one painful way: Mycelium’s Postgres backend has no embedded migrations, so its schema has to be applied by hand once, after the first start. SQLite applies its own. See the database chapter.

Running from somewhere other than the project root

The proxy hands the Docker daemon a host path as the bind-mount source for every container it spawns, so that path has to be one the daemon can resolve — not a path inside the proxy container. The compose file defaults it to ${PWD}/data, which is correct when you run docker compose from the project root and wrong otherwise. If you run from elsewhere, set it explicitly in .env:

CRAB_HOST_DATA_ROOT=/absolute/path/to/zombie-crab-project/data

Resetting to a clean state

To wipe every per-user agent and all templates and let the stack rebuild itself, stop the stack, remove the containers the proxy spawned outside compose, delete the on-disk state, and bring it back up:

docker compose down

# the agents are not compose services, so compose does not remove them
docker rm -f $(docker ps -aq --filter 'name=crabshell') 2>/dev/null

# the tree is written by root-owned processes, hence sudo
sudo rm -rf data/templates data/tenants data/user-secrets data/effective-secrets \
            data/effective-skills data/effective-persona data/managed-skills \
            data/restart data/model-registry.db

docker compose up -d --build

--build is not optional here. The default template the proxy re-bootstraps from is embedded in its binary, so an image that predates a template change would restore the old one.

Then sign in and send a message: the proxy provisions your user again from nothing. Your login survives, because accounts, tenants and roles are in the mycelium-data volume rather than in data/, and so does your conversation list. Add -v to docker compose down only if you want those gone too — you would then have to claim the Staff account again from the beginning.

Where to go next

Configuration covers every file you have just copied or edited and what each variable does. Deployment covers the production overlay and how the two modes differ. If something did not come up, troubleshooting collects the failures people actually hit.

Configuration

Four files decide how this stack behaves. This chapter says which one owns what, walks through the agent catalogue in detail — because that is where you define an agent — and lists the environment variables worth knowing.

The four surfaces

The one thing to internalise: the agent catalogue is baked into the proxy image, so adding or removing an agent means rebuilding the proxy. The gateway configuration is mounted, so a change there needs only a restart of that one service. They have to agree — an agent that exists in one and not the other is either an advertised route with nothing behind it or an agent nobody can reach.

The agent catalogue: config.yaml

This is the file you edit to define an agent. Everything else in it has a working default; the agents: map does not.

Here is a complete agent, with the lines that matter:

agents:
  alpha:
    serviceName: "alpha"                       # must match the gateway's service key
    harness: "ganglion"                        # which runtime answers
    token: { env: "MYC_PICOCLAW_ALPHA_TOKEN" } # read from the environment, never inline
    template: "alpha"                          # <dataRoot>/templates/alpha
    mode: "scale-to-zero"                      # or "continuous"
    idleTimeout: 30s
    model:
      provider: "deepseek"
      name: "deepseek-chat"
      apiKeyEnv: "PICOCLAW_ALPHA_API_KEY"      # the key lives in the environment

serviceName is the value Mycelium injects as x-mycelium-service-name when it forwards a request, and it is the only thing that tells the proxy which agent was addressed. Mycelium takes the first path segment of the incoming URL as the service name and strips it before forwarding, so a member calling /alpha/v1/... reaches the agent whose serviceName is alpha. A request carrying a service name no agent claims is answered 404 with a message telling you to come through the gateway.

harness selects the runtime. The two accepted values are "ganglion" and "picoclaw"; anything else fails the load with a message naming both. An agent that declares no harness at all gets the ganglion, which is what config.DefaultHarness says. Declare it explicitly on every agent anyway, as every agent in this repository’s own catalogue does: a runtime is not something a configuration should choose by omission, and an omission here does not degrade gracefully. A ganglion agent with no image is disabled rather than started, so an agent that inherits the default on a host with no CRAB_GANGLION_IMAGE stops answering instead of quietly running something else. The boot log says so, and names both ways out. See harnesses for how the two runtimes differ.

token shows the pattern this file uses for every secret: { env: "NAME" } reads the value from the proxy’s environment at load time, so nothing confidential is ever written here or baked into the image. A bare string is also accepted, and is the wrong choice outside a test.

template names a subdirectory of <dataRoot>/templates/. For a picoclaw agent the proxy bootstraps a missing template from a default compiled into its binary. A ganglion agent has no template on disk: it is configured through the environment and files written per user.

mode and idleTimeout are the lifecycle. scale-to-zero stops the container after the idle window and starts it again on the next message, with everything preserved. continuous never stops it, which is what picoclaw’s native connectors need — they dial out from inside the container, so the proxy cannot see that activity to keep the agent alive. For a ganglion agent there is no such side door and the mode is a plain cost decision. idleTimeout must be greater than zero when the mode is scale-to-zero; it is ignored otherwise.

model pins the provider and the model name and, crucially, names the environment variable holding the key rather than the key itself. Each agent has its own, so two agents can use different providers and independent credentials. An optional baseUrl overrides the endpoint the proxy would otherwise resolve from the provider — what you want when you run a gateway or a regional endpoint in front of the provider.

What happens when a ganglion agent is not fully configured

This is the behaviour most likely to surprise you, and it is deliberate.

A ganglion agent removes itself from the catalogue, at boot, when this environment cannot run it: no image reference, no token, or an empty value behind its model.apiKeyEnv. It does not take the proxy down, and it is not silent — the boot log carries a line naming the agent and the missing setting, and the agent’s routes then answer 404:

agent "alpha" disabled: PICOCLAW_ALPHA_API_KEY is unset (the agent's model apiKeyEnv) — its routes will answer 404

The reasoning is that one config file should be able to describe several deployments. A ganglion agent reaching a host with no key for it degrades to “that agent does not exist” rather than “the proxy will not boot”, which would take every other agent down with it.

A picoclaw agent behaves differently on purpose: a token it cannot resolve is fatal, because silently dropping one would remove a member’s access with no signal beyond a log line nobody reads until they are locked out. Its model key, by contrast, may be empty — it is written into the member’s own configuration at provisioning time and surfaces as an authentication error on the first message.

The settings around the catalogue

The rest of config.yaml is machine-shaped and mostly supplied by compose. The values worth knowing:

• hostDataRoot is the absolute path on the host of the data tree. The proxy hands it to the Docker daemon as the bind-mount source for the containers it spawns, so a path that only exists inside the proxy will not resolve. This is what CRAB_HOST_DATA_ROOT overrides.
• containerDataRoot is where that same tree is mounted inside the proxy. Defaults to /data.
• network is the Docker network spawned containers join. The compose file pins the network’s real name to zombie_net so it stays stable regardless of the compose project name.
• startupDeadline (default 35 seconds) bounds a cold start. turnIdleTimeout (default 120 seconds, set to 600 in the shipped file) bounds how long the harness may stay silent — not how long a turn may take. A long tool-using turn narrates constantly and resets it on every frame.
• containerPrefix (default crabshell) prefixes every container the proxy manages. The name is <prefix>-<agent>-<hash>; the harness is recorded in the container’s labels, not in its name.
• mediaMaxBytes (default 10 MiB) is the only thing an upload is checked against.

Several fields can be overridden from the environment so the committed file stays portable: CRAB_HOST_DATA_ROOT, CRAB_CONTAINER_DATA_ROOT, CRAB_NETWORK, CRAB_LISTEN, CRAB_PICOCLAW_IMAGE, CRAB_PICOCLAW_USER, CRAB_PICOCLAW_HOME, CRAB_GANGLION_IMAGE, CRAB_MCP_BASE_URL and GANGLION_OTLP_ENDPOINT.

The gateway configuration

deploy/standalone/config.standalone.toml and deploy/prod/config.base.toml configure Mycelium. Per agent, the shape is a service block naming the downstream, a secret block holding the bearer token, and one path block per route:

[[alpha]]
host = "crab-shell-proxy:8080"
healthCheckPath = "/healthz"

[[alpha.secret]]
name = "alpha-authorization-header"
authorizationHeader = { headerName = "Authorization", prefix = "Bearer", token = { env = "MYC_PICOCLAW_ALPHA_TOKEN" } }

[[alpha.path]]
group = { protectedByRoles = [{ name = "alpha", permission = "write" }] }
path = "/v1/chat/completions"
secretName = "alpha-authorization-header"
methods = ["POST"]

Three things follow from that block. The service key (alpha) is both the first path segment callers use and the serviceName the proxy matches. The role named in protectedByRoles is created automatically at boot from this declaration — but granting it to an account is a human action, and until then every request is refused. And the token is resolved from the environment per request, so the committed file holds no secrets.

Adding a route means adding a path block. A route the gateway does not know about is refused before the proxy ever sees it, with a message about the path matching no service — which reads like a routing bug and is really a missing block.

Environment variables

The full, commented list is deploy/standalone/.env.example; copy that file rather than writing one from scratch. These are the ones whose behaviour is not obvious from the name:

Two of those deserve repeating as a rule, because they share it: an unset secret means the feature is absent, not unguarded. A deployment that forgot a value grows no new surface, rather than an endpoint behind a guessable guard.

A ganglion workspace has no .secrets/ directory. Credentials reach that harness as environment variables on the container instead, which is why the proxy binds each workspace separately and nothing above it. Under picoclaw the merged secret view is mounted read-only at workspace/.secrets. See agents and workspaces.

Where to go next

Creating a custom agent walks the catalogue edit and the matching gateway block end to end. Models and providers covers model chains and per-member overrides. Deployment covers what changes in production.

How the stack fits together

This chapter explains the shape of the system: which piece does what, why the pieces are separate, and where the security boundary actually is. Read it once before anything else in this section; every later chapter assumes it.

The problem the shape solves

An AI agent reads and writes files, runs commands, and keeps long-lived memory, all steered by natural language it did not write. Run one agent for several people in one process and one prompt injection, one path-traversal bug or one leaky tool is enough for one person to reach another person’s conversations, files and secrets.

So the stack does not share an agent. Every user gets their own container, with their own directory on disk, and the thing that decides who you are is not the same thing that runs your agent.

Three layers

The stack is three layers, each with exactly one job.

   your browser
        |
        v
+--------------------------------------------+
|  1. EDGE  --  mycelium gateway             |   the only thing exposed
|     authenticates, enforces RBAC,          |
|     injects a verified account profile     |
+--------------------------------------------+
        |  x-mycelium-service-name: alpha
        |  profile (accId, tenant, subscription)
        v
+--------------------------------------------+
|  2. ORCHESTRATION  --  crab-shell-proxy    |   holds the Docker socket
|     resolves (tenant, subscription,        |   runs as root
|     agent, user), starts that user's       |
|     container, proxies the turn            |
+--------------------------------------------+
        |  docker.sock            ^
        v                         |  HTTP / WebSocket on zombie_net
+--------------------------------------------+
|  3. AGENT  --  one harness container       |   non-root, one per user
|     per (tenant, subscription, agent,      |   own volume, own memory
|     user); ganglion or picoclaw            |
+--------------------------------------------+

Alongside those three, two more services run but are not in the request path: chat-webapp, the member-facing chat UI, and harness-sphere, the watcher.

1. Mycelium, the edge

Mycelium is an external API gateway, developed separately from this project and built or pulled as part of the stack. It is the only way into the agent API: every request that reaches crab-shell-proxy came through it, and the proxy trusts nothing else about who is calling. It verifies the caller’s token, enforces role-based access, and injects a verified account profile into the request before forwarding it.

The chat client and mycelium’s own admin UI publish ports of their own, because they are browser applications a person opens. That is not a second door into the agents — both call the gateway like any other client.

That last word is the point. The caller never tells the proxy who they are; mycelium tells the proxy, server-side, and identity flows down from a trusted source rather than up from a request body. The routes are role-protected, so an account must hold the matching guest role to reach an agent at all.

A tenant is an organization in mycelium. A subscription is an account under a tenant that members are invited into. The pair, plus the member’s own account id, is what makes one person’s agent distinct from another’s.

Mycelium’s gateway also exposes a JSON-RPC endpoint at POST /_adm/rpc that the chat webapp uses for identity and membership operations. Requests routed through crab-shell-proxy are the proxy’s own REST API and are a separate surface.

2. crab-shell-proxy, the orchestrator

The proxy reads which agent was addressed from the injected service name and which user is calling from the profile’s account id, then ensures that user’s own container is running — starting it on demand, stopping it when idle — and proxies the turn.

Its unit of isolation is a four-part key: tenant, subscription, role (the agent key, such as alpha), and user account id. That tuple is WorkspaceKey in internal/docker/manager.go, and it names both a directory on disk and a container.

The container name is <prefix>-<role>-<hash>, where the hash is a SHA-256 over the tenant, subscription and user ids. The full tuple carries two UUIDs and would exceed the 63-character DNS label limit, which would make the container unreachable by its own name on the Docker network — so the identity lives in the container’s labels and in a .crab-owner.json marker in the user’s directory, not in the name.

3. The agent, behind a harness contract

The third layer is not one program. It is a harness: an agent runtime behind a fixed contract, chosen per agent. Two are supported — crab-ganglion-harness, which this project wrote, and picoclaw, which it started with and is now deprecating. Which one an agent runs is declared in the proxy’s config.yaml. See Harnesses for the choice and its consequences.

Who holds the Docker socket

This is the whole security argument, so it gets its own section.

crab-shell-proxy holds the Docker socket and runs as root. It is the only component that does. The Docker socket is the host daemon: whoever can write to it can start, stop and exec into any container, and from there reach host root. That makes the proxy the most privileged piece of the stack and its trusted control plane.

Everything the proxy spawns is the opposite. Agent containers run as a non-root uid (picoclawUser: "1000:1000" in the proxy’s config.yaml), get their own process, network and mount namespaces, and get a bind of their own directory and nothing else. If one user’s agent is fully compromised — prompt-injected into running hostile code — it still cannot read another user’s files, memory or conversations. Different container, different directory, no shared surface. The isolation is enforced by the kernel, not by application code deciding what to show whom.

harness-sphere, the observability watcher, never receives a Docker socket, and that is a deliberate standing rule rather than an oversight. It runs as root in order to traverse the proxy-created tenant tree, and three constraints keep that narrow: its /data bind is read-only, it publishes no ports, and it gets no socket. A second socket holder would double the blast radius of the stack’s worst compromise. What it would need the socket for — attributing a container to its tenant — is served instead by GET /v1/instances on the proxy, behind its own token.

The stack is tuned to be easy to read and run locally, not hardened. Before exposing it, isolate the socket (a restricted socket proxy, or a dedicated host), terminate TLS at the edge, and rotate the tokens and keys in .env.

What each repository is responsible for

This repository is a thin top level — the compose files, the deploy profiles, the docs and the fungi/ build overlays for the mycelium side — plus four git submodules under crab/.

Mycelium is not a submodule. The fungi/ directory holds Dockerfiles that fetch mycelium and its admin UI from upstream at image-build time.

Each submodule has its own remote, its own pull requests and its own default branch, and the chain is merged bottom-up: a pointer here may only name a commit reachable from that submodule’s default branch, and a CI check enforces it.

Two things a deployment reader should know now

The agent containers are not started by compose. The proxy creates them through the Docker API, one per member per agent, outside any compose project. docker ps shows them as crabshell-<agent>-<hash>, and docker compose down does not remove them.

Production has no published ganglion image. docker-compose.prod.yaml pulls published images for mycelium, the proxy, the chat webapp and harness-sphere, and sets no CRAB_GANGLION_IMAGE; no workflow in this repository publishes one. The development compose builds the image locally under the tag zombie-crab/crab-ganglion:dev, which exists only on the machine that built it. A production deployment running ganglion agents has to supply that image itself.

Where to go next

Harnesses explains the agent layer and how one is chosen. Agents, workspaces and projects covers what the proxy actually writes to disk. For the components as components, see crab-shell-proxy and harness-sphere.

Harnesses

A harness is the program that actually is the agent: it holds the conversation, calls the model, runs tools, and writes what it learned to disk. This chapter explains what a harness is in this stack, why there are two of them, how one is chosen per agent, and what happens when a harness cannot serve something the API was asked for.

What a harness is here

crab-shell-proxy does not contain an agent. It resolves who is calling, makes sure that person’s container is running, and forwards the turn to whatever is inside it. What is inside is the harness.

The proxy talks to a harness over a fixed contract — start the container, wait for it to answer a health check on a known port, send the turn, stream the reply back — so the agent layer can be replaced without touching the gateway or the orchestrator. That claim stopped being theoretical when a second implementation appeared.

The two harnesses

crab-ganglion-harness — this project’s own runtime, written once picoclaw’s limits started to cost more than they saved. A static Go binary on Alpine, spoken to over native HTTP with server-sent events. It reads its whole configuration from the environment plus one read-only file, and it owns exactly one directory. Its only filesystem tool is a shell, and each command it runs is confined to the turn’s workspace by the kernel (Landlock), so .. and /etc are not refused by a string check — they do not exist as far as the command is concerned.

picoclaw — where this project started. A container spoken to over the Pico Protocol WebSocket, configured through a config.json and a .security.yml that the proxy writes into each user’s directory at provisioning time. It does not run stock here: the image is a patched build, because upstream matches dispatch selectors by exact string equality and per-project agents need a wildcard.

picoclaw is being deprecated, and the ganglion is the harness to use. The proxy’s own source says so: HarnessPicoclaw is documented as “the harness being deprecated – still fully served, still the right value to declare for an agent that needs it, but no longer what an omitted key means.” New work goes to the ganglion, and this book teaches the ganglion.

Choosing one, per agent

The choice is a single key on each agent in the proxy’s config.yaml:

agents:
  alpha:
    serviceName: "alpha"
    harness: "ganglion"
    token: { env: "MYC_PICOCLAW_ALPHA_TOKEN" }
    template: "alpha"
    mode: "scale-to-zero"
    idleTimeout: 30s

Two accepted values, "ganglion" and "picoclaw". Anything else fails the config load outright with a message naming the agent — a stale config naming a runtime the proxy no longer orchestrates would otherwise hand a member a container provisioned for something else.

Declare it explicitly, on every agent. An omitted key does resolve, but relying on that means a future change to the default silently changes which program answers your members.

What an omitted key means today

An agent that declares no harness: gets config.DefaultHarness, and DefaultHarness is the ganglion (internal/config/config.go). The empty value is replaced in applyDefaults, before validation runs, so nothing downstream of the config load ever sees an empty string.

The flip is recent and deliberate — the constant’s own comment records the reasoning: the exit criteria in the harness spec are met, every feature the gate once reserved for picoclaw is now served by the ganglion, and the documentation teaches this harness, so a default that disagreed with the documentation would cost somebody an afternoon.

The consequence an operator must know: an agent that declared nothing used to be a picoclaw agent and is now a ganglion one, so it needs CRAB_GANGLION_IMAGE set. A missing image does not crash the proxy — see below — but it does take that agent out of service.

A ganglion agent that cannot run

A ganglion agent is checked at load for two things it cannot work without: an image reference, and a resolved API key when the agent declares an apiKeyEnv. If either is missing, that one agent is disabled — its routes answer 404 and the boot log names the exact variable to set. It is not a fatal error, because a single misconfigured test agent once put the proxy in a crash loop and took the working agents down with it. The image is never defaulted to a moving tag, on purpose.

picoclaw agents are not subject to that check: a picoclaw agent’s key is written into a per-user .security.yml at provisioning time, and an empty one surfaces as an auth error on the first model call, which is what every existing deployment already depends on.

The feature gate, honestly

Some capabilities the API exposes began life as picoclaw constructs — they were fields in a picoclaw config.json — and a harness that does not read that file cannot serve them by pretending to. The rule is stated in internal/httpapi/harness_gate.go and it is short: a feature a harness cannot serve answers 501, naming the harness. It never quietly succeeds.

That rule exists because of a real failure. An earlier third harness shipped with projects and personal models unimplemented, and both were picoclaw config constructs it never read — so a project could be created, stored, listed and reported active while changing nothing about the agent that answered. The member was told it worked. A 501 is worse to receive and far better to debug.

Four features are named in the gate:

The first three are listed in a table called picoclawOnly. The table is deliberately an allowlist of what works, not a denylist of what does not: a third harness added later is refused by default and has to be declared feature by feature, which fails in the safe direction.

A second table, alsoServedBy, records the harnesses that have since grown one of those features — and the ganglion is listed for all three. Projects, personal model selection and the memory graph all work on the ganglion today. Two tables rather than one deletion, because both statements stay true: the feature is still a picoclaw construct in origin, and a harness that has not implemented it is still refused.

So the practical position is this: with the two harnesses this stack ships, requireHarnessFeature refuses nothing. The 501 is what a third harness would get on its first day, before anyone declared what it can do. That is the state the gate is designed to produce, not a gap.

scheduled tasks is declared in the gate and deliberately absent from picoclawOnly. The read routes under /v1/cron/* need no running container and are never gated: showing an inert schedule is better than hiding one, and the response reports the truth about when a task fires rather than refusing.

Cron writes are ganglion-only

Creating, editing and deleting a scheduled task over the API is a separate gate, written inline in cronWriteScope (internal/httpapi/cron_write.go) rather than in the feature table — and it is the one place where the two harnesses genuinely differ today.

The check is agent.Harness != config.HarnessGanglion, and anything else gets a 501 saying “creating scheduled tasks over this API is not available on the picoclaw harness (agent <key>): its agent creates them itself.”

The reason is ownership. On picoclaw the schedule lives in timers inside the container that this process cannot see; writing its jobs.json from outside would produce a record the member can see and a timer that never changed. On the ganglion the proxy owns the schedule, in a file above the container’s bind, so it can both write it and fire it. That placement also means a turn steered by untrusted text cannot schedule its own future turns.

On picoclaw, the surface that still works is asking the agent in conversation. See Scheduled tasks for the member-facing story.

Other differences worth knowing

• Templates. A picoclaw agent is seeded from a template directory (the proxy embeds a default and auto-bootstraps a missing one). A ganglion agent is provisioned with no template at all, on purpose: the template is a picoclaw config.json plus a .security.yml, and seeding one there would leave two files nothing reads.
• Secrets. picoclaw receives credentials as files under .secrets/. The ganglion receives them as environment variables, and a ganglion workspace has no .secrets/ directory at all.
• What gets mounted. The two harnesses share one on-disk layout and differ in what they bind into the container. That difference is load-bearing and is covered in Agents, workspaces and projects.
• Lifecycle. Each agent is scale-to-zero or continuous. picoclaw’s native connectors dial out from inside the container and bypass the proxy, so an agent reached that way must be continuous. A ganglion agent has no such side door, so the mode is a plain cost decision there.

A production caveat

There is no published ganglion image. The development compose builds one under zombie-crab/crab-ganglion:dev, a tag that lives only on the machine that built it, and docker-compose.prod.yaml sets no CRAB_GANGLION_IMAGE. If you deploy with the prod profile and run ganglion agents, supplying that image is your job.

Where to go next

Agents, workspaces and projects for what each harness sees on disk, Scheduled tasks for the cron surface, and crab-ganglion-harness for the runtime itself.

Agents, workspaces and projects

An agent is one thing to the person using it and a different thing on disk. This chapter covers both, then the directory layout the proxy writes, and finally the one place where the two harnesses deliberately diverge.

What an agent is to a member

When you sign in to the chat client you do not pick a model or a container. You pick an agent — alpha, beta, whatever this deployment declares — and you get a conversation with it.

With no workspace selected, the chat area becomes a picker: one row per tenant, a box per subscription inside it, and the agents you can reach as tiles showing your permissions on each (an eye for read, a pencil for write). Clicking one opens a fresh conversation.

An agent is therefore a kind of assistant, shared in name across everyone who can reach it, and private in substance to each of them. Two members chatting with alpha are talking to the same persona, the same model configuration and the same administrator-published skills — in two different containers, with two different memories, two different file trees, and no path between them.

Your agent is keyed on your account id, not your email. Emails are mutable and are kept only as a human-readable marker for operators; change your email and your agent and its history stay yours.

What an agent is on disk

The proxy’s unit of isolation is a four-part key: tenant, subscription, role (the agent key) and user account id. That key names one directory:

<data root>/tenants/<tenant>/subscriptions/<subscription>/agents/<agent>/users/<user>/

Every component is sanitized before it becomes a path segment, so nothing in a request can grow a separator or a ... The directory is created lazily, on the member’s first chat — or up front for a whole subscription if the optional subscriptionAccount.created webhook is registered with mycelium.

That directory is what this book calls the user directory. It belongs to the proxy. Some of it is mounted into the agent’s container; some of it deliberately is not.

The layout

<user dir>/                       the proxy's, NOT necessarily mounted
├── config.json                   harness configuration, proxy-written
├── .security.yml                 picoclaw only
├── .projects.json                proxy-owned: which projects exist
├── .schedules.json               proxy-owned: scheduled tasks (ganglion)
├── .crab-owner.json              proxy-owned: who this workspace belongs to
├── .crab-model.json              proxy-owned: this member's model choice
├── .crab-mode.json               proxy-owned: this instance's lifecycle override
├── workspace/                    THE MAIN AGENT
│   ├── AGENT.md SOUL.md HEARTBEAT.md USER.md
│   ├── memory/  public/  sessions/  ...
└── workspace-<project-id>/       ONE PER PROJECT, A SIBLING OF workspace/
    ├── AGENT.md SOUL.md HEARTBEAT.md USER.md
    └── memory/  public/  sessions/  ...

Two rules make this layout worth learning rather than looking up.

Every harness lays its per-user directory out the same way. A path that means one thing under picoclaw means the same thing under any other harness this stack spawns. That is not a courtesy to picoclaw; it is what lets the proxy read and write these directories with one set of functions instead of one per harness. A per-harness branch that is missing does not fail loudly — it reads a directory that never existed and reports that the member has no history, no files and no scheduled tasks.

The dotfiles above workspace/ are the proxy’s, not the agent’s. They decide which project identities exist, which model is used, whether the container may be kept alive, and what is scheduled. An agent able to edit them could route a peer’s conversations to itself, pick the endpoint its own keys are sent to, or keep its own container running indefinitely. So they sit outside what the agent can reach — by two different mechanisms, depending on the harness.

.crab-owner.json is a small JSON marker recording the full workspace tuple and the owner’s email, so an operator can find which human a container belongs to. It is needed because the container’s name cannot carry that information: the full tuple is two UUIDs, which exceeds the 63-character DNS label limit.

A project workspace is a sibling

When a member creates a project, the proxy creates a second agent identity for them with its own workspace:

workspace-<project-id>       correct
workspace/projects/<id>      WRONG

workspace-<id> is a sibling of workspace/, never a child of it. The name is not a preference. picoclaw derives a named agent’s workspace as <defaults.workspace>/../workspace-<id> when the agent config does not set one, so its own tooling resolves that path independently of anything the proxy writes. The proxy’s WorkspaceSegment helper returns workspace for the empty project and workspace-<id> otherwise, and it does not branch on the harness.

It used to. The ganglion kept projects under the main workspace for one release, to spare itself a second bind, and the cost was that one function was really two pretending to be one — every caller that reached for a project’s sessions, its public directory or its media inherited the split, and a caller that forgot did not fail: it read a directory that never existed. The legacy child path survives only so cleanup code can find a subtree written before the migration. Nothing may create one.

Each project’s workspace has its own persona files, its own memory, its own sessions and its own public/ directory. A project’s history is not reachable by asking for the main workspace’s, which is the point: a project scopes the transcripts, the context window and the files. See Working with projects for the member-facing side.

What the two harnesses mount, and why it matters

The layout is shared. The bind set is not, and that difference is load-bearing rather than incidental.

picoclaw mounts the whole user directory. It has its own restrict_to_workspace setting, which keeps its agent inside workspace/, so the proxy’s own state can sit beside the workspace and still be out of reach. The proxy additionally binds a read-only .secrets view into each workspace, one per project as well as the main one, plus the persona cascade and the shared skills root.

The ganglion mounts each workspace separately — workspace/, and one bind per workspace-<id> — and nothing above them.

The reason is that the ganglion has no restrict_to_workspace, and could not usefully have one. Its only filesystem tool is /bin/sh -c <string>: there is no path argument to refuse, and any denylist of .. or /etc is defeated by $(echo L2V0Yw== | base64 -d). What confines it instead is a Landlock domain around each command, whose only read-write hierarchy is the turn’s workspace — plus the container boundary itself. So proxy-owned state has to be on the other side of the boundary rather than merely adjacent to it.

The first version of the ganglion path did use one wide bind, and the cost was concrete: it put the container’s own bearer token one level above the shell’s working directory, where cat ../.crab-ganglion.json read it.

The clearest illustration is .schedules.json. It is not a log — it is a standing instruction to the proxy to wake a container and run a turn, on a timer, forever. Inside a ganglion bind, a turn steered by untrusted text could write one. For picoclaw a store inside the workspace is harmless, because the agent writing it is the same process that would have to act on it. Here the agent and the actor are different processes.

Two consequences follow from the per-workspace binds:

• The bind set changes when the project set does, and a bind set is fixed at container creation. So the proxy checks for drift on every ensure and recreates the container when the projects no longer match — otherwise a project created since the container started would be invisible inside it, and the turn would run against a directory the proxy never reads.
• The administrator’s shared skills are mounted into every workspace, not only the main one, because the Landlock root is the turn’s workspace. A skills index pointing at files a project turn cannot open is worse than no index.

What a ganglion workspace contains

A ganglion workspace is seeded with memory/, public/, sessions/ and windows/. It is picoclaw’s set minus what only picoclaw has, plus one of its own:

• no cron/ — the proxy holds the ganglion’s schedules, above the bind;
• no .secrets/ — credentials reach this harness as environment variables, not as files;
• windows/ is the ganglion’s own, holding each conversation’s context window on disk. It has no picoclaw equivalent.

USER.md is seeded only if the workspace does not already have one. It is the one persona file the agent writes back, so overwriting it on every ensure would erase what the agent learned about the member every time the container was recreated — which, for a scale-to-zero agent, is routinely.

Inside the container the data root is /data/.ganglion, and only its workspace children are bound. The credential key file and the model registry land beside that root, read-only, outside every workspace: the workspace is the only hierarchy a command can reach, so a writable model list would let a tool steered by untrusted text choose the endpoint the deployment’s own keys are sent to.

Where to go next

Skills and memory covers what lives inside memory/ and skills/; Files and delivery covers public/. Working with projects is the member’s view of the sibling workspaces described here.

Skills and memory

Two things let an agent be more than a model with a chat box: skills, which are procedures it can look up, and memory, which is what it keeps between conversations. This chapter explains both, where they live on disk, and which copy wins when two of them have the same name.

What a skill is

A skill is a directory containing a file called SKILL.md. The file opens with a small frontmatter block naming it and describing when to use it:

---
name: quarterly-report
description: Build the quarterly sales report from the shared CSV exports. Use when the user asks for a quarterly report or for sales numbers by quarter.
---

# Quarterly report

1. Read the latest export from `workspace/.shared/subscription/`...

Only name and description are read. The format is picoclaw’s, unchanged, so a skill written for one harness loads in the other — the ganglion says so in its own internal/skillfile package, which exists precisely to keep one definition of the format for the code that reads skills and the code that writes them.

A skill’s name must match ^[a-z0-9][a-z0-9._-]{0,63}$ when an administrator uploads one, and shared-content is reserved — it is the name of a skill the platform itself ships.

The two roots, and which one wins

Inside a workspace there are two places a skill can live:

<workspace>/skills/<name>/SKILL.md          yours, writable
<workspace>/shared-skills/<name>/SKILL.md   the administrator's, read-only

skills/ belongs to the agent. It is where a skill the agent wrote for itself lands, and where an operator-approved evolution pass writes its drafts.

shared-skills/ is a read-only bind mount the proxy creates from what administrators published. Read-only at the mount, which is a kernel guarantee the agent cannot argue with: Landlock only ever narrows access, so it can never grant a write into a read-only bind.

The two paths above are what a ganglion workspace looks like, and they are what the shipped skill-creator skill tells the agent. Both harnesses are served from the same merged directory on the host, so one administrator action reaches both; picoclaw mounts it at its own skills root rather than inside the workspace, because that is where picoclaw looks.

A name present in both resolves to the administrator’s copy, and the shadowed workspace copy is logged so an operator can see which one won. The rule is stated in the ganglion’s skill loader and again, in plain words, in the skill-creator skill the agent itself reads. The other way round would let an agent overwrite an administrator’s instruction by writing a file with the same name — a privilege escalation dressed up as a merge rule.

There is a second, separate merge that happens before any of this. An administrator can publish skills at four scopes, and the proxy flattens them into the single read-only directory it mounts: tenant, then tenant-for-this- agent, then subscription, then subscription-for-this-agent, with the later source winning by name. That cascade decides what shared-skills/ contains; the rule above decides what happens when its contents collide with the agent’s own.

Only the index reaches the prompt

This is the part that changes how you write a skill.

Every turn is given an index of the available skills — one line each, name and description and the path to the body — and nothing more. The bodies stay on disk, and the agent reads the one it needs with its shell tool, which already reaches the workspace. In the ganglion the index is bounded at 8 KiB by default, roughly two thousand tokens: enough for a few dozen skills at a sentence each, small enough that the index never competes with the conversation for room.

Two consequences follow, and they are the whole craft of writing one:

• The description is the skill. It is the only part that decides whether the body is ever opened. Write it as the situation it answers, not as a title. “Use when the user asks for a file exported or sent” finds its moment; “File utilities” does not.
• The body can be long, and costs nothing until it is read. Do not compress a procedure into hints to save room. Room is not what you are saving.

Putting every body into every turn’s prompt would spend the context window on instructions for work this turn is not doing, which is how a skill library stops being an asset and becomes a tax.

What ships in the box

The proxy embeds a small set of operator-managed documents in its own binary and bind-mounts them read-only into every workspace it creates. The agent can neither edit them nor keep an edit past a restart.

Three of them are skills:

ganglion-workspace is the only harness-specific entry, and the gating is the same principle the 501 gate uses one level up: a document describing this container’s shell, its image and its layout is a description of the wrong machine for the other harness, and an agent acting on a capability it was told about rather than one it has is exactly the failure to avoid. It stands in for picoclaw’s own bundled workspace skill, which a ganglion agent never gets because a ganglion agent is provisioned with no template at all.

The two shipped skills overlap slightly and disagree in one place on purpose. shared-content describes a .secrets/ directory; a ganglion workspace has none, because credentials reach that harness as environment variables. The ganglion-workspace skill says so in its own words and tells the agent to read the environment instead. It is an acknowledged divergence, not a bug.

Memory

An agent’s memory in this stack is files in workspace/memory/, plus an optional graph. They are not interchangeable, and the difference is which one the member can see.

The memory files

memory/MEMORY.md is the agent’s own notebook — what it has learned, and what it writes to.

memory/MEMORY_CUSTOM.md is the member’s file. It holds their standing notes for the agent: preferences, context, instructions they want kept in mind. The member edits it directly from the chat client’s Workspace panel, at any time, between turns. The proxy writes it on their behalf, through a kernel-confined handle so a swapped path component fails the syscall rather than redirecting a root-owned write. An absent file is an empty document, not an error.

Because the member can change it between any two turns, the shipped shared-content skill tells the agent to re-read the current file whenever it is relevant rather than trusting what it remembers of it.

Three more files in memory/ are the operator’s, mounted read-only:

• CONTEXT_RECOVERY.md tells the agent that its live context can be reset when the container restarts — idle scale-down, a settings change, a redeploy — and that the complete history is preserved for it in workspace/sessions/durable/<session-key>.jsonl, an append-only file that only ever grows. If the agent seems to be missing earlier messages, it is instructed to read that file back rather than guess.
• FILE_DELIVERY.md is the rule about where a produced file goes. See Files and delivery.
• MEMORY_ROUTING.md says which memory to write to, and is mounted only when the memory graph is enabled. With no graph the agent has no mcp_memory_* tools at all, and a document telling it to prefer them would be actively wrong — worse than silent.

These are memory files rather than skills for a reason worth understanding. Skills are loaded by relevance: the agent has to decide to go looking for one, so a rule that must apply on every turn would only be found at the moment it was least needed. The memory directory is read every turn.

The memory graph

When it is enabled, the agent also has a knowledge graph: entities, observations about them, and named relations between them. It is the memory the member can browse, search and audit from the chat client, and it is the only one that records which conversation a fact came from.

It is served by the proxy itself. The proxy writes an MCP server block named memory into the workspace’s config.json, pointing at its own /v1/mcp route with a bearer token that carries the workspace scope plus an HMAC over it — so the proxy stores no tokens, and rotating the signing secret revokes every issued token at once. The ganglion has an MCP client of its own and reaches the same graph a picoclaw container in the same workspace would.

The graph is scoped to the member and spans their projects, rather than being a separate graph per project.

The tools the agent gets are all prefixed mcp_memory_: mcp_memory_create_entities, mcp_memory_add_observations, mcp_memory_create_relations, mcp_memory_search_nodes, mcp_memory_semantic_search, mcp_memory_read_graph and mcp_memory_open_nodes. MEMORY_ROUTING.md gives the agent three rules about using them, each of which came from watching it get them wrong: search before you create, because creating ignores a name that already exists and two spellings produce two entities no single query reunites; create the relation too, because an entity with no relations is a point nothing leads to; and keep entityType consistent, because the member filters the list by it.

Enabling it is one environment variable, CRAB_MCP_TOKEN_SECRET, and leaving it unset is supported: the /v1/mcp route is not registered at all, no MCP block is written into any workspace, and everything else behaves as before. That is deliberate — a deployment that forgot the secret must get no memory rather than an unauthenticated endpoint reachable by every container on the network.

Facts go in the graph; the agent’s own working notes go in MEMORY.md. The routing document also forbids the agent from claiming a save it did not make, which was written after it was observed appending to MEMORY.md and then telling the member it had written to the graph as well.

Where to go next

Files and delivery for public/, the other member-visible directory in a workspace, and the Admin guide for publishing shared skills and files at a scope.

Files and delivery

Files travel in both directions. A member attaches something for the agent to work on; the agent produces something the member has to be able to download. Both use one directory, and this chapter is about that directory and the rules around it.

One directory: public/

Every workspace has a public/ directory, and it is the only directory the member’s interface lists. That single sentence is the whole design. A file written anywhere else in the workspace exists, and is perfectly readable by the agent, and is invisible to the person the agent is working for.

workspace/public/                 what the member sees
workspace/public/attachments/     where the agent delivers

A project’s workspace has its own public/, sibling to the main one, so a file uploaded into a project stays in that project. See Agents, workspaces and projects.

Sending a file to an agent

From the composer in the chat client, attach a file. The client posts it to the proxy’s media route, which:

• caps the request body — the default limit is 10 MiB, configurable as mediaMaxBytes, and an oversized upload is rejected with 413 without being buffered in full;
• guards the type with an extension allowlist, rejecting anything else with 400;
• reduces the filename to a safe base name — directories stripped, unsafe characters collapsed to _, leading dots dropped — and refuses one that would end up empty or contain traversal;
• writes the bytes into your workspace’s public/ directory and chowns it so the non-root agent can read what the root proxy wrote.

The response is the workspace-relative path, public/<name>, and that is the path the turn references, so the agent can open the file by exactly the name it was told.

Two behaviours are worth knowing because they are deliberate:

• Re-uploading the same name overwrites. One file per name, rather than an accumulating pile of report(1).pdf.
• Uploading into a project requires the project to travel with the file. The upload route is the one multipart route on the media surface, so the project arrives as a form field rather than a query parameter. Sending it without one used to land the file in the main workspace, where the project’s agent could never open it.

The write goes through a kernel-confined handle on the public/ directory, so a name that resolves onto a symlink pointing out of the tree fails the syscall instead of writing wherever it pointed. That matters here more than it usually would: the proxy runs as root, the tree is writable by the agent inside the container, and the content comes from the network.

Getting a file back from an agent

There are two ways a file reaches public/attachments/, and a member cannot tell them apart — which is the intent.

The agent writes it there itself. This is the ordinary path on the ganglion. The shell tool’s working directory is the turn’s workspace, so public/attachments/report.pdf is a relative path that lands where the member will find it.

The harness delivers it out of band. picoclaw answers a “send me the file” request with a short sentence and pushes the file itself through its own media channel. The proxy fetches those bytes — with a 60-second timeout and a 64 MiB cap — and writes them into public/attachments/<name> under the same filename sanitization a browser upload gets. Copying rather than proxying on demand is deliberate: the harness’s media store is its own cache with its own lifetime, and a file under public/ is already listable and downloadable by everything that serves a file the member uploaded.

The extension allowlist that applies to uploads is not applied to deliveries. That allowlist constrains what an outside caller may push into a container; a delivered file was written by the agent inside its own workspace, so refusing it there would drop legitimate work while adding no boundary the workspace does not already have.

attachments/ is reserved

public/attachments/ is created, named and populated by the proxy, so the member cannot rename it, move it, delete it or create their own folder by that name — the API refuses with a “managed by the system” error. Renaming it would silently detach every future delivery.

Only the top level is reserved. reports/attachments is an ordinary folder a member may legitimately want; forbidding the word everywhere would be a rule about vocabulary rather than about ownership. And the refusal lives in the API, not only in the interface, because hiding a button is not a permission.

What the member sees

The file appears in the workspace Files panel, in the same list as the files they uploaded themselves, with click-to-download. Nested folders are listed, so an agent that organized its output into subdirectories is rendered as it organized it.

In the conversation itself, the proxy appends a short notice to the reply:

📎 report.pdf — public/attachments/report.pdf

That notice is stream-only. It is injected by the layer between the agent and the member and is not part of the message that gets saved, so after a page reload it is gone. The only durable account of a delivered file is the Files panel — and whatever the agent wrote in its own words.

Which is why the shipped FILE_DELIVERY.md memory document, read on every turn, tells the agent to name the path in its reply:

Salvei o relatório em public/attachments/relatorio-q2.pdf.

That example is in Portuguese because it is quoted verbatim from the shipped document, which is written for a deployment whose members write Portuguese. The instruction it carries is the point, not the language: name the path you wrote to, in the reply.

and forbids it from announcing a file it did not actually write, or a path it did not actually use. picoclaw’s own stock line — “Requested output delivered via tool attachment.” — names nothing, and a reply that says only that becomes, after one reload, a message about a file with no way to find it.

The same document draws the line for the agent in one sentence: anything the member is meant to receive goes in public/attachments/; working files the agent needs only for itself go anywhere else. When in doubt, deliver — a visible file they ignore costs nothing, an invisible one they wanted costs them the whole request.

uploads/ is the legacy name

public/ used to be called uploads/. The rename happened, existing files moved with it, and nothing should be written to uploads/ any more.

The old name survives in three places, each of which is fine:

• Older conversations and older references may say uploads/.... Reading such a path still works where the directory still exists; the agent is explicitly told not to create an uploads/ folder to match one.
• A one-time migration. The proxy routes every access to the member’s public directory — a listing, an upload, a delivery — through one accessor that migrates a pre-rename uploads/ into place first. Hooking the accessor rather than the provisioning step is what makes it reach workspaces created before the rename, since those are never re-provisioned. If both directories somehow exist, they are merged file by file and the newer file wins a collision, mtime being the only evidence available about which copy the member meant to keep.
• Comments and identifiers in the proxy’s source still say “uploads dir” in places. The constants are what to trust: PublicDirName is public and LegacyPublicDirName is uploads, and the only thing that should reference the second is the migration.

A regression test asserts that every shipped skill and memory document names public/attachments and that none of them points a write at the legacy directory. It exists because one of them once did: the shared-content skill told the agent to write deliverables to uploads/attachments while FILE_DELIVERY.md, bound into the same workspace and read every turn, said the opposite.

Where to go next

The chat client for the Files panel in context, Skills and memory for the documents quoted here, and Troubleshooting when a file the agent says it wrote does not appear.

The chat client

This chapter is for the person who uses an agent rather than the person who runs one. It walks through the web chat — crab-exoskeleton-webapp, the compose service named chat-webapp — from signing in to reading a file the agent wrote, and names each part of the screen so the rest of the book can refer to it.

Signing in

There are no passwords. You type your email address, the gateway sends you a message, and you finish the sign-in with a six-digit code: the screen says “Check you@company.com for a link, open it, and enter the 6-digit code it shows.” The two steps live in the URL rather than in the page’s memory, so reloading on the code form keeps you on the code form.

After that the browser holds a session cookie and nothing else — no token, no identity, no upstream address. Every request goes from the browser to the app’s own server layer, from there to the gateway, and only then to the orchestrator.

If the sign-in screen says it cannot reach the gateway, the stack behind the web app is not answering. That is a deployment problem, not an account problem — see Troubleshooting.

Picking a workspace

Your account can reach more than one agent. Until you choose one, the centre of the screen is the picker: a grid titled “Pick a workspace”, grouped by tenant and by subscription, with one card per agent. A tenant is the organisation you belong to; a subscription is an account within that tenant, and an agent belongs to one of them. An agent you can only read is marked “read-only access”; write access is the norm and is not labelled.

Clicking an agent opens a fresh conversation with it. Which agent you are in is kept in the URL fragment, so workspace identifiers never reach the server.

The shape of the screen

Once a workspace is chosen there are three columns, and only the middle one is always present.

• The sidebar, on the left: the brand header, New chat, the list of places you can go, your conversation list, and an account footer.
• The centre, which holds whatever you are reading — a conversation, the landing screen, or the projects screen — with a breadcrumb across the top.
• The pane, on the right, which opens beside the conversation rather than over it. It is a sibling of the centre column in the layout, so the transcript reflows to whatever is left instead of being covered.

On a phone the sidebar and the pane are both full-height drawers, and the one button at the top left toggles the sidebar.

The sidebar, and the rail it collapses to

The sidebar can be dragged wider or narrower by its right edge, and collapsed with the circled arrow in its header. Both the width and the collapsed state are remembered in your browser between visits.

Collapsed, it becomes a 48-pixel rail of icons in three groups separated by hairlines, in the same order the open sidebar reads: New chat, then the destinations, then Conversations. Two behaviours are worth knowing because they are not the usual ones:

• Hovering the Conversations icon slides the conversation list out over the screen as a preview, and you can move the pointer into it and click a conversation. It is the only entry that does this; hovering any other icon closes the preview.
• Hovering any other icon shows a small two-line tooltip — the name, and one line saying what it opens — and chooses which panel the pane would show without expanding the sidebar. The circled arrow above the icons is the only control that expands it.

The destinations

Six rows, and they are not all the same kind of thing. Projects replaces what is in the centre of the screen; the other five open the pane beside it, and clicking the one already open closes the pane again.

Memory and the knowledge graph are two different things with two different names, and Skills and memory owns both. Scheduled tasks have a chapter of their own, Scheduled tasks. If your agent runs on an orchestrator older than projects, the Projects row is not rendered at all rather than rendered and dead.

The breadcrumb

One bar across the top of the centre column says where you are: the workspace, the subscription, the project if you are in one, and the conversation. Each segment is a way back up — from a conversation, “up” is the project’s own screen. The chevron at the end opens the conversation’s actions.

Conversations and history

The conversation list sits under the destinations. It has two views, switched by the List / Tree control: the familiar list by recency, and a tree that draws how the work unfolded over time.

Above the list is a filter. It takes plain text, and it takes four prefixes that are query syntax and are the same in every language: tag:, alias:, text: and date:.

Each row carries the actions that apply to one conversation — rename it, give it an alias and coloured tags, or delete it. The alias is what you called the conversation; the title is what its first message made of it, and the alias wins wherever there is one. Deleting is described plainly: the chat is removed from your list and it cannot be undone.

The heading over the list changes with where you are: “General chats” for the conversations that belong to no project, and “Chats in this project” inside one.

New chat does not create anything. It takes you to the landing screen — a composer with the scope’s conversations listed under it — and the conversation is minted by the first message you send.

The composer

The box at the bottom says “Message your agent… (Shift+Enter for a new line)”. Enter sends; Shift+Enter breaks the line.

Three things can happen as you type. A leading / opens the slash-command menu (/rename sets the conversation’s alias, /tag applies a tag). An @ opens a menu of the files in your workspace, so you can point the agent at one by name. Everything else is an ordinary message. There is also an advanced markdown editor behind its own button, with a live preview and the usual formatting tools, for anything long enough that a one-line box is the wrong shape.

Above the field is a context slot. It shows what the next message will carry besides your prose: a message you chose to reply to, which travels as a quote; a knowledge-graph entity; or a scheduled task or one of its runs, picked from the pane on the right.

While the agent is answering, the send button becomes Stop generating. It is not decorative — the turn is really aborted upstream and rolled out of the transcript, and what you had typed comes back into the box.

Attaching a file

Three gestures, all of which produce the same attachment:

• the paperclip button, which opens your operating system’s file picker;
• pasting a file onto the composer, which is how a screenshot gets in (pasted images are renamed as they arrive, so a second paste cannot overwrite the first);
• dragging files in from outside the browser, which shows a “Drop to attach to this message” overlay over the conversation.

Attached files appear as square tiles above the field, showing the picture itself for an image, so you can see you picked the right screenshot before you send.

Dragging files onto the Files pane instead of onto the conversation is a different act: that pane is filing, not composing, and the overlay says “Drop to add to this workspace”. What becomes of a file after it arrives — and how the agent hands one back to you — is Files and delivery.

The preview pane

Clicking a file, whether in the transcript or in the Files pane, offers Download always and Preview when the format is one the app can draw. Images skip the menu and open straight into the preview.

Some formats get a reader of their own: images, Markdown, HTML, PDF, source code, and the office families .docx/.odt/.odp and .xlsx/.ods. Anything else that is text — including a file with no extension at all, like LICENSE, and a dotfile like .gitignore — is shown as plain text. What stays download-only is what genuinely cannot be read here: archives, audio and video, executables, fonts, the pre-2007 office binaries, and any file whose first bytes turn out to be binary. Even the plain-text reading is safe by construction — text is escaped inside a <pre>, and the orchestrator serves every media file as an attachment your browser will not render, so your files never become a page on this origin.

What the pane can do, once it has the file:

• Markdown and HTML have two readings, and the “How to read this file” control switches between Rendered and Source.
• PDFs are drawn by the app itself, page by page, with previous/next, zoom and fit-to-width. If the browser cannot manage it, the pane says so and offers the download.
• Partial views admit it. A spreadsheet shows “Showing the first n rows”, a presentation says its text was extracted and the deck itself is in the file, and a file too large to preview says to download it instead.
• A file that turns out not to be text says so once its bytes arrive, rather than painting a screen of replacement characters.

Scripts inside an HTML preview are off by default. Turning them on opens a dialog that states both halves honestly: with scripts on, the page can send what the document contains to any address on the internet, but your session, your cookies and the rest of the app stay out of its reach. The permission lasts until you close the browser and covers every HTML file you open in that time.

While the agent is working

The assistant’s band shows what the turn is doing — “Thinking…”, “Using tool”, a collapsed run of narration steps that states its own count, and the model’s own reasoning behind a fold. If the stream is cut, the app says the connection dropped and that the agent is still working, which is a different message from the one it shows when your device goes offline.

Leave a conversation mid-turn and it does not stop. A dock along the bottom lists the conversations running in the background, each with its state — working, reconnecting, reply ready — and clicking one takes you back to it.

A banner above the centre pane appears when something has changed that your agent will only pick up after a restart: a secret you saved, a model or a shared skill an administrator changed, or a restart an administrator asked for. It names the reason and gives you the button, so you choose the moment and no live turn is cut off.

The rest of the chrome

The footer of the sidebar carries your email address, the language switcher, and Log out. Above it are the link to the administration console — shown only if you may reach it — and Install app, because the web app is installable as a PWA. On iPhone and iPad the app explains Safari’s Share → Add to Home Screen flow, because Safari has no install button of its own.

Where to go next

Working with projects explains the one destination this chapter deliberately skipped, and Scheduled tasks the one pane whose behaviour depends on which harness your agent runs. For what the memory pane and the knowledge graph actually hold, read Skills and memory; for uploads and deliveries, Files and delivery.

Working with projects

A project carves one agent into subjects. This chapter is for the member who has noticed that every conversation with their agent shares the same files, the same instructions and the same accumulated notes, and wants one subject kept apart from the rest.

What a project is

Your agent has one workspace: a directory on the host that holds its files, its memory and the transcripts of everything you have said to it. A project is a second workspace beside the first one, belonging to the same agent and the same you, with its own copy of all three.

On disk a project’s workspace is workspace-<id>, a sibling of workspace/ and never a child of it. Both harnesses lay it out the same way, which is the whole reason the layout was changed to this shape — Agents, workspaces and projects covers what is inside.

The app states the idea in one sentence on the projects screen: a project keeps its own files, memory and instructions, and inherits this agent’s model, skills and credentials.

Why you would make one

Because “this thread is about the seed-trial analysis, keep its files and its instructions apart from my other work” is otherwise unsayable. Everything you upload lands in one pile; every standing note you write applies to everything; and an instruction that is right for one subject is noise in every other conversation.

A project answers all three at once. What it is not is a way to give somebody else access: a project belongs to one (tenant, subscription, agent, user) tuple, exactly like the workspace it sits beside, and there is no sharing between members and no delegation between projects.

What changes inside a project

The scheduled-tasks row has a harness caveat: only the ganglion really files a schedule under the project it was created in. picoclaw keeps one schedule store per container, which is a limitation of that harness rather than a decision — see Scheduled tasks.

Two other rows are the ones people get wrong.

The knowledge graph is not per project. The memory pane and the knowledge graph are two different memories with two different names. The notes you write in the memory pane belong to the project you are in; the graph the agent builds for itself is scoped to you and spans your projects — one graph server, reached with the project named in a header rather than a separate graph per project. See Skills and memory.

Administrator-shared files stay in the main workspace. Files and content an operator publishes to a tenant or a subscription cascade into your agent’s own workspace only; they are not copied into project workspaces. Shared skills, the model and your credentials are inherited.

A project also inherits your agent’s identity rather than replacing it. The instructions you type are added to what the agent already knows about itself, and the app says so where you type them: “Say what this project is and how it should behave here.”

On the ganglion harness the separation is enforced by the kernel, not by convention: a turn inside a project runs with its filesystem confinement rooted at that project’s own directory, so the shell tool cannot reach the main workspace or any other project. That is what the sibling layout bought — a project under the main workspace could only ever have been a convention.

Creating one

Open Projects from the sidebar. Unlike the other five destinations, this one replaces what is in the centre of the screen rather than opening a pane beside it — a project is a place you enter, not a panel you consult.

Press New project and fill in two fields:

• Name — for example, “Seed trial 2026”. The identifier is derived from the name by the orchestrator; you never type one.
• Instructions — optional, and the thing worth spending a minute on. The placeholder is a good model: “Always cite the trial protocol and answer with the plot number first.”

Saving creates the workspace. The screen then tells you the one consequence you would otherwise discover as a slow first reply: the agent restarts on your next message. A project changes what the agent’s container has mounted, so the container is rebuilt before the next turn.

Each project is a card on that screen, showing its name, its instructions and the date it was created. The project you are currently inside is marked Current rather than hidden or moved to the front — asking to see the list is not the same as leaving where you are.

Working inside one

Entering a project changes the scope of almost everything on screen. The breadcrumb names it. The conversation list shows “Chats in this project” instead of “General chats”. The landing screen says “Ask anything in this project to get going” rather than naming the agent. The Files, memory and Scheduled tasks panes all address the project’s workspace, not the agent’s.

One rule catches people out: a conversation stays in the project it started in. The project selector is offered when you start a conversation and locked afterwards, because a conversation’s transcript physically lives in that project’s workspace. The app says it plainly — “A conversation stays in the project it started in.”

To move work between projects, hand the file over yourself through the Files pane. The agent cannot copy it across; from inside one project the other one is not there.

Editing and deleting

The pencil on a card edits the name and the instructions. Only what you changed is sent, so renaming a project does not blank its instructions.

Deleting is the destructive one, and the confirmation says exactly what goes: “Its files, memory and every conversation in it are removed. This cannot be undone.” Like creating, it restarts the agent on your next message.

Deleting a project does not delete the scheduled tasks that were filed under it. They keep firing, and the orchestrator deliberately shows them again in the agent’s own task list afterwards — a task you cannot see is a task you cannot stop. Delete a project’s tasks before you delete the project. See Scheduled tasks.

Which agents have projects

Both harnesses this book covers serve projects: picoclaw builds them from its own agent-and-dispatch machinery, and the ganglion takes the project as a request header and works out of the matching sibling workspace. See Harnesses for what a harness is and how one is chosen.

What does not have them is an orchestrator older than the feature. Against one of those the web app reports the agent does not support projects, and both the sidebar row and the screen are omitted entirely rather than shown and dead.

Where to go next

Scheduled tasks is the one project-scoped surface with a constraint of its own. Agents, workspaces and projects describes what a project workspace looks like on disk, and Skills and memory explains the two memories a project does and does not separate.

Scheduled tasks

A scheduled task is a message your agent sends to itself on a timer. This chapter explains what one is, what the Scheduled tasks pane shows you, and the one awkward fact about them: reading tasks works on both harnesses, but creating, editing and deleting them over the API is ganglion-only, and any other harness answers 501.

What a scheduled task is

A task is a stored record: a name, a schedule, and a message. When the schedule comes due, that message is replayed as an ordinary turn — the agent reads it, works, and writes a transcript — and nobody is waiting at the other end. The result is stored and read back later from the pane. A job fired by the orchestrator’s own scheduler, which is how the ganglion works, is delivered to nobody at all by design.

Three kinds of schedule exist, and the pane names each of them the way the record does: a cron expression (Cron 0 7 * * 1), a repeating interval (Every 6h), or a single instant (Once at …). A task can also be marked to remove itself after it runs, which the pane shows as “Removes itself after running”.

This is unattended work. The message is replayed on every occurrence, forever, with nobody reading the result, which is why the orchestrator caps it at 8 KiB and bounds one fired turn at thirty minutes.

The pane

Open Scheduled tasks from the sidebar and it appears beside the conversation, so you can read a past run without losing the chat you are in. Its own subtitle states the scope honestly: “What the agent runs on a schedule, and what each run produced. Read-only: ask the agent to create or change a task.”

Each task shows when it next runs, when it last ran, whether it is disabled, and where it delivers if its record names a target. Under it are its runs. Clicking one opens that run’s transcript — the messages, the tool calls and their results — and Back to tasks returns you to the list.

Two things about runs are worth knowing before you read too much into them:

• Only the most recent run of a live task records a status. Earlier runs show how long they took and how much they logged, and nothing more. There are no success ticks next to them because the store carries no per-run outcome to draw one from.
• A run can outlive its task. A task that removes itself after running leaves its transcripts behind, so the pane groups those under “Removed task” — the task is no longer scheduled, but the work it did is still on record.

The pane also offers Hide finished, which folds away tasks that already ran and will not run again, a Check for new tasks refresh (the agent may schedule something between two visits), and a Reference in chat control on any task or run. That last one drops the task or the run into the composer’s context slot, so you can ask about it in the conversation beside you without copying anything.

Who can create one, and where

This is the part that depends on your agent’s harness. A harness is the program inside the agent container that actually runs the loop; the two this book covers are picoclaw and the ganglion, and Harnesses explains how one is chosen.

Reading works on both. That is deliberate, and the orchestrator’s own code says why: hiding a schedule that will not fire is strictly worse than showing it and saying so, because a task you cannot see is a task you cannot stop. So GET /v1/cron/tasks and GET /v1/cron/runs are served whatever the harness, and they need no running container.

Writing is ganglion-only. POST, PATCH and DELETE on /v1/cron/tasks are refused for any other harness with a 501, naming it:

creating scheduled tasks over this API is not available on the
"picoclaw" harness (agent "beta"): its agent creates them itself

That is not a bug and it is not a stub. On picoclaw the schedule lives in timers inside the container, in the agent’s own memory, where the orchestrator cannot see them — so writing picoclaw’s job file from outside would produce a record you can read and a timer that never changed. A 501 is worse to receive and far better to debug.

The reason the ganglion can be written is the mirror image. It has no scheduler at all; the orchestrator holds the schedule, in a file kept above the container’s workspace mount, and fires the jobs itself. There is no second writer to disagree with. Keeping that file out of the container is also a safety property: nothing inside the agent can reach it, so a turn steered by untrusted text cannot schedule its own future turns.

The web app writes neither. However your agent is configured, the pane today is read-only — the app’s server layer exposes only the two read routes, and there is no create button anywhere in the chat. So in practice:

That second row is the honest state of things, and it is worth spelling out. Asking a ganglion agent to schedule something will not work: it cannot reach the store, and its own workspace guide tells it so — “There is no cron/ either: your scheduled work is held outside this tree, where you cannot reach it.” The web app has no create control. And the write routes cannot be called from the browser either, because the browser holds a session cookie and nothing else: no token, no upstream address. The routes are built and the storage is in place, but the surface that would let a member reach them has not been written.

The pane’s “ask the agent” sentence therefore describes the picoclaw case, and is the one piece of the screen that does not yet distinguish the two harnesses.

What the difference actually costs you

If you only ever read your tasks, the harness barely shows. Where it shows is whether the schedule fires at all.

A picoclaw schedule is in-process timers, so a stopped container fires nothing. On an agent configured to shut down when idle, the tasks are real, listed, and inert — and the pane says so rather than letting you assume the task ran:

These tasks are not running. This instance shuts down when idle, and a schedule only fires while it is up. The tasks below are still recorded — an administrator has to switch this instance to continuous for them to run.

A ganglion schedule is held by the orchestrator, which is always up and which starts the container to deliver the turn. Scale-to-zero is precisely what that design was built around, so the notice never appears for a ganglion agent.

Three more properties of the orchestrator’s scheduler are worth knowing, because they are choices rather than accidents:

• A job is claimed before it runs, not recorded after. An orchestrator that dies mid-turn loses that run rather than re-firing it on every boot. For unattended work a missed daily summary is a gap; a re-fired one is an agent doing real work twice with nobody watching.
• A schedule that was slept through fires once. Replaying every missed occurrence would deliver a weekend of hourly turns at once.
• Two scheduled turns never overlap in one workspace. A job that comes due while another of yours is running is left due and picked up on the next pass. Your own messages are unaffected in both directions — a scheduled run neither blocks them nor is blocked by them.

Tasks and projects

Scheduled tasks are scoped like everything else in a project: inside a project you see that project’s tasks and nothing else, and outside one you see the tasks that belong to the agent’s own workspace. Per-project schedules are something only the ganglion can do — on picoclaw every project’s jobs land in one store, which is a limitation of that harness rather than a decision.

One consequence is worth repeating from the projects chapter. Deleting a project drops its routing but not its scheduled jobs, so they keep firing. Rather than let them become invisible, the orchestrator lists a deleted project’s orphaned jobs in the agent’s own task list, where you can still find and stop them.

Where to go next

Harnesses explains the choice this whole chapter turns on and why an unstated harness: key still means picoclaw today. Working with projects covers the scoping, and the Admin guide covers the instance mode an administrator must change to make a picoclaw schedule fire.

Admin guide

This chapter is for whoever runs a deployment day to day: adding people, giving their agents material to work with, deciding which model they get, and repairing one member’s configuration when it goes wrong. It assumes the stack is already installed — see installation if it is not.

Tenants, subscriptions and scopes

Identity in this stack comes from Mycelium, an external API gateway. A tenant is an organisation; a subscription is one account under a tenant, and it is the only level at which a member list exists. A tenant may have many subscriptions.

Everything an administrator publishes is published at a scope, which is either a tenant or a subscription. A tenant scope reaches everyone under it; a subscription scope reaches only that subscription. Where both have something to say, the narrower one wins.

What you may administer depends on your tier, resolved from the Mycelium profile on every request (CallerTier in crab/crab-shell-proxy/internal/authz/authz.go). Instance staff are authoritative everywhere; a tenant-owner or tenant-manager over one tenant and every subscription under it; a subscriptions-manager over exactly one subscription. The admin area lists only the scopes you can manage, and the proxy checks again on every write — the interface narrowing itself is a convenience, not the gate.

The shape of the admin area

The admin area lives in the chat webapp, at /admin. It has two top-level items: Workspaces, which is everything scoped, and Branding, which is instance-wide (railItems in crab/crab-exoskeleton-webapp/app/admin/admin-nav.ts).

Under Workspaces you choose an agent first, then a scope. That order is deliberate: agents come from the proxy’s configuration and exist before any tenant does, so asking for the scope first implied that agents were a property of a subscription. Every section then acts on that (agent, scope) pair.

The sections are Files, Secrets, Skills, Persona, Model, Config and Members, in that order (SECTION_TABS in crab/crab-exoskeleton-webapp/app/admin/tabs.ts). Which of them a given agent offers is decided in agent-scope.ts, and today a ganglion agent and a picoclaw agent both offer all seven. There is also a legacy “all agents” address holding shared content written before content became per-agent; it offers the content sections only, because a persona write, a model assignment and an invitation all need a real agent to attach to.

The mental model is worth stating once: you edit shared material at a scope, the proxy merges the scopes into an effective view on disk, and each member’s container mounts that view read-only. One member never sees another’s private workspace.

Files

Files uploaded here reach every member’s agent under the scope, as read-only bind mounts at workspace/.shared/<layer> inside the container — one directory per layer, named tenant, subscription, tenant-agent and subscription-agent (sharedFileBinds in crab/crab-shell-proxy/internal/docker/shared.go). The agent can read them; it cannot change them.

Files are the one section whose writes need no container restart. The mount is live, so a new file appears in every running container immediately, and the proxy deliberately does not recreate the container — doing so used to truncate a member’s conversation mid-turn.

Secrets

Secrets are credentials the agents need without them being baked into an image or a template. Four formats are supported (crab/crab-shell-proxy/internal/docker/secrets.go):

The first three are for a skill that reads a credential from somewhere conventional. The native format fills a slot the runtime itself knows about, and only two slot shapes are accepted: web.<provider> for a search provider’s key, and model_list.<model>.api_keys for a model the inventory already knows (validateNativeSlot). Anything else is refused.

A web.<provider> secret is what turns on an agent’s web search. The provider names the two harnesses accept are not identical, so read models and providers before registering one.

Secrets are write-only over the API. You can list the names that exist; nothing reads a value back out.

Skills

A skill is a folder containing a SKILL.md — YAML front matter with a name and a description, then a Markdown body — plus whatever supporting files it needs. Skills and memory explains what they are for.

Publish one at a scope either by writing its SKILL.md inline or by uploading a zip of the folder; the proxy accepts a file part or a body part and refuses a request carrying neither. You can download any published skill back as a zip.

Skills merge across four layers in ascending precedence: tenant, tenant+agent, subscription, subscription+agent (syncEffectiveSkills in crab/crab-shell-proxy/internal/docker/skills.go). A skill folder present at more than one layer is taken from the most specific one, entire — the layers do not merge file by file. The merged result is written into a directory whose inode is kept stable, so an edit reaches running containers without recreating them.

Persona

Four files define an agent’s identity, and the set is fixed and closed (PersonaFiles in crab/crab-shell-proxy/internal/docker/persona.go):

The set is closed because these endpoints write into a workspace root: an arbitrary filename here would be an arbitrary file write reaching every container under the scope.

The first three are resolved per workspace in precedence order — subscription+agent, then tenant+agent, then the agent’s template (resolvePersonaSources) — and mounted read-only. This is precedence, not a merge: two AGENT.md files cannot be combined into one identity. A member cannot edit them, and an edit inside the container never survives a restart.

USER.md is the exception, because the agent writes to it: it is where the agent accumulates what it learns about the member. Setting it here defines what a new workspace starts from, and never overwrites an existing one.

Each row says whether the file is set at this scope or inherited. The editor preloads what the agent actually runs, resolved down the cascade, so you edit a real identity rather than a blank page; saving is what makes it this scope’s. Clearing it lets the workspace fall back to the broader scope, or to the template.

Model

An administrator controls two separate things here: an inventory of models the deployment can serve, and a cascade that decides which of them a given workspace resolves to.

The inventory is one list for the whole proxy, held in one file — model-registry.db under the proxy’s data root. A model is registered once, with its credentials, and every scope points at that record rather than holding a copy. A record carries a provider, a model name used as the handle everywhere else, the identifier actually sent to the endpoint, an API base URL, a write-only API key and that model’s own ordered fallback chain.

The fallback chain is a property of the model, not of a scope: it names other registered models to try after this one. The editor offers only active models as candidates, because the resolver skips a non-active fallback anyway. The order the inventory list is displayed in is presentation only and has no effect on resolution.

A model has one of three statuses. Active is offered normally. Disabled is reversible and requires that nothing references the model — no workspace, no scope default, no other model’s chain — so the proxy refuses with the list of referrers when something does, and the same check blocks deleting a model. Deprecated is for a model people are still using: it requires you to name a replacement, and that replacement is followed only for workspaces that have not already materialised the deprecated model. That single condition is what produces “new members get the successor, existing members keep what they have” without a second code path.

The cascade

Six levels decide which model a workspace gets. Read downwards; each covers fewer people than the one above and overrides it (Resolve in crab/crab-shell-proxy/internal/registry/resolve.go):

global             the whole instance
agent              every tenant running this agent
tenant             one tenant
subscription       one subscription
user (a pin)       one person, set by an administrator
the member's own   one person, registered by themselves

The narrowest level that names a model wins, and that is what a newly provisioned workspace lands on. Levels below a cleared one stay set and take over, which is why the panel draws the whole ladder rather than one level at a time: you can see what your write overrides and what clearing it would fall back to. A level you lack the authority to read is drawn as unreadable rather than as unset — an instance-wide level you cannot see may still be covering your scope.

You may only write the level your scope is sitting on. With a subscription selected you can edit the subscription level and individual pins; with a tenant selected you can edit the tenant level and nothing else (editableLevels in crab/crab-exoskeleton-webapp/lib/models.ts). The agent and global levels are readable here and not writable here — the agent level is labelled with the selected agent’s name but reaches every tenant running that agent, which inside a screen whose rail says “this subscription” reads as something far narrower than it is. The first global or agent default has to be written out of band.

A pin is one person, outranking every administrator-set level above it. Use it for an individual; to move a group, set their scope’s level instead. Pins live under a subscription, and a person appears only once they have a workspace — that is, after their first chat.

A scope default naming a model that no longer exists is skipped and the cascade carries on to the next level, because a stale default imported at boot would otherwise refuse every workspace under that tenant. A pin naming a missing model is not skipped; it is a hard failure, because it was set deliberately.

If nothing resolves at all, what happens depends on the harness. A picoclaw workspace is refused and nothing is written, because picoclaw fails at startup when its default names a model absent from its model list, so a silent default would produce a permanently unbootable container. A ganglion workspace falls back to the model declared in the agent’s own catalog entry, and that fallback is deliberately not recorded as an assignment.

Members’ own models

A member can register models of their own, with their own API keys, and a personal model outranks every level above it, including an administrator’s pin. Their key, their choice — the administrator’s control here is a scope lock, not a per-person veto.

That lock has two independent switches, each of which can be set, cleared, or left to inherit from a wider scope (ScopePolicy in crab/crab-shell-proxy/internal/registry/usermodels.go):

The defaults are deliberately opposite. Personal models are the point of the feature, so they are allowed unless someone objects; a member who cannot name an endpoint cannot aim the instance at one, which is a whole class of risk rather than a governed instance of it. The policy has its own cascade — subscription, tenant, agent, global — and the first level that sets a switch decides it. Flipping a policy restarts nothing; it takes effect the next time each workspace is prepared.

Separately, the User models list lets you disable one member’s one model. It starts enabled: the switch is an intervention, not a gate to pass. Disabling it re-prepares every workspace that was using it and raises a restart notice rather than forcing a bounce. Members have no such switch of their own — their way to stop using a model is to stop selecting it.

A member may register at most ten models, must supply both an endpoint and a key for each, and may only choose from a provider list narrower than the admin inventory’s: the OpenAI-compatible family, with the OAuth and vendor-specific providers deliberately absent.

What happens once a workspace has a model — the fallback chain at turn time, the vision chain, image generation, and which tools appear — is models and providers.

Config

This section reaches the runtime configuration of one member’s instance, or of one key across a whole subscription.

Bulk works on one subscription at a time. You name a dotted path to a single value, read the current distribution first — what each member holds now — and only then write. Instances whose key is absent, blocked by a conflicting path, or whose configuration cannot be read are excluded from the write and listed so you can repair them one at a time.

Keys the proxy owns are refused, because a change there could not survive — the proxy rewrites them every time it prepares a workspace. ManagedConfigPaths in crab/crab-shell-proxy/internal/docker/instance_config.go is the list: the model list, the default provider, model name and fallbacks, the workspace path, the context manager, the pico channel switch and the proxy’s own memory-graph MCP entry.

Single instance opens one member’s configuration as formatted JSON or as a tree, validates it and writes it back.

How that write reaches a ganglion agent differs from picoclaw. A picoclaw workspace’s config.json is seeded once and then edited in place, so an edit simply stays. A ganglion workspace’s configuration is rendered whole every time the container is prepared, so an edit written into the file would be gone on the next turn. The proxy therefore stores the edit in a per-instance overlay beside the file and re-applies it on every render (crab/crab-shell-proxy/internal/docker/ganglion_overlay.go). A broken overlay is logged and skipped rather than treated as fatal.

The overlay is also the only route by which agents.defaults.image_model or agents.defaults.image_gen_model can reach a ganglion agent. The proxy’s renderer never emits either key, so neither appears in the bulk key picker — which is generated from that renderer — and both have to be added by hand in the single-instance editor. The editor diffs the whole document and records every changed leaf that is neither malformed nor proxy-owned, and these two are neither. See models and providers.

A configuration change lands on the instance’s next start. The restart control in the menu decides how that bounce happens: now, the default; schedule, which bounces the scope at a time you pick; or notice, which leaves the moment to the member, who sees a pending-restart notice in chat (crab/crab-shell-proxy/internal/httpapi/restart_policy.go). The change itself always propagates immediately — only the container bounce is deferred.

An instance can also be pinned to a different lifecycle mode than its agent declares. The reason it exists is scheduled tasks that run from timers inside a container: one member who needs a daily report is a reason to keep one container running without forcing the whole agent to.

Members

The Members section lists the people in the selected subscription, with the instances and stored files each of them has. A tenant scope has no member list, because membership is a subscription-level fact.

Inviting is Mycelium’s guest machinery, reached over JSON-RPC; the webapp stores no invitation state of its own. Two facts make it work (crab/crab-exoskeleton-webapp/lib/invitations.ts):

• A guest role’s name is the agent key. The gateway declares protectedByRoles = [{ name = "alpha", permission = "write" }] and Mycelium creates those roles at boot, so “invite someone to agent alpha” is literally “guest them with the role named alpha”.
• The permission lives on the role, not on the invitation call. Choosing read or write is choosing which role id to send. Write is what chatting requires; read is view-only.

The form does not ask which agent, and that is a fix rather than an omission: it uses the agent you selected at the gate. It used to carry an agent picker of its own, three fields deep, which meant two agent selections on one screen and the one that decided access was the invisible one. Revoking is done from the person’s own row in the roster, not from the invite form.

You can see the metadata of a member’s private files — name, size, modified time — and delete one. There is deliberately no way to open, download or preview a member’s private file from here, at any tier. The panel exposes no such affordance and the proxy has no such endpoint.

Branding

Branding is instance-wide and is not scoped to any tenant, agent or subscription. It sets the application name and lets you upload or reset a light logo, a dark logo and an application icon. It appears only for callers who may edit it, because a console offering one thing a caller cannot use reads as a broken screen rather than as an answer about their authority. The proxy is the real gate.

Where to go next

Models and providers is the other half of the Model section. Creating a custom agent covers adding an agent for these sections to administer. Troubleshooting collects the failures people actually report.

Creating a custom agent

This chapter walks through adding a new agent to a deployment, from the first directory to the first chat. It is written as one worked example: an agent called scribe, running the ganglion harness. Read the admin guide first if you have not met tenants, subscriptions and scopes yet.

What an agent is

An agent is a named personality that members chat with. alpha and beta are the two that ship. An agent is not a container: every member who talks to scribe gets their own container, their own workspace and their own history, all cloned from the same starting point. See agents and workspaces for that isolation model.

Three things have to exist before a member can reach a new agent.

1. An entry in the proxy’s agent catalog, which names the agent, the runtime it uses, its lifecycle mode and its default model.
2. A template directory on disk, which supplies the agent’s identity files.
3. A route in the gateway, because Mycelium is the front door and it will not forward a request for a service it has never heard of.

The rest of this chapter is those three, in that order, plus the environment variables that tie them together.

Step 1: the template directory

The proxy resolves an agent’s template at <data-root>/templates/<template>/ (TemplatesDir in crab/crab-shell-proxy/internal/config/config.go). On the host, <data-root> is CRAB_HOST_DATA_ROOT; inside the proxy container the same tree is mounted at CRAB_CONTAINER_DATA_ROOT, which defaults to /data.

For a ganglion agent the part of the template that matters is workspace/, because that directory is the bottom layer of the persona cascade:

data/templates/scribe/
└── workspace/
    ├── AGENT.md       what the agent does and how it behaves
    ├── SOUL.md        its voice
    ├── HEARTBEAT.md   its recurring task list
    └── USER.md        what the agent starts out knowing about the member

Those four names are the complete set. PersonaFiles in crab/crab-shell-proxy/internal/docker/persona.go lists exactly AGENT.md, SOUL.md, HEARTBEAT.md and USER.md, and the first three are delivered as read-only bind mounts while USER.md is seeded once and then left alone — the agent writes to it as it learns about the member.

A ganglion agent has no config.json and no .security.yml in its template. Those are picoclaw’s files. ganglion_config.go states it plainly: the harness’s configuration is not seeded from templates/<agent>/config.json and never was. The proxy renders a configuration file for each workspace instead, and credentials arrive as environment variables. Do not copy a stock agent’s config.json into a ganglion template; nothing will read it.

The template’s workspace/skills/ and workspace/memory/ are likewise picoclaw-only. seedWorkspace in crab/crab-shell-proxy/internal/docker/provision.go copies the config.WorkspaceSeed allowlist — USER.md, memory/ and skills/ — and it is on the picoclaw creation path; createGanglion never calls it. To give a ganglion agent skills, publish them as shared skills from the admin area.

A template: value is still required for every agent, whatever the harness: validate in config.go refuses an agent that declares none. For a ganglion agent it points at the directory holding those identity files.

Step 2: the catalog entry

The catalog is crab/crab-shell-proxy/config.yaml. The Dockerfile copies it to /etc/crab-shell-proxy/config.yaml and sets CRAB_CONFIG to that path, so the committed file is baked into the proxy image. A deployment that mounts its own file over that path, or points CRAB_CONFIG elsewhere, can edit the catalog without rebuilding.

Add the agent under agents::

agents:
  scribe:
    harness: "ganglion"
    serviceName: "scribe"
    token: { env: "MYC_PICOCLAW_SCRIBE_TOKEN" }
    template: "scribe"
    mode: "scale-to-zero"
    idleTimeout: 30s
    model:
      provider: "deepseek"
      name: "deepseek-chat"
      apiKeyEnv: "SCRIBE_API_KEY"

serviceName must match the value Mycelium injects as x-mycelium-service-name, which is the gateway’s service key from step 3. token is the bearer the gateway presents; the proxy rejects any request whose Authorization does not match it.

Declare harness: "ganglion" explicitly. The key is optional and DefaultHarness is the ganglion today (config.go), so omitting it would work — but the same file argues against relying on that: every agent in this repository’s own config.yaml spells its harness out, because a config upgrade should not change an agent’s runtime by omission. Picoclaw is still fully served, and harness: "picoclaw" is still the right value for an agent that needs it; see harnesses for the difference.

mode decides the container lifecycle. scale-to-zero stops the container after idleTimeout of no activity; continuous keeps it running. idleTimeout must be greater than zero when the mode is scale-to-zero, and validate refuses the agent otherwise. A ganglion agent is a good candidate for scale-to-zero: it writes every turn to the transcript before the model is called and rebuilds a missing context window from that transcript, so a stopped container loses nothing.

The model: block is the agent’s floor — the model a workspace runs on when the model inventory resolves nothing for it. You may also list alternatives under models:, which becomes the selectable allowlist; SelectableModels in config.go returns the default followed by that list, deduplicated by provider and name.

Step 3: the image, and what happens without it

A ganglion agent needs CRAB_GANGLION_IMAGE. It has no default, on purpose: the comment on GanglionImage in config.go records that a moving tag once left a deployment running a three-week-old binary for weeks, because the harness image is not a compose service and a redeploy never pulls it. Set it to a digest or a per-commit tag.

When it is unset the agent does not take the proxy down. ganglionUnprovisioned removes the agent from the catalog at load, records it in DisabledAgents with the name of the missing setting, and the agent’s routes then answer 404. The same happens when the agent’s apiKeyEnv resolves to an empty value, or when its token environment variable is unset. Read the proxy’s boot log if a new agent seems not to exist: the reason is there, and it names the variable.

docker-compose.yaml defaults CRAB_GANGLION_IMAGE to zombie-crab/crab-ganglion:dev, which is a locally built tag. docker-compose.prod.yaml sets no ganglion image at all and no workflow in .github/workflows/ publishes one, so a production deployment has to build and push the harness image itself before a ganglion agent can start.

Step 4: the gateway route

Mycelium routes by the first path segment, and that segment is the literal service key. Copy the [[alpha]] block in deploy/standalone/config.standalone.toml — the service block, its [[alpha.secret]] and every [[alpha.path]] — and rename it to scribe. Callers then reach the agent at /scribe/.... Keep host, healthCheckPath, the full path set and the protectedByRoles groups as they are; only the name changes.

Do it in every mode you deploy, because they are separate files: deploy/standalone/config.standalone.toml and deploy/prod/config.base.toml.

The protectedByRoles entries also declare the guest role. A guest role’s name is the agent key — the gateway declares protectedByRoles = [{ name = "alpha", permission = "write" }] and Mycelium creates those roles at boot, which is the fact crab/crab-exoskeleton-webapp/lib/invitations.ts is built on. So a new agent brings its own role into existence, and inviting someone to it is then an ordinary invitation from the Members section of the admin area.

Step 5: the environment

Two variables, in .env and never in a config file:

MYC_PICOCLAW_SCRIBE_TOKEN=<the same shared secret the gateway route uses>
SCRIBE_API_KEY=<the provider API key>

The token authenticates the gateway to the proxy. The API key never reaches a file the agent can read: for a ganglion agent the proxy passes it as GANGLION_API_KEY, or as one GANGLION_MODEL_KEY_<NAME> variable per model when the inventory governs the workspace. Models and providers covers that naming.

Step 6: restart, and the first chat

The gateway configuration is mounted from deploy/<mode>/, so a route change needs a restart. The agent catalog is copied into the proxy image, so a catalog change needs a rebuild — unless your deployment mounts its own catalog, in which case a restart is enough. Locally, the safe catch-all is to rebuild both:

docker compose up -d --build crab-shell-proxy mycelium-gateway

The first time a member with the scribe role sends a message, the proxy creates their workspace: it makes workspace/ with its memory/, public/, sessions/ and windows/ subdirectories, mints a per-user bearer token for the container, seeds USER.md from the persona cascade if anything provides one, renders the harness’s configuration file and starts the container.

Files the agent delivers back to the member go in public/attachments/. See files and delivery.

Checklist

• data/templates/scribe/workspace/{AGENT.md,SOUL.md,HEARTBEAT.md,USER.md}
• Catalog entry in crab/crab-shell-proxy/config.yaml with harness: "ganglion", serviceName, token, template, mode and model
• CRAB_GANGLION_IMAGE set to an immutable reference
• [[scribe]] service block in the gateway config of every mode you deploy
• MYC_PICOCLAW_SCRIBE_TOKEN and the model’s apiKeyEnv in .env
• Proxy rebuilt or its catalog remounted, gateway restarted
• A member invited to the scribe role from the Members section

When it does not work

The agent’s routes answer 404. Either the gateway has no scribe service — check that you edited the file the mode you are running actually mounts — or the proxy disabled the agent at load. The boot log names which.

Requests are rejected as unauthorized. MYC_PICOCLAW_SCRIBE_TOKEN in the proxy’s environment and the token = { env = ... } in the gateway’s secret block must resolve to the same value.

The member chats but the agent has no personality. GANGLION_SYSTEM_FILE points at workspace/AGENT.md inside the container, and that file arrives as a read-only bind from the persona cascade. If no layer of the cascade provides AGENT.md — not the subscription, not the tenant, not the template — no bind is emitted and the agent runs with no identity. Check that your template’s workspace/AGENT.md exists.

A template edit did not reach an existing member. AGENT.md, SOUL.md and HEARTBEAT.md are re-resolved on every ensure, so an edit does reach them. But USER.md is seeded only when the workspace has none, deliberately: it is the file the agent writes back, and overwriting it would erase what the agent has learned. A new value for it reaches new workspaces only.

Where to go next

Models and providers explains how the model in your catalog entry relates to the inventory an administrator manages, and which tools an agent gets. The admin guide covers inviting members and giving the new agent shared skills and shared files.

Models and providers

This chapter explains how a turn ends up talking to a particular model, what happens when that model fails, and why two members of the same deployment can see different tools in the same agent. It is written for the ganglion harness, which is the one this book teaches; see harnesses for the other one.

Two systems are involved and it helps to keep them apart. The proxy decides which models a workspace has — that is the inventory and the cascade described in the admin guide. The harness decides, turn by turn, which of those models actually answers. This chapter is about the second half, plus the plumbing that carries credentials between the two.

How the harness learns about models

Every time the proxy makes sure a member’s container is ready, it resolves that workspace’s model from the inventory and writes a configuration file into the member’s directory: .ganglion-config.json. The file is bound into the container read-only at /data/.ganglion/config.json, and GANGLION_CONFIG_FILE points the harness at it (crab/crab-shell-proxy/internal/docker/ganglion_config.go).

It is read-only, and it sits above the one directory the container can write, for a specific reason: a model list the agent could edit would let a tool steered by untrusted text choose the endpoint its own API keys are sent to.

The file’s shape is picoclaw’s config.json, deliberately, so that one admin screen manages both harnesses. The harness reads model_list, agents.defaults.model_name, agents.defaults.model_fallbacks, agents.defaults.image_model, agents.defaults.image_gen_model, tools.web.* and tools.mcp, and ignores everything else it finds. A missing file is not an error: the harness synthesizes a one-entry list named default from three environment variables and behaves as it did before the file existed (LoadRegistry in crab/crab-ganglion-harness/internal/config/file.go).

The three chains

A turn asks the registry for an ordered list of candidate models. Which list it gets depends on the kind of turn (Kind in the same file):

The text chain

Chain builds the candidate list as follows. If the turn named a model the registry actually knows, that model goes first. Otherwise the list starts with model_name followed by model_fallbacks. Then — for every entry already in the list — that entry’s own fallbacks are appended, one level deep. Entries that are disabled or unknown are dropped, and duplicates are collapsed.

A turn naming a model the registry does not have is the ordinary case, not an anomaly: the proxy fills the turn’s model field with a placeholder. So the harness resolves it silently to the default chain rather than reporting anything to the member.

The one-level rule is worth noticing. A fallback’s fallbacks are not walked recursively, and the expansion iterates over a snapshot, so a cycle in the configuration cannot hang a turn.

An administrator does not edit this file to change the chain. model_fallbacks is written by the proxy from the chain declared on the inventory record of the model that resolved, so the chain is edited in the Model section of the admin guide.

The vision chain

A turn is a vision turn when any message in the context window carries an attachment (hasAttachments in crab/crab-ganglion-harness/internal/runtime/loop.go).

If image_model names an entry, that chain answers. If it does not, the registry returns the text chain instead. That is not a courtesy: whether a model can see is a property of the model, not of a slot, so a deployment whose only model happens to be multimodal needs no second entry and should not be made to write one.

The image-generation chain

This one does not fall back. If image_gen_model names nothing, the chain is empty. The reason is stated in the code: a text model asked to generate an image returns prose describing one, which is worse than an absent tool because it looks like success.

The consequence is the next section.

The proxy’s renderer writes model_name, model_fallbacks, model_list, tools.web and tools.mcp — and not image_model or image_gen_model (ganglionConfigDoc in crab/crab-shell-proxy/internal/docker/ganglion_config.go). For a ganglion agent those two keys reach a workspace only by being added by hand in the single-instance editor of the Config section, which stores them in the per-instance overlay. Neither is a proxy-owned key, so the overlay keeps them and re-applies them on every render. A vision or image-generation chain is therefore per member on this harness, not something a scope can set.

What a fallback means at turn time

A chain is not a retry loop around the whole turn. completeWithFallback in loop.go states the rule: a candidate is abandoned only while nothing has reached the member. Once a single byte of content has been emitted, the turn is committed to that model and its failure surfaces — restarting under another model would splice two voices into one reply, and the member has already read the first half of the first one.

When the chain runs out, the error the member sees is the last provider’s, not a synthetic summary. An operator needs the reason the final attempt failed, and “all 3 models failed” would bury it.

There is one degradation on top of that, and it is a feature rather than politeness. If the whole chain fails on a turn that carries an image, the image is dropped and the turn is retried once, text-only, with the model told what happened. The member sees a progress note saying the image could not be read and gets a degraded answer that says so.

The reason this exists is that the alternative is permanent. The media reference stays in the conversation history, so a harness that simply failed would fail the same way on every later turn in that conversation. This stack met exactly that in production.

Which agent tools exist

This is the question members actually ask — “why can’t my agent search the web”, “why can’t it make me an image” — and the honest answer is that the tool registry is conditional. A tool whose prerequisites are not configured is not present at all. It is not present-and-failing, because a tool the model is told about and that can never answer is worse than no tool: it spends a turn discovering the absence.

tools() in crab/crab-ganglion-harness/cmd/crab-ganglion/main.go is the whole decision:

load_image is unconditional on purpose: an image in the workspace is something any deployment can have. Whether a model can see the result is decided by the vision chain at completion time, not here.

research needs both a dispatcher and search, because without search it is a model asked to recall — which is the failure it exists to replace.

The harness logs each of these decisions at boot. If a member reports a missing capability, the container’s first few log lines say which tools were enabled and which were not.

Why search in particular is easy to get wrong

Two lists of provider names have to agree and they do not fully overlap.

The proxy accepts a native shared secret at slot web.<provider> for these: brave, tavily, kagi, gemini, perplexity, glm_search and baidu_search (webProviders in crab/crab-shell-proxy/internal/docker/secrets.go). Every one of them is written into the ganglion’s configuration as tools.web.<name>: {enabled: true} with its key passed separately.

The harness implements four: brave, tavily, searxng and duckduckgo, in that preference order when tools.web.provider is unset (WebProviderNames in file.go). Anything else in the file is ignored.

So of the providers an administrator can register through the Secrets section, only brave and tavily will actually give a ganglion agent a search tool. Registering a kagi or perplexity key produces a configuration that looks correct and yields no tool.

Readiness differs per provider too (providers.go): brave and tavily need a key, searxng needs a base_url and no key, and duckduckgo needs only to be enabled. A provider block with no enabled: true is off — declaring "brave": {} leaves it present and disabled, which matches picoclaw’s own examples.

Provider keys

Credentials never travel in the configuration file. The proxy splits structure from secrets the way picoclaw does: the file carries endpoints and names, the environment carries keys, one variable per model.

There are three places a key can come from.

The agent’s own key. Each agent in crab/crab-shell-proxy/config.yaml names an apiKeyEnv, and the proxy reads that variable from its own environment. For a ganglion agent this becomes GANGLION_API_KEY, alongside GANGLION_MODEL and GANGLION_BASE_URL. This trio is the floor: what a workspace runs on when the inventory resolves nothing for it.

If that variable is unset, a ganglion agent is disabled at load. The proxy records the reason, naming the variable, and the agent’s routes answer 404 rather than the proxy refusing to boot. A picoclaw agent is deliberately not subject to this — its key is written into a per-user file and an empty one surfaces as an auth error on the first model call.

An inventory model’s key. When the cascade resolves a model from the inventory, its key is passed as GANGLION_MODEL_KEY_<NAME>, one variable per model in the chain. The name is derived by upper-casing the model name and replacing everything outside A-Z0-9 with an underscore. Both sides compute it independently — ganglionModelKeyEnv in the proxy and KeyEnvVar in the harness — and the two agreeing is the entire contract. Two model names differing only in punctuation collide, which is accepted: the alternative is an encoding nobody can read in docker inspect output, which is where these get debugged.

A search provider’s key. Same scheme, different prefix: GANGLION_WEB_KEY_<PROVIDER>, from the native shared secret at slot web.<provider>.

The harness resolves a key from the environment first and only then from the file. The environment wins because that is the path the proxy uses, and because a key that never enters a file cannot be read by anything that gets pointed at the file by mistake.

A model in a chain whose key is empty is simply not offered to the image-generation tool — imagegen.New filters candidates on a non-empty key and returns no tool when none survive. On the text chain the effect is different: the candidate is attempted and fails at call time, and the chain moves on.

Keys may also be stored encrypted. A value beginning enc:// is resolved inside the container from two factors that arrive by different routes — a passphrase in GANGLION_KEY_PASSPHRASE and a key file bound read-only from the host. They are deliberately of different kinds: both as environment would mean one docker inspect yields the plaintext.

Where a model’s endpoint comes from

The ganglion is given a base URL and posts to it; it has no internal table mapping a provider name to an address. Picoclaw did have one, so an agent migrated from picoclaw arrives naming a provider, naming no endpoint, and used to fail on its first turn with unsupported protocol scheme "".

The proxy fills the gap from three sources, in order (resolveGanglionEndpoints in crab/crab-shell-proxy/internal/docker/ganglion.go):

1. The inventory record’s own api_base, which is final. A custom model is custom precisely because its endpoint is not its provider’s default.
2. The agent’s baseUrl from config.yaml.
3. The provider’s default, from an embedded catalog of about thirty provider/model pairs (ProviderEndpoint, model-catalog.json).

Fallback entries are filled too, not just the primary — a chain whose second entry has no endpoint is a chain that works until the day it is needed. A primary with no endpoint anywhere is refused where an operator can see it.

Two caveats from provider_endpoint.go. Azure’s catalogue entry is a fill-in-the-blank shape rather than an address, so it is excluded from the fallback table. And the local runtimes — ollama, lmstudio, vllm, github-copilot — have localhost entries, which inside a container means the container, not the host; using one of those means setting an explicit baseUrl.

Reasoning depth

A model entry may declare thinking_level, one of off, low, medium, high, xhigh or adaptive. Declaring the key is the capability declaration: a model with no thinking_level is never sent a depth field by any path, because the harness cannot discover whether an endpoint accepts one and the operator can.

An unrecognised value is treated as absent and produces a warning, rather than being read as off. That distinction matters: a typo that silently meant “think less” would look like a working configuration right up to the bill.

If a model rejects a reasoning field mid-chain, the field is removed and the same model is asked once more, rather than the chain burning its budget on a field nobody asked for.

Where to go next

The admin guide covers registering models and choosing who gets which. Creating a custom agent shows where an agent’s default model is declared. Troubleshooting collects the symptoms these mechanisms produce.

Deployment

This chapter is for whoever runs the stack on a machine. It describes the two deployment modes that exist, what each one is for, and the exact command that brings each up. It also states plainly where production is not finished yet, so that you find out here rather than from a failed container.

What a mode is

The stack is a set of Docker Compose services. A mode is a combination of compose files and one .env file at the repository root. There is one base file that always participates, docker-compose.yaml, and overlays that change some of its services without replacing them.

Two modes ship as profiles, each with its own directory under deploy/ holding that mode’s .env.example and the gateway configuration it mounts:

A third file, docker-compose.observability.yaml, is not a mode. It is an opt-in overlay that gives the watcher somewhere to send its metrics; it is described in Observability.

Both profiles pin the same Mycelium release. deploy/standalone/.env.example builds the commit 9298ecb44a69ced91cb2d7d3356a716aedca858b, described there as the commit tagged 9.0.0-rc.13, and deploy/prod/.env.example pulls MYCELIUM_IMAGE_TAG=9.0.0-rc.13. Move them together. The gateway configuration is shared vocabulary between the two, and a version skew between what you test and what you deploy is where it breaks.

Development on a laptop: standalone

Copy the profile’s environment file to the repository root and edit it:

cp deploy/standalone/.env.example .env

The values you must set before the first up are the per-agent bearer tokens (MYC_PICOCLAW_ALPHA_TOKEN, MYC_PICOCLAW_BETA_TOKEN), each agent’s own LLM key (PICOCLAW_ALPHA_API_KEY, PICOCLAW_BETA_API_KEY) and MYC_STANDALONE_BOOTSTRAP_SECRET, which gates the one-time Staff claim. A bearer token is what Mycelium injects on a request for that agent and what crab-shell-proxy validates; the LLM key is read from the proxy’s environment and never written into an image or a committed file.

Then:

docker compose up -d --build

Standalone builds rather than pulls. Everything under crab/ — the proxy, the chat webapp, the ganglion harness, the watcher — is built from your working tree, which is the point of the mode: what runs is what you have checked out. Mycelium is the exception. fungi/mycelium/Dockerfile.standalone builds mycelium-api from the upstream git repository at the pinned MYCELIUM_GIT_REF with no local source, and fungi/mycelium-webapp does the same for its admin UI. That is also why the pin and prod’s MYCELIUM_IMAGE_TAG have to move together.

deploy/standalone/.env.example deliberately carries no image tags, because CRAB_SHELL_PROXY_TAG and CHAT_WEBAPP_TAG do nothing in a mode that builds.

The two build-only services

Two services in docker-compose.yaml are not servers. picoclaw-image and ganglion-image each build one image, run /bin/true, and exit zero; crab-shell-proxy declares condition: service_completed_successfully on both, so the images are guaranteed to exist before anything can spawn a container from them.

They exist because the agent containers are created by crab-shell-proxy over the Docker socket, not by Compose, so Compose would otherwise never build or fetch them. Their absence is also what one of the failures in Troubleshooting is about.

ganglion-image’s build runs go vet and go test before it links the binary. A failing test therefore stops the stack from coming up. docker-compose.yaml states that this is the intent for a development compose file: the tests are the image’s acceptance check.

Resetting to zero

To wipe every per-user agent and all templates and let the stack rebuild itself:

docker compose down
docker rm -f $(docker ps -aq --filter 'name=crabshell') 2>/dev/null
sudo rm -rf data/templates data/tenants data/effective-secrets \
            data/effective-skills data/user-secrets data/registered-models
docker compose up -d --build

The docker rm line is needed because the per-user containers were spawned by the proxy rather than by Compose, so docker compose down does not know about them. The sudo is needed because the on-disk tree under data/ is written by the proxy as root. The --build is not optional: the fallback template the proxy re-seeds a wiped data/ from is embedded in the proxy binary.

Accounts and roles are not in data/. They live in named volumes — mycelium-data for Mycelium’s own SQLite database and chat-webapp-postgres-data for the conversation list — so your login survives the wipe. Adding -v to docker compose down resets those too, and you would then re-run the Staff bootstrap. See Database and migrations.

Production: the prod overlay

cp deploy/prod/.env.example .env
docker compose -f docker-compose.yaml -f docker-compose.prod.yaml up -d

The overlay changes four services. mycelium-gateway, crab-shell-proxy, chat-webapp and harness-sphere each get build: !reset null and an image: pointing at a published GHCR image. The !reset matters: a plain image: next to an inherited build: would still build the image locally if it happened to be missing, and the whole posture of this mode is that these four are pulled, never built.

!reset requires Docker Compose v2.24 or newer. This is a prerequisite for the prod mode, not a detail — an older Compose will not understand the file.

Two more things the overlay does are easy to miss and both are deliberate:

• crab-shell-proxy gets ports: !reset []. The base file publishes the proxy on 127.0.0.1:18080 for direct testing, and ports concatenates across -f layers rather than replacing, so without the reset that port would stay live. In production the gateway must be the only entry point, loopback included.
• mycelium-webapp is still built here. It is a browser-side single-page app whose API URL is baked in at build time through the VITE_MYCELIUM_API_URL build argument, so it cannot be a generic prebuilt image.

Production has no published ganglion image today

This is the one thing to be explicit about, because a reader who deploys to a server and configures a ganglion agent can hit it.

docker-compose.prod.yaml does not set CRAB_GANGLION_IMAGE, and deploy/prod/.env.example does not mention it either. Because prod is an overlay on the base file, what actually reaches the proxy is the base file’s default, zombie-crab/crab-ganglion:dev — a tag that exists only on a machine that built it. The agent is therefore not disabled; it is pointed at a name no registry resolves.

Two consequences follow. First, the prod command inherits the ganglion-image build-only service unchanged, since the overlay resets only the four services above. A prod up -d on a host that carries the submodule sources will build the ganglion locally, which is not what “published images” leads you to expect and which requires the source tree and a Go build on that host. Second, in any situation where that build-only service does not run — a docker compose up -d crab-shell-proxy on its own, a docker system prune that removed the local tag, or a deployment that only pulls — the proxy’s EnsureImage finds nothing locally, falls through to a registry pull, and that pull 404s. Every ganglion agent is dead until someone builds the image by hand.

Immutable ganglion images do exist. crab/crab-ganglion-harness’s own .github/workflows/release.yml publishes ghcr.io/lepistabioinformatics/crab-ganglion:sha-<short-sha> on every push to main, deliberately with no :latest and no tag that is ever rebuilt. What is missing is the wiring: nothing in the prod profile points at one. The only release workflow in this repository’s .github/workflows/ is release-picoclaw-glob.yml, which publishes the patched picoclaw image.

Until the profile catches up, set CRAB_GANGLION_IMAGE in your production .env to a specific sha- tag or a digest, and docker pull it before you bring the stack up. crab/crab-shell-proxy’s internal/config/config.go asks for an immutable reference and explains why: a moving tag once left a host running a three-week-old binary, silently, because EnsureImage never pulls what is already present.

The pull is not optional housekeeping. The inherited ganglion-image service takes its image: from whatever you set while keeping its build: context, so if that reference is absent locally, up will build the checked-out submodule source and tag those bytes with the published name — which is the exact wrong-bytes-under-a-trusted-name failure the immutable reference exists to prevent.

Note also that an agent which declares no harness: key at all is a ganglion agent today (DefaultHarness in crab/crab-shell-proxy/internal/config/config.go), so it needs CRAB_GANGLION_IMAGE as well. Declare the harness explicitly on every agent; the config shipped in this repository does, for that reason.

Before a prod deploy

• CRAB_HOST_DATA_ROOT must be an absolute host path. crab-shell-proxy hands it to the host Docker daemon as the bind-mount source for the containers it spawns, so a path that only exists inside the proxy will not resolve.
• Set noreplyEmail and supportEmail in deploy/prod/config.base.toml to the same address as MYC_SMTP_USERNAME. Gmail rejects a mismatched From. The SMTP port is fixed at 465 in that file rather than being an environment variable, because Mycelium parses it as a number and an environment value is a string.
• If you front the stack with a hostname, change domainUrl and allowedOrigins in the same file together with mycelium-webapp’s VITE_MYCELIUM_API_URL build argument. The admin UI calls the gateway directly from the browser, so a mismatch between the two is a CORS wall.
• The agent catalog is baked into the proxy image in both modes (crab/crab-shell-proxy/config.yaml), so adding or dropping an agent means rebuilding it. Binding your own file over /etc/crab-shell-proxy/config.yaml is supported if you would rather mount it.
• Every routed agent needs its bearer token set. Leaving one empty makes the gateway advertise the agent and inject an empty bearer, which fails at request time rather than at boot.

There is no Dokploy profile

There used to be a third profile, docker-compose.dokploy.yaml with its own deploy/dokploy/ directory, and it was removed rather than moved. The deployment it was written for now lives in a separate repository, deploys only the CRAB half of the stack, and had drifted into being the real source of truth while this copy was maintained on faith.

If you are deploying on Dokploy, start from prod, which is the closest profile, and add the Traefik labels and external network your installation needs. git log -- docker-compose.dokploy.yaml deploy/dokploy/ still has the original.

Where to go next

Production on Postgres needs a one-time schema step before anyone can sign in — that is Database and migrations. Once the stack is up, Observability covers the watcher and the optional metrics backend, and Troubleshooting collects the failures people actually run into.

Database and migrations

Most of what this stack persists is not in a database at all, and most operators never run a migration. This chapter says which services do have one, what each stores, and the single one-time schema step that exists — so you can tell quickly whether it applies to you.

Which services store what

Mycelium, the gateway, owns identity. Accounts, tenants, subscriptions, guest roles and the magic-link tokens behind sign-in all live here. Where depends on the deployment mode. In standalone it is SQLite: deploy/standalone/config.standalone.toml sets [sqlite] path = "/data/mycelium.db", and docker-compose.yaml mounts the named volume mycelium-data at /data. In prod it is Postgres: the overlay adds a mycelium-postgres service on postgres:16-alpine, backed by the mycelium-postgres-data volume, and points the gateway at it through MYC_BASE_DATABASE_URL.

chat-webapp owns the conversation list. chat-webapp-postgres, also postgres:16-alpine on the chat-webapp-postgres-data volume, holds one small set of tables: conversations (id, owner email, agent, title, workspace ids, session file, project), conversation_tags, and a single-row branding table for the app name and logos. It is deliberately a separate database from Mycelium’s, with a different lifecycle; the compose file argues the point at the service definition.

crab-shell-proxy owns a small key-value store. At boot it opens model-registry.db under its container data root (crab/crab-shell-proxy/cmd/crab-shell-proxy/main.go). This is a bbolt file, not SQL: internal/registry/registry.go’s Open creates every bucket it needs if they are missing and carries its own schema_version marker with a boot migration. It needs nothing from you.

Everything an agent produces is files, not rows. Transcripts, memory files, projects, skills, delivered attachments and the proxy-owned .schedules.json/.projects.json all live on disk under the data root, in the tenant tree described in Agents, workspaces and projects. That is why a reset is a rm -rf and not a DROP.

deploy/prod/config.base.toml also carries a [redis] block, with hostname = "mycelium-redis" and password = "unused-no-redis-container-in-this-stack". The configuration format requires the keys to exist; no Redis container runs in this stack. Do not go looking for it.

When you need to do anything at all

Three of those four need no action.

Mycelium’s SQLite adapter carries embedded migrations — deploy/prod/config.base.toml says so where it explains that the Postgres one does not — so standalone works from a clean checkout with no schema step.

chat-webapp creates its own schema lazily, at runtime, on the first query. crab/crab-exoskeleton-webapp/lib/db.ts holds one ensureSchema() function that issues CREATE TABLE IF NOT EXISTS and ALTER TABLE ... ADD COLUMN IF NOT EXISTS statements once per process and memoizes the promise. Every statement is idempotent, and new columns are added additively so pre-existing rows survive. There is no migration tool, no migrations directory and no command to run.

The model registry initializes itself, as above.

That leaves exactly one case: Mycelium’s Postgres backend, in prod, once, after the first up.

The one-time schema step, prod only

Mycelium’s Postgres adapter has no embedded migrations, unlike its SQLite one. deploy/prod/config.base.toml records this directly above its [diesel] block, and docker-compose.prod.yaml repeats it in the file header. Until the schema is applied, the gateway has a database it cannot use, so nobody can sign in.

It is two steps, in this order: upstream’s up.sql, then the migration scripts that up.sql does not fold in. Both come from the mycelium repository at the same release this deployment pins.

git clone --depth 1 --branch 9.0.0-rc.13 \
  https://github.com/LepistaBioinformatics/mycelium.git /tmp/myc
cd /path/to/zombie-crab-project && set -a; . ./.env; set +a

Sourcing .env is what puts MYC_DB_USER, MYC_DB_NAME and MYC_DB_PASSWORD into your shell; those are the same three names docker-compose.prod.yaml hands the mycelium-postgres service, so they will match whatever you set in deploy/prod/.env.example.

Step 1 — the base schema.

docker compose -f docker-compose.yaml -f docker-compose.prod.yaml exec -T mycelium-postgres \
  psql -U "$MYC_DB_USER" -d postgres \
       -v db_name="$MYC_DB_NAME" -v db_user="$MYC_DB_USER" \
       -v db_password="$MYC_DB_PASSWORD" -v db_role=service-role-mycelium \
  < /tmp/myc/adapters/diesel_postgres/sql/up.sql

The connection targets the postgres maintenance database because a database cannot be created from inside itself. The root README.md describes what the script then does — create the application database if it is missing, switch into it, and create the roles and tables — and records that it requires -v db_password. The -v flags are psql variables the script substitutes into its own SQL.

Compose has already created the database and the login role by the time you run this, because the mycelium-postgres service declares POSTGRES_DB and POSTGRES_USER. So CREATE USER ... already exists is expected output here, not a failure: psql prints it and keeps going.

Step 2 — the migrations, in filename order.

for m in /tmp/myc/adapters/diesel_postgres/sql/migrations/*.sql; do
  docker compose -f docker-compose.yaml -f docker-compose.prod.yaml exec -T mycelium-postgres \
    psql -U "$MYC_DB_USER" -d "$MYC_DB_NAME" < "$m"
done

The glob expands in lexical order, which is the order these scripts expect. Note the -d here is the application database, not postgres.

This step is not optional at 9.0.0-rc.13. The repository’s own notes — deploy/prod/config.base.toml above [diesel], and the root README.md — record that up.sql at this tag ships kv_artifact and the message_queue claim index but not instance_settings, resource_audit_log, or the tenant.encrypted_dek and kek_version columns that envelope encryption needs. Those exist only as migration scripts.

To check the result, connect and look:

docker compose -f docker-compose.yaml -f docker-compose.prod.yaml exec mycelium-postgres \
  psql -U "$MYC_DB_USER" -d "$MYC_DB_NAME"

\dt should list instance_settings and resource_audit_log; \d tenant should show encrypted_dek and kek_version.

The two commands above are the ones the root README.md gives, and the service name, the MYC_DB_* variable names and the compose file chain in them all match this checkout. What this repository cannot verify on its own is everything inside the mycelium clone: the paths adapters/diesel_postgres/sql/up.sql and adapters/diesel_postgres/sql/migrations/, the db_role=service-role-mycelium value, what up.sql does when it runs, and which tables that tag’s up.sql does and does not create. All of that comes from the upstream repository. If a path has moved in a later release, read the clone rather than this page.

On a Postgres deployment not driven by this repository’s compose file, run the same two steps with docker exec against the mycelium-postgres container directly. Nothing about the SQL changes; only how you reach psql does.

What a reset does and does not touch

The reset described in Deployment removes directories under data/. It does not touch the named volumes, so Mycelium’s accounts and roles and chat-webapp’s conversation list both survive it, and your login still works. That is deliberate: wiping agent state is a routine development action, and losing your Staff account each time would make it much less routine.

Adding -v to docker compose down removes the volumes as well. On standalone that deletes mycelium.db, so you would re-run the Staff bootstrap from scratch. On prod it deletes the Postgres data directory, so you would re-run the two schema steps above as well.

Where to go next

Deployment covers the modes these databases belong to, and Troubleshooting covers what a missing schema or a wiped volume actually looks like from the chat client.

Observability

harness-sphere is the stack’s watcher. This chapter says what it watches, what it shows you, how to run it with and without a metrics backend, and the two rules about it that an operator must not break by accident.

What it is, and what it is not

Before this service existed, the stack emitted nothing — not “not enough”, but nothing: no metrics endpoint, no collector, no dashboard. The first question anyone asks during an incident, was the box under pressure when that got slow?, had no answer.

harness-sphere is a single Rust binary that turns what it finds into standard OpenTelemetry metrics and ships them to whatever backend you point it at. It is exclusive to this stack, by its own README’s first line: collectors with no source here were deleted rather than left configurable.

Three things it deliberately never does, and each is worth knowing before you go looking for the feature:

• It holds no Docker socket. See the rules section below.
• It reports no token cost. picoclaw writes no token counts to disk and exposes no metrics endpoint; the only path that ever existed scraped an endpoint this stack does not run, and it was deleted. This is not deferred work.
• It reads no transcript content. Message bodies are member data. It counts messages, sessions and tool calls; content is not even deserialized.

What it watches

The stack is modelled as exactly six layers: the host, the watcher itself, and the four things the machine runs. Two of them — Host and Watcher — are Critical, meaning the watcher exits non-zero rather than run blind without them. Everything else is Optional: if it is missing or misbehaving it steps aside quietly and never takes the process down.

The host. CPU utilization, memory usage and utilization, and swap. These are the real machine’s numbers, not the container’s, and that was measured rather than assumed — Linux does not namespace /proc, so sysinfo inside the container reads the host. A container capped at -m 512m still reported the host’s 33 GB. This is exactly why the service mounts no /proc and no /sys.

Itself. Its own CPU, resident memory and virtual memory. A watcher whose memory grows without bound is a watcher about to become the incident.

Three service endpoints, by TCP probe. crab/harness-sphere/config.zombie-crab.toml lists them by compose service name, each tagged with the layer it belongs to: mycelium-gateway:8080 (gateway), crab-shell-proxy:8080 (proxy) and chat-webapp:3000 (webapp). Note the third: the repository is called crab-exoskeleton-webapp, but chat-webapp is the compose service and therefore the only name that resolves on zombie_net.

A target that is down reads an honest 0 rather than going absent. That is also why the service declares no depends_on — ordering the watcher behind the health of what it watches would suppress the exact signal it exists to produce.

Per-member AI activity, from the workspace tree. It globs tenants/*/subscriptions/*/agents/*/users/* under the read-only /data mount and runs one session collector per (tenant, subscription, agent, user) tuple, each stamped with that tuple. From the on-disk JSONL transcripts it derives message counts by role, conversation counts, tool-call counts, and scheduled-task runs counted separately. Four distortions are corrected, each with a test behind it: project conversations live in a sibling workspace-<id>/sessions, sessions/durable/ mirrors the live files one-to-one and would exactly double every number, every cron run writes its own session file, and a transcript that shrank is re-read from zero rather than treated as a negative delta.

The intervals are in the same config file: discovery every 30 s, sessions every 60 s, learning material every 300 s, host every 10 s, self every 30 s.

These are Gauges, not Counters. The collector reports the absolute total it finds on disk, re-derived on each scrape, so pushing it through a Counter’s add() would double-count every tick. The consequence to expect is that a value can legitimately fall when transcripts are rotated away.

What it cannot see yet

The watcher reads the tenant tree on disk. It does not yet consume crab-shell-proxy’s live GET /v1/instances inventory, so it can tell you a member’s workspace exists and how much activity is in it, but not whether that member’s container is currently running. Per-instance liveness and per-container CPU and memory both depend on that inventory. harness-sphere never computes the container-name hash itself, because that would duplicate a preimage the proxy owns and silently diverge the day the prefix or the hash changes.

Running it

The watcher is already in the base compose file, so it runs with the stack. Out of the box its exporter is stdout: it prints its signals into its own container log. That is a deliberate default — it proves the entire pipeline with no backend to stand up.

docker compose logs -f harness-sphere

To actually look at numbers over time, bring up the opt-in overlay, which adds an OpenTelemetry Collector, Prometheus and Grafana:

docker compose -f docker-compose.yaml -f docker-compose.observability.yaml up -d

Grafana is then on http://localhost:3001 (GRAFANA_PORT), with anonymous admin access and no login form — it is a local backend on a loopback port, and a login prompt in front of your own CPU graphs has no threat model behind it. If you ever expose it beyond localhost, those three GF_AUTH_* lines are the first thing to remove. Prometheus is published on 9090 so you can run a raw PromQL query when a dashboard disagrees with what you expect.

Two dashboards are provisioned from deploy/observability/grafana/dashboards/, along with the Prometheus datasource, so the stack is useful on first boot with no clicking: zombie-crab — stack, organised by layer, answers “is the stack healthy?”, and zombie-crab — learning, organised by member, answers “where does each instance sit?”.

Query the underscore names. The OTLP-to-Prometheus translation rewrites dots to underscores and appends a unit suffix, so what the watcher emits as system.memory.usage is scraped as system_memory_usage_bytes. Declare instruments with the dotted names; query with the translated ones.

Why there is a collector in the middle

harness-sphere’s OTLP exporter is built with .with_tonic(), so it speaks OTLP over gRPC, while Prometheus’s own OTLP receiver is HTTP-only. The two cannot be wired together directly. The collector receives gRPC on 4317 and re-exposes everything in Prometheus exposition format on 8889, which is the single target deploy/observability/prometheus.yml scrapes.

Its HTTP receiver on 4318 is there for a second producer: crab-ganglion-harness speaks OTLP over HTTP with JSON encoding, which it can do with the Go standard library alone, and the compose file passes GANGLION_OTLP_ENDPOINT (default http://otel-collector:4318) through the proxy into each agent container. Setting it empty disables export inside the harness rather than making it log a failed request every turn.

The pipeline is metrics only. harness-sphere boots sources=3 receivers=0 — no traces, no logs — which is also why the SigNoz stack vendored under crab/harness-sphere/deploy/signoz/ is not what this overlay uses: six services including ClickHouse and Zookeeper, sized for a signal that does not exist yet.

Two settings that will bite you

The -f chain is not optional on any command. Compose applies an overlay only when you name it. Run docker compose up -d, or restart, or up -d harness-sphere without both files, and every service the overlay overrides silently falls back to the base file — the exporter returns to stdout and the config bind-mount disappears. Nothing reports this. Every container stays healthy, the watcher keeps collecting, its logs look busy, and Grafana simply goes empty. Put this in the .env at the repository root and a bare docker compose up -d is correct:

COMPOSE_FILE=docker-compose.yaml:docker-compose.observability.yaml

Set HARNESS_SPHERE_HOST_NAME to the real machine’s name. The watcher reads its hostname to fill the host.name resource attribute, which becomes a label on every series. A container’s hostname defaults to its own id, which changes on every recreate — so every redeploy would fork a brand-new time series and each panel would show the same metric once per container generation. The compose default (zombie-crab-host) is stable rather than accurate, which is the important half: a wrong-but-constant label groups correctly, a correct-but-changing one never does.

The two rules

These are the rules from .claude/CLAUDE.md, and they are stated here because an operator who does not understand why will break one of them while trying to make something work.

It never receives a Docker socket

crab-shell-proxy already mounts /var/run/docker.sock and runs as root. It is the stack’s trusted control plane, and it is the most privileged component there is: a Docker socket is start, stop and exec on any container, and a path to root on the host.

A second socket-mounting service would double the blast radius of the stack’s worst-case compromise, for no gain — because the one thing the watcher would need a socket for, mapping a running container back to its tenant, is served by the proxy’s read-only GET /v1/instances instead.

harness-sphere does run as root in this compose file (user: "0:0"), which overrides the USER 10001:10001 its own Dockerfile sets, and that one decision is argued in place rather than assumed: crab-shell-proxy creates the tenant tree as root:root 0700, and the barrier is at the top of the tree, so a non-root watcher cannot even traverse into it. The alternatives were worse — chmod 0755 would make every workspace path enumerable by any local uid, and the directory names are account UUIDs.

What keeps that narrow is three constraints that are load-bearing together, and a later change must not relax them one at a time:

1. the /data bind stays :ro — it can read the tree, never write it;
2. no Docker socket, ever;
3. no published ports, so there is no inbound surface at all. Every collector pulls.

CRAB_TELEMETRY_TOKEN is not an agent token

CRAB_TELEMETRY_TOKEN authorizes GET /v1/instances on crab-shell-proxy, and nothing else. It is a separate, read-only credential on purpose, and an agent’s bearer token must never be substituted for it.

The reason is what an agent token is actually worth here. The Mycelium profile header is decoded and never verified by the proxy (identity.SDKResolver.Resolve), so the agent’s bearer check is the only thing stopping a caller who reached the proxy directly on zombie_net from asserting any account id it likes. That token gates chatting as any member of any tenant. Handing it to a monitoring component would let that component send messages as other people. handleInstances deliberately does not go through the normal agent resolution for the same reason, and compares the token in constant time.

Leaving it unset is supported and is the safe default. The route is then not registered at all — a 404, not a 401. That is not an oversight: this endpoint discloses the deployment’s whole tenant, subscription and user topology, so a deployment that has not opted in grows no new surface. The watcher keeps reporting the host, itself and the three service probes; it simply cannot attribute per-tenant instances.

Generate a fresh one with openssl rand -hex 32, as deploy/standalone/.env.example and deploy/prod/.env.example both say.

Where to go next

Troubleshooting covers the empty-Grafana failure and others in symptom form. harness-sphere describes the component itself, and Deployment covers the compose modes this overlay sits on.

Troubleshooting

Each entry below is a symptom you can actually observe, the cause behind it, and what to do. They are all failures this stack has real evidence for, recorded in the compose files, the configuration files or the proxy’s own code. If your problem is not here, the boot log of crab-shell-proxy is almost always the right first place to look — it names missing variables on purpose.

A ganglion agent stops answering, and the proxy logs a failed image pull

Symptom. Chatting with an agent that runs the ganglion harness fails. The proxy’s log carries a pull error naming the image, typically zombie-crab/crab-ganglion:dev. Other agents are unaffected. It often starts right after a docker system prune, or on a server that was deployed by pulling images rather than building them.

Cause. EnsureImage in crab/crab-shell-proxy/internal/docker/client.go has a fast path: it asks the daemon for the image locally and, if it is there, returns without contacting any registry. That is what lets a locally built tag work at all. It does not stop there when the image is missing — it falls through to POST /images/create, a real registry pull, which 404s on a tag no registry has.

The default CRAB_GANGLION_IMAGE is zombie-crab/crab-ganglion:dev, a name that exists only on the machine that built it. A prune removes unused images, this one goes, and the agent is dead until the tag comes back.

Fix. Rebuild the tag. The ganglion-image service in docker-compose.yaml exists for exactly this: it is a build-only service that produces the image, runs /bin/true and exits, and crab-shell-proxy waits for its completion. picoclaw-image does the same for the other harness, so both are recovered by the same command:

docker compose up -d --build

On a server, prefer the durable fix: set CRAB_GANGLION_IMAGE in your .env to a published immutable reference — crab/crab-ganglion-harness’s release workflow publishes ghcr.io/lepistabioinformatics/crab-ganglion:sha-<short-sha> — and pull it. See Deployment for why that reference must be immutable rather than a moving tag.

Nothing comes up at all: the gateway never becomes healthy

Symptom. docker compose up -d returns, but mycelium-gateway sits waiting and chat-webapp never starts. Nothing is reachable.

Cause. mycelium-gateway declares depends_on: crab-shell-proxy: condition: service_healthy, and chat-webapp in turn depends on the gateway being healthy, so a proxy that exits at boot takes the whole stack with it.

The proxy exits fatally for a small, specific set of reasons. Its validate() rejects a configuration with no hostDataRoot, no network, an agent with no serviceName or no template, or an agent naming a harness it does not orchestrate. Separately, a picoclaw agent whose bearer token cannot be resolved from the environment is fatal — deliberately, because silently dropping one would lock a member out with no boot-time signal.

Fix.

docker compose logs crab-shell-proxy

The failure names what is missing. Set it in .env and bring the stack up again.

One agent’s routes answer 404 and everything else works

Symptom. A single agent behaves as though it does not exist — its routes return 404 — while the other agents are fine and the proxy is healthy.

Cause. A ganglion agent removes itself at load instead of taking the proxy down, and this is the designed behaviour rather than a bug. One config file can describe several deployments, and an agent that reaches a host with no key for it degrades to “that agent does not exist” rather than “the proxy will not boot”, which would take every other agent down too. There are three reasons it can happen: CRAB_GANGLION_IMAGE is unset, the agent’s model apiKeyEnv variable is unset, or its bearer token cannot be resolved.

Nothing is silently downgraded. The proxy prints one line per disabled agent at boot: agent "<key>" disabled: <reason> — its routes will answer 404, where the reason names the setting, for example that ganglionImage (or CRAB_GANGLION_IMAGE) is unset and has no default on purpose.

Fix. Read the boot log for disabled, set the variable it names, and restart the proxy.

An agent that declares no harness: key is a ganglion agent today (DefaultHarness in crab/crab-shell-proxy/internal/config/config.go), so it is subject to all three checks. Declare the harness explicitly on every agent — a config upgrade should not change a runtime by omission.

Creating a scheduled task answers 501

Symptom. The Tasks panel, or a direct API call, refuses to create, change, disable or delete a scheduled task with a 501 Not Implemented and a message like:

creating scheduled tasks over this API is not available on the picoclaw
harness (agent beta): its agent creates them itself

Cause. crab/crab-shell-proxy/internal/httpapi/cron_write.go refuses every cron write route for an agent whose harness is not the ganglion. On picoclaw the harness owns the job store and holds the live schedule in memory, so a toggle in the panel could disagree with the timers actually running. On the ganglion the proxy owns the schedule, above the container, so it can serve the writes honestly. The read routes are served on both harnesses, which is why you can still see a picoclaw agent’s tasks and their run history.

Fix. On a picoclaw agent, ask the agent in chat to create, change or remove the task; it owns them. If you want the panel’s write controls, use an agent on the ganglion harness. See Scheduled tasks.

The general mechanism is requireHarnessFeature in internal/httpapi/harness_gate.go: a feature a harness cannot serve answers 501 naming the harness, rather than quietly succeeding. That rule exists because a withdrawn harness once let projects be created, stored, listed and reported active while changing nothing about the agent that answered. As the tables in that file stand today it reserves nothing from either shipped harness, so the cron write above is the 501 you will actually meet.

The gateway answers 400 "Request path does not match any service"

Symptom. A request through mycelium-gateway is rejected before the proxy is ever reached, with that exact text.

Cause. Mycelium routes by the first path segment, matched against a service key in its TOML config, and then matches the rest of the path against that service’s [[<agent>.path]] blocks. A path with no matching block is refused here. This is what happens when a proxy route exists but the gateway config was not extended to allow it — the /v1/cron/* read routes are the usual example, and all the profiles under deploy/ already carry a block per agent for them.

Fix. Add the matching [[<agent>.path]] block to the gateway config your mode mounts (deploy/standalone/config.standalone.toml or deploy/prod/config.base.toml) and recreate the gateway.

The same error text appears for an unrelated reason: Mycelium’s own /health route only handles GET and rejects HEAD with this message. If you are probing it with something that sends HEAD — wget --spider, for instance — the 400 is about the method, not about routing. Both healthchecks in docker-compose.yaml use a plain GET for this reason.

Grafana is empty and every container is healthy

Symptom. The dashboards load, the panels render, and there is no data. No container is unhealthy, no log shows an error, and harness-sphere’s own log looks busy.

Cause. The -f chain was omitted on some command. Compose applies an overlay only when you name it, so docker compose up -d, restart, or even up -d harness-sphere without both files reverts every service the overlay overrides back to docker-compose.yaml. For harness-sphere that means the exporter goes back to stdout and its config bind-mount disappears, so it prints metrics into its own logs instead of sending them. Nothing anywhere reports this.

Fix. Bring it up naming both files, and then remove the footgun by putting this in the .env at the repository root, after which a bare docker compose up -d is correct:

COMPOSE_FILE=docker-compose.yaml:docker-compose.observability.yaml

See Observability.

The agent says it saved a file, and the Files panel does not list it

Symptom. The agent reports writing a document, and nothing appears in the member’s Files panel. The file really was written — it is simply somewhere the interface does not look.

Cause. public/ inside a workspace is the only directory a member’s interface lists, and public/attachments/ is where an agent is told to write deliverables. A managed memory file (FILE_DELIVERY.md) carries that rule into every workspace and is read on every turn, precisely because a file written outside public/ is invisible to the member no matter how the deployment is configured.

Fix. Ask the agent to move or re-save the file under public/attachments/, naming the path. Do not write to uploads/: that is the directory’s former name, kept in the code only so a one-time migration can recognise a workspace predating the rename.

A related surprise: the paperclip notice the proxy appends when a file is delivered is stream-only and is never persisted. After a page reload, the only account of a delivered file is whatever the model itself wrote in its reply, which is why the managed rule also tells it to name the path out loud.

The Map and Entities tabs stay empty

Symptom. The knowledge graph never records anything. No error is shown and nothing is logged about it.

Cause. CRAB_MCP_TOKEN_SECRET is empty. That value signs the bearer token a spawned agent presents back to the proxy’s own MCP endpoint. Unset is a supported and deliberate state: /v1/mcp is not registered, no MCP server block is written into any workspace, and everything else behaves normally. A deployment that forgot the secret must get no memory rather than an unauthenticated endpoint reachable by every container on the network.

Fix. Generate one with openssl rand -hex 32, set it in .env, and recreate the proxy. Both .env.example files spell this out.

harness-sphere cannot attribute instances, or /v1/instances 404s

Symptom. The watcher reports the host, itself and the three service probes, but nothing per tenant. A direct request to GET /v1/instances on the proxy returns 404.

Cause. CRAB_TELEMETRY_TOKEN is empty, so the route is not registered at all — absent, not 401. This endpoint discloses the deployment’s whole tenant, subscription and user topology, so a deployment that has not opted in grows no new surface.

Fix. Generate a fresh secret with openssl rand -hex 32 and set it. Do not reuse an agent’s bearer token for this: an agent token gates chatting as any member of any tenant, and the reasoning is in Observability.

rm -rf data/... fails with Permission denied

Symptom. Wiping the on-disk state during a reset fails.

Cause. crab-shell-proxy creates the tenant tree as root, 0700, so the directories are not yours.

Fix. Use sudo for that one command, as the reset sequence in Deployment does. This is also why harness-sphere runs as root with a read-only bind: nothing else could traverse the tree.

No sign-in e-mail arrives in standalone

Symptom. You request a magic link, or the Staff bootstrap code, and no e-mail ever comes.

Cause. Standalone has no real SMTP. It ships a stub transport that writes the message to the log instead of sending it — a deliberate property of the mode.

Fix. Read it out of the gateway’s log. Sign-in links land in the same place.

docker compose logs mycelium-gateway | grep -i bootstrap

Nobody can sign in to a fresh prod deployment

Symptom. The prod stack is up, the gateway is running, and authentication does not work.

Cause. Mycelium’s Postgres adapter has no embedded migrations, unlike its SQLite one, so a freshly created database has no schema.

Fix. Run the one-time, two-step schema application described in Database and migrations. Both steps are required at the release this repository pins; the second is not optional.

Where to go next

If the failure is about how a deployment is assembled, Deployment is the fuller account. If it is about what the watcher does or does not show, Observability covers that surface. For the behaviour of the two harnesses themselves, see Harnesses.

crab-shell-proxy

This page describes the orchestrator as a component: what it owns, what it deliberately does not, and how its code is arranged. Read it before you open the repository for the first time.

What it is

crab-shell-proxy is a small Go service that sits between the Mycelium gateway and the agent containers. It reads which agent a request is for and which member made it, makes sure that member’s own container is running, and relays the conversation to it. The repository is crab/crab-shell-proxy, its Go module is github.com/LepistaBioinformatics/crab-shell-proxy, and the compose service has the same name as the directory.

It is the component that holds the Docker socket, and it runs as root. The Dockerfile’s runtime stage says why in as many words: it has to reach the socket (root:docker, mode 0660), read root-owned template files, and write per-user data directories. Everything else in the stack is arranged so that it does not need those privileges, because this one component already has them. The README’s security note puts it plainly: the proxy is the trusted control plane, and the agents it spawns are the non-root, sandboxed part.

A harness is the program inside an agent container that actually talks to the model and runs tools. The proxy orchestrates harnesses; it is not one. See Harnesses.

What it is responsible for

Resolving identity into a container. The gateway verifies the caller’s token and injects a profile header. The proxy takes the agent from the injected service name and the member from the profile’s accId, and ensures there is one container and one directory for the resulting (tenant, subscription, agent, user) tuple. The account id is used rather than the email because an email is mutable; the email is kept only as a human-readable marker in .crab-owner.json.

Lifecycle. An agent is declared either scale-to-zero, where the container cold-starts on the member’s first request and is stopped after an idle window, or continuous, where it is never stopped automatically. Both are configured per agent in config.yaml.

Choosing the harness. Each agent declares which runtime answers for it:

agents:
  alpha:
    serviceName: "alpha"
    harness: "ganglion"
    template: "alpha"
    mode: "scale-to-zero"
    idleTimeout: 30s

internal/pico runs a turn against picoclaw over its WebSocket protocol; internal/ganglion runs one against the ganglion over HTTP with SSE. Which of the two is used is decided by that one key. An agent that declares no harness: key gets the ganglion: the config loader in internal/config/config.go fills an empty harness: in with DefaultHarness before it validates anything, and requireHarnessFeature in internal/httpapi/harness_gate.go reads the same constant. Declare the key explicitly in every agent you write anyway — a ganglion agent with no image is disabled rather than started, so inheriting the default on an unprepared host takes that agent out of service.

Telling the truth about what a harness cannot do. harness_gate.go keeps a table of features that are not universal, and a feature the agent’s harness cannot serve answers 501 naming the harness rather than quietly succeeding. The file records the incident that produced the rule: a harness once accepted project creation it did not implement, so a project could be created, stored and listed while changing nothing about the agent that answered.

Everything done to a container. Volume provisioning and ownership, secrets materialization, the model registry, the memory-graph MCP server, scheduled tasks and the admin API all live here. The harness specification states this as a permanent boundary: a harness inside a container has no business starting, stopping or provisioning anything, including itself.

The HTTP surface. The member-facing part is OpenAI-shaped — POST /v1/chat/completions, GET /v1/models, GET /v1/sessions/history — with GET /healthz for liveness and GET /doc/openapi.json for the OpenAPI document embedded in the binary. Administration lives under /v1/admin/..., and GET /v1/instances is a read-only inventory of running instances, which exists so that nothing else in the stack has to ask Docker itself.

What it is not responsible for

It does not authenticate anyone. Identity arrives already verified from the gateway, and the proxy’s job is to trust that header rather than to reproduce the check.

It does not run the agent loop. Deciding which tool to call, when to stop, and what the answer is belongs to the harness.

It does not render anything. The member-facing UI is crab-exoskeleton-webapp, which reaches the proxy through the gateway and never talks to an agent container directly.

It does not collect metrics about the stack. That is harness-sphere, and the division is load-bearing: because the proxy already holds a Docker socket, nothing else in the stack is given one.

How it is built and tested

The build is the test gate. The Dockerfile’s build stage runs go vet ./... and go test ./... before it links the binary, so a failing test means no image is produced and therefore nothing is published. release-image.yml is the only workflow in the repository; it builds and pushes that image on a push to main or a version tag. There is no separate pull-request workflow, which means the checks a contributor runs locally are the same ones that gate the image:

go vet ./... && go test ./...

A second suite talks to a real Docker daemon and is kept behind a build tag, so it does not run in the command above and does not run in the image build either:

go test -tags integration ./internal/docker -run TestIntegration -v

config.yaml is baked into the image at /etc/crab-shell-proxy/config.yaml and holds environment-variable names rather than values, so one image stays usable across deployments. See Configuration.

How the code is laid out

cmd/crab-shell-proxy/main.go is the entry point; everything else is under internal/. The packages worth knowing before you start reading:

internal/docker is by a wide margin the largest package, which is a fair signal of where the work is: most of what this service does is careful filesystem and container manipulation on behalf of someone whose request it has already trusted.

Where to go next

Harnesses explains the two runtimes and how one is chosen. Agents, workspaces and projects describes the directory layout this service reads and writes. If you are about to change the code, Working on the stack has the build and test commands for every repository in one place.

crab-ganglion-harness

The ganglion is the agent runtime this project wrote for itself, and the harness this book teaches. This page describes it as a component: what runs inside an agent container, what that program is allowed to do, and how the repository is organised.

What it is

A harness is the program that runs inside a member’s agent container. It holds the conversation, calls the model, runs whatever tools the model asks for, and writes the transcript to disk. crab-shell-proxy starts one such container per member and speaks to it; what happens inside is the harness’s business.

crab-ganglion-harness is that program, written in Go. Its go.mod declares the module, a Go version, and nothing else: the harness has no third-party dependencies at all, and the whole thing compiles to one static binary. It serves HTTP with Server-Sent Events natively, so the proxy’s runner for it needs no protocol translation — internal/ganglion/turn.go in the proxy notes that this is why it has no analogue of the 500-millisecond guess that picoclaw’s runner needs to tell when a turn has ended.

The other harness is picoclaw, a third-party project this stack started on. It is still fully supported and it is on its way out: an agent that declares no harness: key now gets this one, not that one. Harnesses covers the comparison; write harness: "ganglion" explicitly in every agent you create rather than leaning on the default.

What it is responsible for

Serving one turn per request. The HTTP surface is two routes and no more: POST /v1/chat/completions runs a turn, and GET /health answers {"status":"ok"} for the proxy’s health check. The turn ends when the handler returns.

Streaming the work while it happens. As the agent uses its tools, narration and reasoning arrive on a progress channel. The answer is different: a frame is buffered and classified once it is complete, then sent in one piece. The reason is written down in the specification, because it looks like a regression if you do not know it. Whether an iteration’s text is narration or the final answer depends on how the frame ends, and that is not knowable before the stream finishes — the proxy measured seven turns out of a hundred and twelve delivering a whole reply in the same frame that carried a trailing tool call. Emitting optimistically and correcting afterwards would make the reply visibly rewrite itself.

Bounding a turn. The only limit on a single turn is how many times it may go back for another tool. A turn that reaches the cap stops and says so rather than failing silently. Two places set it and the first wins: agents.defaults.max_tool_iterations in the mounted config.json, which is the surface an administrator actually has, and GANGLION_MAX_ITERATIONS on the container, which defaults to 12. The harness prints the number in force and where it came from at boot, because 12 looks identical whether an operator set it, the file set it, or nothing did.

Running tools, confined. The tool that matters is shell. Before running a command, the binary re-executes itself in a sandbox mode, applies a Landlock domain, and only then executes the command. The workspace is read-write; /usr, /bin and /sbin are read-execute; everything else, /proc included, is denied — which is what stops a command from reading /proc/1/environ and with it the harness’s own provider key. If the kernel cannot provide a Landlock domain, the harness refuses to boot rather than serving turns unconfined.

Other tools are registered only when the configuration supports them: load_image and set_reasoning_depth are always present, web_search and web_fetch appear when a search provider is configured, generate_image when an image model is, and subagents and research when sub-turns are enabled. Anything an operator mounted over MCP is added on top. Each boot log line says which of these were switched on.

Persisting the conversation. The transcript is append-only JSONL under workspace/sessions, and the context window is a separate derived artifact. A conversation that has a transcript and no window — the state every conversation migrated from picoclaw arrives in — has its window rebuilt from the transcript, so the member does not see their history on screen while the agent answers as though the conversation had just begun.

Keeping the layout the stack expects. Everything lives under the workspace segment, because that is where the proxy looks from outside the container. A project’s files go in workspace-<id>, a sibling of workspace/, never a child of it; the harness migrates a subtree left at the old path on boot. See Agents, workspaces and projects.

What it is not responsible for

It does not start, stop or provision anything, including itself. Container lifecycle, volumes and their ownership, secrets materialization, mycelium identity and authorization, the model registry, the admin API, projects and scheduled tasks all stay in crab-shell-proxy. The harness specification lists this as permanently out of scope.

It does not decide who may approve a gated action. The harness has the Approver port and the mid-turn suspension behind it, and its first adapter calls back to the proxy, which owns member identity — the harness only asks. The other half of that round trip is not built: there is no approval endpoint on the proxy, and with GANGLION_APPROVAL_ENDPOINT unset the loop installs an allow-all approver. Treat the approval flow as a seam that exists rather than a feature you can switch on.

It has no .secrets/ directory. Credentials reach this harness as environment variables. A value that must not sit in plaintext can be encrypted first — crab-ganglion encrypt reads the plaintext on stdin, never as an argument, and prints the enc:// value to paste into the deployment’s environment.

How it is built and tested

This is the only one of the four repositories with a pull-request gate that runs the test suite. .github/workflows/ci.yml runs, on every pull request and every push to main:

gofmt -l . | tee /dev/stderr | (! read)
go vet ./...
go test -race ./...
go build ./...

-race is deliberate rather than habitual: the SSE writer has two writers by construction, the turn and the heartbeat, and interleaved output would corrupt a frame the client is in the middle of parsing.

The same tests run a second time inside the Dockerfile, before the binary is linked, so the image cannot be built without them passing. release.yml builds with the layer cache switched off for exactly that reason — replaying the tests from a cache would make a green run stop being evidence that they ran for these bytes.

The runtime image is Alpine rather than distroless, and the Dockerfile explains that this was found by running it: distroless has no /bin/sh, so the shell tool could never succeed. The container runs as uid 1000, which matches what the proxy chowns each member’s volume to, and listens on port 18800.

Every published image is addressed by the commit that produced it. There is no :latest and no :main; release.yml publishes ghcr.io/lepistabioinformatics/crab-ganglion:sha-<commit> and nothing that is ever rebuilt. That is this stack’s own scar: the proxy’s EnsureImage returns early when a name resolves locally, and the harness image is not a compose service, so a rebuilt tag can leave a host running old bytes indefinitely with nothing in the logs. It happened once, for three weeks.

The consequence for a deployment is that the image reference is a decision somebody makes. docker-compose.prod.yaml sets no CRAB_GANGLION_IMAGE and supplies no default, so a production deployment has to name a digest or a sha- tag explicitly before a ganglion agent can start. The development compose file takes the other route: a build-only ganglion-image service builds the harness from the submodule and tags it zombie-crab/crab-ganglion:dev, and the proxy waits for it. Because that build runs the harness’s own test suite, a failing test stops the development stack from coming up. See Deployment.

How the code is laid out

The repository is hexagonal, and the shape is enforced by tests rather than by convention.

cmd/crab-ganglion/main.go   the composition root: the only place adapters meet
internal/domain/            entities, the ports, pure policy — standard library only
internal/runtime/           the turn loop, compaction, sub-agents
internal/config/            environment and the mounted config.json
internal/adapter/           one directory per adapter
internal/secret/            the enc:// resolver
internal/skillfile/         reading skill files

internal/domain/arch_test.go holds two rules. The first fails if the domain package imports anything outside the standard library, because the moment the domain imports an HTTP client or a provider SDK the loop stops being testable without them. The second fails if one adapter imports another, which is what keeps a second provider or a second ingress a new file rather than a refactor.

The adapters are grouped by what they drive: httpsse is the way in; provider/openai and provider/router are the way out to a model; store/jsonl and store/window are the transcript and the derived context; tool/* is one directory per tool; approver/proxy, telemetry/otlp, mcp, skills and evolution are the rest. Adding a tool is a file plus a registration line in main.go — not a change to the loop, the port or the registry.

The specification for all of this lives in the product repository, at .specs/features/crab-ganglion-harness/, along with the design notes and the requirement identifiers that the code comments cite.

Where to go next

Harnesses compares the two runtimes and explains what the proxy does when a feature is not available on one of them. Configuration covers the environment this container reads. Working on the stack has the commands.

crab-exoskeleton-webapp

The webapp is the part of the stack a member actually sees. This page describes it as a component: what it is, the one transport rule that governs every change to it, and where its code lives.

What it is

crab-exoskeleton-webapp is a Next.js 15 application using the App Router. It is both the user interface and a backend-for-frontend: the pages under app/ render the chat and the admin console, and the route handlers under app/api/ are a server-side layer that calls upstream on the browser’s behalf.

Its compose service is chat-webapp, not the repository name. So is the name field in its package.json, and so is the published image in docker-compose.prod.yaml. If you are looking for this component in a compose file, in docker compose logs, or in a container listing, look for chat-webapp.

The backend-for-frontend arrangement is the point rather than a detail. The browser holds a session cookie and nothing else — no token, no account id, no upstream URL. Every request goes browser, then route handler, then Mycelium gateway, then crab-shell-proxy, so the same verified identity that protects the backend protects the interface without the interface having to re-implement any of it.

What it is responsible for

Holding the session. Sign-in is a magic link. The route handler completes it and stores the gateway session in an HTTP-only cookie; middleware.ts guards /chat and /onboarding by checking that the cookie parses and that its token has not passed its own expiry, redirecting to /signin and clearing the cookie when it has not. That check is explicitly not validation — a token can be revoked upstream while its expiry is still in the future, so the real answer still comes from the first upstream call, which clears the session on a 401.

The chat experience. Streaming replies, conversation history, search, renaming and tagging, deep links of the form /chat/{agent}/{sessionId}, file upload into the member’s workspace, and timeline and tree views of past activity. See The chat client.

The operator console. Tenants, subscriptions and members, the per-agent model registry and per-user model assignments, shared skills and shared content, secrets, and branding. The gateway still enforces who is allowed to do what; these screens are a surface over the proxy’s admin API, not a second authorization system. See Admin guide.

Its own database. A Postgres connection, configured by DATABASE_URL, carries the conversation index and app-side metadata. In the development compose file this is the separate chat-webapp-postgres service.

The transport rule: always JSON-RPC, never a new REST call

This is the one convention you must know before you write a line of code here.

Mycelium’s gateway exposes both a REST API and a JSON-RPC 2.0 endpoint at POST /_adm/rpc. They are not interchangeable and REST is the wrong default. The gateway’s beginners REST endpoints are external-identity-provider only: for a magic-link user — which is every user of this deployment — they answer 400 "Invalid provider". The RPC dispatcher resolves the internal issuer instead, so it is the only transport that works for this stack’s own members. That was established by trying it, and the evidence is kept in .specs/features/onboarding/context.md in this repository. The RPC surface is also broader: whole operations, such as inviting and uninviting a guest, have no REST equivalent this stack can reach.

In practice this means calling myceliumRpc() from lib/mycelium.ts, never adding a new fetchMycelium() path to a /_adm REST route. Parameters are camelCase, and the authoritative registry of method names is ports/api/src/rpc/method_names.rs in the mycelium source — never guess one, because an invented name fails only at runtime and the failure looks like a permissions problem.

The rule is enforced rather than trusted. .github/workflows/mycelium-transport.yml greps app/ and lib/ on every pull request that touches them and fails the build when a /_adm path is passed to fetchMycelium from a file outside its allowlist. The allowlist is where the exceptions live, each with a reason: the pre-session magic-link request and verify pair, which has no token to authenticate an RPC call with; lib/mycelium.ts itself, because POST /_adm/rpc is the RPC transport; and app/api/tenants/[id], which predates the check and contradicts the rule as written, allowlisted so that it is visible rather than silently tolerated. The workflow’s own comment is honest about its limit: it matches a /_adm path on one line, so a call built across lines or through a variable is not caught. It is a ratchet against the easy way in, not a proof.

One more boundary worth stating: requests to crab-shell-proxy — the /{agent}/v1/... and /alpha/v1/admin/... paths — are the proxy’s own HTTP API and stay REST. “Call mycelium over JSON-RPC” is not “convert the proxy to JSON-RPC”.

What it is not responsible for

It never talks to an agent container. There is no path from this application to a harness that does not go through the gateway and the proxy.

It does not decide who may do what. Authorization is the gateway’s, and the screens reflect it rather than implementing it.

It bakes nothing deployment-specific at build time. MYCELIUM_INTERNAL_URL and DATABASE_URL are read server-side at request time, which is what lets one published image serve every deployment. The Mycelium admin UI that ships beside it, mycelium-webapp, is a pure client-side SPA whose API URL is compiled in; the two are different in this respect and the compose file says so.

How it is built and tested

Development is the ordinary Next.js loop:

yarn install
yarn dev        # http://localhost:3000

Two workflows run in CI, and neither runs the test suite. mycelium-transport.yml is the transport grep described above, on pull requests touching app/ or lib/. release-image.yml builds and pushes the image on a push to main or a version tag; the Dockerfile installs with yarn install --frozen-lockfile and then runs yarn build, so a build error fails the publish and a failing test does not.

yarn test runs the Vitest suite, which is substantial and covers lib/, components/ and parts of app/. It is documented in the repository’s README and it is the check to run before you open a pull request — just be aware that nothing in CI will run it for you.

yarn lint does not work. The script is next lint, but ESLint is not a dependency of this repository: it appears in neither package.json nor yarn.lock, and it is not installed. The script is a leftover. Do not put it in a contribution checklist and do not expect a linter to catch anything here.

tsconfig.json sets noEmit, so TypeScript is a type checker rather than a build step, and there is no script that invokes it on its own; type errors surface through yarn build and through your editor.

The production image is small because next.config.ts sets output: "standalone", which traces only the dependencies actually used at runtime — the final stage copies the traced server and the static assets and needs neither node_modules nor yarn. The Dockerfile carries a long warning against re-enabling corepack, which replaced a working bundled yarn with a shim that resolves its version over the network and turned an offline build into one that needed the npm registry.

How the code is laid out

app/chat/        the chat experience
app/admin/       the operator console
app/api/         the backend-for-frontend route handlers
app/signin/      magic-link sign-in
components/      shared UI, plus the pre-auth landing page
lib/             everything that is not a component: mycelium.ts, session.ts,
                 the model, media, memory and admin helpers, i18n
middleware.ts    the session guard on /chat and /onboarding
.specs/          specifications; start with .specs/project/PROJECT.md

Tests sit beside the code they cover as *.test.ts and *.test.tsx. vitest.config.ts excludes node_modules, .next and .claude by glob rather than by bare name, because a git worktree checked out under .claude/worktrees/ once brought its own node_modules and another branch’s tests into the run.

Styling is Tailwind CSS v4 with class-variance-authority for variants, rather than conditional or interpolated className strings.

Where to go next

The chat client covers using the application as a member, and Admin guide covers the operator screens. Working on the stack has the build and test commands for every repository, and Contributing has the conventions a change has to follow.

harness-sphere

harness-sphere is the watcher. This page describes it as a component: what it observes, the three things it will never do, and how its code is arranged.

What it is

HarnessSphere is a single self-contained Rust binary that watches one stack — the machine it runs on, itself, and the services zombie-crab runs — and turns what it finds into standard OpenTelemetry metrics, shipped to whatever backend you point it at. The binary is called harnesssphere, with three s, and the repository is crab/harness-sphere.

It exists because before it there was nothing. Not “not enough”: the stack emitted no metrics, had no collector and no dashboard, so the first question anyone asks during an incident — was the machine under pressure when that got slow — had no answer at all. Because the watcher sits on the same host and the same timeline as everything else, it can answer the one thing separate tools cannot: whether an agent slowed down because of the agent, or because of the machine underneath it.

It is exclusive to this stack. It began as a general-purpose watcher and is now the observability component of zombie-crab-project. Collectors with no source here were deleted rather than left configurable, and there is no flag that brings them back. Publishing to crates.io is disabled; binary releases continue.

What it is responsible for

The stack is modelled as exactly six layers: the host, the watcher itself, the gateway, the proxy, the webapp, and the agent containers. There is deliberately no catch-all variant — a seventh kind of thing appearing should break the build rather than file itself under “other”.

Two layers are Critical and the rest are Optional, and the distinction is the whole resilience model. Host and Watcher are Critical: if one fails persistently, past a configurable threshold so that a single hiccup is forgiven, the watcher flushes what it can and exits non-zero, loudly. Everything else degrades, backs off and retries, and never brings the process down. A target that is not responding reads an honest zero rather than going absent, which is why the watcher needs no startup ordering and survives booting before the things it watches.

Everything is pull. There is no receive path, because nothing in this stack pushes telemetry at it and a receiver with no sender is code that can only ever be wrong. Host and self metrics come from sysinfo; liveness comes from an active TCP connect per tick; container CPU and memory come from reading cgroup v2 kernel files directly; and per-member activity comes from globbing the tenant tree and reading the on-disk JSONL transcripts incrementally.

That last one is the interesting part, and it has four corrections in it, each with a test. A project’s conversations live in a sibling workspace-<id> directory, and missing it dropped 42% of the conversations on the workspace that was measured. A sessions/durable/ directory mirrors the live files one to one, so counting it exactly doubles every number. Every scheduled-task run writes its own session file, which drifts a count of conversations into a count of conversations plus every cron run since provisioning. And a transcript that shrank is re-read from zero rather than treated as a negative delta.

Message content is never read into a signal. Counts, names and sizes only.

What it is not responsible for

It never receives a Docker socket, by design. The proxy already holds one and runs as root; a second socket-holding service would double the blast radius of the stack’s worst-case compromise. A socket is start, stop and exec on any container and a path to host root, and everything the watcher would need one for — mapping a container to its tenant — is served by the proxy’s read-only GET /v1/instances inventory instead.

It does not report token cost. picoclaw writes no token counts to disk and exposes no metrics endpoint, and the only path that ever existed was scraping an endpoint this stack does not run. That scraper has been deleted. This is not deferred; it is gone. Token accounting is one of the two capabilities that justified writing crab-ganglion-harness, and it will arrive from that direction rather than this one.

It does not read transcript content. Message bodies are member data.

It does not change anything. Every collector pulls, the service publishes no ports, and there is no inbound surface at all.

The proxy’s telemetryToken, supplied as CRAB_TELEMETRY_TOKEN, gates the inventory route and is not an agent token — an agent’s bearer token must never be used in its place. With it unset the route is not registered at all, answering 404 rather than 401, and that is the safe default rather than an oversight.

How it is run in this stack

The development compose file builds the service from the submodule and runs it with a read-only bind of the proxy’s data root at /data, a stable hostname, and no published ports. Three details there are argued at length in the compose file and are worth reading before you change any of them.

The service is given user: "0:0" in this stack, even though its own Dockerfile runs as uid 10001. The reason is that the proxy creates the tenant tree as root:root mode 0700, and the barrier is at the top of the tree rather than at its leaves, so a non-root watcher cannot traverse it at all. Three constraints keep that grant narrow and are load-bearing together: the /data bind stays read-only, there is never a Docker socket, and there are no published ports.

The hostname is pinned because host.name is a label. A container’s hostname defaults to its own id, which changes on every recreate, so every redeploy would fork a new time series — five had accumulated in a single evening of iteration. A wrong-but-constant label groups correctly; a correct-but-changing one never does.

There is deliberately no depends_on. Ordering the watcher behind the health of the things it watches would suppress the exact signal it exists to produce.

To actually look at the numbers, the product repository carries an opt-in overlay with an OpenTelemetry Collector, Prometheus and Grafana:

docker compose -f docker-compose.yaml -f docker-compose.observability.yaml up -d

When you write queries, remember that the names in this book are the emitted OTel names. The OTLP-to-Prometheus translation rewrites dots to underscores and appends a unit suffix, so what the watcher emits as system.memory.usage is scraped as system_memory_usage_bytes.

How it is built and tested

Rust, a Cargo workspace, edition 2024. rust-toolchain.toml asks for stable with rustfmt and clippy.

cargo build --release
./target/release/harnesssphere config.example.toml   # prints signals to your terminal

The stdout exporter is the default and proves the whole pipeline with no backend to stand up. For a real backend, build with the OTLP adapter — otlp is the only feature the binary declares:

cargo build --release --features otlp

Be aware of what CI does and does not check here. audit.yml runs cargo audit --deny warnings when a manifest or lockfile changes, weekly on a schedule, and on dispatch. deepseek-pr-review.yml posts an automated review comment on every pull request. release-image.yml builds and pushes the image. None of them runs cargo test or cargo clippy, and the Dockerfile builds the binary without running the suite. Tests exist — crates/runtime/tests/ and harnesssphere/tests/ — but running them is on you.

The release profile is tuned small, with opt-level = "z", link-time optimization and stripping. Panic unwinding is kept on purpose, because the resilience model depends on catching a panic inside a collector before it can escape.

How the code is laid out

The repository is hexagonal, and the ganglion’s architecture rules were modelled on this crate split.

crates/domain/       canonical signal model, ports, pure policies — no I/O, no OpenTelemetry
crates/runtime/      supervisor, scheduler, circuit breaker, batching drain
crates/collectors/   host and self (Critical); process, endpoint probe, session, container (Optional)
crates/export/       stdout by default, OTLP behind the `otlp` feature
harnesssphere/       the binary: config, wiring, run

The domain holding zero OpenTelemetry dependency is not tidiness. It keeps the important logic — the circuit breaker, the criticality policy, the enrichment — unit-testable without a network, and it keeps a pre-1.0 SDK that is still changing from leaking into the core.

Configuration is a TOML file passed as the first argument, with a few environment-variable overrides: HARNESSSPHERE_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINT and RUST_LOG. Every Optional collector is off until configured, so a fresh run shows Host and Self only. This stack’s own configuration is config.zombie-crab.toml, which probes the gateway, the proxy and the webapp, and deliberately leaves the single-valued container keys empty — pointing them at one arbitrary instance would produce a metric that describes one tenant and reads like it describes the stack.

Where to go next

Observability covers what the numbers mean and how to read them. Deployment covers running the stack with the observability overlay attached.

Working on the stack

This page is for someone about to change the code. It says where the code is, how to get a development environment, and — the part worth reading carefully — exactly what each repository checks, because the four components do not check the same things and one of them documents a command that cannot run.

Where the code is

The product repository is zombie-crab-project, and four of its directories are separate Git repositories brought in as submodules:

The product repository itself holds the compose files, the deployment configuration under deploy/, the Dockerfiles for the Mycelium gateway and its admin UI under fungi/, this book under docs/book/, and the specifications under .specs/.

Because they are submodules, a plain git clone gives you four empty directories. Clone the whole chain at once:

git clone --recurse-submodules https://github.com/LepistaBioinformatics/zombie-crab-project.git
cd zombie-crab-project

If you already cloned without them, git submodule update --init --recursive fills them in.

A development environment

The development compose file builds every service from source, so the only prerequisites for running the stack are Docker with the Compose plugin and an LLM API key. Copy deploy/standalone/.env.example to .env at the repository root, fill in the bearer tokens, the per-agent API keys and the bootstrap secret, and bring it up. Installation walks through this in order, and Configuration covers what each variable does.

Two things about that build are worth knowing before you start.

The first is that some builds run tests. crab-shell-proxy and crab-ganglion-harness both run go vet and their full test suite inside the Docker build, before the binary is linked, so a failing test stops the stack from coming up rather than producing a quietly broken image. That is the intended behaviour for a development compose file, not an accident.

The second is that the two harness images are produced by build-only services — picoclaw-image and ganglion-image — that build a tag and exit, and the proxy waits for both. They exist because those images are not services of their own and nothing pulls them: docker system prune removes an unused image, and without a compose service to rebuild it, every agent using it would be dead until somebody remembered to run docker build by hand.

To work on one component without rebuilding everything, you can develop it natively against the rest of the stack. The webapp is the easiest case — yarn dev on port 3000 against a running gateway — because it reads MYCELIUM_INTERNAL_URL and DATABASE_URL at request time.

The gates, repository by repository

These lists come from each repository’s workflows, its Dockerfile and its package.json. Where a repository has no automated check, this page says so rather than suggesting a command that nothing enforces.

crab-ganglion-harness

The only repository with a pull-request workflow that runs the tests. .github/workflows/ci.yml runs on every pull request and every push to main:

gofmt -l .        # the workflow fails if this prints anything
go vet ./...
go test -race ./...
go build ./...

Run those four before you open a pull request and you have reproduced CI exactly. -race is not decoration: the SSE writer has two writers by construction, the turn and the heartbeat, and an interleaved write would corrupt a frame the client is mid-parse on.

release.yml builds and publishes the image on a push to main or a tag, with the layer cache disabled so that the tests inside the Dockerfile genuinely run for those bytes.

crab-shell-proxy

There is no pull-request workflow at all. The repository’s only workflow, release-image.yml, builds and publishes the image on a push to main or a version tag — and the gate lives inside the Dockerfile, which runs go vet ./... and go test ./... before the build step. A red test means no image.

Locally, those two commands are the whole check:

go vet ./... && go test ./...

A second suite drives a real Docker daemon and is kept behind a build tag, so it runs neither in the command above nor in the image build. Run it when you have changed the Engine API client or anything about container creation:

go test -tags integration ./internal/docker -run TestIntegration -v

It creates and removes a throwaway Alpine container, so it needs a reachable /var/run/docker.sock but no LLM key.

crab-exoskeleton-webapp

Two workflows, and neither runs the tests. mycelium-transport.yml greps app/ and lib/ on pull requests that touch them, failing the build when a new REST call to the mycelium gateway appears outside its allowlist — see the component chapter for what that rule is and why. release-image.yml builds and pushes the image on a push to main or a tag, which runs yarn install --frozen-lockfile and then yarn build.

So the effective CI check on the code is: it builds. The test suite is real and extensive, it is documented in the repository’s README, and running it is on you:

yarn install
yarn test        # Vitest
yarn build       # the same thing the image build runs

Do not run yarn lint. The script is next lint, but ESLint is not a dependency of this repository — it is in neither package.json nor yarn.lock and it is not installed, so the script cannot do anything. It is a leftover from a scaffold.

There is also no type-check script. tsconfig.json sets noEmit, so TypeScript is a checker rather than a build step, and type errors reach you through yarn build and through your editor.

harness-sphere

No workflow runs cargo test, and none runs cargo clippy. What CI does run is cargo audit --deny warnings when a Cargo.toml or Cargo.lock changes, weekly on a schedule, and on manual dispatch; an automated model-written review comment on every pull request; and the image build, whose Dockerfile compiles the release binary without running the suite.

Tests do exist, under crates/runtime/tests/ and harnesssphere/tests/, and rust-toolchain.toml asks for rustfmt and clippy alongside stable Rust. Nothing will run them for you, so run them yourself:

cargo build --release
cargo test

The otlp feature is the only one the binary declares; a build that will ship metrics to a collector wants cargo build --release --features otlp.

Working on this book

The book is an mdBook under docs/book/, with the English source in src/ and the Portuguese translation generated from po/pt-BR.po. English is the source and Portuguese is a translation: translating means editing the catalogue, never writing a second tree of Markdown files that can drift from the first.

cd docs/book
mdbook build                                           # the English book
MDBOOK_BOOK__LANGUAGE=pt-BR mdbook build --dest-dir book/pt-BR

The Portuguese build needs mdbook-gettext from mdbook-i18n-helpers. .github/workflows/deploy-docs.yml builds both on a push to main that touches docs/book/** and publishes them to GitHub Pages, with the Portuguese book nested inside the English one so a single artifact carries both.

Give the Portuguese build a destination of its own, as above: book.toml sets build-dir = "book", so a bare mdbook build in Portuguese would overwrite the English book in place. Note also that create-missing = false is set, so a chapter listed in SUMMARY.md with no file on disk fails the build instead of being created empty.

Where to go next

Contributing covers the conventions a change has to follow, including the submodule chain and how a change moves through it. Troubleshooting covers the failures people actually hit while running the stack.

Contributing

This page collects the conventions a change has to follow: which language to write in, where specifications live, and how a change moves through a chain of five repositories. Most of these rules are enforced by a workflow rather than trusted, and this page says which.

Everything here is written in English

This repository and every repository below it is written in English, without exception by artefact type: code comments, commit messages, pull request titles and bodies, issues and issue comments, .specs/, READMEs, documentation, ADRs, test names, failure messages and changelogs.

This is not a style preference. The repository is public under MIT OR Apache-2.0, it has its own remote, its own pull requests, and readers outside the team. A comment in another language is a barrier to anyone arriving.

The rule applies at every depth of the chain — crab/crab-shell-proxy, crab/crab-ganglion-harness, crab/crab-exoskeleton-webapp and crab/harness-sphere included. The private marketing repository that carries this one as a submodule writes in Portuguese, and the boundary is that repository’s modules/ directory. A consequence worth stating: moving a document across that boundary means translating it, not copying it.

The rule governs artefacts, not conversation. Talking to the project owner in another language while writing an English comment is correct, not inconsistent.

Where specifications live

Specification work lives under .specs/, never at a repository root. The convention is the same in every repository that has one:

.specs/project/            PROJECT.md (vision), ROADMAP.md, STATE.md
.specs/features/<slug>/    spec.md, context.md, design.md, tasks.md, reports

One folder per feature, named with the feature slug. Execution artifacts — progress notes, task reports, implementation notes — go in the same folder as the feature they belong to. Do not leave scratch files at the repository root; if a tool would put one there, move it under the matching feature folder.

The product repository’s .specs/features/ is the largest of these and is where cross-component work is specified. crab-ganglion-harness keeps no .specs/ of its own: its specification lives in the product repository at .specs/features/crab-ganglion-harness/, and its README says so.

A specification is also the right place for a decision that would otherwise be lost. Several of the more surprising behaviours in this stack — why the answer arrives whole rather than word by word, why a harness image carries no moving tag — are documented as requirements with a stated reason, and the code comments cite them by identifier.

The submodule chain

zombie-crab-project carries four submodules, each a separate repository with its own pull requests. A change that touches both a submodule and the product repository is therefore two pull requests, or three, or five.

A pointer may only name a commit reachable from that submodule’s default branch. Reachable, not equal: pointing at an older commit on main is ordinary and allowed. Pointing at a commit that exists only on a pull-request branch is not — that commit disappears when the branch does, and this repository is left describing a tree nothing points at.

So the chain is merged bottom-up, one level at a time. Merge the submodule’s pull request first, then advance the pointer to the merge commit:

git -C crab/<name> checkout main
git -C crab/<name> pull --ff-only
git add crab/<name>
git commit -m "chore(submodule): advance <name> to #<pr>"

One change can have two independent children gating one parent. The harness-sphere-integration work was the first: it needed a pull request in harness-sphere and one in crab-shell-proxy, neither blocking the other, both merged before the parent’s pull request could pass. Siblings have no ordering between them — only children and parents do. Merge them in whatever order review finishes, then bump both pointers.

Opening the parent pull request early is a judgement call rather than a violation. The check is what holds, so “open the whole chain so I can review it” is fine as long as the pull request body says which pointers are branch heads.

What CI enforces

The rules above are enforced by workflows, because the failure they prevent happens at the merge button and an instruction file is not there. A chain of four pull requests was once merged bottom-up by hand and one of them landed with a pointer at a branch head, which needed a second pull request to correct. The instruction file that was supposed to prevent it said the right thing and was read by nobody at the moment that mattered.

.github/workflows/submodule-pointers.yml runs on every pull request that touches crab/** or .gitmodules. For each submodule it reads the pointer out of the tree, asks the GitHub API for that repository’s default branch, and compares the two. The pull request passes only when the comparison says identical — the pointer is the branch tip — or behind, meaning the pointer is an ancestor of the tip, which is the ordinary case for a deliberate lag. Anything else fails, naming the pointer and telling you to merge the child first.

Two details are worth knowing. The workflow clones nothing; it resolves everything through the API, so a private submodule would need no token of its own. And it iterates over .gitmodules, so it covers all four submodules even where a written rule has fallen behind and still names three. If a rule file and a workflow ever disagree, the workflow is the truth.

crab-exoskeleton-webapp/.github/workflows/mycelium-transport.yml fails a pull request that adds a REST call to the mycelium gateway outside its allowlist. The allowlist is the exception list, with a reason for each entry, and adding a line to it is a visible act in the pull request that needs it. See the webapp chapter.

The Dockerfiles of both Go components run go vet and the full test suite before linking the binary, so a red test means no image is built and nothing is published. In crab-shell-proxy, which has no pull-request workflow at all, that build is the gate.

What is not enforced matters just as much. Nothing runs the webapp’s test suite in CI, and nothing runs cargo test or cargo clippy for harness-sphere. Run those locally before you ask for a review; Working on the stack lists the exact commands for each repository.

A change, end to end

1. Write or update the specification under .specs/features/<slug>/ in the repository the change belongs to.
2. Make the change in the submodule, with its tests, and run that repository’s checks locally.
3. Open the submodule’s pull request. Its title, body and commits are in English.
4. When it merges, update the pointer in the product repository to the merge commit and say which pull request it names.
5. Open the product repository’s pull request. If you opened it earlier so the whole chain could be reviewed together, say in the body which pointers are still branch heads.

Where to go next

Working on the stack has the build and test commands. The component chapters — crab-shell-proxy, crab-ganglion-harness, crab-exoskeleton-webapp and harness-sphere — describe the code layout of each repository you might be about to change.