Architecture

Peen is a durable coding-agent service. A client opens a WebSocket, sends a task for a named session, and watches the model, tools, hooks, and child agents work. The default socket is a global event feed. Clients build session tabs by filtering event metadata, or ask the server to send one session only. Peen saves the work as it happens, so reconnecting after a client or process restart keeps the conversation.

WebSocket client
      |
      v
control plane: HTTP server, session hub, SQLite, worker supervisor
      |                                         ^
      | runTurn over a private Unix socket     | durable writes, then events
      v                                         |
session worker process ------------------------ +
      |
      v
agent runtime <--> model provider
      |                  |
      |                  v
      |              model events
      v
harness, tools, hooks, child agents
      |
      +--> workspace files and commands

REST sits beside the WebSocket. It reads stored session state, lists messages, protocol events, notices, compactions, model exchanges, jobs, and child-agent runs, and cancels work. It never accepts a user message. It starts a turn only when a posted notice wakes an idle session through an event handler. The API reference has the contract.

Aichteeteapee supplies Peen’s HTTP server, REST error envelope, and WebSocket hub. Peen gives every socket its own server-controlled identity in that hub. The server sends a session’s events to global sockets and to sockets filtered for that session.

Control plane and workers

One host runs one control plane. It owns SQLite, the REST listener, the global WebSocket hub, session routing, and worker lifecycle. It does not run the model loop. There are no controller IDs and no second controller.

Each session’s turns run in a worker process the control plane starts when the session first needs one. A worker never opens the control plane’s SQLite file. It reaches stored state through the control plane over a private Unix socket for its session, which carries JSON frames to run a turn, cancel it, signal a job, and shut down. Only the controller’s own user can open that socket, and the worker authenticates with a credential made for that one worker. The control plane stores only the SHA-256 hash of the credential, hands the raw value to the worker on stdin, and never writes it to a command line, an environment variable, a label, or a log.

Each session runs in an agent space the deployment offers, and the client names the agent space and nothing else. Images, mounts, network, the Docker socket, and sudo come from deployment configuration. The control agent space starts the same installed Peen binary with its internal worker command, as a child of the controller. A docker:<profile> agent space creates a container named peen-worker-<session-uuid> with two labels, peen.managed=true and peen.session=<session-uuid>, and the supervisor acts on a stored container ID only while both labels still match. Where it runs is the reference for agent spaces and Docker profiles.

A Docker worker runs as the controller’s own host UID, GID, and username. A controller on the host reads that account from the operating system. A controller running in Docker with numeric --user IDs reads PEEN_HOST_USERNAME and PEEN_HOST_HOME, so a worker can recreate the same host account. Every Docker worker container starts as root only long enough for the image entrypoint to create that account and drop to it. A profile that sets sudo: true also gets a passwordless sudo rule for that account. The agent is the host user either way. See sudo.

The controller passes each Docker worker the runtime configuration it needs to run a turn, including provider definitions and named provider credentials. It never passes the controller’s SQLite state, the API token, the controller’s Docker socket, or the entrypoint’s account settings. A Docker profile can grant a worker its own Docker socket with docker: full. That makes the worker host root, and the agent space’s capability warnings say so.

A profile with docker: readonly gets no socket. For each such worker the controller runs a Docker API proxy on a Unix socket in that session’s own directory, which the worker already mounts, and sets the worker’s DOCKER_HOST to it. The proxy forwards only a fixed list of read requests to the controller’s own daemon socket, hides environment values in inspect output, and stops when the worker stops. docker: readonly lists what it answers.

The controller decides at startup whether it has Docker access, by checking whether it can reach a Docker socket. Without one, it registers no way to start containers, and a session in a Docker agent space is refused. Peen never falls back to the control agent space, because quietly giving up a container would run the model’s tools in the controller’s own environment. With a socket the rule runs the other way, and control is never offered, because a control worker could use the controller’s socket to become host root.

Events reach clients only after they are stored. A worker’s turn writes through the control plane, and a feed pump reads each session’s newly committed rows in sequence order and fans them out to the WebSocket hub right after the write lands. A client therefore never sees an event that a reconnect and replay through the session feed would not produce.

One turn

The runtime recovers the root agent’s current directory and resolves its ancestor harness layers first. It builds the model context from the conversation, project rules, the names of the available skills and agents, and current runtime facts. The provider may then ask to use tools. Peen runs those tools, applies matching hooks, records the result, and returns it to the provider until the turn completes or fails.

A replacement worker claims an interrupted child when its next workflow activity starts. The activity and child ownership transfer commit together, retaining the original run identity and progress. A failed claim rolls back both records. Paused executions and children with requested cancellation or terminal state cannot acquire a new activity. A replaced worker cannot write new child checkpoints after ownership transfers. Child recovery describes pending tools, messages and limits.

File changes follow one safety rule. A turn must read an existing file before it can change that file, and Peen checks the file’s hash again right before the change. A new file must not exist yet. These checks stop a write based on an old or unread copy from replacing a file.

The harness

AGENTS.md carries project instructions in the agents.md format, and .agents/rules holds topic rules. .agents/skills lists named procedures that the model can load when needed. .agents/agents defines child agents for bounded jobs. .agents/events tells Peen how to wake a session for an outside event. .agents/hooks.yaml adds mechanical actions around turn and tool events.

Peen applies the configuration directory first, then filesystem layers from the root down to the agent’s current directory. A changeDirectory call reloads every harness piece for that directory and its ancestors. Child agents change independently. The harness guide describes the exact rules.

Durable state and visibility

SQLite is the only stored record of sessions, turns, messages, protocol events, context and prompt snapshots, compactions, child-agent runs and events, notices, process jobs and output, model runs, and individual provider rounds. A model run stores the requested and returned model identity, connection name, settings apart from secrets, messages, text, thinking, usage, retries, cost, timing, and failure details. Job signal requests are stored too. A message that a summary absorbs gets a link to that compaction, and the link never changes. A later summary links to its parent compaction instead of rewriting that older link, so a client can rebuild the summary tree at any time.

Each stored process job retains its source turn and, for a child command, its source child run. Directory changes and workflow restoration preserve that ownership.

Structured logs go to stdout and to daily audit files. The audit log records identifiers and digests that are safe to keep. The transcript holds the verbatim data, so protect its storage and every connected client as well as the workspace itself.

Opening a turn previews pending notices, then consumes its selected batch in the transaction that creates the turn and initial transcript. A rejected admission leaves notices pending. The turn_opening_notices table retains the accepted batch in notice order, with session ownership checked by both storage and foreign keys. Deferred retries restore the original batch for their prompt, leave newer notices queued and publish no duplicate opening delivery event. Live acknowledgement removes only accepted notice IDs. Automatic admission for child events validates its source ancestry in that transaction too; event wake rules describe which states block it.

Runtime startup opens peen.db for session records and workflows.db for the go-workflows backend, and control-core also opens metrics.db for stored observations. Each file has its own migrations. New execution, activity, and control-event tables extend the session schema. Workers access workflow storage over their authenticated session socket, with task leases bound to their worker generation. Runtime shutdown closes both files. Root and descendant agent loops execute through go-workflows and retain their session records for replay and inspection.

Every saved step of a turn carries the turn context: the system prompt, the tool descriptions, and the message history so far. Stored as written, that state grows with the square of the turn’s rounds. Peen stores each JSON string of 512 bytes or more, and each run of array elements of 512 bytes or more, once, in the content_blobs table of peen.db under the SHA-256 of its bytes, and writes a reference in its place. A run ends after an element whose own hash says so, never at a position, so appending to an array leaves every earlier run unchanged, and runs of references fold into runs again, like a tree. A growing message history therefore costs each saved step only its new messages and a few references. This applies to workflow payloads in workflows.db and to the input, output, and context of execution activities and executions in peen.db. Reading a record restores the exact bytes that were saved. A worker stores and reads blobs over its session socket, and the controller accepts only a blob whose bytes match its hash. A worker reads workflow history in pages of at most 8 MiB, so no history is too long to read through the 32 MiB frame limit of that socket.

Child question publication releases the notice gate before requesting a wake. Worker startup and root completion inspect pending questions in history order, recheck their source eligibility and offer one eligible wake. Final turn admission rechecks that source and consumes the retained notice in the same transaction. Paused ancestry and configured queue-only handlers leave the question pending. Missing child records leave their notices pending without blocking startup.

The checkpoint store saves activity input before work starts and saves output before acknowledging completion. It reuses completed output after reopening the database. An interrupted model request can start another attempt; an uncertain tool outcome requires a recorded inspection decision. Checkpoint reads preserve paused state, and a replaced worker cannot write through its old connection. The session-worker loop uses these records over worker RPC. Agents can inspect retained work without resuming it, then explicitly continue it. Resume and inspection decisions validate the calling tool’s live activity and the target’s inspected control revision in the same transaction. Inspection records caller provenance in the existing activity resolution event. Accepted outcomes reuse their saved checkpoint; retry decisions permit another attempt. Neither decision resumes a paused execution.

Each turn lease has a separate holder identity. Reusing a turn ID cannot revive the previous holder’s callbacks. Transcript and finalization transactions keep their lease until they commit, and recovery releases only the matching holder. Root execution lease records retain the holder across controller reopening. A replacement holder must name the previous lease, and pause/resume revisions reject stale restoration requests even in the same worker generation. Worker RPC binds restoration to its authenticated generation. Paused and finished executions remain unavailable for restoration, and an unrelated active turn keeps its lease. Root callbacks wait when restoration refuses admission because another turn owns the session. The refusal leaves the execution admitted, retains committed model and tool results, and records inspection evidence before allowing an unstarted effect to retry. Root completion signals admitted work after retiring its callbacks, and worker startup restores pending signals. Roots and children save their intended workflow identity and original round and parallel-tool limits before submission. Startup and handoff recreate a missing submission from those saved arguments. Paused work stays paused. Failed root submission releases its live turn holder only when the workflow backend confirms that no instance exists; an uncertain lookup preserves the holder. Explicitly resumed roots adopt the current session stop revision when their turn is restored.

A session worker’s stop command pauses unfinished workflow executions and cancels their current activities, including work reconstructed without the original caller. It also cancels activities whose pause was already recorded by the controller. Paused root cleanup drains callbacks and releases only its saved turn holder. Another request can start unrelated work while the old execution remains paused. Reconnecting a client does not resume that execution.

Stop advances a saved session revision and pauses existing unfinished executions in one transaction. Accepted requests carry their revision through controller and worker admission. A later Stop invalidates those requests, including deferred input that has not started. A delayed root workflow creation retains its original turn revision and starts paused after Stop, with its context intact. Cancelled preparations keep their admission position until cleanup finishes. A new message also waits for a stopped turn to release its lease and retire its queue before starting a new turn. Caller cancellation or another Stop cancels that wait without accepting the new message. Stopped deferred input remains stored. Explicit continuation names the inspected deferred request and stop revision, records caller provenance, and admits only that input. A deferred turn created before interruption keeps its identity and opening transcript.

Session-worker child launches use durable workflows too. Each child keeps its own execution ID, saved context and inbox, linked to its immediate parent and original root. Grandchild launches retain the child’s identity and depth. Parallel children can finish after their parent, and completed descendants retain their own transcripts and terminal records. Repeated submission reuses the child identity and preserves paused state. A failure before execution creation ends the accepted run; a scheduling failure after creation retains the durable record for recovery. Startup and root completion schedule eligible deferred input through the existing admission path when no admitted root owns the session.

Roots, children and grandchildren use the same workflow definition. Saved execution identities and activity results retain their original work across worker replacement.

Workflow activity leases expire one minute after their last renewal. Active workers renew them every 25 seconds. A replacement worker can reclaim an expired activity, while its former owner cannot renew or complete it. An interrupted tool handler with an uncertain effect requires inspection before retry; reclaiming its SDK task alone does not resolve that effect.

The workflow execution definition records prepared context separately from the model request. A model retry uses that saved context, including its effective prompt, without repeating the refresh. Saved root and child runtime context retains pending hook injections and denials by tool call. Restored scopes consume those messages and denials once. Preparation reads context and makes idempotent durable writes. Opening, tool, compaction and terminal hooks have separate durable effect checkpoints. A result that cannot be saved releases a retryable claim or records an uncertain tool outcome.

The root model bridge shares transcript, stream and audit callbacks with the turn runner. Audit settings identify the execution, activity, attempt and worker generation. An interrupted stream closes its partial content blocks before another attempt starts; partial output stays in events rather than becoming a completed assistant message. Saved context selects streaming or completion mode. Model activities return tool calls for separate activities, and the workflow refuses to dispatch them after its last permitted model round.

Root tool activities record paired transcript messages and tool content blocks through the turn runner’s callbacks. Their saved context includes the current directory, harness and prompt hashes, and observed file hashes. It also carries a versioned checkpoint of resolved instructions, skill content, agents, hooks, event handlers and budgets. Restoring that checkpoint verifies its digest and uses the saved content rather than reading changed source files. Restored reads retain the existing stale-content checks. Parallel result merging keeps every observed path and uses read revisions to retain the latest hash independently of call order. Conflicting hashes with the same revision refuse the merge. Root preparation applies an accepted directory hop before recording the next model input, including that directory’s prompt and tool definitions.

Saved workflow context also retains skill deliveries and the sequence through which history was compacted. Restoring the ledger preserves current-turn deliveries, while compaction removes only older history deliveries. Parallel tool results merge deliveries without reviving compacted skills. A successful directory change resets the old ledger, including a refresh of the same directory; a failed change preserves it.

Initial root workflow context includes the original request ID, turn and session IDs, caller prompt settings and event origin, alongside the frozen harness. Turn reconstruction reads the original SQLite record, checks the saved identity and accepted task, rebuilds tools and hooks around an independent executor, and claims the original turn under a new lease holder. It restores file-read proofs without opening a new turn or running opening hooks. Restoration validates the saved root agent against retained execution metadata instead of the current deployment default. Restored scopes keep that agent name for prompt and tool selection across directory hops, while new work uses the current default. Original-turn reads remain available for paused and terminal work without claiming it.

Native root callbacks share one scope and inbox arrival channel per execution. Preparation and model callbacks refresh saved state exclusively; tool callbacks can share the scope concurrently. Admission checks the durable holder and control revision again after waiting, and cancellation interrupts that wait. Explicit pause and resume replace the scope and holder without granting old callbacks the new lease. Tool checkpoints retain the effective system prompt, and directory-marker changes carry that prompt into the next model request. Final preparation captures the current root holder, control revision and directory. The completion transaction checks that proof before saving its decision, so holder replacement or pause/resume cannot let an old final callback finish the current execution. After a terminal checkpoint, the native final activity checks ownership again, drains callbacks, closes its live queue and child sink, and removes the cached scope. Continuation and paused decisions retain the scope. Cleanup repeats after checkpoint reuse, and a cleanup error preserves the committed result for retry. Native final checkpoints also commit the original turn state, remaining transcript and events, completion counters and transcript watermark in that transaction. They add the matching turn.completed or turn.failed session event to the final transaction. The controller publishes newly committed final events after the write; continuation, rollback and repeated completion publish none. The active turn lease is released only after a terminal commit. Pending continuation and failed transactions retain it. Repeated final requests validate their original turn payload and reuse the result after reopening. Terminal cleanup also releases the original local worker lease. It appends the retained final event to the embedding turn and marks its pending transcript saved without another database write. Failed turns mark their in-memory assistant and tool messages incomplete, matching the stored transcript. Native failures use agentRun for the turn classification and final event reason; the execution record retains diagnostic detail. Observer notification runs once; a failed sink retains the cached scope for delivery retry while its live queue stays closed.

Native root compaction checkpoints commit the replacement summary, direct message links and activity output together. The summary uses the activity identity, and repeated completion validates the original payload before reusing it. Invalid source history rolls back the entire completion. History holds every turn that has ended, including failed, cancelled and interrupted ones, so source selection spans them too. Every covered message receives a summary link, including the incomplete messages of a turn that did not finish; a turn still running is refused as a source. Named unfinished boundaries and skipped completed prefixes are refused. Assignment conflicts roll back the summary and every link. The workflow records preparation, compaction and model calls as separate activities in that order on each round. Its compaction result supplies the next model context, and completed activities reuse that context after reopening. Saved root context retains a versioned history map and summary chain head, plus the summarizer model, prompt, output limit and timeout. Restoration rebuilds the original compactor even when the deployment now defaults to drop-oldest. Malformed plans are refused during restoration and model preparation. The native root summarizer callback selects completed history with the provider or client token counter, preserves the newest completed turn, and runs the bounded summarizer inside the separate compaction activity. The completion payload advances the saved history map, summary head and skill watermark with the summary checkpoint. Empty or failed summaries retain raw history. A committed result is reused without another summarizer call. Compaction audits retain the execution, activity, attempt and worker identities.

Child summaries validate their message bounds, source count and current predecessor against their own transcript before writing. They refuse skipped complete history and overlapping or stale predecessors. Incomplete messages retain their content, flags and original summary links. Child compaction activity completion uses the execution’s saved child identity and commits the summary, direct links, activity output and execution context together. Reopening storage reuses the original payload, preserves paused state and rejects changed or omitted summary payloads. Invalid completion rolls back the activity result and summary. The authenticated worker transport carries the same completion contract. The native child summarizer callback validates the execution, saved model and prompt, predecessor and source boundaries before a provider call. It uses the saved summarizer settings and records the child, execution, activity and attempt in its audit. Failed or empty summaries leave raw history; completed results reuse the advanced history and skill state.

Native child context retains a versioned launch bootstrap with the child, session, parent turn, parent run, request and tool-call identities; definition, instructions and tool choices; original workspace, prompt and model identity; and maximum and already-consumed rounds. Parent identity and tool choices are detached when copied. Compaction checks the retained launch against its stored run. Directory changes can advance the current workspace and prompt while retaining the original launch. Model preparation checks the remaining round budget and session before resolving a model, so recovery cannot grant already-consumed rounds again. The launch metadata survives SQLite reopening. Initial child context also snapshots the child’s own harness, skill ledger, read observations, prompt, tool schemas and compaction map. Saved schemas and model metadata are detached from live objects. A model request uses the saved model metadata with the current connection client, so a changed catalog default cannot silently switch models. The existing child loop shares the preparation path used to build this initial context.

Cold child scope restoration reads the retained launch and harness checkpoint, creates a fresh executor from the current runtime and restores the child’s own read observations and skills. It keeps the saved prompt, directory, model metadata, compactor settings and round budget without resolving edited harness files or borrowing the parent’s executor or user queue. The restored dependencies use the current runtime and worker generation. Malformed or foreign state is refused. SQLite reopening and scope reconstruction preserve the child’s paused state and control-event count; reconstruction does not claim or resume an activity.

The workflow inbox stores accepted request IDs, source event IDs, original directories and resolved skill content. Acceptance commits the row and its created/queued events together. Preparation saves the queued IDs and their exact content in the model input. Delivery validates those IDs against the saved activity and commits delivered state, transcript messages and events together. Repeating an attempt reuses delivery without another transcript entry or live event. A later message with identical text remains separate and queued. Inbox reads preserve paused state. Both the service and embedded session runners use this durable inbox.

Child workflow inbox acceptance is scoped to the child execution and session. It retains text, resolved skill parts, source event and original directory in one transaction with an inbox.queued execution-history event. It does not require a live parent turn lease, so an unfinished child can accept a message after its parent turn finishes. Repeated identical request IDs reuse the saved row without another history event; changed requests, exhausted capacity, foreign sessions and new acceptance into terminal children are refused. A new message to a paused child resumes the child’s paused subtree, records execution.resumed with the message’s request ID, and queues the rendered pause reason ahead of the message, all in the acceptance transaction. A refused message leaves the child paused. Reopening and repeating an already-accepted request preserves pause. Error notices reach queued and running children through rows that bypass the inbox capacity, with the notice ID as the source event. Model-owned delivery checks the saved activity’s request IDs and commits the child transcript, delivery identity and inbox.delivered history together. A retry of the same activity after storage reopens creates no duplicate transcript. Missing or sibling requests roll back the delivery batch. Paused execution and replaced activity claims cannot deliver messages. The worker transport uses the connection’s authenticated session and generation.

Native child inbox adapters validate the saved launch against the child execution and stored run before reading queued messages. Preparation adds accepted text and resolved skills to the child’s context and skill ledger. Delivery requires the serialized model round to match the activity’s saved input, so later arrivals remain queued. The adapters refuse paused execution, changed ancestry and reset round budgets.

Native child round preparation snapshots the current directory scope’s prompt, tool schemas, harness, read proofs and skill deliveries with the accepted inbox content. It retains the original launch and round budget. Completed preparation checkpoints reuse their saved output when retried, so a later arrival cannot change that round’s model input.

The native child model bridge validates the stored child identity and sink before calling the provider. It delivers only the inbox IDs in the saved model input and uses the saved model metadata on the current connection. Each attempt records a child model audit linked to its execution, activity, attempt and worker generation. Child callbacks record text, reasoning, retries and assistant messages. The model returns tool calls for separate activities. A queued message arriving during the provider call requests another round. Completed model checkpoints reuse their output without another provider invocation. Child model callbacks also buffer the complete assistant message and its event until the model result commits. Partial text, reasoning and provider audit attempts remain visible during the request. A failed result write leaves no completed assistant message; recovery can retry the request. A lost acknowledgement or failed publication reuses the committed output without another provider call. Publication checks the exact saved input and output. Pause retains the result until explicit resume.

Native child tool activities validate the saved call and active attempt before running the existing tool hooks and handler. They record child tool results and injections and retain the current directory, harness, prompt, skills and read proofs in the checkpoint. Completed reads reuse their original output after a file changes. A successful directory hop retains the target’s instructions without replacing the original launch identity. Child context preserves an unrestricted tool choice separately from an explicit empty list, so restoration does not remove originally available file tools.

Native child finalization commits the child run’s terminal response, reasoning, history and usage with the final activity and execution state. The child outcome must match its execution outcome, and a root execution cannot carry a child outcome. Queued messages refuse successful completion or commit a continuation decision while leaving the child run active. Repeating the saved final decision after storage reopens reuses the result; changed child response data conflicts.

The same finalization transaction records the child’s agent.run.completed or agent.run.failed event and increments its durable event count. Failure events retain the failure reason and classification. A repeated completion, including after storage reopens, retains the original event ID, sequence and payload. Event-counter exhaustion rolls back the child outcome and activity result. Child terminal cleanup publishes this committed payload without appending another event row.

Native child final preparation frames its completion or failure as an agent message. The final transaction places that report in the parent execution inbox with the terminal event ID as its request and source identity. Root and nested parents use the same queue. Repeated completion, including after SQLite reopens, retains one report row; changing the report conflicts. Completion reports can be queued when the ordinary message limit is full. A paused parent retains its pause state and control revision. Parent event-counter exhaustion rolls back the child response, activity, event and report together. The authenticated worker transport carries this completion payload. After terminal cleanup, the native dispatcher reconciles queued reports into an already cached parent inbox and signals newly observed request identities once. It checks the current control revision and root holder before notification. Paused, terminal, cancelled or superseded parents receive no arrival signal, and an uncached parent is not reconstructed.

Native child registry restoration retains the stored run ID, ancestry, request, start and end times, lifecycle state and bounded event buffer. Repeating the same identity reuses the entry. Foreign sessions, malformed records and a changed cached identity are refused. Reconstruction writes no durable records and starts no child loop, so a paused execution keeps its state and event count. A restored active entry uses the supplied cancellation function; terminal entries keep a closed completion channel.

Native child inbox bindings reconstruct accepted request IDs and text/skill slot counts from every queued page. Binding checks the owning child and changes no execution state. Message acceptance uses the durable child inbox and signals arrival only for a new queued identity. A model activity clears the in-process arrival count after its saved IDs commit as delivered. Closing the binding leaves queued rows available after storage reopens. The manual provider queue cannot write delivery for a native binding.

Cold child callback assembly validates the saved execution and launch, rebuilds a private actor with its original identity, restores its frozen scope and binds the durable inbox before returning it. It starts no child loop and writes no run record. The supplied cancellation controls that actor. Paused execution is refused. Shared child resolution serializes construction and installs the fully bound actor in the session registry. Repeated resolution at the same control revision retains that actor and inbox. Pause refuses resolution; explicit resume replaces the actor, cancels its old lifetime and closes its old inbox while retaining queued SQLite messages. Another callback owner cannot replace an installed registry actor. Registry cancellation reaches the shared lifetime. Child callback admission links the activity context to registry cancellation and supports shared readers and exclusive state changes. Admission rechecks the saved control revision and registry actor after acquiring the scope. Resume drains old callbacks before installing a replacement. Refresh restores saved scope state while retaining the actor and inbox channel, and reloads queued identities and the current directory under the inbox lock. The child callback set binds preparation, compaction, model, tool and final preparation to this admission path. Preparation and model callbacks refresh their saved scope exclusively; tool callbacks share the scope. Final preparation uses the last model result context, which the workflow schedule retains separately from the round input. The child workflow retains parallel read proofs and directory-hop instructions in its next round and final response history. Terminal cleanup reads the committed child state, end time and last event, checks the cached control revision and drains callbacks. It closes the inbox and completion channel, copies the stored terminal state into the live registry and publishes the retained event payload. A failed live sink retries without another database event or provider call. Repeated cleanup in the same callback instance does not deliver the terminal payload again. Cleanup after SQLite reopening and a fresh runtime reconstructs a terminal registry entry without restoring an active scope.

The native phase dispatcher reads the owned execution record and selects its root or child callback set from the stored kind. Model context cannot select another actor type. Preparation, compaction, model calls, tools and finalization use the same dispatch path. The root route completes the original turn; the child route updates its registry and delivers its committed terminal event. Missing storage, a foreign execution identity or malformed actor context is refused before a provider call. Native worker assembly binds these callbacks, recorded result merging and the workflow definition to one session queue for both workflow and activity tasks. Assembly validates the lifecycle, execution storage, backend and generation before registration. A session worker starts native tasks over its registered controller connection before admitting commands. Shutdown cancels and drains those tasks before closing the protocol read loop. Startup failure also cancels and drains partially started tasks. New native client submissions derive a stable SDK execution ID from the session-scoped instance ID. Repeating creation reuses an existing active or completed instance and its recorded history. After a creation error, the client checks for that exact saved instance before deciding whether submission failed. Startup reconstructs missing root submissions from their saved execution records. The native root submission handoff snapshots the prepared scope, records an execution for the original turn and adopts its existing lease. It binds the durable inbox and retains the prepared callback scope before creating the SDK instance, preserving opening events and the caller sink. The session-worker runner uses this handoff and returns saved text, reasoning, model, finish reason, token usage and retained events. Explicit cancellation records a pause, cancels owned activities and releases the local turn lease so unrelated work can start. Caller interruption without a durable stop request leaves the execution running. A duplicate or foreign-session native worker cannot replace the bound runner. Embedded runtimes start the same native worker for each stored or newly opened session. Their session runtimes share storage, configured extensions and the global turn limit. Closing drains calls, pauses unfinished work and drains workers before closing storage.

The workflow schedule separates each tool’s pre-hook, handler, outcome hook, post-hook and result projection into durable activities. Each phase receives its predecessor’s saved context and result. The phase runner also retains event arguments, messages and handler-error classification. It preserves the call’s original timeout, including an unbounded timeout, across changes to deployment defaults. Cancellation from a handler, hook or injector interrupts the activity before another phase starts. An early failure skips later effect phases and still projects its result. Tool calls retain bounded concurrency. A phase that fails inside Peen reaches the model as an error result saying the call’s outcome is unknown, Peen does not repeat it, and the turn continues. An opening, compaction or terminal hook step that fails inside Peen is recorded as a failed hook in the same way. Hooks describes what each failed step does to the turn. A phase that a crash or a worker shutdown interrupted keeps its unknown effect for inspection. Completed phase results can be reused after storage reopens. Phase state can retain a staged directory change with its frozen target harness, prompt, skills and hook state. Restoring that candidate leaves the current directory intact until the call’s post-hook accepts it. Restored tool bindings do not repeat the pre-hook. The pre-hook checkpoint for waitMessages also records a stable wait identity and its absolute requested deadline. Its prepared passive handler may retry after interruption; hooks retain reconciliation. Root and child waits reuse the saved deadline after SQLite reopens, including an already expired deadline. Invalid argument shapes and durations follow the ordinary tool error path. Root ask commits its question with the pre-hook checkpoint, retaining question text, wait identity and the original capped deadline. Its passive handler waits without asking again and retains the original answer or timeout outcome. The projection checkpoint commits question.answered with the tool result. Question events carry the call ID so history can associate them.

Root and child callbacks use the stored execution kind and validate each phase’s exact activity claim, attempt and worker generation before effects. Each callback restores a private tool scope while retaining the shared root queue or child inbox. Result streaming retains the pre-hook attempt’s stream position across later phase attempts. A child launch must belong to the handler phase of its parent call. New session-worker conversations use this schedule.

Activity completion can atomically append a root transcript payload with its result and completion event. The payload carries the original turn holder and control revision. The controller refuses a replaced holder, a paused execution or a child activity before committing any transcript data. A saved payload hash rejects changed repeats, including omission of a previously committed payload. Identical repeats after SQLite reopens return the saved result without appending messages or publishing events again. This storage path is available over authenticated worker RPC.

Child activity completion can also commit a question and its parent delivery together. A root parent receives a pending agent.question notice; a nested parent receives a queued execution inbox message. Both use the child’s question event ID, so repeating completion after storage reopens preserves the original delivery. The controller derives the destination from stored ancestry and rolls back completion if delivery cannot be saved. A paused parent stays paused, and questions can be retained when its normal message queue is full. Child pre hooks save a stable wait ID, question and original deadline. A separate retryable delivery phase commits the question before its passive handler waits for the answer. Lost acknowledgements reuse that delivery; a replacement runtime can continue the handler after SQLite reopens. The answer commits with the final tool projection. A refused delivery reaches the handler as a saved error and still runs outcome and post hooks. Re-entering the delivery phase publishes the original root notice once per runtime and leaves its durable state pending until consumption. Consumption acknowledges only returned notice identities, preserving newer arrivals in the live queue.

Root tool phases collect tool-start, result, injection and tool stream records in private buffers, then commit them with the phase result. Retained start time preserves elapsed tool duration across phase restoration. After commit, the callback verifies the saved input and output before updating its live transcript. It keeps unrelated pending writes pending. Reusing a phase does not append its records again in the same callback scope; a failed sink retries from the event it could not deliver. A replacement callback can restore these records after SQLite reopens without repeating the phase. A stop between commit and delivery returns the paused decision, retaining the saved result for explicit resume. Root model callbacks collect complete assistant messages in a private turn and commit them with the model result. Live content blocks, audit attempts and inbox handling retain the shared turn. Failed result writes leave no completed assistant row; lost acknowledgements reuse the saved output.

Child activity completion accepts a separate transcript payload for the execution’s own agent run. Child messages, directory metadata, events and their counter commit with the activity result. The controller checks the saved control revision, running state and worker ownership, and refuses stopped children. Payload validation or event-counter exhaustion rolls back every write. Identical completion repeats retain the original messages and event identities after SQLite reopens; changed or omitted payloads conflict. The authenticated worker transport carries this payload. Child tool callbacks collect their records in a private sink until completion, then verify the saved phase and find its events by retained ID across cursor pages. Live publication uses their committed sequence and timestamp. A sink failure can retry without another transcript write. Successful directory changes update the shared inbox after their phase commits. Error results retain their tool message without successful-directory metadata. Child message and event inputs use camelCase JSON keys and continue accepting their earlier field-name spelling. Before phase delivery, the callback rebuilds its bounded event window from SQLite in stored sequence order. Parallel phase publication cannot discard an earlier committed event from that window. Count and byte eviction still apply. A fresh runtime can reuse a committed child pre phase after SQLite reopens without another transcript write. Pausing between commit and publication retains the phase result for explicit resume.

Native queue retries read frozen acceptance by request ID through an indexed query scoped to the session and execution. This read works over authenticated worker RPC for queued or delivered messages and preserves paused state. A missing request leaves new acceptance to resolve its current skill content.

Root executions run turnStop in a separate activity before each completion decision, using the saved final response and original turn identity. Input already queued skips that hook attempt and continues the model loop. A completed hook checkpoint is reused after SQLite reopens. Interrupted hook effects require reconciliation before continuation. Children do not run parent-turn lifecycle hooks. The completion reservation records the inbox watermark before terminal hooks. Later user input is retained for a subsequent turn. Root opening and root/child compaction hooks use the retained checkpoints described in Hooks. Failure finalization uses its own durable checkpoint. Stop pauses work and does not run terminal hook effects; see the lifecycle event contract in Hooks. A durable completion boundary records execution.completionSealed with the exact terminal hook activity, input hash, control revision and inbox watermark. Reservation refuses queued input, unfinished activities and replaced claims; rereading a saved boundary preserves pause after reopening. Workers can reserve this boundary over their authenticated session connection. The controller checks the current worker generation and exact activity lease. Pending input returns an empty reservation; repeated reservations retain the saved boundary without changing paused state. Finalization validates the saved hook input and requires its activity to be completed before applying the inbox watermark. Child terminal reports arriving later remain queued and readable after the parent completes. Their terminal event must belong to a child of that parent; arbitrary source IDs and user messages still block completion. Authenticated worker RPC supports explicit deferral, retaining late user text and resolved skill content in a separate inbox state after validating the reserved completion. The controller checks the current worker and original turn lease, publishes acceptance events once, and preserves invalid-state refusals across the socket. Creating a subsequent root can atomically mark that input forwarded and link its original inbox record to the new execution. The handoff validates the request, text, directory and source completion; replay retains the link even if the target is paused or has changed directory. Deferred root preparation restores the accepted directory, effective model, reasoning setting and frozen skill parts from the completed source. A skill missing from the current harness does not discard accepted input. Normal new messages still validate against the current harness. Root submission carries the deferred inbox identity into the atomic handoff. Authenticated workers can page through unforwarded input from completed roots in source creation order, then inbox sequence order. Reading leaves paused and unfinished work untouched. Preparing a deferred turn emits userMessage.delivered with the original request and source identity. Deferred admission derives one stable turn identity from the inbox record and writes its initial transcript in the same transaction. A retry after reopening reuses that turn and returns no new opening events. A persisted cancellation prevents admission from restarting the turn. Workers reach admission through the authenticated generation-bound connection.

Native root queues accept through that inbox and reconstruct pending arrivals when the original turn is restored. A repeated request ID reuses its original skill content and directory, while changed text or source identity conflicts. Acceptance releases the queue lock before calling event sinks, so a sink can submit another message. Model results record whether queued arrivals require another round, including after an answer without tool calls. Continuation keeps the existing round budget. Successful finalization checks for pending inbox messages in its completion transaction. A validated completed hook boundary excludes only later child terminal reports; other queued messages still require continuation. Pending-inbox refusal has a distinct error identity over worker RPC and still matches the conflict class. It does not classify a stale claim or pause refusal as pending work. The workflow supplies a saved continuation outcome for pending input and starts another model round within its remaining budget. Each final decision has its own stable step ID, so repeated arrivals do not change an earlier activity input.

Workflow model streaming reads the original turn’s retained content block starts and stops before another request. It closes unfinished blocks and continues at the next index. Saved model output records the tool batch’s stream positions, so a reconstructed tool activity can emit its use and result blocks without replaying the provider or old blocks. Parallel tools use independent adapters, and attempts occupy distinct slots. Index translation preserves JSON values exactly and refuses invalid indexes or integer overflow.

Final checkpoints save the activity result, terminal execution state, final context and completion event in one transaction. Reopening storage reuses that result if native workflow completion was not acknowledged. A pause that commits first prevents finalization until explicit resume. Finalization refuses unsettled activities, and success also refuses failed activities or pending inbox messages. A refused inbox check rolls back the activity result and completion event. Failure can retain pending messages for history, including when the round budget is exhausted. A caller can supply an alternative continuation output for pending input. The transaction saves that output and a completion-request digest while leaving the execution unfinished. Without pending input it records terminal output and the same request identity. Repeated requests retain the chosen outcome after reopening; changed output, context, directory or alternative output conflicts. Reusing a saved continuation decision preserves paused state. The workflow consumes the committed branch when deciding whether to schedule another model request. Parallel tool batches wait for every scheduled tool even when another fails. Paused steps recheck their durable decision before waiting, so one explicit resume can continue the execution’s parallel steps.

Process lifecycle

Peen uses Servicepack for process and service lifecycle. Peen owns the agent behavior, public API, storage, and harness. Servicepack’s own details live in its repository.

peen run starts two services:

control-core  ->  control-api

control-core loads and checks the configuration, takes an exclusive lock on <PEEN_STATE_DIR>/controller.lock, opens SQLite, and builds the provider registry, the workspace policy, the agent spaces, and the worker supervisor. It holds the lock for its whole lifetime, so a second controller using the same state directory fails before it can recover or change another controller’s work. The operating system releases the lock when the process exits, including after a crash. It reports ready only after its dependencies exist, and it creates no session, so a controller starts empty.

control-api depends on control-core, so Servicepack starts it second and waits for control-core to be ready first. It owns the REST listener, the global WebSocket hub, and the metrics listener, and it reports ready only once its endpoint accepts a connection. A client that waits for readiness can send a request right away instead of racing a listener that has not started yet.

peen exec first probes the configured controller and reuses it when it answers. Otherwise it runs the same controller services inside the command process. That controller serves REST and WebSocket through an in-memory listener, opens no TCP or metrics listener, and follows the same readiness and shutdown ordering. After the turn, the command waits for its controller to stop, including workers and SQLite, before exiting. A borrowed controller keeps running. The workspace session and its history survive either path.

Servicepack stops the two in reverse order, so the API stops accepting work before control-core cancels active turns, stops running jobs, stops every session worker, and closes SQLite. Workers stop after jobs, so a container is not removed while a process still runs in it, and Peen records each stop, so no worker generation stays marked ready after its worker is gone. Servicepack handles the stop signal and the time limit on the whole shutdown.

In-process embedding

The Go embedding guide describes running the same engine inside another process. That runtime owns its SQLite connections, session workflow workers, jobs and child tasks directly, and its caller closes it with Runtime.Close. Session handles expose stored replay and controls without starting the controller or worker transports. A supplied filesystem affects workspace tools, prompts and harness discovery; configuration, state and audit logs remain in separate host directories. Caller-provided logging and metrics stay local to that runtime.

Terms

Several words in these docs name different things on different pages. This is what each one means.

Term Meaning Where it is defined
hook point A fixed point in a turn where hooks run, such as preWriteFile or turnStart. The top-level keys of hooks.yaml name hook points. The hooks page calls them hook events, and the command stdin field event holds one. Hooks
session event A record of something that happened outside the model’s own tool calls, such as job.exited, agent.finished, or a type a hook or outside caller chooses. Peen hands it to the model as data. The API calls it a notice (/api/v1/session/notices). Session events
protocol event A stored record of one step of a turn, such as turn.started or tool.use. GET /api/v1/session/events lists them, and each one also goes out live as a WebSocket frame. Protocol event types
WebSocket frame One JSON message on the WebSocket, in either direction: a client’s message.send, a protocol event, or a message.completed or message.failed reply. WebSocket
upstream One entry in PEEN_UPSTREAMS. The docs also call it a provider, and the API calls its name connectionName. All three mean the same configured entry. A qualified model name is <upstream name>/<model ID>, such as zai/glm-5.3. Providers
agent space The place a workspace’s agent runs: control, next to Peen, or docker:<profile>, in its own container. A Docker profile is one entry in PEEN_DOCKER_PROFILES. Where it runs
control space Where Peen itself runs when you install it with the installer: host or docker. Where it runs
worker generation One run of a session’s worker process. A session gets a new generation each time Peen starts its worker. Workers

The agent. prefix appears in two places. agent.finished, agent.failed, agent.message, and agent.question are session events. agent.run.started, agent.run.tool.use, and the other agent.run.* names are protocol events of a child agent run.

A child agent runs in its session’s worker, on its own, and can outlive the turn that launched it. It talks to the agent that launched it through messages and session events. See running children.

The field name type also means different things. On a hook action it is the action kind (deny, inject, emitEvent, command). On a session event it is the event name, which an emitEvent action spells eventType. On an upstream it is the wire protocol. On an event handler it is the session event type the handler answers.

Identifiers

Identifier What it names Who creates it Where you see it
session ID One session, which is one workspace’s conversation. Peen, the first time POST /api/v1/sessions/open opens that workspace. X-Session-ID, frame metadata.sessionId, hook stdin sessionId.
request ID One accepted message.send, or one turn an event woke. Peen, when it accepts the frame or starts the woken turn. Frame metadata.requestId, turn records, hook stdin requestId. Every frame of one turn shares it. A queued message keeps its own.
turn ID One stored turn. Peen, when the turn starts. GET /api/v1/session/turns, turnId on stored messages, protocol events, and jobs, hook stdin turnId.
call ID One tool call inside a turn. The model provider. callId on tool.use and tool.result, id and tool_use_id on content blocks, toolCallId on stored messages, hook stdin callId.
agent run ID One launchAgent child run. Peen, when the child starts. GET /api/v1/session/agents/{agentRunId}, runId in agent.* session event data, hook stdin agentRunId.
worker generation ID One run of a session’s worker process. Peen, when it starts the worker. GET /api/v1/session/workers, workerGenerationId on turns, protocol events, and jobs.
HTTP request ID One HTTP call. It has no link to the request ID above. The client in an X-Request-ID request header, or Peen when the client sends none or an invalid one. The X-Request-ID response header and the logs for that call.

External building blocks