Compare commits

...

17 Commits

Author SHA1 Message Date
James Long 36f2b4a939 feat(server): add lazy simulation layer replacements 2026-07-02 20:23:21 +00:00
James Long b52c14ec69 docs: move simulation specs into specs/simulation, drop research archive 2026-07-02 18:51:00 +00:00
James Long e3414b0ea8 refactor(tui): share renderer options with createSimulation 2026-07-02 18:44:42 +00:00
James Long 8b5f7af1b9 refactor(tui): early-return simulation renderer in app.tsx 2026-07-02 18:40:58 +00:00
James Long 3dc36ec38b refactor(tui): collapse simulation activation into renderer creation
app.tsx now has a single simulation touchpoint inside the renderer
acquire: in simulation mode it delegates to Simulation.createSimulation,
which picks the fake or visible renderer and starts the control server.
The server is tied to renderer destroy, so no extra finalizer is needed
in app code.
2026-07-02 18:39:27 +00:00
James Long bab30226b9 refactor(tui): move fake renderer swap into single simulation block 2026-07-02 18:37:20 +00:00
James Long 7a76efc4f8 refactor(tui): keep fake renderer setup out of app code
Move fake renderer creation into simulation/renderer.ts, which keeps the
TestRendererSetup in a module-side WeakMap. createHarness looks it up by
renderer, so app.tsx only picks which renderer factory to call.
2026-07-02 18:26:39 +00:00
James Long 6fbce7b045 feat(tui): support fake renderer in simulation mode
- OPENCODE_SIMULATION_RENDERER=fake creates an OpenTUI test renderer
  (in-memory screen buffer) instead of the terminal renderer; size via
  OPENCODE_SIMULATION_TUI_WIDTH/HEIGHT
- Harness carries mockInput/mockMouse and prefers TestRendererSetup
  APIs (captureCharFrame, renderOnce) over the private
  currentRenderBuffer fallback used for the visible renderer
- Fix Bun.Server generic type argument in simulation server
2026-07-02 18:09:48 +00:00
James Long 1d85015e17 chore: revert stray whitespace change in runtime.lifecycle.ts 2026-07-02 17:42:31 +00:00
James Long da21f109df docs: record anchor-directory and snapshot decisions in simulation specs 2026-07-02 17:31:16 +00:00
James Long 6730dc3a35 feat(tui): wire simulation control server 2026-07-02 14:51:23 +00:00
James Long 6281f17124 fix: avoid simulation hook in mini mode 2026-07-02 14:51:23 +00:00
James Long 640b42b0a1 feat: add simulation control surface 2026-07-02 14:51:23 +00:00
James Long 7fb0f462fa docs: track simulation phase 1 tasks 2026-07-02 14:51:23 +00:00
James Long 59df1b8be6 docs: add simulation implementation phases 2026-07-02 14:51:23 +00:00
James Long 761ca72620 docs: add simulation config generation 2026-07-02 14:51:23 +00:00
James Long 27a0b28858 docs: add simulation architecture spec 2026-07-02 14:51:23 +00:00
10 changed files with 1149 additions and 14 deletions
@@ -0,0 +1,170 @@
# Simulation Implementation Phases
Status: implementation plan for `specs/simulation/simulation.md`.
The full simulation architecture is intentionally broad. This document breaks it into phases that can be implemented and reviewed incrementally.
## Phase 1: Control Surface And Observability
Goal: start the normal app in simulation mode and inspect/drive the TUI through an external WebSocket driver.
This phase proves the core shape without swapping every foundational layer yet.
Implementation checklist:
- [x] Add `OPENCODE_SIMULATION=1` activation in V1/full-TUI startup.
- [x] Add simulation trace service with in-memory append-only records.
- [x] Add OpenTUI UI state extraction for screen, focus, elements, and generated actions.
- [x] Add OpenTUI UI action execution for typing, keys, enter, arrows, focus, and click.
- [x] Add reusable JSON-RPC WebSocket server on `127.0.0.1:40900+`.
- [x] Expose `ui.state`, `ui.action`, `ui.render`.
- [x] Expose `trace.list`, `trace.clear`, `trace.export`.
- [x] Wire visible V1/full-TUI renderer path through the same action protocol.
- [ ] Verify a local driver can inspect state and execute a real TUI input.
Scope:
- Add `OPENCODE_SIMULATION=1` activation.
- Start a TUI-owned JSON-RPC WebSocket server on `127.0.0.1:40900+`.
- Expose `ui.state`, `ui.action`, `ui.render`.
- Use the old simulation action model: type text, press keys, press enter, arrows, focus, click.
- Support fake OpenTUI renderer and visible renderer through the same action protocol.
- Add in-memory append-only trace with `trace.list`, `trace.clear`, `trace.export`.
- Record UI observations, generated actions, executed actions, errors, and render/stabilization events.
Done when:
- `OPENCODE_SIMULATION=1 bun run dev` starts the normal app.
- A local driver can connect to the WebSocket.
- The driver can inspect current screen/elements/actions.
- The driver can execute real TUI inputs.
- The trace shows observations and actions.
Out of scope:
- Backend layer replacement.
- Model-based runner.
- Generated plugin config.
- Deterministic replay tests.
## Phase 2: Foundational Simulation Layers
Goal: make the app safe and controlled by swapping the lowest layers, not app logic.
Scope:
- Wire simulation replacements through `AppNodeBuilder.build(...)` and `AppNodeBuilderV1.build(...)`.
- Create a real, empty anchor directory (`mkdtemp`) and `process.chdir` into it before any command resolves its working directory; skip creation when the runner already spawned the app inside an anchor.
- Root the in-memory filesystem at `process.cwd()` (the anchor). No cwd monkey-patching: cwd, `$PWD`, and `path.resolve()` stay truthful.
- Add snapshot loading from `OPENCODE_SIMULATION_STATE`: read the snapshot directory once at startup and seed the in-memory filesystem (snapshot `project/` paths joined onto the anchor root), config, env, and optional LLM/network state from it.
- Route config/data/state/cache/temp paths into the simulated space using existing env seams (`OPENCODE_CONFIG_DIR`, `OPENCODE_TEST_HOME`, `OPENCODE_DB=:memory:`), set before `packages/core/src/global.ts` import-time path setup runs.
- Deny host filesystem escapes loudly (paths outside the anchor root fail with typed simulation errors).
- Assert the anchor directory on the host is still empty at the end of the run; anything written there means a code path bypassed the simulated filesystem.
- Add simulated network registry and deny unknown external network by default.
- Add scriptable LLM boundary.
- Add simulated process registry:
- shell through `just-bash` against the simulated filesystem.
- minimal fake `git` support for discovery/status paths.
- deny unsupported process spawns.
- Add simulation-gated backend control routes, proxied only through the frontend WebSocket.
- Expose backend methods through the frontend server: filesystem seed/write, network register, LLM enqueue, backend snapshot.
- Trace filesystem, network, LLM, process, and backend control activity.
Done when:
- Unknown network fails with a simulation error.
- Host filesystem escape fails with a simulation error.
- The anchor directory on the host is empty after a run.
- The app boots from a snapshot directory via `OPENCODE_SIMULATION_STATE` and observes the seeded project files, config, and env through normal app paths.
- A driver can seed a project filesystem.
- A driver can enqueue an LLM script and submit a prompt through the TUI.
- The real session/tool path consumes the scripted LLM behavior.
- Shell commands use `just-bash`; unsupported process spawns fail.
- Trace contains backend activity and snapshots.
Out of scope:
- Model-based generation.
- Generated plugin config state.
- Shrinking.
## Phase 3: Generated Config And Model-Based Runner
Goal: explore different app states using generated commands and plugin-provided config state.
Scope:
- Add generated simulation plugins as the primary config-state generation mechanism.
- Support generated plugin domains for:
- agents and defaults.
- provider/model availability.
- tool definitions and scripted tool behavior.
- MCP-like capabilities or endpoints.
- permission policies.
- instructions/system-context-like inputs where supported.
- workspace/project adapters where supported.
- Add runner commands to generate, enable, disable, and inspect generated plugin state.
- Build a custom external model-based runner, not `fast-check` yet.
- Runner command shape: precondition, execute, model update, postcondition.
- Runner model tracks only high-level observational state: screen category, prompt availability, sessions, files, queued LLM scripts, generated plugins, backend status, idle expectation.
- Generate valid command sequences from model state and current `ui.state.actions`.
- Record seed, command distribution, precondition rejections, generated plugin/config domain coverage, UI action coverage, and backend event coverage.
Done when:
- A seeded runner can generate a short valid exploration.
- The runner can generate plugin-provided config state without generating large arbitrary config files.
- The app loads and observes generated plugin state through normal plugin/config paths.
- The runner can type and submit prompts through the TUI using generated actions.
- Basic properties run after commands: no crash, no unknown network, no host FS escape, coherent stabilized state.
- Trace export includes enough state to replay the generated run later.
Out of scope:
- Shrinking.
- Coverage-guided mutation corpus.
- Differential testing.
- CI randomized runs.
## Phase 4: Replay, Promotion, And Campaigns
Goal: turn exploratory simulation into durable tests and prepare for larger campaigns.
Scope:
- Add replay from exported trace.
- Add deterministic replay test generation from successful or failing traces.
- Add stronger trace schema validation.
- Add property families beyond no-crash:
- durable prompt admission is not lost.
- no duplicated visible message IDs.
- no orphan tool results.
- queue/steer semantics hold at stabilization boundaries.
- interrupt/resume does not duplicate promoted inputs.
- Add corpus storage for interesting traces.
- Add simple coverage/novelty scoring over UI states, backend event types, tool outcomes, generated config domains, and errors.
- Add long-running campaign mode outside normal CI.
Done when:
- A trace from Phase 3 can be replayed deterministically.
- A trace can be promoted to a normal test fixture.
- Campaign runs can collect interesting traces without committing randomized tests to CI.
- Failures produce a compact reproduction command and trace export.
Out of scope:
- Full shrinking.
- Deterministic scheduler/clock control.
- Parallel campaigns.
- Differential testing across app versions.
## Later Work
- Shrinking failed traces.
- Coverage-guided mutation of structured traces.
- `fast-check` integration if the custom runner becomes too limited.
- Differential testing across versions, renderers, storage modes, or scheduler policies.
- Deterministic clock/random/scheduler control.
- Parallel isolated workers.
- Model-generated properties with validity/soundness/coverage scoring.
@@ -0,0 +1,489 @@
# Opencode Simulation Architecture
Status: first milestone architecture draft.
## Goal
Build a simulation environment for exploring opencode through the real app, primarily through the TUI, while replacing only the lowest foundational layers needed to make runs controlled, observable, and safe.
The first milestone is an interactive exploration and model-based testing environment. It should be enough to start opencode normally, put the app into generated states, drive real user-level TUI actions, observe what happened, and record an in-memory trace that can later be exported into deterministic replay tests.
This is not intended to be a custom simulated app or a separate `simulate` command. The normal app should run, with simulation enabled by one required flag:
```sh
OPENCODE_SIMULATION=1 bun run dev
```
## Non-Goals
- Do not reimplement the app.
- Do not replace mid-level services like session processing, tool registry, provider orchestration, route trees, or TUI components unless a foundational seam proves impossible.
- Do not build shrinking in the first milestone.
- Do not make generated randomized runs part of CI yet.
- Do not build differential testing in the first milestone.
- Do not expose simulation controls when `OPENCODE_SIMULATION` is not set.
## Design Principles
- Run the real app through normal commands.
- Drive the TUI using real user-level input: typing, keypresses, focus, click, and mouse actions.
- Keep simulation code isolated under a simulation/testing area.
- Touch production app code only at narrow activation points: builders, TUI startup, foundational layers, and simulation-gated backend routes.
- Swap foundational layers, not app logic.
- Make observations rich enough for humans and models.
- Treat traces as first-class artifacts.
- Use a lightweight model of expected high-level behavior, not a clone of opencode internals.
- Generate valid commands from current observed state rather than blindly fuzzing impossible actions.
## Activation
`OPENCODE_SIMULATION=1` is the only required flag.
Initial state is provided through an optional snapshot directory:
```sh
OPENCODE_SIMULATION=1 OPENCODE_SIMULATION_STATE=/path/to/snapshot bun run dev
```
Optional flags can be added later, but should stay minimal. Reasonable optional parameters later include renderer mode, trace output path, seed, or port override.
All simulation parameters are environment variables, not CLI flags. This is a hard requirement: `packages/core/src/global.ts` computes and creates XDG paths at module import time, so anything that redirects paths must be in place before the first import. Environment variables set by the parent process (or read at the very top of startup) satisfy this; CLI flags parsed after imports do not.
When enabled:
- The app creates and changes into a real, empty anchor directory (see Filesystem).
- The app reads the snapshot directory, if provided, and seeds all simulated state from it.
- The app builds with simulation layer replacements.
- The TUI process starts a loopback WebSocket control server.
- Simulation-gated backend control routes become available only to the frontend/control path.
- In-memory trace recording starts automatically.
Path seams reuse existing environment variables where they already exist: `OPENCODE_CONFIG_DIR` for global config, `OPENCODE_TEST_HOME` for home, and `OPENCODE_DB=:memory:` for the database. Simulation mode should set these before foundational modules load rather than inventing parallel mechanisms.
## Control Server
The external control surface lives in the TUI/frontend process, not the backend API server.
This is important because the frontend has direct access to the renderer, screen state, focus state, interactable elements, and user input APIs. The backend remains the normal backend, with only simulation-gated control routes used internally by the frontend when needed.
Protocol:
- JSON-RPC 2.0 over WebSocket.
- Loopback only.
- Start at `127.0.0.1:40900`.
- If occupied, scan upward and report the actual URL.
- External drivers connect only to this frontend WebSocket.
The app should not send JSON-RPC requests back to the driver in the first milestone. The driver sends requests; the app responds and emits notifications/events as useful.
Initial method groups:
- `ui.state`: return screen, elements, focus, and generated possible actions.
- `ui.action`: execute one real user-level action.
- `ui.render`: force or wait for a render and return state.
- `backend.filesystem.seed`: seed project files.
- `backend.filesystem.write`: write one file.
- `backend.network.register`: register a fake network response.
- `backend.llm.enqueue`: queue scripted LLM behavior.
- `backend.snapshot`: return backend simulation state.
- `trace.list`: return trace records.
- `trace.clear`: clear in-memory trace.
- `trace.export`: export trace JSON for replay/test generation.
- `run.stabilize`: wait for frontend/backend quiescence and return observations.
## TUI Actions
The old simulation branch had the right basic shape: observe OpenTUI renderables, derive executable actions, and execute those actions through OpenTUI input/mouse APIs.
The first action vocabulary should stay close to that work:
```ts
type UIAction =
| { type: "typeText"; text: string }
| { type: "pressKey"; key: string; modifiers?: KeyModifiers }
| { type: "pressEnter" }
| { type: "pressArrow"; direction: "up" | "down" | "left" | "right" }
| { type: "focus"; target: number }
| { type: "click"; target: number; x: number; y: number }
```
`ui.state` should return:
- Current screen text.
- Focused renderable/editor state.
- Interactable elements.
- Generated actions valid for the current UI state.
Elements should include stable-enough semantic data where available:
- Renderable ID and numeric target.
- Position and dimensions.
- Focusable/clickable/editor flags.
- Focused flag.
- Text or label when available.
- Role/capability when available.
Both fake OpenTUI renderer and visible terminal renderer should share this protocol. The architecture should support both; the default can be decided later.
## Backend Control
The backend server should be exactly the normal backend server.
Simulation-only backend routes may exist, but only when `OPENCODE_SIMULATION=1`. They are private implementation details for the frontend simulation server to proxy commands like filesystem seeding, LLM scripting, network registration, and snapshots.
External drivers should not use backend simulation routes directly.
## Foundational Layer Replacement
Current `origin/dev` has the right seam: `AppNodeBuilder.build(...)` and `AppNodeBuilderV1.build(...)` accept replacements over `LayerNode`s. Simulation should use those seams instead of adding large alternate app assemblies.
First milestone replacements:
- Filesystem.
- Network / HTTP client.
- LLM boundary.
- Process spawner.
First milestone generated state surfaces:
- Filesystem/project state.
- Network responses.
- LLM scripts.
- Process registry behavior.
- Plugin-generated config state.
Likely later replacements:
- Clock/random.
- Database path/isolation.
- Global paths/temp paths.
The goal is to swap things at the bottom of the app. Everything above these foundational services should behave as production code.
## Filesystem
The filesystem simulation is in-memory, anchored at a real empty directory.
On startup in simulation mode:
1. Create a real, empty anchor directory with `mkdtemp` (for example `$TMPDIR/opencode-sim-XXXXXX`).
2. `process.chdir(anchor)` before any command resolves its working directory.
3. Use `process.cwd()` — now the anchor — as the root of the in-memory filesystem.
4. Seed the in-memory filesystem from the snapshot directory, joining snapshot-relative paths onto the anchor root.
The anchor directory on the host stays empty for the entire run. All file content lives only in the in-memory filesystem.
Rationale for the real anchor:
- `process.cwd()`, `$PWD`, and `path.resolve()` are all genuinely correct with zero patching. The previous simulation branch used a virtual root (`/opencode`) that existed nowhere on the host, which forced monkey-patching `process.cwd` and `$PWD` and left raw `fs` relative-path resolution silently disagreeing with the faked cwd.
- The codebase reads `process.cwd()` at process edges (CLI entry points, TUI frontend, request-fallback in workspace routing) and converts it into an explicit `directory` value early; core never reads it directly. A truthful cwd at startup means every downstream consumer inherits the virtual root without touching those call sites.
- Leak detection is free: the anchor must be empty at the end of the run. Any file that appears there means some code path bypassed the simulated filesystem. This is an assertable invariant.
- Host filesystem bypasses read an empty directory instead of the developer's real project. Bypassed reads fail loudly instead of returning wrong-but-plausible data.
Rationale for in-memory content:
- The run is hermetic: no host writes, no cleanup dependencies, no cross-run contamination.
- Snapshots load and reset quickly, which matters for model-based runs that reset state often.
- The containment check (path must be inside the anchor root) doubles as the host-escape guard with a truthful boundary.
The in-memory filesystem is still controlled and isolated:
- Each run gets its own anchor root.
- Project files, config, data, state, cache, and temp paths should resolve inside that root (via `OPENCODE_CONFIG_DIR`, `OPENCODE_TEST_HOME`, and `OPENCODE_DB=:memory:`).
- Paths outside the anchor root fail loudly with a typed simulation error.
- Trace should record seeded files and file diffs/observations needed for replay.
The anchor may be created by the app itself at activation, or by an external runner that spawns the app with the anchor as its working directory. Both should work: the app creates and enters an anchor only when its current directory is not already a designated anchor.
## Initial State Snapshot
`OPENCODE_SIMULATION_STATE` points at a directory containing one complete initial state. On startup the app slurps this directory once and constructs all simulated state from it. The snapshot is never written back to; it is a pure input.
Proposed layout:
```text
snapshot/
project/... # workspace files, seeded into the in-memory FS under the anchor root
config/opencode.json # global config; the directory backs OPENCODE_CONFIG_DIR
env.json # extra environment values to apply
llm/... # scripted LLM behavior to pre-enqueue (optional)
network/... # network response registrations (optional)
```
Rules:
- Paths inside `project/` are snapshot-relative. The loader joins them onto the anchor root, so absolute virtual paths look like real host paths under the anchor.
- Anything the config references (skills, instructions, reference paths) must exist inside `project/`. A snapshot that references missing files is invalid.
- The snapshot directory format is the contract between external state generators and the app. Generators (such as the opencode-probe project) produce snapshot directories plus a derived expected model; the app consumes only the snapshot.
- Seeding through the control server (`backend.filesystem.seed` and friends) remains available for incremental changes during a run; the snapshot covers initial state.
## Configuration Via Generated Plugins
Generated configuration is a core first-milestone feature.
Much of opencode behavior is driven by config. The simulation runner needs to put the app into many different config-shaped states: different agents, tools, providers, MCP servers, permissions, modes, instructions, formatting settings, feature flags, and other config-dependent behavior.
The runner should not primarily generate arbitrary config files. Instead, the simulation should express config-shaped state as generated plugins.
Rationale:
- Plugins are already a normal extension surface for opencode behavior.
- Generated plugins can produce app states without making the simulation depend on config-file syntax and file layout details.
- Plugin-generated state keeps setup closer to runtime behavior: the app reads config, loads plugins, and observes plugin-provided behavior through normal app paths.
- Plugins are a better unit for model-based generation because they can be named, versioned, traced, reused, and minimized independently.
The first implementation should support generated simulation plugins that can contribute or affect config-equivalent domains such as:
- Agents and agent defaults.
- Provider/model availability.
- Tool definitions and tool behavior.
- MCP-like capabilities or endpoints.
- Permission defaults and policies.
- Instructions/system-context-like inputs where supported.
- Formatting/project behavior where supported.
- Workspace/project adapters where supported.
The simulation can still write the minimal bootstrap state needed for opencode to discover generated plugins, but the interesting generated state should live in plugin definitions rather than large generated `opencode.json` files.
Trace should record:
- Generated plugin IDs.
- Plugin-provided config/state fragments.
- Plugin hooks registered.
- Any plugin load/config errors.
- Which generated plugin state was active for each run.
The model-based runner should include commands for generating and enabling plugin state. These commands should have normal preconditions and postconditions just like UI actions or backend setup commands.
Example command families:
- Generate a provider/model plugin.
- Generate an agent configuration plugin.
- Generate a tool plugin with scripted behavior.
- Generate permission policy state.
- Generate MCP-like tool/resource state.
- Enable or disable a generated plugin for the next app run.
This is the main mechanism for exploring app states driven by configuration.
## Network
Unknown external network should fail loudly by default.
The simulation network should support explicit response registration:
- JSON response.
- Text response.
- Bytes response later if needed.
- Status-only response.
- Handler-style response later if needed.
Loopback traffic needed by the app/frontend/backend may be allowed explicitly.
All network calls should be traceable:
- Method.
- URL.
- Request headers/body where safe.
- Matched simulation route.
- Status.
- Response summary.
- Error if denied.
## LLM
The LLM boundary should be scriptable.
The driver can enqueue scripts that describe model behavior:
- Text chunks.
- Thinking/reasoning chunks if relevant.
- Tool calls.
- Errors.
- Finish reason.
The real session and tool pipeline should consume this behavior through the normal app path. The simulation should not bypass `SessionPrompt`, `SessionProcessor`, or tool execution.
Missing scripted LLM behavior should fail with a clear simulation error unless a default response is explicitly configured.
## Process Spawning
External process spawning should be denied by default.
The first milestone should provide a simulated process registry. This should be inspired by the old branch:
- Shell commands can run through `just-bash` against the simulated filesystem.
- A small fake `git` command set can support project discovery/status paths needed by the app.
- Unsupported process spawns fail loudly.
This preserves the rule that simulation does not spawn arbitrary external programs while still allowing useful shell/tool flows.
## Trace
Trace recording is always on in simulation mode, in memory for the first milestone.
Trace entries should be append-only JSON-compatible records. They do not need to be written to disk initially, but `trace.export` should return a structure suitable for later replay and test generation.
Trace should include:
- Run metadata: seed, app version, renderer mode, WebSocket URL.
- Initial world setup.
- UI observations.
- Generated UI actions.
- Executed UI actions.
- Backend control requests.
- Backend snapshots.
- Network requests and matches/denials.
- LLM scripts enqueued and consumed.
- Tool calls and results.
- Permission decisions.
- Filesystem seed/write/diff summaries.
- Generated plugin/config state and load results.
- Stabilization boundaries.
- Errors and crashes.
- Model command execution and postcondition results.
The trace is the bridge between exploratory simulation and deterministic tests.
## Model-Based Runner
The first runner is an external driver connecting to the frontend WebSocket.
Use a custom runner for now, not `fast-check`. It should still follow the core shape used by property/model-based testing libraries:
```ts
interface Command<Model> {
readonly name: string
check(model: Model): boolean
run(model: Model, app: SimulationClient): Promise<void>
}
```
Basic runner responsibilities:
- Keep a lightweight model of high-level expected state.
- Generate commands whose preconditions match the model and current app observations.
- Execute commands through the WebSocket.
- Update the model.
- Check postconditions/invariants.
- Record all steps in the trace.
- Support seed/replay.
- Track simple distribution stats.
The model should track high-level, observational state only, such as:
- Current screen/route category.
- Whether prompt editor is available.
- Known sessions.
- Known files and expected file contents/diffs.
- Queued LLM scripts.
- Recent backend/session status.
- Whether app is expected to be idle.
The model must not track implementation internals like fibers, exact runner loop state, cache internals, or database implementation details.
Initial command families:
- Seed filesystem.
- Generate and enable plugin config state.
- Register network response.
- Enqueue LLM script.
- Observe UI state.
- Execute one generated UI action.
- Type prompt text.
- Press enter.
- Stabilize.
- Assert no crash.
- Assert visible response or file effect.
- Export trace.
## Generators
The first milestone should include generation, but not shrinking.
Generation should be model-based and state-aware:
- Generate from currently valid `ui.state.actions`.
- Generate backend setup commands from scenario/model state.
- Generate plugin-provided config state.
- Generate LLM scripts that match likely user prompts and tool flows.
- Generate short command sequences using preconditions.
- Use a seed so runs can be replayed.
- Use simple weights to avoid degenerate action selection.
The generator should not attempt to produce arbitrary full app states upfront. It should build state by executing commands through the real app and observing the result.
Important stats to record:
- Seed.
- Command counts.
- Action type distribution.
- Generated plugin/config domain distribution.
- Rejected command/precondition counts.
- UI element/action coverage.
- Backend event type coverage where available.
- Errors and stabilization failures.
## Properties
First milestone properties should be simple and high-signal:
- App does not crash.
- Backend does not crash.
- Unknown network is denied.
- Host filesystem escape is denied.
- Prompt submission can reach a scripted LLM response.
- Stabilization eventually reaches a coherent idle state for the demo flow.
- File effects from scripted tool behavior are observable in the simulated filesystem.
- Trace contains enough information to replay the run.
More advanced model/refinement, metamorphic, and differential properties are future work.
## First Demo Flow
The first major demo should show this system as a real environment for exploring the app in controlled states:
1. Start opencode normally with `OPENCODE_SIMULATION=1`.
2. TUI starts and exposes the simulation WebSocket on `127.0.0.1:40900+`.
3. External runner connects.
4. Runner provides a snapshot directory (or seeds the in-memory project filesystem through the control server).
5. Runner generates and enables plugin-provided config state.
6. Runner queues a scripted LLM response.
7. Runner observes `ui.state` and generated actions.
8. Runner drives real TUI input to type and submit a prompt.
9. App processes the prompt through the real backend/session/tool path.
10. Scripted LLM response appears or executes a file-affecting tool flow.
11. Runner stabilizes the app.
12. Runner inspects trace, backend snapshot, UI state, generated plugin state, and filesystem state.
13. Runner exports a deterministic replay trace.
## Done-When Checklist
- `OPENCODE_SIMULATION=1` starts the normal app with simulation wiring.
- Simulation code is isolated under a dedicated simulation/testing area.
- App changes outside simulation are limited to activation hooks, builder replacements, TUI startup, and gated backend routes.
- TUI exposes JSON-RPC WebSocket on `127.0.0.1:40900+`.
- Driver can call `ui.state`.
- Driver can execute generated UI actions.
- Fake and visible renderer paths use the same action protocol.
- Driver can seed filesystem state.
- Driver can generate and enable plugin-provided config state.
- Driver can register network responses and observe denied unknown network.
- Driver can enqueue LLM scripts.
- External process spawning is denied by default, with shell via `just-bash` and minimal fake process registry support.
- Driver can run a basic model-based generated command sequence.
- In-memory trace records observations/actions/backend interactions.
- Driver can list, clear, and export trace.
- Demo flow succeeds end-to-end.
## Future Directions
- Shrinking failed traces.
- Promote minimized traces into normal committed tests.
- Coverage-guided corpus and structured trace mutation.
- Richer semantic UI grounding for model-driven exploration.
- LLM-generated property proposals with validity/soundness checks.
- Differential testing across app versions, renderers, or storage modes.
- Deterministic scheduler/clock/random control.
- Parallel campaigns with isolated workers.
- File-backed trace persistence and replay CLI.
+23 -10
View File
@@ -16,7 +16,7 @@ import { SdkPlugins } from "@opencode-ai/core/plugin/sdk"
import { ToolOutputStore } from "@opencode-ai/core/tool-output-store"
import { HttpRouter, HttpServer } from "effect/unstable/http"
import { HttpApiBuilder } from "effect/unstable/httpapi"
import { Layer, Option } from "effect"
import { Effect, Layer, Option } from "effect"
import { Api } from "./api"
import { ServerAuth } from "./auth"
import { handlers } from "./handlers"
@@ -58,15 +58,24 @@ function makeRoutes<AuthError, AuthServices>(
sdkPlugins?: SdkPlugins.Store,
) {
const pluginRuntimeCell = PluginRuntime.makeCell()
const serviceLayer = AppNodeBuilder.build(
applicationServices,
[
[SessionExecution.node, SessionExecutionLocal.node],
[PluginRuntime.node, PluginRuntime.layerWithCell(pluginRuntimeCell)],
[PluginRuntime.providerNode, PluginRuntime.providerNodeWithCell(pluginRuntimeCell)],
...(sdkPlugins ? [[SdkPlugins.node, SdkPlugins.layerWithStore(sdkPlugins)] as const] : []),
],
)
const replacements: LayerNode.Replacements = [
[SessionExecution.node, SessionExecutionLocal.node],
[PluginRuntime.node, PluginRuntime.layerWithCell(pluginRuntimeCell)],
[PluginRuntime.providerNode, PluginRuntime.providerNodeWithCell(pluginRuntimeCell)],
...(sdkPlugins ? [[SdkPlugins.node, SdkPlugins.layerWithStore(sdkPlugins)] as const] : []),
]
// Simulation replacements are loaded via dynamic import so the simulation
// module is never eagerly loaded. Layer.unwrap defers both the import and
// the app-node build to layer-build time; when simulation is off the branch
// is byte-for-byte identical to a plain AppNodeBuilder.build call.
const serviceLayer = simulationEnabled()
? Layer.unwrap(
Effect.gen(function* () {
const { simulationReplacements } = yield* Effect.promise(() => import("./simulation"))
return AppNodeBuilder.build(applicationServices, [...replacements, ...simulationReplacements])
}),
)
: AppNodeBuilder.build(applicationServices, replacements)
return HttpApiBuilder.layer(Api, { openapiPath: "/openapi.json" }).pipe(
Layer.provide(handlers),
@@ -79,6 +88,10 @@ function makeRoutes<AuthError, AuthServices>(
)
}
function simulationEnabled() {
return !!process.env.OPENCODE_SIMULATION
}
export const routes = createRoutes()
export const webHandler = () =>
+14
View File
@@ -0,0 +1,14 @@
import { LayerNode } from "@opencode-ai/core/effect/layer-node"
/**
* Layer replacements applied when the server is built in simulation mode.
*
* Empty for now; simulation-mode implementations will populate this with
* replacement nodes/layers that swap real services for simulated ones (e.g.
* a fake filesystem). The server merges these into the app node build when
* `OPENCODE_SIMULATION` is enabled, via a dynamic import so this module is
* never loaded eagerly.
*/
export const simulationReplacements: LayerNode.Replacements = []
export * as Simulation from "./index"
+12 -4
View File
@@ -8,7 +8,7 @@ import { ClipboardProvider, useClipboard } from "./context/clipboard"
import { ExitProvider, useExit } from "./context/exit"
import { EpilogueProvider } from "./context/epilogue"
import * as Selection from "./util/selection"
import { createCliRenderer, MouseButton } from "@opentui/core"
import { createCliRenderer, MouseButton, type CliRendererConfig } from "@opentui/core"
import { RouteProvider, useRoute } from "./context/route"
import {
Switch,
@@ -184,8 +184,8 @@ export const run = Effect.fn("Tui.run")(function* (input: TuiInput) {
Effect.gen(function* () {
const renderer = yield* Effect.acquireRelease(
Effect.tryPromise({
try: () =>
createCliRenderer({
try: async () => {
const options = {
externalOutputMode: "passthrough",
targetFps: 60,
gatherStats: false,
@@ -197,7 +197,15 @@ export const run = Effect.fn("Tui.run")(function* (input: TuiInput) {
consoleOptions: {
keyBindings: [{ name: "y", ctrl: true, action: "copy-selection" }],
},
}),
} satisfies CliRendererConfig
if (process.env.OPENCODE_SIMULATION === "1" || process.env.OPENCODE_SIMULATION === "true") {
const { Simulation } = await import("./simulation/simulation")
return Simulation.createSimulation(options)
}
return createCliRenderer(options)
},
catch: (error) => (error instanceof Error ? error : new Error(String(error))),
}),
(renderer) =>
+177
View File
@@ -0,0 +1,177 @@
import type { CliRenderer, Renderable } from "@opentui/core"
import { createMockKeys, createMockMouse, type MockInput, type MockMouse } from "@opentui/core/testing"
import { SimulationRenderer } from "./renderer"
import { SimulationTrace } from "./trace"
export interface KeyModifiers {
readonly ctrl?: boolean
readonly shift?: boolean
readonly meta?: boolean
readonly super?: boolean
readonly hyper?: boolean
}
export type Action =
| { readonly type: "typeText"; readonly text: string }
| { readonly type: "pressKey"; readonly key: string; readonly modifiers?: KeyModifiers }
| { readonly type: "pressEnter" }
| { readonly type: "pressArrow"; readonly direction: "up" | "down" | "left" | "right" }
| { readonly type: "focus"; readonly target: number }
| { readonly type: "click"; readonly target: number; readonly x: number; readonly y: number }
export interface Element {
readonly id: string
readonly num: number
readonly x: number
readonly y: number
readonly width: number
readonly height: number
readonly focusable: boolean
readonly focused: boolean
readonly clickable: boolean
readonly editor: boolean
}
export interface Harness {
readonly renderer: CliRenderer
readonly mockInput: MockInput
readonly mockMouse: MockMouse
readonly renderOnce: () => Promise<void>
readonly screen: () => string
}
type RenderBuffer = {
readonly width: number
readonly height: number
getRealCharBytes(includeAnsi?: boolean): Uint8Array
}
const decoder = new TextDecoder()
function children(renderable: Renderable) {
return renderable.getChildren().filter((child): child is Renderable => "num" in child)
}
function all(renderable: Renderable): Renderable[] {
return [renderable, ...children(renderable).flatMap(all)]
}
function mouseListeners(renderable: Renderable) {
const general = Reflect.get(renderable, "_mouseListener")
const specific = Reflect.get(renderable, "_mouseListeners")
return Boolean(general) || (specific && typeof specific === "object" && Object.keys(specific).length > 0)
}
function hit(renderer: CliRenderer, renderable: Renderable) {
if (renderable.width <= 0 || renderable.height <= 0) return false
const x = Math.floor(renderable.screenX + renderable.width / 2)
const y = Math.floor(renderable.screenY + renderable.height / 2)
return renderer.hitTest(x, y) === renderable.num
}
/**
* Builds the harness the simulation server drives.
*
* When the renderer is the fake simulation renderer, its TestRendererSetup
* provides the supported testing APIs. For the visible terminal renderer the
* harness falls back to `requestRender` + `idle` and reading the private
* `currentRenderBuffer`.
*/
export function createHarness(renderer: CliRenderer): Harness {
const setup = SimulationRenderer.setupFor(renderer)
return {
renderer,
mockInput: setup?.mockInput ?? createMockKeys(renderer),
mockMouse: setup?.mockMouse ?? createMockMouse(renderer),
renderOnce:
setup?.renderOnce ??
(async () => {
renderer.requestRender()
await renderer.idle()
}),
screen:
setup?.captureCharFrame ??
(() => decoder.decode((Reflect.get(renderer, "currentRenderBuffer") as RenderBuffer).getRealCharBytes(true))),
}
}
export function elements(renderer: CliRenderer): Element[] {
return all(renderer.root)
.filter((renderable) => renderable.visible && !renderable.isDestroyed)
.map((renderable) => {
const clickable = mouseListeners(renderable) && hit(renderer, renderable)
return {
id: renderable.id,
num: renderable.num,
x: renderable.screenX,
y: renderable.screenY,
width: renderable.width,
height: renderable.height,
focusable: renderable.focusable,
focused: renderable.focused,
clickable,
editor: renderer.currentFocusedEditor === renderable,
} satisfies Element
})
.filter((element) => element.focusable || element.clickable || element.editor)
}
export function actions(renderer: CliRenderer, options: { text?: string } = {}): Action[] {
const items = elements(renderer)
return [
...(renderer.currentFocusedEditor
? ([{ type: "typeText", text: options.text ?? "hello" }, { type: "pressEnter" }] satisfies Action[])
: []),
...items.filter((item) => item.focusable && !item.focused).map((item) => ({ type: "focus" as const, target: item.num })),
...items
.filter((item) => item.clickable)
.map((item) => ({
type: "click" as const,
target: item.num,
x: Math.floor(item.x + item.width / 2),
y: Math.floor(item.y + item.height / 2),
})),
{ type: "pressArrow", direction: "down" },
{ type: "pressArrow", direction: "up" },
]
}
export function state(harness: Harness) {
return {
screen: harness.screen(),
focused: {
renderable: harness.renderer.currentFocusedRenderable?.num,
editor: Boolean(harness.renderer.currentFocusedEditor),
},
elements: elements(harness.renderer),
actions: actions(harness.renderer),
}
}
export async function execute(harness: Harness, action: Action) {
SimulationTrace.add("ui.action", { action })
switch (action.type) {
case "typeText":
await harness.mockInput.typeText(action.text)
break
case "pressKey":
harness.mockInput.pressKey(action.key, action.modifiers)
break
case "pressEnter":
harness.mockInput.pressEnter()
break
case "pressArrow":
harness.mockInput.pressArrow(action.direction)
break
case "focus":
all(harness.renderer.root).find((item) => item.num === action.target)?.focus()
break
case "click":
await harness.mockMouse.click(action.x, action.y)
break
}
await harness.renderOnce()
return state(harness)
}
export * as SimulationActions from "./actions"
+26
View File
@@ -0,0 +1,26 @@
import type { CliRenderer, CliRendererConfig } from "@opentui/core"
import { createTestRenderer, type TestRendererSetup } from "@opentui/core/testing"
const setups = new WeakMap<CliRenderer, TestRendererSetup>()
/**
* Creates the fake simulation renderer: a real CliRenderer backed by an
* in-memory screen buffer instead of a terminal. The TestRendererSetup is
* kept module-side (keyed by renderer) so the harness can use the supported
* testing APIs without app code carrying it around.
*/
export async function create(options: CliRendererConfig): Promise<CliRenderer> {
const setup = await createTestRenderer({
...options,
width: Number(process.env.OPENCODE_SIMULATION_TUI_WIDTH) || 100,
height: Number(process.env.OPENCODE_SIMULATION_TUI_HEIGHT) || 40,
})
setups.set(setup.renderer, setup)
return setup.renderer
}
export function setupFor(renderer: CliRenderer): TestRendererSetup | undefined {
return setups.get(renderer)
}
export * as SimulationRenderer from "./renderer"
+174
View File
@@ -0,0 +1,174 @@
import { SimulationActions, type Action, type Harness } from "./actions"
import { SimulationTrace } from "./trace"
const DefaultPort = 40900
const MaxPortAttempts = 100
type JsonRpcRequest = {
readonly jsonrpc: "2.0"
readonly id?: string | number | null
readonly method: string
readonly params?: unknown
}
type JsonRpcResponse = {
readonly jsonrpc: "2.0"
readonly id: string | number | null
readonly result?: unknown
readonly error?: {
readonly code: number
readonly message: string
readonly data?: unknown
}
}
export interface Server {
readonly url: string
readonly stop: () => void
}
function isEnabled() {
return process.env.OPENCODE_SIMULATION === "1" || process.env.OPENCODE_SIMULATION === "true"
}
function isPortUnavailable(error: unknown) {
const message = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase()
return message.includes("eaddrinuse") || message.includes("address already in use") || message.includes(" in use")
}
function parseRequest(input: string | Buffer): JsonRpcRequest {
const value = JSON.parse(typeof input === "string" ? input : input.toString()) as unknown
if (typeof value !== "object" || value === null) throw new Error("Invalid JSON-RPC request")
if (!("jsonrpc" in value) || value.jsonrpc !== "2.0") throw new Error("Invalid JSON-RPC version")
if (!("method" in value) || typeof value.method !== "string") throw new Error("Invalid JSON-RPC method")
return value as JsonRpcRequest
}
function isAction(input: unknown): input is Action {
if (typeof input !== "object" || input === null || !("type" in input)) return false
switch (input.type) {
case "typeText":
return "text" in input && typeof input.text === "string"
case "pressKey":
return "key" in input && typeof input.key === "string"
case "pressEnter":
return true
case "pressArrow":
return "direction" in input && ["up", "down", "left", "right"].includes(String(input.direction))
case "focus":
return "target" in input && typeof input.target === "number"
case "click":
return (
"target" in input &&
typeof input.target === "number" &&
"x" in input &&
typeof input.x === "number" &&
"y" in input &&
typeof input.y === "number"
)
}
return false
}
function actionParam(params: unknown) {
if (typeof params !== "object" || params === null || !("action" in params)) throw new Error("Missing action")
if (!isAction(params.action)) throw new Error("Invalid action")
return params.action
}
function response(id: JsonRpcRequest["id"], result: unknown): JsonRpcResponse | undefined {
if (id === undefined) return undefined
return { jsonrpc: "2.0", id, result }
}
function errorResponse(id: JsonRpcRequest["id"], error: unknown): JsonRpcResponse {
return {
jsonrpc: "2.0",
id: id ?? null,
error: {
code: -32000,
message: error instanceof Error ? error.message : String(error),
},
}
}
async function handle(harness: Harness, request: JsonRpcRequest) {
switch (request.method) {
case "ui.state": {
const result = SimulationActions.state(harness)
SimulationTrace.add("ui.state", { elements: result.elements.length, actions: result.actions.length })
return result
}
case "ui.action":
return SimulationActions.execute(harness, actionParam(request.params))
case "ui.render": {
await harness.renderOnce()
const result = SimulationActions.state(harness)
SimulationTrace.add("ui.render", { elements: result.elements.length, actions: result.actions.length })
return result
}
case "trace.list":
return { records: SimulationTrace.list() }
case "trace.clear":
SimulationTrace.clear()
return { cleared: true }
case "trace.export":
return SimulationTrace.exportTrace()
}
throw new Error(`Unknown simulation method: ${request.method}`)
}
function serve(
harness: Harness,
port = DefaultPort,
attempts = MaxPortAttempts,
): Bun.Server<{ readonly simulation: true }> {
try {
return Bun.serve<{ readonly simulation: true }>({
hostname: "127.0.0.1",
port,
fetch(request, server) {
if (server.upgrade(request, { data: { simulation: true } })) return undefined
return new Response("opencode simulation websocket", { status: 426 })
},
websocket: {
open() {
SimulationTrace.add("control.connect")
},
close() {
SimulationTrace.add("control.disconnect")
},
async message(socket, message) {
let request: JsonRpcRequest | undefined
try {
request = parseRequest(message)
const result = await handle(harness, request)
const next = response(request.id, result)
if (next) socket.send(JSON.stringify(next))
} catch (error) {
socket.send(JSON.stringify(errorResponse(request?.id, error)))
}
},
},
})
} catch (error) {
if (!isPortUnavailable(error) || attempts <= 1 || port >= 65535) throw error
return serve(harness, port + 1, attempts - 1)
}
}
export function start(harness: Harness): Server | undefined {
if (!isEnabled()) return
const server = serve(harness)
const url = `ws://${server.hostname}:${server.port}`
SimulationTrace.add("control.start", { url })
return {
url,
stop: () => {
SimulationTrace.add("control.stop", { url })
server.stop(true)
},
}
}
export * as SimulationServer from "./server"
+27
View File
@@ -0,0 +1,27 @@
import { createCliRenderer, type CliRenderer, type CliRendererConfig } from "@opentui/core"
import { SimulationActions } from "./actions"
import { SimulationRenderer } from "./renderer"
import { SimulationServer } from "./server"
/**
* Simulation-mode renderer entry point.
*
* Creates the renderer (fake when OPENCODE_SIMULATION_RENDERER=fake, the
* normal visible renderer otherwise) and starts the simulation control
* server against it. The server stops when the renderer is destroyed, so the
* caller only manages the renderer lifecycle.
*/
export async function createSimulation(options: CliRendererConfig): Promise<CliRenderer> {
const renderer =
process.env.OPENCODE_SIMULATION_RENDERER === "fake"
? await SimulationRenderer.create(options)
: await createCliRenderer(options)
const server = SimulationServer.start(SimulationActions.createHarness(renderer))
if (server) {
process.stderr.write(`opencode simulation websocket: ${server.url}\n`)
renderer.once("destroy", () => server.stop())
}
return renderer
}
export * as Simulation from "./simulation"
+37
View File
@@ -0,0 +1,37 @@
export type TraceRecord = {
readonly id: number
readonly time: string
readonly type: string
readonly data?: unknown
}
const records: TraceRecord[] = []
let nextID = 0
export function add(type: string, data?: unknown) {
const record = {
id: ++nextID,
time: new Date().toISOString(),
type,
...(data === undefined ? {} : { data }),
} satisfies TraceRecord
records.push(record)
return record
}
export function list() {
return [...records]
}
export function clear() {
records.length = 0
nextID = 0
}
export function exportTrace() {
return {
records: list(),
}
}
export * as SimulationTrace from "./trace"