write down what to test, and what is most likely broken
The serve is the only path now, so this is not a comparison against a fallback. Split by what I have actually driven end to end versus what probes cannot answer. The second list is the real testing: resume from history (never exercised against the serve, and my pick for most likely broken), an idle session, a sidecar restart mid-turn, an officer restart mid-turn, and two conversations at once — that last one because the live event stream is global and a wrong sessionID filter would splice one conversation into another. Known gaps are listed so they do not get reported as bugs, and the one silent failure mode with a single cause — a turn producing nothing at all — points at the credential line from boot, which I have chased twice already. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# OpenCode: what to test tonight
|
||||
|
||||
Every OpenCode turn now runs through the serve (`serve-runner.ts`). The `opencode run` subprocess is
|
||||
deleted, so this is not a comparison against a fallback — it is the only path.
|
||||
|
||||
I have driven each item in **Should already work** end to end through the real chat socket, with a
|
||||
script rather than a browser. Nothing below has been used by a person in a real conversation, and the
|
||||
things in **Unproven** are unproven because probes cannot answer them.
|
||||
|
||||
---
|
||||
|
||||
## Before you start
|
||||
|
||||
```bash
|
||||
cd ~/projects/officer-suite/platform
|
||||
pm2 logs officer-opencode --lines 40 # the sidecar's own account of what happened
|
||||
```
|
||||
|
||||
On boot you should see three lines: `serve healthy on port …`, `connected the opencode credential to the
|
||||
api surface`, and the sidecar registering. **If the credential line is missing or says it could not
|
||||
connect, stop** — paid models will silently do nothing, which is the failure that cost most of an
|
||||
afternoon (`docs/opencode-fork-decision.md`).
|
||||
|
||||
Rolling back is `git revert` of `a3dbda7` (Phase D) and a restart. There is no config flag any more.
|
||||
|
||||
Alpha is unaffected until you pull.
|
||||
|
||||
---
|
||||
|
||||
## Should already work — confirm, do not investigate
|
||||
|
||||
Each verified by me end to end. If one fails, that is new information and worth stopping on.
|
||||
|
||||
- [ ] **A plain turn.** Ask for something short. Text arrives.
|
||||
- [ ] **Streaming.** Ask for something long — "count slowly to 50". Text should appear **progressively**,
|
||||
not in one block at the end. This is the headline change; the old path could not do it.
|
||||
- [ ] **A tool call.** "Run `echo hi` with bash." A tool row appears with the command as its arguments,
|
||||
then its output.
|
||||
- [ ] **Cost.** The turn ends with a token count attached.
|
||||
- [ ] **An image.** Drop a screenshot in and ask what it shows. It must actually be described — the
|
||||
failure mode is a confident answer about nothing, which is what B4 was.
|
||||
- [ ] **Stop.** Press stop mid-turn. The turn ends **and the conversation stays usable** — send another
|
||||
message straight after and it should answer. Previously stop destroyed the session.
|
||||
- [ ] **Mid-turn injection.** While a turn runs, send another message. It should join the RUNNING turn
|
||||
rather than starting a new one or superseding it.
|
||||
|
||||
## Unproven — this is the actual testing
|
||||
|
||||
- [ ] **Resume from history.** Open an older OpenCode conversation from the list and continue it. This
|
||||
exercises `resumeSessionId`, which I never tested against the serve. **Most likely thing to be
|
||||
broken.**
|
||||
- [ ] **An idle session, an hour later.** Send a message to a conversation you have not touched for a
|
||||
while. Warm sessions are new here — the subprocess had nothing to go stale.
|
||||
- [ ] **A sidecar restart mid-turn.** `pm2 restart officer-opencode` while a turn is generating. Expect
|
||||
the transcript to say the turn stopped. **The turn itself keeps running inside the serve** — that
|
||||
is intended, not a bug, but nobody has watched what it looks like from the browser.
|
||||
- [ ] **An officer restart mid-turn.** Different from the above: officer is the relay, the sidecar keeps
|
||||
committing to `chat_session_events`. On reload the transcript should be intact. This is the one I
|
||||
would least like to be wrong about.
|
||||
- [ ] **Two conversations at once.** The live event stream is GLOBAL — one socket carries every session
|
||||
and `serve-runner` filters on `sessionID`. If that filter is wrong, output from one conversation
|
||||
appears in another. Two panels side by side is the test.
|
||||
- [ ] **The Live panel.** A running OpenCode turn should appear, named, and disappear when it ends.
|
||||
- [ ] **A long turn.** Ten-plus minutes. The old path had watchdogs (inactivity, hard cap) that are gone
|
||||
with it; the serve has its own ideas about timeouts and I have not found their edges.
|
||||
- [ ] **A failing tool.** Ask it to run a command that does not exist. The error should land in the
|
||||
transcript as a failed tool, not as a dead turn.
|
||||
|
||||
## Known gaps, so you do not report them as bugs
|
||||
|
||||
- **No durable replay.** `serve-runner` reads the live stream only. Events are still committed to
|
||||
`chat_session_events` as they arrive, so the transcript survives — but recovering a turn *this sidecar
|
||||
process never saw* would need the `?after=` cursor, and that is not built.
|
||||
- **Notifications/thinking/background tasks.** Not implemented on this harness. Thinking is deliberately
|
||||
out of scope for both harnesses.
|
||||
- **`messageCount` shows nothing** for OpenCode rows. The UI renders an `OpenCode` badge instead; not a
|
||||
gap (`docs/opencode-parity.md`).
|
||||
|
||||
## What is useful to tell me
|
||||
|
||||
For anything that misbehaves: what you did, what appeared, and the last twenty lines of
|
||||
`pm2 logs officer-opencode`. The sidecar logs its own errors, and a silent failure with clean logs is a
|
||||
different diagnosis from a loud one.
|
||||
|
||||
If a turn produces **nothing at all** — no text, no error, no spinner ending — check the credential line
|
||||
from boot first. That specific silence has one cause and I have chased it twice.
|
||||
Reference in New Issue
Block a user