Dogfood findings: reliability improvements for the agent workflow #1

New issue

Open

opened 2026-11-12 20:29:54 +00:00 by hermes_forgeron · 1 comment

hermes_forgeron commented

2026-11-12 20:29:54 +00:00

Collaborator

Context

First full dogfood run of the pipeline on serious-devs/test_factory_ai, issue #3 (plan → build → PR). The happy path works end-to-end — plan reply with spec block, human edits the spec, build produces branch agent/issue-N and a mergeable PR — but the run exposed four reliability gaps.

Findings

1. Automatic triggers run but the agent skips (blocks day-to-day usage)

With the thin per-project workflow delegating via uses:, the glue sees the event as workflow_call/<action> instead of the original issues / issue_comment event and, in mode: auto, cannot infer plan vs build (log: not addressed to the agent (unhandled event workflow_call/created) — skipping). Every automatic trigger currently produces a no-op run (~3 min wasted each).

Proposed fix: the glue should read the caller's original event from GITHUB_EVENT_PATH (the payload forwarded to the called workflow is the caller's) and infer the mode from it. The thin per-project workflow stays unchanged.

2. Build agent killed by orphaned background servers (3/3 runs lost)

The build agent wrote correct code and its smoke test passed (HTTP 200 on the target endpoint), but it started the server with npm start &; kill $SERVER_PID kills npm while the node child survives, keeps listening and holds the shell's stdout/stderr open → opencode's shell tool kills the command at its timeout (30 s, 60 s on retry) → opencode exits null → the run is marked failed. Editing the spec to forbid runtime servers made the next run succeed, but that is a manual workaround, not a fix.

Proposed fix: standing guidance in the build prompt: no long-lived/background processes; if a server must be exercised, start node directly with output redirected to a file and kill -9 the exact PID — otherwise prefer static checks (node --check, npm ls) when the spec allows.

3. Completed work is lost when opencode crashes mid-run

In all three failed builds the files were fully written before the crash, but the ephemeral workspace was destroyed with nothing pushed — the human gate never saw usable output.

Proposed fix: salvage path in the glue — if opencode exits non-zero/null after files changed in the workspace, still push agent/issue-N and open a draft PR with an explicit warning (e.g. "agent crashed after implementing — unverified"), so the human decides.

4. (minor) Called-workflow job logs unreachable via API

GET /repos/{o}/{r}/actions/jobs/{id}/logs returns HTTP 500 for the agent job (the reusable workflow's job); only the duplicated agent-1 job's logs are readable. Possibly a Forgejo bug rather than ai-factory code — verify upstream before changing anything here.

Scope boundaries

Glue (glue/), reusable workflow inputs if needed, and the build prompt text only. No changes to opencode itself, no new services.
Security posture unchanged: no new token scopes, no default-branch pushes.

Proposed child issues

glue: infer original event and mode under workflow_call
build prompt: standing no-background-processes / static-checks guidance
glue: salvage path — push branch + draft PR when opencode crashes after writing files
verify Forgejo 500 on called-workflow job logs (upstream?) — drop if confirmed as a Forgejo bug

⏳ Awaiting validation by @mlvtito before any child issue is created or worked on.

## Context First full dogfood run of the pipeline on `serious-devs/test_factory_ai`, issue #3 (plan → build → PR). The happy path works end-to-end — plan reply with spec block, human edits the spec, build produces branch `agent/issue-N` and a mergeable PR — but the run exposed four reliability gaps. ## Findings ### 1. Automatic triggers run but the agent skips (blocks day-to-day usage) With the thin per-project workflow delegating via `uses:`, the glue sees the event as `workflow_call/<action>` instead of the original `issues` / `issue_comment` event and, in `mode: auto`, cannot infer plan vs build (log: `not addressed to the agent (unhandled event workflow_call/created) — skipping`). Every automatic trigger currently produces a no-op run (~3 min wasted each). **Proposed fix:** the glue should read the caller's original event from `GITHUB_EVENT_PATH` (the payload forwarded to the called workflow is the caller's) and infer the mode from it. The thin per-project workflow stays unchanged. ### 2. Build agent killed by orphaned background servers (3/3 runs lost) The build agent wrote correct code and its smoke test passed (HTTP 200 on the target endpoint), but it started the server with `npm start &`; `kill $SERVER_PID` kills npm while the node child survives, keeps listening and holds the shell's stdout/stderr open → opencode's shell tool kills the command at its timeout (30 s, 60 s on retry) → opencode exits `null` → the run is marked failed. Editing the spec to forbid runtime servers made the next run succeed, but that is a manual workaround, not a fix. **Proposed fix:** standing guidance in the build prompt: no long-lived/background processes; if a server must be exercised, start `node` directly with output redirected to a file and `kill -9` the exact PID — otherwise prefer static checks (`node --check`, `npm ls`) when the spec allows. ### 3. Completed work is lost when opencode crashes mid-run In all three failed builds the files were fully written before the crash, but the ephemeral workspace was destroyed with nothing pushed — the human gate never saw usable output. **Proposed fix:** salvage path in the glue — if opencode exits non-zero/null after files changed in the workspace, still push `agent/issue-N` and open a draft PR with an explicit warning (e.g. "agent crashed after implementing — unverified"), so the human decides. ### 4. (minor) Called-workflow job logs unreachable via API `GET /repos/{o}/{r}/actions/jobs/{id}/logs` returns HTTP 500 for the `agent` job (the reusable workflow's job); only the duplicated `agent-1` job's logs are readable. Possibly a Forgejo bug rather than ai-factory code — verify upstream before changing anything here. ## Scope boundaries - Glue (`glue/`), reusable workflow inputs if needed, and the build prompt text only. No changes to opencode itself, no new services. - Security posture unchanged: no new token scopes, no default-branch pushes. ## Proposed child issues 1. glue: infer original event and mode under `workflow_call` 2. build prompt: standing no-background-processes / static-checks guidance 3. glue: salvage path — push branch + draft PR when opencode crashes after writing files 4. verify Forgejo 500 on called-workflow job logs (upstream?) — drop if confirmed as a Forgejo bug ⏳ **Awaiting validation by @mlvtito** before any child issue is created or worked on.

hermes_forgeron referenced this issue

2026-11-12 21:25:12 +00:00

glue: honor the caller's original event when invoked via workflow_call (fix auto-trigger no-op) #2

hermes_forgeron referenced this issue

2026-11-12 21:25:12 +00:00

build agent: language-agnostic rules against orphaned background processes #3

hermes_forgeron referenced this issue

2026-11-12 21:25:12 +00:00

glue: salvage path — push branch + warning PR when the build agent crashes after writing files #4

hermes_forgeron referenced this issue

2026-11-12 21:25:12 +00:00

investigate: called-workflow job logs return HTTP 500 via API (Forgejo 17 / gitea-1.22) #5

hermes_forgeron commented

2026-11-12 21:25:13 +00:00

Author

Collaborator

Child issues created (validated by @mlvtito):

#2 — glue: honor the caller's original event when invoked via workflow_call (fix auto-trigger no-op)
#3 — build agent: language-agnostic rules against orphaned background processes
#4 — glue: salvage path — push branch + warning PR when the build agent crashes after writing files
#5 — investigate: called-workflow job logs return HTTP 500 via API (Forgejo 17 / gitea-1.22)

This mother stays open until all of them land.

Child issues created (validated by @mlvtito): - #2 — glue: honor the caller's original event when invoked via workflow_call (fix auto-trigger no-op) - #3 — build agent: language-agnostic rules against orphaned background processes - #4 — glue: salvage path — push branch + warning PR when the build agent crashes after writing files - #5 — investigate: called-workflow job logs return HTTP 500 via API (Forgejo 17 / gitea-1.22) This mother stays open until all of them land.