harness: scratch graduation pipeline (push side + spec) + flarum first test case
- graduation-push.sh: tar+scp scratch -> BEAST graduation-inbox over Tailscale (decoupled from /save, soft-fail if BEAST off). Tested: 241 files -> BEAST. - docs/graduation-pipeline.md: full spec (push -> Ollama triage on BEAST GPU via API -> reviewed sanitize+git-mv). Secrets never enter git; ride the encrypted link to BEAST only. - tmp-promotion-check.sh: rewritten pure-builtin (0.4s) after the per-file grep/fork loop hung /save for 4 min on Windows at ~240 scratch files. Deep triage moves to the pipeline. - forum-post: GRADUATED the canonical flarum poster from scratch -> skills/forum-post/scripts/flarum-post.py (s9e markdown->XML + DB insert machinery), with the hardcoded IX SSH + Flarum DB passwords swapped to vault lookups. First pipeline test case. - Vaulted the Flarum DB cred (services/flarum-community.sops.yaml) + sanitized the two plaintext copies in forum-post.md. - errorlog: logged the WSL-stub correction + BEAST-Ollama-CPU(vram=0) finding + the promotion-check hang, all via the new log helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
84
.claude/docs/graduation-pipeline.md
Normal file
84
.claude/docs/graduation-pipeline.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Scratch Graduation Pipeline (spec)
|
||||
|
||||
Status: **draft / in progress** (2026-06-15). Push side built + tested; triage validated on
|
||||
the flarum test case; scheduled-on-BEAST wiring + execute helper are the remaining work.
|
||||
|
||||
## Problem
|
||||
|
||||
Scratch dirs (`tmp/`, `temp/`, `.claude/tmp/`) are gitignored, so anything in them is invisible
|
||||
to git and lost on cleanup. The old approach — a **synchronous** `tmp-promotion-check.sh` run
|
||||
inside `/save` and `/scc` — had two fatal flaws:
|
||||
|
||||
1. **Too slow on Windows.** At ~240 scratch files it forked `basename`/`wc`/`grep -r` per file;
|
||||
the "referenced in a session log" check recursed `clients/` + `projects/` (Rust `target/`,
|
||||
`node_modules/`, `.git`) **once per file** and hung `/save` for **4 minutes** (errorlog 2026-06-15).
|
||||
2. **Too dumb.** Extension/size heuristics can't answer the real question — *which* of
|
||||
`flarum_do_insert.py` / `do_insert2.py` / `search_insert.py` is canonical, what's a superseded
|
||||
debug dupe, what holds secrets, where each belongs. That's semantic judgment.
|
||||
|
||||
The interim `tmp-promotion-check.sh` is now a fast (0.4s) pure-builtin "N scripts in scratch" nudge.
|
||||
The real triage is **offloaded and asynchronous**, per this spec.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
workstation BEAST (GURU-BEAST-ROG, best GPU) any Claude session
|
||||
----------- -------------------------------- ------------------
|
||||
graduation-push.sh Ollama @ :11434 (GPU) review proposal
|
||||
tar scratch ──SCP/Tailscale──▶ graduation-inbox/<machine>/*.tgz sanitize secrets
|
||||
(soft-fail if BEAST off) graduation-triage (Ollama classify) git mv keepers
|
||||
─▶ proposal manifest ──coord msg/todo──▶ delete junk → commit
|
||||
```
|
||||
|
||||
1. **Push** (`graduation-push.sh`, built): tars scratch and `scp`s ONE tarball to
|
||||
`guru@100.101.122.4:graduation-inbox/<machine>/scratch-<utc>.tgz` over Tailscale. Decoupled
|
||||
from `/save`; soft-fails if BEAST is unreachable. Centralizes every machine's scratch on the
|
||||
GPU box (archive + lets BEAST batch-process even when the origin machine is off).
|
||||
2. **Triage** (Ollama on BEAST's GPU): for each file, classify
|
||||
`{disposition: graduate|delete|keep-data, canonical?, superseded_by, has_secrets, suggested_home, why}`.
|
||||
Emits a **proposal manifest** (the supersession/secret reasoning the old heuristics couldn't do).
|
||||
The orchestration can run **on BEAST** (Git-bash, scheduled) or on **any machine** against
|
||||
BEAST's Ollama API — the GPU is reached over the HTTP API either way.
|
||||
3. **Review + execute**: a Claude session (or human) reads the manifest, **sanitizes secrets**
|
||||
(hardcoded creds → vault lookups), `git mv`s keepers to permanent homes, deletes junk, commits.
|
||||
*Ollama proposes, human/Claude disposes* (same contract as memory-dream + the Tier-0 routing rule).
|
||||
|
||||
## Transport / environment facts (verified 2026-06-15)
|
||||
|
||||
- BEAST = `guru-beast-rog`, Tailscale `100.101.122.4`. SSH key auth works as **`guru`** (no password).
|
||||
- BEAST default SSH shell = **cmd.exe**; home `C:\Users\guru`. The harness/triage run under
|
||||
**Git-for-Windows MSYS bash** — NOT WSL. (`bash` on PATH resolves to the WindowsApps WSL stub;
|
||||
invoke Git-bash explicitly. The WSL stub also can't reach the Windows-host Ollama on localhost —
|
||||
another reason to avoid it.)
|
||||
- **Ollama** runs on BEAST's Windows side, bound so it's reachable fleet-wide over Tailscale at
|
||||
`http://100.101.122.4:11434`. Models incl. `qwen3:32b`, `qwen3.6:latest` (36B), `gemma3:27b`,
|
||||
`codestral:22b`, `qwen3:14b`, `nomic-embed-text`.
|
||||
- Inbox: `C:\Users\guru\graduation-inbox\<machine>\` (cmd path) — per-machine namespaced.
|
||||
|
||||
## Security (non-negotiable)
|
||||
|
||||
- **Secrets never enter git.** Raw scratch can contain hardcoded creds (the flarum scripts hold the
|
||||
IX root SSH password). It rides the WireGuard-encrypted Tailscale/SSH link and lands ONLY on BEAST
|
||||
(trusted). The transport is deliberately NOT the git repo or a multi-tenant store.
|
||||
- **Sanitize before commit.** Any file graduated into a tracked home gets hardcoded secrets swapped
|
||||
for vault lookups first (`vault.sh get-field ...`). harness-guard would block a plaintext-secret commit.
|
||||
- **Manifest-only returns.** Only the proposal manifest comes back toward git — never the raw files.
|
||||
|
||||
## Components
|
||||
|
||||
| Piece | Path | State |
|
||||
|---|---|---|
|
||||
| Push | `.claude/scripts/graduation-push.sh` | built + tested (241 files → BEAST) |
|
||||
| Interim nudge | `.claude/scripts/tmp-promotion-check.sh` | fast builtin-only (0.4s) |
|
||||
| Triage | `.claude/scripts/graduation-triage.*` | validated ad-hoc on flarum; productize next |
|
||||
| Execute | manual (Claude session) | flarum = first test case |
|
||||
| Schedule | BEAST cron/loop calling triage | TODO |
|
||||
| Return | coord message/todo to origin machine | TODO |
|
||||
|
||||
## Open items
|
||||
|
||||
- Productize `graduation-triage` (general file loop + Ollama classify + manifest) and a `--execute`
|
||||
helper that sanitizes + `git mv`s per an approved manifest.
|
||||
- Wire a scheduled triage run on BEAST (or a `/loop`) + coord-message return.
|
||||
- Decide retention/cleanup of the BEAST inbox + auto-deleting obvious junk to keep scratch bounded.
|
||||
- Consider dropping `tmp-promotion-check` from `/save` entirely once the pipeline is routine.
|
||||
Reference in New Issue
Block a user