From 7431bfd52b53ae7abb800645404a679baf14b47a Mon Sep 17 00:00:00 2001 From: "sysadmin@dataforth.com" Date: Sun, 29 Mar 2026 17:51:55 -0700 Subject: [PATCH] Update CLAUDE.md: Full Dataforth context, credentials, pipeline state, known issues --- .claude/CLAUDE.md | 396 +++++++++++++++++++++++++--------------------- 1 file changed, 219 insertions(+), 177 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 78e79cb..059aed9 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -1,204 +1,246 @@ -# ClaudeTools Project Context +# ClaudeTools on AD2 (Dataforth Domain Controller) -## Identity: You Are a Coordinator +## Identity -You are NOT an executor. You coordinate specialized agents and preserve your context window. +This is the AD2 workstation instance of ClaudeTools. This machine is a Windows Server on the Dataforth LAN (192.168.0.6). Your scope is Dataforth-only -- you do not need context about other clients. -**Delegate ALL significant work:** +## NO EMOJIS -| Operation | Delegate To | -|-----------|------------| -| Database queries/inserts/updates | Database Agent | -| Production code generation | Coding Agent | -| Code review (MANDATORY after changes) | Code Review Agent | -| Test execution | Testing Agent | -| Git commits/push/branch | Gitea Agent | -| Backups/restore | Backup Agent | -| File exploration (broad) | Explore Agent | -| Semantic code search | deep-explore Agent (uses GrepAI) | -| Complex reasoning | General-purpose + Sequential Thinking | - -**Do yourself:** Simple responses, reading 1-2 files, presenting results, planning, decisions. -**Rule:** >500 tokens of work = delegate. Code or database = ALWAYS delegate. - -**DO NOT** query databases directly (no SSH/mysql/curl to API). **DO NOT** write production code. **DO NOT** run tests. **DO NOT** commit/push. Use the appropriate agent. +Use ASCII markers: [OK], [ERROR], [WARNING], [SUCCESS], [INFO] --- -## Project Overview +## Git & Sync -**Type:** MSP Work Tracking System | **Status:** Production-Ready (Phase 5 Complete) -**Database:** MariaDB 10.6.22 @ 172.16.3.30:3306 | **API:** http://172.16.3.30:8001 -**Stats:** 95+ endpoints, 38 tables, JWT auth, AES-256-GCM encryption +### Gitea Credentials (no 1Password on this machine) +- URL: https://git.azcomputerguru.com +- Username: mike@azcomputerguru.com +- Password: Gptf*77ttb123!@#-git +- URL-encoded password: Gptf%2A77ttb123%21%40%23-git +- API Token: 9b1da4b79a38ef782268341d25a4b6880572063f +- Remote: https://mike%40azcomputerguru.com:Gptf%2A77ttb123%21%40%23-git@git.azcomputerguru.com/azcomputerguru/claudetools.git -**DB Connection:** Host: 172.16.3.30:3306 | DB: claudetools | User: claudetools | Password: CT_e8fcd5a3952030a79ed6debae6c954ed -**Details:** `.claude/agents/DATABASE_CONNECTION_INFO.md` +### Branch: ad2 +This machine operates on the `ad2` branch. The main workstation merges into main. ---- +### /save behavior +Save session logs to `session-logs/YYYY-MM-DD-session-ad2.md` (note the -ad2 suffix). +After saving, commit and push to origin/ad2. -## Key Rules - -- **NO EMOJIS** - Use ASCII markers: `[OK]`, `[ERROR]`, `[WARNING]`, `[SUCCESS]`, `[INFO]` -- **No hardcoded credentials** - Use 1Password (`op read "op://Vault/Item/field"`) or encrypted storage -- **SSH:** Use system OpenSSH (on Windows: `C:\Windows\System32\OpenSSH\ssh.exe`, never Git for Windows SSH) -- **Data integrity:** Never use placeholder/fake data. Check credentials.md (op:// refs) or 1Password or ask user. -- **Full coding standards:** `.claude/CODING_GUIDELINES.md` (agents read on-demand, not every session) - ---- - -## Automatic Behaviors - -- **Frontend Design:** Auto-invoke `/frontend-design` skill after ANY UI change (HTML/CSS/JSX/styling) -- **Sequential Thinking:** Use for genuine complexity - rejection loops, 3+ critical issues, architectural decisions, multi-step debugging -- **Task Management:** Complex work (>3 steps) -> TaskCreate. Persist to `.claude/active-tasks.json`. - ---- - -## Context Recovery - -When user references previous work, use `/context` command. Never ask user for info in: -- `credentials.md` - Infrastructure reference with `op://` paths (secrets in 1Password) -- `session-logs/` - Daily work logs (also in `projects/*/session-logs/` and `clients/*/session-logs/`) -- `SESSION_STATE.md` - Project history - -### 1Password Credential Access - -Credentials are stored in 1Password across 4 vaults: **Infrastructure**, **Clients**, **Projects**, **MSP Tools**. - -**To read a secret:** `op read "op://VaultName/ItemTitle/field_name"` - -**Service account (non-interactive):** Set `OP_SERVICE_ACCOUNT_TOKEN` env var. Token stored in `op://Infrastructure/Service Account Auth Token: Agentic_Cli/credential`. The service account has Read & Write on all 4 vaults (except Projects which is read-only -- use desktop app auth for Projects writes). - -**Setup on new machines:** -1. Install 1Password CLI: https://developer.1password.com/docs/cli/get-started/ -2. Sign in: `op signin` (or use desktop app integration) -3. For non-interactive use, add to shell config: `set -gx OP_SERVICE_ACCOUNT_TOKEN "token_value"` - ---- - -## Commands & Skills - -| Command | Purpose | -|---------|---------| -| `/checkpoint` | Dual checkpoint: git commit + database context | -| `/save` | Comprehensive session log (credentials, decisions, changes) | -| `/context` | Search session logs, credentials.md, and 1Password | -| `/1password` | 1Password secrets management integration | -| `/sync` | Sync config from Gitea repository | -| `/create-spec` | Create app specification for AutoCoder | -| `/frontend-design` | Modern frontend design patterns (auto-invoke after UI changes) | - ---- - -## File Placement (Quick Rules) - -- **Dataforth DOS work** -> `projects/dataforth-dos/` -- **ClaudeTools API code** -> `api/`, `migrations/` (existing structure) -- **Client work** -> `clients/[client-name]/` -- **Session logs** -> project or client `session-logs/` subfolder; general -> root `session-logs/` -- **Full guide:** `.claude/FILE_PLACEMENT_GUIDE.md` (read when saving files, not every session) - ---- - -## Local AI (Ollama) - -Ollama runs locally with GPU acceleration. Use it for tasks that don't need Claude-level reasoning. - -### Available Models - -| Model | Size | Use For | -|-------|------|---------| -| `qwen3:14b` | 9.3 GB | General sub-tasks: summarization, classification, data extraction, drafting | -| `codestral:22b` | 12 GB | Code-specific sub-tasks: code generation, refactoring suggestions, docstring generation | -| `nomic-embed-text` | 274 MB | Embeddings only (used by GrepAI, not for direct use) | - -### GrepAI (Semantic Code Search) - -GrepAI indexes the codebase using `nomic-embed-text` embeddings and provides semantic search via MCP server. - -**When to use GrepAI instead of Grep/Glob:** -- Finding code by intent ("how does authentication work") rather than exact text -- Exploring unfamiliar areas of the codebase -- Finding related implementations across files -- Context recovery — searching session logs and credentials by meaning - -**How to use:** -- **MCP tool:** Use the `grepai` MCP server tools directly (available after MCP loads) -- **deep-explore agent:** Delegate to the `deep-explore` agent for thorough semantic exploration -- **CLI fallback:** `grepai search "your query" --json --compact` - -**Maintenance:** The watcher daemon runs in the background and auto-indexes file changes. If search results seem stale, run `grepai watch --stop && grepai watch --background` to restart it. - -### Using Ollama for Sub-Tasks - -For bulk or repetitive work that doesn't require Claude's full reasoning, offload to local models via Ollama's API: - -**When to use Ollama:** -- Processing many items in a loop (e.g., summarizing 50 session logs) -- Generating boilerplate or repetitive code patterns -- Data extraction/classification from structured text -- Draft content that Claude will review/refine -- Any task where speed > quality and results will be verified - -**When NOT to use Ollama (use Claude instead):** -- Architectural decisions or complex reasoning -- Security-sensitive code review -- Tasks requiring tool use or multi-step planning -- Final output that goes directly to production - -**How to call Ollama:** -```bash -# Simple prompt -curl -s http://localhost:11434/api/generate -d '{"model":"qwen3:14b","prompt":"Summarize this: ...","stream":false}' | jq -r '.response' - -# Chat format -curl -s http://localhost:11434/api/chat -d '{"model":"codestral:22b","messages":[{"role":"user","content":"Refactor this function: ..."}],"stream":false}' | jq -r '.message.content' +### /sync behavior +``` +git fetch origin +git rebase origin/main +git push origin ad2 ``` -### Ollama Output Review Policy +--- -The coordinator (Claude) must review Ollama outputs based on impact level. Local models are useful but unreliable — they hallucinate, miss edge cases, and produce subtly wrong code. +## Dataforth Network -**Impact levels and review requirements:** +| Host | IP | Role | Notes | +|------|-----|------|-------| +| AD1 | 192.168.0.27 | Primary DC | Disk at 90%, C:\Engineering = 787 GB | +| **AD2** | **192.168.0.6** | **This machine** | Secondary DC, TestDataDB, file shares | +| D2TESTNAS | 192.168.0.9 | SMB1 proxy for DOS | Debian 13, Samba, SSH root/Paper123!@#-nas | +| UDM | 192.168.0.254 | Gateway/Router | UniFi Dream Machine | +| ESXi-122 | 192.168.0.122 | Hypervisor | ESXi | +| ESXi-124 | 192.168.0.124 | Hypervisor | ESXi | +| DOS stations | TS-01 to TS-30+ | Test stations | DOS 6.22, QuickBASIC ATE software | -| Level | Review | Examples | -|-------|--------|----------| -| **Critical** | ALWAYS review, verify against source | Code touching auth/security/encryption, credential handling, database migrations, production config, anything user-facing | -| **High** | Review for correctness, spot-check details | API endpoint logic, business rules, infrastructure scripts, client-specific work | -| **Medium** | Skim for obvious errors, trust if reasonable | Internal documentation drafts, session log summaries, data extraction from structured input, boilerplate code | -| **Low** | Trust without review | Classification/tagging of items, reformatting text, generating placeholder content for later editing | - -**Review process for Critical/High:** -1. Read Ollama's full output — don't just check if it "looks right" -2. Verify claims against actual files/data (e.g., if it says a function exists, confirm it does) -3. Check for: hallucinated function names, wrong parameter types, missing error handling, security gaps -4. If output is wrong or uncertain, redo the task yourself rather than patching Ollama's attempt - -**Batch processing pattern:** -When using Ollama for bulk tasks (e.g., processing N items), review the first 2-3 results fully before trusting the rest. If any are wrong, switch to doing it yourself or fix the prompt and reprocess. - -**Flag to user:** If Ollama produces output for a Critical task and you are not confident in your review, tell the user explicitly: "This was generated by a local model and I'm not fully confident in [specific concern]." +### Credentials +- AD Sysadmin: INTRANET\sysadmin / Paper123!@# +- D2TESTNAS SSH: root@192.168.0.9 / Paper123!@#-nas +- D2TESTNAS Samba: guest access (no password) +- WINS/NPS: 192.168.0.27:1812/1813 +- M365 Tenant: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584 +- Rsync daemon (NAS): port 873, module "test", user rsync / IQ203s32119 --- -## Memory (Shared Across Machines) +## Local Resources -Claude Code's auto-memory is stored **in-repo** at `.claude/memory/` so it syncs via Gitea to all workstations. - -**IMPORTANT for all machines:** Configure Claude Code to use the repo memory path, NOT the default `~/.claude/projects/` path. When the auto-memory system prompts you to write to `~/.claude/projects/-home-guru-ClaudeTools/memory/`, write to `.claude/memory/` (repo-relative) instead. The index file is `.claude/memory/MEMORY.md`. - -This ensures memory created on one workstation (CachyOS, Mac, Windows) is available on all others after a git pull/sync. +| Resource | Path | +|----------|------| +| TestDataDB app | C:\Shares\testdatadb\ | +| Test database | C:\Shares\testdatadb\database\testdata.db (SQLite, 2.2M+ records) | +| TestDataDB API | http://localhost:3000 | +| Parsers | C:\Shares\testdatadb\parsers\ (multiline.js, csvline.js, shtfile.js, spec-reader.js) | +| Templates | C:\Shares\testdatadb\templates\datasheet-exact.js | +| Import script | C:\Shares\testdatadb\database\import.js | +| Export script | C:\Shares\testdatadb\database\export-datasheets.js | +| Stage import | C:\Shares\testdatadb\import-all-stage.js | +| NAS share | \\D2TESTNAS\test (mapped as T:) | +| Datasheets share | X:\For_Web | +| ProdSW (BAT files) | C:\Shares\test\COMMON\ProdSW\ | +| Sync script | C:\Shares\test\scripts\Sync-FromNAS.ps1 (bidirectional, 15-min schedule) | --- -## Reference (read on-demand, not every session) +## DOS Update System - Batch Files -- **Project structure, endpoints, workflows, troubleshooting:** `.claude/REFERENCE.md` -- **Agent definitions:** `.claude/agents/*.md` -- **MCP servers:** `MCP_SERVERS.md` -- **Coding standards:** `.claude/CODING_GUIDELINES.md` -- **Shared memory:** `.claude/memory/MEMORY.md` (index) + `.claude/memory/*.md` (individual memories) +### Boot Sequence on DOS Machines +``` +AUTOEXEC.BAT (v4.1) + -> STARTNET.BAT (v2.0) -- init network, map T: and X: drives + -> ATESYNC.BAT + -> CTONW.BAT (v5.0) -- upload test data to network + -> CTONWTXT.BAT (v2.3) -- upload C:\STAGE\*.TXT to T:\STAGE\%MACHINE% + -> NWTOC.BAT (v5.0) -- download updates from network +``` + +### Current Production Versions (on AD2 & NAS) +| File | Version | Last Update | Purpose | +|------|---------|-------------|---------| +| AUTOEXEC.BAT | v4.1 | 2026-03-12 | Startup config | +| STARTNET.BAT | v2.0 | 2026-01-20 | Network init | +| NWTOC.BAT | v5.0 | 2026-03-16 | Download updates from network | +| CTONW.BAT | v5.0 | 2026-03-28 | Upload test data (5 steps with echo) | +| CTONWTXT.BAT | v2.3 | 2026-03-28 | Upload Stage TXT files (no MD, dirs pre-created) | +| CHECKUPD.BAT | v1.3 | 2026-01-20 | Check for updates | +| UPDATE.BAT | v2.3 | 2026-01-20 | Full system backup | +| STAGE.BAT | v1.0 | Original | Stage system file updates | +| DEPLOY.BAT | v1.0 | 2026-01-20 | One-time deployment installer | + +### DOS 6.22 Compatibility Rules +- NO `IF NOT` -- unreliable on DOS 6.22. Use positive `IF EXIST` with GOTO +- NO `IF /I` (case-insensitive compare) +- NO `FOR /F` loops +- NO `%COMPUTERNAME%` -- use `%MACHINE%` (set during DEPLOY) +- `XCOPY /D` requires date parameter (`/D:mm-dd-yy`) +- `MD` fails with error on existing directories -- pre-create dirs server-side +- `COPY` without `/Y` hangs on overwrite prompts +- All paths UPPERCASE for Samba compatibility +- Line endings MUST be CRLF (0D 0A) --- -**Last Updated:** 2026-03-22 +## Serial Number Encoding (DOS 8.3 filenames) + +QuickBASIC ATE encodes long serial numbers for 8.3 filenames: +``` +First 2 digits replaced with hex letter if serial too long: + 178236-12 -> H8236-12.TXT (17 -> H, charCode 72 - 55 = 17) + 10819-1 -> A819-1.TXT (10 -> A, charCode 65 - 55 = 10) + +Decode: letter.charCodeAt(0) - 55 = numeric prefix +Only applies when filename starts with [A-Z] followed by digits. + +H-prefix files have decoded SN inside the file (SN: 178236-12) +A-prefix files have encoded SN inside the file (SN: A819-1) -- must decode to 10819-1 +``` + +--- + +## Test Datasheet Pipeline + +### 5-Stage Architecture +1. **DOS Test Programs** -> Write DAT files to C:\ATE\*LOG\ and TXT to C:\STAGE\ +2. **Boot Upload** -> CTONW.BAT copies DAT to T:\%MACHINE%\LOGS\, CTONWTXT copies TXT to T:\STAGE\%MACHINE% +3. **NAS <-> AD2 Sync** -> Rsync every 15 min (Sync-FromNAS.ps1 scheduled task) +4. **TestDataDB Import** -> import.js parses DAT into SQLite; export-datasheets.js generates TXT to X:\For_Web +5. **Web Share** -> X:\For_Web\ holds validated datasheets (501K+ files) + +### import-all-stage.js (ready to run) +Located at `C:\Shares\testdatadb\import-all-stage.js`. Processes ~8,100 TXT files: +- Scans \\D2TESTNAS\test\STAGE\TS-*\*.TXT +- Decodes hex-prefix serial numbers +- Cross-references testdata.db by (serial_number, model_number) +- Inserts missing records as log_type='SHT' +- Copies to X:\For_Web\{decoded_serial}.TXT + +``` +cd C:\Shares\testdatadb +node import-all-stage.js +``` + +### Machine data volumes in STAGE +| Machine | Files | +|---------|-------| +| TS-4L | 3,082 | +| TS-4R | 2,741 | +| TS-1R | 509 | +| TS-8R | 478 | +| TS-3R | 435 | +| TS-11R | 325 | +| TS-8L | 285 | +| TS-11L | 248 | +| TS-27 | 10 (already imported) | +| TS-1L | 1 | + +### Web Share Layout (X:\) +- X:\For_Web -- Validated datasheets (production) +- X:\For_Web_PDF -- PDF versions (4.7K files) +- X:\Test_Datasheets -- Incoming/staging +- X:\Bad_Datasheets -- Invalid files (18K) +- X:\Datasheets_Log -- Processing logs + +--- + +## Known Issues & Pending Work + +### HIGH PRIORITY +1. **Run import-all-stage.js** -- 8,100 TXT files need cross-referencing and ingestion +2. **Website Upload Replacement** -- Old ASP.NET endpoints (Uploader.aspx) return 404. Need new approach. +3. **7B Series Datasheets** -- ~830K records can't generate datasheets (missing 7BMAIN.DAT spec file). Check ENGR share. +4. **Service Permissions** -- testdatadb runs as SYSTEM, causing file permission issues. Change to INTRANET\sysadmin. + +### MEDIUM PRIORITY +5. **C2 IP Blocking** -- iptables rules added to UDM for 80.76.49.18 and 45.88.91.99. Need permanent rules in UniFi UI. +6. **MFA Enforcement** -- 19/38 users ready. Report-only until April 4, 2026. Monitor registration. +7. **Joel Lohr Account** -- Retiring March 31. Disable account post-retirement. Auto-reply set to Dan Center. + +--- + +## Security Incident (2026-03-27) + +**DF-JOEL2 (192.168.0.143) compromised via phishing:** +- Joel Lohr clicked phishing link in personal Yahoo email +- ScreenConnect C2 installed, "Angel Raya" connected remotely +- Two C2 backdoors deployed via PowerShell +- C2 IPs: 80.76.49.18, 45.88.91.99 (AS399486, suspended by host) +- IC3 Complaint: 1c32ade367084be9acd548f23705736f +- ConnectWise Case: 03464184 +- **Remediation complete:** IPs blocked, 3 rogue clients removed, password reset, sessions revoked +- **No lateral movement detected** (32 machines scanned clean) + +--- + +## Key Contacts + +| Person | Email | Role | +|--------|-------|------| +| John Lehman | jlehman@dataforth.com | Engineering, QB code, test specs | +| Dan Center | dcenter@dataforth.com | Operations (replacing Joel) | +| Peter Iliya | pIliya@dataforth.com | Applications Engineer | +| AJ | dataforthgit@... | Engineering contact | +| Ken Hoffman | (unresponsive) | TestDataSheetUploader author | +| Georg Haubner | ghaubner@dataforth.com | Has pre-crypto backup on D: drive | + +--- + +## Quick Reference Commands + +```powershell +# Check BAT files on NAS +ssh root@192.168.0.9 'ls -la /data/test/COMMON/ProdSW/' + +# Trigger NAS sync +Start-ScheduledTask -TaskName 'Sync-FromNAS' + +# Check sync log +Get-Content 'C:\Shares\test\scripts\sync-from-nas.log' -Tail 20 + +# Check TestDataDB health +curl http://localhost:3000/health + +# Query test records +node -e "const db=require('better-sqlite3')('C:\\Shares\\testdatadb\\database\\testdata.db',{readonly:true});console.log(db.prepare('SELECT COUNT(*) as cnt FROM test_records').get())" + +# Check Stage files on NAS +ssh root@192.168.0.9 'find /data/test/STAGE -name "*.TXT" | wc -l' +``` + +--- + +**Last Updated:** 2026-03-29