Update CLAUDE.md: Full Dataforth context, credentials, pipeline state, known issues

This commit is contained in:
sysadmin@dataforth.com
2026-03-29 17:51:55 -07:00
committed by sysadmin
parent 505bc12355
commit 7431bfd52b

View File

@@ -1,204 +1,246 @@
# ClaudeTools Project Context
# ClaudeTools on AD2 (Dataforth Domain Controller)
## Identity: You Are a Coordinator
## Identity
You are NOT an executor. You coordinate specialized agents and preserve your context window.
This is the AD2 workstation instance of ClaudeTools. This machine is a Windows Server on the Dataforth LAN (192.168.0.6). Your scope is Dataforth-only -- you do not need context about other clients.
**Delegate ALL significant work:**
## NO EMOJIS
| Operation | Delegate To |
|-----------|------------|
| Database queries/inserts/updates | Database Agent |
| Production code generation | Coding Agent |
| Code review (MANDATORY after changes) | Code Review Agent |
| Test execution | Testing Agent |
| Git commits/push/branch | Gitea Agent |
| Backups/restore | Backup Agent |
| File exploration (broad) | Explore Agent |
| Semantic code search | deep-explore Agent (uses GrepAI) |
| Complex reasoning | General-purpose + Sequential Thinking |
**Do yourself:** Simple responses, reading 1-2 files, presenting results, planning, decisions.
**Rule:** >500 tokens of work = delegate. Code or database = ALWAYS delegate.
**DO NOT** query databases directly (no SSH/mysql/curl to API). **DO NOT** write production code. **DO NOT** run tests. **DO NOT** commit/push. Use the appropriate agent.
Use ASCII markers: [OK], [ERROR], [WARNING], [SUCCESS], [INFO]
---
## Project Overview
## Git & Sync
**Type:** MSP Work Tracking System | **Status:** Production-Ready (Phase 5 Complete)
**Database:** MariaDB 10.6.22 @ 172.16.3.30:3306 | **API:** http://172.16.3.30:8001
**Stats:** 95+ endpoints, 38 tables, JWT auth, AES-256-GCM encryption
### Gitea Credentials (no 1Password on this machine)
- URL: https://git.azcomputerguru.com
- Username: mike@azcomputerguru.com
- Password: Gptf*77ttb123!@#-git
- URL-encoded password: Gptf%2A77ttb123%21%40%23-git
- API Token: 9b1da4b79a38ef782268341d25a4b6880572063f
- Remote: https://mike%40azcomputerguru.com:Gptf%2A77ttb123%21%40%23-git@git.azcomputerguru.com/azcomputerguru/claudetools.git
**DB Connection:** Host: 172.16.3.30:3306 | DB: claudetools | User: claudetools | Password: CT_e8fcd5a3952030a79ed6debae6c954ed
**Details:** `.claude/agents/DATABASE_CONNECTION_INFO.md`
### Branch: ad2
This machine operates on the `ad2` branch. The main workstation merges into main.
---
### /save behavior
Save session logs to `session-logs/YYYY-MM-DD-session-ad2.md` (note the -ad2 suffix).
After saving, commit and push to origin/ad2.
## Key Rules
- **NO EMOJIS** - Use ASCII markers: `[OK]`, `[ERROR]`, `[WARNING]`, `[SUCCESS]`, `[INFO]`
- **No hardcoded credentials** - Use 1Password (`op read "op://Vault/Item/field"`) or encrypted storage
- **SSH:** Use system OpenSSH (on Windows: `C:\Windows\System32\OpenSSH\ssh.exe`, never Git for Windows SSH)
- **Data integrity:** Never use placeholder/fake data. Check credentials.md (op:// refs) or 1Password or ask user.
- **Full coding standards:** `.claude/CODING_GUIDELINES.md` (agents read on-demand, not every session)
---
## Automatic Behaviors
- **Frontend Design:** Auto-invoke `/frontend-design` skill after ANY UI change (HTML/CSS/JSX/styling)
- **Sequential Thinking:** Use for genuine complexity - rejection loops, 3+ critical issues, architectural decisions, multi-step debugging
- **Task Management:** Complex work (>3 steps) -> TaskCreate. Persist to `.claude/active-tasks.json`.
---
## Context Recovery
When user references previous work, use `/context` command. Never ask user for info in:
- `credentials.md` - Infrastructure reference with `op://` paths (secrets in 1Password)
- `session-logs/` - Daily work logs (also in `projects/*/session-logs/` and `clients/*/session-logs/`)
- `SESSION_STATE.md` - Project history
### 1Password Credential Access
Credentials are stored in 1Password across 4 vaults: **Infrastructure**, **Clients**, **Projects**, **MSP Tools**.
**To read a secret:** `op read "op://VaultName/ItemTitle/field_name"`
**Service account (non-interactive):** Set `OP_SERVICE_ACCOUNT_TOKEN` env var. Token stored in `op://Infrastructure/Service Account Auth Token: Agentic_Cli/credential`. The service account has Read & Write on all 4 vaults (except Projects which is read-only -- use desktop app auth for Projects writes).
**Setup on new machines:**
1. Install 1Password CLI: https://developer.1password.com/docs/cli/get-started/
2. Sign in: `op signin` (or use desktop app integration)
3. For non-interactive use, add to shell config: `set -gx OP_SERVICE_ACCOUNT_TOKEN "token_value"`
---
## Commands & Skills
| Command | Purpose |
|---------|---------|
| `/checkpoint` | Dual checkpoint: git commit + database context |
| `/save` | Comprehensive session log (credentials, decisions, changes) |
| `/context` | Search session logs, credentials.md, and 1Password |
| `/1password` | 1Password secrets management integration |
| `/sync` | Sync config from Gitea repository |
| `/create-spec` | Create app specification for AutoCoder |
| `/frontend-design` | Modern frontend design patterns (auto-invoke after UI changes) |
---
## File Placement (Quick Rules)
- **Dataforth DOS work** -> `projects/dataforth-dos/`
- **ClaudeTools API code** -> `api/`, `migrations/` (existing structure)
- **Client work** -> `clients/[client-name]/`
- **Session logs** -> project or client `session-logs/` subfolder; general -> root `session-logs/`
- **Full guide:** `.claude/FILE_PLACEMENT_GUIDE.md` (read when saving files, not every session)
---
## Local AI (Ollama)
Ollama runs locally with GPU acceleration. Use it for tasks that don't need Claude-level reasoning.
### Available Models
| Model | Size | Use For |
|-------|------|---------|
| `qwen3:14b` | 9.3 GB | General sub-tasks: summarization, classification, data extraction, drafting |
| `codestral:22b` | 12 GB | Code-specific sub-tasks: code generation, refactoring suggestions, docstring generation |
| `nomic-embed-text` | 274 MB | Embeddings only (used by GrepAI, not for direct use) |
### GrepAI (Semantic Code Search)
GrepAI indexes the codebase using `nomic-embed-text` embeddings and provides semantic search via MCP server.
**When to use GrepAI instead of Grep/Glob:**
- Finding code by intent ("how does authentication work") rather than exact text
- Exploring unfamiliar areas of the codebase
- Finding related implementations across files
- Context recovery — searching session logs and credentials by meaning
**How to use:**
- **MCP tool:** Use the `grepai` MCP server tools directly (available after MCP loads)
- **deep-explore agent:** Delegate to the `deep-explore` agent for thorough semantic exploration
- **CLI fallback:** `grepai search "your query" --json --compact`
**Maintenance:** The watcher daemon runs in the background and auto-indexes file changes. If search results seem stale, run `grepai watch --stop && grepai watch --background` to restart it.
### Using Ollama for Sub-Tasks
For bulk or repetitive work that doesn't require Claude's full reasoning, offload to local models via Ollama's API:
**When to use Ollama:**
- Processing many items in a loop (e.g., summarizing 50 session logs)
- Generating boilerplate or repetitive code patterns
- Data extraction/classification from structured text
- Draft content that Claude will review/refine
- Any task where speed > quality and results will be verified
**When NOT to use Ollama (use Claude instead):**
- Architectural decisions or complex reasoning
- Security-sensitive code review
- Tasks requiring tool use or multi-step planning
- Final output that goes directly to production
**How to call Ollama:**
```bash
# Simple prompt
curl -s http://localhost:11434/api/generate -d '{"model":"qwen3:14b","prompt":"Summarize this: ...","stream":false}' | jq -r '.response'
# Chat format
curl -s http://localhost:11434/api/chat -d '{"model":"codestral:22b","messages":[{"role":"user","content":"Refactor this function: ..."}],"stream":false}' | jq -r '.message.content'
### /sync behavior
```
git fetch origin
git rebase origin/main
git push origin ad2
```
### Ollama Output Review Policy
---
The coordinator (Claude) must review Ollama outputs based on impact level. Local models are useful but unreliable — they hallucinate, miss edge cases, and produce subtly wrong code.
## Dataforth Network
**Impact levels and review requirements:**
| Host | IP | Role | Notes |
|------|-----|------|-------|
| AD1 | 192.168.0.27 | Primary DC | Disk at 90%, C:\Engineering = 787 GB |
| **AD2** | **192.168.0.6** | **This machine** | Secondary DC, TestDataDB, file shares |
| D2TESTNAS | 192.168.0.9 | SMB1 proxy for DOS | Debian 13, Samba, SSH root/Paper123!@#-nas |
| UDM | 192.168.0.254 | Gateway/Router | UniFi Dream Machine |
| ESXi-122 | 192.168.0.122 | Hypervisor | ESXi |
| ESXi-124 | 192.168.0.124 | Hypervisor | ESXi |
| DOS stations | TS-01 to TS-30+ | Test stations | DOS 6.22, QuickBASIC ATE software |
| Level | Review | Examples |
|-------|--------|----------|
| **Critical** | ALWAYS review, verify against source | Code touching auth/security/encryption, credential handling, database migrations, production config, anything user-facing |
| **High** | Review for correctness, spot-check details | API endpoint logic, business rules, infrastructure scripts, client-specific work |
| **Medium** | Skim for obvious errors, trust if reasonable | Internal documentation drafts, session log summaries, data extraction from structured input, boilerplate code |
| **Low** | Trust without review | Classification/tagging of items, reformatting text, generating placeholder content for later editing |
**Review process for Critical/High:**
1. Read Ollama's full output — don't just check if it "looks right"
2. Verify claims against actual files/data (e.g., if it says a function exists, confirm it does)
3. Check for: hallucinated function names, wrong parameter types, missing error handling, security gaps
4. If output is wrong or uncertain, redo the task yourself rather than patching Ollama's attempt
**Batch processing pattern:**
When using Ollama for bulk tasks (e.g., processing N items), review the first 2-3 results fully before trusting the rest. If any are wrong, switch to doing it yourself or fix the prompt and reprocess.
**Flag to user:** If Ollama produces output for a Critical task and you are not confident in your review, tell the user explicitly: "This was generated by a local model and I'm not fully confident in [specific concern]."
### Credentials
- AD Sysadmin: INTRANET\sysadmin / Paper123!@#
- D2TESTNAS SSH: root@192.168.0.9 / Paper123!@#-nas
- D2TESTNAS Samba: guest access (no password)
- WINS/NPS: 192.168.0.27:1812/1813
- M365 Tenant: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
- Rsync daemon (NAS): port 873, module "test", user rsync / IQ203s32119
---
## Memory (Shared Across Machines)
## Local Resources
Claude Code's auto-memory is stored **in-repo** at `.claude/memory/` so it syncs via Gitea to all workstations.
**IMPORTANT for all machines:** Configure Claude Code to use the repo memory path, NOT the default `~/.claude/projects/` path. When the auto-memory system prompts you to write to `~/.claude/projects/-home-guru-ClaudeTools/memory/`, write to `.claude/memory/` (repo-relative) instead. The index file is `.claude/memory/MEMORY.md`.
This ensures memory created on one workstation (CachyOS, Mac, Windows) is available on all others after a git pull/sync.
| Resource | Path |
|----------|------|
| TestDataDB app | C:\Shares\testdatadb\ |
| Test database | C:\Shares\testdatadb\database\testdata.db (SQLite, 2.2M+ records) |
| TestDataDB API | http://localhost:3000 |
| Parsers | C:\Shares\testdatadb\parsers\ (multiline.js, csvline.js, shtfile.js, spec-reader.js) |
| Templates | C:\Shares\testdatadb\templates\datasheet-exact.js |
| Import script | C:\Shares\testdatadb\database\import.js |
| Export script | C:\Shares\testdatadb\database\export-datasheets.js |
| Stage import | C:\Shares\testdatadb\import-all-stage.js |
| NAS share | \\D2TESTNAS\test (mapped as T:) |
| Datasheets share | X:\For_Web |
| ProdSW (BAT files) | C:\Shares\test\COMMON\ProdSW\ |
| Sync script | C:\Shares\test\scripts\Sync-FromNAS.ps1 (bidirectional, 15-min schedule) |
---
## Reference (read on-demand, not every session)
## DOS Update System - Batch Files
- **Project structure, endpoints, workflows, troubleshooting:** `.claude/REFERENCE.md`
- **Agent definitions:** `.claude/agents/*.md`
- **MCP servers:** `MCP_SERVERS.md`
- **Coding standards:** `.claude/CODING_GUIDELINES.md`
- **Shared memory:** `.claude/memory/MEMORY.md` (index) + `.claude/memory/*.md` (individual memories)
### Boot Sequence on DOS Machines
```
AUTOEXEC.BAT (v4.1)
-> STARTNET.BAT (v2.0) -- init network, map T: and X: drives
-> ATESYNC.BAT
-> CTONW.BAT (v5.0) -- upload test data to network
-> CTONWTXT.BAT (v2.3) -- upload C:\STAGE\*.TXT to T:\STAGE\%MACHINE%
-> NWTOC.BAT (v5.0) -- download updates from network
```
### Current Production Versions (on AD2 & NAS)
| File | Version | Last Update | Purpose |
|------|---------|-------------|---------|
| AUTOEXEC.BAT | v4.1 | 2026-03-12 | Startup config |
| STARTNET.BAT | v2.0 | 2026-01-20 | Network init |
| NWTOC.BAT | v5.0 | 2026-03-16 | Download updates from network |
| CTONW.BAT | v5.0 | 2026-03-28 | Upload test data (5 steps with echo) |
| CTONWTXT.BAT | v2.3 | 2026-03-28 | Upload Stage TXT files (no MD, dirs pre-created) |
| CHECKUPD.BAT | v1.3 | 2026-01-20 | Check for updates |
| UPDATE.BAT | v2.3 | 2026-01-20 | Full system backup |
| STAGE.BAT | v1.0 | Original | Stage system file updates |
| DEPLOY.BAT | v1.0 | 2026-01-20 | One-time deployment installer |
### DOS 6.22 Compatibility Rules
- NO `IF NOT` -- unreliable on DOS 6.22. Use positive `IF EXIST` with GOTO
- NO `IF /I` (case-insensitive compare)
- NO `FOR /F` loops
- NO `%COMPUTERNAME%` -- use `%MACHINE%` (set during DEPLOY)
- `XCOPY /D` requires date parameter (`/D:mm-dd-yy`)
- `MD` fails with error on existing directories -- pre-create dirs server-side
- `COPY` without `/Y` hangs on overwrite prompts
- All paths UPPERCASE for Samba compatibility
- Line endings MUST be CRLF (0D 0A)
---
**Last Updated:** 2026-03-22
## Serial Number Encoding (DOS 8.3 filenames)
QuickBASIC ATE encodes long serial numbers for 8.3 filenames:
```
First 2 digits replaced with hex letter if serial too long:
178236-12 -> H8236-12.TXT (17 -> H, charCode 72 - 55 = 17)
10819-1 -> A819-1.TXT (10 -> A, charCode 65 - 55 = 10)
Decode: letter.charCodeAt(0) - 55 = numeric prefix
Only applies when filename starts with [A-Z] followed by digits.
H-prefix files have decoded SN inside the file (SN: 178236-12)
A-prefix files have encoded SN inside the file (SN: A819-1) -- must decode to 10819-1
```
---
## Test Datasheet Pipeline
### 5-Stage Architecture
1. **DOS Test Programs** -> Write DAT files to C:\ATE\*LOG\ and TXT to C:\STAGE\
2. **Boot Upload** -> CTONW.BAT copies DAT to T:\%MACHINE%\LOGS\, CTONWTXT copies TXT to T:\STAGE\%MACHINE%
3. **NAS <-> AD2 Sync** -> Rsync every 15 min (Sync-FromNAS.ps1 scheduled task)
4. **TestDataDB Import** -> import.js parses DAT into SQLite; export-datasheets.js generates TXT to X:\For_Web
5. **Web Share** -> X:\For_Web\ holds validated datasheets (501K+ files)
### import-all-stage.js (ready to run)
Located at `C:\Shares\testdatadb\import-all-stage.js`. Processes ~8,100 TXT files:
- Scans \\D2TESTNAS\test\STAGE\TS-*\*.TXT
- Decodes hex-prefix serial numbers
- Cross-references testdata.db by (serial_number, model_number)
- Inserts missing records as log_type='SHT'
- Copies to X:\For_Web\{decoded_serial}.TXT
```
cd C:\Shares\testdatadb
node import-all-stage.js
```
### Machine data volumes in STAGE
| Machine | Files |
|---------|-------|
| TS-4L | 3,082 |
| TS-4R | 2,741 |
| TS-1R | 509 |
| TS-8R | 478 |
| TS-3R | 435 |
| TS-11R | 325 |
| TS-8L | 285 |
| TS-11L | 248 |
| TS-27 | 10 (already imported) |
| TS-1L | 1 |
### Web Share Layout (X:\)
- X:\For_Web -- Validated datasheets (production)
- X:\For_Web_PDF -- PDF versions (4.7K files)
- X:\Test_Datasheets -- Incoming/staging
- X:\Bad_Datasheets -- Invalid files (18K)
- X:\Datasheets_Log -- Processing logs
---
## Known Issues & Pending Work
### HIGH PRIORITY
1. **Run import-all-stage.js** -- 8,100 TXT files need cross-referencing and ingestion
2. **Website Upload Replacement** -- Old ASP.NET endpoints (Uploader.aspx) return 404. Need new approach.
3. **7B Series Datasheets** -- ~830K records can't generate datasheets (missing 7BMAIN.DAT spec file). Check ENGR share.
4. **Service Permissions** -- testdatadb runs as SYSTEM, causing file permission issues. Change to INTRANET\sysadmin.
### MEDIUM PRIORITY
5. **C2 IP Blocking** -- iptables rules added to UDM for 80.76.49.18 and 45.88.91.99. Need permanent rules in UniFi UI.
6. **MFA Enforcement** -- 19/38 users ready. Report-only until April 4, 2026. Monitor registration.
7. **Joel Lohr Account** -- Retiring March 31. Disable account post-retirement. Auto-reply set to Dan Center.
---
## Security Incident (2026-03-27)
**DF-JOEL2 (192.168.0.143) compromised via phishing:**
- Joel Lohr clicked phishing link in personal Yahoo email
- ScreenConnect C2 installed, "Angel Raya" connected remotely
- Two C2 backdoors deployed via PowerShell
- C2 IPs: 80.76.49.18, 45.88.91.99 (AS399486, suspended by host)
- IC3 Complaint: 1c32ade367084be9acd548f23705736f
- ConnectWise Case: 03464184
- **Remediation complete:** IPs blocked, 3 rogue clients removed, password reset, sessions revoked
- **No lateral movement detected** (32 machines scanned clean)
---
## Key Contacts
| Person | Email | Role |
|--------|-------|------|
| John Lehman | jlehman@dataforth.com | Engineering, QB code, test specs |
| Dan Center | dcenter@dataforth.com | Operations (replacing Joel) |
| Peter Iliya | pIliya@dataforth.com | Applications Engineer |
| AJ | dataforthgit@... | Engineering contact |
| Ken Hoffman | (unresponsive) | TestDataSheetUploader author |
| Georg Haubner | ghaubner@dataforth.com | Has pre-crypto backup on D: drive |
---
## Quick Reference Commands
```powershell
# Check BAT files on NAS
ssh root@192.168.0.9 'ls -la /data/test/COMMON/ProdSW/'
# Trigger NAS sync
Start-ScheduledTask -TaskName 'Sync-FromNAS'
# Check sync log
Get-Content 'C:\Shares\test\scripts\sync-from-nas.log' -Tail 20
# Check TestDataDB health
curl http://localhost:3000/health
# Query test records
node -e "const db=require('better-sqlite3')('C:\\Shares\\testdatadb\\database\\testdata.db',{readonly:true});console.log(db.prepare('SELECT COUNT(*) as cnt FROM test_records').get())"
# Check Stage files on NAS
ssh root@192.168.0.9 'find /data/test/STAGE -name "*.TXT" | wc -l'
```
---
**Last Updated:** 2026-03-29