sync: auto-sync from DESKTOP-0O8A1RL at 2026-05-12 07:50:21

Author: Mike Swanson
Machine: DESKTOP-0O8A1RL
Timestamp: 2026-05-12 07:50:21

clients/valleywide/session-logs/2026-05-12-session.md

@@ -0,0 +1,104 @@
+# Session Log — 2026-05-12
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** DESKTOP-0O8A1RL
+- **Role:** admin
+- **Session span:** 2026-05-12, approx 05:30–07:50 MST
+
+## Session Summary
+
+Responded to an emergency service call: VWP servers reported unreachable. Connected via VPN and diagnosed the environment. VWP-QBS (172.16.9.169), DC1 (172.16.9.2), and the XenServer host (192.168.0.104) were all confirmed operational. The HP ProLiant DL360 Gen10 host was found powered off — this machine hosts ADSRVR (192.168.0.25) as a VM, which explained why AD/ADSRVR was unreachable across the network.
+
+Connected to the HP iLO management interface at 172.16.9.125 via SSH using paramiko (Python), which required disabling modern RSA signature algorithms (rsa-sha2-256, rsa-sha2-512) to negotiate with the iLO's legacy host key. Issued the SMASH/RIBCL command `start system1` to power the server on. ADSRVR subsequently came back online.
+
+Created Syncro ticket #32269 for the emergency remote response. VWP is a prepaid block customer (11.5 hrs at start of session). Billed via two timer entries — 1.0 hr for the remote work and 0.5 hr for the emergency/same-day surcharge — totaling 1.5 hrs debited against the block. Invoice #67594 created at $0.00 (prepaid deduction). Block updated from 11.5 → 10.0 hrs. Ticket marked Invoiced.
+
+## Key Decisions
+
+- Used paramiko (Python) to connect to iLO rather than system OpenSSH because the iLO's legacy SSH host key algorithms are incompatible with modern OpenSSH's default settings on Windows. Paramiko allows fine-grained control over accepted algorithms via `disabled_algorithms`.
+- Billed emergency as two separate line items (1.0 hr normal + 0.5 hr surcharge) per user's explicit billing pattern for block clients: "Normal remote and Emergency item at 50% of normal time for 150% charge."
+- Used product 1190473 (Labor - Remote Business) for both entries — NOT product 26184 (Labor - Emergency), which has 1.5x baked into its dollar rate and would double-charge the multiplier for a prepaid customer whose block debits by quantity.
+
+## Problems Encountered
+
+- iLO SSH connection rejected by modern OpenSSH: `ssh-rsa`/`ssh-dss` host key algorithms not accepted by default. Resolved by switching to paramiko with `disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}`.
+- Previous Syncro billing attempts used incorrect endpoints (`/ticket_items`, `/line_items`, `/ticket_charges` all returned 404). Resolved by reading syncro.md which documents `timer_entry → charge_timer_entry` as the required path for time-based billing.
+- `PUT /tickets/{id}` with `{"status": "Invoiced"}` returned 400 when the heredoc was piped inline with jq in the same command. Resolved by capturing the curl response in a variable first, then piping to jq separately.
+
+## Configuration Changes
+
+None. No files modified on disk. Ticket and billing records created in Syncro PSA.
+
+## Credentials & Secrets
+
+- HP iLO (172.16.9.125): credentials from vault (used via paramiko SSH during session — see vault at `clients/valleywide/`)
+- XenServer (192.168.0.104): `root` / `r3tr0gradE99!`
+- Syncro API Key (Mike): `T259810e5c9917386b-52c2aeea7cdb5ff41c6685a73cebbeb3` (also in `.claude/commands/syncro.md`)
+
+## Infrastructure & Servers
+
+| Host | IP | Status | Notes |
+|---|---|---|---|
+| VWP-QBS | 172.16.9.169 | UP | QuickBooks + RDS services running normally |
+| DC1 | 172.16.9.2 | UP | Domain controller |
+| XenServer | 192.168.0.104 | UP | All VMs running: BACKUP-SRV, Server 2012 R2, Server 2003 |
+| HP ProLiant DL360 Gen10 | (LAN) | POWERED OFF → ON | Hosts ADSRVR (192.168.0.25) as VM; SN: MXQ80400X4 |
+| HP iLO | 172.16.9.125 | Accessible | SSH port 22, legacy ssh-rsa host key |
+| ADSRVR | 192.168.0.25 | DOWN → UP | VM on HP ProLiant; came back after host powered on |
+| UDM | 172.16.9.1 | UP | Unifi Dream Machine, VPN gateway |
+
+HP ProLiant DL360 Gen10 SN: MXQ80400X4 (documented in 2026-04-22 session log)
+
+## Commands & Outputs
+
+```python
+# iLO SSH power-on via paramiko (legacy algorithm compatibility)
+import paramiko
+transport = paramiko.Transport(('172.16.9.125', 22))
+transport.disabled_algorithms = {'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}
+transport.connect(username='<ilo_user>', password='<ilo_pass>')
+channel = transport.open_session()
+channel.exec_command('start system1')
+# Result: server powered on, ADSRVR came back online
+```
+
+```bash
+# Syncro billing — timer entry 1 (1.0 hr normal remote)
+# POST /tickets/110159277/timer_entry → id: 39153031, active_duration: 3600
+# POST /tickets/110159277/charge_timer_entry {"timer_entry_id": 39153031}
+# → recorded: true, ticket_line_item_id: 42399068
+
+# Timer entry 2 (0.5 hr emergency surcharge)
+# POST /tickets/110159277/timer_entry → id: 39153048, active_duration: 1800
+# POST /tickets/110159277/charge_timer_entry {"timer_entry_id": 39153048}
+# → recorded: true, ticket_line_item_id: 42399076
+
+# Both line items verified at price_retail: 150.0 (no patching needed)
+
+# Invoice
+# POST /invoices {"ticket_id": 110159277, "customer_id": 31694734, "category": "Standard"}
+# → id: 1650271395, number: "67594", total: "0.0" (prepaid — correct)
+# Customer prepay_hours: 11.5 → 10.0 (-1.5)
+
+# Mark Invoiced
+# PUT /tickets/110159277 {"status": "Invoiced"} → status: "Invoiced"
+```
+
+## Pending / Incomplete Tasks
+
+- Confirm ADSRVR is fully stable and domain connectivity restored (no follow-up from user received)
+- Dataforth (separate project): Add `jlehman@dataforth.com` to `toRecipients` in notify.js and run-pipeline.ps1 once next real pipeline run is confirmed working
+- Dataforth: Optionally `npm uninstall nodemailer` on AD2 (unused after Graph API migration, harmless)
+- Dataforth: Clean diagnostic scripts on AD2: `C:\Shares\testdatadb\database\_*.js`
+
+## Reference Information
+
+- Syncro ticket: #32269 (ID: 110159277), customer: Shelly Dooley / Valley Wide P, customer_id: 31694734
+- Syncro invoice: #67594 (ID: 1650271395)
+- Timer entries: 39153031 (1.0 hr), 39153048 (0.5 hr)
+- Line items: 42399068, 42399076
+- Prior VWP emergency session: `clients/valleywide/session-logs/2026-04-22-hp-server-nvram-corruption-emergency.md`
+- HP ProLiant SN: MXQ80400X4 | iLO: 172.16.9.125
+- Syncro billing reference: `.claude/commands/syncro.md`
+- Dataforth Graph API session log (same date): `projects/dataforth-dos/session-logs/2026-05-12-session.md`