claudetools/clients/valleywide/session-logs/2026-05-12-session.md

Session Log — 2026-05-12

User

• User: Mike Swanson (mike)
• Machine: DESKTOP-0O8A1RL
• Role: admin
• Session span: 2026-05-12, approx 05:30–07:50 MST

Session Summary

Responded to an emergency service call: VWP servers reported unreachable. Connected via VPN and diagnosed the environment. VWP-QBS (172.16.9.169), DC1 (172.16.9.2), and the XenServer host (192.168.0.104) were all confirmed operational. The HP ProLiant DL360 Gen10 host was found powered off — this machine hosts ADSRVR (192.168.0.25) as a VM, which explained why AD/ADSRVR was unreachable across the network.

Connected to the HP iLO management interface at 172.16.9.125 via SSH using paramiko (Python), which required disabling modern RSA signature algorithms (rsa-sha2-256, rsa-sha2-512) to negotiate with the iLO's legacy host key. Issued the SMASH/RIBCL command start system1 to power the server on. ADSRVR subsequently came back online.

Created Syncro ticket #32269 for the emergency remote response. VWP is a prepaid block customer (11.5 hrs at start of session). Billed via two timer entries — 1.0 hr for the remote work and 0.5 hr for the emergency/same-day surcharge — totaling 1.5 hrs debited against the block. Invoice #67594 created at $0.00 (prepaid deduction). Block updated from 11.5 → 10.0 hrs. Ticket marked Invoiced.

Key Decisions

• Used paramiko (Python) to connect to iLO rather than system OpenSSH because the iLO's legacy SSH host key algorithms are incompatible with modern OpenSSH's default settings on Windows. Paramiko allows fine-grained control over accepted algorithms via disabled_algorithms.
• Billed emergency as two separate line items (1.0 hr normal + 0.5 hr surcharge) per user's explicit billing pattern for block clients: "Normal remote and Emergency item at 50% of normal time for 150% charge."
• Used product 1190473 (Labor - Remote Business) for both entries — NOT product 26184 (Labor - Emergency), which has 1.5x baked into its dollar rate and would double-charge the multiplier for a prepaid customer whose block debits by quantity.

Problems Encountered

• iLO SSH connection rejected by modern OpenSSH: ssh-rsa/ssh-dss host key algorithms not accepted by default. Resolved by switching to paramiko with disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}.
• Previous Syncro billing attempts used incorrect endpoints (/ticket_items, /line_items, /ticket_charges all returned 404). Resolved by reading syncro.md which documents timer_entry → charge_timer_entry as the required path for time-based billing.
• PUT /tickets/{id} with {"status": "Invoiced"} returned 400 when the heredoc was piped inline with jq in the same command. Resolved by capturing the curl response in a variable first, then piping to jq separately.

Configuration Changes

None. No files modified on disk. Ticket and billing records created in Syncro PSA.

Credentials & Secrets

• HP iLO (172.16.9.125): credentials from vault (used via paramiko SSH during session — see vault at clients/valleywide/)
• XenServer (192.168.0.104): root / r3tr0gradE99!
• Syncro API Key (Mike): T259810e5c9917386b-52c2aeea7cdb5ff41c6685a73cebbeb3 (also in .claude/commands/syncro.md)

Infrastructure & Servers

Host	IP	Status	Notes
VWP-QBS	172.16.9.169	UP	QuickBooks + RDS services running normally
DC1	172.16.9.2	UP	Domain controller
XenServer	192.168.0.104	UP	All VMs running: BACKUP-SRV, Server 2012 R2, Server 2003
HP ProLiant DL360 Gen10	(LAN)	POWERED OFF → ON	Hosts ADSRVR (192.168.0.25) as VM; SN: MXQ80400X4
HP iLO	172.16.9.125	Accessible	SSH port 22, legacy ssh-rsa host key
ADSRVR	192.168.0.25	DOWN → UP	VM on HP ProLiant; came back after host powered on
UDM	172.16.9.1	UP	Unifi Dream Machine, VPN gateway

HP ProLiant DL360 Gen10 SN: MXQ80400X4 (documented in 2026-04-22 session log)

Commands & Outputs

# iLO SSH power-on via paramiko (legacy algorithm compatibility)
import paramiko
transport = paramiko.Transport(('172.16.9.125', 22))
transport.disabled_algorithms = {'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}
transport.connect(username='<ilo_user>', password='<ilo_pass>')
channel = transport.open_session()
channel.exec_command('start system1')
# Result: server powered on, ADSRVR came back online

# Syncro billing — timer entry 1 (1.0 hr normal remote)
# POST /tickets/110159277/timer_entry → id: 39153031, active_duration: 3600
# POST /tickets/110159277/charge_timer_entry {"timer_entry_id": 39153031}
# → recorded: true, ticket_line_item_id: 42399068

# Timer entry 2 (0.5 hr emergency surcharge)
# POST /tickets/110159277/timer_entry → id: 39153048, active_duration: 1800
# POST /tickets/110159277/charge_timer_entry {"timer_entry_id": 39153048}
# → recorded: true, ticket_line_item_id: 42399076

# Both line items verified at price_retail: 150.0 (no patching needed)

# Invoice
# POST /invoices {"ticket_id": 110159277, "customer_id": 31694734, "category": "Standard"}
# → id: 1650271395, number: "67594", total: "0.0" (prepaid — correct)
# Customer prepay_hours: 11.5 → 10.0 (-1.5)

# Mark Invoiced
# PUT /tickets/110159277 {"status": "Invoiced"} → status: "Invoiced"

Pending / Incomplete Tasks

• Confirm ADSRVR is fully stable and domain connectivity restored (no follow-up from user received)
• Dataforth (separate project): Add jlehman@dataforth.com to toRecipients in notify.js and run-pipeline.ps1 once next real pipeline run is confirmed working
• Dataforth: Optionally npm uninstall nodemailer on AD2 (unused after Graph API migration, harmless)
• Dataforth: Clean diagnostic scripts on AD2: C:\Shares\testdatadb\database\_*.js

Reference Information

• Syncro ticket: #32269 (ID: 110159277), customer: Shelly Dooley / Valley Wide P, customer_id: 31694734
• Syncro invoice: #67594 (ID: 1650271395)
• Timer entries: 39153031 (1.0 hr), 39153048 (0.5 hr)
• Line items: 42399068, 42399076
• Prior VWP emergency session: clients/valleywide/session-logs/2026-04-22-hp-server-nvram-corruption-emergency.md
• HP ProLiant SN: MXQ80400X4 | iLO: 172.16.9.125
• Syncro billing reference: .claude/commands/syncro.md
• Dataforth Graph API session log (same date): projects/dataforth-dos/session-logs/2026-05-12-session.md