claudetools/projects/dataforth-dos/session-logs/2026-03-12-session.md

# Session Log: 2026-03-12 - D2TESTNAS VM Build, NAS Migration, Rsync Sync Fix

## Session Summary

Major infrastructure session: replaced broken SCP-based sync with rsync, built a new Debian 13 VM to replace the aging ReadyNAS, transferred data, and performed IP cutover. Also investigated BTRFS snapshots on old NAS and began DOS machine testing against new Linux-based NAS.

### Key Accomplishments
1. **Fixed Sync-FromNAS.ps1 on AD2** - Replaced broken SCP with rsync daemon protocol, added guards for stray files (TS-21, TS-3R/HVLOG), added log file write retry for AV locking
2. **Disabled old SCP scheduled tasks on AD2** - Killed Sync-FromNAS and BulkSync-Catchup tasks
3. **Built D2TESTNAS replacement VM** on DF-HYPERV-B (Debian 13, Samba SMB1, rsync daemon, BTRFS 512GB data disk)
4. **Transferred data from old NAS** - test/ data (~24GB+), datasheets, home, 82 snapshots (partial ~43GB logical)
5. **IP cutover completed** - New VM now at 192.168.0.9, old NAS on DHCP at 192.168.0.117
6. **WINS/NetBIOS conflict resolved** - Killed nmbd on old NAS, removed auto-restart cron, blocked ports 137/138 via iptables

### Key Decisions
- Chose Hyper-V VM on DF-HYPERV-B over repurposing physical server DF-SVR-D2-SYNC
- Used BTRFS for data disk with subvolumes for test and datasheets
- Single rsync stream to avoid overloading old NAS (ARM processor)
- BTRFS snapshots from old NAS are being flattened (CoW -> full copies) which makes them much larger than ReadyNAS UI reported

### Problems Encountered and Solutions
- **TS-21 stray file**: 1,129-byte DAT file from 2012 existed instead of directory. Renamed, added script guard.
- **TS-3R/LOGS/HVLOG stray file**: 56-byte file from 2013. Same fix.
- **Log file locking**: AV locking sync-from-nas.log. Added 3-retry with 100ms delay.
- **AD2 high latency**: AV causing 685-1056ms ping. Recommended exclusions.
- **NAS freezing under SSH load**: Power cycled, limited to single rsync stream.
- **nmbd auto-restart on old NAS**: Cron `*/5 * * * * pgrep -x nmbd || /usr/sbin/nmbd -D`. Removed cron, blocked ports via iptables.
- **nmcli config didn't save first attempt**: SSH dropped before apply. Re-ran successfully.
- **DOS Error 53 (network path not found)**: Old NAS still broadcasting D2TESTNAS name. Fixed by killing nmbd and blocking NetBIOS ports.

---

## Credentials

### New D2TESTNAS VM (Debian 13)
- **IP**: 192.168.0.9 (static via NetworkManager)
- **SSH**: root / Paper123!@# (also localadmin / Paper123!@#)
- **SSH Key**: ed25519 generated on VM, public key installed on old NAS
- **Key fingerprint**: SHA256:S2Eom4RwHS/8YMu+ePnOmDOJxGhIkxJQ2ocR3WsH24o root@D2TESTNAS

### Mac SSH Key (add to AD2 and D2TESTNAS)
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDrGbr4EwvQ4P3ZtyZW3ZKkuDQOMbqyAQUul2+JE4K4S azcomputerguru@local
```
**Add to:**
- AD2: `C:\Users\sysadmin\.ssh\authorized_keys`
- D2TESTNAS: `/root/.ssh/authorized_keys`

### Rsync Daemon (new VM)
- **Port**: 873
- **Module**: test = /data/test
- **User**: rsync
- **Password**: IQ203s32119
- **Config**: /etc/rsyncd.conf
- **Secrets**: /etc/rsyncd.secrets

### Samba (new VM)
- **Shares**: test (/data/test), datasheets (/data/datasheets), snapshots (/data/test/.snapshots)
- **Protocol**: SMB1 (CORE) through SMB3
- **Auth**: Guest OK on all shares
- **Workgroup**: D2TESTING
- **NetBIOS name**: D2TESTNAS
- **WINS support**: yes

### Old NAS (ReadyNAS)
- **Current IP**: 192.168.0.117 (DHCP, was 192.168.0.9)
- **MAC**: 28:C6:8E:34:4B:5E
- **SSH**: root (key-based auth from new VM)
- **Status**: nmbd killed, cron cleared, NetBIOS ports blocked via iptables. Samba stopped. SSH still works for rsync transfers.

### AD2 (Windows Server)
- **IP**: 192.168.0.6
- **Sync script**: C:\Scripts\Sync-FromNAS-rsync.ps1 (deployed, dry-run validated)
- **Test data path**: C:\Shares\test\
- **cwRsync**: Installed via Chocolatey

### DF-SVR-D2-SYNC (unused physical server)
- **IP**: 192.168.0.93
- **Creds**: sysadmin / Paper123!@#
- **HP ProLiant ML350 G6, 64GB RAM, Server 2019**
- **SMB share**: NAS-BACKUP (was used temporarily for CIFS backup attempt)
- **SMB1 enabled** on this server

### UDM Network
- **WINS server**: 192.168.0.9 (configured in UDM DHCP option 44)

---

## Infrastructure

### New D2TESTNAS VM Configuration
- **Host**: DF-HYPERV-B (dedicated Hyper-V host)
- **OS**: Debian 13 (Trixie)
- **Network**: eth0, static 192.168.0.9/24, gateway 192.168.0.1
- **Disks**:
  - /dev/sda: OS disk
  - /dev/sdb: 512GB BTRFS data disk mounted at /data
- **BTRFS subvolumes**: test, datasheets (under /data)
- **Services**: smbd, nmbd, rsync (daemon), sshd, cron
- **Snapshot cron**:
  ```
  0 * * * * /usr/local/bin/btrfs-snapshot.sh test 48
  0 * * * * /usr/local/bin/btrfs-snapshot.sh datasheets 48
  0 0 * * * /usr/local/bin/btrfs-snapshot.sh test 30
  0 0 * * 0 /usr/local/bin/btrfs-snapshot.sh test 12
  ```

### Key Config Files on New VM
- `/etc/samba/smb.conf` - Samba config (SMB1/CORE, DOS charset CP437, WINS)
- `/etc/rsyncd.conf` - rsync daemon (module "test")
- `/etc/rsyncd.secrets` - rsync auth (rsync:IQ203s32119)
- `/usr/local/bin/btrfs-snapshot.sh` - BTRFS snapshot script

### Data Transfer Status (as of ~18:30)
- **test/ data (excl snapshots)**: ~24 GB transferred, rsync still running (single stream from .117)
- **test/ snapshots**: ~43 GB logical transferred (82 snapshots), transfer was stopped to reduce NAS load - needs restart
- **datasheets/ + snapshots**: Complete (2.3 MB + 82 snapshot dirs)
- **home/**: Complete (612 KB)
- **Disk usage**: ~26 GB actual on BTRFS (CoW dedup), 486 GB free
- **Note**: ReadyNAS UI reported 5.26GB data + 16.28GB snapshots, but actual rsync transfer is MUCH larger due to BTRFS CoW flattening

---

## Files Created/Modified

### New Files
- `D:\ClaudeTools\projects\dataforth-dos\sync-fixes\Sync-FromNAS-rsync.ps1` - Complete rsync-based replacement sync script (deployed to AD2)
- `D:\ClaudeTools\projects\dataforth-dos\d2testnas-vm\setup-d2testnas.sh` - 522-line post-install setup script
- `D:\ClaudeTools\projects\dataforth-dos\d2testnas-vm\README.md` - Hyper-V creation commands, Debian install notes, cutover checklist

### Script Fixes Applied (Sync-FromNAS-rsync.ps1)
1. **Directory-only filter** for NAS station enumeration (line ~125)
2. **Station path guard** - detects stray files where directories expected
3. **Log type directory guard** - renames stray files in LOGS subdirs
4. **Write-Log retry** - 3 attempts with 100ms delay for AV file locking

### Deployed to New VM (via SSH)
- /etc/samba/smb.conf (full Samba config)
- /etc/rsyncd.conf + /etc/rsyncd.secrets
- /usr/local/bin/btrfs-snapshot.sh + cron entries
- SSH key pair generated, public key added to old NAS

---

## Pending/Incomplete Tasks

### Immediate (resume next session)
1. **Monitor test/ data rsync** - Single stream running from old NAS (.117) to new VM (.9). Check with:
   ```bash
   ssh root@192.168.0.9 "ps aux | grep 'rsync -av' | grep -v grep; du -sh /data/test/ --exclude=.snapshots"
   ```
2. **Restart snapshot transfer** after data transfer completes:
   ```bash
   ssh root@192.168.0.9 "nohup bash -c 'rsync -av root@192.168.0.117:/data/test/.snapshots/ /data/test/.snapshots/ 2>&1 | tail -5' &"
   ```
3. **Test DOS machine connectivity** - Error 53 was resolved (old NAS NetBIOS killed). Need to reboot DOS machine and test:
   - `NET USE T: \\D2TESTNAS\TEST`
   - Run CTONW.BAT (copy logs to NAS)
   - Run NWTOC.BAT (download updates from NAS)
   - Verify files appear in /data/test/TS-XX/LOGS/ on new VM

### After Data Transfer Complete
4. **Verify data integrity** - Compare file counts/sizes between old and new NAS
5. **Power off old NAS** once all data confirmed transferred
6. **Set up scheduled task on AD2** - Create 15-minute scheduled task for Sync-FromNAS-rsync.ps1
7. **Run real (non-dry) sync on AD2** - Execute Sync-FromNAS-rsync.ps1 without -DryRun flag
8. **AV exclusions on AD2** - Add exclusions for C:\Shares\test\ and rsync.exe

### Nice to Have
9. **Copy NAS config backup to new VM** (already backed up to DF-SVR-D2-SYNC)
10. **Datto Workplace SmartBadge research** - Researched that SmartBadge add-in for Excel doesn't exist; Workplace integrates via sync client and web, not Excel plugin

---

## DOS Machine Data Flow

```
DOS 6.22 (C:\ATE\) --COPY--> T:\MACHINE\LOGS\ (NAS via SMB1)
                                    |
                                    v (rsync daemon, port 873)
                              AD2 C:\Shares\test\
                                    |
                                    v (future: database ingestion)
                              MariaDB @ 172.16.3.30
```

### Batch Files (DOS -> NAS)
- **CTONW.BAT v3.2** - Uses COPY (not XCOPY) to upload log files from C:\ATE\ to T:\MACHINE\LOGS\
- **NWTOC.BAT v3.5** - Uses COPY to download updates from T:\COMMON\ProdSW\ to C:\BAT\ and C:\ATE\
- **UPDATE.BAT v2.1** - Uses XCOPY for full machine backup (had /D flag fix for DOS 6.22)

### Log Types
5BLOG, 7BLOG, 8BLOG, DSCLOG, SCTLOG, VASLOG, PWRLOG, HVLOG

### Active Stations
TS-3L (most recent activity), TS-4R, TS-3R, TS-11L, TS-GURU, plus many others

---

## Reference

### Key Commands
```bash
# SSH to new D2TESTNAS
ssh root@192.168.0.9

# SSH to old NAS (DHCP)
ssh root@192.168.0.117

# Check rsync transfers on new VM
ssh root@192.168.0.9 "ps aux | grep rsync | grep -v grep"

# Test Samba from Windows
net view \\192.168.0.9
smbclient -L //192.168.0.9 -N

# Test rsync daemon
rsync rsync://rsync@192.168.0.9/test/

# Restart services on new VM
ssh root@192.168.0.9 "systemctl restart smbd nmbd rsync"

# BTRFS snapshot status
ssh root@192.168.0.9 "ls /data/test/.snapshots/"
```

### Old NAS Lockdown Commands (already applied)
```bash
# Block NetBIOS (prevents name conflict)
ssh root@192.168.0.117 "iptables -A INPUT -p udp --dport 137 -j DROP; iptables -A INPUT -p udp --dport 138 -j DROP; iptables -A OUTPUT -p udp --sport 137 -j DROP; iptables -A OUTPUT -p udp --sport 138 -j DROP"

# Remove auto-restart cron
ssh root@192.168.0.117 "crontab -r"
```

---

## Session Timeline
- Started: ~14:00 (context recovery from previous session)
- Rsync script fixes and deployment to AD2
- Disabled old SCP scheduled tasks
- Investigated BTRFS snapshots (81 found)
- Built D2TESTNAS VM on DF-HYPERV-B (Debian 13)
- Configured all services (Samba, rsync, BTRFS, SSH)
- Started data transfer from old NAS
- Killed snapshot transfer to reduce NAS load (single stream)
- IP cutover: new VM .185 -> .9, old NAS .9 -> DHCP .117
- Resolved WINS conflict (killed old NAS nmbd, removed cron, blocked ports)
- DOS machine testing started - Error 53 resolved
- Data transfer ongoing (~24GB+ transferred, snapshots pending restart)
- Session saved: ~18:45

## Update: ~19:30 - Batch File Fix and DOS Machine Testing

### DOS Machine Testing Results
- All 4 tested machines (TS-3L, TS-3R, TS-4L, TS-4R) connected to new Linux NAS successfully
- T: drive mapped via NetBIOS name (after killing old NAS nmbd)
- Files successfully copied (3 .LOG files)
- BUT: "Bad command or file name" (5x) and "Too many parameters" (5x) errors from IF EXIST/IF NOT EXIST commands
- Confirmed CTONW.BAT v3.2 on machine, correct line endings (CR+LF verified via DEBUG)
- Root cause: DOS 6.22 IF EXIST command failing on network paths - likely SMB1 compatibility issue with wildcard queries

### Fix Applied: Batch Files v4.0
Eliminated all IF EXIST/IF NOT EXIST checks from startup batch files. Directories pre-created on server.

**CTONW.BAT v4.0** - Direct COPY commands, no IF EXIST guards. Target dirs pre-created on NAS.
**NWTOC.BAT v4.0** - Direct MD and COPY commands, no IF EXIST guards. MD harmless if dir exists locally.
**AUTOEXEC.BAT v4.0** - Removed IF EXIST around CALL commands, direct MD for local dirs.

All deployed to NAS at `/data/test/COMMON/ProdSW/`. Machines will pick up new versions on next boot via NWTOC download.

### Pre-created Directories on NAS
Ran script to create LOGS/5BLOG, LOGS/7BLOG, LOGS/8BLOG, LOGS/DSCLOG, LOGS/HVLOG, LOGS/PWRLOG, LOGS/SCTLOG, LOGS/VASLOG, and Reports for ALL TS-* station directories.

### Old NAS Status
- DHCP at 192.168.0.117
- nmbd killed, cron removed, NetBIOS ports 137/138 blocked via iptables
- rsync data transfer still running (single stream, ~24GB+ transferred)
- Snapshot transfer stopped (was at ~43GB logical), needs restart after data completes

### Pending
1. Reboot a DOS machine to test v4.0 batch files (second boot needed for NWTOC v4.0)
2. Monitor data transfer completion (rsync single stream still running as of ~20:00)
3. Restart snapshot transfer after data completes
4. Verify test data appears in correct LOGS subdirectories on NAS
5. Set up AD2 scheduled task for rsync sync
6. Run real (non-dry) Sync-FromNAS-rsync.ps1 on AD2

## Update: ~20:00 - DEPLOY Trailing Space Bug and Data Upload Success

### Critical Bug Found: DEPLOY.BAT Trailing Space
- **Root cause of ALL "Too many parameters" errors**: `ECHO SET MACHINE=%MACHINE% >> C:\AUTOEXEC.BAT` includes the space before `>>` in the output
- This sets `MACHINE=TS-3L ` (with trailing space) which causes `T:\TS-3L \LOGS\DSCLOG` to be parsed as two parameters
- **Fix**: DEPLOY v4.1 moves redirect before ECHO: `>>C:\AUTOEXEC.BAT ECHO SET MACHINE=%MACHINE%`
- First line uses `>` (overwrite), rest use `>>` (append)
- DEPLOY v4.1 deployed to NAS at `/data/test/COMMON/ProdSW/DEPLOY.BAT`

### Samba Case Sensitivity - Confirmed OK
- `smb.conf` has `case sensitive = no` and `default case = upper`
- No duplicate directories (only `TS-4L` exists, not `ts-4L`)

### TS-3L Deploy Test
- Ran `T:\UPDATE TS-3L` which calls DEPLOY v4.0 (before trailing space fix)
- DEPLOY completed, files confirmed v4.0 on machine via TYPE
- After reboot: NAS still showed old CTONW.LOG/NWTOC.LOG - MACHINE had trailing space
- Running CTONW manually showed 9x "Too many parameters" on all COPY-to-subdirectory lines
- `COPY C:\ATE\*.LOG T:\%MACHINE%` worked (no subdirectory in path) but `COPY ... T:\%MACHINE%\LOGS\DSCLOG` failed
- This confirmed the trailing space theory - space before `\LOGS\` splits the path

### TS-4L Data Upload - SUCCESS
- TS-4L uploaded data at 20:10 with clean MACHINE variable (no trailing space)
- **84 test data files uploaded to NAS:**
  - 5BLOG: 20 files
  - 7BLOG: 29 files (historical .SHT files)
  - 8BLOG: 10 files
  - DSCLOG: 21 files (including today's 38-02.DAT from 03-12-26)
  - SCTLOG: 2 files
  - VASLOG: 2 files
- **90+ work-order Reports** (.TXT files) uploaded to TS-4L/Reports/
- **3 LOG files** (NWTOC.LOG, CTONW.LOG, CTONWTXT.LOG)
- CTONW.LOG confirms: `CTONW.BAT v4.0 / Machine: TS-4L` (no trailing space)

### Original STARTNET.BAT Found (from TS-3L backup)
The actual STARTNET.BAT on DOS machines loads network drivers manually:
```
LH /L:0;1,45472 /S c:\net\smartdrv.exe /q
c:\net\net initialize
c:\net\netbind.com
lh c:\net\umb.com
c:\net\tcptsr.exe
c:\net\tinyrfc.exe
c:\net\nmtsr.exe
c:\net\emsbfr.exe
c:\net\net start
net use T: \\d2testnas\test
net use X: \\d2testnas\datasheets
```
- `net start` prompts for computer name (pre-populated from SYSTEM.INI)
- Could add `/y` flag to suppress prompt, or use MACHINE variable
- Our v2.0 STARTNET.BAT on ProdSW is a simplified rewrite that was never deployed to machines

### T:\UPDATE.BAT
- Tiny 4-line wrapper at root of test share: `CALL T:\COMMON\ProdSW\DEPLOY.BAT %1`
- Allows running `T:\UPDATE TS-3L` from DOS machines

### Rsync Transfer Status
- Single stream still running from old NAS (.117) to new VM (.9)
- Snapshot transfer still pending restart

### Files Modified This Update
- `D:\ClaudeTools\projects\dataforth-dos\batch-files\DEPLOY.BAT` - v4.1 (trailing space fix)
- Deployed to NAS at `/data/test/COMMON/ProdSW/DEPLOY.BAT`

---

## Pending/Incomplete Tasks (Updated)

### Immediate
1. **Re-deploy TS-3L with DEPLOY v4.1** - needs new deploy + reboot to fix trailing space
2. **Set up AD2 rsync scheduled task** - Sync-FromNAS-rsync.ps1 deployed but no task created (15-min interval planned)
3. **Run real (non-dry) sync** of Sync-FromNAS-rsync.ps1 on AD2
4. **Database ingestion pipeline** - No ingestion exists yet. Data flows: NAS -> AD2 -> MariaDB @ 172.16.3.30

### After Data Transfer Complete
5. **Monitor old NAS rsync completion** - single stream still running
6. **Restart snapshot transfer** after data completes
7. **Verify data integrity** - compare file counts between old and new NAS
8. **Power off old NAS** once confirmed

### Batch File Updates Needed
9. **UPDATE.BAT** (in ProdSW) - has IF EXIST checks, needs v4.0 treatment
10. **ATESYNC.BAT / ATESYNCD.BAT** - have IF EXIST checks, need v4.0 treatment (not currently called by AUTOEXEC)
11. **STARTNET.BAT** - consider deploying updated version or adding `/y` to suppress net start prompt
12. **AV exclusions on AD2** - add exclusions for C:\Shares\test\ and rsync.exe