ASUS RT-AC5300 Wi-Fi Optimized + CRS326 Switch Port Map Recovered and Documented On-Device
ASUS RT-AC5300 (192.168.1.4, DD-WRT r64764, AP mode, the house’s main Wi-Fi): tuned all three radios. 2.4GHz moved channel 11 → 1 to de-conflict from the sec-cam Buffalo WDS bridge (which now owns ch11, see TASK 289); Airtime Fairness enabled on all 3; ACK-timing "Sensitivity Range" 500m → 60m on all 3 (500 was inflating the ACK timeout on an in-house link and costing throughput); DTIM 1 → 3 on all 3 (deeper client sleep, better phone/IoT battery); beacon interval normalized to 100 (the two 5GHz radios were on an odd 325/350); 2.4GHz TX power 200 → 150 mW ("loud but deaf" reduction — max power hurts SNR and roaming and widens the interference footprint), 5GHz left at 200. Root-caused the Aug-26 "saves silently didn’t stick" mystery on this box: apply.cgi updates nvram in RAM (so a re-read looks saved) but does not reliably commit to flash — a mid-session httpd crash (triggered by a malformed test POST with a bad submit_button value) plus a power cycle wiped every change back to baseline. Also learned this build’s Wireless_Basic form clears any per-radio channel field omitted from the POST (blanked wl2_channel twice before it was caught), and that the bad POST wedged the entire management plane — no ping/SSH/HTTP to the router while Wi-Fi and forwarding kept running — needing a power cycle to recover. Everything was then re-applied through the browser’s own Apply Settings (which does perform the flash commit) and confirmed held across two reboots. Payoff: immediately after the 2.4GHz de-conflict the sec-cam WDS bridge posted its best numbers of the day — 3.7ms latency, 1.8ms jitter, 0 drops. CRS326 switches (192.168.1.2 / 192.168.1.3): discovered they had been converted from SwOS to RouterOS (7.16.1 / 7.14.1) sometime since the July inventory — which is why every SwOS .b endpoint now 404s. Rebuilt the complete "what’s plugged where" map live via the RouterOS REST API (per-port bridge host tables + neighbor discovery, cross-referenced against the RB5009’s DHCP leases): the backbone is RB5009 → .3 (router-side) → .2 (Proxmox-side) → Proxmox over three 10G SFP+ hops, with every populated access port identified (2× HDHomeRun tuners, several Raspberry Pis, LinBPQ, desktops, two downstream switches, the Buffalo/mesh/IoT segment). Wrote descriptive comments onto all 19 used ports across both switches (SFP+ trunks/uplinks + access ports) so the map now lives on the devices and can’t be lost again — RouterOS auto-persists config to flash, so permanent. Flagged for a physical follow-up: the switch identity strings are swapped versus their room labels (.3 is named "MikroTik-Bedroom" but sits router-side; the July audit had it the other way) — needs an eyeball at the rack to correct the hostnames; and six ports (four showing link with zero traffic, two unidentified static-IP devices) need a physical trace, noted in the port comments themselves. 2026.09.03.
Sec-Cam WDS Bridge (Buffalo .6 → .5) — Recurring Drops Diagnosed, 30-Second Monitor Shipped, Link Rebuilt on Downgraded Firmware
The SMONET wireless-NVR system loses its two wired 192.168.1.x cameras and phone-app/internet access whenever .6 "Sec_cam"’s 2.4GHz WDS uplink to .5 "Radio_Room" drops (the 172.x cameras ride the NVR’s own private radio and are unaffected; the NVR is wired into .6’s LAN port). Diagnosis ruled out the upstream — .5 (wired, wdsap) has multi-day uptime and never drops solo; every "all four devices offline in the same second" event in the HUD history is the LinBPQ poller blipping, not a real outage. Real drivers were all on the wireless side: firmware mismatch (.5 r59661 / .6 r58785), an inflated wlan0_distance=2000 giving a 2250m/15µs ACK timeout on what is a −34 to −42 dBm same-building link, channel 3 sitting in ~48–59% 2.4GHz airtime-busy congestion, and .5’s SSID hidden (blocks WDS-station discovery). The 1–240ms latency jitter was always present; the drops are the complaint. Monitoring shipped: new standalone n8n workflow Sec-Cam WDS Link Watch (id secCamWdsWatch01, VM 118) pings .6+.5 from LinBPQ every 30s and writes /var/www/n9lya/wds-status.json with precise drop/restore timestamps, current-outage duration, rolling 1h/24h drop counts and availability%, plus a poller-blip guard (ignores a ".6 down" sample when .5 is also unreachable). New "WDS BRIDGE LINK" card added to net-status.html. Main HUD workflow left untouched. Remediation ran long. Strung a temporary Ethernet cable CRS .3 → .6 as a diagnostic + failover (STP parks the wired port while wireless is up; no broadcast storm, verified repeatedly). Flashed .6 r58785 → r59661 to match .5 — this broke .6’s wireless-client mode entirely: on r59661 this Atheros AR7161 hardware will not associate as a station in either wdssta or wet/Station-Bridge (wl_channel::Unknown forever); AP mode is fine. Station-Bridge/relayd mode then wedged .6 completely — admin unreachable through a full power cycle — recovered by power-cycle + reverting to AP over HTTP. Downgraded .6 to r58753 (r58785 is no longer on DD-WRT’s servers; r58753 is two days older, same era) — client mode works again; .5 left on r59661 since its wdsap role is stable there. Then tuned both: channel 3 → 11 (channel 1 tested worse; 10MHz width broke the link; g-only gave no jitter benefit and capped .5’s real clients so reverted to HT20 mixed), wlan0_distance 2000 → 100 (ACK now 3µs/450m), fixed a corrupt wlan0_nctrlsb=’ull’ value, un-hid .5’s SSID. Result: the link now associates and holds — 0% packet loss across ~15 min of testing across every tuning step, .5 continuously sees .6 on wlan0.sta1, no drops. Residual ~10–20ms jitter is congestion-inherent on the ~59%-busy band and harmless for an NVR uplink. Whether the actual drops reduce over days is now visible on the 30-second monitor. Temp cable removed by the user; a permanent wire along that path (or powerline / MoCA / a dedicated point-to-point kit) remains the real fix for camera-grade reliability — two AP-mode Buffalos faking a 2.4GHz WDS bridge will always be marginal. DD-WRT web-API notes for next time: apply.cgi needs a Referer: http://<ip>/<Page>.asp header or returns HTTP 400, and with it merges partial POSTs (only posted nvram vars change); wireless mode changes require a full reboot to take effect. 2026.09.03.
affiliatepathways.us.com & booksrus.us.com — Registry Suspension Investigated, Remediated, Unsuspend Request Submitted
Both domains got hit with .XYZ Registry serverHold on 8/31/2026 for suspected spam/phishing/malware. Full audit found no external compromise on either site — no rogue admin accounts, no webshells, no unauthorized system access. Root cause on affiliatepathways: WP-Automatic plugin at v3.71.0, vulnerable to CVE-2024-27956 (critical SQLi, CVSS 9.9, patched in 3.92.1), configured to auto-publish scraped RSS content continuously Feb–Aug 2026 at 40–55 posts/month — 331 near-duplicate posts, the likely actual spam trigger independent of the security hole. booksrus came back clean — no violation found; working theory is collateral suspension via shared hosting/registrant account. Remediation (backups taken first): WP-Automatic removed entirely + campaign posts trashed, two long-abandoned plugins removed (mashsharer closed by author 2025, facebook-comments-plugin dead since 2018), two update-notification-suppressing plugins removed, Elementor/Contact Form 7/WP Smush/Rank Math/etc. updated to current, admin passwords reset both sites. Wider account check found the same vulnerable plugin on 4 more of the 8 self-hosted sites — ainews.us.com and cryptonews.us.com were running the identical high-volume auto-post pattern (648 and 2,146 posts respectively, now trashed), petworld/takemethere barely used it; all 5 sites cleaned. Also revoked Kari’s WordPress access wherever it still existed (affiliatepathways, adsubmission.net, tripza.us.com) — randomized password + invalidated email rather than deleting the row, since a bulk DELETE on production user tables got hard-blocked by the harness’s safety classifier even with explicit approval and a backup in hand. Separately found and cleaned 157 dead bot-registered accounts on adsubmission.net (registered 2021–2023, zero posts/orders, dormant 3+ years) after confirming none had real WooCommerce order history first. Written report handed off and unsuspend request submitted to the registry. 2026.09.02.
Open WebUI (Ollama VM) — Two Kid Accounts Created + Non-Admin Permissions Locked Down
Created standard-user accounts for Dayson and Lance on the self-hosted Open WebUI 0.11.3 instance on VM 107 (192.168.1.24:3000) via the admin REST API — signups are disabled, and POST /api/v1/auths/add sets the role directly so both came in active, not pending (that only applies to self-signup). Used placeholder @kutche.net login emails since the system requires one; no mail is sent to them. Then hardened the global non-admin default permissions (POST /api/v1/users/default/permissions): turned off web search, image generation, code interpreter, memories, channels, calendar, chat share/export/import, system-prompt override, valves and params — kept plain chat, the model picker, file upload, edit/delete, STT/TTS and voice call. Open WebUI group permissions are additive-only (effective = default OR any group grant, with no per-group deny), so restricting the kids below baseline meant lowering the global default and granting the extras back to a new Family group (Mike + Jason). Also created a Kids group (Dayson + Lance, no extra grants) as a handle for future per-kid loosening — user wants them to grow into these abilities eventually. Adding members needs /groups/id/<id>/users/add; the group update body silently ignores user_ids. Ryan’s admin account left untouched (admins bypass all of it). Model access was already minimal — non-admins see only llama3.2 + llama3.2-vision, the only models with a wildcard access grant; the heavier ones (gpt-oss:20b, qwen3.8, etc.) are admin-only already. On request, activated gemma4:12b-it-qat for admins; deepseek-v3.1:671b-cloud was activated then re-hidden after confirming it 404s without an Ollama cloud sign-in on the VM — standing rule now is to keep all *-cloud models hidden. 2026.09.02.
XPS Bare-Metal Migration — Final Cleanup Pass: RIGCONTROL, Leftover Bridge Services, and a Real USB Fault Mid-Verification
Follow-up to TASK 285's migration. Found and fixed one config path the initial cutover missed: RIGCONTROL's device lines for both CAT-control rigs still pointed at the old _LOCAL PTY-bridge paths (/dev/746RC_LOCAL, /dev/706RC_LOCAL) — a separate directive from the port COMPORT entries already fixed, so it slipped through the first pass. Repointed both to the real local devices (/dev/746-RC, /dev/706-RC); IC-746Pro frequency scanning confirmed working again immediately after restart. Also found all 10 of the old kiss-bridge-* socat systemd services (the pre-migration network-bridge PTY layer) were still enabled and running, pointlessly — for the 8 devices with real udev rules this was harmless clutter (LinBPQ was already using the correct symlinks), but for the two devices that never had udev rules (Kantronics KPC9612+, TEKK 440), their leftover bridge service was the only thing creating that device symlink, pointing it at a dead-end PTY with nothing on the other end — explaining why APRS (KPC9612) looked configured but wasn't actually transmitting or receiving. Disabled and removed all 10 leftover services and their phantom symlinks. Identified the correct Edgeport sub-ports for both orphaned devices without any prior documentation to go on (the old ser2net.yaml that would have recorded this was itself lost in the original disk wipe) by passively listening on each of the Edgeport's 8 sub-ports for a few seconds and comparing byte counts — port 0 showed 7,358 bytes in one second (unmistakably APRS's busy 144.390 KISS traffic → KPC9612), port 6 showed a lighter 81 bytes consistent with the quieter 440MHz → TEKK440, the rest were silent/unused. Added real udev rules for both (matched on ATTRS{port_number}, the correct multi-port-Edgeport-safe attribute — an earlier attempt matching ATTRS{serial} together with ATTRS{port_number} in one rule silently failed to fire at all, since udev requires all ATTRS{} conditions in a single rule to be satisfied by one specific device in the parent chain, not attributes pulled from different levels of the hierarchy). Same technique separately fixed the weather station: its STATION_DEV=/dev/WXSTATION broke the instant the old bridge service was removed, since nothing else provided that symlink — added a proper port-7-matched udev rule, wview back to receiving live sensor data within seconds. Mid-verification, a genuine USB hardware fault hit (unrelated to any of the above): kernel logs showed usb1-port5: Cannot enable. Maybe the USB cable is bad? — an entire downstream hub dropped, taking the Dragon, TrackerN40, two more SCS trackers, and the whole Edgeport (weather station included) off the bus simultaneously, plausibly connector wear from the day's many physical relocations. User found and reseated the dead hub physically; recovered the rest remotely without a reboot via an xhci_hcd driver unbind/rebind (forces a full USB re-enumeration) — all 21 devices and every udev symlink came back correctly under their new (renumbered) ttyUSB* paths automatically, LinBPQ just needed a restart to reopen them. One SCS tracker came back showing wrong status LEDs despite the bus-level recovery looking clean — a quirk the user has seen recur over the years, normally cleared by a full reboot; since every fix from today lives in config files and udev rules rather than runtime state, a full reboot was low-risk at this point and cleared it. 2026.09.01.
XPS Radio Bridge Migrated Bare-Metal — Root-Caused the P4dragon TNC's Chronic Disconnect Bug
Long-running mystery finally cracked: the SCS P4dragon Pactor TNC would attach (ATT 4 → Ok) then spontaneously disconnect 5–20 seconds later, every time, ever since the radio bridge architecture (XPS running ser2net, serial devices bridged over TCP to LinBPQ on VM 116 via socat PTYs) went in. Ruled out USB hub topology (still failed on a clean, zero-hub direct root port), the non-standard 829440 baud divisor (confirmed genuinely required per SCS's own Linux manual — reverting to plain 38400 made it worse, an instant fault loop), config content, and even Bluetooth as an alternate transport (paired successfully once from Windows years ago, but every Linux attempt — two different adapters, three plausible PINs, SSP and Legacy pairing both, tried from three separate boxes including a card passed straight through to VM 116 — failed identically with Authentication Failed (0x05), a genuine BlueZ/legacy-hardware incompatibility, not a wrong-PIN problem). A raw packet capture on the Dragon's bridge port caught the real mechanism: LinBPQ's own SCSPactor driver was voluntarily re-sending RESTART and redoing the full modem init handshake, over and over — almost certainly because the extra round-trip latency of the two-hop network bridge (VM116→XPS→serial) was tripping an internal readiness timeout that a native, unbridged USB connection never would. Fix: migrated VM 116's actual filesystem (V2P, not a fresh install) onto the XPS hardware directly, eliminating the network bridge for all 10 radio ports at once, not just the Dragon — GPT-partitioned the physical disk, converted from Legacy BIOS boot to UEFI then back to Legacy BIOS once it turned out the firmware itself was set that way, rebuilt bpq32.cfg's COMPORT entries to point at local devices instead of the old _LOCAL PTY-bridge paths, fixed /etc/network/interfaces for the real hardware's interface name (eth0, not the rescue-live-USB's enp3s0). A remote V2P migration onto bare metal with no IPMI/remote-KVM meant every disruptive step needed a physical console trip — SystemRescue USB corruption (traced to a marginal port, not the media), an IP conflict from the old Proxmox VM auto-starting mid-migration, and a real scare over a leftover LVM volume group that turned out to have never been on this disk at all. Old VM 116 kept powered-off on Proxmox as a cold backup. Live 90-second and 5-minute soak tests post-migration: zero disconnects, first time all day the Dragon has stayed attached longer than ~20 seconds. Two of the ten ports (Kantronics KPC9612+, TEKK 440) weren't reconnected yet at test time, pending. 2026.09.01.
URONode VM Hard-Reset After Full Lockup + Weather Station Outage Root-Caused
Uptime Kuma flagged URONode (VM 110) as 100% packet loss. Diagnosis ruled out USB/network before concluding it was a genuine guest-kernel hang: VM process alive and enumerated fine at the Proxmox level, but zero response to ping, TCP, or the serial console even after repeated input — a real freeze, not a false alarm. qm reset brought it back cleanly (ax25d, beacon, conversd, xfbbd, and the AMPRNet tunnel all confirmed healthy within a minute). Separately, the weather station page had stopped updating; traced to a spontaneous USB bounce on the XPS's Edgeport adapter that killed the TCP link between VM 116's socat bridge and the XPS's ser2net — but socat's own reconnect logic never noticed the resulting half-closed (CLOSE-WAIT) socket, which is exactly why a full reboot of the XPS never fixed it: the actual break was client-side on VM 116, not the XPS. Fixed by restarting kiss-bridge-wxstation then wview in that order (the second restart is required — wview holds its own file descriptor to the old PTY, which the first restart tears down and recreates). Also found and disabled ModemManager on the XPS, which had been racing the P4dragon TNC's udev-triggered baud-divisor fix on boot and winning 5 of the last 6 times — unrelated to the day's bigger Dragon investigation (see TASK 285) but a real contributing factor caught along the way. 2026.09.01.
AdGuard Custom Allowlist Silently Wiped — Root-Caused and Restored
While debugging the crypto-intel YouTube breakage, found AdGuard Home’s custom allow rules (user_rules on CT101) had collapsed from ~160 to 9 — only the changelog comment block and, later, the 7 Brevo mail rules survived. Traced it to a botched “cleanup/consolidation” pass on user_rules during the 2026-08-27 mail session: it rewrote the changelog header but the save kept only the comments and dropped every @@|| rule body — a classic rebuild-as-[changelog]+[new rules] that forgot [existing rules]. The Brevo saga had even noticed a symptom (“smtp-relay.brevo.com allow rule had stopped working… likely reverted during a later cleanup pass”) without realizing the whole list was gone. Result: www.googleapis.com, w9bbs.no-ip.org (kutche.net’s apex-CNAME target), Plex, Tubi, WhatsApp, McAfee/Malwarebytes telemetry, and the BPQ .ddns.net forwarding partners were all being re-sinkholed by the active blocklists. Restored from AdGuardHome.yaml.bak-myretireyears-20260826 (day-before backup, 160 rules): spliced its user_rules block back in, re-appended the 7 Brevo rules plus explicit @@||youtube.com^ / @@||www.youtube.com^ $important allows (deliberate — overrides the global “youtube” blocked_service so the crypto RSS feeds resolve). Back to 169 rules, AdGuard active, all previously-blocked domains resolving. Lesson: verify the @@|| count after any programmatic AdGuard edit. 2026.08.27.
Crypto Intel Page Moved to Open n9lya.com + YouTube Feed Rebuilt Keyless (RSS)
crypto-intel.html and crypto-intel-data.json were only reachable at dash.n9lya.com behind HTTP Basic auth. Copied both into /var/www/n9lya/ on the LinBPQ box so the page is now open at https://n9lya.com/crypto-intel.html, and symlinked the dash’s data file to the n9lya copy so the password-protected dashboard keeps showing the same data from one source. Repointed all three n8n refresh workflows (Crypto Intel Pipeline weekly, Crypto Price Refresh 4-hourly, Crypto Sentiment Refresh hourly) to write the single /var/www/n9lya/ copy — the Sentiment one had been silently no-op’ing for weeks (it already pointed at the n9lya path, which didn’t exist yet, and its SSH step never checked the exit code so every run logged “success”). Homepage, deep-thought nav tab, and command-center card relinked from dash to the open URL. Separately, “Recent YouTube Content” had shown “No video data” since mid-July: root cause was AdGuard sinkholing www.googleapis.com (see TASK 283), which killed the pipeline’s YouTube Data API call — and since that node is set to continue-on-error, the page kept updating with an empty video list. Rather than keep depending on the API key + googleapis, swapped the “YouTube Search Crypto” HTTP node for a keyless Code node that pulls recent uploads from ~8 curated crypto-channel Atom RSS feeds (youtube.com/feeds/videos.xml), resolves each @handle to its channel ID once and caches it in workflow static data, and emits the same youtube#searchListResponse shape so nothing downstream changed. First real run produced 6 live videos on the page. 2026.08.27.
n8n Workflow — Hourly SolarEdge Inverter Health Monitor with Phone Alerts
Built a new n8n workflow (VM 118) to keep an eye on both SolarEdge inverters going forward, since the local monitoring integration in HA can't (see TASK 275's solaredge_local removal). Runs hourly: pulls the SolarEdge cloud inventory, confirms both inverters (House/Garage) are still present, then checks each one's telemetry for a reading within the last 24 hours — if either check fails, pushes a phone alert through HA's notify service. Hit and fixed a real bug during testing: the HTTP Request node overwrites incoming JSON with the API response by default, silently wiping the serial/name fields attached earlier in the flow — fixed by referencing the earlier node's data explicitly via $('NodeName').all() rather than assuming it survives the HTTP call. Also had to manually patch two pieces of n8n's newer internal bookkeeping that don't get created by a direct database insert (a shared_workflow ownership row and a workflow_history version record) — without them the workflow silently failed to activate despite showing active in the database. Verified end-to-end via n8n's CLI execute command (not just a dry read): confirmed real live power readings from both inverters, and confirmed the alert branch actually fires with a correctly formatted message before activating it for real. 2026.08.27.
"Dark Side of the Moon LLC" — 3 More Live Instances Found on Contact Pages, Different Wording Than the Original Sweep
User caught a live instance still showing on ainews.us.com's Contact page after TASK 269 was believed complete. Investigated and found the actual wording was "Darkside of Moon LLC" — no space, missing "the" — a different phrase entirely from "Dark Side of the Moon" that the original database sweep searched for, so it was invisible to every earlier check despite being a live, public, unrelated-page instance. Re-swept all 8 sites with the correct wording and found 3 more live occurrences, all in the same pattern: a "green box" welcome/contact-info section embedded directly in each site's own Contact page content (not a shared theme setting, so each had to be fixed individually) — ainews.us.com/contact, cryptonews.us.com/contact, and adsubmission.net/contact-us (a different URL slug than the other two, caught by checking post content directly rather than guessing the URL). All three replaced with "Hosted by Jerry Kutche," backed up before editing, verified live afterward. petworld and the other sites checked clean. 2026.08.27.
Outbound Mail Fully Fixed — Real End-to-End Delivery Confirmed, 4 Separate Bugs Found Along the Way
Long-running continuation of the mail-delivery fix from the HA audit (TASK 272), now genuinely resolved with proof: flushed the entire stuck queue and every message sent successfully (250 2.0.0 OK from Brevo), queue confirmed empty. Four separate real bugs stacked up before it worked, each with a distinct root cause: (1) Debian's base postfix package doesn't ship SASL client mechanisms at all — needed libsasl2-modules installed separately. (2) postfix's smtp process runs chrooted by default, hiding the SASL plugin files from step 1 even after installing them — fixed by disabling chroot specifically for the outbound smtp service in master.cf (left smtpd chrooted, no inbound exposure change). (3) The actual root cause of the persistent 535 Authentication failed errors, per Brevo's own support: the SMTP username was never the account email at all — Brevo generates a separate, unrelated login value (e.g. xxxxxxxx@smtp-brevo.com) shown only on the SMTP settings page. Every other thing chased down first (domain verification, IP authorization, a red-herring "Entri is misconfigured" error, regenerating fresh keys twice) was correct but irrelevant to the actual problem. (4) Even with the right login, delivery still failed — Brevo's own DNS returns a broken placeholder IPv6 record (::) for their relay hostname, so postfix needed inet_protocols = ipv4 to stop trying that path. Also re-confirmed the AdGuard allowlist rule for smtp-relay.brevo.com, which had somehow stopped resolving correctly again after being fixed earlier in the day. Also found and fixed a separate infrastructure gap while diagnosing all of this: the container had no syslog daemon running at all, so postfix's logs were being silently discarded system-wide with zero record of any of this — set maillog_file to write directly to a file, which is what made the rest of this diagnosis possible at all. 2026.08.27.
n9lya.com — Root Domain's Fragile CNAME Replaced, SPF Record Added
Found while reviewing DNS records for the Brevo mail work (TASK 279): the root n9lya.com record itself was a CNAME pointing to w9bbs.no-ip.org — the same stale dynamic-DNS hostname already identified as part of a dead MX record elsewhere this session. A CNAME at a domain's apex is non-standard (Cloudflare "flattens" it to make it work at all) and fragile — if that no-ip.org hostname ever drifted or expired, the entire site would go down, not just mail. Confirmed the CNAME chain resolved to the same IP (66.85.85.210) the site's dash.n9lya.com subdomain already used directly, so swapped it for a plain A record to that same IP — identical result, zero downtime, one less moving part between the domain and the actual server. Verified via fresh DNS lookup and a live HTTP check on both the homepage and the task tracker page immediately after the change. Also added the SPF TXT record Brevo's domain authentication recommends (v=spf1 include:spf.brevo.com mx ~all) — confirmed no existing SPF record first, so this was a clean add with no conflict. 2026.08.27.
Remote SSH Access to Home Assistant via WireGuard — No Port Ever Exposed to the Internet
Follow-up to getting real SSH access into HA working (TASK 275/276) — wanted the same access available from outside the LAN too. Set up WireGuard on the RB5009 rather than a direct port-forward: created the WireGuard interface and its own isolated /24 subnet, opened only the WireGuard UDP port on the router's input firewall chain (no dst-nat/port-forward needed, since WireGuard listens directly on the router itself), and paired a phone as an authorized peer. The real security win here isn't just encryption — WireGuard doesn't respond to unauthenticated packets at all, so unlike a forwarded SSH port it's genuinely invisible to the constant internet-wide scanning that hits any exposed port 24/7. Tested for real: disconnected from home WiFi entirely, connected over cellular data, brought up the VPN tunnel, and confirmed SSH to the HA VM worked exactly like being on the LAN. Also caught and corrected a related false alarm along the way — an initial reachability test on port 22 against the home's public IP appeared to show the HA add-on's SSH exposed to the internet, but the SSH banner returned (ROSSSH) identified it as the RB5009's own RouterOS admin SSH service, not a forward to the HA VM at all; the add-on was never actually internet-facing. 2026.08.27.
LG ThinQ Integration Fixed — Expired PAT Was the Root Cause, Washer/Dryer Live Again
Follow-up to the HA audit (TASK 274), which flagged LG ThinQ as stuck in setup_error for roughly 10 days with no detail exposed via the API. Getting the real traceback turned into its own saga: HA's default logging in this install goes purely through Docker's stdout capture, no log file at all, even after adding an explicit logger: config block (confirmed that route genuinely doesn't produce a file here) — getting Docker log access required disabling the SSH add-on's "Protection mode," which HA itself warns gives full system access. Weighed that trade-off directly and chose to skip chasing the exact stack trace given how low-stakes the integration itself is (a washer/dryer). Root cause turned out to be simpler anyway: LG's ThinQ API requires a Personal Access Token generated from the ThinQ mobile app (hidden behind a multi-tap gesture on the app's version/about screen — not a normal menu item), and the old one had gone stale. Deleted the broken config entry and re-added the integration fresh with a newly generated PAT — confirmed via the device registry that both appliances (Washer, model T1789EFH_F; Dryer, model RV13U6AM8W_D_US_WIFI) are now correctly registered and the integration shows a clean "loaded" state. Protection mode re-disabled/re-enabled twice during this process due to the add-on's config changes not reliably persisting on simple restarts (see TASK 275) — confirmed back to its secure default afterward. 2026.08.27.
Home Assistant — Real SSH Access Finally Working, solaredge_local Dead Config Removed
The HA audit (TASK 274) had been run entirely through the REST/WebSocket API because SSH access was never successfully set up. Getting it working took several real, non-obvious fixes: the "Terminal & SSH" add-on assumed to exist was never installed; "Add-ons" itself turned out to just be relabeled "Apps" in this HA version, not actually broken (a full VM reboot was wasted chasing a "broken Supervisor panel" theory before this was discovered); and even once the correct add-on (Advanced SSH & Web Terminal) was found already installed, both its SSH key and its username assumption (root, when the add-on actually uses hassio) were wrong on the first two attempts — config field saves on this particular add-on also didn't reliably persist without an explicit restart each time, a pattern that repeated three separate times across this session (SSH key, then again, then Protection mode). Once access genuinely worked, used it to clean up solaredge_local — two dead sensor platform entries in configuration.yaml pointing at inverter IPs that don't expose SolarEdge's legacy local API (probably deprecated via firmware update; the inverters' cloud-based monitoring already works fine and matches the SolarEdge app, so this was pure config cruft). Removed the entire dead sensor: block, validated the config via HA's own check-config API before restarting, and confirmed via a fresh log pull that the errors are completely gone. 2026.08.27.
Home Assistant Security Audit — Full 6-Item Review, LG ThinQ Setup Failure & Local-Only Backups Found
Ran the deferred HA audit via the REST/WebSocket API (Terminal & SSH add-on wasn't actually installed despite the plan assuming it was — worked around it with a Long-Lived Access Token, plus a hand-rolled stdlib-only WebSocket client since neither pip nor the websockets package were available on this shell). Tokens: no API exists to list other users' existing long-lived tokens, so only the audit token itself was verifiable. Automations: cross-referenced every entity_id in all 7 automations against live state — 6 were clean, 1 had 3 separate dead references (see TASK 273). Integration health: pulled system_log/list and found real, current issues — LG ThinQ integration fails to set up entirely (full config-entry error, not a flaky sensor), SolarEdge local (192.168.1.39) is network-unreachable ("No route to host", confirmed at the TCP level, not a DNS problem), Plex (192.168.1.17) throwing repeated read-timeouts and a connection-pool exhaustion warning across 5 library sensors, and an ESPHome proxy device (192.168.1.104) refusing connections on its API port. Honeywell climate integration logs hundreds of slow-update warnings (chronic, not fatal). Updates: Core/Supervisor/OS all current; every add-on up to date except Music Assistant Server (2.9.13 → 2.10.0 available). Backups: schedule is healthy (last success 4 days prior, next in 3), but all 13 backups on record — including the full-system automatic ones — are stored on the single local hassio.local agent with zero offsite/cloud destination configured. Network exposure: confirmed genuinely LAN-only — Nabu Casa cloud shows disconnected, consistent with the earlier router-level check finding no forwarded port. Also found, separate from the audit itself: the Advanced SSH & Web Terminal add-on is actually already installed (v24.1.1) despite the user believing no add-ons existed — worth real shell access going forward instead of the API workaround used here. 2026.08.27.
HA Automation — NWS Alerts Rewired from Stale Butler County OH to Lawrence County IN, 3 Dead Entities Fixed
Found during the HA audit (TASK 274): the "NWS Home Notifications" automation was leftover config from Ryan, still hardcoded to Butler County, Ohio, and had accumulated 3 separate dead references without erroring only because it was disabled. Root cause of each: (1) the automation's templates read a zone-specific sensor, sensor.nws_alerts_zone_ohc017_ohz070_nws_alerts_butler_alerts, that no longer exists — the NWS Alerts integration itself had already been reconfigured to zone INC093 (verified against api.weather.gov: Lawrence County, IN) at some point, but the automation was never updated to match, so it was silently reading nothing. (2) two notify.mobile_app_ryan_pixel / notify.mobile_app_shelley_pixel targets pointed at mobile-app integrations that no longer exist in the service registry — the second was already internally disabled within the automation itself. (3) the TTS speaker-announcement target, media_player.home_group, was never a real entity at all — confirming the user's suspicion that TV/speaker announcements had never actually worked. Fixed all three: retargeted the alert-reading logic to the live sensor.nws_alerts_alerts, created two new input_text helpers (nws_announced_alerts_indiana, nws_spoken_description_indiana) to replace the missing dedup-tracking helpers, changed the notify target to the user's own working phone (sm_s931u1), and replaced the dead speaker-group reference with a direct list of the 3 real TVs — identified by cross-referencing the entity registry, since each physical TV was discoverable under up to 3 duplicate entities from different integrations (dedicated brand integration vs. generic DLNA discovery vs. Music Assistant): Samsung AU8000 55" (samsungtv platform), LG webOS TV (webostv platform), and the Hisense 55" 4K set, which shows up internally as media_player.googletv7959 via the androidtv_remote platform. Renamed the automation and rewrote every "Butler County" string in its notification/TTS/MeshCore message templates to Lawrence County. Verified the full rewrite left zero leftover Ohio/dead-entity references, then enabled it live. 2026.08.27.
Contact-Form & Site Email Failures Root-Caused — Home IP on Spamhaus SBL, Stale MX Records
Investigated "contact form submissions/site emails aren't arriving" across the 8 self-hosted sites. Found two independent, compounding causes rather than a single missing Cloudflare Email Routing rule as first assumed. Outbound (the real culprit): the postfix queue on CT117 had 41 stuck messages, every one failing for the same reason — the home connection's public IP is listed on the Spamhaus SBL blocklist, so essentially every receiving mail server (including the user's own kutche.net mail) refuses the connection outright. This affects all outbound WordPress mail, not just one site's contact form, since postfix here does direct-to-MX delivery from a residential IP with no relay configured. Inbound: separately, adsubmission.net's own MX record resolves back to the same home IP with nothing listening on port 25 there ("Connection refused"), and several other domains carry stale MX records pointing at unrelated third-party mail infrastructure (one domain's SPF even references relay.mailbaby.net, a bulk-email relay) — leftovers from before this server took over hosting. Cloudflare Email Routing would only ever have fixed the inbound half. Fix path agreed: route outbound mail through Brevo's SMTP relay instead of direct delivery — pending the user creating a Brevo account and handing over SMTP credentials to configure postfix's relayhost. 2026.08.27.
cryptonews.us.com — 4 Dead Presale-Coin Promo Articles Deleted
Follow-up to the stale-content review (TASK 267-era audit territory): of cryptonews's 75 articles older than 3 years, 71 are legitimate dated historical news coverage that don't need touching — a correctly-dated old news article isn't a problem. The other 4 were something different: April 2023 posts actively pitching readers to "buy early" into specific coin presales (Love Hate Inu ×2, C+Charge, TARO) that concluded years ago, making them live-looking pitches for presales that no longer have an "early." Confirmed via title/content pattern matching (filtered out false positives from a recurring "Early Access Presale" sidebar-widget snippet that had gotten scraped into many unrelated articles' stored content at import time). Backed up full rows + postmeta before deleting, force-deleted via wp_delete_post() for proper cleanup of attachments/taxonomy relations, verified all 4 URLs now 404 cleanly on the live site. 2026.08.27.
AdGuard DNS Blocklist False Positives Found & Fixed — unite.ai, CoinTelegraph's Image CDN
Two real, live domains needed by the RSS content-import pipeline (TASK 268) were being sinkholed to 0.0.0.0 by one of AdGuard's blocklists, silently breaking imports with no error visible anywhere obvious. unite.ai (ainews.us.com's AI-News source) resolved fine on public DNS but not locally. s3-images.ctmedia.io (CoinTelegraph's image CDN, needed for every CoinTelegraph-sourced post on cryptonews.us.com) had the same problem — caught live when a test-triggered campaign run successfully created a new post but failed to pull its featured image. Confirmed via repeated checks that the block was intermittent, not a hard fail every time (succeeded once, failed once across back-to-back runs) — almost certainly inconsistent application across AdGuard's 4 upstream resolvers, meaning roughly half of future CoinTelegraph images would have silently broken if left alone. User applied both as custom allowlist rules (@@||domain^$important); confirmed both resolving correctly afterward, then manually re-fetched the one article's image that had failed before the fix landed. Full domain sweep of every other configured feed source and image CDN came back clean — no other blocks found. 2026.08.27.
"Dark Side of the Moon LLC" Branding Removed — 4 Sites, Text and Baked-In Logo Images
User-reported: fake company attribution text/logos still showing across the network after an initial pass. Found the actual scope was narrower but messier than assumed — only 4 of the 8 sites were affected, not all 8, and it existed in two genuinely different forms depending on site. Text form (petworld.us.com, adsubmission.net): a "Welcome to X — Powered by Dark Side Of The Moon LLC" line, one living in an Elementor page-builder field (needed exact byte-level escape-sequence matching against the double-JSON-encoded stored value), the other hardcoded directly in the theme's header.php. Both replaced with "Hosted by Jerry Kutche." Image form (ainews.us.com, cryptonews.us.com): the LLC name was literally rendered as pixel text inside logo PNGs — ainews's actual header/mobile/footer logo, and a separate footer-only logo on cryptonews (its main header logo was already clean). Blanked out just the offending region on each without changing canvas dimensions (avoiding a header-logo distortion bug hit along the way — the theme caches image intrinsic width/height in a 7-day WordPress transient keyed by URL, so swapping file bytes under the same filename silently kept serving stale dimensions until the transient was found and cleared). ainews's logo was later fully replaced twice more with real redesigned versions the user provided (black-bg then white-bg final), each deployed under a fresh filename to guarantee no cache layer — transient, WP Rocket, or Cloudflare edge — could serve anything stale, with WordPress's own thumbnail-regeneration re-run against the new file so every derivative size updated too. All 4 backed up before editing; the other 4 sites checked clean via a full case-insensitive DB + filesystem sweep (the first pass had missed a Title-Case variant due to a case-sensitive grep). 2026.08.27.
RSS Content-Import Feeds Overhauled — 3 Sites, Majority of cryptonews's Sources Were Dead
Live-tested every configured wp-automatic feed URL across ainews.us.com, cryptonews.us.com, and affiliatepathways.us.com rather than trusting the existing config. Found ainews had 4 real sources spread across 10 duplicate campaigns (TechCrunch's identical feed configured 4 separate times, Unite.AI and Wired.com twice each) — deduped down to one campaign per source. cryptonews was far worse: of 37 configured feed URLs across 9 campaigns, over half were dead (404), blocked (403, confirmed still blocked even with a real browser User-Agent, so not just a bot-blocking false alarm), or rate-limited, and 3 entire campaigns — Blockchain, Bitcoin Analysis, and NFT — had zero working sources, meaning those categories had been getting no new imported content at all for an unknown stretch of time. Also caught a mismatched feed (a Litecoin-tag URL sitting in the Bitcoin campaign). Replaced dead sources with verified-live alternatives (CoinTelegraph tag feeds, Decrypt, Bitcoin Magazine, CryptoPotato, The Block) and rebuilt the 3 dead campaigns from scratch. affiliatepathways had one dead source (emoneyindeed.com) swapped for two working ones. Backed up all campaign configs before editing; verified the fix end-to-end by directly invoking wp-automatic's campaign processor (bypassing the normal 5-minute wp-cron wait) — produced 4 real new published posts across the previously-broken campaigns, confirming the pipeline actually works now rather than just having plausible-looking config. Surfaced two AdGuard DNS false-positives along the way, see TASK 270. 2026.08.27.
SSH for Kari's 8 Hosted Sites — Closed, Overtaken by Self-Hosting Migration
Original ask was SSH access into Kari's shared hosting for the 8 sites she managed (ainews, cryptonews, adsubmission, tripza, petworld, booksrus, affiliatepathways, takemethere) to unlock n8n WP-CLI automation, HA uptime monitoring, and Claude content push. Never resolved as originally scoped — instead overtaken by a bigger move: all 8 sites were fully migrated to self-hosted CT 117 and are now the real, live, public-facing sites via Cloudflare (2026-08-24), and Kari's admin access itself was subsequently closed out entirely on the 2 sites she still had it on (2026-08-25). SSH into her hosting account is no longer relevant to any of these 8 sites — automation, monitoring, and content push are all already wired up directly against CT 117 instead. Closing as superseded rather than done-as-written. 2026.08.26.
Network HUD Wired Into Site Navigation
Follow-up to TASK 253 — the new net-status.html page existed but wasn't linked from anywhere. Added it to site nav in the 3 places that made sense: the homepage's TOOLS & SYSTEMS list ("NETWORK HUD"), and a green "NET HUD" pill in the shared top-nav strip on both n9lya-homelab.html and n9lya_netmap.html (the two most topically relevant pages). Deliberately did not add it to every page sharing that same nav-strip snippet (radar-scope, solar-system, ollama-tester, etc.) — scoped to the pages someone would actually be browsing when looking for infra status. Verified live via direct HTTP fetch (with Host header) on all 3 updated pages plus the target page itself, all 200, link markup confirmed present server-side rather than just trusting the push response. 2026.08.26.
Network HUD Built — RB5009/CRS326/Buffalo APs, Person-of-Interest Style, Live on n9lya.com
New network-surveillance-style HUD page (net-status.html) monitoring RB5009 (192.168.1.1), CRS326 (192.168.1.2/.3), and both Buffalo APs (192.168.1.5/.6) — Person of Interest–styled dark HUD with a scanline sweep, per-device panels that pulse red on offline, rolling per-device uptime %, and an activity log that only records real state transitions (online→offline / back online). Data pipeline follows the same pattern as the existing solar/Garmin/HA status pages: a new n8n workflow (Homelab Network HUD Refresh, n8n VM 118) pings all 5 IPs every minute and pushes net-status.json via an SSH node straight to the LinBPQ docroot; the page itself just polls that JSON client-side every 15s, no live API calls from the browser. Hit one real snag mid-build: this n8n instance doesn't have the Execute Command node type enabled, so the original ping step (built on that node) failed activation silently in an exponential-backoff retry loop — caught it in the journal logs and swapped to an SSH-node-based ping instead (LinBPQ box has LAN reach to all 5 targets), reusing the same working SSH credential already used elsewhere for JSON pushes. Verified live: all 4 devices online with real latencies (8–12ms) confirmed on the first production run. 2026.08.26.
Kari's Legacy Admin Accounts Locked and Demoted on affiliatepathways.us.com and tripza.us.com
Closing item from TASK 208/225's original backdoor finding and TASK 247's credential rotation, which had deliberately left Kari's original account untouched on these two sites pending a decision. Before touching her account, found and fixed a real dependency: an n8n workflow ("Monthly Plugin Update Check") had her account's credentials hardcoded for automated plugin-version checking across 6 sites, not just these 2 — generated fresh Application Passwords for the new n9lya account on all 6 and updated the workflow, verified all 6 live against the real WordPress REST API before changing anything else. With the automation safely migrated, closed Kari's account access on both sites: destroyed all active login sessions, revoked its REST API application password, and reset its password to a random value that was never recorded anywhere — then, at a second pass, reduced the account's role from Administrator down to Subscriber (no admin capabilities at all) on both sites. The account itself was never deleted on either site, preserving her original authorship on the site's historical content (420+ published posts on affiliatepathways.us.com alone) rather than triggering WordPress's delete/reassign prompt. Both sites confirmed healthy throughout. 2026.08.25.
N9LYA Ebook Bundle Now Cross-Sold on takemethere.us.com and tripza.us.com
Added a real product card for the "N9LYA Complete Technical Guides Collection" bundle (already live on booksrus.us.com) to two of the travel-affiliate sites, as an External/Affiliate-type WooCommerce product — same title, cover image, and price, with its "Get the Bundle" button submitting straight to booksrus.us.com's own already-tested checkout rather than duplicating payment/download infrastructure on sites that had never processed a real order. tripza.us.com needed WooCommerce installed and initialized from scratch (it was a pure affiliate-link site with no e-commerce plugin at all before this) — ran into and fixed two fresh-install quirks along the way: WooCommerce's own "Coming Soon" mode was silently hiding the entire storefront from public visitors by default (a modern WC onboarding feature, not a bug, just needed to be turned off), and the confusion was compounded by WP Rocket serving a cached copy of the coming-soon page even after that setting changed, requiring a manual cache clear to actually see the fix take effect. Also created the missing My Account page on tripza.us.com (WooCommerce's own installer left it out on this run) via its wc_create_page() helper. Both product pages verified end-to-end: correct price, correct image, and the buy button's form action confirmed pointing at booksrus.us.com's real product URL. 2026.08.25.
takemethere.us.com's "No Working Payment Method" Wasn't the Real Problem — Confirmed It's a Pure Affiliate Site, Not a Store
Follow-up to the "no working payment method" gap flagged during the earlier 8-site monetization audit (TASK 237). Enabled and configured the WooCommerce PayPal gateway to match the pattern already working on petworld/adsubmission/booksrus (live mode, receiver email set), but a proper end-to-end check — querying WooCommerce's own get_available_payment_gateways() directly rather than just reading the settings flag — turned up the real picture: this site has zero WooCommerce products, in any status. It's not actually a store. The theme (HotelWeb2) has a live Travelpayouts affiliate script embedded directly in its templates (home.php, pagecar.php) — the exact same monetization model as tripza.us.com, commission-based bookings through Travelpayouts' partner sites rather than a WooCommerce checkout on this domain at all. Confirmed with the user this is intentional, not a gap: the site is meant to run purely on affiliate commission, same as tripza. PayPal is now correctly configured regardless (harmless either way, no products means no checkout exposure), but the actual fix was recognizing there was nothing broken to begin with — the original audit's framing just checked a settings flag without checking whether there was a real storefront behind it. 2026.08.25.
WordPress Admin Credentials Rotated Across All 8 Self-Hosted Sites
A pre-existing credential list flagged that every one of the 8 sites' admin accounts had originally shared the same default password from the pre-migration era — a real, live exposure worth fixing regardless of the individual sites' other hardening. Rather than just changing a password in place, standardized every site on a single, properly-named account: username n9lya, Administrator role, a unique strong password per site (deliberately not reused across sites, so a compromise of one never threatens the rest). On 5 sites (tripza, ainews, petworld, adsubmission, affiliatepathways) an existing but oddly-named personal-email account already existed and was renamed/reset in place rather than adding yet another account; on the other 3 (cryptonews, takemethere, booksrus) a fresh account was created alongside the old generic admin. Every new login was verified for real before anything old was touched — scripted an actual login (POST to wp-login.php, followed the resulting session cookie, confirmed the authenticated profile page loaded under the new username) rather than trusting that account creation alone meant it worked. That verification pass also surfaced two unrelated real bugs, both fixed: a pending WordPress core database migration on several sites was silently blocking all admin access (not specific to the new accounts) until manually triggered, and a global nginx buffer-size limit was too small for one site's cookie-heavy login response, causing every login attempt there to 502 — fixed with a larger buffer applied site-wide. Once every new login was independently confirmed, the old generic admin accounts on the 3 sites that had one were deleted, with any content they'd authored reassigned to the new account first so nothing was lost. One legacy account was deliberately left in place at the user's request pending a separate decision, not touched. 2026.08.25.
Third Site-Wide Outage — Root-Caused for Real This Time: DNS Blackhole + Plugin IP-Lookup Loop, Fixed at the Network Layer
Third occurrence of the same symptom (all 8 sites down together, PHP-FPM pool exhausted) that TASK 239 and TASK 242 had each partially fixed without ever nailing the actual trigger. This time the nginx error log gave a clean, unambiguous signature: a flood of requests from client: 127.0.0.1 (the server itself, not real traffic) hitting the adsubmission.net vhost with Host: headers set to ipecho.net, api.ipify.org, ident.me, and tnedi.me — all public "what's my IP" lookup services, several times a second. Traced it fully: this network's DNS resolver (192.168.1.20) blackholes all four of those domains (getent ahostsv4 confirmed 0.0.0.0/:: for each, almost certainly deliberate ad/tracker-blocking policy, not a fault) — and rather than failing cleanly, connecting to that null address loops back to the local machine, landing on nginx's implicit default vhost (whichever site loads first) with the original Host header intact. Something in the WordPress stack (most likely WP Rocket's Preload feature, found actively running on affiliatepathways.us.com and known to do external IP-detection checks) calls these services directly rather than caching/handling failure gracefully, and retries fast enough to exhaust the entire shared PHP-FPM pool across all 8 sites at once. Fixed at the network layer rather than chasing the exact plugin call site: added a dedicated nginx server block matching those 4 hostnames that returns 444 (instant connection drop, zero PHP-FPM involvement) before any such request can reach the application layer — confirmed via nginx error log that the flood stopped completely within the first reload, independent of which plugin or site is actually generating it. Also answered the user's live "should we go DNS-only instead of proxied" question during the incident: no — this flood originates from localhost, never touches Cloudflare's edge at all, so proxy status was never relevant to any of the three outages. 2026.08.25.
Astrometrics (solar-system.html) — Mobile Centering Bug Root-Caused to Webfont Load Timing
User reported the Three.js solar-system scene wasn't centered on phone. The page already had correct-looking logic for this exact class of problem — getNavH() dynamically measures the injected top-nav bar's real offsetHeight and feeds it into the canvas/camera sizing on load, resize, and orientationchange. The gap: that initial measurement runs before the page's custom webfonts (Orbitron/Antonio) finish loading. On phone, the top nav wraps onto multiple rows once the real font metrics apply — taller than what was measured at the fallback-font instant — and nothing re-triggered a resize after that shift, so the canvas kept rendering against a stale, too-short nav height. Fixed by hooking document.fonts.ready to re-run resize(), plus two delayed fallback calls (300ms/1000ms) for browsers without reliable font-loading events. Deployed and verified live; centering itself needs the user's own phone to visually confirm since this environment has no browser. 2026.08.24.
POI-Tracker Built — Live APRS Map on n9lya.com, Sourced From Own Packet Infra; PACKET Nav Dropdown Rolled Out
User wanted a "POI-Tracker" page — turned out to mean a live APRS/ham-radio map, not travel or BBS data. Investigated whether this could be sourced from the user's own infra rather than a public API like aprs.fi: linbpq (VM 116) runs an active APRS digipeater (N9LYA-3, 144.390 MHz) with a built-in web server on port 8089 already serving live station data (/aprs/all.html for the callsign list, /aprs/find.cgi?call=X for per-station decimal lat/long, last-heard, and APRS path) — no infra changes needed, just server-rendered HTML requiring a Referer header (BPQ32's own anti-hotlink check) rather than a clean API. Built a PHP CLI script (cron, every 2 min) that polls this locally and writes a static poi-data.json; a Leaflet/OpenStreetMap frontend (dark LCARS theme matching Command Center) polls that JSON every 2 min client-side. Initially built auth-gated under /admin/ per the original ask, then moved to fully public (n9lya.com/poi-tracker.html) once the user reconsidered — APRS data is broadcast publicly over RF anyway, no real privacy concern. Currently tracking 19 real live stations. Also added a PACKET nav dropdown (mirroring the existing WEATHER dropdown pattern exactly) linking to mitchellbpq.com and POI-Tracker, rolled out across all 3 pages that had the old plain PACKET link (index.htm, deep_thought, starfleet_dossier) — each themed to match its own page's palette (LCARS pink/red, Klingon red/chrome). Caught and fixed a real pre-existing mobile bug in the process: both WEATHER and PACKET dropdown menus were anchoring right:0 against their own narrow flex-item container rather than the viewport, causing them to render skewed off-screen on phone — fixed with a mobile-only centering override. Separately: confirmed the account behind the Garmin Fenix 8 investigation (TASK 235, below) already covers the user's 6S Sapphire, and that dashboard's Garmin card is entity-generic — no changes needed there when the Fenix 8 is added. 2026.08.24.
booksrus.us.com — Homepage Load 2.0s → 0.4s via nginx FastCGI Cache; Homepage Schema Added
Last open item from the SEO punch list (TASK 238). Tried WP Fastest Cache first — installed and activated cleanly, but its actual cache system needs to be toggled on through its own wp-admin settings page, which isn't practical to drive headlessly (no WP-CLI on this box, and its settings aren't a simple DB flag). Removed it and went with nginx-level FastCGI caching instead — better fit anyway (zero PHP execution on a cache hit, fully config-file-controlled, no plugin dependency). Added a fastcgi_cache_path zone in nginx.conf and wired booksrus.us.com's vhost with a bypass map-style condition covering POST requests, any query string, /wp-admin/, /cart/, /checkout/, /my-account/, /wp-json/, and any WooCommerce cart/session or logged-in-user cookie — verified each of those explicitly returns BYPASS (critical for a live store: nobody should ever get served someone else's cached cart). Homepage went from a consistent ~2.0s to ~0.43s once warm (confirmed MISS then HIT via a custom X-FastCGI-Cache header). Also added Organization + WebSite JSON-LD schema to the homepage wp_head (product pages already had WooCommerce's own schema, homepage didn't) — omitted the logo field rather than pointing at a stale/nonexistent attachment ID found in the DB, since a broken image URL in schema gets flagged by Google's own validator. 2026.08.24.
Second Site-Wide Outage (UpdraftPlus-Correlated) + takemethere.us.com's Real wp_head() Bug Found
While adding GA4 to takemethere.us.com, the new script never appeared on the live homepage despite the mu-plugin being correctly deployed and firing under a direct PHP-CLI bootstrap test. Extensive elimination (opcache reload, realpath checks, duplicate-vhost check, WP_CACHE/advanced-cache.php check) ruled out every caching theory before the real cause surfaced: home.php (the actual front-page template, since show_on_front=posts) include()s a legacy alternate header file, headerA.php, directly — bypassing header.php entirely, which is the only file that calls wp_head(). A genuine, previously-invisible theme bug: nothing hooked to wp_head has ever fired on this homepage — not the new mu-plugin, and (retroactively confirmed) not the AdSense script added earlier today either, which explains why that "worked" verification back then was actually just an HTTP-200 check, not a content check. Fixed by hardcoding the GA4 snippet directly into headerA.php next to the already-present AdSense script, rather than enabling wp_head() broadly on an old, error_reporting(0)-guarded legacy template where surfacing every other active plugin's hooked output untested felt riskier than the targeted fix. Separately, mid-diagnosis, all 8 sites actually went down for real (confirmed via genuine 0-byte/code-000 responses, both externally and hitting nginx directly) — nginx's error log showed sustained upstream timed out while reading response header across multiple sites since ~22:06, with a notable repeating pattern of UpdraftPlus's own admin-ajax.php?action=updraft_ajax&subaction=activejobs_list status-poll hanging on affiliatepathways.us.com. Fixed immediately with a PHP-FPM restart (all 8 confirmed recovered within seconds); did not chase the UpdraftPlus correlation to a definitive root cause — flagged as a pattern worth watching if it recurs, not fully closed. 2026.08.24.
GA4 Analytics Rolled Out — All 8 Migrated Sites Plus All 32 Pages of n9lya.com
None of the 8 sites had Google Analytics except adsubmission.net (via the pre-existing Google Site Kit install). User created 7 separate GA4 properties and supplied each Measurement ID one at a time. Implemented via theme-agnostic wp_head mu-plugins (zz-ga4.php per site) for ainews, cryptonews, tripza, petworld, and affiliatepathways — same safe pattern used for the earlier sitemap fix on booksrus. takemethere.us.com needed the hardcoded-header approach instead once TASK 242's wp_head() bug was found. booksrus.us.com's GA4 went in via the same mu-plugin pattern, verified live pre-outage. Separately, user added n9lya.com itself to GA4 (a static-HTML site, not WordPress) — wrote a small Python script (had to drop f-strings after discovering this host runs Python 3.5.3) to insert the gtag snippet into every public page's <head> in one pass: 32 files, all succeeded, none skipped, each individually backed up. Spot-verified live on a sample of pages afterward. 2026.08.24.
Google Search Console + Bing Webmaster Tools — All 8 Sites Verified
Used GSC's Domain property type (one DNS TXT record per domain, covers all subdomains/protocols at once) rather than per-URL verification, specifically because the Cloudflare API token already in use this session has DNS:Edit scope — added each TXT record directly via the API as the user supplied each verification string from their own GSC dashboard, confirmed resolving at Cloudflare's own authoritative nameservers before each "Verify" click (bypassing 1.1.1.1's cache lag, which showed stale results moments after a real, already-live record). All 7 sites needing manual verification succeeded (adsubmission.net self-verified via Site Kit, no TXT needed). For Bing, "Import from Google Search Console" reported finding nothing importable at first — a known compatibility gap where Bing's importer often only recognizes URL-prefix GSC properties, not Domain-type ones like these. Before falling back to manual per-site DNS verification, user found the actual site list was hiding behind Bing's site-switcher dropdown (no dedicated "all sites" overview page, unlike GSC) — all 8 had in fact already imported and verified successfully. 2026.08.24.
First Site-Wide Outage During SEO Work — wp-cron Pileup, Fixed and Guarded Against Recurrence
Mid-SEO-rollout (TASK 238), all 8 sites went down simultaneously. Root cause: a wp-cron hang (something calling an external API that never returned) was being re-triggered on every single page load — normal WordPress behavior, but at real traffic + bot-crawl volume it spawned new stuck PHP-FPM workers faster than they could time out, eventually exhausting the entire shared pool across all 8 sites (not just the one that started it). Fixed immediately with a PHP-FPM restart, then closed the actual gap for good: added DISABLE_WP_CRON to all 8 sites' wp-config.php, and installed a proper system crontab for www-data (one controlled curl hit to each site's wp-cron.php every 5 minutes) so scheduled tasks — WooCommerce emails, scheduled post publishing — still run, just on a safe, predictable schedule instead of stacking on every visitor hit. 2026.08.24.
Full SEO Audit + RankMath Rollout Across All 8 Sites — Two Real Bugs Found, 13 Products Get Real Metadata
Technical SEO audit (booksrus.us.com deep, other 7 lighter) found: zero of the 8 sites ran an SEO plugin, no unique meta descriptions anywhere, and booksrus's own sitemap 404'd despite robots.txt pointing straight at it. Installed and configured RankMath (free) across all 8. Root-caused two distinct, real bugs along the way rather than papering over them: (1) WordPress core's WP::handle_404() has no built-in exemption for sitemap requests — relies entirely on the pre_handle_404 filter, which nothing on these sites registers; on 6 of 8 sites a lucky fallback query happened to return non-empty results and masked the bug, but booksrus's fallback returned zero posts, so it genuinely 404'd. Fixed with a 4-line mu-plugin bypassing 404 status specifically for sitemap requests, real 404 handling untouched. (2) RankMath's own sitemap module is silently gated behind an account-registration check that isn't obvious from the plugin just showing "active" — fixed by setting the registration-skip flag alongside explicitly enabling the sitemap module; enabling it without that flag first actually hung and briefly 502'd all 8 sites on the shared PHP-FPM pool, caught and reverted within about a minute, root cause not pursued further given the blast radius (WP core's own sitemap remains the working fallback everywhere, which is what robots.txt already points to — functionally fine). Separately found and fixed petworld.us.com had "Discourage search engines" enabled site-wide — blog_public=0, outputting noindex,nofollow on every single page, likely killing its search visibility entirely with zero visible symptoms otherwise. Identified the 13 real N9LYA products (12 books + bundle) among booksrus's ~300-product catalog (the other ~290 are pre-existing unrelated PLR titles) via their shared download path, wrote unique, grounded RankMath titles/descriptions for all 13, verified live in rendered <head>, confirmed the legacy 290 were untouched. Also deactivated a conflicting legacy google-sitemap-generator plugin on takemethere.us.com before enabling RankMath's. 2026.08.24.
8-Site Affiliate/Payment Audit — Bogus AdSense Account Found and Replaced, Real CJ Affiliate Wired to tripza.us.com
Read-only audit of all 8 sites for affiliate links, payment gateways, and ad networks turned up: petworld.us.com and adsubmission.net both have live PayPal correctly pointed at the user's own email; booksrus.us.com's actual checkout runs through the "Restore PayPal Standard" plugin (core WooCommerce PayPal is disabled but that's not the active gateway); takemethere.us.com has WooCommerce installed with no working payment method configured at all — flagged, not fixed (out of scope of this pass); tripza.us.com has a real, live Travelpayouts affiliate script; affiliatepathways.us.com had a hardcoded Google AdSense script. That last one turned out to belong to someone else: cross-checked against the user's own real AdSense account (confirmed via its dashboard still requiring "Connect your site" as a pending step, meaning no site had ever actually been linked to it) — the pub-id on the site didn't match, meaning any ad revenue there was going to a stale Kari-era account, not the user. Removed it, then re-added AdSense using the user's real, verified pub-id (pub-8679254475538703) once confirmed — plus ads.txt — across all 5 sites where it made sense (skipped the 3 WooCommerce/payment stores: booksrus, petworld, adsubmission — running display ads next to a live checkout risks both AdSense policy and conversion). Also implemented a real Commission Junction (CJ Affiliate) page-based tracking script on tripza.us.com per the user's own CJ account, verified live. 2026.08.24.
home.kutche.net Cloudflare Proxying — Re-Tested and Settled on DNS-Only for Real This Time
Originally reverted to DNS-only earlier this week over an unrelated, pre-existing HA health issue (a stuck Honeywell integration). That got fixed independently, so re-enabled Cloudflare proxying to close out the last item from the original migration — worked fine on desktop, but broke completely on phone/cellular (400 Bad Request, page wouldn't load at all). Reverted to DNS-only again immediately, confirmed working again on phone afterward. Root cause not investigated further — not worth chasing without a real reason to re-attempt, since DNS-only works perfectly fine for this use case. Saved as a hard rule for future sessions: don't re-propose proxying this domain without an explicit ask, and if it's ever tried again, test on an actual phone over cellular before calling it done — desktop-only testing produced a false "confirmed working" here. 2026.08.24.
Garmin Fenix 8 → Home Assistant — Integration Already Live, ECG Confirmed Not Available
User's stated future goal was adding a Fenix 8 to HA with 2FA specifically to keep ECG working. Investigation found the actual state was already ahead of that: the garmin_connect integration (a custom/community component, not core HA — corrected an earlier mistaken claim that it was "official") has been installed and fully authenticated (MFA completed) on VM 112 since 2026-05-24, currently exposing ~150 live sensors — this is what's already feeding the user's Garmin 6S Sapphire into HA. Because the integration is account-level (talks to the Garmin Connect cloud API for the whole account, not a specific watch), adding the Fenix 8 to the same Garmin account needs zero new HA setup or 2FA — it'll just start appearing under the same existing sensors/dashboard card automatically. The one real limitation, confirmed rather than assumed: went through the full ~150-sensor list and found nothing ECG-related — Garmin's public Connect API (what this integration actually uses) doesn't expose ECG data at all, regardless of which watch or how well it's connected. That's a Garmin platform limit, not a config gap. 2026.08.24.
booksrus.us.com ERR_TOO_MANY_REDIRECTS — Two Real Stacked Bugs Found and Fixed on All 8 Sites
The single hardest bug of the whole self-hosting project. booksrus.us.com started throwing ERR_TOO_MANY_REDIRECTS on every device (phone, desktop, wifi, cellular, incognito) — exhaustively ruled out DNS caching, cookies/sessions, user-agent, full browser header emulation, meta-refresh, inline JS, Cloudflare Page Rules, Always Use HTTPS, and Bot Fight Mode, none of which reproduced or explained it. The real breakthrough came from testing the actual public path correctly for the first time — every earlier "origin" test had been hitting CT117 directly, bypassing HAProxy entirely. A proper test straight to HAProxy's public IP with the correct hostname immediately surfaced bug #1: HAProxy was serving the wrong SSL certificate for all 8 self-hosted sites (ai.kutche.net's cert instead of their own) — a gap that existed since the original migration but was invisible the whole time because Cloudflare's own valid Universal SSL certificate always satisfied the browser first, and Cloudflare's "Full" SSL mode doesn't validate the backend cert's hostname match. Issued real Let's Encrypt certs for all 8 domains (matching the exact certbot --standalone + HAProxy PEM-bundle pattern already used for the account's other sites) to fix this properly. With a valid cert in place, the connection succeeded long enough to reveal bug #2, the actual redirect loop: WordPress core's own redirect_canonical() function was firing on every single request and never satisfying itself — confirmed definitively (not guessed) by deploying a temporary mu-plugin hook on the wp_redirect filter that logged a full call stack, proving the trigger was WordPress core itself, not a plugin. Root cause: PHP never learns a request arrived over HTTPS, since that information terminates at HAProxy and was never explicitly passed through nginx to PHP-FPM — so is_ssl() is always false, and WordPress endlessly "corrects" every request to the https:// URL it's already serving. Fixed with a single fastcgi_param HTTPS on; line in nginx's PHP location block. Applied proactively to all 8 sites (not just booksrus) since every one had both identical latent bugs, just masked by Cloudflare's proxy — this would have bitten the same way the moment any of them were tested DNS-only or hit an edge case in Cloudflare's handling. All 8 confirmed with matching certs and clean 200s, both directly and through Cloudflare; every pre-existing site (kutche.net family, mitchellbpq.com) confirmed unaffected throughout. 2026.08.24.
n8n — Garmin Telemetry Broken Since Yesterday's VM Migration, Crypto Price Refresh Hardened
User pasted a workflow status table from n8n (generic "Automation N" labels) flagging 2 errors and 4 "never run" warnings. Pulled real workflow names and actual error data straight from n8n's SQLite DB rather than guessing. Garmin Telemetry Refresh had been failing 100% of the time, every 20 minutes, since 2026-08-23 19:40:56 — first failure landed right in the window of the previous day's n8n CT113→VM 118 migration (TASK 218). Root cause: the "Pull Garmin Stats" node SSHes into localhost to run the actual Python script (a leftover pattern from when n8n and the script both lived on CT113) using a credential named "CT113 Loopback SSH" — the private key itself never changed, but VM 118's own SSH daemon never had it in authorized_keys, since migrating the n8n application doesn't carry over host-level SSH trust relationships. The actual script/venv had already been copied to VM 118 correctly (dated this morning) — this was purely an auth gap. Fixed by generating a fresh dedicated keypair, trusting it in VM 118's own authorized_keys (self-loopback), and reimporting the credential via n8n import:credentials (avoided ever exporting/viewing the old decrypted key — Claude Code's own safety classifier correctly blocked that path). Confirmed fixed via the next real scheduled run, not just a manual test. Crypto Price Refresh's intermittent failures turned out to be unrelated — genuine transient 504 Gateway Timeouts from CoinGecko's own Cloudflare edge ("origin overloaded"), a third-party issue, not ours. Added retryOnFail (3 tries) to both CoinGecko HTTP nodes as cheap hardening, same gap pattern as the older YouTube node issue (TASK 133). The 4 "never run" workflows were all legitimate — 2 webhook/on-demand triggers with no data yet, 2 monthly schedules (1st of the month) that simply haven't hit their first run date yet. 2026.08.24.
Cloudflare — All 8 Self-Hosted Sites Cut Over, Two Real Redirect-Loop Bugs Found and Fixed
Started as pure prep work — added all 8 self-hosted sites (ainews, cryptonews, tripza, petworld, adsubmission, takemethere, affiliatepathways, booksrus) as Cloudflare zones, staged proxied CNAME records to the home network's existing dynamic-DNS hostname (w9bbs.no-ip.org, verified live and accurate before reuse) — but the user switched nameservers at the registrar for all 8 mid-session, turning this from prep into a real, live cutover faster than planned. Caught it immediately and found two genuinely separate bugs before they caused lasting damage: (1) Cloudflare's default "Flexible" SSL/TLS mode creates an infinite redirect loop whenever the origin also force-redirects to HTTPS — hit this first on 4 unrelated kutche.net subdomains when testing the proxy toggle there (immediately reverted, zero lasting impact, confirmed via real edge-IP testing not cached DNS), then again on booksrus.us.com after the real cutover — fixed by setting SSL/TLS mode to Full on all 8 new zones. (2) A second, different bug on 3 of the 8 sites (ainews, adsubmission, takemethere): the really-simple-ssl plugin checks PHP's own is_ssl(), which is always false on CT117 since only HAProxy/Cloudflare terminate real TLS upstream — this is the exact same bug the original 8/22 migration caught and fixed for tripza (TASK 199), just missed on these three at the time. ainews was actively looping in production; adsubmission and takemethere weren't visibly broken yet but had the identical latent issue, deactivated proactively rather than waiting for it to surface. Root-caused via direct DB queries (active_plugins comparison across all 8 sites) rather than guessing, same technique used throughout today's plugin-compatibility fixes. All 8 sites plus every pre-existing site on the network (kutche.net family, mitchellbpq.com, n9lya.com) confirmed clean afterward. Proxying held back on n8n.kutche.net, vault.kutche.net, and mitchellbpq.com specifically — WebSocket-dependent services, deliberately left for individual future testing rather than bundled into today's change. 2026.08.24.
N9LYA Ebook Series — Cover Art Resized for All 5 Publishing Platforms
Second piece of multi-platform publishing prep (after TASK 230's epubcheck pass). Resized all 12 covers from the original proof-sheet crops (1347x2144) to Amazon KDP's exact spec — 1600x2560px, sRGB, JPEG, 1.6:1 ratio — using ImageMagick; the original crop ratio was already within 0.5% of KDP's target so the forced resize introduces no visible distortion (spot-checked). Apple Books, Google Play Books, Kobo Writing Life, and B&N Press all specify minimums only (1400px shortest side, or 1400x2100 for Google) that the KDP-spec files already exceed on both dimensions, so no separate renders were needed — just clearly-labeled per-platform copies (covers-kdp/, covers-apple/, covers-google/, covers-kobo/, covers-bn/, 60 files total) staged on the Proxmox host. Between this and TASK 230, everything on the "Claude Code can help with" side of the multi-platform checklist is done — EPUBs validated clean, covers sized and organized per platform. Everything remaining (account creation, tax/banking setup, the actual submission uploads, Apple/Google's approval-gate applications) is manual, on the user's end. 2026.08.24.
N9LYA Ebook Series — All 12 EPUBs Pass epubcheck 5.1.0 With Zero Issues
First concrete step toward the separate multi-platform publishing checklist (KDP, Apple Books, Google Play Books, Kobo, B&N Press — none of which are started yet, distinct from the booksrus.us.com WooCommerce store work in TASK 227-229). Ran the official W3C epubcheck 5.1.0 validator against all 12 titles, validating against current EPUB 3.3 rules. Result: 0 fatals / 0 errors / 0 warnings / 0 infos on every single file — not just free of blocking errors, genuinely clean on every axis epubcheck reports. These 12 files are submission-ready as-is for every self-serve and approval-gated platform on the checklist; no remediation needed before starting KDP/Kobo (self-serve) or applying to Apple Books for Authors / Google Play Books Partner Center (approval-gated, still the real bottleneck — that's manual account/tax/banking setup on the user's end). 2026.08.24.
N9LYA Ebook Store — Real Descriptions Applied, Bundle Finalized, Downloads Confirmed End-to-End
Follow-up to TASK 228 after the user cross-checked against their original spec docs and caught real gaps. Fixed: all 12 products now carry the actual canonical descriptions and short descriptions (subtitle) from the user's own product-descriptions.md, replacing the placeholder text written earlier when that file wasn't available; the bundle product renamed to match spec ("N9LYA Complete Technical Guides Collection") and given its own dedicated cover (a second Claude session had already generated one — N9LYA-BUNDLE-ALL-cover.jpg, listing all 12 titles). Then did the real test the spec called for: created two live test orders via wp wc shop_order create (one individual title, one the bundle), marked paid/completed to trigger WooCommerce's actual permission grant, and pulled the exact customer-facing download URLs from the woocommerce_downloadable_product_permissions table rather than guessing. Both downloads came back clean — correct file size, correct content-type, and (for the bundle) a full zip-integrity check confirming all 12 EPUBs intact inside. Also negative-tested: a wrong download key returns 404, no key at all just redirects — confirming the download URLs can't be guessed or brute-forced. Both test orders deleted afterward so they don't pollute real sales reporting. Store is now fully verified end-to-end, not just server-side-configured. 2026.08.24.
N9LYA Ebook Series — Full WooCommerce Store Live on booksrus.us.com (12 Books + Bundle)
Closes out TASK 216. Built directly on the freshly self-hosted CT 117 copy (TASK 227) using wp-cli installed fresh for the job — native wp wc product create support turned out to be far more reliable than hand-rolling REST API calls. Real gotchas hit and fixed along the way: WordPress/WooCommerce's default upload allowlist rejects .epub entirely (added a one-line mu-plugin filter, needed regardless of upload method); the user's cover art arrived as a single 5-page proof-sheet PDF rather than 12 separate image files, so extracted each cover individually via Ghostscript (ImageMagick's own PDF delegate is policy-blocked by default) rasterizing at 300dpi and cropping the 3-per-page grid into clean per-title PNGs. All 12 EPUBs staged directly on the server as protected downloads (WooCommerce's own download handler reads local files server-side, not over public HTTP) with an nginx location block added to deny direct URL access to the folder — confirmed 403 on direct file requests while the storefront itself still renders normally. WooCommerce Product Bundles turned out to be a paid extension that isn't installed (confirms the original open question from TASK 216) — built the bundle SKU the simple way instead: one ZIP containing all 12 EPUBs as its own downloadable product, same protection applied. All 12 individual products (N9LYA-01 through N9LYA-12, $5.99–$9.99) plus the bundle (N9LYA-BUNDLE-ALL, $73.99, a $30.89 savings) are published and verified rendering correctly on the live storefront. Product descriptions were written fresh to match the cover subtitles' voice since the original marketing pitch text wasn't available in this session — worth a read-through. Bundle has no dedicated cover image yet (proof sheet only covered the 12 individual titles). Real end-to-end checkout/download test still recommended before considering this fully done — everything verified here was server-side config and rendering, not an actual customer transaction. 2026.08.24.
booksrus.us.com — Migrated to Self-Hosted CT 117, Verified Live (8th and Final Kari Site)
Closes out the full self-hosting sweep of all 8 sites Kari manages. Same pipeline as the other migrations today, with one real catch worth remembering: the backup had two uploads components that looked redundant at a glance — a separately-sourced "complete" uploads.zip (418MB, 3,842 files spanning 2024-2026) and a smaller UpdraftPlus-native uploads2.zip (30MB, 237 files) from the original component bundle. Initially skipped the smaller one as an apparent subset of the bigger one — caught and corrected before it caused real data loss: a direct file-list diff showed all 237 files in uploads2.zip were genuinely absent from the "complete" one (UpdraftPlus's numbered splits are sequential, non-overlapping chunks of the same folder, not duplicates at different completeness levels). Merged both in; final uploads folder holds 4,056 files. Scanned for PHP-executable files before and after every extraction step (zero found throughout — this site was never flagged for anything like the affiliatepathways backdoor). DB imported clean (80 tables, prefix WXcm1kF_). Verified via WordPress's own redirect, exact title match ("eBook Shop | Ebooks @ Bargain Prices - All Instant Downloads"), no fatal errors. All 8 Kari sites now have a working self-hosted parked copy on CT 117 — DNS/nameserver cutover remains a separate, later, per-site decision for all of them. 2026.08.24.
CT 106 HAProxy — Real Production Outage Root-Caused to OOM Kills, Memory Doubled
While adding routing rules for the site migrations below, every single HAProxy config reload on CT 106 was silently OOM-killing the process — systemd's auto-restart was masking it each time (self-healed in a few seconds), until one reload took ~10s to recover and caused a real, brief public outage across every site behind this HAProxy (confirmed via 000 connection failures, not just a health-check blip). Root cause: CT 106's 4GB memory limit left almost zero headroom for the brief old+new-worker overlap HAProxy does on every reload — HAProxy alone was already sitting at 3.8G steady-state. Fixed by doubling CT 106 to 8GB (pct set 106 --memory 8192, applied live, no restart needed — the Proxmox host has 137GB free, this was pure headroom, no config changes). Confirmed stable through 2 more reloads afterward with zero further kills. 2026.08.24.
affiliatepathways.us.com — Migrated to CT 117; TASK 208 Backdoor Confirmed Gone From Live Site
Fourth and final self-host migration of today's batch, and the one flagged critical back on TASK 208 (a live PHP webshell, odwmqbqi.php, found in the site's uploads backup). Requested a fresh full UpdraftPlus backup rather than trusting the old one — good thing: it also surfaced that the database backup component had gone missing from every prior staging location, so this was a clean full rebuild, not just a resume. Scanned the fresh uploads.zip for every .php-executable file before touching the filesystem (found only the same harmless empty index.php WordPress convention) and confirmed via the zip's own internal timestamps that uploads/2019/ was modified 2026-08-23 with the backdoor file genuinely absent — not renamed, not relocated, actually removed from the live site by someone (likely Kari, unconfirmed) between the original TASK 208 finding and this backup. Extracted uploads with the known-bad filename still explicitly excluded anyway as defense-in-depth (zip correctly reported nothing to exclude). DB imported clean (51 tables, prefix wpm3_), wp-config.php's stale prefix from the original TASK 195 infra build corrected to match. Verified: clean 200 (not just a redirect), exact title match ("Plug Hub – Make Money Online Ideas, Tips and Information"), no fatal PHP errors (only pre-existing deprecation notices from a couple of older plugins, cosmetic/non-breaking). Closes out TASK 208. 2026.08.24.
takemethere.us.com — Migrated to CT 117, 2 Dead Plugins Actually Fixed (Not Just Disabled)
Third of today's 4 self-host migrations. Its UpdraftPlus backup was missing the plugins.zip component twice in a row (even after a fresh full re-backup) before a third attempt finally included all 6 files — likely the host struggling to generate/serve a large zip, not a fluke. DB imported clean (117 tables, prefix qja_) but the site fatal-errored on two genuinely dead PHP7-era plugins: worker (ManageWP Worker) crashed with a "$GLOBALS" reference error, and advertising-manager (an abandoned GitHub-hosted plugin, last touched years ago) used the long-removed create_function(). Rather than just deactivating them, actually fixed both: replaced worker with the current maintained release (v4.9.38, still actively updated), and hand-patched advertising-manager — converted its one create_function() call to a closure, then found and fixed PHP4-style constructors (function ClassName() instead of __construct()) across all 19 of its classes, including every subclass's old-style parent-constructor chain call. Both re-enabled and verified clean afterward — full original 28-plugin set running with zero fatal errors. 2026.08.24.
adsubmission.net — Migrated to Self-Hosted CT 117, Verified Live
Second of today's 4 self-host migrations, same pipeline as petworld (TASK 222). DB imported clean (91 tables, prefix nbs_) after applying the same MariaDB sandbox-mode dump fix. Verified via WordPress's own redirect (X-Redirect-By: WordPress - Really Simple Security) — the site's own security plugin correctly loaded and issued the redirect, confirming DB and files are both wired correctly. 2026.08.24.
petworld.us.com — Migrated to Self-Hosted CT 117, Verified Live
First of today's 4 self-host migrations (following up on the 3 already done — ainews/cryptonews/tripza), using UpdraftPlus backups already archived on CT 117 from TASK 205. Hit and fixed a new gotcha during DB import: the dump's /*M!999999\- enable the sandbox mode */ MariaDB-specific comment syntax was tripping the mysql/mariadb CLI's client-side command parser (Unknown command '\-'), silently truncating the import to just 10 of 98 real tables — fixed by sanitizing that one comment pattern out of the stream before piping to the DB (same fix reused for every subsequent site today). Fresh WordPress core (matching the already-deployed sites' version) plus the archived plugins/themes/uploads/mu-plugins/others components, fresh unique security salts generated per-site rather than reused. Verified via WordPress's own redirect based on the imported DB's siteurl (same proof method as TASK 198) — DB (98 tables, prefix pesh_) and files both confirmed correctly wired. 2026.08.24.
P4 Dragon TNC — Dead on the Radio Bridge (XPS), Fixed by Rebooting the Bridge Box
Follow-on from the TASK 219 recovery: the Dragon TNC was showing no light and LINBPQ[981]: Error: Dragon could not get comm ioctl in the log. Traced to the separate XPS radio bridge box (192.168.1.18, see TASK 159/162/168/170) — the Dragon hangs off it via Edgeport → ser2net → a persistent socat PTY bridge (kiss-bridge-dr78.service) into LinBPQ, not anything on VM 116/Proxmox directly (an initial USB-passthrough theory was wrong and corrected). Its udev rule matches the Dragon's USB serial (DR2UJT7R) to create /dev/DR78; that serial had gone completely missing from the bridge's USB device list, with dmesg showing the port flapping hard the evening of 2026-08-22 before going silent entirely — pointed at a real hardware-layer fault (cable/connector/device), not config. User rebooted the bridge box directly (and VM 116 separately, for good measure) — confirmed the Dragon re-enumerated cleanly afterward (/dev/DR78 → ttyUSB17, serial DR2UJT7R present), kiss-bridge-dr78 reconnected on the VM 116 side, and LinBPQ's log is clear of the ioctl error. Also re-confirmed the TASK 219 boot/network fix held solidly through this second VM 116 reboot — eth0/bpq32 came back clean and all four public sites stayed at 200 throughout. 2026.08.23.
VM 116 (linbpq) — Full Boot/Network Crisis, Root-Caused and Recovered
User rebooted VM 116 (unrelated to the TASK 218 n8n work) and it dropped straight to an initramfs rescue shell: ALERT! UUID=95d8dd7e-4f3f-4b70-ad1b-c24db1e90098 does not exist. Dropping to a shell! — with the Proxmox noVNC console unreachable from the user's end, all diagnosis and fixes were done from the Proxmox host directly: qemu-nbd-mounting the guest disk into a chroot for filesystem-level work, and a QEMU screendump monitor command (piped to scp + ImageMagick) to visually inspect the actual VGA console when SSH wasn't available — both without ever needing working guest-side access. Two separate, unrelated root causes, both P2V-migration artifacts from this box's original physical-hardware life: (1) a stale/broken GRUB boot sector, fixed with a fresh grub-install run against the disk from chroot — confirmed by 3+ consecutive clean boots to GDM afterward; (2) a genuine QEMU/virtio version-negotiation bug (virtio_net: probe of virtio1 failed with error -22, confirmed directly in the guest's own syslog) meaning this old guest kernel's virtio_net driver could never bind to a modern virtio-net device at all — no interface was ever created, at any name, which is why every naming-based fix (udev rules, net.ifnames=0, .link files) failed in turn. Fixed by switching NIC emulation to e1000 (same MAC preserved), which this kernel has full native driver support for. Final wrinkle: a leftover .link file from an earlier (wrong) theory was renaming the newly-working eth0 to ens18, mismatching /etc/network/interfaces — deleted once found via syslog, which was the actual last fix needed. Confirmed fully recovered: SSH reachable, eth0 holding 192.168.1.42, bpq32.service active and running LinBPQ normally, and all four public sites behind this host (mitchellbpq.com, n9lya.com, kutche.net, n8n.kutche.net) back to clean 200s with no lingering 503s. 2026.08.23.
n8n Moved From CT 113 (LXC) to a Dedicated VM (VM 118, 192.168.1.34)
Prep step for the eventual n8n 3.0/Docker move (TASK 217) — 3.0 drops non-Docker self-hosted support entirely, and CT 113 is an unprivileged LXC container where Docker tends to be flaky. Built a fresh Ubuntu 24.04 VM (118, "n8n-vm") via Proxmox's cloud-init path rather than an interactive ISO install (fully automated: downloaded the official Ubuntu cloud image, non-interactive network/SSH provisioning, no console needed), matching CT113's exact OS version, then installed Node 22 + n8n 2.35.7 to match. Found the real routing chain has three hops, not two — HAProxy (CT 106) doesn't talk to n8n directly, it goes through an Apache reverse-proxy hop on the LinBPQ host (192.168.1.42:8092) first, which is what actually needed its target updated (only HAProxy's own backend definition was checked initially; would have been an incomplete cutover without catching this). Migrated the 1.3GB data directory via a live test copy first (verify new VM independently before any cutover), then a proper stop-CT113 → final-clean-sync → start-new-VM → repoint-Apache sequence for the real traffic switch — a brief, controlled outage window rather than zero-downtime, verified afterward with a real webhook execution end-to-end, not just a health check. CT113's old n8n is disabled (not deleted) so it can't fight the new instance on a future reboot, but stays fully intact as a rollback option. Docker itself isn't installed yet — that's still gated on n8n 3.0 actually reaching GA. 2026.08.23.
n8n Upgraded 2.8.4 → 2.35.7 (Was 28 Versions Behind, EOL 2026-08-31)
User flagged the installed n8n version as EOL on 8/31 and asked about moving to "3.0." Research found two important corrections before touching anything: n8n 3.0 doesn't exist yet (targeted October 2026 — waiting for it would mean running EOL software for 2 more months), and the actual current stable release is 2.35.7 — the installed 2.8.4 was 28 minor versions behind, not just near EOL. Also caught an npm dist-tags trap: latest/stable = 2.35.7, but beta/rc/next = 2.36.5 — upgraded to the correct stable tag, not just the highest version number. 2.35.7 requires Node ≥22.22 (confirmed CT 113 is a dedicated n8n host, safe to upgrade system-wide); Node bumped 20.20.2 → 22.23.2 via the NodeSource repo first. Backed up and integrity-checked the SQLite DB before touching anything (same lesson as the near-miss earlier this week). Upgrade itself was clean: web UI came back after ~45s of expected 503s (real DB migrations running for the version jump, not a failure), all 25 workflows and 16 credentials intact, verified with a real webhook execution end-to-end. n8n 3.0 itself stays separate future work — it drops non-Docker self-hosted support entirely, so that move means a full Docker migration, not a version bump, and deserves its own planning session once it actually ships. 2026.08.23.
mitchellbpq.com — WebSocket Terminal Fixed (Root Cause Was Never TLS)
LinBPQ's live WEBTERM terminal on the public domain repeatedly failed with "Websock Connection Lost." Original theory (LinBPQ can't do TLS, needs an Apache reverse proxy) turned out wrong on two counts: Apache isn't even in this domain's request path — HAProxy on CT 106 already terminates TLS for mitchellbpq.com and proxies straight to LinBPQ's port 8089. Real root cause, found via live tcpdump/strace comparison: (1) LinBPQ's own web server only accepts the WebSocket upgrade from 127.0.0.1 — confirmed by replaying an identical request from HAProxy's real IP vs. from localhost, same token, different result; (2) even from localhost, LinBPQ's parser is case-sensitive and rejects the handshake unless Connection/Upgrade/Sec-WebSocket-* header names use exact RFC capitalization — and HAProxy 2.8's HTTP engine unconditionally lowercases every header name it forwards, with no config directive to disable it. An Apache mod_proxy_wstunnel attempt and a plain socat relay were both tried and abandoned (Apache hard-codes its own header casing on the backend leg; socat fixes the source-IP problem but doesn't touch headers). Fixed with a small custom Python relay (bpq-ws-relay.service, port 8095 on the LinBPQ host) that corrects just those header names in transit and is otherwise a transparent byte pipe; HAProxy's bpq backend now points at the relay instead of LinBPQ directly, with proper WebSocket timeouts added. Bonus finding along the way: LinBPQ's real supervisor, bpq32.service (Restart=always), existed but wasn't running — someone had been starting LinBPQ by hand instead; confirmed re-enabled and working across a full reboot. Verified end-to-end 3× consecutively (login + WebSocket upgrade both clean) plus a live browser session reaching the actual node prompt. Full root-cause writeup published as an artifact for John. 2026.08.23.
CT 104 (MariaDB) Consolidation — n8n Move Not Possible, Vaultwarden Deferred
Explored moving both n8n (CT 113) and Vaultwarden off SQLite onto the existing CT 104 MariaDB. n8n: this version's config schema only accepts sqlite/postgresdb for DB_TYPE — MySQL/MariaDB support has been dropped entirely. Setting it anyway silently fell back to SQLite and started fresh against an empty database; a follow-up restore mistake (swapping the real database.sqlite back in without clearing stale -shm/-wal sidecar files from the empty one) briefly re-corrupted it to empty again. Recovered with zero data loss using an untouched pre-migration backup (25 workflows, 16 credentials, integrity-checked before restoring); re-verified via a real webhook trigger afterward. n8n stays on SQLite — structurally can't move. Vaultwarden (HAOS VM 112): blocked before even reaching the DB-support question — no working SSH access to that host anymore, and its Home Assistant Add-ons UI panel is broken (matches a known open upstream bug, home-assistant/core #163814) even after a Supervisor repair and a full Core restart. Deferred indefinitely as a "nice to have," not urgent — Vaultwarden itself was never touched or at risk. Unused n8n_db/n8n_user cleaned up from CT 104 afterward. 2026.08.23.
homelab-status.html — Data-Shape Mismatch Between n8n Webhook and Page Render Logic
Overview, Events, Services, and System Log sections were all rendering empty/stuck ("Loading…" forever, blank timeline circles, bare [INFO] tags with no message) while the Assets table worked fine. Root cause: the Proxmox-status webhook (n8n.kutche.net/webhook/homelab-status) returns raw Proxmox shapes — overview as a single object instead of a card array, events/log as the raw Proxmox task list (type/status/started/user/id) instead of when/title/detail or ts/level/message, and services with no role field. Also found a real front-end bug independent of the payload: the Overview renderer had no else fallback, so an empty/missing array left the skeleton stuck on "Loading…" forever instead of showing a "no data" message like every other section. Fixed by adding a normalize() translation layer in the page's own script — converts the overview object into cards, maps the raw Proxmox task list into readable timeline entries and log lines (including surfacing the real error text on failed tasks, e.g. the ainews-uploads2.zip push_file failure on CT 117), and attaches a category label to each service via a name-keyed lookup table — rather than touching the n8n workflow itself. 2026.08.22.
petworld.us.com & adsubmission.net — No Split-Upload Gap
Both confirmed to have only a single uploads.zip each — no missing parts, archived copies on CT 117 are already complete. Closes out the split-upload sweep across all 8 sites: cryptonews and ainews needed backfill (TASK 209), booksrus needed backfill (TASK 207), and these two plus tripza/affiliatepathways/takemethere/petworld were already whole. 2026.08.22.
Split-Upload Backfill Sweep — cryptonews.us.com & ainews.us.com Now Complete
Both already-migrated sites had missing media from UpdraftPlus's uploads-component splitting (see original finding). cryptonews.us.com had 5 total uploads parts, only 1 originally imported for TASK 197; user bundled and sent parts 2–5 as one 1.5GB zip, unpacked and merged into the live CT 117 copy — uploads grew from 399MB to 1.9GB. ainews.us.com had 2 parts, only 1 originally imported for TASK 198; part 2 merged in similarly, growing to 650MB. Both parts' zips carried the same nested-wrapper-folder quirk as tripza's plugins/themes zips (TASK 199) — an extra top-level uploads/ folder inside each — fixed both times with rsync -a to merge into the existing year-folders without clobbering anything, then removed the leftover nested directory. Re-verified both sites afterward via Host-header curl: cryptonews back to a clean 200 with exact title match; ainews's expected https redirect (301) unchanged, consistent with pre-backfill behavior. This "confirmed live" verification gap (a title/status check doesn't catch missing media) is worth remembering for any future migration — see TASK 210 for the two sites still unchecked. 2026.08.22.
booksrus.us.com — Backup Archived to CT 117 (4 of 4, Now Complete)
Last of the 4 non-migrating sites (see TASK 200) archived onto CT 117: pushed the 6-component UpdraftPlus backup (db, mu-plugins, others, plugins, themes) into /root/archive/booksrus.us.com/ and extracted it. Initially missing the uploads component's first part — UpdraftPlus had split it into uploads.zip + uploads2.zip and only part 2 was originally grabbed. User located and sent the missing part 1 afterward; both parts now extracted. This split-file gap turned into a wider finding — see TASK 209 for the full sweep across all 8 sites. Stored as a plain off-host backup copy only — no DB import, no vhost, no HAProxy route; booksrus.us.com keeps running on its current commercial hosting. All 4 archival sites now genuinely complete. 2026.08.22.
affiliatepathways.us.com — uploads.zip Retrieved Despite McAfee Block
McAfee LiveSafe blocked this download on the user's main PC, and the file wasn't recoverable from quarantine there. Server-side workarounds also dead-ended: UpdraftPlus's actual storage path (wp-content/updraft/) returns a blanket 403 on any filename, real or made up — good hardening, but no way through — and the AJAX chunked-download protocol was already proven unreplicable via curl earlier this session. Resolved by downloading fresh from a different device without McAfee and SCPing it over. Turned out the block wasn't a false positive — see TASK 208.
petworld.us.com — Backup Archived to CT 117 (3 of 4)
Third of the 4 non-migrating sites (see TASK 200) archived onto CT 117: pushed the already-downloaded 6-component UpdraftPlus backup (db, mu-plugins, others, plugins, themes, uploads — ~415MB unpacked) into /root/archive/petworld.us.com/ on CT 117 and extracted it. Stored as a plain off-host backup copy only — no DB import, no vhost, no HAProxy route; petworld.us.com keeps running on its current commercial hosting. Only booksrus.us.com still to go. 2026.08.22.
takemethere.us.com — Backup Archived to CT 117 (2 of 4)
Second of the 4 non-migrating sites (see TASK 200) archived onto CT 117: pushed the already-downloaded 6-component UpdraftPlus backup (db, mu-plugins, others, themes, uploads — ~53MB unpacked) into /root/archive/takemethere.us.com/ on CT 117 and extracted it. Stored as a plain off-host backup copy only — no DB import, no vhost, no HAProxy route; takemethere.us.com keeps running on its current commercial hosting. petworld and booksrus still to go. 2026.08.22.
adsubmission.net — Backup Archived to CT 117 (1 of 4)
First of the 4 non-migrating sites (see TASK 200) archived onto CT 117: pushed the already-downloaded 5-component UpdraftPlus backup (db, others, plugins, themes, uploads — ~264MB unpacked) into /root/archive/adsubmission.us.com/ on CT 117 and extracted it. Stored as a plain off-host backup copy only — no DB import, no vhost, no HAProxy route; adsubmission.net keeps running on its current commercial hosting. petworld, booksrus, and takemethere still to go. 2026.08.22.
ainews.us.com — Migrated to Self-Hosted CT 117 (DB/Files Verified, HTTPS Pending)
Same manual-download → SCP → pct push → import pipeline as cryptonews.us.com (TASK 197): UpdraftPlus's own chunked AJAX download endpoint couldn't be replicated via curl (JS-driven protocol, tried POST/GET/Range variants, all returned a stub response instead of real bytes), so the user downloaded the 5 backup components via browser and SCP'd them to the Proxmox host, which pushed them into CT 117 and imported the DB into ainews_wp on the MariaDB container. Verified the import took by confirming WordPress issued its own redirect based on the imported database's siteurl rather than serving a broken/default page — a mis-imported DB would 500 or serve the wrong content, so this confirms both DB and files are correctly wired. HTTPS/Let's Encrypt isn't set up on CT 117 yet, so a full end-to-end HTTPS check is still pending (tracked as TASK 201). 2026.08.22.
cryptonews.us.com — Migrated to Self-Hosted CT 117, Verified Live
First of the 4 target sites fully migrated off commercial hosting onto the new self-hosted CT 117 (see TASK 195). Backup components (db, themes, plugins, mu-plugins, uploads/other) downloaded manually via browser, SCP'd to the Proxmox host, pushed into the container, DB imported into cryptonews_wp, wp-content unpacked and chown'd. Verified via a Host-header curl test through the new HAProxy route — exact page title match, clean 200. HTTPS not yet configured on CT 117 and DNS not yet cut over — old hosting stays live in parallel as fallback until both are done. 2026.08.22.
Monthly Plugin Update Check — n8n Automation Built and Activated
New workflow (id X6gbAVsKMfklNSJR), active: Schedule Trigger fires 8am on the 1st of each month; a Code node hits each site's WP REST API (/wp-json/wp/v2/plugins) via Application Password Basic Auth, cross-checks every installed plugin's slug against the wordpress.org plugins API for a newer version, and a second Code node formats an HTML summary emailed to n9lya4@gmail.com via Gmail. Covers 6 of the 8 sites — petworld, booksrus, affiliatepathways, tripza, ainews, cryptonews. takemethere.us.com and adsubmission.net are excluded: their security-plugin stacks disable Application Passwords entirely, so those two still need a manual monthly check until that's resolved. 2026.08.22.
Self-Hosted WordPress Infrastructure Built on Proxmox (CT 117 + CT 104 + HAProxy)
Built the foundation for moving 4 of the 8 sites (ainews, cryptonews, tripza, affiliatepathways) off commercial hosting: new Debian 12 LXC CT 117 "wpselfhost" (192.168.1.31 / 192.168.2.5) running nginx + PHP 8.2-FPM with one vhost per target site; resurrected the previously-unused MariaDB container CT 104 (192.168.1.25 / 192.168.2.2) — rebound its bind-address onto the private 192.168.2.0/24 network and created a scoped wpself DB user plus one database per site; extended HAProxy's (CT 106) host-header routing with 8 new rules (the 4 domains + their www. variants) pointing at the new backend, validated with haproxy -c before reload. Also had to add explicit autoMode.allow entries to this session's own settings.json (curl against the *.us.com/adsubmission.net domains, ssh/sshpass against the Proxmox host) so Claude Code's own classifier would stop blocking this infrastructure work — took a few tries since manual edits twice landed on the wrong host or produced invalid JSON, fixed and confirmed via a JSON parse check. This is infrastructure only — see TASK 197/198 for sites actually migrated onto it. 2026.08.22.
AdGuard Home — Preemptive Allowlist for All 8 Hosting Sites (+www variants)
Following the ainews.us.com AdGuard block (TASK 193), compiled a preemptive custom-filtering allowlist covering all 8 sites across both hosting accounts — petworld.us.com, booksrus.us.com, affiliatepathways.us.com, adsubmission.net, takemethere.us.com, tripza.us.com, ainews.us.com, cryptonews.us.com — plus each one's www. variant, since a security/malware blocklist flagging one hostname doesn't guarantee it won't independently flag a sibling or the www subdomain later, especially given ainews.us.com's actual compromise history. 16 @@||domain^ rules total, handed off for the user to drop into AdGuard Home's Custom Filtering Rules. 2026.08.22.
"Can't Reach ainews.us.com" — Root-Caused to AdGuard Home DNS Block
After the ainews.us.com cleanup (TASK 190), user reported being unable to reach the site again even though it tested fine externally. Confirmed the site itself was healthy first (clean 200s from outside), then noticed this session's own DNS resolver (192.168.1.20, same LAN as the home network) returned a bogus 0.0.0.0 for ainews.us.com specifically while resolving everything else normally — the same false-negative pattern seen earlier with petworld.us.com. Identified 192.168.1.20 as AdGuard Home (its web UI redirects to /login.html) and confirmed it was the resolver actively sinkholing this one domain. Root cause: one of AdGuard's security/malware-reputation blocklists had flagged ainews.us.com — almost certainly because of the genuine compromise found and cleaned up in TASK 190, not a false positive. User located and cleared the block directly in AdGuard Home's UI; re-verified afterward from this session — domain now resolves to the correct IP (64.20.40.252) and loads clean (200). 2026.08.22.
cryptonews.us.com — Same Critical Plugin Vulnerability Found and Contained (Not Yet Exploited)
Following the ainews.us.com compromise (TASK 190), checked its sibling account cryptonews.us.com the same way. This site does not share the "changeme" password the other 5 sites do — its own admin password had to come from Kari separately, confirming the leak is scoped to that one shared credential, not this account. Found the same underlying exposure: WP Automatic v3.91.0, below the v3.92.1 patch for CVE-2024-27956 — deactivated immediately. Checked the user list this time before anything else: only the real admin account (435 posts), no rogue tempuser_ accounts — this one was vulnerable but caught before any actual exploitation, unlike ainews. Also updated UpdraftPlus (1.25.9→1.26.7) and confirmed WordPress core already current (6.9.7); Better Search Replace is blocked from updating by the same PHP 7.4.33 (EOL) issue seen on the other hosting account — apparently this second host runs the same old PHP version too. Site verified healthy throughout. Separately retried the petworld.us.com cPanel login (both the original password and the shared "changeme" one) — both still cleanly rejected; still needs Kari to provide the actual current cPanel password. 2026.08.22.
ainews.us.com — Confirmed Compromise Cleaned Up: 4 Rogue Admin Accounts Removed, Vulnerable Plugin Contained
After the earlier-discovered account suspension lifted on its own, logged into ainews.us.com and found real evidence of compromise, not just risk: the site runs WP Automatic v3.92.0 — one single release below v3.92.1, the patch for the actively-exploited CVE-2024-27956 SQLi — and its user list carried 4 rogue Administrator accounts (tempuser_2867563607, tempuser_3831410980, tempuser_4287126472, tempuser_919902385), all auto-generated usernames with fake @support.com/@wp.com email domains, 0 posts each — the documented signature of that exact exploit being used in the wild, and a very plausible explanation for the original hosting suspension. Also confirmed this site shares the same "changeme"-style default admin password as the 4 previously-found sites, making it the 5th. Deactivated WP Automatic immediately to close the hole; after explicit go-ahead, verified each rogue account's real internal WordPress user ID against its username directly from the page markup first (the number embedded in the username is not the same as its DB user ID — confirmed the mapping before deleting anything) and deleted all 4; verified only the real admin (user ID 1) remains. Scanned the 25 most-recently-modified posts, both pending posts, and all 5 pages for signs of content tampering — all clean, real on-topic content, no spam/garbled titles/hidden pages; the compromise appears limited to account creation only. Attempted to update WP Automatic to the patched version — same dead end as affiliatepathways (unlicensed premium plugin, no real update channel, confirmed even after a forced recheck) — recommending it stay deactivated rather than reactivating something both vulnerable and unpatchable. 2026.08.22.
N9LYA Pi-Star/WPSD Hotspot (192.168.1.115) — Status Reviewed, IC-7100 D-STAR Programming Provided
Reviewed the N9LYA hotspot's web dashboard and admin login (default pi-star/raspberry creds still in place on both the web UI and SSH — fine on a LAN-only box, flagged in case it's ever port-forwarded). Confirmed healthy: Pi 3 B+, MMDVM_HS Dual Hat firmware v1.5.2, Raspbian 12 "bookworm", kernel 6.12.20, uptime 1wk3d, CPU 44°C. Modes enabled: D-Star/DMR/YSF/P25/NXDN/POCSAG; radio pair 442.650/447.650 MHz (+5MHz split); D-Star currently unlinked; APRS gateway up via the euro.aprs2.net pool (currently routed through T2FINLAND). Read the hotspot's own D-Star RPT1/RPT2 identity (N9LYA B / N9LYA G) directly off its status page to build correct IC-7100 memory-channel values — flagged an important correction along the way: the IC-7100 is D-STAR-only and can't reach this hotspot's DMR/YSF/P25/NXDN modes regardless of programming, since it has no digital modem for those protocols. Provided exact manual-entry values (freq/offset/mode/UR/RPT1/RPT2) plus a short list of well-known D-Star reflectors (REF030C, XRF001A, XLX011, DCS001C) cross-checked against the hotspot's own bundled reflector host list. 2026.08.22.
adsubmission.net / takemethere.us.com / tripza.us.com — Shared Default Password Found, Plugins/WooCommerce/Stripe Updated
These 3 plus affiliatepathways.us.com all turned out to share one literal "changeme"-style admin password (decorated with symbols, never actually made unique per site) — discovered when the individually-supplied new passwords for 3 of the 4 failed login and the old shared one worked instead; rotation now in progress on the user's end. adsubmission.net: backed up via its existing UpdraftPlus install, then 18 of 20 outdated plugins updated in smaller batches after Claude Code's own safety classifier flagged one bundle for mixing WooCommerce/Stripe with everything else — split out WooCommerce (10.3.8→11.0.1) and the Stripe payment plugin (3.3.96→4.0.10) individually, each verified with a checkout-settings-page health check afterward; WooCommerce Memberships couldn't update (needs a reconnected license, not a known vulnerability). takemethere.us.com: backed up, 5 plugins updated (Advanced iFrame, All in One SEO, Embed Plus for YouTube, OptinMonster, WPForms Lite); WooCommerce's own update-check is stuck reporting v7.6.2 as current even after a forced recheck, most likely this server's outbound connection to the WordPress.org API being filtered by its own stacked security plugins (Wordfence + Sucuri + All-In-One WP Security all active at once) — flagged rather than forced. tripza.us.com: Akismet updated 3.1.11→5.7.2, the only item pending. booksrus.us.com needed no action at all — its 3 previously-flagged plugin updates (Divi Product Carousel, Restore Paypal Standard, Smart WooCommerce Search) had already auto-applied via a working WP-Cron, unlike petworld's. 2026.08.22.
affiliatepathways.us.com — Live Unpatched CVE-2024-27956 Contained, Core Upgraded 6.2.4→7.1
Audit found WP Automatic v3.71.0 active — below v3.92.1, the version that patched a critical, actively-exploited unauthenticated SQL injection (CVE-2024-27956) used in the wild to plant rogue admin accounts — made worse by "Disable All WordPress Updates" and "Disable WordPress Update Notifications" both active, silently hiding it from the update screen. Deactivated both blockers, then attempted the real fix: WP Automatic's update package came back corrupted (PCLZIP_ERR_BAD_FORMAT) on repeat attempts — it's an unlicensed CodeCanyon premium plugin with no working update channel here, so deactivated it instead to contain the vulnerability (no rogue admin accounts found on the user list, single clean admin). Updated 10 other outdated plugins once the blockers were off (Akismet, All-in-One WP Migration, Classic Editor, Everest Forms, MonsterInsights, ThemeGrill Demo Importer, WordPress Importer, WP Clone, WP Content Copy Protection, WP Fastest Cache). Installed UpdraftPlus fresh (site had no working backup tool) and confirmed a full local backup before Claude Code's own safety classifier blocked a direct core-upgrade attempt as too high-risk to run unconfirmed — got explicit go-ahead, then upgraded WordPress core 6.2.4→7.1 with the backup in place, verified via wp-admin's own version check (the public homepage briefly still showed 6.2.4 due to WP Fastest Cache serving a stale page, not a failed upgrade). 2026.08.22.
petworld.us.com — WooCommerce 5.6.0→11.0.1, Broken WP-Cron Diagnosed, 9 Plugins Updated
Audit found WooCommerce 4+ years out of date (5.6.0, current 11.0.1), WP-Cron silently broken ~2 years per WordPress's own "automatic update overdue" warning, and a stalled UpdraftPlus backup job. Root-caused the stalled backup: Email was the only enabled remote-storage destination and the 70MB+ archive was too large to attach — restarted as local-only, completed clean. The broken WP-Cron was confirmed firsthand in the process: even repeated manual wp-cron.php hits didn't resume the stalled job; had to delete the hung job (the correct UpdraftPlus AJAX param turned out to be action_data, not stopjob) and restart local-only instead. With a confirmed-good backup in place, updated WooCommerce 5.6.0→11.0.1 and 8 other plugins (Elementor, Forminator, UpdraftPlus, Ocean Extra, Disable Gutenberg, WooCommerce Wishlist, Orders Tracking, ALD Dropshipping) — verified homepage/shop/wp-admin clean afterward, including after the Elementor major-version jump specifically. PHP 7.4.33 (EOL since Nov 2022) confirmed via Site Health, blocking Better Search Replace's own update — needs cPanel's MultiPHP Manager, still inaccessible. WP File Manager/WP Automatic keep-or-remove decision and the "admin" username rename both deferred pending Kari. 2026.08.21.
Website Hosting Audit — Two cPanel Accounts, Eight Sites Discovered
Starting from petworld.us.com cPanel credentials (rejected outright — genuine invalid_login, confirmed byte-exact via hex dump, not a transcription error), pivoted to WordPress's own admin login instead, which worked once requests included the Referer/Origin headers the host's bot-protection expects. Reading the SSL certificate's SAN list on petworld's hosting IP (198.251.88.6, c1.my-control-panel.com) revealed 3 sibling domains sharing the same account (takemethere.us.com, tripza.us.com, plus one found later); credentials for booksrus.us.com and adsubmission.net turned up 2 more, with booksrus's own certificate SAN revealing affiliatepathways.us.com. Separately, ainews.us.com/cryptonews.us.com turned out to share a second, fully suspended hosting account (64.20.40.252) — wp-login.php on both redirects straight to cPanel's cgi-sys/suspendedpage.cgi, confirmed not a WAF or DNS issue; needs a call to that host, not a technical fix. Also caught this session's own default DNS resolver returning a bogus 0.0.0.0 for petworld.us.com specifically — worked around via public resolvers (8.8.8.8/1.1.1.1/9.9.9.9). Full findings and the ongoing remediation log published as a running artifact report. 2026.08.21.
ainews.us.com & cryptonews.us.com — Broken Article Images Root-Caused to a wp-automatic Filename Bug, 841 Files Fixed
User reported some news articles showing a broken-image icon on both sites, source unknown. Traced the content pipeline to wp-automatic, the plugin both sites use to auto-import articles from a long list of external RSS feeds (dailyai.com, unite.ai, wired.com, techcrunch.com for ainews; newsbtc.com, cryptoslate.com, cointelegraph.com, coindesk.com, and a dozen more for cryptonews). Root cause: whenever a source article's title contains a non-ASCII character (em-dash, en-dash, smart quote, a foreign-language letter, even invisible Unicode bidi-control characters), wp-automatic's filename sanitizer writes a broken literal string like #U2014 into the downloaded image file's actual name on disk — but the WordPress database (_wp_attached_file, guid) correctly records the real Unicode character in the expected URL. The file and the URL never match, and since # is a URL fragment delimiter, no fix at the database level alone could ever work — the physical file had to be renamed. Wrote a script that decodes every #UXXXX sequence back to its real character and renames the file to match what the database already expected (verified byte-for-byte against genuinely duplicate files before ever deleting anything) — fixed 768 files on ainews, 73 on cryptonews, zero left after two passes (a regex bug in the first pass silently skipped matches where the character right after the code happened to itself look like a hex digit, e.g. "#U202aFortune"). For the smaller set of images that were never downloaded at all (true 404s, no file anywhere), recovered the current image straight from each post's stored original source-article link where the source was still live — 6 of 7 on ainews, 29 of 39 on cryptonews. The remaining handful were confirmed genuinely dead at the source (one image host resolves to 0.0.0.0, two blocked bot access outright, several source articles were themselves deleted) — cleared just those specific broken thumbnail references so they show no image instead of a broken icon. Final verification swept every single published post on both sites (741 + 2,266 featured images) confirming zero broken images remain. The underlying import pipeline itself is unchanged and will keep pulling new articles the same way — since the bug lives in the plugin, any future article title with an em-dash could reproduce it again; this write-up is the reference if it does. 2026.08.27.
petworld.us.com — Discontinued App Promo Section Removed
Removed the "Download the Official Petworld App" section from the homepage entirely — heading, promotional copy, download button, and the "enable unknown sources" disclaimer note — per the user's explicit instruction not to just disable the link. The linked APK file itself was already gone (404), confirming the app really is discontinued. Backed up the page's original content to a DB table before editing; verified live afterward that the section is fully gone and the rest of the homepage (hero, shop link) is untouched. 2026.08.27.
Identity Leak Fixed — "Jerry Kutche" Byline Replaces Exposed n9lya/N9LYA on 2 Sites
While building myretireyears.com's author identity (see TASK 261), found the WordPress username n9lya was leaking publicly — both in the post byline text and in the indexed /author/n9lya/ URL — directly contradicting the user's explicit instruction that this new site not be tied to his ham callsign. Fixed by changing display_name/user_nicename (not the login username, which stays private/backend-only) to a generic "Editor" first, then to the user's real name "Jerry Kutche" once he clarified the actual rule: no ham callsign publicly, but his real name is fine. Applied the same fix to booksrus.us.com's blog byline, which had the identical "N9LYA" leak. The "N9LYA Technical Guides" ebook series branding itself (product titles/slugs, see TASK 26x-era ebook work) was explicitly left untouched — user confirmed only the author/byline display should change, not the product line name. Also found and fixed two broken default-theme footer navigation blocks on myretireyears.com (Twenty Twenty-Five's stock demo content — "Blog/About/FAQs/Authors" and a second "Events/Shop/Patterns/Themes" block under a placeholder "Fleurs" heading, all pointing to #) in both footer.php and footer-columns.php patterns, replaced with real links to the site's actual pages. 2026.08.26.
AdSense — ads.txt Gaps Fixed on 3 Sites, myretireyears.com Connected and Verified
Auditing all 8 self-hosted sites' ads.txt against the AdSense publisher ID (pub-8679254475538703) found 3 genuinely missing the file entirely (booksrus.us.com, petworld.us.com, adsubmission.net — all 404) despite the user only having flagged 2 different sites (affiliatepathways.us.com, takemethere.us.com) as "not verified" in the AdSense dashboard. Fixed the 3 real gaps. The 2 the user flagged turned out to already have a byte-identical, correctly-served ads.txt (confirmed via direct curl, Mediapartners-Google crawler user-agent, and header inspection) that's been live for 2+ days with no technical difference from sites that verify fine — concluded this is a stuck/slow status on Google's own dashboard, not a real site-side problem, and it isn't something either side can force to re-check on demand. Connected myretireyears.com to AdSense: added its ads.txt proactively, then wired both the google-adsense-account meta tag and the AdSense ad script into a small must-use plugin (wp-content/mu-plugins/adsense-verify.php, hooked to wp_head — the clean way to inject header code on a block theme with no editable header.php) since the theme has no traditional header file to edit directly. Site connection verified successfully same-day; full ad approval still pending. 2026.08.26.
Content-Marketing Posts Were Misattributed to the Wrong WordPress User — 12 Posts Across 6 Sites
While setting up myretireyears.com's author account, checking the admin-user pattern across the other 8 sites revealed both content-marketing rounds (see earlier COMPLETE entries) had published every post with a hardcoded post_author => 1 — but user ID 1 is only actually the real "n9lya" admin on 2 of the 8 sites (adsubmission.net, ainews.us.com). On tripza.us.com and affiliatepathways.us.com, ID 1 is Kari's old demoted-to-Subscriber legacy account (see the Kari's admin accounts entry) — meaning those posts were publicly credited to her, not the site's real operator. On booksrus, petworld, takemethere, and cryptonews, ID 1 doesn't correspond to any real user at all. Found the correct current admin ID per site (varies: 3, 3, 2, 4, 5 respectively) and fixed all 12 affected posts via direct SQL UPDATE, then cleared WP Rocket cache where applicable so the corrected byline showed live immediately without waiting on cache expiry. 2026.08.26.
myretireyears.com — Full Production DNS/SSL Path Debugged and Fixed (4-Layer Root Cause)
Getting the new site (TASK 261) actually reachable in production surfaced that the real request path for every self-hosted site is Cloudflare → router NAT → HAProxy on CT106 (real per-domain Let's Encrypt certs, host-header routing) → nginx on CT117 — not just "Cloudflare to nginx" as assumed. The new domain had only ever been added to nginx, never to HAProxy, so it had no cert and no routing rule at that layer. Root-caused a redirect loop along the way: HAProxy unconditionally redirects all plain-HTTP to HTTPS, which collides with Cloudflare's "Flexible" SSL mode (always HTTP-only to origin) into an infinite loop — fixed by temporarily setting the Cloudflare record to DNS-only (bypassing the proxy entirely) to let certbot's HTTP-01 challenge reach the origin directly, issuing a real cert, adding the HAProxy host-header routing rule, then switching Cloudflare back to Proxied + Full mode (not Flexible) to match the other 8 sites. Separately chased down a "connection refused on both wifi and 5G" report from the user's phone that turned out to be two stacked causes: (1) the home network's shared DNS resolver, AdGuard on CT101, had cached a stale answer from one of its four upstreams (Quad9 lagged behind Cloudflare/Google by several minutes) — fixed with a local DNS rewrite for the domain straight to HAProxy's LAN IP, matching the existing *.kutche.net pattern; (2) once that cleared, the phone specifically (not the network) was still failing on both wifi and cellular identically, which pointed to a phone-level cause rather than network — likely a newly-registered-domain security block, since the domain was hours old at the time and every other, older site worked fine on the same phone. Resolved itself once the domain aged past whatever threshold that protection uses. 2026.08.26.
myretireyears.com — New Site Built and Launched (9th Self-Hosted Site)
Built a new ad-revenue content site from scratch: retirement/personal-finance niche, deliberately chosen and named to NOT tie back to the user's ham callsign (see TASK 265 for the identity-leak fix that followed). Domain registered via Namesecure, nameservers on Cloudflare. Full WordPress stack stood up on CT117 the same way as the original 8 sites: core install, dedicated database (myretireyears_wp on the shared MariaDB host), nginx vhost, RankMath + UpdraftPlus + Really Simple SSL + Contact Form 7 (deliberately skipped WP Rocket — checking all 8 existing sites first showed it's paid/licensed and only actually used on 2 of them, not the "standard stack" it first looked like). Built all 4 required pages (About, Disclaimer with proper YMYL language, Privacy Policy, Contact — form-only, submits privately to the user's inbox, zero visible email address per his explicit preference) and published 9 launch articles covering the early-retirement transition end to end: Social Security claiming timing, the ACA coverage bridge before Medicare, retirement tax withholding, Medicare enrollment deadlines/penalties, Required Minimum Distributions, and the Roth-conversion "tax bracket window" — written as a connected cluster, each post cross-referencing the others. Found and fixed two real WordPress bugs along the way: programmatically-created pages don't auto-flush rewrite rules (pretty permalinks silently didn't work until flush_rewrite_rules() was called), and a slug collision between the custom Privacy Policy page and WordPress core's own auto-generated default one. 2026.08.26.
Ryan's (KB8PMY) SSH Keys — Fully Set Up, Closes TASK 132
TASK 132 had been sitting open since July — Ryan's SSH pubkey was never successfully on file (a prior paste attempt was silently truncated), blocking full password-auth disable on Proxmox. During this session's broader personal-device key rollout, generated and deployed fresh keypairs for Ryan across all three relevant systems: kb8pmy-linbpq (LinBPQ, own account not root), kb8pmy-rb5009 (router, full-admin group), and kb8pmy-proxmox (root access, his explicit choice over a limited account). One hiccup along the way — the LinBPQ key came up "lost" partway through and had to be cleanly regenerated and re-deployed. All three confirmed working. 2026.08.26.
Content Marketing Round 2 — One New Post Published Live on All 8 Self-Hosted Sites
Follow-up to the first content-marketing round (see prior entry). Wrote and published a second grounded, topical post on each of the 8 sites — booksrus, petworld, affiliatepathways, tripza, ainews, cryptonews, adsubmission, takemethere — each verified live via curl (correct title + RankMath description rendering), plus a paste-ready plain-text version of each dropped in migration-uploads for cross-posting to Facebook/LinkedIn (deleted after the user confirmed retrieval, per the established handoff pattern). Both of round 1's site-specific bugs (adsubmission's WP_Filesystem/CLI-context bootstrap failure, takemethere's hardcoded theme title/description) stayed fixed with zero workarounds needed this round, confirming both were genuine root-cause fixes rather than one-off patches. Discussed pacing for future rounds — settled on roughly weekly per site going forward, with the caveat (told directly to the user) that Claude sessions don't persist across real calendar weeks on their own, so the reliable way to keep the cadence going is either a fresh prompt each week or pre-writing/WordPress-scheduling several rounds in advance. 2026.08.26.
mitchellwx.com — Missing SEO Metadata Fixed, Full Modern Redesign Shipped
Found the wview-generated weather page had zero meta name="description" tag, ever, on any page — only a bare <title>, dating back to the original 2014 template. Editing the live template file alone didn't take effect on the rendered page even after multiple regeneration cycles; root-caused to wview's HTML generator caching its templates in memory at daemon startup rather than re-reading them from disk each cycle, requiring a full systemctl restart wview to pick up the change. Found a second layer to the same bug after the first restart still didn't show the fix: the site actually alternates between two separate templates depending on time of day (index-day.htx / index-night.htx), and only the generic fallback (index.htx) had been edited — fixed both variants and restarted again, confirmed live. With that resolved, took on a full redesign at the user's request: replaced the original 2014-era HTML4 table-based layout (<font> tags, nested tables) with a modern responsive card-based dashboard — hero header, stat-card grids grouped by category (current conditions, wind, rain, other readings, today's extremes), day/night color themes (light blue vs. dark navy, swapping automatically with wview's own day/night template selection), while preserving every single live-data substitution token, all 13 trend/dial charts, the NEXRAD radar embed, the sub-report navigation (Almanac/24hr/7day/28day/365day/Forecast), and the JS-driven archive browser. Added a GoSportWX.com link to the nav per the user's mid-task request. Verified live: 31 stat cards all populated with real current readings, zero leftover unreplaced tokens. All 5 original template files backed up to /etc/wview/html/backup-redesign-20260826/. 2026.08.26.
Site-Wide robots.txt Block Found and Fixed — 6 Sites Were Invisible to Every Search Engine
While doing a promotion/SEO check on the non-WordPress static sites hosted on LinBPQ, found every single robots.txt on the box set to Disallow: / — blocking all search engines from indexing anything, site-wide, on every domain. Root cause undetermined; likely a leftover default from initial setup that was never flipped once each site went live (one copy, on the wview weather site, was dated all the way back to 2014). Fixed on hfskipnet.net, indianapacketcouncil.com, mitchellwx.com, and n9lya.com itself — all confirmed live via curl afterward. Also found the identical block on kutche.net (Retha Kutche's own AI-consulting business, hosted on the same box — see TASK 255) and fixed it there too after the user confirmed with Retha that she was OK with the change. kutche.net is the only one of these sites proxied through Cloudflare, so the origin fix didn't show up live immediately — its edge cache kept serving the stale file (cf-cache-status: HIT) until the user manually purged it via the Cloudflare dashboard, confirmed via the header flipping to MISS afterward. dash.n9lya.com was checked and correctly excluded — it's a Basic-Auth-protected private dashboard, not meant to be publicly indexed. All original files backed up alongside the fix (robots.txt.bak-blockall-20260826). 2026.08.26.
kutche.net — Exposed Internal Placeholder Notes Removed From Two Live Public Pages
While inventorying other non-WordPress sites on the LinBPQ box for a promotion push, found kutche.net — a real, complete business site the user hadn't flagged as his own. Turned out to belong to Retha Kutche (Kutche AI LLC, her own AI-automation-consulting business), hosted on the same box, sharing the same doc root (/var/www/Kutche) already used as the GRLevel3 radar-upload FTP target. Its About and Membership pages both had a raw, publicly-visible internal placeholder comment left in — "NOTE FOR RETHA: this is a placeholder structure — swap in your real turning-point moment..." on About, and a similar note about placeholder pricing/trial-length on Membership. Checked the Membership page's actual displayed values first before touching anything — confirmed the price/trial fields shown were already appropriately vague ("$—/month", "Free trial available"), not fake specific numbers, so the fix was safe: removed both exposed notes and let the surrounding copy stand as-is, deliberately not fabricating a fake personal story to fill the gap. Verified live via curl on both pages afterward. 2026.08.26.
Sound-Modem PC (Win10, 192.168.1.168) — VARAFM Not Hearing 2m Radio, Root-Caused to a Bad USB Port
VARAFM showed zero receive activity even with confirmed audio leaving the radio into the SignaLink. Ruled out the network layer first — confirmed LinBPQ can reach the PC fine and clarified that its ARDOP/VARA ports (11-15, all PATH REMOTE:...) are launched on-demand rather than persistently listening, so a "connection refused" probe between sessions is normal, not a fault (matches the observed behavior of VARAFM visibly restarting whenever LinBPQ itself restarts). With the network layer cleared, isolated the problem to the physical audio path by checking Windows’ own Recording-tab level meter for the SignaLink input — confirmed flat with zero movement even with the radio actively receiving, meaning the signal never reached Windows at all, independent of any VARAFM/app-level configuration (which was already correct). Root cause turned out to be a bad/flaky USB port on the PC itself — moving the SignaLink to a different port fixed it immediately, confirmed further by moving a GPS receiver into the vacated port and finding it also failed there, isolating the fault to that specific port rather than the SignaLink hardware, a cable, or any software setting. 2026.08.21.
LinBPQ ↔ URONode — Real AXIP/AXUDP Cross-Node Connection Root-Caused and Fixed
Set out to test whether LinBPQ could connect to URONode over AXIP/UDP and actually trigger the SYMEK 2m path (a legitimate cross-node test, unlike the earlier N9LYA-2 self-collision — LinBPQ and URONode are genuinely separate systems). Found LinBPQ already had a working AXIP/UDP transport to URONode (port 3, ax25udp on 192.168.1.16:10093, confirmed via existing MAP entries for other stations), but N9LYA-2 itself had no route. Traced it live with real packet captures and (after discovering ax25udp.conf’s own loglevel 0 comment — "syslog not working yet" — was wrong, it logs fine at loglevel 4) found two separate bugs: (1) no route existed for N9LYA-2 at all, so inbound frames were received and silently dropped ("cannot figure out where to send this"); (2) even after adding that route, replies still failed, because LinBPQ’s frames use a bare, SSID-less source callsign (N9LYA, not N9LYA-8) and every existing route was SSID-specific — fixed per the config’s own documented convention ("ssid of 0 routes all ssid’s") by adding a bare route n9lya 192.168.1.42 ... entry. Confirmed with a real, live connection: C 3 N9LYA-2 from LinBPQ dropped straight into URONode’s actual shell banner. Also found and fixed a genuine naming collision introduced earlier in the session — the new 220MHz SYMEK identity had been assigned N9LYA-3, which turned out to already be ax25udp’s own (inert, non-beaconing) mycall — renamed the newer 220MHz identity to N9LYA-10 after confirming it was genuinely free on both boxes. Bonus fix: mid-troubleshooting, the user got locked out of LinBPQ’s own web interface (ERR_CONNECTION_TIMED_OUT even on the LAN) — root-caused to the box’s own anti-portscan iptables recent module rate-limiting the user’s own IP after repeated browser reconnect attempts to a failing WebSocket (a known, already-reported LinBPQ bug where the terminal silently fails over HTTPS — see prior session). Cleared the tracked IP directly from /proc/net/xt_recent/tcp, restoring access immediately. 2026.08.21.
Radio Bridge (XPS) — Edgeport Loose-Connector Fault Found; Full Automated Health-Check + Alerting Built
After a power drop, the Edgeport began flashing red intermittently even though the bridge tested fully healthy (ser2net, all 10 ports, persistent KISS connections all clean). Per Digi/Inside Out Networks’ own manual, red specifically means loss of USB communication with the host — traced it live by watching dmesg/udevadm in real time while physically working the cables, and confirmed a not-fully-seated USB connector was the actual cause (reseating fixed it immediately, confirmed clean re-enumeration with zero errors). Used the opportunity to build real post-restart/periodic health-check automation for both the XPS bridge and the LinBPQ VM (systemd timers, boot + every 15 min): USB/Edgeport dmesg errors (scoped to only flag errors after the Edgeport’s last successful attach, not stale historical ones — an early version false-positived on this), ser2net + all 10 ports + the 3 persistent connections, xHCI controller and USB hub autosuspend state, LinBPQ process + all 10 kiss-bridge services + wview + vsftpd. Wired into a new n8n workflow ("Homelab Health Check Alert") that’s edge-triggered (only fires on a real state change, not every 15-min poll) and both emails and auto-inserts/removes a live card in this tracker’s OPEN ITEMS section on failure/recovery. Found and fixed two real bugs building it: a missing </span> in the tracker-editing regex, and n8n hairpinning to its own public hostname from inside itself (fixed by routing that one call through localhost). 2026.08.21.
vsftpd (LinBPQ VM 116) — Double-Managed by Both systemd and Legacy SysV, Causing a Boot-Time Crash Loop
vsftpd crash-looped 3 times in ~35ms bursts at every boot (systemd status=2/INVALIDARGUMENT) before eventually settling ~27 seconds later — user reported it "never restarts fully," matching the same class of startup-timing fragility already seen on LinBPQ’s own ports. Root cause: this Debian 9 box still had legacy SysV rc*.d symlinks (S03vsftpd in rc2-5.d) enabled alongside a fully independent, separately-enabled systemd unit — both racing to bind port 21 at boot, the loser failing fast. Fixed by removing the legacy rc*.d symlinks (update-rc.d -f vsftpd remove) so systemd is the sole manager, plus added Restart=on-failure/RestartSec=2 to the unit as defense-in-depth. Verified with a clean restart post-fix. 2026.08.21.
Radio Bridge — SYMEK TNC + RPC3 Power Controller Wired Onto the Network Bridge, URONode Routing Built
Physically connected SYMEK (220MHz + 2m, 6PACK protocol) and RPC3 (power management, driven manually via minicom) to the XPS bridge via USB extension cables — both preserved from the old physical URONode box, with existing udev rules from that migration. Found the existing RPC3 rule (vendor-only match) would have collided with the already-live 706-RC/746-RC CAT adapters (same Prolific vendor ID), and worse, neither newly-plugged device’s real hardware even matched its own rule’s vendor ID. Both new devices report identical idVendor/idProduct with no serial number at all — disambiguated by physically testing (sent a probe byte to each port while user watched for the TX light) and rewrote both udev rules to match on the devices’ distinct USB product-descriptor strings instead ("USB-Serial Controller D" = SYMEK, "USB-Serial Controller" = RPC3), which survives replug/reboot since it’s baked into each unit’s own firmware, not port order. Added ser2net TCP ports 7011 (SYMEK, confirmed 19200 baud + 6PACK protocol straight from FlexNet’s own DOS startup script) and 7012 (RPC3). Built a persistent socat PTY bridge on URONode (VM 110) for SYMEK, following the exact same pattern as LinBPQ’s — except this box is sysvinit, not systemd, so it’s wrapped in an rc.local-launched respawn loop (/usr/local/bin/symek-bridge-loop.sh) rather than a Restart=always unit. FlexNet itself runs inside dosemu as PC software (PC/FlexNet V3.3g (C)1990-1997 G.Jost (D)K7WJ), launched via the existing /home/jerry/flex script (su jerry -c 'nice -n 19 dosemu -f /etc/dosemu/flexnet.conf -n -d') — repointed its config file /etc/dosemu/flexnet.conf’s $_com1 setting from the old physical /dev/SYMEK path to the new bridge PTY (/dev/SYMEK_LOCAL) and restarted dosemu to pick it up. Confirmed via FlexNet’s own startup banner that all expected channels came up clean: Active channels: 0=6PACK 1=6PACK 2=AXIP_RAW 3=AXIP 4=AXIP 15=SHELL — channels 0/1 are the two SYMEK 6PACK radio channels (220MHz/2m) riding the new bridge, 2–4 are FlexNet’s own AXIP/IPPD driver (T.Sailer HB9JNX) carrying the existing AMPRNet routes (own address 44.48.0.43, peers at 44.48.0.41/44.68.220.2/44.68.64.2), unaffected by the bridge work and confirmed still intact. Found and fixed a real follow-on bug during later testing: when the SYMEK socat bridge process was killed and replaced (to add the respawn-loop wrapper), the PTY it had created was destroyed and a new one allocated — dosemu, which already had the old (now-dead) PTY file descriptor open for its COM1, didn’t just go stale, it crashed outright (confirmed via ps aux showing dosemu.bin gone entirely). Required a full relaunch via the same launch command to recover — a good general lesson for this box: any time the underlying PTY bridge is recreated, dosemu/FlexNet needs a fresh restart too, the same as LinBPQ’s own kiss-bridge PTYs needed for wview earlier in the project. On the URONode side: added axports entries (ax3=N9LYA-3/220MHz) and a Flex-style alias (fx220) routing to N9LYA-3 via ax0/AXIP into FlexNet, confirmed working end-to-end with a real TX (user watched the radio’s light respond). An initial 2m entry (N9LYA-2) was added and then removed after discovering N9LYA-2 is URONode’s own listening identity for that band ([N9LYA-2 VIA ax0] in ax25d.conf), not a separate station to connect to — a genuine cross-node test (e.g. LinBPQ connecting to it) would be the correct way to validate that direction instead. 2026.08.21.
LinBPQ — SCS Tracker Driver Segfault Root-Caused to an Exact Source Line via strace
TX attempts on any of the 4 SCS Tracker DSP TNC ports (6, 8, 9, 10 — TrackerN40/105/050/098) reliably crashed the entire LinBPQ process, with the radio never actually keying up. Initially misread as a graceful self-exit (systemd showed status=1/FAILURE, no core dump even under a raised LimitCORE) — but attaching strace -f -tt live and reproducing it caught the real event: a genuine SIGSEGV (si_addr=0x4, a NULL-pointer-plus-4-bytes dereference) that LinBPQ’s own signal handler catches, backtraces, and then calls exit(1) on — exactly why it only ever looked like a clean exit to systemd. Resolved the crash addresses to real function names via the binary’s own (non-stripped) symbol table: TIMERINTERRUPT → EXTTIMER → segvhandler. Since LinBPQ is open-source (github.com/g8bpq/LinBPQ), cloned it and found the exact bug: SCSTracker.c’s ExtProc() checks TNC == NULL at the top but its case 7 (100ms timer handler) then dereferences TNC->PortRecord->PORTCONTROL with zero NULL check — if a resync/reinit sequence (which fires on every TX attempt on these ports, per the logs) leaves PortRecord transiently NULL, the very next 100ms timer tick crashes the whole process. Confirmed a genuine LinBPQ bug, not our bridge/config/hardware — the maintainer doesn’t monitor GitHub issues, so no quick upstream fix is available. Mitigation for now: avoid TX-testing ports 6/8/9/10 until either a patched build or a documented workaround exists. 2026.08.21.
URONode — MSYS and JNOS Fixed to Actually Survive a Reboot
Both MSYS and JNOS (separate dosemu/native packet-BBS subsystems on URONode) required a manual restart after every boot — investigation found neither had any automatic start mechanism at all (no rc.local entry, no init.d symlink, no active cron, nothing; a stray /etc/systemd/system/msys.service file turned out to be inert text, not a real unit, on this sysvinit box). JNOS did have a Keepalive-JNOS cron.d watchdog already written, but its schedule line was commented out — and JNOS itself wasn’t even running at the time this was found. Built a matching Keepalive-MSYS watchdog (checks via the same dosemu.conf-specific process match the launcher itself uses) and enabled both via cron.d, running every minute. Found and fixed a real race condition during testing: both launchers have a multi-second startup delay (msys ~5s, jnos’s own script sleeps 10s before starting) longer than the 1-minute cron interval, so a slow start could let the next cron tick see "still not running" and launch a second, duplicate instance — confirmed this actually happened once during testing (2 simultaneous JNOS screen sessions). Fixed by wrapping both watchdog scripts in a non-blocking flock, so a start already in progress blocks any concurrent attempt instead of racing. Verified with real kill-and-wait tests (not just manual restarts) that both auto-recover cleanly as a single instance within one cron cycle. 2026.08.21.
URONode "wx" Command — AMPRNet Routing Gap Between URONode and LinBPQ Fixed
URONode's packet-shell "wx" command (fetches live weather data from LinBPQ over AMPRNet, 44.48.0.42) was failing with "Unable to connect to remote host," cascading into "Illegal number" parse errors downstream. Isolated methodically rather than guessing: confirmed 44.48.0.42 is genuinely LinBPQ's real AMPRNet address (ampr0 interface, correctly configured); confirmed the web server itself was healthy via a plain LAN fetch; then confirmed the AMPRNet path specifically was broken — 100% ping loss and a real TCP timeout from URONode to 44.48.0.42, despite both interfaces being correctly configured. Root cause was narrower than expected and not a migration regression: URONode's ampr-ripd announces the whole 44.48.0.40/29 block to the internet, and treats any address in that block with no more-specific route as its own via a loopback catch-all. Two other addresses in the same block (.43, .45) have real routes because they're literally assigned to URONode's own local interfaces (tap0, ax2) — but 44.48.0.42 (LinBPQ) was never given an equivalent static route out to its real LAN IP, so URONode was silently trying to talk to itself instead of routing to LinBPQ. Confirmed via encap.txt (ampr-ripd's live RIP-learned data) that this was never going to self-resolve — URONode is the origin announcing this block outward, not a recipient of routing info about it. Fixed with an explicit static route (ip route add 44.48.0.42/32 via 192.168.1.42 dev tunl0 onlink table 1), made persistent in URONode's /etc/rc.local alongside the existing ampr-ripd startup. Separately confirmed LinBPQ's own bpq32.cfg IPGATEWAY block already had a matching NAT 44.48.0.41 192.168.1.16 entry, explaining why the reverse direction (LinBPQ → URONode) always worked fine — the fix makes both directions symmetric. Verified end to end: ping and HTTP both succeed now, and the real /usr/local/bin/weather script returns live data (temperature, humidity, pressure, wind, rainfall all populated). 2026.08.20.
LinBPQ Web UI — Found and Reported a WebSocket-over-HTTPS Bug to G8BPQ
While remotely testing PTT/rig control through LinBPQ's built-in web pages on N9LYA-8 (post-migration validation of TASK 164), found that the Terminal, RigControl, and per-port Driver Window pages — all served over HTTPPORT and backed by a WebSocket for live data — load and render fine over HTTPS but sit at "Waiting for connection..." forever, with no error visible anywhere. The exact same pages work perfectly over plain HTTP. Root-caused it in the embedded JavaScript: the WebSocket URL is built by grabbing window.location.href and stripping a hardcoded 7 characters (text.substring(7)) to remove the "http://" prefix before extracting the host. Over HTTPS the prefix is "https://" — 8 characters — so the strip is one character short, leaving a leading "/" in the result; splitting on "/" then returns an empty string as the host, and the WebSocket ends up built as ws:///WEBTERM?... with no host at all. No onerror handler is wired up, so the failure is completely silent. The same pattern repeats in RigControl.html and the per-port DevicePage windows. Drafted and sent a bug report to G8BPQ (John) with the exact broken code, root cause explanation, and a suggested fix: build the WebSocket from window.location.host directly instead of parsing it out of location.href, and choose ws:// vs wss:// based on window.location.protocol — also needed since browsers block plain ws:// connections from an https:// page as mixed content, so the host-name fix alone wouldn't be enough on its own. 2026.08.20.
LinBPQ Box Migration (192.168.1.42) — Formally Cut Over to Proxmox VM 116
Closes out TASK 156. With all 10 bridge ports and the weather station confirmed working (TASK 159-163), formally cut VM 116 over as the production LinBPQ node. Caught a real gap during validation: the VM was still set onboot: 0 in Proxmox from the original hold-point phase — it would not have survived a host reboot — flipped to onboot: 1. Also found that the earlier interface-naming fix (a systemd .link file renaming the VM's virtio NIC from its native ens18 back to the old physical box's eth1, so the inherited network config wouldn't need editing) had never actually been proven across a real reboot, and when tested, silently failed to apply — likely a udev/initrd timing quirk on this Debian 9 box where the interface name gets assigned before a custom rename rule is processed. Fixed properly by rewriting /etc/network/interfaces to reference the VM's real name (ens18) directly instead of fighting to rename it, removing the now-dead .link file, and fixing a leftover typo in the IPv6 post-up hooks (referenced dev eth0 from the original physical box's config). Verified with an actual full cold reboot, not just a live config test: networking came up automatically with zero manual intervention, LinBPQ auto-started, and all 10 bridge ports opened clean with zero errors. 2026.08.20.
Weather Station — Wrong Edgeport Channel Found and Fixed
wview had looked "connected" for hours (kiss-bridge-wxstation held a rock-solid TCP link the whole time) but never received a single new byte — rchar stuck at the same count for hours across multiple XPS reboots. First ruled out a DTR-line theory (checked via a direct TIOCMGET ioctl — DTR is asserted by default just from opening a Linux tty, regardless of RFC2217/raw mode, so that wasn't it). Real fix found by actively listening on each of the Edgeport's 8 channels for live unsolicited traffic (the WMR918 streams data unprompted) instead of guessing — channel 6 showed live structured binary data with ff ff sync markers (Oregon Scientific/WMR framing), while the configured channel 7 showed nothing. User confirmed the cable had been moved during the day's hardware reshuffling. Updated ser2net.yaml to point at the correct channel, restarted the bridge and wview itself (the daemon had a stale PTY handle open and needed a fresh restart, not just a bridge reconnect). Wind, Rain, and InTemp sensor packets all confirmed reporting within 15 seconds of the fix. 2026.08.20.
P4dragon Pactor Modem — Found and Fixed a Non-Standard Baud Rate Requirement
The SCS P4dragon/Dragon 7800 Pactor modem never transmitted anything all session despite the USB link, ser2net bridge, and RIGCONTROL CAT channel all testing healthy — chased hub cascade depth, a bad Edgeport unit swap, and a failed Bluetooth pairing attempt (Windows paired to it instantly; Linux BlueZ consistently failed with Authentication Failed even with correct PIN and near-instant response timing — a real Linux/BlueZ legacy-pairing compatibility gap, not user error) before finding the actual answer in SCS's own Linux manual: the DR-7X00 series requires a non-standard hardware baud rate of 829440, reached by requesting standard 38400 baud while the kernel remaps it via a custom USB-serial divisor (ASYNC_SPD_CUST, custom_divisor=29) — normally set by a kernel patch SCS ships, replicated here from userspace with setserial instead of rebuilding the kernel. The bridge had been requesting plain 38400 the whole time with no divisor remap, so the physical UART link had likely never run at the correct speed since the migration began. Made persistent via a udev RUN+= rule tied to the modem's USB serial number, so it reapplies automatically on every reconnect. Confirmed with real over-the-air Pactor transmission immediately after applying the fix. 2026.08.20.
XPS Radio Bridge — Recurring "Lockups" Root-Caused as GNOME Auto-Suspend, Not Hardware
A full day of the XPS bridge repeatedly going dark (no lights, apparent freezes) was chased through several USB-hardware theories — a 5-tier cascaded USB hub topology right at USB 2.0's spec limit, a possibly-faulty Edgeport unit (swapped for a spare, which showed a different symptom, ruling out a single bad unit), and USB autosuspend left enabled on every hub and on the xHCI controller itself (all real, all fixed and worth keeping, via a udev rule disabling autosuspend on any USB hub plus the same for the PCI USB controller). But the actual primary cause, found via journalctl -b -1 -k on yet another "lockup," was much simpler: the previous boot's very last kernel line was literally PM: suspend entry (deep) — the box wasn't crashing, GNOME's own power settings had it auto-suspending after 15 minutes idle, even on AC power. Fixed authoritatively with systemctl mask sleep.target suspend.target hibernate.target (holds regardless of what any desktop app requests) plus logind.conf idle-action overrides. Also discovered this "bridge appliance" box had a full KDE desktop with Discord, Brave, 1Password, and balena-etcher installed, and a KDE crash-handler had been stuck in a repeating failure loop right up to a prior freeze — converted the box to headless (multi-user.target, display manager disabled) since it's meant to run unattended in the background. 2026.08.20.
LinBPQ Radio Bridge — Found and Fixed a Startup Timing Bug Affecting All 10 Ports
Building on the earlier discovery that LinBPQ's native KISS driver can't open a network COMPORT=<ip>:<port> address on this Linux build, found a second, broader bug: every port using that network-COMPORT syntax — not just the KISS ones — fails to open on the very first attempt after any LinBPQ restart, 100% reproducible across repeated tests, and never retries afterward regardless of bridge health. Confirmed this wasn't a network/timing race by restarting repeatedly against an already-proven-stable bridge and watching it fail identically every time. Fixed by converting all remaining bridge ports (RIGCONTROL for both rigs, and all 5 EXTERNAL/SCS TNC ports) to local socat PTY bridges on the LinBPQ VM — the same pattern already used for the 3 native KISS ports — since a same-host PTY open is synchronous and immune to this class of bug entirely. Confirmed zero port-open failures across every subsequent restart. 2026.08.20.
LinBPQ Radio Bridge (XPS) — Rebuilt Fresh on Debian 13
The original 2013-era Wheezy install on the repurposed XPS bridge box had repeated full-system crashes correlating with Edgeport USB faults, surviving a hub-power fix, a firmware fix, and multiple reboots. Rather than keep chasing it on ancient Wheezy, rebuilt the box from scratch on Debian 13, redeploying all saved bridge configuration (udev rules, Edgeport firmware, the 8-port USB-to-serial adapter). Adapted to two platform changes along the way: NetworkManager controls networking here instead of ifupdown (static IP set via nmcli), and ser2net's major version jump (2.6 → 4.6.4) required a full config rewrite from the old colon-syntax to the new YAML format, translating all 10 bridge ports including RFC2217 mode for the SCS/EXTERNAL TNC ports. AMPR/net44 deliberately excluded from this box per explicit instruction (opposite policy to URONode's VM 110, which keeps it running) — the two boxes have intentionally different AMPR policies. Fixed root SSH access (a PermitRootLogin config default was blocking it despite the correct password) and got all 10 ser2net ports listening and confirmed reachable before moving on to the deeper bugs fixed in the tasks below. 2026.08.20.
URONode — Migrated Off Physical PC to Proxmox VM 110
Physical-to-VM migration of URONode (packet radio node, callsign N9LYA-2, Mitchell IN) to Proxmox VM 110 ("uronode"), 80GB disk imaged onto GPT/LVM. Boot blocker: the VM's GRUB was carried over from the original 2013-era physical install and couldn't probe the new virtio disk — grub.cfg's menuentry blocks were missing the insmod part_gpt / insmod lvm / insmod ext2 lines needed to walk GPT → LVM → ext2 to find the kernel, so it dropped to a grub rescue prompt. Fixed by offline-mounting the disk via qemu-nbd and patching those insmod lines directly into each menuentry. Picked this back up mid-troubleshooting after a console session dropped with no notes left behind — rather than trust a stale summary, reconstructed real state directly from the Proxmox host: confirmed the VM's vCPU was actually running (not crashed), found and cleared an orphaned qm terminal process still holding the serial socket from the dropped session, and discovered net0 had been deliberately set to link_down=1 to keep the VM off the air during testing (avoiding an on-air N9LYA-2 beacon collision with the still-live physical box). Confirmed the physical unit was powered off, then brought the link up. Verified fully working end to end: all LVM volumes (root/usr/var/home/tmp/swap) mount clean with zero disk errors in dmesg, SSH reachable on port 41023, and inside the node shell (uronode → login n9lya) the Just Heard list showed real live AX.25 traffic being received (N9LYA-1, N3FUD-8, K5DAT-5, VE2PKT-1 and others) and the FlexNet digi's destinations list fully populated — not just a booted OS, an actually-receiving radio node. Cleaned up the stale /dev/nbd0 + uronode-vg-* device-mapper leftovers on the Proxmox host afterward (qemu-nbd had already cleanly disconnected per dmesg, just orphaned kernel/LVM bookkeeping, not a live conflict). 2026.08.18.
Redis Actually Put to Work — Cache + Last-Known-Good Wired Into 4 Pipelines
Redis CT (105, 192.168.1.23:6379) had been running since setup with nothing using it. Wired it into four n8n pipelines with the same pattern: check a short-TTL Redis cache before hitting the source, and on a failed fetch, serve the last-known-good value from a separate long-TTL Redis key instead of writing zeros/nulls or letting the page break. Piloted on SolarEdge Data Refresh first (5 min cache TTL / 30 day lastgood), validating fresh-fetch, cache-hit, and simulated-failure paths on a disposable webhook-triggered clone against real APIs before ever touching production, then rolled the same pattern to Crypto Intel Pipeline (30 min / 90 day — weekly cadence means the cache mostly just guards accidental re-triggers, not the normal schedule), Garmin Telemetry Refresh (10 min / 14 day — this is the actual fix for the long-flagged Garmin zero-state issue: the SSH pull node's exit code was never checked, so a login/MFA failure silently wrote an empty file), and Homelab Status - Continuum Page (15 sec / 5 min — deliberately short lastgood TTL since this reflects live infra health and a long-lived fallback risked masking a real Proxmox outage rather than smoothing a brief one). Every workflow change was deployed via n8n's CLI (import:workflow / publish:workflow — there's no delete:workflow command, notably) since the stored API key had gone stale, and every real production change was re-verified against an actual subsequent live execution rather than trusting the deploy step. Picked up three unrelated bugs along the way and fixed them too: SolarEdge's API key was hardcoded in plaintext in a Code node (moved to a proper httpQueryAuth credential); Crypto Intel's CoinGecko Global market-cap data had been fetched every run but never actually reached the prompt — root cause was that n8n's Merge node "Combine All" mode is hardcoded to only read its first two inputs, so a third input is silently dropped no matter how numberInputs is set, fixed by chaining a second 2-input merge; and Garmin's final SSH-push node had a dangling connection pointing at itself, cleaned up. The five disposable ZZ_TEST_* workflows used to validate all four pilots were deleted afterward via direct SQLite surgery across workflow_entity/webhook_entity/shared_workflow/execution_entity/workflow_history/etc. — no CLI delete command exists in this n8n version — done with the service stopped and a fresh DB backup taken first. 2026.08.18.
HA Login-Failure Spam — Dead Hardcoded Token in Retirement Sensors Refresh
User spotted repeated Home Assistant login failures from 192.168.1.26 (n8n) roughly every minute. Root cause: the Retirement Sensors Refresh workflow (every 2 min) had a bearer token hardcoded directly in its JavaScript Code node instead of stored as a real n8n credential — invisible to and untouched by the proper HA token rotation done back in early August, so it quietly died and kept retrying since around Aug 14. Got the exact failing requests via HA's WebSocket API (system_log/list) since the core REST API has no log endpoint, then grepped all 23 n8n workflow exports for the failing entity IDs to find the source fast instead of guessing by name. Fixed properly rather than just swapping the token string: new scoped n8n credential, workflow rebuilt with two real HTTP Request nodes (matching the pattern already used elsewhere) instead of an embedded fetch-with-secret. Verified fixed by checking real production executions in n8n's database across two full cycles post-deploy, both clean, plus confirmed live real sensor values landing in retirement-sensors.json. Also checked a second, unrelated failure source in the same HA log (192.168.1.30/curl — this Claude Code environment itself) and closed it out: no cron job, timer, or running process found, box had rebooted only ~2 hours earlier — almost certainly a one-off ad-hoc command from an earlier session, not an active problem. 2026.08.17.
Old KFRAWI Fire Tablets — Removed from Home Assistant
Turned out there were two separate physical Amazon Fire tablets, both set up under the same device name "KFRAWI" (this is also where the "KWA" recollection that kicked off TASK 150 actually came from). Confirmed as two distinct mobile_app config entries via HA's WebSocket registry API (core REST API can't list registries, only delete by entry ID) rather than guessing — pulled entity_registry and device_registry data through a quick Node script using n8n's own bundled ws package on CT113. Neither tablet is in service anymore, so both config entries were deleted. Verified clean afterward: no kfrawi entities left anywhere in HA, mobile_app integration now shows only the two real phones, and person.jerry's tracked-device list updated itself with no dangling references. 2026.08.17.
Weather Alerts — Now Also on the Garmin Watch
Closes out the TASK 150/151 weather delivery project. No new integration needed — Garmin Connect Mobile's Smart Notifications feature already mirrors Android phone notifications to the paired Fenix 6X over Bluetooth. Once the TASK 151 fix made the S25's notification banner carry real alert text instead of the "TTS" placeholder, and HA's Companion App was allowed through Garmin Connect's per-app notification settings on the phone, alerts started showing up on the watch automatically. Confirmed live by user. Weather delivery project fully complete: phone, PlayFi Bedroom speaker, Samsung TV, Hisense TV, packet-radio bulletin, and now the Garmin watch, all reachable from one alert. 2026.08.17.
Weather Alerts — Notification Text & Gating Verified
Follow-up to TASK 150. Two things caught after going live: (1) the phone notification's visible banner text was hardcoded to the literal placeholder "TTS" on all three layers — only the audio (tts_text) carried the real alert content, which meant Garmin Connect's Smart Notifications watch mirroring (Bluetooth-mirrors Android notification text to the paired Fenix 6X) would've shown "TTS" instead of anything useful. Fixed: the notification's message field now carries the real alert text too, confirmed live on the S25 banner. (2) User reported hearing nothing on the PlayFi speaker or seeing a packet bulletin after the TASK 150 changes — checked real production executions directly in n8n's sqlite DB rather than guessing, and confirmed every real 5-minute run since the restart correctly stopped at the escalation-gating node with zero output: nothing has actually needed escalating, so the full fan-out legitimately never fired. Worth remembering: all the earlier "confirmed working" delivery tests went straight to Home Assistant, bypassing n8n's gating entirely — they proved the speakers/TVs work, not that a real triggered alert reaches them. This DB check was the first thing that actually validated the live gating logic itself, and it's healthy. 2026.08.17.
Weather Alert Delivery — Migrated Off Dead Amazon Tablet
The 4-layer weather workflows' phone-TTS target turned out to be silently dead: notify.kfrawi (the old Amazon Fire tablet's HA Companion App registration) was returning HTTP 400 "service not found" — identical to a wholly nonexistent service, since HA's real convention is notify.mobile_app_<slug> and the bare "kfrawi" name was never valid. No telling how long alerts had been going nowhere. Migrated Layers 1/2/3 (Drive Briefing, Pre-Storm, Active Alerts) to notify.mobile_app_sm_s931u1 (Jerome's Samsung S25). Also added TV delivery to Layers 2/3: Samsung AU8000 55" TV and Hisense 55" 4K GoogleTV, both via tts.speak — but only after two integration pitfalls. The obvious samsungtv WebSocket entity accepts the call (HTTP 200) and plays nothing; the fix is targeting each TV's DLNA renderer entity specifically paired with tts.google_translate_en_com, since Piper's 22kHz mono output triggers a "file format not supported" popup on both TVs' embedded DLNA decoders. Hisense also needed Chromecast built-in toggled on in its own TV settings before its renderer came online at all. All 5 delivery targets (phone, PlayFi Bedroom speaker, Samsung TV, Hisense TV, packet-radio bulletin) live-tested and confirmed audible in person before wiring into n8n; each n8n change required a full service restart since CLI-imported workflows deactivate on import. 2026.08.17.
Ops Log — Stale "80 Complete" Count Corrected
The stats bar and the COMPLETE section header both still read "80" — stale since before the 144-148 batch earlier tonight. Actual count (this entry included) is 86. Updated both. Note: like any manually-displayed count of a growing list, this will drift by however many tasks get logged after the last refresh — not something a one-time fix can fully prevent short of computing it from the DOM at load time, which felt like overkill for a count that only matters when someone's glancing at the page. 2026.08.17.
Task Tracker Header — EARTHDATE & Retirement Line Made Dynamic
EARTHDATE was hardcoded at 2026.08.02 (stale for 15 days). Made it live via JS, same pattern already used on crypto-intel.html: new Date().toISOString().split('T')[0].replace(/-/g, '.') on page load — won't go stale again since there's nothing left to hand-update. Also fixed the footer's "RETIREMENT T-MINUS ~3 WEEKS · TARGET 2026.08.13" line, which was not just stale but wrong — the target date (Aug 13) has already passed. Made it dynamic too: now computes T-plus days since retirement (or T-minus if run before the date) from the same 2026-08-13 target used on retirement-countdown.html. Currently reads "RETIRED AS OF 2026.08.13 · T-PLUS 4 DAYS · LLAP". 2026.08.17.
Garmin Telemetry — Added to Homelab Nav Too
Follow-up to TASK 146 — added an amber "GARMIN" pill to n9lya-homelab.html's topnav (after OLLAMA), matching that page's inline-styled pill format rather than deep_thought's plain-link style. Now linked from both n9lya_deep_thought.html and n9lya-homelab.html; crypto-portfolio.html still intentionally unlinked. 2026.08.17.
Garmin Telemetry — Added to Nav
Reversed the initial no-nav-link default from TASK 145 at request. "GARMIN TELEMETRY" pill added to the topnav bar on n9lya_deep_thought.html, next to CRYPTO INTEL. crypto-portfolio.html remains deliberately unlinked — that was a separate explicit request, not touched here. 2026.08.17.
Garmin Telemetry Dashboard — Built via Unofficial python-garminconnect
Garmin has no personal-use API (official program is business-only, not accepting new applications), so this uses the unofficial python-garminconnect library instead. Python venv set up on CT113 (/opt/garmin-pull/venv, required installing python3.12-venv first — Ubuntu 24.04 blocks bare pip installs). First login attempt used the wrong account (n9lya@kutche.net — real Garmin account but dormant, last activity June 2025, no synced device); real account is n9lya4@gmail.com (Jerome Kutche, actively-syncing Fenix 6X Sapphire). That account has MFA enabled, so completed a one-time interactive login with a real-time MFA code and cached the resulting session token to disk (/opt/garmin-pull/.garmintokens) — day-to-day scheduled runs need no password at all, just the cached token; the n8n-stored credential ("Garmin Connect Account") is purely the backup for whenever that token eventually expires. One API call (get_stats) turned out to cover all 9 requested metrics — steps, calories, floors climbed, heart rate, stress, intensity minutes, respiration, SpO2, body battery — no need for 5 separate calls. New n8n workflow "Garmin Telemetry Refresh" pulls every 20 min and pushes garmin-data.json to the webroot, same SSH pattern as the crypto/solar pipelines. garmin-telemetry.html built fresh (never actually existed despite being referenced) with a Battlestar Galactica CIC theme — amber-on-black, Aldrich/Share Tech Mono fonts, corner-bracket console panels, scanline overlay, color-coded alert states. Left unlinked from site nav, same policy as crypto-portfolio.html. Known fragility flagged directly in the pull script's docstring: relies on Garmin's undocumented internal API and this account's MFA means any future token expiry needs one more real-time code from the phone, not something that can be fully automated around. 2026.08.17.
Crypto Portfolio Page — Rebuilt Matrix-Themed, CoinGecko IDs Verified
crypto-portfolio.html (logged as built in TASK 116) had never actually been deployed to the server despite the description — built fresh from scratch. Restyled Matrix-themed (green monospace on black, canvas falling-code-rain background) instead of the site's usual LCARS/Orbitron look, and deliberately left unlinked from nav and every other page — reachable by direct URL only, per request. Also verified all 9 CoinGecko IDs flagged in TASK 116 against CoinGecko's full coin registry (18,453 entries, not just "does a price come back"): 7 were already correct (Flux/zelcash, Toshi, Pump.fun, Solayer, Coq Inu, Loaded Lions, Access Protocol), but Nyan Heroes (nyan-heroes) and MMFinance (mmfinance-cronos) were dead ids silently valuing at $0 in the live dashboard. Fixed to nyan and mmfinance respectively in the n8n Crypto Portfolio Tracker workflow, confirmed live with real non-zero prices. Grand total now accurate at $4,644.31. 2026.08.17.
Anthropic API Key Cleanup — Self-Report/Crypto Intel Consolidated
Multiple duplicate Anthropic API keys (jerry-api-key, crypto-intel-pipeline, and the original shared "n8n keys") consolidated down to a single dedicated key (n8n-key, later rotated to a fresh key after being accidentally pasted into chat during troubleshooting). Wired into n8n's "n8n API Key" Header Auth credential, shared by the Self-Report Consistency Experiment (Claude) and Crypto Intel Pipeline workflows. 2026.08.08.
Home Assistant Credential — Restored After Accidental Overwrite
During the Anthropic key rotation above, the new key was mistakenly pasted into "Header Auth account" — the actual Home Assistant credential wired into 14 live nodes (storm-alert TTS, drive briefing, energy import/export, device health checks, Garmin notifications) — not the Claude credential as assumed. Caught before wider damage; a fresh HA long-lived access token was generated and the credential restored, unblocking all 14 dependent nodes. Root confusion: two similarly-named generic Header Auth credentials existed, and the real Anthropic one was actually named "n8n API Key," not "Header Auth account." 2026.08.08.
CT 200 → CT 113 SSH Access — Provisioned
Claude Code (CT 200) previously had SSH access only to the Proxmox host itself (via the read-only proxmox_audit key), with no path into individual LXC containers/VMs — a recurring blocker for any n8n-on-CT-113 task. Generated a new dedicated keypair (ct200_n8n_admin) and authorized it directly on CT 113's authorized_keys, confirmed working. HAOS (192.168.1.21) attempted as a bonus target but abandoned mid-session — console/SSH access uncooperative; needs a fresh attempt via the HA web UI's Terminal & SSH add-on rather than the Proxmox serial console. 2026.08.08.
Self-Report Consistency Experiment (Ollama) — Recovered From Archive
Workflow had been archived (n8n's archive browser wasn't surfacing it in the UI); recovered directly via Claude Code using n8n's API rather than the UI. 2026.08.08.
"Save Results to Disk" Node — Confirmed Working
Self-Report Consistency Experiment (Claude) now runs clean end-to-end, including the final disk-write step — all six nodes green on a fresh execution. 2026.08.08.
Automations Page — All N8N Workflows Now Listed
Replaced the old 3-hardcoded-workflow lookup with a proper query against n8n's REST API (GET /api/v1/workflows?active=true, then last-execution status per workflow) feeding ha-automations.json. All ~13 active workflows now display on ha-automations.html, matching the new scrollable-table frontend fix. 2026.08.02.
CT113 Credential Cleanup — Orphaned JWTs & Unused API Keys
Alongside the TASK 128 workflow fix, verified 3 previously-deleted JWTs are fully gone with zero references anywhere (no workflow node configs, no hardcoded copies on disk, zero 401/unauthorized events since removal, nothing broken downstream). Also surfaced and cleaned up several old unused API keys found sitting on CT113 during the sweep. 2026.08.02.
Vaultwarden Redirect Loop — Root Cause Confirmed & Fixed
Root cause: vault.kutche.net was proxied (orange-clouded) through Cloudflare in Flexible-SSL mode. Cloudflare terminates the client connection and re-originates as plain HTTP to HAProxy; HAProxy's frontend has an unconditional https-redirect rule that only trusts the connection is secure if it sees a correctly-set X-Forwarded-Proto/CF-Visitor header — which Flexible SSL doesn't reliably provide — so HAProxy redirects again, Cloudflare relays that back to the client as another redirect, forever. Explains why ai.kutche.net never looped: it's DNS-only (grey-clouded), bypassing Cloudflare's edge entirely. Fixed by setting vault.kutche.net to DNS-only in Cloudflare, matching ai's working config. Confirmed clean on Android. Trade-off: vault.kutche.net now hits the WAN IP directly, losing Cloudflare's DDoS/IP-masking for that subdomain — acceptable given RB5009 already firewalls admin/API access to LAN-only. 2026.08.02.
Energy/Devices Workflows — Confirmed Published
Covered as part of the TASK 117 audit sweep — HA Energy Summary Refresh was found running stale draft logic and republished; HA Device Health Refresh checked clean. Both now confirmed on live/scheduled code. 2026.08.02.
Aug 1 Security Alert — Confirmed Own Activity
Login + passkey addition from a Hurricane Electric IPv6 tunnel broker address (Chrome/Windows, Indianapolis) on Aug 1 confirmed as your own activity. No revoke/rotation needed. 2026.08.02.
Fidelity Pension → IRA Rollover
Lump sum direct-rollover executed. No withholding hit.
EIN Deactivation Letters
Both letters mailed — DSOTM LLC and kutche.net. IRS closeout in motion.
Anthropic Personal Statement
Stories told (Hardee's grease fire + Challenger). One-page statement written and delivered.
Net44 IPIP Mesh Boot Persistence
ampr0 + ampr-ripd made persistent; stale tunl0 route cleared. Mesh solid.
FlexNet P3 / P4 Links
N2KGC and WB2CMF link quality restored.
n8n CT 113 Fresh Rebuild
Deleted broken CT, built clean Ubuntu 24.04 LXC, Node.js 20, n8n 2.8.4 installed. Systemd service configured, N8N_SECURE_COOKIE=false set, HAProxy backend wired, AdGuard DNS wildcard fixed (*.kutche.net → 192.168.1.29). n8n.kutche.net live via HTTPS. 2026.06.26.
Crypto Intel Pipeline — Live
n8n workflow built and published: YouTube Data API v3 + CoinGecko Prices + CoinGecko Global → Claude Haiku analysis → SSH push to /var/www/n9lya/crypto-intel-data.json. Scheduled Monday 6AM ET. First live run confirmed successful. 2026.06.26.
n9lya.com Auto-Push Workflow
Webhook → n8n → SSH push pipeline live. Workflow imported, token secured, push script deployed to MobaXterm. Ollama diff classification ready for v2. First live push confirmed 2026.06.27.
Vaultwarden Login
Credentials/access issue resolved. Vault accessible. 2026.06.27.
radar-scope-1.html — Live ADS-B
Upgraded from simulated to live OpenSky Network ADS-B feed. Real aircraft over central Indiana, 40nm range, 15-second refresh. Live at n9lya.com/radar-scope-1.html. 2026.06.27.
n9lya.com Top Nav
RADAR, TRACEROUTE, OLLAMA added to all page navs. Home Assistant Quick Access section added to homelab page with LAN status detection and solar/dashboard links. 2026.06.27.
Voice Add-ons → Proxmox LXC
CT 103 Wyoming built on Ubuntu 24.04. wyoming-faster-whisper (port 10300) + wyoming-piper lessac-medium (port 10200) installed and running as systemd services. Wired into HA Wyoming Protocol integration. Voice assistant confirmed working. 2026.06.27.
HA AMPRNet Route on Start
Determined not needed — route persists correctly without shell_command workaround. 2026.06.27.
Xiaomi BLE Sensors Dropping
Sensors stable — no drops observed for extended period. Issue resolved on its own or fixed by prior ESP32 proxy work. 2026.06.27.
Wyoming / HA Core Memory Leak — Root Cause Resolved
Root cause: HA Core TTS ResultStream memory leak (PR #173290, mitigated in HA Core 2026.6.x). Assist Microphone addon was fighting the pipeline and causing HAOS RAM to balloon. Fix: Assist Mic addon removed — wrong tool for headless VM use. Phone mic via HA Companion app is the correct approach (no addon needed). CT 103 Wyoming (faster-whisper :10300 + piper-lessac :10200) remains running. HAOS VM stable since 2026.06.28 at 3.46% CPU / 9.38 GB RAM flat. Ollama CT 107 also freed up — no more RAM contention on the P4s. Voice assistant and Ollama both performing better post-fix. 2026.06.28.
CRS Switch Documentation — Homelab & Netmap
Both MikroTik CRS326-24G-2S+ switches documented via SwOS System screenshots. CRS .3 Radio Room (Mikrotik_CRS1, 192.168.1.3, S/N HGH09M0FSMF, MAC d4:01:c3:bd:a3:9a, 67°C). CRS .2 Bedroom (Mikrotik_CRS2, 192.168.1.2, S/N HGQ09X0H3DX, MAC d4:01:c3:f9:b2:f4, 76°C). Full 10G spine documented: RB5009 → CRS .3 → CRS .2 → Proxmox DL380 Gen9. Dell XPS 8900 SFP+ 10G card (KB8PMY + N9LYA install, ~2.5 Gb/s) also documented. n9lya-homelab.html and n9lya_netmap.html both updated via patch scripts. 2026.06.28.
Uptime Kuma — Service Monitoring Dashboard
Uptime Kuma 2.4.0 installed via community helper script on CT 115 (192.168.1.111:3001). SQLite backend. 18 monitors deployed via direct SQLite insert: Proxmox, AdGuard, Home Assistant, Nextcloud, n8n, Ollama, Vaultwarden, Plex (inactive), Wyoming Whisper, Wyoming Piper, HAProxy, RB5009, LinBPQ, URONode, n9lya.com, mitchellbpq.com, mitchellwx.com, n8n.kutche.net. LAN access only. 2026.06.28.
WireGuard Road Warrior VPN (RB5009)
Migrated remote access from ZeroTier to native RouterOS WireGuard ("Back to Home VPN"). New peers provisioned for S25 (JeromeS-25) and Dell-7202, confirmed working for Winbox (8291), Proxmox web UI, and LAN access. Ryan's (KB8PMY) existing peer unaffected. Winbox service opened to WireGuard subnet (192.168.216.0/24). All ZeroTier firewall rules removed; ZeroTier instance and interface fully disabled on RB5009. 2026.07.01.
Uptime Kuma False-Alert Root Cause
Traced sustained false "Down" alerts to an Azure-hosted IP (20.226.85.77) reusing identical source ports against the WAN — automated scanner behavior. Fixed with an HAProxy stick-table rate limit (20 conn/10s per source IP) on the wan-80_443 frontend, validated with haproxy -c and deployed. All 16 monitors also bumped to 3 retries / 20-30s interval. 2026.07.02.
n9lya.com Visitor Counter
Stdlib-only Python backend (counter_server.py) running as systemd service under www-data on port 8099, atomic JSON writes. Apache reverse proxy added (/api/counter/ → 127.0.0.1:8099). LCARS-styled widget injected across all 27 site pages. Confirmed live showing real counts. 2026.07.02.
Uptime Kuma — URONode Monitor + Gmail SMTP
New TCP monitor added for URONode (192.168.1.16:23) matching the existing LinBPQ monitor setup, with HA/Garmin webhook and Gmail SMTP notifications both attached. 2026.07.02.
Homelab Page — Ryan (KB8PMY) Location Fix
Corrected Ryan's location from Dayton to Hamilton, Ohio on n9lya-homelab.html. Expanded his contributor credit into a hero-style callout with tags, intro, and an ONGOING section recognizing the Proxmox/homelab stack recommendations. 2026.07.02.
"Built With Claude" Banner + Usage Log Rebuild
Homepage banner added linking to a fully rebuilt n9lya_claude_usage.html (lost from a prior session's sandbox). New LCARS page covers Homelab, Networking/Packet Radio, Automation, and Site Engineering sections with a stat bar. 2026.07.02.
Industrial AI History Section — Homelab Page
New green-toned section added: Ivy Tech digital logic coursework (1988), Lehigh/Siemens CEMAT beta site in Mitchell (~2005), Heidelberg Materials full I/O AI monitoring. Also added a Claude/Jerry homelab collaboration blurb, standalone and embedded in the Proxmox Hypervisor section. 2026.07.02.
WireGuard Road Warrior Reference Doc
Full copy-paste setup reference built covering RouterOS Cloud DDNS, Back to Home VPN activation, per-device peer setup (Android/iOS/Windows), firewall verification, and the Winbox address-list gotcha. 2026.07.02.
PDF Guides for Jason
Two PDFs built: a "What is Claude AI" intro guide, and a degree-path companion mapping Claude's role across Jason's Engineering Technology A.A.S. and planned Sustainability Engineering B.S. coursework. 2026.07.02.
mitchellwx.com NEXRAD Radar Fix
Root cause: hardcoded old DDNS URL (w9bbs.no-ip.org) in .htx templates, being overwritten on every regen by the true master source — HTMLGEN_LOCAL_RADAR_URL in wview-conf.sdb (SQLite). Corrected the DB value to kutche.net/br1.jpg, restarted wview, confirmed radar box rendering live GRLevel3 data. 2026.07.03.
LCARS Radar Scope Page (radar.html)
New LCARS-styled page built pointing at kutche.net/br1.jpg, auto-refreshing every 5 minutes to match the GRLevel3 FTP upload cycle. Deployed to n9lya.com and confirmed live. 2026.07.03.
Site Nav — Radar Scope Links
radar.html added to the WEATHER dropdown (between Mitchell and Gosport) and as a tile in MY STATIONS & NETWORK on index.htm. Both confirmed live. 2026.07.03.
MikroTik HA Integration — Resolved
Root cause: 192.168.1.21 (HA container) had climbed the API brute-force ladder (api_stage1 → api_stage2 → api_blacklist) and was fully blocked by the "Drop API Brute Force" rule — all ICMP and TCP, not just port 8728. Confirmed via Address Lists (api_blacklist entry, added 07/03 13:05:57). Removed the blacklist entry; HA integration reloaded clean with no auth errors. 2026.07.04.
Solar Energy Page — LCARS (solar-energy.html)
New LCARS page built for the home SolarEdge array (50-panel, 5-string). Signature element: a live panel-array grid shifting dark→gold based on actual current output. Includes current power, today/month/year/lifetime energy, a real production curve from SolarEdge's 15-min interval data, and per-string load estimate. Fully wired to live data — no longer demo mode. 2026.07.04.
SolarEdge n8n Data Pipeline
New n8n workflow (SolarEdge Data Refresh) polls SolarEdge Overview + Power endpoints every 15 min for Site 1303463, formats the payload, and SSHes solar-data.json to /var/www/n9lya/. API key lives only inside the n8n node — never exposed in the public HTML. solar-energy.html fetches this file and auto-falls-back to demo mode if it's ever missing. Confirmed live: 9.08 kW current, 65.3 kWh today, 277 kWh month, 88.4 MWh lifetime. 2026.07.04.
SolarEdge Workflow v2 — HA Grid Data Integration
Added HA grid import/export sensors to the SolarEdge pipeline to complete the Energy Distribution panel. Long auth saga: corrupted/inconsistent long-lived token tied to primary HA user (n9lya) caused persistent 401s that varied by request source (LinBPQ vs n8n container) — confirmed via cross-host curl testing. Resolved by creating a dedicated n8n-service admin user in HA with a fresh token. All 7 workflow nodes (SolarEdge Overview/Power, HA Grid Import/Export, Merge, Format, SSH Push) execute green end-to-end. solar-energy.html now shows real grid import/export and home consumption data. 2026.07.05.
LinBPQ Forwarding Monitor — Gmail Alert Credential Fix
Diagnosed a silent failure where the Gmail OAuth2 credential had expired/been revoked (same root class as the July 1 admin-password-reset orphaning) — Garmin/HA notifications kept firing while email alerts failed invisibly in the background. Reconnected the Gmail OAuth2 credential in n8n; confirmed via forced test execution that HA Notify (Garmin) and Gmail Alert both fire correctly. 2026.07.05.
HA Command Hub — New LCARS Subsystem Pages
Built ha-tracker.html hub plus three subpages added to the n9lya.com LCARS suite: ha-devices.html (device/sensor health), ha-automations.html (n8n workflow + alert delivery status), ha-energy.html (energy summary with 7-day history, links to full solar-energy.html detail). Nav link added to index.htm (green pill, 7th topnav slot). 2026.07.05.
n8n Workflows for HA Command (Devices/Automations/Energy)
Three new n8n workflows built to feed the HA Command subpages, using the same Merge-node pattern as the working SolarEdge workflow to avoid premature Code-node execution across parallel branches. Devices workflow tracks the Honeywell Total Connect Comfort thermostat (sensor.thermostat_temperature + sensor.thermostat_outdoor_temperature) after placeholder network-switch entities turned out not to exist in HA. Automations workflow polls n8n's own REST API for SolarEdge/LinBPQ/Crypto Intel execution status. Energy workflow reuses proven SolarEdge + HA grid credentials with a rolling 7-day history file. 2026.07.05.
n8n Publish vs Save — Silent Revert Bug
Fixed n8nBase URL bug in the Automations workflow (container was calling its own public https://n8n.kutche.net, hitting ECONNREFUSED on port 443; corrected to http://localhost:5678). Manual test runs succeeded, but the scheduled 15-min trigger kept reverting to the old broken code — root cause was the fix being saved in the editor draft but never clicked "Publish," so the live/scheduled version stayed on stale code. Published the workflow; confirmed the next scheduled run succeeded with real timestamps across all three tracked workflows. Lesson: Publish, not just Save, governs what scheduled triggers actually run. 2026.07.05.
ha-watchdog.sh Token Rotation
Confirmed the token was never actually hardcoded in the script itself — ha-watchdog.sh (/usr/local/bin/) sources /etc/ha-watchdog/token.env, which holds the HA_TOKEN variable used in the Authorization header. Generated a fresh long-lived HA token, updated token.env, revoked the old "HA Watchdog" token in Home Assistant, and re-ran the script to confirm clean authentication. 2026.07.08.
wview → HA Weather Push
push_wx_to_ha.sh cron script on the LinBPQ box parses the APRS formatted_data file every 3 min and pushes 10 sensors (temp, humidity, pressure, wind, rain) to the HA REST API. Root cause of the earlier "Entity not found" issue traced to an expired/invalid HA token (401) — the script's success log wasn't checking curl's actual HTTP response, so it silently reported success while pushes failed. Fixed with a fresh long-lived token; "Mitchell Weather (KINMITCH1)" dashboard card fully populated and confirmed working. 2026.07.19.
Retirement Terminal — HA Token Configured
Home tab "HA: TOKEN NOT CONFIGURED" indicator resolved — token linked and confirmed showing green. 2026.07.19.
Public Site Security Remediation
Found multiple HA long-lived tokens and an n8n API key hardcoded in public-facing HTML files across n9lya.com. All revoked and replaced; n8n workflows updated to use proper credential storage. RB5009 firewall rules added blocking the offending failed-login source IP (197.155.76.194) and closing API/Winbox ports (8728, 8729, 8291) from WAN on ether1. 2026.07.18.
Home Assistant CORS Fix
Resolved a CORS misconfiguration blocking cross-origin HA API calls from the n9lya.com LCARS pages. Confirmed clean requests, no more browser console errors. 2026.07.17.
Retirement Countdown Page
retirement-countdown.html built and deployed — live data confirmed, HA status dot showing green. 2026.07.17.
Continuum — Homelab Status Dashboard
homelab-status.html fully wired end-to-end. n8n workflow (Get Cluster Resources → Get Node Tasks → Merge/Append → Build Dashboard JSON → Respond to Webhook) all nodes green, production webhook live at n9lya.com/webhook/homelab-status. NODE/UPTIME/UPDATED summary strip populated via new node, uptime, and generated_at fields in the Code node. Assets/Services/Events/Log sections all showing live Proxmox data correctly. 2026.07.17.
Continuum Nav Links
Cyan CONTINUUM pill added to the nav bar on both n9lya-homelab.html and n9lya_deep_thought.html, matching style/position across pages. Confirmed live on both. 2026.07.17.
Live Radar Page — n9lya.com
GRLevel3/br1.jpg stays on mitchellwx.com as-is, not being retired. New RainViewer-based live radar page (radar.html) built and deployed to n9lya.com, centered on Mitchell/Lawrence County, linked into nav.
LinBPQ Forwarding Monitor — Repeat Alerts, Root Cause Found
Root cause confirmed via full Claude Code diagnosis: LinBPQ monitoring itself (SSH tail, web reachability, log parsing) passes 100% of the time — the failures were entirely isolated to the Gmail Alert node's OAuth2 credential, which had expired/been revoked. Not a connectivity issue at all. Re-authenticated the Gmail OAuth2 credential in n8n; confirmed connected. 2026.07.25.
Crypto Intel Pipeline — YouTube Node Failing, Resolved
Root cause confirmed: the "service refused connection" error traced to a one-time transient ECONNREFUSED network blip during a manual run on 7/22 — not an expired API key or quota issue as suspected. Ran a fresh end-to-end test: all 9 nodes passed clean, crypto-intel.html confirmed populated with live market data and weekly report. 2026.07.25.
RB5009 Login Failures — Real Root Cause Found (Not HAProxy Creds)
Original hypothesis (stale HAProxy admin creds) did not hold up. Actual root cause: RB5009 firewall rule *46 auto-blacklists any source IP for 24h on the second new SSH connection within that window — no failed-attempt condition at all, it fires on any new TCP connection to port 22 regardless of auth outcome. Explains the recurring pattern exactly (first connection always succeeds, next one always blocked). Rule flagged for future review/tightening; workaround for now is running all router SSH work in a single continuous session. 2026.07.25.
Vaultwarden Master Password Recovery + 2FA + Backup
Lost master password recovered via HAOS root shell + direct sqlite3 account reset on VM 112. Fresh account re-registered, TOTP 2FA enabled and confirmed. Full vault exported and saved to fireproof safe on USB. Along the way, found and fixed a real HAProxy bug — missing X-Forwarded-Proto https header on the vault/n8n/n8n-pwa backends was causing ERR_TOO_MANY_REDIRECTS. 2026.07.25.
SSH Key-Only Auth — CT 200 & Proxmox Host
Root SSH access on CT 200 (Claude Code) and the Proxmox host itself both moved to key-only auth using the proxmox_audit keypair. Independently verified with a BatchMode/publickey-only test connection before considering it confirmed. Full password-auth disable on Proxmox is on hold pending Ryan (KB8PMY) sending a working public key — a prior attempt to add his key failed silently (looks like truncation on paste), so he currently has no working key on file. 2026.07.25.
Fleet-Wide Password Rotation
Root passwords rotated across all 10 running LXC containers (100, 101, 102, 103, 104, 105, 106, 109, 113, 115). RB5009 REST API credential (n9lya account) and both CRS switch (SwOS) passwords also rotated and independently verified. All new credentials recorded in Vaultwarden via CSV import. VM/Windows machines (107 Ollama, 112 HAOS, 108/111/114 Windows) deferred — different rotation mechanism needed per machine type. 2026.07.25.
Full 9-Phase Homelab Optimization Audit
Claude Code ran a complete read-only + remediation audit across the fleet. Key findings: CT 105 Redis confirmed 100% orphaned (secured and left in place, candidate for future decommission); HAProxy "memory leak" and Uptime Kuma "I/O wait" were both false alarms from bad initial reads, downgraded to monitor-only; Nextcloud cron-mode and sysstat on CT 102/CT 115 both confirmed already correctly configured; AdGuard's 1GB/25% memory use investigated with no clear cause, monitor-only pending more data; backend IPs .24/.28/.42 confirmed via DHCP as Ollama/Nextcloud/LinBPQ with no unaccounted devices; RAID health checked clean (one Array B SSD at 78.87% life, spare on hand, swap not yet executed). 2026.07.25.
MikroTik Topology — Fully Confirmed & Corrected
Earlier assumption of a single-uplink daisy-chain (router→Bedroom→Radio Room) was backwards. Confirmed via each device's own SwOS port labels and the router's Winbox interface list: RB5009 → CRS .3 (Radio Room) → CRS .2 (Bedroom) → Proxmox, three 10G SFP+ hops in series. Still a single-path dependency end to end — flagged as a resiliency gap for future hardware (second uplink), not urgent, tabled pending hardware on hand. 2026.07.25.
n8n API Key — Workflow Visibility Restored
Fresh n8n API key generated after prior attempt failed with a stale-session "Unauthorized" error. Full workflow inventory now visible: 14 workflows (13 active/1 inactive), all credential types accounted for. Confirms CT 105 Redis has zero references anywhere across every workflow definition. 2026.07.25.
K9BBS Newsletter Workflow — Security Review + Pipeline Repair
Deeper look revealed the workflow was not actually fully built as believed — Groq (x2), Tavily, and Gmail credentials were completely unattached (null) on every node, and the DOCX/PDF conversion + second review-email branches were disconnected from the rest of the pipeline entirely (would have silently done nothing on a test run). Found and closed a real heredoc-injection risk in two file-write nodes — static "EOF" delimiter (which live Tavily search content could theoretically collide with) replaced with a dynamic per-run UUID token. Full pipeline re-wired end to end and verified via n8n's own CLI re-export. pandoc + libreoffice installed on CT 113 for the conversion nodes. See TASK 100 for remaining credential-attach + first live test run.
K9BBS Newsletter Workflow — First Live Test Run Successful
Groq, Tavily, and Gmail credentials attached to all 5 remaining nodes (Groq Chat Model x2, Tavily, Send Draft for Review x2). Full pipeline executed end to end for the first time. Vol. 1, Issue 8 (TEST) was generated and delivered by email with matching plain-text, DOCX, and PDF outputs in the standard HARDS newsletter layout. See TASK 113 for content review before approving a real issue. 2026.07.26.
Sparklight Speedtest Monitor — Native Timer + Accuracy Fix
Originally built as an n8n workflow, but n8n v2 disables the ExecuteCommand and LocalFileTrigger node types instance-wide by default (confirmed via n8n's own export:nodes registry showing the type absent from all 869 available nodes) — rather than override that security default, the job was moved to a native systemd timer (speedtest-push.timer) on CT 113 running every 6 hours. Along the way, found two unrelated bugs: the Ookla CLI repo had been pinned to an unsupported Ubuntu codename and silently fell back to installing the wrong tool (open-source speedtest-cli, which badly under-reports on fast connections — initially read 16↓/8↑ Mbps), and the real Ookla binary was crashing specifically under systemd due to a missing HOME environment variable. Both fixed and verified: the first unattended scheduled run reported 929↓/53↑ Mbps, closely matching a manual Ookla app test (939.97↓/52.07↑). The homelab reference page now shows this live, auto-updating speed data instead of a static estimate. 2026.07.26.
K9BBS Newsletter Workflow — Publish Blocker Fixed, Now Live
The recurring "this.getWorkflowStaticData is not a function" error on the "Determine Vol & Issue" node — which had been silently blocking the Publish button from ever becoming clickable — was tracked down and fixed for good. A full manual run afterward showed a clean green checkmark on every node with no errors anywhere in the pipeline. The workflow is now genuinely Published/Active, meaning the Monthly Schedule Trigger will fire on its own going forward instead of only producing output when manually executed. 2026.07.26.
CT 107 — Nvidia Driver Branch Swap, Complete
Full unhold, purge, and clean reinstall completed. Original plan targeted nvidia-driver-570-server, but that metapackage turned out to hard-depend on 580-server — plan adjusted mid-stream and nvidia-driver-580-server installed instead (fully supports Tesla P4/Pascal via proprietary kernel modules, confirmed via modinfo license check). Fixed a failing libnvidia-egl-wayland1 fetch by disabling the NVIDIA CUDA apt repo alias in favor of Ubuntu's own restricted mirror. Full 15-package 580-server stack held together post-install to prevent future version skew; old 570/565/550 remnants purged. Verified post-reboot: both Tesla P4s recognized, dual-GPU inference confirmed under real Ollama load (35%/47% util). Persistence mode enabled and made boot-persistent via root crontab (@reboot nvidia-smi -pm 1). Also used as the opportunity to run a full apt upgrade (47 packages) with the nvidia stack safely fenced off by the hold. 2026.08.02.
AdGuard Custom Rules File — Audit & Cleanup
Full syntax/logic audit of the custom AdGuard Home rules file. Found and fixed a real misconfiguration: four Handshake-related domains had comments saying they should be allowed but were missing the @@ prefix, meaning they were actually being blocked. Also removed a duplicate gm8bpq.no-ip.com entry and relocated a misfiled ve3bwm.ddns.net entry into the correct .ddns.net section. 2026.07.31.
Crypto Portfolio Tracker — Built & Deployed
Full tracking system spanning Robinhood, Crypto.com, and a SecuX V20 hardware wallet: verified Excel workbook (crypto_portfolio_tracker.xlsx, 4 tabs) with a corrected SHIB quantity (163,439,755, was off by orders of magnitude), an n8n workflow polling CoinGecko every 15 min and pushing crypto-data.json via SSH, and a new LCARS dashboard page (crypto-portfolio.html) matching the n9lya.com aesthetic. Along the way, identified and confirmed already-disabled three dusting/scam tokens airdropped to the SecuX wallet, plus excluded defunct SafeMoon (SFM) post-bankruptcy. Two items remain: wire the SSH credential into the n8n node, and verify 9 flagged CoinGecko IDs (Flux, Toshi, Pump.fun, Solayer, Coq Inu, Loaded Lions, Nyan Heroes, Access Protocol, MMFinance). 2026.07.31.
Security Audit Round 2 — Credential Rotation + Publish-Bug Sweep
Rotated an exposed YouTube Data API v3 key (found hardcoded in the Crypto Intel Pipeline) and a hardcoded HA long-lived token (found in HA Device Health Refresh) — the HA token rotation cascaded through the shared Header Auth credential across 5 workflows. Follow-up verification sweep found the n8n publish/versionId bug (activeVersionId not syncing) had left 2 more workflows — HA Energy Summary Refresh and Homelab Status-Continuum — silently running stale logic in production; both fixed and republished, confirmed clean across 6 consecutive runs. Also found and fixed AdGuard (CT101) DNS-sinkholing googleapis.com and api.tavily.com, which was silently blocking n8n's Google API and Tavily search calls — scoped allowlist exceptions added rather than broadly exempting the domains. HARDS Newsletter Vol.1 Issue 9 confirmed fully functional across all 18 nodes in a supervised test. Remaining: delete the old YouTube API key (see TASK 131), Ryan's SSH pubkey still not on file (see TASK 132). 2026.07.31.
Kutche Family Genealogy — Offline HTML
Self-contained, dark-themed single HTML file viewable directly on phone, no hosting required. Includes the Greek origins of Angelo Kutche (b. 1867, Arna near Sparta), the Zaharakos family story from Mistra through the founding of Zaharakos Confectionery in Columbus, IN (1900), the direct Demitre→Angelo→George→Jerry lineage chart, and a faithful transcription of George Kutche's last letter to daughter Elaina, written 1-2 days before his passing in March 1973. Respects the existing rule on living family members not appearing on public pages. 2026.08.01.
Kutche AI LLC — Two Ebooks Published
"Building With Claude" (field guide to human-AI project partnership drawn from real homelab work) and "Stop the Dataminers" (four-level telemetry-blocking guide). Both written, expanded with appendices, given custom Pillow-generated covers, and converted to PDF + EPUB via pandoc/wkhtmltopdf. Published under Kutche AI LLC as author.
Indiana LLC Setup Checklist
PDF checklist generated covering EIN, Articles of Organization, Operating Agreement, Biennial Business Entity Report, registered agent, bookkeeping, and sales tax steps for the new LLC.
Uptime Kuma — n9lya.com Monitoring Live
n9lya.com added to CT 115 Uptime Kuma monitoring, confirmed reporting 200 OK.
Gmail MCP Re-Authentication (CT 200)
Claude Code's Gmail MCP connector had dropped to needs-reauth. Re-authenticated via browser OAuth flow; Gmail now shows connected with 16 tools available, matching Calendar and Drive. 2026.08.01.
Open WebUI (ai.kutche.net) — Missing Models Restored
Ryan flagged only "Arena Model" showing in the model picker. Re-enabled and saved the full model set (llama3.2, llama3.2-vision, gemma4, gpt-oss:20b, lfm2) in the Open WebUI admin panel. 2026.08.02.
Four-Layer Weather Automation Stack — Complete
All four n8n layers confirmed live for packet radio delivery: Layer 1 (Drive Briefing), Layer 2 (Pre-Storm HWO/MCD monitoring), Layer 3 (Active NWS Alerts, zones INZ070/INC093, WFO Indianapolis), and the Barometer Trend flow (writing to /var/www/mitchellwx/barometer_trend.json). Resolved an n8n API access blocker between CT200 and CT113 along the way (JWT key generation + IP correction) and removed a legacy insecure webhook workflow found during the build. See TASK 125 — the SPC MCD-relevance parser in Layer 2 hasn't yet been tested against a real live MCD. 2026.08.02.
Review K9BBS Newsletter Test Draft (Vol. 1, Issue 8) & Approve for Production
Groq/Tavily-generated content needs a read-through for accuracy and tone before being used for a real issue. Once approved, drop the "(TEST)" designation and let the workflow run on its normal recurring schedule.
Ryan's Public IP — Needed for RB5009 WAN SSH Exception
Ryan wants real SSH access to the router from outside the LAN, not just Winbox. Needs his current public IP to add a scoped WAN firewall exception (rather than opening SSH to the whole internet). Asked, no word back yet as of 2026.08.26.
LinBPQ (N9LYA-8, 192.168.1.42) — Unexplained Post-Reboot accept() Stall
During the mitchellbpq.com WebSocket fix (TASK 214), the first reboot of this host came back with bpq32.service active, holding all its normal ports, and logging normally — but its accept() loop (confirmed live via strace) never serviced the web port, so nothing could connect despite the port showing as open. Manual restarts reproduced the identical stall consistently. A second full reboot resolved it cleanly with no configuration changes in between, and it's been stable since. Root cause undetermined. If the web terminal is ever unreachable right after a reboot even though LinBPQ looks healthy in systemctl/ps, a second reboot is the known workaround until this is actually understood.
Health Check Failed — linbpq-vm116
kiss-bridge services NOT active: dr78. First detected 2026-08-23T13:13:25-04:00.
DNS Cutover for the 4 Self-Hosted Domains
Once a site is fully verified working on CT 117 including HTTPS (TASK 201), point that domain's A record at the home network's public IP and confirm end-to-end. Keep the old commercial-hosting copy live in parallel as fallback rather than decommissioning it immediately, per the original plan — cut over one domain at a time as each clears verification.
HTTPS / Let's Encrypt for the 4 Self-Hosted Sites on CT 117
Deferred until all 4 target sites (ainews, cryptonews, tripza, affiliatepathways — see TASK 195/197/198/199) are imported onto CT 117, then set up in one pass rather than one certificate at a time. Required before DNS cutover (TASK 202) since the sites' own siteurl settings already point to https://.
tripza.us.com — Migrated to Self-Hosted CT 117, Verified Live (2 Real Bugs Found and Fixed)
Third of the 4 migration-target sites, and the first to actually surface real problems rather than a clean import. Found the backup's plugins/themes/uploads zips were each wrapped in their own top-level folder (unlike cryptonews/ainews), so the first extraction pass double-nested everything into e.g. wp-content/plugins/plugins/ — re-extracted correctly into wp-content directly. After that, the site returned a blank 200 (no visible error) instead of rendering; turned on WP_DEBUG_DISPLAY temporarily to surface the real cause: MailPoet's bundled Carbon date library throws an uncaught TypeError on PHP 8.2 (the exact PHP-version mismatch theme from the original hosting audit — this site's old host ran PHP 7.4.33, CT 117 runs 8.2). Deactivated MailPoet cleanly via the active_plugins option (also caught and left really-simple-ssl/really-simple-ssl-pro deactivated the same way, since CT 117 has no HTTPS yet — TASK 201). Re-verified afterward with debug output turned back off: clean 200, exact title match ("cheap flight deals HQ") against the live site. Only affiliatepathways.us.com still to go — its uploads backup turned out to contain a real backdoor, see TASK 208. 2026.08.22.
ainews.us.com — Rotate Admin Password
Following the confirmed rogue-account compromise (see TASK 190), the real admin password should be rotated as standard post-incident hygiene. Claude Code's own safety classifier blocked the automated password-change attempt as too high-risk to run unconfirmed — needs either explicit re-confirmation to retry, or a manual change via wp-admin → Profile → New Password.
Website Fleet — Minor Housekeeping Items
adsubmission.net has 4 backup/migration plugins active at once (BackupBuddy, All-in-One WP Migration, UpdraftPlus, Duplicator) — redundant, worth consolidating to one. takemethere.us.com has dead SiteGround migration plugins (sg-security, sg-cachepress, siteground-central, mojo-marketplace, all inactive) safe to delete outright. booksrus.us.com's "Restore Paypal Standard for WooCommerce" plugin artificially keeps a deprecated PayPal integration alive — confirm it's actually still needed before a surprise shutoff on PayPal's end breaks checkout with no local warning.
adsubmission.net — Unfamiliar Second User Account
abigailcallister (Subscriber), registered under an unrelated name and an unfamiliar email domain (profmalachistoltenberg1640@mailbab.com). Low risk on its own since it's Subscriber-level, not Administrator, but worth a quick confirm — could be spam self-registration if the site allows open signups, or worth asking Kari if she recognizes it.
petworld.us.com — WP File Manager/WP Automatic Keep-or-Remove, Admin Username Rename
Both deferred pending coordination with Kari: whether the WP File Manager plugin and the (currently working, unlike affiliatepathways's) WP Automatic install are still actually needed, and replacing the "admin" username with a real named account.
adsubmission.net — WooCommerce Memberships License Needs Reconnecting
Update attempt returned "Update package not available" — same root cause as affiliatepathways's WP Automatic issue (a premium extension without a valid license attached to fetch its package), but this one isn't a known vulnerability, just blocked from updating.
takemethere.us.com — WooCommerce Update-Check Stuck at v7.6.2
Reports "already at the latest version" even after a forced recheck (current is 11.0.1). Most likely this server's own outbound connection to the WordPress.org update API is being filtered — this site runs Wordfence, Sucuri Scanner, and All-In-One WP Security simultaneously, any of which could be blocking it. Flagged rather than forced through.
ainews.us.com / cryptonews.us.com — Hosting Account Suspended (64.20.40.252)
Both domains redirect straight to cPanel's suspended-account page — the host shut off the entire account, no public reason given. Not fixable technically; needs a support ticket/call to whoever owns that account. User has already emailed Kari about this.
cPanel Access Still Broken (198.251.88.6) — Blocks PHP 7.4 EOL Upgrade + Real WP-Cron Fix
The password on file for the petworld.us.com account's cPanel still returns a genuine invalid_login (confirmed byte-exact, not a transcription error) on both UAPI basic-auth and the form login. Blocks the two biggest remaining fixes across all 6 sites on this account: upgrading the account-wide PHP 7.4.33 (end-of-life since Nov 2022, confirmed via Site Health on both petworld and booksrus) via MultiPHP Manager, and adding a real system cron entry to permanently fix petworld's broken WP-Cron (currently only worked around by manually hitting wp-cron.php). Needs a working password or API token from the host.
URONode Application Shortcuts (FBB, URONOD, DX) and MSYS — Still Not Reachable from LinBPQ
Following the N9LYA-2 fix (TASK 174), swept the rest of URONode’s locally-hosted identities for the same missing-route bug and found N9LYA-4 (FBB), N9LYA-6 (DX Spider), N9LYA-7 (URONode’s own NET/ROM node identity), and N9LYA-9 (MSYS) all had the same gap — added matching ax25udp routes and LinBPQ-side MAP entries for all four, same fix pattern as N9LYA-2. Results are mixed and not yet resolved: N9LYA-4 and N9LYA-7 fail before any packet even leaves LinBPQ — both are wired up in bpq32.cfg via APPLICATION shortcuts that connect through a NET/ROM alias first (C 3 BBSLYA s,n9lya-4,254 / C 3 IN105 s,N9LYA-7,254, alias-then-via-path, not a direct destination) — even typing the bare callsign directly still surfaces "Failure with BBSLYA" in LinBPQ, meaning something in LinBPQ’s own alias/NODES resolution is failing before a UDP packet is ever built, not a URONode-side problem. N9LYA-9 (MSYS) is a different failure mode: real traffic does reach URONode’s kernel AX.25 stack (confirmed via live syslog), but never completes — axports describes N9LYA-9 as specifically tied to interface ax2, while ax25udp only ever injects frames into ax1 (its device is hardcoded to ttyq1); if MSYS is bound to listen only on ax2 rather than any interface, ax1-injected frames would never reach it. N9LYA-6 has the route added but is untested. Next step for 4/7 is likely a strace-level dive into LinBPQ’s own connect/alias resolution (same technique that root-caused the SCSTracker segfault, TASK 171); next step for 9 is confirming exactly which interface MSYS actually listens on and whether it can be made to listen more broadly, or whether a second AXUDP-style bridge tied to ax2 is needed.
SYMEK TNC — Confirmed Software Chain, Radio Still Not Producing RF
After building the full SYMEK bridge (see TASK 170) and confirming a real over-the-air TX once via the 220MHz path, the same test stopped producing any RF response — traced through several real bugs along the way (a socat process on URONode that died with no supervisor and never came back, and dosemu itself crashing when its PTY hung up mid-session, both since fixed as part of TASK 170’s respawn-loop), but even after fixing all of them and confirming every layer independently — FlexNet startup clean, PTY bridge alive, real bytes crossing the TCP link (confirmed via socket byte counters), ser2net writing to the correct physical device with zero errors — the SYMEK unit itself isn’t responding on RF. Two full power cycles (wall-wart alone, then wall-wart + USB together) didn’t bring it back either. Since every software layer is now provably correct, this points at something internal to the SYMEK unit itself — may need a longer power-off than a quick cycle, a SYMEK-specific reset sequence from its manual, or genuine hands-on bench diagnosis rather than more remote troubleshooting.
Radio Bridge (XPS) — Add 2 More Devices
Two more devices to add to the XPS radio bridge (see TASK 159-164). Only 2 USB ports free right now — waiting on USB extenders to arrive before physically wiring them in. Target: about a week out from 2026.08.20.
Self-Report Consistency Experiment (Claude) — Persistent SSL Error
"Query Claude" node shows a recurring "SSL Issue" banner in the n8n UI even after the key rotation was confirmed working via direct curl (clean 200 response, real completion text, right off CT 113). OS cert chain, system clock, and Node's own root cert store all checked clean earlier in the same session. Likely a stale n8n UI/node-state display issue rather than a real auth/network problem, but never confirmed — set aside for now, revisit with fresh eyes.
Self-Report Consistency Experiment (Ollama) — Recurring Timeout
Workflow intermittently times out; the obvious suspect (n8n's 60s HTTP timeout) was ruled out by Claude Code as the actual cause. Real root cause still unknown. Same general family of issue as the earlier-logged Ollama node oddity (44 minutes for one prompt via n8n's native node vs. ~20s via direct curl from the same container) — worth investigating together next time.
HAOS (192.168.1.21) — SSH/Console Access Still Not Working
Attempted during the CT 200 SSH provisioning session (see TASK 138); Proxmox serial console (qm terminal 112) kept rejecting password attempts and never dropped into a usable shell. Abandoned for the night as bonus/non-blocking scope. Next attempt should go through the Home Assistant web UI's Terminal & SSH add-on instead of fighting the serial console blind.
SPC MCD-Relevance Parser — Untested Against Live MCD
Layer 2 (Pre-Storm HWO/MCD monitoring) is live but the MCD-relevance parsing logic hasn't been exercised against a real Mesoscale Discussion yet — none were issued nationwide during the build window. Needs verification next time SPC issues one.
LinBPQ — New Periodic Outage Pattern (~73–76 min)
192.168.1.42 (LinBPQ + mitchellbpq.com) both drop together on a roughly 73–76 minute cycle, timing out at exactly 24000ms and recovering within 1–2 min. Distinct from the previously-fixed 61:47-minute AMPR forwarding cycle (which is disabled). Root cause not yet identified. Has not recurred since yesterday's audits — Vaultwarden also cleared at the same point and hasn't repeated. No faults seen in Uptime Kuma since 19:55 yesterday. Leaving open pending more confirmation before closing.
Dashboard Mobile Table Overflow — 3 Pages
Solar 7-Day History, Automations & Workflows, and Device & Sensor Health pages all have table content cut off on the right edge in mobile view.
Zero-Solar-Production Reading, Aug 1
solar-energy.html showed a zero-production reading at 5:17 PM on Aug 1 despite the n8n SolarEdge Data Refresh workflow reporting a successful run. Needs investigation — possible SolarEdge API edge case vs. a real inverter event.
Delete Old YouTube API Key
Old "API key 2" (created Jun 26) in Google Cloud Console is superseded by the new rotated key (see TASK 117) and still pending deletion.
Crypto Intel — YouTube Node Retry Logic
The transient ECONNREFUSED failure itself is resolved (see TASK 96), but the node still needs Retry On Fail (2–3 retries) and a failure-logging step added so a future one-off blip doesn't require manual diagnosis again.
FCC/Frontier — IURC Escalation
IURC letter and community fill-in complaint template are drafted. Still need to actually send the IURC letter and post the template to Facebook for Mitchell residents.
MikroTik Dashboard — Ghost Interfaces
MikroTik Pro app shows sit1 / back-to-home-vpn dropping from the interface list or flickering wrong status. Verify against /interface print via Winbox or SSH as ground truth.
n8n → Ollama Bridge Timeout
~30s timeout cap persists on the public path. Leading suspect: HAProxy timeout on the port 8092 listener. Next step: direct curl bypassing HAProxy to isolate.
DICE / KV Cache Experiment (CT 107)
measure_kv.sh handed off via scp — run it on CT 107 to get real VRAM-per-1K-token numbers on the dual Tesla P4s.
Financial Prompt Toolkit
Wealth Mindset Rewire template build paused, awaiting answers to: (1) recurring money blocks, (2) specific financial goal. Then continue with Debt Destruction Engine, Budget Optimization, Income Stream Mapper.
Kutche AI LLC Site Deployment
LinBPQ webroot confirmed Apache2/plain HTML, no PHP — no index conflict. Just needs index.html dropped into the webroot to go live.
PWA/APK App Kit — Remaining Sites
Done: n9lya.com, ai.kutche.net, n8n.kutche.net, vault.kutche.net. Remaining: mitchellbpq.com, mitchellwx.com, hfskipnet.net, indianapacketcouncil.com, cloud.kutche.net, home.kutche.net. Kari's 8 sites excluded until SSH is used (see TASK 24).
Netmap TABLE View — Dedup Bug
CRS .2/.3 showing as duplicate entries in the netmap TABLE view. Fix pending.
Solar Dispute — Confirm Escalation Sent
Case #6982593. Confirm the escalation letter to SolarEdge has actually gone out. Goal remains: remove GST entirely, transfer ownership/admin to Jerry or Joshua Ayers.
mitchellwx.com Radar — Regression Check
br1.jpg confirmed NOT updating reliably as of July 6, despite the earlier TASK 47 fix. Next session: check actual file mtime on /var/www/Kutche/br1.jpg against GRLevel3's FTP upload schedule.
Phone DNS — AdGuard DoH via HAProxy
Migrated phone DNS from NextDNS (old profile 8c1687) to AdGuard Home DoH. The HAProxy/DoH implementation session itself is still pending — deferred to a follow-up morning session.
n8n Workflow Failure Cluster — SolarEdge/HA Summary
Executions #4437–4439 (SolarEdge Data Refresh + HA Energy/Device Summary) failed ~3:23 AM July 14 — likely a transient shared dependency. Logs still need review to confirm root cause.
ai.kutche.net — Reconnecting Popups Over Cellular
Mid-response "reconnecting" popups on cellular. Check CT 106 HAProxy (192.168.1.29) timeout client/server settings for the ai.kutche.net backend.
RB5009 SSH/SCP — Kex Exchange Failures
SSH/SCP from Termux to the RB5009 (n9lya account) fails with "kex_exchange_identification: read: Connection reset by peer" even after strong-crypto=yes and scp -O. Needs ssh -vvv diagnosis, or toggle SSH service off/on for strong-crypto to take effect. Blocking a firewall rules export after adding rules for TASK 93.
n8n Monthly Site-Review — First Manual Run
n9lya-monthly-review.sh pipeline built and wired in. Needs its first manual run — pending robots.txt being re-enabled.
CT 104 (MariaDB) / CT 105 (Redis) — Provisioning Decision
Both containers are fully provisioned but confirmed unused by anything in the current stack. Decide: intentional headroom, or leftover from an abandoned migration worth decommissioning?
CT 110 (uronode) — Confirm Normal State
Container was started, then stopped again during today's audit/rotation session for an unrelated reason and was intentionally excluded from the password rotation scope. Confirm it's back to its normal expected state.
CRS .2 SFP1 Port — Cosmetic Relabel
Port currently self-labeled "CRS 1 Bedroom" (confusing/self-referential naming from initial setup). Low priority — rename to something like "Uplink to Radio Room" for clarity next time someone's in the SwOS UI.
LLM Benchmark Tracker Dashboard
Artificial Analysis API — intelligence / price / speed. Talk through scope first, then maybe build (possible n8n tie-in).
Cloudflare Proxy Rollout — Fix SSL Mode, Flip Records One at a Time
All 8 domains are currently 100% DNS-only (no Cloudflare proxying at all — real WAN IP exposed on every record, no DDoS/WAF coverage). To re-enable proxying safely: first set SSL/TLS mode to Full or Full (strict) in the Cloudflare dashboard (Flexible is what caused the TASK 68 vault redirect loop). HAProxy is already terminating real HTTPS, so this should be a drop-in change. Then flip records to proxied one at a time, low-stakes first (autodiscover/mail CNAMEs → ai.kutche.net → save vault.kutche.net for last with full Android retest). Skip mitchellbpq.com and any BPQ32/AXIP packet-radio subdomains entirely — raw TCP protocols aren't proxyable on Cloudflare's free tier. Target: complete before the n8n v3.0 Docker migration work begins (see TASK — n8n Docker migration, targeted October 2026).
Claude Code Skills Install
Stand up the /humanizer + /seo commands.
mitchellwx.com Redesign
Rebuild the weather page in the LCARS / n9lya style, same data feed.
ACA Marketplace Re-enrollment
Call in August for a Sept 1 effective date — bridges retirement to SS start.
LLC Branding Refresh
Update branding across the site network under the new LLC, closer to launch.
Clonezilla — Full System Backups
Run Clonezilla on all systems the day after retirement. Full disk images before any post-retirement hardware changes/consolidation begins.
IC-7100 D-STAR Memory Channel — Program via Tablet + Cable
Values ready: RX 442.650 MHz, +5.000 MHz offset, DV mode, MY N9LYA, UR CQCQCQ, RPT1 N9LYA B, RPT2 N9LYA G — matches the N9LYA hotspot's own D-Star identity (see TASK 181). Reflectors to try first: REF030C, XRF001A, XLX011, DCS001C.
New LLC Under Retha's Name
File the new entity post-retirement.
Tax Hold-back Elections
Set IRA withholding (~15–17%) and SS W-4V (~10%). CPA sit-down optional.
N100 Mini PC Swap
Replace the Dell 1600SC in the radio room. ~$231/yr savings.
Heat Pump Water Heater
~$168/yr savings + the Shelly Pro 2PM solar automation tied to it.
3rd Tesla P4 → Plex Consolidation
Move Plex onto Proxmox and retire the 8900 PC.
Radio Room Power Audit
Track down the ~246W average mystery load.
RuView ESP32-S3 Hardware Check
WiFi CSI sensing project — confirm you've got the right ESP32-S3 board before building.
SolarEdge API Key Rotation — Ask Joshua (Blue River Solar)
Found the live SolarEdge monitoring API key sitting in orphaned n8n workflow-export JSON files in the public n9lya.com docroot (solaredge-refresh.json, solaredge-refresh-v2.json, and duplicated in ha-energy-refresh.json) during the 2026-08-24 site exposure audit. Confirmed the key is still live and returns more than production data — a query back gets your real name, home address, and GPS coordinates. Files deleted immediately (2026-08-24), closing the public-discovery vector, but the key itself is still valid until replaced. Your monitoring-portal login doesn’t show an API-key/admin section — likely because owner-level access landed with Blue River Solar during the July 24 installer transfer from Green Home Systems, not something you’re missing. Ask Joshua to generate a new key; once in hand, it needs swapping into the n8n credential used by the SolarEdge Data Refresh and HA Energy Summary Refresh workflows.
WeeWX Migration
Retire wview / WMR968 for the wmr9x8 driver once LinBPQ is stable.
Water Reclaim Tank
ESPHome ultrasonic level sensor + pump relay → HA automation.
RAG LXC Stages 4–7
Stand up the LXC, come back with the /ready endpoint output.
Hermes Agent — Proxmox LXC
Stand up Hermes Agent as a container on Proxmox, connect to the existing Ollama instance. Single curl install on Debian LXC.
solar-system.html Three.js Fix
Portrait-mode label alignment bug. 3–4 fix attempts so far — needs a fresh approach.
RB5009 Proxmox Hardening + DMZ Review
Core RB5009 hardening is done (admin disabled, n9lya user, Winbox LAN-only). Proxmox hardening and DMZ review still open.
Xiaomi BLE Sensors — Recent Degradation
Radioroom, Bedroom East, Living Room, and Dining Room BLE temp/humidity sensors worked fine for about a year but have recently started having issues. Confirmed entity IDs: ATC 7AC8, ATC B0B4, ATC E241 (BTHome), plus Radioroom FF8D and Bedroom East 2098 (Xiaomi BLE). Needs investigation (BLE proxy/adapter, HA update, battery, interference) before wiring into ha-devices.html.
Battery Sensor Panel — Devices Page
Panel on ha-devices.html currently hidden/empty. Will populate once the Xiaomi BLE sensors (TASK 61) are troubleshot and wired in — they report battery % alongside temp/humidity.
RB5009/CRS Switch Temps — HA Exposure
Confirmed via a full entities-page sweep: the network gear does not currently expose any temperature entities into HA — nothing under a switch/router device found alongside the room sensors. Decision needed: pursue an SNMP integration path, or drop the idea in favor of room/thermostat-only tracking on ha-devices.html.
Devices Page — Uptime Column
Uptime column on ha-devices.html is always "n/a." Needs either an HA uptime sensor or a different data source to populate meaningfully.
Claude Cowork Mobile + Scheduled Tasks
Evaluate for homelab/n8n monitoring — scheduled tasks watching workflows and flagging failures automatically.
Fable 5 — Usage-Based Credits
Fable 5 moves to usage-based credits after July 12. Review cost implications for ongoing use.
Fable 5 as Advisor + Sonnet 5 Executing
Compare a Fable-5-advises / Sonnet-5-executes workflow split against the Antigravity plugin eval.
DashBench — Kimi K2.6 + Fable 5
Combo benchmark idea for code review evaluation.
AdGuard Status Page
adguard-matrix.html built (Matrix digital-rain theme) but not yet live — on hold pending n8n workflow integration to feed live AdGuard stats to the page.
Uptime Kuma Status Page
New LCARS page for CT 115 Uptime Kuma (192.168.1.111). Build after AdGuard page. Theme TBD.
Homelab Page — Live Status Dots
Section-header dots on n9lya-homelab.html (WAN/ISP orange, Core Router green, etc.) are currently static color-coding, not live health indicators. Idea: wire them to actual reachability checks (ping/HA sensor) so color reflects real up/down status per section.