lakemonster · internal viewer go.7.fish:None

Issues 16 open · 24 resolved · source: docs/ISSUES.md · Last groomed: 2026-09-08

The tracker is docs/ISSUES.md in the repo — edit it there (or ask the assistant to log an issue) and this page follows. One entry per issue: What / Impact / Next step.

ISS-1 · Lake 9822 has no valid shore met point — `FAIL` in met_validation
What: `_shore_met_point` finds no land cell within 300 m of lake elevation among the bbox-edge/midway candidates (steep canyon reservoir). Air-temp forcing falls back to the centroid water cell.
Impact: GLM forcing + displayed air temp for 9822 may carry the weather model's internal-lake-temp contamination.
Next: widen the probe ring (more candidates along the shoreline polygon) or relax the elevation tolerance for canyon lakes; re-run `scripts/validate_lake_met.py 9822`.
ISS-2 · 77 lakes `NO_BBOX` — met validation can't run
What: fleet met validation (2026-08-15: 117 OK / 77 WARN / 1 FAIL) warns on lakes without `data/processed/{sid}/bbox.json` — partially onboarded lakes.
Impact: low today (those lakes ship no forecast products), but they'll onboard eventually.
Next: the `met_validation` workflow step requires `bbox_compute`, so full onboarding covers this; optionally run `bbox_compute` + validation for the 77 to pre-clear them.
ISS-3 · 64 lakes have no satellite temp anchor
What: no MODIS calibrated series and no recent ECOSTRESS/Landsat scenes — `latest_anchor()` returns None (audit 2026-08-15).
Impact: their GLM headlines are unverifiable and uncorrected; only the climatology display cap protects them (several showed 33–36 °C raw before capping).
Next: extend the MODIS extract/calibration pipeline to these siteids (pooled global model already allows zero-pair calibration).
ISS-4 · Utah Lake far-tail heat-wave peaks still read a few °F hot
What: GLM afternoon peaks 5+ days out exceed the observed envelope under calm heat waves; the climatology soft cap compresses but doesn't eliminate.
Impact: cosmetic-to-moderate on the forecast chart tail.
Next: revisit calibration with wave-breaking/convective mixing params, or fit sw_factor seasonally; re-run `scratchpad`-style level+diel sweep after the heat wave passes with fresh MODIS verification.
ISS-5 · Echo deep-hole hypolimnion disagreement
What: tiled tier-2 hindcast holds ~6–8 °C at 9 m+ in the deepest hole; the live whole-lake forecast says bottom ~17 °C.
Impact: cosmetic — visible in the 3D cross-section only (~65 of ~9k points).
Next: compare against any deep DWQ cast; likely the tiled deep column under-mixes (coef_mix_hyp for deep tiles).
ISS-18 · 18:47 UTC GLM cycle timed out daily (Sep 2–8) — cap raised, slowness unexplained
What: `lakemonster-glm-cycle.service` had `TimeoutStartSec=30min`. The 00/06/12 UTC cycles finish in 22–28 min; the 18:47 UTC (12:47 MDT) cycle hit the cap and was SIGTERMed on Sep 2, 3, 5, 6, 7, 8 — status 143, and because the pool only prints per-lake JSON after all workers finish, the log shows *nothing* for the killed run (232 per-lake logs were touched in the window vs 87 for the morning run, so it was running, just slower).
Impact: no fresh GLM forecast for ~6 h every afternoon; the +30 min anchor correction ran against stale warm state.
Fix (2026-09-08): `TimeoutStartSec=55min`; anchor-correct timer moved to +60 min (`01,07,13,19:47 UTC`). `systemctl --user reset-failed` applied.
Next: find why the 18 UTC cycle is slower (metro open-meteo HRRR 18Z ingest? concurrent `forecasts`/`bloom-tags` timers at :55/:12?). Run the pool with `-v` once so per-lake `runtime=` lines land in the log, or check `data/logs/glm_cycle/*.log` mtimes in the 18:47–19:40 UTC window.
ISS-19 · Lake Merwin #519 `terrain_rgb_sample` fails the brightness gate on every retry
What: all 8 probed S2 candidates have lake-area mean brightness 0.128–0.136, below the 0.15 floor (`derive_terrain_rgb.py --min-brightness`), so the reonboard workflow defers 3600 s and repeats (2× so far on 2026-09-08; it *passed* on 2026-09-02).
Impact: #519 registry workflow never completes; one deferral log line per hour.
Next: per-lake brightness floor override in lake_config, or accept the brightest candidate when all fail by a small margin (<0.03); or widen the candidate window (`max_candidates`).
ISS-8 · Anchor dead band leaves ±1.5–3 °C persistent bias
What: confidence-scaled nudge thresholds (1.5 °C med/high, 3 °C low) mean a stable model bias just under threshold persists (Rockport pattern). Display-time obs bias corrects the shown value; the model state stays biased.
Next: consider a slow integral-style nudge (e.g. 0.3 °C/cycle toward anchor regardless of threshold) so state converges without churn.
ISS-9 · Legacy UTC remnants until natural re-push
What: timestamp localization ships on change-detection: objects that rarely change (and pre-change day cubes/tier2_3d cached files) keep UTC strings until re-rendered/re-pushed.
Impact: shrinking daily; day cubes refresh on the hourly timer.
Next: none needed unless a stale page is user-visible; spot-check in a week.
ISS-11 · Confirm fleet tier-2 rebuild completion
What: the full-fleet tier-2 rolling rebuild (started early 2026-08-15 evening) runs with a 4 h budget + requeue; leftovers heal via nightly timer/watchdog.
Next: check `data/logs/tier2_rolling.log` + per-lake logs for failures; re-run stragglers with `--single`.
ISS-12 · lakemonster-web restarts still drop in-flight requests
What: deploys restart uvicorn (`--no-reload`); each restart is a brief 502 blip behind go.7.fish. Mitigated 2026-08-15: peer session added `TimeoutStopSec=10` (was ~2 min drain) and the homepage now cold-serves in 15 ms from a persisted snapshot — but the blip itself remains.
Impact: seconds of 502s per deploy; only matters when deploying during active use.
Next: graceful reload (uvicorn workers + SIGHUP) or a blue-green pair behind the proxy; batch code changes into single restarts meanwhile.
ISS-13 · Learn per-lake diel air-temp bias from consensus residuals
What: conditions-now air temp is a median of 3 estimators (station obs / metro-HRRR shore / WeatherKit); per-source residuals vs the consensus now accrue in `derived/air_temp_residuals.jsonl` per lake. Mountain-valley lakes show systematic hour-of-day model bias (inversion cold pools: metro +2–3 °C mornings — fleet comparison 2026-08-16: median agreement 1.3 °C, valley outliers to 4.7 °C).
Impact: consensus already caps single-source errors; learned diel corrections would tighten the last ~1–2 °C.
Next: after ~2 weeks of residuals, fit per-lake per-hour-of-day offsets for the metro source (extend `derive_met_bias.py`) and apply before the median.
ISS-14 · WWTP nutrient-loading context from EPA ECHO
What: EPA ECHO REST + DMR services expose the 7 major CWA dischargers around Utah Lake (and equivalents fleet-wide) with monthly effluent flow + nutrient DMRs.
Impact: would add seasonal nutrient-loading markers/context to the Algae page (monthly resolution — not per-scene attribution).
Next: small fetcher caching per-lake NPDES DMRs; band markers on the algae graph; correlate seasonal loading vs bloom onset.
ISS-17 · Sea-level/tidal lakes never promote — `bed_elev_navd88_5m.tif` skipped (FIXED)
What: `bathy publish` only wrote the NAVD88 bed-elevation raster when `pool_elev_m > 0`, but `promote_to_bathymetry` hard-requires it. Any water body whose surface elevation is ≤ 0 or whose lakemon record stored `elevation_m` as `0`/null (tidal/coastal/low ponds) got its bed DEM silently skipped and then permanently failed promote with the misleading "publish step must have failed". Pre-existing since ≥2026-08-02.
Affected (from logs): St Johns River #3342 (Jacksonville tidal river, surface ≈ 0 m NAVD88 — parked/retried for a month), Stink Lake #13588, Crystal Springs #13649, Goode Ski Lake #14886.
Fix (2026-09-08, `~/bathy` `cli.py` publish): treat surface elevation as an `Optional[float]` — `None` = genuinely unknown (skip), any *known* value including ≤ 0 produces the bed DEM (`bed = surface − depth`, correctly landing below the datum). A stored `0` is treated as missing and falls through to the authoritative USGS EPQS lookup. Validated: #3342 republished with `bed_elev_navd88_5m.tif` (surface 0 m, bed 0 to −11.9 m); unparked to complete. Whole class fixed.
Follow-up: `promote_to_bathymetry`'s error text ("publish step must have failed") is misleading when publish succeeded but legitimately had no elevation anchor — reword, and consider a non-parking "no NAVD88 anchor" skip for genuinely unknown-elevation lakes.
ISS-16 · `get_shape` API serves degenerate geometry for some lakes (Chelan #90, Pickwick #2468)
What: `vps.lakemon.com/.../get_shape&siteid=<id>` returns valid JSON but a degenerate polygon — Chelan #90 ≈ 105 m × 82 m (a real 80 km fjord), Pickwick #2468 ≈ 30 m — instead of the true lake extent (confirmed consistent across retries 2026-09-07). Left unchecked this yielded a 1–2 tile "chart" and an all-nodata DEM that crashed `fuse` downstream (that was the real cause of both lakes' bathy parks, on top of the ISS-15 OOM history they also had).
Not the deep_water shape-refresh pipeline: zero entries for 90/2468 in `shape_refresh*.log` — the S2 refresh never touched them, so the bad geometry lives in the VPS lake database from the original seed/import.
Client mitigation (2026-09-07, `~/bathy` `sources/lakemon_api.py`): `fetch_lake` now (a) retries `get_shape`/meta on empty/non-JSON bodies (the other intermittent failure mode — an empty body → JSONDecodeError), and (b) rejects a degenerate geometry (bbox < `BATHY_MIN_AOI_BBOX_M`, default 200 m, or empty) with a clear error, so it fails fast at `fetch-aoi` instead of publishing garbage. `navionics-dem` also hard-fails on zero valid pixels (see ISS-15).
Audit (2026-09-08): `scripts/audit_lake_shapes.py` sweeps `get_shape` for every lake in the bathy queue and classifies OK / DEGENERATE / MISSING_GEOMETRY / NOT_FOUND / ERROR → `docs/triage/shape_audit.md`. Current active-queue result (25 lakes, 21 OK): **4 bad shapes, all need a server-side fix on vps.lakemon.com:** - #90 Lake Chelan — DEGENERATE (bbox ≈ 70×82 m) - #496 Lake Wallula — DEGENERATE (bbox ≈ 82×63 m) - #2468 Pickwick Lake — DEGENERATE (bbox ≈ 30×30 m) - #16998 Lake Higgins — NOT_FOUND (HTTP 404, no shape at all)
Next (server-side, needs operator): regenerate the 4 shapes above in the vps.lakemon.com lake DB (create one for #16998). The client guards make all 4 fail fast + clean until then. To cover lakes not currently in the queue, extend the audit to the full onboarded fleet (it accepts `--siteids`, or wire it to the all_lakes list).
ISS-15 · Big-lake `navionics-dem` needs >124 GB — OOM loop crashed the host
What: the bathy CLI's `navionics-dem` (sometimes `fetch-navionics`) step allocates ~124 GB+ on very large lakes (Superior, Okeechobee, Oahe, Sakakawea, Great Salt Lake, Ozarks, Hartwell #1029, Guntersville #1186, Chickamauga #959, …). The kernel OOM-killed it 1,889 times between Aug 15 and Sep 7 2026; each 6 h-backoff retry re-ran the same doomed job, and with ~20+ such lakes rotating the host OOMed every ~15–20 min until the desktop froze on 2026-09-07 and needed a reboot.
Impact: host instability (the Sep 7 "crash"), the dedicated bathy lane spends most of its time on jobs that can never finish, and those lakes never get bathymetry.
Mitigation (2026-09-07): systemd drop-in `~/.config/systemd/user/lakemonster-bathy-dispatcher.service.d/memory-cap.conf` sets `MemoryMax=64G` + `MemorySwapMax=0`, so a runaway job dies inside its own cgroup in ~2 min instead of starving the host.
Mitigation 2 (2026-09-07): max-attempts park — after `LAKEMON_BATHY_MAX_ATTEMPTS` (default 10) failures the dispatcher parks the lake (status `parked`, queue score +inf; seeders can't resurrect it) and kicks `lakemonster-bathy-triage.service`, which runs a read-only headless Claude investigation and writes `docs/triage/bathy_<sid>.md` (2 h sweep timer catches stragglers). Review parks with `scripts/dispatch_bathy_queue.py --list-parked`; requeue after a fix with `--unpark <siteid>`.
Fix (2026-09-07, in `~/bathy`, uncommitted): size-aware zoom via a shared `_memory_capped_zoom()` helper in `cli.py`. BOTH steps apply it: `fetch-navionics` estimates mosaic pixels from the AOI before fetching, and `navionics-dem` caps its zoom search too (a stale oversized z17 chart from an old attempt must not be loaded — that was the gap that re-OOMed Eufaula/St. Clair). Auto-drops the zoom (floor z11) to stay under `BATHY_MAX_MOSAIC_PX`. **Budget calibrated by measurement:** the DEM step's peak RSS is ~66 bytes/mosaic-px (siteid 307: 0.78 Gpx → 51.2 GB). Default budget **0.65 Gpx ≈ 43 GB peak**, ~20 GB headroom under the 64 GB cgroup cap. (An initial 1.5 Gpx guess was too high — Sakakawea #231 OOMed at z15/1.35 Gpx before recalibration.) Resulting zooms: Sakakawea z14, Superior z15, Okeechobee/Ozarks z16, Erie z15. Fusion terrain preview falls back across z17..z14; raw tile dict freed before the GeoTIFF write. - **Watch out (cost me a cascade 2026-09-07 21:45):** the helpers must sit ABOVE the `@main.command("fetch-navionics")` decorator block. If placed between the decorators and `def fetch_navionics_cmd`, the decorators bind to the helper instead, silently turning `fetch-navionics` into a no-op that fails EVERY lake with rc=1. `bathy fetch-navionics --help` exiting 0 is the quick smoke check.
Validated 2026-09-07 22:19: Lake of the Woods #307 (12.3 Gpx at z17, the largest yet) completed end-to-end — auto-reduced to z15, real DEM (298M valid px, 66 rings), fused `bed_elev_30m.tif` + hypsograph uploaded. All OOM-class lakes (Erie, Mead, Michigan, Eufaula, St. Clair, Pickwick, …) unparked; queue is clear of parks.
Also hardened (`navionics-dem`): now hard-fails when segmentation yields zero valid pixels instead of uploading an all-nodata DEM that crashes `fuse` downstream — surfaces the real cause at the source. (From the Pickwick #2468 triage report; that lake's 82nd attempt was a transient degenerate 1-tile AOI from the lakemon `SHAPE_URL` API, on top of its underlying ISS-15 OOM.)
Next / follow-ups: - ~~Commit the `~/bathy` changes~~ — done (`947c851` size-aware zoom + tile-archive manifest, `518cf34`, `567bd09`; verified 2026-09-08). - Add an AOI-size sanity guard in `fetch-navionics` (reject/retry when `aoi_to_tile_range` yields a suspiciously tiny tile count vs the lake's known extent) — catches the degenerate-AOI glitch one step earlier; and investigate why `SHAPE_URL` returned a near-empty geometry for #2468 on that call. - Optional: 256 GB RAM would let the mid-tier reservoirs (Chickamauga-class) run at full z17 (1.2 m/px) instead of z16.
Resolved (24)