Issues
47 open · 28 resolved
· source: docs/ISSUES.md · Last groomed: 2026-09-28
The tracker is docs/ISSUES.md in the repo — edit it there (or
ask the assistant to log an issue) and this page follows. One entry per
issue: What / Impact / Next step.
ISS-1 · Lake 9822 has no valid shore met point — `FAIL` in met_validation
What:
`_shore_met_point` finds no land cell within 300 m of lake elevation among the bbox-edge/midway candidates (steep canyon reservoir). Air-temp forcing falls back to the centroid water cell.
Impact:
GLM forcing + displayed air temp for 9822 may carry the weather model's internal-lake-temp contamination.
Next:
widen the probe ring (more candidates along the shoreline polygon) or relax the elevation tolerance for canyon lakes; re-run `scripts/validate_lake_met.py 9822`.
ISS-2 · 77 lakes `NO_BBOX` — met validation can't run
What:
fleet met validation (2026-08-15: 117 OK / 77 WARN / 1 FAIL) warns on lakes without `data/processed/{sid}/bbox.json` — partially onboarded lakes.
Impact:
low today (those lakes ship no forecast products), but they'll onboard eventually.
Next:
the `met_validation` workflow step requires `bbox_compute`, so full onboarding covers this; optionally run `bbox_compute` + validation for the 77 to pre-clear them.
ISS-3 · 64 lakes have no satellite temp anchor
What:
no MODIS calibrated series and no recent ECOSTRESS/Landsat scenes — `latest_anchor()` returns None (audit 2026-08-15).
Impact:
their GLM headlines are unverifiable and uncorrected; only the climatology display cap protects them (several showed 33–36 °C raw before capping).
Next:
extend the MODIS extract/calibration pipeline to these siteids (pooled global model already allows zero-pair calibration).
ISS-4 · Utah Lake far-tail heat-wave peaks still read a few °F hot
What:
GLM afternoon peaks 5+ days out exceed the observed envelope under calm heat waves; the climatology soft cap compresses but doesn't eliminate.
Impact:
cosmetic-to-moderate on the forecast chart tail.
Next:
revisit calibration with wave-breaking/convective mixing params, or fit sw_factor seasonally; re-run `scratchpad`-style level+diel sweep after the heat wave passes with fresh MODIS verification.
ISS-5 · Echo deep-hole hypolimnion disagreement
What:
tiled tier-2 hindcast holds ~6–8 °C at 9 m+ in the deepest hole; the live whole-lake forecast says bottom ~17 °C.
Impact:
cosmetic — visible in the 3D cross-section only (~65 of ~9k points).
Next:
compare against any deep DWQ cast; likely the tiled deep column under-mixes (coef_mix_hyp for deep tiles).
ISS-18 · 18:47 UTC GLM cycle timed out daily (Sep 2–8) — cap raised, slowness unexplained
What:
`lakemonster-glm-cycle.service` had `TimeoutStartSec=30min`. The 00/06/12 UTC cycles finish in 22–28 min; the 18:47 UTC (12:47 MDT) cycle hit the cap and was SIGTERMed on Sep 2, 3, 5, 6, 7, 8 — status 143, and because the pool only prints per-lake JSON after all workers finish, the log shows *nothing* for the killed run (232 per-lake logs were touched in the window vs 87 for the morning run, so it was running, just slower).
Impact:
no fresh GLM forecast for ~6 h every afternoon; the +30 min anchor correction ran against stale warm state.
Fix (2026-09-08):
`TimeoutStartSec=55min`; anchor-correct timer moved to +60 min (`01,07,13,19:47 UTC`). `systemctl --user reset-failed` applied.
Next:
find why the 18 UTC cycle is slower (metro open-meteo HRRR 18Z ingest? concurrent `forecasts`/`bloom-tags` timers at :55/:12?). Run the pool with `-v` once so per-lake `runtime=` lines land in the log, or check `data/logs/glm_cycle/*.log` mtimes in the 18:47–19:40 UTC window.
ISS-19 · Lake Merwin #519 `terrain_rgb_sample` fails the brightness gate on every retry — FIXED for wide lakes
2026-10-10:
the probe averaged the whole bbox window, water included — on a wide lake that is mostly dark water, so every clear scene read 0.09–0.10 against the 0.15 floor (Sakakawea, Guntersville, Lake of the Woods, Belle Fourche, Carmi, Upper North Crow). The probe now masks the lake polygon out and judges the land: Sakakawea's July scene reads 0.277. The colour is also sampled through the TCI's 30 m overview (the terrain cells are 30 m), 9x less to read, and the terrain cloud itself is capped near 1.5 M points (Sakakawea's was 42.5 M / 1.2 GB).
What:
all 8 probed S2 candidates have lake-area mean brightness 0.128–0.136, below the 0.15 floor (`derive_terrain_rgb.py --min-brightness`), so the reonboard workflow defers 3600 s and repeats (2× so far on 2026-09-08; it *passed* on 2026-09-02).
Impact:
#519 registry workflow never completes; one deferral log line per hour.
Next:
per-lake brightness floor override in lake_config, or accept the brightest candidate when all fail by a small margin (<0.03); or widen the candidate window (`max_candidates`).
ISS-20 · 35 lakes carry a degenerate/wrong canonical polygon in local `lakes.geom` → empty day cubes, empty 3D, failed syncs
What:
`lakes.geom` (pinned at onboarding from vps `get_shape`, see ISS-16) is a ~30–45 m box for **35 of 482 lakes** (#52, 130, 184, 187, 197, 199, 210, 255, 315, 342, 406, 431, 440, 473, 487, 537, 541, 542, 590, 604, 609, 613, 664, 710, 766, 868, 880, 881, 960, 1000, 1073, 1082, 1136, 1164, 1567) and points at the WRONG water body for ~12 more (#40: a 0.23 km² pond 5 km from the bed raster; also 35, 107, 545, 565, 656, 663, 781, 800, 1107, 1183, 1318, 1379, 1525). The bed rasters are fine (tiling uses the raster mask), but `_shrunken_polygon` + `_filter_by_buffer` dropped every pixel, so **47 lakes shipped 24 empty frames per day cube** (`n=0`, `polygon_pixels` > 0) and the live `tier2_3d` / surface endpoints rendered nothing. Downstream, `vps_lake_sync.build_view3d_frame` raised "zero-size array" in `draco_encode`, which failed the WHOLE lake sync — 51 lakes, ~300 failures/day, every 15 min.
Code guards (2026-09-20, `web/app.py`):
`_shrunken_polygon` detects a degenerate geometry (projected bbox < `DEGENERATE_POLYGON_M` = 200 m), substitutes the current `lake_polygon_history` shape, or returns `None`; new `polygon_keep_mask(siteid, buffer_m, lat, lon)` returns all-True (warn once) whenever the polygon would keep 0 of N > 0 pixels — used by `build_tier2_day.py` and both 3D endpoints. `vps_lake_sync`: empty frames are skipped (object omitted, lake still ships), every object builder is isolated in try/except, and the envelope is NaN/Inf-sanitised (`allow_nan=False`) after 10 "Invalid JSON body." rejections. All 47 cubes rebuilt; #881 `tier2_3d` 0 → 9,886 points; #960/#5517 sync OK.
Also (2026-09-21):
these polygons made the nightly SWOT ingest download thousands of granules per lake and intersect them with a 30 m box — six such lakes (210, 330, 342, 406, 440, 473) each hit the 900 s per-lake cap and ate the ENTIRE 90 min budget (50 real lakes done, 430 requeued). `refresh_swot_shape_attribution.py` now skips polygons < 300 m with one warning; `DEGENERATE_POLYGON_M` raised to 300 (#330 Lake Almanor is a 233 m box). #330 and #766 added to the affected list.
Next (server-side, needs operator):
regenerate these shapes in the vps.lakemon.com lake DB (extend `scripts/audit_lake_shapes.py` to the onboarded fleet — the 35 above are `DEGENERATE`, the offset ones need an area-vs-bathymetry cross-check), then re-pin each with `refresh_lake_polygon_canonical.py --force`. Until then the polygon buffer is effectively off for these lakes (slightly softer shoreline).
ISS-21 · `swot-shape-attr` (daily) and `terrain-refresh` (weekly) SIGTERMed on every run for weeks — FIXED
What:
`lakemonster-swot-shape-attr.service` hit its 2 h `TimeoutStartSec` 10 of the last 11 days; `lakemonster-terrain-refresh` hit 3 h on 3 of its last 4 weekly runs. Both were fleet loops over 482 lakes with no per-lake bound and no resume cursor: the SWOT driver had a budget flag but a single uncached lake downloading NASA granules ran 30+ min straight through it; the terrain driver had neither. Worse, each run restarted at siteid 1, so the tail never ran — only **75/482** lakes had a SWOT refresh in the 3 days before the fix; terrain never got past ~#1447.
Fix (2026-09-20):
`refresh_swot_shape_attribution.py` gained `--lake-timeout-s` (SIGALRM, 900 s; `LakeTimeout` is a *BaseException* so the extractor's per-granule `except Exception` can't swallow it), a round-robin `cursor` in `data/swot_attr/state.json`, INFO progress lines with timestamps, per-lake `runtime_s`, and a done-summary. `refresh_terrain.py` gained `--run-budget-s` (9000 under 3 h), a cursor in `data/terrain_refresh/state.json`, a one-shot DEM fetch for the 36 lakes without a tile (they failed every week before), and exit-code policy. SWOT extract checkpoints are now written atomically (tmp + rename). Cursors seeded to the last lake each killed run reached so the starved tail goes first. Validated: SWOT `--run-budget-s 150 --lake-timeout-s 60` → ok=11 timeout=3 requeued=468 exit 0 in 154 s; terrain `--lake 1449` (no DEM) fetched the tile and refreshed in 49 s.
Also:
`ops/systemd/*.service` in the repo had drifted from the installed copies — every `--run-budget-s` (elevation, backfill, heal, modis-lst, process-queue, shape daily/weekly, vps-sync) lived only in hand-edited files under `~/.config/systemd/user`, so a re-run of `install.sh` (`ln -sf`) would have silently removed all of them. Repo units updated to match, `bathy-triage` + `smoke-web` units added to `install.sh`, and all installed units are symlinks again.
Next:
none required; watch the two units for one cycle (`journalctl --user -u lakemonster-swot-shape-attr -u lakemonster-terrain-refresh`).
ISS-22 · 9 lakes: bed raster datum 3–14 m too high (built on a wrong registry `elevation_m`) → bottom mask dropped every pixel — FIXED (republished)
What:
`physics_field.build_polygon_dense_grid` computes `local_depth = crest − bed` with crest = `COALESCE(spillway_elevation_m, elevation_m)`. For #781 the spillway (272.1 m) sits below 100 % of the bed raster (275–277 m), so every depth was negative, the bottom mask kept nothing and the cube/3D stayed empty even after the ISS-20 polygon fix. Same signature in #542, 320, 247, 1102, 519, 2380, 799, 1073. The dense-grid JSON cache also never expired, so a corrected crest or a republished bed raster never reached the endpoints.
Audit (2026-09-20, EPQS at the deepest raster pixel):
the SPILLWAY is the right number; the bed rasters are wrong. All 9 spillways come from `satellite_observed_max` and agree with published full pool (Buckeye 891 ft = 271.6 m, Merwin 239.5 ft = 73 m, Wylie 569.4 ft = 173.6 m, Minnetonka 929.4 ft = 283.3 m, Alum Creek 888 ft = 270.7 m). The `~/bathy` `publish` step wrote `bed = registry elevation − depth` using the vps registry `elevation_m`, which is an integer 3–14 m too high for these lakes (781: 277 vs EPQS 271.0; 542: 109 vs 102.6; 320: 8 vs 5.0; 247: 289 vs 283.0; 799: 276 vs 270.2; 519: 87 vs 74.7; 2380: none → 3105 vs 3055.4; 1102 and 1073 are ambiguous, EPQS sides with the registry there). So the bed rasters — and the NAVD88 hypsographs derived from them — for these lakes sit 3–14 m above reality.
Render guard (2026-09-20):
when the crest lies below more than half the in-polygon bed (+1 m), the grid builder logs `datum mismatch` and uses the raster surface as crest (depth below surface is then correct; absolute elevations are not); `load_polygon_dense_grid` rebuilds a cache older than the bed raster or with negative depth almost everywhere. All 9 cubes rebuilt (#781: 0 → 3,230 points/frame; live `tier2_3d` 2,426).
Fixed (2026-09-20, `~/bathy` `c59e965` + republish):
`bathy publish` gained `--surface-elev-m` (operator-verified surface) and an automatic cross-check of the registry elevation against USGS EPQS at the deepest pixel — EPQS wins when they disagree by > `--surface-tolerance-m` (3 m); provenance now records `lake_surface_source`, `registry_elevation_m`, `epqs_elevation_m_at_deepest`. All 9 lakes republished with their verified spillway as surface, promoted to `bathymetry/bed_elev_30m.tif` + `hypsograph_1m.csv`, re-synced locally, extended hypsographs rebuilt, dense grids self-healed and cubes rebuilt. Result: crest == bed surface for all 9, zero negative depths, e.g. #519 Merwin depth max 15.0 m, #247 Minnetonka 29.5 m. GLM / stage math / storage pick up the corrected hypsographs on their next scheduled runs.
Still open:
correct `elevation_m` in the vps registry for the 7 confirmed-wrong lakes (781, 542, 320, 247, 519, 799, and none for 2380), so future re-derivations start from the right number.
ISS-23 · GLM saw 3,600× the real rainfall (NLDAS `Rainf` unit error) → 42 lakes' tier-2 hindcasts stalled at their first cold snap, reported `ok` — FIXED
Symptom:
`refresh_tier2_days` logged `parquet frontier <date> implausibly old — skipping this cycle` for 47 lakes on 2026-09-21 (42 on 09-27; frontiers 2025-12-12..2026-08-12, clustered Dec–Mar). Per-lake GLM logs: `Array bounds error - too many layers. NumLayers = 1, M = 500` — the column collapsed to one layer and GLM aborted; the rolling driver's 2-day-step resume died at the same date every time and then returned `status=ok` with `last_t` months old. No cube built since; 3D frozen.
Root cause (reproduced on #99, 2026-01-25..02-20):
`glm.nldas2_to_glm_met` treated NLDAS-2 `Rainf` as a kg/m²/s flux (× 86.4 → m/day). `io/nldas2.py` documents it correctly as an HOURLY ACCUMULATION in kg/m² (= mm/h), so every NLDAS-forced GLM run received 3,600× the real precipitation — Huron 3,461 m/yr, Bear 2,410, Utah Lake 1,611 (should be 0.96 / 0.67 / 0.45) — and all of it as liquid rain even at −5 °C (no snow split). In the repro tile 0 grew 5.7 m of white ice in a week, the lake level fell 3.6 m, then the layer stack collapsed. Same forcing fed `run_glm_live_cycle` (NLDAS spin-up portion of `lake_meteorology_continuous`), `build_tier2_forecast`, `build_hourly_reanalysis`, `rebuild_glm_hindcast` and `calibrate_glm`.
Fix (2026-09-27):
`Rain = Rainf × 24 / 1000` m/day with the same 1 °C rain/snow split as the HRRR path. Repro now runs the full window: 4 tiles to 02-21, ice-free, level 11.16 → 11.13 m. `build_tier2_rolling` reports `status=stalled` (+ `frontier_gap_days`, exit 2 under `--single`) when `last_t` is > 7 d short of the window end (`STALL_GAP_DAYS`), so the pool/heal/status see it. The 42 stalled lakes were rebuilt 2026-09-27 22:00–23:00 MDT (`tier2-iss23-rebuild` transient unit, `data/logs/tier2_iss23_rebuild.log`): **42/42 `ok`, every frontier at 2026-09-29**, no stalls; cubes follow on the hourly tier2-days timer. The live GLM cycle picks up the corrected forcing at its next 6 h run.
Follow-ups:
(1) per-lake GLM calibrations (kw, mixing) were fitted with the rain error present — recheck the anchor residuals over the next weeks and re-run `calibrate_glm.py` for lakes whose bias shifts; (2) the ice-albedo-lock breaker and "deep-freeze collapse" comments in `build_tier2_rolling.py` describe symptoms of this same bug — they can likely be simplified once a full winter passes clean.
ISS-24 · GLM live cycle drained lakes to half depth; the 50 % warm-state floor then froze them (267 lakes) — FIXED
What:
`save_warm_state` rejects a run whose water column is below `0.5 × hypsograph max depth` (`_degenerate_state_threshold_m`) and quarantines it to `latest.broken.*.json`. Fleet audit 2026-09-27: of 519 warm states, **267 sit at 0.50–0.55 of hypsograph depth** (median 0.54; p10/p25 exactly 0.50), 146 are > 7 d old, and quarantine files run at ~2,000/day (7,756 in Aug, 39,760 in Sep; 266 lakes' latest run flagged today). Pattern in every log: `lake_depth=11.916 m … floor=12.00`, `6.363 … 6.40` — the column is *just* under the floor.
Mechanism:
the live cycle has no water-balance closure for lakes without gauged inflows: evaporation drains the modelled level a few cm per cycle and the warm state carries the decline forward. Once the level crosses the 50 % floor every subsequent save is rejected, so `latest.json` freezes (e.g. #9822 stuck at 6.47 m since 09-17, #2374 at 11.63 m since 09-01) and each cycle re-runs from that frozen half-depth state — or, after 30 d, cold-starts (`no_warm_state` jumped 9 → 82/day on 09-26). `_anchor_level_to_stage` re-pins the level only when a gauge/SWOT stage exists (Bear 54.9/60 m and Rockport 21.2 m track their observations correctly); most lakes have none, so nothing ever pulls the level back up. The rain-unit bug (ISS-23) hid this on NLDAS-forced runs by over-filling the lake; the live cycle mostly runs on HRRR (NLDAS lags the forecast window), so it drained regardless.
Impact:
thermal structure for ~half the fleet is computed on a half-depth column (wrong thermocline depth, bottom temps, mixing volume); warm-state continuity is lost, so anchor nudges evaporate.
Fix (2026-09-27):
`_anchor_level_to_stage` now runs with a full-pool fallback — every cycle's init profile has its depth axis rescaled to the observed stage when one exists, else to the hypsograph full-pool depth (the same convention as cold starts and the tier-2 hindcast); an implausible stage also falls back to full pool instead of leaving the level unpinned. Validated on the two frozen examples: #9822 6.47 → 12.77 m, #2374 11.63 → 22.78 m, both saved fresh warm states. The fleet re-anchors on the next 6 h cycle. Purged 47,353 stale `latest.broken.*` files (newest kept for 302 lakes).
Watch:
ungauged reservoirs that really are drawn down will now be modelled at full pool (as cold starts always were); a per-lake climatological level would be the refinement. Check anchor residuals after a week — thermal structure for ~half the fleet just changed.
Related:
the exit-2 queue entries (`run_glm_live_cycle exited 2`, 45 since 09-25) are lakes whose single-lake HRRR-triggered runs ended degenerate for this reason.
ISS-25 · Fleet producers starved the tail of the fleet; onboarding lane capped at ~8 lakes/day — FIXED
What (audit 2026-09-27, fleet = 555 lakes):
product health showed MODIS `nodata` for 416 lakes, SWOT/elevation 278, ECOSTRESS rolling series 172, shape 228, tier-2 35; 324 lakes "waiting". Three separate causes: (a) budgeted fleet drivers walked `ORDER BY siteid` and requeued leftovers — elevation reached ~264 lakes per 3.5 h run, MODIS ~55 per 40 min run — so lakes above the cut never ran, ever; (b) the registry workflow lane ran ONE ~2.2 h workflow at a time against a 382-lake queue (~32 days), the two ~45 min NLDAS pulls dominating; (c) 30 lakes whose bathymetry had finished in MinIO never had it synced locally, so the hard `bathymetry_sister_sync` gate kept them half-onboarded (no bbox, no tier-2) while the queue re-deferred them.
Fix:
`lakemonster.workflow.cursor` round-robin cursor (state in `data/pool_cursors/`) wired into `run_pool`, the elevation and GLM-cycle pools and the MODIS serial loop — every budgeted run resumes after the last lake the previous run started (SWOT/terrain already had ad-hoc cursors). `dispatch_workflow_queue` runs `LAKEMON_ONBOARDING_MAX_CONCURRENT` worker threads (set to 3 in `.env`; per-lake lease keeps them apart; `run_one` skips leased lakes). The 30 lakes' bathymetry synced + hypsographs extended; tier-2 builds launched (`tier2-nobed-build`); the ECOSTRESS rolling series backfilled fleet-wide (`lst-rolling-backfill`, no timer had ever produced it for pre-existing lakes). GLM cycle unit now passes `--run-budget-s 3000` (it timed out again at 55 min on 2026-09-27 12:47 with 555 lakes).
Superseded 2026-09-28 (ISS-28):
the three onboarding workers were turned back to one; the lane's problem was not concurrency.
Expect:
MODIS coverage completes in ~2.5 days of rotation, elevation in ~1 day. Permanent step failures still loop every 2 h → 12 h backoff (#137 Tempe Town Lake: no ECOSTRESS scene ever; #442 Cheat Lake: S2 brightness gate, ISS-19 class; #373 Kegonsa: polygon covers the whole DEM tile) — those need a skip-after-N-attempts rule or a per-lake override.
Blocked on the VPS:
Superior, Sakakawea, St Clair, Saginaw Bay, Toledo Bend, St Johns River defer on `master_shape/master_bbox/ lake_identity` prerequisites (ISS-16/20 shapes).
ISS-26 · `get_shape` serves a fragment for 104 of 558 lakes, and the number is growing — onboarding built lakes on 50 m polygons
What (audit 2026-09-28, `docs/triage/master_shape_audit_2026-09-28.md`):
the master polygon pinned in `lakes.geom` is unusable for **104 lakes** (88 under a 200 m bbox, the rest under half the area their bathymetry was built from) — Choke Canyon #547 0.6 acres for a 25,670-acre reservoir, Flaming Gorge #11 0.2 acres, Lake Crescent #325 0.2 acres. ISS-20 counted 35 on 2026-09-20. 102 of the 104 were good when the bathymetry lane fetched them (2026-08-02 … 09-08 — the snapshot in `s3://bathy/{sid}/shape.geojson` is a whole lake), and on the VPS the fragments carry `shape_updated` stamps of **2026-09-07 … 09-26, 15–21 per day on the last four days**: something on vps.lakemon.com is rewriting good shapes into fragments right now. deep_water only ever reads `get_shape` (no write path exists in `deep_water` or `~/bathy`).
Impact:
this is what most "failed" onboardings of the last week actually were. `ecostress_pixel_prewarm` errors on every scene (clip is one pixel wide) → `climatology_derive` "no cached ECOSTRESS scenes"; `terrain_pointcloud_extract` "polygon covers the entire DEM tile"; `terrain_rgb_sample` brightness gate; `met_validation` `NO_SHORE_POINT`. Worse, lakes whose steps all exit 0 **complete on the fragment**: 36 lakes are marked onboarded this way (Cass Lake #1541: 33 min of `polygon_history_derive`, 295 scenes rejected, 0 shapes). The three lakes in flight when this was found (547, 584, 1459) were all on fragments.
Fix in deep_water (2026-09-28):
`lakemonster.lakes.master_shape` — pre-flight, `seed_lake.py` and `refresh_lake_polygon_canonical.py` refuse a fragment and use the polygon the lake's bathymetry was built from (`s3://bathy/{sid}/shape.geojson`, the `get_shape` polygon as it was when the bathymetry lane ran); `attrs.polygon_source` records it. 102 of the 104 have one. A good `lakes.geom` can no longer be overwritten by a fragment. its shape.** The master polygon is now the USGS NHD water body at the registry lat/lon (`lakemonster.lakes.nhd_shape`), merged with the bathymetry polygon; then the bathymetry polygon; `get_shape` only as a checked last resort. On a 40-lake random sample NHD returned the right lake for 37 (bathymetry polygon 83–100 % inside it), nothing for 1 (Swan Falls), a wrong neighbour for 1 (California Delta) and an outline smaller than the bathymetry polygon for 1 (Lake Pueblo) — all three caught by the overlap test or covered by the merge. #440 Summersville and #441 Burnsville, which had no fallback, resolve from NHD (2,843 and 828 acres). `~/bathy` `fetch_lake` uses the same lookup (dfecd60); the four lakes parked in the bathymetry lane for bad shapes (#90 Chelan, #496 Wallula, #2468 Pickwick, #16998 Higgins) were unparked. `scripts/audit_master_shapes.py` → `docs/triage/master_shape_sources.md`.
Fleet audit (2026-09-28, 720 lakes = `lakes` + the queue):
681 resolve from NHD, 39 from the bathymetry polygon (17 are one bay of a larger lake, 14 where NHD returned a different water body, 5 where it returned a sliver, 3 where it has nothing), none unresolved. For 438 lakes the polygon now pinned differs materially from the master polygon (NHD is the normal-pool outline — median 1.14x the pinned area); their caches are purged when they are next onboarded. Two lakes' bathymetry was built somewhere other than the lake (#208 Toledo Bend: a 16-acre polygon; #1531 Belleville Lake) — sent back to the bathymetry lane.
Still worth doing server-side:
find what is rewriting shapes on vps.lakemon.com. deep_water no longer depends on it, but the website draws those shapes.
Open:
lakes that completed before this change keep the polygon they were pinned with until they are re-onboarded.
Re-queued (2026-09-28):
the 36 lakes that had completed on a fragment were taken out of the completed set and put at the end of the registry-workflow queue (`source: iss26_requeue`); the audit doc lists them. Queue 382 → 418.
ISS-27 · `ecostress_pixel_prewarm` reports `ok` when every scene fails — FIXED
What:
`prewarm_lst_pixels.py` exits 0 with `{'scenes': 100, 'with_pixels': 0, 'empty': 0, 'errors': 100}`. 60 of 535 onboarding runs had every scene error, 69 more had some.
Impact:
the failure is invisible at the step that caused it and shows up later as `climatology_derive` "no cached ECOSTRESS scenes", which reads like "this lake has no ECOSTRESS coverage".
Cause of the all-error runs seen so far:
the fragment polygons of ISS-26 (rioxarray: "clipped raster x,y coordinates has only one point").
Fix (2026-09-28):
`prewarm_lst_pixels.py --single` exits 2 when every scene raised. The step is non-blocking, so the lake still onboards and the failure is listed under its degraded steps.
Next:
re-check the 69 partial-error lakes once the ISS-26 polygons are corrected.
ISS-28 · Onboarding lane made no real progress: ~6 lakes/day against a 382-lake queue — FIXED
What (re-evaluation 2026-09-28):
(a) every lake downloaded all NLDAS-2 granules itself — whole-CONUS files, ~84 GB for the raw pull and ~44 GB again for the met build, whose window/bbox never matched the raw pull's cache key — 88 of ~125 min per lake; (b) any failed step failed the lake and the retry cleared everything and started over, so about half of all 2 h runs produced nothing; (c) a vps.lakemon.com outage was charged to every queued lake as a failed attempt (340 deferrals 08-29, 423 on 09-25); (d) failures went to the back of the queue; (e) a dispatcher restart killed the running lake and left its lock held for 2 h; (f) clearing ran before the pre-flight; (g) the master polygon was never sanity-checked (ISS-26). ISS-25's answer — three concurrent workers — tripled the downloads and the garbage, not the progress.
Fix:
one lake at a time, in order (`LAKEMON_ONBOARDING_MAX_CONCURRENT=1`); shared NLDAS-2 granule store (`lakemonster.io.nldas2_store`, bit-identical crops, ~1 min for three years); resumable runs with per-step checkpoints; blocking vs non-blocking steps; park after 3 failed attempts; pre-flight first; VPS outage holds the lane; place in line kept; interrupt-safe runs; steps killed as a process group; Overpass mirrors. Full description: `docs/ONBOARDING_LANE.md`.
Queue migration (2026-09-28):
attempt counts reset (they were all outage charges), the 13 lakes the lane had already worked on moved back to the front.
ISS-29 · GLM live cycle refused to start: unit passed `--run-budget-s`, script did not accept it — FIXED
What:
5c97e70 (2026-09-27 23:04) added `--run-budget-s 3000` to `lakemonster-glm-cycle.service`, but `run_glm_live_cycle.py` builds its own argument parser and never declared the flag (it only read `args.run_budget_s`), nor passed a budget to `LakeWorkerPool`. The first timer firing afterwards (2026-09-28 00:48) exited 2 in 2 s: `unrecognized arguments: --run-budget-s`.
Impact:
no GLM cycle for the whole fleet from the 00:48 firing until the fix; every later firing would have failed the same way.
Fix (2026-09-28 01:07):
flag declared and passed to the pool; cycle started by hand (`glm_cycle-20260928T070731`).
ISS-30 · SWOT elevation: every lake downloads 36 MB continent-wide granules one by one and throws them away
What:
`_extract_swot_one` downloads a whole SWOT LakeSP granule (a zipped shapefile of every lake on that pass over North America, ~36 MB) into `/tmp`, keeps the one or two features that touch the lake, and deletes it — ~5 s per granule. A first full scan of the passes that cover a lake is ~600 granules, about an hour and ~20 GB per lake, and the same granules are downloaded again for every other lake under that pass. Same shape of problem as NLDAS-2 was (ISS-28).
Made far worse by ISS-26:
the pass-coverage table is probed against `lakes.geom`. Probed against a fragment it records "no pass covers this lake", and an empty coverage set silently falls back to the WHOLE catalog — Tygart Lake #228: 6,949 granules, ~10 h, against a 2 h step timeout. The parquet also remembers every granule that missed the fragment and skips it forever, so the lake could never get SWOT history even after its polygon was corrected. The onboarding step never ran the probe at all.
Fixed (2026-09-28):
`ingest_swot_elevation.py` records the polygon the cache was built for (`data/cache/lakes/{sid}/swot_polygon.json`); when `lakes.geom` changes it re-probes the passes and rebuilds the parquet. "Probed, nothing covers" is re-probed instead of trusted. The onboarding step runs with `--budget-s 900`: newest granules first, the recurring SWOT jobs fill in the rest (skip-seen). Earthdata login is reused for 30 min instead of once per granule (1.7 s each).
Open — the real fix:
keep SWOT granules (or a reduced copy: the features near any registry lake) in a shared store like `nldas2_conus`. Raw it is large — order of 30,000 North America granules, ~1 TB; reduced it is small. Needs a decision on which.
Also open:
the 36 lakes re-queued under ISS-26 and any other lake whose polygon changes get their SWOT cache rebuilt automatically on their next run; nothing to do by hand.
ISS-31 · Per-lake caches outlive the polygon they were cut to — corrected lakes kept rebuilding on the fragment — FIXED
What:
caches keyed by `(siteid, granule)` or `(siteid, kind)` hold data already cropped to the master polygon and have no polygon in the key: clipped band rasters (`data/cache/polygon_bands/`), imagery WebPs, ECOSTRESS/Landsat/MODIS pixels, wind shelter, SWOT/ATL13 elevation, SWOT pass coverage. After ISS-26 gave a lake its real polygon they still served the fragment. Lake Cushman #521 and Clear Lake #1217 "completed" on 2026-09-28 with **0 shapes** — every scene rejected as "polygon area 0.2 km² outside band [5.9, 23.7]" because the 250 cached band rasters were 1.2 km squares.
Fix (2026-09-28):
`lakemonster.lakes.polygon_caches.purge(siteid)` drops the polygon-derived caches; the queue lane calls it whenever the pinned `lakes.geom` is not the master polygon the pre-flight resolved. `band_cache.cached_clip(cover_4326=…)` also discards any cached raster under a quarter of the lake's bbox area, so that cache heals itself. Migration 020 indexes `lake_notes.pointer_id` (deleting one lake's ~150 pointers took a minute against 4.6 M un-indexed notes).
Re-run:
the five lakes onboarded on corrected polygons before the fix (228, 1008, 521, 1217, 325) were purged and put back at the front of the queue.
Open:
lakes whose `lakes.geom` was swapped to a derived shape (`polygon_source` = `landsat_oli_l2` etc.) were not audited; a raster extent check flags ~35 more lakes whose band cache is smaller than their current polygon bbox, for reasons not yet looked at.
ISS-32 · NLDAS-2 crop crashed under `nldas2_backfill`'s three threads — 89 lakes parked — FIXED
What:
`nldas2_store.subset_window` read the grid and variable attributes with netCDF4 in the calling process without the HDF lock; `nldas2_backfill.py --workers 3` called it from three threads. HDF5 is not thread-safe: "NetCDF: Can't open HDF5 attribute", "HDF error", "double free or corruption", SIGSEGV (rc 139/134), 2026-09-29 → 10-07. Three attempts a few seconds apart, then parked — 89 lakes.
Fix (2026-10-08):
the parent-side reads take `_HDF_LOCK`; the backfill's default is one thread (the crop already fans out over worker processes). The 89 lakes were unparked and resume from their checkpoints.
ISS-33 · Big-lake shoreline derive found no scenes: STAC search body too large — FIXED
What:
the search posts the lake polygon as `intersects`. An NHD outline of a large reservoir has thousands of vertices and Planetary Computer answers `{"error":"Request body is too large."}` — the existing 413 fallback matched only the old wording. Every month failed on every provider and the lake catalogued nothing: 13 lakes "completed" with 0 shoreline shapes (Powell #9, Sakakawea #231, Lake of the Ozarks #37, Clarks Hill #1026, …). The web smoke test has flagged Powell's history page (0 scenes) since.
Fix (2026-10-08):
outlines over 500 vertices are generalised to ~50 m for the request, and the bbox-plus-post-filter fallback also triggers on "too large". Powell: 45 scenes for Aug 2026. The 13 lakes' 3-year shoreline backfill is being re-run serially (`data/logs/polygon_history_refill.log`).
ISS-34 · `cloud-ledger` ran the fleet serially with no budget — timed out at 2 h daily — FIXED
What:
`analyze_imagery_clouds.py --all` walked every lake in siteid order; the unit's 2 h `TimeoutStartSec` SIGTERMed it (exit 143) before the tail of the fleet, every day.
Fix (2026-10-08):
`--run-budget-s` + round-robin cursor (`cloud_ledger`), unit passes 6600 s.
ISS-35 · Three lakes park on `tier2_thermal_build` `empty_run`: no chart soundings — FIXED
What:
#2363 Cedar Creek, #2377 Whitney, #14886 Goode Ski Lake: every GLM tile is skipped for "insufficient depth" and the run is empty. Their `bathymetry/hypsograph_1m.csv` holds one or two rows — the Navionics derive found essentially no depth for these small, obscure water bodies — and the bathymetry polygon covers only 33–69 % of the master polygon. Not a pipeline bug; there is no bathymetry.
Fix (2026-10-08):
`lakemonster.lakes.synthetic_bathymetry` writes a parametric bowl inside the master polygon — depth grows with distance from shore to the registry `max_depth_m`, or to an area-based estimate (2.6 m for the 9 ha pond, 6.6 m for the 958-acre reservoir) — as `bed_elev_30m.tif` + `hypsograph_1m.csv` locally and on the bucket, chart output kept as `*.navionics.*`, `synthetic.json` marker, `bathy_confidence = parametric`. `sync_lake_bathymetry.py` does this automatically when the chart hypsograph is flat. Tier-2 builds for #14886: 6 tiles, 8,784 steps. All three unparked.
ISS-36 · Pickwick #2468 parked in the bathymetry lane: a 2.5 KB stub raster shadowed the real DEM — FIXED
What:
an OOM-killed z17 attempt left `outputs/navionics_dem_z17.tif` as a 2,560-byte stub; `fuse` tries z17 before z16 and crashed on it ten times ("not recognized as being in a supported file format").
Fix (2026-10-08, `~/bathy`):
`_try_load` skips an object that does not open as a raster and falls through to the next zoom. Unparked.
Again (2026-10-09):
the stub *does* open — it is a valid raster with no data — so it still won over z16 and `fuse` failed on an empty array nine more times. A DEM layer with no finite pixel is now skipped too.
ISS-37 · Bathymetry marked complete on the wrong outline was never rebuilt — FIXED
What:
`trigger_bathy_derive` short-circuits when the bucket already holds the two bathymetry files, so #208 Toledo Bend and #1531 Belleville — sent back for a rebuild on 2026-09-28 because their AOI was a fragment — "completed" in one second with the old 16-acre bathymetry.
Fix (2026-10-08):
the short-circuit first compares the bucket's `shape.geojson` with the lake's master polygon (`material_change`) and rebuilds when they differ. Both re-sent.
ISS-38 · Degraded steps were never retried — FIXED
What:
a lake that finished with a non-blocking step failed (Overpass timeout, brightness gate) kept that gap until someone read `--list-degraded`: 36 lakes by 2026-10-08.
Fix:
`scripts/retry_degraded_steps.py` re-runs each degraded step (not `polygon_history_derive: empty`, which the shoreline refill and the daily shape refresh own) and clears it on success; `lakemonster-degraded-retry.timer` daily.
2026-10-09:
the first run spent its whole budget on two 20-min overlay timeouts and reached 3 lakes. Overlay retries now run last, after every lake's cheap steps; a step that timed out is retried weekly; budget 2 h. Big-lake overlays (bbox > 0.15 deg²) fetch major roads and rivers only.
Not retried on purpose:
`met_validation` `WIND_VS_REALITY` (#1109, #2050, #4205 — our shore wind 8–12 mph against 3 mph observed) is a QA flag on the met forcing, the same class as ISS-1/ISS-2.
ISS-39 · DEM tile and terrain cloud cut for the old polygon were kept through re-onboards — FIXED
What:
`data/processed/{sid}/dem/` is not cleared by a re-onboard and `fetch_dem_tile` / `derive_terrain_pointcloud` keep whatever exists. A lake given its real polygon kept a DEM tile cut for the fragment and rebuilt a terrain cloud of a few dozen points from it (Clear Lake #1217: 29 points), then the overlay step queried a 200 m bbox. Audit 2026-10-08: 15 DEM tiles and 21 terrain clouds under a quarter of their lake's bbox, 22 lakes in all.
Fix:
the DEM tile is refetched when it does not cover the canonical bbox (and a sister-bucket tile that does not cover it is passed over for 3DEP); the terrain cloud is rebuilt when its extent is under a quarter of the polygon's; `dem/` joins the polygon-cache purge. The 22 lakes' checkpoints carry `stale` entries for the four terrain steps, which `retry_degraded_steps.py` re-runs with `--force` tonight.
Also:
when every Overpass instance is down the overlay step now takes roads from the USGS National Transportation Dataset and named streams from NHD flowlines instead of failing (`metadata.source` records which).
ISS-40 · Big lakes derive no shorelines: the cloud gate is sized for small lakes — FIXED
What:
the polygon-aware cloud guard rejects a scene when more than 1 % (S-2) / 2 % (Landsat) of the lake AOI is cloud-classified. On a 500–1,300 km² reservoir some cloud sits over the water in nearly every scene: the 2026-10-08 refill blocked 1,000–1,300 scenes each for Sakakawea #231, Wheeler #2470, Kerr #538, Powell #9 and derived nothing. Separately, one failed band read (expired blob token, 403) aborted a whole lake's backfill, and Powell's refill was cut by the script's 4 h cap.
Fix (2026-10-09):
threshold scaled by sqrt(area / 10 km²), capped at 12x (Powell ≈ 7 %/14 %); the area band in the derive still rejects a pass where cloud eats into the outline. A failing cloud or coverage probe skips that scene instead of the lake.
Still nothing (2026-10-10):
with cloud relaxed, every scene on those six lakes was `partial_swath` — a lake 1–4° across is never inside one Sentinel-2 tile or Landsat path, so the 95 % coverage gate rejected all 1,100–1,500 scenes per lake. 25 lakes span more than 0.5°.
Fix:
`lakemonster.geometry.swath_fill` — the derive records the scene's valid-data footprint and completes the shape with the canonical outline where the scene had no data (`observed_frac` in the result); the coverage gate passes scenes that see ≥ 20 % of a lake wider than 0.5°. A whole-lake shape per scene, current where seen, canonical elsewhere. A true same-day tile mosaic is the follow-up. Refill re-run Powell first (`data/logs/polygon_history_refill3.log`).
ISS-41 · Live GLM cycle crashed on lakes deeper than ~100 m (Seneca #2633, Fremont #1003) — FIXED
What:
the namelist is cloned from glmpy's Sparkling Lake example: sediment `zone_heights` [10, 20, 30] m and layers of 0.15–0.5 m with `max_layers` 500. A 188 m lake needs 376+ layers at 0.5 m; GLM 3.3 "adds an extra zone to compensate" and dies with "double free or corruption" (Seneca: 40 failed inbox cycles; Fremont: 34). The two lakes' forecasts showed `error` on /status.
Fix (2026-10-10, `build_glm_nml`):
for lakes deeper than 30 m the top sediment zone reaches the bed; deeper than 100 m the layers are thickened (depth/1000 … depth/300 m) so the column stays near 300 layers. Seneca: 78,115 rows, 8 days.
Also seen in the same failures:
queued lakes whose chart bathymetry is flat (#1023) fail every inbox cycle with "GLM produced empty output" until the lane onboards them and synthesises a bed (ISS-35).
ISS-42 · Shoreline derive restarted from zero on every retry; Lake Tahoe parked — FIXED
What:
the registry ran `backfill_polygon_history.py --years 3 --reset`. `--reset` wipes the lake's polygon history before deriving, so each of Tahoe #258's three attempts (two S-2 tiles, three years: more than the 2 h step timeout) started over and was killed; the lane parked the lake after attempt 3 with nothing to show. The clearing phase of a first attempt already wipes the same tables, so the flag only ever hurt resumed attempts.
Fix (2026-10-11):
`--reset` dropped from the step; the backfill takes `--run-budget-s` (registry: 6,000 s) and stops between scenes when it is spent, exits 0 and writes `data/processed/{sid}/derived/polygon_history_backfill.json` with `partial: true`. The lane lists such a lake as degraded (`polygon_history_derive: partial`) instead of parking it; the nightly degraded retry re-runs the step last, and it resumes from the granules already decided. Tahoe unparked.
ISS-43 · SWOT/ATL13 granule temp dirs leak into /tmp — FIXED
What:
`lake_elevation` downloads each granule into a `mkdtemp` dir and removes it in a `finally`; when the elevation pool kills a worker at its 1,200 s step timeout (SIGTERM, then SIGKILL) mid download, the `finally` never runs. 57 `/tmp/swot_*` dirs, 3.3 GB, from the daily and weekly elevation services on 2026-10-11.
Fix:
`sweep_stale_tempdirs()` removes `swot_*`/`atl13_*` dirs older than 3 h at the start of `refresh_lake_elevation.py` (parent only) and `ingest_swot_elevation.py`.
ISS-44 · MODIS refresh could not cover the fleet: a download per lake per scene — FIXED
What:
`extract_modis_pixels` downloaded the MOD11A1/MYD11A1 tile (and every LANCE NRT swath) into a private temp dir for each lake, ~10 s a scene, although a tile serves dozens of lakes. The 6-hourly refresh (2,400 s budget) managed 6–66 lakes a run against 680, so 170 lakes' `modis` product read `old` (>2 d) on /status and the attention count jumped to 227.
Fix (2026-10-11):
shared granule store `data/cache/modis_hdf/` (`_fetch_hdf_shared`: per-file flock, download to scratch, rename into place; pruned after 10 days at each refresh start); the service budget raised to 4,500 s (four 75 min runs a day). A lake with a cached tile now costs the pyhdf read plus the per-scene S3 register.
ISS-45 · Shape freshness is structural, not a fault (431 lakes "waiting")
What:
the status page marks a lake's `shape` old after 10 days without an accepted shoreline. The daily refresh (90 min budget, 2 workers, ~69 s a lake) reaches ~150 lakes a day, and only ~20–40 lakes a day get a clear, fully-covered scene accepted (the rest are cloud, partial swath, or scenes at a tile edge whose polygon is disjoint from the outline). 468 heal markers say `waiting_upstream`; the cells show as `waiting`, not attention.
Next:
more budget/workers for the daily refresh once the lane's backlog clears; a same-day tile mosaic (ISS-40) would turn the partial-swath rejections into shapes.
ISS-46 · Water level missing on drawn-down lakes: zero registry depth and a SWOT step that never finished — FIXED
What (Lake Powell #9, 2026-11-04):
two faults stacked. (a) `latest_known_elevation` and `/api/lake/{sid}/elevation` reject readings below `full_pool − max_depth_m − 5 m`; the registry has no depth for 517 of 722 lakes (`max_depth_m` NULL/0), so the floor sat 5 m under the registry level and every Powell reading since June (lake at 1095 m against 1102 m) was dropped — the lake page showed a June level and the 3D water plane sat at the last reading that passed. (b) The daily elevation refresh (`lakemonster-elevation`, pool step timeout 1,200 s) killed Powell's worker mid-SWOT every day since 10-08 (55 timeouts; #332, #210, #406, #330 … likewise), so the merged track was never rewritten and froze at 10-07.
Fix:
`plausible_elevation_window()` — full pool from the verified spillway, depth from the registry, else the hypsograph span, else 150 m; used by both readers. `refresh_one(swot_budget_s=…)`: the worker gives SWOT half the step timeout so the merge always runs. Powell's registry set to full pool 1,127.76 m / 171 m deep (was 1,102 / 0). Note Powell's hypsograph is the Wahweap-bay snapshot (166 km², 33 m) — pct-of-full is still meaningless until a whole-lake bathymetry exists (ISS-15).
ISS-47 · Dark dots over the lake in the 3D viewer: survey-time water surface kept as terrain — FIXED
What:
the terrain cloud masks only the *current* shoreline polygon. Inside the normal-pool outline but outside that polygon (Powell: 160 km², 53 holes), the DEM carries the water surface at survey time (~1,101.5 m, 2 m above today's 1,099.3 m) — rendered ×5 as a terrace of dark, water-coloured points over the lake. Bed substitution only helps where bathymetry exists (Powell: the bay).
Fix (`derive_terrain_pointcloud`):
inside the outline keep only cells that are land now — bed the bathymetry puts above the water, or DEM standing above the survey-time surface (median DEM over the current polygon). Underwater and survey-surface cells are dropped.
ISS-48 · Wide lakes' current shape flip-flops between tiles
What:
Powell's current polygon alternated between tiles (518 → 484 → 358 km², 8 parts, 53 holes) because a partial scene that misses water inside its footprint (turbid arms, glint) still passes the 0.4–1.6× area band and `|Δarea| > 1 %` makes it current; the thermal field and 3D view then show scattered fragments.
Fix (2026-11-04):
`decide_action_daily(observed_frac=…)` — a scene that saw < 90 % of the lake may not shrink the current shape by more than 10 %; it is archived as an outlier (evidence kept). Powell's 518 km² 07-27/10-05 shape promoted back to current by hand.
Next:
the same-day tile mosaic (ISS-40) is the real answer.
ISS-49 · Gauged lakes were shifted onto the SWOT datum, not the other way round — FIXED
What:
the merger reconciled a daily operator gauge with SWOT by shifting **the gauge** by the median same-day difference. Where SWOT's matched features are not the lake's own water surface, that moves the one trustworthy source onto a wrong datum. Lake Powell: USGS 09379900 read 3,533.7 ft (1,077.1 m) on 11-03 while the site served 1,099.3 m, and its three bay entries 1,106.8 / 1,117.0 / 1,120.5 m — the same reservoir, four different surfaces, all 22–43 m high. The SWOT offsets were +22.2 (#9), +20.8 (#2878), +39.5 (#2877), +43.0 m (#2876).
Fix (2026-11-05):
the **gauge is the datum** when it publishes a surveyed elevation; SWOT is shifted onto it, and dropped outright when the two differ by more than `_MAX_SWOT_GAUGE_OFFSET_M` (5 m) — that much daylight means the matched SWOT features are not this water. `_gauge_reads_absolute_elevation()` makes the distinction: a gauge whose own numbers fall inside the lake's plausibility window reads elevation; one that does not is stage on a local zero (USBR quotes Utah Lake in ft-compromise, +1,368.7 m; the Lake Tahoe gauge reads above the 6,223 ft lake datum, +1,888.8 m) and is still lifted onto SWOT, as before. Verified: Powell + 3 bays → 1,077.47 m (3,535 ft), Utah Lake 1,367.33 m, Tahoe 1,894.50 m, all unchanged or corrected.
Fleet scan of the shifts the old code applied:
156 lakes have both sources — 141 under 1 m (harmless), 4 at 1–2 m, 2 at 2–5 m, and 9 of 5 m or more. The nine large ones are the local-datum gauges (#258 +1,889, #857 +1,798, #820 +1,372, #10 +1,369, #868 +884, #742 +154, #245 −68.5, #2636 +11.2, #791 −5.3) and are now classified rather than guessed at. The nightly elevation refresh heals the rest.
ISS-50 · Lake Powell's 3D view: white fans over the water — FIXED
What:
the thermocline plane took **the single nearest sample's** depth per mesh vertex. On a lake whose mesh is far larger than the sampled area (Powell: 12 k vertices against 3.5 k thermal points strung along the main channel) adjacent vertices land in different Voronoi cells, so 0.75 m sits beside 15.9 m and the ×150 column exaggeration renders the step as a 2,200-unit wall — the white fans the operator saw over the lake. The water sheet and shore skirt made it worse by triangulating the single longest ring of a 28,561-point, 5-part, 53-hole outline, which earcut fills with spikes.
Fix:
`_thermoDepthsAt()` — inverse-distance weighting over the six nearest samples, falling back to the lake's median depth beyond a sample-spacing-derived radius, for both the mesh and its outline loop. `_lakeShapes()` — rings split at the null separators, each simplified (Douglas–Peucker, tolerance span/4000, min 3 m), nested by containment so islands become holes, every part getting its own `THREE.Shape`. Powell: 1 ring of 28,561 points → 26 shapes, outer contour 6,225 points; the skirt is built from the same simplified contour.
Also:
`lakes.max_depth_m`/`elevation_m` for the four Powell entries were 0 m / 1,102 m; set to 171 m / 1,127.76 m (3,700 ft) with `spillway_elevation_source` recorded. USBR hydrodata site 919 wired into all four configs, so pct-of-full comes from measured storage (6.19 MAF / 24.32 MAF = 25.5 %) instead of the Wahweap-bay hypsograph, which spans 1,069–1,102 m and read 0 %.
ISS-8 · Anchor dead band leaves ±1.5–3 °C persistent bias
What:
confidence-scaled nudge thresholds (1.5 °C med/high, 3 °C low) mean a stable model bias just under threshold persists (Rockport pattern). Display-time obs bias corrects the shown value; the model state stays biased.
Next:
consider a slow integral-style nudge (e.g. 0.3 °C/cycle toward anchor regardless of threshold) so state converges without churn.
ISS-9 · Legacy UTC remnants until natural re-push
What:
timestamp localization ships on change-detection: objects that rarely change (and pre-change day cubes/tier2_3d cached files) keep UTC strings until re-rendered/re-pushed.
Impact:
shrinking daily; day cubes refresh on the hourly timer.
Next:
none needed unless a stale page is user-visible; spot-check in a week.
ISS-11 · Confirm fleet tier-2 rebuild completion
What:
the full-fleet tier-2 rolling rebuild (started early 2026-08-15 evening) runs with a 4 h budget + requeue; leftovers heal via nightly timer/watchdog.
Next:
check `data/logs/tier2_rolling.log` + per-lake logs for failures; re-run stragglers with `--single`.
ISS-12 · lakemonster-web restarts still drop in-flight requests
What:
deploys restart uvicorn (`--no-reload`); each restart is a brief 502 blip behind go.7.fish. Mitigated 2026-08-15: peer session added `TimeoutStopSec=10` (was ~2 min drain) and the homepage now cold-serves in 15 ms from a persisted snapshot — but the blip itself remains.
Impact:
seconds of 502s per deploy; only matters when deploying during active use.
Next:
graceful reload (uvicorn workers + SIGHUP) or a blue-green pair behind the proxy; batch code changes into single restarts meanwhile.
ISS-13 · Learn per-lake diel air-temp bias from consensus residuals
What:
conditions-now air temp is a median of 3 estimators (station obs / metro-HRRR shore / WeatherKit); per-source residuals vs the consensus now accrue in `derived/air_temp_residuals.jsonl` per lake. Mountain-valley lakes show systematic hour-of-day model bias (inversion cold pools: metro +2–3 °C mornings — fleet comparison 2026-08-16: median agreement 1.3 °C, valley outliers to 4.7 °C).
Impact:
consensus already caps single-source errors; learned diel corrections would tighten the last ~1–2 °C.
Next:
after ~2 weeks of residuals, fit per-lake per-hour-of-day offsets for the metro source (extend `derive_met_bias.py`) and apply before the median.
ISS-14 · WWTP nutrient-loading context from EPA ECHO
What:
EPA ECHO REST + DMR services expose the 7 major CWA dischargers around Utah Lake (and equivalents fleet-wide) with monthly effluent flow + nutrient DMRs.
Impact:
would add seasonal nutrient-loading markers/context to the Algae page (monthly resolution — not per-scene attribution).
Next:
small fetcher caching per-lake NPDES DMRs; band markers on the algae graph; correlate seasonal loading vs bloom onset.
ISS-17 · Sea-level/tidal lakes never promote — `bed_elev_navd88_5m.tif` skipped (FIXED)
What:
`bathy publish` only wrote the NAVD88 bed-elevation raster when `pool_elev_m > 0`, but `promote_to_bathymetry` hard-requires it. Any water body whose surface elevation is ≤ 0 or whose lakemon record stored `elevation_m` as `0`/null (tidal/coastal/low ponds) got its bed DEM silently skipped and then permanently failed promote with the misleading "publish step must have failed". Pre-existing since ≥2026-08-02.
Affected (from logs):
St Johns River #3342 (Jacksonville tidal river, surface ≈ 0 m NAVD88 — parked/retried for a month), Stink Lake #13588, Crystal Springs #13649, Goode Ski Lake #14886.
Fix (2026-09-08, `~/bathy` `cli.py` publish):
treat surface elevation as an `Optional[float]` — `None` = genuinely unknown (skip), any *known* value including ≤ 0 produces the bed DEM (`bed = surface − depth`, correctly landing below the datum). A stored `0` is treated as missing and falls through to the authoritative USGS EPQS lookup. Validated: #3342 republished with `bed_elev_navd88_5m.tif` (surface 0 m, bed 0 to −11.9 m); unparked to complete. Whole class fixed.
Follow-up:
`promote_to_bathymetry`'s error text ("publish step must have failed") is misleading when publish succeeded but legitimately had no elevation anchor — reword, and consider a non-parking "no NAVD88 anchor" skip for genuinely unknown-elevation lakes.
ISS-16 · `get_shape` API serves degenerate geometry for some lakes (Chelan #90, Pickwick #2468) — see also ISS-20 (35 onboarded lakes affected locally)
What:
`vps.lakemon.com/.../get_shape&siteid=<id>` returns valid JSON but a degenerate polygon — Chelan #90 ≈ 105 m × 82 m (a real 80 km fjord), Pickwick #2468 ≈ 30 m — instead of the true lake extent (confirmed consistent across retries 2026-09-07). Left unchecked this yielded a 1–2 tile "chart" and an all-nodata DEM that crashed `fuse` downstream (that was the real cause of both lakes' bathy parks, on top of the ISS-15 OOM history they also had).
Not the deep_water shape-refresh pipeline:
zero entries for 90/2468 in `shape_refresh*.log` — the S2 refresh never touched them, so the bad geometry lives in the VPS lake database from the original seed/import.
Client mitigation (2026-09-07, `~/bathy` `sources/lakemon_api.py`):
`fetch_lake` now (a) retries `get_shape`/meta on empty/non-JSON bodies (the other intermittent failure mode — an empty body → JSONDecodeError), and (b) rejects a degenerate geometry (bbox < `BATHY_MIN_AOI_BBOX_M`, default 200 m, or empty) with a clear error, so it fails fast at `fetch-aoi` instead of publishing garbage. `navionics-dem` also hard-fails on zero valid pixels (see ISS-15).
Audit (2026-09-08):
`scripts/audit_lake_shapes.py` sweeps `get_shape` for every lake in the bathy queue and classifies OK / DEGENERATE / MISSING_GEOMETRY / NOT_FOUND / ERROR → `docs/triage/shape_audit.md`. Current active-queue result (25 lakes, 21 OK): **4 bad shapes, all need a server-side fix on vps.lakemon.com:** - #90 Lake Chelan — DEGENERATE (bbox ≈ 70×82 m) - #496 Lake Wallula — DEGENERATE (bbox ≈ 82×63 m) - #2468 Pickwick Lake — DEGENERATE (bbox ≈ 30×30 m) - #16998 Lake Higgins — NOT_FOUND (HTTP 404, no shape at all)
Next (server-side, needs operator):
regenerate the 4 shapes above in the vps.lakemon.com lake DB (create one for #16998). The client guards make all 4 fail fast + clean until then. To cover lakes not currently in the queue, extend the audit to the full onboarded fleet (it accepts `--siteids`, or wire it to the all_lakes list).
ISS-15 · Big-lake `navionics-dem` needs >124 GB — OOM loop crashed the host
What:
the bathy CLI's `navionics-dem` (sometimes `fetch-navionics`) step allocates ~124 GB+ on very large lakes (Superior, Okeechobee, Oahe, Sakakawea, Great Salt Lake, Ozarks, Hartwell #1029, Guntersville #1186, Chickamauga #959, …). The kernel OOM-killed it 1,889 times between Aug 15 and Sep 7 2026; each 6 h-backoff retry re-ran the same doomed job, and with ~20+ such lakes rotating the host OOMed every ~15–20 min until the desktop froze on 2026-09-07 and needed a reboot.
Impact:
host instability (the Sep 7 "crash"), the dedicated bathy lane spends most of its time on jobs that can never finish, and those lakes never get bathymetry.
Mitigation (2026-09-07):
systemd drop-in `~/.config/systemd/user/lakemonster-bathy-dispatcher.service.d/memory-cap.conf` sets `MemoryMax=64G` + `MemorySwapMax=0`, so a runaway job dies inside its own cgroup in ~2 min instead of starving the host.
Mitigation 2 (2026-09-07):
max-attempts park — after `LAKEMON_BATHY_MAX_ATTEMPTS` (default 10) failures the dispatcher parks the lake (status `parked`, queue score +inf; seeders can't resurrect it) and kicks `lakemonster-bathy-triage.service`, which runs a read-only headless Claude investigation and writes `docs/triage/bathy_<sid>.md` (2 h sweep timer catches stragglers). Review parks with `scripts/dispatch_bathy_queue.py --list-parked`; requeue after a fix with `--unpark <siteid>`.
Fix (2026-09-07, in `~/bathy`, uncommitted):
size-aware zoom via a shared `_memory_capped_zoom()` helper in `cli.py`. BOTH steps apply it: `fetch-navionics` estimates mosaic pixels from the AOI before fetching, and `navionics-dem` caps its zoom search too (a stale oversized z17 chart from an old attempt must not be loaded — that was the gap that re-OOMed Eufaula/St. Clair). Auto-drops the zoom (floor z11) to stay under `BATHY_MAX_MOSAIC_PX`. **Budget calibrated by measurement:** the DEM step's peak RSS is ~66 bytes/mosaic-px (siteid 307: 0.78 Gpx → 51.2 GB). Default budget **0.65 Gpx ≈ 43 GB peak**, ~20 GB headroom under the 64 GB cgroup cap. (An initial 1.5 Gpx guess was too high — Sakakawea #231 OOMed at z15/1.35 Gpx before recalibration.) Resulting zooms: Sakakawea z14, Superior z15, Okeechobee/Ozarks z16, Erie z15. Fusion terrain preview falls back across z17..z14; raw tile dict freed before the GeoTIFF write. - **Watch out (cost me a cascade 2026-09-07 21:45):** the helpers must sit ABOVE the `@main.command("fetch-navionics")` decorator block. If placed between the decorators and `def fetch_navionics_cmd`, the decorators bind to the helper instead, silently turning `fetch-navionics` into a no-op that fails EVERY lake with rc=1. `bathy fetch-navionics --help` exiting 0 is the quick smoke check.
Validated 2026-09-07 22:19:
Lake of the Woods #307 (12.3 Gpx at z17, the largest yet) completed end-to-end — auto-reduced to z15, real DEM (298M valid px, 66 rings), fused `bed_elev_30m.tif` + hypsograph uploaded. All OOM-class lakes (Erie, Mead, Michigan, Eufaula, St. Clair, Pickwick, …) unparked; queue is clear of parks.
Also hardened (`navionics-dem`):
now hard-fails when segmentation yields zero valid pixels instead of uploading an all-nodata DEM that crashes `fuse` downstream — surfaces the real cause at the source. (From the Pickwick #2468 triage report; that lake's 82nd attempt was a transient degenerate 1-tile AOI from the lakemon `SHAPE_URL` API, on top of its underlying ISS-15 OOM.)
Next / follow-ups:
- ~~Commit the `~/bathy` changes~~ — done (`947c851` size-aware zoom + tile-archive manifest, `518cf34`, `567bd09`; verified 2026-09-08). - Add an AOI-size sanity guard in `fetch-navionics` (reject/retry when `aoi_to_tile_range` yields a suspiciously tiny tile count vs the lake's known extent) — catches the degenerate-AOI glitch one step earlier; and investigate why `SHAPE_URL` returned a near-empty geometry for #2468 on that call. - Optional: 256 GB RAM would let the mid-tier reservoirs (Chickamauga-class) run at full z17 (1.2 m/px) instead of z16.
Resolved (28)
- **SWOT fill values poisoned shape elevations.** 174 obs rows carried the LakeSP fill (−999999999999 m, or fill-mixed granule means) and 31 shapes' `swot_elev_mean_m` read −1.4e10 m (shipping to the lake page's ≥3-pass override). `swot_ingest` now drops rows outside −500..9000 m / sigma outside 0..100 m (`_plausible_swot_rows`), aggregates ignore out-of-band rows, bad rows deleted and the 31 shapes recomputed. The per-row shape lookup (1–3 queries × ~500 rows × 482 lakes) is now one query per lake (`_ShapeResolver`, bisect).
- **tier2-days met-bias piggyback aborted fleet-wide on one lake.** The `derive_met_bias` loop in `refresh_tier2_days.py` had its try/except OUTSIDE the `for`, so #14537/#14628/#14886 (no `bbox.json`, onboarding incomplete) threw `FileNotFoundError` hourly and every lake sorted after them never got a met-bias refresh. Per-lake isolation + skip-with-warning for lakes without a bbox.
- ISS-20 code guards and the vps_sync per-object isolation / NaN-safe envelope (see ISS-20).
- ISS-21 (see above).
- **ISS-6 root cause found — disk-full was the bathy contour tileset rebuild.** `dispatch_bathy_queue.py` re-ran `build_bathy_vector_tiles.py` after EVERY completed lake: dump all ~865 lakes' isolines into one ~45 GB GeoJSONL in `/tmp/bathy_tiles_*`, tippecanoe the whole fleet, upload. The build crossed the dispatcher's 3600 s `subprocess.run` timeout on 2026-09-08 (62 min); a SIGKILLed child never runs `TemporaryDirectory` cleanup, so each hourly kill leaked 43 GB — 13 dirs / **546 GB in /tmp** by 17:20 (disk 42 %). Same mechanism as the 2026-08-14 ENOSPC. Fix: (a) 12 orphan dirs removed (→ 29 %); (b) builder rewritten as **incremental per-lake tiles** — each lake's isolines → tippecanoe → `bathy/{sid}/published/contours.mbtiles` (+ `.json` source stamp) uploaded to MinIO immediately, scratch deleted per lake; the fleet tileset is now a `tile-join --no-tile-size-limit` of those small files (`--lake`, `--fleet-only`, `--force-lakes` modes); (c) dispatcher cuts a lake's tiles right after it publishes (seconds) and flags a Redis-pending fleet merge that runs when the lane is idle or ≥30 min since the last good merge (`LAKEMON_BATHY_TILES_MIN_INTERVAL_S`), in its own process group with `killpg` on timeout (`LAKEMON_BATHY_TILES_TIMEOUT_S`, 2 h) — no more orphaned tippecanoe/tile-join; (d) builder + heal watchdog reap `bathy_tiles_*`/`bathy_lake_*`/`bathy_merge_*`/`swot_*` scratch older than 12 h; (e) heal watchdog summary now carries a `disk` block and logs `DISK ALARM` under 400 GB free / over 85 % (`LAKEMON_DISK_ALARM_*`); (f) per-lake refresh uses a fork process pool (thread pool serialized behind the GIL at ~4 lakes/min; now ~60/min — fleet backfill ≈ 15 min); (g) `~/bathy` SDB step leaked one `nav.tif` per run via bare `mkdtemp()` — scoped to a `TemporaryDirectory` (committed in `~/bathy`).
- **ISS-7 climatology builder scheduled:** `lakemonster-temp-climatology.timer` monthly on the 3rd 10:40 UTC (`ops/systemd/`, in `install.sh`).
- **ISS-10 fallback safety block** now carries `timezone`/`tz_abbrev` (`build_safety` in `vps_lake_sync.py`).
- **Service logs never rotated** (`data/logs/*.log`: web 172 MB, vps_sync 214 MB, process_queue 135 MB, 1.8 GB total). Added unprivileged `ops/logrotate/lakemonster.conf` (100 MB × 4, compress, copytruncate) run daily by `lakemonster-logrotate.timer`; first rotation applied.
- Side effect to know about: re-installing the anchor-correct timer with a new `OnCalendar` + `Persistent=true` fired one immediate catch-up run at 17:31 MDT on 2026-09-08 (harmless).
- Fleet water-level date audit (2026-08-27): 21/27 gauge lakes fully fresh; 6 "stale" lakes traced to USBR ITSELF not publishing past Aug 24 (verified at source — their QA lag, not ours). Format audit caught `nwis_stage` (22 lakes) missing from the date-only daily list → added; all 50 gauge lakes force-resnapshotted so chart dates and "as of" labels are correct immediately. Satellite-only lakes (122) sparse by nature. Lake 10's morning elevation refresh hit the 1200 s step timeout (requeue convention heals it; data was already source-fresh).
- Water-level "as of" off by one day (Rockport showed Aug 24 while holding the Aug 25 reading): daily gauge readings are stamped midnight-UTC and the localize-times walker shifted them to the previous local day. Fix: daily-cadence sources (usbr_daily, twdb_daily, nwis_dv, state_monthly) now ship DATE-ONLY ``t``/ ``as_of`` strings, which the localizer leaves untouched. Fleet transitions on each lake's next daily snapshot. (2026-08-26)
- Rockport GLM recalibrated FROM user ground truth (2026-08-26): two consecutive noon boat readings (75 °F) vs raw-model ~73 °F exposed a diel phase error; the 2024 cal had picked kw=0.6 at the sweep-grid EDGE against flat ECOSTRESS scores. Diel-scored resweep (16 runs, hour-matched vs boat + MODIS day/night) → kw=1.0 (`glm_calibration_2026.json`): midday cold bias halved (+2.3 → +1.7 °C raw; boat + day-MODIS agree independently), surface diel range now 3.8 °C. Next-noon forecast 74.4 °F vs the boat's consistent 75. Night-MODIS argues for lower kw but carries the documented narrow-reservoir night land-mixing cold bias — trusted the boat. Mixing params had NO diel effect (negative result).
- Rockport display 3 °F cold vs boat (75 °F noon 2026-08-25): raw median in `_thermal_obs_bias` let 2 night MODIS obs outvote 1 day scene (night pixels mix cooling shoreline land; 3-5 px on a narrow reservoir). Fix: class-balanced bias (mean of per-class medians: sat-day / sat-night / ground-truth) + the ground-truth ledger is now a first-class obs source. Display 72.3 → 75.6 °F; fleet spot-checks sane (10: +1.5 C n=22, 2: +1.9 n=7, 9: +0.5 n=18); smoke OK. (2026-08-25)
- Rockport fire release annotated + generic lake-events mechanism: curated `data/events/lake_events.yaml` + auto rapid-drawdown detector (`lakemonster/lakes/events.py`: 7-day mean rate ≥2× month median AND ≥12 cm/day, ≥7 days; curated suppresses overlapping auto). Shipped in `/api/lake/{sid}/elevation` → shaded bands on the ops chart, and in the sync elevation `current.events` for lakemonster.com. Context: regression (122 obs, 2018-24) attributes Rockport's 77 °F to ~+1.8 °F drawdown (−0.27 °F/m, p=1e-4) + ~+1.1 °F warm week. (2026-08-25)
- Sync wind moved to METRO PRIMARY (user directive) — `_fetch_safety_wind` now calls the self-hosted metro grid first (72 h wind, 48 h gusts+CAPE, no 429s), public 3-model ensemble demoted to fallback; weather_code filled by a single-point public side-call (None on 429, cosmetic). Cache v10. Verified on Rockport: full rows, no fallback warnings.
- Rockport headline temp GROUND-TRUTHED: boat thermometer 77 °F at 4 pm 2026-08-24 vs site 77 °F — exact match. Ledger: `data/ground_truth/water_temp.jsonl`. (2026-08-24)
- Utah Lake water level "as of Aug 20" on Aug 23 — layered lag: USBR posts day D−1 around midday, but `lakemonster-elevation.timer` ran once daily at 05:13 (before USBR posts), so the freshest local point was D−2; the user additionally caught the page before the morning vps sync pushed it (D−3 visible). Fix: second daily elevation refresh at 14:13 local (+ immediate manual run), so the site carries D−1 from mid-afternoon. The 15-min sync picks it up automatically. (2026-08-23)
- Algae history index truncation — the imagery hook's 120-day lookback rewrote every index with only 120 days (Powell 83 → 7; 107 lakes truncated). Lookback now bounds rendering only; smoke test asserts history depth. (2026-08-18)
- Lake grid 503s from Open-Meteo 429 rate limiting — metro fallback + last-good cache in `lakewind.fetch_wind_grid`. The earlier "transient" smoke failure on lakegrid[10] was this. (2026-08-18)
- Echo 3D viewer 54 °F pixels — tier-2 segment-stitch ghost profiles + per-tile ice-lock + level-correction blowout. Fixed in `build_tier2_rolling.py` / `build_tier2_day.py`.
- Rockport hot / Utah Lake cold headlines — mountain-cell shore met point, anchor race, destroyed MODIS series, MODIS night-scene anchor bias. Fixed: elevation-validated shore points, inline cycle anchoring, blended MODIS anchors, series rebuilt.
- Utah Lake 17 °C diel swing — recalibrated (kw 1.5, wind_factor 1.2, sw_factor 0.88) against dense MODIS series.
- 94 °F forecast peaks — lead-tapered obs bias + per-lake climatology soft cap (`build_temp_climatology.py`).
- Surface-grid scrubber 55–96 °F + vertical stripes — winsorization + same display corrections; de-striped `_nearest_anomaly` (radius mean, ±4 °C clamp).
- UTC times everywhere — shared `lakemonster/lakes/timezones.py`; all VPS payloads, advisories (day-qualified), 3D cubes (`ts_local`), viewer endpoints localized; all 194 lakes' zones in DB.
- Fleet v5 rescore (2026-08-17): 23,050 renders / 214 lakes, ~2.3 h. After: 17,174 history entries, 56 % good / 22 % dark (dark had dominated most clear lakes); 83 composites. Spot checks: sid 9 12→63 good, sid 956 6→33, sid 38 47→145; sid 10 (turbid) unchanged behavior. Post-run smoke FAIL on lakegrid[10] was a transient race with rescore disk churn — endpoint healthy (14 cells, 2.6 s) and rerun smoke OK.
- Lake Powell "very few readings" — two fleet-wide causes. (a) Tile-split coverage starvation: S2 MGRS tiles / Landsat path-rows each fail the coverage gate on big lakes (Powell: 484/566 unusable) → same-sensor scene stitching (`build_scene_composites`, window 3 d, cloud-clearing fill, sensor-classified `S2X_CMP_*`/`LC08X_CMP_*` stems), hooked into `update_algae_history` for every lake. (b) Absolute dark gate hid clear-water lakes’ archives fleet-wide (Powell bright_med ~62 year-round → 61/82 scenes hidden; many mountain lakes 60–95 %) → v5 lake-relative gate (0.7 × archive-median bright_med, floor 35). Powell: 12 → 63 good scenes. See `docs/ALGAE.md` decision log.
- Advisory wind 2–3× reality — single sample point in terrain-accelerated corner → area-scaled polygon grid median (`lakemonster/io/lakewind.py`), then 3-model ensemble median (best_match + ECMWF + ICON) after GFS phantom-storm false alarm. Wind grid ops page at `/lake/{sid}/windgrid`; spatial-variation risk line in advisories; `met_validation` onboarding step.