Files
cimtechniques-service-suite/docs/REBUILD-STATUS.md

247 lines
24 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Rebuild Status
## TL;DR
The full rebuild is **functionally complete and runs end-to-end against the
simulator**. All four planned phases are done: pure protocol core, transport +
simulator, domain + controller, the PySide6 UI, plus packaging (a working
standalone `.exe`) and docs. DA-12 is now **module #1 of the `cim_suite` monorepo**
— the suite shell (card launcher + `SuiteWindow`) and shared `core` package exist.
**DA-07 ("eLink") was rebuilt as module #2** and **IOModbus as module #3** (a
config-driven Modbus RTU master — same five-layer pattern, against its own simulator;
see Phases 89 below). The suite-wide **Instrument design system rollout is
complete** — all six phases, shell + all three modules (see Phases 1116 and
[BL-DS-P6](BACKLOG.md#bl-ds-p6--instrument-design-system-phase-6-iomodbus-adoption--launcher--p2--done)).
**1037 tests pass across `tests/core`, `tests/da12`,
`tests/da07`, `tests/iomodbus`, and `tests/shell`; ruff is clean.**
What remains is **verification against real DA-12, DA-07, and Modbus hardware**
guided by `docs/HARDWARE-VERIFICATION.md`.
All modules carry **contextual `ⓘ` hover help** distilled from the legacy DA-12
`Help.rtf` (replacing the old Station-tab help pane): a small `ⓘ` marker on grid
headers, Station setting rows, buttons, and the calibration dialog, with the text on
hover (IOModbus uses the same mechanism on its register grids + calibration dialog).
The DA-12/DA-07 simulators are seeded with their canonical station settings so the
Station tab is populated in `--simulate`. The DA-07 Station tab help (21 of 28 settings) was
keyed from a real DA-07 capture (`docs/samples/`); 7 device-specific mode codes and 3
DA-06-tainted DA-12 network settings are parked for a hardware/SME pass (BL-8 DONE /
BL-D3, BL-E4 TODO).
The suite shell now keeps opened modules **warm**: navigating away `suspend()`s a
module (releasing its cable) and returning `resume()`s it without re-running the slow
station load — **Refresh** is the explicit reload. App exit releases all warm ports.
See BL-1 (DONE) and BL-7 in the backlog.
An **unexpected** disconnect (cable yanked mid-session) is now reported honestly: the
shared transport carries a `connection_callback`, the serial reader fires it on a read
failure, and both controllers bridge it to `connectionChanged` — the label flips to
"Disconnected" and the pill greys, same as a deliberate stop. Report-only; auto-reconnect
is filed as BL-9. See BL-2/BL-5 (DONE).
**End-user distribution (BL-P1, 2026-06-08):** the suite now builds two artifacts a
non-admin user can run on a locked-down laptop — a **per-user no-admin installer**
(`%LOCALAPPDATA%\Programs`) and a **portable zip** — via `packaging\build.ps1`. The exe
carries the brand icon + version metadata, and a `READ-ME-FIRST.txt` walks users past the
SmartScreen warning. Code signing is wired but **inert** pending a certificate purchase
(**BL-P2** — OV / Azure Trusted Signing, not EV). See `docs/RELEASE-PACKAGING.md`.
App-wide **serial → model recognition + channel grouping** is now implemented (BL-D1
DONE). `cim_suite/core/sensor_models.py` maps any sensor serial to a model name and
channel type via prefix lookup, and clusters related channels by shared serial body.
A reusable `group_band_delegate.py` paints a left accent bar + separator at group
boundaries without inserting extra model rows. DA-12 gains a Model column on all four
data tabs and visual grouping on the Sensors tab; DA-07's Channels tab gains Serial +
Model columns with within-device grouping. Unrecognized serials show a blank Model and
are not grouped. The DA-07 per-channel serial is decoded as an optional
backward-compatible token on the `'E'` frame (HW-pending). **304 tests pass; ruff
clean.**
A **DA-07 polish pass** (2026-06-04) followed: the Devices tab now shows the **full
assembled serial** (station prefix + 6-hex + `"00"`, edited via a dialog) and a **Type
dropdown** of catalog model labels (e.g. CS-31); two previously-deferred legacy tabs
were revived — **Alarm** (PA-series alarm-indicator group editor, `M`/`N`/`Q` frames)
and **Traffic** (read-only RS-485 monitor, `Y` frames); and the **Channels** tab is now
gated until a device row is selected (Export-All still covers all devices via a new
optional `export_sheets()` plural tab hook). The `M`/`N`/`Y` layouts, the serial-prefix
label match, and the device-type catalog id↔label mapping are reconstructed from the VB6
and verified against the **simulator only** (HW-pending — see `HARDWARE-VERIFICATION.md`
DA-07 polish-pass items and BL-E2). The legacy Pictures tab remains deferred (binary
`.frx` assets). **444 tests pass; ruff clean.**
A **universal right-click Copy** was added suite-wide (2026-06-04): a single shared
helper (`cim_suite/core/ui/copy_menu.py`) provides a right-click "Copy" context menu on
table cells (cell / row / cell-with-header / selection block), the DA-07 traffic log,
and readable labels, wired via `TableTab` and a per-window `enable_copy_in`. All three
main windows and seven dialogs opt in. No protocol or model changes — view-layer only.
**467 tests pass; ruff clean.** See
`docs/superpowers/specs/2026-06-04-universal-right-click-copy-design.md`.
The DA-12 **`{F}` station-health signals** were surfaced (BL-D6, 2026-06-04).
`decoder._decode_status` was reconciled with the firmware-authoritative byte layout
(`docs/da12c_status.py::parse_f_frame`): `StationStatus` now carries named fields
(active-sensor count, a big-endian station clock, per-interval throughput counters,
a free-running comm byte, and a little-endian buffered-`record_count` tail) instead
of an opaque counter list, and the BL-D4 server-link monitor was rewired to the named
fields. The status bar (status-bar-only by design) now shows **Sensors N**, **Buffered
N** (tinted amber when the upload backlog grows), the station clock, and a live **`⚡`
activity indicator** with the throughput counters on hover. The `F`-frame layout is now
aligned to the firmware source but its HW-verification flag stays until a real `{F}` is
captured. **470 tests pass; ruff clean.** See
`docs/superpowers/specs/2026-06-04-da12-station-health-signals-design.md`.
A **per-sensor History dialog** was added to the DA-12 Sensors tab (BL-D5, 2026-06-05):
double-click a row (or right-click → **"Show history…"**) to open a live QtCharts trend
chart (Average + Current series) seeded by a `{c}``{J}` buffered-history backfill,
with alarm-limit guide lines and gap/dropout markers. The data table exports to `.xlsx`
via the shared export engine; the chart exports to PNG. This is the first trend/history
view the tool has ever provided. The `{J}` decode follows the firmware reference
(`docs/da12c_status.py`) and is pending a live hardware capture to retire the HW-
verification flag (see `docs/HARDWARE-VERIFICATION.md` section 6).
Two small cross-module items landed (2026-06-05): **BL-S3** — a one-time DA-12 data
migration that **moves** legacy `DACal.csv` + `DA-Logs\` from the pre-reshape Roaming
data dir into the current Local one (no-clobber, flag-guarded, run once from
`Da12Module.create_widget`; config was already in the right place, so only data moved).
**BL-I3** — IOModbus enum/digital register cells now edit via a strict pick-list
`QComboBox` that preserves off-list device values; view-layer only,
the chosen label routes through the existing write path. (Since Phase 6 the channels
grid keeps `PicklistDelegate` — rebased onto the kit `InstrumentDelegate`, off-list
insertion retained — while Device Settings moved to the kit `SettingsList`, whose
choice editors carry the same off-catalog preservation in the kit `SettingsDelegate`.) **593 tests pass; ruff clean.**
See `docs/superpowers/specs/2026-06-05-config-migration-and-picklist-dropdown-design.md`.
**IOModbus catalog device management (Spec #1)** landed 2026-06-05: a JSON user layer
merged on load lets users add or override devices at runtime without a rebuild; full
import/export of the legacy `IOModbus.txt` format; a **Catalog** menu in the IOModbus
window (Import / Export / Manage User Devices); and lenient import validation that skips
structurally-invalid devices. User-layer devices shadow factory devices by id. The rich
per-field device **editor** (the real IOBuilder revival) is deferred to Spec #2. **653
tests pass; ruff clean.** See BL-I2 in `docs/BACKLOG.md`.
The **Manage User Devices** dialog was replaced by a read-only **Supported Devices**
browser (Catalog → Supported Devices…): a sortable table listing every factory + user
device with columns Name / ID / Manufacturer / Min FW / Channels / Settings / Origin —
Delete is limited to user-layer rows, and factory rows that have been overridden by a
user entry are marked "Factory (overridden)". **662 tests pass; ruff clean.**
- **Device settings repository** (2026-06-06): suite-wide SQLite store
(`%LOCALAPPDATA%\CIMTechniques\Service Suite\devices.db`, `devices.sim.db` under
`--simulate`) change-logs DA-12/DA-07 configuration settings keyed by MAC, with a
"Setting history…" dialog (per-row + whole-device), with right-click revert to any
recorded value (BL-R1, 2026-06-08).
> **Open work items** (tweaks, fixes, planned features) live in
> **[`docs/BACKLOG.md`](BACKLOG.md)** — this file is the point-in-time status; the
> backlog is the live to-do list. Keep them in sync: when a backlog item is
> finished, reflect it here if it changes the overall status.
## What works right now
```powershell
.venv\Scripts\python -m cim_suite.shell.app --module da12 --simulate
```
launches the DA-12 tool inside the suite shell: 5 tabs (Station, Sensors, Alarm
Limits, Statistics, Calibration), live updating values, inline editing that
round-trips to the (simulated) station, a working calibration dialog that writes
scale/offset and logs history, and per-sensor measurement logging. Any data tab can
be exported to an Excel `.xlsx` workbook ("Export This Tab" / "Export All Tabs").
Running without `--module` opens the suite launcher landing page (a card grid).
A standalone build also runs with no Python/VB6/OCX installed:
`packaging\dist\CIM-Service-Suite\CIM-Service-Suite.exe --module da12 --simulate`.
## Phase-by-phase
| Phase | Status | Notes |
|---|---|---|
| 0 — Scaffolding | ✅ | pyproject, venv, pytest/ruff |
| 1 — Protocol core | ✅ | framing/codecs/decoder/encoder; 33 tests |
| 2 — Transport + domain | ✅ | pyserial + simulator; models/calibration/logger/controller |
| 3 — PySide6 UI | ✅ | 5 tabs, dialogs, toolbar, status bar; app boots |
| 4 — Packaging + docs | ✅ | PyInstaller spec (built & verified), Inno Setup script, RUNNING.md, HARDWARE-VERIFICATION.md |
| 5 — Suite monorepo reshape | ✅ | DA-12 → `cim_suite/modules/da12/`; shared `cim_suite/core/` (theme, TableTab, transport, codecs, generic config, Module contract); `cim_suite/shell/` (card launcher + SuiteWindow); suite-wide packaging (`suite.spec``CIM-Service-Suite.exe`); 88 tests pass across tests/core, tests/da12, tests/shell |
| 6 — Dashboard service-cable detection | ✅ | Suite launcher scans COM ports on dashboard entry; detected service cables (FTDI VID 0x0403 flagged) shown in an inline `CablePanel`; module entry gated until one is selected; `NoCableDialog` warns with Rescan/Quit when none are present; selected source (COM port or simulator) handed to the opened module. 112 tests pass. See `docs/superpowers/specs/2026-06-02-dashboard-service-cable-detection-design.md`. |
| 7 — Spreadsheet export | ✅ | Any data tab exports to an Excel `.xlsx` (metadata block + grid as shown, incl. alarm colors); "Export This Tab" / "Export All Tabs" toolbar actions. Suite-wide engine `cim_suite/core/export/` + reusable `add_export_actions` helper; `TableTab.export_sheet()` so new columns export for free. 127 tests pass. See `docs/EXPORT.md` and `docs/superpowers/specs/2026-06-02-spreadsheet-export-design.md`. |
| 8 — DA-07 module #2 | ✅ | DA-07 ("eLink") rebuilt against the simulator following the DA-12 pattern. Its own pure protocol layer (different wire format: `~<payload><checksum>\r`, little-endian, IEEE-754 floats), Station→Device→Channel domain models, simulator, and 4-tab UI (Station/Devices/Channels/Calibration). Registered in the shell; `--module da07 --simulate` boots. `core` reused as-is. 229 tests pass; ruff clean. **2026-06-03: fixed the DA-07 frame-per-ACK handshake** (rebuild ACK'd nothing, so real hardware loaded only the first frame — the simulator hid it; see HARDWARE-VERIFICATION DA-07 + BL-E1). See `docs/superpowers/specs/2026-06-02-da07-service-tool-rebuild-design.md`. |
| 9 — IOModbus module #3 | ✅ | IOModbus rebuilt against the simulator — a **standard, config-driven Modbus RTU master** (unlike the DA-12/DA-07 streams). Pure protocol layer (Modbus CRC-16, the 11 register data-type codecs, an 8-byte request builder, a transaction-scoped `ResponseAssembler`, and an `IOModbus.txt` catalog parser bundled as a resource), an in-memory Modbus-**slave** simulator, domain models (catalog → discovered devices → live register cells), a timer-stepped **scan + poll** controller (request/response with an `_awaiting` interlock), calibration + logging, and a single-screen UI (comm toolbar / Available Devices / Device Settings + I/O Channels grids / User Alerts). Registered in the shell; `--module iomodbus --simulate` boots and auto-scans the seeded devices. `core` reused as-is. 88 IOModbus tests; full suite 408 green; ruff clean. See `docs/superpowers/specs/2026-06-03-iomodbus-service-tool-rebuild-design.md`. |
| 10 — IOModbus catalog management (Spec #1) | ✅ | JSON user layer merged on load; full import/export of the legacy `IOModbus.txt` format; Catalog menu (Import / Export / **Supported Devices…**); lenient import validation (skips structurally-invalid devices). User-layer devices shadow factory devices by id. The former Manage-User-Devices dialog replaced by a read-only **Supported Devices** browser — sortable table, all factory + user devices, Origin column, Delete gated to user rows, overridden factory rows flagged. Rich per-field editor deferred to Spec #2. 662 tests; ruff clean. See `docs/superpowers/specs/2026-06-05-iomodbus-catalog-device-management-design.md`. |
| 11 — Instrument design system Phase 1 (foundation) | ✅ | Two-theme token system (light/dark), IBM Plex Sans/Mono fonts bundled, regenerated QSS from the token layer, theme manager with persistence, all painters migrated to live tokens, launcher dark/light toggle. Phases 26 (frameless chrome, component kit, per-module adoption, launcher redesign) pending — see `docs/superpowers/specs/2026-06-10-instrument-design-system-rollout-design.md`. |
| 12 — Instrument design system Phase 2 (window chrome) | ✅ | Merged 2026-06-10. Frameless SuiteWindow (PySideSix-Frameless-Window 0.8.1) with the Instrument title bar (logo, breadcrumb, theme toggle, 42px window buttons); 2px brand accent strip; declarative StatusFooter adopted by DA-12/DA-07/IOModbus (legacy ConnPill QSS retired); Instrument chrome + breadcrumb titles on all 13 child dialogs; chrome-framed confirm/info/warning dialogs replacing QMessageBox. Frozen-exe selftest re-verified with the new frameless dependency (all three modules exit=0; no extra hiddenimports needed). Manual Windows 11 snap/drag/DPI checklist outstanding — tracked in `docs/superpowers/plans/2026-06-10-instrument-phase2-chrome.md` Task 11 Step 3. Phases 36 pending. |
| 13 — Instrument design system Phase 3 (component kit) | ✅ | Merged 2026-06-10. The core component kit at `cim_suite/core/ui/kit/`: InstrumentDelegate (cell kinds, §1.2 status tags, edge bars, alarm tint, hover pencil chip, styled editor + validators, Tab-move, write-feedback flash), suite-wide single-click editing via TableTab (checkable columns now render as ToggleSwitches), UnitsHeaderView (microcaps headers + units line; ⓘ markers retired from headers), InstrumentTabWidget (microcaps tabs, closes BL-DS3), SummaryStrip, SettingsList, ActivityLogCard, Sidebar, tinted SVG icons. Bonus: fixed the QSS suppression of item-brush backgrounds and two pre-existing rebuild write-storms (hardware-relevant; see BL-DS-P3). Frozen-exe selftest re-verified (all three modules exit=0 with the bundled SVGs + QtSvg). Module adoption is Phases 46; the Phase 2 manual Win11 snap/DPI checklist remains the only open chrome item. Plan: `docs/superpowers/plans/2026-06-10-instrument-phase3-component-kit.md`. |
| 14 — Instrument design system Phase 4 (DA-12 adoption) | ✅ | Merged 2026-06-11. DA-12 fully adopted on the Instrument system. §5.2 toolbar (`← Suite` button, `ConnectionChip`, confirm-guarded "Station commands ▾" menu, primary Connect/Disconnect); Station tab → `SettingsList` with spec §5.5 groups (Identity/Communication/Sensors/Alarms & beeper/OP05/Advanced) driven by curated metadata in `da12/ui/station_settings_meta.py`; Sensors/Alarm Limits/Statistics/Calibration adopt the delegate kit (status tags, `ALARM_ROW_ROLE` alarm-only row tint, units headers, summary strips, toggle for Enabled, resolve-on-echo write feedback); history dialog restyled per §5.10. BL-DS-P3 carry-overs closed: live-refill editor guard, group-band edge-bar precedence (DA-07 picks this up too), `SettingsDelegate` exported. Two follow-ups deferred: [BL-DS4](#bl-ds4--sim-disconnect-has-no-toolbar-reconnect-path--p3--todo) (SIM reconnect path) and [BL-DS5](#bl-ds5--toolbar-responsiveness-below-1180px--p3--todo) (toolbar compaction). Phases 5 (DA-07) and 6 (IOModbus + launcher) remain. See [BL-DS-P4](BACKLOG.md#bl-ds-p4--instrument-design-system-phase-4-da-12-adoption--p2--done). 960 tests pass; ruff clean. |
| 15 — Instrument design system Phase 5 (DA-07 adoption) | ✅ | Merged 2026-06-11. DA-07 fully adopted on the Instrument system. §5.2 toolbar (`← Suite`, `ConnectionChip` fed COM/SIM labels, confirm-guarded "Station commands ▾" holding all four rare commands, primary Connect/Disconnect); Station tab → `SettingsList` with groups Identity/Network/Server/Timing & polling/Measurement/Advanced from `da07/ui/station_settings_meta.py` (wire-governed read-only; only Poll Devices is a toggle — unverified mode codes stay numeric); Devices tab per spec §6 (16 slots always visible, quiet empty slots via the new kit `QUIET_ROLE`, no zebra, status tags, type-cell add affordance); Channels/Alarm/Calibration adopt the kit (status tags + alarm rows, Active toggles, mono addresses, units, summaries, resolve-on-echo write feedback). BL-DS-P3 carry-over closed: `ComboBoxDelegate` rebased onto `InstrumentDelegate`. Phase 6 (IOModbus + launcher) remains. See [BL-DS-P5](BACKLOG.md#bl-ds-p5--instrument-design-system-phase-5-da-07-adoption--p2--done). 1004 tests pass; ruff clean. |
| 16 — Instrument design system Phase 6 (IOModbus adoption + launcher) | ✅ | Merged 2026-06-11 — **closes the Instrument rollout; all six phases shipped.** IOModbus §5.2 toolbar (`← Suite`, `ConnectionChip`, the legacy Catalog menubar folded into a "Catalog ▾" toolbar menu, data exports in their own "Export ▾" menu via the new shared `add_export_menu_actions`, primary Connect/Disconnect; themed toolbar 1185px vs the 1240px budget, regression-asserted — closes BL-DS5); device list → kit `Sidebar` (§5.7) with bus-parameters footer; User Alerts → kit `ActivityLogCard` (§5.8); Device Settings → grouped kit `SettingsList` (§5.5) with RO chips and choice rows; Channels grid kit kinds + summary + §5.6 echo-resolved write feedback; launcher redesigned per §5.9 (header, cable-card port rows, 3-up tool cards). A PySide6 export-action slot-lifetime bug fixed (closure slots → bound-method slots crash; regression-tested), and the kit `SettingsDelegate` now preserves off-catalog values in choice editors suite-wide. `docs/DESIGN-SYSTEM.md` rewritten around the Instrument spec. See [BL-DS-P6](BACKLOG.md#bl-ds-p6--instrument-design-system-phase-6-iomodbus-adoption--launcher--p2--done). 1037 tests pass; ruff clean. |
## What I could NOT do without hardware (your turn)
Work through `docs/HARDWARE-VERIFICATION.md` with a DA-12 connected. The five
flagged items, each isolated to a small, clearly-marked spot in the code:
1. **Station clock epoch/format** — verify Set Clock → read-back. Fix in
`protocol/codecs.py` clock helpers if off.
2. **Sensor type codes + Calc enum** — compare against the legacy tool.
3. **16-channel message/letter mapping** — confirm reads (`A/G/H/D/E`) and each
write command actually changes the station.
4. **`F` status frame layout** — capture a real frame; adjust
`decoder._decode_status`.
5. **Save real frames as test fixtures** (no usable captures existed in the repo —
`Exceptions.txt`/`VB187.tmp` are a crash log and an old form backup).
## Deliberately dropped (documented in the spec, restorable)
Wireless sensors, OP-05 annunciator, the Debug tab, and factory serial-number
setting — with exact legacy source locations in
`docs/superpowers/specs/2026-06-01-da12-service-tool-rebuild-design.md`
("Dropped / Deferred Features").
## Code review
A senior-reviewer pass was run on the whole branch. It confirmed the outbound
command mapping, 'A'-record field order, calibration math, logger format, and Qt
threading are faithful to the VB6 source. Two real protocol bugs it found are
**fixed** (with regression tests):
- Sensor **scale/offset** were wrongly treating `8000` as the no-value sentinel;
the legacy `AddFloat` has no sentinel there. Added `codecs.decode_float`.
- `encode_scaled` truncated toward zero; VB `Int()` floors toward −∞ (matters for
negative calibration offsets). Now uses `math.floor`.
## Known gaps / nice-to-haves (low priority)
*(These are candidates for [`BACKLOG.md`](BACKLOG.md) — promote any you intend to
act on into a tracked backlog item.)*
- **Type-F status signals** are now surfaced in the status bar (BL-D6): active-sensor
count, buffered-upload backlog, station clock, and the per-interval throughput
counters (on the activity-indicator tooltip). The decode is aligned to the firmware
layout but the F-frame HW-verification flag remains until a real capture confirms it
(HARDWARE-VERIFICATION #4). Per-interval counters are shown as activity/deltas, not
accumulated cumulative totals as the legacy did (intentional — the firmware zeroes
them each frame).
- Calc column as a labeled dropdown (currently a numeric text field).
- **Sensors tab Input + Refresh columns restored** (BL-D2, 2026-06-02): raw value and seconds-since-update columns re-added; Refresh cell tinted by staleness via new `STALENESS` theme token.
- App icon (PyInstaller spec has `icon=None`).
- Code-signing the exe/installer (commented hook in `installer.iss`).
## How to pick up
1. `docs/RUNNING.md` — setup, run, test, build.
2. `docs/HARDWARE-VERIFICATION.md` — the hardware checklist.
3. Branch: `main` is the Python rebuild mainline; the VB6 baseline is preserved
per-module under `cim_suite/modules/<id>/legacy/` (read-only).
## Next direction
The monorepo reshape is done, the first three VB6 tools are rebuilt (DA-12, DA-07,
IOModbus), and the **Instrument design system rollout is complete** — all six phases,
shell/launcher + all three modules (Phases 46, 2026-06-11 —
[BL-DS-P4](BACKLOG.md#bl-ds-p4--instrument-design-system-phase-4-da-12-adoption--p2--done),
[BL-DS-P5](BACKLOG.md#bl-ds-p5--instrument-design-system-phase-5-da-07-adoption--p2--done),
[BL-DS-P6](BACKLOG.md#bl-ds-p6--instrument-design-system-phase-6-iomodbus-adoption--launcher--p2--done)).
The outstanding hardware milestone is **verification** of all three modules
(`docs/HARDWARE-VERIFICATION.md`). Further VB6 tools follow the same pattern — drop the
source into `cim_suite/modules/<app>/legacy/`, then spec → plan → implement; `core` grows
only as real sharing reveals itself. Design and sequencing: **`docs/SUITE-ARCHITECTURE.md`**.