docs: add LLM analytics tracking design

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 13:08:10 +00:00
parent 6712942436
commit 703b940e9d
1 changed files with 84 additions and 0 deletions
@@ -0,0 +1,84 @@
+# LLM Analytics Tracking Design
+
+## Problem
+
+With 8 NPCs running overnight, LLM requests range from 67-309/hour with an upward trend. No visibility into which call types are responsible or what they cost.
+
+## Solution
+
+Track every LLM call in SQLite with per-type analytics surfaced in the admin panel.
+
+## Data Model
+
+New table `llm_call_log`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | INTEGER PK AUTOINCREMENT | |
+| tick | INTEGER | Game tick when call was made |
+| template_name | TEXT | Call type (backstoryAndDesires, socialNarration, batchedThoughts, desireGeneration, invention) |
+| input_tokens | INTEGER | Prompt tokens |
+| output_tokens | INTEGER | Completion tokens |
+| cost_usd | REAL | Calculated cost based on model pricing |
+| model | TEXT | Model used (for price lookup) |
+| retries | INTEGER | Number of retries before success (0 = first attempt worked) |
+| failed | INTEGER | 0 = succeeded, 1 = ultimately failed |
+| timestamp | TEXT | ISO timestamp |
+
+Index on `template_name` for fast aggregation.
+
+Tracking toggle stored in `metadata` table (key: `llm_tracking_enabled`, default `'1'`).
+
+## Server Instrumentation
+
+### openRouterClient.ts
+Change return type from `string | null` to structured result:
+`{ text: string | null, inputTokens: number, outputTokens: number, model: string, retries: number, failed: boolean }`
+
+### llmStatsService.ts (new)
+- `record(templateName, inputTokens, outputTokens, model, retries, failed, tick)` — computes cost, inserts row (if tracking enabled)
+- `getStats()` — aggregate SQL query returning per-type stats + totals
+- `setTrackingEnabled(enabled)` / `isTrackingEnabled()` — toggle persistence via metadata table
+
+### llmService.ts
+After each `generate()` call, record via `llmStatsService.record()`.
+
+### Model Pricing (llmConfig.ts)
+```
+trinity-large: input $0, output $0
+gpt-oss-120b: input $0.039/M, output $0.19/M
+```
+
+## Admin Panel UI
+
+### LeftPanel behavior
+- Admin tab active: panel widens to 680px
+- All other tabs: 340px
+
+### Two-column layout (independently scrollable)
+
+**Left column (existing):** Tunable constants editor, unchanged.
+
+**Right column (new):** LLM Analytics
+- Toggle: "LLM Tracking: ON/OFF"
+- Stats table per call type:
+  - Friendly label (Backstory, Social, Thoughts, Desires, Invention)
+  - Call count
+  - Retry % and Fail %
+  - Min / Avg / Max cost (USD, micro-dollar precision)
+  - Total cost
+- Totals row at bottom
+
+Data refreshes on tab open via socket request/response.
+
+## Socket Protocol
+
+- `admin-llm-stats` → `admin-llm-stats-result` (aggregated data)
+- `admin-toggle-llm-tracking` { enabled: boolean } → `admin-llm-tracking-toggled` { enabled: boolean }
+
+## Schema Migration
+
+- Bump CURRENT_SCHEMA_VERSION from 1 to 2
+- Migration 1→2: create `llm_call_log` table + insert tracking toggle into metadata
+- `initSchema()` updated to include new table for fresh worlds
+- Stats reset with `--new-world` (table created fresh)