diff --git a/docs/plans/2026-03-09-world-persistence-design.md b/docs/plans/2026-03-09-world-persistence-design.md new file mode 100644 index 0000000..0e78788 --- /dev/null +++ b/docs/plans/2026-03-09-world-persistence-design.md @@ -0,0 +1,200 @@ +# World Persistence Design + +## Overview + +Add SQLite-based persistence so the server can stop and restart without losing world state. Event history (narration, NPC memory, stockpile log, inventions) is written immediately as it occurs. Entity state (NPCs, structures, relationships) is batch-saved every 30 seconds. A graceful shutdown save triggers on SIGINT/SIGTERM. + +## Storage + +- **Engine:** SQLite via `better-sqlite3` (synchronous, embedded, single-file) +- **Location:** `saves/default.db` (one file per region, only one region for now) +- **Schema versioning:** `schema_version` integer in metadata table, sequential migration functions + +## Database Schema + +### `metadata` +| Column | Type | Description | +|--------|------|-------------| +| key | TEXT PK | e.g. `schema_version`, `tick`, `created_at`, `last_saved_at` | +| value | TEXT | Stored as text, parsed by application | + +### `tiles` +| Column | Type | Description | +|--------|------|-------------| +| x | INTEGER | Tile x coordinate | +| y | INTEGER | Tile y coordinate | +| terrain | TEXT | Terrain type (grass/water/dirt/stone) | +| decoration | TEXT NULL | Decoration type | +| resource_type | TEXT NULL | Resource marker type | + +Primary key: `(x, y)` + +### `entities` +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK | Entity ID (preserved across saves) | +| type | TEXT | `npc`, `structure` | +| name | TEXT NULL | Entity name | +| x | REAL | Position x | +| y | REAL | Position y | + +### `components` +| Column | Type | Description | +|--------|------|-------------| +| entity_id | INTEGER | FK to entities | +| component_name | TEXT | Component type key | +| data | TEXT | JSON blob | + +Primary key: `(entity_id, component_name)` + +### `relationships` +| Column | Type | Description | +|--------|------|-------------| +| entity_id | INTEGER | Source NPC | +| target_id | INTEGER | Target NPC | +| value | REAL | -100 to 100 | +| interactions | INTEGER | Interaction count | +| last_interaction_tick | INTEGER | Last interaction tick | +| status | TEXT | Relationship status | + +Primary key: `(entity_id, target_id)` + +### `bonds` +| Column | Type | Description | +|--------|------|-------------| +| entity_a | INTEGER | First partner | +| entity_b | INTEGER | Second partner | +| type | TEXT | Bond type | +| formed_tick | INTEGER | When bond formed | + +### `narration_events` +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK AUTOINCREMENT | Event ID | +| tick | INTEGER | Game tick | +| type | TEXT | `social`, `proposal`, `invention` | +| entity_ids | TEXT | JSON array of entity IDs | +| outcome | TEXT | Outcome string | +| narration | TEXT | Narration text | + +### `memory_events` +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK AUTOINCREMENT | Event ID | +| entity_id | INTEGER | NPC this memory belongs to | +| tick | INTEGER | Game tick | +| type | TEXT | Event type | +| other_entity_id | INTEGER NULL | Other NPC involved | +| other_name | TEXT NULL | Other NPC name | +| detail | TEXT NULL | Event detail | +| old_tier | TEXT NULL | Previous relationship tier | +| new_tier | TEXT NULL | New relationship tier | + +### `stockpile_log` +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK AUTOINCREMENT | Entry ID | +| tick | INTEGER | Game tick | +| npc_name | TEXT | NPC name | +| action | TEXT | `dropoff` or `pickup` | +| item_id | TEXT | Item identifier | +| quantity | INTEGER | Amount | + +### `inventions` +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK AUTOINCREMENT | Entry ID | +| tick | INTEGER | Game tick | +| inventor_name | TEXT | NPC name | +| recipe_id | TEXT | Recipe identifier | +| item_data | TEXT | JSON blob with full invention data | + +## Save Strategy + +### Event-driven (immediate writes) +- Narration events → `narration_events` table +- NPC memory events → `memory_events` table +- Stockpile actions → `stockpile_log` table +- Inventions → `inventions` table + +### Periodic batch (every 30 seconds) +- All entities → `DELETE` + bulk `INSERT` in `entities` + `components` tables within a transaction +- All relationships → `DELETE` + bulk `INSERT` in `relationships` table +- Bond registry → `DELETE` + bulk `INSERT` in `bonds` table +- Metadata (tick count, timestamp) → `UPDATE` + +### Shutdown save +- `SIGINT` and `SIGTERM` handlers trigger a final entity state save, close the DB, then `process.exit(0)` + +### Map tiles +- Saved once on world creation +- Only re-saved if runtime tile modification is ever added + +## Load Strategy + +### Startup sequence +1. Parse CLI args for `--new-world` flag +2. If `--new-world` OR no `saves/default.db`: + - Create DB and init schema + - `generateMap()`, save tiles to DB + - `spawnInitialNPCs()`, first entity save +3. Else (existing save): + - Open DB, check `schema_version`, run migrations if needed + - Load tiles → reconstruct map (skip `generateMap()`) + - Load entities + components → populate ECS World with original entity IDs + - Load relationships + bonds → restore Maps and bond registry + - Load last N entries from event tables → populate in-memory buffers + - Restore tick counter from metadata + +### Entity ID preservation +- Entity IDs are restored exactly as saved +- `World.nextId` set to `max(saved entity IDs) + 1` + +### In-memory buffers +- On startup: `SELECT ... ORDER BY tick DESC LIMIT 50` populates each buffer +- During runtime: buffers work as before, but each new event also gets an INSERT +- DB holds full history; buffers are the hot working set + +## Serialization Details + +### Component JSON serialization +- Most components serialize directly via `JSON.stringify` +- `Map` types (`inventory`, `pairCooldowns`) → `Object.fromEntries()` for save, `new Map(Object.entries())` for load + +### Skipped on save (recalculated on load) +- `Movement.path` — Pathfinding recalculates; just save position +- `SocialState.phase` / active interaction state — NPCs resume from idle +- `BuildingState` active progress — Builder re-targets structure (structure's `buildProgress` IS saved) +- `playerControlled` entities — Players reconnect fresh; only NPCs and structures persist + +## CLI Usage + +- `npm -w server run dev` — loads existing save or creates new world +- `npm -w server run dev -- --new-world` — creates fresh world (overwrites existing save) + +## New Files + +- `server/src/persistence/database.ts` — SQLite connection, schema creation, migration runner +- `server/src/persistence/schema.ts` — Table definitions, migration functions +- `server/src/persistence/saveManager.ts` — Orchestrates periodic saves, shutdown saves +- `server/src/persistence/worldSerializer.ts` — Save/load map tiles and metadata +- `server/src/persistence/entitySerializer.ts` — Save/load entities, components, relationships, bonds +- `server/src/persistence/eventSerializer.ts` — Save/load narration, memory, stockpile, invention events + +## Modified Files + +- `server/src/game/GameLoop.ts` — Integrate SaveManager (init, periodic save timer, shutdown hook) +- `server/src/map/mapGenerator.ts` — Add ability to reconstruct map from saved tile data +- `server/src/llm/narrationService.ts` — Add DB write on `recordInteraction()`, load from DB on init +- `server/src/llm/eventMemoryService.ts` — Add DB write on `record()`, load from DB on init +- `server/src/industry/stockpileLog.ts` — Add DB write on `record()`, load from DB on init +- `server/src/industry/inventionTimeline.ts` — Add DB write on `record()`, load from DB on init +- `server/package.json` — Add `better-sqlite3` dependency + +## Key Principles + +- Persistence layer wraps existing systems; minimal changes to game logic +- ECS systems don't need to know about SQLite +- Serializers translate between ECS World state and database rows +- `better-sqlite3` is synchronous — no async overhead, no race conditions +- At current data volumes (50 NPCs, 4K tiles), saves take single-digit milliseconds