diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 3e12790..1ff630b 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -18,8 +18,8 @@ - [x] **STOR-01**: API stores config snapshots in `router_config_snapshots` table with SHA256 hash - [x] **STOR-02**: Duplicate snapshots (same hash as previous) are skipped, no diff generated -- [ ] **STOR-03**: Snapshots retained for 90 days (configurable via `CONFIG_RETENTION_DAYS`) -- [ ] **STOR-04**: Older snapshots automatically deleted by retention cleanup +- [x] **STOR-03**: Snapshots retained for 90 days (configurable via `CONFIG_RETENTION_DAYS`) +- [x] **STOR-04**: Older snapshots automatically deleted by retention cleanup - [x] **STOR-05**: Snapshots encrypted at rest, accessible only through RBAC ### Diff & Parsing @@ -76,8 +76,8 @@ | COLL-06 | Phase 2: Poller Config Collection | Complete | | STOR-01 | Phase 1: Database Schema | Complete | | STOR-02 | Phase 3: Snapshot Ingestion | Complete | -| STOR-03 | Phase 9: Retention & Cleanup | Pending | -| STOR-04 | Phase 9: Retention & Cleanup | Pending | +| STOR-03 | Phase 9: Retention & Cleanup | Complete | +| STOR-04 | Phase 9: Retention & Cleanup | Complete | | STOR-05 | Phase 1: Database Schema | Complete | | DIFF-01 | Phase 5: Diff Engine | Complete | | DIFF-02 | Phase 5: Diff Engine | Complete | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index cc787a6..d37c40d 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -20,7 +20,7 @@ Decimal phases appear between their surrounding integers in numeric order. - [x] **Phase 6: History API** - REST endpoints for timeline, snapshot view, and diff retrieval with RBAC (completed 2026-03-13) - [x] **Phase 7: Config History UI** - Timeline section on device page with change summaries (completed 2026-03-13) - [ ] **Phase 8: Diff Viewer & Download** - Unified diff display with syntax highlighting and .rsc download -- [ ] **Phase 9: Retention & Cleanup** - 90-day retention policy with automatic snapshot deletion +- [x] **Phase 9: Retention & Cleanup** - 90-day retention policy with automatic snapshot deletion (completed 2026-03-13) - [ ] **Phase 10: Audit & Observability** - Audit event logging for all config backup operations ## Phase Details @@ -147,10 +147,10 @@ Plans: 1. Snapshots older than 90 days (default) are automatically deleted along with their associated diffs and changes 2. Retention period is configurable via `CONFIG_RETENTION_DAYS` environment variable 3. Cleanup runs on a scheduled interval without blocking normal operations -**Plans**: TBD +**Plans**: 1 plan Plans: -- [ ] 09-01: Retention cleanup scheduler and cascading deletion +- [ ] 09-01-PLAN.md — Retention cleanup service with APScheduler, configurable retention period, and cascading deletion ### Phase 10: Audit & Observability **Goal**: All config backup operations are logged as audit events for compliance and troubleshooting @@ -182,5 +182,5 @@ Note: Phase 9 depends only on Phase 3 and Phase 10 depends on Phases 3/4/5, so P | 6. History API | 2/2 | Complete | 2026-03-13 | | 7. Config History UI | 1/1 | Complete | 2026-03-13 | | 8. Diff Viewer & Download | 1/2 | In Progress| | -| 9. Retention & Cleanup | 0/1 | Not started | - | +| 9. Retention & Cleanup | 1/1 | Complete | 2026-03-13 | | 10. Audit & Observability | 0/1 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index beae16e..58a5ded 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,15 +3,15 @@ gsd_state_version: 1.0 milestone: v9.6 milestone_name: milestone status: completed -stopped_at: Completed 08-02-PLAN.md -last_updated: "2026-03-13T04:24:44.396Z" -last_activity: 2026-03-13 -- Completed 08-02 snapshot download +stopped_at: Completed 09-01-PLAN.md +last_updated: "2026-03-13T04:34:12Z" +last_activity: 2026-03-13 -- Completed 09-01 retention cleanup progress: total_phases: 10 - completed_phases: 8 - total_plans: 12 - completed_plans: 12 - percent: 92 + completed_phases: 9 + total_plans: 13 + completed_plans: 13 + percent: 100 --- # Project State @@ -21,14 +21,14 @@ progress: See: .planning/PROJECT.md (updated 2026-03-12) **Core value:** Operators can see exactly what changed on a router and when, with reliable config snapshots for download -**Current focus:** Phase 8: Diff Viewer & Download +**Current focus:** Phase 9: Retention & Cleanup -- COMPLETE ## Current Position -Phase: 8 of 10 (Diff Viewer & Download) -- COMPLETE -Plan: 2 of 2 in current phase -Status: Phase 08 complete -Last activity: 2026-03-13 -- Completed 08-02 snapshot download +Phase: 9 of 10 (Retention & Cleanup) -- COMPLETE +Plan: 1 of 1 in current phase +Status: Phase 09 complete +Last activity: 2026-03-13 -- Completed 09-01 retention cleanup Progress: [██████████] 100% @@ -60,6 +60,7 @@ Progress: [██████████] 100% | Phase 07 P01 | 3min | 2 tasks | 3 files | | Phase 08 P01 | 1min | 2 tasks | 3 files | | Phase 08 P02 | 1min | 1 tasks | 3 files | +| Phase 09 P01 | 2min | 2 tasks | 4 files | ## Accumulated Context @@ -94,6 +95,8 @@ Recent decisions affecting current work: - [Phase 08]: DiffViewer rendered inline above timeline (not modal) for context preservation - [Phase 08]: Line classification function for unified diff: +green, -red, @@blue, ---/+++ muted - [Phase 08]: Blob URL download pattern consistent with existing exportMyData and auditLogsApi.exportCsv patterns +- [Phase 09]: make_interval(days => :days) for parameterized PostgreSQL interval in retention cleanup +- [Phase 09]: 24h IntervalTrigger with 1h jitter for stagger; AdminAsyncSessionLocal for cross-tenant cleanup ### Pending Todos @@ -105,6 +108,6 @@ None yet. ## Session Continuity -Last session: 2026-03-13T04:24:44.393Z -Stopped at: Completed 08-02-PLAN.md +Last session: 2026-03-13T04:34:12Z +Stopped at: Completed 09-01-PLAN.md Resume file: None diff --git a/.planning/phases/09-retention-cleanup/09-01-SUMMARY.md b/.planning/phases/09-retention-cleanup/09-01-SUMMARY.md new file mode 100644 index 0000000..fec5ad3 --- /dev/null +++ b/.planning/phases/09-retention-cleanup/09-01-SUMMARY.md @@ -0,0 +1,98 @@ +--- +phase: 09-retention-cleanup +plan: 01 +subsystem: database +tags: [apscheduler, retention, postgresql, prometheus, cascade-delete] + +# Dependency graph +requires: + - phase: 01-database-schema + provides: router_config_snapshots table with CASCADE FK constraints +provides: + - Automatic retention cleanup of expired config snapshots + - CONFIG_RETENTION_DAYS env var for configurable retention period + - Prometheus metrics for cleanup observability +affects: [] + +# Tech tracking +tech-stack: + added: [] + patterns: [APScheduler IntervalTrigger for periodic maintenance jobs] + +key-files: + created: + - backend/app/services/retention_service.py + - backend/tests/test_retention_service.py + modified: + - backend/app/config.py + - backend/app/main.py + +key-decisions: + - "make_interval(days => :days) for parameterized PostgreSQL interval (no string concatenation)" + - "24h IntervalTrigger with 1h jitter to stagger cleanup across instances" + - "AdminAsyncSessionLocal (bypasses RLS) since retention is cross-tenant system operation" + +patterns-established: + - "IntervalTrigger pattern for periodic maintenance jobs (vs CronTrigger for scheduled backups)" + +requirements-completed: [STOR-03, STOR-04] + +# Metrics +duration: 2min +completed: 2026-03-13 +--- + +# Phase 9 Plan 1: Retention Cleanup Summary + +**Daily APScheduler job deletes config snapshots older than CONFIG_RETENTION_DAYS (default 90) with CASCADE FK cleanup of diffs and changes** + +## Performance + +- **Duration:** 2 min +- **Started:** 2026-03-13T04:31:48Z +- **Completed:** 2026-03-13T04:34:12Z +- **Tasks:** 2 +- **Files modified:** 4 + +## Accomplishments +- Retention service with parameterized SQL DELETE using make_interval for safe interval binding +- APScheduler IntervalTrigger running every 24h with 1h jitter for stagger +- Prometheus counter and histogram for cleanup observability +- Wired into main.py lifespan with non-fatal startup pattern + +## Task Commits + +Each task was committed atomically: + +1. **Task 1 (RED): Add failing tests** - `00bdde9` (test) +2. **Task 1 (GREEN): Implement retention service + config setting** - `a9f7a45` (feat) +3. **Task 2: Wire retention scheduler into lifespan** - `4d62bc9` (feat) + +## Files Created/Modified +- `backend/app/services/retention_service.py` - Retention cleanup logic, scheduler, Prometheus metrics +- `backend/tests/test_retention_service.py` - 4 unit tests for cleanup function +- `backend/app/config.py` - Added CONFIG_RETENTION_DAYS setting (default 90) +- `backend/app/main.py` - Wired start/stop retention scheduler into lifespan + +## Decisions Made +- Used make_interval(days => :days) for parameterized PostgreSQL interval (avoids string concatenation SQL injection risk) +- 24h IntervalTrigger with 1h jitter to stagger cleanup across instances +- AdminAsyncSessionLocal bypasses RLS since retention is a cross-tenant system operation + +## Deviations from Plan + +None - plan executed exactly as written. + +## Issues Encountered +None + +## User Setup Required +None - no external service configuration required. CONFIG_RETENTION_DAYS defaults to 90 if not set. + +## Next Phase Readiness +- Retention cleanup is fully operational, ready for phase 10 +- No blockers + +--- +*Phase: 09-retention-cleanup* +*Completed: 2026-03-13*