the-other-dude/.planning/codebase/ARCHITECTURE.md

Architecture

Analysis Date: 2026-03-12

Pattern Overview

Overall: Event-driven microservice architecture with asynchronous pub/sub messaging

Key Characteristics:

• Three independent microservices: Go Poller, Python FastAPI Backend, React/TypeScript Frontend
• NATS JetStream as central event bus for all inter-service communication
• PostgreSQL with Row-Level Security (RLS) for multi-tenant isolation at database layer
• Real-time Server-Sent Events (SSE) for frontend event streaming
• Distributed task coordination using Redis distributed locks
• Per-tenant encryption via OpenBao Transit KMS engine

Layers

Device Polling Layer (Go Poller):

• Purpose: Connects to RouterOS devices via binary API (port 8729), detects status/version, collects metrics, pushes configs, manages WinBox/SSH tunnels
• Location: poller/
• Contains: Device client, scheduler, SSH relay, WinBox tunnel manager, NATS publisher, Redis credential cache, OpenBao vault client
• Depends on: NATS JetStream, Redis, PostgreSQL (read-only for device list), OpenBao
• Used by: Publishes events to backend via NATS

Event Bus Layer (NATS JetStream):

• Purpose: Central publish/subscribe message broker for all service-to-service communication
• Streams: DEVICE_EVENTS, OPERATION_EVENTS, ALERT_EVENTS
• Contains: Device status changes, metrics, config change notifications, push rollback triggers, alert events, session audit events
• All events include device_id and tenant_id for multi-tenant routing

Backend API Layer (Python FastAPI):

• Purpose: RESTful API, business logic, database persistence, event subscription and processing
• Location: backend/app/
• Contains: FastAPI routers, SQLAlchemy ORM models, async services, NATS subscribers, middleware (RBAC, tenant context, rate limiting)
• Depends on: PostgreSQL (via RLS-enforced app_user connection), NATS JetStream, Redis, OpenBao, email/webhook services
• Used by: Frontend (REST API), poller (reads device list, writes operation results)

Data Persistence Layer (PostgreSQL + TimescaleDB):

• Purpose: Multi-tenant relational data store with RLS-enforced isolation
• Connection: Two engines in backend/app/database.py
  • Admin engine (superuser): Migrations, bootstrap, admin operations
  • App engine (app_user role): All tenant-scoped API requests, RLS enforced
• Row-Level Security: SET LOCAL app.current_tenant set per-request by get_current_user dependency
• Contains: Devices, users, tenants, alerts, config backups, templates, VPN peers, certificates, audit logs, metrics aggregates

Caching/Locking Layer (Redis):

• Purpose: Distributed locks (poller prevents duplicate device polls), session management, temporary data
• Usage: redislock package in poller for per-device poll coordination across replicas

Secret Management Layer (OpenBao):

• Purpose: Transit KMS for per-tenant envelope encryption, credential storage access control
• Mode: Transit secret engine wrapping credentials for envelope encryption
• Accessed by: Poller (fetch decrypted credentials), backend (re-encrypt on password change)

Frontend Layer (React 19 + TanStack):

• Purpose: Web UI for fleet management, device control, configuration, monitoring
• Location: frontend/src/
• Contains: TanStack Router, TanStack Query, Tailwind CSS, SSE event stream integration, WebSocket tunnels
• Depends on: Backend REST API, Server-Sent Events for real-time updates, WebSocket for terminal/remote access
• Entry point: frontend/src/routes/__root.tsx (QueryClientProvider, root layout)

Data Flow

Device Status Polling (Poller → NATS → Backend):

1. Poller scheduler periodically fetches device list from PostgreSQL
2. For each device, poller's Worker connects to RouterOS binary API (port 8729 TLS)
3. Worker collects device status (online/offline), version, system metrics
4. Worker publishes DeviceStatusEvent to NATS stream DEVICE_EVENTS topic device.status.{device_id}
5. Backend subscribes to device.status.> via nats_subscriber.py
6. Subscriber updates device record in PostgreSQL via admin session (bypasses RLS)
7. Frontend receives update via SSE subscription to /api/sse?topics=device_status

Configuration Push (Frontend → Backend → Poller → Router):

1. Frontend calls POST /api/tenants/{tenant_id}/devices/{device_id}/config with new configuration
2. Backend stores config in PostgreSQL, publishes ConfigPushEvent to OPERATION_EVENTS
3. Poller subscribes to push operation events, receives config delta
4. Poller connects to device via binary API, executes RouterOS commands (two-phase: backup, apply, verify)
5. On completion, poller publishes ConfigPushCompletedEvent to NATS
6. Backend subscriber updates operation record with success/failure
7. Frontend notifies user via SSE

Metrics Collection (Poller → NATS → Backend → Frontend):

1. Poller collects health metrics (CPU, memory, disk), interface stats, wireless stats per poll cycle
2. Publishes DeviceMetricsEvent to DEVICE_EVENTS topic device.metrics.{type}.{device_id}
3. Backend metrics_subscriber.py aggregates into TimescaleDB hypertables
4. Frontend queries /api/tenants/{tenant_id}/devices/{device_id}/metrics for graphs
5. Alternatively, frontend SSE stream pushes metric updates for real-time graphs

Real-Time Event Streaming (Backend → Frontend via SSE):

1. Frontend calls POST /api/auth/sse-token to exchange session cookie for short-lived SSE bearer token
2. Token valid for 25 seconds (refreshed every 25 seconds before expiry)
3. Frontend opens EventSource to /api/sse?topics=device_status,alert_fired,config_push,firmware_progress,metric_update
4. Backend maintains SSE connections, pushes events from NATS subscribers
5. Reconnection on disconnect with exponential backoff (1s → 30s max)

Multi-Tenant Isolation (Request → Middleware → RLS):

1. Frontend sends JWT token in Authorization header or httpOnly cookie
2. Backend tenant_context.py middleware extracts user from JWT, determines tenant_id
3. Middleware calls SET LOCAL app.current_tenant = '{tenant_id}' on the database session
4. All subsequent queries automatically filtered by RLS policy (tenant_id = current_setting('app.current_tenant'))
5. Superadmin can re-set tenant context to access any tenant
6. Admin sessions (migrations, NATS subscribers) use superuser connection, handle tenant routing explicitly

State Management:

• Frontend: TanStack Query for server state (device list, metrics, config), React Context for session/auth state
• Backend: Async SQLAlchemy ORM with automatic transaction management per request
• Poller: In-memory device state map with per-device circuit breaker tracking failures and backoff
• Shared: Redis for distributed locks, NATS for event persistence (JetStream replays)

Key Abstractions

Device Client (poller/internal/device/):

• Purpose: Binary API communication with RouterOS devices
• Files: client.go, version.go, health.go, interfaces.go, wireless.go, firmware.go, cert_deploy.go, sftp.go
• Pattern: RouterOS binary API command execution, metric parsing and extraction
• Usage: Worker polls device state and metrics in parallel goroutines

Scheduler & Worker (poller/internal/poller/scheduler.go, worker.go):

• Purpose: Orchestrate per-device polling goroutines with circuit breaker resilience
• Pattern: Per-device goroutine with Redis distributed locking to prevent duplicate polls across replicas
• Lifecycle: Discover new devices from DB, create goroutine; remove devices, cancel goroutine
• Circuit Breaker: Exponential backoff after N consecutive failures, resets on success

NATS Publisher (poller/internal/bus/publisher.go):

• Purpose: Publish typed device events to JetStream streams
• Event types: DeviceStatusEvent, DeviceMetricsEvent, ConfigChangedEvent, PushRollbackEvent, PushAlertEvent
• Each event includes device_id and tenant_id for multi-tenant routing
• Consumers: Backend subscribers, audit logging, alert evaluation

Tunnel Manager (poller/internal/tunnel/manager.go):

• Purpose: Manage WinBox TCP tunnels to devices (port-forwarded SOCKS proxies)
• Port pool: Allocate ephemeral local ports for tunnel endpoints
• Pattern: Accept local connections on port, tunnel to device's WinBox port via binary API

SSH Relay (poller/internal/sshrelay/server.go, session.go, bridge.go):

• Purpose: SSH terminal access to RouterOS devices for remote management
• Pattern: SSH server on poller, bridges SSH sessions to RouterOS via binary API terminal protocol
• Authentication: SSH key or password relay from frontend

FastAPI Router Pattern (backend/app/routers/):

• Files: devices.py, auth.py, alerts.py, config_editor.py, templates.py, metrics.py, etc.
• Pattern: APIRouter with Depends() for RBAC, tenant context, rate limiting
• All routes tenant-scoped under /api/tenants/{tenant_id}/...
• RLS enforcement: Automatic via SET LOCAL app.current_tenant in get_current_user middleware

Async Service Layer (backend/app/services/):

• Purpose: Business logic, database operations, integration with external systems
• Files: device.py, auth.py, backup_service.py, ca_service.py, alert_evaluator.py, etc.
• Pattern: Async functions using AsyncSession, composable for multiple operations in single transaction
• NATS Integration: Subscribers consume events, services update database accordingly

NATS Subscribers (backend/app/services/*_subscriber.py):

• Purpose: Consume events from NATS JetStream, update application state
• Lifecycle: Started/stopped in FastAPI lifespan context manager
• Examples: nats_subscriber.py (device status), metrics_subscriber.py (metrics aggregation), firmware_subscriber.py (firmware update tracking)
• Pattern: JetStream consumer with durable name, explicit message acking for reliability

Frontend Router (frontend/src/routes/):

• Pattern: TanStack Router file-based routing
• Structure: _authenticated.tsx (layout for logged-in users), _authenticated/tenants/$tenantId/devices/... (device management)
• Entry: __root.tsx (QueryClientProvider setup), _authenticated.tsx (auth check + layout)

Frontend Event Stream Hook (frontend/src/hooks/useEventStream.ts):

• Purpose: Manage SSE connection lifecycle, handle reconnection, parse event payloads
• Pattern: useRef for connection state, setInterval for token refresh, EventSource API
• Callbacks: Per-event-type handlers registered by components
• State: Managed in EventStreamContext for app-wide access

Entry Points

Poller Binary (poller/cmd/poller/main.go):

• Location: poller/cmd/poller/main.go
• Triggers: Docker container start, Kubernetes pod initialization
• Responsibilities: Load config, initialize NATS/Redis/PostgreSQL connections, start scheduler, setup observability (Prometheus metrics, structured logging)
• Config source: Environment variables (see poller/internal/config/config.go)

Backend API (backend/app/main.py):

• Location: backend/app/main.py
• Triggers: Docker container start, uvicorn ASGI server
• Responsibilities: Configure logging, run migrations, bootstrap first admin, start NATS subscribers, setup middleware, register routers
• Lifespan: Async context manager handles startup/shutdown of services
• Health check: /api/health endpoint, /api/readiness for k8s

Frontend Entry (frontend/src/routes/__root.tsx):

• Location: frontend/src/routes/__root.tsx
• Triggers: Browser loads app at /
• Responsibilities: Wrap app in QueryClientProvider (TanStack Query), setup root error boundary
• Auth flow: Routes under _authenticated check JWT token, redirect to login if missing
• Real-time setup: Establish SSE connection via useEventStream hook in layout

Error Handling

Strategy: Three-tier error handling across services

Patterns:

• Poller: Circuit breaker exponential backoff for device connection failures. Logs all errors to structured JSON with context (device_id, tenant_id, attempt number). Publishes failure events to NATS for alerting.

• Backend: FastAPI exception handlers convert service errors to HTTP responses. RLS violations return 403 Forbidden. Invalid tenant access returns 404. Database errors logged via structlog with request_id middleware for correlation.

• Frontend: TanStack Query retry logic (1 retry by default), error boundaries catch component crashes, toast notifications display user-friendly error messages, RequestID middleware propagates correlation IDs

Cross-Cutting Concerns

Logging:

• Poller: log/slog with JSON handler, structured fields (service, device_id, tenant_id, operation)
• Backend: structlog with async logger, JSON output in production
• Frontend: Browser console + error tracking (if configured)

Validation:

• Backend: Pydantic models (app/schemas/) enforce request shape and types, custom validators for business logic (e.g., SRP challenge validation)
• Frontend: TanStack Form for client-side validation before submission
• Database: PostgreSQL CHECK constraints and unique indexes

Authentication:

• Zero-knowledge SRP-6a for initial password enrollment (client never sends plaintext)
• JWT tokens issued after SRP enrollment, stored as httpOnly cookies
• Optional API keys with scoped access for programmatic use
• SSE token exchange for event stream access (short-lived, single-use)

Authorization (RBAC):

• Four roles: super_admin (all access), tenant_admin (full tenant access), operator (read+config), viewer (read-only)
• Role hierarchy enforced by require_role() dependency in routers
• API key scopes: subset of operator permissions (read, write_device, write_config, etc.)

Rate Limiting:

• Backend: Token bucket limiter on sensitive endpoints (login, token generation, device operations)
• Configuration: app/middleware/rate_limit.py defines limits per endpoint
• Redis-backed for distributed rate limit state

Multi-Tenancy:

• Database RLS: All tables have tenant_id, policy enforces current_tenant filter
• Tenant context: Middleware extracts from JWT, sets app.current_tenant local variable
• Superadmin bypass: Can re-set tenant context to access any tenant
• Admin operations: Use superuser connection, explicit tenant routing

---

Architecture analysis: 2026-03-12