275 lines
13 KiB
Markdown
275 lines
13 KiB
Markdown
# Per-Tenant VPN Network Isolation — Design Spec
|
||
|
||
## Overview
|
||
|
||
Isolate WireGuard VPN networks per tenant so that devices in one tenant's VPN cannot reach devices in another tenant's VPN. Each tenant gets a unique `/24` subnet auto-allocated from `10.10.0.0/16`, with iptables rules blocking cross-subnet traffic.
|
||
|
||
**Branch:** `main` (this is a security fix, not SaaS-specific)
|
||
|
||
## Design Decisions
|
||
|
||
- **Single `wg0` interface** — WireGuard handles thousands of peers on one interface with negligible performance impact. No need for per-tenant interfaces.
|
||
- **Per-tenant `/24` subnets** — allocated from `10.10.0.0/16`, giving 255 tenants (index 1–255). Index 0 is reserved. Expandable to `10.0.0.0/8` if needed (note: `_next_available_ip()` materializes all hosts in the subnet, so subnets larger than `/24` require refactoring that function).
|
||
- **Auto-allocation only** — `setup_vpn()` picks the next available subnet. No manual override.
|
||
- **Global config sync** — one `wg0.conf` with all tenants' peers. Rebuilt on any VPN change. Protected by a PostgreSQL advisory lock to prevent concurrent writes.
|
||
- **Global server keypair** — a single WireGuard server keypair stored in `system_settings`, replacing per-tenant server keys. Generated on first `setup_vpn()` call or during migration.
|
||
- **iptables isolation** — cross-subnet traffic blocked at the WireGuard container's firewall. IPv6 blocked too.
|
||
- **Device-side config is untrusted** — isolation relies entirely on server-side enforcement (AllowedIPs `/32` + iptables DROP). A malicious device operator changing their `allowed-address` to `10.10.0.0/16` on their router gains nothing — the server only routes their assigned `/32`.
|
||
|
||
## Data Model Changes
|
||
|
||
### Modified: `vpn_config`
|
||
|
||
| Column | Change | Description |
|
||
|--------|--------|-------------|
|
||
| `subnet_index` | **New column**, integer, unique, not null | Maps to third octet: index 1 = `10.10.1.0/24` |
|
||
| `subnet` | Default changes | No longer `10.10.0.0/24`; derived from `subnet_index` |
|
||
| `server_address` | Default changes | No longer `10.10.0.1/24`; derived as `10.10.{index}.1/24` |
|
||
| `server_private_key` | **Deprecated** | Kept in table for rollback safety but no longer used. Global key in `system_settings` is authoritative. |
|
||
| `server_public_key` | **Deprecated** | Same — kept but unused. All peers use the global public key. |
|
||
|
||
### New: `system_settings` entries
|
||
|
||
| Key | Description |
|
||
|-----|-------------|
|
||
| `vpn_server_private_key` | Global WireGuard server private key (encrypted with CREDENTIAL_ENCRYPTION_KEY) |
|
||
| `vpn_server_public_key` | Global WireGuard server public key (plaintext) |
|
||
|
||
### Allocation Logic
|
||
|
||
```
|
||
subnet_index = first available integer in range [1, 255] not already in vpn_config
|
||
subnet = 10.10.{subnet_index}.0/24
|
||
server_address = 10.10.{subnet_index}.1/24
|
||
```
|
||
|
||
Allocation query (atomic, gap-filling):
|
||
```sql
|
||
SELECT MIN(x) FROM generate_series(1, 255) AS x
|
||
WHERE x NOT IN (SELECT subnet_index FROM vpn_config)
|
||
```
|
||
|
||
If no index available → 422 "VPN subnet pool exhausted".
|
||
|
||
Unique constraint on `subnet_index` provides safety against race conditions. On conflict, retry once.
|
||
|
||
## VPN Service Changes
|
||
|
||
### `setup_vpn(db, tenant_id, endpoint)`
|
||
|
||
Current behavior: creates VpnConfig with hardcoded `10.10.0.0/24` and generates a per-tenant server keypair.
|
||
|
||
New behavior:
|
||
1. **Get or create global server keypair:** check `system_settings` for `vpn_server_private_key`. If not found, generate a new keypair and store both the private key (encrypted) and public key. This happens on the first `setup_vpn()` call on a fresh install.
|
||
2. Allocate next `subnet_index` using the gap-filling query
|
||
3. Set `subnet = 10.10.{index}.0/24`
|
||
4. Set `server_address = 10.10.{index}.1/24`
|
||
5. Store the global public key in `server_public_key` (for backward compat / display)
|
||
6. Call `sync_wireguard_config(db)` (global, not per-tenant)
|
||
|
||
### `sync_wireguard_config(db)`
|
||
|
||
Current signature: `sync_wireguard_config(db, tenant_id)` — builds config for one tenant.
|
||
|
||
New signature: `sync_wireguard_config(db)` — builds config for ALL tenants.
|
||
|
||
**Concurrency protection:** acquire a PostgreSQL advisory lock (`pg_advisory_xact_lock(hash)`) before writing. This prevents two simultaneous peer additions from producing a corrupt `wg0.conf`.
|
||
|
||
**Atomic write:** write to a temp file, then `os.rename()` to `wg0.conf`. This prevents the WireGuard container from reading a partially-written file.
|
||
|
||
New behavior:
|
||
1. Acquire advisory lock
|
||
2. Read global server private key from `system_settings` (decrypt it)
|
||
3. Query ALL enabled `VpnConfig` rows (across all tenants, using admin engine to bypass RLS)
|
||
4. For each, query enabled `VpnPeer` rows
|
||
5. Build single `wg0.conf`:
|
||
|
||
```ini
|
||
[Interface]
|
||
Address = 10.10.0.1/16
|
||
ListenPort = 51820
|
||
PrivateKey = {global_server_private_key}
|
||
|
||
# --- Tenant: {tenant_name} (10.10.1.0/24) ---
|
||
[Peer]
|
||
PublicKey = {peer_public_key}
|
||
PresharedKey = {preshared_key}
|
||
AllowedIPs = 10.10.1.2/32
|
||
|
||
# --- Tenant: {tenant_name_2} (10.10.2.0/24) ---
|
||
[Peer]
|
||
PublicKey = {peer_public_key}
|
||
PresharedKey = {preshared_key}
|
||
AllowedIPs = 10.10.2.2/32
|
||
```
|
||
|
||
6. Write to temp file, `os.rename()` to `wg0.conf`
|
||
7. Touch `.reload` flag
|
||
8. Release advisory lock
|
||
|
||
### `_next_available_ip(db, tenant_id, config)`
|
||
|
||
No changes needed — already scoped to `tenant_id` and uses the config's subnet. With unique subnets per tenant, IPs are naturally isolated. Note: this function materializes all `/24` hosts into a list, which is fine for `/24` (253 entries) but must be refactored if subnets larger than `/24` are ever used.
|
||
|
||
### `add_peer(db, tenant_id, device_id, ...)`
|
||
|
||
Changes:
|
||
- Calls `sync_wireguard_config(db)` instead of `sync_wireguard_config(db, tenant_id)`
|
||
- **Validate `additional_allowed_ips`:** if provided, reject any subnet that overlaps with `10.10.0.0/16` (the VPN address space). Only non-VPN subnets are allowed (e.g., `192.168.1.0/24` for site-to-site routing). This prevents a tenant from claiming another tenant's VPN subnet in their AllowedIPs.
|
||
|
||
### `remove_peer(db, tenant_id, peer_id)`
|
||
|
||
Minor change: calls `sync_wireguard_config(db)` instead of `sync_wireguard_config(db, tenant_id)`.
|
||
|
||
### Tenant deletion hook
|
||
|
||
When a tenant is deleted (CASCADE deletes vpn_config and vpn_peers), call `sync_wireguard_config(db)` to regenerate `wg0.conf` without the deleted tenant's peers. Add this to the tenant deletion endpoint.
|
||
|
||
### `read_wg_status()`
|
||
|
||
No changes — status is keyed by peer public key, which is unique globally. The existing `get_peer_handshake()` lookup continues to work.
|
||
|
||
## WireGuard Container Changes
|
||
|
||
### iptables Isolation Rules
|
||
|
||
Update `docker-data/wireguard/custom-cont-init.d/10-forwarding.sh`:
|
||
|
||
```bash
|
||
#!/bin/sh
|
||
# Enable forwarding between Docker network and WireGuard tunnel
|
||
# Idempotent: check before adding to prevent duplicates on restart
|
||
iptables -C FORWARD -i eth0 -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i eth0 -o wg0 -j ACCEPT
|
||
iptables -C FORWARD -i wg0 -o eth0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -o eth0 -j ACCEPT
|
||
|
||
# Block cross-subnet traffic on wg0 (tenant isolation)
|
||
# Peers in 10.10.1.0/24 cannot reach peers in 10.10.2.0/24
|
||
iptables -C FORWARD -i wg0 -o wg0 -j DROP 2>/dev/null || iptables -A FORWARD -i wg0 -o wg0 -j DROP
|
||
|
||
# Block IPv6 forwarding on wg0 (prevent link-local bypass)
|
||
ip6tables -C FORWARD -i wg0 -j DROP 2>/dev/null || ip6tables -A FORWARD -i wg0 -j DROP
|
||
|
||
# NAT for return traffic
|
||
iptables -C POSTROUTING -t nat -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
|
||
|
||
echo "WireGuard forwarding and tenant isolation rules applied"
|
||
```
|
||
|
||
Rules use `iptables -C` (check) before `-A` (append) to be idempotent across container restarts.
|
||
|
||
The key isolation layers:
|
||
|
||
1. **WireGuard AllowedIPs** — each peer can only send to its own `/32` IP (cryptographic enforcement)
|
||
2. **iptables `wg0 → wg0` DROP** — blocks any traffic that enters and exits the tunnel interface (peer-to-peer)
|
||
3. **iptables IPv6 DROP** — prevents link-local IPv6 bypass
|
||
4. **Separate subnets** — no IP collisions between tenants
|
||
5. **`additional_allowed_ips` validation** — blocks tenants from claiming VPN address space
|
||
|
||
### Server Address
|
||
|
||
The `[Interface] Address` changes from `10.10.0.1/24` to `10.10.0.1/16` so the server can route to all tenant subnets.
|
||
|
||
## Routing Changes
|
||
|
||
### Poller & API
|
||
|
||
No changes needed. Both already route `10.10.0.0/16` via the WireGuard container.
|
||
|
||
### setup.py
|
||
|
||
Update `prepare_data_dirs()` to write the updated forwarding script with idempotent rules and IPv6 blocking.
|
||
|
||
## RouterOS Command Generation
|
||
|
||
### `onboard_device()` and `get_peer_config()`
|
||
|
||
These generate RouterOS commands for device setup. Changes:
|
||
|
||
- `allowed-address` changes from `10.10.0.0/24` to `10.10.{index}.0/24` (tenant's specific subnet)
|
||
- `endpoint-address` and `endpoint-port` unchanged
|
||
- Server public key changes to the global server public key (read from `system_settings`)
|
||
|
||
## Migration
|
||
|
||
### Database Migration
|
||
|
||
1. Generate global server keypair:
|
||
- Create keypair using `generate_wireguard_keypair()`
|
||
- Store in `system_settings`: `vpn_server_private_key` (encrypted), `vpn_server_public_key` (plaintext)
|
||
2. Add `subnet_index` column to `vpn_config` (integer, unique, not null)
|
||
3. For existing VpnConfig rows (may be multiple if multiple tenants have VPN):
|
||
- Assign sequential `subnet_index` values starting from 1
|
||
- Update `subnet` to `10.10.{index}.0/24`
|
||
- Update `server_address` to `10.10.{index}.1/24`
|
||
4. For existing VpnPeer rows:
|
||
- Remap IPs: `10.10.0.X` → `10.10.{tenant's index}.X` (preserve the host octet)
|
||
- Example: Tenant A (index 1) peer at `10.10.0.2` → `10.10.1.2`. Tenant B (index 2) peer at `10.10.0.2` → `10.10.2.2`. No collision.
|
||
5. Regenerate `wg0.conf` using the new global sync function
|
||
|
||
### Device-Side Update Required
|
||
|
||
This is a **breaking change** for existing VPN peers. After migration:
|
||
- Devices need updated RouterOS commands:
|
||
- New server public key (global key replaces per-tenant key)
|
||
- New VPN IP address (`10.10.0.X` → `10.10.{index}.X`)
|
||
- New allowed-address (`10.10.{index}.0/24`)
|
||
- The API should expose a "regenerate commands" endpoint or show a banner in the UI indicating that VPN reconfiguration is needed.
|
||
|
||
### Migration Communication
|
||
|
||
After the migration runs:
|
||
- Log a warning with the list of affected devices
|
||
- Show a banner in the VPN UI: "VPN network updated — devices need reconfiguration. Click here for updated commands."
|
||
- The existing "View Setup Commands" button in the UI will show the correct updated commands.
|
||
|
||
## API Changes
|
||
|
||
### Modified Endpoints
|
||
|
||
| Method | Path | Change |
|
||
|--------|------|--------|
|
||
| `POST` | `/api/tenants/{id}/vpn` | `setup_vpn` allocates subnet_index, uses global server key |
|
||
| `GET` | `/api/tenants/{id}/vpn` | Returns tenant's specific subnet info |
|
||
| `GET` | `/api/tenants/{id}/vpn/peers/{id}/config` | Returns commands with tenant-specific subnet and global server key |
|
||
| `POST` | `/api/tenants/{id}/vpn/peers` | Validates `additional_allowed_ips` doesn't overlap `10.10.0.0/16` |
|
||
| `DELETE` | `/api/tenants/{id}` | Calls `sync_wireguard_config(db)` after cascade delete |
|
||
|
||
### No New Endpoints
|
||
|
||
The isolation is transparent — tenants don't need to know about it.
|
||
|
||
## Error Handling
|
||
|
||
| Scenario | HTTP Status | Message |
|
||
|----------|-------------|---------|
|
||
| No available subnet index (255 tenants with VPN) | 422 | "VPN subnet pool exhausted" |
|
||
| Subnet index conflict (race condition) | — | Retry allocation once |
|
||
| `additional_allowed_ips` overlaps VPN space | 422 | "Additional allowed IPs must not overlap the VPN address space (10.10.0.0/16)" |
|
||
|
||
## Testing
|
||
|
||
- Create two tenants with VPN enabled → verify they get different subnets (`10.10.1.0/24`, `10.10.2.0/24`)
|
||
- Add peers in both → verify IPs don't collide
|
||
- From tenant A's device, attempt to ping tenant B's device → verify it's blocked
|
||
- Verify `wg0.conf` contains peers from both tenants with correct subnets
|
||
- Verify iptables rules are in place after container restart (idempotent)
|
||
- Verify `additional_allowed_ips` with `10.10.x.x` subnet is rejected
|
||
- Delete a tenant → verify `wg0.conf` is regenerated without its peers
|
||
- Disable a tenant's VPN → verify peers excluded from `wg0.conf`
|
||
- Empty state (no enabled tenants) → verify `wg0.conf` has only `[Interface]` section
|
||
- Migration: multiple tenants sharing `10.10.0.0/24` → verify correct remapping to unique subnets
|
||
|
||
## Audit Logging
|
||
|
||
- Subnet allocated (tenant_id, subnet_index, subnet)
|
||
- Global server keypair generated (first-run event)
|
||
- VPN config regenerated (triggered by which operation)
|
||
|
||
## Out of Scope
|
||
|
||
- Multiple WireGuard interfaces (not needed at current scale)
|
||
- Manual subnet assignment
|
||
- IPv6 VPN support (IPv6 is blocked as a security measure)
|
||
- Per-tenant WireGuard listen ports
|
||
- VPN-level rate limiting or bandwidth quotas
|