Compare commits

...
3 Commits
Author SHA1 Message Date
Julian Boldt 4eea6424e7 Release v1.32.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
  Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
  ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
  das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
  ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
  nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
  und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
  Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
  **ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
  (Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
  vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
  damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
  vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
2026-10-23 11:27:47 +02:00
Julian Boldt a8ca0a14ef Release v1.31.1
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
  (`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
  SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
  ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
2026-10-22 23:45:57 +02:00
Julian Boldt b020e38d0a Release v1.31.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Regel-Bereinigung: Entfernen, Bestätigen, Löschen, Wiederherstellen:** Im Tab **Bereinigung** (Regelwerk › Regel-Analyse) lassen sich Kandidaten jetzt auswählen
  (Checkbox, höchstens so viele Regeln je Lauf wie in den Einstellungen; „vermutlich“ nur mit einer ausdrücklichen Einzelbestätigung je Regel, „unbekannt“ nie).
  **Vorschau** zeigt je Firewall (im Cluster zuerst der Backup) die Regeln mit Befund, Sicherheit und Schutzprüfung; **Ausrollen** deaktiviert die Regeln und wendet an.
  Danach läuft ein Countdown: **Bestätigen** heißt „ich erreiche die Firewall noch“, ohne Bestätigung setzt OPNexus **automatisch zurück**; bis dahin geht auch
  **Jetzt zurücksetzen**. Bestätigte Regeln lassen sich nach der Wartezeit (einstellbar, Standard 7 Tage) **löschen**; vorher sichert OPNexus den Regelinhalt, und
  unter **Verlauf der Bereinigungen** kommt die Regel mit **Wiederherstellen** deaktiviert und möglichst an alter Position zurück (sonst mit Hinweis am Ende der Gruppe);
  **Wieder aktivieren** hat wieder ein Bestätigungsfenster. Wichtig: **Ausrollen, Wiederholen und Wiederherstellen darf nur ein Administrator**, eine Vorschau gehört
  dem, der sie erstellt hat (ein Admin öffnet selbst eine neue); **Bestätigen geht nur in der Browser-Sitzung**, nie per API-Token. **Die Bereinigung bleibt gesperrt,
  bis unter Einstellungen die Verwaltungsquellen** (IP-Adressen oder Netze, von denen aus verwaltet wird) **eingetragen sind**: Regeln, die diesen Zugang treffen
  könnten, deaktiviert OPNexus nie. Dort stehen auch die Obergrenze je Lauf, die Wartezeit bis zum Löschen und **weitere Verwaltungsports** (zusätzlich zu
  API-Port, 443, 80, 22). Tragen Sie die Adresse ein, mit der OPNexus selbst bei den Firewalls ankommt (im Cluster für jeden Knoten). Quellen, Ports und
  Firewall-Adressen sehen nur Administratoren. Der Hinweis „Entfernen folgt“ entfällt.
- **Update: neuere CLI aus dem Zielrelease:** `opnexus update` (auch per Klick im Webinterface) holt sich nach dem Pull die CLI aus dem Updater-Image des
  Zielreleases und startet sich damit neu, damit Korrekturen am Update-Ablauf sofort greifen statt erst beim übernächsten Update. Das geschieht nur, wenn
  `cosign` und `cosign.pub` vorhanden sind und die Signatur des Image-**Digests** in diesem Lauf geprüft wurde (ausgeführt wird genau dieser Digest);
  sonst und bei jedem Fehler läuft das Update mit der vorhandenen CLI weiter. Abschalten: `OPNEXUS_NO_CLI_REFRESH=1`. Wirksam wird es ab dem Update
  *von* dieser Version.
2026-10-22 21:04:32 +02:00
125 changed files with 14691 additions and 229 deletions

No files matched your search

+47 -1
View File
@@ -29,7 +29,53 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
den Einstellungen.
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
ansteht): Merlin …
ansteht): Saker …
## [1.32.0] "Merlin" — 2026-10-23
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
## [1.31.1] "Vulture" — 2026-10-22
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
## [1.31.0] "Vulture" — 2026-10-22
- **Regel-Bereinigung: Entfernen, Bestätigen, Löschen, Wiederherstellen:** Im Tab **Bereinigung** (Regelwerk › Regel-Analyse) lassen sich Kandidaten jetzt auswählen
(Checkbox, höchstens so viele Regeln je Lauf wie in den Einstellungen; „vermutlich“ nur mit einer ausdrücklichen Einzelbestätigung je Regel, „unbekannt“ nie).
**Vorschau** zeigt je Firewall (im Cluster zuerst der Backup) die Regeln mit Befund, Sicherheit und Schutzprüfung; **Ausrollen** deaktiviert die Regeln und wendet an.
Danach läuft ein Countdown: **Bestätigen** heißt „ich erreiche die Firewall noch“, ohne Bestätigung setzt OPNexus **automatisch zurück**; bis dahin geht auch
**Jetzt zurücksetzen**. Bestätigte Regeln lassen sich nach der Wartezeit (einstellbar, Standard 7 Tage) **löschen**; vorher sichert OPNexus den Regelinhalt, und
unter **Verlauf der Bereinigungen** kommt die Regel mit **Wiederherstellen** deaktiviert und möglichst an alter Position zurück (sonst mit Hinweis am Ende der Gruppe);
**Wieder aktivieren** hat wieder ein Bestätigungsfenster. Wichtig: **Ausrollen, Wiederholen und Wiederherstellen darf nur ein Administrator**, eine Vorschau gehört
dem, der sie erstellt hat (ein Admin öffnet selbst eine neue); **Bestätigen geht nur in der Browser-Sitzung**, nie per API-Token. **Die Bereinigung bleibt gesperrt,
bis unter Einstellungen die Verwaltungsquellen** (IP-Adressen oder Netze, von denen aus verwaltet wird) **eingetragen sind**: Regeln, die diesen Zugang treffen
könnten, deaktiviert OPNexus nie. Dort stehen auch die Obergrenze je Lauf, die Wartezeit bis zum Löschen und **weitere Verwaltungsports** (zusätzlich zu
API-Port, 443, 80, 22). Tragen Sie die Adresse ein, mit der OPNexus selbst bei den Firewalls ankommt (im Cluster für jeden Knoten). Quellen, Ports und
Firewall-Adressen sehen nur Administratoren. Der Hinweis „Entfernen folgt“ entfällt.
- **Update: neuere CLI aus dem Zielrelease:** `opnexus update` (auch per Klick im Webinterface) holt sich nach dem Pull die CLI aus dem Updater-Image des
Zielreleases und startet sich damit neu, damit Korrekturen am Update-Ablauf sofort greifen statt erst beim übernächsten Update. Das geschieht nur, wenn
`cosign` und `cosign.pub` vorhanden sind und die Signatur des Image-**Digests** in diesem Lauf geprüft wurde (ausgeführt wird genau dieser Digest);
sonst und bei jedem Fehler läuft das Update mit der vorhandenen CLI weiter. Abschalten: `OPNEXUS_NO_CLI_REFRESH=1`. Wirksam wird es ab dem Update
*von* dieser Version.
## [1.30.0] "Owl" — 2026-10-22
+5 -4
View File
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
![Zentrale Firewall-Regeln](docs/screenshots/readme-rules-de.webp)
> **Stand:** `1.30.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
> **Stand:** `1.32.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
## Funktionen
@@ -21,12 +21,13 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit), und jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit); ein Administrator kann ausgewählte Regeln abschalten (Vorschau, dann Ausrollen mit automatischer Rücknahme ohne Bestätigung), später löschen und aus einer Sicherung des Regelinhalts wiederherstellen. Jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
@@ -65,7 +66,7 @@ Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelie
```bash
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
OPNEXUS_VERSION=1.30.0
OPNEXUS_VERSION=1.32.0
```
```bash
+5 -4
View File
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
![Central firewall rules](docs/screenshots/readme-rules-en.webp)
> **Status:** `1.30.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
> **Status:** `1.32.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
## What it does
@@ -21,12 +21,13 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence) and every rule has a change history, including changes made outside OPNexus.
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence); an administrator can switch selected rules off (preview, then roll-out with an automatic revert unless confirmed), delete them later and restore them from a backup of their content. Every rule has a change history, including changes made outside OPNexus.
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
@@ -67,7 +68,7 @@ For access from other machines and for production use with TLS (bundled Caddy pr
```bash
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
OPNEXUS_VERSION=1.30.0
OPNEXUS_VERSION=1.32.0
```
```bash
+6 -3
View File
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
"/api/settings",
# Read-only JSON-RPC tools (see routers/mcp.py).
"/api/mcp",
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
"/api/config/import/plan",
# Self-scoped: a user managing their own second factor.
"/api/auth/2fa/setup",
"/api/auth/2fa/enable",
@@ -121,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
)
if user is not None:
parts = request.url.path.strip("/").split("/")
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
@@ -131,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
if len(parts) > 2 and parts[1] == "settings":
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
entity = parts[2] if len(parts) > 2 and kind != "config" else None
event = {"kind": kind, "action": action, "entity_id": entity,
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
"details": {"method": request.method, "path": request.url.path}}
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
if (
user.role == "read_only"
+5
View File
@@ -0,0 +1,5 @@
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
"""
+228
View File
@@ -0,0 +1,228 @@
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
"""
import argparse
import asyncio
import json
import re
import sys
import uuid
from app.config_code import apply as applier
from app.config_code import plan as planner
from app.config_code import schema
from app.config_code.export import export_yaml
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
ACTOR = "cli (Host-Konsole)"
TOKEN_MARK = "# plan-token: "
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
ERRORS_DE = {
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
"yaml_empty": "Die Datei ist leer",
"yaml_syntax": "YAML-Syntaxfehler",
"duplicate_key": "Doppelter Schlüssel",
"non_string_key": "Schlüssel muss Text sein",
"too_deep": "Zu tief verschachtelt",
"too_many_nodes": "Zu viele Elemente",
"scalar_too_long": "Wert zu lang",
"too_many_objects": "Zu viele Objekte",
"too_many_entries": "Zu viele Einträge",
"too_many_variables": "Zu viele Variablen für diesen Host",
"too_large": "Datei zu groß",
"bad_encoding": "Kein gültiges UTF-8",
"not_a_mapping": "Die Datei muss ein Mapping sein",
"missing_version": "opnexus_config: 1 fehlt",
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
"unknown_field": "Unbekanntes Feld",
"missing_field": "Pflichtfeld fehlt",
"bad_type": "Falscher Typ",
"bad_value": "Ungültiger Wert",
"bad_format": "Falsches Format",
"too_short": "Zu kurz",
"too_long": "Zu lang",
"control_character": "Steuerzeichen nicht erlaubt",
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
"bad_variable_name": "Ungültiger Variablenname",
"bad_variable_value": "Ungültiger Variablenwert",
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
"duplicate_entry": "Eintrag kommt doppelt vor",
"unknown_host": "Unbekannter Host",
"unknown_group": "Unbekannte Gerätegruppe",
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
}
REASONS_DE = {
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
"prune_referenced": "wird noch verwendet",
}
WARNINGS_DE = {
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
"unknown_alias_reference": "{value}: kein Alias dieses Namens",
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
}
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
def safe(text) -> str:
"""Nothing from a foreign file reaches the terminal as a control sequence."""
return _ANSI.sub("?", str(text))
def _short(value) -> str:
return safe(display(json.dumps(value, ensure_ascii=False), 60))
def format_errors(errors: list[dict]) -> str:
lines = []
for e in errors:
where = f"Zeile {e['line']}: " if e.get("line") else ""
path = f"{safe(e['path'])}: " if e.get("path") else ""
detail = ""
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
if params:
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
return "\n".join(lines)
def format_plan(public: dict) -> str:
lines = []
for item in public["items"]:
action = item["action"]
planned = item["planned_action"]
name = f"{item['kind']} {safe(item['key'])}"
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
if action == "blocked":
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
elif planned in LABEL:
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
else:
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
if change["field"]:
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
else:
lines.append(f"~ {name}{suffix}")
if item.get("implicit"):
lines[-1] += " (Folge der Reihenfolge)"
for warning in public["warnings"]:
template = WARNINGS_DE.get(warning["code"])
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
text = template.format(**params) if template else warning["code"]
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
lines.append(f"? {where}{text}")
s = public["summary"]
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
return "\n".join(lines)
def _read_stdin() -> bytes:
data = sys.stdin.buffer.read(MAX_BYTES + 1)
return data
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
parser.add_argument("--kinds", default=None)
parser.add_argument("--prune", action="store_true")
parser.add_argument("--json", action="store_true")
parser.add_argument("--plan-token", default=None)
try:
args = parser.parse_args(argv)
except SystemExit:
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
if args.command == "schema":
return 0, schema.json_schema_text(), ""
from app.database import SessionLocal
from app.services import audit_service
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
try:
async with SessionLocal() as db:
if args.command == "export":
wanted = None
if args.kinds:
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
if not wanted <= set(schema.KINDS):
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
return 0, await export_yaml(db, wanted), ""
raw = stdin if stdin is not None else _read_stdin()
try:
parsed = parse(raw)
if args.command == "plan":
plan = await planner.build_plan(db, parsed, args.prune)
await db.rollback()
public = plan.public()
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
s = public["summary"]
if not plan.applicable:
return 4, out, ""
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
if not args.plan_token:
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
try:
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
except applier.PlanStale:
await db.rollback()
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
except applier.PlanBlocked as exc:
await db.rollback()
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
except applier.SelfCheckFailed as exc:
await db.rollback()
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
except Exception:
await db.rollback()
raise
await db.commit()
s = plan.summary()
if args.json:
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
except ConfigError as exc:
if args.json:
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
finally:
audit_service.context.reset(context)
def main(argv: list[str]) -> int:
try:
code, out, err = asyncio.run(run(argv))
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
return 1
sys.stdout.write(out)
sys.stderr.write(err)
return code
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+312
View File
@@ -0,0 +1,312 @@
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
Den Commit macht der Aufrufer.
"""
from datetime import datetime, timezone
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.config_code import plan as planner
from app.config_code.export import canonical
from app.config_code.loader import Parsed
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.host_variable import HostVariable
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
from app.services import audit_service
NOTE = "Import (Konfiguration als Code)"
AUDIT_KIND = {
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
}
class PlanStale(Exception):
pass
class PlanBlocked(Exception):
def __init__(self, plan: planner.Plan):
self.plan = plan
super().__init__("plan blocked")
class SelfCheckFailed(Exception):
def __init__(self, differences: list[str]):
self.differences = differences
super().__init__("; ".join(differences[:5]))
def _revision_writer(kind: str):
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
"""
if kind == "rules":
from app.services.policy_service import snapshot
return RuleRevision, "rule_id", snapshot
if kind == "aliases":
from app.services.alias_policy_service import snapshot
return AliasRevision, "alias_id", snapshot
if kind == "port_forwards":
from app.services.port_forward_policy_service import snapshot
return PortForwardRevision, "port_forward_id", snapshot
if kind == "one_to_one":
from app.services.one_to_one_policy_service import snapshot
return OneToOneRevision, "one_to_one_id", snapshot
if kind == "outbound_nat":
from app.services.outbound_nat_policy_service import snapshot
return OutboundNatRevision, "outbound_nat_id", snapshot
from app.services.policy_hierarchy_service import snapshot
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
SCALARS = {
"aliases": ("name", "type", "description", "enabled"),
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
"destination_net", "destination_port", "enabled", "log"),
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
"destination_port", "target", "local_port", "natreflection", "enabled"),
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
"destination_not", "natreflection", "log", "enabled"),
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
}
class _Writer:
def __init__(self, db: AsyncSession, plan: planner.Plan):
self.db = db
self.plan = plan
self.now = datetime.now(timezone.utc)
self.events: list[dict] = []
async def load(self) -> None:
db = self.db
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
self.groups = {g.name: g for g in (await db.execute(
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
self.rows: dict[str, dict] = {}
for kind, model in MODELS.items():
options = [selectinload(model.deployments)]
if kind != "policy_hierarchy_rules":
options += [selectinload(model.targets), selectinload(model.groups)]
rows = (await db.execute(select(model).options(*options))).scalars().all()
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
def items(self, kind: str, *actions: str) -> list[planner.Item]:
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
def event(self, item: planner.Item, entity_id, name: str) -> None:
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
# -- groups ----------------------------------------------------------------------------------------------------------
async def hierarchy_groups(self) -> None:
items = self.items("policy_hierarchy_groups", "create", "update")
for item in items:
if item.action == "create":
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
self.db.add(group)
self.hierarchy[item.key] = group
await self.db.flush()
for item in items:
group = self.hierarchy[item.key]
group.description = item.after["description"]
parent = item.after["parent"]
group.parent_id = self.hierarchy[parent].id if parent else None
self.event(item, group.id, group.name)
await self.db.flush()
async def device_groups(self) -> None:
for item in self.items("device_groups", "create", "update"):
group = self.groups.get(item.key)
if group is None:
group = DeviceGroup(name=item.key)
self.db.add(group)
self.groups[item.key] = group
group.description = item.after["description"]
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
await self.db.flush()
self.event(item, group.id, group.name)
# -- objects ---------------------------------------------------------------------------------------------------------
async def objects(self, kind: str) -> None:
model = MODELS[kind]
revision_model, fk, snapshot = _revision_writer(kind)
for item in self.items(kind, "create", "update"):
after = item.after
row = self.rows[kind].get(item.key)
if row is None:
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
if kind != "aliases":
row.config_key = item.key
self.db.add(row)
self.rows[kind][item.key] = row
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
for name in SCALARS[kind]:
setattr(row, name, after[name])
if kind == "aliases":
row.content = "\n".join(after["entries"])
if kind == "policy_hierarchy_rules":
row.hierarchy_group_id = self.hierarchy[after["group"]].id
else:
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
if "hierarchy_group" in after["targets"]:
hierarchy = after["targets"]["hierarchy_group"]
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
await self.db.flush()
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
self.event(item, row.id, row.name if kind == "aliases" else row.description)
async def hierarchy_positions(self) -> None:
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
if "policy_hierarchy_rules" not in self.plan.touched:
return
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
for key, rank in self.plan.positions.items():
row = self.rows["policy_hierarchy_rules"].get(key)
if row is not None and key not in deleted and row.sequence != rank:
row.sequence = rank
await self.db.flush()
async def host_hierarchy(self) -> None:
for host_name, group_name in self.plan.host_hierarchy.items():
host = self.hosts[host_name]
wanted = self.hierarchy[group_name].id if group_name else None
if host.policy_hierarchy_group_id != wanted:
host.policy_hierarchy_group_id = wanted
await self.db.flush()
async def variables(self) -> None:
if "variables" not in self.plan.touched:
return
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
for item in self.items("variables", "create", "update"):
host = self.hosts[item.after["host"]]
row = existing.get((host.id, item.after["name"]))
if row is None:
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
else:
row.value = item.after["value"]
self.event(item, None, item.key)
for item in self.items("variables", "delete"):
host = self.hosts[item.before["host"]]
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
self.event(item, None, item.key)
await self.db.flush()
# -- deletions -------------------------------------------------------------------------------------------------------
async def deletions(self) -> None:
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
for item in self.items(kind, "delete"):
row = self.rows[kind][item.key]
if row.deployments: # the planner blocks this; never undeploy from here
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
self.event(item, row.id, row.name if kind == "aliases" else row.description)
await self.db.delete(row)
await self.db.flush()
for item in self.items("device_groups", "delete"):
row = self.groups[item.key]
self.event(item, row.id, row.name)
await self.db.delete(row)
await self.db.flush()
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
by_id = {g.id: g for g in self.hierarchy.values()}
def depth(group) -> int:
n, current = 0, group
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
n, current = n + 1, by_id[current.parent_id]
return n
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
self.event(item, row.id, row.name)
await self.db.delete(row)
await self.db.flush()
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
db.expire_all() # compare what is in the database, not what the session believes
actual = await canonical(db, set(plan.touched))
differences: list[str] = []
for kind in plan.touched:
if kind == "variables":
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
for h, values in actual["variables"].items() for n, v in values.items()}
else:
ident = planner.KIND_SPECS[kind].ident
got = {item[ident]: item for item in actual[kind]}
want = plan.expected[kind]
for key in sorted(set(got) | set(want)):
if got.get(key) != want.get(key):
differences.append(f"{kind}:{key}")
if "policy_hierarchy_rules" in plan.touched:
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
for key, rank in ranks.items():
if plan.positions.get(key) != rank:
differences.append(f"policy_hierarchy_rules:{key}:position")
if differences:
raise SelfCheckFailed(differences)
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
ConfigError; on any exception the caller must roll back."""
from app.services.policy_service import policy_lock
await policy_lock(db)
plan = await planner.build_plan(db, parsed, prune)
if plan.token != token:
raise PlanStale()
if not plan.applicable:
raise PlanBlocked(plan)
writer = _Writer(db, plan)
await writer.load()
await writer.hierarchy_groups()
await writer.device_groups()
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
if kind in plan.touched:
await writer.objects(kind)
await writer.hierarchy_positions()
await writer.host_hierarchy()
await writer.variables()
await writer.deletions()
await _self_check(db, plan)
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
"summary": plan.summary()}))
for values in writer.events:
db.add(audit_service.make_event(**values))
await db.flush()
return plan
+154
View File
@@ -0,0 +1,154 @@
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
"""
import yaml
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.config_code import schema
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
from app.version import APP_VERSION
HEADER = (
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
)
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
fields = list(doc.model_fields)
if set(values) != set(fields):
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
return {name: values[name] for name in fields}
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
if "hierarchy_group" in doc.model_fields:
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
return _ordered(doc, values)
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
values = dict(extra)
for name in doc.model_fields:
if name in values:
continue
values[name] = obj.config_key if name == "key" else getattr(obj, name)
return _ordered(doc, values)
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
"""Parents before children, siblings by name; the same rule the loader expects."""
children: dict = {}
for group in groups:
children.setdefault(group.parent_id, []).append(group)
ordered: list[PolicyHierarchyGroup] = []
def walk(parent_id) -> None:
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
ordered.append(group)
walk(group.id)
walk(None)
return ordered
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
"""{kind: content} for the requested kinds (default all), in file order."""
wanted = set(schema.KINDS) if kinds is None else kinds
out: dict[str, list | dict] = {}
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
hierarchy_names = {g.id: g.name for g in all_groups}
if "device_groups" in wanted:
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
out["device_groups"] = [
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
for g in sorted(groups, key=lambda g: g.name)
]
if "policy_hierarchy_groups" in wanted:
out["policy_hierarchy_groups"] = [
_ordered(schema.HierarchyGroupDoc, {
"name": g.name,
"parent": hierarchy_names.get(g.parent_id),
"description": g.description or "",
"hosts": sorted(h.name for h in g.hosts),
})
for g in _hierarchy_order(list(all_groups))
]
if "aliases" in wanted:
aliases = (await db.execute(
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
)).scalars().all()
out["aliases"] = [
_ordered(schema.AliasDoc, {
"name": a.name,
"type": a.type,
"description": a.description or "",
"enabled": a.enabled,
"entries": [line for line in (a.content or "").split("\n") if line],
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
})
for a in sorted(aliases, key=lambda a: a.name)
]
simple = (
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
)
for kind, model, doc, targets_doc in simple:
if kind not in wanted:
continue
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
out[kind] = [
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
for row in sorted(rows, key=lambda r: r.config_key)
]
if "policy_hierarchy_rules" in wanted:
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
out["policy_hierarchy_rules"] = [
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
]
if "variables" in wanted:
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
out["variables"] = {
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
for h in sorted(hosts, key=lambda h: h.name)
if h.variables
}
return out
def dump_yaml(content: dict[str, list | dict]) -> str:
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
return dump_yaml(await canonical(db, kinds))
+306
View File
@@ -0,0 +1,306 @@
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
"""
import hashlib
import re
from typing import NamedTuple
import yaml
from pydantic import ValidationError
from yaml.composer import Composer
from yaml.constructor import SafeConstructor
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
from yaml.parser import Parser
from yaml.reader import Reader
from yaml.resolver import BaseResolver
from yaml.scanner import Scanner
from app.config_code import schema
MAX_BYTES = 1024 * 1024
MAX_DEPTH = 8
MAX_NODES = 100_000
MAX_SCALAR = schema.MAX_SCALAR
MAX_OBJECTS_PER_KIND = 5000
MAX_VARIABLE_HOSTS = 500
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
MAX_ALIAS_ENTRIES = 2000
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
_STR = "tag:yaml.org,2002:str"
class ConfigError(Exception):
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
def __init__(self, code: str, errors: list[dict]):
self.code = code
self.errors = errors
super().__init__(f"{code}: {errors[:3]}")
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
return {"code": code, "path": path, "line": line, "params": params}
def display(value, limit: int = 80) -> str:
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
text = value if isinstance(value, str) else repr(value)
text = text.encode("unicode_escape").decode("ascii")
return text if len(text) <= limit else text[: limit - 1] + "…"
def _path(parts) -> str:
out = ""
for part in parts:
if isinstance(part, int):
out += f"[{part}]"
else:
out += ("." if out else "") + str(part)
return out
class _Rejected(yaml.YAMLError):
def __init__(self, code: str, mark=None, path=(), **params):
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
super().__init__(code)
class StrictResolver(BaseResolver):
yaml_implicit_resolvers: dict = {}
StrictResolver.add_implicit_resolver(
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
StrictResolver.add_implicit_resolver(
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
StrictResolver.add_implicit_resolver(
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
class StrictConstructor(SafeConstructor):
yaml_constructors: dict = {}
for _tag, _fn in (
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
(_STR, SafeConstructor.construct_yaml_str),
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
):
StrictConstructor.add_constructor(_tag, _fn)
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
def __init__(self, stream: str):
Reader.__init__(self, stream)
Scanner.__init__(self)
Parser.__init__(self)
Composer.__init__(self)
StrictConstructor.__init__(self)
StrictResolver.__init__(self)
self._depth = 0
self._nodes = 0
self._path: list = []
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
# -- document level ---------------------------------------------------------------------------------------------
def get_single_node(self):
self.get_event() # StreamStart
if self.check_event(StreamEndEvent):
raise _Rejected("yaml_empty")
start = self.peek_event()
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
raise _Rejected("yaml_directive_forbidden", start.start_mark)
node = self.compose_document()
if not self.check_event(StreamEndEvent):
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
self.get_event()
return node
# -- node level ---------------------------------------------------------------------------------------------------
def compose_node(self, parent, index):
event = self.peek_event()
if isinstance(event, AliasEvent):
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
if getattr(event, "anchor", None) is not None:
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
if getattr(event, "tag", None) is not None:
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
self._nodes += 1
if self._nodes > MAX_NODES:
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
if isinstance(index, ScalarNode):
segment = index.value
elif isinstance(index, int) and not isinstance(index, bool):
segment = index
else:
segment = None # a mapping key
if segment is not None:
self._path.append(segment)
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
self._depth += 1
try:
if self._depth > MAX_DEPTH:
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
return super().compose_node(parent, index)
finally:
self._depth -= 1
if segment is not None:
self._path.pop()
def compose_scalar_node(self, anchor):
event = self.peek_event()
if len(event.value) > MAX_SCALAR:
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
return super().compose_scalar_node(anchor)
def compose_mapping_node(self, anchor):
node = super().compose_mapping_node(anchor)
seen: set[str] = set()
for key, _value in node.value:
if not isinstance(key, ScalarNode) or key.tag != _STR:
raise _Rejected("non_string_key", key.start_mark, self._path)
if key.value == "<<":
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
if key.value in seen:
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
seen.add(key.value)
return node
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
if isinstance(raw, str):
raw = raw.encode("utf-8")
if len(raw) > MAX_BYTES:
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
try:
text = raw.decode("utf-8", errors="strict")
except UnicodeDecodeError as exc:
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
if text.startswith(""):
text = text[1:]
loader = None
try:
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
node = loader.get_single_node()
data = loader.construct_document(node)
except _Rejected as exc:
line = exc.mark.line + 1 if exc.mark is not None else None
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
except yaml.MarkedYAMLError as exc:
mark = exc.problem_mark or exc.context_mark
line = mark.line + 1 if mark is not None else None
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
finally:
if loader is not None:
loader.dispose()
return data, loader.lines
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
for end in range(len(loc), 0, -1):
if loc[:end] in lines:
return lines[loc[:end]]
return None
_PYDANTIC_CODES = {
"extra_forbidden": "unknown_field",
"missing": "missing_field",
"literal_error": "bad_value",
"string_too_short": "too_short",
"string_too_long": "too_long",
"string_pattern_mismatch": "bad_format",
}
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
out = []
for error in exc.errors(include_url=False):
loc = tuple(error["loc"])
code = _PYDANTIC_CODES.get(error["type"])
if code is None and error["type"] == "value_error":
message = str(error.get("ctx", {}).get("error", ""))
code = message if message in _OWN_CODES else "bad_value"
if code is None:
code = "bad_type"
params = {}
if error["type"] == "extra_forbidden":
params["field"] = display(loc[-1]) if loc else ""
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
params["value"] = display(error["input"])
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
return out
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
errors = []
for kind in schema.KINDS:
section = data.get(kind)
if kind == "variables":
if isinstance(section, dict):
if len(section) > MAX_VARIABLE_HOSTS:
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
for host, values in section.items():
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
max=MAX_VARIABLES_PER_HOST))
continue
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
for i, alias in enumerate(data.get("aliases") or []):
entries = alias.get("entries") if isinstance(alias, dict) else None
if isinstance(entries, list):
loc = ("aliases", i, "entries")
if len(entries) > MAX_ALIAS_ENTRIES:
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
return errors
class Parsed(NamedTuple):
doc: schema.ConfigDoc
sha256: str # of the raw bytes, shown in the plan and stored in the audit
lines: dict[tuple, int] # path -> line, for messages of the planner
def parse(raw: bytes | str) -> Parsed:
"""Load, limit and validate a file. Raises ConfigError."""
if isinstance(raw, str):
raw = raw.encode("utf-8")
sha256 = hashlib.sha256(raw).hexdigest()
data, lines = load_yaml(raw)
if not isinstance(data, dict):
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
version = data.get("opnexus_config")
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
"opnexus_config", _line_for(lines, ("opnexus_config",)),
**({"value": display(version)} if version is not None else {}))])
errors = _check_counts(data, lines)
if errors:
raise ConfigError("CONFIG_INVALID", errors)
try:
doc = schema.ConfigDoc.model_validate(data)
except ValidationError as exc:
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
return Parsed(doc, sha256, lines)
@@ -0,0 +1,933 @@
{
"$defs": {
"AliasDoc": {
"additionalProperties": false,
"properties": {
"name": {
"maxLength": 64,
"minLength": 1,
"pattern": "^[A-Za-z0-9_]+$",
"title": "Name",
"type": "string"
},
"type": {
"default": "host",
"enum": [
"host",
"network",
"port",
"networkgroup",
"mac",
"asn"
],
"title": "Type",
"type": "string"
},
"description": {
"default": "",
"maxLength": 255,
"title": "Description",
"type": "string"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"entries": {
"items": {
"maxLength": 8000,
"minLength": 1,
"type": "string"
},
"title": "Entries",
"type": "array"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"name"
],
"title": "AliasDoc",
"type": "object"
},
"DeviceGroupDoc": {
"additionalProperties": false,
"properties": {
"name": {
"maxLength": 120,
"minLength": 1,
"title": "Name",
"type": "string"
},
"description": {
"default": "",
"maxLength": 2000,
"title": "Description",
"type": "string"
},
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
}
},
"required": [
"name"
],
"title": "DeviceGroupDoc",
"type": "object"
},
"HierarchyGroupDoc": {
"additionalProperties": false,
"properties": {
"name": {
"maxLength": 120,
"minLength": 1,
"title": "Name",
"type": "string"
},
"parent": {
"anyOf": [
{
"maxLength": 120,
"minLength": 1,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Parent"
},
"description": {
"default": "",
"maxLength": 2000,
"title": "Description",
"type": "string"
},
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
}
},
"required": [
"name"
],
"title": "HierarchyGroupDoc",
"type": "object"
},
"HierarchyRuleDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"group": {
"maxLength": 120,
"minLength": 1,
"title": "Group",
"type": "string"
},
"rulebase": {
"default": "pre",
"enum": [
"pre",
"post"
],
"title": "Rulebase",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"direction": {
"default": "in",
"enum": [
"in",
"out"
],
"title": "Direction",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "any",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"action": {
"default": "pass",
"enum": [
"pass",
"block",
"reject"
],
"title": "Action",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
}
},
"required": [
"key",
"group",
"description",
"interface"
],
"title": "HierarchyRuleDoc",
"type": "object"
},
"OneToOneDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"type": {
"default": "binat",
"enum": [
"binat",
"nat"
],
"title": "Type",
"type": "string"
},
"external": {
"maxLength": 128,
"minLength": 1,
"title": "External",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"source_not": {
"default": false,
"title": "Source Not",
"type": "boolean"
},
"destination_not": {
"default": false,
"title": "Destination Not",
"type": "boolean"
},
"natreflection": {
"default": "",
"enum": [
"",
"enable",
"disable"
],
"title": "Natreflection",
"type": "string"
},
"log": {
"default": false,
"title": "Log",
"type": "boolean"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"key",
"description",
"interface",
"external"
],
"title": "OneToOneDoc",
"type": "object"
},
"OutboundNatDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "any",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"source_not": {
"default": false,
"title": "Source Not",
"type": "boolean"
},
"destination_not": {
"default": false,
"title": "Destination Not",
"type": "boolean"
},
"target": {
"default": "",
"maxLength": 128,
"title": "Target",
"type": "string"
},
"target_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Target Port"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"key",
"description",
"interface"
],
"title": "OutboundNatDoc",
"type": "object"
},
"PortForwardDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "tcp",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"target": {
"maxLength": 128,
"minLength": 1,
"title": "Target",
"type": "string"
},
"local_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Local Port"
},
"natreflection": {
"default": "",
"enum": [
"",
"purenat",
"disable"
],
"title": "Natreflection",
"type": "string"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"key",
"description",
"interface",
"target"
],
"title": "PortForwardDoc",
"type": "object"
},
"RuleDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"direction": {
"default": "in",
"enum": [
"in",
"out"
],
"title": "Direction",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "any",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"action": {
"default": "pass",
"enum": [
"pass",
"block",
"reject"
],
"title": "Action",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"log": {
"default": false,
"title": "Log",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/Targets"
}
},
"required": [
"key",
"description",
"interface"
],
"title": "RuleDoc",
"type": "object"
},
"Targets": {
"additionalProperties": false,
"properties": {
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
},
"groups": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Groups",
"type": "array"
}
},
"title": "Targets",
"type": "object"
},
"TargetsWithHierarchy": {
"additionalProperties": false,
"properties": {
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
},
"groups": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Groups",
"type": "array"
},
"hierarchy_group": {
"anyOf": [
{
"maxLength": 120,
"minLength": 1,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Hierarchy Group"
}
},
"title": "TargetsWithHierarchy",
"type": "object"
}
},
"additionalProperties": false,
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
"properties": {
"opnexus_config": {
"const": 1,
"title": "Opnexus Config",
"type": "integer"
},
"device_groups": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/DeviceGroupDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Device Groups"
},
"policy_hierarchy_groups": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/HierarchyGroupDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Policy Hierarchy Groups"
},
"aliases": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/AliasDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Aliases"
},
"rules": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/RuleDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Rules"
},
"port_forwards": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/PortForwardDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Port Forwards"
},
"one_to_one": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/OneToOneDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "One To One"
},
"outbound_nat": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/OutboundNatDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Outbound Nat"
},
"policy_hierarchy_rules": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/HierarchyRuleDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Policy Hierarchy Rules"
},
"variables": {
"anyOf": [
{
"additionalProperties": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"propertyNames": {
"maxLength": 120,
"minLength": 1
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Variables"
}
},
"required": [
"opnexus_config"
],
"title": "OPNexus-Konfiguration (Schema v1)",
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
+570
View File
@@ -0,0 +1,570 @@
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
`prune` gleicht die Art ab.
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
"""
import hashlib
import json
import re
import uuid
from dataclasses import dataclass, field
from pydantic import ValidationError
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.config_code import schema
from app.config_code.export import canonical
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
from app.database import Base
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
from app.schemas.alias import AliasCreate
from app.schemas.one_to_one import OneToOneCreate
from app.schemas.outbound_nat import OutboundNatCreate
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
from app.schemas.port_forward import PortForwardCreate
from app.schemas.rule import RuleCreate
from app.services.variables_service import names_in
BUSY = ("pending", "revert_failed")
DEPLOYED = ("success", "pending", "revert_failed")
@dataclass(frozen=True)
class Kind:
section: str
model: type | None
ident: str # identity field in the file ("key"/"name")
api: type | None = None # API create schema, run on every file object (same validators as the UI)
deployable: bool = False
KIND_SPECS = {
"device_groups": Kind("device_groups", DeviceGroup, "name"),
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
"variables": Kind("variables", None, "key"),
}
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
@dataclass
class Item:
kind: str
key: str
action: str # create | update | delete | unchanged
before: dict | None = None
after: dict | None = None
changes: list = field(default_factory=list)
reasons: list = field(default_factory=list)
deployed_hosts: int = 0
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
db_id: uuid.UUID | None = None
def public(self) -> dict:
return {
"kind": self.kind,
"key": self.key,
"action": "blocked" if self.reasons else self.action,
"planned_action": self.action,
"changes": self.changes,
"deployed_hosts": self.deployed_hosts,
"reasons": self.reasons,
"implicit": self.implicit,
}
@dataclass
class Plan:
sha256: str
prune: bool
touched: list[str]
items: list[Item]
warnings: list[dict]
token: str = ""
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
expected: dict = field(default_factory=dict)
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
@property
def applicable(self) -> bool:
return not any(i.reasons for i in self.items)
def summary(self) -> dict:
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
for item in self.items:
out["blocked" if item.reasons else item.action] += 1
return out
def public(self) -> dict:
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
return {
"token": self.token,
"sha256": self.sha256,
"prune": self.prune,
"kinds": self.touched,
"summary": self.summary(),
"items": [i.public() for i in changed],
"warnings": self.warnings,
"applicable": self.applicable,
}
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
section = getattr(doc, kind)
if kind == "variables":
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
for host, values in section.items() for name, value in values.items()]
return [item.model_dump(mode="python") for item in section]
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
def sorted_unique(values: list, where: tuple) -> list:
seen = set()
for i, value in enumerate(values):
if value in seen:
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
seen.add(value)
return sorted(seen)
item = dict(item)
if "targets" in item:
targets = dict(item["targets"])
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
item["targets"] = targets
if kind in ("device_groups", "policy_hierarchy_groups"):
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
return item
def _changes(before: dict, after: dict) -> list[dict]:
out = []
for name in after:
if name == "targets":
for sub in after["targets"]:
if before["targets"].get(sub) != after["targets"][sub]:
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
elif before.get(name) != after[name]:
out.append({"field": name, "before": before.get(name), "after": after[name]})
return out
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
buckets: dict = {}
for row in rows:
buckets.setdefault(bucket_of(row), []).append(row)
out = {}
for members in buckets.values():
for rank, row in enumerate(sorted(members, key=order_of)):
out[key_of(row)] = rank
return out
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
found: dict = {}
if not ids:
return found
for table in Base.metadata.sorted_tables:
if table.name in skip:
continue
for fk in table.foreign_keys:
if fk.column.table.name != target_table:
continue
column = fk.parent
rows = (await db.execute(
select(column, func.count()).where(column.in_(ids)).group_by(column)
)).all()
for ref_id, _count in rows:
found.setdefault(ref_id, []).append(table.name)
return found
async def _load(db: AsyncSession) -> dict:
loaded: dict = {}
for kind in TARGET_KINDS:
model = KIND_SPECS[kind].model
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
return loaded
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
doc, lines = parsed.doc, parsed.lines
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
errors: list[dict] = []
warnings: list[dict] = []
db_canon = await canonical(db)
loaded = await _load(db)
hosts = loaded["hosts"]
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
file_objects: dict[str, dict[str, dict]] = {}
file_locs: dict[tuple, tuple] = {}
for kind in touched:
spec = KIND_SPECS[kind]
objects: dict[str, dict] = {}
for index, raw in enumerate(_file_items(doc, kind)):
if kind == "variables":
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
else:
ident, loc = raw[spec.ident], (kind, index)
if ident in objects:
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
continue
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
file_locs[(kind, ident)] = loc
file_objects[kind] = objects
def db_items(kind: str) -> dict[str, dict]:
if kind == "variables":
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
for h, values in db_canon["variables"].items() for n, v in values.items()}
ident = KIND_SPECS[kind].ident
return {item[ident]: item for item in db_canon[kind]}
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
post: dict[str, dict[str, dict]] = {}
for kind in schema.KINDS:
if kind in file_objects:
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
if prune:
deleted[kind] = set(remaining)
remaining = {}
post[kind] = {**remaining, **file_objects[kind]}
else:
post[kind] = dict(db_objects[kind])
host_names = set(hosts)
group_names = set(post["device_groups"])
hierarchy_names = set(post["policy_hierarchy_groups"])
# -- references in the file ------------------------------------------------------------------------------------------
for kind in touched:
for ident, item in file_objects[kind].items():
loc = file_locs[(kind, ident)]
if kind == "variables":
if item["host"] not in host_names:
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
continue
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
for name in member_hosts:
if name not in host_names:
_err(errors, "unknown_host", loc, lines, host=display(name))
for name in item.get("targets", {}).get("groups", []):
if name not in group_names:
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
if ref is not None and ref not in hierarchy_names:
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
if kind == "policy_hierarchy_rules":
for name, value in item.items():
if names_in(value):
_err(errors, "placeholder_not_supported", loc + (name,), lines)
spec = KIND_SPECS[kind]
if spec.api is not None:
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
for name in member_hosts:
host = hosts.get(name)
if host is not None and not host.enabled:
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
if "policy_hierarchy_groups" in file_objects:
file_groups = file_objects["policy_hierarchy_groups"]
for host_name, group in list(host_hierarchy.items()):
if prune or group in file_groups:
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
host_hierarchy[host_name] = None
listed: dict[str, str] = {}
for name, group in file_groups.items():
for host_name in group["hosts"]:
if host_name in listed:
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
lines, host=display(host_name), first=display(listed[host_name]))
listed[host_name] = name
if host_name in host_hierarchy:
host_hierarchy[host_name] = name
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
for start in parents:
seen, current = set(), start
while current is not None and current not in seen:
seen.add(current)
current = parents.get(current)
if current is not None and current == start:
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
lines, group=display(start))
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
per_host: dict[str, int] = {}
for item in post["variables"].values():
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
for host_name, count in sorted(per_host.items()):
if count > MAX_VARIABLES_PER_HOST:
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
max=MAX_VARIABLES_PER_HOST, count=count)
if errors:
raise ConfigError("CONFIG_INVALID", errors[:100])
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
db_rule_rows = loaded["policy_hierarchy_rules"]
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
positions = dict(db_positions)
if "policy_hierarchy_rules" in file_objects:
buckets: dict[tuple, list[str]] = {}
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
for key, item in post["policy_hierarchy_rules"].items():
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
buckets.setdefault((item["group"], item["rulebase"]), [])
positions = {}
extras: dict[tuple, list[str]] = {}
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
key=lambda k: db_positions.get(k, 0)):
item = post["policy_hierarchy_rules"][key]
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
for bucket in set(buckets) | set(extras):
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
positions[key] = rank
# -- items -----------------------------------------------------------------------------------------------------------
items: list[Item] = []
for kind in touched:
rows = loaded.get(kind, {})
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
row = rows.get(ident) if kind != "variables" else None
before = db_objects[kind].get(ident)
after = post[kind].get(ident) if ident not in deleted[kind] else None
if before is None:
item = Item(kind, ident, "create", None, after)
elif after is None:
item = Item(kind, ident, "delete", before, None)
else:
changes = _changes(before, after)
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
if row is not None:
item.db_id = row.id
deployments = getattr(row, "deployments", None) or []
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
if item.action in ("update", "delete") and any(d.status in BUSY for d in deployments):
item.reasons.append("busy_deployment")
if item.action == "delete" and deployments:
item.reasons.append("prune_deployed")
items.append(item)
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
if redeploy:
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
warnings += _alias_reference_warnings(post, file_objects, file_locs)
warnings += _variable_warnings(post, deleted)
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
file_objects=file_objects)
plan.expected = {kind: post[kind] for kind in touched}
plan.token = _token(plan, db_canon, db_positions, loaded)
return plan
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
out = set()
if kind == "policy_hierarchy_rules":
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
if kind == "policy_hierarchy_groups":
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
return out
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
if kind == "aliases":
values["content"] = "\n".join(item["entries"])
if kind == "policy_hierarchy_rules":
values["hierarchy_group_id"] = uuid.UUID(int=0)
try:
cleaned = KIND_SPECS[kind].api.model_validate(values)
except ValidationError as exc:
for error in exc.errors(include_url=False):
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
return item
out = dict(item)
for name in item:
if name in values and name != "content" and hasattr(cleaned, name):
out[name] = getattr(cleaned, name)
if kind == "aliases":
out["entries"] = [e for e in cleaned.content.split("\n") if e]
return out
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
by_key = {(i.kind, i.key): i for i in items}
# device groups: post-state objects of the import, plus every other table that points at the group
if deleted["device_groups"]:
users: dict[str, list[str]] = {}
for kind in TARGET_KINDS:
for ident, item in post[kind].items():
for group in item["targets"]["groups"]:
users.setdefault(group, []).append(f"{kind}:{ident}")
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
users.setdefault(ids[ref_id], []).extend(tables)
for name in deleted["device_groups"]:
if users.get(name):
by_key[("device_groups", name)].reasons.append("prune_referenced")
if deleted["policy_hierarchy_groups"]:
users = {}
for kind in HIERARCHY_TARGET_KINDS:
for ident, item in post[kind].items():
if item["targets"]["hierarchy_group"]:
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
for ident, item in post["policy_hierarchy_rules"].items():
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
for name, item in post["policy_hierarchy_groups"].items():
if item["parent"]:
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
for host_name, group in host_hierarchy.items():
if group:
users.setdefault(group, []).append(f"host:{host_name}")
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
"policy_hierarchy_rules", "opnsense_hosts"}
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
users.setdefault(ids[ref_id], []).extend(tables)
for name in deleted["policy_hierarchy_groups"]:
if users.get(name):
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
if deleted["aliases"]:
used: set[str] = set()
for kind in RULE_KINDS:
for item in post[kind].values():
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
for item in post["aliases"].values():
used.update(item["entries"])
for name in deleted["aliases"]:
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
by_key[("aliases", name)].reasons.append("prune_referenced")
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
aliases = set(post["aliases"])
out = []
for kind in RULE_KINDS:
for ident, item in file_objects.get(kind, {}).items():
for name in ALIAS_REF_FIELDS:
value = item.get(name)
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
"params": {"value": display(value)}})
return out
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
if not deleted["variables"]:
return []
used: set[str] = set()
for kind in RULE_KINDS + ("aliases",):
for item in post[kind].values():
for value in item.values():
if isinstance(value, str):
used |= names_in(value)
elif isinstance(value, list):
for entry in value:
used |= names_in(entry)
out = []
for key in sorted(deleted["variables"]):
host, name = key.split("/", 1)
if name in used:
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
return out
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
deployments = {
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
for ident, row in loaded[kind].items()}
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
}
material = {
"file": plan.file_objects,
"prune": plan.prune,
"db": db_canon,
"positions": db_positions,
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
"deployments": deployments,
"items": [i.public() for i in plan.items],
"warnings": plan.warnings,
}
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
+242
View File
@@ -0,0 +1,242 @@
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
"""
import re
from typing import Annotated, Literal
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
from app.models.config_key import CONFIG_KEY_RE
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
SCHEMA_VERSION = 1
MAX_SCALAR = 8000
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
def _clean(value: str) -> str:
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
if _CONTROL.search(value):
raise ValueError("control_character")
if value != value.strip():
raise ValueError("surrounding_whitespace")
return value
def _clean_value(value: str) -> str:
if _CONTROL.search(value):
raise ValueError("control_character")
return value
def _clean_text(value: str) -> str:
if _CONTROL_TEXT.search(value):
raise ValueError("control_character")
return value
def _port(value):
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
Text = Annotated[str, AfterValidator(_clean)]
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
Port = Annotated[PortText | None, BeforeValidator(_port)]
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
IpProtocol = Literal["inet", "inet6"]
class _Doc(BaseModel):
model_config = ConfigDict(extra="forbid", strict=True)
class Targets(_Doc):
hosts: list[HostName] = Field(default_factory=list)
groups: list[GroupName] = Field(default_factory=list)
class TargetsWithHierarchy(Targets):
hierarchy_group: GroupName | None = None
class DeviceGroupDoc(_Doc):
name: GroupName
description: GroupDescription = ""
hosts: list[HostName] = Field(default_factory=list)
class HierarchyGroupDoc(_Doc):
name: GroupName
parent: GroupName | None = None
description: GroupDescription = ""
hosts: list[HostName] = Field(default_factory=list)
class AliasDoc(_Doc):
name: AliasName
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
description: Description = ""
enabled: bool = True
entries: list[AliasEntry] = Field(default_factory=list)
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class RuleDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
direction: Literal["in", "out"] = "in"
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "any"
action: Literal["pass", "block", "reject"] = "pass"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
enabled: bool = True
log: bool = False
targets: Targets = Field(default_factory=Targets)
class PortForwardDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "tcp"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
target: Net
local_port: Port = None
natreflection: Literal["", "purenat", "disable"] = ""
enabled: bool = True
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class OneToOneDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
type: Literal["binat", "nat"] = "binat"
external: Net
source_net: Net = "any"
destination_net: Net = "any"
source_not: bool = False
destination_not: bool = False
natreflection: Literal["", "enable", "disable"] = ""
log: bool = False
enabled: bool = True
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class OutboundNatDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "any"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
source_not: bool = False
destination_not: bool = False
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
target_port: Port = None
enabled: bool = True
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class HierarchyRuleDoc(_Doc):
key: Key
group: GroupName
rulebase: Literal["pre", "post"] = "pre"
description: RuleDescription
interface: Interface
direction: Literal["in", "out"] = "in"
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "any"
action: Literal["pass", "block", "reject"] = "pass"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
enabled: bool = True
def _variable_name(value: str) -> str:
if not VARIABLE_NAME_RE.match(value):
raise ValueError("bad_variable_name")
return value
def _variable_value(value: str) -> str:
if not VARIABLE_VALUE_RE.match(value):
raise ValueError("bad_variable_value")
return value
VariableName = Annotated[str, AfterValidator(_variable_name)]
VariableValue = Annotated[str, AfterValidator(_variable_value)]
class ConfigDoc(_Doc):
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
opnexus_config: Literal[1]
device_groups: list[DeviceGroupDoc] | None = None
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
aliases: list[AliasDoc] | None = None
rules: list[RuleDoc] | None = None
port_forwards: list[PortForwardDoc] | None = None
one_to_one: list[OneToOneDoc] | None = None
outbound_nat: list[OutboundNatDoc] | None = None
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
KINDS = (
"device_groups",
"policy_hierarchy_groups",
"aliases",
"rules",
"port_forwards",
"one_to_one",
"outbound_nat",
"policy_hierarchy_rules",
"variables",
)
SCHEMA_FILE = "opnexus-config-v1.schema.json"
def json_schema_text() -> str:
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
import json
schema = ConfigDoc.model_json_schema()
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
+32
View File
@@ -171,6 +171,7 @@ class ErrorCode(str, Enum):
OPEN_DEPLOYMENT_DRAFT_UNCHANGED = "OPEN_DEPLOYMENT_DRAFT_UNCHANGED"
OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY = "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
OPEN_RESTORE_BLOCKS_DEPLOY = "OPEN_RESTORE_BLOCKS_DEPLOY"
OPEN_CLEANUP_BLOCKS_DEPLOY = "OPEN_CLEANUP_BLOCKS_DEPLOY"
UPSTREAM_REJECTED = "UPSTREAM_REJECTED"
INVALID_CHARACTERS = "INVALID_CHARACTERS"
SERVICE_NOT_FOUND = "SERVICE_NOT_FOUND"
@@ -222,6 +223,37 @@ class ErrorCode(str, Enum):
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
# Rule cleanup (1.31, services/rule_cleanup_service.py)
BEREINIGUNG_NICHT_GEFUNDEN = "BEREINIGUNG_NICHT_GEFUNDEN"
BEREINIGUNG_AUSWAHL_UNGUELTIG = "BEREINIGUNG_AUSWAHL_UNGUELTIG"
BEREINIGUNG_VORSCHAU_VERALTET = "BEREINIGUNG_VORSCHAU_VERALTET"
BEREINIGUNG_REGEL_NICHT_AENDERBAR = "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
BEREINIGUNG_OFFENE_OPERATION = "BEREINIGUNG_OFFENE_OPERATION"
BEREINIGUNG_ZU_VIELE_REGELN = "BEREINIGUNG_ZU_VIELE_REGELN"
BEREINIGUNG_BESTAETIGUNG_FEHLT = "BEREINIGUNG_BESTAETIGUNG_FEHLT"
BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN = "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG = "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG"
BEREINIGUNG_FRIST_ABGELAUFEN = "BEREINIGUNG_FRIST_ABGELAUFEN"
BEREINIGUNG_NICHT_BESTAETIGBAR = "BEREINIGUNG_NICHT_BESTAETIGBAR"
BEREINIGUNG_LIVESTAND_ABWEICHEND = "BEREINIGUNG_LIVESTAND_ABWEICHEND"
BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR = "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT = "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT"
BEREINIGUNG_SNAPSHOT_FEHLT = "BEREINIGUNG_SNAPSHOT_FEHLT"
BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR = "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
BEREINIGUNG_AKTION_NICHT_MOEGLICH = "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
BEREINIGUNG_NICHT_AUSGEFUEHRT = "BEREINIGUNG_NICHT_AUSGEFUEHRT"
BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN = "BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN"
BEREINIGUNG_AUSGANG_UNKLAR = "BEREINIGUNG_AUSGANG_UNKLAR"
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
CONFIG_INVALID = "CONFIG_INVALID"
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
CONFIG_BLOCKED = "CONFIG_BLOCKED"
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
CONFIG_BUSY = "CONFIG_BUSY"
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
+12
View File
@@ -53,6 +53,8 @@ from app.routers import scheduled_maintenance as scheduled_maintenance_router
from app.routers import maintenance_windows as maintenance_windows_router
from app.routers import rollouts as rollouts_router
from app.routers import rule_analysis as rule_analysis_router
from app.routers import rule_cleanup as rule_cleanup_router
from app.routers import config_code as config_code_router
from app.routers import gui_access as gui_access_router
from app.services import scheduled_maintenance_service
from app.services import rollout_engine
@@ -62,6 +64,7 @@ from app.services import overlay_service, variables_service
from app.services import backup_schedule_service
from app.services import carp_event_service, carp_maintenance_window_service
from app.services import rule_history_service
from app.services import rule_cleanup_service
# Without this, Python's root logger defaults to WARNING with no handler,
# so every logger.info() call in the 5 background loops below (CARP role
@@ -142,6 +145,13 @@ async def _deploy_revert_sweep_loop():
logger.info("Auto-reverted %d unconfirmed policy-hierarchy deployment(s)", len(reverted))
except Exception:
logger.exception("Policy-hierarchy deploy-revert sweep failed")
try:
async with SessionLocal() as db:
reverted = await rule_cleanup_service.sweep_expired_cleanups(db)
if reverted:
logger.info("Regel-Bereinigung: %d Operation(en) automatisch zurückgenommen/abgeschlossen", len(reverted))
except Exception:
logger.exception("Rule-cleanup revert sweep failed")
await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
@@ -561,5 +571,7 @@ app.include_router(scheduled_maintenance_router.router, prefix="/api")
app.include_router(maintenance_windows_router.router, prefix="/api")
app.include_router(rollouts_router.router, prefix="/api")
app.include_router(rule_analysis_router.router, prefix="/api")
app.include_router(rule_cleanup_router.router, prefix="/api")
app.include_router(config_code_router.router, prefix="/api")
app.include_router(gui_access_router.router, prefix="/api")
app.include_router(gui_access_router.internal_router)
+8
View File
@@ -105,3 +105,11 @@ __all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
from app.models.rule_history import LiveRuleState, RuleChangeEvent
__all__ += ["LiveRuleState", "RuleChangeEvent"]
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
from app.models.host_variable import HostVariable
__all__ += ["HostVariable"]
+34
View File
@@ -0,0 +1,34 @@
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
"""
import re
import uuid
from sqlalchemy import String, event
from sqlalchemy.orm import Mapped, mapped_column
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
def config_key_column() -> Mapped[str]:
return mapped_column(String(64), unique=True)
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
return f"{prefix}-{object_id.hex[:12]}"
def register_config_key(model: type, prefix: str) -> None:
"""Fills an empty `config_key` just before INSERT (and the id, which is
otherwise only assigned while the INSERT is built)."""
@event.listens_for(model, "before_insert")
def _fill(_mapper, _connection, target) -> None:
if target.id is None:
target.id = uuid.uuid4()
if not target.config_key:
target.config_key = default_config_key(prefix, target.id)
+12 -1
View File
@@ -1,5 +1,5 @@
"""Separation of duties (singleton row): may operators roll changes out themselves?"""
from sqlalchemy import Boolean, Integer
from sqlalchemy import JSON, Boolean, Integer
from sqlalchemy.orm import Mapped, mapped_column
from app.database import Base
@@ -11,3 +11,14 @@ class GovernanceSettings(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True)
# False: an operator prepares changes (edit, preview) but only an admin may deploy them.
operators_may_deploy: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
# Rule cleanup (1.31). At most this many rules per cleanup operation (default 10, ceiling 25).
cleanup_max_rules: Mapped[int] = mapped_column(Integer, nullable=False, default=10)
# A rule must have been switched off (and confirmed) this many days before it may be deleted (minimum 1).
cleanup_delete_min_days: Mapped[int] = mapped_column(Integer, nullable=False, default=7)
# IPs/CIDRs the admin manages the firewalls from; no rule that could match them may be cleaned up. Empty list =
# the cleanup is locked (fail closed). Only an admin may change it.
cleanup_management_sources: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
# Extra TCP ports the firewalls are managed on (besides the API port of base_url, 443, 80 and 22): a rule that
# could match a management source on one of them is protected too. Only an admin may change it.
cleanup_management_ports: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.one_to_one_target import one_to_one_targets
from app.models.policy import one_to_one_groups
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
__tablename__ = "one_to_one_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
source_net: Mapped[str] = mapped_column(String(128))
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
)
register_config_key(OneToOneRule, "nat1")
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.outbound_nat_target import outbound_nat_targets
from app.models.policy import outbound_nat_groups
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
__tablename__ = "outbound_nat_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
source_net: Mapped[str] = mapped_column(String(128))
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
)
register_config_key(OutboundNatRule, "out")
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
class PolicyHierarchyRule(Base):
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
__tablename__ = "policy_hierarchy_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
)
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
)
register_config_key(PolicyHierarchyRule, "ph")
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.port_forward_target import port_forward_targets
from app.models.policy import port_forward_groups
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
__tablename__ = "port_forward_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
)
register_config_key(PortForwardRule, "pf")
+2
View File
@@ -42,6 +42,8 @@ class RolloutPlan(Base):
# is activated AND unchanged since (activated_fingerprint).
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.rule_target import rule_targets
from app.models.policy import rule_groups
@@ -18,6 +19,7 @@ class FirewallRule(Base):
__tablename__ = "firewall_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
direction: Mapped[str] = mapped_column(String(8))
@@ -44,3 +46,6 @@ class FirewallRule(Base):
deployments: Mapped[list["RuleDeployment"]] = relationship(
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
)
register_config_key(FirewallRule, "r")
+106
View File
@@ -0,0 +1,106 @@
"""Rule cleanup (1.31, B4 part 2): switching rules off or deleting them through preview -> deploy -> confirm.
``RuleCleanupOperation`` is one cleanup run on a host (or a cluster pair, anchored at one host); its id doubles as
the preview token. ``RuleCleanupItem`` is one rule on one node of that run. The columns only carry what the design
(DESIGN-1.31) needs to resume, revert and restore after a crash; the logic lives in services/rule_cleanup_service.py.
"""
import uuid
from datetime import datetime, timezone
from typing import Optional
from sqlalchemy import JSON, DateTime, ForeignKey, Index, Integer, String, Text, text
from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column
from app.database import Base
# kind: disable | enable (re-enable after a restore) | delete | restore
OPERATION_KINDS = ("disable", "enable", "delete", "restore")
# preview -> running -> awaiting_confirm -> confirmed; the rest are the side exits.
OPERATION_STATUSES = (
"preview", "running", "awaiting_confirm", "confirmed", "partial", "apply_failed", "reverting", "reverted",
"revert_failed", "done", "failed", "cancelled", "expired",
)
ITEM_STATES = (
"queued", "disabling", "disabled", "confirmed", "reverting", "reverted", "deleting", "deleted", "restoring",
"restored", "failed", "unknown", "revert_failed",
# never attempted (the run stopped at an earlier error); not an open state, so the rule is free again
"skipped",
)
# A rule in one of these states is being worked on or its outcome is unresolved: no second item may claim it.
OPEN_ITEM_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
def _now() -> datetime:
return datetime.now(timezone.utc)
class RuleCleanupOperation(Base):
__tablename__ = "rule_cleanup_operations"
__table_args__ = (
Index("ix_rule_cleanup_operations_host", "host_id", "created_at"),
Index("ix_rule_cleanup_operations_status", "status"),
)
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
cluster_name: Mapped[Optional[str]] = mapped_column(String(120), nullable=True)
kind: Mapped[str] = mapped_column(String(16), nullable=False)
status: Mapped[str] = mapped_column(String(20), nullable=False, default="preview")
actor: Mapped[str] = mapped_column(String(150), nullable=False)
# The stable account id next to the name (see migration 0026); no foreign key, a deleted account stays gone.
actor_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
started_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
# Written at every checkpoint, so another process can tell an abandoned ``running`` operation from a live one.
heartbeat_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
# Auto-revert deadline (disable/enable only); NULL once confirmed or for kinds without a window.
revert_deadline: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
confirmed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
# sha256 over everything the preview showed; the deploy recomputes it and refuses on any difference.
preview_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
# unused_days, acknowledged rules (with the text version), protected management sources.
options: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
class RuleCleanupItem(Base):
__tablename__ = "rule_cleanup_items"
__table_args__ = (
Index("ix_rule_cleanup_items_operation", "operation_id"),
# Independent of any lock: one rule on one node can be in only one unfinished item at a time.
Index(
"uq_cleanup_rule_open", "host_id", "rule_uuid", unique=True,
postgresql_where=text("state IN (" + ", ".join(f"'{s}'" for s in OPEN_ITEM_STATES) + ")"),
),
)
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
operation_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rule_cleanup_operations.id", ondelete="CASCADE"), nullable=False)
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
seq: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
carp_role: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
state: Mapped[str] = mapped_column(String(16), nullable=False, default="queued")
# What the preview showed.
description: Mapped[str] = mapped_column(Text, nullable=False, default="")
categories: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
confidence: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
row_before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
fp_before: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
# shadowed/duplicate: the covering rule must stay unchanged until the deploy.
cover_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
cover_fp: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
# Flattened getRule, raw answer and firmware; written BEFORE a rule is deleted.
snapshot: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
# Where the rule sat (neighbours in the same prio_group by sort_order), for the restore.
anchor_next_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
anchor_prev_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
prio_group: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
# delete -> the confirmed disable item it builds on; restore -> the delete item it undoes. Plain id, no foreign key.
source_item_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
restored_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
changed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
+2
View File
@@ -205,6 +205,8 @@ async def remove_previous_target(alias_id: uuid.UUID, host_id: uuid.UUID, approv
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
deployment = next(d for d in alias.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
await opnsense_client.del_alias_item(deployment.host, deployment.opnsense_alias_uuid)
+17 -4
View File
@@ -1,14 +1,15 @@
from datetime import datetime
from fastapi import APIRouter, Depends, Query
from fastapi import APIRouter, Depends, Query, Request
from sqlalchemy import select, func, or_
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models.audit import AuditEvent
from app.services.audit_service import ADMIN_ONLY_DETAILS
router = APIRouter(prefix="/audit", tags=["audit"])
@router.get("")
async def history(kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
async def history(request: Request, kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
status: str | None = None, action: str | None = None, actor: str | None = None,
before: datetime | None = None, since: datetime | None = None,
q: str = Query(default="", max_length=200), offset: int = Query(default=0, ge=0),
@@ -24,5 +25,17 @@ async def history(kind: str | None = None, host_id: str | None = None, entity_id
clauses.append(or_(*(c.ilike('%'+q+'%') for c in [AuditEvent.entity_name, AuditEvent.host_name, AuditEvent.actor, AuditEvent.message])))
total = await db.scalar(select(func.count()).select_from(AuditEvent).where(*clauses))
rows = (await db.scalars(select(AuditEvent).where(*clauses).order_by(AuditEvent.created_at.desc(), AuditEvent.id.desc()).offset(offset).limit(limit))).all()
return {"total": total, "offset": offset, "limit": limit,
"items": [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]}
items = [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]
if not full_view(request):
for item in items:
if isinstance(item.get("details"), dict) and ADMIN_ONLY_DETAILS in item["details"]:
item["details"] = {**{k: v for k, v in item["details"].items() if k != ADMIN_ONLY_DETAILS},
ADMIN_ONLY_DETAILS: "redacted"}
return {"total": total, "offset": offset, "limit": limit, "items": items}
def full_view(request: Request | None) -> bool:
"""An admin account, and with a token only one of scope admin."""
user = getattr(getattr(request, "state", None), "user", None)
token = getattr(getattr(request, "state", None), "api_token", None)
return getattr(user, "role", None) == "admin" and (token is None or token.scope == "admin")
+154
View File
@@ -0,0 +1,154 @@
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
"""
import asyncio
import json
from fastapi import APIRouter, Depends, Query, Request
from fastapi.concurrency import run_in_threadpool
from fastapi.responses import Response
from pydantic import BaseModel, ConfigDict, Field, ValidationError
from sqlalchemy.ext.asyncio import AsyncSession
from app.config_code import apply as applier
from app.config_code import plan as planner
from app.config_code import schema
from app.config_code.export import export_yaml
from app.config_code.loader import MAX_BYTES, ConfigError, parse
from app.database import get_db
from app.errors import ErrorCode, api_error
router = APIRouter(prefix="/config", tags=["config"])
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
# and read before parsing so a huge body never reaches the JSON parser.
MAX_REQUEST_BYTES = 4 * MAX_BYTES
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
# open to read-only accounts and read tokens.
_busy = asyncio.Semaphore(1)
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
MAX_WAITING = 4
_waiting = 0
class _Slot:
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
async def __aenter__(self):
global _waiting
if _waiting >= MAX_WAITING:
raise api_error(429, ErrorCode.CONFIG_BUSY)
_waiting += 1
return self
async def run(self, coro_fn):
async with _busy:
return await coro_fn()
async def __aexit__(self, *exc):
global _waiting
_waiting -= 1
def _encode(text: str) -> bytes:
try:
return text.encode("utf-8")
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
class ImportRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
yaml: str
prune: bool = False
class ApplyRequest(ImportRequest):
plan_token: str = Field(min_length=64, max_length=64)
def _config_error(exc: ConfigError):
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
async def _read_body(request: Request, model: type[BaseModel]):
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
chunks, size = [], 0
async for chunk in request.stream():
size += len(chunk)
if size > MAX_REQUEST_BYTES:
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
chunks.append(chunk)
try:
return model.model_validate(json.loads(b"".join(chunks)))
except (ValueError, ValidationError):
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
@router.get("/export")
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
wanted = None
if kinds:
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
if not wanted <= set(schema.KINDS):
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
text = await export_yaml(db, wanted)
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
@router.post("/import/plan")
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
async with _Slot() as slot:
body = await _read_body(request, ImportRequest)
raw = _encode(body.yaml)
async def work():
try:
parsed = await run_in_threadpool(parse, raw)
return await planner.build_plan(db, parsed, body.prune)
except ConfigError as exc:
raise _config_error(exc) from None
finally:
await db.rollback() # never leave anything in the session: plan is read-only
result = await slot.run(work)
return result.public()
@router.post("/import/apply")
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
async with _Slot() as slot:
body = await _read_body(request, ApplyRequest)
raw = _encode(body.yaml)
async def work():
try:
parsed = await run_in_threadpool(parse, raw)
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
except ConfigError as exc:
await db.rollback()
raise _config_error(exc) from None
except applier.PlanStale:
await db.rollback()
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
except applier.PlanBlocked as exc:
await db.rollback()
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
except applier.SelfCheckFailed as exc:
await db.rollback()
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
except Exception:
await db.rollback()
raise
await db.commit()
return parsed, result
parsed, result = await slot.run(work)
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
+105 -6
View File
@@ -1,6 +1,15 @@
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it)."""
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it).
The rule-cleanup settings (1.31) live here too: how many rules one cleanup may touch, how long a rule must stay
switched off before it may be deleted, the management sources (addresses the firewalls are managed from; no rule
that could match them is ever cleaned up) and extra management ports (besides the API port, 443, 80 and 22). Only an admin writes any of it (middleware: ``/api/settings/...`` is
admin-only for operators and tokens below scope admin; checked again here) and every change is audited with the
old and the new values."""
import ipaddress
from typing import Any
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
@@ -8,12 +17,83 @@ from app.errors import ErrorCode, api_error
from app.models.governance import GovernanceSettings
from app.models.user import AdminUser
from app.routers.auth import current_user
from app.services import audit_service
router = APIRouter(prefix="/settings/governance", tags=["governance"])
MAX_CLEANUP_RULES = 25 # same ceiling as services/rule_cleanup_service.MAX_RULES_CEILING
MAX_DELETE_MIN_DAYS = 365
MAX_MANAGEMENT_SOURCES = 20
MAX_MANAGEMENT_PORTS = 20
class GovernanceRead(BaseModel):
operators_may_deploy: bool
cleanup_max_rules: int = 10
cleanup_delete_min_days: int = 7
# The list itself is shown to admins only; everybody else sees just whether it is set.
cleanup_management_sources: list[str] = Field(default_factory=list)
cleanup_management_sources_configured: bool = False
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
cleanup_management_ports: list[int] = Field(default_factory=list)
cleanup_management_ports_configured: bool = False
class GovernanceUpdate(BaseModel):
operators_may_deploy: bool | None = None
cleanup_max_rules: int | None = None
cleanup_delete_min_days: int | None = None
cleanup_management_sources: list[str] | None = None
cleanup_management_ports: list[Any] | None = None # validated in clean_management_ports (no bool coercion)
def _bad(field: str, reason: str):
return api_error(422, ErrorCode.BEREINIGUNG_EINSTELLUNG_UNGUELTIG, field=field, reason=reason)
def clean_management_sources(values: list[str]) -> list[str]:
"""IP addresses and CIDR networks only (no host names: they could resolve differently later), canonical form,
without duplicates; not the whole address space, which would protect every rule."""
if len(values) > MAX_MANAGEMENT_SOURCES:
raise _bad("cleanup_management_sources", "too_many")
cleaned: list[str] = []
for raw in values:
text = str(raw).strip()
try:
net = ipaddress.ip_network(text, strict=False)
except ValueError:
raise _bad("cleanup_management_sources", "not_an_address") from None
if net.prefixlen == 0:
raise _bad("cleanup_management_sources", "everything")
entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)
if entry not in cleaned:
cleaned.append(entry)
return cleaned
def clean_management_ports(values) -> list[int]:
"""TCP ports 1..65535, at most MAX_MANAGEMENT_PORTS, without duplicates, sorted."""
if len(values) > MAX_MANAGEMENT_PORTS:
raise _bad("cleanup_management_ports", "too_many")
ports: set[int] = set()
for raw in values:
text = str(raw).strip()
if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:
raise _bad("cleanup_management_ports", "not_a_port")
ports.add(int(text))
return sorted(ports)
def _read(row: GovernanceSettings, user: AdminUser) -> GovernanceRead:
sources = list(row.cleanup_management_sources or [])
ports = list(row.cleanup_management_ports or [])
return GovernanceRead(
operators_may_deploy=row.operators_may_deploy, cleanup_max_rules=row.cleanup_max_rules or 10,
cleanup_delete_min_days=row.cleanup_delete_min_days or 7,
cleanup_management_sources=sources if user.role == "admin" else [],
cleanup_management_sources_configured=bool(sources),
cleanup_management_ports=ports if user.role == "admin" else [],
cleanup_management_ports_configured=bool(ports))
async def get_settings(db: AsyncSession) -> GovernanceSettings:
@@ -28,14 +108,33 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
@router.get("", response_model=GovernanceRead)
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
return await get_settings(db)
return _read(await get_settings(db), user)
@router.put("", response_model=GovernanceRead)
async def update(payload: GovernanceRead, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
if user.role != "admin":
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
row = await get_settings(db)
row.operators_may_deploy = payload.operators_may_deploy
changes: dict[str, dict] = {}
if payload.cleanup_max_rules is not None:
if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:
raise _bad("cleanup_max_rules", f"1..{MAX_CLEANUP_RULES}")
if payload.cleanup_delete_min_days is not None:
if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:
raise _bad("cleanup_delete_min_days", f"1..{MAX_DELETE_MIN_DAYS}")
sources = clean_management_sources(payload.cleanup_management_sources) if payload.cleanup_management_sources is not None else None
ports = clean_management_ports(payload.cleanup_management_ports) if payload.cleanup_management_ports is not None else None
for field, value in (("cleanup_max_rules", payload.cleanup_max_rules), ("cleanup_delete_min_days", payload.cleanup_delete_min_days),
("cleanup_management_sources", sources), ("cleanup_management_ports", ports)):
if value is not None and value != getattr(row, field):
changes[field] = {"old": getattr(row, field), "new": value}
setattr(row, field, value)
if payload.operators_may_deploy is not None:
row.operators_may_deploy = payload.operators_may_deploy
if changes:
db.add(audit_service.make_event(
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
await db.commit()
return row
return _read(row, user)
+8
View File
@@ -147,6 +147,11 @@ async def get_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
host = await _get_host_or_404(db, host_id)
if (payload.base_url is not None or payload.enabled is not None or payload.clear_cluster_name
or payload.cluster_name is not None):
# An open rule cleanup (1.31) reverts through this address and this cluster pairing: not while it is open.
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [host])
if payload.name is not None:
existing = await db.scalar(
@@ -188,6 +193,9 @@ async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
host = await db.get(OpnsenseHost, host_id)
if not host:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [host])
# Both deployment tables FK to opnsense_hosts with ondelete=CASCADE, so
# deleting the host would silently drop OPNexus's only record of what
# it pushed there — the rule/alias itself stays live on the actual
+2
View File
@@ -193,6 +193,8 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_child_uuid:
+102 -8
View File
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
from app.database import get_db
from app.models import OpnsenseHost
from app.models.host_operation import HostOperation
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
class Preview(BaseModel):
action: Literal['reboot','update','major_update']
action: Literal['reboot','update','major_update','firmware_rollback']
class Execute(BaseModel):
preview_id: uuid.UUID
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
def read_operation(op):
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
async def snapshot_or_502(db, host):
try: return await inspect_host(db,host)
async def snapshot_or_502(db, host, action=None):
try:
state=await inspect_host(db,host)
if action=='firmware_rollback':
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
rollback=await snapshot_service.rollback_state(host)
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
state['rollback']=rollback
state['actions']['firmware_rollback']=state['actions']['reboot']+(
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
return state
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
async def _changes_since(db, host, target):
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
if not target:
return []
since=_point_time(target)
if since is None:
return []
out=[]
for kind,model in _DEPLOYMENT_MODELS():
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
return sorted(out,key=lambda r:r['deployed_at'])[:50]
def _point_time(target):
try:
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
except (KeyError, TypeError, ValueError):
return None
def _DEPLOYMENT_MODELS():
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
PortForwardDeployment, RuleDeployment)
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
async def _mark_rolled_back(db, host, target):
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
really writes, and the UI shows the object as not in sync."""
since=_point_time(target)
if since is None:
return 0
count=0
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
for _kind,model in _DEPLOYMENT_MODELS():
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
row.status='error'; row.message=note[:500]; row.last_payload=None
if hasattr(model,'revert_deadline'): row.revert_deadline=None
count+=1
return count
async def carp_snapshot_or_502(db, host):
try: return await inspect_carp_switch(db,host)
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
host=await host_or_404(db,host_id)
state=await snapshot_or_502(db,host)
state=await snapshot_or_502(db,host,payload.action)
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
state=await snapshot_or_502(db,host)
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
state=await snapshot_or_502(db,host,op.action)
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
# replaying a reboot/update automatically could interrupt recovery.
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
if op.action in ('update','major_update'):
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
try:
point=await snapshot_service.create_rollback_point(host,state.get('version'))
except client.OpnsenseAPIError as exc:
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
await db.commit()
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
return read_operation(op)
op.snapshot={**op.snapshot,'rollback_point':point}
await db.commit()
if op.action=='firmware_rollback':
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
# Fail closed, like the backup before an update.
try:
backup_store.save_snapshot(host.id, await client.download_backup(host))
except client.OpnsenseAPIError as exc:
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
await db.commit()
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
return read_operation(op)
rollback=state['rollback']
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
else:
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
try:
result=await action_fn(host)
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
op.status='submitted';op.message=done
if op.action=='firmware_rollback':
marked=await _mark_rolled_back(db,host,rollback['target'])
if marked:
op.message+=f' {marked} Deployment(s) als unklar markiert.'
except client.OpnsenseAPIError as exc:
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
result={}
+3
View File
@@ -50,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
source_net=rule.source_net,
@@ -231,6 +232,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
deployment = next(d for d in rule.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
await opnsense_client.del_one_to_one_rule(deployment.host, deployment.opnsense_one_to_one_uuid)
+2
View File
@@ -195,6 +195,8 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
deployment = next(d for d in instance.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_instance_uuid:
+3
View File
@@ -51,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
source_net=rule.source_net,
@@ -238,6 +239,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
deployment = next(d for d in rule.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
await opnsense_client.del_outbound_nat_rule(deployment.host, deployment.opnsense_outbound_nat_uuid)
+3
View File
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
hierarchy_group_id=rule.hierarchy_group_id,
rulebase=rule.rulebase,
sequence=rule.sequence,
@@ -330,6 +331,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
deployment = next(d for d in rule.deployments if d.host_id == host_id)
from app.services import opnsense_client
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
try:
await opnsense_client.del_rule(deployment.host, deployment.opnsense_policy_hierarchy_uuid)
await opnsense_client.apply(deployment.host)
+3
View File
@@ -50,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
ipprotocol=rule.ipprotocol,
@@ -233,6 +234,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
deployment = next(d for d in rule.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
await opnsense_client.del_port_forward_rule(deployment.host, deployment.opnsense_port_forward_uuid)
+27 -2
View File
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
from typing import Literal
from fastapi import APIRouter, Depends, Request
from pydantic import BaseModel, Field
from pydantic import BaseModel, Field, model_validator
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -20,7 +20,7 @@ from app.models.maintenance_window import MaintenanceWindow
from app.models.policy import DeviceGroup
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
@@ -36,6 +36,19 @@ class WaveIn(BaseModel):
require_approval: bool = False
class ProbeIn(BaseModel):
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
type: Literal["ping", "tcp"]
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
port: int | None = Field(default=None, ge=1, le=65535)
@model_validator(mode="after")
def _port_only_for_tcp(self):
if (self.type == "tcp") != (self.port is not None):
raise ValueError("port_required_for_tcp_only")
return self
class PlanIn(BaseModel):
name: str = Field(min_length=1, max_length=120)
description: str = Field(default="", max_length=2000)
@@ -43,6 +56,7 @@ class PlanIn(BaseModel):
window_id: uuid.UUID | None = None
auto_start: bool = False
gate_checks: dict[str, bool] = Field(default_factory=dict)
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
compliance_mode: Literal["warn", "fail"] = "warn"
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
@@ -64,6 +78,7 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
return {
**rollout_service.definition(plan),
"gate_probes": list(plan.gate_probes or []),
"description": plan.description,
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
@@ -81,6 +96,15 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
}
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
out: list[dict] = []
for probe in probes:
value = {"type": probe.type, "target": probe.target, "port": probe.port}
if value not in out:
out.append(value)
return out
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
name = payload.name.strip()
if not name:
@@ -99,6 +123,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
plan.gate_probes = _unique_probes(payload.gate_probes)
plan.compliance_mode = payload.compliance_mode
plan.waves = [
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
+194
View File
@@ -0,0 +1,194 @@
"""Rule cleanup (1.31, B4 part 2): switch hand-made rules off, delete them later, restore them (DESIGN-1.31 §4).
POST /hosts/{id}/rule-cleanup/preview operator, admin prepare (writes nothing to a firewall)
POST /hosts/{id}/rule-cleanup/{op}/cancel operator, admin the preview's own actor
POST /hosts/{id}/rule-cleanup/{op}/deploy ADMIN, token scope admin
POST /hosts/{id}/rule-cleanup/{op}/confirm operator, admin BROWSER SESSION ONLY, never a token
POST /hosts/{id}/rule-cleanup/{op}/revert admin; operator only with operators_may_deploy
POST /hosts/{id}/rule-cleanup/{op}/retry ADMIN, token scope admin
POST /hosts/{id}/rule-cleanup/items/{item}/restore ADMIN, token scope admin
GET /hosts/{id}/rule-cleanup, /{op} everybody signed in (read tokens too)
Who may do what is enforced twice: AuthMiddleware (services/api_token_service.py: _ADMIN_ONLY_CLEANUP,
_SESSION_ONLY, _ROLLOUT_POST) and here, so a gap in one layer does not open the route. The audit trail is the
service's (one ``rule_cleanup`` entry per step and rule) plus the middleware's request entry (kind ``host``,
action = last path segment; ``/preview`` is not logged by it, the service writes its own entry).
The services raise ``api_error``; nothing is translated here. MCP stays read-only (no tool, ``api_get`` is GET).
"""
import uuid
from fastapi import APIRouter, Depends, Query, Request
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.models.user import AdminUser
from app.routers import governance
from app.routers.auth import resolve_user
from app.schemas.rule_cleanup import CleanupOperationRead, CleanupPreviewRead, ConfirmName, PreviewRequest
from app.services import rule_cleanup_service as service
from app.services.rule_cleanup_service import ACK_TEXT_VERSION
router = APIRouter(prefix="/hosts/{host_id}/rule-cleanup", tags=["rule-cleanup"])
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
host = await db.get(OpnsenseHost, host_id)
if host is None:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
return host
def _token(request: Request):
return getattr(request.state, "api_token", None)
def _writer(user: AdminUser, request: Request) -> None:
"""Any write: not for a read-only account, not with a read-scope token."""
if user.role == "read_only":
raise api_error(403, ErrorCode.READ_ONLY_ACCESS)
token = _token(request)
if token is not None and token.scope == "read":
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
def _admin_only(user: AdminUser, request: Request) -> None:
"""Deploy, retry, restore: an admin account, and with a token only one of scope admin."""
_writer(user, request)
if user.role != "admin":
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
token = _token(request)
if token is not None and token.scope != "admin":
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
async def _deployer(user: AdminUser, request: Request, db: AsyncSession) -> None:
"""Revert: an admin, or an operator while the governance switch operators_may_deploy is on."""
_writer(user, request)
if user.role == "admin":
return
if user.role == "operator" and (await governance.get_settings(db)).operators_may_deploy:
return
raise api_error(403, ErrorCode.DEPLOY_NEEDS_ADMIN)
def _session_only(user: AdminUser, request: Request) -> None:
"""Confirm: "I still have access" is a human decision in a browser session, never a token's."""
_writer(user, request)
token = _token(request)
if token is not None:
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
def _full_view(request: Request) -> bool:
"""Admins (session or admin-scope token) see the management sources/ports and firewall addresses in a plan."""
from app.routers.audit import full_view
return full_view(request)
async def _names(db: AsyncSession, operations: list[dict], request: Request) -> list[dict]:
"""The firewall names the UI shows next to the ids; for non-admins without the security-relevant plan parts."""
if not _full_view(request):
operations = [service.redact(op) for op in operations]
ids = {uuid.UUID(op["host_id"]) for op in operations} | {uuid.UUID(i["host_id"]) for op in operations for i in op["items"]}
names = {h.id: h.name for h in (await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)))).all()} if ids else {}
for op in operations:
op["host_name"] = names.get(uuid.UUID(op["host_id"]))
for item in op["items"]:
item["host_name"] = names.get(uuid.UUID(item["host_id"]))
return operations
async def _one(db: AsyncSession, operation: dict, request: Request) -> dict:
return (await _names(db, [operation], request))[0]
@router.get("", response_model=list[CleanupOperationRead])
async def list_operations(host_id: uuid.UUID, request: Request, limit: int = Query(50, ge=1, le=100),
db: AsyncSession = Depends(get_db)):
"""The newest operations of this host (and its cluster peers) with their items; unexpired previews included."""
host = await _host(db, host_id)
return await _names(db, await service.list_operations(db, host, limit), request)
@router.post("/preview", response_model=CleanupPreviewRead)
async def preview(host_id: uuid.UUID, payload: PreviewRequest, request: Request, db: AsyncSession = Depends(get_db)):
"""Recompute the plan from the live firewall(s) and store a 5-minute preview bound to the caller."""
user = await resolve_user(request, db)
_writer(user, request)
host = await _host(db, host_id)
kind, uuids, acknowledge = payload.resolved()
if kind is None:
raise api_error(422, ErrorCode.BEREINIGUNG_AUSWAHL_UNGUELTIG, field="kind")
result = await service.preview(
db, host, actor=user.username, actor_id=user.id, kind=kind, uuids=uuids, acknowledge=acknowledge,
**({"unused_days": payload.unused_days} if payload.unused_days else {}),
ack_text_version=payload.ack_text_version or ACK_TEXT_VERSION)
result["operation"] = await _one(db, await service.get_operation(db, host, uuid.UUID(result["preview_id"])), request)
return result if _full_view(request) else service.redact(result)
@router.get("/{op_id}", response_model=CleanupOperationRead)
async def get_operation(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
host = await _host(db, host_id)
return await _one(db, await service.get_operation(db, host, op_id), request)
@router.post("/{op_id}/cancel", response_model=CleanupOperationRead)
async def cancel(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
"""Drop an unused preview (only its own actor can)."""
user = await resolve_user(request, db)
_writer(user, request)
host = await _host(db, host_id)
return await _one(db, await service.cancel(db, host, op_id, actor=user.username), request)
@router.post("/{op_id}/deploy", response_model=CleanupOperationRead)
async def deploy(host_id: uuid.UUID, op_id: uuid.UUID, payload: ConfirmName, request: Request,
db: AsyncSession = Depends(get_db)):
"""Execute a preview. Admin only; the live state must still give the fingerprint of the preview."""
user = await resolve_user(request, db)
_admin_only(user, request)
host = await _host(db, host_id)
return await _one(db, await service.deploy(db, host, op_id, actor=user.username, confirm_name=payload.confirm_name), request)
@router.post("/{op_id}/confirm", response_model=CleanupOperationRead)
async def confirm(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
""""I still have access": re-reads every node first. Browser session only, never with an API token."""
user = await resolve_user(request, db)
_session_only(user, request)
host = await _host(db, host_id)
return await _one(db, await service.confirm(db, host, op_id), request)
@router.post("/{op_id}/revert", response_model=CleanupOperationRead)
async def revert(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
"""Take the change back now (the same code the sweeper runs when the window passes)."""
user = await resolve_user(request, db)
await _deployer(user, request, db)
host = await _host(db, host_id)
return await _one(db, await service.revert_now(db, host, op_id), request)
@router.post("/{op_id}/retry", response_model=CleanupOperationRead)
async def retry(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
"""Revert again after revert_failed, or read the firewall again for a delete/restore with an unclear item."""
user = await resolve_user(request, db)
_admin_only(user, request)
host = await _host(db, host_id)
return await _one(db, await service.retry(db, host, op_id), request)
@router.post("/items/{item_id}/restore", response_model=CleanupOperationRead)
async def restore(host_id: uuid.UUID, item_id: uuid.UUID, payload: ConfirmName, request: Request,
db: AsyncSession = Depends(get_db)):
"""Bring a deleted rule back, switched OFF, at its old position (a new UUID). The result is the restore run;
``warnings`` can hold BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR."""
user = await resolve_user(request, db)
_admin_only(user, request)
host = await _host(db, host_id)
return await _one(db, await service.restore_item(
db, host, item_id, actor=user.username, actor_id=user.id, confirm_name=payload.confirm_name), request)
+7
View File
@@ -52,6 +52,7 @@ def _to_read(
return RuleRead(
workflow_state=state,
workflow_counts=counts, id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
direction=rule.direction,
@@ -108,6 +109,10 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
host = await db.get(OpnsenseHost, payload.host_id)
if not host:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
# Taking over a hand-made rule that a rule cleanup (1.31) is switching off/deleting would make it "managed"
# under the cleanup's feet: not while one is open on this host or its cluster.
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [host])
try:
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
except rule_adopt_service.RuleNotFoundOnHost as exc:
@@ -276,6 +281,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
deployment = next(d for d in rule.deployments if d.host_id == host_id)
from app.services import opnsense_client
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
try:
await opnsense_client.del_rule(deployment.host, deployment.opnsense_rule_uuid)
await opnsense_client.apply(deployment.host)
+2
View File
@@ -207,6 +207,8 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_peer_uuid:
+1
View File
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
source_net: str
+1
View File
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
source_net: str
+1
View File
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
hierarchy_group_id: uuid.UUID
rulebase: Literal["pre", "post"]
sequence: int
+1
View File
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
ipprotocol: str
+1
View File
@@ -48,6 +48,7 @@ class RuleRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
direction: str
+117
View File
@@ -0,0 +1,117 @@
"""Request and response shapes of the rule-cleanup API (routers/rule_cleanup.py, DESIGN-1.31 §4).
Responses mirror services/rule_cleanup_service.read_operation. Texts are not part of the API: ``message`` holds an
ErrorCode name (``errors.<CODE>`` in the UI, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR) or the firewall's
own wording; ``warnings`` is a list of ErrorCode names.
"""
from __future__ import annotations
from typing import Any, Literal
from pydantic import BaseModel, Field, model_validator
CleanupKind = Literal["disable", "enable", "delete"]
# Longest list the API accepts at all; the effective limit is the setting cleanup_max_rules (ceiling 25).
MAX_SELECTION = 100
class PreviewItem(BaseModel):
"""One rule of the selection. ``kind`` (alias ``mode``) must be the same for all items of one preview."""
uuid: str = Field(min_length=1, max_length=64)
kind: CleanupKind | None = None
mode: CleanupKind | None = None
# The per-rule acknowledgement for a "probable" rule (the UI's checkbox).
ack: bool = False
class PreviewRequest(BaseModel):
"""``{kind, uuids, acknowledge}`` or ``{items:[{uuid, kind|mode, ack}]}`` (or both: they are merged)."""
kind: CleanupKind | None = None
uuids: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
items: list[PreviewItem] = Field(default_factory=list, max_length=MAX_SELECTION)
acknowledge: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
unused_days: int | None = Field(default=None, ge=1, le=3650)
ack_text_version: str | None = Field(default=None, max_length=16)
@model_validator(mode="after")
def _single_kind(self):
kinds = {k for k in [self.kind, *[i.kind or i.mode for i in self.items]] if k}
if len(kinds) > 1:
raise ValueError("kind must be the same for every item")
return self
def resolved(self) -> tuple[str | None, list[str], list[str]]:
kind = self.kind or next((i.kind or i.mode for i in self.items if i.kind or i.mode), None)
uuids = [*self.uuids, *[i.uuid for i in self.items]]
acknowledge = [*self.acknowledge, *[i.uuid for i in self.items if i.ack]]
return kind, list(dict.fromkeys(u.strip() for u in uuids)), list(dict.fromkeys(a.strip() for a in acknowledge))
class ConfirmName(BaseModel):
"""Deploy and restore: the exact host name, typed by a human."""
confirm_name: str = Field(max_length=255)
class CleanupItemRead(BaseModel):
id: str
host_id: str
host_name: str | None = None
rule_uuid: str
seq: int
carp_role: str | None = None
# queued, disabling, disabled, confirmed, reverting, reverted, deleting, deleted, restoring, restored, failed,
# unknown, revert_failed, skipped (never attempted: the run stopped earlier)
state: str
description: str | None = None
categories: Any = None
confidence: str | None = None
cover_uuid: str | None = None
prio_group: str | None = None
anchor_next_uuid: str | None = None
anchor_prev_uuid: str | None = None
source_item_id: str | None = None
restored_uuid: str | None = None
changed_at: str | None = None
message: str | None = None
has_snapshot: bool = False
# A deleted rule with a stored snapshot that has not been restored yet (POST items/{id}/restore, admin).
restorable: bool = False
class CleanupOperationRead(BaseModel):
id: str
host_id: str
host_name: str | None = None
cluster_name: str | None = None
kind: str # disable | enable | delete | restore
# preview, running, awaiting_confirm, confirmed, partial, apply_failed, reverting, reverted, revert_failed, done,
# failed, cancelled, expired
status: str
actor: str
created_at: str | None = None
expires_at: str | None = None
started_at: str | None = None
heartbeat_at: str | None = None
revert_deadline: str | None = None
confirmed_at: str | None = None
finished_at: str | None = None
message: str | None = None
options: dict[str, Any] = Field(default_factory=dict)
# Only while status == "preview".
plan: dict[str, Any] | None = None
items: list[CleanupItemRead] = Field(default_factory=list)
can_confirm: bool = False
can_revert: bool = False
can_retry: bool = False
# ErrorCode names, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR (restore only).
warnings: list[str] = Field(default_factory=list)
class CleanupPreviewRead(BaseModel):
preview_id: str
expires_at: str | None = None
kind: str
plan: dict[str, Any]
max_rules: int
window_seconds: int
operation: CleanupOperationRead
@@ -130,6 +130,9 @@ async def sweep_expired_alias_deployments(db: AsyncSession) -> list[AliasDeploym
async def undeploy_alias(db: AsyncSession, alias: FirewallAlias) -> None:
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)
failures = []
result = await db.execute(select(AliasDeployment).where(AliasDeployment.alias_id == alias.id))
for deployment in result.scalars().all():
+22 -6
View File
@@ -39,12 +39,15 @@ _LAST_USED_GRANULARITY = timedelta(minutes=1)
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$")
# Rule cleanup (1.31): "I still have access" is a human decision, so confirming a cleanup is never possible with a
# token, whatever its scope (a script must not be able to confirm blindly).
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$")
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
# notification settings the webhook URL/SMTP data): admin scope only.
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
_MCP_PATH = "/api/mcp"
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
# it compares a posted file with the database and writes nothing.
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
_ADMIN_ONLY_WRITES = re.compile(
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
r"|^/api/logs/settings$"
@@ -52,9 +55,18 @@ _ADMIN_ONLY_WRITES = re.compile(
r"|^/api/hosts/[^/]+/gui-ticket$"
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
r"|^/api/config/import/apply$"
)
# Rule cleanup: executing a cleanup, retrying a failed revert/reconcile and restoring a deleted rule are admin-only.
# operators_may_deploy defaults to True, so _ROLLOUT_POST alone cannot express "admin only"; this pattern is checked
# for operator accounts (operator_denies) and for tokens below scope admin (denies). Not part of _ADMIN_ONLY_WRITES:
# that one means "who may access OPNexus / where data goes". The router checks the same rule once more.
_ADMIN_ONLY_CLEANUP = re.compile(r"^/api/hosts/[^/]+/rule-cleanup/([^/]+/(deploy|retry)|items/[^/]+/restore)/?$")
def _now() -> datetime:
return datetime.now(timezone.utc)
@@ -67,13 +79,13 @@ def denies(scope: str, method: str, path: str) -> bool:
"""True if a token with this scope must not make this request."""
if _SESSION_ONLY.match(path):
return True
if path == _MCP_PATH:
if path in _READ_ONLY_POSTS:
return False
if method in _SAFE_METHODS:
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
if scope == "read":
return True
if scope == "deploy" and _ADMIN_ONLY_WRITES.match(path):
if scope == "deploy" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):
return True
return False
@@ -82,7 +94,8 @@ def operator_denies(method: str, path: str) -> bool:
"""True if an `operator` account must not make this request: the writes that
change who may access OPNexus or where it sends data. (Their own theme at
PUT /api/settings stays open; account endpoints are not matched here.)"""
return method not in _SAFE_METHODS and path != "/api/settings" and bool(_ADMIN_ONLY_WRITES.match(path))
return method not in _SAFE_METHODS and path != "/api/settings" and bool(
_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
@@ -92,6 +105,9 @@ def operator_denies(method: str, path: str) -> bool:
_ROLLOUT_POST = re.compile(
r"/(deploy|push|sign|poll|register|reassert-ordering"
r"|deployments/[^/]+/(remove|retry-rollback)"
# Rule cleanup (1.31): switching rules off (/deploy matches above) and taking it back. Confirming and
# previewing stay open, like confirm_rule.
r"|rule-cleanup/[^/]+/(revert|retry)"
r"|maintenance/(execute|carp-execute|schedule)"
r"|network/changes"
r"|services/[^/]+/(start|stop|restart)"
+3
View File
@@ -13,6 +13,9 @@ from app.database import SessionLocal
from app.models.audit import AuditEvent
context = ContextVar("audit_context", default=None)
# A key in ``details`` whose content only admins (session or admin-scope token) get to read (routers/audit.py):
# firewall addresses and management sources of the rule cleanup.
ADMIN_ONLY_DETAILS = "admin_only"
# Column widths of audit_events. entity_id/host_id/action are taken from the
@@ -34,7 +34,27 @@ async def peers(db, host):
return [host]
async def ensure_no_maintenance(db, hosts):
async def ensure_no_open_cleanup(db, hosts):
"""A rule cleanup (1.31) that is running, waiting for its confirmation or unresolved owns the host and its
cluster peers: nothing else may write to the firewall until it is confirmed, reverted or settled."""
from app.services.rule_cleanup_service import has_open_cleanup
ids = set()
for host in hosts:
ids.update(h.id for h in await peers(db, host))
if ids and await has_open_cleanup(db, ids):
raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)
async def ensure_no_open_cleanup_on(db, model, condition):
"""ensure_no_open_cleanup for every host that has a deployment row of ``model`` matching ``condition`` (the
undeploy paths: deleting a central object removes it from all its firewalls)."""
ids = set((await db.scalars(select(model.host_id).where(condition))).all())
hosts = [h for h in [await db.get(OpnsenseHost, i) for i in ids] if h is not None]
await ensure_no_open_cleanup(db, hosts)
async def ensure_no_maintenance(db, hosts, *, cleanup=True):
"""cleanup=False: only the cleanup service itself, which reports an open cleanup with its own error code."""
ids = set()
for host in hosts:
ids.update(h.id for h in await peers(db, host))
@@ -42,6 +62,8 @@ async def ensure_no_maintenance(db, hosts):
raise api_error(409, ErrorCode.OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY)
if ids and await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
raise api_error(409, ErrorCode.OPEN_RESTORE_BLOCKS_DEPLOY)
if cleanup and ids:
await ensure_no_open_cleanup(db, hosts)
async def _shared_change_blockers(db, fleet):
@@ -56,6 +78,9 @@ async def _shared_change_blockers(db, fleet):
blocked.append("Eine frühere Wartungsaktion im Host/Cluster muss zuerst geprüft und abgeschlossen werden.")
if await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(fleet_ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
blocked.append("Eine Wiederherstellung im Host/Cluster ist offen und muss zuerst bestätigt oder abgebrochen werden.")
from app.services.rule_cleanup_service import has_open_cleanup
if await has_open_cleanup(db, fleet_ids):
blocked.append("Eine Regel-Bereinigung im Host/Cluster ist offen und muss zuerst bestätigt oder zurückgenommen werden.")
return blocked
@@ -303,6 +303,9 @@ async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
for this entity to gate on) — deletes Child/Local/Remote before their
parent Connection (the ModelRelationField dependency direction), then
the PreSharedKey (independent of the other 4), then reconfigures."""
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, IpsecTunnelDeployment, IpsecTunnelDeployment.tunnel_id == tunnel.id)
failures = []
result = await db.execute(select(IpsecTunnelDeployment).where(IpsecTunnelDeployment.tunnel_id == tunnel.id))
for deployment in result.scalars().all():
@@ -201,6 +201,62 @@ async def check_revert_failed(db: AsyncSession) -> dict[str, dict]:
"host_name": host.name if host else None,
"message": deployment.message,
}
problems.update(await _cleanup_problems(db))
return problems
# Rule cleanup (1.31) states that need a human: the automatic revert failed, or a run ended half-way.
_CLEANUP_STILL_CHANGED = ("revert_failed", "unknown", "disabling", "disabled", "failed")
async def _cleanup_problems(db: AsyncSession) -> dict[str, dict]:
"""Rule cleanups whose automatic revert failed (per rule that could not be switched back) and runs stuck in
``partial``/``apply_failed`` (a decision is needed before the window ends), plus delete/restore runs with an
unresolved item. Same ``revert_failed`` alert as the deployments. Keys: ``rule_cleanup:<item id>`` for one rule,
``rule_cleanup_op:<operation id>`` for a whole run, so one open problem is one message."""
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
problems: dict[str, dict] = {}
operations = (await db.scalars(select(RuleCleanupOperation).where(
RuleCleanupOperation.status.in_(("revert_failed", "partial", "apply_failed", "failed"))))).all()
for op in operations:
items = list((await db.scalars(select(RuleCleanupItem).where(RuleCleanupItem.operation_id == op.id)
.order_by(RuleCleanupItem.seq))).all())
host = await db.get(OpnsenseHost, op.host_id)
base = {
"deployment_type": "rule_cleanup", "deployment_id": str(op.id), "host_id": str(op.host_id),
"host_name": host.name if host else None, "operation": op.kind, "operation_status": op.status,
}
if op.status == "revert_failed":
stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]
for item in stuck:
member = await db.get(OpnsenseHost, item.host_id)
problems[f"rule_cleanup:{item.id}"] = {
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
"message": item.message or op.message,
"hinweis": "Bereinigung: Die Regel konnte nicht automatisch zurückgesetzt werden und ist möglicherweise "
"noch geändert. Auf der Firewall prüfen und in OPNexus „Erneut zurücknehmen“ wählen.",
}
if not stuck:
problems[f"rule_cleanup_op:{op.id}"] = {**base, "entity_id": str(op.id), "message": op.message,
"hinweis": "Bereinigung: Das automatische Zurücknehmen ist fehlgeschlagen. Zustand der Firewall prüfen."}
elif op.status in ("partial", "apply_failed"):
problems[f"rule_cleanup_op:{op.id}"] = {
**base, "entity_id": str(op.id), "message": op.message,
"regeln": ", ".join(i.rule_uuid for i in items if i.state in ("failed", "unknown")) or None,
"hinweis": "Bereinigung nur teilweise ausgeführt (oder das Anwenden ist fehlgeschlagen). Bestätigen ist gesperrt; "
"jetzt zurücknehmen oder die automatische Rücknahme am Fristende abwarten.",
}
elif op.kind in ("delete", "restore"):
for item in (i for i in items if i.state == "unknown"):
member = await db.get(OpnsenseHost, item.host_id)
problems[f"rule_cleanup:{item.id}"] = {
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
"message": item.message or op.message,
"hinweis": "Bereinigung: Der Ausgang beim Löschen/Wiederherstellen ist unklar. Mit „Erneut prüfen“ nachlesen.",
}
return problems
@@ -135,6 +135,9 @@ async def sweep_expired_one_to_one_deployments(db: AsyncSession) -> list[OneToOn
async def undeploy_one_to_one(db: AsyncSession, rule: OneToOneRule) -> None:
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, OneToOneDeployment, OneToOneDeployment.one_to_one_id == rule.id)
failures = []
result = await db.execute(select(OneToOneDeployment).where(OneToOneDeployment.one_to_one_id == rule.id))
for deployment in result.scalars().all():
@@ -371,6 +371,9 @@ async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None
for this entity to gate on) — deletes the Instance before its
referenced StaticKey (the ModelRelationField dependency direction),
then reconfigures."""
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, OpenVpnInstanceDeployment, OpenVpnInstanceDeployment.instance_id == instance.id)
failures = []
result = await db.execute(select(OpenVpnInstanceDeployment).where(OpenVpnInstanceDeployment.instance_id == instance.id))
for deployment in result.scalars().all():
+137 -4
View File
@@ -10,6 +10,7 @@ Auth is HTTP Basic: username = API key, password = API secret.
"""
import logging
import json
import re
from typing import Any
import httpx
@@ -33,6 +34,11 @@ class OpnsenseAmbiguousError(OpnsenseAPIError):
outcome of a write is unknown and must be reconciled, not retried blindly."""
class OpnsenseNotFoundError(OpnsenseAPIError):
"""The firewall answered, but there is no such object (an empty `getRule` answer: unknown UUID, or a rule that is
not part of the rules model - legacy or automatic). A refusal, never an unknown outcome."""
class OpnsenseForbiddenError(OpnsenseAPIError):
"""The firewall answered 403: the API key lacks the privilege for this endpoint. A refusal, not an unknown
outcome; callers that only read can tell "no rights" from "unreachable". 401 (wrong key or secret) stays a
@@ -241,6 +247,51 @@ def _mutation_result(data: Any, operation: str, results: set[str]) -> dict:
return data
# -- Filter rules (FilterController -> ApiMutableModelControllerBase, rules.rule) --------------------------------
# Evidence for add/set/get/del/toggle/moveRuleBefore, checked 2026-10-22 against BOTH
# source opnsense/core tags 26.7.7, 26.1.9 and 25.7.11, src/opnsense/mvc/app/controllers/OPNsense/:
# Firewall/Api/FilterController.php (the *RuleAction methods, moveRuleBeforeAction, ittrRules),
# Firewall/Api/FilterBaseController.php (applyAction),
# Base/ApiMutableModelControllerBase.php (addBase, setBase, getBase, delBase, toggleBase)
# live OPNsense 26.7.7 on two lab test firewalls (scratch rules whose description began with
# `opnexus-b4-probe`, deleted again and applied afterwards; nothing was left behind).
# Same in all three source lines unless said otherwise; the live checks cover 26.7.7 only.
#
# addRule POST, body {"rule": {...}}. Answer {"result": "saved", "uuid": "<new>"}; a validation failure is HTTP 200
# {"result": "failed", "validations": {"rule.<field>": "<text>"}} (so _mutation_result raises).
# `sequence` is a FilterSequenceField (models/OPNsense/Firewall/Filter.xml: 1..999999, default 1;
# `sort_order` and `prio_group` are volatile, derived). Live: without `sequence` the rule gets the
# highest sequence of ALL rules + 100 (566, 666, 766, ...), so a copy always lands at the END of its
# prio_group; an explicit `sequence` IS honoured (sequence "3" -> sort_order 400000.0000003) and is not
# checked for uniqueness. A restore can therefore either send the old number or call moveRuleBefore.
# setRule POST /{uuid}, body {"rule": {...}}; answer {"result": "saved"}. It is an upsert (a well-formed unknown
# UUID creates the rule) - never call it for a rule that may be gone.
# delRule POST /{uuid}. {"result": "deleted"} or {"result": "not found"} (unknown, legacy and automatic UUIDs all
# answer "not found", HTTP 200, nothing changes). 26.1 and 26.7 accept a comma-separated list; 25.7 a
# single UUID. Consequence: "not found" proves nothing about a rule being gone - re-read it.
# getRule GET /{uuid} -> {"rule": {...}} (every field of the rules model, select fields as
# {"<option>": {"value": "<label>", "selected": 0|1}}, `categories` a select dict - or [] when the rule has
# none -; no `uuid` key; read-only: `sequence`, `sort_order`, `prio_group` and the nested `audit`).
# Live: an unknown, legacy (hex label) or automatic UUID answers HTTP 200 with a bare `[]`. 25.7 and 26.1
# also know `?fetchmode=copy` (pre-fills `sequence`); 26.7.7 no longer has it - not used here.
# Without a UUID it returns the template of a new rule (see get_rule_template above).
# toggleRule POST /{uuid}/{enabled} - `enabled` is "0" or "1" and EXPLICIT; without it the rule flips (never do that).
# Answer {"result": "Enabled"|"Disabled", "changed": true|false}. The explicit form is idempotent: repeating
# it answers the same result with "changed": false (live: 0,0 -> true,false; 1,1 -> true,false). Any other
# value answers {"result": "failed", "changed": false}; an unknown or legacy UUID, a GET, or a model that
# has no such node answers {"result": "failed"} without "changed" - all HTTP 200. It only edits the
# configuration; nothing reaches pf before apply() (the rule keeps its sort_order, searchRule shows
# enabled "0"/"1" at once). 26.1/26.7 take a list of UUIDs (explicit value only); 25.7 one UUID.
# moveRuleBefore POST /{selected}/{target}. {"status": "ok"} (no "result" key). Both rules must be in the same
# `prio_group` - the sort_order prefix - which is NOT the interface: every single-interface rule of every
# interface shares 400000 (lan and wan rules can be moved before each other), floating rules (several or
# no interface) are 200000, interface-group rules 3000xx. Live: unknown UUID, same UUID twice and
# different groups all answer HTTP 500 {"errorMessage": "Unexpected error, check log for details",
# "errorTitle": "Filter"}, a GET answers 200 {"status": "error"}. A 500 is Ambiguous in _do_request, so a
# caller that wants to tell "refused" from "unknown" must read sort_order back. The move only touches
# `sequence` of the moved rule (and, when the gap is too small, of the rules behind the target); legacy
# rules are not in the model and cannot be an anchor.
# apply POST. {"status": "OK\n\n"} after `filter reload skip_alias`. FilterBaseController has no savepoint here.
async def add_rule(host: OpnsenseHost, payload: dict) -> dict:
return _mutation_result(await _request(host, "POST", "/firewall/filter/addRule", json={"rule": payload}), "Regel anlegen", {"saved"})
@@ -253,18 +304,93 @@ async def del_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
return _mutation_result(await _request(host, "POST", f"/firewall/filter/delRule/{rule_uuid}"), "Regel entfernen", {"deleted", "not found"})
_RULE_REF = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
def _rule_ref(rule_uuid: str) -> str:
"""A rule reference ends up in the URL path: refuse anything that could leave it."""
if not isinstance(rule_uuid, str) or not _RULE_REF.match(rule_uuid):
raise OpnsenseAPIError("Ungültige Regel-UUID")
return rule_uuid
async def get_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
"""The full model record of one rule (`getRule/{uuid}`, the inner `rule` dict, select fields still in
OPNsense's option form - reduce them with rule_write_payload()). A rule that is not in the rules model (unknown,
legacy hex label, automatic) is answered with `[]`: raises OpnsenseNotFoundError."""
data = await _request(host, "GET", f"/firewall/filter/getRule/{_rule_ref(rule_uuid)}")
rule = data.get("rule") if isinstance(data, dict) else None
if isinstance(rule, dict) and rule:
return rule
if data == [] or data == {}:
raise OpnsenseNotFoundError(f"Regel {rule_uuid} gibt es auf {host.name} nicht (oder sie ist keine bearbeitbare Regel)")
raise OpnsenseAPIError(f"Regel lesen: Keine auswertbare Antwort von {host.name}")
async def toggle_rule(host: OpnsenseHost, rule_uuid: str, enabled: bool) -> dict:
"""Sets the rule's `enabled` flag to exactly `enabled` (`toggleRule/{uuid}/1|0`; never the flipping form) and
returns OPNsense's answer, e.g. {"result": "Disabled", "changed": False}. Repeating it is harmless. Only the
configuration changes; call apply() afterwards. A refusal (unknown or legacy UUID) is a plain OpnsenseAPIError,
a lost answer an OpnsenseAmbiguousError - then read the rule back before doing anything else."""
if not isinstance(enabled, bool):
raise TypeError("toggle_rule: 'enabled' must be a bool (an implicit flip would not be idempotent)")
data = await _request(host, "POST", f"/firewall/filter/toggleRule/{_rule_ref(rule_uuid)}/{1 if enabled else 0}")
return _mutation_result(data, "Regel ein-/ausschalten", {"enabled" if enabled else "disabled"})
# Fields of a getRule record that a write must not send: the UUID is the rule's identity, `sequence` is assigned
# on creation (sort_order and prio_group are derived from it), `audit` is metadata OPNsense maintains itself.
RULE_READONLY_FIELDS = frozenset({"uuid", "sequence", "sort_order", "prio_group", "audit"})
def rule_write_payload(rule: dict) -> dict[str, str]:
"""Turns a getRule record into the flat form addRule/setRule accept: select fields become their selected
option(s), comma-separated (`lan,wan`; none selected -> ""), the empty-list form of an empty multi-select
becomes "", read-only fields are left out. Live round trip on 26.7.7 (getRule -> this -> addRule -> getRule)
gave an identical record except uuid/sequence/sort_order/audit for single-interface, multi-interface (floating),
no-interface (floating), interface-group, ICMP and any-protocol rules with categories, aliases, port ranges,
TCP flags, tags and state limits."""
out: dict[str, str] = {}
for key, value in rule.items():
if key in RULE_READONLY_FIELDS:
continue
if isinstance(value, dict):
if value and all(isinstance(o, dict) and "selected" in o for o in value.values()):
out[key] = ",".join(k for k, o in value.items() if o.get("selected") and k != "")
# any other nested dict is metadata and is not sent
elif isinstance(value, list):
out[key] = ",".join(str(v) for v in value)
else:
out[key] = "" if value is None else str(value)
return out
async def apply(host: OpnsenseHost) -> dict:
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
async def get_hasync(host: OpnsenseHost) -> dict:
"""System > High Availability settings, read-only: `core/hasync/get` (Core/Api/HasyncController extends
ApiMutableModelControllerBase, model Core/Hasync.xml; identical in opnsense/core 25.7.11, 26.1.9 and 26.7.7).
Answer {"hasync": {"pfsyncinterface": {"": {"value": "Disabled", ...}, "lan": {"value": "LAN", "selected": 1},
...}, "pfsyncpeerip": "<address or empty>", "synchronizetoip": ..., "syncitems": {...}}} - `pfsyncinterface` is an
InterfaceField (the selected option, "" = disabled), `pfsyncpeerip` a NetworkField without mask. Live on 26.7.7
(both nodes of the lab cluster): pfsyncinterface `lan`, a peer address, synchronizetoip empty. Used by the rule
cleanup to keep pfsync rules out of reach (rule_analysis/protection.py); it reads nothing secret (the sync
password field comes back, but is never stored or logged by the caller)."""
return await _request(host, "GET", "/core/hasync/get")
async def move_rule_before(host: OpnsenseHost, selected_uuid: str, target_uuid: str) -> dict:
"""`firewall/filter/moveRuleBefore/{selected_uuid}/{target_uuid}`
(FilterController::moveRuleBeforeAction) — repositions `selected_uuid`
to sit directly before `target_uuid` via gap-numbered `sequence`
floats, scoped to rules sharing the same `prio_group` (interface/type).
Verified against OPNsense core source: no "move to top/bottom"
sentinel exists, and both UUIDs must resolve to existing rules on the
SAME interface/type or the controller throws. On success it returns
floats, scoped to rules sharing the same `prio_group` (the sort_order
prefix, see the evidence block above - not the interface). Verified against OPNsense core source and live
(26.7.7): no "move to top/bottom"
sentinel exists, and both UUIDs must resolve to existing rules in the
SAME prio_group or the controller throws (HTTP 500, which this client
reports as OpnsenseAmbiguousError). On success it returns
`{"status": "ok"}` (confirmed against FilterController.php directly —
unlike addRule/setRule/delRule this has no "result" key at all), so
this is dispatched through _mutation_result with an empty `results`
@@ -1328,6 +1454,13 @@ async def create_snapshot(host: OpnsenseHost, name: str, note: str = "") -> None
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
async def activate_snapshot(host: OpnsenseHost, uuid: str) -> None:
"""Boot environment for every following start (bectl activate, not -t); takes effect at the next reboot."""
data = await _request(host, "POST", f"/core/snapshots/activate/{uuid}")
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
raise OpnsenseAPIError(f"Snapshot aktivieren von Firewall nicht bestätigt: {redact_data(data)}")
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
@@ -147,6 +147,9 @@ async def sweep_expired_outbound_nat_deployments(db: AsyncSession) -> list[Outbo
async def undeploy_outbound_nat(db: AsyncSession, rule: OutboundNatRule) -> None:
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, OutboundNatDeployment, OutboundNatDeployment.outbound_nat_id == rule.id)
failures = []
result = await db.execute(select(OutboundNatDeployment).where(OutboundNatDeployment.outbound_nat_id == rule.id))
for deployment in result.scalars().all():
@@ -185,6 +185,9 @@ async def reassert_group_ordering(db: AsyncSession, group: PolicyHierarchyGroup)
for h in g.hosts:
hosts[h.id] = h
errors: list[str] = []
# moveRuleBefore changes rule positions: not while a rule cleanup (1.31) owns one of the firewalls.
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [h for h in hosts.values() if h.enabled])
for host in sorted(hosts.values(), key=lambda h: (h.name, str(h.id))):
if not host.enabled:
continue
@@ -300,9 +303,13 @@ async def sweep_expired_hierarchy_deployments(db: AsyncSession) -> list[PolicyHi
async def undeploy_hierarchy_rule(db: AsyncSession, rule: PolicyHierarchyRule) -> None:
from app.services.host_maintenance_service import ensure_no_open_cleanup
failures = []
result = await db.execute(select(PolicyHierarchyRuleDeployment).where(PolicyHierarchyRuleDeployment.policy_hierarchy_id == rule.id))
for deployment in result.scalars().all():
deployments = list(result.scalars().all())
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them.
await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])
for deployment in deployments:
host = await db.get(OpnsenseHost, deployment.host_id)
if host and deployment.opnsense_policy_hierarchy_uuid:
try:
@@ -147,6 +147,9 @@ async def sweep_expired_port_forward_deployments(db: AsyncSession) -> list[PortF
async def undeploy_port_forward(db: AsyncSession, rule: PortForwardRule) -> None:
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, PortForwardDeployment, PortForwardDeployment.port_forward_id == rule.id)
failures = []
result = await db.execute(select(PortForwardDeployment).where(PortForwardDeployment.port_forward_id == rule.id))
for deployment in result.scalars().all():
+7 -3
View File
@@ -196,6 +196,7 @@ async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, sta
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
"window_id": str(plan.window_id) if plan.window_id else None,
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
"gate_probes": list(plan.gate_probes or []),
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
}
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
@@ -381,7 +382,8 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
baselines.update((w.baseline or {}))
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
checks=run.plan_snapshot.get("gate_checks"),
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
probes=run.plan_snapshot.get("gate_probes"))
async def apply(db, run):
if gate["verdict"] == "fail":
@@ -537,7 +539,7 @@ async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | Non
versions = {}
for rh_id in offers:
host = usable[rh_id]
baselines[str(host.id)] = await rollout_health.baseline(db, host)
baselines[str(host.id)] = await rollout_health.baseline(db, host, run.plan_snapshot.get("gate_probes"))
versions[rh_id] = baselines[str(host.id)].get("version")
roles = await get_statuses([usable[i] for i in offers])
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
@@ -630,6 +632,7 @@ async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision |
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
gate = await rollout_health.evaluate(
db, done_peers + [host], baselines, target_version=run.target_version,
# No probes here: the peers' probes were checked in their own gate, and this runs on every tick.
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
@@ -718,7 +721,8 @@ async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
checks=run.plan_snapshot.get("gate_checks"),
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
probes=run.plan_snapshot.get("gate_probes"))
async def apply(db, run):
w = wave_row(run, index)
+63 -2
View File
@@ -18,13 +18,19 @@ the configuration like the 6-hourly security check does):
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
update may legitimately change defaults.
* probes — optional per plan (1.32): ping/TCP probes run FROM the firewall (uplink, DNS, an internal
server). A probe that worked in the baseline and fails now fails the gate; one that failed
already before only warns. This is reachability from the firewall, not traffic through it.
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
"""
import asyncio
import uuid
from collections import defaultdict
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -41,6 +47,8 @@ DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": Tru
"compliance": False}
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
MAX_PROBES = 10
PROBE_PING_COUNT = 3
def _r(result: str, code: str, **params) -> dict:
@@ -64,7 +72,55 @@ async def _services(host) -> dict | None:
return None if listing is None else _services_compact(listing)
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
def _probe_key(probe: dict) -> str:
return f"{probe['type']} {probe['target']}" + (f":{probe['port']}" if probe.get("port") else "")
async def run_probes(host: OpnsenseHost, probes: list[dict] | None) -> list[dict]:
"""Every probe from the firewall, one after the other; a probe that cannot run counts as failed. Audited by
diagnostics_service like a probe from the host page, as `system:rollout` when no request context exists."""
from app.services import audit_service, diagnostics_service
if not probes:
return []
token = None
if audit_service.context.get() is None:
token = audit_service.context.set({"actor": "system:rollout", "id": str(uuid.uuid4()), "action": "probe"})
try:
async def one(probe: dict) -> dict:
ipproto = "inet6" if ":" in probe["target"] else "inet"
try:
if probe["type"] == "ping":
result = await diagnostics_service.ping(host, probe["target"], ipproto, PROBE_PING_COUNT, None, None)
ok = int(result.get("received") or 0) > 0
else:
result = await diagnostics_service.portprobe(host, probe["target"], int(probe["port"]), ipproto, None)
ok = bool(result.get("open"))
except (HTTPException, client.OpnsenseAPIError):
ok = False
return {"key": _probe_key(probe), "ok": ok}
# Concurrently (diagnostics_service caps it at a few at a time per process), results in plan order.
return list(await asyncio.gather(*(one(p) for p in probes[:MAX_PROBES])))
finally:
if token is not None:
audit_service.context.reset(token)
def _check_probes(before: list | None, now: list | None) -> dict:
if now is None:
return _r("fail", "unreadable")
was = {p["key"]: p["ok"] for p in (before or [])}
regressed = [p["key"] for p in now if not p["ok"] and was.get(p["key"], True)]
still = [p["key"] for p in now if not p["ok"] and was.get(p["key"]) is False]
if regressed:
return _r("fail", "probe_regressed", probes=", ".join(regressed))
if still:
return _r("warn", "probe_was_failing", probes=", ".join(still))
return _r("ok", "ok", count=len(now))
async def baseline(db: AsyncSession, host: OpnsenseHost, probes: list[dict] | None = None) -> dict:
info, gateways, services = await asyncio.gather(
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
@@ -73,6 +129,7 @@ async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
"gateways": gateways,
"services": services,
"compliance_score": score,
"probes": await run_probes(host, probes) if probes else None,
}
@@ -137,7 +194,7 @@ async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dic
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
checks: dict | None = None, compliance_mode: str = "warn", probes: list[dict] | None = None) -> dict:
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
enabled = {**DEFAULT_CHECKS, **(checks or {})}
for name in ALWAYS_ON:
@@ -190,6 +247,10 @@ async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict,
if enabled["services"]:
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
if probes:
# A probe new since the baseline (plan edited mid-run) has nothing to compare with: treated as "worked before".
row["probes"] = _check_probes(before.get("probes"), await run_probes(host, probes) if reachable else None)
blockers = await _shared_change_blockers(db, fleets[hid])
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
+2
View File
@@ -110,6 +110,8 @@ def definition(plan: RolloutPlan) -> dict:
return {
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
# Only when set: an activated plan from before 1.32 must keep its fingerprint (else its automatic start stops).
**({"gate_probes": list(plan.gate_probes)} if plan.gate_probes else {}),
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
}
@@ -15,7 +15,9 @@ Confidence (never higher than what is proven; what is not proven is never "sure"
sampled recently and over the whole window; shadowed/duplicate = literal comparison, order from
``sort_order``, the covering rule is loaded; disabled_long = observed unchanged for N days.
* ``probable`` relies on a guess or on static alias content, or pf's loaded ruleset is not available, or the rule
has a schedule (a hit might happen at another time).
has a schedule (a hit might happen at another time), or (cluster) some node has no usage data for
the rule. In a cluster "unused" is judged over all nodes (rule_hit_service.merge_cluster_hits):
the CARP backup sees almost no traffic, so its zero alone never makes a rule unused.
* ``unknown`` there is a hint but no support: the rule is not loaded in pf (no counters can exist), or the
counters are stale.
@@ -57,6 +59,8 @@ def unused_confidence(rule: NormRule, hit, now: datetime) -> str:
return "unknown" # stale counters say nothing about the present
if rule.loaded is None or rule.sched:
return "probable" # pf's ruleset not available / the rule only works at certain times
if getattr(hit, "peer_missing", False):
return "probable" # cluster: some node has no usage data for this rule (rule_hit_service.ClusterHit)
return "sure"
@@ -78,7 +82,10 @@ def build(rules: list[NormRule], ctx: Context, *, hits: dict, states: dict, mana
key = rule.uuid.lower()
hit = hits.get(key)
if hit is not None and rule_hit_service.classify(hit, now, unused_days) == "unused":
categories.append({"kind": "unused", "confidence": unused_confidence(rule, hit, now), "days": unused_days})
category = {"kind": "unused", "confidence": unused_confidence(rule, hit, now), "days": unused_days}
if getattr(hit, "peer_missing", False):
category["notes"] = ["CLUSTER_PEER_USAGE_UNKNOWN"]
categories.append(category)
overlap = overlaps.get(id(rule))
if overlap is not None:
categories.append({
@@ -117,6 +117,31 @@ def rule_fields(r: NormRule, q: Query, direction: str, iface: str, ctx: Context,
return fields
def match_rule(r: NormRule, q: Query, ctx: Context, *, iface: str | None = None,
source: Outcome | None = None, overrides: dict[str, Outcome] | None = None) -> Outcome:
"""Could the rule's own fields match the packet ``q`` arriving on ``iface``? Order, ``quick``, other rules and
whether the rule is enabled or loaded play no part: this is the question "is this rule about such a packet at
all", which the cleanup asks about the way the admin reaches the firewall (DESIGN-1.31 §3.2).
``iface`` defaults to ``q.iface`` and then to the interface guessed from the address; an interface that cannot
be resolved is UNKNOWN. ``source`` replaces the source field (the caller judged it for a whole network, see
rule_analysis/protection.py); ``overrides`` replaces further fields by name (e.g. ``interface`` for a packet
arriving on a device that is not an assigned interface). Tags set by earlier rules are not known here, so ``tagged`` is UNKNOWN. Only
``False`` is a proof that the rule does not match; ``None`` (unknown) must be treated like ``True``."""
iface = iface or q.iface or guess_iface(q.src if q.direction == "in" else q.dst, ctx)
if iface is None:
return Outcome(None, ["IFACE_UNRESOLVED"])
fields = rule_fields(r, q, q.direction, iface, ctx, set(), None)
if source is not None:
fields["source"] = source
for name, outcome in (overrides or {}).items():
if name in fields or name == "interface":
fields[name] = outcome
value = t_and(*(f.value for f in fields.values()))
reasons = sorted({reason for f in fields.values() if f.value is None for reason in f.reasons}) if value is None else []
return Outcome(value, reasons)
def _step(r: NormRule, fields: dict[str, Outcome], result, effect: str) -> dict:
return {
"rule": r.summary(), "result": tri_str(result), "effect": effect,
@@ -0,0 +1,256 @@
"""May a rule be switched off without touching the way the admin reaches the firewall? (DESIGN-1.31 §3.2/§3.3)
Pure functions on ``NormRule`` and the analysis ``Context``; no I/O.
Management access (§3.2). For every configured management source (IP or network), every address of the
firewall (the address in ``host.base_url`` plus every interface address and VIP) and every management port
(the API port of ``base_url``, 443, 80, 22) and every interface the packet could arrive on, the rule's own
fields are compared with that packet (``engine.match_rule``). A rule is safe only when EVERY such comparison is
``False``. ``True`` or unknown means the rule is about management traffic: it is protected.
Why that is enough without simulating "the ruleset afterwards": a rule that does not match a packet takes no part
in its evaluation at all - it neither decides it nor tags it (pf only tags on a match) - so removing it cannot
change what happens to that packet. This holds whatever the verdict for the packet is, even when the analysis
of the whole ruleset is only "probable". The packet's ingress interface is a routing question OPNexus cannot prove
(asymmetric routes, VPNs, failover), so every interface is tried: that only ever protects more.
A management source that is a network is judged for all of its addresses at once: up to 256 addresses are
enumerated (exact, with every resolver rule of the analysis), larger networks are compared as address ranges
with the rule's literal/alias/interface addresses; anything that cannot be resolved is unknown = protected.
Infrastructure (§3.3, reason ``infrastructure``): CARP (IP protocol 112) and pfsync (240) rules, every rule that
could match CARP advertisements (to 224.0.0.18 / ff02::12 from the interface's own networks) on an interface with
a CARP VIP, and every rule that could match pfsync traffic on the pfsync interface (System > High Availability,
``core/hasync/get``; when that cannot be read on a cluster node, on every interface). The second leash of the
design ("(self) with port 22/80/443/API" when §3.2 is not False) is part of §3.2 already: such a rule is
protected there.
"""
from __future__ import annotations
import ipaddress
from urllib.parse import urlsplit
from app.services.rule_analysis import coverage, resolve
from app.services.rule_analysis.context import BUILTIN_GROUP_DEVICES, Context
from app.services.rule_analysis.engine import Query, match_rule
from app.services.rule_analysis.model import Endpoint, IPAddr, IPNet, NormRule, Outcome, o_or, t_not
MANAGEMENT_PORTS = (443, 80, 22)
# A management network up to this size is checked address by address (exact); larger ones as address ranges.
ENUMERATE_LIMIT = 256
CARP_PROTO, PFSYNC_PROTO = "112", "240"
CARP_GROUPS = {4: ipaddress.ip_address("224.0.0.18"), 6: ipaddress.ip_address("ff02::12")}
PFSYNC_GROUP = ipaddress.ip_address("224.0.0.240")
_LOOPBACK = {ipaddress.ip_address("127.0.0.1"), ipaddress.ip_address("::1")}
def parse_sources(values) -> list[IPNet]:
"""The configured management sources as networks (an address becomes a /32 or /128). Invalid entries are
dropped; the caller treats an empty result as "not configured" (the cleanup is then locked)."""
nets: list[IPNet] = []
for value in values or []:
try:
net = ipaddress.ip_network(str(value).strip(), strict=False)
except ValueError:
continue
if net not in nets:
nets.append(net)
return nets
def management_targets(base_url: str, ctx: Context, extra_ports=()) -> tuple[list[IPAddr], list[int]]:
"""Firewall addresses and ports the management traffic goes to; ``extra_ports`` = the admin's
``cleanup_management_ports`` (an SSH or GUI port moved away from the defaults)."""
parts = urlsplit(base_url or "")
addrs: list[IPAddr] = []
try:
addrs.append(ipaddress.ip_address((parts.hostname or "").strip("[]")))
except ValueError:
pass # a DNS name: the interface addresses below cover the address it resolves to on the firewall
for addr in ctx.self_addrs:
if addr not in _LOOPBACK and addr not in addrs:
addrs.append(addr)
try:
api_port = parts.port or 443
except ValueError:
api_port = 443
ports = [api_port]
for port in (*MANAGEMENT_PORTS, *extra_ports):
if isinstance(port, int) and 1 <= port <= 65535 and port not in ports:
ports.append(port)
return addrs, ports
def _interfaces(ctx: Context) -> list[str]:
return sorted(i for i in ctx.interfaces if i != "lo0")
# A packet can also arrive on a device that is not an assigned interface: a WireGuard instance (wg0), an OpenVPN
# server (ovpns1), IPsec (enc0). They have no identifier, so they are not in ``ctx.interfaces``, and the built-in
# groups that stand for them (``wireguard``, ``openvpn``, ``enc0``) list no members in ``firewall/group/searchItem``
# (live fixture tests/fixtures/rule_analysis_fw.json: members ""). One extra, virtual ingress is checked per query:
UNASSIGNED = "(unassigned)"
def unassigned_interface(rule: NormRule, ctx: Context) -> Outcome:
"""Could the rule's interface field match a packet arriving on an unassigned device? An assigned interface or a
user-defined group (its members are assigned interfaces) cannot; a built-in group or an unknown name might
(members unknown = protected); no interface (floating "any") always does."""
if not rule.interfaces:
return Outcome(True)
results = []
for name in rule.interfaces:
if name in ctx.interfaces:
results.append(Outcome(False))
elif name in BUILTIN_GROUP_DEVICES or ctx.groups is None or name not in ctx.groups:
results.append(Outcome(None, ["IFACE_MEMBERS_UNKNOWN"]))
else:
results.append(Outcome(False))
result = o_or(results)
return Outcome(t_not(result.value), result.reasons) if rule.if_not else result
def _ingress(rule: NormRule, ctx: Context) -> list[tuple[str, dict | None]]:
"""(interface, field overrides) for every ingress a management packet could take."""
virtual = {"interface": unassigned_interface(rule, ctx)}
if rule.received_on:
virtual["received_on"] = Outcome(None, ["RECEIVED_ON"])
return [(i, None) for i in _interfaces(ctx)] + [(UNASSIGNED, virtual)]
def _item_ranges(token: str, ctx: Context) -> list[coverage.Range] | None:
"""Address ranges an endpoint item stands for; None when that is not static."""
ranges, _alias = coverage._ranges_of(token, ctx)
if ranges is not None:
return ranges
name = token.strip().lower()
if name in ("(self)", "self", "this firewall"):
return [(a.version, int(a), int(a)) for a in ctx.self_addrs]
if name.endswith("ip") and name[:-2] in ctx.interfaces:
info = ctx.interfaces[name[:-2]]
return [(a.version, int(a), int(a)) for a in (*info.addrs, *info.vips)]
if name.startswith("__") and name.endswith("_network"):
name = name[2:-len("_network")]
if name in ctx.interfaces:
return [(n.version, int(n.network_address), int(n.broadcast_address)) for n in ctx.interfaces[name].nets]
return None
def _covers_all(ranges: list[coverage.Range], version: int, low: int, high: int) -> bool:
cursor = low
for _v, a, b in sorted(r for r in ranges if r[0] == version):
if a > cursor:
return False
cursor = max(cursor, b + 1)
if cursor > high:
return True
return cursor > high
def source_outcome(endpoint: Endpoint, net: IPNet, ctx: Context) -> Outcome:
"""Does the endpoint match at least one address of ``net``? (False is a proof for every address.)"""
if endpoint.is_any:
return Outcome(True)
if net.num_addresses <= ENUMERATE_LIMIT:
return o_or([resolve.endpoint_match(endpoint, ip, ctx) for ip in net])
ranges: list[coverage.Range] = []
for item in endpoint.items:
got = _item_ranges(item, ctx)
if got is None:
return Outcome(None, ["SOURCE_NOT_RESOLVABLE"])
ranges += got
low, high = int(net.network_address), int(net.broadcast_address)
if endpoint.negated:
return Outcome(not _covers_all(ranges, net.version, low, high))
return Outcome(any(v == net.version and a <= high and b >= low for v, a, b in ranges))
def _query(src: IPAddr, dst: IPAddr, proto: str, label: str, dport: int | None) -> Query:
return Query(src=src, dst=dst, proto=proto, proto_label=label, dport=dport, direction="in", egress="none")
def management_check(rule: NormRule, ctx: Context, sources: list[IPNet], targets: list[IPAddr],
ports: list[int]) -> dict:
"""{"result": "false" | "true" | "unknown", "queries": n, "reasons": [...], "match": {...} | None}.
Only "false" makes the rule removable."""
reasons: set[str] = set()
queries = 0
unknown_at = None
interfaces = _ingress(rule, ctx)
for net in sources:
src = source_outcome(rule.src, net, ctx)
if src.value is False:
dsts = [d for d in targets if d.version == net.version]
queries += len(dsts) * len(ports) * len(interfaces)
continue
for dst in (d for d in targets if d.version == net.version):
for port in ports:
q = _query(net.network_address, dst, "6", "tcp", port)
for iface, overrides in interfaces:
queries += 1
out = match_rule(rule, q, ctx, iface=iface, source=src, overrides=overrides)
where = {"source": str(net), "destination": str(dst), "port": port, "interface": iface}
if out.value is True:
return {"result": "true", "queries": queries, "reasons": [], "match": where}
if out.value is None:
reasons.update(out.reasons)
unknown_at = unknown_at or where
if not queries:
# No address of the firewall in the family of any source: nothing could be proven.
return {"result": "unknown", "queries": 0, "reasons": ["NO_MANAGEMENT_QUERY"], "match": None}
if unknown_at is not None:
return {"result": "unknown", "queries": queries, "reasons": sorted(reasons), "match": unknown_at}
return {"result": "false", "queries": queries, "reasons": [], "match": None}
def _hasync_interface(hasync: dict | None) -> str | None:
"""The pfsync interface from ``core/hasync/get`` ("" = HA sync disabled); None when not readable."""
node = (hasync or {}).get("hasync") if isinstance(hasync, dict) else None
if not isinstance(node, dict) or "pfsyncinterface" not in node:
return None
value = node.get("pfsyncinterface")
if isinstance(value, dict):
return next((k for k, o in value.items() if isinstance(o, dict) and str(o.get("selected")) == "1"), "")
return str(value or "")
def _hasync_peer(hasync: dict | None) -> IPAddr | None:
node = (hasync or {}).get("hasync") if isinstance(hasync, dict) else None
try:
return ipaddress.ip_address(str((node or {}).get("pfsyncpeerip") or "").strip())
except ValueError:
return None
def infrastructure_reason(rule: NormRule, ctx: Context, *, hasync: dict | None, clustered: bool) -> str | None:
"""'infrastructure' when the rule could carry CARP or pfsync traffic, else None."""
if rule.protos is not None and rule.protos & {CARP_PROTO, PFSYNC_PROTO}:
return "infrastructure"
for info in ctx.interfaces.values():
if info.identifier == "lo0" or not info.vips:
continue
for net in info.nets:
group = CARP_GROUPS.get(net.version)
out = match_rule(rule, _query(net.network_address, group, CARP_PROTO, "carp", None), ctx,
iface=info.identifier, source=source_outcome(rule.src, net, ctx))
if out.value is not False:
return "infrastructure"
pfsync_if = _hasync_interface(hasync)
if pfsync_if == "" or (pfsync_if is None and not clustered):
return None # HA sync off (or a single firewall whose HA settings are not readable)
peer = _hasync_peer(hasync)
candidates = [ctx.interfaces[pfsync_if]] if pfsync_if in ctx.interfaces else \
[i for i in ctx.interfaces.values() if i.identifier != "lo0"] # unknown: every interface
for info in candidates:
for net in info.nets:
if peer is not None and peer.version == net.version:
src = Outcome(resolve.endpoint_match(rule.src, peer, ctx).value)
src_ip = peer
else:
src, src_ip = source_outcome(rule.src, net, ctx), net.network_address
for dst in [*(a for a in info.addrs if a.version == net.version)] + ([PFSYNC_GROUP] if net.version == 4 else []):
out = match_rule(rule, _query(src_ip, dst, PFSYNC_PROTO, "pfsync", None), ctx,
iface=info.identifier, source=src)
if out.value is not False:
return "infrastructure"
return None
+35 -5
View File
@@ -107,17 +107,47 @@ async def _managed_rules(db: AsyncSession, host: OpnsenseHost) -> dict[str, dict
return managed
async def cleanup_candidates(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
refresh: bool = False, now: datetime | None = None) -> dict:
"""Read-only. Raises when the firewall cannot be read (callers map that)."""
async def _hits(db: AsyncSession, host_id) -> dict:
return {r.rule_uuid.lower(): r for r in (await db.scalars(select(RuleHit).where(RuleHit.host_id == host_id))).all()}
async def cluster_hits(db: AsyncSession, host: OpnsenseHost) -> tuple[dict, dict]:
"""(usage to judge "unused" with, the host's own hit rows). For a cluster member the usage is merged over all
nodes (DESIGN-1.31 §3.4): "unused" only when it holds on every node."""
own = await _hits(db, host.id)
if not host.cluster_name:
return own, own
others = (await db.scalars(select(OpnsenseHost.id).where(
OpnsenseHost.cluster_name == host.cluster_name, OpnsenseHost.id != host.id))).all()
if not others:
return own, own
return rule_hit_service.merge_cluster_hits(own, [await _hits(db, other) for other in others]), own
async def cleanup_view(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
refresh: bool = False, now: datetime | None = None):
"""(context, normalized rules, candidate payload, managed rules) - the cleanup service needs all of them."""
now = now or datetime.now(timezone.utc)
ctx = await ctx_mod.load_context(host, use_cache=not refresh)
rules, notes = normalize(ctx)
hits = {r.rule_uuid.lower(): r for r in (await db.scalars(select(RuleHit).where(RuleHit.host_id == host.id))).all()}
usage, hits = await cluster_hits(db, host)
states = {s.rule_uuid: s for s in (await db.scalars(select(LiveRuleState).where(LiveRuleState.host_id == host.id))).all()}
rows = cleanup.build(rules, ctx, hits=hits, states=states, managed=await _managed_rules(db, host), now=now,
managed = await _managed_rules(db, host)
rows = cleanup.build(rules, ctx, hits=usage, states=states, managed=managed, now=now,
unused_days=unused_days, order_guessed=coverage.order_doubtful(notes))
return ctx, rules, _payload(host, rules, rows, hits, states, notes, ctx, unused_days, now), managed
async def cleanup_candidates(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
refresh: bool = False, now: datetime | None = None) -> dict:
"""Read-only. Raises when the firewall cannot be read (callers map that)."""
return (await cleanup_view(db, host, unused_days=unused_days, refresh=refresh, now=now))[2]
def _payload(host, rules, rows, hits, states, notes, ctx, unused_days, now) -> dict:
all_notes = sorted({*notes, *ctx.notes})
if host.cluster_name and any("CLUSTER_PEER_USAGE_UNKNOWN" in c.get("notes", ()) for r in rows for c in r["categories"]):
all_notes = sorted({*all_notes, "CLUSTER_PEER_USAGE_UNKNOWN"})
return {
"host_id": str(host.id), "host_name": host.name, "cluster_name": host.cluster_name,
"unused_days": max(cleanup.MIN_DAYS, min(cleanup.MAX_DAYS, int(unused_days))), "generated_at": now,
File diff suppressed because it is too large. Load diff
+5 -1
View File
@@ -155,9 +155,13 @@ async def sweep_expired_deployments(db: AsyncSession) -> list[RuleDeployment]:
async def undeploy_rule(db: AsyncSession, rule: FirewallRule) -> None:
from app.services.host_maintenance_service import ensure_no_open_cleanup
failures = []
result = await db.execute(select(RuleDeployment).where(RuleDeployment.rule_id == rule.id))
for deployment in result.scalars().all():
deployments = list(result.scalars().all())
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them.
await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])
for deployment in deployments:
host = await db.get(OpnsenseHost, deployment.host_id)
if host and deployment.opnsense_rule_uuid:
try:
+10 -5
View File
@@ -40,10 +40,12 @@ logger = logging.getLogger(__name__)
BASELINE_KEY = "__baseline__"
RETENTION_DAYS = 365
# Audit kinds whose events carry the UUID of a rule on a firewall in details["remote_uuid"].
AUDIT_KINDS = ("rule", "policy_hierarchy")
# Audit statuses that mean the firewall's rule was (or was meant to be) touched.
AUDIT_STATUSES_NO_CHANGE = ("requested", "interrupted", "error")
# Audit kinds whose events carry the UUID of a rule on a firewall in details["remote_uuid"]. "rule_cleanup": the
# cleanup assistant (rule_cleanup_service) switching a hand-made rule off/on, deleting or restoring it.
AUDIT_KINDS = ("rule", "policy_hierarchy", "rule_cleanup")
# Audit statuses that mean the firewall's rule was NOT changed by that event (so it cannot explain a change).
# "confirmed": the cleanup's confirmation only re-reads the firewall.
AUDIT_STATUSES_NO_CHANGE = ("requested", "interrupted", "error", "confirmed")
# The audit entry is written right after the firewall changed; allow for the observer running just before it.
AUDIT_SLACK = timedelta(minutes=2)
RECHECK_WINDOW = timedelta(minutes=45)
@@ -274,8 +276,11 @@ async def prune(db: AsyncSession, retention_days: int = RETENTION_DAYS) -> int:
events = await db.execute(delete(RuleChangeEvent).where(RuleChangeEvent.observed_at < cutoff))
gone = await db.execute(delete(LiveRuleState).where(
LiveRuleState.present.is_(False), LiveRuleState.rule_uuid != BASELINE_KEY, LiveRuleState.changed_at < cutoff))
# Finished cleanup operations (with the snapshots of deleted rules) are kept as long as the history.
from app.services.rule_cleanup_service import prune_operations
cleanups = await prune_operations(db, cutoff)
await db.commit()
return (events.rowcount or 0) + (gone.rowcount or 0)
return (events.rowcount or 0) + (gone.rowcount or 0) + cleanups
# ── Reading the history of one rule ───────────────────────────────────────
+39 -1
View File
@@ -9,7 +9,9 @@ releases without that endpoint fall back to parsing ``diagnostics/firewall/pfSta
* A counter that grew since the last sample is a hit "now" (at most one sampling interval late).
* A counter that is *lower* than before means pf reloaded its ruleset (an apply or a reboot), which
zeroes everything. Whatever it shows above zero was hit since the reload, so that also counts as a hit now;
zero tells nothing, the old ``last_hit_at`` stays.
zero tells nothing, the old ``last_hit_at`` stays. (Live, 2026-10-22, OPNsense 26.7.7: ``firewall/filter/apply``
WITHOUT any pending change already zeroes every ``ruleStats`` counter - 16228 -> 0 packets on the busiest rule -
so every apply is such a reload; writers sample right before their own apply, see rule_cleanup_service.)
* A rule is called unused only after OPNexus has watched it for ``UNUSED_AFTER_DAYS`` without a hit.
A new rule, or a brand-new installation, is "still collecting data" instead of a false alarm.
@@ -20,6 +22,7 @@ never tracked.
"""
import logging
import re
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, select, update
@@ -165,6 +168,41 @@ async def sample_all(db: AsyncSession) -> int:
return sampled
@dataclass
class ClusterHit:
"""The usage of one rule over all nodes of a cluster (DESIGN-1.31 §3.4). Has the attributes ``classify`` and
``cleanup.unused_confidence`` read from a ``RuleHit``. Traffic goes through the CARP master, a backup sees
almost nothing: a rule is only unused when it is unused everywhere, so the newest hit of any node counts, the
window starts at the latest start of any node and the counters are only as fresh as the oldest node's."""
rule_uuid: str
enabled: bool
first_seen_at: datetime
last_hit_at: datetime | None
last_sampled_at: datetime
# The rule is not tracked on some node (missing there, or the node was never sampled): nothing is known
# about its traffic there, so "unused" is never better than "probable".
peer_missing: bool = False
def merge_cluster_hits(own: dict[str, RuleHit], peers: list[dict[str, RuleHit]]) -> dict[str, ClusterHit]:
"""Per rule over the union of all nodes' rows (a rule this node has no row for is still judged, as
``peer_missing``: what this node saw of it is unknown)."""
merged: dict[str, ClusterHit] = {}
for key in {*own, *(k for p in peers for k in p)}:
rows = [own.get(key), *(p.get(key) for p in peers)]
present = [r for r in rows if r is not None]
hits = [r.last_hit_at for r in present if r.last_hit_at is not None]
mine = own.get(key) or present[0]
merged[key] = ClusterHit(
rule_uuid=mine.rule_uuid, enabled=mine.enabled,
first_seen_at=max(r.first_seen_at for r in present),
last_hit_at=max(hits) if hits else None,
last_sampled_at=min(r.last_sampled_at for r in present),
peer_missing=len(present) < len(rows),
)
return merged
def classify(row: RuleHit, now: datetime, days: int = UNUSED_AFTER_DAYS) -> str:
"""'used' | 'unused' | 'collecting' | 'disabled'; ``days`` is the observation window."""
if not row.enabled:
@@ -40,7 +40,7 @@ from app.models.host_operation import HostOperation
from app.models.scheduled_maintenance import ScheduledMaintenance
from contextlib import contextmanager
from app.services import audit_service, backup_store, opnsense_client as client
from app.services import audit_service, backup_store, opnsense_client as client, snapshot_service
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
from app.services.host_status_service import get_statuses
@@ -202,6 +202,17 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
await db.commit()
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
return
if sched.action in _BACKUP_BEFORE:
# Way back (1.32, B5): a boot environment of the running system, after every other check passed.
# Fail closed like the backup: no rollback point on a ZFS firewall, no update.
try:
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
except client.OpnsenseAPIError as exc:
sched.status = "failed"
sched.result = {"error": f"Rückweg-Punkt (Boot-Environment) vor dem Update nicht angelegt: {str(exc)[:300]}"}
await db.commit()
logger.warning("Geplante Wartung %s: Rückweg-Punkt fehlgeschlagen", sched.id)
return
boot_before = await _boot_time(host) if sched.auto_verify else None
op = HostOperation(
@@ -302,6 +313,9 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
backup_store.save_snapshot(host.id, await client.download_backup(host))
state = await inspect_host(db, host, skip_role_checks=True)
blocked = list(dict.fromkeys(blocked + state["actions"][sched.action]))
if not blocked and sched.action in _BACKUP_BEFORE:
# Way back (1.32, B5) while nothing has been toggled yet; a failure ends here like a failed backup.
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
except client.OpnsenseAPIError as exc:
await fail(f"Vorbereitung der CARP-Übergabe fehlgeschlagen, nichts wurde verändert: {str(exc)[:300]}")
return
+115 -9
View File
@@ -1,17 +1,22 @@
"""ZFS snapshots (boot environments) of one firewall: list, create, delete.
Snapshots are a safety net before bigger changes, not a rollback button:
activating one needs a reboot and discards everything since, so that action is
deliberately NOT offered here. Deleting refuses the active/next-boot snapshot
Snapshots are a safety net before bigger changes. Activating an arbitrary one is
deliberately NOT offered (it needs a reboot and discards everything since); the
only way back is the guided firmware rollback to a point OPNexus itself created
right before an update (see "Rollback points" below, maintenance action
`firmware_rollback`). Deleting refuses the active/next-boot snapshot
(the firewall refuses too; we say so up front). Every step is audited.
"""
import logging
import re
from datetime import datetime, timezone
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.services import audit_service, opnsense_client
from app.services.opnsense_client import OpnsenseAPIError
logger = logging.getLogger(__name__)
NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$")
@@ -23,20 +28,121 @@ def _row(row: dict) -> dict:
}
async def list_snapshots(host: OpnsenseHost) -> dict:
async def supported(host: OpnsenseHost) -> bool:
try:
supported = (await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported")
return bool((await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported"))
except OpnsenseAPIError as exc:
if "404" in str(exc):
return {"supported": False, "rows": []}
if "404" in str(exc): # OPNsense without the snapshots API
return False
raise
if not supported:
async def list_snapshots(host: OpnsenseHost) -> dict:
if not await supported(host):
return {"supported": False, "rows": []}
return {"supported": True, "rows": [_row(r) for r in await opnsense_client.list_snapshots(host)]}
# ── Rollback points (1.32, B5) ──────────────────────────────────────────────
# Before every firmware update/upgrade OPNexus creates a boot environment of the running system. Going back to it
# (activate + reboot) restores the old firmware AND the old configuration of that moment: everything changed on the
# firewall since is gone. Only these points are offered for going back, never boot environments made by hand.
ROLLBACK_PREFIX = "opnexus-pre-"
KEEP_ROLLBACK_POINTS = 2
_POINT = re.compile(r"^opnexus-pre-(?P<version>[A-Za-z0-9._-]+)-(?P<stamp>\d{8}-\d{6})$")
def _is_active(row: dict) -> bool:
# N = active now, R = active after reboot.
return str(row.get("active", "")).strip() not in ("", "-")
def rollback_points(rows: list[dict]) -> list[dict]:
"""OPNexus rollback points, newest first: {uuid, name, version, stamp, active, created}."""
out = []
for row in rows:
match = _POINT.match(str(row.get("name", "")))
if match:
out.append({"uuid": str(row.get("uuid", "")), "name": match.group(0), "version": match.group("version"),
"stamp": match.group("stamp"), "active": str(row.get("active", "")).strip(),
"created": str(row.get("created_str", ""))})
return sorted(out, key=lambda p: p["stamp"], reverse=True)
async def create_rollback_point(host: OpnsenseHost, version: str | None) -> dict | None:
"""Boot environment right before a firmware action. None when the firewall has no ZFS snapshots (UFS install):
the update may still run, without a way back. Raises OpnsenseAPIError when creating it fails (callers abort the
update: fail closed). Older OPNexus points beyond KEEP_ROLLBACK_POINTS are removed afterwards."""
if not await supported(host):
return None
clean_version = re.sub(r"[^A-Za-z0-9._-]", "-", version or "unbekannt")[:30] or "unbekannt"
name = f"{ROLLBACK_PREFIX}{clean_version}-{datetime.now(timezone.utc):%Y%m%d-%H%M%S}"
audit = {"kind": "snapshot", "action": "snapshot.rollback_point", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
await audit_service.persist(**audit, status="started", message=None, details={"version": version})
try:
await opnsense_client.create_snapshot(host, name, f"OPNexus: Rückweg-Punkt vor dem Firmware-Update von {version}")
rows = await opnsense_client.list_snapshots(host)
except OpnsenseAPIError as exc:
await audit_service.persist(**audit, status="rejected", message=str(exc)[:500], details={})
raise
point = next((p for p in rollback_points(rows) if p["name"] == name), None)
if point is None:
await audit_service.persist(**audit, status="unverified", message=None, details={})
raise OpnsenseAPIError(f"Rückweg-Punkt {name} nach dem Anlegen nicht in der Liste")
await audit_service.persist(**audit, status="applied", message=None, details={})
for old in [p for p in rollback_points(rows) if not _is_active(p)][KEEP_ROLLBACK_POINTS:]:
try: # housekeeping only: a point that cannot be removed stays, the update goes on
await opnsense_client.delete_snapshot(host, old["uuid"])
await audit_service.persist(kind="snapshot", action="snapshot.del", entity_name=old["name"], host_id=str(host.id),
host_name=host.name, status="applied", message="Älterer Rückweg-Punkt aufgeräumt", details={})
except OpnsenseAPIError as exc:
logger.warning("Rückweg-Punkt %s auf %s nicht gelöscht: %s", old["name"], host.name, exc)
return {"uuid": point["uuid"], "name": point["name"], "version": version}
async def rollback_state(host: OpnsenseHost) -> dict:
"""What going back would do: the newest OPNexus point that is not running now, and the boot environment that is."""
if not await supported(host):
return {"supported": False, "target": None, "current": None}
rows = await opnsense_client.list_snapshots(host)
current = next((r for r in rows if "N" in str(r.get("active", ""))), None) # running now (R alone = next start)
target = next((p for p in rollback_points(rows) if "N" not in p["active"]), None)
return {"supported": True, "target": target,
"current": {"uuid": str(current.get("uuid", "")), "name": str(current.get("name", ""))} if current else None}
async def activate_and_reboot(host: OpnsenseHost, rollback: dict) -> dict:
"""Activate the rollback point, check it is marked for the next start, reboot. If the reboot cannot be handed over,
the previous boot environment is activated again: an activated old one must never wait for the next unplanned
reboot."""
target, current = rollback["target"], rollback["current"]
async def undo() -> None:
if current:
try:
await opnsense_client.activate_snapshot(host, current["uuid"])
except OpnsenseAPIError:
logger.warning("Rückweg auf %s: vorheriges Boot-Environment nicht reaktiviert", host.name)
await opnsense_client.activate_snapshot(host, target["uuid"])
try:
rows = await opnsense_client.list_snapshots(host)
except OpnsenseAPIError:
await undo() # cannot confirm the mark: never leave it waiting for an unplanned reboot
raise
marked = next((r for r in rows if str(r.get("uuid", "")) == target["uuid"]), None)
if marked is None or "R" not in str(marked.get("active", "")):
await undo()
raise OpnsenseAPIError(f"Boot-Environment {target['name']} ist nicht für den nächsten Start vorgemerkt")
try:
return await opnsense_client.reboot_host(host)
except OpnsenseAPIError:
await undo()
raise
async def create(host: OpnsenseHost, name: str, note: str) -> dict:
if not NAME.match(name):
if not NAME.match(name) or name.startswith(ROLLBACK_PREFIX): # that prefix marks OPNexus' own rollback points
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field="name")
audit = {"kind": "snapshot", "action": "snapshot.add", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
await audit_service.persist(**audit, status="started", message=None, details={"note": note[:200]})
@@ -374,6 +374,9 @@ async def undeploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel) -> None:
dependency direction; deleting the Peer first also auto-unlinks it
from the Server's `peers` list server-side, per delClientAction's own
behavior — see module docstring), then reconfigures."""
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
await ensure_no_open_cleanup_on(db, WireguardTunnelDeployment, WireguardTunnelDeployment.tunnel_id == tunnel.id)
failures = []
result = await db.execute(select(WireguardTunnelDeployment).where(WireguardTunnelDeployment.tunnel_id == tunnel.id))
for deployment in result.scalars().all():
+2 -2
View File
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
Versioning scheme: see CHANGELOG.md ("Versionierung").
"""
APP_VERSION = "1.30.0"
APP_CODENAME = "Owl"
APP_VERSION = "1.32.0"
APP_CODENAME = "Merlin"
@@ -0,0 +1,103 @@
"""Regel-Bereinigung (1.31, B4 Teil 2): Operationen, Einträge, Einstellungen.
Revision ID: 0029
Revises: 0028
Create Date: 2026-10-22
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision = "0029"
down_revision = "0028"
branch_labels = None
depends_on = None
UUID = postgresql.UUID(as_uuid=True)
TS = sa.DateTime(timezone=True)
# Must equal OPEN_ITEM_STATES in app/models/rule_cleanup.py (the schema comparison of the tests notices a drift).
_OPEN_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
_SETTINGS = [
("cleanup_max_rules", "integer NOT NULL DEFAULT 10"),
("cleanup_delete_min_days", "integer NOT NULL DEFAULT 7"),
("cleanup_management_sources", "json NOT NULL DEFAULT '[]'::json"),
("cleanup_management_ports", "json NOT NULL DEFAULT '[]'::json"),
]
def upgrade() -> None:
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which may already have made all of it.
inspector = sa.inspect(op.get_bind())
if not inspector.has_table("rule_cleanup_operations"):
op.create_table(
"rule_cleanup_operations",
sa.Column("id", UUID, nullable=False),
sa.Column("host_id", UUID, nullable=False),
sa.Column("cluster_name", sa.String(120), nullable=True),
sa.Column("kind", sa.String(16), nullable=False),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("actor", sa.String(150), nullable=False),
sa.Column("actor_id", UUID, nullable=True),
sa.Column("created_at", TS, nullable=False),
sa.Column("expires_at", TS, nullable=False),
sa.Column("started_at", TS, nullable=True),
sa.Column("heartbeat_at", TS, nullable=True),
sa.Column("revert_deadline", TS, nullable=True),
sa.Column("confirmed_at", TS, nullable=True),
sa.Column("finished_at", TS, nullable=True),
sa.Column("message", sa.String(1000), nullable=True),
sa.Column("preview_fingerprint", sa.String(64), nullable=False),
sa.Column("options", sa.JSON(), nullable=False),
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("ix_rule_cleanup_operations_host", "rule_cleanup_operations", ["host_id", "created_at"])
op.create_index("ix_rule_cleanup_operations_status", "rule_cleanup_operations", ["status"])
if not inspector.has_table("rule_cleanup_items"):
op.create_table(
"rule_cleanup_items",
sa.Column("id", UUID, nullable=False),
sa.Column("operation_id", UUID, nullable=False),
sa.Column("host_id", UUID, nullable=False),
sa.Column("rule_uuid", sa.String(64), nullable=False),
sa.Column("seq", sa.Integer(), nullable=False),
sa.Column("carp_role", sa.String(16), nullable=True),
sa.Column("state", sa.String(16), nullable=False),
sa.Column("description", sa.Text(), nullable=False),
sa.Column("categories", sa.JSON(), nullable=False),
sa.Column("confidence", sa.String(16), nullable=True),
sa.Column("row_before", sa.JSON(), nullable=True),
sa.Column("fp_before", sa.String(64), nullable=True),
sa.Column("cover_uuid", sa.String(64), nullable=True),
sa.Column("cover_fp", sa.String(64), nullable=True),
sa.Column("snapshot", sa.JSON(), nullable=True),
sa.Column("anchor_next_uuid", sa.String(64), nullable=True),
sa.Column("anchor_prev_uuid", sa.String(64), nullable=True),
sa.Column("prio_group", sa.String(32), nullable=True),
sa.Column("source_item_id", UUID, nullable=True),
sa.Column("restored_uuid", sa.String(64), nullable=True),
sa.Column("changed_at", TS, nullable=True),
sa.Column("message", sa.String(1000), nullable=True),
sa.ForeignKeyConstraint(["operation_id"], ["rule_cleanup_operations.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("ix_rule_cleanup_items_operation", "rule_cleanup_items", ["operation_id"])
op.create_index(
"uq_cleanup_rule_open", "rule_cleanup_items", ["host_id", "rule_uuid"], unique=True,
postgresql_where=sa.text("state IN (" + ", ".join(f"'{s}'" for s in _OPEN_STATES) + ")"),
)
for name, ddl in _SETTINGS:
op.execute(f"ALTER TABLE governance_settings ADD COLUMN IF NOT EXISTS {name} {ddl}")
# The default only fills the existing row; the model supplies it for new ones (and the schema comparison
# cannot compare a JSON server default).
op.execute(f"ALTER TABLE governance_settings ALTER COLUMN {name} DROP DEFAULT")
def downgrade() -> None:
for name, _ddl in reversed(_SETTINGS):
op.execute(f"ALTER TABLE governance_settings DROP COLUMN IF EXISTS {name}")
op.execute("DROP TABLE IF EXISTS rule_cleanup_items")
op.execute("DROP TABLE IF EXISTS rule_cleanup_operations")
@@ -0,0 +1,45 @@
"""Stabile Schlüssel (config_key) für Regeln und NAT-Objekte (1.32, Konfiguration als Code).
Revision ID: 0030
Revises: 0029
Create Date: 2026-10-23
"""
from alembic import op
revision = "0030"
down_revision = "0029"
branch_labels = None
depends_on = None
_TABLES = (
("firewall_rules", "r"),
("port_forward_rules", "pf"),
("one_to_one_rules", "nat1"),
("outbound_nat_rules", "out"),
("policy_hierarchy_rules", "ph"),
)
def upgrade() -> None:
# IF NOT EXISTS / DO-Block: ein Altbestand ohne Alembic wird zuerst per create_all() angehoben und hat Spalte samt
# Constraint dann schon.
for table, prefix in _TABLES:
op.execute(f"ALTER TABLE {table} ADD COLUMN IF NOT EXISTS config_key varchar(64)")
op.execute(
f"UPDATE {table} SET config_key = '{prefix}-' || substr(replace(id::text, '-', ''), 1, 12) "
"WHERE config_key IS NULL"
)
op.execute(f"ALTER TABLE {table} ALTER COLUMN config_key SET NOT NULL")
op.execute(
f"""DO $$ BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '{table}_config_key_key') THEN
ALTER TABLE {table} ADD CONSTRAINT {table}_config_key_key UNIQUE (config_key);
END IF;
END $$"""
)
def downgrade() -> None:
for table, _prefix in _TABLES:
op.execute(f"ALTER TABLE {table} DROP CONSTRAINT IF EXISTS {table}_config_key_key")
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS config_key")
@@ -0,0 +1,23 @@
"""Rollout-Gate: Erreichbarkeitsproben von der Firewall aus (1.32, B5).
Revision ID: 0031
Revises: 0030
Create Date: 2026-10-23
"""
from alembic import op
revision = "0031"
down_revision = "0030"
branch_labels = None
depends_on = None
def upgrade() -> None:
# IF NOT EXISTS: a pre-Alembic install is bridged with create_all() first, which may already have the column.
op.execute("ALTER TABLE rollout_plans ADD COLUMN IF NOT EXISTS gate_probes json")
op.execute("UPDATE rollout_plans SET gate_probes = '[]'::json WHERE gate_probes IS NULL")
op.execute("ALTER TABLE rollout_plans ALTER COLUMN gate_probes SET NOT NULL")
def downgrade() -> None:
op.execute("ALTER TABLE rollout_plans DROP COLUMN IF EXISTS gate_probes")
+1
View File
@@ -14,3 +14,4 @@ cryptography==50.0.2
alembic==1.20.0
PyJWT==2.10.1
tzdata==2025.2
PyYAML==6.0.3
+53
View File
@@ -0,0 +1,53 @@
"""Seed data for the configuration-as-code tests: every object kind, hierarchy, placeholders, variables."""
from app.database import SessionLocal
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.host_variable import HostVariable
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
async def seed_everything() -> dict:
"""Two hosts, a device group, a hierarchy (Organisation > Nord), and one object of every kind."""
async with SessionLocal() as db:
fw1 = OpnsenseHost(name="fw-filiale-1", base_url="https://secret-host.example:8443", api_key="AKIA-SECRET-KEY")
fw2 = OpnsenseHost(name="fw-filiale-2", base_url="https://other.example", api_key="k2")
group = DeviceGroup(name="Filialen", description="alle Filialen", hosts=[fw1, fw2])
org = PolicyHierarchyGroup(name="Organisation", description="")
db.add_all([fw1, fw2, group, org])
await db.flush()
nord = PolicyHierarchyGroup(name="Nord", description="Region Nord", parent_id=org.id)
db.add(nord)
await db.flush()
fw1.policy_hierarchy_group_id = nord.id
db.add_all([
HostVariable(host_id=fw1.id, name="LAN_NET", value="10.20.30.0/24"),
HostVariable(host_id=fw1.id, name="SITE", value="Filiale Nord"),
FirewallAlias(name="WEB_SERVERS", type="host", description="DMZ", enabled=True,
content="10.20.30.10\n203.0.113.20", groups=[group], policy_hierarchy_group_id=None),
FirewallRule(config_key="allow-dns-lan", description="DNS aus {{SITE}}", interface="lan", direction="in",
source_net="{{LAN_NET}}", destination_net="WEB_SERVERS", destination_port="53", protocol="TCP/UDP",
groups=[group], targets=[]),
FirewallRule(config_key="block-all", description="alles sperren", interface="wan", direction="in", action="block",
log=True, targets=[fw2], groups=[]),
PortForwardRule(config_key="pf-web", description="Web DMZ", interface="wan", destination_net="wanip",
destination_port="443", target="10.20.30.10", local_port="443", targets=[fw1], groups=[]),
OneToOneRule(config_key="nat1-mail", description="Mail", interface="wan", external="203.0.113.25",
source_net="10.20.30.25", destination_net="any", groups=[group], targets=[]),
OutboundNatRule(config_key="out-lan", description="LAN raus", interface="wan", protocol="any",
source_net="{{LAN_NET}}", destination_net="any", target="wanip", groups=[group], targets=[],
policy_hierarchy_group_id=None),
PolicyHierarchyRule(config_key="ph-block-bogons", hierarchy_group_id=org.id, rulebase="pre", sequence=0,
description="Bogons", interface="wan", direction="in", action="block", source_net="bogons"),
PolicyHierarchyRule(config_key="ph-allow-mgmt", hierarchy_group_id=org.id, rulebase="pre", sequence=1,
description="Mgmt", interface="lan", direction="in", action="pass"),
PolicyHierarchyRule(config_key="ph-post-deny", hierarchy_group_id=nord.id, rulebase="post", sequence=0,
description="Standard sperren", interface="lan", direction="in", action="block"),
])
await db.commit()
return {"fw1": "fw-filiale-1", "fw2": "fw-filiale-2"}
+2 -1
View File
@@ -6,6 +6,7 @@ Each mutation: {"file": ..., "old": ..., "new": ..., "tests": "pytest selection"
The file is patched, the selected tests must FAIL, then the file is restored.
"""
import json
import shlex
import subprocess
import sys
from pathlib import Path
@@ -23,7 +24,7 @@ def main(spec_path: str) -> int:
continue
try:
path.write_text(original.replace(m["old"], m["new"]))
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", *m["tests"].split()],
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", *shlex.split(m["tests"])],
capture_output=True, text=True)
finally:
path.write_text(original)
+380
View File
@@ -0,0 +1,380 @@
[
{
"name": "fingerprint not compared at deploy",
"file": "app/services/rule_cleanup_service.py",
"old": " if plan_fingerprint(plan) != op.preview_fingerprint:",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_deploy_refuses_a_stale_preview_without_writing"
},
{
"name": "management UNKNOWN treated as FALSE (service)",
"file": "app/services/rule_cleanup_service.py",
"old": " if check[\"result\"] != \"false\":",
"new": " if check[\"result\"] == \"true\":",
"tests": "tests/test_rule_cleanup.py::test_rules_that_must_not_be_switched_off_are_refused"
},
{
"name": "management UNKNOWN treated as FALSE (protection)",
"file": "app/services/rule_analysis/protection.py",
"old": " return {\"result\": \"unknown\", \"queries\": queries, \"reasons\": sorted(reasons), \"match\": unknown_at}",
"new": " return {\"result\": \"false\", \"queries\": queries, \"reasons\": [], \"match\": None}",
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_rules_that_must_not_be_switched_off_are_refused"
},
{
"name": "network source: ranges ignored",
"file": "app/services/rule_analysis/protection.py",
"old": " return Outcome(any(v == net.version and a <= high and b >= low for v, a, b in ranges))",
"new": " return Outcome(False)",
"tests": "tests/test_rule_cleanup_protection.py"
},
{
"name": "CARP/pfsync protocol not excluded",
"file": "app/services/rule_analysis/protection.py",
"old": " if rule.protos is not None and rule.protos & {CARP_PROTO, PFSYNC_PROTO}:",
"new": " if False:",
"tests": "tests/test_rule_cleanup_protection.py"
},
{
"name": "pfsync interface not checked",
"file": "app/services/rule_analysis/protection.py",
"old": " if pfsync_if == \"\" or (pfsync_if is None and not clustered):",
"new": " if True:",
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_cluster_pfsync_interface_rules_are_infrastructure"
},
{
"name": "no management sources not locked",
"file": "app/services/rule_cleanup_service.py",
"old": " if not sources:\n raise api_error(409, ErrorCode.BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN)\n clustered",
"new": " if False:\n raise api_error(409, ErrorCode.BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN)\n clustered",
"tests": "tests/test_rule_cleanup.py::test_without_management_sources_everything_is_locked"
},
{
"name": "unused sure below 30 days",
"file": "app/services/rule_cleanup_service.py",
"old": " if unused_days < MIN_SURE_UNUSED_DAYS:",
"new": " if unused_days < 7:",
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_short_window_and_external_change_cap_at_probable"
},
{
"name": "unused block rule not capped",
"file": "app/services/rule_cleanup_service.py",
"old": " if rule.action in (\"block\", \"reject\"):",
"new": " if rule.action in (\"reject\",):",
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_probable_needs_an_explicit_acknowledgement_per_rule"
},
{
"name": "external change not capped",
"file": "app/services/rule_cleanup_service.py",
"old": " if external and best is not None:",
"new": " if False:",
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_short_window_and_external_change_cap_at_probable"
},
{
"name": "probable without acknowledgement",
"file": "app/services/rule_cleanup_service.py",
"old": " if level == \"probable\" and given not in acknowledge:",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_probable_needs_an_explicit_acknowledgement_per_rule"
},
{
"name": "unknown confidence selectable",
"file": "app/services/rule_cleanup_service.py",
"old": " if level == \"unknown\":\n refuse(\"confidence\")",
"new": " if False:\n refuse(\"confidence\")",
"tests": "tests/test_rule_cleanup.py::test_managed_legacy_and_unknown_confidence_are_refused"
},
{
"name": "cluster usage merge removed",
"file": "app/services/rule_analysis/service.py",
"old": " return rule_hit_service.merge_cluster_hits(own, [await _hits(db, other) for other in others]), own",
"new": " return own, own",
"tests": "tests/test_rule_cleanup.py::test_cluster_usage_is_judged_over_all_nodes"
},
{
"name": "write order master first",
"file": "app/services/rule_cleanup_service.py",
"old": " return sorted(nodes, key=lambda n: (n.role == \"MASTER\", n.host.name))",
"new": " return sorted(nodes, key=lambda n: (n.role != \"MASTER\", n.host.name))",
"tests": "tests/test_rule_cleanup.py::test_cluster_writes_backup_first_and_reverts_master_first"
},
{
"name": "revert order backup first",
"file": "app/services/rule_cleanup_service.py",
"old": " for item in sorted(items, key=lambda i: (i.carp_role != \"MASTER\", str(i.host_id))):",
"new": " for item in sorted(items, key=lambda i: (i.carp_role == \"MASTER\", str(i.host_id))):",
"tests": "tests/test_rule_cleanup.py::test_cluster_writes_backup_first_and_reverts_master_first"
},
{
"name": "no apply after a partial failure",
"file": "app/services/rule_cleanup_service.py",
"old": " if any(i.state in (\"disabled\", \"unknown\") for i in node_items):\n # Also after an error",
"new": " if not stop and any(i.state in (\"disabled\", \"unknown\") for i in node_items):\n # Also after an error",
"tests": "tests/test_rule_cleanup.py::test_refused_toggle_stops_writing_but_applies_what_is_switched"
},
{
"name": "write continues after a refusal",
"file": "app/services/rule_cleanup_service.py",
"old": " item.state, item.message = \"failed\", str(exc)[:1000]\n _audit(db, op, item, node.host, action=op.kind, status=\"error\", message=item.message)\n stop = True",
"new": " item.state, item.message = \"failed\", str(exc)[:1000]\n _audit(db, op, item, node.host, action=op.kind, status=\"error\", message=item.message)",
"tests": "tests/test_rule_cleanup.py::test_refused_toggle_stops_writing_but_applies_what_is_switched"
},
{
"name": "unclear toggle counted as done",
"file": "app/services/rule_cleanup_service.py",
"old": " item.state = \"disabled\" if seen is target else (\"failed\" if seen is not None else \"unknown\")",
"new": " item.state = \"disabled\"",
"tests": "tests/test_rule_cleanup.py::test_unclear_toggle_is_settled_by_reading"
},
{
"name": "confirm with unreachable firewall",
"file": "app/services/rule_cleanup_service.py",
"old": " except OpnsenseAPIError as exc:\n raise api_error(502, ErrorCode.BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR, host=member.name) from exc\n if not rule_hit_service.looks_complete(search):",
"new": " except OpnsenseAPIError as exc:\n continue\n if not rule_hit_service.looks_complete(search):",
"tests": "tests/test_rule_cleanup.py::test_confirm_reads_the_firewall_and_refuses_when_unreachable_or_different"
},
{
"name": "confirm ignores re-enabled rule",
"file": "app/services/rule_cleanup_service.py",
"old": " if row is None or _truthy(row.get(\"enabled\", \"1\")) is not target \\\n or _content_fp",
"new": " if row is None \\\n or _content_fp",
"tests": "tests/test_rule_cleanup.py::test_confirm_reads_the_firewall_and_refuses_when_unreachable_or_different"
},
{
"name": "confirm after the deadline",
"file": "app/services/rule_cleanup_service.py",
"old": " RuleCleanupOperation.revert_deadline > now).values(\n status=\"confirmed\"",
"new": " RuleCleanupOperation.revert_deadline > now - timedelta(days=1)).values(\n status=\"confirmed\"",
"tests": "tests/test_rule_cleanup.py::test_deadline_passing_while_confirm_reads_the_firewall_refuses tests/test_rule_cleanup.py::test_confirm_and_sweeper_at_the_deadline_decide_once"
},
{
"name": "sweeper does not revert expired",
"file": "app/services/rule_cleanup_service.py",
"old": " or_(and_(RuleCleanupOperation.status.in_(SWEEP_STATUSES), RuleCleanupOperation.revert_deadline <= now),",
"new": " or_(and_(RuleCleanupOperation.status.in_(SWEEP_STATUSES), RuleCleanupOperation.revert_deadline <= now - timedelta(days=1)),",
"tests": "tests/test_rule_cleanup.py::test_confirm_after_the_deadline_is_refused_and_the_sweeper_reverts tests/test_rule_cleanup.py::test_failed_apply_keeps_the_deadline_and_the_sweeper_reverts"
},
{
"name": "sweeper retries revert_failed",
"file": "app/services/rule_cleanup_service.py",
"old": "SWEEP_STATUSES = (\"awaiting_confirm\", \"partial\", \"apply_failed\")",
"new": "SWEEP_STATUSES = (\"awaiting_confirm\", \"partial\", \"apply_failed\", \"revert_failed\")",
"tests": "tests/test_rule_cleanup.py::test_revert_now_and_failed_revert_is_not_retried_automatically"
},
{
"name": "heartbeat not checked",
"file": "app/services/rule_cleanup_service.py",
"old": " if not _stale(op, len(items), nodes, now):\n # alive",
"new": " if False:\n # alive",
"tests": "tests/test_rule_cleanup.py::test_sweeper_skips_a_live_run_and_takes_over_an_abandoned_one"
},
{
"name": "delete before the waiting time",
"file": "app/services/rule_cleanup_service.py",
"old": " if now < due:",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_delete_needs_the_waiting_time_an_unchanged_rule_and_no_external_change"
},
{
"name": "delete of a changed rule",
"file": "app/services/rule_cleanup_service.py",
"old": " if _content_fp(row, False) != _content_fp(item.row_before or {}, False):",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_delete_needs_the_waiting_time_an_unchanged_rule_and_no_external_change"
},
{
"name": "snapshot not committed before delRule",
"file": "app/services/rule_cleanup_service.py",
"old": " item.state, item.changed_at = \"deleting\", _now()\n await _commit(db) # heartbeat + snapshot + intent, durable before delRule\n",
"new": " item.state, item.changed_at = \"deleting\", _now()\n",
"tests": "tests/test_rule_cleanup.py::test_delete_commits_the_snapshot_before_delrule"
},
{
"name": "unclear delete counted as deleted",
"file": "app/services/rule_cleanup_service.py",
"old": " return \"failed\" # still there",
"new": " return \"deleted\"",
"tests": "tests/test_rule_cleanup.py::test_unclear_delete_is_settled_with_getrule"
},
{
"name": "restore anchor ignored",
"file": "app/services/rule_cleanup_service.py",
"old": " await client.move_rule_before(member, new_uuid, anchor)",
"new": " pass",
"tests": "tests/test_rule_cleanup.py::test_restore_adds_disabled_at_the_anchor_with_a_new_uuid"
},
{
"name": "restore position not verified",
"file": "app/services/rule_cleanup_service.py",
"old": " return i > 0 and position[i - 1] == new_uuid.lower()",
"new": " return True",
"tests": "tests/test_rule_cleanup.py::test_restore_with_a_refused_move_is_verified_by_reading_and_only_warns"
},
{
"name": "R1 unassigned ingress not checked",
"file": "app/services/rule_analysis/protection.py",
"old": " return [(i, None) for i in _interfaces(ctx)] + [(UNASSIGNED, virtual)]",
"new": " return [(i, None) for i in _interfaces(ctx)]",
"tests": "tests/test_rule_cleanup_protection.py"
},
{
"name": "R1 builtin group members treated as known",
"file": "app/services/rule_analysis/protection.py",
"old": " elif name in BUILTIN_GROUP_DEVICES or ctx.groups is None or name not in ctx.groups:",
"new": " elif ctx.groups is None or name not in ctx.groups:",
"tests": "tests/test_rule_cleanup_protection.py"
},
{
"name": "R2 sweeper takes over a live run past its deadline",
"file": "app/services/rule_cleanup_service.py",
"old": " if not _stale(op, len(items), nodes, now):",
"new": " if not (op.revert_deadline is not None and op.revert_deadline <= now) and not _stale(op, len(items), nodes, now):",
"tests": "tests/test_rule_cleanup.py::test_sweeper_leaves_a_live_run_alone_even_past_its_deadline"
},
{
"name": "R2 heartbeat before a write not conditional",
"file": "app/services/rule_cleanup_service.py",
"old": " RuleCleanupOperation.id == op.id, RuleCleanupOperation.status == expect).values(heartbeat_at=now)",
"new": " RuleCleanupOperation.id == op.id).values(heartbeat_at=now)",
"tests": "tests/test_rule_cleanup.py::test_a_run_taken_over_by_another_backend_writes_nothing_more"
},
{
"name": "R3 extra management ports ignored",
"file": "app/services/rule_analysis/protection.py",
"old": " for port in (*MANAGEMENT_PORTS, *extra_ports):",
"new": " for port in MANAGEMENT_PORTS:",
"tests": "tests/test_rule_cleanup.py::test_extra_management_ports_protect_rules_on_them"
},
{
"name": "R3 management ports shown to non-admins",
"file": "app/routers/governance.py",
"old": " cleanup_management_ports=ports if user.role == \"admin\" else [],",
"new": " cleanup_management_ports=ports,",
"tests": "tests/test_rule_cleanup_api.py::test_management_ports_are_validated_normalised_audited_and_hidden"
},
{
"name": "R3 port range not validated",
"file": "app/routers/governance.py",
"old": " if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:",
"new": " if not text.isdigit():",
"tests": "tests/test_rule_cleanup_api.py::test_management_ports_are_validated_normalised_audited_and_hidden"
},
{
"name": "R4 any host may act on any operation",
"file": "app/services/rule_cleanup_service.py",
"old": " if op.host_id != host.id and not (cluster and host.cluster_name and op.host_id in {h.id for h in await _fleet(db, host)}):",
"new": " if op.host_id != host.id and not cluster:",
"tests": "tests/test_rule_cleanup.py::test_cluster_peer_may_confirm_and_read_but_strangers_may_not"
},
{
"name": "R4 peers refused",
"file": "app/services/rule_cleanup_service.py",
"old": " if op.host_id != host.id and not (cluster and host.cluster_name and op.host_id in {h.id for h in await _fleet(db, host)}):",
"new": " if op.host_id != host.id:",
"tests": "tests/test_rule_cleanup.py::test_cluster_peer_may_confirm_and_read_but_strangers_may_not"
},
{
"name": "R5 half-deleted node not skipped",
"file": "app/services/rule_cleanup_service.py",
"old": " if clustered and kind == \"delete\" and await _already_deleted(db, node.host.id, u):",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_finished"
},
{
"name": "R5 failed delete counts as basis",
"file": "app/services/rule_cleanup_service.py",
"old": " if not (o.kind == \"delete\" and i.state in (\"failed\", \"skipped\"))]",
"new": " ]",
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_finished"
},
{
"name": "R5 nothing left still planned",
"file": "app/services/rule_cleanup_service.py",
"old": " if not any(i[\"given\"] == given for n in plan_nodes for i in n[\"items\"]):",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_finished"
},
{
"name": "R5 restore needs all siblings deleted",
"file": "app/services/rule_cleanup_service.py",
"old": " if s.state == \"deleted\" and not s.restored_uuid]",
"new": " ]\n if any(s.state != \"deleted\" or s.restored_uuid for s in siblings):\n raise api_error(409, ErrorCode.BEREINIGUNG_AKTION_NICHT_MOEGLICH, status=\"partial\")",
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_restored_where_it_was_deleted"
},
{
"name": "R7 enabling a block rule without acknowledgement",
"file": "app/services/rule_cleanup_service.py",
"old": " if rule.action != \"pass\" and item[\"protection\"][\"result\"] != \"false\":",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_switching_a_block_rule_back_on_needs_an_acknowledgement_unless_proven_harmless"
},
{
"name": "R7 unknown protection counted as harmless",
"file": "app/services/rule_cleanup_service.py",
"old": " if rule.action != \"pass\" and item[\"protection\"][\"result\"] != \"false\":",
"new": " if rule.action != \"pass\" and item[\"protection\"][\"result\"] == \"true\":",
"tests": "tests/test_rule_cleanup.py::test_switching_a_block_rule_back_on_needs_an_acknowledgement_unless_proven_harmless"
},
{
"name": "R8 router does not redact for non-admins",
"file": "app/routers/rule_cleanup.py",
"old": " if not _full_view(request):\n operations = [service.redact(op) for op in operations]",
"new": " if False:\n operations = [service.redact(op) for op in operations]",
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
},
{
"name": "R8 preview answer not redacted",
"file": "app/routers/rule_cleanup.py",
"old": " return result if _full_view(request) else service.redact(result)",
"new": " return result",
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
},
{
"name": "R8 audit admin_only not stripped",
"file": "app/routers/audit.py",
"old": " if not full_view(request):",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
},
{
"name": "R8 read token of an admin sees everything",
"file": "app/routers/audit.py",
"old": " return getattr(user, \"role\", None) == \"admin\" and (token is None or token.scope == \"admin\")",
"new": " return getattr(user, \"role\", None) == \"admin\"",
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
},
{
"name": "R8 base_url kept in plan",
"file": "app/services/rule_cleanup_service.py",
"old": " out[\"nodes\"] = [{**{k: v for k, v in n.items() if k != \"base_url\"},",
"new": " out[\"nodes\"] = [{**{k: v for k, v in n.items()},",
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
},
{
"name": "R8 protection match kept",
"file": "app/services/rule_cleanup_service.py",
"old": " return {k: v for k, v in check.items() if k != \"match\"}",
"new": " return check",
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
},
{
"name": "R10 stored restored_uuid ignored",
"file": "app/services/rule_cleanup_service.py",
"old": " if item.restored_uuid:\n try:\n await client.get_rule(member, item.restored_uuid)",
"new": " if False:\n try:\n await client.get_rule(member, item.restored_uuid)",
"tests": "tests/test_rule_cleanup.py::test_unclear_restore_uses_the_stored_uuid_before_searching_by_content"
},
{
"name": "R10 twins resolved to the first",
"file": "app/services/rule_cleanup_service.py",
"old": " if len(found) == 1:\n return \"found\", found[0]",
"new": " if found:\n return \"found\", found[0]",
"tests": "tests/test_rule_cleanup.py::test_unclear_restore_with_twins_and_no_stored_uuid_stays_unknown"
},
{
"name": "R11 any acknowledgement text version accepted",
"file": "app/services/rule_cleanup_service.py",
"old": " if str(ack_text_version) != ACK_TEXT_VERSION:",
"new": " if False:",
"tests": "tests/test_rule_cleanup.py::test_an_outdated_acknowledgement_text_is_refused"
}
]
@@ -0,0 +1,373 @@
[
{
"name": "middleware: confirm not session-only",
"file": "app/services/api_token_service.py",
"old": "|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$\")",
"new": "\")",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "middleware: retry not admin-only",
"file": "app/services/api_token_service.py",
"old": "(deploy|retry)|items",
"new": "(deploy)|items",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "middleware: restore not admin-only",
"file": "app/services/api_token_service.py",
"old": "|items/[^/]+/restore)/?$\")",
"new": ")/?$\")",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "middleware: deploy-scope token passes admin-only cleanup routes",
"file": "app/services/api_token_service.py",
"old": "if scope == \"deploy\" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):",
"new": "if scope == \"deploy\" and (_ADMIN_ONLY_WRITES.match(path)):",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "middleware: operator passes admin-only cleanup routes",
"file": "app/services/api_token_service.py",
"old": " _ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))",
"new": " _ADMIN_ONLY_WRITES.match(path))",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "middleware: revert not a rollout (operators_may_deploy ignored)",
"file": "app/services/api_token_service.py",
"old": "r\"|rule-cleanup/[^/]+/(revert|retry)\"",
"new": "r\"|rule-cleanup/[^/]+/(retry)\"",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "middleware: retry not a rollout",
"file": "app/services/api_token_service.py",
"old": "r\"|rule-cleanup/[^/]+/(revert|retry)\"",
"new": "r\"|rule-cleanup/[^/]+/(revert)\"",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: deploy only needs a writer",
"file": "app/routers/rule_cleanup.py",
"old": " _admin_only(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.deploy(",
"new": " _writer(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.deploy(",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: retry only needs a writer",
"file": "app/routers/rule_cleanup.py",
"old": " _admin_only(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.retry(",
"new": " _writer(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.retry(",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: restore only needs a writer",
"file": "app/routers/rule_cleanup.py",
"old": " _admin_only(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.restore_item(",
"new": " _writer(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.restore_item(",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: confirm only needs a writer (tokens may confirm)",
"file": "app/routers/rule_cleanup.py",
"old": " _session_only(user, request)\n",
"new": " _writer(user, request)\n",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: revert needs no deployer",
"file": "app/routers/rule_cleanup.py",
"old": " await _deployer(user, request, db)\n",
"new": " _writer(user, request)\n",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: admin role not checked",
"file": "app/routers/rule_cleanup.py",
"old": " if user.role != \"admin\":\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n token = _token(request)",
"new": " if False:\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n token = _token(request)",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: admin token scope not checked",
"file": "app/routers/rule_cleanup.py",
"old": " if token is not None and token.scope != \"admin\":",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: session-only token check removed",
"file": "app/routers/rule_cleanup.py",
"old": " if token is not None:\n raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)",
"new": " if False:\n raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: operator revert ignores operators_may_deploy",
"file": "app/routers/rule_cleanup.py",
"old": "if user.role == \"operator\" and (await governance.get_settings(db)).operators_may_deploy:",
"new": "if user.role == \"operator\":",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: read-only account may write",
"file": "app/routers/rule_cleanup.py",
"old": " if user.role == \"read_only\":\n raise api_error(403, ErrorCode.READ_ONLY_ACCESS)",
"new": " if False:\n raise api_error(403, ErrorCode.READ_ONLY_ACCESS)",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: read-scope token may write",
"file": "app/routers/rule_cleanup.py",
"old": " if token is not None and token.scope == \"read\":",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: preview needs no writer",
"file": "app/routers/rule_cleanup.py",
"old": " user = await resolve_user(request, db)\n _writer(user, request)\n host = await _host(db, host_id)\n kind, uuids",
"new": " user = await resolve_user(request, db)\n host = await _host(db, host_id)\n kind, uuids",
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
},
{
"name": "router: actor is not the account name",
"file": "app/routers/rule_cleanup.py",
"old": "actor=user.username, actor_id=user.id, kind=kind",
"new": "actor='x', actor_id=user.id, kind=kind",
"tests": "tests/test_rule_cleanup_api.py -k 'preview_belongs_to_its_actor or shapes_and_audit'"
},
{
"name": "settings: any role may write",
"file": "app/routers/governance.py",
"old": " if user.role != \"admin\":\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n row = await get_settings(db)\n changes",
"new": " if False:\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n row = await get_settings(db)\n changes",
"tests": "tests/test_rule_cleanup_api.py -k only_an_admin_writes"
},
{
"name": "settings: whole address space accepted",
"file": "app/routers/governance.py",
"old": " if net.prefixlen == 0:",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: host names accepted",
"file": "app/routers/governance.py",
"old": " raise _bad(\"cleanup_management_sources\", \"not_an_address\") from None",
"new": " continue",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: unlimited list",
"file": "app/routers/governance.py",
"old": " if len(values) > MAX_MANAGEMENT_SOURCES:",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: sources not canonical",
"file": "app/routers/governance.py",
"old": " entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)",
"new": " entry = text",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: sources shown to everybody",
"file": "app/routers/governance.py",
"old": "cleanup_management_sources=sources if user.role == \"admin\" else [],",
"new": "cleanup_management_sources=sources,",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: change not audited",
"file": "app/routers/governance.py",
"old": " if changes:\n db.add(audit_service.make_event(",
"new": " if False:\n db.add(audit_service.make_event(",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: max rules unbounded",
"file": "app/routers/governance.py",
"old": " if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "settings: min days unbounded",
"file": "app/routers/governance.py",
"old": " if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
},
{
"name": "lock: ensure_no_maintenance ignores a cleanup",
"file": "app/services/host_maintenance_service.py",
"old": " if cleanup and ids:\n await ensure_no_open_cleanup(db, hosts)",
"new": " if False:\n await ensure_no_open_cleanup(db, hosts)",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: ensure_no_open_cleanup never raises",
"file": "app/services/host_maintenance_service.py",
"old": " if ids and await has_open_cleanup(db, ids):\n raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)",
"new": " if False:\n raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: maintenance inspection ignores a cleanup",
"file": "app/services/host_maintenance_service.py",
"old": " if await has_open_cleanup(db, fleet_ids):\n blocked.append(",
"new": " if False:\n blocked.append(",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: host delete ignores a cleanup",
"file": "app/routers/hosts.py",
"old": " await ensure_no_open_cleanup(db, [host])\n # Both deployment tables FK",
"new": " # Both deployment tables FK",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: host re-addressing ignores a cleanup",
"file": "app/routers/hosts.py",
"old": " await ensure_no_open_cleanup(db, [host])\n\n if payload.name is not None:",
"new": " pass\n\n if payload.name is not None:",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: rule deletion ignores a cleanup",
"file": "app/services/rule_deploy_service.py",
"old": " await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])",
"new": " pass",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: hierarchy rule deletion ignores a cleanup",
"file": "app/services/policy_hierarchy_deploy_service.py",
"old": " await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])",
"new": " pass",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: hierarchy reordering ignores a cleanup",
"file": "app/services/policy_hierarchy_deploy_service.py",
"old": " await ensure_no_open_cleanup(db, [h for h in hosts.values() if h.enabled])",
"new": " pass",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: removing a rule target ignores a cleanup",
"file": "app/routers/rules.py",
"old": " await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes",
"new": " pass",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: removing a hierarchy target ignores a cleanup",
"file": "app/routers/policy_hierarchy.py",
"old": " await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes",
"new": " pass",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: adoption ignores a cleanup",
"file": "app/routers/rules.py",
"old": " await ensure_no_open_cleanup(db, [host])\n try:\n rule = await rule_adopt_service.adopt(",
"new": " try:\n rule = await rule_adopt_service.adopt(",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
},
{
"name": "lock: cleanup service trips over itself",
"file": "app/services/rule_cleanup_service.py",
"old": "await ensure_no_maintenance(db, [host], cleanup=False)",
"new": "await ensure_no_maintenance(db, [host])",
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or preview_input_forms or shapes_and_audit or other_open_work'"
},
{
"name": "service: lost revert claim hits an expired row",
"file": "app/services/rule_cleanup_service.py",
"old": "\n await db.refresh(op) # _revert rolled back when it lost the claim: the status is expired\n raise api_error(409, ErrorCode.BEREINIGUNG_AKTION_NICHT_MOEGLICH, status=op.status)\n await db.commit()",
"new": "\n raise api_error(409, ErrorCode.BEREINIGUNG_AKTION_NICHT_MOEGLICH, status=op.status)\n await db.commit()",
"tests": "tests/test_rule_cleanup_api.py -k revert_is_for_admins"
},
{
"name": "service: lost retry claim hits an expired row",
"file": "app/services/rule_cleanup_service.py",
"old": "\n await db.refresh(op) # _revert rolled back when it lost the claim: the status is expired",
"new": "",
"tests": "tests/test_rule_cleanup_api.py -k retry_after_a_failed"
},
{
"name": "notify: failed revert not reported",
"file": "app/services/notification_checks.py",
"old": " if op.status == \"revert_failed\":",
"new": " if False:",
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
},
{
"name": "notify: partial run not reported",
"file": "app/services/notification_checks.py",
"old": " elif op.status in (\"partial\", \"apply_failed\"):",
"new": " elif False:",
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
},
{
"name": "notify: unclear delete not reported",
"file": "app/services/notification_checks.py",
"old": " elif op.kind in (\"delete\", \"restore\"):",
"new": " elif False:",
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
},
{
"name": "notify: every item of a failed revert is reported",
"file": "app/services/notification_checks.py",
"old": " stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]",
"new": " stuck = list(items)",
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
},
{
"name": "notify: cleanup problems not part of the revert alert",
"file": "app/services/notification_checks.py",
"old": " problems.update(await _cleanup_problems(db))",
"new": " pass",
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
},
{
"name": "preview writes no own audit entry",
"file": "app/services/rule_cleanup_service.py",
"old": " db.add(audit_service.make_event(\n kind=\"rule_cleanup\", action=\"preview\"",
"new": " (lambda *a, **k: None)(audit_service.make_event(\n kind=\"rule_cleanup\", action=\"preview\"",
"tests": "tests/test_rule_cleanup_api.py -k shapes_and_audit"
},
{
"name": "R9 alias undeploy without cleanup lock",
"file": "app/services/alias_deploy_service.py",
"old": " await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)\n",
"new": "",
"tests": "tests/test_rule_cleanup_api.py::test_alias_delete_and_target_removal_wait_for_the_cleanup"
},
{
"name": "R9 alias target removal without cleanup lock",
"file": "app/routers/aliases.py",
"old": " await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes\n",
"new": "",
"tests": "tests/test_rule_cleanup_api.py::test_alias_delete_and_target_removal_wait_for_the_cleanup"
},
{
"name": "R9 wireguard undeploy without cleanup lock",
"file": "app/services/wireguard_deploy_service.py",
"old": " await ensure_no_open_cleanup_on(db, WireguardTunnelDeployment, WireguardTunnelDeployment.tunnel_id == tunnel.id)\n",
"new": "",
"tests": "tests/test_rule_cleanup_api.py::test_every_undeploy_and_target_removal_checks_for_an_open_cleanup_first"
},
{
"name": "R9 helper checks nothing",
"file": "app/services/host_maintenance_service.py",
"old": " ids = set((await db.scalars(select(model.host_id).where(condition))).all())",
"new": " ids = set()",
"tests": "tests/test_rule_cleanup_api.py::test_alias_delete_and_target_removal_wait_for_the_cleanup"
}
]
+241
View File
@@ -0,0 +1,241 @@
"""A small stateful OPNsense for the rule-cleanup tests (respx side effects).
It keeps its rules in memory and answers searchRule, getRule, toggleRule, delRule, addRule, moveRuleBefore, apply,
ruleStats, pfStatistics (generated from the enabled rules, like pf after an apply), interfacesInfo, groups, aliases,
port forwards, getVipStatus and core/hasync/get in the shapes seen live (OPNsense 26.7, see opnsense_client.py).
``fail`` injects errors per call: {("toggle", uuid): httpx.ReadTimeout | httpx.Response | callable}.
Addresses: documentation/private ranges only.
"""
from __future__ import annotations
import copy
import re
import uuid as uuid_lib
import httpx
R_BASE = "aaaaaaaa-0000-4000-8000-000000000100"
R_DUP = "aaaaaaaa-0000-4000-8000-000000000200"
R_UNUSED = "aaaaaaaa-0000-4000-8000-000000000300"
R_BLOCK = "aaaaaaaa-0000-4000-8000-000000000400"
R_MGMT = "aaaaaaaa-0000-4000-8000-000000000500"
R_CARP = "aaaaaaaa-0000-4000-8000-000000000600"
R_ALIAS = "aaaaaaaa-0000-4000-8000-000000000700"
R_TAIL = "aaaaaaaa-0000-4000-8000-000000000800"
R_LEGACY = "0123456789abcdef0123456789abcdef"
AUTO = "02f4bab031b57d1e30553ce08e0ec131"
MGMT_SOURCE = "203.0.113.10"
DEVICES = {"wan": "vtnet0", "lan": "vtnet1"}
def rule(uuid, seq, *, action="pass", interface="lan", proto="TCP", src="any", dst="any", dport="", enabled="1",
description=None, **extra) -> dict:
row = {
"uuid": uuid, "sort_order": f"400000.{seq:07d}", "prio_group": "400000", "sequence": str(seq),
"enabled": enabled, "action": action, "quick": "1", "interface": interface, "interfacenot": "0",
"direction": "in", "ipprotocol": "inet", "protocol": proto, "source_net": src, "source_not": "0",
"source_port": "", "destination_net": dst, "destination_not": "0", "destination_port": dport,
"gateway": "", "sched": "", "statetype": "keep", "tagged": "", "log": "0", "categories": "",
"description": description or f"cleanup-{seq}",
}
row.update(extra)
return row
def default_rules() -> list[dict]:
return [
{"uuid": AUTO, "sort_order": "000001.1000001", "legacy": True, "is_automatic": True, "enabled": "1",
"action": "block", "quick": False, "direction": "in", "ipprotocol": "inet46", "protocol": "",
"source_net": "", "source_port": "", "destination_net": "", "destination_port": "",
"description": "Default deny / state violation rule"},
rule(R_BASE, 100, src="10.20.30.0/24", dport="9000", description="base 9000"),
rule(R_DUP, 200, src="10.20.30.0/24", dport="9000", description="dup 9000"),
rule(R_UNUSED, 300, src="10.20.30.0/24", dport="8081", description="unused 8081"),
rule(R_BLOCK, 400, action="block", proto="UDP", dport="9999", description="block 9999"),
rule(R_MGMT, 500, interface="wan", src="203.0.113.0/24", dport="443", description="mgmt 443"),
rule(R_CARP, 600, proto="CARP", description="carp"),
rule(R_ALIAS, 700, interface="wan", src="tbl", dport="443", description="alias 443"),
rule(R_TAIL, 800, src="10.20.30.7", dport="7000", description="tail 7000"),
]
class FakeFirewall:
def __init__(self, base: str, *, role: str | None = None, rules: list[dict] | None = None,
pfsync: str = "", peer: str = "", lan_addr: str = "10.20.30.2", wan_addr: str = "203.0.113.57",
log: list | None = None):
self.base = base
self.log = log # shared across firewalls: the order of calls over a cluster
self.role = role
self.rules: list[dict] = copy.deepcopy(rules if rules is not None else default_rules())
self.pfsync, self.peer = pfsync, peer
self.lan_addr, self.wan_addr = lan_addr, wan_addr
self.fail: dict = {}
self.calls: list[tuple[str, str]] = []
self.applied = 0
self.down = False
self.on_call = None # callable(kind, uuid) run before a write is answered (e.g. look at the database)
# ── state helpers ──
def get(self, uuid):
return next((r for r in self.rules if r["uuid"].lower() == uuid.lower()), None)
def _inject(self, key):
effect = self.fail.get(key)
if effect is None:
return None
if isinstance(effect, list):
effect = effect.pop(0) if effect else None
if effect is None:
return None
if isinstance(effect, type) and issubclass(effect, Exception):
raise effect("injected")
if isinstance(effect, Exception):
raise effect
if callable(effect) and not isinstance(effect, httpx.Response):
return effect()
return effect
def pf_lines(self) -> dict:
lines = {"@0 scrub in all fragment reassemble": {}}
n = 1
for r in sorted(self.rules, key=lambda r: r["sort_order"]):
if str(r.get("enabled")) != "1":
continue
if r.get("is_automatic"):
lines[f'@{n} block drop in log inet all label "{r["uuid"]}"'] = {"evaluations": 0}
else:
words = ["block" if r["action"] == "block" else "pass", "in", "quick", f"on {DEVICES.get(r['interface'], r['interface'])}", "inet"]
if r["protocol"] not in ("", "any"):
words.append(f"proto {r['protocol'].lower()}")
words.append(f'all label "{r["uuid"]}"')
lines[f"@{n} " + " ".join(words)] = {"evaluations": 0, "packets": 0}
n += 1
return {"rules": {"filter rules": lines}}
def model(self, row: dict) -> dict:
"""getRule shape: select fields as option dicts, no uuid key."""
out = {k: v for k, v in row.items() if k not in ("uuid", "is_automatic", "legacy")}
out["action"] = {a: {"value": a, "selected": 1 if row["action"] == a else 0} for a in ("pass", "block", "reject")}
out["interface"] = {i: {"value": i.upper(), "selected": 1 if i in str(row["interface"]).split(",") else 0} for i in ("lan", "wan")}
out["categories"] = []
out["audit"] = {"created": "x", "updated": "y"}
return out
# ── respx wiring ──
def mount(self, respx_mock):
respx_mock.route(url__startswith=self.base).mock(side_effect=self.handle)
return self
def handle(self, request: httpx.Request) -> httpx.Response:
path = request.url.path.removeprefix("/api")
method = request.method
self.calls.append((method, path))
if self.log is not None:
self.log.append((self.base, method, path))
if self.down:
raise httpx.ConnectError("down")
if method == "GET" and path == "/diagnostics/interface/getVipStatus":
rows = [] if self.role is None else [{"interface": "LAN", "vhid": "10", "status": self.role}]
return httpx.Response(200, json={"rows": rows, "carp": {"allow": "1", "maintenancemode": False}})
if method == "GET" and path == "/firewall/filter/searchRule":
got = self._inject(("search", ""))
if got is not None:
return got
return httpx.Response(200, json={"rows": copy.deepcopy(sorted(self.rules, key=lambda r: r["sort_order"]))})
if method == "GET" and path == "/diagnostics/firewall/pfStatistics/rules":
return httpx.Response(200, json=self.pf_lines())
if method == "GET" and path == "/firewall/filter_util/ruleStats":
return httpx.Response(200, json={"status": "ok", "stats": {}})
if method == "GET" and path == "/interfaces/overview/interfacesInfo":
return httpx.Response(200, json={"rows": [
{"identifier": "wan", "device": "vtnet0", "description": "WAN", "ipv4": [{"ipaddr": f"{self.wan_addr}/24"}],
"ipv6": [], "gateways": ["203.0.113.1"], "routes": ["default"]},
{"identifier": "lan", "device": "vtnet1", "description": "LAN",
"ipv4": [{"ipaddr": f"{self.lan_addr}/24"}, {"ipaddr": "10.20.30.1/24", "vhid": "10"}],
"ipv6": [], "gateways": [], "routes": ["10.20.30.0/24"]},
{"identifier": "lo0", "device": "lo0", "description": "Loopback", "ipv4": [{"ipaddr": "127.0.0.1/8"}], "ipv6": []},
]})
if method == "GET" and path == "/firewall/group/searchItem":
return httpx.Response(200, json={"rows": []})
if method == "GET" and path == "/firewall/alias/searchItem":
return httpx.Response(200, json={"rows": [{"name": "tbl", "type": "urltable", "content": "https://203.0.113.9/list"}]})
if method == "GET" and path == "/firewall/d_nat/searchRule":
return httpx.Response(200, json={"rows": []})
if method == "GET" and path == "/core/hasync/get":
sel = {"": {"value": "Disabled", "selected": 1 if not self.pfsync else 0},
"lan": {"value": "LAN", "selected": 1 if self.pfsync == "lan" else 0}}
return httpx.Response(200, json={"hasync": {"pfsyncinterface": sel, "pfsyncpeerip": self.peer, "password": "x"}})
m = re.fullmatch(r"/firewall/filter/getRule/([^/]+)", path)
if method == "GET" and m:
got = self._inject(("get", m.group(1)))
if got is not None:
return got
row = self.get(m.group(1))
if row is None or row.get("is_automatic") or row.get("legacy"):
return httpx.Response(200, json=[])
return httpx.Response(200, json={"rule": self.model(row)})
m = re.fullmatch(r"/firewall/filter/toggleRule/([^/]+)/([01])", path)
if method == "POST" and m:
if self.on_call:
self.on_call("toggle", m.group(1))
row = self.get(m.group(1))
if row is not None and not row.get("legacy"):
pending = ("toggle", m.group(1))
if pending in self.fail and self.fail.get(("toggle_applies", m.group(1))):
row["enabled"] = m.group(2) # applied on the firewall, but the answer is lost
got = self._inject(pending)
if got is not None:
return got
changed = row["enabled"] != m.group(2)
row["enabled"] = m.group(2)
return httpx.Response(200, json={"result": "Enabled" if m.group(2) == "1" else "Disabled", "changed": changed})
return httpx.Response(200, json={"result": "failed"})
m = re.fullmatch(r"/firewall/filter/delRule/([^/]+)", path)
if method == "POST" and m:
if self.on_call:
self.on_call("delete", m.group(1))
row = self.get(m.group(1))
applies = self.fail.get(("delete_applies", m.group(1)))
if applies and row is not None:
self.rules.remove(row)
got = self._inject(("delete", m.group(1)))
if got is not None:
return got
if row is None or row.get("legacy"):
return httpx.Response(200, json={"result": "not found"})
self.rules.remove(row)
return httpx.Response(200, json={"result": "deleted"})
if method == "POST" and path == "/firewall/filter/addRule":
got = self._inject(("add", ""))
if got is not None:
return got
import json as _json
body = _json.loads(request.content)["rule"]
new = str(uuid_lib.uuid4())
top = max(int(float(r.get("sequence") or 0)) for r in self.rules if not r.get("is_automatic"))
row = {**body, "uuid": new, "sequence": str(top + 100), "sort_order": f"400000.{top + 100:07d}", "prio_group": "400000"}
self.rules.append(row)
return httpx.Response(200, json={"result": "saved", "uuid": new})
m = re.fullmatch(r"/firewall/filter/moveRuleBefore/([^/]+)/([^/]+)", path)
if method == "POST" and m:
got = self._inject(("move", m.group(1))) or self._inject(("move", "*"))
if got is not None:
return got
moving, target = self.get(m.group(1)), self.get(m.group(2))
if moving is None or target is None or moving is target:
return httpx.Response(500, json={"errorMessage": "Unexpected error, check log for details"})
seq = int(target["sequence"]) - 1
moving["sequence"], moving["sort_order"] = str(seq), f"400000.{seq:07d}"
return httpx.Response(200, json={"status": "ok"})
if method == "POST" and path == "/firewall/filter/apply":
if self.on_call:
self.on_call("apply", "")
got = self._inject(("apply", ""))
if got is not None:
return got
self.applied += 1
return httpx.Response(200, json={"status": "OK\n\n"})
return httpx.Response(404, json={"message": f"unexpected {method} {path}"})
def writes(self) -> list[tuple[str, str]]:
return [(m, p) for m, p in self.calls if m == "POST"]
+169
View File
@@ -0,0 +1,169 @@
"""HTTP API of configuration as code (1.32 P7): rights matrix, audit, errors, limits."""
import pytest
import yaml
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from app.config import settings
from app.database import SessionLocal
from app.main import app
from app.models.audit import AuditEvent
from app.models.rule import FirewallRule
from app.services import api_token_service
from tests.config_code_seed import seed_everything
PW = "config-code-password-1"
async def _login(username: str, password: str) -> AsyncClient:
ac = AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
resp = await ac.post("/api/auth/login", json={"username": username, "password": password})
assert resp.status_code == 200, resp.text
return ac
def _bearer(token: str) -> AsyncClient:
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test", headers={"Authorization": f"Bearer {token}"})
async def _token(client, scope: str) -> str:
resp = await client.post("/api/tokens", json={"name": f"t-{scope}", "scope": scope, "current_password": settings.admin_password})
assert resp.status_code == 201, resp.text
return resp.json()["token"]
async def _changed_file(client) -> str:
text = (await client.get("/api/config/export")).text
data = yaml.safe_load(text)
data["rules"][0]["description"] = "über die API"
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
@pytest.mark.parametrize("scope,method,path,denied", [
("read", "GET", "/api/config/export", False),
("read", "POST", "/api/config/import/plan", False),
("read", "POST", "/api/config/import/apply", True),
("deploy", "POST", "/api/config/import/plan", False),
("deploy", "POST", "/api/config/import/apply", True),
("admin", "POST", "/api/config/import/apply", False),
# the read-only exception is exactly this path, not its neighbours
("read", "POST", "/api/config/import/plan/x", True),
("read", "POST", "/api/config/import", True),
])
def test_token_scope_matrix(scope, method, path, denied):
assert api_token_service.denies(scope, method, path) is denied
def test_operators_may_plan_but_not_apply():
assert not api_token_service.operator_denies("POST", "/api/config/import/plan")
assert api_token_service.operator_denies("POST", "/api/config/import/apply")
async def test_export_plan_apply_as_admin_with_audit(client):
await seed_everything()
resp = await client.get("/api/config/export")
assert resp.status_code == 200 and resp.headers["content-type"].startswith("text/yaml")
assert "attachment" in resp.headers["content-disposition"] and resp.headers["cache-control"] == "no-store"
text = await _changed_file(client)
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
assert plan["summary"]["update"] == 1 and plan["applicable"] and len(plan["token"]) == 64
assert [i["key"] for i in plan["items"]] == ["allow-dns-lan"] # unchanged objects are not listed
resp = await client.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan["token"]})
assert resp.status_code == 200 and resp.json()["summary"]["update"] == 1
async with SessionLocal() as db:
events = (await db.execute(select(AuditEvent))).scalars().all()
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
assert rule.description == "über die API"
requests = [e for e in events if e.kind == "config" and e.action == "apply"]
assert [e.status for e in requests] == ["requested", "accepted"] and requests[0].entity_id is None
assert not any(e.action == "plan" for e in events) # planning is like a preview: not audited
assert all("über die API" not in str(e.details) for e in events if e.kind == "config")
assert any(e.kind == "rule" and e.action == "import_update" and e.actor == settings.admin_username for e in events)
async def test_stale_blocked_invalid_and_too_large(client):
await seed_everything()
text = await _changed_file(client)
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
resp = await client.post("/api/config/import/apply", json={"yaml": text + "\n# anders\n", "plan_token": "0" * 64})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_PLAN_STALE"
resp = await client.post("/api/config/import/plan", json={"yaml": "a: &x 1\n"})
detail = resp.json()["detail"]
assert resp.status_code == 422 and detail["code"] == "CONFIG_INVALID" and detail["params"]["errors"][0]["code"] == "yaml_anchor_forbidden"
resp = await client.post("/api/config/import/plan", json={"yaml": "#" * (1024 * 1024 + 1)})
assert resp.status_code == 413 and resp.json()["detail"]["code"] == "CONFIG_TOO_LARGE"
resp = await client.post("/api/config/import/plan", content=b"x" * (4 * 1024 * 1024 + 1), headers={"content-type": "application/json"})
assert resp.status_code == 413
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 9\n"})
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "CONFIG_UNSUPPORTED_VERSION"
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n", "extra": 1})
assert resp.status_code == 422 and resp.json()["detail"]["params"]["errors"][0]["code"] == "bad_request"
resp = await client.get("/api/config/export?kinds=rules,secrets")
assert resp.status_code == 422
blocked = "opnexus_config: 1\naliases: []\n"
plan = (await client.post("/api/config/import/plan", json={"yaml": blocked, "prune": True})).json()
assert not plan["applicable"] and plan["items"][0]["action"] == "blocked"
resp = await client.post("/api/config/import/apply", json={"yaml": blocked, "prune": True, "plan_token": plan["token"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_BLOCKED"
async def test_read_only_account_can_export_and_plan_but_not_apply(client):
await seed_everything()
assert (await client.post("/api/users", json={"username": "cfg-reader", "password": PW, "role": "read_only"})).status_code == 201
text = await _changed_file(client)
reader = await _login("cfg-reader", PW)
try:
assert (await reader.get("/api/config/export")).status_code == 200
plan = await reader.post("/api/config/import/plan", json={"yaml": text})
assert plan.status_code == 200
resp = await reader.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "READ_ONLY_ACCESS"
finally:
await reader.aclose()
async def test_operator_can_plan_but_not_apply(client):
await seed_everything()
assert (await client.post("/api/users", json={"username": "cfg-operator", "password": PW, "role": "operator"})).status_code == 201
text = await _changed_file(client)
op = await _login("cfg-operator", PW)
try:
plan = await op.post("/api/config/import/plan", json={"yaml": text})
assert plan.status_code == 200
resp = await op.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "ADMIN_REQUIRED"
finally:
await op.aclose()
async with SessionLocal() as db:
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "config"))).scalars().all()
assert [e.status for e in events] == ["denied"]
@pytest.mark.parametrize("scope,plan_status,apply_status", [("read", 200, 403), ("deploy", 200, 403), ("admin", 200, 200)])
async def test_tokens(client, scope, plan_status, apply_status):
await seed_everything()
text = await _changed_file(client)
async with _bearer(await _token(client, scope)) as api:
assert (await api.get("/api/config/export")).status_code == 200
plan = await api.post("/api/config/import/plan", json={"yaml": text})
assert plan.status_code == plan_status
resp = await api.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
assert resp.status_code == apply_status, resp.text
async def test_group_changes_are_audited_now(client):
resp = await client.post("/api/groups", json={"name": "Neu", "description": "", "host_ids": []})
assert resp.status_code == 201
async with SessionLocal() as db:
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "group"))).scalars().all()
assert [e.status for e in events] == ["requested", "accepted"]
async def test_too_many_waiting_plans_are_turned_away(client, monkeypatch):
from app.routers import config_code
monkeypatch.setattr(config_code, "_waiting", config_code.MAX_WAITING)
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})
assert resp.status_code == 429 and resp.json()["detail"]["code"] == "CONFIG_BUSY"
monkeypatch.setattr(config_code, "_waiting", 0)
assert (await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})).status_code == 200
+434
View File
@@ -0,0 +1,434 @@
"""Planner and apply (1.32 P5/P6): round trip, partial files, prune protection, conflicts, atomicity, no firewall."""
import ast
import copy
from pathlib import Path
import pytest
import respx
import yaml
from sqlalchemy import func, select
from sqlalchemy.orm import selectinload
from app.config_code import apply as applier
from app.config_code import plan as planner
from app.config_code.export import export_yaml
from app.config_code.loader import ConfigError, parse
from app.database import SessionLocal
from app.models.alias import FirewallAlias
from app.models.audit import AuditEvent
from app.models.host import OpnsenseHost
from app.models.host_variable import HostVariable
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup, RuleRevision
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rollout import RolloutPlan, RolloutPlanWave
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
from tests.config_code_seed import seed_everything
CENTRAL = (PolicyHierarchyRule, OutboundNatRule, OneToOneRule, PortForwardRule, FirewallRule, FirewallAlias, DeviceGroup)
async def export() -> str:
async with SessionLocal() as db:
return await export_yaml(db)
async def plan(text: str, prune: bool = False) -> planner.Plan:
async with SessionLocal() as db:
return await planner.build_plan(db, parse(text), prune)
async def apply(text: str, prune: bool = False, token: str | None = None) -> planner.Plan:
token = token or (await plan(text, prune)).token
async with SessionLocal() as db:
try:
result = await applier.apply_config(db, parse(text), prune, token)
except Exception:
await db.rollback()
raise
await db.commit()
return result
def edit(text: str, fn) -> str:
data = yaml.safe_load(text)
fn(data)
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
def by_key(p: planner.Plan) -> dict:
return {(i.kind, i.key): i for i in p.items}
async def wipe_central_objects() -> None:
async with SessionLocal() as db:
await db.execute(HostVariable.__table__.delete())
for model in CENTRAL:
await db.execute(model.__table__.delete())
await db.execute(PolicyHierarchyGroup.__table__.update().values(parent_id=None))
await db.execute(OpnsenseHost.__table__.update().values(policy_hierarchy_group_id=None))
await db.execute(PolicyHierarchyGroup.__table__.delete())
await db.commit()
async def payloads() -> dict:
"""What every object would push to every host: the ROADMAP criterion "identical preview" after a round trip."""
from app.services.alias_deploy_service import _build_payload as alias_payload
from app.services.one_to_one_deploy_service import _build_payload as one_payload
from app.services.outbound_nat_deploy_service import _build_payload as out_payload
from app.services.port_forward_deploy_service import _build_payload as pf_payload
from app.services.rule_deploy_service import _build_payload as rule_payload
out = {}
async with SessionLocal() as db:
hosts = (await db.execute(select(OpnsenseHost))).scalars().all()
for model, builder, ident in ((FirewallRule, rule_payload, "config_key"), (PortForwardRule, pf_payload, "config_key"),
(OneToOneRule, one_payload, "config_key"), (OutboundNatRule, out_payload, "config_key"),
(FirewallAlias, alias_payload, "name")):
for row in (await db.execute(select(model))).scalars().all():
for host in hosts:
try:
out[(model.__name__, getattr(row, ident), host.name)] = builder(row, host)
except Exception as exc: # a missing variable is part of the result too
out[(model.__name__, getattr(row, ident), host.name)] = type(exc).__name__
return out
# -- round trip ---------------------------------------------------------------------------------------------------------
async def test_export_then_plan_shows_no_changes_and_apply_is_a_no_op():
await seed_everything()
text = await export()
p = await plan(text)
assert p.applicable and p.summary()["create"] == p.summary()["update"] == p.summary()["delete"] == 0
await apply(text, prune=True)
assert await export() == text
async def test_import_into_an_instance_without_central_objects_restores_everything_identically():
await seed_everything()
text, before = await export(), await payloads()
await wipe_central_objects()
p = await plan(text, prune=True)
assert p.applicable and p.summary()["create"] > 10
await apply(text, prune=True)
assert await export() == text
assert await payloads() == before
assert (await plan(text, prune=True)).summary()["create"] == 0
async def test_apply_writes_revisions_and_audit_events_without_the_file_content():
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="geändert"))
await apply(text)
async with SessionLocal() as db:
notes = (await db.execute(select(RuleRevision.note))).scalars().all()
events = (await db.execute(select(AuditEvent))).scalars().all()
assert "Import (Konfiguration als Code)" in notes and "Vor erster Bearbeitung" in notes
summary = next(e for e in events if e.kind == "config")
assert summary.action == "import" and summary.details["summary"]["update"] == 1
assert any(e.kind == "rule" and e.action == "import_update" and e.details["fields"] == ["description"] for e in events)
assert all("geändert" not in str(e.details) for e in events if e.kind == "config")
# -- partial files and prune ---------------------------------------------------------------------------------------------
async def test_a_missing_section_leaves_the_kind_alone_and_an_empty_one_only_deletes_with_prune():
await seed_everything()
only_aliases = "opnexus_config: 1\naliases: []\n"
p = await plan(only_aliases)
assert p.touched == ["aliases"] and p.summary() == {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
await apply(only_aliases)
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
p = await plan(only_aliases, prune=True)
assert by_key(p)[("aliases", "WEB_SERVERS")].reasons == ["prune_referenced"] # a rule still names it
rules_without_alias = edit(await export(), lambda d: d["rules"][0].update(destination_net="any"))
await apply(rules_without_alias)
await apply(only_aliases, prune=True)
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 0
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
async def test_prune_never_deletes_a_deployed_object_and_nothing_changes():
await seed_everything()
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="reverted", opnsense_rule_uuid="u"))
await db.commit()
text = edit(await export(), lambda d: d.update(rules=[r for r in d["rules"] if r["key"] != "block-all"]))
p = await plan(text, prune=True)
assert by_key(p)[("rules", "block-all")].reasons == ["prune_deployed"] and not p.applicable
with pytest.raises(applier.PlanBlocked):
await apply(text, prune=True, token=p.token)
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
async def test_an_object_with_an_open_deployment_cannot_be_changed():
await seed_everything()
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="pending", opnsense_rule_uuid="u"))
await db.commit()
text = edit(await export(), lambda d: d["rules"][1].update(description="neu"))
item = by_key(await plan(text))[("rules", "block-all")]
assert item.reasons == ["busy_deployment"] and item.deployed_hosts == 1
unchanged = await plan(await export())
assert unchanged.applicable # an unchanged busy object does not block
async def test_pruning_a_group_still_used_by_a_rollout_wave_is_blocked():
await seed_everything()
async with SessionLocal() as db:
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
rollout = RolloutPlan(name="r", action="update", created_by="t")
db.add(rollout)
await db.flush()
wave = RolloutPlanWave(plan_id=rollout.id, position=0, label="w", groups=[group])
db.add(wave)
await db.commit()
data = yaml.safe_load(await export())
for section in ("aliases", "rules", "one_to_one", "outbound_nat"):
for obj in data[section]:
obj["targets"]["groups"] = []
data["device_groups"] = []
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("device_groups", "Filialen")]
assert item.reasons == ["prune_referenced"]
async def test_pruning_a_hierarchy_group_is_blocked_while_hosts_rules_or_children_remain():
await seed_everything()
data = yaml.safe_load(await export())
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Organisation"]
with pytest.raises(ConfigError) as info: # Nord still names Organisation as parent
await plan(yaml.safe_dump(data), prune=True)
assert info.value.errors[0]["code"] == "unknown_hierarchy_group"
data = yaml.safe_load(await export())
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Nord"]
text = yaml.safe_dump(data)
with pytest.raises(ConfigError): # the hierarchy rule ph-post-deny still sits in Nord
await plan(text, prune=True)
data["policy_hierarchy_rules"] = [r for r in data["policy_hierarchy_rules"] if r["group"] != "Nord"]
p = await plan(yaml.safe_dump(data), prune=True)
assert p.applicable # fw-filiale-1 drops out of the hierarchy with its group
await apply(yaml.safe_dump(data), prune=True)
async with SessionLocal() as db:
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
assert host.policy_hierarchy_group_id is None
data = yaml.safe_load(await export())
data.pop("policy_hierarchy_rules")
data["policy_hierarchy_groups"] = []
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("policy_hierarchy_groups", "Organisation")]
assert item.reasons == ["prune_referenced"] # its hierarchy rules stay in the database
# -- validation against the database ------------------------------------------------------------------------------------
@pytest.mark.parametrize("change,code", [
(lambda d: d["rules"][0]["targets"]["hosts"].append("nope"), "unknown_host"),
(lambda d: d["rules"][0]["targets"]["groups"].append("nope"), "unknown_group"),
(lambda d: d["rules"].append(copy.deepcopy(d["rules"][0])), "duplicate_object_key"),
(lambda d: d["rules"][0]["targets"]["groups"].append("Filialen"), "duplicate_entry"),
(lambda d: d["policy_hierarchy_rules"][0].update(description="{{SITE}}"), "placeholder_not_supported"),
(lambda d: d["policy_hierarchy_groups"][0].update(parent="Nord"), "hierarchy_cycle"),
(lambda d: d["policy_hierarchy_groups"][0]["hosts"].append("fw-filiale-1"), "host_in_two_hierarchy_groups"),
(lambda d: d["variables"].update({"nope": {"X": "1"}}), "unknown_host"),
(lambda d: d["aliases"][0]["targets"].update(hierarchy_group="nope"), "unknown_hierarchy_group"),
])
async def test_file_errors_are_reported_with_a_path(change, code):
await seed_everything()
with pytest.raises(ConfigError) as info:
await plan(edit(await export(), change))
assert code in [e["code"] for e in info.value.errors]
assert all(e["path"] for e in info.value.errors)
async def test_warnings_for_unknown_aliases_removed_variables_and_redeploys():
await seed_everything()
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="success", opnsense_rule_uuid="u"))
await db.commit()
def change(d):
d["rules"][0].update(destination_net="MISSING_ALIAS", source_net="{{LAN_NET}}")
d["rules"][1].update(source_net="lan", destination_net="wanip")
d["variables"] = {"fw-filiale-1": {"SITE": "x"}}
p = await plan(edit(await export(), change), prune=True)
codes = {w["code"]: w for w in p.warnings}
assert codes["unknown_alias_reference"]["params"]["value"] == "MISSING_ALIAS"
assert codes["variable_removed_in_use"]["params"]["name"] == "LAN_NET"
assert codes["redeploy_needed"]["params"]["count"] == 1
assert [w for w in p.warnings if w["code"] == "unknown_alias_reference"] == [codes["unknown_alias_reference"]]
# -- hierarchy order -----------------------------------------------------------------------------------------------------
async def test_hierarchy_rule_order_is_the_order_in_the_file():
await seed_everything()
data = yaml.safe_load(await export())
rules = data["policy_hierarchy_rules"]
rules[0], rules[1] = rules[1], rules[0] # both in Organisation/pre
text = yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
p = await plan(text)
changed = {i.key: i.changes for i in p.items if i.action == "update"}
assert set(changed) == {"ph-block-bogons", "ph-allow-mgmt"} and changed["ph-allow-mgmt"][0]["field"] == "position"
await apply(text)
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]] == ["ph-allow-mgmt", "ph-block-bogons", "ph-post-deny"]
assert (await plan(text)).summary()["update"] == 0
async def test_new_hierarchy_rules_go_before_rules_the_file_does_not_name():
await seed_everything()
text = ("opnexus_config: 1\npolicy_hierarchy_rules:\n"
"- {key: ph-new, group: Organisation, rulebase: pre, description: neu, interface: lan}\n")
p = await plan(text)
assert {i.key: i.action for i in p.items} == {"ph-new": "create", "ph-block-bogons": "update", "ph-allow-mgmt": "update"}
assert all(i.implicit for i in p.items if i.key != "ph-new")
await apply(text)
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]][:3] == ["ph-new", "ph-block-bogons", "ph-allow-mgmt"]
# -- variables -----------------------------------------------------------------------------------------------------------
async def test_variables_are_merged_per_name_and_only_removed_with_prune():
await seed_everything()
text = "opnexus_config: 1\nvariables:\n fw-filiale-2:\n LAN_NET: 10.9.0.0/24\n fw-filiale-1:\n SITE: Neu\n"
await apply(text)
data = yaml.safe_load(await export())["variables"]
assert data == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
await apply(text, prune=True)
assert yaml.safe_load(await export())["variables"] == {"fw-filiale-1": {"SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
# -- conflicts, atomicity, self-check -----------------------------------------------------------------------------------
async def test_a_change_between_plan_and_apply_makes_the_plan_stale():
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="aus der Datei"))
token = (await plan(text)).token
async with SessionLocal() as db:
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen").options(selectinload(DeviceGroup.hosts)))
group.hosts = group.hosts[:1] # only a group member: no updated_at anywhere
await db.commit()
with pytest.raises(applier.PlanStale):
await apply(text, token=token)
async with SessionLocal() as db:
db.add(FirewallRule(description="neu in der UI", interface="lan", direction="in"))
await db.commit()
token = (await plan(text)).token
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.description == "neu in der UI"))
rule.description = "noch einmal geändert"
await db.commit()
with pytest.raises(applier.PlanStale):
await apply(text, token=token)
async def test_a_failure_in_the_middle_leaves_the_database_unchanged(monkeypatch):
await seed_everything()
before = await export()
text = edit(before, lambda d: [r.update(description=f"neu {i}") for i, r in enumerate(d["rules"])] and
d["aliases"][0].update(description="neu"))
original = applier._Writer.objects
async def broken(self, kind):
await original(self, kind)
if kind == "rules":
raise RuntimeError("injected")
monkeypatch.setattr(applier._Writer, "objects", broken)
with pytest.raises(RuntimeError):
await apply(text)
assert await export() == before
async def test_the_self_check_catches_a_writer_that_drops_a_field(monkeypatch):
await seed_everything()
before = await export()
text = edit(before, lambda d: d["rules"][0].update(log=True))
monkeypatch.setitem(applier.SCALARS, "rules", tuple(f for f in applier.SCALARS["rules"] if f != "log"))
with pytest.raises(applier.SelfCheckFailed) as info:
await apply(text)
assert info.value.differences == ["rules:allow-dns-lan"]
assert await export() == before
async def test_the_writer_covers_every_field_of_the_schema():
from app.config_code import schema
docs = {"aliases": schema.AliasDoc, "rules": schema.RuleDoc, "port_forwards": schema.PortForwardDoc,
"one_to_one": schema.OneToOneDoc, "outbound_nat": schema.OutboundNatDoc, "policy_hierarchy_rules": schema.HierarchyRuleDoc}
computed = {"key", "targets", "entries", "group"}
for kind, doc in docs.items():
assert set(applier.SCALARS[kind]) == set(doc.model_fields) - computed, kind
# -- never a firewall ----------------------------------------------------------------------------------------------------
async def test_plan_and_apply_never_talk_to_a_firewall():
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="x"))
with respx.mock(assert_all_called=False) as mock: # no routes: any HTTP request fails the test
await apply(text, prune=True)
await wipe_central_objects()
await apply(text, prune=True)
assert mock.calls.call_count == 0
def test_the_config_code_package_does_not_import_a_firewall_client():
forbidden = ("opnsense_client", "_deploy_service", "rollout_engine", "host_maintenance_service")
for path in Path(planner.__file__).parent.glob("*.py"):
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if isinstance(node, (ast.Import, ast.ImportFrom)):
names = [a.name for a in node.names] + ([node.module] if isinstance(node, ast.ImportFrom) and node.module else [])
assert not any(f in n for n in names for f in forbidden), f"{path.name}: {names}"
async def test_two_concurrent_applies_of_the_same_plan_serialize_and_the_second_is_stale():
import asyncio
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="parallel"))
token = (await plan(text)).token
results = await asyncio.gather(apply(text, token=token), apply(text, token=token), return_exceptions=True)
assert sorted(type(r).__name__ for r in results) == ["Plan", "PlanStale"]
async def test_variables_merged_beyond_the_per_host_limit_are_refused():
"""The host page allows 200 variables per host; merging a file into existing values must not get past that."""
await seed_everything() # fw-filiale-1 has 2 already
values = "".join(f" V{i}: x\n" for i in range(199))
with pytest.raises(ConfigError) as info:
await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values)
assert [e["code"] for e in info.value.errors] == ["too_many_variables"]
assert (await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values, prune=True)).applicable
async def test_what_the_ui_can_store_survives_the_round_trip_and_padding_is_cleaned_like_the_ui_does():
await seed_everything()
async with SessionLocal() as db:
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
group.description = "zwei\nZeilen " # GroupWrite does not strip descriptions
alias = await db.scalar(select(FirewallAlias).where(FirewallAlias.name == "WEB_SERVERS"))
alias.description = " eingerückt"
await db.commit()
text = await export()
assert (await plan(text)).summary()["update"] == 0
padded = edit(text, lambda d: d["rules"][0].update(description=" DNS aus {{SITE}} ", interface=" lan "))
p = await plan(padded)
assert p.summary()["update"] == 0 # RuleCreate strips these fields, so this is the same rule
with_entries = edit(text, lambda d: d["aliases"][0].update(entries=[" 10.20.30.10 ", "203.0.113.20"]))
assert (await plan(with_entries)).summary()["update"] == 0
+106
View File
@@ -0,0 +1,106 @@
"""Host console of configuration as code (1.32 P8): exit codes, text diff, actor, no control sequences."""
import copy
import json
import pytest
import yaml
from sqlalchemy import select
from app.config_code.__main__ import TOKEN_MARK, format_errors, run
from app.database import SessionLocal
from app.models.audit import AuditEvent
from app.models.rule import FirewallRule
from tests.config_code_seed import seed_everything
async def cli(*argv, stdin: str | bytes | None = None):
data = stdin.encode() if isinstance(stdin, str) else stdin
return await run(list(argv), data)
def token_of(out: str) -> str:
return next(line[len(TOKEN_MARK):] for line in out.splitlines() if line.startswith(TOKEN_MARK))
async def export_text() -> str:
code, out, _ = await cli("export")
assert code == 0
return out
def edit(text: str, fn) -> str:
data = yaml.safe_load(text)
fn(data)
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
async def test_export_plan_without_changes_exits_0_and_a_change_exits_2_with_a_readable_diff():
await seed_everything()
text = await export_text()
code, out, _ = await cli("plan", stdin=text)
assert code == 0 and "Plan: 0 neu, 0 geändert, 0 entfernt" in out and token_of(out)
def change(d):
d["rules"][0]["destination_port"] = 853
d["rules"].append({**copy.deepcopy(d["rules"][0]), "key": "new-rule"}) # no shared objects: YAML would write anchors
changed = edit(text, change)
code, out, _ = await cli("plan", stdin=changed)
assert code == 2
assert '+ rules new-rule (neu)' in out
assert '~ rules allow-dns-lan destination_port: "53" → "853"' in out
assert out.rstrip().splitlines()[-2].startswith("Plan: 1 neu, 1 geändert, 0 entfernt")
async def test_json_output_is_the_api_answer():
await seed_everything()
code, out, _ = await cli("plan", "--json", stdin=await export_text())
body = json.loads(out)
assert code == 0 and body["applicable"] and len(body["token"]) == 64 and body["summary"]["create"] == 0
async def test_apply_needs_the_token_of_the_plan_and_writes_audit_as_the_console():
await seed_everything()
changed = edit(await export_text(), lambda d: d["rules"][0].update(description="per CLI"))
code, _, err = await cli("apply", stdin=changed)
assert code == 1 and "--plan-token" in err
code, out, _ = await cli("plan", stdin=changed)
token = token_of(out)
code, out, err = await cli("apply", "--plan-token", "0" * 64, stdin=changed)
assert code == 4 and "neu planen" in err
code, out, err = await cli("apply", "--plan-token", token, stdin=changed)
assert code == 0 and "Übernommen: 0 neu, 1 geändert, 0 entfernt" in out and "nichts auf Firewalls geschrieben" in out
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
events = (await db.execute(select(AuditEvent))).scalars().all()
assert rule.description == "per CLI"
assert events and {e.actor for e in events} == {"cli (Host-Konsole)"}
async def test_invalid_files_exit_3_and_name_line_and_path_without_control_characters():
await seed_everything()
bad = "opnexus_config: 1\nrules:\n- key: r-1\n description: \"x\\e[31mred\"\n interface: lan\n nope: 1\n"
code, out, err = await cli("plan", stdin=bad)
assert code == 3 and not out
assert "Zeile 6: rules[0].nope: Unbekanntes Feld (field=nope)" in err and "\x1b" not in err
code, out, _ = await cli("plan", "--json", stdin="a: &x 1\n")
assert code == 3 and json.loads(out)["errors"][0]["code"] == "yaml_anchor_forbidden"
assert "\x1b" not in format_errors([{"code": "x", "path": "p\x1b[2J", "line": 1, "params": {"value": "v\x1b[0m"}}])
async def test_blocked_plans_exit_4_and_apply_refuses_them():
await seed_everything()
only_aliases = "opnexus_config: 1\naliases: []\n"
code, out, _ = await cli("plan", "--prune", stdin=only_aliases)
assert code == 4 and "! aliases WEB_SERVERS blockiert (entfernen): wird noch verwendet" in out
code, out, err = await cli("apply", "--prune", "--plan-token", token_of(out), stdin=only_aliases)
assert code == 4 and "blockierte Objekte" in err
async def test_kinds_schema_and_usage():
await seed_everything()
code, out, _ = await cli("export", "--kinds", "aliases")
assert code == 0 and list(yaml.safe_load(out)) == ["opnexus_config", "aliases"]
assert (await cli("export", "--kinds", "nope"))[0] == 1
code, out, _ = await cli("schema")
assert code == 0 and json.loads(out)["title"].startswith("OPNexus-Konfiguration")
assert (await cli("frobnicate"))[0] == 1
+129
View File
@@ -0,0 +1,129 @@
"""Export (1.32 P3): deterministic, complete, schema-conform, and free of anything secret."""
import re
import pytest
import yaml
from app.config_code import schema
from app.config_code.export import canonical, dump_yaml, export_yaml
from app.database import SessionLocal
from app.models.rule import FirewallRule
from tests.config_code_seed import seed_everything
GOLDEN = '''\
# OPNexus-Konfiguration (Schema v1), exportiert von OPNexus {version}.
# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run
opnexus_config: 1
device_groups:
- name: Filialen
description: alle Filialen
hosts:
- fw-filiale-1
- fw-filiale-2
policy_hierarchy_groups:
- name: Organisation
parent: null
description: ''
hosts: []
- name: Nord
parent: Organisation
description: Region Nord
hosts:
- fw-filiale-1
aliases:
- name: WEB_SERVERS
type: host
description: DMZ
enabled: true
entries:
- 10.20.30.10
- 203.0.113.20
targets:
hosts: []
groups:
- Filialen
hierarchy_group: null
'''
async def _export(**kw) -> str:
async with SessionLocal() as db:
return await export_yaml(db, **kw)
async def test_export_starts_with_the_fixed_header_and_the_expected_sections():
from app.version import APP_VERSION
await seed_everything()
text = await _export()
assert text.startswith(GOLDEN.format(version=APP_VERSION))
assert [k for k in yaml.safe_load(text) if k != "opnexus_config"] == list(schema.KINDS)
assert text.endswith("\n") and "\r" not in text
async def test_export_is_deterministic_and_validates_against_the_schema():
await seed_everything()
first, second = await _export(), await _export()
assert first == second
doc = schema.ConfigDoc.model_validate(yaml.safe_load(first)) # strict: unknown fields, wrong types, control chars
assert [r.key for r in doc.rules] == ["allow-dns-lan", "block-all"]
assert doc.variables == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Filiale Nord"}}
async def test_hierarchy_rules_are_ordered_by_group_rulebase_and_sequence_and_keep_placeholders_unresolved():
await seed_everything()
data = yaml.safe_load(await _export())
assert [r["key"] for r in data["policy_hierarchy_rules"]] == ["ph-block-bogons", "ph-allow-mgmt", "ph-post-deny"]
rule = next(r for r in data["rules"] if r["key"] == "allow-dns-lan")
assert rule["source_net"] == "{{LAN_NET}}" and rule["destination_port"] == "53"
async def test_ports_are_written_as_strings_so_they_survive_the_round_trip():
await seed_everything()
text = await _export()
assert re.search(r"destination_port: '53'", text)
assert schema.ConfigDoc.model_validate(yaml.safe_load(text)).rules[0].destination_port == "53"
async def test_export_never_contains_secrets_or_connection_data():
await seed_everything()
text = (await _export()).lower()
for forbidden in ("akia-secret-key", "secret-host.example", "https://", "api_key", "api_secret", "private key", "opx_", "password"):
assert forbidden not in text
async def test_kinds_limits_the_sections():
await seed_everything()
data = yaml.safe_load(await _export(kinds={"aliases", "rules"}))
assert list(data) == ["opnexus_config", "aliases", "rules"]
async def test_new_objects_get_a_key_and_an_empty_database_exports_empty_sections():
async with SessionLocal() as db:
rule = FirewallRule(description="x", interface="lan", direction="in")
db.add(rule)
await db.commit()
assert re.fullmatch(r"r-[0-9a-f]{12}", rule.config_key) and rule.config_key == f"r-{rule.id.hex[:12]}"
data = yaml.safe_load(await _export(kinds={"aliases", "variables"}))
assert data["aliases"] == [] and data["variables"] == {}
async def test_exporter_field_lists_match_the_schema():
"""A field added to the schema but not to the exporter (or vice versa) must fail loudly, not export silently wrong."""
await seed_everything()
async with SessionLocal() as db:
content = await canonical(db)
for kind, doc in (("rules", schema.RuleDoc), ("port_forwards", schema.PortForwardDoc), ("one_to_one", schema.OneToOneDoc),
("outbound_nat", schema.OutboundNatDoc), ("policy_hierarchy_rules", schema.HierarchyRuleDoc),
("aliases", schema.AliasDoc), ("device_groups", schema.DeviceGroupDoc),
("policy_hierarchy_groups", schema.HierarchyGroupDoc)):
assert all(list(item) == list(doc.model_fields) for item in content[kind]), kind
assert dump_yaml(content) == await _export()
async def test_objects_are_sorted_by_key_not_by_insertion_order():
async with SessionLocal() as db:
for key in ("zz-last", "aa-first", "mm-middle"):
db.add(FirewallRule(config_key=key, description=key, interface="lan", direction="in"))
await db.commit()
assert [r["key"] for r in yaml.safe_load(await _export())["rules"]] == ["aa-first", "mm-middle", "zz-last"]
+171
View File
@@ -0,0 +1,171 @@
"""Strict YAML loader (1.32 P2): everything a hostile file could try is refused with a stable code."""
import time
import pytest
from app.config_code import loader
from app.config_code.loader import ConfigError, load_yaml, parse
HEAD = "opnexus_config: 1\n"
def _code(raw) -> tuple[str, str]:
with pytest.raises(ConfigError) as info:
parse(raw)
return info.value.code, info.value.errors[0]["code"]
def _codes(raw) -> list[str]:
with pytest.raises(ConfigError) as info:
parse(raw)
return [e["code"] for e in info.value.errors]
RULE = "rules:\n- key: r-1\n description: d\n interface: lan\n"
def test_a_plain_file_loads():
doc, sha, _lines = parse(HEAD + RULE)
assert doc.rules[0].key == "r-1" and len(sha) == 64
@pytest.mark.parametrize("raw,code", [
# Billion laughs and every other use of anchors/aliases
("a: &x [1, 2]\nb: *x\n", "yaml_anchor_forbidden"),
("a: *x\n", "yaml_alias_forbidden"),
('lol1: &l1 ["x","x"]\nlol2: &l2 [*l1,*l1]\nlol3: [*l2,*l2]\n', "yaml_anchor_forbidden"),
# merge keys
("a: {b: 1}\nc:\n <<: {d: 1}\n", "yaml_merge_forbidden"),
# explicit tags of any kind
("a: !!python/object/apply:os.system ['id']\n", "yaml_tag_forbidden"),
("a: !!python/name:os.system\n", "yaml_tag_forbidden"),
("a: !!binary aGVsbG8=\n", "yaml_tag_forbidden"),
("a: !!set {x: null}\n", "yaml_tag_forbidden"),
("a: !!str 1\n", "yaml_tag_forbidden"),
("a: ! x\n", "yaml_tag_forbidden"),
("a: !local x\n", "yaml_tag_forbidden"),
("!!map {a: 1}\n", "yaml_tag_forbidden"),
# directives and multiple documents
("%YAML 1.1\n---\na: 1\n", "yaml_directive_forbidden"),
("%TAG !e! tag:example.com,2000:\n---\na: 1\n", "yaml_directive_forbidden"),
("a: 1\n---\nb: 2\n", "yaml_multiple_documents"),
("", "yaml_empty"),
("# only a comment\n", "yaml_empty"),
# keys
("a: 1\na: 2\n", "duplicate_key"),
("rules:\n- key: x\n key: y\n", "duplicate_key"),
("1: a\n", "non_string_key"),
("true: a\n", "non_string_key"),
("? [a]\n: b\n", "non_string_key"),
("null: a\n", "non_string_key"),
# structure limits
("a: " + "[" * 20 + "]" * 20 + "\n", "too_deep"),
("a: x\0y\n", "yaml_syntax"),
("a: 'x\x1b[31m'\n", "yaml_syntax"),
("a: [1, 2\n", "yaml_syntax"),
("a:\n\t- b\n", "yaml_syntax"),
])
def test_hostile_yaml_is_refused(raw, code):
assert _code(raw) == ("CONFIG_INVALID", code)
def test_errors_name_line_and_path():
with pytest.raises(ConfigError) as info:
parse(HEAD + RULE + " targets:\n hosts: [a]\n hosts: [b]\n")
error = info.value.errors[0]
assert error["code"] == "duplicate_key" and error["line"] == 8 and error["path"] == "rules[0].targets"
with pytest.raises(ConfigError) as info:
parse(HEAD + RULE + " destination_port: [1]\n secret_ref: x\n")
by_code = {e["code"]: e for e in info.value.errors}
assert by_code["unknown_field"]["path"] == "rules[0].secret_ref" and by_code["unknown_field"]["line"] == 7
assert by_code["unknown_field"]["params"] == {"field": "secret_ref"}
assert by_code["bad_type"]["path"].startswith("rules[0].destination_port") and by_code["bad_type"]["line"] == 6
def test_yaml_1_1_surprises_stay_text_or_are_type_errors():
data, _ = load_yaml("a: on\nb: yes\nc: 1.5\nd: 2026-10-22\ne: 0755\nf: 1:30\ng: 0x1F\nh: .inf\ni: Off\nj: 1e3\n")
assert data == {"a": "on", "b": "yes", "c": "1.5", "d": "2026-10-22", "e": "0755", "f": "1:30", "g": "0x1F",
"h": ".inf", "i": "Off", "j": "1e3"}
data, _ = load_yaml("a: true\nb: False\nc: null\nd: ~\ne:\nf: 53\ng: -1\nh: 0\n")
assert data == {"a": True, "b": False, "c": None, "d": None, "e": None, "f": 53, "g": -1, "h": 0}
assert "enabled" in _field_error(RULE + " enabled: yes\n") # strict bool: "yes" is not true
def _field_error(body: str) -> str:
with pytest.raises(ConfigError) as info:
parse(HEAD + body)
return " ".join(e["path"] for e in info.value.errors)
def test_huge_integers_stay_text_and_do_not_hit_pythons_int_limit():
data, _ = load_yaml("a: " + "9" * 5000 + "\n")
assert data["a"] == "9" * 5000
@pytest.mark.parametrize("raw,code", [
(b"a: \xff\xfe\n", "bad_encoding"),
(b"a: \xc3\x28\n", "bad_encoding"),
])
def test_invalid_utf8_is_refused(raw, code):
assert _code(raw) == ("CONFIG_INVALID", code)
def test_a_bom_is_accepted():
doc = parse("\ufeff" + HEAD).doc
assert doc.rules is None
def test_size_limits():
assert _code(b"#" * (loader.MAX_BYTES + 1)) == ("CONFIG_TOO_LARGE", "too_large")
assert _code("a: " + "x" * (loader.MAX_SCALAR + 1) + "\n") == ("CONFIG_INVALID", "scalar_too_long")
many = "a: [" + ",".join(["1"] * (loader.MAX_NODES + 1)) + "]\n"
assert _code(many) == ("CONFIG_INVALID", "too_many_nodes")
def test_object_and_entry_limits():
rules = "rules:\n" + "".join(f"- {{key: r-{i}, description: d, interface: lan}}\n" for i in range(loader.MAX_OBJECTS_PER_KIND + 1))
assert "too_many_objects" in _codes(HEAD + rules)
entries = "aliases:\n- name: A\n entries: [" + ",".join(f"10.0.{i // 250}.{i % 250}" for i in range(loader.MAX_ALIAS_ENTRIES + 1)) + "]\n"
assert "too_many_entries" in _codes(HEAD + entries)
long_entries = "aliases:\n- name: A\n entries: [" + ",".join(["x" * 999] * 9) + "]\n"
assert "too_long" in _codes(HEAD + long_entries)
variables = "variables:\n fw:\n" + "".join(f" V{i}: x\n" for i in range(loader.MAX_VARIABLES_PER_HOST + 1))
assert "too_many_objects" in _codes(HEAD + variables)
@pytest.mark.parametrize("raw,code,top", [
("- a\n", "not_a_mapping", "CONFIG_INVALID"),
("rules: []\n", "missing_version", "CONFIG_INVALID"),
("opnexus_config: '1'\n", "missing_version", "CONFIG_INVALID"),
("opnexus_config: true\n", "missing_version", "CONFIG_INVALID"),
("opnexus_config: 2\n", "unsupported_version", "CONFIG_UNSUPPORTED_VERSION"),
])
def test_version_and_shape(raw, code, top):
assert _code(raw) == (top, code)
def test_error_values_are_shortened_and_escaped():
with pytest.raises(ConfigError) as info:
parse(HEAD + RULE + " direction: \"" + "\\u001b[2J" + "x" * 200 + "\"\n")
value = info.value.errors[0]["params"]["value"]
assert len(value) <= 80 and "\x1b" not in value and value.startswith("\\x1b")
def test_one_mebibyte_parses_in_reasonable_time():
"""Pure-Python PyYAML; design risk (4). A full-size file of realistic rules must stay well under the request timeout."""
line = "- {key: r-%05d, description: Regel %05d mit etwas Text, interface: lan, source_net: 10.0.0.0/24, destination_port: '443'}\n"
body = HEAD + "rules:\n"
i = 0
while len(body) + len(line) < loader.MAX_BYTES - 200 and i < loader.MAX_OBJECTS_PER_KIND:
body += line % (i, i)
i += 1
body += "aliases:\n- name: BIG\n entries:\n" + "".join(f" - 10.{n // 65536}.{n // 256 % 256}.{n % 256}\n" for n in range(loader.MAX_ALIAS_ENTRIES))
raw = body.encode()[: loader.MAX_BYTES]
started = time.monotonic()
try:
parse(raw)
except ConfigError:
pass # truncation may cut the last line; the parse time is what counts
elapsed = time.monotonic() - started
print(f"parse {len(raw)} bytes: {elapsed:.2f}s")
assert elapsed < 8
+125
View File
@@ -0,0 +1,125 @@
"""Schema v1 (1.32 P3): strict, equal to the API limits, JSON Schema file up to date."""
import typing
from pathlib import Path
import pytest
from pydantic import ValidationError
from app.config_code import schema
from app.schemas.alias import AliasCreate
from app.schemas.one_to_one import OneToOneCreate
from app.schemas.outbound_nat import OutboundNatCreate
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
from app.schemas.port_forward import PortForwardCreate
from app.schemas.rule import RuleCreate
# (file model, API model, fields that exist only on one side)
PAIRS = [
(schema.RuleDoc, RuleCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids"}),
(schema.PortForwardDoc, PortForwardCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
(schema.OneToOneDoc, OneToOneCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
(schema.OutboundNatDoc, OutboundNatCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
(schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, {"key", "group"}, {"hierarchy_group_id"}),
(schema.AliasDoc, AliasCreate, {"entries", "targets"}, {"content", "target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
]
@pytest.mark.parametrize("doc,api,only_doc,only_api", PAIRS, ids=[p[0].__name__ for p in PAIRS])
def test_file_model_has_the_same_fields_defaults_and_limits_as_the_api_model(doc, api, only_doc, only_api):
assert set(doc.model_fields) - only_doc == set(api.model_fields) - only_api
for name in set(doc.model_fields) - only_doc:
a, b = doc.model_fields[name], api.model_fields[name]
assert a.default == b.default, name
assert _limits(a) == _limits(b), name
def _limits(field) -> dict:
"""Length/pattern limits of a field, also when they sit on the str branch of `str | None`."""
metas = list(field.metadata)
for arg in typing.get_args(field.annotation):
metas += list(getattr(arg, "__metadata__", ()))
metas = [m for meta in metas for m in (meta.metadata if hasattr(meta, "metadata") else [meta])] # unwrap Field(...)
out = {}
for meta in metas:
for attr in ("min_length", "max_length", "pattern"):
if getattr(meta, attr, None) is not None:
out[attr] = getattr(meta, attr)
return out
def test_enumerations_match_the_api():
for doc, api, name in ((schema.RuleDoc, RuleCreate, "action"), (schema.PortForwardDoc, PortForwardCreate, "natreflection"),
(schema.OneToOneDoc, OneToOneCreate, "natreflection"), (schema.OneToOneDoc, OneToOneCreate, "type"),
(schema.AliasDoc, AliasCreate, "type"), (schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, "rulebase")):
assert doc.model_fields[name].annotation == api.model_fields[name].annotation, (doc.__name__, name)
def _rule(**over):
return {"key": "r-1", "description": "d", "interface": "lan", **over}
def test_unknown_fields_wrong_types_and_secret_ref_are_refused():
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(secret_ref="x"))
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(enabled="yes")) # strict: no truthy strings
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(source_port=True)) # bool is never a port
with pytest.raises(ValidationError):
schema.ConfigDoc.model_validate({"opnexus_config": 2})
def test_ports_accept_numbers_and_ranges_and_become_strings():
assert schema.RuleDoc.model_validate(_rule(destination_port=53)).destination_port == "53"
assert schema.RuleDoc.model_validate(_rule(destination_port="8000-8100")).destination_port == "8000-8100"
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(destination_port="1" * 33))
@pytest.mark.parametrize("bad", ["a\x00b", "a\x1b[31m", "a\x7fb", "a\rb", "a\x0bb"])
def test_control_characters_are_refused(bad):
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(description=bad))
with pytest.raises(ValidationError):
schema.AliasDoc.model_validate({"name": "A", "entries": [bad]})
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(interface=bad))
def test_descriptions_may_have_several_lines_and_padding_but_names_and_keys_not():
assert schema.RuleDoc.model_validate(_rule(description="a\n\tb ")).description == "a\n\tb "
assert schema.DeviceGroupDoc.model_validate({"name": "G", "description": " zwei\nZeilen "}).description == " zwei\nZeilen "
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(interface="a\nb")) # a newline only in free text
for bad in (" G", "G "):
with pytest.raises(ValidationError):
schema.DeviceGroupDoc.model_validate({"name": bad})
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(targets={"hosts": [" fw"]}))
@pytest.mark.parametrize("key", ["", "Upper", "-lead", "a b", "x" * 65, "ü"])
def test_keys_must_be_lowercase_slugs(key):
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(key=key))
def test_variable_names_and_values_use_the_same_rules_as_the_host_page():
ok = {"opnexus_config": 1, "variables": {"fw": {"LAN_NET": "10.0.0.0/24"}}}
assert schema.ConfigDoc.model_validate(ok).variables == {"fw": {"LAN_NET": "10.0.0.0/24"}}
for bad in ({"lan": "x"}, {"LAN": "{{X}}"}, {"LAN": "a\nb"}, {"LAN": ""}):
with pytest.raises(ValidationError):
schema.ConfigDoc.model_validate({"opnexus_config": 1, "variables": {"fw": bad}})
def test_a_missing_section_is_not_an_empty_section():
doc = schema.ConfigDoc.model_validate({"opnexus_config": 1, "aliases": []})
assert doc.aliases == [] and doc.rules is None
def test_checked_in_json_schema_is_current():
path = Path(schema.__file__).with_name(schema.SCHEMA_FILE)
assert path.read_text(encoding="utf-8") == schema.json_schema_text(), (
"Schema geändert: `docker compose run --rm --no-deps -v \"$PWD/backend:/app\" backend python -m app.config_code schema "
f"> backend/app/config_code/{schema.SCHEMA_FILE}`"
)
+266
View File
@@ -0,0 +1,266 @@
"""Way back to the previous firmware (1.32, B5): rollback point before every update, guided rollback."""
import uuid
from datetime import datetime, timedelta, timezone
import httpx
import pytest
from sqlalchemy import select
from app.database import SessionLocal
from app.models.host_operation import HostOperation
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import backup_store, scheduled_maintenance_service
from tests.test_maintenance import execute, preview
from tests.test_rules import _create_host
from tests.test_scheduled_maintenance import _due_update
BASE = "https://fw1.test"
class Snapshots:
"""A firewall with ZFS boot environments, as Core/Api/SnapshotsController answers."""
def __init__(self, mock, *, version="26.7.6", status="update", fail_add=False, hide_new=False, fail_reboot=False,
mark_on_activate=True):
self.rows = [{"uuid": "be-default", "name": "default", "active": "NR", "created_str": "2026-10-01 10:00"}]
self.fail_add, self.hide_new, self.fail_reboot, self.mark_on_activate = fail_add, hide_new, fail_reboot, mark_on_activate
self.activated: list[str] = []
mock.get(f"{BASE}/api/diagnostics/interface/getVipStatus").respond(200, json={"rows": []})
mock.get(f"{BASE}/api/core/firmware/info").respond(200, json={"product_version": version})
mock.get(f"{BASE}/api/core/firmware/status").respond(200, json={"status": status, "all_packages": {}, "status_msg": "ok"})
mock.get(f"{BASE}/api/core/firmware/running").respond(200, json={"status": "ready"})
mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": True})
self.fail_list = False
mock.get(f"{BASE}/api/core/snapshots/search").mock(side_effect=self._search)
self.add = mock.post(f"{BASE}/api/core/snapshots/add").mock(side_effect=self._add)
self.delete = mock.post(url__regex=rf"{BASE}/api/core/snapshots/del/.*").mock(side_effect=self._del)
self.activate = mock.post(url__regex=rf"{BASE}/api/core/snapshots/activate/.*").mock(side_effect=self._activate)
self.update = mock.post(f"{BASE}/api/core/firmware/update").respond(200, json={"status": "ok"})
self.upgrade = mock.post(f"{BASE}/api/core/firmware/upgrade").respond(200, json={"status": "ok"})
self.reboot = mock.post(f"{BASE}/api/core/system/reboot").mock(side_effect=self._reboot)
self.download = mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
def _search(self, request):
if self.fail_list:
return httpx.Response(502, json={})
return httpx.Response(200, json={"rows": self.rows})
def point(self, name: str, active: str = "-") -> None:
self.rows.append({"uuid": f"be-{name}", "name": name, "active": active, "created_str": "x"})
def _add(self, request):
if self.fail_add:
return httpx.Response(200, json={"status": "failed", "validations": {"snapshot.name": "nope"}})
import json
name = json.loads(request.content)["name"]
if not self.hide_new:
self.point(name)
return httpx.Response(200, json={"status": "ok"})
def _del(self, request):
uid = str(request.url).rsplit("/", 1)[-1]
self.rows = [r for r in self.rows if r["uuid"] != uid]
return httpx.Response(200, json={"status": "ok"})
def _activate(self, request):
uid = str(request.url).rsplit("/", 1)[-1]
self.activated.append(uid)
if getattr(self, "fail_list_after_activate", False) and len(self.activated) == 1:
self.fail_list = True
if self.mark_on_activate:
for r in self.rows:
r["active"] = ("N" if "N" in r["active"] else "") + ("R" if r["uuid"] == uid else "")
r["active"] = r["active"] or "-"
return httpx.Response(200, json={"status": "ok"})
def _reboot(self, request):
if self.fail_reboot:
return httpx.Response(500, json={})
return httpx.Response(200, json={"status": "ok"})
async def _host_with_backup(client):
h = await _create_host(client, "fw1", BASE)
backup_store.save_snapshot(uuid.UUID(h["id"]), b"<opnsense/>")
return h
async def _op(op_id) -> HostOperation:
async with SessionLocal() as db:
return await db.get(HostOperation, uuid.UUID(str(op_id)))
async def test_a_manual_update_creates_a_rollback_point_first_and_keeps_only_two(client, respx_mock):
fw = Snapshots(respx_mock)
fw.point("opnexus-pre-26.7.4-20261001-100000")
fw.point("opnexus-pre-26.7.5-20261010-100000")
fw.point("handmade") # never touched
h = await _host_with_backup(client)
p = await preview(client, h, "update")
r = await execute(client, h, p.json())
assert r.json()["status"] == "submitted" and fw.update.call_count == 1
names = [row["name"] for row in fw.rows]
new = next(n for n in names if n.startswith("opnexus-pre-26.7.6-"))
assert "opnexus-pre-26.7.4-20261001-100000" not in names # the oldest of three is cleaned up
assert {"default", "handmade", "opnexus-pre-26.7.5-20261010-100000", new} == set(names)
op = await _op(r.json()["id"])
assert op.snapshot["rollback_point"]["name"] == new and op.snapshot["rollback_point"]["version"] == "26.7.6"
assert fw.add.call_count == 1
order = [str(c.request.url) for c in respx_mock.calls if c.request.method == "POST"]
assert order.index(f"{BASE}/api/core/snapshots/add") < order.index(f"{BASE}/api/core/firmware/update")
@pytest.mark.parametrize("problem", ["fail_add", "hide_new"])
async def test_no_rollback_point_no_update(client, respx_mock, problem):
fw = Snapshots(respx_mock, **{problem: True})
h = await _host_with_backup(client)
p = await preview(client, h, "update")
r = await execute(client, h, p.json())
assert r.json()["status"] == "cancelled" and "Update nicht gestartet" in r.json()["message"]
assert fw.update.call_count == 0
async def test_without_zfs_the_update_runs_and_says_there_is_no_way_back(client, respx_mock):
fw = Snapshots(respx_mock)
respx_mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": False})
h = await _host_with_backup(client)
r = await execute(client, h, (await preview(client, h, "update")).json())
assert r.json()["status"] == "submitted" and "Kein Rückweg-Punkt" in r.json()["message"]
assert fw.add.call_count == 0 and fw.update.call_count == 1
async def test_a_scheduled_update_fails_closed_without_a_rollback_point(client, respx_mock):
fw = Snapshots(respx_mock, fail_add=True)
respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
h = await _create_host(client, "fw1", BASE)
sched_id = await _due_update(h)
async with SessionLocal() as db:
await scheduled_maintenance_service.fire_due(db)
async with SessionLocal() as db:
sched = await db.get(ScheduledMaintenance, sched_id)
assert sched.status == "failed" and "Rückweg-Punkt" in sched.result["error"]
assert fw.update.call_count == 0
async def test_rollback_needs_a_point_from_opnexus(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.point("handmade")
h = await _create_host(client, "fw1", BASE)
r = await preview(client, h, "firmware_rollback")
assert r.status_code == 409 and "Kein Rückweg-Punkt" in r.json()["detail"]["params"]["reason"]
assert (await client.post(f"/api/hosts/{h['id']}/snapshots", json={"name": "opnexus-pre-1-20260101-000000"})).status_code == 422
async def test_rollback_activates_the_newest_point_reboots_and_lists_what_is_lost(client, respx_mock):
fw = Snapshots(respx_mock, version="26.7.7", status="none")
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
fw.point(f"opnexus-pre-26.7.5-{(stamp - timedelta(days=5)):%Y%m%d-%H%M%S}")
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
h = await _create_host(client, "fw1", BASE)
async with SessionLocal() as db:
rule = FirewallRule(description="nachher", interface="lan", direction="in")
db.add(rule)
await db.flush()
db.add(RuleDeployment(rule_id=rule.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u"))
await db.commit()
p = await preview(client, h, "firmware_rollback")
assert p.status_code == 200, p.text
rollback = p.json()["state"]["rollback"]
assert rollback["target"]["version"] == "26.7.6" and rollback["current"]["name"] == "default"
assert [c["kind"] for c in rollback["changes_since"]] == ["rule"]
r = await execute(client, h, p.json())
assert r.json()["status"] == "submitted" and "erwartet 26.7.6" in r.json()["message"]
assert fw.activated == [rollback["target"]["uuid"]] and fw.reboot.call_count == 1
async def test_rollback_that_is_not_marked_for_the_next_start_never_reboots(client, respx_mock):
fw = Snapshots(respx_mock, status="none", mark_on_activate=False)
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
async def test_a_failed_reboot_reactivates_the_current_boot_environment(client, respx_mock):
fw = Snapshots(respx_mock, status="none", fail_reboot=True)
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "unknown"
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
assert next(row for row in fw.rows if row["uuid"] == "be-default")["active"].endswith("R")
async def test_rollback_on_a_carp_master_is_refused_like_a_reboot(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
r = await preview(client, h, "firmware_rollback")
assert r.status_code == 409 and fw.activate.call_count == 0
async def test_the_rollback_endpoint_needs_the_deploy_right_like_every_maintenance_execute():
from app.services import api_token_service
assert api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/execute")
assert not api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/preview")
async def test_rollback_saves_the_current_configuration_first_and_refuses_without_it(client, respx_mock):
"""Going back discards every change since the point on the firewall: OPNexus keeps a copy of the config first."""
fw = Snapshots(respx_mock, status="none")
fw.point("opnexus-pre-26.7.6-20261020-100000")
download = respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense>now</opnsense>")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "submitted" and download.call_count == 1
assert backup_store.list_snapshots(uuid.UUID(h["id"])) # stored locally
order = [str(c.request.url) for c in respx_mock.calls]
assert order.index(f"{BASE}/api/core/backup/download/this") < order.index(f"{BASE}/api/core/snapshots/activate/be-opnexus-pre-26.7.6-20261020-100000")
async def test_rollback_without_a_configuration_copy_does_not_start(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.point("opnexus-pre-26.7.6-20261020-100000")
respx_mock.get(f"{BASE}/api/core/backup/download/this").mock(side_effect=httpx.ConnectError("down"))
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "cancelled" and fw.activate.call_count == 0 and fw.reboot.call_count == 0
async def test_an_unreadable_list_after_activating_switches_back_and_never_reboots(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.fail_list_after_activate = True
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
async def test_after_a_rollback_newer_deployments_are_marked_so_the_next_deploy_really_writes(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
h = await _create_host(client, "fw1", BASE)
async with SessionLocal() as db:
old, new = FirewallRule(description="alt", interface="lan", direction="in"), FirewallRule(description="neu", interface="lan", direction="in")
db.add_all([old, new])
await db.flush()
db.add_all([
RuleDeployment(rule_id=old.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u1",
last_payload={"x": 1}, deployed_at=stamp - timedelta(days=1)),
RuleDeployment(rule_id=new.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u2",
last_payload={"x": 2}, deployed_at=stamp + timedelta(minutes=30)),
])
await db.commit()
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "submitted" and "1 Deployment(s) als unklar markiert" in r.json()["message"]
async with SessionLocal() as db:
rows = {d.opnsense_rule_uuid: d for d in (await db.scalars(select(RuleDeployment))).all()}
assert rows["u1"].status == "success" and rows["u1"].last_payload == {"x": 1} # older than the point: untouched
assert rows["u2"].status == "error" and rows["u2"].last_payload is None and "Firmware-Rückweg" in rows["u2"].message
+1
View File
@@ -14,6 +14,7 @@ def mock_reads(mock,base='https://fw1.test',carp=None,status='none',running='rea
mock.get(f'{base}/api/core/firmware/info').respond(200,json={'product_version':version})
mock.get(f'{base}/api/core/firmware/status').respond(200,json={'status':status,'all_packages':{},'status_msg':'cached result'})
mock.get(f'{base}/api/core/firmware/running').respond(200,json={'status':running})
mock.get(f'{base}/api/core/snapshots/isSupported').respond(200,json={'supported':False}) # UFS: no rollback point (1.32)
async def preview(client,h,action='reboot'):
return await client.post(f"/api/hosts/{h['id']}/maintenance/preview",json={'action':action})
+111 -1
View File
@@ -378,7 +378,7 @@ async def test_0028_adds_the_event_window_and_is_idempotent(scratch):
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'rule_change_events'"))).scalars().all())
assert "window_start" not in cols
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() == "0028" and await _schema_diff(scratch) == []
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
await _alembic(scratch, "downgrade", "0027")
async with scratch.connect() as conn:
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'rule_change_events'"))).scalars().all())
@@ -386,3 +386,113 @@ async def test_0028_adds_the_event_window_and_is_idempotent(scratch):
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async def test_0029_creates_the_rule_cleanup_tables_and_settings_and_is_idempotent(scratch):
"""1.30 -> 1.31: two tables with a partial unique index (one unfinished item per rule and node) and three
columns on governance_settings; the downgrade removes all of it again."""
await _alembic(scratch, "upgrade", "0028")
async with scratch.begin() as conn:
await conn.execute(text("INSERT INTO governance_settings (id, operators_may_deploy) VALUES (1, true)"))
tables = set((await conn.execute(text("SELECT tablename FROM pg_tables WHERE schemaname = 'public'"))).scalars().all())
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'governance_settings'"))).scalars().all())
assert "rule_cleanup_operations" not in tables and "rule_cleanup_items" not in tables
assert "cleanup_max_rules" not in cols
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async with scratch.connect() as conn: # the existing settings row got the defaults of the design
row = (await conn.execute(text(
"SELECT cleanup_max_rules, cleanup_delete_min_days, cleanup_management_sources, cleanup_management_ports "
"FROM governance_settings"))).one()
assert tuple(row) == (10, 7, [], [])
host, op_id = "11111111-1111-1111-1111-111111111111", "22222222-2222-2222-2222-222222222222"
async with scratch.begin() as conn:
await conn.execute(text(
"INSERT INTO opnsense_hosts (id, name, base_url, api_key, verify_tls, enabled, created_at) "
"VALUES (:h, 'fw', 'https://fw.test', 'k', true, true, now())"), {"h": host})
await conn.execute(text(
"INSERT INTO rule_cleanup_operations (id, host_id, kind, status, actor, created_at, expires_at, preview_fingerprint, options) "
"VALUES (:o, :h, 'disable', 'preview', 'admin', now(), now(), 'f', '{}')"), {"o": op_id, "h": host})
async def item(conn, rule: str, state: str):
await conn.execute(text(
"INSERT INTO rule_cleanup_items (id, operation_id, host_id, rule_uuid, seq, state, description, categories) "
f"VALUES (gen_random_uuid(), :o, :h, '{rule}', 0, '{state}', 'r', '[]')"), {"o": op_id, "h": host})
async with scratch.begin() as conn:
await item(conn, "r1", "disabled")
await item(conn, "r2", "disabled") # another rule is fine
await item(conn, "r1", "confirmed") # finished items do not count
await item(conn, "r1", "reverted")
for blocked in ("queued", "unknown", "revert_failed", "deleting"): # r1 is still open as "disabled"
with pytest.raises(DBAPIError):
async with scratch.begin() as conn:
await item(conn, "r1", blocked)
async with scratch.begin() as conn: # on another node the same UUID is a different rule
await conn.execute(text(
"INSERT INTO opnsense_hosts (id, name, base_url, api_key, verify_tls, enabled, created_at) "
"VALUES ('33333333-3333-3333-3333-333333333333', 'fw2', 'https://fw2.test', 'k', true, true, now())"))
await conn.execute(text(
"INSERT INTO rule_cleanup_items (id, operation_id, host_id, rule_uuid, seq, state, description, categories) "
"VALUES (gen_random_uuid(), :o, '33333333-3333-3333-3333-333333333333', 'r1', 0, 'disabled', 'r', '[]')"), {"o": op_id})
async with scratch.begin() as conn: # the host takes its operations and their items with it
await conn.execute(text("DELETE FROM opnsense_hosts WHERE id = :h"), {"h": host})
assert (await conn.execute(text("SELECT count(*) FROM rule_cleanup_operations"))).scalar() == 0
assert (await conn.execute(text("SELECT count(*) FROM rule_cleanup_items"))).scalar() == 0
await _alembic(scratch, "downgrade", "0028")
async with scratch.connect() as conn:
tables = set((await conn.execute(text("SELECT tablename FROM pg_tables WHERE schemaname = 'public'"))).scalars().all())
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'governance_settings'"))).scalars().all())
assert "rule_cleanup_operations" not in tables and "rule_cleanup_items" not in tables
assert not {"cleanup_max_rules", "cleanup_delete_min_days", "cleanup_management_sources", "cleanup_management_ports"} & cols
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head") # second run: nothing to do
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async def test_0030_backfills_config_keys_and_is_idempotent(scratch):
"""1.31 -> 1.32: every rule/NAT table gets a unique, readable key derived from the UUID; the downgrade drops it."""
await _alembic(scratch, "upgrade", "0029")
rule_id = "3f2a9c1d-0000-4000-8000-000000000001"
async with scratch.begin() as conn:
await conn.execute(text(
"INSERT INTO firewall_rules (id, description, interface, direction, ipprotocol, protocol, action, "
"source_net, destination_net, enabled, log, created_at, updated_at) VALUES "
f"('{rule_id}', 'r', 'lan', 'in', 'inet', 'any', 'pass', 'any', 'any', true, false, now(), now())"))
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async with scratch.connect() as conn:
key = (await conn.execute(text("SELECT config_key FROM firewall_rules"))).scalar_one()
nullable = (await conn.execute(text(
"SELECT is_nullable FROM information_schema.columns "
"WHERE table_name = 'port_forward_rules' AND column_name = 'config_key'"))).scalar_one()
assert key == "r-3f2a9c1d0000" and nullable == "NO"
await _alembic(scratch, "downgrade", "0029")
async with scratch.connect() as conn:
cols = set((await conn.execute(text(
"SELECT column_name FROM information_schema.columns WHERE table_name = 'firewall_rules'"))).scalars().all())
assert "config_key" not in cols
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async def test_0031_adds_gate_probes_to_existing_plans_and_is_idempotent(scratch):
await _alembic(scratch, "upgrade", "0030")
async with scratch.begin() as conn:
await conn.execute(text(
"INSERT INTO rollout_plans (id, name, description, action, auto_start, gate_checks, compliance_mode, created_by, created_at, updated_at) "
"VALUES (gen_random_uuid(), 'p', '', 'update', false, '{}', 'warn', 'admin', now(), now())"))
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async with scratch.connect() as conn:
assert (await conn.execute(text("SELECT gate_probes::text FROM rollout_plans"))).scalar_one() == "[]"
await _alembic(scratch, "downgrade", "0030")
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
+16
View File
@@ -0,0 +1,16 @@
import subprocess
import sys
def test_importing_models_alone_configures_all_mappers():
"""`python -m app.reset_password` / `app.reset_2fa` import only part of the app: the mapper registry must still be complete."""
code = "import app.models; from sqlalchemy.orm import configure_mappers; configure_mappers()"
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
assert result.returncode == 0, result.stderr[-800:]
def test_reset_scripts_import_cleanly_and_configure_mappers():
for module in ("app.reset_password", "app.reset_2fa"):
code = f"import {module}; from sqlalchemy.orm import configure_mappers; configure_mappers()"
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
assert result.returncode == 0, (module, result.stderr[-800:])
+2 -2
View File
@@ -146,9 +146,9 @@ def test_what_counts_as_a_rollout(method, path, rollout):
async def test_with_separation_of_duties_operators_prepare_but_only_admins_deploy(client):
op = await _operator(client, "dutyop")
try:
assert (await op.get("/api/settings/governance")).json() == {"operators_may_deploy": True}
assert (await op.get("/api/settings/governance")).json()["operators_may_deploy"] is True
assert (await op.put("/api/settings/governance", json={"operators_may_deploy": False})).status_code == 403
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": False})).json() == {"operators_may_deploy": False}
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": False})).json()["operators_may_deploy"] is False
# Preparing still works ...
assert (await op.post("/api/groups", json={"name": "prepared"})).status_code == 201
# ... rolling out does not, whatever the target (the 403 comes before any lookup).
+246
View File
@@ -0,0 +1,246 @@
"""Client calls for filter rules that the cleanup assistant (1.31) builds on: toggleRule, getRule, the
getRule -> addRule payload, and the request shape / failure classification of add/del/move/apply.
The JSON fixtures are shaped like live answers of OPNsense 26.7.7 (checked 2026-10-22 on a test firewall) with
documentation addresses and made-up UUIDs. Evidence for each endpoint: comment block in services/opnsense_client.py.
"""
import json
import uuid
import httpx
import pytest
from app.models.host import OpnsenseHost
from app.services import opnsense_client as c
BASE = "https://fw.test/api/firewall/filter"
RULE = "5c6f3bfa-1718-4216-81c8-04a75527960f"
OTHER = "74190f58-f35f-4402-8794-5410e99ea364"
@pytest.fixture
def host(monkeypatch):
monkeypatch.setattr(c.opnsense_secret_store, "read_secret", lambda host_id: "secret")
return OpnsenseHost(id=uuid.uuid4(), name="fw-a", base_url="https://fw.test", api_key="key", verify_tls=False)
def _opts(selected: list[str], options: dict[str, str]) -> dict:
return {k: {"value": v, "selected": 1 if k in selected else 0} for k, v in options.items()}
def live_rule() -> dict:
"""A `getRule/<uuid>` record: select fields as option dicts, `categories` an option dict, read-only extras."""
return {
"enabled": "1", "sequence": "566", "sort_order": "200000.0000566", "prio_group": "200000", "quick": "1",
"action": _opts(["pass"], {"pass": "Pass", "block": "Block", "reject": "Reject"}),
"interface": _opts(["lan", "wan"], {"lan": "LAN", "wan": "WAN", "wireguard": "WireGuard (Group)"}),
"direction": _opts(["in"], {"in": "In", "out": "Out", "any": "Both"}),
"protocol": _opts(["TCP"], {"any": "any", "TCP": "TCP", "UDP": "UDP"}),
"gateway": _opts([""], {"": "None", "WAN_GW": "WAN_GW - 203.0.113.1"}),
"categories": _opts(["c1", "c2"], {"c1": "one", "c2": "two", "c3": "three"}),
"tcpflags1": _opts([], {"syn": "SYN", "ack": "ACK"}),
"source_net": "bogons", "destination_net": "203.0.113.9", "destination_port": "8080-8443", "source_port": "",
"description": "opnexus-b4-probe", "statetimeout": None, "log": 1,
"audit": {"created": {"username": "", "time": "", "description": ""}, "userdata": {"note": ""}},
}
def _body(route) -> dict:
return json.loads(route.calls.last.request.content)
# ── toggle_rule ────────────────────────────────────────────────────────────
@pytest.mark.asyncio
@pytest.mark.parametrize("enabled, tail, answer", [(False, "0", "Disabled"), (True, "1", "Enabled")])
async def test_toggle_rule_sends_the_explicit_value_and_returns_the_answer(host, respx_mock, enabled, tail, answer):
route = respx_mock.post(f"{BASE}/toggleRule/{RULE}/{tail}").mock(
return_value=httpx.Response(200, json={"result": answer, "changed": True}))
assert await c.toggle_rule(host, RULE, enabled) == {"result": answer, "changed": True}
assert route.call_count == 1 and not route.calls.last.request.content # never the flipping form, no body
@pytest.mark.asyncio
async def test_toggle_rule_is_idempotent_an_unchanged_answer_is_success(host, respx_mock):
respx_mock.post(f"{BASE}/toggleRule/{RULE}/0").mock(
return_value=httpx.Response(200, json={"result": "Disabled", "changed": False}))
assert (await c.toggle_rule(host, RULE, False))["changed"] is False
@pytest.mark.asyncio
@pytest.mark.parametrize("bad", [0, 1, "0", "1", None, "false"])
async def test_toggle_rule_refuses_anything_but_a_bool_without_a_request(host, respx_mock, bad):
route = respx_mock.post(url__regex=r".*/toggleRule/.*").mock(return_value=httpx.Response(200, json={}))
with pytest.raises(TypeError):
await c.toggle_rule(host, RULE, bad)
assert route.call_count == 0
@pytest.mark.asyncio
async def test_toggle_rule_answer_for_the_other_state_is_not_accepted(host, respx_mock):
respx_mock.post(f"{BASE}/toggleRule/{RULE}/0").mock(return_value=httpx.Response(200, json={"result": "Enabled", "changed": True}))
with pytest.raises(c.OpnsenseAPIError) as info:
await c.toggle_rule(host, RULE, False)
assert not isinstance(info.value, c.OpnsenseAmbiguousError)
@pytest.mark.asyncio
@pytest.mark.parametrize("answer", [{"result": "failed"}, {"result": "failed", "changed": False}])
async def test_toggle_rule_unknown_uuid_is_a_plain_refusal(host, respx_mock, answer):
"""Live: an unknown or legacy UUID answers HTTP 200 {"result": "failed"} (no "changed" key)."""
respx_mock.post(f"{BASE}/toggleRule/{RULE}/0").mock(return_value=httpx.Response(200, json=answer))
with pytest.raises(c.OpnsenseAPIError) as info:
await c.toggle_rule(host, RULE, False)
assert type(info.value) is c.OpnsenseAPIError
@pytest.mark.asyncio
async def test_toggle_rule_failure_classes(host, respx_mock):
route = respx_mock.post(f"{BASE}/toggleRule/{RULE}/0")
route.mock(return_value=httpx.Response(403, json={"status": 403, "message": "Forbidden"}))
with pytest.raises(c.OpnsenseForbiddenError):
await c.toggle_rule(host, RULE, False)
route.mock(return_value=httpx.Response(500, text="boom"))
with pytest.raises(c.OpnsenseAmbiguousError): # the change may have been saved
await c.toggle_rule(host, RULE, False)
route.mock(side_effect=httpx.ReadTimeout("slow"))
with pytest.raises(c.OpnsenseAmbiguousError):
await c.toggle_rule(host, RULE, False)
route.mock(side_effect=httpx.ConnectError("down")) # never sent: a plain error, not an unknown outcome
with pytest.raises(c.OpnsenseAPIError) as info:
await c.toggle_rule(host, RULE, False)
assert not isinstance(info.value, c.OpnsenseAmbiguousError)
@pytest.mark.asyncio
@pytest.mark.parametrize("bad", ["../x", "a/b", "a?b", "", "x" * 65, "a b"])
async def test_rule_references_cannot_leave_the_url_path(host, respx_mock, bad):
route = respx_mock.route().mock(return_value=httpx.Response(200, json={}))
with pytest.raises(c.OpnsenseAPIError):
await c.toggle_rule(host, bad, False)
with pytest.raises(c.OpnsenseAPIError):
await c.get_rule(host, bad)
assert route.call_count == 0
# ── get_rule ───────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_get_rule_returns_the_inner_record(host, respx_mock):
route = respx_mock.get(f"{BASE}/getRule/{RULE}").mock(return_value=httpx.Response(200, json={"rule": live_rule()}))
rule = await c.get_rule(host, RULE)
assert route.call_count == 1 and rule["description"] == "opnexus-b4-probe" and rule["interface"]["lan"]["selected"] == 1
@pytest.mark.asyncio
@pytest.mark.parametrize("answer", [[], {}])
async def test_get_rule_empty_answer_means_not_found(host, respx_mock, answer):
"""Live: unknown, legacy (hex label) and automatic UUIDs answer HTTP 200 with a bare `[]`."""
respx_mock.get(f"{BASE}/getRule/{RULE}").mock(return_value=httpx.Response(200, json=answer))
with pytest.raises(c.OpnsenseNotFoundError):
await c.get_rule(host, RULE)
@pytest.mark.asyncio
async def test_get_rule_failure_classes(host, respx_mock):
route = respx_mock.get(f"{BASE}/getRule/{RULE}")
route.mock(return_value=httpx.Response(200, json={"rule": "nonsense"}))
with pytest.raises(c.OpnsenseAPIError) as info:
await c.get_rule(host, RULE)
assert not isinstance(info.value, c.OpnsenseNotFoundError) # an unreadable answer is not proof that the rule is gone
route.mock(return_value=httpx.Response(403, text="no"))
with pytest.raises(c.OpnsenseForbiddenError):
await c.get_rule(host, RULE)
route.mock(return_value=httpx.Response(502, text="bad gateway"))
with pytest.raises(c.OpnsenseAmbiguousError):
await c.get_rule(host, RULE)
# ── rule_write_payload (getRule -> addRule) ───────────────────────────────
def test_rule_write_payload_reduces_selects_and_drops_read_only_fields():
payload = c.rule_write_payload(live_rule())
assert payload["interface"] == "lan,wan" # multi-interface: comma separated
assert payload["categories"] == "c1,c2"
assert payload["action"] == "pass" and payload["protocol"] == "TCP" and payload["direction"] == "in"
assert payload["gateway"] == "" # the "None" option has the key ""
assert payload["tcpflags1"] == "" # nothing selected
assert payload["statetimeout"] == "" and payload["log"] == "1" and payload["source_port"] == ""
assert payload["enabled"] == "1" and payload["destination_port"] == "8080-8443"
assert not set(payload) & {"uuid", "sequence", "sort_order", "prio_group", "audit"}
assert all(isinstance(v, str) for v in payload.values())
def test_rule_write_payload_empty_category_list_becomes_an_empty_string():
"""Live: a rule without categories answers `"categories": []` (PHP's empty array), not an option dict."""
rule = live_rule() | {"categories": []}
assert c.rule_write_payload(rule)["categories"] == ""
def test_rule_write_payload_does_not_mutate_its_input():
rule = live_rule()
before = json.dumps(rule, sort_keys=True)
c.rule_write_payload(rule)
assert json.dumps(rule, sort_keys=True) == before
@pytest.mark.asyncio
async def test_a_copy_is_added_with_exactly_the_payload_and_stays_disabled(host, respx_mock):
"""The restore path: getRule -> payload -> addRule(enabled=0). The body must carry the payload unchanged."""
respx_mock.get(f"{BASE}/getRule/{RULE}").mock(return_value=httpx.Response(200, json={"rule": live_rule()}))
add = respx_mock.post(f"{BASE}/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": OTHER}))
payload = c.rule_write_payload(await c.get_rule(host, RULE)) | {"enabled": "0"}
assert (await c.add_rule(host, payload))["uuid"] == OTHER
sent = _body(add)["rule"]
assert sent == payload and sent["enabled"] == "0" and "sequence" not in sent and "uuid" not in sent
# ── add / set / del / move / apply: request shape and classification ─────
@pytest.mark.asyncio
async def test_add_rule_validation_failure_is_a_plain_refusal(host, respx_mock):
respx_mock.post(f"{BASE}/addRule").mock(return_value=httpx.Response(
200, json={"result": "failed", "validations": {"rule.destination_port": "Please specify a valid portnumber"}}))
with pytest.raises(c.OpnsenseAPIError) as info:
await c.add_rule(host, {"destination_port": "1,2"})
assert type(info.value) is c.OpnsenseAPIError and "destination_port" in str(info.value)
@pytest.mark.asyncio
@pytest.mark.parametrize("answer", ["deleted", "not found"])
async def test_del_rule_accepts_deleted_and_not_found(host, respx_mock, answer):
route = respx_mock.post(f"{BASE}/delRule/{RULE}").mock(return_value=httpx.Response(200, json={"result": answer}))
assert (await c.del_rule(host, RULE))["result"] == answer and route.call_count == 1
@pytest.mark.asyncio
async def test_del_rule_failed_answer_and_5xx(host, respx_mock):
route = respx_mock.post(f"{BASE}/delRule/{RULE}")
route.mock(return_value=httpx.Response(200, json={"result": "failed"}))
with pytest.raises(c.OpnsenseAPIError):
await c.del_rule(host, RULE)
route.mock(return_value=httpx.Response(500, text="x"))
with pytest.raises(c.OpnsenseAmbiguousError):
await c.del_rule(host, RULE)
@pytest.mark.asyncio
async def test_move_rule_before_url_and_answer(host, respx_mock):
route = respx_mock.post(f"{BASE}/moveRuleBefore/{RULE}/{OTHER}").mock(return_value=httpx.Response(200, json={"status": "ok"}))
assert await c.move_rule_before(host, RULE, OTHER) == {"status": "ok"} and route.call_count == 1
@pytest.mark.asyncio
async def test_move_rule_before_refusal_is_http_500_and_therefore_ambiguous(host, respx_mock):
"""Live: unknown UUID, the same UUID twice and rules of different prio_groups all answer HTTP 500
{"errorMessage": "Unexpected error, check log for details"}. Callers must read the position back."""
respx_mock.post(f"{BASE}/moveRuleBefore/{RULE}/{OTHER}").mock(return_value=httpx.Response(
500, json={"errorMessage": "Unexpected error, check log for details", "errorTitle": "Filter", "errorLevel": "warning"}))
with pytest.raises(c.OpnsenseAmbiguousError):
await c.move_rule_before(host, RULE, OTHER)
@pytest.mark.asyncio
async def test_apply_url_and_answer(host, respx_mock):
route = respx_mock.post(f"{BASE}/apply").mock(return_value=httpx.Response(200, json={"status": "OK\n\n"}))
assert (await c.apply(host))["status"].strip() == "OK" and route.call_count == 1
+26
View File
@@ -1,4 +1,6 @@
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
import uuid
import pytest
from tests.test_maintenance_windows import _WEEKLY
@@ -145,3 +147,27 @@ async def test_role_matrix_for_plans(client):
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
finally:
await ro.aclose()
@pytest.mark.asyncio
async def test_gate_probes_are_validated_stored_and_frozen_into_the_fingerprint(client):
"""1.32 (B5): probes per plan; a plan without probes keeps the fingerprint of earlier versions."""
from app.database import SessionLocal
from app.models.rollout import RolloutPlan
from app.services import rollout_service
_hosts, groups = await _fleet(client)
plain = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
assert plain["gate_probes"] == []
async with SessionLocal() as db:
assert "gate_probes" not in rollout_service.definition(await db.get(RolloutPlan, uuid.UUID(plain["id"])))
probes = [{"type": "ping", "target": "1.1.1.1"}, {"type": "tcp", "target": "intranet.example", "port": 443},
{"type": "ping", "target": "1.1.1.1"}]
resp = await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=probes))
assert resp.status_code == 200, resp.text
assert resp.json()["gate_probes"] == [{"type": "ping", "target": "1.1.1.1", "port": None},
{"type": "tcp", "target": "intranet.example", "port": 443}]
for bad in ([{"type": "tcp", "target": "x"}], [{"type": "ping", "target": "x", "port": 1}],
[{"type": "ping", "target": "-rf"}], [{"type": "ping", "target": "a b"}], [{"type": "udp", "target": "x"}],
[{"type": "ping", "target": f"h{i}"} for i in range(11)]):
assert (await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=bad))).status_code == 422, bad
+26
View File
@@ -31,6 +31,7 @@ class Fw:
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
r, b = fleet.mock, self.base
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
r.get(f"{b}/api/core/snapshots/isSupported").mock(side_effect=lambda q: self._ok({"supported": False}))
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
@@ -711,3 +712,28 @@ async def test_rows_without_an_account_id_still_work_by_name(client, respx_mock,
assert await engine.may_roll_out(db, "admin") is True
assert await engine.may_roll_out(db, "nobody-here") is False
assert await engine.may_roll_out(db, "admin", uuid.uuid4()) is False # a given id that does not exist is final
@pytest.mark.asyncio
async def test_a_probe_that_breaks_after_the_update_stops_the_rollout(client, respx_mock, clock, monkeypatch):
"""1.32 (B5): the plan's probes are frozen into the run, taken in the baseline and compared in the gate."""
from app.services import diagnostics_service
reachable = {"ok": True}
calls = []
async def portprobe(host, target, port, ipproto, source):
calls.append(host.name)
return {"open": reachable["ok"] or host.name != "c1"}
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, gate_probes=[{"type": "tcp", "target": "10.0.0.5", "port": 443}])
run = await _start(client, plan)
assert (await _run(run["id"])).plan_snapshot["gate_probes"] == [{"type": "tcp", "target": "10.0.0.5", "port": 443}]
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
assert "c1" in calls # baseline
reachable["ok"] = False # the update broke the path to the server
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "gate_failed"
assert paused.waves[0].gate_result["hosts"][str(_host(paused, "c1").host_id)]["probes"]["code"] == "probe_regressed"
assert fleet.fw["ha-b"].update.call_count == 0
+44
View File
@@ -193,3 +193,47 @@ async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeyp
def datetime_now():
from datetime import datetime, timezone
return datetime.now(timezone.utc)
@pytest.mark.asyncio
async def test_probes_from_the_firewall_compare_against_the_baseline(client, respx_mock, monkeypatch):
"""1.32 (B5): reachability probes. Worked before and fails now = fail; failed already before = warn."""
from app.services import audit_service, diagnostics_service
reachable = {"ping 1.1.1.1": True, "tcp 10.0.0.5:443": True, "ping 192.0.2.1": False}
actors = []
async def ping(host, target, ipproto, count, packetsize, source):
actors.append((audit_service.context.get() or {}).get("actor"))
return {"received": count if reachable[f"ping {target}"] else 0}
async def portprobe(host, target, port, ipproto, source):
if target == "broken":
raise diagnostics_service.OpnsenseAPIError("job failed")
return {"open": reachable[f"tcp {target}:{port}"]}
monkeypatch.setattr(diagnostics_service, "ping", ping)
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
Fw(respx_mock, A)
a, _ = await _hosts(client, cluster=False)
probes = [{"type": "ping", "target": "1.1.1.1", "port": None}, {"type": "tcp", "target": "10.0.0.5", "port": 443},
{"type": "ping", "target": "192.0.2.1", "port": None}]
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, uuid.UUID(a["id"]))
base = {a["id"]: await rh.baseline(db, host, probes)}
assert [p["ok"] for p in base[a["id"]]["probes"]] == [True, True, False] and set(actors) == {"system:rollout"}
gate, _ = await _eval([a["id"]], base, probes=probes)
assert gate["hosts"][a["id"]]["probes"] == {"result": "warn", "code": "probe_was_failing", "params": {"probes": "ping 192.0.2.1"}}
reachable["tcp 10.0.0.5:443"] = False
gate, _ = await _eval([a["id"]], base, probes=probes)
assert gate["verdict"] == "fail"
assert gate["hosts"][a["id"]]["probes"] == {"result": "fail", "code": "probe_regressed", "params": {"probes": "tcp 10.0.0.5:443"}}
broken = probes + [{"type": "tcp", "target": "broken", "port": 22}] # cannot run = failed; new since the baseline = worked before
gate, _ = await _eval([a["id"]], base, probes=broken)
assert "tcp broken:22" in gate["hosts"][a["id"]]["probes"]["params"]["probes"]
gate, _ = await _eval([a["id"]], base)
assert "probes" not in gate["hosts"][a["id"]] # a plan without probes runs none
fw_down = Fw(respx_mock, A)
fw_down.up = False
gate, _ = await _eval([a["id"]], base, probes=probes)
assert gate["hosts"][a["id"]]["probes"]["code"] == "unreadable"
+994
View File
@@ -0,0 +1,994 @@
"""Rule cleanup (1.31, B4 part 2): preview, deploy, confirm, revert/sweeper, delete and restore against a stateful
fake firewall (tests/rule_cleanup_fake.py) and the test database. Numbers refer to DESIGN-1.31 §6. Roles and API
tokens are checked in the router (package 6), not here."""
import asyncio
import uuid as uuid_lib
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
import httpx
import pytest
import pytest_asyncio
from fastapi import HTTPException
from sqlalchemy import select, update
from app.config import settings
from app.database import SessionLocal
from app.models.audit import AuditEvent
from app.models.governance import GovernanceSettings
from app.models.host import OpnsenseHost
from app.models.host_operation import HostOperation
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
from app.models.rule_hit import RuleHit
from app.models.rule_history import LiveRuleState, RuleChangeEvent
from app.services import rule_cleanup_service as svc
from app.services import rule_history_service as rhs
from app.services.rule_analysis import context as ctx_mod
from tests.rule_cleanup_fake import (
AUTO, MGMT_SOURCE, R_ALIAS, R_BASE, R_BLOCK, R_CARP, R_DUP, R_LEGACY, R_MGMT, R_TAIL, R_UNUSED, FakeFirewall,
default_rules, rule,
)
from tests.test_rules import _create_host
FW1, FW2 = "https://203.0.113.57", "https://203.0.113.58"
UNUSED_SEEDED = (R_UNUSED, R_BLOCK, R_MGMT, R_CARP, R_ALIAS, R_TAIL)
@pytest.fixture(autouse=True)
def _fresh_cache():
ctx_mod.clear_cache()
yield
ctx_mod.clear_cache()
def _now():
return datetime.now(timezone.utc)
async def _settings(sources=(MGMT_SOURCE,), max_rules=10, min_days=7, ports=()):
async with SessionLocal() as db:
row = await db.get(GovernanceSettings, 1)
if row is None:
row = GovernanceSettings(id=1, operators_may_deploy=True)
db.add(row)
row.cleanup_management_sources = list(sources)
row.cleanup_management_ports = list(ports)
row.cleanup_max_rules, row.cleanup_delete_min_days = max_rules, min_days
await db.commit()
async def _seed_hits(host_id, uuids, *, days=100, hit_days=None, sampled_minutes=10):
async with SessionLocal() as db:
now = _now()
for u in uuids:
db.add(RuleHit(host_id=host_id, rule_uuid=u.lower(), description=u, enabled=True,
first_seen_at=now - timedelta(days=days), last_packets=0, last_bytes=0,
last_sampled_at=now - timedelta(minutes=sampled_minutes),
last_hit_at=None if hit_days is None else now - timedelta(days=hit_days)))
await db.commit()
async def call(host_id, fn, *args, **kw):
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, host_id)
return await fn(db, host, *args, **kw)
async def refused(host_id, fn, *args, **kw) -> dict:
with pytest.raises(HTTPException) as info:
await call(host_id, fn, *args, **kw)
return {"status": info.value.status_code, **info.value.detail}
async def preview(host_id, uuids, actor="admin", **kw):
kw.setdefault("kind", "disable")
return await call(host_id, svc.preview, actor=actor, uuids=uuids, **kw)
async def deploy(host_id, preview_id, actor="admin", name=None):
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, host_id)
return await svc.deploy(db, host, uuid_lib.UUID(preview_id), actor=actor, confirm_name=name or host.name)
async def _items(op_id):
async with SessionLocal() as db:
return {i.rule_uuid: i for i in (await db.scalars(select(RuleCleanupItem).where(
RuleCleanupItem.operation_id == uuid_lib.UUID(str(op_id))))).all()}
async def _op(op_id):
async with SessionLocal() as db:
return await db.get(RuleCleanupOperation, uuid_lib.UUID(str(op_id)))
async def _backdate(op_id, **fields):
async with SessionLocal() as db:
await db.execute(update(RuleCleanupOperation).where(RuleCleanupOperation.id == uuid_lib.UUID(str(op_id))).values(**fields))
await db.commit()
async def _sweep():
async with SessionLocal() as db:
return await svc.sweep_expired_cleanups(db)
@pytest_asyncio.fixture
async def env(client, respx_mock):
log: list = []
fw = FakeFirewall(FW1, log=log).mount(respx_mock)
host_id = uuid_lib.UUID((await _create_host(client, "fw-a", FW1))["id"])
await _settings()
await _seed_hits(host_id, UNUSED_SEEDED)
fw.calls.clear()
log.clear()
return SimpleNamespace(fw=fw, host_id=host_id, log=log)
@pytest_asyncio.fixture
async def cluster(client, respx_mock):
log: list = []
backup = FakeFirewall(FW2, role="BACKUP", pfsync="lan", peer="10.20.30.2", lan_addr="10.20.30.3",
wan_addr="203.0.113.58", log=log).mount(respx_mock)
master = FakeFirewall(FW1, role="MASTER", pfsync="lan", peer="10.20.30.3", log=log).mount(respx_mock)
a = uuid_lib.UUID((await _create_host(client, "fw-a", FW1))["id"])
b = uuid_lib.UUID((await _create_host(client, "fw-b", FW2))["id"])
async with SessionLocal() as db:
await db.execute(update(OpnsenseHost).where(OpnsenseHost.id.in_([a, b])).values(cluster_name="ha-test"))
await db.commit()
await _settings()
await _seed_hits(a, UNUSED_SEEDED)
await _seed_hits(b, UNUSED_SEEDED)
for f in (master, backup):
f.calls.clear()
log.clear()
return SimpleNamespace(master=master, backup=backup, a=a, b=b, log=log)
def _writes(log):
return [(base, path) for base, method, path in log if method == "POST"]
# ── Preview ───────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_preview_recomputes_from_the_live_firewall_and_stores_a_bound_token(env):
result = await preview(env.host_id, [R_UNUSED, R_DUP])
plan = result["plan"]
items = {i["uuid"]: i for i in plan["nodes"][0]["items"]}
assert set(items) == {R_UNUSED, R_DUP}
assert items[R_UNUSED]["level"] == "sure" and items[R_UNUSED]["protection"]["result"] == "false"
assert items[R_DUP]["cover_uuid"] == R_BASE and items[R_DUP]["cover_fp"]
assert env.fw.writes() == [] # nothing is written by a preview
assert ("GET", "/firewall/filter/searchRule") in env.fw.calls # read live, not from a cache or the request
op = await _op(result["preview_id"])
assert op.status == "preview" and op.actor == "admin" and len(op.preview_fingerprint) == 64
assert timedelta(minutes=4) < op.expires_at - _now() <= timedelta(minutes=5)
async with SessionLocal() as db:
audit = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rule_cleanup"))).all()
assert [(a.action, a.status) for a in audit] == [("preview", "requested")]
assert "remote_uuid" not in audit[0].details # a preview never appears as a change in the rule history
# a second preview of the same actor replaces the first
again = await preview(env.host_id, [R_UNUSED])
assert (await _op(result["preview_id"])).status == "cancelled" and (await _op(again["preview_id"])).status == "preview"
@pytest.mark.asyncio
async def test_without_management_sources_everything_is_locked(env):
await _settings(sources=())
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
await _settings(sources=("not-an-address",))
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
@pytest.mark.asyncio
@pytest.mark.parametrize("uuid,reason", [
(R_MGMT, "management"), # allows the management source to the firewall's 443
(R_ALIAS, "management"), # source is a URL table: match unknown = protected (case 18)
(R_CARP, "infrastructure"), # CARP protocol
(R_BASE, "not_candidate"), # nothing says it is removable
])
async def test_rules_that_must_not_be_switched_off_are_refused(env, uuid, reason):
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[uuid], acknowledge=[uuid])
assert got["status"] == 409 and got["code"] == "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
assert got["params"]["reason"] == reason and got["params"]["uuid"] == uuid
assert env.fw.writes() == []
@pytest.mark.asyncio
async def test_managed_legacy_and_unknown_confidence_are_refused(env):
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
env.fw.rules.append(rule(R_LEGACY, 900, src="10.20.30.0/24", dport="6000", legacy=True))
await _seed_hits(env.host_id, [R_LEGACY])
async with SessionLocal() as db:
managed = FirewallRule(description="Managed", interface="lan", action="pass", direction="in", protocol="tcp",
source_net="any", destination_net="any", destination_port="8081")
db.add(managed)
await db.flush()
db.add(RuleDeployment(rule_id=managed.id, host_id=env.host_id, status="success", opnsense_rule_uuid=R_UNUSED))
# stale counters (sampled two days ago): "unused" is unknown, never selectable
await db.execute(update(RuleHit).where(RuleHit.rule_uuid == R_TAIL).values(last_sampled_at=_now() - timedelta(days=2)))
await db.commit()
for uuid, reason in ((R_UNUSED, "managed"), (R_LEGACY, "legacy"), (R_TAIL, "confidence")):
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[uuid], acknowledge=[uuid])
assert got["params"]["reason"] == reason, (uuid, got)
@pytest.mark.asyncio
async def test_probable_needs_an_explicit_acknowledgement_per_rule(env):
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_BLOCK, R_UNUSED])
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT" and got["params"]["uuids"] == R_BLOCK
ok = await preview(env.host_id, [R_BLOCK, R_UNUSED], acknowledge=[R_BLOCK], ack_text_version="1")
item = next(i for i in ok["plan"]["nodes"][0]["items"] if i["uuid"] == R_BLOCK)
assert item["level"] == "probable" and item["needs_ack"] and "UNUSED_BLOCK_RULE" in item["notes"]
op = await _op(ok["preview_id"])
assert op.options["acknowledged"] == [{"uuid": R_BLOCK, "text_version": "1"}]
@pytest.mark.asyncio
async def test_short_window_and_external_change_cap_at_probable(env):
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED], unused_days=14)
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
async with SessionLocal() as db:
db.add(RuleChangeEvent(host_id=env.host_id, rule_uuid=R_UNUSED, seq=1, observed_at=_now() - timedelta(days=2),
kind="changed", before={}, after={}, origin="external"))
await db.commit()
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
ok = await preview(env.host_id, [R_UNUSED], acknowledge=[R_UNUSED])
assert "EXTERNAL_CHANGE" in ok["plan"]["nodes"][0]["items"][0]["notes"]
@pytest.mark.asyncio
async def test_rule_limit_and_selection_validation(env):
await _settings(max_rules=2)
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED, R_DUP, R_TAIL])
assert got["status"] == 422 and got["code"] == "BEREINIGUNG_ZU_VIELE_REGELN" and got["params"]["max"] == 2
for uuids in ([], ["../apply"]):
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=uuids)
assert got["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG"
got = await refused(env.host_id, svc.preview, actor="admin", kind="wipe", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG"
@pytest.mark.asyncio
async def test_maintenance_open_deployments_and_open_cleanups_block(env):
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
async with SessionLocal() as db:
op = HostOperation(host_id=env.host_id, host_name="fw-a", actor="admin", action="reboot", status="submitted",
expires_at=_now(), fingerprint="x", snapshot={})
db.add(op)
await db.commit()
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
async with SessionLocal() as db:
await db.execute(update(HostOperation).values(status="done"))
managed = FirewallRule(description="x", interface="lan", action="pass", direction="in", protocol="tcp",
source_net="any", destination_net="any")
db.add(managed)
await db.flush()
dep = RuleDeployment(rule_id=managed.id, host_id=env.host_id, status="pending", opnsense_rule_uuid=R_TAIL)
db.add(dep)
await db.commit()
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_OFFENE_OPERATION" and got["params"]["reason"] == "deployment"
async with SessionLocal() as db:
await db.execute(update(RuleDeployment).values(status="success"))
await db.commit()
first = await preview(env.host_id, [R_UNUSED])
await deploy(env.host_id, first["preview_id"])
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_DUP])
assert got["code"] == "BEREINIGUNG_OFFENE_OPERATION" and got["params"]["reason"] == "cleanup"
async with SessionLocal() as db:
assert await svc.has_open_cleanup(db, [env.host_id])
# ── Deploy ────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_deploy_switches_off_samples_before_apply_and_waits_for_confirmation(env):
p = await preview(env.host_id, [R_UNUSED, R_DUP])
env.fw.calls.clear()
result = await deploy(env.host_id, p["preview_id"])
assert result["status"] == "awaiting_confirm" and result["can_confirm"]
writes = env.fw.writes()
assert sorted(writes[:2]) == sorted([("POST", f"/firewall/filter/toggleRule/{R_DUP}/0"),
("POST", f"/firewall/filter/toggleRule/{R_UNUSED}/0")])
assert writes[2:] == [("POST", "/firewall/filter/apply")] # one apply per node, after the toggles
apply_at = env.fw.calls.index(("POST", "/firewall/filter/apply"))
last_toggle = max(i for i, c in enumerate(env.fw.calls) if "toggleRule" in c[1])
assert ("GET", "/firewall/filter_util/ruleStats") in env.fw.calls[last_toggle:apply_at] # sampled before the apply
assert env.fw.get(R_UNUSED)["enabled"] == "0" and env.fw.get(R_DUP)["enabled"] == "0"
op = await _op(p["preview_id"])
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
assert window - timedelta(seconds=10) < op.revert_deadline - _now() <= window
items = await _items(p["preview_id"])
assert {i.state for i in items.values()} == {"disabled"}
assert items[R_UNUSED].fp_before and items[R_UNUSED].row_before["enabled"] == "1"
async with SessionLocal() as db:
audits = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rule_cleanup", AuditEvent.action == "disable"))).all()
assert {(a.details["remote_uuid"], a.status) for a in audits} == {(R_UNUSED, "accepted"), (R_DUP, "accepted")}
assert all(a.host_id == str(env.host_id) and a.details["preview_fingerprint"] == op.preview_fingerprint for a in audits)
@pytest.mark.asyncio
async def test_rule_history_shows_the_cleanup_as_opnexus(env):
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, env.host_id)
await rhs.observe_rows(db, host, {"rows": env.fw.rules}) # baseline
p = await preview(env.host_id, [R_UNUSED])
await deploy(env.host_id, p["preview_id"])
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, env.host_id)
await rhs.observe_rows(db, host, {"rows": env.fw.rules})
event = (await db.scalars(select(RuleChangeEvent).where(RuleChangeEvent.rule_uuid == R_UNUSED))).one()
assert event.kind == "disabled" and event.origin == "opnexus" and event.audit_id is not None
@pytest.mark.asyncio
@pytest.mark.parametrize("change", ["edited", "moved", "cover_edited", "hit"])
async def test_deploy_refuses_a_stale_preview_without_writing(env, change):
p = await preview(env.host_id, [R_UNUSED, R_DUP])
if change == "edited":
env.fw.get(R_UNUSED)["destination_port"] = "8082"
elif change == "moved":
env.fw.get(R_UNUSED)["sort_order"] = "400000.0000301"
elif change == "cover_edited":
env.fw.get(R_BASE)["description"] = "base changed"
else: # a hit since the preview: the fresh sample at deploy sees the counter grow
async with SessionLocal() as db:
await db.execute(update(RuleHit).where(RuleHit.rule_uuid == R_UNUSED).values(last_hit_at=_now()))
await db.commit()
env.fw.calls.clear()
got = await refused(env.host_id, svc.deploy, uuid_lib.UUID(p["preview_id"]), actor="admin", confirm_name="fw-a")
expected = R_DUP if change == "cover_edited" else R_UNUSED
if change == "hit":
assert got["code"] == "BEREINIGUNG_REGEL_NICHT_AENDERBAR" and got["params"]["uuid"] == R_UNUSED
else:
assert got["code"] == "BEREINIGUNG_VORSCHAU_VERALTET" and got["params"]["uuids"] == expected
assert env.fw.writes() == []
assert (await _op(p["preview_id"])).status == "expired"
@pytest.mark.asyncio
async def test_preview_token_is_single_use_bound_to_actor_and_name(env):
p = await preview(env.host_id, [R_UNUSED])
pid = uuid_lib.UUID(p["preview_id"])
got = await refused(env.host_id, svc.deploy, pid, actor="someone-else", confirm_name="fw-a")
assert got["status"] == 404
got = await refused(env.host_id, svc.deploy, pid, actor="admin", confirm_name="FW-A")
assert got["code"] == "HOSTNAMEN_EXAKT_EINGEBEN"
await deploy(env.host_id, p["preview_id"])
got = await refused(env.host_id, svc.deploy, pid, actor="admin", confirm_name="fw-a")
assert got["code"] == "VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET"
# an expired preview
await call(env.host_id, svc.revert_now, pid)
q = await preview(env.host_id, [R_DUP])
await _backdate(q["preview_id"], expires_at=_now() - timedelta(seconds=1))
got = await refused(env.host_id, svc.deploy, uuid_lib.UUID(q["preview_id"]), actor="admin", confirm_name="fw-a")
assert got["code"] == "VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET"
@pytest.mark.asyncio
async def test_refused_toggle_stops_writing_but_applies_what_is_switched(env):
uuids = [R_DUP, R_UNUSED, R_BLOCK, R_TAIL]
p = await preview(env.host_id, uuids, acknowledge=[R_BLOCK])
env.fw.fail[("toggle", R_BLOCK)] = httpx.Response(200, json={"result": "failed"})
env.fw.calls.clear()
result = await deploy(env.host_id, p["preview_id"])
states = {i.rule_uuid: i.state for i in (await _items(p["preview_id"])).values()}
assert states == {R_DUP: "disabled", R_UNUSED: "disabled", R_BLOCK: "failed", R_TAIL: "queued"}
assert ("POST", f"/firewall/filter/toggleRule/{R_TAIL}/0") not in env.fw.calls # nothing after the error
assert env.fw.writes()[-1] == ("POST", "/firewall/filter/apply") # but the switched ones are applied
assert result["status"] == "partial" and not result["can_confirm"] and result["can_revert"]
assert (await _op(p["preview_id"])).revert_deadline > _now()
got = await refused(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
assert got["code"] == "BEREINIGUNG_NICHT_BESTAETIGBAR"
@pytest.mark.asyncio
@pytest.mark.parametrize("applied,readable,state", [(True, True, "disabled"), (False, True, "failed"), (False, False, "unknown")])
async def test_unclear_toggle_is_settled_by_reading(env, applied, readable, state):
p = await preview(env.host_id, [R_UNUSED, R_DUP])
items_order = [i["uuid"] for i in p["plan"]["nodes"][0]["items"]]
first = items_order[0]
env.fw.fail[("toggle", first)] = httpx.ReadTimeout
env.fw.fail[("toggle_applies", first)] = applied
if not readable:
env.fw.fail[("get", first)] = httpx.ConnectError
result = await deploy(env.host_id, p["preview_id"])
items = await _items(p["preview_id"])
assert items[first].state == state
# no further write after an unclear answer (never attempted: "skipped" once nothing changed at all)
assert items[items_order[1]].state == ("skipped" if state == "failed" else "queued")
expect_apply = state in ("disabled", "unknown")
assert (("POST", "/firewall/filter/apply") in env.fw.calls) is expect_apply
assert result["status"] == {"disabled": "partial", "failed": "failed", "unknown": "partial"}[state]
assert not result["can_confirm"]
@pytest.mark.asyncio
async def test_failed_apply_keeps_the_deadline_and_the_sweeper_reverts(env):
p = await preview(env.host_id, [R_UNUSED])
env.fw.fail[("apply", "")] = [httpx.Response(500, json={}), None]
result = await deploy(env.host_id, p["preview_id"])
assert result["status"] == "apply_failed" and result["revert_deadline"]
await _sweep()
assert (await _op(p["preview_id"])).status == "apply_failed" # deadline not reached yet
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
env.fw.calls.clear()
await _sweep()
assert (await _op(p["preview_id"])).status == "reverted"
assert env.fw.writes() == [("POST", f"/firewall/filter/toggleRule/{R_UNUSED}/1"), ("POST", "/firewall/filter/apply")]
assert env.fw.get(R_UNUSED)["enabled"] == "1"
# ── Confirm ───────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_confirm_reads_the_firewall_and_refuses_when_unreachable_or_different(env):
p = await preview(env.host_id, [R_UNUSED, R_DUP])
await deploy(env.host_id, p["preview_id"])
pid = uuid_lib.UUID(p["preview_id"])
env.fw.down = True
got = await refused(env.host_id, svc.confirm, pid)
assert got["status"] == 502 and got["code"] == "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
assert (await _op(pid)).status == "awaiting_confirm" # stays open: the sweeper reverts it
env.fw.down = False
env.fw.get(R_DUP)["enabled"] = "1" # someone switched it back on in the GUI
got = await refused(env.host_id, svc.confirm, pid)
assert got["code"] == "BEREINIGUNG_LIVESTAND_ABWEICHEND" and got["params"]["uuids"] == R_DUP
env.fw.get(R_DUP)["enabled"] = "0"
env.fw.calls.clear()
result = await call(env.host_id, svc.confirm, pid)
assert result["status"] == "confirmed" and {i["state"] for i in result["items"]} == {"confirmed"}
assert env.fw.writes() == [] # confirming only reads
op = await _op(pid)
assert op.revert_deadline is None and op.confirmed_at is not None
assert await _sweep() == []
@pytest.mark.asyncio
async def test_confirm_after_the_deadline_is_refused_and_the_sweeper_reverts(env):
p = await preview(env.host_id, [R_UNUSED])
await deploy(env.host_id, p["preview_id"])
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
got = await refused(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
assert got["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
assert len(await _sweep()) == 1
assert env.fw.get(R_UNUSED)["enabled"] == "1"
assert {i.state for i in (await _items(p["preview_id"])).values()} == {"reverted"}
async with SessionLocal() as db:
audit = (await db.scalars(select(AuditEvent).where(AuditEvent.action == "revert"))).one()
assert audit.actor == "system:rollback" and audit.details["remote_uuid"] == R_UNUSED
@pytest.mark.asyncio
async def test_revert_now_and_failed_revert_is_not_retried_automatically(env):
p = await preview(env.host_id, [R_UNUSED, R_DUP])
await deploy(env.host_id, p["preview_id"])
env.fw.fail[("toggle", R_DUP)] = httpx.Response(200, json={"result": "failed"})
result = await call(env.host_id, svc.revert_now, uuid_lib.UUID(p["preview_id"]))
assert result["status"] == "revert_failed" and result["can_retry"]
states = {i["rule_uuid"]: i["state"] for i in result["items"]}
assert states == {R_UNUSED: "reverted", R_DUP: "revert_failed"} # the others are still reverted
assert env.fw.get(R_UNUSED)["enabled"] == "1" and env.fw.writes()[-1] == ("POST", "/firewall/filter/apply")
env.fw.calls.clear()
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(minutes=5))
await _sweep()
assert env.fw.writes() == [] # never retried by the sweeper
del env.fw.fail[("toggle", R_DUP)]
result = await call(env.host_id, svc.retry, uuid_lib.UUID(p["preview_id"]))
assert result["status"] == "reverted" and env.fw.get(R_DUP)["enabled"] == "1"
# ── Sweeper and two backends ──────────────────────────────────────────────
@pytest.mark.asyncio
async def test_sweeper_skips_a_live_run_and_takes_over_an_abandoned_one(env):
p = await preview(env.host_id, [R_UNUSED])
await deploy(env.host_id, p["preview_id"])
# simulate a backend that died mid-run: running, item half-way
async with SessionLocal() as db:
await db.execute(update(RuleCleanupOperation).values(status="running", heartbeat_at=_now(),
revert_deadline=_now() + timedelta(minutes=10)))
await db.execute(update(RuleCleanupItem).values(state="disabling"))
await db.commit()
env.fw.calls.clear()
assert await _sweep() == [] # (b) fresh heartbeat: someone is still working on it
assert env.fw.writes() == []
await _backdate(p["preview_id"], heartbeat_at=_now() - timedelta(hours=1))
assert await _sweep() == [p["preview_id"]] # (c) stale heartbeat: taken over and reverted
assert env.fw.writes() == [("POST", f"/firewall/filter/toggleRule/{R_UNUSED}/1"), ("POST", "/firewall/filter/apply")]
assert (await _op(p["preview_id"])).status == "reverted"
@pytest.mark.asyncio
async def test_two_deploys_of_one_preview_write_once(env):
p = await preview(env.host_id, [R_UNUSED, R_DUP])
outcomes = await asyncio.gather(deploy(env.host_id, p["preview_id"]), deploy(env.host_id, p["preview_id"]),
return_exceptions=True)
errors = [o for o in outcomes if isinstance(o, HTTPException)]
assert len(errors) == 1 and errors[0].detail["code"] == "VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET"
assert sum(1 for c in env.fw.calls if "toggleRule" in c[1]) == 2 # (a) each rule exactly once
@pytest.mark.asyncio
async def test_second_operation_on_the_same_rule_is_refused(env):
a = await preview(env.host_id, [R_UNUSED], actor="admin")
b = await preview(env.host_id, [R_UNUSED], actor="admin2")
await deploy(env.host_id, a["preview_id"])
got = await refused(env.host_id, svc.deploy, uuid_lib.UUID(b["preview_id"]), actor="admin2", confirm_name="fw-a")
assert got["code"] == "BEREINIGUNG_OFFENE_OPERATION" # (d)
# and the partial unique index holds even without the service's checks
async with SessionLocal() as db:
op = await db.get(RuleCleanupOperation, uuid_lib.UUID(b["preview_id"]))
db.add(RuleCleanupItem(operation_id=op.id, host_id=env.host_id, rule_uuid=R_UNUSED, state="queued", description=""))
with pytest.raises(Exception):
await db.commit()
@pytest.mark.asyncio
async def test_confirm_and_sweeper_at_the_deadline_decide_once(env):
p = await preview(env.host_id, [R_UNUSED])
await deploy(env.host_id, p["preview_id"])
await _backdate(p["preview_id"], revert_deadline=_now() + timedelta(milliseconds=300))
async def late_confirm():
await asyncio.sleep(0.4)
return await call(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
async def late_sweep():
await asyncio.sleep(0.4)
return await _sweep()
outcomes = await asyncio.gather(late_confirm(), late_sweep(), return_exceptions=True)
op = await _op(p["preview_id"])
assert op.status == "reverted" # (e) the deadline passed: confirm refused, exactly one revert
assert isinstance(outcomes[0], HTTPException) and outcomes[0].detail["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
assert sum(1 for c in env.fw.calls if c[1].endswith(f"{R_UNUSED}/1")) == 1
# ── Cluster ───────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_cluster_writes_backup_first_and_reverts_master_first(cluster):
p = await preview(cluster.a, [R_UNUSED])
assert [n["role"] for n in p["plan"]["nodes"]] == ["BACKUP", "MASTER"]
await deploy(cluster.a, p["preview_id"])
writes = _writes(cluster.log)
assert writes == [(FW2, f"/firewall/filter/toggleRule/{R_UNUSED}/0"), (FW2, "/firewall/filter/apply"),
(FW1, f"/firewall/filter/toggleRule/{R_UNUSED}/0"), (FW1, "/firewall/filter/apply")]
items = await _items(p["preview_id"])
assert len(items) == 1 # (same rule_uuid on both nodes: keyed by uuid here)
async with SessionLocal() as db:
roles = {i.host_id: i.carp_role for i in (await db.scalars(select(RuleCleanupItem))).all()}
assert roles == {cluster.a: "MASTER", cluster.b: "BACKUP"}
cluster.log.clear()
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
await _sweep()
assert _writes(cluster.log) == [(FW1, f"/firewall/filter/toggleRule/{R_UNUSED}/1"), (FW1, "/firewall/filter/apply"),
(FW2, f"/firewall/filter/toggleRule/{R_UNUSED}/1"), (FW2, "/firewall/filter/apply")]
@pytest.mark.asyncio
async def test_cluster_master_failure_leaves_backup_switched_and_unconfirmable(cluster):
p = await preview(cluster.a, [R_UNUSED])
cluster.master.fail[("toggle", R_UNUSED)] = httpx.Response(200, json={"result": "failed"})
result = await deploy(cluster.a, p["preview_id"])
assert result["status"] == "partial" and not result["can_confirm"]
assert cluster.backup.get(R_UNUSED)["enabled"] == "0" and cluster.master.get(R_UNUSED)["enabled"] == "1"
@pytest.mark.asyncio
async def test_cluster_refuses_missing_uuid_offline_peer_and_two_masters(cluster):
cluster.backup.rules = [r for r in cluster.backup.rules if r["uuid"] != R_UNUSED]
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG" and got["params"]["reason"] == "uuid_missing"
cluster.backup.rules = default_rules()
cluster.backup.role = "MASTER"
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG" and got["params"]["reason"] == "roles"
cluster.backup.role = "BACKUP"
cluster.backup.down = True
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["status"] == 502 and got["params"]["host"] == "fw-b"
@pytest.mark.asyncio
async def test_cluster_usage_is_judged_over_all_nodes(cluster):
# case 16: the backup saw no hit, the master did: not unused
async with SessionLocal() as db:
await db.execute(update(RuleHit).where(RuleHit.host_id == cluster.a, RuleHit.rule_uuid == R_UNUSED)
.values(last_hit_at=_now() - timedelta(days=1)))
await db.commit()
got = await refused(cluster.b, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
assert got["params"]["reason"] == "not_candidate"
# no usage data on one node: at best probable
async with SessionLocal() as db:
await db.execute(RuleHit.__table__.delete().where(RuleHit.host_id == cluster.a, RuleHit.rule_uuid == R_TAIL))
await db.commit()
got = await refused(cluster.b, svc.preview, actor="admin", kind="disable", uuids=[R_TAIL])
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
@pytest.mark.asyncio
async def test_cluster_pfsync_interface_rules_are_infrastructure(cluster):
# "any" from the LAN net on the pfsync interface (LAN) would also carry pfsync
for fw in (cluster.master, cluster.backup):
fw.rules.append(rule("aaaaaaaa-0000-4000-8000-000000000900", 900, proto="any", src="10.20.30.0/24", description="lan any"))
await _seed_hits(cluster.a, ["aaaaaaaa-0000-4000-8000-000000000900"])
await _seed_hits(cluster.b, ["aaaaaaaa-0000-4000-8000-000000000900"])
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=["aaaaaaaa-0000-4000-8000-000000000900"])
assert got["params"]["reason"] == "infrastructure"
# ── Delete and restore ────────────────────────────────────────────────────
async def _confirmed_disable(host_id, uuids, *, days_ago=8):
p = await preview(host_id, uuids)
await deploy(host_id, p["preview_id"])
await call(host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
await _backdate(p["preview_id"], confirmed_at=_now() - timedelta(days=days_ago))
return p["preview_id"]
@pytest.mark.asyncio
async def test_delete_needs_the_waiting_time_an_unchanged_rule_and_no_external_change(env):
await _confirmed_disable(env.host_id, [R_UNUSED], days_ago=2)
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT" and got["params"]["days"] == 5
async with SessionLocal() as db:
await db.execute(update(RuleCleanupOperation).values(confirmed_at=_now() - timedelta(days=8)))
await db.commit()
ok = await preview(env.host_id, [R_UNUSED], kind="delete")
assert ok["plan"]["nodes"][0]["items"][0]["basis"]["type"] == "confirmed"
env.fw.get(R_UNUSED)["destination_port"] = "8099"
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_VORSCHAU_VERALTET"
env.fw.get(R_UNUSED)["destination_port"] = "8081"
async with SessionLocal() as db:
db.add(RuleChangeEvent(host_id=env.host_id, rule_uuid=R_UNUSED, seq=5, observed_at=_now() - timedelta(days=1),
kind="changed", before={}, after={}, origin="external"))
await db.commit()
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED])
assert got["code"] == "BEREINIGUNG_VORSCHAU_VERALTET"
env.fw.get(R_DUP)["enabled"] = "1"
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_DUP])
assert got["params"]["reason"] == "not_candidate" # an enabled rule is never deleted
@pytest.mark.asyncio
async def test_delete_commits_the_snapshot_before_delrule(env, monkeypatch):
from app.services import opnsense_client as client_mod
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
seen: list = []
real_del = client_mod.del_rule
async def spy(host, uuid): # what another session sees in the database at the moment delRule goes out
async with SessionLocal() as db:
item = (await db.scalars(select(RuleCleanupItem).where(
RuleCleanupItem.operation_id == uuid_lib.UUID(p["preview_id"])))).one()
seen.append((item.state, bool(item.snapshot), item.anchor_next_uuid))
return await real_del(host, uuid)
monkeypatch.setattr(client_mod, "del_rule", spy)
result = await deploy(env.host_id, p["preview_id"])
assert seen == [("deleting", True, R_BLOCK)] # durable before the remote delete
assert result["status"] == "done"
item = next(iter((await _items(p["preview_id"])).values()))
assert item.state == "deleted" and item.snapshot["rule"]["destination_port"] == "8081"
assert item.snapshot["rule"]["action"] == "pass" and "uuid" not in item.snapshot["rule"]
assert item.prio_group == "400000" and item.anchor_prev_uuid == R_DUP
assert env.fw.get(R_UNUSED) is None and env.fw.writes()[-1] == ("POST", "/firewall/filter/apply")
@pytest.mark.asyncio
@pytest.mark.parametrize("applied,state", [(True, "deleted"), (False, "failed")])
async def test_unclear_delete_is_settled_with_getrule(env, applied, state):
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
env.fw.fail[("delete", R_UNUSED)] = httpx.ReadTimeout
env.fw.fail[("delete_applies", R_UNUSED)] = applied
result = await deploy(env.host_id, p["preview_id"])
assert next(iter((await _items(p["preview_id"])).values())).state == state
assert result["status"] == ("done" if state == "deleted" else "failed")
@pytest.mark.asyncio
async def test_restore_adds_disabled_at_the_anchor_with_a_new_uuid(env):
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
await deploy(env.host_id, p["preview_id"])
deleted = next(iter((await _items(p["preview_id"])).values()))
env.fw.calls.clear()
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert result["status"] == "done" and result["warnings"] == []
item = result["items"][0]
new = item["restored_uuid"]
assert new and new != R_UNUSED and item["state"] == "restored"
restored = env.fw.get(new)
assert restored["enabled"] == "0" and restored["destination_port"] == "8081"
assert ("POST", f"/firewall/filter/moveRuleBefore/{new}/{R_BLOCK}") in env.fw.calls
order = [r["uuid"] for r in sorted(env.fw.rules, key=lambda r: r["sort_order"]) if not r.get("is_automatic")]
assert order.index(new) == order.index(R_BLOCK) - 1 and order.index(new) == order.index(R_DUP) + 1
async with SessionLocal() as db:
audit = (await db.scalars(select(AuditEvent).where(AuditEvent.action == "restore"))).one()
source = await db.get(RuleCleanupItem, deleted.id)
assert audit.details["remote_uuid"] == new and audit.details["restored_from"] == R_UNUSED
assert source.restored_uuid == new
got = await refused(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert got["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH" # restored once only
# re-enabling the restored rule is an operation with a window
e = await preview(env.host_id, [new], kind="enable")
env.fw.calls.clear()
r = await deploy(env.host_id, e["preview_id"])
assert r["status"] == "awaiting_confirm" and ("POST", f"/firewall/filter/toggleRule/{new}/1") in env.fw.calls
@pytest.mark.asyncio
async def test_restore_without_anchor_warns_and_keeps_the_rule(env):
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
await deploy(env.host_id, p["preview_id"])
deleted = next(iter((await _items(p["preview_id"])).values()))
env.fw.rules = [r for r in env.fw.rules if r["uuid"] != R_BLOCK] # the anchor is gone meanwhile
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert result["status"] == "done" and result["warnings"] == ["BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"]
assert result["items"][0]["message"] == "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
assert env.fw.get(result["items"][0]["restored_uuid"])["enabled"] == "0"
@pytest.mark.asyncio
async def test_restore_refused_by_the_firewall_is_failed_with_its_message(env):
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
await deploy(env.host_id, p["preview_id"])
deleted = next(iter((await _items(p["preview_id"])).values()))
env.fw.fail[("add", "")] = httpx.Response(200, json={"result": "failed", "validations": {"rule.source_net": "alias missing"}})
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert result["status"] == "failed" and "alias missing" in result["items"][0]["message"]
async with SessionLocal() as db:
assert (await db.get(RuleCleanupItem, deleted.id)).restored_uuid is None # can be tried again
@pytest.mark.asyncio
async def test_cluster_delete_restore_and_reenable_follow_the_restore_link(cluster):
p = await preview(cluster.a, [R_UNUSED])
await deploy(cluster.a, p["preview_id"])
await call(cluster.a, svc.confirm, uuid_lib.UUID(p["preview_id"]))
await _backdate(p["preview_id"], confirmed_at=_now() - timedelta(days=8))
d = await preview(cluster.a, [R_UNUSED], kind="delete")
cluster.log.clear()
assert (await deploy(cluster.a, d["preview_id"]))["status"] == "done"
assert [w for w in _writes(cluster.log) if "delRule" in w[1]] == [(FW2, f"/firewall/filter/delRule/{R_UNUSED}"),
(FW1, f"/firewall/filter/delRule/{R_UNUSED}")]
async with SessionLocal() as db:
deleted = (await db.scalars(select(RuleCleanupItem).where(
RuleCleanupItem.operation_id == uuid_lib.UUID(d["preview_id"]), RuleCleanupItem.host_id == cluster.a))).one()
restored = await call(cluster.a, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert restored["status"] == "done" and len(restored["items"]) == 2 # both nodes, backup first
new = {i["host_id"]: i["restored_uuid"] for i in restored["items"]}
assert new[str(cluster.a)] != new[str(cluster.b)] # addRule: a new UUID on every node
assert cluster.backup.get(new[str(cluster.b)])["enabled"] == "0" and cluster.master.get(new[str(cluster.a)])["enabled"] == "0"
# re-enable through the master's UUID: the backup's copy is found through the restore operation
e = await preview(cluster.a, [new[str(cluster.a)]], kind="enable")
assert {n["host_name"]: n["items"][0]["uuid"] for n in e["plan"]["nodes"]} == {"fw-a": new[str(cluster.a)], "fw-b": new[str(cluster.b)]}
cluster.log.clear()
r = await deploy(cluster.a, e["preview_id"])
assert r["status"] == "awaiting_confirm"
assert [w for w in _writes(cluster.log) if "toggleRule" in w[1]] == [
(FW2, f"/firewall/filter/toggleRule/{new[str(cluster.b)]}/1"), (FW1, f"/firewall/filter/toggleRule/{new[str(cluster.a)]}/1")]
@pytest.mark.asyncio
async def test_restore_with_a_refused_move_is_verified_by_reading_and_only_warns(env):
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
await deploy(env.host_id, p["preview_id"])
deleted = next(iter((await _items(p["preview_id"])).values()))
# moveRuleBefore answers HTTP 500 (an Ambiguous error for the client, P3): the position is read back
env.fw.fail[("move", "*")] = httpx.Response(500, json={"errorMessage": "Unexpected error, check log for details"})
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert result["status"] == "done" and result["warnings"] == ["BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"]
@pytest.mark.asyncio
async def test_deadline_passing_while_confirm_reads_the_firewall_refuses(env):
import time
p = await preview(env.host_id, [R_UNUSED])
await deploy(env.host_id, p["preview_id"])
await _backdate(p["preview_id"], revert_deadline=_now() + timedelta(milliseconds=300))
env.fw.fail[("search", "")] = lambda: time.sleep(0.6) # a slow firewall: the window closes during the read
got = await refused(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
assert got["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
assert (await _op(p["preview_id"])).status == "awaiting_confirm" # left to the sweeper
@pytest.mark.asyncio
async def test_sweeper_leaves_a_live_run_alone_even_past_its_deadline(env, monkeypatch): # review finding 2
from app.services import opnsense_client as client_mod
p = await preview(env.host_id, [R_UNUSED, R_DUP])
real = client_mod.toggle_rule
swept: list = []
async def toggle(host, uuid, enabled): # a second backend's sweeper starts while the first toggle is on the wire
if not swept:
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
swept.append(asyncio.create_task(_sweep())) # it gets the lock in the deploy's next checkpoint gap
await asyncio.sleep(0.05)
return await real(host, uuid, enabled)
monkeypatch.setattr(client_mod, "toggle_rule", toggle)
result = await deploy(env.host_id, p["preview_id"])
assert await swept[0] == [] # fresh heartbeat: not taken over although the deadline passed
assert result["status"] == "awaiting_confirm"
assert {i.state for i in (await _items(p["preview_id"])).values()} == {"disabled"}
@pytest.mark.asyncio
async def test_a_run_taken_over_by_another_backend_writes_nothing_more(env, monkeypatch): # review finding 2
from app.services import opnsense_client as client_mod
p = await preview(env.host_id, [R_UNUSED, R_DUP])
real = client_mod.toggle_rule
calls: list = []
async def toggle(host, uuid, enabled):
calls.append(uuid)
if len(calls) == 1: # another backend took the operation over while this toggle was on the wire
await _backdate(p["preview_id"], status="reverting")
return await real(host, uuid, enabled)
monkeypatch.setattr(client_mod, "toggle_rule", toggle)
await deploy(env.host_id, p["preview_id"])
assert len(calls) == 1 # the conditional heartbeat before the second toggle found the operation gone
assert (await _op(p["preview_id"])).status == "reverting" # left to the new owner, not overwritten
states = sorted(i.state for i in (await _items(p["preview_id"])).values())
assert states == ["disabled", "queued"]
@pytest.mark.asyncio
async def test_extra_management_ports_protect_rules_on_them(env): # review finding 3
custom = "aaaaaaaa-0000-4000-8000-000000000910"
env.fw.rules.append(rule(custom, 910, interface="wan", src="203.0.113.0/24", dport="8443", description="gui 8443"))
await _seed_hits(env.host_id, [custom])
ok = await preview(env.host_id, [custom]) # 8443 is not one of the fixed ports: not about management
assert ok["plan"]["ports"] == []
await _settings(ports=(8443,))
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[custom])
assert got["params"]["reason"] == "management"
@pytest.mark.asyncio
async def test_cluster_peer_may_confirm_and_read_but_strangers_may_not(cluster, client, respx_mock): # review finding 4
p = await preview(cluster.a, [R_UNUSED])
await deploy(cluster.a, p["preview_id"])
pid = uuid_lib.UUID(p["preview_id"])
assert (await call(cluster.b, svc.get_operation, pid))["id"] == p["preview_id"]
FakeFirewall("https://203.0.113.59").mount(respx_mock)
other = uuid_lib.UUID((await _create_host(client, "fw-c", "https://203.0.113.59"))["id"])
for fn in (svc.get_operation, svc.confirm, svc.revert_now, svc.retry):
assert (await refused(other, fn, pid))["status"] == 404
got = await refused(cluster.b, svc.cancel, pid, actor="admin") # previews/deploys stay bound to the anchor
assert got["status"] == 404
result = await call(cluster.b, svc.confirm, pid)
assert result["status"] == "confirmed"
async def _half_deleted_cluster(cluster):
p = await preview(cluster.a, [R_UNUSED])
await deploy(cluster.a, p["preview_id"])
await call(cluster.a, svc.confirm, uuid_lib.UUID(p["preview_id"]))
await _backdate(p["preview_id"], confirmed_at=_now() - timedelta(days=8))
d = await preview(cluster.a, [R_UNUSED], kind="delete")
cluster.master.fail[("delete", R_UNUSED)] = httpx.Response(200, json={"result": "failed"})
result = await deploy(cluster.a, d["preview_id"])
assert result["status"] == "failed"
assert cluster.backup.get(R_UNUSED) is None and cluster.master.get(R_UNUSED) is not None
del cluster.master.fail[("delete", R_UNUSED)]
return d["preview_id"]
@pytest.mark.asyncio
async def test_half_done_cluster_delete_can_be_finished(cluster): # review finding 5
await _half_deleted_cluster(cluster)
again = await preview(cluster.a, [R_UNUSED], kind="delete")
assert [(n["host_name"], len(n["items"])) for n in again["plan"]["nodes"]] == [("fw-b", 0), ("fw-a", 1)]
cluster.log.clear()
assert (await deploy(cluster.a, again["preview_id"]))["status"] == "done"
assert [w for w in _writes(cluster.log) if "delRule" in w[1]] == [(FW1, f"/firewall/filter/delRule/{R_UNUSED}")]
assert cluster.master.get(R_UNUSED) is None
got = await refused(cluster.a, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED]) # nothing left
assert got["params"]["reason"] == "not_candidate"
@pytest.mark.asyncio
async def test_half_done_cluster_delete_can_be_restored_where_it_was_deleted(cluster): # review finding 5
delete_op = await _half_deleted_cluster(cluster)
async with SessionLocal() as db:
deleted = (await db.scalars(select(RuleCleanupItem).where(
RuleCleanupItem.operation_id == uuid_lib.UUID(delete_op), RuleCleanupItem.state == "deleted"))).one()
result = await call(cluster.a, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert result["status"] == "done" and [i["host_id"] for i in result["items"]] == [str(cluster.b)]
assert cluster.backup.get(result["items"][0]["restored_uuid"])["enabled"] == "0"
assert cluster.master.get(R_UNUSED)["enabled"] == "0" # the master's copy was never deleted: untouched
@pytest.mark.asyncio
async def test_switching_a_block_rule_back_on_needs_an_acknowledgement_unless_proven_harmless(env): # review finding 7
lock_out = "aaaaaaaa-0000-4000-8000-000000000920"
env.fw.rules.append(rule(lock_out, 920, action="block", interface="wan", src="203.0.113.0/24", dport="443",
enabled="0", description="block mgmt"))
unknown = "aaaaaaaa-0000-4000-8000-000000000940" # source is a URL table: match unknown
env.fw.rules.append(rule(unknown, 940, action="block", interface="wan", src="tbl", dport="443", enabled="0",
description="block table"))
harmless = "aaaaaaaa-0000-4000-8000-000000000930"
env.fw.rules.append(rule(harmless, 930, action="block", proto="UDP", dport="5353", enabled="0", description="block mdns"))
async with SessionLocal() as db: # both were switched off by a confirmed cleanup earlier
op = RuleCleanupOperation(host_id=env.host_id, kind="disable", status="confirmed", actor="admin",
expires_at=_now(), preview_fingerprint="x", options={}, confirmed_at=_now())
db.add(op)
await db.flush()
for u in (lock_out, harmless, unknown):
db.add(RuleCleanupItem(operation_id=op.id, host_id=env.host_id, rule_uuid=u, state="confirmed", description=u))
await db.commit()
got = await refused(env.host_id, svc.preview, actor="admin", kind="enable", uuids=[lock_out, harmless, unknown])
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT" and got["params"]["uuids"] == ", ".join(sorted([lock_out, unknown]))
ok = await preview(env.host_id, [lock_out, harmless, unknown], kind="enable", acknowledge=[lock_out, unknown])
items = {i["uuid"]: i for i in ok["plan"]["nodes"][0]["items"]}
assert items[lock_out]["needs_ack"] and items[lock_out]["notes"] == ["ENABLE_MAY_BLOCK_MANAGEMENT"]
assert not items[harmless]["needs_ack"] and items[harmless]["protection"]["result"] == "false"
async def _unknown_restore(env):
"""A restore whose addRule answer was lost: the item stays unknown."""
await _confirmed_disable(env.host_id, [R_UNUSED])
p = await preview(env.host_id, [R_UNUSED], kind="delete")
await deploy(env.host_id, p["preview_id"])
deleted = next(iter((await _items(p["preview_id"])).values()))
snapshot_row = dict(deleted.snapshot["row"])
env.fw.fail[("add", "")] = httpx.ReadTimeout
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
assert result["items"][0]["state"] == "unknown"
del env.fw.fail[("add", "")]
return result, snapshot_row
@pytest.mark.asyncio
async def test_unclear_restore_uses_the_stored_uuid_before_searching_by_content(env): # review finding 10
result, row = await _unknown_restore(env)
# on the firewall: the real copy and a content-identical twin someone made by hand
real, twin = "cccccccc-0000-4000-8000-000000000001", "cccccccc-0000-4000-8000-000000000002"
for u, seq in ((real, 950), (twin, 960)):
env.fw.rules.append({**row, "uuid": u, "sequence": str(seq), "sort_order": f"400000.{seq:07d}"})
async with SessionLocal() as db:
await db.execute(update(RuleCleanupItem).where(RuleCleanupItem.id == uuid_lib.UUID(result["items"][0]["id"]))
.values(restored_uuid=real))
await db.commit()
done = await call(env.host_id, svc.retry, uuid_lib.UUID(result["id"]))
assert done["items"][0]["state"] == "restored" and done["items"][0]["restored_uuid"] == real
@pytest.mark.asyncio
async def test_unclear_restore_with_twins_and_no_stored_uuid_stays_unknown(env): # review finding 10
result, row = await _unknown_restore(env)
for u, seq in (("cccccccc-0000-4000-8000-000000000003", 950), ("cccccccc-0000-4000-8000-000000000004", 960)):
env.fw.rules.append({**row, "uuid": u, "sequence": str(seq), "sort_order": f"400000.{seq:07d}"})
done = await call(env.host_id, svc.retry, uuid_lib.UUID(result["id"]))
assert done["items"][0]["state"] == "unknown" and done["items"][0]["message"] == "BEREINIGUNG_AUSGANG_UNKLAR"
env.fw.rules = [r for r in env.fw.rules if not r["uuid"].startswith("cccccccc")]
done = await call(env.host_id, svc.retry, uuid_lib.UUID(result["id"])) # nothing there: it was never created
assert done["items"][0]["state"] == "failed"
@pytest.mark.asyncio
async def test_an_outdated_acknowledgement_text_is_refused(env): # review finding 11
for version in ("0", "2", ""):
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_BLOCK],
acknowledge=[R_BLOCK], ack_text_version=version)
assert got["status"] == 409 and got["code"] == "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET", version
assert env.fw.calls == [] # refused before anything is read
ok = await preview(env.host_id, [R_BLOCK], acknowledge=[R_BLOCK], ack_text_version=svc.ACK_TEXT_VERSION)
assert ok["plan"]["acknowledged"] == [R_BLOCK]
+910
View File
@@ -0,0 +1,910 @@
"""Rule cleanup API (1.31, package 6): router, rights, locks in both directions, notification, settings.
Who may do what is enforced by TWO layers (AuthMiddleware via services/api_token_service.py, and the router itself).
The role matrix therefore runs three times: with both layers, with the middleware's rights checks switched off (the
router alone must refuse) and with the router's checks switched off (the middleware alone must refuse). A mutation
of either layer is then caught by one of the variants (tests/mutations_cleanup_1_31_api.json).
Addresses: documentation/private ranges only."""
import uuid as uuid_lib
from datetime import timedelta
import httpx
import pytest
import pytest_asyncio
from sqlalchemy import select, update
from app import auth_middleware
from app.database import SessionLocal
from app.models.audit import AuditEvent
from app.models.host import OpnsenseHost
from app.models.host_operation import HostOperation
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
from app.models.rule_deployment import RuleDeployment
from app.routers import rule_cleanup as router_mod
from app.schemas.rule_cleanup import CleanupItemRead, CleanupOperationRead
from app.services import api_token_service, notification_checks
from app.services import rule_cleanup_service as svc
from tests.rule_cleanup_fake import MGMT_SOURCE, R_ALIAS, R_BASE, R_BLOCK, R_DUP, R_MGMT, R_TAIL, R_UNUSED
from tests.test_api_tokens import _bearer, _login, _make
from tests.test_rule_cleanup import ( # noqa: F401 (fixtures are used by name)
FW1, FW2, _backdate, _confirmed_disable, _fresh_cache, _items, _now, _op, _settings, cluster, env, refused,
)
PW = "cleanup-api-test-pw-1"
NOHOST = "00000000-0000-0000-0000-0000000000aa"
NOID = "00000000-0000-0000-0000-0000000000bb"
# ── Principals ────────────────────────────────────────────────────────────
async def _user(client, name: str, role: str):
resp = await client.post("/api/users", json={"username": name, "password": PW, "role": role})
assert resp.status_code in (201, 409), resp.text # the account survives between tests
return await _login(name, PW)
async def _token_of(session, name: str, scope: str):
resp = await session.post("/api/tokens", json={"name": name, "scope": scope, "current_password": PW})
assert resp.status_code == 201, resp.text
return _bearer(resp.json()["token"])
@pytest_asyncio.fixture
async def who(client):
"""Every kind of caller: sessions of an admin, an operator and a read-only account, and tokens of each scope
(an admin's read/deploy/admin, an operator's deploy, a read-only account's read)."""
operator = await _user(client, "cl-operator", "operator")
read_only = await _user(client, "cl-readonly", "read_only")
callers = {
"admin": client, "operator": operator, "read_only": read_only,
"tok_read": _bearer((await _make(client, "read", name="cl-read"))["token"]),
"tok_deploy": _bearer((await _make(client, "deploy", name="cl-deploy"))["token"]),
"tok_admin": _bearer((await _make(client, "admin", name="cl-admin"))["token"]),
"tok_op_deploy": await _token_of(operator, "cl-op-deploy", "deploy"),
"tok_ro_read": await _token_of(read_only, "cl-ro-read", "read"),
}
yield callers
for name, session in callers.items():
if name != "admin":
await session.aclose()
async def _governance(client, may_deploy: bool):
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": may_deploy})).status_code == 200
# ── The role matrix ───────────────────────────────────────────────────────
ROUTES = {
"list": ("GET", "", None),
"get": ("GET", "/{op}", None),
"preview": ("POST", "/preview", {"kind": "disable", "uuids": ["a"]}),
"cancel": ("POST", "/{op}/cancel", None),
"deploy": ("POST", "/{op}/deploy", {"confirm_name": "x"}),
"confirm": ("POST", "/{op}/confirm", None),
"revert": ("POST", "/{op}/revert", None),
"retry": ("POST", "/{op}/retry", None),
"restore": ("POST", "/items/{op}/restore", {"confirm_name": "x"}),
}
READ_ONLY_CODES = {"READ_ONLY_ACCESS", "API_TOKEN_SCOPE_FORBIDDEN"}
def verdict(route: str, caller: str, may_deploy: bool):
"""None: the caller passes the gate (the unknown host then answers 404). Otherwise the acceptable 403 codes."""
if route in ("list", "get"):
return None
if caller in ("read_only", "tok_ro_read"):
return READ_ONLY_CODES
if caller == "tok_read":
return {"API_TOKEN_SCOPE_FORBIDDEN"}
if route in ("preview", "cancel"):
return None
if route == "confirm": # a human in a browser session; no token, whatever its scope
return None if caller in ("admin", "operator") else {"API_TOKEN_SCOPE_FORBIDDEN"}
if route in ("deploy", "retry", "restore"): # admin only, even though operators_may_deploy is on
if caller in ("admin", "tok_admin"):
return None
return {"admin": None, "operator": {"ADMIN_REQUIRED"}, "tok_deploy": {"API_TOKEN_SCOPE_FORBIDDEN"},
"tok_op_deploy": {"API_TOKEN_SCOPE_FORBIDDEN", "ADMIN_REQUIRED"}}[caller]
assert route == "revert" # an admin, or an operator while operators_may_deploy is on
if caller in ("admin", "tok_deploy", "tok_admin"):
return None
return None if may_deploy else {"DEPLOY_NEEDS_ADMIN"}
async def _gate(session, route: str):
method, path, body = ROUTES[route]
url = f"/api/hosts/{NOHOST}/rule-cleanup" + path.format(op=NOID)
resp = await session.request(method, url, **({"json": body} if body is not None else {}))
code = (resp.json().get("detail") or {}).get("code") if resp.headers.get("content-type", "").startswith("application/json") else None
return resp.status_code, code
@pytest.mark.asyncio
@pytest.mark.parametrize("may_deploy", [True, False])
@pytest.mark.parametrize("variant", ["both", "router_only", "middleware_only"])
async def test_every_route_for_every_caller(client, who, monkeypatch, variant, may_deploy):
await _governance(client, may_deploy)
try:
if variant == "router_only": # switch the middleware's rights checks off: the router must still refuse
monkeypatch.setattr(api_token_service, "denies", lambda *a, **k: False)
monkeypatch.setattr(api_token_service, "operator_denies", lambda *a, **k: False)
monkeypatch.setattr(api_token_service, "is_rollout", lambda *a, **k: False)
class _Everything: # the read-only gate of the middleware lets this request through
def __contains__(self, _):
return True
monkeypatch.setattr(auth_middleware, "_READ_ONLY_ALLOWED_WRITES", _Everything())
if variant == "middleware_only": # switch the router's checks off: the middleware must still refuse
async def _nothing_async(*a, **k):
return None
for name in ("_writer", "_admin_only", "_session_only"):
monkeypatch.setattr(router_mod, name, lambda *a, **k: None)
monkeypatch.setattr(router_mod, "_deployer", _nothing_async)
problems = []
for route in ROUTES:
for caller, session in who.items():
want = verdict(route, caller, may_deploy)
got = await _gate(session, route)
ok = got == (404, "HOST_NICHT_GEFUNDEN") if want is None else (got[0] == 403 and got[1] in want)
if not ok:
problems.append((route, caller, got, want))
assert not problems, problems
finally:
await _governance(client, True)
@pytest.mark.parametrize("method,path,session_only,admin_only,rollout", [
("POST", "/api/hosts/h/rule-cleanup/preview", False, False, False),
("POST", "/api/hosts/h/rule-cleanup/o/cancel", False, False, False),
("POST", "/api/hosts/h/rule-cleanup/o/confirm", True, False, False),
("POST", "/api/hosts/h/rule-cleanup/o/confirm/", True, False, False),
("POST", "/api/hosts/h/rule-cleanup/o/deploy", False, True, True),
("POST", "/api/hosts/h/rule-cleanup/o/deploy/", False, True, False),
("POST", "/api/hosts/h/rule-cleanup/o/revert", False, False, True),
("POST", "/api/hosts/h/rule-cleanup/o/retry", False, True, True),
("POST", "/api/hosts/h/rule-cleanup/items/i/restore", False, True, False),
("GET", "/api/hosts/h/rule-cleanup", False, False, False),
("GET", "/api/hosts/h/rule-cleanup/o", False, False, False),
])
def test_pattern_table(method, path, session_only, admin_only, rollout):
denied_for_read = method == "POST"
assert api_token_service.denies("read", method, path) is denied_for_read
# a token below scope admin: admin-only routes and confirm are closed
assert api_token_service.denies("deploy", method, path) is (session_only or admin_only)
# an admin-scope token reaches everything except what only a session may do
assert api_token_service.denies("admin", method, path) is session_only
assert api_token_service.operator_denies(method, path) is admin_only
assert api_token_service.is_rollout(method, path) is rollout
def test_governance_settings_are_admin_writes_for_operators_and_deploy_tokens():
assert api_token_service.operator_denies("PUT", "/api/settings/governance") is True
assert api_token_service.denies("deploy", "PUT", "/api/settings/governance") is True
assert api_token_service.denies("admin", "PUT", "/api/settings/governance") is False
# ── A full run over the API: shapes, audit, input forms ──────────────────
def _base(env_or_id) -> str:
return f"/api/hosts/{getattr(env_or_id, 'host_id', env_or_id)}/rule-cleanup"
async def _audit(**filters):
async with SessionLocal() as db:
rows = (await db.scalars(select(AuditEvent).where(*[getattr(AuditEvent, k) == v for k, v in filters.items()])
.order_by(AuditEvent.created_at))).all()
return rows
async def _open(session, env, uuids=(R_UNUSED, R_DUP), name="fw-a"):
"""Preview + deploy over the API; the run waits for its confirmation."""
pv = await session.post(_base(env) + "/preview", json={"kind": "disable", "uuids": list(uuids)})
assert pv.status_code == 200, pv.text
op = await session.post(f"{_base(env)}/{pv.json()['preview_id']}/deploy", json={"confirm_name": name})
assert op.status_code == 200, op.text
return op.json()
@pytest.mark.asyncio
async def test_preview_deploy_confirm_over_the_api_with_shapes_and_audit(client, env):
pv = await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED, R_DUP]})
assert pv.status_code == 200, pv.text
body = pv.json()
assert {"preview_id", "expires_at", "kind", "plan", "max_rules", "window_seconds", "operation"} <= set(body)
assert body["max_rules"] == 10 and body["kind"] == "disable" and body["window_seconds"] > 0
assert {i["uuid"] for i in body["plan"]["nodes"][0]["items"]} == {R_UNUSED, R_DUP}
operation = body["operation"]
assert set(operation) == set(CleanupOperationRead.model_fields) # the UI gets exactly the documented fields
assert operation["status"] == "preview" and operation["plan"] and operation["items"] == []
assert operation["actor"] == "admin" and operation["host_name"] == "fw-a"
# The middleware does not log /preview; the service writes the one entry (and it is no history change).
assert [(a.action, a.status) for a in await _audit(kind="rule_cleanup")] == [("preview", "requested")]
assert await _audit(kind="host", action="preview") == []
assert env.fw.writes() == []
pid = body["preview_id"]
wrong = await client.post(f"{_base(env)}/{pid}/deploy", json={"confirm_name": "fw-x"})
assert wrong.status_code == 409 and wrong.json()["detail"]["code"] == "HOSTNAMEN_EXAKT_EINGEBEN"
deployed = await client.post(f"{_base(env)}/{pid}/deploy", json={"confirm_name": "fw-a"})
assert deployed.status_code == 200, deployed.text
op = deployed.json()
assert set(op) == set(CleanupOperationRead.model_fields) and all(set(i) == set(CleanupItemRead.model_fields) for i in op["items"])
assert op["status"] == "awaiting_confirm" and op["revert_deadline"] and op["plan"] is None
assert (op["can_confirm"], op["can_revert"], op["can_retry"]) == (True, True, False)
assert {(i["rule_uuid"], i["state"], i["host_name"]) for i in op["items"]} == {(R_UNUSED, "disabled", "fw-a"), (R_DUP, "disabled", "fw-a")}
assert env.fw.get(R_UNUSED)["enabled"] == "0"
# Audit: one request pair from the middleware (kind host), one entry per rule from the service - no duplicates.
# the wrong-name attempt and the real one: a request entry and an outcome each, from the middleware only
assert sorted(a.status for a in await _audit(kind="host", action="deploy")) == ["accepted", "rejected", "requested", "requested"]
assert [a.status for a in await _audit(kind="rule_cleanup", action="disable")] == ["accepted", "accepted"]
assert await _audit(kind="rule_cleanup", action="deploy") == [] and await _audit(kind="host", action="disable") == []
listing = (await client.get(_base(env))).json()
assert [o["id"] for o in listing] == [pid] and listing[0]["status"] == "awaiting_confirm"
one = (await client.get(f"{_base(env)}/{pid}")).json()
assert one == listing[0]
confirmed = await client.post(f"{_base(env)}/{pid}/confirm")
assert confirmed.status_code == 200, confirmed.text
c = confirmed.json()
assert c["status"] == "confirmed" and c["revert_deadline"] is None and {i["state"] for i in c["items"]} == {"confirmed"}
assert (c["can_confirm"], c["can_revert"], c["can_retry"]) == (False, False, False)
assert [a.status for a in await _audit(kind="rule_cleanup", action="confirm")] == ["confirmed", "confirmed"]
assert sorted(a.status for a in await _audit(kind="host", action="confirm")) == ["accepted", "requested"]
again = await client.post(f"{_base(env)}/{pid}/confirm")
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
missing = await client.get(f"{_base(env)}/{NOID}")
assert missing.status_code == 404 and missing.json()["detail"]["code"] == "BEREINIGUNG_NICHT_GEFUNDEN"
assert (await client.get(f"/api/hosts/{NOHOST}/rule-cleanup")).status_code == 404
@pytest.mark.asyncio
async def test_preview_input_forms_and_validation(client, env):
url = _base(env) + "/preview"
# items[] with per-rule kind/mode and acknowledgement; a "probable" rule needs the acknowledgement
plain = {"items": [{"uuid": R_UNUSED, "mode": "disable"}, {"uuid": R_BLOCK, "kind": "disable"}]}
got = await client.post(url, json=plain)
assert got.status_code == 409 and got.json()["detail"]["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
assert R_BLOCK in got.json()["detail"]["params"]["uuids"]
ack = {"items": [{"uuid": R_UNUSED, "mode": "disable"}, {"uuid": R_BLOCK, "kind": "disable", "ack": True}]}
done = await client.post(url, json=ack)
assert done.status_code == 200, done.text
assert {i["uuid"] for i in done.json()["plan"]["nodes"][0]["items"]} == {R_UNUSED, R_BLOCK}
assert done.json()["plan"]["acknowledged"] == [R_BLOCK]
# the old form: kind + uuids + acknowledge
old = await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED, R_BLOCK], "acknowledge": [R_BLOCK]})
assert old.status_code == 200 and (await _op(done.json()["preview_id"])).status == "cancelled" # one preview per actor
# bad input
for bad in ({"uuids": [R_UNUSED]}, {"items": []}):
r = await client.post(url, json=bad)
assert r.status_code == 422 and r.json()["detail"]["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG", bad
for bad in ({"kind": "disable", "uuids": []}, {"kind": "disable", "uuids": ["../x"]}):
r = await client.post(url, json=bad)
assert r.status_code == 422 and r.json()["detail"]["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG", bad
mixed = {"items": [{"uuid": R_UNUSED, "kind": "disable"}, {"uuid": R_DUP, "kind": "delete"}]}
assert (await client.post(url, json=mixed)).status_code == 422
assert (await client.post(url, json={"kind": "restore", "uuids": [R_UNUSED]})).status_code == 422
assert (await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED], "unused_days": 0})).status_code == 422
assert (await client.post(f"/api/hosts/{NOHOST}/rule-cleanup/preview", json={"kind": "disable", "uuids": [R_UNUSED]})).status_code == 404
# the settings reach the service: the limit and the fail-closed lock without management sources
assert (await client.put("/api/settings/governance", json={"cleanup_max_rules": 1})).status_code == 200
many = await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED, R_DUP]})
assert many.status_code == 422 and many.json()["detail"]["code"] == "BEREINIGUNG_ZU_VIELE_REGELN" and many.json()["detail"]["params"]["max"] == 1
assert (await client.put("/api/settings/governance", json={"cleanup_management_sources": []})).status_code == 200
locked = await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED]})
assert locked.status_code == 409 and locked.json()["detail"]["code"] == "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
@pytest.mark.asyncio
async def test_a_preview_belongs_to_its_actor_and_can_be_cancelled(client, env, who):
operator = who["operator"]
mine = (await operator.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})).json()
assert mine["operation"]["actor"] == "cl-operator"
# the admin neither sees it as his own nor deploys it; the operator cannot deploy at all
stranger = await client.post(f"{_base(env)}/{mine['preview_id']}/deploy", json={"confirm_name": "fw-a"})
assert stranger.status_code == 404 and stranger.json()["detail"]["code"] == "BEREINIGUNG_NICHT_GEFUNDEN"
refused_op = await operator.post(f"{_base(env)}/{mine['preview_id']}/deploy", json={"confirm_name": "fw-a"})
assert refused_op.status_code == 403 and refused_op.json()["detail"]["code"] == "ADMIN_REQUIRED"
assert (await client.post(f"{_base(env)}/{mine['preview_id']}/cancel")).status_code == 404 # not the admin's
cancelled = await operator.post(f"{_base(env)}/{mine['preview_id']}/cancel")
assert cancelled.status_code == 200 and cancelled.json()["status"] == "cancelled"
# the admin's refused attempt (not his preview) and the operator's real cancel: a request entry and an outcome each
assert sorted(a.status for a in await _audit(kind="host", action="cancel")) == ["accepted", "rejected", "requested", "requested"]
again = await operator.post(f"{_base(env)}/{mine['preview_id']}/cancel")
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
gone = await client.post(f"{_base(env)}/{mine['preview_id']}/deploy", json={"confirm_name": "fw-a"})
assert gone.status_code == 404 # (not the admin's preview)
assert env.fw.writes() == []
# an operator prepares, the admin prepares his own and executes; the operator may confirm afterwards
own = (await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})).json()
deployed = await client.post(f"{_base(env)}/{own['preview_id']}/deploy", json={"confirm_name": "fw-a"})
assert deployed.status_code == 200
confirmed = await operator.post(f"{_base(env)}/{own['preview_id']}/confirm")
assert confirmed.status_code == 200 and confirmed.json()["status"] == "confirmed"
@pytest.mark.asyncio
async def test_confirm_never_works_with_a_token_and_changes_nothing(client, env, who):
op = await _open(client, env)
for name in ("tok_admin", "tok_deploy", "tok_op_deploy"):
r = await who[name].post(f"{_base(env)}/{op['id']}/confirm")
assert r.status_code == 403 and r.json()["detail"]["code"] == "API_TOKEN_SCOPE_FORBIDDEN", name
state = await _op(op["id"])
assert state.status == "awaiting_confirm" and state.revert_deadline is not None
assert {i.state for i in (await _items(op["id"])).values()} == {"disabled"}
# the same token may read the run, and the browser session confirms
assert (await who["tok_read"].get(f"{_base(env)}/{op['id']}")).json()["can_confirm"] is True
assert (await client.post(f"{_base(env)}/{op['id']}/confirm")).json()["status"] == "confirmed"
denied = await _audit(kind="host", action="confirm", status="rejected")
assert not denied # the middleware refused before the router: no request entry for a refused token confirm
@pytest.mark.asyncio
async def test_revert_is_for_admins_and_for_operators_only_while_they_may_deploy(client, env, who):
op = await _open(client, env)
url = f"{_base(env)}/{op['id']}/revert"
await _governance(client, False)
try:
assert (await who["operator"].post(url)).status_code == 403
assert (await who["tok_op_deploy"].post(url)).status_code == 403
assert (await who["read_only"].post(url)).status_code == 403
assert (await _op(op["id"])).status == "awaiting_confirm" and env.fw.get(R_UNUSED)["enabled"] == "0"
finally:
await _governance(client, True)
reverted = await who["operator"].post(url)
assert reverted.status_code == 200, reverted.text
r = reverted.json()
assert r["status"] == "reverted" and {i["state"] for i in r["items"]} == {"reverted"} and not r["can_revert"]
assert env.fw.get(R_UNUSED)["enabled"] == "1" and env.fw.get(R_DUP)["enabled"] == "1"
again = await client.post(url)
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
@pytest.mark.asyncio
async def test_retry_after_a_failed_revert_is_admin_only(client, env, who):
op = await _open(client, env)
env.fw.fail[("toggle", R_DUP)] = httpx.Response(200, json={"result": "failed"})
failed = await client.post(f"{_base(env)}/{op['id']}/revert")
assert failed.status_code == 200 and failed.json()["status"] == "revert_failed" and failed.json()["can_retry"] is True
del env.fw.fail[("toggle", R_DUP)]
url = f"{_base(env)}/{op['id']}/retry"
for name, code in (("operator", "ADMIN_REQUIRED"), ("tok_deploy", "API_TOKEN_SCOPE_FORBIDDEN"), ("read_only", "READ_ONLY_ACCESS")):
r = await who[name].post(url)
assert r.status_code == 403 and r.json()["detail"]["code"] == code, name
assert (await _op(op["id"])).status == "revert_failed"
done = await who["tok_admin"].post(url)
assert done.status_code == 200 and done.json()["status"] == "reverted" and env.fw.get(R_DUP)["enabled"] == "1"
nothing = await client.post(url)
assert nothing.status_code == 409 and nothing.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
@pytest.mark.asyncio
async def test_delete_and_restore_over_the_api(client, env, who):
await _confirmed_disable(env.host_id, [R_UNUSED])
pv = (await client.post(_base(env) + "/preview", json={"kind": "delete", "uuids": [R_UNUSED]})).json()
assert pv["operation"]["kind"] == "delete"
done = await client.post(f"{_base(env)}/{pv['preview_id']}/deploy", json={"confirm_name": "fw-a"})
assert done.status_code == 200, done.text
op = done.json()
assert op["status"] == "done" and op["kind"] == "delete" and (op["can_confirm"], op["can_revert"], op["can_retry"]) == (False,) * 3
item = op["items"][0]
assert item["state"] == "deleted" and item["restorable"] is True and item["has_snapshot"] is True
assert env.fw.get(R_UNUSED) is None
assert [a.status for a in await _audit(kind="rule_cleanup", action="delete")] == ["accepted"]
url = f"{_base(env)}/items/{item['id']}/restore"
for name, code in (("operator", "ADMIN_REQUIRED"), ("tok_deploy", "API_TOKEN_SCOPE_FORBIDDEN"), ("read_only", "READ_ONLY_ACCESS")):
r = await who[name].post(url, json={"confirm_name": "fw-a"})
assert r.status_code == 403 and r.json()["detail"]["code"] == code, name
wrong = await client.post(url, json={"confirm_name": "nope"})
assert wrong.status_code == 409 and wrong.json()["detail"]["code"] == "HOSTNAMEN_EXAKT_EINGEBEN"
assert env.fw.get(R_UNUSED) is None
restored = await who["tok_admin"].post(url, json={"confirm_name": "fw-a"})
assert restored.status_code == 200, restored.text
r = restored.json()
assert r["kind"] == "restore" and r["status"] == "done" and r["warnings"] == []
new = r["items"][0]["restored_uuid"]
assert new and r["items"][0]["state"] == "restored" and env.fw.get(new)["enabled"] == "0"
listed = {o["id"]: o for o in (await client.get(_base(env))).json()}
source = next(i for i in listed[op["id"]]["items"] if i["id"] == item["id"])
assert source["restorable"] is False and source["restored_uuid"] == new
assert [a.status for a in await _audit(kind="rule_cleanup", action="restore")] == ["accepted"]
again = await client.post(url, json={"confirm_name": "fw-a"})
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
nothing = await client.post(f"{_base(env)}/{op['id']}/retry")
assert nothing.status_code == 409 # nothing unclear in a finished delete
@pytest.mark.asyncio
async def test_restore_reports_a_lost_position_as_a_warning(client, env):
await _confirmed_disable(env.host_id, [R_UNUSED])
pv = (await client.post(_base(env) + "/preview", json={"kind": "delete", "uuids": [R_UNUSED]})).json()
op = (await client.post(f"{_base(env)}/{pv['preview_id']}/deploy", json={"confirm_name": "fw-a"})).json()
env.fw.rules = [r for r in env.fw.rules if r["uuid"] != R_BLOCK] # the anchor is gone
r = await client.post(f"{_base(env)}/items/{op['items'][0]['id']}/restore", json={"confirm_name": "fw-a"})
assert r.status_code == 200, r.text
assert r.json()["warnings"] == ["BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"]
assert r.json()["items"][0]["message"] == "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
missing = await client.post(f"{_base(env)}/items/{NOID}/restore", json={"confirm_name": "fw-a"})
assert missing.status_code == 404 and missing.json()["detail"]["code"] == "BEREINIGUNG_NICHT_GEFUNDEN"
# ── Settings ──────────────────────────────────────────────────────────────
GOV = "/api/settings/governance"
@pytest.mark.asyncio
async def test_management_sources_are_validated_normalised_and_audited(client):
put = await client.put(GOV, json={"cleanup_management_sources": [" 203.0.113.10 ", "10.20.30.5/24", "203.0.113.10", "2001:db8::1"],
"cleanup_max_rules": 25, "cleanup_delete_min_days": 30})
assert put.status_code == 200, put.text
got = put.json()
assert got["cleanup_management_sources"] == ["203.0.113.10", "10.20.30.0/24", "2001:db8::1"]
assert got["cleanup_management_sources_configured"] is True and got["cleanup_max_rules"] == 25 and got["cleanup_delete_min_days"] == 30
assert (await client.get(GOV)).json() == got
audit = await _audit(kind="settings", action="cleanup_settings")
assert len(audit) == 1 and audit[0].actor == "admin"
changes = audit[0].details["changes"]
assert changes["cleanup_management_sources"]["new"] == got["cleanup_management_sources"]
assert changes["cleanup_max_rules"] == {"old": 10, "new": 25}
# an unchanged value writes no further entry; a partial update leaves the rest alone
assert (await client.put(GOV, json={"cleanup_max_rules": 25, "operators_may_deploy": True})).json()["cleanup_delete_min_days"] == 30
assert len(await _audit(kind="settings", action="cleanup_settings")) == 1
bad_sources = [["fw.example.test"], ["203.0.113.999"], ["0.0.0.0/0"], ["::/0"], ["203.0.113.1", ""], [f"10.20.30.{i}" for i in range(21)], ["10.20.30.0/33"]]
for sources in bad_sources:
r = await client.put(GOV, json={"cleanup_management_sources": sources})
assert r.status_code == 422 and r.json()["detail"]["code"] == "BEREINIGUNG_EINSTELLUNG_UNGUELTIG", sources
assert r.json()["detail"]["params"]["field"] == "cleanup_management_sources"
for field, value in (("cleanup_max_rules", 0), ("cleanup_max_rules", 26), ("cleanup_delete_min_days", 0), ("cleanup_delete_min_days", 366)):
r = await client.put(GOV, json={field: value})
assert r.status_code == 422 and r.json()["detail"]["params"]["field"] == field, (field, value)
assert (await client.get(GOV)).json() == got # nothing of the refused requests was stored
assert len(await _audit(kind="settings", action="cleanup_settings")) == 1
assert (await client.put(GOV, json={"cleanup_management_sources": [f"10.20.30.{i}" for i in range(20)]})).status_code == 200
@pytest.mark.asyncio
async def test_management_ports_are_validated_normalised_audited_and_hidden(client, who): # review finding 3
put = await client.put(GOV, json={"cleanup_management_ports": [8443, "2222", 8443]})
assert put.status_code == 200, put.text
assert put.json()["cleanup_management_ports"] == [2222, 8443] and put.json()["cleanup_management_ports_configured"] is True
audit = await _audit(kind="settings", action="cleanup_settings")
assert audit[-1].details["changes"]["cleanup_management_ports"] == {"old": [], "new": [2222, 8443]}
for ports in ([0], [65536], ["ssh"], [True], [-1], list(range(1, 22))):
r = await client.put(GOV, json={"cleanup_management_ports": ports})
assert r.status_code == 422 and r.json()["detail"]["params"]["field"] == "cleanup_management_ports", ports
assert (await client.get(GOV)).json()["cleanup_management_ports"] == [2222, 8443]
for name in ("operator", "read_only"):
seen = (await who[name].get(GOV)).json()
assert seen["cleanup_management_ports"] == [] and seen["cleanup_management_ports_configured"] is True, name
r = await who[name].put(GOV, json={"cleanup_management_ports": [1]})
assert r.status_code == 403, name
assert (await client.put(GOV, json={"cleanup_management_ports": []})).json()["cleanup_management_ports_configured"] is False
@pytest.mark.asyncio
@pytest.mark.parametrize("variant", ["both", "router_only"])
async def test_only_an_admin_writes_the_cleanup_settings(client, who, monkeypatch, variant):
await client.put(GOV, json={"cleanup_management_sources": ["203.0.113.10"]})
if variant == "router_only":
monkeypatch.setattr(api_token_service, "denies", lambda *a, **k: False)
monkeypatch.setattr(api_token_service, "operator_denies", lambda *a, **k: False)
# router_only: an admin-owned deploy/read token IS an admin account: only the token layer (switched off) stops it
attempts = {"operator": {"ADMIN_REQUIRED"}, "tok_op_deploy": {"API_TOKEN_SCOPE_FORBIDDEN", "ADMIN_REQUIRED"},
"read_only": {"READ_ONLY_ACCESS", "ADMIN_REQUIRED"}}
if variant == "both":
attempts.update({"tok_deploy": {"API_TOKEN_SCOPE_FORBIDDEN"}, "tok_read": {"API_TOKEN_SCOPE_FORBIDDEN"}})
for name, codes in attempts.items():
r = await who[name].put(GOV, json={"cleanup_management_sources": ["198.51.100.7"], "cleanup_max_rules": 3})
assert r.status_code == 403 and r.json()["detail"]["code"] in codes, (name, r.text)
assert (await client.get(GOV)).json()["cleanup_management_sources"] == ["203.0.113.10"]
assert (await client.get(GOV)).json()["cleanup_max_rules"] == 10
# an admin-scope token acts as the admin and the audit names the token
ok = await who["tok_admin"].put(GOV, json={"cleanup_management_sources": ["198.51.100.7"]})
assert ok.status_code == 200, ok.text
assert [a.actor for a in await _audit(kind="settings", action="cleanup_settings")][-1] == "admin (API: cl-admin)"
@pytest.mark.asyncio
async def test_others_only_see_whether_management_sources_are_set(client, who):
await client.put(GOV, json={"cleanup_management_sources": ["203.0.113.10"]})
assert (await client.get(GOV)).json()["cleanup_management_sources"] == ["203.0.113.10"]
for name in ("operator", "read_only"):
seen = (await who[name].get(GOV)).json()
assert seen["cleanup_management_sources"] == [] and seen["cleanup_management_sources_configured"] is True, name
assert seen["cleanup_max_rules"] == 10 and seen["operators_may_deploy"] is True
# ── Locks in both directions ──────────────────────────────────────────────
async def _central_rule(client, env, **extra):
resp = await client.post("/api/rules", json={
"description": "managed ssh", "interface": "lan", "direction": "in", "protocol": "tcp", "action": "pass",
"source_net": "any", "destination_net": "lan", "destination_port": "22",
"target_host_ids": [str(env.host_id)], **extra})
assert resp.status_code == 201, resp.text
return resp.json()
async def _deployed_row(rule_id, host_id, firewall_uuid, status="success"):
async with SessionLocal() as db:
dep = RuleDeployment(rule_id=uuid_lib.UUID(rule_id), host_id=host_id, status=status, opnsense_rule_uuid=firewall_uuid)
db.add(dep)
await db.commit()
@pytest.mark.asyncio
async def test_an_open_cleanup_blocks_other_writes_to_the_firewall(client, env):
rule = await _central_rule(client, env)
alias = await client.post("/api/aliases", json={"name": "cl_alias", "type": "host", "content": "203.0.113.5",
"target_host_ids": [str(env.host_id)]})
assert alias.status_code == 201, alias.text
hid = env.host_id
await _open(client, env)
writes = list(env.fw.writes())
blocked = [
("POST", f"/api/rules/{rule['id']}/deploy", None),
("POST", f"/api/aliases/{alias.json()['id']}/deploy", None),
("POST", f"/api/hosts/{hid}/snapshots", {"name": "x"}),
("POST", f"/api/hosts/{hid}/services/dhcpd/restart", None),
("POST", f"/api/hosts/{hid}/maintenance/check-updates", None),
("POST", f"/api/hosts/{hid}/network/changes", {"kind": "vlan", "action": "add", "fields": {}, "preview_token": "0" * 64}),
("DELETE", f"/api/hosts/{hid}/snapshots/abc", None),
]
for method, path, body in blocked:
r = await client.request(method, path, **({"json": body} if body is not None else {}))
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", (path, r.text)
assert env.fw.writes() == writes # nothing reached the firewall
# the cleanup's own follow-up steps are not blocked by itself, and a second cleanup is refused with its own code
second = await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_BLOCK], "acknowledge": [R_BLOCK]})
assert second.status_code == 409 and second.json()["detail"]["code"] == "BEREINIGUNG_OFFENE_OPERATION"
assert second.json()["detail"]["params"]["reason"] == "cleanup"
op = (await client.get(_base(env))).json()[0]
assert (await client.post(f"{_base(env)}/{op['id']}/revert")).status_code == 200
again = await client.post(f"/api/hosts/{hid}/maintenance/check-updates")
assert again.status_code != 409 or again.json()["detail"]["code"] != "OPEN_CLEANUP_BLOCKS_DEPLOY"
@pytest.mark.asyncio
async def test_the_block_covers_the_cluster_and_the_maintenance_inspection(cluster, client):
from app.services.host_maintenance_service import _shared_change_blockers, ensure_no_maintenance, ensure_no_open_cleanup, peers
pv = await client.post(_base(cluster.a) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})
assert pv.status_code == 200, pv.text
assert (await client.post(f"{_base(cluster.a)}/{pv.json()['preview_id']}/deploy", json={"confirm_name": "fw-a"})).status_code == 200
async with SessionLocal() as db:
a, b = await db.get(OpnsenseHost, cluster.a), await db.get(OpnsenseHost, cluster.b)
for host in (a, b): # the partner node is just as busy
for call in (ensure_no_maintenance, ensure_no_open_cleanup):
with pytest.raises(Exception) as info:
await call(db, [host])
assert info.value.detail["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
blockers = await _shared_change_blockers(db, await peers(db, b))
assert any("Bereinigung" in text for text in blockers)
await ensure_no_maintenance(db, [a], cleanup=False) # the cleanup service's own check does not trip over itself
# deleting or re-addressing either node is refused while it is open
for host_id in (cluster.a, cluster.b):
r = await client.delete(f"/api/hosts/{host_id}")
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", host_id
op = (await client.get(_base(cluster.a))).json()[0]
assert (await client.post(f"{_base(cluster.a)}/{op['id']}/revert")).status_code == 200
async with SessionLocal() as db:
assert await _shared_change_blockers(db, await peers(db, await db.get(OpnsenseHost, cluster.b))) == []
await ensure_no_maintenance(db, [await db.get(OpnsenseHost, cluster.b)])
@pytest.mark.asyncio
async def test_host_delete_and_readdressing_are_refused_while_a_cleanup_is_open(client, env):
hid = env.host_id
await _open(client, env)
gone = await client.delete(f"/api/hosts/{hid}")
assert gone.status_code == 409 and gone.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
for body in ({"base_url": "https://203.0.113.99"}, {"enabled": False}, {"cluster_name": "ha-x"}, {"clear_cluster_name": True}):
r = await client.patch(f"/api/hosts/{hid}", json=body)
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", body
assert (await client.patch(f"/api/hosts/{hid}", json={"name": "fw-a2"})).status_code == 200 # a rename is harmless
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, hid)
assert host.base_url == FW1 and host.enabled and host.name == "fw-a2"
op = (await client.get(_base(env))).json()[0]
assert (await client.post(f"{_base(env)}/{op['id']}/revert")).status_code == 200
assert (await client.delete(f"/api/hosts/{hid}")).status_code == 204 # settled: the host may go (its runs go with it)
async with SessionLocal() as db:
assert (await db.scalars(select(RuleCleanupOperation))).all() == []
@pytest.mark.asyncio
async def test_a_confirmed_cleanup_blocks_nothing(client, env):
op = await _open(client, env)
assert (await client.post(f"{_base(env)}/{op['id']}/confirm")).status_code == 200
async with SessionLocal() as db:
assert not await svc.has_open_cleanup(db, [env.host_id])
r = await client.post(f"/api/hosts/{env.host_id}/maintenance/check-updates")
assert r.status_code != 409 or r.json()["detail"]["code"] != "OPEN_CLEANUP_BLOCKS_DEPLOY"
@pytest.mark.asyncio
async def test_rule_deletion_and_removal_of_targets_wait_for_the_cleanup(client, env):
rule = await _central_rule(client, env, target_host_ids=[])
await _deployed_row(rule["id"], env.host_id, R_TAIL)
token = (await client.get(f"/api/rules/{rule['id']}/preview")).json()["token"]
await _open(client, env)
env.fw.calls.clear()
deleted = await client.delete(f"/api/rules/{rule['id']}")
assert deleted.status_code == 409 and deleted.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
removed = await client.post(f"/api/rules/{rule['id']}/deployments/{env.host_id}/remove", json={"preview_token": token})
assert removed.status_code == 409 and removed.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", removed.text
assert env.fw.writes() == [] and env.fw.get(R_TAIL) is not None
op = (await client.get(_base(env))).json()[0]
await client.post(f"{_base(env)}/{op['id']}/revert")
assert (await client.delete(f"/api/rules/{rule['id']}")).status_code == 204 # the same call works afterwards
assert env.fw.get(R_TAIL) is None
@pytest.mark.asyncio
async def test_hierarchy_rule_deletion_and_reordering_wait_for_the_cleanup(client, env):
from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeployment
from tests.test_policy_hierarchy import _create_group, _hrule_payload
group = await _create_group(client, "Cleanup-Root")
assert (await client.patch(f"/api/hosts/{env.host_id}", json={"policy_hierarchy_group_id": group["id"]})).status_code == 200
rule = (await client.post("/api/policy-hierarchy/rules", json=_hrule_payload(group["id"], description="guard"))).json()
async with SessionLocal() as db:
db.add(PolicyHierarchyRuleDeployment(policy_hierarchy_id=uuid_lib.UUID(rule["id"]), host_id=env.host_id, status="success",
opnsense_policy_hierarchy_uuid=R_TAIL))
await db.commit()
await _open(client, env)
env.fw.calls.clear()
deleted = await client.delete(f"/api/policy-hierarchy/rules/{rule['id']}")
assert deleted.status_code == 409 and deleted.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", deleted.text
reorder = await client.post(f"/api/policy-hierarchy/groups/{group['id']}/reassert-ordering")
assert reorder.status_code == 409 and reorder.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", reorder.text
assert env.fw.writes() == []
@pytest.mark.asyncio
async def test_removing_a_former_hierarchy_target_waits_for_the_cleanup(client, env):
from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeployment
from tests.test_policy_hierarchy import _create_group, _hrule_payload
group = await _create_group(client, "Cleanup-Former")
rule = (await client.post("/api/policy-hierarchy/rules", json=_hrule_payload(group["id"], description="former"))).json()
async with SessionLocal() as db: # deployed on the host, which is not in the group (any more): "retained"
db.add(PolicyHierarchyRuleDeployment(policy_hierarchy_id=uuid_lib.UUID(rule["id"]), host_id=env.host_id, status="success",
opnsense_policy_hierarchy_uuid=R_TAIL))
await db.commit()
plan = (await client.get(f"/api/policy-hierarchy/rules/{rule['id']}/preview")).json()
assert [h["can_remove"] for h in plan["hosts"]] == [True]
await _open(client, env)
env.fw.calls.clear()
r = await client.post(f"/api/policy-hierarchy/rules/{rule['id']}/deployments/{env.host_id}/remove", json={"preview_token": plan["token"]})
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
assert env.fw.writes() == [] and env.fw.get(R_TAIL) is not None
@pytest.mark.asyncio
async def test_adopting_a_rule_waits_for_the_cleanup(client, env):
await _open(client, env)
r = await client.post("/api/rules/adopt", json={"host_id": str(env.host_id), "rule_uuid": R_BASE})
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
@pytest.mark.asyncio
async def test_other_open_work_blocks_a_new_cleanup_over_the_api(client, env):
url = _base(env) + "/preview"
body = {"kind": "disable", "uuids": [R_UNUSED]}
async with SessionLocal() as db:
db.add(HostOperation(host_id=env.host_id, host_name="fw-a", actor="admin", action="reboot", status="submitted",
expires_at=_now(), fingerprint="x", snapshot={}))
await db.commit()
r = await client.post(url, json=body)
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
async with SessionLocal() as db:
await db.execute(update(HostOperation).values(status="done"))
await db.commit()
rule = await _central_rule(client, env)
await _deployed_row(rule["id"], env.host_id, R_TAIL, status="pending")
r = await client.post(url, json=body)
assert r.status_code == 409 and r.json()["detail"]["code"] == "BEREINIGUNG_OFFENE_OPERATION" and r.json()["detail"]["params"]["reason"] == "deployment"
async with SessionLocal() as db:
await db.execute(update(RuleDeployment).values(status="success"))
await db.commit()
assert (await client.post(url, json=body)).status_code == 200
assert env.fw.writes() == []
# ── Notification ──────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_failed_reverts_and_unclear_runs_are_reported_with_stable_keys(client, env):
op = await _open(client, env)
async with SessionLocal() as db:
assert not [k for k in await notification_checks.check_revert_failed(db) if k.startswith("rule_cleanup")]
# a run that ended half-way (decision needed): one problem for the run
async with SessionLocal() as db:
await db.execute(update(RuleCleanupOperation).where(RuleCleanupOperation.id == uuid_lib.UUID(op["id"])).values(status="partial"))
await db.commit()
problems = await notification_checks.check_revert_failed(db)
key = f"rule_cleanup_op:{op['id']}"
assert set(problems) == {key} and problems[key]["deployment_type"] == "rule_cleanup"
assert problems[key]["host_name"] == "fw-a" and "teilweise" in problems[key]["hinweis"]
# the automatic revert failed: one problem per rule that is still changed (the key is the item, not the run)
env.fw.fail[("toggle", R_DUP)] = httpx.Response(200, json={"result": "failed"})
await client.post(f"{_base(env)}/{op['id']}/revert")
items = await _items(op["id"])
assert (await _op(op["id"])).status == "revert_failed"
async with SessionLocal() as db:
first = await notification_checks.check_revert_failed(db)
second = await notification_checks.check_revert_failed(db)
stuck = f"rule_cleanup:{items[R_DUP].id}"
assert set(first) == {stuck} and set(second) == set(first) # stable: the same problem is not reported twice
assert first[stuck]["entity_id"] == R_DUP and first[stuck]["host_name"] == "fw-a" and "Erneut zurücknehmen" in first[stuck]["hinweis"]
assert first[stuck]["operation_status"] == "revert_failed" and first[stuck]["item_state"] == "revert_failed"
# solved: gone
del env.fw.fail[("toggle", R_DUP)]
assert (await client.post(f"{_base(env)}/{op['id']}/retry")).json()["status"] == "reverted"
async with SessionLocal() as db:
assert not [k for k in await notification_checks.check_revert_failed(db) if k.startswith("rule_cleanup")]
@pytest.mark.asyncio
async def test_an_unclear_delete_is_reported_until_it_is_read_again(client, env):
await _confirmed_disable(env.host_id, [R_UNUSED])
pv = (await client.post(_base(env) + "/preview", json={"kind": "delete", "uuids": [R_UNUSED]})).json()
env.fw.fail[("delete", R_UNUSED)] = httpx.ReadTimeout
def _then_unreadable(kind, rule_uuid): # the firewall stops answering once delRule went out
if kind == "delete":
env.fw.fail[("get", R_UNUSED)] = httpx.ReadTimeout
env.fw.on_call = _then_unreadable
op = (await client.post(f"{_base(env)}/{pv['preview_id']}/deploy", json={"confirm_name": "fw-a"})).json()
assert op["status"] == "failed" and op["items"][0]["state"] == "unknown" and op["can_retry"] is True
async with SessionLocal() as db:
problems = await notification_checks.check_revert_failed(db)
key = f"rule_cleanup:{op['items'][0]['id']}"
assert set(problems) == {key} and problems[key]["operation"] == "delete" and "unklar" in problems[key]["hinweis"]
env.fw.on_call = None
del env.fw.fail[("get", R_UNUSED)]
del env.fw.fail[("delete", R_UNUSED)]
assert (await client.post(f"{_base(env)}/{op['id']}/retry")).status_code == 200
async with SessionLocal() as db:
assert not [k for k in await notification_checks.check_revert_failed(db) if k.startswith("rule_cleanup")]
# ── MCP and the schema ────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_mcp_stays_read_only_for_the_cleanup(client, env, who):
from app.routers.mcp import TOOLS
names = {t["name"] for t in TOOLS}
assert not [n for n in names if "cleanup" in n and n != "rule_cleanup_candidates"]
assert all(t.get("path", "").startswith("/api/") for t in TOOLS if "path" in t)
op = await _open(client, env)
def call(path):
return {"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "api_get", "arguments": {"path": path}}}
async with _bearer((await _make(client, "admin", name="mcp-admin"))["token"]) as api:
for tail in ("/preview", f"/{op['id']}/confirm", f"/{op['id']}/revert", f"/{op['id']}/deploy", f"/{op['id']}/retry"):
result = (await api.post("/api/mcp", json=call(f"/api/hosts/{env.host_id}/rule-cleanup{tail}"))).json()["result"]
assert result["isError"] is True, tail # a GET on a POST route: refused, nothing runs
listing = (await api.post("/api/mcp", json=call(f"/api/hosts/{env.host_id}/rule-cleanup"))).json()["result"]
assert listing["isError"] is False and op["id"] in listing["content"][0]["text"]
assert (await _op(op["id"])).status == "awaiting_confirm"
# a plain POST of a tool name that would write does not exist
async with _bearer((await _make(client, "admin", name="mcp-admin2"))["token"]) as api:
r = (await api.post("/api/mcp", json={"jsonrpc": "2.0", "id": 2, "method": "tools/call",
"params": {"name": "rule_cleanup_deploy", "arguments": {}}})).json()
assert "error" in r or r["result"]["isError"] is True
@pytest.mark.asyncio
async def test_openapi_knows_the_routes_and_their_models(client):
schema = (await client.get("/api/openapi.json")).json()
base = "/api/hosts/{host_id}/rule-cleanup"
paths = schema["paths"]
assert set(paths[base]) == {"get"} and set(paths[base + "/preview"]) == {"post"} and set(paths[base + "/{op_id}"]) == {"get"}
for tail in ("cancel", "deploy", "confirm", "revert", "retry"):
assert set(paths[f"{base}/{{op_id}}/{tail}"]) == {"post"}, tail
assert set(paths[base + "/items/{item_id}/restore"]) == {"post"}
assert {"CleanupOperationRead", "CleanupItemRead", "CleanupPreviewRead", "PreviewRequest", "ConfirmName"} <= set(schema["components"]["schemas"])
ops = schema["components"]["schemas"]["CleanupOperationRead"]["properties"]
assert {"can_confirm", "can_revert", "can_retry", "plan", "warnings", "items", "revert_deadline", "message"} <= set(ops)
assert {"restorable", "state", "host_name", "has_snapshot"} <= set(schema["components"]["schemas"]["CleanupItemRead"]["properties"])
def test_every_new_error_code_has_a_text_in_both_languages():
import json
import pathlib
from app.errors import ErrorCode
root = pathlib.Path(__file__).resolve().parents[2] / "frontend" / "lib" / "i18n"
if not root.is_dir():
pytest.skip("the frontend is not part of this image (scripts/check_i18n.py checks the same from the repo root)")
for lang in ("de", "en"):
errors = json.loads((root / f"{lang}.json").read_text(encoding="utf-8"))["errors"]
for code in ErrorCode:
if code.name.startswith("BEREINIGUNG_") or code.name == "OPEN_CLEANUP_BLOCKS_DEPLOY":
assert errors.get(code.value), (lang, code.value)
@pytest.mark.asyncio
async def test_a_refused_operator_attempt_is_audited_and_writes_nothing(client, env, who):
op = await _open(client, env)
r = await who["operator"].post(f"{_base(env)}/{op['id']}/retry")
assert r.status_code == 403
denied = await _audit(kind="host", action="retry", status="denied")
assert [a.actor for a in denied] == ["cl-operator"] and denied[0].host_id == str(env.host_id)
assert (await _op(op["id"])).status == "awaiting_confirm"
def _leaks(value) -> bool:
"""Anything that names a management source or a firewall's base URL."""
import json as _json
text = _json.dumps(value)
return MGMT_SOURCE in text or "203.0.113.57" in text
@pytest.mark.asyncio
async def test_sources_and_firewall_addresses_are_for_admins_only(client, env, who): # review finding 8
mine = await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})
assert mine.json()["plan"]["sources"] == [f"{MGMT_SOURCE}/32"] and mine.json()["plan"]["nodes"][0]["base_url"]
theirs = await who["operator"].post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_DUP]})
assert theirs.status_code == 200, theirs.text
body = theirs.json()
assert "sources" not in body["plan"] and body["plan"]["sources_configured"] is True and not _leaks(body)
assert body["plan"]["nodes"][0]["items"][0]["protection"]["result"] == "false"
for name in ("operator", "read_only", "tok_read", "tok_deploy", "tok_ro_read"):
listed = await who[name].get(_base(env))
assert listed.status_code == 200 and listed.json() and not _leaks(listed.json()), name
one = await who[name].get(f"{_base(env)}/{mine.json()['preview_id']}")
assert one.status_code == 200 and not _leaks(one.json()), name
audit = await who[name].get("/api/audit", params={"kind": "rule_cleanup"})
assert audit.status_code == 200 and audit.json()["items"] and not _leaks(audit.json()), name
assert {i["details"]["admin_only"] for i in audit.json()["items"]} == {"redacted"}, name
for name in ("admin", "tok_admin"):
assert _leaks((await who[name].get(_base(env))).json()), name
audit = (await who[name].get("/api/audit", params={"kind": "rule_cleanup"})).json()
assert all(isinstance(i["details"]["admin_only"], dict) for i in audit["items"]), name
# Review finding 9: undeploying a central object and removing a former target write to the firewall too.
_UNDEPLOY_SITES = [
("app.services.alias_deploy_service", "undeploy_alias"), ("app.services.openvpn_deploy_service", "undeploy_instance"),
("app.services.wireguard_deploy_service", "undeploy_tunnel"), ("app.services.port_forward_deploy_service", "undeploy_port_forward"),
("app.services.outbound_nat_deploy_service", "undeploy_outbound_nat"), ("app.services.one_to_one_deploy_service", "undeploy_one_to_one"),
("app.services.ipsec_deploy_service", "undeploy_tunnel"),
] + [(f"app.routers.{r}", "remove_previous_target") for r in
("aliases", "port_forward", "one_to_one", "outbound_nat", "wireguard", "openvpn", "ipsec")]
@pytest.mark.parametrize("module,name", _UNDEPLOY_SITES)
def test_every_undeploy_and_target_removal_checks_for_an_open_cleanup_first(module, name):
import importlib
import inspect
source = inspect.getsource(getattr(importlib.import_module(module), name))
lock = min(i for i in (source.find("ensure_no_open_cleanup_on("), source.find("ensure_no_open_cleanup(")) if i >= 0)
first_write = source.find("opnsense_client.")
first_write = min(i for i in (first_write, source.find("client.del_"), source.find("_client.del")) if i >= 0)
assert 0 <= lock < first_write, (module, name)
@pytest.mark.asyncio
async def test_alias_delete_and_target_removal_wait_for_the_cleanup(client, env):
from app.models.alias_deployment import AliasDeployment
deployed = await client.post("/api/aliases", json={"name": "cl_gone", "type": "host", "content": "203.0.113.5",
"target_host_ids": [str(env.host_id)]})
former = await client.post("/api/aliases", json={"name": "cl_former", "type": "host", "content": "203.0.113.6",
"target_host_ids": []})
assert deployed.status_code == 201 and former.status_code == 201, (deployed.text, former.text)
async with SessionLocal() as db:
for alias in (deployed.json(), former.json()):
db.add(AliasDeployment(alias_id=uuid_lib.UUID(alias["id"]), host_id=env.host_id, status="success",
opnsense_alias_uuid="aaaaaaaa-1111-4111-8111-111111111111", last_payload={"name": alias["name"]}))
await db.commit()
preview = (await client.get(f"/api/aliases/{former.json()['id']}/preview")).json()
assert next(h for h in preview["hosts"] if h["host_id"] == str(env.host_id))["can_remove"], preview
await _open(client, env)
writes = list(env.fw.writes())
r = await client.delete(f"/api/aliases/{deployed.json()['id']}")
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
preview = (await client.get(f"/api/aliases/{former.json()['id']}/preview")).json()
r = await client.post(f"/api/aliases/{former.json()['id']}/deployments/{env.host_id}/remove", json={"preview_token": preview["token"]})
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
assert env.fw.writes() == writes
@@ -0,0 +1,193 @@
"""Pure parts of the rule cleanup (1.31): engine.match_rule, the management-access protection, the infrastructure
exclusion, the confidence caps and the cluster usage merge. No database, no firewall."""
import ipaddress
from datetime import datetime, timedelta, timezone
from app.services import rule_hit_service
from app.services.rule_analysis import cleanup, protection
from app.services.rule_analysis.context import Context, IfInfo
from app.services.rule_analysis.engine import Query, match_rule
from app.services.rule_analysis.model import Endpoint, NormRule
from app.services.rule_cleanup_service import selectable_level
NOW = datetime(2026, 10, 22, 12, 0, tzinfo=timezone.utc)
ip = ipaddress.ip_address
net = ipaddress.ip_network
def ctx(**kw) -> Context:
interfaces = {
"wan": IfInfo("wan", "vtnet0", "WAN", nets=[net("203.0.113.0/24")], addrs=[ip("203.0.113.57")], has_gateway=True),
"lan": IfInfo("lan", "vtnet1", "LAN", nets=[net("10.20.30.0/24")], addrs=[ip("10.20.30.2")], vips=[ip("10.20.30.1")]),
}
return Context(rows=[], pf=None, interfaces=interfaces, groups={}, aliases=kw.get("aliases", {}), port_forwards=[])
def nrule(*, src="", dst="", dport="", protos=frozenset({"6"}), interfaces=frozenset({"wan"}), action="pass", negated=False,
tagged="", uuid="u1") -> NormRule:
items = [i for i in src.split(",") if i]
return NormRule(uuid=uuid, label=uuid, description=uuid, order=(400000, 1), order_known=True, section="interface",
enabled=True, action=action, quick=True, interfaces=set(interfaces), if_not=False, direction="in",
af={4}, protos=None if protos is None else set(protos), src=Endpoint(raw=src, negated=negated, items=items),
dst=Endpoint(raw=dst, items=[i for i in dst.split(",") if i]), sport="", dport=dport, tagged=tagged)
SOURCES = [net("203.0.113.10/32")]
def check(rule, c=None, sources=SOURCES):
c = c or ctx()
targets, ports = protection.management_targets("https://203.0.113.57:4443", c)
return protection.management_check(rule, c, sources, targets, ports)
def test_targets_are_the_api_address_and_port_plus_every_firewall_address():
targets, ports = protection.management_targets("https://203.0.113.57:4443", ctx())
assert targets[0] == ip("203.0.113.57") and ip("10.20.30.1") in targets and ip("127.0.0.1") not in targets
assert ports == [4443, 443, 80, 22]
def test_a_rule_for_the_management_path_is_protected():
assert check(nrule(src="203.0.113.0/24", dport="443"))["result"] == "true"
assert check(nrule(src="", dport="4443"))["result"] == "true" # any source, API port
assert check(nrule(src="203.0.113.10", dport="", protos=None))["result"] == "true"
def test_only_a_proven_non_match_is_removable():
assert check(nrule(src="10.20.30.0/24", dport="443"))["result"] == "false" # other source
assert check(nrule(src="203.0.113.0/24", dport="8081"))["result"] == "false" # other port
assert check(nrule(src="203.0.113.0/24", dport="443", protos={"17"}))["result"] == "false" # udp
def test_unknown_is_protected(): # case 18
unknown = check(nrule(src="tbl", dport="443"), ctx(aliases={"tbl": {"name": "tbl", "type": "urltable"}}))
assert unknown["result"] == "unknown" and unknown["match"]["port"] in (4443, 443)
tagged = check(nrule(src="", dport="443", tagged="mark"))
assert tagged["result"] == "unknown" and "TAGGED" in tagged["reasons"]
q = Query(src=ip("203.0.113.10"), dst=ip("203.0.113.57"), proto="6", proto_label="tcp", dport=443)
assert match_rule(nrule(interfaces={"nosuchgroup"}), q, ctx(), iface="wan").value is None
def test_network_sources_are_judged_for_every_address():
small = [net("203.0.113.0/28")]
assert check(nrule(src="203.0.113.14", dport="443"), sources=small)["result"] == "true" # one address inside
assert check(nrule(src="203.0.113.20", dport="443"), sources=small)["result"] == "false"
big = [net("10.0.0.0/8")]
assert check(nrule(src="10.20.30.0/24", dport="443"), sources=big)["result"] == "true" # ranges overlap
assert check(nrule(src="192.0.2.0/24", dport="443"), sources=big)["result"] == "false"
assert check(nrule(src="lan", dport="443"), sources=big)["result"] == "true" # interface network
assert check(nrule(src="10.0.0.0/8", dport="443", negated=True), sources=big)["result"] == "false" # !whole net
assert check(nrule(src="tbl", dport="443"), ctx(aliases={"tbl": {"type": "urltable"}}), sources=big)["result"] == "unknown"
def test_no_query_possible_is_protected():
got = check(nrule(src="", dport="443"), sources=[net("2001:db8::/64")]) # no IPv6 address on the firewall
assert got["result"] == "unknown" and got["reasons"] == ["NO_MANAGEMENT_QUERY"]
def test_carp_and_pfsync_rules_are_infrastructure():
c = ctx()
assert protection.infrastructure_reason(nrule(protos={"112"}), c, hasync=None, clustered=False) == "infrastructure"
assert protection.infrastructure_reason(nrule(protos={"240"}), c, hasync=None, clustered=False) == "infrastructure"
lan_any = nrule(src="10.20.30.0/24", protos=None, interfaces={"lan"})
assert protection.infrastructure_reason(lan_any, c, hasync=None, clustered=False) == "infrastructure" # CARP adverts
lan_tcp = nrule(src="10.20.30.0/24", dport="8081", interfaces={"lan"})
assert protection.infrastructure_reason(lan_tcp, c, hasync=None, clustered=True) is None
wan_any = nrule(src="", protos=None, interfaces={"wan"})
off = {"hasync": {"pfsyncinterface": {"": {"selected": 1}, "wan": {"selected": 0}}, "pfsyncpeerip": ""}}
on = {"hasync": {"pfsyncinterface": {"": {"selected": 0}, "wan": {"selected": 1}}, "pfsyncpeerip": "203.0.113.58"}}
assert protection.infrastructure_reason(wan_any, c, hasync=off, clustered=True) is None
assert protection.infrastructure_reason(wan_any, c, hasync=on, clustered=True) == "infrastructure"
# HA settings not readable on a cluster node: every interface counts as a possible pfsync interface
assert protection.infrastructure_reason(wan_any, c, hasync=None, clustered=True) == "infrastructure"
def test_confidence_caps():
rule = nrule()
unused = {"categories": [{"kind": "unused", "confidence": "sure"}]}
assert selectable_level(unused, rule, unused_days=30, external=False)[0] == "sure"
assert selectable_level(unused, rule, unused_days=29, external=False)[0] == "probable"
assert selectable_level(unused, nrule(action="block"), unused_days=90, external=False)[0] == "probable"
assert selectable_level(unused, nrule(action="reject"), unused_days=90, external=False)[0] == "probable"
assert selectable_level(unused, rule, unused_days=90, external=True)[0] == "probable"
both = {"categories": [{"kind": "unused", "confidence": "unknown"}, {"kind": "duplicate", "confidence": "sure"}]}
assert selectable_level(both, nrule(action="block"), unused_days=7, external=False)[0] == "sure" # best category
assert selectable_level({"categories": [{"kind": "shadowed", "confidence": "unknown"}]}, rule, unused_days=30, external=False)[0] == "unknown"
assert selectable_level({"categories": [{"kind": "disabled_long", "confidence": "sure"}]}, rule, unused_days=30, external=False)[0] is None
class Hit:
def __init__(self, uuid, first_days, hit_days=None, sampled_minutes=10, enabled=True):
self.rule_uuid, self.enabled = uuid, enabled
self.first_seen_at = NOW - timedelta(days=first_days)
self.last_hit_at = None if hit_days is None else NOW - timedelta(days=hit_days)
self.last_sampled_at = NOW - timedelta(minutes=sampled_minutes)
def test_cluster_usage_merge(): # case 16
master = {"r": Hit("r", 100, hit_days=1)}
backup = {"r": Hit("r", 100)}
merged = rule_hit_service.merge_cluster_hits(backup, [master])
assert rule_hit_service.classify(merged["r"], NOW, 30) == "used"
both_idle = rule_hit_service.merge_cluster_hits({"r": Hit("r", 100)}, [{"r": Hit("r", 20)}])
assert rule_hit_service.classify(both_idle["r"], NOW, 30) == "collecting" # window = latest start
stale = rule_hit_service.merge_cluster_hits({"r": Hit("r", 100)}, [{"r": Hit("r", 100, sampled_minutes=60 * 48)}])
assert stale["r"].last_sampled_at == NOW - timedelta(hours=48) # as fresh as the oldest node
missing = rule_hit_service.merge_cluster_hits({"r": Hit("r", 100)}, [{}])
assert missing["r"].peer_missing and rule_hit_service.classify(missing["r"], NOW, 30) == "unused"
rule = nrule()
rule.loaded = True
assert cleanup.unused_confidence(rule, missing["r"], NOW) == "probable"
assert cleanup.unused_confidence(rule, both_idle["r"], NOW) == "sure"
def _live_rule(**kw):
"""A rule normalized inside the live fixture (tests/fixtures/rule_analysis_fw.json: built-in groups wireguard,
openvpn, enc0 with members "" and unassigned devices without identifier)."""
from app.services.rule_analysis.normalize import normalize
from tests.test_rule_analysis import build, mvc
row = mvc("bbbbbbbb-0000-4000-8000-000000000001", seq=900, proto="tcp", **kw)
c = build([row])
rules, _ = normalize(c)
return next(r for r in rules if r.uuid == row["uuid"]), c
def test_rules_on_builtin_groups_without_known_members_are_protected(): # review finding 1
for group in ("wireguard", "openvpn", "enc0"):
rule, c = _live_rule(interface=group, src="203.0.113.10", dst="(self)", dport="443")
got = check(rule, c)
assert got["result"] == "unknown" and "IFACE_MEMBERS_UNKNOWN" in got["reasons"], (group, got)
assert got["match"]["interface"] == protection.UNASSIGNED
# the virtual ingress does not make every rule protected
rule, c = _live_rule(interface="wireguard", src="203.0.113.10", dst="(self)", dport="8081")
assert check(rule, c)["result"] == "false"
rule, c = _live_rule(interface="lan", src="10.20.30.0/24", dst="(self)", dport="443")
assert check(rule, c)["result"] == "false"
# floating "any interface" and "not lan" also match a packet on wg0
rule, c = _live_rule(interface="", src="203.0.113.10", dst="(self)", dport="22")
assert check(rule, c)["result"] == "true"
rule, c = _live_rule(interface="lan", interfacenot="1", src="203.0.113.10", dst="(self)", dport="22")
assert check(rule, c)["result"] == "true"
def test_unassigned_interface_outcome():
c = ctx()
c.groups = {"mygroup": {"lan"}, "wireguard": set()}
assert protection.unassigned_interface(nrule(interfaces={"lan"}), c).value is False
assert protection.unassigned_interface(nrule(interfaces={"mygroup"}), c).value is False
assert protection.unassigned_interface(nrule(interfaces={"wireguard"}), c).value is None
assert protection.unassigned_interface(nrule(interfaces={"nosuch"}), c).value is None
assert protection.unassigned_interface(nrule(interfaces=set()), c).value is True
def test_redacted_plan_names_no_source_port_or_firewall_address(): # review finding 8
from app.services.rule_cleanup_service import redact
plan = {"kind": "enable", "sources": ["203.0.113.10/32"], "ports": [2222], "nodes": [
{"host_id": "h", "base_url": "https://203.0.113.57", "items": [
{"uuid": "u", "protection": {"result": "true", "queries": 3, "reasons": [],
"match": {"source": "203.0.113.10/32", "destination": "203.0.113.57"}}}]}]}
out = redact({"plan": plan, "operation": {"plan": plan}})
for p in (out["plan"], out["operation"]["plan"]):
assert "203.0.113" not in str(p) and "2222" not in str(p)
assert p["sources_configured"] is True and p["nodes"][0]["items"][0]["protection"]["result"] == "true"
assert plan["sources"] and plan["nodes"][0]["base_url"] # the stored plan itself is untouched
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff. Show more