Compare commits

..
4 Commits
Author SHA1 Message Date
Julian Boldt d93b154660 Release v1.33.0
Release / verify (push) Waiting to run
Release / test (push) Blocked by required conditions
CI / backend (push) Blocked by required conditions
CI / frontend (push) Blocked by required conditions
CI / checks (push) Blocked by required conditions
- **Aus dem Testlauf 1.31.0 behoben (T2 bis T6, B15):**
  - **Benutzerliste:** `GET /api/users` zeigte jedem angemeldeten Konto (auch Nur-Lese, auch per Token) alle Benutzernamen, Rollen, letzte Anmeldung und 2FA-Status.
    Jetzt sehen **nur Admins** die ganze Liste, alle anderen nur das eigene Konto.
  - **Hosts:** Dieselbe Adresse zweimal einzutragen wird abgelehnt (`HOST_URL_EXISTS`, auch beim Ändern der Adresse). Die Hosts-Seite und der Einrichtungsassistent
    prüfen die Verbindung beim Anlegen (Schlüssel abgelehnt / zu wenig Rechte / Zertifikat / keine Verbindung), speichern bei einem Fehler nichts und bieten
    „Trotzdem anlegen“. Über die API ist das `verify_connection: true` (Standard aus). **Löschen verlangt jetzt auch am Server den Hostnamen:**
    `DELETE /api/hosts/{id}?confirm_name=<Name>` (sonst 409); Skripte, die Hosts löschen, müssen das anpassen.
  - **Zertifikate:** `cert_type: v3_ca` wird nicht mehr angenommen (es entstand ein Zertifikat mit CA-Erweiterung, keine nutzbare CA); die Oberfläche bot es nie an.
  - **Traceroute:** Zwischenstationen ohne Antwort stehen als Zeile „keine Antwort“ in der Liste (vorher sprang die Zählung 1 → 3); ein Weg ganz ohne Antwort wird erklärt.
  - **Netzwerk:** Die Vorschau prüft Zahlenbereiche vorab (VLAN-Tag 1–4094, PCP 0–7, VHID 1–255, Basis 1–254, Skew 0–254, gegen das Modell der OPNsense gelesen): `422` mit dem
    Feldnamen statt `502` von der Firewall.
  - **Gruppen:** Eine Gruppe, die ein VPN-Objekt als Ziel hat, lässt sich nicht mehr löschen (die Verknüpfung fiel still weg).
  - Das Skript `scripts/verify_i18n.mjs` läuft jetzt auch gegen die HTTPS-Test-Instanz.
- **Regel-Bereinigung: OPNexus selbst ist automatisch geschützt (B10).** Beim Planen fragt die Bereinigung jede Firewall, von welcher Adresse sie OPNexus kommen sieht
  (die eigene API-Verbindung in der Zustandstabelle), und schützt diese Adresse zusätzlich zu den Verwaltungsquellen. Vorher konnte eine Regel abgeschaltet werden, über
  die nur OPNexus die Firewall erreicht, wenn in den Quellen nur Arbeitsplätze standen; dann scheiterte auch das automatische Zurücksetzen. Die Vorschau zeigt die Adresse
  je Firewall (nur Admins) oder sagt, dass sie sich nicht ermitteln ließ (API-Recht „Diagnose: Zustände“, Management-Tunnel).
- **Keine verlorenen Regeltreffer mehr (B11):** Jedes Anwenden setzt die Regelzähler der Firewall zurück. OPNexus liest sie jetzt unmittelbar vor jedem Anwenden, das es
  selbst auslöst (Regeln, NAT, Aliase, IPsec/OpenVPN/WireGuard, Netzwerk); vorher konnte eine Regel deshalb fälschlich als „ungenutzt“ gelten.
- **Einstellungen › Regel-Bereinigung:** schlägt die eigene Browser-Adresse als Verwaltungsquelle vor (B12). Geprüft: OPNsense 26.7 hat kein „savepoint“ zum Zurücknehmen
  durch die Firewall selbst (Quellcode und live 404), das automatische Zurücksetzen bleibt bei OPNexus.

- **Installation unter Windows:** Neue Anleitung auf der Website (`/docs/windows/`, auch auf Englisch) und im Handbuch: Docker Desktop mit WSL 2, Zertifizierungsstelle importieren, Zugriff aus dem Netz (Windows-Firewall), Updates mit `opnexus` in WSL und Fehlerbehebung. Entwickelt und getestet wird weiterhin unter Linux; die Anleitung sagt das offen.
- **Sicherheit und Stabilität (aus einer externen Prüfung):**
  - **Rückweg bleibt erhalten:** Wer nach „Konfiguration geschrieben, Anwenden fehlgeschlagen“ erneut ausrollte, überschrieb den ursprünglichen Rückweg; die automatische
    Rücknahme stellte dann den unbestätigten Stand wieder her. Jetzt bleibt der Rückweg der Stand vor dem ersten Versuch. Wartung, Regel-Bereinigung und
    „Konfiguration als Code“ behandeln eine solche Zeile wie eine offene Bestätigung. Gilt für Regeln, Aliase, Port-Forwards, 1:1-NAT, Outbound-NAT und Hierarchie-Regeln.
  - **Anmeldung und 2FA:** Eine Passwort-Anmeldung, die genau während der Aktivierung von 2FA lief, bekam eine Sitzung ohne zweiten Faktor. Jetzt wird sie mit `TOTP_REQUIRED` abgewiesen.
  - **Letzter Administrator:** Zwei Administratoren, die sich gleichzeitig herabstuften oder gegenseitig löschten, konnten alle Administratoren entfernen. Beides ist jetzt serialisiert.
  - **Operatoren:** Mit ausgeschaltetem „Operatoren dürfen ausrollen“ konnten sie noch Snapshots (Boot-Umgebungen) anlegen. Ein Test hält künftig jede schreibende Route
    fest, die keiner Rechte-Regel zugeordnet ist.
  - **Zertifikate und ACME:** Der Ziel-Host ist nach dem ersten Push fest (`409 ZIEL_HOST_NACH_AUSROLLEN_FEST`); vorher meldete ein Host-Wechsel Erfolg für einen Host, der nie
    etwas erhielt. Ein ACME-Konto mit Anfragen lässt sich nicht auf einen anderen Host verschieben (`ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET`).
  - **VPN-Ausrollen:** IPsec, OpenVPN und WireGuard sichern jede von der Firewall vergebene Kennung sofort; bei einem Abbruch entstehen beim nächsten Versuch keine Duplikate.
    Pro VPN läuft nur ein Ausrollen oder Entfernen zugleich (`409 VPN_DEPLOY_LAEUFT_BEREITS`).
  - **Konfiguration als Code:** Gültiges YAML mit falsch geformtem Abschnitt (z. B. `aliases: 42`) ergibt `422 CONFIG_INVALID` statt eines Serverfehlers.
  - **`.env`:** Die Anleitungen legen sie mit `install -m 600` an; `opnexus update` und `opnexus status` setzen eine zu offene Datei auf 0600. `opnexus rollback` liest seinen
    Update-Zustand nur noch als Text und führt ihn nie aus.
  - **Abhängigkeiten:** Next.js 15.5.28, PyJWT 2.15.1.
2026-10-24 00:22:22 +02:00
Julian Boldt cf9981c6eb Release v1.32.1
Release / verify (push) Waiting to run
Release / test (push) Blocked by required conditions
CI / backend (push) Blocked by required conditions
CI / frontend (push) Blocked by required conditions
CI / checks (push) Blocked by required conditions
- **Korrektur: Rollouts erkannten die Zielversion bei anderen Release-Typen falsch.** OPNexus las die neue Version nur am Kernpaket `opnsense`. Bei der
  Business-Edition (`opnsense-business`) und beim Wechsel des Release-Typs (z. B. auf `opnsense-devel`, dann als neues Paket angeboten) fiel es auf
  `product_latest` zurück; das Gate hielt den Rollout dann fälschlich mit „Version passt nicht“ an. Live gefunden und belegt (Rollout mit Cluster in einer Welle).
- **Plan-Hinweis zu Alias-Namen ehrlich formuliert:** „kein zentraler Alias in OPNexus“ statt „kein Alias dieses Namens“ (von Hand auf der Firewall
  angelegte Aliase sind in Ordnung; der Planer fragt bewusst keine Firewall).
2026-10-23 12:40:49 +02:00
Julian Boldt 4eea6424e7 Release v1.32.0
Release / verify (push) Waiting to run
Release / test (push) Blocked by required conditions
CI / backend (push) Blocked by required conditions
CI / frontend (push) Blocked by required conditions
CI / checks (push) Blocked by required conditions
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
  Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
  ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
  das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
  ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
  nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
  und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
  Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
  **ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
  (Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
  vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
  damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
  vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
2026-10-23 11:27:47 +02:00
Julian Boldt a8ca0a14ef Release v1.31.1
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
  (`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
  SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
  ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
2026-10-22 23:45:57 +02:00
130 changed files with 7977 additions and 329 deletions

No files matched your search

+77 -1
View File
@@ -29,7 +29,83 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
den Einstellungen.
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
ansteht): Merlin …
ansteht): Saker …
## [1.33.0] "Saker" — 2026-10-24
- **Aus dem Testlauf 1.31.0 behoben (T2 bis T6, B15):**
- **Benutzerliste:** `GET /api/users` zeigte jedem angemeldeten Konto (auch Nur-Lese, auch per Token) alle Benutzernamen, Rollen, letzte Anmeldung und 2FA-Status.
Jetzt sehen **nur Admins** die ganze Liste, alle anderen nur das eigene Konto.
- **Hosts:** Dieselbe Adresse zweimal einzutragen wird abgelehnt (`HOST_URL_EXISTS`, auch beim Ändern der Adresse). Die Hosts-Seite und der Einrichtungsassistent
prüfen die Verbindung beim Anlegen (Schlüssel abgelehnt / zu wenig Rechte / Zertifikat / keine Verbindung), speichern bei einem Fehler nichts und bieten
„Trotzdem anlegen“. Über die API ist das `verify_connection: true` (Standard aus). **Löschen verlangt jetzt auch am Server den Hostnamen:**
`DELETE /api/hosts/{id}?confirm_name=<Name>` (sonst 409); Skripte, die Hosts löschen, müssen das anpassen.
- **Zertifikate:** `cert_type: v3_ca` wird nicht mehr angenommen (es entstand ein Zertifikat mit CA-Erweiterung, keine nutzbare CA); die Oberfläche bot es nie an.
- **Traceroute:** Zwischenstationen ohne Antwort stehen als Zeile „keine Antwort“ in der Liste (vorher sprang die Zählung 1 → 3); ein Weg ganz ohne Antwort wird erklärt.
- **Netzwerk:** Die Vorschau prüft Zahlenbereiche vorab (VLAN-Tag 1–4094, PCP 0–7, VHID 1–255, Basis 1–254, Skew 0–254, gegen das Modell der OPNsense gelesen): `422` mit dem
Feldnamen statt `502` von der Firewall.
- **Gruppen:** Eine Gruppe, die ein VPN-Objekt als Ziel hat, lässt sich nicht mehr löschen (die Verknüpfung fiel still weg).
- Das Skript `scripts/verify_i18n.mjs` läuft jetzt auch gegen die HTTPS-Test-Instanz.
- **Regel-Bereinigung: OPNexus selbst ist automatisch geschützt (B10).** Beim Planen fragt die Bereinigung jede Firewall, von welcher Adresse sie OPNexus kommen sieht
(die eigene API-Verbindung in der Zustandstabelle), und schützt diese Adresse zusätzlich zu den Verwaltungsquellen. Vorher konnte eine Regel abgeschaltet werden, über
die nur OPNexus die Firewall erreicht, wenn in den Quellen nur Arbeitsplätze standen; dann scheiterte auch das automatische Zurücksetzen. Die Vorschau zeigt die Adresse
je Firewall (nur Admins) oder sagt, dass sie sich nicht ermitteln ließ (API-Recht „Diagnose: Zustände“, Management-Tunnel).
- **Keine verlorenen Regeltreffer mehr (B11):** Jedes Anwenden setzt die Regelzähler der Firewall zurück. OPNexus liest sie jetzt unmittelbar vor jedem Anwenden, das es
selbst auslöst (Regeln, NAT, Aliase, IPsec/OpenVPN/WireGuard, Netzwerk); vorher konnte eine Regel deshalb fälschlich als „ungenutzt“ gelten.
- **Einstellungen › Regel-Bereinigung:** schlägt die eigene Browser-Adresse als Verwaltungsquelle vor (B12). Geprüft: OPNsense 26.7 hat kein „savepoint“ zum Zurücknehmen
durch die Firewall selbst (Quellcode und live 404), das automatische Zurücksetzen bleibt bei OPNexus.
- **Installation unter Windows:** Neue Anleitung auf der Website (`/docs/windows/`, auch auf Englisch) und im Handbuch: Docker Desktop mit WSL 2, Zertifizierungsstelle importieren, Zugriff aus dem Netz (Windows-Firewall), Updates mit `opnexus` in WSL und Fehlerbehebung. Entwickelt und getestet wird weiterhin unter Linux; die Anleitung sagt das offen.
- **Sicherheit und Stabilität (aus einer externen Prüfung):**
- **Rückweg bleibt erhalten:** Wer nach „Konfiguration geschrieben, Anwenden fehlgeschlagen“ erneut ausrollte, überschrieb den ursprünglichen Rückweg; die automatische
Rücknahme stellte dann den unbestätigten Stand wieder her. Jetzt bleibt der Rückweg der Stand vor dem ersten Versuch. Wartung, Regel-Bereinigung und
„Konfiguration als Code“ behandeln eine solche Zeile wie eine offene Bestätigung. Gilt für Regeln, Aliase, Port-Forwards, 1:1-NAT, Outbound-NAT und Hierarchie-Regeln.
- **Anmeldung und 2FA:** Eine Passwort-Anmeldung, die genau während der Aktivierung von 2FA lief, bekam eine Sitzung ohne zweiten Faktor. Jetzt wird sie mit `TOTP_REQUIRED` abgewiesen.
- **Letzter Administrator:** Zwei Administratoren, die sich gleichzeitig herabstuften oder gegenseitig löschten, konnten alle Administratoren entfernen. Beides ist jetzt serialisiert.
- **Operatoren:** Mit ausgeschaltetem „Operatoren dürfen ausrollen“ konnten sie noch Snapshots (Boot-Umgebungen) anlegen. Ein Test hält künftig jede schreibende Route
fest, die keiner Rechte-Regel zugeordnet ist.
- **Zertifikate und ACME:** Der Ziel-Host ist nach dem ersten Push fest (`409 ZIEL_HOST_NACH_AUSROLLEN_FEST`); vorher meldete ein Host-Wechsel Erfolg für einen Host, der nie
etwas erhielt. Ein ACME-Konto mit Anfragen lässt sich nicht auf einen anderen Host verschieben (`ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET`).
- **VPN-Ausrollen:** IPsec, OpenVPN und WireGuard sichern jede von der Firewall vergebene Kennung sofort; bei einem Abbruch entstehen beim nächsten Versuch keine Duplikate.
Pro VPN läuft nur ein Ausrollen oder Entfernen zugleich (`409 VPN_DEPLOY_LAEUFT_BEREITS`).
- **Konfiguration als Code:** Gültiges YAML mit falsch geformtem Abschnitt (z. B. `aliases: 42`) ergibt `422 CONFIG_INVALID` statt eines Serverfehlers.
- **`.env`:** Die Anleitungen legen sie mit `install -m 600` an; `opnexus update` und `opnexus status` setzen eine zu offene Datei auf 0600. `opnexus rollback` liest seinen
Update-Zustand nur noch als Text und führt ihn nie aus.
- **Abhängigkeiten:** Next.js 15.5.28, PyJWT 2.15.1.
## [1.32.1] "Merlin" — 2026-10-23
- **Korrektur: Rollouts erkannten die Zielversion bei anderen Release-Typen falsch.** OPNexus las die neue Version nur am Kernpaket `opnsense`. Bei der
Business-Edition (`opnsense-business`) und beim Wechsel des Release-Typs (z. B. auf `opnsense-devel`, dann als neues Paket angeboten) fiel es auf
`product_latest` zurück; das Gate hielt den Rollout dann fälschlich mit „Version passt nicht“ an. Live gefunden und belegt (Rollout mit Cluster in einer Welle).
- **Plan-Hinweis zu Alias-Namen ehrlich formuliert:** „kein zentraler Alias in OPNexus“ statt „kein Alias dieses Namens“ (von Hand auf der Firewall
angelegte Aliase sind in Ordnung; der Planer fragt bewusst keine Firewall).
## [1.32.0] "Merlin" — 2026-10-23
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
## [1.31.1] "Vulture" — 2026-10-22
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
## [1.31.0] "Vulture" — 2026-10-22
+7 -4
View File
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
![Zentrale Firewall-Regeln](docs/screenshots/readme-rules-de.webp)
> **Stand:** `1.31.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
> **Stand:** `1.33.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
## Funktionen
@@ -21,7 +21,8 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
@@ -47,7 +48,7 @@ Voraussetzung: Docker mit Compose-Plugin.
```bash
git clone https://git.opnexus.dev/opnexus/opnexus.git
cd opnexus
cp .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen
install -m 600 .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen; nur für dich lesbar
docker compose up -d --build
```
@@ -59,13 +60,15 @@ scripts/opnexus password
Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassistent**: eigenes Passwort setzen, optional Zwei-Faktor-Anmeldung einschalten, die erste Firewall verbinden (samt Anleitung für den API-Key in OPNsense). Er lässt sich überspringen und jederzeit unter System › Einrichtung wieder öffnen.
**Windows:** läuft in Docker Desktop mit WSL 2, siehe die [Installationsanleitung für Windows](https://opnexus.dev/docs/windows/). Entwickelt und getestet wird unter Linux; für den Dauerbetrieb ist ein Linux-Server oder eine Linux-VM besser.
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
```bash
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
OPNEXUS_VERSION=1.31.0
OPNEXUS_VERSION=1.33.0
```
```bash
+7 -4
View File
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
![Central firewall rules](docs/screenshots/readme-rules-en.webp)
> **Status:** `1.31.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
> **Status:** `1.33.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
## What it does
@@ -21,7 +21,8 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
@@ -49,7 +50,7 @@ Requires Docker with the Compose plugin.
```bash
git clone https://git.opnexus.dev/opnexus/opnexus.git
cd opnexus
cp .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD)
install -m 600 .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD); readable by you only
docker compose up -d --build
```
@@ -61,13 +62,15 @@ scripts/opnexus password
After the first sign-in a short **setup assistant** opens on its own: set your own password, optionally enable two-factor sign-in, connect your first firewall (including how to create the API key in OPNsense). You can skip it and reopen it any time under System › Setup.
**Windows:** run it in Docker Desktop with WSL 2 – see the [installation guide for Windows](https://opnexus.dev/en/docs/windows/). OPNexus is developed and tested on Linux; for permanent operation a Linux server or VM is the better choice.
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
```bash
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
OPNEXUS_VERSION=1.31.0
OPNEXUS_VERSION=1.33.0
```
```bash
+6 -3
View File
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
"/api/settings",
# Read-only JSON-RPC tools (see routers/mcp.py).
"/api/mcp",
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
"/api/config/import/plan",
# Self-scoped: a user managing their own second factor.
"/api/auth/2fa/setup",
"/api/auth/2fa/enable",
@@ -121,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
)
if user is not None:
parts = request.url.path.strip("/").split("/")
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
@@ -131,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
if len(parts) > 2 and parts[1] == "settings":
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
entity = parts[2] if len(parts) > 2 and kind != "config" else None
event = {"kind": kind, "action": action, "entity_id": entity,
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
"details": {"method": request.method, "path": request.url.path}}
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
if (
user.role == "read_only"
+5
View File
@@ -0,0 +1,5 @@
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
"""
+228
View File
@@ -0,0 +1,228 @@
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
"""
import argparse
import asyncio
import json
import re
import sys
import uuid
from app.config_code import apply as applier
from app.config_code import plan as planner
from app.config_code import schema
from app.config_code.export import export_yaml
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
ACTOR = "cli (Host-Konsole)"
TOKEN_MARK = "# plan-token: "
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
ERRORS_DE = {
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
"yaml_empty": "Die Datei ist leer",
"yaml_syntax": "YAML-Syntaxfehler",
"duplicate_key": "Doppelter Schlüssel",
"non_string_key": "Schlüssel muss Text sein",
"too_deep": "Zu tief verschachtelt",
"too_many_nodes": "Zu viele Elemente",
"scalar_too_long": "Wert zu lang",
"too_many_objects": "Zu viele Objekte",
"too_many_entries": "Zu viele Einträge",
"too_many_variables": "Zu viele Variablen für diesen Host",
"too_large": "Datei zu groß",
"bad_encoding": "Kein gültiges UTF-8",
"not_a_mapping": "Die Datei muss ein Mapping sein",
"missing_version": "opnexus_config: 1 fehlt",
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
"unknown_field": "Unbekanntes Feld",
"missing_field": "Pflichtfeld fehlt",
"bad_type": "Falscher Typ",
"bad_value": "Ungültiger Wert",
"bad_format": "Falsches Format",
"too_short": "Zu kurz",
"too_long": "Zu lang",
"control_character": "Steuerzeichen nicht erlaubt",
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
"bad_variable_name": "Ungültiger Variablenname",
"bad_variable_value": "Ungültiger Variablenwert",
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
"duplicate_entry": "Eintrag kommt doppelt vor",
"unknown_host": "Unbekannter Host",
"unknown_group": "Unbekannte Gerätegruppe",
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
}
REASONS_DE = {
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
"prune_referenced": "wird noch verwendet",
}
WARNINGS_DE = {
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
"unknown_alias_reference": "{value}: kein zentraler Alias in OPNexus (auf der Firewall von Hand angelegt? dann in Ordnung)",
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
}
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
def safe(text) -> str:
"""Nothing from a foreign file reaches the terminal as a control sequence."""
return _ANSI.sub("?", str(text))
def _short(value) -> str:
return safe(display(json.dumps(value, ensure_ascii=False), 60))
def format_errors(errors: list[dict]) -> str:
lines = []
for e in errors:
where = f"Zeile {e['line']}: " if e.get("line") else ""
path = f"{safe(e['path'])}: " if e.get("path") else ""
detail = ""
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
if params:
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
return "\n".join(lines)
def format_plan(public: dict) -> str:
lines = []
for item in public["items"]:
action = item["action"]
planned = item["planned_action"]
name = f"{item['kind']} {safe(item['key'])}"
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
if action == "blocked":
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
elif planned in LABEL:
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
else:
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
if change["field"]:
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
else:
lines.append(f"~ {name}{suffix}")
if item.get("implicit"):
lines[-1] += " (Folge der Reihenfolge)"
for warning in public["warnings"]:
template = WARNINGS_DE.get(warning["code"])
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
text = template.format(**params) if template else warning["code"]
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
lines.append(f"? {where}{text}")
s = public["summary"]
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
return "\n".join(lines)
def _read_stdin() -> bytes:
data = sys.stdin.buffer.read(MAX_BYTES + 1)
return data
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
parser.add_argument("--kinds", default=None)
parser.add_argument("--prune", action="store_true")
parser.add_argument("--json", action="store_true")
parser.add_argument("--plan-token", default=None)
try:
args = parser.parse_args(argv)
except SystemExit:
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
if args.command == "schema":
return 0, schema.json_schema_text(), ""
from app.database import SessionLocal
from app.services import audit_service
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
try:
async with SessionLocal() as db:
if args.command == "export":
wanted = None
if args.kinds:
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
if not wanted <= set(schema.KINDS):
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
return 0, await export_yaml(db, wanted), ""
raw = stdin if stdin is not None else _read_stdin()
try:
parsed = parse(raw)
if args.command == "plan":
plan = await planner.build_plan(db, parsed, args.prune)
await db.rollback()
public = plan.public()
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
s = public["summary"]
if not plan.applicable:
return 4, out, ""
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
if not args.plan_token:
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
try:
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
except applier.PlanStale:
await db.rollback()
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
except applier.PlanBlocked as exc:
await db.rollback()
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
except applier.SelfCheckFailed as exc:
await db.rollback()
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
except Exception:
await db.rollback()
raise
await db.commit()
s = plan.summary()
if args.json:
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
except ConfigError as exc:
if args.json:
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
finally:
audit_service.context.reset(context)
def main(argv: list[str]) -> int:
try:
code, out, err = asyncio.run(run(argv))
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
return 1
sys.stdout.write(out)
sys.stderr.write(err)
return code
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+312
View File
@@ -0,0 +1,312 @@
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
Den Commit macht der Aufrufer.
"""
from datetime import datetime, timezone
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.config_code import plan as planner
from app.config_code.export import canonical
from app.config_code.loader import Parsed
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.host_variable import HostVariable
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
from app.services import audit_service
NOTE = "Import (Konfiguration als Code)"
AUDIT_KIND = {
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
}
class PlanStale(Exception):
pass
class PlanBlocked(Exception):
def __init__(self, plan: planner.Plan):
self.plan = plan
super().__init__("plan blocked")
class SelfCheckFailed(Exception):
def __init__(self, differences: list[str]):
self.differences = differences
super().__init__("; ".join(differences[:5]))
def _revision_writer(kind: str):
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
"""
if kind == "rules":
from app.services.policy_service import snapshot
return RuleRevision, "rule_id", snapshot
if kind == "aliases":
from app.services.alias_policy_service import snapshot
return AliasRevision, "alias_id", snapshot
if kind == "port_forwards":
from app.services.port_forward_policy_service import snapshot
return PortForwardRevision, "port_forward_id", snapshot
if kind == "one_to_one":
from app.services.one_to_one_policy_service import snapshot
return OneToOneRevision, "one_to_one_id", snapshot
if kind == "outbound_nat":
from app.services.outbound_nat_policy_service import snapshot
return OutboundNatRevision, "outbound_nat_id", snapshot
from app.services.policy_hierarchy_service import snapshot
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
SCALARS = {
"aliases": ("name", "type", "description", "enabled"),
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
"destination_net", "destination_port", "enabled", "log"),
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
"destination_port", "target", "local_port", "natreflection", "enabled"),
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
"destination_not", "natreflection", "log", "enabled"),
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
}
class _Writer:
def __init__(self, db: AsyncSession, plan: planner.Plan):
self.db = db
self.plan = plan
self.now = datetime.now(timezone.utc)
self.events: list[dict] = []
async def load(self) -> None:
db = self.db
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
self.groups = {g.name: g for g in (await db.execute(
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
self.rows: dict[str, dict] = {}
for kind, model in MODELS.items():
options = [selectinload(model.deployments)]
if kind != "policy_hierarchy_rules":
options += [selectinload(model.targets), selectinload(model.groups)]
rows = (await db.execute(select(model).options(*options))).scalars().all()
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
def items(self, kind: str, *actions: str) -> list[planner.Item]:
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
def event(self, item: planner.Item, entity_id, name: str) -> None:
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
# -- groups ----------------------------------------------------------------------------------------------------------
async def hierarchy_groups(self) -> None:
items = self.items("policy_hierarchy_groups", "create", "update")
for item in items:
if item.action == "create":
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
self.db.add(group)
self.hierarchy[item.key] = group
await self.db.flush()
for item in items:
group = self.hierarchy[item.key]
group.description = item.after["description"]
parent = item.after["parent"]
group.parent_id = self.hierarchy[parent].id if parent else None
self.event(item, group.id, group.name)
await self.db.flush()
async def device_groups(self) -> None:
for item in self.items("device_groups", "create", "update"):
group = self.groups.get(item.key)
if group is None:
group = DeviceGroup(name=item.key)
self.db.add(group)
self.groups[item.key] = group
group.description = item.after["description"]
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
await self.db.flush()
self.event(item, group.id, group.name)
# -- objects ---------------------------------------------------------------------------------------------------------
async def objects(self, kind: str) -> None:
model = MODELS[kind]
revision_model, fk, snapshot = _revision_writer(kind)
for item in self.items(kind, "create", "update"):
after = item.after
row = self.rows[kind].get(item.key)
if row is None:
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
if kind != "aliases":
row.config_key = item.key
self.db.add(row)
self.rows[kind][item.key] = row
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
for name in SCALARS[kind]:
setattr(row, name, after[name])
if kind == "aliases":
row.content = "\n".join(after["entries"])
if kind == "policy_hierarchy_rules":
row.hierarchy_group_id = self.hierarchy[after["group"]].id
else:
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
if "hierarchy_group" in after["targets"]:
hierarchy = after["targets"]["hierarchy_group"]
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
await self.db.flush()
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
self.event(item, row.id, row.name if kind == "aliases" else row.description)
async def hierarchy_positions(self) -> None:
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
if "policy_hierarchy_rules" not in self.plan.touched:
return
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
for key, rank in self.plan.positions.items():
row = self.rows["policy_hierarchy_rules"].get(key)
if row is not None and key not in deleted and row.sequence != rank:
row.sequence = rank
await self.db.flush()
async def host_hierarchy(self) -> None:
for host_name, group_name in self.plan.host_hierarchy.items():
host = self.hosts[host_name]
wanted = self.hierarchy[group_name].id if group_name else None
if host.policy_hierarchy_group_id != wanted:
host.policy_hierarchy_group_id = wanted
await self.db.flush()
async def variables(self) -> None:
if "variables" not in self.plan.touched:
return
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
for item in self.items("variables", "create", "update"):
host = self.hosts[item.after["host"]]
row = existing.get((host.id, item.after["name"]))
if row is None:
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
else:
row.value = item.after["value"]
self.event(item, None, item.key)
for item in self.items("variables", "delete"):
host = self.hosts[item.before["host"]]
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
self.event(item, None, item.key)
await self.db.flush()
# -- deletions -------------------------------------------------------------------------------------------------------
async def deletions(self) -> None:
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
for item in self.items(kind, "delete"):
row = self.rows[kind][item.key]
if row.deployments: # the planner blocks this; never undeploy from here
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
self.event(item, row.id, row.name if kind == "aliases" else row.description)
await self.db.delete(row)
await self.db.flush()
for item in self.items("device_groups", "delete"):
row = self.groups[item.key]
self.event(item, row.id, row.name)
await self.db.delete(row)
await self.db.flush()
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
by_id = {g.id: g for g in self.hierarchy.values()}
def depth(group) -> int:
n, current = 0, group
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
n, current = n + 1, by_id[current.parent_id]
return n
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
self.event(item, row.id, row.name)
await self.db.delete(row)
await self.db.flush()
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
db.expire_all() # compare what is in the database, not what the session believes
actual = await canonical(db, set(plan.touched))
differences: list[str] = []
for kind in plan.touched:
if kind == "variables":
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
for h, values in actual["variables"].items() for n, v in values.items()}
else:
ident = planner.KIND_SPECS[kind].ident
got = {item[ident]: item for item in actual[kind]}
want = plan.expected[kind]
for key in sorted(set(got) | set(want)):
if got.get(key) != want.get(key):
differences.append(f"{kind}:{key}")
if "policy_hierarchy_rules" in plan.touched:
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
for key, rank in ranks.items():
if plan.positions.get(key) != rank:
differences.append(f"policy_hierarchy_rules:{key}:position")
if differences:
raise SelfCheckFailed(differences)
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
ConfigError; on any exception the caller must roll back."""
from app.services.policy_service import policy_lock
await policy_lock(db)
plan = await planner.build_plan(db, parsed, prune)
if plan.token != token:
raise PlanStale()
if not plan.applicable:
raise PlanBlocked(plan)
writer = _Writer(db, plan)
await writer.load()
await writer.hierarchy_groups()
await writer.device_groups()
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
if kind in plan.touched:
await writer.objects(kind)
await writer.hierarchy_positions()
await writer.host_hierarchy()
await writer.variables()
await writer.deletions()
await _self_check(db, plan)
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
"summary": plan.summary()}))
for values in writer.events:
db.add(audit_service.make_event(**values))
await db.flush()
return plan
+154
View File
@@ -0,0 +1,154 @@
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
"""
import yaml
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.config_code import schema
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
from app.version import APP_VERSION
HEADER = (
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
)
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
fields = list(doc.model_fields)
if set(values) != set(fields):
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
return {name: values[name] for name in fields}
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
if "hierarchy_group" in doc.model_fields:
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
return _ordered(doc, values)
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
values = dict(extra)
for name in doc.model_fields:
if name in values:
continue
values[name] = obj.config_key if name == "key" else getattr(obj, name)
return _ordered(doc, values)
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
"""Parents before children, siblings by name; the same rule the loader expects."""
children: dict = {}
for group in groups:
children.setdefault(group.parent_id, []).append(group)
ordered: list[PolicyHierarchyGroup] = []
def walk(parent_id) -> None:
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
ordered.append(group)
walk(group.id)
walk(None)
return ordered
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
"""{kind: content} for the requested kinds (default all), in file order."""
wanted = set(schema.KINDS) if kinds is None else kinds
out: dict[str, list | dict] = {}
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
hierarchy_names = {g.id: g.name for g in all_groups}
if "device_groups" in wanted:
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
out["device_groups"] = [
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
for g in sorted(groups, key=lambda g: g.name)
]
if "policy_hierarchy_groups" in wanted:
out["policy_hierarchy_groups"] = [
_ordered(schema.HierarchyGroupDoc, {
"name": g.name,
"parent": hierarchy_names.get(g.parent_id),
"description": g.description or "",
"hosts": sorted(h.name for h in g.hosts),
})
for g in _hierarchy_order(list(all_groups))
]
if "aliases" in wanted:
aliases = (await db.execute(
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
)).scalars().all()
out["aliases"] = [
_ordered(schema.AliasDoc, {
"name": a.name,
"type": a.type,
"description": a.description or "",
"enabled": a.enabled,
"entries": [line for line in (a.content or "").split("\n") if line],
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
})
for a in sorted(aliases, key=lambda a: a.name)
]
simple = (
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
)
for kind, model, doc, targets_doc in simple:
if kind not in wanted:
continue
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
out[kind] = [
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
for row in sorted(rows, key=lambda r: r.config_key)
]
if "policy_hierarchy_rules" in wanted:
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
out["policy_hierarchy_rules"] = [
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
]
if "variables" in wanted:
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
out["variables"] = {
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
for h in sorted(hosts, key=lambda h: h.name)
if h.variables
}
return out
def dump_yaml(content: dict[str, list | dict]) -> str:
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
return dump_yaml(await canonical(db, kinds))
+307
View File
@@ -0,0 +1,307 @@
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
"""
import hashlib
import re
from typing import NamedTuple
import yaml
from pydantic import ValidationError
from yaml.composer import Composer
from yaml.constructor import SafeConstructor
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
from yaml.parser import Parser
from yaml.reader import Reader
from yaml.resolver import BaseResolver
from yaml.scanner import Scanner
from app.config_code import schema
MAX_BYTES = 1024 * 1024
MAX_DEPTH = 8
MAX_NODES = 100_000
MAX_SCALAR = schema.MAX_SCALAR
MAX_OBJECTS_PER_KIND = 5000
MAX_VARIABLE_HOSTS = 500
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
MAX_ALIAS_ENTRIES = 2000
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
_STR = "tag:yaml.org,2002:str"
class ConfigError(Exception):
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
def __init__(self, code: str, errors: list[dict]):
self.code = code
self.errors = errors
super().__init__(f"{code}: {errors[:3]}")
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
return {"code": code, "path": path, "line": line, "params": params}
def display(value, limit: int = 80) -> str:
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
text = value if isinstance(value, str) else repr(value)
text = text.encode("unicode_escape").decode("ascii")
return text if len(text) <= limit else text[: limit - 1] + "…"
def _path(parts) -> str:
out = ""
for part in parts:
if isinstance(part, int):
out += f"[{part}]"
else:
out += ("." if out else "") + str(part)
return out
class _Rejected(yaml.YAMLError):
def __init__(self, code: str, mark=None, path=(), **params):
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
super().__init__(code)
class StrictResolver(BaseResolver):
yaml_implicit_resolvers: dict = {}
StrictResolver.add_implicit_resolver(
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
StrictResolver.add_implicit_resolver(
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
StrictResolver.add_implicit_resolver(
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
class StrictConstructor(SafeConstructor):
yaml_constructors: dict = {}
for _tag, _fn in (
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
(_STR, SafeConstructor.construct_yaml_str),
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
):
StrictConstructor.add_constructor(_tag, _fn)
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
def __init__(self, stream: str):
Reader.__init__(self, stream)
Scanner.__init__(self)
Parser.__init__(self)
Composer.__init__(self)
StrictConstructor.__init__(self)
StrictResolver.__init__(self)
self._depth = 0
self._nodes = 0
self._path: list = []
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
# -- document level ---------------------------------------------------------------------------------------------
def get_single_node(self):
self.get_event() # StreamStart
if self.check_event(StreamEndEvent):
raise _Rejected("yaml_empty")
start = self.peek_event()
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
raise _Rejected("yaml_directive_forbidden", start.start_mark)
node = self.compose_document()
if not self.check_event(StreamEndEvent):
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
self.get_event()
return node
# -- node level ---------------------------------------------------------------------------------------------------
def compose_node(self, parent, index):
event = self.peek_event()
if isinstance(event, AliasEvent):
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
if getattr(event, "anchor", None) is not None:
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
if getattr(event, "tag", None) is not None:
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
self._nodes += 1
if self._nodes > MAX_NODES:
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
if isinstance(index, ScalarNode):
segment = index.value
elif isinstance(index, int) and not isinstance(index, bool):
segment = index
else:
segment = None # a mapping key
if segment is not None:
self._path.append(segment)
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
self._depth += 1
try:
if self._depth > MAX_DEPTH:
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
return super().compose_node(parent, index)
finally:
self._depth -= 1
if segment is not None:
self._path.pop()
def compose_scalar_node(self, anchor):
event = self.peek_event()
if len(event.value) > MAX_SCALAR:
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
return super().compose_scalar_node(anchor)
def compose_mapping_node(self, anchor):
node = super().compose_mapping_node(anchor)
seen: set[str] = set()
for key, _value in node.value:
if not isinstance(key, ScalarNode) or key.tag != _STR:
raise _Rejected("non_string_key", key.start_mark, self._path)
if key.value == "<<":
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
if key.value in seen:
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
seen.add(key.value)
return node
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
if isinstance(raw, str):
raw = raw.encode("utf-8")
if len(raw) > MAX_BYTES:
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
try:
text = raw.decode("utf-8", errors="strict")
except UnicodeDecodeError as exc:
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
if text.startswith(""):
text = text[1:]
loader = None
try:
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
node = loader.get_single_node()
data = loader.construct_document(node)
except _Rejected as exc:
line = exc.mark.line + 1 if exc.mark is not None else None
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
except yaml.MarkedYAMLError as exc:
mark = exc.problem_mark or exc.context_mark
line = mark.line + 1 if mark is not None else None
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
finally:
if loader is not None:
loader.dispose()
return data, loader.lines
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
for end in range(len(loc), 0, -1):
if loc[:end] in lines:
return lines[loc[:end]]
return None
_PYDANTIC_CODES = {
"extra_forbidden": "unknown_field",
"missing": "missing_field",
"literal_error": "bad_value",
"string_too_short": "too_short",
"string_too_long": "too_long",
"string_pattern_mismatch": "bad_format",
}
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
out = []
for error in exc.errors(include_url=False):
loc = tuple(error["loc"])
code = _PYDANTIC_CODES.get(error["type"])
if code is None and error["type"] == "value_error":
message = str(error.get("ctx", {}).get("error", ""))
code = message if message in _OWN_CODES else "bad_value"
if code is None:
code = "bad_type"
params = {}
if error["type"] == "extra_forbidden":
params["field"] = display(loc[-1]) if loc else ""
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
params["value"] = display(error["input"])
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
return out
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
errors = []
for kind in schema.KINDS:
section = data.get(kind)
if kind == "variables":
if isinstance(section, dict):
if len(section) > MAX_VARIABLE_HOSTS:
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
for host, values in section.items():
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
max=MAX_VARIABLES_PER_HOST))
continue
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
aliases = data.get("aliases")
for i, alias in enumerate(aliases if isinstance(aliases, list) else []): # any other shape is for the schema check
entries = alias.get("entries") if isinstance(alias, dict) else None
if isinstance(entries, list):
loc = ("aliases", i, "entries")
if len(entries) > MAX_ALIAS_ENTRIES:
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
return errors
class Parsed(NamedTuple):
doc: schema.ConfigDoc
sha256: str # of the raw bytes, shown in the plan and stored in the audit
lines: dict[tuple, int] # path -> line, for messages of the planner
def parse(raw: bytes | str) -> Parsed:
"""Load, limit and validate a file. Raises ConfigError."""
if isinstance(raw, str):
raw = raw.encode("utf-8")
sha256 = hashlib.sha256(raw).hexdigest()
data, lines = load_yaml(raw)
if not isinstance(data, dict):
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
version = data.get("opnexus_config")
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
"opnexus_config", _line_for(lines, ("opnexus_config",)),
**({"value": display(version)} if version is not None else {}))])
errors = _check_counts(data, lines)
if errors:
raise ConfigError("CONFIG_INVALID", errors)
try:
doc = schema.ConfigDoc.model_validate(data)
except ValidationError as exc:
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
return Parsed(doc, sha256, lines)
@@ -0,0 +1,933 @@
{
"$defs": {
"AliasDoc": {
"additionalProperties": false,
"properties": {
"name": {
"maxLength": 64,
"minLength": 1,
"pattern": "^[A-Za-z0-9_]+$",
"title": "Name",
"type": "string"
},
"type": {
"default": "host",
"enum": [
"host",
"network",
"port",
"networkgroup",
"mac",
"asn"
],
"title": "Type",
"type": "string"
},
"description": {
"default": "",
"maxLength": 255,
"title": "Description",
"type": "string"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"entries": {
"items": {
"maxLength": 8000,
"minLength": 1,
"type": "string"
},
"title": "Entries",
"type": "array"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"name"
],
"title": "AliasDoc",
"type": "object"
},
"DeviceGroupDoc": {
"additionalProperties": false,
"properties": {
"name": {
"maxLength": 120,
"minLength": 1,
"title": "Name",
"type": "string"
},
"description": {
"default": "",
"maxLength": 2000,
"title": "Description",
"type": "string"
},
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
}
},
"required": [
"name"
],
"title": "DeviceGroupDoc",
"type": "object"
},
"HierarchyGroupDoc": {
"additionalProperties": false,
"properties": {
"name": {
"maxLength": 120,
"minLength": 1,
"title": "Name",
"type": "string"
},
"parent": {
"anyOf": [
{
"maxLength": 120,
"minLength": 1,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Parent"
},
"description": {
"default": "",
"maxLength": 2000,
"title": "Description",
"type": "string"
},
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
}
},
"required": [
"name"
],
"title": "HierarchyGroupDoc",
"type": "object"
},
"HierarchyRuleDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"group": {
"maxLength": 120,
"minLength": 1,
"title": "Group",
"type": "string"
},
"rulebase": {
"default": "pre",
"enum": [
"pre",
"post"
],
"title": "Rulebase",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"direction": {
"default": "in",
"enum": [
"in",
"out"
],
"title": "Direction",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "any",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"action": {
"default": "pass",
"enum": [
"pass",
"block",
"reject"
],
"title": "Action",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
}
},
"required": [
"key",
"group",
"description",
"interface"
],
"title": "HierarchyRuleDoc",
"type": "object"
},
"OneToOneDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"type": {
"default": "binat",
"enum": [
"binat",
"nat"
],
"title": "Type",
"type": "string"
},
"external": {
"maxLength": 128,
"minLength": 1,
"title": "External",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"source_not": {
"default": false,
"title": "Source Not",
"type": "boolean"
},
"destination_not": {
"default": false,
"title": "Destination Not",
"type": "boolean"
},
"natreflection": {
"default": "",
"enum": [
"",
"enable",
"disable"
],
"title": "Natreflection",
"type": "string"
},
"log": {
"default": false,
"title": "Log",
"type": "boolean"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"key",
"description",
"interface",
"external"
],
"title": "OneToOneDoc",
"type": "object"
},
"OutboundNatDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "any",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"source_not": {
"default": false,
"title": "Source Not",
"type": "boolean"
},
"destination_not": {
"default": false,
"title": "Destination Not",
"type": "boolean"
},
"target": {
"default": "",
"maxLength": 128,
"title": "Target",
"type": "string"
},
"target_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Target Port"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"key",
"description",
"interface"
],
"title": "OutboundNatDoc",
"type": "object"
},
"PortForwardDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "tcp",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"target": {
"maxLength": 128,
"minLength": 1,
"title": "Target",
"type": "string"
},
"local_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Local Port"
},
"natreflection": {
"default": "",
"enum": [
"",
"purenat",
"disable"
],
"title": "Natreflection",
"type": "string"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/TargetsWithHierarchy"
}
},
"required": [
"key",
"description",
"interface",
"target"
],
"title": "PortForwardDoc",
"type": "object"
},
"RuleDoc": {
"additionalProperties": false,
"properties": {
"key": {
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
"title": "Key",
"type": "string"
},
"description": {
"maxLength": 255,
"minLength": 1,
"title": "Description",
"type": "string"
},
"interface": {
"maxLength": 64,
"minLength": 1,
"title": "Interface",
"type": "string"
},
"direction": {
"default": "in",
"enum": [
"in",
"out"
],
"title": "Direction",
"type": "string"
},
"ipprotocol": {
"default": "inet",
"enum": [
"inet",
"inet6"
],
"title": "Ipprotocol",
"type": "string"
},
"protocol": {
"default": "any",
"maxLength": 16,
"minLength": 1,
"title": "Protocol",
"type": "string"
},
"action": {
"default": "pass",
"enum": [
"pass",
"block",
"reject"
],
"title": "Action",
"type": "string"
},
"source_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Source Net",
"type": "string"
},
"source_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Source Port"
},
"destination_net": {
"default": "any",
"maxLength": 128,
"minLength": 1,
"title": "Destination Net",
"type": "string"
},
"destination_port": {
"anyOf": [
{
"maxLength": 32,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Destination Port"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"log": {
"default": false,
"title": "Log",
"type": "boolean"
},
"targets": {
"$ref": "#/$defs/Targets"
}
},
"required": [
"key",
"description",
"interface"
],
"title": "RuleDoc",
"type": "object"
},
"Targets": {
"additionalProperties": false,
"properties": {
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
},
"groups": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Groups",
"type": "array"
}
},
"title": "Targets",
"type": "object"
},
"TargetsWithHierarchy": {
"additionalProperties": false,
"properties": {
"hosts": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Hosts",
"type": "array"
},
"groups": {
"items": {
"maxLength": 120,
"minLength": 1,
"type": "string"
},
"title": "Groups",
"type": "array"
},
"hierarchy_group": {
"anyOf": [
{
"maxLength": 120,
"minLength": 1,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Hierarchy Group"
}
},
"title": "TargetsWithHierarchy",
"type": "object"
}
},
"additionalProperties": false,
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
"properties": {
"opnexus_config": {
"const": 1,
"title": "Opnexus Config",
"type": "integer"
},
"device_groups": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/DeviceGroupDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Device Groups"
},
"policy_hierarchy_groups": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/HierarchyGroupDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Policy Hierarchy Groups"
},
"aliases": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/AliasDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Aliases"
},
"rules": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/RuleDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Rules"
},
"port_forwards": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/PortForwardDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Port Forwards"
},
"one_to_one": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/OneToOneDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "One To One"
},
"outbound_nat": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/OutboundNatDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Outbound Nat"
},
"policy_hierarchy_rules": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/HierarchyRuleDoc"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Policy Hierarchy Rules"
},
"variables": {
"anyOf": [
{
"additionalProperties": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"propertyNames": {
"maxLength": 120,
"minLength": 1
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Variables"
}
},
"required": [
"opnexus_config"
],
"title": "OPNexus-Konfiguration (Schema v1)",
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
+570
View File
@@ -0,0 +1,570 @@
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
`prune` gleicht die Art ab.
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
"""
import hashlib
import json
import re
import uuid
from dataclasses import dataclass, field
from pydantic import ValidationError
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.config_code import schema
from app.config_code.export import canonical
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
from app.database import Base
from app.services.deploy_flow import is_unsettled
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
from app.schemas.alias import AliasCreate
from app.schemas.one_to_one import OneToOneCreate
from app.schemas.outbound_nat import OutboundNatCreate
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
from app.schemas.port_forward import PortForwardCreate
from app.schemas.rule import RuleCreate
from app.services.variables_service import names_in
DEPLOYED = ("success", "pending", "revert_failed")
@dataclass(frozen=True)
class Kind:
section: str
model: type | None
ident: str # identity field in the file ("key"/"name")
api: type | None = None # API create schema, run on every file object (same validators as the UI)
deployable: bool = False
KIND_SPECS = {
"device_groups": Kind("device_groups", DeviceGroup, "name"),
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
"variables": Kind("variables", None, "key"),
}
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
@dataclass
class Item:
kind: str
key: str
action: str # create | update | delete | unchanged
before: dict | None = None
after: dict | None = None
changes: list = field(default_factory=list)
reasons: list = field(default_factory=list)
deployed_hosts: int = 0
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
db_id: uuid.UUID | None = None
def public(self) -> dict:
return {
"kind": self.kind,
"key": self.key,
"action": "blocked" if self.reasons else self.action,
"planned_action": self.action,
"changes": self.changes,
"deployed_hosts": self.deployed_hosts,
"reasons": self.reasons,
"implicit": self.implicit,
}
@dataclass
class Plan:
sha256: str
prune: bool
touched: list[str]
items: list[Item]
warnings: list[dict]
token: str = ""
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
expected: dict = field(default_factory=dict)
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
@property
def applicable(self) -> bool:
return not any(i.reasons for i in self.items)
def summary(self) -> dict:
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
for item in self.items:
out["blocked" if item.reasons else item.action] += 1
return out
def public(self) -> dict:
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
return {
"token": self.token,
"sha256": self.sha256,
"prune": self.prune,
"kinds": self.touched,
"summary": self.summary(),
"items": [i.public() for i in changed],
"warnings": self.warnings,
"applicable": self.applicable,
}
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
section = getattr(doc, kind)
if kind == "variables":
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
for host, values in section.items() for name, value in values.items()]
return [item.model_dump(mode="python") for item in section]
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
def sorted_unique(values: list, where: tuple) -> list:
seen = set()
for i, value in enumerate(values):
if value in seen:
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
seen.add(value)
return sorted(seen)
item = dict(item)
if "targets" in item:
targets = dict(item["targets"])
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
item["targets"] = targets
if kind in ("device_groups", "policy_hierarchy_groups"):
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
return item
def _changes(before: dict, after: dict) -> list[dict]:
out = []
for name in after:
if name == "targets":
for sub in after["targets"]:
if before["targets"].get(sub) != after["targets"][sub]:
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
elif before.get(name) != after[name]:
out.append({"field": name, "before": before.get(name), "after": after[name]})
return out
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
buckets: dict = {}
for row in rows:
buckets.setdefault(bucket_of(row), []).append(row)
out = {}
for members in buckets.values():
for rank, row in enumerate(sorted(members, key=order_of)):
out[key_of(row)] = rank
return out
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
found: dict = {}
if not ids:
return found
for table in Base.metadata.sorted_tables:
if table.name in skip:
continue
for fk in table.foreign_keys:
if fk.column.table.name != target_table:
continue
column = fk.parent
rows = (await db.execute(
select(column, func.count()).where(column.in_(ids)).group_by(column)
)).all()
for ref_id, _count in rows:
found.setdefault(ref_id, []).append(table.name)
return found
async def _load(db: AsyncSession) -> dict:
loaded: dict = {}
for kind in TARGET_KINDS:
model = KIND_SPECS[kind].model
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
return loaded
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
doc, lines = parsed.doc, parsed.lines
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
errors: list[dict] = []
warnings: list[dict] = []
db_canon = await canonical(db)
loaded = await _load(db)
hosts = loaded["hosts"]
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
file_objects: dict[str, dict[str, dict]] = {}
file_locs: dict[tuple, tuple] = {}
for kind in touched:
spec = KIND_SPECS[kind]
objects: dict[str, dict] = {}
for index, raw in enumerate(_file_items(doc, kind)):
if kind == "variables":
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
else:
ident, loc = raw[spec.ident], (kind, index)
if ident in objects:
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
continue
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
file_locs[(kind, ident)] = loc
file_objects[kind] = objects
def db_items(kind: str) -> dict[str, dict]:
if kind == "variables":
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
for h, values in db_canon["variables"].items() for n, v in values.items()}
ident = KIND_SPECS[kind].ident
return {item[ident]: item for item in db_canon[kind]}
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
post: dict[str, dict[str, dict]] = {}
for kind in schema.KINDS:
if kind in file_objects:
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
if prune:
deleted[kind] = set(remaining)
remaining = {}
post[kind] = {**remaining, **file_objects[kind]}
else:
post[kind] = dict(db_objects[kind])
host_names = set(hosts)
group_names = set(post["device_groups"])
hierarchy_names = set(post["policy_hierarchy_groups"])
# -- references in the file ------------------------------------------------------------------------------------------
for kind in touched:
for ident, item in file_objects[kind].items():
loc = file_locs[(kind, ident)]
if kind == "variables":
if item["host"] not in host_names:
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
continue
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
for name in member_hosts:
if name not in host_names:
_err(errors, "unknown_host", loc, lines, host=display(name))
for name in item.get("targets", {}).get("groups", []):
if name not in group_names:
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
if ref is not None and ref not in hierarchy_names:
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
if kind == "policy_hierarchy_rules":
for name, value in item.items():
if names_in(value):
_err(errors, "placeholder_not_supported", loc + (name,), lines)
spec = KIND_SPECS[kind]
if spec.api is not None:
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
for name in member_hosts:
host = hosts.get(name)
if host is not None and not host.enabled:
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
if "policy_hierarchy_groups" in file_objects:
file_groups = file_objects["policy_hierarchy_groups"]
for host_name, group in list(host_hierarchy.items()):
if prune or group in file_groups:
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
host_hierarchy[host_name] = None
listed: dict[str, str] = {}
for name, group in file_groups.items():
for host_name in group["hosts"]:
if host_name in listed:
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
lines, host=display(host_name), first=display(listed[host_name]))
listed[host_name] = name
if host_name in host_hierarchy:
host_hierarchy[host_name] = name
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
for start in parents:
seen, current = set(), start
while current is not None and current not in seen:
seen.add(current)
current = parents.get(current)
if current is not None and current == start:
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
lines, group=display(start))
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
per_host: dict[str, int] = {}
for item in post["variables"].values():
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
for host_name, count in sorted(per_host.items()):
if count > MAX_VARIABLES_PER_HOST:
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
max=MAX_VARIABLES_PER_HOST, count=count)
if errors:
raise ConfigError("CONFIG_INVALID", errors[:100])
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
db_rule_rows = loaded["policy_hierarchy_rules"]
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
positions = dict(db_positions)
if "policy_hierarchy_rules" in file_objects:
buckets: dict[tuple, list[str]] = {}
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
for key, item in post["policy_hierarchy_rules"].items():
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
buckets.setdefault((item["group"], item["rulebase"]), [])
positions = {}
extras: dict[tuple, list[str]] = {}
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
key=lambda k: db_positions.get(k, 0)):
item = post["policy_hierarchy_rules"][key]
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
for bucket in set(buckets) | set(extras):
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
positions[key] = rank
# -- items -----------------------------------------------------------------------------------------------------------
items: list[Item] = []
for kind in touched:
rows = loaded.get(kind, {})
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
row = rows.get(ident) if kind != "variables" else None
before = db_objects[kind].get(ident)
after = post[kind].get(ident) if ident not in deleted[kind] else None
if before is None:
item = Item(kind, ident, "create", None, after)
elif after is None:
item = Item(kind, ident, "delete", before, None)
else:
changes = _changes(before, after)
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
if row is not None:
item.db_id = row.id
deployments = getattr(row, "deployments", None) or []
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
if item.action in ("update", "delete") and any(is_unsettled(d) for d in deployments):
item.reasons.append("busy_deployment")
if item.action == "delete" and deployments:
item.reasons.append("prune_deployed")
items.append(item)
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
if redeploy:
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
warnings += _alias_reference_warnings(post, file_objects, file_locs)
warnings += _variable_warnings(post, deleted)
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
file_objects=file_objects)
plan.expected = {kind: post[kind] for kind in touched}
plan.token = _token(plan, db_canon, db_positions, loaded)
return plan
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
out = set()
if kind == "policy_hierarchy_rules":
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
if kind == "policy_hierarchy_groups":
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
return out
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
if kind == "aliases":
values["content"] = "\n".join(item["entries"])
if kind == "policy_hierarchy_rules":
values["hierarchy_group_id"] = uuid.UUID(int=0)
try:
cleaned = KIND_SPECS[kind].api.model_validate(values)
except ValidationError as exc:
for error in exc.errors(include_url=False):
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
return item
out = dict(item)
for name in item:
if name in values and name != "content" and hasattr(cleaned, name):
out[name] = getattr(cleaned, name)
if kind == "aliases":
out["entries"] = [e for e in cleaned.content.split("\n") if e]
return out
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
by_key = {(i.kind, i.key): i for i in items}
# device groups: post-state objects of the import, plus every other table that points at the group
if deleted["device_groups"]:
users: dict[str, list[str]] = {}
for kind in TARGET_KINDS:
for ident, item in post[kind].items():
for group in item["targets"]["groups"]:
users.setdefault(group, []).append(f"{kind}:{ident}")
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
users.setdefault(ids[ref_id], []).extend(tables)
for name in deleted["device_groups"]:
if users.get(name):
by_key[("device_groups", name)].reasons.append("prune_referenced")
if deleted["policy_hierarchy_groups"]:
users = {}
for kind in HIERARCHY_TARGET_KINDS:
for ident, item in post[kind].items():
if item["targets"]["hierarchy_group"]:
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
for ident, item in post["policy_hierarchy_rules"].items():
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
for name, item in post["policy_hierarchy_groups"].items():
if item["parent"]:
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
for host_name, group in host_hierarchy.items():
if group:
users.setdefault(group, []).append(f"host:{host_name}")
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
"policy_hierarchy_rules", "opnsense_hosts"}
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
users.setdefault(ids[ref_id], []).extend(tables)
for name in deleted["policy_hierarchy_groups"]:
if users.get(name):
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
if deleted["aliases"]:
used: set[str] = set()
for kind in RULE_KINDS:
for item in post[kind].values():
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
for item in post["aliases"].values():
used.update(item["entries"])
for name in deleted["aliases"]:
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
by_key[("aliases", name)].reasons.append("prune_referenced")
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
aliases = set(post["aliases"])
out = []
for kind in RULE_KINDS:
for ident, item in file_objects.get(kind, {}).items():
for name in ALIAS_REF_FIELDS:
value = item.get(name)
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
"params": {"value": display(value)}})
return out
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
if not deleted["variables"]:
return []
used: set[str] = set()
for kind in RULE_KINDS + ("aliases",):
for item in post[kind].values():
for value in item.values():
if isinstance(value, str):
used |= names_in(value)
elif isinstance(value, list):
for entry in value:
used |= names_in(entry)
out = []
for key in sorted(deleted["variables"]):
host, name = key.split("/", 1)
if name in used:
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
return out
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
deployments = {
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
for ident, row in loaded[kind].items()}
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
}
material = {
"file": plan.file_objects,
"prune": plan.prune,
"db": db_canon,
"positions": db_positions,
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
"deployments": deployments,
"items": [i.public() for i in plan.items],
"warnings": plan.warnings,
}
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
+242
View File
@@ -0,0 +1,242 @@
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
"""
import re
from typing import Annotated, Literal
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
from app.models.config_key import CONFIG_KEY_RE
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
SCHEMA_VERSION = 1
MAX_SCALAR = 8000
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
def _clean(value: str) -> str:
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
if _CONTROL.search(value):
raise ValueError("control_character")
if value != value.strip():
raise ValueError("surrounding_whitespace")
return value
def _clean_value(value: str) -> str:
if _CONTROL.search(value):
raise ValueError("control_character")
return value
def _clean_text(value: str) -> str:
if _CONTROL_TEXT.search(value):
raise ValueError("control_character")
return value
def _port(value):
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
Text = Annotated[str, AfterValidator(_clean)]
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
Port = Annotated[PortText | None, BeforeValidator(_port)]
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
IpProtocol = Literal["inet", "inet6"]
class _Doc(BaseModel):
model_config = ConfigDict(extra="forbid", strict=True)
class Targets(_Doc):
hosts: list[HostName] = Field(default_factory=list)
groups: list[GroupName] = Field(default_factory=list)
class TargetsWithHierarchy(Targets):
hierarchy_group: GroupName | None = None
class DeviceGroupDoc(_Doc):
name: GroupName
description: GroupDescription = ""
hosts: list[HostName] = Field(default_factory=list)
class HierarchyGroupDoc(_Doc):
name: GroupName
parent: GroupName | None = None
description: GroupDescription = ""
hosts: list[HostName] = Field(default_factory=list)
class AliasDoc(_Doc):
name: AliasName
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
description: Description = ""
enabled: bool = True
entries: list[AliasEntry] = Field(default_factory=list)
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class RuleDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
direction: Literal["in", "out"] = "in"
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "any"
action: Literal["pass", "block", "reject"] = "pass"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
enabled: bool = True
log: bool = False
targets: Targets = Field(default_factory=Targets)
class PortForwardDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "tcp"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
target: Net
local_port: Port = None
natreflection: Literal["", "purenat", "disable"] = ""
enabled: bool = True
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class OneToOneDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
type: Literal["binat", "nat"] = "binat"
external: Net
source_net: Net = "any"
destination_net: Net = "any"
source_not: bool = False
destination_not: bool = False
natreflection: Literal["", "enable", "disable"] = ""
log: bool = False
enabled: bool = True
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class OutboundNatDoc(_Doc):
key: Key
description: RuleDescription
interface: Interface
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "any"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
source_not: bool = False
destination_not: bool = False
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
target_port: Port = None
enabled: bool = True
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
class HierarchyRuleDoc(_Doc):
key: Key
group: GroupName
rulebase: Literal["pre", "post"] = "pre"
description: RuleDescription
interface: Interface
direction: Literal["in", "out"] = "in"
ipprotocol: IpProtocol = "inet"
protocol: Protocol = "any"
action: Literal["pass", "block", "reject"] = "pass"
source_net: Net = "any"
source_port: Port = None
destination_net: Net = "any"
destination_port: Port = None
enabled: bool = True
def _variable_name(value: str) -> str:
if not VARIABLE_NAME_RE.match(value):
raise ValueError("bad_variable_name")
return value
def _variable_value(value: str) -> str:
if not VARIABLE_VALUE_RE.match(value):
raise ValueError("bad_variable_value")
return value
VariableName = Annotated[str, AfterValidator(_variable_name)]
VariableValue = Annotated[str, AfterValidator(_variable_value)]
class ConfigDoc(_Doc):
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
opnexus_config: Literal[1]
device_groups: list[DeviceGroupDoc] | None = None
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
aliases: list[AliasDoc] | None = None
rules: list[RuleDoc] | None = None
port_forwards: list[PortForwardDoc] | None = None
one_to_one: list[OneToOneDoc] | None = None
outbound_nat: list[OutboundNatDoc] | None = None
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
KINDS = (
"device_groups",
"policy_hierarchy_groups",
"aliases",
"rules",
"port_forwards",
"one_to_one",
"outbound_nat",
"policy_hierarchy_rules",
"variables",
)
SCHEMA_FILE = "opnexus-config-v1.schema.json"
def json_schema_text() -> str:
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
import json
schema = ConfigDoc.model_json_schema()
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
+13
View File
@@ -62,6 +62,7 @@ class ErrorCode(str, Enum):
GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET"
GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET"
GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET"
GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET"
HOSTNAMEN_EXAKT_EINGEBEN = "HOSTNAMEN_EXAKT_EINGEBEN"
HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN = "HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN"
HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0 = "HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0"
@@ -72,6 +73,8 @@ class ErrorCode(str, Enum):
HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN = "HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN"
HOST_IST_DEAKTIVIERT = "HOST_IST_DEAKTIVIERT"
HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS = "HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS"
HOST_URL_EXISTS = "HOST_URL_EXISTS"
HOST_CONNECTION_FAILED = "HOST_CONNECTION_FAILED"
HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B = "HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B"
HOST_NICHT_GEFUNDEN = "HOST_NICHT_GEFUNDEN"
HOST_REQUEST_FAILED = "HOST_REQUEST_FAILED"
@@ -247,6 +250,16 @@ class ErrorCode(str, Enum):
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
CONFIG_INVALID = "CONFIG_INVALID"
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
CONFIG_BLOCKED = "CONFIG_BLOCKED"
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
CONFIG_BUSY = "CONFIG_BUSY"
ZIEL_HOST_NACH_AUSROLLEN_FEST = "ZIEL_HOST_NACH_AUSROLLEN_FEST"
ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET = "ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET"
VPN_DEPLOY_LAEUFT_BEREITS = "VPN_DEPLOY_LAEUFT_BEREITS"
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
+2
View File
@@ -54,6 +54,7 @@ from app.routers import maintenance_windows as maintenance_windows_router
from app.routers import rollouts as rollouts_router
from app.routers import rule_analysis as rule_analysis_router
from app.routers import rule_cleanup as rule_cleanup_router
from app.routers import config_code as config_code_router
from app.routers import gui_access as gui_access_router
from app.services import scheduled_maintenance_service
from app.services import rollout_engine
@@ -571,5 +572,6 @@ app.include_router(maintenance_windows_router.router, prefix="/api")
app.include_router(rollouts_router.router, prefix="/api")
app.include_router(rule_analysis_router.router, prefix="/api")
app.include_router(rule_cleanup_router.router, prefix="/api")
app.include_router(config_code_router.router, prefix="/api")
app.include_router(gui_access_router.router, prefix="/api")
app.include_router(gui_access_router.internal_router)
+5
View File
@@ -108,3 +108,8 @@ __all__ += ["LiveRuleState", "RuleChangeEvent"]
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
from app.models.host_variable import HostVariable
__all__ += ["HostVariable"]
+34
View File
@@ -0,0 +1,34 @@
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
"""
import re
import uuid
from sqlalchemy import String, event
from sqlalchemy.orm import Mapped, mapped_column
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
def config_key_column() -> Mapped[str]:
return mapped_column(String(64), unique=True)
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
return f"{prefix}-{object_id.hex[:12]}"
def register_config_key(model: type, prefix: str) -> None:
"""Fills an empty `config_key` just before INSERT (and the id, which is
otherwise only assigned while the INSERT is built)."""
@event.listens_for(model, "before_insert")
def _fill(_mapper, _connection, target) -> None:
if target.id is None:
target.id = uuid.uuid4()
if not target.config_key:
target.config_key = default_config_key(prefix, target.id)
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.one_to_one_target import one_to_one_targets
from app.models.policy import one_to_one_groups
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
__tablename__ = "one_to_one_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
source_net: Mapped[str] = mapped_column(String(128))
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
)
register_config_key(OneToOneRule, "nat1")
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.outbound_nat_target import outbound_nat_targets
from app.models.policy import outbound_nat_groups
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
__tablename__ = "outbound_nat_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
source_net: Mapped[str] = mapped_column(String(128))
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
)
register_config_key(OutboundNatRule, "out")
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
class PolicyHierarchyRule(Base):
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
__tablename__ = "policy_hierarchy_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
)
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
)
register_config_key(PolicyHierarchyRule, "ph")
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.port_forward_target import port_forward_targets
from app.models.policy import port_forward_groups
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
__tablename__ = "port_forward_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
)
register_config_key(PortForwardRule, "pf")
+2
View File
@@ -42,6 +42,8 @@ class RolloutPlan(Base):
# is activated AND unchanged since (activated_fingerprint).
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
+5
View File
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
from app.models.config_key import config_key_column, register_config_key
from app.models.rule_target import rule_targets
from app.models.policy import rule_groups
@@ -18,6 +19,7 @@ class FirewallRule(Base):
__tablename__ = "firewall_rules"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
config_key: Mapped[str] = config_key_column()
description: Mapped[str] = mapped_column(String(255))
interface: Mapped[str] = mapped_column(String(64))
direction: Mapped[str] = mapped_column(String(8))
@@ -44,3 +46,6 @@ class FirewallRule(Base):
deployments: Mapped[list["RuleDeployment"]] = relationship(
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
)
register_config_key(FirewallRule, "r")
+6 -2
View File
@@ -139,7 +139,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
verified_hash = user.password_hash
if user.totp_enabled:
second_factor_checked = bool(user.totp_enabled) # the state this login is decided on; create_session re-checks it
if second_factor_checked:
# Only revealed after the password was correct, so it cannot be used
# to find out which usernames exist or have 2FA.
if not payload.totp_code:
@@ -153,7 +154,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
auth_service.throttle.reset(ip, payload.username)
failed_since_last_login = auth_service.throttle.pop_failures_since_success(payload.username)
token, expires_at = await auth_service.create_session(
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash,
verified_totp=second_factor_checked,
)
_set_session_cookie(response, token)
logger.info("Successful login for '%s' from %s", user.username, ip)
@@ -252,6 +254,8 @@ async def totp_enable(
_account_failure(user, "2fa-enable")
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.TOTP_INVALID)
_account_success(user, "2fa-enable")
# Same row lock as create_session: a sign-in that decided "no second factor needed" is ordered before or after this.
await db.execute(select(AdminUser.id).where(AdminUser.id == user.id).with_for_update())
plain, hashes = totp_service.new_recovery_codes()
totp_service.promote_pending(user.id)
user.totp_enabled = True
+14
View File
@@ -37,6 +37,13 @@ acme_accounts_router = APIRouter(prefix="/acme/accounts", tags=["acme"])
acme_certificates_router = APIRouter(prefix="/acme/certificates", tags=["acme"])
def _ensure_host_unchanged_once_deployed(obj, new_host_id: uuid.UUID) -> None:
"""The firewall-local UUID, the stored payload and the success status all belong to the host the object was pushed
to. Re-pointing it would report success for a host that never received it and orphan the object on the old one."""
if new_host_id != obj.host_id and obj.opnsense_uuid:
raise api_error(409, ErrorCode.ZIEL_HOST_NACH_AUSROLLEN_FEST)
async def _get_host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
host = await db.get(OpnsenseHost, host_id)
if not host:
@@ -82,6 +89,7 @@ async def edit_managed_certificate(cert_id: uuid.UUID, payload: ManagedCertifica
if cert.updated_at != payload.expected_updated_at:
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
await _get_host(db, payload.host_id)
_ensure_host_unchanged_once_deployed(cert, payload.host_id)
for field, value in payload.model_dump(exclude={"expected_updated_at", "private_key"}).items():
setattr(cert, field, value)
if payload.private_key:
@@ -146,6 +154,11 @@ async def edit_acme_account(account_id: uuid.UUID, payload: AcmeAccountEdit, db:
if account.updated_at != payload.expected_updated_at:
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
await _get_host(db, payload.host_id)
_ensure_host_unchanged_once_deployed(account, payload.host_id)
if payload.host_id != account.host_id and await db.scalar(
select(AcmeCertificateRequest.id).where(AcmeCertificateRequest.account_id == account.id).limit(1)
):
raise api_error(409, ErrorCode.ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET)
for field, value in payload.model_dump(exclude={"expected_updated_at", "eab_hmac"}).items():
setattr(account, field, value)
if payload.eab_hmac:
@@ -236,6 +249,7 @@ async def edit_acme_certificate_request(request_id: uuid.UUID, payload: AcmeCert
if req.updated_at != payload.expected_updated_at:
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
await _get_host(db, payload.host_id)
_ensure_host_unchanged_once_deployed(req, payload.host_id)
await _get_account(db, payload.account_id, payload.host_id)
for field, value in payload.model_dump(exclude={"expected_updated_at"}).items():
setattr(req, field, value)
+154
View File
@@ -0,0 +1,154 @@
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
"""
import asyncio
import json
from fastapi import APIRouter, Depends, Query, Request
from fastapi.concurrency import run_in_threadpool
from fastapi.responses import Response
from pydantic import BaseModel, ConfigDict, Field, ValidationError
from sqlalchemy.ext.asyncio import AsyncSession
from app.config_code import apply as applier
from app.config_code import plan as planner
from app.config_code import schema
from app.config_code.export import export_yaml
from app.config_code.loader import MAX_BYTES, ConfigError, parse
from app.database import get_db
from app.errors import ErrorCode, api_error
router = APIRouter(prefix="/config", tags=["config"])
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
# and read before parsing so a huge body never reaches the JSON parser.
MAX_REQUEST_BYTES = 4 * MAX_BYTES
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
# open to read-only accounts and read tokens.
_busy = asyncio.Semaphore(1)
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
MAX_WAITING = 4
_waiting = 0
class _Slot:
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
async def __aenter__(self):
global _waiting
if _waiting >= MAX_WAITING:
raise api_error(429, ErrorCode.CONFIG_BUSY)
_waiting += 1
return self
async def run(self, coro_fn):
async with _busy:
return await coro_fn()
async def __aexit__(self, *exc):
global _waiting
_waiting -= 1
def _encode(text: str) -> bytes:
try:
return text.encode("utf-8")
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
class ImportRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
yaml: str
prune: bool = False
class ApplyRequest(ImportRequest):
plan_token: str = Field(min_length=64, max_length=64)
def _config_error(exc: ConfigError):
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
async def _read_body(request: Request, model: type[BaseModel]):
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
chunks, size = [], 0
async for chunk in request.stream():
size += len(chunk)
if size > MAX_REQUEST_BYTES:
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
chunks.append(chunk)
try:
return model.model_validate(json.loads(b"".join(chunks)))
except (ValueError, ValidationError):
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
@router.get("/export")
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
wanted = None
if kinds:
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
if not wanted <= set(schema.KINDS):
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
text = await export_yaml(db, wanted)
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
@router.post("/import/plan")
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
async with _Slot() as slot:
body = await _read_body(request, ImportRequest)
raw = _encode(body.yaml)
async def work():
try:
parsed = await run_in_threadpool(parse, raw)
return await planner.build_plan(db, parsed, body.prune)
except ConfigError as exc:
raise _config_error(exc) from None
finally:
await db.rollback() # never leave anything in the session: plan is read-only
result = await slot.run(work)
return result.public()
@router.post("/import/apply")
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
async with _Slot() as slot:
body = await _read_body(request, ApplyRequest)
raw = _encode(body.yaml)
async def work():
try:
parsed = await run_in_threadpool(parse, raw)
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
except ConfigError as exc:
await db.rollback()
raise _config_error(exc) from None
except applier.PlanStale:
await db.rollback()
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
except applier.PlanBlocked as exc:
await db.rollback()
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
except applier.SelfCheckFailed as exc:
await db.rollback()
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
except Exception:
await db.rollback()
raise
await db.commit()
return parsed, result
parsed, result = await slot.run(work)
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
+18 -6
View File
@@ -8,7 +8,7 @@ old and the new values."""
import ipaddress
from typing import Any
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, Request
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
@@ -16,7 +16,7 @@ from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.governance import GovernanceSettings
from app.models.user import AdminUser
from app.routers.auth import current_user
from app.routers.auth import client_ip, current_user
from app.services import audit_service
router = APIRouter(prefix="/settings/governance", tags=["governance"])
@@ -37,6 +37,9 @@ class GovernanceRead(BaseModel):
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
cleanup_management_ports: list[int] = Field(default_factory=list)
cleanup_management_ports_configured: bool = False
# The address this request came from, as a suggestion for the management sources (B12; admins only). Only a
# suggestion: the firewalls may see the workstation under another address (NAT, VPN).
your_address: str | None = None
class GovernanceUpdate(BaseModel):
@@ -105,14 +108,23 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
return row
def _suggestion(request: Request, user: AdminUser) -> str | None:
if user.role != "admin":
return None
try:
return str(ipaddress.ip_address(client_ip(request)))
except ValueError:
return None
@router.get("", response_model=GovernanceRead)
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
async def read(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
return _read(await get_settings(db), user)
return _read(await get_settings(db), user).model_copy(update={"your_address": _suggestion(request, user)})
@router.put("", response_model=GovernanceRead)
async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
async def update(payload: GovernanceUpdate, request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
if user.role != "admin":
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
row = await get_settings(db)
@@ -137,4 +149,4 @@ async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db),
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
await db.commit()
return _read(row, user)
return _read(row, user).model_copy(update={"your_address": _suggestion(request, user)})
+6
View File
@@ -99,6 +99,12 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
# VPN objects target groups too; their link rows are ON DELETE CASCADE, so without this check the group would
# vanish and the VPN would silently lose its targets (B15, found while designing the config import).
from app.models.policy import ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups
for assignment in (ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups):
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET)
from app.models.rollout import rollout_wave_groups
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
+41 -2
View File
@@ -5,7 +5,7 @@ from app.models.port_forward_deployment import PortForwardDeployment
from app.models.one_to_one_deployment import OneToOneDeployment
from app.models.outbound_nat_deployment import OutboundNatDeployment
from fastapi import APIRouter, Depends, HTTPException, Response
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -107,11 +107,37 @@ async def tls_probe(payload: TlsProbeRequest):
raise api_error(502, ErrorCode.TLS_PROBE_FAILED)
def _same_url(a: str, b: str) -> bool:
return a.strip().rstrip("/").lower() == b.strip().rstrip("/").lower()
async def _ensure_url_free(db: AsyncSession, base_url: str, exclude_id: uuid.UUID | None = None) -> None:
"""One firewall, one entry: the same address twice would double every rollout, alert and backup (T4)."""
for other in (await db.scalars(select(OpnsenseHost))).all():
if other.id != exclude_id and _same_url(other.base_url, base_url):
raise api_error(409, ErrorCode.HOST_URL_EXISTS, host=other.name)
def _connection_failure_reason(exc: Exception) -> str:
"""A short machine code for the UI (hosts.connection.<code>): why the first call to the firewall failed."""
text = str(exc)
if isinstance(exc, opnsense_client.OpnsenseForbiddenError):
return "forbidden"
if text.startswith("401 ") or text.startswith("403 "):
return "credentials" if text.startswith("401 ") else "forbidden"
if "TLS-Zertifikat" in text or "CERTIFICATE_VERIFY_FAILED" in text:
return "tls"
if "Verbindung zu" in text:
return "unreachable"
return "other"
@router.post("", response_model=HostRead, status_code=201)
async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
existing = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == payload.name))
if existing:
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
await _ensure_url_free(db, payload.base_url)
if payload.policy_hierarchy_group_id and not await db.get(PolicyHierarchyGroup, payload.policy_hierarchy_group_id):
raise api_error(400, ErrorCode.POLICY_HIERARCHIE_GRUPPE_NICHT_GEFUNDEN)
@@ -130,6 +156,14 @@ async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
db.add(host)
await db.flush()
opnsense_secret_store.save_secret(host.id, payload.api_secret)
if payload.verify_connection:
# The call that decides "online" in the host list: if it works here, the host shows up online right away.
try:
await opnsense_client.get_vip_status(host)
except opnsense_client.OpnsenseAPIError as exc:
await db.rollback()
opnsense_secret_store.delete_secret(host.id)
raise api_error(422, ErrorCode.HOST_CONNECTION_FAILED, reason=_connection_failure_reason(exc)) from exc
await db.commit()
statuses = await get_statuses([host])
@@ -161,6 +195,7 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
host.name = payload.name
if payload.base_url is not None:
await _ensure_url_free(db, payload.base_url, exclude_id=host_id)
host.base_url = payload.base_url
if payload.verify_tls is not None:
host.verify_tls = payload.verify_tls
@@ -188,11 +223,15 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
@router.delete("/{host_id}", status_code=204)
async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
async def delete_host(host_id: uuid.UUID, confirm_name: str = Query(default="", max_length=120), db: AsyncSession = Depends(get_db)):
"""Deleting a host drops its stored credentials at once, so the caller must name it (like every other destructive
action in OPNexus), not only a click in the UI: `DELETE /api/hosts/{id}?confirm_name=<name>` (T4)."""
await policy_lock(db)
host = await db.get(OpnsenseHost, host_id)
if not host:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
if confirm_name != host.name:
raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [host])
+24 -22
View File
@@ -16,6 +16,7 @@ from app.services import ipsec_psk_store
from app.services.ipsec_service import effective_hosts, preview, revision, snapshot, workflow, IPSEC_FIELDS
from app.services.ipsec_deploy_service import deploy_tunnel, undeploy_tunnel
from app.services.opnsense_client import OpnsenseAPIError
from app.services.deploy_lease import resource_lease
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/ipsec/tunnels", tags=["ipsec"])
@@ -192,26 +193,27 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_child_uuid:
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
if deployment.opnsense_local_uuid:
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
if deployment.opnsense_remote_uuid:
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
if deployment.opnsense_connection_uuid:
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
if deployment.opnsense_psk_uuid:
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
await opnsense_client.reconfigure_ipsec(deployment.host)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
async with resource_lease("ipsec", tunnel.id):
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_child_uuid:
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
if deployment.opnsense_local_uuid:
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
if deployment.opnsense_remote_uuid:
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
if deployment.opnsense_connection_uuid:
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
if deployment.opnsense_psk_uuid:
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
await opnsense_client.reconfigure_ipsec(deployment.host)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
await db.commit()
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
await db.delete(deployment)
await db.commit()
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
await db.delete(deployment)
await db.commit()
+102 -8
View File
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
from app.database import get_db
from app.models import OpnsenseHost
from app.models.host_operation import HostOperation
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
class Preview(BaseModel):
action: Literal['reboot','update','major_update']
action: Literal['reboot','update','major_update','firmware_rollback']
class Execute(BaseModel):
preview_id: uuid.UUID
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
def read_operation(op):
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
async def snapshot_or_502(db, host):
try: return await inspect_host(db,host)
async def snapshot_or_502(db, host, action=None):
try:
state=await inspect_host(db,host)
if action=='firmware_rollback':
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
rollback=await snapshot_service.rollback_state(host)
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
state['rollback']=rollback
state['actions']['firmware_rollback']=state['actions']['reboot']+(
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
return state
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
async def _changes_since(db, host, target):
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
if not target:
return []
since=_point_time(target)
if since is None:
return []
out=[]
for kind,model in _DEPLOYMENT_MODELS():
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
return sorted(out,key=lambda r:r['deployed_at'])[:50]
def _point_time(target):
try:
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
except (KeyError, TypeError, ValueError):
return None
def _DEPLOYMENT_MODELS():
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
PortForwardDeployment, RuleDeployment)
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
async def _mark_rolled_back(db, host, target):
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
really writes, and the UI shows the object as not in sync."""
since=_point_time(target)
if since is None:
return 0
count=0
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
for _kind,model in _DEPLOYMENT_MODELS():
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
row.status='error'; row.message=note[:500]; row.last_payload=None
if hasattr(model,'revert_deadline'): row.revert_deadline=None
count+=1
return count
async def carp_snapshot_or_502(db, host):
try: return await inspect_carp_switch(db,host)
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
host=await host_or_404(db,host_id)
state=await snapshot_or_502(db,host)
state=await snapshot_or_502(db,host,payload.action)
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
state=await snapshot_or_502(db,host)
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
state=await snapshot_or_502(db,host,op.action)
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
# replaying a reboot/update automatically could interrupt recovery.
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
if op.action in ('update','major_update'):
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
try:
point=await snapshot_service.create_rollback_point(host,state.get('version'))
except client.OpnsenseAPIError as exc:
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
await db.commit()
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
return read_operation(op)
op.snapshot={**op.snapshot,'rollback_point':point}
await db.commit()
if op.action=='firmware_rollback':
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
# Fail closed, like the backup before an update.
try:
backup_store.save_snapshot(host.id, await client.download_backup(host))
except client.OpnsenseAPIError as exc:
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
await db.commit()
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
return read_operation(op)
rollback=state['rollback']
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
else:
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
try:
result=await action_fn(host)
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
op.status='submitted';op.message=done
if op.action=='firmware_rollback':
marked=await _mark_rolled_back(db,host,rollback['target'])
if marked:
op.message+=f' {marked} Deployment(s) als unklar markiert.'
except client.OpnsenseAPIError as exc:
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
result={}
+1
View File
@@ -50,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
source_net=rule.source_net,
+18 -16
View File
@@ -18,6 +18,7 @@ from app.services import openvpn_key_store
from app.services.openvpn_service import effective_hosts, preview, revision, snapshot, workflow, OPENVPN_FIELDS
from app.services.openvpn_deploy_service import deploy_instance, undeploy_instance
from app.services.opnsense_client import OpnsenseAPIError
from app.services.deploy_lease import resource_lease
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/openvpn/instances", tags=["openvpn"])
@@ -194,20 +195,21 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
deployment = next(d for d in instance.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_instance_uuid:
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
if deployment.opnsense_statickey_uuid:
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
await opnsense_client.reconfigure_openvpn(deployment.host)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
async with resource_lease("openvpn", instance.id):
deployment = next(d for d in instance.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_instance_uuid:
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
if deployment.opnsense_statickey_uuid:
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
await opnsense_client.reconfigure_openvpn(deployment.host)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
await db.commit()
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
await db.delete(deployment)
await db.commit()
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
await db.delete(deployment)
await db.commit()
+1
View File
@@ -51,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
source_net=rule.source_net,
+1
View File
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
hierarchy_group_id=rule.hierarchy_group_id,
rulebase=rule.rulebase,
sequence=rule.sequence,
+1
View File
@@ -50,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
workflow_state=state,
workflow_counts=counts,
id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
ipprotocol=rule.ipprotocol,
+27 -2
View File
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
from typing import Literal
from fastapi import APIRouter, Depends, Request
from pydantic import BaseModel, Field
from pydantic import BaseModel, Field, model_validator
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -20,7 +20,7 @@ from app.models.maintenance_window import MaintenanceWindow
from app.models.policy import DeviceGroup
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
@@ -36,6 +36,19 @@ class WaveIn(BaseModel):
require_approval: bool = False
class ProbeIn(BaseModel):
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
type: Literal["ping", "tcp"]
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
port: int | None = Field(default=None, ge=1, le=65535)
@model_validator(mode="after")
def _port_only_for_tcp(self):
if (self.type == "tcp") != (self.port is not None):
raise ValueError("port_required_for_tcp_only")
return self
class PlanIn(BaseModel):
name: str = Field(min_length=1, max_length=120)
description: str = Field(default="", max_length=2000)
@@ -43,6 +56,7 @@ class PlanIn(BaseModel):
window_id: uuid.UUID | None = None
auto_start: bool = False
gate_checks: dict[str, bool] = Field(default_factory=dict)
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
compliance_mode: Literal["warn", "fail"] = "warn"
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
@@ -64,6 +78,7 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
return {
**rollout_service.definition(plan),
"gate_probes": list(plan.gate_probes or []),
"description": plan.description,
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
@@ -81,6 +96,15 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
}
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
out: list[dict] = []
for probe in probes:
value = {"type": probe.type, "target": probe.target, "port": probe.port}
if value not in out:
out.append(value)
return out
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
name = payload.name.strip()
if not name:
@@ -99,6 +123,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
plan.gate_probes = _unique_probes(payload.gate_probes)
plan.compliance_mode = payload.compliance_mode
plan.waves = [
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
+1
View File
@@ -52,6 +52,7 @@ def _to_read(
return RuleRead(
workflow_state=state,
workflow_counts=counts, id=rule.id,
config_key=rule.config_key,
description=rule.description,
interface=rule.interface,
direction=rule.direction,
+35 -12
View File
@@ -1,7 +1,7 @@
import uuid
from fastapi import APIRouter, Depends
from sqlalchemy import func, select
from sqlalchemy import func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.errors import ErrorCode, api_error
@@ -21,10 +21,29 @@ async def _admin_count(db: AsyncSession, exclude_id: uuid.UUID | None = None) ->
return await db.scalar(query)
_ADMIN_LOCK_KEY = 728194032 # next to policy_lock's key; only admin role changes and deletions take it
async def _lock_admins(db: AsyncSession, current: AdminUser) -> None:
"""Serialize every change that can reduce the number of administrators and re-read the caller under the lock.
The "at least one administrator" check counts rows; two requests that each see the other's admin row would both
pass it. Held until commit/rollback (transaction-scoped advisory lock). The caller may have been demoted or
deleted by the request that ran first, so it must still be an admin to proceed."""
await db.execute(text("SELECT pg_advisory_xact_lock(:key)"), {"key": _ADMIN_LOCK_KEY})
fresh = await db.scalar(select(AdminUser).where(AdminUser.id == current.id).execution_options(populate_existing=True))
if fresh is None or fresh.role != "admin":
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
@router.get("", response_model=list[UserRead])
async def list_users(db: AsyncSession = Depends(get_db)):
rows = (await db.scalars(select(AdminUser).order_by(AdminUser.username))).all()
return rows
async def list_users(user: AdminUser = Depends(current_user), db: AsyncSession = Depends(get_db)):
"""Admins see everyone. Every other role sees only its own account: the full list (names, roles, last login,
2FA status) is a directory an attacker with any low-privilege account or token could use to pick targets."""
query = select(AdminUser).order_by(AdminUser.username)
if user.role != "admin":
query = query.where(AdminUser.id == user.id)
return (await db.scalars(query)).all()
@router.post("", response_model=UserRead, status_code=201)
@@ -43,8 +62,12 @@ async def create_user(payload: UserCreate, db: AsyncSession = Depends(get_db)):
@router.patch("/{user_id}/role", response_model=UserRead)
async def update_role(user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db)):
target = await db.get(AdminUser, user_id)
async def update_role(
user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db),
current: AdminUser = Depends(current_user),
):
await _lock_admins(db, current)
target = await db.get(AdminUser, user_id, populate_existing=True)
if target is None:
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
if target.auth_source == "oidc":
@@ -96,13 +119,13 @@ async def delete_user(
):
if user_id == current.id:
raise api_error(400, ErrorCode.DER_EIGENE_ACCOUNT_KANN_NICHT_GELOESCHT_WERDEN)
target = await db.get(AdminUser, user_id)
await _lock_admins(db, current)
target = await db.get(AdminUser, user_id, populate_existing=True)
if target is None:
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
# No separate "last admin" guard here: only an admin can reach this
# endpoint (read_only is rejected earlier, in AuthMiddleware) and the
# self-delete check above already guarantees target != current — so
# `current` is always a still-existing admin afterward, meaning the
# admin count can never reach zero this way.
# Two admins deleting each other (or demoting/deleting in parallel) would each still see the other: the lock above
# orders them, and the guard below counts what is left without the target.
if target.role == "admin" and await _admin_count(db, exclude_id=target.id) == 0:
raise api_error(409, ErrorCode.MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN)
await db.delete(target)
await db.commit()
+18 -16
View File
@@ -18,6 +18,7 @@ from app.services import wireguard_key_store
from app.services.wireguard_service import effective_hosts, preview, revision, snapshot, workflow, WIREGUARD_FIELDS
from app.services.wireguard_deploy_service import deploy_tunnel, undeploy_tunnel
from app.services.opnsense_client import OpnsenseAPIError
from app.services.deploy_lease import resource_lease
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/wireguard/tunnels", tags=["wireguard"])
@@ -206,20 +207,21 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
if not entry["can_remove"]:
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_peer_uuid:
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
if deployment.opnsense_server_uuid:
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
await opnsense_client.reconfigure_wireguard(deployment.host)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
async with resource_lease("wireguard", tunnel.id):
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
from app.services.host_maintenance_service import ensure_no_open_cleanup
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
from app.services import opnsense_client
try:
if deployment.opnsense_peer_uuid:
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
if deployment.opnsense_server_uuid:
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
await opnsense_client.reconfigure_wireguard(deployment.host)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
await db.commit()
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
await db.delete(deployment)
await db.commit()
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
await db.delete(deployment)
await db.commit()
+3 -1
View File
@@ -12,7 +12,9 @@ CertAction = Literal["import", "internal", "external", "sign_csr", "import_csr",
# EC curves) — see ManagedCertificate's docstring.
CertKeyType = Literal["2048", "3072", "4096", "prime256v1", "secp384r1", "secp521r1"]
CertDigest = Literal["sha1", "sha224", "sha256", "sha384", "sha512"]
CertType = Literal["usr_cert", "server_cert", "combined_server_client", "v3_ca"]
# No "v3_ca": OPNsense would create a certificate with the CA extension, not a CA that can be used as `caref`
# (found by the 1.31.0 test run, T3). Creating a CA is not offered; the UI never did.
CertType = Literal["usr_cert", "server_cert", "combined_server_client"]
_DN_ACTIONS = {"internal", "external", "reissue"}
_CRT_ACTIONS = {"import", "import_csr", "manual"}
+3
View File
@@ -18,6 +18,9 @@ class HostCreate(BaseModel):
enabled: bool = True
cluster_name: str | None = Field(default=None, max_length=120)
policy_hierarchy_group_id: uuid.UUID | None = None
# Ask the firewall once with these credentials before saving (the UI and the setup assistant do). Off by default so
# scripts can still register a firewall that is not reachable yet (behind NAT, tunnel not up).
verify_connection: bool = False
@field_validator("base_url")
@classmethod
+1
View File
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
source_net: str
+1
View File
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
source_net: str
+1
View File
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
hierarchy_group_id: uuid.UUID
rulebase: Literal["pre", "post"]
sequence: int
+1
View File
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
ipprotocol: str
+1
View File
@@ -48,6 +48,7 @@ class RuleRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
config_key: str
description: str
interface: str
direction: str
+2 -1
View File
@@ -13,6 +13,7 @@ from app.models.alias_deployment import AliasDeployment
from app.models.host import OpnsenseHost
from app.schemas.host import AliasOption
from app.services import opnsense_client
from app.services.deploy_flow import unsettled
from app.services.opnsense_client import OpnsenseAPIError
@@ -87,7 +88,7 @@ async def ensure_aliases_settled(db: AsyncSession, hosts: list[OpnsenseHost], va
.where(
FirewallAlias.name.in_(names),
AliasDeployment.host_id.in_([host.id for host in hosts]),
AliasDeployment.status.in_(("pending", "revert_failed")),
unsettled(AliasDeployment),
)
.order_by(FirewallAlias.name, OpnsenseHost.name)
)).all()
+8 -4
View File
@@ -45,8 +45,9 @@ _SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-tick
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
# notification settings the webhook URL/SMTP data): admin scope only.
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
_MCP_PATH = "/api/mcp"
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
# it compares a posted file with the database and writes nothing.
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
_ADMIN_ONLY_WRITES = re.compile(
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
r"|^/api/logs/settings$"
@@ -54,6 +55,8 @@ _ADMIN_ONLY_WRITES = re.compile(
r"|^/api/hosts/[^/]+/gui-ticket$"
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
r"|^/api/config/import/apply$"
)
@@ -76,7 +79,7 @@ def denies(scope: str, method: str, path: str) -> bool:
"""True if a token with this scope must not make this request."""
if _SESSION_ONLY.match(path):
return True
if path == _MCP_PATH:
if path in _READ_ONLY_POSTS:
return False
if method in _SAFE_METHODS:
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
@@ -98,7 +101,7 @@ def operator_denies(method: str, path: str) -> bool:
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
# maintenance (reboot/update/CARP switch), scheduling it, applying network changes, pushing certificates and
# ACME accounts, re-asserting the hierarchy order, controlling services, and DELETING a centrally managed object
# or snapshot (deleting a rule removes it from every firewall it was deployed to).
# or snapshot (deleting a rule removes it from every firewall it was deployed to), and creating a snapshot.
_ROLLOUT_POST = re.compile(
r"/(deploy|push|sign|poll|register|reassert-ordering"
r"|deployments/[^/]+/(remove|retry-rollback)"
@@ -107,6 +110,7 @@ _ROLLOUT_POST = re.compile(
r"|rule-cleanup/[^/]+/(revert|retry)"
r"|maintenance/(execute|carp-execute|schedule)"
r"|network/changes"
r"|snapshots" # creating a boot environment (deleting one is in _ROLLOUT_DELETE)
r"|services/[^/]+/(start|stop|restart)"
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
r"|rollouts/plans/[^/]+/(start|activate)"
+13 -5
View File
@@ -236,6 +236,7 @@ async def create_session(
user_agent: str | None = None,
ip_address: str | None = None,
verified_hash: str | None = None,
verified_totp: bool | None = None,
lifetime_seconds: int | None = None,
) -> tuple[str, datetime]:
"""Create a session, returning the raw token (to be set as a cookie) and its expiry.
@@ -244,15 +245,22 @@ async def create_session(
against. The user row is re-read under a row lock first: a password
change/reset that committed while the (slow) password check was running
changed the hash, and the login must not outlive that revocation.
``verified_totp`` is whether the caller checked a second factor. Two-factor
enabled in the meantime (it also revoked every session) means this login
was decided without it and must start over.
"""
user_id = user.id
if verified_hash is not None:
locked = await db.scalar(
select(AdminUser.password_hash).where(AdminUser.id == user_id).with_for_update()
)
if locked != verified_hash:
if verified_hash is not None or verified_totp is not None:
locked_hash, locked_totp = (await db.execute(
select(AdminUser.password_hash, AdminUser.totp_enabled).where(AdminUser.id == user_id).with_for_update()
)).one()
if verified_hash is not None and locked_hash != verified_hash:
await db.rollback()
raise api_error(401, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
if locked_totp and not verified_totp:
await db.rollback()
raise api_error(401, ErrorCode.TOTP_REQUIRED)
token = secrets.token_urlsafe(SESSION_TOKEN_BYTES)
expires_at = _now() + timedelta(seconds=lifetime_seconds or settings.session_lifetime)
session = Session(
+24 -5
View File
@@ -29,6 +29,7 @@ and the sweeper away in the gap (pending/unknown rows are never re-deployed).
from datetime import datetime, timedelta, timezone
from typing import Any, Awaitable, Callable
from sqlalchemy import and_, or_
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
@@ -46,7 +47,21 @@ def _now() -> datetime:
return datetime.now(timezone.utc)
async def _checkpoint(db: AsyncSession) -> None:
def is_unsettled(row) -> bool:
"""A change that may still be live and whose rollback is not yet done: pending, outcome unknown, or an
"error" row whose revert deadline is still armed (written but apply/ordering not confirmed)."""
return row.status in ("pending", "revert_failed") or (row.status == "error" and row.revert_deadline is not None)
def unsettled(model):
"""``is_unsettled`` as a SQL condition for the deployment model."""
return or_(
model.status.in_(("pending", "revert_failed")),
and_(model.status == "error", model.revert_deadline.isnot(None)),
)
async def checkpoint(db: AsyncSession) -> None:
from app.services.policy_service import policy_lock
await db.commit()
@@ -67,12 +82,16 @@ async def push_to_host(
) -> Any:
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
before = {name: getattr(existing, name) for name in _FIELDS} if existing else None
# An armed "error" row (written, apply not confirmed) holds the UNCONFIRMED payload as last_payload: a retry must
# keep the original rollback target instead of promoting that payload to it.
armed = existing is not None and existing.status == "error" and existing.revert_deadline is not None
rollback_target = existing.previous_payload if armed else prior_live_payload
# 1. durable intent
if existing is not None:
row = existing
row.status, row.message = "pending", None
row.previous_payload, row.last_payload = prior_live_payload, payload
row.previous_payload, row.last_payload = rollback_target, payload
row.revert_deadline, row.deployed_at = _now() + window, _now()
else:
row = make(
@@ -83,7 +102,7 @@ async def push_to_host(
# The intent is a technical safety record, not a state change worth a history
# entry of its own (the request itself is already audited as "requested").
row._skip_audit = True
await _checkpoint(db)
await checkpoint(db)
row._skip_audit = False
wrote = False
@@ -93,9 +112,9 @@ async def push_to_host(
# 3. recoverable by UUID from here on
setattr(row, uuid_attr, new_uuid)
row.status, row.message = "pending", None
row.previous_payload = prior_live_payload if existing is not None else None
row.previous_payload = rollback_target if existing is not None else None
row.last_payload, row.revert_deadline, row.deployed_at = payload, _now() + window, _now()
await _checkpoint(db)
await checkpoint(db)
await apply()
except OpnsenseAPIError as exc:
ambiguous = isinstance(exc, OpnsenseAmbiguousError)
+39
View File
@@ -0,0 +1,39 @@
"""An exclusive per-object lock for flows that make several remote calls and commit in between (VPN deploys).
``policy_lock`` is transaction-scoped and is released by every checkpoint commit; this one is a session-level advisory
lock on a connection of its own, so it spans the commits of the request session. If the process dies the connection
closes and PostgreSQL frees the lock. It never waits: a second deploy/removal of the same object is refused at once.
"""
import uuid
import zlib
from contextlib import asynccontextmanager
from sqlalchemy import text
from app.database import engine
from app.errors import ErrorCode, api_error
# Written when the first row is committed, before any remote call: a crash or cancellation leaves this, and a new
# deploy continues from the UUIDs recorded after it.
VPN_INTERRUPTED = (
"Ausrollen unterbrochen oder Ausgang unklar: Die Firewall hat möglicherweise schon Teile der Konfiguration "
"erhalten. Erneutes Ausrollen setzt dort fort."
)
@asynccontextmanager
async def resource_lease(kind: str, resource_id: uuid.UUID):
params = {"a": zlib.crc32(kind.encode()) & 0x7FFFFFFF, "b": resource_id.int & 0x7FFFFFFF}
async with engine.connect() as conn:
if not await conn.scalar(text("SELECT pg_try_advisory_lock(:a, :b)"), params):
raise api_error(409, ErrorCode.VPN_DEPLOY_LAEUFT_BEREITS)
try:
yield
finally:
try:
await conn.execute(text("SELECT pg_advisory_unlock(:a, :b)"), params)
except Exception:
await conn.invalidate() # never hand a connection that may still hold the lock back to the pool
except BaseException: # cancelled while unlocking
await conn.invalidate()
raise
+14 -2
View File
@@ -63,6 +63,17 @@ async def _guarded(host: OpnsenseHost, tool: str, target: str, params: dict, cal
return result
def _with_silent_hops(hops: list[dict]) -> list[dict]:
"""The firewall leaves out hops that never answered (TTL jumps 1 -> 3, T5 of the 1.31.0 test run). Put them back as
explicit rows so a gap reads as "no answer here", not as a missing hop. A route where nothing answered at all stays
an empty list; the UI says so."""
seen = {int(h["ttl"]): h for h in hops if h["ttl"].isdigit()}
if not seen:
return hops
silent = {"address": "", "host": "", "as": "", "probes": "", "silent": True}
return [seen.get(ttl) or {**silent, "ttl": str(ttl)} for ttl in range(1, max(seen) + 1)]
async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: str, source: str | None) -> dict:
target, source = clean_target(target), clean_source(source)
params = {"target": target, "ipproto": ipproto, "protocol": protocol, "source": source}
@@ -73,8 +84,9 @@ async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: st
{"hostname": target, "ipproto": ipproto, "protocol": protocol, "source_address": source})
resp = data.get("response") or {}
hops = [{"ttl": str(r.get("ttl", "")), "address": str(r.get("address", "")), "host": str(r.get("host", "")),
"as": str(r.get("AS", "")), "probes": str(r.get("probes", ""))} for r in resp.get("rows", []) if isinstance(r, dict)]
return {"target": target, "notice": str(resp.get("notice", "")), "hops": hops}
"as": str(r.get("AS", "")), "probes": str(r.get("probes", "")), "silent": False}
for r in resp.get("rows", []) if isinstance(r, dict)]
return {"target": target, "notice": str(resp.get("notice", "")), "hops": _with_silent_hops(hops)}
return await _guarded(host, "traceroute", target, params, call)
@@ -12,6 +12,7 @@ from app.models import OpnsenseHost, RuleDeployment, AliasDeployment, PortForwar
from app.models.host_operation import HostOperation
from app.models.restore_operation import RestoreOperation
from app.services import opnsense_client as client, backup_store, carp_maintenance_window_service
from app.services.deploy_flow import unsettled
from app.services.host_status_service import get_statuses
DEPLOYMENTS = (RuleDeployment, AliasDeployment, PortForwardDeployment, OneToOneDeployment, OutboundNatDeployment)
@@ -70,7 +71,7 @@ async def _shared_change_blockers(db, fleet):
blocked = []
fleet_ids = [h.id for h in fleet]
for model in DEPLOYMENTS:
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), model.status.in_(("pending", "revert_failed"))).limit(1)):
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), unsettled(model)).limit(1)):
blocked.append("Offene Bestätigung oder fehlgeschlagener Rollback im Host/Cluster.")
break
operations = list((await db.scalars(select(HostOperation).where(HostOperation.host_id.in_(fleet_ids), HostOperation.status.in_(ACTIVE)))).all())
+25 -11
View File
@@ -109,6 +109,8 @@ from app.models.ipsec_tunnel import IpsecTunnel
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
from app.services import opnsense_client, ipsec_psk_store
from app.services.opnsense_client import OpnsenseAPIError
from app.services.deploy_flow import checkpoint
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
_VERSION_MAP = {"ike": "0", "ikev1": "1", "ikev2": "2"}
@@ -176,7 +178,7 @@ def build_payloads(tunnel: IpsecTunnel) -> dict:
}
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
"""Runs the 5 config-write calls in dependency order, persisting each
UUID onto `deployment` as soon as it's known so a later retry (after a
partial failure) updates rather than duplicates. Raises
@@ -189,6 +191,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
deployment.opnsense_connection_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_connection_uuid:
raise OpnsenseAPIError("Firewall hat keine Connection-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
# `payloads` (built by build_payloads()) is also what gets persisted as
# last_payload for no-op-redeploy comparison — the injected `connection`
@@ -209,6 +212,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
deployment.opnsense_local_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_local_uuid:
raise OpnsenseAPIError("Firewall hat keine Local-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
if deployment.opnsense_remote_uuid:
await opnsense_client.set_ipsec_remote(host, deployment.opnsense_remote_uuid, remote_wire)
@@ -217,6 +221,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
deployment.opnsense_remote_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_remote_uuid:
raise OpnsenseAPIError("Firewall hat keine Remote-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
if deployment.opnsense_child_uuid:
await opnsense_client.set_ipsec_child(host, deployment.opnsense_child_uuid, child_wire)
@@ -225,6 +230,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
deployment.opnsense_child_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_child_uuid:
raise OpnsenseAPIError("Firewall hat keine Child-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
if deployment.opnsense_psk_uuid:
await opnsense_client.set_ipsec_psk(host, deployment.opnsense_psk_uuid, payloads["psk"])
@@ -233,6 +239,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
deployment.opnsense_psk_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_psk_uuid:
raise OpnsenseAPIError("Firewall hat keine PSK-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseHost) -> IpsecTunnelDeployment:
@@ -241,7 +248,7 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
existing = await db.scalar(
select(IpsecTunnelDeployment).where(
IpsecTunnelDeployment.tunnel_id == tunnel.id, IpsecTunnelDeployment.host_id == host.id
)
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
)
if existing and existing.status == "success" and existing.last_payload == comparable:
# Nothing actually changed on this host — re-affirm without
@@ -252,8 +259,15 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
deployment = existing or IpsecTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
if not existing:
db.add(deployment)
await checkpoint(db)
try:
await _write_config(host, payloads, deployment)
await _write_config(db, host, payloads, deployment)
# Same trap as WireGuard: with the global IPsec switch off the config
# is stored but nothing runs, and reconfigure still answers ok.
await opnsense_client.ensure_ipsec_enabled(host)
@@ -261,8 +275,6 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
deployment.status = "error"
deployment.message = str(exc)
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
try:
@@ -277,28 +289,30 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
deployment.last_payload = comparable
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
deployment.status = "success"
deployment.message = None
deployment.last_payload = comparable
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
async def deploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel, hosts: list[OpnsenseHost]) -> list[IpsecTunnelDeployment]:
from app.services.host_maintenance_service import ensure_no_maintenance
await ensure_no_maintenance(db, hosts)
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
await db.commit()
async with resource_lease("ipsec", tunnel.id):
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
await db.commit()
return deployments
async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
async with resource_lease("ipsec", tunnel.id):
await _undeploy_tunnel(db, tunnel)
async def _undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
"""Explicit action, not gated by anything (there is no lockout state
for this entity to gate on) — deletes Child/Local/Remote before their
parent Connection (the ModelRelationField dependency direction), then
@@ -87,6 +87,13 @@ OBJECT_COLUMNS: dict[str, list[str]] = {
# These change how the network is wired; the others (DNS/DHCP data) cannot cut the connection to the firewall.
WIRING_KINDS = {"vlan", "vip", "gateway", "route", "interface", "lagg", "bridge", "gre", "gif", "vxlan"}
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
# Integer ranges of the firewall's own model (Interfaces/Vlan.xml, Interfaces/Vip.xml, read 2026-10-23). Checked before the
# preview so a wrong number is a 422 with the field name, not a 502 after the firewall refused it (T6 of the 1.31.0 test
# run). Only fields whose range is stated in the model; an empty value is left to the firewall (e.g. vhid of a non-CARP VIP).
RANGES: dict[tuple[str, str], tuple[int, int]] = {
("vlan", "tag"): (1, 4094), ("vlan", "pcp"): (0, 7),
("vip", "vhid"): (1, 255), ("vip", "advbase"): (1, 254), ("vip", "advskew"): (0, 254),
}
def validate_fields(kind: str, fields: dict) -> dict[str, str]:
@@ -104,6 +111,9 @@ def validate_fields(kind: str, fields: dict) -> dict[str, str]:
text = str(value).strip()
if len(text) > 255 or _CONTROL.search(text):
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
bounds = RANGES.get((kind, key))
if bounds and text and not (text.isdecimal() and bounds[0] <= int(text) <= bounds[1]):
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
clean[key] = text
return clean
+23 -11
View File
@@ -165,6 +165,8 @@ from app.models.openvpn_instance import OpenVpnInstance
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
from app.services import opnsense_client, openvpn_key_store
from app.services.opnsense_client import OpnsenseAPIError
from app.services.deploy_flow import checkpoint
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
def _now() -> datetime:
@@ -252,7 +254,7 @@ async def _next_vpnid(host: OpnsenseHost) -> str:
return str(max(used, default=0) + 1)
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnInstanceDeployment) -> None:
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: OpenVpnInstanceDeployment) -> None:
"""Writes the StaticKey (if enabled) then the Instance, in that
dependency order, persisting each UUID onto `deployment` as soon as
it's known so a later retry (after a partial failure) updates rather
@@ -267,11 +269,13 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnI
deployment.opnsense_statickey_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_statickey_uuid:
raise OpnsenseAPIError("Firewall hat keine StaticKey-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
elif deployment.opnsense_statickey_uuid:
# tls_key hardening was switched off after previously being on —
# remove the now-orphaned StaticKey rather than leaving it behind.
await opnsense_client.del_openvpn_static_key(host, deployment.opnsense_statickey_uuid)
deployment.opnsense_statickey_uuid = None
await checkpoint(db)
# Overlay the deploy-time-known tls_key UUID (or "" to explicitly
# clear a previously-set relation — see module docstring) onto a
@@ -292,13 +296,14 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnI
deployment.opnsense_instance_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_instance_uuid:
raise OpnsenseAPIError("Firewall hat keine Instance-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: OpnsenseHost) -> OpenVpnInstanceDeployment:
existing = await db.scalar(
select(OpenVpnInstanceDeployment).where(
OpenVpnInstanceDeployment.instance_id == instance.id, OpenVpnInstanceDeployment.host_id == host.id
)
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
)
try:
payloads = build_payloads(instance, host.id)
@@ -323,14 +328,19 @@ async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: Opn
deployment = existing or OpenVpnInstanceDeployment(instance_id=instance.id, host_id=host.id, status="error")
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
if not existing:
db.add(deployment)
await checkpoint(db)
try:
await _write_config(host, payloads, deployment)
await _write_config(db, host, payloads, deployment)
except OpnsenseAPIError as exc:
deployment.status = "error"
deployment.message = str(exc)
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
try:
@@ -345,28 +355,30 @@ async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: Opn
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
deployment.last_payload = comparable
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
deployment.status = "success"
deployment.message = None
deployment.last_payload = comparable
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
async def deploy_instance(db: AsyncSession, instance: OpenVpnInstance, hosts: list[OpnsenseHost]) -> list[OpenVpnInstanceDeployment]:
from app.services.host_maintenance_service import ensure_no_maintenance
await ensure_no_maintenance(db, hosts)
deployments = [await _deploy_to_host(db, instance, host) for host in hosts]
await db.commit()
async with resource_lease("openvpn", instance.id):
deployments = [await _deploy_to_host(db, instance, host) for host in hosts]
await db.commit()
return deployments
async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
async with resource_lease("openvpn", instance.id):
await _undeploy_instance(db, instance)
async def _undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
"""Explicit action, not gated by anything (there is no lockout state
for this entity to gate on) — deletes the Instance before its
referenced StaticKey (the ModelRelationField dependency direction),
+100
View File
@@ -8,7 +8,10 @@ required.
Auth is HTTP Basic: username = API key, password = API secret.
"""
import ipaddress
import logging
from contextlib import contextmanager
from contextvars import ContextVar
import json
import re
from typing import Any
@@ -63,6 +66,34 @@ def _client(host: OpnsenseHost) -> httpx.AsyncClient:
)
# ── Rule counters before a reload (B11) ────────────────────────────────────
# Every request below that makes OPNsense reload pf (filter/NAT apply; alias and IPsec reconfigure run
# `filter reload skip_alias` too, verified in core stable/26.7; OpenVPN/WireGuard/network may through interface
# registration) zeroes every rule counter. Whatever was hit since the last observation would be lost and a rule
# could later look "unused". So the counters are read right before, through rule_hit_service. A writer that has
# just sampled itself (the rule cleanup) switches this off for its own call with ``counters_already_sampled()``.
_sampled_already: ContextVar[bool] = ContextVar("opnexus_counters_sampled", default=False)
@contextmanager
def counters_already_sampled():
token = _sampled_already.set(True)
try:
yield
finally:
_sampled_already.reset(token)
async def _before_reload(host: OpnsenseHost) -> None:
if _sampled_already.get():
return
from app.services.rule_hit_service import sample_before_reload # needs the database; this module does not
try:
await sample_before_reload(host)
except Exception as exc: # noqa: BLE001 - a failed sample must never block or fail the write itself
logger.info("Zählerprobe vor dem Anwenden auf %s fehlgeschlagen: %s", host.name, exc)
def _refuse_unresolved_placeholders(method: str, path: str, kwargs: dict) -> None:
"""Last line of defence for site variables: a payload that still contains a {{NAME}} placeholder was never
resolved for this host and must not reach a firewall."""
@@ -221,6 +252,7 @@ async def del_alias_item(host: OpnsenseHost, alias_uuid: str) -> dict:
async def reconfigure_aliases(host: OpnsenseHost) -> dict:
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/firewall/alias/reconfigure"), "Aliase anwenden", set())
@@ -366,6 +398,7 @@ def rule_write_payload(rule: dict) -> dict[str, str]:
async def apply(host: OpnsenseHost) -> dict:
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
@@ -471,6 +504,7 @@ async def apply_port_forward(host: OpnsenseHost) -> dict:
apply (verified in core source: neither DNatController nor
FilterController override applyAction). Called via its own semantically
named endpoint anyway, matching what OPNsense's own UI does."""
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/firewall/d_nat/apply"), "Änderungen anwenden", set())
@@ -510,6 +544,7 @@ async def del_one_to_one_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
async def apply_one_to_one(host: OpnsenseHost) -> dict:
"""Apply through the type-specific controller, using FilterBaseController reload."""
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/firewall/one_to_one/apply"), "Änderungen anwenden", set())
@@ -532,6 +567,7 @@ async def del_outbound_nat_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
async def apply_outbound_nat(host: OpnsenseHost) -> dict:
"""Apply through the type-specific controller, using FilterBaseController reload."""
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/firewall/source_nat/apply"), "Änderungen anwenden", set())
@@ -567,6 +603,59 @@ async def get_pf_rule_statistics(host: OpnsenseHost) -> dict:
# - Firewall/Api/GroupController::searchItemAction -> searchBase("ifgroupentry"): rows with
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
# wireguard) appear with their name as uuid and empty members.
def _stream_addr(response: httpx.Response, which: str) -> tuple[str, int] | None:
stream = response.extensions.get("network_stream")
try:
addr = stream.get_extra_info(which) if stream is not None else None
except Exception: # noqa: BLE001 - transport without that information
return None
return (str(addr[0]), int(addr[1])) if isinstance(addr, (tuple, list)) and len(addr) >= 2 else None
async def own_source_address(host: OpnsenseHost) -> str | None:
"""The address this firewall sees OPNexus's API requests come from (B10): behind NAT or the management tunnel it is
not what OPNexus knows about itself, so the firewall is asked. Two requests on one keep-alive connection: the first
tells the firewall address and the local port of this connection, the second lists pf's states for that address
(`diagnostics/firewall/query_states`, POST, read-only; Diagnostics/Api/FirewallController, core stable/26.7) and the
state whose source port is our local port is this very connection. Live 2026-10-23 on test-fw-1/3: one ESTABLISHED
state, src 192.0.2.10 = the OPNexus server, port preserved by Docker's NAT.
None when it cannot be told for sure: through the overlay's CONNECT proxy (the local port belongs to the proxy
connection), a NAT that rewrote the port, or an API key without the states privilege (that raises
OpnsenseAPIError)."""
async with _client(host) as http:
try:
first = await http.get("/diagnostics/firewall/pf_states")
first.raise_for_status()
server = _stream_addr(first, "server_addr")
if server is None:
return None
second = await http.post("/diagnostics/firewall/query_states",
json={"searchPhrase": server[0], "rowCount": 500, "current": 1})
second.raise_for_status()
local, server = _stream_addr(second, "client_addr"), _stream_addr(second, "server_addr")
rows = second.json().get("rows") or []
except (httpx.HTTPError, ValueError, AttributeError) as exc:
raise OpnsenseAPIError(f"Zustandstabelle von {host.name} nicht lesbar: {str(exc)[:200]}") from exc
if local is None or server is None:
return None
def same_ip(a, b) -> bool:
try:
return ipaddress.ip_address(str(a)) == ipaddress.ip_address(str(b))
except ValueError:
return False
for row in rows if isinstance(rows, list) else []:
if (isinstance(row, dict) and same_ip(row.get("dst_addr"), server[0]) and str(row.get("dst_port")) == str(server[1])
and str(row.get("src_port")) == str(local[1]) and str(row.get("state", "")).startswith("ESTABLISHED")):
try:
return str(ipaddress.ip_address(str(row.get("src_addr"))))
except ValueError:
return None
return None
async def get_rule_stats(host: OpnsenseHost) -> dict:
"""Per-label counters, already summed over every pf line of a rule. Used for the rule usage sampling
(rule_hit_service). Live check 2026-10-22 on all three test firewalls (26.7, one with unmigrated legacy
@@ -801,6 +890,7 @@ async def reconfigure_ipsec(host: OpnsenseHost) -> dict:
Deliberately its own call, separate from the 5 config-write calls
above — see ipsec_deploy_service.py for why this app never bundles it
silently."""
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/ipsec/service/reconfigure"), "IPsec-Dienst neu laden", set())
@@ -898,6 +988,7 @@ async def reconfigure_openvpn(host: OpnsenseHost) -> dict:
section comment above for why this is NOT assumed symmetric with
IPsec's `{"status": "ok"}`), dispatched via _mutation_result with the
"result"-based branch (results={"ok"})."""
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/openvpn/service/reconfigure"), "OpenVPN-Dienst neu laden", {"ok"})
@@ -1074,6 +1165,7 @@ async def reconfigure_wireguard(host: OpnsenseHost) -> dict:
IPsec's `{"status": "ok"}`, and matches OpenVPN's shape instead),
dispatched via _mutation_result with the "result"-based branch
(results={"ok"})."""
await _before_reload(host)
return _mutation_result(await _request(host, "POST", "/wireguard/service/reconfigure"), "WireGuard-Dienst neu laden", {"ok"})
@@ -1394,6 +1486,7 @@ async def write_network_item(host: OpnsenseHost, kind: str, action: str, uuid: s
async def reconfigure_network(host: OpnsenseHost, kind: str) -> dict:
await _before_reload(host)
return _mutation_result(
await _request(host, "POST", NETWORK_ITEM_APIS[kind]["reconfigure"]), "Änderung anwenden", set())
@@ -1454,6 +1547,13 @@ async def create_snapshot(host: OpnsenseHost, name: str, note: str = "") -> None
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
async def activate_snapshot(host: OpnsenseHost, uuid: str) -> None:
"""Boot environment for every following start (bectl activate, not -t); takes effect at the next reboot."""
data = await _request(host, "POST", f"/core/snapshots/activate/{uuid}")
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
raise OpnsenseAPIError(f"Snapshot aktivieren von Firewall nicht bestätigt: {redact_data(data)}")
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
+23 -10
View File
@@ -196,6 +196,7 @@ async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, sta
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
"window_id": str(plan.window_id) if plan.window_id else None,
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
"gate_probes": list(plan.gate_probes or []),
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
}
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
@@ -381,7 +382,8 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
baselines.update((w.baseline or {}))
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
checks=run.plan_snapshot.get("gate_checks"),
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
probes=run.plan_snapshot.get("gate_probes"))
async def apply(db, run):
if gate["verdict"] == "fail":
@@ -407,21 +409,30 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
return apply
# The core package of each OPNsense release type (community, development, business edition). Switching the type
# installs another core package: live 2026-10-23, community -> devel offered `opnsense-devel` 27.1.a_483 as a NEW
# package while product_latest still said 26.7.7 (the rollout expected 26.7.7 and stopped at the gate).
CORE_PACKAGES = ("opnsense", "opnsense-devel", "opnsense-business")
def target_of(status: dict, action: str) -> str | None:
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
core package's new version comes first; product_latest is only a fallback."""
core package's new version comes first (whichever release type it is, see CORE_PACKAGES, also when it
arrives as a new package); product_latest is only a fallback."""
if action == "major_update":
return status.get("upgrade_major_version") or None
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
if core and core.get("new") and core.get("new") != "N/A":
return str(core["new"])
for row in status.get("upgrade_packages") or []:
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
return str(row["new_version"])
for name in CORE_PACKAGES:
core = packages.get(name) if isinstance(packages.get(name), dict) else None
if core and core.get("new") and core.get("new") != "N/A":
return str(core["new"])
for key, field in (("upgrade_packages", "new_version"), ("new_packages", "version")):
for row in status.get(key) or []:
if isinstance(row, dict) and row.get("name") in CORE_PACKAGES and row.get(field):
return str(row[field])
product = status.get("product") if isinstance(status.get("product"), dict) else {}
return product.get("product_latest") or status.get("product_latest") or None
@@ -537,7 +548,7 @@ async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | Non
versions = {}
for rh_id in offers:
host = usable[rh_id]
baselines[str(host.id)] = await rollout_health.baseline(db, host)
baselines[str(host.id)] = await rollout_health.baseline(db, host, run.plan_snapshot.get("gate_probes"))
versions[rh_id] = baselines[str(host.id)].get("version")
roles = await get_statuses([usable[i] for i in offers])
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
@@ -630,6 +641,7 @@ async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision |
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
gate = await rollout_health.evaluate(
db, done_peers + [host], baselines, target_version=run.target_version,
# No probes here: the peers' probes were checked in their own gate, and this runs on every tick.
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
@@ -718,7 +730,8 @@ async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
checks=run.plan_snapshot.get("gate_checks"),
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
probes=run.plan_snapshot.get("gate_probes"))
async def apply(db, run):
w = wave_row(run, index)
+63 -2
View File
@@ -18,13 +18,19 @@ the configuration like the 6-hourly security check does):
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
update may legitimately change defaults.
* probes — optional per plan (1.32): ping/TCP probes run FROM the firewall (uplink, DNS, an internal
server). A probe that worked in the baseline and fails now fails the gate; one that failed
already before only warns. This is reachability from the firewall, not traffic through it.
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
"""
import asyncio
import uuid
from collections import defaultdict
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -41,6 +47,8 @@ DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": Tru
"compliance": False}
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
MAX_PROBES = 10
PROBE_PING_COUNT = 3
def _r(result: str, code: str, **params) -> dict:
@@ -64,7 +72,55 @@ async def _services(host) -> dict | None:
return None if listing is None else _services_compact(listing)
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
def _probe_key(probe: dict) -> str:
return f"{probe['type']} {probe['target']}" + (f":{probe['port']}" if probe.get("port") else "")
async def run_probes(host: OpnsenseHost, probes: list[dict] | None) -> list[dict]:
"""Every probe from the firewall, one after the other; a probe that cannot run counts as failed. Audited by
diagnostics_service like a probe from the host page, as `system:rollout` when no request context exists."""
from app.services import audit_service, diagnostics_service
if not probes:
return []
token = None
if audit_service.context.get() is None:
token = audit_service.context.set({"actor": "system:rollout", "id": str(uuid.uuid4()), "action": "probe"})
try:
async def one(probe: dict) -> dict:
ipproto = "inet6" if ":" in probe["target"] else "inet"
try:
if probe["type"] == "ping":
result = await diagnostics_service.ping(host, probe["target"], ipproto, PROBE_PING_COUNT, None, None)
ok = int(result.get("received") or 0) > 0
else:
result = await diagnostics_service.portprobe(host, probe["target"], int(probe["port"]), ipproto, None)
ok = bool(result.get("open"))
except (HTTPException, client.OpnsenseAPIError):
ok = False
return {"key": _probe_key(probe), "ok": ok}
# Concurrently (diagnostics_service caps it at a few at a time per process), results in plan order.
return list(await asyncio.gather(*(one(p) for p in probes[:MAX_PROBES])))
finally:
if token is not None:
audit_service.context.reset(token)
def _check_probes(before: list | None, now: list | None) -> dict:
if now is None:
return _r("fail", "unreadable")
was = {p["key"]: p["ok"] for p in (before or [])}
regressed = [p["key"] for p in now if not p["ok"] and was.get(p["key"], True)]
still = [p["key"] for p in now if not p["ok"] and was.get(p["key"]) is False]
if regressed:
return _r("fail", "probe_regressed", probes=", ".join(regressed))
if still:
return _r("warn", "probe_was_failing", probes=", ".join(still))
return _r("ok", "ok", count=len(now))
async def baseline(db: AsyncSession, host: OpnsenseHost, probes: list[dict] | None = None) -> dict:
info, gateways, services = await asyncio.gather(
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
@@ -73,6 +129,7 @@ async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
"gateways": gateways,
"services": services,
"compliance_score": score,
"probes": await run_probes(host, probes) if probes else None,
}
@@ -137,7 +194,7 @@ async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dic
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
checks: dict | None = None, compliance_mode: str = "warn", probes: list[dict] | None = None) -> dict:
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
enabled = {**DEFAULT_CHECKS, **(checks or {})}
for name in ALWAYS_ON:
@@ -190,6 +247,10 @@ async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict,
if enabled["services"]:
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
if probes:
# A probe new since the baseline (plan edited mid-run) has nothing to compare with: treated as "worked before".
row["probes"] = _check_probes(before.get("probes"), await run_probes(host, probes) if reachable else None)
blockers = await _shared_change_blockers(db, fleets[hid])
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
+2
View File
@@ -110,6 +110,8 @@ def definition(plan: RolloutPlan) -> dict:
return {
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
# Only when set: an activated plan from before 1.32 must keep its fingerprint (else its automatic start stops).
**({"gate_probes": list(plan.gate_probes)} if plan.gate_probes else {}),
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
}
+36 -10
View File
@@ -75,6 +75,7 @@ Item and operation ``message`` hold an ErrorCode name (``errors.<CODE>`` in the
from __future__ import annotations
import hashlib
import ipaddress
import json
import logging
import re
@@ -96,6 +97,7 @@ from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeplo
from app.models.rule_cleanup import OPEN_ITEM_STATES, RuleCleanupItem, RuleCleanupOperation
from app.models.rule_history import LiveRuleState, RuleChangeEvent
from app.services import audit_service
from app.services.deploy_flow import unsettled
from app.services import opnsense_client as client
from app.services import rule_history_service as rhs
from app.services import rule_hit_service
@@ -236,7 +238,7 @@ async def _ensure_free(db: AsyncSession, host: OpnsenseHost, fleet: list[Opnsens
await ensure_no_maintenance(db, [host], cleanup=False)
ids = [h.id for h in fleet]
for model in (*DEPLOYMENTS, PolicyHierarchyRuleDeployment):
if await db.scalar(select(model.id).where(model.host_id.in_(ids), model.status.in_(("pending", "revert_failed"))).limit(1)):
if await db.scalar(select(model.id).where(model.host_id.in_(ids), unsettled(model)).limit(1)):
raise api_error(409, ErrorCode.BEREINIGUNG_OFFENE_OPERATION, reason="deployment")
if await has_open_cleanup(db, ids, exclude_op=exclude_op):
raise api_error(409, ErrorCode.BEREINIGUNG_OFFENE_OPERATION, reason="cleanup")
@@ -255,6 +257,7 @@ class Node:
managed: dict[str, dict]
hasync: dict | None
states: dict = field(default_factory=dict)
own_source: str | None = None # where this firewall sees OPNexus's API requests come from (B10)
def _hasync_brief(data) -> dict | None:
@@ -312,12 +315,17 @@ async def _load_nodes(db: AsyncSession, host: OpnsenseHost, *, unused_days: int,
except OpnsenseAPIError:
hasync = None # protection.infrastructure_reason then assumes the worst on a cluster node
states = {s.rule_uuid: s for s in (await db.scalars(select(LiveRuleState).where(LiveRuleState.host_id == member.id))).all()}
try:
own_source = await client.own_source_address(member)
except OpnsenseAPIError as exc:
own_source = None # shown as "unknown" in the preview; the configured sources still protect
logger.info("Bereinigung: OPNexus-Adresse auf %s nicht ermittelbar: %s", member.name, exc)
nodes.append(Node(
host=member, role=roles.get(member.id), ctx=ctx,
rules={r.uuid.lower(): r for r in rules if r.uuid and not r.uuid.startswith("pf@")},
rows={str(r.get("uuid")).lower(): r for r in ctx.rows if r.get("uuid")},
candidates={str(c["uuid"]).lower(): c for c in view["candidates"] if c.get("uuid")},
managed=managed, hasync=hasync, states=states,
managed=managed, hasync=hasync, states=states, own_source=own_source,
))
return _write_order(nodes)
@@ -394,6 +402,18 @@ async def _uuid_map(db: AsyncSession, nodes: list[Node], kind: str, uuids: list[
return mapping
def _with_own_source(sources: list, own: str | None) -> tuple[list, dict]:
"""The configured management sources plus OPNexus's own address on this firewall (B10): a rule that only an
admin's workstation was listed for could otherwise cut OPNexus off, and then the automatic revert cannot reach the
firewall any more. Unknown address: the configured sources alone, and the preview says so."""
if not own:
return sources, {"address": None, "status": "unknown"}
addr = ipaddress.ip_address(own)
if any(addr.version == net.version and addr in net for net in sources):
return sources, {"address": own, "status": "configured"}
return [*sources, ipaddress.ip_network(own)], {"address": own, "status": "added"}
async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[str], unused_days: int,
acknowledge: set[str], limits: Limits, now: datetime) -> dict:
"""What the operation would do, recomputed from the live state, or a refusal (api_error)."""
@@ -408,6 +428,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
missing_ack: list[str] = []
for node in nodes:
targets, ports = protection.management_targets(node.host.base_url, node.ctx, limits.ports)
node_sources, opnexus_source = _with_own_source(sources, node.own_source)
node_uuids = [mapping[u][node.host.id] for u in uuids]
rows = node.rows
covers = {c.get("by", {}).get("uuid", "").lower() for u in node_uuids
@@ -452,7 +473,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
if infra:
refuse(infra)
continue
check = protection.management_check(rule, node.ctx, sources, targets, ports)
check = protection.management_check(rule, node.ctx, node_sources, targets, ports)
item["protection"] = check
if check["result"] != "false":
refuse("management")
@@ -487,7 +508,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
if not eligible:
refuse("not_candidate")
continue
item["protection"] = protection.management_check(rule, node.ctx, sources, targets, ports)
item["protection"] = protection.management_check(rule, node.ctx, node_sources, targets, ports)
# Switching a block/reject rule back ON can cut off the management path; the auto-revert then only
# works if OPNexus itself still reaches the firewall. Unless the rule is proven not to touch
# management traffic: an explicit acknowledgement per rule (review finding 7).
@@ -506,7 +527,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
continue
items.append(item)
plan_nodes.append({"host_id": str(node.host.id), "host_name": node.host.name, "base_url": node.host.base_url,
"role": node.role, "items": items})
"role": node.role, "opnexus_source": opnexus_source, "items": items})
if problems:
first = problems[0]
if first.get("code") == "wait":
@@ -677,7 +698,8 @@ def redact_plan(plan: dict | None) -> dict | None:
return plan
out = {k: v for k, v in plan.items() if k not in ("sources", "ports")}
out["sources_configured"] = bool(plan.get("sources"))
out["nodes"] = [{**{k: v for k, v in n.items() if k != "base_url"},
out["nodes"] = [{**{k: v for k, v in n.items() if k not in ("base_url", "opnexus_source")},
"opnexus_source": {"status": (n.get("opnexus_source") or {}).get("status")},
"items": [{**i, "protection": _brief_protection(i.get("protection"))} for i in n.get("items", [])]}
for n in plan.get("nodes", [])]
return out
@@ -963,7 +985,8 @@ async def _run_switch(db: AsyncSession, op: RuleCleanupOperation, nodes: list[No
# Also after an error: what is switched in the configuration must not wait for a foreign apply.
await _sample(db, node.host, strict=False)
try:
await client.apply(node.host)
with client.counters_already_sampled(): # _sample just ran: no second probe
await client.apply(node.host)
except OpnsenseAPIError as exc:
apply_failed = True
op.message = f"{ErrorCode.BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN.value}: {node.host.name}: {exc}"[:1000]
@@ -1110,7 +1133,8 @@ async def _revert(db: AsyncSession, op: RuleCleanupOperation, *, allowed: tuple,
if touched and member is not None:
await _sample(db, member, strict=False)
try:
await client.apply(member)
with client.counters_already_sampled(): # _sample just ran: no second probe
await client.apply(member)
except OpnsenseAPIError as exc:
for item in node_items:
if item.state == "reverted":
@@ -1322,7 +1346,8 @@ async def _run_delete(db: AsyncSession, op: RuleCleanupOperation, nodes: list[No
# and leaves no "pending changes" in the GUI. It still zeroes the counters: sample first.
await _sample(db, node.host, strict=False)
try:
await client.apply(node.host)
with client.counters_already_sampled(): # _sample just ran: no second probe
await client.apply(node.host)
except OpnsenseAPIError as exc:
op.message = f"{ErrorCode.BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN.value}: {node.host.name}: {exc}"[:1000]
stop = True
@@ -1467,7 +1492,8 @@ async def restore_item(db: AsyncSession, host: OpnsenseHost, item_id, *, actor:
warnings.append(f"{member.name}:{item.rule_uuid}")
await _sample(db, member, strict=False)
try:
await client.apply(member)
with client.counters_already_sampled(): # _sample just ran: no second probe
await client.apply(member)
except OpnsenseAPIError as exc: # the rule is disabled: pf does not change; the GUI shows pending changes
op.message = f"{ErrorCode.BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN.value}: {member.name}: {exc}"[:1000]
item.state, item.changed_at = "restored", _now()
+9
View File
@@ -142,6 +142,15 @@ async def apply_sample(db: AsyncSession, host: OpnsenseHost, counters: dict[str,
return len(rules)
async def sample_before_reload(host: OpnsenseHost) -> None:
"""Called by opnsense_client right before any request that makes the firewall reload pf (B11): read the counters
while they still hold what was hit since the last sample. Own session, so the writer's transaction stays as it is."""
from app.database import SessionLocal
async with SessionLocal() as db:
await sample_host(db, host)
async def restart_window(db: AsyncSession, host_id, rule_uuids, now: datetime | None = None) -> int:
"""The rules were edited, enabled or created: what OPNexus saw before does not describe them any more, so the
observation window for "unused" starts over (an earlier last hit is kept)."""
@@ -40,7 +40,7 @@ from app.models.host_operation import HostOperation
from app.models.scheduled_maintenance import ScheduledMaintenance
from contextlib import contextmanager
from app.services import audit_service, backup_store, opnsense_client as client
from app.services import audit_service, backup_store, opnsense_client as client, snapshot_service
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
from app.services.host_status_service import get_statuses
@@ -202,6 +202,17 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
await db.commit()
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
return
if sched.action in _BACKUP_BEFORE:
# Way back (1.32, B5): a boot environment of the running system, after every other check passed.
# Fail closed like the backup: no rollback point on a ZFS firewall, no update.
try:
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
except client.OpnsenseAPIError as exc:
sched.status = "failed"
sched.result = {"error": f"Rückweg-Punkt (Boot-Environment) vor dem Update nicht angelegt: {str(exc)[:300]}"}
await db.commit()
logger.warning("Geplante Wartung %s: Rückweg-Punkt fehlgeschlagen", sched.id)
return
boot_before = await _boot_time(host) if sched.auto_verify else None
op = HostOperation(
@@ -302,6 +313,9 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
backup_store.save_snapshot(host.id, await client.download_backup(host))
state = await inspect_host(db, host, skip_role_checks=True)
blocked = list(dict.fromkeys(blocked + state["actions"][sched.action]))
if not blocked and sched.action in _BACKUP_BEFORE:
# Way back (1.32, B5) while nothing has been toggled yet; a failure ends here like a failed backup.
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
except client.OpnsenseAPIError as exc:
await fail(f"Vorbereitung der CARP-Übergabe fehlgeschlagen, nichts wurde verändert: {str(exc)[:300]}")
return
+115 -9
View File
@@ -1,17 +1,22 @@
"""ZFS snapshots (boot environments) of one firewall: list, create, delete.
Snapshots are a safety net before bigger changes, not a rollback button:
activating one needs a reboot and discards everything since, so that action is
deliberately NOT offered here. Deleting refuses the active/next-boot snapshot
Snapshots are a safety net before bigger changes. Activating an arbitrary one is
deliberately NOT offered (it needs a reboot and discards everything since); the
only way back is the guided firmware rollback to a point OPNexus itself created
right before an update (see "Rollback points" below, maintenance action
`firmware_rollback`). Deleting refuses the active/next-boot snapshot
(the firewall refuses too; we say so up front). Every step is audited.
"""
import logging
import re
from datetime import datetime, timezone
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.services import audit_service, opnsense_client
from app.services.opnsense_client import OpnsenseAPIError
logger = logging.getLogger(__name__)
NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$")
@@ -23,20 +28,121 @@ def _row(row: dict) -> dict:
}
async def list_snapshots(host: OpnsenseHost) -> dict:
async def supported(host: OpnsenseHost) -> bool:
try:
supported = (await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported")
return bool((await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported"))
except OpnsenseAPIError as exc:
if "404" in str(exc):
return {"supported": False, "rows": []}
if "404" in str(exc): # OPNsense without the snapshots API
return False
raise
if not supported:
async def list_snapshots(host: OpnsenseHost) -> dict:
if not await supported(host):
return {"supported": False, "rows": []}
return {"supported": True, "rows": [_row(r) for r in await opnsense_client.list_snapshots(host)]}
# ── Rollback points (1.32, B5) ──────────────────────────────────────────────
# Before every firmware update/upgrade OPNexus creates a boot environment of the running system. Going back to it
# (activate + reboot) restores the old firmware AND the old configuration of that moment: everything changed on the
# firewall since is gone. Only these points are offered for going back, never boot environments made by hand.
ROLLBACK_PREFIX = "opnexus-pre-"
KEEP_ROLLBACK_POINTS = 2
_POINT = re.compile(r"^opnexus-pre-(?P<version>[A-Za-z0-9._-]+)-(?P<stamp>\d{8}-\d{6})$")
def _is_active(row: dict) -> bool:
# N = active now, R = active after reboot.
return str(row.get("active", "")).strip() not in ("", "-")
def rollback_points(rows: list[dict]) -> list[dict]:
"""OPNexus rollback points, newest first: {uuid, name, version, stamp, active, created}."""
out = []
for row in rows:
match = _POINT.match(str(row.get("name", "")))
if match:
out.append({"uuid": str(row.get("uuid", "")), "name": match.group(0), "version": match.group("version"),
"stamp": match.group("stamp"), "active": str(row.get("active", "")).strip(),
"created": str(row.get("created_str", ""))})
return sorted(out, key=lambda p: p["stamp"], reverse=True)
async def create_rollback_point(host: OpnsenseHost, version: str | None) -> dict | None:
"""Boot environment right before a firmware action. None when the firewall has no ZFS snapshots (UFS install):
the update may still run, without a way back. Raises OpnsenseAPIError when creating it fails (callers abort the
update: fail closed). Older OPNexus points beyond KEEP_ROLLBACK_POINTS are removed afterwards."""
if not await supported(host):
return None
clean_version = re.sub(r"[^A-Za-z0-9._-]", "-", version or "unbekannt")[:30] or "unbekannt"
name = f"{ROLLBACK_PREFIX}{clean_version}-{datetime.now(timezone.utc):%Y%m%d-%H%M%S}"
audit = {"kind": "snapshot", "action": "snapshot.rollback_point", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
await audit_service.persist(**audit, status="started", message=None, details={"version": version})
try:
await opnsense_client.create_snapshot(host, name, f"OPNexus: Rückweg-Punkt vor dem Firmware-Update von {version}")
rows = await opnsense_client.list_snapshots(host)
except OpnsenseAPIError as exc:
await audit_service.persist(**audit, status="rejected", message=str(exc)[:500], details={})
raise
point = next((p for p in rollback_points(rows) if p["name"] == name), None)
if point is None:
await audit_service.persist(**audit, status="unverified", message=None, details={})
raise OpnsenseAPIError(f"Rückweg-Punkt {name} nach dem Anlegen nicht in der Liste")
await audit_service.persist(**audit, status="applied", message=None, details={})
for old in [p for p in rollback_points(rows) if not _is_active(p)][KEEP_ROLLBACK_POINTS:]:
try: # housekeeping only: a point that cannot be removed stays, the update goes on
await opnsense_client.delete_snapshot(host, old["uuid"])
await audit_service.persist(kind="snapshot", action="snapshot.del", entity_name=old["name"], host_id=str(host.id),
host_name=host.name, status="applied", message="Älterer Rückweg-Punkt aufgeräumt", details={})
except OpnsenseAPIError as exc:
logger.warning("Rückweg-Punkt %s auf %s nicht gelöscht: %s", old["name"], host.name, exc)
return {"uuid": point["uuid"], "name": point["name"], "version": version}
async def rollback_state(host: OpnsenseHost) -> dict:
"""What going back would do: the newest OPNexus point that is not running now, and the boot environment that is."""
if not await supported(host):
return {"supported": False, "target": None, "current": None}
rows = await opnsense_client.list_snapshots(host)
current = next((r for r in rows if "N" in str(r.get("active", ""))), None) # running now (R alone = next start)
target = next((p for p in rollback_points(rows) if "N" not in p["active"]), None)
return {"supported": True, "target": target,
"current": {"uuid": str(current.get("uuid", "")), "name": str(current.get("name", ""))} if current else None}
async def activate_and_reboot(host: OpnsenseHost, rollback: dict) -> dict:
"""Activate the rollback point, check it is marked for the next start, reboot. If the reboot cannot be handed over,
the previous boot environment is activated again: an activated old one must never wait for the next unplanned
reboot."""
target, current = rollback["target"], rollback["current"]
async def undo() -> None:
if current:
try:
await opnsense_client.activate_snapshot(host, current["uuid"])
except OpnsenseAPIError:
logger.warning("Rückweg auf %s: vorheriges Boot-Environment nicht reaktiviert", host.name)
await opnsense_client.activate_snapshot(host, target["uuid"])
try:
rows = await opnsense_client.list_snapshots(host)
except OpnsenseAPIError:
await undo() # cannot confirm the mark: never leave it waiting for an unplanned reboot
raise
marked = next((r for r in rows if str(r.get("uuid", "")) == target["uuid"]), None)
if marked is None or "R" not in str(marked.get("active", "")):
await undo()
raise OpnsenseAPIError(f"Boot-Environment {target['name']} ist nicht für den nächsten Start vorgemerkt")
try:
return await opnsense_client.reboot_host(host)
except OpnsenseAPIError:
await undo()
raise
async def create(host: OpnsenseHost, name: str, note: str) -> dict:
if not NAME.match(name):
if not NAME.match(name) or name.startswith(ROLLBACK_PREFIX): # that prefix marks OPNexus' own rollback points
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field="name")
audit = {"kind": "snapshot", "action": "snapshot.add", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
await audit_service.persist(**audit, status="started", message=None, details={"note": note[:200]})
@@ -211,6 +211,8 @@ from app.models.wireguard_tunnel import WireguardTunnel
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
from app.services import opnsense_client, wireguard_key_store
from app.services.opnsense_client import OpnsenseAPIError
from app.services.deploy_flow import checkpoint
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
def _now() -> datetime:
@@ -274,7 +276,7 @@ def build_payloads(tunnel: WireguardTunnel) -> dict:
}
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: WireguardTunnelDeployment) -> None:
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: WireguardTunnelDeployment) -> None:
"""Writes the Server then the Peer, in that dependency order (the
Peer's payload needs the Server's UUID for the `servers` field — see
module docstring), persisting each UUID onto `deployment` as soon as
@@ -288,6 +290,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: Wireguar
deployment.opnsense_server_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_server_uuid:
raise OpnsenseAPIError("Firewall hat keine Server-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
# Overlay the deploy-time-known Server UUID onto a shallow copy, never
# mutating payloads["peer"] itself — the same reasoning ipsec_deploy_
@@ -303,6 +306,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: Wireguar
deployment.opnsense_peer_uuid = result.get("uuid") if isinstance(result, dict) else None
if not deployment.opnsense_peer_uuid:
raise OpnsenseAPIError("Firewall hat keine Peer-UUID zurückgegeben; Stand manuell prüfen")
await checkpoint(db) # the UUID is durable before the next remote call
async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: OpnsenseHost) -> WireguardTunnelDeployment:
@@ -311,7 +315,7 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
existing = await db.scalar(
select(WireguardTunnelDeployment).where(
WireguardTunnelDeployment.tunnel_id == tunnel.id, WireguardTunnelDeployment.host_id == host.id
)
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
)
if existing and existing.status == "success" and existing.last_payload == comparable:
# Nothing actually changed on this host — re-affirm without
@@ -322,8 +326,15 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
deployment = existing or WireguardTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
if not existing:
db.add(deployment)
await checkpoint(db)
try:
await _write_config(host, payloads, deployment)
await _write_config(db, host, payloads, deployment)
# Without the global switch the config is stored but no interface
# ever comes up (live-verified), so a "success" would be a lie.
await opnsense_client.ensure_wireguard_enabled(host)
@@ -331,8 +342,6 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
deployment.status = "error"
deployment.message = str(exc)
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
try:
@@ -347,28 +356,30 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
deployment.last_payload = comparable
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
deployment.status = "success"
deployment.message = None
deployment.last_payload = comparable
deployment.deployed_at = _now()
if not existing:
db.add(deployment)
return deployment
async def deploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel, hosts: list[OpnsenseHost]) -> list[WireguardTunnelDeployment]:
from app.services.host_maintenance_service import ensure_no_maintenance
await ensure_no_maintenance(db, hosts)
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
await db.commit()
async with resource_lease("wireguard", tunnel.id):
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
await db.commit()
return deployments
async def undeploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel) -> None:
async with resource_lease("wireguard", tunnel.id):
await _undeploy_tunnel(db, tunnel)
async def _undeploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel) -> None:
"""Explicit action, not gated by anything (there is no lockout state
for this entity to gate on) — deletes the Peer before the Server (the
dependency direction; deleting the Peer first also auto-unlinks it
+2 -2
View File
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
Versioning scheme: see CHANGELOG.md ("Versionierung").
"""
APP_VERSION = "1.31.0"
APP_CODENAME = "Vulture"
APP_VERSION = "1.33.0"
APP_CODENAME = "Saker"
@@ -0,0 +1,45 @@
"""Stabile Schlüssel (config_key) für Regeln und NAT-Objekte (1.32, Konfiguration als Code).
Revision ID: 0030
Revises: 0029
Create Date: 2026-10-23
"""
from alembic import op
revision = "0030"
down_revision = "0029"
branch_labels = None
depends_on = None
_TABLES = (
("firewall_rules", "r"),
("port_forward_rules", "pf"),
("one_to_one_rules", "nat1"),
("outbound_nat_rules", "out"),
("policy_hierarchy_rules", "ph"),
)
def upgrade() -> None:
# IF NOT EXISTS / DO-Block: ein Altbestand ohne Alembic wird zuerst per create_all() angehoben und hat Spalte samt
# Constraint dann schon.
for table, prefix in _TABLES:
op.execute(f"ALTER TABLE {table} ADD COLUMN IF NOT EXISTS config_key varchar(64)")
op.execute(
f"UPDATE {table} SET config_key = '{prefix}-' || substr(replace(id::text, '-', ''), 1, 12) "
"WHERE config_key IS NULL"
)
op.execute(f"ALTER TABLE {table} ALTER COLUMN config_key SET NOT NULL")
op.execute(
f"""DO $$ BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '{table}_config_key_key') THEN
ALTER TABLE {table} ADD CONSTRAINT {table}_config_key_key UNIQUE (config_key);
END IF;
END $$"""
)
def downgrade() -> None:
for table, _prefix in _TABLES:
op.execute(f"ALTER TABLE {table} DROP CONSTRAINT IF EXISTS {table}_config_key_key")
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS config_key")
@@ -0,0 +1,23 @@
"""Rollout-Gate: Erreichbarkeitsproben von der Firewall aus (1.32, B5).
Revision ID: 0031
Revises: 0030
Create Date: 2026-10-23
"""
from alembic import op
revision = "0031"
down_revision = "0030"
branch_labels = None
depends_on = None
def upgrade() -> None:
# IF NOT EXISTS: a pre-Alembic install is bridged with create_all() first, which may already have the column.
op.execute("ALTER TABLE rollout_plans ADD COLUMN IF NOT EXISTS gate_probes json")
op.execute("UPDATE rollout_plans SET gate_probes = '[]'::json WHERE gate_probes IS NULL")
op.execute("ALTER TABLE rollout_plans ALTER COLUMN gate_probes SET NOT NULL")
def downgrade() -> None:
op.execute("ALTER TABLE rollout_plans DROP COLUMN IF EXISTS gate_probes")
+2 -1
View File
@@ -12,5 +12,6 @@ pytest-asyncio==1.4.0
respx==0.23.1
cryptography==50.0.2
alembic==1.20.0
PyJWT==2.10.1
PyJWT==2.15.1
tzdata==2025.2
PyYAML==6.0.3
+53
View File
@@ -0,0 +1,53 @@
"""Seed data for the configuration-as-code tests: every object kind, hierarchy, placeholders, variables."""
from app.database import SessionLocal
from app.models.alias import FirewallAlias
from app.models.host import OpnsenseHost
from app.models.host_variable import HostVariable
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rule import FirewallRule
async def seed_everything() -> dict:
"""Two hosts, a device group, a hierarchy (Organisation > Nord), and one object of every kind."""
async with SessionLocal() as db:
fw1 = OpnsenseHost(name="fw-filiale-1", base_url="https://secret-host.example:8443", api_key="AKIA-SECRET-KEY")
fw2 = OpnsenseHost(name="fw-filiale-2", base_url="https://other.example", api_key="k2")
group = DeviceGroup(name="Filialen", description="alle Filialen", hosts=[fw1, fw2])
org = PolicyHierarchyGroup(name="Organisation", description="")
db.add_all([fw1, fw2, group, org])
await db.flush()
nord = PolicyHierarchyGroup(name="Nord", description="Region Nord", parent_id=org.id)
db.add(nord)
await db.flush()
fw1.policy_hierarchy_group_id = nord.id
db.add_all([
HostVariable(host_id=fw1.id, name="LAN_NET", value="10.20.30.0/24"),
HostVariable(host_id=fw1.id, name="SITE", value="Filiale Nord"),
FirewallAlias(name="WEB_SERVERS", type="host", description="DMZ", enabled=True,
content="10.20.30.10\n203.0.113.20", groups=[group], policy_hierarchy_group_id=None),
FirewallRule(config_key="allow-dns-lan", description="DNS aus {{SITE}}", interface="lan", direction="in",
source_net="{{LAN_NET}}", destination_net="WEB_SERVERS", destination_port="53", protocol="TCP/UDP",
groups=[group], targets=[]),
FirewallRule(config_key="block-all", description="alles sperren", interface="wan", direction="in", action="block",
log=True, targets=[fw2], groups=[]),
PortForwardRule(config_key="pf-web", description="Web DMZ", interface="wan", destination_net="wanip",
destination_port="443", target="10.20.30.10", local_port="443", targets=[fw1], groups=[]),
OneToOneRule(config_key="nat1-mail", description="Mail", interface="wan", external="203.0.113.25",
source_net="10.20.30.25", destination_net="any", groups=[group], targets=[]),
OutboundNatRule(config_key="out-lan", description="LAN raus", interface="wan", protocol="any",
source_net="{{LAN_NET}}", destination_net="any", target="wanip", groups=[group], targets=[],
policy_hierarchy_group_id=None),
PolicyHierarchyRule(config_key="ph-block-bogons", hierarchy_group_id=org.id, rulebase="pre", sequence=0,
description="Bogons", interface="wan", direction="in", action="block", source_net="bogons"),
PolicyHierarchyRule(config_key="ph-allow-mgmt", hierarchy_group_id=org.id, rulebase="pre", sequence=1,
description="Mgmt", interface="lan", direction="in", action="pass"),
PolicyHierarchyRule(config_key="ph-post-deny", hierarchy_group_id=nord.id, rulebase="post", sequence=0,
description="Standard sperren", interface="lan", direction="in", action="block"),
])
await db.commit()
return {"fw1": "fw-filiale-1", "fw2": "fw-filiale-2"}
+13
View File
@@ -7,3 +7,16 @@ async def deploy_approved(client, deploy_url: str):
preview = await client.get(deploy_url.rsplit("/deploy", 1)[0] + "/preview")
assert preview.status_code == 200, preview.text
return await client.post(deploy_url, json={"preview_token": preview.json()["token"]})
async def delete_host(client, host_id):
"""DELETE /api/hosts/{id} with the confirmation the API now requires (T4): the host's own name, read from the DB
so the helper does not touch the firewall mocks."""
import uuid as _uuid
from app.database import SessionLocal
from app.models.host import OpnsenseHost
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, _uuid.UUID(str(host_id)))
return await client.delete(f"/api/hosts/{host_id}", params={"confirm_name": host.name if host else ""})
+6
View File
@@ -161,6 +161,12 @@ class FakeFirewall:
return httpx.Response(200, json={"rows": [{"name": "tbl", "type": "urltable", "content": "https://203.0.113.9/list"}]})
if method == "GET" and path == "/firewall/d_nat/searchRule":
return httpx.Response(200, json={"rows": []})
# B10: OPNexus asks for its own source address. respx has no real socket, so the state cannot be matched to
# this connection: the address stays unknown unless a test sets ``own_source`` (then the fake answers it).
if method == "GET" and path == "/diagnostics/firewall/pf_states":
return httpx.Response(200, json={"current": 5, "limit": "100000"})
if method == "POST" and path == "/diagnostics/firewall/query_states":
return httpx.Response(200, json={"rows": [], "rowCount": 0, "total": 0, "current": 1})
if method == "GET" and path == "/core/hasync/get":
sel = {"": {"value": "Disabled", "selected": 1 if not self.pfsync else 0},
"lan": {"value": "LAN", "selected": 1 if self.pfsync == "lan" else 0}}
+27
View File
@@ -326,3 +326,30 @@ async def test_poll_certificate_status_updates_remote_fields(client, respx_mock)
assert body["remote_status_code"] == 200
assert body["remote_status_last_update"] is not None
assert body["remote_last_update"] is not None
@pytest.mark.asyncio
async def test_acme_target_host_is_fixed_once_deployed_or_used_by_requests(client, respx_mock):
"""Audit AUD-03: the same re-pointing problem for ACME accounts and certificate requests."""
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
a = await _create_host(client, "acme-a", "https://acme-a.test")
b = await _create_host(client, "acme-b", "https://acme-b.test")
acct = (await client.post("/api/acme/accounts", json=_account_payload(a["id"]))).json()
free = await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(b["id"], expected_updated_at=acct["updated_at"]))
assert free.status_code == 200 and free.json()["host_id"] == b["id"] # draft, no requests: free to move
acct = (await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(a["id"], expected_updated_at=free.json()["updated_at"]))).json()
req = (await client.post("/api/acme/certificates", json=_cert_request_payload(a["id"], acct["id"]))).json()
used = await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(b["id"], expected_updated_at=acct["updated_at"]))
assert used.status_code == 409 and used.json()["detail"]["code"] == "ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET"
respx_mock.post(url__regex=r".*/acmeclient/accounts/add$").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "acct-uuid-1"}))
pushed_account = (await client.post(f"/api/acme/accounts/{acct['id']}/push")).json()
fixed = await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(b["id"], expected_updated_at=pushed_account["updated_at"]))
assert fixed.status_code == 409 and fixed.json()["detail"]["code"] == "ZIEL_HOST_NACH_AUSROLLEN_FEST"
respx_mock.post(url__regex=r".*/acmeclient/certificates/add$").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "req-uuid-1"}))
pushed_request = (await client.post(f"/api/acme/certificates/{req['id']}/push")).json()
assert pushed_request["status"] == "success"
moved = await client.put(f"/api/acme/certificates/{req['id']}", json=_cert_request_payload(b["id"], acct["id"], expected_updated_at=pushed_request["updated_at"]))
assert moved.status_code == 409 and moved.json()["detail"]["code"] == "ZIEL_HOST_NACH_AUSROLLEN_FEST"
@@ -18,6 +18,7 @@ from tests.test_monitoring_alerts import (
)
from tests.test_notifications import _create_host
from tests.test_resource_monitor import _mock_resource_endpoints, _mock_vip
from tests.helpers import delete_host
async def _host_row(host_id: str) -> OpnsenseHost:
@@ -413,5 +414,5 @@ async def test_deleting_a_host_removes_its_gateway_mutes(client):
gone = await _create_host(client, "fw-gone", "https://fwgone.test")
for h in (keep, gone):
assert (await client.put(f"/api/hosts/{h['id']}/gateways/WAN_GW/alert", json={"enabled": False})).status_code == 200
assert (await client.delete(f"/api/hosts/{gone['id']}")).status_code == 204
assert (await delete_host(client, gone['id'])).status_code == 204
assert (await client.get("/api/notifications/settings")).json()["gateway_alert_muted"] == [f"{keep['id']}:WAN_GW"]
+2 -1
View File
@@ -7,6 +7,7 @@ from app.database import SessionLocal
from app.models.audit import AuditEvent
from tests.test_rules import _create_host
from tests.test_port_forward import _pf_payload, _mock_pf_push
from tests.helpers import delete_host
@pytest.mark.asyncio
async def test_audit_retains_payload_actor_after_rule_and_host_deletion(client,respx_mock):
@@ -18,7 +19,7 @@ async def test_audit_retains_payload_actor_after_rule_and_host_deletion(client,r
assert (await client.post(f"/api/port-forward/{r['id']}/confirm")).status_code==200
respx_mock.post('https://fw1.test/api/firewall/d_nat/delRule/pf-uuid-1').respond(200,json={'result':'deleted'})
assert (await client.delete(f"/api/port-forward/{r['id']}")).status_code==204
assert (await client.delete(f"/api/hosts/{h['id']}")).status_code==204
assert (await delete_host(client, h['id'])).status_code==204
data=(await client.get('/api/audit',params={'entity_id':r['id'],'limit':200})).json()
state=[e for e in data['items'] if e['host_id']==h['id']]
assert {e['status'] for e in state} == {'pending','success','removed'}
+32
View File
@@ -166,3 +166,35 @@ async def test_delete_in_use_certificate_returns_stable_error_code(client, respx
assert detail["code"] == "UPSTREAM_REJECTED"
assert "Item in use by" in detail["params"]["reason"]
assert "roadwarrior" in detail["params"]["reason"]
async def test_a_ca_cannot_be_requested_through_the_managed_certificate_api(client, respx_mock):
"""T3 (1.31.0 test run): cert_type=v3_ca created a certificate with the CA extension, not a usable CA."""
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = await _create_host(client, "fw-ca", "https://fw-ca.test")
resp = await client.post("/api/certificates/managed", json=_cert_payload(host["id"], cert_type="v3_ca"))
assert resp.status_code == 422
for allowed in ("usr_cert", "server_cert", "combined_server_client"):
assert (await client.post("/api/certificates/managed", json=_cert_payload(host["id"], cert_type=allowed))).status_code == 201
@pytest.mark.asyncio
async def test_target_host_of_a_deployed_certificate_is_fixed_but_free_before_the_first_push(client, respx_mock):
"""Audit AUD-03: re-pointing a deployed certificate reported success for a host that never received it."""
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
a = await _create_host(client, "fix-a", "https://fix-a.test")
b = await _create_host(client, "fix-b", "https://fix-b.test")
cert = (await client.post("/api/certificates/managed", json=_cert_payload(a["id"]))).json()
moved = await client.put(f"/api/certificates/managed/{cert['id']}", json=_cert_payload(b["id"], expected_updated_at=cert["updated_at"]))
assert moved.status_code == 200 and moved.json()["host_id"] == b["id"] # never pushed: still free to move
add = respx_mock.post("https://fix-b.test/api/trust/cert/add").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "uuid-on-b"}))
pushed = (await client.post(f"/api/certificates/managed/{cert['id']}/push")).json()
assert pushed["status"] == "success" and add.call_count == 1
calls = len(respx_mock.calls)
refused = await client.put(f"/api/certificates/managed/{cert['id']}", json=_cert_payload(a["id"], expected_updated_at=pushed["updated_at"]))
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ZIEL_HOST_NACH_AUSROLLEN_FEST"
assert len(respx_mock.calls) == calls # nothing was sent anywhere
same_host = await client.put(f"/api/certificates/managed/{cert['id']}", json=_cert_payload(b["id"], description="renamed", expected_updated_at=pushed["updated_at"]))
assert same_host.status_code == 200 # everything else stays editable
+176
View File
@@ -0,0 +1,176 @@
"""HTTP API of configuration as code (1.32 P7): rights matrix, audit, errors, limits."""
import pytest
import yaml
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from app.config import settings
from app.database import SessionLocal
from app.main import app
from app.models.audit import AuditEvent
from app.models.rule import FirewallRule
from app.services import api_token_service
from tests.config_code_seed import seed_everything
PW = "config-code-password-1"
async def _login(username: str, password: str) -> AsyncClient:
ac = AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
resp = await ac.post("/api/auth/login", json={"username": username, "password": password})
assert resp.status_code == 200, resp.text
return ac
def _bearer(token: str) -> AsyncClient:
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test", headers={"Authorization": f"Bearer {token}"})
async def _token(client, scope: str) -> str:
resp = await client.post("/api/tokens", json={"name": f"t-{scope}", "scope": scope, "current_password": settings.admin_password})
assert resp.status_code == 201, resp.text
return resp.json()["token"]
async def _changed_file(client) -> str:
text = (await client.get("/api/config/export")).text
data = yaml.safe_load(text)
data["rules"][0]["description"] = "über die API"
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
@pytest.mark.parametrize("scope,method,path,denied", [
("read", "GET", "/api/config/export", False),
("read", "POST", "/api/config/import/plan", False),
("read", "POST", "/api/config/import/apply", True),
("deploy", "POST", "/api/config/import/plan", False),
("deploy", "POST", "/api/config/import/apply", True),
("admin", "POST", "/api/config/import/apply", False),
# the read-only exception is exactly this path, not its neighbours
("read", "POST", "/api/config/import/plan/x", True),
("read", "POST", "/api/config/import", True),
])
def test_token_scope_matrix(scope, method, path, denied):
assert api_token_service.denies(scope, method, path) is denied
def test_operators_may_plan_but_not_apply():
assert not api_token_service.operator_denies("POST", "/api/config/import/plan")
assert api_token_service.operator_denies("POST", "/api/config/import/apply")
async def test_export_plan_apply_as_admin_with_audit(client):
await seed_everything()
resp = await client.get("/api/config/export")
assert resp.status_code == 200 and resp.headers["content-type"].startswith("text/yaml")
assert "attachment" in resp.headers["content-disposition"] and resp.headers["cache-control"] == "no-store"
text = await _changed_file(client)
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
assert plan["summary"]["update"] == 1 and plan["applicable"] and len(plan["token"]) == 64
assert [i["key"] for i in plan["items"]] == ["allow-dns-lan"] # unchanged objects are not listed
resp = await client.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan["token"]})
assert resp.status_code == 200 and resp.json()["summary"]["update"] == 1
async with SessionLocal() as db:
events = (await db.execute(select(AuditEvent))).scalars().all()
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
assert rule.description == "über die API"
requests = [e for e in events if e.kind == "config" and e.action == "apply"]
assert [e.status for e in requests] == ["requested", "accepted"] and requests[0].entity_id is None
assert not any(e.action == "plan" for e in events) # planning is like a preview: not audited
assert all("über die API" not in str(e.details) for e in events if e.kind == "config")
assert any(e.kind == "rule" and e.action == "import_update" and e.actor == settings.admin_username for e in events)
async def test_stale_blocked_invalid_and_too_large(client):
await seed_everything()
text = await _changed_file(client)
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
resp = await client.post("/api/config/import/apply", json={"yaml": text + "\n# anders\n", "plan_token": "0" * 64})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_PLAN_STALE"
resp = await client.post("/api/config/import/plan", json={"yaml": "a: &x 1\n"})
detail = resp.json()["detail"]
assert resp.status_code == 422 and detail["code"] == "CONFIG_INVALID" and detail["params"]["errors"][0]["code"] == "yaml_anchor_forbidden"
resp = await client.post("/api/config/import/plan", json={"yaml": "#" * (1024 * 1024 + 1)})
assert resp.status_code == 413 and resp.json()["detail"]["code"] == "CONFIG_TOO_LARGE"
resp = await client.post("/api/config/import/plan", content=b"x" * (4 * 1024 * 1024 + 1), headers={"content-type": "application/json"})
assert resp.status_code == 413
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 9\n"})
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "CONFIG_UNSUPPORTED_VERSION"
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n", "extra": 1})
assert resp.status_code == 422 and resp.json()["detail"]["params"]["errors"][0]["code"] == "bad_request"
resp = await client.get("/api/config/export?kinds=rules,secrets")
assert resp.status_code == 422
blocked = "opnexus_config: 1\naliases: []\n"
plan = (await client.post("/api/config/import/plan", json={"yaml": blocked, "prune": True})).json()
assert not plan["applicable"] and plan["items"][0]["action"] == "blocked"
resp = await client.post("/api/config/import/apply", json={"yaml": blocked, "prune": True, "plan_token": plan["token"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_BLOCKED"
async def test_read_only_account_can_export_and_plan_but_not_apply(client):
await seed_everything()
assert (await client.post("/api/users", json={"username": "cfg-reader", "password": PW, "role": "read_only"})).status_code == 201
text = await _changed_file(client)
reader = await _login("cfg-reader", PW)
try:
assert (await reader.get("/api/config/export")).status_code == 200
plan = await reader.post("/api/config/import/plan", json={"yaml": text})
assert plan.status_code == 200
resp = await reader.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "READ_ONLY_ACCESS"
finally:
await reader.aclose()
async def test_operator_can_plan_but_not_apply(client):
await seed_everything()
assert (await client.post("/api/users", json={"username": "cfg-operator", "password": PW, "role": "operator"})).status_code == 201
text = await _changed_file(client)
op = await _login("cfg-operator", PW)
try:
plan = await op.post("/api/config/import/plan", json={"yaml": text})
assert plan.status_code == 200
resp = await op.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "ADMIN_REQUIRED"
finally:
await op.aclose()
async with SessionLocal() as db:
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "config"))).scalars().all()
assert [e.status for e in events] == ["denied"]
@pytest.mark.parametrize("scope,plan_status,apply_status", [("read", 200, 403), ("deploy", 200, 403), ("admin", 200, 200)])
async def test_tokens(client, scope, plan_status, apply_status):
await seed_everything()
text = await _changed_file(client)
async with _bearer(await _token(client, scope)) as api:
assert (await api.get("/api/config/export")).status_code == 200
plan = await api.post("/api/config/import/plan", json={"yaml": text})
assert plan.status_code == plan_status
resp = await api.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
assert resp.status_code == apply_status, resp.text
async def test_group_changes_are_audited_now(client):
resp = await client.post("/api/groups", json={"name": "Neu", "description": "", "host_ids": []})
assert resp.status_code == 201
async with SessionLocal() as db:
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "group"))).scalars().all()
assert [e.status for e in events] == ["requested", "accepted"]
async def test_too_many_waiting_plans_are_turned_away(client, monkeypatch):
from app.routers import config_code
monkeypatch.setattr(config_code, "_waiting", config_code.MAX_WAITING)
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})
assert resp.status_code == 429 and resp.json()["detail"]["code"] == "CONFIG_BUSY"
monkeypatch.setattr(config_code, "_waiting", 0)
assert (await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})).status_code == 200
async def test_a_wrong_shaped_section_is_answered_with_422_not_a_server_error(client):
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\naliases: 42\n"})
assert resp.status_code == 422, resp.text
detail = resp.json()["detail"]
assert detail["code"] == "CONFIG_INVALID" and detail["params"]["errors"]
+434
View File
@@ -0,0 +1,434 @@
"""Planner and apply (1.32 P5/P6): round trip, partial files, prune protection, conflicts, atomicity, no firewall."""
import ast
import copy
from pathlib import Path
import pytest
import respx
import yaml
from sqlalchemy import func, select
from sqlalchemy.orm import selectinload
from app.config_code import apply as applier
from app.config_code import plan as planner
from app.config_code.export import export_yaml
from app.config_code.loader import ConfigError, parse
from app.database import SessionLocal
from app.models.alias import FirewallAlias
from app.models.audit import AuditEvent
from app.models.host import OpnsenseHost
from app.models.host_variable import HostVariable
from app.models.one_to_one import OneToOneRule
from app.models.outbound_nat import OutboundNatRule
from app.models.policy import DeviceGroup, RuleRevision
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
from app.models.port_forward import PortForwardRule
from app.models.rollout import RolloutPlan, RolloutPlanWave
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
from tests.config_code_seed import seed_everything
CENTRAL = (PolicyHierarchyRule, OutboundNatRule, OneToOneRule, PortForwardRule, FirewallRule, FirewallAlias, DeviceGroup)
async def export() -> str:
async with SessionLocal() as db:
return await export_yaml(db)
async def plan(text: str, prune: bool = False) -> planner.Plan:
async with SessionLocal() as db:
return await planner.build_plan(db, parse(text), prune)
async def apply(text: str, prune: bool = False, token: str | None = None) -> planner.Plan:
token = token or (await plan(text, prune)).token
async with SessionLocal() as db:
try:
result = await applier.apply_config(db, parse(text), prune, token)
except Exception:
await db.rollback()
raise
await db.commit()
return result
def edit(text: str, fn) -> str:
data = yaml.safe_load(text)
fn(data)
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
def by_key(p: planner.Plan) -> dict:
return {(i.kind, i.key): i for i in p.items}
async def wipe_central_objects() -> None:
async with SessionLocal() as db:
await db.execute(HostVariable.__table__.delete())
for model in CENTRAL:
await db.execute(model.__table__.delete())
await db.execute(PolicyHierarchyGroup.__table__.update().values(parent_id=None))
await db.execute(OpnsenseHost.__table__.update().values(policy_hierarchy_group_id=None))
await db.execute(PolicyHierarchyGroup.__table__.delete())
await db.commit()
async def payloads() -> dict:
"""What every object would push to every host: the ROADMAP criterion "identical preview" after a round trip."""
from app.services.alias_deploy_service import _build_payload as alias_payload
from app.services.one_to_one_deploy_service import _build_payload as one_payload
from app.services.outbound_nat_deploy_service import _build_payload as out_payload
from app.services.port_forward_deploy_service import _build_payload as pf_payload
from app.services.rule_deploy_service import _build_payload as rule_payload
out = {}
async with SessionLocal() as db:
hosts = (await db.execute(select(OpnsenseHost))).scalars().all()
for model, builder, ident in ((FirewallRule, rule_payload, "config_key"), (PortForwardRule, pf_payload, "config_key"),
(OneToOneRule, one_payload, "config_key"), (OutboundNatRule, out_payload, "config_key"),
(FirewallAlias, alias_payload, "name")):
for row in (await db.execute(select(model))).scalars().all():
for host in hosts:
try:
out[(model.__name__, getattr(row, ident), host.name)] = builder(row, host)
except Exception as exc: # a missing variable is part of the result too
out[(model.__name__, getattr(row, ident), host.name)] = type(exc).__name__
return out
# -- round trip ---------------------------------------------------------------------------------------------------------
async def test_export_then_plan_shows_no_changes_and_apply_is_a_no_op():
await seed_everything()
text = await export()
p = await plan(text)
assert p.applicable and p.summary()["create"] == p.summary()["update"] == p.summary()["delete"] == 0
await apply(text, prune=True)
assert await export() == text
async def test_import_into_an_instance_without_central_objects_restores_everything_identically():
await seed_everything()
text, before = await export(), await payloads()
await wipe_central_objects()
p = await plan(text, prune=True)
assert p.applicable and p.summary()["create"] > 10
await apply(text, prune=True)
assert await export() == text
assert await payloads() == before
assert (await plan(text, prune=True)).summary()["create"] == 0
async def test_apply_writes_revisions_and_audit_events_without_the_file_content():
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="geändert"))
await apply(text)
async with SessionLocal() as db:
notes = (await db.execute(select(RuleRevision.note))).scalars().all()
events = (await db.execute(select(AuditEvent))).scalars().all()
assert "Import (Konfiguration als Code)" in notes and "Vor erster Bearbeitung" in notes
summary = next(e for e in events if e.kind == "config")
assert summary.action == "import" and summary.details["summary"]["update"] == 1
assert any(e.kind == "rule" and e.action == "import_update" and e.details["fields"] == ["description"] for e in events)
assert all("geändert" not in str(e.details) for e in events if e.kind == "config")
# -- partial files and prune ---------------------------------------------------------------------------------------------
async def test_a_missing_section_leaves_the_kind_alone_and_an_empty_one_only_deletes_with_prune():
await seed_everything()
only_aliases = "opnexus_config: 1\naliases: []\n"
p = await plan(only_aliases)
assert p.touched == ["aliases"] and p.summary() == {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
await apply(only_aliases)
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
p = await plan(only_aliases, prune=True)
assert by_key(p)[("aliases", "WEB_SERVERS")].reasons == ["prune_referenced"] # a rule still names it
rules_without_alias = edit(await export(), lambda d: d["rules"][0].update(destination_net="any"))
await apply(rules_without_alias)
await apply(only_aliases, prune=True)
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 0
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
async def test_prune_never_deletes_a_deployed_object_and_nothing_changes():
await seed_everything()
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="reverted", opnsense_rule_uuid="u"))
await db.commit()
text = edit(await export(), lambda d: d.update(rules=[r for r in d["rules"] if r["key"] != "block-all"]))
p = await plan(text, prune=True)
assert by_key(p)[("rules", "block-all")].reasons == ["prune_deployed"] and not p.applicable
with pytest.raises(applier.PlanBlocked):
await apply(text, prune=True, token=p.token)
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
async def test_an_object_with_an_open_deployment_cannot_be_changed():
await seed_everything()
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="pending", opnsense_rule_uuid="u"))
await db.commit()
text = edit(await export(), lambda d: d["rules"][1].update(description="neu"))
item = by_key(await plan(text))[("rules", "block-all")]
assert item.reasons == ["busy_deployment"] and item.deployed_hosts == 1
unchanged = await plan(await export())
assert unchanged.applicable # an unchanged busy object does not block
async def test_pruning_a_group_still_used_by_a_rollout_wave_is_blocked():
await seed_everything()
async with SessionLocal() as db:
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
rollout = RolloutPlan(name="r", action="update", created_by="t")
db.add(rollout)
await db.flush()
wave = RolloutPlanWave(plan_id=rollout.id, position=0, label="w", groups=[group])
db.add(wave)
await db.commit()
data = yaml.safe_load(await export())
for section in ("aliases", "rules", "one_to_one", "outbound_nat"):
for obj in data[section]:
obj["targets"]["groups"] = []
data["device_groups"] = []
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("device_groups", "Filialen")]
assert item.reasons == ["prune_referenced"]
async def test_pruning_a_hierarchy_group_is_blocked_while_hosts_rules_or_children_remain():
await seed_everything()
data = yaml.safe_load(await export())
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Organisation"]
with pytest.raises(ConfigError) as info: # Nord still names Organisation as parent
await plan(yaml.safe_dump(data), prune=True)
assert info.value.errors[0]["code"] == "unknown_hierarchy_group"
data = yaml.safe_load(await export())
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Nord"]
text = yaml.safe_dump(data)
with pytest.raises(ConfigError): # the hierarchy rule ph-post-deny still sits in Nord
await plan(text, prune=True)
data["policy_hierarchy_rules"] = [r for r in data["policy_hierarchy_rules"] if r["group"] != "Nord"]
p = await plan(yaml.safe_dump(data), prune=True)
assert p.applicable # fw-filiale-1 drops out of the hierarchy with its group
await apply(yaml.safe_dump(data), prune=True)
async with SessionLocal() as db:
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
assert host.policy_hierarchy_group_id is None
data = yaml.safe_load(await export())
data.pop("policy_hierarchy_rules")
data["policy_hierarchy_groups"] = []
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("policy_hierarchy_groups", "Organisation")]
assert item.reasons == ["prune_referenced"] # its hierarchy rules stay in the database
# -- validation against the database ------------------------------------------------------------------------------------
@pytest.mark.parametrize("change,code", [
(lambda d: d["rules"][0]["targets"]["hosts"].append("nope"), "unknown_host"),
(lambda d: d["rules"][0]["targets"]["groups"].append("nope"), "unknown_group"),
(lambda d: d["rules"].append(copy.deepcopy(d["rules"][0])), "duplicate_object_key"),
(lambda d: d["rules"][0]["targets"]["groups"].append("Filialen"), "duplicate_entry"),
(lambda d: d["policy_hierarchy_rules"][0].update(description="{{SITE}}"), "placeholder_not_supported"),
(lambda d: d["policy_hierarchy_groups"][0].update(parent="Nord"), "hierarchy_cycle"),
(lambda d: d["policy_hierarchy_groups"][0]["hosts"].append("fw-filiale-1"), "host_in_two_hierarchy_groups"),
(lambda d: d["variables"].update({"nope": {"X": "1"}}), "unknown_host"),
(lambda d: d["aliases"][0]["targets"].update(hierarchy_group="nope"), "unknown_hierarchy_group"),
])
async def test_file_errors_are_reported_with_a_path(change, code):
await seed_everything()
with pytest.raises(ConfigError) as info:
await plan(edit(await export(), change))
assert code in [e["code"] for e in info.value.errors]
assert all(e["path"] for e in info.value.errors)
async def test_warnings_for_unknown_aliases_removed_variables_and_redeploys():
await seed_everything()
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="success", opnsense_rule_uuid="u"))
await db.commit()
def change(d):
d["rules"][0].update(destination_net="MISSING_ALIAS", source_net="{{LAN_NET}}")
d["rules"][1].update(source_net="lan", destination_net="wanip")
d["variables"] = {"fw-filiale-1": {"SITE": "x"}}
p = await plan(edit(await export(), change), prune=True)
codes = {w["code"]: w for w in p.warnings}
assert codes["unknown_alias_reference"]["params"]["value"] == "MISSING_ALIAS"
assert codes["variable_removed_in_use"]["params"]["name"] == "LAN_NET"
assert codes["redeploy_needed"]["params"]["count"] == 1
assert [w for w in p.warnings if w["code"] == "unknown_alias_reference"] == [codes["unknown_alias_reference"]]
# -- hierarchy order -----------------------------------------------------------------------------------------------------
async def test_hierarchy_rule_order_is_the_order_in_the_file():
await seed_everything()
data = yaml.safe_load(await export())
rules = data["policy_hierarchy_rules"]
rules[0], rules[1] = rules[1], rules[0] # both in Organisation/pre
text = yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
p = await plan(text)
changed = {i.key: i.changes for i in p.items if i.action == "update"}
assert set(changed) == {"ph-block-bogons", "ph-allow-mgmt"} and changed["ph-allow-mgmt"][0]["field"] == "position"
await apply(text)
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]] == ["ph-allow-mgmt", "ph-block-bogons", "ph-post-deny"]
assert (await plan(text)).summary()["update"] == 0
async def test_new_hierarchy_rules_go_before_rules_the_file_does_not_name():
await seed_everything()
text = ("opnexus_config: 1\npolicy_hierarchy_rules:\n"
"- {key: ph-new, group: Organisation, rulebase: pre, description: neu, interface: lan}\n")
p = await plan(text)
assert {i.key: i.action for i in p.items} == {"ph-new": "create", "ph-block-bogons": "update", "ph-allow-mgmt": "update"}
assert all(i.implicit for i in p.items if i.key != "ph-new")
await apply(text)
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]][:3] == ["ph-new", "ph-block-bogons", "ph-allow-mgmt"]
# -- variables -----------------------------------------------------------------------------------------------------------
async def test_variables_are_merged_per_name_and_only_removed_with_prune():
await seed_everything()
text = "opnexus_config: 1\nvariables:\n fw-filiale-2:\n LAN_NET: 10.9.0.0/24\n fw-filiale-1:\n SITE: Neu\n"
await apply(text)
data = yaml.safe_load(await export())["variables"]
assert data == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
await apply(text, prune=True)
assert yaml.safe_load(await export())["variables"] == {"fw-filiale-1": {"SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
# -- conflicts, atomicity, self-check -----------------------------------------------------------------------------------
async def test_a_change_between_plan_and_apply_makes_the_plan_stale():
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="aus der Datei"))
token = (await plan(text)).token
async with SessionLocal() as db:
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen").options(selectinload(DeviceGroup.hosts)))
group.hosts = group.hosts[:1] # only a group member: no updated_at anywhere
await db.commit()
with pytest.raises(applier.PlanStale):
await apply(text, token=token)
async with SessionLocal() as db:
db.add(FirewallRule(description="neu in der UI", interface="lan", direction="in"))
await db.commit()
token = (await plan(text)).token
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.description == "neu in der UI"))
rule.description = "noch einmal geändert"
await db.commit()
with pytest.raises(applier.PlanStale):
await apply(text, token=token)
async def test_a_failure_in_the_middle_leaves_the_database_unchanged(monkeypatch):
await seed_everything()
before = await export()
text = edit(before, lambda d: [r.update(description=f"neu {i}") for i, r in enumerate(d["rules"])] and
d["aliases"][0].update(description="neu"))
original = applier._Writer.objects
async def broken(self, kind):
await original(self, kind)
if kind == "rules":
raise RuntimeError("injected")
monkeypatch.setattr(applier._Writer, "objects", broken)
with pytest.raises(RuntimeError):
await apply(text)
assert await export() == before
async def test_the_self_check_catches_a_writer_that_drops_a_field(monkeypatch):
await seed_everything()
before = await export()
text = edit(before, lambda d: d["rules"][0].update(log=True))
monkeypatch.setitem(applier.SCALARS, "rules", tuple(f for f in applier.SCALARS["rules"] if f != "log"))
with pytest.raises(applier.SelfCheckFailed) as info:
await apply(text)
assert info.value.differences == ["rules:allow-dns-lan"]
assert await export() == before
async def test_the_writer_covers_every_field_of_the_schema():
from app.config_code import schema
docs = {"aliases": schema.AliasDoc, "rules": schema.RuleDoc, "port_forwards": schema.PortForwardDoc,
"one_to_one": schema.OneToOneDoc, "outbound_nat": schema.OutboundNatDoc, "policy_hierarchy_rules": schema.HierarchyRuleDoc}
computed = {"key", "targets", "entries", "group"}
for kind, doc in docs.items():
assert set(applier.SCALARS[kind]) == set(doc.model_fields) - computed, kind
# -- never a firewall ----------------------------------------------------------------------------------------------------
async def test_plan_and_apply_never_talk_to_a_firewall():
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="x"))
with respx.mock(assert_all_called=False) as mock: # no routes: any HTTP request fails the test
await apply(text, prune=True)
await wipe_central_objects()
await apply(text, prune=True)
assert mock.calls.call_count == 0
def test_the_config_code_package_does_not_import_a_firewall_client():
forbidden = ("opnsense_client", "_deploy_service", "rollout_engine", "host_maintenance_service")
for path in Path(planner.__file__).parent.glob("*.py"):
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if isinstance(node, (ast.Import, ast.ImportFrom)):
names = [a.name for a in node.names] + ([node.module] if isinstance(node, ast.ImportFrom) and node.module else [])
assert not any(f in n for n in names for f in forbidden), f"{path.name}: {names}"
async def test_two_concurrent_applies_of_the_same_plan_serialize_and_the_second_is_stale():
import asyncio
await seed_everything()
text = edit(await export(), lambda d: d["rules"][0].update(description="parallel"))
token = (await plan(text)).token
results = await asyncio.gather(apply(text, token=token), apply(text, token=token), return_exceptions=True)
assert sorted(type(r).__name__ for r in results) == ["Plan", "PlanStale"]
async def test_variables_merged_beyond_the_per_host_limit_are_refused():
"""The host page allows 200 variables per host; merging a file into existing values must not get past that."""
await seed_everything() # fw-filiale-1 has 2 already
values = "".join(f" V{i}: x\n" for i in range(199))
with pytest.raises(ConfigError) as info:
await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values)
assert [e["code"] for e in info.value.errors] == ["too_many_variables"]
assert (await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values, prune=True)).applicable
async def test_what_the_ui_can_store_survives_the_round_trip_and_padding_is_cleaned_like_the_ui_does():
await seed_everything()
async with SessionLocal() as db:
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
group.description = "zwei\nZeilen " # GroupWrite does not strip descriptions
alias = await db.scalar(select(FirewallAlias).where(FirewallAlias.name == "WEB_SERVERS"))
alias.description = " eingerückt"
await db.commit()
text = await export()
assert (await plan(text)).summary()["update"] == 0
padded = edit(text, lambda d: d["rules"][0].update(description=" DNS aus {{SITE}} ", interface=" lan "))
p = await plan(padded)
assert p.summary()["update"] == 0 # RuleCreate strips these fields, so this is the same rule
with_entries = edit(text, lambda d: d["aliases"][0].update(entries=[" 10.20.30.10 ", "203.0.113.20"]))
assert (await plan(with_entries)).summary()["update"] == 0
+106
View File
@@ -0,0 +1,106 @@
"""Host console of configuration as code (1.32 P8): exit codes, text diff, actor, no control sequences."""
import copy
import json
import pytest
import yaml
from sqlalchemy import select
from app.config_code.__main__ import TOKEN_MARK, format_errors, run
from app.database import SessionLocal
from app.models.audit import AuditEvent
from app.models.rule import FirewallRule
from tests.config_code_seed import seed_everything
async def cli(*argv, stdin: str | bytes | None = None):
data = stdin.encode() if isinstance(stdin, str) else stdin
return await run(list(argv), data)
def token_of(out: str) -> str:
return next(line[len(TOKEN_MARK):] for line in out.splitlines() if line.startswith(TOKEN_MARK))
async def export_text() -> str:
code, out, _ = await cli("export")
assert code == 0
return out
def edit(text: str, fn) -> str:
data = yaml.safe_load(text)
fn(data)
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
async def test_export_plan_without_changes_exits_0_and_a_change_exits_2_with_a_readable_diff():
await seed_everything()
text = await export_text()
code, out, _ = await cli("plan", stdin=text)
assert code == 0 and "Plan: 0 neu, 0 geändert, 0 entfernt" in out and token_of(out)
def change(d):
d["rules"][0]["destination_port"] = 853
d["rules"].append({**copy.deepcopy(d["rules"][0]), "key": "new-rule"}) # no shared objects: YAML would write anchors
changed = edit(text, change)
code, out, _ = await cli("plan", stdin=changed)
assert code == 2
assert '+ rules new-rule (neu)' in out
assert '~ rules allow-dns-lan destination_port: "53" → "853"' in out
assert out.rstrip().splitlines()[-2].startswith("Plan: 1 neu, 1 geändert, 0 entfernt")
async def test_json_output_is_the_api_answer():
await seed_everything()
code, out, _ = await cli("plan", "--json", stdin=await export_text())
body = json.loads(out)
assert code == 0 and body["applicable"] and len(body["token"]) == 64 and body["summary"]["create"] == 0
async def test_apply_needs_the_token_of_the_plan_and_writes_audit_as_the_console():
await seed_everything()
changed = edit(await export_text(), lambda d: d["rules"][0].update(description="per CLI"))
code, _, err = await cli("apply", stdin=changed)
assert code == 1 and "--plan-token" in err
code, out, _ = await cli("plan", stdin=changed)
token = token_of(out)
code, out, err = await cli("apply", "--plan-token", "0" * 64, stdin=changed)
assert code == 4 and "neu planen" in err
code, out, err = await cli("apply", "--plan-token", token, stdin=changed)
assert code == 0 and "Übernommen: 0 neu, 1 geändert, 0 entfernt" in out and "nichts auf Firewalls geschrieben" in out
async with SessionLocal() as db:
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
events = (await db.execute(select(AuditEvent))).scalars().all()
assert rule.description == "per CLI"
assert events and {e.actor for e in events} == {"cli (Host-Konsole)"}
async def test_invalid_files_exit_3_and_name_line_and_path_without_control_characters():
await seed_everything()
bad = "opnexus_config: 1\nrules:\n- key: r-1\n description: \"x\\e[31mred\"\n interface: lan\n nope: 1\n"
code, out, err = await cli("plan", stdin=bad)
assert code == 3 and not out
assert "Zeile 6: rules[0].nope: Unbekanntes Feld (field=nope)" in err and "\x1b" not in err
code, out, _ = await cli("plan", "--json", stdin="a: &x 1\n")
assert code == 3 and json.loads(out)["errors"][0]["code"] == "yaml_anchor_forbidden"
assert "\x1b" not in format_errors([{"code": "x", "path": "p\x1b[2J", "line": 1, "params": {"value": "v\x1b[0m"}}])
async def test_blocked_plans_exit_4_and_apply_refuses_them():
await seed_everything()
only_aliases = "opnexus_config: 1\naliases: []\n"
code, out, _ = await cli("plan", "--prune", stdin=only_aliases)
assert code == 4 and "! aliases WEB_SERVERS blockiert (entfernen): wird noch verwendet" in out
code, out, err = await cli("apply", "--prune", "--plan-token", token_of(out), stdin=only_aliases)
assert code == 4 and "blockierte Objekte" in err
async def test_kinds_schema_and_usage():
await seed_everything()
code, out, _ = await cli("export", "--kinds", "aliases")
assert code == 0 and list(yaml.safe_load(out)) == ["opnexus_config", "aliases"]
assert (await cli("export", "--kinds", "nope"))[0] == 1
code, out, _ = await cli("schema")
assert code == 0 and json.loads(out)["title"].startswith("OPNexus-Konfiguration")
assert (await cli("frobnicate"))[0] == 1
+129
View File
@@ -0,0 +1,129 @@
"""Export (1.32 P3): deterministic, complete, schema-conform, and free of anything secret."""
import re
import pytest
import yaml
from app.config_code import schema
from app.config_code.export import canonical, dump_yaml, export_yaml
from app.database import SessionLocal
from app.models.rule import FirewallRule
from tests.config_code_seed import seed_everything
GOLDEN = '''\
# OPNexus-Konfiguration (Schema v1), exportiert von OPNexus {version}.
# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run
opnexus_config: 1
device_groups:
- name: Filialen
description: alle Filialen
hosts:
- fw-filiale-1
- fw-filiale-2
policy_hierarchy_groups:
- name: Organisation
parent: null
description: ''
hosts: []
- name: Nord
parent: Organisation
description: Region Nord
hosts:
- fw-filiale-1
aliases:
- name: WEB_SERVERS
type: host
description: DMZ
enabled: true
entries:
- 10.20.30.10
- 203.0.113.20
targets:
hosts: []
groups:
- Filialen
hierarchy_group: null
'''
async def _export(**kw) -> str:
async with SessionLocal() as db:
return await export_yaml(db, **kw)
async def test_export_starts_with_the_fixed_header_and_the_expected_sections():
from app.version import APP_VERSION
await seed_everything()
text = await _export()
assert text.startswith(GOLDEN.format(version=APP_VERSION))
assert [k for k in yaml.safe_load(text) if k != "opnexus_config"] == list(schema.KINDS)
assert text.endswith("\n") and "\r" not in text
async def test_export_is_deterministic_and_validates_against_the_schema():
await seed_everything()
first, second = await _export(), await _export()
assert first == second
doc = schema.ConfigDoc.model_validate(yaml.safe_load(first)) # strict: unknown fields, wrong types, control chars
assert [r.key for r in doc.rules] == ["allow-dns-lan", "block-all"]
assert doc.variables == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Filiale Nord"}}
async def test_hierarchy_rules_are_ordered_by_group_rulebase_and_sequence_and_keep_placeholders_unresolved():
await seed_everything()
data = yaml.safe_load(await _export())
assert [r["key"] for r in data["policy_hierarchy_rules"]] == ["ph-block-bogons", "ph-allow-mgmt", "ph-post-deny"]
rule = next(r for r in data["rules"] if r["key"] == "allow-dns-lan")
assert rule["source_net"] == "{{LAN_NET}}" and rule["destination_port"] == "53"
async def test_ports_are_written_as_strings_so_they_survive_the_round_trip():
await seed_everything()
text = await _export()
assert re.search(r"destination_port: '53'", text)
assert schema.ConfigDoc.model_validate(yaml.safe_load(text)).rules[0].destination_port == "53"
async def test_export_never_contains_secrets_or_connection_data():
await seed_everything()
text = (await _export()).lower()
for forbidden in ("akia-secret-key", "secret-host.example", "https://", "api_key", "api_secret", "private key", "opx_", "password"):
assert forbidden not in text
async def test_kinds_limits_the_sections():
await seed_everything()
data = yaml.safe_load(await _export(kinds={"aliases", "rules"}))
assert list(data) == ["opnexus_config", "aliases", "rules"]
async def test_new_objects_get_a_key_and_an_empty_database_exports_empty_sections():
async with SessionLocal() as db:
rule = FirewallRule(description="x", interface="lan", direction="in")
db.add(rule)
await db.commit()
assert re.fullmatch(r"r-[0-9a-f]{12}", rule.config_key) and rule.config_key == f"r-{rule.id.hex[:12]}"
data = yaml.safe_load(await _export(kinds={"aliases", "variables"}))
assert data["aliases"] == [] and data["variables"] == {}
async def test_exporter_field_lists_match_the_schema():
"""A field added to the schema but not to the exporter (or vice versa) must fail loudly, not export silently wrong."""
await seed_everything()
async with SessionLocal() as db:
content = await canonical(db)
for kind, doc in (("rules", schema.RuleDoc), ("port_forwards", schema.PortForwardDoc), ("one_to_one", schema.OneToOneDoc),
("outbound_nat", schema.OutboundNatDoc), ("policy_hierarchy_rules", schema.HierarchyRuleDoc),
("aliases", schema.AliasDoc), ("device_groups", schema.DeviceGroupDoc),
("policy_hierarchy_groups", schema.HierarchyGroupDoc)):
assert all(list(item) == list(doc.model_fields) for item in content[kind]), kind
assert dump_yaml(content) == await _export()
async def test_objects_are_sorted_by_key_not_by_insertion_order():
async with SessionLocal() as db:
for key in ("zz-last", "aa-first", "mm-middle"):
db.add(FirewallRule(config_key=key, description=key, interface="lan", direction="in"))
await db.commit()
assert [r["key"] for r in yaml.safe_load(await _export())["rules"]] == ["aa-first", "mm-middle", "zz-last"]
+184
View File
@@ -0,0 +1,184 @@
"""Strict YAML loader (1.32 P2): everything a hostile file could try is refused with a stable code."""
import time
import pytest
from app.config_code import loader
from app.config_code.loader import ConfigError, load_yaml, parse
HEAD = "opnexus_config: 1\n"
def _code(raw) -> tuple[str, str]:
with pytest.raises(ConfigError) as info:
parse(raw)
return info.value.code, info.value.errors[0]["code"]
def _codes(raw) -> list[str]:
with pytest.raises(ConfigError) as info:
parse(raw)
return [e["code"] for e in info.value.errors]
RULE = "rules:\n- key: r-1\n description: d\n interface: lan\n"
def test_a_plain_file_loads():
doc, sha, _lines = parse(HEAD + RULE)
assert doc.rules[0].key == "r-1" and len(sha) == 64
@pytest.mark.parametrize("raw,code", [
# Billion laughs and every other use of anchors/aliases
("a: &x [1, 2]\nb: *x\n", "yaml_anchor_forbidden"),
("a: *x\n", "yaml_alias_forbidden"),
('lol1: &l1 ["x","x"]\nlol2: &l2 [*l1,*l1]\nlol3: [*l2,*l2]\n', "yaml_anchor_forbidden"),
# merge keys
("a: {b: 1}\nc:\n <<: {d: 1}\n", "yaml_merge_forbidden"),
# explicit tags of any kind
("a: !!python/object/apply:os.system ['id']\n", "yaml_tag_forbidden"),
("a: !!python/name:os.system\n", "yaml_tag_forbidden"),
("a: !!binary aGVsbG8=\n", "yaml_tag_forbidden"),
("a: !!set {x: null}\n", "yaml_tag_forbidden"),
("a: !!str 1\n", "yaml_tag_forbidden"),
("a: ! x\n", "yaml_tag_forbidden"),
("a: !local x\n", "yaml_tag_forbidden"),
("!!map {a: 1}\n", "yaml_tag_forbidden"),
# directives and multiple documents
("%YAML 1.1\n---\na: 1\n", "yaml_directive_forbidden"),
("%TAG !e! tag:example.com,2000:\n---\na: 1\n", "yaml_directive_forbidden"),
("a: 1\n---\nb: 2\n", "yaml_multiple_documents"),
("", "yaml_empty"),
("# only a comment\n", "yaml_empty"),
# keys
("a: 1\na: 2\n", "duplicate_key"),
("rules:\n- key: x\n key: y\n", "duplicate_key"),
("1: a\n", "non_string_key"),
("true: a\n", "non_string_key"),
("? [a]\n: b\n", "non_string_key"),
("null: a\n", "non_string_key"),
# structure limits
("a: " + "[" * 20 + "]" * 20 + "\n", "too_deep"),
("a: x\0y\n", "yaml_syntax"),
("a: 'x\x1b[31m'\n", "yaml_syntax"),
("a: [1, 2\n", "yaml_syntax"),
("a:\n\t- b\n", "yaml_syntax"),
])
def test_hostile_yaml_is_refused(raw, code):
assert _code(raw) == ("CONFIG_INVALID", code)
def test_errors_name_line_and_path():
with pytest.raises(ConfigError) as info:
parse(HEAD + RULE + " targets:\n hosts: [a]\n hosts: [b]\n")
error = info.value.errors[0]
assert error["code"] == "duplicate_key" and error["line"] == 8 and error["path"] == "rules[0].targets"
with pytest.raises(ConfigError) as info:
parse(HEAD + RULE + " destination_port: [1]\n secret_ref: x\n")
by_code = {e["code"]: e for e in info.value.errors}
assert by_code["unknown_field"]["path"] == "rules[0].secret_ref" and by_code["unknown_field"]["line"] == 7
assert by_code["unknown_field"]["params"] == {"field": "secret_ref"}
assert by_code["bad_type"]["path"].startswith("rules[0].destination_port") and by_code["bad_type"]["line"] == 6
def test_yaml_1_1_surprises_stay_text_or_are_type_errors():
data, _ = load_yaml("a: on\nb: yes\nc: 1.5\nd: 2026-10-22\ne: 0755\nf: 1:30\ng: 0x1F\nh: .inf\ni: Off\nj: 1e3\n")
assert data == {"a": "on", "b": "yes", "c": "1.5", "d": "2026-10-22", "e": "0755", "f": "1:30", "g": "0x1F",
"h": ".inf", "i": "Off", "j": "1e3"}
data, _ = load_yaml("a: true\nb: False\nc: null\nd: ~\ne:\nf: 53\ng: -1\nh: 0\n")
assert data == {"a": True, "b": False, "c": None, "d": None, "e": None, "f": 53, "g": -1, "h": 0}
assert "enabled" in _field_error(RULE + " enabled: yes\n") # strict bool: "yes" is not true
def _field_error(body: str) -> str:
with pytest.raises(ConfigError) as info:
parse(HEAD + body)
return " ".join(e["path"] for e in info.value.errors)
def test_huge_integers_stay_text_and_do_not_hit_pythons_int_limit():
data, _ = load_yaml("a: " + "9" * 5000 + "\n")
assert data["a"] == "9" * 5000
@pytest.mark.parametrize("raw,code", [
(b"a: \xff\xfe\n", "bad_encoding"),
(b"a: \xc3\x28\n", "bad_encoding"),
])
def test_invalid_utf8_is_refused(raw, code):
assert _code(raw) == ("CONFIG_INVALID", code)
def test_a_bom_is_accepted():
doc = parse("\ufeff" + HEAD).doc
assert doc.rules is None
def test_size_limits():
assert _code(b"#" * (loader.MAX_BYTES + 1)) == ("CONFIG_TOO_LARGE", "too_large")
assert _code("a: " + "x" * (loader.MAX_SCALAR + 1) + "\n") == ("CONFIG_INVALID", "scalar_too_long")
many = "a: [" + ",".join(["1"] * (loader.MAX_NODES + 1)) + "]\n"
assert _code(many) == ("CONFIG_INVALID", "too_many_nodes")
def test_object_and_entry_limits():
rules = "rules:\n" + "".join(f"- {{key: r-{i}, description: d, interface: lan}}\n" for i in range(loader.MAX_OBJECTS_PER_KIND + 1))
assert "too_many_objects" in _codes(HEAD + rules)
entries = "aliases:\n- name: A\n entries: [" + ",".join(f"10.0.{i // 250}.{i % 250}" for i in range(loader.MAX_ALIAS_ENTRIES + 1)) + "]\n"
assert "too_many_entries" in _codes(HEAD + entries)
long_entries = "aliases:\n- name: A\n entries: [" + ",".join(["x" * 999] * 9) + "]\n"
assert "too_long" in _codes(HEAD + long_entries)
variables = "variables:\n fw:\n" + "".join(f" V{i}: x\n" for i in range(loader.MAX_VARIABLES_PER_HOST + 1))
assert "too_many_objects" in _codes(HEAD + variables)
@pytest.mark.parametrize("raw,code,top", [
("- a\n", "not_a_mapping", "CONFIG_INVALID"),
("rules: []\n", "missing_version", "CONFIG_INVALID"),
("opnexus_config: '1'\n", "missing_version", "CONFIG_INVALID"),
("opnexus_config: true\n", "missing_version", "CONFIG_INVALID"),
("opnexus_config: 2\n", "unsupported_version", "CONFIG_UNSUPPORTED_VERSION"),
])
def test_version_and_shape(raw, code, top):
assert _code(raw) == (top, code)
def test_error_values_are_shortened_and_escaped():
with pytest.raises(ConfigError) as info:
parse(HEAD + RULE + " direction: \"" + "\\u001b[2J" + "x" * 200 + "\"\n")
value = info.value.errors[0]["params"]["value"]
assert len(value) <= 80 and "\x1b" not in value and value.startswith("\\x1b")
def test_one_mebibyte_parses_in_reasonable_time():
"""Pure-Python PyYAML; design risk (4). A full-size file of realistic rules must stay well under the request timeout."""
line = "- {key: r-%05d, description: Regel %05d mit etwas Text, interface: lan, source_net: 10.0.0.0/24, destination_port: '443'}\n"
body = HEAD + "rules:\n"
i = 0
while len(body) + len(line) < loader.MAX_BYTES - 200 and i < loader.MAX_OBJECTS_PER_KIND:
body += line % (i, i)
i += 1
body += "aliases:\n- name: BIG\n entries:\n" + "".join(f" - 10.{n // 65536}.{n // 256 % 256}.{n % 256}\n" for n in range(loader.MAX_ALIAS_ENTRIES))
raw = body.encode()[: loader.MAX_BYTES]
started = time.monotonic()
try:
parse(raw)
except ConfigError:
pass # truncation may cut the last line; the parse time is what counts
elapsed = time.monotonic() - started
print(f"parse {len(raw)} bytes: {elapsed:.2f}s")
assert elapsed < 8
WRONG_SHAPES = ["42", '"text"', "true", "{a: 1}", "[1, 2]", "[[1]]", "[{entries: 5}]", "[{entries: [1, [2]]}]", "null"]
@pytest.mark.parametrize("shape", WRONG_SHAPES)
@pytest.mark.parametrize("kind", __import__("app.config_code.schema", fromlist=["KINDS"]).KINDS)
def test_every_section_with_the_wrong_shape_is_a_config_error_never_another_exception(kind, shape):
"""Valid YAML of the wrong form used to end in a TypeError before the schema check (audit AUD-07)."""
try:
parse(f"{HEAD}{kind}: {shape}\n")
except ConfigError:
pass # the expected, orderly refusal (null parses as an empty section)
+125
View File
@@ -0,0 +1,125 @@
"""Schema v1 (1.32 P3): strict, equal to the API limits, JSON Schema file up to date."""
import typing
from pathlib import Path
import pytest
from pydantic import ValidationError
from app.config_code import schema
from app.schemas.alias import AliasCreate
from app.schemas.one_to_one import OneToOneCreate
from app.schemas.outbound_nat import OutboundNatCreate
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
from app.schemas.port_forward import PortForwardCreate
from app.schemas.rule import RuleCreate
# (file model, API model, fields that exist only on one side)
PAIRS = [
(schema.RuleDoc, RuleCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids"}),
(schema.PortForwardDoc, PortForwardCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
(schema.OneToOneDoc, OneToOneCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
(schema.OutboundNatDoc, OutboundNatCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
(schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, {"key", "group"}, {"hierarchy_group_id"}),
(schema.AliasDoc, AliasCreate, {"entries", "targets"}, {"content", "target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
]
@pytest.mark.parametrize("doc,api,only_doc,only_api", PAIRS, ids=[p[0].__name__ for p in PAIRS])
def test_file_model_has_the_same_fields_defaults_and_limits_as_the_api_model(doc, api, only_doc, only_api):
assert set(doc.model_fields) - only_doc == set(api.model_fields) - only_api
for name in set(doc.model_fields) - only_doc:
a, b = doc.model_fields[name], api.model_fields[name]
assert a.default == b.default, name
assert _limits(a) == _limits(b), name
def _limits(field) -> dict:
"""Length/pattern limits of a field, also when they sit on the str branch of `str | None`."""
metas = list(field.metadata)
for arg in typing.get_args(field.annotation):
metas += list(getattr(arg, "__metadata__", ()))
metas = [m for meta in metas for m in (meta.metadata if hasattr(meta, "metadata") else [meta])] # unwrap Field(...)
out = {}
for meta in metas:
for attr in ("min_length", "max_length", "pattern"):
if getattr(meta, attr, None) is not None:
out[attr] = getattr(meta, attr)
return out
def test_enumerations_match_the_api():
for doc, api, name in ((schema.RuleDoc, RuleCreate, "action"), (schema.PortForwardDoc, PortForwardCreate, "natreflection"),
(schema.OneToOneDoc, OneToOneCreate, "natreflection"), (schema.OneToOneDoc, OneToOneCreate, "type"),
(schema.AliasDoc, AliasCreate, "type"), (schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, "rulebase")):
assert doc.model_fields[name].annotation == api.model_fields[name].annotation, (doc.__name__, name)
def _rule(**over):
return {"key": "r-1", "description": "d", "interface": "lan", **over}
def test_unknown_fields_wrong_types_and_secret_ref_are_refused():
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(secret_ref="x"))
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(enabled="yes")) # strict: no truthy strings
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(source_port=True)) # bool is never a port
with pytest.raises(ValidationError):
schema.ConfigDoc.model_validate({"opnexus_config": 2})
def test_ports_accept_numbers_and_ranges_and_become_strings():
assert schema.RuleDoc.model_validate(_rule(destination_port=53)).destination_port == "53"
assert schema.RuleDoc.model_validate(_rule(destination_port="8000-8100")).destination_port == "8000-8100"
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(destination_port="1" * 33))
@pytest.mark.parametrize("bad", ["a\x00b", "a\x1b[31m", "a\x7fb", "a\rb", "a\x0bb"])
def test_control_characters_are_refused(bad):
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(description=bad))
with pytest.raises(ValidationError):
schema.AliasDoc.model_validate({"name": "A", "entries": [bad]})
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(interface=bad))
def test_descriptions_may_have_several_lines_and_padding_but_names_and_keys_not():
assert schema.RuleDoc.model_validate(_rule(description="a\n\tb ")).description == "a\n\tb "
assert schema.DeviceGroupDoc.model_validate({"name": "G", "description": " zwei\nZeilen "}).description == " zwei\nZeilen "
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(interface="a\nb")) # a newline only in free text
for bad in (" G", "G "):
with pytest.raises(ValidationError):
schema.DeviceGroupDoc.model_validate({"name": bad})
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(targets={"hosts": [" fw"]}))
@pytest.mark.parametrize("key", ["", "Upper", "-lead", "a b", "x" * 65, "ü"])
def test_keys_must_be_lowercase_slugs(key):
with pytest.raises(ValidationError):
schema.RuleDoc.model_validate(_rule(key=key))
def test_variable_names_and_values_use_the_same_rules_as_the_host_page():
ok = {"opnexus_config": 1, "variables": {"fw": {"LAN_NET": "10.0.0.0/24"}}}
assert schema.ConfigDoc.model_validate(ok).variables == {"fw": {"LAN_NET": "10.0.0.0/24"}}
for bad in ({"lan": "x"}, {"LAN": "{{X}}"}, {"LAN": "a\nb"}, {"LAN": ""}):
with pytest.raises(ValidationError):
schema.ConfigDoc.model_validate({"opnexus_config": 1, "variables": {"fw": bad}})
def test_a_missing_section_is_not_an_empty_section():
doc = schema.ConfigDoc.model_validate({"opnexus_config": 1, "aliases": []})
assert doc.aliases == [] and doc.rules is None
def test_checked_in_json_schema_is_current():
path = Path(schema.__file__).with_name(schema.SCHEMA_FILE)
assert path.read_text(encoding="utf-8") == schema.json_schema_text(), (
"Schema geändert: `docker compose run --rm --no-deps -v \"$PWD/backend:/app\" backend python -m app.config_code schema "
f"> backend/app/config_code/{schema.SCHEMA_FILE}`"
)
+122
View File
@@ -196,3 +196,125 @@ async def test_sweep_marks_revert_failed_when_host_unreachable(client, respx_moc
still_there = (await db.execute(select(RuleDeployment))).scalar_one_or_none()
assert still_there is not None
assert still_there.status == "revert_failed"
# ── A retry after "written, apply failed" must keep the ORIGINAL rollback target (audit 2026-10-23) ──
def _patch_rule_calls(monkeypatch, *, apply_results):
from unittest.mock import AsyncMock
from app.services import opnsense_client
calls = {
"add": AsyncMock(return_value={"uuid": "rule-uuid-1"}),
"set": AsyncMock(return_value={}),
"delete": AsyncMock(return_value={}),
"apply": AsyncMock(side_effect=apply_results),
}
monkeypatch.setattr(opnsense_client, "add_rule", calls["add"])
monkeypatch.setattr(opnsense_client, "set_rule", calls["set"])
monkeypatch.setattr(opnsense_client, "del_rule", calls["delete"])
monkeypatch.setattr(opnsense_client, "apply", calls["apply"])
return calls
@pytest.mark.asyncio
async def test_retry_after_failed_apply_still_removes_a_first_installation(client, respx_mock, monkeypatch):
from app.services.opnsense_client import OpnsenseAPIError
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = await _create_host(client, "fw1", "https://fw1.test")
rule_id = (await client.post("/api/rules", json=_rule_payload([host["id"]]))).json()["id"]
calls = _patch_rule_calls(monkeypatch, apply_results=[OpnsenseAPIError("apply failed"), {}, {}])
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
async with SessionLocal() as db:
dep = (await db.execute(select(RuleDeployment))).scalar_one()
assert dep.status == "error" and dep.revert_deadline is not None and dep.previous_payload is None
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
async with SessionLocal() as db:
dep = (await db.execute(select(RuleDeployment))).scalar_one()
assert dep.status == "pending"
assert dep.previous_payload is None # the rollback is still "remove the rule"
dep.revert_deadline = datetime.now(timezone.utc) - timedelta(seconds=1)
await db.commit()
await sweep_expired_deployments(db)
assert (await db.execute(select(RuleDeployment))).scalar_one_or_none() is None
assert calls["delete"].call_count == 1
assert calls["delete"].call_args.args[1] == "rule-uuid-1"
@pytest.mark.asyncio
async def test_retry_after_failed_apply_restores_the_confirmed_payload_of_an_existing_rule(client, respx_mock, monkeypatch):
from app.services.opnsense_client import OpnsenseAPIError
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = await _create_host(client, "fw1", "https://fw1.test")
rule_id = (await client.post("/api/rules", json=_rule_payload([host["id"]]))).json()["id"]
confirmed = {"description": "confirmed before", "action": "pass"}
async with SessionLocal() as db:
db.add(RuleDeployment(
rule_id=uuid.UUID(rule_id), host_id=uuid.UUID(host["id"]), status="success",
opnsense_rule_uuid="existing-uuid", last_payload=confirmed,
))
await db.commit()
calls = _patch_rule_calls(monkeypatch, apply_results=[OpnsenseAPIError("apply failed"), {}, {}])
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
async with SessionLocal() as db:
dep = (await db.execute(select(RuleDeployment))).scalar_one()
assert dep.status == "pending"
assert dep.previous_payload == confirmed
assert dep.last_payload != confirmed
dep.revert_deadline = datetime.now(timezone.utc) - timedelta(seconds=1)
await db.commit()
await sweep_expired_deployments(db)
assert calls["delete"].call_count == 0
assert calls["set"].call_args.args[2] == confirmed # the last set_rule is the rollback to the confirmed payload
@pytest.mark.asyncio
async def test_an_armed_error_row_blocks_cleanup_and_maintenance_like_a_pending_one(client, respx_mock, monkeypatch):
from app.models.rule_deployment import RuleDeployment as Dep
from app.services.deploy_flow import is_unsettled, unsettled
now = datetime.now(timezone.utc)
assert is_unsettled(Dep(status="error", revert_deadline=now + timedelta(minutes=1)))
assert not is_unsettled(Dep(status="error", revert_deadline=None))
assert not is_unsettled(Dep(status="success", revert_deadline=None))
assert is_unsettled(Dep(status="revert_failed", revert_deadline=None))
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = await _create_host(client, "fw1", "https://fw1.test")
rule_id = (await client.post("/api/rules", json=_rule_payload([host["id"]]))).json()["id"]
async with SessionLocal() as db:
db.add(Dep(rule_id=uuid.UUID(rule_id), host_id=uuid.UUID(host["id"]), status="error",
opnsense_rule_uuid="u", last_payload={"a": 1}, revert_deadline=now + timedelta(minutes=1)))
await db.commit()
assert await db.scalar(select(Dep.id).where(unsettled(Dep)))
@pytest.mark.asyncio
async def test_alias_retry_after_failed_reconfigure_still_removes_a_first_installation(client, respx_mock, monkeypatch):
from unittest.mock import AsyncMock
from app.models.alias_deployment import AliasDeployment
from app.services import opnsense_client
from app.services.alias_deploy_service import sweep_expired_alias_deployments
from app.services.opnsense_client import OpnsenseAPIError
from tests.test_aliases import _alias_payload
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = await _create_host(client, "fw1", "https://fw1.test")
alias_id = (await client.post("/api/aliases", json=_alias_payload([host["id"]]))).json()["id"]
delete = AsyncMock(return_value={})
monkeypatch.setattr(opnsense_client, "add_alias_item", AsyncMock(return_value={"uuid": "alias-uuid-1"}))
monkeypatch.setattr(opnsense_client, "set_alias_item", AsyncMock(return_value={}))
monkeypatch.setattr(opnsense_client, "del_alias_item", delete)
monkeypatch.setattr(opnsense_client, "reconfigure_aliases", AsyncMock(side_effect=[OpnsenseAPIError("reconfigure failed"), {}, {}]))
assert (await client.post(f"/api/aliases/{alias_id}/deploy")).status_code == 200
assert (await client.post(f"/api/aliases/{alias_id}/deploy")).status_code == 200
async with SessionLocal() as db:
dep = (await db.execute(select(AliasDeployment))).scalar_one()
assert dep.previous_payload is None
dep.revert_deadline = datetime.now(timezone.utc) - timedelta(seconds=1)
await db.commit()
await sweep_expired_alias_deployments(db)
assert delete.call_count == 1
+15 -1
View File
@@ -42,7 +42,7 @@ async def test_traceroute_is_parsed_and_audited(client, respx_mock):
{"address": "10.0.0.1", "ttl": "1", "AS": "AS0", "host": "gw.lan", "probes": "0.4 ms"}]}}))
resp = await client.post(f"/api/hosts/{host_id}/diagnostics/traceroute", json={"target": "example.com", "protocol": "icmp"})
assert resp.status_code == 200, resp.text
assert resp.json()["hops"] == [{"ttl": "1", "address": "10.0.0.1", "host": "gw.lan", "as": "AS0", "probes": "0.4 ms"}]
assert resp.json()["hops"] == [{"ttl": "1", "address": "10.0.0.1", "host": "gw.lan", "as": "AS0", "probes": "0.4 ms", "silent": False}]
assert json.loads(sent.calls.last.request.content)["traceroute"]["settings"]["hostname"] == "example.com"
events = await _events(host_id)
assert [(e.action, e.status, e.entity_name) for e in events] == [("diag.traceroute", "ok", "example.com")]
@@ -141,3 +141,17 @@ async def test_only_admins_can_run_probes(anon_client, client, respx_mock):
assert not probe.called
finally:
await client.delete(f"/api/users/{created.json()['id']}")
@pytest.mark.asyncio
async def test_hops_that_never_answered_come_back_as_explicit_silent_rows(client, respx_mock):
"""T5: the firewall skips them (TTL 1 -> 3); an all-silent route is an empty list the UI explains."""
host_id = await _host(client, respx_mock, "fw-diag-silent")
route = respx_mock.post(_url("/diagnostics/traceroute/set"))
route.mock(return_value=httpx.Response(200, json={"result": "ok", "response": {"notice": "n", "rows": [
{"address": "10.0.0.1", "ttl": "1", "AS": "", "host": "", "probes": "0.4 ms"},
{"address": "192.0.2.9", "ttl": "3", "AS": "", "host": "", "probes": "9 ms"}]}}))
hops = (await client.post(f"/api/hosts/{host_id}/diagnostics/traceroute", json={"target": "example.com"})).json()["hops"]
assert [(h["ttl"], h["address"], h["silent"]) for h in hops] == [("1", "10.0.0.1", False), ("2", "", True), ("3", "192.0.2.9", False)]
route.mock(return_value=httpx.Response(200, json={"result": "ok", "response": {"notice": "n", "rows": []}}))
assert (await client.post(f"/api/hosts/{host_id}/diagnostics/traceroute", json={"target": "example.com"})).json()["hops"] == []
+266
View File
@@ -0,0 +1,266 @@
"""Way back to the previous firmware (1.32, B5): rollback point before every update, guided rollback."""
import uuid
from datetime import datetime, timedelta, timezone
import httpx
import pytest
from sqlalchemy import select
from app.database import SessionLocal
from app.models.host_operation import HostOperation
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import backup_store, scheduled_maintenance_service
from tests.test_maintenance import execute, preview
from tests.test_rules import _create_host
from tests.test_scheduled_maintenance import _due_update
BASE = "https://fw1.test"
class Snapshots:
"""A firewall with ZFS boot environments, as Core/Api/SnapshotsController answers."""
def __init__(self, mock, *, version="26.7.6", status="update", fail_add=False, hide_new=False, fail_reboot=False,
mark_on_activate=True):
self.rows = [{"uuid": "be-default", "name": "default", "active": "NR", "created_str": "2026-10-01 10:00"}]
self.fail_add, self.hide_new, self.fail_reboot, self.mark_on_activate = fail_add, hide_new, fail_reboot, mark_on_activate
self.activated: list[str] = []
mock.get(f"{BASE}/api/diagnostics/interface/getVipStatus").respond(200, json={"rows": []})
mock.get(f"{BASE}/api/core/firmware/info").respond(200, json={"product_version": version})
mock.get(f"{BASE}/api/core/firmware/status").respond(200, json={"status": status, "all_packages": {}, "status_msg": "ok"})
mock.get(f"{BASE}/api/core/firmware/running").respond(200, json={"status": "ready"})
mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": True})
self.fail_list = False
mock.get(f"{BASE}/api/core/snapshots/search").mock(side_effect=self._search)
self.add = mock.post(f"{BASE}/api/core/snapshots/add").mock(side_effect=self._add)
self.delete = mock.post(url__regex=rf"{BASE}/api/core/snapshots/del/.*").mock(side_effect=self._del)
self.activate = mock.post(url__regex=rf"{BASE}/api/core/snapshots/activate/.*").mock(side_effect=self._activate)
self.update = mock.post(f"{BASE}/api/core/firmware/update").respond(200, json={"status": "ok"})
self.upgrade = mock.post(f"{BASE}/api/core/firmware/upgrade").respond(200, json={"status": "ok"})
self.reboot = mock.post(f"{BASE}/api/core/system/reboot").mock(side_effect=self._reboot)
self.download = mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
def _search(self, request):
if self.fail_list:
return httpx.Response(502, json={})
return httpx.Response(200, json={"rows": self.rows})
def point(self, name: str, active: str = "-") -> None:
self.rows.append({"uuid": f"be-{name}", "name": name, "active": active, "created_str": "x"})
def _add(self, request):
if self.fail_add:
return httpx.Response(200, json={"status": "failed", "validations": {"snapshot.name": "nope"}})
import json
name = json.loads(request.content)["name"]
if not self.hide_new:
self.point(name)
return httpx.Response(200, json={"status": "ok"})
def _del(self, request):
uid = str(request.url).rsplit("/", 1)[-1]
self.rows = [r for r in self.rows if r["uuid"] != uid]
return httpx.Response(200, json={"status": "ok"})
def _activate(self, request):
uid = str(request.url).rsplit("/", 1)[-1]
self.activated.append(uid)
if getattr(self, "fail_list_after_activate", False) and len(self.activated) == 1:
self.fail_list = True
if self.mark_on_activate:
for r in self.rows:
r["active"] = ("N" if "N" in r["active"] else "") + ("R" if r["uuid"] == uid else "")
r["active"] = r["active"] or "-"
return httpx.Response(200, json={"status": "ok"})
def _reboot(self, request):
if self.fail_reboot:
return httpx.Response(500, json={})
return httpx.Response(200, json={"status": "ok"})
async def _host_with_backup(client):
h = await _create_host(client, "fw1", BASE)
backup_store.save_snapshot(uuid.UUID(h["id"]), b"<opnsense/>")
return h
async def _op(op_id) -> HostOperation:
async with SessionLocal() as db:
return await db.get(HostOperation, uuid.UUID(str(op_id)))
async def test_a_manual_update_creates_a_rollback_point_first_and_keeps_only_two(client, respx_mock):
fw = Snapshots(respx_mock)
fw.point("opnexus-pre-26.7.4-20261001-100000")
fw.point("opnexus-pre-26.7.5-20261010-100000")
fw.point("handmade") # never touched
h = await _host_with_backup(client)
p = await preview(client, h, "update")
r = await execute(client, h, p.json())
assert r.json()["status"] == "submitted" and fw.update.call_count == 1
names = [row["name"] for row in fw.rows]
new = next(n for n in names if n.startswith("opnexus-pre-26.7.6-"))
assert "opnexus-pre-26.7.4-20261001-100000" not in names # the oldest of three is cleaned up
assert {"default", "handmade", "opnexus-pre-26.7.5-20261010-100000", new} == set(names)
op = await _op(r.json()["id"])
assert op.snapshot["rollback_point"]["name"] == new and op.snapshot["rollback_point"]["version"] == "26.7.6"
assert fw.add.call_count == 1
order = [str(c.request.url) for c in respx_mock.calls if c.request.method == "POST"]
assert order.index(f"{BASE}/api/core/snapshots/add") < order.index(f"{BASE}/api/core/firmware/update")
@pytest.mark.parametrize("problem", ["fail_add", "hide_new"])
async def test_no_rollback_point_no_update(client, respx_mock, problem):
fw = Snapshots(respx_mock, **{problem: True})
h = await _host_with_backup(client)
p = await preview(client, h, "update")
r = await execute(client, h, p.json())
assert r.json()["status"] == "cancelled" and "Update nicht gestartet" in r.json()["message"]
assert fw.update.call_count == 0
async def test_without_zfs_the_update_runs_and_says_there_is_no_way_back(client, respx_mock):
fw = Snapshots(respx_mock)
respx_mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": False})
h = await _host_with_backup(client)
r = await execute(client, h, (await preview(client, h, "update")).json())
assert r.json()["status"] == "submitted" and "Kein Rückweg-Punkt" in r.json()["message"]
assert fw.add.call_count == 0 and fw.update.call_count == 1
async def test_a_scheduled_update_fails_closed_without_a_rollback_point(client, respx_mock):
fw = Snapshots(respx_mock, fail_add=True)
respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
h = await _create_host(client, "fw1", BASE)
sched_id = await _due_update(h)
async with SessionLocal() as db:
await scheduled_maintenance_service.fire_due(db)
async with SessionLocal() as db:
sched = await db.get(ScheduledMaintenance, sched_id)
assert sched.status == "failed" and "Rückweg-Punkt" in sched.result["error"]
assert fw.update.call_count == 0
async def test_rollback_needs_a_point_from_opnexus(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.point("handmade")
h = await _create_host(client, "fw1", BASE)
r = await preview(client, h, "firmware_rollback")
assert r.status_code == 409 and "Kein Rückweg-Punkt" in r.json()["detail"]["params"]["reason"]
assert (await client.post(f"/api/hosts/{h['id']}/snapshots", json={"name": "opnexus-pre-1-20260101-000000"})).status_code == 422
async def test_rollback_activates_the_newest_point_reboots_and_lists_what_is_lost(client, respx_mock):
fw = Snapshots(respx_mock, version="26.7.7", status="none")
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
fw.point(f"opnexus-pre-26.7.5-{(stamp - timedelta(days=5)):%Y%m%d-%H%M%S}")
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
h = await _create_host(client, "fw1", BASE)
async with SessionLocal() as db:
rule = FirewallRule(description="nachher", interface="lan", direction="in")
db.add(rule)
await db.flush()
db.add(RuleDeployment(rule_id=rule.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u"))
await db.commit()
p = await preview(client, h, "firmware_rollback")
assert p.status_code == 200, p.text
rollback = p.json()["state"]["rollback"]
assert rollback["target"]["version"] == "26.7.6" and rollback["current"]["name"] == "default"
assert [c["kind"] for c in rollback["changes_since"]] == ["rule"]
r = await execute(client, h, p.json())
assert r.json()["status"] == "submitted" and "erwartet 26.7.6" in r.json()["message"]
assert fw.activated == [rollback["target"]["uuid"]] and fw.reboot.call_count == 1
async def test_rollback_that_is_not_marked_for_the_next_start_never_reboots(client, respx_mock):
fw = Snapshots(respx_mock, status="none", mark_on_activate=False)
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
async def test_a_failed_reboot_reactivates_the_current_boot_environment(client, respx_mock):
fw = Snapshots(respx_mock, status="none", fail_reboot=True)
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "unknown"
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
assert next(row for row in fw.rows if row["uuid"] == "be-default")["active"].endswith("R")
async def test_rollback_on_a_carp_master_is_refused_like_a_reboot(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
r = await preview(client, h, "firmware_rollback")
assert r.status_code == 409 and fw.activate.call_count == 0
async def test_the_rollback_endpoint_needs_the_deploy_right_like_every_maintenance_execute():
from app.services import api_token_service
assert api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/execute")
assert not api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/preview")
async def test_rollback_saves_the_current_configuration_first_and_refuses_without_it(client, respx_mock):
"""Going back discards every change since the point on the firewall: OPNexus keeps a copy of the config first."""
fw = Snapshots(respx_mock, status="none")
fw.point("opnexus-pre-26.7.6-20261020-100000")
download = respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense>now</opnsense>")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "submitted" and download.call_count == 1
assert backup_store.list_snapshots(uuid.UUID(h["id"])) # stored locally
order = [str(c.request.url) for c in respx_mock.calls]
assert order.index(f"{BASE}/api/core/backup/download/this") < order.index(f"{BASE}/api/core/snapshots/activate/be-opnexus-pre-26.7.6-20261020-100000")
async def test_rollback_without_a_configuration_copy_does_not_start(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.point("opnexus-pre-26.7.6-20261020-100000")
respx_mock.get(f"{BASE}/api/core/backup/download/this").mock(side_effect=httpx.ConnectError("down"))
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "cancelled" and fw.activate.call_count == 0 and fw.reboot.call_count == 0
async def test_an_unreadable_list_after_activating_switches_back_and_never_reboots(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
fw.fail_list_after_activate = True
fw.point("opnexus-pre-26.7.6-20261020-100000")
h = await _create_host(client, "fw1", BASE)
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
async def test_after_a_rollback_newer_deployments_are_marked_so_the_next_deploy_really_writes(client, respx_mock):
fw = Snapshots(respx_mock, status="none")
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
h = await _create_host(client, "fw1", BASE)
async with SessionLocal() as db:
old, new = FirewallRule(description="alt", interface="lan", direction="in"), FirewallRule(description="neu", interface="lan", direction="in")
db.add_all([old, new])
await db.flush()
db.add_all([
RuleDeployment(rule_id=old.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u1",
last_payload={"x": 1}, deployed_at=stamp - timedelta(days=1)),
RuleDeployment(rule_id=new.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u2",
last_payload={"x": 2}, deployed_at=stamp + timedelta(minutes=30)),
])
await db.commit()
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
assert r.json()["status"] == "submitted" and "1 Deployment(s) als unklar markiert" in r.json()["message"]
async with SessionLocal() as db:
rows = {d.opnsense_rule_uuid: d for d in (await db.scalars(select(RuleDeployment))).all()}
assert rows["u1"].status == "success" and rows["u1"].last_payload == {"x": 1} # older than the point: untouched
assert rows["u2"].status == "error" and rows["u2"].last_payload is None and "Firmware-Rückweg" in rows["u2"].message
+15
View File
@@ -218,3 +218,18 @@ async def test_rejected_draft_can_be_corrected_then_pushed(client, respx_mock):
respx_mock.post('https://fw1.test/api/firewall/filter/apply').mock(return_value=httpx.Response(200, json={'status':'OK\n'}))
pushed = await client.post(f"/api/rules/{r['id']}/deploy")
assert pushed.json()['deployments'][0]['status'] == 'pending'
async def test_a_group_that_targets_a_vpn_object_cannot_be_deleted(client, respx_mock):
"""B15: the VPN link tables cascade, so the delete used to succeed and the VPN silently lost its targets."""
from tests.test_ipsec import _tunnel_payload
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = await _create_host(client, "fw-vpn", "https://fw-vpn.test")
target = await group(client, [host["id"]], name="VPN-Ziel")
resp = await client.post("/api/ipsec/tunnels", json=_tunnel_payload([], target_group_ids=[target["id"]]))
assert resp.status_code == 201, resp.text
refused = await client.delete(f"/api/groups/{target['id']}")
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET"
assert (await client.delete(f"/api/ipsec/tunnels/{resp.json()['id']}")).status_code == 204
assert (await client.delete(f"/api/groups/{target['id']}")).status_code == 204
+55 -5
View File
@@ -5,6 +5,7 @@ import pytest
from tests.test_rules import _rule_payload
from tests.test_aliases import _alias_payload, _mock_alias_push
from tests.helpers import delete_host
@pytest.mark.asyncio
@@ -18,12 +19,12 @@ async def test_delete_host_with_active_rule_deployment_rejected(client, respx_mo
await client.post(f"/api/rules/{rule['id']}/deploy")
await client.post(f"/api/rules/{rule['id']}/confirm")
resp = await client.delete(f"/api/hosts/{host['id']}")
resp = await delete_host(client, host['id'])
assert resp.status_code == 409, resp.text
respx_mock.post("https://fw-k.test/api/firewall/filter/delRule/r1").mock(return_value=httpx.Response(200, json={"result": "deleted"}))
await client.delete(f"/api/rules/{rule['id']}") # undeploys + removes the rule first
assert (await client.delete(f"/api/hosts/{host['id']}")).status_code == 204
assert (await delete_host(client, host['id'])).status_code == 204
@pytest.mark.asyncio
@@ -36,12 +37,12 @@ async def test_delete_host_with_active_alias_deployment_rejected(client, respx_m
await client.post(f"/api/aliases/{alias['id']}/deploy")
await client.post(f"/api/aliases/{alias['id']}/confirm")
resp = await client.delete(f"/api/hosts/{host['id']}")
resp = await delete_host(client, host['id'])
assert resp.status_code == 409, resp.text
respx_mock.post("https://fw-l.test/api/firewall/alias/delItem/alias-uuid-1").mock(return_value=httpx.Response(200, json={"result": "deleted"}))
await client.delete(f"/api/aliases/{alias['id']}")
assert (await client.delete(f"/api/hosts/{host['id']}")).status_code == 204
assert (await delete_host(client, host['id'])).status_code == 204
@pytest.mark.asyncio
@@ -72,7 +73,7 @@ async def test_create_list_delete_host(client, respx_mock):
names = [h["name"] for h in resp.json()]
assert "fw1" in names
resp = await client.delete(f"/api/hosts/{host_id}")
resp = await delete_host(client, host_id)
assert resp.status_code == 204
resp = await client.get("/api/hosts")
@@ -587,3 +588,52 @@ async def test_host_address_can_be_changed_after_the_firewall_moved(client, resp
for bad in ("ftp://x", "http://10.9.9.3", "https://user:[email protected]", "https://10.9.9.3/?a=b"):
assert (await client.patch(f"/api/hosts/{host['id']}", json={"base_url": bad})).status_code == 422, bad
assert next(h for h in (await client.get("/api/hosts")).json() if h["id"] == host["id"])["base_url"] == "https://10.9.9.2"
@pytest.mark.asyncio
async def test_a_second_host_with_the_same_address_is_refused(client, respx_mock):
"""T4: the same firewall twice would double every rollout, alert and backup."""
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
first = await client.post("/api/hosts", json={"name": "dup-a", "base_url": "https://dup.test", "api_key": "k", "api_secret": "s"})
assert first.status_code == 201
for url in ("https://dup.test", "https://DUP.test/", "https://dup.test//"):
again = await client.post("/api/hosts", json={"name": "dup-b", "base_url": url, "api_key": "k", "api_secret": "s"})
assert again.status_code == 409, url
assert again.json()["detail"] == {"code": "HOST_URL_EXISTS", "params": {"host": "dup-a"}}
other = (await client.post("/api/hosts", json={"name": "dup-c", "base_url": "https://dup-c.test", "api_key": "k", "api_secret": "s"})).json()
moved = await client.patch(f"/api/hosts/{other['id']}", json={"base_url": "https://dup.test"})
assert moved.status_code == 409 and moved.json()["detail"]["code"] == "HOST_URL_EXISTS"
assert (await client.patch(f"/api/hosts/{other['id']}", json={"base_url": "https://dup-c.test/"})).status_code == 200 # itself
@pytest.mark.asyncio
@pytest.mark.parametrize("failure,reason", [
(httpx.Response(401, json={"status": 401}), "credentials"),
(httpx.Response(403, json={"status": 403}), "forbidden"),
(httpx.ConnectError("no route"), "unreachable"),
(httpx.Response(200, json={"rows": []}), None),
])
async def test_verify_connection_asks_the_firewall_once_and_keeps_nothing_on_failure(client, respx_mock, failure, reason):
route = respx_mock.get(url__regex=r".*/getVipStatus")
route.mock(side_effect=failure) if isinstance(failure, Exception) else route.mock(return_value=failure)
body = {"name": f"vc-{reason}", "base_url": f"https://vc-{reason}.test", "api_key": "k", "api_secret": "s", "verify_connection": True}
resp = await client.post("/api/hosts", json=body)
names = [h["name"] for h in (await client.get("/api/hosts")).json()]
if reason is None:
assert resp.status_code == 201 and body["name"] in names
else:
assert resp.status_code == 422 and resp.json()["detail"] == {"code": "HOST_CONNECTION_FAILED", "params": {"reason": reason}}
assert body["name"] not in names # nothing saved, so a corrected retry is not blocked by a leftover
assert (await client.post("/api/hosts", json={**body, "verify_connection": False})).status_code == 201 # "create anyway"
@pytest.mark.asyncio
async def test_deleting_a_host_needs_its_name_on_the_server_side_too(client, respx_mock):
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
host = (await client.post("/api/hosts", json={"name": "del-me", "base_url": "https://del-me.test", "api_key": "k", "api_secret": "s"})).json()
for params in ({}, {"confirm_name": ""}, {"confirm_name": "DEL-ME"}, {"confirm_name": "other"}):
refused = await client.delete(f"/api/hosts/{host['id']}", params=params)
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "HOSTNAMEN_EXAKT_EINGEBEN", params
assert (await client.get(f"/api/hosts/{host['id']}")).status_code == 200
assert (await client.delete(f"/api/hosts/{host['id']}", params={"confirm_name": "del-me"})).status_code == 204
assert (await client.delete(f"/api/hosts/{host['id']}", params={"confirm_name": "del-me"})).status_code == 404
+1
View File
@@ -14,6 +14,7 @@ def mock_reads(mock,base='https://fw1.test',carp=None,status='none',running='rea
mock.get(f'{base}/api/core/firmware/info').respond(200,json={'product_version':version})
mock.get(f'{base}/api/core/firmware/status').respond(200,json={'status':status,'all_packages':{},'status_msg':'cached result'})
mock.get(f'{base}/api/core/firmware/running').respond(200,json={'status':running})
mock.get(f'{base}/api/core/snapshots/isSupported').respond(200,json={'supported':False}) # UFS: no rollback point (1.32)
async def preview(client,h,action='reboot'):
return await client.post(f"/api/hosts/{h['id']}/maintenance/preview",json={'action':action})
+46 -1
View File
@@ -450,4 +450,49 @@ async def test_0029_creates_the_rule_cleanup_tables_and_settings_and_is_idempote
assert not {"cleanup_max_rules", "cleanup_delete_min_days", "cleanup_management_sources", "cleanup_management_ports"} & cols
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head") # second run: nothing to do
assert await _version(scratch) == _head() == "0029" and await _schema_diff(scratch) == []
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async def test_0030_backfills_config_keys_and_is_idempotent(scratch):
"""1.31 -> 1.32: every rule/NAT table gets a unique, readable key derived from the UUID; the downgrade drops it."""
await _alembic(scratch, "upgrade", "0029")
rule_id = "3f2a9c1d-0000-4000-8000-000000000001"
async with scratch.begin() as conn:
await conn.execute(text(
"INSERT INTO firewall_rules (id, description, interface, direction, ipprotocol, protocol, action, "
"source_net, destination_net, enabled, log, created_at, updated_at) VALUES "
f"('{rule_id}', 'r', 'lan', 'in', 'inet', 'any', 'pass', 'any', 'any', true, false, now(), now())"))
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async with scratch.connect() as conn:
key = (await conn.execute(text("SELECT config_key FROM firewall_rules"))).scalar_one()
nullable = (await conn.execute(text(
"SELECT is_nullable FROM information_schema.columns "
"WHERE table_name = 'port_forward_rules' AND column_name = 'config_key'"))).scalar_one()
assert key == "r-3f2a9c1d0000" and nullable == "NO"
await _alembic(scratch, "downgrade", "0029")
async with scratch.connect() as conn:
cols = set((await conn.execute(text(
"SELECT column_name FROM information_schema.columns WHERE table_name = 'firewall_rules'"))).scalars().all())
assert "config_key" not in cols
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async def test_0031_adds_gate_probes_to_existing_plans_and_is_idempotent(scratch):
await _alembic(scratch, "upgrade", "0030")
async with scratch.begin() as conn:
await conn.execute(text(
"INSERT INTO rollout_plans (id, name, description, action, auto_start, gate_checks, compliance_mode, created_by, created_at, updated_at) "
"VALUES (gen_random_uuid(), 'p', '', 'update', false, '{}', 'warn', 'admin', now(), now())"))
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
async with scratch.connect() as conn:
assert (await conn.execute(text("SELECT gate_probes::text FROM rollout_plans"))).scalar_one() == "[]"
await _alembic(scratch, "downgrade", "0030")
await _alembic(scratch, "upgrade", "head")
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
+16
View File
@@ -0,0 +1,16 @@
import subprocess
import sys
def test_importing_models_alone_configures_all_mappers():
"""`python -m app.reset_password` / `app.reset_2fa` import only part of the app: the mapper registry must still be complete."""
code = "import app.models; from sqlalchemy.orm import configure_mappers; configure_mappers()"
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
assert result.returncode == 0, result.stderr[-800:]
def test_reset_scripts_import_cleanly_and_configure_mappers():
for module in ("app.reset_password", "app.reset_2fa"):
code = f"import {module}; from sqlalchemy.orm import configure_mappers; configure_mappers()"
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
assert result.returncode == 0, (module, result.stderr[-800:])
+3 -2
View File
@@ -1,6 +1,7 @@
import httpx
import pytest
from tests.test_rules import _create_host
from tests.helpers import delete_host
@pytest.mark.asyncio
@pytest.mark.parametrize('kind,remote,fields',[('port-forward','d_nat',{'target':'192.0.2.20'}),('one-to-one','one_to_one',{'external':'192.0.2.20'}),('outbound-nat','source_nat',{'target':'192.0.2.20'})])
@@ -14,7 +15,7 @@ async def test_nat_groups_and_deletion_guards(client,respx_mock,outbound_mode_hy
respx_mock.post(f'https://fw1.test/api/firewall/{remote}/addRule').respond(200,json={'result':'saved','uuid':'nat1'})
respx_mock.post(f'https://fw1.test/api/firewall/{remote}/apply').respond(200,json={'status':'ok'})
assert (await client.post(f"/api/{kind}/{r['id']}/deploy")).status_code==200
assert (await client.delete('/api/hosts/'+h['id'])).status_code==409
assert (await delete_host(client, h['id'])).status_code==409
impact=(await client.post(f"/api/groups/{g['id']}/preview",json={'name':'NAT group','description':'','host_ids':[]})).json()
assert impact['nat'][0]['retained_host_ids']==[h['id']]
assert impact['nat'][0]['kind']==kind.replace('-','_')
@@ -25,7 +26,7 @@ async def test_nat_groups_and_deletion_guards(client,respx_mock,outbound_mode_hy
respx_mock.post(f'https://fw1.test/api/firewall/{remote}/delRule/nat1').respond(200,json={'result':'deleted'})
p=(await client.get(f"/api/{kind}/{r['id']}/preview")).json()
assert (await client.post(f"/api/{kind}/{r['id']}/deployments/{h['id']}/remove",json={'preview_token':p['token']})).status_code==204
assert (await client.delete('/api/hosts/'+h['id'])).status_code==204
assert (await delete_host(client, h['id'])).status_code==204
@pytest.mark.asyncio
async def test_pf_disabled_polarity_and_readonly_automatic_rows(client,respx_mock):
+22
View File
@@ -600,3 +600,25 @@ async def test_vxlan_and_gif_are_listed_and_changed_like_the_other_devices(clien
done = await _execute(client, host_id, plan["token"], **change)
assert done.status_code == 200 and done.json()["verified"] is True, done.text
assert _body(add.calls.last) == {"gif": change["fields"]}
@pytest.mark.asyncio
@pytest.mark.parametrize("kind,field,bad,good", [
("vlan", "tag", ["0", "4095", "-1", "abc", "20.5", "99999"], ["1", "20", "4094"]),
("vlan", "pcp", ["8", "-1"], ["0", "7"]),
("vip", "vhid", ["0", "256"], ["1", "255", ""]),
("vip", "advbase", ["0", "255"], ["1", "254"]),
("vip", "advskew", ["255", "-5"], ["0", "254"]),
])
async def test_numbers_outside_the_firewalls_range_are_refused_in_the_preview(client, respx_mock, kind, field, bad, good):
"""T6: a VLAN tag of 4095 used to come back from the firewall as 502 UPSTREAM_REJECTED; now 422 with the field name,
before the firewall is asked anything (ranges from Interfaces/Vlan.xml and Vip.xml)."""
host_id = await _setup(client, respx_mock, f"fw-r-{kind}-{field}")
base_fields = {"vlan": {"if": "vtnet1", "tag": "20"}, "vip": {"interface": "lan", "mode": "carp", "network": "192.0.2.1/24"}}[kind]
for value in bad:
resp = await _preview(client, host_id, kind=kind, action="add", fields={**base_fields, field: value})
assert resp.status_code == 422, (field, value)
assert resp.json()["detail"] == {"code": "NETWORK_FIELD_INVALID", "params": {"field": field}}
for value in good:
resp = await _preview(client, host_id, kind=kind, action="add", fields={**base_fields, field: value})
assert resp.status_code == 200, (field, value, resp.text)
+11 -1
View File
@@ -2,6 +2,7 @@
confused login could succeed."""
import base64
import hashlib
import hmac
import json
import time
from urllib.parse import parse_qs, urlsplit
@@ -72,6 +73,15 @@ def _claims(sent_nonce, **over):
return {k: v for k, v in base.items() if v is not None}
def _forged_hs256(claims, secret: str, kid="k1"):
"""What an attacker sends: PyJWT >= 2.12 itself refuses to sign with a JWK-looking secret, so build the token by hand."""
def b64(raw: bytes) -> bytes:
return base64.urlsafe_b64encode(raw).rstrip(b"=")
signing_input = b64(json.dumps({"alg": "HS256", "typ": "JWT", "kid": kid}).encode()) + b"." + b64(json.dumps(claims).encode())
signature = hmac.new(secret.encode(), signing_input, hashlib.sha256).digest()
return (signing_input + b"." + b64(signature)).decode()
def _id_token(claims, key=KEY, alg="RS256", kid="k1"):
return jwt.encode(claims, key, algorithm=alg, headers={"kid": kid})
@@ -221,7 +231,7 @@ async def test_signature_attacks_are_rejected(client, anon_client, respx_mock):
if spec is None:
token = jwt.encode(claims, None, algorithm="none", headers={"kid": "k1"})
elif spec == "hs":
token = jwt.encode(claims, json.dumps(JWK), algorithm="HS256", headers={"kid": "k1"})
token = _forged_hs256(claims, json.dumps(JWK))
else:
token = _id_token(claims, **spec)
resp, _ = await _callback(anon_client, respx_mock, query, token_response={"id_token": token})
+28 -1
View File
@@ -1,4 +1,5 @@
"""The operator role: rolls out and operates, but cannot change access or destinations."""
import httpx
import pytest
from sqlalchemy import select
@@ -42,7 +43,8 @@ async def test_operator_keeps_self_service(client):
assert (await op.put("/api/settings", json={"theme": "dark"})).status_code == 200
resp = await op.post("/api/auth/change-password", json={"current_password": PW, "new_password": "another-password-1"})
assert resp.status_code == 200, resp.text
assert (await op.get("/api/users")).status_code == 200
listed = await op.get("/api/users") # self-service: only the own account, never the directory (T2)
assert listed.status_code == 200 and [u["username"] for u in listed.json()] == ["selfserve"]
finally:
await op.aclose()
@@ -136,6 +138,8 @@ async def test_demoting_an_admin_to_operator_takes_effect_at_once(client):
("POST", "/api/rules/abc/confirm", False),
("GET", "/api/rules/abc/deploy", False),
("POST", "/api/hosts/abc/network/changes/preview", False),
("POST", "/api/hosts/abc/snapshots", True),
("GET", "/api/hosts/abc/snapshots", False),
])
def test_what_counts_as_a_rollout(method, path, rollout):
from app.services import api_token_service
@@ -206,3 +210,26 @@ async def test_channel_test_button_is_admin_only_on_the_server_too(client):
await op.aclose()
if ro is not None:
await ro.aclose()
@pytest.mark.asyncio
async def test_creating_a_snapshot_is_a_rollout_and_follows_the_separation_of_duties(client, respx_mock):
"""A boot environment is created ON the firewall: with deploying admin-only an operator may not do it (AUD-05)."""
from tests.test_snapshots import BASE, ROW, _host
host_id = await _host(client, respx_mock, "fw-oper-snap")
new = {"uuid": "s9", "name": "by_operator", "active": "-", "mountpoint": "-", "size": "0", "created_str": "now"}
respx_mock.get(f"{BASE}/api/core/snapshots/isSupported").mock(return_value=httpx.Response(200, json={"supported": True}))
respx_mock.get(f"{BASE}/api/core/snapshots/search").mock(return_value=httpx.Response(200, json={"rows": [ROW, new]}))
add = respx_mock.post(f"{BASE}/api/core/snapshots/add").mock(return_value=httpx.Response(200, json="OK"))
op = await _operator(client, "snapop")
try:
assert (await op.post(f"/api/hosts/{host_id}/snapshots", json={"name": "by_operator"})).status_code == 201
assert add.call_count == 1
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": False})).status_code == 200
resp = await op.post(f"/api/hosts/{host_id}/snapshots", json={"name": "by_operator"})
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
assert add.call_count == 1 # nothing was sent to the firewall
assert (await client.post(f"/api/hosts/{host_id}/snapshots", json={"name": "by_operator"})).status_code == 201 # an admin may
finally:
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
await op.aclose()
+26
View File
@@ -1,4 +1,6 @@
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
import uuid
import pytest
from tests.test_maintenance_windows import _WEEKLY
@@ -145,3 +147,27 @@ async def test_role_matrix_for_plans(client):
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
finally:
await ro.aclose()
@pytest.mark.asyncio
async def test_gate_probes_are_validated_stored_and_frozen_into_the_fingerprint(client):
"""1.32 (B5): probes per plan; a plan without probes keeps the fingerprint of earlier versions."""
from app.database import SessionLocal
from app.models.rollout import RolloutPlan
from app.services import rollout_service
_hosts, groups = await _fleet(client)
plain = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
assert plain["gate_probes"] == []
async with SessionLocal() as db:
assert "gate_probes" not in rollout_service.definition(await db.get(RolloutPlan, uuid.UUID(plain["id"])))
probes = [{"type": "ping", "target": "1.1.1.1"}, {"type": "tcp", "target": "intranet.example", "port": 443},
{"type": "ping", "target": "1.1.1.1"}]
resp = await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=probes))
assert resp.status_code == 200, resp.text
assert resp.json()["gate_probes"] == [{"type": "ping", "target": "1.1.1.1", "port": None},
{"type": "tcp", "target": "intranet.example", "port": 443}]
for bad in ([{"type": "tcp", "target": "x"}], [{"type": "ping", "target": "x", "port": 1}],
[{"type": "ping", "target": "-rf"}], [{"type": "ping", "target": "a b"}], [{"type": "udp", "target": "x"}],
[{"type": "ping", "target": f"h{i}"} for i in range(11)]):
assert (await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=bad))).status_code == 422, bad
+39
View File
@@ -31,6 +31,7 @@ class Fw:
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
r, b = fleet.mock, self.base
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
r.get(f"{b}/api/core/snapshots/isSupported").mock(side_effect=lambda q: self._ok({"supported": False}))
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
@@ -620,6 +621,19 @@ def test_target_version_comes_from_the_core_package_not_product_latest():
assert engine.target_of({"upgrade_major_version": "27.1"}, "major_update") == "27.1"
def test_target_version_of_other_release_types_and_a_switched_core_package():
# Shape seen live on test-fw-1 (2026-10-23) after switching the release type community -> development.
switch = {"status": "update", "product": {"product_id": "opnsense", "product_version": "26.7.7", "product_latest": "26.7.7"},
"all_packages": {"opnsense-devel": {"new": "27.1.a_483", "reason": "new", "name": "opnsense-devel", "old": "N/A"}},
"upgrade_packages": [], "new_packages": [{"name": "opnsense-devel", "repository": "OPNsense", "version": "27.1.a_483"}]}
assert engine.target_of(switch, "update") == "27.1.a_483"
assert engine.target_of({**switch, "all_packages": {}}, "update") == "27.1.a_483" # from new_packages
business = {"status": "update", "product": {"product_latest": "26.10"},
"all_packages": {"opnsense-business": {"old": "26.10", "new": "26.10.1", "reason": "upgrade"}}}
assert engine.target_of(business, "update") == "26.10.1"
assert engine.target_of({"all_packages": {"unrelated": {"new": "9"}}, "product": {"product_latest": "26.7.7"}}, "update") == "26.7.7"
# ── 1.29 (S6): window exceptions in the fingerprint, stable account id ────────
async def _activate(client, plan):
@@ -711,3 +725,28 @@ async def test_rows_without_an_account_id_still_work_by_name(client, respx_mock,
assert await engine.may_roll_out(db, "admin") is True
assert await engine.may_roll_out(db, "nobody-here") is False
assert await engine.may_roll_out(db, "admin", uuid.uuid4()) is False # a given id that does not exist is final
@pytest.mark.asyncio
async def test_a_probe_that_breaks_after_the_update_stops_the_rollout(client, respx_mock, clock, monkeypatch):
"""1.32 (B5): the plan's probes are frozen into the run, taken in the baseline and compared in the gate."""
from app.services import diagnostics_service
reachable = {"ok": True}
calls = []
async def portprobe(host, target, port, ipproto, source):
calls.append(host.name)
return {"open": reachable["ok"] or host.name != "c1"}
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, gate_probes=[{"type": "tcp", "target": "10.0.0.5", "port": 443}])
run = await _start(client, plan)
assert (await _run(run["id"])).plan_snapshot["gate_probes"] == [{"type": "tcp", "target": "10.0.0.5", "port": 443}]
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
assert "c1" in calls # baseline
reachable["ok"] = False # the update broke the path to the server
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "gate_failed"
assert paused.waves[0].gate_result["hosts"][str(_host(paused, "c1").host_id)]["probes"]["code"] == "probe_regressed"
assert fleet.fw["ha-b"].update.call_count == 0
+44
View File
@@ -193,3 +193,47 @@ async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeyp
def datetime_now():
from datetime import datetime, timezone
return datetime.now(timezone.utc)
@pytest.mark.asyncio
async def test_probes_from_the_firewall_compare_against_the_baseline(client, respx_mock, monkeypatch):
"""1.32 (B5): reachability probes. Worked before and fails now = fail; failed already before = warn."""
from app.services import audit_service, diagnostics_service
reachable = {"ping 1.1.1.1": True, "tcp 10.0.0.5:443": True, "ping 192.0.2.1": False}
actors = []
async def ping(host, target, ipproto, count, packetsize, source):
actors.append((audit_service.context.get() or {}).get("actor"))
return {"received": count if reachable[f"ping {target}"] else 0}
async def portprobe(host, target, port, ipproto, source):
if target == "broken":
raise diagnostics_service.OpnsenseAPIError("job failed")
return {"open": reachable[f"tcp {target}:{port}"]}
monkeypatch.setattr(diagnostics_service, "ping", ping)
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
Fw(respx_mock, A)
a, _ = await _hosts(client, cluster=False)
probes = [{"type": "ping", "target": "1.1.1.1", "port": None}, {"type": "tcp", "target": "10.0.0.5", "port": 443},
{"type": "ping", "target": "192.0.2.1", "port": None}]
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, uuid.UUID(a["id"]))
base = {a["id"]: await rh.baseline(db, host, probes)}
assert [p["ok"] for p in base[a["id"]]["probes"]] == [True, True, False] and set(actors) == {"system:rollout"}
gate, _ = await _eval([a["id"]], base, probes=probes)
assert gate["hosts"][a["id"]]["probes"] == {"result": "warn", "code": "probe_was_failing", "params": {"probes": "ping 192.0.2.1"}}
reachable["tcp 10.0.0.5:443"] = False
gate, _ = await _eval([a["id"]], base, probes=probes)
assert gate["verdict"] == "fail"
assert gate["hosts"][a["id"]]["probes"] == {"result": "fail", "code": "probe_regressed", "params": {"probes": "tcp 10.0.0.5:443"}}
broken = probes + [{"type": "tcp", "target": "broken", "port": 22}] # cannot run = failed; new since the baseline = worked before
gate, _ = await _eval([a["id"]], base, probes=broken)
assert "tcp broken:22" in gate["hosts"][a["id"]]["probes"]["params"]["probes"]
gate, _ = await _eval([a["id"]], base)
assert "probes" not in gate["hosts"][a["id"]] # a plan without probes runs none
fw_down = Fw(respx_mock, A)
fw_down.up = False
gate, _ = await _eval([a["id"]], base, probes=probes)
assert gate["hosts"][a["id"]]["probes"]["code"] == "unreadable"
+68
View File
@@ -2,6 +2,7 @@
fake firewall (tests/rule_cleanup_fake.py) and the test database. Numbers refer to DESIGN-1.31 §6. Roles and API
tokens are checked in the router (package 6), not here."""
import asyncio
import json
import uuid as uuid_lib
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
@@ -992,3 +993,70 @@ async def test_an_outdated_acknowledgement_text_is_refused(env): # review findi
assert env.fw.calls == [] # refused before anything is read
ok = await preview(env.host_id, [R_BLOCK], acknowledge=[R_BLOCK], ack_text_version=svc.ACK_TEXT_VERSION)
assert ok["plan"]["acknowledged"] == [R_BLOCK]
@pytest.mark.asyncio
async def test_opnexus_own_address_is_protected_even_when_only_workstations_are_configured(env, monkeypatch):
"""B10: the management sources may list only the admins' workstations. A rule that is the only way OPNexus itself
reaches the firewall would then look harmless, and after switching it off the automatic revert could not reach the
firewall any more. The address the firewall sees OPNexus come from is protected as well."""
from app.services import opnsense_client
from app.services.rule_cleanup_service import redact
custom = "aaaaaaaa-0000-4000-8000-000000000920"
env.fw.rules.append(rule(custom, 920, interface="wan", src="198.51.100.0/24", dport="443", description="opnexus server net"))
await _seed_hits(env.host_id, [custom])
ok = await preview(env.host_id, [custom]) # address unknown (respx has no socket): configured sources only
assert ok["plan"]["nodes"][0]["opnexus_source"] == {"address": None, "status": "unknown"}
async def seen_from(_host):
return "198.51.100.7"
monkeypatch.setattr(opnsense_client, "own_source_address", seen_from)
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[custom])
assert got["code"] == "BEREINIGUNG_REGEL_NICHT_AENDERBAR" and got["params"]["reason"] == "management"
assert env.fw.writes() == []
async def configured(_host):
return MGMT_SOURCE
monkeypatch.setattr(opnsense_client, "own_source_address", configured)
p = await preview(env.host_id, [R_UNUSED])
node = p["plan"]["nodes"][0]
assert node["opnexus_source"] == {"address": MGMT_SOURCE, "status": "configured"}
hidden = redact(p)["plan"]["nodes"][0] # an address is security-relevant like the sources: admins only
assert hidden["opnexus_source"] == {"status": "configured"}
@pytest.mark.asyncio
async def test_own_source_address_matches_exactly_this_connection(monkeypatch, respx_mock):
"""The state whose source port is the local port of the request; nothing else (other clients on 443, old states)."""
import httpx as _httpx
from app.models.host import OpnsenseHost
from app.services import opnsense_client, opnsense_secret_store
host = OpnsenseHost(id=uuid_lib.uuid4(), name="fw-src", base_url="https://192.0.2.1", api_key="k", verify_tls=False)
opnsense_secret_store.save_secret(host.id, "s")
respx_mock.get("https://192.0.2.1/api/diagnostics/firewall/pf_states").mock(return_value=_httpx.Response(200, json={"current": 3}))
states = respx_mock.post("https://192.0.2.1/api/diagnostics/firewall/query_states")
rows = [
{"src_addr": "203.0.113.50", "src_port": "40000", "dst_addr": "192.0.2.1", "dst_port": "443", "state": "ESTABLISHED:ESTABLISHED"}, # a browser
{"src_addr": "192.0.2.10", "src_port": "55555", "dst_addr": "192.0.2.1", "dst_port": "443", "state": "TIME_WAIT:TIME_WAIT"}, # old
{"src_addr": "192.0.2.10", "src_port": "33160", "dst_addr": "192.0.2.1", "dst_port": "443", "state": "ESTABLISHED:ESTABLISHED"}, # us
]
states.mock(return_value=_httpx.Response(200, json={"rows": rows}))
addrs = {"server_addr": ("192.0.2.1", 443), "client_addr": ("172.22.0.5", 33160)}
monkeypatch.setattr(opnsense_client, "_stream_addr", lambda _resp, which: addrs.get(which))
assert await opnsense_client.own_source_address(host) == "192.0.2.10"
assert json.loads(states.calls.last.request.content)["searchPhrase"] == "192.0.2.1"
addrs["client_addr"] = ("172.22.0.5", 40001) # NAT rewrote the port: no guessing
assert await opnsense_client.own_source_address(host) is None
addrs["server_addr"] = None # through the overlay's CONNECT proxy
assert await opnsense_client.own_source_address(host) is None
states.mock(return_value=_httpx.Response(403, json={"message": "forbidden"}))
addrs["server_addr"] = ("192.0.2.1", 443)
with pytest.raises(opnsense_client.OpnsenseAPIError):
await opnsense_client.own_source_address(host)
opnsense_secret_store.delete_secret(host.id)
+24 -6
View File
@@ -24,6 +24,7 @@ from app.routers import rule_cleanup as router_mod
from app.schemas.rule_cleanup import CleanupItemRead, CleanupOperationRead
from app.services import api_token_service, notification_checks
from app.services import rule_cleanup_service as svc
from tests.helpers import delete_host
from tests.rule_cleanup_fake import MGMT_SOURCE, R_ALIAS, R_BASE, R_BLOCK, R_DUP, R_MGMT, R_TAIL, R_UNUSED
from tests.test_api_tokens import _bearer, _login, _make
from tests.test_rule_cleanup import ( # noqa: F401 (fixtures are used by name)
@@ -583,7 +584,7 @@ async def test_the_block_covers_the_cluster_and_the_maintenance_inspection(clust
await ensure_no_maintenance(db, [a], cleanup=False) # the cleanup service's own check does not trip over itself
# deleting or re-addressing either node is refused while it is open
for host_id in (cluster.a, cluster.b):
r = await client.delete(f"/api/hosts/{host_id}")
r = await delete_host(client, host_id)
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", host_id
op = (await client.get(_base(cluster.a))).json()[0]
assert (await client.post(f"{_base(cluster.a)}/{op['id']}/revert")).status_code == 200
@@ -596,7 +597,7 @@ async def test_the_block_covers_the_cluster_and_the_maintenance_inspection(clust
async def test_host_delete_and_readdressing_are_refused_while_a_cleanup_is_open(client, env):
hid = env.host_id
await _open(client, env)
gone = await client.delete(f"/api/hosts/{hid}")
gone = await delete_host(client, hid)
assert gone.status_code == 409 and gone.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
for body in ({"base_url": "https://203.0.113.99"}, {"enabled": False}, {"cluster_name": "ha-x"}, {"clear_cluster_name": True}):
r = await client.patch(f"/api/hosts/{hid}", json=body)
@@ -607,7 +608,7 @@ async def test_host_delete_and_readdressing_are_refused_while_a_cleanup_is_open(
assert host.base_url == FW1 and host.enabled and host.name == "fw-a2"
op = (await client.get(_base(env))).json()[0]
assert (await client.post(f"{_base(env)}/{op['id']}/revert")).status_code == 200
assert (await client.delete(f"/api/hosts/{hid}")).status_code == 204 # settled: the host may go (its runs go with it)
assert (await delete_host(client, hid)).status_code == 204 # settled: the host may go (its runs go with it)
async with SessionLocal() as db:
assert (await db.scalars(select(RuleCleanupOperation))).all() == []
@@ -865,11 +866,12 @@ async def test_sources_and_firewall_addresses_are_for_admins_only(client, env, w
# Review finding 9: undeploying a central object and removing a former target write to the firewall too.
# (The VPN undeploy_* are thin wrappers that take the per-object lease; the work and the check live in _undeploy_*.)
_UNDEPLOY_SITES = [
("app.services.alias_deploy_service", "undeploy_alias"), ("app.services.openvpn_deploy_service", "undeploy_instance"),
("app.services.wireguard_deploy_service", "undeploy_tunnel"), ("app.services.port_forward_deploy_service", "undeploy_port_forward"),
("app.services.alias_deploy_service", "undeploy_alias"), ("app.services.openvpn_deploy_service", "_undeploy_instance"),
("app.services.wireguard_deploy_service", "_undeploy_tunnel"), ("app.services.port_forward_deploy_service", "undeploy_port_forward"),
("app.services.outbound_nat_deploy_service", "undeploy_outbound_nat"), ("app.services.one_to_one_deploy_service", "undeploy_one_to_one"),
("app.services.ipsec_deploy_service", "undeploy_tunnel"),
("app.services.ipsec_deploy_service", "_undeploy_tunnel"),
] + [(f"app.routers.{r}", "remove_previous_target") for r in
("aliases", "port_forward", "one_to_one", "outbound_nat", "wireguard", "openvpn", "ipsec")]
@@ -908,3 +910,19 @@ async def test_alias_delete_and_target_removal_wait_for_the_cleanup(client, env)
r = await client.post(f"/api/aliases/{former.json()['id']}/deployments/{env.host_id}/remove", json={"preview_token": preview["token"]})
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
assert env.fw.writes() == writes
async def test_admins_get_their_own_address_as_a_source_suggestion(client):
"""B12: the settings suggest the address the request came from; only to admins (an address is security-relevant
like the sources themselves)."""
from app.main import app
got = (await client.get("/api/settings/governance")).json()
assert got["your_address"] == "127.0.0.1" # ASGI test client
saved = (await client.put("/api/settings/governance", json={"cleanup_management_sources": ["203.0.113.10"]})).json()
assert saved["your_address"] == "127.0.0.1"
assert (await client.post("/api/users", json={"username": "b12-op", "password": "password-123456", "role": "operator"})).status_code == 201
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as op:
await op.post("/api/auth/login", json={"username": "b12-op", "password": "password-123456"})
seen = (await op.get("/api/settings/governance")).json()
assert seen["your_address"] is None and seen["cleanup_management_sources"] == []
+54
View File
@@ -625,3 +625,57 @@ def test_firmware_without_the_automatic_flag_is_not_rejected_as_incomplete():
assert not rule_hit_service.looks_complete({"rows": []})
assert not rule_hit_service.looks_complete({"rows": [{"uuid": "u1", "is_automatic": "0"}]}) # flagged firmware, no automatic rule: partial
assert rule_hit_service.looks_complete({"rows": [{"uuid": "u1", "is_automatic": "0"}, {"uuid": "a", "is_automatic": "1"}]})
# ── B11: counters are read right before every apply that reloads pf ───────────
@pytest.mark.asyncio
async def test_every_reloading_apply_samples_the_counters_first_and_a_failed_sample_never_blocks(client, respx_mock):
"""`apply` (and NAT/alias/VPN/network reconfigure) zeroes all rule counters; hits since the last sample would be
lost. Each of these client calls now reads ruleStats + searchRule right before its own POST."""
from app.services import opnsense_client
host_id = await host_of(client, respx_mock, url=FW)
stats = respx_mock.get(f"{FW}/api/firewall/filter_util/ruleStats").mock(return_value=httpx.Response(200, json={
"status": "ok", "stats": {R1: {"packets": 42, "bytes": 4200, "evaluations": 50, "pf_rules": 1}}}))
respx_mock.get(f"{FW}/api/firewall/filter/searchRule").mock(return_value=httpx.Response(200, json=answer(rule(R1))))
reloads = {
"apply": "/firewall/filter/apply", "apply_port_forward": "/firewall/d_nat/apply",
"apply_one_to_one": "/firewall/one_to_one/apply", "apply_outbound_nat": "/firewall/source_nat/apply",
"reconfigure_aliases": "/firewall/alias/reconfigure", "reconfigure_ipsec": "/ipsec/service/reconfigure",
"reconfigure_openvpn": "/openvpn/service/reconfigure", "reconfigure_wireguard": "/wireguard/service/reconfigure",
}
for path in reloads.values():
respx_mock.post(f"{FW}/api{path}").mock(return_value=httpx.Response(200, json={"status": "ok"}))
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, host_id)
for name, path in reloads.items():
before = len(respx_mock.calls)
await getattr(opnsense_client, name)(host)
order = [c.request.url.path for c in list(respx_mock.calls)[before:]]
assert order[-1] == f"/api{path}", name
assert "/api/firewall/filter_util/ruleStats" in order[:-1] and "/api/firewall/filter/searchRule" in order[:-1], (name, order)
async with SessionLocal() as db:
row = (await db.scalars(select(RuleHit).where(RuleHit.host_id == host_id))).one()
assert row.rule_uuid == R1 and row.last_packets == 42 and row.last_hit_at is not None
stats.mock(return_value=httpx.Response(500, json={})) # the probe fails: the write goes on anyway
respx_mock.get(f"{FW}/api/diagnostics/firewall/pfStatistics/rules").mock(return_value=httpx.Response(500, json={}))
assert (await opnsense_client.apply(host))["status"] == "ok"
@pytest.mark.asyncio
async def test_a_writer_that_just_sampled_itself_does_not_probe_twice(client, respx_mock):
from app.services import opnsense_client
host_id = await host_of(client, respx_mock, url=FW)
stats = respx_mock.get(f"{FW}/api/firewall/filter_util/ruleStats").mock(return_value=httpx.Response(200, json={"status": "ok", "stats": {}}))
respx_mock.get(f"{FW}/api/firewall/filter/searchRule").mock(return_value=httpx.Response(200, json=answer()))
respx_mock.post(f"{FW}/api/firewall/filter/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
async with SessionLocal() as db:
host = await db.get(OpnsenseHost, host_id)
with opnsense_client.counters_already_sampled():
await opnsense_client.apply(host)
assert stats.call_count == 0
await opnsense_client.apply(host)
assert stats.call_count == 1
Loaded 100 of 130 files, more files were not shown because too many files have changed in this diff. Show more