Compare commits

...
10 Commits
Author SHA1 Message Date
Julian Boldt 8128d14da4 Release v1.28.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Übernehmen: mit vorhandenem zentralem Objekt verknüpfen:** Dieselbe von Hand angelegte Regel, derselbe Alias oder dieselbe NAT-Regel auf mehreren
  Firewalls (z. B. beide HA-Knoten) wird nicht mehr als zweites Objekt übernommen. Im Dialog „Von Firewall übernehmen“ erscheint bei exakter
  Übereinstimmung (nach Einsetzen der Host-Variablen, bei Aliasen inkl. Verweisen) die Aktion **Verknüpfen**: die Firewall wird weiteres Ziel des
  vorhandenen Objekts, die Vorschau zeigt „keine Änderung“, und der nächste Rollout aktualisiert alle Firewalls an einem Objekt. Auf der Firewall wird
  dabei nichts geändert. Unterscheidet sich ein gleichnamiges Objekt, zeigt der Dialog die abweichenden Felder; verknüpft wird dann nie. „Alle
  übernehmbaren“ fragt vorher nach („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“). Aliase, die bisher wegen „Name existiert“ abgelehnt wurden,
  lassen sich so übernehmen.
  Wartet das vorhandene Objekt gerade auf eine Bestätigung, wird nicht verknüpft; „Alle übernehmbaren“ lässt den Eintrag dann aus. Verknüpfen
  macht ein gerade geöffnetes Bearbeiten-Formular des Objekts ungültig, damit die neue Firewall nicht versehentlich wieder aus den Zielen fällt.

- **Tooltips überarbeitet:** Der deaktivierte Button „Oberfläche öffnen“ nennt jetzt den Grund im Klartext samt nächstem Schritt (statt nur einer
  Variablen wie `GUI_GATEWAY_TOKEN`); dasselbe gilt für ausgeschaltete Funktion, deaktivierte Firewall, Nicht-HTTPS-Adresse und ausgeschaltete
  Zertifikatsprüfung. Texte durchgehend in Du-Form, kürzere VPN-Einleitungen, verständlichere Hinweise bei gesperrten Buttons (Zertifikate, Benutzer,
  Dienste). Icon-Buttons (Bearbeiten, Löschen, Schließen, Reihenfolge) zeigen beim Darüberfahren ihren Namen.
2026-10-21 23:05:16 +02:00
Julian Boldt 95f297d3f2 Release v1.27.2
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **`opnexus update` prüft den Proxy vor dem Wechsel:** Nach dem Ziehen der Images validiert das Werkzeug die Caddy-Konfiguration des neuen
  Proxy-Images mit den Umgebungsvariablen *dieser* Installation. Ist sie ungültig, bricht das Update ab, bevor etwas gewechselt wurde.
  Der Fall aus 1.27.0 (Compose-Datei mit leerem `OPNEXUS_GUI_SITES`) wird dabei automatisch repariert: Das Werkzeug trägt den Platzhalter
  `OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` ein und sagt es. Installationen, die noch die Compose-Datei aus 1.27.0 haben,
  müssen also nur einmal `opnexus update` ausführen.
- **`opnexus update` erkennt einen abstürzenden Proxy:** Ein Dienst ohne Healthcheck (der Proxy) gilt jetzt erst nach 10 Sekunden
  durchgehender Laufzeit als gesund. Bisher meldete das Update „Fertig“, obwohl der Proxy in einer Neustart-Schleife hing, und das
  automatische Zurückrollen griff nicht.
2026-10-21 21:23:53 +02:00
Julian Boldt ad17dcad25 Release v1.27.1
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Fix (1.27.1): Proxy startete mit der neuen Compose-Datei nicht.** Mit der `docker-compose.yml` aus 1.27.0 gab Compose die Variable
  `OPNEXUS_GUI_SITES` als leeren Text an den Proxy weiter; Caddy nutzt seinen Standardwert aber nur bei *nicht gesetzter* Variable und
  brach mit „server block without any key“ ab, OPNexus war dann nicht mehr erreichbar. Die Compose-Datei setzt jetzt einen gültigen
  Platzhalter als Standard. Betroffen waren nur Installationen, die die neue Compose-Datei aus 1.27.0 übernommen hatten (ohne
  `OPNEXUS_GUI_SITES` in der `.env`); wer nur die Images aktualisiert hatte, war nicht betroffen. Sofort-Abhilfe ohne Update:
  `OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` eintragen und `docker compose up -d proxy` ausführen.
2026-10-21 20:44:14 +02:00
Julian Boldt 5cb2ebe1c6 Release v1.27.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
Vorgesehen als MINOR-Release **1.27.0** „Firewall-Oberfläche über OPNexus“.

- **Firewall-Oberfläche über OPNexus:** Auf der Host-Seite öffnet **Oberfläche öffnen** die Weboberfläche der Firewall
  in einem neuen Tab, auch bei Firewalls hinter NAT im Management-Tunnel. Nur für Admins; jede Firewall läuft in einer
  eigenen Browser-Origin (Platz = Port), Cookies bleiben auf dem Server, das Ticket gilt einmalig 30 Sekunden,
  Inaktivitäts- und Gesamtlimit sind einstellbar, Ticket, Start und Ende stehen im Audit (Art „GUI-Zugriff“).
  Standardmäßig **aus**: braucht das Compose-Profil `gui`, `OPNEXUS_GUI_SITES`, `GUI_PORTS`, `GUI_GATEWAY_TOKEN`,
  `COOKIE_SECURE=true` und den Schalter unter Einstellungen (Handbuch, „Firewall-Oberfläche über OPNexus“).
  Live getestet auf OPNsense 26.7.
- **Einschränkungen:** kein WebSocket, keine automatische Anmeldung, höchstens 4 Plätze gleichzeitig. OPNexus-Adresse
  vorher in die Whitelist des Anmeldeschutzes (sshguard) der Firewall eintragen, sonst können Fehlanmeldungen die
  API-Zugriffe von OPNexus aussperren. Nach einem Neustart des wg-hub braucht der Tunnel 2 bis 4 Minuten.
- **Hinweis für Betreiber mit eigenen Skripten:** Bei HTTPS heißt das Sitzungs-Cookie jetzt `__Host-opnexus-session`
  (bestehende Sitzungen werden ohne Abmelden übernommen), und Cookie-Anfragen an `/api` mit `Sec-Fetch-Site: same-site/cross-site`
  werden abgelehnt. Zugriffe mit API-Token sind nicht betroffen.
2026-10-21 20:14:06 +02:00
Julian Boldt 18d489693d Release v1.26.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
Geplant als MINOR-Release **1.26.0** (Regel-Analyse). Backend, MCP und Oberfläche.

- **Regel-Analyse „Welche Regel greift?“ (API):** `GET /api/hosts/{id}/rule-analysis/match` und
  `GET /api/rule-analysis/match` (ganze Flotte, Auswahl oder Gerätegruppe) beantworten für Quelle, Ziel, Protokoll
  und Port, ob eine Firewall das erste Paket einer neuen Verbindung durchlässt oder blockiert, welche Regel das
  entscheidet und warum (Prüfprotokoll Regel für Regel). Grundlage ist der tatsächlich geladene pf-Regelsatz
  inklusive automatischer Regeln, Floating- und Gruppenregeln, `quick`/letzter Treffer, Negation, verschachtelter
  Aliase (pf-Tabellentest der Firewall), Port-Aliase und Adressfamilien. Jede Antwort trägt eine ehrliche
  Konfidenz **sicher / vermutlich / unbekannt** mit Begründungscodes; was nicht belegbar ist (z. B. `tagged`,
  nicht auflösbare Aliase, geratenes Interface, NAT in der Nähe), wird nie als „sicher“ gemeldet. Nur lesend,
  auch mit `read`-Token nutzbar; es wird nichts gespeichert und nichts an der Firewall geändert.
- **MCP:** neues Werkzeug `rule_match` für KI-Assistenten (gleiche Abfrage, ohne Prüfprotokoll).
- **Seite „Regel-Analyse“ (Regelwerk):** Formular für Hosts oder Gerätegruppe, Quell-/Ziel-IP, Protokoll, Ports,
  Eingangs-Interface („automatisch“ oder Liste) und optional die Ausgangsseite. Die Abfrage steht in der Adresszeile
  (teilbar, wird beim Öffnen sofort ausgeführt). Je Firewall eine Karte mit Erlaubt/Blockiert/Abgewiesen, Sicherheit
  des Urteils, entscheidender Regel und Hinweisen; ein „unbekannt“-Ergebnis erscheint bewusst nicht als Urteil.
  Aufklappbares „Warum?“ zeigt jede geprüfte Regel mit ✓/✗/? je Feld, überstimmte Regeln durchgestrichen. Auch mit
  Nur-Lesen-Rolle nutzbar.
2026-10-21 13:12:11 +02:00
Julian Boldt a9f562eec6 Release v1.25.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).

- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
  `ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
  bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
  gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
  folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
  Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
  (nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
  Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
  Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
  Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
  nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
  eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
  älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
  Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
  zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
  (nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
  der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
  vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
  erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
  sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
  gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
  laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
  sicher, wird aber angekündigt.
- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
  26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
  Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
  Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
  übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
  Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
  Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.
2026-10-21 11:59:41 +02:00
Julian Boldt a3274f64ae Release v1.24.0
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Alarme für Gateways, Ressourcen und Dienste:** Neue Ereignisse in Einstellungen › Benachrichtigungen.
  **Gateway gestört** (standardmäßig **an**) meldet ein Gateway, das OPNsense als „down“ führt oder dessen Paketverlust (ab 20 %) bzw.
  Latenz (ab 500 ms) über der Schwelle liegt; bewusst abgeschaltete und nicht überwachte Gateways bleiben still. **Ressourcen-Schwelle**
  (standardmäßig aus) prüft Festplatte (ab 90 %), Arbeitsspeicher (ab 95 %), Swap (ab 50 %) und CPU-Last (ab 2 pro Thread), **Dienst ausgefallen**
  (standardmäßig aus) eine einstellbare Liste (Unbound, Dnsmasq, DHCP, NTP, OpenVPN, WireGuard, IPsec …) und meldet nur Dienste, die zuvor
  liefen. Alle drei stützen sich auf die Minutenmessung und lösen erst aus, wenn die letzten 3 (Ressourcen: 5) Messungen übereinstimmen, und
  enden erst, wenn alle wieder in Ordnung sind: Einzelspitzen und flackernde Verbindungen schicken nichts. Der Gateway-Status (Verlust, Latenz,
  Abweichung) steht außerdem auf der Host-Seite, als Kachel auf dem Dashboard und als Prometheus-Metriken (`opnexus_host_gateway_*`,
  `opnexus_host_swap_used_percent`, `opnexus_host_service_running`). Die Messung speichert dafür zusätzlich Swap, CPU-Threads, Gateways
  und Dienste in der vorhandenen Messreihe (keine neue Tabelle).
- **Weitere Benachrichtigungskanäle:** ntfy, Gotify, Telegram, Microsoft Teams (Workflows-/Incoming-Webhook) und Matrix, beliebig viele
  je Typ, mit „Testen“-Knopf pro Kanal. Zugangsdaten (Tokens, die Teams-Adresse) liegen wie das SMTP-Passwort in der Geheimnis-Ablage, nie
  in der Datenbank oder in Antworten, und tauchen nicht in Fehlermeldungen oder Logs auf. Webhook und neue Kanäle prüfen die Zieladresse:
  nur http(s), keine Zugangsdaten in der Adresse, keine lokalen/Link-Local-/Metadaten-Adressen; Weiterleitungen werden nicht verfolgt.
  Betrifft die bestehende Webhook-Adresse: `localhost`, Loopback, 169.254.x.x und Zugangsdaten in der Adresse (`https://user:pass@…`)
  werden jetzt abgelehnt. Eine solche gespeicherte Adresse wird nicht mehr beliefert (nur ein Log-Eintrag), und die
  Benachrichtigungs-Einstellungen lassen sich erst wieder speichern, wenn sie korrigiert oder geleert ist (die Meldung nennt den Grund).
  Neue Einstellungen kommen mit Migration 0020.

- **Bestand übernehmen für Aliase und NAT:** Auf den Seiten Aliase und NAT (Port Forward, 1:1, Outbound) gibt es jetzt **Von Firewall übernehmen**
  wie bei den Regeln. Der Dialog zeigt, was von Hand in OPNsense angelegt wurde, einzeln zum Übernehmen oder mit „Alle übernehmbaren“ in einem Schritt;
  verschachtelte Aliase werden in der richtigen Reihenfolge übernommen. Die Übernahme schreibt nichts auf die Firewall: Die Vorschau zeigt direkt
  danach „keine Änderung“, der nächste Rollout aktualisiert dasselbe Objekt (kein Duplikat). Nicht übernommen werden OPNsense-eigene Aliase,
  Typen, die OPNexus nicht kennt (z. B. URL, GeoIP), Aliase mit Verweis auf nicht übernommene Aliase sowie NAT-Regeln, die das Modell verfälschen würden
  (z. B. „Kein Redirect“, verknüpfte Filterregel, mehrere Schnittstellen, „Kein NAT“); die Oberfläche nennt den Grund je Objekt.
  Outbound-NAT lässt sich nur übernehmen, wenn der Modus der Firewall Hybrid oder Manuell ist. Einstellungen, die OPNexus nicht verwaltet (Protokollierung,
  Pool-Optionen, Tags, Kategorien), bleiben unverändert und werden angezeigt.

- **Images für ARM:** Alle Release-Images (backend, frontend, proxy, updater) gibt es jetzt für `linux/amd64` und `linux/arm64`
  (Raspberry Pi 4/5, ARM-Server, Apple-Silicon-Docker).
2026-10-21 01:57:30 +02:00
Julian Boldt 9ba6357193 Release v1.23.11
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Updates per Klick prüfen jetzt ebenfalls die Signatur:** Der Updater-Container enthält `cosign` (feste Version, Prüfsumme beim Bau
  geprüft) und verifiziert die neuen Images vor dem Wechsel, wenn `cosign.pub` im Projektverzeichnis liegt. Das Image wird dadurch größer
  (rund 266 statt 123 MB).
2026-10-20 15:46:34 +02:00
Julian Boldt 23d3a00ae1 Release v1.23.10
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- **Signierte Release-Images:** Die Images (backend, frontend, proxy, updater) werden mit cosign signiert, beginnend mit 1.23.9. Der öffentliche
  Schlüssel liegt als `cosign.pub` im Repository und im Release-Bundle. `opnexus update` prüft die Signaturen vor dem Wechsel, wenn `cosign`
  installiert ist. Anleitung: Handbuch, „Images prüfen“.
2026-10-20 15:38:35 +02:00
Julian Boldt 4379a17b15 Release v1.23.9
Release / verify (push) Canceled after 0s
CI / checks (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / backend (push) Canceled after 0s
Release / test (push) Canceled after 0s
Release / publish (backend) (push) Canceled after 0s
Release / publish (frontend) (push) Canceled after 0s
Release / publish (proxy) (push) Canceled after 0s
Release / publish (updater) (push) Canceled after 0s
- Update-Karte: Der Hinweis für Installationen ohne Release-Images nennt jetzt die tatsächliche Ursache („OPNEXUS_IMAGE_REPO ist nicht gesetzt“)
  statt „baut aus dem Quellcode“.
- `cosign.pub`: öffentlicher Schlüssel für die Signaturen der Release-Images (das Release-Skript kann Images jetzt signieren).
2026-10-20 15:16:51 +02:00
149 changed files with 19342 additions and 268 deletions

No files matched your search

+23
View File
@@ -100,3 +100,26 @@ TRUST_PROXY_HEADERS=false
# Zum Aktivieren (Handbuch „Firewalls hinter NAT“): COMPOSE_PROFILES=overlay
# und den UDP-Port aus dem Internet erreichbar machen.
#OVERLAY_PORT=51820
# ── GUI-Fernzugriff auf verwaltete Firewalls (optional, Standard AUS) ──
# Öffnet die Weboberfläche einer Firewall über OPNexus, auch hinter NAT
# (Management-Tunnel), jeweils in einem eigenen Browser-Ursprung (Slot-Port).
# Dreifach aus, bis alles hier gesetzt ist UND ein Admin den Schalter unter
# Einstellungen einschaltet. Voraussetzungen: HTTPS über den Proxy und
# COOKIE_SECURE=true. Siehe Handbuch, „Firewall-Oberfläche über OPNexus“.
# 1. Profil aktivieren (mit anderen Profilen kommagetrennt, z. B. overlay,gui):
# COMPOSE_PROFILES=gui
# 2. Slot-Ursprünge: dieselbe Adresse wie in OPNEXUS_SITES, je Slot ein Port
# aus GUI_PORTS. Darf NICHT in FRONTEND_ORIGIN stehen (Start wird sonst verweigert).
# OPNEXUS_GUI_SITES=https://10.0.0.5:9441, https://10.0.0.5:9442, https://10.0.0.5:9443, https://10.0.0.5:9444
# 3. Gemeinsames Geheimnis Backend <-> Gateway, mind. 32 Zeichen, z. B.
# `openssl rand -hex 32`:
# GUI_GATEWAY_TOKEN=
# 4. Wo die Slot-Ports lauschen. Standard nur lokal (127.0.0.1); für den
# Zugriff aus dem Netz die Adresse dieses Hosts eintragen (wie bei
# PROXY_BIND_ADDRESS). GUI_PORTS ist ein Bereich, Host- und Container-Port
# sind gleich. GUI_PORTS zum Einschalten SETZEN: ohne den Eintrag nimmt
# Docker zufällige freie Ports (damit ein Update nie an belegten Ports wie
# 9443 scheitert), und die Slot-Adressen oben wären nicht erreichbar.
# GUI_BIND_ADDRESS=127.0.0.1
# GUI_PORTS=9441-9444
+167
View File
@@ -31,6 +31,173 @@ den Einstellungen.
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
ansteht): Merlin …
## [1.28.0] "Sparrowhawk" — 2026-10-21
- **Übernehmen: mit vorhandenem zentralem Objekt verknüpfen:** Dieselbe von Hand angelegte Regel, derselbe Alias oder dieselbe NAT-Regel auf mehreren
Firewalls (z. B. beide HA-Knoten) wird nicht mehr als zweites Objekt übernommen. Im Dialog „Von Firewall übernehmen“ erscheint bei exakter
Übereinstimmung (nach Einsetzen der Host-Variablen, bei Aliasen inkl. Verweisen) die Aktion **Verknüpfen**: die Firewall wird weiteres Ziel des
vorhandenen Objekts, die Vorschau zeigt „keine Änderung“, und der nächste Rollout aktualisiert alle Firewalls an einem Objekt. Auf der Firewall wird
dabei nichts geändert. Unterscheidet sich ein gleichnamiges Objekt, zeigt der Dialog die abweichenden Felder; verknüpft wird dann nie. „Alle
übernehmbaren“ fragt vorher nach („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“). Aliase, die bisher wegen „Name existiert“ abgelehnt wurden,
lassen sich so übernehmen.
Wartet das vorhandene Objekt gerade auf eine Bestätigung, wird nicht verknüpft; „Alle übernehmbaren“ lässt den Eintrag dann aus. Verknüpfen
macht ein gerade geöffnetes Bearbeiten-Formular des Objekts ungültig, damit die neue Firewall nicht versehentlich wieder aus den Zielen fällt.
- **Tooltips überarbeitet:** Der deaktivierte Button „Oberfläche öffnen“ nennt jetzt den Grund im Klartext samt nächstem Schritt (statt nur einer
Variablen wie `GUI_GATEWAY_TOKEN`); dasselbe gilt für ausgeschaltete Funktion, deaktivierte Firewall, Nicht-HTTPS-Adresse und ausgeschaltete
Zertifikatsprüfung. Texte durchgehend in Du-Form, kürzere VPN-Einleitungen, verständlichere Hinweise bei gesperrten Buttons (Zertifikate, Benutzer,
Dienste). Icon-Buttons (Bearbeiten, Löschen, Schließen, Reihenfolge) zeigen beim Darüberfahren ihren Namen.
## [1.27.2] "Buzzard" — 2026-10-21
- **`opnexus update` prüft den Proxy vor dem Wechsel:** Nach dem Ziehen der Images validiert das Werkzeug die Caddy-Konfiguration des neuen
Proxy-Images mit den Umgebungsvariablen *dieser* Installation. Ist sie ungültig, bricht das Update ab, bevor etwas gewechselt wurde.
Der Fall aus 1.27.0 (Compose-Datei mit leerem `OPNEXUS_GUI_SITES`) wird dabei automatisch repariert: Das Werkzeug trägt den Platzhalter
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` ein und sagt es. Installationen, die noch die Compose-Datei aus 1.27.0 haben,
müssen also nur einmal `opnexus update` ausführen.
- **`opnexus update` erkennt einen abstürzenden Proxy:** Ein Dienst ohne Healthcheck (der Proxy) gilt jetzt erst nach 10 Sekunden
durchgehender Laufzeit als gesund. Bisher meldete das Update „Fertig“, obwohl der Proxy in einer Neustart-Schleife hing, und das
automatische Zurückrollen griff nicht.
## [1.27.1] "Buzzard" — 2026-10-21
- **Fix (1.27.1): Proxy startete mit der neuen Compose-Datei nicht.** Mit der `docker-compose.yml` aus 1.27.0 gab Compose die Variable
`OPNEXUS_GUI_SITES` als leeren Text an den Proxy weiter; Caddy nutzt seinen Standardwert aber nur bei *nicht gesetzter* Variable und
brach mit „server block without any key“ ab, OPNexus war dann nicht mehr erreichbar. Die Compose-Datei setzt jetzt einen gültigen
Platzhalter als Standard. Betroffen waren nur Installationen, die die neue Compose-Datei aus 1.27.0 übernommen hatten (ohne
`OPNEXUS_GUI_SITES` in der `.env`); wer nur die Images aktualisiert hatte, war nicht betroffen. Sofort-Abhilfe ohne Update:
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` eintragen und `docker compose up -d proxy` ausführen.
## [1.27.0] "Buzzard" — 2026-10-21
Vorgesehen als MINOR-Release **1.27.0** „Firewall-Oberfläche über OPNexus“.
- **Firewall-Oberfläche über OPNexus:** Auf der Host-Seite öffnet **Oberfläche öffnen** die Weboberfläche der Firewall
in einem neuen Tab, auch bei Firewalls hinter NAT im Management-Tunnel. Nur für Admins; jede Firewall läuft in einer
eigenen Browser-Origin (Platz = Port), Cookies bleiben auf dem Server, das Ticket gilt einmalig 30 Sekunden,
Inaktivitäts- und Gesamtlimit sind einstellbar, Ticket, Start und Ende stehen im Audit (Art „GUI-Zugriff“).
Standardmäßig **aus**: braucht das Compose-Profil `gui`, `OPNEXUS_GUI_SITES`, `GUI_PORTS`, `GUI_GATEWAY_TOKEN`,
`COOKIE_SECURE=true` und den Schalter unter Einstellungen (Handbuch, „Firewall-Oberfläche über OPNexus“).
Live getestet auf OPNsense 26.7.
- **Einschränkungen:** kein WebSocket, keine automatische Anmeldung, höchstens 4 Plätze gleichzeitig. OPNexus-Adresse
vorher in die Whitelist des Anmeldeschutzes (sshguard) der Firewall eintragen, sonst können Fehlanmeldungen die
API-Zugriffe von OPNexus aussperren. Nach einem Neustart des wg-hub braucht der Tunnel 2 bis 4 Minuten.
- **Hinweis für Betreiber mit eigenen Skripten:** Bei HTTPS heißt das Sitzungs-Cookie jetzt `__Host-opnexus-session`
(bestehende Sitzungen werden ohne Abmelden übernommen), und Cookie-Anfragen an `/api` mit `Sec-Fetch-Site: same-site/cross-site`
werden abgelehnt. Zugriffe mit API-Token sind nicht betroffen.
## [1.26.0] "Kite" — 2026-10-21
Geplant als MINOR-Release **1.26.0** (Regel-Analyse). Backend, MCP und Oberfläche.
- **Regel-Analyse „Welche Regel greift?“ (API):** `GET /api/hosts/{id}/rule-analysis/match` und
`GET /api/rule-analysis/match` (ganze Flotte, Auswahl oder Gerätegruppe) beantworten für Quelle, Ziel, Protokoll
und Port, ob eine Firewall das erste Paket einer neuen Verbindung durchlässt oder blockiert, welche Regel das
entscheidet und warum (Prüfprotokoll Regel für Regel). Grundlage ist der tatsächlich geladene pf-Regelsatz
inklusive automatischer Regeln, Floating- und Gruppenregeln, `quick`/letzter Treffer, Negation, verschachtelter
Aliase (pf-Tabellentest der Firewall), Port-Aliase und Adressfamilien. Jede Antwort trägt eine ehrliche
Konfidenz **sicher / vermutlich / unbekannt** mit Begründungscodes; was nicht belegbar ist (z. B. `tagged`,
nicht auflösbare Aliase, geratenes Interface, NAT in der Nähe), wird nie als „sicher“ gemeldet. Nur lesend,
auch mit `read`-Token nutzbar; es wird nichts gespeichert und nichts an der Firewall geändert.
- **MCP:** neues Werkzeug `rule_match` für KI-Assistenten (gleiche Abfrage, ohne Prüfprotokoll).
- **Seite „Regel-Analyse“ (Regelwerk):** Formular für Hosts oder Gerätegruppe, Quell-/Ziel-IP, Protokoll, Ports,
Eingangs-Interface („automatisch“ oder Liste) und optional die Ausgangsseite. Die Abfrage steht in der Adresszeile
(teilbar, wird beim Öffnen sofort ausgeführt). Je Firewall eine Karte mit Erlaubt/Blockiert/Abgewiesen, Sicherheit
des Urteils, entscheidender Regel und Hinweisen; ein „unbekannt“-Ergebnis erscheint bewusst nicht als Urteil.
Aufklappbares „Warum?“ zeigt jede geprüfte Regel mit ✓/✗/? je Feld, überstimmte Regeln durchgestrichen. Auch mit
Nur-Lesen-Rolle nutzbar.
## [1.25.0] "Goshawk" — 2026-10-21
Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).
- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
`ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
(nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
(nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
sicher, wird aber angekündigt.
- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.
## [1.24.0] "Merlin" — 2026-10-21
- **Alarme für Gateways, Ressourcen und Dienste:** Neue Ereignisse in Einstellungen › Benachrichtigungen.
**Gateway gestört** (standardmäßig **an**) meldet ein Gateway, das OPNsense als „down“ führt oder dessen Paketverlust (ab 20 %) bzw.
Latenz (ab 500 ms) über der Schwelle liegt; bewusst abgeschaltete und nicht überwachte Gateways bleiben still. **Ressourcen-Schwelle**
(standardmäßig aus) prüft Festplatte (ab 90 %), Arbeitsspeicher (ab 95 %), Swap (ab 50 %) und CPU-Last (ab 2 pro Thread), **Dienst ausgefallen**
(standardmäßig aus) eine einstellbare Liste (Unbound, Dnsmasq, DHCP, NTP, OpenVPN, WireGuard, IPsec …) und meldet nur Dienste, die zuvor
liefen. Alle drei stützen sich auf die Minutenmessung und lösen erst aus, wenn die letzten 3 (Ressourcen: 5) Messungen übereinstimmen, und
enden erst, wenn alle wieder in Ordnung sind: Einzelspitzen und flackernde Verbindungen schicken nichts. Der Gateway-Status (Verlust, Latenz,
Abweichung) steht außerdem auf der Host-Seite, als Kachel auf dem Dashboard und als Prometheus-Metriken (`opnexus_host_gateway_*`,
`opnexus_host_swap_used_percent`, `opnexus_host_service_running`). Die Messung speichert dafür zusätzlich Swap, CPU-Threads, Gateways
und Dienste in der vorhandenen Messreihe (keine neue Tabelle).
- **Weitere Benachrichtigungskanäle:** ntfy, Gotify, Telegram, Microsoft Teams (Workflows-/Incoming-Webhook) und Matrix, beliebig viele
je Typ, mit „Testen“-Knopf pro Kanal. Zugangsdaten (Tokens, die Teams-Adresse) liegen wie das SMTP-Passwort in der Geheimnis-Ablage, nie
in der Datenbank oder in Antworten, und tauchen nicht in Fehlermeldungen oder Logs auf. Webhook und neue Kanäle prüfen die Zieladresse:
nur http(s), keine Zugangsdaten in der Adresse, keine lokalen/Link-Local-/Metadaten-Adressen; Weiterleitungen werden nicht verfolgt.
Betrifft die bestehende Webhook-Adresse: `localhost`, Loopback, 169.254.x.x und Zugangsdaten in der Adresse (`https://user:pass@…`)
werden jetzt abgelehnt. Eine solche gespeicherte Adresse wird nicht mehr beliefert (nur ein Log-Eintrag), und die
Benachrichtigungs-Einstellungen lassen sich erst wieder speichern, wenn sie korrigiert oder geleert ist (die Meldung nennt den Grund).
Neue Einstellungen kommen mit Migration 0020.
- **Bestand übernehmen für Aliase und NAT:** Auf den Seiten Aliase und NAT (Port Forward, 1:1, Outbound) gibt es jetzt **Von Firewall übernehmen**
wie bei den Regeln. Der Dialog zeigt, was von Hand in OPNsense angelegt wurde, einzeln zum Übernehmen oder mit „Alle übernehmbaren“ in einem Schritt;
verschachtelte Aliase werden in der richtigen Reihenfolge übernommen. Die Übernahme schreibt nichts auf die Firewall: Die Vorschau zeigt direkt
danach „keine Änderung“, der nächste Rollout aktualisiert dasselbe Objekt (kein Duplikat). Nicht übernommen werden OPNsense-eigene Aliase,
Typen, die OPNexus nicht kennt (z. B. URL, GeoIP), Aliase mit Verweis auf nicht übernommene Aliase sowie NAT-Regeln, die das Modell verfälschen würden
(z. B. „Kein Redirect“, verknüpfte Filterregel, mehrere Schnittstellen, „Kein NAT“); die Oberfläche nennt den Grund je Objekt.
Outbound-NAT lässt sich nur übernehmen, wenn der Modus der Firewall Hybrid oder Manuell ist. Einstellungen, die OPNexus nicht verwaltet (Protokollierung,
Pool-Optionen, Tags, Kategorien), bleiben unverändert und werden angezeigt.
- **Images für ARM:** Alle Release-Images (backend, frontend, proxy, updater) gibt es jetzt für `linux/amd64` und `linux/arm64`
(Raspberry Pi 4/5, ARM-Server, Apple-Silicon-Docker).
## [1.23.11] "Harrier" — 2026-10-20
- **Updates per Klick prüfen jetzt ebenfalls die Signatur:** Der Updater-Container enthält `cosign` (feste Version, Prüfsumme beim Bau
geprüft) und verifiziert die neuen Images vor dem Wechsel, wenn `cosign.pub` im Projektverzeichnis liegt. Das Image wird dadurch größer
(rund 266 statt 123 MB).
## [1.23.10] "Harrier" — 2026-10-20
- **Signierte Release-Images:** Die Images (backend, frontend, proxy, updater) werden mit cosign signiert, beginnend mit 1.23.9. Der öffentliche
Schlüssel liegt als `cosign.pub` im Repository und im Release-Bundle. `opnexus update` prüft die Signaturen vor dem Wechsel, wenn `cosign`
installiert ist. Anleitung: Handbuch, „Images prüfen“.
## [1.23.9] "Harrier" — 2026-10-20
- Update-Karte: Der Hinweis für Installationen ohne Release-Images nennt jetzt die tatsächliche Ursache („OPNEXUS_IMAGE_REPO ist nicht gesetzt“)
statt „baut aus dem Quellcode“.
- `cosign.pub`: öffentlicher Schlüssel für die Signaturen der Release-Images (das Release-Skript kann Images jetzt signieren).
## [1.23.8] "Harrier" — 2026-10-20
- **Einstellungen › OPNexus aktualisieren:** Auf einer Installation, die aus dem Quellcode gebaut wird (kein `OPNEXUS_IMAGE_REPO`), erscheint nicht mehr
+8 -5
View File
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
![Zentrale Firewall-Regeln](docs/screenshots/readme-rules-de.webp)
> **Stand:** `1.23.8` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
> **Stand:** `1.28.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
## Funktionen
@@ -16,14 +16,17 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
- **Sichere Änderungen** — jede Änderung läuft *Prüfen → Ausrollen → Bestätigen*; bleibt die Bestätigung im Zeitfenster aus, rollt OPNexus zurück.
- **Hierarchische Policies** — Pre-/Post-Rulebases mit Vererbung über Gruppenbäume.
- **VPN** — IPsec-Site-to-Site, OpenVPN und WireGuard, einmal definiert und ausgerollt.
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln lassen sich übernehmen.
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln, Aliase und NAT-Regeln lassen sich übernehmen.
- **Zertifikate** — Trust-Store für alle Firewalls und Ausstellung per ACME / Let's Encrypt.
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
- **Update-Center** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus.
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend.
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
@@ -58,11 +61,11 @@ Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassis
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
**Fertige Images** (amd64) liegen in der Registry dieses Gitea. In der `.env`:
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
```bash
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
OPNEXUS_VERSION=1.23.8
OPNEXUS_VERSION=1.28.0
```
```bash
+8 -5
View File
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
![Central firewall rules](docs/screenshots/readme-rules-en.webp)
> **Status:** `1.23.8` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
> **Status:** `1.28.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
## What it does
@@ -16,14 +16,17 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
- **Safe changes** — every change goes *preview → deploy → confirm*; without your confirmation inside the time window OPNexus rolls it back.
- **Hierarchical policies** — pre/post rulebases with inheritance across group trees.
- **VPN** — IPsec site-to-site, OpenVPN and WireGuard, defined once and rolled out.
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules created by hand on a firewall can be adopted.
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules, aliases and NAT rules created by hand on a firewall can be adopted.
- **Certificates** — trust store for all firewalls and ACME / Let's Encrypt issuance.
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
- **Update center** — firmware updates for the fleet, aware of HA order and maintenance mode.
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only.
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
@@ -60,11 +63,11 @@ After the first sign-in a short **setup assistant** opens on its own: set your o
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
**Prebuilt images** (amd64) are published to this Gitea's registry. In `.env`:
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
```bash
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
OPNEXUS_VERSION=1.23.8
OPNEXUS_VERSION=1.28.0
```
```bash
+9 -5
View File
@@ -7,7 +7,7 @@ from starlette.responses import JSONResponse
from app.database import SessionLocal
from app.errors import ErrorCode
from app.routers.auth import COOKIE_NAME
from app.routers.auth import session_token
from app.routers import governance
from app.routers.auth import client_ip
from app.services import api_token_service, auth_service
@@ -56,9 +56,9 @@ _READ_ONLY_ALLOWED_WRITES = {
_SECURITY_KINDS = {
"ipsec": "ipsec_tunnel", "openvpn": "openvpn_instance", "wireguard": "wireguard_tunnel",
"certificates": "certificate", "acme": "acme", "notifications": "notification", "auth": "auth",
"tokens": "api_token",
"tokens": "api_token", "rollouts": "rollout",
}
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings"}
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings", "plans", "runs"}
def _is_uuid(value: str) -> bool:
@@ -94,10 +94,14 @@ class AuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
if request.url.path in _PUBLIC_PATHS:
return await call_next(request)
if request.url.path.startswith("/internal/gui/"):
# GUI gateway only: never routed by the proxy, authenticated by the
# shared gateway token inside routers/gui_access.py.
return await call_next(request)
api_token = None
bearer = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
token = request.cookies.get(COOKIE_NAME, "")
token = session_token(request)
if bearer is not None:
# An API token wins over a cookie; a bad one never falls back to it.
async with SessionLocal() as db:
@@ -117,7 +121,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
)
if user is not None:
parts = request.url.path.strip("/").split("/")
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup"}
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
+19
View File
@@ -6,6 +6,17 @@ class Settings(BaseSettings):
def allowed_origins(self) -> list[str]:
return [o.strip().rstrip("/") for o in self.frontend_origin.split(",") if o.strip()]
@property
def gui_site_origins(self) -> list[str]:
return [o.strip().rstrip("/") for o in self.opnexus_gui_sites.split(",") if o.strip()]
def gui_origin_conflicts(self) -> list[str]:
"""Slot origins of the GUI gateway that are also listed in FRONTEND_ORIGIN.
Such an origin would get CORS with credentials and pass the Sec-Fetch-Site
exception — a firewall GUI could then read OPNexus' API. Startup is refused."""
allowed = {o.lower() for o in self.allowed_origins}
return [o for o in self.gui_site_origins if o.lower() in allowed]
database_url: str = (
"postgresql+asyncpg://opnexus:opnexus@postgres:5432/opnexus"
)
@@ -92,6 +103,14 @@ class Settings(BaseSettings):
# per-IP lockout entirely.
trust_proxy_headers: bool = False
# ── GUI gateway (1.27, optional container "gui-gateway") ─────────
# Slot origins the gateway is served on, comma-separated, e.g.
# "https://192.0.2.10:9441,https://192.0.2.10:9442". Empty = feature unavailable.
opnexus_gui_sites: str = ""
# Shared secret between backend and gateway for /internal/gui/* (at least
# 32 characters). Empty = the internal endpoints answer 404.
gui_gateway_token: str = ""
# ── Deploy lockout protection ────────────────────────────────────
deploy_confirm_window_seconds: int = 180
+45
View File
@@ -87,6 +87,13 @@ class ErrorCode(str, Enum):
MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN = "MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN"
NICHT_AUTHENTIFIZIERT = "NICHT_AUTHENTIFIZIERT"
CROSS_ORIGIN_BLOCKED = "CROSS_ORIGIN_BLOCKED"
CROSS_SITE_REQUEST_BLOCKED = "CROSS_SITE_REQUEST_BLOCKED"
GUI_SESSION_REQUIRED = "GUI_SESSION_REQUIRED"
GUI_GATEWAY_NOT_CONFIGURED = "GUI_GATEWAY_NOT_CONFIGURED"
GUI_REQUIRES_HTTPS = "GUI_REQUIRES_HTTPS"
GUI_ACCESS_DISABLED = "GUI_ACCESS_DISABLED"
GUI_HOST_NOT_ELIGIBLE = "GUI_HOST_NOT_ELIGIBLE"
GUI_NO_FREE_SLOT = "GUI_NO_FREE_SLOT"
TLS_PROBE_FAILED = "TLS_PROBE_FAILED"
TLS_PEM_INVALID = "TLS_PEM_INVALID"
TLS_PIN_MISMATCH = "TLS_PIN_MISMATCH"
@@ -116,6 +123,13 @@ class ErrorCode(str, Enum):
NETWORK_FIELD_INVALID = "NETWORK_FIELD_INVALID"
NETWORK_ITEM_NOT_FOUND = "NETWORK_ITEM_NOT_FOUND"
RULE_NOT_ADOPTABLE = "RULE_NOT_ADOPTABLE"
OBJECT_NOT_ADOPTABLE = "OBJECT_NOT_ADOPTABLE"
ADOPT_OBJECT_NOT_FOUND = "ADOPT_OBJECT_NOT_FOUND"
OUTBOUND_MODE_BLOCKS_ADOPT = "OUTBOUND_MODE_BLOCKS_ADOPT"
ADOPT_LINK_TARGET_NOT_FOUND = "ADOPT_LINK_TARGET_NOT_FOUND"
ADOPT_LINK_TARGET_CHANGED = "ADOPT_LINK_TARGET_CHANGED"
ADOPT_LINK_HOST_HAS_OBJECT = "ADOPT_LINK_HOST_HAS_OBJECT"
ADOPT_LINK_BUSY = "ADOPT_LINK_BUSY"
REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0 = "REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0"
RESTORE_VORGANG_NICHT_GEFUNDEN = "RESTORE_VORGANG_NICHT_GEFUNDEN"
ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL = "ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL"
@@ -162,6 +176,11 @@ class ErrorCode(str, Enum):
VARIABLES_MISSING = "VARIABLES_MISSING"
VARIABLE_INVALID = "VARIABLE_INVALID"
OVERLAY_INVALID = "OVERLAY_INVALID"
NOTIFICATION_CHANNEL_INVALID = "NOTIFICATION_CHANNEL_INVALID"
NOTIFICATION_CHANNEL_NOT_FOUND = "NOTIFICATION_CHANNEL_NOT_FOUND"
NOTIFICATION_CHANNEL_NAME_TAKEN = "NOTIFICATION_CHANNEL_NAME_TAKEN"
NOTIFICATION_CHANNEL_TEST_FAILED = "NOTIFICATION_CHANNEL_TEST_FAILED"
NOTIFICATION_WEBHOOK_URL_INVALID = "NOTIFICATION_WEBHOOK_URL_INVALID"
OVERLAY_SUBNET_LOCKED = "OVERLAY_SUBNET_LOCKED"
OVERLAY_NAME_TAKEN = "OVERLAY_NAME_TAKEN"
OVERLAY_PEER_NOT_FOUND = "OVERLAY_PEER_NOT_FOUND"
@@ -171,6 +190,32 @@ class ErrorCode(str, Enum):
API_TOKEN_NOT_FOUND = "API_TOKEN_NOT_FOUND"
API_TOKEN_SCOPE_FORBIDDEN = "API_TOKEN_SCOPE_FORBIDDEN"
API_TOKEN_SCOPE_TOO_HIGH = "API_TOKEN_SCOPE_TOO_HIGH"
WINDOW_NOT_FOUND = "WINDOW_NOT_FOUND"
WINDOW_INVALID = "WINDOW_INVALID"
WINDOW_NAME_TAKEN = "WINDOW_NAME_TAKEN"
WINDOW_IN_USE = "WINDOW_IN_USE"
WINDOW_EXCEPTION_INVALID = "WINDOW_EXCEPTION_INVALID"
WINDOW_EXCEPTION_EXISTS = "WINDOW_EXCEPTION_EXISTS"
WINDOW_EXCEPTION_NOT_FOUND = "WINDOW_EXCEPTION_NOT_FOUND"
ROLLOUT_PLAN_NOT_FOUND = "ROLLOUT_PLAN_NOT_FOUND"
ROLLOUT_INVALID = "ROLLOUT_INVALID"
ROLLOUT_NAME_TAKEN = "ROLLOUT_NAME_TAKEN"
ROLLOUT_PLAN_INVALID = "ROLLOUT_PLAN_INVALID"
ROLLOUT_PREVIEW_STALE = "ROLLOUT_PREVIEW_STALE"
ROLLOUT_PLAN_BUSY = "ROLLOUT_PLAN_BUSY"
ROLLOUT_GROUP_IN_USE = "ROLLOUT_GROUP_IN_USE"
ROLLOUT_RUN_NOT_FOUND = "ROLLOUT_RUN_NOT_FOUND"
ROLLOUT_ALREADY_ACTIVE = "ROLLOUT_ALREADY_ACTIVE"
ROLLOUT_CONFIRM_NAME = "ROLLOUT_CONFIRM_NAME"
ROLLOUT_ACTION_NOT_ALLOWED = "ROLLOUT_ACTION_NOT_ALLOWED"
ROLLOUT_HOST_NOT_IN_RUN = "ROLLOUT_HOST_NOT_IN_RUN"
ROLLOUT_HOST_RUNNING = "ROLLOUT_HOST_RUNNING"
ROLLOUT_HOST_FAILED = "ROLLOUT_HOST_FAILED"
ANALYSE_ADRESSE_UNGUELTIG = "ANALYSE_ADRESSE_UNGUELTIG"
ANALYSE_FAMILIEN_MISCHUNG = "ANALYSE_FAMILIEN_MISCHUNG"
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
+587
View File
@@ -0,0 +1,587 @@
"""The ``gui-gateway`` container: firewall web interfaces in their own browser origin (1.27).
Runs from the backend image (``python -m app.gui_gateway``) behind the proxy, which terminates TLS for the
slot ports and sets ``X-Onx-Slot`` (overwriting any client value). It never touches the database or the
secret stores; it only talks to the backend's internal API (``/internal/gui/*``, shared GUI_GATEWAY_TOKEN)
and, per gateway session, to exactly one firewall.
Security model (see the 1.27 design):
* Every slot port is its own origin. The firewall GUI therefore never runs in the OPNexus origin.
* Cookies never cross: the browser's ``Cookie`` header is not forwarded (it carries the OPNexus session),
and the firewall's ``Set-Cookie`` never reaches the browser — a server-side jar per gateway session
(memory only) holds them. The browser only gets ``__Host-onx-gw-<slot>``.
* Each upstream request is built from scratch: method, path and query from the browser, a small header
allow-list, the jar's cookies; target origin, pinned certificate and overlay proxy from the backend.
Absolute-form targets, CONNECT/TRACE, ``Upgrade`` and service-worker script requests are refused.
* ``Sec-Fetch-Site`` must be same-origin/none (same-site only for the redeem step), fail-closed;
non-GET/HEAD requests also need the slot's own ``Origin``.
* ``Location`` pointing anywhere but the firewall is dropped; responses get CSP, X-Frame-Options DENY,
``Cache-Control: no-store``.
* At most 3 failed GUI logins per firewall in 5 minutes (protects OPNexus' API access from the firewall's
login lockout, which would see all attempts coming from OPNexus).
* Idle and absolute limits; every 20 s the backend re-checks the session (OPNexus session, admin role,
switch, host). Logs carry method, path (no query), status and duration only.
OPNsense behaviour this relies on (opnsense/core, src/www/authgui.inc, master as of 2026-10-21):
* session_auth(): a login is ``isset($_POST['login']) && !empty($_POST['usernamefld']) &&
!empty($_POST['passwordfld'])`` posted to the current URL (login form has no action). Success sends a
``Location:`` header and exits (302); a failure re-renders the login form (200) with "Wrong username or
password." and an auth_log entry.
* check_security_http_referer_enforcement() only runs when HTTP_REFERER is set, and wants the firewall's
own host name/IP — so the gateway sends no Referer (and no Origin) upstream at all.
* check_security_dns_rebind() accepts an IP address or the configured host name as Host — the gateway
sends the Host of the host record's base URL (httpx derives it from the URL).
"""
import asyncio
import hashlib
import logging
import os
import secrets
import ssl
import time
from collections import deque
from dataclasses import dataclass, field
from http.cookies import CookieError, SimpleCookie
from typing import Awaitable, Callable
from urllib.parse import parse_qs, urlsplit
import httpx
logger = logging.getLogger("gui_gateway")
# httpx/httpcore log full request URLs (query strings) at INFO/DEBUG: never in this process.
for _noisy in ("httpx", "httpcore"):
logging.getLogger(_noisy).setLevel(logging.WARNING)
LISTEN_PORT = 8080
COOKIE_PREFIX = "__Host-onx-gw-"
OPNEXUS_SESSION_COOKIE = "__Host-opnexus-session"
BODY_LIMIT = 64 * 1024 * 1024 # config restore uploads
LOGIN_FAILURE_LIMIT = 3
LOGIN_FAILURE_WINDOW = 300.0
UPSTREAM_TIMEOUT = 60.0
MAX_SESSIONS = 64
# Request headers copied from the browser; everything else (Cookie, Authorization, Origin, Referer,
# X-Forwarded-*, hop-by-hop, Sec-*, ...) is dropped.
_FORWARD_REQUEST_HEADERS = {
"accept", "accept-language", "content-type", "user-agent", "x-requested-with", "x-csrftoken", "range",
}
# Response headers copied from the firewall. Set-Cookie goes to the jar, Location is rewritten.
_FORWARD_RESPONSE_HEADERS = {
"content-type", "content-length", "content-encoding", "content-disposition", "content-language",
"content-range", "accept-ranges", "content-security-policy",
}
_GATEWAY_CSP = ("connect-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'self'; "
"object-src 'none'; worker-src 'none'")
_SECURITY_HEADERS = [
(b"content-security-policy", _GATEWAY_CSP.encode()),
(b"x-frame-options", b"DENY"),
(b"x-content-type-options", b"nosniff"),
(b"cache-control", b"no-store"),
(b"pragma", b"no-cache"),
(b"referrer-policy", b"same-origin"),
(b"cross-origin-opener-policy", b"same-origin"),
(b"cross-origin-resource-policy", b"same-origin"),
(b"permissions-policy", b"camera=(), microphone=(), geolocation=(), payment=(), usb=()"),
]
def _sha(value: str) -> str:
return hashlib.sha256(value.encode()).hexdigest()
def _origin_of(url: str) -> str | None:
try:
parts = urlsplit(url)
port = parts.port
except ValueError:
return None
if parts.scheme not in ("http", "https") or not parts.hostname:
return None
default = 443 if parts.scheme == "https" else 80
host = parts.hostname.lower()
if ":" in host:
host = f"[{host}]"
return f"{parts.scheme}://{host}" + (f":{port}" if port and port != default else "")
def slot_origins_from(sites: str) -> dict[int, set[str]]:
"""OPNEXUS_GUI_SITES -> {port: {normalised origins}} (https only, explicit port)."""
slots: dict[int, set[str]] = {}
for raw in sites.split(","):
raw = raw.strip().rstrip("/")
try:
parts = urlsplit(raw)
port = parts.port
except ValueError:
continue
if parts.scheme == "https" and parts.hostname and port:
slots.setdefault(port, set()).add(_origin_of(raw))
return slots
@dataclass
class GatewaySession:
gui_session_id: str
session_hash: str
slot: int
upstream_origin: str
ca_pem: str | None
proxy_url: str | None
idle_timeout: float
absolute_deadline: float
revalidate_every: float
last_activity: float = field(default_factory=time.monotonic)
next_check: float = 0.0
jar: dict[str, str] = field(default_factory=dict)
client: httpx.AsyncClient | None = None
class BackendClient:
"""The backend's internal GUI API. Raises httpx.HTTPError when the backend is unreachable."""
def __init__(self, base_url: str, token: str, transport: httpx.AsyncBaseTransport | None = None):
self._client = httpx.AsyncClient(base_url=base_url, headers={"X-Onx-Gateway-Token": token},
timeout=10.0, transport=transport)
async def redeem(self, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
resp = await self._client.post("/internal/gui/redeem", json={
"ticket_hash": ticket_hash, "session_hash": session_hash, "slot": slot, "client_ip": client_ip})
if resp.status_code == 200:
return resp.json()
if resp.status_code == 403:
return {"ok": False, "reason": (resp.json().get("detail") or {}).get("reason", "denied")}
resp.raise_for_status()
return {"ok": False, "reason": f"http_{resp.status_code}"}
async def revalidate(self, gs: GatewaySession) -> dict:
resp = await self._client.post("/internal/gui/revalidate",
json={"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash})
resp.raise_for_status()
return resp.json()
async def end(self, gs: GatewaySession, reason: str) -> None:
try:
await self._client.post("/internal/gui/end", json={
"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash, "reason": reason})
except httpx.HTTPError as exc:
logger.warning("Could not report the end of a GUI session (%s)", type(exc).__name__)
async def aclose(self) -> None:
await self._client.aclose()
def build_upstream_client(gs: GatewaySession) -> httpx.AsyncClient:
"""Same transport rules as opnsense_client._client: pinned certificate (or system CAs), overlay proxy."""
verify: ssl.SSLContext | bool
if gs.ca_pem:
from app.services.tls_pin_service import pinned_context # no database layer behind this import
verify = pinned_context(gs.ca_pem)
else:
verify = True
return httpx.AsyncClient(verify=verify, proxy=gs.proxy_url, timeout=UPSTREAM_TIMEOUT,
follow_redirects=False, trust_env=False)
class LoginLimiter:
"""At most LOGIN_FAILURE_LIMIT failed GUI logins per firewall within LOGIN_FAILURE_WINDOW seconds."""
def __init__(self, clock: Callable[[], float] = time.monotonic):
self._failures: dict[str, deque] = {}
self._clock = clock
def _recent(self, key: str) -> deque:
now = self._clock()
q = self._failures.setdefault(key, deque())
while q and now - q[0] > LOGIN_FAILURE_WINDOW:
q.popleft()
return q
def blocked(self, key: str) -> bool:
return len(self._recent(key)) >= LOGIN_FAILURE_LIMIT
def record_failure(self, key: str) -> None:
self._recent(key).append(self._clock())
def is_login_attempt(method: str, content_type: str, body: bytes) -> bool:
if method != "POST":
return False
if content_type.lower().startswith("multipart/form-data"):
# PHP fills $_POST from multipart bodies too, so authgui.inc accepts a login posted this way.
return all(f'name="{field}"'.encode() in body for field in ("login", "usernamefld", "passwordfld"))
if not content_type.lower().startswith("application/x-www-form-urlencoded"):
return False
try:
form = parse_qs(body.decode("latin-1"), keep_blank_values=True)
except ValueError:
return False
return "login" in form and bool(form.get("usernamefld", [""])[0]) and bool(form.get("passwordfld", [""])[0])
def rewrite_location(location: str, upstream_origin: str) -> str | None:
"""Relative target for the browser, or None when the redirect leaves the firewall."""
location = location.strip()
if not location or "\\" in location or any(ord(c) < 0x20 for c in location):
return None
if location.startswith("//"):
location = "https:" + location
parts = urlsplit(location)
if parts.scheme or parts.netloc:
if _origin_of(location) != _origin_of(upstream_origin):
return None
path = parts.path or "/"
return path + (f"?{parts.query}" if parts.query else "") + (f"#{parts.fragment}" if parts.fragment else "")
return location
def _page(title: str, text_de: str, text_en: str) -> bytes:
return (f"<!doctype html><html lang=\"de\"><head><meta charset=\"utf-8\"><title>{title}</title></head>"
f"<body style=\"font-family:sans-serif;max-width:40rem;margin:3rem auto\"><h1>{title}</h1>"
f"<p>{text_de}</p><p lang=\"en\">{text_en}</p></body></html>").encode()
_SESSION_GONE = _page("OPNexus", "Diese Sitzung ist abgelaufen oder ungültig. Öffne die Oberfläche erneut aus OPNexus.",
"This session has expired or is invalid. Open the interface again from OPNexus.")
_FORBIDDEN = _page("OPNexus", "Diese Anfrage wurde vom GUI-Gateway abgelehnt.", "This request was refused by the GUI gateway.")
_BAD_REDIRECT = _page("OPNexus", "Die Firewall wollte auf eine fremde Adresse umleiten; die Umleitung wurde verworfen.",
"The firewall tried to redirect to a foreign address; the redirect was dropped.")
_LOGIN_LOCKED = _page("OPNexus", "Zu viele fehlgeschlagene Anmeldungen an dieser Firewall. Bitte warte 5 Minuten.",
"Too many failed sign-ins at this firewall. Please wait 5 minutes.")
_UNAVAILABLE = _page("OPNexus", "Die Firewall oder OPNexus ist gerade nicht erreichbar.",
"The firewall or OPNexus is not reachable right now.")
_REDEEMED = (b"<!doctype html><html><head><meta charset=\"utf-8\"><meta http-equiv=\"refresh\" content=\"0;url=/\">"
b"<title>OPNexus</title></head><body><a href=\"/\">&rarr;</a></body></html>")
class GuiGateway:
def __init__(self, *, backend: BackendClient, slot_origins: dict[int, set[str]],
upstream_factory: Callable[[GatewaySession], httpx.AsyncClient] = build_upstream_client,
clock: Callable[[], float] = time.monotonic, wall_clock: Callable[[], float] = time.time):
self.backend = backend
self.slot_origins = slot_origins
self.upstream_factory = upstream_factory
self.clock = clock
self.wall_clock = wall_clock
self.sessions: dict[str, GatewaySession] = {} # sha256(gateway cookie) -> session
self.limiter = LoginLimiter(clock)
self._loop_task: asyncio.Task | None = None
# ── ASGI ──────────────────────────────────────────────────────────────
async def __call__(self, scope, receive, send):
if scope["type"] == "lifespan":
await self._lifespan(receive, send)
return
if scope["type"] != "http":
# WebSocket and anything else: not supported (design: no WebSocket).
if scope["type"] == "websocket":
await send({"type": "websocket.close", "code": 1008})
return
started = time.monotonic()
status = await self._handle(scope, receive, send)
# Method, path and status only: never the query string, headers or bodies.
logger.info("%s %s %d %.0fms", scope.get("method", "?"), scope.get("path", "?")[:200], status,
(time.monotonic() - started) * 1000)
async def _lifespan(self, receive, send):
while True:
message = await receive()
if message["type"] == "lifespan.startup":
self._loop_task = asyncio.create_task(self._revalidate_loop())
await send({"type": "lifespan.startup.complete"})
elif message["type"] == "lifespan.shutdown":
if self._loop_task:
self._loop_task.cancel()
for key in list(self.sessions):
await self._drop(key, "shutdown")
await self.backend.aclose()
await send({"type": "lifespan.shutdown.complete"})
return
async def _revalidate_loop(self):
while True:
await asyncio.sleep(10)
try:
await self.sweep()
except Exception: # never let the loop die
logger.exception("GUI session sweep failed")
async def sweep(self) -> None:
"""Idle/absolute limits and the periodic backend re-check for every gateway session."""
for key, gs in list(self.sessions.items()):
reason = self._expired(gs)
if reason:
await self._drop(key, reason)
elif self.clock() >= gs.next_check:
await self._recheck(key, gs)
# ── helpers ───────────────────────────────────────────────────────────
@staticmethod
def _headers(scope) -> dict[str, str]:
result: dict[str, str] = {}
for k, v in scope.get("headers", []):
name = k.decode("latin-1").lower()
value = v.decode("latin-1")
if name in result:
value = f"{result[name]}{'; ' if name == 'cookie' else ', '}{value}"
result[name] = value
return result
@staticmethod
def _cookies(header: str) -> dict[str, str]:
cookies: dict[str, str] = {}
for part in header.split(";"):
name, sep, value = part.strip().partition("=")
if sep and name and name not in cookies:
cookies[name] = value
return cookies
async def _respond(self, send, status: int, body: bytes = b"", headers: list | None = None,
content_type: bytes = b"text/html; charset=utf-8") -> int:
out = [(b"content-type", content_type), (b"content-length", str(len(body)).encode())]
out += list(headers or []) + _SECURITY_HEADERS
await send({"type": "http.response.start", "status": status, "headers": out})
await send({"type": "http.response.body", "body": body})
return status
def _expired(self, gs: GatewaySession) -> str | None:
if self.wall_clock() >= gs.absolute_deadline:
return "absolute"
if self.clock() - gs.last_activity >= gs.idle_timeout:
return "idle"
return None
async def _drop(self, key: str, reason: str, *, report: bool = True) -> None:
gs = self.sessions.pop(key, None)
if gs is None:
return
gs.jar.clear()
if gs.client is not None:
await gs.client.aclose()
if report:
await self.backend.end(gs, reason)
async def _recheck(self, key: str, gs: GatewaySession) -> bool | None:
"""True = still valid, False = ended (dropped), None = backend unreachable (fail closed, keep for retry)."""
try:
result = await self.backend.revalidate(gs)
except httpx.HTTPError as exc:
logger.warning("GUI session re-check failed (%s)", type(exc).__name__)
return None
if not result.get("ok"):
await self._drop(key, str(result.get("reason", "revoked")), report=False) # backend already ended it
return False
gs.absolute_deadline = min(gs.absolute_deadline, float(result.get("absolute_deadline", gs.absolute_deadline)))
gs.idle_timeout = float(result.get("idle_timeout_seconds", gs.idle_timeout))
gs.next_check = self.clock() + gs.revalidate_every
return True
# ── request handling ──────────────────────────────────────────────────
async def _handle(self, scope, receive, send) -> int:
method = scope.get("method", "")
path = scope.get("path", "")
raw_path = scope.get("raw_path") or path.encode()
headers = self._headers(scope)
if method in ("CONNECT", "TRACE"):
return await self._respond(send, 501, _FORBIDDEN)
# Origin-form only: "GET http://elsewhere/ HTTP/1.1" (absolute-form) or "//host" never reach upstream.
if not path.startswith("/") or path.startswith("//") or not raw_path.startswith(b"/") \
or raw_path.startswith(b"//") or "\\" in path:
return await self._respond(send, 400, _FORBIDDEN)
if "upgrade" in headers or "upgrade" in headers.get("connection", "").lower():
return await self._respond(send, 400, _FORBIDDEN)
if headers.get("service-worker", "").lower() == "script":
return await self._respond(send, 403, _FORBIDDEN)
try:
slot = int(headers.get("x-onx-slot", ""))
except ValueError:
slot = 0
own_origins = self.slot_origins.get(slot)
if not own_origins:
return await self._respond(send, 400, _FORBIDDEN) # not coming through the proxy's slot block
site = headers.get("sec-fetch-site", "").strip().lower()
if path == "/__onx/redeem":
if method != "GET" or site not in ("same-site", "same-origin", "none"):
return await self._respond(send, 403, _FORBIDDEN)
return await self._redeem(scope, headers, slot, send)
if site not in ("same-origin", "none"):
return await self._respond(send, 403, _FORBIDDEN) # missing header = old/odd client: fail closed
if method not in ("GET", "HEAD"):
origin = headers.get("origin", "")
if _origin_of(origin) not in own_origins:
return await self._respond(send, 403, _FORBIDDEN)
if path.startswith("/__onx/"):
return await self._respond(send, 404, _FORBIDDEN)
cookies = self._cookies(headers.get("cookie", ""))
token = cookies.get(f"{COOKIE_PREFIX}{slot}", "")
key = _sha(token) if token else ""
gs = self.sessions.get(key) if key else None
if gs is None or gs.slot != slot:
return await self._respond(send, 401, _SESSION_GONE)
reason = self._expired(gs)
if reason:
await self._drop(key, reason)
return await self._respond(send, 401, _SESSION_GONE)
if self.clock() >= gs.next_check:
state = await self._recheck(key, gs)
if state is False:
return await self._respond(send, 401, _SESSION_GONE)
if state is None:
return await self._respond(send, 503, _UNAVAILABLE)
gs.last_activity = self.clock()
return await self._proxy(scope, receive, send, headers, gs)
async def _redeem(self, scope, headers, slot: int, send) -> int:
query = parse_qs(scope.get("query_string", b"").decode("latin-1"))
ticket = (query.get("t") or [""])[0]
browser_session = self._cookies(headers.get("cookie", "")).get(OPNEXUS_SESSION_COOKIE, "")
if not ticket or len(ticket) > 128 or not browser_session:
return await self._respond(send, 403, _SESSION_GONE)
client_ip = (headers.get("x-forwarded-for", "").split(",")[0].strip() or None)
try:
result = await self.backend.redeem(_sha(ticket), _sha(browser_session), slot, client_ip and client_ip[:45])
except httpx.HTTPError as exc:
logger.warning("Ticket redeem failed (%s)", type(exc).__name__)
return await self._respond(send, 503, _UNAVAILABLE)
if not result.get("ok"):
return await self._respond(send, 403, _SESSION_GONE)
# One browser session, one gateway session per slot: an older one in this slot is replaced.
for old_key, old in list(self.sessions.items()):
if old.slot == slot and old.session_hash == _sha(browser_session):
await self._drop(old_key, "superseded", report=False)
if len(self.sessions) >= MAX_SESSIONS:
oldest = min(self.sessions, key=lambda k: self.sessions[k].last_activity)
await self._drop(oldest, "superseded")
value = secrets.token_urlsafe(32)
gs = GatewaySession(
gui_session_id=result["gui_session_id"], session_hash=_sha(browser_session), slot=slot,
upstream_origin=result["upstream_origin"], ca_pem=result.get("ca_pem"), proxy_url=result.get("proxy_url"),
idle_timeout=float(result["idle_timeout_seconds"]), absolute_deadline=float(result["absolute_deadline"]),
revalidate_every=float(result.get("revalidate_seconds", 20)),
)
gs.last_activity = self.clock()
gs.next_check = gs.last_activity + gs.revalidate_every
self.sessions[_sha(value)] = gs
cookie = f"{COOKIE_PREFIX}{slot}={value}; Path=/; Secure; HttpOnly; SameSite=Strict"
# 200 + meta refresh instead of a 303: the follow-up request then starts in this origin
# (Sec-Fetch-Site: same-origin) and the ticket leaves the address bar.
return await self._respond(send, 200, _REDEEMED, headers=[
(b"set-cookie", cookie.encode()), (b"clear-site-data", b'"cache", "storage"'),
])
async def _read_body(self, receive) -> bytes | None:
chunks, size = [], 0
while True:
message = await receive()
if message["type"] == "http.disconnect":
return None
chunk = message.get("body", b"")
size += len(chunk)
if size > BODY_LIMIT:
return None
chunks.append(chunk)
if not message.get("more_body"):
return b"".join(chunks)
def _store_cookies(self, gs: GatewaySession, response: httpx.Response) -> None:
for header in response.headers.get_list("set-cookie"):
jar = SimpleCookie()
try:
jar.load(header)
except CookieError:
continue
for name, morsel in jar.items():
expired = morsel["max-age"] in ("0",) or morsel.value in ("", "deleted")
if not expired and morsel["expires"]:
try:
from email.utils import parsedate_to_datetime
expired = parsedate_to_datetime(morsel["expires"]).timestamp() <= self.wall_clock()
except (TypeError, ValueError):
pass
if expired:
gs.jar.pop(name, None)
else:
gs.jar[name] = morsel.value
async def _proxy(self, scope, receive, send, headers: dict, gs: GatewaySession) -> int:
length = headers.get("content-length", "")
if length.isdigit() and int(length) > BODY_LIMIT:
return await self._respond(send, 413, _FORBIDDEN)
body = await self._read_body(receive)
if body is None:
return await self._respond(send, 413, _FORBIDDEN)
method = scope["method"]
login = is_login_attempt(method, headers.get("content-type", ""), body)
if login and self.limiter.blocked(gs.upstream_origin):
return await self._respond(send, 429, _LOGIN_LOCKED)
out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS}
out_headers["accept-encoding"] = "identity"
if gs.jar:
out_headers["cookie"] = "; ".join(f"{k}={v}" for k, v in gs.jar.items())
query = scope.get("query_string", b"").decode("latin-1")
# The path exactly as the browser sent it (still percent-encoded): the decoded scope["path"] would
# turn %3F/%23 into query/fragment, %2F into a separator and decode %25xx a second time upstream.
raw_path = (scope.get("raw_path") or scope["path"].encode()).split(b"?", 1)[0].decode("latin-1")
url = gs.upstream_origin + raw_path + (f"?{query}" if query else "")
if gs.client is None:
gs.client = self.upstream_factory(gs)
try:
request = gs.client.build_request(method, url, headers=out_headers, content=body)
except httpx.InvalidURL:
return await self._respond(send, 400, _FORBIDDEN) # e.g. control characters in the path
try:
response = await gs.client.send(request, stream=True)
except (httpx.HTTPError, ssl.SSLError, OSError) as exc:
logger.warning("Upstream request failed (%s)", type(exc).__name__)
return await self._respond(send, 502, _UNAVAILABLE)
try:
self._store_cookies(gs, response)
if login and not (300 <= response.status_code < 400):
self.limiter.record_failure(gs.upstream_origin)
out = []
for name, value in response.headers.multi_items():
lname = name.lower()
if lname in _FORWARD_RESPONSE_HEADERS:
out.append((lname.encode("latin-1"), value.encode("latin-1")))
if "location" in response.headers:
target = rewrite_location(response.headers["location"], gs.upstream_origin)
if target is None:
await response.aclose()
return await self._respond(send, 502, _BAD_REDIRECT)
out.append((b"location", target.encode("latin-1")))
await send({"type": "http.response.start", "status": response.status_code, "headers": out + _SECURITY_HEADERS})
async for chunk in response.aiter_raw():
await send({"type": "http.response.body", "body": chunk, "more_body": True})
await send({"type": "http.response.body", "body": b""})
return response.status_code
finally:
await response.aclose()
def create_app(transport: httpx.AsyncBaseTransport | None = None) -> GuiGateway:
token = os.environ.get("GUI_GATEWAY_TOKEN", "")
if len(token) < 32:
raise SystemExit("GUI_GATEWAY_TOKEN (at least 32 characters) is required")
slots = slot_origins_from(os.environ.get("OPNEXUS_GUI_SITES", ""))
if not slots:
raise SystemExit("OPNEXUS_GUI_SITES must list the https slot origins")
backend = BackendClient(os.environ.get("GUI_BACKEND_URL", "http://backend:8000"), token, transport)
return GuiGateway(backend=backend, slot_origins=slots)
def main() -> None:
import uvicorn
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
# h11: an absolute-form target stays visible in scope["path"] (and is refused); access_log off because
# uvicorn's access log would contain query strings (the redeem ticket).
uvicorn.run(create_app(), host="0.0.0.0", port=LISTEN_PORT, http="h11", ws="none", access_log=False,
server_header=False, date_header=False, proxy_headers=False, lifespan="on", log_config=None)
if __name__ == "__main__":
main()
+22
View File
@@ -46,10 +46,16 @@ from app.version import APP_VERSION
from app.routers import dashboard as dashboard_router
from app.routers import cluster as cluster_router
from app.routers import services as services_router
from app.routers import gateways as gateways_router
from app.routers import resource_samples as resource_samples_router
from app.services import resource_monitor_service
from app.routers import scheduled_maintenance as scheduled_maintenance_router
from app.routers import maintenance_windows as maintenance_windows_router
from app.routers import rollouts as rollouts_router
from app.routers import rule_analysis as rule_analysis_router
from app.routers import gui_access as gui_access_router
from app.services import scheduled_maintenance_service
from app.services import rollout_engine
from app.services import compliance_service
from app.services import audit_export_service
from app.services import overlay_service, variables_service
@@ -254,6 +260,12 @@ async def _scheduled_maintenance_poll_loop():
except Exception:
logger.exception("Geplante-Wartung-Poll fehlgeschlagen")
in_progress = 0
# Firmware rollouts (1.25) only ever create schedules; the next tick of the lines above fires them.
try:
async with SessionLocal() as db:
in_progress += await rollout_engine.advance_runs(db)
except Exception:
logger.exception("Rollout-Poll fehlgeschlagen")
# A handover run is stepped quickly (the partner should take over within a minute or two);
# otherwise the relaxed cadence is plenty.
await asyncio.sleep(_RUNNING_MAINTENANCE_POLL_SECONDS if in_progress else _SCHEDULED_MAINTENANCE_POLL_SECONDS)
@@ -312,6 +324,10 @@ async def _backup_poll_loop():
@asynccontextmanager
async def lifespan(app: FastAPI):
conflicts = settings.gui_origin_conflicts()
if conflicts:
# A GUI-gateway slot must never be a trusted OPNexus origin (CORS with credentials).
raise RuntimeError(f"OPNEXUS_GUI_SITES overlaps FRONTEND_ORIGIN: {', '.join(conflicts)} — refusing to start")
await migrate.run()
async with SessionLocal() as db:
await auth_service.bootstrap_admin(db)
@@ -510,9 +526,15 @@ app.include_router(restore_router.router, prefix="/api")
app.include_router(dashboard_router.router, prefix="/api")
app.include_router(cluster_router.router, prefix="/api")
app.include_router(services_router.router, prefix="/api")
app.include_router(gateways_router.router, prefix="/api")
app.include_router(network_router.router, prefix="/api")
app.include_router(snapshots_router.router, prefix="/api")
app.include_router(network_drift_router.router, prefix="/api")
app.include_router(diagnostics_router.router, prefix="/api")
app.include_router(resource_samples_router.router, prefix="/api")
app.include_router(scheduled_maintenance_router.router, prefix="/api")
app.include_router(maintenance_windows_router.router, prefix="/api")
app.include_router(rollouts_router.router, prefix="/api")
app.include_router(rule_analysis_router.router, prefix="/api")
app.include_router(gui_access_router.router, prefix="/api")
app.include_router(gui_access_router.internal_router)
+9
View File
@@ -93,3 +93,12 @@ __all__ += ["HostResourceSample"]
from app.models.scheduled_maintenance import ScheduledMaintenance
__all__ += ["ScheduledMaintenance"]
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
__all__ += ["MaintenanceWindow", "MaintenanceWindowException"]
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun, RolloutRunHost, RolloutRunWave, rollout_wave_groups
__all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "RolloutRunWave", "rollout_wave_groups"]
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
__all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
+65
View File
@@ -0,0 +1,65 @@
"""GUI access to managed firewalls through the separate gui-gateway container (1.27).
Settings are a singleton row (id fixed at 1), off by default. Tickets and gateway
sessions only ever store hashes: the raw ticket travels once through the browser,
the OPNexus session is referenced by the same SHA-256 that `sessions.token_hash` holds.
"""
import uuid
from datetime import datetime
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String
from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column
from app.database import Base
class GuiAccessSettings(Base):
__tablename__ = "gui_access_settings"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
idle_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=15, server_default="15")
absolute_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
slot_count: Mapped[int] = mapped_column(Integer, nullable=False, default=4, server_default="4")
updated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class GuiTicket(Base):
"""One-time, 30-second ticket, bound to the OPNexus session, host and slot."""
__tablename__ = "gui_tickets"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
ticket_hash: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
user_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
)
host_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
)
slot: Mapped[int] = mapped_column(Integer, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
redeemed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class GuiSession(Base):
"""A redeemed ticket: one firewall GUI open in one slot of one browser session."""
__tablename__ = "gui_sessions"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
user_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
)
host_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
)
slot: Mapped[int] = mapped_column(Integer, nullable=False)
client_ip: Mapped[str | None] = mapped_column(String(45), nullable=True)
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
last_checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
end_reason: Mapped[str | None] = mapped_column(String(32), nullable=True)
+65
View File
@@ -0,0 +1,65 @@
"""Recurring maintenance windows (1.25): "every Sunday 02:00-05:00 Europe/Berlin",
"first Tuesday of the month 22:00 for 3 h". A window on its own changes nothing on any
firewall; rollout plans (models/rollout.py) use it to decide WHEN they may start work on a host.
Times are wall-clock times in the window's own IANA time zone, so a window stays at 02:00
local time across daylight-saving changes (see services/window_service.py for the rules).
"""
import uuid
from datetime import date, datetime, time, timezone
from sqlalchemy import JSON, Boolean, Date, DateTime, ForeignKey, Integer, String, Time, UniqueConstraint
from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
def _now() -> datetime:
return datetime.now(timezone.utc)
class MaintenanceWindow(Base):
__tablename__ = "maintenance_windows"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
name: Mapped[str] = mapped_column(String(120), unique=True)
timezone: Mapped[str] = mapped_column(String(64))
recurrence: Mapped[str] = mapped_column(String(16)) # weekly | monthly
# weekly: weekdays 0 (Monday) .. 6 (Sunday). JSON columns get a Python default only (Alembic gotcha, see 0007).
weekdays: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
# monthly: "nth_weekday" (month_nth 1-4 or -1 = last, month_weekday 0-6) or "day" (month_day 1-28)
month_mode: Mapped[str | None] = mapped_column(String(16), nullable=True)
month_nth: Mapped[int | None] = mapped_column(Integer, nullable=True)
month_weekday: Mapped[int | None] = mapped_column(Integer, nullable=True)
month_day: Mapped[int | None] = mapped_column(Integer, nullable=True)
start_time: Mapped[time] = mapped_column(Time)
duration_minutes: Mapped[int] = mapped_column(Integer)
# After this many minutes into the window no NEW host action starts (running ones finish).
start_cutoff_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
created_by: Mapped[str] = mapped_column(String(150))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
exceptions: Mapped[list["MaintenanceWindowException"]] = relationship(
"MaintenanceWindowException", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
order_by="MaintenanceWindowException.occurrence_date",
)
class MaintenanceWindowException(Base):
"""One occurrence skipped or moved. Keyed by the occurrence's LOCAL date."""
__tablename__ = "maintenance_window_exceptions"
__table_args__ = (UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),)
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
window_id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="CASCADE"), index=True
)
occurrence_date: Mapped[date] = mapped_column(Date)
kind: Mapped[str] = mapped_column(String(8)) # skip | move
moved_start: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
reason: Mapped[str] = mapped_column(String(300), default="")
actor: Mapped[str] = mapped_column(String(150))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
+50 -1
View File
@@ -6,7 +6,7 @@ OPNsense API secrets stay off the database entirely.
import uuid
from datetime import datetime, timezone
from sqlalchemy import JSON, Boolean, DateTime, Integer, String, UniqueConstraint
from sqlalchemy import JSON, Boolean, DateTime, Float, Integer, String, UniqueConstraint
from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column
@@ -17,6 +17,13 @@ def _now() -> datetime:
return datetime.now(timezone.utc)
DEFAULT_SERVICE_WATCHLIST = (
"unbound", "dnsmasq", "dhcpd", "dhcpd6", "kea-dhcp4", "kea-dhcp6", "ntpd", "openvpn", "wireguard", "strongswan",
)
CHANNEL_TYPES = ("ntfy", "gotify", "telegram", "teams", "matrix")
class NotificationSettings(Base):
"""Single row, id fixed at 1 — see notification_settings_service.get_settings
for the get-or-create helper. The only admin-editable operational config
@@ -55,6 +62,30 @@ class NotificationSettings(Base):
# Off by default: switching it on for an existing fleet reports every long-standing finding at once.
compliance_alerts_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# Monitoring alerts (1.24). The gateway alert is on by default (a WAN outage is the one thing every
# operator wants to hear about); resource thresholds and service checks are off, like compliance alerts:
# switching them on for an existing fleet may report long-standing conditions all at once.
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
# OPNexus threshold on the measured values (0 = off).
gateway_loss_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=20, server_default="20")
gateway_delay_threshold_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=500, server_default="500")
# Resources (0 = that threshold is off). Load is per CPU thread.
disk_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=90, server_default="90")
memory_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=95, server_default="95")
swap_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=50, server_default="50")
load_per_cpu_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=2.0, server_default="2.0")
# Debouncing: consecutive 60 s resource samples that must agree before a problem opens or closes.
alert_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=3, server_default="3")
resource_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5")
# OPNsense service names that are reported when they stop (the API cannot tell a crash from a stop on purpose).
# NULL = the default list (an old row never got one).
service_watchlist: Mapped[list | None] = mapped_column(JSON, nullable=True, default=lambda: list(DEFAULT_SERVICE_WATCHLIST))
poll_interval_seconds: Mapped[int] = mapped_column(Integer, default=300)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
@@ -85,3 +116,21 @@ class NotificationProblem(Base):
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
resolved_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class NotificationChannel(Base):
"""An additional delivery channel (ntfy, Gotify, Telegram, Microsoft Teams, Matrix) next to the SMTP
and generic-webhook settings above. Several channels of one type are allowed. ``config`` holds only
non-secret parameters (server URL, topic, chat id, room id); the credential — token, or the whole
webhook URL for Teams — lives in notification_secret_store under ``channel_<id>``, never here."""
__tablename__ = "notification_channels"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
name: Mapped[str] = mapped_column(String(100), unique=True)
channel_type: Mapped[str] = mapped_column(String(16))
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
secret_set: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+149
View File
@@ -0,0 +1,149 @@
"""Firmware rollouts in waves (1.25): plans (what, in which order, when) and runs (what happened).
A plan's waves point at device groups ("rings": canary -> pilot -> prod). A run freezes the plan
(``plan_snapshot``) and the resolved hosts (``RolloutRunHost``) when it starts, so later edits of
the plan or of a group never change a run that is under way. The run never fires firmware actions
itself: every host step is an ordinary ScheduledMaintenance row (origin="rollout",
auto_verify=True) linked from ``RolloutRunHost.scheduled_maintenance_id`` — see
services/rollout_service.py.
"""
import uuid
from datetime import datetime, timezone
from sqlalchemy import (JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Table, Text,
UniqueConstraint)
from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
def _now() -> datetime:
return datetime.now(timezone.utc)
rollout_wave_groups = Table(
"rollout_wave_groups", Base.metadata,
Column("wave_id", PGUUID(as_uuid=True), ForeignKey("rollout_plan_waves.id", ondelete="CASCADE"), primary_key=True),
Column("group_id", PGUUID(as_uuid=True), ForeignKey("device_groups.id", ondelete="CASCADE"), primary_key=True),
)
class RolloutPlan(Base):
__tablename__ = "rollout_plans"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
name: Mapped[str] = mapped_column(String(120), unique=True)
description: Mapped[str] = mapped_column(Text, default="")
action: Mapped[str] = mapped_column(String(16)) # update | major_update
window_id: Mapped[uuid.UUID | None] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="SET NULL"), nullable=True, index=True)
# Start a run by itself at every occurrence of the window. Only for "update" and only while the plan
# is activated AND unchanged since (activated_fingerprint).
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
auto_start_error: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_by: Mapped[str] = mapped_column(String(150))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
waves: Mapped[list["RolloutPlanWave"]] = relationship(
"RolloutPlanWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
order_by="RolloutPlanWave.position")
class RolloutPlanWave(Base):
__tablename__ = "rollout_plan_waves"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
plan_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="CASCADE"), index=True)
position: Mapped[int] = mapped_column(Integer)
label: Mapped[str] = mapped_column(String(60))
soak_minutes: Mapped[int] = mapped_column(Integer, default=30)
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# Wait for a human "release" before this wave starts (not used for the first wave: starting is the release).
require_approval: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
groups: Mapped[list["DeviceGroup"]] = relationship("DeviceGroup", secondary=rollout_wave_groups, lazy="selectin") # noqa: F821
class RolloutRun(Base):
__tablename__ = "rollout_runs"
__table_args__ = (UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),)
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
plan_id: Mapped[uuid.UUID | None] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="SET NULL"), nullable=True, index=True)
plan_name: Mapped[str] = mapped_column(String(120))
plan_snapshot: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
action: Mapped[str] = mapped_column(String(16))
# waiting_window | preflight | wave_running | soaking | gate | awaiting_approval | resuming | paused
# | succeeded | cancelled
status: Mapped[str] = mapped_column(String(20), index=True)
paused_from: Mapped[str | None] = mapped_column(String(20), nullable=True)
current_wave: Mapped[int] = mapped_column(Integer, default=0)
target_version: Mapped[str | None] = mapped_column(String(40), nullable=True)
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
started_by: Mapped[str] = mapped_column(String(150))
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
progress: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
step_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
waves: Mapped[list["RolloutRunWave"]] = relationship(
"RolloutRunWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin", order_by="RolloutRunWave.index")
hosts: Mapped[list["RolloutRunHost"]] = relationship(
"RolloutRunHost", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
order_by=lambda: [RolloutRunHost.wave_index, RolloutRunHost.order])
class RolloutRunWave(Base):
__tablename__ = "rollout_run_waves"
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
index: Mapped[int] = mapped_column(Integer)
label: Mapped[str] = mapped_column(String(60))
# pending | awaiting_approval | preflight | running | soaking | gate | passed | skipped | cancelled
status: Mapped[str] = mapped_column(String(20), default="pending")
baseline: Mapped[dict | None] = mapped_column(JSON, nullable=True)
gate_result: Mapped[dict | None] = mapped_column(JSON, nullable=True)
soak_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
gate_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
approved_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class RolloutRunHost(Base):
__tablename__ = "rollout_run_hosts"
__table_args__ = (UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),)
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
wave_index: Mapped[int] = mapped_column(Integer)
host_id: Mapped[uuid.UUID | None] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="SET NULL"), nullable=True, index=True)
host_name: Mapped[str] = mapped_column(String(120))
cluster_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
order: Mapped[int] = mapped_column(Integer, default=0)
# queued | running | verifying | succeeded | skipped | failed | blocked | cancelled
status: Mapped[str] = mapped_column(String(16), default="queued")
scheduled_maintenance_id: Mapped[uuid.UUID | None] = mapped_column(
PGUUID(as_uuid=True), ForeignKey("scheduled_maintenance.id", ondelete="SET NULL"), nullable=True, index=True)
version_before: Mapped[str | None] = mapped_column(String(40), nullable=True)
version_after: Mapped[str | None] = mapped_column(String(40), nullable=True)
# {"code": ..., "params": {...}} — translated in the UI (rollouts.hostNote.*)
detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+6 -1
View File
@@ -42,7 +42,12 @@ class ScheduledMaintenance(Base):
host_operation_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
# Opt-in: hand the traffic over to the CARP partner first, run the action, then switch back.
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# Only while status == "running" with a handover: handover_wait | job_wait | leave | leave_wait.
# 1.25: check the result automatically (job_wait -> verify) and close the operation, instead of
# leaving it "submitted" for a human. Every rollout step sets it; off for existing plans.
auto_verify: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# manual | rollout (created by a rollout run, see models/rollout.py RolloutRunHost.scheduled_maintenance_id)
origin: Mapped[str] = mapped_column(String(16), nullable=False, default="manual", server_default="manual")
# Only while status == "running" with a handover or auto_verify: handover_wait | job_wait | verify | leave | leave_wait.
# State lives here, not in memory, so a backend restart in the middle of a run resumes cleanly.
phase: Mapped[str | None] = mapped_column(String(16), nullable=True)
phase_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+77
View File
@@ -0,0 +1,77 @@
"""The three "take over what exists on the firewall" endpoints that aliases, port forwards, 1:1 NAT and
outbound NAT share (services/object_adopt_service.py has the logic and the evidence):
GET <prefix>/unmanaged?host_id=… candidates found live on the host, with reasons for the refused ones
POST <prefix>/adopt take over one object {host_id, uuid}; returns the new central object
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
Like rule adoption this only creates central records; nothing is written to the firewall."""
import uuid
from typing import Callable
from fastapi import APIRouter, Depends
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.schemas.adopt import AdoptAllRequest, AdoptAllResult, AdoptObjectRequest, UnmanagedObject
from app.services import object_adopt_service
from app.services.adopt_common import LINK_ERRORS, link_api_error
from app.services.opnsense_client import OpnsenseAPIError
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
from app.services.policy_service import policy_lock
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
host = await db.get(OpnsenseHost, host_id)
if not host:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
return host
def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Callable, to_read: Callable, read_model) -> None:
@router.get("/unmanaged", response_model=list[UnmanagedObject])
async def list_unmanaged(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
host = await _host(db, host_id)
try:
return await object_adopt_service.list_unmanaged(db, kind, host)
except OpnsenseAPIError as exc:
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
@router.post("/adopt", response_model=read_model, status_code=201)
async def adopt(payload: AdoptObjectRequest, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
host = await _host(db, payload.host_id)
try:
created = await object_adopt_service.adopt(db, kind, host, payload.uuid, link_to=payload.link_to)
except object_adopt_service.NotFoundOnHost as exc:
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
except object_adopt_service.NotAdoptable as exc:
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
except LINK_ERRORS as exc:
raise link_api_error(exc) from exc
except OpnsenseAPIError as exc:
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
await db.commit()
# The new object sits in the session without its relationships loaded; db.get() would return it as-is.
created = (await db.execute(
select(model).where(model.id == created.id).options(*load_options())
.execution_options(populate_existing=True)
)).scalar_one()
await hydrate_hierarchy_targets(db, [created])
return to_read(created, created.targets, created.deployments)
@router.post("/adopt-all", response_model=AdoptAllResult)
async def adopt_all(payload: AdoptAllRequest, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
host = await _host(db, payload.host_id)
try:
result = await object_adopt_service.adopt_all(db, kind, host, link_existing=payload.link_existing)
except LINK_ERRORS as exc:
raise link_api_error(exc) from exc
except OpnsenseAPIError as exc:
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
await db.commit()
return result
+4
View File
@@ -17,6 +17,7 @@ from app.schemas.alias import DeploymentResult, AliasCreate, AliasRead, AliasEdi
from app.services.alias_deploy_service import confirm_alias, deploy_alias, undeploy_alias, revert_alias_deployment
from app.services.opnsense_client import OpnsenseAPIError
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
from app.routers.adopt_routes import add_adopt_routes
router = APIRouter(prefix="/aliases", tags=["aliases"])
@@ -70,6 +71,9 @@ def _to_read(alias: FirewallAlias, hosts: list[OpnsenseHost], deployments: list[
)
add_adopt_routes(router, kind="alias", model=FirewallAlias, load_options=_load_options, to_read=_to_read, read_model=AliasRead)
@router.get("", response_model=list[AliasRead])
async def list_aliases(db: AsyncSession = Depends(get_db)):
result = await db.execute(select(FirewallAlias).options(*_load_options()))
+46 -8
View File
@@ -15,7 +15,36 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/auth", tags=["auth"])
# Plain name: used on HTTP-only (lab) deployments and accepted for a while on
# HTTPS deployments so that sessions issued before 1.27 survive the upgrade.
COOKIE_NAME = "opnexus-session"
# HTTPS deployments: the __Host- prefix makes the browser refuse any cookie of
# this name that is not Secure, has a Domain attribute or a Path other than
# "/". A page on another port of the same host (the GUI gateway slots, any
# other service on that machine) therefore cannot toss a narrower-path cookie
# that shadows the real session, and cannot overwrite it from script (HttpOnly).
SECURE_COOKIE_NAME = "__Host-opnexus-session"
def session_cookie_name() -> str:
return SECURE_COOKIE_NAME if settings.cookie_secure else COOKIE_NAME
def session_token(request: Request) -> str:
"""The session token the browser sent. On HTTPS the __Host- cookie always
wins; the plain name is only a migration fallback when the browser has no
__Host- cookie yet (it is replaced on the next /api/auth/me)."""
if settings.cookie_secure:
return request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME, "")
return request.cookies.get(COOKIE_NAME, "")
def uses_legacy_cookie(request: Request) -> bool:
return settings.cookie_secure and not request.cookies.get(SECURE_COOKIE_NAME) and bool(request.cookies.get(COOKIE_NAME))
def has_session_cookie(request: Request) -> bool:
return bool(request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME))
def client_ip(request: Request) -> str:
@@ -32,7 +61,7 @@ def client_ip(request: Request) -> str:
def _set_session_cookie(response: Response, token: str, max_age: int | None = None) -> None:
response.set_cookie(
key=COOKIE_NAME,
key=session_cookie_name(),
value=token,
max_age=max_age or settings.session_lifetime,
httponly=True,
@@ -40,10 +69,15 @@ def _set_session_cookie(response: Response, token: str, max_age: int | None = No
secure=settings.cookie_secure,
path="/",
)
if settings.cookie_secure:
# Migration: a pre-1.27 plain-name cookie must not linger next to the new one.
response.delete_cookie(key=COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
def _clear_session_cookie(response: Response) -> None:
response.delete_cookie(key=COOKIE_NAME, path="/")
response.delete_cookie(key=COOKIE_NAME, path="/", secure=settings.cookie_secure, httponly=True, samesite="lax")
if settings.cookie_secure:
response.delete_cookie(key=SECURE_COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
def _check_account_budget(user: AdminUser, action: str) -> None:
@@ -71,7 +105,7 @@ async def resolve_user(request: Request, db: AsyncSession) -> AdminUser:
user = await db.get(AdminUser, resolved.id)
if user is not None:
return user
token = request.cookies.get(COOKIE_NAME, "")
token = session_token(request)
user = await auth_service.resolve_session(db, token)
if user is None:
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.NICHT_AUTHENTIFIZIERT)
@@ -229,7 +263,7 @@ async def totp_enable(
user.totp_last_step = step
user.totp_recovery_hashes = hashes
# Other sessions were established without the second factor — end them.
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
await db.commit()
logger.info("Two-factor enabled for '%s'", user.username)
return {"enabled": True, "recovery_codes": plain}
@@ -254,7 +288,7 @@ async def totp_disable(
user.totp_enabled = False
user.totp_recovery_hashes = None
user.totp_last_step = 0
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
await db.commit()
logger.info("Two-factor disabled for '%s'", user.username)
return {"enabled": False}
@@ -262,14 +296,14 @@ async def totp_disable(
@router.post("/logout")
async def logout(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
token = request.cookies.get(COOKIE_NAME, "")
token = session_token(request)
await auth_service.revoke_session(db, token)
_clear_session_cookie(response)
return {"authenticated": False}
@router.get("/me")
async def me(request: Request, db: AsyncSession = Depends(get_db)):
async def me(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
raw = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
if raw is not None:
resolved = await api_token_service.resolve(db, raw, client_ip(request))
@@ -278,10 +312,14 @@ async def me(request: Request, db: AsyncSession = Depends(get_db)):
user, api_token = resolved
return {"authenticated": True, "username": user.username, "role": user.role,
"via": "token", "scope": api_token.scope, "token_name": api_token.name}
token = request.cookies.get(COOKIE_NAME, "")
token = session_token(request)
user = await auth_service.resolve_session(db, token)
if user is None:
return {"authenticated": False}
if uses_legacy_cookie(request):
# Soft migration (1.27): the same server-side session moves to the
# __Host- cookie and the plain-name cookie is deleted — nobody is logged out.
_set_session_cookie(response, token)
return {"authenticated": True, "username": user.username, "role": user.role}
+25
View File
@@ -18,6 +18,8 @@ locks are deliberately NOT included here (no fleet-wide list endpoint
exists for those, only per-host, and it's a rarely-triggered flow) —
a documented v1 gap, not an oversight.
"""
from datetime import datetime, timezone
from fastapi import APIRouter, Depends
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -46,7 +48,10 @@ from app.models.rule_deployment import RuleDeployment
from app.models.wireguard_tunnel import WireguardTunnel
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
from app.models.resource_sample import HostResourceSample
from app.services import gateway_health_service
from app.services.host_status_service import get_statuses
from app.services.monitor_checks import STALE_AFTER
router = APIRouter(prefix="/dashboard", tags=["dashboard"])
@@ -202,14 +207,34 @@ async def collect_pending_actions(db: AsyncSession) -> list[PendingAction]:
return pending_actions
async def _gateways_by_host(db: AsyncSession, hosts) -> dict:
"""Gateway state from each host's newest resource sample; hosts without a fresh sample are left out."""
result: dict = {}
now = datetime.now(timezone.utc)
for host in hosts:
row = (
await db.execute(
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json())
.where(HostResourceSample.host_id == host.id)
.order_by(HostResourceSample.sampled_at.desc())
.limit(1)
)
).first()
if row is not None and row[1] is not None and now - row[0] <= STALE_AFTER:
result[host.id] = gateway_health_service.from_compact(row[1])
return result
@router.get("", response_model=DashboardSummary)
async def summary(db: AsyncSession = Depends(get_db)):
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
statuses = await get_statuses(hosts)
gateways = await _gateways_by_host(db, hosts)
host_summaries = [
DashboardHost(
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
gateways=gateways.get(host.id),
)
for host in hosts
]
+25
View File
@@ -0,0 +1,25 @@
import uuid
from fastapi import APIRouter, Depends
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.schemas.gateway import GatewayHealth
from app.services import gateway_health_service
from app.services.opnsense_client import OpnsenseAPIError
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
@router.get("", response_model=list[GatewayHealth])
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
"""Live gateway state (loss, delay, deviation, OPNsense' verdict) of one firewall."""
host = await db.get(OpnsenseHost, host_id)
if host is None:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
try:
return await gateway_health_service.get_gateway_health(host)
except OpnsenseAPIError as exc:
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
+3
View File
@@ -99,6 +99,9 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
from app.models.rollout import rollout_wave_groups
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
await db.delete(group)
await db.commit()
+185
View File
@@ -0,0 +1,185 @@
"""Firewall GUI access through the gui-gateway container (1.27).
Public API (session cookie, admin only, never API tokens):
* POST /api/hosts/{id}/gui-ticket — one-time ticket + the slot URL the browser opens (noopener).
* GET/PUT /api/settings/gui-access — switch, idle/absolute timeout, number of slots.
Internal API for the gateway only (NOT under /api, so the proxy never forwards it; shared secret
GUI_GATEWAY_TOKEN in the X-Onx-Gateway-Token header, compared in constant time):
* POST /internal/gui/redeem, /internal/gui/revalidate, /internal/gui/end.
The answers never contain firewall credentials: only the firewall origin, its pinned (public) CA
certificate, the overlay proxy address and the timeouts.
"""
import hashlib
import hmac
import uuid
from datetime import datetime, timezone
from typing import Literal
from fastapi import APIRouter, Depends, Header, HTTPException, Request, status
from pydantic import BaseModel, Field, model_validator
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.models.user import AdminUser
from app.routers.auth import SECURE_COOKIE_NAME, current_user
from app.services import api_token_service, gui_ticket_service as svc
router = APIRouter(tags=["gui-access"])
internal_router = APIRouter(prefix="/internal/gui", include_in_schema=False)
def _require_admin_session(request: Request, user: AdminUser) -> None:
if getattr(request.state, "api_token", None) is not None or api_token_service.bearer_from_header(
request.headers.get("authorization", "")
) is not None:
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.GUI_SESSION_REQUIRED)
if user.role != "admin":
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.ADMIN_REQUIRED)
class GuiTicketRead(BaseModel):
url: str
slot: int
expires_in: int
@router.post("/hosts/{host_id}/gui-ticket", response_model=GuiTicketRead)
async def create_ticket(host_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db),
user: AdminUser = Depends(current_user)):
_require_admin_session(request, user)
if not svc.gateway_configured():
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED)
session_cookie = request.cookies.get(SECURE_COOKIE_NAME, "")
if not settings.cookie_secure or not session_cookie:
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_REQUIRES_HTTPS)
row = await svc.get_settings(db)
if not row.enabled:
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_ACCESS_DISABLED)
host = await db.get(OpnsenseHost, host_id)
if host is None:
raise api_error(status.HTTP_404_NOT_FOUND, ErrorCode.HOST_NICHT_GEFUNDEN)
refusal = svc.host_refusal(host)
if refusal:
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_HOST_NOT_ELIGIBLE, reason=refusal)
try:
return await svc.issue_ticket(db, user=user, session_token=session_cookie, host=host,
request_host=request.headers.get("host", ""))
except svc.GuiAccessError as exc:
raise api_error(exc.status, exc.code, **exc.params)
class GuiAccessSettingsRead(BaseModel):
enabled: bool
idle_timeout_minutes: int
absolute_timeout_minutes: int
slot_count: int
available: bool
unavailable_reasons: list[str]
slots: list[int]
updated_by: str | None
updated_at: datetime | None
class GuiAccessSettingsUpdate(BaseModel):
enabled: bool
idle_timeout_minutes: int = Field(ge=1, le=240)
absolute_timeout_minutes: int = Field(ge=5, le=720)
slot_count: int = Field(ge=1, le=svc.MAX_SLOTS)
@model_validator(mode="after")
def _idle_within_absolute(self):
if self.idle_timeout_minutes > self.absolute_timeout_minutes:
raise ValueError("idle_timeout_minutes must not exceed absolute_timeout_minutes")
return self
def _settings_read(row) -> dict:
reasons = svc.unavailable_reasons()
return {"enabled": row.enabled, "idle_timeout_minutes": row.idle_timeout_minutes,
"absolute_timeout_minutes": row.absolute_timeout_minutes, "slot_count": row.slot_count,
"available": not reasons, "unavailable_reasons": reasons, "slots": svc.active_slots(row),
"updated_by": row.updated_by, "updated_at": row.updated_at}
@router.get("/settings/gui-access", response_model=GuiAccessSettingsRead)
async def read_settings(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
_require_admin_session(request, user)
row = await svc.get_settings(db)
await db.commit()
return _settings_read(row)
@router.put("/settings/gui-access", response_model=GuiAccessSettingsRead)
async def update_settings(payload: GuiAccessSettingsUpdate, request: Request, db: AsyncSession = Depends(get_db),
user: AdminUser = Depends(current_user)):
_require_admin_session(request, user)
if payload.enabled and svc.unavailable_reasons():
# Never switchable on without HTTPS cookies and a configured gateway.
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED,
reasons=", ".join(svc.unavailable_reasons()))
row = await svc.get_settings(db)
row.enabled = payload.enabled
row.idle_timeout_minutes = payload.idle_timeout_minutes
row.absolute_timeout_minutes = payload.absolute_timeout_minutes
row.slot_count = payload.slot_count
row.updated_by = user.username
row.updated_at = datetime.now(timezone.utc)
if not payload.enabled:
await svc.end_all(db, "disabled")
await db.commit()
return _settings_read(row)
# ── internal: gateway only ────────────────────────────────────────────────
def _gateway_auth(x_onx_gateway_token: str = Header(default="")) -> None:
expected = settings.gui_gateway_token
if len(expected) < svc.MIN_GATEWAY_TOKEN_LENGTH:
raise HTTPException(status_code=404)
# Fixed-length digests: the comparison time does not depend on the presented value.
if not hmac.compare_digest(hashlib.sha256(x_onx_gateway_token.encode()).digest(),
hashlib.sha256(expected.encode()).digest()):
raise HTTPException(status_code=401)
_HEX64 = r"^[0-9a-f]{64}$"
class RedeemRequest(BaseModel):
ticket_hash: str = Field(pattern=_HEX64)
session_hash: str = Field(pattern=_HEX64)
slot: int = Field(ge=1, le=65535)
client_ip: str | None = Field(default=None, max_length=45)
class RevalidateRequest(BaseModel):
gui_session_id: uuid.UUID
session_hash: str = Field(pattern=_HEX64)
class EndRequest(RevalidateRequest):
reason: Literal["idle", "absolute", "logout", "revoked", "superseded", "shutdown"]
@internal_router.post("/redeem", dependencies=[Depends(_gateway_auth)])
async def internal_redeem(payload: RedeemRequest, db: AsyncSession = Depends(get_db)):
result = await svc.redeem(db, ticket_hash=payload.ticket_hash, session_hash=payload.session_hash,
slot=payload.slot, client_ip=payload.client_ip)
if not result["ok"]:
raise HTTPException(status_code=403, detail={"ok": False, "reason": result["reason"]})
return result
@internal_router.post("/revalidate", dependencies=[Depends(_gateway_auth)])
async def internal_revalidate(payload: RevalidateRequest, db: AsyncSession = Depends(get_db)):
return await svc.revalidate(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash)
@internal_router.post("/end", dependencies=[Depends(_gateway_auth)])
async def internal_end(payload: EndRequest, db: AsyncSession = Depends(get_db)):
return {"ok": await svc.end(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
reason=payload.reason)}
+244
View File
@@ -0,0 +1,244 @@
"""Recurring maintenance windows and their exceptions (1.25).
Every write here counts as a rollout action (api_token_service.is_rollout): a window decides WHEN
unattended firmware actions may run, so an operator needs the same right as for deploying.
"""
import re
import uuid
from datetime import date, datetime, time, timedelta, timezone
from typing import Literal
from fastapi import APIRouter, Depends, Query, Request
from pydantic import BaseModel, Field
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
from app.services import audit_service, window_service
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/maintenance-windows", tags=["maintenance-windows"])
_TIME = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
class WindowIn(BaseModel):
name: str = Field(min_length=1, max_length=120)
timezone: str = Field(min_length=1, max_length=64)
recurrence: Literal["weekly", "monthly"]
weekdays: list[int] = Field(default_factory=list, max_length=7)
month_mode: Literal["nth_weekday", "day"] | None = None
month_nth: int | None = None
month_weekday: int | None = None
month_day: int | None = None
start_time: str
duration_minutes: int = Field(ge=15, le=window_service.MAX_DURATION_MINUTES)
start_cutoff_minutes: int = Field(default=60, ge=5, le=window_service.MAX_DURATION_MINUTES)
enabled: bool = True
class ExceptionIn(BaseModel):
occurrence_date: date
kind: Literal["skip", "move"]
moved_start: datetime | None = None
reason: str = Field(default="", max_length=300)
def _invalid(field: str):
return api_error(422, ErrorCode.WINDOW_INVALID, field=field)
def _validated(payload: WindowIn) -> dict:
name = payload.name.strip()
if not name:
raise _invalid("name")
if not window_service.valid_timezone(payload.timezone):
raise _invalid("timezone")
match = _TIME.match(payload.start_time)
if not match:
raise _invalid("start_time")
values = {
"name": name, "timezone": payload.timezone, "recurrence": payload.recurrence,
"weekdays": [], "month_mode": None, "month_nth": None, "month_weekday": None, "month_day": None,
"start_time": time(int(match.group(1)), int(match.group(2))),
"duration_minutes": payload.duration_minutes,
"start_cutoff_minutes": min(payload.start_cutoff_minutes, payload.duration_minutes),
"enabled": payload.enabled,
}
if payload.recurrence == "weekly":
days = sorted(set(payload.weekdays))
if not days or any(d < 0 or d > 6 for d in days):
raise _invalid("weekdays")
values["weekdays"] = days
elif payload.month_mode == "day":
if payload.month_day is None or not 1 <= payload.month_day <= 28:
raise _invalid("month_day")
values.update(month_mode="day", month_day=payload.month_day)
elif payload.month_mode == "nth_weekday":
if payload.month_nth not in (1, 2, 3, 4, -1):
raise _invalid("month_nth")
if payload.month_weekday is None or not 0 <= payload.month_weekday <= 6:
raise _invalid("month_weekday")
values.update(month_mode="nth_weekday", month_nth=payload.month_nth, month_weekday=payload.month_weekday)
else:
raise _invalid("month_mode")
return values
async def _plans_using(db: AsyncSession, window_id) -> list[dict]:
from app.models.rollout import RolloutPlan
rows = (await db.scalars(select(RolloutPlan).where(RolloutPlan.window_id == window_id).order_by(RolloutPlan.name))).all()
return [{"id": str(p.id), "name": p.name, "auto_start": p.auto_start, "active": p.activated_at is not None} for p in rows]
def read_window(window: MaintenanceWindow, *, now: datetime | None = None, count: int = 5) -> dict:
now = now or datetime.now(timezone.utc)
return {
"id": str(window.id), "name": window.name, "timezone": window.timezone, "recurrence": window.recurrence,
"weekdays": window.weekdays or [], "month_mode": window.month_mode, "month_nth": window.month_nth,
"month_weekday": window.month_weekday, "month_day": window.month_day,
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
"created_by": window.created_by, "created_at": window.created_at,
"open_now": window_service.current(window, now) is not None,
"next": [o.as_dict() for o in window_service.occurrences(window, now, count, with_skipped=True)] if window.enabled else [],
"exceptions": [
{"id": str(e.id), "occurrence_date": e.occurrence_date.isoformat(), "kind": e.kind,
"moved_start": e.moved_start, "reason": e.reason, "actor": e.actor, "created_at": e.created_at}
for e in window.exceptions
],
}
async def _window_or_404(db: AsyncSession, window_id: uuid.UUID) -> MaintenanceWindow:
window = await db.get(MaintenanceWindow, window_id)
if window is None:
raise api_error(404, ErrorCode.WINDOW_NOT_FOUND)
return window
async def _name_taken(db: AsyncSession, name: str, own_id=None) -> bool:
other = await db.scalar(select(MaintenanceWindow.id).where(MaintenanceWindow.name == name))
return other is not None and other != own_id
@router.get("")
async def list_windows(db: AsyncSession = Depends(get_db)):
rows = (await db.scalars(select(MaintenanceWindow).order_by(MaintenanceWindow.name))).all()
result = []
for w in rows:
item = read_window(w)
item["used_by"] = await _plans_using(db, w.id)
result.append(item)
return result
@router.post("", status_code=201)
async def create_window(payload: WindowIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
values = _validated(payload)
if await _name_taken(db, values["name"]):
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
window = MaintenanceWindow(**values, created_by=request.state.user.username, created_at=datetime.now(timezone.utc))
window.exceptions = []
db.add(window)
await db.commit()
item = read_window(await _window_or_404(db, window.id))
item["used_by"] = []
return item
@router.get("/{window_id}")
async def get_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
item = read_window(await _window_or_404(db, window_id))
item["used_by"] = await _plans_using(db, window_id)
return item
@router.put("/{window_id}")
async def update_window(window_id: uuid.UUID, payload: WindowIn, db: AsyncSession = Depends(get_db)):
"""Changing a window changes the fingerprint of every plan that uses it, so an activated
automatic start stops until the plan is activated again (see rollout_service.fingerprint)."""
await policy_lock(db)
window = await _window_or_404(db, window_id)
values = _validated(payload)
if await _name_taken(db, values["name"], window.id):
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
for key, value in values.items():
setattr(window, key, value)
await db.commit()
item = read_window(await _window_or_404(db, window_id))
item["used_by"] = await _plans_using(db, window_id)
return item
@router.delete("/{window_id}", status_code=204)
async def delete_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
window = await _window_or_404(db, window_id)
# A plan without a window means "no time limit": deleting its window must never turn it into one.
if await _plans_using(db, window_id):
raise api_error(409, ErrorCode.WINDOW_IN_USE)
await db.delete(window)
await db.commit()
@router.get("/{window_id}/occurrences")
async def window_occurrences(window_id: uuid.UUID, count: int = Query(10, ge=1, le=50), db: AsyncSession = Depends(get_db)):
window = await _window_or_404(db, window_id)
return [o.as_dict() for o in window_service.occurrences(window, datetime.now(timezone.utc), count, with_skipped=True)]
@router.post("/{window_id}/exceptions", status_code=201)
async def add_exception(window_id: uuid.UUID, payload: ExceptionIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
window = await _window_or_404(db, window_id)
now = datetime.now(timezone.utc)
if not window_service.is_occurrence_date(window, payload.occurrence_date):
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
# Only future occurrences (or the one open now) can be changed; history stays as it was.
if window_service.local_start(window, payload.occurrence_date) + timedelta(minutes=window.duration_minutes) <= now:
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
moved = None
if payload.kind == "move":
if payload.moved_start is None:
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
moved = payload.moved_start if payload.moved_start.tzinfo else payload.moved_start.replace(tzinfo=timezone.utc)
if moved <= now:
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
if any(e.occurrence_date == payload.occurrence_date for e in window.exceptions):
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
exc = MaintenanceWindowException(
window_id=window.id, occurrence_date=payload.occurrence_date, kind=payload.kind, moved_start=moved,
reason=payload.reason.strip(), actor=request.state.user.username, created_at=now,
)
db.add(exc)
try:
await db.commit()
except IntegrityError:
await db.rollback()
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
await audit_service.persist(
kind="maintenance_window", action=f"exception_{payload.kind}", status="saved", entity_id=str(window.id),
entity_name=window.name, details={"occurrence_date": payload.occurrence_date.isoformat(),
"moved_start": moved.isoformat() if moved else None, "reason": exc.reason},
)
db.expire_all()
return read_window(await _window_or_404(db, window_id))
@router.delete("/{window_id}/exceptions/{exception_id}")
async def delete_exception(window_id: uuid.UUID, exception_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
window = await _window_or_404(db, window_id)
exc = await db.get(MaintenanceWindowException, exception_id)
if exc is None or exc.window_id != window.id:
raise api_error(404, ErrorCode.WINDOW_EXCEPTION_NOT_FOUND)
await db.delete(exc)
await db.commit()
db.expire_all()
return read_window(await _window_or_404(db, window_id))
+25
View File
@@ -88,6 +88,31 @@ TOOLS = [
"path": "/api/search",
"query": ("q",),
},
{
"name": "rule_match",
"description": "Which firewall rule decides a packet (first packet of a new connection), per firewall, "
"with the verdict (pass/block/reject), the deciding rule and an honest confidence "
"(sure/probable/unknown plus reason codes). Reads only; nothing is changed.",
"inputSchema": {
"type": "object", "required": ["src", "dst"],
"properties": {
"src": {"type": "string", "description": "source IP address"},
"dst": {"type": "string", "description": "destination IP address"},
"proto": {"type": "string", "description": "tcp, udp, icmp, ipv6-icmp or a protocol number", "default": "tcp"},
"dport": {"type": "integer", "minimum": 0, "maximum": 65535},
"sport": {"type": "integer", "minimum": 0, "maximum": 65535, "description": "default: any ephemeral port"},
"icmptype": {"type": "string", "description": "e.g. echoreq or 8"},
"iface": {"type": "string", "description": "ingress interface identifier (e.g. lan); default: guessed per firewall"},
"egress": {"type": "string", "description": "also check the outgoing side: auto or an interface identifier"},
"host_ids": {"type": "string", "description": "comma separated firewall ids (default: all)"},
"group_id": {"type": "string", "description": "device group id instead of host_ids"},
},
},
"path": "/api/rule-analysis/match",
"query": ("src", "dst", "proto", "dport", "sport", "icmptype", "iface", "egress", "host_ids", "group_id"),
# The per-rule trace is for the UI; the verdict, deciding rule and reasons are what matters here.
"drop": ("trace",),
},
{
"name": "api_get",
"description": "Read any other OPNexus API endpoint (GET only), e.g. /api/rules, /api/aliases, /api/logs. "
+112 -2
View File
@@ -1,21 +1,26 @@
import uuid
from urllib.parse import urlsplit
from fastapi import APIRouter, Depends, Query
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.errors import ErrorCode, api_error
from app.database import get_db
from app.models.user import AdminUser
from app.routers.auth import current_user
from app.models.notification import NotificationProblem
from app.models.notification import NotificationChannel, NotificationProblem
from app.schemas.notification import (
NotificationChannelCreate,
NotificationChannelRead,
NotificationChannelUpdate,
NotificationProblemRead,
NotificationSettingsRead,
NotificationSettingsUpdate,
TestNotificationRequest,
)
from app.services import notification_delivery, notification_settings_service
from app.services import notification_channels, notification_delivery, notification_secret_store, notification_settings_service, url_guard
router = APIRouter(prefix="/notifications", tags=["notifications"])
@@ -43,6 +48,11 @@ async def read_settings(db: AsyncSession = Depends(get_db), user: AdminUser = De
@router.put("/settings", response_model=NotificationSettingsRead)
async def update_settings(payload: NotificationSettingsUpdate, db: AsyncSession = Depends(get_db)):
if payload.webhook_url:
try:
payload.webhook_url = url_guard.validate_url(payload.webhook_url)
except ValueError as exc:
raise api_error(422, ErrorCode.NOTIFICATION_WEBHOOK_URL_INVALID, reason=str(exc)) from None
return await notification_settings_service.update_settings(db, payload)
@@ -79,3 +89,103 @@ async def test_notification(payload: TestNotificationRequest, db: AsyncSession =
except notification_delivery.NotificationDeliveryError as exc:
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
return {"success": True}
# ── Additional channels (ntfy, Gotify, Telegram, Teams, Matrix) ────────────────
def _channel_read(row: NotificationChannel, user: AdminUser) -> NotificationChannelRead:
read = NotificationChannelRead.model_validate(row)
# An ntfy topic or a chat id is as good as a password for whoever knows it: only admins see the settings.
return read if user.role == "admin" else read.model_copy(update={"config": {}})
async def _get_channel(db: AsyncSession, channel_id: uuid.UUID) -> NotificationChannel:
row = await db.get(NotificationChannel, channel_id)
if row is None:
raise api_error(404, ErrorCode.NOTIFICATION_CHANNEL_NOT_FOUND)
return row
def _invalid(exc: ValueError):
return api_error(422, ErrorCode.NOTIFICATION_CHANNEL_INVALID, reason=str(exc))
@router.get("/channels", response_model=list[NotificationChannelRead])
async def list_channels(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
rows = await db.scalars(select(NotificationChannel).order_by(NotificationChannel.name))
return [_channel_read(r, user) for r in rows.all()]
@router.post("/channels", response_model=NotificationChannelRead, status_code=201)
async def create_channel(payload: NotificationChannelCreate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
try:
config = notification_channels.clean_config(payload.channel_type, payload.config)
secret = notification_channels.clean_secret(payload.channel_type, payload.secret) if payload.secret else None
except ValueError as exc:
raise _invalid(exc) from None
row = NotificationChannel(
name=payload.name.strip(), channel_type=payload.channel_type, enabled=payload.enabled, config=config,
secret_set=secret is not None,
)
row.id = uuid.uuid4()
if secret is None and notification_channels.missing_requirements(row) == "Zugangsdaten fehlen":
raise _invalid(ValueError("Zugangsdaten fehlen"))
db.add(row)
try:
await db.flush()
except IntegrityError:
await db.rollback()
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
if secret is not None:
notification_secret_store.save_secret(notification_channels.secret_name(row.id), secret)
await db.commit()
return _channel_read(row, user)
@router.put("/channels/{channel_id}", response_model=NotificationChannelRead)
async def update_channel(
channel_id: uuid.UUID, payload: NotificationChannelUpdate,
db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user),
):
row = await _get_channel(db, channel_id)
try:
if payload.config is not None:
row.config = notification_channels.clean_config(row.channel_type, payload.config)
new_secret = notification_channels.clean_secret(row.channel_type, payload.secret) if payload.secret else None
except ValueError as exc:
raise _invalid(exc) from None
if payload.name is not None:
row.name = payload.name.strip()
if payload.enabled is not None:
row.enabled = payload.enabled
if payload.clear_secret:
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
row.secret_set = False
elif new_secret is not None:
notification_secret_store.save_secret(notification_channels.secret_name(row.id), new_secret)
row.secret_set = True
try:
await db.commit()
except IntegrityError:
await db.rollback()
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
return _channel_read(row, user)
@router.delete("/channels/{channel_id}", status_code=204)
async def delete_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
row = await _get_channel(db, channel_id)
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
await db.delete(row)
await db.commit()
@router.post("/channels/{channel_id}/test")
async def test_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
row = await _get_channel(db, channel_id)
try:
await notification_channels.send_test(row)
except notification_delivery.NotificationDeliveryError as exc:
# The text names the channel and the HTTP status; it never contains a URL or a token.
raise api_error(502, ErrorCode.NOTIFICATION_CHANNEL_TEST_FAILED, reason=str(exc)) from None
return {"success": True}
+4
View File
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
from app.services.opnsense_client import OpnsenseAPIError
from app.services.one_to_one_deploy_service import confirm_one_to_one, deploy_one_to_one, undeploy_one_to_one, revert_deployment
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
from app.routers.adopt_routes import add_adopt_routes
router = APIRouter(prefix="/one-to-one", tags=["one-to-one"])
@@ -79,6 +80,9 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
)
add_adopt_routes(router, kind="one_to_one", model=OneToOneRule, load_options=_load_options, to_read=_to_read, read_model=OneToOneRead)
@router.get("", response_model=list[OneToOneRead])
async def list_one_to_one_rules(db: AsyncSession = Depends(get_db)):
result = await db.execute(select(OneToOneRule).options(*_load_options()))
+4
View File
@@ -21,6 +21,7 @@ from app.services.interface_service import check_interface_consistency
from app.services.opnsense_client import OpnsenseAPIError
from app.services.outbound_nat_deploy_service import confirm_outbound_nat, deploy_outbound_nat, undeploy_outbound_nat, revert_deployment
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
from app.routers.adopt_routes import add_adopt_routes
router = APIRouter(prefix="/outbound-nat", tags=["outbound-nat"])
@@ -82,6 +83,9 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
)
add_adopt_routes(router, kind="outbound_nat", model=OutboundNatRule, load_options=_load_options, to_read=_to_read, read_model=OutboundNatRead)
@router.get("", response_model=list[OutboundNatRead])
async def list_outbound_nat_rules(db: AsyncSession = Depends(get_db)):
result = await db.execute(select(OutboundNatRule).options(*_load_options()))
+4
View File
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
from app.services.opnsense_client import OpnsenseAPIError
from app.services.port_forward_deploy_service import confirm_port_forward, deploy_port_forward, undeploy_port_forward, revert_deployment
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
from app.routers.adopt_routes import add_adopt_routes
router = APIRouter(prefix="/port-forward", tags=["port-forward"])
@@ -80,6 +81,9 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
)
add_adopt_routes(router, kind="port_forward", model=PortForwardRule, load_options=_load_options, to_read=_to_read, read_model=PortForwardRead)
@router.get("", response_model=list[PortForwardRead])
async def list_port_forward_rules(db: AsyncSession = Depends(get_db)):
result = await db.execute(select(PortForwardRule).options(*_load_options()))
+432
View File
@@ -0,0 +1,432 @@
"""Firmware rollouts in waves (1.25): plans and runs.
Roles (enforced in AuthMiddleware, see api_token_service.is_rollout): everybody signed in reads;
operators prepare plans; starting/activating a plan and resuming/approving/skipping/aborting a run
need the deploy right (admin, or operator while operators may deploy). Pausing is deliberately NOT
a rollout action: anybody who may write at all may always brake.
"""
import uuid
from datetime import datetime, timezone
from typing import Literal
from fastapi import APIRouter, Depends, Request
from pydantic import BaseModel, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.maintenance_window import MaintenanceWindow
from app.models.policy import DeviceGroup
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
from app.services.policy_service import policy_lock
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
ACTIVE_RUN = rollout_engine.ACTIVE
class WaveIn(BaseModel):
label: str = Field(min_length=1, max_length=60)
group_ids: list[uuid.UUID] = Field(min_length=1, max_length=20)
soak_minutes: int = Field(default=30, ge=0, le=rollout_service.MAX_SOAK_MINUTES)
carp_handover: bool = False
require_approval: bool = False
class PlanIn(BaseModel):
name: str = Field(min_length=1, max_length=120)
description: str = Field(default="", max_length=2000)
action: Literal["update", "major_update"] = "update"
window_id: uuid.UUID | None = None
auto_start: bool = False
gate_checks: dict[str, bool] = Field(default_factory=dict)
compliance_mode: Literal["warn", "fail"] = "warn"
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
class TokenIn(BaseModel):
preview_token: str = Field(min_length=64, max_length=64)
async def _plan_or_404(db: AsyncSession, plan_id: uuid.UUID) -> RolloutPlan:
plan = await rollout_service.load_plan(db, plan_id)
if plan is None:
raise api_error(404, ErrorCode.ROLLOUT_PLAN_NOT_FOUND)
return plan
async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None = None) -> dict:
resolved = resolved or await rollout_service.resolve(db, plan)
active = plan.activated_at is not None
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
return {
**rollout_service.definition(plan),
"description": plan.description,
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
"group_ids": [str(g.id) for g in w.groups], "groups": [g.name for g in w.groups],
"host_count": len(resolved["waves"][i]["hosts"]) if i < len(resolved["waves"]) else 0}
for i, w in enumerate(plan.waves)],
"active": active, "activated_by": plan.activated_by, "activated_at": plan.activated_at,
# Activated but the plan, its groups or its window changed since: the automatic start is held back.
"fingerprint_ok": (not active) or plan.activated_fingerprint == rollout_service.fingerprint(plan, resolved),
"auto_start_error": plan.auto_start_error,
"window": resolved["window"], "errors": resolved["errors"], "warnings": resolved["warnings"],
"host_count": sum(len(w["hosts"]) for w in resolved["waves"]),
"last_run": {"id": str(last_run.id), "status": last_run.status, "created_at": last_run.created_at} if last_run else None,
"created_by": plan.created_by, "created_at": plan.created_at, "updated_at": plan.updated_at,
}
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
name = payload.name.strip()
if not name:
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="name")
other = await db.scalar(select(RolloutPlan.id).where(RolloutPlan.name == name))
if other is not None and other != plan.id:
raise api_error(409, ErrorCode.ROLLOUT_NAME_TAKEN)
if payload.window_id is not None and await db.get(MaintenanceWindow, payload.window_id) is None:
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="window_id")
if any(k not in rollout_health.CHECKS for k in payload.gate_checks):
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="gate_checks")
wanted = {g for w in payload.waves for g in w.group_ids}
groups = {g.id: g for g in (await db.scalars(select(DeviceGroup).where(DeviceGroup.id.in_(wanted)))).all()}
if len(groups) != len(wanted):
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="group_ids")
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
plan.compliance_mode = payload.compliance_mode
plan.waves = [
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
require_approval=w.require_approval, groups=[groups[g] for g in dict.fromkeys(w.group_ids)])
for i, w in enumerate(payload.waves)
]
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
plan.activated_at = plan.activated_by = plan.activated_fingerprint = None
plan.auto_start_error = None
@router.get("/plans")
async def list_plans(db: AsyncSession = Depends(get_db)):
plans = (await db.scalars(select(RolloutPlan).order_by(RolloutPlan.name))).all()
return [await read_plan(db, p) for p in plans]
@router.post("/plans", status_code=201)
async def create_plan(payload: PlanIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
plan = RolloutPlan(created_by=request.state.user.username, created_at=datetime.now(timezone.utc), waves=[])
await _apply(db, plan, payload)
db.add(plan)
await db.commit()
return await read_plan(db, await _plan_or_404(db, plan.id))
@router.get("/plans/{plan_id}")
async def get_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
return await read_plan(db, await _plan_or_404(db, plan_id))
@router.put("/plans/{plan_id}")
async def update_plan(plan_id: uuid.UUID, payload: PlanIn, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
plan = await _plan_or_404(db, plan_id)
was_active = plan.activated_at is not None
await _apply(db, plan, payload)
await db.commit()
if was_active:
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id),
entity_name=plan.name, message="Plan geändert: automatischer Start deaktiviert")
return await read_plan(db, await _plan_or_404(db, plan_id))
@router.delete("/plans/{plan_id}", status_code=204)
async def delete_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
plan = await _plan_or_404(db, plan_id)
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.status.in_(ACTIVE_RUN)).limit(1)):
raise api_error(409, ErrorCode.ROLLOUT_PLAN_BUSY)
await db.delete(plan)
await db.commit()
@router.post("/plans/{plan_id}/preview")
async def preview_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
plan = await _plan_or_404(db, plan_id)
resolved = await rollout_service.resolve(db, plan)
return {**resolved, "preview_token": rollout_service.fingerprint(plan, resolved),
"can_start": not resolved["errors"], "plan": await read_plan(db, plan, resolved)}
async def _checked_token(db: AsyncSession, plan: RolloutPlan, token: str) -> tuple[dict, str]:
resolved = await rollout_service.resolve(db, plan)
current = rollout_service.fingerprint(plan, resolved)
if token != current:
raise api_error(409, ErrorCode.ROLLOUT_PREVIEW_STALE)
if resolved["errors"]:
raise api_error(422, ErrorCode.ROLLOUT_PLAN_INVALID, reason=resolved["errors"][0]["code"])
return resolved, current
@router.post("/plans/{plan_id}/activate")
async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
plan = await _plan_or_404(db, plan_id)
if not plan.auto_start:
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
_resolved, current = await _checked_token(db, plan, payload.preview_token)
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
plan.auto_start_error = None
await db.commit()
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
entity_name=plan.name, details={"fingerprint": current})
return await read_plan(db, await _plan_or_404(db, plan_id))
@router.post("/plans/{plan_id}/deactivate")
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
plan = await _plan_or_404(db, plan_id)
plan.activated_by = plan.activated_at = plan.activated_fingerprint = None
await db.commit()
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
return await read_plan(db, await _plan_or_404(db, plan_id))
# ── runs ─────────────────────────────────────────────────────────────────────
class StartIn(TokenIn):
confirm_name: str = Field(min_length=1, max_length=120)
acknowledge_unattended: Literal[True]
class ReasonIn(BaseModel):
reason: str = Field(default="", max_length=500)
class SkipIn(BaseModel):
host_id: uuid.UUID
reason: str = Field(min_length=3, max_length=500)
class AbortIn(BaseModel):
reason: str = Field(min_length=3, max_length=500)
confirm_name: str = Field(min_length=1, max_length=120)
def _read_run_summary(run: RolloutRun) -> dict:
counts: dict[str, int] = {}
for h in run.hosts:
counts[h.status] = counts.get(h.status, 0) + 1
return {
"id": str(run.id), "plan_id": str(run.plan_id) if run.plan_id else None, "plan_name": run.plan_name,
"action": run.action, "status": run.status, "paused_from": run.paused_from, "current_wave": run.current_wave,
"wave_count": len(run.waves), "current_wave_label": next((w.label for w in run.waves if w.index == run.current_wave), None),
"target_version": run.target_version, "pause_reason": run.pause_reason, "pause_detail": run.pause_detail,
"started_by": run.started_by, "trigger": run.trigger, "window_occurrence": run.window_occurrence,
"created_at": run.created_at, "finished_at": run.finished_at, "step_at": run.step_at, "host_counts": counts,
"active": run.status in rollout_engine.ACTIVE,
}
async def read_run(db: AsyncSession, run: RolloutRun) -> dict:
scheds = {}
ids = [h.scheduled_maintenance_id for h in run.hosts if h.scheduled_maintenance_id]
if ids:
scheds = {s.id: s for s in (await db.scalars(select(ScheduledMaintenance).where(ScheduledMaintenance.id.in_(ids)))).all()}
soak_until = next((w.soak_until for w in run.waves if w.index == run.current_wave), None)
return {
**_read_run_summary(run),
"soak_until": soak_until, "progress": {k: v for k, v in (run.progress or {}).items()
if k in ("gate_recheck_at", "cluster_recheck_at", "blocked_since", "resume_gate",
"cluster_gate", "preflight_started", "resume_recheck_at")},
"snapshot": run.plan_snapshot,
"waves": [{"index": w.index, "label": w.label, "status": w.status, "gate_result": w.gate_result,
"gate_attempts": w.gate_attempts, "soak_until": w.soak_until, "approved_by": w.approved_by,
"started_at": w.started_at, "finished_at": w.finished_at,
"spec": run.plan_snapshot["waves"][w.index] if w.index < len(run.plan_snapshot.get("waves", [])) else None}
for w in run.waves],
"hosts": [{"id": str(h.id), "host_id": str(h.host_id) if h.host_id else None, "host_name": h.host_name,
"cluster_name": h.cluster_name, "wave_index": h.wave_index, "order": h.order, "status": h.status,
"version_before": h.version_before, "version_after": h.version_after, "detail": h.detail,
"started_at": h.started_at, "finished_at": h.finished_at,
"schedule": ({"id": str(s.id), "status": s.status, "phase": s.phase, "carp_handover": s.carp_handover,
"host_operation_id": str(s.host_operation_id) if s.host_operation_id else None}
if (s := scheds.get(h.scheduled_maintenance_id)) else None)}
for h in sorted(run.hosts, key=lambda h: (h.wave_index, h.order, h.host_name))],
}
async def _cancel_if_pending(db: AsyncSession, sched_id) -> bool:
"""Cancel a host's schedule only if it has not fired. Locked: the poll loop may be firing it right now
(fire_due holds the row), and a blind write would turn a running action into "cancelled"."""
if sched_id is None:
return False
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sched_id)
.with_for_update().execution_options(populate_existing=True))
if sched is not None and sched.status == "pending":
sched.status = "cancelled"
return True
return False
async def _run_or_404(db: AsyncSession, run_id: uuid.UUID, *, lock: bool = False) -> RolloutRun:
stmt = select(RolloutRun).where(RolloutRun.id == run_id).execution_options(populate_existing=True)
if lock:
stmt = stmt.with_for_update()
run = await db.scalar(stmt)
if run is None:
raise api_error(404, ErrorCode.ROLLOUT_RUN_NOT_FOUND)
return run
def _not_now(run: RolloutRun):
return api_error(409, ErrorCode.ROLLOUT_ACTION_NOT_ALLOWED, status=run.status)
async def _done(db: AsyncSession, run: RolloutRun, action: str, status: str, request: Request, **details) -> dict:
run.revision += 1
await db.commit()
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id), entity_name=run.plan_name,
details={"wave": run.current_wave, **details})
return await read_run(db, await _run_or_404(db, run.id))
@router.post("/plans/{plan_id}/start", status_code=201)
async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
plan = await _plan_or_404(db, plan_id)
if payload.confirm_name != plan.name:
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
resolved, _token = await _checked_token(db, plan, payload.preview_token)
if (other := await rollout_engine.active_run(db)) is not None:
# One run at a time: two runs could otherwise touch the same cluster at once.
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username, trigger="manual")
await db.commit()
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
"waves": [w["label"] for w in resolved["waves"]]})
return await read_run(db, await _run_or_404(db, run.id))
@router.get("/runs")
async def list_runs(db: AsyncSession = Depends(get_db)):
runs = (await db.scalars(select(RolloutRun).order_by(RolloutRun.created_at.desc()).limit(50))).all()
return [_read_run_summary(r) for r in runs]
@router.get("/status")
async def rollout_status(db: AsyncSession = Depends(get_db)):
"""For /updates and the host pages: the active run and the hosts it still owns."""
run = await rollout_engine.active_run(db)
if run is None:
return {"active": None, "host_ids": [], "current_host_ids": []}
return {
"active": _read_run_summary(run),
"host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.status in ("queued", "blocked", "running", "verifying", "failed")],
"current_host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.wave_index == run.current_wave
and h.status in ("queued", "blocked", "running", "verifying", "failed")],
}
@router.get("/runs/{run_id}")
async def get_run(run_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
return await read_run(db, await _run_or_404(db, run_id))
@router.post("/runs/{run_id}/pause")
async def pause_run(run_id: uuid.UUID, payload: ReasonIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
run = await _run_or_404(db, run_id, lock=True)
if run.status not in rollout_engine.ACTIVE or run.status == "paused":
raise _not_now(run)
# A brake that lets the next host fire anyway is no brake: a step not fired yet goes back to the queue.
requeued = [h.host_name for h in run.hosts if await rollout_engine.requeue_if_pending(db, h)]
rollout_engine.pause(run, "manual", by=request.state.user.username, text=payload.reason.strip()[:300])
return await _done(db, run, "pause", "paused", request, reason="manual", text=payload.reason.strip()[:300], requeued=requeued)
@router.post("/runs/{run_id}/resume")
async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
run = await _run_or_404(db, run_id, lock=True)
if run.status != "paused":
raise _not_now(run)
failed = [h.host_name for h in run.hosts if h.status == "failed"]
if failed:
# A failed host is never retried by itself: check it, then skip it explicitly.
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
previous = run.started_by
# Whoever resumes takes over the responsibility (and the permission check) from here on.
run.started_by = request.state.user.username
run.status = "resuming"
run.pause_reason = run.pause_detail = None
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
return await _done(db, run, "resume", "resuming", request, previous_owner=previous)
@router.post("/runs/{run_id}/approve")
async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
run = await _run_or_404(db, run_id, lock=True)
if run.status != "awaiting_approval":
raise _not_now(run)
wave = rollout_engine.wave_row(run, run.current_wave)
wave.approved_by, wave.status = request.state.user.username, "preflight"
run.started_by = request.state.user.username
run.status = "preflight"
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
@router.post("/runs/{run_id}/skip-host")
async def skip_host(run_id: uuid.UUID, payload: SkipIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
run = await _run_or_404(db, run_id, lock=True)
if run.status not in rollout_engine.ACTIVE:
raise _not_now(run)
host = next((h for h in run.hosts if h.host_id == payload.host_id), None)
if host is None:
raise api_error(404, ErrorCode.ROLLOUT_HOST_NOT_IN_RUN)
if host.status in ("running", "verifying"):
if not await _cancel_if_pending(db, host.scheduled_maintenance_id):
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
# not fired yet: nothing happened on the firewall
elif host.status not in ("queued", "blocked", "failed"):
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
host.status = "skipped"
host.detail = {"code": "skipped_by", "params": {"user": request.state.user.username, "reason": payload.reason.strip()[:300]}}
host.finished_at = datetime.now(timezone.utc)
return await _done(db, run, "skip_host", "skipped", request, host=host.host_name, reason=payload.reason.strip()[:300])
@router.post("/runs/{run_id}/abort")
async def abort_run(run_id: uuid.UUID, payload: AbortIn, request: Request, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
run = await _run_or_404(db, run_id, lock=True)
if run.status not in rollout_engine.ACTIVE:
raise _not_now(run)
if payload.confirm_name != run.plan_name:
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
now = datetime.now(timezone.utc)
still_running = []
for host in run.hosts:
if host.status in ("running", "verifying") and host.scheduled_maintenance_id:
if await _cancel_if_pending(db, host.scheduled_maintenance_id):
host.status, host.finished_at = "cancelled", now
else:
still_running.append(host.host_name) # an action already on the firewall is finished in a controlled way
elif host.status in ("queued", "blocked"):
host.status, host.finished_at = "cancelled", now
for wave in run.waves:
if wave.status not in ("passed", "skipped"):
wave.status = "cancelled"
run.status, run.finished_at = "cancelled", now
run.pause_reason, run.pause_detail = "aborted", {"by": request.state.user.username, "reason": payload.reason.strip()[:300],
"still_running": still_running}
return await _done(db, run, "abort", "cancelled", request, reason=payload.reason.strip()[:300], still_running=still_running)
+91
View File
@@ -0,0 +1,91 @@
"""Rule analysis: which rule decides a packet (read-only, GET so ``read`` tokens and MCP can use it).
Nothing is stored and nothing is written to a firewall; every request reads the rules, the
loaded pf ruleset and the interfaces (cached for 60 s per firewall, ``refresh=true`` skips
the cache) and evaluates them with services/rule_analysis/engine.py.
"""
import asyncio
import logging
import uuid
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.models.policy import DeviceGroup
from app.services.rule_analysis import service
logger = logging.getLogger(__name__)
router = APIRouter(tags=["rule-analysis"])
_PARALLEL = 5
def _query(src: str, dst: str, proto: str, sport: str | None, dport: str | None, icmptype: str | None,
iface: str | None, direction: str, egress: str | None):
return service.build_query(src=src, dst=dst, proto=proto, sport=sport, dport=dport, icmptype=icmptype,
iface=iface, direction=direction, egress=egress)
async def _run(host: OpnsenseHost, q, refresh: bool) -> dict:
head = {"host_id": str(host.id), "host_name": host.name, "cluster_name": host.cluster_name}
try:
return {**head, **await service.analyze_host(host, q, refresh=refresh), "error": None}
except HTTPException as exc:
return {**head, "error": exc.detail}
except Exception as exc: # noqa: BLE001 - one unreachable firewall must not fail the fleet answer
logger.info("rule analysis failed for %s: %s", host.name, exc)
return {**head, "error": {"code": ErrorCode.ANALYSE_FIREWALL_NICHT_LESBAR.value, "params": {}}}
@router.get("/hosts/{host_id}/rule-analysis/match")
async def host_match(
host_id: uuid.UUID, src: str, dst: str, proto: str = "tcp", sport: str | None = None, dport: str | None = None,
icmptype: str | None = None, iface: str | None = None, direction: str = "in", egress: str | None = None,
refresh: bool = False, db: AsyncSession = Depends(get_db),
):
host = await db.get(OpnsenseHost, host_id)
if host is None:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
q = _query(src, dst, proto, sport, dport, icmptype, iface, direction, egress)
result = await _run(host, q, refresh)
error = result.get("error")
if error:
status = 422 if error.get("code") == ErrorCode.ANALYSE_INTERFACE_UNBEKANNT.value else 502
raise HTTPException(status_code=status, detail=error)
return result
@router.get("/rule-analysis/match")
async def fleet_match(
src: str, dst: str, proto: str = "tcp", sport: str | None = None, dport: str | None = None,
icmptype: str | None = None, iface: str | None = None, direction: str = "in", egress: str | None = None,
host_ids: str | None = Query(None, description="comma separated host ids; default: all enabled hosts"),
group_id: uuid.UUID | None = None, refresh: bool = False, db: AsyncSession = Depends(get_db),
):
q = _query(src, dst, proto, sport, dport, icmptype, iface, direction, egress)
if group_id is not None:
group = await db.scalar(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)).where(DeviceGroup.id == group_id))
if group is None:
raise api_error(404, ErrorCode.GRUPPE_NICHT_GEFUNDEN)
hosts = sorted(group.hosts, key=lambda h: h.name)
elif host_ids:
try:
ids = [uuid.UUID(part.strip()) for part in host_ids.split(",") if part.strip()]
except ValueError:
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="host_ids", value=host_ids[:64]) from None
hosts = list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)).order_by(OpnsenseHost.name))).all())
if len(hosts) != len(set(ids)):
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
else:
hosts = list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)).order_by(OpnsenseHost.name))).all())
gate = asyncio.Semaphore(_PARALLEL)
async def one(host):
async with gate:
return await _run(host, q, refresh)
return list(await asyncio.gather(*(one(h) for h in hosts)))
+4 -1
View File
@@ -18,6 +18,7 @@ from app.services.interface_service import check_interface_consistency
from app.services.opnsense_client import OpnsenseAPIError
from app.services.rule_deploy_service import confirm_rule, deploy_rule, undeploy_rule, revert_deployment
from app.services import rule_adopt_service
from app.services.adopt_common import LINK_ERRORS, link_api_error
from app.schemas.rule import AdoptRequest, UnmanagedRule
router = APIRouter(prefix="/rules", tags=["rules"])
@@ -107,11 +108,13 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
if not host:
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
try:
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid)
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
except rule_adopt_service.RuleNotFoundOnHost as exc:
raise api_error(404, ErrorCode.REGEL_NICHT_GEFUNDEN) from exc
except rule_adopt_service.NotAdoptable as exc:
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
except LINK_ERRORS as exc:
raise link_api_error(exc) from exc
except OpnsenseAPIError as exc:
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
await db.commit()
+8 -1
View File
@@ -24,6 +24,8 @@ class Schedule(BaseModel):
acknowledge_unattended: Literal[True]
# Hand the traffic over to the CARP partner first and switch back afterwards (cluster nodes only).
carp_handover: bool = False
# Check the result automatically afterwards and close the operation (no manual "Wartung abschließen").
auto_verify: bool = False
def _read(sched: ScheduledMaintenance) -> dict:
@@ -32,6 +34,8 @@ def _read(sched: ScheduledMaintenance) -> dict:
"status": sched.status, "actor": sched.actor, "created_at": sched.created_at,
"result": sched.result, "host_operation_id": sched.host_operation_id,
"carp_handover": sched.carp_handover, "phase": sched.phase,
"auto_verify": sched.auto_verify, "origin": sched.origin,
"rollout_run_id": ((sched.progress or {}).get("rollout") or {}).get("run_id"),
}
@@ -80,6 +84,7 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
sched = ScheduledMaintenance(
host_id=host.id, host_name=host.name, action=payload.action, run_at=run_at,
actor=request.state.user.username, created_at=now, carp_handover=payload.carp_handover,
auto_verify=payload.auto_verify,
)
db.add(sched)
await db.commit()
@@ -90,7 +95,9 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
async def cancel_scheduled(host_id: uuid.UUID, schedule_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
await policy_lock(db)
await _host_or_404(db, host_id)
sched = await db.get(ScheduledMaintenance, schedule_id)
# Locked and re-read: the poll loop may be firing this very row right now (it holds the lock meanwhile).
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == schedule_id)
.with_for_update().execution_options(populate_existing=True))
if not sched or sched.host_id != host_id:
raise api_error(404, ErrorCode.GEPLANTE_WARTUNG_NICHT_GEFUNDEN)
if sched.status != "pending":
+4
View File
@@ -7,6 +7,7 @@ from app.database import get_db
from app.errors import ErrorCode, api_error
from app.models.user import AdminUser
from app.routers.auth import current_user
from app.services import rollout_engine
from app.services import update_check_service, updater_client
router = APIRouter(prefix="/system", tags=["system"])
@@ -99,6 +100,9 @@ async def updater_status(db: AsyncSession = Depends(get_db), user: AdminUser = D
"can_update": False, "major_change": False,
"job": None, "phases": [], "log": [], "services": {},
"rollback_available": False, "rollback_loses_data": False, "rollback_version": None,
# A restart in the middle of a firmware rollout is safe (its state is in the database), but the
# UI warns: the run pauses its 15-second steps for as long as OPNexus is down.
"active_rollouts": await rollout_engine.count_active(db),
}
try:
updater = await updater_client.status()
+76
View File
@@ -0,0 +1,76 @@
import uuid
from pydantic import BaseModel, Field
class AdoptObjectRequest(BaseModel):
host_id: uuid.UUID
# Link to this existing central object instead of creating a new one (services/adopt_link_service.py).
# Declared before `uuid`: that field's name shadows the module inside the class body.
link_to: uuid.UUID | None = None
uuid: str = Field(min_length=1, max_length=64) # the object's UUID on the firewall
class AdoptAllRequest(BaseModel):
host_id: uuid.UUID
# Link entries that have exactly one exact central twin instead of creating a duplicate.
link_existing: bool = False
class LinkDiff(BaseModel):
field: str # model field ("content", "destination_port", ...) or "variables" (a {{VARIABLE}} has no value on this host)
central: str
live: str
class LinkTarget(BaseModel):
"""An existing central object that a live object could be linked to (exact match) or is similar to."""
id: uuid.UUID
name: str
targets: list[str] # names of the hosts it is deployed to
diff: list[LinkDiff] = Field(default_factory=list) # empty for an exact match
class UnmanagedObject(BaseModel):
"""An alias / NAT rule found on a host that OPNexus does not manage yet (see object_adopt_service.py)."""
uuid: str
title: str
enabled: bool
adoptable: bool # can be adopted on its own right now
# Adoptable in the same "adopt all" run although not on its own (an alias whose referenced aliases come first).
bulk_adoptable: bool
# Codes such as "unsupported_type" or "no_target"; the UI turns them into text.
reasons: list[str]
reason_details: dict[str, str] = Field(default_factory=dict) # code -> extra text (alias names, type, ...)
# Settings OPNexus does not manage that stay untouched on the firewall (log, pool options, tags, ...).
kept: list[str] = Field(default_factory=list)
# Facts for display, depend on the object type.
info: dict[str, str] = Field(default_factory=dict)
# Central objects whose content, resolved for this host, equals the live object exactly (first = preferred).
matches: list[LinkTarget] = Field(default_factory=list)
# Objects with the same name/description but different content: a hint with the differing fields, never linked.
similar: list[LinkTarget] = Field(default_factory=list)
linkable: bool = False # can be linked to `matches[0]` (or another match) on its own
# What "adopt all" would do: "link" | "new" | "skip" (None only on the adopt-all result).
bulk_action: str | None = None
class AdoptedObject(BaseModel):
uuid: str
title: str
id: uuid.UUID
linked: bool = False # linked to an existing central object instead of newly created
class SkippedObject(BaseModel):
uuid: str
title: str
reasons: list[str]
reason_details: dict[str, str] = Field(default_factory=dict)
class AdoptAllResult(BaseModel):
adopted: list[AdoptedObject]
skipped: list[SkippedObject]
+4
View File
@@ -3,6 +3,8 @@ from datetime import datetime
from pydantic import BaseModel
from app.schemas.gateway import GatewayHealth
class DashboardHost(BaseModel):
id: uuid.UUID
@@ -10,6 +12,8 @@ class DashboardHost(BaseModel):
status: str
carp_status: str | None
cluster_name: str | None
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
gateways: list[GatewayHealth] | None = None
class PendingAction(BaseModel):
+19
View File
@@ -0,0 +1,19 @@
from typing import Literal
from pydantic import BaseModel
# online: all fine · degraded: OPNsense flags loss and/or latency above the gateway's own limits ·
# offline: down · forced_down: switched off by the admin on purpose · unknown: nothing measured
# (monitoring disabled) or the verdict is still pending.
GatewayState = Literal["online", "degraded", "offline", "forced_down", "unknown"]
class GatewayHealth(BaseModel):
name: str
address: str | None = None
status: str # OPNsense' raw verdict: none | down | force_down | delay | loss | delay+loss | …
state: GatewayState
loss: float | None = None # percent
delay: float | None = None # milliseconds
stddev: float | None = None # milliseconds
monitor: str | None = None # the monitored address
+104 -1
View File
@@ -1,10 +1,12 @@
import re
import uuid
from datetime import datetime
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
from pydantic import BaseModel, ConfigDict, Field, field_validator
from app.config import settings
from app.models.notification import DEFAULT_SERVICE_WATCHLIST
SmtpSecurity = Literal["none", "starttls", "ssl"]
ProblemType = Literal[
@@ -14,7 +16,10 @@ ProblemType = Literal[
"backup_stale",
"backup_offsite_failed",
"compliance_finding",
"gateway_down", "resource_threshold", "service_down",
"rollout_paused", "rollout_start_failed",
]
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
class NotificationSettingsRead(BaseModel):
@@ -45,10 +50,28 @@ class NotificationSettingsRead(BaseModel):
scheduled_maintenance_failed_enabled: bool = True
backup_failed_enabled: bool = True
compliance_alerts_enabled: bool = False
gateway_down_enabled: bool = True
resource_threshold_enabled: bool = False
service_down_enabled: bool = False
rollout_problems_enabled: bool = True
gateway_loss_threshold_pct: int = 20
gateway_delay_threshold_ms: int = 500
disk_threshold_pct: int = 90
memory_threshold_pct: int = 95
swap_threshold_pct: int = 50
load_per_cpu_threshold: float = 2.0
alert_confirm_samples: int = 3
resource_confirm_samples: int = 5
service_watchlist: list[str] = Field(default_factory=lambda: list(DEFAULT_SERVICE_WATCHLIST))
poll_interval_seconds: int
updated_at: datetime
@field_validator("service_watchlist", mode="before")
@classmethod
def _default_watchlist(cls, value):
return list(DEFAULT_SERVICE_WATCHLIST) if value is None else value
class NotificationSettingsUpdate(BaseModel):
"""Partial update — everything optional, only provided fields change.
@@ -84,6 +107,34 @@ class NotificationSettingsUpdate(BaseModel):
scheduled_maintenance_failed_enabled: bool | None = None
backup_failed_enabled: bool | None = None
compliance_alerts_enabled: bool | None = None
gateway_down_enabled: bool | None = None
resource_threshold_enabled: bool | None = None
service_down_enabled: bool | None = None
rollout_problems_enabled: bool | None = None
# 0 switches that single threshold off.
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
disk_threshold_pct: int | None = Field(default=None, ge=0, le=100)
memory_threshold_pct: int | None = Field(default=None, ge=0, le=100)
swap_threshold_pct: int | None = Field(default=None, ge=0, le=100)
load_per_cpu_threshold: float | None = Field(default=None, ge=0, le=1000)
alert_confirm_samples: int | None = Field(default=None, ge=1, le=60)
resource_confirm_samples: int | None = Field(default=None, ge=1, le=60)
service_watchlist: list[str] | None = Field(default=None, max_length=100)
@field_validator("service_watchlist")
@classmethod
def _clean_watchlist(cls, value):
if value is None:
return None
names = []
for raw in value:
name = str(raw).strip()
if not re.fullmatch(r"[A-Za-z0-9._+-]{1,64}", name):
raise ValueError("Dienstnamen bestehen aus Buchstaben, Ziffern und . _ + -")
if name not in names:
names.append(name)
return names
poll_interval_seconds: int | None = Field(
default=None, ge=settings.notification_poll_min_interval_seconds
@@ -106,3 +157,55 @@ class NotificationProblemRead(BaseModel):
class TestNotificationRequest(BaseModel):
channel: Literal["webhook", "email"]
class NotificationChannelRead(BaseModel):
"""Never carries the credential; ``config`` is withheld from non-admins (an ntfy topic is a password)."""
model_config = ConfigDict(from_attributes=True)
id: uuid.UUID
name: str
channel_type: ChannelType
enabled: bool
config: dict
secret_set: bool
created_at: datetime
updated_at: datetime
def _channel_name(value: str | None) -> str | None:
if value is None:
return None
if not value.strip():
raise ValueError("Name darf nicht leer sein")
return value.strip()
class NotificationChannelCreate(BaseModel):
name: str = Field(min_length=1, max_length=100)
channel_type: ChannelType
enabled: bool = True
config: dict = Field(default_factory=dict)
# Token, or for Teams the webhook address. Write-only.
secret: str | None = Field(default=None, max_length=2100)
@field_validator("name")
@classmethod
def _clean_name(cls, value):
return _channel_name(value)
class NotificationChannelUpdate(BaseModel):
"""Partial: omitted fields stay. ``secret`` replaces the credential, ``clear_secret`` removes it."""
name: str | None = Field(default=None, min_length=1, max_length=100)
enabled: bool | None = None
config: dict | None = None
secret: str | None = Field(default=None, max_length=2100)
clear_secret: bool = False
@field_validator("name")
@classmethod
def _clean_name(cls, value):
return _channel_name(value)
+8
View File
@@ -4,6 +4,8 @@ from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator
from app.schemas.adopt import LinkTarget
class RuleCreate(BaseModel):
description: str = Field(min_length=1, max_length=255)
@@ -78,6 +80,8 @@ class DeployApproval(BaseModel):
class AdoptRequest(BaseModel):
host_id: uuid.UUID
# Link to this existing central rule instead of creating a new one (services/adopt_link_service.py).
link_to: uuid.UUID | None = None
rule_uuid: str = Field(min_length=1, max_length=64)
@@ -98,3 +102,7 @@ class UnmanagedRule(BaseModel):
adoptable: bool
# Codes such as "negated_source" or "not_quick"; the UI turns them into text.
reasons: list[str]
# Central rules equal to this one (resolved for the host), same-description rules that differ, link possible.
matches: list[LinkTarget] = Field(default_factory=list)
similar: list[LinkTarget] = Field(default_factory=list)
linkable: bool = False
+44
View File
@@ -7,9 +7,19 @@ how the proxy setup works for IP *and* name access) nor an explicitly allowed
origin. Requests without either header (curl, scripts) are not browsers, so
they cannot be riding on a victim's cookie and pass through to normal auth.
Sec-Fetch-Site (1.27): a page on another port of the same host (e.g. a
firewall GUI behind the GUI gateway) is *same-site* to OPNexus, so the
browser attaches the session cookie to its requests and SameSite=Lax does
not help. Cookie-carrying /api requests that the browser itself marks as
same-site or cross-site are therefore refused — unless the Origin is an
explicitly configured frontend origin (development setup with the frontend
on its own port). Clients that send no Sec-Fetch-Site header (curl,
scripts, API tokens) are not browsers and stay unaffected.
SecurityHeadersMiddleware: API responses must never be cached by browsers or
proxies and must not be sniffed or framed.
"""
import re
from urllib.parse import urlparse
from starlette.middleware.base import BaseHTTPMiddleware
@@ -18,6 +28,7 @@ from starlette.responses import JSONResponse
from app.config import settings
from app.errors import ErrorCode
from app.routers.auth import has_session_cookie
_SAFE_METHODS = {"GET", "HEAD", "OPTIONS"}
@@ -35,8 +46,41 @@ def _origin_allowed(origin: str, host_header: str) -> bool:
return f"{parsed.scheme}://{parsed.netloc}".lower() in {o.lower() for o in settings.allowed_origins}
# The SSO provider sends the browser back here by a cross-site navigation;
# the endpoint authenticates by its own state cookie and creates the session.
_SEC_FETCH_EXEMPT = {"/api/auth/oidc/callback"}
def _explicit_frontend_origin(origin: str) -> bool:
origin = origin.rstrip("/").lower()
if not origin or origin == "null":
return False
if settings.allowed_origins:
return origin in {o.lower() for o in settings.allowed_origins}
return re.fullmatch(settings.cors_origin_regex, origin) is not None
def sec_fetch_blocked(request: Request) -> bool:
path = request.url.path
if not (path == "/api" or path.startswith("/api/")) or path in _SEC_FETCH_EXEMPT:
return False
if not has_session_cookie(request):
return False
site = request.headers.get("sec-fetch-site", "").strip().lower()
if site == "cross-site":
return True
if site == "same-site":
return not _explicit_frontend_origin(request.headers.get("origin", ""))
return False # same-origin, none (typed URL/bookmark) or no header (not a browser)
class OriginCheckMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
if sec_fetch_blocked(request):
return JSONResponse(
status_code=403,
content={"detail": {"code": ErrorCode.CROSS_SITE_REQUEST_BLOCKED.value, "params": {}}},
)
if request.method not in _SAFE_METHODS:
source = request.headers.get("origin") or request.headers.get("referer")
if source and source != "null" and not _origin_allowed(source, request.headers.get("host", "")):
+63
View File
@@ -0,0 +1,63 @@
"""Small pieces shared by every "take over what was made by hand on a firewall" feature
(rule_adopt_service.py for filter rules, object_adopt_service.py for aliases and NAT)."""
class NotAdoptable(Exception):
"""The object exists on the host but cannot be described without distortion.
`reasons` are machine-readable codes (the UI translates them); `details` maps a code to
extra text the translation can show (e.g. the alias names a reference points to)."""
def __init__(self, reasons: list[str], details: dict[str, str] | None = None):
super().__init__(", ".join(reasons))
self.reasons = reasons
self.details = details or {}
class NotFoundOnHost(Exception):
"""The UUID is not (or no longer) present in the host's live configuration."""
def on(value) -> bool:
"""OPNsense booleans arrive as "1"/"0" strings, sometimes as real booleans."""
return value in ("1", 1, True)
def clean(value) -> str:
return "" if value is None else str(value).strip()
class LinkTargetNotFound(Exception):
"""The central object chosen to link to was deleted in the meantime."""
class LinkTargetChanged(Exception):
"""The central object no longer equals what is live on the host. `diff` lists the differing fields."""
def __init__(self, diff: list[dict]):
super().__init__(", ".join(d["field"] for d in diff))
self.diff = diff
class HostAlreadyHasObject(Exception):
"""The central object is already deployed to / managed on this host under another UUID."""
class LinkTargetBusy(Exception):
"""The central object has an unconfirmed deployment or a failed rollback; linking waits until it is settled."""
LINK_ERRORS = (LinkTargetNotFound, LinkTargetChanged, HostAlreadyHasObject, LinkTargetBusy)
def link_api_error(exc: Exception):
"""The HTTP error for a failed link (shared by the rule and the object endpoints)."""
from app.errors import ErrorCode, api_error
if isinstance(exc, LinkTargetNotFound):
return api_error(404, ErrorCode.ADOPT_LINK_TARGET_NOT_FOUND)
if isinstance(exc, LinkTargetChanged):
return api_error(409, ErrorCode.ADOPT_LINK_TARGET_CHANGED, fields=", ".join(d["field"] for d in exc.diff))
if isinstance(exc, HostAlreadyHasObject):
return api_error(409, ErrorCode.ADOPT_LINK_HOST_HAS_OBJECT)
return api_error(409, ErrorCode.ADOPT_LINK_BUSY)
+151
View File
@@ -0,0 +1,151 @@
"""Links a hand-made object that was found on a firewall to a central object that already exists, instead
of creating a second central object for it ("Mit vorhandenem zentralem Objekt verknüpfen").
Typical case: the same rule/alias/NAT entry was made by hand on several firewalls (or on both nodes of an HA
pair). The first adoption creates the central object; every further firewall is LINKED to it: the host
becomes another target of that object and gets a deployment row holding the live UUID (remote_uuid) and
the payload OPNexus would send. Like adopting, linking writes NOTHING to the firewall. The live state is
read fresh from the host at link time (the browser only names the central object), so the preview right
afterwards says "unchanged" and a later rollout updates the one object on every host without a duplicate.
A central object is an EXACT match when every field OPNexus manages is equal after the object's
{{VARIABLE}} placeholders are replaced with this host's values (variables_service.resolve). An object with
the same identity (alias name / rule description) but different content is only shown as a hint with the
differing fields and is NEVER linked automatically.
Callers hold policy_lock (same serialisation as adopting), which also makes a repeated link idempotent.
"""
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Callable
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.models.host import OpnsenseHost
from app.models.policy import DeviceGroup
from app.services import variables_service
from app.services.adopt_common import HostAlreadyHasObject, LinkTargetBusy, LinkTargetChanged, LinkTargetNotFound
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
from app.services.policy_service import effective_hosts
LINK_MESSAGE = "Mit vorhandenem Objekt verknüpft"
BUSY_STATUSES = ("pending", "revert_failed") # the central object waits for a confirmation / failed to roll back
@dataclass(frozen=True)
class LinkSpec:
model: type
deployment: type
fk: str # deployment column pointing at the central object
uuid_attr: str # deployment column holding the OPNsense UUID (remote_uuid)
identity: str # attribute naming the object: "name" (alias) or "description"
payload: Callable[[Any, OpnsenseHost], dict]
revision: Callable
@dataclass
class Match:
obj: Any
diff: list[dict] # empty = exact match
busy: bool = False # the central object waits for a confirmation: linking is refused until that is settled
def _norm(value) -> Any:
if value is None:
return ""
return value if isinstance(value, bool) else str(value)
def _show(value) -> str:
return ("true" if value else "false") if isinstance(value, bool) else ("" if value is None else str(value))
def diff_fields(obj, values: dict, host: OpnsenseHost) -> list[dict]:
"""Fields in which the central object (resolved for `host`) differs from the live `values`."""
raw = {key: getattr(obj, key) for key in values}
try:
central = variables_service.resolve(raw, host)
except variables_service.MissingVariables as exc:
# Cannot be compared (and could not be deployed to this host either): report, never match.
return [{"field": "variables", "central": ", ".join(exc.missing[host.name]), "live": ""}]
return [{"field": key, "central": _show(central[key]), "live": _show(values[key])}
for key in values if _norm(central[key]) != _norm(values[key])]
def _options(spec: LinkSpec):
return [
selectinload(spec.model.targets),
selectinload(spec.model.groups).selectinload(DeviceGroup.hosts),
selectinload(spec.model.deployments),
]
async def load_central(db: AsyncSession, spec: LinkSpec) -> list:
objs = list((await db.scalars(select(spec.model).options(*_options(spec)))).unique().all())
await hydrate_hierarchy_targets(db, objs)
return objs
def find(objs: list, spec: LinkSpec, host: OpnsenseHost, values: dict) -> tuple[list[Match], list[Match]]:
"""(exact matches, same-identity objects that differ). Objects that already have a deployment on
this host are not candidates: linking would collide with that deployment."""
exact: list[Match] = []
similar: list[Match] = []
for obj in objs:
if any(d.host_id == host.id for d in obj.deployments):
continue
if _norm(getattr(obj, spec.identity)) != _norm(values[spec.identity]):
continue
d = diff_fields(obj, values, host)
busy = any(dep.status in BUSY_STATUSES for dep in obj.deployments)
(similar if d else exact).append(Match(obj, d, busy))
# prefer the object that already reaches the most hosts (the "real" one), then a stable order
exact.sort(key=lambda m: (-len(effective_hosts(m.obj)), str(m.obj.id)))
return exact, similar
def describe(spec: LinkSpec, match: Match) -> dict:
return {
"id": match.obj.id,
"name": getattr(match.obj, spec.identity),
"targets": [h.name for h in effective_hosts(match.obj)],
"diff": match.diff,
}
async def link(db: AsyncSession, spec: LinkSpec, host: OpnsenseHost, object_id, remote_uuid: str, values: dict):
"""Links the live object `remote_uuid` (whose normalised fields are `values`, read fresh by the caller)
to the central object `object_id`. Returns it; a repeated call for the same pair changes nothing."""
obj = (await db.execute(
select(spec.model).where(spec.model.id == object_id).options(*_options(spec))
.execution_options(populate_existing=True)
)).unique().scalar_one_or_none()
if obj is None:
raise LinkTargetNotFound(str(object_id))
await hydrate_hierarchy_targets(db, [obj])
existing = next((d for d in obj.deployments if d.host_id == host.id), None)
if existing is not None:
if getattr(existing, spec.uuid_attr) == remote_uuid:
return obj # already linked: nothing to do
raise HostAlreadyHasObject(str(object_id))
if any(d.status in BUSY_STATUSES for d in obj.deployments):
raise LinkTargetBusy(str(object_id))
changed = diff_fields(obj, values, host)
if changed:
raise LinkTargetChanged(changed)
if host.id not in {h.id for h in effective_hosts(obj)}:
obj.targets.append(host)
# an association-only change does not touch the row; stale edit forms must still be refused (see edit_rule)
obj.updated_at = datetime.now(timezone.utc)
await db.flush()
spec.revision(db, obj, f"Mit {host.name} verknüpft")
db.add(spec.deployment(**{
spec.fk: obj.id, "host_id": host.id, "status": "success", "message": LINK_MESSAGE,
spec.uuid_attr: remote_uuid, "last_payload": spec.payload(obj, host), "previous_payload": None,
}))
await db.flush()
return obj
+13 -2
View File
@@ -38,7 +38,8 @@ MAX_TOKENS_PER_USER = 50
_LAST_USED_GRANULARITY = timedelta(minutes=1)
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)")
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$")
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
# notification settings the webhook URL/SMTP data): admin scope only.
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
@@ -48,6 +49,7 @@ _ADMIN_ONLY_WRITES = re.compile(
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
r"|^/api/logs/settings$"
r"|^/api/hosts(/tls-probe|/[^/]+)?$"
r"|^/api/hosts/[^/]+/gui-ticket$"
)
@@ -90,7 +92,10 @@ _ROLLOUT_POST = re.compile(
r"|deployments/[^/]+/(remove|retry-rollback)"
r"|maintenance/(execute|carp-execute|schedule)"
r"|network/changes"
r"|services/[^/]+/(start|stop|restart))$"
r"|services/[^/]+/(start|stop|restart)"
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
r"|rollouts/plans/[^/]+/(start|activate)"
r"|rollouts/runs/[^/]+/(resume|approve|skip-host|abort))$"
)
_ROLLOUT_DELETE = re.compile(
r"^/api/(rules|aliases|port-forward|one-to-one|outbound-nat|policy-hierarchy/rules|ipsec/tunnels|openvpn/instances"
@@ -99,7 +104,13 @@ _ROLLOUT_DELETE = re.compile(
)
# Recurring maintenance windows decide WHEN unattended firmware actions run (1.25): every write counts.
_ROLLOUT_ANY_WRITE = re.compile(r"^/api/maintenance-windows(/|$)")
def is_rollout(method: str, path: str) -> bool:
if method in ("POST", "PUT", "PATCH", "DELETE") and _ROLLOUT_ANY_WRITE.match(path):
return True
if method == "POST":
return bool(_ROLLOUT_POST.search(path))
return method == "DELETE" and bool(_ROLLOUT_DELETE.match(path))
@@ -0,0 +1,97 @@
"""Gateway health of a firewall (dpinger numbers, as OPNsense reports them).
``get_gateway_health(host)`` is the one reusable entry point: a live read of
/routes/gateway/status, one :class:`GatewayHealth` per gateway. The notification
engine, the host page and the metrics read the SAME numbers from the periodic
resource sample (``compact`` / ``from_compact``), so a scrape or a page view never
fans out to the firewalls; the 1.25 health gate can call ``get_gateway_health``
directly. Parsing is lenient on purpose: "~" means "nothing measured", never zero.
"""
import re
from app.models.host import OpnsenseHost
from app.schemas.gateway import GatewayHealth, GatewayState
from app.services import opnsense_client as client
_NUMBER = re.compile(r"-?\d+(?:\.\d+)?")
_DEGRADED = {"delay", "loss", "delay+loss"}
def _number(value) -> float | None:
if value is None:
return None
match = _NUMBER.search(str(value))
return float(match.group()) if match else None
def _state(status: str, measured: bool) -> GatewayState:
if status == "down":
return "offline"
if status == "force_down":
return "forced_down"
if status in _DEGRADED:
return "degraded"
if status == "none":
# "none" is also what a gateway without monitoring reports; only numbers make it "online".
return "online" if measured else "unknown"
return "unknown"
def parse_gateway_status(payload) -> list[GatewayHealth]:
items = payload.get("items") if isinstance(payload, dict) else None
result: list[GatewayHealth] = []
for item in items or []:
if not isinstance(item, dict) or not item.get("name"):
continue
status = str(item.get("status") or "").lower()
loss, delay, stddev = _number(item.get("loss")), _number(item.get("delay")), _number(item.get("stddev"))
if loss is not None and loss >= 100:
delay = stddev = None # dpinger prints 0.0 ms when nothing answers: no round trip, not a fast one
measured = loss is not None or delay is not None
address = item.get("address")
monitor = item.get("monitor")
result.append(GatewayHealth(
name=str(item["name"]),
address=None if address in (None, "", "~") else str(address),
status=status or "unknown",
state=_state(status, measured),
loss=loss, delay=delay, stddev=stddev,
monitor=None if monitor in (None, "", "~") else str(monitor),
))
return result
async def get_gateway_health(host: OpnsenseHost) -> list[GatewayHealth]:
"""Live gateway state of ``host``. Raises OpnsenseAPIError if the firewall cannot be asked."""
return parse_gateway_status(await client.get_gateway_status(host))
def compact(gateways: list[GatewayHealth]) -> list[dict]:
"""What the resource sampler stores per sample (small, no addresses)."""
return [
{"name": g.name, "status": g.status, "state": g.state, "loss": g.loss, "delay": g.delay, "stddev": g.stddev}
for g in gateways
]
def from_compact(rows) -> list[GatewayHealth]:
return [
GatewayHealth(
name=r["name"], status=r.get("status") or "unknown", state=r.get("state") or "unknown",
loss=r.get("loss"), delay=r.get("delay"), stddev=r.get("stddev"),
)
for r in rows or [] if isinstance(r, dict) and r.get("name")
]
def is_bad(gateway: GatewayHealth, loss_threshold_pct: float = 0, delay_threshold_ms: float = 0) -> bool:
"""Alarm-worthy: down, or (when the thresholds are on) measured loss / latency beyond them.
A gateway forced down by the admin and one that is not measured are not alarms."""
if gateway.state == "offline":
return True
if loss_threshold_pct and gateway.loss is not None and gateway.loss >= loss_threshold_pct:
return True
if delay_threshold_ms and gateway.delay is not None and gateway.delay >= delay_threshold_ms:
return True
return False
+320
View File
@@ -0,0 +1,320 @@
"""Tickets and gateway sessions for the firewall GUI access (1.27, gui-gateway container).
Flow (see docs in routers/gui_access.py):
* An admin asks for a ticket (POST /api/hosts/{id}/gui-ticket). 32 random bytes, valid 30 s, single use;
only its SHA-256 is stored, bound to the OPNexus session (same hash as sessions.token_hash), the user,
the host and a slot (gateway port = its own browser origin).
* The gateway redeems it through /internal/gui/redeem and gets what it needs to reach the firewall —
base URL, pinned CA, overlay proxy, timeouts. Never the API key or secret.
* Every 20 s the gateway re-checks the gateway session (/internal/gui/revalidate): OPNexus session still
valid, user still admin, switch still on, host still eligible, absolute limit not reached.
A slot stays assigned to one firewall for as long as that browser session has the firewall open; another
firewall gets another slot (another origin), so two firewall GUIs never share an origin at the same time.
"""
import hashlib
import secrets
import uuid
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from urllib.parse import urlsplit
from sqlalchemy import delete, func, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.errors import ErrorCode
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
from app.models.host import OpnsenseHost
from app.models.user import AdminUser, Session
from app.services import audit_service, firewall_url, overlay_service
TICKET_TTL = timedelta(seconds=30)
# 20 s: a switch-off, logout or role change must end a gateway session within 30 s of the change, and
# the last re-check can lie up to one interval before it (live test 2026-10-21: 30 s gave 31 s).
REVALIDATE_SECONDS = 20
MAX_SLOTS = 4
MIN_GATEWAY_TOKEN_LENGTH = 32
AUDIT_KIND = "gui_access"
@dataclass
class GuiAccessError(Exception):
status: int
code: ErrorCode
params: dict
def hash_value(raw: str) -> str:
return hashlib.sha256(raw.encode()).hexdigest()
def _now() -> datetime:
return datetime.now(timezone.utc)
# ── configuration ─────────────────────────────────────────────────────────
def slot_sites() -> dict[int, list[str]]:
"""Slot port -> configured https origins with that port (OPNEXUS_GUI_SITES)."""
sites: dict[int, list[str]] = {}
for origin in settings.gui_site_origins:
try:
parts = urlsplit(origin)
port = parts.port
except ValueError:
continue
if parts.scheme != "https" or not parts.hostname or not port or parts.path not in ("", "/"):
continue
sites.setdefault(port, []).append(f"https://{parts.netloc}")
return sites
def gateway_configured() -> bool:
return len(settings.gui_gateway_token) >= MIN_GATEWAY_TOKEN_LENGTH and bool(slot_sites())
def unavailable_reasons() -> list[str]:
reasons = []
if not settings.cookie_secure:
reasons.append("cookie_secure")
if len(settings.gui_gateway_token) < MIN_GATEWAY_TOKEN_LENGTH:
reasons.append("gateway_token")
if not slot_sites():
reasons.append("gui_sites")
return reasons
def active_slots(row: GuiAccessSettings) -> list[int]:
return sorted(slot_sites())[: max(1, min(row.slot_count, MAX_SLOTS))]
async def get_settings(db: AsyncSession) -> GuiAccessSettings:
row = await db.get(GuiAccessSettings, 1)
if row is None:
row = GuiAccessSettings(id=1, enabled=False, idle_timeout_minutes=15, absolute_timeout_minutes=60, slot_count=MAX_SLOTS)
db.add(row)
await db.flush()
return row
def host_refusal(host: OpnsenseHost) -> str | None:
"""Why this firewall must not be opened through the gateway (None = eligible)."""
if not host.enabled:
return "disabled"
if urlsplit(host.base_url.strip()).scheme != "https" or firewall_url.check_base_url(host.base_url):
return "https"
if not host.verify_tls:
return "tls"
return None
def firewall_origin(base_url: str) -> str:
parts = urlsplit(base_url.strip())
return f"https://{parts.netloc.rsplit('@', 1)[-1]}"
# ── audit ─────────────────────────────────────────────────────────────────
async def _audit(actor: str, *, action: str, status: str, host: OpnsenseHost | None = None,
host_id=None, entity_id=None, message: str | None = None, details: dict | None = None) -> None:
token = audit_service.context.set({"actor": actor, "id": str(uuid.uuid4()), "action": action})
try:
await audit_service.persist(
kind=AUDIT_KIND, action=action, status=status, entity_id=str(entity_id) if entity_id else None,
host_id=str(host.id) if host is not None else (str(host_id) if host_id else None),
host_name=host.name if host is not None else None, message=message, details=details or {},
)
finally:
audit_service.context.reset(token)
# ── tickets ───────────────────────────────────────────────────────────────
def _redeem_origin(slot: int, request_host: str) -> str:
candidates = slot_sites().get(slot, [])
wanted = (urlsplit(f"//{request_host}").hostname or "").lower()
for origin in candidates:
if (urlsplit(origin).hostname or "").lower() == wanted:
return origin
return candidates[0]
async def _pick_slot(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id) -> int:
now = _now()
slots = active_slots(row)
absolute = timedelta(minutes=row.absolute_timeout_minutes)
in_use: dict[int, uuid.UUID] = {}
open_sessions = await db.scalars(select(GuiSession).where(
GuiSession.session_hash == session_hash, GuiSession.ended_at.is_(None), GuiSession.started_at > now - absolute))
for gs in open_sessions:
in_use[gs.slot] = gs.host_id
pending = await db.scalars(select(GuiTicket).where(
GuiTicket.session_hash == session_hash, GuiTicket.redeemed_at.is_(None), GuiTicket.expires_at > now))
for ticket in pending:
in_use.setdefault(ticket.slot, ticket.host_id)
for slot in slots:
if in_use.get(slot) == host_id:
return slot # this firewall is already open in this browser session: same origin again
free = [s for s in slots if s not in in_use]
if not free:
raise GuiAccessError(409, ErrorCode.GUI_NO_FREE_SLOT, {"slots": str(len(slots))})
# Prefer the slot this user had this firewall on before, then the slot the user used least recently
# (an old tab of another firewall on a reused origin is the documented residual risk).
history = (await db.execute(
select(GuiSession.slot, GuiSession.host_id, func.max(GuiSession.started_at))
.where(GuiSession.user_id == user_id).group_by(GuiSession.slot, GuiSession.host_id)
)).all()
for slot, h_id, _ in sorted(history, key=lambda r: r[2], reverse=True):
if h_id == host_id and slot in free:
return slot
last_used = {}
for slot, _, started in history:
last_used[slot] = max(last_used.get(slot, started), started)
epoch = datetime.min.replace(tzinfo=timezone.utc)
return min(free, key=lambda s: (last_used.get(s, epoch), s))
async def issue_ticket(db: AsyncSession, *, user: AdminUser, session_token: str, host: OpnsenseHost,
request_host: str) -> dict:
row = await get_settings(db)
now = _now()
await db.execute(delete(GuiTicket).where(GuiTicket.expires_at < now - timedelta(hours=1)))
session_hash = hash_value(session_token)
slot = await _pick_slot(db, row, session_hash, user.id, host.id)
raw = secrets.token_urlsafe(32)
db.add(GuiTicket(ticket_hash=hash_value(raw), session_hash=session_hash, user_id=user.id, host_id=host.id,
slot=slot, created_at=now, expires_at=now + TICKET_TTL))
await db.commit()
await _audit(user.username, action="ticket", status="accepted", host=host, details={"slot": slot})
return {"url": f"{_redeem_origin(slot, request_host)}/__onx/redeem?t={raw}", "slot": slot,
"expires_in": int(TICKET_TTL.total_seconds())}
# ── gateway side (internal endpoints) ─────────────────────────────────────
async def _live_check(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id,
slot: int) -> tuple[str | None, AdminUser | None, OpnsenseHost | None]:
session = await db.scalar(select(Session).where(Session.token_hash == session_hash))
if session is None or session.expires_at <= _now() or session.user_id != user_id:
return "session", None, None
user = await db.get(AdminUser, user_id)
if user is None or user.role != "admin":
return "role", user, None
if not row.enabled:
return "disabled", user, None
if not settings.cookie_secure or not gateway_configured():
return "config", user, None
host = await db.get(OpnsenseHost, host_id)
if host is None or host_refusal(host):
return "host", user, host
if slot not in active_slots(row):
return "slot", user, host
return None, user, host
def _timing(row: GuiAccessSettings, started_at: datetime) -> dict:
deadline = started_at + timedelta(minutes=row.absolute_timeout_minutes)
return {"idle_timeout_seconds": row.idle_timeout_minutes * 60,
"absolute_deadline": int(deadline.timestamp()),
"revalidate_seconds": REVALIDATE_SECONDS}
async def redeem(db: AsyncSession, *, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
"""{'ok': True, ...upstream data...} or {'ok': False, 'reason': ...}. Any attempt burns the ticket."""
now = _now()
ticket = await db.scalar(select(GuiTicket).where(GuiTicket.ticket_hash == ticket_hash).with_for_update())
if ticket is None:
await db.rollback()
return {"ok": False, "reason": "unknown"}
replayed = ticket.redeemed_at is not None
ticket.redeemed_at = ticket.redeemed_at or now
row = await get_settings(db)
reason = None
if replayed:
reason = "replayed"
elif ticket.expires_at <= now:
reason = "expired"
elif not secrets.compare_digest(ticket.session_hash, session_hash):
reason = "session"
elif ticket.slot != slot:
reason = "slot"
user = host = None
if reason is None:
reason, user, host = await _live_check(db, row, ticket.session_hash, ticket.user_id, ticket.host_id, slot)
if user is None:
user = await db.get(AdminUser, ticket.user_id)
if host is None:
host = await db.get(OpnsenseHost, ticket.host_id)
actor = user.username if user is not None else "unknown"
if reason is not None:
await db.commit()
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
message=reason, details={"slot": slot, "client_ip": client_ip, "reason": reason})
return {"ok": False, "reason": reason}
# The same browser session reopening this slot replaces the earlier gateway session there.
superseded = (await db.scalars(select(GuiSession).where(
GuiSession.session_hash == session_hash, GuiSession.slot == slot, GuiSession.ended_at.is_(None)))).all()
for old in superseded:
old.ended_at, old.end_reason = now, "superseded"
gs = GuiSession(session_hash=session_hash, user_id=user.id, host_id=host.id, slot=slot,
client_ip=(client_ip or None) and client_ip[:45], started_at=now, last_checked_at=now)
db.add(gs)
await db.commit()
await _audit(actor, action="redeem", status="accepted", host=host, entity_id=gs.id,
details={"slot": slot, "client_ip": client_ip})
return {
"ok": True, "gui_session_id": str(gs.id), "host_id": str(host.id), "host_name": host.name,
"upstream_origin": firewall_origin(host.base_url),
"ca_pem": host.tls_ca_pem or None, # public certificate (pin); None = system trust store
"proxy_url": overlay_service.proxy_for(host.base_url),
**_timing(row, now),
}
async def _finish(db: AsyncSession, gs: GuiSession, reason: str) -> None:
now = _now()
if gs.ended_at is None:
gs.ended_at, gs.end_reason = now, reason[:32]
await db.commit()
user = await db.get(AdminUser, gs.user_id)
host = await db.get(OpnsenseHost, gs.host_id)
await _audit(user.username if user else "unknown", action="end", status="accepted", host=host,
host_id=gs.host_id, entity_id=gs.id, message=reason,
details={"slot": gs.slot, "client_ip": gs.client_ip, "reason": reason,
"duration_seconds": int((gs.ended_at - gs.started_at).total_seconds())})
async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str) -> dict:
gs = await db.get(GuiSession, gui_session_id)
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
return {"ok": False, "reason": "unknown"}
if gs.ended_at is not None:
return {"ok": False, "reason": gs.end_reason or "ended"}
row = await get_settings(db)
if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):
await _finish(db, gs, "absolute")
return {"ok": False, "reason": "absolute"}
reason, _, _ = await _live_check(db, row, gs.session_hash, gs.user_id, gs.host_id, gs.slot)
if reason is not None:
await _finish(db, gs, reason)
return {"ok": False, "reason": reason}
gs.last_checked_at = _now()
await db.commit()
return {"ok": True, **_timing(row, gs.started_at)}
async def end(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str, reason: str) -> bool:
gs = await db.get(GuiSession, gui_session_id)
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
return False
if gs.ended_at is None:
await _finish(db, gs, reason)
return True
async def end_all(db: AsyncSession, reason: str) -> int:
"""Switch turned off: mark every open gateway session ended (the gateway drops them on its next check)."""
result = await db.execute(update(GuiSession).where(GuiSession.ended_at.is_(None))
.values(ended_at=_now(), end_reason=reason[:32]))
return result.rowcount or 0
+27 -3
View File
@@ -5,7 +5,7 @@ fan out live calls to every firewall (an offline host would stall it past the
scraper's timeout and a 15 s scrape interval would hammer the fleet). The
inputs are what OPNexus' own pollers already store:
* host reachability and resources — the 60 s resource sampler (a host that
* host reachability, resources, gateway health and service state — the 60 s resource sampler (a host that
produced no sample for ``STALE_AFTER`` is reported down),
* CARP role — the latest observed transition (carp_events),
* open problems — the notification engine's problem table,
@@ -19,11 +19,11 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.models.carp_event import CarpEvent
from app.models.compliance_result import ComplianceResult
from app.models.host import OpnsenseHost
from app.models.notification import NotificationProblem
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
from app.models.resource_sample import HostResourceSample
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.routers.dashboard import collect_pending_actions
from app.services import compliance_service
from app.services import compliance_service, gateway_health_service, notification_settings_service
from app.version import APP_VERSION
CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
@@ -99,6 +99,14 @@ async def render(db: AsyncSession) -> str:
out.family("opnexus_host_memory_used_percent", "Memory in use, percent (last sample).")
out.family("opnexus_host_load1", "1-minute load average (last sample).")
out.family("opnexus_host_disk_used_percent", "Fullest disk, percent (last sample).")
out.family("opnexus_host_swap_used_percent", "Swap in use, percent (last sample; absent without swap).")
out.family("opnexus_host_gateway_up", "1 if the gateway is not down (online, degraded or not monitored), 0 if OPNsense reports it down.")
out.family("opnexus_host_gateway_loss_percent", "Packet loss to the gateway monitor address, percent (last sample).")
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
configured = (await notification_settings_service.get_settings(db)).service_watchlist
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
for host in hosts:
labels = {"host": host.name, "cluster": host.cluster_name or ""}
@@ -115,6 +123,22 @@ async def render(db: AsyncSession) -> str:
):
if value is not None:
out.sample(name, value, **labels)
if sample is not None:
raw = sample.raw or {}
if raw.get("swap_used_pct") is not None:
out.sample("opnexus_host_swap_used_percent", raw["swap_used_pct"], **labels)
for gw in gateway_health_service.from_compact(raw.get("gateways")):
gw_labels = {**labels, "gateway": gw.name}
out.sample("opnexus_host_gateway_up", 0 if gw.state == "offline" else 1, **gw_labels)
for name, value, factor in (
("opnexus_host_gateway_loss_percent", gw.loss, 1), ("opnexus_host_gateway_delay_seconds", gw.delay, 0.001),
("opnexus_host_gateway_stddev_seconds", gw.stddev, 0.001),
):
if value is not None:
out.sample(name, round(value * factor, 6), **gw_labels)
for key, running in sorted((raw.get("services") or {}).items()):
if key.split("/")[0] in watchlist:
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
if carp_state.get(host.id) in ("MASTER", "BACKUP"):
out.sample("opnexus_host_carp_master", 1 if carp_state[host.id] == "MASTER" else 0, **labels)
+234
View File
@@ -0,0 +1,234 @@
"""Monitoring alerts (1.24): gateway down, resource thresholds, stopped services.
Same contract as notification_checks.py (each check returns {dedup_key: context} of what is
bad right now), with one addition: they never judge a single reading. The inputs are the
60-second samples the resource sampler stores (resource_monitor_service, HostResourceSample),
and a problem needs the last N samples to agree before it opens AND before it closes again:
* all N bad -> problem,
* all N good -> no problem,
* anything in between -> keep the current state (open stays open, closed stays closed).
So a single spike never alerts, a flapping link does not open and close the alert every few
minutes, and a sample that could not be read ("unknown") neither raises nor clears anything.
A host without a fresh sample is not judged at all (check_host_offline reports it); its open
problems are kept.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.host import OpnsenseHost
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
from app.models.resource_sample import HostResourceSample
from app.services import gateway_health_service, notification_settings_service
from app.services.notification_dedup import Problems
# The sampler runs every 60 s; three missed rounds mean the host is not reachable (same limit as metrics_service).
STALE_AFTER = timedelta(seconds=180)
# How far back a service must have been seen running before its absence is reported (samples, ~2 h at 60 s).
SERVICE_EVIDENCE_SAMPLES = 120
_DISK_TYPES = {"ufs", "zfs"}
def verdict(flags: list[bool | None], confirm: int, is_open: bool) -> bool:
"""flags: newest first; True = bad, False = good, None = unknown."""
window = flags[:confirm]
if len(window) >= confirm:
if all(f is True for f in window):
return True
if all(f is False for f in window):
return False
return is_open
async def _enabled_hosts(db: AsyncSession) -> list[OpnsenseHost]:
return list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)))).all())
async def _open_keys(db: AsyncSession, problem_type: str) -> set[str]:
rows = await db.scalars(
select(NotificationProblem.dedup_key).where(
NotificationProblem.problem_type == problem_type, NotificationProblem.status == "open"
)
)
return set(rows.all())
async def _recent(db: AsyncSession, host_id, limit: int, *raw_keys: str) -> list[dict]:
"""The newest ``limit`` samples of a host, newest first, with just the JSON keys asked for."""
columns = [HostResourceSample.sampled_at, HostResourceSample.mem_used_pct, HostResourceSample.load_avg_1m]
columns += [HostResourceSample.raw[key].as_json() for key in raw_keys]
rows = (
await db.execute(
select(*columns).where(HostResourceSample.host_id == host_id).order_by(HostResourceSample.sampled_at.desc()).limit(limit)
)
).all()
return [
{"sampled_at": r[0], "mem_used_pct": r[1], "load_avg_1m": r[2], **{k: r[3 + i] for i, k in enumerate(raw_keys)}}
for r in rows
]
def _fresh(samples: list[dict]) -> bool:
return bool(samples) and datetime.now(timezone.utc) - samples[0]["sampled_at"] <= STALE_AFTER
def _keys_of_host(open_keys: set[str], host: OpnsenseHost) -> set[str]:
prefix = f"{host.id}:"
return {k[len(prefix):] for k in open_keys if k.startswith(prefix)}
# ── Gateways ────────────────────────────────────────────────────────────────
def _gateway_reason(g) -> str:
if g.state == "offline":
return "Gateway offline"
parts = []
if g.loss is not None:
parts.append(f"Paketverlust {g.loss:g} %")
if g.delay is not None:
parts.append(f"Latenz {g.delay:g} ms")
return ", ".join(parts) or g.status
async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
cfg = await notification_settings_service.get_settings(db)
confirm = max(1, cfg.alert_confirm_samples)
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
open_keys = await _open_keys(db, "gateway_down")
problems = Problems()
for host in await _enabled_hosts(db):
samples = await _recent(db, host.id, confirm, "gateways")
if not _fresh(samples):
problems.preserve_prefixes.add(f"{host.id}:")
continue
parsed = [
None if s["gateways"] is None else {g.name: g for g in gateway_health_service.from_compact(s["gateways"])}
for s in samples
]
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
for name in sorted(names):
flags: list[bool | None] = []
for p in parsed:
if p is None:
flags.append(None)
else:
g = p.get(name) # a gateway that no longer exists is "fine": its problem resolves
flags.append(False if g is None else gateway_health_service.is_bad(g, loss_limit, delay_limit))
key = f"{host.id}:{name}"
if not verdict(flags, confirm, key in open_keys):
continue
latest = next((p[name] for p in parsed if p and name in p), None)
problems[key] = {
"host_id": str(host.id), "host_name": host.name, "gateway": name,
"grund": _gateway_reason(latest) if latest else "keine aktuellen Messwerte",
"status": latest.status if latest else None,
"paketverlust_prozent": latest.loss if latest else None,
"latenz_ms": latest.delay if latest else None,
}
return problems
# ── Resources ───────────────────────────────────────────────────────────────
def disk_groups(devices) -> dict[str, tuple[str, float]]:
"""{group: (label, used %)} — one entry per ZFS pool / UFS device, the fullest dataset of a pool wins."""
groups: dict[str, tuple[str, float]] = {}
for dev in devices or []:
if not isinstance(dev, dict) or dev.get("type") not in _DISK_TYPES:
continue
try:
pct = float(dev["used_pct"])
except (KeyError, TypeError, ValueError):
continue
device = str(dev.get("device") or dev.get("mountpoint") or "?")
group = device.split("/")[0] if dev["type"] == "zfs" else device
label = group if dev["type"] == "zfs" else str(dev.get("mountpoint") or device)
if group not in groups or pct > groups[group][1]:
groups[group] = (label, pct)
return groups
async def check_resource_threshold(db: AsyncSession) -> dict[str, dict]:
cfg = await notification_settings_service.get_settings(db)
confirm = max(1, cfg.resource_confirm_samples)
open_keys = await _open_keys(db, "resource_threshold")
problems = Problems()
def over(value, limit) -> bool | None:
if not limit or value is None:
return None if value is None else False
return value >= limit
for host in await _enabled_hosts(db):
samples = await _recent(db, host.id, confirm, "disk", "swap_used_pct", "cpu_threads")
if not _fresh(samples):
problems.preserve_prefixes.add(f"{host.id}:")
continue
base = {"host_id": str(host.id), "host_name": host.name}
was_open = _keys_of_host(open_keys, host)
# Disk: one problem per pool/device.
disk = [None if s["disk"] is None else disk_groups(s["disk"]) for s in samples]
group_names = {g for d in disk if d for g in d} | {k[5:] for k in was_open if k.startswith("disk:")}
for group in sorted(group_names):
flags = [None if d is None else (over(d[group][1], cfg.disk_threshold_pct) if group in d else False) for d in disk]
if verdict(flags, confirm, f"disk:{group}" in was_open):
label, pct = next((d[group] for d in disk if d and group in d), (group, None))
problems[f"{host.id}:disk:{group}"] = {
**base, "ressource": f"Festplatte {label}", "wert_prozent": pct, "schwelle_prozent": cfg.disk_threshold_pct,
}
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
if verdict(flags, confirm, key in was_open):
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": f"{value}{unit}", "schwelle": f"{threshold}{unit}"}
latest = samples[0]
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
latest["mem_used_pct"], cfg.memory_threshold_pct, " %")
simple("swap", "Swap", [over(s["swap_used_pct"], cfg.swap_threshold_pct) for s in samples],
latest["swap_used_pct"], cfg.swap_threshold_pct, " %")
def per_cpu(s):
if s["load_avg_1m"] is None or not s["cpu_threads"]:
return None # thread count unknown (older firmware): the load cannot be judged
return s["load_avg_1m"] / s["cpu_threads"]
simple("cpu", "CPU-Last (pro Thread)", [over(per_cpu(s), cfg.load_per_cpu_threshold) for s in samples],
None if per_cpu(latest) is None else round(per_cpu(latest), 2), cfg.load_per_cpu_threshold, "")
return problems
# ── Services ────────────────────────────────────────────────────────────────
async def check_service_down(db: AsyncSession) -> dict[str, dict]:
cfg = await notification_settings_service.get_settings(db)
confirm = max(1, cfg.alert_confirm_samples)
watch = set(cfg.service_watchlist if cfg.service_watchlist is not None else DEFAULT_SERVICE_WATCHLIST)
open_keys = await _open_keys(db, "service_down")
problems = Problems()
for host in await _enabled_hosts(db):
samples = await _recent(db, host.id, confirm + SERVICE_EVIDENCE_SAMPLES, "services")
if not _fresh(samples):
problems.preserve_prefixes.add(f"{host.id}:")
continue
listings = [s["services"] for s in samples]
was_open = _keys_of_host(open_keys, host)
candidates = {k for k in (listings[0] or {}) if k.split("/")[0] in watch} | was_open
for svc in sorted(candidates):
if svc.split("/")[0] not in watch:
continue # taken off the watchlist: the problem resolves
flags = [None if sv is None else (sv[svc] == 0 if svc in sv else False) for sv in listings]
bad = verdict(flags, confirm, svc in was_open)
# Only a service that was seen running before counts: one that never ran (enabled but idle,
# e.g. DHCPv6 without a prefix) or was stopped on purpose long ago is not a failure.
if bad and svc not in was_open and not any(sv and sv.get(svc) == 1 for sv in listings[confirm:]):
bad = False
if bad:
problems[f"{host.id}:{svc}"] = {
"host_id": str(host.id), "host_name": host.name, "dienst": svc.split("/")[0],
"hinweis": "Dienst läuft nicht mehr; absichtlich gestoppt? Dann aus der Überwachungsliste nehmen.",
}
return problems
@@ -0,0 +1,250 @@
"""Extra notification channels: ntfy, Gotify, Telegram, Microsoft Teams, Matrix.
They sit beside SMTP and the generic webhook (notification_delivery.py) and reuse its message
text. Each channel is a row in ``notification_channels`` (see models/notification.py); the
credential never touches the database: it is stored through notification_secret_store as
``channel_<id>`` (mode 0600 file) and read only at send time. Where the credential is part of the
URL (a Telegram bot token, the whole Teams webhook URL) the URL is built per request, and the
HTTP request log is muted while it is in flight (``url_guard.muted``), so neither the logs nor an
error message ever contain it.
API formats, checked against the vendors' documentation:
* ntfy — JSON publish: POST <server>/ with {"topic", "title", "message", "priority", "tags"}
(docs.ntfy.sh/publish, "Publish as JSON"); optional ``Authorization: Bearer <token>``.
* Gotify — POST <server>/message, header ``X-Gotify-Key: <app token>`` (keeps the token out of
the URL), JSON {"title", "message", "priority"} (gotify.net/docs/pushmsg).
* Telegram — POST <api>/bot<token>/sendMessage, JSON {"chat_id", "text"}; text limit 4096
characters; answers {"ok": true, ...} (core.telegram.org/bots/api#sendmessage).
* Teams — POST to the (Workflows / incoming) webhook URL with a message envelope carrying an
Adaptive Card (learn.microsoft.com ... add-incoming-webhook); the URL is the secret.
* Matrix — PUT <homeserver>/_matrix/client/v3/rooms/<roomId>/send/m.room.message/<txnId> with
``Authorization: Bearer <access token>``, JSON {"msgtype": "m.text", "body"}; a fresh
transaction id per message (Matrix client-server API, "Sending events to a room").
"""
import re
import uuid
from urllib.parse import quote
import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.models.notification import CHANNEL_TYPES, NotificationChannel, NotificationProblem
from app.services import notification_delivery, notification_secret_store, url_guard
from app.services.notification_delivery import NotificationDeliveryError
TELEGRAM_LIMIT = 4096
TEAMS_LIMIT = 20000
_NTFY_TOPIC = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
_MATRIX_ROOM = re.compile(r"^![^\s:]+:[^\s/]+$")
# type -> (required config keys, secret required?)
_SPEC = {
"ntfy": ({"topic"}, False),
"gotify": ({"server_url"}, True),
"telegram": ({"chat_id"}, True),
"teams": (set(), True),
"matrix": ({"homeserver_url", "room_id"}, True),
}
assert set(_SPEC) == set(CHANNEL_TYPES)
def secret_name(channel_id) -> str:
return f"channel_{channel_id}"
def clean_config(channel_type: str, config: dict) -> dict:
"""Validates and normalises the non-secret settings of a channel. Raises ValueError (German reason)."""
if channel_type not in _SPEC:
raise ValueError("Unbekannter Kanaltyp")
config = dict(config or {})
out: dict = {}
def url(key: str, default: str | None = None) -> str | None:
raw = str(config.get(key) or "").strip() or default
if raw is None:
return None
return url_guard.validate_url(raw).rstrip("/")
def integer(key: str, low: int, high: int, default: int) -> int:
value = config.get(key)
if value in (None, ""):
return default
try:
number = int(value)
except (TypeError, ValueError):
raise ValueError(f"{key}: ganze Zahl erwartet") from None
if not low <= number <= high:
raise ValueError(f"{key}: Wert zwischen {low} und {high} erwartet")
return number
if channel_type == "ntfy":
out["server_url"] = url("server_url", "https://ntfy.sh")
topic = str(config.get("topic") or "").strip()
if not _NTFY_TOPIC.match(topic):
raise ValueError("Thema: 1–64 Zeichen aus Buchstaben, Ziffern, _ und -")
out["topic"] = topic
out["priority"] = integer("priority", 1, 5, 4)
elif channel_type == "gotify":
out["server_url"] = url("server_url")
if not out["server_url"]:
raise ValueError("Serveradresse fehlt")
out["priority"] = integer("priority", 0, 10, 5)
elif channel_type == "telegram":
chat_id = str(config.get("chat_id") or "").strip()
if not re.match(r"^(-?\d{1,20}|@[A-Za-z0-9_]{4,64})$", chat_id):
raise ValueError("Chat-ID: Zahl (z. B. -1001234567890) oder @Kanalname")
out["chat_id"] = chat_id
out["api_base"] = url("api_base", "https://api.telegram.org")
elif channel_type == "matrix":
out["homeserver_url"] = url("homeserver_url")
if not out["homeserver_url"]:
raise ValueError("Homeserver-Adresse fehlt")
room = str(config.get("room_id") or "").strip()
if not _MATRIX_ROOM.match(room):
raise ValueError("Raum-ID: Form !abc123:example.org (nicht der Alias #raum:…)")
out["room_id"] = room
return out
def clean_secret(channel_type: str, secret: str) -> str:
value = (secret or "").strip()
if not value:
raise ValueError("Zugangsdaten fehlen")
if channel_type == "teams":
value = url_guard.validate_url(value, max_length=2000)
if not value.startswith("https://"):
raise ValueError("Die Teams-Webhook-Adresse muss mit https:// beginnen")
elif any(c.isspace() for c in value) or len(value) > 500:
raise ValueError("Token: keine Leerzeichen, höchstens 500 Zeichen")
return value
def missing_requirements(channel: NotificationChannel) -> str | None:
required, needs_secret = _SPEC[channel.channel_type]
if any(not channel.config.get(k) for k in required):
return "Konfiguration unvollständig"
if needs_secret and not channel.secret_set:
return "Zugangsdaten fehlen"
return None
# ── Message formatting (one function per channel, pure — unit-testable) ───────────
def _parts(problem: NotificationProblem, transition: str) -> tuple[str, str]:
"""(title, details) — the title repeated at the top of render_message's body is dropped."""
subject, body = notification_delivery.render_message(problem, transition)
details = body.split("\n", 2)[2] if body.count("\n") >= 2 else ""
return subject, details.strip()
def _clip(text: str, limit: int) -> str:
return text if len(text) <= limit else text[: limit - 1] + "…"
def format_ntfy(config: dict, problem, transition: str) -> dict:
title, details = _parts(problem, transition)
resolved = transition == "resolved"
return {
"topic": config["topic"], "title": title, "message": details or title,
"priority": 3 if resolved else int(config.get("priority", 4)),
"tags": ["white_check_mark"] if resolved else ["warning"],
}
def format_gotify(config: dict, problem, transition: str) -> dict:
title, details = _parts(problem, transition)
return {"title": title, "message": details or title, "priority": 1 if transition == "resolved" else int(config.get("priority", 5))}
def format_telegram(config: dict, problem, transition: str) -> dict:
title, details = _parts(problem, transition)
return {
"chat_id": config["chat_id"], "text": _clip(f"{title}\n\n{details}".strip(), TELEGRAM_LIMIT),
"disable_web_page_preview": True,
}
def format_teams(config: dict, problem, transition: str) -> dict:
title, details = _parts(problem, transition)
body: list[dict] = [{"type": "TextBlock", "text": title, "weight": "Bolder", "size": "Medium", "wrap": True}]
if details:
body.append({"type": "TextBlock", "text": _clip(details, TEAMS_LIMIT), "wrap": True})
return {
"type": "message",
"attachments": [{
"contentType": "application/vnd.microsoft.card.adaptive", "contentUrl": None,
"content": {
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"type": "AdaptiveCard", "version": "1.4", "body": body,
},
}],
}
def format_matrix(config: dict, problem, transition: str) -> dict:
title, details = _parts(problem, transition)
return {"msgtype": "m.text", "body": f"{title}\n\n{details}".strip()}
# ── Sending ───────────────────────────────────────────────────────────────────────
def _request_for(channel: NotificationChannel, secret: str, problem, transition: str) -> tuple[str, str, dict, dict, str | None]:
"""(method, url, headers, json body, expected "ok" check key)."""
cfg, kind = channel.config, channel.channel_type
if kind == "ntfy":
headers = {"Authorization": f"Bearer {secret}"} if secret else {}
return "POST", cfg["server_url"].rstrip("/") + "/", headers, format_ntfy(cfg, problem, transition), None
if kind == "gotify":
return "POST", cfg["server_url"].rstrip("/") + "/message", {"X-Gotify-Key": secret}, format_gotify(cfg, problem, transition), None
if kind == "telegram":
return "POST", f"{cfg['api_base'].rstrip('/')}/bot{secret}/sendMessage", {}, format_telegram(cfg, problem, transition), "ok"
if kind == "teams":
return "POST", secret, {}, format_teams(cfg, problem, transition), None
room = quote(cfg["room_id"], safe="")
url = f"{cfg['homeserver_url'].rstrip('/')}/_matrix/client/v3/rooms/{room}/send/m.room.message/{uuid.uuid4().hex}"
return "PUT", url, {"Authorization": f"Bearer {secret}"}, format_matrix(cfg, problem, transition), None
async def send(channel: NotificationChannel, problem: NotificationProblem, transition: str) -> None:
problem_missing = missing_requirements(channel)
if problem_missing:
raise NotificationDeliveryError(f"Kanal „{channel.name}“: {problem_missing}")
secret = notification_secret_store.read_secret(secret_name(channel.id)) or ""
if channel.secret_set and not secret:
raise NotificationDeliveryError(f"Kanal „{channel.name}“: Zugangsdaten nicht lesbar – bitte neu setzen")
method, url, headers, payload, ok_key = _request_for(channel, secret, problem, transition)
label = f"Kanal „{channel.name}“ ({channel.channel_type})"
with url_guard.muted():
try:
url_guard.validate_url(url, max_length=2100)
await url_guard.ensure_resolved_safe(url)
except ValueError as exc:
raise NotificationDeliveryError(f"{label}: {exc}") from None
try:
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
resp = await client.request(method, url, json=payload, headers=headers)
resp.raise_for_status()
if ok_key and not (resp.json() or {}).get(ok_key):
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
except httpx.HTTPError as exc:
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
except ValueError:
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
async def send_test(channel: NotificationChannel) -> None:
fake = NotificationProblem(
problem_type="test", dedup_key="test",
context={"message": "Dies ist eine Testbenachrichtigung von OPNexus."},
)
await send(channel, fake, "opened")
async def enabled_channels(db: AsyncSession) -> list[NotificationChannel]:
rows = await db.scalars(select(NotificationChannel).where(NotificationChannel.enabled.is_(True)).order_by(NotificationChannel.name))
return list(rows.all())
@@ -363,3 +363,49 @@ async def check_compliance(db: AsyncSession) -> dict[str, dict]:
"beispiele": ", ".join(params.get("examples", [])) or None,
}
return problems
# ── Rollouts (1.25) ─────────────────────────────────────────────────────────
def _short(detail) -> str | None:
if not detail:
return None
parts = []
for key, value in detail.items():
if isinstance(value, list):
value = ", ".join(str(v) for v in value[:8])
parts.append(f"{key}: {value}")
return "; ".join(parts)[:500]
async def check_rollout_paused(db: AsyncSession) -> dict[str, dict]:
"""Open while a rollout run stands still waiting for a human (gate, failed host, window gone...)."""
from app.models.rollout import RolloutRun
problems: dict[str, dict] = {}
for run in (await db.scalars(select(RolloutRun).where(RolloutRun.status == "paused"))).all():
wave = next((w.label for w in run.waves if w.index == run.current_wave), None)
problems[str(run.id)] = {
"run_id": str(run.id), "plan": run.plan_name, "grund": run.pause_reason, "welle": wave,
"zielversion": run.target_version, "details": _short(run.pause_detail),
}
return problems
ROLLOUT_START_FAILURE_WINDOW = timedelta(days=3)
async def check_rollout_start_failed(db: AsyncSession) -> dict[str, dict]:
"""An automatic start at a window occurrence did not happen (event: closed without a 'resolved' notice)."""
from app.models.rollout import RolloutPlan
since = datetime.now(timezone.utc) - ROLLOUT_START_FAILURE_WINDOW
problems: dict[str, dict] = {}
for plan in (await db.scalars(select(RolloutPlan).where(RolloutPlan.auto_start_error.is_not(None)))).all():
error = plan.auto_start_error or {}
at = datetime.fromisoformat(error["at"]) if error.get("at") else None
if at is None or at < since:
continue
problems[f"{plan.id}:{error.get('occurrence')}"] = {
"plan_id": str(plan.id), "plan": plan.name, "grund": error.get("code"), "termin": error.get("occurrence"),
"details": _short(error.get("params")),
}
return problems
+17 -4
View File
@@ -16,7 +16,7 @@ import httpx
from app.config import settings
from app.models.notification import NotificationProblem, NotificationSettings
from app.services import notification_secret_store
from app.services import notification_secret_store, url_guard
class NotificationDeliveryError(Exception):
@@ -37,6 +37,12 @@ _LABELS = {
"backup_stale": "Kein aktuelles Backup",
"backup_offsite_failed": "Offsite-Backup fehlgeschlagen",
"compliance_finding": "Sicherheitsbefund",
"gateway_down": "Gateway gestört",
"resource_threshold": "Ressourcen-Schwelle überschritten",
"service_down": "Dienst ausgefallen",
"rollout_paused": "Rollout angehalten",
"rollout_start_failed": "Rollout nicht automatisch gestartet",
"test": "Testbenachrichtigung",
}
@@ -84,9 +90,16 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
headers["X-OPNexus-Signature"] = f"sha256={signature}"
try:
async with httpx.AsyncClient(timeout=settings.opnsense_timeout) as client:
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
resp.raise_for_status()
url_guard.validate_url(settings_row.webhook_url)
await url_guard.ensure_resolved_safe(settings_row.webhook_url)
except ValueError as exc:
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
try:
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
with url_guard.muted():
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
resp.raise_for_status()
except httpx.HTTPError as exc:
# str(exc) names the full URL, which often is the credential itself
# (token in path/query) — keep it out of messages and logs.
+21 -6
View File
@@ -8,8 +8,8 @@ from datetime import datetime, timezone
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.notification import NotificationProblem, NotificationSettings
from app.services import notification_checks, notification_dedup, notification_delivery
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
from app.services import monitor_checks, notification_channels, notification_checks, notification_dedup, notification_delivery
from app.services import notification_settings_service
logger = logging.getLogger(__name__)
@@ -34,14 +34,21 @@ _CHECKS = [
("backup_stale", "backup_failed_enabled", notification_checks.check_backup_stale),
("backup_offsite_failed", "backup_failed_enabled", notification_checks.check_backup_offsite),
("compliance_finding", "compliance_alerts_enabled", notification_checks.check_compliance),
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
("service_down", "service_down_enabled", monitor_checks.check_service_down),
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
]
# Event-like problems: when they drop out of the window nothing was "fixed", so no
# "behoben" message goes out for them.
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed"}
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
async def _deliver(settings_row: NotificationSettings, problem: NotificationProblem, transition: str) -> None:
async def _deliver(
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
) -> None:
"""Best-effort, not an audit trail: attempted once per transition and
never retried, even on failure — avoids retry-storming a broken webhook
URL or SMTP relay every poll cycle."""
@@ -60,6 +67,13 @@ async def _deliver(settings_row: NotificationSettings, problem: NotificationProb
except notification_delivery.NotificationDeliveryError:
logger.exception("Email delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
for channel in channels:
try:
await notification_channels.send(channel, problem, transition)
except notification_delivery.NotificationDeliveryError:
# The error text names the channel and the HTTP status, never the URL or token.
logger.exception("Channel %s (%s) delivery failed for %s/%s", channel.name, channel.channel_type, problem.problem_type, problem.dedup_key)
if transition == "resolved":
problem.resolved_notified_at = _now()
else:
@@ -69,6 +83,7 @@ async def _deliver(settings_row: NotificationSettings, problem: NotificationProb
async def run_notification_checks(db: AsyncSession) -> dict:
settings_row = await notification_settings_service.get_settings(db)
result = {"opened": 0, "resolved": 0}
channels = await notification_channels.enabled_channels(db)
for problem_type, enabled_flag, check_fn in _CHECKS:
if not getattr(settings_row, enabled_flag):
@@ -85,9 +100,9 @@ async def run_notification_checks(db: AsyncSession) -> dict:
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
for problem in opened:
await _deliver(settings_row, problem, "opened")
await _deliver(settings_row, problem, "opened", channels)
for problem in resolved:
await _deliver(settings_row, problem, "resolved")
await _deliver(settings_row, problem, "resolved", channels)
result["opened"] += len(opened)
result["resolved"] += len(resolved)
@@ -26,6 +26,10 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled",
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
"service_watchlist",
"poll_interval_seconds",
]
for field in simple_fields:
@@ -0,0 +1,627 @@
"""Takes over aliases, port forwards, 1:1 NAT and outbound NAT rules that were created by hand on a
firewall, so they can be edited, versioned and deployed from OPNexus like any other object. This is
the generalisation of rule_adopt_service.py (filter rules keep their own module: their model has
more special cases and the code is relied upon).
Adoption itself writes NOTHING to the firewall. It records the object as it is right now (read fresh
from the host at adoption time, never from what the browser sent) together with its OPNsense UUID
(`remote_uuid` = the deployment's opnsense_*_uuid), and stores the payload OPNexus would send for it as
the deployment's last_payload. So the preview directly afterwards says "unchanged", a later deploy
updates the same object via setItem/setRule (no duplicate), and the normal preview → push →
confirm/auto-revert flow applies from the first change on.
An object that already exists centrally for another firewall is LINKED instead of adopted a second time
(adopt_link_service.py: exact match after resolving the host's variables, never automatic for a mere
resemblance).
Only objects OPNexus can describe without distortion are adopted. Anything the model cannot express
is refused with machine-readable reasons (the UI translates them) instead of being imported in a way
that would silently change its meaning on the next edit. Fields OPNexus does not manage (log flag of a
port forward, pool options, tags, categories, ...) are never sent, and OPNsense's set*/setItem only
overwrites the posted fields (ApiMutableModelControllerBase::setBase -> setNodes), so they stay as
they are; the candidate list shows them as `kept`.
Evidence (OPNsense core master, src/opnsense/mvc/app/…, read 2026-10-20, and live against OPNsense
26.7 on test-fw-3 on the same day with hand-made objects of every case below):
* Aliases — controllers/OPNsense/Firewall/Api/AliasController.php::searchItemAction -> searchBase
("aliases.alias"), model models/OPNsense/Firewall/Alias.xml. Row keys used: uuid, name, type
("host", "network", "port", "networkgroup", "mac", "asn", plus the unsupported "url", "urltable",
"urljson", "geoip", "dynipv6host", "authgroup", and the system types "internal"/"external"),
enabled "1"/"0", content (entries separated by "\n", live-verified; for "authgroup" the controller
rewrites it to group names), description, proto (only meaningful for asn/geoip). The built-in rows
are bogons, bogonsv6, sshlockout, virusprot (type external) and one __<if>_network per interface
(type internal); they are OPNsense's own and never adoptable. A "networkgroup" lists alias NAMES;
host/network/port aliases may also contain alias names (nesting).
* Port forward — controllers/OPNsense/Firewall/Api/DNatController.php::searchRuleAction merges the
configured rules (getFlatNodes: dotted keys "source.network", "source.port", "source.not",
"destination.network", "destination.port", "destination.not", plus descr, disabled, interface,
ipprotocol, protocol, target, local-port, natreflection, nordr, pass, log, poolopts, tag, tagged,
nosync, categories, sequence) with synthetic automatic rows (uuid "automatic_*"/"lockout_*",
is_automatic true) that setRule/delRule cannot target. models/OPNsense/Firewall/DNat.xml: interface is
Multiple=Y (comma separated), ipprotocol may be blank/inet/inet6/inet46, protocol blank = any,
`nordr` ("no redirect") requires an empty target/local-port (live: "Target must be empty when No
RDR is set"), `pass` ("" manual / "pass" / "rule") links an associated filter rule, natreflection is
""/purenat/disable. OPNexus sends pass "" and not "0", so a rule using one of those would change
meaning on its first edit and is refused.
* 1:1 NAT — OneToOneController::searchRuleAction (flat keys), model Filter.xml onetoone.rule:
enabled, log, interface, type (binat/nat), source_net/source_not, destination_net/destination_not,
external, natreflection ""/enable/disable, description. All of these are managed by OPNexus.
* Outbound NAT — SourceNatController::searchRuleAction returns the configured rules ONLY when the
global mode (filter.settings.nat.snat_mode) is hybrid or advanced; in automatic/disabled mode the
manual rules do not show up at all, so adopting is refused up front (outbound_nat_mode.py) instead
of presenting an empty list. Model Filter.xml snatrules.rule: `nonat` rules ("do not NAT") are not
expressible and refused; staticnatport, poolopts, log, tag, tagged, nosync, endpoint-independent
are left alone.
"""
import re
from dataclasses import dataclass, field
from typing import Any, Awaitable, Callable, get_args
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.errors import ErrorCode, api_error
from app.models.alias import FirewallAlias
from app.models.alias_deployment import AliasDeployment
from app.models.host import OpnsenseHost
from app.models.one_to_one import OneToOneRule
from app.models.one_to_one_deployment import OneToOneDeployment
from app.models.outbound_nat import OutboundNatRule
from app.models.outbound_nat_deployment import OutboundNatDeployment
from app.models.port_forward import PortForwardRule
from app.models.port_forward_deployment import PortForwardDeployment
from app.schemas.alias import AliasType
from app.services import alias_deploy_service, alias_policy_service
from app.services import one_to_one_deploy_service, one_to_one_policy_service
from app.services import opnsense_client, outbound_nat_deploy_service, outbound_nat_policy_service
from app.services import port_forward_deploy_service, port_forward_policy_service
from app.services import adopt_link_service
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on
from app.services.nat_live_service import split_rows
from app.services.outbound_nat_mode import host_mode
ADOPT_NOTE = "Von Firewall übernommen"
ADOPT_MESSAGE = "Von der Firewall übernommen"
ALIAS_NAME = re.compile(r"^[A-Za-z0-9_]+$") # schemas/alias.py AliasCreate.name
ALIAS_TYPES = frozenset(get_args(AliasType))
ALIAS_CONTENT_MAX = 8000 # schemas/alias.py AliasCreate.content
# OPNsense's own aliases that are not user objects (Firewall/Alias.php: bogons & co. are `external`
# rows created by the system, __<if>_network are `internal`).
SYSTEM_ALIAS_NAMES = frozenset({"bogons", "bogonsv6", "sshlockout", "virusprot"})
LINK_TOLERATED = frozenset({"name_exists", "references_unmanaged_alias"})
@dataclass
class Entry:
row: dict
reasons: list[str] = field(default_factory=list)
details: dict[str, str] = field(default_factory=dict)
bulk: bool = False # not adoptable alone yet, but adoptable in the same run (its references come first)
matches: list = field(default_factory=list) # central objects equal to the live object (link candidates)
similar: list = field(default_factory=list) # same name/description but different content (hint only)
@property
def linkable(self) -> bool:
"""An exact central twin exists and nothing but "the name is taken" / "its references are not
managed here yet" speaks against it (both are moot when the twin is linked, its content is equal)."""
return any(not m.busy for m in self.matches) and set(self.reasons) <= LINK_TOLERATED
@property
def busy_twin(self) -> bool:
"""The one exact twin waits for a confirmation: \"adopt all\" leaves this entry alone (neither link nor duplicate)."""
return len(self.matches) == 1 and self.matches[0].busy
@property
def uuid(self) -> str:
return clean(self.row.get("uuid"))
@dataclass(frozen=True)
class Kind:
key: str
model: type
deployment: type
fk: str # deployment column pointing at the central object
uuid_attr: str # deployment column holding the OPNsense UUID (= remote_uuid)
fetch: Callable[[OpnsenseHost], Awaitable[list[dict]]]
check: Callable[[dict, dict], tuple[list[str], dict[str, str]]]
title: Callable[[dict], str]
info: Callable[[dict], dict[str, str]]
build: Callable[[dict], dict]
payload: Callable[[Any], dict]
revision: Callable
kept: Callable[[dict], list[str]] = lambda row: []
context: Callable[[AsyncSession, list[dict]], Awaitable[dict]] | None = None
refs: Callable[[dict], set[str]] | None = None
name: Callable[[dict], str] | None = None
identity: str = "description" # attribute that names the object (the alias name for aliases)
@property
def link_spec(self) -> adopt_link_service.LinkSpec:
return adopt_link_service.LinkSpec(
model=self.model, deployment=self.deployment, fk=self.fk, uuid_attr=self.uuid_attr,
identity=self.identity, payload=self.payload, revision=self.revision,
)
# ── shared helpers ───────────────────────────────────────────────────────
def _single_interface(row: dict) -> bool:
interface = clean(row.get("interface"))
return bool(interface) and "," not in interface
def _too_long(kind_model: type, values: dict) -> list[str]:
"""Columns are bounded; a live value that does not fit would fail (or be cut) on the first edit."""
columns = kind_model.__table__.columns
return [name for name, value in values.items()
if name in columns and getattr(columns[name].type, "length", None)
and isinstance(value, str) and len(value) > columns[name].type.length]
def _port(value) -> str | None:
return clean(value) or None
def _kept(row: dict, keys: tuple[str, ...]) -> list[str]:
return [key for key in keys if clean(row.get(key)) not in ("", "0")]
def _fetch_nat(search) -> Callable[[OpnsenseHost], Awaitable[list[dict]]]:
async def fetch(host: OpnsenseHost) -> list[dict]:
data = await search(host)
# split_rows drops automatic_*/lockout_* and is_automatic rows: they are OPNsense's own
return [r for r in split_rows(data)["rules"] if clean(r.get("uuid"))]
return fetch
# ── aliases ──────────────────────────────────────────────────────────────
def _alias_content(row: dict) -> str:
# same normalisation as AliasCreate.clean_content
return "\n".join(line.strip() for line in str(row.get("content") or "").splitlines() if line.strip())
def _alias_is_system(row: dict) -> bool:
name = clean(row.get("name"))
return clean(row.get("type")) == "internal" or name.startswith("__") or name in SYSTEM_ALIAS_NAMES
async def _fetch_aliases(host: OpnsenseHost) -> list[dict]:
data = await opnsense_client.search_aliases(host)
rows = data.get("rows", []) if isinstance(data, dict) else []
return [r for r in rows if isinstance(r, dict) and clean(r.get("uuid")) and clean(r.get("name"))]
async def _alias_context(db: AsyncSession, rows: list[dict]) -> dict:
return {"central_names": set((await db.scalars(select(FirewallAlias.name))).all())}
def _alias_refs(row: dict) -> set[str]:
return {line.lstrip("!").strip() for line in _alias_content(row).splitlines()} - {""}
def _alias_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
if _alias_is_system(row):
return ["system"], {}
reasons: list[str] = []
details: dict[str, str] = {}
name, type_ = clean(row.get("name")), clean(row.get("type"))
if type_ not in ALIAS_TYPES:
reasons.append("unsupported_type")
details["unsupported_type"] = clean(row.get("%type")) or type_
if not ALIAS_NAME.fullmatch(name) or len(name) > 64:
reasons.append("name_invalid")
if name in ctx["central_names"]:
reasons.append("name_exists")
if len(_alias_content(row)) > ALIAS_CONTENT_MAX:
reasons.append("content_too_large")
if type_ == "asn" and clean(row.get("proto")):
reasons.append("proto_set") # OPNexus has no field for it: a deploy elsewhere would drop the restriction
if _too_long(FirewallAlias, {"description": clean(row.get("description"))}):
reasons.append("value_too_long")
return reasons, details
def _alias_info(row: dict) -> dict[str, str]:
lines = _alias_content(row).splitlines()
return {
"type": clean(row.get("type")),
"type_label": clean(row.get("%type")) or clean(row.get("type")),
"description": clean(row.get("description")),
"content": "\n".join(lines[:6]) + ("\n…" if len(lines) > 6 else ""),
"entries": str(len(lines)),
}
def _alias_build(row: dict) -> dict:
return {
"name": clean(row.get("name")),
"type": clean(row.get("type")),
"content": _alias_content(row),
"description": clean(row.get("description")),
"enabled": on(row.get("enabled")),
}
# ── port forward ─────────────────────────────────────────────────────────
def _pf_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
reasons = []
if not _single_interface(row):
reasons.append("interface_not_single")
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
reasons.append("ipprotocol_unsupported") # blank / inet46 have no counterpart in OPNexus
if not clean(row.get("protocol")):
reasons.append("protocol_any")
if on(row.get("nordr")):
reasons.append("no_redirect")
elif not clean(row.get("target")):
reasons.append("no_target")
if clean(row.get("pass")):
reasons.append("filter_rule_linked") # OPNexus sends pass "" (manual) and would cut the link
if on(row.get("source.not")):
reasons.append("negated_source")
if on(row.get("destination.not")):
reasons.append("negated_destination")
if clean(row.get("natreflection")) not in ("", "purenat", "disable"):
reasons.append("natreflection_unsupported")
if not clean(row.get("descr")):
reasons.append("no_description")
if _too_long(PortForwardRule, _pf_build(row)):
reasons.append("value_too_long")
return reasons, {}
def _pf_build(row: dict) -> dict:
return {
"description": clean(row.get("descr")),
"interface": clean(row.get("interface")),
"ipprotocol": clean(row.get("ipprotocol")),
"protocol": clean(row.get("protocol")),
"source_net": clean(row.get("source.network")) or "any",
"source_port": _port(row.get("source.port")),
"destination_net": clean(row.get("destination.network")) or "any",
"destination_port": _port(row.get("destination.port")),
"target": clean(row.get("target")),
"local_port": _port(row.get("local-port")),
"natreflection": clean(row.get("natreflection")),
"enabled": not on(row.get("disabled")),
}
def _pf_info(row: dict) -> dict[str, str]:
return {
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
"protocol": clean(row.get("%protocol")) or clean(row.get("protocol")) or "any",
"source": clean(row.get("source.network")) or "any",
"source_port": clean(row.get("source.port")),
"destination": clean(row.get("destination.network")) or "any",
"destination_port": clean(row.get("destination.port")),
"target": clean(row.get("target")),
"target_port": clean(row.get("local-port")),
}
# ── 1:1 NAT ──────────────────────────────────────────────────────────────
def _o2o_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
reasons = []
if not _single_interface(row):
reasons.append("interface_not_single")
if clean(row.get("type")) not in ("binat", "nat"):
reasons.append("type_unsupported")
if clean(row.get("natreflection")) not in ("", "enable", "disable"):
reasons.append("natreflection_unsupported")
if not clean(row.get("source_net")) or not clean(row.get("external")):
reasons.append("incomplete")
if not clean(row.get("description")):
reasons.append("no_description")
if _too_long(OneToOneRule, _o2o_build(row)):
reasons.append("value_too_long")
return reasons, {}
def _o2o_build(row: dict) -> dict:
return {
"description": clean(row.get("description")),
"interface": clean(row.get("interface")),
"source_net": clean(row.get("source_net")),
"destination_net": clean(row.get("destination_net")) or "any",
"source_not": on(row.get("source_not")),
"destination_not": on(row.get("destination_not")),
"enabled": on(row.get("enabled")),
"type": clean(row.get("type")),
"external": clean(row.get("external")),
"natreflection": clean(row.get("natreflection")),
"log": on(row.get("log")),
}
def _o2o_info(row: dict) -> dict[str, str]:
return {
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
"type": clean(row.get("type")),
"source": ("! " if on(row.get("source_not")) else "") + clean(row.get("source_net")),
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
"external": clean(row.get("external")),
}
# ── outbound NAT ─────────────────────────────────────────────────────────
async def _fetch_outbound(host: OpnsenseHost) -> list[dict]:
# In automatic/disabled mode OPNsense does not list manual rules at all (see module docstring):
# say so instead of showing an empty list.
state = await host_mode(host)
if not state["can_deploy"]:
raise api_error(409, ErrorCode.OUTBOUND_MODE_BLOCKS_ADOPT, host=host.name, mode=state["mode"])
return await _fetch_nat(opnsense_client.search_outbound_nat_rules)(host)
def _snat_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
reasons = []
if not _single_interface(row):
reasons.append("interface_not_single")
if on(row.get("nonat")):
reasons.append("no_nat_rule")
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
reasons.append("ipprotocol_unsupported")
if not clean(row.get("protocol")):
reasons.append("protocol_missing")
if not clean(row.get("description")):
reasons.append("no_description")
if _too_long(OutboundNatRule, _snat_build(row)):
reasons.append("value_too_long")
return reasons, {}
def _snat_build(row: dict) -> dict:
return {
"description": clean(row.get("description")),
"interface": clean(row.get("interface")),
"source_net": clean(row.get("source_net")) or "any",
"destination_net": clean(row.get("destination_net")) or "any",
"source_not": on(row.get("source_not")),
"destination_not": on(row.get("destination_not")),
"enabled": on(row.get("enabled")),
"ipprotocol": clean(row.get("ipprotocol")),
"protocol": clean(row.get("protocol")),
"source_port": _port(row.get("source_port")),
"destination_port": _port(row.get("destination_port")),
"target": clean(row.get("target")),
"target_port": _port(row.get("target_port")),
}
def _snat_info(row: dict) -> dict[str, str]:
return {
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
"protocol": clean(row.get("protocol")) or "any",
"source": ("! " if on(row.get("source_not")) else "") + (clean(row.get("source_net")) or "any"),
"source_port": clean(row.get("source_port")),
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
"destination_port": clean(row.get("destination_port")),
"target": clean(row.get("target")),
"target_port": clean(row.get("target_port")),
}
KINDS: dict[str, Kind] = {
"alias": Kind(
key="alias", model=FirewallAlias, deployment=AliasDeployment, fk="alias_id",
uuid_attr="opnsense_alias_uuid", fetch=_fetch_aliases, check=_alias_check,
title=lambda row: clean(row.get("name")), info=_alias_info, build=_alias_build,
payload=alias_deploy_service._build_payload, revision=alias_policy_service.revision,
context=_alias_context, refs=_alias_refs, name=lambda row: clean(row.get("name")), identity="name",
),
"port_forward": Kind(
key="port_forward", model=PortForwardRule, deployment=PortForwardDeployment, fk="port_forward_id",
uuid_attr="opnsense_port_forward_uuid", fetch=_fetch_nat(opnsense_client.search_port_forward_rules),
check=_pf_check, title=lambda row: clean(row.get("descr")), info=_pf_info, build=_pf_build,
payload=port_forward_deploy_service._build_payload, revision=port_forward_policy_service.revision,
kept=lambda row: _kept(row, ("log", "poolopts", "tag", "tagged", "nosync", "categories")),
),
"one_to_one": Kind(
key="one_to_one", model=OneToOneRule, deployment=OneToOneDeployment, fk="one_to_one_id",
uuid_attr="opnsense_one_to_one_uuid", fetch=_fetch_nat(opnsense_client.search_one_to_one_rules),
check=_o2o_check, title=lambda row: clean(row.get("description")), info=_o2o_info, build=_o2o_build,
payload=one_to_one_deploy_service._build_payload, revision=one_to_one_policy_service.revision,
kept=lambda row: _kept(row, ("categories",)),
),
"outbound_nat": Kind(
key="outbound_nat", model=OutboundNatRule, deployment=OutboundNatDeployment, fk="outbound_nat_id",
uuid_attr="opnsense_outbound_nat_uuid", fetch=_fetch_outbound,
check=_snat_check, title=lambda row: clean(row.get("description")), info=_snat_info, build=_snat_build,
payload=outbound_nat_deploy_service._build_payload, revision=outbound_nat_policy_service.revision,
kept=lambda row: _kept(row, ("log", "staticnatport", "poolopts", "poolopts_sourcehashkey", "tag", "tagged",
"nosync", "endpoint-independent", "categories")),
),
}
# ── engine ───────────────────────────────────────────────────────────────
async def _managed_uuids(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> set[str]:
column = getattr(kind.deployment, kind.uuid_attr)
rows = await db.scalars(select(column).where(kind.deployment.host_id == host.id, column.is_not(None)))
return set(rows.all())
def _resolve_references(kind: Kind, entries: list[Entry], rows: list[dict], managed: set[str], ctx: dict) -> list[Entry]:
"""Aliases may refer to other aliases by name. A reference to a live alias OPNexus does not manage
(and that is not OPNsense's own) would be dangling in the central model, so the referring alias
waits until its target is managed. Returns the entries in an order in which they can be adopted
in ONE run (dependencies first); entries whose references can never be satisfied (target is a
url alias, a user-defined external alias, ...) keep the reason `references_unmanaged_alias`."""
live = {kind.name(r): r for r in rows}
available = set(ctx["central_names"])
available |= {n for n, r in live.items() if _alias_is_system(r) or clean(r.get("uuid")) in managed}
candidates = [e for e in entries if not e.reasons]
for entry in candidates:
missing = {ref for ref in kind.refs(entry.row) if ref in live and ref not in available}
if missing:
entry.reasons.append("references_unmanaged_alias")
entry.details["references_unmanaged_alias"] = ", ".join(sorted(missing))
order: list[Entry] = []
ready = set(available)
progress = True
while progress:
progress = False
for entry in candidates:
if entry in order:
continue
if all(ref not in live or ref in ready for ref in kind.refs(entry.row)):
order.append(entry)
ready.add(kind.name(entry.row))
progress = True
for entry in order:
entry.bulk = "references_unmanaged_alias" in entry.reasons
return order
async def _plan(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> tuple[list[Entry], list[Entry]]:
"""(all unmanaged entries with their reasons, the ones adoptable in one run in a safe order)."""
rows = await kind.fetch(host)
managed = await _managed_uuids(db, kind, host)
ctx = await kind.context(db, rows) if kind.context else {}
central = await adopt_link_service.load_central(db, kind.link_spec)
entries = []
for row in rows:
if clean(row.get("uuid")) in managed:
continue
reasons, details = kind.check(row, ctx)
entry = Entry(row, list(reasons), dict(details))
if reasons != ["system"]:
entry.matches, entry.similar = adopt_link_service.find(central, kind.link_spec, host, kind.build(row))
entries.append(entry)
if kind.refs:
order = _resolve_references(kind, entries, rows, managed, ctx)
else:
order = [e for e in entries if not e.reasons]
return entries, order
def _bulk_actions(entries: list[Entry], order: list[Entry], link_existing: bool) -> dict[str, tuple[str, Any]]:
"""What "adopt all" does with each entry: ("link", central object) when exactly one central twin exists
(and no other entry of this run took it), ("new", None), or nothing. One function for the dialog's
summary and for the real run, so they cannot disagree. Several twins are never guessed between."""
actions: dict[str, tuple[str, Any]] = {}
taken: set = set()
if link_existing:
for entry in entries:
if entry.linkable and len(entry.matches) == 1 and entry.matches[0].obj.id not in taken:
actions[entry.uuid] = ("link", entry.matches[0].obj)
taken.add(entry.matches[0].obj.id)
for entry in order:
if link_existing and entry.busy_twin:
continue
actions.setdefault(entry.uuid, ("new", None))
return actions
def _describe(kind: Kind, entry: Entry, action: str | None = None) -> dict:
spec = kind.link_spec
return {
"matches": [adopt_link_service.describe(spec, m) for m in entry.matches],
"similar": [adopt_link_service.describe(spec, m) for m in entry.similar],
"linkable": entry.linkable,
"bulk_action": action,
"uuid": entry.uuid,
"title": kind.title(entry.row),
"enabled": on(entry.row.get("enabled")) if "disabled" not in entry.row else not on(entry.row.get("disabled")),
"adoptable": not entry.reasons,
"bulk_adoptable": not entry.reasons or entry.bulk,
"reasons": entry.reasons,
"reason_details": entry.details,
"kept": kind.kept(entry.row),
"info": kind.info(entry.row),
}
async def list_unmanaged(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> list[dict]:
kind = KINDS[kind_key]
entries, order = await _plan(db, kind, host)
actions = _bulk_actions(entries, order, True)
return [_describe(kind, e, actions.get(e.uuid, ("skip", None))[0]) for e in entries]
def _create(db: AsyncSession, kind: Kind, host: OpnsenseHost, entry: Entry):
obj = kind.model(**kind.build(entry.row), targets=[host], groups=[])
db.add(obj)
return obj
async def _record(db: AsyncSession, kind: Kind, host: OpnsenseHost, obj, remote_uuid: str) -> None:
await db.flush()
kind.revision(db, obj, ADOPT_NOTE)
db.add(kind.deployment(**{
kind.fk: obj.id, "host_id": host.id, "status": "success", "message": ADOPT_MESSAGE,
kind.uuid_attr: remote_uuid, "last_payload": kind.payload(obj), "previous_payload": None,
}))
await db.flush()
async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid: str, link_to=None):
"""Takes over one object as a NEW central object, or - with `link_to` - links it to that existing
central object (adopt_link_service). Either way the live state is read fresh here."""
kind = KINDS[kind_key]
if link_to is not None:
existing = await db.scalar(select(kind.deployment).where(
kind.deployment.host_id == host.id, getattr(kind.deployment, kind.uuid_attr) == remote_uuid))
if existing is not None:
if getattr(existing, kind.fk) == link_to:
return await db.get(kind.model, link_to) # second click: already linked, nothing to do
raise NotAdoptable(["already_managed"])
entries, _ = await _plan(db, kind, host)
entry = next((e for e in entries if e.uuid == remote_uuid), None)
if entry is None:
# either gone from the host or already managed
if remote_uuid in await _managed_uuids(db, kind, host):
raise NotAdoptable(["already_managed"])
raise NotFoundOnHost(remote_uuid)
if link_to is not None:
if not set(entry.reasons) <= LINK_TOLERATED:
raise NotAdoptable(entry.reasons, entry.details)
obj = await adopt_link_service.link(db, kind.link_spec, host, link_to, remote_uuid, kind.build(entry.row))
return obj
if entry.reasons:
raise NotAdoptable(entry.reasons, entry.details)
obj = _create(db, kind, host, entry)
await _record(db, kind, host, obj, remote_uuid)
return obj
async def adopt_all(db: AsyncSession, kind_key: str, host: OpnsenseHost, link_existing: bool = False) -> dict:
"""Adopts everything adoptable in one transaction (the caller commits), nested aliases in
dependency order. With `link_existing`, entries with exactly one exact central twin are linked to it
instead (see _bulk_actions). System objects are not reported; everything else that stays behind is,
with reasons."""
kind = KINDS[kind_key]
entries, order = await _plan(db, kind, host)
actions = _bulk_actions(entries, order, link_existing)
adopted = []
done: set[str] = set()
for entry in entries: # linked first: their names are central already, so nested aliases can refer to them
action, target = actions.get(entry.uuid, ("skip", None))
if action == "link":
obj = await adopt_link_service.link(db, kind.link_spec, host, target.id, entry.uuid, kind.build(entry.row))
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": True})
done.add(entry.uuid)
for entry in order:
if actions.get(entry.uuid, ("skip", None))[0] != "new":
continue
obj = _create(db, kind, host, entry)
await _record(db, kind, host, obj, entry.uuid)
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": False})
done.add(entry.uuid)
skipped = [
{"uuid": e.uuid, "title": kind.title(e.row),
"reasons": e.reasons + (["twin_busy"] if link_existing and e.busy_twin else []), "reason_details": e.details}
for e in entries if e.uuid not in done and e.reasons != ["system"]
]
return {"adopted": adopted, "skipped": skipped}
+64
View File
@@ -145,6 +145,34 @@ async def get_interface_statistics(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/diagnostics/interface/getInterfaceStatistics")
# Swap and CPU count, for the resource-threshold alert (1.24). Verified against core source
# (Diagnostics/Api/SystemController::systemSwapAction -> configd "system show swapinfo") and live
# (read-only GET, test-fw-3, OPNsense 26.7):
# systemSwap -> {"swap": [{"device": "/dev/da0p3", "total": "8388608", "used": "0"}]} (KiB, strings;
# an empty list when the firewall has no swap)
# cpu_usage/getCPUType -> ["QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)"]
async def get_system_swap(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/diagnostics/system/systemSwap")
async def get_cpu_type(host: OpnsenseHost) -> Any:
return await _request(host, "GET", "/diagnostics/cpu_usage/getCPUType")
# Gateway health (read-only). Routes\Api\GatewayController::statusAction runs configd "interface gateways status"
# -> scripts/routes/gateway_status.php, which merges the configured gateways with dpinger's live numbers
# (src/etc/inc/plugins.inc.d/dpinger.inc, dpinger_status()). Live-verified on test-fw-3 (26.7):
# {"items": [{"name": "WAN_GW", "address": "203.0.113.1", "status": "none", "loss": "~", "delay": "~",
# "stddev": "~", "monitor": "~", "status_translated": "Online"}, ...], "status": "ok"}
# "status" is the machine-readable verdict: none (fine), down, force_down (switched off by the admin), delay,
# loss, delay+loss (thresholds of the gateway itself), anything else = pending. loss/delay/stddev are "12.3 %" /
# "4.5 ms" strings, or "~" while nothing is measured (e.g. monitoring disabled on that gateway). "status" is
# "failed" with no items when the firewall has no gateways. searchGateway carries the same numbers but with a
# translated status text ("Online"), so it is not used here.
async def get_gateway_status(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/routes/gateway/status")
async def toggle_carp_maintenance(host: OpnsenseHost) -> dict:
"""Toggle persistent CARP maintenance mode.
@@ -391,6 +419,42 @@ async def get_pf_rule_statistics(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/diagnostics/firewall/pfStatistics/rules")
# ── Rule analysis (read-only) ─────────────────────────────────────────────
# Checked against opnsense/core master (2026-10-21) and live on OPNsense 26.7:
# - Firewall/Api/FilterUtilController::ruleStatsAction runs configd `!filter rule stats` and
# returns {"status": "ok", "stats": {"<label>": {pf_rules, evaluations, packets, bytes, states}}};
# the label is the rule UUID (MVC), an md5 (automatic) or a 32-hex label (unmigrated legacy).
# - Firewall/Api/AliasUtilController::findReferencesAction requires POST with "ip" (only hex,
# ":", ".", "/", "_" accepted, else {"status": "Not an IP address!"}), runs configd
# `filter find_table_references` (a pf table test over all aliases) and returns
# {"status": "ok", "matches": ["<alias>", ...]}. It changes nothing; POST is only the
# controller's calling convention.
# - Firewall/Api/FilterController::getInterfaceListAction: {"floating"|"groups"|"interfaces"|"any":
# {"label", "items": [{"value", "label", "type"}]}} (lo0 omitted).
# - Firewall/Api/GroupController::searchItemAction -> searchBase("ifgroupentry"): rows with
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
# wireguard) appear with their name as uuid and empty members.
async def get_rule_stats(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/firewall/filter_util/ruleStats")
async def find_alias_references(host: OpnsenseHost, ip: str) -> list[str] | None:
"""Names of the aliases (pf tables) that contain ``ip`` right now; None when the firewall
could not answer (callers then treat table membership as unknown)."""
data = await _request(host, "POST", "/firewall/alias_util/findReferences", json={"ip": ip})
if not isinstance(data, dict) or data.get("status") != "ok" or not isinstance(data.get("matches"), list):
return None
return [str(name) for name in data["matches"]]
async def get_filter_interface_list(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/firewall/filter/get_interface_list")
async def search_interface_groups(host: OpnsenseHost) -> dict:
return await _request(host, "GET", "/firewall/group/searchItem", params={"rowCount": -1})
async def get_firmware_status(host: OpnsenseHost) -> dict:
# GET reads cached results. POST would trigger a synchronous probe.
return await _request(host, "GET", "/core/firmware/status")
@@ -5,7 +5,9 @@ poll_all_hosts. v1 deliberately has no admin-editable poll interval (see
config.py) — just a fixed constant, to keep this feature's first cut
small.
"""
import asyncio
import logging
import re
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete, select
@@ -13,6 +15,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.models.host import OpnsenseHost
from app.models.resource_sample import HostResourceSample
from app.services import gateway_health_service
from app.services import opnsense_client as client
from app.services.opnsense_client import OpnsenseAPIError
@@ -55,11 +58,59 @@ def _disk_used_pct(disk: dict) -> float | None:
return None
def _swap_used_pct(swap) -> float | None:
"""systemSwap: {"swap": [{"device", "total", "used"}]} in KiB; None when there is no swap at all."""
try:
rows = swap.get("swap") or []
total = sum(int(r["total"]) for r in rows)
used = sum(int(r["used"]) for r in rows)
except (AttributeError, KeyError, TypeError, ValueError):
return None
return round(used / total * 100, 1) if total > 0 else None
_CPU_THREADS = re.compile(r"(\d+)\s+threads?", re.IGNORECASE)
_CPU_CORES = re.compile(r"(\d+)\s+cores?", re.IGNORECASE)
def _cpu_threads(cpu_type) -> int | None:
"""getCPUType: ["<model> (6 cores, 6 threads)"]."""
text = " ".join(str(x) for x in cpu_type) if isinstance(cpu_type, list) else str(cpu_type or "")
match = _CPU_THREADS.search(text) or _CPU_CORES.search(text)
return int(match.group(1)) if match else None
def _services_compact(listing) -> dict[str, int]:
"""{"unbound": 1, "openvpn/<uuid>": 0}: 1 = running. Only services an admin can stop (not 'locked')."""
result: dict[str, int] = {}
for row in (listing.get("rows") if isinstance(listing, dict) else None) or []:
if not isinstance(row, dict) or not row.get("name") or str(row.get("locked")) in ("1", "True", "true"):
continue
name, id_ = str(row["name"]), str(row.get("id") or row["name"])
result[name if id_ == name else f"{name}/{id_}"] = 1 if str(row.get("running")) in ("1", "True", "true") else 0
return result
async def _optional(operation):
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing)."""
try:
return await operation
except Exception as exc:
logger.debug("Optionale Messung fehlgeschlagen: %s", type(exc).__name__)
return None
async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
resources = await client.get_system_resources(host)
system_time = await client.get_system_time(host)
disk = await client.get_system_disk(host)
interfaces = await client.get_interface_statistics(host)
swap, cpu_type, gateways, services = await asyncio.gather(
_optional(client.get_system_swap(host)),
_optional(client.get_cpu_type(host)),
_optional(gateway_health_service.get_gateway_health(host)),
_optional(client.search_services(host)),
)
db.add(
HostResourceSample(
host_id=host.id,
@@ -67,7 +118,14 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
mem_used_pct=_mem_used_pct(resources),
load_avg_1m=_load_avg_1m(system_time),
disk_used_pct=_disk_used_pct(disk),
raw={"memory": resources.get("memory"), "disk": disk.get("devices"), "interfaces": interfaces.get("statistics")},
raw={
"memory": resources.get("memory"), "disk": disk.get("devices"), "interfaces": interfaces.get("statistics"),
# None = not readable this round (the alert checks treat that as "unknown", not as "fine").
"swap_used_pct": _swap_used_pct(swap) if swap is not None else None,
"cpu_threads": _cpu_threads(cpu_type) if cpu_type is not None else None,
"gateways": gateway_health_service.compact(gateways) if gateways is not None else None,
"services": _services_compact(services) if services is not None else None,
},
)
)
+805
View File
@@ -0,0 +1,805 @@
"""The rollout run engine (1.25): moves runs forward, one small step per poll tick.
Ground rules (an unattended engine that triggers firmware updates is the riskiest thing in OPNexus):
* It never calls a firmware action itself. Each host step is an ordinary ScheduledMaintenance
(run_at=now, origin="rollout", auto_verify=True) — locks, backup before the update, CARP handover,
audit and restart safety all come from that existing path.
* Never twice: a host gets a schedule only while its ``scheduled_maintenance_id`` is empty, written
in the same transaction as the schedule itself, under the run's row lock.
* Never in parallel: one host at a time per run, and only one active run at all; nodes of a cluster
go BACKUP first, the MASTER only with the wave's CARP handover, and before every node of an
already touched cluster the cluster is checked again.
* When in doubt, stop: anything unexpected pauses the run (``pause_reason``) and notifies; nothing
is retried on its own. A paused run only continues when a human resumes it, and resuming first
checks the health of everything done so far.
* Every step re-checks that the person responsible for the run may still roll out.
* Hard limits: waves/hosts (rollout_service), soak time, preflight/host timeouts, run age.
Concurrency: a tick (1) does DB-only bookkeeping under the run's row lock, (2) decides with the
lock released (remote reads may take seconds), and (3) applies the decision only if the run's
``revision`` is still the one it decided on — a pause/abort in between, or a second backend, wins.
"""
import logging
import uuid
from datetime import datetime, timedelta, timezone
from typing import Awaitable, Callable
from fastapi import HTTPException
from sqlalchemy import func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import SessionLocal
from app.models.host import OpnsenseHost
from app.models.maintenance_window import MaintenanceWindow
from app.models.rollout import RolloutPlan, RolloutRun, RolloutRunHost, RolloutRunWave
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.models.user import AdminUser
from app.services import audit_service, rollout_health, rollout_service, window_service
from app.services import opnsense_client as client
from app.services.host_maintenance_service import ensure_no_maintenance, peers
from app.services.host_status_service import get_statuses
from app.services.policy_service import policy_lock
logger = logging.getLogger(__name__)
ACTIVE = ("waiting_window", "preflight", "wave_running", "soaking", "gate", "awaiting_approval", "resuming", "paused")
TERMINAL = ("succeeded", "cancelled")
# Statuses in which the poll loop should tick quickly.
BUSY = ("preflight", "wave_running", "gate", "resuming")
HOST_OPEN = ("running", "verifying")
PREFLIGHT_CHECK_DELAY = timedelta(seconds=20) # after asking for an update check, before reading the result
PREFLIGHT_TIMEOUT = timedelta(minutes=10)
GATE_RECHECK = timedelta(minutes=2) # a failed gate is looked at once more (slow start after a reboot)
BLOCKED_PATIENCE = timedelta(minutes=15) # someone else's maintenance in the cluster
HOST_START_TIMEOUT = timedelta(minutes=10) # the schedule never fired
HOST_MAX_DURATION = timedelta(hours=4) # backstop above the schedule's own timeouts
MAX_RUN_AGE = timedelta(days=14)
def _now() -> datetime:
return datetime.now(timezone.utc)
def _iso(value: datetime | None) -> str | None:
return value.isoformat() if value else None
def _ts(value: str | None) -> datetime | None:
return datetime.fromisoformat(value) if value else None
def _note(code: str, **params) -> dict:
return {"code": code, "params": params}
# ── small helpers on a loaded run ─────────────────────────────────────────────
def wave_row(run: RolloutRun, index: int) -> RolloutRunWave:
return next(w for w in run.waves if w.index == index)
def wave_spec(run: RolloutRun, index: int) -> dict:
return run.plan_snapshot["waves"][index]
def hosts_of(run: RolloutRun, index: int) -> list[RolloutRunHost]:
return sorted((h for h in run.hosts if h.wave_index == index), key=lambda h: (h.order, h.host_name))
def _named(run: RolloutRun, failed: list[str]) -> list[str]:
""""<host-id>:<check>" -> "<host name>:<check>" for pause details people read."""
names = {str(h.host_id): h.host_name for h in run.hosts if h.host_id}
return [f"{names.get(item.split(':', 1)[0], item.split(':', 1)[0])}:{item.split(':', 1)[1]}" for item in failed]
def _progress(run: RolloutRun, **changes) -> None:
run.progress = {**(run.progress or {}), **changes}
async def _audit(run: RolloutRun, action: str, status: str, message: str | None = None, **details) -> None:
token = audit_service.context.set({"actor": f"{run.started_by} (Rollout)"[:150], "id": str(run.id)})
try:
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id),
entity_name=run.plan_name, message=message,
details={"wave": run.current_wave, **details})
finally:
audit_service.context.reset(token)
def _requeue(host: RolloutRunHost, code: str) -> None:
"""Back to the queue: only for a host whose schedule provably never fired (cancelled under its row lock,
or refused by fire_due as overdue). Clearing the schedule id lets a resumed run plan it once more."""
host.status, host.scheduled_maintenance_id, host.started_at, host.detail = "queued", None, None, _note(code)
async def requeue_if_pending(db: AsyncSession, host: RolloutRunHost) -> bool:
"""Cancel a host's schedule that has not fired yet and put the host back in the queue. Locked like the
router's abort: fire_due holds the row while firing, and a running action is never touched."""
if host.status not in HOST_OPEN or host.scheduled_maintenance_id is None:
return False
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == host.scheduled_maintenance_id)
.with_for_update().execution_options(populate_existing=True))
if sched is None or sched.status != "pending":
return False
sched.status = "cancelled"
sched.result = {"error": "Rollout angehalten, nicht ausgeführt", "not_fired": True}
_requeue(host, "not_fired")
return True
def pause(run: RolloutRun, reason: str, **detail) -> None:
if run.status != "paused":
run.paused_from = run.status
run.status = "paused"
run.pause_reason = reason
run.pause_detail = detail
_progress(run, paused_at=_iso(_now()))
async def may_roll_out(db: AsyncSession, username: str) -> bool:
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout."""
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
if user is None:
return False
if user.role == "admin":
return True
if user.role == "operator":
from app.routers import governance
return (await governance.get_settings(db)).operators_may_deploy
return False
async def _window(db: AsyncSession, run: RolloutRun) -> tuple[bool, MaintenanceWindow | None]:
"""(window configured, window). Configured but gone means: stop, never 'no time limit'."""
window_id = run.plan_snapshot.get("window_id")
if not window_id:
return False, None
return True, await db.get(MaintenanceWindow, window_id)
async def _may_start_now(db: AsyncSession, run: RolloutRun) -> str | None:
"""None if new host work may start now, else the reason ('window_closed' / 'window_missing')."""
configured, window = await _window(db, run)
if not configured:
return None
if window is None:
return "window_missing"
return None if window_service.startable(window, _now()) else "window_closed"
async def _lock(db: AsyncSession, run_id, revision: int | None = None) -> RolloutRun | None:
stmt = select(RolloutRun).where(RolloutRun.id == run_id)
if revision is not None:
stmt = stmt.where(RolloutRun.revision == revision)
return await db.scalar(stmt.with_for_update().execution_options(populate_existing=True))
# ── creating a run ────────────────────────────────────────────────────────────
async def active_run(db: AsyncSession) -> RolloutRun | None:
return await db.scalar(select(RolloutRun).where(RolloutRun.status.in_(ACTIVE)).order_by(RolloutRun.created_at).limit(1))
async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
occurrence: datetime | None = None) -> RolloutRun:
"""Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
now = _now()
snapshot = {
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
"window_id": str(plan.window_id) if plan.window_id else None,
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
}
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
current_wave=0, started_by=started_by, trigger=trigger, window_occurrence=occurrence,
progress={}, created_at=now, step_at=now, revision=0)
run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
for w in resolved["waves"]]
run.hosts = [RolloutRunHost(wave_index=w["index"], host_id=uuid.UUID(h["id"]), host_name=h["name"], cluster_name=h["cluster_name"],
order=i, status="queued")
for w in resolved["waves"] for i, h in enumerate(w["hosts"])]
db.add(run)
await db.flush()
return run
# ── the tick ──────────────────────────────────────────────────────────────────
async def advance_runs(db: AsyncSession) -> int:
"""One step for every run. Returns how many runs want a quick next tick."""
try:
await auto_start(db)
except Exception:
logger.exception("Rollout: automatischer Start fehlgeschlagen")
await db.rollback()
ids = list((await db.scalars(select(RolloutRun.id).where(
(RolloutRun.status.in_(ACTIVE)) | RolloutRun.hosts.any(RolloutRunHost.status.in_(HOST_OPEN))))).all())
await db.commit()
busy = 0
for run_id in ids:
try:
await tick(run_id)
except Exception as exc:
logger.exception("Rollout %s: Schritt fehlgeschlagen", run_id)
async with SessionLocal() as s:
run = await _lock(s, run_id)
if run is not None and run.status in ACTIVE and run.status != "paused":
pause(run, "internal_error", error=type(exc).__name__)
run.revision += 1
await s.commit()
await _audit(run, "pause", "paused", "Interner Fehler: Rollout angehalten", reason="internal_error")
async with SessionLocal() as s:
status = await s.scalar(select(RolloutRun.status).where(RolloutRun.id == run_id))
busy += status in BUSY
return busy
async def tick(run_id) -> None:
async with SessionLocal() as db:
# (1) bookkeeping under the lock: what did the schedules do?
run = await _lock(db, run_id)
if run is None:
return
changed = await _sync_hosts(db, run)
if (run.status in ACTIVE and run.status != "paused" and _now() - run.created_at > MAX_RUN_AGE
and not any(h.status in HOST_OPEN for h in run.hosts)):
pause(run, "run_too_old", days=MAX_RUN_AGE.days)
changed = True
if changed:
run.revision += 1
await db.commit()
if changed and run.status == "paused" and run.pause_reason in ("host_failed", "host_timeout", "host_not_started", "run_too_old"):
await _audit(run, "pause", "paused", reason=run.pause_reason, detail=run.pause_detail)
if run.status not in ACTIVE or run.status in ("paused", "awaiting_approval"):
return
revision = run.revision
# (2) decide, without holding the lock
decision = await _HANDLERS[run.status](db, run)
await db.commit()
if decision is None:
return
# (3) apply only on the revision it was decided on
run = await _lock(db, run_id, revision)
if run is None:
await db.commit()
return
events = await decision(db, run)
run.revision += 1
run.step_at = _now()
await db.commit()
for event in events or []:
await _audit(run, *event[:2], **(event[2] if len(event) > 2 else {}))
Decision = Callable[[AsyncSession, RolloutRun], Awaitable[list | None]]
async def _sync_hosts(db: AsyncSession, run: RolloutRun) -> bool:
"""Carry the outcome of each host's schedule over to the run. Returns True if anything changed."""
changed = False
now = _now()
for host in run.hosts:
if host.status not in HOST_OPEN:
continue
sched = await db.get(ScheduledMaintenance, host.scheduled_maintenance_id, populate_existing=True) \
if host.scheduled_maintenance_id else None
problem = None
if sched is None:
host.status, host.detail, host.finished_at = "failed", _note("schedule_missing"), now
problem = ("host_failed", {"host": host.host_name, "error": "schedule_missing"})
elif sched.status == "pending":
if host.started_at and now - host.started_at > HOST_START_TIMEOUT:
# Never leave it pending behind a paused run: it would fire whenever the poll loop comes back.
if await requeue_if_pending(db, host):
problem = ("host_not_started", {"host": host.host_name})
changed = True
elif (sched.result or {}).get("not_fired"):
_requeue(host, "not_fired") # fire_due refused it as overdue: nothing happened on the firewall
problem = ("host_not_started", {"host": host.host_name})
changed = True
elif sched.status == "running":
want = "verifying" if sched.phase == "verify" else "running"
if host.status != want:
host.status = want
changed = True
if host.started_at and now - host.started_at > HOST_MAX_DURATION:
problem = ("host_timeout", {"host": host.host_name})
elif sched.status == "succeeded":
result = sched.result or {}
host.status, host.finished_at = "succeeded", now
host.version_after = result.get("version_after")
host.detail = _note("verified", version=result.get("version_after"), rebooted=bool(result.get("rebooted")))
changed = True
else: # failed / cancelled
error = (sched.result or {}).get("error") or sched.status
host.status, host.finished_at = "failed", now
host.detail = _note("host_failed", error=error[:300])
problem = ("host_failed", {"host": host.host_name, "error": error[:300]})
if problem and run.status in ACTIVE and run.status != "paused":
pause(run, problem[0], **problem[1])
changed = True
elif problem and host.status == "failed":
changed = True
return changed
# ── handlers: one per run status; each returns an apply-function or None ──────
async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
if await may_roll_out(db, run.started_by):
return None
async def apply(db, run):
pause(run, "actor_invalid", user=run.started_by)
return [("pause", "paused", {"reason": "actor_invalid"})]
return apply
def _to_waiting(reason: str) -> Decision:
async def apply(db, run):
if reason == "window_missing":
pause(run, "window_missing")
return [("pause", "paused", {"reason": reason})]
run.paused_from = run.status
run.status = "waiting_window"
return [("waiting_window", "waiting")]
return apply
async def _handle_waiting_window(db: AsyncSession, run: RolloutRun) -> Decision | None:
reason = await _may_start_now(db, run)
if reason == "window_missing":
return _to_waiting(reason)
if reason:
return None
async def apply(db, run):
_progress(run, resume_recheck_at=None, resume_attempts=0)
run.status = "resuming" # the gate of everything done so far is checked again first
return [("window_open", "resuming")]
return apply
async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None:
recheck = _ts((run.progress or {}).get("resume_recheck_at"))
if recheck and _now() < recheck:
return None
done = [h for h in run.hosts if h.status == "succeeded" and h.host_id]
gate = {"hosts": {}, "verdict": "ok"}
if done:
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
baselines = {}
for w in run.waves:
baselines.update((w.baseline or {}))
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
checks=run.plan_snapshot.get("gate_checks"),
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
async def apply(db, run):
if gate["verdict"] == "fail":
attempts = (run.progress or {}).get("resume_attempts", 0) + 1
_progress(run, resume_gate=gate, resume_attempts=attempts)
if attempts < 2:
_progress(run, resume_recheck_at=_iso(_now() + GATE_RECHECK))
return [("resume_gate", "recheck", {"failed": rollout_health.failed_checks(gate)})]
run.paused_from = run.paused_from or "preflight"
_progress(run, resume_recheck_at=None)
run.status = "paused"
run.pause_reason, run.pause_detail = "gate_failed", {"failed": _named(run, rollout_health.failed_checks(gate)), "on_resume": True}
return [("resume_gate", "failed", {"failed": rollout_health.failed_checks(gate)})]
_progress(run, resume_gate=gate, resume_recheck_at=None, resume_attempts=0)
target = run.paused_from if run.paused_from in ("preflight", "wave_running", "soaking", "gate") else "preflight"
if target == "gate":
wave_row(run, run.current_wave).gate_attempts = 0
_progress(run, gate_recheck_at=None)
if target == "preflight":
_progress(run, preflight_started=None, preflight_requested_at=None)
run.status, run.paused_from, run.pause_reason, run.pause_detail = target, None, None, None
return [("resume_gate", "passed", {"continue_with": target})]
return apply
def target_of(status: dict, action: str) -> str | None:
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
core package's new version comes first; product_latest is only a fallback."""
if action == "major_update":
return status.get("upgrade_major_version") or None
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
if core and core.get("new") and core.get("new") != "N/A":
return str(core["new"])
for row in status.get("upgrade_packages") or []:
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
return str(row["new_version"])
product = status.get("product") if isinstance(status.get("product"), dict) else {}
return product.get("product_latest") or status.get("product_latest") or None
async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | None:
if problem := await _actor_problem(db, run):
return problem
reason = await _may_start_now(db, run)
if reason:
return _to_waiting(reason)
index = run.current_wave
prog = run.progress or {}
started = _ts(prog.get("preflight_started")) or _now()
requested = _ts(prog.get("preflight_requested_at"))
candidates = [h for h in hosts_of(run, index) if h.status in ("queued", "blocked")]
now = _now()
# Which hosts are usable at all?
usable: dict = {}
skips: dict = {}
for rh in candidates:
host = await db.get(OpnsenseHost, rh.host_id) if rh.host_id else None
if host is None:
skips[rh.id] = _note("host_deleted")
elif not host.enabled:
skips[rh.id] = _note("host_disabled")
else:
usable[rh.id] = host
if requested is None:
# Step 1: ask every usable host to look for updates (a firmware *check*, changes nothing installed).
asked = []
for rh_id, host in usable.items():
try:
await ensure_no_maintenance(db, [host])
if (await client.get_firmware_running(host)).get("status") == "ready":
await client.check_firmware_updates(host)
asked.append(str(rh_id))
except (client.OpnsenseAPIError, HTTPException):
pass # judged in step 2 (and blocked after the timeout)
async def apply(db, run):
_progress(run, preflight_started=_iso(started), preflight_requested_at=_iso(_now()), preflight_asked=asked)
w = wave_row(run, index)
w.status = "preflight"
w.started_at = w.started_at or _now()
return [("preflight", "started", {"hosts": len(usable)})]
return apply
if now - requested < PREFLIGHT_CHECK_DELAY:
return None
# Step 2: read what each host offers.
offers: dict = {}
waiting: dict = {}
for rh_id, host in usable.items():
try:
await ensure_no_maintenance(db, [host])
except HTTPException:
waiting[rh_id] = _note("locked")
continue
try:
running = (await client.get_firmware_running(host)).get("status")
if running != "ready":
waiting[rh_id] = _note("firmware_busy")
continue
status = await client.get_firmware_status(host)
except client.OpnsenseAPIError:
waiting[rh_id] = _note("offline")
continue
fw = status.get("status")
wanted = "upgrade" if run.action == "major_update" else "update"
if fw == wanted:
offers[rh_id] = target_of(status, run.action)
elif fw in ("none", "update", "upgrade"):
skips[rh_id] = _note("no_update_offered", status=fw)
else:
waiting[rh_id] = _note("firmware_status", status=str(fw)[:40])
if waiting:
if now - started < PREFLIGHT_TIMEOUT:
return None
async def apply_blocked(db, run):
names = []
for rh in hosts_of(run, index):
if rh.id in waiting:
rh.status, rh.detail = "blocked", waiting[rh.id]
names.append(rh.host_name)
pause(run, "preflight_blocked", hosts=", ".join(names))
_progress(run, preflight_started=None, preflight_requested_at=None)
return [("pause", "paused", {"reason": "preflight_blocked", "hosts": names})]
return apply_blocked
targets = {t for t in offers.values()}
expected = run.target_version
mismatch = None
if None in targets:
mismatch = {"offered": "?"}
elif expected is None and len(targets) > 1:
mismatch = {"offered": ", ".join(sorted(targets))}
elif expected is not None and targets - {expected}:
mismatch = {"expected": expected, "offered": ", ".join(sorted(targets - {expected}))}
if mismatch:
async def apply_mismatch(db, run):
# OPNsense always installs the newest offered version; a later ring must not get one the earlier ring never ran.
pause(run, "target_version_changed", **mismatch)
_progress(run, preflight_started=None, preflight_requested_at=None)
return [("pause", "paused", {"reason": "target_version_changed", **mismatch})]
return apply_mismatch
# Baselines and order (BACKUP before MASTER inside a cluster).
baselines = {}
versions = {}
for rh_id in offers:
host = usable[rh_id]
baselines[str(host.id)] = await rollout_health.baseline(db, host)
versions[rh_id] = baselines[str(host.id)].get("version")
roles = await get_statuses([usable[i] for i in offers])
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
def order_key(rh_id):
host = usable[rh_id]
role = roles.get(host.id, {}).get("carp_status") if host.cluster_name else None
return (1 if host.cluster_name else 0, host.cluster_name or "", rank.get(role, 1), host.name)
ordered = sorted(offers, key=order_key)
target = next(iter(targets)) if targets else expected
async def apply_ready(db, run):
for rh in hosts_of(run, index):
if rh.id in skips:
rh.status, rh.detail = "skipped", skips[rh.id]
elif rh.id in offers:
rh.status, rh.detail = "queued", None
rh.order = ordered.index(rh.id)
rh.version_before = versions.get(rh.id)
w = wave_row(run, index)
w.baseline = baselines
if run.target_version is None and target:
run.target_version = target
_progress(run, preflight_started=None, preflight_requested_at=None, blocked_since=None, cluster_recheck_at=None)
if not offers:
w.status, w.finished_at = "skipped", _now()
events = [("wave", "skipped", {"wave": index})]
events += _next_wave(run)
return events
w.status = "running"
run.status = "wave_running"
return [("wave", "started", {"wave": index, "target_version": run.target_version,
"hosts": [usable[i].name for i in ordered], "skipped": len(skips)})]
return apply_ready
def _next_wave(run: RolloutRun) -> list:
index = run.current_wave + 1
if index >= len(run.waves):
run.status, run.finished_at = "succeeded", _now()
return [("finish", "succeeded")]
run.current_wave = index
w = wave_row(run, index)
if wave_spec(run, index).get("require_approval"):
w.status = run.status = "awaiting_approval"
return [("wave", "awaiting_approval", {"wave": index})]
w.status, run.status = "preflight", "preflight"
return []
async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision | None:
if any(h.status in HOST_OPEN for h in run.hosts):
return None # strictly one host at a time
index = run.current_wave
queued = [h for h in hosts_of(run, index) if h.status == "queued"]
if not queued:
soak = int(wave_spec(run, index).get("soak_minutes") or 0)
async def apply_soak(db, run):
w = wave_row(run, index)
w.status, w.soak_until = "soaking", _now() + timedelta(minutes=soak)
run.status = "soaking"
return [("wave", "soaking", {"wave": index, "minutes": soak})]
return apply_soak
if problem := await _actor_problem(db, run):
return problem
reason = await _may_start_now(db, run)
if reason:
return _to_waiting(reason)
nxt = queued[0]
host = await db.get(OpnsenseHost, nxt.host_id) if nxt.host_id else None
if host is None or not host.enabled:
async def apply_skip(db, run):
rh = next(h for h in run.hosts if h.id == nxt.id)
rh.status, rh.detail = "skipped", _note("host_deleted" if host is None else "host_disabled")
return [("host", "skipped", {"host": nxt.host_name})]
return apply_skip
spec = wave_spec(run, index)
prog = run.progress or {}
if host.cluster_name:
fleet = await peers(db, host)
statuses = await get_statuses(fleet)
done_ids = {h.host_id for h in run.hosts if h.status == "succeeded"}
done_peers = [p for p in fleet if p.id in done_ids and p.id != host.id]
baselines = {}
for w in run.waves:
baselines.update(w.baseline or {})
# The cluster as a whole must be healthy before a node is touched; nodes done earlier in this run
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
gate = await rollout_health.evaluate(
db, done_peers + [host], baselines, target_version=run.target_version,
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
if r["result"] == "fail" and not (hid == str(host.id) and c in ("version", "gateways", "services"))]
if failed:
recheck = _ts(prog.get("cluster_recheck_at"))
if recheck is None:
async def apply_recheck(db, run):
_progress(run, cluster_recheck_at=_iso(_now() + GATE_RECHECK), cluster_gate=gate)
return [("cluster_gate", "recheck", {"host": host.name, "failed": failed})]
return apply_recheck
if _now() < recheck:
return None
async def apply_cluster_fail(db, run):
pause(run, "cluster_gate_failed", cluster=host.cluster_name, host=host.name, failed=_named(run, failed))
_progress(run, cluster_recheck_at=None, cluster_gate=gate)
return [("cluster_gate", "failed", {"host": host.name, "failed": failed})]
return apply_cluster_fail
if statuses.get(host.id, {}).get("carp_status") == "MASTER" and not spec.get("carp_handover"):
async def apply_master(db, run):
pause(run, "master_needs_handover", host=host.name, cluster=host.cluster_name)
return [("pause", "paused", {"reason": "master_needs_handover", "host": host.name})]
return apply_master
try:
await ensure_no_maintenance(db, [host])
except HTTPException:
since = _ts(prog.get("blocked_since"))
if since is None:
async def apply_blocked_start(db, run):
_progress(run, blocked_since=_iso(_now()))
return [("host", "waiting_for_lock", {"host": host.name})]
return apply_blocked_start
if _now() - since < BLOCKED_PATIENCE:
return None
async def apply_blocked(db, run):
pause(run, "blocked_by_maintenance", host=host.name)
_progress(run, blocked_since=None)
return [("pause", "paused", {"reason": "blocked_by_maintenance", "host": host.name})]
return apply_blocked
async def apply_start(db, run):
await policy_lock(db)
rh = next(h for h in run.hosts if h.id == nxt.id)
# Idempotency guard: a host that already has a schedule never gets a second one.
if rh.status != "queued" or rh.scheduled_maintenance_id is not None:
return []
now = _now()
sched = ScheduledMaintenance(
host_id=host.id, host_name=host.name, action=run.action, run_at=now, status="pending",
actor=run.started_by, created_at=now, carp_handover=bool(spec.get("carp_handover") and host.cluster_name),
auto_verify=True, origin="rollout",
progress={"rollout": {"run_id": str(run.id), "wave": index, "plan": run.plan_name}},
)
db.add(sched)
await db.flush()
rh.scheduled_maintenance_id, rh.status, rh.started_at = sched.id, "running", now
_progress(run, blocked_since=None, cluster_recheck_at=None)
return [("host", "started", {"host": host.name, "schedule_id": str(sched.id), "handover": sched.carp_handover})]
return apply_start
async def _handle_soaking(db: AsyncSession, run: RolloutRun) -> Decision | None:
w = wave_row(run, run.current_wave)
if w.soak_until and _now() < w.soak_until:
return None
async def apply(db, run):
w = wave_row(run, run.current_wave)
w.status, w.gate_attempts = "gate", 0
run.status = "gate"
_progress(run, gate_recheck_at=None)
return []
return apply
async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
recheck = _ts((run.progress or {}).get("gate_recheck_at"))
if recheck and _now() < recheck:
return None
index = run.current_wave
w = wave_row(run, index)
done = [h for h in hosts_of(run, index) if h.status == "succeeded" and h.host_id]
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
checks=run.plan_snapshot.get("gate_checks"),
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
async def apply(db, run):
w = wave_row(run, index)
w.gate_result = gate
w.gate_attempts = (w.gate_attempts or 0) + 1
failed = rollout_health.failed_checks(gate)
events = [("gate", gate["verdict"], {"wave": index, "attempt": w.gate_attempts, "failed": failed, "result": gate})]
if gate["verdict"] == "fail":
if w.gate_attempts < 2:
_progress(run, gate_recheck_at=_iso(_now() + GATE_RECHECK))
return events
_progress(run, gate_recheck_at=None)
pause(run, "gate_failed", wave=w.label, failed=_named(run, failed))
return events + [("pause", "paused", {"reason": "gate_failed"})]
w.status, w.finished_at = "passed", _now()
_progress(run, gate_recheck_at=None)
return events + _next_wave(run)
return apply
_HANDLERS = {
"waiting_window": _handle_waiting_window,
"resuming": _handle_resuming,
"preflight": _handle_preflight,
"wave_running": _handle_wave_running,
"soaking": _handle_soaking,
"gate": _handle_gate,
}
# ── automatic start at a window occurrence ────────────────────────────────────
async def auto_start(db: AsyncSession) -> int:
started = 0
plans = (await db.scalars(select(RolloutPlan).where(
RolloutPlan.auto_start.is_(True), RolloutPlan.activated_at.is_not(None), RolloutPlan.window_id.is_not(None)))).all()
for plan in plans:
window = await db.get(MaintenanceWindow, plan.window_id)
occ = window_service.startable(window, _now()) if window else None
if occ is None:
continue
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
continue
if (plan.auto_start_error or {}).get("occurrence") == _iso(occ.start):
continue # already reported for this occurrence
await policy_lock(db)
plan = await rollout_service.load_plan(db, plan.id)
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
continue # another backend started it while this one waited for the lock (not a refusal)
resolved = await rollout_service.resolve(db, plan)
problem = None
if plan.action != "update":
problem = _note("auto_start_update_only")
elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
problem = _note("plan_changed") # someone changed the plan, its groups or its window since activation
elif not await may_roll_out(db, plan.activated_by or ""):
problem = _note("actor_invalid", user=plan.activated_by)
elif resolved["errors"]:
problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
elif (other := await active_run(db)) is not None:
problem = _note("other_run_active", plan=other.plan_name)
if problem:
plan.auto_start_error = {"occurrence": _iso(occ.start), "at": _iso(_now()), **problem}
await db.commit()
token = audit_service.context.set({"actor": "system:rollout", "id": str(plan.id)})
try:
await audit_service.persist(kind="rollout", action="auto_start", status="refused", entity_id=str(plan.id),
entity_name=plan.name, details=plan.auto_start_error)
finally:
audit_service.context.reset(token)
continue
try:
run = await create_run(db, plan, resolved, started_by=plan.activated_by, trigger="window", occurrence=occ.start)
plan.auto_start_error = None
await db.commit()
except IntegrityError:
await db.rollback() # another backend started it for this occurrence
continue
started += 1
await _audit(run, "start", "started", trigger="window", occurrence=_iso(occ.start),
hosts=sum(len(w["hosts"]) for w in resolved["waves"]))
await db.commit()
return started
# ── human actions (called by the router under policy_lock) ────────────────────
async def count_active(db: AsyncSession) -> int:
return await db.scalar(select(func.count()).select_from(RolloutRun).where(RolloutRun.status.in_(ACTIVE))) or 0
+218
View File
@@ -0,0 +1,218 @@
"""Health gate of a firmware rollout (1.25): is a host, after its update, as healthy as before?
``baseline(db, host)`` is taken right before a wave starts, ``evaluate(...)`` after the soak time
(and before each node of an already touched cluster, and on resume). Every check of every host
yields ``{"result": ok|warn|fail|skip, "code": <machine code>, "params": {...}}``; the UI translates
the code (rollouts.gate.code.*), so no German prose travels through here.
What the checks look at (all read-only on the firewall, except that the compliance check downloads
the configuration like the 6-hourly security check does):
* online — enabled, the API answers, firmware job idle (``running == ready``).
* version — the installed version is the rollout's target version.
* carp — per cluster: every node online, exactly one MASTER per VHID, no node in CARP
maintenance mode, no CARP auto-revert timer running.
* gateways — gateways that were up in the baseline are up again; a gateway that was already bad
before only warns (1.24 ``get_gateway_health``).
* services — services that ran in the baseline run again.
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
update may legitimately change defaults.
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
"""
import asyncio
from collections import defaultdict
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.carp_maintenance_window import CarpMaintenanceWindow
from app.models.compliance_result import ComplianceResult
from app.models.host import OpnsenseHost
from app.services import compliance_service, gateway_health_service, opnsense_client as client
from app.services.host_maintenance_service import _shared_change_blockers, peers
from app.services.host_status_service import get_statuses
from app.services.resource_monitor_service import _services_compact
CHECKS = ("online", "version", "carp", "gateways", "services", "locks", "compliance")
DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": True, "services": True, "locks": True,
"compliance": False}
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
def _r(result: str, code: str, **params) -> dict:
return {"result": result, "code": code, "params": params}
async def _optional(coro):
try:
return await coro
except client.OpnsenseAPIError:
return None
async def _gateways(host) -> dict | None:
gws = await _optional(gateway_health_service.get_gateway_health(host))
return None if gws is None else {g.name: g.state for g in gws}
async def _services(host) -> dict | None:
listing = await _optional(client.search_services(host))
return None if listing is None else _services_compact(listing)
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
info, gateways, services = await asyncio.gather(
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
return {
"version": (info or {}).get("product_version"),
"gateways": gateways,
"services": services,
"compliance_score": score,
}
def _check_gateways(before: dict | None, now: dict | None) -> dict:
if now is None:
return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
if before is None:
bad = sorted(n for n, s in now.items() if s == "offline")
return _r("warn", "gateway_down", names=", ".join(bad)) if bad else _r("ok", "no_baseline")
down = sorted(n for n, s in before.items() if s in ("online", "unknown") and now.get(n) == "offline")
gone = sorted(n for n in before if n not in now)
degraded = sorted(n for n, s in now.items() if s == "degraded" and before.get(n) != "degraded")
still_bad = sorted(n for n, s in before.items() if s in ("offline", "degraded") and now.get(n) in ("offline", "degraded"))
# "unknown" (not monitored) going to "offline" is a real change too: monitoring just started and reports down.
if down:
return _r("fail", "gateway_down", names=", ".join(down))
if gone:
return _r("warn", "gateway_missing", names=", ".join(gone))
if degraded:
return _r("warn", "gateway_degraded", names=", ".join(degraded))
if still_bad:
return _r("warn", "gateway_bad_before", names=", ".join(still_bad))
return _r("ok", "ok")
def _check_services(before: dict | None, now: dict | None) -> dict:
if now is None:
return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
if before is None:
return _r("ok", "no_baseline")
stopped = sorted(name.split("/")[0] for name, running in before.items() if running == 1 and now.get(name) != 1)
if stopped:
return _r("fail", "service_stopped", names=", ".join(dict.fromkeys(stopped)))
return _r("ok", "ok")
async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dict) -> dict:
offline = sorted(h.name for h in fleet if h.enabled and statuses.get(h.id, {}).get("status") != "online")
if offline:
return _r("fail", "node_offline", names=", ".join(offline))
online = [h for h in fleet if statuses.get(h.id, {}).get("status") == "online"]
demoted = sorted(h.name for h in online if statuses[h.id].get("carp_maintenance_mode"))
if demoted:
return _r("fail", "maintenance_mode", names=", ".join(demoted))
timers = (await db.scalars(select(CarpMaintenanceWindow.host_name).where(
CarpMaintenanceWindow.host_id.in_([h.id for h in fleet]), CarpMaintenanceWindow.status == "active"))).all()
if timers:
return _r("fail", "auto_revert_active", names=", ".join(sorted(timers)))
masters: dict[str, int] = defaultdict(int)
broken: set[str] = set()
for h in online:
for vhid, state in (statuses[h.id].get("carp_vips") or {}).items():
masters[vhid] += state == "MASTER"
if state not in ("MASTER", "BACKUP"):
broken.add(vhid)
wrong = sorted(set(v for v, n in masters.items() if n != 1) | broken, key=lambda v: (len(v), v))
if wrong:
return _r("fail", "vhid_roles", vhids=", ".join(wrong))
if not masters and not any(statuses[h.id].get("carp_status") == "MASTER" for h in online):
return _r("fail", "no_master")
return _r("ok", "ok")
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
enabled = {**DEFAULT_CHECKS, **(checks or {})}
for name in ALWAYS_ON:
enabled[name] = True
results: dict[str, dict] = {}
fleets: dict[str, list[OpnsenseHost]] = {}
for host in hosts:
fleets[str(host.id)] = await peers(db, host) if host.cluster_name else [host]
every = {h.id: h for fleet in fleets.values() for h in fleet}
statuses = await get_statuses(list(every.values()))
carp_cache: dict[str, dict] = {}
for host in hosts:
hid = str(host.id)
before = baselines.get(hid) or {}
row: dict[str, dict] = {}
st = statuses.get(host.id, {})
running = await _optional(client.get_firmware_running(host)) if st.get("status") == "online" else None
if not host.enabled:
row["online"] = _r("fail", "disabled")
elif st.get("status") != "online":
row["online"] = _r("fail", "offline")
elif (running or {}).get("status") != "ready":
row["online"] = _r("fail", "firmware_busy")
else:
row["online"] = _r("ok", "ok")
reachable = row["online"]["result"] == "ok"
if enabled["version"]:
info = await _optional(client.get_firmware_info(host)) if reachable else None
version = (info or {}).get("product_version")
if not target_version:
row["version"] = _r("skip", "no_target")
elif version is None:
row["version"] = _r("fail", "unreadable")
elif version != target_version:
row["version"] = _r("fail", "version_mismatch", expected=target_version, actual=version)
else:
row["version"] = _r("ok", "ok", actual=version)
if enabled["carp"]:
if not host.cluster_name:
row["carp"] = _r("skip", "no_cluster")
else:
if host.cluster_name not in carp_cache:
carp_cache[host.cluster_name] = await _check_carp(db, fleets[hid], statuses)
row["carp"] = carp_cache[host.cluster_name]
if enabled["gateways"]:
row["gateways"] = _check_gateways(before.get("gateways"), await _gateways(host) if reachable else None)
if enabled["services"]:
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
blockers = await _shared_change_blockers(db, fleets[hid])
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
if enabled["compliance"]:
base_score = before.get("compliance_score")
if base_score is None:
row["compliance"] = _r("skip", "no_baseline")
elif not reachable:
row["compliance"] = _r("fail", "unreadable")
else:
fresh = await compliance_service.check_host(db, host)
if fresh.score is None:
row["compliance"] = _r("warn", "unreadable")
elif fresh.score < base_score:
row["compliance"] = _r("fail" if compliance_mode == "fail" else "warn", "score_lower",
before=base_score, after=fresh.score)
else:
row["compliance"] = _r("ok", "ok", before=base_score, after=fresh.score)
results[hid] = row
worst = max((_RANK[c["result"]] for row in results.values() for c in row.values()), default=0)
return {"hosts": results, "verdict": ("ok", "warn", "fail")[worst]}
def failed_checks(gate: dict) -> list[str]:
"""Flat "host-id:check" list of everything that failed (for notifications and the audit trail)."""
return [f"{hid}:{name}" for hid, row in (gate.get("hosts") or {}).items() for name, c in row.items() if c["result"] == "fail"]
+138
View File
@@ -0,0 +1,138 @@
"""Rollout plans: resolving waves to hosts, validation, preview token / activation fingerprint.
The run engine (rollout_engine.py) builds on ``resolve``: what a preview shows is exactly what a run
freezes when it starts.
"""
import hashlib
import json
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.models.host import OpnsenseHost
from app.models.maintenance_window import MaintenanceWindow
from app.models.policy import DeviceGroup
from app.models.rollout import RolloutPlan
from app.services import rollout_health, window_service
# Hard limits: an unattended engine that fires firmware updates must stay small and explainable.
MAX_WAVES = 10
MAX_HOSTS = 100
MAX_SOAK_MINUTES = 24 * 60
def _issue(code: str, **params) -> dict:
return {"code": code, "params": params}
async def resolve(db: AsyncSession, plan: RolloutPlan) -> dict:
"""Waves with their hosts (a host in several waves belongs to the earliest), errors that block a
start/activation, and warnings."""
errors: list[dict] = []
warnings: list[dict] = []
seen: dict = {}
duplicates: set[str] = set()
waves = []
group_ids = {g.id for w in plan.waves for g in w.groups}
groups = {g.id: g for g in (await db.scalars(
select(DeviceGroup).where(DeviceGroup.id.in_(group_ids)).options(selectinload(DeviceGroup.hosts)))).all()} if group_ids else {}
for index, wave in enumerate(plan.waves):
hosts: list[OpnsenseHost] = []
for group in sorted(wave.groups, key=lambda g: g.name):
for host in sorted(groups[group.id].hosts, key=lambda h: h.name):
if host.id in seen:
if seen[host.id] != index:
duplicates.add(host.name)
continue
seen[host.id] = index
hosts.append(host)
if not hosts:
warnings.append(_issue("empty_wave", wave=wave.label))
clusters = sorted({h.cluster_name for h in hosts if h.cluster_name})
if clusters and not wave.carp_handover:
warnings.append(_issue("master_needs_handover", wave=wave.label, clusters=", ".join(clusters)))
waves.append({
"index": index, "label": wave.label, "soak_minutes": wave.soak_minutes, "carp_handover": wave.carp_handover,
"require_approval": wave.require_approval, "groups": sorted(g.name for g in wave.groups),
"hosts": [{"id": str(h.id), "name": h.name, "cluster_name": h.cluster_name, "enabled": h.enabled} for h in hosts],
})
if duplicates:
warnings.append(_issue("duplicate_hosts", names=", ".join(sorted(duplicates))))
all_hosts = [h for w in waves for h in w["hosts"]]
disabled = sorted(h["name"] for h in all_hosts if not h["enabled"])
if disabled:
warnings.append(_issue("disabled_hosts", names=", ".join(disabled)))
if not plan.waves:
errors.append(_issue("no_waves"))
elif not all_hosts:
errors.append(_issue("no_hosts"))
if len(plan.waves) > MAX_WAVES:
errors.append(_issue("too_many_waves", limit=MAX_WAVES))
if len(all_hosts) > MAX_HOSTS:
errors.append(_issue("too_many_hosts", limit=MAX_HOSTS))
# Clusters: where are their nodes?
cluster_names = sorted({h["cluster_name"] for h in all_hosts if h["cluster_name"]})
for cluster in cluster_names:
members = (await db.scalars(select(OpnsenseHost.id).where(OpnsenseHost.cluster_name == cluster))).all()
in_plan = {h["id"]: w["index"] for w in waves for h in w["hosts"] if h["cluster_name"] == cluster}
partial = len(in_plan) < len(members)
split = len(set(in_plan.values())) > 1
if plan.action == "major_update" and (partial or split):
# Never two major releases side by side in one cluster longer than one wave.
errors.append(_issue("cluster_split", cluster=cluster))
elif partial:
warnings.append(_issue("cluster_partial", cluster=cluster))
elif split:
warnings.append(_issue("mixed_versions", cluster=cluster, wave=waves[max(in_plan.values())]["label"]))
window = await db.get(MaintenanceWindow, plan.window_id) if plan.window_id else None
if plan.auto_start and plan.action != "update":
errors.append(_issue("auto_start_update_only"))
if plan.auto_start and window is None:
errors.append(_issue("auto_start_needs_window"))
if window is None:
warnings.append(_issue("no_window"))
elif not window.enabled:
warnings.append(_issue("window_disabled"))
now = datetime.now(timezone.utc)
return {
"waves": waves, "errors": errors, "warnings": warnings,
"window": {"id": str(window.id), "name": window.name, "definition": window_service.definition(window),
"next": [o.as_dict() for o in window_service.occurrences(window, now, 3)] if window.enabled else []}
if window else None,
}
def definition(plan: RolloutPlan) -> dict:
return {
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
}
def effective_checks(checks: dict | None) -> dict:
merged = {**rollout_health.DEFAULT_CHECKS, **{k: bool(v) for k, v in (checks or {}).items() if k in rollout_health.CHECKS}}
for name in rollout_health.ALWAYS_ON:
merged[name] = True
return merged
def fingerprint(plan: RolloutPlan, resolved: dict) -> str:
"""Covers the plan, every resolved host (id, name, cluster, enabled) and the window definition: a change
to any of them invalidates a preview and stops an activated automatic start."""
state = {
"plan": definition(plan),
"hosts": [[w["index"], h["id"], h["name"], h["cluster_name"], h["enabled"]] for w in resolved["waves"] for h in w["hosts"]],
"window": (resolved.get("window") or {}).get("definition"),
}
return hashlib.sha256(json.dumps(state, sort_keys=True, default=str).encode()).hexdigest()
async def load_plan(db: AsyncSession, plan_id) -> RolloutPlan | None:
return await db.scalar(
select(RolloutPlan).where(RolloutPlan.id == plan_id).execution_options(populate_existing=True))
+46 -31
View File
@@ -19,27 +19,18 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.models.host import OpnsenseHost
from app.models.rule import FirewallRule
from app.models.rule_deployment import RuleDeployment
from app.services import opnsense_client
from app.services import adopt_link_service, opnsense_client
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on # noqa: F401 (NotAdoptable is re-exported for routers/rules.py)
from app.services.policy_service import revision
from app.services.rule_deploy_service import _build_payload
class RuleNotFoundOnHost(Exception):
class RuleNotFoundOnHost(NotFoundOnHost):
pass
class NotAdoptable(Exception):
def __init__(self, reasons: list[str]):
super().__init__(", ".join(reasons))
self.reasons = reasons
def _on(value) -> bool:
return value in ("1", 1, True)
def _clean(value) -> str:
return "" if value is None else str(value).strip()
_on = on
_clean = clean
def reasons_not_adoptable(row: dict, managed_uuids: set[str]) -> list[str]:
@@ -73,6 +64,30 @@ def reasons_not_adoptable(row: dict, managed_uuids: set[str]) -> list[str]:
return reasons
def _values(row: dict) -> dict:
"""The fields OPNexus manages, normalised exactly as adopting stores them (also what a link compares)."""
return {
"description": _clean(row.get("description")),
"interface": _clean(row.get("interface")),
"direction": _clean(row.get("direction")),
"ipprotocol": _clean(row.get("ipprotocol")),
"protocol": _clean(row.get("protocol")) or "any",
"action": _clean(row.get("action")),
"source_net": _clean(row.get("source_net")) or "any",
"source_port": _clean(row.get("source_port")) or None,
"destination_net": _clean(row.get("destination_net")) or "any",
"destination_port": _clean(row.get("destination_port")) or None,
"enabled": _on(row.get("enabled")),
"log": _on(row.get("log")),
}
LINK_SPEC = adopt_link_service.LinkSpec(
model=FirewallRule, deployment=RuleDeployment, fk="rule_id", uuid_attr="opnsense_rule_uuid",
identity="description", payload=_build_payload, revision=revision,
)
async def _managed_uuids(db: AsyncSession, host: OpnsenseHost) -> set[str]:
rows = await db.scalars(
select(RuleDeployment.opnsense_rule_uuid).where(
@@ -90,6 +105,7 @@ async def _live_rows(host: OpnsenseHost) -> list[dict]:
async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
managed = await _managed_uuids(db, host)
central = await adopt_link_service.load_central(db, LINK_SPEC)
result = []
for row in await _live_rows(host):
if row.get("is_automatic"):
@@ -97,7 +113,11 @@ async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
reasons = reasons_not_adoptable(row, managed)
if "already_managed" in reasons:
continue
matches, similar = adopt_link_service.find(central, LINK_SPEC, host, _values(row))
result.append({
"matches": [adopt_link_service.describe(LINK_SPEC, m) for m in matches],
"similar": [adopt_link_service.describe(LINK_SPEC, m) for m in similar],
"linkable": any(not m.busy for m in matches) and not reasons,
"uuid": _clean(row.get("uuid")),
"description": _clean(row.get("description")),
"interface": _clean(row.get("interface")),
@@ -117,7 +137,15 @@ async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
return result
async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str) -> FirewallRule:
async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str, link_to=None) -> FirewallRule:
"""Takes the rule over as a new central rule, or - with `link_to` - links it to that existing rule."""
if link_to is not None:
existing = await db.scalar(select(RuleDeployment).where(
RuleDeployment.host_id == host.id, RuleDeployment.opnsense_rule_uuid == rule_uuid))
if existing is not None:
if existing.rule_id == link_to:
return await db.get(FirewallRule, link_to) # second click: already linked
raise NotAdoptable(["already_managed"])
managed = await _managed_uuids(db, host)
row = next((r for r in await _live_rows(host) if _clean(r.get("uuid")) == rule_uuid), None)
if row is None:
@@ -125,23 +153,10 @@ async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str) -> Firewal
reasons = reasons_not_adoptable(row, managed)
if reasons:
raise NotAdoptable(reasons)
if link_to is not None:
return await adopt_link_service.link(db, LINK_SPEC, host, link_to, rule_uuid, _values(row))
rule = FirewallRule(
description=_clean(row.get("description")),
interface=_clean(row.get("interface")),
direction=_clean(row.get("direction")),
ipprotocol=_clean(row.get("ipprotocol")),
protocol=_clean(row.get("protocol")) or "any",
action=_clean(row.get("action")),
source_net=_clean(row.get("source_net")) or "any",
source_port=_clean(row.get("source_port")) or None,
destination_net=_clean(row.get("destination_net")) or "any",
destination_port=_clean(row.get("destination_port")) or None,
enabled=_on(row.get("enabled")),
log=_on(row.get("log")),
targets=[host],
groups=[],
)
rule = FirewallRule(**_values(row), targets=[host], groups=[])
db.add(rule)
await db.flush()
revision(db, rule, "Von Firewall übernommen")
@@ -0,0 +1 @@
"""Rule analysis (1.26): which firewall rule decides a packet, with honest confidence."""
@@ -0,0 +1,159 @@
"""Everything about one firewall the analysis needs, fetched read-only and cached briefly.
Sources (see the evidence block in opnsense_client.py): ``firewall/filter/searchRule`` (all
rules incl. automatic ones, with ``sort_order``), ``diagnostics/firewall/pfStatistics/rules``
(what pf has loaded), ``interfaces/overview/interfacesInfo`` (identifier, device, addresses
incl. CARP VIPs, gateways), ``firewall/group/searchItem`` (interface groups),
``firewall/alias/searchItem`` (alias definitions), ``firewall/d_nat/searchRule`` (port
forwards, only to warn about NAT) and, per queried IP, ``firewall/alias_util/findReferences``.
Optional sources may fail; the analysis then lowers its confidence instead of guessing.
"""
from __future__ import annotations
import asyncio
import ipaddress
import logging
import time
from dataclasses import dataclass, field
from app.models.host import OpnsenseHost
from app.services import opnsense_client
from app.services.rule_analysis import pf_ruleset
from app.services.rule_analysis.model import IPAddr, IPNet
logger = logging.getLogger(__name__)
CACHE_SECONDS = 60
# Built-in interface groups whose members are the matching dynamic devices.
BUILTIN_GROUP_DEVICES = {"wireguard": ("wg",), "openvpn": ("ovpns", "ovpnc"), "enc0": ("enc0",)}
@dataclass
class IfInfo:
identifier: str
device: str
description: str
nets: list[IPNet] = field(default_factory=list)
addrs: list[IPAddr] = field(default_factory=list) # primary addresses
vips: list[IPAddr] = field(default_factory=list) # CARP / alias VIPs on this interface
has_gateway: bool = False
@dataclass
class Context:
rows: list[dict]
pf: pf_ruleset.PfRuleset | None
interfaces: dict[str, IfInfo]
groups: dict[str, set[str]] | None # name -> member identifiers; None = unknown
aliases: dict[str, dict]
port_forwards: list[dict] | None
refs: dict[str, list[str] | None] = field(default_factory=dict)
notes: list[str] = field(default_factory=list)
loaded_at: float = 0.0
@property
def self_addrs(self) -> list[IPAddr]:
addrs = [a for i in self.interfaces.values() for a in (*i.addrs, *i.vips)]
return addrs + [ipaddress.ip_address("127.0.0.1"), ipaddress.ip_address("::1")]
def by_device(self, device: str) -> IfInfo | None:
return next((i for i in self.interfaces.values() if i.device == device), None)
def by_label(self, label: str) -> IfInfo | None:
lowered = label.strip().lower()
return self.interfaces.get(lowered) or next(
(i for i in self.interfaces.values() if i.description.lower() == lowered), None)
def _interfaces(info: dict) -> dict[str, IfInfo]:
result: dict[str, IfInfo] = {}
for row in (info or {}).get("rows", []) or []:
identifier = str(row.get("identifier") or "").strip()
if not identifier:
continue
item = IfInfo(identifier=identifier, device=str(row.get("device") or ""),
description=str(row.get("description") or identifier),
has_gateway=bool(row.get("gateways")) or "default" in (row.get("routes") or []))
for entry in [*(row.get("ipv4") or []), *(row.get("ipv6") or [])]:
try:
iface = ipaddress.ip_interface(str(entry.get("ipaddr")).split("%")[0])
except ValueError:
continue
(item.vips if entry.get("vhid") or entry.get("vip") else item.addrs).append(iface.ip)
if iface.network not in item.nets:
item.nets.append(iface.network)
result[identifier] = item
return result
def _groups(data: dict | None, interfaces: dict[str, IfInfo]) -> dict[str, set[str]] | None:
if not isinstance(data, dict) or not isinstance(data.get("rows"), list):
return None
groups: dict[str, set[str]] = {}
for row in data["rows"]:
name = str(row.get("ifname") or "")
members = {m.strip() for m in str(row.get("members") or "").split(",") if m.strip()}
prefixes = BUILTIN_GROUP_DEVICES.get(name)
if prefixes:
members |= {i.identifier for i in interfaces.values() if i.device.startswith(prefixes)}
if name:
groups[name] = members
return groups
async def _optional(coro, note: str, notes: list[str]):
try:
return await coro
except Exception as exc: # noqa: BLE001 - optional source: degrade, never fail the analysis
logger.info("rule analysis: optional source failed (%s): %s", note, exc)
notes.append(note)
return None
_CACHE: dict[str, Context] = {}
async def load_context(host: OpnsenseHost, *, use_cache: bool = True) -> Context:
key = str(host.id)
cached = _CACHE.get(key)
if use_cache and cached and time.monotonic() - cached.loaded_at < CACHE_SECONDS:
return cached
notes: list[str] = []
rules, info = await asyncio.gather(
opnsense_client.search_rules(host, all_rows=True), opnsense_client.get_interfaces_info(host))
stats, groups, aliases, forwards = await asyncio.gather(
_optional(opnsense_client.get_pf_rule_statistics(host), "PF_RULESET_UNAVAILABLE", notes),
_optional(opnsense_client.search_interface_groups(host), "GROUPS_UNAVAILABLE", notes),
_optional(opnsense_client._request(host, "GET", "/firewall/alias/searchItem", params={"rowCount": -1}),
"ALIASES_UNAVAILABLE", notes),
_optional(opnsense_client.search_port_forward_rules(host), "NAT_UNAVAILABLE", notes),
)
interfaces = _interfaces(info)
pf = pf_ruleset.parse(stats)
if stats is not None and pf is None:
notes.append("PF_RULESET_UNAVAILABLE")
ctx = Context(
rows=list((rules or {}).get("rows") or []), pf=pf, interfaces=interfaces,
groups=_groups(groups, interfaces),
aliases={str(a.get("name")): a for a in ((aliases or {}).get("rows") or []) if a.get("name")},
port_forwards=None if forwards is None else list(forwards.get("rows") or []),
notes=notes, loaded_at=time.monotonic(),
)
_CACHE[key] = ctx
return ctx
async def ensure_refs(host: OpnsenseHost, ctx: Context, ips: list[str]) -> None:
"""Ask pf which tables contain each IP (cached in the context)."""
for ip in ips:
if ip in ctx.refs:
continue
try:
ctx.refs[ip] = await opnsense_client.find_alias_references(host, ip)
except Exception as exc: # noqa: BLE001 - table membership becomes "unknown"
logger.info("rule analysis: findReferences failed: %s", exc)
ctx.refs[ip] = None
def clear_cache() -> None:
_CACHE.clear()
@@ -0,0 +1,317 @@
"""Which rule decides a packet, following pf semantics.
pf walks the loaded ruleset top to bottom; the *last* matching rule wins unless a matching
rule is ``quick``, which ends the walk. OPNsense orders rules by ``sort_order`` (system,
floating, groups, interfaces). The analysis looks at the first packet of a new connection.
Confidence:
- ``rule_confidence`` is "sure" only when no rule that could still change the outcome
matched with an unknown result;
- ``confidence`` (the verdict) is "sure" when every such unknown rule has the same action as
the deciding rule (then the verdict holds either way), "unknown" otherwise;
- notes about guesses (interface guessed from the address, NAT nearby, pf ruleset missing,
TCP flags, rate limits) lower a "sure" to "probable".
"""
from __future__ import annotations
import ipaddress
from dataclasses import dataclass, field
from app.services.rule_analysis import resolve
from app.services.rule_analysis.context import Context
from app.services.rule_analysis.model import IPAddr, NormRule, Outcome, t_and, tri_str
from app.services.rule_analysis.normalize import norm_proto, normalize
SURE, PROBABLE, UNKNOWN = "sure", "probable", "unknown"
_RANK = {SURE: 2, PROBABLE: 1, UNKNOWN: 0}
DOWNGRADING_NOTES = {
"IFACE_GUESSED", "EGRESS_GUESSED", "NAT_NEARBY", "TCP_FLAGS", "RATE_LIMIT", "ORDER_GUESSED",
"ORDER_MISMATCH", "PF_RULESET_UNAVAILABLE", "NAT_UNAVAILABLE", "QUICK_UNKNOWN",
}
_PORT_PROTOS = {"6", "17"}
_ICMP_PROTOS = {"1", "58"}
ICMP_NAMES = {
4: {"0": "echorep", "3": "unreach", "4": "squench", "5": "redir", "8": "echoreq", "9": "routeradv",
"10": "routersol", "11": "timex", "12": "paramprob", "13": "timereq", "14": "timerep"},
6: {"1": "unreach", "2": "toobig", "3": "timex", "4": "paramprob", "128": "echoreq", "129": "echorep",
"133": "routersol", "134": "routeradv", "135": "neighbrsol", "136": "neighbradv", "137": "redir"},
}
@dataclass
class Query:
src: IPAddr
dst: IPAddr
proto: str # normalized ("6", "17", "1", "58", other number or name)
proto_label: str
sport: int | None = None
dport: int | None = None
icmptype: str | None = None
iface: str | None = None # None = guess from the source address
direction: str = "in"
egress: str | None = "none" # "none" = only the first leg, None = guess, else identifier
@property
def af(self) -> int:
return self.src.version
@dataclass
class LegResult:
direction: str
iface: str | None
iface_source: str
action: str | None
decisive: NormRule | None
confidence: str
rule_confidence: str
notes: list[str] = field(default_factory=list)
unknowns: list[str] = field(default_factory=list)
steps: list[dict] = field(default_factory=list)
def _min_conf(*values: str) -> str:
return min(values, key=lambda v: _RANK[v])
def _max_conf(*values: str) -> str:
return max(values, key=lambda v: _RANK[v])
def _icmp_name(q: Query) -> str | None:
if q.icmptype is None:
return None
return ICMP_NAMES[q.af].get(q.icmptype, q.icmptype)
def rule_fields(r: NormRule, q: Query, direction: str, iface: str, ctx: Context,
tags: set[str], first_leg_iface: str | None) -> dict[str, Outcome]:
fields: dict[str, Outcome] = {
"interface": resolve.iface_match(r.interfaces, r.if_not, iface, ctx),
"direction": Outcome(r.direction in ("any", direction)),
"family": Outcome(q.af in r.af),
"protocol": Outcome(True if r.protos is None else q.proto in r.protos),
"source": resolve.endpoint_match(r.src, q.src, ctx),
"destination": resolve.endpoint_match(r.dst, q.dst, ctx),
}
if q.proto in _PORT_PROTOS:
fields["source_port"] = resolve.port_match(r.sport, q.sport, ctx, ephemeral_default=True)
fields["destination_port"] = resolve.port_match(r.dport, q.dport, ctx)
elif r.sport or r.dport:
fields["ports"] = Outcome(False) # pf only generates port rules for tcp/udp
if r.icmptypes is not None and q.proto in _ICMP_PROTOS:
name = _icmp_name(q)
fields["icmp_type"] = Outcome(None, ["ICMP_TYPE_NOT_GIVEN"]) if name is None else \
Outcome(name in r.icmptypes or q.icmptype in r.icmptypes)
if r.tagged:
fields["tagged"] = Outcome(True) if r.tagged in tags else Outcome(None, ["TAGGED"])
if r.received_on:
if direction == "in":
fields["received_on"] = Outcome(r.received_on == iface)
elif first_leg_iface:
fields["received_on"] = Outcome(r.received_on == first_leg_iface)
else:
fields["received_on"] = Outcome(None, ["RECEIVED_ON"])
if r.action is None:
fields["action"] = Outcome(None, ["ACTION_UNKNOWN"])
return fields
def _step(r: NormRule, fields: dict[str, Outcome], result, effect: str) -> dict:
return {
"rule": r.summary(), "result": tri_str(result), "effect": effect,
"fields": {k: tri_str(v.value) for k, v in fields.items()},
"reasons": sorted({reason for v in fields.values() if v.value is None for reason in v.reasons}),
}
def eval_leg(rules: list[NormRule], q: Query, ctx: Context, direction: str, iface: str,
first_leg_iface: str | None = None) -> LegResult:
steps: list[dict] = []
candidate: NormRule | None = None
possible: list[tuple[NormRule, bool]] = [] # (rule, came after the current candidate)
tags: set[str] = set()
final: NormRule | None = None
for r in rules:
if not r.enabled:
steps.append({"rule": r.summary(), "result": "false", "effect": "disabled", "fields": {}, "reasons": []})
continue
if r.loaded is False:
steps.append({"rule": r.summary(), "result": "false", "effect": "not_loaded", "fields": {}, "reasons": []})
continue
fields = rule_fields(r, q, direction, iface, ctx, tags, first_leg_iface)
result = t_and(*(f.value for f in fields.values()))
if result is False:
steps.append(_step(r, fields, result, "no_match"))
continue
if result is None or r.quick is None:
possible.append((r, True))
steps.append(_step(r, fields, result, "possible"))
continue
if r.quick:
final = r
for s in steps:
if s["effect"] == "candidate":
s["effect"] = "overridden"
steps.append(_step(r, fields, result, "final"))
break
if candidate is not None:
for s in steps:
if s["effect"] == "candidate":
s["effect"] = "overridden"
candidate = r
possible = [(p, False) for p, _ in possible]
if r.tag:
tags.add(r.tag)
steps.append(_step(r, fields, result, "candidate"))
decisive = final or candidate
if final is None:
for s in steps:
if s["effect"] == "candidate":
s["effect"] = "decisive"
else:
steps[-1]["effect"] = "decisive"
action = decisive.action if decisive else "pass" # pf without any matching rule passes
# Unknown matches that could still change the outcome: quick (or quick-unknown) ones always,
# non-quick ones only after the deciding candidate (an earlier one is overridden anyway).
relevant = [p for p, after in possible if p.quick is not False or (final is None and after)]
rule_conf = SURE if not relevant else PROBABLE
conf = SURE if all(p.action == action for p in relevant) else UNKNOWN
unknowns = sorted({reason for s in steps if s["effect"] == "possible" for reason in s["reasons"]})
if any(p.quick is None for p, _ in possible):
unknowns = sorted({*unknowns, "QUICK_UNKNOWN"})
notes: list[str] = []
if decisive is None:
notes.append("PF_DEFAULT_PASS")
else:
notes += decisive.extra_notes
if decisive.gateway and action == "pass":
notes.append("POLICY_ROUTE")
if decisive.statetype.lower() == "none" or "no state" in getattr(decisive.pf_line, "text", ""):
notes.append("STATELESS")
if decisive.sched:
notes.append("SCHEDULE_NOW")
if any(s["effect"] == "not_loaded" and s["rule"]["loaded"] is False for s in steps):
notes.append("RULES_NOT_LOADED")
return LegResult(direction=direction, iface=iface, iface_source="given", action=action, decisive=decisive,
confidence=conf, rule_confidence=rule_conf, notes=notes, unknowns=unknowns, steps=steps)
def guess_iface(ip: IPAddr, ctx: Context) -> str | None:
"""The interface whose connected network holds ``ip`` (longest prefix), else the one with
the default gateway."""
best, best_len = None, -1
for info in ctx.interfaces.values():
if info.identifier == "lo0":
continue
for net in info.nets:
if net.version == ip.version and ip in net and net.prefixlen > best_len and not net.is_link_local:
best, best_len = info.identifier, net.prefixlen
if best:
return best
gateways = [i.identifier for i in ctx.interfaces.values() if i.has_gateway]
return gateways[0] if len(gateways) == 1 else None
def _nat_nearby(q: Query, iface: str, ctx: Context) -> bool:
for row in ctx.port_forwards or []:
if str(row.get("disabled") or "0") == "1" or str(row.get("enabled", "1")) == "0":
continue
if str(row.get("nordr") or "0") == "1": # "no rdr" exclusions (anti-lockout) redirect nothing
continue
if q.dport is not None and resolve.port_match(str(row.get("destination.port") or ""), q.dport, ctx).value is False:
continue
ifaces = {i.strip().lower() for i in str(row.get("interface") or "").split(",") if i.strip()}
if ifaces and iface not in ifaces:
continue
proto = str(row.get("protocol") or "").lower()
if proto and proto not in ("any", "tcp/udp") and norm_proto(proto) != q.proto:
continue
return True
return False
def evaluate(ctx: Context, q: Query) -> dict:
rules, order_notes = normalize(ctx)
context_notes = [*ctx.notes, *order_notes]
legs: list[LegResult] = []
iface, iface_source = q.iface, "given"
if iface is None:
iface, iface_source = guess_iface(q.src if q.direction == "in" else q.dst, ctx), "guessed"
if iface is None:
return _result(q, legs, context_notes + ["IFACE_UNRESOLVED"], unresolved=True)
first = eval_leg(rules, q, ctx, q.direction, iface)
first.iface_source = iface_source
if iface_source == "guessed":
first.notes.append("IFACE_GUESSED")
if q.direction == "in" and (ctx.port_forwards is None or _nat_nearby(q, iface, ctx)):
first.notes.append("NAT_NEARBY" if ctx.port_forwards is not None else "NAT_UNAVAILABLE")
legs.append(first)
if q.direction == "in" and q.egress != "none":
egress, source = q.egress, "given"
if egress is None:
egress, source = guess_iface(q.dst, ctx), "guessed"
if egress is None:
context_notes.append("EGRESS_UNRESOLVED")
else:
out = eval_leg(rules, q, ctx, "out", egress, first_leg_iface=iface)
out.iface_source = source
if source == "guessed":
out.notes.append("EGRESS_GUESSED")
legs.append(out)
return _result(q, legs, context_notes)
def _leg_conf(leg: LegResult, context_notes: list[str]) -> tuple[str, str]:
downgrade = any(n in DOWNGRADING_NOTES for n in (*leg.notes, *context_notes))
conf, rule_conf = leg.confidence, leg.rule_confidence
if downgrade:
conf = _min_conf(conf, PROBABLE)
rule_conf = _min_conf(rule_conf, PROBABLE)
return conf, rule_conf
def _result(q: Query, legs: list[LegResult], context_notes: list[str], unresolved: bool = False) -> dict:
leg_dicts, confs = [], []
for leg in legs:
conf, rule_conf = _leg_conf(leg, context_notes)
confs.append((leg, conf, rule_conf))
leg_dicts.append({
"direction": leg.direction, "interface": leg.iface, "interface_source": leg.iface_source,
"action": leg.action, "confidence": conf, "rule_confidence": rule_conf,
"decisive_rule": leg.decisive.summary() if leg.decisive else None,
"notes": sorted(set(leg.notes)), "unknowns": leg.unknowns,
})
if unresolved or not legs:
action, conf, rule_conf, decisive = None, UNKNOWN, UNKNOWN, None
else:
blocking = [c for c in confs if c[0].action != "pass"]
if blocking:
leg, _, rule_conf = blocking[0]
action, decisive = leg.action, leg.decisive
# Blocked if any blocking leg is right: the best of their confidences counts.
conf = _max_conf(*(c[1] for c in blocking))
if conf == SURE and any(c[0].action != action for c in blocking):
conf = PROBABLE
else:
leg, _, rule_conf = confs[0]
action, decisive = "pass", leg.decisive
conf = _min_conf(*(c[1] for c in confs))
return {
"query": {
"src": str(q.src), "dst": str(q.dst), "proto": q.proto_label, "sport": q.sport, "dport": q.dport,
"icmptype": q.icmptype, "direction": q.direction,
"assumptions": ["FIRST_PACKET"] + (["SRC_PORT_EPHEMERAL"] if q.sport is None and q.proto in _PORT_PROTOS else []),
},
"action": action, "confidence": conf, "rule_confidence": rule_conf,
"decisive_rule": decisive.summary() if decisive else None,
"notes": sorted({*context_notes, *(n for leg in legs for n in leg.notes)}),
"unknowns": sorted({u for leg in legs for u in leg.unknowns}),
"legs": leg_dicts,
"trace": [{"leg": i, "direction": leg.direction, "interface": leg.iface, "steps": leg.steps}
for i, leg in enumerate(legs)],
}
def parse_ip(value: str) -> IPAddr:
return ipaddress.ip_address(value.strip())
+122
View File
@@ -0,0 +1,122 @@
"""Shared types of the rule analysis: three-valued logic and the normalized rule.
``Tri`` is Kleene logic with ``True``/``False``/``None`` (unknown). An unknown never turns
into a definite answer by negation, and an AND is only False when one side is definitely
False. This is what keeps the engine from claiming "sure" for something it cannot prove.
"""
from __future__ import annotations
import ipaddress
from dataclasses import dataclass, field
from typing import Optional
Tri = Optional[bool]
IPAddr = ipaddress.IPv4Address | ipaddress.IPv6Address
IPNet = ipaddress.IPv4Network | ipaddress.IPv6Network
def t_and(*values: Tri) -> Tri:
if any(v is False for v in values):
return False
if any(v is None for v in values):
return None
return True
def t_or(*values: Tri) -> Tri:
if any(v is True for v in values):
return True
if any(v is None for v in values):
return None
return False
def t_not(value: Tri) -> Tri:
return None if value is None else not value
def tri_str(value: Tri) -> str:
return "unknown" if value is None else ("true" if value else "false")
class Outcome:
"""A Tri plus the reason codes that made it unknown (or noteworthy)."""
__slots__ = ("value", "reasons")
def __init__(self, value: Tri, reasons: tuple[str, ...] | list[str] = ()):
self.value = value
self.reasons = list(reasons)
def __repr__(self) -> str: # pragma: no cover - debugging aid
return f"Outcome({self.value!r}, {self.reasons!r})"
def o_or(outcomes: list[Outcome]) -> Outcome:
value = t_or(*(o.value for o in outcomes)) if outcomes else False
reasons = [] if value is True else [r for o in outcomes if o.value is None for r in o.reasons]
return Outcome(value, reasons)
SECTION_SYSTEM, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE = "system", "floating", "group", "interface"
SECTION_NAT_AUTO, SECTION_INVALID, SECTION_ANTISPOOF, SECTION_OTHER = "nat_auto", "invalid", "antispoof", "other"
@dataclass
class Endpoint:
raw: str
negated: bool = False
items: list[str] = field(default_factory=list) # empty = any
@property
def is_any(self) -> bool:
return not self.items
@dataclass
class NormRule:
uuid: str
label: str
description: str
order: tuple[int, int]
order_known: bool
section: str
enabled: bool
action: str | None # pass | block | reject | None (unknown)
quick: bool | None # None = unknown (automatic rule without pf ruleset)
interfaces: set[str] # identifiers or group names; empty = every interface
if_not: bool
direction: str # in | out | any
af: set[int] # {4}, {6} or {4, 6}
protos: set[str] | None # None = any; normalized names/numbers as strings
src: Endpoint
dst: Endpoint
sport: str
dport: str
icmptypes: set[str] | None = None # None = no restriction
tag: str = ""
tagged: str = ""
received_on: str = ""
gateway: str = ""
statetype: str = ""
sched: str = ""
automatic: bool = False
legacy: bool = False
loaded: bool | None = None # None = pf ruleset not available
pf_numbers: list[int] = field(default_factory=list)
extra_notes: list[str] = field(default_factory=list) # e.g. TCP_FLAGS, RATE_LIMIT from pf text
pf_line: object = None # set for unlabeled pf lines (antispoof)
def summary(self) -> dict:
return {
"uuid": self.uuid, "label": self.label, "description": self.description,
"section": self.section, "sort_order": f"{self.order[0]:06d}.{self.order[1]:07d}" if self.order_known else None,
"pf_numbers": self.pf_numbers, "action": self.action, "quick": self.quick,
"interfaces": sorted(self.interfaces), "interface_not": self.if_not, "direction": self.direction,
"automatic": self.automatic, "legacy": self.legacy, "loaded": self.loaded,
"editable": not self.automatic and not self.legacy and self.section != SECTION_ANTISPOOF,
"source": ("!" if self.src.negated else "") + (self.src.raw or "any"),
"destination": ("!" if self.dst.negated else "") + (self.dst.raw or "any"),
"source_port": self.sport or None, "destination_port": self.dport or None,
"gateway": self.gateway or None, "schedule": self.sched or None,
}
@@ -0,0 +1,211 @@
"""Turn ``searchRule`` rows (+ the loaded pf ruleset) into ``NormRule`` objects.
Format pitfalls seen live on OPNsense 26.7:
- automatic rows carry the interface *description* (``"LAN"``), MVC rows the identifier;
- automatic rows often have ``quick``/``action`` empty; the pf lines with the same label
know the truth (``pass in quick on vtnet1_vlan10 ...``);
- empty source/destination means any; address lists are comma separated;
- unmigrated legacy rules have a 32-hex label instead of a UUID.
Order is ``sort_order`` (``prio.0seq`` from FilterRuleField::getPriority: 1/5 system,
10000 force gateway, 200000 floating, 300000+n groups, 400000 interface, 500000 automatic
NAT, 600000 = interface missing, never loaded).
"""
from __future__ import annotations
from app.services.rule_analysis.context import Context
from app.services.rule_analysis.model import (
SECTION_ANTISPOOF, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE, SECTION_INVALID,
SECTION_NAT_AUTO, SECTION_OTHER, SECTION_SYSTEM, Endpoint, NormRule,
)
from app.services.rule_analysis.pf_ruleset import PfLine
PROTO_NUMBERS = {
"icmp": "1", "igmp": "2", "ggp": "3", "ipencap": "4", "tcp": "6", "egp": "8", "udp": "17",
"ipv6": "41", "gre": "47", "esp": "50", "ah": "51", "ipv6-icmp": "58", "icmp6": "58",
"icmpv6": "58", "ospf": "89", "pim": "103", "vrrp": "112", "carp": "112", "sctp": "132", "pfsync": "240",
}
def norm_proto(value: str) -> str:
value = str(value).strip().lower()
return PROTO_NUMBERS.get(value, value)
def _truthy(value) -> bool:
return value is True or str(value).strip().lower() in ("1", "true", "yes", "on")
def _section(prio: int) -> str:
if prio < 10001:
return SECTION_SYSTEM
if prio == 200000:
return SECTION_FLOATING
if 300000 <= prio < 400000:
return SECTION_GROUP
if prio == 400000:
return SECTION_INTERFACE
if prio == 500000:
return SECTION_NAT_AUTO
if prio >= 600000:
return SECTION_INVALID
return SECTION_OTHER
def _order(row: dict) -> tuple[tuple[int, int], bool]:
raw = str(row.get("sort_order") or "")
if "." in raw:
head, _, tail = raw.partition(".")
if head.isdigit() and tail.isdigit():
return (int(head), int(tail)), True
# Older releases without sort_order: best effort from prio_group/sequence (confidence drops).
prio = str(row.get("prio_group") or row.get("#priority") or "")
seq = str(row.get("sequence") or "")
return (int(prio) if prio.isdigit() else 400000, int(seq) if seq.isdigit() else 0), False
def _endpoint(row: dict, side: str) -> Endpoint:
raw = str(row.get(f"{side}_net") or "").strip()
items = [] if raw.lower() in ("", "any") else [i.strip() for i in raw.split(",") if i.strip()]
return Endpoint(raw=raw, negated=_truthy(row.get(f"{side}_not")), items=items)
def _af(value: str) -> set[int]:
value = str(value or "").strip().lower()
return {4} if value == "inet" else ({6} if value == "inet6" else {4, 6})
def _protos(value: str) -> set[str] | None:
value = str(value or "").strip().lower()
if value in ("", "any"):
return None
return {norm_proto(p) for p in value.split("/") if p}
def _pf_interfaces(lines: list[PfLine], ctx: Context) -> tuple[set[str], bool] | None:
devices = {(line.on_device, line.on_negated) for line in lines}
if any(device is None for device, _ in devices):
return set(), False
if len({neg for _, neg in devices}) != 1:
return None
names: set[str] = set()
for device, _ in devices:
info = ctx.by_device(device)
if info is not None:
names.add(info.identifier)
elif ctx.groups is not None and device in ctx.groups:
names.add(device)
else:
return None
return names, next(iter(devices))[1]
def normalize_row(row: dict, ctx: Context) -> NormRule:
order, order_known = _order(row)
label = str(row.get("uuid") or "")
automatic = _truthy(row.get("is_automatic"))
lines = ctx.pf.by_label.get(label, []) if ctx.pf else []
rule = NormRule(
uuid=label, label=label, description=str(row.get("description") or ""),
order=order, order_known=order_known, section=_section(order[0]),
enabled=_truthy(row.get("enabled", "1")),
action=(str(row.get("action") or "").lower() or None),
quick=None, interfaces=set(), if_not=_truthy(row.get("interfacenot")),
direction=(str(row.get("direction") or "in").lower() or "in"),
af=_af(row.get("ipprotocol")), protos=_protos(row.get("protocol")),
src=_endpoint(row, "source"), dst=_endpoint(row, "destination"),
sport=str(row.get("source_port") or "").strip(), dport=str(row.get("destination_port") or "").strip(),
tag=str(row.get("tag") or ""), tagged=str(row.get("tagged") or ""),
received_on=str(row.get("received-on") or row.get("received_on") or ""),
gateway=str(row.get("gateway") or ""), statetype=str(row.get("statetype") or ""),
sched=str(row.get("sched") or ""), automatic=automatic, legacy=_truthy(row.get("legacy")),
pf_numbers=[line.number for line in lines],
)
if rule.action not in ("pass", "block", "reject"):
rule.action = None
if row.get("quick") not in (None, ""):
rule.quick = _truthy(row.get("quick"))
raw_if = str(row.get("interface") or "").strip()
names = {n.strip() for n in raw_if.split(",") if n.strip()}
# Automatic and unmigrated legacy rows carry descriptions ("LAN"): map them to identifiers.
# Identifiers and group names stay as they are; the pf "on" clause decides when available.
mapped = set()
for name in names:
info = None if name in ctx.interfaces or name in (ctx.groups or {}) else ctx.by_label(name)
mapped.add(info.identifier if info else name)
names = mapped
rule.interfaces = names
if ctx.pf is not None:
rule.loaded = bool(lines) and rule.enabled
if lines:
actions = {line.action for line in lines}
if len(actions) == 1 and next(iter(actions)) in ("pass", "block", "reject"):
pf_action = next(iter(actions))
if rule.action is None or automatic:
rule.action = pf_action
quick = {line.quick for line in lines}
rule.quick = next(iter(quick)) if len(quick) == 1 else None
if automatic or rule.legacy:
on = _pf_interfaces(lines, ctx)
if on is not None:
rule.interfaces, rule.if_not = on
directions = {line.direction for line in lines}
if len(directions) == 1:
rule.direction = next(iter(directions))
families = {line.af for line in lines}
if None not in families:
rule.af &= families
types = {t for line in lines for t in line.icmptypes}
if types:
rule.icmptypes = types
if any(line.flags not in (None, "S/SA", "any") for line in lines):
rule.extra_notes.append("TCP_FLAGS")
if any(word in line.text for line in lines for word in ("max-src-", "overload", "max-pkt-rate")):
rule.extra_notes.append("RATE_LIMIT")
elif rule.section == SECTION_INVALID:
rule.loaded = False
if rule.icmptypes is None and str(row.get("icmptype") or "").strip():
rule.icmptypes = {t.strip() for t in str(row["icmptype"]).split(",") if t.strip()}
return rule
def antispoof_rule(line: PfLine, order: tuple[int, int]) -> NormRule:
"""An unlabeled pf line OPNsense generated itself (antispoof / own-address block)."""
return NormRule(
uuid=f"pf@{line.number}", label="", description=line.text, order=order, order_known=True,
section=SECTION_ANTISPOOF, enabled=True, action=line.action if line.action != "other" else None,
quick=line.quick, interfaces=set(), if_not=False, direction=line.direction,
af={line.af} if line.af else {4, 6}, protos=None,
src=Endpoint(raw="" if line.src == "any" else line.src, items=[] if line.src == "any" else [line.src]),
dst=Endpoint(raw="" if line.dst == "any" else line.dst, items=[] if line.dst == "any" else [line.dst]),
sport="", dport="", automatic=True, loaded=True, pf_numbers=[line.number], pf_line=line,
)
def normalize(ctx: Context) -> tuple[list[NormRule], list[str]]:
"""All rules in evaluation order plus context notes (order problems)."""
notes: list[str] = []
rules = [normalize_row(row, ctx) for row in ctx.rows]
if any(not r.order_known for r in rules):
notes.append("ORDER_GUESSED")
rules.sort(key=lambda r: (r.order, r.uuid))
if ctx.pf is not None:
loaded = [r for r in rules if r.loaded and r.pf_numbers]
firsts = [min(r.pf_numbers) for r in loaded]
if firsts != sorted(firsts):
notes.append("ORDER_MISMATCH")
# Unlabeled pf lines: placed before the first configured rule that pf evaluates later.
extra: list[tuple[tuple, NormRule]] = []
for line in ctx.pf.unlabeled:
after = next((r for r in loaded if min(r.pf_numbers) > line.number), None)
key = after.order if after else (999999, 9999999)
spoof = antispoof_rule(line, key)
if line.on_device:
info = ctx.by_device(line.on_device)
spoof.interfaces = {info.identifier if info else line.on_device}
spoof.if_not = line.on_negated
extra.append(((key, 0, line.number), spoof))
keyed = [((r.order, 1, 0), r) for r in rules] + extra
keyed.sort(key=lambda item: item[0])
rules = [r for _, r in keyed]
return rules, notes
@@ -0,0 +1,93 @@
"""The ruleset pf has actually loaded (``diagnostics/firewall/pfStatistics/rules``).
Each key looks like ``@76 pass in quick on vtnet1_vlan10 inet from (...) to any flags S/SA
keep state label "<uuid>"``. One configured rule can expand to several pf lines (address
families, ICMP types, port lists); they share one label. Lines without a label are generated
by OPNsense itself (antispoof: ``block drop in log on ! vtnet1_vlan10 inet from 192.0.2.0/24
to any``). Only what the analysis needs is parsed: position, action, quick, direction, the
``on`` interface, address family, ICMP types and a few pf options that make the answer
uncertain.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
_HEAD = re.compile(r"^@(\d+)\s+(.*)$")
_LABEL = re.compile(r'label "([^"]+)"')
_ON = re.compile(r"\bon (!\s*)?([A-Za-z0-9_.]+)")
_ICMP = re.compile(r"\bicmp6?-type (\S+)")
_FROM = re.compile(r"\bfrom (\S+) to (\S+)")
_FLAGS = re.compile(r"\bflags (\S+)")
@dataclass
class PfLine:
number: int
text: str
action: str # pass | block | reject | match | scrub | other
quick: bool
direction: str # in | out | any
on_negated: bool
on_device: str | None
af: int | None # 4, 6 or None (both)
label: str | None
icmptypes: list[str] = field(default_factory=list)
src: str = "any"
dst: str = "any"
flags: str | None = None
def parse_line(number: int, text: str) -> PfLine:
words = text.split()
first = words[0] if words else ""
action = first if first in ("pass", "match", "scrub") else "other"
if first == "block":
action = "reject" if len(words) > 1 and words[1].startswith("return") else "block"
direction = "in" if " in " in f" {text} " else ("out" if " out " in f" {text} " else "any")
on = _ON.search(text)
af = 4 if re.search(r"\binet\b", text) else (6 if re.search(r"\binet6\b", text) else None)
label = _LABEL.search(text)
endpoints = _FROM.search(text)
flags = _FLAGS.search(text)
return PfLine(
number=number, text=text, action=action, quick=" quick " in f" {text} ",
direction=direction, on_negated=bool(on and on.group(1)), on_device=on.group(2) if on else None,
af=af, label=label.group(1) if label else None, icmptypes=_ICMP.findall(text),
src=endpoints.group(1) if endpoints else ("any" if " all" in f" {text}" else "?"),
dst=endpoints.group(2) if endpoints else ("any" if " all" in f" {text}" else "?"),
flags=flags.group(1) if flags else None,
)
@dataclass
class PfRuleset:
lines: list[PfLine]
by_label: dict[str, list[PfLine]]
@property
def unlabeled(self) -> list[PfLine]:
return [line for line in self.lines if line.label is None and line.action in ("pass", "block", "reject")]
def parse(stats: dict | None) -> PfRuleset | None:
"""None when the answer does not look like a pf ruleset at all."""
rules = (stats or {}).get("rules") if isinstance(stats, dict) else None
filter_rules = rules.get("filter rules") if isinstance(rules, dict) else None
if not isinstance(filter_rules, dict):
return None
lines: list[PfLine] = []
for key in filter_rules:
head = _HEAD.match(str(key).strip())
if not head:
continue
line = parse_line(int(head.group(1)), head.group(2))
if line.action == "scrub":
continue
lines.append(line)
lines.sort(key=lambda line: line.number)
by_label: dict[str, list[PfLine]] = {}
for line in lines:
if line.label:
by_label.setdefault(line.label, []).append(line)
return PfRuleset(lines=lines, by_label=by_label)
@@ -0,0 +1,211 @@
"""Does a rule field match the queried packet? Addresses, interfaces, aliases, ports.
Every function answers with an ``Outcome`` (True/False/None + reason codes). Whatever cannot
be proven from the firewall's own data is ``None`` (unknown), never a guess.
"""
from __future__ import annotations
import ipaddress
import re
from app.services.rule_analysis.context import Context
from app.services.rule_analysis.model import IPAddr, Outcome, o_or, t_not
MAX_ALIAS_DEPTH = 10
TABLE_ONLY_TYPES = {"url", "urltable", "geoip", "asn", "mac", "dynipv6host", "authgroup", "external", "internal"}
ADDRESS_TYPES = {"host", "network", "networkgroup"}
SERVICES = {
"ftp-data": 20, "ftp": 21, "ssh": 22, "telnet": 23, "smtp": 25, "domain": 53, "dns": 53, "bootps": 67,
"bootpc": 68, "tftp": 69, "http": 80, "www": 80, "kerberos": 88, "pop3": 110, "ntp": 123, "imap": 143,
"snmp": 161, "snmptrap": 162, "ldap": 389, "https": 443, "smb": 445, "microsoft-ds": 445, "isakmp": 500,
"syslog": 514, "submission": 587, "ldaps": 636, "imaps": 993, "pop3s": 995, "openvpn": 1194,
"ms-sql-s": 1433, "radius": 1812, "radius-acct": 1813, "nfs": 2049, "mysql": 3306, "rdp": 3389,
"ipsec-nat-t": 4500, "sip": 5060, "postgresql": 5432, "dhcpv6-client": 546, "dhcpv6-server": 547,
"http-alt": 8080, "wireguard": 51820,
}
EPHEMERAL = (1024, 65535)
def _ip_in(token: str, ip: IPAddr) -> bool | None:
"""Literal IP / CIDR / range; None when the token is not a literal."""
token = token.strip()
if "-" in token and not token.startswith("-"):
low, _, high = token.partition("-")
try:
low_ip, high_ip = ipaddress.ip_address(low.strip()), ipaddress.ip_address(high.strip())
except ValueError:
return None
return low_ip.version == ip.version and low_ip <= ip <= high_ip
try:
net = ipaddress.ip_network(token, strict=False)
except ValueError:
return None
return net.version == ip.version and ip in net
def _interface_ref(token: str, ip: IPAddr, ctx: Context) -> Outcome | None:
name = token.strip()
lower = name.lower()
m = re.fullmatch(r"__(.+)_network", lower)
if m and m.group(1) in ctx.interfaces:
lower = m.group(1)
info = ctx.interfaces.get(lower)
if info is not None: # "lan" = the interface's networks (pf: (dev:network), incl. VIP subnets)
return Outcome(any(ip in net for net in info.nets))
if lower.endswith("ip") and lower[:-2] in ctx.interfaces: # "lanip" = the interface address
info = ctx.interfaces[lower[:-2]]
if ip in info.addrs:
return Outcome(True)
if ip in info.vips:
return Outcome(None, ["VIP_AS_INTERFACE_ADDRESS"])
return Outcome(False)
return None
def addr_item(token: str, ip: IPAddr, ctx: Context, *, depth: int = 0, seen: tuple = ()) -> Outcome:
token = token.strip()
if token.lower() in ("any", ""):
return Outcome(True)
if token.lower() in ("(self)", "self", "this firewall"):
return Outcome(ip in ctx.self_addrs)
literal = _ip_in(token, ip)
if literal is not None:
return Outcome(literal)
ref = _interface_ref(token, ip, ctx)
if ref is not None:
return ref
return alias_contains(token, ip, ctx, depth=depth, seen=seen)
def alias_contains(name: str, ip: IPAddr, ctx: Context, *, depth: int = 0, seen: tuple = ()) -> Outcome:
if name in seen:
return Outcome(None, ["ALIAS_CYCLE"])
if depth > MAX_ALIAS_DEPTH:
return Outcome(None, ["ALIAS_TOO_DEEP"])
alias = ctx.aliases.get(name)
refs = ctx.refs.get(str(ip))
if alias is None:
# pf knows tables the alias list may not show (e.g. internal ones): trust a positive hit.
if refs is not None and name in refs:
return Outcome(True)
return Outcome(None, ["ALIAS_MISSING"])
kind = str(alias.get("type") or "").lower()
if depth == 0 and refs is not None:
# pf's own table test: authoritative for every alias type (nesting, "!" entries, FQDNs).
return Outcome(name in refs)
if kind in ADDRESS_TYPES:
return _alias_content(alias, ip, ctx, depth, (*seen, name))
if refs is not None:
return Outcome(name in refs)
return Outcome(None, ["ALIAS_TABLE_UNAVAILABLE"])
def _alias_content(alias: dict, ip: IPAddr, ctx: Context, depth: int, seen: tuple) -> Outcome:
entries = [e.strip() for e in re.split(r"[\n,]", str(alias.get("content") or "")) if e.strip()]
if any(e.startswith("!") for e in entries):
return Outcome(None, ["ALIAS_EXCLUSION"])
results = []
for entry in entries:
literal = _ip_in(entry, ip)
if literal is not None:
results.append(Outcome(literal))
elif entry in ctx.aliases or entry in seen:
results.append(alias_contains(entry, ip, ctx, depth=depth + 1, seen=seen))
elif "." in entry or ":" in entry:
results.append(Outcome(None, ["FQDN_UNRESOLVED"]))
else:
results.append(Outcome(None, ["ALIAS_MISSING"]))
return o_or(results)
def endpoint_match(endpoint, ip: IPAddr, ctx: Context) -> Outcome:
if endpoint.is_any:
return Outcome(True)
result = o_or([addr_item(item, ip, ctx) for item in endpoint.items])
if endpoint.negated:
return Outcome(t_not(result.value), result.reasons)
return result
# ── Ports ─────────────────────────────────────────────────────────────────
def port_ranges(spec: str, ctx: Context, *, depth: int = 0, seen: tuple = ()) -> tuple[list[tuple[int, int]] | None, list[str]]:
"""The port ranges a spec stands for; (None, reasons) when that cannot be known."""
ranges: list[tuple[int, int]] = []
reasons: list[str] = []
for part in [p.strip() for p in re.split(r"[\n,]", spec) if p.strip()]:
m = re.fullmatch(r"(\w[\w-]*?)\s*[-:]\s*(\w[\w-]*)", part) if not part.isdigit() and part not in ctx.aliases else None
if part.isdigit():
ranges.append((int(part), int(part)))
elif m and _port_number(m.group(1)) is not None and _port_number(m.group(2)) is not None:
ranges.append((_port_number(m.group(1)), _port_number(m.group(2))))
elif part.lower() in SERVICES:
ranges.append((SERVICES[part.lower()],) * 2)
elif part in ctx.aliases:
if part in seen or depth >= MAX_ALIAS_DEPTH:
return None, ["ALIAS_CYCLE" if part in seen else "ALIAS_TOO_DEEP"]
alias = ctx.aliases[part]
if str(alias.get("type") or "").lower() != "port":
return None, ["PORT_ALIAS_UNRESOLVED"]
sub, why = port_ranges(str(alias.get("content") or ""), ctx, depth=depth + 1, seen=(*seen, part))
if sub is None:
return None, why
ranges += sub
else:
return None, ["PORT_ALIAS_UNRESOLVED" if part.isidentifier() else "PORT_UNPARSEABLE"]
return ranges, reasons
def _port_number(token: str) -> int | None:
if token.isdigit():
return int(token)
return SERVICES.get(token.lower())
def port_match(spec: str, port: int | None, ctx: Context, *, ephemeral_default: bool = False) -> Outcome:
spec = (spec or "").strip()
if spec.lower() in ("", "any"):
return Outcome(True)
ranges, reasons = port_ranges(spec, ctx)
if ranges is None:
return Outcome(None, reasons)
if port is not None:
return Outcome(any(low <= port <= high for low, high in ranges))
if not ephemeral_default:
return Outcome(None, ["PORT_NOT_GIVEN"])
# No source port given: the query stands for an ephemeral client port (1024-65535).
low_e, high_e = EPHEMERAL
overlap = [(max(low, low_e), min(high, high_e)) for low, high in ranges if high >= low_e and low <= high_e]
if not overlap:
return Outcome(False)
covered, cursor = True, low_e
for low, high in sorted(overlap):
if low > cursor:
covered = False
break
cursor = max(cursor, high + 1)
if covered and cursor > high_e:
return Outcome(True)
return Outcome(None, ["SRC_PORT_NOT_GIVEN"])
# ── Interfaces ────────────────────────────────────────────────────────────
def iface_match(interfaces: set[str], if_not: bool, iface: str, ctx: Context) -> Outcome:
if not interfaces:
return Outcome(True)
results = []
for name in interfaces:
if name == iface:
results.append(Outcome(True))
elif name in ctx.interfaces:
results.append(Outcome(False))
elif ctx.groups is not None and name in ctx.groups:
results.append(Outcome(iface in ctx.groups[name]))
else:
results.append(Outcome(None, ["IFACE_UNRESOLVED" if ctx.groups is not None else "GROUPS_UNAVAILABLE"]))
result = o_or(results)
if if_not:
return Outcome(t_not(result.value), result.reasons)
return result
@@ -0,0 +1,76 @@
"""Entry point used by the API: validate a query, load the firewall's context, evaluate."""
from __future__ import annotations
import ipaddress
import logging
from app.errors import ErrorCode, api_error
from app.models.host import OpnsenseHost
from app.services.rule_analysis import context as ctx_mod
from app.services.rule_analysis.engine import Query, evaluate
from app.services.rule_analysis.normalize import PROTO_NUMBERS, norm_proto
logger = logging.getLogger(__name__)
def _ip(value: str | None, field: str):
try:
return ipaddress.ip_address(str(value or "").strip())
except ValueError:
raise api_error(422, ErrorCode.ANALYSE_ADRESSE_UNGUELTIG, field=field, value=str(value or "")[:64]) from None
def _port(value: str | int | None, field: str) -> int | None:
if value in (None, ""):
return None
text = str(value).strip()
if not text.isdigit() or int(text) > 65535:
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field=field, value=text[:32])
return int(text)
def build_query(*, src: str, dst: str, proto: str = "tcp", sport=None, dport=None, icmptype: str | None = None,
iface: str | None = None, direction: str = "in", egress: str | None = None) -> Query:
source, destination = _ip(src, "src"), _ip(dst, "dst")
if source.version != destination.version:
raise api_error(422, ErrorCode.ANALYSE_FAMILIEN_MISCHUNG)
label = str(proto or "tcp").strip().lower()
if label == "icmp" and source.version == 6:
label = "ipv6-icmp"
number = norm_proto(label)
if not (number.isdigit() and int(number) <= 255) and label not in PROTO_NUMBERS:
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="proto", value=label[:32])
q = Query(src=source, dst=destination, proto=number, proto_label=label,
sport=_port(sport, "sport"), dport=_port(dport, "dport"))
if (q.sport is not None or q.dport is not None) and number not in ("6", "17"):
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="dport", value=str(dport or sport))
if icmptype not in (None, ""):
if number not in ("1", "58"):
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="icmptype", value=str(icmptype)[:32])
q.icmptype = str(icmptype).strip().lower()
if direction not in ("in", "out"):
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="direction", value=str(direction)[:8])
q.direction = direction
q.iface = None if str(iface or "").strip().lower() in ("", "auto") else str(iface).strip()
egress_value = str(egress or "").strip().lower()
q.egress = "none" if egress_value in ("", "none") else (None if egress_value == "auto" else str(egress).strip())
return q
def check_interfaces(q: Query, ctx: ctx_mod.Context) -> None:
for name in (q.iface, None if q.egress in (None, "none") else q.egress):
if name is not None and name not in ctx.interfaces:
raise api_error(422, ErrorCode.ANALYSE_INTERFACE_UNBEKANNT, interface=name[:64],
known=", ".join(sorted(i for i in ctx.interfaces if i != "lo0")))
async def analyze_host(host: OpnsenseHost, q: Query, *, refresh: bool = False) -> dict:
"""Raises api_error for a bad interface; OpnsenseAPIError/httpx errors when the firewall
cannot be read (callers map that)."""
ctx = await ctx_mod.load_context(host, use_cache=not refresh)
check_interfaces(q, ctx)
await ctx_mod.ensure_refs(host, ctx, [str(q.src), str(q.dst)])
result = evaluate(ctx, q)
if any(ctx.refs.get(ip) is None for ip in (str(q.src), str(q.dst))):
result["notes"] = sorted({*result["notes"], "ALIAS_REFERENCES_UNAVAILABLE"})
return result
@@ -15,18 +15,31 @@ the UI) but is NOT wired into the NotificationSettings webhook/email
system — that system is built around persistent NotificationProblem rows
with their own admin-configurable toggles, and adding a new problem kind
for this is a bigger change than this feature's own scope. Flagged here,
not silently dropped, in case it's wanted later.
not silently dropped, in case it's wanted later. (Later: the failure is reported by
notification_checks.check_scheduled_maintenance_failed since 1.17.)
auto_verify (1.25, set for every rollout step): instead of leaving the operation "submitted"
for a human, the run waits for the job (phase job_wait) and then checks the result itself
(phase verify: firmware idle for a while, a reboot really happened when one was due, version
readable). Only then the operation is closed as "acknowledged"; anything unclear ends as
"unknown" (stays locked, notification). Without auto_verify nothing changes.
Concurrency: due rows and running rows are each taken with SELECT ... FOR UPDATE SKIP LOCKED,
one row per transaction, so a second backend (briefly possible during a web update) can never
fire or step the same row at the same time.
"""
import logging
from datetime import datetime, timedelta, timezone
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy import select, true
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.host import OpnsenseHost
from app.models.host_operation import HostOperation
from app.models.scheduled_maintenance import ScheduledMaintenance
from contextlib import contextmanager
from app.services import audit_service, backup_store, opnsense_client as client
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
from app.services.host_status_service import get_statuses
@@ -51,40 +64,93 @@ HANDOVER_TIMEOUT = timedelta(minutes=3)
LEAVE_TIMEOUT = timedelta(minutes=5)
JOB_GRACE = timedelta(minutes=5) # no sign of a job after this long: treat a package update as already done
JOB_TIMEOUT = {"reboot": timedelta(minutes=20), "update": timedelta(minutes=60), "major_update": timedelta(minutes=150)}
# auto_verify: the firmware must report "ready" for this long in a row (a reboot at the end of an update
# can follow a short "ready" moment), and the whole check gives up after VERIFY_TIMEOUT.
VERIFY_SETTLE = timedelta(seconds=60)
VERIFY_TIMEOUT = timedelta(minutes=20)
# A rollout step is created with run_at=now and fires on the next poll. Found much later (OPNexus was down,
# e.g. during an update), it no longer belongs to the decision that created it: the window may have closed
# and the run may have been paused meanwhile. It is then not fired; the rollout engine re-queues the host.
ROLLOUT_FIRE_GRACE = timedelta(minutes=5)
def _now() -> datetime:
return datetime.now(timezone.utc)
@contextmanager
def _as_actor(sched: ScheduledMaintenance):
"""Audit events of an unattended run name the person who planned it (not the rollback worker)."""
token = audit_service.context.set({"actor": f"{sched.actor} (geplant)"[:150], "id": str(sched.id)})
try:
yield
finally:
audit_service.context.reset(token)
def _rollout_details(sched: ScheduledMaintenance) -> dict:
rollout = (sched.progress or {}).get("rollout")
return {"rollout": rollout} if rollout else {}
async def fire_due(db: AsyncSession) -> dict:
due = list(
(
await db.scalars(
select(ScheduledMaintenance).where(
ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now()
)
)
).all()
)
fired = 0
blocked = 0
for sched in due:
try:
await _fire_one(db, sched)
except Exception:
logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
sched.status = "failed"
sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
due = fired = blocked = 0
seen: set = set()
while True:
# One row per transaction, locked: a second backend skips it instead of firing it again.
sched = await db.scalar(
select(ScheduledMaintenance)
.where(ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now(),
ScheduledMaintenance.id.not_in(seen) if seen else true())
.order_by(ScheduledMaintenance.run_at)
.limit(1)
.with_for_update(skip_locked=True)
.execution_options(populate_existing=True)
)
if sched is None:
await db.commit()
break
seen.add(sched.id)
due += 1
with _as_actor(sched):
try:
await _fire_one(db, sched)
except Exception:
logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
await db.rollback()
sched = await db.get(ScheduledMaintenance, sched.id, populate_existing=True)
if sched.status in ("pending", "running") and sched.phase is None:
sched.status = "failed"
sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
await db.commit()
await db.commit() # never keep the row lock past this row
if sched.status == "failed":
blocked += 1
else:
fired += 1
return {"due": len(due), "fired": fired, "blocked_or_failed": blocked}
return {"due": due, "fired": fired, "blocked_or_failed": blocked}
def _reboot_expected(action: str, state: dict) -> bool:
return action in ("reboot", "major_update") or bool(state.get("reboot_required"))
async def _boot_time(host: OpnsenseHost) -> str | None:
"""systemTime's "boottime" (Diagnostics SystemController::systemTimeAction, read-only, already used by the
resource sampler). A different value afterwards proves a reboot even if no poll caught the host offline."""
try:
return (await client.get_system_time(host)).get("boottime") or None
except client.OpnsenseAPIError:
return None
async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
if sched.origin == "rollout" and _now() - sched.run_at > ROLLOUT_FIRE_GRACE:
sched.status = "failed"
sched.result = {"error": "Rollout-Schritt nicht rechtzeitig gestartet, nichts ausgeführt", "not_fired": True}
await db.commit()
logger.warning("Geplante Wartung %s: Rollout-Schritt überfällig, nicht ausgeführt", sched.id)
return
host = await db.get(OpnsenseHost, sched.host_id)
if not host or not host.enabled:
sched.status = "failed"
@@ -137,6 +203,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
return
boot_before = await _boot_time(host) if sched.auto_verify else None
op = HostOperation(
host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
expires_at=_now(), fingerprint="", snapshot=state, status="starting", started_at=_now(),
@@ -147,14 +214,23 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
await audit_service.persist(
kind="host", action=sched.action, status="requested",
host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state},
details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state, **_rollout_details(sched)},
)
try:
result = await _ACTION_FN[sched.action](host)
op.status = "submitted"
op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
sched.status = "succeeded"
sched.result = {"response": result}
if sched.auto_verify:
# Not done yet: the poll loop waits for the job and checks the result (job_wait -> verify).
op.message = "Geplant übergeben. Abschluss wird automatisch geprüft."
sched.phase, sched.phase_since = "job_wait", _now()
_progress(sched, handover=False, job_started_at=_now().isoformat(), seen_down=False, seen_running=False,
version_before=state.get("version"), boot_before=boot_before,
reboot_expected=_reboot_expected(sched.action, state))
sched.result = {"response": result}
else:
op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
sched.status = "succeeded"
sched.result = {"response": result}
except client.OpnsenseAPIError as exc:
op.status = "unknown"
op.message = "Ausgang unklar: " + str(exc)[:800] + ". Nicht automatisch wiederholen."
@@ -166,7 +242,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
await audit_service.persist(
kind="host", action=sched.action, status=op.status,
host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result},
message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result, **_rollout_details(sched)},
)
@@ -182,21 +258,24 @@ async def _audit(host, sched, op, action, status, message=None, **details):
await audit_service.persist(
kind="host", action=action, status=status, host_id=str(host.id), host_name=host.name,
entity_id=str(op.id) if op else None, message=message,
details={"scheduled": True, "schedule_id": str(sched.id), "handover": True, **details},
details={"scheduled": True, "schedule_id": str(sched.id), "handover": bool((sched.progress or {}).get("handover")),
**_rollout_details(sched), **details},
)
async def _end(db, sched, host, op, *, ok, message, op_status=None, **result):
handover = bool((sched.progress or {}).get("handover"))
sched.status = "succeeded" if ok else "failed"
sched.phase = None
sched.result = {"handover": True, **result} if ok else {"error": message[:900], **result}
sched.result = ({"handover": handover, "auto_verified": bool(sched.auto_verify), **result} if ok
else {"error": message[:900], **result})
if op is not None:
op.status = op_status or ("acknowledged" if ok else "unknown")
op.message = message[:1000]
await db.commit()
await _audit(host, sched, op, sched.action, "completed" if ok else (op_status or "unknown"), message)
if not ok:
logger.warning("Geplante Wartung %s (CARP-Übergabe) beendet: %s", sched.id, message)
logger.warning("Geplante Wartung %s beendet: %s", sched.id, message)
async def _toggle(host, expected: str) -> None:
@@ -230,6 +309,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
await fail(" ".join(blocked))
return
boot_before = await _boot_time(host) if sched.auto_verify else None
now = _now()
op = HostOperation(
host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
@@ -237,6 +317,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
)
db.add(op)
sched.status = "running"
_progress(sched, handover=True, boot_before=boot_before, reboot_expected=_reboot_expected(sched.action, state))
await db.commit()
sched.host_operation_id = op.id
await _audit(host, sched, op, "carp_enter", "requested", snapshot=state)
@@ -255,32 +336,56 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
async def advance_running(db: AsyncSession) -> int:
"""One step for every handover run in progress. Returns how many are still running."""
rows = list((await db.scalars(
select(ScheduledMaintenance).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
"""One step for every run in progress (handover and/or auto_verify). Returns how many are still running."""
ids = list((await db.scalars(
select(ScheduledMaintenance.id).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
)).all())
for sched in rows:
host = await db.get(OpnsenseHost, sched.host_id)
op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
try:
if host is None:
raise RuntimeError("Host nicht mehr vorhanden")
if op is not None and op.status not in ACTIVE:
# A human closed or cancelled the operation: stop here and touch nothing more.
sched.status, sched.phase = "failed", None
sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt. CARP-Rolle von {host.name} prüfen."}
await db.commit()
continue
await _STEPS[sched.phase](db, sched, host, op)
except Exception as exc: # a stuck run must not loop forever or take the poll loop down
logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
if host is not None:
await _end(db, sched, host, op, ok=False,
message=f"Unerwarteter Fehler während der CARP-Übergabe ({type(exc).__name__}). "
f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen.")
return sum(1 for s in rows if s.status == "running")
await db.commit()
still_running = 0
for sched_id in ids:
# Locked for the whole step: a second backend skips the row instead of toggling or submitting twice.
sched = await db.scalar(
select(ScheduledMaintenance)
.where(ScheduledMaintenance.id == sched_id, ScheduledMaintenance.status == "running",
ScheduledMaintenance.phase.is_not(None))
.with_for_update(skip_locked=True)
.execution_options(populate_existing=True)
)
if sched is None:
await db.commit()
still_running += 1 if await db.scalar(select(ScheduledMaintenance.status).where(ScheduledMaintenance.id == sched_id)) == "running" else 0
await db.commit()
continue
with _as_actor(sched):
await _advance_one(db, sched)
await db.commit()
if sched.status == "running":
still_running += 1
return still_running
async def _advance_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
host = await db.get(OpnsenseHost, sched.host_id)
op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
try:
if host is None:
raise RuntimeError("Host nicht mehr vorhanden")
if op is not None and op.status not in ACTIVE:
# A human closed or cancelled the operation: stop here and touch nothing more.
hint = f" CARP-Rolle von {host.name} prüfen." if (sched.progress or {}).get("handover") else ""
sched.status, sched.phase = "failed", None
sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt.{hint}"}
await db.commit()
return
await _STEPS[sched.phase](db, sched, host, op)
except Exception as exc: # a stuck run must not loop forever or take the poll loop down
logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
if host is not None:
hint = (f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen."
if (sched.progress or {}).get("handover") else f"Zustand von {host.name} prüfen.")
await _end(db, sched, host, op, ok=False,
message=f"Unerwarteter Fehler während der geplanten Wartung ({type(exc).__name__}). {hint}")
def _since(sched: ScheduledMaintenance) -> timedelta:
return _now() - (sched.phase_since or _now())
@@ -352,17 +457,75 @@ async def _step_job_wait(db, sched, host, op):
else:
done = bool(prog.get("seen_down") or prog.get("seen_running")) or elapsed >= JOB_GRACE
if done:
sched.phase, sched.phase_since = "leave", _now()
sched.phase = "verify" if sched.auto_verify else "leave"
sched.phase_since = _now()
await db.commit()
return
if elapsed >= JOB_TIMEOUT[sched.action]:
await _end(db, sched, host, op, ok=False,
message=f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
if prog.get("handover"):
text = (f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
else:
text = f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. Bitte {host.name} prüfen."
await _end(db, sched, host, op, ok=False, message=text)
return
await db.commit()
async def _step_verify(db, sched, host, op):
"""auto_verify: is the job really over and the host healthy? Read-only, never repeats the action."""
prog = dict(sched.progress or {})
handover = bool(prog.get("handover"))
async def wait_or_give_up(why: str):
if _since(sched) >= VERIFY_TIMEOUT:
tail = (f" {host.name} bleibt im CARP-Wartungsmodus, der Partner trägt den Verkehr." if handover else "")
await _end(db, sched, host, op, ok=False,
message=f"Automatische Prüfung nach „{sched.action}“ nicht bestanden: {why}.{tail} "
f"Bitte {host.name} prüfen und die Wartung von Hand abschließen.")
return
_progress(sched, waiting_for=why)
await db.commit()
try:
running = (await client.get_firmware_running(host)).get("status")
except client.OpnsenseAPIError:
running = None
_progress(sched, seen_down=True)
if running != "ready":
_progress(sched, ready_since=None)
await wait_or_give_up("Firmware-Auftrag läuft noch oder Host nicht erreichbar")
return
if not prog.get("ready_since"):
_progress(sched, ready_since=_now().isoformat())
await db.commit()
return
if _now() - datetime.fromisoformat(prog["ready_since"]) < VERIFY_SETTLE:
await db.commit()
return
boot_now = await _boot_time(host)
rebooted = bool(prog.get("seen_down")) or bool(prog.get("boot_before") and boot_now and boot_now != prog["boot_before"])
if prog.get("reboot_expected") and not rebooted:
await wait_or_give_up("der erwartete Neustart wurde nicht beobachtet")
return
try:
version = (await client.get_firmware_info(host)).get("product_version")
except client.OpnsenseAPIError:
version = None
online = (await get_statuses([host])).get(host.id, {}).get("status") == "online"
if not version or not online:
await wait_or_give_up("Host nicht vollständig erreichbar (Version/Status nicht lesbar)")
return
_progress(sched, version_after=version, boot_after=boot_now, rebooted=rebooted, waiting_for=None)
if handover:
sched.phase, sched.phase_since = "leave", _now() # only now take the traffic back
await db.commit()
return
await _end(db, sched, host, op, ok=True,
message=f"Automatisch geprüft: {host.name} erreichbar, Firmware-Auftrag beendet, Version {version}.",
version_before=prog.get("version_before"), version_after=version, rebooted=rebooted)
async def _step_leave(db, sched, host, op):
me, _partner = await _roles(db, host)
if me.get("status") != "online":
@@ -397,6 +560,8 @@ async def _step_leave_wait(db, sched, host, op):
except client.OpnsenseAPIError:
pass
extra["version_before"] = (sched.progress or {}).get("version_before")
if sched.auto_verify:
extra["rebooted"] = bool((sched.progress or {}).get("rebooted")) # what verify saw before leaving
text = "Automatisch abgeschlossen: Übergabe, Auftrag und Zurückschalten bestätigt."
if extra.get("reboot_required"):
text += " Ein Neustart ist noch erforderlich."
@@ -410,6 +575,7 @@ async def _step_leave_wait(db, sched, host, op):
_STEPS = {
"handover_wait": _step_handover_wait,
"job_wait": _step_job_wait,
"verify": _step_verify,
"leave": _step_leave,
"leave_wait": _step_leave_wait,
}
+3 -1
View File
@@ -18,7 +18,7 @@ from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.x509.oid import ExtensionOID, NameOID
from app.services import overlay_service, proxy_connect
from app.services import proxy_connect
MAX_PEM_BYTES = 32 * 1024
@@ -94,6 +94,8 @@ def verify_argument(verify_tls: bool, tls_ca_pem: str | None):
async def _open_tls(base_url: str, parsed, ctx: ssl.SSLContext, timeout: float):
"""TLS connection to the firewall: directly, or through the overlay hub's proxy for overlay addresses."""
# Imported here so that pinned_context() stays usable without the database layer (gui-gateway container).
from app.services import overlay_service
proxy = overlay_service.proxy_for(base_url)
port = parsed.port or 443
if proxy is None:
+94
View File
@@ -0,0 +1,94 @@
"""Destination check for URLs OPNexus is told to call (notification channels, webhook).
Admins legitimately point these at servers on their own network (a self-hosted ntfy, Gotify or
Matrix), so private ranges are allowed. What is refused is what cannot be a notification target:
anything but http(s), credentials inside the URL, and addresses that only ever mean "this machine"
or "the cloud metadata service" (loopback, link-local such as 169.254.169.254, unspecified,
multicast). Redirects are never followed by the callers, so a public host cannot bounce a request
to such an address. The hostname is resolved at send time and every address is checked (a name that
does not resolve is let through; the request then fails on its own).
"""
import asyncio
import contextlib
import contextvars
import ipaddress
import logging
import socket
from urllib.parse import urlsplit
# httpx logs every request line ("HTTP Request: POST https://api.telegram.org/bot<token>/sendMessage"). For
# targets whose URL is the credential that line must not reach the logs: callers wrap the request in muted().
_muted = contextvars.ContextVar("url_guard_muted", default=False)
class _MuteFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
return not _muted.get()
for _name in ("httpx", "httpcore"):
logging.getLogger(_name).addFilter(_MuteFilter())
@contextlib.contextmanager
def muted():
token = _muted.set(True)
try:
yield
finally:
_muted.reset(token)
def _forbidden(ip: ipaddress._BaseAddress) -> bool:
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
return ip.is_loopback or ip.is_link_local or ip.is_unspecified or ip.is_multicast
def validate_url(url: str, *, max_length: int = 500) -> str:
"""Returns the cleaned URL or raises ValueError with a short reason (shown to the admin)."""
value = (url or "").strip()
if not value or len(value) > max_length or any(c.isspace() for c in value):
raise ValueError("Die Adresse ist leer, zu lang oder enthält Leerzeichen")
try:
parts = urlsplit(value)
host = parts.hostname
parts.port # noqa: B018 - raises ValueError for an invalid port
except ValueError:
raise ValueError("Die Adresse ist ungültig") from None
if parts.scheme not in ("http", "https") or not host:
raise ValueError("Die Adresse muss mit http:// oder https:// beginnen")
if parts.username or parts.password:
raise ValueError("Zugangsdaten gehören nicht in die Adresse")
try:
literal = ipaddress.ip_address(host)
except ValueError:
literal = None
if literal is not None and _forbidden(literal):
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
if literal is None and host.lower() in ("localhost", "localhost.localdomain"):
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
return value
async def ensure_resolved_safe(url: str) -> None:
"""Send-time check against DNS answers pointing at forbidden addresses. Raises ValueError."""
host = urlsplit(url).hostname
if not host:
raise ValueError("Die Adresse ist ungültig")
try:
ipaddress.ip_address(host)
return # a literal was already checked by validate_url
except ValueError:
pass
try:
infos = await asyncio.get_running_loop().getaddrinfo(host, None, type=socket.SOCK_STREAM)
except OSError:
return
for info in infos:
try:
ip = ipaddress.ip_address(info[4][0].split("%")[0])
except ValueError:
continue
if _forbidden(ip):
raise ValueError("Der Name zeigt auf eine nicht erlaubte Adresse (lokal, Link-Local oder Metadaten-Dienst)")
+147
View File
@@ -0,0 +1,147 @@
"""Occurrences of recurring maintenance windows.
Rules (deliberately simple and written down, because a firmware rollout depends on them):
* A window is a wall-clock start time in its own IANA time zone plus a duration. "Sunday 02:00
Europe/Berlin" stays 02:00 local time in summer and in winter.
* Daylight saving: a local start time that does not exist on that day (spring forward, e.g. 02:30 on
the last Sunday of March in Europe/Berlin) moves FORWARD by the gap (03:30). A local time that
exists twice (fall back) means the FIRST of the two (fold=0, still summer time). Both follow from
Python's zoneinfo with fold=0 and are covered by tests.
* The duration is real elapsed time (a 3 h window over the fall-back night still lasts 3 h).
* Exceptions are keyed by the occurrence's local date: "skip" drops it, "move" replaces its start
with an absolute instant (same duration and cut-off).
* New host actions may only START within the first ``start_cutoff_minutes`` of an occurrence
(``startable``); actions already running finish even after the window has closed.
"""
from dataclasses import dataclass
from datetime import date, datetime, timedelta, timezone
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError, available_timezones
from app.models.maintenance_window import MaintenanceWindow
# How far ahead occurrences are searched. A monthly "5th ..." does not exist; the longest real gap
# between two occurrences is a month, so two years is plenty and bounds the work per call.
_SEARCH_DAYS = 2 * 366
MAX_DURATION_MINUTES = 24 * 60
@dataclass(frozen=True)
class Occurrence:
local_date: date
start: datetime # UTC
end: datetime # UTC
cutoff: datetime # UTC: no new host action after this instant
moved: bool = False
skipped: bool = False
def as_dict(self) -> dict:
return {"local_date": self.local_date.isoformat(), "start": self.start.isoformat(), "end": self.end.isoformat(),
"cutoff": self.cutoff.isoformat(), "moved": self.moved, "skipped": self.skipped}
def valid_timezone(name: str) -> bool:
if not name or name not in available_timezones():
return False
try:
ZoneInfo(name)
except (ZoneInfoNotFoundError, ValueError):
return False
return True
def _matches(window: MaintenanceWindow, day: date) -> bool:
if window.recurrence == "weekly":
return day.weekday() in (window.weekdays or [])
if window.recurrence == "monthly":
if window.month_mode == "day":
return day.day == window.month_day
if window.month_mode == "nth_weekday":
if day.weekday() != window.month_weekday:
return False
if window.month_nth == -1:
return (day + timedelta(days=7)).month != day.month
return (day.day - 1) // 7 + 1 == window.month_nth
return False
def local_start(window: MaintenanceWindow, day: date) -> datetime:
"""The UTC instant of the window's start on local date ``day`` (DST rules see module docstring)."""
tz = ZoneInfo(window.timezone)
wall = datetime.combine(day, window.start_time, tzinfo=tz).replace(fold=0)
return wall.astimezone(timezone.utc)
def _occurrence(window: MaintenanceWindow, day: date, start: datetime, *, moved: bool = False, skipped: bool = False) -> Occurrence:
duration = timedelta(minutes=window.duration_minutes)
cutoff = timedelta(minutes=min(window.start_cutoff_minutes, window.duration_minutes))
return Occurrence(local_date=day, start=start, end=start + duration, cutoff=start + cutoff, moved=moved, skipped=skipped)
def occurrences(window: MaintenanceWindow, after: datetime, n: int = 10, *, with_skipped: bool = False) -> list[Occurrence]:
"""The next ``n`` occurrences whose END lies after ``after`` (so one that is open right now is first).
``with_skipped`` lists skipped ones too (flagged, for the UI); the engine never sees them."""
if n <= 0:
return []
after = after.astimezone(timezone.utc)
tz = ZoneInfo(window.timezone)
exceptions = {e.occurrence_date: e for e in (window.exceptions or [])}
duration = timedelta(minutes=window.duration_minutes)
# Start one day early: an occurrence that began yesterday (local) may still be open.
day = (after - duration).astimezone(tz).date() - timedelta(days=1)
found: list[Occurrence] = []
# Moved occurrences can land anywhere, so they are collected separately and merged at the end.
for exc in exceptions.values():
if exc.kind == "move" and exc.moved_start is not None and _matches(window, exc.occurrence_date):
occ = _occurrence(window, exc.occurrence_date, exc.moved_start.astimezone(timezone.utc), moved=True)
if occ.end > after:
found.append(occ)
regular = 0
for _ in range(_SEARCH_DAYS):
if regular >= n:
break
if _matches(window, day):
exc = exceptions.get(day)
if exc is None or (with_skipped and exc.kind == "skip"):
occ = _occurrence(window, day, local_start(window, day), skipped=exc is not None)
if occ.end > after:
found.append(occ)
regular += 1
day += timedelta(days=1)
found.sort(key=lambda o: o.start)
return found[:n]
def current(window: MaintenanceWindow, now: datetime) -> Occurrence | None:
"""The occurrence open at ``now`` (start <= now < end), if any."""
if not window.enabled:
return None
for occ in occurrences(window, now, 3):
if occ.start <= now < occ.end:
return occ
return None
def startable(window: MaintenanceWindow | None, now: datetime) -> Occurrence | None:
"""The occurrence in which a NEW host action may start right now (start <= now < cutoff)."""
if window is None or not window.enabled:
return None
occ = current(window, now)
if occ is not None and occ.start <= now < occ.cutoff:
return occ
return None
def is_occurrence_date(window: MaintenanceWindow, day: date) -> bool:
return _matches(window, day)
def definition(window: MaintenanceWindow) -> dict:
"""What a rollout plan's fingerprint covers: changing any of this re-requires activation."""
return {
"id": str(window.id), "timezone": window.timezone, "recurrence": window.recurrence,
"weekdays": sorted(window.weekdays or []), "month_mode": window.month_mode, "month_nth": window.month_nth,
"month_weekday": window.month_weekday, "month_day": window.month_day,
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
}
+2 -2
View File
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
Versioning scheme: see CHANGELOG.md ("Versionierung").
"""
APP_VERSION = "1.23.8"
APP_CODENAME = "Harrier"
APP_VERSION = "1.28.0"
APP_CODENAME = "Sparrowhawk"
@@ -0,0 +1,55 @@
"""Alarme für Gateways, Ressourcen und Dienste; zusätzliche Benachrichtigungskanäle.
Revision ID: 0020
Revises: 0019
Create Date: 2026-10-21
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision = "0020"
down_revision = "0019"
branch_labels = None
depends_on = None
_COLUMNS = [
("gateway_down_enabled", "boolean NOT NULL DEFAULT true"),
("resource_threshold_enabled", "boolean NOT NULL DEFAULT false"),
("service_down_enabled", "boolean NOT NULL DEFAULT false"),
("gateway_loss_threshold_pct", "integer NOT NULL DEFAULT 20"),
("gateway_delay_threshold_ms", "integer NOT NULL DEFAULT 500"),
("disk_threshold_pct", "integer NOT NULL DEFAULT 90"),
("memory_threshold_pct", "integer NOT NULL DEFAULT 95"),
("swap_threshold_pct", "integer NOT NULL DEFAULT 50"),
("load_per_cpu_threshold", "double precision NOT NULL DEFAULT 2.0"),
("alert_confirm_samples", "integer NOT NULL DEFAULT 3"),
("resource_confirm_samples", "integer NOT NULL DEFAULT 5"),
("service_watchlist", "json"),
]
def upgrade() -> None:
for name, ddl in _COLUMNS:
op.execute(f"ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS {name} {ddl}")
if sa.inspect(op.get_bind()).has_table("notification_channels"):
return
op.create_table(
"notification_channels",
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
sa.Column("name", sa.String(length=100), nullable=False),
sa.Column("channel_type", sa.String(length=16), nullable=False),
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
sa.Column("config", sa.JSON(), nullable=False),
sa.Column("secret_set", sa.Boolean(), server_default="false", nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
)
def downgrade() -> None:
op.execute("DROP TABLE IF EXISTS notification_channels")
for name, _ddl in reversed(_COLUMNS):
op.execute(f"ALTER TABLE notification_settings DROP COLUMN IF EXISTS {name}")
@@ -0,0 +1,60 @@
"""Wiederkehrende Wartungsfenster mit Ausnahmen (1.25).
Revision ID: 0021
Revises: 0020
Create Date: 2026-10-21
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision = "0021"
down_revision = "0020"
branch_labels = None
depends_on = None
def upgrade() -> None:
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
if sa.inspect(op.get_bind()).has_table("maintenance_windows"):
return
op.create_table(
"maintenance_windows",
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
sa.Column("name", sa.String(length=120), nullable=False),
sa.Column("timezone", sa.String(length=64), nullable=False),
sa.Column("recurrence", sa.String(length=16), nullable=False),
sa.Column("weekdays", sa.JSON(), nullable=False),
sa.Column("month_mode", sa.String(length=16), nullable=True),
sa.Column("month_nth", sa.Integer(), nullable=True),
sa.Column("month_weekday", sa.Integer(), nullable=True),
sa.Column("month_day", sa.Integer(), nullable=True),
sa.Column("start_time", sa.Time(), nullable=False),
sa.Column("duration_minutes", sa.Integer(), nullable=False),
sa.Column("start_cutoff_minutes", sa.Integer(), server_default="60", nullable=False),
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
sa.Column("created_by", sa.String(length=150), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
)
op.create_table(
"maintenance_window_exceptions",
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
sa.Column("window_id", postgresql.UUID(as_uuid=True), nullable=False),
sa.Column("occurrence_date", sa.Date(), nullable=False),
sa.Column("kind", sa.String(length=8), nullable=False),
sa.Column("moved_start", sa.DateTime(timezone=True), nullable=True),
sa.Column("reason", sa.String(length=300), nullable=False),
sa.Column("actor", sa.String(length=150), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),
)
op.create_index("ix_maintenance_window_exceptions_window_id", "maintenance_window_exceptions", ["window_id"])
def downgrade() -> None:
op.execute("DROP TABLE IF EXISTS maintenance_window_exceptions")
op.execute("DROP TABLE IF EXISTS maintenance_windows")
@@ -0,0 +1,22 @@
"""Geplante Wartung: automatische Prüfung nach dem Auftrag (auto_verify) und Herkunft (manuell/Rollout).
Revision ID: 0022
Revises: 0021
Create Date: 2026-10-21
"""
from alembic import op
revision = "0022"
down_revision = "0021"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS auto_verify boolean NOT NULL DEFAULT false")
op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS origin varchar(16) NOT NULL DEFAULT 'manual'")
def downgrade() -> None:
op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS origin")
op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS auto_verify")
@@ -0,0 +1,143 @@
"""Firmware-Rollouts in Wellen: Pläne, Läufe, Hosts je Lauf; Benachrichtigung bei Rollout-Problemen (1.25).
Revision ID: 0023
Revises: 0022
Create Date: 2026-10-21
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision = "0023"
down_revision = "0022"
branch_labels = None
depends_on = None
UUID = postgresql.UUID(as_uuid=True)
TS = sa.DateTime(timezone=True)
def upgrade() -> None:
op.execute("ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS rollout_problems_enabled boolean NOT NULL DEFAULT true")
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
if sa.inspect(op.get_bind()).has_table("rollout_plans"):
return
op.create_table(
"rollout_plans",
sa.Column("id", UUID, nullable=False),
sa.Column("name", sa.String(120), nullable=False),
sa.Column("description", sa.Text(), nullable=False),
sa.Column("action", sa.String(16), nullable=False),
sa.Column("window_id", UUID, nullable=True),
sa.Column("auto_start", sa.Boolean(), server_default="false", nullable=False),
sa.Column("gate_checks", sa.JSON(), nullable=False),
sa.Column("compliance_mode", sa.String(8), server_default="warn", nullable=False),
sa.Column("activated_by", sa.String(150), nullable=True),
sa.Column("activated_at", TS, nullable=True),
sa.Column("activated_fingerprint", sa.String(64), nullable=True),
sa.Column("auto_start_error", sa.JSON(), nullable=True),
sa.Column("created_by", sa.String(150), nullable=False),
sa.Column("created_at", TS, nullable=False),
sa.Column("updated_at", TS, nullable=False),
sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="SET NULL"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
)
op.create_index("ix_rollout_plans_window_id", "rollout_plans", ["window_id"])
op.create_table(
"rollout_plan_waves",
sa.Column("id", UUID, nullable=False),
sa.Column("plan_id", UUID, nullable=False),
sa.Column("position", sa.Integer(), nullable=False),
sa.Column("label", sa.String(60), nullable=False),
sa.Column("soak_minutes", sa.Integer(), nullable=False),
sa.Column("carp_handover", sa.Boolean(), server_default="false", nullable=False),
sa.Column("require_approval", sa.Boolean(), server_default="false", nullable=False),
sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("ix_rollout_plan_waves_plan_id", "rollout_plan_waves", ["plan_id"])
op.create_table(
"rollout_wave_groups",
sa.Column("wave_id", UUID, nullable=False),
sa.Column("group_id", UUID, nullable=False),
sa.ForeignKeyConstraint(["wave_id"], ["rollout_plan_waves.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["group_id"], ["device_groups.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("wave_id", "group_id"),
)
op.create_table(
"rollout_runs",
sa.Column("id", UUID, nullable=False),
sa.Column("plan_id", UUID, nullable=True),
sa.Column("plan_name", sa.String(120), nullable=False),
sa.Column("plan_snapshot", sa.JSON(), nullable=False),
sa.Column("action", sa.String(16), nullable=False),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("paused_from", sa.String(20), nullable=True),
sa.Column("current_wave", sa.Integer(), nullable=False),
sa.Column("target_version", sa.String(40), nullable=True),
sa.Column("pause_reason", sa.String(40), nullable=True),
sa.Column("pause_detail", sa.JSON(), nullable=True),
sa.Column("started_by", sa.String(150), nullable=False),
sa.Column("trigger", sa.String(8), nullable=False),
sa.Column("window_occurrence", TS, nullable=True),
sa.Column("revision", sa.Integer(), server_default="0", nullable=False),
sa.Column("progress", sa.JSON(), nullable=False),
sa.Column("step_at", TS, nullable=True),
sa.Column("created_at", TS, nullable=False),
sa.Column("finished_at", TS, nullable=True),
sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="SET NULL"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),
)
op.create_index("ix_rollout_runs_plan_id", "rollout_runs", ["plan_id"])
op.create_index("ix_rollout_runs_status", "rollout_runs", ["status"])
op.create_table(
"rollout_run_waves",
sa.Column("id", UUID, nullable=False),
sa.Column("run_id", UUID, nullable=False),
sa.Column("index", sa.Integer(), nullable=False),
sa.Column("label", sa.String(60), nullable=False),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("baseline", sa.JSON(), nullable=True),
sa.Column("gate_result", sa.JSON(), nullable=True),
sa.Column("soak_until", TS, nullable=True),
sa.Column("gate_attempts", sa.Integer(), server_default="0", nullable=False),
sa.Column("approved_by", sa.String(150), nullable=True),
sa.Column("started_at", TS, nullable=True),
sa.Column("finished_at", TS, nullable=True),
sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("ix_rollout_run_waves_run_id", "rollout_run_waves", ["run_id"])
op.create_table(
"rollout_run_hosts",
sa.Column("id", UUID, nullable=False),
sa.Column("run_id", UUID, nullable=False),
sa.Column("wave_index", sa.Integer(), nullable=False),
sa.Column("host_id", UUID, nullable=True),
sa.Column("host_name", sa.String(120), nullable=False),
sa.Column("cluster_name", sa.String(120), nullable=True),
sa.Column("order", sa.Integer(), nullable=False),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("scheduled_maintenance_id", UUID, nullable=True),
sa.Column("version_before", sa.String(40), nullable=True),
sa.Column("version_after", sa.String(40), nullable=True),
sa.Column("detail", sa.JSON(), nullable=True),
sa.Column("started_at", TS, nullable=True),
sa.Column("finished_at", TS, nullable=True),
sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="SET NULL"),
sa.ForeignKeyConstraint(["scheduled_maintenance_id"], ["scheduled_maintenance.id"], ondelete="SET NULL"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),
)
op.create_index("ix_rollout_run_hosts_run_id", "rollout_run_hosts", ["run_id"])
op.create_index("ix_rollout_run_hosts_host_id", "rollout_run_hosts", ["host_id"])
op.create_index("ix_rollout_run_hosts_scheduled_maintenance_id", "rollout_run_hosts", ["scheduled_maintenance_id"])
def downgrade() -> None:
for table in ("rollout_run_hosts", "rollout_run_waves", "rollout_runs", "rollout_wave_groups", "rollout_plan_waves", "rollout_plans"):
op.execute(f"DROP TABLE IF EXISTS {table}")
op.execute("ALTER TABLE notification_settings DROP COLUMN IF EXISTS rollout_problems_enabled")
@@ -0,0 +1,78 @@
"""GUI-Zugriff über das gui-gateway (1.27): Einstellungen, Einmal-Tickets, Gateway-Sitzungen.
Revision ID: 0024
Revises: 0023
Create Date: 2026-10-21
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision = "0024"
down_revision = "0023"
branch_labels = None
depends_on = None
UUID = postgresql.UUID(as_uuid=True)
TS = sa.DateTime(timezone=True)
def upgrade() -> None:
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
if sa.inspect(op.get_bind()).has_table("gui_access_settings"):
return
op.create_table(
"gui_access_settings",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("enabled", sa.Boolean(), server_default="false", nullable=False),
sa.Column("idle_timeout_minutes", sa.Integer(), server_default="15", nullable=False),
sa.Column("absolute_timeout_minutes", sa.Integer(), server_default="60", nullable=False),
sa.Column("slot_count", sa.Integer(), server_default="4", nullable=False),
sa.Column("updated_by", sa.String(150), nullable=True),
sa.Column("updated_at", TS, nullable=True),
sa.PrimaryKeyConstraint("id"),
)
op.create_table(
"gui_tickets",
sa.Column("id", UUID, nullable=False),
sa.Column("ticket_hash", sa.String(64), nullable=False),
sa.Column("session_hash", sa.String(64), nullable=False),
sa.Column("user_id", UUID, nullable=False),
sa.Column("host_id", UUID, nullable=False),
sa.Column("slot", sa.Integer(), nullable=False),
sa.Column("created_at", TS, nullable=False),
sa.Column("expires_at", TS, nullable=False),
sa.Column("redeemed_at", TS, nullable=True),
sa.ForeignKeyConstraint(["user_id"], ["admin_users.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("ticket_hash"),
)
op.create_index("ix_gui_tickets_session_hash", "gui_tickets", ["session_hash"])
op.create_index("ix_gui_tickets_user_id", "gui_tickets", ["user_id"])
op.create_index("ix_gui_tickets_host_id", "gui_tickets", ["host_id"])
op.create_table(
"gui_sessions",
sa.Column("id", UUID, nullable=False),
sa.Column("session_hash", sa.String(64), nullable=False),
sa.Column("user_id", UUID, nullable=False),
sa.Column("host_id", UUID, nullable=False),
sa.Column("slot", sa.Integer(), nullable=False),
sa.Column("client_ip", sa.String(45), nullable=True),
sa.Column("started_at", TS, nullable=False),
sa.Column("last_checked_at", TS, nullable=False),
sa.Column("ended_at", TS, nullable=True),
sa.Column("end_reason", sa.String(32), nullable=True),
sa.ForeignKeyConstraint(["user_id"], ["admin_users.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("ix_gui_sessions_session_hash", "gui_sessions", ["session_hash"])
op.create_index("ix_gui_sessions_user_id", "gui_sessions", ["user_id"])
op.create_index("ix_gui_sessions_host_id", "gui_sessions", ["host_id"])
def downgrade() -> None:
op.drop_table("gui_sessions")
op.drop_table("gui_tickets")
op.drop_table("gui_access_settings")
+1
View File
@@ -13,3 +13,4 @@ respx==0.23.1
cryptography==50.0.2
alembic==1.20.0
PyJWT==2.10.1
tzdata==2025.2
File diff suppressed because it is too large. Load diff
+39
View File
@@ -0,0 +1,39 @@
"""Mutation check helper (run manually, not collected by pytest).
python tests/mutation_check.py tests/mutations_gui.json
Each mutation: {"file": ..., "old": ..., "new": ..., "tests": "pytest selection"}.
The file is patched, the selected tests must FAIL, then the file is restored.
"""
import json
import subprocess
import sys
from pathlib import Path
def main(spec_path: str) -> int:
mutations = json.loads(Path(spec_path).read_text())
survivors = []
for i, m in enumerate(mutations, 1):
path = Path(m["file"])
original = path.read_text()
if original.count(m["old"]) != 1:
print(f"[{i}] SKIP (anchor not unique/found): {m['file']}: {m['old'][:60]!r}")
survivors.append(m)
continue
try:
path.write_text(original.replace(m["old"], m["new"]))
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", *m["tests"].split()],
capture_output=True, text=True)
finally:
path.write_text(original)
killed = result.returncode != 0
print(f"[{i}] {'KILLED ' if killed else 'SURVIVED'} {m.get('name', m['old'][:50])}")
if not killed:
survivors.append(m)
print(f"{len(mutations) - len(survivors)}/{len(mutations)} mutations killed")
return 1 if survivors else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1]))
+226
View File
@@ -0,0 +1,226 @@
[
{
"name": "GW browser Cookie forwarded",
"file": "app/gui_gateway.py",
"old": " out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS}",
"new": " out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS or k == \"cookie\"}",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW Referer/Authorization forwarded",
"file": "app/gui_gateway.py",
"old": "\"accept\", \"accept-language\", \"content-type\", \"user-agent\", \"x-requested-with\", \"x-csrftoken\", \"range\",",
"new": "\"accept\", \"accept-language\", \"content-type\", \"user-agent\", \"x-requested-with\", \"x-csrftoken\", \"range\", \"referer\", \"authorization\",",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW Set-Cookie passed to browser",
"file": "app/gui_gateway.py",
"old": "\"content-range\", \"accept-ranges\", \"content-security-policy\",",
"new": "\"content-range\", \"accept-ranges\", \"content-security-policy\", \"set-cookie\",",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW jar never forgets deleted cookies",
"file": "app/gui_gateway.py",
"old": " if expired:\n gs.jar.pop(name, None)",
"new": " if False:\n gs.jar.pop(name, None)",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW foreign Location kept",
"file": "app/gui_gateway.py",
"old": " if _origin_of(location) != _origin_of(upstream_origin):\n return None",
"new": " pass",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW backslash Location kept",
"file": "app/gui_gateway.py",
"old": " if not location or \"\\\\\" in location",
"new": " if not location",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW Sec-Fetch-Site lax",
"file": "app/gui_gateway.py",
"old": " if site not in (\"same-origin\", \"none\"):",
"new": " if site in (\"cross-site\",):",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW redeem Sec-Fetch-Site unchecked",
"file": "app/gui_gateway.py",
"old": " if method != \"GET\" or site not in (\"same-site\", \"same-origin\", \"none\"):",
"new": " if method != \"GET\":",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW redeem without OPNexus cookie",
"file": "app/gui_gateway.py",
"old": " if not ticket or len(ticket) > 128 or not browser_session:",
"new": " if not ticket or len(ticket) > 128:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW Origin unchecked for writes",
"file": "app/gui_gateway.py",
"old": " if _origin_of(origin) not in own_origins:",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW absolute-form accepted",
"file": "app/gui_gateway.py",
"old": " if not path.startswith(\"/\") or path.startswith(\"//\") or not raw_path.startswith(b\"/\") \\",
"new": " if False \\",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW CONNECT accepted",
"file": "app/gui_gateway.py",
"old": " if method in (\"CONNECT\", \"TRACE\"):",
"new": " if method in (\"TRACE\",):",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW Upgrade accepted",
"file": "app/gui_gateway.py",
"old": " if \"upgrade\" in headers or \"upgrade\" in headers.get(\"connection\", \"\").lower():",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW service worker accepted",
"file": "app/gui_gateway.py",
"old": " if headers.get(\"service-worker\", \"\").lower() == \"script\":",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW slot header optional",
"file": "app/gui_gateway.py",
"old": " if not own_origins:\n return await self._respond(send, 400, _FORBIDDEN)",
"new": " if False:\n return await self._respond(send, 400, _FORBIDDEN)",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW foreign slot cookie accepted",
"file": "app/gui_gateway.py",
"old": " if gs is None or gs.slot != slot:",
"new": " if gs is None:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW login rate limit off",
"file": "app/gui_gateway.py",
"old": " if login and self.limiter.blocked(gs.upstream_origin):",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW login failures not counted",
"file": "app/gui_gateway.py",
"old": " if login and not (300 <= response.status_code < 400):",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW no periodic re-check",
"file": "app/gui_gateway.py",
"old": " if self.clock() >= gs.next_check:\n state = await self._recheck(key, gs)",
"new": " if False:\n state = await self._recheck(key, gs)",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW fail-open when backend down",
"file": "app/gui_gateway.py",
"old": " if state is None:\n return await self._respond(send, 503, _UNAVAILABLE)",
"new": "",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW idle timeout off",
"file": "app/gui_gateway.py",
"old": " if self.clock() - gs.last_activity >= gs.idle_timeout:",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW absolute timeout off",
"file": "app/gui_gateway.py",
"old": " if self.wall_clock() >= gs.absolute_deadline:",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW no CSP",
"file": "app/gui_gateway.py",
"old": " (b\"content-security-policy\", _GATEWAY_CSP.encode()),",
"new": "",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW no X-Frame-Options",
"file": "app/gui_gateway.py",
"old": " (b\"x-frame-options\", b\"DENY\"),",
"new": "",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW no no-store",
"file": "app/gui_gateway.py",
"old": " (b\"cache-control\", b\"no-store\"),",
"new": "",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW streaming body limit off",
"file": "app/gui_gateway.py",
"old": " if size > BODY_LIMIT:\n return None",
"new": "",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW query logged",
"file": "app/gui_gateway.py",
"old": "scope.get(\"path\", \"?\")[:200], status,",
"new": "scope.get(\"path\", \"?\")[:200] + \"?\" + scope.get(\"query_string\", b\"\").decode(), status,",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW httpx URL logging on",
"file": "app/gui_gateway.py",
"old": " logging.getLogger(_noisy).setLevel(logging.WARNING)",
"new": " pass",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW weak gateway cookie",
"file": "app/gui_gateway.py",
"old": "={value}; Path=/; Secure; HttpOnly; SameSite=Strict\"",
"new": "={value}; Path=/\"",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW no Clear-Site-Data",
"file": "app/gui_gateway.py",
"old": " (b\"set-cookie\", cookie.encode()), (b\"clear-site-data\", b'\"cache\", \"storage\"'),",
"new": " (b\"set-cookie\", cookie.encode()),",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW pin not used",
"file": "app/gui_gateway.py",
"old": " verify = pinned_context(gs.ca_pem)",
"new": " verify = False",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "GW access log on",
"file": "app/gui_gateway.py",
"old": "http=\"h11\", ws=\"none\", access_log=False,",
"new": "http=\"h11\", ws=\"none\", access_log=True,",
"tests": "tests/test_gui_gateway.py"
}
]
+170
View File
@@ -0,0 +1,170 @@
[
{
"name": "AP1 sec-fetch: cross-site not blocked",
"file": "app/security_middleware.py",
"old": " if site == \"cross-site\":\n return True",
"new": " if site == \"cross-site\":\n return False",
"tests": "tests/test_session_hardening.py"
},
{
"name": "AP1 sec-fetch: same-site always allowed",
"file": "app/security_middleware.py",
"old": " return not _explicit_frontend_origin(request.headers.get(\"origin\", \"\"))",
"new": " return False",
"tests": "tests/test_session_hardening.py"
},
{
"name": "AP1 sec-fetch: check not wired",
"file": "app/security_middleware.py",
"old": " if sec_fetch_blocked(request):",
"new": " if False:",
"tests": "tests/test_session_hardening.py"
},
{
"name": "AP1 cookie: plain name falls back even with __Host-",
"file": "app/routers/auth.py",
"old": " return request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME, \"\")",
"new": " return request.cookies.get(COOKIE_NAME) or request.cookies.get(SECURE_COOKIE_NAME, \"\")",
"tests": "tests/test_session_hardening.py"
},
{
"name": "AP1 cookie: no __Host- prefix",
"file": "app/routers/auth.py",
"old": " return SECURE_COOKIE_NAME if settings.cookie_secure else COOKIE_NAME",
"new": " return COOKIE_NAME",
"tests": "tests/test_session_hardening.py"
},
{
"name": "AP1 startup: conflict check removed",
"file": "app/main.py",
"old": " if conflicts:\n",
"new": " if False:\n",
"tests": "tests/test_session_hardening.py"
},
{
"name": "AP2 ticket expiry ignored",
"file": "app/services/gui_ticket_service.py",
"old": " elif ticket.expires_at <= now:",
"new": " elif False:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 replay allowed",
"file": "app/services/gui_ticket_service.py",
"old": " if replayed:\n reason = \"replayed\"",
"new": " if False:\n reason = \"replayed\"",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 session binding ignored",
"file": "app/services/gui_ticket_service.py",
"old": " elif not secrets.compare_digest(ticket.session_hash, session_hash):",
"new": " elif False:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 slot binding ignored",
"file": "app/services/gui_ticket_service.py",
"old": " elif ticket.slot != slot:",
"new": " elif False:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 revalidate: role not rechecked",
"file": "app/services/gui_ticket_service.py",
"old": " if user is None or user.role != \"admin\":",
"new": " if user is None:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 revalidate: session not rechecked",
"file": "app/services/gui_ticket_service.py",
"old": " if session is None or session.expires_at <= _now() or session.user_id != user_id:",
"new": " if False:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 revalidate: switch not rechecked",
"file": "app/services/gui_ticket_service.py",
"old": " if not row.enabled:\n return \"disabled\", user, None",
"new": "",
"tests": "tests/test_gui_access.py::test_revalidate_fails_when_switched_off"
},
{
"name": "AP2 absolute limit ignored",
"file": "app/services/gui_ticket_service.py",
"old": " if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):",
"new": " if False:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 host tls requirement dropped",
"file": "app/services/gui_ticket_service.py",
"old": " if not host.verify_tls:\n return \"tls\"",
"new": "",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 slot limit ignored",
"file": "app/services/gui_ticket_service.py",
"old": " if not free:\n raise",
"new": " if False:\n raise",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 gateway token not compared",
"file": "app/routers/gui_access.py",
"old": " if not hmac.compare_digest(",
"new": " if False and not hmac.compare_digest(",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 switch not checked for tickets",
"file": "app/routers/gui_access.py",
"old": " if not row.enabled:\n raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_ACCESS_DISABLED)",
"new": "",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 legacy cookie accepted for tickets",
"file": "app/routers/gui_access.py",
"old": " session_cookie = request.cookies.get(SECURE_COOKIE_NAME, \"\")",
"new": " session_cookie = request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(\"opnexus-session\", \"\")",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 enabling without https allowed",
"file": "app/routers/gui_access.py",
"old": " if payload.enabled and svc.unavailable_reasons():",
"new": " if False:",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP2 redeem leaks api key",
"file": "app/services/gui_ticket_service.py",
"old": " \"upstream_origin\": firewall_origin(host.base_url),",
"new": " \"upstream_origin\": firewall_origin(host.base_url), \"k\": host.api_key,",
"tests": "tests/test_gui_access.py"
},
{
"name": "AP7 revalidation interval back to 30 s (switch-off took 31 s live)",
"file": "app/services/gui_ticket_service.py",
"old": "REVALIDATE_SECONDS = 20",
"new": "REVALIDATE_SECONDS = 30",
"tests": "tests/test_gui_access.py"
},
{
"name": "Review: multipart login not counted",
"file": "app/gui_gateway.py",
"old": " if content_type.lower().startswith(\"multipart/form-data\"):",
"new": " if False:",
"tests": "tests/test_gui_gateway.py"
},
{
"name": "Review: decoded path forwarded upstream",
"file": "app/gui_gateway.py",
"old": "url = gs.upstream_origin + raw_path +",
"new": "url = gs.upstream_origin + scope[\"path\"] +",
"tests": "tests/test_gui_gateway.py"
}
]
+486
View File
@@ -0,0 +1,486 @@
"""Linking hand-made objects on a second firewall to the central object that already exists
(services/adopt_link_service.py), instead of creating a duplicate."""
import json
import httpx
import pytest
from sqlalchemy import func, select
from app.database import SessionLocal
from app.models.alias import FirewallAlias
from app.models.audit import AuditEvent
from app.models.port_forward import PortForwardRule
from tests.helpers import deploy_approved
from tests.test_api_tokens import _login
from tests.test_object_adopt import AUTOMATIC_NAT, NAT_KINDS, SYSTEM_ALIASES, alias_row, o2o_row, pf_row, snat_row
from tests.test_object_adopt import _mode
from tests.test_rule_adopt import _row as rule_row
A = "https://fw-link-a.test"
B = "https://fw-link-b.test"
async def _host(client, respx_mock, base, name):
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
r = await client.post("/api/hosts", json={"name": name, "base_url": base, "api_key": "k", "api_secret": "s"})
assert r.status_code == 201, r.text
return r.json()["id"]
def _search(respx_mock, base, path, rows):
return respx_mock.get(f"{base}/api/firewall/{path}").mock(return_value=httpx.Response(200, json={"rows": rows, "total": len(rows)}))
def _live_writes(respx_mock):
return [c for c in respx_mock.calls if c.request.method != "GET" and "/firewall/" in str(c.request.url)]
@pytest.fixture
async def two(client, respx_mock):
a = await _host(client, respx_mock, A, "fw-link-a")
b = await _host(client, respx_mock, B, "fw-link-b")
_mode(respx_mock) # outbound NAT mode is read per base url below
for base in (A, B):
respx_mock.get(f"{base}/api/firewall/source_nat/get").mock(return_value=httpx.Response(
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == "hybrid" else 0} for m in
("automatic", "hybrid", "advanced", "disabled")}}}}))
return a, b
async def _put_vars(client, host_id, values):
r = await client.put(f"/api/hosts/{host_id}/variables", json={"variables": values})
assert r.status_code == 200, r.text
# ── aliases ─────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_alias_is_linked_not_duplicated_and_rolls_out_to_both_hosts(client, respx_mock, two):
a, b = two
live_a = alias_row("web_servers", "host", "10.0.0.1\n10.0.0.2", uuid="uuid-a")
live_b = alias_row("web_servers", "host", "10.0.0.1\n10.0.0.2", uuid="uuid-b")
_search(respx_mock, A, "alias/searchItem", SYSTEM_ALIASES + [live_a])
_search(respx_mock, B, "alias/searchItem", SYSTEM_ALIASES + [live_b])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
# host B: the candidate carries the exact twin (name_exists no longer decides alone)
cand = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()
row = next(r for r in cand if r["uuid"] == "uuid-b")
assert row["linkable"] is True and row["reasons"] == ["name_exists"]
assert [(m["id"], m["name"], m["targets"], m["diff"]) for m in row["matches"]] == [(first["id"], "web_servers", ["fw-link-a"], [])]
assert row["similar"] == [] and row["bulk_action"] == "link"
# adopting it as an own object is still refused (the name is taken)
assert (await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b"})).status_code == 409
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": first["id"]})
assert resp.status_code == 201, resp.text
linked = resp.json()
assert linked["id"] == first["id"] and sorted(linked["target_host_ids"]) == sorted([a, b])
assert {d["host_id"]: d["status"] for d in linked["deployments"]} == {a: "success", b: "success"}
assert not _live_writes(respx_mock) # nothing is written to a firewall
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "alias"))).all()
assert any(e.message == "Mit vorhandenem Objekt verknüpft" and e.details["remote_uuid"] == "uuid-b" for e in events)
revisions = (await client.get(f"/api/aliases/{first['id']}/revisions")).json()
assert any("fw-link-b verknüpft" in r["note"] for r in revisions)
# preview: unchanged on both hosts; it is no longer offered on B
preview = (await client.get(f"/api/aliases/{first['id']}/preview")).json()
assert sorted(h["action"] for h in preview["hosts"]) == ["unchanged", "unchanged"]
assert [r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json() if r["reasons"] != ["system"]] == []
# one edit rolls out to both remote objects, never as a new one
edit = {"name": "web_servers", "type": "host", "content": "10.0.0.1\n10.0.0.3", "description": "web_servers desc", "enabled": True,
"target_host_ids": [a, b], "target_group_ids": [], "expected_updated_at": linked["updated_at"]}
assert (await client.put(f"/api/aliases/{first['id']}", json=edit)).status_code == 200
add = respx_mock.post(url__regex=r".*/alias/addItem").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
set_a = respx_mock.post(f"{A}/api/firewall/alias/setItem/uuid-a").mock(return_value=httpx.Response(200, json={"result": "saved"}))
set_b = respx_mock.post(f"{B}/api/firewall/alias/setItem/uuid-b").mock(return_value=httpx.Response(200, json={"result": "saved"}))
respx_mock.post(url__regex=r".*/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
assert (await deploy_approved(client, f"/api/aliases/{first['id']}/deploy")).status_code == 200
assert set_a.called and set_b.called and not add.called
@pytest.mark.asyncio
async def test_link_is_idempotent(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("dup_me", uuid="uuid-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("dup_me", uuid="uuid-b")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
body = {"host_id": b, "uuid": "uuid-b", "link_to": first["id"]}
assert (await client.post("/api/aliases/adopt", json=body)).status_code == 201
again = await client.post("/api/aliases/adopt", json=body)
assert again.status_code == 201, again.text
assert len(again.json()["deployments"]) == 2 and len(again.json()["target_host_ids"]) == 2
# linking the same live object to ANOTHER central object is refused
other = (await client.post("/api/aliases", json={"name": "other", "type": "host", "content": "1.1.1.1"})).json()
assert (await client.post("/api/aliases/adopt", json={**body, "link_to": other["id"]})).status_code == 409
@pytest.mark.asyncio
async def test_different_content_shows_a_diff_and_is_never_linked(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("ntp", "host", "10.0.0.1", uuid="uuid-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("ntp", "host", "10.0.0.1\n10.0.0.9", uuid="uuid-b", description="other text")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
row = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]
assert row["linkable"] is False and row["matches"] == [] and row["bulk_action"] == "skip"
sim = row["similar"][0]
assert sim["id"] == first["id"]
assert {d["field"]: (d["central"], d["live"]) for d in sim["diff"]} == {
"content": ("10.0.0.1", "10.0.0.1\n10.0.0.9"), "description": ("ntp desc", "other text")}
# forcing the link by id is refused too: the live state is compared again on the server
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": first["id"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
assert "content" in resp.json()["detail"]["params"]["fields"]
# nothing was linked
alias = next(x for x in (await client.get("/api/aliases")).json() if x["id"] == first["id"])
assert alias["target_host_ids"] == [a] and len(alias["deployments"]) == 1
@pytest.mark.asyncio
async def test_browser_state_is_not_trusted_live_state_decides(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("srv", "host", "10.0.0.1", uuid="uuid-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("srv", "host", "10.0.0.1", uuid="uuid-b")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
assert (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]["linkable"] is True
# between listing and clicking somebody changes the alias on host B by hand
_search(respx_mock, B, "alias/searchItem", [alias_row("srv", "host", "10.0.0.1\n10.0.0.77", uuid="uuid-b")])
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": first["id"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
@pytest.mark.asyncio
async def test_variables_are_resolved_per_host(client, respx_mock, two):
a, b = two
await _put_vars(client, a, {"LAN_NET": "192.168.1.0/24"})
await _put_vars(client, b, {"LAN_NET": "192.168.2.0/24"})
central = (await client.post("/api/aliases", json={"name": "lan_net", "type": "network", "content": "{{LAN_NET}}", "description": "lan_net desc",
"target_host_ids": [a]})).json()
_search(respx_mock, B, "alias/searchItem", [alias_row("lan_net", "network", "192.168.2.0/24", uuid="uuid-b")])
row = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]
assert row["linkable"] is True and row["matches"][0]["id"] == central["id"]
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": central["id"]})
assert resp.status_code == 201, resp.text
dep_b = next(d for d in resp.json()["deployments"] if d["host_id"] == b)
assert dep_b["status"] == "success"
async with SessionLocal() as db:
from app.models.alias_deployment import AliasDeployment
deps = {str(d.host_id): d for d in (await db.scalars(select(AliasDeployment))).all()}
assert deps[b].opnsense_alias_uuid == "uuid-b" and deps[b].last_payload["content"] == "192.168.2.0/24"
preview = (await client.get(f"/api/aliases/{central['id']}/preview")).json()
assert {h["host_name"]: h["action"] for h in preview["hosts"]}["fw-link-b"] == "unchanged" # resolved payload == live
# a host with a DIFFERENT value for the variable does not match; the diff shows the resolved value
_search(respx_mock, B, "alias/searchItem", [alias_row("lan_net", "network", "192.168.9.0/24", uuid="uuid-x")])
# (host B already has the object, so ask for it through a third host)
c = await _host(client, respx_mock, "https://fw-link-c.test", "fw-link-c")
await _put_vars(client, c, {"LAN_NET": "192.168.3.0/24"})
_search(respx_mock, "https://fw-link-c.test", "alias/searchItem", [alias_row("lan_net", "network", "192.168.9.0/24", uuid="uuid-c")])
row = (await client.get(f"/api/aliases/unmanaged?host_id={c}")).json()[0]
assert row["matches"] == [] and row["similar"][0]["diff"] == [{"field": "content", "central": "192.168.3.0/24", "live": "192.168.9.0/24"}]
@pytest.mark.asyncio
async def test_missing_variable_on_the_host_never_matches(client, respx_mock, two):
a, b = two
await _put_vars(client, a, {"LAN_NET": "192.168.1.0/24"})
await client.post("/api/aliases", json={"name": "lan_net", "type": "network", "content": "{{LAN_NET}}", "description": "lan_net desc", "target_host_ids": [a]})
_search(respx_mock, B, "alias/searchItem", [alias_row("lan_net", "network", "192.168.2.0/24", uuid="uuid-b")])
row = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]
assert row["matches"] == [] and row["similar"][0]["diff"][0]["field"] == "variables"
@pytest.mark.asyncio
async def test_nested_alias_references_are_part_of_the_comparison(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [
alias_row("inner", "network", "10.1.0.0/16", uuid="in-a"),
alias_row("outer", "networkgroup", "inner\n10.2.0.0/16", uuid="out-a")])
result = (await client.post("/api/aliases/adopt-all", json={"host_id": a})).json()
assert len(result["adopted"]) == 2
_search(respx_mock, B, "alias/searchItem", [
alias_row("inner", "network", "10.1.0.0/16", uuid="in-b"),
alias_row("outer", "networkgroup", "inner\n10.2.0.0/16", uuid="out-b")])
rows = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()}
assert rows["inner"]["linkable"] and rows["outer"]["linkable"]
# a different reference is a diff, not a match
_search(respx_mock, B, "alias/searchItem", [
alias_row("inner", "network", "10.1.0.0/16", uuid="in-b"),
alias_row("outer", "networkgroup", "other_alias\n10.2.0.0/16", uuid="out-b"),
alias_row("other_alias", "network", "10.5.0.0/16", uuid="oth-b")])
rows = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()}
assert rows["outer"]["matches"] == [] and rows["outer"]["similar"][0]["diff"][0]["field"] == "content"
# "adopt all" with link: both linked, nothing created
_search(respx_mock, B, "alias/searchItem", [
alias_row("inner", "network", "10.1.0.0/16", uuid="in-b"),
alias_row("outer", "networkgroup", "inner\n10.2.0.0/16", uuid="out-b")])
done = (await client.post("/api/aliases/adopt-all", json={"host_id": b, "link_existing": True})).json()
assert sorted((x["title"], x["linked"]) for x in done["adopted"]) == [("inner", True), ("outer", True)] and done["skipped"] == []
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 2
@pytest.mark.asyncio
async def test_adopt_all_without_link_flag_keeps_old_behaviour(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("x1", uuid="u-a")])
await client.post("/api/aliases/adopt-all", json={"host_id": a})
_search(respx_mock, B, "alias/searchItem", [alias_row("x1", uuid="u-b")])
done = (await client.post("/api/aliases/adopt-all", json={"host_id": b})).json()
assert done["adopted"] == [] and done["skipped"][0]["reasons"] == ["name_exists"]
# ── NAT + rules, every kind ─────────────────────────────────────────────
KINDS = {
"port-forward": ("d_nat/searchRule", pf_row, {"descr": "Web"}, {"destination.port": "9999"}, "local_port"),
"one-to-one": ("one_to_one/searchRule", o2o_row, {"description": "Mail 1:1"}, {"external": "203.0.113.77"}, "external"),
"outbound-nat": ("source_nat/searchRule", snat_row, {"description": "LAN out"}, {"source_net": "192.168.9.0/24"}, "source_net"),
}
@pytest.mark.asyncio
@pytest.mark.parametrize("prefix", list(KINDS))
async def test_nat_kinds_link_diff_and_bulk(client, respx_mock, two, prefix):
a, b = two
path, factory, _, change, _ = KINDS[prefix]
_search(respx_mock, A, path, [factory(uuid="a-1"), *AUTOMATIC_NAT])
first = (await client.post(f"/api/{prefix}/adopt", json={"host_id": a, "uuid": "a-1"})).json()
# different on B -> hint + diff, no link possible
_search(respx_mock, B, path, [factory(uuid="b-1", **change)])
row = (await client.get(f"/api/{prefix}/unmanaged?host_id={b}")).json()[0]
assert row["adoptable"] is True and row["linkable"] is False and row["matches"] == []
assert row["similar"][0]["id"] == first["id"] and row["similar"][0]["diff"]
resp = await client.post(f"/api/{prefix}/adopt", json={"host_id": b, "uuid": "b-1", "link_to": first["id"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
adopt_all = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": b, "link_existing": True})).json()
assert adopt_all["adopted"][0]["linked"] is False # a different object is adopted as its own, not linked
# fresh start for the exact case on a third host
c = await _host(client, respx_mock, "https://fw-link-c.test", "fw-link-c")
respx_mock.get("https://fw-link-c.test/api/firewall/source_nat/get").mock(return_value=httpx.Response(
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == "hybrid" else 0} for m in
("automatic", "hybrid", "advanced", "disabled")}}}}))
_search(respx_mock, "https://fw-link-c.test", path, [factory(uuid="c-1")])
row = (await client.get(f"/api/{prefix}/unmanaged?host_id={c}")).json()[0]
ids = [m["id"] for m in row["matches"]]
assert ids == [first["id"]] and row["bulk_action"] == "link" and row["linkable"]
resp = await client.post(f"/api/{prefix}/adopt", json={"host_id": c, "uuid": "c-1", "link_to": first["id"]})
assert resp.status_code == 201, resp.text
assert resp.json()["id"] == first["id"] and c in resp.json()["target_host_ids"]
preview = (await client.get(f"/api/{prefix}/{first['id']}/preview")).json()
assert {h["host_name"]: h["action"] for h in preview["hosts"]}["fw-link-c"] == "unchanged"
assert not _live_writes(respx_mock)
listed = (await client.get(f"/api/{prefix}")).json()
assert len(listed) == 2 # first (A + C) and the different one from B
@pytest.mark.asyncio
async def test_port_forward_link_updates_remote_object_without_duplicate(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "d_nat/searchRule", [pf_row(uuid="pf-a")])
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="pf-b")])
first = (await client.post("/api/port-forward/adopt", json={"host_id": a, "uuid": "pf-a"})).json()
linked = (await client.post("/api/port-forward/adopt", json={"host_id": b, "uuid": "pf-b", "link_to": first["id"]})).json()
edit = {k: linked[k] for k in ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port",
"destination_net", "destination_port", "target", "local_port", "natreflection", "enabled")}
for base in (A, B): # alias lookups and the interface consistency check of a multi-host rollout
_search(respx_mock, base, "alias/searchItem", [])
for base in (A, B): # the interface consistency check of a multi-host rollout
respx_mock.get(f"{base}/api/firewall/filter/getRule").mock(return_value=httpx.Response(
200, json={"rule": {"interface": {"wan": {"value": "WAN", "selected": 1}}}}))
edit.update(target="192.168.1.99", target_host_ids=[a, b], target_group_ids=[], expected_updated_at=linked["updated_at"])
assert (await client.put(f"/api/port-forward/{first['id']}", json=edit)).status_code == 200
add = respx_mock.post(url__regex=r".*/d_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
sa = respx_mock.post(f"{A}/api/firewall/d_nat/setRule/pf-a").mock(return_value=httpx.Response(200, json={"result": "saved"}))
sb = respx_mock.post(f"{B}/api/firewall/d_nat/setRule/pf-b").mock(return_value=httpx.Response(200, json={"result": "saved"}))
respx_mock.post(url__regex=r".*/d_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
assert (await deploy_approved(client, f"/api/port-forward/{first['id']}/deploy")).status_code == 200
assert sa.called and sb.called and not add.called
assert json.loads(sb.calls.last.request.content)["rule"]["target"] == "192.168.1.99"
@pytest.mark.asyncio
async def test_rule_link_diff_and_errors(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "filter/searchRule", [rule_row("r-a")])
_search(respx_mock, B, "filter/searchRule", [rule_row("r-b")])
first = (await client.post("/api/rules/adopt", json={"host_id": a, "rule_uuid": "r-a"})).json()
row = (await client.get(f"/api/rules/unmanaged?host_id={b}")).json()[0]
assert row["linkable"] is True and [m["id"] for m in row["matches"]] == [first["id"]]
resp = await client.post("/api/rules/adopt", json={"host_id": b, "rule_uuid": "r-b", "link_to": first["id"]})
assert resp.status_code == 201, resp.text
assert resp.json()["id"] == first["id"] and sorted(resp.json()["target_host_ids"]) == sorted([a, b])
again = await client.post("/api/rules/adopt", json={"host_id": b, "rule_uuid": "r-b", "link_to": first["id"]})
assert again.status_code == 201 and len(again.json()["deployments"]) == 2
preview = (await client.get(f"/api/rules/{first['id']}/preview")).json()
assert sorted(h["action"] for h in preview["hosts"]) == ["unchanged", "unchanged"]
assert not _live_writes(respx_mock)
# different port -> diff only
c = await _host(client, respx_mock, "https://fw-link-c.test", "fw-link-c")
_search(respx_mock, "https://fw-link-c.test", "filter/searchRule", [rule_row("r-c", destination_port="2222")])
row = (await client.get(f"/api/rules/unmanaged?host_id={c}")).json()[0]
assert row["linkable"] is False and row["similar"][0]["diff"] == [{"field": "destination_port", "central": "22", "live": "2222"}]
refused = await client.post("/api/rules/adopt", json={"host_id": c, "rule_uuid": "r-c", "link_to": first["id"]})
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
# ── error cases ─────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_deleted_target_and_host_already_has_object_and_busy(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("gone", uuid="u-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("gone", uuid="u-b")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
# busy: an unconfirmed deployment on the central object
respx_mock.post(f"{A}/api/firewall/alias/setItem/u-a").mock(return_value=httpx.Response(200, json={"result": "saved"}))
respx_mock.post(url__regex=r".*/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
edit = {"name": "gone", "type": "host", "content": "10.9.9.1\n10.9.9.2", "description": "gone desc", "enabled": False,
"target_host_ids": [a], "target_group_ids": [], "expected_updated_at": first["updated_at"]}
assert (await client.put(f"/api/aliases/{first['id']}", json=edit)).status_code == 200
assert (await deploy_approved(client, f"/api/aliases/{first['id']}/deploy")).status_code == 200 # pending now
busy = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
assert busy.status_code == 409 # content differs now AND it is pending; either way nothing is linked
await client.post(f"/api/aliases/{first['id']}/confirm")
# deleted in the meantime
respx_mock.post(url__regex=r".*/alias/delItem/.*").mock(return_value=httpx.Response(200, json={"result": "deleted"}))
assert (await client.delete(f"/api/aliases/{first['id']}")).status_code == 204
gone = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
assert gone.status_code == 404 and gone.json()["detail"]["code"] == "ADOPT_LINK_TARGET_NOT_FOUND"
@pytest.mark.asyncio
async def test_pending_deployment_blocks_the_link(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("busy1", uuid="u-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("busy1", uuid="u-b")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
async with SessionLocal() as db:
from app.models.alias_deployment import AliasDeployment
dep = (await db.scalars(select(AliasDeployment))).one()
dep.status = "pending"
await db.commit()
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_BUSY"
@pytest.mark.asyncio
async def test_host_that_already_has_the_object_is_not_a_candidate_and_cannot_link(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("twice", uuid="u-a")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
# a second, differently-UUID'd copy on host A itself: A already has the object
_search(respx_mock, A, "alias/searchItem", [alias_row("twice", uuid="u-a"), alias_row("twice", uuid="u-a2")])
rows = (await client.get(f"/api/aliases/unmanaged?host_id={a}")).json()
assert [r["uuid"] for r in rows] == ["u-a2"] and rows[0]["matches"] == [] and rows[0]["linkable"] is False
resp = await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a2", "link_to": first["id"]})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_HOST_HAS_OBJECT"
@pytest.mark.asyncio
async def test_bulk_never_guesses_between_two_twins_and_links_each_object_once(client, respx_mock, two):
a, b = two
# two identical central objects exist from earlier separate adoptions
for _ in range(2):
await client.post("/api/port-forward", json={
"description": "Web", "interface": "wan", "ipprotocol": "inet", "protocol": "tcp", "source_net": "any",
"destination_net": "wanip", "destination_port": "8443", "target": "192.168.1.50", "local_port": "443",
"natreflection": "", "enabled": True, "target_host_ids": [a], "target_group_ids": []})
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="pf-b")])
row = (await client.get(f"/api/port-forward/unmanaged?host_id={b}")).json()[0]
assert len(row["matches"]) == 2 and row["bulk_action"] == "new" # ask the user, do not guess
done = (await client.post("/api/port-forward/adopt-all", json={"host_id": b, "link_existing": True})).json()
assert [x["linked"] for x in done["adopted"]] == [False]
@pytest.mark.asyncio
async def test_bulk_links_each_central_object_only_once(client, respx_mock, two):
a, b = two
await client.post("/api/port-forward", json={
"description": "Web", "interface": "wan", "ipprotocol": "inet", "protocol": "tcp", "source_net": "any",
"destination_net": "wanip", "destination_port": "8443", "target": "192.168.1.50", "local_port": "443",
"natreflection": "", "enabled": True, "target_host_ids": [a], "target_group_ids": []})
# two identical live rows on B, ONE twin: the first links, the second becomes its own object
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="b-1"), pf_row(uuid="b-2")])
rows = (await client.get(f"/api/port-forward/unmanaged?host_id={b}")).json()
assert sorted(r["bulk_action"] for r in rows) == ["link", "new"] # the dialog's summary equals what happens
done = (await client.post("/api/port-forward/adopt-all", json={"host_id": b, "link_existing": True})).json()
assert sorted(x["linked"] for x in done["adopted"]) == [False, True]
assert len((await client.get("/api/port-forward")).json()) == 2
@pytest.mark.asyncio
async def test_roles_for_linking(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("roles1", uuid="u-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("roles1", uuid="u-b")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
for role, expected in (("read_only", 403), ("operator", 201)):
user = await client.post("/api/users", json={"username": f"lnk-{role}", "password": "adopt-test-password-1", "role": role})
assert user.status_code == 201, user.text
session = await _login(f"lnk-{role}", "adopt-test-password-1")
try:
resp = await session.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
assert resp.status_code == expected, (role, resp.text)
finally:
await session.aclose()
@pytest.mark.asyncio
async def test_bulk_leaves_a_twin_alone_that_waits_for_a_confirmation(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("busy2", uuid="u-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("busy2", uuid="u-b")])
await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})
async with SessionLocal() as db:
from app.models.alias_deployment import AliasDeployment
(await db.scalars(select(AliasDeployment))).one().status = "pending"
await db.commit()
row = next(r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json() if r["uuid"] == "u-b")
assert row["linkable"] is False and row["bulk_action"] == "skip"
resp = await client.post("/api/aliases/adopt-all", json={"host_id": b, "link_existing": True})
assert resp.status_code == 200, resp.text
assert resp.json()["adopted"] == []
assert "twin_busy" in resp.json()["skipped"][0]["reasons"]
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
@pytest.mark.asyncio
async def test_linking_invalidates_an_open_edit_form(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "alias/searchItem", [alias_row("stale1", uuid="u-a")])
_search(respx_mock, B, "alias/searchItem", [alias_row("stale1", uuid="u-b")])
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
linked = (await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})).json()
assert linked["updated_at"] != first["updated_at"]
stale = {"name": "stale1", "type": "host", "content": linked["content"], "description": linked["description"],
"enabled": linked["enabled"], "target_host_ids": [a], "target_group_ids": [],
"expected_updated_at": first["updated_at"]}
assert (await client.put(f"/api/aliases/{first['id']}", json=stale)).status_code == 409
@pytest.mark.asyncio
async def test_bulk_neither_links_nor_duplicates_a_busy_nat_twin(client, respx_mock, two):
a, b = two
_search(respx_mock, A, "d_nat/searchRule", [pf_row(uuid="pf-a")])
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="pf-b")])
await client.post("/api/port-forward/adopt", json={"host_id": a, "uuid": "pf-a"})
async with SessionLocal() as db:
from app.models.port_forward_deployment import PortForwardDeployment
(await db.scalars(select(PortForwardDeployment))).one().status = "pending"
await db.commit()
resp = await client.post("/api/port-forward/adopt-all", json={"host_id": b, "link_existing": True})
assert resp.status_code == 200, resp.text
assert resp.json()["adopted"] == [] and "twin_busy" in resp.json()["skipped"][0]["reasons"]
async with SessionLocal() as db:
assert await db.scalar(select(func.count()).select_from(PortForwardRule)) == 1
+330
View File
@@ -0,0 +1,330 @@
"""GUI access (1.27): settings, one-time tickets and the gateway's internal redeem/revalidate/end API."""
import hashlib
import uuid
from datetime import datetime, timedelta, timezone
import pytest
import pytest_asyncio
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select, update
from app.config import settings
from app.database import SessionLocal
from app.main import app
from app.models.audit import AuditEvent
from app.models.gui_access import GuiSession, GuiTicket
from app.models.host import OpnsenseHost
from app.routers.auth import SECURE_COOKIE_NAME
GW_TOKEN = "gateway-token-" + "x" * 40
SITES = "https://opnexus.example:9441,https://opnexus.example:9442,https://opnexus.example:9443"
PW = "gui-user-password-1"
def sha(v: str) -> str:
return hashlib.sha256(v.encode()).hexdigest()
@pytest.fixture
def gui_env(monkeypatch):
monkeypatch.setattr(settings, "cookie_secure", True)
monkeypatch.setattr(settings, "gui_gateway_token", GW_TOKEN)
monkeypatch.setattr(settings, "opnexus_gui_sites", SITES)
async def _https_login(username: str, password: str) -> AsyncClient:
ac = AsyncClient(transport=ASGITransport(app=app), base_url="https://opnexus.example")
resp = await ac.post("/api/auth/login", json={"username": username, "password": password})
assert resp.status_code == 200, resp.text
assert ac.cookies.get(SECURE_COOKIE_NAME)
return ac
@pytest_asyncio.fixture
async def admin(gui_env):
ac = await _https_login(settings.admin_username, settings.admin_password)
yield ac
await ac.aclose()
async def _user(admin: AsyncClient, name: str, role: str) -> AsyncClient:
resp = await admin.post("/api/users", json={"username": name, "password": PW, "role": role})
if resp.status_code == 409 or (resp.status_code == 400 and "BENUTZERNAME" in resp.text):
users = (await admin.get("/api/users")).json()
uid = next(u["id"] for u in users if u["username"] == name)
await admin.patch(f"/api/users/{uid}/role", json={"role": role})
else:
assert resp.status_code == 201, resp.text
return await _https_login(name, PW)
async def _host(name="fw-gui", base_url="https://fw-gui.test", verify_tls=True, enabled=True) -> uuid.UUID:
async with SessionLocal() as db:
host = OpnsenseHost(name=name, base_url=base_url, api_key="apikey-must-not-leak", verify_tls=verify_tls,
enabled=enabled, tls_ca_pem="-----BEGIN CERTIFICATE-----\nPIN\n-----END CERTIFICATE-----\n")
db.add(host)
await db.commit()
return host.id
async def _enable(admin: AsyncClient, **over) -> dict:
body = {"enabled": True, "idle_timeout_minutes": 15, "absolute_timeout_minutes": 60, "slot_count": 4, **over}
resp = await admin.put("/api/settings/gui-access", json=body)
assert resp.status_code == 200, resp.text
return resp.json()
def _ticket_from(url: str) -> str:
return url.split("?t=", 1)[1]
async def _internal(path: str, body: dict, token: str = GW_TOKEN):
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://backend:8000") as ac:
return await ac.post(f"/internal/gui/{path}", json=body, headers={"X-Onx-Gateway-Token": token})
async def _redeem(admin: AsyncClient, url: str, slot: int | None = None, session_hash: str | None = None):
return await _internal("redeem", {
"ticket_hash": sha(_ticket_from(url)), "slot": slot or int(url.split(":")[2].split("/")[0]),
"session_hash": session_hash or sha(admin.cookies.get(SECURE_COOKIE_NAME)), "client_ip": "192.0.2.10",
})
# ── settings ──────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_settings_default_off_and_switchable(admin):
data = (await admin.get("/api/settings/gui-access")).json()
assert data["enabled"] is False and data["available"] is True and data["slots"] == [9441, 9442, 9443]
assert (await _enable(admin, slot_count=2))["slots"] == [9441, 9442]
bad = await admin.put("/api/settings/gui-access", json={"enabled": True, "idle_timeout_minutes": 90,
"absolute_timeout_minutes": 60, "slot_count": 2})
assert bad.status_code == 422
@pytest.mark.asyncio
async def test_cannot_enable_without_https_cookies(admin, monkeypatch):
monkeypatch.setattr(settings, "cookie_secure", False)
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as plain:
await plain.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
resp = await plain.put("/api/settings/gui-access", json={"enabled": True, "idle_timeout_minutes": 15,
"absolute_timeout_minutes": 60, "slot_count": 4})
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_GATEWAY_NOT_CONFIGURED"
assert "cookie_secure" in (await plain.get("/api/settings/gui-access")).json()["unavailable_reasons"]
@pytest.mark.asyncio
async def test_settings_admin_only(admin):
op = await _user(admin, "gui-op-settings", "operator")
assert (await op.get("/api/settings/gui-access")).status_code == 403
assert (await op.put("/api/settings/gui-access", json={"enabled": False, "idle_timeout_minutes": 15,
"absolute_timeout_minutes": 60, "slot_count": 4})).status_code == 403
# ── tickets ───────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_ticket_issued_and_only_hash_stored(admin):
host_id = await _host()
await _enable(admin)
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 200, resp.text
data = resp.json()
assert data["url"].startswith("https://opnexus.example:9441/__onx/redeem?t=") and data["expires_in"] == 30
raw = _ticket_from(data["url"])
async with SessionLocal() as db:
ticket = (await db.scalars(select(GuiTicket))).one()
assert ticket.ticket_hash == sha(raw) and raw not in repr(ticket.__dict__)
assert ticket.session_hash == sha(admin.cookies.get(SECURE_COOKIE_NAME))
assert (ticket.expires_at - ticket.created_at) == timedelta(seconds=30)
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "gui_access"))).all()
assert [e.action for e in events] == ["ticket"]
@pytest.mark.asyncio
@pytest.mark.parametrize("role,code", [("operator", "ADMIN_REQUIRED"), ("read_only", "READ_ONLY_ACCESS")])
async def test_ticket_refused_for_non_admins(admin, role, code):
host_id = await _host()
await _enable(admin)
other = await _user(admin, f"gui-{role}", role)
resp = await other.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 403 and resp.json()["detail"]["code"] == code
@pytest.mark.asyncio
async def test_ticket_refused_for_api_tokens(admin, anon_client):
host_id = await _host()
await _enable(admin)
token = (await admin.post("/api/tokens", json={"name": "gui", "scope": "admin", "current_password": settings.admin_password})).json()["token"]
resp = await anon_client.post(f"/api/hosts/{host_id}/gui-ticket", headers={"Authorization": f"Bearer {token}"})
assert resp.status_code == 403
@pytest.mark.asyncio
async def test_ticket_refused_when_switch_off_or_not_configured(admin, monkeypatch):
host_id = await _host()
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_ACCESS_DISABLED"
await _enable(admin)
monkeypatch.setattr(settings, "gui_gateway_token", "short")
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_GATEWAY_NOT_CONFIGURED"
@pytest.mark.asyncio
async def test_ticket_refused_without_secure_cookie(admin, monkeypatch):
host_id = await _host()
await _enable(admin)
monkeypatch.setattr(settings, "cookie_secure", False)
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as plain:
await plain.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
resp = await plain.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 409 and resp.json()["detail"]["code"] in ("GUI_REQUIRES_HTTPS", "GUI_GATEWAY_NOT_CONFIGURED")
monkeypatch.setattr(settings, "cookie_secure", True)
# A legacy plain-name cookie (pre-migration) is not enough either.
async with AsyncClient(transport=ASGITransport(app=app), base_url="https://opnexus.example") as legacy:
legacy.cookies.set("opnexus-session", admin.cookies.get(SECURE_COOKIE_NAME))
resp = await legacy.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_REQUIRES_HTTPS"
@pytest.mark.asyncio
@pytest.mark.parametrize("kwargs,reason", [
({"verify_tls": False}, "tls"), ({"base_url": "http://fw-plain.test"}, "https"), ({"enabled": False}, "disabled"),
])
async def test_ticket_refused_for_ineligible_host(admin, kwargs, reason):
host_id = await _host(**kwargs)
await _enable(admin)
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
assert resp.status_code == 409 and resp.json()["detail"] == {"code": "GUI_HOST_NOT_ELIGIBLE", "params": {"reason": reason}}
@pytest.mark.asyncio
async def test_slots_fixed_per_firewall_and_limited(admin):
a, b, c = await _host("fw-a"), await _host("fw-b", "https://fw-b.test"), await _host("fw-c", "https://fw-c.test")
await _enable(admin, slot_count=2)
slot_a = (await admin.post(f"/api/hosts/{a}/gui-ticket")).json()["slot"]
slot_b = (await admin.post(f"/api/hosts/{b}/gui-ticket")).json()["slot"]
assert slot_a != slot_b and {slot_a, slot_b} == {9441, 9442}
assert (await admin.post(f"/api/hosts/{a}/gui-ticket")).json()["slot"] == slot_a
full = await admin.post(f"/api/hosts/{c}/gui-ticket")
assert full.status_code == 409 and full.json()["detail"]["code"] == "GUI_NO_FREE_SLOT"
# ── internal API ──────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_internal_api_needs_gateway_token(admin, monkeypatch):
body = {"gui_session_id": str(uuid.uuid4()), "session_hash": "0" * 64}
assert (await _internal("revalidate", body, token="wrong")).status_code == 401
assert (await _internal("revalidate", body, token="")).status_code == 401
monkeypatch.setattr(settings, "gui_gateway_token", "")
assert (await _internal("revalidate", body, token="")).status_code == 404
# Not part of the public API surface.
assert "/internal/gui/redeem" not in (await admin.get("/api/openapi.json")).text
@pytest.mark.asyncio
async def test_redeem_returns_no_credentials_and_is_single_use(admin):
host_id = await _host()
await _enable(admin)
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
resp = await _redeem(admin, url)
assert resp.status_code == 200, resp.text
data = resp.json()
assert data["upstream_origin"] == "https://fw-gui.test" and "PIN" in data["ca_pem"]
assert data["idle_timeout_seconds"] == 900 and data["revalidate_seconds"] == 20
assert "apikey-must-not-leak" not in resp.text and "secret" not in resp.text.lower() and "api_key" not in resp.text
again = await _redeem(admin, url)
assert again.status_code == 403 and again.json()["detail"]["reason"] == "replayed"
async with SessionLocal() as db:
actions = [e.action for e in (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "gui_access")
.order_by(AuditEvent.created_at))).all()]
assert actions[:2] == ["ticket", "redeem"] and actions.count("redeem") == 2
@pytest.mark.asyncio
async def test_redeem_bound_to_session_slot_and_time(admin):
host_id = await _host()
await _enable(admin)
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
wrong = await _redeem(admin, url, session_hash=sha("another-browser-session"))
assert wrong.status_code == 403 and wrong.json()["detail"]["reason"] == "session"
assert (await _redeem(admin, url)).json()["detail"]["reason"] == "replayed" # burned by the failed attempt
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
assert (await _redeem(admin, url, slot=9443)).json()["detail"]["reason"] == "slot"
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
async with SessionLocal() as db:
await db.execute(update(GuiTicket).where(GuiTicket.ticket_hash == sha(_ticket_from(url)))
.values(expires_at=datetime.now(timezone.utc) - timedelta(seconds=1)))
await db.commit()
assert (await _redeem(admin, url)).json()["detail"]["reason"] == "expired"
unknown = await _internal("redeem", {"ticket_hash": sha("nope"), "session_hash": sha("x"), "slot": 9441})
assert unknown.status_code == 403 and unknown.json()["detail"]["reason"] == "unknown"
async def _open(admin: AsyncClient, host_id) -> dict:
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
data = (await _redeem(admin, url)).json()
return {"gui_session_id": data["gui_session_id"], "session_hash": sha(admin.cookies.get(SECURE_COOKIE_NAME))}
@pytest.mark.asyncio
async def test_revalidate_ok_then_logout_ends_it(admin):
host_id = await _host()
await _enable(admin)
ref = await _open(admin, host_id)
ok = (await _internal("revalidate", ref)).json()
assert ok["ok"] is True and ok["revalidate_seconds"] == 20
await admin.post("/api/auth/logout")
assert (await _internal("revalidate", ref)).json() == {"ok": False, "reason": "session"}
assert (await _internal("revalidate", ref)).json()["ok"] is False
async with SessionLocal() as db:
ends = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "gui_access", AuditEvent.action == "end"))).all()
assert len(ends) == 1 and ends[0].details["reason"] == "session" and "duration_seconds" in ends[0].details
@pytest.mark.asyncio
async def test_revalidate_fails_after_demotion(admin):
host_id = await _host()
await _enable(admin)
second = await _user(admin, "gui-second-admin", "admin")
ref = await _open(second, host_id)
users = (await admin.get("/api/users")).json()
uid = next(u["id"] for u in users if u["username"] == "gui-second-admin")
assert (await admin.patch(f"/api/users/{uid}/role", json={"role": "operator"})).status_code == 200
assert (await _internal("revalidate", ref)).json()["ok"] is False
@pytest.mark.asyncio
async def test_revalidate_fails_when_switched_off(admin):
host_id = await _host()
await _enable(admin)
ref = await _open(admin, host_id)
await _enable(admin, enabled=False)
assert (await _internal("revalidate", ref)).json()["ok"] is False
# The live re-check alone (switch flipped without the end-all sweep) also ends it.
await _enable(admin)
ref = await _open(admin, host_id)
from app.models.gui_access import GuiAccessSettings
async with SessionLocal() as db:
await db.execute(update(GuiAccessSettings).values(enabled=False))
await db.commit()
assert (await _internal("revalidate", ref)).json() == {"ok": False, "reason": "disabled"}
@pytest.mark.asyncio
async def test_revalidate_enforces_absolute_limit_and_end(admin):
host_id = await _host()
await _enable(admin)
ref = await _open(admin, host_id)
async with SessionLocal() as db:
await db.execute(update(GuiSession).values(started_at=datetime.now(timezone.utc) - timedelta(minutes=61)))
await db.commit()
assert (await _internal("revalidate", ref)).json() == {"ok": False, "reason": "absolute"}
ref2 = await _open(admin, host_id)
assert (await _internal("end", {**ref2, "reason": "idle"})).json() == {"ok": True}
assert (await _internal("revalidate", ref2)).json() == {"ok": False, "reason": "idle"}
assert (await _internal("end", {**ref2, "session_hash": sha("x"), "reason": "idle"})).json() == {"ok": False}
+470
View File
@@ -0,0 +1,470 @@
"""GUI gateway (1.27): every counter-measure of the design's threat table against a mock firewall,
with the real backend app behind the internal API."""
import datetime
import logging
import subprocess
import sys
import httpx
import pytest
import pytest_asyncio
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.x509.oid import NameOID
from httpx import ASGITransport, AsyncClient
from sqlalchemy import select
from starlette.requests import Request
from starlette.responses import HTMLResponse, Response, StreamingResponse
from app import gui_gateway as gw
from app.config import settings
from app.database import SessionLocal
from app.main import app as backend_app
from app.models.gui_access import GuiSession
from app.routers.auth import SECURE_COOKIE_NAME
from tests.test_gui_access import GW_TOKEN, SITES, _enable, _host, _https_login, gui_env # noqa: F401
PASSWORD = "S3cret-Testpasswort!"
OWN = "https://opnexus.example:9441"
class MockFirewall:
"""Records what reaches the "firewall" and answers like OPNsense's authgui.inc."""
def __init__(self):
self.requests: list[dict] = []
async def __call__(self, scope, receive, send):
request = Request(scope, receive)
body = await request.body()
self.requests.append({"method": request.method, "path": request.url.path, "query": request.url.query,
"headers": dict(request.headers), "body": body})
path = request.url.path
if path == "/" and request.method == "POST":
from urllib.parse import parse_qs
if parse_qs(body.decode()).get("passwordfld") == ["right"]:
response = Response(status_code=302, headers={"Location": "/ui/core/dashboard"})
else:
response = HTMLResponse("<form>Wrong username or password.</form>")
elif path == "/redir":
response = Response(status_code=302, headers={"Location": request.query_params["to"]})
elif path == "/logout":
response = Response(status_code=302, headers={"Location": "/"})
response.headers.append("set-cookie", "PHPSESSID=deleted; expires=Thu, 01 Jan 1970 00:00:01 GMT; path=/")
elif path == "/big":
async def chunks():
for i in range(50):
yield (b"%04d" % i) * 1024
response = StreamingResponse(chunks(), media_type="application/octet-stream")
else:
response = HTMLResponse("<html>firewall</html>", headers={"Content-Security-Policy": "default-src 'self'"})
response.headers.append("set-cookie", "PHPSESSID=fw-session-1; path=/; secure; HttpOnly")
response.headers.append("x-powered-by", "PHP")
await response(scope, receive, send)
class Clock:
def __init__(self):
self.now = 1000.0
def __call__(self):
return self.now
@pytest.fixture
def firewall():
return MockFirewall()
@pytest_asyncio.fixture
async def setup(gui_env, firewall): # noqa: F811
admin = await _https_login(settings.admin_username, settings.admin_password)
host_id = await _host()
await _enable(admin)
clock = Clock()
gateway = gw.GuiGateway(
backend=gw.BackendClient("http://backend:8000", GW_TOKEN, transport=ASGITransport(app=backend_app)),
slot_origins=gw.slot_origins_from(SITES),
upstream_factory=lambda gs: httpx.AsyncClient(transport=ASGITransport(app=firewall)),
clock=clock,
)
browser = AsyncClient(transport=ASGITransport(app=gateway), base_url=OWN)
yield {"admin": admin, "host_id": host_id, "gateway": gateway, "browser": browser, "clock": clock, "fw": firewall}
await browser.aclose()
await admin.aclose()
async def _ticket(s) -> tuple[str, int]:
data = (await s["admin"].post(f"/api/hosts/{s['host_id']}/gui-ticket")).json()
return data["url"].split("?t=", 1)[1], data["slot"]
def _opnexus_cookie(s) -> str:
return f"{SECURE_COOKIE_NAME}={s['admin'].cookies.get(SECURE_COOKIE_NAME)}"
async def _redeem(s, ticket: str, slot: int = 9441, site: str = "same-site", cookie: str | None = None):
return await s["browser"].get(f"/__onx/redeem?t={ticket}", headers={
"X-Onx-Slot": str(slot), "Sec-Fetch-Site": site, "Cookie": _opnexus_cookie(s) if cookie is None else cookie})
async def _open(s) -> str:
ticket, slot = await _ticket(s)
resp = await _redeem(s, ticket, slot)
assert resp.status_code == 200, resp.text
raw = resp.headers["set-cookie"]
return raw.split(";")[0] # "__Host-onx-gw-9441=<value>"
async def _get(s, path: str, gw_cookie: str, slot: int = 9441, site: str | None = "same-origin", extra: dict | None = None,
method: str = "GET", **kwargs):
headers = {"X-Onx-Slot": str(slot), "Cookie": f"{_opnexus_cookie(s)}; {gw_cookie}; other=1", **(extra or {})}
if site is not None:
headers["Sec-Fetch-Site"] = site
return await s["browser"].request(method, path, headers=headers, **kwargs)
# ── redeem ────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_redeem_sets_host_cookie_and_clears_site_data(setup):
ticket, slot = await _ticket(setup)
resp = await _redeem(setup, ticket, slot)
assert resp.status_code == 200
cookie = resp.headers["set-cookie"]
assert cookie.startswith(f"__Host-onx-gw-{slot}=")
for attr in ("Path=/", "Secure", "HttpOnly", "SameSite=Strict"):
assert attr in cookie
assert "Domain" not in cookie and ticket not in resp.text
assert resp.headers["clear-site-data"] == '"cache", "storage"'
assert 'http-equiv="refresh"' in resp.text and resp.headers["cache-control"] == "no-store"
@pytest.mark.asyncio
async def test_redeem_refused_without_session_cookie_cross_site_or_twice(setup):
ticket, _ = await _ticket(setup)
# Refused by the gateway itself, before the backend sees (and burns) the ticket:
assert (await _redeem(setup, ticket, cookie="")).status_code == 403
assert (await _redeem(setup, ticket, site="cross-site")).status_code == 403
assert (await _redeem(setup, ticket, site="")).status_code == 403
assert (await _redeem(setup, ticket)).status_code == 200
assert (await _redeem(setup, ticket)).status_code == 403 # single use
other = await _ticket(setup)
assert (await _redeem(setup, other[0], slot=9442)).status_code == 403 # bound to its slot
# ── cookies never cross ───────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_browser_cookies_never_upstream_and_firewall_cookies_never_downstream(setup):
cookie = await _open(setup)
first = await _get(setup, "/ui/core/dashboard", cookie)
assert first.status_code == 200 and "set-cookie" not in first.headers and "x-powered-by" not in first.headers
second = await _get(setup, "/ui/x", cookie)
assert second.status_code == 200
up = setup["fw"].requests
assert "cookie" not in up[0]["headers"]
assert up[1]["headers"]["cookie"] == "PHPSESSID=fw-session-1"
session_value = setup["admin"].cookies.get(SECURE_COOKIE_NAME)
for r in up:
flat = repr(r["headers"])
assert session_value not in flat and "onx-gw" not in flat and "other=1" not in flat
# A firewall logout deletes the cookie from the jar.
await _get(setup, "/logout", cookie)
await _get(setup, "/ui/y", cookie)
assert "cookie" not in setup["fw"].requests[-1]["headers"]
@pytest.mark.asyncio
async def test_request_rebuilt_with_header_allow_list(setup):
cookie = await _open(setup)
await _get(setup, "/ui/a?b=1", cookie, extra={
"Authorization": "Basic Zm9vOmJhcg==", "Referer": f"{OWN}/ui/prev", "X-Forwarded-For": "192.0.2.99",
"X-CSRFToken": "csrf-1", "Accept-Language": "de", "Proxy-Authorization": "x",
})
r = setup["fw"].requests[-1]
assert r["path"] == "/ui/a" and r["query"] == "b=1"
h = r["headers"]
assert h["host"] == "fw-gui.test" and h["x-csrftoken"] == "csrf-1" and h["accept-language"] == "de"
for dropped in ("authorization", "referer", "origin", "x-forwarded-for", "proxy-authorization", "x-onx-slot",
"sec-fetch-site"):
assert dropped not in h
# ── redirects ─────────────────────────────────────────────────────────────
@pytest.mark.asyncio
@pytest.mark.parametrize("target,expected", [
("https://evil.example/x", None), ("//evil.example/x", None), ("/\\evil.example", None),
("https://fw-gui.test:8443/x", None), ("http://fw-gui.test/x", None),
("https://fw-gui.test/ui/x?y=1", "/ui/x?y=1"), ("https://FW-GUI.test:443/", "/"), ("/ui/z", "/ui/z"),
])
async def test_location_rewritten_or_dropped(setup, target, expected):
cookie = await _open(setup)
resp = await _get(setup, "/redir", cookie, params={"to": target})
if expected is None:
assert resp.status_code == 502 and "location" not in resp.headers
else:
assert resp.status_code == 302 and resp.headers["location"] == expected
# ── request shape ─────────────────────────────────────────────────────────
async def _raw(gateway, method="GET", path="/", raw_path=None, headers=None, scope_type="http"):
sent = []
async def receive():
return {"type": "http.request", "body": b"", "more_body": False}
async def send(message):
sent.append(message)
scope = {"type": scope_type, "method": method, "path": path, "raw_path": raw_path if raw_path is not None else path.encode(),
"query_string": b"", "headers": [(k.encode(), v.encode()) for k, v in (headers or {}).items()]}
await gateway(scope, receive, send)
return sent
@pytest.mark.asyncio
async def test_absolute_form_connect_upgrade_websocket_service_worker_refused(setup):
g = setup["gateway"]
base = {"x-onx-slot": "9441", "sec-fetch-site": "same-origin"}
assert (await _raw(g, path="http://evil.example/", headers=base))[0]["status"] == 400
assert (await _raw(g, path="/", raw_path=b"http://evil.example/", headers=base))[0]["status"] == 400
assert (await _raw(g, path="//evil.example/", headers=base))[0]["status"] == 400
assert (await _raw(g, method="CONNECT", path="/", headers=base))[0]["status"] == 501
assert (await _raw(g, method="TRACE", path="/", headers=base))[0]["status"] == 501
assert (await _raw(g, headers={**base, "upgrade": "websocket", "connection": "Upgrade"}))[0]["status"] == 400
assert (await _raw(g, headers={**base, "service-worker": "script"}))[0]["status"] == 403
assert (await _raw(g, scope_type="websocket"))[0]["type"] == "websocket.close"
assert setup["fw"].requests == []
@pytest.mark.asyncio
@pytest.mark.parametrize("slot", ["", "abc", "9999"])
async def test_slot_header_required(setup, slot):
sent = await _raw(setup["gateway"], headers={"x-onx-slot": slot, "sec-fetch-site": "same-origin"})
assert sent[0]["status"] == 400
# ── Sec-Fetch-Site and Origin ─────────────────────────────────────────────
@pytest.mark.asyncio
@pytest.mark.parametrize("site,expected", [("same-site", 403), ("cross-site", 403), (None, 403), ("none", 200), ("same-origin", 200)])
async def test_sec_fetch_site_enforced(setup, site, expected):
cookie = await _open(setup)
assert (await _get(setup, "/ui/x", cookie, site=site)).status_code == expected
@pytest.mark.asyncio
async def test_origin_required_for_writes(setup):
cookie = await _open(setup)
assert (await _get(setup, "/api/x", cookie, method="POST", content=b"{}")).status_code == 403
other_slot = await _get(setup, "/api/x", cookie, method="POST", content=b"{}", extra={"Origin": "https://opnexus.example:9442"})
assert other_slot.status_code == 403
opnexus = await _get(setup, "/api/x", cookie, method="POST", content=b"{}", extra={"Origin": "https://opnexus.example"})
assert opnexus.status_code == 403
assert (await _get(setup, "/api/x", cookie, method="POST", content=b"{}", extra={"Origin": OWN})).status_code == 200
@pytest.mark.asyncio
async def test_foreign_slot_cookie_rejected(setup):
cookie = await _open(setup) # slot 9441
value = cookie.split("=", 1)[1]
resp = await _get(setup, "/ui/x", f"__Host-onx-gw-9442={value}", slot=9442)
assert resp.status_code == 401
assert (await _get(setup, "/ui/x", "", slot=9441)).status_code == 401
assert (await _get(setup, "/__onx/anything", cookie)).status_code == 404
# ── response hardening ────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_security_headers_and_streaming(setup):
cookie = await _open(setup)
resp = await _get(setup, "/ui/x", cookie)
csps = resp.headers.get_list("content-security-policy")
assert any("frame-ancestors 'none'" in c and "connect-src 'self'" in c and "form-action 'self'" in c for c in csps)
assert "default-src 'self'" in csps # the firewall's own CSP applies as well
assert resp.headers["x-frame-options"] == "DENY" and resp.headers["cache-control"] == "no-store"
big = await _get(setup, "/big", cookie)
assert big.status_code == 200 and len(big.content) == 50 * 4 * 1024 and big.content.endswith(b"0049" * 1024)
@pytest.mark.asyncio
async def test_body_limit(setup, monkeypatch):
cookie = await _open(setup)
monkeypatch.setattr(gw, "BODY_LIMIT", 1000)
resp = await _get(setup, "/api/x", cookie, method="POST", content=b"x" * 2000, extra={"Origin": OWN})
assert resp.status_code == 413 and setup["fw"].requests == []
async def stream():
for _ in range(4):
yield b"y" * 600
chunked = await _get(setup, "/api/x", cookie, method="POST", content=stream(), extra={"Origin": OWN})
assert chunked.status_code == 413 and setup["fw"].requests == []
# ── login protection and logging ──────────────────────────────────────────
def _login(password: str) -> dict:
return {"content": f"usernamefld=root&passwordfld={password}&login=1".encode(),
"extra": {"Origin": OWN, "Content-Type": "application/x-www-form-urlencoded"}}
@pytest.mark.asyncio
async def test_login_rate_limit_per_firewall(setup):
cookie = await _open(setup)
for _ in range(3):
assert (await _get(setup, "/", cookie, method="POST", **_login("wrong"))).status_code == 200
blocked = await _get(setup, "/", cookie, method="POST", **_login("right"))
assert blocked.status_code == 429
assert sum(1 for r in setup["fw"].requests if r["method"] == "POST") == 3
setup["clock"].now += gw.LOGIN_FAILURE_WINDOW + 1
await _get(setup, "/ui/x", cookie) # (session re-check due; still valid)
assert (await _get(setup, "/", cookie, method="POST", **_login("right"))).status_code == 302
@pytest.mark.asyncio
async def test_multipart_login_counted_too(setup):
# PHP's $_POST is filled from multipart bodies as well: a login posted that way must not bypass the limit.
cookie = await _open(setup)
files = {"login": (None, "1"), "usernamefld": (None, "root"), "passwordfld": (None, "wrong")}
codes = [(await _get(setup, "/", cookie, method="POST", files=files, extra={"Origin": OWN})).status_code
for _ in range(4)]
assert codes == [200, 200, 200, 429]
assert not gw.is_login_attempt("POST", "multipart/form-data; boundary=x", b'name="file"; name="login"')
@pytest.mark.asyncio
async def test_path_forwarded_as_sent_and_control_characters_refused(setup):
cookie = await _open(setup)
for sent, seen in (("/a%3Fb=1", "/a?b=1"), ("/api/x%2Fy", "/api/x/y"), ("/a%2541", "/a%41"), ("/a%23f", "/a#f")):
resp = await _get(setup, sent, cookie)
last = setup["fw"].requests[-1]
# Reaches the firewall still encoded (or decoded exactly once): never split into a query, never
# decoded twice (%2541 must not become "A").
assert resp.status_code == 200 and last["path"] in (sent, seen) and last["query"] == "", (sent, last)
before = len(setup["fw"].requests)
resp = await _get(setup, "/a%0d%0aX-Injected:%201", cookie)
assert resp.status_code in (200, 400)
assert all("x-injected" not in r["headers"] for r in setup["fw"].requests[before:])
@pytest.mark.asyncio
async def test_successful_login_not_counted(setup):
cookie = await _open(setup)
for _ in range(5):
assert (await _get(setup, "/", cookie, method="POST", **_login("right"))).status_code == 302
@pytest.mark.asyncio
async def test_password_and_ticket_never_logged(setup, caplog):
caplog.set_level(logging.DEBUG)
ticket, slot = await _ticket(setup)
resp = await _redeem(setup, ticket, slot)
cookie = resp.headers["set-cookie"].split(";")[0]
await _get(setup, "/", cookie, method="POST", **_login(PASSWORD))
await _get(setup, "/ui/x?secretquery=abc", cookie)
text = caplog.text
assert "POST / 200" in text and "GET /ui/x 200" in text
for leaked in (PASSWORD, ticket, "secretquery", "usernamefld", cookie.split("=", 1)[1], "fw-session-1"):
assert leaked not in text
def test_uvicorn_runs_without_access_log(monkeypatch):
captured = {}
monkeypatch.setenv("GUI_GATEWAY_TOKEN", GW_TOKEN)
monkeypatch.setenv("OPNEXUS_GUI_SITES", SITES)
import uvicorn
monkeypatch.setattr(uvicorn, "run", lambda app, **kw: captured.update(kw))
gw.main()
assert captured["access_log"] is False and captured["http"] == "h11" and captured["proxy_headers"] is False
def test_gateway_refuses_to_start_without_token(monkeypatch):
monkeypatch.setenv("GUI_GATEWAY_TOKEN", "short")
monkeypatch.setenv("OPNEXUS_GUI_SITES", SITES)
with pytest.raises(SystemExit):
gw.create_app()
# ── session lifetime ──────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_revalidation_failure_ends_session(setup):
cookie = await _open(setup)
assert (await _get(setup, "/ui/x", cookie)).status_code == 200
await setup["admin"].post("/api/auth/logout")
assert (await _get(setup, "/ui/x", cookie)).status_code == 200 # not yet due (≤ 30 s)
setup["clock"].now += 31
assert (await _get(setup, "/ui/x", cookie)).status_code == 401
assert setup["gateway"].sessions == {}
@pytest.mark.asyncio
async def test_sweep_ends_idle_sessions_and_reports(setup):
cookie = await _open(setup)
setup["clock"].now += 15 * 60
await setup["gateway"].sweep()
assert setup["gateway"].sessions == {}
assert (await _get(setup, "/ui/x", cookie)).status_code == 401
async with SessionLocal() as db:
assert (await db.scalars(select(GuiSession.end_reason))).one() == "idle"
@pytest.mark.asyncio
async def test_absolute_deadline(setup):
cookie = await _open(setup)
gs = next(iter(setup["gateway"].sessions.values()))
gs.absolute_deadline = setup["gateway"].wall_clock() - 1
assert (await _get(setup, "/ui/x", cookie)).status_code == 401
assert setup["gateway"].sessions == {}
@pytest.mark.asyncio
async def test_backend_unreachable_fails_closed(setup):
cookie = await _open(setup)
async def broken(gs):
raise httpx.ConnectError("down")
setup["gateway"].backend.revalidate = broken
setup["clock"].now += 31
assert (await _get(setup, "/ui/x", cookie)).status_code == 503
assert len(setup["fw"].requests) == 0
# ── upstream transport and isolation ──────────────────────────────────────
def _pem() -> str:
key = ec.generate_private_key(ec.SECP256R1())
name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "OPNsense.localdomain")])
now = datetime.datetime.now(datetime.timezone.utc)
cert = (x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
.serial_number(1).not_valid_before(now).not_valid_after(now + datetime.timedelta(days=1)).sign(key, hashes.SHA256()))
return cert.public_bytes(serialization.Encoding.PEM).decode()
def test_upstream_client_pins_and_uses_overlay_proxy(monkeypatch):
captured = {}
monkeypatch.setattr(gw.httpx, "AsyncClient", lambda **kw: captured.update(kw) or "client")
gs = gw.GatewaySession(gui_session_id="x", session_hash="y", slot=9441, upstream_origin="https://203.0.113.5",
ca_pem=_pem(), proxy_url="http://wg-hub:8888", idle_timeout=900, absolute_deadline=0,
revalidate_every=30)
assert gw.build_upstream_client(gs) == "client"
ctx = captured["verify"]
assert ctx.check_hostname is False and ctx.verify_mode.name == "CERT_REQUIRED"
assert captured["proxy"] == "http://wg-hub:8888" and captured["follow_redirects"] is False and captured["trust_env"] is False
gs.ca_pem, gs.proxy_url = None, None
gw.build_upstream_client(gs)
assert captured["verify"] is True and captured["proxy"] is None
def test_gateway_never_loads_the_database_layer():
code = ("import sys, app.gui_gateway as g; "
"g.build_upstream_client(g.GatewaySession('x','y',9441,'https://203.0.113.5',None,None,1,1,1)); "
"from app.services import tls_pin_service; "
"bad=[m for m in ('app.database','sqlalchemy','app.models') if m in sys.modules]; "
"print(bad); sys.exit(1 if bad else 0)")
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True)
assert result.returncode == 0, result.stdout + result.stderr
+203
View File
@@ -0,0 +1,203 @@
"""Recurring maintenance windows: occurrence maths (weekly, monthly, DST, exceptions, cut-off)."""
from datetime import date, datetime, time, timedelta, timezone
import pytest
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
from app.services import window_service as ws
UTC = timezone.utc
def _w(**kw) -> MaintenanceWindow:
base = dict(name="w", timezone="Europe/Berlin", recurrence="weekly", weekdays=[6], start_time=time(2, 0),
duration_minutes=180, start_cutoff_minutes=60, enabled=True, created_by="admin")
base.update(kw)
w = MaintenanceWindow(**base)
w.exceptions = kw.get("exceptions", [])
return w
def _starts(w, after, n=4, **kw):
return [o.start for o in ws.occurrences(w, after, n, **kw)]
def test_weekly_sunday_in_summer_and_winter_keeps_local_time():
w = _w()
# 2026-10-18 is a Sunday (CEST, +2), 2026-11-01 a Sunday (CET, +1)
starts = _starts(w, datetime(2026, 10, 12, tzinfo=UTC), 4)
assert starts[0] == datetime(2026, 10, 18, 0, 0, tzinfo=UTC)
assert starts[2] == datetime(2026, 11, 1, 1, 0, tzinfo=UTC)
assert all(s.astimezone(ws.ZoneInfo("Europe/Berlin")).hour == 2 for s in starts)
def test_weekly_several_days_in_order():
w = _w(weekdays=[1, 3], timezone="UTC", start_time=time(22, 0), duration_minutes=60)
starts = _starts(w, datetime(2026, 10, 19, tzinfo=UTC), 3) # Monday
assert starts == [datetime(2026, 10, 20, 22, tzinfo=UTC), datetime(2026, 10, 22, 22, tzinfo=UTC),
datetime(2026, 10, 27, 22, tzinfo=UTC)]
def test_monthly_nth_weekday_first_and_last():
first_tue = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=1, month_weekday=1, timezone="UTC")
assert [s.date() for s in _starts(first_tue, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
date(2026, 10, 6), date(2026, 11, 3), date(2026, 12, 1)]
last_sun = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=-1, month_weekday=6, timezone="UTC")
assert [s.date() for s in _starts(last_sun, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
date(2026, 10, 25), date(2026, 11, 29), date(2026, 12, 27)]
def test_monthly_fixed_day():
w = _w(recurrence="monthly", weekdays=[], month_mode="day", month_day=15, timezone="UTC")
assert [s.date() for s in _starts(w, datetime(2026, 10, 16, tzinfo=UTC), 2)] == [date(2026, 11, 15), date(2026, 12, 15)]
def test_spring_forward_missing_time_moves_forward():
# 2026-03-29 (last Sunday of March): 02:00-03:00 local does not exist in Europe/Berlin.
w = _w(start_time=time(2, 30))
occ = ws.occurrences(w, datetime(2026, 3, 28, tzinfo=UTC), 1)[0]
assert occ.local_date == date(2026, 3, 29)
assert occ.start == datetime(2026, 3, 29, 1, 30, tzinfo=UTC) # = 03:30 CEST
assert occ.start.astimezone(ws.ZoneInfo("Europe/Berlin")).strftime("%H:%M") == "03:30"
def test_fall_back_ambiguous_time_uses_the_first_and_duration_is_real_time():
# 2026-10-25 (last Sunday of October): 02:00-03:00 local happens twice.
w = _w(start_time=time(2, 30), duration_minutes=120)
occ = ws.occurrences(w, datetime(2026, 10, 24, tzinfo=UTC), 1)[0]
assert occ.start == datetime(2026, 10, 25, 0, 30, tzinfo=UTC) # 02:30 CEST (first pass)
assert occ.end - occ.start == timedelta(hours=2)
def test_skip_and_move_exceptions():
moved = datetime(2026, 10, 31, 20, 0, tzinfo=UTC) # Saturday evening instead of Sunday night
w = _w(exceptions=[
MaintenanceWindowException(occurrence_date=date(2026, 10, 25), kind="skip", reason="x", actor="a"),
MaintenanceWindowException(occurrence_date=date(2026, 11, 1), kind="move", moved_start=moved, reason="x", actor="a"),
])
occs = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3)
assert [o.local_date for o in occs] == [date(2026, 11, 1), date(2026, 11, 8), date(2026, 11, 15)]
assert occs[0].moved and occs[0].start == moved and occs[0].end == moved + timedelta(minutes=180)
listed = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3, with_skipped=True)
assert [(o.local_date, o.skipped, o.moved) for o in listed] == [
(date(2026, 10, 25), True, False), (date(2026, 11, 1), False, True), (date(2026, 11, 8), False, False)]
def test_open_window_is_current_and_cutoff_limits_new_starts():
w = _w(timezone="UTC", weekdays=[6], start_time=time(2, 0), duration_minutes=180, start_cutoff_minutes=60)
start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
assert ws.current(w, start - timedelta(seconds=1)) is None
assert ws.current(w, start).start == start
assert ws.startable(w, start + timedelta(minutes=59)) is not None
assert ws.startable(w, start + timedelta(minutes=60)) is None # past the cut-off: nothing new starts
assert ws.current(w, start + timedelta(minutes=179)) is not None # still open, running work may finish
assert ws.current(w, start + timedelta(minutes=180)) is None
# an occurrence that is open right now is the first one listed
assert ws.occurrences(w, start + timedelta(minutes=30), 1)[0].start == start
def test_cutoff_never_exceeds_duration_and_disabled_window_is_never_open():
w = _w(timezone="UTC", duration_minutes=30, start_cutoff_minutes=90)
start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
assert ws.occurrences(w, start, 1)[0].cutoff == start + timedelta(minutes=30)
w.enabled = False
assert ws.current(w, start) is None and ws.startable(w, start) is None
@pytest.mark.parametrize("name,ok", [("Europe/Berlin", True), ("UTC", True), ("Mars/Olympus", False), ("", False), ("../etc", False)])
def test_timezone_validation(name, ok):
assert ws.valid_timezone(name) is ok
# ── API ─────────────────────────────────────────────────────────────────────
from sqlalchemy import select # noqa: E402
from app.database import SessionLocal # noqa: E402
from app.models.audit import AuditEvent # noqa: E402
_WEEKLY = {"name": "Sonntagnacht", "timezone": "Europe/Berlin", "recurrence": "weekly", "weekdays": [6],
"start_time": "02:00", "duration_minutes": 180, "start_cutoff_minutes": 60}
@pytest.mark.asyncio
async def test_window_crud_and_next_occurrences(client):
resp = await client.post("/api/maintenance-windows", json=_WEEKLY)
assert resp.status_code == 201, resp.text
w = resp.json()
assert w["start_time"] == "02:00" and len(w["next"]) == 5 and w["exceptions"] == [] and w["used_by"] == []
assert (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()["detail"]["code"] == "WINDOW_NAME_TAKEN"
upd = await client.put(f"/api/maintenance-windows/{w['id']}", json={**_WEEKLY, "start_time": "03:15", "start_cutoff_minutes": 999})
assert upd.status_code == 200 and upd.json()["start_time"] == "03:15"
assert upd.json()["start_cutoff_minutes"] == 180 # never longer than the window itself
occ = await client.get(f"/api/maintenance-windows/{w['id']}/occurrences?count=12")
assert occ.status_code == 200 and len(occ.json()) == 12
assert (await client.delete(f"/api/maintenance-windows/{w['id']}")).status_code == 204
assert (await client.get(f"/api/maintenance-windows/{w['id']}")).status_code == 404
@pytest.mark.parametrize("change,field", [
({"timezone": "Mars/Base"}, "timezone"),
({"start_time": "25:00"}, "start_time"),
({"weekdays": []}, "weekdays"),
({"weekdays": [7]}, "weekdays"),
({"recurrence": "monthly", "month_mode": "day", "month_day": 31}, "month_day"),
({"recurrence": "monthly", "month_mode": "nth_weekday", "month_nth": 5, "month_weekday": 1}, "month_nth"),
({"recurrence": "monthly"}, "month_mode"),
])
@pytest.mark.asyncio
async def test_window_validation(client, change, field):
resp = await client.post("/api/maintenance-windows", json={**_WEEKLY, **change})
assert resp.status_code == 422 and resp.json()["detail"] == {"code": "WINDOW_INVALID", "params": {"field": field}}
@pytest.mark.asyncio
async def test_exceptions_skip_move_and_audit(client):
w = (await client.post("/api/maintenance-windows", json={**_WEEKLY, "timezone": "UTC"})).json()
first = date.fromisoformat(w["next"][0]["local_date"])
second = date.fromisoformat(w["next"][1]["local_date"])
bad = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first + timedelta(days=1)).isoformat(), "kind": "skip"})
assert bad.status_code == 422 and bad.json()["detail"]["params"]["field"] == "occurrence_date"
past = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first - timedelta(days=14)).isoformat(), "kind": "skip"})
assert past.status_code == 422
skip = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip", "reason": "Feiertag"})
assert skip.status_code == 201, skip.text
assert skip.json()["next"][0]["skipped"] is True
dup = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip"})
assert dup.status_code == 409 and dup.json()["detail"]["code"] == "WINDOW_EXCEPTION_EXISTS"
no_target = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move"})
assert no_target.status_code == 422 and no_target.json()["detail"]["params"]["field"] == "moved_start"
target = datetime.combine(second, time(20, 0), tzinfo=UTC) - timedelta(days=1)
moved = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move", "moved_start": target.isoformat()})
assert moved.status_code == 201
regular = [o for o in moved.json()["next"] if not o["skipped"]]
assert regular[0]["moved"] is True and datetime.fromisoformat(regular[0]["start"]) == target
eid = moved.json()["exceptions"][1]["id"]
after = await client.delete(f"/api/maintenance-windows/{w['id']}/exceptions/{eid}")
assert after.status_code == 200 and len(after.json()["exceptions"]) == 1
async with SessionLocal() as db:
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "maintenance_window"))).all()
assert any(e.action == "exception_skip" and e.details["reason"] == "Feiertag" for e in events)
assert any(e.status == "accepted" and e.action == "delete" for e in events)
@pytest.mark.asyncio
async def test_window_writes_follow_the_deploy_right(client):
from tests.test_operator_role import _operator
op = await _operator(client, "winop")
try:
made = await op.post("/api/maintenance-windows", json={**_WEEKLY, "name": "op-window"})
assert made.status_code == 201 # operators may prepare windows while they may deploy
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
for method, path, body in [
("POST", "/api/maintenance-windows", {**_WEEKLY, "name": "x"}),
("PUT", f"/api/maintenance-windows/{made.json()['id']}", _WEEKLY),
("DELETE", f"/api/maintenance-windows/{made.json()['id']}", None),
("POST", f"/api/maintenance-windows/{made.json()['id']}/exceptions", {"occurrence_date": "2030-01-06", "kind": "skip"}),
]:
resp = await op.request(method, path, json=body)
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN", path
assert (await op.get("/api/maintenance-windows")).status_code == 200
finally:
await op.aclose()
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
+50
View File
@@ -206,3 +206,53 @@ async def test_fresh_database_needs_no_backup(scratch, monkeypatch):
await migrate.run() # würde ohne Backup-Ziel scheitern, wenn eins verlangt würde
from pathlib import Path
assert not Path(settings.migration_backup_dir).exists()
async def _alembic(engine, action: str, revision: str) -> None:
async with engine.begin() as conn:
await conn.run_sync(lambda sync_conn: migrate._run_alembic(sync_conn, action, revision))
async def test_0020_upgrade_keeps_existing_settings_and_downgrade_is_clean(scratch):
"""1.23 -> 1.24 with a real settings row: the new alert columns get their defaults, the stored webhook
and flags stay, the channel table appears; a downgrade removes exactly that again and re-upgrading works."""
await _alembic(scratch, "upgrade", "0019")
async with scratch.begin() as conn:
# Every NOT NULL column of the 0019 table gets a value, so the row is what a real 1.23 install has.
required = (await conn.execute(text(
"SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'notification_settings' "
"AND is_nullable = 'NO' AND column_default IS NULL"
))).all()
by_type = {"boolean": "true", "integer": "300", "json": "'[]'", "timestamp with time zone": "now()"}
values = {name: by_type.get(kind, "'x'") for name, kind in required}
values.update({"id": "1", "webhook_enabled": "true", "webhook_url": "'https://hooks.example/x'"})
await conn.execute(text(
f"INSERT INTO notification_settings ({', '.join(values)}) VALUES ({', '.join(values.values())})"
))
await _alembic(scratch, "upgrade", "head")
async with scratch.connect() as conn:
row = (await conn.execute(text(
"SELECT webhook_enabled, webhook_url, gateway_down_enabled, resource_threshold_enabled, service_down_enabled, "
"disk_threshold_pct, load_per_cpu_threshold, alert_confirm_samples, resource_confirm_samples, service_watchlist "
"FROM notification_settings"
))).one()
assert tuple(row) == (True, "https://hooks.example/x", True, False, False, 90, 2.0, 3, 5, None)
await conn.execute(text(
"INSERT INTO notification_channels (id, name, channel_type, config, created_at, updated_at) "
"VALUES (gen_random_uuid(), 'n', 'ntfy', '{\"topic\": \"t\"}', now(), now())"
))
await conn.commit()
assert await _schema_diff(scratch) == []
await _alembic(scratch, "downgrade", "0019")
async with scratch.connect() as conn:
assert (await conn.execute(text("SELECT to_regclass('notification_channels')"))).scalar() is None
columns = set((await conn.execute(text(
"SELECT column_name FROM information_schema.columns WHERE table_name = 'notification_settings'"
))).scalars().all())
assert "gateway_down_enabled" not in columns and "service_watchlist" not in columns
assert (await conn.execute(text("SELECT webhook_url FROM notification_settings"))).scalar() == "https://hooks.example/x"
await _alembic(scratch, "upgrade", "head")
assert await _version(scratch) == _head()
assert await _schema_diff(scratch) == []
+717
View File
@@ -0,0 +1,717 @@
"""Monitoring alerts (1.24): gateway health, resource thresholds, stopped services, extra channels."""
import logging
import uuid
from datetime import datetime, timedelta, timezone
import httpx
import pytest
from sqlalchemy import select
from app.database import SessionLocal
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
from app.models.resource_sample import HostResourceSample
from app.services import (
gateway_health_service, monitor_checks, notification_channels, notification_secret_store, resource_monitor_service,
url_guard,
)
from app.services.notification_delivery import NotificationDeliveryError
from app.services.notification_service import run_notification_checks
from tests.test_notifications import _create_host
from tests.test_resource_monitor import _mock_resource_endpoints, _mock_vip
# What routes/gateway/status returns on a real 26.7 firewall (test-fw-3, live), plus a degraded and a dead gateway.
STATUS_PAYLOAD = {"status": "ok", "items": [
{"name": "WAN_GW", "address": "203.0.113.1", "status": "none", "loss": "0.0 %", "delay": "1.9 ms", "stddev": "0.3 ms", "monitor": "203.0.113.1"},
{"name": "LTE_GW", "address": "192.0.2.9", "status": "delay+loss", "loss": "12.5 %", "delay": "310.2 ms", "stddev": "40.0 ms", "monitor": "192.0.2.9"},
{"name": "DOWN_GW", "address": "192.0.2.1", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~", "monitor": "192.0.2.1"},
{"name": "WAN_DHCP6", "address": "~", "status": "none", "loss": "~", "delay": "~", "stddev": "~", "monitor": "~"},
{"name": "OFF_GW", "address": "192.0.2.5", "status": "force_down", "loss": "~", "delay": "~", "stddev": "~", "monitor": "~"},
]}
# ── Gateway parsing ──────────────────────────────────────────────────────────
def test_parse_gateway_status_maps_states_and_numbers():
by_name = {g.name: g for g in gateway_health_service.parse_gateway_status(STATUS_PAYLOAD)}
assert by_name["WAN_GW"].state == "online" and by_name["WAN_GW"].loss == 0.0 and by_name["WAN_GW"].delay == 1.9
assert by_name["WAN_GW"].stddev == 0.3
assert by_name["LTE_GW"].state == "degraded" and by_name["LTE_GW"].loss == 12.5
assert by_name["DOWN_GW"].state == "offline" and by_name["DOWN_GW"].delay is None
# nothing measured -> unknown, never "online" and never 0 ms
assert by_name["WAN_DHCP6"].state == "unknown" and by_name["WAN_DHCP6"].loss is None and by_name["WAN_DHCP6"].address is None
assert by_name["OFF_GW"].state == "forced_down"
def test_dead_gateway_has_no_round_trip_time():
# dpinger reports "100.0 %" loss together with "0.0 ms" when nothing answers: 0 ms must not look fast
payload = {"items": [{"name": "GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms", "monitor": "192.0.2.1"}]}
gw = gateway_health_service.parse_gateway_status(payload)[0]
assert gw.state == "offline" and gw.loss == 100.0 and gw.delay is None and gw.stddev is None
def test_parse_gateway_status_tolerates_failed_answer():
assert gateway_health_service.parse_gateway_status({"items": [], "status": "failed"}) == []
assert gateway_health_service.parse_gateway_status(None) == []
def test_is_bad_thresholds():
by_name = {g.name: g for g in gateway_health_service.parse_gateway_status(STATUS_PAYLOAD)}
assert gateway_health_service.is_bad(by_name["DOWN_GW"])
assert not gateway_health_service.is_bad(by_name["OFF_GW"]) # switched off on purpose
assert not gateway_health_service.is_bad(by_name["WAN_DHCP6"])
assert not gateway_health_service.is_bad(by_name["LTE_GW"]) # degraded by OPNsense' limits only, own thresholds off
assert gateway_health_service.is_bad(by_name["LTE_GW"], loss_threshold_pct=10)
assert gateway_health_service.is_bad(by_name["LTE_GW"], delay_threshold_ms=300)
assert not gateway_health_service.is_bad(by_name["WAN_GW"], loss_threshold_pct=10, delay_threshold_ms=300)
@pytest.mark.asyncio
async def test_get_gateway_health_reads_live(client, respx_mock):
_mock_vip(respx_mock)
host = await _create_host(client, "fw1", "https://fw1.test")
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(200, json=STATUS_PAYLOAD)
resp = await client.get(f"/api/hosts/{host['id']}/gateways")
assert resp.status_code == 200, resp.text
rows = {g["name"]: g for g in resp.json()}
assert rows["DOWN_GW"]["state"] == "offline" and rows["WAN_GW"]["delay"] == 1.9
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(500)
assert (await client.get(f"/api/hosts/{host['id']}/gateways")).status_code == 502
# ── Sampler stores the extra readings ────────────────────────────────────────────
@pytest.mark.asyncio
async def test_sampler_stores_gateways_services_swap_and_cpu(client, respx_mock):
_mock_vip(respx_mock)
await _create_host(client, "fw1", "https://fw1.test")
_mock_resource_endpoints(respx_mock)
respx_mock.get("https://fw1.test/api/diagnostics/system/systemSwap").respond(200, json={"swap": [{"device": "/dev/da0p3", "total": "8388608", "used": "2097152"}]})
respx_mock.get("https://fw1.test/api/diagnostics/cpu_usage/getCPUType").respond(200, json=["QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)"])
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(200, json=STATUS_PAYLOAD)
respx_mock.get("https://fw1.test/api/core/service/search").respond(200, json={"rows": [
{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
{"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0},
{"id": "pf", "name": "pf", "locked": 1, "running": 1},
{"id": "3f2a", "name": "openvpn", "locked": 0, "running": 0},
]})
async with SessionLocal() as db:
await resource_monitor_service.poll_all_hosts(db)
raw = (await db.scalars(select(HostResourceSample))).one().raw
assert raw["swap_used_pct"] == 25.0 and raw["cpu_threads"] == 6
assert {g["name"]: g["state"] for g in raw["gateways"]}["DOWN_GW"] == "offline"
assert raw["services"] == {"unbound": 1, "dhcpd6": 0, "openvpn/3f2a": 0} # locked services are not recorded
@pytest.mark.asyncio
async def test_sampler_survives_missing_optional_endpoints(client, respx_mock):
_mock_vip(respx_mock)
await _create_host(client, "fw1", "https://fw1.test")
_mock_resource_endpoints(respx_mock) # nothing for swap/cpu/gateways/services -> unmocked routes fail
async with SessionLocal() as db:
result = await resource_monitor_service.poll_all_hosts(db)
row = (await db.scalars(select(HostResourceSample))).one()
assert result["sampled"] == 1 and row.mem_used_pct == 25.0
assert row.raw["gateways"] is None and row.raw["services"] is None and row.raw["swap_used_pct"] is None
# ── Debouncing ───────────────────────────────────────────────────────────────
def test_verdict_needs_agreement_and_keeps_state_otherwise():
v = monitor_checks.verdict
assert v([True, True, True], 3, False) is True
assert v([True, True, False], 3, False) is False # no single spike / short blip
assert v([True, True], 3, False) is False # not enough history yet
assert v([False, False, False], 3, True) is False # needs the full window to close
assert v([False, True, False], 3, True) is True # flapping keeps an open alert open
assert v([True, None, True], 3, False) is False # unreadable sample is no evidence
assert v([None, None, None], 3, True) is True # ... and does not clear either
assert v([True], 1, False) is True
async def _add_samples(host_id, rows: list[dict], *, age_minutes: int = 0) -> None:
"""rows: newest first; each {mem, load, disk, raw...}. One sample per minute, the newest ``age_minutes`` old."""
now = datetime.now(timezone.utc)
async with SessionLocal() as db:
for i, row in enumerate(rows):
raw = {k: v for k, v in row.items() if k not in ("mem", "load")}
db.add(HostResourceSample(
host_id=host_id, sampled_at=now - timedelta(minutes=age_minutes + i),
mem_used_pct=row.get("mem"), load_avg_1m=row.get("load"), disk_used_pct=None, raw=raw,
))
await db.commit()
def _gw(state="online", loss=0.0, delay=2.0, name="WAN_GW", status=None):
return {"name": name, "status": status or {"offline": "down", "online": "none"}.get(state, state), "state": state,
"loss": loss, "delay": delay, "stddev": 0.1}
async def _set(**fields):
async with SessionLocal() as db:
row = await db.get(NotificationSettings, 1) or NotificationSettings(id=1)
for k, v in fields.items():
setattr(row, k, v)
db.add(row)
await db.commit()
_ALL_FLAGS = [
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled", "revert_failed_enabled", "maintenance_unknown_enabled",
"carp_health_enabled", "nat_drift_enabled", "network_drift_enabled", "carp_auto_revert_failed_enabled",
"scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled",
]
async def _only(*enabled: str):
"""Run run_notification_checks with just these checks on (the mocked firewalls are unreachable, which would
otherwise also open host_offline problems)."""
await _set(**{flag: flag in enabled for flag in _ALL_FLAGS})
async def _open_count(db, problem_type: str | None = None) -> int:
query = select(NotificationProblem).where(NotificationProblem.status == "open")
if problem_type:
query = query.where(NotificationProblem.problem_type == problem_type)
return len((await db.scalars(query)).all())
async def _gateway_problems():
async with SessionLocal() as db:
return await monitor_checks.check_gateway_down(db)
@pytest.mark.asyncio
async def test_gateway_down_needs_consecutive_samples(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
await _add_samples(hid, [{"gateways": [_gw("offline", loss=100.0, delay=None)]}, {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}])
assert dict(await _gateway_problems()) == {} # only two bad samples in a row
await _add_samples(hid, [{"gateways": [_gw("offline", 100.0, None)]}] * 3, age_minutes=-3) # three newer bad ones
problems = await _gateway_problems()
assert list(problems) == [f"{hid}:WAN_GW"]
ctx = problems[f"{hid}:WAN_GW"]
assert ctx["host_name"] == "fw1" and ctx["gateway"] == "WAN_GW" and ctx["grund"] == "Gateway offline"
@pytest.mark.asyncio
async def test_gateway_alert_resolves_only_after_full_recovery_and_ignores_unknown(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
bad, good = {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}
await _only("gateway_down_enabled")
await _add_samples(hid, [bad, bad, bad, good, good])
async with SessionLocal() as db:
await run_notification_checks(db)
assert await _open_count(db) == 1
# one good sample after the outage, then a relapse: still open (flapping link must not re-notify)
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [bad, good, bad, bad, good])
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"]
# a sample that could not read the gateways does not clear it either
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [{"gateways": None}] * 3)
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"]
# three good ones close it
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [good, good, good])
assert dict(await _gateway_problems()) == {}
@pytest.mark.asyncio
async def test_gateway_thresholds_and_forced_down(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
lossy = {"gateways": [_gw("degraded", loss=35.0, delay=20.0, status="loss"), _gw("forced_down", None, None, name="OFF", status="force_down")]}
await _add_samples(hid, [lossy] * 3)
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"] # 35 % >= 20 %; the switched-off gateway is not reported
await _set(gateway_loss_threshold_pct=0)
assert dict(await _gateway_problems()) == {} # own loss threshold off; OPNsense says "degraded", not "down"
@pytest.mark.asyncio
async def test_stale_samples_are_not_judged_and_keep_open_problems(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
bad = {"gateways": [_gw("offline", 100.0, None)]}
await _only("gateway_down_enabled")
await _add_samples(hid, [bad] * 3)
async with SessionLocal() as db:
await run_notification_checks(db)
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [{"gateways": [_gw()]}] * 3, age_minutes=30) # the host stopped answering 30 min ago
async with SessionLocal() as db:
result = await run_notification_checks(db)
assert result["resolved"] == 0
assert await _open_count(db) == 1
@pytest.mark.asyncio
async def test_gateway_alert_is_delivered_once_and_resolution_follows(client, respx_mock):
_mock_vip(respx_mock)
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
await _only("gateway_down_enabled")
await _set(webhook_enabled=True, webhook_url="https://hooks.test/n")
hook = respx_mock.post("https://hooks.test/n").mock(return_value=httpx.Response(200))
bad, good = {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}
await _add_samples(hid, [bad] * 3)
async with SessionLocal() as db:
assert (await run_notification_checks(db))["opened"] == 1
async with SessionLocal() as db:
assert (await run_notification_checks(db))["opened"] == 0
assert hook.call_count == 1 and b"Gateway" in hook.calls.last.request.content
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [good] * 3)
async with SessionLocal() as db:
assert (await run_notification_checks(db))["resolved"] == 1
assert hook.call_count == 2 and b"behoben" in hook.calls.last.request.content
@pytest.mark.asyncio
async def test_gateway_check_is_on_by_default_and_the_others_are_off(client):
data = (await client.get("/api/notifications/settings")).json()
assert data["gateway_down_enabled"] is True
assert data["resource_threshold_enabled"] is False and data["service_down_enabled"] is False
assert data["service_watchlist"][:2] == ["unbound", "dnsmasq"]
# ── Resource thresholds ──────────────────────────────────────────────────────────
DISK_OK = [{"device": "zroot/ROOT/default", "type": "zfs", "used_pct": 40, "mountpoint": "/"}, {"device": "devfs", "type": "devfs", "used_pct": 100, "mountpoint": "/dev"}]
DISK_FULL = [{"device": "zroot/ROOT/default", "type": "zfs", "used_pct": 93, "mountpoint": "/"},
{"device": "zroot/var/log", "type": "zfs", "used_pct": 91, "mountpoint": "/var/log"},
{"device": "devfs", "type": "devfs", "used_pct": 100, "mountpoint": "/dev"}]
async def _resource_problems():
async with SessionLocal() as db:
return await monitor_checks.check_resource_threshold(db)
@pytest.mark.asyncio
async def test_disk_threshold_one_alert_per_pool_and_ignores_devfs(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
await _add_samples(hid, [{"disk": DISK_FULL, "mem": 10, "load": 0.1}] * 5)
problems = await _resource_problems()
assert list(problems) == [f"{hid}:disk:zroot"]
assert problems[f"{hid}:disk:zroot"]["wert_prozent"] == 93 and problems[f"{hid}:disk:zroot"]["schwelle_prozent"] == 90
@pytest.mark.asyncio
async def test_memory_spike_does_not_alert_but_sustained_use_does(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
calm = {"disk": DISK_OK, "mem": 40, "load": 0.1}
await _add_samples(hid, [{"disk": DISK_OK, "mem": 99, "load": 0.1}, calm, calm, calm, calm])
assert dict(await _resource_problems()) == {}
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [{"disk": DISK_OK, "mem": 96, "load": 0.1}] * 5)
assert list(await _resource_problems()) == [f"{hid}:memory"]
await _set(memory_threshold_pct=0) # 0 = off
assert dict(await _resource_problems()) == {}
@pytest.mark.asyncio
async def test_swap_and_cpu_per_thread(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
# load 6 on 2 threads = 3 per thread (>= 2.0); load 6 on 8 threads would be fine
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 6.0, "swap_used_pct": 70.0, "cpu_threads": 2}] * 5)
assert sorted(await _resource_problems()) == sorted([f"{hid}:cpu", f"{hid}:swap"])
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 6.0, "swap_used_pct": None, "cpu_threads": 8}] * 5)
assert dict(await _resource_problems()) == {}
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
# unknown thread count: the load cannot be judged, so it is not reported
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 50.0, "swap_used_pct": None, "cpu_threads": None}] * 5)
assert dict(await _resource_problems()) == {}
# ── Services ─────────────────────────────────────────────────────────────────────
async def _service_problems():
async with SessionLocal() as db:
return await monitor_checks.check_service_down(db)
@pytest.mark.asyncio
async def test_service_down_only_after_it_was_seen_running(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
up = {"services": {"unbound": 1, "dhcpd6": 0, "cron": 0}}
down = {"services": {"unbound": 0, "dhcpd6": 0, "cron": 0}}
await _add_samples(hid, [down, down, down, up, up])
problems = await _service_problems()
# unbound stopped; dhcpd6 never ran; cron is not on the watchlist
assert list(problems) == [f"{hid}:unbound"]
assert problems[f"{hid}:unbound"]["dienst"] == "unbound"
@pytest.mark.asyncio
async def test_service_that_never_ran_or_stopped_long_ago_is_not_reported(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
await _add_samples(hid, [{"services": {"unbound": 0}}] * 6)
assert dict(await _service_problems()) == {}
@pytest.mark.asyncio
async def test_service_alert_resolves_when_it_runs_again_or_leaves_the_watchlist(client):
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
up, down = {"services": {"unbound": 1}}, {"services": {"unbound": 0}}
await _only("service_down_enabled")
await _add_samples(hid, [down, down, down, up])
async with SessionLocal() as db:
assert (await run_notification_checks(db))["opened"] == 1
await _set(service_watchlist=["ntpd"])
async with SessionLocal() as db:
assert (await run_notification_checks(db))["resolved"] == 1
await _set(service_watchlist=["unbound"])
async with SessionLocal() as db:
assert (await run_notification_checks(db))["opened"] == 1 # reopens (the row is reused)
async with SessionLocal() as db:
await db.execute(HostResourceSample.__table__.delete())
await db.commit()
await _add_samples(hid, [up, up, up, down])
async with SessionLocal() as db:
assert (await run_notification_checks(db))["resolved"] == 1
@pytest.mark.asyncio
async def test_service_watchlist_validation(client):
ok = await client.put("/api/notifications/settings", json={"service_watchlist": [" unbound ", "openvpn", "unbound"]})
assert ok.status_code == 200 and ok.json()["service_watchlist"] == ["unbound", "openvpn"]
assert (await client.put("/api/notifications/settings", json={"service_watchlist": ["bad name!"]})).status_code == 422
assert (await client.put("/api/notifications/settings", json={"disk_threshold_pct": 101})).status_code == 422
assert (await client.put("/api/notifications/settings", json={"alert_confirm_samples": 0})).status_code == 422
# ── Metrics and dashboard ─────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_metrics_and_dashboard_show_gateways_and_services(client):
host = await _create_host(client, "fw1", "https://fw1.test", cluster_name=None)
hid = uuid.UUID(host["id"])
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 0.1, "swap_used_pct": 5.0,
"gateways": [_gw(), _gw("offline", 100.0, None, name="DOWN_GW")],
"services": {"unbound": 1, "openvpn/3f2a": 0, "cron": 0}}])
text = (await client.get("/api/metrics")).text
assert 'opnexus_host_gateway_up{host="fw1",cluster="",gateway="WAN_GW"} 1' in text
assert 'opnexus_host_gateway_up{host="fw1",cluster="",gateway="DOWN_GW"} 0' in text
assert 'opnexus_host_gateway_delay_seconds{host="fw1",cluster="",gateway="WAN_GW"} 0.002' in text
assert 'opnexus_host_gateway_loss_percent{host="fw1",cluster="",gateway="DOWN_GW"} 100' in text
assert 'opnexus_host_swap_used_percent{host="fw1",cluster=""} 5' in text
assert 'opnexus_host_service_running{host="fw1",cluster="",service="unbound"} 1' in text
assert 'service="openvpn",instance="3f2a"} 0' in text
assert 'service="cron"' not in text # not on the watchlist
summary = (await client.get("/api/dashboard")).json()
gateways = {g["name"]: g for g in summary["hosts"][0]["gateways"]}
assert gateways["DOWN_GW"]["state"] == "offline"
@pytest.mark.asyncio
async def test_dashboard_has_no_gateways_without_a_fresh_sample(client):
host = await _create_host(client, "fw1", "https://fw1.test")
await _add_samples(uuid.UUID(host["id"]), [{"gateways": [_gw()]}], age_minutes=20)
assert (await client.get("/api/dashboard")).json()["hosts"][0]["gateways"] is None
# ── URL guard ────────────────────────────────────────────────────────────────────
@pytest.mark.parametrize("url", [
"http://169.254.169.254/latest/meta-data", "http://127.0.0.1:8000/x", "http://localhost/x", "http://[::1]/x",
"http://0.0.0.0/x", "http://[::ffff:169.254.169.254]/x", "ftp://ntfy.example/x", "file:///etc/passwd",
"https://user:[email protected]/", "https://", "not a url", "http://224.0.0.1/x", "http://[fe80::1]/x",
])
def test_url_guard_refuses(url):
with pytest.raises(ValueError):
url_guard.validate_url(url)
@pytest.mark.parametrize("url", ["https://ntfy.sh", "http://10.20.30.5:8080/path", "https://gotify.lan.example/", "http://192.168.1.10"])
def test_url_guard_allows_private_and_public_targets(url):
assert url_guard.validate_url(url) == url
@pytest.mark.asyncio
async def test_url_guard_refuses_names_resolving_to_metadata_address(monkeypatch):
async def fake(self, host, port, **kw):
return [(2, 1, 6, "", ("169.254.169.254", 0))]
monkeypatch.setattr("asyncio.base_events.BaseEventLoop.getaddrinfo", fake)
with pytest.raises(ValueError):
await url_guard.ensure_resolved_safe("https://innocent.example/hook")
@pytest.mark.asyncio
async def test_webhook_url_is_validated_when_saved_and_sent(client, respx_mock):
resp = await client.put("/api/notifications/settings", json={"webhook_url": "http://169.254.169.254/x"})
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "NOTIFICATION_WEBHOOK_URL_INVALID"
row = NotificationSettings(webhook_url="http://127.0.0.1:9/x")
from app.services import notification_delivery
with pytest.raises(NotificationDeliveryError):
await notification_delivery.send_webhook(row, NotificationProblem(problem_type="test", dedup_key="t", context={}), "opened")
# ── Channels: formats ────────────────────────────────────────────────────────────
PROBLEM = NotificationProblem(problem_type="gateway_down", dedup_key="k", context={"host_name": "fw1", "gateway": "WAN_GW", "grund": "Gateway offline"})
def _channel(kind: str, config: dict, secret: str | None, name: str = "c1") -> NotificationChannel:
ch = NotificationChannel(id=uuid.uuid4(), name=name, channel_type=kind, enabled=True, config=config, secret_set=secret is not None)
if secret is not None:
notification_secret_store.save_secret(notification_channels.secret_name(ch.id), secret)
return ch
@pytest.mark.asyncio
async def test_ntfy_posts_json_to_root_with_bearer(respx_mock):
ch = _channel("ntfy", {"server_url": "https://ntfy.test", "topic": "alerts", "priority": 5}, "tk_secret")
route = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200, json={"id": "x"}))
await notification_channels.send(ch, PROBLEM, "opened")
req = route.calls.last.request
assert req.headers["authorization"] == "Bearer tk_secret"
body = httpx.Response(200, content=req.content).json()
assert body["topic"] == "alerts" and body["priority"] == 5 and body["tags"] == ["warning"]
assert body["title"] == "OPNexus: Gateway gestört" and "gateway: WAN_GW" in body["message"]
await notification_channels.send(ch, PROBLEM, "resolved")
resolved = httpx.Response(200, content=route.calls.last.request.content).json()
assert resolved["priority"] == 3 and resolved["title"].endswith("behoben") and resolved["tags"] == ["white_check_mark"]
@pytest.mark.asyncio
async def test_ntfy_without_token_sends_no_auth_header(respx_mock):
ch = _channel("ntfy", {"server_url": "https://ntfy.test", "topic": "alerts"}, None)
route = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200))
await notification_channels.send(ch, PROBLEM, "opened")
assert "authorization" not in route.calls.last.request.headers
@pytest.mark.asyncio
async def test_gotify_uses_key_header_not_query(respx_mock):
ch = _channel("gotify", {"server_url": "https://gotify.test/", "priority": 8}, "AppTok123")
route = respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(200, json={"id": 1}))
await notification_channels.send(ch, PROBLEM, "opened")
req = route.calls.last.request
assert req.headers["x-gotify-key"] == "AppTok123" and "AppTok123" not in str(req.url)
body = httpx.Response(200, content=req.content).json()
assert body["priority"] == 8 and body["title"] == "OPNexus: Gateway gestört"
@pytest.mark.asyncio
async def test_telegram_sendmessage_and_ok_flag(respx_mock):
ch = _channel("telegram", {"chat_id": "-1001234", "api_base": "https://api.telegram.test"}, "123456:ABC-token")
route = respx_mock.post("https://api.telegram.test/bot123456:ABC-token/sendMessage").mock(
return_value=httpx.Response(200, json={"ok": True, "result": {}}))
await notification_channels.send(ch, PROBLEM, "opened")
body = httpx.Response(200, content=route.calls.last.request.content).json()
assert body["chat_id"] == "-1001234" and body["text"].startswith("OPNexus: Gateway gestört\n\n") and len(body["text"]) <= 4096
route.mock(return_value=httpx.Response(200, json={"ok": False, "description": "chat not found"}))
with pytest.raises(NotificationDeliveryError):
await notification_channels.send(ch, PROBLEM, "opened")
def test_telegram_text_is_clipped_to_the_api_limit():
big = NotificationProblem(problem_type="gateway_down", dedup_key="k", context={"x": "y" * 9000})
text = notification_channels.format_telegram({"chat_id": "1"}, big, "opened")["text"]
assert len(text) == 4096 and text.endswith("…")
@pytest.mark.asyncio
async def test_teams_adaptive_card_envelope(respx_mock):
url = "https://prod-12.westeurope.logic.azure.test/workflows/abc/triggers/manual/paths/invoke?sig=SECRETSIG"
ch = _channel("teams", {}, url)
route = respx_mock.post(url.split("?")[0]).mock(return_value=httpx.Response(202))
await notification_channels.send(ch, PROBLEM, "opened")
body = httpx.Response(200, content=route.calls.last.request.content).json()
assert body["type"] == "message"
att = body["attachments"][0]
assert att["contentType"] == "application/vnd.microsoft.card.adaptive" and att["content"]["type"] == "AdaptiveCard"
assert att["content"]["body"][0]["text"] == "OPNexus: Gateway gestört"
assert "sig=SECRETSIG" in str(route.calls.last.request.url)
@pytest.mark.asyncio
async def test_matrix_put_with_transaction_id_and_encoded_room(respx_mock):
ch = _channel("matrix", {"homeserver_url": "https://matrix.test", "room_id": "!abc123:matrix.test"}, "syt_token")
route = respx_mock.put(url__regex=r"^https://matrix\.test/_matrix/client/v3/rooms/%21abc123%3Amatrix\.test/send/m\.room\.message/[0-9a-f]{32}$").mock(
return_value=httpx.Response(200, json={"event_id": "$e"}))
await notification_channels.send(ch, PROBLEM, "opened")
await notification_channels.send(ch, PROBLEM, "opened")
first, second = (c.request for c in route.calls)
assert first.headers["authorization"] == "Bearer syt_token"
assert first.url.path != second.url.path # a new transaction id per message
body = httpx.Response(200, content=first.content).json()
assert body["msgtype"] == "m.text" and body["body"].startswith("OPNexus: Gateway gestört")
@pytest.mark.asyncio
async def test_failures_and_logs_never_contain_the_credential(respx_mock, caplog):
token = "123456:SECRET-BOT-TOKEN"
ch = _channel("telegram", {"chat_id": "42", "api_base": "https://api.telegram.test"}, token)
respx_mock.post(f"https://api.telegram.test/bot{token}/sendMessage").mock(return_value=httpx.Response(401, json={"ok": False}))
with caplog.at_level(logging.DEBUG):
with pytest.raises(NotificationDeliveryError) as exc_info:
await notification_channels.send(ch, PROBLEM, "opened")
respx_mock.post(f"https://api.telegram.test/bot{token}/sendMessage").mock(side_effect=httpx.ConnectError("boom"))
with pytest.raises(NotificationDeliveryError) as exc_info2:
await notification_channels.send(ch, PROBLEM, "opened")
assert "HTTP 401" in str(exc_info.value) and "ConnectError" in str(exc_info2.value)
for text in (str(exc_info.value), str(exc_info2.value), caplog.text):
assert "SECRET-BOT-TOKEN" not in text
assert exc_info.value.__cause__ is None and exc_info.value.__suppress_context__
@pytest.mark.asyncio
async def test_missing_or_unreadable_secret_fails_closed(respx_mock):
ch = _channel("gotify", {"server_url": "https://gotify.test"}, "tok")
notification_secret_store.delete_secret(notification_channels.secret_name(ch.id))
with pytest.raises(NotificationDeliveryError, match="nicht lesbar"):
await notification_channels.send(ch, PROBLEM, "opened")
ch.secret_set = False
with pytest.raises(NotificationDeliveryError, match="fehlen"):
await notification_channels.send(ch, PROBLEM, "opened")
@pytest.mark.asyncio
async def test_redirects_are_not_followed(respx_mock):
ch = _channel("gotify", {"server_url": "https://gotify.test"}, "tok")
respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(302, headers={"location": "http://169.254.169.254/"}))
with pytest.raises(NotificationDeliveryError, match="HTTP 302"):
await notification_channels.send(ch, PROBLEM, "opened")
# ── Channels: API ────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_channel_crud_keeps_the_secret_out_of_database_and_responses(client, respx_mock):
resp = await client.post("/api/notifications/channels", json={
"name": "Telegram Ops", "channel_type": "telegram", "config": {"chat_id": "-100123"}, "secret": "111:TOKEN-VALUE",
})
assert resp.status_code == 201, resp.text
created = resp.json()
assert created["secret_set"] is True and "TOKEN-VALUE" not in resp.text
assert created["config"] == {"chat_id": "-100123", "api_base": "https://api.telegram.org"}
cid = created["id"]
assert notification_secret_store.read_secret(f"channel_{cid}") == "111:TOKEN-VALUE"
async with SessionLocal() as db:
row = await db.get(NotificationChannel, uuid.UUID(cid))
assert "TOKEN-VALUE" not in str(row.config) and "TOKEN-VALUE" not in repr(row.__dict__)
assert "TOKEN-VALUE" not in (await client.get("/api/notifications/channels")).text
# test button
route = respx_mock.post("https://api.telegram.org/bot111:TOKEN-VALUE/sendMessage").mock(return_value=httpx.Response(200, json={"ok": True}))
assert (await client.post(f"/api/notifications/channels/{cid}/test")).json() == {"success": True}
assert b"Testbenachrichtigung" in route.calls.last.request.content
route.mock(return_value=httpx.Response(500))
failed = await client.post(f"/api/notifications/channels/{cid}/test")
assert failed.status_code == 502 and failed.json()["detail"]["code"] == "NOTIFICATION_CHANNEL_TEST_FAILED"
assert "TOKEN-VALUE" not in failed.text
# update: rename, disable, rotate secret, clear secret
upd = await client.put(f"/api/notifications/channels/{cid}", json={"name": "Telegram", "enabled": False, "secret": "222:NEW"})
assert upd.status_code == 200 and upd.json()["enabled"] is False
assert notification_secret_store.read_secret(f"channel_{cid}") == "222:NEW"
assert (await client.put(f"/api/notifications/channels/{cid}", json={"clear_secret": True})).json()["secret_set"] is False
assert notification_secret_store.read_secret(f"channel_{cid}") is None
assert (await client.delete(f"/api/notifications/channels/{cid}")).status_code == 204
assert (await client.get("/api/notifications/channels")).json() == []
assert (await client.delete(f"/api/notifications/channels/{cid}")).status_code == 404
@pytest.mark.asyncio
@pytest.mark.parametrize("payload", [
{"name": "a", "channel_type": "ntfy", "config": {"topic": "bad topic!"}},
{"name": "a", "channel_type": "ntfy", "config": {"topic": "ok", "server_url": "http://169.254.169.254"}},
{"name": "a", "channel_type": "ntfy", "config": {"topic": "ok", "priority": 9}},
{"name": "a", "channel_type": "gotify", "config": {"server_url": "https://g.test"}}, # token missing
{"name": "a", "channel_type": "gotify", "config": {}, "secret": "t"}, # server missing
{"name": "a", "channel_type": "telegram", "config": {"chat_id": "abc"}, "secret": "t"},
{"name": "a", "channel_type": "teams", "config": {}, "secret": "http://insecure.test/hook"},
{"name": "a", "channel_type": "teams", "config": {}},
{"name": "a", "channel_type": "matrix", "config": {"homeserver_url": "https://m.test", "room_id": "#alias:m.test"}, "secret": "t"},
{"name": "a", "channel_type": "carrier-pigeon", "config": {}},
{"name": " ", "channel_type": "ntfy", "config": {"topic": "ok"}}, # name only blanks
])
async def test_channel_validation(client, payload):
resp = await client.post("/api/notifications/channels", json=payload)
assert resp.status_code == 422, resp.text
@pytest.mark.asyncio
async def test_channel_names_are_unique_and_ntfy_needs_no_secret(client):
body = {"name": "Phone", "channel_type": "ntfy", "config": {"topic": "opnexus-alerts"}}
first = await client.post("/api/notifications/channels", json=body)
assert first.status_code == 201 and first.json()["secret_set"] is False
assert first.json()["config"]["server_url"] == "https://ntfy.sh" and first.json()["config"]["priority"] == 4
dup = await client.post("/api/notifications/channels", json=body)
assert dup.status_code == 409 and dup.json()["detail"]["code"] == "NOTIFICATION_CHANNEL_NAME_TAKEN"
@pytest.mark.asyncio
async def test_operator_cannot_change_channels_and_does_not_see_their_settings(client):
from tests.test_operator_role import PW, _operator
created = await client.post("/api/notifications/channels", json={"name": "Phone", "channel_type": "ntfy", "config": {"topic": "secret-topic"}})
op = await _operator(client, "chan-op")
try:
for method, path, body in [
("POST", "/api/notifications/channels", {"name": "x", "channel_type": "ntfy", "config": {"topic": "t"}}),
("PUT", f"/api/notifications/channels/{created.json()['id']}", {"enabled": False}),
("DELETE", f"/api/notifications/channels/{created.json()['id']}", None),
("POST", f"/api/notifications/channels/{created.json()['id']}/test", {}),
]:
resp = await op.request(method, path, json=body) if body is not None else await op.request(method, path)
assert resp.status_code == 403, (method, path, resp.text)
listing = await op.get("/api/notifications/channels")
assert listing.status_code == 200 and listing.json()[0]["config"] == {} and "secret-topic" not in listing.text
finally:
await op.aclose()
@pytest.mark.asyncio
async def test_alerts_reach_every_enabled_channel_and_one_failure_does_not_block_the_rest(client, respx_mock):
_mock_vip(respx_mock)
host = await _create_host(client, "fw1", "https://fw1.test")
hid = uuid.UUID(host["id"])
async with SessionLocal() as db:
for kind, cfg, secret, name in [
("gotify", {"server_url": "https://gotify.test"}, "g-tok", "A gotify"),
("ntfy", {"server_url": "https://ntfy.test", "topic": "t"}, None, "B ntfy"),
]:
ch = _channel(kind, cfg, secret, name)
db.add(ch)
off = _channel("ntfy", {"server_url": "https://off.test", "topic": "t"}, None, "C disabled")
off.enabled = False
db.add(off)
await db.commit()
await _only("gateway_down_enabled")
gotify = respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(500))
ntfy = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200))
await _add_samples(hid, [{"gateways": [_gw("offline", 100.0, None)]}] * 3)
async with SessionLocal() as db:
assert (await run_notification_checks(db))["opened"] == 1
assert gotify.call_count == 1 and ntfy.call_count == 1
assert "off.test" not in str([c.request.url for c in respx_mock.calls])
+452
View File
@@ -0,0 +1,452 @@
"""Taking over aliases, port forwards, 1:1 NAT and outbound NAT that were created by hand on a firewall.
The live rows below have the shape recorded from OPNsense 26.7 (test-fw-3, 2026-10-20), see the module
docstring of app/services/object_adopt_service.py."""
import json
import httpx
import pytest
from sqlalchemy import select
from app.database import SessionLocal
from app.models.audit import AuditEvent
from tests.helpers import deploy_approved
from tests.test_api_tokens import _login
BASE = "https://fw-adopt-obj.test"
API = f"{BASE}/api/firewall"
async def _host(client, respx_mock, name="fw-obj"):
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
r = await client.post("/api/hosts", json={"name": name, "base_url": BASE, "api_key": "k", "api_secret": "s"})
assert r.status_code == 201, r.text
return r.json()["id"]
def _search(respx_mock, path, rows):
return respx_mock.get(f"{API}/{path}").mock(return_value=httpx.Response(200, json={"rows": rows, "total": len(rows)}))
def _mode(respx_mock, mode="hybrid"):
respx_mock.get(f"{API}/source_nat/get").mock(return_value=httpx.Response(
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == mode else 0} for m in
("automatic", "hybrid", "advanced", "disabled")}}}}))
def _writes(respx_mock):
return [c for c in respx_mock.calls if c.request.method != "GET"]
# ── live rows (shapes captured from OPNsense 26.7) ──────────────────────
def alias_row(name, type_="host", content="10.9.9.1\n10.9.9.2", **over):
row = {"uuid": f"uuid-{name}", "enabled": "1", "name": name, "type": type_, "%type": type_.title(),
"content": content, "description": f"{name} desc", "proto": "", "interface": "", "counters": "0",
"updatefreq": "", "categories": ""}
row.update(over)
return row
SYSTEM_ALIASES = [
alias_row("bogons", "external", "", description="bogon networks (internal)"),
alias_row("sshlockout", "external", "", description="abuse lockout table (internal)"),
alias_row("__lan_network", "internal", "", description="lan net"),
]
def pf_row(**over):
row = {"uuid": "pf-1", "ipprotocol": "inet", "protocol": "tcp", "%protocol": "TCP", "disabled": "0", "nordr": "0",
"interface": "wan", "%interface": "WAN", "source.network": "any", "source.port": "", "source.not": "0",
"destination.network": "wanip", "destination.port": "8443", "destination.not": "0", "target": "192.168.1.50",
"local-port": "443", "natreflection": "", "descr": "Web", "pass": "", "log": "0", "poolopts": "", "tag": "",
"tagged": "", "nosync": "0", "categories": "", "is_automatic": False}
row.update(over)
return row
def o2o_row(**over):
row = {"uuid": "o2o-1", "enabled": "1", "interface": "wan", "%interface": "WAN", "type": "binat",
"source_net": "192.168.1.10", "source_not": "0", "destination_net": "any", "destination_not": "0",
"external": "203.0.113.201", "natreflection": "", "log": "0", "description": "Mail 1:1", "categories": ""}
row.update(over)
return row
def snat_row(**over):
row = {"uuid": "snat-1", "enabled": "1", "interface": "wan", "%interface": "WAN", "ipprotocol": "inet",
"protocol": "any", "source_net": "192.168.1.0/24", "source_not": "0", "source_port": "",
"destination_net": "any", "destination_not": "0", "destination_port": "", "target": "", "target_port": "",
"nonat": "0", "nosync": "0", "staticnatport": "0", "log": "0", "tag": "", "tagged": "",
"description": "LAN out", "categories": ""}
row.update(over)
return row
AUTOMATIC_NAT = [
{"uuid": "lockout_0", "interface": "lan", "descr": "Anti-Lockout Rule", "is_automatic": True},
{"uuid": "automatic_wan", "interface": "wan", "description": "Auto created rule", "is_automatic": True},
]
# kind -> (url prefix, search path, row factory, id field of the preview)
NAT_KINDS = {
"port-forward": ("d_nat/searchRule", pf_row, "Web"),
"one-to-one": ("one_to_one/searchRule", o2o_row, "Mail 1:1"),
"outbound-nat": ("source_nat/searchRule", snat_row, "LAN out"),
}
# ── aliases ─────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_alias_candidates_mark_system_unsupported_and_nested(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
alias_row("ok_hosts"),
alias_row("ok_nets", "network", "10.0.0.0/8\n192.168.0.0/16"),
alias_row("ok_ports", "port", "8080\n9000:9010"),
alias_row("ok_group", "networkgroup", "ok_nets\n__lan_network"),
alias_row("a_url", "url", "https://example.org/list.txt", updatefreq="1"),
alias_row("geo", "geoip", "DE", proto="IPv4"),
alias_row("uses_url", "host", "a_url"),
alias_row("uses_hosts", "host", "ok_hosts\n10.9.9.3"),
alias_row("asn_v4", "asn", "64496", proto="IPv4"),
alias_row("bad name", "host", "10.1.1.1"),
alias_row("off", "host", "10.9.9.9", enabled="0"),
])
resp = await client.get(f"/api/aliases/unmanaged?host_id={host_id}")
assert resp.status_code == 200, resp.text
by = {r["title"]: r for r in resp.json()}
for name in ("bogons", "sshlockout", "__lan_network"):
assert by[name]["reasons"] == ["system"] and not by[name]["adoptable"]
for name in ("ok_hosts", "ok_nets", "ok_ports", "off"):
assert by[name]["adoptable"] is True and by[name]["reasons"] == []
assert by["off"]["enabled"] is False
# __lan_network is OPNsense's own and fine; ok_nets is a live alias that is not managed yet
assert by["ok_group"]["reasons"] == ["references_unmanaged_alias"] and by["ok_group"]["bulk_adoptable"] is True
assert by["ok_group"]["reason_details"]["references_unmanaged_alias"] == "ok_nets"
assert by["a_url"]["reasons"] == ["unsupported_type"] and by["geo"]["reasons"] == ["unsupported_type"]
assert by["a_url"]["reason_details"]["unsupported_type"]
assert by["uses_url"]["reasons"] == ["references_unmanaged_alias"]
assert by["uses_url"]["reason_details"]["references_unmanaged_alias"] == "a_url"
assert by["uses_url"]["bulk_adoptable"] is False # its target can never be adopted
assert by["asn_v4"]["reasons"] == ["proto_set"]
assert by["bad name"]["reasons"] == ["name_invalid"]
@pytest.mark.asyncio
async def test_nested_alias_waits_for_its_target_but_bulk_orders_them(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "alias/searchItem", [
alias_row("outer", "host", "inner\n10.0.0.9"),
alias_row("inner", "host", "base"),
alias_row("base", "host", "10.0.0.1"),
])
by = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()}
assert by["base"]["adoptable"] is True
for name, target in (("inner", "base"), ("outer", "inner")):
assert by[name]["adoptable"] is False and by[name]["bulk_adoptable"] is True
assert by[name]["reason_details"]["references_unmanaged_alias"] == target
# alone it is refused ...
refused = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-outer"})
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "OBJECT_NOT_ADOPTABLE"
assert "references_unmanaged_alias" in refused.json()["detail"]["params"]["reasons"]
# ... after its target is managed it works
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-base"})).status_code == 201
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-inner"})).status_code == 201
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-outer"})).status_code == 201
@pytest.mark.asyncio
async def test_adopt_alias_records_it_without_writing_and_preview_is_unchanged(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
alias_row("web_servers", "host", "10.0.0.1\n 10.0.0.2 \n\nexample.org", description="Web tier"),
])
resp = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})
assert resp.status_code == 201, resp.text
alias = resp.json()
assert (alias["name"], alias["type"], alias["description"], alias["enabled"]) == ("web_servers", "host", "Web tier", True)
assert alias["content"] == "10.0.0.1\n10.0.0.2\nexample.org"
assert alias["target_host_ids"] == [host_id]
assert [d["status"] for d in alias["deployments"]] == ["success"]
assert alias["workflow_state"] == "synced"
assert not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
preview = (await client.get(f"/api/aliases/{alias['id']}/preview")).json()
assert [(h["action"], h["changes"]) for h in preview["hosts"]] == [("unchanged", [])]
assert preview["can_deploy"] is False
# no longer offered; adopting again is refused
assert [r["title"] for r in (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()
if r["reasons"] != ["system"]] == []
again = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})
assert again.status_code == 409 and "already_managed" in again.json()["detail"]["params"]["reasons"]
# audit: the deployment record (kind alias, remote_uuid) is written with the adoption
async with SessionLocal() as db:
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "alias"))).all()
assert any(e.entity_name == "web_servers" and e.details["remote_uuid"] == "uuid-web_servers"
and e.message == "Von der Firewall übernommen" for e in events)
@pytest.mark.asyncio
async def test_edit_after_alias_adopt_updates_the_same_remote_object(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "alias/searchItem", [alias_row("web_servers", "host", "10.0.0.1")])
alias = (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})).json()
edit = {"name": "web_servers", "type": "host", "content": "10.0.0.1\n10.0.0.2", "description": "web_servers desc",
"enabled": True, "target_host_ids": [host_id], "target_group_ids": [], "expected_updated_at": alias["updated_at"]}
assert (await client.put(f"/api/aliases/{alias['id']}", json=edit)).status_code == 200
add = respx_mock.post(f"{API}/alias/addItem").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
setter = respx_mock.post(f"{API}/alias/setItem/uuid-web_servers").mock(return_value=httpx.Response(200, json={"result": "saved"}))
respx_mock.post(f"{API}/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
resp = await deploy_approved(client, f"/api/aliases/{alias['id']}/deploy")
assert resp.status_code == 200, resp.text
assert setter.called and not add.called # no duplicate
assert json.loads(setter.calls.last.request.content)["alias"]["content"] == "10.0.0.1\n10.0.0.2"
confirm = await client.post(f"/api/aliases/{alias['id']}/confirm")
assert [d["status"] for d in confirm.json()["deployments"]] == ["success"]
@pytest.mark.asyncio
async def test_alias_name_that_exists_centrally_is_refused(client, respx_mock):
host_id = await _host(client, respx_mock)
assert (await client.post("/api/aliases", json={"name": "dup", "type": "host", "content": "1.1.1.1"})).status_code == 201
_search(respx_mock, "alias/searchItem", [alias_row("dup", "host", "2.2.2.2")])
row = (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()[0]
assert row["reasons"] == ["name_exists"]
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-dup"})).status_code == 409
@pytest.mark.asyncio
async def test_adopt_all_aliases_orders_dependencies_and_reports_skipped(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
alias_row("outer", "host", "inner"),
alias_row("inner", "host", "10.0.0.1"),
alias_row("a_url", "url", "https://example.org/x"),
alias_row("uses_url", "host", "a_url"),
alias_row("plain", "port", "443"),
])
resp = await client.post("/api/aliases/adopt-all", json={"host_id": host_id})
assert resp.status_code == 200, resp.text
body = resp.json()
assert sorted(a["title"] for a in body["adopted"]) == ["inner", "outer", "plain"]
skipped = {s["title"]: s for s in body["skipped"]}
assert set(skipped) == {"a_url", "uses_url"} # system aliases are not reported
assert skipped["a_url"]["reasons"] == ["unsupported_type"]
assert skipped["uses_url"]["reasons"] == ["references_unmanaged_alias"]
aliases = {a["name"]: a for a in (await client.get("/api/aliases")).json()}
assert set(aliases) == {"inner", "outer", "plain"}
assert all([d["status"] for d in a["deployments"]] == ["success"] and a["workflow_state"] == "synced" for a in aliases.values())
# a second run finds nothing left to take over
again = (await client.post("/api/aliases/adopt-all", json={"host_id": host_id})).json()
assert again["adopted"] == [] and {s["title"] for s in again["skipped"]} == {"a_url", "uses_url"}
@pytest.mark.asyncio
async def test_adopt_unknown_alias_is_404(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "alias/searchItem", [alias_row("x")])
resp = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "nope"})
assert resp.status_code == 404 and resp.json()["detail"]["code"] == "ADOPT_OBJECT_NOT_FOUND"
# ── NAT ─────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_automatic_nat_rows_are_never_offered(client, respx_mock):
host_id = await _host(client, respx_mock)
_mode(respx_mock)
for prefix, (path, factory, _) in NAT_KINDS.items():
_search(respx_mock, path, [factory(), *AUTOMATIC_NAT])
rows = (await client.get(f"/api/{prefix}/unmanaged?host_id={host_id}")).json()
assert [r["uuid"] for r in rows] == [factory()["uuid"]], prefix
assert rows[0]["adoptable"] is True
@pytest.mark.asyncio
async def test_port_forward_refusals(client, respx_mock):
host_id = await _host(client, respx_mock)
cases = {
"multi": (dict(interface="wan,lan"), "interface_not_single"),
"v46": (dict(ipprotocol="inet46"), "ipprotocol_unsupported"),
"anyproto": (dict(protocol=""), "protocol_any"),
"nordr": (dict(nordr="1", target="", **{"local-port": ""}), "no_redirect"),
"notarget": (dict(target=""), "no_target"),
"linked": (dict(**{"pass": "pass"}), "filter_rule_linked"),
"srcnot": (dict(**{"source.not": "1"}), "negated_source"),
"dstnot": (dict(**{"destination.not": "1"}), "negated_destination"),
"refl": (dict(natreflection="enable"), "natreflection_unsupported"),
"nodesc": (dict(descr=""), "no_description"),
"long": (dict(target="x" * 200), "value_too_long"),
}
_search(respx_mock, "d_nat/searchRule", [pf_row(**{"uuid": k, "descr": f"rule {k}", **over})
for k, (over, _) in cases.items()] + [pf_row(uuid="fine")])
rows = {r["uuid"]: r for r in (await client.get(f"/api/port-forward/unmanaged?host_id={host_id}")).json()}
assert rows["fine"]["adoptable"] is True
for key, (_, reason) in cases.items():
assert reason in rows[key]["reasons"] and not rows[key]["adoptable"], key
@pytest.mark.asyncio
async def test_port_forward_adopt_maps_fields_and_keeps_unmanaged_ones(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "d_nat/searchRule", [
pf_row(disabled="1", natreflection="purenat", log="1", poolopts="random", tag="tg1", **{"source.network": "10.0.0.0/8"}),
pf_row(uuid="pf-2", descr="Alias based", target="web_servers", **{"destination.port": "web_ports", "local-port": ""}),
*AUTOMATIC_NAT,
])
candidates = {r["uuid"]: r for r in (await client.get(f"/api/port-forward/unmanaged?host_id={host_id}")).json()}
assert sorted(candidates["pf-1"]["kept"]) == ["log", "poolopts", "tag"]
resp = await client.post("/api/port-forward/adopt", json={"host_id": host_id, "uuid": "pf-1"})
assert resp.status_code == 201, resp.text
rule = resp.json()
assert (rule["description"], rule["interface"], rule["ipprotocol"], rule["protocol"]) == ("Web", "wan", "inet", "tcp")
assert (rule["source_net"], rule["destination_net"], rule["destination_port"]) == ("10.0.0.0/8", "wanip", "8443")
assert (rule["target"], rule["local_port"], rule["natreflection"], rule["enabled"]) == ("192.168.1.50", "443", "purenat", False)
assert rule["workflow_state"] == "synced" and [d["status"] for d in rule["deployments"]] == ["success"]
preview = (await client.get(f"/api/port-forward/{rule['id']}/preview")).json()
assert [h["action"] for h in preview["hosts"]] == ["unchanged"] and preview["can_deploy"] is False
assert not _writes(respx_mock) or not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
# edit -> deploy updates the same remote rule and never sends the fields it does not manage
edit = {k: rule[k] for k in ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port",
"destination_net", "destination_port", "target", "local_port", "natreflection", "enabled")}
edit.update(target="192.168.1.51", target_host_ids=[host_id], target_group_ids=[], expected_updated_at=rule["updated_at"])
assert (await client.put(f"/api/port-forward/{rule['id']}", json=edit)).status_code == 200
add = respx_mock.post(f"{API}/d_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
setter = respx_mock.post(f"{API}/d_nat/setRule/pf-1").mock(return_value=httpx.Response(200, json={"result": "saved"}))
respx_mock.post(f"{API}/d_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
assert (await deploy_approved(client, f"/api/port-forward/{rule['id']}/deploy")).status_code == 200
assert setter.called and not add.called
sent = json.loads(setter.calls.last.request.content)["rule"]
assert sent["target"] == "192.168.1.51" and sent["disabled"] == "1"
assert not {"log", "poolopts", "tag", "tagged", "nosync", "categories", "nordr"} & set(sent)
@pytest.mark.asyncio
async def test_one_to_one_adopt_and_refusals(client, respx_mock):
host_id = await _host(client, respx_mock)
_search(respx_mock, "one_to_one/searchRule", [
o2o_row(),
o2o_row(uuid="o2", description="NAT off", type="nat", enabled="0", natreflection="enable", log="1",
source_not="1", destination_net="10.0.0.0/8", destination_not="1"),
o2o_row(uuid="o3", interface="wan,lan", description="multi"),
o2o_row(uuid="o4", description=""),
o2o_row(uuid="o5", description="weird", type="other"),
o2o_row(uuid="o6", description="refl", natreflection="purenat"),
*AUTOMATIC_NAT,
])
rows = {r["uuid"]: r for r in (await client.get(f"/api/one-to-one/unmanaged?host_id={host_id}")).json()}
assert rows["o2o-1"]["adoptable"] and rows["o2"]["adoptable"]
assert rows["o3"]["reasons"] == ["interface_not_single"] and rows["o4"]["reasons"] == ["no_description"]
assert rows["o5"]["reasons"] == ["type_unsupported"] and rows["o6"]["reasons"] == ["natreflection_unsupported"]
rule = (await client.post("/api/one-to-one/adopt", json={"host_id": host_id, "uuid": "o2"})).json()
assert (rule["type"], rule["enabled"], rule["natreflection"], rule["log"], rule["source_not"], rule["destination_not"]) == (
"nat", False, "enable", True, True, True)
assert rule["destination_net"] == "10.0.0.0/8" and rule["external"] == "203.0.113.201"
preview = (await client.get(f"/api/one-to-one/{rule['id']}/preview")).json()
assert [h["action"] for h in preview["hosts"]] == ["unchanged"]
@pytest.mark.asyncio
async def test_outbound_nat_adopt_and_refusals(client, respx_mock):
host_id = await _host(client, respx_mock)
_mode(respx_mock, "hybrid")
_search(respx_mock, "source_nat/searchRule", [
snat_row(),
snat_row(uuid="s2", description="to pool", protocol="TCP", destination_port="443", target="203.0.113.205",
target_port="1024", staticnatport="1", log="1", enabled="0", destination_not="1", destination_net="10.0.0.0/8"),
snat_row(uuid="s3", description="no nat", nonat="1"),
snat_row(uuid="s4", description="v46", ipprotocol="inet46"),
snat_row(uuid="s5", description=""),
*AUTOMATIC_NAT,
])
rows = {r["uuid"]: r for r in (await client.get(f"/api/outbound-nat/unmanaged?host_id={host_id}")).json()}
assert rows["snat-1"]["adoptable"] and rows["s2"]["adoptable"]
assert sorted(rows["s2"]["kept"]) == ["log", "staticnatport"]
assert rows["s3"]["reasons"] == ["no_nat_rule"] and rows["s4"]["reasons"] == ["ipprotocol_unsupported"]
assert rows["s5"]["reasons"] == ["no_description"]
rule = (await client.post("/api/outbound-nat/adopt", json={"host_id": host_id, "uuid": "s2"})).json()
assert (rule["protocol"], rule["destination_port"], rule["target"], rule["target_port"], rule["enabled"],
rule["destination_not"]) == ("TCP", "443", "203.0.113.205", "1024", False, True)
assert rule["source_port"] is None
preview = (await client.get(f"/api/outbound-nat/{rule['id']}/preview")).json()
assert [h["action"] for h in preview["hosts"]] == ["unchanged"]
# edit -> deploy: setRule on the same uuid, nonat/log/staticnatport are not part of the payload
edit = {k: rule[k] for k in ("description", "interface", "source_net", "destination_net", "source_not", "destination_not",
"enabled", "ipprotocol", "protocol", "source_port", "destination_port", "target", "target_port")}
edit.update(enabled=True, target_host_ids=[host_id], target_group_ids=[], expected_updated_at=rule["updated_at"])
assert (await client.put(f"/api/outbound-nat/{rule['id']}", json=edit)).status_code == 200
add = respx_mock.post(f"{API}/source_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
setter = respx_mock.post(f"{API}/source_nat/setRule/s2").mock(return_value=httpx.Response(200, json={"result": "saved"}))
respx_mock.post(f"{API}/source_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
assert (await deploy_approved(client, f"/api/outbound-nat/{rule['id']}/deploy")).status_code == 200
assert setter.called and not add.called
sent = json.loads(setter.calls.last.request.content)["rule"]
assert sent["enabled"] == "1" and not {"nonat", "log", "staticnatport", "nosync", "tag"} & set(sent)
@pytest.mark.asyncio
@pytest.mark.parametrize("mode", ["automatic", "disabled"])
async def test_outbound_nat_adoption_needs_a_manual_capable_mode(client, respx_mock, mode):
host_id = await _host(client, respx_mock)
_mode(respx_mock, mode)
search = _search(respx_mock, "source_nat/searchRule", [snat_row()])
for call in (client.get(f"/api/outbound-nat/unmanaged?host_id={host_id}"),
client.post("/api/outbound-nat/adopt", json={"host_id": host_id, "uuid": "snat-1"}),
client.post("/api/outbound-nat/adopt-all", json={"host_id": host_id})):
resp = await call
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "OUTBOUND_MODE_BLOCKS_ADOPT", resp.text
assert resp.json()["detail"]["params"]["mode"] == mode
assert not search.called
assert (await client.get("/api/outbound-nat")).json() == []
@pytest.mark.asyncio
@pytest.mark.parametrize("prefix", list(NAT_KINDS))
async def test_adopt_all_nat_takes_what_is_adoptable_and_is_idempotent(client, respx_mock, prefix):
host_id = await _host(client, respx_mock)
_mode(respx_mock)
path, factory, title = NAT_KINDS[prefix]
bad = {"descr": ""} if prefix == "port-forward" else {"description": ""}
_search(respx_mock, path, [factory(), factory(uuid="second", **({"descr": "Second"} if prefix == "port-forward" else {"description": "Second"})),
factory(uuid="bad", **bad), *AUTOMATIC_NAT])
result = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": host_id})).json()
assert sorted(a["title"] for a in result["adopted"]) == sorted([title, "Second"])
assert [(s["uuid"], s["reasons"]) for s in result["skipped"]] == [("bad", ["no_description"])]
listed = (await client.get(f"/api/{prefix}")).json()
assert len(listed) == 2 and all(r["workflow_state"] == "synced" for r in listed)
again = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": host_id})).json()
assert again["adopted"] == [] and len((await client.get(f"/api/{prefix}")).json()) == 2
assert not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
@pytest.mark.asyncio
@pytest.mark.parametrize("prefix", ["aliases", *NAT_KINDS])
async def test_roles_and_unknown_host(client, respx_mock, prefix):
host_id = await _host(client, respx_mock)
_mode(respx_mock)
path = "alias/searchItem" if prefix == "aliases" else NAT_KINDS[prefix][0]
row = alias_row("a1") if prefix == "aliases" else NAT_KINDS[prefix][1]()
_search(respx_mock, path, [row])
unknown = "00000000-0000-0000-0000-000000000000"
assert (await client.get(f"/api/{prefix}/unmanaged?host_id={unknown}")).status_code == 404
assert (await client.post(f"/api/{prefix}/adopt", json={"host_id": unknown, "uuid": "x"})).status_code == 404
# read-only users cannot adopt (it creates central objects), operators can (it writes nothing to a firewall)
for role, expected in (("read_only", 403), ("operator", 201)):
user = await client.post("/api/users", json={"username": f"u-{role}-{prefix}", "password": "adopt-test-password-1", "role": role})
assert user.status_code == 201, user.text
session = await _login(f"u-{role}-{prefix}", "adopt-test-password-1")
try:
assert (await session.get(f"/api/{prefix}/unmanaged?host_id={host_id}")).status_code == 200
resp = await session.post(f"/api/{prefix}/adopt", json={"host_id": host_id, "uuid": row["uuid"]})
assert resp.status_code == expected, (role, resp.text)
finally:
await session.aclose()
+147
View File
@@ -0,0 +1,147 @@
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
import pytest
from tests.test_maintenance_windows import _WEEKLY
from tests.test_rules import _create_host
async def _fleet(client):
"""canary: c1 · pilot: p1 + ha-b (BACKUP) · prod: ha-a; ha-a/ha-b form cluster "ha"."""
hosts = {}
for name in ("c1", "p1", "ha-a", "ha-b"):
hosts[name] = await _create_host(client, name, f"https://{name}.test")
for name in ("ha-a", "ha-b"):
await client.patch(f"/api/hosts/{hosts[name]['id']}", json={"cluster_name": "ha"})
groups = {}
for gname, members in (("ring-canary", ["c1"]), ("ring-pilot", ["p1", "ha-b"]), ("ring-prod", ["ha-a", "c1"])):
resp = await client.post("/api/groups", json={"name": gname, "host_ids": [hosts[m]["id"] for m in members]})
assert resp.status_code == 201, resp.text
groups[gname] = resp.json()["id"]
return hosts, groups
def _plan_body(groups, **kw):
body = {"name": "Ringe", "action": "update", "waves": [
{"label": "canary", "group_ids": [groups["ring-canary"]], "soak_minutes": 10},
{"label": "pilot", "group_ids": [groups["ring-pilot"]], "soak_minutes": 10},
{"label": "prod", "group_ids": [groups["ring-prod"]], "soak_minutes": 0, "carp_handover": True, "require_approval": True},
]}
body.update(kw)
return body
@pytest.mark.asyncio
async def test_plan_crud_and_preview_resolves_rings(client):
_hosts, groups = await _fleet(client)
resp = await client.post("/api/rollouts/plans", json=_plan_body(groups))
assert resp.status_code == 201, resp.text
plan = resp.json()
assert [w["label"] for w in plan["waves"]] == ["canary", "pilot", "prod"] and plan["host_count"] == 4
prev = await client.post(f"/api/rollouts/plans/{plan['id']}/preview")
assert prev.status_code == 200
body = prev.json()
assert [[h["name"] for h in w["hosts"]] for w in body["waves"]] == [["c1"], ["ha-b", "p1"], ["ha-a"]] # c1 only once
codes = {w["code"] for w in body["warnings"]}
assert {"duplicate_hosts", "mixed_versions", "master_needs_handover", "no_window"} <= codes
assert body["errors"] == [] and body["can_start"] and len(body["preview_token"]) == 64
assert (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()["detail"]["code"] == "ROLLOUT_NAME_TAKEN"
assert (await client.delete(f"/api/groups/{groups['ring-canary']}")).json()["detail"]["code"] == "ROLLOUT_GROUP_IN_USE"
assert (await client.delete(f"/api/rollouts/plans/{plan['id']}")).status_code == 204
assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).status_code == 404
@pytest.mark.asyncio
async def test_major_update_must_keep_a_cluster_in_one_wave(client):
_hosts, groups = await _fleet(client)
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, action="major_update"))).json()
assert {e["code"] for e in plan["errors"]} == {"cluster_split"}
prev = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()
assert prev["can_start"] is False
@pytest.mark.asyncio
async def test_activation_needs_auto_start_a_window_and_the_current_token(client):
_hosts, groups = await _fleet(client)
window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == "auto_start"
plan = (await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"]))).json()
stale = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
assert stale.status_code == 409 and stale.json()["detail"]["code"] == "ROLLOUT_PREVIEW_STALE"
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
active = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
assert active.status_code == 200 and active.json()["active"] and active.json()["fingerprint_ok"]
assert active.json()["activated_by"] == "admin"
# A group change behind the plan's back: still "active", but the fingerprint no longer matches.
await client.put(f"/api/groups/{groups['ring-canary']}", json={"name": "ring-canary", "host_ids": []})
assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).json()["fingerprint_ok"] is False
# A window that a plan uses cannot be deleted (the plan would silently lose its time limit).
assert (await client.delete(f"/api/maintenance-windows/{window['id']}")).json()["detail"]["code"] == "WINDOW_IN_USE"
# Any edit disarms it.
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
edited = await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"], name="Ringe 2"))
assert edited.json()["active"] is False
off = await client.post(f"/api/rollouts/plans/{plan['id']}/deactivate")
assert off.status_code == 200 and off.json()["active"] is False
@pytest.mark.asyncio
async def test_auto_start_only_for_minor_updates_with_a_window(client):
_hosts, groups = await _fleet(client)
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True))).json()
assert "auto_start_needs_window" in {e["code"] for e in plan["errors"]}
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "ROLLOUT_PLAN_INVALID"
@pytest.mark.parametrize("change,field", [
({"gate_checks": {"nonsense": True}}, "gate_checks"),
({"window_id": "00000000-0000-0000-0000-000000000000"}, "window_id"),
])
@pytest.mark.asyncio
async def test_plan_validation(client, change, field):
_hosts, groups = await _fleet(client)
resp = await client.post("/api/rollouts/plans", json=_plan_body(groups, **change))
assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == field
bad_group = _plan_body(groups)
bad_group["waves"][0]["group_ids"] = ["00000000-0000-0000-0000-000000000000"]
assert (await client.post("/api/rollouts/plans", json=bad_group)).json()["detail"]["params"]["field"] == "group_ids"
too_many = _plan_body(groups, waves=[{"label": str(i), "group_ids": [groups["ring-canary"]]} for i in range(11)])
assert (await client.post("/api/rollouts/plans", json=too_many)).status_code == 422
@pytest.mark.asyncio
async def test_role_matrix_for_plans(client):
from tests.test_operator_role import _operator
_hosts, groups = await _fleet(client)
window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
op = await _operator(client, "rollop")
try:
made = await op.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True, window_id=window["id"]))
assert made.status_code == 201 # preparing is fine
pid = made.json()["id"]
token = (await op.post(f"/api/rollouts/plans/{pid}/preview")).json()["preview_token"]
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
denied = await op.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})
assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
assert (await op.post(f"/api/rollouts/plans/{pid}/deactivate")).status_code == 200 # braking is always allowed
assert (await client.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})).status_code == 200
finally:
await op.aclose()
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
from tests.test_api_tokens import _login
assert (await client.post("/api/users", json={"username": "rollro", "password": "readonly-pass-1", "role": "read_only"})).status_code == 201
ro = await _login("rollro", "readonly-pass-1")
try:
assert (await ro.get("/api/rollouts/plans")).status_code == 200
for path in ("/api/rollouts/plans", f"/api/rollouts/plans/{pid}/deactivate"):
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
finally:
await ro.aclose()
+620
View File
@@ -0,0 +1,620 @@
"""The rollout engine end to end against a fake fleet: c1 (stand-alone), ha-a (MASTER) + ha-b (BACKUP)
in cluster "ha". A fake clock drives timeouts, soak and re-checks; every tick runs exactly what the
poll loop runs (fire due schedules, step running ones, step the rollout)."""
import asyncio
import uuid
from datetime import datetime, timedelta, timezone
import httpx
import pytest
from sqlalchemy import select
from app.database import SessionLocal
from app.models.audit import AuditEvent
from app.models.host_operation import HostOperation
from app.models.rollout import RolloutPlan, RolloutRun
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import rollout_engine as engine
from app.services import scheduled_maintenance_service as svc
from tests.test_rules import _create_host
class Fw:
def __init__(self, fleet, name, *, role=None, version="26.7.6", latest="26.7.7"):
self.fleet, self.name, self.base = fleet, name, f"https://{name}.test"
self.role, self.maint = role, False
self.version, self.latest, self.offer = version, latest, "update"
self.job, self.job_ticks, self.boot, self.up = "ready", 0, "boot-1", True
self.update_error = False
self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
{"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
r, b = fleet.mock, self.base
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
r.get(f"{b}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
r.get(f"{b}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
r.get(f"{b}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
r.get(f"{b}/api/core/firmware/status").mock(side_effect=self._status)
r.get(f"{b}/api/diagnostics/system/systemTime").mock(side_effect=lambda q: self._ok({"boottime": self.boot}))
r.get(f"{b}/api/routes/gateway/status").mock(side_effect=lambda q: self._ok({"items": self.gateways, "status": "ok"}))
r.get(f"{b}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
@property
def down(self):
return self.job == "down" or not self.up
def _ok(self, data):
if self.down:
raise httpx.ConnectError("down")
return httpx.Response(200, json=data)
def _raw(self, content):
if self.down:
raise httpx.ConnectError("down")
return httpx.Response(200, content=content)
def _status(self, request):
if self.offer == "update":
return self._ok({"status": "update", "status_msg": "x", "status_reboot": "1", "all_packages": {},
"product": {"product_latest": self.latest}})
return self._ok({"status": self.offer, "status_msg": "x", "status_reboot": "0", "all_packages": {}, "product": {}})
def _update(self, request):
if self.update_error:
return httpx.Response(500, json={"status": "error"})
self.job, self.job_ticks = "running", 2
return self._ok({"status": "ok"})
def _vip(self, request):
rows = [{"vhid": "1", "status": self.role}] if self.role else []
return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "240" if self.maint else "0",
"status_msg": ""}})
def _toggle(self, request):
self.maint = not self.maint
partner = self.fleet.partner(self)
if self.maint:
self.role, partner.role = "BACKUP", "MASTER"
else:
self.role, partner.role = "MASTER", "BACKUP" # preempt: takes MASTER back
return self._ok({"status": "ok", "action": "enter_maintenance" if self.maint else "leave_maintenance"})
def progress(self):
if self.job == "running":
self.job_ticks -= 1
if self.job_ticks <= 0:
self.job, self.job_ticks = "down", 2
elif self.job == "down":
self.job_ticks -= 1
if self.job_ticks <= 0:
self.job, self.version, self.offer, self.boot = "ready", self.latest, "none", self.boot + "+"
class Fleet:
def __init__(self, respx_mock):
self.mock = respx_mock
self.fw = {"c1": Fw(self, "c1"), "ha-a": Fw(self, "ha-a", role="MASTER"), "ha-b": Fw(self, "ha-b", role="BACKUP")}
def partner(self, fw):
return self.fw["ha-b"] if fw.name == "ha-a" else self.fw["ha-a"]
def progress(self):
for fw in self.fw.values():
fw.progress()
class Clock:
def __init__(self):
self.now = datetime.now(timezone.utc)
def __call__(self):
return self.now
@pytest.fixture
def clock(monkeypatch):
c = Clock()
monkeypatch.setattr(engine, "_now", c)
monkeypatch.setattr(svc, "_now", c)
return c
async def _setup(client, respx_mock, *, waves=None, **plan_kw):
fleet = Fleet(respx_mock)
hosts = {n: await _create_host(client, n, f"https://{n}.test") for n in ("c1", "ha-a", "ha-b")}
for n in ("ha-a", "ha-b"):
await client.patch(f"/api/hosts/{hosts[n]['id']}", json={"cluster_name": "ha"})
groups = {}
for g, members in (("canary", ["c1"]), ("pilot", ["ha-b"]), ("prod", ["ha-a"]), ("cluster", ["ha-a", "ha-b"])):
groups[g] = (await client.post("/api/groups", json={"name": g, "host_ids": [hosts[m]["id"] for m in members]})).json()["id"]
waves = waves or [
{"label": "canary", "group_ids": [groups["canary"]], "soak_minutes": 5},
{"label": "pilot", "group_ids": [groups["pilot"]], "soak_minutes": 5},
{"label": "prod", "group_ids": [groups["prod"]], "soak_minutes": 0, "carp_handover": True},
]
waves = [{**w, "group_ids": [groups[g] if g in groups else g for g in w["group_ids"]]} for w in waves]
plan = (await client.post("/api/rollouts/plans", json={"name": "ringe", "action": "update", "waves": waves, **plan_kw})).json()
return fleet, hosts, groups, plan
async def _start(client, plan, api=None):
api = api or client
token = (await api.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
resp = await api.post(f"/api/rollouts/plans/{plan['id']}/start",
json={"preview_token": token, "confirm_name": plan["name"], "acknowledge_unattended": True})
assert resp.status_code == 201, resp.text
return resp.json()
async def _tick(fleet, clock, seconds=30):
fleet.progress()
async with SessionLocal() as db:
await svc.fire_due(db)
async with SessionLocal() as db:
await svc.advance_running(db)
async with SessionLocal() as db:
await engine.advance_runs(db)
clock.now += timedelta(seconds=seconds)
async def _run(run_id):
async with SessionLocal() as db:
return await db.scalar(select(RolloutRun).where(RolloutRun.id == uuid.UUID(str(run_id))).execution_options(populate_existing=True))
async def _until(fleet, clock, run_id, pred, max_ticks=400, check=None):
for _ in range(max_ticks):
run = await _run(run_id)
if pred(run):
return run
if check:
await check()
await _tick(fleet, clock)
raise AssertionError(f"not reached: status={run.status} reason={run.pause_reason} detail={run.pause_detail}")
def _host(run, name):
return next(h for h in run.hosts if h.host_name == name)
async def _scheds():
async with SessionLocal() as db:
return list((await db.scalars(select(ScheduledMaintenance).order_by(ScheduledMaintenance.created_at))).all())
async def _no_two_at_once():
active = [s for s in await _scheds() if s.status in ("pending", "running")]
assert len(active) <= 1, [(s.host_name, s.status, s.phase) for s in active]
# ── happy path ────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_three_rings_run_through_and_the_master_hands_over(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
assert done.target_version == "26.7.7"
assert [(h.host_name, h.status, h.version_before, h.version_after) for h in sorted(done.hosts, key=lambda h: h.wave_index)] == [
("c1", "succeeded", "26.7.6", "26.7.7"), ("ha-b", "succeeded", "26.7.6", "26.7.7"), ("ha-a", "succeeded", "26.7.6", "26.7.7")]
assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
# The MASTER went through the CARP handover and is MASTER again; the BACKUP never needed one.
assert fleet.fw["ha-a"].toggles.call_count == 2 and fleet.fw["ha-b"].toggles.call_count == 0
assert (fleet.fw["ha-a"].role, fleet.fw["ha-b"].role) == ("MASTER", "BACKUP")
scheds = await _scheds()
assert [s.host_name for s in scheds] == ["c1", "ha-b", "ha-a"]
assert all(s.origin == "rollout" and s.auto_verify and s.status == "succeeded" for s in scheds)
async with SessionLocal() as db:
ops = (await db.scalars(select(HostOperation))).all()
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rollout"))).all()
assert {o.status for o in ops} == {"acknowledged"} # nothing left locked
assert sum(1 for e in events if e.action == "gate" and e.status == "ok") == 3
assert any(e.action == "finish" and e.status == "succeeded" for e in events)
host_events = [e for e in (await _audit_host()) if (e.details or {}).get("rollout")]
assert host_events and all(e.details["rollout"]["run_id"] == run["id"] for e in host_events)
async def _audit_host():
async with SessionLocal() as db:
return (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
# ── the gate stops the rollout ─────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_gate_failure_in_the_canary_ring_stops_everything_after_it(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
fleet.fw["c1"].services[1]["running"] = 0 # ntpd did not come back
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "gate_failed" and paused.paused_from == "gate"
assert paused.waves[0].gate_attempts == 2 # looked at twice (slow start), then stopped
assert paused.waves[0].gate_result["hosts"][_host(paused, "c1").host_id.__str__()]["services"]["code"] == "service_stopped"
for _ in range(20): # time passes, nothing moves
await _tick(fleet, clock)
assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
assert [s.host_name for s in await _scheds()] == ["c1"]
# Notification while paused.
from app.services import notification_checks
async with SessionLocal() as db:
problems = await notification_checks.check_rollout_paused(db)
assert problems[run["id"]]["grund"] == "gate_failed"
# Fixed by hand, resumed: the gate is checked again first, then it goes on.
fleet.fw["c1"].services[1]["running"] = 1
resp = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
assert resp.status_code == 200 and resp.json()["status"] == "resuming"
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
assert fleet.fw["ha-a"].update.call_count == 1
@pytest.mark.asyncio
async def test_resume_rechecks_and_pauses_again_if_still_broken(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
fleet.fw["c1"].gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100 %", "delay": "~", "stddev": "~"}
await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
again = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert again.pause_reason == "gate_failed" and again.pause_detail.get("on_resume") is True
assert fleet.fw["ha-b"].update.call_count == 0
# ── never twice, never in parallel ─────────────────────────────────────────────
@pytest.mark.asyncio
async def test_restart_in_the_middle_of_job_wait_submits_exactly_once(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
async def in_job_wait(r):
return any(s.host_name == "ha-b" and s.phase == "job_wait" for s in await _scheds())
for _ in range(300):
if await in_job_wait(None):
break
await _tick(fleet, clock)
# "Restart": nothing survives but the database (each tick opens fresh sessions anyway); two backends
# even step the same run at the very same moment.
await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(3)))
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded"
assert fleet.fw["ha-b"].update.call_count == 1 and len([s for s in await _scheds() if s.host_name == "ha-b"]) == 1
@pytest.mark.asyncio
async def test_two_backends_deciding_at_once_create_one_schedule(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
await _until(fleet, clock, run["id"], lambda r: r.status == "wave_running")
await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(4)))
assert len(await _scheds()) == 1
@pytest.mark.asyncio
async def test_a_cluster_in_one_wave_goes_backup_first_and_never_both_at_once(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
run = await _start(client, plan)
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
assert [s.host_name for s in await _scheds()] == ["ha-b", "ha-a"]
@pytest.mark.asyncio
async def test_master_without_handover_pauses_before_anything_happens_to_it(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": False}])
run = await _start(client, plan)
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "master_needs_handover" and paused.pause_detail["host"] == "ha-a"
assert fleet.fw["ha-b"].update.call_count == 1 and fleet.fw["ha-a"].update.call_count == 0
assert fleet.fw["ha-a"].toggles.call_count == 0
@pytest.mark.asyncio
async def test_cluster_gate_before_the_second_node(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
run = await _start(client, plan)
async def break_backup_after_update():
if fleet.fw["ha-b"].version == "26.7.7":
fleet.fw["ha-b"].services[0]["running"] = 0 # unbound dead on the freshly updated node
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused", check=break_backup_after_update)
assert paused.pause_reason == "cluster_gate_failed" and paused.pause_detail["host"] == "ha-a"
assert fleet.fw["ha-a"].update.call_count == 0 and fleet.fw["ha-a"].toggles.call_count == 0
# ── stop when in doubt ─────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_a_different_target_version_in_a_later_ring_pauses(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
fleet.fw["ha-b"].latest = "26.7.8" # a newer release appeared after the canary ran
run = await _start(client, plan)
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "target_version_changed"
assert paused.pause_detail == {"expected": "26.7.7", "offered": "26.7.8"}
assert fleet.fw["ha-b"].update.call_count == 0
@pytest.mark.asyncio
async def test_hosts_without_an_update_are_skipped(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
fleet.fw["ha-b"].offer = "none"
run = await _start(client, plan)
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded" and _host(done, "ha-b").status == "skipped"
assert _host(done, "ha-b").detail["code"] == "no_update_offered" and done.waves[1].status == "skipped"
@pytest.mark.asyncio
async def test_failed_host_pauses_and_resume_needs_an_explicit_skip(client, respx_mock, clock):
fleet, hosts, _groups, plan = await _setup(client, respx_mock)
fleet.fw["c1"].update_error = True
run = await _start(client, plan)
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "host_failed" and _host(paused, "c1").status == "failed"
refused = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ROLLOUT_HOST_FAILED"
skip = await client.post(f"/api/rollouts/runs/{run['id']}/skip-host", json={"host_id": hosts["c1"]["id"], "reason": "geprüft, später"})
assert skip.status_code == 200
# The failed operation stays "unknown" (locks c1, notifies) until a human closes it; it is not retried.
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
assert fleet.fw["c1"].update.call_count == 1 and _host(done, "c1").detail["code"] == "skipped_by"
@pytest.mark.asyncio
async def test_unreachable_host_in_preflight_is_blocked_then_paused(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
fleet.fw["c1"].up = False
run = await _start(client, plan)
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "preflight_blocked" and _host(paused, "c1").status == "blocked"
assert clock.now - datetime.fromisoformat(paused.progress["paused_at"]) < timedelta(minutes=1)
fleet.fw["c1"].up = True
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded"
@pytest.mark.asyncio
async def test_someone_elses_open_maintenance_blocks_then_pauses(client, respx_mock, clock):
fleet, hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
await _until(fleet, clock, run["id"], lambda r: r.current_wave == 1 and r.status == "wave_running")
async with SessionLocal() as db:
db.add(HostOperation(host_id=uuid.UUID(hosts["ha-a"]["id"]), host_name="ha-a", actor="someone", action="reboot",
status="submitted", expires_at=clock.now, fingerprint="", snapshot={}))
await db.commit()
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason in ("blocked_by_maintenance", "cluster_gate_failed")
assert fleet.fw["ha-b"].update.call_count == 0
@pytest.mark.asyncio
async def test_actor_who_lost_the_right_pauses_the_run(client, respx_mock, clock):
from tests.test_operator_role import _operator
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
op = await _operator(client, "rollengine-op")
try:
run = await _start(client, plan, api=op)
assert run["started_by"] == "rollengine-op"
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
assert paused.pause_reason == "actor_invalid" and fleet.fw["ha-b"].update.call_count == 0
# An admin takes over by resuming.
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).json()["started_by"] == "admin"
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded"
finally:
await op.aclose()
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
# ── windows, approval, skip, abort, automatic start ───────────────────────────
async def _window_now(client, clock, *, cutoff=60, duration=120):
local = clock.now
body = {"name": f"jetzt-{uuid.uuid4().hex[:6]}", "timezone": "UTC", "recurrence": "weekly", "weekdays": [local.weekday()],
"start_time": (local - timedelta(minutes=1)).strftime("%H:%M"), "duration_minutes": duration, "start_cutoff_minutes": cutoff}
if (local - timedelta(minutes=1)).date() != local.date():
pytest.skip("too close to midnight for a same-day window")
resp = await client.post("/api/maintenance-windows", json=body)
assert resp.status_code == 201, resp.text
return resp.json()
@pytest.mark.asyncio
async def test_closed_window_waits_and_the_next_occurrence_resumes(client, respx_mock, clock):
window = await _window_now(client, clock, cutoff=15)
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"])
run = await _start(client, plan)
waiting = await _until(fleet, clock, run["id"], lambda r: r.status in ("waiting_window", "paused", "succeeded"))
assert waiting.status == "waiting_window" and waiting.current_wave >= 1
started = {s.host_name for s in await _scheds()}
assert "c1" in started and "ha-a" not in started
clock.now += timedelta(days=7) - timedelta(minutes=40) # one week later, a few minutes into the next occurrence
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
@pytest.mark.asyncio
async def test_approval_holds_the_wave(client, respx_mock, clock):
fleet, _hosts, groups, plan = await _setup(client, respx_mock, waves=[
{"label": "canary", "group_ids": ["canary"], "soak_minutes": 1},
{"label": "rest", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True, "require_approval": True}])
run = await _start(client, plan)
held = await _until(fleet, clock, run["id"], lambda r: r.status == "awaiting_approval")
for _ in range(10):
await _tick(fleet, clock)
assert fleet.fw["ha-b"].update.call_count == 0
resp = await client.post(f"/api/rollouts/runs/{run['id']}/approve")
assert resp.status_code == 200 and resp.json()["waves"][1]["approved_by"] == "admin"
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
assert done.status == "succeeded" and held.current_wave == 1
@pytest.mark.asyncio
async def test_abort_cancels_what_has_not_started_and_lets_the_running_host_finish(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
await _until(fleet, clock, run["id"], lambda r: any(h.status == "running" for h in r.hosts))
bad = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "falsch"})
assert bad.status_code == 409 and bad.json()["detail"]["code"] == "ROLLOUT_CONFIRM_NAME"
resp = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "ringe"})
assert resp.status_code == 200 and resp.json()["status"] == "cancelled"
for _ in range(40):
await _tick(fleet, clock)
run_now = await _run(run["id"])
assert run_now.status == "cancelled"
assert {h.host_name: h.status for h in run_now.hosts} in (
{"c1": "succeeded", "ha-b": "cancelled", "ha-a": "cancelled"}, {"c1": "cancelled", "ha-b": "cancelled", "ha-a": "cancelled"})
assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 409
@pytest.mark.asyncio
async def test_pause_by_hand_and_operator_may_always_brake(client, respx_mock, clock):
from tests.test_operator_role import _operator
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
op = await _operator(client, "brake-op")
try:
resp = await op.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "Bauchgefühl"})
assert resp.status_code == 200 and resp.json()["pause_reason"] == "manual"
denied = await op.post(f"/api/rollouts/runs/{run['id']}/resume")
assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
finally:
await op.aclose()
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
for _ in range(10):
await _tick(fleet, clock)
assert sum(fw.update.call_count for fw in fleet.fw.values()) == 0
second = await client.post(f"/api/rollouts/plans/{plan['id']}/start", json={
"preview_token": (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"],
"confirm_name": "ringe", "acknowledge_unattended": True})
assert second.status_code == 409 and second.json()["detail"]["code"] == "ROLLOUT_ALREADY_ACTIVE"
async def _until_step_pending(fleet, clock, run_id):
"""Right after the engine created a host's schedule (fire_due has not seen it yet)."""
run = await _until(fleet, clock, run_id, lambda r: any(h.status == "running" for h in r.hosts))
name = next(h.host_name for h in run.hosts if h.status == "running")
assert [s.status for s in await _scheds()] == ["pending"]
return name
async def _resume_to_the_end(client, fleet, clock, run_id):
assert (await client.post(f"/api/rollouts/runs/{run_id}/resume")).status_code == 200
done = await _until(fleet, clock, run_id, lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
@pytest.mark.asyncio
async def test_pause_while_a_step_is_pending_really_brakes_and_resume_runs_it_once(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
name = await _until_step_pending(fleet, clock, run["id"])
resp = await client.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "halt"})
assert resp.status_code == 200
for _ in range(10):
await _tick(fleet, clock)
assert fleet.fw[name].update.call_count == 0 # the pending step did not fire behind the pause
paused = await _run(run["id"])
assert paused.status == "paused" and _host(paused, name).status == "queued"
assert _host(paused, name).scheduled_maintenance_id is None
assert [s.status for s in await _scheds()] == ["cancelled"]
await _resume_to_the_end(client, fleet, clock, run["id"])
@pytest.mark.asyncio
async def test_a_step_found_long_overdue_after_an_outage_is_not_fired(client, respx_mock, clock):
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
run = await _start(client, plan)
name = await _until_step_pending(fleet, clock, run["id"])
clock.now += timedelta(minutes=30) # OPNexus was down (e.g. a web update); the window may be over by now
await _tick(fleet, clock)
assert fleet.fw[name].update.call_count == 0
paused = await _run(run["id"])
assert (paused.status, paused.pause_reason) == ("paused", "host_not_started")
assert _host(paused, name).status == "queued" and _host(paused, name).detail["code"] == "not_fired"
(sched,) = await _scheds()
assert sched.status == "failed" and sched.result["not_fired"]
await _resume_to_the_end(client, fleet, clock, run["id"])
@pytest.mark.asyncio
async def test_automatic_start_once_per_occurrence_and_only_while_unchanged(client, respx_mock, clock):
window = await _window_now(client, clock)
fleet, _hosts, groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
for _ in range(3):
async with SessionLocal() as db:
await engine.auto_start(db)
async with SessionLocal() as db:
runs = (await db.scalars(select(RolloutRun))).all()
assert len(runs) == 1 and runs[0].trigger == "window" and runs[0].started_by == "admin"
assert runs[0].window_occurrence is not None
# Next week, but somebody added a host to the canary group meanwhile: no start, a notification instead.
async with SessionLocal() as db:
(await db.get(RolloutRun, runs[0].id)).status = "cancelled"
await db.commit()
await client.put(f"/api/groups/{groups['canary']}", json={"name": "canary", "host_ids": [_hosts["c1"]["id"], _hosts["ha-a"]["id"]]})
clock.now += timedelta(days=7)
async with SessionLocal() as db:
await engine.auto_start(db)
async with SessionLocal() as db:
assert len((await db.scalars(select(RolloutRun))).all()) == 1
stored = await db.get(RolloutPlan, uuid.UUID(plan["id"]))
assert stored.auto_start_error["code"] == "plan_changed"
from app.services import notification_checks
async with SessionLocal() as db:
problems = await notification_checks.check_rollout_start_failed(db)
assert list(problems.values())[0]["grund"] == "plan_changed"
@pytest.mark.asyncio
async def test_a_second_backend_waiting_for_the_lock_reports_no_refusal(client, respx_mock, clock, monkeypatch):
window = await _window_now(client, clock)
_fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
real_lock = engine.policy_lock
async def racing_lock(db):
monkeypatch.setattr(engine, "policy_lock", real_lock)
async with SessionLocal() as other: # backend A starts the run while B waits for the lock
assert await engine.auto_start(other) == 1
await real_lock(db)
monkeypatch.setattr(engine, "policy_lock", racing_lock)
async with SessionLocal() as db:
assert await engine.auto_start(db) == 0
async with SessionLocal() as db:
assert len((await db.scalars(select(RolloutRun))).all()) == 1
assert (await db.get(RolloutPlan, uuid.UUID(plan["id"]))).auto_start_error is None
def test_target_version_comes_from_the_core_package_not_product_latest():
# Shape seen live on test-fw-3 (26.7.1_1): product_latest lagged behind what would really be installed.
status = {"status": "update", "product": {"product_latest": "26.7.2"},
"all_packages": {"opnsense": {"old": "26.7.1_1", "new": "26.7.7", "reason": "upgrade"}},
"upgrade_packages": [{"name": "opnsense", "current_version": "26.7.1_1", "new_version": "26.7.7"}]}
assert engine.target_of(status, "update") == "26.7.7"
assert engine.target_of({**status, "all_packages": {}}, "update") == "26.7.7"
assert engine.target_of({"product": {"product_latest": "26.7.7"}}, "update") == "26.7.7"
assert engine.target_of({"upgrade_major_version": "27.1"}, "major_update") == "27.1"
+195
View File
@@ -0,0 +1,195 @@
"""Health gate checks of the rollout engine, one by one, against a fake firewall pair."""
import uuid
import httpx
import pytest
from app.database import SessionLocal
from app.models.carp_maintenance_window import CarpMaintenanceWindow
from app.models.compliance_result import ComplianceResult
from app.models.host import OpnsenseHost
from app.models.host_operation import HostOperation
from app.services import rollout_health as rh
from tests.test_rules import _create_host
A, B = "https://gw-a.test", "https://gw-b.test"
class Fw:
def __init__(self, respx_mock, base, *, role="MASTER", version="26.7.7"):
self.base, self.role, self.version, self.up, self.job, self.maint = base, role, version, True, "ready", False
self.vips = None
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1.0 ms", "stddev": "0.1 ms"}]
self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
{"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
self.gw_error = False
r = respx_mock
r.get(f"{base}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
r.get(f"{base}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
r.get(f"{base}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
r.get(f"{base}/api/routes/gateway/status").mock(side_effect=self._gw)
r.get(f"{base}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
def _ok(self, data):
if not self.up:
raise httpx.ConnectError("down")
return httpx.Response(200, json=data)
def _gw(self, request):
if self.gw_error:
return httpx.Response(403, json={"message": "forbidden"})
return self._ok({"items": self.gateways, "status": "ok"})
def _vip(self, request):
rows = [{"vhid": v, "status": s} for v, s in (self.vips or {"1": self.role}).items()]
return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "0", "status_msg": ""}})
async def _hosts(client, cluster=True):
a = await _create_host(client, "gw-a", A)
if not cluster:
return a, None
b = await _create_host(client, "gw-b", B)
if cluster:
for h in (a, b):
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
return a, b
async def _eval(host_ids, baselines=None, target="26.7.7", **kw):
async with SessionLocal() as db:
hosts = [await db.get(OpnsenseHost, uuid.UUID(h)) for h in host_ids]
if baselines is None:
baselines = {str(h.id): await rh.baseline(db, h) for h in hosts}
return await rh.evaluate(db, hosts, baselines, target_version=target, **kw), baselines
@pytest.mark.asyncio
async def test_healthy_cluster_passes_every_check(client, respx_mock):
Fw(respx_mock, A)
Fw(respx_mock, B, role="BACKUP")
a, _ = await _hosts(client)
gate, base = await _eval([a["id"]])
assert gate["verdict"] == "ok", gate
row = gate["hosts"][a["id"]]
assert {k: v["result"] for k, v in row.items()} == {
"online": "ok", "version": "ok", "carp": "ok", "gateways": "ok", "services": "ok", "locks": "ok"}
assert base[a["id"]]["gateways"] == {"WAN_GW": "online"} and base[a["id"]]["services"]["ntpd"] == 1
@pytest.mark.asyncio
async def test_offline_or_busy_host_and_wrong_version_fail(client, respx_mock):
fa = Fw(respx_mock, A)
a, _ = await _hosts(client, cluster=False)
fa.version = "26.7.6"
gate, base = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["version"] == {"result": "fail", "code": "version_mismatch",
"params": {"expected": "26.7.7", "actual": "26.7.6"}}
fa.job = "busy"
gate, _ = await _eval([a["id"]], base)
assert gate["hosts"][a["id"]]["online"]["code"] == "firmware_busy"
fa.up = False
gate, _ = await _eval([a["id"]], base)
assert gate["hosts"][a["id"]]["online"]["code"] == "offline" and gate["verdict"] == "fail"
assert gate["hosts"][a["id"]]["gateways"]["result"] == "fail" # readable before, not now
assert rh.failed_checks(gate)[0].startswith(a["id"])
@pytest.mark.asyncio
async def test_carp_split_brain_maintenance_mode_and_timer_fail(client, respx_mock):
fa = Fw(respx_mock, A)
fb = Fw(respx_mock, B, role="BACKUP")
a, b = await _hosts(client)
fb.role = "MASTER" # both MASTER on VHID 1
gate, _ = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "vhid_roles", "params": {"vhids": "1"}}
fb.role = "BACKUP"
fa.vips, fb.vips = {"1": "MASTER", "2": "MASTER"}, {"1": "BACKUP", "2": "MASTER"} # partial failover on VHID 2
gate, _ = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["carp"]["params"] == {"vhids": "2"}
fa.vips = fb.vips = None
fb.maint = True
gate, _ = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["carp"]["code"] == "maintenance_mode"
fb.maint = False
async with SessionLocal() as db:
db.add(CarpMaintenanceWindow(host_id=uuid.UUID(b["id"]), host_name="gw-b", status="active"))
await db.commit()
gate, _ = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["carp"]["code"] == "auto_revert_active"
fb.up = False
gate, _ = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "node_offline", "params": {"names": "gw-b"}}
@pytest.mark.asyncio
async def test_gateways_compare_against_the_baseline(client, respx_mock):
fa = Fw(respx_mock, A)
fa.gateways.append({"name": "OLD_GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms"})
a, _ = await _hosts(client, cluster=False)
_gate, base = await _eval([a["id"]])
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["code"] == "gateway_bad_before" # only warns
fa.gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~"}
gate, _ = await _eval([a["id"]], base)
assert gate["hosts"][a["id"]]["gateways"] == {"result": "fail", "code": "gateway_down", "params": {"names": "WAN_GW"}}
fa.gateways[0] = {"name": "WAN_GW", "status": "loss", "loss": "30 %", "delay": "5 ms", "stddev": "1 ms"}
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "warn"
fa.gw_error = True
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "fail"
gate, _ = await _eval([a["id"]], {a["id"]: {**base[a["id"]], "gateways": None}})
assert gate["hosts"][a["id"]]["gateways"]["result"] == "skip" # no permission before either: nothing to compare
@pytest.mark.asyncio
async def test_services_that_ran_before_must_run_again(client, respx_mock):
fa = Fw(respx_mock, A)
fa.services.append({"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0})
a, _ = await _hosts(client, cluster=False)
_gate, base = await _eval([a["id"]])
fa.services[1] = {"id": "ntpd", "name": "ntpd", "locked": 0, "running": 0}
gate, _ = await _eval([a["id"]], base)
assert gate["hosts"][a["id"]]["services"] == {"result": "fail", "code": "service_stopped", "params": {"names": "ntpd"}}
fa.services[1]["running"] = 1
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["services"]["result"] == "ok" # dhcpd6 never ran: fine
@pytest.mark.asyncio
async def test_open_operation_in_the_cluster_is_a_lock(client, respx_mock):
Fw(respx_mock, A)
Fw(respx_mock, B, role="BACKUP")
a, b = await _hosts(client)
from datetime import datetime, timezone
async with SessionLocal() as db:
db.add(HostOperation(host_id=uuid.UUID(b["id"]), host_name="gw-b", actor="x", action="reboot", status="submitted",
expires_at=datetime.now(timezone.utc), fingerprint="", snapshot={}))
await db.commit()
gate, _ = await _eval([a["id"]])
assert gate["hosts"][a["id"]]["locks"] == {"result": "fail", "code": "open_locks", "params": {"count": 1}}
@pytest.mark.parametrize("mode,expected", [("warn", "warn"), ("fail", "fail")])
@pytest.mark.asyncio
async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeypatch, mode, expected):
Fw(respx_mock, A)
a, _ = await _hosts(client, cluster=False)
async with SessionLocal() as db:
db.add(ComplianceResult(host_id=uuid.UUID(a["id"]), checked_at=datetime_now(), score=80, findings=[]))
await db.commit()
async def fake_check(db, host):
row = await db.get(ComplianceResult, (await db.scalar(__import__("sqlalchemy").select(ComplianceResult.id))))
row.score = 70
await db.commit()
return row
monkeypatch.setattr(rh.compliance_service, "check_host", fake_check)
gate, _ = await _eval([a["id"]], checks={"compliance": True}, compliance_mode=mode)
assert gate["hosts"][a["id"]]["compliance"] == {"result": expected, "code": "score_lower", "params": {"before": 80, "after": 70}}
assert gate["verdict"] == expected
# the gate cannot be switched off where it matters
gate, _ = await _eval([a["id"]], checks={"online": False, "version": False, "locks": False})
assert {"online", "version", "locks"} <= set(gate["hosts"][a["id"]])
def datetime_now():
from datetime import datetime, timezone
return datetime.now(timezone.utc)
+334
View File
@@ -0,0 +1,334 @@
"""Rule analysis (1.26): normalization, resolver and pf semantics of the engine.
Pure functions against an anonymized live fixture (OPNsense 26.7: automatic rules, antispoof
lines, a CARP VIP on LAN). Synthetic rules get pf lines in sort order, as pf would load them.
The numbers in the test names follow the test matrix of the design (section 8).
"""
import copy
import ipaddress
import json
from pathlib import Path
import pytest
from app.services.rule_analysis import context as ctx_mod
from app.services.rule_analysis import pf_ruleset, resolve
from app.services.rule_analysis.engine import evaluate
from app.services.rule_analysis.normalize import normalize
from app.services.rule_analysis.service import build_query
FIXTURE = json.loads((Path(__file__).parent / "fixtures" / "rule_analysis_fw.json").read_text())
DEVICES = {"lan": "vtnet1_vlan10", "wan": "vtnet0"}
LAN_CLIENT, INTERNET, WAN_IP = "10.20.30.50", "198.51.100.10", "203.0.113.57"
def mvc(uuid, *, prio=400000, seq=1, action="pass", quick="1", interface="lan", src="any", dst="any",
proto="any", ipprotocol="inet", dport="", sport="", direction="in", loaded=True, **extra):
row = {
"uuid": uuid, "sort_order": f"{prio:06d}.{seq:07d}", "prio_group": str(prio), "sequence": str(seq),
"enabled": "1", "action": action, "quick": quick, "interface": interface, "interfacenot": "0",
"direction": direction, "ipprotocol": ipprotocol, "protocol": proto, "source_net": src, "source_not": "0",
"source_port": sport, "destination_net": dst, "destination_not": "0", "destination_port": dport,
"gateway": "", "sched": "", "statetype": "keep", "tagged": "", "description": uuid,
}
row.update(extra)
row["_loaded"] = loaded
return row
def build(extra_rows=(), *, drop=(), aliases=None, refs=None, groups=None, pf=True, port_forwards=()):
"""A Context from the fixture plus synthetic MVC rules. pf lines of MVC rules are generated
in sort order (pf numbers follow the evaluation order like on a real firewall)."""
data = copy.deepcopy(FIXTURE)
rows = [r for r in data["rules"]["rows"] if r["description"] not in drop] + [dict(r) for r in extra_rows]
interfaces = ctx_mod._interfaces(data["interfaces"])
group_rows = data["groups"]["rows"] + [{"ifname": k, "members": ",".join(v)} for k, v in (groups or {}).items()]
stats = None
if pf:
auto = {k: v for k, v in data["pf"]["rules"]["filter rules"].items()
if not any(r.get("uuid") in k for r in data["rules"]["rows"] if not r.get("is_automatic"))}
lines = sorted(((int(k.split()[0][1:]), k.split(" ", 1)[1]) for k in auto), key=lambda x: x[0])
number = max(n for n, _ in lines) + 1
mvc_rows = sorted((r for r in rows if not r.get("is_automatic") and r.get("_loaded", True)
and r.get("enabled") == "1"), key=lambda r: r.get("sort_order") or "400000." + r["sequence"].zfill(7))
for r in mvc_rows:
words = ["block" if r["action"] == "block" else ("block return" if r["action"] == "reject" else "pass"),
r["direction"] if r["direction"] != "any" else ""]
if r["quick"] == "1":
words.append("quick")
ifs = [i for i in str(r["interface"]).split(",") if i]
if len(ifs) == 1 and ifs[0] in DEVICES:
words.append(f"on {DEVICES[ifs[0]]}")
if r["ipprotocol"] in ("inet", "inet6"):
words.append(r["ipprotocol"])
words.append(f'all label "{r["uuid"]}"')
lines.append((number, " ".join(w for w in words if w)))
number += 1
stats = {"rules": {"filter rules": {f"@{n} {t}": {} for n, t in lines}}}
for r in rows:
r.pop("_loaded", None)
alias_rows = {a["name"]: a for a in data["aliases"]["rows"]}
alias_rows.update({name: {"name": name, **a} for name, a in (aliases or {}).items()})
ctx = ctx_mod.Context(rows=rows, pf=pf_ruleset.parse(stats) if pf else None, interfaces=interfaces,
groups=ctx_mod._groups({"rows": group_rows}, interfaces), aliases=alias_rows,
port_forwards=list(port_forwards), notes=[] if pf else ["PF_RULESET_UNAVAILABLE"])
default_refs = {LAN_CLIENT: ["__lan_network"], INTERNET: [], WAN_IP: ["__wan_network"]}
ctx.refs = {**default_refs, **(refs or {})}
return ctx
def run(ctx, src=LAN_CLIENT, dst=INTERNET, proto="tcp", dport="443", iface="lan", **kw):
q = build_query(src=src, dst=dst, proto=proto, dport=dport, iface=iface, **kw)
for ip in (str(q.src), str(q.dst)):
ctx.refs.setdefault(ip, [])
return evaluate(ctx, q)
def decided_by(result):
return result["decisive_rule"]["description"] if result["decisive_rule"] else None
# ── Normalization ─────────────────────────────────────────────────────────
def test_normalize_live_fixture_uses_pf_for_automatic_rules():
rules, notes = normalize(build())
by_desc = {}
for r in rules:
by_desc.setdefault(r.description, r)
assert notes == []
deny = by_desc["Default deny / state violation rule"]
assert deny.quick is False and deny.action == "block" and deny.loaded and deny.pf_numbers
dhcp = by_desc["allow access to DHCP server"]
assert dhcp.action == "pass" and dhcp.quick is True and dhcp.interfaces == {"lan"} # "LAN" + empty action
lockout = by_desc["anti-lockout rule"]
assert lockout.interfaces == {"lan"} and lockout.automatic and lockout.legacy
allow = by_desc["Default allow LAN to any rule"]
assert allow.section == "interface" and allow.quick and allow.af == {4} and not allow.automatic
spoof = [r for r in rules if r.section == "antispoof"]
assert spoof and spoof[0].interfaces == {"lan"} and spoof[0].if_not and spoof[0].quick is False
# Antispoof lines sit before the first automatic rule, like in pf.
assert rules.index(spoof[0]) < rules.index(deny)
def test_normalize_reports_order_mismatch_and_missing_sort_order():
ctx = build([mvc("r-a", seq=1)])
# Swap pf positions: the rule now appears in pf before the system rules.
key = next(k for k in ctx.pf.by_label if k == "r-a")
ctx.pf.by_label[key][0].number = -1
assert "ORDER_MISMATCH" in normalize(ctx)[1]
row = mvc("r-b")
row.pop("sort_order")
assert "ORDER_GUESSED" in normalize(build([row]))[1]
def test_unmigrated_legacy_rule_with_interface_description():
# Seen live: legacy user rules have a 32-hex label and "WAN" instead of "wan".
legacy = mvc("0123456789abcdef0123456789abcdef", seq=1, action="block", interface="WAN", legacy=True)
r = run(build([legacy], drop=("Allow ALL",)), src=INTERNET, dst=WAN_IP, iface="wan")
assert decided_by(r) == legacy["uuid"] and r["confidence"] == "sure"
assert r["decisive_rule"]["legacy"] and not r["decisive_rule"]["editable"]
# ── Engine: matrix 1-8, 14-20 ─────────────────────────────────────────────
def test_01_lan_client_to_internet_is_default_allow_sure():
r = run(build())
assert r["action"] == "pass" and decided_by(r) == "Default allow LAN to any rule"
assert r["confidence"] == "sure" and r["rule_confidence"] == "sure" and r["unknowns"] == []
def test_02_wan_to_firewall_ssh_hits_default_deny():
r = run(build(drop=("Allow ALL",)), src=INTERNET, dst=WAN_IP, dport="22", iface="wan")
assert r["action"] == "block" and decided_by(r) == "Default deny / state violation rule"
assert r["confidence"] == "sure" and r["decisive_rule"]["quick"] is False
def test_03_floating_non_quick_block_loses_to_interface_quick_pass():
ctx = build([mvc("float-block", prio=200000, action="block", quick="0", interface="")])
r = run(ctx)
assert r["action"] == "pass" and decided_by(r) == "Default allow LAN to any rule"
effects = {s["rule"]["description"]: s["effect"] for s in r["trace"][0]["steps"]}
assert effects["float-block"] == "overridden"
def test_04_floating_quick_block_on_lan_and_wan_wins():
r = run(build([mvc("float-quick", prio=200000, action="block", quick="1", interface="lan,wan")]))
assert r["action"] == "block" and decided_by(r) == "float-quick" and r["confidence"] == "sure"
assert r["decisive_rule"]["section"] == "floating"
def test_05_floating_interfacenot_applies_to_the_other_interfaces():
ctx = build([mvc("not-lan", prio=200000, action="block", interface="lan", interfacenot="1")])
assert decided_by(run(ctx)) == "Default allow LAN to any rule"
wan = run(ctx, src=INTERNET, dst=WAN_IP, dport="22", iface="wan")
assert wan["action"] == "block" and decided_by(wan) == "not-lan"
def test_06_group_rule_comes_before_interface_rules():
ctx = build([mvc("grp-block", prio=300005, seq=900, action="block", interface="office")], groups={"office": ["lan"]})
r = run(ctx)
assert r["action"] == "block" and decided_by(r) == "grp-block" and r["decisive_rule"]["section"] == "group"
def test_07_last_non_quick_match_wins():
ctx = build([mvc("first-pass", seq=1, quick="0", interface="wan"), mvc("then-block", seq=2, action="block", quick="0", interface="wan")],
drop=("Allow ALL",))
r = run(ctx, src=INTERNET, dst=WAN_IP, iface="wan")
assert r["action"] == "block" and decided_by(r) == "then-block" and r["confidence"] == "sure"
def test_08_negated_source_alias():
aliases = {"BADHOSTS": {"type": "host", "content": "10.20.30.66"}}
rule = mvc("not-bad", seq=1, action="block", src="BADHOSTS", source_not="1")
ctx = build([rule], aliases=aliases, refs={LAN_CLIENT: ["__lan_network"], "10.20.30.66": ["BADHOSTS", "__lan_network"]})
assert run(ctx)["action"] == "block" # .50 is not in BADHOSTS -> !BADHOSTS matches
assert run(ctx, src="10.20.30.66")["action"] == "pass" # in BADHOSTS -> negation does not match
# Table membership unknown: the negation stays unknown, the verdict is not "sure".
unknown = build([dict(rule, source_net="FEED")], aliases={"FEED": {"type": "urltable", "content": ""}},
refs={LAN_CLIENT: None})
r = run(unknown)
assert r["action"] == "pass" and r["confidence"] == "unknown" and "ALIAS_TABLE_UNAVAILABLE" in r["unknowns"]
def test_14_rule_not_loaded_is_skipped():
ctx = build([mvc("scheduled", seq=1, action="block", sched="weekends", loaded=False)])
r = run(ctx)
assert decided_by(r) == "Default allow LAN to any rule"
step = next(s for s in r["trace"][0]["steps"] if s["rule"]["description"] == "scheduled")
assert step["effect"] == "not_loaded" and "RULES_NOT_LOADED" in r["notes"]
def test_15_16_gateway_and_stateless_notes():
r = run(build([mvc("via-gw", seq=1, gateway="WAN_GW", statetype="none")]))
assert decided_by(r) == "via-gw" and {"POLICY_ROUTE", "STATELESS"} <= set(r["notes"])
assert r["confidence"] == "sure"
def test_17_tagged_rule_is_never_sure():
r = run(build([mvc("tagged-block", seq=1, action="block", tagged="FROM_NAT")]))
assert r["confidence"] == "unknown" and "TAGGED" in r["unknowns"]
def test_18_unknown_rule_with_same_action_keeps_the_verdict_sure():
ctx = build([mvc("maybe-pass", seq=1, src="GHOST")]) # alias unknown on the firewall
r = run(ctx)
assert r["action"] == "pass" and r["confidence"] == "sure" and r["rule_confidence"] == "probable"
def test_19_unknown_rule_with_other_action_makes_the_verdict_unknown():
ctx = build([mvc("maybe-block", seq=1, action="block", src="GHOST")])
r = run(ctx)
assert r["action"] == "pass" and r["confidence"] == "unknown" and "ALIAS_MISSING" in r["unknowns"]
assert any(s["effect"] == "possible" for s in r["trace"][0]["steps"])
def test_20_rule_with_missing_interface_never_matches():
row = mvc("orphan", prio=600000, action="pass", interface="opt9", loaded=False)
r = run(build([row], drop=("Allow ALL",)), src=INTERNET, dst=WAN_IP, iface="wan")
assert decided_by(r) == "Default deny / state violation rule"
assert next(s for s in r["trace"][0]["steps"] if s["rule"]["description"] == "orphan")["effect"] == "not_loaded"
# Even without a pf ruleset a 600000 rule counts as not loaded; the verdict is no longer sure.
no_pf = run(build([row], pf=False))
assert decided_by(no_pf) == "Default allow LAN to any rule" and no_pf["confidence"] != "sure"
def test_guessed_interface_and_nat_nearby_lower_confidence():
r = run(build(), iface=None)
assert r["legs"][0]["interface"] == "lan" and "IFACE_GUESSED" in r["notes"] and r["confidence"] == "probable"
nat = run(build(port_forwards=[{"interface": "lan", "protocol": "tcp", "destination.port": "443"}]))
assert "NAT_NEARBY" in nat["notes"] and nat["confidence"] == "probable"
# "no rdr" exclusions and forwards for other ports do not count.
other = run(build(port_forwards=[{"interface": "lan", "protocol": "tcp", "destination.port": "22", "nordr": "0"},
{"interface": "lan", "protocol": "tcp", "destination.port": "443", "nordr": "1"}]))
assert "NAT_NEARBY" not in other["notes"] and other["confidence"] == "sure"
def test_egress_leg_blocks_when_out_rule_blocks():
ctx = build([mvc("no-out", prio=200000, action="block", interface="wan", direction="out")])
r = run(ctx, egress="wan")
assert [leg["action"] for leg in r["legs"]] == ["pass", "block"]
assert r["action"] == "block" and decided_by(r) == "no-out" and r["confidence"] == "sure"
# Without the block the automatic "let out anything" rule passes the out leg.
ok = run(build(), egress="wan")
assert ok["action"] == "pass" and ok["legs"][1]["decisive_rule"]["description"] == "let out anything from firewall host itself"
# ── Resolver: matrix 9-13 ─────────────────────────────────────────────────
def _ip(text):
return ipaddress.ip_address(text)
def test_09_nested_aliases_and_cycles():
ctx = build(aliases={
"OUTER": {"type": "host", "content": "MIDDLE\n192.0.2.9"},
"MIDDLE": {"type": "network", "content": "10.20.30.0/25\nINNER"},
"INNER": {"type": "host", "content": "10.20.30.200"},
"LOOP_A": {"type": "host", "content": "LOOP_B"}, "LOOP_B": {"type": "host", "content": "LOOP_A"},
"NAMED": {"type": "host", "content": "server.example.org"},
})
ctx.refs = {}
assert resolve.alias_contains("OUTER", _ip("10.20.30.5"), ctx).value is True
assert resolve.alias_contains("OUTER", _ip("10.20.30.200"), ctx).value is True
assert resolve.alias_contains("OUTER", _ip("10.20.30.150"), ctx).value is False
loop = resolve.alias_contains("LOOP_A", _ip("10.20.30.5"), ctx)
assert loop.value is None and "ALIAS_CYCLE" in loop.reasons
fqdn = resolve.alias_contains("NAMED", _ip("10.20.30.5"), ctx)
assert fqdn.value is None and "FQDN_UNRESOLVED" in fqdn.reasons
assert resolve.alias_contains("NOPE", _ip("10.20.30.5"), ctx).value is None
def test_10_table_aliases_need_find_references():
ctx = build(aliases={"FEED": {"type": "urltable", "content": "https://example.org/list"}})
ctx.refs = {"192.0.2.7": ["FEED"], "192.0.2.8": []}
assert resolve.alias_contains("FEED", _ip("192.0.2.7"), ctx).value is True
assert resolve.alias_contains("FEED", _ip("192.0.2.8"), ctx).value is False
ctx.refs = {"192.0.2.7": None}
out = resolve.alias_contains("FEED", _ip("192.0.2.7"), ctx)
assert out.value is None and out.reasons == ["ALIAS_TABLE_UNAVAILABLE"]
def test_11_address_family():
ctx = build([mvc("v4-only", seq=1, action="block", interface="wan", ipprotocol="inet")], drop=("Allow ALL",))
v6 = run(ctx, src="2001:db8::5", dst="2001:db8:1::1", iface="wan")
assert decided_by(v6) == "Default deny / state violation rule"
both = build([mvc("v46", seq=1, action="reject", interface="wan", ipprotocol="inet46")], drop=("Allow ALL",))
assert decided_by(run(both, src="2001:db8::5", dst="2001:db8:1::1", iface="wan")) == "v46"
def test_12_port_ranges_protocol_pairs_and_nested_port_aliases():
ctx = build(aliases={"WEB": {"type": "port", "content": "1000:2000\nMORE"}, "MORE": {"type": "port", "content": "https\n8000-8001"}})
for port, expected in ((999, False), (1000, True), (2000, True), (2001, False), (443, True), (8001, True), (8002, False)):
assert resolve.port_match("WEB", port, ctx).value is expected, port
assert resolve.port_match("1000:2000", 1500, ctx).value is True
rule = mvc("tcpudp", seq=1, action="block", proto="TCP/UDP", dport="1000-2000")
assert decided_by(run(build([rule]), proto="udp", dport="1000")) == "tcpudp"
assert decided_by(run(build([rule]), proto="tcp", dport="2001")) == "Default allow LAN to any rule"
# No source port given: assumed ephemeral, so "port 0" rules do not match, a narrow one is unknown.
assert resolve.port_match("0", None, ctx, ephemeral_default=True).value is False
assert resolve.port_match("5000", None, ctx, ephemeral_default=True).value is None
assert resolve.port_match("1024-65535", None, ctx, ephemeral_default=True).value is True
def test_13_interface_networks_addresses_self_and_carp_vip():
ctx = build()
vip, primary, client = _ip("10.20.30.1"), _ip("10.20.30.2"), _ip("10.20.30.50")
assert resolve.addr_item("lan", vip, ctx).value is True and resolve.addr_item("lan", client, ctx).value is True
assert resolve.addr_item("lanip", primary, ctx).value is True
assert resolve.addr_item("lanip", client, ctx).value is False
assert resolve.addr_item("lanip", vip, ctx).value is None # VIP vs interface address: not provable
assert resolve.addr_item("(self)", vip, ctx).value is True
assert resolve.addr_item("(self)", _ip(WAN_IP), ctx).value is True
assert resolve.addr_item("(self)", client, ctx).value is False
assert resolve.addr_item("__wan_network", _ip("203.0.113.9"), ctx).value is True
def test_query_validation_errors():
from fastapi import HTTPException
for kwargs, code in (({"src": "host.example", "dst": INTERNET}, "ANALYSE_ADRESSE_UNGUELTIG"),
({"src": "2001:db8::1", "dst": INTERNET}, "ANALYSE_FAMILIEN_MISCHUNG"),
({"src": LAN_CLIENT, "dst": INTERNET, "dport": "70000"}, "ANALYSE_PARAMETER_UNGUELTIG"),
({"src": LAN_CLIENT, "dst": INTERNET, "proto": "icmp", "dport": "80"}, "ANALYSE_PARAMETER_UNGUELTIG")):
with pytest.raises(HTTPException) as exc:
build_query(**kwargs)
assert exc.value.detail["code"] == code
+88
View File
@@ -0,0 +1,88 @@
"""Rule analysis API + MCP tool (matrix case 25): GET only, read tokens allowed, nothing written."""
import json
import httpx
import pytest
from app.services.rule_analysis import context as ctx_mod
from tests.test_api_tokens import _bearer, _make
from tests.test_rule_analysis import FIXTURE
from tests.test_rules import _create_host
FW, DOWN = "https://198.51.100.200", "https://198.51.100.201"
@pytest.fixture(autouse=True)
def _fresh_cache():
ctx_mod.clear_cache()
yield
ctx_mod.clear_cache()
def _mock_fw(respx_mock, base=FW):
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
respx_mock.get(f"{base}/api/firewall/filter/searchRule").mock(return_value=httpx.Response(200, json=FIXTURE["rules"]))
respx_mock.get(f"{base}/api/diagnostics/firewall/pfStatistics/rules").mock(return_value=httpx.Response(200, json=FIXTURE["pf"]))
respx_mock.get(f"{base}/api/interfaces/overview/interfacesInfo").mock(return_value=httpx.Response(200, json=FIXTURE["interfaces"]))
respx_mock.get(f"{base}/api/firewall/group/searchItem").mock(return_value=httpx.Response(200, json=FIXTURE["groups"]))
respx_mock.get(f"{base}/api/firewall/alias/searchItem").mock(return_value=httpx.Response(200, json=FIXTURE["aliases"]))
respx_mock.get(f"{base}/api/firewall/d_nat/searchRule").mock(return_value=httpx.Response(200, json={"rows": []}))
return respx_mock.post(f"{base}/api/firewall/alias_util/findReferences").mock(
return_value=httpx.Response(200, json={"status": "ok", "matches": []}))
Q = {"src": "10.20.30.50", "dst": "198.51.100.10", "proto": "tcp", "dport": "443", "iface": "lan"}
@pytest.mark.asyncio
async def test_host_match_and_validation(client, respx_mock):
refs = _mock_fw(respx_mock)
host = await _create_host(client, "fw-a", FW)
resp = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params=Q)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["action"] == "pass" and body["confidence"] == "sure" and body["error"] is None
assert body["decisive_rule"]["description"] == "Default allow LAN to any rule"
assert body["legs"][0]["interface"] == "lan" and body["trace"][0]["steps"]
assert refs.call_count == 2 and json.loads(refs.calls[0].request.content) == {"ip": "10.20.30.50"}
bad = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params={**Q, "src": "pc.example"})
assert bad.status_code == 422 and bad.json()["detail"]["code"] == "ANALYSE_ADRESSE_UNGUELTIG"
mixed = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params={**Q, "dst": "2001:db8::1"})
assert mixed.json()["detail"]["code"] == "ANALYSE_FAMILIEN_MISCHUNG"
iface = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params={**Q, "iface": "opt7"})
assert iface.status_code == 422 and iface.json()["detail"]["code"] == "ANALYSE_INTERFACE_UNBEKANNT"
@pytest.mark.asyncio
async def test_fleet_match_reports_unreachable_hosts_per_host(client, respx_mock):
_mock_fw(respx_mock)
respx_mock.route(url__startswith=DOWN).mock(side_effect=httpx.ConnectError("down"))
await _create_host(client, "fw-a", FW)
await _create_host(client, "fw-down", DOWN)
resp = await client.get("/api/rule-analysis/match", params={**Q, "iface": "auto"})
assert resp.status_code == 200, resp.text
by_name = {r["host_name"]: r for r in resp.json()}
assert by_name["fw-a"]["action"] == "pass" and "IFACE_GUESSED" in by_name["fw-a"]["notes"]
assert by_name["fw-a"]["confidence"] == "probable"
assert by_name["fw-down"]["error"]["code"] == "ANALYSE_FIREWALL_NICHT_LESBAR"
@pytest.mark.asyncio
async def test_25_mcp_rule_match_with_a_read_token_changes_nothing(client, respx_mock):
_mock_fw(respx_mock)
host = await _create_host(client, "fw-a", FW)
token = await _make(client, "read")
async with _bearer(token["token"]) as api:
direct = await api.get(f"/api/hosts/{host['id']}/rule-analysis/match", params=Q)
assert direct.status_code == 200
rpc = {"jsonrpc": "2.0", "id": 1, "method": "tools/call",
"params": {"name": "rule_match", "arguments": {**Q, "dport": 443, "host_ids": host["id"]}}}
result = (await api.post("/api/mcp", json=rpc)).json()["result"]
tools = (await api.post("/api/mcp", json={"jsonrpc": "2.0", "id": 2, "method": "tools/list"})).json()["result"]["tools"]
assert result["isError"] is False
payload = json.loads(result["content"][0]["text"])
assert payload[0]["action"] == "pass" and "trace" not in payload[0]
assert "rule_match" in {t["name"] for t in tools}
# The only POST that reached the firewall is the read-only pf table lookup.
posts = {c.request.url.path for c in respx_mock.calls if c.request.method != "GET"}
assert posts == {"/api/firewall/alias_util/findReferences"}
+306
View File
@@ -0,0 +1,306 @@
"""auto_verify (1.25): an unattended action checks its own result and closes the operation,
so the host/cluster is not left locked until a human clicks "Wartung abschließen"."""
import asyncio
import uuid
from datetime import datetime, timedelta, timezone
import httpx
import pytest
from sqlalchemy import select
from app.database import SessionLocal
from app.models import HostOperation
from app.models.audit import AuditEvent
from app.models.scheduled_maintenance import ScheduledMaintenance
from app.services import scheduled_maintenance_service as svc
from tests.test_rules import _create_host
from tests.test_scheduled_handover import FakeCluster, _setup
H = "https://fw-solo.test"
class FakeHost:
"""One stand-alone firewall: job = ready | running | down; boot = what systemTime reports."""
def __init__(self, respx_mock, *, firmware="update", reboot_flag="1", version="26.7.5"):
self.job, self.boot, self.version = "ready", "Mon Oct 5 10:00:00 UTC 2026", version
self.update = respx_mock.post(f"{H}/api/core/firmware/update").respond(200, json={"status": "ok"})
self.reboot = respx_mock.post(f"{H}/api/core/system/reboot").respond(200, json={"status": "ok"})
respx_mock.get(f"{H}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
respx_mock.get(f"{H}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
respx_mock.get(f"{H}/api/core/firmware/info").mock(side_effect=lambda r: self._up(json={"product_version": self.version}))
respx_mock.get(f"{H}/api/core/firmware/status").respond(
200, json={"status": firmware, "all_packages": {}, "status_msg": "x", "status_reboot": reboot_flag})
respx_mock.get(f"{H}/api/core/firmware/running").mock(side_effect=lambda r: self._up(json={"status": self.job}))
respx_mock.get(f"{H}/api/diagnostics/system/systemTime").mock(side_effect=lambda r: self._up(json={"boottime": self.boot}))
def _up(self, **kw):
if self.job == "down":
raise httpx.ConnectError("rebooting")
return httpx.Response(200, **kw)
def _vip(self, request):
return self._up(json={"rows": [], "carp": {"allow": "1", "maintenancemode": False, "demotion": "0", "status_msg": ""}})
def reboot_now(self, new_boot="Wed Oct 21 03:00:00 UTC 2026", version=None):
self.boot = new_boot
if version:
self.version = version
async def _plan(host, action="update", *, auto_verify=True, handover=False):
async with SessionLocal() as db:
s = ScheduledMaintenance(
host_id=uuid.UUID(host["id"]), host_name=host["name"], action=action, carp_handover=handover,
auto_verify=auto_verify, run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin",
created_at=datetime.now(timezone.utc))
db.add(s)
await db.commit()
return s.id
async def _fire():
async with SessionLocal() as db:
return await svc.fire_due(db)
async def _advance():
async with SessionLocal() as db:
return await svc.advance_running(db)
async def _get(sid):
async with SessionLocal() as db:
s = await db.get(ScheduledMaintenance, sid)
op = await db.get(HostOperation, s.host_operation_id) if s.host_operation_id else None
return s, op
async def _shift(sid, *, phase=None, ready=None):
"""Pretend the phase began / the firmware turned ready that long ago."""
async with SessionLocal() as db:
s = await db.get(ScheduledMaintenance, sid)
if phase is not None:
s.phase_since = datetime.now(timezone.utc) - phase
if ready is not None:
s.progress = {**s.progress, "ready_since": (datetime.now(timezone.utc) - ready).isoformat()}
await db.commit()
async def _settle(sid):
"""verify needs the firmware idle for VERIFY_SETTLE: one tick to notice, then pretend it has been a while."""
await _advance()
await _shift(sid, ready=svc.VERIFY_SETTLE + timedelta(seconds=1))
await _advance()
@pytest.mark.asyncio
async def test_update_is_verified_and_the_operation_closed(client, respx_mock):
fw = FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host)
await _fire()
s, op = await _get(sid)
assert (s.status, s.phase, op.status, fw.update.call_count) == ("running", "job_wait", "submitted", 1)
assert s.progress["reboot_expected"] is True and s.progress["boot_before"] == fw.boot
fw.job = "running"
await _advance()
fw.job = "down"
await _advance()
fw.job = "ready"
fw.reboot_now(version="26.7.7")
await _advance()
assert (await _get(sid))[0].phase == "verify"
await _settle(sid)
s, op = await _get(sid)
assert s.status == "succeeded" and s.phase is None
assert s.result["auto_verified"] is True and s.result["version_after"] == "26.7.7" and s.result["version_before"] == "26.7.5"
assert op.status == "acknowledged" and "Automatisch geprüft" in op.message
assert fw.update.call_count == 1
# the host is free again: a new preview is not blocked by a forgotten operation
resp = await client.get(f"/api/hosts/{host['id']}/maintenance")
assert not any("frühere Wartungsaktion" in b for b in resp.json()["state"]["blocked"])
async with SessionLocal() as db:
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
assert {e.actor for e in events if (e.details or {}).get("scheduled")} == {"admin (geplant)"}
@pytest.mark.asyncio
async def test_boot_time_proves_a_reboot_the_poll_never_saw(client, respx_mock):
fw = FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host)
await _fire()
await _shift(sid, phase=timedelta(0))
fw.reboot_now() # rebooted between two polls: never seen offline
async with SessionLocal() as db: # the job grace has passed
s = await db.get(ScheduledMaintenance, sid)
s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
await db.commit()
await _advance()
assert (await _get(sid))[0].phase == "verify"
await _settle(sid)
s, op = await _get(sid)
assert s.status == "succeeded" and s.result["rebooted"] is True and op.status == "acknowledged"
@pytest.mark.asyncio
async def test_expected_reboot_never_happened_ends_unknown_after_timeout(client, respx_mock):
FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host)
await _fire()
async with SessionLocal() as db:
s = await db.get(ScheduledMaintenance, sid)
s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
await db.commit()
await _advance()
await _settle(sid)
s, op = await _get(sid)
assert (s.status, s.phase, op.status) == ("running", "verify", "submitted") # waiting, not done
assert "Neustart" in s.progress["waiting_for"]
await _shift(sid, phase=svc.VERIFY_TIMEOUT)
await _advance()
s, op = await _get(sid)
assert s.status == "failed" and op.status == "unknown" and "nicht bestanden" in s.result["error"]
@pytest.mark.asyncio
async def test_ready_must_hold_a_while_a_short_ready_before_the_reboot_does_not_count(client, respx_mock):
fw = FakeHost(respx_mock, reboot_flag="0")
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host)
await _fire()
fw.job = "running"
await _advance()
fw.job = "ready"
await _advance() # job_wait -> verify
await _advance() # ready noticed
s, _ = await _get(sid)
assert s.phase == "verify" and s.progress["ready_since"]
fw.job = "running" # busy again (e.g. the reboot at the end of the update)
await _advance()
assert (await _get(sid))[0].progress["ready_since"] is None
fw.job = "ready"
await _settle(sid)
assert (await _get(sid))[0].status == "succeeded"
@pytest.mark.asyncio
async def test_without_auto_verify_nothing_changes(client, respx_mock):
fw = FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host, auto_verify=False)
await _fire()
s, op = await _get(sid)
assert (s.status, s.phase, op.status, fw.update.call_count) == ("succeeded", None, "submitted", 1)
assert await _advance() == 0
@pytest.mark.asyncio
async def test_human_closing_the_operation_stops_the_check(client, respx_mock):
FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host)
await _fire()
s, op = await _get(sid)
async with SessionLocal() as db:
(await db.get(HostOperation, op.id)).status = "cancelled"
await db.commit()
await _advance()
s, op = await _get(sid)
assert s.status == "failed" and "von Hand beendet" in s.result["error"] and op.status == "cancelled"
@pytest.mark.asyncio
async def test_handover_verifies_before_taking_the_traffic_back(client, respx_mock):
cluster = FakeCluster(respx_mock)
cluster.boot = "before"
def system_time(request):
if cluster.job == "down":
raise httpx.ConnectError("down")
return httpx.Response(200, json={"boottime": cluster.boot})
respx_mock.get("https://fw-a.test/api/diagnostics/system/systemTime").mock(side_effect=system_time)
a, _b = await _setup(client)
async with SessionLocal() as db:
s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True, auto_verify=True,
run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
db.add(s)
await db.commit()
sid = s.id
await _fire()
await _advance() # partner MASTER -> job submitted
cluster.job = "down"
await _advance()
cluster.job = "ready"
cluster.boot = "after"
await _advance()
s, _ = await _get(sid)
assert s.phase == "verify" and cluster.toggles.call_count == 1 # still demoted while checking
await _settle(sid)
assert (await _get(sid))[0].phase == "leave" and cluster.toggles.call_count == 1
await _advance() # leave
await _advance() # leave_wait -> done
s, op = await _get(sid)
assert s.status == "succeeded" and s.result["handover"] is True and s.result["auto_verified"] is True
assert s.result["rebooted"] is True # found live: the handover path dropped what verify had seen
assert op.status == "acknowledged" and cluster.toggles.call_count == 2 and cluster.update.call_count == 1
@pytest.mark.asyncio
async def test_a_due_row_locked_by_another_backend_is_not_fired_twice(client, respx_mock):
fw = FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
sid = await _plan(host)
async with SessionLocal() as other: # another backend is firing this very row right now
await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
# Without the row lock this would try to update the locked row and hang: bounded, so it fails instead.
result = await asyncio.wait_for(_fire(), timeout=10)
assert result["due"] == 0 and fw.update.call_count == 0
await other.rollback()
await _fire()
assert fw.update.call_count == 1
@pytest.mark.asyncio
async def test_a_running_row_locked_by_another_backend_is_not_stepped_twice(client, respx_mock):
cluster = FakeCluster(respx_mock)
a, _b = await _setup(client)
async with SessionLocal() as db:
s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True,
run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
db.add(s)
await db.commit()
sid = s.id
await _fire()
async with SessionLocal() as other:
await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
assert await asyncio.wait_for(_advance(), timeout=10) == 1 # still counted as running, but not touched
assert cluster.update.call_count == 0
await other.rollback()
await _advance()
assert cluster.update.call_count == 1
@pytest.mark.asyncio
async def test_cancel_waits_for_a_firing_row_and_never_overwrites_running(client, respx_mock):
FakeHost(respx_mock)
host = await _create_host(client, "fw-solo", H)
async with SessionLocal() as db:
s = ScheduledMaintenance(host_id=uuid.UUID(host["id"]), host_name="fw-solo", action="update", auto_verify=True,
run_at=datetime.now(timezone.utc) + timedelta(hours=1), actor="admin", created_at=datetime.now(timezone.utc))
db.add(s)
await db.commit()
sid = s.id
async with SessionLocal() as firing: # the poll loop has the row and is about to mark it running
row = await firing.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
cancel = asyncio.create_task(client.delete(f"/api/hosts/{host['id']}/maintenance/schedule/{sid}"))
await asyncio.sleep(0.5)
row.status = "running"
await firing.commit()
resp = await asyncio.wait_for(cancel, timeout=10)
assert resp.status_code == 409
assert (await _get(sid))[0].status == "running"
+137
View File
@@ -0,0 +1,137 @@
"""1.27 prerequisite hardening: __Host- session cookie (with soft migration) and
the Sec-Fetch-Site check for cookie-authenticated /api requests."""
import pytest
from httpx import ASGITransport, AsyncClient
from app import main
from app.config import settings
from app.main import app
from app.routers.auth import COOKIE_NAME, SECURE_COOKIE_NAME
PW = "tokpass-12345678"
def _https_client() -> AsyncClient:
return AsyncClient(transport=ASGITransport(app=app), base_url="https://test")
async def _login(ac: AsyncClient) -> None:
resp = await ac.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
assert resp.status_code == 200, resp.text
def _set_cookies(resp) -> list[str]:
return resp.headers.get_list("set-cookie")
@pytest.mark.asyncio
async def test_secure_login_sets_host_prefixed_cookie_and_drops_legacy(monkeypatch):
monkeypatch.setattr(settings, "cookie_secure", True)
async with _https_client() as ac:
resp = await ac.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
assert resp.status_code == 200
cookies = _set_cookies(resp)
new = [c for c in cookies if c.startswith(f"{SECURE_COOKIE_NAME}=")]
assert new and "Secure" in new[0] and "Path=/" in new[0] and "HttpOnly" in new[0] and "Domain" not in new[0]
assert any(c.startswith(f'{COOKIE_NAME}=""') or c.startswith(f"{COOKIE_NAME}=;") or "Max-Age=0" in c
for c in cookies if c.startswith(f"{COOKIE_NAME}="))
assert (await ac.get("/api/hosts")).status_code == 200
@pytest.mark.asyncio
async def test_plain_name_cookie_without_https():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
resp = await ac.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
assert any(c.startswith(f"{COOKIE_NAME}=") for c in _set_cookies(resp))
assert not any(c.startswith(SECURE_COOKIE_NAME) for c in _set_cookies(resp))
@pytest.mark.asyncio
async def test_legacy_cookie_is_migrated_not_logged_out(monkeypatch):
# A session issued before the upgrade (plain cookie name) ...
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
await _login(ac)
legacy = ac.cookies.get(COOKIE_NAME)
assert legacy
monkeypatch.setattr(settings, "cookie_secure", True)
async with _https_client() as ac:
ac.cookies.set(COOKIE_NAME, legacy)
assert (await ac.get("/api/hosts")).status_code == 200 # still accepted
me = await ac.get("/api/auth/me")
assert me.json()["authenticated"] is True
moved = [c for c in _set_cookies(me) if c.startswith(f"{SECURE_COOKIE_NAME}={legacy}")]
assert moved and "Secure" in moved[0]
@pytest.mark.asyncio
async def test_host_cookie_wins_over_tossed_plain_cookie(monkeypatch):
monkeypatch.setattr(settings, "cookie_secure", True)
async with _https_client() as ac:
await _login(ac)
real = ac.cookies.get(SECURE_COOKIE_NAME)
async with _https_client() as ac:
# A script on another port may set a plain-name cookie; it is ignored ...
ac.cookies.set(SECURE_COOKIE_NAME, real)
ac.cookies.set(COOKIE_NAME, "attacker-token")
assert (await ac.get("/api/hosts")).status_code == 200
async with _https_client() as ac:
# ... and a bad __Host- cookie never falls back to the plain one.
ac.cookies.set(SECURE_COOKIE_NAME, "garbage")
ac.cookies.set(COOKIE_NAME, real)
assert (await ac.get("/api/hosts")).status_code == 401
@pytest.mark.asyncio
@pytest.mark.parametrize("site,expected", [
("same-site", 403), ("cross-site", 403), ("Cross-Site", 403),
("same-origin", 200), ("none", 200), (None, 200),
])
async def test_sec_fetch_site_on_cookie_requests(client, site, expected):
headers = {"Sec-Fetch-Site": site} if site else {}
resp = await client.get("/api/hosts", headers=headers)
assert resp.status_code == expected, resp.text
if expected == 403:
assert resp.json()["detail"]["code"] == "CROSS_SITE_REQUEST_BLOCKED"
@pytest.mark.asyncio
async def test_sec_fetch_same_site_allowed_only_for_configured_frontend(client, monkeypatch):
monkeypatch.setattr(settings, "frontend_origin", "https://opnexus.example:3100")
ok = await client.get("/api/hosts", headers={"Sec-Fetch-Site": "same-site", "Origin": "https://opnexus.example:3100"})
assert ok.status_code == 200
slot = await client.get("/api/hosts", headers={"Sec-Fetch-Site": "same-site", "Origin": "https://opnexus.example:9441"})
assert slot.status_code == 403
# Writes from a slot page are refused as well (before auth even runs).
write = await client.post("/api/auth/logout", headers={"Sec-Fetch-Site": "same-site", "Origin": "https://opnexus.example:9441"})
assert write.status_code == 403
@pytest.mark.asyncio
async def test_sec_fetch_does_not_touch_token_requests(client, anon_client):
created = (await client.post("/api/tokens", json={"name": "sfs", "scope": "read", "current_password": settings.admin_password})).json()
resp = await anon_client.get("/api/hosts", headers={"Authorization": f"Bearer {created['token']}", "Sec-Fetch-Site": "cross-site"})
assert resp.status_code == 200
@pytest.mark.asyncio
async def test_sec_fetch_exempts_sso_callback(client):
resp = await client.get("/api/auth/oidc/callback?code=x&state=y", headers={"Sec-Fetch-Site": "cross-site"}, follow_redirects=False)
assert resp.status_code != 403 or resp.json().get("detail", {}).get("code") != "CROSS_SITE_REQUEST_BLOCKED"
@pytest.mark.asyncio
async def test_startup_refused_when_slot_origin_is_a_frontend_origin(monkeypatch):
monkeypatch.setattr(settings, "frontend_origin", "https://opnexus.example, https://opnexus.example:9441")
monkeypatch.setattr(settings, "opnexus_gui_sites", "https://opnexus.example:9441,https://opnexus.example:9442")
assert settings.gui_origin_conflicts() == ["https://opnexus.example:9441"]
called = []
async def _no_migrate():
called.append(1)
monkeypatch.setattr(main.migrate, "run", _no_migrate)
with pytest.raises(RuntimeError):
async with main.lifespan(app):
pass
assert not called
monkeypatch.setattr(settings, "opnexus_gui_sites", "https://opnexus.example:9442")
assert settings.gui_origin_conflicts() == []
+1
View File
@@ -47,6 +47,7 @@ async def test_status_reports_update_and_can_update(client, respx_mock):
body = (await client.get("/api/system/updater")).json()
assert body["available"] is True and body["can_update"] is True
assert body["latest"] == _bump(minor=1) and body["major_change"] is False
assert body["active_rollouts"] == 0 # the UI warns before restarting OPNexus in the middle of a rollout
@pytest.mark.asyncio
+4
View File
@@ -0,0 +1,4 @@
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEA4pZWkNMkAtCGc54M2wuWy7CXwkT
3PCysv1/1IbkyJRPdM8+DFf26fBxWG9P9PLKfwavhqVYtHILxxzxKmtjeQ==
-----END PUBLIC KEY-----
+46
View File
@@ -124,6 +124,10 @@ services:
# Optionaler Updater-Dienst (Profil "updater", siehe unten). Ohne ihn
# zeigt das Webinterface nur an, wie man ihn aktiviert.
UPDATER_URL: ${UPDATER_URL:-http://updater:8080}
# Optionaler GUI-Fernzugriff (Profil "gui", siehe gui-gateway unten).
# Leer = aus; das Backend lässt den Schalter dann nicht einschalten.
OPNEXUS_GUI_SITES: ${OPNEXUS_GUI_SITES:-}
GUI_GATEWAY_TOKEN: ${GUI_GATEWAY_TOKEN:-}
volumes:
- secrets:/data/opnsense-secrets
- backups:/data/opnsense-backups
@@ -197,9 +201,19 @@ services:
environment:
OPNEXUS_SITES: ${OPNEXUS_SITES:-localhost}
OPNEXUS_DEFAULT_SNI: ${OPNEXUS_DEFAULT_SNI:-localhost}
OPNEXUS_GUI_SITES: ${OPNEXUS_GUI_SITES:-https://localhost:9441}
ports:
- "${PROXY_BIND_ADDRESS:-0.0.0.0}:${PROXY_HTTPS_PORT:-443}:443"
- "${PROXY_BIND_ADDRESS:-0.0.0.0}:${PROXY_HTTP_PORT:-80}:80"
# Slot-Ports des GUI-Fernzugriffs (Profil "gui"). Standardmäßig nur auf
# 127.0.0.1; ohne den gui-gateway-Dienst antworten sie nur mit 502.
# Host- und Container-Port sind absichtlich gleich: die Ports stehen so
# auch in OPNEXUS_GUI_SITES, und der Proxy meldet dem Gateway den Port,
# auf dem die Verbindung ankam.
# Ohne GUI_PORTS belegt Docker zufällige freie Host-Ports: ein Update darf
# den Proxy nicht daran scheitern lassen, dass 9441-9444 schon vergeben
# sind (z. B. Portainer auf 0.0.0.0:9443). Zum Einschalten GUI_PORTS setzen.
- "${GUI_BIND_ADDRESS:-127.0.0.1}:${GUI_PORTS:-}:${GUI_PORTS:-9441-9444}"
volumes:
- caddy_data:/data
- caddy_config:/config
@@ -228,6 +242,38 @@ services:
ports:
- "${OVERLAY_PORT:-51820}:51820/udp"
# ── Optional: GUI-Fernzugriff auf verwaltete Firewalls ──────────────
# Standardmäßig NICHT gestartet und dreifach aus: Profil "gui"
# (COMPOSE_PROFILES=gui), GUI_BIND_ADDRESS (Standard 127.0.0.1) und der
# Schalter unter Einstellungen. Gleiches Image wie das Backend, aber ohne
# Datenbank, ohne Secret-Volumes und ohne eigene Ports: der Proxy terminiert
# TLS auf den Slot-Ports und leitet hierher weiter; mit dem Backend spricht
# das Gateway nur über /internal/gui/* (GUI_GATEWAY_TOKEN). Siehe Handbuch,
# "Firewall-Oberfläche über OPNexus".
gui-gateway:
image: ${OPNEXUS_IMAGE_REPO:-local}/opnexus-backend:${OPNEXUS_VERSION:-latest}
profiles: ["gui"]
restart: unless-stopped
logging: *default-logging
command: ["python", "-m", "app.gui_gateway"]
depends_on:
backend:
condition: service_healthy
environment:
GUI_GATEWAY_TOKEN: ${GUI_GATEWAY_TOKEN:-}
OPNEXUS_GUI_SITES: ${OPNEXUS_GUI_SITES:-}
GUI_BACKEND_URL: http://backend:8000
user: "65534:65534"
read_only: true
cap_drop: ["ALL"]
security_opt: ["no-new-privileges:true"]
healthcheck:
test: ["CMD", "python", "-c", "import socket; socket.create_connection(('127.0.0.1', 8080), 3).close()"]
interval: 10s
timeout: 5s
retries: 5
start_period: 5s
# ── Optional: Updates aus dem Webinterface ──────────────────────────
# Standardmäßig NICHT gestartet. Aktivieren mit COMPOSE_PROFILES=updater in
# .env und `docker compose up -d`. Dies ist der einzige Container mit
+305 -5
View File
@@ -74,6 +74,32 @@ Alternativ lassen sich auch aus einem Checkout die fertigen Images verwenden: in
der `.env` `OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus` und `OPNEXUS_VERSION=<version>`
setzen, dann `docker compose pull && docker compose up -d --no-build`.
### Images prüfen (Signatur)
Die Release-Images sind mit [cosign](https://github.com/sigstore/cosign) signiert (ab 1.23.9).
Damit lässt sich prüfen, dass ein Image wirklich von OPNexus stammt und seit dem Bau nicht
verändert wurde. Der öffentliche Schlüssel liegt als `cosign.pub` im Repository und im
Release-Bundle:
```bash
cosign verify --key cosign.pub --insecure-ignore-tlog=true \
git.opnexus.dev/opnexus/opnexus-backend:<version>
```
Das Gleiche gilt für `opnexus-frontend`, `opnexus-proxy` und `opnexus-updater`. Eine
erfolgreiche Prüfung gibt die Signatur als JSON aus; bei einem fremden oder fehlenden
Schlüssel endet der Befehl mit einem Fehler. `--insecure-ignore-tlog=true` ist hier gewollt:
OPNexus lädt Signaturen nicht in das öffentliche Transparenzlog, damit die Prüfung auch in
abgeschotteten Netzen funktioniert. Die Signatur gilt dem Inhalt (Digest) des Images, nicht
dem Tag.
`opnexus update` prüft die Signaturen selbst, bevor es etwas wechselt, sobald `cosign`
installiert ist und `cosign.pub` neben der `docker-compose.yml` liegt (`cosign.pub` liegt im Bundle
bereits dabei, `cosign` selbst musst du installieren). Der Updater-Container für Updates per Klick
bringt `cosign` mit und prüft deshalb ebenfalls, solange `cosign.pub` im Projektverzeichnis liegt. Schlägt die Prüfung fehl, bricht das
Update ab und es wurde nichts verändert. Fehlt `cosign`, sagt `opnexus update` das und
macht ohne Prüfung weiter.
### Erster Start und Passwort
Ohne `ADMIN_PASSWORD` in der `.env` erzeugt OPNexus beim ersten Start ein
@@ -384,6 +410,83 @@ angesprochen (keine DNS-Namen). Nicht enthalten: Die Einrichtung auf der Firewal
(bei einer Firewall hinter NAT kann es sie ja nicht erreichen); bei einer erreichbaren Firewall kannst du
alternativ unter VPN › WireGuard einen Tunnel mit denselben Werten verteilen.
### Firewall-Oberfläche über OPNexus
Mit **Oberfläche öffnen** auf der Host-Seite öffnest du die Weboberfläche einer Firewall in einem neuen
Browser-Tab, und zwar **über OPNexus**. Das lohnt sich vor allem bei Firewalls hinter NAT, die nur über den
[Management-Tunnel](#firewalls-hinter-nat-management-tunnel) erreichbar sind: Dein Browser braucht keinen eigenen
Zugang zum Tunnel-Netz. Du meldest dich an der Firewall wie gewohnt mit deren Benutzer und Passwort an; OPNexus
kennt und speichert diese Zugangsdaten nicht und meldet nichts automatisch an.
**Standardmäßig ist die Funktion aus** und doppelt gesichert: Sie braucht eine Einrichtung auf dem Server und
einen Schalter in den Einstellungen.
**Einrichten** (in der `.env` im OPNexus-Verzeichnis):
1. **HTTPS und `COOKIE_SECURE=true`** (siehe [Zugriff aus dem Netz und TLS](#zugriff-aus-dem-netz-und-tls)).
Ohne sichere Cookies lässt sich die Funktion nicht einschalten.
2. **Compose-Profil `gui`** aktivieren: `COMPOSE_PROFILES=gui` (mit weiteren Profilen kommagetrennt, z. B.
`overlay,gui`). Das startet den Container `gui-gateway`.
3. **Slot-Adressen** festlegen: `OPNEXUS_GUI_SITES=https://10.0.0.5:9441, https://10.0.0.5:9442, https://10.0.0.5:9443, https://10.0.0.5:9444`.
Es ist dieselbe Adresse wie in `OPNEXUS_SITES`, je Slot ein Port. Diese Ursprünge dürfen **nicht** in
`FRONTEND_ORIGIN` stehen (OPNexus startet sonst nicht).
4. **Gemeinsames Geheimnis** `GUI_GATEWAY_TOKEN` (mindestens 32 Zeichen, z. B. `openssl rand -hex 32`).
5. **Ports:** `GUI_PORTS=9441-9444` (die Ports aus `OPNEXUS_GUI_SITES`) sind die Slot-Ports am Proxy. Setze den
Eintrag ausdrücklich: ohne ihn vergibt Docker zufällige freie Ports, damit ein Update nicht an schon belegten
Ports (z. B. Portainer auf 9443) scheitert. `GUI_BIND_ADDRESS` bestimmt, wo
sie lauschen; Standard ist `127.0.0.1` (nur lokal). Für den Zugriff aus dem Netz trägst du die Adresse des
OPNexus-Rechners ein und gibst die Ports in der Firewall des Rechners nur für Administratoren frei.
6. `docker compose up -d`, dann unter **Einstellungen › Firewall-Oberfläche über OPNexus** den Schalter einschalten.
Fehlt etwas, nennt die Einstellungsseite genau, was.
In den Einstellungen legst du außerdem fest: **Inaktivität** (Standard 15 Minuten, 1 bis 240), **Gesamtdauer**
(Standard 60 Minuten, 5 bis 720) und die Zahl der **gleichzeitigen Plätze** (Standard 4, 1 bis 4). Ausschalten
beendet alle offenen Sitzungen (das Gateway bemerkt es spätestens nach 20 Sekunden).
**So ist es abgesichert:**
- **Eigene Origin:** Jede Firewall-Oberfläche läuft auf einem eigenen Port (einem „Platz“) und damit in einem
eigenen Browser-Ursprung, getrennt von OPNexus. Skripte der Firewall-Oberfläche können OPNexus nicht
ansprechen; OPNexus lehnt außerdem Anfragen mit Anmelde-Cookie von fremden Ursprüngen ab.
- **Cookies bleiben auf dem Server:** Das Gateway reicht die OPNexus-Sitzung nie an die Firewall weiter und
die Cookies der Firewall nie an deinen Browser. Die Firewall-Sitzung liegt nur im Arbeitsspeicher des Gateways.
- **Nur Admins**, nur mit angemeldeter Browsersitzung (nie mit einem API-Token). Das Gateway prüft alle 20
Sekunden nach: Abmelden, Entzug der Admin-Rolle oder Ausschalten beendet die Sitzung.
- **Einmal-Ticket:** Der Klick erzeugt ein Ticket, das nur **30 Sekunden** und **genau einmal** gilt und an deine
Sitzung und den Platz gebunden ist. Danach verschwindet es aus der Adresszeile.
- **Nur sichere Firewalls:** https-Adresse und geprüftes oder festgesetztes Zertifikat (siehe Pinning); sonst
bleibt der Knopf deaktiviert und nennt den Grund.
- **Zeitlimits** (Inaktivität und Gesamtdauer) und **Audit:** Unter **Monitoring › Audit** (Art „GUI-Zugriff“)
stehen Ticket, Start und Ende jeder Sitzung mit Benutzer, Firewall, Platz und Client-IP. Was du in der
Oberfläche tust, protokolliert die Firewall selbst, dort aber mit der Adresse von OPNexus; das OPNexus-Audit
zeigt, wer dahinter stand.
- **Anmeldeschutz:** Das Gateway lässt höchstens drei fehlgeschlagene Anmeldungen pro Firewall in fünf Minuten zu.
**Wichtig: OPNexus-Adresse beim Anmeldeschutz der Firewall eintragen.** Über das Gateway kommen alle
Anmeldungen von der Adresse von OPNexus (bei Firewalls im Tunnel von der des Hubs). Die Anmeldesperre von
OPNsense (sshguard) sperrt nach mehreren Fehlversuchen diese Adresse, und damit auch die **API-Zugriffe von
OPNexus**. Trage sie deshalb auf jeder Firewall unter **System › Einstellungen › Administration** (Login
Protection) in die Whitelist ein, bevor du die Funktion benutzt.
**Einschränkungen:**
- **Kein WebSocket** (Funktionen, die einen brauchen, gehen nicht) und **keine automatische Anmeldung**.
- **Höchstens 4 Plätze gleichzeitig.** Ein Platz bleibt pro OPNexus-Sitzung bei derselben Firewall; sind alle
belegt, schließe eine Oberfläche oder warte, bis sie abläuft. Ein noch offener Tab einer alten Sitzung auf
einem wiederverwendeten Platz bleibt ein theoretisches Restrisiko; schließe Tabs nach der Arbeit.
- Nach einem **Neustart des wg-hub** braucht der Management-Tunnel 2 bis 4 Minuten, bis die Firewalls
sich wieder verbunden haben; bis dahin lässt sich die Oberfläche nicht öffnen.
- Mit `COOKIE_SECURE=false` ist die Funktion nicht verfügbar.
**Hinweis für Betreiber mit eigenen Skripten:** Seit dieser Funktion heißt das Sitzungs-Cookie bei HTTPS
`__Host-opnexus-session` (bestehende Sitzungen werden beim nächsten Seitenaufruf übernommen, niemand wird abgemeldet), und Anfragen an `/api` mit Cookie
und `Sec-Fetch-Site: same-site` oder `cross-site` werden abgelehnt. Skripte mit API-Token (`Authorization: Bearer`)
sind nicht betroffen.
**Alternative ohne diese Funktion:** Du richtest dir einen eigenen WireGuard-Zugang in das Netz der Firewalls ein
(z. B. am Standort oder auf der Firewall selbst) und rufst die Oberfläche direkt auf. Das ist unabhängig von
OPNexus, hat aber keine Rollenprüfung und kein Audit durch OPNexus.
---
## Regeln sicher ausrollen
@@ -462,6 +565,71 @@ Beschreibung und Systemregeln. Die Oberfläche nennt den Grund je Regel.
> strikt „erster Treffer gewinnt“ aus. Die Reihenfolge ist dadurch
> sicherheitsrelevant, siehe [hierarchische Policies](#gerätegruppen-und-hierarchische-policies).
### Aliase und NAT von der Firewall übernehmen
Wie die Regeln lassen sich auch **Aliase** und **NAT-Regeln** (Port Forward, 1:1, Outbound) übernehmen, die von Hand
in OPNsense entstanden sind: auf der jeweiligen Seite **Von Firewall übernehmen**, Firewall wählen, dann entweder ein
Objekt mit **Übernehmen** oder alle auf einmal mit **Alle übernehmbaren übernehmen**. So kommt eine bestehende Firewall
ohne Abtippen unter zentrale Verwaltung.
Die Übernahme **schreibt nichts auf die Firewall**. OPNexus liest das Objekt frisch von der Firewall, legt es zentral an
(Ziel: genau diese Firewall) und merkt sich die UUID aus OPNsense. Deshalb zeigt die Vorschau direkt danach „keine Änderung“,
und der nächste Rollout nach einer Änderung aktualisiert dasselbe Objekt, statt ein zweites anzulegen. Die Übernahme steht im
Audit-Log; Operatoren dürfen übernehmen (sie legt nur zentrale Einträge an), Nur-Lese-Benutzer nicht.
Nicht übernommen wird, was das Modell verfälschen würde. Die Oberfläche nennt den Grund je Objekt:
| Objekt | Nicht übernehmbar, weil … |
|---|---|
| Aliase | OPNsense-eigen (`bogons`, `sshlockout`, `__lan_network` …); Typ unbekannt für OPNexus (URL, URL-Tabelle, GeoIP, dynamische IPv6-Hosts, OpenVPN-Gruppe, „Extern“); der Name existiert in OPNexus schon; Name mit Sonderzeichen; BGP-ASN mit IPv4/IPv6-Einschränkung; Verweis auf einen Alias, der nicht übernommen ist |
| Port Forward | mehrere oder keine Schnittstelle; IP-Version weder IPv4 noch IPv6; Protokoll „beliebig“; „Kein Redirect“; kein Ziel; verknüpfte Filterregel; invertierte Quelle oder Ziel; NAT-Reflection-Wert unbekannt; Beschreibung fehlt |
| 1:1-NAT | mehrere oder keine Schnittstelle; unbekannter Typ oder NAT-Reflection-Wert; Pflichtangaben oder Beschreibung fehlen |
| Outbound-NAT | mehrere oder keine Schnittstelle; „Kein NAT“-Regel (`nonat`); IP-Version weder IPv4 noch IPv6; Beschreibung fehlt |
Automatisch erzeugte NAT-Regeln (Anti-Lockout, „Auto created rule“) erscheinen nie in der Liste. **Verschachtelte Aliase:**
Ein Alias, der einen anderen, noch nicht übernommenen Alias enthält, wartet auf diesen; „Alle übernehmbaren übernehmen“
übernimmt beide in der richtigen Reihenfolge, einzeln zuerst den enthaltenen Alias. Verweist er auf einen Alias, der nie
übernommen werden kann (z. B. vom Typ URL), bleibt er mit diesem Grund zurück. **Outbound-NAT** lässt sich nur übernehmen,
wenn der Outbound-Modus der Firewall Hybrid oder Manuell ist; im Modus Automatisch zeigt OPNsense manuelle Regeln gar nicht an,
OPNexus verweist dann darauf und ändert den Modus nicht.
Einstellungen, die OPNexus nicht verwaltet (Protokollierung und Pool-Optionen beim Port Forward, Tags, Kategorien, statischer
NAT-Port), werden nie überschrieben und im Dialog als „Bleibt auf der Firewall unverändert“ angezeigt.
#### Gleiches Objekt auf mehreren Firewalls: verknüpfen statt doppelt übernehmen
Gibt es dieselbe Regel, denselben Alias oder dieselbe NAT-Regel auf mehreren Firewalls (typisch: beide Knoten eines
HA-Clusters, mehrere Standorte mit gleicher Vorgabe), übernimmst du sie **einmal** und **verknüpfst** sie auf den anderen
Firewalls mit dem zentralen Objekt. Das gilt für Regeln, Aliase, Port Forward, 1:1-NAT und Outbound-NAT.
1. Auf der ersten Firewall normal **übernehmen**.
2. Im Dialog die zweite Firewall wählen. Bei jedem Eintrag, der einem zentralen Objekt **exakt** entspricht, steht
„Gibt es schon zentral“ und die Aktion **Verknüpfen** (daneben **Als eigenes Objekt übernehmen**). Verknüpfen trägt die
Firewall als weiteres Ziel in das Objekt ein und merkt sich die UUID dort. Auf der Firewall wird nichts geändert.
3. Die Vorschau zeigt danach für diese Firewall „keine Änderung“. Eine spätere Änderung am Objekt rollst du wie gewohnt für
alle Ziele gemeinsam aus; es entsteht kein Duplikat.
**Was „exakt“ heißt:** Alle von OPNexus verwalteten Felder sind gleich, nachdem die `{{VARIABLEN}}` des zentralen Objekts mit
den Werten **dieser** Firewall ersetzt wurden (ein Alias `{{LAN_NET}}` passt also zu `192.168.2.0/24`, wenn das die Variable
dieser Firewall ist). Bei Aliasen müssen Name, Typ, Inhalt einschließlich der Verweise auf andere Aliase, Beschreibung und
Aktiv-Status stimmen. Fehlt der Firewall ein Variablenwert, passt das Objekt nicht, der Dialog nennt die Variable.
**Abweichungen:** Gibt es ein Objekt mit gleichem Namen (Alias) bzw. gleicher Beschreibung (Regeln, NAT), das sich im Inhalt
unterscheidet, zeigt der Dialog „Ähnlich“ mit den abweichenden Feldern (zentral gegenüber Firewall). Verknüpft wird dann **nie**,
auch nicht automatisch. Gleiche entweder die Firewall oder das zentrale Objekt an, oder übernimm das Objekt getrennt (bei Aliasen
geht das nur nach einer Umbenennung, der Name ist zentral eindeutig).
**Alle übernehmbaren:** Der Dialog zeigt vorher, was passiert („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“), und du
bestätigst. Verknüpft wird nur, wenn genau **ein** zentrales Objekt exakt passt; gibt es mehrere identische (Altlast aus früheren
Einzelübernahmen), entscheidest du bei dem Eintrag selbst, und „Alle“ übernimmt ihn als neues Objekt. Ein zentrales Objekt wird in
einem Lauf höchstens einmal verknüpft.
**Sicherheit:** Die Firewall wird beim Verknüpfen frisch gelesen und noch einmal exakt verglichen; hat sich zwischen Anzeige und Klick
etwas geändert, wird nichts verknüpft und die Meldung nennt die Felder. Ein zweiter Klick ändert nichts. Die Aktion steht im
Audit-Log und in den Revisionen des Objekts („Mit <Firewall> verknüpft“). Rollen wie beim Übernehmen: Operatoren dürfen, Nur-Lese-Benutzer
nicht. Hat das zentrale Objekt eine offene Bestätigung oder einen offenen Rollback, wird erst nach deren Klärung verknüpft.
„Alle übernehmbaren“ lässt solche Einträge dann aus (Hinweis im Ergebnis). **Wichtig:** Ein verknüpftes Objekt wird wie jedes verwaltete Objekt behandelt: Wird das zentrale Objekt gelöscht oder die Firewall aus dessen Zielen entfernt, löscht der nächste Rollout das Objekt auch auf dieser Firewall.
---
## Gerätegruppen und hierarchische Policies
@@ -765,7 +933,15 @@ frische Backup erneut. Eine blockierte Aktion wird als „fehlgeschlagen“ mark
und **nicht** wiederholt. Sie steht im Status der geplanten Wartung und meldet sich
zusätzlich per Benachrichtigung („Geplante Wartung nicht ausgeführt“, einzeln
schaltbar), damit man es am nächsten Morgen nicht erst beim Nachsehen merkt.
Wiederkehrende Zeitpläne gibt es nicht.
Wiederkehrende Termine und ganze Flotten laufen über [Rollouts](#rollouts-in-wellen).
**Ergebnis automatisch prüfen und abschließen** (im Dialog vorausgewählt): Nach dem Auftrag
wartet OPNexus, bis der Firmware-Auftrag beendet ist, und prüft dann selbst: Firmware
mindestens eine Minute am Stück bereit, ein erwarteter Neustart hat wirklich stattgefunden
(Host offline gesehen oder neue Boot-Zeit), Version lesbar. Erst dann wird die Wartungsaktion
als abgeschlossen markiert und sperrt Host und Cluster nicht mehr. Gelingt das nicht innerhalb
von 20 Minuten, bleibt sie „unklar“ (gesperrt, mit Benachrichtigung). Ohne die Option bleibt
sie wie bisher „übergeben“, bis jemand sie von Hand abschließt.
In einem Cluster läuft unbeaufsichtigte Wartung standardmäßig **nur auf dem
CARP-BACKUP**. Ist der Host beim Einplanen gerade MASTER, lehnt OPNexus das sofort mit
@@ -792,7 +968,72 @@ der Plan steht auf „fehlgeschlagen“, die Wartungsaktion auf „unklar“, un
Benachrichtigung. Der CARP-Umschalter wird nie blind wiederholt. Beendest du die
Wartungsaktion von Hand, stoppt der Ablauf und fasst nichts mehr an. Bei Erfolg schließt
OPNexus die Wartungsaktion selbst ab. Sie läuft auch einen Neustart des OPNexus-Backends
mitten im Ablauf weiter (der Zustand steht in der Datenbank).
mitten im Ablauf weiter (der Zustand steht in der Datenbank). Mit „Ergebnis automatisch
prüfen“ kommt die Prüfung **vor** dem Verlassen des Wartungsmodus: Der Host bekommt den
Verkehr erst zurück, wenn er nachweislich fertig ist.
### Rollouts in Wellen
**System › Rollouts** verteilt ein Firmware-Update (stabiles Update oder Major-Upgrade) in
Wellen über die Flotte. Die Wellen bestehen aus [Gerätegruppen](#gerätegruppen), die man
als Ringe anlegt, zum Beispiel `ring-canary` (eine unwichtige Firewall), `ring-pilot`
(einige) und `ring-prod` (der Rest). Steht ein Host in mehreren Wellen, gilt die früheste.
**Ablauf je Welle:**
1. **Vorprüfung:** Update-Suche auf den Hosts der Welle, dann Bestand aufnehmen (Version,
Gateways, laufende Dienste, Sicherheits-Score). Hosts ohne passendes Update werden
übersprungen. Die Version, die in der ersten Welle installiert wird, ist die
**Zielversion**; bietet ein Host einer späteren Welle eine andere an, hält der Rollout an
(OPNsense installiert immer die neueste, sonst bekäme „prod“ eine Version, die im
Canary-Ring nie lief).
2. **Hosts nacheinander:** Jeder Host läuft als [geplante Wartung](#geplante-wartung) mit
automatischer Prüfung (Backup vor dem Update, Sperren, Audit). Immer nur ein Host,
immer nur ein Rollout gleichzeitig. Im Cluster zuerst der BACKUP, dann der MASTER;
der MASTER nur, wenn die Welle die CARP-Übergabe erlaubt, sonst hält der Rollout an.
Vor dem zweiten Knoten eines Clusters werden der Cluster (alle Knoten erreichbar,
genau ein MASTER je VHID, kein Wartungsmodus) und der schon aktualisierte Knoten
erneut geprüft.
3. **Beobachtungszeit** (je Welle einstellbar), dann die **Gesundheitsprüfung** der
aktualisierten Hosts: erreichbar und Firmware bereit, Zielversion installiert, CARP
gesund, Gateways und Dienste, die vorher liefen, laufen wieder (vorher schon gestörte
nur als Hinweis), keine offenen Sperren, optional Sicherheits-Score nicht gesunken
(Hinweis oder Halt). Ist etwas rot, prüft OPNexus zwei Minuten später noch einmal
(langsamer Start nach dem Neustart) und hält dann an.
4. **Freigabe** (optional je Welle): Die Welle beginnt erst, wenn jemand auf „Welle
freigeben“ klickt.
**Angehalten** wird bei jeder Unklarheit, nie wird etwas von selbst wiederholt: Gate rot,
Host fehlgeschlagen oder ungewöhnlich lange unterwegs, Host in der Vorprüfung nicht bereit,
abweichende Zielversion, MASTER ohne Übergabe, eine fremde offene Wartung im Cluster länger
als 15 Minuten, die verantwortliche Person darf nicht mehr ausrollen, das Fenster wurde
gelöscht, der Lauf ist älter als 14 Tage. Es kommt die Benachrichtigung „Rollout
angehalten“. **Fortsetzen** prüft zuerst alle bisher aktualisierten Hosts erneut; wer
fortsetzt, ist ab dann verantwortlich. Ein fehlgeschlagener Host muss vorher ausdrücklich
**übersprungen** werden (seine Wartungsaktion bleibt „unklar“, bis du sie auf der
Host-Seite abschließt). **Abbrechen** verwirft alles, was noch nicht begonnen hat; eine
Aktion, die schon auf einer Firewall läuft, wird geführt zu Ende gebracht. **Anhalten**
darf jeder mit Schreibrecht, alles andere braucht das Recht zum Ausrollen.
**Wartungsfenster** (Reiter „Wartungsfenster“): wöchentlich an bestimmten Tagen oder
monatlich (erster…vierter oder letzter Wochentag, oder ein fester Tag 1–28), Beginn in einer
Zeitzone, Dauer und Abschneidezeit: Nach den ersten N Minuten beginnt kein neuer Host mehr,
laufende Updates laufen zu Ende, und der Rollout wartet auf den nächsten Termin (dann wird
zuerst das Bisherige erneut geprüft). Sommerzeit: Eine Uhrzeit, die es an dem Tag nicht gibt,
rückt vor (02:30 → 03:30); eine doppelte gilt beim ersten Mal. Einzelne Termine lassen sich
auslassen oder verschieben. Ein Fenster, das ein Plan nutzt, lässt sich nicht löschen.
**Automatischer Start:** Ein Plan mit Fenster und stabilem Update kann zu jedem Termin von
selbst starten. Dazu muss ihn jemand mit Ausroll-Recht nach der Vorschau **aktivieren**.
OPNexus merkt sich dabei einen Fingerabdruck von Plan, aufgelösten Hosts und Fenster; ändert
sich davon etwas (auch die Mitglieder einer Gruppe), startet nichts, und es kommt „Rollout
nicht automatisch gestartet“. Jede Bearbeitung des Plans deaktiviert ihn. So kann niemand
die Ziele eines aktivierten Plans unbemerkt ändern.
Das **Update Center** zeigt den laufenden Rollout als Banner; Hosts, die er noch bearbeitet,
sind dort für manuelle Firmware-Aktionen gesperrt. Ein Neustart oder Update von OPNexus
während eines Rollouts ist sicher (der Zustand steht in der Datenbank), hält ihn aber für
diese Zeit an.
### Dienste und Systemauslastung
@@ -918,18 +1159,70 @@ Einstellungen › **Benachrichtigungen**. Kanäle:
Absender und Empfängern. Zertifikat und Hostname werden geprüft; ein Mailserver
mit privater CA braucht `SMTP_CA_FILE`. Anmeldung ohne Verschlüsselung bleibt
gesperrt. Mit **testen** prüfst du jeden Kanal.
- **Weitere Kanäle** (Abschnitt „Weitere Kanäle“, beliebig viele je Typ, jeweils mit **Testen**):
| Typ | Angaben | Zugangsdaten |
|---|---|---|
| **ntfy** | Serveradresse (Standard `https://ntfy.sh`), Thema, Priorität 1–5 | Zugriffstoken, optional |
| **Gotify** | Serveradresse, Priorität 0–10 | App-Token |
| **Telegram** | Chat-ID (Zahl oder `@kanal`), optional eine eigene API-Adresse | Bot-Token (von @BotFather) |
| **Microsoft Teams** | – | Webhook-Adresse eines Workflows („Beim Empfang einer Webhook-Anfrage in einem Kanal posten“) oder eines Incoming-Webhooks |
| **Matrix** | Homeserver-Adresse, Raum-ID (`!abc:example.org`, nicht der Alias) | Zugriffstoken eines Benutzers, der im Raum ist |
Die Zugangsdaten liegen wie das SMTP-Passwort in der Geheimnis-Ablage des Servers, nie in der
Datenbank, werden nie wieder angezeigt (nur „gesetzt“) und stehen nicht in Fehlermeldungen oder Logs.
Weil bei Telegram und Teams die Adresse selbst das Geheimnis ist, wird sie wie ein Token behandelt.
Ziele müssen `http(s)` sein, ohne Zugangsdaten in der Adresse; lokale Adressen (`localhost`, Loopback,
Link-Local, Metadaten-Dienst) werden abgelehnt, Ziele im eigenen Netz sind erlaubt. Weiterleitungen werden
nicht verfolgt. Ein ausgefallener Kanal hält die anderen nicht auf; zugestellt wird einmal je Ereignis.
Ereignisse (jeweils einzeln schaltbar): Host nicht erreichbar, Zertifikat läuft ab,
Regel-Drift im Cluster, NAT-Drift im Cluster, Netzwerk-Abweichung im Cluster,
fehlgeschlagener Rollback, Wartungsergebnis unklar, CARP-Zustand kritisch
(Split-Brain/kein Master), CARP-Auto-Revert fehlgeschlagen, geplante Wartung nicht
ausgeführt. Geprüft wird im eingestellten Intervall (Standard 300 s). Ein Problem
ausgeführt, Gateway gestört, Ressourcen-Schwelle überschritten, Dienst ausgefallen
(die letzten drei sind unten beschrieben). Geprüft wird im eingestellten Intervall (Standard 300 s). Ein Problem
meldet sich einmal und löst sich selbst auf, sobald es behoben ist; die
Dashboard-Karte „Warnungen“ zeigt die offenen. Ausnahme „geplante Wartung nicht
ausgeführt“: Sie bleibt drei Tage nach dem geplanten Zeitpunkt offen und schließt dann
ohne „behoben“-Meldung, weil sich ein verpasster Termin nicht von selbst behebt. Der
Text der Benachrichtigungen ist nicht übersetzt.
#### Gateway-, Ressourcen- und Dienst-Alarme
Diese drei Ereignisse lesen die **Minutenmessung** (dieselbe wie die Auslastungs-Diagramme), fragen die
Firewalls also nicht zusätzlich ab. Damit Einzelspitzen und flackernde Verbindungen nichts melden,
öffnet ein Alarm erst, wenn die letzten *N* Messungen alle schlecht sind, und schließt erst, wenn die letzten
*N* alle wieder gut sind; dazwischen bleibt der Zustand, wie er ist. *N* ist einstellbar (Standard 3 für
Gateways und Dienste, 5 für Ressourcen; die Messung läuft jede Minute). Meldet eine Firewall gerade nicht,
beurteilt OPNexus sie nicht (das meldet „Host nicht erreichbar“) und lässt offene Alarme stehen.
- **Gateway gestört** – **standardmäßig an**, weil ein WAN-Ausfall das ist, was jeder wissen will. Ein Alarm je Gateway, wenn OPNsense es als
*down* führt, oder wenn der Paketverlust (Standard ab 20 %) oder die Latenz (Standard ab 500 ms) darüber liegt (0 schaltet die jeweilige Schwelle ab).
OPNsense selbst entscheidet über „down“ nach den Grenzen im Gateway (Einstellung „Gateway-Überwachung“). Von Hand abgeschaltete
Gateways („Gateway deaktivieren/erzwungen aus“) und Gateways ohne Überwachung (kein Monitor-IP, Werte „~“) melden nichts.
Der Status steht auch auf der Host-Seite (Abschnitt „Gateways“, live gelesen), als Kachel auf dem Dashboard und in Prometheus.
- **Ressourcen-Schwelle** – standardmäßig **aus**: Auf einer bestehenden Flotte würde das Einschalten sonst gleich alles melden, was schon länger voll
ist. Schwellen: Festplatte ab 90 % (je ZFS-Pool bzw. UFS-Gerät ein Alarm), Arbeitsspeicher ab 95 %, Swap ab 50 %, CPU-Last ab 2,0 je Thread
(1-Minuten-Last geteilt durch die Zahl der CPU-Threads). Jede Schwelle lässt sich mit 0 abschalten.
- **Dienst ausgefallen** – standardmäßig **aus**. OPNsense liefert nur „läuft / läuft nicht“ und „gesperrt“, aber nicht, ob jemand einen Dienst
absichtlich gestoppt hat. Deshalb gilt eine **Liste überwachter Dienste** (Standard: unbound, dnsmasq, dhcpd, dhcpd6, kea-dhcp4, kea-dhcp6, ntpd,
openvpn, wireguard, strongswan), und gemeldet wird nur ein Dienst, der in den letzten rund zwei Stunden lief und jetzt nicht mehr. Ein Dienst, der
nie lief (etwa DHCPv6 ohne Präfix), bleibt still; einen absichtlich gestoppten Dienst nimmst du aus der Liste. Eine Firewall listet nur Dienste, die
in ihrer Konfiguration aktiviert sind, ein nicht eingerichteter Dienst taucht also gar nicht auf.
### Regel-Analyse
Unter **Regelwerk › Regel-Analyse** fragst du, welche Regel für eine Verbindung greift: Quell- und Ziel-IP,
Protokoll, Zielport und das Eingangs-Interface („automatisch“ rät es aus der Quell-IP und kennzeichnet das als
Hinweis). Mit „Ausgangsseite prüfen“ werden zusätzlich die Regeln für ausgehenden Verkehr geprüft. Geprüft wird
das erste Paket einer neuen Verbindung gegen den tatsächlich geladenen Regelsatz der Firewall; es wird nur gelesen.
Je Firewall erscheint eine Karte mit dem Urteil (Erlaubt, Blockiert, Abgewiesen), der **Sicherheit** (sicher,
vermutlich, unbekannt; Tooltip mit den Gründen) und der entscheidenden Regel. Ist das Ergebnis „unbekannt“
(zum Beispiel fehlt der Zielport oder ein Alias ist nicht auflösbar), zeigt die Karte bewusst kein Urteil, sondern
die offenen Punkte. Unter „Warum?“ steht jede geprüfte Regel in der Prüfreihenfolge mit ✓/✗/? je Feld.
Die Abfrage steht in der Adresszeile und lässt sich als Link teilen.
### Sicherheitslage
Unter **Monitoring › Sicherheitslage** prüft OPNexus jede Firewall auf typische
@@ -1030,6 +1323,10 @@ scrape_configs:
| `opnexus_host_enabled`, `…_last_sample_timestamp_seconds` | Überwachung an; Zeit der letzten Messung |
| `opnexus_host_memory_used_percent`, `opnexus_host_load1`, `opnexus_host_disk_used_percent` | Letzte Messung |
| `opnexus_host_carp_master` | 1 = MASTER, 0 = BACKUP (fehlt, wenn unbekannt) |
| `opnexus_host_swap_used_percent` | Swap-Belegung der letzten Messung (fehlt ohne Swap) |
| `opnexus_host_gateway_up{gateway}` | 1, solange OPNsense das Gateway nicht als down führt |
| `opnexus_host_gateway_loss_percent`, `…_delay_seconds`, `…_stddev_seconds` `{gateway}` | Paketverlust, Laufzeit und Abweichung zum Monitor (fehlen ohne Messung) |
| `opnexus_host_service_running{service,instance}` | 1 = läuft, nur für Dienste aus der Überwachungsliste |
| `opnexus_host_compliance_score`, `opnexus_host_compliance_findings{severity}` | Sicherheitslage |
| `opnexus_pending_actions{kind,status}` | Offene Aktionen wie auf dem Dashboard |
| `opnexus_problems_open{type}` | Offene Probleme der Benachrichtigungen |
@@ -1324,8 +1621,11 @@ nach einem Neustart erhalten. Alternativ legt ihn OPNexus unter
OPNsense, keine Ausführung durch OPNexus.
- **Logs:** kein Live-Streaming (Polling im Sekundenbereich), die Freitextsuche ist
eine einfache Teilstring-Suche ohne Index.
- **Geplante Wartung:** nur einmalige Zeitpunkte, ohne Benachrichtigung bei
Fehlschlag.
- **Geplante Wartung und Rollouts:** Rollouts kennen keine Zielversion „von außen“
(OPNsense installiert immer die neueste angebotene) und keinen automatischen Rückweg zur
alten Version. Major-Upgrade-Rollouts sind nur gegen eine nachgestellte Firewall getestet.
Die Gesundheitsprüfung sieht die Firewall aus Sicht der Verwaltung; ob der Nutzverkehr
durchgeht, prüft sie nicht.
- **Dienste:** keine Auswahlliste, alle Dienste sind steuerbar.
- **VPN:** Umfang je Verfahren siehe [VPN](#vpn-ipsec-openvpn-wireguard).
- **Zertifikate:** kein Push auf mehrere Hosts, keine Verwaltung von DNS-Providern
+9 -2
View File
@@ -1,9 +1,11 @@
FROM node:20-alpine AS deps
# deps und builder laufen immer auf der Build-Plattform (nativ schnell, `next build` unter QEMU-Emulation
# hängt); das Ergebnis (.next/standalone) ist reines JavaScript und gilt für jede Zielarchitektur.
FROM --platform=$BUILDPLATFORM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --no-audit --no-fund
FROM node:20-alpine AS builder
FROM --platform=$BUILDPLATFORM node:20-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
@@ -11,10 +13,15 @@ ENV NEXT_TELEMETRY_DISABLED=1
RUN npm run build
FROM node:20-alpine AS runner
ARG TARGETARCH
ARG BUILDARCH
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
# Beim Cross-Build liegen die nativen sharp-Binärdateien der Build-Architektur im Standalone-Ordner; die App nutzt
# next/image nicht, sie wären nur totes Gewicht.
RUN if [ "$TARGETARCH" != "$BUILDARCH" ]; then rm -rf node_modules/@img node_modules/sharp; fi
EXPOSE 3000
CMD ["node", "server.js"]
Loaded 100 of 149 files, more files were not shown because too many files have changed in this diff. Show more