Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8128d14da4 | ||
|
|
95f297d3f2 | ||
|
|
ad17dcad25 | ||
|
|
5cb2ebe1c6 | ||
|
|
18d489693d | ||
|
|
a9f562eec6 | ||
|
|
a3274f64ae | ||
|
|
9ba6357193 | ||
|
|
23d3a00ae1 | ||
|
|
4379a17b15 |
No files matched your search
@@ -100,3 +100,26 @@ TRUST_PROXY_HEADERS=false
|
||||
# Zum Aktivieren (Handbuch „Firewalls hinter NAT“): COMPOSE_PROFILES=overlay
|
||||
# und den UDP-Port aus dem Internet erreichbar machen.
|
||||
#OVERLAY_PORT=51820
|
||||
|
||||
# ── GUI-Fernzugriff auf verwaltete Firewalls (optional, Standard AUS) ──
|
||||
# Öffnet die Weboberfläche einer Firewall über OPNexus, auch hinter NAT
|
||||
# (Management-Tunnel), jeweils in einem eigenen Browser-Ursprung (Slot-Port).
|
||||
# Dreifach aus, bis alles hier gesetzt ist UND ein Admin den Schalter unter
|
||||
# Einstellungen einschaltet. Voraussetzungen: HTTPS über den Proxy und
|
||||
# COOKIE_SECURE=true. Siehe Handbuch, „Firewall-Oberfläche über OPNexus“.
|
||||
# 1. Profil aktivieren (mit anderen Profilen kommagetrennt, z. B. overlay,gui):
|
||||
# COMPOSE_PROFILES=gui
|
||||
# 2. Slot-Ursprünge: dieselbe Adresse wie in OPNEXUS_SITES, je Slot ein Port
|
||||
# aus GUI_PORTS. Darf NICHT in FRONTEND_ORIGIN stehen (Start wird sonst verweigert).
|
||||
# OPNEXUS_GUI_SITES=https://10.0.0.5:9441, https://10.0.0.5:9442, https://10.0.0.5:9443, https://10.0.0.5:9444
|
||||
# 3. Gemeinsames Geheimnis Backend <-> Gateway, mind. 32 Zeichen, z. B.
|
||||
# `openssl rand -hex 32`:
|
||||
# GUI_GATEWAY_TOKEN=
|
||||
# 4. Wo die Slot-Ports lauschen. Standard nur lokal (127.0.0.1); für den
|
||||
# Zugriff aus dem Netz die Adresse dieses Hosts eintragen (wie bei
|
||||
# PROXY_BIND_ADDRESS). GUI_PORTS ist ein Bereich, Host- und Container-Port
|
||||
# sind gleich. GUI_PORTS zum Einschalten SETZEN: ohne den Eintrag nimmt
|
||||
# Docker zufällige freie Ports (damit ein Update nie an belegten Ports wie
|
||||
# 9443 scheitert), und die Slot-Adressen oben wären nicht erreichbar.
|
||||
# GUI_BIND_ADDRESS=127.0.0.1
|
||||
# GUI_PORTS=9441-9444
|
||||
+167
@@ -31,6 +31,173 @@ den Einstellungen.
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Merlin …
|
||||
|
||||
## [1.28.0] "Sparrowhawk" — 2026-10-21
|
||||
|
||||
- **Übernehmen: mit vorhandenem zentralem Objekt verknüpfen:** Dieselbe von Hand angelegte Regel, derselbe Alias oder dieselbe NAT-Regel auf mehreren
|
||||
Firewalls (z. B. beide HA-Knoten) wird nicht mehr als zweites Objekt übernommen. Im Dialog „Von Firewall übernehmen“ erscheint bei exakter
|
||||
Übereinstimmung (nach Einsetzen der Host-Variablen, bei Aliasen inkl. Verweisen) die Aktion **Verknüpfen**: die Firewall wird weiteres Ziel des
|
||||
vorhandenen Objekts, die Vorschau zeigt „keine Änderung“, und der nächste Rollout aktualisiert alle Firewalls an einem Objekt. Auf der Firewall wird
|
||||
dabei nichts geändert. Unterscheidet sich ein gleichnamiges Objekt, zeigt der Dialog die abweichenden Felder; verknüpft wird dann nie. „Alle
|
||||
übernehmbaren“ fragt vorher nach („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“). Aliase, die bisher wegen „Name existiert“ abgelehnt wurden,
|
||||
lassen sich so übernehmen.
|
||||
Wartet das vorhandene Objekt gerade auf eine Bestätigung, wird nicht verknüpft; „Alle übernehmbaren“ lässt den Eintrag dann aus. Verknüpfen
|
||||
macht ein gerade geöffnetes Bearbeiten-Formular des Objekts ungültig, damit die neue Firewall nicht versehentlich wieder aus den Zielen fällt.
|
||||
|
||||
- **Tooltips überarbeitet:** Der deaktivierte Button „Oberfläche öffnen“ nennt jetzt den Grund im Klartext samt nächstem Schritt (statt nur einer
|
||||
Variablen wie `GUI_GATEWAY_TOKEN`); dasselbe gilt für ausgeschaltete Funktion, deaktivierte Firewall, Nicht-HTTPS-Adresse und ausgeschaltete
|
||||
Zertifikatsprüfung. Texte durchgehend in Du-Form, kürzere VPN-Einleitungen, verständlichere Hinweise bei gesperrten Buttons (Zertifikate, Benutzer,
|
||||
Dienste). Icon-Buttons (Bearbeiten, Löschen, Schließen, Reihenfolge) zeigen beim Darüberfahren ihren Namen.
|
||||
|
||||
## [1.27.2] "Buzzard" — 2026-10-21
|
||||
|
||||
- **`opnexus update` prüft den Proxy vor dem Wechsel:** Nach dem Ziehen der Images validiert das Werkzeug die Caddy-Konfiguration des neuen
|
||||
Proxy-Images mit den Umgebungsvariablen *dieser* Installation. Ist sie ungültig, bricht das Update ab, bevor etwas gewechselt wurde.
|
||||
Der Fall aus 1.27.0 (Compose-Datei mit leerem `OPNEXUS_GUI_SITES`) wird dabei automatisch repariert: Das Werkzeug trägt den Platzhalter
|
||||
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` ein und sagt es. Installationen, die noch die Compose-Datei aus 1.27.0 haben,
|
||||
müssen also nur einmal `opnexus update` ausführen.
|
||||
- **`opnexus update` erkennt einen abstürzenden Proxy:** Ein Dienst ohne Healthcheck (der Proxy) gilt jetzt erst nach 10 Sekunden
|
||||
durchgehender Laufzeit als gesund. Bisher meldete das Update „Fertig“, obwohl der Proxy in einer Neustart-Schleife hing, und das
|
||||
automatische Zurückrollen griff nicht.
|
||||
|
||||
## [1.27.1] "Buzzard" — 2026-10-21
|
||||
|
||||
- **Fix (1.27.1): Proxy startete mit der neuen Compose-Datei nicht.** Mit der `docker-compose.yml` aus 1.27.0 gab Compose die Variable
|
||||
`OPNEXUS_GUI_SITES` als leeren Text an den Proxy weiter; Caddy nutzt seinen Standardwert aber nur bei *nicht gesetzter* Variable und
|
||||
brach mit „server block without any key“ ab, OPNexus war dann nicht mehr erreichbar. Die Compose-Datei setzt jetzt einen gültigen
|
||||
Platzhalter als Standard. Betroffen waren nur Installationen, die die neue Compose-Datei aus 1.27.0 übernommen hatten (ohne
|
||||
`OPNEXUS_GUI_SITES` in der `.env`); wer nur die Images aktualisiert hatte, war nicht betroffen. Sofort-Abhilfe ohne Update:
|
||||
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` eintragen und `docker compose up -d proxy` ausführen.
|
||||
|
||||
## [1.27.0] "Buzzard" — 2026-10-21
|
||||
|
||||
Vorgesehen als MINOR-Release **1.27.0** „Firewall-Oberfläche über OPNexus“.
|
||||
|
||||
- **Firewall-Oberfläche über OPNexus:** Auf der Host-Seite öffnet **Oberfläche öffnen** die Weboberfläche der Firewall
|
||||
in einem neuen Tab, auch bei Firewalls hinter NAT im Management-Tunnel. Nur für Admins; jede Firewall läuft in einer
|
||||
eigenen Browser-Origin (Platz = Port), Cookies bleiben auf dem Server, das Ticket gilt einmalig 30 Sekunden,
|
||||
Inaktivitäts- und Gesamtlimit sind einstellbar, Ticket, Start und Ende stehen im Audit (Art „GUI-Zugriff“).
|
||||
Standardmäßig **aus**: braucht das Compose-Profil `gui`, `OPNEXUS_GUI_SITES`, `GUI_PORTS`, `GUI_GATEWAY_TOKEN`,
|
||||
`COOKIE_SECURE=true` und den Schalter unter Einstellungen (Handbuch, „Firewall-Oberfläche über OPNexus“).
|
||||
Live getestet auf OPNsense 26.7.
|
||||
- **Einschränkungen:** kein WebSocket, keine automatische Anmeldung, höchstens 4 Plätze gleichzeitig. OPNexus-Adresse
|
||||
vorher in die Whitelist des Anmeldeschutzes (sshguard) der Firewall eintragen, sonst können Fehlanmeldungen die
|
||||
API-Zugriffe von OPNexus aussperren. Nach einem Neustart des wg-hub braucht der Tunnel 2 bis 4 Minuten.
|
||||
- **Hinweis für Betreiber mit eigenen Skripten:** Bei HTTPS heißt das Sitzungs-Cookie jetzt `__Host-opnexus-session`
|
||||
(bestehende Sitzungen werden ohne Abmelden übernommen), und Cookie-Anfragen an `/api` mit `Sec-Fetch-Site: same-site/cross-site`
|
||||
werden abgelehnt. Zugriffe mit API-Token sind nicht betroffen.
|
||||
|
||||
## [1.26.0] "Kite" — 2026-10-21
|
||||
|
||||
Geplant als MINOR-Release **1.26.0** (Regel-Analyse). Backend, MCP und Oberfläche.
|
||||
|
||||
- **Regel-Analyse „Welche Regel greift?“ (API):** `GET /api/hosts/{id}/rule-analysis/match` und
|
||||
`GET /api/rule-analysis/match` (ganze Flotte, Auswahl oder Gerätegruppe) beantworten für Quelle, Ziel, Protokoll
|
||||
und Port, ob eine Firewall das erste Paket einer neuen Verbindung durchlässt oder blockiert, welche Regel das
|
||||
entscheidet und warum (Prüfprotokoll Regel für Regel). Grundlage ist der tatsächlich geladene pf-Regelsatz
|
||||
inklusive automatischer Regeln, Floating- und Gruppenregeln, `quick`/letzter Treffer, Negation, verschachtelter
|
||||
Aliase (pf-Tabellentest der Firewall), Port-Aliase und Adressfamilien. Jede Antwort trägt eine ehrliche
|
||||
Konfidenz **sicher / vermutlich / unbekannt** mit Begründungscodes; was nicht belegbar ist (z. B. `tagged`,
|
||||
nicht auflösbare Aliase, geratenes Interface, NAT in der Nähe), wird nie als „sicher“ gemeldet. Nur lesend,
|
||||
auch mit `read`-Token nutzbar; es wird nichts gespeichert und nichts an der Firewall geändert.
|
||||
- **MCP:** neues Werkzeug `rule_match` für KI-Assistenten (gleiche Abfrage, ohne Prüfprotokoll).
|
||||
- **Seite „Regel-Analyse“ (Regelwerk):** Formular für Hosts oder Gerätegruppe, Quell-/Ziel-IP, Protokoll, Ports,
|
||||
Eingangs-Interface („automatisch“ oder Liste) und optional die Ausgangsseite. Die Abfrage steht in der Adresszeile
|
||||
(teilbar, wird beim Öffnen sofort ausgeführt). Je Firewall eine Karte mit Erlaubt/Blockiert/Abgewiesen, Sicherheit
|
||||
des Urteils, entscheidender Regel und Hinweisen; ein „unbekannt“-Ergebnis erscheint bewusst nicht als Urteil.
|
||||
Aufklappbares „Warum?“ zeigt jede geprüfte Regel mit ✓/✗/? je Feld, überstimmte Regeln durchgestrichen. Auch mit
|
||||
Nur-Lesen-Rolle nutzbar.
|
||||
|
||||
## [1.25.0] "Goshawk" — 2026-10-21
|
||||
|
||||
Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).
|
||||
|
||||
- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
|
||||
`ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
|
||||
bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
|
||||
gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
|
||||
folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
|
||||
Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
|
||||
(nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
|
||||
Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
|
||||
Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
|
||||
- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
|
||||
Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
|
||||
nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
|
||||
eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
|
||||
älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
|
||||
- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
|
||||
Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
|
||||
zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
|
||||
(nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
|
||||
der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
|
||||
- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
|
||||
vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
|
||||
erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
|
||||
sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
|
||||
- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
|
||||
- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
|
||||
gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
|
||||
laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
|
||||
sicher, wird aber angekündigt.
|
||||
- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
|
||||
26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
|
||||
Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
|
||||
Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
|
||||
übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
|
||||
Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
|
||||
Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.
|
||||
|
||||
## [1.24.0] "Merlin" — 2026-10-21
|
||||
|
||||
- **Alarme für Gateways, Ressourcen und Dienste:** Neue Ereignisse in Einstellungen › Benachrichtigungen.
|
||||
**Gateway gestört** (standardmäßig **an**) meldet ein Gateway, das OPNsense als „down“ führt oder dessen Paketverlust (ab 20 %) bzw.
|
||||
Latenz (ab 500 ms) über der Schwelle liegt; bewusst abgeschaltete und nicht überwachte Gateways bleiben still. **Ressourcen-Schwelle**
|
||||
(standardmäßig aus) prüft Festplatte (ab 90 %), Arbeitsspeicher (ab 95 %), Swap (ab 50 %) und CPU-Last (ab 2 pro Thread), **Dienst ausgefallen**
|
||||
(standardmäßig aus) eine einstellbare Liste (Unbound, Dnsmasq, DHCP, NTP, OpenVPN, WireGuard, IPsec …) und meldet nur Dienste, die zuvor
|
||||
liefen. Alle drei stützen sich auf die Minutenmessung und lösen erst aus, wenn die letzten 3 (Ressourcen: 5) Messungen übereinstimmen, und
|
||||
enden erst, wenn alle wieder in Ordnung sind: Einzelspitzen und flackernde Verbindungen schicken nichts. Der Gateway-Status (Verlust, Latenz,
|
||||
Abweichung) steht außerdem auf der Host-Seite, als Kachel auf dem Dashboard und als Prometheus-Metriken (`opnexus_host_gateway_*`,
|
||||
`opnexus_host_swap_used_percent`, `opnexus_host_service_running`). Die Messung speichert dafür zusätzlich Swap, CPU-Threads, Gateways
|
||||
und Dienste in der vorhandenen Messreihe (keine neue Tabelle).
|
||||
- **Weitere Benachrichtigungskanäle:** ntfy, Gotify, Telegram, Microsoft Teams (Workflows-/Incoming-Webhook) und Matrix, beliebig viele
|
||||
je Typ, mit „Testen“-Knopf pro Kanal. Zugangsdaten (Tokens, die Teams-Adresse) liegen wie das SMTP-Passwort in der Geheimnis-Ablage, nie
|
||||
in der Datenbank oder in Antworten, und tauchen nicht in Fehlermeldungen oder Logs auf. Webhook und neue Kanäle prüfen die Zieladresse:
|
||||
nur http(s), keine Zugangsdaten in der Adresse, keine lokalen/Link-Local-/Metadaten-Adressen; Weiterleitungen werden nicht verfolgt.
|
||||
Betrifft die bestehende Webhook-Adresse: `localhost`, Loopback, 169.254.x.x und Zugangsdaten in der Adresse (`https://user:pass@…`)
|
||||
werden jetzt abgelehnt. Eine solche gespeicherte Adresse wird nicht mehr beliefert (nur ein Log-Eintrag), und die
|
||||
Benachrichtigungs-Einstellungen lassen sich erst wieder speichern, wenn sie korrigiert oder geleert ist (die Meldung nennt den Grund).
|
||||
Neue Einstellungen kommen mit Migration 0020.
|
||||
|
||||
- **Bestand übernehmen für Aliase und NAT:** Auf den Seiten Aliase und NAT (Port Forward, 1:1, Outbound) gibt es jetzt **Von Firewall übernehmen**
|
||||
wie bei den Regeln. Der Dialog zeigt, was von Hand in OPNsense angelegt wurde, einzeln zum Übernehmen oder mit „Alle übernehmbaren“ in einem Schritt;
|
||||
verschachtelte Aliase werden in der richtigen Reihenfolge übernommen. Die Übernahme schreibt nichts auf die Firewall: Die Vorschau zeigt direkt
|
||||
danach „keine Änderung“, der nächste Rollout aktualisiert dasselbe Objekt (kein Duplikat). Nicht übernommen werden OPNsense-eigene Aliase,
|
||||
Typen, die OPNexus nicht kennt (z. B. URL, GeoIP), Aliase mit Verweis auf nicht übernommene Aliase sowie NAT-Regeln, die das Modell verfälschen würden
|
||||
(z. B. „Kein Redirect“, verknüpfte Filterregel, mehrere Schnittstellen, „Kein NAT“); die Oberfläche nennt den Grund je Objekt.
|
||||
Outbound-NAT lässt sich nur übernehmen, wenn der Modus der Firewall Hybrid oder Manuell ist. Einstellungen, die OPNexus nicht verwaltet (Protokollierung,
|
||||
Pool-Optionen, Tags, Kategorien), bleiben unverändert und werden angezeigt.
|
||||
|
||||
- **Images für ARM:** Alle Release-Images (backend, frontend, proxy, updater) gibt es jetzt für `linux/amd64` und `linux/arm64`
|
||||
(Raspberry Pi 4/5, ARM-Server, Apple-Silicon-Docker).
|
||||
|
||||
## [1.23.11] "Harrier" — 2026-10-20
|
||||
|
||||
- **Updates per Klick prüfen jetzt ebenfalls die Signatur:** Der Updater-Container enthält `cosign` (feste Version, Prüfsumme beim Bau
|
||||
geprüft) und verifiziert die neuen Images vor dem Wechsel, wenn `cosign.pub` im Projektverzeichnis liegt. Das Image wird dadurch größer
|
||||
(rund 266 statt 123 MB).
|
||||
|
||||
## [1.23.10] "Harrier" — 2026-10-20
|
||||
|
||||
- **Signierte Release-Images:** Die Images (backend, frontend, proxy, updater) werden mit cosign signiert, beginnend mit 1.23.9. Der öffentliche
|
||||
Schlüssel liegt als `cosign.pub` im Repository und im Release-Bundle. `opnexus update` prüft die Signaturen vor dem Wechsel, wenn `cosign`
|
||||
installiert ist. Anleitung: Handbuch, „Images prüfen“.
|
||||
|
||||
## [1.23.9] "Harrier" — 2026-10-20
|
||||
|
||||
- Update-Karte: Der Hinweis für Installationen ohne Release-Images nennt jetzt die tatsächliche Ursache („OPNEXUS_IMAGE_REPO ist nicht gesetzt“)
|
||||
statt „baut aus dem Quellcode“.
|
||||
- `cosign.pub`: öffentlicher Schlüssel für die Signaturen der Release-Images (das Release-Skript kann Images jetzt signieren).
|
||||
|
||||
## [1.23.8] "Harrier" — 2026-10-20
|
||||
|
||||
- **Einstellungen › OPNexus aktualisieren:** Auf einer Installation, die aus dem Quellcode gebaut wird (kein `OPNEXUS_IMAGE_REPO`), erscheint nicht mehr
|
||||
|
||||
+8
-5
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||

|
||||
|
||||
> **Stand:** `1.23.8` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.28.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -16,14 +16,17 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
- **Sichere Änderungen** — jede Änderung läuft *Prüfen → Ausrollen → Bestätigen*; bleibt die Bestätigung im Zeitfenster aus, rollt OPNexus zurück.
|
||||
- **Hierarchische Policies** — Pre-/Post-Rulebases mit Vererbung über Gruppenbäume.
|
||||
- **VPN** — IPsec-Site-to-Site, OpenVPN und WireGuard, einmal definiert und ausgerollt.
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln lassen sich übernehmen.
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln, Aliase und NAT-Regeln lassen sich übernehmen.
|
||||
- **Zertifikate** — Trust-Store für alle Firewalls und Ausstellung per ACME / Let's Encrypt.
|
||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||
- **Update-Center** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
|
||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
|
||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend.
|
||||
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
|
||||
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
|
||||
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
|
||||
@@ -58,11 +61,11 @@ Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassis
|
||||
|
||||
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
|
||||
|
||||
**Fertige Images** (amd64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.23.8
|
||||
OPNEXUS_VERSION=1.28.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||

|
||||
|
||||
> **Status:** `1.23.8` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.28.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -16,14 +16,17 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
- **Safe changes** — every change goes *preview → deploy → confirm*; without your confirmation inside the time window OPNexus rolls it back.
|
||||
- **Hierarchical policies** — pre/post rulebases with inheritance across group trees.
|
||||
- **VPN** — IPsec site-to-site, OpenVPN and WireGuard, defined once and rolled out.
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules created by hand on a firewall can be adopted.
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules, aliases and NAT rules created by hand on a firewall can be adopted.
|
||||
- **Certificates** — trust store for all firewalls and ACME / Let's Encrypt issuance.
|
||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||
- **Update center** — firmware updates for the fleet, aware of HA order and maintenance mode.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
|
||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
|
||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only.
|
||||
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
|
||||
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
|
||||
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
|
||||
@@ -60,11 +63,11 @@ After the first sign-in a short **setup assistant** opens on its own: set your o
|
||||
|
||||
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
|
||||
|
||||
**Prebuilt images** (amd64) are published to this Gitea's registry. In `.env`:
|
||||
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.23.8
|
||||
OPNEXUS_VERSION=1.28.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ from starlette.responses import JSONResponse
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.errors import ErrorCode
|
||||
from app.routers.auth import COOKIE_NAME
|
||||
from app.routers.auth import session_token
|
||||
from app.routers import governance
|
||||
from app.routers.auth import client_ip
|
||||
from app.services import api_token_service, auth_service
|
||||
@@ -56,9 +56,9 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
_SECURITY_KINDS = {
|
||||
"ipsec": "ipsec_tunnel", "openvpn": "openvpn_instance", "wireguard": "wireguard_tunnel",
|
||||
"certificates": "certificate", "acme": "acme", "notifications": "notification", "auth": "auth",
|
||||
"tokens": "api_token",
|
||||
"tokens": "api_token", "rollouts": "rollout",
|
||||
}
|
||||
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings"}
|
||||
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings", "plans", "runs"}
|
||||
|
||||
|
||||
def _is_uuid(value: str) -> bool:
|
||||
@@ -94,10 +94,14 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if request.url.path in _PUBLIC_PATHS:
|
||||
return await call_next(request)
|
||||
if request.url.path.startswith("/internal/gui/"):
|
||||
# GUI gateway only: never routed by the proxy, authenticated by the
|
||||
# shared gateway token inside routers/gui_access.py.
|
||||
return await call_next(request)
|
||||
|
||||
api_token = None
|
||||
bearer = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
if bearer is not None:
|
||||
# An API token wins over a cookie; a bad one never falls back to it.
|
||||
async with SessionLocal() as db:
|
||||
@@ -117,7 +121,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
)
|
||||
if user is not None:
|
||||
parts = request.url.path.strip("/").split("/")
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup"}
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
|
||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||
|
||||
@@ -6,6 +6,17 @@ class Settings(BaseSettings):
|
||||
def allowed_origins(self) -> list[str]:
|
||||
return [o.strip().rstrip("/") for o in self.frontend_origin.split(",") if o.strip()]
|
||||
|
||||
@property
|
||||
def gui_site_origins(self) -> list[str]:
|
||||
return [o.strip().rstrip("/") for o in self.opnexus_gui_sites.split(",") if o.strip()]
|
||||
|
||||
def gui_origin_conflicts(self) -> list[str]:
|
||||
"""Slot origins of the GUI gateway that are also listed in FRONTEND_ORIGIN.
|
||||
Such an origin would get CORS with credentials and pass the Sec-Fetch-Site
|
||||
exception — a firewall GUI could then read OPNexus' API. Startup is refused."""
|
||||
allowed = {o.lower() for o in self.allowed_origins}
|
||||
return [o for o in self.gui_site_origins if o.lower() in allowed]
|
||||
|
||||
database_url: str = (
|
||||
"postgresql+asyncpg://opnexus:opnexus@postgres:5432/opnexus"
|
||||
)
|
||||
@@ -92,6 +103,14 @@ class Settings(BaseSettings):
|
||||
# per-IP lockout entirely.
|
||||
trust_proxy_headers: bool = False
|
||||
|
||||
# ── GUI gateway (1.27, optional container "gui-gateway") ─────────
|
||||
# Slot origins the gateway is served on, comma-separated, e.g.
|
||||
# "https://192.0.2.10:9441,https://192.0.2.10:9442". Empty = feature unavailable.
|
||||
opnexus_gui_sites: str = ""
|
||||
# Shared secret between backend and gateway for /internal/gui/* (at least
|
||||
# 32 characters). Empty = the internal endpoints answer 404.
|
||||
gui_gateway_token: str = ""
|
||||
|
||||
# ── Deploy lockout protection ────────────────────────────────────
|
||||
deploy_confirm_window_seconds: int = 180
|
||||
|
||||
|
||||
@@ -87,6 +87,13 @@ class ErrorCode(str, Enum):
|
||||
MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN = "MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN"
|
||||
NICHT_AUTHENTIFIZIERT = "NICHT_AUTHENTIFIZIERT"
|
||||
CROSS_ORIGIN_BLOCKED = "CROSS_ORIGIN_BLOCKED"
|
||||
CROSS_SITE_REQUEST_BLOCKED = "CROSS_SITE_REQUEST_BLOCKED"
|
||||
GUI_SESSION_REQUIRED = "GUI_SESSION_REQUIRED"
|
||||
GUI_GATEWAY_NOT_CONFIGURED = "GUI_GATEWAY_NOT_CONFIGURED"
|
||||
GUI_REQUIRES_HTTPS = "GUI_REQUIRES_HTTPS"
|
||||
GUI_ACCESS_DISABLED = "GUI_ACCESS_DISABLED"
|
||||
GUI_HOST_NOT_ELIGIBLE = "GUI_HOST_NOT_ELIGIBLE"
|
||||
GUI_NO_FREE_SLOT = "GUI_NO_FREE_SLOT"
|
||||
TLS_PROBE_FAILED = "TLS_PROBE_FAILED"
|
||||
TLS_PEM_INVALID = "TLS_PEM_INVALID"
|
||||
TLS_PIN_MISMATCH = "TLS_PIN_MISMATCH"
|
||||
@@ -116,6 +123,13 @@ class ErrorCode(str, Enum):
|
||||
NETWORK_FIELD_INVALID = "NETWORK_FIELD_INVALID"
|
||||
NETWORK_ITEM_NOT_FOUND = "NETWORK_ITEM_NOT_FOUND"
|
||||
RULE_NOT_ADOPTABLE = "RULE_NOT_ADOPTABLE"
|
||||
OBJECT_NOT_ADOPTABLE = "OBJECT_NOT_ADOPTABLE"
|
||||
ADOPT_OBJECT_NOT_FOUND = "ADOPT_OBJECT_NOT_FOUND"
|
||||
OUTBOUND_MODE_BLOCKS_ADOPT = "OUTBOUND_MODE_BLOCKS_ADOPT"
|
||||
ADOPT_LINK_TARGET_NOT_FOUND = "ADOPT_LINK_TARGET_NOT_FOUND"
|
||||
ADOPT_LINK_TARGET_CHANGED = "ADOPT_LINK_TARGET_CHANGED"
|
||||
ADOPT_LINK_HOST_HAS_OBJECT = "ADOPT_LINK_HOST_HAS_OBJECT"
|
||||
ADOPT_LINK_BUSY = "ADOPT_LINK_BUSY"
|
||||
REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0 = "REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0"
|
||||
RESTORE_VORGANG_NICHT_GEFUNDEN = "RESTORE_VORGANG_NICHT_GEFUNDEN"
|
||||
ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL = "ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL"
|
||||
@@ -162,6 +176,11 @@ class ErrorCode(str, Enum):
|
||||
VARIABLES_MISSING = "VARIABLES_MISSING"
|
||||
VARIABLE_INVALID = "VARIABLE_INVALID"
|
||||
OVERLAY_INVALID = "OVERLAY_INVALID"
|
||||
NOTIFICATION_CHANNEL_INVALID = "NOTIFICATION_CHANNEL_INVALID"
|
||||
NOTIFICATION_CHANNEL_NOT_FOUND = "NOTIFICATION_CHANNEL_NOT_FOUND"
|
||||
NOTIFICATION_CHANNEL_NAME_TAKEN = "NOTIFICATION_CHANNEL_NAME_TAKEN"
|
||||
NOTIFICATION_CHANNEL_TEST_FAILED = "NOTIFICATION_CHANNEL_TEST_FAILED"
|
||||
NOTIFICATION_WEBHOOK_URL_INVALID = "NOTIFICATION_WEBHOOK_URL_INVALID"
|
||||
OVERLAY_SUBNET_LOCKED = "OVERLAY_SUBNET_LOCKED"
|
||||
OVERLAY_NAME_TAKEN = "OVERLAY_NAME_TAKEN"
|
||||
OVERLAY_PEER_NOT_FOUND = "OVERLAY_PEER_NOT_FOUND"
|
||||
@@ -171,6 +190,32 @@ class ErrorCode(str, Enum):
|
||||
API_TOKEN_NOT_FOUND = "API_TOKEN_NOT_FOUND"
|
||||
API_TOKEN_SCOPE_FORBIDDEN = "API_TOKEN_SCOPE_FORBIDDEN"
|
||||
API_TOKEN_SCOPE_TOO_HIGH = "API_TOKEN_SCOPE_TOO_HIGH"
|
||||
WINDOW_NOT_FOUND = "WINDOW_NOT_FOUND"
|
||||
WINDOW_INVALID = "WINDOW_INVALID"
|
||||
WINDOW_NAME_TAKEN = "WINDOW_NAME_TAKEN"
|
||||
WINDOW_IN_USE = "WINDOW_IN_USE"
|
||||
WINDOW_EXCEPTION_INVALID = "WINDOW_EXCEPTION_INVALID"
|
||||
WINDOW_EXCEPTION_EXISTS = "WINDOW_EXCEPTION_EXISTS"
|
||||
WINDOW_EXCEPTION_NOT_FOUND = "WINDOW_EXCEPTION_NOT_FOUND"
|
||||
ROLLOUT_PLAN_NOT_FOUND = "ROLLOUT_PLAN_NOT_FOUND"
|
||||
ROLLOUT_INVALID = "ROLLOUT_INVALID"
|
||||
ROLLOUT_NAME_TAKEN = "ROLLOUT_NAME_TAKEN"
|
||||
ROLLOUT_PLAN_INVALID = "ROLLOUT_PLAN_INVALID"
|
||||
ROLLOUT_PREVIEW_STALE = "ROLLOUT_PREVIEW_STALE"
|
||||
ROLLOUT_PLAN_BUSY = "ROLLOUT_PLAN_BUSY"
|
||||
ROLLOUT_GROUP_IN_USE = "ROLLOUT_GROUP_IN_USE"
|
||||
ROLLOUT_RUN_NOT_FOUND = "ROLLOUT_RUN_NOT_FOUND"
|
||||
ROLLOUT_ALREADY_ACTIVE = "ROLLOUT_ALREADY_ACTIVE"
|
||||
ROLLOUT_CONFIRM_NAME = "ROLLOUT_CONFIRM_NAME"
|
||||
ROLLOUT_ACTION_NOT_ALLOWED = "ROLLOUT_ACTION_NOT_ALLOWED"
|
||||
ROLLOUT_HOST_NOT_IN_RUN = "ROLLOUT_HOST_NOT_IN_RUN"
|
||||
ROLLOUT_HOST_RUNNING = "ROLLOUT_HOST_RUNNING"
|
||||
ROLLOUT_HOST_FAILED = "ROLLOUT_HOST_FAILED"
|
||||
ANALYSE_ADRESSE_UNGUELTIG = "ANALYSE_ADRESSE_UNGUELTIG"
|
||||
ANALYSE_FAMILIEN_MISCHUNG = "ANALYSE_FAMILIEN_MISCHUNG"
|
||||
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
|
||||
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
|
||||
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
|
||||
|
||||
|
||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||
|
||||
@@ -0,0 +1,587 @@
|
||||
"""The ``gui-gateway`` container: firewall web interfaces in their own browser origin (1.27).
|
||||
|
||||
Runs from the backend image (``python -m app.gui_gateway``) behind the proxy, which terminates TLS for the
|
||||
slot ports and sets ``X-Onx-Slot`` (overwriting any client value). It never touches the database or the
|
||||
secret stores; it only talks to the backend's internal API (``/internal/gui/*``, shared GUI_GATEWAY_TOKEN)
|
||||
and, per gateway session, to exactly one firewall.
|
||||
|
||||
Security model (see the 1.27 design):
|
||||
* Every slot port is its own origin. The firewall GUI therefore never runs in the OPNexus origin.
|
||||
* Cookies never cross: the browser's ``Cookie`` header is not forwarded (it carries the OPNexus session),
|
||||
and the firewall's ``Set-Cookie`` never reaches the browser — a server-side jar per gateway session
|
||||
(memory only) holds them. The browser only gets ``__Host-onx-gw-<slot>``.
|
||||
* Each upstream request is built from scratch: method, path and query from the browser, a small header
|
||||
allow-list, the jar's cookies; target origin, pinned certificate and overlay proxy from the backend.
|
||||
Absolute-form targets, CONNECT/TRACE, ``Upgrade`` and service-worker script requests are refused.
|
||||
* ``Sec-Fetch-Site`` must be same-origin/none (same-site only for the redeem step), fail-closed;
|
||||
non-GET/HEAD requests also need the slot's own ``Origin``.
|
||||
* ``Location`` pointing anywhere but the firewall is dropped; responses get CSP, X-Frame-Options DENY,
|
||||
``Cache-Control: no-store``.
|
||||
* At most 3 failed GUI logins per firewall in 5 minutes (protects OPNexus' API access from the firewall's
|
||||
login lockout, which would see all attempts coming from OPNexus).
|
||||
* Idle and absolute limits; every 20 s the backend re-checks the session (OPNexus session, admin role,
|
||||
switch, host). Logs carry method, path (no query), status and duration only.
|
||||
|
||||
OPNsense behaviour this relies on (opnsense/core, src/www/authgui.inc, master as of 2026-10-21):
|
||||
* session_auth(): a login is ``isset($_POST['login']) && !empty($_POST['usernamefld']) &&
|
||||
!empty($_POST['passwordfld'])`` posted to the current URL (login form has no action). Success sends a
|
||||
``Location:`` header and exits (302); a failure re-renders the login form (200) with "Wrong username or
|
||||
password." and an auth_log entry.
|
||||
* check_security_http_referer_enforcement() only runs when HTTP_REFERER is set, and wants the firewall's
|
||||
own host name/IP — so the gateway sends no Referer (and no Origin) upstream at all.
|
||||
* check_security_dns_rebind() accepts an IP address or the configured host name as Host — the gateway
|
||||
sends the Host of the host record's base URL (httpx derives it from the URL).
|
||||
"""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
import ssl
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from http.cookies import CookieError, SimpleCookie
|
||||
from typing import Awaitable, Callable
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("gui_gateway")
|
||||
# httpx/httpcore log full request URLs (query strings) at INFO/DEBUG: never in this process.
|
||||
for _noisy in ("httpx", "httpcore"):
|
||||
logging.getLogger(_noisy).setLevel(logging.WARNING)
|
||||
|
||||
LISTEN_PORT = 8080
|
||||
COOKIE_PREFIX = "__Host-onx-gw-"
|
||||
OPNEXUS_SESSION_COOKIE = "__Host-opnexus-session"
|
||||
BODY_LIMIT = 64 * 1024 * 1024 # config restore uploads
|
||||
LOGIN_FAILURE_LIMIT = 3
|
||||
LOGIN_FAILURE_WINDOW = 300.0
|
||||
UPSTREAM_TIMEOUT = 60.0
|
||||
MAX_SESSIONS = 64
|
||||
|
||||
# Request headers copied from the browser; everything else (Cookie, Authorization, Origin, Referer,
|
||||
# X-Forwarded-*, hop-by-hop, Sec-*, ...) is dropped.
|
||||
_FORWARD_REQUEST_HEADERS = {
|
||||
"accept", "accept-language", "content-type", "user-agent", "x-requested-with", "x-csrftoken", "range",
|
||||
}
|
||||
# Response headers copied from the firewall. Set-Cookie goes to the jar, Location is rewritten.
|
||||
_FORWARD_RESPONSE_HEADERS = {
|
||||
"content-type", "content-length", "content-encoding", "content-disposition", "content-language",
|
||||
"content-range", "accept-ranges", "content-security-policy",
|
||||
}
|
||||
_GATEWAY_CSP = ("connect-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'self'; "
|
||||
"object-src 'none'; worker-src 'none'")
|
||||
_SECURITY_HEADERS = [
|
||||
(b"content-security-policy", _GATEWAY_CSP.encode()),
|
||||
(b"x-frame-options", b"DENY"),
|
||||
(b"x-content-type-options", b"nosniff"),
|
||||
(b"cache-control", b"no-store"),
|
||||
(b"pragma", b"no-cache"),
|
||||
(b"referrer-policy", b"same-origin"),
|
||||
(b"cross-origin-opener-policy", b"same-origin"),
|
||||
(b"cross-origin-resource-policy", b"same-origin"),
|
||||
(b"permissions-policy", b"camera=(), microphone=(), geolocation=(), payment=(), usb=()"),
|
||||
]
|
||||
|
||||
|
||||
def _sha(value: str) -> str:
|
||||
return hashlib.sha256(value.encode()).hexdigest()
|
||||
|
||||
|
||||
def _origin_of(url: str) -> str | None:
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
return None
|
||||
if parts.scheme not in ("http", "https") or not parts.hostname:
|
||||
return None
|
||||
default = 443 if parts.scheme == "https" else 80
|
||||
host = parts.hostname.lower()
|
||||
if ":" in host:
|
||||
host = f"[{host}]"
|
||||
return f"{parts.scheme}://{host}" + (f":{port}" if port and port != default else "")
|
||||
|
||||
|
||||
def slot_origins_from(sites: str) -> dict[int, set[str]]:
|
||||
"""OPNEXUS_GUI_SITES -> {port: {normalised origins}} (https only, explicit port)."""
|
||||
slots: dict[int, set[str]] = {}
|
||||
for raw in sites.split(","):
|
||||
raw = raw.strip().rstrip("/")
|
||||
try:
|
||||
parts = urlsplit(raw)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
continue
|
||||
if parts.scheme == "https" and parts.hostname and port:
|
||||
slots.setdefault(port, set()).add(_origin_of(raw))
|
||||
return slots
|
||||
|
||||
|
||||
@dataclass
|
||||
class GatewaySession:
|
||||
gui_session_id: str
|
||||
session_hash: str
|
||||
slot: int
|
||||
upstream_origin: str
|
||||
ca_pem: str | None
|
||||
proxy_url: str | None
|
||||
idle_timeout: float
|
||||
absolute_deadline: float
|
||||
revalidate_every: float
|
||||
last_activity: float = field(default_factory=time.monotonic)
|
||||
next_check: float = 0.0
|
||||
jar: dict[str, str] = field(default_factory=dict)
|
||||
client: httpx.AsyncClient | None = None
|
||||
|
||||
|
||||
class BackendClient:
|
||||
"""The backend's internal GUI API. Raises httpx.HTTPError when the backend is unreachable."""
|
||||
|
||||
def __init__(self, base_url: str, token: str, transport: httpx.AsyncBaseTransport | None = None):
|
||||
self._client = httpx.AsyncClient(base_url=base_url, headers={"X-Onx-Gateway-Token": token},
|
||||
timeout=10.0, transport=transport)
|
||||
|
||||
async def redeem(self, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
|
||||
resp = await self._client.post("/internal/gui/redeem", json={
|
||||
"ticket_hash": ticket_hash, "session_hash": session_hash, "slot": slot, "client_ip": client_ip})
|
||||
if resp.status_code == 200:
|
||||
return resp.json()
|
||||
if resp.status_code == 403:
|
||||
return {"ok": False, "reason": (resp.json().get("detail") or {}).get("reason", "denied")}
|
||||
resp.raise_for_status()
|
||||
return {"ok": False, "reason": f"http_{resp.status_code}"}
|
||||
|
||||
async def revalidate(self, gs: GatewaySession) -> dict:
|
||||
resp = await self._client.post("/internal/gui/revalidate",
|
||||
json={"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash})
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def end(self, gs: GatewaySession, reason: str) -> None:
|
||||
try:
|
||||
await self._client.post("/internal/gui/end", json={
|
||||
"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash, "reason": reason})
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Could not report the end of a GUI session (%s)", type(exc).__name__)
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._client.aclose()
|
||||
|
||||
|
||||
def build_upstream_client(gs: GatewaySession) -> httpx.AsyncClient:
|
||||
"""Same transport rules as opnsense_client._client: pinned certificate (or system CAs), overlay proxy."""
|
||||
verify: ssl.SSLContext | bool
|
||||
if gs.ca_pem:
|
||||
from app.services.tls_pin_service import pinned_context # no database layer behind this import
|
||||
verify = pinned_context(gs.ca_pem)
|
||||
else:
|
||||
verify = True
|
||||
return httpx.AsyncClient(verify=verify, proxy=gs.proxy_url, timeout=UPSTREAM_TIMEOUT,
|
||||
follow_redirects=False, trust_env=False)
|
||||
|
||||
|
||||
class LoginLimiter:
|
||||
"""At most LOGIN_FAILURE_LIMIT failed GUI logins per firewall within LOGIN_FAILURE_WINDOW seconds."""
|
||||
|
||||
def __init__(self, clock: Callable[[], float] = time.monotonic):
|
||||
self._failures: dict[str, deque] = {}
|
||||
self._clock = clock
|
||||
|
||||
def _recent(self, key: str) -> deque:
|
||||
now = self._clock()
|
||||
q = self._failures.setdefault(key, deque())
|
||||
while q and now - q[0] > LOGIN_FAILURE_WINDOW:
|
||||
q.popleft()
|
||||
return q
|
||||
|
||||
def blocked(self, key: str) -> bool:
|
||||
return len(self._recent(key)) >= LOGIN_FAILURE_LIMIT
|
||||
|
||||
def record_failure(self, key: str) -> None:
|
||||
self._recent(key).append(self._clock())
|
||||
|
||||
|
||||
def is_login_attempt(method: str, content_type: str, body: bytes) -> bool:
|
||||
if method != "POST":
|
||||
return False
|
||||
if content_type.lower().startswith("multipart/form-data"):
|
||||
# PHP fills $_POST from multipart bodies too, so authgui.inc accepts a login posted this way.
|
||||
return all(f'name="{field}"'.encode() in body for field in ("login", "usernamefld", "passwordfld"))
|
||||
if not content_type.lower().startswith("application/x-www-form-urlencoded"):
|
||||
return False
|
||||
try:
|
||||
form = parse_qs(body.decode("latin-1"), keep_blank_values=True)
|
||||
except ValueError:
|
||||
return False
|
||||
return "login" in form and bool(form.get("usernamefld", [""])[0]) and bool(form.get("passwordfld", [""])[0])
|
||||
|
||||
|
||||
def rewrite_location(location: str, upstream_origin: str) -> str | None:
|
||||
"""Relative target for the browser, or None when the redirect leaves the firewall."""
|
||||
location = location.strip()
|
||||
if not location or "\\" in location or any(ord(c) < 0x20 for c in location):
|
||||
return None
|
||||
if location.startswith("//"):
|
||||
location = "https:" + location
|
||||
parts = urlsplit(location)
|
||||
if parts.scheme or parts.netloc:
|
||||
if _origin_of(location) != _origin_of(upstream_origin):
|
||||
return None
|
||||
path = parts.path or "/"
|
||||
return path + (f"?{parts.query}" if parts.query else "") + (f"#{parts.fragment}" if parts.fragment else "")
|
||||
return location
|
||||
|
||||
|
||||
def _page(title: str, text_de: str, text_en: str) -> bytes:
|
||||
return (f"<!doctype html><html lang=\"de\"><head><meta charset=\"utf-8\"><title>{title}</title></head>"
|
||||
f"<body style=\"font-family:sans-serif;max-width:40rem;margin:3rem auto\"><h1>{title}</h1>"
|
||||
f"<p>{text_de}</p><p lang=\"en\">{text_en}</p></body></html>").encode()
|
||||
|
||||
|
||||
_SESSION_GONE = _page("OPNexus", "Diese Sitzung ist abgelaufen oder ungültig. Öffne die Oberfläche erneut aus OPNexus.",
|
||||
"This session has expired or is invalid. Open the interface again from OPNexus.")
|
||||
_FORBIDDEN = _page("OPNexus", "Diese Anfrage wurde vom GUI-Gateway abgelehnt.", "This request was refused by the GUI gateway.")
|
||||
_BAD_REDIRECT = _page("OPNexus", "Die Firewall wollte auf eine fremde Adresse umleiten; die Umleitung wurde verworfen.",
|
||||
"The firewall tried to redirect to a foreign address; the redirect was dropped.")
|
||||
_LOGIN_LOCKED = _page("OPNexus", "Zu viele fehlgeschlagene Anmeldungen an dieser Firewall. Bitte warte 5 Minuten.",
|
||||
"Too many failed sign-ins at this firewall. Please wait 5 minutes.")
|
||||
_UNAVAILABLE = _page("OPNexus", "Die Firewall oder OPNexus ist gerade nicht erreichbar.",
|
||||
"The firewall or OPNexus is not reachable right now.")
|
||||
_REDEEMED = (b"<!doctype html><html><head><meta charset=\"utf-8\"><meta http-equiv=\"refresh\" content=\"0;url=/\">"
|
||||
b"<title>OPNexus</title></head><body><a href=\"/\">→</a></body></html>")
|
||||
|
||||
|
||||
class GuiGateway:
|
||||
def __init__(self, *, backend: BackendClient, slot_origins: dict[int, set[str]],
|
||||
upstream_factory: Callable[[GatewaySession], httpx.AsyncClient] = build_upstream_client,
|
||||
clock: Callable[[], float] = time.monotonic, wall_clock: Callable[[], float] = time.time):
|
||||
self.backend = backend
|
||||
self.slot_origins = slot_origins
|
||||
self.upstream_factory = upstream_factory
|
||||
self.clock = clock
|
||||
self.wall_clock = wall_clock
|
||||
self.sessions: dict[str, GatewaySession] = {} # sha256(gateway cookie) -> session
|
||||
self.limiter = LoginLimiter(clock)
|
||||
self._loop_task: asyncio.Task | None = None
|
||||
|
||||
# ── ASGI ──────────────────────────────────────────────────────────────
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] == "lifespan":
|
||||
await self._lifespan(receive, send)
|
||||
return
|
||||
if scope["type"] != "http":
|
||||
# WebSocket and anything else: not supported (design: no WebSocket).
|
||||
if scope["type"] == "websocket":
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
return
|
||||
started = time.monotonic()
|
||||
status = await self._handle(scope, receive, send)
|
||||
# Method, path and status only: never the query string, headers or bodies.
|
||||
logger.info("%s %s %d %.0fms", scope.get("method", "?"), scope.get("path", "?")[:200], status,
|
||||
(time.monotonic() - started) * 1000)
|
||||
|
||||
async def _lifespan(self, receive, send):
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] == "lifespan.startup":
|
||||
self._loop_task = asyncio.create_task(self._revalidate_loop())
|
||||
await send({"type": "lifespan.startup.complete"})
|
||||
elif message["type"] == "lifespan.shutdown":
|
||||
if self._loop_task:
|
||||
self._loop_task.cancel()
|
||||
for key in list(self.sessions):
|
||||
await self._drop(key, "shutdown")
|
||||
await self.backend.aclose()
|
||||
await send({"type": "lifespan.shutdown.complete"})
|
||||
return
|
||||
|
||||
async def _revalidate_loop(self):
|
||||
while True:
|
||||
await asyncio.sleep(10)
|
||||
try:
|
||||
await self.sweep()
|
||||
except Exception: # never let the loop die
|
||||
logger.exception("GUI session sweep failed")
|
||||
|
||||
async def sweep(self) -> None:
|
||||
"""Idle/absolute limits and the periodic backend re-check for every gateway session."""
|
||||
for key, gs in list(self.sessions.items()):
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
elif self.clock() >= gs.next_check:
|
||||
await self._recheck(key, gs)
|
||||
|
||||
# ── helpers ───────────────────────────────────────────────────────────
|
||||
@staticmethod
|
||||
def _headers(scope) -> dict[str, str]:
|
||||
result: dict[str, str] = {}
|
||||
for k, v in scope.get("headers", []):
|
||||
name = k.decode("latin-1").lower()
|
||||
value = v.decode("latin-1")
|
||||
if name in result:
|
||||
value = f"{result[name]}{'; ' if name == 'cookie' else ', '}{value}"
|
||||
result[name] = value
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _cookies(header: str) -> dict[str, str]:
|
||||
cookies: dict[str, str] = {}
|
||||
for part in header.split(";"):
|
||||
name, sep, value = part.strip().partition("=")
|
||||
if sep and name and name not in cookies:
|
||||
cookies[name] = value
|
||||
return cookies
|
||||
|
||||
async def _respond(self, send, status: int, body: bytes = b"", headers: list | None = None,
|
||||
content_type: bytes = b"text/html; charset=utf-8") -> int:
|
||||
out = [(b"content-type", content_type), (b"content-length", str(len(body)).encode())]
|
||||
out += list(headers or []) + _SECURITY_HEADERS
|
||||
await send({"type": "http.response.start", "status": status, "headers": out})
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
return status
|
||||
|
||||
def _expired(self, gs: GatewaySession) -> str | None:
|
||||
if self.wall_clock() >= gs.absolute_deadline:
|
||||
return "absolute"
|
||||
if self.clock() - gs.last_activity >= gs.idle_timeout:
|
||||
return "idle"
|
||||
return None
|
||||
|
||||
async def _drop(self, key: str, reason: str, *, report: bool = True) -> None:
|
||||
gs = self.sessions.pop(key, None)
|
||||
if gs is None:
|
||||
return
|
||||
gs.jar.clear()
|
||||
if gs.client is not None:
|
||||
await gs.client.aclose()
|
||||
if report:
|
||||
await self.backend.end(gs, reason)
|
||||
|
||||
async def _recheck(self, key: str, gs: GatewaySession) -> bool | None:
|
||||
"""True = still valid, False = ended (dropped), None = backend unreachable (fail closed, keep for retry)."""
|
||||
try:
|
||||
result = await self.backend.revalidate(gs)
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("GUI session re-check failed (%s)", type(exc).__name__)
|
||||
return None
|
||||
if not result.get("ok"):
|
||||
await self._drop(key, str(result.get("reason", "revoked")), report=False) # backend already ended it
|
||||
return False
|
||||
gs.absolute_deadline = min(gs.absolute_deadline, float(result.get("absolute_deadline", gs.absolute_deadline)))
|
||||
gs.idle_timeout = float(result.get("idle_timeout_seconds", gs.idle_timeout))
|
||||
gs.next_check = self.clock() + gs.revalidate_every
|
||||
return True
|
||||
|
||||
# ── request handling ──────────────────────────────────────────────────
|
||||
async def _handle(self, scope, receive, send) -> int:
|
||||
method = scope.get("method", "")
|
||||
path = scope.get("path", "")
|
||||
raw_path = scope.get("raw_path") or path.encode()
|
||||
headers = self._headers(scope)
|
||||
|
||||
if method in ("CONNECT", "TRACE"):
|
||||
return await self._respond(send, 501, _FORBIDDEN)
|
||||
# Origin-form only: "GET http://elsewhere/ HTTP/1.1" (absolute-form) or "//host" never reach upstream.
|
||||
if not path.startswith("/") or path.startswith("//") or not raw_path.startswith(b"/") \
|
||||
or raw_path.startswith(b"//") or "\\" in path:
|
||||
return await self._respond(send, 400, _FORBIDDEN)
|
||||
if "upgrade" in headers or "upgrade" in headers.get("connection", "").lower():
|
||||
return await self._respond(send, 400, _FORBIDDEN)
|
||||
if headers.get("service-worker", "").lower() == "script":
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
|
||||
try:
|
||||
slot = int(headers.get("x-onx-slot", ""))
|
||||
except ValueError:
|
||||
slot = 0
|
||||
own_origins = self.slot_origins.get(slot)
|
||||
if not own_origins:
|
||||
return await self._respond(send, 400, _FORBIDDEN) # not coming through the proxy's slot block
|
||||
|
||||
site = headers.get("sec-fetch-site", "").strip().lower()
|
||||
if path == "/__onx/redeem":
|
||||
if method != "GET" or site not in ("same-site", "same-origin", "none"):
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
return await self._redeem(scope, headers, slot, send)
|
||||
if site not in ("same-origin", "none"):
|
||||
return await self._respond(send, 403, _FORBIDDEN) # missing header = old/odd client: fail closed
|
||||
if method not in ("GET", "HEAD"):
|
||||
origin = headers.get("origin", "")
|
||||
if _origin_of(origin) not in own_origins:
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
if path.startswith("/__onx/"):
|
||||
return await self._respond(send, 404, _FORBIDDEN)
|
||||
|
||||
cookies = self._cookies(headers.get("cookie", ""))
|
||||
token = cookies.get(f"{COOKIE_PREFIX}{slot}", "")
|
||||
key = _sha(token) if token else ""
|
||||
gs = self.sessions.get(key) if key else None
|
||||
if gs is None or gs.slot != slot:
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
if self.clock() >= gs.next_check:
|
||||
state = await self._recheck(key, gs)
|
||||
if state is False:
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
if state is None:
|
||||
return await self._respond(send, 503, _UNAVAILABLE)
|
||||
gs.last_activity = self.clock()
|
||||
return await self._proxy(scope, receive, send, headers, gs)
|
||||
|
||||
async def _redeem(self, scope, headers, slot: int, send) -> int:
|
||||
query = parse_qs(scope.get("query_string", b"").decode("latin-1"))
|
||||
ticket = (query.get("t") or [""])[0]
|
||||
browser_session = self._cookies(headers.get("cookie", "")).get(OPNEXUS_SESSION_COOKIE, "")
|
||||
if not ticket or len(ticket) > 128 or not browser_session:
|
||||
return await self._respond(send, 403, _SESSION_GONE)
|
||||
client_ip = (headers.get("x-forwarded-for", "").split(",")[0].strip() or None)
|
||||
try:
|
||||
result = await self.backend.redeem(_sha(ticket), _sha(browser_session), slot, client_ip and client_ip[:45])
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Ticket redeem failed (%s)", type(exc).__name__)
|
||||
return await self._respond(send, 503, _UNAVAILABLE)
|
||||
if not result.get("ok"):
|
||||
return await self._respond(send, 403, _SESSION_GONE)
|
||||
# One browser session, one gateway session per slot: an older one in this slot is replaced.
|
||||
for old_key, old in list(self.sessions.items()):
|
||||
if old.slot == slot and old.session_hash == _sha(browser_session):
|
||||
await self._drop(old_key, "superseded", report=False)
|
||||
if len(self.sessions) >= MAX_SESSIONS:
|
||||
oldest = min(self.sessions, key=lambda k: self.sessions[k].last_activity)
|
||||
await self._drop(oldest, "superseded")
|
||||
value = secrets.token_urlsafe(32)
|
||||
gs = GatewaySession(
|
||||
gui_session_id=result["gui_session_id"], session_hash=_sha(browser_session), slot=slot,
|
||||
upstream_origin=result["upstream_origin"], ca_pem=result.get("ca_pem"), proxy_url=result.get("proxy_url"),
|
||||
idle_timeout=float(result["idle_timeout_seconds"]), absolute_deadline=float(result["absolute_deadline"]),
|
||||
revalidate_every=float(result.get("revalidate_seconds", 20)),
|
||||
)
|
||||
gs.last_activity = self.clock()
|
||||
gs.next_check = gs.last_activity + gs.revalidate_every
|
||||
self.sessions[_sha(value)] = gs
|
||||
cookie = f"{COOKIE_PREFIX}{slot}={value}; Path=/; Secure; HttpOnly; SameSite=Strict"
|
||||
# 200 + meta refresh instead of a 303: the follow-up request then starts in this origin
|
||||
# (Sec-Fetch-Site: same-origin) and the ticket leaves the address bar.
|
||||
return await self._respond(send, 200, _REDEEMED, headers=[
|
||||
(b"set-cookie", cookie.encode()), (b"clear-site-data", b'"cache", "storage"'),
|
||||
])
|
||||
|
||||
async def _read_body(self, receive) -> bytes | None:
|
||||
chunks, size = [], 0
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] == "http.disconnect":
|
||||
return None
|
||||
chunk = message.get("body", b"")
|
||||
size += len(chunk)
|
||||
if size > BODY_LIMIT:
|
||||
return None
|
||||
chunks.append(chunk)
|
||||
if not message.get("more_body"):
|
||||
return b"".join(chunks)
|
||||
|
||||
def _store_cookies(self, gs: GatewaySession, response: httpx.Response) -> None:
|
||||
for header in response.headers.get_list("set-cookie"):
|
||||
jar = SimpleCookie()
|
||||
try:
|
||||
jar.load(header)
|
||||
except CookieError:
|
||||
continue
|
||||
for name, morsel in jar.items():
|
||||
expired = morsel["max-age"] in ("0",) or morsel.value in ("", "deleted")
|
||||
if not expired and morsel["expires"]:
|
||||
try:
|
||||
from email.utils import parsedate_to_datetime
|
||||
expired = parsedate_to_datetime(morsel["expires"]).timestamp() <= self.wall_clock()
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
if expired:
|
||||
gs.jar.pop(name, None)
|
||||
else:
|
||||
gs.jar[name] = morsel.value
|
||||
|
||||
async def _proxy(self, scope, receive, send, headers: dict, gs: GatewaySession) -> int:
|
||||
length = headers.get("content-length", "")
|
||||
if length.isdigit() and int(length) > BODY_LIMIT:
|
||||
return await self._respond(send, 413, _FORBIDDEN)
|
||||
body = await self._read_body(receive)
|
||||
if body is None:
|
||||
return await self._respond(send, 413, _FORBIDDEN)
|
||||
method = scope["method"]
|
||||
login = is_login_attempt(method, headers.get("content-type", ""), body)
|
||||
if login and self.limiter.blocked(gs.upstream_origin):
|
||||
return await self._respond(send, 429, _LOGIN_LOCKED)
|
||||
|
||||
out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS}
|
||||
out_headers["accept-encoding"] = "identity"
|
||||
if gs.jar:
|
||||
out_headers["cookie"] = "; ".join(f"{k}={v}" for k, v in gs.jar.items())
|
||||
query = scope.get("query_string", b"").decode("latin-1")
|
||||
# The path exactly as the browser sent it (still percent-encoded): the decoded scope["path"] would
|
||||
# turn %3F/%23 into query/fragment, %2F into a separator and decode %25xx a second time upstream.
|
||||
raw_path = (scope.get("raw_path") or scope["path"].encode()).split(b"?", 1)[0].decode("latin-1")
|
||||
url = gs.upstream_origin + raw_path + (f"?{query}" if query else "")
|
||||
if gs.client is None:
|
||||
gs.client = self.upstream_factory(gs)
|
||||
try:
|
||||
request = gs.client.build_request(method, url, headers=out_headers, content=body)
|
||||
except httpx.InvalidURL:
|
||||
return await self._respond(send, 400, _FORBIDDEN) # e.g. control characters in the path
|
||||
try:
|
||||
response = await gs.client.send(request, stream=True)
|
||||
except (httpx.HTTPError, ssl.SSLError, OSError) as exc:
|
||||
logger.warning("Upstream request failed (%s)", type(exc).__name__)
|
||||
return await self._respond(send, 502, _UNAVAILABLE)
|
||||
try:
|
||||
self._store_cookies(gs, response)
|
||||
if login and not (300 <= response.status_code < 400):
|
||||
self.limiter.record_failure(gs.upstream_origin)
|
||||
out = []
|
||||
for name, value in response.headers.multi_items():
|
||||
lname = name.lower()
|
||||
if lname in _FORWARD_RESPONSE_HEADERS:
|
||||
out.append((lname.encode("latin-1"), value.encode("latin-1")))
|
||||
if "location" in response.headers:
|
||||
target = rewrite_location(response.headers["location"], gs.upstream_origin)
|
||||
if target is None:
|
||||
await response.aclose()
|
||||
return await self._respond(send, 502, _BAD_REDIRECT)
|
||||
out.append((b"location", target.encode("latin-1")))
|
||||
await send({"type": "http.response.start", "status": response.status_code, "headers": out + _SECURITY_HEADERS})
|
||||
async for chunk in response.aiter_raw():
|
||||
await send({"type": "http.response.body", "body": chunk, "more_body": True})
|
||||
await send({"type": "http.response.body", "body": b""})
|
||||
return response.status_code
|
||||
finally:
|
||||
await response.aclose()
|
||||
|
||||
|
||||
def create_app(transport: httpx.AsyncBaseTransport | None = None) -> GuiGateway:
|
||||
token = os.environ.get("GUI_GATEWAY_TOKEN", "")
|
||||
if len(token) < 32:
|
||||
raise SystemExit("GUI_GATEWAY_TOKEN (at least 32 characters) is required")
|
||||
slots = slot_origins_from(os.environ.get("OPNEXUS_GUI_SITES", ""))
|
||||
if not slots:
|
||||
raise SystemExit("OPNEXUS_GUI_SITES must list the https slot origins")
|
||||
backend = BackendClient(os.environ.get("GUI_BACKEND_URL", "http://backend:8000"), token, transport)
|
||||
return GuiGateway(backend=backend, slot_origins=slots)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import uvicorn
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
|
||||
# h11: an absolute-form target stays visible in scope["path"] (and is refused); access_log off because
|
||||
# uvicorn's access log would contain query strings (the redeem ticket).
|
||||
uvicorn.run(create_app(), host="0.0.0.0", port=LISTEN_PORT, http="h11", ws="none", access_log=False,
|
||||
server_header=False, date_header=False, proxy_headers=False, lifespan="on", log_config=None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -46,10 +46,16 @@ from app.version import APP_VERSION
|
||||
from app.routers import dashboard as dashboard_router
|
||||
from app.routers import cluster as cluster_router
|
||||
from app.routers import services as services_router
|
||||
from app.routers import gateways as gateways_router
|
||||
from app.routers import resource_samples as resource_samples_router
|
||||
from app.services import resource_monitor_service
|
||||
from app.routers import scheduled_maintenance as scheduled_maintenance_router
|
||||
from app.routers import maintenance_windows as maintenance_windows_router
|
||||
from app.routers import rollouts as rollouts_router
|
||||
from app.routers import rule_analysis as rule_analysis_router
|
||||
from app.routers import gui_access as gui_access_router
|
||||
from app.services import scheduled_maintenance_service
|
||||
from app.services import rollout_engine
|
||||
from app.services import compliance_service
|
||||
from app.services import audit_export_service
|
||||
from app.services import overlay_service, variables_service
|
||||
@@ -254,6 +260,12 @@ async def _scheduled_maintenance_poll_loop():
|
||||
except Exception:
|
||||
logger.exception("Geplante-Wartung-Poll fehlgeschlagen")
|
||||
in_progress = 0
|
||||
# Firmware rollouts (1.25) only ever create schedules; the next tick of the lines above fires them.
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
in_progress += await rollout_engine.advance_runs(db)
|
||||
except Exception:
|
||||
logger.exception("Rollout-Poll fehlgeschlagen")
|
||||
# A handover run is stepped quickly (the partner should take over within a minute or two);
|
||||
# otherwise the relaxed cadence is plenty.
|
||||
await asyncio.sleep(_RUNNING_MAINTENANCE_POLL_SECONDS if in_progress else _SCHEDULED_MAINTENANCE_POLL_SECONDS)
|
||||
@@ -312,6 +324,10 @@ async def _backup_poll_loop():
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
conflicts = settings.gui_origin_conflicts()
|
||||
if conflicts:
|
||||
# A GUI-gateway slot must never be a trusted OPNexus origin (CORS with credentials).
|
||||
raise RuntimeError(f"OPNEXUS_GUI_SITES overlaps FRONTEND_ORIGIN: {', '.join(conflicts)} — refusing to start")
|
||||
await migrate.run()
|
||||
async with SessionLocal() as db:
|
||||
await auth_service.bootstrap_admin(db)
|
||||
@@ -510,9 +526,15 @@ app.include_router(restore_router.router, prefix="/api")
|
||||
app.include_router(dashboard_router.router, prefix="/api")
|
||||
app.include_router(cluster_router.router, prefix="/api")
|
||||
app.include_router(services_router.router, prefix="/api")
|
||||
app.include_router(gateways_router.router, prefix="/api")
|
||||
app.include_router(network_router.router, prefix="/api")
|
||||
app.include_router(snapshots_router.router, prefix="/api")
|
||||
app.include_router(network_drift_router.router, prefix="/api")
|
||||
app.include_router(diagnostics_router.router, prefix="/api")
|
||||
app.include_router(resource_samples_router.router, prefix="/api")
|
||||
app.include_router(scheduled_maintenance_router.router, prefix="/api")
|
||||
app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||
app.include_router(rollouts_router.router, prefix="/api")
|
||||
app.include_router(rule_analysis_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.internal_router)
|
||||
@@ -93,3 +93,12 @@ __all__ += ["HostResourceSample"]
|
||||
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
__all__ += ["ScheduledMaintenance"]
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
__all__ += ["MaintenanceWindow", "MaintenanceWindowException"]
|
||||
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun, RolloutRunHost, RolloutRunWave, rollout_wave_groups
|
||||
__all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "RolloutRunWave", "rollout_wave_groups"]
|
||||
|
||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||
__all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
|
||||
@@ -0,0 +1,65 @@
|
||||
"""GUI access to managed firewalls through the separate gui-gateway container (1.27).
|
||||
|
||||
Settings are a singleton row (id fixed at 1), off by default. Tickets and gateway
|
||||
sessions only ever store hashes: the raw ticket travels once through the browser,
|
||||
the OPNexus session is referenced by the same SHA-256 that `sessions.token_hash` holds.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
class GuiAccessSettings(Base):
|
||||
__tablename__ = "gui_access_settings"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
idle_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=15, server_default="15")
|
||||
absolute_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
|
||||
slot_count: Mapped[int] = mapped_column(Integer, nullable=False, default=4, server_default="4")
|
||||
updated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class GuiTicket(Base):
|
||||
"""One-time, 30-second ticket, bound to the OPNexus session, host and slot."""
|
||||
__tablename__ = "gui_tickets"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
ticket_hash: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
slot: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
redeemed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class GuiSession(Base):
|
||||
"""A redeemed ticket: one firewall GUI open in one slot of one browser session."""
|
||||
__tablename__ = "gui_sessions"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
slot: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
client_ip: Mapped[str | None] = mapped_column(String(45), nullable=True)
|
||||
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
last_checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
end_reason: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Recurring maintenance windows (1.25): "every Sunday 02:00-05:00 Europe/Berlin",
|
||||
"first Tuesday of the month 22:00 for 3 h". A window on its own changes nothing on any
|
||||
firewall; rollout plans (models/rollout.py) use it to decide WHEN they may start work on a host.
|
||||
|
||||
Times are wall-clock times in the window's own IANA time zone, so a window stays at 02:00
|
||||
local time across daylight-saving changes (see services/window_service.py for the rules).
|
||||
"""
|
||||
import uuid
|
||||
from datetime import date, datetime, time, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, Date, DateTime, ForeignKey, Integer, String, Time, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class MaintenanceWindow(Base):
|
||||
__tablename__ = "maintenance_windows"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(120), unique=True)
|
||||
timezone: Mapped[str] = mapped_column(String(64))
|
||||
recurrence: Mapped[str] = mapped_column(String(16)) # weekly | monthly
|
||||
# weekly: weekdays 0 (Monday) .. 6 (Sunday). JSON columns get a Python default only (Alembic gotcha, see 0007).
|
||||
weekdays: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# monthly: "nth_weekday" (month_nth 1-4 or -1 = last, month_weekday 0-6) or "day" (month_day 1-28)
|
||||
month_mode: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
month_nth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
month_weekday: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
month_day: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
start_time: Mapped[time] = mapped_column(Time)
|
||||
duration_minutes: Mapped[int] = mapped_column(Integer)
|
||||
# After this many minutes into the window no NEW host action starts (running ones finish).
|
||||
start_cutoff_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
created_by: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
|
||||
exceptions: Mapped[list["MaintenanceWindowException"]] = relationship(
|
||||
"MaintenanceWindowException", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by="MaintenanceWindowException.occurrence_date",
|
||||
)
|
||||
|
||||
|
||||
class MaintenanceWindowException(Base):
|
||||
"""One occurrence skipped or moved. Keyed by the occurrence's LOCAL date."""
|
||||
|
||||
__tablename__ = "maintenance_window_exceptions"
|
||||
__table_args__ = (UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
window_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
occurrence_date: Mapped[date] = mapped_column(Date)
|
||||
kind: Mapped[str] = mapped_column(String(8)) # skip | move
|
||||
moved_start: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
reason: Mapped[str] = mapped_column(String(300), default="")
|
||||
actor: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
@@ -6,7 +6,7 @@ OPNsense API secrets stay off the database entirely.
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Integer, String, UniqueConstraint
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Float, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -17,6 +17,13 @@ def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
DEFAULT_SERVICE_WATCHLIST = (
|
||||
"unbound", "dnsmasq", "dhcpd", "dhcpd6", "kea-dhcp4", "kea-dhcp6", "ntpd", "openvpn", "wireguard", "strongswan",
|
||||
)
|
||||
|
||||
CHANNEL_TYPES = ("ntfy", "gotify", "telegram", "teams", "matrix")
|
||||
|
||||
|
||||
class NotificationSettings(Base):
|
||||
"""Single row, id fixed at 1 — see notification_settings_service.get_settings
|
||||
for the get-or-create helper. The only admin-editable operational config
|
||||
@@ -55,6 +62,30 @@ class NotificationSettings(Base):
|
||||
# Off by default: switching it on for an existing fleet reports every long-standing finding at once.
|
||||
compliance_alerts_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
# Monitoring alerts (1.24). The gateway alert is on by default (a WAN outage is the one thing every
|
||||
# operator wants to hear about); resource thresholds and service checks are off, like compliance alerts:
|
||||
# switching them on for an existing fleet may report long-standing conditions all at once.
|
||||
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
|
||||
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
|
||||
# OPNexus threshold on the measured values (0 = off).
|
||||
gateway_loss_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=20, server_default="20")
|
||||
gateway_delay_threshold_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=500, server_default="500")
|
||||
# Resources (0 = that threshold is off). Load is per CPU thread.
|
||||
disk_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=90, server_default="90")
|
||||
memory_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=95, server_default="95")
|
||||
swap_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=50, server_default="50")
|
||||
load_per_cpu_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=2.0, server_default="2.0")
|
||||
# Debouncing: consecutive 60 s resource samples that must agree before a problem opens or closes.
|
||||
alert_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=3, server_default="3")
|
||||
resource_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5")
|
||||
# OPNsense service names that are reported when they stop (the API cannot tell a crash from a stop on purpose).
|
||||
# NULL = the default list (an old row never got one).
|
||||
service_watchlist: Mapped[list | None] = mapped_column(JSON, nullable=True, default=lambda: list(DEFAULT_SERVICE_WATCHLIST))
|
||||
|
||||
poll_interval_seconds: Mapped[int] = mapped_column(Integer, default=300)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
|
||||
@@ -85,3 +116,21 @@ class NotificationProblem(Base):
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
last_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
resolved_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class NotificationChannel(Base):
|
||||
"""An additional delivery channel (ntfy, Gotify, Telegram, Microsoft Teams, Matrix) next to the SMTP
|
||||
and generic-webhook settings above. Several channels of one type are allowed. ``config`` holds only
|
||||
non-secret parameters (server URL, topic, chat id, room id); the credential — token, or the whole
|
||||
webhook URL for Teams — lives in notification_secret_store under ``channel_<id>``, never here."""
|
||||
|
||||
__tablename__ = "notification_channels"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||||
channel_type: Mapped[str] = mapped_column(String(16))
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
secret_set: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Firmware rollouts in waves (1.25): plans (what, in which order, when) and runs (what happened).
|
||||
|
||||
A plan's waves point at device groups ("rings": canary -> pilot -> prod). A run freezes the plan
|
||||
(``plan_snapshot``) and the resolved hosts (``RolloutRunHost``) when it starts, so later edits of
|
||||
the plan or of a group never change a run that is under way. The run never fires firmware actions
|
||||
itself: every host step is an ordinary ScheduledMaintenance row (origin="rollout",
|
||||
auto_verify=True) linked from ``RolloutRunHost.scheduled_maintenance_id`` — see
|
||||
services/rollout_service.py.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import (JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Table, Text,
|
||||
UniqueConstraint)
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
rollout_wave_groups = Table(
|
||||
"rollout_wave_groups", Base.metadata,
|
||||
Column("wave_id", PGUUID(as_uuid=True), ForeignKey("rollout_plan_waves.id", ondelete="CASCADE"), primary_key=True),
|
||||
Column("group_id", PGUUID(as_uuid=True), ForeignKey("device_groups.id", ondelete="CASCADE"), primary_key=True),
|
||||
)
|
||||
|
||||
|
||||
class RolloutPlan(Base):
|
||||
__tablename__ = "rollout_plans"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(120), unique=True)
|
||||
description: Mapped[str] = mapped_column(Text, default="")
|
||||
action: Mapped[str] = mapped_column(String(16)) # update | major_update
|
||||
window_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
# Start a run by itself at every occurrence of the window. Only for "update" and only while the plan
|
||||
# is activated AND unchanged since (activated_fingerprint).
|
||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
|
||||
auto_start_error: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
created_by: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
|
||||
waves: Mapped[list["RolloutPlanWave"]] = relationship(
|
||||
"RolloutPlanWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by="RolloutPlanWave.position")
|
||||
|
||||
|
||||
class RolloutPlanWave(Base):
|
||||
__tablename__ = "rollout_plan_waves"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
plan_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="CASCADE"), index=True)
|
||||
position: Mapped[int] = mapped_column(Integer)
|
||||
label: Mapped[str] = mapped_column(String(60))
|
||||
soak_minutes: Mapped[int] = mapped_column(Integer, default=30)
|
||||
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Wait for a human "release" before this wave starts (not used for the first wave: starting is the release).
|
||||
require_approval: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
groups: Mapped[list["DeviceGroup"]] = relationship("DeviceGroup", secondary=rollout_wave_groups, lazy="selectin") # noqa: F821
|
||||
|
||||
|
||||
class RolloutRun(Base):
|
||||
__tablename__ = "rollout_runs"
|
||||
__table_args__ = (UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
plan_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
plan_name: Mapped[str] = mapped_column(String(120))
|
||||
plan_snapshot: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
action: Mapped[str] = mapped_column(String(16))
|
||||
# waiting_window | preflight | wave_running | soaking | gate | awaiting_approval | resuming | paused
|
||||
# | succeeded | cancelled
|
||||
status: Mapped[str] = mapped_column(String(20), index=True)
|
||||
paused_from: Mapped[str | None] = mapped_column(String(20), nullable=True)
|
||||
current_wave: Mapped[int] = mapped_column(Integer, default=0)
|
||||
target_version: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_by: Mapped[str] = mapped_column(String(150))
|
||||
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
|
||||
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
|
||||
revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
progress: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
step_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
waves: Mapped[list["RolloutRunWave"]] = relationship(
|
||||
"RolloutRunWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin", order_by="RolloutRunWave.index")
|
||||
hosts: Mapped[list["RolloutRunHost"]] = relationship(
|
||||
"RolloutRunHost", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by=lambda: [RolloutRunHost.wave_index, RolloutRunHost.order])
|
||||
|
||||
|
||||
class RolloutRunWave(Base):
|
||||
__tablename__ = "rollout_run_waves"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
|
||||
index: Mapped[int] = mapped_column(Integer)
|
||||
label: Mapped[str] = mapped_column(String(60))
|
||||
# pending | awaiting_approval | preflight | running | soaking | gate | passed | skipped | cancelled
|
||||
status: Mapped[str] = mapped_column(String(20), default="pending")
|
||||
baseline: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
gate_result: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
soak_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
gate_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
approved_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class RolloutRunHost(Base):
|
||||
__tablename__ = "rollout_run_hosts"
|
||||
__table_args__ = (UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
|
||||
wave_index: Mapped[int] = mapped_column(Integer)
|
||||
host_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
host_name: Mapped[str] = mapped_column(String(120))
|
||||
cluster_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
||||
order: Mapped[int] = mapped_column(Integer, default=0)
|
||||
# queued | running | verifying | succeeded | skipped | failed | blocked | cancelled
|
||||
status: Mapped[str] = mapped_column(String(16), default="queued")
|
||||
scheduled_maintenance_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("scheduled_maintenance.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
version_before: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
version_after: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
# {"code": ..., "params": {...}} — translated in the UI (rollouts.hostNote.*)
|
||||
detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
@@ -42,7 +42,12 @@ class ScheduledMaintenance(Base):
|
||||
host_operation_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
# Opt-in: hand the traffic over to the CARP partner first, run the action, then switch back.
|
||||
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Only while status == "running" with a handover: handover_wait | job_wait | leave | leave_wait.
|
||||
# 1.25: check the result automatically (job_wait -> verify) and close the operation, instead of
|
||||
# leaving it "submitted" for a human. Every rollout step sets it; off for existing plans.
|
||||
auto_verify: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# manual | rollout (created by a rollout run, see models/rollout.py RolloutRunHost.scheduled_maintenance_id)
|
||||
origin: Mapped[str] = mapped_column(String(16), nullable=False, default="manual", server_default="manual")
|
||||
# Only while status == "running" with a handover or auto_verify: handover_wait | job_wait | verify | leave | leave_wait.
|
||||
# State lives here, not in memory, so a backend restart in the middle of a run resumes cleanly.
|
||||
phase: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
phase_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
"""The three "take over what exists on the firewall" endpoints that aliases, port forwards, 1:1 NAT and
|
||||
outbound NAT share (services/object_adopt_service.py has the logic and the evidence):
|
||||
|
||||
GET <prefix>/unmanaged?host_id=… candidates found live on the host, with reasons for the refused ones
|
||||
POST <prefix>/adopt take over one object {host_id, uuid}; returns the new central object
|
||||
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
|
||||
|
||||
Like rule adoption this only creates central records; nothing is written to the firewall."""
|
||||
import uuid
|
||||
from typing import Callable
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.adopt import AdoptAllRequest, AdoptAllResult, AdoptObjectRequest, UnmanagedObject
|
||||
from app.services import object_adopt_service
|
||||
from app.services.adopt_common import LINK_ERRORS, link_api_error
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
|
||||
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
return host
|
||||
|
||||
|
||||
def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Callable, to_read: Callable, read_model) -> None:
|
||||
@router.get("/unmanaged", response_model=list[UnmanagedObject])
|
||||
async def list_unmanaged(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
host = await _host(db, host_id)
|
||||
try:
|
||||
return await object_adopt_service.list_unmanaged(db, kind, host)
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
|
||||
@router.post("/adopt", response_model=read_model, status_code=201)
|
||||
async def adopt(payload: AdoptObjectRequest, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
created = await object_adopt_service.adopt(db, kind, host, payload.uuid, link_to=payload.link_to)
|
||||
except object_adopt_service.NotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
|
||||
except object_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
# The new object sits in the session without its relationships loaded; db.get() would return it as-is.
|
||||
created = (await db.execute(
|
||||
select(model).where(model.id == created.id).options(*load_options())
|
||||
.execution_options(populate_existing=True)
|
||||
)).scalar_one()
|
||||
await hydrate_hierarchy_targets(db, [created])
|
||||
return to_read(created, created.targets, created.deployments)
|
||||
|
||||
@router.post("/adopt-all", response_model=AdoptAllResult)
|
||||
async def adopt_all(payload: AdoptAllRequest, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
result = await object_adopt_service.adopt_all(db, kind, host, link_existing=payload.link_existing)
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
return result
|
||||
@@ -17,6 +17,7 @@ from app.schemas.alias import DeploymentResult, AliasCreate, AliasRead, AliasEdi
|
||||
from app.services.alias_deploy_service import confirm_alias, deploy_alias, undeploy_alias, revert_alias_deployment
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/aliases", tags=["aliases"])
|
||||
|
||||
@@ -70,6 +71,9 @@ def _to_read(alias: FirewallAlias, hosts: list[OpnsenseHost], deployments: list[
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="alias", model=FirewallAlias, load_options=_load_options, to_read=_to_read, read_model=AliasRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[AliasRead])
|
||||
async def list_aliases(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(FirewallAlias).options(*_load_options()))
|
||||
|
||||
@@ -15,7 +15,36 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
||||
|
||||
# Plain name: used on HTTP-only (lab) deployments and accepted for a while on
|
||||
# HTTPS deployments so that sessions issued before 1.27 survive the upgrade.
|
||||
COOKIE_NAME = "opnexus-session"
|
||||
# HTTPS deployments: the __Host- prefix makes the browser refuse any cookie of
|
||||
# this name that is not Secure, has a Domain attribute or a Path other than
|
||||
# "/". A page on another port of the same host (the GUI gateway slots, any
|
||||
# other service on that machine) therefore cannot toss a narrower-path cookie
|
||||
# that shadows the real session, and cannot overwrite it from script (HttpOnly).
|
||||
SECURE_COOKIE_NAME = "__Host-opnexus-session"
|
||||
|
||||
|
||||
def session_cookie_name() -> str:
|
||||
return SECURE_COOKIE_NAME if settings.cookie_secure else COOKIE_NAME
|
||||
|
||||
|
||||
def session_token(request: Request) -> str:
|
||||
"""The session token the browser sent. On HTTPS the __Host- cookie always
|
||||
wins; the plain name is only a migration fallback when the browser has no
|
||||
__Host- cookie yet (it is replaced on the next /api/auth/me)."""
|
||||
if settings.cookie_secure:
|
||||
return request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME, "")
|
||||
return request.cookies.get(COOKIE_NAME, "")
|
||||
|
||||
|
||||
def uses_legacy_cookie(request: Request) -> bool:
|
||||
return settings.cookie_secure and not request.cookies.get(SECURE_COOKIE_NAME) and bool(request.cookies.get(COOKIE_NAME))
|
||||
|
||||
|
||||
def has_session_cookie(request: Request) -> bool:
|
||||
return bool(request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME))
|
||||
|
||||
|
||||
def client_ip(request: Request) -> str:
|
||||
@@ -32,7 +61,7 @@ def client_ip(request: Request) -> str:
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, max_age: int | None = None) -> None:
|
||||
response.set_cookie(
|
||||
key=COOKIE_NAME,
|
||||
key=session_cookie_name(),
|
||||
value=token,
|
||||
max_age=max_age or settings.session_lifetime,
|
||||
httponly=True,
|
||||
@@ -40,10 +69,15 @@ def _set_session_cookie(response: Response, token: str, max_age: int | None = No
|
||||
secure=settings.cookie_secure,
|
||||
path="/",
|
||||
)
|
||||
if settings.cookie_secure:
|
||||
# Migration: a pre-1.27 plain-name cookie must not linger next to the new one.
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||
|
||||
|
||||
def _clear_session_cookie(response: Response) -> None:
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/")
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=settings.cookie_secure, httponly=True, samesite="lax")
|
||||
if settings.cookie_secure:
|
||||
response.delete_cookie(key=SECURE_COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||
|
||||
|
||||
def _check_account_budget(user: AdminUser, action: str) -> None:
|
||||
@@ -71,7 +105,7 @@ async def resolve_user(request: Request, db: AsyncSession) -> AdminUser:
|
||||
user = await db.get(AdminUser, resolved.id)
|
||||
if user is not None:
|
||||
return user
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
user = await auth_service.resolve_session(db, token)
|
||||
if user is None:
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.NICHT_AUTHENTIFIZIERT)
|
||||
@@ -229,7 +263,7 @@ async def totp_enable(
|
||||
user.totp_last_step = step
|
||||
user.totp_recovery_hashes = hashes
|
||||
# Other sessions were established without the second factor — end them.
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
|
||||
await db.commit()
|
||||
logger.info("Two-factor enabled for '%s'", user.username)
|
||||
return {"enabled": True, "recovery_codes": plain}
|
||||
@@ -254,7 +288,7 @@ async def totp_disable(
|
||||
user.totp_enabled = False
|
||||
user.totp_recovery_hashes = None
|
||||
user.totp_last_step = 0
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
|
||||
await db.commit()
|
||||
logger.info("Two-factor disabled for '%s'", user.username)
|
||||
return {"enabled": False}
|
||||
@@ -262,14 +296,14 @@ async def totp_disable(
|
||||
|
||||
@router.post("/logout")
|
||||
async def logout(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
await auth_service.revoke_session(db, token)
|
||||
_clear_session_cookie(response)
|
||||
return {"authenticated": False}
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
async def me(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async def me(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||
raw = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
|
||||
if raw is not None:
|
||||
resolved = await api_token_service.resolve(db, raw, client_ip(request))
|
||||
@@ -278,10 +312,14 @@ async def me(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
user, api_token = resolved
|
||||
return {"authenticated": True, "username": user.username, "role": user.role,
|
||||
"via": "token", "scope": api_token.scope, "token_name": api_token.name}
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
user = await auth_service.resolve_session(db, token)
|
||||
if user is None:
|
||||
return {"authenticated": False}
|
||||
if uses_legacy_cookie(request):
|
||||
# Soft migration (1.27): the same server-side session moves to the
|
||||
# __Host- cookie and the plain-name cookie is deleted — nobody is logged out.
|
||||
_set_session_cookie(response, token)
|
||||
return {"authenticated": True, "username": user.username, "role": user.role}
|
||||
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@ locks are deliberately NOT included here (no fleet-wide list endpoint
|
||||
exists for those, only per-host, and it's a rarely-triggered flow) —
|
||||
a documented v1 gap, not an oversight.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -46,7 +48,10 @@ from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.wireguard_tunnel import WireguardTunnel
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.monitor_checks import STALE_AFTER
|
||||
|
||||
router = APIRouter(prefix="/dashboard", tags=["dashboard"])
|
||||
|
||||
@@ -202,14 +207,34 @@ async def collect_pending_actions(db: AsyncSession) -> list[PendingAction]:
|
||||
return pending_actions
|
||||
|
||||
|
||||
async def _gateways_by_host(db: AsyncSession, hosts) -> dict:
|
||||
"""Gateway state from each host's newest resource sample; hosts without a fresh sample are left out."""
|
||||
result: dict = {}
|
||||
now = datetime.now(timezone.utc)
|
||||
for host in hosts:
|
||||
row = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json())
|
||||
.where(HostResourceSample.host_id == host.id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).first()
|
||||
if row is not None and row[1] is not None and now - row[0] <= STALE_AFTER:
|
||||
result[host.id] = gateway_health_service.from_compact(row[1])
|
||||
return result
|
||||
|
||||
|
||||
@router.get("", response_model=DashboardSummary)
|
||||
async def summary(db: AsyncSession = Depends(get_db)):
|
||||
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
||||
statuses = await get_statuses(hosts)
|
||||
gateways = await _gateways_by_host(db, hosts)
|
||||
host_summaries = [
|
||||
DashboardHost(
|
||||
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
|
||||
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
|
||||
gateways=gateways.get(host.id),
|
||||
)
|
||||
for host in hosts
|
||||
]
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.gateway import GatewayHealth
|
||||
from app.services import gateway_health_service
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
|
||||
|
||||
|
||||
@router.get("", response_model=list[GatewayHealth])
|
||||
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
"""Live gateway state (loss, delay, deviation, OPNsense' verdict) of one firewall."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
return await gateway_health_service.get_gateway_health(host)
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
@@ -99,6 +99,9 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
|
||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
|
||||
from app.models.rollout import rollout_wave_groups
|
||||
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
|
||||
await db.delete(group)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
"""Firewall GUI access through the gui-gateway container (1.27).
|
||||
|
||||
Public API (session cookie, admin only, never API tokens):
|
||||
* POST /api/hosts/{id}/gui-ticket — one-time ticket + the slot URL the browser opens (noopener).
|
||||
* GET/PUT /api/settings/gui-access — switch, idle/absolute timeout, number of slots.
|
||||
|
||||
Internal API for the gateway only (NOT under /api, so the proxy never forwards it; shared secret
|
||||
GUI_GATEWAY_TOKEN in the X-Onx-Gateway-Token header, compared in constant time):
|
||||
* POST /internal/gui/redeem, /internal/gui/revalidate, /internal/gui/end.
|
||||
The answers never contain firewall credentials: only the firewall origin, its pinned (public) CA
|
||||
certificate, the overlay proxy address and the timeouts.
|
||||
"""
|
||||
import hashlib
|
||||
import hmac
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Request, status
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import SECURE_COOKIE_NAME, current_user
|
||||
from app.services import api_token_service, gui_ticket_service as svc
|
||||
|
||||
router = APIRouter(tags=["gui-access"])
|
||||
internal_router = APIRouter(prefix="/internal/gui", include_in_schema=False)
|
||||
|
||||
|
||||
def _require_admin_session(request: Request, user: AdminUser) -> None:
|
||||
if getattr(request.state, "api_token", None) is not None or api_token_service.bearer_from_header(
|
||||
request.headers.get("authorization", "")
|
||||
) is not None:
|
||||
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.GUI_SESSION_REQUIRED)
|
||||
if user.role != "admin":
|
||||
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.ADMIN_REQUIRED)
|
||||
|
||||
|
||||
class GuiTicketRead(BaseModel):
|
||||
url: str
|
||||
slot: int
|
||||
expires_in: int
|
||||
|
||||
|
||||
@router.post("/hosts/{host_id}/gui-ticket", response_model=GuiTicketRead)
|
||||
async def create_ticket(host_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db),
|
||||
user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
if not svc.gateway_configured():
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED)
|
||||
session_cookie = request.cookies.get(SECURE_COOKIE_NAME, "")
|
||||
if not settings.cookie_secure or not session_cookie:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_REQUIRES_HTTPS)
|
||||
row = await svc.get_settings(db)
|
||||
if not row.enabled:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_ACCESS_DISABLED)
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(status.HTTP_404_NOT_FOUND, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
refusal = svc.host_refusal(host)
|
||||
if refusal:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_HOST_NOT_ELIGIBLE, reason=refusal)
|
||||
try:
|
||||
return await svc.issue_ticket(db, user=user, session_token=session_cookie, host=host,
|
||||
request_host=request.headers.get("host", ""))
|
||||
except svc.GuiAccessError as exc:
|
||||
raise api_error(exc.status, exc.code, **exc.params)
|
||||
|
||||
|
||||
class GuiAccessSettingsRead(BaseModel):
|
||||
enabled: bool
|
||||
idle_timeout_minutes: int
|
||||
absolute_timeout_minutes: int
|
||||
slot_count: int
|
||||
available: bool
|
||||
unavailable_reasons: list[str]
|
||||
slots: list[int]
|
||||
updated_by: str | None
|
||||
updated_at: datetime | None
|
||||
|
||||
|
||||
class GuiAccessSettingsUpdate(BaseModel):
|
||||
enabled: bool
|
||||
idle_timeout_minutes: int = Field(ge=1, le=240)
|
||||
absolute_timeout_minutes: int = Field(ge=5, le=720)
|
||||
slot_count: int = Field(ge=1, le=svc.MAX_SLOTS)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _idle_within_absolute(self):
|
||||
if self.idle_timeout_minutes > self.absolute_timeout_minutes:
|
||||
raise ValueError("idle_timeout_minutes must not exceed absolute_timeout_minutes")
|
||||
return self
|
||||
|
||||
|
||||
def _settings_read(row) -> dict:
|
||||
reasons = svc.unavailable_reasons()
|
||||
return {"enabled": row.enabled, "idle_timeout_minutes": row.idle_timeout_minutes,
|
||||
"absolute_timeout_minutes": row.absolute_timeout_minutes, "slot_count": row.slot_count,
|
||||
"available": not reasons, "unavailable_reasons": reasons, "slots": svc.active_slots(row),
|
||||
"updated_by": row.updated_by, "updated_at": row.updated_at}
|
||||
|
||||
|
||||
@router.get("/settings/gui-access", response_model=GuiAccessSettingsRead)
|
||||
async def read_settings(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
row = await svc.get_settings(db)
|
||||
await db.commit()
|
||||
return _settings_read(row)
|
||||
|
||||
|
||||
@router.put("/settings/gui-access", response_model=GuiAccessSettingsRead)
|
||||
async def update_settings(payload: GuiAccessSettingsUpdate, request: Request, db: AsyncSession = Depends(get_db),
|
||||
user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
if payload.enabled and svc.unavailable_reasons():
|
||||
# Never switchable on without HTTPS cookies and a configured gateway.
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED,
|
||||
reasons=", ".join(svc.unavailable_reasons()))
|
||||
row = await svc.get_settings(db)
|
||||
row.enabled = payload.enabled
|
||||
row.idle_timeout_minutes = payload.idle_timeout_minutes
|
||||
row.absolute_timeout_minutes = payload.absolute_timeout_minutes
|
||||
row.slot_count = payload.slot_count
|
||||
row.updated_by = user.username
|
||||
row.updated_at = datetime.now(timezone.utc)
|
||||
if not payload.enabled:
|
||||
await svc.end_all(db, "disabled")
|
||||
await db.commit()
|
||||
return _settings_read(row)
|
||||
|
||||
|
||||
# ── internal: gateway only ────────────────────────────────────────────────
|
||||
|
||||
def _gateway_auth(x_onx_gateway_token: str = Header(default="")) -> None:
|
||||
expected = settings.gui_gateway_token
|
||||
if len(expected) < svc.MIN_GATEWAY_TOKEN_LENGTH:
|
||||
raise HTTPException(status_code=404)
|
||||
# Fixed-length digests: the comparison time does not depend on the presented value.
|
||||
if not hmac.compare_digest(hashlib.sha256(x_onx_gateway_token.encode()).digest(),
|
||||
hashlib.sha256(expected.encode()).digest()):
|
||||
raise HTTPException(status_code=401)
|
||||
|
||||
|
||||
_HEX64 = r"^[0-9a-f]{64}$"
|
||||
|
||||
|
||||
class RedeemRequest(BaseModel):
|
||||
ticket_hash: str = Field(pattern=_HEX64)
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
slot: int = Field(ge=1, le=65535)
|
||||
client_ip: str | None = Field(default=None, max_length=45)
|
||||
|
||||
|
||||
class RevalidateRequest(BaseModel):
|
||||
gui_session_id: uuid.UUID
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
|
||||
|
||||
class EndRequest(RevalidateRequest):
|
||||
reason: Literal["idle", "absolute", "logout", "revoked", "superseded", "shutdown"]
|
||||
|
||||
|
||||
@internal_router.post("/redeem", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_redeem(payload: RedeemRequest, db: AsyncSession = Depends(get_db)):
|
||||
result = await svc.redeem(db, ticket_hash=payload.ticket_hash, session_hash=payload.session_hash,
|
||||
slot=payload.slot, client_ip=payload.client_ip)
|
||||
if not result["ok"]:
|
||||
raise HTTPException(status_code=403, detail={"ok": False, "reason": result["reason"]})
|
||||
return result
|
||||
|
||||
|
||||
@internal_router.post("/revalidate", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_revalidate(payload: RevalidateRequest, db: AsyncSession = Depends(get_db)):
|
||||
return await svc.revalidate(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash)
|
||||
|
||||
|
||||
@internal_router.post("/end", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_end(payload: EndRequest, db: AsyncSession = Depends(get_db)):
|
||||
return {"ok": await svc.end(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
|
||||
reason=payload.reason)}
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Recurring maintenance windows and their exceptions (1.25).
|
||||
|
||||
Every write here counts as a rollout action (api_token_service.is_rollout): a window decides WHEN
|
||||
unattended firmware actions may run, so an operator needs the same right as for deploying.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
from datetime import date, datetime, time, timedelta, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
from app.services import audit_service, window_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/maintenance-windows", tags=["maintenance-windows"])
|
||||
|
||||
_TIME = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
|
||||
|
||||
|
||||
class WindowIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
timezone: str = Field(min_length=1, max_length=64)
|
||||
recurrence: Literal["weekly", "monthly"]
|
||||
weekdays: list[int] = Field(default_factory=list, max_length=7)
|
||||
month_mode: Literal["nth_weekday", "day"] | None = None
|
||||
month_nth: int | None = None
|
||||
month_weekday: int | None = None
|
||||
month_day: int | None = None
|
||||
start_time: str
|
||||
duration_minutes: int = Field(ge=15, le=window_service.MAX_DURATION_MINUTES)
|
||||
start_cutoff_minutes: int = Field(default=60, ge=5, le=window_service.MAX_DURATION_MINUTES)
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
class ExceptionIn(BaseModel):
|
||||
occurrence_date: date
|
||||
kind: Literal["skip", "move"]
|
||||
moved_start: datetime | None = None
|
||||
reason: str = Field(default="", max_length=300)
|
||||
|
||||
|
||||
def _invalid(field: str):
|
||||
return api_error(422, ErrorCode.WINDOW_INVALID, field=field)
|
||||
|
||||
|
||||
def _validated(payload: WindowIn) -> dict:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
raise _invalid("name")
|
||||
if not window_service.valid_timezone(payload.timezone):
|
||||
raise _invalid("timezone")
|
||||
match = _TIME.match(payload.start_time)
|
||||
if not match:
|
||||
raise _invalid("start_time")
|
||||
values = {
|
||||
"name": name, "timezone": payload.timezone, "recurrence": payload.recurrence,
|
||||
"weekdays": [], "month_mode": None, "month_nth": None, "month_weekday": None, "month_day": None,
|
||||
"start_time": time(int(match.group(1)), int(match.group(2))),
|
||||
"duration_minutes": payload.duration_minutes,
|
||||
"start_cutoff_minutes": min(payload.start_cutoff_minutes, payload.duration_minutes),
|
||||
"enabled": payload.enabled,
|
||||
}
|
||||
if payload.recurrence == "weekly":
|
||||
days = sorted(set(payload.weekdays))
|
||||
if not days or any(d < 0 or d > 6 for d in days):
|
||||
raise _invalid("weekdays")
|
||||
values["weekdays"] = days
|
||||
elif payload.month_mode == "day":
|
||||
if payload.month_day is None or not 1 <= payload.month_day <= 28:
|
||||
raise _invalid("month_day")
|
||||
values.update(month_mode="day", month_day=payload.month_day)
|
||||
elif payload.month_mode == "nth_weekday":
|
||||
if payload.month_nth not in (1, 2, 3, 4, -1):
|
||||
raise _invalid("month_nth")
|
||||
if payload.month_weekday is None or not 0 <= payload.month_weekday <= 6:
|
||||
raise _invalid("month_weekday")
|
||||
values.update(month_mode="nth_weekday", month_nth=payload.month_nth, month_weekday=payload.month_weekday)
|
||||
else:
|
||||
raise _invalid("month_mode")
|
||||
return values
|
||||
|
||||
|
||||
async def _plans_using(db: AsyncSession, window_id) -> list[dict]:
|
||||
from app.models.rollout import RolloutPlan
|
||||
rows = (await db.scalars(select(RolloutPlan).where(RolloutPlan.window_id == window_id).order_by(RolloutPlan.name))).all()
|
||||
return [{"id": str(p.id), "name": p.name, "auto_start": p.auto_start, "active": p.activated_at is not None} for p in rows]
|
||||
|
||||
|
||||
def read_window(window: MaintenanceWindow, *, now: datetime | None = None, count: int = 5) -> dict:
|
||||
now = now or datetime.now(timezone.utc)
|
||||
return {
|
||||
"id": str(window.id), "name": window.name, "timezone": window.timezone, "recurrence": window.recurrence,
|
||||
"weekdays": window.weekdays or [], "month_mode": window.month_mode, "month_nth": window.month_nth,
|
||||
"month_weekday": window.month_weekday, "month_day": window.month_day,
|
||||
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
|
||||
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
|
||||
"created_by": window.created_by, "created_at": window.created_at,
|
||||
"open_now": window_service.current(window, now) is not None,
|
||||
"next": [o.as_dict() for o in window_service.occurrences(window, now, count, with_skipped=True)] if window.enabled else [],
|
||||
"exceptions": [
|
||||
{"id": str(e.id), "occurrence_date": e.occurrence_date.isoformat(), "kind": e.kind,
|
||||
"moved_start": e.moved_start, "reason": e.reason, "actor": e.actor, "created_at": e.created_at}
|
||||
for e in window.exceptions
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def _window_or_404(db: AsyncSession, window_id: uuid.UUID) -> MaintenanceWindow:
|
||||
window = await db.get(MaintenanceWindow, window_id)
|
||||
if window is None:
|
||||
raise api_error(404, ErrorCode.WINDOW_NOT_FOUND)
|
||||
return window
|
||||
|
||||
|
||||
async def _name_taken(db: AsyncSession, name: str, own_id=None) -> bool:
|
||||
other = await db.scalar(select(MaintenanceWindow.id).where(MaintenanceWindow.name == name))
|
||||
return other is not None and other != own_id
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_windows(db: AsyncSession = Depends(get_db)):
|
||||
rows = (await db.scalars(select(MaintenanceWindow).order_by(MaintenanceWindow.name))).all()
|
||||
result = []
|
||||
for w in rows:
|
||||
item = read_window(w)
|
||||
item["used_by"] = await _plans_using(db, w.id)
|
||||
result.append(item)
|
||||
return result
|
||||
|
||||
|
||||
@router.post("", status_code=201)
|
||||
async def create_window(payload: WindowIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
values = _validated(payload)
|
||||
if await _name_taken(db, values["name"]):
|
||||
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
|
||||
window = MaintenanceWindow(**values, created_by=request.state.user.username, created_at=datetime.now(timezone.utc))
|
||||
window.exceptions = []
|
||||
db.add(window)
|
||||
await db.commit()
|
||||
item = read_window(await _window_or_404(db, window.id))
|
||||
item["used_by"] = []
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{window_id}")
|
||||
async def get_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
item = read_window(await _window_or_404(db, window_id))
|
||||
item["used_by"] = await _plans_using(db, window_id)
|
||||
return item
|
||||
|
||||
|
||||
@router.put("/{window_id}")
|
||||
async def update_window(window_id: uuid.UUID, payload: WindowIn, db: AsyncSession = Depends(get_db)):
|
||||
"""Changing a window changes the fingerprint of every plan that uses it, so an activated
|
||||
automatic start stops until the plan is activated again (see rollout_service.fingerprint)."""
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
values = _validated(payload)
|
||||
if await _name_taken(db, values["name"], window.id):
|
||||
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
|
||||
for key, value in values.items():
|
||||
setattr(window, key, value)
|
||||
await db.commit()
|
||||
item = read_window(await _window_or_404(db, window_id))
|
||||
item["used_by"] = await _plans_using(db, window_id)
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{window_id}", status_code=204)
|
||||
async def delete_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
# A plan without a window means "no time limit": deleting its window must never turn it into one.
|
||||
if await _plans_using(db, window_id):
|
||||
raise api_error(409, ErrorCode.WINDOW_IN_USE)
|
||||
await db.delete(window)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.get("/{window_id}/occurrences")
|
||||
async def window_occurrences(window_id: uuid.UUID, count: int = Query(10, ge=1, le=50), db: AsyncSession = Depends(get_db)):
|
||||
window = await _window_or_404(db, window_id)
|
||||
return [o.as_dict() for o in window_service.occurrences(window, datetime.now(timezone.utc), count, with_skipped=True)]
|
||||
|
||||
|
||||
@router.post("/{window_id}/exceptions", status_code=201)
|
||||
async def add_exception(window_id: uuid.UUID, payload: ExceptionIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
now = datetime.now(timezone.utc)
|
||||
if not window_service.is_occurrence_date(window, payload.occurrence_date):
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
|
||||
# Only future occurrences (or the one open now) can be changed; history stays as it was.
|
||||
if window_service.local_start(window, payload.occurrence_date) + timedelta(minutes=window.duration_minutes) <= now:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
|
||||
moved = None
|
||||
if payload.kind == "move":
|
||||
if payload.moved_start is None:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
|
||||
moved = payload.moved_start if payload.moved_start.tzinfo else payload.moved_start.replace(tzinfo=timezone.utc)
|
||||
if moved <= now:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
|
||||
if any(e.occurrence_date == payload.occurrence_date for e in window.exceptions):
|
||||
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
|
||||
exc = MaintenanceWindowException(
|
||||
window_id=window.id, occurrence_date=payload.occurrence_date, kind=payload.kind, moved_start=moved,
|
||||
reason=payload.reason.strip(), actor=request.state.user.username, created_at=now,
|
||||
)
|
||||
db.add(exc)
|
||||
try:
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
|
||||
await audit_service.persist(
|
||||
kind="maintenance_window", action=f"exception_{payload.kind}", status="saved", entity_id=str(window.id),
|
||||
entity_name=window.name, details={"occurrence_date": payload.occurrence_date.isoformat(),
|
||||
"moved_start": moved.isoformat() if moved else None, "reason": exc.reason},
|
||||
)
|
||||
db.expire_all()
|
||||
return read_window(await _window_or_404(db, window_id))
|
||||
|
||||
|
||||
@router.delete("/{window_id}/exceptions/{exception_id}")
|
||||
async def delete_exception(window_id: uuid.UUID, exception_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
exc = await db.get(MaintenanceWindowException, exception_id)
|
||||
if exc is None or exc.window_id != window.id:
|
||||
raise api_error(404, ErrorCode.WINDOW_EXCEPTION_NOT_FOUND)
|
||||
await db.delete(exc)
|
||||
await db.commit()
|
||||
db.expire_all()
|
||||
return read_window(await _window_or_404(db, window_id))
|
||||
|
||||
@@ -88,6 +88,31 @@ TOOLS = [
|
||||
"path": "/api/search",
|
||||
"query": ("q",),
|
||||
},
|
||||
{
|
||||
"name": "rule_match",
|
||||
"description": "Which firewall rule decides a packet (first packet of a new connection), per firewall, "
|
||||
"with the verdict (pass/block/reject), the deciding rule and an honest confidence "
|
||||
"(sure/probable/unknown plus reason codes). Reads only; nothing is changed.",
|
||||
"inputSchema": {
|
||||
"type": "object", "required": ["src", "dst"],
|
||||
"properties": {
|
||||
"src": {"type": "string", "description": "source IP address"},
|
||||
"dst": {"type": "string", "description": "destination IP address"},
|
||||
"proto": {"type": "string", "description": "tcp, udp, icmp, ipv6-icmp or a protocol number", "default": "tcp"},
|
||||
"dport": {"type": "integer", "minimum": 0, "maximum": 65535},
|
||||
"sport": {"type": "integer", "minimum": 0, "maximum": 65535, "description": "default: any ephemeral port"},
|
||||
"icmptype": {"type": "string", "description": "e.g. echoreq or 8"},
|
||||
"iface": {"type": "string", "description": "ingress interface identifier (e.g. lan); default: guessed per firewall"},
|
||||
"egress": {"type": "string", "description": "also check the outgoing side: auto or an interface identifier"},
|
||||
"host_ids": {"type": "string", "description": "comma separated firewall ids (default: all)"},
|
||||
"group_id": {"type": "string", "description": "device group id instead of host_ids"},
|
||||
},
|
||||
},
|
||||
"path": "/api/rule-analysis/match",
|
||||
"query": ("src", "dst", "proto", "dport", "sport", "icmptype", "iface", "egress", "host_ids", "group_id"),
|
||||
# The per-rule trace is for the UI; the verdict, deciding rule and reasons are what matters here.
|
||||
"drop": ("trace",),
|
||||
},
|
||||
{
|
||||
"name": "api_get",
|
||||
"description": "Read any other OPNexus API endpoint (GET only), e.g. /api/rules, /api/aliases, /api/logs. "
|
||||
|
||||
@@ -1,21 +1,26 @@
|
||||
import uuid
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.models.notification import NotificationProblem
|
||||
from app.models.notification import NotificationChannel, NotificationProblem
|
||||
from app.schemas.notification import (
|
||||
NotificationChannelCreate,
|
||||
NotificationChannelRead,
|
||||
NotificationChannelUpdate,
|
||||
NotificationProblemRead,
|
||||
NotificationSettingsRead,
|
||||
NotificationSettingsUpdate,
|
||||
TestNotificationRequest,
|
||||
)
|
||||
from app.services import notification_delivery, notification_settings_service
|
||||
from app.services import notification_channels, notification_delivery, notification_secret_store, notification_settings_service, url_guard
|
||||
|
||||
router = APIRouter(prefix="/notifications", tags=["notifications"])
|
||||
|
||||
@@ -43,6 +48,11 @@ async def read_settings(db: AsyncSession = Depends(get_db), user: AdminUser = De
|
||||
|
||||
@router.put("/settings", response_model=NotificationSettingsRead)
|
||||
async def update_settings(payload: NotificationSettingsUpdate, db: AsyncSession = Depends(get_db)):
|
||||
if payload.webhook_url:
|
||||
try:
|
||||
payload.webhook_url = url_guard.validate_url(payload.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise api_error(422, ErrorCode.NOTIFICATION_WEBHOOK_URL_INVALID, reason=str(exc)) from None
|
||||
return await notification_settings_service.update_settings(db, payload)
|
||||
|
||||
|
||||
@@ -79,3 +89,103 @@ async def test_notification(payload: TestNotificationRequest, db: AsyncSession =
|
||||
except notification_delivery.NotificationDeliveryError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ── Additional channels (ntfy, Gotify, Telegram, Teams, Matrix) ────────────────
|
||||
|
||||
def _channel_read(row: NotificationChannel, user: AdminUser) -> NotificationChannelRead:
|
||||
read = NotificationChannelRead.model_validate(row)
|
||||
# An ntfy topic or a chat id is as good as a password for whoever knows it: only admins see the settings.
|
||||
return read if user.role == "admin" else read.model_copy(update={"config": {}})
|
||||
|
||||
|
||||
async def _get_channel(db: AsyncSession, channel_id: uuid.UUID) -> NotificationChannel:
|
||||
row = await db.get(NotificationChannel, channel_id)
|
||||
if row is None:
|
||||
raise api_error(404, ErrorCode.NOTIFICATION_CHANNEL_NOT_FOUND)
|
||||
return row
|
||||
|
||||
|
||||
def _invalid(exc: ValueError):
|
||||
return api_error(422, ErrorCode.NOTIFICATION_CHANNEL_INVALID, reason=str(exc))
|
||||
|
||||
|
||||
@router.get("/channels", response_model=list[NotificationChannelRead])
|
||||
async def list_channels(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
rows = await db.scalars(select(NotificationChannel).order_by(NotificationChannel.name))
|
||||
return [_channel_read(r, user) for r in rows.all()]
|
||||
|
||||
|
||||
@router.post("/channels", response_model=NotificationChannelRead, status_code=201)
|
||||
async def create_channel(payload: NotificationChannelCreate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
try:
|
||||
config = notification_channels.clean_config(payload.channel_type, payload.config)
|
||||
secret = notification_channels.clean_secret(payload.channel_type, payload.secret) if payload.secret else None
|
||||
except ValueError as exc:
|
||||
raise _invalid(exc) from None
|
||||
row = NotificationChannel(
|
||||
name=payload.name.strip(), channel_type=payload.channel_type, enabled=payload.enabled, config=config,
|
||||
secret_set=secret is not None,
|
||||
)
|
||||
row.id = uuid.uuid4()
|
||||
if secret is None and notification_channels.missing_requirements(row) == "Zugangsdaten fehlen":
|
||||
raise _invalid(ValueError("Zugangsdaten fehlen"))
|
||||
db.add(row)
|
||||
try:
|
||||
await db.flush()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
|
||||
if secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(row.id), secret)
|
||||
await db.commit()
|
||||
return _channel_read(row, user)
|
||||
|
||||
|
||||
@router.put("/channels/{channel_id}", response_model=NotificationChannelRead)
|
||||
async def update_channel(
|
||||
channel_id: uuid.UUID, payload: NotificationChannelUpdate,
|
||||
db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user),
|
||||
):
|
||||
row = await _get_channel(db, channel_id)
|
||||
try:
|
||||
if payload.config is not None:
|
||||
row.config = notification_channels.clean_config(row.channel_type, payload.config)
|
||||
new_secret = notification_channels.clean_secret(row.channel_type, payload.secret) if payload.secret else None
|
||||
except ValueError as exc:
|
||||
raise _invalid(exc) from None
|
||||
if payload.name is not None:
|
||||
row.name = payload.name.strip()
|
||||
if payload.enabled is not None:
|
||||
row.enabled = payload.enabled
|
||||
if payload.clear_secret:
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
|
||||
row.secret_set = False
|
||||
elif new_secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(row.id), new_secret)
|
||||
row.secret_set = True
|
||||
try:
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
|
||||
return _channel_read(row, user)
|
||||
|
||||
|
||||
@router.delete("/channels/{channel_id}", status_code=204)
|
||||
async def delete_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
row = await _get_channel(db, channel_id)
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
|
||||
await db.delete(row)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.post("/channels/{channel_id}/test")
|
||||
async def test_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
row = await _get_channel(db, channel_id)
|
||||
try:
|
||||
await notification_channels.send_test(row)
|
||||
except notification_delivery.NotificationDeliveryError as exc:
|
||||
# The text names the channel and the HTTP status; it never contains a URL or a token.
|
||||
raise api_error(502, ErrorCode.NOTIFICATION_CHANNEL_TEST_FAILED, reason=str(exc)) from None
|
||||
return {"success": True}
|
||||
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.one_to_one_deploy_service import confirm_one_to_one, deploy_one_to_one, undeploy_one_to_one, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/one-to-one", tags=["one-to-one"])
|
||||
|
||||
@@ -79,6 +80,9 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="one_to_one", model=OneToOneRule, load_options=_load_options, to_read=_to_read, read_model=OneToOneRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[OneToOneRead])
|
||||
async def list_one_to_one_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(OneToOneRule).options(*_load_options()))
|
||||
|
||||
@@ -21,6 +21,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.outbound_nat_deploy_service import confirm_outbound_nat, deploy_outbound_nat, undeploy_outbound_nat, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/outbound-nat", tags=["outbound-nat"])
|
||||
|
||||
@@ -82,6 +83,9 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="outbound_nat", model=OutboundNatRule, load_options=_load_options, to_read=_to_read, read_model=OutboundNatRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[OutboundNatRead])
|
||||
async def list_outbound_nat_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(OutboundNatRule).options(*_load_options()))
|
||||
|
||||
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.port_forward_deploy_service import confirm_port_forward, deploy_port_forward, undeploy_port_forward, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/port-forward", tags=["port-forward"])
|
||||
|
||||
@@ -80,6 +81,9 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="port_forward", model=PortForwardRule, load_options=_load_options, to_read=_to_read, read_model=PortForwardRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[PortForwardRead])
|
||||
async def list_port_forward_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(PortForwardRule).options(*_load_options()))
|
||||
|
||||
@@ -0,0 +1,432 @@
|
||||
"""Firmware rollouts in waves (1.25): plans and runs.
|
||||
|
||||
Roles (enforced in AuthMiddleware, see api_token_service.is_rollout): everybody signed in reads;
|
||||
operators prepare plans; starting/activating a plan and resuming/approving/skipping/aborting a run
|
||||
need the deploy right (admin, or operator while operators may deploy). Pausing is deliberately NOT
|
||||
a rollout action: anybody who may write at all may always brake.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||
|
||||
ACTIVE_RUN = rollout_engine.ACTIVE
|
||||
|
||||
|
||||
class WaveIn(BaseModel):
|
||||
label: str = Field(min_length=1, max_length=60)
|
||||
group_ids: list[uuid.UUID] = Field(min_length=1, max_length=20)
|
||||
soak_minutes: int = Field(default=30, ge=0, le=rollout_service.MAX_SOAK_MINUTES)
|
||||
carp_handover: bool = False
|
||||
require_approval: bool = False
|
||||
|
||||
|
||||
class PlanIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
description: str = Field(default="", max_length=2000)
|
||||
action: Literal["update", "major_update"] = "update"
|
||||
window_id: uuid.UUID | None = None
|
||||
auto_start: bool = False
|
||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||
|
||||
|
||||
class TokenIn(BaseModel):
|
||||
preview_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
async def _plan_or_404(db: AsyncSession, plan_id: uuid.UUID) -> RolloutPlan:
|
||||
plan = await rollout_service.load_plan(db, plan_id)
|
||||
if plan is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_PLAN_NOT_FOUND)
|
||||
return plan
|
||||
|
||||
|
||||
async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None = None) -> dict:
|
||||
resolved = resolved or await rollout_service.resolve(db, plan)
|
||||
active = plan.activated_at is not None
|
||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||
return {
|
||||
**rollout_service.definition(plan),
|
||||
"description": plan.description,
|
||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||
"group_ids": [str(g.id) for g in w.groups], "groups": [g.name for g in w.groups],
|
||||
"host_count": len(resolved["waves"][i]["hosts"]) if i < len(resolved["waves"]) else 0}
|
||||
for i, w in enumerate(plan.waves)],
|
||||
"active": active, "activated_by": plan.activated_by, "activated_at": plan.activated_at,
|
||||
# Activated but the plan, its groups or its window changed since: the automatic start is held back.
|
||||
"fingerprint_ok": (not active) or plan.activated_fingerprint == rollout_service.fingerprint(plan, resolved),
|
||||
"auto_start_error": plan.auto_start_error,
|
||||
"window": resolved["window"], "errors": resolved["errors"], "warnings": resolved["warnings"],
|
||||
"host_count": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
"last_run": {"id": str(last_run.id), "status": last_run.status, "created_at": last_run.created_at} if last_run else None,
|
||||
"created_by": plan.created_by, "created_at": plan.created_at, "updated_at": plan.updated_at,
|
||||
}
|
||||
|
||||
|
||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="name")
|
||||
other = await db.scalar(select(RolloutPlan.id).where(RolloutPlan.name == name))
|
||||
if other is not None and other != plan.id:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_NAME_TAKEN)
|
||||
if payload.window_id is not None and await db.get(MaintenanceWindow, payload.window_id) is None:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="window_id")
|
||||
if any(k not in rollout_health.CHECKS for k in payload.gate_checks):
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="gate_checks")
|
||||
wanted = {g for w in payload.waves for g in w.group_ids}
|
||||
groups = {g.id: g for g in (await db.scalars(select(DeviceGroup).where(DeviceGroup.id.in_(wanted)))).all()}
|
||||
if len(groups) != len(wanted):
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="group_ids")
|
||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||
plan.compliance_mode = payload.compliance_mode
|
||||
plan.waves = [
|
||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||
require_approval=w.require_approval, groups=[groups[g] for g in dict.fromkeys(w.group_ids)])
|
||||
for i, w in enumerate(payload.waves)
|
||||
]
|
||||
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
|
||||
plan.activated_at = plan.activated_by = plan.activated_fingerprint = None
|
||||
plan.auto_start_error = None
|
||||
|
||||
|
||||
@router.get("/plans")
|
||||
async def list_plans(db: AsyncSession = Depends(get_db)):
|
||||
plans = (await db.scalars(select(RolloutPlan).order_by(RolloutPlan.name))).all()
|
||||
return [await read_plan(db, p) for p in plans]
|
||||
|
||||
|
||||
@router.post("/plans", status_code=201)
|
||||
async def create_plan(payload: PlanIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = RolloutPlan(created_by=request.state.user.username, created_at=datetime.now(timezone.utc), waves=[])
|
||||
await _apply(db, plan, payload)
|
||||
db.add(plan)
|
||||
await db.commit()
|
||||
return await read_plan(db, await _plan_or_404(db, plan.id))
|
||||
|
||||
|
||||
@router.get("/plans/{plan_id}")
|
||||
async def get_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.put("/plans/{plan_id}")
|
||||
async def update_plan(plan_id: uuid.UUID, payload: PlanIn, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
was_active = plan.activated_at is not None
|
||||
await _apply(db, plan, payload)
|
||||
await db.commit()
|
||||
if was_active:
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id),
|
||||
entity_name=plan.name, message="Plan geändert: automatischer Start deaktiviert")
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.delete("/plans/{plan_id}", status_code=204)
|
||||
async def delete_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.status.in_(ACTIVE_RUN)).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_PLAN_BUSY)
|
||||
await db.delete(plan)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/preview")
|
||||
async def preview_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
return {**resolved, "preview_token": rollout_service.fingerprint(plan, resolved),
|
||||
"can_start": not resolved["errors"], "plan": await read_plan(db, plan, resolved)}
|
||||
|
||||
|
||||
async def _checked_token(db: AsyncSession, plan: RolloutPlan, token: str) -> tuple[dict, str]:
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
current = rollout_service.fingerprint(plan, resolved)
|
||||
if token != current:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_PREVIEW_STALE)
|
||||
if resolved["errors"]:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_PLAN_INVALID, reason=resolved["errors"][0]["code"])
|
||||
return resolved, current
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/activate")
|
||||
async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if not plan.auto_start:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
|
||||
_resolved, current = await _checked_token(db, plan, payload.preview_token)
|
||||
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
|
||||
entity_name=plan.name, details={"fingerprint": current})
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/deactivate")
|
||||
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
plan.activated_by = plan.activated_at = plan.activated_fingerprint = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
# ── runs ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
class StartIn(TokenIn):
|
||||
confirm_name: str = Field(min_length=1, max_length=120)
|
||||
acknowledge_unattended: Literal[True]
|
||||
|
||||
|
||||
class ReasonIn(BaseModel):
|
||||
reason: str = Field(default="", max_length=500)
|
||||
|
||||
|
||||
class SkipIn(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
reason: str = Field(min_length=3, max_length=500)
|
||||
|
||||
|
||||
class AbortIn(BaseModel):
|
||||
reason: str = Field(min_length=3, max_length=500)
|
||||
confirm_name: str = Field(min_length=1, max_length=120)
|
||||
|
||||
|
||||
def _read_run_summary(run: RolloutRun) -> dict:
|
||||
counts: dict[str, int] = {}
|
||||
for h in run.hosts:
|
||||
counts[h.status] = counts.get(h.status, 0) + 1
|
||||
return {
|
||||
"id": str(run.id), "plan_id": str(run.plan_id) if run.plan_id else None, "plan_name": run.plan_name,
|
||||
"action": run.action, "status": run.status, "paused_from": run.paused_from, "current_wave": run.current_wave,
|
||||
"wave_count": len(run.waves), "current_wave_label": next((w.label for w in run.waves if w.index == run.current_wave), None),
|
||||
"target_version": run.target_version, "pause_reason": run.pause_reason, "pause_detail": run.pause_detail,
|
||||
"started_by": run.started_by, "trigger": run.trigger, "window_occurrence": run.window_occurrence,
|
||||
"created_at": run.created_at, "finished_at": run.finished_at, "step_at": run.step_at, "host_counts": counts,
|
||||
"active": run.status in rollout_engine.ACTIVE,
|
||||
}
|
||||
|
||||
|
||||
async def read_run(db: AsyncSession, run: RolloutRun) -> dict:
|
||||
scheds = {}
|
||||
ids = [h.scheduled_maintenance_id for h in run.hosts if h.scheduled_maintenance_id]
|
||||
if ids:
|
||||
scheds = {s.id: s for s in (await db.scalars(select(ScheduledMaintenance).where(ScheduledMaintenance.id.in_(ids)))).all()}
|
||||
soak_until = next((w.soak_until for w in run.waves if w.index == run.current_wave), None)
|
||||
return {
|
||||
**_read_run_summary(run),
|
||||
"soak_until": soak_until, "progress": {k: v for k, v in (run.progress or {}).items()
|
||||
if k in ("gate_recheck_at", "cluster_recheck_at", "blocked_since", "resume_gate",
|
||||
"cluster_gate", "preflight_started", "resume_recheck_at")},
|
||||
"snapshot": run.plan_snapshot,
|
||||
"waves": [{"index": w.index, "label": w.label, "status": w.status, "gate_result": w.gate_result,
|
||||
"gate_attempts": w.gate_attempts, "soak_until": w.soak_until, "approved_by": w.approved_by,
|
||||
"started_at": w.started_at, "finished_at": w.finished_at,
|
||||
"spec": run.plan_snapshot["waves"][w.index] if w.index < len(run.plan_snapshot.get("waves", [])) else None}
|
||||
for w in run.waves],
|
||||
"hosts": [{"id": str(h.id), "host_id": str(h.host_id) if h.host_id else None, "host_name": h.host_name,
|
||||
"cluster_name": h.cluster_name, "wave_index": h.wave_index, "order": h.order, "status": h.status,
|
||||
"version_before": h.version_before, "version_after": h.version_after, "detail": h.detail,
|
||||
"started_at": h.started_at, "finished_at": h.finished_at,
|
||||
"schedule": ({"id": str(s.id), "status": s.status, "phase": s.phase, "carp_handover": s.carp_handover,
|
||||
"host_operation_id": str(s.host_operation_id) if s.host_operation_id else None}
|
||||
if (s := scheds.get(h.scheduled_maintenance_id)) else None)}
|
||||
for h in sorted(run.hosts, key=lambda h: (h.wave_index, h.order, h.host_name))],
|
||||
}
|
||||
|
||||
|
||||
async def _cancel_if_pending(db: AsyncSession, sched_id) -> bool:
|
||||
"""Cancel a host's schedule only if it has not fired. Locked: the poll loop may be firing it right now
|
||||
(fire_due holds the row), and a blind write would turn a running action into "cancelled"."""
|
||||
if sched_id is None:
|
||||
return False
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sched_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if sched is not None and sched.status == "pending":
|
||||
sched.status = "cancelled"
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _run_or_404(db: AsyncSession, run_id: uuid.UUID, *, lock: bool = False) -> RolloutRun:
|
||||
stmt = select(RolloutRun).where(RolloutRun.id == run_id).execution_options(populate_existing=True)
|
||||
if lock:
|
||||
stmt = stmt.with_for_update()
|
||||
run = await db.scalar(stmt)
|
||||
if run is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_RUN_NOT_FOUND)
|
||||
return run
|
||||
|
||||
|
||||
def _not_now(run: RolloutRun):
|
||||
return api_error(409, ErrorCode.ROLLOUT_ACTION_NOT_ALLOWED, status=run.status)
|
||||
|
||||
|
||||
async def _done(db: AsyncSession, run: RolloutRun, action: str, status: str, request: Request, **details) -> dict:
|
||||
run.revision += 1
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id), entity_name=run.plan_name,
|
||||
details={"wave": run.current_wave, **details})
|
||||
return await read_run(db, await _run_or_404(db, run.id))
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/start", status_code=201)
|
||||
async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if payload.confirm_name != plan.name:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
|
||||
resolved, _token = await _checked_token(db, plan, payload.preview_token)
|
||||
if (other := await rollout_engine.active_run(db)) is not None:
|
||||
# One run at a time: two runs could otherwise touch the same cluster at once.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
|
||||
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username, trigger="manual")
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
|
||||
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
"waves": [w["label"] for w in resolved["waves"]]})
|
||||
return await read_run(db, await _run_or_404(db, run.id))
|
||||
|
||||
|
||||
@router.get("/runs")
|
||||
async def list_runs(db: AsyncSession = Depends(get_db)):
|
||||
runs = (await db.scalars(select(RolloutRun).order_by(RolloutRun.created_at.desc()).limit(50))).all()
|
||||
return [_read_run_summary(r) for r in runs]
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def rollout_status(db: AsyncSession = Depends(get_db)):
|
||||
"""For /updates and the host pages: the active run and the hosts it still owns."""
|
||||
run = await rollout_engine.active_run(db)
|
||||
if run is None:
|
||||
return {"active": None, "host_ids": [], "current_host_ids": []}
|
||||
return {
|
||||
"active": _read_run_summary(run),
|
||||
"host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.status in ("queued", "blocked", "running", "verifying", "failed")],
|
||||
"current_host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.wave_index == run.current_wave
|
||||
and h.status in ("queued", "blocked", "running", "verifying", "failed")],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}")
|
||||
async def get_run(run_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
return await read_run(db, await _run_or_404(db, run_id))
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/pause")
|
||||
async def pause_run(run_id: uuid.UUID, payload: ReasonIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE or run.status == "paused":
|
||||
raise _not_now(run)
|
||||
# A brake that lets the next host fire anyway is no brake: a step not fired yet goes back to the queue.
|
||||
requeued = [h.host_name for h in run.hosts if await rollout_engine.requeue_if_pending(db, h)]
|
||||
rollout_engine.pause(run, "manual", by=request.state.user.username, text=payload.reason.strip()[:300])
|
||||
return await _done(db, run, "pause", "paused", request, reason="manual", text=payload.reason.strip()[:300], requeued=requeued)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/resume")
|
||||
async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status != "paused":
|
||||
raise _not_now(run)
|
||||
failed = [h.host_name for h in run.hosts if h.status == "failed"]
|
||||
if failed:
|
||||
# A failed host is never retried by itself: check it, then skip it explicitly.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
|
||||
previous = run.started_by
|
||||
# Whoever resumes takes over the responsibility (and the permission check) from here on.
|
||||
run.started_by = request.state.user.username
|
||||
run.status = "resuming"
|
||||
run.pause_reason = run.pause_detail = None
|
||||
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
|
||||
return await _done(db, run, "resume", "resuming", request, previous_owner=previous)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/approve")
|
||||
async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status != "awaiting_approval":
|
||||
raise _not_now(run)
|
||||
wave = rollout_engine.wave_row(run, run.current_wave)
|
||||
wave.approved_by, wave.status = request.state.user.username, "preflight"
|
||||
run.started_by = request.state.user.username
|
||||
run.status = "preflight"
|
||||
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
|
||||
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/skip-host")
|
||||
async def skip_host(run_id: uuid.UUID, payload: SkipIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE:
|
||||
raise _not_now(run)
|
||||
host = next((h for h in run.hosts if h.host_id == payload.host_id), None)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_HOST_NOT_IN_RUN)
|
||||
if host.status in ("running", "verifying"):
|
||||
if not await _cancel_if_pending(db, host.scheduled_maintenance_id):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
|
||||
# not fired yet: nothing happened on the firewall
|
||||
elif host.status not in ("queued", "blocked", "failed"):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
|
||||
host.status = "skipped"
|
||||
host.detail = {"code": "skipped_by", "params": {"user": request.state.user.username, "reason": payload.reason.strip()[:300]}}
|
||||
host.finished_at = datetime.now(timezone.utc)
|
||||
return await _done(db, run, "skip_host", "skipped", request, host=host.host_name, reason=payload.reason.strip()[:300])
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/abort")
|
||||
async def abort_run(run_id: uuid.UUID, payload: AbortIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE:
|
||||
raise _not_now(run)
|
||||
if payload.confirm_name != run.plan_name:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
|
||||
now = datetime.now(timezone.utc)
|
||||
still_running = []
|
||||
for host in run.hosts:
|
||||
if host.status in ("running", "verifying") and host.scheduled_maintenance_id:
|
||||
if await _cancel_if_pending(db, host.scheduled_maintenance_id):
|
||||
host.status, host.finished_at = "cancelled", now
|
||||
else:
|
||||
still_running.append(host.host_name) # an action already on the firewall is finished in a controlled way
|
||||
elif host.status in ("queued", "blocked"):
|
||||
host.status, host.finished_at = "cancelled", now
|
||||
for wave in run.waves:
|
||||
if wave.status not in ("passed", "skipped"):
|
||||
wave.status = "cancelled"
|
||||
run.status, run.finished_at = "cancelled", now
|
||||
run.pause_reason, run.pause_detail = "aborted", {"by": request.state.user.username, "reason": payload.reason.strip()[:300],
|
||||
"still_running": still_running}
|
||||
return await _done(db, run, "abort", "cancelled", request, reason=payload.reason.strip()[:300], still_running=still_running)
|
||||
@@ -0,0 +1,91 @@
|
||||
"""Rule analysis: which rule decides a packet (read-only, GET so ``read`` tokens and MCP can use it).
|
||||
|
||||
Nothing is stored and nothing is written to a firewall; every request reads the rules, the
|
||||
loaded pf ruleset and the interfaces (cached for 60 s per firewall, ``refresh=true`` skips
|
||||
the cache) and evaluates them with services/rule_analysis/engine.py.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.services.rule_analysis import service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["rule-analysis"])
|
||||
_PARALLEL = 5
|
||||
|
||||
|
||||
def _query(src: str, dst: str, proto: str, sport: str | None, dport: str | None, icmptype: str | None,
|
||||
iface: str | None, direction: str, egress: str | None):
|
||||
return service.build_query(src=src, dst=dst, proto=proto, sport=sport, dport=dport, icmptype=icmptype,
|
||||
iface=iface, direction=direction, egress=egress)
|
||||
|
||||
|
||||
async def _run(host: OpnsenseHost, q, refresh: bool) -> dict:
|
||||
head = {"host_id": str(host.id), "host_name": host.name, "cluster_name": host.cluster_name}
|
||||
try:
|
||||
return {**head, **await service.analyze_host(host, q, refresh=refresh), "error": None}
|
||||
except HTTPException as exc:
|
||||
return {**head, "error": exc.detail}
|
||||
except Exception as exc: # noqa: BLE001 - one unreachable firewall must not fail the fleet answer
|
||||
logger.info("rule analysis failed for %s: %s", host.name, exc)
|
||||
return {**head, "error": {"code": ErrorCode.ANALYSE_FIREWALL_NICHT_LESBAR.value, "params": {}}}
|
||||
|
||||
|
||||
@router.get("/hosts/{host_id}/rule-analysis/match")
|
||||
async def host_match(
|
||||
host_id: uuid.UUID, src: str, dst: str, proto: str = "tcp", sport: str | None = None, dport: str | None = None,
|
||||
icmptype: str | None = None, iface: str | None = None, direction: str = "in", egress: str | None = None,
|
||||
refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
q = _query(src, dst, proto, sport, dport, icmptype, iface, direction, egress)
|
||||
result = await _run(host, q, refresh)
|
||||
error = result.get("error")
|
||||
if error:
|
||||
status = 422 if error.get("code") == ErrorCode.ANALYSE_INTERFACE_UNBEKANNT.value else 502
|
||||
raise HTTPException(status_code=status, detail=error)
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/rule-analysis/match")
|
||||
async def fleet_match(
|
||||
src: str, dst: str, proto: str = "tcp", sport: str | None = None, dport: str | None = None,
|
||||
icmptype: str | None = None, iface: str | None = None, direction: str = "in", egress: str | None = None,
|
||||
host_ids: str | None = Query(None, description="comma separated host ids; default: all enabled hosts"),
|
||||
group_id: uuid.UUID | None = None, refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
q = _query(src, dst, proto, sport, dport, icmptype, iface, direction, egress)
|
||||
if group_id is not None:
|
||||
group = await db.scalar(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)).where(DeviceGroup.id == group_id))
|
||||
if group is None:
|
||||
raise api_error(404, ErrorCode.GRUPPE_NICHT_GEFUNDEN)
|
||||
hosts = sorted(group.hosts, key=lambda h: h.name)
|
||||
elif host_ids:
|
||||
try:
|
||||
ids = [uuid.UUID(part.strip()) for part in host_ids.split(",") if part.strip()]
|
||||
except ValueError:
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="host_ids", value=host_ids[:64]) from None
|
||||
hosts = list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)).order_by(OpnsenseHost.name))).all())
|
||||
if len(hosts) != len(set(ids)):
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
else:
|
||||
hosts = list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)).order_by(OpnsenseHost.name))).all())
|
||||
gate = asyncio.Semaphore(_PARALLEL)
|
||||
|
||||
async def one(host):
|
||||
async with gate:
|
||||
return await _run(host, q, refresh)
|
||||
|
||||
return list(await asyncio.gather(*(one(h) for h in hosts)))
|
||||
@@ -18,6 +18,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.rule_deploy_service import confirm_rule, deploy_rule, undeploy_rule, revert_deployment
|
||||
from app.services import rule_adopt_service
|
||||
from app.services.adopt_common import LINK_ERRORS, link_api_error
|
||||
from app.schemas.rule import AdoptRequest, UnmanagedRule
|
||||
|
||||
router = APIRouter(prefix="/rules", tags=["rules"])
|
||||
@@ -107,11 +108,13 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid)
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
|
||||
except rule_adopt_service.RuleNotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.REGEL_NICHT_GEFUNDEN) from exc
|
||||
except rule_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
|
||||
@@ -24,6 +24,8 @@ class Schedule(BaseModel):
|
||||
acknowledge_unattended: Literal[True]
|
||||
# Hand the traffic over to the CARP partner first and switch back afterwards (cluster nodes only).
|
||||
carp_handover: bool = False
|
||||
# Check the result automatically afterwards and close the operation (no manual "Wartung abschließen").
|
||||
auto_verify: bool = False
|
||||
|
||||
|
||||
def _read(sched: ScheduledMaintenance) -> dict:
|
||||
@@ -32,6 +34,8 @@ def _read(sched: ScheduledMaintenance) -> dict:
|
||||
"status": sched.status, "actor": sched.actor, "created_at": sched.created_at,
|
||||
"result": sched.result, "host_operation_id": sched.host_operation_id,
|
||||
"carp_handover": sched.carp_handover, "phase": sched.phase,
|
||||
"auto_verify": sched.auto_verify, "origin": sched.origin,
|
||||
"rollout_run_id": ((sched.progress or {}).get("rollout") or {}).get("run_id"),
|
||||
}
|
||||
|
||||
|
||||
@@ -80,6 +84,7 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
|
||||
sched = ScheduledMaintenance(
|
||||
host_id=host.id, host_name=host.name, action=payload.action, run_at=run_at,
|
||||
actor=request.state.user.username, created_at=now, carp_handover=payload.carp_handover,
|
||||
auto_verify=payload.auto_verify,
|
||||
)
|
||||
db.add(sched)
|
||||
await db.commit()
|
||||
@@ -90,7 +95,9 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
|
||||
async def cancel_scheduled(host_id: uuid.UUID, schedule_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
await _host_or_404(db, host_id)
|
||||
sched = await db.get(ScheduledMaintenance, schedule_id)
|
||||
# Locked and re-read: the poll loop may be firing this very row right now (it holds the lock meanwhile).
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == schedule_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if not sched or sched.host_id != host_id:
|
||||
raise api_error(404, ErrorCode.GEPLANTE_WARTUNG_NICHT_GEFUNDEN)
|
||||
if sched.status != "pending":
|
||||
|
||||
@@ -7,6 +7,7 @@ from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.services import rollout_engine
|
||||
from app.services import update_check_service, updater_client
|
||||
|
||||
router = APIRouter(prefix="/system", tags=["system"])
|
||||
@@ -99,6 +100,9 @@ async def updater_status(db: AsyncSession = Depends(get_db), user: AdminUser = D
|
||||
"can_update": False, "major_change": False,
|
||||
"job": None, "phases": [], "log": [], "services": {},
|
||||
"rollback_available": False, "rollback_loses_data": False, "rollback_version": None,
|
||||
# A restart in the middle of a firmware rollout is safe (its state is in the database), but the
|
||||
# UI warns: the run pauses its 15-second steps for as long as OPNexus is down.
|
||||
"active_rollouts": await rollout_engine.count_active(db),
|
||||
}
|
||||
try:
|
||||
updater = await updater_client.status()
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
import uuid
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AdoptObjectRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link to this existing central object instead of creating a new one (services/adopt_link_service.py).
|
||||
# Declared before `uuid`: that field's name shadows the module inside the class body.
|
||||
link_to: uuid.UUID | None = None
|
||||
uuid: str = Field(min_length=1, max_length=64) # the object's UUID on the firewall
|
||||
|
||||
|
||||
class AdoptAllRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link entries that have exactly one exact central twin instead of creating a duplicate.
|
||||
link_existing: bool = False
|
||||
|
||||
|
||||
class LinkDiff(BaseModel):
|
||||
field: str # model field ("content", "destination_port", ...) or "variables" (a {{VARIABLE}} has no value on this host)
|
||||
central: str
|
||||
live: str
|
||||
|
||||
|
||||
class LinkTarget(BaseModel):
|
||||
"""An existing central object that a live object could be linked to (exact match) or is similar to."""
|
||||
|
||||
id: uuid.UUID
|
||||
name: str
|
||||
targets: list[str] # names of the hosts it is deployed to
|
||||
diff: list[LinkDiff] = Field(default_factory=list) # empty for an exact match
|
||||
|
||||
|
||||
class UnmanagedObject(BaseModel):
|
||||
"""An alias / NAT rule found on a host that OPNexus does not manage yet (see object_adopt_service.py)."""
|
||||
|
||||
uuid: str
|
||||
title: str
|
||||
enabled: bool
|
||||
adoptable: bool # can be adopted on its own right now
|
||||
# Adoptable in the same "adopt all" run although not on its own (an alias whose referenced aliases come first).
|
||||
bulk_adoptable: bool
|
||||
# Codes such as "unsupported_type" or "no_target"; the UI turns them into text.
|
||||
reasons: list[str]
|
||||
reason_details: dict[str, str] = Field(default_factory=dict) # code -> extra text (alias names, type, ...)
|
||||
# Settings OPNexus does not manage that stay untouched on the firewall (log, pool options, tags, ...).
|
||||
kept: list[str] = Field(default_factory=list)
|
||||
# Facts for display, depend on the object type.
|
||||
info: dict[str, str] = Field(default_factory=dict)
|
||||
# Central objects whose content, resolved for this host, equals the live object exactly (first = preferred).
|
||||
matches: list[LinkTarget] = Field(default_factory=list)
|
||||
# Objects with the same name/description but different content: a hint with the differing fields, never linked.
|
||||
similar: list[LinkTarget] = Field(default_factory=list)
|
||||
linkable: bool = False # can be linked to `matches[0]` (or another match) on its own
|
||||
# What "adopt all" would do: "link" | "new" | "skip" (None only on the adopt-all result).
|
||||
bulk_action: str | None = None
|
||||
|
||||
|
||||
class AdoptedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
id: uuid.UUID
|
||||
linked: bool = False # linked to an existing central object instead of newly created
|
||||
|
||||
|
||||
class SkippedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
reasons: list[str]
|
||||
reason_details: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AdoptAllResult(BaseModel):
|
||||
adopted: list[AdoptedObject]
|
||||
skipped: list[SkippedObject]
|
||||
@@ -3,6 +3,8 @@ from datetime import datetime
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.schemas.gateway import GatewayHealth
|
||||
|
||||
|
||||
class DashboardHost(BaseModel):
|
||||
id: uuid.UUID
|
||||
@@ -10,6 +12,8 @@ class DashboardHost(BaseModel):
|
||||
status: str
|
||||
carp_status: str | None
|
||||
cluster_name: str | None
|
||||
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
|
||||
gateways: list[GatewayHealth] | None = None
|
||||
|
||||
|
||||
class PendingAction(BaseModel):
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
# online: all fine · degraded: OPNsense flags loss and/or latency above the gateway's own limits ·
|
||||
# offline: down · forced_down: switched off by the admin on purpose · unknown: nothing measured
|
||||
# (monitoring disabled) or the verdict is still pending.
|
||||
GatewayState = Literal["online", "degraded", "offline", "forced_down", "unknown"]
|
||||
|
||||
|
||||
class GatewayHealth(BaseModel):
|
||||
name: str
|
||||
address: str | None = None
|
||||
status: str # OPNsense' raw verdict: none | down | force_down | delay | loss | delay+loss | …
|
||||
state: GatewayState
|
||||
loss: float | None = None # percent
|
||||
delay: float | None = None # milliseconds
|
||||
stddev: float | None = None # milliseconds
|
||||
monitor: str | None = None # the monitored address
|
||||
@@ -1,10 +1,12 @@
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST
|
||||
|
||||
SmtpSecurity = Literal["none", "starttls", "ssl"]
|
||||
ProblemType = Literal[
|
||||
@@ -14,7 +16,10 @@ ProblemType = Literal[
|
||||
"backup_stale",
|
||||
"backup_offsite_failed",
|
||||
"compliance_finding",
|
||||
"gateway_down", "resource_threshold", "service_down",
|
||||
"rollout_paused", "rollout_start_failed",
|
||||
]
|
||||
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
||||
|
||||
|
||||
class NotificationSettingsRead(BaseModel):
|
||||
@@ -45,10 +50,28 @@ class NotificationSettingsRead(BaseModel):
|
||||
scheduled_maintenance_failed_enabled: bool = True
|
||||
backup_failed_enabled: bool = True
|
||||
compliance_alerts_enabled: bool = False
|
||||
gateway_down_enabled: bool = True
|
||||
resource_threshold_enabled: bool = False
|
||||
service_down_enabled: bool = False
|
||||
rollout_problems_enabled: bool = True
|
||||
gateway_loss_threshold_pct: int = 20
|
||||
gateway_delay_threshold_ms: int = 500
|
||||
disk_threshold_pct: int = 90
|
||||
memory_threshold_pct: int = 95
|
||||
swap_threshold_pct: int = 50
|
||||
load_per_cpu_threshold: float = 2.0
|
||||
alert_confirm_samples: int = 3
|
||||
resource_confirm_samples: int = 5
|
||||
service_watchlist: list[str] = Field(default_factory=lambda: list(DEFAULT_SERVICE_WATCHLIST))
|
||||
|
||||
poll_interval_seconds: int
|
||||
updated_at: datetime
|
||||
|
||||
@field_validator("service_watchlist", mode="before")
|
||||
@classmethod
|
||||
def _default_watchlist(cls, value):
|
||||
return list(DEFAULT_SERVICE_WATCHLIST) if value is None else value
|
||||
|
||||
|
||||
class NotificationSettingsUpdate(BaseModel):
|
||||
"""Partial update — everything optional, only provided fields change.
|
||||
@@ -84,6 +107,34 @@ class NotificationSettingsUpdate(BaseModel):
|
||||
scheduled_maintenance_failed_enabled: bool | None = None
|
||||
backup_failed_enabled: bool | None = None
|
||||
compliance_alerts_enabled: bool | None = None
|
||||
gateway_down_enabled: bool | None = None
|
||||
resource_threshold_enabled: bool | None = None
|
||||
service_down_enabled: bool | None = None
|
||||
rollout_problems_enabled: bool | None = None
|
||||
# 0 switches that single threshold off.
|
||||
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
||||
disk_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
memory_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
swap_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
load_per_cpu_threshold: float | None = Field(default=None, ge=0, le=1000)
|
||||
alert_confirm_samples: int | None = Field(default=None, ge=1, le=60)
|
||||
resource_confirm_samples: int | None = Field(default=None, ge=1, le=60)
|
||||
service_watchlist: list[str] | None = Field(default=None, max_length=100)
|
||||
|
||||
@field_validator("service_watchlist")
|
||||
@classmethod
|
||||
def _clean_watchlist(cls, value):
|
||||
if value is None:
|
||||
return None
|
||||
names = []
|
||||
for raw in value:
|
||||
name = str(raw).strip()
|
||||
if not re.fullmatch(r"[A-Za-z0-9._+-]{1,64}", name):
|
||||
raise ValueError("Dienstnamen bestehen aus Buchstaben, Ziffern und . _ + -")
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
poll_interval_seconds: int | None = Field(
|
||||
default=None, ge=settings.notification_poll_min_interval_seconds
|
||||
@@ -106,3 +157,55 @@ class NotificationProblemRead(BaseModel):
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
channel: Literal["webhook", "email"]
|
||||
|
||||
|
||||
class NotificationChannelRead(BaseModel):
|
||||
"""Never carries the credential; ``config`` is withheld from non-admins (an ntfy topic is a password)."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
name: str
|
||||
channel_type: ChannelType
|
||||
enabled: bool
|
||||
config: dict
|
||||
secret_set: bool
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
def _channel_name(value: str | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if not value.strip():
|
||||
raise ValueError("Name darf nicht leer sein")
|
||||
return value.strip()
|
||||
|
||||
|
||||
class NotificationChannelCreate(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=100)
|
||||
channel_type: ChannelType
|
||||
enabled: bool = True
|
||||
config: dict = Field(default_factory=dict)
|
||||
# Token, or for Teams the webhook address. Write-only.
|
||||
secret: str | None = Field(default=None, max_length=2100)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def _clean_name(cls, value):
|
||||
return _channel_name(value)
|
||||
|
||||
|
||||
class NotificationChannelUpdate(BaseModel):
|
||||
"""Partial: omitted fields stay. ``secret`` replaces the credential, ``clear_secret`` removes it."""
|
||||
|
||||
name: str | None = Field(default=None, min_length=1, max_length=100)
|
||||
enabled: bool | None = None
|
||||
config: dict | None = None
|
||||
secret: str | None = Field(default=None, max_length=2100)
|
||||
clear_secret: bool = False
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def _clean_name(cls, value):
|
||||
return _channel_name(value)
|
||||
@@ -4,6 +4,8 @@ from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from app.schemas.adopt import LinkTarget
|
||||
|
||||
|
||||
class RuleCreate(BaseModel):
|
||||
description: str = Field(min_length=1, max_length=255)
|
||||
@@ -78,6 +80,8 @@ class DeployApproval(BaseModel):
|
||||
|
||||
class AdoptRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link to this existing central rule instead of creating a new one (services/adopt_link_service.py).
|
||||
link_to: uuid.UUID | None = None
|
||||
rule_uuid: str = Field(min_length=1, max_length=64)
|
||||
|
||||
|
||||
@@ -98,3 +102,7 @@ class UnmanagedRule(BaseModel):
|
||||
adoptable: bool
|
||||
# Codes such as "negated_source" or "not_quick"; the UI turns them into text.
|
||||
reasons: list[str]
|
||||
# Central rules equal to this one (resolved for the host), same-description rules that differ, link possible.
|
||||
matches: list[LinkTarget] = Field(default_factory=list)
|
||||
similar: list[LinkTarget] = Field(default_factory=list)
|
||||
linkable: bool = False
|
||||
@@ -7,9 +7,19 @@ how the proxy setup works for IP *and* name access) nor an explicitly allowed
|
||||
origin. Requests without either header (curl, scripts) are not browsers, so
|
||||
they cannot be riding on a victim's cookie and pass through to normal auth.
|
||||
|
||||
Sec-Fetch-Site (1.27): a page on another port of the same host (e.g. a
|
||||
firewall GUI behind the GUI gateway) is *same-site* to OPNexus, so the
|
||||
browser attaches the session cookie to its requests and SameSite=Lax does
|
||||
not help. Cookie-carrying /api requests that the browser itself marks as
|
||||
same-site or cross-site are therefore refused — unless the Origin is an
|
||||
explicitly configured frontend origin (development setup with the frontend
|
||||
on its own port). Clients that send no Sec-Fetch-Site header (curl,
|
||||
scripts, API tokens) are not browsers and stay unaffected.
|
||||
|
||||
SecurityHeadersMiddleware: API responses must never be cached by browsers or
|
||||
proxies and must not be sniffed or framed.
|
||||
"""
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
@@ -18,6 +28,7 @@ from starlette.responses import JSONResponse
|
||||
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode
|
||||
from app.routers.auth import has_session_cookie
|
||||
|
||||
_SAFE_METHODS = {"GET", "HEAD", "OPTIONS"}
|
||||
|
||||
@@ -35,8 +46,41 @@ def _origin_allowed(origin: str, host_header: str) -> bool:
|
||||
return f"{parsed.scheme}://{parsed.netloc}".lower() in {o.lower() for o in settings.allowed_origins}
|
||||
|
||||
|
||||
# The SSO provider sends the browser back here by a cross-site navigation;
|
||||
# the endpoint authenticates by its own state cookie and creates the session.
|
||||
_SEC_FETCH_EXEMPT = {"/api/auth/oidc/callback"}
|
||||
|
||||
|
||||
def _explicit_frontend_origin(origin: str) -> bool:
|
||||
origin = origin.rstrip("/").lower()
|
||||
if not origin or origin == "null":
|
||||
return False
|
||||
if settings.allowed_origins:
|
||||
return origin in {o.lower() for o in settings.allowed_origins}
|
||||
return re.fullmatch(settings.cors_origin_regex, origin) is not None
|
||||
|
||||
|
||||
def sec_fetch_blocked(request: Request) -> bool:
|
||||
path = request.url.path
|
||||
if not (path == "/api" or path.startswith("/api/")) or path in _SEC_FETCH_EXEMPT:
|
||||
return False
|
||||
if not has_session_cookie(request):
|
||||
return False
|
||||
site = request.headers.get("sec-fetch-site", "").strip().lower()
|
||||
if site == "cross-site":
|
||||
return True
|
||||
if site == "same-site":
|
||||
return not _explicit_frontend_origin(request.headers.get("origin", ""))
|
||||
return False # same-origin, none (typed URL/bookmark) or no header (not a browser)
|
||||
|
||||
|
||||
class OriginCheckMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if sec_fetch_blocked(request):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": {"code": ErrorCode.CROSS_SITE_REQUEST_BLOCKED.value, "params": {}}},
|
||||
)
|
||||
if request.method not in _SAFE_METHODS:
|
||||
source = request.headers.get("origin") or request.headers.get("referer")
|
||||
if source and source != "null" and not _origin_allowed(source, request.headers.get("host", "")):
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Small pieces shared by every "take over what was made by hand on a firewall" feature
|
||||
(rule_adopt_service.py for filter rules, object_adopt_service.py for aliases and NAT)."""
|
||||
|
||||
|
||||
class NotAdoptable(Exception):
|
||||
"""The object exists on the host but cannot be described without distortion.
|
||||
|
||||
`reasons` are machine-readable codes (the UI translates them); `details` maps a code to
|
||||
extra text the translation can show (e.g. the alias names a reference points to)."""
|
||||
|
||||
def __init__(self, reasons: list[str], details: dict[str, str] | None = None):
|
||||
super().__init__(", ".join(reasons))
|
||||
self.reasons = reasons
|
||||
self.details = details or {}
|
||||
|
||||
|
||||
class NotFoundOnHost(Exception):
|
||||
"""The UUID is not (or no longer) present in the host's live configuration."""
|
||||
|
||||
|
||||
def on(value) -> bool:
|
||||
"""OPNsense booleans arrive as "1"/"0" strings, sometimes as real booleans."""
|
||||
return value in ("1", 1, True)
|
||||
|
||||
|
||||
def clean(value) -> str:
|
||||
return "" if value is None else str(value).strip()
|
||||
|
||||
|
||||
class LinkTargetNotFound(Exception):
|
||||
"""The central object chosen to link to was deleted in the meantime."""
|
||||
|
||||
|
||||
class LinkTargetChanged(Exception):
|
||||
"""The central object no longer equals what is live on the host. `diff` lists the differing fields."""
|
||||
|
||||
def __init__(self, diff: list[dict]):
|
||||
super().__init__(", ".join(d["field"] for d in diff))
|
||||
self.diff = diff
|
||||
|
||||
|
||||
class HostAlreadyHasObject(Exception):
|
||||
"""The central object is already deployed to / managed on this host under another UUID."""
|
||||
|
||||
|
||||
class LinkTargetBusy(Exception):
|
||||
"""The central object has an unconfirmed deployment or a failed rollback; linking waits until it is settled."""
|
||||
|
||||
|
||||
LINK_ERRORS = (LinkTargetNotFound, LinkTargetChanged, HostAlreadyHasObject, LinkTargetBusy)
|
||||
|
||||
|
||||
def link_api_error(exc: Exception):
|
||||
"""The HTTP error for a failed link (shared by the rule and the object endpoints)."""
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
if isinstance(exc, LinkTargetNotFound):
|
||||
return api_error(404, ErrorCode.ADOPT_LINK_TARGET_NOT_FOUND)
|
||||
if isinstance(exc, LinkTargetChanged):
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_TARGET_CHANGED, fields=", ".join(d["field"] for d in exc.diff))
|
||||
if isinstance(exc, HostAlreadyHasObject):
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_HOST_HAS_OBJECT)
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_BUSY)
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Links a hand-made object that was found on a firewall to a central object that already exists, instead
|
||||
of creating a second central object for it ("Mit vorhandenem zentralem Objekt verknüpfen").
|
||||
|
||||
Typical case: the same rule/alias/NAT entry was made by hand on several firewalls (or on both nodes of an HA
|
||||
pair). The first adoption creates the central object; every further firewall is LINKED to it: the host
|
||||
becomes another target of that object and gets a deployment row holding the live UUID (remote_uuid) and
|
||||
the payload OPNexus would send. Like adopting, linking writes NOTHING to the firewall. The live state is
|
||||
read fresh from the host at link time (the browser only names the central object), so the preview right
|
||||
afterwards says "unchanged" and a later rollout updates the one object on every host without a duplicate.
|
||||
|
||||
A central object is an EXACT match when every field OPNexus manages is equal after the object's
|
||||
{{VARIABLE}} placeholders are replaced with this host's values (variables_service.resolve). An object with
|
||||
the same identity (alias name / rule description) but different content is only shown as a hint with the
|
||||
differing fields and is NEVER linked automatically.
|
||||
|
||||
Callers hold policy_lock (same serialisation as adopting), which also makes a repeated link idempotent.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Callable
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.services import variables_service
|
||||
from app.services.adopt_common import HostAlreadyHasObject, LinkTargetBusy, LinkTargetChanged, LinkTargetNotFound
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import effective_hosts
|
||||
|
||||
LINK_MESSAGE = "Mit vorhandenem Objekt verknüpft"
|
||||
BUSY_STATUSES = ("pending", "revert_failed") # the central object waits for a confirmation / failed to roll back
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LinkSpec:
|
||||
model: type
|
||||
deployment: type
|
||||
fk: str # deployment column pointing at the central object
|
||||
uuid_attr: str # deployment column holding the OPNsense UUID (remote_uuid)
|
||||
identity: str # attribute naming the object: "name" (alias) or "description"
|
||||
payload: Callable[[Any, OpnsenseHost], dict]
|
||||
revision: Callable
|
||||
|
||||
|
||||
@dataclass
|
||||
class Match:
|
||||
obj: Any
|
||||
diff: list[dict] # empty = exact match
|
||||
busy: bool = False # the central object waits for a confirmation: linking is refused until that is settled
|
||||
|
||||
|
||||
def _norm(value) -> Any:
|
||||
if value is None:
|
||||
return ""
|
||||
return value if isinstance(value, bool) else str(value)
|
||||
|
||||
|
||||
def _show(value) -> str:
|
||||
return ("true" if value else "false") if isinstance(value, bool) else ("" if value is None else str(value))
|
||||
|
||||
|
||||
def diff_fields(obj, values: dict, host: OpnsenseHost) -> list[dict]:
|
||||
"""Fields in which the central object (resolved for `host`) differs from the live `values`."""
|
||||
raw = {key: getattr(obj, key) for key in values}
|
||||
try:
|
||||
central = variables_service.resolve(raw, host)
|
||||
except variables_service.MissingVariables as exc:
|
||||
# Cannot be compared (and could not be deployed to this host either): report, never match.
|
||||
return [{"field": "variables", "central": ", ".join(exc.missing[host.name]), "live": ""}]
|
||||
return [{"field": key, "central": _show(central[key]), "live": _show(values[key])}
|
||||
for key in values if _norm(central[key]) != _norm(values[key])]
|
||||
|
||||
|
||||
def _options(spec: LinkSpec):
|
||||
return [
|
||||
selectinload(spec.model.targets),
|
||||
selectinload(spec.model.groups).selectinload(DeviceGroup.hosts),
|
||||
selectinload(spec.model.deployments),
|
||||
]
|
||||
|
||||
|
||||
async def load_central(db: AsyncSession, spec: LinkSpec) -> list:
|
||||
objs = list((await db.scalars(select(spec.model).options(*_options(spec)))).unique().all())
|
||||
await hydrate_hierarchy_targets(db, objs)
|
||||
return objs
|
||||
|
||||
|
||||
def find(objs: list, spec: LinkSpec, host: OpnsenseHost, values: dict) -> tuple[list[Match], list[Match]]:
|
||||
"""(exact matches, same-identity objects that differ). Objects that already have a deployment on
|
||||
this host are not candidates: linking would collide with that deployment."""
|
||||
exact: list[Match] = []
|
||||
similar: list[Match] = []
|
||||
for obj in objs:
|
||||
if any(d.host_id == host.id for d in obj.deployments):
|
||||
continue
|
||||
if _norm(getattr(obj, spec.identity)) != _norm(values[spec.identity]):
|
||||
continue
|
||||
d = diff_fields(obj, values, host)
|
||||
busy = any(dep.status in BUSY_STATUSES for dep in obj.deployments)
|
||||
(similar if d else exact).append(Match(obj, d, busy))
|
||||
# prefer the object that already reaches the most hosts (the "real" one), then a stable order
|
||||
exact.sort(key=lambda m: (-len(effective_hosts(m.obj)), str(m.obj.id)))
|
||||
return exact, similar
|
||||
|
||||
|
||||
def describe(spec: LinkSpec, match: Match) -> dict:
|
||||
return {
|
||||
"id": match.obj.id,
|
||||
"name": getattr(match.obj, spec.identity),
|
||||
"targets": [h.name for h in effective_hosts(match.obj)],
|
||||
"diff": match.diff,
|
||||
}
|
||||
|
||||
|
||||
async def link(db: AsyncSession, spec: LinkSpec, host: OpnsenseHost, object_id, remote_uuid: str, values: dict):
|
||||
"""Links the live object `remote_uuid` (whose normalised fields are `values`, read fresh by the caller)
|
||||
to the central object `object_id`. Returns it; a repeated call for the same pair changes nothing."""
|
||||
obj = (await db.execute(
|
||||
select(spec.model).where(spec.model.id == object_id).options(*_options(spec))
|
||||
.execution_options(populate_existing=True)
|
||||
)).unique().scalar_one_or_none()
|
||||
if obj is None:
|
||||
raise LinkTargetNotFound(str(object_id))
|
||||
await hydrate_hierarchy_targets(db, [obj])
|
||||
|
||||
existing = next((d for d in obj.deployments if d.host_id == host.id), None)
|
||||
if existing is not None:
|
||||
if getattr(existing, spec.uuid_attr) == remote_uuid:
|
||||
return obj # already linked: nothing to do
|
||||
raise HostAlreadyHasObject(str(object_id))
|
||||
if any(d.status in BUSY_STATUSES for d in obj.deployments):
|
||||
raise LinkTargetBusy(str(object_id))
|
||||
changed = diff_fields(obj, values, host)
|
||||
if changed:
|
||||
raise LinkTargetChanged(changed)
|
||||
|
||||
if host.id not in {h.id for h in effective_hosts(obj)}:
|
||||
obj.targets.append(host)
|
||||
# an association-only change does not touch the row; stale edit forms must still be refused (see edit_rule)
|
||||
obj.updated_at = datetime.now(timezone.utc)
|
||||
await db.flush()
|
||||
spec.revision(db, obj, f"Mit {host.name} verknüpft")
|
||||
db.add(spec.deployment(**{
|
||||
spec.fk: obj.id, "host_id": host.id, "status": "success", "message": LINK_MESSAGE,
|
||||
spec.uuid_attr: remote_uuid, "last_payload": spec.payload(obj, host), "previous_payload": None,
|
||||
}))
|
||||
await db.flush()
|
||||
return obj
|
||||
@@ -38,7 +38,8 @@ MAX_TOKENS_PER_USER = 50
|
||||
_LAST_USED_GRANULARITY = timedelta(minutes=1)
|
||||
|
||||
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)")
|
||||
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$")
|
||||
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
||||
# notification settings the webhook URL/SMTP data): admin scope only.
|
||||
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
||||
@@ -48,6 +49,7 @@ _ADMIN_ONLY_WRITES = re.compile(
|
||||
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
||||
r"|^/api/logs/settings$"
|
||||
r"|^/api/hosts(/tls-probe|/[^/]+)?$"
|
||||
r"|^/api/hosts/[^/]+/gui-ticket$"
|
||||
)
|
||||
|
||||
|
||||
@@ -90,7 +92,10 @@ _ROLLOUT_POST = re.compile(
|
||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||
r"|maintenance/(execute|carp-execute|schedule)"
|
||||
r"|network/changes"
|
||||
r"|services/[^/]+/(start|stop|restart))$"
|
||||
r"|services/[^/]+/(start|stop|restart)"
|
||||
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
|
||||
r"|rollouts/plans/[^/]+/(start|activate)"
|
||||
r"|rollouts/runs/[^/]+/(resume|approve|skip-host|abort))$"
|
||||
)
|
||||
_ROLLOUT_DELETE = re.compile(
|
||||
r"^/api/(rules|aliases|port-forward|one-to-one|outbound-nat|policy-hierarchy/rules|ipsec/tunnels|openvpn/instances"
|
||||
@@ -99,7 +104,13 @@ _ROLLOUT_DELETE = re.compile(
|
||||
)
|
||||
|
||||
|
||||
# Recurring maintenance windows decide WHEN unattended firmware actions run (1.25): every write counts.
|
||||
_ROLLOUT_ANY_WRITE = re.compile(r"^/api/maintenance-windows(/|$)")
|
||||
|
||||
|
||||
def is_rollout(method: str, path: str) -> bool:
|
||||
if method in ("POST", "PUT", "PATCH", "DELETE") and _ROLLOUT_ANY_WRITE.match(path):
|
||||
return True
|
||||
if method == "POST":
|
||||
return bool(_ROLLOUT_POST.search(path))
|
||||
return method == "DELETE" and bool(_ROLLOUT_DELETE.match(path))
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Gateway health of a firewall (dpinger numbers, as OPNsense reports them).
|
||||
|
||||
``get_gateway_health(host)`` is the one reusable entry point: a live read of
|
||||
/routes/gateway/status, one :class:`GatewayHealth` per gateway. The notification
|
||||
engine, the host page and the metrics read the SAME numbers from the periodic
|
||||
resource sample (``compact`` / ``from_compact``), so a scrape or a page view never
|
||||
fans out to the firewalls; the 1.25 health gate can call ``get_gateway_health``
|
||||
directly. Parsing is lenient on purpose: "~" means "nothing measured", never zero.
|
||||
"""
|
||||
import re
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.gateway import GatewayHealth, GatewayState
|
||||
from app.services import opnsense_client as client
|
||||
|
||||
_NUMBER = re.compile(r"-?\d+(?:\.\d+)?")
|
||||
_DEGRADED = {"delay", "loss", "delay+loss"}
|
||||
|
||||
|
||||
def _number(value) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
match = _NUMBER.search(str(value))
|
||||
return float(match.group()) if match else None
|
||||
|
||||
|
||||
def _state(status: str, measured: bool) -> GatewayState:
|
||||
if status == "down":
|
||||
return "offline"
|
||||
if status == "force_down":
|
||||
return "forced_down"
|
||||
if status in _DEGRADED:
|
||||
return "degraded"
|
||||
if status == "none":
|
||||
# "none" is also what a gateway without monitoring reports; only numbers make it "online".
|
||||
return "online" if measured else "unknown"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def parse_gateway_status(payload) -> list[GatewayHealth]:
|
||||
items = payload.get("items") if isinstance(payload, dict) else None
|
||||
result: list[GatewayHealth] = []
|
||||
for item in items or []:
|
||||
if not isinstance(item, dict) or not item.get("name"):
|
||||
continue
|
||||
status = str(item.get("status") or "").lower()
|
||||
loss, delay, stddev = _number(item.get("loss")), _number(item.get("delay")), _number(item.get("stddev"))
|
||||
if loss is not None and loss >= 100:
|
||||
delay = stddev = None # dpinger prints 0.0 ms when nothing answers: no round trip, not a fast one
|
||||
measured = loss is not None or delay is not None
|
||||
address = item.get("address")
|
||||
monitor = item.get("monitor")
|
||||
result.append(GatewayHealth(
|
||||
name=str(item["name"]),
|
||||
address=None if address in (None, "", "~") else str(address),
|
||||
status=status or "unknown",
|
||||
state=_state(status, measured),
|
||||
loss=loss, delay=delay, stddev=stddev,
|
||||
monitor=None if monitor in (None, "", "~") else str(monitor),
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
async def get_gateway_health(host: OpnsenseHost) -> list[GatewayHealth]:
|
||||
"""Live gateway state of ``host``. Raises OpnsenseAPIError if the firewall cannot be asked."""
|
||||
return parse_gateway_status(await client.get_gateway_status(host))
|
||||
|
||||
|
||||
def compact(gateways: list[GatewayHealth]) -> list[dict]:
|
||||
"""What the resource sampler stores per sample (small, no addresses)."""
|
||||
return [
|
||||
{"name": g.name, "status": g.status, "state": g.state, "loss": g.loss, "delay": g.delay, "stddev": g.stddev}
|
||||
for g in gateways
|
||||
]
|
||||
|
||||
|
||||
def from_compact(rows) -> list[GatewayHealth]:
|
||||
return [
|
||||
GatewayHealth(
|
||||
name=r["name"], status=r.get("status") or "unknown", state=r.get("state") or "unknown",
|
||||
loss=r.get("loss"), delay=r.get("delay"), stddev=r.get("stddev"),
|
||||
)
|
||||
for r in rows or [] if isinstance(r, dict) and r.get("name")
|
||||
]
|
||||
|
||||
|
||||
def is_bad(gateway: GatewayHealth, loss_threshold_pct: float = 0, delay_threshold_ms: float = 0) -> bool:
|
||||
"""Alarm-worthy: down, or (when the thresholds are on) measured loss / latency beyond them.
|
||||
A gateway forced down by the admin and one that is not measured are not alarms."""
|
||||
if gateway.state == "offline":
|
||||
return True
|
||||
if loss_threshold_pct and gateway.loss is not None and gateway.loss >= loss_threshold_pct:
|
||||
return True
|
||||
if delay_threshold_ms and gateway.delay is not None and gateway.delay >= delay_threshold_ms:
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -0,0 +1,320 @@
|
||||
"""Tickets and gateway sessions for the firewall GUI access (1.27, gui-gateway container).
|
||||
|
||||
Flow (see docs in routers/gui_access.py):
|
||||
* An admin asks for a ticket (POST /api/hosts/{id}/gui-ticket). 32 random bytes, valid 30 s, single use;
|
||||
only its SHA-256 is stored, bound to the OPNexus session (same hash as sessions.token_hash), the user,
|
||||
the host and a slot (gateway port = its own browser origin).
|
||||
* The gateway redeems it through /internal/gui/redeem and gets what it needs to reach the firewall —
|
||||
base URL, pinned CA, overlay proxy, timeouts. Never the API key or secret.
|
||||
* Every 20 s the gateway re-checks the gateway session (/internal/gui/revalidate): OPNexus session still
|
||||
valid, user still admin, switch still on, host still eligible, absolute limit not reached.
|
||||
|
||||
A slot stays assigned to one firewall for as long as that browser session has the firewall open; another
|
||||
firewall gets another slot (another origin), so two firewall GUIs never share an origin at the same time.
|
||||
"""
|
||||
import hashlib
|
||||
import secrets
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import delete, func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode
|
||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser, Session
|
||||
from app.services import audit_service, firewall_url, overlay_service
|
||||
|
||||
TICKET_TTL = timedelta(seconds=30)
|
||||
# 20 s: a switch-off, logout or role change must end a gateway session within 30 s of the change, and
|
||||
# the last re-check can lie up to one interval before it (live test 2026-10-21: 30 s gave 31 s).
|
||||
REVALIDATE_SECONDS = 20
|
||||
MAX_SLOTS = 4
|
||||
MIN_GATEWAY_TOKEN_LENGTH = 32
|
||||
AUDIT_KIND = "gui_access"
|
||||
|
||||
|
||||
@dataclass
|
||||
class GuiAccessError(Exception):
|
||||
status: int
|
||||
code: ErrorCode
|
||||
params: dict
|
||||
|
||||
|
||||
def hash_value(raw: str) -> str:
|
||||
return hashlib.sha256(raw.encode()).hexdigest()
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
# ── configuration ─────────────────────────────────────────────────────────
|
||||
|
||||
def slot_sites() -> dict[int, list[str]]:
|
||||
"""Slot port -> configured https origins with that port (OPNEXUS_GUI_SITES)."""
|
||||
sites: dict[int, list[str]] = {}
|
||||
for origin in settings.gui_site_origins:
|
||||
try:
|
||||
parts = urlsplit(origin)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
continue
|
||||
if parts.scheme != "https" or not parts.hostname or not port or parts.path not in ("", "/"):
|
||||
continue
|
||||
sites.setdefault(port, []).append(f"https://{parts.netloc}")
|
||||
return sites
|
||||
|
||||
|
||||
def gateway_configured() -> bool:
|
||||
return len(settings.gui_gateway_token) >= MIN_GATEWAY_TOKEN_LENGTH and bool(slot_sites())
|
||||
|
||||
|
||||
def unavailable_reasons() -> list[str]:
|
||||
reasons = []
|
||||
if not settings.cookie_secure:
|
||||
reasons.append("cookie_secure")
|
||||
if len(settings.gui_gateway_token) < MIN_GATEWAY_TOKEN_LENGTH:
|
||||
reasons.append("gateway_token")
|
||||
if not slot_sites():
|
||||
reasons.append("gui_sites")
|
||||
return reasons
|
||||
|
||||
|
||||
def active_slots(row: GuiAccessSettings) -> list[int]:
|
||||
return sorted(slot_sites())[: max(1, min(row.slot_count, MAX_SLOTS))]
|
||||
|
||||
|
||||
async def get_settings(db: AsyncSession) -> GuiAccessSettings:
|
||||
row = await db.get(GuiAccessSettings, 1)
|
||||
if row is None:
|
||||
row = GuiAccessSettings(id=1, enabled=False, idle_timeout_minutes=15, absolute_timeout_minutes=60, slot_count=MAX_SLOTS)
|
||||
db.add(row)
|
||||
await db.flush()
|
||||
return row
|
||||
|
||||
|
||||
def host_refusal(host: OpnsenseHost) -> str | None:
|
||||
"""Why this firewall must not be opened through the gateway (None = eligible)."""
|
||||
if not host.enabled:
|
||||
return "disabled"
|
||||
if urlsplit(host.base_url.strip()).scheme != "https" or firewall_url.check_base_url(host.base_url):
|
||||
return "https"
|
||||
if not host.verify_tls:
|
||||
return "tls"
|
||||
return None
|
||||
|
||||
|
||||
def firewall_origin(base_url: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
return f"https://{parts.netloc.rsplit('@', 1)[-1]}"
|
||||
|
||||
|
||||
# ── audit ─────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _audit(actor: str, *, action: str, status: str, host: OpnsenseHost | None = None,
|
||||
host_id=None, entity_id=None, message: str | None = None, details: dict | None = None) -> None:
|
||||
token = audit_service.context.set({"actor": actor, "id": str(uuid.uuid4()), "action": action})
|
||||
try:
|
||||
await audit_service.persist(
|
||||
kind=AUDIT_KIND, action=action, status=status, entity_id=str(entity_id) if entity_id else None,
|
||||
host_id=str(host.id) if host is not None else (str(host_id) if host_id else None),
|
||||
host_name=host.name if host is not None else None, message=message, details=details or {},
|
||||
)
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
# ── tickets ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _redeem_origin(slot: int, request_host: str) -> str:
|
||||
candidates = slot_sites().get(slot, [])
|
||||
wanted = (urlsplit(f"//{request_host}").hostname or "").lower()
|
||||
for origin in candidates:
|
||||
if (urlsplit(origin).hostname or "").lower() == wanted:
|
||||
return origin
|
||||
return candidates[0]
|
||||
|
||||
|
||||
async def _pick_slot(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id) -> int:
|
||||
now = _now()
|
||||
slots = active_slots(row)
|
||||
absolute = timedelta(minutes=row.absolute_timeout_minutes)
|
||||
in_use: dict[int, uuid.UUID] = {}
|
||||
open_sessions = await db.scalars(select(GuiSession).where(
|
||||
GuiSession.session_hash == session_hash, GuiSession.ended_at.is_(None), GuiSession.started_at > now - absolute))
|
||||
for gs in open_sessions:
|
||||
in_use[gs.slot] = gs.host_id
|
||||
pending = await db.scalars(select(GuiTicket).where(
|
||||
GuiTicket.session_hash == session_hash, GuiTicket.redeemed_at.is_(None), GuiTicket.expires_at > now))
|
||||
for ticket in pending:
|
||||
in_use.setdefault(ticket.slot, ticket.host_id)
|
||||
for slot in slots:
|
||||
if in_use.get(slot) == host_id:
|
||||
return slot # this firewall is already open in this browser session: same origin again
|
||||
free = [s for s in slots if s not in in_use]
|
||||
if not free:
|
||||
raise GuiAccessError(409, ErrorCode.GUI_NO_FREE_SLOT, {"slots": str(len(slots))})
|
||||
# Prefer the slot this user had this firewall on before, then the slot the user used least recently
|
||||
# (an old tab of another firewall on a reused origin is the documented residual risk).
|
||||
history = (await db.execute(
|
||||
select(GuiSession.slot, GuiSession.host_id, func.max(GuiSession.started_at))
|
||||
.where(GuiSession.user_id == user_id).group_by(GuiSession.slot, GuiSession.host_id)
|
||||
)).all()
|
||||
for slot, h_id, _ in sorted(history, key=lambda r: r[2], reverse=True):
|
||||
if h_id == host_id and slot in free:
|
||||
return slot
|
||||
last_used = {}
|
||||
for slot, _, started in history:
|
||||
last_used[slot] = max(last_used.get(slot, started), started)
|
||||
epoch = datetime.min.replace(tzinfo=timezone.utc)
|
||||
return min(free, key=lambda s: (last_used.get(s, epoch), s))
|
||||
|
||||
|
||||
async def issue_ticket(db: AsyncSession, *, user: AdminUser, session_token: str, host: OpnsenseHost,
|
||||
request_host: str) -> dict:
|
||||
row = await get_settings(db)
|
||||
now = _now()
|
||||
await db.execute(delete(GuiTicket).where(GuiTicket.expires_at < now - timedelta(hours=1)))
|
||||
session_hash = hash_value(session_token)
|
||||
slot = await _pick_slot(db, row, session_hash, user.id, host.id)
|
||||
raw = secrets.token_urlsafe(32)
|
||||
db.add(GuiTicket(ticket_hash=hash_value(raw), session_hash=session_hash, user_id=user.id, host_id=host.id,
|
||||
slot=slot, created_at=now, expires_at=now + TICKET_TTL))
|
||||
await db.commit()
|
||||
await _audit(user.username, action="ticket", status="accepted", host=host, details={"slot": slot})
|
||||
return {"url": f"{_redeem_origin(slot, request_host)}/__onx/redeem?t={raw}", "slot": slot,
|
||||
"expires_in": int(TICKET_TTL.total_seconds())}
|
||||
|
||||
|
||||
# ── gateway side (internal endpoints) ─────────────────────────────────────
|
||||
|
||||
async def _live_check(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id,
|
||||
slot: int) -> tuple[str | None, AdminUser | None, OpnsenseHost | None]:
|
||||
session = await db.scalar(select(Session).where(Session.token_hash == session_hash))
|
||||
if session is None or session.expires_at <= _now() or session.user_id != user_id:
|
||||
return "session", None, None
|
||||
user = await db.get(AdminUser, user_id)
|
||||
if user is None or user.role != "admin":
|
||||
return "role", user, None
|
||||
if not row.enabled:
|
||||
return "disabled", user, None
|
||||
if not settings.cookie_secure or not gateway_configured():
|
||||
return "config", user, None
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None or host_refusal(host):
|
||||
return "host", user, host
|
||||
if slot not in active_slots(row):
|
||||
return "slot", user, host
|
||||
return None, user, host
|
||||
|
||||
|
||||
def _timing(row: GuiAccessSettings, started_at: datetime) -> dict:
|
||||
deadline = started_at + timedelta(minutes=row.absolute_timeout_minutes)
|
||||
return {"idle_timeout_seconds": row.idle_timeout_minutes * 60,
|
||||
"absolute_deadline": int(deadline.timestamp()),
|
||||
"revalidate_seconds": REVALIDATE_SECONDS}
|
||||
|
||||
|
||||
async def redeem(db: AsyncSession, *, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
|
||||
"""{'ok': True, ...upstream data...} or {'ok': False, 'reason': ...}. Any attempt burns the ticket."""
|
||||
now = _now()
|
||||
ticket = await db.scalar(select(GuiTicket).where(GuiTicket.ticket_hash == ticket_hash).with_for_update())
|
||||
if ticket is None:
|
||||
await db.rollback()
|
||||
return {"ok": False, "reason": "unknown"}
|
||||
replayed = ticket.redeemed_at is not None
|
||||
ticket.redeemed_at = ticket.redeemed_at or now
|
||||
row = await get_settings(db)
|
||||
reason = None
|
||||
if replayed:
|
||||
reason = "replayed"
|
||||
elif ticket.expires_at <= now:
|
||||
reason = "expired"
|
||||
elif not secrets.compare_digest(ticket.session_hash, session_hash):
|
||||
reason = "session"
|
||||
elif ticket.slot != slot:
|
||||
reason = "slot"
|
||||
user = host = None
|
||||
if reason is None:
|
||||
reason, user, host = await _live_check(db, row, ticket.session_hash, ticket.user_id, ticket.host_id, slot)
|
||||
if user is None:
|
||||
user = await db.get(AdminUser, ticket.user_id)
|
||||
if host is None:
|
||||
host = await db.get(OpnsenseHost, ticket.host_id)
|
||||
actor = user.username if user is not None else "unknown"
|
||||
if reason is not None:
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
|
||||
message=reason, details={"slot": slot, "client_ip": client_ip, "reason": reason})
|
||||
return {"ok": False, "reason": reason}
|
||||
# The same browser session reopening this slot replaces the earlier gateway session there.
|
||||
superseded = (await db.scalars(select(GuiSession).where(
|
||||
GuiSession.session_hash == session_hash, GuiSession.slot == slot, GuiSession.ended_at.is_(None)))).all()
|
||||
for old in superseded:
|
||||
old.ended_at, old.end_reason = now, "superseded"
|
||||
gs = GuiSession(session_hash=session_hash, user_id=user.id, host_id=host.id, slot=slot,
|
||||
client_ip=(client_ip or None) and client_ip[:45], started_at=now, last_checked_at=now)
|
||||
db.add(gs)
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="accepted", host=host, entity_id=gs.id,
|
||||
details={"slot": slot, "client_ip": client_ip})
|
||||
return {
|
||||
"ok": True, "gui_session_id": str(gs.id), "host_id": str(host.id), "host_name": host.name,
|
||||
"upstream_origin": firewall_origin(host.base_url),
|
||||
"ca_pem": host.tls_ca_pem or None, # public certificate (pin); None = system trust store
|
||||
"proxy_url": overlay_service.proxy_for(host.base_url),
|
||||
**_timing(row, now),
|
||||
}
|
||||
|
||||
|
||||
async def _finish(db: AsyncSession, gs: GuiSession, reason: str) -> None:
|
||||
now = _now()
|
||||
if gs.ended_at is None:
|
||||
gs.ended_at, gs.end_reason = now, reason[:32]
|
||||
await db.commit()
|
||||
user = await db.get(AdminUser, gs.user_id)
|
||||
host = await db.get(OpnsenseHost, gs.host_id)
|
||||
await _audit(user.username if user else "unknown", action="end", status="accepted", host=host,
|
||||
host_id=gs.host_id, entity_id=gs.id, message=reason,
|
||||
details={"slot": gs.slot, "client_ip": gs.client_ip, "reason": reason,
|
||||
"duration_seconds": int((gs.ended_at - gs.started_at).total_seconds())})
|
||||
|
||||
|
||||
async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str) -> dict:
|
||||
gs = await db.get(GuiSession, gui_session_id)
|
||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||
return {"ok": False, "reason": "unknown"}
|
||||
if gs.ended_at is not None:
|
||||
return {"ok": False, "reason": gs.end_reason or "ended"}
|
||||
row = await get_settings(db)
|
||||
if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):
|
||||
await _finish(db, gs, "absolute")
|
||||
return {"ok": False, "reason": "absolute"}
|
||||
reason, _, _ = await _live_check(db, row, gs.session_hash, gs.user_id, gs.host_id, gs.slot)
|
||||
if reason is not None:
|
||||
await _finish(db, gs, reason)
|
||||
return {"ok": False, "reason": reason}
|
||||
gs.last_checked_at = _now()
|
||||
await db.commit()
|
||||
return {"ok": True, **_timing(row, gs.started_at)}
|
||||
|
||||
|
||||
async def end(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str, reason: str) -> bool:
|
||||
gs = await db.get(GuiSession, gui_session_id)
|
||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||
return False
|
||||
if gs.ended_at is None:
|
||||
await _finish(db, gs, reason)
|
||||
return True
|
||||
|
||||
|
||||
async def end_all(db: AsyncSession, reason: str) -> int:
|
||||
"""Switch turned off: mark every open gateway session ended (the gateway drops them on its next check)."""
|
||||
result = await db.execute(update(GuiSession).where(GuiSession.ended_at.is_(None))
|
||||
.values(ended_at=_now(), end_reason=reason[:32]))
|
||||
return result.rowcount or 0
|
||||
@@ -5,7 +5,7 @@ fan out live calls to every firewall (an offline host would stall it past the
|
||||
scraper's timeout and a 15 s scrape interval would hammer the fleet). The
|
||||
inputs are what OPNexus' own pollers already store:
|
||||
|
||||
* host reachability and resources — the 60 s resource sampler (a host that
|
||||
* host reachability, resources, gateway health and service state — the 60 s resource sampler (a host that
|
||||
produced no sample for ``STALE_AFTER`` is reported down),
|
||||
* CARP role — the latest observed transition (carp_events),
|
||||
* open problems — the notification engine's problem table,
|
||||
@@ -19,11 +19,11 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.models.carp_event import CarpEvent
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import NotificationProblem
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.routers.dashboard import collect_pending_actions
|
||||
from app.services import compliance_service
|
||||
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
||||
from app.version import APP_VERSION
|
||||
|
||||
CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
|
||||
@@ -99,6 +99,14 @@ async def render(db: AsyncSession) -> str:
|
||||
out.family("opnexus_host_memory_used_percent", "Memory in use, percent (last sample).")
|
||||
out.family("opnexus_host_load1", "1-minute load average (last sample).")
|
||||
out.family("opnexus_host_disk_used_percent", "Fullest disk, percent (last sample).")
|
||||
out.family("opnexus_host_swap_used_percent", "Swap in use, percent (last sample; absent without swap).")
|
||||
out.family("opnexus_host_gateway_up", "1 if the gateway is not down (online, degraded or not monitored), 0 if OPNsense reports it down.")
|
||||
out.family("opnexus_host_gateway_loss_percent", "Packet loss to the gateway monitor address, percent (last sample).")
|
||||
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
|
||||
configured = (await notification_settings_service.get_settings(db)).service_watchlist
|
||||
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
|
||||
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
|
||||
for host in hosts:
|
||||
labels = {"host": host.name, "cluster": host.cluster_name or ""}
|
||||
@@ -115,6 +123,22 @@ async def render(db: AsyncSession) -> str:
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, value, **labels)
|
||||
if sample is not None:
|
||||
raw = sample.raw or {}
|
||||
if raw.get("swap_used_pct") is not None:
|
||||
out.sample("opnexus_host_swap_used_percent", raw["swap_used_pct"], **labels)
|
||||
for gw in gateway_health_service.from_compact(raw.get("gateways")):
|
||||
gw_labels = {**labels, "gateway": gw.name}
|
||||
out.sample("opnexus_host_gateway_up", 0 if gw.state == "offline" else 1, **gw_labels)
|
||||
for name, value, factor in (
|
||||
("opnexus_host_gateway_loss_percent", gw.loss, 1), ("opnexus_host_gateway_delay_seconds", gw.delay, 0.001),
|
||||
("opnexus_host_gateway_stddev_seconds", gw.stddev, 0.001),
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, round(value * factor, 6), **gw_labels)
|
||||
for key, running in sorted((raw.get("services") or {}).items()):
|
||||
if key.split("/")[0] in watchlist:
|
||||
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
|
||||
if carp_state.get(host.id) in ("MASTER", "BACKUP"):
|
||||
out.sample("opnexus_host_carp_master", 1 if carp_state[host.id] == "MASTER" else 0, **labels)
|
||||
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
"""Monitoring alerts (1.24): gateway down, resource thresholds, stopped services.
|
||||
|
||||
Same contract as notification_checks.py (each check returns {dedup_key: context} of what is
|
||||
bad right now), with one addition: they never judge a single reading. The inputs are the
|
||||
60-second samples the resource sampler stores (resource_monitor_service, HostResourceSample),
|
||||
and a problem needs the last N samples to agree before it opens AND before it closes again:
|
||||
|
||||
* all N bad -> problem,
|
||||
* all N good -> no problem,
|
||||
* anything in between -> keep the current state (open stays open, closed stays closed).
|
||||
|
||||
So a single spike never alerts, a flapping link does not open and close the alert every few
|
||||
minutes, and a sample that could not be read ("unknown") neither raises nor clears anything.
|
||||
A host without a fresh sample is not judged at all (check_host_offline reports it); its open
|
||||
problems are kept.
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service, notification_settings_service
|
||||
from app.services.notification_dedup import Problems
|
||||
|
||||
# The sampler runs every 60 s; three missed rounds mean the host is not reachable (same limit as metrics_service).
|
||||
STALE_AFTER = timedelta(seconds=180)
|
||||
# How far back a service must have been seen running before its absence is reported (samples, ~2 h at 60 s).
|
||||
SERVICE_EVIDENCE_SAMPLES = 120
|
||||
_DISK_TYPES = {"ufs", "zfs"}
|
||||
|
||||
|
||||
def verdict(flags: list[bool | None], confirm: int, is_open: bool) -> bool:
|
||||
"""flags: newest first; True = bad, False = good, None = unknown."""
|
||||
window = flags[:confirm]
|
||||
if len(window) >= confirm:
|
||||
if all(f is True for f in window):
|
||||
return True
|
||||
if all(f is False for f in window):
|
||||
return False
|
||||
return is_open
|
||||
|
||||
|
||||
async def _enabled_hosts(db: AsyncSession) -> list[OpnsenseHost]:
|
||||
return list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)))).all())
|
||||
|
||||
|
||||
async def _open_keys(db: AsyncSession, problem_type: str) -> set[str]:
|
||||
rows = await db.scalars(
|
||||
select(NotificationProblem.dedup_key).where(
|
||||
NotificationProblem.problem_type == problem_type, NotificationProblem.status == "open"
|
||||
)
|
||||
)
|
||||
return set(rows.all())
|
||||
|
||||
|
||||
async def _recent(db: AsyncSession, host_id, limit: int, *raw_keys: str) -> list[dict]:
|
||||
"""The newest ``limit`` samples of a host, newest first, with just the JSON keys asked for."""
|
||||
columns = [HostResourceSample.sampled_at, HostResourceSample.mem_used_pct, HostResourceSample.load_avg_1m]
|
||||
columns += [HostResourceSample.raw[key].as_json() for key in raw_keys]
|
||||
rows = (
|
||||
await db.execute(
|
||||
select(*columns).where(HostResourceSample.host_id == host_id).order_by(HostResourceSample.sampled_at.desc()).limit(limit)
|
||||
)
|
||||
).all()
|
||||
return [
|
||||
{"sampled_at": r[0], "mem_used_pct": r[1], "load_avg_1m": r[2], **{k: r[3 + i] for i, k in enumerate(raw_keys)}}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def _fresh(samples: list[dict]) -> bool:
|
||||
return bool(samples) and datetime.now(timezone.utc) - samples[0]["sampled_at"] <= STALE_AFTER
|
||||
|
||||
|
||||
def _keys_of_host(open_keys: set[str], host: OpnsenseHost) -> set[str]:
|
||||
prefix = f"{host.id}:"
|
||||
return {k[len(prefix):] for k in open_keys if k.startswith(prefix)}
|
||||
|
||||
|
||||
# ── Gateways ────────────────────────────────────────────────────────────────
|
||||
|
||||
def _gateway_reason(g) -> str:
|
||||
if g.state == "offline":
|
||||
return "Gateway offline"
|
||||
parts = []
|
||||
if g.loss is not None:
|
||||
parts.append(f"Paketverlust {g.loss:g} %")
|
||||
if g.delay is not None:
|
||||
parts.append(f"Latenz {g.delay:g} ms")
|
||||
return ", ".join(parts) or g.status
|
||||
|
||||
|
||||
async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
|
||||
open_keys = await _open_keys(db, "gateway_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "gateways")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
parsed = [
|
||||
None if s["gateways"] is None else {g.name: g for g in gateway_health_service.from_compact(s["gateways"])}
|
||||
for s in samples
|
||||
]
|
||||
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
|
||||
for name in sorted(names):
|
||||
flags: list[bool | None] = []
|
||||
for p in parsed:
|
||||
if p is None:
|
||||
flags.append(None)
|
||||
else:
|
||||
g = p.get(name) # a gateway that no longer exists is "fine": its problem resolves
|
||||
flags.append(False if g is None else gateway_health_service.is_bad(g, loss_limit, delay_limit))
|
||||
key = f"{host.id}:{name}"
|
||||
if not verdict(flags, confirm, key in open_keys):
|
||||
continue
|
||||
latest = next((p[name] for p in parsed if p and name in p), None)
|
||||
problems[key] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "gateway": name,
|
||||
"grund": _gateway_reason(latest) if latest else "keine aktuellen Messwerte",
|
||||
"status": latest.status if latest else None,
|
||||
"paketverlust_prozent": latest.loss if latest else None,
|
||||
"latenz_ms": latest.delay if latest else None,
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Resources ───────────────────────────────────────────────────────────────
|
||||
|
||||
def disk_groups(devices) -> dict[str, tuple[str, float]]:
|
||||
"""{group: (label, used %)} — one entry per ZFS pool / UFS device, the fullest dataset of a pool wins."""
|
||||
groups: dict[str, tuple[str, float]] = {}
|
||||
for dev in devices or []:
|
||||
if not isinstance(dev, dict) or dev.get("type") not in _DISK_TYPES:
|
||||
continue
|
||||
try:
|
||||
pct = float(dev["used_pct"])
|
||||
except (KeyError, TypeError, ValueError):
|
||||
continue
|
||||
device = str(dev.get("device") or dev.get("mountpoint") or "?")
|
||||
group = device.split("/")[0] if dev["type"] == "zfs" else device
|
||||
label = group if dev["type"] == "zfs" else str(dev.get("mountpoint") or device)
|
||||
if group not in groups or pct > groups[group][1]:
|
||||
groups[group] = (label, pct)
|
||||
return groups
|
||||
|
||||
|
||||
async def check_resource_threshold(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.resource_confirm_samples)
|
||||
open_keys = await _open_keys(db, "resource_threshold")
|
||||
problems = Problems()
|
||||
|
||||
def over(value, limit) -> bool | None:
|
||||
if not limit or value is None:
|
||||
return None if value is None else False
|
||||
return value >= limit
|
||||
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "disk", "swap_used_pct", "cpu_threads")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
base = {"host_id": str(host.id), "host_name": host.name}
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
|
||||
# Disk: one problem per pool/device.
|
||||
disk = [None if s["disk"] is None else disk_groups(s["disk"]) for s in samples]
|
||||
group_names = {g for d in disk if d for g in d} | {k[5:] for k in was_open if k.startswith("disk:")}
|
||||
for group in sorted(group_names):
|
||||
flags = [None if d is None else (over(d[group][1], cfg.disk_threshold_pct) if group in d else False) for d in disk]
|
||||
if verdict(flags, confirm, f"disk:{group}" in was_open):
|
||||
label, pct = next((d[group] for d in disk if d and group in d), (group, None))
|
||||
problems[f"{host.id}:disk:{group}"] = {
|
||||
**base, "ressource": f"Festplatte {label}", "wert_prozent": pct, "schwelle_prozent": cfg.disk_threshold_pct,
|
||||
}
|
||||
|
||||
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
|
||||
if verdict(flags, confirm, key in was_open):
|
||||
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": f"{value}{unit}", "schwelle": f"{threshold}{unit}"}
|
||||
|
||||
latest = samples[0]
|
||||
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
|
||||
latest["mem_used_pct"], cfg.memory_threshold_pct, " %")
|
||||
simple("swap", "Swap", [over(s["swap_used_pct"], cfg.swap_threshold_pct) for s in samples],
|
||||
latest["swap_used_pct"], cfg.swap_threshold_pct, " %")
|
||||
|
||||
def per_cpu(s):
|
||||
if s["load_avg_1m"] is None or not s["cpu_threads"]:
|
||||
return None # thread count unknown (older firmware): the load cannot be judged
|
||||
return s["load_avg_1m"] / s["cpu_threads"]
|
||||
|
||||
simple("cpu", "CPU-Last (pro Thread)", [over(per_cpu(s), cfg.load_per_cpu_threshold) for s in samples],
|
||||
None if per_cpu(latest) is None else round(per_cpu(latest), 2), cfg.load_per_cpu_threshold, "")
|
||||
return problems
|
||||
|
||||
|
||||
# ── Services ────────────────────────────────────────────────────────────────
|
||||
|
||||
async def check_service_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
watch = set(cfg.service_watchlist if cfg.service_watchlist is not None else DEFAULT_SERVICE_WATCHLIST)
|
||||
open_keys = await _open_keys(db, "service_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm + SERVICE_EVIDENCE_SAMPLES, "services")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
listings = [s["services"] for s in samples]
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
candidates = {k for k in (listings[0] or {}) if k.split("/")[0] in watch} | was_open
|
||||
for svc in sorted(candidates):
|
||||
if svc.split("/")[0] not in watch:
|
||||
continue # taken off the watchlist: the problem resolves
|
||||
flags = [None if sv is None else (sv[svc] == 0 if svc in sv else False) for sv in listings]
|
||||
bad = verdict(flags, confirm, svc in was_open)
|
||||
# Only a service that was seen running before counts: one that never ran (enabled but idle,
|
||||
# e.g. DHCPv6 without a prefix) or was stopped on purpose long ago is not a failure.
|
||||
if bad and svc not in was_open and not any(sv and sv.get(svc) == 1 for sv in listings[confirm:]):
|
||||
bad = False
|
||||
if bad:
|
||||
problems[f"{host.id}:{svc}"] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "dienst": svc.split("/")[0],
|
||||
"hinweis": "Dienst läuft nicht mehr; absichtlich gestoppt? Dann aus der Überwachungsliste nehmen.",
|
||||
}
|
||||
return problems
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Extra notification channels: ntfy, Gotify, Telegram, Microsoft Teams, Matrix.
|
||||
|
||||
They sit beside SMTP and the generic webhook (notification_delivery.py) and reuse its message
|
||||
text. Each channel is a row in ``notification_channels`` (see models/notification.py); the
|
||||
credential never touches the database: it is stored through notification_secret_store as
|
||||
``channel_<id>`` (mode 0600 file) and read only at send time. Where the credential is part of the
|
||||
URL (a Telegram bot token, the whole Teams webhook URL) the URL is built per request, and the
|
||||
HTTP request log is muted while it is in flight (``url_guard.muted``), so neither the logs nor an
|
||||
error message ever contain it.
|
||||
|
||||
API formats, checked against the vendors' documentation:
|
||||
|
||||
* ntfy — JSON publish: POST <server>/ with {"topic", "title", "message", "priority", "tags"}
|
||||
(docs.ntfy.sh/publish, "Publish as JSON"); optional ``Authorization: Bearer <token>``.
|
||||
* Gotify — POST <server>/message, header ``X-Gotify-Key: <app token>`` (keeps the token out of
|
||||
the URL), JSON {"title", "message", "priority"} (gotify.net/docs/pushmsg).
|
||||
* Telegram — POST <api>/bot<token>/sendMessage, JSON {"chat_id", "text"}; text limit 4096
|
||||
characters; answers {"ok": true, ...} (core.telegram.org/bots/api#sendmessage).
|
||||
* Teams — POST to the (Workflows / incoming) webhook URL with a message envelope carrying an
|
||||
Adaptive Card (learn.microsoft.com ... add-incoming-webhook); the URL is the secret.
|
||||
* Matrix — PUT <homeserver>/_matrix/client/v3/rooms/<roomId>/send/m.room.message/<txnId> with
|
||||
``Authorization: Bearer <access token>``, JSON {"msgtype": "m.text", "body"}; a fresh
|
||||
transaction id per message (Matrix client-server API, "Sending events to a room").
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import CHANNEL_TYPES, NotificationChannel, NotificationProblem
|
||||
from app.services import notification_delivery, notification_secret_store, url_guard
|
||||
from app.services.notification_delivery import NotificationDeliveryError
|
||||
|
||||
TELEGRAM_LIMIT = 4096
|
||||
TEAMS_LIMIT = 20000
|
||||
_NTFY_TOPIC = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||
_MATRIX_ROOM = re.compile(r"^![^\s:]+:[^\s/]+$")
|
||||
|
||||
# type -> (required config keys, secret required?)
|
||||
_SPEC = {
|
||||
"ntfy": ({"topic"}, False),
|
||||
"gotify": ({"server_url"}, True),
|
||||
"telegram": ({"chat_id"}, True),
|
||||
"teams": (set(), True),
|
||||
"matrix": ({"homeserver_url", "room_id"}, True),
|
||||
}
|
||||
assert set(_SPEC) == set(CHANNEL_TYPES)
|
||||
|
||||
|
||||
def secret_name(channel_id) -> str:
|
||||
return f"channel_{channel_id}"
|
||||
|
||||
|
||||
def clean_config(channel_type: str, config: dict) -> dict:
|
||||
"""Validates and normalises the non-secret settings of a channel. Raises ValueError (German reason)."""
|
||||
if channel_type not in _SPEC:
|
||||
raise ValueError("Unbekannter Kanaltyp")
|
||||
config = dict(config or {})
|
||||
out: dict = {}
|
||||
|
||||
def url(key: str, default: str | None = None) -> str | None:
|
||||
raw = str(config.get(key) or "").strip() or default
|
||||
if raw is None:
|
||||
return None
|
||||
return url_guard.validate_url(raw).rstrip("/")
|
||||
|
||||
def integer(key: str, low: int, high: int, default: int) -> int:
|
||||
value = config.get(key)
|
||||
if value in (None, ""):
|
||||
return default
|
||||
try:
|
||||
number = int(value)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError(f"{key}: ganze Zahl erwartet") from None
|
||||
if not low <= number <= high:
|
||||
raise ValueError(f"{key}: Wert zwischen {low} und {high} erwartet")
|
||||
return number
|
||||
|
||||
if channel_type == "ntfy":
|
||||
out["server_url"] = url("server_url", "https://ntfy.sh")
|
||||
topic = str(config.get("topic") or "").strip()
|
||||
if not _NTFY_TOPIC.match(topic):
|
||||
raise ValueError("Thema: 1–64 Zeichen aus Buchstaben, Ziffern, _ und -")
|
||||
out["topic"] = topic
|
||||
out["priority"] = integer("priority", 1, 5, 4)
|
||||
elif channel_type == "gotify":
|
||||
out["server_url"] = url("server_url")
|
||||
if not out["server_url"]:
|
||||
raise ValueError("Serveradresse fehlt")
|
||||
out["priority"] = integer("priority", 0, 10, 5)
|
||||
elif channel_type == "telegram":
|
||||
chat_id = str(config.get("chat_id") or "").strip()
|
||||
if not re.match(r"^(-?\d{1,20}|@[A-Za-z0-9_]{4,64})$", chat_id):
|
||||
raise ValueError("Chat-ID: Zahl (z. B. -1001234567890) oder @Kanalname")
|
||||
out["chat_id"] = chat_id
|
||||
out["api_base"] = url("api_base", "https://api.telegram.org")
|
||||
elif channel_type == "matrix":
|
||||
out["homeserver_url"] = url("homeserver_url")
|
||||
if not out["homeserver_url"]:
|
||||
raise ValueError("Homeserver-Adresse fehlt")
|
||||
room = str(config.get("room_id") or "").strip()
|
||||
if not _MATRIX_ROOM.match(room):
|
||||
raise ValueError("Raum-ID: Form !abc123:example.org (nicht der Alias #raum:…)")
|
||||
out["room_id"] = room
|
||||
return out
|
||||
|
||||
|
||||
def clean_secret(channel_type: str, secret: str) -> str:
|
||||
value = (secret or "").strip()
|
||||
if not value:
|
||||
raise ValueError("Zugangsdaten fehlen")
|
||||
if channel_type == "teams":
|
||||
value = url_guard.validate_url(value, max_length=2000)
|
||||
if not value.startswith("https://"):
|
||||
raise ValueError("Die Teams-Webhook-Adresse muss mit https:// beginnen")
|
||||
elif any(c.isspace() for c in value) or len(value) > 500:
|
||||
raise ValueError("Token: keine Leerzeichen, höchstens 500 Zeichen")
|
||||
return value
|
||||
|
||||
|
||||
def missing_requirements(channel: NotificationChannel) -> str | None:
|
||||
required, needs_secret = _SPEC[channel.channel_type]
|
||||
if any(not channel.config.get(k) for k in required):
|
||||
return "Konfiguration unvollständig"
|
||||
if needs_secret and not channel.secret_set:
|
||||
return "Zugangsdaten fehlen"
|
||||
return None
|
||||
|
||||
|
||||
# ── Message formatting (one function per channel, pure — unit-testable) ───────────
|
||||
|
||||
def _parts(problem: NotificationProblem, transition: str) -> tuple[str, str]:
|
||||
"""(title, details) — the title repeated at the top of render_message's body is dropped."""
|
||||
subject, body = notification_delivery.render_message(problem, transition)
|
||||
details = body.split("\n", 2)[2] if body.count("\n") >= 2 else ""
|
||||
return subject, details.strip()
|
||||
|
||||
|
||||
def _clip(text: str, limit: int) -> str:
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def format_ntfy(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
resolved = transition == "resolved"
|
||||
return {
|
||||
"topic": config["topic"], "title": title, "message": details or title,
|
||||
"priority": 3 if resolved else int(config.get("priority", 4)),
|
||||
"tags": ["white_check_mark"] if resolved else ["warning"],
|
||||
}
|
||||
|
||||
|
||||
def format_gotify(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {"title": title, "message": details or title, "priority": 1 if transition == "resolved" else int(config.get("priority", 5))}
|
||||
|
||||
|
||||
def format_telegram(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {
|
||||
"chat_id": config["chat_id"], "text": _clip(f"{title}\n\n{details}".strip(), TELEGRAM_LIMIT),
|
||||
"disable_web_page_preview": True,
|
||||
}
|
||||
|
||||
|
||||
def format_teams(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
body: list[dict] = [{"type": "TextBlock", "text": title, "weight": "Bolder", "size": "Medium", "wrap": True}]
|
||||
if details:
|
||||
body.append({"type": "TextBlock", "text": _clip(details, TEAMS_LIMIT), "wrap": True})
|
||||
return {
|
||||
"type": "message",
|
||||
"attachments": [{
|
||||
"contentType": "application/vnd.microsoft.card.adaptive", "contentUrl": None,
|
||||
"content": {
|
||||
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||
"type": "AdaptiveCard", "version": "1.4", "body": body,
|
||||
},
|
||||
}],
|
||||
}
|
||||
|
||||
|
||||
def format_matrix(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {"msgtype": "m.text", "body": f"{title}\n\n{details}".strip()}
|
||||
|
||||
|
||||
# ── Sending ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _request_for(channel: NotificationChannel, secret: str, problem, transition: str) -> tuple[str, str, dict, dict, str | None]:
|
||||
"""(method, url, headers, json body, expected "ok" check key)."""
|
||||
cfg, kind = channel.config, channel.channel_type
|
||||
if kind == "ntfy":
|
||||
headers = {"Authorization": f"Bearer {secret}"} if secret else {}
|
||||
return "POST", cfg["server_url"].rstrip("/") + "/", headers, format_ntfy(cfg, problem, transition), None
|
||||
if kind == "gotify":
|
||||
return "POST", cfg["server_url"].rstrip("/") + "/message", {"X-Gotify-Key": secret}, format_gotify(cfg, problem, transition), None
|
||||
if kind == "telegram":
|
||||
return "POST", f"{cfg['api_base'].rstrip('/')}/bot{secret}/sendMessage", {}, format_telegram(cfg, problem, transition), "ok"
|
||||
if kind == "teams":
|
||||
return "POST", secret, {}, format_teams(cfg, problem, transition), None
|
||||
room = quote(cfg["room_id"], safe="")
|
||||
url = f"{cfg['homeserver_url'].rstrip('/')}/_matrix/client/v3/rooms/{room}/send/m.room.message/{uuid.uuid4().hex}"
|
||||
return "PUT", url, {"Authorization": f"Bearer {secret}"}, format_matrix(cfg, problem, transition), None
|
||||
|
||||
|
||||
async def send(channel: NotificationChannel, problem: NotificationProblem, transition: str) -> None:
|
||||
problem_missing = missing_requirements(channel)
|
||||
if problem_missing:
|
||||
raise NotificationDeliveryError(f"Kanal „{channel.name}“: {problem_missing}")
|
||||
secret = notification_secret_store.read_secret(secret_name(channel.id)) or ""
|
||||
if channel.secret_set and not secret:
|
||||
raise NotificationDeliveryError(f"Kanal „{channel.name}“: Zugangsdaten nicht lesbar – bitte neu setzen")
|
||||
method, url, headers, payload, ok_key = _request_for(channel, secret, problem, transition)
|
||||
label = f"Kanal „{channel.name}“ ({channel.channel_type})"
|
||||
with url_guard.muted():
|
||||
try:
|
||||
url_guard.validate_url(url, max_length=2100)
|
||||
await url_guard.ensure_resolved_safe(url)
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
resp = await client.request(method, url, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
if ok_key and not (resp.json() or {}).get(ok_key):
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
|
||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
|
||||
except ValueError:
|
||||
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
|
||||
|
||||
|
||||
async def send_test(channel: NotificationChannel) -> None:
|
||||
fake = NotificationProblem(
|
||||
problem_type="test", dedup_key="test",
|
||||
context={"message": "Dies ist eine Testbenachrichtigung von OPNexus."},
|
||||
)
|
||||
await send(channel, fake, "opened")
|
||||
|
||||
|
||||
async def enabled_channels(db: AsyncSession) -> list[NotificationChannel]:
|
||||
rows = await db.scalars(select(NotificationChannel).where(NotificationChannel.enabled.is_(True)).order_by(NotificationChannel.name))
|
||||
return list(rows.all())
|
||||
@@ -363,3 +363,49 @@ async def check_compliance(db: AsyncSession) -> dict[str, dict]:
|
||||
"beispiele": ", ".join(params.get("examples", [])) or None,
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Rollouts (1.25) ─────────────────────────────────────────────────────────
|
||||
|
||||
def _short(detail) -> str | None:
|
||||
if not detail:
|
||||
return None
|
||||
parts = []
|
||||
for key, value in detail.items():
|
||||
if isinstance(value, list):
|
||||
value = ", ".join(str(v) for v in value[:8])
|
||||
parts.append(f"{key}: {value}")
|
||||
return "; ".join(parts)[:500]
|
||||
|
||||
|
||||
async def check_rollout_paused(db: AsyncSession) -> dict[str, dict]:
|
||||
"""Open while a rollout run stands still waiting for a human (gate, failed host, window gone...)."""
|
||||
from app.models.rollout import RolloutRun
|
||||
problems: dict[str, dict] = {}
|
||||
for run in (await db.scalars(select(RolloutRun).where(RolloutRun.status == "paused"))).all():
|
||||
wave = next((w.label for w in run.waves if w.index == run.current_wave), None)
|
||||
problems[str(run.id)] = {
|
||||
"run_id": str(run.id), "plan": run.plan_name, "grund": run.pause_reason, "welle": wave,
|
||||
"zielversion": run.target_version, "details": _short(run.pause_detail),
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
ROLLOUT_START_FAILURE_WINDOW = timedelta(days=3)
|
||||
|
||||
|
||||
async def check_rollout_start_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"""An automatic start at a window occurrence did not happen (event: closed without a 'resolved' notice)."""
|
||||
from app.models.rollout import RolloutPlan
|
||||
since = datetime.now(timezone.utc) - ROLLOUT_START_FAILURE_WINDOW
|
||||
problems: dict[str, dict] = {}
|
||||
for plan in (await db.scalars(select(RolloutPlan).where(RolloutPlan.auto_start_error.is_not(None)))).all():
|
||||
error = plan.auto_start_error or {}
|
||||
at = datetime.fromisoformat(error["at"]) if error.get("at") else None
|
||||
if at is None or at < since:
|
||||
continue
|
||||
problems[f"{plan.id}:{error.get('occurrence')}"] = {
|
||||
"plan_id": str(plan.id), "plan": plan.name, "grund": error.get("code"), "termin": error.get("occurrence"),
|
||||
"details": _short(error.get("params")),
|
||||
}
|
||||
return problems
|
||||
@@ -16,7 +16,7 @@ import httpx
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
|
||||
|
||||
class NotificationDeliveryError(Exception):
|
||||
@@ -37,6 +37,12 @@ _LABELS = {
|
||||
"backup_stale": "Kein aktuelles Backup",
|
||||
"backup_offsite_failed": "Offsite-Backup fehlgeschlagen",
|
||||
"compliance_finding": "Sicherheitsbefund",
|
||||
"gateway_down": "Gateway gestört",
|
||||
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
||||
"service_down": "Dienst ausgefallen",
|
||||
"rollout_paused": "Rollout angehalten",
|
||||
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
||||
"test": "Testbenachrichtigung",
|
||||
}
|
||||
|
||||
|
||||
@@ -84,9 +90,16 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
|
||||
headers["X-OPNexus-Signature"] = f"sha256={signature}"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout) as client:
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
url_guard.validate_url(settings_row.webhook_url)
|
||||
await url_guard.ensure_resolved_safe(settings_row.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
||||
try:
|
||||
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
with url_guard.muted():
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which often is the credential itself
|
||||
# (token in path/query) — keep it out of messages and logs.
|
||||
|
||||
@@ -8,8 +8,8 @@ from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_checks, notification_dedup, notification_delivery
|
||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||
from app.services import monitor_checks, notification_channels, notification_checks, notification_dedup, notification_delivery
|
||||
from app.services import notification_settings_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -34,14 +34,21 @@ _CHECKS = [
|
||||
("backup_stale", "backup_failed_enabled", notification_checks.check_backup_stale),
|
||||
("backup_offsite_failed", "backup_failed_enabled", notification_checks.check_backup_offsite),
|
||||
("compliance_finding", "compliance_alerts_enabled", notification_checks.check_compliance),
|
||||
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
|
||||
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
||||
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
||||
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
||||
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
||||
]
|
||||
|
||||
# Event-like problems: when they drop out of the window nothing was "fixed", so no
|
||||
# "behoben" message goes out for them.
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed"}
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
|
||||
|
||||
|
||||
async def _deliver(settings_row: NotificationSettings, problem: NotificationProblem, transition: str) -> None:
|
||||
async def _deliver(
|
||||
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
|
||||
) -> None:
|
||||
"""Best-effort, not an audit trail: attempted once per transition and
|
||||
never retried, even on failure — avoids retry-storming a broken webhook
|
||||
URL or SMTP relay every poll cycle."""
|
||||
@@ -60,6 +67,13 @@ async def _deliver(settings_row: NotificationSettings, problem: NotificationProb
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("Email delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
||||
|
||||
for channel in channels:
|
||||
try:
|
||||
await notification_channels.send(channel, problem, transition)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
# The error text names the channel and the HTTP status, never the URL or token.
|
||||
logger.exception("Channel %s (%s) delivery failed for %s/%s", channel.name, channel.channel_type, problem.problem_type, problem.dedup_key)
|
||||
|
||||
if transition == "resolved":
|
||||
problem.resolved_notified_at = _now()
|
||||
else:
|
||||
@@ -69,6 +83,7 @@ async def _deliver(settings_row: NotificationSettings, problem: NotificationProb
|
||||
async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
settings_row = await notification_settings_service.get_settings(db)
|
||||
result = {"opened": 0, "resolved": 0}
|
||||
channels = await notification_channels.enabled_channels(db)
|
||||
|
||||
for problem_type, enabled_flag, check_fn in _CHECKS:
|
||||
if not getattr(settings_row, enabled_flag):
|
||||
@@ -85,9 +100,9 @@ async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
|
||||
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
|
||||
for problem in opened:
|
||||
await _deliver(settings_row, problem, "opened")
|
||||
await _deliver(settings_row, problem, "opened", channels)
|
||||
for problem in resolved:
|
||||
await _deliver(settings_row, problem, "resolved")
|
||||
await _deliver(settings_row, problem, "resolved", channels)
|
||||
result["opened"] += len(opened)
|
||||
result["resolved"] += len(resolved)
|
||||
|
||||
|
||||
@@ -26,6 +26,10 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
|
||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
|
||||
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
||||
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled",
|
||||
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
||||
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
||||
"service_watchlist",
|
||||
"poll_interval_seconds",
|
||||
]
|
||||
for field in simple_fields:
|
||||
|
||||
@@ -0,0 +1,627 @@
|
||||
"""Takes over aliases, port forwards, 1:1 NAT and outbound NAT rules that were created by hand on a
|
||||
firewall, so they can be edited, versioned and deployed from OPNexus like any other object. This is
|
||||
the generalisation of rule_adopt_service.py (filter rules keep their own module: their model has
|
||||
more special cases and the code is relied upon).
|
||||
|
||||
Adoption itself writes NOTHING to the firewall. It records the object as it is right now (read fresh
|
||||
from the host at adoption time, never from what the browser sent) together with its OPNsense UUID
|
||||
(`remote_uuid` = the deployment's opnsense_*_uuid), and stores the payload OPNexus would send for it as
|
||||
the deployment's last_payload. So the preview directly afterwards says "unchanged", a later deploy
|
||||
updates the same object via setItem/setRule (no duplicate), and the normal preview → push →
|
||||
confirm/auto-revert flow applies from the first change on.
|
||||
|
||||
An object that already exists centrally for another firewall is LINKED instead of adopted a second time
|
||||
(adopt_link_service.py: exact match after resolving the host's variables, never automatic for a mere
|
||||
resemblance).
|
||||
|
||||
Only objects OPNexus can describe without distortion are adopted. Anything the model cannot express
|
||||
is refused with machine-readable reasons (the UI translates them) instead of being imported in a way
|
||||
that would silently change its meaning on the next edit. Fields OPNexus does not manage (log flag of a
|
||||
port forward, pool options, tags, categories, ...) are never sent, and OPNsense's set*/setItem only
|
||||
overwrites the posted fields (ApiMutableModelControllerBase::setBase -> setNodes), so they stay as
|
||||
they are; the candidate list shows them as `kept`.
|
||||
|
||||
Evidence (OPNsense core master, src/opnsense/mvc/app/…, read 2026-10-20, and live against OPNsense
|
||||
26.7 on test-fw-3 on the same day with hand-made objects of every case below):
|
||||
|
||||
* Aliases — controllers/OPNsense/Firewall/Api/AliasController.php::searchItemAction -> searchBase
|
||||
("aliases.alias"), model models/OPNsense/Firewall/Alias.xml. Row keys used: uuid, name, type
|
||||
("host", "network", "port", "networkgroup", "mac", "asn", plus the unsupported "url", "urltable",
|
||||
"urljson", "geoip", "dynipv6host", "authgroup", and the system types "internal"/"external"),
|
||||
enabled "1"/"0", content (entries separated by "\n", live-verified; for "authgroup" the controller
|
||||
rewrites it to group names), description, proto (only meaningful for asn/geoip). The built-in rows
|
||||
are bogons, bogonsv6, sshlockout, virusprot (type external) and one __<if>_network per interface
|
||||
(type internal); they are OPNsense's own and never adoptable. A "networkgroup" lists alias NAMES;
|
||||
host/network/port aliases may also contain alias names (nesting).
|
||||
* Port forward — controllers/OPNsense/Firewall/Api/DNatController.php::searchRuleAction merges the
|
||||
configured rules (getFlatNodes: dotted keys "source.network", "source.port", "source.not",
|
||||
"destination.network", "destination.port", "destination.not", plus descr, disabled, interface,
|
||||
ipprotocol, protocol, target, local-port, natreflection, nordr, pass, log, poolopts, tag, tagged,
|
||||
nosync, categories, sequence) with synthetic automatic rows (uuid "automatic_*"/"lockout_*",
|
||||
is_automatic true) that setRule/delRule cannot target. models/OPNsense/Firewall/DNat.xml: interface is
|
||||
Multiple=Y (comma separated), ipprotocol may be blank/inet/inet6/inet46, protocol blank = any,
|
||||
`nordr` ("no redirect") requires an empty target/local-port (live: "Target must be empty when No
|
||||
RDR is set"), `pass` ("" manual / "pass" / "rule") links an associated filter rule, natreflection is
|
||||
""/purenat/disable. OPNexus sends pass "" and not "0", so a rule using one of those would change
|
||||
meaning on its first edit and is refused.
|
||||
* 1:1 NAT — OneToOneController::searchRuleAction (flat keys), model Filter.xml onetoone.rule:
|
||||
enabled, log, interface, type (binat/nat), source_net/source_not, destination_net/destination_not,
|
||||
external, natreflection ""/enable/disable, description. All of these are managed by OPNexus.
|
||||
* Outbound NAT — SourceNatController::searchRuleAction returns the configured rules ONLY when the
|
||||
global mode (filter.settings.nat.snat_mode) is hybrid or advanced; in automatic/disabled mode the
|
||||
manual rules do not show up at all, so adopting is refused up front (outbound_nat_mode.py) instead
|
||||
of presenting an empty list. Model Filter.xml snatrules.rule: `nonat` rules ("do not NAT") are not
|
||||
expressible and refused; staticnatport, poolopts, log, tag, tagged, nosync, endpoint-independent
|
||||
are left alone.
|
||||
"""
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Awaitable, Callable, get_args
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.one_to_one_deployment import OneToOneDeployment
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.port_forward_deployment import PortForwardDeployment
|
||||
from app.schemas.alias import AliasType
|
||||
from app.services import alias_deploy_service, alias_policy_service
|
||||
from app.services import one_to_one_deploy_service, one_to_one_policy_service
|
||||
from app.services import opnsense_client, outbound_nat_deploy_service, outbound_nat_policy_service
|
||||
from app.services import port_forward_deploy_service, port_forward_policy_service
|
||||
from app.services import adopt_link_service
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on
|
||||
from app.services.nat_live_service import split_rows
|
||||
from app.services.outbound_nat_mode import host_mode
|
||||
|
||||
ADOPT_NOTE = "Von Firewall übernommen"
|
||||
ADOPT_MESSAGE = "Von der Firewall übernommen"
|
||||
|
||||
ALIAS_NAME = re.compile(r"^[A-Za-z0-9_]+$") # schemas/alias.py AliasCreate.name
|
||||
ALIAS_TYPES = frozenset(get_args(AliasType))
|
||||
ALIAS_CONTENT_MAX = 8000 # schemas/alias.py AliasCreate.content
|
||||
# OPNsense's own aliases that are not user objects (Firewall/Alias.php: bogons & co. are `external`
|
||||
# rows created by the system, __<if>_network are `internal`).
|
||||
SYSTEM_ALIAS_NAMES = frozenset({"bogons", "bogonsv6", "sshlockout", "virusprot"})
|
||||
LINK_TOLERATED = frozenset({"name_exists", "references_unmanaged_alias"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entry:
|
||||
row: dict
|
||||
reasons: list[str] = field(default_factory=list)
|
||||
details: dict[str, str] = field(default_factory=dict)
|
||||
bulk: bool = False # not adoptable alone yet, but adoptable in the same run (its references come first)
|
||||
matches: list = field(default_factory=list) # central objects equal to the live object (link candidates)
|
||||
similar: list = field(default_factory=list) # same name/description but different content (hint only)
|
||||
|
||||
@property
|
||||
def linkable(self) -> bool:
|
||||
"""An exact central twin exists and nothing but "the name is taken" / "its references are not
|
||||
managed here yet" speaks against it (both are moot when the twin is linked, its content is equal)."""
|
||||
return any(not m.busy for m in self.matches) and set(self.reasons) <= LINK_TOLERATED
|
||||
|
||||
@property
|
||||
def busy_twin(self) -> bool:
|
||||
"""The one exact twin waits for a confirmation: \"adopt all\" leaves this entry alone (neither link nor duplicate)."""
|
||||
return len(self.matches) == 1 and self.matches[0].busy
|
||||
|
||||
@property
|
||||
def uuid(self) -> str:
|
||||
return clean(self.row.get("uuid"))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
key: str
|
||||
model: type
|
||||
deployment: type
|
||||
fk: str # deployment column pointing at the central object
|
||||
uuid_attr: str # deployment column holding the OPNsense UUID (= remote_uuid)
|
||||
fetch: Callable[[OpnsenseHost], Awaitable[list[dict]]]
|
||||
check: Callable[[dict, dict], tuple[list[str], dict[str, str]]]
|
||||
title: Callable[[dict], str]
|
||||
info: Callable[[dict], dict[str, str]]
|
||||
build: Callable[[dict], dict]
|
||||
payload: Callable[[Any], dict]
|
||||
revision: Callable
|
||||
kept: Callable[[dict], list[str]] = lambda row: []
|
||||
context: Callable[[AsyncSession, list[dict]], Awaitable[dict]] | None = None
|
||||
refs: Callable[[dict], set[str]] | None = None
|
||||
name: Callable[[dict], str] | None = None
|
||||
identity: str = "description" # attribute that names the object (the alias name for aliases)
|
||||
|
||||
@property
|
||||
def link_spec(self) -> adopt_link_service.LinkSpec:
|
||||
return adopt_link_service.LinkSpec(
|
||||
model=self.model, deployment=self.deployment, fk=self.fk, uuid_attr=self.uuid_attr,
|
||||
identity=self.identity, payload=self.payload, revision=self.revision,
|
||||
)
|
||||
|
||||
|
||||
# ── shared helpers ───────────────────────────────────────────────────────
|
||||
|
||||
def _single_interface(row: dict) -> bool:
|
||||
interface = clean(row.get("interface"))
|
||||
return bool(interface) and "," not in interface
|
||||
|
||||
|
||||
def _too_long(kind_model: type, values: dict) -> list[str]:
|
||||
"""Columns are bounded; a live value that does not fit would fail (or be cut) on the first edit."""
|
||||
columns = kind_model.__table__.columns
|
||||
return [name for name, value in values.items()
|
||||
if name in columns and getattr(columns[name].type, "length", None)
|
||||
and isinstance(value, str) and len(value) > columns[name].type.length]
|
||||
|
||||
|
||||
def _port(value) -> str | None:
|
||||
return clean(value) or None
|
||||
|
||||
|
||||
def _kept(row: dict, keys: tuple[str, ...]) -> list[str]:
|
||||
return [key for key in keys if clean(row.get(key)) not in ("", "0")]
|
||||
|
||||
|
||||
def _fetch_nat(search) -> Callable[[OpnsenseHost], Awaitable[list[dict]]]:
|
||||
async def fetch(host: OpnsenseHost) -> list[dict]:
|
||||
data = await search(host)
|
||||
# split_rows drops automatic_*/lockout_* and is_automatic rows: they are OPNsense's own
|
||||
return [r for r in split_rows(data)["rules"] if clean(r.get("uuid"))]
|
||||
return fetch
|
||||
|
||||
|
||||
# ── aliases ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _alias_content(row: dict) -> str:
|
||||
# same normalisation as AliasCreate.clean_content
|
||||
return "\n".join(line.strip() for line in str(row.get("content") or "").splitlines() if line.strip())
|
||||
|
||||
|
||||
def _alias_is_system(row: dict) -> bool:
|
||||
name = clean(row.get("name"))
|
||||
return clean(row.get("type")) == "internal" or name.startswith("__") or name in SYSTEM_ALIAS_NAMES
|
||||
|
||||
|
||||
async def _fetch_aliases(host: OpnsenseHost) -> list[dict]:
|
||||
data = await opnsense_client.search_aliases(host)
|
||||
rows = data.get("rows", []) if isinstance(data, dict) else []
|
||||
return [r for r in rows if isinstance(r, dict) and clean(r.get("uuid")) and clean(r.get("name"))]
|
||||
|
||||
|
||||
async def _alias_context(db: AsyncSession, rows: list[dict]) -> dict:
|
||||
return {"central_names": set((await db.scalars(select(FirewallAlias.name))).all())}
|
||||
|
||||
|
||||
def _alias_refs(row: dict) -> set[str]:
|
||||
return {line.lstrip("!").strip() for line in _alias_content(row).splitlines()} - {""}
|
||||
|
||||
|
||||
def _alias_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
if _alias_is_system(row):
|
||||
return ["system"], {}
|
||||
reasons: list[str] = []
|
||||
details: dict[str, str] = {}
|
||||
name, type_ = clean(row.get("name")), clean(row.get("type"))
|
||||
if type_ not in ALIAS_TYPES:
|
||||
reasons.append("unsupported_type")
|
||||
details["unsupported_type"] = clean(row.get("%type")) or type_
|
||||
if not ALIAS_NAME.fullmatch(name) or len(name) > 64:
|
||||
reasons.append("name_invalid")
|
||||
if name in ctx["central_names"]:
|
||||
reasons.append("name_exists")
|
||||
if len(_alias_content(row)) > ALIAS_CONTENT_MAX:
|
||||
reasons.append("content_too_large")
|
||||
if type_ == "asn" and clean(row.get("proto")):
|
||||
reasons.append("proto_set") # OPNexus has no field for it: a deploy elsewhere would drop the restriction
|
||||
if _too_long(FirewallAlias, {"description": clean(row.get("description"))}):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, details
|
||||
|
||||
|
||||
def _alias_info(row: dict) -> dict[str, str]:
|
||||
lines = _alias_content(row).splitlines()
|
||||
return {
|
||||
"type": clean(row.get("type")),
|
||||
"type_label": clean(row.get("%type")) or clean(row.get("type")),
|
||||
"description": clean(row.get("description")),
|
||||
"content": "\n".join(lines[:6]) + ("\n…" if len(lines) > 6 else ""),
|
||||
"entries": str(len(lines)),
|
||||
}
|
||||
|
||||
|
||||
def _alias_build(row: dict) -> dict:
|
||||
return {
|
||||
"name": clean(row.get("name")),
|
||||
"type": clean(row.get("type")),
|
||||
"content": _alias_content(row),
|
||||
"description": clean(row.get("description")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
}
|
||||
|
||||
|
||||
# ── port forward ─────────────────────────────────────────────────────────
|
||||
|
||||
def _pf_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
|
||||
reasons.append("ipprotocol_unsupported") # blank / inet46 have no counterpart in OPNexus
|
||||
if not clean(row.get("protocol")):
|
||||
reasons.append("protocol_any")
|
||||
if on(row.get("nordr")):
|
||||
reasons.append("no_redirect")
|
||||
elif not clean(row.get("target")):
|
||||
reasons.append("no_target")
|
||||
if clean(row.get("pass")):
|
||||
reasons.append("filter_rule_linked") # OPNexus sends pass "" (manual) and would cut the link
|
||||
if on(row.get("source.not")):
|
||||
reasons.append("negated_source")
|
||||
if on(row.get("destination.not")):
|
||||
reasons.append("negated_destination")
|
||||
if clean(row.get("natreflection")) not in ("", "purenat", "disable"):
|
||||
reasons.append("natreflection_unsupported")
|
||||
if not clean(row.get("descr")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(PortForwardRule, _pf_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _pf_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("descr")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"ipprotocol": clean(row.get("ipprotocol")),
|
||||
"protocol": clean(row.get("protocol")),
|
||||
"source_net": clean(row.get("source.network")) or "any",
|
||||
"source_port": _port(row.get("source.port")),
|
||||
"destination_net": clean(row.get("destination.network")) or "any",
|
||||
"destination_port": _port(row.get("destination.port")),
|
||||
"target": clean(row.get("target")),
|
||||
"local_port": _port(row.get("local-port")),
|
||||
"natreflection": clean(row.get("natreflection")),
|
||||
"enabled": not on(row.get("disabled")),
|
||||
}
|
||||
|
||||
|
||||
def _pf_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"protocol": clean(row.get("%protocol")) or clean(row.get("protocol")) or "any",
|
||||
"source": clean(row.get("source.network")) or "any",
|
||||
"source_port": clean(row.get("source.port")),
|
||||
"destination": clean(row.get("destination.network")) or "any",
|
||||
"destination_port": clean(row.get("destination.port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": clean(row.get("local-port")),
|
||||
}
|
||||
|
||||
|
||||
# ── 1:1 NAT ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _o2o_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if clean(row.get("type")) not in ("binat", "nat"):
|
||||
reasons.append("type_unsupported")
|
||||
if clean(row.get("natreflection")) not in ("", "enable", "disable"):
|
||||
reasons.append("natreflection_unsupported")
|
||||
if not clean(row.get("source_net")) or not clean(row.get("external")):
|
||||
reasons.append("incomplete")
|
||||
if not clean(row.get("description")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(OneToOneRule, _o2o_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _o2o_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("description")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"source_net": clean(row.get("source_net")),
|
||||
"destination_net": clean(row.get("destination_net")) or "any",
|
||||
"source_not": on(row.get("source_not")),
|
||||
"destination_not": on(row.get("destination_not")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
"type": clean(row.get("type")),
|
||||
"external": clean(row.get("external")),
|
||||
"natreflection": clean(row.get("natreflection")),
|
||||
"log": on(row.get("log")),
|
||||
}
|
||||
|
||||
|
||||
def _o2o_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"type": clean(row.get("type")),
|
||||
"source": ("! " if on(row.get("source_not")) else "") + clean(row.get("source_net")),
|
||||
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
|
||||
"external": clean(row.get("external")),
|
||||
}
|
||||
|
||||
|
||||
# ── outbound NAT ─────────────────────────────────────────────────────────
|
||||
|
||||
async def _fetch_outbound(host: OpnsenseHost) -> list[dict]:
|
||||
# In automatic/disabled mode OPNsense does not list manual rules at all (see module docstring):
|
||||
# say so instead of showing an empty list.
|
||||
state = await host_mode(host)
|
||||
if not state["can_deploy"]:
|
||||
raise api_error(409, ErrorCode.OUTBOUND_MODE_BLOCKS_ADOPT, host=host.name, mode=state["mode"])
|
||||
return await _fetch_nat(opnsense_client.search_outbound_nat_rules)(host)
|
||||
|
||||
|
||||
def _snat_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if on(row.get("nonat")):
|
||||
reasons.append("no_nat_rule")
|
||||
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
|
||||
reasons.append("ipprotocol_unsupported")
|
||||
if not clean(row.get("protocol")):
|
||||
reasons.append("protocol_missing")
|
||||
if not clean(row.get("description")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(OutboundNatRule, _snat_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _snat_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("description")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"source_net": clean(row.get("source_net")) or "any",
|
||||
"destination_net": clean(row.get("destination_net")) or "any",
|
||||
"source_not": on(row.get("source_not")),
|
||||
"destination_not": on(row.get("destination_not")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
"ipprotocol": clean(row.get("ipprotocol")),
|
||||
"protocol": clean(row.get("protocol")),
|
||||
"source_port": _port(row.get("source_port")),
|
||||
"destination_port": _port(row.get("destination_port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": _port(row.get("target_port")),
|
||||
}
|
||||
|
||||
|
||||
def _snat_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"protocol": clean(row.get("protocol")) or "any",
|
||||
"source": ("! " if on(row.get("source_not")) else "") + (clean(row.get("source_net")) or "any"),
|
||||
"source_port": clean(row.get("source_port")),
|
||||
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
|
||||
"destination_port": clean(row.get("destination_port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": clean(row.get("target_port")),
|
||||
}
|
||||
|
||||
|
||||
KINDS: dict[str, Kind] = {
|
||||
"alias": Kind(
|
||||
key="alias", model=FirewallAlias, deployment=AliasDeployment, fk="alias_id",
|
||||
uuid_attr="opnsense_alias_uuid", fetch=_fetch_aliases, check=_alias_check,
|
||||
title=lambda row: clean(row.get("name")), info=_alias_info, build=_alias_build,
|
||||
payload=alias_deploy_service._build_payload, revision=alias_policy_service.revision,
|
||||
context=_alias_context, refs=_alias_refs, name=lambda row: clean(row.get("name")), identity="name",
|
||||
),
|
||||
"port_forward": Kind(
|
||||
key="port_forward", model=PortForwardRule, deployment=PortForwardDeployment, fk="port_forward_id",
|
||||
uuid_attr="opnsense_port_forward_uuid", fetch=_fetch_nat(opnsense_client.search_port_forward_rules),
|
||||
check=_pf_check, title=lambda row: clean(row.get("descr")), info=_pf_info, build=_pf_build,
|
||||
payload=port_forward_deploy_service._build_payload, revision=port_forward_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("log", "poolopts", "tag", "tagged", "nosync", "categories")),
|
||||
),
|
||||
"one_to_one": Kind(
|
||||
key="one_to_one", model=OneToOneRule, deployment=OneToOneDeployment, fk="one_to_one_id",
|
||||
uuid_attr="opnsense_one_to_one_uuid", fetch=_fetch_nat(opnsense_client.search_one_to_one_rules),
|
||||
check=_o2o_check, title=lambda row: clean(row.get("description")), info=_o2o_info, build=_o2o_build,
|
||||
payload=one_to_one_deploy_service._build_payload, revision=one_to_one_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("categories",)),
|
||||
),
|
||||
"outbound_nat": Kind(
|
||||
key="outbound_nat", model=OutboundNatRule, deployment=OutboundNatDeployment, fk="outbound_nat_id",
|
||||
uuid_attr="opnsense_outbound_nat_uuid", fetch=_fetch_outbound,
|
||||
check=_snat_check, title=lambda row: clean(row.get("description")), info=_snat_info, build=_snat_build,
|
||||
payload=outbound_nat_deploy_service._build_payload, revision=outbound_nat_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("log", "staticnatport", "poolopts", "poolopts_sourcehashkey", "tag", "tagged",
|
||||
"nosync", "endpoint-independent", "categories")),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ── engine ───────────────────────────────────────────────────────────────
|
||||
|
||||
async def _managed_uuids(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> set[str]:
|
||||
column = getattr(kind.deployment, kind.uuid_attr)
|
||||
rows = await db.scalars(select(column).where(kind.deployment.host_id == host.id, column.is_not(None)))
|
||||
return set(rows.all())
|
||||
|
||||
|
||||
def _resolve_references(kind: Kind, entries: list[Entry], rows: list[dict], managed: set[str], ctx: dict) -> list[Entry]:
|
||||
"""Aliases may refer to other aliases by name. A reference to a live alias OPNexus does not manage
|
||||
(and that is not OPNsense's own) would be dangling in the central model, so the referring alias
|
||||
waits until its target is managed. Returns the entries in an order in which they can be adopted
|
||||
in ONE run (dependencies first); entries whose references can never be satisfied (target is a
|
||||
url alias, a user-defined external alias, ...) keep the reason `references_unmanaged_alias`."""
|
||||
live = {kind.name(r): r for r in rows}
|
||||
available = set(ctx["central_names"])
|
||||
available |= {n for n, r in live.items() if _alias_is_system(r) or clean(r.get("uuid")) in managed}
|
||||
|
||||
candidates = [e for e in entries if not e.reasons]
|
||||
for entry in candidates:
|
||||
missing = {ref for ref in kind.refs(entry.row) if ref in live and ref not in available}
|
||||
if missing:
|
||||
entry.reasons.append("references_unmanaged_alias")
|
||||
entry.details["references_unmanaged_alias"] = ", ".join(sorted(missing))
|
||||
|
||||
order: list[Entry] = []
|
||||
ready = set(available)
|
||||
progress = True
|
||||
while progress:
|
||||
progress = False
|
||||
for entry in candidates:
|
||||
if entry in order:
|
||||
continue
|
||||
if all(ref not in live or ref in ready for ref in kind.refs(entry.row)):
|
||||
order.append(entry)
|
||||
ready.add(kind.name(entry.row))
|
||||
progress = True
|
||||
for entry in order:
|
||||
entry.bulk = "references_unmanaged_alias" in entry.reasons
|
||||
return order
|
||||
|
||||
|
||||
async def _plan(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> tuple[list[Entry], list[Entry]]:
|
||||
"""(all unmanaged entries with their reasons, the ones adoptable in one run in a safe order)."""
|
||||
rows = await kind.fetch(host)
|
||||
managed = await _managed_uuids(db, kind, host)
|
||||
ctx = await kind.context(db, rows) if kind.context else {}
|
||||
central = await adopt_link_service.load_central(db, kind.link_spec)
|
||||
entries = []
|
||||
for row in rows:
|
||||
if clean(row.get("uuid")) in managed:
|
||||
continue
|
||||
reasons, details = kind.check(row, ctx)
|
||||
entry = Entry(row, list(reasons), dict(details))
|
||||
if reasons != ["system"]:
|
||||
entry.matches, entry.similar = adopt_link_service.find(central, kind.link_spec, host, kind.build(row))
|
||||
entries.append(entry)
|
||||
if kind.refs:
|
||||
order = _resolve_references(kind, entries, rows, managed, ctx)
|
||||
else:
|
||||
order = [e for e in entries if not e.reasons]
|
||||
return entries, order
|
||||
|
||||
|
||||
def _bulk_actions(entries: list[Entry], order: list[Entry], link_existing: bool) -> dict[str, tuple[str, Any]]:
|
||||
"""What "adopt all" does with each entry: ("link", central object) when exactly one central twin exists
|
||||
(and no other entry of this run took it), ("new", None), or nothing. One function for the dialog's
|
||||
summary and for the real run, so they cannot disagree. Several twins are never guessed between."""
|
||||
actions: dict[str, tuple[str, Any]] = {}
|
||||
taken: set = set()
|
||||
if link_existing:
|
||||
for entry in entries:
|
||||
if entry.linkable and len(entry.matches) == 1 and entry.matches[0].obj.id not in taken:
|
||||
actions[entry.uuid] = ("link", entry.matches[0].obj)
|
||||
taken.add(entry.matches[0].obj.id)
|
||||
for entry in order:
|
||||
if link_existing and entry.busy_twin:
|
||||
continue
|
||||
actions.setdefault(entry.uuid, ("new", None))
|
||||
return actions
|
||||
|
||||
|
||||
def _describe(kind: Kind, entry: Entry, action: str | None = None) -> dict:
|
||||
spec = kind.link_spec
|
||||
return {
|
||||
"matches": [adopt_link_service.describe(spec, m) for m in entry.matches],
|
||||
"similar": [adopt_link_service.describe(spec, m) for m in entry.similar],
|
||||
"linkable": entry.linkable,
|
||||
"bulk_action": action,
|
||||
"uuid": entry.uuid,
|
||||
"title": kind.title(entry.row),
|
||||
"enabled": on(entry.row.get("enabled")) if "disabled" not in entry.row else not on(entry.row.get("disabled")),
|
||||
"adoptable": not entry.reasons,
|
||||
"bulk_adoptable": not entry.reasons or entry.bulk,
|
||||
"reasons": entry.reasons,
|
||||
"reason_details": entry.details,
|
||||
"kept": kind.kept(entry.row),
|
||||
"info": kind.info(entry.row),
|
||||
}
|
||||
|
||||
|
||||
async def list_unmanaged(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> list[dict]:
|
||||
kind = KINDS[kind_key]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
actions = _bulk_actions(entries, order, True)
|
||||
return [_describe(kind, e, actions.get(e.uuid, ("skip", None))[0]) for e in entries]
|
||||
|
||||
|
||||
def _create(db: AsyncSession, kind: Kind, host: OpnsenseHost, entry: Entry):
|
||||
obj = kind.model(**kind.build(entry.row), targets=[host], groups=[])
|
||||
db.add(obj)
|
||||
return obj
|
||||
|
||||
|
||||
async def _record(db: AsyncSession, kind: Kind, host: OpnsenseHost, obj, remote_uuid: str) -> None:
|
||||
await db.flush()
|
||||
kind.revision(db, obj, ADOPT_NOTE)
|
||||
db.add(kind.deployment(**{
|
||||
kind.fk: obj.id, "host_id": host.id, "status": "success", "message": ADOPT_MESSAGE,
|
||||
kind.uuid_attr: remote_uuid, "last_payload": kind.payload(obj), "previous_payload": None,
|
||||
}))
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid: str, link_to=None):
|
||||
"""Takes over one object as a NEW central object, or - with `link_to` - links it to that existing
|
||||
central object (adopt_link_service). Either way the live state is read fresh here."""
|
||||
kind = KINDS[kind_key]
|
||||
if link_to is not None:
|
||||
existing = await db.scalar(select(kind.deployment).where(
|
||||
kind.deployment.host_id == host.id, getattr(kind.deployment, kind.uuid_attr) == remote_uuid))
|
||||
if existing is not None:
|
||||
if getattr(existing, kind.fk) == link_to:
|
||||
return await db.get(kind.model, link_to) # second click: already linked, nothing to do
|
||||
raise NotAdoptable(["already_managed"])
|
||||
entries, _ = await _plan(db, kind, host)
|
||||
entry = next((e for e in entries if e.uuid == remote_uuid), None)
|
||||
if entry is None:
|
||||
# either gone from the host or already managed
|
||||
if remote_uuid in await _managed_uuids(db, kind, host):
|
||||
raise NotAdoptable(["already_managed"])
|
||||
raise NotFoundOnHost(remote_uuid)
|
||||
if link_to is not None:
|
||||
if not set(entry.reasons) <= LINK_TOLERATED:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = await adopt_link_service.link(db, kind.link_spec, host, link_to, remote_uuid, kind.build(entry.row))
|
||||
return obj
|
||||
if entry.reasons:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, remote_uuid)
|
||||
return obj
|
||||
|
||||
|
||||
async def adopt_all(db: AsyncSession, kind_key: str, host: OpnsenseHost, link_existing: bool = False) -> dict:
|
||||
"""Adopts everything adoptable in one transaction (the caller commits), nested aliases in
|
||||
dependency order. With `link_existing`, entries with exactly one exact central twin are linked to it
|
||||
instead (see _bulk_actions). System objects are not reported; everything else that stays behind is,
|
||||
with reasons."""
|
||||
kind = KINDS[kind_key]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
actions = _bulk_actions(entries, order, link_existing)
|
||||
adopted = []
|
||||
done: set[str] = set()
|
||||
for entry in entries: # linked first: their names are central already, so nested aliases can refer to them
|
||||
action, target = actions.get(entry.uuid, ("skip", None))
|
||||
if action == "link":
|
||||
obj = await adopt_link_service.link(db, kind.link_spec, host, target.id, entry.uuid, kind.build(entry.row))
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": True})
|
||||
done.add(entry.uuid)
|
||||
for entry in order:
|
||||
if actions.get(entry.uuid, ("skip", None))[0] != "new":
|
||||
continue
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, entry.uuid)
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": False})
|
||||
done.add(entry.uuid)
|
||||
skipped = [
|
||||
{"uuid": e.uuid, "title": kind.title(e.row),
|
||||
"reasons": e.reasons + (["twin_busy"] if link_existing and e.busy_twin else []), "reason_details": e.details}
|
||||
for e in entries if e.uuid not in done and e.reasons != ["system"]
|
||||
]
|
||||
return {"adopted": adopted, "skipped": skipped}
|
||||
@@ -145,6 +145,34 @@ async def get_interface_statistics(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/diagnostics/interface/getInterfaceStatistics")
|
||||
|
||||
|
||||
# Swap and CPU count, for the resource-threshold alert (1.24). Verified against core source
|
||||
# (Diagnostics/Api/SystemController::systemSwapAction -> configd "system show swapinfo") and live
|
||||
# (read-only GET, test-fw-3, OPNsense 26.7):
|
||||
# systemSwap -> {"swap": [{"device": "/dev/da0p3", "total": "8388608", "used": "0"}]} (KiB, strings;
|
||||
# an empty list when the firewall has no swap)
|
||||
# cpu_usage/getCPUType -> ["QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)"]
|
||||
async def get_system_swap(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/diagnostics/system/systemSwap")
|
||||
|
||||
|
||||
async def get_cpu_type(host: OpnsenseHost) -> Any:
|
||||
return await _request(host, "GET", "/diagnostics/cpu_usage/getCPUType")
|
||||
|
||||
|
||||
# Gateway health (read-only). Routes\Api\GatewayController::statusAction runs configd "interface gateways status"
|
||||
# -> scripts/routes/gateway_status.php, which merges the configured gateways with dpinger's live numbers
|
||||
# (src/etc/inc/plugins.inc.d/dpinger.inc, dpinger_status()). Live-verified on test-fw-3 (26.7):
|
||||
# {"items": [{"name": "WAN_GW", "address": "203.0.113.1", "status": "none", "loss": "~", "delay": "~",
|
||||
# "stddev": "~", "monitor": "~", "status_translated": "Online"}, ...], "status": "ok"}
|
||||
# "status" is the machine-readable verdict: none (fine), down, force_down (switched off by the admin), delay,
|
||||
# loss, delay+loss (thresholds of the gateway itself), anything else = pending. loss/delay/stddev are "12.3 %" /
|
||||
# "4.5 ms" strings, or "~" while nothing is measured (e.g. monitoring disabled on that gateway). "status" is
|
||||
# "failed" with no items when the firewall has no gateways. searchGateway carries the same numbers but with a
|
||||
# translated status text ("Online"), so it is not used here.
|
||||
async def get_gateway_status(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/routes/gateway/status")
|
||||
|
||||
|
||||
async def toggle_carp_maintenance(host: OpnsenseHost) -> dict:
|
||||
"""Toggle persistent CARP maintenance mode.
|
||||
|
||||
@@ -391,6 +419,42 @@ async def get_pf_rule_statistics(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/diagnostics/firewall/pfStatistics/rules")
|
||||
|
||||
|
||||
# ── Rule analysis (read-only) ─────────────────────────────────────────────
|
||||
# Checked against opnsense/core master (2026-10-21) and live on OPNsense 26.7:
|
||||
# - Firewall/Api/FilterUtilController::ruleStatsAction runs configd `!filter rule stats` and
|
||||
# returns {"status": "ok", "stats": {"<label>": {pf_rules, evaluations, packets, bytes, states}}};
|
||||
# the label is the rule UUID (MVC), an md5 (automatic) or a 32-hex label (unmigrated legacy).
|
||||
# - Firewall/Api/AliasUtilController::findReferencesAction requires POST with "ip" (only hex,
|
||||
# ":", ".", "/", "_" accepted, else {"status": "Not an IP address!"}), runs configd
|
||||
# `filter find_table_references` (a pf table test over all aliases) and returns
|
||||
# {"status": "ok", "matches": ["<alias>", ...]}. It changes nothing; POST is only the
|
||||
# controller's calling convention.
|
||||
# - Firewall/Api/FilterController::getInterfaceListAction: {"floating"|"groups"|"interfaces"|"any":
|
||||
# {"label", "items": [{"value", "label", "type"}]}} (lo0 omitted).
|
||||
# - Firewall/Api/GroupController::searchItemAction -> searchBase("ifgroupentry"): rows with
|
||||
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
|
||||
# wireguard) appear with their name as uuid and empty members.
|
||||
async def get_rule_stats(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/firewall/filter_util/ruleStats")
|
||||
|
||||
|
||||
async def find_alias_references(host: OpnsenseHost, ip: str) -> list[str] | None:
|
||||
"""Names of the aliases (pf tables) that contain ``ip`` right now; None when the firewall
|
||||
could not answer (callers then treat table membership as unknown)."""
|
||||
data = await _request(host, "POST", "/firewall/alias_util/findReferences", json={"ip": ip})
|
||||
if not isinstance(data, dict) or data.get("status") != "ok" or not isinstance(data.get("matches"), list):
|
||||
return None
|
||||
return [str(name) for name in data["matches"]]
|
||||
|
||||
|
||||
async def get_filter_interface_list(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/firewall/filter/get_interface_list")
|
||||
|
||||
|
||||
async def search_interface_groups(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/firewall/group/searchItem", params={"rowCount": -1})
|
||||
|
||||
|
||||
async def get_firmware_status(host: OpnsenseHost) -> dict:
|
||||
# GET reads cached results. POST would trigger a synchronous probe.
|
||||
return await _request(host, "GET", "/core/firmware/status")
|
||||
|
||||
@@ -5,7 +5,9 @@ poll_all_hosts. v1 deliberately has no admin-editable poll interval (see
|
||||
config.py) — just a fixed constant, to keep this feature's first cut
|
||||
small.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
@@ -13,6 +15,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
@@ -55,11 +58,59 @@ def _disk_used_pct(disk: dict) -> float | None:
|
||||
return None
|
||||
|
||||
|
||||
def _swap_used_pct(swap) -> float | None:
|
||||
"""systemSwap: {"swap": [{"device", "total", "used"}]} in KiB; None when there is no swap at all."""
|
||||
try:
|
||||
rows = swap.get("swap") or []
|
||||
total = sum(int(r["total"]) for r in rows)
|
||||
used = sum(int(r["used"]) for r in rows)
|
||||
except (AttributeError, KeyError, TypeError, ValueError):
|
||||
return None
|
||||
return round(used / total * 100, 1) if total > 0 else None
|
||||
|
||||
|
||||
_CPU_THREADS = re.compile(r"(\d+)\s+threads?", re.IGNORECASE)
|
||||
_CPU_CORES = re.compile(r"(\d+)\s+cores?", re.IGNORECASE)
|
||||
|
||||
|
||||
def _cpu_threads(cpu_type) -> int | None:
|
||||
"""getCPUType: ["<model> (6 cores, 6 threads)"]."""
|
||||
text = " ".join(str(x) for x in cpu_type) if isinstance(cpu_type, list) else str(cpu_type or "")
|
||||
match = _CPU_THREADS.search(text) or _CPU_CORES.search(text)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def _services_compact(listing) -> dict[str, int]:
|
||||
"""{"unbound": 1, "openvpn/<uuid>": 0}: 1 = running. Only services an admin can stop (not 'locked')."""
|
||||
result: dict[str, int] = {}
|
||||
for row in (listing.get("rows") if isinstance(listing, dict) else None) or []:
|
||||
if not isinstance(row, dict) or not row.get("name") or str(row.get("locked")) in ("1", "True", "true"):
|
||||
continue
|
||||
name, id_ = str(row["name"]), str(row.get("id") or row["name"])
|
||||
result[name if id_ == name else f"{name}/{id_}"] = 1 if str(row.get("running")) in ("1", "True", "true") else 0
|
||||
return result
|
||||
|
||||
|
||||
async def _optional(operation):
|
||||
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing)."""
|
||||
try:
|
||||
return await operation
|
||||
except Exception as exc:
|
||||
logger.debug("Optionale Messung fehlgeschlagen: %s", type(exc).__name__)
|
||||
return None
|
||||
|
||||
|
||||
async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
resources = await client.get_system_resources(host)
|
||||
system_time = await client.get_system_time(host)
|
||||
disk = await client.get_system_disk(host)
|
||||
interfaces = await client.get_interface_statistics(host)
|
||||
swap, cpu_type, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_system_swap(host)),
|
||||
_optional(client.get_cpu_type(host)),
|
||||
_optional(gateway_health_service.get_gateway_health(host)),
|
||||
_optional(client.search_services(host)),
|
||||
)
|
||||
db.add(
|
||||
HostResourceSample(
|
||||
host_id=host.id,
|
||||
@@ -67,7 +118,14 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
mem_used_pct=_mem_used_pct(resources),
|
||||
load_avg_1m=_load_avg_1m(system_time),
|
||||
disk_used_pct=_disk_used_pct(disk),
|
||||
raw={"memory": resources.get("memory"), "disk": disk.get("devices"), "interfaces": interfaces.get("statistics")},
|
||||
raw={
|
||||
"memory": resources.get("memory"), "disk": disk.get("devices"), "interfaces": interfaces.get("statistics"),
|
||||
# None = not readable this round (the alert checks treat that as "unknown", not as "fine").
|
||||
"swap_used_pct": _swap_used_pct(swap) if swap is not None else None,
|
||||
"cpu_threads": _cpu_threads(cpu_type) if cpu_type is not None else None,
|
||||
"gateways": gateway_health_service.compact(gateways) if gateways is not None else None,
|
||||
"services": _services_compact(services) if services is not None else None,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,805 @@
|
||||
"""The rollout run engine (1.25): moves runs forward, one small step per poll tick.
|
||||
|
||||
Ground rules (an unattended engine that triggers firmware updates is the riskiest thing in OPNexus):
|
||||
|
||||
* It never calls a firmware action itself. Each host step is an ordinary ScheduledMaintenance
|
||||
(run_at=now, origin="rollout", auto_verify=True) — locks, backup before the update, CARP handover,
|
||||
audit and restart safety all come from that existing path.
|
||||
* Never twice: a host gets a schedule only while its ``scheduled_maintenance_id`` is empty, written
|
||||
in the same transaction as the schedule itself, under the run's row lock.
|
||||
* Never in parallel: one host at a time per run, and only one active run at all; nodes of a cluster
|
||||
go BACKUP first, the MASTER only with the wave's CARP handover, and before every node of an
|
||||
already touched cluster the cluster is checked again.
|
||||
* When in doubt, stop: anything unexpected pauses the run (``pause_reason``) and notifies; nothing
|
||||
is retried on its own. A paused run only continues when a human resumes it, and resuming first
|
||||
checks the health of everything done so far.
|
||||
* Every step re-checks that the person responsible for the run may still roll out.
|
||||
* Hard limits: waves/hosts (rollout_service), soak time, preflight/host timeouts, run age.
|
||||
|
||||
Concurrency: a tick (1) does DB-only bookkeeping under the run's row lock, (2) decides with the
|
||||
lock released (remote reads may take seconds), and (3) applies the decision only if the run's
|
||||
``revision`` is still the one it decided on — a pause/abort in between, or a second backend, wins.
|
||||
"""
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Awaitable, Callable
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.rollout import RolloutPlan, RolloutRun, RolloutRunHost, RolloutRunWave
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.models.user import AdminUser
|
||||
from app.services import audit_service, rollout_health, rollout_service, window_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.host_maintenance_service import ensure_no_maintenance, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ACTIVE = ("waiting_window", "preflight", "wave_running", "soaking", "gate", "awaiting_approval", "resuming", "paused")
|
||||
TERMINAL = ("succeeded", "cancelled")
|
||||
# Statuses in which the poll loop should tick quickly.
|
||||
BUSY = ("preflight", "wave_running", "gate", "resuming")
|
||||
HOST_OPEN = ("running", "verifying")
|
||||
|
||||
PREFLIGHT_CHECK_DELAY = timedelta(seconds=20) # after asking for an update check, before reading the result
|
||||
PREFLIGHT_TIMEOUT = timedelta(minutes=10)
|
||||
GATE_RECHECK = timedelta(minutes=2) # a failed gate is looked at once more (slow start after a reboot)
|
||||
BLOCKED_PATIENCE = timedelta(minutes=15) # someone else's maintenance in the cluster
|
||||
HOST_START_TIMEOUT = timedelta(minutes=10) # the schedule never fired
|
||||
HOST_MAX_DURATION = timedelta(hours=4) # backstop above the schedule's own timeouts
|
||||
MAX_RUN_AGE = timedelta(days=14)
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
return value.isoformat() if value else None
|
||||
|
||||
|
||||
def _ts(value: str | None) -> datetime | None:
|
||||
return datetime.fromisoformat(value) if value else None
|
||||
|
||||
|
||||
def _note(code: str, **params) -> dict:
|
||||
return {"code": code, "params": params}
|
||||
|
||||
|
||||
# ── small helpers on a loaded run ─────────────────────────────────────────────
|
||||
|
||||
def wave_row(run: RolloutRun, index: int) -> RolloutRunWave:
|
||||
return next(w for w in run.waves if w.index == index)
|
||||
|
||||
|
||||
def wave_spec(run: RolloutRun, index: int) -> dict:
|
||||
return run.plan_snapshot["waves"][index]
|
||||
|
||||
|
||||
def hosts_of(run: RolloutRun, index: int) -> list[RolloutRunHost]:
|
||||
return sorted((h for h in run.hosts if h.wave_index == index), key=lambda h: (h.order, h.host_name))
|
||||
|
||||
|
||||
def _named(run: RolloutRun, failed: list[str]) -> list[str]:
|
||||
""""<host-id>:<check>" -> "<host name>:<check>" for pause details people read."""
|
||||
names = {str(h.host_id): h.host_name for h in run.hosts if h.host_id}
|
||||
return [f"{names.get(item.split(':', 1)[0], item.split(':', 1)[0])}:{item.split(':', 1)[1]}" for item in failed]
|
||||
|
||||
|
||||
def _progress(run: RolloutRun, **changes) -> None:
|
||||
run.progress = {**(run.progress or {}), **changes}
|
||||
|
||||
|
||||
async def _audit(run: RolloutRun, action: str, status: str, message: str | None = None, **details) -> None:
|
||||
token = audit_service.context.set({"actor": f"{run.started_by} (Rollout)"[:150], "id": str(run.id)})
|
||||
try:
|
||||
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id),
|
||||
entity_name=run.plan_name, message=message,
|
||||
details={"wave": run.current_wave, **details})
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _requeue(host: RolloutRunHost, code: str) -> None:
|
||||
"""Back to the queue: only for a host whose schedule provably never fired (cancelled under its row lock,
|
||||
or refused by fire_due as overdue). Clearing the schedule id lets a resumed run plan it once more."""
|
||||
host.status, host.scheduled_maintenance_id, host.started_at, host.detail = "queued", None, None, _note(code)
|
||||
|
||||
|
||||
async def requeue_if_pending(db: AsyncSession, host: RolloutRunHost) -> bool:
|
||||
"""Cancel a host's schedule that has not fired yet and put the host back in the queue. Locked like the
|
||||
router's abort: fire_due holds the row while firing, and a running action is never touched."""
|
||||
if host.status not in HOST_OPEN or host.scheduled_maintenance_id is None:
|
||||
return False
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == host.scheduled_maintenance_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if sched is None or sched.status != "pending":
|
||||
return False
|
||||
sched.status = "cancelled"
|
||||
sched.result = {"error": "Rollout angehalten, nicht ausgeführt", "not_fired": True}
|
||||
_requeue(host, "not_fired")
|
||||
return True
|
||||
|
||||
|
||||
def pause(run: RolloutRun, reason: str, **detail) -> None:
|
||||
if run.status != "paused":
|
||||
run.paused_from = run.status
|
||||
run.status = "paused"
|
||||
run.pause_reason = reason
|
||||
run.pause_detail = detail
|
||||
_progress(run, paused_at=_iso(_now()))
|
||||
|
||||
|
||||
async def may_roll_out(db: AsyncSession, username: str) -> bool:
|
||||
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout."""
|
||||
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
|
||||
if user is None:
|
||||
return False
|
||||
if user.role == "admin":
|
||||
return True
|
||||
if user.role == "operator":
|
||||
from app.routers import governance
|
||||
return (await governance.get_settings(db)).operators_may_deploy
|
||||
return False
|
||||
|
||||
|
||||
async def _window(db: AsyncSession, run: RolloutRun) -> tuple[bool, MaintenanceWindow | None]:
|
||||
"""(window configured, window). Configured but gone means: stop, never 'no time limit'."""
|
||||
window_id = run.plan_snapshot.get("window_id")
|
||||
if not window_id:
|
||||
return False, None
|
||||
return True, await db.get(MaintenanceWindow, window_id)
|
||||
|
||||
|
||||
async def _may_start_now(db: AsyncSession, run: RolloutRun) -> str | None:
|
||||
"""None if new host work may start now, else the reason ('window_closed' / 'window_missing')."""
|
||||
configured, window = await _window(db, run)
|
||||
if not configured:
|
||||
return None
|
||||
if window is None:
|
||||
return "window_missing"
|
||||
return None if window_service.startable(window, _now()) else "window_closed"
|
||||
|
||||
|
||||
async def _lock(db: AsyncSession, run_id, revision: int | None = None) -> RolloutRun | None:
|
||||
stmt = select(RolloutRun).where(RolloutRun.id == run_id)
|
||||
if revision is not None:
|
||||
stmt = stmt.where(RolloutRun.revision == revision)
|
||||
return await db.scalar(stmt.with_for_update().execution_options(populate_existing=True))
|
||||
|
||||
|
||||
# ── creating a run ────────────────────────────────────────────────────────────
|
||||
|
||||
async def active_run(db: AsyncSession) -> RolloutRun | None:
|
||||
return await db.scalar(select(RolloutRun).where(RolloutRun.status.in_(ACTIVE)).order_by(RolloutRun.created_at).limit(1))
|
||||
|
||||
|
||||
async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
|
||||
occurrence: datetime | None = None) -> RolloutRun:
|
||||
"""Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
|
||||
now = _now()
|
||||
snapshot = {
|
||||
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
||||
"window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
||||
}
|
||||
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
||||
current_wave=0, started_by=started_by, trigger=trigger, window_occurrence=occurrence,
|
||||
progress={}, created_at=now, step_at=now, revision=0)
|
||||
run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
|
||||
for w in resolved["waves"]]
|
||||
run.hosts = [RolloutRunHost(wave_index=w["index"], host_id=uuid.UUID(h["id"]), host_name=h["name"], cluster_name=h["cluster_name"],
|
||||
order=i, status="queued")
|
||||
for w in resolved["waves"] for i, h in enumerate(w["hosts"])]
|
||||
db.add(run)
|
||||
await db.flush()
|
||||
return run
|
||||
|
||||
|
||||
# ── the tick ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def advance_runs(db: AsyncSession) -> int:
|
||||
"""One step for every run. Returns how many runs want a quick next tick."""
|
||||
try:
|
||||
await auto_start(db)
|
||||
except Exception:
|
||||
logger.exception("Rollout: automatischer Start fehlgeschlagen")
|
||||
await db.rollback()
|
||||
ids = list((await db.scalars(select(RolloutRun.id).where(
|
||||
(RolloutRun.status.in_(ACTIVE)) | RolloutRun.hosts.any(RolloutRunHost.status.in_(HOST_OPEN))))).all())
|
||||
await db.commit()
|
||||
busy = 0
|
||||
for run_id in ids:
|
||||
try:
|
||||
await tick(run_id)
|
||||
except Exception as exc:
|
||||
logger.exception("Rollout %s: Schritt fehlgeschlagen", run_id)
|
||||
async with SessionLocal() as s:
|
||||
run = await _lock(s, run_id)
|
||||
if run is not None and run.status in ACTIVE and run.status != "paused":
|
||||
pause(run, "internal_error", error=type(exc).__name__)
|
||||
run.revision += 1
|
||||
await s.commit()
|
||||
await _audit(run, "pause", "paused", "Interner Fehler: Rollout angehalten", reason="internal_error")
|
||||
async with SessionLocal() as s:
|
||||
status = await s.scalar(select(RolloutRun.status).where(RolloutRun.id == run_id))
|
||||
busy += status in BUSY
|
||||
return busy
|
||||
|
||||
|
||||
async def tick(run_id) -> None:
|
||||
async with SessionLocal() as db:
|
||||
# (1) bookkeeping under the lock: what did the schedules do?
|
||||
run = await _lock(db, run_id)
|
||||
if run is None:
|
||||
return
|
||||
changed = await _sync_hosts(db, run)
|
||||
if (run.status in ACTIVE and run.status != "paused" and _now() - run.created_at > MAX_RUN_AGE
|
||||
and not any(h.status in HOST_OPEN for h in run.hosts)):
|
||||
pause(run, "run_too_old", days=MAX_RUN_AGE.days)
|
||||
changed = True
|
||||
if changed:
|
||||
run.revision += 1
|
||||
await db.commit()
|
||||
if changed and run.status == "paused" and run.pause_reason in ("host_failed", "host_timeout", "host_not_started", "run_too_old"):
|
||||
await _audit(run, "pause", "paused", reason=run.pause_reason, detail=run.pause_detail)
|
||||
if run.status not in ACTIVE or run.status in ("paused", "awaiting_approval"):
|
||||
return
|
||||
revision = run.revision
|
||||
# (2) decide, without holding the lock
|
||||
decision = await _HANDLERS[run.status](db, run)
|
||||
await db.commit()
|
||||
if decision is None:
|
||||
return
|
||||
# (3) apply only on the revision it was decided on
|
||||
run = await _lock(db, run_id, revision)
|
||||
if run is None:
|
||||
await db.commit()
|
||||
return
|
||||
events = await decision(db, run)
|
||||
run.revision += 1
|
||||
run.step_at = _now()
|
||||
await db.commit()
|
||||
for event in events or []:
|
||||
await _audit(run, *event[:2], **(event[2] if len(event) > 2 else {}))
|
||||
|
||||
|
||||
Decision = Callable[[AsyncSession, RolloutRun], Awaitable[list | None]]
|
||||
|
||||
|
||||
async def _sync_hosts(db: AsyncSession, run: RolloutRun) -> bool:
|
||||
"""Carry the outcome of each host's schedule over to the run. Returns True if anything changed."""
|
||||
changed = False
|
||||
now = _now()
|
||||
for host in run.hosts:
|
||||
if host.status not in HOST_OPEN:
|
||||
continue
|
||||
sched = await db.get(ScheduledMaintenance, host.scheduled_maintenance_id, populate_existing=True) \
|
||||
if host.scheduled_maintenance_id else None
|
||||
problem = None
|
||||
if sched is None:
|
||||
host.status, host.detail, host.finished_at = "failed", _note("schedule_missing"), now
|
||||
problem = ("host_failed", {"host": host.host_name, "error": "schedule_missing"})
|
||||
elif sched.status == "pending":
|
||||
if host.started_at and now - host.started_at > HOST_START_TIMEOUT:
|
||||
# Never leave it pending behind a paused run: it would fire whenever the poll loop comes back.
|
||||
if await requeue_if_pending(db, host):
|
||||
problem = ("host_not_started", {"host": host.host_name})
|
||||
changed = True
|
||||
elif (sched.result or {}).get("not_fired"):
|
||||
_requeue(host, "not_fired") # fire_due refused it as overdue: nothing happened on the firewall
|
||||
problem = ("host_not_started", {"host": host.host_name})
|
||||
changed = True
|
||||
elif sched.status == "running":
|
||||
want = "verifying" if sched.phase == "verify" else "running"
|
||||
if host.status != want:
|
||||
host.status = want
|
||||
changed = True
|
||||
if host.started_at and now - host.started_at > HOST_MAX_DURATION:
|
||||
problem = ("host_timeout", {"host": host.host_name})
|
||||
elif sched.status == "succeeded":
|
||||
result = sched.result or {}
|
||||
host.status, host.finished_at = "succeeded", now
|
||||
host.version_after = result.get("version_after")
|
||||
host.detail = _note("verified", version=result.get("version_after"), rebooted=bool(result.get("rebooted")))
|
||||
changed = True
|
||||
else: # failed / cancelled
|
||||
error = (sched.result or {}).get("error") or sched.status
|
||||
host.status, host.finished_at = "failed", now
|
||||
host.detail = _note("host_failed", error=error[:300])
|
||||
problem = ("host_failed", {"host": host.host_name, "error": error[:300]})
|
||||
if problem and run.status in ACTIVE and run.status != "paused":
|
||||
pause(run, problem[0], **problem[1])
|
||||
changed = True
|
||||
elif problem and host.status == "failed":
|
||||
changed = True
|
||||
return changed
|
||||
|
||||
|
||||
# ── handlers: one per run status; each returns an apply-function or None ──────
|
||||
|
||||
async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if await may_roll_out(db, run.started_by):
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
pause(run, "actor_invalid", user=run.started_by)
|
||||
return [("pause", "paused", {"reason": "actor_invalid"})]
|
||||
return apply
|
||||
|
||||
|
||||
def _to_waiting(reason: str) -> Decision:
|
||||
async def apply(db, run):
|
||||
if reason == "window_missing":
|
||||
pause(run, "window_missing")
|
||||
return [("pause", "paused", {"reason": reason})]
|
||||
run.paused_from = run.status
|
||||
run.status = "waiting_window"
|
||||
return [("waiting_window", "waiting")]
|
||||
return apply
|
||||
|
||||
|
||||
async def _handle_waiting_window(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
reason = await _may_start_now(db, run)
|
||||
if reason == "window_missing":
|
||||
return _to_waiting(reason)
|
||||
if reason:
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
_progress(run, resume_recheck_at=None, resume_attempts=0)
|
||||
run.status = "resuming" # the gate of everything done so far is checked again first
|
||||
return [("window_open", "resuming")]
|
||||
return apply
|
||||
|
||||
|
||||
async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
recheck = _ts((run.progress or {}).get("resume_recheck_at"))
|
||||
if recheck and _now() < recheck:
|
||||
return None
|
||||
done = [h for h in run.hosts if h.status == "succeeded" and h.host_id]
|
||||
gate = {"hosts": {}, "verdict": "ok"}
|
||||
if done:
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
baselines = {}
|
||||
for w in run.waves:
|
||||
baselines.update((w.baseline or {}))
|
||||
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
|
||||
async def apply(db, run):
|
||||
if gate["verdict"] == "fail":
|
||||
attempts = (run.progress or {}).get("resume_attempts", 0) + 1
|
||||
_progress(run, resume_gate=gate, resume_attempts=attempts)
|
||||
if attempts < 2:
|
||||
_progress(run, resume_recheck_at=_iso(_now() + GATE_RECHECK))
|
||||
return [("resume_gate", "recheck", {"failed": rollout_health.failed_checks(gate)})]
|
||||
run.paused_from = run.paused_from or "preflight"
|
||||
_progress(run, resume_recheck_at=None)
|
||||
run.status = "paused"
|
||||
run.pause_reason, run.pause_detail = "gate_failed", {"failed": _named(run, rollout_health.failed_checks(gate)), "on_resume": True}
|
||||
return [("resume_gate", "failed", {"failed": rollout_health.failed_checks(gate)})]
|
||||
_progress(run, resume_gate=gate, resume_recheck_at=None, resume_attempts=0)
|
||||
target = run.paused_from if run.paused_from in ("preflight", "wave_running", "soaking", "gate") else "preflight"
|
||||
if target == "gate":
|
||||
wave_row(run, run.current_wave).gate_attempts = 0
|
||||
_progress(run, gate_recheck_at=None)
|
||||
if target == "preflight":
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None)
|
||||
run.status, run.paused_from, run.pause_reason, run.pause_detail = target, None, None, None
|
||||
return [("resume_gate", "passed", {"continue_with": target})]
|
||||
return apply
|
||||
|
||||
|
||||
def target_of(status: dict, action: str) -> str | None:
|
||||
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
|
||||
|
||||
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
|
||||
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
|
||||
core package's new version comes first; product_latest is only a fallback."""
|
||||
if action == "major_update":
|
||||
return status.get("upgrade_major_version") or None
|
||||
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
|
||||
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
|
||||
if core and core.get("new") and core.get("new") != "N/A":
|
||||
return str(core["new"])
|
||||
for row in status.get("upgrade_packages") or []:
|
||||
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
|
||||
return str(row["new_version"])
|
||||
product = status.get("product") if isinstance(status.get("product"), dict) else {}
|
||||
return product.get("product_latest") or status.get("product_latest") or None
|
||||
|
||||
|
||||
async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if problem := await _actor_problem(db, run):
|
||||
return problem
|
||||
reason = await _may_start_now(db, run)
|
||||
if reason:
|
||||
return _to_waiting(reason)
|
||||
index = run.current_wave
|
||||
prog = run.progress or {}
|
||||
started = _ts(prog.get("preflight_started")) or _now()
|
||||
requested = _ts(prog.get("preflight_requested_at"))
|
||||
candidates = [h for h in hosts_of(run, index) if h.status in ("queued", "blocked")]
|
||||
now = _now()
|
||||
|
||||
# Which hosts are usable at all?
|
||||
usable: dict = {}
|
||||
skips: dict = {}
|
||||
for rh in candidates:
|
||||
host = await db.get(OpnsenseHost, rh.host_id) if rh.host_id else None
|
||||
if host is None:
|
||||
skips[rh.id] = _note("host_deleted")
|
||||
elif not host.enabled:
|
||||
skips[rh.id] = _note("host_disabled")
|
||||
else:
|
||||
usable[rh.id] = host
|
||||
|
||||
if requested is None:
|
||||
# Step 1: ask every usable host to look for updates (a firmware *check*, changes nothing installed).
|
||||
asked = []
|
||||
for rh_id, host in usable.items():
|
||||
try:
|
||||
await ensure_no_maintenance(db, [host])
|
||||
if (await client.get_firmware_running(host)).get("status") == "ready":
|
||||
await client.check_firmware_updates(host)
|
||||
asked.append(str(rh_id))
|
||||
except (client.OpnsenseAPIError, HTTPException):
|
||||
pass # judged in step 2 (and blocked after the timeout)
|
||||
|
||||
async def apply(db, run):
|
||||
_progress(run, preflight_started=_iso(started), preflight_requested_at=_iso(_now()), preflight_asked=asked)
|
||||
w = wave_row(run, index)
|
||||
w.status = "preflight"
|
||||
w.started_at = w.started_at or _now()
|
||||
return [("preflight", "started", {"hosts": len(usable)})]
|
||||
return apply
|
||||
if now - requested < PREFLIGHT_CHECK_DELAY:
|
||||
return None
|
||||
|
||||
# Step 2: read what each host offers.
|
||||
offers: dict = {}
|
||||
waiting: dict = {}
|
||||
for rh_id, host in usable.items():
|
||||
try:
|
||||
await ensure_no_maintenance(db, [host])
|
||||
except HTTPException:
|
||||
waiting[rh_id] = _note("locked")
|
||||
continue
|
||||
try:
|
||||
running = (await client.get_firmware_running(host)).get("status")
|
||||
if running != "ready":
|
||||
waiting[rh_id] = _note("firmware_busy")
|
||||
continue
|
||||
status = await client.get_firmware_status(host)
|
||||
except client.OpnsenseAPIError:
|
||||
waiting[rh_id] = _note("offline")
|
||||
continue
|
||||
fw = status.get("status")
|
||||
wanted = "upgrade" if run.action == "major_update" else "update"
|
||||
if fw == wanted:
|
||||
offers[rh_id] = target_of(status, run.action)
|
||||
elif fw in ("none", "update", "upgrade"):
|
||||
skips[rh_id] = _note("no_update_offered", status=fw)
|
||||
else:
|
||||
waiting[rh_id] = _note("firmware_status", status=str(fw)[:40])
|
||||
|
||||
if waiting:
|
||||
if now - started < PREFLIGHT_TIMEOUT:
|
||||
return None
|
||||
|
||||
async def apply_blocked(db, run):
|
||||
names = []
|
||||
for rh in hosts_of(run, index):
|
||||
if rh.id in waiting:
|
||||
rh.status, rh.detail = "blocked", waiting[rh.id]
|
||||
names.append(rh.host_name)
|
||||
pause(run, "preflight_blocked", hosts=", ".join(names))
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None)
|
||||
return [("pause", "paused", {"reason": "preflight_blocked", "hosts": names})]
|
||||
return apply_blocked
|
||||
|
||||
targets = {t for t in offers.values()}
|
||||
expected = run.target_version
|
||||
mismatch = None
|
||||
if None in targets:
|
||||
mismatch = {"offered": "?"}
|
||||
elif expected is None and len(targets) > 1:
|
||||
mismatch = {"offered": ", ".join(sorted(targets))}
|
||||
elif expected is not None and targets - {expected}:
|
||||
mismatch = {"expected": expected, "offered": ", ".join(sorted(targets - {expected}))}
|
||||
if mismatch:
|
||||
async def apply_mismatch(db, run):
|
||||
# OPNsense always installs the newest offered version; a later ring must not get one the earlier ring never ran.
|
||||
pause(run, "target_version_changed", **mismatch)
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None)
|
||||
return [("pause", "paused", {"reason": "target_version_changed", **mismatch})]
|
||||
return apply_mismatch
|
||||
|
||||
# Baselines and order (BACKUP before MASTER inside a cluster).
|
||||
baselines = {}
|
||||
versions = {}
|
||||
for rh_id in offers:
|
||||
host = usable[rh_id]
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host)
|
||||
versions[rh_id] = baselines[str(host.id)].get("version")
|
||||
roles = await get_statuses([usable[i] for i in offers])
|
||||
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
||||
|
||||
def order_key(rh_id):
|
||||
host = usable[rh_id]
|
||||
role = roles.get(host.id, {}).get("carp_status") if host.cluster_name else None
|
||||
return (1 if host.cluster_name else 0, host.cluster_name or "", rank.get(role, 1), host.name)
|
||||
ordered = sorted(offers, key=order_key)
|
||||
target = next(iter(targets)) if targets else expected
|
||||
|
||||
async def apply_ready(db, run):
|
||||
for rh in hosts_of(run, index):
|
||||
if rh.id in skips:
|
||||
rh.status, rh.detail = "skipped", skips[rh.id]
|
||||
elif rh.id in offers:
|
||||
rh.status, rh.detail = "queued", None
|
||||
rh.order = ordered.index(rh.id)
|
||||
rh.version_before = versions.get(rh.id)
|
||||
w = wave_row(run, index)
|
||||
w.baseline = baselines
|
||||
if run.target_version is None and target:
|
||||
run.target_version = target
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None, blocked_since=None, cluster_recheck_at=None)
|
||||
if not offers:
|
||||
w.status, w.finished_at = "skipped", _now()
|
||||
events = [("wave", "skipped", {"wave": index})]
|
||||
events += _next_wave(run)
|
||||
return events
|
||||
w.status = "running"
|
||||
run.status = "wave_running"
|
||||
return [("wave", "started", {"wave": index, "target_version": run.target_version,
|
||||
"hosts": [usable[i].name for i in ordered], "skipped": len(skips)})]
|
||||
return apply_ready
|
||||
|
||||
|
||||
def _next_wave(run: RolloutRun) -> list:
|
||||
index = run.current_wave + 1
|
||||
if index >= len(run.waves):
|
||||
run.status, run.finished_at = "succeeded", _now()
|
||||
return [("finish", "succeeded")]
|
||||
run.current_wave = index
|
||||
w = wave_row(run, index)
|
||||
if wave_spec(run, index).get("require_approval"):
|
||||
w.status = run.status = "awaiting_approval"
|
||||
return [("wave", "awaiting_approval", {"wave": index})]
|
||||
w.status, run.status = "preflight", "preflight"
|
||||
return []
|
||||
|
||||
|
||||
async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if any(h.status in HOST_OPEN for h in run.hosts):
|
||||
return None # strictly one host at a time
|
||||
index = run.current_wave
|
||||
queued = [h for h in hosts_of(run, index) if h.status == "queued"]
|
||||
if not queued:
|
||||
soak = int(wave_spec(run, index).get("soak_minutes") or 0)
|
||||
|
||||
async def apply_soak(db, run):
|
||||
w = wave_row(run, index)
|
||||
w.status, w.soak_until = "soaking", _now() + timedelta(minutes=soak)
|
||||
run.status = "soaking"
|
||||
return [("wave", "soaking", {"wave": index, "minutes": soak})]
|
||||
return apply_soak
|
||||
if problem := await _actor_problem(db, run):
|
||||
return problem
|
||||
reason = await _may_start_now(db, run)
|
||||
if reason:
|
||||
return _to_waiting(reason)
|
||||
nxt = queued[0]
|
||||
host = await db.get(OpnsenseHost, nxt.host_id) if nxt.host_id else None
|
||||
if host is None or not host.enabled:
|
||||
async def apply_skip(db, run):
|
||||
rh = next(h for h in run.hosts if h.id == nxt.id)
|
||||
rh.status, rh.detail = "skipped", _note("host_deleted" if host is None else "host_disabled")
|
||||
return [("host", "skipped", {"host": nxt.host_name})]
|
||||
return apply_skip
|
||||
spec = wave_spec(run, index)
|
||||
prog = run.progress or {}
|
||||
|
||||
if host.cluster_name:
|
||||
fleet = await peers(db, host)
|
||||
statuses = await get_statuses(fleet)
|
||||
done_ids = {h.host_id for h in run.hosts if h.status == "succeeded"}
|
||||
done_peers = [p for p in fleet if p.id in done_ids and p.id != host.id]
|
||||
baselines = {}
|
||||
for w in run.waves:
|
||||
baselines.update(w.baseline or {})
|
||||
# The cluster as a whole must be healthy before a node is touched; nodes done earlier in this run
|
||||
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
||||
gate = await rollout_health.evaluate(
|
||||
db, done_peers + [host], baselines, target_version=run.target_version,
|
||||
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
||||
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
||||
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
||||
if r["result"] == "fail" and not (hid == str(host.id) and c in ("version", "gateways", "services"))]
|
||||
if failed:
|
||||
recheck = _ts(prog.get("cluster_recheck_at"))
|
||||
if recheck is None:
|
||||
async def apply_recheck(db, run):
|
||||
_progress(run, cluster_recheck_at=_iso(_now() + GATE_RECHECK), cluster_gate=gate)
|
||||
return [("cluster_gate", "recheck", {"host": host.name, "failed": failed})]
|
||||
return apply_recheck
|
||||
if _now() < recheck:
|
||||
return None
|
||||
|
||||
async def apply_cluster_fail(db, run):
|
||||
pause(run, "cluster_gate_failed", cluster=host.cluster_name, host=host.name, failed=_named(run, failed))
|
||||
_progress(run, cluster_recheck_at=None, cluster_gate=gate)
|
||||
return [("cluster_gate", "failed", {"host": host.name, "failed": failed})]
|
||||
return apply_cluster_fail
|
||||
if statuses.get(host.id, {}).get("carp_status") == "MASTER" and not spec.get("carp_handover"):
|
||||
async def apply_master(db, run):
|
||||
pause(run, "master_needs_handover", host=host.name, cluster=host.cluster_name)
|
||||
return [("pause", "paused", {"reason": "master_needs_handover", "host": host.name})]
|
||||
return apply_master
|
||||
|
||||
try:
|
||||
await ensure_no_maintenance(db, [host])
|
||||
except HTTPException:
|
||||
since = _ts(prog.get("blocked_since"))
|
||||
if since is None:
|
||||
async def apply_blocked_start(db, run):
|
||||
_progress(run, blocked_since=_iso(_now()))
|
||||
return [("host", "waiting_for_lock", {"host": host.name})]
|
||||
return apply_blocked_start
|
||||
if _now() - since < BLOCKED_PATIENCE:
|
||||
return None
|
||||
|
||||
async def apply_blocked(db, run):
|
||||
pause(run, "blocked_by_maintenance", host=host.name)
|
||||
_progress(run, blocked_since=None)
|
||||
return [("pause", "paused", {"reason": "blocked_by_maintenance", "host": host.name})]
|
||||
return apply_blocked
|
||||
|
||||
async def apply_start(db, run):
|
||||
await policy_lock(db)
|
||||
rh = next(h for h in run.hosts if h.id == nxt.id)
|
||||
# Idempotency guard: a host that already has a schedule never gets a second one.
|
||||
if rh.status != "queued" or rh.scheduled_maintenance_id is not None:
|
||||
return []
|
||||
now = _now()
|
||||
sched = ScheduledMaintenance(
|
||||
host_id=host.id, host_name=host.name, action=run.action, run_at=now, status="pending",
|
||||
actor=run.started_by, created_at=now, carp_handover=bool(spec.get("carp_handover") and host.cluster_name),
|
||||
auto_verify=True, origin="rollout",
|
||||
progress={"rollout": {"run_id": str(run.id), "wave": index, "plan": run.plan_name}},
|
||||
)
|
||||
db.add(sched)
|
||||
await db.flush()
|
||||
rh.scheduled_maintenance_id, rh.status, rh.started_at = sched.id, "running", now
|
||||
_progress(run, blocked_since=None, cluster_recheck_at=None)
|
||||
return [("host", "started", {"host": host.name, "schedule_id": str(sched.id), "handover": sched.carp_handover})]
|
||||
return apply_start
|
||||
|
||||
|
||||
async def _handle_soaking(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
w = wave_row(run, run.current_wave)
|
||||
if w.soak_until and _now() < w.soak_until:
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, run.current_wave)
|
||||
w.status, w.gate_attempts = "gate", 0
|
||||
run.status = "gate"
|
||||
_progress(run, gate_recheck_at=None)
|
||||
return []
|
||||
return apply
|
||||
|
||||
|
||||
async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
recheck = _ts((run.progress or {}).get("gate_recheck_at"))
|
||||
if recheck and _now() < recheck:
|
||||
return None
|
||||
index = run.current_wave
|
||||
w = wave_row(run, index)
|
||||
done = [h for h in hosts_of(run, index) if h.status == "succeeded" and h.host_id]
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, index)
|
||||
w.gate_result = gate
|
||||
w.gate_attempts = (w.gate_attempts or 0) + 1
|
||||
failed = rollout_health.failed_checks(gate)
|
||||
events = [("gate", gate["verdict"], {"wave": index, "attempt": w.gate_attempts, "failed": failed, "result": gate})]
|
||||
if gate["verdict"] == "fail":
|
||||
if w.gate_attempts < 2:
|
||||
_progress(run, gate_recheck_at=_iso(_now() + GATE_RECHECK))
|
||||
return events
|
||||
_progress(run, gate_recheck_at=None)
|
||||
pause(run, "gate_failed", wave=w.label, failed=_named(run, failed))
|
||||
return events + [("pause", "paused", {"reason": "gate_failed"})]
|
||||
w.status, w.finished_at = "passed", _now()
|
||||
_progress(run, gate_recheck_at=None)
|
||||
return events + _next_wave(run)
|
||||
return apply
|
||||
|
||||
|
||||
_HANDLERS = {
|
||||
"waiting_window": _handle_waiting_window,
|
||||
"resuming": _handle_resuming,
|
||||
"preflight": _handle_preflight,
|
||||
"wave_running": _handle_wave_running,
|
||||
"soaking": _handle_soaking,
|
||||
"gate": _handle_gate,
|
||||
}
|
||||
|
||||
|
||||
# ── automatic start at a window occurrence ────────────────────────────────────
|
||||
|
||||
async def auto_start(db: AsyncSession) -> int:
|
||||
started = 0
|
||||
plans = (await db.scalars(select(RolloutPlan).where(
|
||||
RolloutPlan.auto_start.is_(True), RolloutPlan.activated_at.is_not(None), RolloutPlan.window_id.is_not(None)))).all()
|
||||
for plan in plans:
|
||||
window = await db.get(MaintenanceWindow, plan.window_id)
|
||||
occ = window_service.startable(window, _now()) if window else None
|
||||
if occ is None:
|
||||
continue
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
|
||||
continue
|
||||
if (plan.auto_start_error or {}).get("occurrence") == _iso(occ.start):
|
||||
continue # already reported for this occurrence
|
||||
await policy_lock(db)
|
||||
plan = await rollout_service.load_plan(db, plan.id)
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
|
||||
continue # another backend started it while this one waited for the lock (not a refusal)
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
problem = None
|
||||
if plan.action != "update":
|
||||
problem = _note("auto_start_update_only")
|
||||
elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
|
||||
problem = _note("plan_changed") # someone changed the plan, its groups or its window since activation
|
||||
elif not await may_roll_out(db, plan.activated_by or ""):
|
||||
problem = _note("actor_invalid", user=plan.activated_by)
|
||||
elif resolved["errors"]:
|
||||
problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
|
||||
elif (other := await active_run(db)) is not None:
|
||||
problem = _note("other_run_active", plan=other.plan_name)
|
||||
if problem:
|
||||
plan.auto_start_error = {"occurrence": _iso(occ.start), "at": _iso(_now()), **problem}
|
||||
await db.commit()
|
||||
token = audit_service.context.set({"actor": "system:rollout", "id": str(plan.id)})
|
||||
try:
|
||||
await audit_service.persist(kind="rollout", action="auto_start", status="refused", entity_id=str(plan.id),
|
||||
entity_name=plan.name, details=plan.auto_start_error)
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
continue
|
||||
try:
|
||||
run = await create_run(db, plan, resolved, started_by=plan.activated_by, trigger="window", occurrence=occ.start)
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback() # another backend started it for this occurrence
|
||||
continue
|
||||
started += 1
|
||||
await _audit(run, "start", "started", trigger="window", occurrence=_iso(occ.start),
|
||||
hosts=sum(len(w["hosts"]) for w in resolved["waves"]))
|
||||
await db.commit()
|
||||
return started
|
||||
|
||||
|
||||
# ── human actions (called by the router under policy_lock) ────────────────────
|
||||
|
||||
async def count_active(db: AsyncSession) -> int:
|
||||
return await db.scalar(select(func.count()).select_from(RolloutRun).where(RolloutRun.status.in_(ACTIVE))) or 0
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Health gate of a firmware rollout (1.25): is a host, after its update, as healthy as before?
|
||||
|
||||
``baseline(db, host)`` is taken right before a wave starts, ``evaluate(...)`` after the soak time
|
||||
(and before each node of an already touched cluster, and on resume). Every check of every host
|
||||
yields ``{"result": ok|warn|fail|skip, "code": <machine code>, "params": {...}}``; the UI translates
|
||||
the code (rollouts.gate.code.*), so no German prose travels through here.
|
||||
|
||||
What the checks look at (all read-only on the firewall, except that the compliance check downloads
|
||||
the configuration like the 6-hourly security check does):
|
||||
|
||||
* online — enabled, the API answers, firmware job idle (``running == ready``).
|
||||
* version — the installed version is the rollout's target version.
|
||||
* carp — per cluster: every node online, exactly one MASTER per VHID, no node in CARP
|
||||
maintenance mode, no CARP auto-revert timer running.
|
||||
* gateways — gateways that were up in the baseline are up again; a gateway that was already bad
|
||||
before only warns (1.24 ``get_gateway_health``).
|
||||
* services — services that ran in the baseline run again.
|
||||
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
||||
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
||||
update may legitimately change defaults.
|
||||
|
||||
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
||||
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
||||
"""
|
||||
import asyncio
|
||||
from collections import defaultdict
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.carp_maintenance_window import CarpMaintenanceWindow
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import compliance_service, gateway_health_service, opnsense_client as client
|
||||
from app.services.host_maintenance_service import _shared_change_blockers, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.resource_monitor_service import _services_compact
|
||||
|
||||
CHECKS = ("online", "version", "carp", "gateways", "services", "locks", "compliance")
|
||||
DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": True, "services": True, "locks": True,
|
||||
"compliance": False}
|
||||
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
||||
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
||||
|
||||
|
||||
def _r(result: str, code: str, **params) -> dict:
|
||||
return {"result": result, "code": code, "params": params}
|
||||
|
||||
|
||||
async def _optional(coro):
|
||||
try:
|
||||
return await coro
|
||||
except client.OpnsenseAPIError:
|
||||
return None
|
||||
|
||||
|
||||
async def _gateways(host) -> dict | None:
|
||||
gws = await _optional(gateway_health_service.get_gateway_health(host))
|
||||
return None if gws is None else {g.name: g.state for g in gws}
|
||||
|
||||
|
||||
async def _services(host) -> dict | None:
|
||||
listing = await _optional(client.search_services(host))
|
||||
return None if listing is None else _services_compact(listing)
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
info, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
||||
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
||||
return {
|
||||
"version": (info or {}).get("product_version"),
|
||||
"gateways": gateways,
|
||||
"services": services,
|
||||
"compliance_score": score,
|
||||
}
|
||||
|
||||
|
||||
def _check_gateways(before: dict | None, now: dict | None) -> dict:
|
||||
if now is None:
|
||||
return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
|
||||
if before is None:
|
||||
bad = sorted(n for n, s in now.items() if s == "offline")
|
||||
return _r("warn", "gateway_down", names=", ".join(bad)) if bad else _r("ok", "no_baseline")
|
||||
down = sorted(n for n, s in before.items() if s in ("online", "unknown") and now.get(n) == "offline")
|
||||
gone = sorted(n for n in before if n not in now)
|
||||
degraded = sorted(n for n, s in now.items() if s == "degraded" and before.get(n) != "degraded")
|
||||
still_bad = sorted(n for n, s in before.items() if s in ("offline", "degraded") and now.get(n) in ("offline", "degraded"))
|
||||
# "unknown" (not monitored) going to "offline" is a real change too: monitoring just started and reports down.
|
||||
if down:
|
||||
return _r("fail", "gateway_down", names=", ".join(down))
|
||||
if gone:
|
||||
return _r("warn", "gateway_missing", names=", ".join(gone))
|
||||
if degraded:
|
||||
return _r("warn", "gateway_degraded", names=", ".join(degraded))
|
||||
if still_bad:
|
||||
return _r("warn", "gateway_bad_before", names=", ".join(still_bad))
|
||||
return _r("ok", "ok")
|
||||
|
||||
|
||||
def _check_services(before: dict | None, now: dict | None) -> dict:
|
||||
if now is None:
|
||||
return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
|
||||
if before is None:
|
||||
return _r("ok", "no_baseline")
|
||||
stopped = sorted(name.split("/")[0] for name, running in before.items() if running == 1 and now.get(name) != 1)
|
||||
if stopped:
|
||||
return _r("fail", "service_stopped", names=", ".join(dict.fromkeys(stopped)))
|
||||
return _r("ok", "ok")
|
||||
|
||||
|
||||
async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dict) -> dict:
|
||||
offline = sorted(h.name for h in fleet if h.enabled and statuses.get(h.id, {}).get("status") != "online")
|
||||
if offline:
|
||||
return _r("fail", "node_offline", names=", ".join(offline))
|
||||
online = [h for h in fleet if statuses.get(h.id, {}).get("status") == "online"]
|
||||
demoted = sorted(h.name for h in online if statuses[h.id].get("carp_maintenance_mode"))
|
||||
if demoted:
|
||||
return _r("fail", "maintenance_mode", names=", ".join(demoted))
|
||||
timers = (await db.scalars(select(CarpMaintenanceWindow.host_name).where(
|
||||
CarpMaintenanceWindow.host_id.in_([h.id for h in fleet]), CarpMaintenanceWindow.status == "active"))).all()
|
||||
if timers:
|
||||
return _r("fail", "auto_revert_active", names=", ".join(sorted(timers)))
|
||||
masters: dict[str, int] = defaultdict(int)
|
||||
broken: set[str] = set()
|
||||
for h in online:
|
||||
for vhid, state in (statuses[h.id].get("carp_vips") or {}).items():
|
||||
masters[vhid] += state == "MASTER"
|
||||
if state not in ("MASTER", "BACKUP"):
|
||||
broken.add(vhid)
|
||||
wrong = sorted(set(v for v, n in masters.items() if n != 1) | broken, key=lambda v: (len(v), v))
|
||||
if wrong:
|
||||
return _r("fail", "vhid_roles", vhids=", ".join(wrong))
|
||||
if not masters and not any(statuses[h.id].get("carp_status") == "MASTER" for h in online):
|
||||
return _r("fail", "no_master")
|
||||
return _r("ok", "ok")
|
||||
|
||||
|
||||
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
||||
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
|
||||
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
||||
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
||||
for name in ALWAYS_ON:
|
||||
enabled[name] = True
|
||||
results: dict[str, dict] = {}
|
||||
fleets: dict[str, list[OpnsenseHost]] = {}
|
||||
for host in hosts:
|
||||
fleets[str(host.id)] = await peers(db, host) if host.cluster_name else [host]
|
||||
every = {h.id: h for fleet in fleets.values() for h in fleet}
|
||||
statuses = await get_statuses(list(every.values()))
|
||||
carp_cache: dict[str, dict] = {}
|
||||
for host in hosts:
|
||||
hid = str(host.id)
|
||||
before = baselines.get(hid) or {}
|
||||
row: dict[str, dict] = {}
|
||||
st = statuses.get(host.id, {})
|
||||
running = await _optional(client.get_firmware_running(host)) if st.get("status") == "online" else None
|
||||
if not host.enabled:
|
||||
row["online"] = _r("fail", "disabled")
|
||||
elif st.get("status") != "online":
|
||||
row["online"] = _r("fail", "offline")
|
||||
elif (running or {}).get("status") != "ready":
|
||||
row["online"] = _r("fail", "firmware_busy")
|
||||
else:
|
||||
row["online"] = _r("ok", "ok")
|
||||
reachable = row["online"]["result"] == "ok"
|
||||
|
||||
if enabled["version"]:
|
||||
info = await _optional(client.get_firmware_info(host)) if reachable else None
|
||||
version = (info or {}).get("product_version")
|
||||
if not target_version:
|
||||
row["version"] = _r("skip", "no_target")
|
||||
elif version is None:
|
||||
row["version"] = _r("fail", "unreadable")
|
||||
elif version != target_version:
|
||||
row["version"] = _r("fail", "version_mismatch", expected=target_version, actual=version)
|
||||
else:
|
||||
row["version"] = _r("ok", "ok", actual=version)
|
||||
|
||||
if enabled["carp"]:
|
||||
if not host.cluster_name:
|
||||
row["carp"] = _r("skip", "no_cluster")
|
||||
else:
|
||||
if host.cluster_name not in carp_cache:
|
||||
carp_cache[host.cluster_name] = await _check_carp(db, fleets[hid], statuses)
|
||||
row["carp"] = carp_cache[host.cluster_name]
|
||||
|
||||
if enabled["gateways"]:
|
||||
row["gateways"] = _check_gateways(before.get("gateways"), await _gateways(host) if reachable else None)
|
||||
if enabled["services"]:
|
||||
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
||||
|
||||
blockers = await _shared_change_blockers(db, fleets[hid])
|
||||
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
||||
|
||||
if enabled["compliance"]:
|
||||
base_score = before.get("compliance_score")
|
||||
if base_score is None:
|
||||
row["compliance"] = _r("skip", "no_baseline")
|
||||
elif not reachable:
|
||||
row["compliance"] = _r("fail", "unreadable")
|
||||
else:
|
||||
fresh = await compliance_service.check_host(db, host)
|
||||
if fresh.score is None:
|
||||
row["compliance"] = _r("warn", "unreadable")
|
||||
elif fresh.score < base_score:
|
||||
row["compliance"] = _r("fail" if compliance_mode == "fail" else "warn", "score_lower",
|
||||
before=base_score, after=fresh.score)
|
||||
else:
|
||||
row["compliance"] = _r("ok", "ok", before=base_score, after=fresh.score)
|
||||
results[hid] = row
|
||||
worst = max((_RANK[c["result"]] for row in results.values() for c in row.values()), default=0)
|
||||
return {"hosts": results, "verdict": ("ok", "warn", "fail")[worst]}
|
||||
|
||||
|
||||
def failed_checks(gate: dict) -> list[str]:
|
||||
"""Flat "host-id:check" list of everything that failed (for notifications and the audit trail)."""
|
||||
return [f"{hid}:{name}" for hid, row in (gate.get("hosts") or {}).items() for name, c in row.items() if c["result"] == "fail"]
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Rollout plans: resolving waves to hosts, validation, preview token / activation fingerprint.
|
||||
|
||||
The run engine (rollout_engine.py) builds on ``resolve``: what a preview shows is exactly what a run
|
||||
freezes when it starts.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan
|
||||
from app.services import rollout_health, window_service
|
||||
|
||||
# Hard limits: an unattended engine that fires firmware updates must stay small and explainable.
|
||||
MAX_WAVES = 10
|
||||
MAX_HOSTS = 100
|
||||
MAX_SOAK_MINUTES = 24 * 60
|
||||
|
||||
|
||||
def _issue(code: str, **params) -> dict:
|
||||
return {"code": code, "params": params}
|
||||
|
||||
|
||||
async def resolve(db: AsyncSession, plan: RolloutPlan) -> dict:
|
||||
"""Waves with their hosts (a host in several waves belongs to the earliest), errors that block a
|
||||
start/activation, and warnings."""
|
||||
errors: list[dict] = []
|
||||
warnings: list[dict] = []
|
||||
seen: dict = {}
|
||||
duplicates: set[str] = set()
|
||||
waves = []
|
||||
group_ids = {g.id for w in plan.waves for g in w.groups}
|
||||
groups = {g.id: g for g in (await db.scalars(
|
||||
select(DeviceGroup).where(DeviceGroup.id.in_(group_ids)).options(selectinload(DeviceGroup.hosts)))).all()} if group_ids else {}
|
||||
for index, wave in enumerate(plan.waves):
|
||||
hosts: list[OpnsenseHost] = []
|
||||
for group in sorted(wave.groups, key=lambda g: g.name):
|
||||
for host in sorted(groups[group.id].hosts, key=lambda h: h.name):
|
||||
if host.id in seen:
|
||||
if seen[host.id] != index:
|
||||
duplicates.add(host.name)
|
||||
continue
|
||||
seen[host.id] = index
|
||||
hosts.append(host)
|
||||
if not hosts:
|
||||
warnings.append(_issue("empty_wave", wave=wave.label))
|
||||
clusters = sorted({h.cluster_name for h in hosts if h.cluster_name})
|
||||
if clusters and not wave.carp_handover:
|
||||
warnings.append(_issue("master_needs_handover", wave=wave.label, clusters=", ".join(clusters)))
|
||||
waves.append({
|
||||
"index": index, "label": wave.label, "soak_minutes": wave.soak_minutes, "carp_handover": wave.carp_handover,
|
||||
"require_approval": wave.require_approval, "groups": sorted(g.name for g in wave.groups),
|
||||
"hosts": [{"id": str(h.id), "name": h.name, "cluster_name": h.cluster_name, "enabled": h.enabled} for h in hosts],
|
||||
})
|
||||
if duplicates:
|
||||
warnings.append(_issue("duplicate_hosts", names=", ".join(sorted(duplicates))))
|
||||
all_hosts = [h for w in waves for h in w["hosts"]]
|
||||
disabled = sorted(h["name"] for h in all_hosts if not h["enabled"])
|
||||
if disabled:
|
||||
warnings.append(_issue("disabled_hosts", names=", ".join(disabled)))
|
||||
if not plan.waves:
|
||||
errors.append(_issue("no_waves"))
|
||||
elif not all_hosts:
|
||||
errors.append(_issue("no_hosts"))
|
||||
if len(plan.waves) > MAX_WAVES:
|
||||
errors.append(_issue("too_many_waves", limit=MAX_WAVES))
|
||||
if len(all_hosts) > MAX_HOSTS:
|
||||
errors.append(_issue("too_many_hosts", limit=MAX_HOSTS))
|
||||
|
||||
# Clusters: where are their nodes?
|
||||
cluster_names = sorted({h["cluster_name"] for h in all_hosts if h["cluster_name"]})
|
||||
for cluster in cluster_names:
|
||||
members = (await db.scalars(select(OpnsenseHost.id).where(OpnsenseHost.cluster_name == cluster))).all()
|
||||
in_plan = {h["id"]: w["index"] for w in waves for h in w["hosts"] if h["cluster_name"] == cluster}
|
||||
partial = len(in_plan) < len(members)
|
||||
split = len(set(in_plan.values())) > 1
|
||||
if plan.action == "major_update" and (partial or split):
|
||||
# Never two major releases side by side in one cluster longer than one wave.
|
||||
errors.append(_issue("cluster_split", cluster=cluster))
|
||||
elif partial:
|
||||
warnings.append(_issue("cluster_partial", cluster=cluster))
|
||||
elif split:
|
||||
warnings.append(_issue("mixed_versions", cluster=cluster, wave=waves[max(in_plan.values())]["label"]))
|
||||
|
||||
window = await db.get(MaintenanceWindow, plan.window_id) if plan.window_id else None
|
||||
if plan.auto_start and plan.action != "update":
|
||||
errors.append(_issue("auto_start_update_only"))
|
||||
if plan.auto_start and window is None:
|
||||
errors.append(_issue("auto_start_needs_window"))
|
||||
if window is None:
|
||||
warnings.append(_issue("no_window"))
|
||||
elif not window.enabled:
|
||||
warnings.append(_issue("window_disabled"))
|
||||
now = datetime.now(timezone.utc)
|
||||
return {
|
||||
"waves": waves, "errors": errors, "warnings": warnings,
|
||||
"window": {"id": str(window.id), "name": window.name, "definition": window_service.definition(window),
|
||||
"next": [o.as_dict() for o in window_service.occurrences(window, now, 3)] if window.enabled else []}
|
||||
if window else None,
|
||||
}
|
||||
|
||||
|
||||
def definition(plan: RolloutPlan) -> dict:
|
||||
return {
|
||||
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
||||
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
||||
}
|
||||
|
||||
|
||||
def effective_checks(checks: dict | None) -> dict:
|
||||
merged = {**rollout_health.DEFAULT_CHECKS, **{k: bool(v) for k, v in (checks or {}).items() if k in rollout_health.CHECKS}}
|
||||
for name in rollout_health.ALWAYS_ON:
|
||||
merged[name] = True
|
||||
return merged
|
||||
|
||||
|
||||
def fingerprint(plan: RolloutPlan, resolved: dict) -> str:
|
||||
"""Covers the plan, every resolved host (id, name, cluster, enabled) and the window definition: a change
|
||||
to any of them invalidates a preview and stops an activated automatic start."""
|
||||
state = {
|
||||
"plan": definition(plan),
|
||||
"hosts": [[w["index"], h["id"], h["name"], h["cluster_name"], h["enabled"]] for w in resolved["waves"] for h in w["hosts"]],
|
||||
"window": (resolved.get("window") or {}).get("definition"),
|
||||
}
|
||||
return hashlib.sha256(json.dumps(state, sort_keys=True, default=str).encode()).hexdigest()
|
||||
|
||||
|
||||
async def load_plan(db: AsyncSession, plan_id) -> RolloutPlan | None:
|
||||
return await db.scalar(
|
||||
select(RolloutPlan).where(RolloutPlan.id == plan_id).execution_options(populate_existing=True))
|
||||
@@ -19,27 +19,18 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.services import opnsense_client
|
||||
from app.services import adopt_link_service, opnsense_client
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on # noqa: F401 (NotAdoptable is re-exported for routers/rules.py)
|
||||
from app.services.policy_service import revision
|
||||
from app.services.rule_deploy_service import _build_payload
|
||||
|
||||
|
||||
class RuleNotFoundOnHost(Exception):
|
||||
class RuleNotFoundOnHost(NotFoundOnHost):
|
||||
pass
|
||||
|
||||
|
||||
class NotAdoptable(Exception):
|
||||
def __init__(self, reasons: list[str]):
|
||||
super().__init__(", ".join(reasons))
|
||||
self.reasons = reasons
|
||||
|
||||
|
||||
def _on(value) -> bool:
|
||||
return value in ("1", 1, True)
|
||||
|
||||
|
||||
def _clean(value) -> str:
|
||||
return "" if value is None else str(value).strip()
|
||||
_on = on
|
||||
_clean = clean
|
||||
|
||||
|
||||
def reasons_not_adoptable(row: dict, managed_uuids: set[str]) -> list[str]:
|
||||
@@ -73,6 +64,30 @@ def reasons_not_adoptable(row: dict, managed_uuids: set[str]) -> list[str]:
|
||||
return reasons
|
||||
|
||||
|
||||
def _values(row: dict) -> dict:
|
||||
"""The fields OPNexus manages, normalised exactly as adopting stores them (also what a link compares)."""
|
||||
return {
|
||||
"description": _clean(row.get("description")),
|
||||
"interface": _clean(row.get("interface")),
|
||||
"direction": _clean(row.get("direction")),
|
||||
"ipprotocol": _clean(row.get("ipprotocol")),
|
||||
"protocol": _clean(row.get("protocol")) or "any",
|
||||
"action": _clean(row.get("action")),
|
||||
"source_net": _clean(row.get("source_net")) or "any",
|
||||
"source_port": _clean(row.get("source_port")) or None,
|
||||
"destination_net": _clean(row.get("destination_net")) or "any",
|
||||
"destination_port": _clean(row.get("destination_port")) or None,
|
||||
"enabled": _on(row.get("enabled")),
|
||||
"log": _on(row.get("log")),
|
||||
}
|
||||
|
||||
|
||||
LINK_SPEC = adopt_link_service.LinkSpec(
|
||||
model=FirewallRule, deployment=RuleDeployment, fk="rule_id", uuid_attr="opnsense_rule_uuid",
|
||||
identity="description", payload=_build_payload, revision=revision,
|
||||
)
|
||||
|
||||
|
||||
async def _managed_uuids(db: AsyncSession, host: OpnsenseHost) -> set[str]:
|
||||
rows = await db.scalars(
|
||||
select(RuleDeployment.opnsense_rule_uuid).where(
|
||||
@@ -90,6 +105,7 @@ async def _live_rows(host: OpnsenseHost) -> list[dict]:
|
||||
|
||||
async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
|
||||
managed = await _managed_uuids(db, host)
|
||||
central = await adopt_link_service.load_central(db, LINK_SPEC)
|
||||
result = []
|
||||
for row in await _live_rows(host):
|
||||
if row.get("is_automatic"):
|
||||
@@ -97,7 +113,11 @@ async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
|
||||
reasons = reasons_not_adoptable(row, managed)
|
||||
if "already_managed" in reasons:
|
||||
continue
|
||||
matches, similar = adopt_link_service.find(central, LINK_SPEC, host, _values(row))
|
||||
result.append({
|
||||
"matches": [adopt_link_service.describe(LINK_SPEC, m) for m in matches],
|
||||
"similar": [adopt_link_service.describe(LINK_SPEC, m) for m in similar],
|
||||
"linkable": any(not m.busy for m in matches) and not reasons,
|
||||
"uuid": _clean(row.get("uuid")),
|
||||
"description": _clean(row.get("description")),
|
||||
"interface": _clean(row.get("interface")),
|
||||
@@ -117,7 +137,15 @@ async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
|
||||
return result
|
||||
|
||||
|
||||
async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str) -> FirewallRule:
|
||||
async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str, link_to=None) -> FirewallRule:
|
||||
"""Takes the rule over as a new central rule, or - with `link_to` - links it to that existing rule."""
|
||||
if link_to is not None:
|
||||
existing = await db.scalar(select(RuleDeployment).where(
|
||||
RuleDeployment.host_id == host.id, RuleDeployment.opnsense_rule_uuid == rule_uuid))
|
||||
if existing is not None:
|
||||
if existing.rule_id == link_to:
|
||||
return await db.get(FirewallRule, link_to) # second click: already linked
|
||||
raise NotAdoptable(["already_managed"])
|
||||
managed = await _managed_uuids(db, host)
|
||||
row = next((r for r in await _live_rows(host) if _clean(r.get("uuid")) == rule_uuid), None)
|
||||
if row is None:
|
||||
@@ -125,23 +153,10 @@ async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str) -> Firewal
|
||||
reasons = reasons_not_adoptable(row, managed)
|
||||
if reasons:
|
||||
raise NotAdoptable(reasons)
|
||||
if link_to is not None:
|
||||
return await adopt_link_service.link(db, LINK_SPEC, host, link_to, rule_uuid, _values(row))
|
||||
|
||||
rule = FirewallRule(
|
||||
description=_clean(row.get("description")),
|
||||
interface=_clean(row.get("interface")),
|
||||
direction=_clean(row.get("direction")),
|
||||
ipprotocol=_clean(row.get("ipprotocol")),
|
||||
protocol=_clean(row.get("protocol")) or "any",
|
||||
action=_clean(row.get("action")),
|
||||
source_net=_clean(row.get("source_net")) or "any",
|
||||
source_port=_clean(row.get("source_port")) or None,
|
||||
destination_net=_clean(row.get("destination_net")) or "any",
|
||||
destination_port=_clean(row.get("destination_port")) or None,
|
||||
enabled=_on(row.get("enabled")),
|
||||
log=_on(row.get("log")),
|
||||
targets=[host],
|
||||
groups=[],
|
||||
)
|
||||
rule = FirewallRule(**_values(row), targets=[host], groups=[])
|
||||
db.add(rule)
|
||||
await db.flush()
|
||||
revision(db, rule, "Von Firewall übernommen")
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Rule analysis (1.26): which firewall rule decides a packet, with honest confidence."""
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Everything about one firewall the analysis needs, fetched read-only and cached briefly.
|
||||
|
||||
Sources (see the evidence block in opnsense_client.py): ``firewall/filter/searchRule`` (all
|
||||
rules incl. automatic ones, with ``sort_order``), ``diagnostics/firewall/pfStatistics/rules``
|
||||
(what pf has loaded), ``interfaces/overview/interfacesInfo`` (identifier, device, addresses
|
||||
incl. CARP VIPs, gateways), ``firewall/group/searchItem`` (interface groups),
|
||||
``firewall/alias/searchItem`` (alias definitions), ``firewall/d_nat/searchRule`` (port
|
||||
forwards, only to warn about NAT) and, per queried IP, ``firewall/alias_util/findReferences``.
|
||||
Optional sources may fail; the analysis then lowers its confidence instead of guessing.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import opnsense_client
|
||||
from app.services.rule_analysis import pf_ruleset
|
||||
from app.services.rule_analysis.model import IPAddr, IPNet
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CACHE_SECONDS = 60
|
||||
# Built-in interface groups whose members are the matching dynamic devices.
|
||||
BUILTIN_GROUP_DEVICES = {"wireguard": ("wg",), "openvpn": ("ovpns", "ovpnc"), "enc0": ("enc0",)}
|
||||
|
||||
|
||||
@dataclass
|
||||
class IfInfo:
|
||||
identifier: str
|
||||
device: str
|
||||
description: str
|
||||
nets: list[IPNet] = field(default_factory=list)
|
||||
addrs: list[IPAddr] = field(default_factory=list) # primary addresses
|
||||
vips: list[IPAddr] = field(default_factory=list) # CARP / alias VIPs on this interface
|
||||
has_gateway: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class Context:
|
||||
rows: list[dict]
|
||||
pf: pf_ruleset.PfRuleset | None
|
||||
interfaces: dict[str, IfInfo]
|
||||
groups: dict[str, set[str]] | None # name -> member identifiers; None = unknown
|
||||
aliases: dict[str, dict]
|
||||
port_forwards: list[dict] | None
|
||||
refs: dict[str, list[str] | None] = field(default_factory=dict)
|
||||
notes: list[str] = field(default_factory=list)
|
||||
loaded_at: float = 0.0
|
||||
|
||||
@property
|
||||
def self_addrs(self) -> list[IPAddr]:
|
||||
addrs = [a for i in self.interfaces.values() for a in (*i.addrs, *i.vips)]
|
||||
return addrs + [ipaddress.ip_address("127.0.0.1"), ipaddress.ip_address("::1")]
|
||||
|
||||
def by_device(self, device: str) -> IfInfo | None:
|
||||
return next((i for i in self.interfaces.values() if i.device == device), None)
|
||||
|
||||
def by_label(self, label: str) -> IfInfo | None:
|
||||
lowered = label.strip().lower()
|
||||
return self.interfaces.get(lowered) or next(
|
||||
(i for i in self.interfaces.values() if i.description.lower() == lowered), None)
|
||||
|
||||
|
||||
def _interfaces(info: dict) -> dict[str, IfInfo]:
|
||||
result: dict[str, IfInfo] = {}
|
||||
for row in (info or {}).get("rows", []) or []:
|
||||
identifier = str(row.get("identifier") or "").strip()
|
||||
if not identifier:
|
||||
continue
|
||||
item = IfInfo(identifier=identifier, device=str(row.get("device") or ""),
|
||||
description=str(row.get("description") or identifier),
|
||||
has_gateway=bool(row.get("gateways")) or "default" in (row.get("routes") or []))
|
||||
for entry in [*(row.get("ipv4") or []), *(row.get("ipv6") or [])]:
|
||||
try:
|
||||
iface = ipaddress.ip_interface(str(entry.get("ipaddr")).split("%")[0])
|
||||
except ValueError:
|
||||
continue
|
||||
(item.vips if entry.get("vhid") or entry.get("vip") else item.addrs).append(iface.ip)
|
||||
if iface.network not in item.nets:
|
||||
item.nets.append(iface.network)
|
||||
result[identifier] = item
|
||||
return result
|
||||
|
||||
|
||||
def _groups(data: dict | None, interfaces: dict[str, IfInfo]) -> dict[str, set[str]] | None:
|
||||
if not isinstance(data, dict) or not isinstance(data.get("rows"), list):
|
||||
return None
|
||||
groups: dict[str, set[str]] = {}
|
||||
for row in data["rows"]:
|
||||
name = str(row.get("ifname") or "")
|
||||
members = {m.strip() for m in str(row.get("members") or "").split(",") if m.strip()}
|
||||
prefixes = BUILTIN_GROUP_DEVICES.get(name)
|
||||
if prefixes:
|
||||
members |= {i.identifier for i in interfaces.values() if i.device.startswith(prefixes)}
|
||||
if name:
|
||||
groups[name] = members
|
||||
return groups
|
||||
|
||||
|
||||
async def _optional(coro, note: str, notes: list[str]):
|
||||
try:
|
||||
return await coro
|
||||
except Exception as exc: # noqa: BLE001 - optional source: degrade, never fail the analysis
|
||||
logger.info("rule analysis: optional source failed (%s): %s", note, exc)
|
||||
notes.append(note)
|
||||
return None
|
||||
|
||||
|
||||
_CACHE: dict[str, Context] = {}
|
||||
|
||||
|
||||
async def load_context(host: OpnsenseHost, *, use_cache: bool = True) -> Context:
|
||||
key = str(host.id)
|
||||
cached = _CACHE.get(key)
|
||||
if use_cache and cached and time.monotonic() - cached.loaded_at < CACHE_SECONDS:
|
||||
return cached
|
||||
notes: list[str] = []
|
||||
rules, info = await asyncio.gather(
|
||||
opnsense_client.search_rules(host, all_rows=True), opnsense_client.get_interfaces_info(host))
|
||||
stats, groups, aliases, forwards = await asyncio.gather(
|
||||
_optional(opnsense_client.get_pf_rule_statistics(host), "PF_RULESET_UNAVAILABLE", notes),
|
||||
_optional(opnsense_client.search_interface_groups(host), "GROUPS_UNAVAILABLE", notes),
|
||||
_optional(opnsense_client._request(host, "GET", "/firewall/alias/searchItem", params={"rowCount": -1}),
|
||||
"ALIASES_UNAVAILABLE", notes),
|
||||
_optional(opnsense_client.search_port_forward_rules(host), "NAT_UNAVAILABLE", notes),
|
||||
)
|
||||
interfaces = _interfaces(info)
|
||||
pf = pf_ruleset.parse(stats)
|
||||
if stats is not None and pf is None:
|
||||
notes.append("PF_RULESET_UNAVAILABLE")
|
||||
ctx = Context(
|
||||
rows=list((rules or {}).get("rows") or []), pf=pf, interfaces=interfaces,
|
||||
groups=_groups(groups, interfaces),
|
||||
aliases={str(a.get("name")): a for a in ((aliases or {}).get("rows") or []) if a.get("name")},
|
||||
port_forwards=None if forwards is None else list(forwards.get("rows") or []),
|
||||
notes=notes, loaded_at=time.monotonic(),
|
||||
)
|
||||
_CACHE[key] = ctx
|
||||
return ctx
|
||||
|
||||
|
||||
async def ensure_refs(host: OpnsenseHost, ctx: Context, ips: list[str]) -> None:
|
||||
"""Ask pf which tables contain each IP (cached in the context)."""
|
||||
for ip in ips:
|
||||
if ip in ctx.refs:
|
||||
continue
|
||||
try:
|
||||
ctx.refs[ip] = await opnsense_client.find_alias_references(host, ip)
|
||||
except Exception as exc: # noqa: BLE001 - table membership becomes "unknown"
|
||||
logger.info("rule analysis: findReferences failed: %s", exc)
|
||||
ctx.refs[ip] = None
|
||||
|
||||
|
||||
def clear_cache() -> None:
|
||||
_CACHE.clear()
|
||||
@@ -0,0 +1,317 @@
|
||||
"""Which rule decides a packet, following pf semantics.
|
||||
|
||||
pf walks the loaded ruleset top to bottom; the *last* matching rule wins unless a matching
|
||||
rule is ``quick``, which ends the walk. OPNsense orders rules by ``sort_order`` (system,
|
||||
floating, groups, interfaces). The analysis looks at the first packet of a new connection.
|
||||
|
||||
Confidence:
|
||||
- ``rule_confidence`` is "sure" only when no rule that could still change the outcome
|
||||
matched with an unknown result;
|
||||
- ``confidence`` (the verdict) is "sure" when every such unknown rule has the same action as
|
||||
the deciding rule (then the verdict holds either way), "unknown" otherwise;
|
||||
- notes about guesses (interface guessed from the address, NAT nearby, pf ruleset missing,
|
||||
TCP flags, rate limits) lower a "sure" to "probable".
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from app.services.rule_analysis import resolve
|
||||
from app.services.rule_analysis.context import Context
|
||||
from app.services.rule_analysis.model import IPAddr, NormRule, Outcome, t_and, tri_str
|
||||
from app.services.rule_analysis.normalize import norm_proto, normalize
|
||||
|
||||
SURE, PROBABLE, UNKNOWN = "sure", "probable", "unknown"
|
||||
_RANK = {SURE: 2, PROBABLE: 1, UNKNOWN: 0}
|
||||
DOWNGRADING_NOTES = {
|
||||
"IFACE_GUESSED", "EGRESS_GUESSED", "NAT_NEARBY", "TCP_FLAGS", "RATE_LIMIT", "ORDER_GUESSED",
|
||||
"ORDER_MISMATCH", "PF_RULESET_UNAVAILABLE", "NAT_UNAVAILABLE", "QUICK_UNKNOWN",
|
||||
}
|
||||
_PORT_PROTOS = {"6", "17"}
|
||||
_ICMP_PROTOS = {"1", "58"}
|
||||
ICMP_NAMES = {
|
||||
4: {"0": "echorep", "3": "unreach", "4": "squench", "5": "redir", "8": "echoreq", "9": "routeradv",
|
||||
"10": "routersol", "11": "timex", "12": "paramprob", "13": "timereq", "14": "timerep"},
|
||||
6: {"1": "unreach", "2": "toobig", "3": "timex", "4": "paramprob", "128": "echoreq", "129": "echorep",
|
||||
"133": "routersol", "134": "routeradv", "135": "neighbrsol", "136": "neighbradv", "137": "redir"},
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Query:
|
||||
src: IPAddr
|
||||
dst: IPAddr
|
||||
proto: str # normalized ("6", "17", "1", "58", other number or name)
|
||||
proto_label: str
|
||||
sport: int | None = None
|
||||
dport: int | None = None
|
||||
icmptype: str | None = None
|
||||
iface: str | None = None # None = guess from the source address
|
||||
direction: str = "in"
|
||||
egress: str | None = "none" # "none" = only the first leg, None = guess, else identifier
|
||||
|
||||
@property
|
||||
def af(self) -> int:
|
||||
return self.src.version
|
||||
|
||||
|
||||
@dataclass
|
||||
class LegResult:
|
||||
direction: str
|
||||
iface: str | None
|
||||
iface_source: str
|
||||
action: str | None
|
||||
decisive: NormRule | None
|
||||
confidence: str
|
||||
rule_confidence: str
|
||||
notes: list[str] = field(default_factory=list)
|
||||
unknowns: list[str] = field(default_factory=list)
|
||||
steps: list[dict] = field(default_factory=list)
|
||||
|
||||
|
||||
def _min_conf(*values: str) -> str:
|
||||
return min(values, key=lambda v: _RANK[v])
|
||||
|
||||
|
||||
def _max_conf(*values: str) -> str:
|
||||
return max(values, key=lambda v: _RANK[v])
|
||||
|
||||
|
||||
def _icmp_name(q: Query) -> str | None:
|
||||
if q.icmptype is None:
|
||||
return None
|
||||
return ICMP_NAMES[q.af].get(q.icmptype, q.icmptype)
|
||||
|
||||
|
||||
def rule_fields(r: NormRule, q: Query, direction: str, iface: str, ctx: Context,
|
||||
tags: set[str], first_leg_iface: str | None) -> dict[str, Outcome]:
|
||||
fields: dict[str, Outcome] = {
|
||||
"interface": resolve.iface_match(r.interfaces, r.if_not, iface, ctx),
|
||||
"direction": Outcome(r.direction in ("any", direction)),
|
||||
"family": Outcome(q.af in r.af),
|
||||
"protocol": Outcome(True if r.protos is None else q.proto in r.protos),
|
||||
"source": resolve.endpoint_match(r.src, q.src, ctx),
|
||||
"destination": resolve.endpoint_match(r.dst, q.dst, ctx),
|
||||
}
|
||||
if q.proto in _PORT_PROTOS:
|
||||
fields["source_port"] = resolve.port_match(r.sport, q.sport, ctx, ephemeral_default=True)
|
||||
fields["destination_port"] = resolve.port_match(r.dport, q.dport, ctx)
|
||||
elif r.sport or r.dport:
|
||||
fields["ports"] = Outcome(False) # pf only generates port rules for tcp/udp
|
||||
if r.icmptypes is not None and q.proto in _ICMP_PROTOS:
|
||||
name = _icmp_name(q)
|
||||
fields["icmp_type"] = Outcome(None, ["ICMP_TYPE_NOT_GIVEN"]) if name is None else \
|
||||
Outcome(name in r.icmptypes or q.icmptype in r.icmptypes)
|
||||
if r.tagged:
|
||||
fields["tagged"] = Outcome(True) if r.tagged in tags else Outcome(None, ["TAGGED"])
|
||||
if r.received_on:
|
||||
if direction == "in":
|
||||
fields["received_on"] = Outcome(r.received_on == iface)
|
||||
elif first_leg_iface:
|
||||
fields["received_on"] = Outcome(r.received_on == first_leg_iface)
|
||||
else:
|
||||
fields["received_on"] = Outcome(None, ["RECEIVED_ON"])
|
||||
if r.action is None:
|
||||
fields["action"] = Outcome(None, ["ACTION_UNKNOWN"])
|
||||
return fields
|
||||
|
||||
|
||||
def _step(r: NormRule, fields: dict[str, Outcome], result, effect: str) -> dict:
|
||||
return {
|
||||
"rule": r.summary(), "result": tri_str(result), "effect": effect,
|
||||
"fields": {k: tri_str(v.value) for k, v in fields.items()},
|
||||
"reasons": sorted({reason for v in fields.values() if v.value is None for reason in v.reasons}),
|
||||
}
|
||||
|
||||
|
||||
def eval_leg(rules: list[NormRule], q: Query, ctx: Context, direction: str, iface: str,
|
||||
first_leg_iface: str | None = None) -> LegResult:
|
||||
steps: list[dict] = []
|
||||
candidate: NormRule | None = None
|
||||
possible: list[tuple[NormRule, bool]] = [] # (rule, came after the current candidate)
|
||||
tags: set[str] = set()
|
||||
final: NormRule | None = None
|
||||
for r in rules:
|
||||
if not r.enabled:
|
||||
steps.append({"rule": r.summary(), "result": "false", "effect": "disabled", "fields": {}, "reasons": []})
|
||||
continue
|
||||
if r.loaded is False:
|
||||
steps.append({"rule": r.summary(), "result": "false", "effect": "not_loaded", "fields": {}, "reasons": []})
|
||||
continue
|
||||
fields = rule_fields(r, q, direction, iface, ctx, tags, first_leg_iface)
|
||||
result = t_and(*(f.value for f in fields.values()))
|
||||
if result is False:
|
||||
steps.append(_step(r, fields, result, "no_match"))
|
||||
continue
|
||||
if result is None or r.quick is None:
|
||||
possible.append((r, True))
|
||||
steps.append(_step(r, fields, result, "possible"))
|
||||
continue
|
||||
if r.quick:
|
||||
final = r
|
||||
for s in steps:
|
||||
if s["effect"] == "candidate":
|
||||
s["effect"] = "overridden"
|
||||
steps.append(_step(r, fields, result, "final"))
|
||||
break
|
||||
if candidate is not None:
|
||||
for s in steps:
|
||||
if s["effect"] == "candidate":
|
||||
s["effect"] = "overridden"
|
||||
candidate = r
|
||||
possible = [(p, False) for p, _ in possible]
|
||||
if r.tag:
|
||||
tags.add(r.tag)
|
||||
steps.append(_step(r, fields, result, "candidate"))
|
||||
decisive = final or candidate
|
||||
if final is None:
|
||||
for s in steps:
|
||||
if s["effect"] == "candidate":
|
||||
s["effect"] = "decisive"
|
||||
else:
|
||||
steps[-1]["effect"] = "decisive"
|
||||
action = decisive.action if decisive else "pass" # pf without any matching rule passes
|
||||
# Unknown matches that could still change the outcome: quick (or quick-unknown) ones always,
|
||||
# non-quick ones only after the deciding candidate (an earlier one is overridden anyway).
|
||||
relevant = [p for p, after in possible if p.quick is not False or (final is None and after)]
|
||||
rule_conf = SURE if not relevant else PROBABLE
|
||||
conf = SURE if all(p.action == action for p in relevant) else UNKNOWN
|
||||
unknowns = sorted({reason for s in steps if s["effect"] == "possible" for reason in s["reasons"]})
|
||||
if any(p.quick is None for p, _ in possible):
|
||||
unknowns = sorted({*unknowns, "QUICK_UNKNOWN"})
|
||||
notes: list[str] = []
|
||||
if decisive is None:
|
||||
notes.append("PF_DEFAULT_PASS")
|
||||
else:
|
||||
notes += decisive.extra_notes
|
||||
if decisive.gateway and action == "pass":
|
||||
notes.append("POLICY_ROUTE")
|
||||
if decisive.statetype.lower() == "none" or "no state" in getattr(decisive.pf_line, "text", ""):
|
||||
notes.append("STATELESS")
|
||||
if decisive.sched:
|
||||
notes.append("SCHEDULE_NOW")
|
||||
if any(s["effect"] == "not_loaded" and s["rule"]["loaded"] is False for s in steps):
|
||||
notes.append("RULES_NOT_LOADED")
|
||||
return LegResult(direction=direction, iface=iface, iface_source="given", action=action, decisive=decisive,
|
||||
confidence=conf, rule_confidence=rule_conf, notes=notes, unknowns=unknowns, steps=steps)
|
||||
|
||||
|
||||
def guess_iface(ip: IPAddr, ctx: Context) -> str | None:
|
||||
"""The interface whose connected network holds ``ip`` (longest prefix), else the one with
|
||||
the default gateway."""
|
||||
best, best_len = None, -1
|
||||
for info in ctx.interfaces.values():
|
||||
if info.identifier == "lo0":
|
||||
continue
|
||||
for net in info.nets:
|
||||
if net.version == ip.version and ip in net and net.prefixlen > best_len and not net.is_link_local:
|
||||
best, best_len = info.identifier, net.prefixlen
|
||||
if best:
|
||||
return best
|
||||
gateways = [i.identifier for i in ctx.interfaces.values() if i.has_gateway]
|
||||
return gateways[0] if len(gateways) == 1 else None
|
||||
|
||||
|
||||
def _nat_nearby(q: Query, iface: str, ctx: Context) -> bool:
|
||||
for row in ctx.port_forwards or []:
|
||||
if str(row.get("disabled") or "0") == "1" or str(row.get("enabled", "1")) == "0":
|
||||
continue
|
||||
if str(row.get("nordr") or "0") == "1": # "no rdr" exclusions (anti-lockout) redirect nothing
|
||||
continue
|
||||
if q.dport is not None and resolve.port_match(str(row.get("destination.port") or ""), q.dport, ctx).value is False:
|
||||
continue
|
||||
ifaces = {i.strip().lower() for i in str(row.get("interface") or "").split(",") if i.strip()}
|
||||
if ifaces and iface not in ifaces:
|
||||
continue
|
||||
proto = str(row.get("protocol") or "").lower()
|
||||
if proto and proto not in ("any", "tcp/udp") and norm_proto(proto) != q.proto:
|
||||
continue
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def evaluate(ctx: Context, q: Query) -> dict:
|
||||
rules, order_notes = normalize(ctx)
|
||||
context_notes = [*ctx.notes, *order_notes]
|
||||
legs: list[LegResult] = []
|
||||
iface, iface_source = q.iface, "given"
|
||||
if iface is None:
|
||||
iface, iface_source = guess_iface(q.src if q.direction == "in" else q.dst, ctx), "guessed"
|
||||
if iface is None:
|
||||
return _result(q, legs, context_notes + ["IFACE_UNRESOLVED"], unresolved=True)
|
||||
first = eval_leg(rules, q, ctx, q.direction, iface)
|
||||
first.iface_source = iface_source
|
||||
if iface_source == "guessed":
|
||||
first.notes.append("IFACE_GUESSED")
|
||||
if q.direction == "in" and (ctx.port_forwards is None or _nat_nearby(q, iface, ctx)):
|
||||
first.notes.append("NAT_NEARBY" if ctx.port_forwards is not None else "NAT_UNAVAILABLE")
|
||||
legs.append(first)
|
||||
if q.direction == "in" and q.egress != "none":
|
||||
egress, source = q.egress, "given"
|
||||
if egress is None:
|
||||
egress, source = guess_iface(q.dst, ctx), "guessed"
|
||||
if egress is None:
|
||||
context_notes.append("EGRESS_UNRESOLVED")
|
||||
else:
|
||||
out = eval_leg(rules, q, ctx, "out", egress, first_leg_iface=iface)
|
||||
out.iface_source = source
|
||||
if source == "guessed":
|
||||
out.notes.append("EGRESS_GUESSED")
|
||||
legs.append(out)
|
||||
return _result(q, legs, context_notes)
|
||||
|
||||
|
||||
def _leg_conf(leg: LegResult, context_notes: list[str]) -> tuple[str, str]:
|
||||
downgrade = any(n in DOWNGRADING_NOTES for n in (*leg.notes, *context_notes))
|
||||
conf, rule_conf = leg.confidence, leg.rule_confidence
|
||||
if downgrade:
|
||||
conf = _min_conf(conf, PROBABLE)
|
||||
rule_conf = _min_conf(rule_conf, PROBABLE)
|
||||
return conf, rule_conf
|
||||
|
||||
|
||||
def _result(q: Query, legs: list[LegResult], context_notes: list[str], unresolved: bool = False) -> dict:
|
||||
leg_dicts, confs = [], []
|
||||
for leg in legs:
|
||||
conf, rule_conf = _leg_conf(leg, context_notes)
|
||||
confs.append((leg, conf, rule_conf))
|
||||
leg_dicts.append({
|
||||
"direction": leg.direction, "interface": leg.iface, "interface_source": leg.iface_source,
|
||||
"action": leg.action, "confidence": conf, "rule_confidence": rule_conf,
|
||||
"decisive_rule": leg.decisive.summary() if leg.decisive else None,
|
||||
"notes": sorted(set(leg.notes)), "unknowns": leg.unknowns,
|
||||
})
|
||||
if unresolved or not legs:
|
||||
action, conf, rule_conf, decisive = None, UNKNOWN, UNKNOWN, None
|
||||
else:
|
||||
blocking = [c for c in confs if c[0].action != "pass"]
|
||||
if blocking:
|
||||
leg, _, rule_conf = blocking[0]
|
||||
action, decisive = leg.action, leg.decisive
|
||||
# Blocked if any blocking leg is right: the best of their confidences counts.
|
||||
conf = _max_conf(*(c[1] for c in blocking))
|
||||
if conf == SURE and any(c[0].action != action for c in blocking):
|
||||
conf = PROBABLE
|
||||
else:
|
||||
leg, _, rule_conf = confs[0]
|
||||
action, decisive = "pass", leg.decisive
|
||||
conf = _min_conf(*(c[1] for c in confs))
|
||||
return {
|
||||
"query": {
|
||||
"src": str(q.src), "dst": str(q.dst), "proto": q.proto_label, "sport": q.sport, "dport": q.dport,
|
||||
"icmptype": q.icmptype, "direction": q.direction,
|
||||
"assumptions": ["FIRST_PACKET"] + (["SRC_PORT_EPHEMERAL"] if q.sport is None and q.proto in _PORT_PROTOS else []),
|
||||
},
|
||||
"action": action, "confidence": conf, "rule_confidence": rule_conf,
|
||||
"decisive_rule": decisive.summary() if decisive else None,
|
||||
"notes": sorted({*context_notes, *(n for leg in legs for n in leg.notes)}),
|
||||
"unknowns": sorted({u for leg in legs for u in leg.unknowns}),
|
||||
"legs": leg_dicts,
|
||||
"trace": [{"leg": i, "direction": leg.direction, "interface": leg.iface, "steps": leg.steps}
|
||||
for i, leg in enumerate(legs)],
|
||||
}
|
||||
|
||||
|
||||
def parse_ip(value: str) -> IPAddr:
|
||||
return ipaddress.ip_address(value.strip())
|
||||
@@ -0,0 +1,122 @@
|
||||
"""Shared types of the rule analysis: three-valued logic and the normalized rule.
|
||||
|
||||
``Tri`` is Kleene logic with ``True``/``False``/``None`` (unknown). An unknown never turns
|
||||
into a definite answer by negation, and an AND is only False when one side is definitely
|
||||
False. This is what keeps the engine from claiming "sure" for something it cannot prove.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
Tri = Optional[bool]
|
||||
|
||||
IPAddr = ipaddress.IPv4Address | ipaddress.IPv6Address
|
||||
IPNet = ipaddress.IPv4Network | ipaddress.IPv6Network
|
||||
|
||||
|
||||
def t_and(*values: Tri) -> Tri:
|
||||
if any(v is False for v in values):
|
||||
return False
|
||||
if any(v is None for v in values):
|
||||
return None
|
||||
return True
|
||||
|
||||
|
||||
def t_or(*values: Tri) -> Tri:
|
||||
if any(v is True for v in values):
|
||||
return True
|
||||
if any(v is None for v in values):
|
||||
return None
|
||||
return False
|
||||
|
||||
|
||||
def t_not(value: Tri) -> Tri:
|
||||
return None if value is None else not value
|
||||
|
||||
|
||||
def tri_str(value: Tri) -> str:
|
||||
return "unknown" if value is None else ("true" if value else "false")
|
||||
|
||||
|
||||
class Outcome:
|
||||
"""A Tri plus the reason codes that made it unknown (or noteworthy)."""
|
||||
__slots__ = ("value", "reasons")
|
||||
|
||||
def __init__(self, value: Tri, reasons: tuple[str, ...] | list[str] = ()):
|
||||
self.value = value
|
||||
self.reasons = list(reasons)
|
||||
|
||||
def __repr__(self) -> str: # pragma: no cover - debugging aid
|
||||
return f"Outcome({self.value!r}, {self.reasons!r})"
|
||||
|
||||
|
||||
def o_or(outcomes: list[Outcome]) -> Outcome:
|
||||
value = t_or(*(o.value for o in outcomes)) if outcomes else False
|
||||
reasons = [] if value is True else [r for o in outcomes if o.value is None for r in o.reasons]
|
||||
return Outcome(value, reasons)
|
||||
|
||||
|
||||
SECTION_SYSTEM, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE = "system", "floating", "group", "interface"
|
||||
SECTION_NAT_AUTO, SECTION_INVALID, SECTION_ANTISPOOF, SECTION_OTHER = "nat_auto", "invalid", "antispoof", "other"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Endpoint:
|
||||
raw: str
|
||||
negated: bool = False
|
||||
items: list[str] = field(default_factory=list) # empty = any
|
||||
|
||||
@property
|
||||
def is_any(self) -> bool:
|
||||
return not self.items
|
||||
|
||||
|
||||
@dataclass
|
||||
class NormRule:
|
||||
uuid: str
|
||||
label: str
|
||||
description: str
|
||||
order: tuple[int, int]
|
||||
order_known: bool
|
||||
section: str
|
||||
enabled: bool
|
||||
action: str | None # pass | block | reject | None (unknown)
|
||||
quick: bool | None # None = unknown (automatic rule without pf ruleset)
|
||||
interfaces: set[str] # identifiers or group names; empty = every interface
|
||||
if_not: bool
|
||||
direction: str # in | out | any
|
||||
af: set[int] # {4}, {6} or {4, 6}
|
||||
protos: set[str] | None # None = any; normalized names/numbers as strings
|
||||
src: Endpoint
|
||||
dst: Endpoint
|
||||
sport: str
|
||||
dport: str
|
||||
icmptypes: set[str] | None = None # None = no restriction
|
||||
tag: str = ""
|
||||
tagged: str = ""
|
||||
received_on: str = ""
|
||||
gateway: str = ""
|
||||
statetype: str = ""
|
||||
sched: str = ""
|
||||
automatic: bool = False
|
||||
legacy: bool = False
|
||||
loaded: bool | None = None # None = pf ruleset not available
|
||||
pf_numbers: list[int] = field(default_factory=list)
|
||||
extra_notes: list[str] = field(default_factory=list) # e.g. TCP_FLAGS, RATE_LIMIT from pf text
|
||||
pf_line: object = None # set for unlabeled pf lines (antispoof)
|
||||
|
||||
def summary(self) -> dict:
|
||||
return {
|
||||
"uuid": self.uuid, "label": self.label, "description": self.description,
|
||||
"section": self.section, "sort_order": f"{self.order[0]:06d}.{self.order[1]:07d}" if self.order_known else None,
|
||||
"pf_numbers": self.pf_numbers, "action": self.action, "quick": self.quick,
|
||||
"interfaces": sorted(self.interfaces), "interface_not": self.if_not, "direction": self.direction,
|
||||
"automatic": self.automatic, "legacy": self.legacy, "loaded": self.loaded,
|
||||
"editable": not self.automatic and not self.legacy and self.section != SECTION_ANTISPOOF,
|
||||
"source": ("!" if self.src.negated else "") + (self.src.raw or "any"),
|
||||
"destination": ("!" if self.dst.negated else "") + (self.dst.raw or "any"),
|
||||
"source_port": self.sport or None, "destination_port": self.dport or None,
|
||||
"gateway": self.gateway or None, "schedule": self.sched or None,
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Turn ``searchRule`` rows (+ the loaded pf ruleset) into ``NormRule`` objects.
|
||||
|
||||
Format pitfalls seen live on OPNsense 26.7:
|
||||
- automatic rows carry the interface *description* (``"LAN"``), MVC rows the identifier;
|
||||
- automatic rows often have ``quick``/``action`` empty; the pf lines with the same label
|
||||
know the truth (``pass in quick on vtnet1_vlan10 ...``);
|
||||
- empty source/destination means any; address lists are comma separated;
|
||||
- unmigrated legacy rules have a 32-hex label instead of a UUID.
|
||||
|
||||
Order is ``sort_order`` (``prio.0seq`` from FilterRuleField::getPriority: 1/5 system,
|
||||
10000 force gateway, 200000 floating, 300000+n groups, 400000 interface, 500000 automatic
|
||||
NAT, 600000 = interface missing, never loaded).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from app.services.rule_analysis.context import Context
|
||||
from app.services.rule_analysis.model import (
|
||||
SECTION_ANTISPOOF, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE, SECTION_INVALID,
|
||||
SECTION_NAT_AUTO, SECTION_OTHER, SECTION_SYSTEM, Endpoint, NormRule,
|
||||
)
|
||||
from app.services.rule_analysis.pf_ruleset import PfLine
|
||||
|
||||
PROTO_NUMBERS = {
|
||||
"icmp": "1", "igmp": "2", "ggp": "3", "ipencap": "4", "tcp": "6", "egp": "8", "udp": "17",
|
||||
"ipv6": "41", "gre": "47", "esp": "50", "ah": "51", "ipv6-icmp": "58", "icmp6": "58",
|
||||
"icmpv6": "58", "ospf": "89", "pim": "103", "vrrp": "112", "carp": "112", "sctp": "132", "pfsync": "240",
|
||||
}
|
||||
|
||||
|
||||
def norm_proto(value: str) -> str:
|
||||
value = str(value).strip().lower()
|
||||
return PROTO_NUMBERS.get(value, value)
|
||||
|
||||
|
||||
def _truthy(value) -> bool:
|
||||
return value is True or str(value).strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def _section(prio: int) -> str:
|
||||
if prio < 10001:
|
||||
return SECTION_SYSTEM
|
||||
if prio == 200000:
|
||||
return SECTION_FLOATING
|
||||
if 300000 <= prio < 400000:
|
||||
return SECTION_GROUP
|
||||
if prio == 400000:
|
||||
return SECTION_INTERFACE
|
||||
if prio == 500000:
|
||||
return SECTION_NAT_AUTO
|
||||
if prio >= 600000:
|
||||
return SECTION_INVALID
|
||||
return SECTION_OTHER
|
||||
|
||||
|
||||
def _order(row: dict) -> tuple[tuple[int, int], bool]:
|
||||
raw = str(row.get("sort_order") or "")
|
||||
if "." in raw:
|
||||
head, _, tail = raw.partition(".")
|
||||
if head.isdigit() and tail.isdigit():
|
||||
return (int(head), int(tail)), True
|
||||
# Older releases without sort_order: best effort from prio_group/sequence (confidence drops).
|
||||
prio = str(row.get("prio_group") or row.get("#priority") or "")
|
||||
seq = str(row.get("sequence") or "")
|
||||
return (int(prio) if prio.isdigit() else 400000, int(seq) if seq.isdigit() else 0), False
|
||||
|
||||
|
||||
def _endpoint(row: dict, side: str) -> Endpoint:
|
||||
raw = str(row.get(f"{side}_net") or "").strip()
|
||||
items = [] if raw.lower() in ("", "any") else [i.strip() for i in raw.split(",") if i.strip()]
|
||||
return Endpoint(raw=raw, negated=_truthy(row.get(f"{side}_not")), items=items)
|
||||
|
||||
|
||||
def _af(value: str) -> set[int]:
|
||||
value = str(value or "").strip().lower()
|
||||
return {4} if value == "inet" else ({6} if value == "inet6" else {4, 6})
|
||||
|
||||
|
||||
def _protos(value: str) -> set[str] | None:
|
||||
value = str(value or "").strip().lower()
|
||||
if value in ("", "any"):
|
||||
return None
|
||||
return {norm_proto(p) for p in value.split("/") if p}
|
||||
|
||||
|
||||
def _pf_interfaces(lines: list[PfLine], ctx: Context) -> tuple[set[str], bool] | None:
|
||||
devices = {(line.on_device, line.on_negated) for line in lines}
|
||||
if any(device is None for device, _ in devices):
|
||||
return set(), False
|
||||
if len({neg for _, neg in devices}) != 1:
|
||||
return None
|
||||
names: set[str] = set()
|
||||
for device, _ in devices:
|
||||
info = ctx.by_device(device)
|
||||
if info is not None:
|
||||
names.add(info.identifier)
|
||||
elif ctx.groups is not None and device in ctx.groups:
|
||||
names.add(device)
|
||||
else:
|
||||
return None
|
||||
return names, next(iter(devices))[1]
|
||||
|
||||
|
||||
def normalize_row(row: dict, ctx: Context) -> NormRule:
|
||||
order, order_known = _order(row)
|
||||
label = str(row.get("uuid") or "")
|
||||
automatic = _truthy(row.get("is_automatic"))
|
||||
lines = ctx.pf.by_label.get(label, []) if ctx.pf else []
|
||||
rule = NormRule(
|
||||
uuid=label, label=label, description=str(row.get("description") or ""),
|
||||
order=order, order_known=order_known, section=_section(order[0]),
|
||||
enabled=_truthy(row.get("enabled", "1")),
|
||||
action=(str(row.get("action") or "").lower() or None),
|
||||
quick=None, interfaces=set(), if_not=_truthy(row.get("interfacenot")),
|
||||
direction=(str(row.get("direction") or "in").lower() or "in"),
|
||||
af=_af(row.get("ipprotocol")), protos=_protos(row.get("protocol")),
|
||||
src=_endpoint(row, "source"), dst=_endpoint(row, "destination"),
|
||||
sport=str(row.get("source_port") or "").strip(), dport=str(row.get("destination_port") or "").strip(),
|
||||
tag=str(row.get("tag") or ""), tagged=str(row.get("tagged") or ""),
|
||||
received_on=str(row.get("received-on") or row.get("received_on") or ""),
|
||||
gateway=str(row.get("gateway") or ""), statetype=str(row.get("statetype") or ""),
|
||||
sched=str(row.get("sched") or ""), automatic=automatic, legacy=_truthy(row.get("legacy")),
|
||||
pf_numbers=[line.number for line in lines],
|
||||
)
|
||||
if rule.action not in ("pass", "block", "reject"):
|
||||
rule.action = None
|
||||
if row.get("quick") not in (None, ""):
|
||||
rule.quick = _truthy(row.get("quick"))
|
||||
raw_if = str(row.get("interface") or "").strip()
|
||||
names = {n.strip() for n in raw_if.split(",") if n.strip()}
|
||||
# Automatic and unmigrated legacy rows carry descriptions ("LAN"): map them to identifiers.
|
||||
# Identifiers and group names stay as they are; the pf "on" clause decides when available.
|
||||
mapped = set()
|
||||
for name in names:
|
||||
info = None if name in ctx.interfaces or name in (ctx.groups or {}) else ctx.by_label(name)
|
||||
mapped.add(info.identifier if info else name)
|
||||
names = mapped
|
||||
rule.interfaces = names
|
||||
if ctx.pf is not None:
|
||||
rule.loaded = bool(lines) and rule.enabled
|
||||
if lines:
|
||||
actions = {line.action for line in lines}
|
||||
if len(actions) == 1 and next(iter(actions)) in ("pass", "block", "reject"):
|
||||
pf_action = next(iter(actions))
|
||||
if rule.action is None or automatic:
|
||||
rule.action = pf_action
|
||||
quick = {line.quick for line in lines}
|
||||
rule.quick = next(iter(quick)) if len(quick) == 1 else None
|
||||
if automatic or rule.legacy:
|
||||
on = _pf_interfaces(lines, ctx)
|
||||
if on is not None:
|
||||
rule.interfaces, rule.if_not = on
|
||||
directions = {line.direction for line in lines}
|
||||
if len(directions) == 1:
|
||||
rule.direction = next(iter(directions))
|
||||
families = {line.af for line in lines}
|
||||
if None not in families:
|
||||
rule.af &= families
|
||||
types = {t for line in lines for t in line.icmptypes}
|
||||
if types:
|
||||
rule.icmptypes = types
|
||||
if any(line.flags not in (None, "S/SA", "any") for line in lines):
|
||||
rule.extra_notes.append("TCP_FLAGS")
|
||||
if any(word in line.text for line in lines for word in ("max-src-", "overload", "max-pkt-rate")):
|
||||
rule.extra_notes.append("RATE_LIMIT")
|
||||
elif rule.section == SECTION_INVALID:
|
||||
rule.loaded = False
|
||||
if rule.icmptypes is None and str(row.get("icmptype") or "").strip():
|
||||
rule.icmptypes = {t.strip() for t in str(row["icmptype"]).split(",") if t.strip()}
|
||||
return rule
|
||||
|
||||
|
||||
def antispoof_rule(line: PfLine, order: tuple[int, int]) -> NormRule:
|
||||
"""An unlabeled pf line OPNsense generated itself (antispoof / own-address block)."""
|
||||
return NormRule(
|
||||
uuid=f"pf@{line.number}", label="", description=line.text, order=order, order_known=True,
|
||||
section=SECTION_ANTISPOOF, enabled=True, action=line.action if line.action != "other" else None,
|
||||
quick=line.quick, interfaces=set(), if_not=False, direction=line.direction,
|
||||
af={line.af} if line.af else {4, 6}, protos=None,
|
||||
src=Endpoint(raw="" if line.src == "any" else line.src, items=[] if line.src == "any" else [line.src]),
|
||||
dst=Endpoint(raw="" if line.dst == "any" else line.dst, items=[] if line.dst == "any" else [line.dst]),
|
||||
sport="", dport="", automatic=True, loaded=True, pf_numbers=[line.number], pf_line=line,
|
||||
)
|
||||
|
||||
|
||||
def normalize(ctx: Context) -> tuple[list[NormRule], list[str]]:
|
||||
"""All rules in evaluation order plus context notes (order problems)."""
|
||||
notes: list[str] = []
|
||||
rules = [normalize_row(row, ctx) for row in ctx.rows]
|
||||
if any(not r.order_known for r in rules):
|
||||
notes.append("ORDER_GUESSED")
|
||||
rules.sort(key=lambda r: (r.order, r.uuid))
|
||||
if ctx.pf is not None:
|
||||
loaded = [r for r in rules if r.loaded and r.pf_numbers]
|
||||
firsts = [min(r.pf_numbers) for r in loaded]
|
||||
if firsts != sorted(firsts):
|
||||
notes.append("ORDER_MISMATCH")
|
||||
# Unlabeled pf lines: placed before the first configured rule that pf evaluates later.
|
||||
extra: list[tuple[tuple, NormRule]] = []
|
||||
for line in ctx.pf.unlabeled:
|
||||
after = next((r for r in loaded if min(r.pf_numbers) > line.number), None)
|
||||
key = after.order if after else (999999, 9999999)
|
||||
spoof = antispoof_rule(line, key)
|
||||
if line.on_device:
|
||||
info = ctx.by_device(line.on_device)
|
||||
spoof.interfaces = {info.identifier if info else line.on_device}
|
||||
spoof.if_not = line.on_negated
|
||||
extra.append(((key, 0, line.number), spoof))
|
||||
keyed = [((r.order, 1, 0), r) for r in rules] + extra
|
||||
keyed.sort(key=lambda item: item[0])
|
||||
rules = [r for _, r in keyed]
|
||||
return rules, notes
|
||||
@@ -0,0 +1,93 @@
|
||||
"""The ruleset pf has actually loaded (``diagnostics/firewall/pfStatistics/rules``).
|
||||
|
||||
Each key looks like ``@76 pass in quick on vtnet1_vlan10 inet from (...) to any flags S/SA
|
||||
keep state label "<uuid>"``. One configured rule can expand to several pf lines (address
|
||||
families, ICMP types, port lists); they share one label. Lines without a label are generated
|
||||
by OPNsense itself (antispoof: ``block drop in log on ! vtnet1_vlan10 inet from 192.0.2.0/24
|
||||
to any``). Only what the analysis needs is parsed: position, action, quick, direction, the
|
||||
``on`` interface, address family, ICMP types and a few pf options that make the answer
|
||||
uncertain.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
_HEAD = re.compile(r"^@(\d+)\s+(.*)$")
|
||||
_LABEL = re.compile(r'label "([^"]+)"')
|
||||
_ON = re.compile(r"\bon (!\s*)?([A-Za-z0-9_.]+)")
|
||||
_ICMP = re.compile(r"\bicmp6?-type (\S+)")
|
||||
_FROM = re.compile(r"\bfrom (\S+) to (\S+)")
|
||||
_FLAGS = re.compile(r"\bflags (\S+)")
|
||||
|
||||
|
||||
@dataclass
|
||||
class PfLine:
|
||||
number: int
|
||||
text: str
|
||||
action: str # pass | block | reject | match | scrub | other
|
||||
quick: bool
|
||||
direction: str # in | out | any
|
||||
on_negated: bool
|
||||
on_device: str | None
|
||||
af: int | None # 4, 6 or None (both)
|
||||
label: str | None
|
||||
icmptypes: list[str] = field(default_factory=list)
|
||||
src: str = "any"
|
||||
dst: str = "any"
|
||||
flags: str | None = None
|
||||
|
||||
|
||||
def parse_line(number: int, text: str) -> PfLine:
|
||||
words = text.split()
|
||||
first = words[0] if words else ""
|
||||
action = first if first in ("pass", "match", "scrub") else "other"
|
||||
if first == "block":
|
||||
action = "reject" if len(words) > 1 and words[1].startswith("return") else "block"
|
||||
direction = "in" if " in " in f" {text} " else ("out" if " out " in f" {text} " else "any")
|
||||
on = _ON.search(text)
|
||||
af = 4 if re.search(r"\binet\b", text) else (6 if re.search(r"\binet6\b", text) else None)
|
||||
label = _LABEL.search(text)
|
||||
endpoints = _FROM.search(text)
|
||||
flags = _FLAGS.search(text)
|
||||
return PfLine(
|
||||
number=number, text=text, action=action, quick=" quick " in f" {text} ",
|
||||
direction=direction, on_negated=bool(on and on.group(1)), on_device=on.group(2) if on else None,
|
||||
af=af, label=label.group(1) if label else None, icmptypes=_ICMP.findall(text),
|
||||
src=endpoints.group(1) if endpoints else ("any" if " all" in f" {text}" else "?"),
|
||||
dst=endpoints.group(2) if endpoints else ("any" if " all" in f" {text}" else "?"),
|
||||
flags=flags.group(1) if flags else None,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PfRuleset:
|
||||
lines: list[PfLine]
|
||||
by_label: dict[str, list[PfLine]]
|
||||
|
||||
@property
|
||||
def unlabeled(self) -> list[PfLine]:
|
||||
return [line for line in self.lines if line.label is None and line.action in ("pass", "block", "reject")]
|
||||
|
||||
|
||||
def parse(stats: dict | None) -> PfRuleset | None:
|
||||
"""None when the answer does not look like a pf ruleset at all."""
|
||||
rules = (stats or {}).get("rules") if isinstance(stats, dict) else None
|
||||
filter_rules = rules.get("filter rules") if isinstance(rules, dict) else None
|
||||
if not isinstance(filter_rules, dict):
|
||||
return None
|
||||
lines: list[PfLine] = []
|
||||
for key in filter_rules:
|
||||
head = _HEAD.match(str(key).strip())
|
||||
if not head:
|
||||
continue
|
||||
line = parse_line(int(head.group(1)), head.group(2))
|
||||
if line.action == "scrub":
|
||||
continue
|
||||
lines.append(line)
|
||||
lines.sort(key=lambda line: line.number)
|
||||
by_label: dict[str, list[PfLine]] = {}
|
||||
for line in lines:
|
||||
if line.label:
|
||||
by_label.setdefault(line.label, []).append(line)
|
||||
return PfRuleset(lines=lines, by_label=by_label)
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Does a rule field match the queried packet? Addresses, interfaces, aliases, ports.
|
||||
|
||||
Every function answers with an ``Outcome`` (True/False/None + reason codes). Whatever cannot
|
||||
be proven from the firewall's own data is ``None`` (unknown), never a guess.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import re
|
||||
|
||||
from app.services.rule_analysis.context import Context
|
||||
from app.services.rule_analysis.model import IPAddr, Outcome, o_or, t_not
|
||||
|
||||
MAX_ALIAS_DEPTH = 10
|
||||
TABLE_ONLY_TYPES = {"url", "urltable", "geoip", "asn", "mac", "dynipv6host", "authgroup", "external", "internal"}
|
||||
ADDRESS_TYPES = {"host", "network", "networkgroup"}
|
||||
|
||||
SERVICES = {
|
||||
"ftp-data": 20, "ftp": 21, "ssh": 22, "telnet": 23, "smtp": 25, "domain": 53, "dns": 53, "bootps": 67,
|
||||
"bootpc": 68, "tftp": 69, "http": 80, "www": 80, "kerberos": 88, "pop3": 110, "ntp": 123, "imap": 143,
|
||||
"snmp": 161, "snmptrap": 162, "ldap": 389, "https": 443, "smb": 445, "microsoft-ds": 445, "isakmp": 500,
|
||||
"syslog": 514, "submission": 587, "ldaps": 636, "imaps": 993, "pop3s": 995, "openvpn": 1194,
|
||||
"ms-sql-s": 1433, "radius": 1812, "radius-acct": 1813, "nfs": 2049, "mysql": 3306, "rdp": 3389,
|
||||
"ipsec-nat-t": 4500, "sip": 5060, "postgresql": 5432, "dhcpv6-client": 546, "dhcpv6-server": 547,
|
||||
"http-alt": 8080, "wireguard": 51820,
|
||||
}
|
||||
EPHEMERAL = (1024, 65535)
|
||||
|
||||
|
||||
def _ip_in(token: str, ip: IPAddr) -> bool | None:
|
||||
"""Literal IP / CIDR / range; None when the token is not a literal."""
|
||||
token = token.strip()
|
||||
if "-" in token and not token.startswith("-"):
|
||||
low, _, high = token.partition("-")
|
||||
try:
|
||||
low_ip, high_ip = ipaddress.ip_address(low.strip()), ipaddress.ip_address(high.strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return low_ip.version == ip.version and low_ip <= ip <= high_ip
|
||||
try:
|
||||
net = ipaddress.ip_network(token, strict=False)
|
||||
except ValueError:
|
||||
return None
|
||||
return net.version == ip.version and ip in net
|
||||
|
||||
|
||||
def _interface_ref(token: str, ip: IPAddr, ctx: Context) -> Outcome | None:
|
||||
name = token.strip()
|
||||
lower = name.lower()
|
||||
m = re.fullmatch(r"__(.+)_network", lower)
|
||||
if m and m.group(1) in ctx.interfaces:
|
||||
lower = m.group(1)
|
||||
info = ctx.interfaces.get(lower)
|
||||
if info is not None: # "lan" = the interface's networks (pf: (dev:network), incl. VIP subnets)
|
||||
return Outcome(any(ip in net for net in info.nets))
|
||||
if lower.endswith("ip") and lower[:-2] in ctx.interfaces: # "lanip" = the interface address
|
||||
info = ctx.interfaces[lower[:-2]]
|
||||
if ip in info.addrs:
|
||||
return Outcome(True)
|
||||
if ip in info.vips:
|
||||
return Outcome(None, ["VIP_AS_INTERFACE_ADDRESS"])
|
||||
return Outcome(False)
|
||||
return None
|
||||
|
||||
|
||||
def addr_item(token: str, ip: IPAddr, ctx: Context, *, depth: int = 0, seen: tuple = ()) -> Outcome:
|
||||
token = token.strip()
|
||||
if token.lower() in ("any", ""):
|
||||
return Outcome(True)
|
||||
if token.lower() in ("(self)", "self", "this firewall"):
|
||||
return Outcome(ip in ctx.self_addrs)
|
||||
literal = _ip_in(token, ip)
|
||||
if literal is not None:
|
||||
return Outcome(literal)
|
||||
ref = _interface_ref(token, ip, ctx)
|
||||
if ref is not None:
|
||||
return ref
|
||||
return alias_contains(token, ip, ctx, depth=depth, seen=seen)
|
||||
|
||||
|
||||
def alias_contains(name: str, ip: IPAddr, ctx: Context, *, depth: int = 0, seen: tuple = ()) -> Outcome:
|
||||
if name in seen:
|
||||
return Outcome(None, ["ALIAS_CYCLE"])
|
||||
if depth > MAX_ALIAS_DEPTH:
|
||||
return Outcome(None, ["ALIAS_TOO_DEEP"])
|
||||
alias = ctx.aliases.get(name)
|
||||
refs = ctx.refs.get(str(ip))
|
||||
if alias is None:
|
||||
# pf knows tables the alias list may not show (e.g. internal ones): trust a positive hit.
|
||||
if refs is not None and name in refs:
|
||||
return Outcome(True)
|
||||
return Outcome(None, ["ALIAS_MISSING"])
|
||||
kind = str(alias.get("type") or "").lower()
|
||||
if depth == 0 and refs is not None:
|
||||
# pf's own table test: authoritative for every alias type (nesting, "!" entries, FQDNs).
|
||||
return Outcome(name in refs)
|
||||
if kind in ADDRESS_TYPES:
|
||||
return _alias_content(alias, ip, ctx, depth, (*seen, name))
|
||||
if refs is not None:
|
||||
return Outcome(name in refs)
|
||||
return Outcome(None, ["ALIAS_TABLE_UNAVAILABLE"])
|
||||
|
||||
|
||||
def _alias_content(alias: dict, ip: IPAddr, ctx: Context, depth: int, seen: tuple) -> Outcome:
|
||||
entries = [e.strip() for e in re.split(r"[\n,]", str(alias.get("content") or "")) if e.strip()]
|
||||
if any(e.startswith("!") for e in entries):
|
||||
return Outcome(None, ["ALIAS_EXCLUSION"])
|
||||
results = []
|
||||
for entry in entries:
|
||||
literal = _ip_in(entry, ip)
|
||||
if literal is not None:
|
||||
results.append(Outcome(literal))
|
||||
elif entry in ctx.aliases or entry in seen:
|
||||
results.append(alias_contains(entry, ip, ctx, depth=depth + 1, seen=seen))
|
||||
elif "." in entry or ":" in entry:
|
||||
results.append(Outcome(None, ["FQDN_UNRESOLVED"]))
|
||||
else:
|
||||
results.append(Outcome(None, ["ALIAS_MISSING"]))
|
||||
return o_or(results)
|
||||
|
||||
|
||||
def endpoint_match(endpoint, ip: IPAddr, ctx: Context) -> Outcome:
|
||||
if endpoint.is_any:
|
||||
return Outcome(True)
|
||||
result = o_or([addr_item(item, ip, ctx) for item in endpoint.items])
|
||||
if endpoint.negated:
|
||||
return Outcome(t_not(result.value), result.reasons)
|
||||
return result
|
||||
|
||||
|
||||
# ── Ports ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def port_ranges(spec: str, ctx: Context, *, depth: int = 0, seen: tuple = ()) -> tuple[list[tuple[int, int]] | None, list[str]]:
|
||||
"""The port ranges a spec stands for; (None, reasons) when that cannot be known."""
|
||||
ranges: list[tuple[int, int]] = []
|
||||
reasons: list[str] = []
|
||||
for part in [p.strip() for p in re.split(r"[\n,]", spec) if p.strip()]:
|
||||
m = re.fullmatch(r"(\w[\w-]*?)\s*[-:]\s*(\w[\w-]*)", part) if not part.isdigit() and part not in ctx.aliases else None
|
||||
if part.isdigit():
|
||||
ranges.append((int(part), int(part)))
|
||||
elif m and _port_number(m.group(1)) is not None and _port_number(m.group(2)) is not None:
|
||||
ranges.append((_port_number(m.group(1)), _port_number(m.group(2))))
|
||||
elif part.lower() in SERVICES:
|
||||
ranges.append((SERVICES[part.lower()],) * 2)
|
||||
elif part in ctx.aliases:
|
||||
if part in seen or depth >= MAX_ALIAS_DEPTH:
|
||||
return None, ["ALIAS_CYCLE" if part in seen else "ALIAS_TOO_DEEP"]
|
||||
alias = ctx.aliases[part]
|
||||
if str(alias.get("type") or "").lower() != "port":
|
||||
return None, ["PORT_ALIAS_UNRESOLVED"]
|
||||
sub, why = port_ranges(str(alias.get("content") or ""), ctx, depth=depth + 1, seen=(*seen, part))
|
||||
if sub is None:
|
||||
return None, why
|
||||
ranges += sub
|
||||
else:
|
||||
return None, ["PORT_ALIAS_UNRESOLVED" if part.isidentifier() else "PORT_UNPARSEABLE"]
|
||||
return ranges, reasons
|
||||
|
||||
|
||||
def _port_number(token: str) -> int | None:
|
||||
if token.isdigit():
|
||||
return int(token)
|
||||
return SERVICES.get(token.lower())
|
||||
|
||||
|
||||
def port_match(spec: str, port: int | None, ctx: Context, *, ephemeral_default: bool = False) -> Outcome:
|
||||
spec = (spec or "").strip()
|
||||
if spec.lower() in ("", "any"):
|
||||
return Outcome(True)
|
||||
ranges, reasons = port_ranges(spec, ctx)
|
||||
if ranges is None:
|
||||
return Outcome(None, reasons)
|
||||
if port is not None:
|
||||
return Outcome(any(low <= port <= high for low, high in ranges))
|
||||
if not ephemeral_default:
|
||||
return Outcome(None, ["PORT_NOT_GIVEN"])
|
||||
# No source port given: the query stands for an ephemeral client port (1024-65535).
|
||||
low_e, high_e = EPHEMERAL
|
||||
overlap = [(max(low, low_e), min(high, high_e)) for low, high in ranges if high >= low_e and low <= high_e]
|
||||
if not overlap:
|
||||
return Outcome(False)
|
||||
covered, cursor = True, low_e
|
||||
for low, high in sorted(overlap):
|
||||
if low > cursor:
|
||||
covered = False
|
||||
break
|
||||
cursor = max(cursor, high + 1)
|
||||
if covered and cursor > high_e:
|
||||
return Outcome(True)
|
||||
return Outcome(None, ["SRC_PORT_NOT_GIVEN"])
|
||||
|
||||
|
||||
# ── Interfaces ────────────────────────────────────────────────────────────
|
||||
|
||||
def iface_match(interfaces: set[str], if_not: bool, iface: str, ctx: Context) -> Outcome:
|
||||
if not interfaces:
|
||||
return Outcome(True)
|
||||
results = []
|
||||
for name in interfaces:
|
||||
if name == iface:
|
||||
results.append(Outcome(True))
|
||||
elif name in ctx.interfaces:
|
||||
results.append(Outcome(False))
|
||||
elif ctx.groups is not None and name in ctx.groups:
|
||||
results.append(Outcome(iface in ctx.groups[name]))
|
||||
else:
|
||||
results.append(Outcome(None, ["IFACE_UNRESOLVED" if ctx.groups is not None else "GROUPS_UNAVAILABLE"]))
|
||||
result = o_or(results)
|
||||
if if_not:
|
||||
return Outcome(t_not(result.value), result.reasons)
|
||||
return result
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Entry point used by the API: validate a query, load the firewall's context, evaluate."""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services.rule_analysis import context as ctx_mod
|
||||
from app.services.rule_analysis.engine import Query, evaluate
|
||||
from app.services.rule_analysis.normalize import PROTO_NUMBERS, norm_proto
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _ip(value: str | None, field: str):
|
||||
try:
|
||||
return ipaddress.ip_address(str(value or "").strip())
|
||||
except ValueError:
|
||||
raise api_error(422, ErrorCode.ANALYSE_ADRESSE_UNGUELTIG, field=field, value=str(value or "")[:64]) from None
|
||||
|
||||
|
||||
def _port(value: str | int | None, field: str) -> int | None:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
text = str(value).strip()
|
||||
if not text.isdigit() or int(text) > 65535:
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field=field, value=text[:32])
|
||||
return int(text)
|
||||
|
||||
|
||||
def build_query(*, src: str, dst: str, proto: str = "tcp", sport=None, dport=None, icmptype: str | None = None,
|
||||
iface: str | None = None, direction: str = "in", egress: str | None = None) -> Query:
|
||||
source, destination = _ip(src, "src"), _ip(dst, "dst")
|
||||
if source.version != destination.version:
|
||||
raise api_error(422, ErrorCode.ANALYSE_FAMILIEN_MISCHUNG)
|
||||
label = str(proto or "tcp").strip().lower()
|
||||
if label == "icmp" and source.version == 6:
|
||||
label = "ipv6-icmp"
|
||||
number = norm_proto(label)
|
||||
if not (number.isdigit() and int(number) <= 255) and label not in PROTO_NUMBERS:
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="proto", value=label[:32])
|
||||
q = Query(src=source, dst=destination, proto=number, proto_label=label,
|
||||
sport=_port(sport, "sport"), dport=_port(dport, "dport"))
|
||||
if (q.sport is not None or q.dport is not None) and number not in ("6", "17"):
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="dport", value=str(dport or sport))
|
||||
if icmptype not in (None, ""):
|
||||
if number not in ("1", "58"):
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="icmptype", value=str(icmptype)[:32])
|
||||
q.icmptype = str(icmptype).strip().lower()
|
||||
if direction not in ("in", "out"):
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="direction", value=str(direction)[:8])
|
||||
q.direction = direction
|
||||
q.iface = None if str(iface or "").strip().lower() in ("", "auto") else str(iface).strip()
|
||||
egress_value = str(egress or "").strip().lower()
|
||||
q.egress = "none" if egress_value in ("", "none") else (None if egress_value == "auto" else str(egress).strip())
|
||||
return q
|
||||
|
||||
|
||||
def check_interfaces(q: Query, ctx: ctx_mod.Context) -> None:
|
||||
for name in (q.iface, None if q.egress in (None, "none") else q.egress):
|
||||
if name is not None and name not in ctx.interfaces:
|
||||
raise api_error(422, ErrorCode.ANALYSE_INTERFACE_UNBEKANNT, interface=name[:64],
|
||||
known=", ".join(sorted(i for i in ctx.interfaces if i != "lo0")))
|
||||
|
||||
|
||||
async def analyze_host(host: OpnsenseHost, q: Query, *, refresh: bool = False) -> dict:
|
||||
"""Raises api_error for a bad interface; OpnsenseAPIError/httpx errors when the firewall
|
||||
cannot be read (callers map that)."""
|
||||
ctx = await ctx_mod.load_context(host, use_cache=not refresh)
|
||||
check_interfaces(q, ctx)
|
||||
await ctx_mod.ensure_refs(host, ctx, [str(q.src), str(q.dst)])
|
||||
result = evaluate(ctx, q)
|
||||
if any(ctx.refs.get(ip) is None for ip in (str(q.src), str(q.dst))):
|
||||
result["notes"] = sorted({*result["notes"], "ALIAS_REFERENCES_UNAVAILABLE"})
|
||||
return result
|
||||
@@ -15,18 +15,31 @@ the UI) but is NOT wired into the NotificationSettings webhook/email
|
||||
system — that system is built around persistent NotificationProblem rows
|
||||
with their own admin-configurable toggles, and adding a new problem kind
|
||||
for this is a bigger change than this feature's own scope. Flagged here,
|
||||
not silently dropped, in case it's wanted later.
|
||||
not silently dropped, in case it's wanted later. (Later: the failure is reported by
|
||||
notification_checks.check_scheduled_maintenance_failed since 1.17.)
|
||||
|
||||
auto_verify (1.25, set for every rollout step): instead of leaving the operation "submitted"
|
||||
for a human, the run waits for the job (phase job_wait) and then checks the result itself
|
||||
(phase verify: firmware idle for a while, a reboot really happened when one was due, version
|
||||
readable). Only then the operation is closed as "acknowledged"; anything unclear ends as
|
||||
"unknown" (stays locked, notification). Without auto_verify nothing changes.
|
||||
|
||||
Concurrency: due rows and running rows are each taken with SELECT ... FOR UPDATE SKIP LOCKED,
|
||||
one row per transaction, so a second backend (briefly possible during a web update) can never
|
||||
fire or step the same row at the same time.
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import select, true
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from contextlib import contextmanager
|
||||
|
||||
from app.services import audit_service, backup_store, opnsense_client as client
|
||||
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
@@ -51,40 +64,93 @@ HANDOVER_TIMEOUT = timedelta(minutes=3)
|
||||
LEAVE_TIMEOUT = timedelta(minutes=5)
|
||||
JOB_GRACE = timedelta(minutes=5) # no sign of a job after this long: treat a package update as already done
|
||||
JOB_TIMEOUT = {"reboot": timedelta(minutes=20), "update": timedelta(minutes=60), "major_update": timedelta(minutes=150)}
|
||||
# auto_verify: the firmware must report "ready" for this long in a row (a reboot at the end of an update
|
||||
# can follow a short "ready" moment), and the whole check gives up after VERIFY_TIMEOUT.
|
||||
VERIFY_SETTLE = timedelta(seconds=60)
|
||||
VERIFY_TIMEOUT = timedelta(minutes=20)
|
||||
# A rollout step is created with run_at=now and fires on the next poll. Found much later (OPNexus was down,
|
||||
# e.g. during an update), it no longer belongs to the decision that created it: the window may have closed
|
||||
# and the run may have been paused meanwhile. It is then not fired; the rollout engine re-queues the host.
|
||||
ROLLOUT_FIRE_GRACE = timedelta(minutes=5)
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _as_actor(sched: ScheduledMaintenance):
|
||||
"""Audit events of an unattended run name the person who planned it (not the rollback worker)."""
|
||||
token = audit_service.context.set({"actor": f"{sched.actor} (geplant)"[:150], "id": str(sched.id)})
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _rollout_details(sched: ScheduledMaintenance) -> dict:
|
||||
rollout = (sched.progress or {}).get("rollout")
|
||||
return {"rollout": rollout} if rollout else {}
|
||||
|
||||
|
||||
async def fire_due(db: AsyncSession) -> dict:
|
||||
due = list(
|
||||
(
|
||||
await db.scalars(
|
||||
select(ScheduledMaintenance).where(
|
||||
ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now()
|
||||
)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
fired = 0
|
||||
blocked = 0
|
||||
for sched in due:
|
||||
try:
|
||||
await _fire_one(db, sched)
|
||||
except Exception:
|
||||
logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
|
||||
due = fired = blocked = 0
|
||||
seen: set = set()
|
||||
while True:
|
||||
# One row per transaction, locked: a second backend skips it instead of firing it again.
|
||||
sched = await db.scalar(
|
||||
select(ScheduledMaintenance)
|
||||
.where(ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now(),
|
||||
ScheduledMaintenance.id.not_in(seen) if seen else true())
|
||||
.order_by(ScheduledMaintenance.run_at)
|
||||
.limit(1)
|
||||
.with_for_update(skip_locked=True)
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if sched is None:
|
||||
await db.commit()
|
||||
break
|
||||
seen.add(sched.id)
|
||||
due += 1
|
||||
with _as_actor(sched):
|
||||
try:
|
||||
await _fire_one(db, sched)
|
||||
except Exception:
|
||||
logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
|
||||
await db.rollback()
|
||||
sched = await db.get(ScheduledMaintenance, sched.id, populate_existing=True)
|
||||
if sched.status in ("pending", "running") and sched.phase is None:
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
|
||||
await db.commit()
|
||||
await db.commit() # never keep the row lock past this row
|
||||
if sched.status == "failed":
|
||||
blocked += 1
|
||||
else:
|
||||
fired += 1
|
||||
return {"due": len(due), "fired": fired, "blocked_or_failed": blocked}
|
||||
return {"due": due, "fired": fired, "blocked_or_failed": blocked}
|
||||
|
||||
|
||||
def _reboot_expected(action: str, state: dict) -> bool:
|
||||
return action in ("reboot", "major_update") or bool(state.get("reboot_required"))
|
||||
|
||||
|
||||
async def _boot_time(host: OpnsenseHost) -> str | None:
|
||||
"""systemTime's "boottime" (Diagnostics SystemController::systemTimeAction, read-only, already used by the
|
||||
resource sampler). A different value afterwards proves a reboot even if no poll caught the host offline."""
|
||||
try:
|
||||
return (await client.get_system_time(host)).get("boottime") or None
|
||||
except client.OpnsenseAPIError:
|
||||
return None
|
||||
|
||||
|
||||
async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
if sched.origin == "rollout" and _now() - sched.run_at > ROLLOUT_FIRE_GRACE:
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": "Rollout-Schritt nicht rechtzeitig gestartet, nichts ausgeführt", "not_fired": True}
|
||||
await db.commit()
|
||||
logger.warning("Geplante Wartung %s: Rollout-Schritt überfällig, nicht ausgeführt", sched.id)
|
||||
return
|
||||
host = await db.get(OpnsenseHost, sched.host_id)
|
||||
if not host or not host.enabled:
|
||||
sched.status = "failed"
|
||||
@@ -137,6 +203,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
|
||||
return
|
||||
|
||||
boot_before = await _boot_time(host) if sched.auto_verify else None
|
||||
op = HostOperation(
|
||||
host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
|
||||
expires_at=_now(), fingerprint="", snapshot=state, status="starting", started_at=_now(),
|
||||
@@ -147,14 +214,23 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
await audit_service.persist(
|
||||
kind="host", action=sched.action, status="requested",
|
||||
host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state},
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state, **_rollout_details(sched)},
|
||||
)
|
||||
try:
|
||||
result = await _ACTION_FN[sched.action](host)
|
||||
op.status = "submitted"
|
||||
op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
|
||||
sched.status = "succeeded"
|
||||
sched.result = {"response": result}
|
||||
if sched.auto_verify:
|
||||
# Not done yet: the poll loop waits for the job and checks the result (job_wait -> verify).
|
||||
op.message = "Geplant übergeben. Abschluss wird automatisch geprüft."
|
||||
sched.phase, sched.phase_since = "job_wait", _now()
|
||||
_progress(sched, handover=False, job_started_at=_now().isoformat(), seen_down=False, seen_running=False,
|
||||
version_before=state.get("version"), boot_before=boot_before,
|
||||
reboot_expected=_reboot_expected(sched.action, state))
|
||||
sched.result = {"response": result}
|
||||
else:
|
||||
op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
|
||||
sched.status = "succeeded"
|
||||
sched.result = {"response": result}
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status = "unknown"
|
||||
op.message = "Ausgang unklar: " + str(exc)[:800] + ". Nicht automatisch wiederholen."
|
||||
@@ -166,7 +242,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
await audit_service.persist(
|
||||
kind="host", action=sched.action, status=op.status,
|
||||
host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
|
||||
message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result},
|
||||
message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result, **_rollout_details(sched)},
|
||||
)
|
||||
|
||||
|
||||
@@ -182,21 +258,24 @@ async def _audit(host, sched, op, action, status, message=None, **details):
|
||||
await audit_service.persist(
|
||||
kind="host", action=action, status=status, host_id=str(host.id), host_name=host.name,
|
||||
entity_id=str(op.id) if op else None, message=message,
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "handover": True, **details},
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "handover": bool((sched.progress or {}).get("handover")),
|
||||
**_rollout_details(sched), **details},
|
||||
)
|
||||
|
||||
|
||||
async def _end(db, sched, host, op, *, ok, message, op_status=None, **result):
|
||||
handover = bool((sched.progress or {}).get("handover"))
|
||||
sched.status = "succeeded" if ok else "failed"
|
||||
sched.phase = None
|
||||
sched.result = {"handover": True, **result} if ok else {"error": message[:900], **result}
|
||||
sched.result = ({"handover": handover, "auto_verified": bool(sched.auto_verify), **result} if ok
|
||||
else {"error": message[:900], **result})
|
||||
if op is not None:
|
||||
op.status = op_status or ("acknowledged" if ok else "unknown")
|
||||
op.message = message[:1000]
|
||||
await db.commit()
|
||||
await _audit(host, sched, op, sched.action, "completed" if ok else (op_status or "unknown"), message)
|
||||
if not ok:
|
||||
logger.warning("Geplante Wartung %s (CARP-Übergabe) beendet: %s", sched.id, message)
|
||||
logger.warning("Geplante Wartung %s beendet: %s", sched.id, message)
|
||||
|
||||
|
||||
async def _toggle(host, expected: str) -> None:
|
||||
@@ -230,6 +309,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
await fail(" ".join(blocked))
|
||||
return
|
||||
|
||||
boot_before = await _boot_time(host) if sched.auto_verify else None
|
||||
now = _now()
|
||||
op = HostOperation(
|
||||
host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
|
||||
@@ -237,6 +317,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
)
|
||||
db.add(op)
|
||||
sched.status = "running"
|
||||
_progress(sched, handover=True, boot_before=boot_before, reboot_expected=_reboot_expected(sched.action, state))
|
||||
await db.commit()
|
||||
sched.host_operation_id = op.id
|
||||
await _audit(host, sched, op, "carp_enter", "requested", snapshot=state)
|
||||
@@ -255,32 +336,56 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
|
||||
|
||||
async def advance_running(db: AsyncSession) -> int:
|
||||
"""One step for every handover run in progress. Returns how many are still running."""
|
||||
rows = list((await db.scalars(
|
||||
select(ScheduledMaintenance).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
|
||||
"""One step for every run in progress (handover and/or auto_verify). Returns how many are still running."""
|
||||
ids = list((await db.scalars(
|
||||
select(ScheduledMaintenance.id).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
|
||||
)).all())
|
||||
for sched in rows:
|
||||
host = await db.get(OpnsenseHost, sched.host_id)
|
||||
op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
|
||||
try:
|
||||
if host is None:
|
||||
raise RuntimeError("Host nicht mehr vorhanden")
|
||||
if op is not None and op.status not in ACTIVE:
|
||||
# A human closed or cancelled the operation: stop here and touch nothing more.
|
||||
sched.status, sched.phase = "failed", None
|
||||
sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt. CARP-Rolle von {host.name} prüfen."}
|
||||
await db.commit()
|
||||
continue
|
||||
await _STEPS[sched.phase](db, sched, host, op)
|
||||
except Exception as exc: # a stuck run must not loop forever or take the poll loop down
|
||||
logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
|
||||
if host is not None:
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Unerwarteter Fehler während der CARP-Übergabe ({type(exc).__name__}). "
|
||||
f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen.")
|
||||
return sum(1 for s in rows if s.status == "running")
|
||||
await db.commit()
|
||||
still_running = 0
|
||||
for sched_id in ids:
|
||||
# Locked for the whole step: a second backend skips the row instead of toggling or submitting twice.
|
||||
sched = await db.scalar(
|
||||
select(ScheduledMaintenance)
|
||||
.where(ScheduledMaintenance.id == sched_id, ScheduledMaintenance.status == "running",
|
||||
ScheduledMaintenance.phase.is_not(None))
|
||||
.with_for_update(skip_locked=True)
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if sched is None:
|
||||
await db.commit()
|
||||
still_running += 1 if await db.scalar(select(ScheduledMaintenance.status).where(ScheduledMaintenance.id == sched_id)) == "running" else 0
|
||||
await db.commit()
|
||||
continue
|
||||
with _as_actor(sched):
|
||||
await _advance_one(db, sched)
|
||||
await db.commit()
|
||||
if sched.status == "running":
|
||||
still_running += 1
|
||||
return still_running
|
||||
|
||||
|
||||
async def _advance_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
host = await db.get(OpnsenseHost, sched.host_id)
|
||||
op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
|
||||
try:
|
||||
if host is None:
|
||||
raise RuntimeError("Host nicht mehr vorhanden")
|
||||
if op is not None and op.status not in ACTIVE:
|
||||
# A human closed or cancelled the operation: stop here and touch nothing more.
|
||||
hint = f" CARP-Rolle von {host.name} prüfen." if (sched.progress or {}).get("handover") else ""
|
||||
sched.status, sched.phase = "failed", None
|
||||
sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt.{hint}"}
|
||||
await db.commit()
|
||||
return
|
||||
await _STEPS[sched.phase](db, sched, host, op)
|
||||
except Exception as exc: # a stuck run must not loop forever or take the poll loop down
|
||||
logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
|
||||
if host is not None:
|
||||
hint = (f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen."
|
||||
if (sched.progress or {}).get("handover") else f"Zustand von {host.name} prüfen.")
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Unerwarteter Fehler während der geplanten Wartung ({type(exc).__name__}). {hint}")
|
||||
|
||||
def _since(sched: ScheduledMaintenance) -> timedelta:
|
||||
return _now() - (sched.phase_since or _now())
|
||||
|
||||
@@ -352,17 +457,75 @@ async def _step_job_wait(db, sched, host, op):
|
||||
else:
|
||||
done = bool(prog.get("seen_down") or prog.get("seen_running")) or elapsed >= JOB_GRACE
|
||||
if done:
|
||||
sched.phase, sched.phase_since = "leave", _now()
|
||||
sched.phase = "verify" if sched.auto_verify else "leave"
|
||||
sched.phase_since = _now()
|
||||
await db.commit()
|
||||
return
|
||||
if elapsed >= JOB_TIMEOUT[sched.action]:
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
|
||||
f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
|
||||
if prog.get("handover"):
|
||||
text = (f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
|
||||
f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
|
||||
else:
|
||||
text = f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. Bitte {host.name} prüfen."
|
||||
await _end(db, sched, host, op, ok=False, message=text)
|
||||
return
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _step_verify(db, sched, host, op):
|
||||
"""auto_verify: is the job really over and the host healthy? Read-only, never repeats the action."""
|
||||
prog = dict(sched.progress or {})
|
||||
handover = bool(prog.get("handover"))
|
||||
|
||||
async def wait_or_give_up(why: str):
|
||||
if _since(sched) >= VERIFY_TIMEOUT:
|
||||
tail = (f" {host.name} bleibt im CARP-Wartungsmodus, der Partner trägt den Verkehr." if handover else "")
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Automatische Prüfung nach „{sched.action}“ nicht bestanden: {why}.{tail} "
|
||||
f"Bitte {host.name} prüfen und die Wartung von Hand abschließen.")
|
||||
return
|
||||
_progress(sched, waiting_for=why)
|
||||
await db.commit()
|
||||
|
||||
try:
|
||||
running = (await client.get_firmware_running(host)).get("status")
|
||||
except client.OpnsenseAPIError:
|
||||
running = None
|
||||
_progress(sched, seen_down=True)
|
||||
if running != "ready":
|
||||
_progress(sched, ready_since=None)
|
||||
await wait_or_give_up("Firmware-Auftrag läuft noch oder Host nicht erreichbar")
|
||||
return
|
||||
if not prog.get("ready_since"):
|
||||
_progress(sched, ready_since=_now().isoformat())
|
||||
await db.commit()
|
||||
return
|
||||
if _now() - datetime.fromisoformat(prog["ready_since"]) < VERIFY_SETTLE:
|
||||
await db.commit()
|
||||
return
|
||||
boot_now = await _boot_time(host)
|
||||
rebooted = bool(prog.get("seen_down")) or bool(prog.get("boot_before") and boot_now and boot_now != prog["boot_before"])
|
||||
if prog.get("reboot_expected") and not rebooted:
|
||||
await wait_or_give_up("der erwartete Neustart wurde nicht beobachtet")
|
||||
return
|
||||
try:
|
||||
version = (await client.get_firmware_info(host)).get("product_version")
|
||||
except client.OpnsenseAPIError:
|
||||
version = None
|
||||
online = (await get_statuses([host])).get(host.id, {}).get("status") == "online"
|
||||
if not version or not online:
|
||||
await wait_or_give_up("Host nicht vollständig erreichbar (Version/Status nicht lesbar)")
|
||||
return
|
||||
_progress(sched, version_after=version, boot_after=boot_now, rebooted=rebooted, waiting_for=None)
|
||||
if handover:
|
||||
sched.phase, sched.phase_since = "leave", _now() # only now take the traffic back
|
||||
await db.commit()
|
||||
return
|
||||
await _end(db, sched, host, op, ok=True,
|
||||
message=f"Automatisch geprüft: {host.name} erreichbar, Firmware-Auftrag beendet, Version {version}.",
|
||||
version_before=prog.get("version_before"), version_after=version, rebooted=rebooted)
|
||||
|
||||
|
||||
async def _step_leave(db, sched, host, op):
|
||||
me, _partner = await _roles(db, host)
|
||||
if me.get("status") != "online":
|
||||
@@ -397,6 +560,8 @@ async def _step_leave_wait(db, sched, host, op):
|
||||
except client.OpnsenseAPIError:
|
||||
pass
|
||||
extra["version_before"] = (sched.progress or {}).get("version_before")
|
||||
if sched.auto_verify:
|
||||
extra["rebooted"] = bool((sched.progress or {}).get("rebooted")) # what verify saw before leaving
|
||||
text = "Automatisch abgeschlossen: Übergabe, Auftrag und Zurückschalten bestätigt."
|
||||
if extra.get("reboot_required"):
|
||||
text += " Ein Neustart ist noch erforderlich."
|
||||
@@ -410,6 +575,7 @@ async def _step_leave_wait(db, sched, host, op):
|
||||
_STEPS = {
|
||||
"handover_wait": _step_handover_wait,
|
||||
"job_wait": _step_job_wait,
|
||||
"verify": _step_verify,
|
||||
"leave": _step_leave,
|
||||
"leave_wait": _step_leave_wait,
|
||||
}
|
||||
@@ -18,7 +18,7 @@ from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.x509.oid import ExtensionOID, NameOID
|
||||
|
||||
from app.services import overlay_service, proxy_connect
|
||||
from app.services import proxy_connect
|
||||
|
||||
MAX_PEM_BYTES = 32 * 1024
|
||||
|
||||
@@ -94,6 +94,8 @@ def verify_argument(verify_tls: bool, tls_ca_pem: str | None):
|
||||
|
||||
async def _open_tls(base_url: str, parsed, ctx: ssl.SSLContext, timeout: float):
|
||||
"""TLS connection to the firewall: directly, or through the overlay hub's proxy for overlay addresses."""
|
||||
# Imported here so that pinned_context() stays usable without the database layer (gui-gateway container).
|
||||
from app.services import overlay_service
|
||||
proxy = overlay_service.proxy_for(base_url)
|
||||
port = parsed.port or 443
|
||||
if proxy is None:
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Destination check for URLs OPNexus is told to call (notification channels, webhook).
|
||||
|
||||
Admins legitimately point these at servers on their own network (a self-hosted ntfy, Gotify or
|
||||
Matrix), so private ranges are allowed. What is refused is what cannot be a notification target:
|
||||
anything but http(s), credentials inside the URL, and addresses that only ever mean "this machine"
|
||||
or "the cloud metadata service" (loopback, link-local such as 169.254.169.254, unspecified,
|
||||
multicast). Redirects are never followed by the callers, so a public host cannot bounce a request
|
||||
to such an address. The hostname is resolved at send time and every address is checked (a name that
|
||||
does not resolve is let through; the request then fails on its own).
|
||||
"""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import contextvars
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
# httpx logs every request line ("HTTP Request: POST https://api.telegram.org/bot<token>/sendMessage"). For
|
||||
# targets whose URL is the credential that line must not reach the logs: callers wrap the request in muted().
|
||||
_muted = contextvars.ContextVar("url_guard_muted", default=False)
|
||||
|
||||
|
||||
class _MuteFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
return not _muted.get()
|
||||
|
||||
|
||||
for _name in ("httpx", "httpcore"):
|
||||
logging.getLogger(_name).addFilter(_MuteFilter())
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def muted():
|
||||
token = _muted.set(True)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_muted.reset(token)
|
||||
|
||||
|
||||
def _forbidden(ip: ipaddress._BaseAddress) -> bool:
|
||||
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
|
||||
ip = ip.ipv4_mapped
|
||||
return ip.is_loopback or ip.is_link_local or ip.is_unspecified or ip.is_multicast
|
||||
|
||||
|
||||
def validate_url(url: str, *, max_length: int = 500) -> str:
|
||||
"""Returns the cleaned URL or raises ValueError with a short reason (shown to the admin)."""
|
||||
value = (url or "").strip()
|
||||
if not value or len(value) > max_length or any(c.isspace() for c in value):
|
||||
raise ValueError("Die Adresse ist leer, zu lang oder enthält Leerzeichen")
|
||||
try:
|
||||
parts = urlsplit(value)
|
||||
host = parts.hostname
|
||||
parts.port # noqa: B018 - raises ValueError for an invalid port
|
||||
except ValueError:
|
||||
raise ValueError("Die Adresse ist ungültig") from None
|
||||
if parts.scheme not in ("http", "https") or not host:
|
||||
raise ValueError("Die Adresse muss mit http:// oder https:// beginnen")
|
||||
if parts.username or parts.password:
|
||||
raise ValueError("Zugangsdaten gehören nicht in die Adresse")
|
||||
try:
|
||||
literal = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
literal = None
|
||||
if literal is not None and _forbidden(literal):
|
||||
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
|
||||
if literal is None and host.lower() in ("localhost", "localhost.localdomain"):
|
||||
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
|
||||
return value
|
||||
|
||||
|
||||
async def ensure_resolved_safe(url: str) -> None:
|
||||
"""Send-time check against DNS answers pointing at forbidden addresses. Raises ValueError."""
|
||||
host = urlsplit(url).hostname
|
||||
if not host:
|
||||
raise ValueError("Die Adresse ist ungültig")
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
return # a literal was already checked by validate_url
|
||||
except ValueError:
|
||||
pass
|
||||
try:
|
||||
infos = await asyncio.get_running_loop().getaddrinfo(host, None, type=socket.SOCK_STREAM)
|
||||
except OSError:
|
||||
return
|
||||
for info in infos:
|
||||
try:
|
||||
ip = ipaddress.ip_address(info[4][0].split("%")[0])
|
||||
except ValueError:
|
||||
continue
|
||||
if _forbidden(ip):
|
||||
raise ValueError("Der Name zeigt auf eine nicht erlaubte Adresse (lokal, Link-Local oder Metadaten-Dienst)")
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Occurrences of recurring maintenance windows.
|
||||
|
||||
Rules (deliberately simple and written down, because a firmware rollout depends on them):
|
||||
|
||||
* A window is a wall-clock start time in its own IANA time zone plus a duration. "Sunday 02:00
|
||||
Europe/Berlin" stays 02:00 local time in summer and in winter.
|
||||
* Daylight saving: a local start time that does not exist on that day (spring forward, e.g. 02:30 on
|
||||
the last Sunday of March in Europe/Berlin) moves FORWARD by the gap (03:30). A local time that
|
||||
exists twice (fall back) means the FIRST of the two (fold=0, still summer time). Both follow from
|
||||
Python's zoneinfo with fold=0 and are covered by tests.
|
||||
* The duration is real elapsed time (a 3 h window over the fall-back night still lasts 3 h).
|
||||
* Exceptions are keyed by the occurrence's local date: "skip" drops it, "move" replaces its start
|
||||
with an absolute instant (same duration and cut-off).
|
||||
* New host actions may only START within the first ``start_cutoff_minutes`` of an occurrence
|
||||
(``startable``); actions already running finish even after the window has closed.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError, available_timezones
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
|
||||
# How far ahead occurrences are searched. A monthly "5th ..." does not exist; the longest real gap
|
||||
# between two occurrences is a month, so two years is plenty and bounds the work per call.
|
||||
_SEARCH_DAYS = 2 * 366
|
||||
MAX_DURATION_MINUTES = 24 * 60
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Occurrence:
|
||||
local_date: date
|
||||
start: datetime # UTC
|
||||
end: datetime # UTC
|
||||
cutoff: datetime # UTC: no new host action after this instant
|
||||
moved: bool = False
|
||||
skipped: bool = False
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {"local_date": self.local_date.isoformat(), "start": self.start.isoformat(), "end": self.end.isoformat(),
|
||||
"cutoff": self.cutoff.isoformat(), "moved": self.moved, "skipped": self.skipped}
|
||||
|
||||
|
||||
def valid_timezone(name: str) -> bool:
|
||||
if not name or name not in available_timezones():
|
||||
return False
|
||||
try:
|
||||
ZoneInfo(name)
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _matches(window: MaintenanceWindow, day: date) -> bool:
|
||||
if window.recurrence == "weekly":
|
||||
return day.weekday() in (window.weekdays or [])
|
||||
if window.recurrence == "monthly":
|
||||
if window.month_mode == "day":
|
||||
return day.day == window.month_day
|
||||
if window.month_mode == "nth_weekday":
|
||||
if day.weekday() != window.month_weekday:
|
||||
return False
|
||||
if window.month_nth == -1:
|
||||
return (day + timedelta(days=7)).month != day.month
|
||||
return (day.day - 1) // 7 + 1 == window.month_nth
|
||||
return False
|
||||
|
||||
|
||||
def local_start(window: MaintenanceWindow, day: date) -> datetime:
|
||||
"""The UTC instant of the window's start on local date ``day`` (DST rules see module docstring)."""
|
||||
tz = ZoneInfo(window.timezone)
|
||||
wall = datetime.combine(day, window.start_time, tzinfo=tz).replace(fold=0)
|
||||
return wall.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def _occurrence(window: MaintenanceWindow, day: date, start: datetime, *, moved: bool = False, skipped: bool = False) -> Occurrence:
|
||||
duration = timedelta(minutes=window.duration_minutes)
|
||||
cutoff = timedelta(minutes=min(window.start_cutoff_minutes, window.duration_minutes))
|
||||
return Occurrence(local_date=day, start=start, end=start + duration, cutoff=start + cutoff, moved=moved, skipped=skipped)
|
||||
|
||||
|
||||
def occurrences(window: MaintenanceWindow, after: datetime, n: int = 10, *, with_skipped: bool = False) -> list[Occurrence]:
|
||||
"""The next ``n`` occurrences whose END lies after ``after`` (so one that is open right now is first).
|
||||
``with_skipped`` lists skipped ones too (flagged, for the UI); the engine never sees them."""
|
||||
if n <= 0:
|
||||
return []
|
||||
after = after.astimezone(timezone.utc)
|
||||
tz = ZoneInfo(window.timezone)
|
||||
exceptions = {e.occurrence_date: e for e in (window.exceptions or [])}
|
||||
duration = timedelta(minutes=window.duration_minutes)
|
||||
# Start one day early: an occurrence that began yesterday (local) may still be open.
|
||||
day = (after - duration).astimezone(tz).date() - timedelta(days=1)
|
||||
found: list[Occurrence] = []
|
||||
# Moved occurrences can land anywhere, so they are collected separately and merged at the end.
|
||||
for exc in exceptions.values():
|
||||
if exc.kind == "move" and exc.moved_start is not None and _matches(window, exc.occurrence_date):
|
||||
occ = _occurrence(window, exc.occurrence_date, exc.moved_start.astimezone(timezone.utc), moved=True)
|
||||
if occ.end > after:
|
||||
found.append(occ)
|
||||
regular = 0
|
||||
for _ in range(_SEARCH_DAYS):
|
||||
if regular >= n:
|
||||
break
|
||||
if _matches(window, day):
|
||||
exc = exceptions.get(day)
|
||||
if exc is None or (with_skipped and exc.kind == "skip"):
|
||||
occ = _occurrence(window, day, local_start(window, day), skipped=exc is not None)
|
||||
if occ.end > after:
|
||||
found.append(occ)
|
||||
regular += 1
|
||||
day += timedelta(days=1)
|
||||
found.sort(key=lambda o: o.start)
|
||||
return found[:n]
|
||||
|
||||
|
||||
def current(window: MaintenanceWindow, now: datetime) -> Occurrence | None:
|
||||
"""The occurrence open at ``now`` (start <= now < end), if any."""
|
||||
if not window.enabled:
|
||||
return None
|
||||
for occ in occurrences(window, now, 3):
|
||||
if occ.start <= now < occ.end:
|
||||
return occ
|
||||
return None
|
||||
|
||||
|
||||
def startable(window: MaintenanceWindow | None, now: datetime) -> Occurrence | None:
|
||||
"""The occurrence in which a NEW host action may start right now (start <= now < cutoff)."""
|
||||
if window is None or not window.enabled:
|
||||
return None
|
||||
occ = current(window, now)
|
||||
if occ is not None and occ.start <= now < occ.cutoff:
|
||||
return occ
|
||||
return None
|
||||
|
||||
|
||||
def is_occurrence_date(window: MaintenanceWindow, day: date) -> bool:
|
||||
return _matches(window, day)
|
||||
|
||||
|
||||
def definition(window: MaintenanceWindow) -> dict:
|
||||
"""What a rollout plan's fingerprint covers: changing any of this re-requires activation."""
|
||||
return {
|
||||
"id": str(window.id), "timezone": window.timezone, "recurrence": window.recurrence,
|
||||
"weekdays": sorted(window.weekdays or []), "month_mode": window.month_mode, "month_nth": window.month_nth,
|
||||
"month_weekday": window.month_weekday, "month_day": window.month_day,
|
||||
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
|
||||
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
|
||||
}
|
||||
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
|
||||
Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
"""
|
||||
|
||||
APP_VERSION = "1.23.8"
|
||||
APP_CODENAME = "Harrier"
|
||||
APP_VERSION = "1.28.0"
|
||||
APP_CODENAME = "Sparrowhawk"
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Alarme für Gateways, Ressourcen und Dienste; zusätzliche Benachrichtigungskanäle.
|
||||
|
||||
Revision ID: 0020
|
||||
Revises: 0019
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0020"
|
||||
down_revision = "0019"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
_COLUMNS = [
|
||||
("gateway_down_enabled", "boolean NOT NULL DEFAULT true"),
|
||||
("resource_threshold_enabled", "boolean NOT NULL DEFAULT false"),
|
||||
("service_down_enabled", "boolean NOT NULL DEFAULT false"),
|
||||
("gateway_loss_threshold_pct", "integer NOT NULL DEFAULT 20"),
|
||||
("gateway_delay_threshold_ms", "integer NOT NULL DEFAULT 500"),
|
||||
("disk_threshold_pct", "integer NOT NULL DEFAULT 90"),
|
||||
("memory_threshold_pct", "integer NOT NULL DEFAULT 95"),
|
||||
("swap_threshold_pct", "integer NOT NULL DEFAULT 50"),
|
||||
("load_per_cpu_threshold", "double precision NOT NULL DEFAULT 2.0"),
|
||||
("alert_confirm_samples", "integer NOT NULL DEFAULT 3"),
|
||||
("resource_confirm_samples", "integer NOT NULL DEFAULT 5"),
|
||||
("service_watchlist", "json"),
|
||||
]
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for name, ddl in _COLUMNS:
|
||||
op.execute(f"ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS {name} {ddl}")
|
||||
if sa.inspect(op.get_bind()).has_table("notification_channels"):
|
||||
return
|
||||
op.create_table(
|
||||
"notification_channels",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("name", sa.String(length=100), nullable=False),
|
||||
sa.Column("channel_type", sa.String(length=16), nullable=False),
|
||||
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
|
||||
sa.Column("config", sa.JSON(), nullable=False),
|
||||
sa.Column("secret_set", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DROP TABLE IF EXISTS notification_channels")
|
||||
for name, _ddl in reversed(_COLUMNS):
|
||||
op.execute(f"ALTER TABLE notification_settings DROP COLUMN IF EXISTS {name}")
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Wiederkehrende Wartungsfenster mit Ausnahmen (1.25).
|
||||
|
||||
Revision ID: 0021
|
||||
Revises: 0020
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0021"
|
||||
down_revision = "0020"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
|
||||
if sa.inspect(op.get_bind()).has_table("maintenance_windows"):
|
||||
return
|
||||
op.create_table(
|
||||
"maintenance_windows",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("name", sa.String(length=120), nullable=False),
|
||||
sa.Column("timezone", sa.String(length=64), nullable=False),
|
||||
sa.Column("recurrence", sa.String(length=16), nullable=False),
|
||||
sa.Column("weekdays", sa.JSON(), nullable=False),
|
||||
sa.Column("month_mode", sa.String(length=16), nullable=True),
|
||||
sa.Column("month_nth", sa.Integer(), nullable=True),
|
||||
sa.Column("month_weekday", sa.Integer(), nullable=True),
|
||||
sa.Column("month_day", sa.Integer(), nullable=True),
|
||||
sa.Column("start_time", sa.Time(), nullable=False),
|
||||
sa.Column("duration_minutes", sa.Integer(), nullable=False),
|
||||
sa.Column("start_cutoff_minutes", sa.Integer(), server_default="60", nullable=False),
|
||||
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
|
||||
sa.Column("created_by", sa.String(length=150), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
)
|
||||
op.create_table(
|
||||
"maintenance_window_exceptions",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("window_id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("occurrence_date", sa.Date(), nullable=False),
|
||||
sa.Column("kind", sa.String(length=8), nullable=False),
|
||||
sa.Column("moved_start", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("reason", sa.String(length=300), nullable=False),
|
||||
sa.Column("actor", sa.String(length=150), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),
|
||||
)
|
||||
op.create_index("ix_maintenance_window_exceptions_window_id", "maintenance_window_exceptions", ["window_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DROP TABLE IF EXISTS maintenance_window_exceptions")
|
||||
op.execute("DROP TABLE IF EXISTS maintenance_windows")
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Geplante Wartung: automatische Prüfung nach dem Auftrag (auto_verify) und Herkunft (manuell/Rollout).
|
||||
|
||||
Revision ID: 0022
|
||||
Revises: 0021
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0022"
|
||||
down_revision = "0021"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS auto_verify boolean NOT NULL DEFAULT false")
|
||||
op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS origin varchar(16) NOT NULL DEFAULT 'manual'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS origin")
|
||||
op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS auto_verify")
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Firmware-Rollouts in Wellen: Pläne, Läufe, Hosts je Lauf; Benachrichtigung bei Rollout-Problemen (1.25).
|
||||
|
||||
Revision ID: 0023
|
||||
Revises: 0022
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0023"
|
||||
down_revision = "0022"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
UUID = postgresql.UUID(as_uuid=True)
|
||||
TS = sa.DateTime(timezone=True)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute("ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS rollout_problems_enabled boolean NOT NULL DEFAULT true")
|
||||
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
|
||||
if sa.inspect(op.get_bind()).has_table("rollout_plans"):
|
||||
return
|
||||
op.create_table(
|
||||
"rollout_plans",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("name", sa.String(120), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=False),
|
||||
sa.Column("action", sa.String(16), nullable=False),
|
||||
sa.Column("window_id", UUID, nullable=True),
|
||||
sa.Column("auto_start", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("gate_checks", sa.JSON(), nullable=False),
|
||||
sa.Column("compliance_mode", sa.String(8), server_default="warn", nullable=False),
|
||||
sa.Column("activated_by", sa.String(150), nullable=True),
|
||||
sa.Column("activated_at", TS, nullable=True),
|
||||
sa.Column("activated_fingerprint", sa.String(64), nullable=True),
|
||||
sa.Column("auto_start_error", sa.JSON(), nullable=True),
|
||||
sa.Column("created_by", sa.String(150), nullable=False),
|
||||
sa.Column("created_at", TS, nullable=False),
|
||||
sa.Column("updated_at", TS, nullable=False),
|
||||
sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
)
|
||||
op.create_index("ix_rollout_plans_window_id", "rollout_plans", ["window_id"])
|
||||
op.create_table(
|
||||
"rollout_plan_waves",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("plan_id", UUID, nullable=False),
|
||||
sa.Column("position", sa.Integer(), nullable=False),
|
||||
sa.Column("label", sa.String(60), nullable=False),
|
||||
sa.Column("soak_minutes", sa.Integer(), nullable=False),
|
||||
sa.Column("carp_handover", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("require_approval", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_rollout_plan_waves_plan_id", "rollout_plan_waves", ["plan_id"])
|
||||
op.create_table(
|
||||
"rollout_wave_groups",
|
||||
sa.Column("wave_id", UUID, nullable=False),
|
||||
sa.Column("group_id", UUID, nullable=False),
|
||||
sa.ForeignKeyConstraint(["wave_id"], ["rollout_plan_waves.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["group_id"], ["device_groups.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("wave_id", "group_id"),
|
||||
)
|
||||
op.create_table(
|
||||
"rollout_runs",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("plan_id", UUID, nullable=True),
|
||||
sa.Column("plan_name", sa.String(120), nullable=False),
|
||||
sa.Column("plan_snapshot", sa.JSON(), nullable=False),
|
||||
sa.Column("action", sa.String(16), nullable=False),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("paused_from", sa.String(20), nullable=True),
|
||||
sa.Column("current_wave", sa.Integer(), nullable=False),
|
||||
sa.Column("target_version", sa.String(40), nullable=True),
|
||||
sa.Column("pause_reason", sa.String(40), nullable=True),
|
||||
sa.Column("pause_detail", sa.JSON(), nullable=True),
|
||||
sa.Column("started_by", sa.String(150), nullable=False),
|
||||
sa.Column("trigger", sa.String(8), nullable=False),
|
||||
sa.Column("window_occurrence", TS, nullable=True),
|
||||
sa.Column("revision", sa.Integer(), server_default="0", nullable=False),
|
||||
sa.Column("progress", sa.JSON(), nullable=False),
|
||||
sa.Column("step_at", TS, nullable=True),
|
||||
sa.Column("created_at", TS, nullable=False),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),
|
||||
)
|
||||
op.create_index("ix_rollout_runs_plan_id", "rollout_runs", ["plan_id"])
|
||||
op.create_index("ix_rollout_runs_status", "rollout_runs", ["status"])
|
||||
op.create_table(
|
||||
"rollout_run_waves",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("run_id", UUID, nullable=False),
|
||||
sa.Column("index", sa.Integer(), nullable=False),
|
||||
sa.Column("label", sa.String(60), nullable=False),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("baseline", sa.JSON(), nullable=True),
|
||||
sa.Column("gate_result", sa.JSON(), nullable=True),
|
||||
sa.Column("soak_until", TS, nullable=True),
|
||||
sa.Column("gate_attempts", sa.Integer(), server_default="0", nullable=False),
|
||||
sa.Column("approved_by", sa.String(150), nullable=True),
|
||||
sa.Column("started_at", TS, nullable=True),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_rollout_run_waves_run_id", "rollout_run_waves", ["run_id"])
|
||||
op.create_table(
|
||||
"rollout_run_hosts",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("run_id", UUID, nullable=False),
|
||||
sa.Column("wave_index", sa.Integer(), nullable=False),
|
||||
sa.Column("host_id", UUID, nullable=True),
|
||||
sa.Column("host_name", sa.String(120), nullable=False),
|
||||
sa.Column("cluster_name", sa.String(120), nullable=True),
|
||||
sa.Column("order", sa.Integer(), nullable=False),
|
||||
sa.Column("status", sa.String(16), nullable=False),
|
||||
sa.Column("scheduled_maintenance_id", UUID, nullable=True),
|
||||
sa.Column("version_before", sa.String(40), nullable=True),
|
||||
sa.Column("version_after", sa.String(40), nullable=True),
|
||||
sa.Column("detail", sa.JSON(), nullable=True),
|
||||
sa.Column("started_at", TS, nullable=True),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["scheduled_maintenance_id"], ["scheduled_maintenance.id"], ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),
|
||||
)
|
||||
op.create_index("ix_rollout_run_hosts_run_id", "rollout_run_hosts", ["run_id"])
|
||||
op.create_index("ix_rollout_run_hosts_host_id", "rollout_run_hosts", ["host_id"])
|
||||
op.create_index("ix_rollout_run_hosts_scheduled_maintenance_id", "rollout_run_hosts", ["scheduled_maintenance_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for table in ("rollout_run_hosts", "rollout_run_waves", "rollout_runs", "rollout_wave_groups", "rollout_plan_waves", "rollout_plans"):
|
||||
op.execute(f"DROP TABLE IF EXISTS {table}")
|
||||
op.execute("ALTER TABLE notification_settings DROP COLUMN IF EXISTS rollout_problems_enabled")
|
||||
@@ -0,0 +1,78 @@
|
||||
"""GUI-Zugriff über das gui-gateway (1.27): Einstellungen, Einmal-Tickets, Gateway-Sitzungen.
|
||||
|
||||
Revision ID: 0024
|
||||
Revises: 0023
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0024"
|
||||
down_revision = "0023"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
UUID = postgresql.UUID(as_uuid=True)
|
||||
TS = sa.DateTime(timezone=True)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
|
||||
if sa.inspect(op.get_bind()).has_table("gui_access_settings"):
|
||||
return
|
||||
op.create_table(
|
||||
"gui_access_settings",
|
||||
sa.Column("id", sa.Integer(), nullable=False),
|
||||
sa.Column("enabled", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("idle_timeout_minutes", sa.Integer(), server_default="15", nullable=False),
|
||||
sa.Column("absolute_timeout_minutes", sa.Integer(), server_default="60", nullable=False),
|
||||
sa.Column("slot_count", sa.Integer(), server_default="4", nullable=False),
|
||||
sa.Column("updated_by", sa.String(150), nullable=True),
|
||||
sa.Column("updated_at", TS, nullable=True),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_table(
|
||||
"gui_tickets",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("ticket_hash", sa.String(64), nullable=False),
|
||||
sa.Column("session_hash", sa.String(64), nullable=False),
|
||||
sa.Column("user_id", UUID, nullable=False),
|
||||
sa.Column("host_id", UUID, nullable=False),
|
||||
sa.Column("slot", sa.Integer(), nullable=False),
|
||||
sa.Column("created_at", TS, nullable=False),
|
||||
sa.Column("expires_at", TS, nullable=False),
|
||||
sa.Column("redeemed_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["user_id"], ["admin_users.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("ticket_hash"),
|
||||
)
|
||||
op.create_index("ix_gui_tickets_session_hash", "gui_tickets", ["session_hash"])
|
||||
op.create_index("ix_gui_tickets_user_id", "gui_tickets", ["user_id"])
|
||||
op.create_index("ix_gui_tickets_host_id", "gui_tickets", ["host_id"])
|
||||
op.create_table(
|
||||
"gui_sessions",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("session_hash", sa.String(64), nullable=False),
|
||||
sa.Column("user_id", UUID, nullable=False),
|
||||
sa.Column("host_id", UUID, nullable=False),
|
||||
sa.Column("slot", sa.Integer(), nullable=False),
|
||||
sa.Column("client_ip", sa.String(45), nullable=True),
|
||||
sa.Column("started_at", TS, nullable=False),
|
||||
sa.Column("last_checked_at", TS, nullable=False),
|
||||
sa.Column("ended_at", TS, nullable=True),
|
||||
sa.Column("end_reason", sa.String(32), nullable=True),
|
||||
sa.ForeignKeyConstraint(["user_id"], ["admin_users.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_gui_sessions_session_hash", "gui_sessions", ["session_hash"])
|
||||
op.create_index("ix_gui_sessions_user_id", "gui_sessions", ["user_id"])
|
||||
op.create_index("ix_gui_sessions_host_id", "gui_sessions", ["host_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("gui_sessions")
|
||||
op.drop_table("gui_tickets")
|
||||
op.drop_table("gui_access_settings")
|
||||
@@ -13,3 +13,4 @@ respx==0.23.1
|
||||
cryptography==50.0.2
|
||||
alembic==1.20.0
|
||||
PyJWT==2.10.1
|
||||
tzdata==2025.2
|
||||
+1003
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,39 @@
|
||||
"""Mutation check helper (run manually, not collected by pytest).
|
||||
|
||||
python tests/mutation_check.py tests/mutations_gui.json
|
||||
|
||||
Each mutation: {"file": ..., "old": ..., "new": ..., "tests": "pytest selection"}.
|
||||
The file is patched, the selected tests must FAIL, then the file is restored.
|
||||
"""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main(spec_path: str) -> int:
|
||||
mutations = json.loads(Path(spec_path).read_text())
|
||||
survivors = []
|
||||
for i, m in enumerate(mutations, 1):
|
||||
path = Path(m["file"])
|
||||
original = path.read_text()
|
||||
if original.count(m["old"]) != 1:
|
||||
print(f"[{i}] SKIP (anchor not unique/found): {m['file']}: {m['old'][:60]!r}")
|
||||
survivors.append(m)
|
||||
continue
|
||||
try:
|
||||
path.write_text(original.replace(m["old"], m["new"]))
|
||||
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", *m["tests"].split()],
|
||||
capture_output=True, text=True)
|
||||
finally:
|
||||
path.write_text(original)
|
||||
killed = result.returncode != 0
|
||||
print(f"[{i}] {'KILLED ' if killed else 'SURVIVED'} {m.get('name', m['old'][:50])}")
|
||||
if not killed:
|
||||
survivors.append(m)
|
||||
print(f"{len(mutations) - len(survivors)}/{len(mutations)} mutations killed")
|
||||
return 1 if survivors else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1]))
|
||||
@@ -0,0 +1,226 @@
|
||||
[
|
||||
{
|
||||
"name": "GW browser Cookie forwarded",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS}",
|
||||
"new": " out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS or k == \"cookie\"}",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW Referer/Authorization forwarded",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": "\"accept\", \"accept-language\", \"content-type\", \"user-agent\", \"x-requested-with\", \"x-csrftoken\", \"range\",",
|
||||
"new": "\"accept\", \"accept-language\", \"content-type\", \"user-agent\", \"x-requested-with\", \"x-csrftoken\", \"range\", \"referer\", \"authorization\",",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW Set-Cookie passed to browser",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": "\"content-range\", \"accept-ranges\", \"content-security-policy\",",
|
||||
"new": "\"content-range\", \"accept-ranges\", \"content-security-policy\", \"set-cookie\",",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW jar never forgets deleted cookies",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if expired:\n gs.jar.pop(name, None)",
|
||||
"new": " if False:\n gs.jar.pop(name, None)",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW foreign Location kept",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if _origin_of(location) != _origin_of(upstream_origin):\n return None",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW backslash Location kept",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if not location or \"\\\\\" in location",
|
||||
"new": " if not location",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW Sec-Fetch-Site lax",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if site not in (\"same-origin\", \"none\"):",
|
||||
"new": " if site in (\"cross-site\",):",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW redeem Sec-Fetch-Site unchecked",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if method != \"GET\" or site not in (\"same-site\", \"same-origin\", \"none\"):",
|
||||
"new": " if method != \"GET\":",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW redeem without OPNexus cookie",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if not ticket or len(ticket) > 128 or not browser_session:",
|
||||
"new": " if not ticket or len(ticket) > 128:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW Origin unchecked for writes",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if _origin_of(origin) not in own_origins:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW absolute-form accepted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if not path.startswith(\"/\") or path.startswith(\"//\") or not raw_path.startswith(b\"/\") \\",
|
||||
"new": " if False \\",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW CONNECT accepted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if method in (\"CONNECT\", \"TRACE\"):",
|
||||
"new": " if method in (\"TRACE\",):",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW Upgrade accepted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if \"upgrade\" in headers or \"upgrade\" in headers.get(\"connection\", \"\").lower():",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW service worker accepted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if headers.get(\"service-worker\", \"\").lower() == \"script\":",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW slot header optional",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if not own_origins:\n return await self._respond(send, 400, _FORBIDDEN)",
|
||||
"new": " if False:\n return await self._respond(send, 400, _FORBIDDEN)",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW foreign slot cookie accepted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if gs is None or gs.slot != slot:",
|
||||
"new": " if gs is None:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW login rate limit off",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if login and self.limiter.blocked(gs.upstream_origin):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW login failures not counted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if login and not (300 <= response.status_code < 400):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW no periodic re-check",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if self.clock() >= gs.next_check:\n state = await self._recheck(key, gs)",
|
||||
"new": " if False:\n state = await self._recheck(key, gs)",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW fail-open when backend down",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if state is None:\n return await self._respond(send, 503, _UNAVAILABLE)",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW idle timeout off",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if self.clock() - gs.last_activity >= gs.idle_timeout:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW absolute timeout off",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if self.wall_clock() >= gs.absolute_deadline:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW no CSP",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " (b\"content-security-policy\", _GATEWAY_CSP.encode()),",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW no X-Frame-Options",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " (b\"x-frame-options\", b\"DENY\"),",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW no no-store",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " (b\"cache-control\", b\"no-store\"),",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW streaming body limit off",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if size > BODY_LIMIT:\n return None",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW query logged",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": "scope.get(\"path\", \"?\")[:200], status,",
|
||||
"new": "scope.get(\"path\", \"?\")[:200] + \"?\" + scope.get(\"query_string\", b\"\").decode(), status,",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW httpx URL logging on",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " logging.getLogger(_noisy).setLevel(logging.WARNING)",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW weak gateway cookie",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": "={value}; Path=/; Secure; HttpOnly; SameSite=Strict\"",
|
||||
"new": "={value}; Path=/\"",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW no Clear-Site-Data",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " (b\"set-cookie\", cookie.encode()), (b\"clear-site-data\", b'\"cache\", \"storage\"'),",
|
||||
"new": " (b\"set-cookie\", cookie.encode()),",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW pin not used",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " verify = pinned_context(gs.ca_pem)",
|
||||
"new": " verify = False",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "GW access log on",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": "http=\"h11\", ws=\"none\", access_log=False,",
|
||||
"new": "http=\"h11\", ws=\"none\", access_log=True,",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,170 @@
|
||||
[
|
||||
{
|
||||
"name": "AP1 sec-fetch: cross-site not blocked",
|
||||
"file": "app/security_middleware.py",
|
||||
"old": " if site == \"cross-site\":\n return True",
|
||||
"new": " if site == \"cross-site\":\n return False",
|
||||
"tests": "tests/test_session_hardening.py"
|
||||
},
|
||||
{
|
||||
"name": "AP1 sec-fetch: same-site always allowed",
|
||||
"file": "app/security_middleware.py",
|
||||
"old": " return not _explicit_frontend_origin(request.headers.get(\"origin\", \"\"))",
|
||||
"new": " return False",
|
||||
"tests": "tests/test_session_hardening.py"
|
||||
},
|
||||
{
|
||||
"name": "AP1 sec-fetch: check not wired",
|
||||
"file": "app/security_middleware.py",
|
||||
"old": " if sec_fetch_blocked(request):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_session_hardening.py"
|
||||
},
|
||||
{
|
||||
"name": "AP1 cookie: plain name falls back even with __Host-",
|
||||
"file": "app/routers/auth.py",
|
||||
"old": " return request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME, \"\")",
|
||||
"new": " return request.cookies.get(COOKIE_NAME) or request.cookies.get(SECURE_COOKIE_NAME, \"\")",
|
||||
"tests": "tests/test_session_hardening.py"
|
||||
},
|
||||
{
|
||||
"name": "AP1 cookie: no __Host- prefix",
|
||||
"file": "app/routers/auth.py",
|
||||
"old": " return SECURE_COOKIE_NAME if settings.cookie_secure else COOKIE_NAME",
|
||||
"new": " return COOKIE_NAME",
|
||||
"tests": "tests/test_session_hardening.py"
|
||||
},
|
||||
{
|
||||
"name": "AP1 startup: conflict check removed",
|
||||
"file": "app/main.py",
|
||||
"old": " if conflicts:\n",
|
||||
"new": " if False:\n",
|
||||
"tests": "tests/test_session_hardening.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 ticket expiry ignored",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " elif ticket.expires_at <= now:",
|
||||
"new": " elif False:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 replay allowed",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if replayed:\n reason = \"replayed\"",
|
||||
"new": " if False:\n reason = \"replayed\"",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 session binding ignored",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " elif not secrets.compare_digest(ticket.session_hash, session_hash):",
|
||||
"new": " elif False:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 slot binding ignored",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " elif ticket.slot != slot:",
|
||||
"new": " elif False:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 revalidate: role not rechecked",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if user is None or user.role != \"admin\":",
|
||||
"new": " if user is None:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 revalidate: session not rechecked",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if session is None or session.expires_at <= _now() or session.user_id != user_id:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 revalidate: switch not rechecked",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if not row.enabled:\n return \"disabled\", user, None",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_access.py::test_revalidate_fails_when_switched_off"
|
||||
},
|
||||
{
|
||||
"name": "AP2 absolute limit ignored",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 host tls requirement dropped",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if not host.verify_tls:\n return \"tls\"",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 slot limit ignored",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " if not free:\n raise",
|
||||
"new": " if False:\n raise",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 gateway token not compared",
|
||||
"file": "app/routers/gui_access.py",
|
||||
"old": " if not hmac.compare_digest(",
|
||||
"new": " if False and not hmac.compare_digest(",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 switch not checked for tickets",
|
||||
"file": "app/routers/gui_access.py",
|
||||
"old": " if not row.enabled:\n raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_ACCESS_DISABLED)",
|
||||
"new": "",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 legacy cookie accepted for tickets",
|
||||
"file": "app/routers/gui_access.py",
|
||||
"old": " session_cookie = request.cookies.get(SECURE_COOKIE_NAME, \"\")",
|
||||
"new": " session_cookie = request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(\"opnexus-session\", \"\")",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 enabling without https allowed",
|
||||
"file": "app/routers/gui_access.py",
|
||||
"old": " if payload.enabled and svc.unavailable_reasons():",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP2 redeem leaks api key",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": " \"upstream_origin\": firewall_origin(host.base_url),",
|
||||
"new": " \"upstream_origin\": firewall_origin(host.base_url), \"k\": host.api_key,",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "AP7 revalidation interval back to 30 s (switch-off took 31 s live)",
|
||||
"file": "app/services/gui_ticket_service.py",
|
||||
"old": "REVALIDATE_SECONDS = 20",
|
||||
"new": "REVALIDATE_SECONDS = 30",
|
||||
"tests": "tests/test_gui_access.py"
|
||||
},
|
||||
{
|
||||
"name": "Review: multipart login not counted",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": " if content_type.lower().startswith(\"multipart/form-data\"):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
},
|
||||
{
|
||||
"name": "Review: decoded path forwarded upstream",
|
||||
"file": "app/gui_gateway.py",
|
||||
"old": "url = gs.upstream_origin + raw_path +",
|
||||
"new": "url = gs.upstream_origin + scope[\"path\"] +",
|
||||
"tests": "tests/test_gui_gateway.py"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,486 @@
|
||||
"""Linking hand-made objects on a second firewall to the central object that already exists
|
||||
(services/adopt_link_service.py), instead of creating a duplicate."""
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from tests.helpers import deploy_approved
|
||||
from tests.test_api_tokens import _login
|
||||
from tests.test_object_adopt import AUTOMATIC_NAT, NAT_KINDS, SYSTEM_ALIASES, alias_row, o2o_row, pf_row, snat_row
|
||||
from tests.test_object_adopt import _mode
|
||||
from tests.test_rule_adopt import _row as rule_row
|
||||
|
||||
A = "https://fw-link-a.test"
|
||||
B = "https://fw-link-b.test"
|
||||
|
||||
|
||||
async def _host(client, respx_mock, base, name):
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
r = await client.post("/api/hosts", json={"name": name, "base_url": base, "api_key": "k", "api_secret": "s"})
|
||||
assert r.status_code == 201, r.text
|
||||
return r.json()["id"]
|
||||
|
||||
|
||||
def _search(respx_mock, base, path, rows):
|
||||
return respx_mock.get(f"{base}/api/firewall/{path}").mock(return_value=httpx.Response(200, json={"rows": rows, "total": len(rows)}))
|
||||
|
||||
|
||||
def _live_writes(respx_mock):
|
||||
return [c for c in respx_mock.calls if c.request.method != "GET" and "/firewall/" in str(c.request.url)]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def two(client, respx_mock):
|
||||
a = await _host(client, respx_mock, A, "fw-link-a")
|
||||
b = await _host(client, respx_mock, B, "fw-link-b")
|
||||
_mode(respx_mock) # outbound NAT mode is read per base url below
|
||||
for base in (A, B):
|
||||
respx_mock.get(f"{base}/api/firewall/source_nat/get").mock(return_value=httpx.Response(
|
||||
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == "hybrid" else 0} for m in
|
||||
("automatic", "hybrid", "advanced", "disabled")}}}}))
|
||||
return a, b
|
||||
|
||||
|
||||
async def _put_vars(client, host_id, values):
|
||||
r = await client.put(f"/api/hosts/{host_id}/variables", json={"variables": values})
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
|
||||
# ── aliases ─────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_is_linked_not_duplicated_and_rolls_out_to_both_hosts(client, respx_mock, two):
|
||||
a, b = two
|
||||
live_a = alias_row("web_servers", "host", "10.0.0.1\n10.0.0.2", uuid="uuid-a")
|
||||
live_b = alias_row("web_servers", "host", "10.0.0.1\n10.0.0.2", uuid="uuid-b")
|
||||
_search(respx_mock, A, "alias/searchItem", SYSTEM_ALIASES + [live_a])
|
||||
_search(respx_mock, B, "alias/searchItem", SYSTEM_ALIASES + [live_b])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
|
||||
|
||||
# host B: the candidate carries the exact twin (name_exists no longer decides alone)
|
||||
cand = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()
|
||||
row = next(r for r in cand if r["uuid"] == "uuid-b")
|
||||
assert row["linkable"] is True and row["reasons"] == ["name_exists"]
|
||||
assert [(m["id"], m["name"], m["targets"], m["diff"]) for m in row["matches"]] == [(first["id"], "web_servers", ["fw-link-a"], [])]
|
||||
assert row["similar"] == [] and row["bulk_action"] == "link"
|
||||
|
||||
# adopting it as an own object is still refused (the name is taken)
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b"})).status_code == 409
|
||||
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": first["id"]})
|
||||
assert resp.status_code == 201, resp.text
|
||||
linked = resp.json()
|
||||
assert linked["id"] == first["id"] and sorted(linked["target_host_ids"]) == sorted([a, b])
|
||||
assert {d["host_id"]: d["status"] for d in linked["deployments"]} == {a: "success", b: "success"}
|
||||
assert not _live_writes(respx_mock) # nothing is written to a firewall
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "alias"))).all()
|
||||
assert any(e.message == "Mit vorhandenem Objekt verknüpft" and e.details["remote_uuid"] == "uuid-b" for e in events)
|
||||
revisions = (await client.get(f"/api/aliases/{first['id']}/revisions")).json()
|
||||
assert any("fw-link-b verknüpft" in r["note"] for r in revisions)
|
||||
|
||||
# preview: unchanged on both hosts; it is no longer offered on B
|
||||
preview = (await client.get(f"/api/aliases/{first['id']}/preview")).json()
|
||||
assert sorted(h["action"] for h in preview["hosts"]) == ["unchanged", "unchanged"]
|
||||
assert [r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json() if r["reasons"] != ["system"]] == []
|
||||
|
||||
# one edit rolls out to both remote objects, never as a new one
|
||||
edit = {"name": "web_servers", "type": "host", "content": "10.0.0.1\n10.0.0.3", "description": "web_servers desc", "enabled": True,
|
||||
"target_host_ids": [a, b], "target_group_ids": [], "expected_updated_at": linked["updated_at"]}
|
||||
assert (await client.put(f"/api/aliases/{first['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(url__regex=r".*/alias/addItem").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
set_a = respx_mock.post(f"{A}/api/firewall/alias/setItem/uuid-a").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
set_b = respx_mock.post(f"{B}/api/firewall/alias/setItem/uuid-b").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(url__regex=r".*/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert (await deploy_approved(client, f"/api/aliases/{first['id']}/deploy")).status_code == 200
|
||||
assert set_a.called and set_b.called and not add.called
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_link_is_idempotent(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("dup_me", uuid="uuid-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("dup_me", uuid="uuid-b")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
|
||||
body = {"host_id": b, "uuid": "uuid-b", "link_to": first["id"]}
|
||||
assert (await client.post("/api/aliases/adopt", json=body)).status_code == 201
|
||||
again = await client.post("/api/aliases/adopt", json=body)
|
||||
assert again.status_code == 201, again.text
|
||||
assert len(again.json()["deployments"]) == 2 and len(again.json()["target_host_ids"]) == 2
|
||||
# linking the same live object to ANOTHER central object is refused
|
||||
other = (await client.post("/api/aliases", json={"name": "other", "type": "host", "content": "1.1.1.1"})).json()
|
||||
assert (await client.post("/api/aliases/adopt", json={**body, "link_to": other["id"]})).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_different_content_shows_a_diff_and_is_never_linked(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("ntp", "host", "10.0.0.1", uuid="uuid-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("ntp", "host", "10.0.0.1\n10.0.0.9", uuid="uuid-b", description="other text")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
|
||||
row = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]
|
||||
assert row["linkable"] is False and row["matches"] == [] and row["bulk_action"] == "skip"
|
||||
sim = row["similar"][0]
|
||||
assert sim["id"] == first["id"]
|
||||
assert {d["field"]: (d["central"], d["live"]) for d in sim["diff"]} == {
|
||||
"content": ("10.0.0.1", "10.0.0.1\n10.0.0.9"), "description": ("ntp desc", "other text")}
|
||||
# forcing the link by id is refused too: the live state is compared again on the server
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": first["id"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
|
||||
assert "content" in resp.json()["detail"]["params"]["fields"]
|
||||
# nothing was linked
|
||||
alias = next(x for x in (await client.get("/api/aliases")).json() if x["id"] == first["id"])
|
||||
assert alias["target_host_ids"] == [a] and len(alias["deployments"]) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_state_is_not_trusted_live_state_decides(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("srv", "host", "10.0.0.1", uuid="uuid-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("srv", "host", "10.0.0.1", uuid="uuid-b")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "uuid-a"})).json()
|
||||
assert (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]["linkable"] is True
|
||||
# between listing and clicking somebody changes the alias on host B by hand
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("srv", "host", "10.0.0.1\n10.0.0.77", uuid="uuid-b")])
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": first["id"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_variables_are_resolved_per_host(client, respx_mock, two):
|
||||
a, b = two
|
||||
await _put_vars(client, a, {"LAN_NET": "192.168.1.0/24"})
|
||||
await _put_vars(client, b, {"LAN_NET": "192.168.2.0/24"})
|
||||
central = (await client.post("/api/aliases", json={"name": "lan_net", "type": "network", "content": "{{LAN_NET}}", "description": "lan_net desc",
|
||||
"target_host_ids": [a]})).json()
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("lan_net", "network", "192.168.2.0/24", uuid="uuid-b")])
|
||||
row = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]
|
||||
assert row["linkable"] is True and row["matches"][0]["id"] == central["id"]
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "uuid-b", "link_to": central["id"]})
|
||||
assert resp.status_code == 201, resp.text
|
||||
dep_b = next(d for d in resp.json()["deployments"] if d["host_id"] == b)
|
||||
assert dep_b["status"] == "success"
|
||||
async with SessionLocal() as db:
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
deps = {str(d.host_id): d for d in (await db.scalars(select(AliasDeployment))).all()}
|
||||
assert deps[b].opnsense_alias_uuid == "uuid-b" and deps[b].last_payload["content"] == "192.168.2.0/24"
|
||||
preview = (await client.get(f"/api/aliases/{central['id']}/preview")).json()
|
||||
assert {h["host_name"]: h["action"] for h in preview["hosts"]}["fw-link-b"] == "unchanged" # resolved payload == live
|
||||
|
||||
# a host with a DIFFERENT value for the variable does not match; the diff shows the resolved value
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("lan_net", "network", "192.168.9.0/24", uuid="uuid-x")])
|
||||
# (host B already has the object, so ask for it through a third host)
|
||||
c = await _host(client, respx_mock, "https://fw-link-c.test", "fw-link-c")
|
||||
await _put_vars(client, c, {"LAN_NET": "192.168.3.0/24"})
|
||||
_search(respx_mock, "https://fw-link-c.test", "alias/searchItem", [alias_row("lan_net", "network", "192.168.9.0/24", uuid="uuid-c")])
|
||||
row = (await client.get(f"/api/aliases/unmanaged?host_id={c}")).json()[0]
|
||||
assert row["matches"] == [] and row["similar"][0]["diff"] == [{"field": "content", "central": "192.168.3.0/24", "live": "192.168.9.0/24"}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_variable_on_the_host_never_matches(client, respx_mock, two):
|
||||
a, b = two
|
||||
await _put_vars(client, a, {"LAN_NET": "192.168.1.0/24"})
|
||||
await client.post("/api/aliases", json={"name": "lan_net", "type": "network", "content": "{{LAN_NET}}", "description": "lan_net desc", "target_host_ids": [a]})
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("lan_net", "network", "192.168.2.0/24", uuid="uuid-b")])
|
||||
row = (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()[0]
|
||||
assert row["matches"] == [] and row["similar"][0]["diff"][0]["field"] == "variables"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_nested_alias_references_are_part_of_the_comparison(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [
|
||||
alias_row("inner", "network", "10.1.0.0/16", uuid="in-a"),
|
||||
alias_row("outer", "networkgroup", "inner\n10.2.0.0/16", uuid="out-a")])
|
||||
result = (await client.post("/api/aliases/adopt-all", json={"host_id": a})).json()
|
||||
assert len(result["adopted"]) == 2
|
||||
_search(respx_mock, B, "alias/searchItem", [
|
||||
alias_row("inner", "network", "10.1.0.0/16", uuid="in-b"),
|
||||
alias_row("outer", "networkgroup", "inner\n10.2.0.0/16", uuid="out-b")])
|
||||
rows = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()}
|
||||
assert rows["inner"]["linkable"] and rows["outer"]["linkable"]
|
||||
# a different reference is a diff, not a match
|
||||
_search(respx_mock, B, "alias/searchItem", [
|
||||
alias_row("inner", "network", "10.1.0.0/16", uuid="in-b"),
|
||||
alias_row("outer", "networkgroup", "other_alias\n10.2.0.0/16", uuid="out-b"),
|
||||
alias_row("other_alias", "network", "10.5.0.0/16", uuid="oth-b")])
|
||||
rows = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json()}
|
||||
assert rows["outer"]["matches"] == [] and rows["outer"]["similar"][0]["diff"][0]["field"] == "content"
|
||||
|
||||
# "adopt all" with link: both linked, nothing created
|
||||
_search(respx_mock, B, "alias/searchItem", [
|
||||
alias_row("inner", "network", "10.1.0.0/16", uuid="in-b"),
|
||||
alias_row("outer", "networkgroup", "inner\n10.2.0.0/16", uuid="out-b")])
|
||||
done = (await client.post("/api/aliases/adopt-all", json={"host_id": b, "link_existing": True})).json()
|
||||
assert sorted((x["title"], x["linked"]) for x in done["adopted"]) == [("inner", True), ("outer", True)] and done["skipped"] == []
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_all_without_link_flag_keeps_old_behaviour(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("x1", uuid="u-a")])
|
||||
await client.post("/api/aliases/adopt-all", json={"host_id": a})
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("x1", uuid="u-b")])
|
||||
done = (await client.post("/api/aliases/adopt-all", json={"host_id": b})).json()
|
||||
assert done["adopted"] == [] and done["skipped"][0]["reasons"] == ["name_exists"]
|
||||
|
||||
|
||||
# ── NAT + rules, every kind ─────────────────────────────────────────────
|
||||
|
||||
KINDS = {
|
||||
"port-forward": ("d_nat/searchRule", pf_row, {"descr": "Web"}, {"destination.port": "9999"}, "local_port"),
|
||||
"one-to-one": ("one_to_one/searchRule", o2o_row, {"description": "Mail 1:1"}, {"external": "203.0.113.77"}, "external"),
|
||||
"outbound-nat": ("source_nat/searchRule", snat_row, {"description": "LAN out"}, {"source_net": "192.168.9.0/24"}, "source_net"),
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("prefix", list(KINDS))
|
||||
async def test_nat_kinds_link_diff_and_bulk(client, respx_mock, two, prefix):
|
||||
a, b = two
|
||||
path, factory, _, change, _ = KINDS[prefix]
|
||||
_search(respx_mock, A, path, [factory(uuid="a-1"), *AUTOMATIC_NAT])
|
||||
first = (await client.post(f"/api/{prefix}/adopt", json={"host_id": a, "uuid": "a-1"})).json()
|
||||
|
||||
# different on B -> hint + diff, no link possible
|
||||
_search(respx_mock, B, path, [factory(uuid="b-1", **change)])
|
||||
row = (await client.get(f"/api/{prefix}/unmanaged?host_id={b}")).json()[0]
|
||||
assert row["adoptable"] is True and row["linkable"] is False and row["matches"] == []
|
||||
assert row["similar"][0]["id"] == first["id"] and row["similar"][0]["diff"]
|
||||
resp = await client.post(f"/api/{prefix}/adopt", json={"host_id": b, "uuid": "b-1", "link_to": first["id"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
|
||||
adopt_all = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": b, "link_existing": True})).json()
|
||||
assert adopt_all["adopted"][0]["linked"] is False # a different object is adopted as its own, not linked
|
||||
|
||||
# fresh start for the exact case on a third host
|
||||
c = await _host(client, respx_mock, "https://fw-link-c.test", "fw-link-c")
|
||||
respx_mock.get("https://fw-link-c.test/api/firewall/source_nat/get").mock(return_value=httpx.Response(
|
||||
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == "hybrid" else 0} for m in
|
||||
("automatic", "hybrid", "advanced", "disabled")}}}}))
|
||||
_search(respx_mock, "https://fw-link-c.test", path, [factory(uuid="c-1")])
|
||||
row = (await client.get(f"/api/{prefix}/unmanaged?host_id={c}")).json()[0]
|
||||
ids = [m["id"] for m in row["matches"]]
|
||||
assert ids == [first["id"]] and row["bulk_action"] == "link" and row["linkable"]
|
||||
resp = await client.post(f"/api/{prefix}/adopt", json={"host_id": c, "uuid": "c-1", "link_to": first["id"]})
|
||||
assert resp.status_code == 201, resp.text
|
||||
assert resp.json()["id"] == first["id"] and c in resp.json()["target_host_ids"]
|
||||
preview = (await client.get(f"/api/{prefix}/{first['id']}/preview")).json()
|
||||
assert {h["host_name"]: h["action"] for h in preview["hosts"]}["fw-link-c"] == "unchanged"
|
||||
assert not _live_writes(respx_mock)
|
||||
listed = (await client.get(f"/api/{prefix}")).json()
|
||||
assert len(listed) == 2 # first (A + C) and the different one from B
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_port_forward_link_updates_remote_object_without_duplicate(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "d_nat/searchRule", [pf_row(uuid="pf-a")])
|
||||
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="pf-b")])
|
||||
first = (await client.post("/api/port-forward/adopt", json={"host_id": a, "uuid": "pf-a"})).json()
|
||||
linked = (await client.post("/api/port-forward/adopt", json={"host_id": b, "uuid": "pf-b", "link_to": first["id"]})).json()
|
||||
edit = {k: linked[k] for k in ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "target", "local_port", "natreflection", "enabled")}
|
||||
for base in (A, B): # alias lookups and the interface consistency check of a multi-host rollout
|
||||
_search(respx_mock, base, "alias/searchItem", [])
|
||||
for base in (A, B): # the interface consistency check of a multi-host rollout
|
||||
respx_mock.get(f"{base}/api/firewall/filter/getRule").mock(return_value=httpx.Response(
|
||||
200, json={"rule": {"interface": {"wan": {"value": "WAN", "selected": 1}}}}))
|
||||
edit.update(target="192.168.1.99", target_host_ids=[a, b], target_group_ids=[], expected_updated_at=linked["updated_at"])
|
||||
assert (await client.put(f"/api/port-forward/{first['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(url__regex=r".*/d_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
sa = respx_mock.post(f"{A}/api/firewall/d_nat/setRule/pf-a").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
sb = respx_mock.post(f"{B}/api/firewall/d_nat/setRule/pf-b").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(url__regex=r".*/d_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert (await deploy_approved(client, f"/api/port-forward/{first['id']}/deploy")).status_code == 200
|
||||
assert sa.called and sb.called and not add.called
|
||||
assert json.loads(sb.calls.last.request.content)["rule"]["target"] == "192.168.1.99"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_link_diff_and_errors(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "filter/searchRule", [rule_row("r-a")])
|
||||
_search(respx_mock, B, "filter/searchRule", [rule_row("r-b")])
|
||||
first = (await client.post("/api/rules/adopt", json={"host_id": a, "rule_uuid": "r-a"})).json()
|
||||
row = (await client.get(f"/api/rules/unmanaged?host_id={b}")).json()[0]
|
||||
assert row["linkable"] is True and [m["id"] for m in row["matches"]] == [first["id"]]
|
||||
resp = await client.post("/api/rules/adopt", json={"host_id": b, "rule_uuid": "r-b", "link_to": first["id"]})
|
||||
assert resp.status_code == 201, resp.text
|
||||
assert resp.json()["id"] == first["id"] and sorted(resp.json()["target_host_ids"]) == sorted([a, b])
|
||||
again = await client.post("/api/rules/adopt", json={"host_id": b, "rule_uuid": "r-b", "link_to": first["id"]})
|
||||
assert again.status_code == 201 and len(again.json()["deployments"]) == 2
|
||||
preview = (await client.get(f"/api/rules/{first['id']}/preview")).json()
|
||||
assert sorted(h["action"] for h in preview["hosts"]) == ["unchanged", "unchanged"]
|
||||
assert not _live_writes(respx_mock)
|
||||
|
||||
# different port -> diff only
|
||||
c = await _host(client, respx_mock, "https://fw-link-c.test", "fw-link-c")
|
||||
_search(respx_mock, "https://fw-link-c.test", "filter/searchRule", [rule_row("r-c", destination_port="2222")])
|
||||
row = (await client.get(f"/api/rules/unmanaged?host_id={c}")).json()[0]
|
||||
assert row["linkable"] is False and row["similar"][0]["diff"] == [{"field": "destination_port", "central": "22", "live": "2222"}]
|
||||
refused = await client.post("/api/rules/adopt", json={"host_id": c, "rule_uuid": "r-c", "link_to": first["id"]})
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ADOPT_LINK_TARGET_CHANGED"
|
||||
|
||||
|
||||
# ── error cases ─────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_deleted_target_and_host_already_has_object_and_busy(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("gone", uuid="u-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("gone", uuid="u-b")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
|
||||
# busy: an unconfirmed deployment on the central object
|
||||
respx_mock.post(f"{A}/api/firewall/alias/setItem/u-a").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(url__regex=r".*/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
edit = {"name": "gone", "type": "host", "content": "10.9.9.1\n10.9.9.2", "description": "gone desc", "enabled": False,
|
||||
"target_host_ids": [a], "target_group_ids": [], "expected_updated_at": first["updated_at"]}
|
||||
assert (await client.put(f"/api/aliases/{first['id']}", json=edit)).status_code == 200
|
||||
assert (await deploy_approved(client, f"/api/aliases/{first['id']}/deploy")).status_code == 200 # pending now
|
||||
busy = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
|
||||
assert busy.status_code == 409 # content differs now AND it is pending; either way nothing is linked
|
||||
await client.post(f"/api/aliases/{first['id']}/confirm")
|
||||
|
||||
# deleted in the meantime
|
||||
respx_mock.post(url__regex=r".*/alias/delItem/.*").mock(return_value=httpx.Response(200, json={"result": "deleted"}))
|
||||
assert (await client.delete(f"/api/aliases/{first['id']}")).status_code == 204
|
||||
gone = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
|
||||
assert gone.status_code == 404 and gone.json()["detail"]["code"] == "ADOPT_LINK_TARGET_NOT_FOUND"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pending_deployment_blocks_the_link(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("busy1", uuid="u-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("busy1", uuid="u-b")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
|
||||
async with SessionLocal() as db:
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
dep = (await db.scalars(select(AliasDeployment))).one()
|
||||
dep.status = "pending"
|
||||
await db.commit()
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_BUSY"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_host_that_already_has_the_object_is_not_a_candidate_and_cannot_link(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("twice", uuid="u-a")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
|
||||
# a second, differently-UUID'd copy on host A itself: A already has the object
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("twice", uuid="u-a"), alias_row("twice", uuid="u-a2")])
|
||||
rows = (await client.get(f"/api/aliases/unmanaged?host_id={a}")).json()
|
||||
assert [r["uuid"] for r in rows] == ["u-a2"] and rows[0]["matches"] == [] and rows[0]["linkable"] is False
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a2", "link_to": first["id"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "ADOPT_LINK_HOST_HAS_OBJECT"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bulk_never_guesses_between_two_twins_and_links_each_object_once(client, respx_mock, two):
|
||||
a, b = two
|
||||
# two identical central objects exist from earlier separate adoptions
|
||||
for _ in range(2):
|
||||
await client.post("/api/port-forward", json={
|
||||
"description": "Web", "interface": "wan", "ipprotocol": "inet", "protocol": "tcp", "source_net": "any",
|
||||
"destination_net": "wanip", "destination_port": "8443", "target": "192.168.1.50", "local_port": "443",
|
||||
"natreflection": "", "enabled": True, "target_host_ids": [a], "target_group_ids": []})
|
||||
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="pf-b")])
|
||||
row = (await client.get(f"/api/port-forward/unmanaged?host_id={b}")).json()[0]
|
||||
assert len(row["matches"]) == 2 and row["bulk_action"] == "new" # ask the user, do not guess
|
||||
done = (await client.post("/api/port-forward/adopt-all", json={"host_id": b, "link_existing": True})).json()
|
||||
assert [x["linked"] for x in done["adopted"]] == [False]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bulk_links_each_central_object_only_once(client, respx_mock, two):
|
||||
a, b = two
|
||||
await client.post("/api/port-forward", json={
|
||||
"description": "Web", "interface": "wan", "ipprotocol": "inet", "protocol": "tcp", "source_net": "any",
|
||||
"destination_net": "wanip", "destination_port": "8443", "target": "192.168.1.50", "local_port": "443",
|
||||
"natreflection": "", "enabled": True, "target_host_ids": [a], "target_group_ids": []})
|
||||
# two identical live rows on B, ONE twin: the first links, the second becomes its own object
|
||||
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="b-1"), pf_row(uuid="b-2")])
|
||||
rows = (await client.get(f"/api/port-forward/unmanaged?host_id={b}")).json()
|
||||
assert sorted(r["bulk_action"] for r in rows) == ["link", "new"] # the dialog's summary equals what happens
|
||||
done = (await client.post("/api/port-forward/adopt-all", json={"host_id": b, "link_existing": True})).json()
|
||||
assert sorted(x["linked"] for x in done["adopted"]) == [False, True]
|
||||
assert len((await client.get("/api/port-forward")).json()) == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_roles_for_linking(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("roles1", uuid="u-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("roles1", uuid="u-b")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
|
||||
for role, expected in (("read_only", 403), ("operator", 201)):
|
||||
user = await client.post("/api/users", json={"username": f"lnk-{role}", "password": "adopt-test-password-1", "role": role})
|
||||
assert user.status_code == 201, user.text
|
||||
session = await _login(f"lnk-{role}", "adopt-test-password-1")
|
||||
try:
|
||||
resp = await session.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})
|
||||
assert resp.status_code == expected, (role, resp.text)
|
||||
finally:
|
||||
await session.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bulk_leaves_a_twin_alone_that_waits_for_a_confirmation(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("busy2", uuid="u-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("busy2", uuid="u-b")])
|
||||
await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})
|
||||
async with SessionLocal() as db:
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
(await db.scalars(select(AliasDeployment))).one().status = "pending"
|
||||
await db.commit()
|
||||
row = next(r for r in (await client.get(f"/api/aliases/unmanaged?host_id={b}")).json() if r["uuid"] == "u-b")
|
||||
assert row["linkable"] is False and row["bulk_action"] == "skip"
|
||||
resp = await client.post("/api/aliases/adopt-all", json={"host_id": b, "link_existing": True})
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["adopted"] == []
|
||||
assert "twin_busy" in resp.json()["skipped"][0]["reasons"]
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_linking_invalidates_an_open_edit_form(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "alias/searchItem", [alias_row("stale1", uuid="u-a")])
|
||||
_search(respx_mock, B, "alias/searchItem", [alias_row("stale1", uuid="u-b")])
|
||||
first = (await client.post("/api/aliases/adopt", json={"host_id": a, "uuid": "u-a"})).json()
|
||||
linked = (await client.post("/api/aliases/adopt", json={"host_id": b, "uuid": "u-b", "link_to": first["id"]})).json()
|
||||
assert linked["updated_at"] != first["updated_at"]
|
||||
stale = {"name": "stale1", "type": "host", "content": linked["content"], "description": linked["description"],
|
||||
"enabled": linked["enabled"], "target_host_ids": [a], "target_group_ids": [],
|
||||
"expected_updated_at": first["updated_at"]}
|
||||
assert (await client.put(f"/api/aliases/{first['id']}", json=stale)).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bulk_neither_links_nor_duplicates_a_busy_nat_twin(client, respx_mock, two):
|
||||
a, b = two
|
||||
_search(respx_mock, A, "d_nat/searchRule", [pf_row(uuid="pf-a")])
|
||||
_search(respx_mock, B, "d_nat/searchRule", [pf_row(uuid="pf-b")])
|
||||
await client.post("/api/port-forward/adopt", json={"host_id": a, "uuid": "pf-a"})
|
||||
async with SessionLocal() as db:
|
||||
from app.models.port_forward_deployment import PortForwardDeployment
|
||||
(await db.scalars(select(PortForwardDeployment))).one().status = "pending"
|
||||
await db.commit()
|
||||
resp = await client.post("/api/port-forward/adopt-all", json={"host_id": b, "link_existing": True})
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["adopted"] == [] and "twin_busy" in resp.json()["skipped"][0]["reasons"]
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(PortForwardRule)) == 1
|
||||
@@ -0,0 +1,330 @@
|
||||
"""GUI access (1.27): settings, one-time tickets and the gateway's internal redeem/revalidate/end API."""
|
||||
import hashlib
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.main import app
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.gui_access import GuiSession, GuiTicket
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.routers.auth import SECURE_COOKIE_NAME
|
||||
|
||||
GW_TOKEN = "gateway-token-" + "x" * 40
|
||||
SITES = "https://opnexus.example:9441,https://opnexus.example:9442,https://opnexus.example:9443"
|
||||
PW = "gui-user-password-1"
|
||||
|
||||
|
||||
def sha(v: str) -> str:
|
||||
return hashlib.sha256(v.encode()).hexdigest()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gui_env(monkeypatch):
|
||||
monkeypatch.setattr(settings, "cookie_secure", True)
|
||||
monkeypatch.setattr(settings, "gui_gateway_token", GW_TOKEN)
|
||||
monkeypatch.setattr(settings, "opnexus_gui_sites", SITES)
|
||||
|
||||
|
||||
async def _https_login(username: str, password: str) -> AsyncClient:
|
||||
ac = AsyncClient(transport=ASGITransport(app=app), base_url="https://opnexus.example")
|
||||
resp = await ac.post("/api/auth/login", json={"username": username, "password": password})
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert ac.cookies.get(SECURE_COOKIE_NAME)
|
||||
return ac
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def admin(gui_env):
|
||||
ac = await _https_login(settings.admin_username, settings.admin_password)
|
||||
yield ac
|
||||
await ac.aclose()
|
||||
|
||||
|
||||
async def _user(admin: AsyncClient, name: str, role: str) -> AsyncClient:
|
||||
resp = await admin.post("/api/users", json={"username": name, "password": PW, "role": role})
|
||||
if resp.status_code == 409 or (resp.status_code == 400 and "BENUTZERNAME" in resp.text):
|
||||
users = (await admin.get("/api/users")).json()
|
||||
uid = next(u["id"] for u in users if u["username"] == name)
|
||||
await admin.patch(f"/api/users/{uid}/role", json={"role": role})
|
||||
else:
|
||||
assert resp.status_code == 201, resp.text
|
||||
return await _https_login(name, PW)
|
||||
|
||||
|
||||
async def _host(name="fw-gui", base_url="https://fw-gui.test", verify_tls=True, enabled=True) -> uuid.UUID:
|
||||
async with SessionLocal() as db:
|
||||
host = OpnsenseHost(name=name, base_url=base_url, api_key="apikey-must-not-leak", verify_tls=verify_tls,
|
||||
enabled=enabled, tls_ca_pem="-----BEGIN CERTIFICATE-----\nPIN\n-----END CERTIFICATE-----\n")
|
||||
db.add(host)
|
||||
await db.commit()
|
||||
return host.id
|
||||
|
||||
|
||||
async def _enable(admin: AsyncClient, **over) -> dict:
|
||||
body = {"enabled": True, "idle_timeout_minutes": 15, "absolute_timeout_minutes": 60, "slot_count": 4, **over}
|
||||
resp = await admin.put("/api/settings/gui-access", json=body)
|
||||
assert resp.status_code == 200, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
def _ticket_from(url: str) -> str:
|
||||
return url.split("?t=", 1)[1]
|
||||
|
||||
|
||||
async def _internal(path: str, body: dict, token: str = GW_TOKEN):
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://backend:8000") as ac:
|
||||
return await ac.post(f"/internal/gui/{path}", json=body, headers={"X-Onx-Gateway-Token": token})
|
||||
|
||||
|
||||
async def _redeem(admin: AsyncClient, url: str, slot: int | None = None, session_hash: str | None = None):
|
||||
return await _internal("redeem", {
|
||||
"ticket_hash": sha(_ticket_from(url)), "slot": slot or int(url.split(":")[2].split("/")[0]),
|
||||
"session_hash": session_hash or sha(admin.cookies.get(SECURE_COOKIE_NAME)), "client_ip": "192.0.2.10",
|
||||
})
|
||||
|
||||
|
||||
# ── settings ──────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_settings_default_off_and_switchable(admin):
|
||||
data = (await admin.get("/api/settings/gui-access")).json()
|
||||
assert data["enabled"] is False and data["available"] is True and data["slots"] == [9441, 9442, 9443]
|
||||
assert (await _enable(admin, slot_count=2))["slots"] == [9441, 9442]
|
||||
bad = await admin.put("/api/settings/gui-access", json={"enabled": True, "idle_timeout_minutes": 90,
|
||||
"absolute_timeout_minutes": 60, "slot_count": 2})
|
||||
assert bad.status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cannot_enable_without_https_cookies(admin, monkeypatch):
|
||||
monkeypatch.setattr(settings, "cookie_secure", False)
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as plain:
|
||||
await plain.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
|
||||
resp = await plain.put("/api/settings/gui-access", json={"enabled": True, "idle_timeout_minutes": 15,
|
||||
"absolute_timeout_minutes": 60, "slot_count": 4})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_GATEWAY_NOT_CONFIGURED"
|
||||
assert "cookie_secure" in (await plain.get("/api/settings/gui-access")).json()["unavailable_reasons"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_settings_admin_only(admin):
|
||||
op = await _user(admin, "gui-op-settings", "operator")
|
||||
assert (await op.get("/api/settings/gui-access")).status_code == 403
|
||||
assert (await op.put("/api/settings/gui-access", json={"enabled": False, "idle_timeout_minutes": 15,
|
||||
"absolute_timeout_minutes": 60, "slot_count": 4})).status_code == 403
|
||||
|
||||
|
||||
# ── tickets ───────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ticket_issued_and_only_hash_stored(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = resp.json()
|
||||
assert data["url"].startswith("https://opnexus.example:9441/__onx/redeem?t=") and data["expires_in"] == 30
|
||||
raw = _ticket_from(data["url"])
|
||||
async with SessionLocal() as db:
|
||||
ticket = (await db.scalars(select(GuiTicket))).one()
|
||||
assert ticket.ticket_hash == sha(raw) and raw not in repr(ticket.__dict__)
|
||||
assert ticket.session_hash == sha(admin.cookies.get(SECURE_COOKIE_NAME))
|
||||
assert (ticket.expires_at - ticket.created_at) == timedelta(seconds=30)
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "gui_access"))).all()
|
||||
assert [e.action for e in events] == ["ticket"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("role,code", [("operator", "ADMIN_REQUIRED"), ("read_only", "READ_ONLY_ACCESS")])
|
||||
async def test_ticket_refused_for_non_admins(admin, role, code):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
other = await _user(admin, f"gui-{role}", role)
|
||||
resp = await other.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == code
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ticket_refused_for_api_tokens(admin, anon_client):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
token = (await admin.post("/api/tokens", json={"name": "gui", "scope": "admin", "current_password": settings.admin_password})).json()["token"]
|
||||
resp = await anon_client.post(f"/api/hosts/{host_id}/gui-ticket", headers={"Authorization": f"Bearer {token}"})
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ticket_refused_when_switch_off_or_not_configured(admin, monkeypatch):
|
||||
host_id = await _host()
|
||||
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_ACCESS_DISABLED"
|
||||
await _enable(admin)
|
||||
monkeypatch.setattr(settings, "gui_gateway_token", "short")
|
||||
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_GATEWAY_NOT_CONFIGURED"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ticket_refused_without_secure_cookie(admin, monkeypatch):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
monkeypatch.setattr(settings, "cookie_secure", False)
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as plain:
|
||||
await plain.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
|
||||
resp = await plain.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] in ("GUI_REQUIRES_HTTPS", "GUI_GATEWAY_NOT_CONFIGURED")
|
||||
monkeypatch.setattr(settings, "cookie_secure", True)
|
||||
# A legacy plain-name cookie (pre-migration) is not enough either.
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="https://opnexus.example") as legacy:
|
||||
legacy.cookies.set("opnexus-session", admin.cookies.get(SECURE_COOKIE_NAME))
|
||||
resp = await legacy.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "GUI_REQUIRES_HTTPS"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("kwargs,reason", [
|
||||
({"verify_tls": False}, "tls"), ({"base_url": "http://fw-plain.test"}, "https"), ({"enabled": False}, "disabled"),
|
||||
])
|
||||
async def test_ticket_refused_for_ineligible_host(admin, kwargs, reason):
|
||||
host_id = await _host(**kwargs)
|
||||
await _enable(admin)
|
||||
resp = await admin.post(f"/api/hosts/{host_id}/gui-ticket")
|
||||
assert resp.status_code == 409 and resp.json()["detail"] == {"code": "GUI_HOST_NOT_ELIGIBLE", "params": {"reason": reason}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_slots_fixed_per_firewall_and_limited(admin):
|
||||
a, b, c = await _host("fw-a"), await _host("fw-b", "https://fw-b.test"), await _host("fw-c", "https://fw-c.test")
|
||||
await _enable(admin, slot_count=2)
|
||||
slot_a = (await admin.post(f"/api/hosts/{a}/gui-ticket")).json()["slot"]
|
||||
slot_b = (await admin.post(f"/api/hosts/{b}/gui-ticket")).json()["slot"]
|
||||
assert slot_a != slot_b and {slot_a, slot_b} == {9441, 9442}
|
||||
assert (await admin.post(f"/api/hosts/{a}/gui-ticket")).json()["slot"] == slot_a
|
||||
full = await admin.post(f"/api/hosts/{c}/gui-ticket")
|
||||
assert full.status_code == 409 and full.json()["detail"]["code"] == "GUI_NO_FREE_SLOT"
|
||||
|
||||
|
||||
# ── internal API ──────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_internal_api_needs_gateway_token(admin, monkeypatch):
|
||||
body = {"gui_session_id": str(uuid.uuid4()), "session_hash": "0" * 64}
|
||||
assert (await _internal("revalidate", body, token="wrong")).status_code == 401
|
||||
assert (await _internal("revalidate", body, token="")).status_code == 401
|
||||
monkeypatch.setattr(settings, "gui_gateway_token", "")
|
||||
assert (await _internal("revalidate", body, token="")).status_code == 404
|
||||
# Not part of the public API surface.
|
||||
assert "/internal/gui/redeem" not in (await admin.get("/api/openapi.json")).text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redeem_returns_no_credentials_and_is_single_use(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
|
||||
resp = await _redeem(admin, url)
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = resp.json()
|
||||
assert data["upstream_origin"] == "https://fw-gui.test" and "PIN" in data["ca_pem"]
|
||||
assert data["idle_timeout_seconds"] == 900 and data["revalidate_seconds"] == 20
|
||||
assert "apikey-must-not-leak" not in resp.text and "secret" not in resp.text.lower() and "api_key" not in resp.text
|
||||
again = await _redeem(admin, url)
|
||||
assert again.status_code == 403 and again.json()["detail"]["reason"] == "replayed"
|
||||
async with SessionLocal() as db:
|
||||
actions = [e.action for e in (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "gui_access")
|
||||
.order_by(AuditEvent.created_at))).all()]
|
||||
assert actions[:2] == ["ticket", "redeem"] and actions.count("redeem") == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redeem_bound_to_session_slot_and_time(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
|
||||
wrong = await _redeem(admin, url, session_hash=sha("another-browser-session"))
|
||||
assert wrong.status_code == 403 and wrong.json()["detail"]["reason"] == "session"
|
||||
assert (await _redeem(admin, url)).json()["detail"]["reason"] == "replayed" # burned by the failed attempt
|
||||
|
||||
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
|
||||
assert (await _redeem(admin, url, slot=9443)).json()["detail"]["reason"] == "slot"
|
||||
|
||||
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(GuiTicket).where(GuiTicket.ticket_hash == sha(_ticket_from(url)))
|
||||
.values(expires_at=datetime.now(timezone.utc) - timedelta(seconds=1)))
|
||||
await db.commit()
|
||||
assert (await _redeem(admin, url)).json()["detail"]["reason"] == "expired"
|
||||
unknown = await _internal("redeem", {"ticket_hash": sha("nope"), "session_hash": sha("x"), "slot": 9441})
|
||||
assert unknown.status_code == 403 and unknown.json()["detail"]["reason"] == "unknown"
|
||||
|
||||
|
||||
async def _open(admin: AsyncClient, host_id) -> dict:
|
||||
url = (await admin.post(f"/api/hosts/{host_id}/gui-ticket")).json()["url"]
|
||||
data = (await _redeem(admin, url)).json()
|
||||
return {"gui_session_id": data["gui_session_id"], "session_hash": sha(admin.cookies.get(SECURE_COOKIE_NAME))}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revalidate_ok_then_logout_ends_it(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
ref = await _open(admin, host_id)
|
||||
ok = (await _internal("revalidate", ref)).json()
|
||||
assert ok["ok"] is True and ok["revalidate_seconds"] == 20
|
||||
await admin.post("/api/auth/logout")
|
||||
assert (await _internal("revalidate", ref)).json() == {"ok": False, "reason": "session"}
|
||||
assert (await _internal("revalidate", ref)).json()["ok"] is False
|
||||
async with SessionLocal() as db:
|
||||
ends = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "gui_access", AuditEvent.action == "end"))).all()
|
||||
assert len(ends) == 1 and ends[0].details["reason"] == "session" and "duration_seconds" in ends[0].details
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revalidate_fails_after_demotion(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
second = await _user(admin, "gui-second-admin", "admin")
|
||||
ref = await _open(second, host_id)
|
||||
users = (await admin.get("/api/users")).json()
|
||||
uid = next(u["id"] for u in users if u["username"] == "gui-second-admin")
|
||||
assert (await admin.patch(f"/api/users/{uid}/role", json={"role": "operator"})).status_code == 200
|
||||
assert (await _internal("revalidate", ref)).json()["ok"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revalidate_fails_when_switched_off(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
ref = await _open(admin, host_id)
|
||||
await _enable(admin, enabled=False)
|
||||
assert (await _internal("revalidate", ref)).json()["ok"] is False
|
||||
# The live re-check alone (switch flipped without the end-all sweep) also ends it.
|
||||
await _enable(admin)
|
||||
ref = await _open(admin, host_id)
|
||||
from app.models.gui_access import GuiAccessSettings
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(GuiAccessSettings).values(enabled=False))
|
||||
await db.commit()
|
||||
assert (await _internal("revalidate", ref)).json() == {"ok": False, "reason": "disabled"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revalidate_enforces_absolute_limit_and_end(admin):
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
ref = await _open(admin, host_id)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(GuiSession).values(started_at=datetime.now(timezone.utc) - timedelta(minutes=61)))
|
||||
await db.commit()
|
||||
assert (await _internal("revalidate", ref)).json() == {"ok": False, "reason": "absolute"}
|
||||
ref2 = await _open(admin, host_id)
|
||||
assert (await _internal("end", {**ref2, "reason": "idle"})).json() == {"ok": True}
|
||||
assert (await _internal("revalidate", ref2)).json() == {"ok": False, "reason": "idle"}
|
||||
assert (await _internal("end", {**ref2, "session_hash": sha("x"), "reason": "idle"})).json() == {"ok": False}
|
||||
@@ -0,0 +1,470 @@
|
||||
"""GUI gateway (1.27): every counter-measure of the design's threat table against a mock firewall,
|
||||
with the real backend app behind the internal API."""
|
||||
import datetime
|
||||
import logging
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
from cryptography.x509.oid import NameOID
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import select
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import HTMLResponse, Response, StreamingResponse
|
||||
|
||||
from app import gui_gateway as gw
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.main import app as backend_app
|
||||
from app.models.gui_access import GuiSession
|
||||
from app.routers.auth import SECURE_COOKIE_NAME
|
||||
from tests.test_gui_access import GW_TOKEN, SITES, _enable, _host, _https_login, gui_env # noqa: F401
|
||||
|
||||
PASSWORD = "S3cret-Testpasswort!"
|
||||
OWN = "https://opnexus.example:9441"
|
||||
|
||||
|
||||
class MockFirewall:
|
||||
"""Records what reaches the "firewall" and answers like OPNsense's authgui.inc."""
|
||||
|
||||
def __init__(self):
|
||||
self.requests: list[dict] = []
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
request = Request(scope, receive)
|
||||
body = await request.body()
|
||||
self.requests.append({"method": request.method, "path": request.url.path, "query": request.url.query,
|
||||
"headers": dict(request.headers), "body": body})
|
||||
path = request.url.path
|
||||
if path == "/" and request.method == "POST":
|
||||
from urllib.parse import parse_qs
|
||||
if parse_qs(body.decode()).get("passwordfld") == ["right"]:
|
||||
response = Response(status_code=302, headers={"Location": "/ui/core/dashboard"})
|
||||
else:
|
||||
response = HTMLResponse("<form>Wrong username or password.</form>")
|
||||
elif path == "/redir":
|
||||
response = Response(status_code=302, headers={"Location": request.query_params["to"]})
|
||||
elif path == "/logout":
|
||||
response = Response(status_code=302, headers={"Location": "/"})
|
||||
response.headers.append("set-cookie", "PHPSESSID=deleted; expires=Thu, 01 Jan 1970 00:00:01 GMT; path=/")
|
||||
elif path == "/big":
|
||||
async def chunks():
|
||||
for i in range(50):
|
||||
yield (b"%04d" % i) * 1024
|
||||
response = StreamingResponse(chunks(), media_type="application/octet-stream")
|
||||
else:
|
||||
response = HTMLResponse("<html>firewall</html>", headers={"Content-Security-Policy": "default-src 'self'"})
|
||||
response.headers.append("set-cookie", "PHPSESSID=fw-session-1; path=/; secure; HttpOnly")
|
||||
response.headers.append("x-powered-by", "PHP")
|
||||
await response(scope, receive, send)
|
||||
|
||||
|
||||
class Clock:
|
||||
def __init__(self):
|
||||
self.now = 1000.0
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def firewall():
|
||||
return MockFirewall()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def setup(gui_env, firewall): # noqa: F811
|
||||
admin = await _https_login(settings.admin_username, settings.admin_password)
|
||||
host_id = await _host()
|
||||
await _enable(admin)
|
||||
clock = Clock()
|
||||
gateway = gw.GuiGateway(
|
||||
backend=gw.BackendClient("http://backend:8000", GW_TOKEN, transport=ASGITransport(app=backend_app)),
|
||||
slot_origins=gw.slot_origins_from(SITES),
|
||||
upstream_factory=lambda gs: httpx.AsyncClient(transport=ASGITransport(app=firewall)),
|
||||
clock=clock,
|
||||
)
|
||||
browser = AsyncClient(transport=ASGITransport(app=gateway), base_url=OWN)
|
||||
yield {"admin": admin, "host_id": host_id, "gateway": gateway, "browser": browser, "clock": clock, "fw": firewall}
|
||||
await browser.aclose()
|
||||
await admin.aclose()
|
||||
|
||||
|
||||
async def _ticket(s) -> tuple[str, int]:
|
||||
data = (await s["admin"].post(f"/api/hosts/{s['host_id']}/gui-ticket")).json()
|
||||
return data["url"].split("?t=", 1)[1], data["slot"]
|
||||
|
||||
|
||||
def _opnexus_cookie(s) -> str:
|
||||
return f"{SECURE_COOKIE_NAME}={s['admin'].cookies.get(SECURE_COOKIE_NAME)}"
|
||||
|
||||
|
||||
async def _redeem(s, ticket: str, slot: int = 9441, site: str = "same-site", cookie: str | None = None):
|
||||
return await s["browser"].get(f"/__onx/redeem?t={ticket}", headers={
|
||||
"X-Onx-Slot": str(slot), "Sec-Fetch-Site": site, "Cookie": _opnexus_cookie(s) if cookie is None else cookie})
|
||||
|
||||
|
||||
async def _open(s) -> str:
|
||||
ticket, slot = await _ticket(s)
|
||||
resp = await _redeem(s, ticket, slot)
|
||||
assert resp.status_code == 200, resp.text
|
||||
raw = resp.headers["set-cookie"]
|
||||
return raw.split(";")[0] # "__Host-onx-gw-9441=<value>"
|
||||
|
||||
|
||||
async def _get(s, path: str, gw_cookie: str, slot: int = 9441, site: str | None = "same-origin", extra: dict | None = None,
|
||||
method: str = "GET", **kwargs):
|
||||
headers = {"X-Onx-Slot": str(slot), "Cookie": f"{_opnexus_cookie(s)}; {gw_cookie}; other=1", **(extra or {})}
|
||||
if site is not None:
|
||||
headers["Sec-Fetch-Site"] = site
|
||||
return await s["browser"].request(method, path, headers=headers, **kwargs)
|
||||
|
||||
|
||||
# ── redeem ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redeem_sets_host_cookie_and_clears_site_data(setup):
|
||||
ticket, slot = await _ticket(setup)
|
||||
resp = await _redeem(setup, ticket, slot)
|
||||
assert resp.status_code == 200
|
||||
cookie = resp.headers["set-cookie"]
|
||||
assert cookie.startswith(f"__Host-onx-gw-{slot}=")
|
||||
for attr in ("Path=/", "Secure", "HttpOnly", "SameSite=Strict"):
|
||||
assert attr in cookie
|
||||
assert "Domain" not in cookie and ticket not in resp.text
|
||||
assert resp.headers["clear-site-data"] == '"cache", "storage"'
|
||||
assert 'http-equiv="refresh"' in resp.text and resp.headers["cache-control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redeem_refused_without_session_cookie_cross_site_or_twice(setup):
|
||||
ticket, _ = await _ticket(setup)
|
||||
# Refused by the gateway itself, before the backend sees (and burns) the ticket:
|
||||
assert (await _redeem(setup, ticket, cookie="")).status_code == 403
|
||||
assert (await _redeem(setup, ticket, site="cross-site")).status_code == 403
|
||||
assert (await _redeem(setup, ticket, site="")).status_code == 403
|
||||
assert (await _redeem(setup, ticket)).status_code == 200
|
||||
assert (await _redeem(setup, ticket)).status_code == 403 # single use
|
||||
other = await _ticket(setup)
|
||||
assert (await _redeem(setup, other[0], slot=9442)).status_code == 403 # bound to its slot
|
||||
|
||||
|
||||
# ── cookies never cross ───────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_cookies_never_upstream_and_firewall_cookies_never_downstream(setup):
|
||||
cookie = await _open(setup)
|
||||
first = await _get(setup, "/ui/core/dashboard", cookie)
|
||||
assert first.status_code == 200 and "set-cookie" not in first.headers and "x-powered-by" not in first.headers
|
||||
second = await _get(setup, "/ui/x", cookie)
|
||||
assert second.status_code == 200
|
||||
up = setup["fw"].requests
|
||||
assert "cookie" not in up[0]["headers"]
|
||||
assert up[1]["headers"]["cookie"] == "PHPSESSID=fw-session-1"
|
||||
session_value = setup["admin"].cookies.get(SECURE_COOKIE_NAME)
|
||||
for r in up:
|
||||
flat = repr(r["headers"])
|
||||
assert session_value not in flat and "onx-gw" not in flat and "other=1" not in flat
|
||||
# A firewall logout deletes the cookie from the jar.
|
||||
await _get(setup, "/logout", cookie)
|
||||
await _get(setup, "/ui/y", cookie)
|
||||
assert "cookie" not in setup["fw"].requests[-1]["headers"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_rebuilt_with_header_allow_list(setup):
|
||||
cookie = await _open(setup)
|
||||
await _get(setup, "/ui/a?b=1", cookie, extra={
|
||||
"Authorization": "Basic Zm9vOmJhcg==", "Referer": f"{OWN}/ui/prev", "X-Forwarded-For": "192.0.2.99",
|
||||
"X-CSRFToken": "csrf-1", "Accept-Language": "de", "Proxy-Authorization": "x",
|
||||
})
|
||||
r = setup["fw"].requests[-1]
|
||||
assert r["path"] == "/ui/a" and r["query"] == "b=1"
|
||||
h = r["headers"]
|
||||
assert h["host"] == "fw-gui.test" and h["x-csrftoken"] == "csrf-1" and h["accept-language"] == "de"
|
||||
for dropped in ("authorization", "referer", "origin", "x-forwarded-for", "proxy-authorization", "x-onx-slot",
|
||||
"sec-fetch-site"):
|
||||
assert dropped not in h
|
||||
|
||||
|
||||
# ── redirects ─────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("target,expected", [
|
||||
("https://evil.example/x", None), ("//evil.example/x", None), ("/\\evil.example", None),
|
||||
("https://fw-gui.test:8443/x", None), ("http://fw-gui.test/x", None),
|
||||
("https://fw-gui.test/ui/x?y=1", "/ui/x?y=1"), ("https://FW-GUI.test:443/", "/"), ("/ui/z", "/ui/z"),
|
||||
])
|
||||
async def test_location_rewritten_or_dropped(setup, target, expected):
|
||||
cookie = await _open(setup)
|
||||
resp = await _get(setup, "/redir", cookie, params={"to": target})
|
||||
if expected is None:
|
||||
assert resp.status_code == 502 and "location" not in resp.headers
|
||||
else:
|
||||
assert resp.status_code == 302 and resp.headers["location"] == expected
|
||||
|
||||
|
||||
# ── request shape ─────────────────────────────────────────────────────────
|
||||
|
||||
async def _raw(gateway, method="GET", path="/", raw_path=None, headers=None, scope_type="http"):
|
||||
sent = []
|
||||
|
||||
async def receive():
|
||||
return {"type": "http.request", "body": b"", "more_body": False}
|
||||
|
||||
async def send(message):
|
||||
sent.append(message)
|
||||
scope = {"type": scope_type, "method": method, "path": path, "raw_path": raw_path if raw_path is not None else path.encode(),
|
||||
"query_string": b"", "headers": [(k.encode(), v.encode()) for k, v in (headers or {}).items()]}
|
||||
await gateway(scope, receive, send)
|
||||
return sent
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_absolute_form_connect_upgrade_websocket_service_worker_refused(setup):
|
||||
g = setup["gateway"]
|
||||
base = {"x-onx-slot": "9441", "sec-fetch-site": "same-origin"}
|
||||
assert (await _raw(g, path="http://evil.example/", headers=base))[0]["status"] == 400
|
||||
assert (await _raw(g, path="/", raw_path=b"http://evil.example/", headers=base))[0]["status"] == 400
|
||||
assert (await _raw(g, path="//evil.example/", headers=base))[0]["status"] == 400
|
||||
assert (await _raw(g, method="CONNECT", path="/", headers=base))[0]["status"] == 501
|
||||
assert (await _raw(g, method="TRACE", path="/", headers=base))[0]["status"] == 501
|
||||
assert (await _raw(g, headers={**base, "upgrade": "websocket", "connection": "Upgrade"}))[0]["status"] == 400
|
||||
assert (await _raw(g, headers={**base, "service-worker": "script"}))[0]["status"] == 403
|
||||
assert (await _raw(g, scope_type="websocket"))[0]["type"] == "websocket.close"
|
||||
assert setup["fw"].requests == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("slot", ["", "abc", "9999"])
|
||||
async def test_slot_header_required(setup, slot):
|
||||
sent = await _raw(setup["gateway"], headers={"x-onx-slot": slot, "sec-fetch-site": "same-origin"})
|
||||
assert sent[0]["status"] == 400
|
||||
|
||||
|
||||
# ── Sec-Fetch-Site and Origin ─────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("site,expected", [("same-site", 403), ("cross-site", 403), (None, 403), ("none", 200), ("same-origin", 200)])
|
||||
async def test_sec_fetch_site_enforced(setup, site, expected):
|
||||
cookie = await _open(setup)
|
||||
assert (await _get(setup, "/ui/x", cookie, site=site)).status_code == expected
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_origin_required_for_writes(setup):
|
||||
cookie = await _open(setup)
|
||||
assert (await _get(setup, "/api/x", cookie, method="POST", content=b"{}")).status_code == 403
|
||||
other_slot = await _get(setup, "/api/x", cookie, method="POST", content=b"{}", extra={"Origin": "https://opnexus.example:9442"})
|
||||
assert other_slot.status_code == 403
|
||||
opnexus = await _get(setup, "/api/x", cookie, method="POST", content=b"{}", extra={"Origin": "https://opnexus.example"})
|
||||
assert opnexus.status_code == 403
|
||||
assert (await _get(setup, "/api/x", cookie, method="POST", content=b"{}", extra={"Origin": OWN})).status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_foreign_slot_cookie_rejected(setup):
|
||||
cookie = await _open(setup) # slot 9441
|
||||
value = cookie.split("=", 1)[1]
|
||||
resp = await _get(setup, "/ui/x", f"__Host-onx-gw-9442={value}", slot=9442)
|
||||
assert resp.status_code == 401
|
||||
assert (await _get(setup, "/ui/x", "", slot=9441)).status_code == 401
|
||||
assert (await _get(setup, "/__onx/anything", cookie)).status_code == 404
|
||||
|
||||
|
||||
# ── response hardening ────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_security_headers_and_streaming(setup):
|
||||
cookie = await _open(setup)
|
||||
resp = await _get(setup, "/ui/x", cookie)
|
||||
csps = resp.headers.get_list("content-security-policy")
|
||||
assert any("frame-ancestors 'none'" in c and "connect-src 'self'" in c and "form-action 'self'" in c for c in csps)
|
||||
assert "default-src 'self'" in csps # the firewall's own CSP applies as well
|
||||
assert resp.headers["x-frame-options"] == "DENY" and resp.headers["cache-control"] == "no-store"
|
||||
big = await _get(setup, "/big", cookie)
|
||||
assert big.status_code == 200 and len(big.content) == 50 * 4 * 1024 and big.content.endswith(b"0049" * 1024)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_body_limit(setup, monkeypatch):
|
||||
cookie = await _open(setup)
|
||||
monkeypatch.setattr(gw, "BODY_LIMIT", 1000)
|
||||
resp = await _get(setup, "/api/x", cookie, method="POST", content=b"x" * 2000, extra={"Origin": OWN})
|
||||
assert resp.status_code == 413 and setup["fw"].requests == []
|
||||
|
||||
async def stream():
|
||||
for _ in range(4):
|
||||
yield b"y" * 600
|
||||
chunked = await _get(setup, "/api/x", cookie, method="POST", content=stream(), extra={"Origin": OWN})
|
||||
assert chunked.status_code == 413 and setup["fw"].requests == []
|
||||
|
||||
|
||||
# ── login protection and logging ──────────────────────────────────────────
|
||||
|
||||
def _login(password: str) -> dict:
|
||||
return {"content": f"usernamefld=root&passwordfld={password}&login=1".encode(),
|
||||
"extra": {"Origin": OWN, "Content-Type": "application/x-www-form-urlencoded"}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_login_rate_limit_per_firewall(setup):
|
||||
cookie = await _open(setup)
|
||||
for _ in range(3):
|
||||
assert (await _get(setup, "/", cookie, method="POST", **_login("wrong"))).status_code == 200
|
||||
blocked = await _get(setup, "/", cookie, method="POST", **_login("right"))
|
||||
assert blocked.status_code == 429
|
||||
assert sum(1 for r in setup["fw"].requests if r["method"] == "POST") == 3
|
||||
setup["clock"].now += gw.LOGIN_FAILURE_WINDOW + 1
|
||||
await _get(setup, "/ui/x", cookie) # (session re-check due; still valid)
|
||||
assert (await _get(setup, "/", cookie, method="POST", **_login("right"))).status_code == 302
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multipart_login_counted_too(setup):
|
||||
# PHP's $_POST is filled from multipart bodies as well: a login posted that way must not bypass the limit.
|
||||
cookie = await _open(setup)
|
||||
files = {"login": (None, "1"), "usernamefld": (None, "root"), "passwordfld": (None, "wrong")}
|
||||
codes = [(await _get(setup, "/", cookie, method="POST", files=files, extra={"Origin": OWN})).status_code
|
||||
for _ in range(4)]
|
||||
assert codes == [200, 200, 200, 429]
|
||||
assert not gw.is_login_attempt("POST", "multipart/form-data; boundary=x", b'name="file"; name="login"')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_path_forwarded_as_sent_and_control_characters_refused(setup):
|
||||
cookie = await _open(setup)
|
||||
for sent, seen in (("/a%3Fb=1", "/a?b=1"), ("/api/x%2Fy", "/api/x/y"), ("/a%2541", "/a%41"), ("/a%23f", "/a#f")):
|
||||
resp = await _get(setup, sent, cookie)
|
||||
last = setup["fw"].requests[-1]
|
||||
# Reaches the firewall still encoded (or decoded exactly once): never split into a query, never
|
||||
# decoded twice (%2541 must not become "A").
|
||||
assert resp.status_code == 200 and last["path"] in (sent, seen) and last["query"] == "", (sent, last)
|
||||
before = len(setup["fw"].requests)
|
||||
resp = await _get(setup, "/a%0d%0aX-Injected:%201", cookie)
|
||||
assert resp.status_code in (200, 400)
|
||||
assert all("x-injected" not in r["headers"] for r in setup["fw"].requests[before:])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_successful_login_not_counted(setup):
|
||||
cookie = await _open(setup)
|
||||
for _ in range(5):
|
||||
assert (await _get(setup, "/", cookie, method="POST", **_login("right"))).status_code == 302
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_password_and_ticket_never_logged(setup, caplog):
|
||||
caplog.set_level(logging.DEBUG)
|
||||
ticket, slot = await _ticket(setup)
|
||||
resp = await _redeem(setup, ticket, slot)
|
||||
cookie = resp.headers["set-cookie"].split(";")[0]
|
||||
await _get(setup, "/", cookie, method="POST", **_login(PASSWORD))
|
||||
await _get(setup, "/ui/x?secretquery=abc", cookie)
|
||||
text = caplog.text
|
||||
assert "POST / 200" in text and "GET /ui/x 200" in text
|
||||
for leaked in (PASSWORD, ticket, "secretquery", "usernamefld", cookie.split("=", 1)[1], "fw-session-1"):
|
||||
assert leaked not in text
|
||||
|
||||
|
||||
def test_uvicorn_runs_without_access_log(monkeypatch):
|
||||
captured = {}
|
||||
monkeypatch.setenv("GUI_GATEWAY_TOKEN", GW_TOKEN)
|
||||
monkeypatch.setenv("OPNEXUS_GUI_SITES", SITES)
|
||||
import uvicorn
|
||||
monkeypatch.setattr(uvicorn, "run", lambda app, **kw: captured.update(kw))
|
||||
gw.main()
|
||||
assert captured["access_log"] is False and captured["http"] == "h11" and captured["proxy_headers"] is False
|
||||
|
||||
|
||||
def test_gateway_refuses_to_start_without_token(monkeypatch):
|
||||
monkeypatch.setenv("GUI_GATEWAY_TOKEN", "short")
|
||||
monkeypatch.setenv("OPNEXUS_GUI_SITES", SITES)
|
||||
with pytest.raises(SystemExit):
|
||||
gw.create_app()
|
||||
|
||||
|
||||
# ── session lifetime ──────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revalidation_failure_ends_session(setup):
|
||||
cookie = await _open(setup)
|
||||
assert (await _get(setup, "/ui/x", cookie)).status_code == 200
|
||||
await setup["admin"].post("/api/auth/logout")
|
||||
assert (await _get(setup, "/ui/x", cookie)).status_code == 200 # not yet due (≤ 30 s)
|
||||
setup["clock"].now += 31
|
||||
assert (await _get(setup, "/ui/x", cookie)).status_code == 401
|
||||
assert setup["gateway"].sessions == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweep_ends_idle_sessions_and_reports(setup):
|
||||
cookie = await _open(setup)
|
||||
setup["clock"].now += 15 * 60
|
||||
await setup["gateway"].sweep()
|
||||
assert setup["gateway"].sessions == {}
|
||||
assert (await _get(setup, "/ui/x", cookie)).status_code == 401
|
||||
async with SessionLocal() as db:
|
||||
assert (await db.scalars(select(GuiSession.end_reason))).one() == "idle"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_absolute_deadline(setup):
|
||||
cookie = await _open(setup)
|
||||
gs = next(iter(setup["gateway"].sessions.values()))
|
||||
gs.absolute_deadline = setup["gateway"].wall_clock() - 1
|
||||
assert (await _get(setup, "/ui/x", cookie)).status_code == 401
|
||||
assert setup["gateway"].sessions == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_backend_unreachable_fails_closed(setup):
|
||||
cookie = await _open(setup)
|
||||
|
||||
async def broken(gs):
|
||||
raise httpx.ConnectError("down")
|
||||
setup["gateway"].backend.revalidate = broken
|
||||
setup["clock"].now += 31
|
||||
assert (await _get(setup, "/ui/x", cookie)).status_code == 503
|
||||
assert len(setup["fw"].requests) == 0
|
||||
|
||||
|
||||
# ── upstream transport and isolation ──────────────────────────────────────
|
||||
|
||||
def _pem() -> str:
|
||||
key = ec.generate_private_key(ec.SECP256R1())
|
||||
name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "OPNsense.localdomain")])
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
cert = (x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
|
||||
.serial_number(1).not_valid_before(now).not_valid_after(now + datetime.timedelta(days=1)).sign(key, hashes.SHA256()))
|
||||
return cert.public_bytes(serialization.Encoding.PEM).decode()
|
||||
|
||||
|
||||
def test_upstream_client_pins_and_uses_overlay_proxy(monkeypatch):
|
||||
captured = {}
|
||||
monkeypatch.setattr(gw.httpx, "AsyncClient", lambda **kw: captured.update(kw) or "client")
|
||||
gs = gw.GatewaySession(gui_session_id="x", session_hash="y", slot=9441, upstream_origin="https://203.0.113.5",
|
||||
ca_pem=_pem(), proxy_url="http://wg-hub:8888", idle_timeout=900, absolute_deadline=0,
|
||||
revalidate_every=30)
|
||||
assert gw.build_upstream_client(gs) == "client"
|
||||
ctx = captured["verify"]
|
||||
assert ctx.check_hostname is False and ctx.verify_mode.name == "CERT_REQUIRED"
|
||||
assert captured["proxy"] == "http://wg-hub:8888" and captured["follow_redirects"] is False and captured["trust_env"] is False
|
||||
gs.ca_pem, gs.proxy_url = None, None
|
||||
gw.build_upstream_client(gs)
|
||||
assert captured["verify"] is True and captured["proxy"] is None
|
||||
|
||||
|
||||
def test_gateway_never_loads_the_database_layer():
|
||||
code = ("import sys, app.gui_gateway as g; "
|
||||
"g.build_upstream_client(g.GatewaySession('x','y',9441,'https://203.0.113.5',None,None,1,1,1)); "
|
||||
"from app.services import tls_pin_service; "
|
||||
"bad=[m for m in ('app.database','sqlalchemy','app.models') if m in sys.modules]; "
|
||||
"print(bad); sys.exit(1 if bad else 0)")
|
||||
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Recurring maintenance windows: occurrence maths (weekly, monthly, DST, exceptions, cut-off)."""
|
||||
from datetime import date, datetime, time, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
from app.services import window_service as ws
|
||||
|
||||
UTC = timezone.utc
|
||||
|
||||
|
||||
def _w(**kw) -> MaintenanceWindow:
|
||||
base = dict(name="w", timezone="Europe/Berlin", recurrence="weekly", weekdays=[6], start_time=time(2, 0),
|
||||
duration_minutes=180, start_cutoff_minutes=60, enabled=True, created_by="admin")
|
||||
base.update(kw)
|
||||
w = MaintenanceWindow(**base)
|
||||
w.exceptions = kw.get("exceptions", [])
|
||||
return w
|
||||
|
||||
|
||||
def _starts(w, after, n=4, **kw):
|
||||
return [o.start for o in ws.occurrences(w, after, n, **kw)]
|
||||
|
||||
|
||||
def test_weekly_sunday_in_summer_and_winter_keeps_local_time():
|
||||
w = _w()
|
||||
# 2026-10-18 is a Sunday (CEST, +2), 2026-11-01 a Sunday (CET, +1)
|
||||
starts = _starts(w, datetime(2026, 10, 12, tzinfo=UTC), 4)
|
||||
assert starts[0] == datetime(2026, 10, 18, 0, 0, tzinfo=UTC)
|
||||
assert starts[2] == datetime(2026, 11, 1, 1, 0, tzinfo=UTC)
|
||||
assert all(s.astimezone(ws.ZoneInfo("Europe/Berlin")).hour == 2 for s in starts)
|
||||
|
||||
|
||||
def test_weekly_several_days_in_order():
|
||||
w = _w(weekdays=[1, 3], timezone="UTC", start_time=time(22, 0), duration_minutes=60)
|
||||
starts = _starts(w, datetime(2026, 10, 19, tzinfo=UTC), 3) # Monday
|
||||
assert starts == [datetime(2026, 10, 20, 22, tzinfo=UTC), datetime(2026, 10, 22, 22, tzinfo=UTC),
|
||||
datetime(2026, 10, 27, 22, tzinfo=UTC)]
|
||||
|
||||
|
||||
def test_monthly_nth_weekday_first_and_last():
|
||||
first_tue = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=1, month_weekday=1, timezone="UTC")
|
||||
assert [s.date() for s in _starts(first_tue, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
|
||||
date(2026, 10, 6), date(2026, 11, 3), date(2026, 12, 1)]
|
||||
last_sun = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=-1, month_weekday=6, timezone="UTC")
|
||||
assert [s.date() for s in _starts(last_sun, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
|
||||
date(2026, 10, 25), date(2026, 11, 29), date(2026, 12, 27)]
|
||||
|
||||
|
||||
def test_monthly_fixed_day():
|
||||
w = _w(recurrence="monthly", weekdays=[], month_mode="day", month_day=15, timezone="UTC")
|
||||
assert [s.date() for s in _starts(w, datetime(2026, 10, 16, tzinfo=UTC), 2)] == [date(2026, 11, 15), date(2026, 12, 15)]
|
||||
|
||||
|
||||
def test_spring_forward_missing_time_moves_forward():
|
||||
# 2026-03-29 (last Sunday of March): 02:00-03:00 local does not exist in Europe/Berlin.
|
||||
w = _w(start_time=time(2, 30))
|
||||
occ = ws.occurrences(w, datetime(2026, 3, 28, tzinfo=UTC), 1)[0]
|
||||
assert occ.local_date == date(2026, 3, 29)
|
||||
assert occ.start == datetime(2026, 3, 29, 1, 30, tzinfo=UTC) # = 03:30 CEST
|
||||
assert occ.start.astimezone(ws.ZoneInfo("Europe/Berlin")).strftime("%H:%M") == "03:30"
|
||||
|
||||
|
||||
def test_fall_back_ambiguous_time_uses_the_first_and_duration_is_real_time():
|
||||
# 2026-10-25 (last Sunday of October): 02:00-03:00 local happens twice.
|
||||
w = _w(start_time=time(2, 30), duration_minutes=120)
|
||||
occ = ws.occurrences(w, datetime(2026, 10, 24, tzinfo=UTC), 1)[0]
|
||||
assert occ.start == datetime(2026, 10, 25, 0, 30, tzinfo=UTC) # 02:30 CEST (first pass)
|
||||
assert occ.end - occ.start == timedelta(hours=2)
|
||||
|
||||
|
||||
def test_skip_and_move_exceptions():
|
||||
moved = datetime(2026, 10, 31, 20, 0, tzinfo=UTC) # Saturday evening instead of Sunday night
|
||||
w = _w(exceptions=[
|
||||
MaintenanceWindowException(occurrence_date=date(2026, 10, 25), kind="skip", reason="x", actor="a"),
|
||||
MaintenanceWindowException(occurrence_date=date(2026, 11, 1), kind="move", moved_start=moved, reason="x", actor="a"),
|
||||
])
|
||||
occs = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3)
|
||||
assert [o.local_date for o in occs] == [date(2026, 11, 1), date(2026, 11, 8), date(2026, 11, 15)]
|
||||
assert occs[0].moved and occs[0].start == moved and occs[0].end == moved + timedelta(minutes=180)
|
||||
listed = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3, with_skipped=True)
|
||||
assert [(o.local_date, o.skipped, o.moved) for o in listed] == [
|
||||
(date(2026, 10, 25), True, False), (date(2026, 11, 1), False, True), (date(2026, 11, 8), False, False)]
|
||||
|
||||
|
||||
def test_open_window_is_current_and_cutoff_limits_new_starts():
|
||||
w = _w(timezone="UTC", weekdays=[6], start_time=time(2, 0), duration_minutes=180, start_cutoff_minutes=60)
|
||||
start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
|
||||
assert ws.current(w, start - timedelta(seconds=1)) is None
|
||||
assert ws.current(w, start).start == start
|
||||
assert ws.startable(w, start + timedelta(minutes=59)) is not None
|
||||
assert ws.startable(w, start + timedelta(minutes=60)) is None # past the cut-off: nothing new starts
|
||||
assert ws.current(w, start + timedelta(minutes=179)) is not None # still open, running work may finish
|
||||
assert ws.current(w, start + timedelta(minutes=180)) is None
|
||||
# an occurrence that is open right now is the first one listed
|
||||
assert ws.occurrences(w, start + timedelta(minutes=30), 1)[0].start == start
|
||||
|
||||
|
||||
def test_cutoff_never_exceeds_duration_and_disabled_window_is_never_open():
|
||||
w = _w(timezone="UTC", duration_minutes=30, start_cutoff_minutes=90)
|
||||
start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
|
||||
assert ws.occurrences(w, start, 1)[0].cutoff == start + timedelta(minutes=30)
|
||||
w.enabled = False
|
||||
assert ws.current(w, start) is None and ws.startable(w, start) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,ok", [("Europe/Berlin", True), ("UTC", True), ("Mars/Olympus", False), ("", False), ("../etc", False)])
|
||||
def test_timezone_validation(name, ok):
|
||||
assert ws.valid_timezone(name) is ok
|
||||
|
||||
|
||||
# ── API ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
from sqlalchemy import select # noqa: E402
|
||||
|
||||
from app.database import SessionLocal # noqa: E402
|
||||
from app.models.audit import AuditEvent # noqa: E402
|
||||
|
||||
_WEEKLY = {"name": "Sonntagnacht", "timezone": "Europe/Berlin", "recurrence": "weekly", "weekdays": [6],
|
||||
"start_time": "02:00", "duration_minutes": 180, "start_cutoff_minutes": 60}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_window_crud_and_next_occurrences(client):
|
||||
resp = await client.post("/api/maintenance-windows", json=_WEEKLY)
|
||||
assert resp.status_code == 201, resp.text
|
||||
w = resp.json()
|
||||
assert w["start_time"] == "02:00" and len(w["next"]) == 5 and w["exceptions"] == [] and w["used_by"] == []
|
||||
assert (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()["detail"]["code"] == "WINDOW_NAME_TAKEN"
|
||||
upd = await client.put(f"/api/maintenance-windows/{w['id']}", json={**_WEEKLY, "start_time": "03:15", "start_cutoff_minutes": 999})
|
||||
assert upd.status_code == 200 and upd.json()["start_time"] == "03:15"
|
||||
assert upd.json()["start_cutoff_minutes"] == 180 # never longer than the window itself
|
||||
occ = await client.get(f"/api/maintenance-windows/{w['id']}/occurrences?count=12")
|
||||
assert occ.status_code == 200 and len(occ.json()) == 12
|
||||
assert (await client.delete(f"/api/maintenance-windows/{w['id']}")).status_code == 204
|
||||
assert (await client.get(f"/api/maintenance-windows/{w['id']}")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change,field", [
|
||||
({"timezone": "Mars/Base"}, "timezone"),
|
||||
({"start_time": "25:00"}, "start_time"),
|
||||
({"weekdays": []}, "weekdays"),
|
||||
({"weekdays": [7]}, "weekdays"),
|
||||
({"recurrence": "monthly", "month_mode": "day", "month_day": 31}, "month_day"),
|
||||
({"recurrence": "monthly", "month_mode": "nth_weekday", "month_nth": 5, "month_weekday": 1}, "month_nth"),
|
||||
({"recurrence": "monthly"}, "month_mode"),
|
||||
])
|
||||
@pytest.mark.asyncio
|
||||
async def test_window_validation(client, change, field):
|
||||
resp = await client.post("/api/maintenance-windows", json={**_WEEKLY, **change})
|
||||
assert resp.status_code == 422 and resp.json()["detail"] == {"code": "WINDOW_INVALID", "params": {"field": field}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_exceptions_skip_move_and_audit(client):
|
||||
w = (await client.post("/api/maintenance-windows", json={**_WEEKLY, "timezone": "UTC"})).json()
|
||||
first = date.fromisoformat(w["next"][0]["local_date"])
|
||||
second = date.fromisoformat(w["next"][1]["local_date"])
|
||||
bad = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first + timedelta(days=1)).isoformat(), "kind": "skip"})
|
||||
assert bad.status_code == 422 and bad.json()["detail"]["params"]["field"] == "occurrence_date"
|
||||
past = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first - timedelta(days=14)).isoformat(), "kind": "skip"})
|
||||
assert past.status_code == 422
|
||||
skip = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip", "reason": "Feiertag"})
|
||||
assert skip.status_code == 201, skip.text
|
||||
assert skip.json()["next"][0]["skipped"] is True
|
||||
dup = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip"})
|
||||
assert dup.status_code == 409 and dup.json()["detail"]["code"] == "WINDOW_EXCEPTION_EXISTS"
|
||||
no_target = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move"})
|
||||
assert no_target.status_code == 422 and no_target.json()["detail"]["params"]["field"] == "moved_start"
|
||||
target = datetime.combine(second, time(20, 0), tzinfo=UTC) - timedelta(days=1)
|
||||
moved = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move", "moved_start": target.isoformat()})
|
||||
assert moved.status_code == 201
|
||||
regular = [o for o in moved.json()["next"] if not o["skipped"]]
|
||||
assert regular[0]["moved"] is True and datetime.fromisoformat(regular[0]["start"]) == target
|
||||
eid = moved.json()["exceptions"][1]["id"]
|
||||
after = await client.delete(f"/api/maintenance-windows/{w['id']}/exceptions/{eid}")
|
||||
assert after.status_code == 200 and len(after.json()["exceptions"]) == 1
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "maintenance_window"))).all()
|
||||
assert any(e.action == "exception_skip" and e.details["reason"] == "Feiertag" for e in events)
|
||||
assert any(e.status == "accepted" and e.action == "delete" for e in events)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_window_writes_follow_the_deploy_right(client):
|
||||
from tests.test_operator_role import _operator
|
||||
op = await _operator(client, "winop")
|
||||
try:
|
||||
made = await op.post("/api/maintenance-windows", json={**_WEEKLY, "name": "op-window"})
|
||||
assert made.status_code == 201 # operators may prepare windows while they may deploy
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
for method, path, body in [
|
||||
("POST", "/api/maintenance-windows", {**_WEEKLY, "name": "x"}),
|
||||
("PUT", f"/api/maintenance-windows/{made.json()['id']}", _WEEKLY),
|
||||
("DELETE", f"/api/maintenance-windows/{made.json()['id']}", None),
|
||||
("POST", f"/api/maintenance-windows/{made.json()['id']}/exceptions", {"occurrence_date": "2030-01-06", "kind": "skip"}),
|
||||
]:
|
||||
resp = await op.request(method, path, json=body)
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN", path
|
||||
assert (await op.get("/api/maintenance-windows")).status_code == 200
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
@@ -206,3 +206,53 @@ async def test_fresh_database_needs_no_backup(scratch, monkeypatch):
|
||||
await migrate.run() # würde ohne Backup-Ziel scheitern, wenn eins verlangt würde
|
||||
from pathlib import Path
|
||||
assert not Path(settings.migration_backup_dir).exists()
|
||||
|
||||
|
||||
async def _alembic(engine, action: str, revision: str) -> None:
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(lambda sync_conn: migrate._run_alembic(sync_conn, action, revision))
|
||||
|
||||
|
||||
async def test_0020_upgrade_keeps_existing_settings_and_downgrade_is_clean(scratch):
|
||||
"""1.23 -> 1.24 with a real settings row: the new alert columns get their defaults, the stored webhook
|
||||
and flags stay, the channel table appears; a downgrade removes exactly that again and re-upgrading works."""
|
||||
await _alembic(scratch, "upgrade", "0019")
|
||||
async with scratch.begin() as conn:
|
||||
# Every NOT NULL column of the 0019 table gets a value, so the row is what a real 1.23 install has.
|
||||
required = (await conn.execute(text(
|
||||
"SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'notification_settings' "
|
||||
"AND is_nullable = 'NO' AND column_default IS NULL"
|
||||
))).all()
|
||||
by_type = {"boolean": "true", "integer": "300", "json": "'[]'", "timestamp with time zone": "now()"}
|
||||
values = {name: by_type.get(kind, "'x'") for name, kind in required}
|
||||
values.update({"id": "1", "webhook_enabled": "true", "webhook_url": "'https://hooks.example/x'"})
|
||||
await conn.execute(text(
|
||||
f"INSERT INTO notification_settings ({', '.join(values)}) VALUES ({', '.join(values.values())})"
|
||||
))
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
async with scratch.connect() as conn:
|
||||
row = (await conn.execute(text(
|
||||
"SELECT webhook_enabled, webhook_url, gateway_down_enabled, resource_threshold_enabled, service_down_enabled, "
|
||||
"disk_threshold_pct, load_per_cpu_threshold, alert_confirm_samples, resource_confirm_samples, service_watchlist "
|
||||
"FROM notification_settings"
|
||||
))).one()
|
||||
assert tuple(row) == (True, "https://hooks.example/x", True, False, False, 90, 2.0, 3, 5, None)
|
||||
await conn.execute(text(
|
||||
"INSERT INTO notification_channels (id, name, channel_type, config, created_at, updated_at) "
|
||||
"VALUES (gen_random_uuid(), 'n', 'ntfy', '{\"topic\": \"t\"}', now(), now())"
|
||||
))
|
||||
await conn.commit()
|
||||
assert await _schema_diff(scratch) == []
|
||||
|
||||
await _alembic(scratch, "downgrade", "0019")
|
||||
async with scratch.connect() as conn:
|
||||
assert (await conn.execute(text("SELECT to_regclass('notification_channels')"))).scalar() is None
|
||||
columns = set((await conn.execute(text(
|
||||
"SELECT column_name FROM information_schema.columns WHERE table_name = 'notification_settings'"
|
||||
))).scalars().all())
|
||||
assert "gateway_down_enabled" not in columns and "service_watchlist" not in columns
|
||||
assert (await conn.execute(text("SELECT webhook_url FROM notification_settings"))).scalar() == "https://hooks.example/x"
|
||||
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head()
|
||||
assert await _schema_diff(scratch) == []
|
||||
@@ -0,0 +1,717 @@
|
||||
"""Monitoring alerts (1.24): gateway health, resource thresholds, stopped services, extra channels."""
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import (
|
||||
gateway_health_service, monitor_checks, notification_channels, notification_secret_store, resource_monitor_service,
|
||||
url_guard,
|
||||
)
|
||||
from app.services.notification_delivery import NotificationDeliveryError
|
||||
from app.services.notification_service import run_notification_checks
|
||||
from tests.test_notifications import _create_host
|
||||
from tests.test_resource_monitor import _mock_resource_endpoints, _mock_vip
|
||||
|
||||
# What routes/gateway/status returns on a real 26.7 firewall (test-fw-3, live), plus a degraded and a dead gateway.
|
||||
STATUS_PAYLOAD = {"status": "ok", "items": [
|
||||
{"name": "WAN_GW", "address": "203.0.113.1", "status": "none", "loss": "0.0 %", "delay": "1.9 ms", "stddev": "0.3 ms", "monitor": "203.0.113.1"},
|
||||
{"name": "LTE_GW", "address": "192.0.2.9", "status": "delay+loss", "loss": "12.5 %", "delay": "310.2 ms", "stddev": "40.0 ms", "monitor": "192.0.2.9"},
|
||||
{"name": "DOWN_GW", "address": "192.0.2.1", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~", "monitor": "192.0.2.1"},
|
||||
{"name": "WAN_DHCP6", "address": "~", "status": "none", "loss": "~", "delay": "~", "stddev": "~", "monitor": "~"},
|
||||
{"name": "OFF_GW", "address": "192.0.2.5", "status": "force_down", "loss": "~", "delay": "~", "stddev": "~", "monitor": "~"},
|
||||
]}
|
||||
|
||||
|
||||
# ── Gateway parsing ──────────────────────────────────────────────────────────
|
||||
|
||||
def test_parse_gateway_status_maps_states_and_numbers():
|
||||
by_name = {g.name: g for g in gateway_health_service.parse_gateway_status(STATUS_PAYLOAD)}
|
||||
assert by_name["WAN_GW"].state == "online" and by_name["WAN_GW"].loss == 0.0 and by_name["WAN_GW"].delay == 1.9
|
||||
assert by_name["WAN_GW"].stddev == 0.3
|
||||
assert by_name["LTE_GW"].state == "degraded" and by_name["LTE_GW"].loss == 12.5
|
||||
assert by_name["DOWN_GW"].state == "offline" and by_name["DOWN_GW"].delay is None
|
||||
# nothing measured -> unknown, never "online" and never 0 ms
|
||||
assert by_name["WAN_DHCP6"].state == "unknown" and by_name["WAN_DHCP6"].loss is None and by_name["WAN_DHCP6"].address is None
|
||||
assert by_name["OFF_GW"].state == "forced_down"
|
||||
|
||||
|
||||
def test_dead_gateway_has_no_round_trip_time():
|
||||
# dpinger reports "100.0 %" loss together with "0.0 ms" when nothing answers: 0 ms must not look fast
|
||||
payload = {"items": [{"name": "GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms", "monitor": "192.0.2.1"}]}
|
||||
gw = gateway_health_service.parse_gateway_status(payload)[0]
|
||||
assert gw.state == "offline" and gw.loss == 100.0 and gw.delay is None and gw.stddev is None
|
||||
|
||||
|
||||
def test_parse_gateway_status_tolerates_failed_answer():
|
||||
assert gateway_health_service.parse_gateway_status({"items": [], "status": "failed"}) == []
|
||||
assert gateway_health_service.parse_gateway_status(None) == []
|
||||
|
||||
|
||||
def test_is_bad_thresholds():
|
||||
by_name = {g.name: g for g in gateway_health_service.parse_gateway_status(STATUS_PAYLOAD)}
|
||||
assert gateway_health_service.is_bad(by_name["DOWN_GW"])
|
||||
assert not gateway_health_service.is_bad(by_name["OFF_GW"]) # switched off on purpose
|
||||
assert not gateway_health_service.is_bad(by_name["WAN_DHCP6"])
|
||||
assert not gateway_health_service.is_bad(by_name["LTE_GW"]) # degraded by OPNsense' limits only, own thresholds off
|
||||
assert gateway_health_service.is_bad(by_name["LTE_GW"], loss_threshold_pct=10)
|
||||
assert gateway_health_service.is_bad(by_name["LTE_GW"], delay_threshold_ms=300)
|
||||
assert not gateway_health_service.is_bad(by_name["WAN_GW"], loss_threshold_pct=10, delay_threshold_ms=300)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_gateway_health_reads_live(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(200, json=STATUS_PAYLOAD)
|
||||
resp = await client.get(f"/api/hosts/{host['id']}/gateways")
|
||||
assert resp.status_code == 200, resp.text
|
||||
rows = {g["name"]: g for g in resp.json()}
|
||||
assert rows["DOWN_GW"]["state"] == "offline" and rows["WAN_GW"]["delay"] == 1.9
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(500)
|
||||
assert (await client.get(f"/api/hosts/{host['id']}/gateways")).status_code == 502
|
||||
|
||||
|
||||
# ── Sampler stores the extra readings ────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sampler_stores_gateways_services_swap_and_cpu(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
await _create_host(client, "fw1", "https://fw1.test")
|
||||
_mock_resource_endpoints(respx_mock)
|
||||
respx_mock.get("https://fw1.test/api/diagnostics/system/systemSwap").respond(200, json={"swap": [{"device": "/dev/da0p3", "total": "8388608", "used": "2097152"}]})
|
||||
respx_mock.get("https://fw1.test/api/diagnostics/cpu_usage/getCPUType").respond(200, json=["QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)"])
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(200, json=STATUS_PAYLOAD)
|
||||
respx_mock.get("https://fw1.test/api/core/service/search").respond(200, json={"rows": [
|
||||
{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
|
||||
{"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0},
|
||||
{"id": "pf", "name": "pf", "locked": 1, "running": 1},
|
||||
{"id": "3f2a", "name": "openvpn", "locked": 0, "running": 0},
|
||||
]})
|
||||
async with SessionLocal() as db:
|
||||
await resource_monitor_service.poll_all_hosts(db)
|
||||
raw = (await db.scalars(select(HostResourceSample))).one().raw
|
||||
assert raw["swap_used_pct"] == 25.0 and raw["cpu_threads"] == 6
|
||||
assert {g["name"]: g["state"] for g in raw["gateways"]}["DOWN_GW"] == "offline"
|
||||
assert raw["services"] == {"unbound": 1, "dhcpd6": 0, "openvpn/3f2a": 0} # locked services are not recorded
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sampler_survives_missing_optional_endpoints(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
await _create_host(client, "fw1", "https://fw1.test")
|
||||
_mock_resource_endpoints(respx_mock) # nothing for swap/cpu/gateways/services -> unmocked routes fail
|
||||
async with SessionLocal() as db:
|
||||
result = await resource_monitor_service.poll_all_hosts(db)
|
||||
row = (await db.scalars(select(HostResourceSample))).one()
|
||||
assert result["sampled"] == 1 and row.mem_used_pct == 25.0
|
||||
assert row.raw["gateways"] is None and row.raw["services"] is None and row.raw["swap_used_pct"] is None
|
||||
|
||||
|
||||
# ── Debouncing ───────────────────────────────────────────────────────────────
|
||||
|
||||
def test_verdict_needs_agreement_and_keeps_state_otherwise():
|
||||
v = monitor_checks.verdict
|
||||
assert v([True, True, True], 3, False) is True
|
||||
assert v([True, True, False], 3, False) is False # no single spike / short blip
|
||||
assert v([True, True], 3, False) is False # not enough history yet
|
||||
assert v([False, False, False], 3, True) is False # needs the full window to close
|
||||
assert v([False, True, False], 3, True) is True # flapping keeps an open alert open
|
||||
assert v([True, None, True], 3, False) is False # unreadable sample is no evidence
|
||||
assert v([None, None, None], 3, True) is True # ... and does not clear either
|
||||
assert v([True], 1, False) is True
|
||||
|
||||
|
||||
async def _add_samples(host_id, rows: list[dict], *, age_minutes: int = 0) -> None:
|
||||
"""rows: newest first; each {mem, load, disk, raw...}. One sample per minute, the newest ``age_minutes`` old."""
|
||||
now = datetime.now(timezone.utc)
|
||||
async with SessionLocal() as db:
|
||||
for i, row in enumerate(rows):
|
||||
raw = {k: v for k, v in row.items() if k not in ("mem", "load")}
|
||||
db.add(HostResourceSample(
|
||||
host_id=host_id, sampled_at=now - timedelta(minutes=age_minutes + i),
|
||||
mem_used_pct=row.get("mem"), load_avg_1m=row.get("load"), disk_used_pct=None, raw=raw,
|
||||
))
|
||||
await db.commit()
|
||||
|
||||
|
||||
def _gw(state="online", loss=0.0, delay=2.0, name="WAN_GW", status=None):
|
||||
return {"name": name, "status": status or {"offline": "down", "online": "none"}.get(state, state), "state": state,
|
||||
"loss": loss, "delay": delay, "stddev": 0.1}
|
||||
|
||||
|
||||
async def _set(**fields):
|
||||
async with SessionLocal() as db:
|
||||
row = await db.get(NotificationSettings, 1) or NotificationSettings(id=1)
|
||||
for k, v in fields.items():
|
||||
setattr(row, k, v)
|
||||
db.add(row)
|
||||
await db.commit()
|
||||
|
||||
|
||||
_ALL_FLAGS = [
|
||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled", "revert_failed_enabled", "maintenance_unknown_enabled",
|
||||
"carp_health_enabled", "nat_drift_enabled", "network_drift_enabled", "carp_auto_revert_failed_enabled",
|
||||
"scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled",
|
||||
]
|
||||
|
||||
|
||||
async def _only(*enabled: str):
|
||||
"""Run run_notification_checks with just these checks on (the mocked firewalls are unreachable, which would
|
||||
otherwise also open host_offline problems)."""
|
||||
await _set(**{flag: flag in enabled for flag in _ALL_FLAGS})
|
||||
|
||||
|
||||
async def _open_count(db, problem_type: str | None = None) -> int:
|
||||
query = select(NotificationProblem).where(NotificationProblem.status == "open")
|
||||
if problem_type:
|
||||
query = query.where(NotificationProblem.problem_type == problem_type)
|
||||
return len((await db.scalars(query)).all())
|
||||
|
||||
|
||||
async def _gateway_problems():
|
||||
async with SessionLocal() as db:
|
||||
return await monitor_checks.check_gateway_down(db)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_down_needs_consecutive_samples(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"gateways": [_gw("offline", loss=100.0, delay=None)]}, {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}])
|
||||
assert dict(await _gateway_problems()) == {} # only two bad samples in a row
|
||||
|
||||
await _add_samples(hid, [{"gateways": [_gw("offline", 100.0, None)]}] * 3, age_minutes=-3) # three newer bad ones
|
||||
problems = await _gateway_problems()
|
||||
assert list(problems) == [f"{hid}:WAN_GW"]
|
||||
ctx = problems[f"{hid}:WAN_GW"]
|
||||
assert ctx["host_name"] == "fw1" and ctx["gateway"] == "WAN_GW" and ctx["grund"] == "Gateway offline"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_alert_resolves_only_after_full_recovery_and_ignores_unknown(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
bad, good = {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}
|
||||
await _only("gateway_down_enabled")
|
||||
await _add_samples(hid, [bad, bad, bad, good, good])
|
||||
async with SessionLocal() as db:
|
||||
await run_notification_checks(db)
|
||||
assert await _open_count(db) == 1
|
||||
|
||||
# one good sample after the outage, then a relapse: still open (flapping link must not re-notify)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [bad, good, bad, bad, good])
|
||||
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"]
|
||||
# a sample that could not read the gateways does not clear it either
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"gateways": None}] * 3)
|
||||
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"]
|
||||
# three good ones close it
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [good, good, good])
|
||||
assert dict(await _gateway_problems()) == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_thresholds_and_forced_down(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
lossy = {"gateways": [_gw("degraded", loss=35.0, delay=20.0, status="loss"), _gw("forced_down", None, None, name="OFF", status="force_down")]}
|
||||
await _add_samples(hid, [lossy] * 3)
|
||||
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"] # 35 % >= 20 %; the switched-off gateway is not reported
|
||||
await _set(gateway_loss_threshold_pct=0)
|
||||
assert dict(await _gateway_problems()) == {} # own loss threshold off; OPNsense says "degraded", not "down"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stale_samples_are_not_judged_and_keep_open_problems(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
bad = {"gateways": [_gw("offline", 100.0, None)]}
|
||||
await _only("gateway_down_enabled")
|
||||
await _add_samples(hid, [bad] * 3)
|
||||
async with SessionLocal() as db:
|
||||
await run_notification_checks(db)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"gateways": [_gw()]}] * 3, age_minutes=30) # the host stopped answering 30 min ago
|
||||
async with SessionLocal() as db:
|
||||
result = await run_notification_checks(db)
|
||||
assert result["resolved"] == 0
|
||||
assert await _open_count(db) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_alert_is_delivered_once_and_resolution_follows(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _only("gateway_down_enabled")
|
||||
await _set(webhook_enabled=True, webhook_url="https://hooks.test/n")
|
||||
hook = respx_mock.post("https://hooks.test/n").mock(return_value=httpx.Response(200))
|
||||
bad, good = {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}
|
||||
await _add_samples(hid, [bad] * 3)
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 0
|
||||
assert hook.call_count == 1 and b"Gateway" in hook.calls.last.request.content
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [good] * 3)
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["resolved"] == 1
|
||||
assert hook.call_count == 2 and b"behoben" in hook.calls.last.request.content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_check_is_on_by_default_and_the_others_are_off(client):
|
||||
data = (await client.get("/api/notifications/settings")).json()
|
||||
assert data["gateway_down_enabled"] is True
|
||||
assert data["resource_threshold_enabled"] is False and data["service_down_enabled"] is False
|
||||
assert data["service_watchlist"][:2] == ["unbound", "dnsmasq"]
|
||||
|
||||
|
||||
# ── Resource thresholds ──────────────────────────────────────────────────────────
|
||||
|
||||
DISK_OK = [{"device": "zroot/ROOT/default", "type": "zfs", "used_pct": 40, "mountpoint": "/"}, {"device": "devfs", "type": "devfs", "used_pct": 100, "mountpoint": "/dev"}]
|
||||
DISK_FULL = [{"device": "zroot/ROOT/default", "type": "zfs", "used_pct": 93, "mountpoint": "/"},
|
||||
{"device": "zroot/var/log", "type": "zfs", "used_pct": 91, "mountpoint": "/var/log"},
|
||||
{"device": "devfs", "type": "devfs", "used_pct": 100, "mountpoint": "/dev"}]
|
||||
|
||||
|
||||
async def _resource_problems():
|
||||
async with SessionLocal() as db:
|
||||
return await monitor_checks.check_resource_threshold(db)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_disk_threshold_one_alert_per_pool_and_ignores_devfs(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"disk": DISK_FULL, "mem": 10, "load": 0.1}] * 5)
|
||||
problems = await _resource_problems()
|
||||
assert list(problems) == [f"{hid}:disk:zroot"]
|
||||
assert problems[f"{hid}:disk:zroot"]["wert_prozent"] == 93 and problems[f"{hid}:disk:zroot"]["schwelle_prozent"] == 90
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_memory_spike_does_not_alert_but_sustained_use_does(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
calm = {"disk": DISK_OK, "mem": 40, "load": 0.1}
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 99, "load": 0.1}, calm, calm, calm, calm])
|
||||
assert dict(await _resource_problems()) == {}
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 96, "load": 0.1}] * 5)
|
||||
assert list(await _resource_problems()) == [f"{hid}:memory"]
|
||||
await _set(memory_threshold_pct=0) # 0 = off
|
||||
assert dict(await _resource_problems()) == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_swap_and_cpu_per_thread(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
# load 6 on 2 threads = 3 per thread (>= 2.0); load 6 on 8 threads would be fine
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 6.0, "swap_used_pct": 70.0, "cpu_threads": 2}] * 5)
|
||||
assert sorted(await _resource_problems()) == sorted([f"{hid}:cpu", f"{hid}:swap"])
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 6.0, "swap_used_pct": None, "cpu_threads": 8}] * 5)
|
||||
assert dict(await _resource_problems()) == {}
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
# unknown thread count: the load cannot be judged, so it is not reported
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 50.0, "swap_used_pct": None, "cpu_threads": None}] * 5)
|
||||
assert dict(await _resource_problems()) == {}
|
||||
|
||||
|
||||
# ── Services ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _service_problems():
|
||||
async with SessionLocal() as db:
|
||||
return await monitor_checks.check_service_down(db)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_down_only_after_it_was_seen_running(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
up = {"services": {"unbound": 1, "dhcpd6": 0, "cron": 0}}
|
||||
down = {"services": {"unbound": 0, "dhcpd6": 0, "cron": 0}}
|
||||
await _add_samples(hid, [down, down, down, up, up])
|
||||
problems = await _service_problems()
|
||||
# unbound stopped; dhcpd6 never ran; cron is not on the watchlist
|
||||
assert list(problems) == [f"{hid}:unbound"]
|
||||
assert problems[f"{hid}:unbound"]["dienst"] == "unbound"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_that_never_ran_or_stopped_long_ago_is_not_reported(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"services": {"unbound": 0}}] * 6)
|
||||
assert dict(await _service_problems()) == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_alert_resolves_when_it_runs_again_or_leaves_the_watchlist(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
up, down = {"services": {"unbound": 1}}, {"services": {"unbound": 0}}
|
||||
await _only("service_down_enabled")
|
||||
await _add_samples(hid, [down, down, down, up])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1
|
||||
await _set(service_watchlist=["ntpd"])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["resolved"] == 1
|
||||
await _set(service_watchlist=["unbound"])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1 # reopens (the row is reused)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [up, up, up, down])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["resolved"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_watchlist_validation(client):
|
||||
ok = await client.put("/api/notifications/settings", json={"service_watchlist": [" unbound ", "openvpn", "unbound"]})
|
||||
assert ok.status_code == 200 and ok.json()["service_watchlist"] == ["unbound", "openvpn"]
|
||||
assert (await client.put("/api/notifications/settings", json={"service_watchlist": ["bad name!"]})).status_code == 422
|
||||
assert (await client.put("/api/notifications/settings", json={"disk_threshold_pct": 101})).status_code == 422
|
||||
assert (await client.put("/api/notifications/settings", json={"alert_confirm_samples": 0})).status_code == 422
|
||||
|
||||
|
||||
# ── Metrics and dashboard ─────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_metrics_and_dashboard_show_gateways_and_services(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test", cluster_name=None)
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 0.1, "swap_used_pct": 5.0,
|
||||
"gateways": [_gw(), _gw("offline", 100.0, None, name="DOWN_GW")],
|
||||
"services": {"unbound": 1, "openvpn/3f2a": 0, "cron": 0}}])
|
||||
text = (await client.get("/api/metrics")).text
|
||||
assert 'opnexus_host_gateway_up{host="fw1",cluster="",gateway="WAN_GW"} 1' in text
|
||||
assert 'opnexus_host_gateway_up{host="fw1",cluster="",gateway="DOWN_GW"} 0' in text
|
||||
assert 'opnexus_host_gateway_delay_seconds{host="fw1",cluster="",gateway="WAN_GW"} 0.002' in text
|
||||
assert 'opnexus_host_gateway_loss_percent{host="fw1",cluster="",gateway="DOWN_GW"} 100' in text
|
||||
assert 'opnexus_host_swap_used_percent{host="fw1",cluster=""} 5' in text
|
||||
assert 'opnexus_host_service_running{host="fw1",cluster="",service="unbound"} 1' in text
|
||||
assert 'service="openvpn",instance="3f2a"} 0' in text
|
||||
assert 'service="cron"' not in text # not on the watchlist
|
||||
|
||||
summary = (await client.get("/api/dashboard")).json()
|
||||
gateways = {g["name"]: g for g in summary["hosts"][0]["gateways"]}
|
||||
assert gateways["DOWN_GW"]["state"] == "offline"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashboard_has_no_gateways_without_a_fresh_sample(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
await _add_samples(uuid.UUID(host["id"]), [{"gateways": [_gw()]}], age_minutes=20)
|
||||
assert (await client.get("/api/dashboard")).json()["hosts"][0]["gateways"] is None
|
||||
|
||||
|
||||
# ── URL guard ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("url", [
|
||||
"http://169.254.169.254/latest/meta-data", "http://127.0.0.1:8000/x", "http://localhost/x", "http://[::1]/x",
|
||||
"http://0.0.0.0/x", "http://[::ffff:169.254.169.254]/x", "ftp://ntfy.example/x", "file:///etc/passwd",
|
||||
"https://user:[email protected]/", "https://", "not a url", "http://224.0.0.1/x", "http://[fe80::1]/x",
|
||||
])
|
||||
def test_url_guard_refuses(url):
|
||||
with pytest.raises(ValueError):
|
||||
url_guard.validate_url(url)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url", ["https://ntfy.sh", "http://10.20.30.5:8080/path", "https://gotify.lan.example/", "http://192.168.1.10"])
|
||||
def test_url_guard_allows_private_and_public_targets(url):
|
||||
assert url_guard.validate_url(url) == url
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_url_guard_refuses_names_resolving_to_metadata_address(monkeypatch):
|
||||
async def fake(self, host, port, **kw):
|
||||
return [(2, 1, 6, "", ("169.254.169.254", 0))]
|
||||
monkeypatch.setattr("asyncio.base_events.BaseEventLoop.getaddrinfo", fake)
|
||||
with pytest.raises(ValueError):
|
||||
await url_guard.ensure_resolved_safe("https://innocent.example/hook")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_webhook_url_is_validated_when_saved_and_sent(client, respx_mock):
|
||||
resp = await client.put("/api/notifications/settings", json={"webhook_url": "http://169.254.169.254/x"})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "NOTIFICATION_WEBHOOK_URL_INVALID"
|
||||
row = NotificationSettings(webhook_url="http://127.0.0.1:9/x")
|
||||
from app.services import notification_delivery
|
||||
with pytest.raises(NotificationDeliveryError):
|
||||
await notification_delivery.send_webhook(row, NotificationProblem(problem_type="test", dedup_key="t", context={}), "opened")
|
||||
|
||||
|
||||
# ── Channels: formats ────────────────────────────────────────────────────────────
|
||||
|
||||
PROBLEM = NotificationProblem(problem_type="gateway_down", dedup_key="k", context={"host_name": "fw1", "gateway": "WAN_GW", "grund": "Gateway offline"})
|
||||
|
||||
|
||||
def _channel(kind: str, config: dict, secret: str | None, name: str = "c1") -> NotificationChannel:
|
||||
ch = NotificationChannel(id=uuid.uuid4(), name=name, channel_type=kind, enabled=True, config=config, secret_set=secret is not None)
|
||||
if secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(ch.id), secret)
|
||||
return ch
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ntfy_posts_json_to_root_with_bearer(respx_mock):
|
||||
ch = _channel("ntfy", {"server_url": "https://ntfy.test", "topic": "alerts", "priority": 5}, "tk_secret")
|
||||
route = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200, json={"id": "x"}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
req = route.calls.last.request
|
||||
assert req.headers["authorization"] == "Bearer tk_secret"
|
||||
body = httpx.Response(200, content=req.content).json()
|
||||
assert body["topic"] == "alerts" and body["priority"] == 5 and body["tags"] == ["warning"]
|
||||
assert body["title"] == "OPNexus: Gateway gestört" and "gateway: WAN_GW" in body["message"]
|
||||
await notification_channels.send(ch, PROBLEM, "resolved")
|
||||
resolved = httpx.Response(200, content=route.calls.last.request.content).json()
|
||||
assert resolved["priority"] == 3 and resolved["title"].endswith("behoben") and resolved["tags"] == ["white_check_mark"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ntfy_without_token_sends_no_auth_header(respx_mock):
|
||||
ch = _channel("ntfy", {"server_url": "https://ntfy.test", "topic": "alerts"}, None)
|
||||
route = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
assert "authorization" not in route.calls.last.request.headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gotify_uses_key_header_not_query(respx_mock):
|
||||
ch = _channel("gotify", {"server_url": "https://gotify.test/", "priority": 8}, "AppTok123")
|
||||
route = respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(200, json={"id": 1}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
req = route.calls.last.request
|
||||
assert req.headers["x-gotify-key"] == "AppTok123" and "AppTok123" not in str(req.url)
|
||||
body = httpx.Response(200, content=req.content).json()
|
||||
assert body["priority"] == 8 and body["title"] == "OPNexus: Gateway gestört"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_telegram_sendmessage_and_ok_flag(respx_mock):
|
||||
ch = _channel("telegram", {"chat_id": "-1001234", "api_base": "https://api.telegram.test"}, "123456:ABC-token")
|
||||
route = respx_mock.post("https://api.telegram.test/bot123456:ABC-token/sendMessage").mock(
|
||||
return_value=httpx.Response(200, json={"ok": True, "result": {}}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
body = httpx.Response(200, content=route.calls.last.request.content).json()
|
||||
assert body["chat_id"] == "-1001234" and body["text"].startswith("OPNexus: Gateway gestört\n\n") and len(body["text"]) <= 4096
|
||||
route.mock(return_value=httpx.Response(200, json={"ok": False, "description": "chat not found"}))
|
||||
with pytest.raises(NotificationDeliveryError):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
|
||||
|
||||
def test_telegram_text_is_clipped_to_the_api_limit():
|
||||
big = NotificationProblem(problem_type="gateway_down", dedup_key="k", context={"x": "y" * 9000})
|
||||
text = notification_channels.format_telegram({"chat_id": "1"}, big, "opened")["text"]
|
||||
assert len(text) == 4096 and text.endswith("…")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_teams_adaptive_card_envelope(respx_mock):
|
||||
url = "https://prod-12.westeurope.logic.azure.test/workflows/abc/triggers/manual/paths/invoke?sig=SECRETSIG"
|
||||
ch = _channel("teams", {}, url)
|
||||
route = respx_mock.post(url.split("?")[0]).mock(return_value=httpx.Response(202))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
body = httpx.Response(200, content=route.calls.last.request.content).json()
|
||||
assert body["type"] == "message"
|
||||
att = body["attachments"][0]
|
||||
assert att["contentType"] == "application/vnd.microsoft.card.adaptive" and att["content"]["type"] == "AdaptiveCard"
|
||||
assert att["content"]["body"][0]["text"] == "OPNexus: Gateway gestört"
|
||||
assert "sig=SECRETSIG" in str(route.calls.last.request.url)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_matrix_put_with_transaction_id_and_encoded_room(respx_mock):
|
||||
ch = _channel("matrix", {"homeserver_url": "https://matrix.test", "room_id": "!abc123:matrix.test"}, "syt_token")
|
||||
route = respx_mock.put(url__regex=r"^https://matrix\.test/_matrix/client/v3/rooms/%21abc123%3Amatrix\.test/send/m\.room\.message/[0-9a-f]{32}$").mock(
|
||||
return_value=httpx.Response(200, json={"event_id": "$e"}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
first, second = (c.request for c in route.calls)
|
||||
assert first.headers["authorization"] == "Bearer syt_token"
|
||||
assert first.url.path != second.url.path # a new transaction id per message
|
||||
body = httpx.Response(200, content=first.content).json()
|
||||
assert body["msgtype"] == "m.text" and body["body"].startswith("OPNexus: Gateway gestört")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failures_and_logs_never_contain_the_credential(respx_mock, caplog):
|
||||
token = "123456:SECRET-BOT-TOKEN"
|
||||
ch = _channel("telegram", {"chat_id": "42", "api_base": "https://api.telegram.test"}, token)
|
||||
respx_mock.post(f"https://api.telegram.test/bot{token}/sendMessage").mock(return_value=httpx.Response(401, json={"ok": False}))
|
||||
with caplog.at_level(logging.DEBUG):
|
||||
with pytest.raises(NotificationDeliveryError) as exc_info:
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
respx_mock.post(f"https://api.telegram.test/bot{token}/sendMessage").mock(side_effect=httpx.ConnectError("boom"))
|
||||
with pytest.raises(NotificationDeliveryError) as exc_info2:
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
assert "HTTP 401" in str(exc_info.value) and "ConnectError" in str(exc_info2.value)
|
||||
for text in (str(exc_info.value), str(exc_info2.value), caplog.text):
|
||||
assert "SECRET-BOT-TOKEN" not in text
|
||||
assert exc_info.value.__cause__ is None and exc_info.value.__suppress_context__
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_or_unreadable_secret_fails_closed(respx_mock):
|
||||
ch = _channel("gotify", {"server_url": "https://gotify.test"}, "tok")
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(ch.id))
|
||||
with pytest.raises(NotificationDeliveryError, match="nicht lesbar"):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
ch.secret_set = False
|
||||
with pytest.raises(NotificationDeliveryError, match="fehlen"):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redirects_are_not_followed(respx_mock):
|
||||
ch = _channel("gotify", {"server_url": "https://gotify.test"}, "tok")
|
||||
respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(302, headers={"location": "http://169.254.169.254/"}))
|
||||
with pytest.raises(NotificationDeliveryError, match="HTTP 302"):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
|
||||
|
||||
# ── Channels: API ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_crud_keeps_the_secret_out_of_database_and_responses(client, respx_mock):
|
||||
resp = await client.post("/api/notifications/channels", json={
|
||||
"name": "Telegram Ops", "channel_type": "telegram", "config": {"chat_id": "-100123"}, "secret": "111:TOKEN-VALUE",
|
||||
})
|
||||
assert resp.status_code == 201, resp.text
|
||||
created = resp.json()
|
||||
assert created["secret_set"] is True and "TOKEN-VALUE" not in resp.text
|
||||
assert created["config"] == {"chat_id": "-100123", "api_base": "https://api.telegram.org"}
|
||||
cid = created["id"]
|
||||
assert notification_secret_store.read_secret(f"channel_{cid}") == "111:TOKEN-VALUE"
|
||||
async with SessionLocal() as db:
|
||||
row = await db.get(NotificationChannel, uuid.UUID(cid))
|
||||
assert "TOKEN-VALUE" not in str(row.config) and "TOKEN-VALUE" not in repr(row.__dict__)
|
||||
assert "TOKEN-VALUE" not in (await client.get("/api/notifications/channels")).text
|
||||
|
||||
# test button
|
||||
route = respx_mock.post("https://api.telegram.org/bot111:TOKEN-VALUE/sendMessage").mock(return_value=httpx.Response(200, json={"ok": True}))
|
||||
assert (await client.post(f"/api/notifications/channels/{cid}/test")).json() == {"success": True}
|
||||
assert b"Testbenachrichtigung" in route.calls.last.request.content
|
||||
route.mock(return_value=httpx.Response(500))
|
||||
failed = await client.post(f"/api/notifications/channels/{cid}/test")
|
||||
assert failed.status_code == 502 and failed.json()["detail"]["code"] == "NOTIFICATION_CHANNEL_TEST_FAILED"
|
||||
assert "TOKEN-VALUE" not in failed.text
|
||||
|
||||
# update: rename, disable, rotate secret, clear secret
|
||||
upd = await client.put(f"/api/notifications/channels/{cid}", json={"name": "Telegram", "enabled": False, "secret": "222:NEW"})
|
||||
assert upd.status_code == 200 and upd.json()["enabled"] is False
|
||||
assert notification_secret_store.read_secret(f"channel_{cid}") == "222:NEW"
|
||||
assert (await client.put(f"/api/notifications/channels/{cid}", json={"clear_secret": True})).json()["secret_set"] is False
|
||||
assert notification_secret_store.read_secret(f"channel_{cid}") is None
|
||||
|
||||
assert (await client.delete(f"/api/notifications/channels/{cid}")).status_code == 204
|
||||
assert (await client.get("/api/notifications/channels")).json() == []
|
||||
assert (await client.delete(f"/api/notifications/channels/{cid}")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [
|
||||
{"name": "a", "channel_type": "ntfy", "config": {"topic": "bad topic!"}},
|
||||
{"name": "a", "channel_type": "ntfy", "config": {"topic": "ok", "server_url": "http://169.254.169.254"}},
|
||||
{"name": "a", "channel_type": "ntfy", "config": {"topic": "ok", "priority": 9}},
|
||||
{"name": "a", "channel_type": "gotify", "config": {"server_url": "https://g.test"}}, # token missing
|
||||
{"name": "a", "channel_type": "gotify", "config": {}, "secret": "t"}, # server missing
|
||||
{"name": "a", "channel_type": "telegram", "config": {"chat_id": "abc"}, "secret": "t"},
|
||||
{"name": "a", "channel_type": "teams", "config": {}, "secret": "http://insecure.test/hook"},
|
||||
{"name": "a", "channel_type": "teams", "config": {}},
|
||||
{"name": "a", "channel_type": "matrix", "config": {"homeserver_url": "https://m.test", "room_id": "#alias:m.test"}, "secret": "t"},
|
||||
{"name": "a", "channel_type": "carrier-pigeon", "config": {}},
|
||||
{"name": " ", "channel_type": "ntfy", "config": {"topic": "ok"}}, # name only blanks
|
||||
])
|
||||
async def test_channel_validation(client, payload):
|
||||
resp = await client.post("/api/notifications/channels", json=payload)
|
||||
assert resp.status_code == 422, resp.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_names_are_unique_and_ntfy_needs_no_secret(client):
|
||||
body = {"name": "Phone", "channel_type": "ntfy", "config": {"topic": "opnexus-alerts"}}
|
||||
first = await client.post("/api/notifications/channels", json=body)
|
||||
assert first.status_code == 201 and first.json()["secret_set"] is False
|
||||
assert first.json()["config"]["server_url"] == "https://ntfy.sh" and first.json()["config"]["priority"] == 4
|
||||
dup = await client.post("/api/notifications/channels", json=body)
|
||||
assert dup.status_code == 409 and dup.json()["detail"]["code"] == "NOTIFICATION_CHANNEL_NAME_TAKEN"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_operator_cannot_change_channels_and_does_not_see_their_settings(client):
|
||||
from tests.test_operator_role import PW, _operator
|
||||
created = await client.post("/api/notifications/channels", json={"name": "Phone", "channel_type": "ntfy", "config": {"topic": "secret-topic"}})
|
||||
op = await _operator(client, "chan-op")
|
||||
try:
|
||||
for method, path, body in [
|
||||
("POST", "/api/notifications/channels", {"name": "x", "channel_type": "ntfy", "config": {"topic": "t"}}),
|
||||
("PUT", f"/api/notifications/channels/{created.json()['id']}", {"enabled": False}),
|
||||
("DELETE", f"/api/notifications/channels/{created.json()['id']}", None),
|
||||
("POST", f"/api/notifications/channels/{created.json()['id']}/test", {}),
|
||||
]:
|
||||
resp = await op.request(method, path, json=body) if body is not None else await op.request(method, path)
|
||||
assert resp.status_code == 403, (method, path, resp.text)
|
||||
listing = await op.get("/api/notifications/channels")
|
||||
assert listing.status_code == 200 and listing.json()[0]["config"] == {} and "secret-topic" not in listing.text
|
||||
finally:
|
||||
await op.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alerts_reach_every_enabled_channel_and_one_failure_does_not_block_the_rest(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
async with SessionLocal() as db:
|
||||
for kind, cfg, secret, name in [
|
||||
("gotify", {"server_url": "https://gotify.test"}, "g-tok", "A gotify"),
|
||||
("ntfy", {"server_url": "https://ntfy.test", "topic": "t"}, None, "B ntfy"),
|
||||
]:
|
||||
ch = _channel(kind, cfg, secret, name)
|
||||
db.add(ch)
|
||||
off = _channel("ntfy", {"server_url": "https://off.test", "topic": "t"}, None, "C disabled")
|
||||
off.enabled = False
|
||||
db.add(off)
|
||||
await db.commit()
|
||||
await _only("gateway_down_enabled")
|
||||
gotify = respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(500))
|
||||
ntfy = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200))
|
||||
await _add_samples(hid, [{"gateways": [_gw("offline", 100.0, None)]}] * 3)
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1
|
||||
assert gotify.call_count == 1 and ntfy.call_count == 1
|
||||
assert "off.test" not in str([c.request.url for c in respx_mock.calls])
|
||||
@@ -0,0 +1,452 @@
|
||||
"""Taking over aliases, port forwards, 1:1 NAT and outbound NAT that were created by hand on a firewall.
|
||||
|
||||
The live rows below have the shape recorded from OPNsense 26.7 (test-fw-3, 2026-10-20), see the module
|
||||
docstring of app/services/object_adopt_service.py."""
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from tests.helpers import deploy_approved
|
||||
from tests.test_api_tokens import _login
|
||||
|
||||
BASE = "https://fw-adopt-obj.test"
|
||||
API = f"{BASE}/api/firewall"
|
||||
|
||||
|
||||
async def _host(client, respx_mock, name="fw-obj"):
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
r = await client.post("/api/hosts", json={"name": name, "base_url": BASE, "api_key": "k", "api_secret": "s"})
|
||||
assert r.status_code == 201, r.text
|
||||
return r.json()["id"]
|
||||
|
||||
|
||||
def _search(respx_mock, path, rows):
|
||||
return respx_mock.get(f"{API}/{path}").mock(return_value=httpx.Response(200, json={"rows": rows, "total": len(rows)}))
|
||||
|
||||
|
||||
def _mode(respx_mock, mode="hybrid"):
|
||||
respx_mock.get(f"{API}/source_nat/get").mock(return_value=httpx.Response(
|
||||
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == mode else 0} for m in
|
||||
("automatic", "hybrid", "advanced", "disabled")}}}}))
|
||||
|
||||
|
||||
def _writes(respx_mock):
|
||||
return [c for c in respx_mock.calls if c.request.method != "GET"]
|
||||
|
||||
|
||||
# ── live rows (shapes captured from OPNsense 26.7) ──────────────────────
|
||||
|
||||
def alias_row(name, type_="host", content="10.9.9.1\n10.9.9.2", **over):
|
||||
row = {"uuid": f"uuid-{name}", "enabled": "1", "name": name, "type": type_, "%type": type_.title(),
|
||||
"content": content, "description": f"{name} desc", "proto": "", "interface": "", "counters": "0",
|
||||
"updatefreq": "", "categories": ""}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
SYSTEM_ALIASES = [
|
||||
alias_row("bogons", "external", "", description="bogon networks (internal)"),
|
||||
alias_row("sshlockout", "external", "", description="abuse lockout table (internal)"),
|
||||
alias_row("__lan_network", "internal", "", description="lan net"),
|
||||
]
|
||||
|
||||
|
||||
def pf_row(**over):
|
||||
row = {"uuid": "pf-1", "ipprotocol": "inet", "protocol": "tcp", "%protocol": "TCP", "disabled": "0", "nordr": "0",
|
||||
"interface": "wan", "%interface": "WAN", "source.network": "any", "source.port": "", "source.not": "0",
|
||||
"destination.network": "wanip", "destination.port": "8443", "destination.not": "0", "target": "192.168.1.50",
|
||||
"local-port": "443", "natreflection": "", "descr": "Web", "pass": "", "log": "0", "poolopts": "", "tag": "",
|
||||
"tagged": "", "nosync": "0", "categories": "", "is_automatic": False}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
def o2o_row(**over):
|
||||
row = {"uuid": "o2o-1", "enabled": "1", "interface": "wan", "%interface": "WAN", "type": "binat",
|
||||
"source_net": "192.168.1.10", "source_not": "0", "destination_net": "any", "destination_not": "0",
|
||||
"external": "203.0.113.201", "natreflection": "", "log": "0", "description": "Mail 1:1", "categories": ""}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
def snat_row(**over):
|
||||
row = {"uuid": "snat-1", "enabled": "1", "interface": "wan", "%interface": "WAN", "ipprotocol": "inet",
|
||||
"protocol": "any", "source_net": "192.168.1.0/24", "source_not": "0", "source_port": "",
|
||||
"destination_net": "any", "destination_not": "0", "destination_port": "", "target": "", "target_port": "",
|
||||
"nonat": "0", "nosync": "0", "staticnatport": "0", "log": "0", "tag": "", "tagged": "",
|
||||
"description": "LAN out", "categories": ""}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
AUTOMATIC_NAT = [
|
||||
{"uuid": "lockout_0", "interface": "lan", "descr": "Anti-Lockout Rule", "is_automatic": True},
|
||||
{"uuid": "automatic_wan", "interface": "wan", "description": "Auto created rule", "is_automatic": True},
|
||||
]
|
||||
|
||||
# kind -> (url prefix, search path, row factory, id field of the preview)
|
||||
NAT_KINDS = {
|
||||
"port-forward": ("d_nat/searchRule", pf_row, "Web"),
|
||||
"one-to-one": ("one_to_one/searchRule", o2o_row, "Mail 1:1"),
|
||||
"outbound-nat": ("source_nat/searchRule", snat_row, "LAN out"),
|
||||
}
|
||||
|
||||
|
||||
# ── aliases ─────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_candidates_mark_system_unsupported_and_nested(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
|
||||
alias_row("ok_hosts"),
|
||||
alias_row("ok_nets", "network", "10.0.0.0/8\n192.168.0.0/16"),
|
||||
alias_row("ok_ports", "port", "8080\n9000:9010"),
|
||||
alias_row("ok_group", "networkgroup", "ok_nets\n__lan_network"),
|
||||
alias_row("a_url", "url", "https://example.org/list.txt", updatefreq="1"),
|
||||
alias_row("geo", "geoip", "DE", proto="IPv4"),
|
||||
alias_row("uses_url", "host", "a_url"),
|
||||
alias_row("uses_hosts", "host", "ok_hosts\n10.9.9.3"),
|
||||
alias_row("asn_v4", "asn", "64496", proto="IPv4"),
|
||||
alias_row("bad name", "host", "10.1.1.1"),
|
||||
alias_row("off", "host", "10.9.9.9", enabled="0"),
|
||||
])
|
||||
resp = await client.get(f"/api/aliases/unmanaged?host_id={host_id}")
|
||||
assert resp.status_code == 200, resp.text
|
||||
by = {r["title"]: r for r in resp.json()}
|
||||
|
||||
for name in ("bogons", "sshlockout", "__lan_network"):
|
||||
assert by[name]["reasons"] == ["system"] and not by[name]["adoptable"]
|
||||
for name in ("ok_hosts", "ok_nets", "ok_ports", "off"):
|
||||
assert by[name]["adoptable"] is True and by[name]["reasons"] == []
|
||||
assert by["off"]["enabled"] is False
|
||||
# __lan_network is OPNsense's own and fine; ok_nets is a live alias that is not managed yet
|
||||
assert by["ok_group"]["reasons"] == ["references_unmanaged_alias"] and by["ok_group"]["bulk_adoptable"] is True
|
||||
assert by["ok_group"]["reason_details"]["references_unmanaged_alias"] == "ok_nets"
|
||||
assert by["a_url"]["reasons"] == ["unsupported_type"] and by["geo"]["reasons"] == ["unsupported_type"]
|
||||
assert by["a_url"]["reason_details"]["unsupported_type"]
|
||||
assert by["uses_url"]["reasons"] == ["references_unmanaged_alias"]
|
||||
assert by["uses_url"]["reason_details"]["references_unmanaged_alias"] == "a_url"
|
||||
assert by["uses_url"]["bulk_adoptable"] is False # its target can never be adopted
|
||||
assert by["asn_v4"]["reasons"] == ["proto_set"]
|
||||
assert by["bad name"]["reasons"] == ["name_invalid"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_nested_alias_waits_for_its_target_but_bulk_orders_them(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", [
|
||||
alias_row("outer", "host", "inner\n10.0.0.9"),
|
||||
alias_row("inner", "host", "base"),
|
||||
alias_row("base", "host", "10.0.0.1"),
|
||||
])
|
||||
by = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()}
|
||||
assert by["base"]["adoptable"] is True
|
||||
for name, target in (("inner", "base"), ("outer", "inner")):
|
||||
assert by[name]["adoptable"] is False and by[name]["bulk_adoptable"] is True
|
||||
assert by[name]["reason_details"]["references_unmanaged_alias"] == target
|
||||
# alone it is refused ...
|
||||
refused = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-outer"})
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "OBJECT_NOT_ADOPTABLE"
|
||||
assert "references_unmanaged_alias" in refused.json()["detail"]["params"]["reasons"]
|
||||
# ... after its target is managed it works
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-base"})).status_code == 201
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-inner"})).status_code == 201
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-outer"})).status_code == 201
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_alias_records_it_without_writing_and_preview_is_unchanged(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
|
||||
alias_row("web_servers", "host", "10.0.0.1\n 10.0.0.2 \n\nexample.org", description="Web tier"),
|
||||
])
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})
|
||||
assert resp.status_code == 201, resp.text
|
||||
alias = resp.json()
|
||||
assert (alias["name"], alias["type"], alias["description"], alias["enabled"]) == ("web_servers", "host", "Web tier", True)
|
||||
assert alias["content"] == "10.0.0.1\n10.0.0.2\nexample.org"
|
||||
assert alias["target_host_ids"] == [host_id]
|
||||
assert [d["status"] for d in alias["deployments"]] == ["success"]
|
||||
assert alias["workflow_state"] == "synced"
|
||||
assert not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
|
||||
|
||||
preview = (await client.get(f"/api/aliases/{alias['id']}/preview")).json()
|
||||
assert [(h["action"], h["changes"]) for h in preview["hosts"]] == [("unchanged", [])]
|
||||
assert preview["can_deploy"] is False
|
||||
# no longer offered; adopting again is refused
|
||||
assert [r["title"] for r in (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()
|
||||
if r["reasons"] != ["system"]] == []
|
||||
again = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})
|
||||
assert again.status_code == 409 and "already_managed" in again.json()["detail"]["params"]["reasons"]
|
||||
|
||||
# audit: the deployment record (kind alias, remote_uuid) is written with the adoption
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "alias"))).all()
|
||||
assert any(e.entity_name == "web_servers" and e.details["remote_uuid"] == "uuid-web_servers"
|
||||
and e.message == "Von der Firewall übernommen" for e in events)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_edit_after_alias_adopt_updates_the_same_remote_object(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", [alias_row("web_servers", "host", "10.0.0.1")])
|
||||
alias = (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})).json()
|
||||
edit = {"name": "web_servers", "type": "host", "content": "10.0.0.1\n10.0.0.2", "description": "web_servers desc",
|
||||
"enabled": True, "target_host_ids": [host_id], "target_group_ids": [], "expected_updated_at": alias["updated_at"]}
|
||||
assert (await client.put(f"/api/aliases/{alias['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(f"{API}/alias/addItem").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
setter = respx_mock.post(f"{API}/alias/setItem/uuid-web_servers").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(f"{API}/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
resp = await deploy_approved(client, f"/api/aliases/{alias['id']}/deploy")
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert setter.called and not add.called # no duplicate
|
||||
assert json.loads(setter.calls.last.request.content)["alias"]["content"] == "10.0.0.1\n10.0.0.2"
|
||||
confirm = await client.post(f"/api/aliases/{alias['id']}/confirm")
|
||||
assert [d["status"] for d in confirm.json()["deployments"]] == ["success"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_name_that_exists_centrally_is_refused(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
assert (await client.post("/api/aliases", json={"name": "dup", "type": "host", "content": "1.1.1.1"})).status_code == 201
|
||||
_search(respx_mock, "alias/searchItem", [alias_row("dup", "host", "2.2.2.2")])
|
||||
row = (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()[0]
|
||||
assert row["reasons"] == ["name_exists"]
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-dup"})).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_all_aliases_orders_dependencies_and_reports_skipped(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
|
||||
alias_row("outer", "host", "inner"),
|
||||
alias_row("inner", "host", "10.0.0.1"),
|
||||
alias_row("a_url", "url", "https://example.org/x"),
|
||||
alias_row("uses_url", "host", "a_url"),
|
||||
alias_row("plain", "port", "443"),
|
||||
])
|
||||
resp = await client.post("/api/aliases/adopt-all", json={"host_id": host_id})
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert sorted(a["title"] for a in body["adopted"]) == ["inner", "outer", "plain"]
|
||||
skipped = {s["title"]: s for s in body["skipped"]}
|
||||
assert set(skipped) == {"a_url", "uses_url"} # system aliases are not reported
|
||||
assert skipped["a_url"]["reasons"] == ["unsupported_type"]
|
||||
assert skipped["uses_url"]["reasons"] == ["references_unmanaged_alias"]
|
||||
aliases = {a["name"]: a for a in (await client.get("/api/aliases")).json()}
|
||||
assert set(aliases) == {"inner", "outer", "plain"}
|
||||
assert all([d["status"] for d in a["deployments"]] == ["success"] and a["workflow_state"] == "synced" for a in aliases.values())
|
||||
# a second run finds nothing left to take over
|
||||
again = (await client.post("/api/aliases/adopt-all", json={"host_id": host_id})).json()
|
||||
assert again["adopted"] == [] and {s["title"] for s in again["skipped"]} == {"a_url", "uses_url"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_unknown_alias_is_404(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", [alias_row("x")])
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "nope"})
|
||||
assert resp.status_code == 404 and resp.json()["detail"]["code"] == "ADOPT_OBJECT_NOT_FOUND"
|
||||
|
||||
|
||||
# ── NAT ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automatic_nat_rows_are_never_offered(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock)
|
||||
for prefix, (path, factory, _) in NAT_KINDS.items():
|
||||
_search(respx_mock, path, [factory(), *AUTOMATIC_NAT])
|
||||
rows = (await client.get(f"/api/{prefix}/unmanaged?host_id={host_id}")).json()
|
||||
assert [r["uuid"] for r in rows] == [factory()["uuid"]], prefix
|
||||
assert rows[0]["adoptable"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_port_forward_refusals(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
cases = {
|
||||
"multi": (dict(interface="wan,lan"), "interface_not_single"),
|
||||
"v46": (dict(ipprotocol="inet46"), "ipprotocol_unsupported"),
|
||||
"anyproto": (dict(protocol=""), "protocol_any"),
|
||||
"nordr": (dict(nordr="1", target="", **{"local-port": ""}), "no_redirect"),
|
||||
"notarget": (dict(target=""), "no_target"),
|
||||
"linked": (dict(**{"pass": "pass"}), "filter_rule_linked"),
|
||||
"srcnot": (dict(**{"source.not": "1"}), "negated_source"),
|
||||
"dstnot": (dict(**{"destination.not": "1"}), "negated_destination"),
|
||||
"refl": (dict(natreflection="enable"), "natreflection_unsupported"),
|
||||
"nodesc": (dict(descr=""), "no_description"),
|
||||
"long": (dict(target="x" * 200), "value_too_long"),
|
||||
}
|
||||
_search(respx_mock, "d_nat/searchRule", [pf_row(**{"uuid": k, "descr": f"rule {k}", **over})
|
||||
for k, (over, _) in cases.items()] + [pf_row(uuid="fine")])
|
||||
rows = {r["uuid"]: r for r in (await client.get(f"/api/port-forward/unmanaged?host_id={host_id}")).json()}
|
||||
assert rows["fine"]["adoptable"] is True
|
||||
for key, (_, reason) in cases.items():
|
||||
assert reason in rows[key]["reasons"] and not rows[key]["adoptable"], key
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_port_forward_adopt_maps_fields_and_keeps_unmanaged_ones(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "d_nat/searchRule", [
|
||||
pf_row(disabled="1", natreflection="purenat", log="1", poolopts="random", tag="tg1", **{"source.network": "10.0.0.0/8"}),
|
||||
pf_row(uuid="pf-2", descr="Alias based", target="web_servers", **{"destination.port": "web_ports", "local-port": ""}),
|
||||
*AUTOMATIC_NAT,
|
||||
])
|
||||
candidates = {r["uuid"]: r for r in (await client.get(f"/api/port-forward/unmanaged?host_id={host_id}")).json()}
|
||||
assert sorted(candidates["pf-1"]["kept"]) == ["log", "poolopts", "tag"]
|
||||
resp = await client.post("/api/port-forward/adopt", json={"host_id": host_id, "uuid": "pf-1"})
|
||||
assert resp.status_code == 201, resp.text
|
||||
rule = resp.json()
|
||||
assert (rule["description"], rule["interface"], rule["ipprotocol"], rule["protocol"]) == ("Web", "wan", "inet", "tcp")
|
||||
assert (rule["source_net"], rule["destination_net"], rule["destination_port"]) == ("10.0.0.0/8", "wanip", "8443")
|
||||
assert (rule["target"], rule["local_port"], rule["natreflection"], rule["enabled"]) == ("192.168.1.50", "443", "purenat", False)
|
||||
assert rule["workflow_state"] == "synced" and [d["status"] for d in rule["deployments"]] == ["success"]
|
||||
preview = (await client.get(f"/api/port-forward/{rule['id']}/preview")).json()
|
||||
assert [h["action"] for h in preview["hosts"]] == ["unchanged"] and preview["can_deploy"] is False
|
||||
assert not _writes(respx_mock) or not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
|
||||
|
||||
# edit -> deploy updates the same remote rule and never sends the fields it does not manage
|
||||
edit = {k: rule[k] for k in ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "target", "local_port", "natreflection", "enabled")}
|
||||
edit.update(target="192.168.1.51", target_host_ids=[host_id], target_group_ids=[], expected_updated_at=rule["updated_at"])
|
||||
assert (await client.put(f"/api/port-forward/{rule['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(f"{API}/d_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
setter = respx_mock.post(f"{API}/d_nat/setRule/pf-1").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(f"{API}/d_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert (await deploy_approved(client, f"/api/port-forward/{rule['id']}/deploy")).status_code == 200
|
||||
assert setter.called and not add.called
|
||||
sent = json.loads(setter.calls.last.request.content)["rule"]
|
||||
assert sent["target"] == "192.168.1.51" and sent["disabled"] == "1"
|
||||
assert not {"log", "poolopts", "tag", "tagged", "nosync", "categories", "nordr"} & set(sent)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_one_to_one_adopt_and_refusals(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "one_to_one/searchRule", [
|
||||
o2o_row(),
|
||||
o2o_row(uuid="o2", description="NAT off", type="nat", enabled="0", natreflection="enable", log="1",
|
||||
source_not="1", destination_net="10.0.0.0/8", destination_not="1"),
|
||||
o2o_row(uuid="o3", interface="wan,lan", description="multi"),
|
||||
o2o_row(uuid="o4", description=""),
|
||||
o2o_row(uuid="o5", description="weird", type="other"),
|
||||
o2o_row(uuid="o6", description="refl", natreflection="purenat"),
|
||||
*AUTOMATIC_NAT,
|
||||
])
|
||||
rows = {r["uuid"]: r for r in (await client.get(f"/api/one-to-one/unmanaged?host_id={host_id}")).json()}
|
||||
assert rows["o2o-1"]["adoptable"] and rows["o2"]["adoptable"]
|
||||
assert rows["o3"]["reasons"] == ["interface_not_single"] and rows["o4"]["reasons"] == ["no_description"]
|
||||
assert rows["o5"]["reasons"] == ["type_unsupported"] and rows["o6"]["reasons"] == ["natreflection_unsupported"]
|
||||
|
||||
rule = (await client.post("/api/one-to-one/adopt", json={"host_id": host_id, "uuid": "o2"})).json()
|
||||
assert (rule["type"], rule["enabled"], rule["natreflection"], rule["log"], rule["source_not"], rule["destination_not"]) == (
|
||||
"nat", False, "enable", True, True, True)
|
||||
assert rule["destination_net"] == "10.0.0.0/8" and rule["external"] == "203.0.113.201"
|
||||
preview = (await client.get(f"/api/one-to-one/{rule['id']}/preview")).json()
|
||||
assert [h["action"] for h in preview["hosts"]] == ["unchanged"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_outbound_nat_adopt_and_refusals(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock, "hybrid")
|
||||
_search(respx_mock, "source_nat/searchRule", [
|
||||
snat_row(),
|
||||
snat_row(uuid="s2", description="to pool", protocol="TCP", destination_port="443", target="203.0.113.205",
|
||||
target_port="1024", staticnatport="1", log="1", enabled="0", destination_not="1", destination_net="10.0.0.0/8"),
|
||||
snat_row(uuid="s3", description="no nat", nonat="1"),
|
||||
snat_row(uuid="s4", description="v46", ipprotocol="inet46"),
|
||||
snat_row(uuid="s5", description=""),
|
||||
*AUTOMATIC_NAT,
|
||||
])
|
||||
rows = {r["uuid"]: r for r in (await client.get(f"/api/outbound-nat/unmanaged?host_id={host_id}")).json()}
|
||||
assert rows["snat-1"]["adoptable"] and rows["s2"]["adoptable"]
|
||||
assert sorted(rows["s2"]["kept"]) == ["log", "staticnatport"]
|
||||
assert rows["s3"]["reasons"] == ["no_nat_rule"] and rows["s4"]["reasons"] == ["ipprotocol_unsupported"]
|
||||
assert rows["s5"]["reasons"] == ["no_description"]
|
||||
|
||||
rule = (await client.post("/api/outbound-nat/adopt", json={"host_id": host_id, "uuid": "s2"})).json()
|
||||
assert (rule["protocol"], rule["destination_port"], rule["target"], rule["target_port"], rule["enabled"],
|
||||
rule["destination_not"]) == ("TCP", "443", "203.0.113.205", "1024", False, True)
|
||||
assert rule["source_port"] is None
|
||||
preview = (await client.get(f"/api/outbound-nat/{rule['id']}/preview")).json()
|
||||
assert [h["action"] for h in preview["hosts"]] == ["unchanged"]
|
||||
|
||||
# edit -> deploy: setRule on the same uuid, nonat/log/staticnatport are not part of the payload
|
||||
edit = {k: rule[k] for k in ("description", "interface", "source_net", "destination_net", "source_not", "destination_not",
|
||||
"enabled", "ipprotocol", "protocol", "source_port", "destination_port", "target", "target_port")}
|
||||
edit.update(enabled=True, target_host_ids=[host_id], target_group_ids=[], expected_updated_at=rule["updated_at"])
|
||||
assert (await client.put(f"/api/outbound-nat/{rule['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(f"{API}/source_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
setter = respx_mock.post(f"{API}/source_nat/setRule/s2").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(f"{API}/source_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert (await deploy_approved(client, f"/api/outbound-nat/{rule['id']}/deploy")).status_code == 200
|
||||
assert setter.called and not add.called
|
||||
sent = json.loads(setter.calls.last.request.content)["rule"]
|
||||
assert sent["enabled"] == "1" and not {"nonat", "log", "staticnatport", "nosync", "tag"} & set(sent)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("mode", ["automatic", "disabled"])
|
||||
async def test_outbound_nat_adoption_needs_a_manual_capable_mode(client, respx_mock, mode):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock, mode)
|
||||
search = _search(respx_mock, "source_nat/searchRule", [snat_row()])
|
||||
for call in (client.get(f"/api/outbound-nat/unmanaged?host_id={host_id}"),
|
||||
client.post("/api/outbound-nat/adopt", json={"host_id": host_id, "uuid": "snat-1"}),
|
||||
client.post("/api/outbound-nat/adopt-all", json={"host_id": host_id})):
|
||||
resp = await call
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "OUTBOUND_MODE_BLOCKS_ADOPT", resp.text
|
||||
assert resp.json()["detail"]["params"]["mode"] == mode
|
||||
assert not search.called
|
||||
assert (await client.get("/api/outbound-nat")).json() == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("prefix", list(NAT_KINDS))
|
||||
async def test_adopt_all_nat_takes_what_is_adoptable_and_is_idempotent(client, respx_mock, prefix):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock)
|
||||
path, factory, title = NAT_KINDS[prefix]
|
||||
bad = {"descr": ""} if prefix == "port-forward" else {"description": ""}
|
||||
_search(respx_mock, path, [factory(), factory(uuid="second", **({"descr": "Second"} if prefix == "port-forward" else {"description": "Second"})),
|
||||
factory(uuid="bad", **bad), *AUTOMATIC_NAT])
|
||||
result = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": host_id})).json()
|
||||
assert sorted(a["title"] for a in result["adopted"]) == sorted([title, "Second"])
|
||||
assert [(s["uuid"], s["reasons"]) for s in result["skipped"]] == [("bad", ["no_description"])]
|
||||
listed = (await client.get(f"/api/{prefix}")).json()
|
||||
assert len(listed) == 2 and all(r["workflow_state"] == "synced" for r in listed)
|
||||
again = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": host_id})).json()
|
||||
assert again["adopted"] == [] and len((await client.get(f"/api/{prefix}")).json()) == 2
|
||||
assert not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("prefix", ["aliases", *NAT_KINDS])
|
||||
async def test_roles_and_unknown_host(client, respx_mock, prefix):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock)
|
||||
path = "alias/searchItem" if prefix == "aliases" else NAT_KINDS[prefix][0]
|
||||
row = alias_row("a1") if prefix == "aliases" else NAT_KINDS[prefix][1]()
|
||||
_search(respx_mock, path, [row])
|
||||
unknown = "00000000-0000-0000-0000-000000000000"
|
||||
assert (await client.get(f"/api/{prefix}/unmanaged?host_id={unknown}")).status_code == 404
|
||||
assert (await client.post(f"/api/{prefix}/adopt", json={"host_id": unknown, "uuid": "x"})).status_code == 404
|
||||
|
||||
# read-only users cannot adopt (it creates central objects), operators can (it writes nothing to a firewall)
|
||||
for role, expected in (("read_only", 403), ("operator", 201)):
|
||||
user = await client.post("/api/users", json={"username": f"u-{role}-{prefix}", "password": "adopt-test-password-1", "role": role})
|
||||
assert user.status_code == 201, user.text
|
||||
session = await _login(f"u-{role}-{prefix}", "adopt-test-password-1")
|
||||
try:
|
||||
assert (await session.get(f"/api/{prefix}/unmanaged?host_id={host_id}")).status_code == 200
|
||||
resp = await session.post(f"/api/{prefix}/adopt", json={"host_id": host_id, "uuid": row["uuid"]})
|
||||
assert resp.status_code == expected, (role, resp.text)
|
||||
finally:
|
||||
await session.aclose()
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
|
||||
import pytest
|
||||
|
||||
from tests.test_maintenance_windows import _WEEKLY
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
|
||||
async def _fleet(client):
|
||||
"""canary: c1 · pilot: p1 + ha-b (BACKUP) · prod: ha-a; ha-a/ha-b form cluster "ha"."""
|
||||
hosts = {}
|
||||
for name in ("c1", "p1", "ha-a", "ha-b"):
|
||||
hosts[name] = await _create_host(client, name, f"https://{name}.test")
|
||||
for name in ("ha-a", "ha-b"):
|
||||
await client.patch(f"/api/hosts/{hosts[name]['id']}", json={"cluster_name": "ha"})
|
||||
groups = {}
|
||||
for gname, members in (("ring-canary", ["c1"]), ("ring-pilot", ["p1", "ha-b"]), ("ring-prod", ["ha-a", "c1"])):
|
||||
resp = await client.post("/api/groups", json={"name": gname, "host_ids": [hosts[m]["id"] for m in members]})
|
||||
assert resp.status_code == 201, resp.text
|
||||
groups[gname] = resp.json()["id"]
|
||||
return hosts, groups
|
||||
|
||||
|
||||
def _plan_body(groups, **kw):
|
||||
body = {"name": "Ringe", "action": "update", "waves": [
|
||||
{"label": "canary", "group_ids": [groups["ring-canary"]], "soak_minutes": 10},
|
||||
{"label": "pilot", "group_ids": [groups["ring-pilot"]], "soak_minutes": 10},
|
||||
{"label": "prod", "group_ids": [groups["ring-prod"]], "soak_minutes": 0, "carp_handover": True, "require_approval": True},
|
||||
]}
|
||||
body.update(kw)
|
||||
return body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_plan_crud_and_preview_resolves_rings(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
resp = await client.post("/api/rollouts/plans", json=_plan_body(groups))
|
||||
assert resp.status_code == 201, resp.text
|
||||
plan = resp.json()
|
||||
assert [w["label"] for w in plan["waves"]] == ["canary", "pilot", "prod"] and plan["host_count"] == 4
|
||||
prev = await client.post(f"/api/rollouts/plans/{plan['id']}/preview")
|
||||
assert prev.status_code == 200
|
||||
body = prev.json()
|
||||
assert [[h["name"] for h in w["hosts"]] for w in body["waves"]] == [["c1"], ["ha-b", "p1"], ["ha-a"]] # c1 only once
|
||||
codes = {w["code"] for w in body["warnings"]}
|
||||
assert {"duplicate_hosts", "mixed_versions", "master_needs_handover", "no_window"} <= codes
|
||||
assert body["errors"] == [] and body["can_start"] and len(body["preview_token"]) == 64
|
||||
assert (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()["detail"]["code"] == "ROLLOUT_NAME_TAKEN"
|
||||
assert (await client.delete(f"/api/groups/{groups['ring-canary']}")).json()["detail"]["code"] == "ROLLOUT_GROUP_IN_USE"
|
||||
assert (await client.delete(f"/api/rollouts/plans/{plan['id']}")).status_code == 204
|
||||
assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_major_update_must_keep_a_cluster_in_one_wave(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, action="major_update"))).json()
|
||||
assert {e["code"] for e in plan["errors"]} == {"cluster_split"}
|
||||
prev = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()
|
||||
assert prev["can_start"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_activation_needs_auto_start_a_window_and_the_current_token(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
|
||||
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == "auto_start"
|
||||
|
||||
plan = (await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"]))).json()
|
||||
stale = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert stale.status_code == 409 and stale.json()["detail"]["code"] == "ROLLOUT_PREVIEW_STALE"
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
active = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert active.status_code == 200 and active.json()["active"] and active.json()["fingerprint_ok"]
|
||||
assert active.json()["activated_by"] == "admin"
|
||||
|
||||
# A group change behind the plan's back: still "active", but the fingerprint no longer matches.
|
||||
await client.put(f"/api/groups/{groups['ring-canary']}", json={"name": "ring-canary", "host_ids": []})
|
||||
assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).json()["fingerprint_ok"] is False
|
||||
# A window that a plan uses cannot be deleted (the plan would silently lose its time limit).
|
||||
assert (await client.delete(f"/api/maintenance-windows/{window['id']}")).json()["detail"]["code"] == "WINDOW_IN_USE"
|
||||
|
||||
# Any edit disarms it.
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
edited = await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"], name="Ringe 2"))
|
||||
assert edited.json()["active"] is False
|
||||
off = await client.post(f"/api/rollouts/plans/{plan['id']}/deactivate")
|
||||
assert off.status_code == 200 and off.json()["active"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_auto_start_only_for_minor_updates_with_a_window(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True))).json()
|
||||
assert "auto_start_needs_window" in {e["code"] for e in plan["errors"]}
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "ROLLOUT_PLAN_INVALID"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change,field", [
|
||||
({"gate_checks": {"nonsense": True}}, "gate_checks"),
|
||||
({"window_id": "00000000-0000-0000-0000-000000000000"}, "window_id"),
|
||||
])
|
||||
@pytest.mark.asyncio
|
||||
async def test_plan_validation(client, change, field):
|
||||
_hosts, groups = await _fleet(client)
|
||||
resp = await client.post("/api/rollouts/plans", json=_plan_body(groups, **change))
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == field
|
||||
bad_group = _plan_body(groups)
|
||||
bad_group["waves"][0]["group_ids"] = ["00000000-0000-0000-0000-000000000000"]
|
||||
assert (await client.post("/api/rollouts/plans", json=bad_group)).json()["detail"]["params"]["field"] == "group_ids"
|
||||
too_many = _plan_body(groups, waves=[{"label": str(i), "group_ids": [groups["ring-canary"]]} for i in range(11)])
|
||||
assert (await client.post("/api/rollouts/plans", json=too_many)).status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_role_matrix_for_plans(client):
|
||||
from tests.test_operator_role import _operator
|
||||
_hosts, groups = await _fleet(client)
|
||||
window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
|
||||
op = await _operator(client, "rollop")
|
||||
try:
|
||||
made = await op.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True, window_id=window["id"]))
|
||||
assert made.status_code == 201 # preparing is fine
|
||||
pid = made.json()["id"]
|
||||
token = (await op.post(f"/api/rollouts/plans/{pid}/preview")).json()["preview_token"]
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
denied = await op.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})
|
||||
assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
|
||||
assert (await op.post(f"/api/rollouts/plans/{pid}/deactivate")).status_code == 200 # braking is always allowed
|
||||
assert (await client.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})).status_code == 200
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
from tests.test_api_tokens import _login
|
||||
assert (await client.post("/api/users", json={"username": "rollro", "password": "readonly-pass-1", "role": "read_only"})).status_code == 201
|
||||
ro = await _login("rollro", "readonly-pass-1")
|
||||
try:
|
||||
assert (await ro.get("/api/rollouts/plans")).status_code == 200
|
||||
for path in ("/api/rollouts/plans", f"/api/rollouts/plans/{pid}/deactivate"):
|
||||
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
|
||||
finally:
|
||||
await ro.aclose()
|
||||
@@ -0,0 +1,620 @@
|
||||
"""The rollout engine end to end against a fake fleet: c1 (stand-alone), ha-a (MASTER) + ha-b (BACKUP)
|
||||
in cluster "ha". A fake clock drives timeouts, soak and re-checks; every tick runs exactly what the
|
||||
poll loop runs (fire due schedules, step running ones, step the rollout)."""
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.rollout import RolloutPlan, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import rollout_engine as engine
|
||||
from app.services import scheduled_maintenance_service as svc
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
|
||||
class Fw:
|
||||
def __init__(self, fleet, name, *, role=None, version="26.7.6", latest="26.7.7"):
|
||||
self.fleet, self.name, self.base = fleet, name, f"https://{name}.test"
|
||||
self.role, self.maint = role, False
|
||||
self.version, self.latest, self.offer = version, latest, "update"
|
||||
self.job, self.job_ticks, self.boot, self.up = "ready", 0, "boot-1", True
|
||||
self.update_error = False
|
||||
self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
|
||||
{"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
|
||||
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
|
||||
r, b = fleet.mock, self.base
|
||||
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
|
||||
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
|
||||
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
|
||||
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
|
||||
r.get(f"{b}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
|
||||
r.get(f"{b}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
|
||||
r.get(f"{b}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
|
||||
r.get(f"{b}/api/core/firmware/status").mock(side_effect=self._status)
|
||||
r.get(f"{b}/api/diagnostics/system/systemTime").mock(side_effect=lambda q: self._ok({"boottime": self.boot}))
|
||||
r.get(f"{b}/api/routes/gateway/status").mock(side_effect=lambda q: self._ok({"items": self.gateways, "status": "ok"}))
|
||||
r.get(f"{b}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
|
||||
|
||||
@property
|
||||
def down(self):
|
||||
return self.job == "down" or not self.up
|
||||
|
||||
def _ok(self, data):
|
||||
if self.down:
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, json=data)
|
||||
|
||||
def _raw(self, content):
|
||||
if self.down:
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, content=content)
|
||||
|
||||
def _status(self, request):
|
||||
if self.offer == "update":
|
||||
return self._ok({"status": "update", "status_msg": "x", "status_reboot": "1", "all_packages": {},
|
||||
"product": {"product_latest": self.latest}})
|
||||
return self._ok({"status": self.offer, "status_msg": "x", "status_reboot": "0", "all_packages": {}, "product": {}})
|
||||
|
||||
def _update(self, request):
|
||||
if self.update_error:
|
||||
return httpx.Response(500, json={"status": "error"})
|
||||
self.job, self.job_ticks = "running", 2
|
||||
return self._ok({"status": "ok"})
|
||||
|
||||
def _vip(self, request):
|
||||
rows = [{"vhid": "1", "status": self.role}] if self.role else []
|
||||
return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "240" if self.maint else "0",
|
||||
"status_msg": ""}})
|
||||
|
||||
def _toggle(self, request):
|
||||
self.maint = not self.maint
|
||||
partner = self.fleet.partner(self)
|
||||
if self.maint:
|
||||
self.role, partner.role = "BACKUP", "MASTER"
|
||||
else:
|
||||
self.role, partner.role = "MASTER", "BACKUP" # preempt: takes MASTER back
|
||||
return self._ok({"status": "ok", "action": "enter_maintenance" if self.maint else "leave_maintenance"})
|
||||
|
||||
def progress(self):
|
||||
if self.job == "running":
|
||||
self.job_ticks -= 1
|
||||
if self.job_ticks <= 0:
|
||||
self.job, self.job_ticks = "down", 2
|
||||
elif self.job == "down":
|
||||
self.job_ticks -= 1
|
||||
if self.job_ticks <= 0:
|
||||
self.job, self.version, self.offer, self.boot = "ready", self.latest, "none", self.boot + "+"
|
||||
|
||||
|
||||
class Fleet:
|
||||
def __init__(self, respx_mock):
|
||||
self.mock = respx_mock
|
||||
self.fw = {"c1": Fw(self, "c1"), "ha-a": Fw(self, "ha-a", role="MASTER"), "ha-b": Fw(self, "ha-b", role="BACKUP")}
|
||||
|
||||
def partner(self, fw):
|
||||
return self.fw["ha-b"] if fw.name == "ha-a" else self.fw["ha-a"]
|
||||
|
||||
def progress(self):
|
||||
for fw in self.fw.values():
|
||||
fw.progress()
|
||||
|
||||
|
||||
class Clock:
|
||||
def __init__(self):
|
||||
self.now = datetime.now(timezone.utc)
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clock(monkeypatch):
|
||||
c = Clock()
|
||||
monkeypatch.setattr(engine, "_now", c)
|
||||
monkeypatch.setattr(svc, "_now", c)
|
||||
return c
|
||||
|
||||
|
||||
async def _setup(client, respx_mock, *, waves=None, **plan_kw):
|
||||
fleet = Fleet(respx_mock)
|
||||
hosts = {n: await _create_host(client, n, f"https://{n}.test") for n in ("c1", "ha-a", "ha-b")}
|
||||
for n in ("ha-a", "ha-b"):
|
||||
await client.patch(f"/api/hosts/{hosts[n]['id']}", json={"cluster_name": "ha"})
|
||||
groups = {}
|
||||
for g, members in (("canary", ["c1"]), ("pilot", ["ha-b"]), ("prod", ["ha-a"]), ("cluster", ["ha-a", "ha-b"])):
|
||||
groups[g] = (await client.post("/api/groups", json={"name": g, "host_ids": [hosts[m]["id"] for m in members]})).json()["id"]
|
||||
waves = waves or [
|
||||
{"label": "canary", "group_ids": [groups["canary"]], "soak_minutes": 5},
|
||||
{"label": "pilot", "group_ids": [groups["pilot"]], "soak_minutes": 5},
|
||||
{"label": "prod", "group_ids": [groups["prod"]], "soak_minutes": 0, "carp_handover": True},
|
||||
]
|
||||
waves = [{**w, "group_ids": [groups[g] if g in groups else g for g in w["group_ids"]]} for w in waves]
|
||||
plan = (await client.post("/api/rollouts/plans", json={"name": "ringe", "action": "update", "waves": waves, **plan_kw})).json()
|
||||
return fleet, hosts, groups, plan
|
||||
|
||||
|
||||
async def _start(client, plan, api=None):
|
||||
api = api or client
|
||||
token = (await api.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
resp = await api.post(f"/api/rollouts/plans/{plan['id']}/start",
|
||||
json={"preview_token": token, "confirm_name": plan["name"], "acknowledge_unattended": True})
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
async def _tick(fleet, clock, seconds=30):
|
||||
fleet.progress()
|
||||
async with SessionLocal() as db:
|
||||
await svc.fire_due(db)
|
||||
async with SessionLocal() as db:
|
||||
await svc.advance_running(db)
|
||||
async with SessionLocal() as db:
|
||||
await engine.advance_runs(db)
|
||||
clock.now += timedelta(seconds=seconds)
|
||||
|
||||
|
||||
async def _run(run_id):
|
||||
async with SessionLocal() as db:
|
||||
return await db.scalar(select(RolloutRun).where(RolloutRun.id == uuid.UUID(str(run_id))).execution_options(populate_existing=True))
|
||||
|
||||
|
||||
async def _until(fleet, clock, run_id, pred, max_ticks=400, check=None):
|
||||
for _ in range(max_ticks):
|
||||
run = await _run(run_id)
|
||||
if pred(run):
|
||||
return run
|
||||
if check:
|
||||
await check()
|
||||
await _tick(fleet, clock)
|
||||
raise AssertionError(f"not reached: status={run.status} reason={run.pause_reason} detail={run.pause_detail}")
|
||||
|
||||
|
||||
def _host(run, name):
|
||||
return next(h for h in run.hosts if h.host_name == name)
|
||||
|
||||
|
||||
async def _scheds():
|
||||
async with SessionLocal() as db:
|
||||
return list((await db.scalars(select(ScheduledMaintenance).order_by(ScheduledMaintenance.created_at))).all())
|
||||
|
||||
|
||||
async def _no_two_at_once():
|
||||
active = [s for s in await _scheds() if s.status in ("pending", "running")]
|
||||
assert len(active) <= 1, [(s.host_name, s.status, s.phase) for s in active]
|
||||
|
||||
|
||||
# ── happy path ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_three_rings_run_through_and_the_master_hands_over(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert done.target_version == "26.7.7"
|
||||
assert [(h.host_name, h.status, h.version_before, h.version_after) for h in sorted(done.hosts, key=lambda h: h.wave_index)] == [
|
||||
("c1", "succeeded", "26.7.6", "26.7.7"), ("ha-b", "succeeded", "26.7.6", "26.7.7"), ("ha-a", "succeeded", "26.7.6", "26.7.7")]
|
||||
assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
|
||||
# The MASTER went through the CARP handover and is MASTER again; the BACKUP never needed one.
|
||||
assert fleet.fw["ha-a"].toggles.call_count == 2 and fleet.fw["ha-b"].toggles.call_count == 0
|
||||
assert (fleet.fw["ha-a"].role, fleet.fw["ha-b"].role) == ("MASTER", "BACKUP")
|
||||
scheds = await _scheds()
|
||||
assert [s.host_name for s in scheds] == ["c1", "ha-b", "ha-a"]
|
||||
assert all(s.origin == "rollout" and s.auto_verify and s.status == "succeeded" for s in scheds)
|
||||
async with SessionLocal() as db:
|
||||
ops = (await db.scalars(select(HostOperation))).all()
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rollout"))).all()
|
||||
assert {o.status for o in ops} == {"acknowledged"} # nothing left locked
|
||||
assert sum(1 for e in events if e.action == "gate" and e.status == "ok") == 3
|
||||
assert any(e.action == "finish" and e.status == "succeeded" for e in events)
|
||||
host_events = [e for e in (await _audit_host()) if (e.details or {}).get("rollout")]
|
||||
assert host_events and all(e.details["rollout"]["run_id"] == run["id"] for e in host_events)
|
||||
|
||||
|
||||
async def _audit_host():
|
||||
async with SessionLocal() as db:
|
||||
return (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
|
||||
|
||||
|
||||
# ── the gate stops the rollout ─────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gate_failure_in_the_canary_ring_stops_everything_after_it(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
fleet.fw["c1"].services[1]["running"] = 0 # ntpd did not come back
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "gate_failed" and paused.paused_from == "gate"
|
||||
assert paused.waves[0].gate_attempts == 2 # looked at twice (slow start), then stopped
|
||||
assert paused.waves[0].gate_result["hosts"][_host(paused, "c1").host_id.__str__()]["services"]["code"] == "service_stopped"
|
||||
for _ in range(20): # time passes, nothing moves
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
|
||||
assert [s.host_name for s in await _scheds()] == ["c1"]
|
||||
# Notification while paused.
|
||||
from app.services import notification_checks
|
||||
async with SessionLocal() as db:
|
||||
problems = await notification_checks.check_rollout_paused(db)
|
||||
assert problems[run["id"]]["grund"] == "gate_failed"
|
||||
|
||||
# Fixed by hand, resumed: the gate is checked again first, then it goes on.
|
||||
fleet.fw["c1"].services[1]["running"] = 1
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
|
||||
assert resp.status_code == 200 and resp.json()["status"] == "resuming"
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert fleet.fw["ha-a"].update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resume_rechecks_and_pauses_again_if_still_broken(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
fleet.fw["c1"].gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100 %", "delay": "~", "stddev": "~"}
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
|
||||
again = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert again.pause_reason == "gate_failed" and again.pause_detail.get("on_resume") is True
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
|
||||
|
||||
# ── never twice, never in parallel ─────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restart_in_the_middle_of_job_wait_submits_exactly_once(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
|
||||
async def in_job_wait(r):
|
||||
return any(s.host_name == "ha-b" and s.phase == "job_wait" for s in await _scheds())
|
||||
for _ in range(300):
|
||||
if await in_job_wait(None):
|
||||
break
|
||||
await _tick(fleet, clock)
|
||||
# "Restart": nothing survives but the database (each tick opens fresh sessions anyway); two backends
|
||||
# even step the same run at the very same moment.
|
||||
await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(3)))
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded"
|
||||
assert fleet.fw["ha-b"].update.call_count == 1 and len([s for s in await _scheds() if s.host_name == "ha-b"]) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_two_backends_deciding_at_once_create_one_schedule(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "wave_running")
|
||||
await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(4)))
|
||||
assert len(await _scheds()) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_cluster_in_one_wave_goes_backup_first_and_never_both_at_once(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
|
||||
run = await _start(client, plan)
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert [s.host_name for s in await _scheds()] == ["ha-b", "ha-a"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_master_without_handover_pauses_before_anything_happens_to_it(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": False}])
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "master_needs_handover" and paused.pause_detail["host"] == "ha-a"
|
||||
assert fleet.fw["ha-b"].update.call_count == 1 and fleet.fw["ha-a"].update.call_count == 0
|
||||
assert fleet.fw["ha-a"].toggles.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_gate_before_the_second_node(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
|
||||
run = await _start(client, plan)
|
||||
|
||||
async def break_backup_after_update():
|
||||
if fleet.fw["ha-b"].version == "26.7.7":
|
||||
fleet.fw["ha-b"].services[0]["running"] = 0 # unbound dead on the freshly updated node
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused", check=break_backup_after_update)
|
||||
assert paused.pause_reason == "cluster_gate_failed" and paused.pause_detail["host"] == "ha-a"
|
||||
assert fleet.fw["ha-a"].update.call_count == 0 and fleet.fw["ha-a"].toggles.call_count == 0
|
||||
|
||||
|
||||
# ── stop when in doubt ─────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_different_target_version_in_a_later_ring_pauses(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["ha-b"].latest = "26.7.8" # a newer release appeared after the canary ran
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "target_version_changed"
|
||||
assert paused.pause_detail == {"expected": "26.7.7", "offered": "26.7.8"}
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hosts_without_an_update_are_skipped(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["ha-b"].offer = "none"
|
||||
run = await _start(client, plan)
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded" and _host(done, "ha-b").status == "skipped"
|
||||
assert _host(done, "ha-b").detail["code"] == "no_update_offered" and done.waves[1].status == "skipped"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_host_pauses_and_resume_needs_an_explicit_skip(client, respx_mock, clock):
|
||||
fleet, hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["c1"].update_error = True
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "host_failed" and _host(paused, "c1").status == "failed"
|
||||
refused = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ROLLOUT_HOST_FAILED"
|
||||
skip = await client.post(f"/api/rollouts/runs/{run['id']}/skip-host", json={"host_id": hosts["c1"]["id"], "reason": "geprüft, später"})
|
||||
assert skip.status_code == 200
|
||||
# The failed operation stays "unknown" (locks c1, notifies) until a human closes it; it is not retried.
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert fleet.fw["c1"].update.call_count == 1 and _host(done, "c1").detail["code"] == "skipped_by"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unreachable_host_in_preflight_is_blocked_then_paused(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["c1"].up = False
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "preflight_blocked" and _host(paused, "c1").status == "blocked"
|
||||
assert clock.now - datetime.fromisoformat(paused.progress["paused_at"]) < timedelta(minutes=1)
|
||||
fleet.fw["c1"].up = True
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_someone_elses_open_maintenance_blocks_then_pauses(client, respx_mock, clock):
|
||||
fleet, hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.current_wave == 1 and r.status == "wave_running")
|
||||
async with SessionLocal() as db:
|
||||
db.add(HostOperation(host_id=uuid.UUID(hosts["ha-a"]["id"]), host_name="ha-a", actor="someone", action="reboot",
|
||||
status="submitted", expires_at=clock.now, fingerprint="", snapshot={}))
|
||||
await db.commit()
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason in ("blocked_by_maintenance", "cluster_gate_failed")
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_actor_who_lost_the_right_pauses_the_run(client, respx_mock, clock):
|
||||
from tests.test_operator_role import _operator
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
op = await _operator(client, "rollengine-op")
|
||||
try:
|
||||
run = await _start(client, plan, api=op)
|
||||
assert run["started_by"] == "rollengine-op"
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "actor_invalid" and fleet.fw["ha-b"].update.call_count == 0
|
||||
# An admin takes over by resuming.
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).json()["started_by"] == "admin"
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded"
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
|
||||
|
||||
# ── windows, approval, skip, abort, automatic start ───────────────────────────
|
||||
|
||||
async def _window_now(client, clock, *, cutoff=60, duration=120):
|
||||
local = clock.now
|
||||
body = {"name": f"jetzt-{uuid.uuid4().hex[:6]}", "timezone": "UTC", "recurrence": "weekly", "weekdays": [local.weekday()],
|
||||
"start_time": (local - timedelta(minutes=1)).strftime("%H:%M"), "duration_minutes": duration, "start_cutoff_minutes": cutoff}
|
||||
if (local - timedelta(minutes=1)).date() != local.date():
|
||||
pytest.skip("too close to midnight for a same-day window")
|
||||
resp = await client.post("/api/maintenance-windows", json=body)
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_closed_window_waits_and_the_next_occurrence_resumes(client, respx_mock, clock):
|
||||
window = await _window_now(client, clock, cutoff=15)
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"])
|
||||
run = await _start(client, plan)
|
||||
waiting = await _until(fleet, clock, run["id"], lambda r: r.status in ("waiting_window", "paused", "succeeded"))
|
||||
assert waiting.status == "waiting_window" and waiting.current_wave >= 1
|
||||
started = {s.host_name for s in await _scheds()}
|
||||
assert "c1" in started and "ha-a" not in started
|
||||
clock.now += timedelta(days=7) - timedelta(minutes=40) # one week later, a few minutes into the next occurrence
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_approval_holds_the_wave(client, respx_mock, clock):
|
||||
fleet, _hosts, groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "canary", "group_ids": ["canary"], "soak_minutes": 1},
|
||||
{"label": "rest", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True, "require_approval": True}])
|
||||
run = await _start(client, plan)
|
||||
held = await _until(fleet, clock, run["id"], lambda r: r.status == "awaiting_approval")
|
||||
for _ in range(10):
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/approve")
|
||||
assert resp.status_code == 200 and resp.json()["waves"][1]["approved_by"] == "admin"
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded" and held.current_wave == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_abort_cancels_what_has_not_started_and_lets_the_running_host_finish(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: any(h.status == "running" for h in r.hosts))
|
||||
bad = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "falsch"})
|
||||
assert bad.status_code == 409 and bad.json()["detail"]["code"] == "ROLLOUT_CONFIRM_NAME"
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "ringe"})
|
||||
assert resp.status_code == 200 and resp.json()["status"] == "cancelled"
|
||||
for _ in range(40):
|
||||
await _tick(fleet, clock)
|
||||
run_now = await _run(run["id"])
|
||||
assert run_now.status == "cancelled"
|
||||
assert {h.host_name: h.status for h in run_now.hosts} in (
|
||||
{"c1": "succeeded", "ha-b": "cancelled", "ha-a": "cancelled"}, {"c1": "cancelled", "ha-b": "cancelled", "ha-a": "cancelled"})
|
||||
assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pause_by_hand_and_operator_may_always_brake(client, respx_mock, clock):
|
||||
from tests.test_operator_role import _operator
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
op = await _operator(client, "brake-op")
|
||||
try:
|
||||
resp = await op.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "Bauchgefühl"})
|
||||
assert resp.status_code == 200 and resp.json()["pause_reason"] == "manual"
|
||||
denied = await op.post(f"/api/rollouts/runs/{run['id']}/resume")
|
||||
assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
for _ in range(10):
|
||||
await _tick(fleet, clock)
|
||||
assert sum(fw.update.call_count for fw in fleet.fw.values()) == 0
|
||||
second = await client.post(f"/api/rollouts/plans/{plan['id']}/start", json={
|
||||
"preview_token": (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"],
|
||||
"confirm_name": "ringe", "acknowledge_unattended": True})
|
||||
assert second.status_code == 409 and second.json()["detail"]["code"] == "ROLLOUT_ALREADY_ACTIVE"
|
||||
|
||||
|
||||
async def _until_step_pending(fleet, clock, run_id):
|
||||
"""Right after the engine created a host's schedule (fire_due has not seen it yet)."""
|
||||
run = await _until(fleet, clock, run_id, lambda r: any(h.status == "running" for h in r.hosts))
|
||||
name = next(h.host_name for h in run.hosts if h.status == "running")
|
||||
assert [s.status for s in await _scheds()] == ["pending"]
|
||||
return name
|
||||
|
||||
|
||||
async def _resume_to_the_end(client, fleet, clock, run_id):
|
||||
assert (await client.post(f"/api/rollouts/runs/{run_id}/resume")).status_code == 200
|
||||
done = await _until(fleet, clock, run_id, lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pause_while_a_step_is_pending_really_brakes_and_resume_runs_it_once(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
name = await _until_step_pending(fleet, clock, run["id"])
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "halt"})
|
||||
assert resp.status_code == 200
|
||||
for _ in range(10):
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw[name].update.call_count == 0 # the pending step did not fire behind the pause
|
||||
paused = await _run(run["id"])
|
||||
assert paused.status == "paused" and _host(paused, name).status == "queued"
|
||||
assert _host(paused, name).scheduled_maintenance_id is None
|
||||
assert [s.status for s in await _scheds()] == ["cancelled"]
|
||||
await _resume_to_the_end(client, fleet, clock, run["id"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_step_found_long_overdue_after_an_outage_is_not_fired(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
name = await _until_step_pending(fleet, clock, run["id"])
|
||||
clock.now += timedelta(minutes=30) # OPNexus was down (e.g. a web update); the window may be over by now
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw[name].update.call_count == 0
|
||||
paused = await _run(run["id"])
|
||||
assert (paused.status, paused.pause_reason) == ("paused", "host_not_started")
|
||||
assert _host(paused, name).status == "queued" and _host(paused, name).detail["code"] == "not_fired"
|
||||
(sched,) = await _scheds()
|
||||
assert sched.status == "failed" and sched.result["not_fired"]
|
||||
await _resume_to_the_end(client, fleet, clock, run["id"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automatic_start_once_per_occurrence_and_only_while_unchanged(client, respx_mock, clock):
|
||||
window = await _window_now(client, clock)
|
||||
fleet, _hosts, groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
|
||||
for _ in range(3):
|
||||
async with SessionLocal() as db:
|
||||
await engine.auto_start(db)
|
||||
async with SessionLocal() as db:
|
||||
runs = (await db.scalars(select(RolloutRun))).all()
|
||||
assert len(runs) == 1 and runs[0].trigger == "window" and runs[0].started_by == "admin"
|
||||
assert runs[0].window_occurrence is not None
|
||||
|
||||
# Next week, but somebody added a host to the canary group meanwhile: no start, a notification instead.
|
||||
async with SessionLocal() as db:
|
||||
(await db.get(RolloutRun, runs[0].id)).status = "cancelled"
|
||||
await db.commit()
|
||||
await client.put(f"/api/groups/{groups['canary']}", json={"name": "canary", "host_ids": [_hosts["c1"]["id"], _hosts["ha-a"]["id"]]})
|
||||
clock.now += timedelta(days=7)
|
||||
async with SessionLocal() as db:
|
||||
await engine.auto_start(db)
|
||||
async with SessionLocal() as db:
|
||||
assert len((await db.scalars(select(RolloutRun))).all()) == 1
|
||||
stored = await db.get(RolloutPlan, uuid.UUID(plan["id"]))
|
||||
assert stored.auto_start_error["code"] == "plan_changed"
|
||||
from app.services import notification_checks
|
||||
async with SessionLocal() as db:
|
||||
problems = await notification_checks.check_rollout_start_failed(db)
|
||||
assert list(problems.values())[0]["grund"] == "plan_changed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_second_backend_waiting_for_the_lock_reports_no_refusal(client, respx_mock, clock, monkeypatch):
|
||||
window = await _window_now(client, clock)
|
||||
_fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
|
||||
real_lock = engine.policy_lock
|
||||
|
||||
async def racing_lock(db):
|
||||
monkeypatch.setattr(engine, "policy_lock", real_lock)
|
||||
async with SessionLocal() as other: # backend A starts the run while B waits for the lock
|
||||
assert await engine.auto_start(other) == 1
|
||||
await real_lock(db)
|
||||
monkeypatch.setattr(engine, "policy_lock", racing_lock)
|
||||
async with SessionLocal() as db:
|
||||
assert await engine.auto_start(db) == 0
|
||||
async with SessionLocal() as db:
|
||||
assert len((await db.scalars(select(RolloutRun))).all()) == 1
|
||||
assert (await db.get(RolloutPlan, uuid.UUID(plan["id"]))).auto_start_error is None
|
||||
|
||||
|
||||
def test_target_version_comes_from_the_core_package_not_product_latest():
|
||||
# Shape seen live on test-fw-3 (26.7.1_1): product_latest lagged behind what would really be installed.
|
||||
status = {"status": "update", "product": {"product_latest": "26.7.2"},
|
||||
"all_packages": {"opnsense": {"old": "26.7.1_1", "new": "26.7.7", "reason": "upgrade"}},
|
||||
"upgrade_packages": [{"name": "opnsense", "current_version": "26.7.1_1", "new_version": "26.7.7"}]}
|
||||
assert engine.target_of(status, "update") == "26.7.7"
|
||||
assert engine.target_of({**status, "all_packages": {}}, "update") == "26.7.7"
|
||||
assert engine.target_of({"product": {"product_latest": "26.7.7"}}, "update") == "26.7.7"
|
||||
assert engine.target_of({"upgrade_major_version": "27.1"}, "major_update") == "27.1"
|
||||
@@ -0,0 +1,195 @@
|
||||
"""Health gate checks of the rollout engine, one by one, against a fake firewall pair."""
|
||||
import uuid
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.carp_maintenance_window import CarpMaintenanceWindow
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.services import rollout_health as rh
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
A, B = "https://gw-a.test", "https://gw-b.test"
|
||||
|
||||
|
||||
class Fw:
|
||||
def __init__(self, respx_mock, base, *, role="MASTER", version="26.7.7"):
|
||||
self.base, self.role, self.version, self.up, self.job, self.maint = base, role, version, True, "ready", False
|
||||
self.vips = None
|
||||
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1.0 ms", "stddev": "0.1 ms"}]
|
||||
self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
|
||||
{"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
|
||||
self.gw_error = False
|
||||
r = respx_mock
|
||||
r.get(f"{base}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
|
||||
r.get(f"{base}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
|
||||
r.get(f"{base}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
|
||||
r.get(f"{base}/api/routes/gateway/status").mock(side_effect=self._gw)
|
||||
r.get(f"{base}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
|
||||
|
||||
def _ok(self, data):
|
||||
if not self.up:
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, json=data)
|
||||
|
||||
def _gw(self, request):
|
||||
if self.gw_error:
|
||||
return httpx.Response(403, json={"message": "forbidden"})
|
||||
return self._ok({"items": self.gateways, "status": "ok"})
|
||||
|
||||
def _vip(self, request):
|
||||
rows = [{"vhid": v, "status": s} for v, s in (self.vips or {"1": self.role}).items()]
|
||||
return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "0", "status_msg": ""}})
|
||||
|
||||
|
||||
async def _hosts(client, cluster=True):
|
||||
a = await _create_host(client, "gw-a", A)
|
||||
if not cluster:
|
||||
return a, None
|
||||
b = await _create_host(client, "gw-b", B)
|
||||
if cluster:
|
||||
for h in (a, b):
|
||||
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
|
||||
return a, b
|
||||
|
||||
|
||||
async def _eval(host_ids, baselines=None, target="26.7.7", **kw):
|
||||
async with SessionLocal() as db:
|
||||
hosts = [await db.get(OpnsenseHost, uuid.UUID(h)) for h in host_ids]
|
||||
if baselines is None:
|
||||
baselines = {str(h.id): await rh.baseline(db, h) for h in hosts}
|
||||
return await rh.evaluate(db, hosts, baselines, target_version=target, **kw), baselines
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_healthy_cluster_passes_every_check(client, respx_mock):
|
||||
Fw(respx_mock, A)
|
||||
Fw(respx_mock, B, role="BACKUP")
|
||||
a, _ = await _hosts(client)
|
||||
gate, base = await _eval([a["id"]])
|
||||
assert gate["verdict"] == "ok", gate
|
||||
row = gate["hosts"][a["id"]]
|
||||
assert {k: v["result"] for k, v in row.items()} == {
|
||||
"online": "ok", "version": "ok", "carp": "ok", "gateways": "ok", "services": "ok", "locks": "ok"}
|
||||
assert base[a["id"]]["gateways"] == {"WAN_GW": "online"} and base[a["id"]]["services"]["ntpd"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_offline_or_busy_host_and_wrong_version_fail(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
fa.version = "26.7.6"
|
||||
gate, base = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["version"] == {"result": "fail", "code": "version_mismatch",
|
||||
"params": {"expected": "26.7.7", "actual": "26.7.6"}}
|
||||
fa.job = "busy"
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["online"]["code"] == "firmware_busy"
|
||||
fa.up = False
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["online"]["code"] == "offline" and gate["verdict"] == "fail"
|
||||
assert gate["hosts"][a["id"]]["gateways"]["result"] == "fail" # readable before, not now
|
||||
assert rh.failed_checks(gate)[0].startswith(a["id"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_carp_split_brain_maintenance_mode_and_timer_fail(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
fb = Fw(respx_mock, B, role="BACKUP")
|
||||
a, b = await _hosts(client)
|
||||
fb.role = "MASTER" # both MASTER on VHID 1
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "vhid_roles", "params": {"vhids": "1"}}
|
||||
fb.role = "BACKUP"
|
||||
fa.vips, fb.vips = {"1": "MASTER", "2": "MASTER"}, {"1": "BACKUP", "2": "MASTER"} # partial failover on VHID 2
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"]["params"] == {"vhids": "2"}
|
||||
fa.vips = fb.vips = None
|
||||
fb.maint = True
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"]["code"] == "maintenance_mode"
|
||||
fb.maint = False
|
||||
async with SessionLocal() as db:
|
||||
db.add(CarpMaintenanceWindow(host_id=uuid.UUID(b["id"]), host_name="gw-b", status="active"))
|
||||
await db.commit()
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"]["code"] == "auto_revert_active"
|
||||
fb.up = False
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "node_offline", "params": {"names": "gw-b"}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateways_compare_against_the_baseline(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
fa.gateways.append({"name": "OLD_GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms"})
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
_gate, base = await _eval([a["id"]])
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["code"] == "gateway_bad_before" # only warns
|
||||
fa.gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~"}
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["gateways"] == {"result": "fail", "code": "gateway_down", "params": {"names": "WAN_GW"}}
|
||||
fa.gateways[0] = {"name": "WAN_GW", "status": "loss", "loss": "30 %", "delay": "5 ms", "stddev": "1 ms"}
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "warn"
|
||||
fa.gw_error = True
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "fail"
|
||||
gate, _ = await _eval([a["id"]], {a["id"]: {**base[a["id"]], "gateways": None}})
|
||||
assert gate["hosts"][a["id"]]["gateways"]["result"] == "skip" # no permission before either: nothing to compare
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_services_that_ran_before_must_run_again(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
fa.services.append({"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0})
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
_gate, base = await _eval([a["id"]])
|
||||
fa.services[1] = {"id": "ntpd", "name": "ntpd", "locked": 0, "running": 0}
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["services"] == {"result": "fail", "code": "service_stopped", "params": {"names": "ntpd"}}
|
||||
fa.services[1]["running"] = 1
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["services"]["result"] == "ok" # dhcpd6 never ran: fine
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_open_operation_in_the_cluster_is_a_lock(client, respx_mock):
|
||||
Fw(respx_mock, A)
|
||||
Fw(respx_mock, B, role="BACKUP")
|
||||
a, b = await _hosts(client)
|
||||
from datetime import datetime, timezone
|
||||
async with SessionLocal() as db:
|
||||
db.add(HostOperation(host_id=uuid.UUID(b["id"]), host_name="gw-b", actor="x", action="reboot", status="submitted",
|
||||
expires_at=datetime.now(timezone.utc), fingerprint="", snapshot={}))
|
||||
await db.commit()
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["locks"] == {"result": "fail", "code": "open_locks", "params": {"count": 1}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode,expected", [("warn", "warn"), ("fail", "fail")])
|
||||
@pytest.mark.asyncio
|
||||
async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeypatch, mode, expected):
|
||||
Fw(respx_mock, A)
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
async with SessionLocal() as db:
|
||||
db.add(ComplianceResult(host_id=uuid.UUID(a["id"]), checked_at=datetime_now(), score=80, findings=[]))
|
||||
await db.commit()
|
||||
|
||||
async def fake_check(db, host):
|
||||
row = await db.get(ComplianceResult, (await db.scalar(__import__("sqlalchemy").select(ComplianceResult.id))))
|
||||
row.score = 70
|
||||
await db.commit()
|
||||
return row
|
||||
monkeypatch.setattr(rh.compliance_service, "check_host", fake_check)
|
||||
gate, _ = await _eval([a["id"]], checks={"compliance": True}, compliance_mode=mode)
|
||||
assert gate["hosts"][a["id"]]["compliance"] == {"result": expected, "code": "score_lower", "params": {"before": 80, "after": 70}}
|
||||
assert gate["verdict"] == expected
|
||||
# the gate cannot be switched off where it matters
|
||||
gate, _ = await _eval([a["id"]], checks={"online": False, "version": False, "locks": False})
|
||||
assert {"online", "version", "locks"} <= set(gate["hosts"][a["id"]])
|
||||
|
||||
|
||||
def datetime_now():
|
||||
from datetime import datetime, timezone
|
||||
return datetime.now(timezone.utc)
|
||||
@@ -0,0 +1,334 @@
|
||||
"""Rule analysis (1.26): normalization, resolver and pf semantics of the engine.
|
||||
|
||||
Pure functions against an anonymized live fixture (OPNsense 26.7: automatic rules, antispoof
|
||||
lines, a CARP VIP on LAN). Synthetic rules get pf lines in sort order, as pf would load them.
|
||||
The numbers in the test names follow the test matrix of the design (section 8).
|
||||
"""
|
||||
import copy
|
||||
import ipaddress
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.rule_analysis import context as ctx_mod
|
||||
from app.services.rule_analysis import pf_ruleset, resolve
|
||||
from app.services.rule_analysis.engine import evaluate
|
||||
from app.services.rule_analysis.normalize import normalize
|
||||
from app.services.rule_analysis.service import build_query
|
||||
|
||||
FIXTURE = json.loads((Path(__file__).parent / "fixtures" / "rule_analysis_fw.json").read_text())
|
||||
DEVICES = {"lan": "vtnet1_vlan10", "wan": "vtnet0"}
|
||||
LAN_CLIENT, INTERNET, WAN_IP = "10.20.30.50", "198.51.100.10", "203.0.113.57"
|
||||
|
||||
|
||||
def mvc(uuid, *, prio=400000, seq=1, action="pass", quick="1", interface="lan", src="any", dst="any",
|
||||
proto="any", ipprotocol="inet", dport="", sport="", direction="in", loaded=True, **extra):
|
||||
row = {
|
||||
"uuid": uuid, "sort_order": f"{prio:06d}.{seq:07d}", "prio_group": str(prio), "sequence": str(seq),
|
||||
"enabled": "1", "action": action, "quick": quick, "interface": interface, "interfacenot": "0",
|
||||
"direction": direction, "ipprotocol": ipprotocol, "protocol": proto, "source_net": src, "source_not": "0",
|
||||
"source_port": sport, "destination_net": dst, "destination_not": "0", "destination_port": dport,
|
||||
"gateway": "", "sched": "", "statetype": "keep", "tagged": "", "description": uuid,
|
||||
}
|
||||
row.update(extra)
|
||||
row["_loaded"] = loaded
|
||||
return row
|
||||
|
||||
|
||||
def build(extra_rows=(), *, drop=(), aliases=None, refs=None, groups=None, pf=True, port_forwards=()):
|
||||
"""A Context from the fixture plus synthetic MVC rules. pf lines of MVC rules are generated
|
||||
in sort order (pf numbers follow the evaluation order like on a real firewall)."""
|
||||
data = copy.deepcopy(FIXTURE)
|
||||
rows = [r for r in data["rules"]["rows"] if r["description"] not in drop] + [dict(r) for r in extra_rows]
|
||||
interfaces = ctx_mod._interfaces(data["interfaces"])
|
||||
group_rows = data["groups"]["rows"] + [{"ifname": k, "members": ",".join(v)} for k, v in (groups or {}).items()]
|
||||
stats = None
|
||||
if pf:
|
||||
auto = {k: v for k, v in data["pf"]["rules"]["filter rules"].items()
|
||||
if not any(r.get("uuid") in k for r in data["rules"]["rows"] if not r.get("is_automatic"))}
|
||||
lines = sorted(((int(k.split()[0][1:]), k.split(" ", 1)[1]) for k in auto), key=lambda x: x[0])
|
||||
number = max(n for n, _ in lines) + 1
|
||||
mvc_rows = sorted((r for r in rows if not r.get("is_automatic") and r.get("_loaded", True)
|
||||
and r.get("enabled") == "1"), key=lambda r: r.get("sort_order") or "400000." + r["sequence"].zfill(7))
|
||||
for r in mvc_rows:
|
||||
words = ["block" if r["action"] == "block" else ("block return" if r["action"] == "reject" else "pass"),
|
||||
r["direction"] if r["direction"] != "any" else ""]
|
||||
if r["quick"] == "1":
|
||||
words.append("quick")
|
||||
ifs = [i for i in str(r["interface"]).split(",") if i]
|
||||
if len(ifs) == 1 and ifs[0] in DEVICES:
|
||||
words.append(f"on {DEVICES[ifs[0]]}")
|
||||
if r["ipprotocol"] in ("inet", "inet6"):
|
||||
words.append(r["ipprotocol"])
|
||||
words.append(f'all label "{r["uuid"]}"')
|
||||
lines.append((number, " ".join(w for w in words if w)))
|
||||
number += 1
|
||||
stats = {"rules": {"filter rules": {f"@{n} {t}": {} for n, t in lines}}}
|
||||
for r in rows:
|
||||
r.pop("_loaded", None)
|
||||
alias_rows = {a["name"]: a for a in data["aliases"]["rows"]}
|
||||
alias_rows.update({name: {"name": name, **a} for name, a in (aliases or {}).items()})
|
||||
ctx = ctx_mod.Context(rows=rows, pf=pf_ruleset.parse(stats) if pf else None, interfaces=interfaces,
|
||||
groups=ctx_mod._groups({"rows": group_rows}, interfaces), aliases=alias_rows,
|
||||
port_forwards=list(port_forwards), notes=[] if pf else ["PF_RULESET_UNAVAILABLE"])
|
||||
default_refs = {LAN_CLIENT: ["__lan_network"], INTERNET: [], WAN_IP: ["__wan_network"]}
|
||||
ctx.refs = {**default_refs, **(refs or {})}
|
||||
return ctx
|
||||
|
||||
|
||||
def run(ctx, src=LAN_CLIENT, dst=INTERNET, proto="tcp", dport="443", iface="lan", **kw):
|
||||
q = build_query(src=src, dst=dst, proto=proto, dport=dport, iface=iface, **kw)
|
||||
for ip in (str(q.src), str(q.dst)):
|
||||
ctx.refs.setdefault(ip, [])
|
||||
return evaluate(ctx, q)
|
||||
|
||||
|
||||
def decided_by(result):
|
||||
return result["decisive_rule"]["description"] if result["decisive_rule"] else None
|
||||
|
||||
|
||||
# ── Normalization ─────────────────────────────────────────────────────────
|
||||
|
||||
def test_normalize_live_fixture_uses_pf_for_automatic_rules():
|
||||
rules, notes = normalize(build())
|
||||
by_desc = {}
|
||||
for r in rules:
|
||||
by_desc.setdefault(r.description, r)
|
||||
assert notes == []
|
||||
deny = by_desc["Default deny / state violation rule"]
|
||||
assert deny.quick is False and deny.action == "block" and deny.loaded and deny.pf_numbers
|
||||
dhcp = by_desc["allow access to DHCP server"]
|
||||
assert dhcp.action == "pass" and dhcp.quick is True and dhcp.interfaces == {"lan"} # "LAN" + empty action
|
||||
lockout = by_desc["anti-lockout rule"]
|
||||
assert lockout.interfaces == {"lan"} and lockout.automatic and lockout.legacy
|
||||
allow = by_desc["Default allow LAN to any rule"]
|
||||
assert allow.section == "interface" and allow.quick and allow.af == {4} and not allow.automatic
|
||||
spoof = [r for r in rules if r.section == "antispoof"]
|
||||
assert spoof and spoof[0].interfaces == {"lan"} and spoof[0].if_not and spoof[0].quick is False
|
||||
# Antispoof lines sit before the first automatic rule, like in pf.
|
||||
assert rules.index(spoof[0]) < rules.index(deny)
|
||||
|
||||
|
||||
def test_normalize_reports_order_mismatch_and_missing_sort_order():
|
||||
ctx = build([mvc("r-a", seq=1)])
|
||||
# Swap pf positions: the rule now appears in pf before the system rules.
|
||||
key = next(k for k in ctx.pf.by_label if k == "r-a")
|
||||
ctx.pf.by_label[key][0].number = -1
|
||||
assert "ORDER_MISMATCH" in normalize(ctx)[1]
|
||||
row = mvc("r-b")
|
||||
row.pop("sort_order")
|
||||
assert "ORDER_GUESSED" in normalize(build([row]))[1]
|
||||
|
||||
|
||||
def test_unmigrated_legacy_rule_with_interface_description():
|
||||
# Seen live: legacy user rules have a 32-hex label and "WAN" instead of "wan".
|
||||
legacy = mvc("0123456789abcdef0123456789abcdef", seq=1, action="block", interface="WAN", legacy=True)
|
||||
r = run(build([legacy], drop=("Allow ALL",)), src=INTERNET, dst=WAN_IP, iface="wan")
|
||||
assert decided_by(r) == legacy["uuid"] and r["confidence"] == "sure"
|
||||
assert r["decisive_rule"]["legacy"] and not r["decisive_rule"]["editable"]
|
||||
|
||||
|
||||
# ── Engine: matrix 1-8, 14-20 ─────────────────────────────────────────────
|
||||
|
||||
def test_01_lan_client_to_internet_is_default_allow_sure():
|
||||
r = run(build())
|
||||
assert r["action"] == "pass" and decided_by(r) == "Default allow LAN to any rule"
|
||||
assert r["confidence"] == "sure" and r["rule_confidence"] == "sure" and r["unknowns"] == []
|
||||
|
||||
|
||||
def test_02_wan_to_firewall_ssh_hits_default_deny():
|
||||
r = run(build(drop=("Allow ALL",)), src=INTERNET, dst=WAN_IP, dport="22", iface="wan")
|
||||
assert r["action"] == "block" and decided_by(r) == "Default deny / state violation rule"
|
||||
assert r["confidence"] == "sure" and r["decisive_rule"]["quick"] is False
|
||||
|
||||
|
||||
def test_03_floating_non_quick_block_loses_to_interface_quick_pass():
|
||||
ctx = build([mvc("float-block", prio=200000, action="block", quick="0", interface="")])
|
||||
r = run(ctx)
|
||||
assert r["action"] == "pass" and decided_by(r) == "Default allow LAN to any rule"
|
||||
effects = {s["rule"]["description"]: s["effect"] for s in r["trace"][0]["steps"]}
|
||||
assert effects["float-block"] == "overridden"
|
||||
|
||||
|
||||
def test_04_floating_quick_block_on_lan_and_wan_wins():
|
||||
r = run(build([mvc("float-quick", prio=200000, action="block", quick="1", interface="lan,wan")]))
|
||||
assert r["action"] == "block" and decided_by(r) == "float-quick" and r["confidence"] == "sure"
|
||||
assert r["decisive_rule"]["section"] == "floating"
|
||||
|
||||
|
||||
def test_05_floating_interfacenot_applies_to_the_other_interfaces():
|
||||
ctx = build([mvc("not-lan", prio=200000, action="block", interface="lan", interfacenot="1")])
|
||||
assert decided_by(run(ctx)) == "Default allow LAN to any rule"
|
||||
wan = run(ctx, src=INTERNET, dst=WAN_IP, dport="22", iface="wan")
|
||||
assert wan["action"] == "block" and decided_by(wan) == "not-lan"
|
||||
|
||||
|
||||
def test_06_group_rule_comes_before_interface_rules():
|
||||
ctx = build([mvc("grp-block", prio=300005, seq=900, action="block", interface="office")], groups={"office": ["lan"]})
|
||||
r = run(ctx)
|
||||
assert r["action"] == "block" and decided_by(r) == "grp-block" and r["decisive_rule"]["section"] == "group"
|
||||
|
||||
|
||||
def test_07_last_non_quick_match_wins():
|
||||
ctx = build([mvc("first-pass", seq=1, quick="0", interface="wan"), mvc("then-block", seq=2, action="block", quick="0", interface="wan")],
|
||||
drop=("Allow ALL",))
|
||||
r = run(ctx, src=INTERNET, dst=WAN_IP, iface="wan")
|
||||
assert r["action"] == "block" and decided_by(r) == "then-block" and r["confidence"] == "sure"
|
||||
|
||||
|
||||
def test_08_negated_source_alias():
|
||||
aliases = {"BADHOSTS": {"type": "host", "content": "10.20.30.66"}}
|
||||
rule = mvc("not-bad", seq=1, action="block", src="BADHOSTS", source_not="1")
|
||||
ctx = build([rule], aliases=aliases, refs={LAN_CLIENT: ["__lan_network"], "10.20.30.66": ["BADHOSTS", "__lan_network"]})
|
||||
assert run(ctx)["action"] == "block" # .50 is not in BADHOSTS -> !BADHOSTS matches
|
||||
assert run(ctx, src="10.20.30.66")["action"] == "pass" # in BADHOSTS -> negation does not match
|
||||
# Table membership unknown: the negation stays unknown, the verdict is not "sure".
|
||||
unknown = build([dict(rule, source_net="FEED")], aliases={"FEED": {"type": "urltable", "content": ""}},
|
||||
refs={LAN_CLIENT: None})
|
||||
r = run(unknown)
|
||||
assert r["action"] == "pass" and r["confidence"] == "unknown" and "ALIAS_TABLE_UNAVAILABLE" in r["unknowns"]
|
||||
|
||||
|
||||
def test_14_rule_not_loaded_is_skipped():
|
||||
ctx = build([mvc("scheduled", seq=1, action="block", sched="weekends", loaded=False)])
|
||||
r = run(ctx)
|
||||
assert decided_by(r) == "Default allow LAN to any rule"
|
||||
step = next(s for s in r["trace"][0]["steps"] if s["rule"]["description"] == "scheduled")
|
||||
assert step["effect"] == "not_loaded" and "RULES_NOT_LOADED" in r["notes"]
|
||||
|
||||
|
||||
def test_15_16_gateway_and_stateless_notes():
|
||||
r = run(build([mvc("via-gw", seq=1, gateway="WAN_GW", statetype="none")]))
|
||||
assert decided_by(r) == "via-gw" and {"POLICY_ROUTE", "STATELESS"} <= set(r["notes"])
|
||||
assert r["confidence"] == "sure"
|
||||
|
||||
|
||||
def test_17_tagged_rule_is_never_sure():
|
||||
r = run(build([mvc("tagged-block", seq=1, action="block", tagged="FROM_NAT")]))
|
||||
assert r["confidence"] == "unknown" and "TAGGED" in r["unknowns"]
|
||||
|
||||
|
||||
def test_18_unknown_rule_with_same_action_keeps_the_verdict_sure():
|
||||
ctx = build([mvc("maybe-pass", seq=1, src="GHOST")]) # alias unknown on the firewall
|
||||
r = run(ctx)
|
||||
assert r["action"] == "pass" and r["confidence"] == "sure" and r["rule_confidence"] == "probable"
|
||||
|
||||
|
||||
def test_19_unknown_rule_with_other_action_makes_the_verdict_unknown():
|
||||
ctx = build([mvc("maybe-block", seq=1, action="block", src="GHOST")])
|
||||
r = run(ctx)
|
||||
assert r["action"] == "pass" and r["confidence"] == "unknown" and "ALIAS_MISSING" in r["unknowns"]
|
||||
assert any(s["effect"] == "possible" for s in r["trace"][0]["steps"])
|
||||
|
||||
|
||||
def test_20_rule_with_missing_interface_never_matches():
|
||||
row = mvc("orphan", prio=600000, action="pass", interface="opt9", loaded=False)
|
||||
r = run(build([row], drop=("Allow ALL",)), src=INTERNET, dst=WAN_IP, iface="wan")
|
||||
assert decided_by(r) == "Default deny / state violation rule"
|
||||
assert next(s for s in r["trace"][0]["steps"] if s["rule"]["description"] == "orphan")["effect"] == "not_loaded"
|
||||
# Even without a pf ruleset a 600000 rule counts as not loaded; the verdict is no longer sure.
|
||||
no_pf = run(build([row], pf=False))
|
||||
assert decided_by(no_pf) == "Default allow LAN to any rule" and no_pf["confidence"] != "sure"
|
||||
|
||||
|
||||
def test_guessed_interface_and_nat_nearby_lower_confidence():
|
||||
r = run(build(), iface=None)
|
||||
assert r["legs"][0]["interface"] == "lan" and "IFACE_GUESSED" in r["notes"] and r["confidence"] == "probable"
|
||||
nat = run(build(port_forwards=[{"interface": "lan", "protocol": "tcp", "destination.port": "443"}]))
|
||||
assert "NAT_NEARBY" in nat["notes"] and nat["confidence"] == "probable"
|
||||
# "no rdr" exclusions and forwards for other ports do not count.
|
||||
other = run(build(port_forwards=[{"interface": "lan", "protocol": "tcp", "destination.port": "22", "nordr": "0"},
|
||||
{"interface": "lan", "protocol": "tcp", "destination.port": "443", "nordr": "1"}]))
|
||||
assert "NAT_NEARBY" not in other["notes"] and other["confidence"] == "sure"
|
||||
|
||||
|
||||
def test_egress_leg_blocks_when_out_rule_blocks():
|
||||
ctx = build([mvc("no-out", prio=200000, action="block", interface="wan", direction="out")])
|
||||
r = run(ctx, egress="wan")
|
||||
assert [leg["action"] for leg in r["legs"]] == ["pass", "block"]
|
||||
assert r["action"] == "block" and decided_by(r) == "no-out" and r["confidence"] == "sure"
|
||||
# Without the block the automatic "let out anything" rule passes the out leg.
|
||||
ok = run(build(), egress="wan")
|
||||
assert ok["action"] == "pass" and ok["legs"][1]["decisive_rule"]["description"] == "let out anything from firewall host itself"
|
||||
|
||||
|
||||
# ── Resolver: matrix 9-13 ─────────────────────────────────────────────────
|
||||
|
||||
def _ip(text):
|
||||
return ipaddress.ip_address(text)
|
||||
|
||||
|
||||
def test_09_nested_aliases_and_cycles():
|
||||
ctx = build(aliases={
|
||||
"OUTER": {"type": "host", "content": "MIDDLE\n192.0.2.9"},
|
||||
"MIDDLE": {"type": "network", "content": "10.20.30.0/25\nINNER"},
|
||||
"INNER": {"type": "host", "content": "10.20.30.200"},
|
||||
"LOOP_A": {"type": "host", "content": "LOOP_B"}, "LOOP_B": {"type": "host", "content": "LOOP_A"},
|
||||
"NAMED": {"type": "host", "content": "server.example.org"},
|
||||
})
|
||||
ctx.refs = {}
|
||||
assert resolve.alias_contains("OUTER", _ip("10.20.30.5"), ctx).value is True
|
||||
assert resolve.alias_contains("OUTER", _ip("10.20.30.200"), ctx).value is True
|
||||
assert resolve.alias_contains("OUTER", _ip("10.20.30.150"), ctx).value is False
|
||||
loop = resolve.alias_contains("LOOP_A", _ip("10.20.30.5"), ctx)
|
||||
assert loop.value is None and "ALIAS_CYCLE" in loop.reasons
|
||||
fqdn = resolve.alias_contains("NAMED", _ip("10.20.30.5"), ctx)
|
||||
assert fqdn.value is None and "FQDN_UNRESOLVED" in fqdn.reasons
|
||||
assert resolve.alias_contains("NOPE", _ip("10.20.30.5"), ctx).value is None
|
||||
|
||||
|
||||
def test_10_table_aliases_need_find_references():
|
||||
ctx = build(aliases={"FEED": {"type": "urltable", "content": "https://example.org/list"}})
|
||||
ctx.refs = {"192.0.2.7": ["FEED"], "192.0.2.8": []}
|
||||
assert resolve.alias_contains("FEED", _ip("192.0.2.7"), ctx).value is True
|
||||
assert resolve.alias_contains("FEED", _ip("192.0.2.8"), ctx).value is False
|
||||
ctx.refs = {"192.0.2.7": None}
|
||||
out = resolve.alias_contains("FEED", _ip("192.0.2.7"), ctx)
|
||||
assert out.value is None and out.reasons == ["ALIAS_TABLE_UNAVAILABLE"]
|
||||
|
||||
|
||||
def test_11_address_family():
|
||||
ctx = build([mvc("v4-only", seq=1, action="block", interface="wan", ipprotocol="inet")], drop=("Allow ALL",))
|
||||
v6 = run(ctx, src="2001:db8::5", dst="2001:db8:1::1", iface="wan")
|
||||
assert decided_by(v6) == "Default deny / state violation rule"
|
||||
both = build([mvc("v46", seq=1, action="reject", interface="wan", ipprotocol="inet46")], drop=("Allow ALL",))
|
||||
assert decided_by(run(both, src="2001:db8::5", dst="2001:db8:1::1", iface="wan")) == "v46"
|
||||
|
||||
|
||||
def test_12_port_ranges_protocol_pairs_and_nested_port_aliases():
|
||||
ctx = build(aliases={"WEB": {"type": "port", "content": "1000:2000\nMORE"}, "MORE": {"type": "port", "content": "https\n8000-8001"}})
|
||||
for port, expected in ((999, False), (1000, True), (2000, True), (2001, False), (443, True), (8001, True), (8002, False)):
|
||||
assert resolve.port_match("WEB", port, ctx).value is expected, port
|
||||
assert resolve.port_match("1000:2000", 1500, ctx).value is True
|
||||
rule = mvc("tcpudp", seq=1, action="block", proto="TCP/UDP", dport="1000-2000")
|
||||
assert decided_by(run(build([rule]), proto="udp", dport="1000")) == "tcpudp"
|
||||
assert decided_by(run(build([rule]), proto="tcp", dport="2001")) == "Default allow LAN to any rule"
|
||||
# No source port given: assumed ephemeral, so "port 0" rules do not match, a narrow one is unknown.
|
||||
assert resolve.port_match("0", None, ctx, ephemeral_default=True).value is False
|
||||
assert resolve.port_match("5000", None, ctx, ephemeral_default=True).value is None
|
||||
assert resolve.port_match("1024-65535", None, ctx, ephemeral_default=True).value is True
|
||||
|
||||
|
||||
def test_13_interface_networks_addresses_self_and_carp_vip():
|
||||
ctx = build()
|
||||
vip, primary, client = _ip("10.20.30.1"), _ip("10.20.30.2"), _ip("10.20.30.50")
|
||||
assert resolve.addr_item("lan", vip, ctx).value is True and resolve.addr_item("lan", client, ctx).value is True
|
||||
assert resolve.addr_item("lanip", primary, ctx).value is True
|
||||
assert resolve.addr_item("lanip", client, ctx).value is False
|
||||
assert resolve.addr_item("lanip", vip, ctx).value is None # VIP vs interface address: not provable
|
||||
assert resolve.addr_item("(self)", vip, ctx).value is True
|
||||
assert resolve.addr_item("(self)", _ip(WAN_IP), ctx).value is True
|
||||
assert resolve.addr_item("(self)", client, ctx).value is False
|
||||
assert resolve.addr_item("__wan_network", _ip("203.0.113.9"), ctx).value is True
|
||||
|
||||
|
||||
def test_query_validation_errors():
|
||||
from fastapi import HTTPException
|
||||
for kwargs, code in (({"src": "host.example", "dst": INTERNET}, "ANALYSE_ADRESSE_UNGUELTIG"),
|
||||
({"src": "2001:db8::1", "dst": INTERNET}, "ANALYSE_FAMILIEN_MISCHUNG"),
|
||||
({"src": LAN_CLIENT, "dst": INTERNET, "dport": "70000"}, "ANALYSE_PARAMETER_UNGUELTIG"),
|
||||
({"src": LAN_CLIENT, "dst": INTERNET, "proto": "icmp", "dport": "80"}, "ANALYSE_PARAMETER_UNGUELTIG")):
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
build_query(**kwargs)
|
||||
assert exc.value.detail["code"] == code
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Rule analysis API + MCP tool (matrix case 25): GET only, read tokens allowed, nothing written."""
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.rule_analysis import context as ctx_mod
|
||||
from tests.test_api_tokens import _bearer, _make
|
||||
from tests.test_rule_analysis import FIXTURE
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
FW, DOWN = "https://198.51.100.200", "https://198.51.100.201"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_cache():
|
||||
ctx_mod.clear_cache()
|
||||
yield
|
||||
ctx_mod.clear_cache()
|
||||
|
||||
|
||||
def _mock_fw(respx_mock, base=FW):
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
respx_mock.get(f"{base}/api/firewall/filter/searchRule").mock(return_value=httpx.Response(200, json=FIXTURE["rules"]))
|
||||
respx_mock.get(f"{base}/api/diagnostics/firewall/pfStatistics/rules").mock(return_value=httpx.Response(200, json=FIXTURE["pf"]))
|
||||
respx_mock.get(f"{base}/api/interfaces/overview/interfacesInfo").mock(return_value=httpx.Response(200, json=FIXTURE["interfaces"]))
|
||||
respx_mock.get(f"{base}/api/firewall/group/searchItem").mock(return_value=httpx.Response(200, json=FIXTURE["groups"]))
|
||||
respx_mock.get(f"{base}/api/firewall/alias/searchItem").mock(return_value=httpx.Response(200, json=FIXTURE["aliases"]))
|
||||
respx_mock.get(f"{base}/api/firewall/d_nat/searchRule").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
return respx_mock.post(f"{base}/api/firewall/alias_util/findReferences").mock(
|
||||
return_value=httpx.Response(200, json={"status": "ok", "matches": []}))
|
||||
|
||||
|
||||
Q = {"src": "10.20.30.50", "dst": "198.51.100.10", "proto": "tcp", "dport": "443", "iface": "lan"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_host_match_and_validation(client, respx_mock):
|
||||
refs = _mock_fw(respx_mock)
|
||||
host = await _create_host(client, "fw-a", FW)
|
||||
resp = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params=Q)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["action"] == "pass" and body["confidence"] == "sure" and body["error"] is None
|
||||
assert body["decisive_rule"]["description"] == "Default allow LAN to any rule"
|
||||
assert body["legs"][0]["interface"] == "lan" and body["trace"][0]["steps"]
|
||||
assert refs.call_count == 2 and json.loads(refs.calls[0].request.content) == {"ip": "10.20.30.50"}
|
||||
bad = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params={**Q, "src": "pc.example"})
|
||||
assert bad.status_code == 422 and bad.json()["detail"]["code"] == "ANALYSE_ADRESSE_UNGUELTIG"
|
||||
mixed = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params={**Q, "dst": "2001:db8::1"})
|
||||
assert mixed.json()["detail"]["code"] == "ANALYSE_FAMILIEN_MISCHUNG"
|
||||
iface = await client.get(f"/api/hosts/{host['id']}/rule-analysis/match", params={**Q, "iface": "opt7"})
|
||||
assert iface.status_code == 422 and iface.json()["detail"]["code"] == "ANALYSE_INTERFACE_UNBEKANNT"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fleet_match_reports_unreachable_hosts_per_host(client, respx_mock):
|
||||
_mock_fw(respx_mock)
|
||||
respx_mock.route(url__startswith=DOWN).mock(side_effect=httpx.ConnectError("down"))
|
||||
await _create_host(client, "fw-a", FW)
|
||||
await _create_host(client, "fw-down", DOWN)
|
||||
resp = await client.get("/api/rule-analysis/match", params={**Q, "iface": "auto"})
|
||||
assert resp.status_code == 200, resp.text
|
||||
by_name = {r["host_name"]: r for r in resp.json()}
|
||||
assert by_name["fw-a"]["action"] == "pass" and "IFACE_GUESSED" in by_name["fw-a"]["notes"]
|
||||
assert by_name["fw-a"]["confidence"] == "probable"
|
||||
assert by_name["fw-down"]["error"]["code"] == "ANALYSE_FIREWALL_NICHT_LESBAR"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_25_mcp_rule_match_with_a_read_token_changes_nothing(client, respx_mock):
|
||||
_mock_fw(respx_mock)
|
||||
host = await _create_host(client, "fw-a", FW)
|
||||
token = await _make(client, "read")
|
||||
async with _bearer(token["token"]) as api:
|
||||
direct = await api.get(f"/api/hosts/{host['id']}/rule-analysis/match", params=Q)
|
||||
assert direct.status_code == 200
|
||||
rpc = {"jsonrpc": "2.0", "id": 1, "method": "tools/call",
|
||||
"params": {"name": "rule_match", "arguments": {**Q, "dport": 443, "host_ids": host["id"]}}}
|
||||
result = (await api.post("/api/mcp", json=rpc)).json()["result"]
|
||||
tools = (await api.post("/api/mcp", json={"jsonrpc": "2.0", "id": 2, "method": "tools/list"})).json()["result"]["tools"]
|
||||
assert result["isError"] is False
|
||||
payload = json.loads(result["content"][0]["text"])
|
||||
assert payload[0]["action"] == "pass" and "trace" not in payload[0]
|
||||
assert "rule_match" in {t["name"] for t in tools}
|
||||
# The only POST that reached the firewall is the read-only pf table lookup.
|
||||
posts = {c.request.url.path for c in respx_mock.calls if c.request.method != "GET"}
|
||||
assert posts == {"/api/firewall/alias_util/findReferences"}
|
||||
@@ -0,0 +1,306 @@
|
||||
"""auto_verify (1.25): an unattended action checks its own result and closes the operation,
|
||||
so the host/cluster is not left locked until a human clicks "Wartung abschließen"."""
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models import HostOperation
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import scheduled_maintenance_service as svc
|
||||
from tests.test_rules import _create_host
|
||||
from tests.test_scheduled_handover import FakeCluster, _setup
|
||||
|
||||
H = "https://fw-solo.test"
|
||||
|
||||
|
||||
class FakeHost:
|
||||
"""One stand-alone firewall: job = ready | running | down; boot = what systemTime reports."""
|
||||
|
||||
def __init__(self, respx_mock, *, firmware="update", reboot_flag="1", version="26.7.5"):
|
||||
self.job, self.boot, self.version = "ready", "Mon Oct 5 10:00:00 UTC 2026", version
|
||||
self.update = respx_mock.post(f"{H}/api/core/firmware/update").respond(200, json={"status": "ok"})
|
||||
self.reboot = respx_mock.post(f"{H}/api/core/system/reboot").respond(200, json={"status": "ok"})
|
||||
respx_mock.get(f"{H}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
|
||||
respx_mock.get(f"{H}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
|
||||
respx_mock.get(f"{H}/api/core/firmware/info").mock(side_effect=lambda r: self._up(json={"product_version": self.version}))
|
||||
respx_mock.get(f"{H}/api/core/firmware/status").respond(
|
||||
200, json={"status": firmware, "all_packages": {}, "status_msg": "x", "status_reboot": reboot_flag})
|
||||
respx_mock.get(f"{H}/api/core/firmware/running").mock(side_effect=lambda r: self._up(json={"status": self.job}))
|
||||
respx_mock.get(f"{H}/api/diagnostics/system/systemTime").mock(side_effect=lambda r: self._up(json={"boottime": self.boot}))
|
||||
|
||||
def _up(self, **kw):
|
||||
if self.job == "down":
|
||||
raise httpx.ConnectError("rebooting")
|
||||
return httpx.Response(200, **kw)
|
||||
|
||||
def _vip(self, request):
|
||||
return self._up(json={"rows": [], "carp": {"allow": "1", "maintenancemode": False, "demotion": "0", "status_msg": ""}})
|
||||
|
||||
def reboot_now(self, new_boot="Wed Oct 21 03:00:00 UTC 2026", version=None):
|
||||
self.boot = new_boot
|
||||
if version:
|
||||
self.version = version
|
||||
|
||||
|
||||
async def _plan(host, action="update", *, auto_verify=True, handover=False):
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(
|
||||
host_id=uuid.UUID(host["id"]), host_name=host["name"], action=action, carp_handover=handover,
|
||||
auto_verify=auto_verify, run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin",
|
||||
created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
return s.id
|
||||
|
||||
|
||||
async def _fire():
|
||||
async with SessionLocal() as db:
|
||||
return await svc.fire_due(db)
|
||||
|
||||
|
||||
async def _advance():
|
||||
async with SessionLocal() as db:
|
||||
return await svc.advance_running(db)
|
||||
|
||||
|
||||
async def _get(sid):
|
||||
async with SessionLocal() as db:
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
op = await db.get(HostOperation, s.host_operation_id) if s.host_operation_id else None
|
||||
return s, op
|
||||
|
||||
|
||||
async def _shift(sid, *, phase=None, ready=None):
|
||||
"""Pretend the phase began / the firmware turned ready that long ago."""
|
||||
async with SessionLocal() as db:
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
if phase is not None:
|
||||
s.phase_since = datetime.now(timezone.utc) - phase
|
||||
if ready is not None:
|
||||
s.progress = {**s.progress, "ready_since": (datetime.now(timezone.utc) - ready).isoformat()}
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _settle(sid):
|
||||
"""verify needs the firmware idle for VERIFY_SETTLE: one tick to notice, then pretend it has been a while."""
|
||||
await _advance()
|
||||
await _shift(sid, ready=svc.VERIFY_SETTLE + timedelta(seconds=1))
|
||||
await _advance()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_is_verified_and_the_operation_closed(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
s, op = await _get(sid)
|
||||
assert (s.status, s.phase, op.status, fw.update.call_count) == ("running", "job_wait", "submitted", 1)
|
||||
assert s.progress["reboot_expected"] is True and s.progress["boot_before"] == fw.boot
|
||||
|
||||
fw.job = "running"
|
||||
await _advance()
|
||||
fw.job = "down"
|
||||
await _advance()
|
||||
fw.job = "ready"
|
||||
fw.reboot_now(version="26.7.7")
|
||||
await _advance()
|
||||
assert (await _get(sid))[0].phase == "verify"
|
||||
await _settle(sid)
|
||||
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "succeeded" and s.phase is None
|
||||
assert s.result["auto_verified"] is True and s.result["version_after"] == "26.7.7" and s.result["version_before"] == "26.7.5"
|
||||
assert op.status == "acknowledged" and "Automatisch geprüft" in op.message
|
||||
assert fw.update.call_count == 1
|
||||
# the host is free again: a new preview is not blocked by a forgotten operation
|
||||
resp = await client.get(f"/api/hosts/{host['id']}/maintenance")
|
||||
assert not any("frühere Wartungsaktion" in b for b in resp.json()["state"]["blocked"])
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
|
||||
assert {e.actor for e in events if (e.details or {}).get("scheduled")} == {"admin (geplant)"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_boot_time_proves_a_reboot_the_poll_never_saw(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
await _shift(sid, phase=timedelta(0))
|
||||
fw.reboot_now() # rebooted between two polls: never seen offline
|
||||
async with SessionLocal() as db: # the job grace has passed
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
|
||||
await db.commit()
|
||||
await _advance()
|
||||
assert (await _get(sid))[0].phase == "verify"
|
||||
await _settle(sid)
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "succeeded" and s.result["rebooted"] is True and op.status == "acknowledged"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_expected_reboot_never_happened_ends_unknown_after_timeout(client, respx_mock):
|
||||
FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
async with SessionLocal() as db:
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
|
||||
await db.commit()
|
||||
await _advance()
|
||||
await _settle(sid)
|
||||
s, op = await _get(sid)
|
||||
assert (s.status, s.phase, op.status) == ("running", "verify", "submitted") # waiting, not done
|
||||
assert "Neustart" in s.progress["waiting_for"]
|
||||
await _shift(sid, phase=svc.VERIFY_TIMEOUT)
|
||||
await _advance()
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "failed" and op.status == "unknown" and "nicht bestanden" in s.result["error"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ready_must_hold_a_while_a_short_ready_before_the_reboot_does_not_count(client, respx_mock):
|
||||
fw = FakeHost(respx_mock, reboot_flag="0")
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
fw.job = "running"
|
||||
await _advance()
|
||||
fw.job = "ready"
|
||||
await _advance() # job_wait -> verify
|
||||
await _advance() # ready noticed
|
||||
s, _ = await _get(sid)
|
||||
assert s.phase == "verify" and s.progress["ready_since"]
|
||||
fw.job = "running" # busy again (e.g. the reboot at the end of the update)
|
||||
await _advance()
|
||||
assert (await _get(sid))[0].progress["ready_since"] is None
|
||||
fw.job = "ready"
|
||||
await _settle(sid)
|
||||
assert (await _get(sid))[0].status == "succeeded"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_without_auto_verify_nothing_changes(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host, auto_verify=False)
|
||||
await _fire()
|
||||
s, op = await _get(sid)
|
||||
assert (s.status, s.phase, op.status, fw.update.call_count) == ("succeeded", None, "submitted", 1)
|
||||
assert await _advance() == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_human_closing_the_operation_stops_the_check(client, respx_mock):
|
||||
FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
s, op = await _get(sid)
|
||||
async with SessionLocal() as db:
|
||||
(await db.get(HostOperation, op.id)).status = "cancelled"
|
||||
await db.commit()
|
||||
await _advance()
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "failed" and "von Hand beendet" in s.result["error"] and op.status == "cancelled"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handover_verifies_before_taking_the_traffic_back(client, respx_mock):
|
||||
cluster = FakeCluster(respx_mock)
|
||||
cluster.boot = "before"
|
||||
|
||||
def system_time(request):
|
||||
if cluster.job == "down":
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, json={"boottime": cluster.boot})
|
||||
respx_mock.get("https://fw-a.test/api/diagnostics/system/systemTime").mock(side_effect=system_time)
|
||||
a, _b = await _setup(client)
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True, auto_verify=True,
|
||||
run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
sid = s.id
|
||||
await _fire()
|
||||
await _advance() # partner MASTER -> job submitted
|
||||
cluster.job = "down"
|
||||
await _advance()
|
||||
cluster.job = "ready"
|
||||
cluster.boot = "after"
|
||||
await _advance()
|
||||
s, _ = await _get(sid)
|
||||
assert s.phase == "verify" and cluster.toggles.call_count == 1 # still demoted while checking
|
||||
await _settle(sid)
|
||||
assert (await _get(sid))[0].phase == "leave" and cluster.toggles.call_count == 1
|
||||
await _advance() # leave
|
||||
await _advance() # leave_wait -> done
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "succeeded" and s.result["handover"] is True and s.result["auto_verified"] is True
|
||||
assert s.result["rebooted"] is True # found live: the handover path dropped what verify had seen
|
||||
assert op.status == "acknowledged" and cluster.toggles.call_count == 2 and cluster.update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_due_row_locked_by_another_backend_is_not_fired_twice(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
async with SessionLocal() as other: # another backend is firing this very row right now
|
||||
await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
|
||||
# Without the row lock this would try to update the locked row and hang: bounded, so it fails instead.
|
||||
result = await asyncio.wait_for(_fire(), timeout=10)
|
||||
assert result["due"] == 0 and fw.update.call_count == 0
|
||||
await other.rollback()
|
||||
await _fire()
|
||||
assert fw.update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_running_row_locked_by_another_backend_is_not_stepped_twice(client, respx_mock):
|
||||
cluster = FakeCluster(respx_mock)
|
||||
a, _b = await _setup(client)
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True,
|
||||
run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
sid = s.id
|
||||
await _fire()
|
||||
async with SessionLocal() as other:
|
||||
await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
|
||||
assert await asyncio.wait_for(_advance(), timeout=10) == 1 # still counted as running, but not touched
|
||||
assert cluster.update.call_count == 0
|
||||
await other.rollback()
|
||||
await _advance()
|
||||
assert cluster.update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancel_waits_for_a_firing_row_and_never_overwrites_running(client, respx_mock):
|
||||
FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(host_id=uuid.UUID(host["id"]), host_name="fw-solo", action="update", auto_verify=True,
|
||||
run_at=datetime.now(timezone.utc) + timedelta(hours=1), actor="admin", created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
sid = s.id
|
||||
async with SessionLocal() as firing: # the poll loop has the row and is about to mark it running
|
||||
row = await firing.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
|
||||
cancel = asyncio.create_task(client.delete(f"/api/hosts/{host['id']}/maintenance/schedule/{sid}"))
|
||||
await asyncio.sleep(0.5)
|
||||
row.status = "running"
|
||||
await firing.commit()
|
||||
resp = await asyncio.wait_for(cancel, timeout=10)
|
||||
assert resp.status_code == 409
|
||||
assert (await _get(sid))[0].status == "running"
|
||||
@@ -0,0 +1,137 @@
|
||||
"""1.27 prerequisite hardening: __Host- session cookie (with soft migration) and
|
||||
the Sec-Fetch-Site check for cookie-authenticated /api requests."""
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app import main
|
||||
from app.config import settings
|
||||
from app.main import app
|
||||
from app.routers.auth import COOKIE_NAME, SECURE_COOKIE_NAME
|
||||
|
||||
PW = "tokpass-12345678"
|
||||
|
||||
|
||||
def _https_client() -> AsyncClient:
|
||||
return AsyncClient(transport=ASGITransport(app=app), base_url="https://test")
|
||||
|
||||
|
||||
async def _login(ac: AsyncClient) -> None:
|
||||
resp = await ac.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
|
||||
def _set_cookies(resp) -> list[str]:
|
||||
return resp.headers.get_list("set-cookie")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_secure_login_sets_host_prefixed_cookie_and_drops_legacy(monkeypatch):
|
||||
monkeypatch.setattr(settings, "cookie_secure", True)
|
||||
async with _https_client() as ac:
|
||||
resp = await ac.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
|
||||
assert resp.status_code == 200
|
||||
cookies = _set_cookies(resp)
|
||||
new = [c for c in cookies if c.startswith(f"{SECURE_COOKIE_NAME}=")]
|
||||
assert new and "Secure" in new[0] and "Path=/" in new[0] and "HttpOnly" in new[0] and "Domain" not in new[0]
|
||||
assert any(c.startswith(f'{COOKIE_NAME}=""') or c.startswith(f"{COOKIE_NAME}=;") or "Max-Age=0" in c
|
||||
for c in cookies if c.startswith(f"{COOKIE_NAME}="))
|
||||
assert (await ac.get("/api/hosts")).status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_plain_name_cookie_without_https():
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
|
||||
resp = await ac.post("/api/auth/login", json={"username": settings.admin_username, "password": settings.admin_password})
|
||||
assert any(c.startswith(f"{COOKIE_NAME}=") for c in _set_cookies(resp))
|
||||
assert not any(c.startswith(SECURE_COOKIE_NAME) for c in _set_cookies(resp))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legacy_cookie_is_migrated_not_logged_out(monkeypatch):
|
||||
# A session issued before the upgrade (plain cookie name) ...
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
|
||||
await _login(ac)
|
||||
legacy = ac.cookies.get(COOKIE_NAME)
|
||||
assert legacy
|
||||
monkeypatch.setattr(settings, "cookie_secure", True)
|
||||
async with _https_client() as ac:
|
||||
ac.cookies.set(COOKIE_NAME, legacy)
|
||||
assert (await ac.get("/api/hosts")).status_code == 200 # still accepted
|
||||
me = await ac.get("/api/auth/me")
|
||||
assert me.json()["authenticated"] is True
|
||||
moved = [c for c in _set_cookies(me) if c.startswith(f"{SECURE_COOKIE_NAME}={legacy}")]
|
||||
assert moved and "Secure" in moved[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_host_cookie_wins_over_tossed_plain_cookie(monkeypatch):
|
||||
monkeypatch.setattr(settings, "cookie_secure", True)
|
||||
async with _https_client() as ac:
|
||||
await _login(ac)
|
||||
real = ac.cookies.get(SECURE_COOKIE_NAME)
|
||||
async with _https_client() as ac:
|
||||
# A script on another port may set a plain-name cookie; it is ignored ...
|
||||
ac.cookies.set(SECURE_COOKIE_NAME, real)
|
||||
ac.cookies.set(COOKIE_NAME, "attacker-token")
|
||||
assert (await ac.get("/api/hosts")).status_code == 200
|
||||
async with _https_client() as ac:
|
||||
# ... and a bad __Host- cookie never falls back to the plain one.
|
||||
ac.cookies.set(SECURE_COOKIE_NAME, "garbage")
|
||||
ac.cookies.set(COOKIE_NAME, real)
|
||||
assert (await ac.get("/api/hosts")).status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("site,expected", [
|
||||
("same-site", 403), ("cross-site", 403), ("Cross-Site", 403),
|
||||
("same-origin", 200), ("none", 200), (None, 200),
|
||||
])
|
||||
async def test_sec_fetch_site_on_cookie_requests(client, site, expected):
|
||||
headers = {"Sec-Fetch-Site": site} if site else {}
|
||||
resp = await client.get("/api/hosts", headers=headers)
|
||||
assert resp.status_code == expected, resp.text
|
||||
if expected == 403:
|
||||
assert resp.json()["detail"]["code"] == "CROSS_SITE_REQUEST_BLOCKED"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sec_fetch_same_site_allowed_only_for_configured_frontend(client, monkeypatch):
|
||||
monkeypatch.setattr(settings, "frontend_origin", "https://opnexus.example:3100")
|
||||
ok = await client.get("/api/hosts", headers={"Sec-Fetch-Site": "same-site", "Origin": "https://opnexus.example:3100"})
|
||||
assert ok.status_code == 200
|
||||
slot = await client.get("/api/hosts", headers={"Sec-Fetch-Site": "same-site", "Origin": "https://opnexus.example:9441"})
|
||||
assert slot.status_code == 403
|
||||
# Writes from a slot page are refused as well (before auth even runs).
|
||||
write = await client.post("/api/auth/logout", headers={"Sec-Fetch-Site": "same-site", "Origin": "https://opnexus.example:9441"})
|
||||
assert write.status_code == 403
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sec_fetch_does_not_touch_token_requests(client, anon_client):
|
||||
created = (await client.post("/api/tokens", json={"name": "sfs", "scope": "read", "current_password": settings.admin_password})).json()
|
||||
resp = await anon_client.get("/api/hosts", headers={"Authorization": f"Bearer {created['token']}", "Sec-Fetch-Site": "cross-site"})
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sec_fetch_exempts_sso_callback(client):
|
||||
resp = await client.get("/api/auth/oidc/callback?code=x&state=y", headers={"Sec-Fetch-Site": "cross-site"}, follow_redirects=False)
|
||||
assert resp.status_code != 403 or resp.json().get("detail", {}).get("code") != "CROSS_SITE_REQUEST_BLOCKED"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_startup_refused_when_slot_origin_is_a_frontend_origin(monkeypatch):
|
||||
monkeypatch.setattr(settings, "frontend_origin", "https://opnexus.example, https://opnexus.example:9441")
|
||||
monkeypatch.setattr(settings, "opnexus_gui_sites", "https://opnexus.example:9441,https://opnexus.example:9442")
|
||||
assert settings.gui_origin_conflicts() == ["https://opnexus.example:9441"]
|
||||
called = []
|
||||
|
||||
async def _no_migrate():
|
||||
called.append(1)
|
||||
monkeypatch.setattr(main.migrate, "run", _no_migrate)
|
||||
with pytest.raises(RuntimeError):
|
||||
async with main.lifespan(app):
|
||||
pass
|
||||
assert not called
|
||||
monkeypatch.setattr(settings, "opnexus_gui_sites", "https://opnexus.example:9442")
|
||||
assert settings.gui_origin_conflicts() == []
|
||||
@@ -47,6 +47,7 @@ async def test_status_reports_update_and_can_update(client, respx_mock):
|
||||
body = (await client.get("/api/system/updater")).json()
|
||||
assert body["available"] is True and body["can_update"] is True
|
||||
assert body["latest"] == _bump(minor=1) and body["major_change"] is False
|
||||
assert body["active_rollouts"] == 0 # the UI warns before restarting OPNexus in the middle of a rollout
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
-----BEGIN PUBLIC KEY-----
|
||||
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEA4pZWkNMkAtCGc54M2wuWy7CXwkT
|
||||
3PCysv1/1IbkyJRPdM8+DFf26fBxWG9P9PLKfwavhqVYtHILxxzxKmtjeQ==
|
||||
-----END PUBLIC KEY-----
|
||||
@@ -124,6 +124,10 @@ services:
|
||||
# Optionaler Updater-Dienst (Profil "updater", siehe unten). Ohne ihn
|
||||
# zeigt das Webinterface nur an, wie man ihn aktiviert.
|
||||
UPDATER_URL: ${UPDATER_URL:-http://updater:8080}
|
||||
# Optionaler GUI-Fernzugriff (Profil "gui", siehe gui-gateway unten).
|
||||
# Leer = aus; das Backend lässt den Schalter dann nicht einschalten.
|
||||
OPNEXUS_GUI_SITES: ${OPNEXUS_GUI_SITES:-}
|
||||
GUI_GATEWAY_TOKEN: ${GUI_GATEWAY_TOKEN:-}
|
||||
volumes:
|
||||
- secrets:/data/opnsense-secrets
|
||||
- backups:/data/opnsense-backups
|
||||
@@ -197,9 +201,19 @@ services:
|
||||
environment:
|
||||
OPNEXUS_SITES: ${OPNEXUS_SITES:-localhost}
|
||||
OPNEXUS_DEFAULT_SNI: ${OPNEXUS_DEFAULT_SNI:-localhost}
|
||||
OPNEXUS_GUI_SITES: ${OPNEXUS_GUI_SITES:-https://localhost:9441}
|
||||
ports:
|
||||
- "${PROXY_BIND_ADDRESS:-0.0.0.0}:${PROXY_HTTPS_PORT:-443}:443"
|
||||
- "${PROXY_BIND_ADDRESS:-0.0.0.0}:${PROXY_HTTP_PORT:-80}:80"
|
||||
# Slot-Ports des GUI-Fernzugriffs (Profil "gui"). Standardmäßig nur auf
|
||||
# 127.0.0.1; ohne den gui-gateway-Dienst antworten sie nur mit 502.
|
||||
# Host- und Container-Port sind absichtlich gleich: die Ports stehen so
|
||||
# auch in OPNEXUS_GUI_SITES, und der Proxy meldet dem Gateway den Port,
|
||||
# auf dem die Verbindung ankam.
|
||||
# Ohne GUI_PORTS belegt Docker zufällige freie Host-Ports: ein Update darf
|
||||
# den Proxy nicht daran scheitern lassen, dass 9441-9444 schon vergeben
|
||||
# sind (z. B. Portainer auf 0.0.0.0:9443). Zum Einschalten GUI_PORTS setzen.
|
||||
- "${GUI_BIND_ADDRESS:-127.0.0.1}:${GUI_PORTS:-}:${GUI_PORTS:-9441-9444}"
|
||||
volumes:
|
||||
- caddy_data:/data
|
||||
- caddy_config:/config
|
||||
@@ -228,6 +242,38 @@ services:
|
||||
ports:
|
||||
- "${OVERLAY_PORT:-51820}:51820/udp"
|
||||
|
||||
# ── Optional: GUI-Fernzugriff auf verwaltete Firewalls ──────────────
|
||||
# Standardmäßig NICHT gestartet und dreifach aus: Profil "gui"
|
||||
# (COMPOSE_PROFILES=gui), GUI_BIND_ADDRESS (Standard 127.0.0.1) und der
|
||||
# Schalter unter Einstellungen. Gleiches Image wie das Backend, aber ohne
|
||||
# Datenbank, ohne Secret-Volumes und ohne eigene Ports: der Proxy terminiert
|
||||
# TLS auf den Slot-Ports und leitet hierher weiter; mit dem Backend spricht
|
||||
# das Gateway nur über /internal/gui/* (GUI_GATEWAY_TOKEN). Siehe Handbuch,
|
||||
# "Firewall-Oberfläche über OPNexus".
|
||||
gui-gateway:
|
||||
image: ${OPNEXUS_IMAGE_REPO:-local}/opnexus-backend:${OPNEXUS_VERSION:-latest}
|
||||
profiles: ["gui"]
|
||||
restart: unless-stopped
|
||||
logging: *default-logging
|
||||
command: ["python", "-m", "app.gui_gateway"]
|
||||
depends_on:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
GUI_GATEWAY_TOKEN: ${GUI_GATEWAY_TOKEN:-}
|
||||
OPNEXUS_GUI_SITES: ${OPNEXUS_GUI_SITES:-}
|
||||
GUI_BACKEND_URL: http://backend:8000
|
||||
user: "65534:65534"
|
||||
read_only: true
|
||||
cap_drop: ["ALL"]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import socket; socket.create_connection(('127.0.0.1', 8080), 3).close()"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 5s
|
||||
|
||||
# ── Optional: Updates aus dem Webinterface ──────────────────────────
|
||||
# Standardmäßig NICHT gestartet. Aktivieren mit COMPOSE_PROFILES=updater in
|
||||
# .env und `docker compose up -d`. Dies ist der einzige Container mit
|
||||
|
||||
+305
-5
@@ -74,6 +74,32 @@ Alternativ lassen sich auch aus einem Checkout die fertigen Images verwenden: in
|
||||
der `.env` `OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus` und `OPNEXUS_VERSION=<version>`
|
||||
setzen, dann `docker compose pull && docker compose up -d --no-build`.
|
||||
|
||||
### Images prüfen (Signatur)
|
||||
|
||||
Die Release-Images sind mit [cosign](https://github.com/sigstore/cosign) signiert (ab 1.23.9).
|
||||
Damit lässt sich prüfen, dass ein Image wirklich von OPNexus stammt und seit dem Bau nicht
|
||||
verändert wurde. Der öffentliche Schlüssel liegt als `cosign.pub` im Repository und im
|
||||
Release-Bundle:
|
||||
|
||||
```bash
|
||||
cosign verify --key cosign.pub --insecure-ignore-tlog=true \
|
||||
git.opnexus.dev/opnexus/opnexus-backend:<version>
|
||||
```
|
||||
|
||||
Das Gleiche gilt für `opnexus-frontend`, `opnexus-proxy` und `opnexus-updater`. Eine
|
||||
erfolgreiche Prüfung gibt die Signatur als JSON aus; bei einem fremden oder fehlenden
|
||||
Schlüssel endet der Befehl mit einem Fehler. `--insecure-ignore-tlog=true` ist hier gewollt:
|
||||
OPNexus lädt Signaturen nicht in das öffentliche Transparenzlog, damit die Prüfung auch in
|
||||
abgeschotteten Netzen funktioniert. Die Signatur gilt dem Inhalt (Digest) des Images, nicht
|
||||
dem Tag.
|
||||
|
||||
`opnexus update` prüft die Signaturen selbst, bevor es etwas wechselt, sobald `cosign`
|
||||
installiert ist und `cosign.pub` neben der `docker-compose.yml` liegt (`cosign.pub` liegt im Bundle
|
||||
bereits dabei, `cosign` selbst musst du installieren). Der Updater-Container für Updates per Klick
|
||||
bringt `cosign` mit und prüft deshalb ebenfalls, solange `cosign.pub` im Projektverzeichnis liegt. Schlägt die Prüfung fehl, bricht das
|
||||
Update ab und es wurde nichts verändert. Fehlt `cosign`, sagt `opnexus update` das und
|
||||
macht ohne Prüfung weiter.
|
||||
|
||||
### Erster Start und Passwort
|
||||
|
||||
Ohne `ADMIN_PASSWORD` in der `.env` erzeugt OPNexus beim ersten Start ein
|
||||
@@ -384,6 +410,83 @@ angesprochen (keine DNS-Namen). Nicht enthalten: Die Einrichtung auf der Firewal
|
||||
(bei einer Firewall hinter NAT kann es sie ja nicht erreichen); bei einer erreichbaren Firewall kannst du
|
||||
alternativ unter VPN › WireGuard einen Tunnel mit denselben Werten verteilen.
|
||||
|
||||
### Firewall-Oberfläche über OPNexus
|
||||
|
||||
Mit **Oberfläche öffnen** auf der Host-Seite öffnest du die Weboberfläche einer Firewall in einem neuen
|
||||
Browser-Tab, und zwar **über OPNexus**. Das lohnt sich vor allem bei Firewalls hinter NAT, die nur über den
|
||||
[Management-Tunnel](#firewalls-hinter-nat-management-tunnel) erreichbar sind: Dein Browser braucht keinen eigenen
|
||||
Zugang zum Tunnel-Netz. Du meldest dich an der Firewall wie gewohnt mit deren Benutzer und Passwort an; OPNexus
|
||||
kennt und speichert diese Zugangsdaten nicht und meldet nichts automatisch an.
|
||||
|
||||
**Standardmäßig ist die Funktion aus** und doppelt gesichert: Sie braucht eine Einrichtung auf dem Server und
|
||||
einen Schalter in den Einstellungen.
|
||||
|
||||
**Einrichten** (in der `.env` im OPNexus-Verzeichnis):
|
||||
|
||||
1. **HTTPS und `COOKIE_SECURE=true`** (siehe [Zugriff aus dem Netz und TLS](#zugriff-aus-dem-netz-und-tls)).
|
||||
Ohne sichere Cookies lässt sich die Funktion nicht einschalten.
|
||||
2. **Compose-Profil `gui`** aktivieren: `COMPOSE_PROFILES=gui` (mit weiteren Profilen kommagetrennt, z. B.
|
||||
`overlay,gui`). Das startet den Container `gui-gateway`.
|
||||
3. **Slot-Adressen** festlegen: `OPNEXUS_GUI_SITES=https://10.0.0.5:9441, https://10.0.0.5:9442, https://10.0.0.5:9443, https://10.0.0.5:9444`.
|
||||
Es ist dieselbe Adresse wie in `OPNEXUS_SITES`, je Slot ein Port. Diese Ursprünge dürfen **nicht** in
|
||||
`FRONTEND_ORIGIN` stehen (OPNexus startet sonst nicht).
|
||||
4. **Gemeinsames Geheimnis** `GUI_GATEWAY_TOKEN` (mindestens 32 Zeichen, z. B. `openssl rand -hex 32`).
|
||||
5. **Ports:** `GUI_PORTS=9441-9444` (die Ports aus `OPNEXUS_GUI_SITES`) sind die Slot-Ports am Proxy. Setze den
|
||||
Eintrag ausdrücklich: ohne ihn vergibt Docker zufällige freie Ports, damit ein Update nicht an schon belegten
|
||||
Ports (z. B. Portainer auf 9443) scheitert. `GUI_BIND_ADDRESS` bestimmt, wo
|
||||
sie lauschen; Standard ist `127.0.0.1` (nur lokal). Für den Zugriff aus dem Netz trägst du die Adresse des
|
||||
OPNexus-Rechners ein und gibst die Ports in der Firewall des Rechners nur für Administratoren frei.
|
||||
6. `docker compose up -d`, dann unter **Einstellungen › Firewall-Oberfläche über OPNexus** den Schalter einschalten.
|
||||
Fehlt etwas, nennt die Einstellungsseite genau, was.
|
||||
|
||||
In den Einstellungen legst du außerdem fest: **Inaktivität** (Standard 15 Minuten, 1 bis 240), **Gesamtdauer**
|
||||
(Standard 60 Minuten, 5 bis 720) und die Zahl der **gleichzeitigen Plätze** (Standard 4, 1 bis 4). Ausschalten
|
||||
beendet alle offenen Sitzungen (das Gateway bemerkt es spätestens nach 20 Sekunden).
|
||||
|
||||
**So ist es abgesichert:**
|
||||
|
||||
- **Eigene Origin:** Jede Firewall-Oberfläche läuft auf einem eigenen Port (einem „Platz“) und damit in einem
|
||||
eigenen Browser-Ursprung, getrennt von OPNexus. Skripte der Firewall-Oberfläche können OPNexus nicht
|
||||
ansprechen; OPNexus lehnt außerdem Anfragen mit Anmelde-Cookie von fremden Ursprüngen ab.
|
||||
- **Cookies bleiben auf dem Server:** Das Gateway reicht die OPNexus-Sitzung nie an die Firewall weiter und
|
||||
die Cookies der Firewall nie an deinen Browser. Die Firewall-Sitzung liegt nur im Arbeitsspeicher des Gateways.
|
||||
- **Nur Admins**, nur mit angemeldeter Browsersitzung (nie mit einem API-Token). Das Gateway prüft alle 20
|
||||
Sekunden nach: Abmelden, Entzug der Admin-Rolle oder Ausschalten beendet die Sitzung.
|
||||
- **Einmal-Ticket:** Der Klick erzeugt ein Ticket, das nur **30 Sekunden** und **genau einmal** gilt und an deine
|
||||
Sitzung und den Platz gebunden ist. Danach verschwindet es aus der Adresszeile.
|
||||
- **Nur sichere Firewalls:** https-Adresse und geprüftes oder festgesetztes Zertifikat (siehe Pinning); sonst
|
||||
bleibt der Knopf deaktiviert und nennt den Grund.
|
||||
- **Zeitlimits** (Inaktivität und Gesamtdauer) und **Audit:** Unter **Monitoring › Audit** (Art „GUI-Zugriff“)
|
||||
stehen Ticket, Start und Ende jeder Sitzung mit Benutzer, Firewall, Platz und Client-IP. Was du in der
|
||||
Oberfläche tust, protokolliert die Firewall selbst, dort aber mit der Adresse von OPNexus; das OPNexus-Audit
|
||||
zeigt, wer dahinter stand.
|
||||
- **Anmeldeschutz:** Das Gateway lässt höchstens drei fehlgeschlagene Anmeldungen pro Firewall in fünf Minuten zu.
|
||||
|
||||
**Wichtig: OPNexus-Adresse beim Anmeldeschutz der Firewall eintragen.** Über das Gateway kommen alle
|
||||
Anmeldungen von der Adresse von OPNexus (bei Firewalls im Tunnel von der des Hubs). Die Anmeldesperre von
|
||||
OPNsense (sshguard) sperrt nach mehreren Fehlversuchen diese Adresse, und damit auch die **API-Zugriffe von
|
||||
OPNexus**. Trage sie deshalb auf jeder Firewall unter **System › Einstellungen › Administration** (Login
|
||||
Protection) in die Whitelist ein, bevor du die Funktion benutzt.
|
||||
|
||||
**Einschränkungen:**
|
||||
|
||||
- **Kein WebSocket** (Funktionen, die einen brauchen, gehen nicht) und **keine automatische Anmeldung**.
|
||||
- **Höchstens 4 Plätze gleichzeitig.** Ein Platz bleibt pro OPNexus-Sitzung bei derselben Firewall; sind alle
|
||||
belegt, schließe eine Oberfläche oder warte, bis sie abläuft. Ein noch offener Tab einer alten Sitzung auf
|
||||
einem wiederverwendeten Platz bleibt ein theoretisches Restrisiko; schließe Tabs nach der Arbeit.
|
||||
- Nach einem **Neustart des wg-hub** braucht der Management-Tunnel 2 bis 4 Minuten, bis die Firewalls
|
||||
sich wieder verbunden haben; bis dahin lässt sich die Oberfläche nicht öffnen.
|
||||
- Mit `COOKIE_SECURE=false` ist die Funktion nicht verfügbar.
|
||||
|
||||
**Hinweis für Betreiber mit eigenen Skripten:** Seit dieser Funktion heißt das Sitzungs-Cookie bei HTTPS
|
||||
`__Host-opnexus-session` (bestehende Sitzungen werden beim nächsten Seitenaufruf übernommen, niemand wird abgemeldet), und Anfragen an `/api` mit Cookie
|
||||
und `Sec-Fetch-Site: same-site` oder `cross-site` werden abgelehnt. Skripte mit API-Token (`Authorization: Bearer`)
|
||||
sind nicht betroffen.
|
||||
|
||||
**Alternative ohne diese Funktion:** Du richtest dir einen eigenen WireGuard-Zugang in das Netz der Firewalls ein
|
||||
(z. B. am Standort oder auf der Firewall selbst) und rufst die Oberfläche direkt auf. Das ist unabhängig von
|
||||
OPNexus, hat aber keine Rollenprüfung und kein Audit durch OPNexus.
|
||||
|
||||
---
|
||||
|
||||
## Regeln sicher ausrollen
|
||||
@@ -462,6 +565,71 @@ Beschreibung und Systemregeln. Die Oberfläche nennt den Grund je Regel.
|
||||
> strikt „erster Treffer gewinnt“ aus. Die Reihenfolge ist dadurch
|
||||
> sicherheitsrelevant, siehe [hierarchische Policies](#gerätegruppen-und-hierarchische-policies).
|
||||
|
||||
### Aliase und NAT von der Firewall übernehmen
|
||||
|
||||
Wie die Regeln lassen sich auch **Aliase** und **NAT-Regeln** (Port Forward, 1:1, Outbound) übernehmen, die von Hand
|
||||
in OPNsense entstanden sind: auf der jeweiligen Seite **Von Firewall übernehmen**, Firewall wählen, dann entweder ein
|
||||
Objekt mit **Übernehmen** oder alle auf einmal mit **Alle übernehmbaren übernehmen**. So kommt eine bestehende Firewall
|
||||
ohne Abtippen unter zentrale Verwaltung.
|
||||
|
||||
Die Übernahme **schreibt nichts auf die Firewall**. OPNexus liest das Objekt frisch von der Firewall, legt es zentral an
|
||||
(Ziel: genau diese Firewall) und merkt sich die UUID aus OPNsense. Deshalb zeigt die Vorschau direkt danach „keine Änderung“,
|
||||
und der nächste Rollout nach einer Änderung aktualisiert dasselbe Objekt, statt ein zweites anzulegen. Die Übernahme steht im
|
||||
Audit-Log; Operatoren dürfen übernehmen (sie legt nur zentrale Einträge an), Nur-Lese-Benutzer nicht.
|
||||
|
||||
Nicht übernommen wird, was das Modell verfälschen würde. Die Oberfläche nennt den Grund je Objekt:
|
||||
|
||||
| Objekt | Nicht übernehmbar, weil … |
|
||||
|---|---|
|
||||
| Aliase | OPNsense-eigen (`bogons`, `sshlockout`, `__lan_network` …); Typ unbekannt für OPNexus (URL, URL-Tabelle, GeoIP, dynamische IPv6-Hosts, OpenVPN-Gruppe, „Extern“); der Name existiert in OPNexus schon; Name mit Sonderzeichen; BGP-ASN mit IPv4/IPv6-Einschränkung; Verweis auf einen Alias, der nicht übernommen ist |
|
||||
| Port Forward | mehrere oder keine Schnittstelle; IP-Version weder IPv4 noch IPv6; Protokoll „beliebig“; „Kein Redirect“; kein Ziel; verknüpfte Filterregel; invertierte Quelle oder Ziel; NAT-Reflection-Wert unbekannt; Beschreibung fehlt |
|
||||
| 1:1-NAT | mehrere oder keine Schnittstelle; unbekannter Typ oder NAT-Reflection-Wert; Pflichtangaben oder Beschreibung fehlen |
|
||||
| Outbound-NAT | mehrere oder keine Schnittstelle; „Kein NAT“-Regel (`nonat`); IP-Version weder IPv4 noch IPv6; Beschreibung fehlt |
|
||||
|
||||
Automatisch erzeugte NAT-Regeln (Anti-Lockout, „Auto created rule“) erscheinen nie in der Liste. **Verschachtelte Aliase:**
|
||||
Ein Alias, der einen anderen, noch nicht übernommenen Alias enthält, wartet auf diesen; „Alle übernehmbaren übernehmen“
|
||||
übernimmt beide in der richtigen Reihenfolge, einzeln zuerst den enthaltenen Alias. Verweist er auf einen Alias, der nie
|
||||
übernommen werden kann (z. B. vom Typ URL), bleibt er mit diesem Grund zurück. **Outbound-NAT** lässt sich nur übernehmen,
|
||||
wenn der Outbound-Modus der Firewall Hybrid oder Manuell ist; im Modus Automatisch zeigt OPNsense manuelle Regeln gar nicht an,
|
||||
OPNexus verweist dann darauf und ändert den Modus nicht.
|
||||
|
||||
Einstellungen, die OPNexus nicht verwaltet (Protokollierung und Pool-Optionen beim Port Forward, Tags, Kategorien, statischer
|
||||
NAT-Port), werden nie überschrieben und im Dialog als „Bleibt auf der Firewall unverändert“ angezeigt.
|
||||
|
||||
#### Gleiches Objekt auf mehreren Firewalls: verknüpfen statt doppelt übernehmen
|
||||
|
||||
Gibt es dieselbe Regel, denselben Alias oder dieselbe NAT-Regel auf mehreren Firewalls (typisch: beide Knoten eines
|
||||
HA-Clusters, mehrere Standorte mit gleicher Vorgabe), übernimmst du sie **einmal** und **verknüpfst** sie auf den anderen
|
||||
Firewalls mit dem zentralen Objekt. Das gilt für Regeln, Aliase, Port Forward, 1:1-NAT und Outbound-NAT.
|
||||
|
||||
1. Auf der ersten Firewall normal **übernehmen**.
|
||||
2. Im Dialog die zweite Firewall wählen. Bei jedem Eintrag, der einem zentralen Objekt **exakt** entspricht, steht
|
||||
„Gibt es schon zentral“ und die Aktion **Verknüpfen** (daneben **Als eigenes Objekt übernehmen**). Verknüpfen trägt die
|
||||
Firewall als weiteres Ziel in das Objekt ein und merkt sich die UUID dort. Auf der Firewall wird nichts geändert.
|
||||
3. Die Vorschau zeigt danach für diese Firewall „keine Änderung“. Eine spätere Änderung am Objekt rollst du wie gewohnt für
|
||||
alle Ziele gemeinsam aus; es entsteht kein Duplikat.
|
||||
|
||||
**Was „exakt“ heißt:** Alle von OPNexus verwalteten Felder sind gleich, nachdem die `{{VARIABLEN}}` des zentralen Objekts mit
|
||||
den Werten **dieser** Firewall ersetzt wurden (ein Alias `{{LAN_NET}}` passt also zu `192.168.2.0/24`, wenn das die Variable
|
||||
dieser Firewall ist). Bei Aliasen müssen Name, Typ, Inhalt einschließlich der Verweise auf andere Aliase, Beschreibung und
|
||||
Aktiv-Status stimmen. Fehlt der Firewall ein Variablenwert, passt das Objekt nicht, der Dialog nennt die Variable.
|
||||
|
||||
**Abweichungen:** Gibt es ein Objekt mit gleichem Namen (Alias) bzw. gleicher Beschreibung (Regeln, NAT), das sich im Inhalt
|
||||
unterscheidet, zeigt der Dialog „Ähnlich“ mit den abweichenden Feldern (zentral gegenüber Firewall). Verknüpft wird dann **nie**,
|
||||
auch nicht automatisch. Gleiche entweder die Firewall oder das zentrale Objekt an, oder übernimm das Objekt getrennt (bei Aliasen
|
||||
geht das nur nach einer Umbenennung, der Name ist zentral eindeutig).
|
||||
|
||||
**Alle übernehmbaren:** Der Dialog zeigt vorher, was passiert („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“), und du
|
||||
bestätigst. Verknüpft wird nur, wenn genau **ein** zentrales Objekt exakt passt; gibt es mehrere identische (Altlast aus früheren
|
||||
Einzelübernahmen), entscheidest du bei dem Eintrag selbst, und „Alle“ übernimmt ihn als neues Objekt. Ein zentrales Objekt wird in
|
||||
einem Lauf höchstens einmal verknüpft.
|
||||
|
||||
**Sicherheit:** Die Firewall wird beim Verknüpfen frisch gelesen und noch einmal exakt verglichen; hat sich zwischen Anzeige und Klick
|
||||
etwas geändert, wird nichts verknüpft und die Meldung nennt die Felder. Ein zweiter Klick ändert nichts. Die Aktion steht im
|
||||
Audit-Log und in den Revisionen des Objekts („Mit <Firewall> verknüpft“). Rollen wie beim Übernehmen: Operatoren dürfen, Nur-Lese-Benutzer
|
||||
nicht. Hat das zentrale Objekt eine offene Bestätigung oder einen offenen Rollback, wird erst nach deren Klärung verknüpft.
|
||||
„Alle übernehmbaren“ lässt solche Einträge dann aus (Hinweis im Ergebnis). **Wichtig:** Ein verknüpftes Objekt wird wie jedes verwaltete Objekt behandelt: Wird das zentrale Objekt gelöscht oder die Firewall aus dessen Zielen entfernt, löscht der nächste Rollout das Objekt auch auf dieser Firewall.
|
||||
|
||||
---
|
||||
|
||||
## Gerätegruppen und hierarchische Policies
|
||||
@@ -765,7 +933,15 @@ frische Backup erneut. Eine blockierte Aktion wird als „fehlgeschlagen“ mark
|
||||
und **nicht** wiederholt. Sie steht im Status der geplanten Wartung und meldet sich
|
||||
zusätzlich per Benachrichtigung („Geplante Wartung nicht ausgeführt“, einzeln
|
||||
schaltbar), damit man es am nächsten Morgen nicht erst beim Nachsehen merkt.
|
||||
Wiederkehrende Zeitpläne gibt es nicht.
|
||||
Wiederkehrende Termine und ganze Flotten laufen über [Rollouts](#rollouts-in-wellen).
|
||||
|
||||
**Ergebnis automatisch prüfen und abschließen** (im Dialog vorausgewählt): Nach dem Auftrag
|
||||
wartet OPNexus, bis der Firmware-Auftrag beendet ist, und prüft dann selbst: Firmware
|
||||
mindestens eine Minute am Stück bereit, ein erwarteter Neustart hat wirklich stattgefunden
|
||||
(Host offline gesehen oder neue Boot-Zeit), Version lesbar. Erst dann wird die Wartungsaktion
|
||||
als abgeschlossen markiert und sperrt Host und Cluster nicht mehr. Gelingt das nicht innerhalb
|
||||
von 20 Minuten, bleibt sie „unklar“ (gesperrt, mit Benachrichtigung). Ohne die Option bleibt
|
||||
sie wie bisher „übergeben“, bis jemand sie von Hand abschließt.
|
||||
|
||||
In einem Cluster läuft unbeaufsichtigte Wartung standardmäßig **nur auf dem
|
||||
CARP-BACKUP**. Ist der Host beim Einplanen gerade MASTER, lehnt OPNexus das sofort mit
|
||||
@@ -792,7 +968,72 @@ der Plan steht auf „fehlgeschlagen“, die Wartungsaktion auf „unklar“, un
|
||||
Benachrichtigung. Der CARP-Umschalter wird nie blind wiederholt. Beendest du die
|
||||
Wartungsaktion von Hand, stoppt der Ablauf und fasst nichts mehr an. Bei Erfolg schließt
|
||||
OPNexus die Wartungsaktion selbst ab. Sie läuft auch einen Neustart des OPNexus-Backends
|
||||
mitten im Ablauf weiter (der Zustand steht in der Datenbank).
|
||||
mitten im Ablauf weiter (der Zustand steht in der Datenbank). Mit „Ergebnis automatisch
|
||||
prüfen“ kommt die Prüfung **vor** dem Verlassen des Wartungsmodus: Der Host bekommt den
|
||||
Verkehr erst zurück, wenn er nachweislich fertig ist.
|
||||
|
||||
### Rollouts in Wellen
|
||||
|
||||
**System › Rollouts** verteilt ein Firmware-Update (stabiles Update oder Major-Upgrade) in
|
||||
Wellen über die Flotte. Die Wellen bestehen aus [Gerätegruppen](#gerätegruppen), die man
|
||||
als Ringe anlegt, zum Beispiel `ring-canary` (eine unwichtige Firewall), `ring-pilot`
|
||||
(einige) und `ring-prod` (der Rest). Steht ein Host in mehreren Wellen, gilt die früheste.
|
||||
|
||||
**Ablauf je Welle:**
|
||||
|
||||
1. **Vorprüfung:** Update-Suche auf den Hosts der Welle, dann Bestand aufnehmen (Version,
|
||||
Gateways, laufende Dienste, Sicherheits-Score). Hosts ohne passendes Update werden
|
||||
übersprungen. Die Version, die in der ersten Welle installiert wird, ist die
|
||||
**Zielversion**; bietet ein Host einer späteren Welle eine andere an, hält der Rollout an
|
||||
(OPNsense installiert immer die neueste, sonst bekäme „prod“ eine Version, die im
|
||||
Canary-Ring nie lief).
|
||||
2. **Hosts nacheinander:** Jeder Host läuft als [geplante Wartung](#geplante-wartung) mit
|
||||
automatischer Prüfung (Backup vor dem Update, Sperren, Audit). Immer nur ein Host,
|
||||
immer nur ein Rollout gleichzeitig. Im Cluster zuerst der BACKUP, dann der MASTER;
|
||||
der MASTER nur, wenn die Welle die CARP-Übergabe erlaubt, sonst hält der Rollout an.
|
||||
Vor dem zweiten Knoten eines Clusters werden der Cluster (alle Knoten erreichbar,
|
||||
genau ein MASTER je VHID, kein Wartungsmodus) und der schon aktualisierte Knoten
|
||||
erneut geprüft.
|
||||
3. **Beobachtungszeit** (je Welle einstellbar), dann die **Gesundheitsprüfung** der
|
||||
aktualisierten Hosts: erreichbar und Firmware bereit, Zielversion installiert, CARP
|
||||
gesund, Gateways und Dienste, die vorher liefen, laufen wieder (vorher schon gestörte
|
||||
nur als Hinweis), keine offenen Sperren, optional Sicherheits-Score nicht gesunken
|
||||
(Hinweis oder Halt). Ist etwas rot, prüft OPNexus zwei Minuten später noch einmal
|
||||
(langsamer Start nach dem Neustart) und hält dann an.
|
||||
4. **Freigabe** (optional je Welle): Die Welle beginnt erst, wenn jemand auf „Welle
|
||||
freigeben“ klickt.
|
||||
|
||||
**Angehalten** wird bei jeder Unklarheit, nie wird etwas von selbst wiederholt: Gate rot,
|
||||
Host fehlgeschlagen oder ungewöhnlich lange unterwegs, Host in der Vorprüfung nicht bereit,
|
||||
abweichende Zielversion, MASTER ohne Übergabe, eine fremde offene Wartung im Cluster länger
|
||||
als 15 Minuten, die verantwortliche Person darf nicht mehr ausrollen, das Fenster wurde
|
||||
gelöscht, der Lauf ist älter als 14 Tage. Es kommt die Benachrichtigung „Rollout
|
||||
angehalten“. **Fortsetzen** prüft zuerst alle bisher aktualisierten Hosts erneut; wer
|
||||
fortsetzt, ist ab dann verantwortlich. Ein fehlgeschlagener Host muss vorher ausdrücklich
|
||||
**übersprungen** werden (seine Wartungsaktion bleibt „unklar“, bis du sie auf der
|
||||
Host-Seite abschließt). **Abbrechen** verwirft alles, was noch nicht begonnen hat; eine
|
||||
Aktion, die schon auf einer Firewall läuft, wird geführt zu Ende gebracht. **Anhalten**
|
||||
darf jeder mit Schreibrecht, alles andere braucht das Recht zum Ausrollen.
|
||||
|
||||
**Wartungsfenster** (Reiter „Wartungsfenster“): wöchentlich an bestimmten Tagen oder
|
||||
monatlich (erster…vierter oder letzter Wochentag, oder ein fester Tag 1–28), Beginn in einer
|
||||
Zeitzone, Dauer und Abschneidezeit: Nach den ersten N Minuten beginnt kein neuer Host mehr,
|
||||
laufende Updates laufen zu Ende, und der Rollout wartet auf den nächsten Termin (dann wird
|
||||
zuerst das Bisherige erneut geprüft). Sommerzeit: Eine Uhrzeit, die es an dem Tag nicht gibt,
|
||||
rückt vor (02:30 → 03:30); eine doppelte gilt beim ersten Mal. Einzelne Termine lassen sich
|
||||
auslassen oder verschieben. Ein Fenster, das ein Plan nutzt, lässt sich nicht löschen.
|
||||
|
||||
**Automatischer Start:** Ein Plan mit Fenster und stabilem Update kann zu jedem Termin von
|
||||
selbst starten. Dazu muss ihn jemand mit Ausroll-Recht nach der Vorschau **aktivieren**.
|
||||
OPNexus merkt sich dabei einen Fingerabdruck von Plan, aufgelösten Hosts und Fenster; ändert
|
||||
sich davon etwas (auch die Mitglieder einer Gruppe), startet nichts, und es kommt „Rollout
|
||||
nicht automatisch gestartet“. Jede Bearbeitung des Plans deaktiviert ihn. So kann niemand
|
||||
die Ziele eines aktivierten Plans unbemerkt ändern.
|
||||
|
||||
Das **Update Center** zeigt den laufenden Rollout als Banner; Hosts, die er noch bearbeitet,
|
||||
sind dort für manuelle Firmware-Aktionen gesperrt. Ein Neustart oder Update von OPNexus
|
||||
während eines Rollouts ist sicher (der Zustand steht in der Datenbank), hält ihn aber für
|
||||
diese Zeit an.
|
||||
|
||||
### Dienste und Systemauslastung
|
||||
|
||||
@@ -918,18 +1159,70 @@ Einstellungen › **Benachrichtigungen**. Kanäle:
|
||||
Absender und Empfängern. Zertifikat und Hostname werden geprüft; ein Mailserver
|
||||
mit privater CA braucht `SMTP_CA_FILE`. Anmeldung ohne Verschlüsselung bleibt
|
||||
gesperrt. Mit **testen** prüfst du jeden Kanal.
|
||||
- **Weitere Kanäle** (Abschnitt „Weitere Kanäle“, beliebig viele je Typ, jeweils mit **Testen**):
|
||||
|
||||
| Typ | Angaben | Zugangsdaten |
|
||||
|---|---|---|
|
||||
| **ntfy** | Serveradresse (Standard `https://ntfy.sh`), Thema, Priorität 1–5 | Zugriffstoken, optional |
|
||||
| **Gotify** | Serveradresse, Priorität 0–10 | App-Token |
|
||||
| **Telegram** | Chat-ID (Zahl oder `@kanal`), optional eine eigene API-Adresse | Bot-Token (von @BotFather) |
|
||||
| **Microsoft Teams** | – | Webhook-Adresse eines Workflows („Beim Empfang einer Webhook-Anfrage in einem Kanal posten“) oder eines Incoming-Webhooks |
|
||||
| **Matrix** | Homeserver-Adresse, Raum-ID (`!abc:example.org`, nicht der Alias) | Zugriffstoken eines Benutzers, der im Raum ist |
|
||||
|
||||
Die Zugangsdaten liegen wie das SMTP-Passwort in der Geheimnis-Ablage des Servers, nie in der
|
||||
Datenbank, werden nie wieder angezeigt (nur „gesetzt“) und stehen nicht in Fehlermeldungen oder Logs.
|
||||
Weil bei Telegram und Teams die Adresse selbst das Geheimnis ist, wird sie wie ein Token behandelt.
|
||||
Ziele müssen `http(s)` sein, ohne Zugangsdaten in der Adresse; lokale Adressen (`localhost`, Loopback,
|
||||
Link-Local, Metadaten-Dienst) werden abgelehnt, Ziele im eigenen Netz sind erlaubt. Weiterleitungen werden
|
||||
nicht verfolgt. Ein ausgefallener Kanal hält die anderen nicht auf; zugestellt wird einmal je Ereignis.
|
||||
|
||||
Ereignisse (jeweils einzeln schaltbar): Host nicht erreichbar, Zertifikat läuft ab,
|
||||
Regel-Drift im Cluster, NAT-Drift im Cluster, Netzwerk-Abweichung im Cluster,
|
||||
fehlgeschlagener Rollback, Wartungsergebnis unklar, CARP-Zustand kritisch
|
||||
(Split-Brain/kein Master), CARP-Auto-Revert fehlgeschlagen, geplante Wartung nicht
|
||||
ausgeführt. Geprüft wird im eingestellten Intervall (Standard 300 s). Ein Problem
|
||||
ausgeführt, Gateway gestört, Ressourcen-Schwelle überschritten, Dienst ausgefallen
|
||||
(die letzten drei sind unten beschrieben). Geprüft wird im eingestellten Intervall (Standard 300 s). Ein Problem
|
||||
meldet sich einmal und löst sich selbst auf, sobald es behoben ist; die
|
||||
Dashboard-Karte „Warnungen“ zeigt die offenen. Ausnahme „geplante Wartung nicht
|
||||
ausgeführt“: Sie bleibt drei Tage nach dem geplanten Zeitpunkt offen und schließt dann
|
||||
ohne „behoben“-Meldung, weil sich ein verpasster Termin nicht von selbst behebt. Der
|
||||
Text der Benachrichtigungen ist nicht übersetzt.
|
||||
|
||||
#### Gateway-, Ressourcen- und Dienst-Alarme
|
||||
|
||||
Diese drei Ereignisse lesen die **Minutenmessung** (dieselbe wie die Auslastungs-Diagramme), fragen die
|
||||
Firewalls also nicht zusätzlich ab. Damit Einzelspitzen und flackernde Verbindungen nichts melden,
|
||||
öffnet ein Alarm erst, wenn die letzten *N* Messungen alle schlecht sind, und schließt erst, wenn die letzten
|
||||
*N* alle wieder gut sind; dazwischen bleibt der Zustand, wie er ist. *N* ist einstellbar (Standard 3 für
|
||||
Gateways und Dienste, 5 für Ressourcen; die Messung läuft jede Minute). Meldet eine Firewall gerade nicht,
|
||||
beurteilt OPNexus sie nicht (das meldet „Host nicht erreichbar“) und lässt offene Alarme stehen.
|
||||
|
||||
- **Gateway gestört** – **standardmäßig an**, weil ein WAN-Ausfall das ist, was jeder wissen will. Ein Alarm je Gateway, wenn OPNsense es als
|
||||
*down* führt, oder wenn der Paketverlust (Standard ab 20 %) oder die Latenz (Standard ab 500 ms) darüber liegt (0 schaltet die jeweilige Schwelle ab).
|
||||
OPNsense selbst entscheidet über „down“ nach den Grenzen im Gateway (Einstellung „Gateway-Überwachung“). Von Hand abgeschaltete
|
||||
Gateways („Gateway deaktivieren/erzwungen aus“) und Gateways ohne Überwachung (kein Monitor-IP, Werte „~“) melden nichts.
|
||||
Der Status steht auch auf der Host-Seite (Abschnitt „Gateways“, live gelesen), als Kachel auf dem Dashboard und in Prometheus.
|
||||
- **Ressourcen-Schwelle** – standardmäßig **aus**: Auf einer bestehenden Flotte würde das Einschalten sonst gleich alles melden, was schon länger voll
|
||||
ist. Schwellen: Festplatte ab 90 % (je ZFS-Pool bzw. UFS-Gerät ein Alarm), Arbeitsspeicher ab 95 %, Swap ab 50 %, CPU-Last ab 2,0 je Thread
|
||||
(1-Minuten-Last geteilt durch die Zahl der CPU-Threads). Jede Schwelle lässt sich mit 0 abschalten.
|
||||
- **Dienst ausgefallen** – standardmäßig **aus**. OPNsense liefert nur „läuft / läuft nicht“ und „gesperrt“, aber nicht, ob jemand einen Dienst
|
||||
absichtlich gestoppt hat. Deshalb gilt eine **Liste überwachter Dienste** (Standard: unbound, dnsmasq, dhcpd, dhcpd6, kea-dhcp4, kea-dhcp6, ntpd,
|
||||
openvpn, wireguard, strongswan), und gemeldet wird nur ein Dienst, der in den letzten rund zwei Stunden lief und jetzt nicht mehr. Ein Dienst, der
|
||||
nie lief (etwa DHCPv6 ohne Präfix), bleibt still; einen absichtlich gestoppten Dienst nimmst du aus der Liste. Eine Firewall listet nur Dienste, die
|
||||
in ihrer Konfiguration aktiviert sind, ein nicht eingerichteter Dienst taucht also gar nicht auf.
|
||||
|
||||
### Regel-Analyse
|
||||
|
||||
Unter **Regelwerk › Regel-Analyse** fragst du, welche Regel für eine Verbindung greift: Quell- und Ziel-IP,
|
||||
Protokoll, Zielport und das Eingangs-Interface („automatisch“ rät es aus der Quell-IP und kennzeichnet das als
|
||||
Hinweis). Mit „Ausgangsseite prüfen“ werden zusätzlich die Regeln für ausgehenden Verkehr geprüft. Geprüft wird
|
||||
das erste Paket einer neuen Verbindung gegen den tatsächlich geladenen Regelsatz der Firewall; es wird nur gelesen.
|
||||
Je Firewall erscheint eine Karte mit dem Urteil (Erlaubt, Blockiert, Abgewiesen), der **Sicherheit** (sicher,
|
||||
vermutlich, unbekannt; Tooltip mit den Gründen) und der entscheidenden Regel. Ist das Ergebnis „unbekannt“
|
||||
(zum Beispiel fehlt der Zielport oder ein Alias ist nicht auflösbar), zeigt die Karte bewusst kein Urteil, sondern
|
||||
die offenen Punkte. Unter „Warum?“ steht jede geprüfte Regel in der Prüfreihenfolge mit ✓/✗/? je Feld.
|
||||
Die Abfrage steht in der Adresszeile und lässt sich als Link teilen.
|
||||
|
||||
### Sicherheitslage
|
||||
|
||||
Unter **Monitoring › Sicherheitslage** prüft OPNexus jede Firewall auf typische
|
||||
@@ -1030,6 +1323,10 @@ scrape_configs:
|
||||
| `opnexus_host_enabled`, `…_last_sample_timestamp_seconds` | Überwachung an; Zeit der letzten Messung |
|
||||
| `opnexus_host_memory_used_percent`, `opnexus_host_load1`, `opnexus_host_disk_used_percent` | Letzte Messung |
|
||||
| `opnexus_host_carp_master` | 1 = MASTER, 0 = BACKUP (fehlt, wenn unbekannt) |
|
||||
| `opnexus_host_swap_used_percent` | Swap-Belegung der letzten Messung (fehlt ohne Swap) |
|
||||
| `opnexus_host_gateway_up{gateway}` | 1, solange OPNsense das Gateway nicht als down führt |
|
||||
| `opnexus_host_gateway_loss_percent`, `…_delay_seconds`, `…_stddev_seconds` `{gateway}` | Paketverlust, Laufzeit und Abweichung zum Monitor (fehlen ohne Messung) |
|
||||
| `opnexus_host_service_running{service,instance}` | 1 = läuft, nur für Dienste aus der Überwachungsliste |
|
||||
| `opnexus_host_compliance_score`, `opnexus_host_compliance_findings{severity}` | Sicherheitslage |
|
||||
| `opnexus_pending_actions{kind,status}` | Offene Aktionen wie auf dem Dashboard |
|
||||
| `opnexus_problems_open{type}` | Offene Probleme der Benachrichtigungen |
|
||||
@@ -1324,8 +1621,11 @@ nach einem Neustart erhalten. Alternativ legt ihn OPNexus unter
|
||||
OPNsense, keine Ausführung durch OPNexus.
|
||||
- **Logs:** kein Live-Streaming (Polling im Sekundenbereich), die Freitextsuche ist
|
||||
eine einfache Teilstring-Suche ohne Index.
|
||||
- **Geplante Wartung:** nur einmalige Zeitpunkte, ohne Benachrichtigung bei
|
||||
Fehlschlag.
|
||||
- **Geplante Wartung und Rollouts:** Rollouts kennen keine Zielversion „von außen“
|
||||
(OPNsense installiert immer die neueste angebotene) und keinen automatischen Rückweg zur
|
||||
alten Version. Major-Upgrade-Rollouts sind nur gegen eine nachgestellte Firewall getestet.
|
||||
Die Gesundheitsprüfung sieht die Firewall aus Sicht der Verwaltung; ob der Nutzverkehr
|
||||
durchgeht, prüft sie nicht.
|
||||
- **Dienste:** keine Auswahlliste, alle Dienste sind steuerbar.
|
||||
- **VPN:** Umfang je Verfahren siehe [VPN](#vpn-ipsec-openvpn-wireguard).
|
||||
- **Zertifikate:** kein Push auf mehrere Hosts, keine Verwaltung von DNS-Providern
|
||||
|
||||
+9
-2
@@ -1,9 +1,11 @@
|
||||
FROM node:20-alpine AS deps
|
||||
# deps und builder laufen immer auf der Build-Plattform (nativ schnell, `next build` unter QEMU-Emulation
|
||||
# hängt); das Ergebnis (.next/standalone) ist reines JavaScript und gilt für jede Zielarchitektur.
|
||||
FROM --platform=$BUILDPLATFORM node:20-alpine AS deps
|
||||
WORKDIR /app
|
||||
COPY package.json package-lock.json ./
|
||||
RUN npm ci --no-audit --no-fund
|
||||
|
||||
FROM node:20-alpine AS builder
|
||||
FROM --platform=$BUILDPLATFORM node:20-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
@@ -11,10 +13,15 @@ ENV NEXT_TELEMETRY_DISABLED=1
|
||||
RUN npm run build
|
||||
|
||||
FROM node:20-alpine AS runner
|
||||
ARG TARGETARCH
|
||||
ARG BUILDARCH
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production
|
||||
COPY --from=builder /app/.next/standalone ./
|
||||
COPY --from=builder /app/.next/static ./.next/static
|
||||
# Beim Cross-Build liegen die nativen sharp-Binärdateien der Build-Architektur im Standalone-Ordner; die App nutzt
|
||||
# next/image nicht, sie wären nur totes Gewicht.
|
||||
RUN if [ "$TARGETARCH" != "$BUILDARCH" ]; then rm -rf node_modules/@img node_modules/sharp; fi
|
||||
|
||||
EXPOSE 3000
|
||||
CMD ["node", "server.js"]
|
||||
Loaded 100 of 149 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user