Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Julian Boldt
|
cf9981c6eb | ||
|
Julian Boldt
|
4eea6424e7 | ||
|
Julian Boldt
|
a8ca0a14ef | ||
|
Julian Boldt
|
b020e38d0a | ||
|
Julian Boldt
|
abd15318a3 | ||
|
Julian Boldt
|
ec20940139 | ||
|
Julian Boldt
|
8128d14da4 | ||
|
Julian Boldt
|
95f297d3f2 | ||
|
Julian Boldt
|
ad17dcad25 | ||
|
Julian Boldt
|
5cb2ebe1c6 |
No files matched your search
@@ -100,3 +100,26 @@ TRUST_PROXY_HEADERS=false
|
||||
# Zum Aktivieren (Handbuch „Firewalls hinter NAT“): COMPOSE_PROFILES=overlay
|
||||
# und den UDP-Port aus dem Internet erreichbar machen.
|
||||
#OVERLAY_PORT=51820
|
||||
|
||||
# ── GUI-Fernzugriff auf verwaltete Firewalls (optional, Standard AUS) ──
|
||||
# Öffnet die Weboberfläche einer Firewall über OPNexus, auch hinter NAT
|
||||
# (Management-Tunnel), jeweils in einem eigenen Browser-Ursprung (Slot-Port).
|
||||
# Dreifach aus, bis alles hier gesetzt ist UND ein Admin den Schalter unter
|
||||
# Einstellungen einschaltet. Voraussetzungen: HTTPS über den Proxy und
|
||||
# COOKIE_SECURE=true. Siehe Handbuch, „Firewall-Oberfläche über OPNexus“.
|
||||
# 1. Profil aktivieren (mit anderen Profilen kommagetrennt, z. B. overlay,gui):
|
||||
# COMPOSE_PROFILES=gui
|
||||
# 2. Slot-Ursprünge: dieselbe Adresse wie in OPNEXUS_SITES, je Slot ein Port
|
||||
# aus GUI_PORTS. Darf NICHT in FRONTEND_ORIGIN stehen (Start wird sonst verweigert).
|
||||
# OPNEXUS_GUI_SITES=https://10.0.0.5:9441, https://10.0.0.5:9442, https://10.0.0.5:9443, https://10.0.0.5:9444
|
||||
# 3. Gemeinsames Geheimnis Backend <-> Gateway, mind. 32 Zeichen, z. B.
|
||||
# `openssl rand -hex 32`:
|
||||
# GUI_GATEWAY_TOKEN=
|
||||
# 4. Wo die Slot-Ports lauschen. Standard nur lokal (127.0.0.1); für den
|
||||
# Zugriff aus dem Netz die Adresse dieses Hosts eintragen (wie bei
|
||||
# PROXY_BIND_ADDRESS). GUI_PORTS ist ein Bereich, Host- und Container-Port
|
||||
# sind gleich. GUI_PORTS zum Einschalten SETZEN: ohne den Eintrag nimmt
|
||||
# Docker zufällige freie Ports (damit ein Update nie an belegten Ports wie
|
||||
# 9443 scheitert), und die Slot-Adressen oben wären nicht erreichbar.
|
||||
# GUI_BIND_ADDRESS=127.0.0.1
|
||||
# GUI_PORTS=9441-9444
|
||||
+183
-1
@@ -29,7 +29,189 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
|
||||
den Einstellungen.
|
||||
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Merlin …
|
||||
ansteht): Saker …
|
||||
|
||||
## [1.32.1] "Merlin" — 2026-10-23
|
||||
|
||||
- **Korrektur: Rollouts erkannten die Zielversion bei anderen Release-Typen falsch.** OPNexus las die neue Version nur am Kernpaket `opnsense`. Bei der
|
||||
Business-Edition (`opnsense-business`) und beim Wechsel des Release-Typs (z. B. auf `opnsense-devel`, dann als neues Paket angeboten) fiel es auf
|
||||
`product_latest` zurück; das Gate hielt den Rollout dann fälschlich mit „Version passt nicht“ an. Live gefunden und belegt (Rollout mit Cluster in einer Welle).
|
||||
- **Plan-Hinweis zu Alias-Namen ehrlich formuliert:** „kein zentraler Alias in OPNexus“ statt „kein Alias dieses Namens“ (von Hand auf der Firewall
|
||||
angelegte Aliase sind in Ordnung; der Planer fragt bewusst keine Firewall).
|
||||
|
||||
## [1.32.0] "Merlin" — 2026-10-23
|
||||
|
||||
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
|
||||
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
|
||||
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
|
||||
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
|
||||
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
|
||||
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
|
||||
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
|
||||
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
|
||||
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
|
||||
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
|
||||
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
|
||||
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
|
||||
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
|
||||
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
|
||||
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
|
||||
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
|
||||
|
||||
## [1.31.1] "Vulture" — 2026-10-22
|
||||
|
||||
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
|
||||
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
|
||||
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
|
||||
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
|
||||
|
||||
## [1.31.0] "Vulture" — 2026-10-22
|
||||
|
||||
- **Regel-Bereinigung: Entfernen, Bestätigen, Löschen, Wiederherstellen:** Im Tab **Bereinigung** (Regelwerk › Regel-Analyse) lassen sich Kandidaten jetzt auswählen
|
||||
(Checkbox, höchstens so viele Regeln je Lauf wie in den Einstellungen; „vermutlich“ nur mit einer ausdrücklichen Einzelbestätigung je Regel, „unbekannt“ nie).
|
||||
**Vorschau** zeigt je Firewall (im Cluster zuerst der Backup) die Regeln mit Befund, Sicherheit und Schutzprüfung; **Ausrollen** deaktiviert die Regeln und wendet an.
|
||||
Danach läuft ein Countdown: **Bestätigen** heißt „ich erreiche die Firewall noch“, ohne Bestätigung setzt OPNexus **automatisch zurück**; bis dahin geht auch
|
||||
**Jetzt zurücksetzen**. Bestätigte Regeln lassen sich nach der Wartezeit (einstellbar, Standard 7 Tage) **löschen**; vorher sichert OPNexus den Regelinhalt, und
|
||||
unter **Verlauf der Bereinigungen** kommt die Regel mit **Wiederherstellen** deaktiviert und möglichst an alter Position zurück (sonst mit Hinweis am Ende der Gruppe);
|
||||
**Wieder aktivieren** hat wieder ein Bestätigungsfenster. Wichtig: **Ausrollen, Wiederholen und Wiederherstellen darf nur ein Administrator**, eine Vorschau gehört
|
||||
dem, der sie erstellt hat (ein Admin öffnet selbst eine neue); **Bestätigen geht nur in der Browser-Sitzung**, nie per API-Token. **Die Bereinigung bleibt gesperrt,
|
||||
bis unter Einstellungen die Verwaltungsquellen** (IP-Adressen oder Netze, von denen aus verwaltet wird) **eingetragen sind**: Regeln, die diesen Zugang treffen
|
||||
könnten, deaktiviert OPNexus nie. Dort stehen auch die Obergrenze je Lauf, die Wartezeit bis zum Löschen und **weitere Verwaltungsports** (zusätzlich zu
|
||||
API-Port, 443, 80, 22). Tragen Sie die Adresse ein, mit der OPNexus selbst bei den Firewalls ankommt (im Cluster für jeden Knoten). Quellen, Ports und
|
||||
Firewall-Adressen sehen nur Administratoren. Der Hinweis „Entfernen folgt“ entfällt.
|
||||
- **Update: neuere CLI aus dem Zielrelease:** `opnexus update` (auch per Klick im Webinterface) holt sich nach dem Pull die CLI aus dem Updater-Image des
|
||||
Zielreleases und startet sich damit neu, damit Korrekturen am Update-Ablauf sofort greifen statt erst beim übernächsten Update. Das geschieht nur, wenn
|
||||
`cosign` und `cosign.pub` vorhanden sind und die Signatur des Image-**Digests** in diesem Lauf geprüft wurde (ausgeführt wird genau dieser Digest);
|
||||
sonst und bei jedem Fehler läuft das Update mit der vorhandenen CLI weiter. Abschalten: `OPNEXUS_NO_CLI_REFRESH=1`. Wirksam wird es ab dem Update
|
||||
*von* dieser Version.
|
||||
|
||||
## [1.30.0] "Owl" — 2026-10-22
|
||||
|
||||
- **Regel-Analyse: Bereinigung (nur Ansicht):** Auf **Regelwerk › Regel-Analyse** gibt es den Tab **Bereinigung** (`?tab=cleanup`). Er listet je Firewall die
|
||||
Regeln, die einen Blick wert sind: **ungenutzt** (OPNexus hat in 7/30/90/180 Tagen keinen Treffer gesehen), **verdeckt** (eine frühere Quick-Regel
|
||||
trifft schon alles, was sie trifft), **doppelt** und **lange deaktiviert**. Jeder Befund trägt eine Sicherheit (**sicher / vermutlich / unbekannt**, Tooltip mit
|
||||
dem Grund): „sicher“ gibt es nur mit vollständigen, frischen Belegen, bei einer Regel mit Zeitplan, ohne geladenen pf-Regelsatz oder bei Aliasen bleibt es
|
||||
höchstens „vermutlich“. Die Tabelle zeigt auch den letzten Treffer und ob die Regel entfernbar wäre (nicht: automatische Regeln, Regeln im Altformat, von OPNexus
|
||||
verwaltete Regeln – die entfernt man über die Regelseite). Es wird nur gelesen; **Entfernen folgt in einer späteren Version**. Die Sicherheitslage verlinkt ihre
|
||||
Befunde „ungenutzt/verdeckt/doppelt“ direkt dorthin; per API `GET /api/hosts/{id}/rule-analysis/cleanup?unused_days=30`, per MCP `rule_cleanup_candidates`.
|
||||
- **Änderungsverlauf je Regel:** OPNexus liest alle 15 Minuten die Regeln jeder Firewall und merkt sich, was sich geändert hat – auch bei Regeln, die nicht von OPNexus
|
||||
stammen: angelegt, geändert (mit Vorher/Nachher), aktiviert, deaktiviert, entfernt, jeweils mit der Herkunft „über OPNexus“ (passender Audit-Eintrag) oder „extern“.
|
||||
Auf der Live-Regel-Ansicht (Regelwerk › Live über alle Hosts) öffnet die Zeilenaktion **Verlauf** die Ereignisse zusammen mit den gespeicherten Fassungen und dem Audit der
|
||||
verwalteten Regel. Die erste Beobachtung speichert nur den Ausgangszustand; der Verlauf wird 365 Tage aufbewahrt (`GET /api/hosts/{id}/live-rules/{uuid}/history`).
|
||||
Eine Umsortierung (andere Reihenfolge oder Gruppe) gilt als „verschoben“, nicht als Änderung. Eine leere oder unvollständige Antwort der Firewall (ohne
|
||||
automatische Regeln) wird verworfen; verschwinden mehr als die Hälfte der Regeln auf einmal, zählt erst die zweite gleiche Beobachtung. Die Herkunft „über OPNexus“
|
||||
erklärt nur Audit-Einträge seit der vorherigen Beobachtung, und jeder Eintrag nur eine Änderung.
|
||||
Schemaänderungen 0027 und 0028 (Tabellen und ein Feld für den Beobachtungszeitraum) laufen beim Start.
|
||||
- **Regelnutzung genauer:** Die Zähler kommen jetzt aus `ruleStats` (von der Firewall schon je Regel summiert; ältere Versionen weiterhin über den pf-Regelsatz) und werden alle
|
||||
15 Minuten statt alle 6 Stunden gelesen. Regeln im Altformat (noch nicht migriert) werden ebenfalls erfasst. Das Beobachtungsfenster für „ungenutzt“ beginnt neu, wenn eine Regel eingeschaltet oder geändert wird, und nach einem pf-Neuladen mit langer Lücke bei den Zählern.
|
||||
- **Verdeckt/doppelt in der Sicherheitslage genauer:** Der Vergleich kennt jetzt die echte Reihenfolge (Floating vor Interface-Gruppen vor Interface-Regeln), Netze und
|
||||
Portbereiche als Mengen (z. B. 10.1.0.0/16 liegt in 10.0.0.0/8), `inet46` und `tcp/udp`, und übergeht Regeln mit Zeitplan, Markierung oder Limits. Es findet dadurch
|
||||
mehr verdeckte Regeln als bisher, nie weniger belegte.
|
||||
|
||||
## [1.29.0] "Caracara" — 2026-10-22
|
||||
|
||||
- **Fehlende API-Rechte werden sichtbar:** Darf der API-Schlüssel einer Firewall den Gateway- oder Dienststatus nicht lesen (OPNsense antwortet
|
||||
mit 403), fiel das bisher niemandem auf, und der standardmäßig aktive Gateway-Alarm konnte nie auslösen. Jetzt zeigen die Host-Seite
|
||||
(Abschnitt „Gateways“) und die Dashboard-Karte „Fleet-/CARP-Gesundheit“ ein Warn-Badge „API-Rechte fehlen: Gateways, Dienste“; der Tooltip nennt die
|
||||
OPNsense-Berechtigung, die beim Benutzer oder bei der Gruppe fehlt (System: Gateways, Status: Services, Lobby: Dashboard). Zusätzlich gibt es
|
||||
den neuen Alarm **API-Rechte fehlen** (standardmäßig an, abschaltbar unter Einstellungen › Benachrichtigungen, gleiche Entprellung wie die
|
||||
anderen Messungs-Alarme) und die Prometheus-Metrik `opnexus_host_api_rights_missing{host,read}`. Nur eine ausdrückliche Ablehnung zählt;
|
||||
Zeitüberschreitungen oder Serverfehler bleiben „unbekannt“.
|
||||
- **Gateway-Alarm pro Gateway abschaltbar:** In der Gateway-Liste der Host-Seite schaltet eine Glocke den Alarm für ein einzelnes Gateway aus
|
||||
(z. B. ruhendes Zweit-WAN oder tote IPv6-Gateways); ein „Alarm aus“-Badge zeigt es an. Ein offener Alarm dieses Gateways löst sich dabei
|
||||
auf. Nur Administratoren, die Änderung steht im Audit-Log. Das Dashboard zeigt bei mehr als acht Gateways „+ N weitere“.
|
||||
- **Firewall-Oberfläche (GUI-Zugriff) härter:** Ein frei gewordener Platz wird für eine **andere** Firewall erst 5 Minuten nach dem Ende der letzten
|
||||
Sitzung vergeben (Schutz, falls dort noch ein Tab offen ist, der die Cookies des Platzes teilt); dieselbe Firewall bekommt ihn sofort wieder.
|
||||
Sind alle Plätze belegt oder noch in der Abkühlzeit, nennt die Meldung den Grund. Ändern sich Adresse, festgesetztes Zertifikat oder Tunnelweg
|
||||
einer Firewall, während ihre Oberfläche offen ist, endet die Sitzung spätestens nach 20 Sekunden. Das Limit von 64 gleichzeitigen Sitzungen
|
||||
verdrängt keine laufende Sitzung mehr: eine neue wird abgelehnt (mit Meldung), bis eine endet. **Hinweis:** Das GUI-Gateway-Image muss zur
|
||||
Backend-Version passen (wie bisher beim Update beider Container).
|
||||
- **Anmeldung (Cookies) härter:** Das alte Sitzungs-Cookie `opnexus-session` wird bei HTTPS **nicht mehr akzeptiert**. **Alle, die sich seit der
|
||||
Umstellung auf 1.27 nicht neu angemeldet haben, müssen sich einmal neu anmelden**; Skripte mit rohem Sitzungs-Cookie brauchen den Namen
|
||||
`__Host-opnexus-session` (API-Tokens sind nicht betroffen, ohne HTTPS bleibt `opnexus-session` der normale Name). Beim Abmelden wird ein altes
|
||||
Cookie weiterhin gelöscht. Das Cookie, das die SSO-Anmeldung an den Browser bindet, heißt bei HTTPS jetzt `__Host-opnexus-oidc-state`; der
|
||||
Rückweg vom Anbieter wird nur noch als Seitenaufruf (Navigation) von einer fremden Seite zugelassen, und ohne dieses Cookie scheitert er immer.
|
||||
- **Webhook, Benachrichtigungskanäle und SSO-Anbieter: Schutz vor DNS-Rebinding:** Bisher prüfte OPNexus die Adresse, auf die ein Name zeigt, und
|
||||
der HTTP-Client löste den Namen danach noch einmal auf; ein Server unter Fremdkontrolle konnte beim zweiten Mal eine interne Adresse liefern. Jetzt
|
||||
verbindet OPNexus genau zu der geprüften Adresse (Host-Name, TLS-Zertifikatsprüfung und Anfrage bleiben beim Namen); zeigt eine Antwort auch nur
|
||||
teilweise auf Loopback, Link-Local oder einen Cloud-Metadaten-Dienst (auch AWS-IPv6 und Alibaba), kommt keine Verbindung zustande und die Zustellung meldet „Zieladresse nicht
|
||||
erlaubt“. Private Netze bleiben erlaubt (eigener ntfy, Gotify, Matrix). Proxy-Umgebungsvariablen werden für diese Aufrufe nicht mehr beachtet.
|
||||
- **Benachrichtigungen parallel zugestellt:** Webhook, E-Mail und alle Kanäle werden gleichzeitig bedient, jeder mit eigener Zeitgrenze (20 Sekunden),
|
||||
und ein Prüfdurchlauf sendet insgesamt höchstens 90 Sekunden. Ein hängender Kanal verzögert weder die anderen noch die nächsten Prüfungen; im Log
|
||||
steht dann „timed out“ (ohne Adresse oder Token). Wie bisher wird einmal je Ereignis zugestellt, ohne Wiederholung, **außer** der Durchlauf war so voll, dass
|
||||
für ein Ereignis noch gar kein Versuch stattfand: es wird im nächsten Durchlauf zugestellt. Die Zustellung beginnt jetzt nach allen Prüfungen
|
||||
eines Durchlaufs statt mitten darin.
|
||||
- **Rollouts: Fingerabdruck, verantwortliche Person, überlappende Termine:** Der Fingerabdruck eines aktivierten Plans umfasst jetzt auch die
|
||||
ausgelassenen und verschobenen Termine seines Fensters: **wer einen Termin auslässt oder verschiebt, muss einen automatischen Plan danach neu
|
||||
aktivieren** (ohne solche Ausnahmen bleiben bestehende Aktivierungen gültig; **Pläne, deren Fenster schon Ausnahmen haben, nach dem Update
|
||||
einmal neu aktivieren**, sonst meldet der Plan „Plan geändert“ und startet nicht automatisch). Die verantwortliche Person (Aktivierende, Startende, Freigebende)
|
||||
wird zusätzlich als Konto-ID gespeichert (Migration 0026, bestehende Einträge werden über den Namen zugeordnet): ein gelöschtes und unter gleichem
|
||||
Namen neu angelegtes Konto gilt nicht mehr als dieselbe Person, der Lauf pausiert bzw. der automatische Start unterbleibt mit „Person darf nicht mehr
|
||||
ausrollen“. Angezeigt wird weiter der Benutzername. Überlappen sich zwei Termine eines Wartungsfensters, zählt der zuletzt begonnene (bisher der
|
||||
früheste), sodass ein älterer, längst abgeschnittener Termin keinen neuen Start mehr blockiert.
|
||||
- **Kleinigkeiten:** Ressourcen-Alarme zeigen bei fehlendem Messwert keine Zahl mehr („None %“ entfällt). Der Dialog „Von Firewall übernehmen“ zeigt ohne aktive
|
||||
Firewall einen Hinweis statt dauerhaft „Lädt…“. Fehlermeldungen beim Übernehmen und Verknüpfen nennen die Gründe in Klartext (Deutsch/Englisch) statt
|
||||
roher Codes wie `name_exists`; dasselbe gilt für „GUI-Zugriff nicht eingerichtet“ und blockierte Netzwerkänderungen. Der Test-Knopf der
|
||||
Benachrichtigungskanäle ist für Nicht-Admins ausgeblendet (der Server verweigert den Test ohnehin). Der Tooltip der aktiven Plätze nennt keine
|
||||
Umgebungsvariable mehr. Eine Gateway-Sitzung, die der Gateway-Container nicht mehr bestätigt (Absturz), zählt für die Abkühlzeit der Plätze als
|
||||
beendet.
|
||||
- **Grafana-Dashboard erweitert:** neue Felder für Gateways (erreichbar, Paketverlust, Latenz), Swap, gestoppte Dienste, fehlende API-Rechte und
|
||||
Rollouts nach Status; dazu die Metrik `opnexus_rollout_runs{status}`.
|
||||
|
||||
## [1.28.0] "Sparrowhawk" — 2026-10-21
|
||||
|
||||
- **Übernehmen: mit vorhandenem zentralem Objekt verknüpfen:** Dieselbe von Hand angelegte Regel, derselbe Alias oder dieselbe NAT-Regel auf mehreren
|
||||
Firewalls (z. B. beide HA-Knoten) wird nicht mehr als zweites Objekt übernommen. Im Dialog „Von Firewall übernehmen“ erscheint bei exakter
|
||||
Übereinstimmung (nach Einsetzen der Host-Variablen, bei Aliasen inkl. Verweisen) die Aktion **Verknüpfen**: die Firewall wird weiteres Ziel des
|
||||
vorhandenen Objekts, die Vorschau zeigt „keine Änderung“, und der nächste Rollout aktualisiert alle Firewalls an einem Objekt. Auf der Firewall wird
|
||||
dabei nichts geändert. Unterscheidet sich ein gleichnamiges Objekt, zeigt der Dialog die abweichenden Felder; verknüpft wird dann nie. „Alle
|
||||
übernehmbaren“ fragt vorher nach („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“). Aliase, die bisher wegen „Name existiert“ abgelehnt wurden,
|
||||
lassen sich so übernehmen.
|
||||
Wartet das vorhandene Objekt gerade auf eine Bestätigung, wird nicht verknüpft; „Alle übernehmbaren“ lässt den Eintrag dann aus. Verknüpfen
|
||||
macht ein gerade geöffnetes Bearbeiten-Formular des Objekts ungültig, damit die neue Firewall nicht versehentlich wieder aus den Zielen fällt.
|
||||
|
||||
- **Tooltips überarbeitet:** Der deaktivierte Button „Oberfläche öffnen“ nennt jetzt den Grund im Klartext samt nächstem Schritt (statt nur einer
|
||||
Variablen wie `GUI_GATEWAY_TOKEN`); dasselbe gilt für ausgeschaltete Funktion, deaktivierte Firewall, Nicht-HTTPS-Adresse und ausgeschaltete
|
||||
Zertifikatsprüfung. Texte durchgehend in Du-Form, kürzere VPN-Einleitungen, verständlichere Hinweise bei gesperrten Buttons (Zertifikate, Benutzer,
|
||||
Dienste). Icon-Buttons (Bearbeiten, Löschen, Schließen, Reihenfolge) zeigen beim Darüberfahren ihren Namen.
|
||||
|
||||
## [1.27.2] "Buzzard" — 2026-10-21
|
||||
|
||||
- **`opnexus update` prüft den Proxy vor dem Wechsel:** Nach dem Ziehen der Images validiert das Werkzeug die Caddy-Konfiguration des neuen
|
||||
Proxy-Images mit den Umgebungsvariablen *dieser* Installation. Ist sie ungültig, bricht das Update ab, bevor etwas gewechselt wurde.
|
||||
Der Fall aus 1.27.0 (Compose-Datei mit leerem `OPNEXUS_GUI_SITES`) wird dabei automatisch repariert: Das Werkzeug trägt den Platzhalter
|
||||
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` ein und sagt es. Installationen, die noch die Compose-Datei aus 1.27.0 haben,
|
||||
müssen also nur einmal `opnexus update` ausführen.
|
||||
- **`opnexus update` erkennt einen abstürzenden Proxy:** Ein Dienst ohne Healthcheck (der Proxy) gilt jetzt erst nach 10 Sekunden
|
||||
durchgehender Laufzeit als gesund. Bisher meldete das Update „Fertig“, obwohl der Proxy in einer Neustart-Schleife hing, und das
|
||||
automatische Zurückrollen griff nicht.
|
||||
|
||||
## [1.27.1] "Buzzard" — 2026-10-21
|
||||
|
||||
- **Fix (1.27.1): Proxy startete mit der neuen Compose-Datei nicht.** Mit der `docker-compose.yml` aus 1.27.0 gab Compose die Variable
|
||||
`OPNEXUS_GUI_SITES` als leeren Text an den Proxy weiter; Caddy nutzt seinen Standardwert aber nur bei *nicht gesetzter* Variable und
|
||||
brach mit „server block without any key“ ab, OPNexus war dann nicht mehr erreichbar. Die Compose-Datei setzt jetzt einen gültigen
|
||||
Platzhalter als Standard. Betroffen waren nur Installationen, die die neue Compose-Datei aus 1.27.0 übernommen hatten (ohne
|
||||
`OPNEXUS_GUI_SITES` in der `.env`); wer nur die Images aktualisiert hatte, war nicht betroffen. Sofort-Abhilfe ohne Update:
|
||||
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` eintragen und `docker compose up -d proxy` ausführen.
|
||||
|
||||
## [1.27.0] "Buzzard" — 2026-10-21
|
||||
|
||||
Vorgesehen als MINOR-Release **1.27.0** „Firewall-Oberfläche über OPNexus“.
|
||||
|
||||
- **Firewall-Oberfläche über OPNexus:** Auf der Host-Seite öffnet **Oberfläche öffnen** die Weboberfläche der Firewall
|
||||
in einem neuen Tab, auch bei Firewalls hinter NAT im Management-Tunnel. Nur für Admins; jede Firewall läuft in einer
|
||||
eigenen Browser-Origin (Platz = Port), Cookies bleiben auf dem Server, das Ticket gilt einmalig 30 Sekunden,
|
||||
Inaktivitäts- und Gesamtlimit sind einstellbar, Ticket, Start und Ende stehen im Audit (Art „GUI-Zugriff“).
|
||||
Standardmäßig **aus**: braucht das Compose-Profil `gui`, `OPNEXUS_GUI_SITES`, `GUI_PORTS`, `GUI_GATEWAY_TOKEN`,
|
||||
`COOKIE_SECURE=true` und den Schalter unter Einstellungen (Handbuch, „Firewall-Oberfläche über OPNexus“).
|
||||
Live getestet auf OPNsense 26.7.
|
||||
- **Einschränkungen:** kein WebSocket, keine automatische Anmeldung, höchstens 4 Plätze gleichzeitig. OPNexus-Adresse
|
||||
vorher in die Whitelist des Anmeldeschutzes (sshguard) der Firewall eintragen, sonst können Fehlanmeldungen die
|
||||
API-Zugriffe von OPNexus aussperren. Nach einem Neustart des wg-hub braucht der Tunnel 2 bis 4 Minuten.
|
||||
- **Hinweis für Betreiber mit eigenen Skripten:** Bei HTTPS heißt das Sitzungs-Cookie jetzt `__Host-opnexus-session`
|
||||
(bestehende Sitzungen werden ohne Abmelden übernommen), und Cookie-Anfragen an `/api` mit `Sec-Fetch-Site: same-site/cross-site`
|
||||
werden abgelehnt. Zugriffe mit API-Token sind nicht betroffen.
|
||||
|
||||
## [1.26.0] "Kite" — 2026-10-21
|
||||
|
||||
|
||||
+6
-4
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||
|
||||
|
||||
> **Stand:** `1.26.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.32.1` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -19,13 +19,15 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln, Aliase und NAT-Regeln lassen sich übernehmen.
|
||||
- **Zertifikate** — Trust-Store für alle Firewalls und Ausstellung per ACME / Let's Encrypt.
|
||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
|
||||
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
|
||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
|
||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend.
|
||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit); ein Administrator kann ausgewählte Regeln abschalten (Vorschau, dann Ausrollen mit automatischer Rücknahme ohne Bestätigung), später löschen und aus einer Sicherung des Regelinhalts wiederherstellen. Jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
|
||||
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
|
||||
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
|
||||
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
|
||||
@@ -64,7 +66,7 @@ Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelie
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.26.0
|
||||
OPNEXUS_VERSION=1.32.1
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||
|
||||
|
||||
> **Status:** `1.26.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.32.1` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -19,13 +19,15 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules, aliases and NAT rules created by hand on a firewall can be adopted.
|
||||
- **Certificates** — trust store for all firewalls and ACME / Let's Encrypt issuance.
|
||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
|
||||
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
|
||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
|
||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only.
|
||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence); an administrator can switch selected rules off (preview, then roll-out with an automatic revert unless confirmed), delete them later and restore them from a backup of their content. Every rule has a change history, including changes made outside OPNexus.
|
||||
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
|
||||
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
|
||||
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
|
||||
@@ -66,7 +68,7 @@ For access from other machines and for production use with TLS (bundled Caddy pr
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.26.0
|
||||
OPNEXUS_VERSION=1.32.1
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ from starlette.responses import JSONResponse
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.errors import ErrorCode
|
||||
from app.routers.auth import COOKIE_NAME
|
||||
from app.routers.auth import session_token
|
||||
from app.routers import governance
|
||||
from app.routers.auth import client_ip
|
||||
from app.services import api_token_service, auth_service
|
||||
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
"/api/settings",
|
||||
# Read-only JSON-RPC tools (see routers/mcp.py).
|
||||
"/api/mcp",
|
||||
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
|
||||
"/api/config/import/plan",
|
||||
# Self-scoped: a user managing their own second factor.
|
||||
"/api/auth/2fa/setup",
|
||||
"/api/auth/2fa/enable",
|
||||
@@ -94,10 +96,14 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if request.url.path in _PUBLIC_PATHS:
|
||||
return await call_next(request)
|
||||
if request.url.path.startswith("/internal/gui/"):
|
||||
# GUI gateway only: never routed by the proxy, authenticated by the
|
||||
# shared gateway token inside routers/gui_access.py.
|
||||
return await call_next(request)
|
||||
|
||||
api_token = None
|
||||
bearer = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
if bearer is not None:
|
||||
# An API token wins over a cookie; a bad one never falls back to it.
|
||||
async with SessionLocal() as db:
|
||||
@@ -117,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
)
|
||||
if user is not None:
|
||||
parts = request.url.path.strip("/").split("/")
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
|
||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||
@@ -127,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
if len(parts) > 2 and parts[1] == "settings":
|
||||
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
|
||||
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
|
||||
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
|
||||
entity = parts[2] if len(parts) > 2 and kind != "config" else None
|
||||
event = {"kind": kind, "action": action, "entity_id": entity,
|
||||
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
|
||||
"details": {"method": request.method, "path": request.url.path}}
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
|
||||
|
||||
if (
|
||||
user.role == "read_only"
|
||||
|
||||
@@ -6,6 +6,17 @@ class Settings(BaseSettings):
|
||||
def allowed_origins(self) -> list[str]:
|
||||
return [o.strip().rstrip("/") for o in self.frontend_origin.split(",") if o.strip()]
|
||||
|
||||
@property
|
||||
def gui_site_origins(self) -> list[str]:
|
||||
return [o.strip().rstrip("/") for o in self.opnexus_gui_sites.split(",") if o.strip()]
|
||||
|
||||
def gui_origin_conflicts(self) -> list[str]:
|
||||
"""Slot origins of the GUI gateway that are also listed in FRONTEND_ORIGIN.
|
||||
Such an origin would get CORS with credentials and pass the Sec-Fetch-Site
|
||||
exception — a firewall GUI could then read OPNexus' API. Startup is refused."""
|
||||
allowed = {o.lower() for o in self.allowed_origins}
|
||||
return [o for o in self.gui_site_origins if o.lower() in allowed]
|
||||
|
||||
database_url: str = (
|
||||
"postgresql+asyncpg://opnexus:opnexus@postgres:5432/opnexus"
|
||||
)
|
||||
@@ -92,6 +103,14 @@ class Settings(BaseSettings):
|
||||
# per-IP lockout entirely.
|
||||
trust_proxy_headers: bool = False
|
||||
|
||||
# ── GUI gateway (1.27, optional container "gui-gateway") ─────────
|
||||
# Slot origins the gateway is served on, comma-separated, e.g.
|
||||
# "https://192.0.2.10:9441,https://192.0.2.10:9442". Empty = feature unavailable.
|
||||
opnexus_gui_sites: str = ""
|
||||
# Shared secret between backend and gateway for /internal/gui/* (at least
|
||||
# 32 characters). Empty = the internal endpoints answer 404.
|
||||
gui_gateway_token: str = ""
|
||||
|
||||
# ── Deploy lockout protection ────────────────────────────────────
|
||||
deploy_confirm_window_seconds: int = 180
|
||||
|
||||
@@ -141,6 +160,13 @@ class Settings(BaseSettings):
|
||||
# wireguard_key_store.py.
|
||||
wireguard_key_secret_dir: str = "/data/wireguard-key-secrets"
|
||||
|
||||
# ── Rule observation (usage counters + change history of the live rules) ─
|
||||
# Two read-only GETs per enabled firewall per cycle (searchRule, ruleStats). The interval is an operator
|
||||
# setting (env RULE_OBSERVE_INTERVAL_SECONDS); it can not go below the floor, which keeps a typo from
|
||||
# polling every firewall in a tight loop.
|
||||
rule_observe_interval_seconds: int = 900
|
||||
rule_observe_min_interval_seconds: int = 60
|
||||
|
||||
# ── Resource monitoring ──────────────────────────────────────────────
|
||||
# Fixed interval, no admin-editable settings row/UI for this — v1
|
||||
# deliberately kept minimal (see resource_monitor_service.py).
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
|
||||
|
||||
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
|
||||
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
|
||||
"""
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
|
||||
|
||||
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
|
||||
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
|
||||
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
|
||||
|
||||
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
|
||||
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
|
||||
|
||||
ACTOR = "cli (Host-Konsole)"
|
||||
TOKEN_MARK = "# plan-token: "
|
||||
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
|
||||
|
||||
ERRORS_DE = {
|
||||
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
|
||||
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
|
||||
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
|
||||
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
|
||||
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
|
||||
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
|
||||
"yaml_empty": "Die Datei ist leer",
|
||||
"yaml_syntax": "YAML-Syntaxfehler",
|
||||
"duplicate_key": "Doppelter Schlüssel",
|
||||
"non_string_key": "Schlüssel muss Text sein",
|
||||
"too_deep": "Zu tief verschachtelt",
|
||||
"too_many_nodes": "Zu viele Elemente",
|
||||
"scalar_too_long": "Wert zu lang",
|
||||
"too_many_objects": "Zu viele Objekte",
|
||||
"too_many_entries": "Zu viele Einträge",
|
||||
"too_many_variables": "Zu viele Variablen für diesen Host",
|
||||
"too_large": "Datei zu groß",
|
||||
"bad_encoding": "Kein gültiges UTF-8",
|
||||
"not_a_mapping": "Die Datei muss ein Mapping sein",
|
||||
"missing_version": "opnexus_config: 1 fehlt",
|
||||
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
|
||||
"unknown_field": "Unbekanntes Feld",
|
||||
"missing_field": "Pflichtfeld fehlt",
|
||||
"bad_type": "Falscher Typ",
|
||||
"bad_value": "Ungültiger Wert",
|
||||
"bad_format": "Falsches Format",
|
||||
"too_short": "Zu kurz",
|
||||
"too_long": "Zu lang",
|
||||
"control_character": "Steuerzeichen nicht erlaubt",
|
||||
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
|
||||
"bad_variable_name": "Ungültiger Variablenname",
|
||||
"bad_variable_value": "Ungültiger Variablenwert",
|
||||
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
|
||||
"duplicate_entry": "Eintrag kommt doppelt vor",
|
||||
"unknown_host": "Unbekannter Host",
|
||||
"unknown_group": "Unbekannte Gerätegruppe",
|
||||
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
|
||||
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
|
||||
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
|
||||
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
|
||||
}
|
||||
REASONS_DE = {
|
||||
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
|
||||
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
|
||||
"prune_referenced": "wird noch verwendet",
|
||||
}
|
||||
WARNINGS_DE = {
|
||||
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
|
||||
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
|
||||
"unknown_alias_reference": "{value}: kein zentraler Alias in OPNexus (auf der Firewall von Hand angelegt? dann in Ordnung)",
|
||||
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
|
||||
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
|
||||
}
|
||||
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
|
||||
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
|
||||
|
||||
|
||||
def safe(text) -> str:
|
||||
"""Nothing from a foreign file reaches the terminal as a control sequence."""
|
||||
return _ANSI.sub("?", str(text))
|
||||
|
||||
|
||||
def _short(value) -> str:
|
||||
return safe(display(json.dumps(value, ensure_ascii=False), 60))
|
||||
|
||||
|
||||
def format_errors(errors: list[dict]) -> str:
|
||||
lines = []
|
||||
for e in errors:
|
||||
where = f"Zeile {e['line']}: " if e.get("line") else ""
|
||||
path = f"{safe(e['path'])}: " if e.get("path") else ""
|
||||
detail = ""
|
||||
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
|
||||
if params:
|
||||
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
|
||||
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_plan(public: dict) -> str:
|
||||
lines = []
|
||||
for item in public["items"]:
|
||||
action = item["action"]
|
||||
planned = item["planned_action"]
|
||||
name = f"{item['kind']} {safe(item['key'])}"
|
||||
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
|
||||
if action == "blocked":
|
||||
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
|
||||
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
|
||||
elif planned in LABEL:
|
||||
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
|
||||
else:
|
||||
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
|
||||
if change["field"]:
|
||||
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
|
||||
else:
|
||||
lines.append(f"~ {name}{suffix}")
|
||||
if item.get("implicit"):
|
||||
lines[-1] += " (Folge der Reihenfolge)"
|
||||
for warning in public["warnings"]:
|
||||
template = WARNINGS_DE.get(warning["code"])
|
||||
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
|
||||
text = template.format(**params) if template else warning["code"]
|
||||
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
|
||||
lines.append(f"? {where}{text}")
|
||||
s = public["summary"]
|
||||
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _read_stdin() -> bytes:
|
||||
data = sys.stdin.buffer.read(MAX_BYTES + 1)
|
||||
return data
|
||||
|
||||
|
||||
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
|
||||
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
|
||||
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
|
||||
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
|
||||
parser.add_argument("--kinds", default=None)
|
||||
parser.add_argument("--prune", action="store_true")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--plan-token", default=None)
|
||||
try:
|
||||
args = parser.parse_args(argv)
|
||||
except SystemExit:
|
||||
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
|
||||
if args.command == "schema":
|
||||
return 0, schema.json_schema_text(), ""
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.services import audit_service
|
||||
|
||||
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
if args.command == "export":
|
||||
wanted = None
|
||||
if args.kinds:
|
||||
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
|
||||
return 0, await export_yaml(db, wanted), ""
|
||||
|
||||
raw = stdin if stdin is not None else _read_stdin()
|
||||
try:
|
||||
parsed = parse(raw)
|
||||
if args.command == "plan":
|
||||
plan = await planner.build_plan(db, parsed, args.prune)
|
||||
await db.rollback()
|
||||
public = plan.public()
|
||||
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
|
||||
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
|
||||
s = public["summary"]
|
||||
if not plan.applicable:
|
||||
return 4, out, ""
|
||||
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
|
||||
|
||||
if not args.plan_token:
|
||||
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
|
||||
try:
|
||||
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
s = plan.summary()
|
||||
if args.json:
|
||||
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
|
||||
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
|
||||
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
|
||||
except ConfigError as exc:
|
||||
if args.json:
|
||||
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
|
||||
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
|
||||
finally:
|
||||
audit_service.context.reset(context)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
try:
|
||||
code, out, err = asyncio.run(run(argv))
|
||||
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
|
||||
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
|
||||
return 1
|
||||
sys.stdout.write(out)
|
||||
sys.stderr.write(err)
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,312 @@
|
||||
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
|
||||
|
||||
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
|
||||
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
|
||||
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
|
||||
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
|
||||
|
||||
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
|
||||
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
|
||||
Den Commit macht der Aufrufer.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import Parsed
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
|
||||
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.services import audit_service
|
||||
|
||||
NOTE = "Import (Konfiguration als Code)"
|
||||
AUDIT_KIND = {
|
||||
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
|
||||
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
|
||||
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
|
||||
}
|
||||
|
||||
|
||||
class PlanStale(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PlanBlocked(Exception):
|
||||
def __init__(self, plan: planner.Plan):
|
||||
self.plan = plan
|
||||
super().__init__("plan blocked")
|
||||
|
||||
|
||||
class SelfCheckFailed(Exception):
|
||||
def __init__(self, differences: list[str]):
|
||||
self.differences = differences
|
||||
super().__init__("; ".join(differences[:5]))
|
||||
|
||||
|
||||
def _revision_writer(kind: str):
|
||||
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
|
||||
|
||||
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
|
||||
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
|
||||
"""
|
||||
if kind == "rules":
|
||||
from app.services.policy_service import snapshot
|
||||
return RuleRevision, "rule_id", snapshot
|
||||
if kind == "aliases":
|
||||
from app.services.alias_policy_service import snapshot
|
||||
return AliasRevision, "alias_id", snapshot
|
||||
if kind == "port_forwards":
|
||||
from app.services.port_forward_policy_service import snapshot
|
||||
return PortForwardRevision, "port_forward_id", snapshot
|
||||
if kind == "one_to_one":
|
||||
from app.services.one_to_one_policy_service import snapshot
|
||||
return OneToOneRevision, "one_to_one_id", snapshot
|
||||
if kind == "outbound_nat":
|
||||
from app.services.outbound_nat_policy_service import snapshot
|
||||
return OutboundNatRevision, "outbound_nat_id", snapshot
|
||||
from app.services.policy_hierarchy_service import snapshot
|
||||
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
|
||||
|
||||
|
||||
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
|
||||
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
|
||||
SCALARS = {
|
||||
"aliases": ("name", "type", "description", "enabled"),
|
||||
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "enabled", "log"),
|
||||
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "target", "local_port", "natreflection", "enabled"),
|
||||
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
|
||||
"destination_not", "natreflection", "log", "enabled"),
|
||||
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
|
||||
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
|
||||
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
|
||||
}
|
||||
|
||||
|
||||
class _Writer:
|
||||
def __init__(self, db: AsyncSession, plan: planner.Plan):
|
||||
self.db = db
|
||||
self.plan = plan
|
||||
self.now = datetime.now(timezone.utc)
|
||||
self.events: list[dict] = []
|
||||
|
||||
async def load(self) -> None:
|
||||
db = self.db
|
||||
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
self.groups = {g.name: g for g in (await db.execute(
|
||||
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
|
||||
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
self.rows: dict[str, dict] = {}
|
||||
for kind, model in MODELS.items():
|
||||
options = [selectinload(model.deployments)]
|
||||
if kind != "policy_hierarchy_rules":
|
||||
options += [selectinload(model.targets), selectinload(model.groups)]
|
||||
rows = (await db.execute(select(model).options(*options))).scalars().all()
|
||||
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
|
||||
def items(self, kind: str, *actions: str) -> list[planner.Item]:
|
||||
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
|
||||
|
||||
def event(self, item: planner.Item, entity_id, name: str) -> None:
|
||||
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
|
||||
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
|
||||
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
|
||||
|
||||
# -- groups ----------------------------------------------------------------------------------------------------------
|
||||
async def hierarchy_groups(self) -> None:
|
||||
items = self.items("policy_hierarchy_groups", "create", "update")
|
||||
for item in items:
|
||||
if item.action == "create":
|
||||
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
|
||||
self.db.add(group)
|
||||
self.hierarchy[item.key] = group
|
||||
await self.db.flush()
|
||||
for item in items:
|
||||
group = self.hierarchy[item.key]
|
||||
group.description = item.after["description"]
|
||||
parent = item.after["parent"]
|
||||
group.parent_id = self.hierarchy[parent].id if parent else None
|
||||
self.event(item, group.id, group.name)
|
||||
await self.db.flush()
|
||||
|
||||
async def device_groups(self) -> None:
|
||||
for item in self.items("device_groups", "create", "update"):
|
||||
group = self.groups.get(item.key)
|
||||
if group is None:
|
||||
group = DeviceGroup(name=item.key)
|
||||
self.db.add(group)
|
||||
self.groups[item.key] = group
|
||||
group.description = item.after["description"]
|
||||
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
|
||||
await self.db.flush()
|
||||
self.event(item, group.id, group.name)
|
||||
|
||||
# -- objects ---------------------------------------------------------------------------------------------------------
|
||||
async def objects(self, kind: str) -> None:
|
||||
model = MODELS[kind]
|
||||
revision_model, fk, snapshot = _revision_writer(kind)
|
||||
for item in self.items(kind, "create", "update"):
|
||||
after = item.after
|
||||
row = self.rows[kind].get(item.key)
|
||||
if row is None:
|
||||
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
|
||||
if kind != "aliases":
|
||||
row.config_key = item.key
|
||||
self.db.add(row)
|
||||
self.rows[kind][item.key] = row
|
||||
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
|
||||
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
|
||||
for name in SCALARS[kind]:
|
||||
setattr(row, name, after[name])
|
||||
if kind == "aliases":
|
||||
row.content = "\n".join(after["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
row.hierarchy_group_id = self.hierarchy[after["group"]].id
|
||||
else:
|
||||
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
|
||||
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
|
||||
if "hierarchy_group" in after["targets"]:
|
||||
hierarchy = after["targets"]["hierarchy_group"]
|
||||
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
|
||||
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
|
||||
await self.db.flush()
|
||||
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
|
||||
async def hierarchy_positions(self) -> None:
|
||||
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
|
||||
if "policy_hierarchy_rules" not in self.plan.touched:
|
||||
return
|
||||
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
|
||||
for key, rank in self.plan.positions.items():
|
||||
row = self.rows["policy_hierarchy_rules"].get(key)
|
||||
if row is not None and key not in deleted and row.sequence != rank:
|
||||
row.sequence = rank
|
||||
await self.db.flush()
|
||||
|
||||
async def host_hierarchy(self) -> None:
|
||||
for host_name, group_name in self.plan.host_hierarchy.items():
|
||||
host = self.hosts[host_name]
|
||||
wanted = self.hierarchy[group_name].id if group_name else None
|
||||
if host.policy_hierarchy_group_id != wanted:
|
||||
host.policy_hierarchy_group_id = wanted
|
||||
await self.db.flush()
|
||||
|
||||
async def variables(self) -> None:
|
||||
if "variables" not in self.plan.touched:
|
||||
return
|
||||
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
|
||||
for item in self.items("variables", "create", "update"):
|
||||
host = self.hosts[item.after["host"]]
|
||||
row = existing.get((host.id, item.after["name"]))
|
||||
if row is None:
|
||||
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
|
||||
else:
|
||||
row.value = item.after["value"]
|
||||
self.event(item, None, item.key)
|
||||
for item in self.items("variables", "delete"):
|
||||
host = self.hosts[item.before["host"]]
|
||||
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
|
||||
self.event(item, None, item.key)
|
||||
await self.db.flush()
|
||||
|
||||
# -- deletions -------------------------------------------------------------------------------------------------------
|
||||
async def deletions(self) -> None:
|
||||
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
|
||||
for item in self.items(kind, "delete"):
|
||||
row = self.rows[kind][item.key]
|
||||
if row.deployments: # the planner blocks this; never undeploy from here
|
||||
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
for item in self.items("device_groups", "delete"):
|
||||
row = self.groups[item.key]
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
|
||||
by_id = {g.id: g for g in self.hierarchy.values()}
|
||||
|
||||
def depth(group) -> int:
|
||||
n, current = 0, group
|
||||
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
|
||||
n, current = n + 1, by_id[current.parent_id]
|
||||
return n
|
||||
|
||||
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
|
||||
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
|
||||
|
||||
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
|
||||
db.expire_all() # compare what is in the database, not what the session believes
|
||||
actual = await canonical(db, set(plan.touched))
|
||||
differences: list[str] = []
|
||||
for kind in plan.touched:
|
||||
if kind == "variables":
|
||||
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in actual["variables"].items() for n, v in values.items()}
|
||||
else:
|
||||
ident = planner.KIND_SPECS[kind].ident
|
||||
got = {item[ident]: item for item in actual[kind]}
|
||||
want = plan.expected[kind]
|
||||
for key in sorted(set(got) | set(want)):
|
||||
if got.get(key) != want.get(key):
|
||||
differences.append(f"{kind}:{key}")
|
||||
if "policy_hierarchy_rules" in plan.touched:
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
|
||||
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
|
||||
for key, rank in ranks.items():
|
||||
if plan.positions.get(key) != rank:
|
||||
differences.append(f"policy_hierarchy_rules:{key}:position")
|
||||
if differences:
|
||||
raise SelfCheckFailed(differences)
|
||||
|
||||
|
||||
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
|
||||
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
|
||||
ConfigError; on any exception the caller must roll back."""
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
await policy_lock(db)
|
||||
plan = await planner.build_plan(db, parsed, prune)
|
||||
if plan.token != token:
|
||||
raise PlanStale()
|
||||
if not plan.applicable:
|
||||
raise PlanBlocked(plan)
|
||||
writer = _Writer(db, plan)
|
||||
await writer.load()
|
||||
await writer.hierarchy_groups()
|
||||
await writer.device_groups()
|
||||
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
|
||||
if kind in plan.touched:
|
||||
await writer.objects(kind)
|
||||
await writer.hierarchy_positions()
|
||||
await writer.host_hierarchy()
|
||||
await writer.variables()
|
||||
await writer.deletions()
|
||||
await _self_check(db, plan)
|
||||
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
|
||||
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
|
||||
"summary": plan.summary()}))
|
||||
for values in writer.events:
|
||||
db.add(audit_service.make_event(**values))
|
||||
await db.flush()
|
||||
return plan
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
|
||||
|
||||
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
|
||||
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
|
||||
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
|
||||
"""
|
||||
import yaml
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.version import APP_VERSION
|
||||
|
||||
HEADER = (
|
||||
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
|
||||
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
|
||||
)
|
||||
|
||||
|
||||
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
|
||||
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
|
||||
fields = list(doc.model_fields)
|
||||
if set(values) != set(fields):
|
||||
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
|
||||
return {name: values[name] for name in fields}
|
||||
|
||||
|
||||
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
|
||||
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
|
||||
if "hierarchy_group" in doc.model_fields:
|
||||
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
|
||||
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
|
||||
values = dict(extra)
|
||||
for name in doc.model_fields:
|
||||
if name in values:
|
||||
continue
|
||||
values[name] = obj.config_key if name == "key" else getattr(obj, name)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
|
||||
"""Parents before children, siblings by name; the same rule the loader expects."""
|
||||
children: dict = {}
|
||||
for group in groups:
|
||||
children.setdefault(group.parent_id, []).append(group)
|
||||
ordered: list[PolicyHierarchyGroup] = []
|
||||
|
||||
def walk(parent_id) -> None:
|
||||
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
|
||||
ordered.append(group)
|
||||
walk(group.id)
|
||||
|
||||
walk(None)
|
||||
return ordered
|
||||
|
||||
|
||||
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
|
||||
"""{kind: content} for the requested kinds (default all), in file order."""
|
||||
wanted = set(schema.KINDS) if kinds is None else kinds
|
||||
out: dict[str, list | dict] = {}
|
||||
|
||||
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
|
||||
hierarchy_names = {g.id: g.name for g in all_groups}
|
||||
|
||||
if "device_groups" in wanted:
|
||||
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
|
||||
out["device_groups"] = [
|
||||
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
|
||||
for g in sorted(groups, key=lambda g: g.name)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_groups" in wanted:
|
||||
out["policy_hierarchy_groups"] = [
|
||||
_ordered(schema.HierarchyGroupDoc, {
|
||||
"name": g.name,
|
||||
"parent": hierarchy_names.get(g.parent_id),
|
||||
"description": g.description or "",
|
||||
"hosts": sorted(h.name for h in g.hosts),
|
||||
})
|
||||
for g in _hierarchy_order(list(all_groups))
|
||||
]
|
||||
|
||||
if "aliases" in wanted:
|
||||
aliases = (await db.execute(
|
||||
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
|
||||
)).scalars().all()
|
||||
out["aliases"] = [
|
||||
_ordered(schema.AliasDoc, {
|
||||
"name": a.name,
|
||||
"type": a.type,
|
||||
"description": a.description or "",
|
||||
"enabled": a.enabled,
|
||||
"entries": [line for line in (a.content or "").split("\n") if line],
|
||||
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
|
||||
})
|
||||
for a in sorted(aliases, key=lambda a: a.name)
|
||||
]
|
||||
|
||||
simple = (
|
||||
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
|
||||
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
|
||||
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
|
||||
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
|
||||
)
|
||||
for kind, model, doc, targets_doc in simple:
|
||||
if kind not in wanted:
|
||||
continue
|
||||
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
|
||||
out[kind] = [
|
||||
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
|
||||
for row in sorted(rows, key=lambda r: r.config_key)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_rules" in wanted:
|
||||
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
|
||||
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
|
||||
out["policy_hierarchy_rules"] = [
|
||||
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
|
||||
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
|
||||
]
|
||||
|
||||
if "variables" in wanted:
|
||||
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
|
||||
out["variables"] = {
|
||||
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
|
||||
for h in sorted(hosts, key=lambda h: h.name)
|
||||
if h.variables
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def dump_yaml(content: dict[str, list | dict]) -> str:
|
||||
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
|
||||
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
|
||||
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
|
||||
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
|
||||
|
||||
|
||||
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
|
||||
return dump_yaml(await canonical(db, kinds))
|
||||
@@ -0,0 +1,306 @@
|
||||
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
|
||||
|
||||
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
|
||||
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
|
||||
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
|
||||
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
|
||||
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
|
||||
|
||||
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
|
||||
"""
|
||||
import hashlib
|
||||
import re
|
||||
from typing import NamedTuple
|
||||
|
||||
import yaml
|
||||
from pydantic import ValidationError
|
||||
from yaml.composer import Composer
|
||||
from yaml.constructor import SafeConstructor
|
||||
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
|
||||
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
|
||||
from yaml.parser import Parser
|
||||
from yaml.reader import Reader
|
||||
from yaml.resolver import BaseResolver
|
||||
from yaml.scanner import Scanner
|
||||
|
||||
from app.config_code import schema
|
||||
|
||||
MAX_BYTES = 1024 * 1024
|
||||
MAX_DEPTH = 8
|
||||
MAX_NODES = 100_000
|
||||
MAX_SCALAR = schema.MAX_SCALAR
|
||||
MAX_OBJECTS_PER_KIND = 5000
|
||||
MAX_VARIABLE_HOSTS = 500
|
||||
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
|
||||
MAX_ALIAS_ENTRIES = 2000
|
||||
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
|
||||
|
||||
_STR = "tag:yaml.org,2002:str"
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
|
||||
|
||||
def __init__(self, code: str, errors: list[dict]):
|
||||
self.code = code
|
||||
self.errors = errors
|
||||
super().__init__(f"{code}: {errors[:3]}")
|
||||
|
||||
|
||||
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
|
||||
return {"code": code, "path": path, "line": line, "params": params}
|
||||
|
||||
|
||||
def display(value, limit: int = 80) -> str:
|
||||
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
|
||||
text = value if isinstance(value, str) else repr(value)
|
||||
text = text.encode("unicode_escape").decode("ascii")
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def _path(parts) -> str:
|
||||
out = ""
|
||||
for part in parts:
|
||||
if isinstance(part, int):
|
||||
out += f"[{part}]"
|
||||
else:
|
||||
out += ("." if out else "") + str(part)
|
||||
return out
|
||||
|
||||
|
||||
class _Rejected(yaml.YAMLError):
|
||||
def __init__(self, code: str, mark=None, path=(), **params):
|
||||
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
class StrictResolver(BaseResolver):
|
||||
yaml_implicit_resolvers: dict = {}
|
||||
|
||||
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
|
||||
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
|
||||
|
||||
|
||||
class StrictConstructor(SafeConstructor):
|
||||
yaml_constructors: dict = {}
|
||||
|
||||
|
||||
for _tag, _fn in (
|
||||
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
|
||||
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
|
||||
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
|
||||
(_STR, SafeConstructor.construct_yaml_str),
|
||||
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
|
||||
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
|
||||
):
|
||||
StrictConstructor.add_constructor(_tag, _fn)
|
||||
|
||||
|
||||
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
|
||||
def __init__(self, stream: str):
|
||||
Reader.__init__(self, stream)
|
||||
Scanner.__init__(self)
|
||||
Parser.__init__(self)
|
||||
Composer.__init__(self)
|
||||
StrictConstructor.__init__(self)
|
||||
StrictResolver.__init__(self)
|
||||
self._depth = 0
|
||||
self._nodes = 0
|
||||
self._path: list = []
|
||||
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
|
||||
|
||||
# -- document level ---------------------------------------------------------------------------------------------
|
||||
def get_single_node(self):
|
||||
self.get_event() # StreamStart
|
||||
if self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_empty")
|
||||
start = self.peek_event()
|
||||
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
|
||||
raise _Rejected("yaml_directive_forbidden", start.start_mark)
|
||||
node = self.compose_document()
|
||||
if not self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
|
||||
self.get_event()
|
||||
return node
|
||||
|
||||
# -- node level ---------------------------------------------------------------------------------------------------
|
||||
def compose_node(self, parent, index):
|
||||
event = self.peek_event()
|
||||
if isinstance(event, AliasEvent):
|
||||
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "anchor", None) is not None:
|
||||
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "tag", None) is not None:
|
||||
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
|
||||
self._nodes += 1
|
||||
if self._nodes > MAX_NODES:
|
||||
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
|
||||
|
||||
if isinstance(index, ScalarNode):
|
||||
segment = index.value
|
||||
elif isinstance(index, int) and not isinstance(index, bool):
|
||||
segment = index
|
||||
else:
|
||||
segment = None # a mapping key
|
||||
if segment is not None:
|
||||
self._path.append(segment)
|
||||
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
|
||||
self._depth += 1
|
||||
try:
|
||||
if self._depth > MAX_DEPTH:
|
||||
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
|
||||
return super().compose_node(parent, index)
|
||||
finally:
|
||||
self._depth -= 1
|
||||
if segment is not None:
|
||||
self._path.pop()
|
||||
|
||||
def compose_scalar_node(self, anchor):
|
||||
event = self.peek_event()
|
||||
if len(event.value) > MAX_SCALAR:
|
||||
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
|
||||
return super().compose_scalar_node(anchor)
|
||||
|
||||
def compose_mapping_node(self, anchor):
|
||||
node = super().compose_mapping_node(anchor)
|
||||
seen: set[str] = set()
|
||||
for key, _value in node.value:
|
||||
if not isinstance(key, ScalarNode) or key.tag != _STR:
|
||||
raise _Rejected("non_string_key", key.start_mark, self._path)
|
||||
if key.value == "<<":
|
||||
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
|
||||
if key.value in seen:
|
||||
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
|
||||
seen.add(key.value)
|
||||
return node
|
||||
|
||||
|
||||
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
|
||||
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
if len(raw) > MAX_BYTES:
|
||||
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
|
||||
try:
|
||||
text = raw.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
|
||||
if text.startswith(""):
|
||||
text = text[1:]
|
||||
loader = None
|
||||
try:
|
||||
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
|
||||
node = loader.get_single_node()
|
||||
data = loader.construct_document(node)
|
||||
except _Rejected as exc:
|
||||
line = exc.mark.line + 1 if exc.mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
|
||||
except yaml.MarkedYAMLError as exc:
|
||||
mark = exc.problem_mark or exc.context_mark
|
||||
line = mark.line + 1 if mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
|
||||
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
|
||||
finally:
|
||||
if loader is not None:
|
||||
loader.dispose()
|
||||
return data, loader.lines
|
||||
|
||||
|
||||
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
|
||||
for end in range(len(loc), 0, -1):
|
||||
if loc[:end] in lines:
|
||||
return lines[loc[:end]]
|
||||
return None
|
||||
|
||||
|
||||
_PYDANTIC_CODES = {
|
||||
"extra_forbidden": "unknown_field",
|
||||
"missing": "missing_field",
|
||||
"literal_error": "bad_value",
|
||||
"string_too_short": "too_short",
|
||||
"string_too_long": "too_long",
|
||||
"string_pattern_mismatch": "bad_format",
|
||||
}
|
||||
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
|
||||
|
||||
|
||||
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
|
||||
out = []
|
||||
for error in exc.errors(include_url=False):
|
||||
loc = tuple(error["loc"])
|
||||
code = _PYDANTIC_CODES.get(error["type"])
|
||||
if code is None and error["type"] == "value_error":
|
||||
message = str(error.get("ctx", {}).get("error", ""))
|
||||
code = message if message in _OWN_CODES else "bad_value"
|
||||
if code is None:
|
||||
code = "bad_type"
|
||||
params = {}
|
||||
if error["type"] == "extra_forbidden":
|
||||
params["field"] = display(loc[-1]) if loc else ""
|
||||
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
|
||||
params["value"] = display(error["input"])
|
||||
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
|
||||
return out
|
||||
|
||||
|
||||
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
|
||||
errors = []
|
||||
for kind in schema.KINDS:
|
||||
section = data.get(kind)
|
||||
if kind == "variables":
|
||||
if isinstance(section, dict):
|
||||
if len(section) > MAX_VARIABLE_HOSTS:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
|
||||
for host, values in section.items():
|
||||
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
|
||||
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
|
||||
max=MAX_VARIABLES_PER_HOST))
|
||||
continue
|
||||
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
|
||||
for i, alias in enumerate(data.get("aliases") or []):
|
||||
entries = alias.get("entries") if isinstance(alias, dict) else None
|
||||
if isinstance(entries, list):
|
||||
loc = ("aliases", i, "entries")
|
||||
if len(entries) > MAX_ALIAS_ENTRIES:
|
||||
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
|
||||
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
|
||||
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
|
||||
return errors
|
||||
|
||||
|
||||
class Parsed(NamedTuple):
|
||||
doc: schema.ConfigDoc
|
||||
sha256: str # of the raw bytes, shown in the plan and stored in the audit
|
||||
lines: dict[tuple, int] # path -> line, for messages of the planner
|
||||
|
||||
|
||||
def parse(raw: bytes | str) -> Parsed:
|
||||
"""Load, limit and validate a file. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
sha256 = hashlib.sha256(raw).hexdigest()
|
||||
data, lines = load_yaml(raw)
|
||||
if not isinstance(data, dict):
|
||||
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
|
||||
version = data.get("opnexus_config")
|
||||
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
|
||||
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
|
||||
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
|
||||
"opnexus_config", _line_for(lines, ("opnexus_config",)),
|
||||
**({"value": display(version)} if version is not None else {}))])
|
||||
errors = _check_counts(data, lines)
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors)
|
||||
try:
|
||||
doc = schema.ConfigDoc.model_validate(data)
|
||||
except ValidationError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
|
||||
return Parsed(doc, sha256, lines)
|
||||
@@ -0,0 +1,933 @@
|
||||
{
|
||||
"$defs": {
|
||||
"AliasDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"pattern": "^[A-Za-z0-9_]+$",
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "host",
|
||||
"enum": [
|
||||
"host",
|
||||
"network",
|
||||
"port",
|
||||
"networkgroup",
|
||||
"mac",
|
||||
"asn"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 255,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"entries": {
|
||||
"items": {
|
||||
"maxLength": 8000,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Entries",
|
||||
"type": "array"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "AliasDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"DeviceGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "DeviceGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"parent": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Parent"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "HierarchyGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyRuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"group": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Group",
|
||||
"type": "string"
|
||||
},
|
||||
"rulebase": {
|
||||
"default": "pre",
|
||||
"enum": [
|
||||
"pre",
|
||||
"post"
|
||||
],
|
||||
"title": "Rulebase",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"group",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "HierarchyRuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OneToOneDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "binat",
|
||||
"enum": [
|
||||
"binat",
|
||||
"nat"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"external": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "External",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"enable",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"external"
|
||||
],
|
||||
"title": "OneToOneDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OutboundNatDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"target": {
|
||||
"default": "",
|
||||
"maxLength": 128,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"target_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Target Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "OutboundNatDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"PortForwardDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "tcp",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"target": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"local_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Local Port"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"purenat",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"target"
|
||||
],
|
||||
"title": "PortForwardDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"RuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/Targets"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "RuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"Targets": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"title": "Targets",
|
||||
"type": "object"
|
||||
},
|
||||
"TargetsWithHierarchy": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
},
|
||||
"hierarchy_group": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Hierarchy Group"
|
||||
}
|
||||
},
|
||||
"title": "TargetsWithHierarchy",
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
|
||||
"properties": {
|
||||
"opnexus_config": {
|
||||
"const": 1,
|
||||
"title": "Opnexus Config",
|
||||
"type": "integer"
|
||||
},
|
||||
"device_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/DeviceGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Device Groups"
|
||||
},
|
||||
"policy_hierarchy_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Groups"
|
||||
},
|
||||
"aliases": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/AliasDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Aliases"
|
||||
},
|
||||
"rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/RuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Rules"
|
||||
},
|
||||
"port_forwards": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/PortForwardDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Port Forwards"
|
||||
},
|
||||
"one_to_one": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OneToOneDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "One To One"
|
||||
},
|
||||
"outbound_nat": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OutboundNatDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Outbound Nat"
|
||||
},
|
||||
"policy_hierarchy_rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyRuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Rules"
|
||||
},
|
||||
"variables": {
|
||||
"anyOf": [
|
||||
{
|
||||
"additionalProperties": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"propertyNames": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Variables"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"opnexus_config"
|
||||
],
|
||||
"title": "OPNexus-Konfiguration (Schema v1)",
|
||||
"type": "object",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
}
|
||||
@@ -0,0 +1,570 @@
|
||||
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
|
||||
|
||||
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
|
||||
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
|
||||
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
|
||||
`prune` gleicht die Art ab.
|
||||
|
||||
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
|
||||
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
|
||||
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
|
||||
|
||||
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
|
||||
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
|
||||
from app.database import Base
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.schemas.alias import AliasCreate
|
||||
from app.schemas.one_to_one import OneToOneCreate
|
||||
from app.schemas.outbound_nat import OutboundNatCreate
|
||||
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||
from app.schemas.port_forward import PortForwardCreate
|
||||
from app.schemas.rule import RuleCreate
|
||||
from app.services.variables_service import names_in
|
||||
|
||||
BUSY = ("pending", "revert_failed")
|
||||
DEPLOYED = ("success", "pending", "revert_failed")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
section: str
|
||||
model: type | None
|
||||
ident: str # identity field in the file ("key"/"name")
|
||||
api: type | None = None # API create schema, run on every file object (same validators as the UI)
|
||||
deployable: bool = False
|
||||
|
||||
|
||||
KIND_SPECS = {
|
||||
"device_groups": Kind("device_groups", DeviceGroup, "name"),
|
||||
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
|
||||
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
|
||||
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
|
||||
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
|
||||
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
|
||||
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
|
||||
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
|
||||
"variables": Kind("variables", None, "key"),
|
||||
}
|
||||
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
|
||||
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
|
||||
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
|
||||
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
|
||||
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
|
||||
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
|
||||
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
|
||||
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
kind: str
|
||||
key: str
|
||||
action: str # create | update | delete | unchanged
|
||||
before: dict | None = None
|
||||
after: dict | None = None
|
||||
changes: list = field(default_factory=list)
|
||||
reasons: list = field(default_factory=list)
|
||||
deployed_hosts: int = 0
|
||||
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
|
||||
db_id: uuid.UUID | None = None
|
||||
|
||||
def public(self) -> dict:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"key": self.key,
|
||||
"action": "blocked" if self.reasons else self.action,
|
||||
"planned_action": self.action,
|
||||
"changes": self.changes,
|
||||
"deployed_hosts": self.deployed_hosts,
|
||||
"reasons": self.reasons,
|
||||
"implicit": self.implicit,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Plan:
|
||||
sha256: str
|
||||
prune: bool
|
||||
touched: list[str]
|
||||
items: list[Item]
|
||||
warnings: list[dict]
|
||||
token: str = ""
|
||||
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
|
||||
expected: dict = field(default_factory=dict)
|
||||
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
|
||||
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
|
||||
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
|
||||
|
||||
@property
|
||||
def applicable(self) -> bool:
|
||||
return not any(i.reasons for i in self.items)
|
||||
|
||||
def summary(self) -> dict:
|
||||
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||
for item in self.items:
|
||||
out["blocked" if item.reasons else item.action] += 1
|
||||
return out
|
||||
|
||||
def public(self) -> dict:
|
||||
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
|
||||
return {
|
||||
"token": self.token,
|
||||
"sha256": self.sha256,
|
||||
"prune": self.prune,
|
||||
"kinds": self.touched,
|
||||
"summary": self.summary(),
|
||||
"items": [i.public() for i in changed],
|
||||
"warnings": self.warnings,
|
||||
"applicable": self.applicable,
|
||||
}
|
||||
|
||||
|
||||
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
|
||||
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
|
||||
|
||||
|
||||
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
|
||||
section = getattr(doc, kind)
|
||||
if kind == "variables":
|
||||
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
|
||||
for host, values in section.items() for name, value in values.items()]
|
||||
return [item.model_dump(mode="python") for item in section]
|
||||
|
||||
|
||||
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
|
||||
def sorted_unique(values: list, where: tuple) -> list:
|
||||
seen = set()
|
||||
for i, value in enumerate(values):
|
||||
if value in seen:
|
||||
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
|
||||
seen.add(value)
|
||||
return sorted(seen)
|
||||
|
||||
item = dict(item)
|
||||
if "targets" in item:
|
||||
targets = dict(item["targets"])
|
||||
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
|
||||
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
|
||||
item["targets"] = targets
|
||||
if kind in ("device_groups", "policy_hierarchy_groups"):
|
||||
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
|
||||
return item
|
||||
|
||||
|
||||
def _changes(before: dict, after: dict) -> list[dict]:
|
||||
out = []
|
||||
for name in after:
|
||||
if name == "targets":
|
||||
for sub in after["targets"]:
|
||||
if before["targets"].get(sub) != after["targets"][sub]:
|
||||
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
|
||||
elif before.get(name) != after[name]:
|
||||
out.append({"field": name, "before": before.get(name), "after": after[name]})
|
||||
return out
|
||||
|
||||
|
||||
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
|
||||
buckets: dict = {}
|
||||
for row in rows:
|
||||
buckets.setdefault(bucket_of(row), []).append(row)
|
||||
out = {}
|
||||
for members in buckets.values():
|
||||
for rank, row in enumerate(sorted(members, key=order_of)):
|
||||
out[key_of(row)] = rank
|
||||
return out
|
||||
|
||||
|
||||
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
|
||||
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
|
||||
found: dict = {}
|
||||
if not ids:
|
||||
return found
|
||||
for table in Base.metadata.sorted_tables:
|
||||
if table.name in skip:
|
||||
continue
|
||||
for fk in table.foreign_keys:
|
||||
if fk.column.table.name != target_table:
|
||||
continue
|
||||
column = fk.parent
|
||||
rows = (await db.execute(
|
||||
select(column, func.count()).where(column.in_(ids)).group_by(column)
|
||||
)).all()
|
||||
for ref_id, _count in rows:
|
||||
found.setdefault(ref_id, []).append(table.name)
|
||||
return found
|
||||
|
||||
|
||||
async def _load(db: AsyncSession) -> dict:
|
||||
loaded: dict = {}
|
||||
for kind in TARGET_KINDS:
|
||||
model = KIND_SPECS[kind].model
|
||||
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
|
||||
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
|
||||
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
|
||||
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
|
||||
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
return loaded
|
||||
|
||||
|
||||
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
|
||||
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
|
||||
doc, lines = parsed.doc, parsed.lines
|
||||
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
|
||||
errors: list[dict] = []
|
||||
warnings: list[dict] = []
|
||||
|
||||
db_canon = await canonical(db)
|
||||
loaded = await _load(db)
|
||||
hosts = loaded["hosts"]
|
||||
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
|
||||
|
||||
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
|
||||
file_objects: dict[str, dict[str, dict]] = {}
|
||||
file_locs: dict[tuple, tuple] = {}
|
||||
for kind in touched:
|
||||
spec = KIND_SPECS[kind]
|
||||
objects: dict[str, dict] = {}
|
||||
for index, raw in enumerate(_file_items(doc, kind)):
|
||||
if kind == "variables":
|
||||
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
|
||||
else:
|
||||
ident, loc = raw[spec.ident], (kind, index)
|
||||
if ident in objects:
|
||||
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
|
||||
continue
|
||||
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
|
||||
file_locs[(kind, ident)] = loc
|
||||
file_objects[kind] = objects
|
||||
|
||||
def db_items(kind: str) -> dict[str, dict]:
|
||||
if kind == "variables":
|
||||
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in db_canon["variables"].items() for n, v in values.items()}
|
||||
ident = KIND_SPECS[kind].ident
|
||||
return {item[ident]: item for item in db_canon[kind]}
|
||||
|
||||
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
|
||||
|
||||
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
|
||||
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
|
||||
post: dict[str, dict[str, dict]] = {}
|
||||
for kind in schema.KINDS:
|
||||
if kind in file_objects:
|
||||
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
|
||||
if prune:
|
||||
deleted[kind] = set(remaining)
|
||||
remaining = {}
|
||||
post[kind] = {**remaining, **file_objects[kind]}
|
||||
else:
|
||||
post[kind] = dict(db_objects[kind])
|
||||
|
||||
host_names = set(hosts)
|
||||
group_names = set(post["device_groups"])
|
||||
hierarchy_names = set(post["policy_hierarchy_groups"])
|
||||
|
||||
# -- references in the file ------------------------------------------------------------------------------------------
|
||||
for kind in touched:
|
||||
for ident, item in file_objects[kind].items():
|
||||
loc = file_locs[(kind, ident)]
|
||||
if kind == "variables":
|
||||
if item["host"] not in host_names:
|
||||
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
|
||||
continue
|
||||
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
|
||||
for name in member_hosts:
|
||||
if name not in host_names:
|
||||
_err(errors, "unknown_host", loc, lines, host=display(name))
|
||||
for name in item.get("targets", {}).get("groups", []):
|
||||
if name not in group_names:
|
||||
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
|
||||
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
|
||||
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
|
||||
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
|
||||
if ref is not None and ref not in hierarchy_names:
|
||||
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
|
||||
if kind == "policy_hierarchy_rules":
|
||||
for name, value in item.items():
|
||||
if names_in(value):
|
||||
_err(errors, "placeholder_not_supported", loc + (name,), lines)
|
||||
spec = KIND_SPECS[kind]
|
||||
if spec.api is not None:
|
||||
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
|
||||
for name in member_hosts:
|
||||
host = hosts.get(name)
|
||||
if host is not None and not host.enabled:
|
||||
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
|
||||
|
||||
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
|
||||
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
|
||||
if "policy_hierarchy_groups" in file_objects:
|
||||
file_groups = file_objects["policy_hierarchy_groups"]
|
||||
for host_name, group in list(host_hierarchy.items()):
|
||||
if prune or group in file_groups:
|
||||
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
|
||||
host_hierarchy[host_name] = None
|
||||
listed: dict[str, str] = {}
|
||||
for name, group in file_groups.items():
|
||||
for host_name in group["hosts"]:
|
||||
if host_name in listed:
|
||||
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
|
||||
lines, host=display(host_name), first=display(listed[host_name]))
|
||||
listed[host_name] = name
|
||||
if host_name in host_hierarchy:
|
||||
host_hierarchy[host_name] = name
|
||||
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
|
||||
for start in parents:
|
||||
seen, current = set(), start
|
||||
while current is not None and current not in seen:
|
||||
seen.add(current)
|
||||
current = parents.get(current)
|
||||
if current is not None and current == start:
|
||||
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
|
||||
lines, group=display(start))
|
||||
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
|
||||
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
|
||||
|
||||
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
|
||||
per_host: dict[str, int] = {}
|
||||
for item in post["variables"].values():
|
||||
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
|
||||
for host_name, count in sorted(per_host.items()):
|
||||
if count > MAX_VARIABLES_PER_HOST:
|
||||
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
|
||||
max=MAX_VARIABLES_PER_HOST, count=count)
|
||||
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors[:100])
|
||||
|
||||
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
|
||||
db_rule_rows = loaded["policy_hierarchy_rules"]
|
||||
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
|
||||
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
|
||||
positions = dict(db_positions)
|
||||
if "policy_hierarchy_rules" in file_objects:
|
||||
buckets: dict[tuple, list[str]] = {}
|
||||
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
|
||||
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for key, item in post["policy_hierarchy_rules"].items():
|
||||
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
|
||||
buckets.setdefault((item["group"], item["rulebase"]), [])
|
||||
positions = {}
|
||||
extras: dict[tuple, list[str]] = {}
|
||||
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
|
||||
key=lambda k: db_positions.get(k, 0)):
|
||||
item = post["policy_hierarchy_rules"][key]
|
||||
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for bucket in set(buckets) | set(extras):
|
||||
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
|
||||
positions[key] = rank
|
||||
|
||||
# -- items -----------------------------------------------------------------------------------------------------------
|
||||
items: list[Item] = []
|
||||
for kind in touched:
|
||||
rows = loaded.get(kind, {})
|
||||
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
|
||||
row = rows.get(ident) if kind != "variables" else None
|
||||
before = db_objects[kind].get(ident)
|
||||
after = post[kind].get(ident) if ident not in deleted[kind] else None
|
||||
if before is None:
|
||||
item = Item(kind, ident, "create", None, after)
|
||||
elif after is None:
|
||||
item = Item(kind, ident, "delete", before, None)
|
||||
else:
|
||||
changes = _changes(before, after)
|
||||
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
|
||||
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
|
||||
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
|
||||
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
|
||||
if row is not None:
|
||||
item.db_id = row.id
|
||||
deployments = getattr(row, "deployments", None) or []
|
||||
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
|
||||
if item.action in ("update", "delete") and any(d.status in BUSY for d in deployments):
|
||||
item.reasons.append("busy_deployment")
|
||||
if item.action == "delete" and deployments:
|
||||
item.reasons.append("prune_deployed")
|
||||
items.append(item)
|
||||
|
||||
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
|
||||
|
||||
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
|
||||
if redeploy:
|
||||
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
|
||||
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
|
||||
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
|
||||
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
|
||||
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
|
||||
warnings += _alias_reference_warnings(post, file_objects, file_locs)
|
||||
warnings += _variable_warnings(post, deleted)
|
||||
|
||||
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
|
||||
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
|
||||
file_objects=file_objects)
|
||||
plan.expected = {kind: post[kind] for kind in touched}
|
||||
plan.token = _token(plan, db_canon, db_positions, loaded)
|
||||
return plan
|
||||
|
||||
|
||||
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
|
||||
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
|
||||
out = set()
|
||||
if kind == "policy_hierarchy_rules":
|
||||
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
|
||||
if kind == "policy_hierarchy_groups":
|
||||
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
|
||||
return out
|
||||
|
||||
|
||||
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
|
||||
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
|
||||
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
|
||||
if kind == "aliases":
|
||||
values["content"] = "\n".join(item["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
values["hierarchy_group_id"] = uuid.UUID(int=0)
|
||||
try:
|
||||
cleaned = KIND_SPECS[kind].api.model_validate(values)
|
||||
except ValidationError as exc:
|
||||
for error in exc.errors(include_url=False):
|
||||
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
|
||||
return item
|
||||
out = dict(item)
|
||||
for name in item:
|
||||
if name in values and name != "content" and hasattr(cleaned, name):
|
||||
out[name] = getattr(cleaned, name)
|
||||
if kind == "aliases":
|
||||
out["entries"] = [e for e in cleaned.content.split("\n") if e]
|
||||
return out
|
||||
|
||||
|
||||
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
|
||||
by_key = {(i.kind, i.key): i for i in items}
|
||||
|
||||
# device groups: post-state objects of the import, plus every other table that points at the group
|
||||
if deleted["device_groups"]:
|
||||
users: dict[str, list[str]] = {}
|
||||
for kind in TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
for group in item["targets"]["groups"]:
|
||||
users.setdefault(group, []).append(f"{kind}:{ident}")
|
||||
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
|
||||
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
|
||||
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["device_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("device_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["policy_hierarchy_groups"]:
|
||||
users = {}
|
||||
for kind in HIERARCHY_TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
if item["targets"]["hierarchy_group"]:
|
||||
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
|
||||
for ident, item in post["policy_hierarchy_rules"].items():
|
||||
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
|
||||
for name, item in post["policy_hierarchy_groups"].items():
|
||||
if item["parent"]:
|
||||
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
|
||||
for host_name, group in host_hierarchy.items():
|
||||
if group:
|
||||
users.setdefault(group, []).append(f"host:{host_name}")
|
||||
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
|
||||
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
|
||||
"policy_hierarchy_rules", "opnsense_hosts"}
|
||||
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["policy_hierarchy_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["aliases"]:
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS:
|
||||
for item in post[kind].values():
|
||||
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
|
||||
for item in post["aliases"].values():
|
||||
used.update(item["entries"])
|
||||
for name in deleted["aliases"]:
|
||||
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
|
||||
by_key[("aliases", name)].reasons.append("prune_referenced")
|
||||
|
||||
|
||||
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
|
||||
aliases = set(post["aliases"])
|
||||
out = []
|
||||
for kind in RULE_KINDS:
|
||||
for ident, item in file_objects.get(kind, {}).items():
|
||||
for name in ALIAS_REF_FIELDS:
|
||||
value = item.get(name)
|
||||
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
|
||||
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
|
||||
"params": {"value": display(value)}})
|
||||
return out
|
||||
|
||||
|
||||
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
|
||||
if not deleted["variables"]:
|
||||
return []
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS + ("aliases",):
|
||||
for item in post[kind].values():
|
||||
for value in item.values():
|
||||
if isinstance(value, str):
|
||||
used |= names_in(value)
|
||||
elif isinstance(value, list):
|
||||
for entry in value:
|
||||
used |= names_in(entry)
|
||||
out = []
|
||||
for key in sorted(deleted["variables"]):
|
||||
host, name = key.split("/", 1)
|
||||
if name in used:
|
||||
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
|
||||
return out
|
||||
|
||||
|
||||
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
|
||||
deployments = {
|
||||
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
|
||||
for ident, row in loaded[kind].items()}
|
||||
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
|
||||
}
|
||||
material = {
|
||||
"file": plan.file_objects,
|
||||
"prune": plan.prune,
|
||||
"db": db_canon,
|
||||
"positions": db_positions,
|
||||
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
|
||||
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
|
||||
"deployments": deployments,
|
||||
"items": [i.public() for i in plan.items],
|
||||
"warnings": plan.warnings,
|
||||
}
|
||||
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
|
||||
@@ -0,0 +1,242 @@
|
||||
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
|
||||
|
||||
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
|
||||
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
|
||||
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
|
||||
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
|
||||
"""
|
||||
import re
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
|
||||
|
||||
from app.models.config_key import CONFIG_KEY_RE
|
||||
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
MAX_SCALAR = 8000
|
||||
|
||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
|
||||
|
||||
|
||||
def _clean(value: str) -> str:
|
||||
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
if value != value.strip():
|
||||
raise ValueError("surrounding_whitespace")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_value(value: str) -> str:
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_text(value: str) -> str:
|
||||
if _CONTROL_TEXT.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _port(value):
|
||||
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
|
||||
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
|
||||
|
||||
|
||||
Text = Annotated[str, AfterValidator(_clean)]
|
||||
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
|
||||
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
|
||||
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
|
||||
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
|
||||
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
|
||||
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
|
||||
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
|
||||
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
|
||||
Port = Annotated[PortText | None, BeforeValidator(_port)]
|
||||
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
|
||||
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
|
||||
|
||||
IpProtocol = Literal["inet", "inet6"]
|
||||
|
||||
|
||||
class _Doc(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
class Targets(_Doc):
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
groups: list[GroupName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class TargetsWithHierarchy(Targets):
|
||||
hierarchy_group: GroupName | None = None
|
||||
|
||||
|
||||
class DeviceGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class HierarchyGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
parent: GroupName | None = None
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AliasDoc(_Doc):
|
||||
name: AliasName
|
||||
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
|
||||
description: Description = ""
|
||||
enabled: bool = True
|
||||
entries: list[AliasEntry] = Field(default_factory=list)
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class RuleDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
log: bool = False
|
||||
targets: Targets = Field(default_factory=Targets)
|
||||
|
||||
|
||||
class PortForwardDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "tcp"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
target: Net
|
||||
local_port: Port = None
|
||||
natreflection: Literal["", "purenat", "disable"] = ""
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OneToOneDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
type: Literal["binat", "nat"] = "binat"
|
||||
external: Net
|
||||
source_net: Net = "any"
|
||||
destination_net: Net = "any"
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
natreflection: Literal["", "enable", "disable"] = ""
|
||||
log: bool = False
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OutboundNatDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
|
||||
target_port: Port = None
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class HierarchyRuleDoc(_Doc):
|
||||
key: Key
|
||||
group: GroupName
|
||||
rulebase: Literal["pre", "post"] = "pre"
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
def _variable_name(value: str) -> str:
|
||||
if not VARIABLE_NAME_RE.match(value):
|
||||
raise ValueError("bad_variable_name")
|
||||
return value
|
||||
|
||||
|
||||
def _variable_value(value: str) -> str:
|
||||
if not VARIABLE_VALUE_RE.match(value):
|
||||
raise ValueError("bad_variable_value")
|
||||
return value
|
||||
|
||||
|
||||
VariableName = Annotated[str, AfterValidator(_variable_name)]
|
||||
VariableValue = Annotated[str, AfterValidator(_variable_value)]
|
||||
|
||||
|
||||
class ConfigDoc(_Doc):
|
||||
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
|
||||
|
||||
opnexus_config: Literal[1]
|
||||
device_groups: list[DeviceGroupDoc] | None = None
|
||||
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
|
||||
aliases: list[AliasDoc] | None = None
|
||||
rules: list[RuleDoc] | None = None
|
||||
port_forwards: list[PortForwardDoc] | None = None
|
||||
one_to_one: list[OneToOneDoc] | None = None
|
||||
outbound_nat: list[OutboundNatDoc] | None = None
|
||||
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
|
||||
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
|
||||
|
||||
|
||||
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
|
||||
KINDS = (
|
||||
"device_groups",
|
||||
"policy_hierarchy_groups",
|
||||
"aliases",
|
||||
"rules",
|
||||
"port_forwards",
|
||||
"one_to_one",
|
||||
"outbound_nat",
|
||||
"policy_hierarchy_rules",
|
||||
"variables",
|
||||
)
|
||||
|
||||
|
||||
SCHEMA_FILE = "opnexus-config-v1.schema.json"
|
||||
|
||||
|
||||
def json_schema_text() -> str:
|
||||
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
|
||||
import json
|
||||
|
||||
schema = ConfigDoc.model_json_schema()
|
||||
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
|
||||
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
|
||||
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
|
||||
@@ -87,6 +87,15 @@ class ErrorCode(str, Enum):
|
||||
MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN = "MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN"
|
||||
NICHT_AUTHENTIFIZIERT = "NICHT_AUTHENTIFIZIERT"
|
||||
CROSS_ORIGIN_BLOCKED = "CROSS_ORIGIN_BLOCKED"
|
||||
CROSS_SITE_REQUEST_BLOCKED = "CROSS_SITE_REQUEST_BLOCKED"
|
||||
GUI_SESSION_REQUIRED = "GUI_SESSION_REQUIRED"
|
||||
GUI_GATEWAY_NOT_CONFIGURED = "GUI_GATEWAY_NOT_CONFIGURED"
|
||||
GUI_REQUIRES_HTTPS = "GUI_REQUIRES_HTTPS"
|
||||
GUI_ACCESS_DISABLED = "GUI_ACCESS_DISABLED"
|
||||
GUI_HOST_NOT_ELIGIBLE = "GUI_HOST_NOT_ELIGIBLE"
|
||||
GUI_NO_FREE_SLOT = "GUI_NO_FREE_SLOT"
|
||||
GUI_SLOT_COOLING_DOWN = "GUI_SLOT_COOLING_DOWN"
|
||||
GUI_TOO_MANY_SESSIONS = "GUI_TOO_MANY_SESSIONS"
|
||||
TLS_PROBE_FAILED = "TLS_PROBE_FAILED"
|
||||
TLS_PEM_INVALID = "TLS_PEM_INVALID"
|
||||
TLS_PIN_MISMATCH = "TLS_PIN_MISMATCH"
|
||||
@@ -119,6 +128,10 @@ class ErrorCode(str, Enum):
|
||||
OBJECT_NOT_ADOPTABLE = "OBJECT_NOT_ADOPTABLE"
|
||||
ADOPT_OBJECT_NOT_FOUND = "ADOPT_OBJECT_NOT_FOUND"
|
||||
OUTBOUND_MODE_BLOCKS_ADOPT = "OUTBOUND_MODE_BLOCKS_ADOPT"
|
||||
ADOPT_LINK_TARGET_NOT_FOUND = "ADOPT_LINK_TARGET_NOT_FOUND"
|
||||
ADOPT_LINK_TARGET_CHANGED = "ADOPT_LINK_TARGET_CHANGED"
|
||||
ADOPT_LINK_HOST_HAS_OBJECT = "ADOPT_LINK_HOST_HAS_OBJECT"
|
||||
ADOPT_LINK_BUSY = "ADOPT_LINK_BUSY"
|
||||
REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0 = "REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0"
|
||||
RESTORE_VORGANG_NICHT_GEFUNDEN = "RESTORE_VORGANG_NICHT_GEFUNDEN"
|
||||
ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL = "ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL"
|
||||
@@ -134,6 +147,10 @@ class ErrorCode(str, Enum):
|
||||
UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN = "UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN"
|
||||
UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT = "UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT"
|
||||
UPSTREAM_REQUEST_FAILED = "UPSTREAM_REQUEST_FAILED"
|
||||
API_RIGHTS_MISSING = "API_RIGHTS_MISSING"
|
||||
GATEWAY_NAME_INVALID = "GATEWAY_NAME_INVALID"
|
||||
GATEWAY_UNKNOWN = "GATEWAY_UNKNOWN"
|
||||
GATEWAY_MUTE_LIMIT = "GATEWAY_MUTE_LIMIT"
|
||||
VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN = "VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN"
|
||||
VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN = "VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN"
|
||||
VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN = "VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN"
|
||||
@@ -154,6 +171,7 @@ class ErrorCode(str, Enum):
|
||||
OPEN_DEPLOYMENT_DRAFT_UNCHANGED = "OPEN_DEPLOYMENT_DRAFT_UNCHANGED"
|
||||
OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY = "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
||||
OPEN_RESTORE_BLOCKS_DEPLOY = "OPEN_RESTORE_BLOCKS_DEPLOY"
|
||||
OPEN_CLEANUP_BLOCKS_DEPLOY = "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
UPSTREAM_REJECTED = "UPSTREAM_REJECTED"
|
||||
INVALID_CHARACTERS = "INVALID_CHARACTERS"
|
||||
SERVICE_NOT_FOUND = "SERVICE_NOT_FOUND"
|
||||
@@ -205,6 +223,37 @@ class ErrorCode(str, Enum):
|
||||
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
|
||||
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
|
||||
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
|
||||
# Rule cleanup (1.31, services/rule_cleanup_service.py)
|
||||
BEREINIGUNG_NICHT_GEFUNDEN = "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||
BEREINIGUNG_AUSWAHL_UNGUELTIG = "BEREINIGUNG_AUSWAHL_UNGUELTIG"
|
||||
BEREINIGUNG_VORSCHAU_VERALTET = "BEREINIGUNG_VORSCHAU_VERALTET"
|
||||
BEREINIGUNG_REGEL_NICHT_AENDERBAR = "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
|
||||
BEREINIGUNG_OFFENE_OPERATION = "BEREINIGUNG_OFFENE_OPERATION"
|
||||
BEREINIGUNG_ZU_VIELE_REGELN = "BEREINIGUNG_ZU_VIELE_REGELN"
|
||||
BEREINIGUNG_BESTAETIGUNG_FEHLT = "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN = "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||
BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG = "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG"
|
||||
BEREINIGUNG_FRIST_ABGELAUFEN = "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
BEREINIGUNG_NICHT_BESTAETIGBAR = "BEREINIGUNG_NICHT_BESTAETIGBAR"
|
||||
BEREINIGUNG_LIVESTAND_ABWEICHEND = "BEREINIGUNG_LIVESTAND_ABWEICHEND"
|
||||
BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR = "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
|
||||
BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT = "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT"
|
||||
BEREINIGUNG_SNAPSHOT_FEHLT = "BEREINIGUNG_SNAPSHOT_FEHLT"
|
||||
BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR = "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
|
||||
BEREINIGUNG_AKTION_NICHT_MOEGLICH = "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
BEREINIGUNG_NICHT_AUSGEFUEHRT = "BEREINIGUNG_NICHT_AUSGEFUEHRT"
|
||||
BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN = "BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN"
|
||||
BEREINIGUNG_AUSGANG_UNKLAR = "BEREINIGUNG_AUSGANG_UNKLAR"
|
||||
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
|
||||
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
|
||||
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
|
||||
CONFIG_INVALID = "CONFIG_INVALID"
|
||||
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
|
||||
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
|
||||
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
|
||||
CONFIG_BLOCKED = "CONFIG_BLOCKED"
|
||||
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
|
||||
CONFIG_BUSY = "CONFIG_BUSY"
|
||||
|
||||
|
||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||
|
||||
@@ -0,0 +1,609 @@
|
||||
"""The ``gui-gateway`` container: firewall web interfaces in their own browser origin (1.27).
|
||||
|
||||
Runs from the backend image (``python -m app.gui_gateway``) behind the proxy, which terminates TLS for the
|
||||
slot ports and sets ``X-Onx-Slot`` (overwriting any client value). It never touches the database or the
|
||||
secret stores; it only talks to the backend's internal API (``/internal/gui/*``, shared GUI_GATEWAY_TOKEN)
|
||||
and, per gateway session, to exactly one firewall.
|
||||
|
||||
Security model (see the 1.27 design):
|
||||
* Every slot port is its own origin. The firewall GUI therefore never runs in the OPNexus origin.
|
||||
* Cookies never cross: the browser's ``Cookie`` header is not forwarded (it carries the OPNexus session),
|
||||
and the firewall's ``Set-Cookie`` never reaches the browser — a server-side jar per gateway session
|
||||
(memory only) holds them. The browser only gets ``__Host-onx-gw-<slot>``.
|
||||
* Each upstream request is built from scratch: method, path and query from the browser, a small header
|
||||
allow-list, the jar's cookies; target origin, pinned certificate and overlay proxy from the backend.
|
||||
Absolute-form targets, CONNECT/TRACE, ``Upgrade`` and service-worker script requests are refused.
|
||||
* ``Sec-Fetch-Site`` must be same-origin/none (same-site only for the redeem step), fail-closed;
|
||||
non-GET/HEAD requests also need the slot's own ``Origin``.
|
||||
* ``Location`` pointing anywhere but the firewall is dropped; responses get CSP, X-Frame-Options DENY,
|
||||
``Cache-Control: no-store``.
|
||||
* At most 3 failed GUI logins per firewall in 5 minutes (protects OPNexus' API access from the firewall's
|
||||
login lockout, which would see all attempts coming from OPNexus).
|
||||
* Idle and absolute limits; every 20 s the backend re-checks the session (OPNexus session, admin role,
|
||||
switch, host, and that the firewall's address, pinned certificate and overlay route are still the ones the
|
||||
session started with).
|
||||
* At most MAX_SESSIONS sessions at once: expired ones are cleared first, a new session is refused otherwise —
|
||||
a running session is never evicted by a newer one. Logs carry method, path (no query), status and duration only.
|
||||
|
||||
OPNsense behaviour this relies on (opnsense/core, src/www/authgui.inc, master as of 2026-10-21):
|
||||
* session_auth(): a login is ``isset($_POST['login']) && !empty($_POST['usernamefld']) &&
|
||||
!empty($_POST['passwordfld'])`` posted to the current URL (login form has no action). Success sends a
|
||||
``Location:`` header and exits (302); a failure re-renders the login form (200) with "Wrong username or
|
||||
password." and an auth_log entry.
|
||||
* check_security_http_referer_enforcement() only runs when HTTP_REFERER is set, and wants the firewall's
|
||||
own host name/IP — so the gateway sends no Referer (and no Origin) upstream at all.
|
||||
* check_security_dns_rebind() accepts an IP address or the configured host name as Host — the gateway
|
||||
sends the Host of the host record's base URL (httpx derives it from the URL).
|
||||
"""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
import ssl
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from http.cookies import CookieError, SimpleCookie
|
||||
from typing import Awaitable, Callable
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("gui_gateway")
|
||||
# httpx/httpcore log full request URLs (query strings) at INFO/DEBUG: never in this process.
|
||||
for _noisy in ("httpx", "httpcore"):
|
||||
logging.getLogger(_noisy).setLevel(logging.WARNING)
|
||||
|
||||
LISTEN_PORT = 8080
|
||||
COOKIE_PREFIX = "__Host-onx-gw-"
|
||||
OPNEXUS_SESSION_COOKIE = "__Host-opnexus-session"
|
||||
BODY_LIMIT = 64 * 1024 * 1024 # config restore uploads
|
||||
LOGIN_FAILURE_LIMIT = 3
|
||||
LOGIN_FAILURE_WINDOW = 300.0
|
||||
UPSTREAM_TIMEOUT = 60.0
|
||||
MAX_SESSIONS = 64
|
||||
|
||||
# Request headers copied from the browser; everything else (Cookie, Authorization, Origin, Referer,
|
||||
# X-Forwarded-*, hop-by-hop, Sec-*, ...) is dropped.
|
||||
_FORWARD_REQUEST_HEADERS = {
|
||||
"accept", "accept-language", "content-type", "user-agent", "x-requested-with", "x-csrftoken", "range",
|
||||
}
|
||||
# Response headers copied from the firewall. Set-Cookie goes to the jar, Location is rewritten.
|
||||
_FORWARD_RESPONSE_HEADERS = {
|
||||
"content-type", "content-length", "content-encoding", "content-disposition", "content-language",
|
||||
"content-range", "accept-ranges", "content-security-policy",
|
||||
}
|
||||
_GATEWAY_CSP = ("connect-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'self'; "
|
||||
"object-src 'none'; worker-src 'none'")
|
||||
_SECURITY_HEADERS = [
|
||||
(b"content-security-policy", _GATEWAY_CSP.encode()),
|
||||
(b"x-frame-options", b"DENY"),
|
||||
(b"x-content-type-options", b"nosniff"),
|
||||
(b"cache-control", b"no-store"),
|
||||
(b"pragma", b"no-cache"),
|
||||
(b"referrer-policy", b"same-origin"),
|
||||
(b"cross-origin-opener-policy", b"same-origin"),
|
||||
(b"cross-origin-resource-policy", b"same-origin"),
|
||||
(b"permissions-policy", b"camera=(), microphone=(), geolocation=(), payment=(), usb=()"),
|
||||
]
|
||||
|
||||
|
||||
def _sha(value: str) -> str:
|
||||
return hashlib.sha256(value.encode()).hexdigest()
|
||||
|
||||
|
||||
def _origin_of(url: str) -> str | None:
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
return None
|
||||
if parts.scheme not in ("http", "https") or not parts.hostname:
|
||||
return None
|
||||
default = 443 if parts.scheme == "https" else 80
|
||||
host = parts.hostname.lower()
|
||||
if ":" in host:
|
||||
host = f"[{host}]"
|
||||
return f"{parts.scheme}://{host}" + (f":{port}" if port and port != default else "")
|
||||
|
||||
|
||||
def slot_origins_from(sites: str) -> dict[int, set[str]]:
|
||||
"""OPNEXUS_GUI_SITES -> {port: {normalised origins}} (https only, explicit port)."""
|
||||
slots: dict[int, set[str]] = {}
|
||||
for raw in sites.split(","):
|
||||
raw = raw.strip().rstrip("/")
|
||||
try:
|
||||
parts = urlsplit(raw)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
continue
|
||||
if parts.scheme == "https" and parts.hostname and port:
|
||||
slots.setdefault(port, set()).add(_origin_of(raw))
|
||||
return slots
|
||||
|
||||
|
||||
@dataclass
|
||||
class GatewaySession:
|
||||
gui_session_id: str
|
||||
session_hash: str
|
||||
slot: int
|
||||
upstream_origin: str
|
||||
ca_pem: str | None
|
||||
proxy_url: str | None
|
||||
idle_timeout: float
|
||||
absolute_deadline: float
|
||||
revalidate_every: float
|
||||
target_fp: str | None = None
|
||||
last_activity: float = field(default_factory=time.monotonic)
|
||||
next_check: float = 0.0
|
||||
jar: dict[str, str] = field(default_factory=dict)
|
||||
client: httpx.AsyncClient | None = None
|
||||
|
||||
|
||||
class BackendClient:
|
||||
"""The backend's internal GUI API. Raises httpx.HTTPError when the backend is unreachable."""
|
||||
|
||||
def __init__(self, base_url: str, token: str, transport: httpx.AsyncBaseTransport | None = None):
|
||||
self._client = httpx.AsyncClient(base_url=base_url, headers={"X-Onx-Gateway-Token": token},
|
||||
timeout=10.0, transport=transport)
|
||||
|
||||
async def redeem(self, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
|
||||
resp = await self._client.post("/internal/gui/redeem", json={
|
||||
"ticket_hash": ticket_hash, "session_hash": session_hash, "slot": slot, "client_ip": client_ip})
|
||||
if resp.status_code == 200:
|
||||
return resp.json()
|
||||
if resp.status_code == 403:
|
||||
return {"ok": False, "reason": (resp.json().get("detail") or {}).get("reason", "denied")}
|
||||
resp.raise_for_status()
|
||||
return {"ok": False, "reason": f"http_{resp.status_code}"}
|
||||
|
||||
async def revalidate(self, gs: GatewaySession) -> dict:
|
||||
resp = await self._client.post("/internal/gui/revalidate",
|
||||
json={"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash,
|
||||
"target_fp": gs.target_fp})
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def end(self, gs: GatewaySession, reason: str) -> None:
|
||||
try:
|
||||
await self._client.post("/internal/gui/end", json={
|
||||
"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash, "reason": reason})
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Could not report the end of a GUI session (%s)", type(exc).__name__)
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._client.aclose()
|
||||
|
||||
|
||||
def build_upstream_client(gs: GatewaySession) -> httpx.AsyncClient:
|
||||
"""Same transport rules as opnsense_client._client: pinned certificate (or system CAs), overlay proxy."""
|
||||
verify: ssl.SSLContext | bool
|
||||
if gs.ca_pem:
|
||||
from app.services.tls_pin_service import pinned_context # no database layer behind this import
|
||||
verify = pinned_context(gs.ca_pem)
|
||||
else:
|
||||
verify = True
|
||||
return httpx.AsyncClient(verify=verify, proxy=gs.proxy_url, timeout=UPSTREAM_TIMEOUT,
|
||||
follow_redirects=False, trust_env=False)
|
||||
|
||||
|
||||
class LoginLimiter:
|
||||
"""At most LOGIN_FAILURE_LIMIT failed GUI logins per firewall within LOGIN_FAILURE_WINDOW seconds."""
|
||||
|
||||
def __init__(self, clock: Callable[[], float] = time.monotonic):
|
||||
self._failures: dict[str, deque] = {}
|
||||
self._clock = clock
|
||||
|
||||
def _recent(self, key: str) -> deque:
|
||||
now = self._clock()
|
||||
q = self._failures.setdefault(key, deque())
|
||||
while q and now - q[0] > LOGIN_FAILURE_WINDOW:
|
||||
q.popleft()
|
||||
return q
|
||||
|
||||
def blocked(self, key: str) -> bool:
|
||||
return len(self._recent(key)) >= LOGIN_FAILURE_LIMIT
|
||||
|
||||
def record_failure(self, key: str) -> None:
|
||||
self._recent(key).append(self._clock())
|
||||
|
||||
|
||||
def is_login_attempt(method: str, content_type: str, body: bytes) -> bool:
|
||||
if method != "POST":
|
||||
return False
|
||||
if content_type.lower().startswith("multipart/form-data"):
|
||||
# PHP fills $_POST from multipart bodies too, so authgui.inc accepts a login posted this way.
|
||||
return all(f'name="{field}"'.encode() in body for field in ("login", "usernamefld", "passwordfld"))
|
||||
if not content_type.lower().startswith("application/x-www-form-urlencoded"):
|
||||
return False
|
||||
try:
|
||||
form = parse_qs(body.decode("latin-1"), keep_blank_values=True)
|
||||
except ValueError:
|
||||
return False
|
||||
return "login" in form and bool(form.get("usernamefld", [""])[0]) and bool(form.get("passwordfld", [""])[0])
|
||||
|
||||
|
||||
def rewrite_location(location: str, upstream_origin: str) -> str | None:
|
||||
"""Relative target for the browser, or None when the redirect leaves the firewall."""
|
||||
location = location.strip()
|
||||
if not location or "\\" in location or any(ord(c) < 0x20 for c in location):
|
||||
return None
|
||||
if location.startswith("//"):
|
||||
location = "https:" + location
|
||||
parts = urlsplit(location)
|
||||
if parts.scheme or parts.netloc:
|
||||
if _origin_of(location) != _origin_of(upstream_origin):
|
||||
return None
|
||||
path = parts.path or "/"
|
||||
return path + (f"?{parts.query}" if parts.query else "") + (f"#{parts.fragment}" if parts.fragment else "")
|
||||
return location
|
||||
|
||||
|
||||
def _page(title: str, text_de: str, text_en: str) -> bytes:
|
||||
return (f"<!doctype html><html lang=\"de\"><head><meta charset=\"utf-8\"><title>{title}</title></head>"
|
||||
f"<body style=\"font-family:sans-serif;max-width:40rem;margin:3rem auto\"><h1>{title}</h1>"
|
||||
f"<p>{text_de}</p><p lang=\"en\">{text_en}</p></body></html>").encode()
|
||||
|
||||
|
||||
_SESSION_GONE = _page("OPNexus", "Diese Sitzung ist abgelaufen oder ungültig. Öffne die Oberfläche erneut aus OPNexus.",
|
||||
"This session has expired or is invalid. Open the interface again from OPNexus.")
|
||||
_FORBIDDEN = _page("OPNexus", "Diese Anfrage wurde vom GUI-Gateway abgelehnt.", "This request was refused by the GUI gateway.")
|
||||
_BAD_REDIRECT = _page("OPNexus", "Die Firewall wollte auf eine fremde Adresse umleiten; die Umleitung wurde verworfen.",
|
||||
"The firewall tried to redirect to a foreign address; the redirect was dropped.")
|
||||
_LOGIN_LOCKED = _page("OPNexus", "Zu viele fehlgeschlagene Anmeldungen an dieser Firewall. Bitte warte 5 Minuten.",
|
||||
"Too many failed sign-ins at this firewall. Please wait 5 minutes.")
|
||||
_UNAVAILABLE = _page("OPNexus", "Die Firewall oder OPNexus ist gerade nicht erreichbar.",
|
||||
"The firewall or OPNexus is not reachable right now.")
|
||||
_TOO_MANY = _page("OPNexus", "Es laufen schon zu viele Firewall-Oberflächen gleichzeitig. Versuche es später erneut.",
|
||||
"Too many firewall interfaces are open at the same time. Try again later.")
|
||||
_REDEEMED = (b"<!doctype html><html><head><meta charset=\"utf-8\"><meta http-equiv=\"refresh\" content=\"0;url=/\">"
|
||||
b"<title>OPNexus</title></head><body><a href=\"/\">→</a></body></html>")
|
||||
|
||||
|
||||
class GuiGateway:
|
||||
def __init__(self, *, backend: BackendClient, slot_origins: dict[int, set[str]],
|
||||
upstream_factory: Callable[[GatewaySession], httpx.AsyncClient] = build_upstream_client,
|
||||
clock: Callable[[], float] = time.monotonic, wall_clock: Callable[[], float] = time.time):
|
||||
self.backend = backend
|
||||
self.slot_origins = slot_origins
|
||||
self.upstream_factory = upstream_factory
|
||||
self.clock = clock
|
||||
self.wall_clock = wall_clock
|
||||
self.sessions: dict[str, GatewaySession] = {} # sha256(gateway cookie) -> session
|
||||
self.limiter = LoginLimiter(clock)
|
||||
self._loop_task: asyncio.Task | None = None
|
||||
|
||||
# ── ASGI ──────────────────────────────────────────────────────────────
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] == "lifespan":
|
||||
await self._lifespan(receive, send)
|
||||
return
|
||||
if scope["type"] != "http":
|
||||
# WebSocket and anything else: not supported (design: no WebSocket).
|
||||
if scope["type"] == "websocket":
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
return
|
||||
started = time.monotonic()
|
||||
status = await self._handle(scope, receive, send)
|
||||
# Method, path and status only: never the query string, headers or bodies.
|
||||
logger.info("%s %s %d %.0fms", scope.get("method", "?"), scope.get("path", "?")[:200], status,
|
||||
(time.monotonic() - started) * 1000)
|
||||
|
||||
async def _lifespan(self, receive, send):
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] == "lifespan.startup":
|
||||
self._loop_task = asyncio.create_task(self._revalidate_loop())
|
||||
await send({"type": "lifespan.startup.complete"})
|
||||
elif message["type"] == "lifespan.shutdown":
|
||||
if self._loop_task:
|
||||
self._loop_task.cancel()
|
||||
for key in list(self.sessions):
|
||||
await self._drop(key, "shutdown")
|
||||
await self.backend.aclose()
|
||||
await send({"type": "lifespan.shutdown.complete"})
|
||||
return
|
||||
|
||||
async def _revalidate_loop(self):
|
||||
while True:
|
||||
await asyncio.sleep(10)
|
||||
try:
|
||||
await self.sweep()
|
||||
except Exception: # never let the loop die
|
||||
logger.exception("GUI session sweep failed")
|
||||
|
||||
async def sweep(self) -> None:
|
||||
"""Idle/absolute limits and the periodic backend re-check for every gateway session."""
|
||||
for key, gs in list(self.sessions.items()):
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
elif self.clock() >= gs.next_check:
|
||||
await self._recheck(key, gs)
|
||||
|
||||
# ── helpers ───────────────────────────────────────────────────────────
|
||||
@staticmethod
|
||||
def _headers(scope) -> dict[str, str]:
|
||||
result: dict[str, str] = {}
|
||||
for k, v in scope.get("headers", []):
|
||||
name = k.decode("latin-1").lower()
|
||||
value = v.decode("latin-1")
|
||||
if name in result:
|
||||
value = f"{result[name]}{'; ' if name == 'cookie' else ', '}{value}"
|
||||
result[name] = value
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _cookies(header: str) -> dict[str, str]:
|
||||
cookies: dict[str, str] = {}
|
||||
for part in header.split(";"):
|
||||
name, sep, value = part.strip().partition("=")
|
||||
if sep and name and name not in cookies:
|
||||
cookies[name] = value
|
||||
return cookies
|
||||
|
||||
async def _respond(self, send, status: int, body: bytes = b"", headers: list | None = None,
|
||||
content_type: bytes = b"text/html; charset=utf-8") -> int:
|
||||
out = [(b"content-type", content_type), (b"content-length", str(len(body)).encode())]
|
||||
out += list(headers or []) + _SECURITY_HEADERS
|
||||
await send({"type": "http.response.start", "status": status, "headers": out})
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
return status
|
||||
|
||||
def _expired(self, gs: GatewaySession) -> str | None:
|
||||
if self.wall_clock() >= gs.absolute_deadline:
|
||||
return "absolute"
|
||||
if self.clock() - gs.last_activity >= gs.idle_timeout:
|
||||
return "idle"
|
||||
return None
|
||||
|
||||
async def _drop(self, key: str, reason: str, *, report: bool = True) -> None:
|
||||
gs = self.sessions.pop(key, None)
|
||||
if gs is None:
|
||||
return
|
||||
gs.jar.clear()
|
||||
if gs.client is not None:
|
||||
await gs.client.aclose()
|
||||
if report:
|
||||
await self.backend.end(gs, reason)
|
||||
|
||||
async def _recheck(self, key: str, gs: GatewaySession) -> bool | None:
|
||||
"""True = still valid, False = ended (dropped), None = backend unreachable (fail closed, keep for retry)."""
|
||||
try:
|
||||
result = await self.backend.revalidate(gs)
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("GUI session re-check failed (%s)", type(exc).__name__)
|
||||
return None
|
||||
if not result.get("ok"):
|
||||
await self._drop(key, str(result.get("reason", "revoked")), report=False) # backend already ended it
|
||||
return False
|
||||
gs.absolute_deadline = min(gs.absolute_deadline, float(result.get("absolute_deadline", gs.absolute_deadline)))
|
||||
gs.idle_timeout = float(result.get("idle_timeout_seconds", gs.idle_timeout))
|
||||
gs.next_check = self.clock() + gs.revalidate_every
|
||||
return True
|
||||
|
||||
async def _clear_expired(self) -> None:
|
||||
for key, gs in list(self.sessions.items()):
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
|
||||
def _has_room(self, browser_hash: str, slot: int) -> bool:
|
||||
"""Room for one more session; the one this redeem would replace (same browser session and slot) does not count."""
|
||||
replaced = sum(1 for g in self.sessions.values() if g.slot == slot and g.session_hash == browser_hash)
|
||||
return len(self.sessions) - replaced < MAX_SESSIONS
|
||||
|
||||
# ── request handling ──────────────────────────────────────────────────
|
||||
async def _handle(self, scope, receive, send) -> int:
|
||||
method = scope.get("method", "")
|
||||
path = scope.get("path", "")
|
||||
raw_path = scope.get("raw_path") or path.encode()
|
||||
headers = self._headers(scope)
|
||||
|
||||
if method in ("CONNECT", "TRACE"):
|
||||
return await self._respond(send, 501, _FORBIDDEN)
|
||||
# Origin-form only: "GET http://elsewhere/ HTTP/1.1" (absolute-form) or "//host" never reach upstream.
|
||||
if not path.startswith("/") or path.startswith("//") or not raw_path.startswith(b"/") \
|
||||
or raw_path.startswith(b"//") or "\\" in path:
|
||||
return await self._respond(send, 400, _FORBIDDEN)
|
||||
if "upgrade" in headers or "upgrade" in headers.get("connection", "").lower():
|
||||
return await self._respond(send, 400, _FORBIDDEN)
|
||||
if headers.get("service-worker", "").lower() == "script":
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
|
||||
try:
|
||||
slot = int(headers.get("x-onx-slot", ""))
|
||||
except ValueError:
|
||||
slot = 0
|
||||
own_origins = self.slot_origins.get(slot)
|
||||
if not own_origins:
|
||||
return await self._respond(send, 400, _FORBIDDEN) # not coming through the proxy's slot block
|
||||
|
||||
site = headers.get("sec-fetch-site", "").strip().lower()
|
||||
if path == "/__onx/redeem":
|
||||
if method != "GET" or site not in ("same-site", "same-origin", "none"):
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
return await self._redeem(scope, headers, slot, send)
|
||||
if site not in ("same-origin", "none"):
|
||||
return await self._respond(send, 403, _FORBIDDEN) # missing header = old/odd client: fail closed
|
||||
if method not in ("GET", "HEAD"):
|
||||
origin = headers.get("origin", "")
|
||||
if _origin_of(origin) not in own_origins:
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
if path.startswith("/__onx/"):
|
||||
return await self._respond(send, 404, _FORBIDDEN)
|
||||
|
||||
cookies = self._cookies(headers.get("cookie", ""))
|
||||
token = cookies.get(f"{COOKIE_PREFIX}{slot}", "")
|
||||
key = _sha(token) if token else ""
|
||||
gs = self.sessions.get(key) if key else None
|
||||
if gs is None or gs.slot != slot:
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
if self.clock() >= gs.next_check:
|
||||
state = await self._recheck(key, gs)
|
||||
if state is False:
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
if state is None:
|
||||
return await self._respond(send, 503, _UNAVAILABLE)
|
||||
gs.last_activity = self.clock()
|
||||
return await self._proxy(scope, receive, send, headers, gs)
|
||||
|
||||
async def _redeem(self, scope, headers, slot: int, send) -> int:
|
||||
query = parse_qs(scope.get("query_string", b"").decode("latin-1"))
|
||||
ticket = (query.get("t") or [""])[0]
|
||||
browser_session = self._cookies(headers.get("cookie", "")).get(OPNEXUS_SESSION_COOKIE, "")
|
||||
if not ticket or len(ticket) > 128 or not browser_session:
|
||||
return await self._respond(send, 403, _SESSION_GONE)
|
||||
client_ip = (headers.get("x-forwarded-for", "").split(",")[0].strip() or None)
|
||||
# Room first, before the ticket is burned: expired sessions go, running ones are never evicted.
|
||||
await self._clear_expired()
|
||||
if not self._has_room(_sha(browser_session), slot):
|
||||
return await self._respond(send, 503, _TOO_MANY)
|
||||
try:
|
||||
result = await self.backend.redeem(_sha(ticket), _sha(browser_session), slot, client_ip and client_ip[:45])
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Ticket redeem failed (%s)", type(exc).__name__)
|
||||
return await self._respond(send, 503, _UNAVAILABLE)
|
||||
if not result.get("ok"):
|
||||
return await self._respond(send, 403, _SESSION_GONE)
|
||||
# One browser session, one gateway session per slot: an older one in this slot is replaced.
|
||||
for old_key, old in list(self.sessions.items()):
|
||||
if old.slot == slot and old.session_hash == _sha(browser_session):
|
||||
await self._drop(old_key, "superseded", report=False)
|
||||
value = secrets.token_urlsafe(32)
|
||||
gs = GatewaySession(
|
||||
gui_session_id=result["gui_session_id"], session_hash=_sha(browser_session), slot=slot,
|
||||
upstream_origin=result["upstream_origin"], ca_pem=result.get("ca_pem"), proxy_url=result.get("proxy_url"),
|
||||
idle_timeout=float(result["idle_timeout_seconds"]), absolute_deadline=float(result["absolute_deadline"]),
|
||||
revalidate_every=float(result.get("revalidate_seconds", 20)), target_fp=result.get("target_fp"),
|
||||
)
|
||||
if len(self.sessions) >= MAX_SESSIONS: # filled up while the backend was answering
|
||||
await self.backend.end(gs, "revoked")
|
||||
return await self._respond(send, 503, _TOO_MANY)
|
||||
gs.last_activity = self.clock()
|
||||
gs.next_check = gs.last_activity + gs.revalidate_every
|
||||
self.sessions[_sha(value)] = gs
|
||||
cookie = f"{COOKIE_PREFIX}{slot}={value}; Path=/; Secure; HttpOnly; SameSite=Strict"
|
||||
# 200 + meta refresh instead of a 303: the follow-up request then starts in this origin
|
||||
# (Sec-Fetch-Site: same-origin) and the ticket leaves the address bar.
|
||||
return await self._respond(send, 200, _REDEEMED, headers=[
|
||||
(b"set-cookie", cookie.encode()), (b"clear-site-data", b'"cache", "storage"'),
|
||||
])
|
||||
|
||||
async def _read_body(self, receive) -> bytes | None:
|
||||
chunks, size = [], 0
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] == "http.disconnect":
|
||||
return None
|
||||
chunk = message.get("body", b"")
|
||||
size += len(chunk)
|
||||
if size > BODY_LIMIT:
|
||||
return None
|
||||
chunks.append(chunk)
|
||||
if not message.get("more_body"):
|
||||
return b"".join(chunks)
|
||||
|
||||
def _store_cookies(self, gs: GatewaySession, response: httpx.Response) -> None:
|
||||
for header in response.headers.get_list("set-cookie"):
|
||||
jar = SimpleCookie()
|
||||
try:
|
||||
jar.load(header)
|
||||
except CookieError:
|
||||
continue
|
||||
for name, morsel in jar.items():
|
||||
expired = morsel["max-age"] in ("0",) or morsel.value in ("", "deleted")
|
||||
if not expired and morsel["expires"]:
|
||||
try:
|
||||
from email.utils import parsedate_to_datetime
|
||||
expired = parsedate_to_datetime(morsel["expires"]).timestamp() <= self.wall_clock()
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
if expired:
|
||||
gs.jar.pop(name, None)
|
||||
else:
|
||||
gs.jar[name] = morsel.value
|
||||
|
||||
async def _proxy(self, scope, receive, send, headers: dict, gs: GatewaySession) -> int:
|
||||
length = headers.get("content-length", "")
|
||||
if length.isdigit() and int(length) > BODY_LIMIT:
|
||||
return await self._respond(send, 413, _FORBIDDEN)
|
||||
body = await self._read_body(receive)
|
||||
if body is None:
|
||||
return await self._respond(send, 413, _FORBIDDEN)
|
||||
method = scope["method"]
|
||||
login = is_login_attempt(method, headers.get("content-type", ""), body)
|
||||
if login and self.limiter.blocked(gs.upstream_origin):
|
||||
return await self._respond(send, 429, _LOGIN_LOCKED)
|
||||
|
||||
out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS}
|
||||
out_headers["accept-encoding"] = "identity"
|
||||
if gs.jar:
|
||||
out_headers["cookie"] = "; ".join(f"{k}={v}" for k, v in gs.jar.items())
|
||||
query = scope.get("query_string", b"").decode("latin-1")
|
||||
# The path exactly as the browser sent it (still percent-encoded): the decoded scope["path"] would
|
||||
# turn %3F/%23 into query/fragment, %2F into a separator and decode %25xx a second time upstream.
|
||||
raw_path = (scope.get("raw_path") or scope["path"].encode()).split(b"?", 1)[0].decode("latin-1")
|
||||
url = gs.upstream_origin + raw_path + (f"?{query}" if query else "")
|
||||
if gs.client is None:
|
||||
gs.client = self.upstream_factory(gs)
|
||||
try:
|
||||
request = gs.client.build_request(method, url, headers=out_headers, content=body)
|
||||
except httpx.InvalidURL:
|
||||
return await self._respond(send, 400, _FORBIDDEN) # e.g. control characters in the path
|
||||
try:
|
||||
response = await gs.client.send(request, stream=True)
|
||||
except (httpx.HTTPError, ssl.SSLError, OSError) as exc:
|
||||
logger.warning("Upstream request failed (%s)", type(exc).__name__)
|
||||
return await self._respond(send, 502, _UNAVAILABLE)
|
||||
try:
|
||||
self._store_cookies(gs, response)
|
||||
if login and not (300 <= response.status_code < 400):
|
||||
self.limiter.record_failure(gs.upstream_origin)
|
||||
out = []
|
||||
for name, value in response.headers.multi_items():
|
||||
lname = name.lower()
|
||||
if lname in _FORWARD_RESPONSE_HEADERS:
|
||||
out.append((lname.encode("latin-1"), value.encode("latin-1")))
|
||||
if "location" in response.headers:
|
||||
target = rewrite_location(response.headers["location"], gs.upstream_origin)
|
||||
if target is None:
|
||||
await response.aclose()
|
||||
return await self._respond(send, 502, _BAD_REDIRECT)
|
||||
out.append((b"location", target.encode("latin-1")))
|
||||
await send({"type": "http.response.start", "status": response.status_code, "headers": out + _SECURITY_HEADERS})
|
||||
async for chunk in response.aiter_raw():
|
||||
await send({"type": "http.response.body", "body": chunk, "more_body": True})
|
||||
await send({"type": "http.response.body", "body": b""})
|
||||
return response.status_code
|
||||
finally:
|
||||
await response.aclose()
|
||||
|
||||
|
||||
def create_app(transport: httpx.AsyncBaseTransport | None = None) -> GuiGateway:
|
||||
token = os.environ.get("GUI_GATEWAY_TOKEN", "")
|
||||
if len(token) < 32:
|
||||
raise SystemExit("GUI_GATEWAY_TOKEN (at least 32 characters) is required")
|
||||
slots = slot_origins_from(os.environ.get("OPNEXUS_GUI_SITES", ""))
|
||||
if not slots:
|
||||
raise SystemExit("OPNEXUS_GUI_SITES must list the https slot origins")
|
||||
backend = BackendClient(os.environ.get("GUI_BACKEND_URL", "http://backend:8000"), token, transport)
|
||||
return GuiGateway(backend=backend, slot_origins=slots)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import uvicorn
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
|
||||
# h11: an absolute-form target stays visible in scope["path"] (and is refused); access_log off because
|
||||
# uvicorn's access log would contain query strings (the redeem ticket).
|
||||
uvicorn.run(create_app(), host="0.0.0.0", port=LISTEN_PORT, http="h11", ws="none", access_log=False,
|
||||
server_header=False, date_header=False, proxy_headers=False, lifespan="on", log_config=None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+45
-1
@@ -53,6 +53,9 @@ from app.routers import scheduled_maintenance as scheduled_maintenance_router
|
||||
from app.routers import maintenance_windows as maintenance_windows_router
|
||||
from app.routers import rollouts as rollouts_router
|
||||
from app.routers import rule_analysis as rule_analysis_router
|
||||
from app.routers import rule_cleanup as rule_cleanup_router
|
||||
from app.routers import config_code as config_code_router
|
||||
from app.routers import gui_access as gui_access_router
|
||||
from app.services import scheduled_maintenance_service
|
||||
from app.services import rollout_engine
|
||||
from app.services import compliance_service
|
||||
@@ -60,6 +63,8 @@ from app.services import audit_export_service
|
||||
from app.services import overlay_service, variables_service
|
||||
from app.services import backup_schedule_service
|
||||
from app.services import carp_event_service, carp_maintenance_window_service
|
||||
from app.services import rule_history_service
|
||||
from app.services import rule_cleanup_service
|
||||
|
||||
# Without this, Python's root logger defaults to WARNING with no handler,
|
||||
# so every logger.info() call in the 5 background loops below (CARP role
|
||||
@@ -140,6 +145,13 @@ async def _deploy_revert_sweep_loop():
|
||||
logger.info("Auto-reverted %d unconfirmed policy-hierarchy deployment(s)", len(reverted))
|
||||
except Exception:
|
||||
logger.exception("Policy-hierarchy deploy-revert sweep failed")
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
reverted = await rule_cleanup_service.sweep_expired_cleanups(db)
|
||||
if reverted:
|
||||
logger.info("Regel-Bereinigung: %d Operation(en) automatisch zurückgenommen/abgeschlossen", len(reverted))
|
||||
except Exception:
|
||||
logger.exception("Rule-cleanup revert sweep failed")
|
||||
await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
|
||||
|
||||
|
||||
@@ -233,6 +245,9 @@ async def _resource_poll_loop():
|
||||
deleted = await resource_monitor_service.prune_old_samples(db, settings.resource_sample_retention_days)
|
||||
if deleted:
|
||||
logger.info("Resource-Sample-Retention: %d alte Einträge gelöscht", deleted)
|
||||
pruned = await rule_history_service.prune(db)
|
||||
if pruned:
|
||||
logger.info("Regelverlauf-Retention: %d alte Einträge gelöscht", pruned)
|
||||
except Exception:
|
||||
logger.exception("Resource-Poll fehlgeschlagen")
|
||||
await asyncio.sleep(settings.resource_poll_interval_seconds)
|
||||
@@ -321,8 +336,31 @@ async def _backup_poll_loop():
|
||||
await asyncio.sleep(_BACKUP_POLL_SECONDS)
|
||||
|
||||
|
||||
_RULE_OBSERVE_FIRST_RUN_SECONDS = 150
|
||||
|
||||
|
||||
async def _rule_observe_loop():
|
||||
"""Ninth background loop: reads the rules and their counters of every enabled firewall (two GETs each),
|
||||
keeps the usage sampling current and records what changed since the last look (services/rule_history_service.py).
|
||||
Writes only to the database. The first look at a firewall only stores the starting point."""
|
||||
await asyncio.sleep(_RULE_OBSERVE_FIRST_RUN_SECONDS)
|
||||
while True:
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
result = await rule_history_service.observe_all(db)
|
||||
if result["events"] or result["failed"]:
|
||||
logger.info("Regelverlauf: %s", result)
|
||||
except Exception:
|
||||
logger.exception("Regelverlauf fehlgeschlagen")
|
||||
await asyncio.sleep(max(settings.rule_observe_interval_seconds, settings.rule_observe_min_interval_seconds))
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
conflicts = settings.gui_origin_conflicts()
|
||||
if conflicts:
|
||||
# A GUI-gateway slot must never be a trusted OPNexus origin (CORS with credentials).
|
||||
raise RuntimeError(f"OPNEXUS_GUI_SITES overlaps FRONTEND_ORIGIN: {', '.join(conflicts)} — refusing to start")
|
||||
await migrate.run()
|
||||
async with SessionLocal() as db:
|
||||
await auth_service.bootstrap_admin(db)
|
||||
@@ -346,6 +384,7 @@ async def lifespan(app: FastAPI):
|
||||
compliance_task = asyncio.create_task(_compliance_poll_loop())
|
||||
audit_export_task = asyncio.create_task(_audit_export_poll_loop())
|
||||
backup_task = asyncio.create_task(_backup_poll_loop())
|
||||
rule_observe_task = asyncio.create_task(_rule_observe_loop())
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
@@ -357,7 +396,8 @@ async def lifespan(app: FastAPI):
|
||||
compliance_task.cancel()
|
||||
audit_export_task.cancel()
|
||||
backup_task.cancel()
|
||||
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task):
|
||||
rule_observe_task.cancel()
|
||||
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task, rule_observe_task):
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
@@ -531,3 +571,7 @@ app.include_router(scheduled_maintenance_router.router, prefix="/api")
|
||||
app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||
app.include_router(rollouts_router.router, prefix="/api")
|
||||
app.include_router(rule_analysis_router.router, prefix="/api")
|
||||
app.include_router(rule_cleanup_router.router, prefix="/api")
|
||||
app.include_router(config_code_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.internal_router)
|
||||
@@ -99,3 +99,17 @@ __all__ += ["MaintenanceWindow", "MaintenanceWindowException"]
|
||||
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun, RolloutRunHost, RolloutRunWave, rollout_wave_groups
|
||||
__all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "RolloutRunWave", "rollout_wave_groups"]
|
||||
|
||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||
__all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
|
||||
|
||||
from app.models.rule_history import LiveRuleState, RuleChangeEvent
|
||||
__all__ += ["LiveRuleState", "RuleChangeEvent"]
|
||||
|
||||
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
|
||||
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
|
||||
|
||||
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
|
||||
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
|
||||
from app.models.host_variable import HostVariable
|
||||
__all__ += ["HostVariable"]
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
|
||||
|
||||
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
|
||||
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
|
||||
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
|
||||
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import String, event
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
|
||||
|
||||
|
||||
def config_key_column() -> Mapped[str]:
|
||||
return mapped_column(String(64), unique=True)
|
||||
|
||||
|
||||
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
|
||||
return f"{prefix}-{object_id.hex[:12]}"
|
||||
|
||||
|
||||
def register_config_key(model: type, prefix: str) -> None:
|
||||
"""Fills an empty `config_key` just before INSERT (and the id, which is
|
||||
otherwise only assigned while the INSERT is built)."""
|
||||
|
||||
@event.listens_for(model, "before_insert")
|
||||
def _fill(_mapper, _connection, target) -> None:
|
||||
if target.id is None:
|
||||
target.id = uuid.uuid4()
|
||||
if not target.config_key:
|
||||
target.config_key = default_config_key(prefix, target.id)
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Separation of duties (singleton row): may operators roll changes out themselves?"""
|
||||
from sqlalchemy import Boolean, Integer
|
||||
from sqlalchemy import JSON, Boolean, Integer
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
@@ -11,3 +11,14 @@ class GovernanceSettings(Base):
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
# False: an operator prepares changes (edit, preview) but only an admin may deploy them.
|
||||
operators_may_deploy: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
|
||||
# Rule cleanup (1.31). At most this many rules per cleanup operation (default 10, ceiling 25).
|
||||
cleanup_max_rules: Mapped[int] = mapped_column(Integer, nullable=False, default=10)
|
||||
# A rule must have been switched off (and confirmed) this many days before it may be deleted (minimum 1).
|
||||
cleanup_delete_min_days: Mapped[int] = mapped_column(Integer, nullable=False, default=7)
|
||||
# IPs/CIDRs the admin manages the firewalls from; no rule that could match them may be cleaned up. Empty list =
|
||||
# the cleanup is locked (fail closed). Only an admin may change it.
|
||||
cleanup_management_sources: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# Extra TCP ports the firewalls are managed on (besides the API port of base_url, 443, 80 and 22): a rule that
|
||||
# could match a management source on one of them is protected too. Only an admin may change it.
|
||||
cleanup_management_ports: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""GUI access to managed firewalls through the separate gui-gateway container (1.27).
|
||||
|
||||
Settings are a singleton row (id fixed at 1), off by default. Tickets and gateway
|
||||
sessions only ever store hashes: the raw ticket travels once through the browser,
|
||||
the OPNexus session is referenced by the same SHA-256 that `sessions.token_hash` holds.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
class GuiAccessSettings(Base):
|
||||
__tablename__ = "gui_access_settings"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
idle_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=15, server_default="15")
|
||||
absolute_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
|
||||
slot_count: Mapped[int] = mapped_column(Integer, nullable=False, default=4, server_default="4")
|
||||
updated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class GuiTicket(Base):
|
||||
"""One-time, 30-second ticket, bound to the OPNexus session, host and slot."""
|
||||
__tablename__ = "gui_tickets"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
ticket_hash: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
slot: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
redeemed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class GuiSession(Base):
|
||||
"""A redeemed ticket: one firewall GUI open in one slot of one browser session."""
|
||||
__tablename__ = "gui_sessions"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
slot: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
client_ip: Mapped[str | None] = mapped_column(String(45), nullable=True)
|
||||
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
last_checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
end_reason: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
@@ -68,6 +68,10 @@ class NotificationSettings(Base):
|
||||
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# An API key that cannot read gateway / service state makes those alerts silently blind (1.29): say so.
|
||||
api_rights_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
# Gateways whose alert is switched off, "<host_id>:<gateway name>" (1.29), e.g. an idle second WAN.
|
||||
gateway_alert_muted: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
|
||||
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
|
||||
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.one_to_one_target import one_to_one_targets
|
||||
from app.models.policy import one_to_one_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
|
||||
__tablename__ = "one_to_one_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
|
||||
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
|
||||
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OneToOneRule, "nat1")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.outbound_nat_target import outbound_nat_targets
|
||||
from app.models.policy import outbound_nat_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
|
||||
__tablename__ = "outbound_nat_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
|
||||
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
|
||||
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OutboundNatRule, "out")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
|
||||
|
||||
class PolicyHierarchyRule(Base):
|
||||
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
|
||||
__tablename__ = "policy_hierarchy_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
|
||||
)
|
||||
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
|
||||
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
|
||||
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PolicyHierarchyRule, "ph")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.port_forward_target import port_forward_targets
|
||||
from app.models.policy import port_forward_groups
|
||||
|
||||
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
|
||||
__tablename__ = "port_forward_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
|
||||
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
|
||||
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
|
||||
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PortForwardRule, "pf")
|
||||
@@ -42,8 +42,12 @@ class RolloutPlan(Base):
|
||||
# is activated AND unchanged since (activated_fingerprint).
|
||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
|
||||
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
|
||||
activated_by_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
|
||||
@@ -91,6 +95,7 @@ class RolloutRun(Base):
|
||||
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_by: Mapped[str] = mapped_column(String(150))
|
||||
started_by_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True) # see activated_by_id
|
||||
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
|
||||
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
|
||||
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.rule_target import rule_targets
|
||||
from app.models.policy import rule_groups
|
||||
|
||||
@@ -18,6 +19,7 @@ class FirewallRule(Base):
|
||||
__tablename__ = "firewall_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
direction: Mapped[str] = mapped_column(String(8))
|
||||
@@ -44,3 +46,6 @@ class FirewallRule(Base):
|
||||
deployments: Mapped[list["RuleDeployment"]] = relationship(
|
||||
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(FirewallRule, "r")
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): switching rules off or deleting them through preview -> deploy -> confirm.
|
||||
|
||||
``RuleCleanupOperation`` is one cleanup run on a host (or a cluster pair, anchored at one host); its id doubles as
|
||||
the preview token. ``RuleCleanupItem`` is one rule on one node of that run. The columns only carry what the design
|
||||
(DESIGN-1.31) needs to resume, revert and restore after a crash; the logic lives in services/rule_cleanup_service.py.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import JSON, DateTime, ForeignKey, Index, Integer, String, Text, text
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
# kind: disable | enable (re-enable after a restore) | delete | restore
|
||||
OPERATION_KINDS = ("disable", "enable", "delete", "restore")
|
||||
# preview -> running -> awaiting_confirm -> confirmed; the rest are the side exits.
|
||||
OPERATION_STATUSES = (
|
||||
"preview", "running", "awaiting_confirm", "confirmed", "partial", "apply_failed", "reverting", "reverted",
|
||||
"revert_failed", "done", "failed", "cancelled", "expired",
|
||||
)
|
||||
ITEM_STATES = (
|
||||
"queued", "disabling", "disabled", "confirmed", "reverting", "reverted", "deleting", "deleted", "restoring",
|
||||
"restored", "failed", "unknown", "revert_failed",
|
||||
# never attempted (the run stopped at an earlier error); not an open state, so the rule is free again
|
||||
"skipped",
|
||||
)
|
||||
# A rule in one of these states is being worked on or its outcome is unresolved: no second item may claim it.
|
||||
OPEN_ITEM_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class RuleCleanupOperation(Base):
|
||||
__tablename__ = "rule_cleanup_operations"
|
||||
__table_args__ = (
|
||||
Index("ix_rule_cleanup_operations_host", "host_id", "created_at"),
|
||||
Index("ix_rule_cleanup_operations_status", "status"),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
cluster_name: Mapped[Optional[str]] = mapped_column(String(120), nullable=True)
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="preview")
|
||||
actor: Mapped[str] = mapped_column(String(150), nullable=False)
|
||||
# The stable account id next to the name (see migration 0026); no foreign key, a deleted account stays gone.
|
||||
actor_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
started_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Written at every checkpoint, so another process can tell an abandoned ``running`` operation from a live one.
|
||||
heartbeat_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Auto-revert deadline (disable/enable only); NULL once confirmed or for kinds without a window.
|
||||
revert_deadline: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
confirmed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||
# sha256 over everything the preview showed; the deploy recomputes it and refuses on any difference.
|
||||
preview_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# unused_days, acknowledged rules (with the text version), protected management sources.
|
||||
options: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
|
||||
|
||||
class RuleCleanupItem(Base):
|
||||
__tablename__ = "rule_cleanup_items"
|
||||
__table_args__ = (
|
||||
Index("ix_rule_cleanup_items_operation", "operation_id"),
|
||||
# Independent of any lock: one rule on one node can be in only one unfinished item at a time.
|
||||
Index(
|
||||
"uq_cleanup_rule_open", "host_id", "rule_uuid", unique=True,
|
||||
postgresql_where=text("state IN (" + ", ".join(f"'{s}'" for s in OPEN_ITEM_STATES) + ")"),
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
operation_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rule_cleanup_operations.id", ondelete="CASCADE"), nullable=False)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
seq: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
carp_role: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
state: Mapped[str] = mapped_column(String(16), nullable=False, default="queued")
|
||||
# What the preview showed.
|
||||
description: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
categories: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
confidence: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
row_before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
fp_before: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# shadowed/duplicate: the covering rule must stay unchanged until the deploy.
|
||||
cover_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
cover_fp: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# Flattened getRule, raw answer and firmware; written BEFORE a rule is deleted.
|
||||
snapshot: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
# Where the rule sat (neighbours in the same prio_group by sort_order), for the restore.
|
||||
anchor_next_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
anchor_prev_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
prio_group: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
# delete -> the confirmed disable item it builds on; restore -> the delete item it undoes. Plain id, no foreign key.
|
||||
source_item_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
restored_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
changed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Change history of the rules that are live on a firewall (all rules, also hand-made ones).
|
||||
|
||||
``live_rule_states`` is the last observed state per rule (a fingerprint plus the row itself), maintained by the
|
||||
``rule_observe`` loop; ``rule_change_events`` is what changed between two observations. See
|
||||
services/rule_history_service.py. Neither table holds anything the firewall itself would not show on its rule page.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Index, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
class LiveRuleState(Base):
|
||||
__tablename__ = "live_rule_states"
|
||||
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), primary_key=True)
|
||||
# The rule's UUID/label on the firewall; BASELINE_KEY marks "this host has been observed once".
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
row: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
# False = the rule is gone (kept as a marker so a returning UUID continues the numbering).
|
||||
present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
# Counts the observed changes of this rule; the event of a change gets the new number, which makes the
|
||||
# observation idempotent when two processes see the same change.
|
||||
version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
changed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
|
||||
|
||||
class RuleChangeEvent(Base):
|
||||
__tablename__ = "rule_change_events"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("host_id", "rule_uuid", "seq", name="uq_rule_change_event"),
|
||||
Index("ix_rule_change_events_rule", "host_id", "rule_uuid", "observed_at"),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
seq: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
observed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True)
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False) # added | changed | removed | enabled | disabled
|
||||
before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
after: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
origin: Mapped[str] = mapped_column(String(16), nullable=False) # opnexus | external
|
||||
actor: Mapped[Optional[str]] = mapped_column(String(150), nullable=True)
|
||||
# The audit entry that explains an OPNexus-made change; no foreign key, the audit trail outlives everything.
|
||||
audit_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
# Start of the time since the previous observation of the host: only audit entries from then on can explain
|
||||
# this change (a later reconcile must not reach back to an older deployment).
|
||||
window_start: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
@@ -6,6 +6,7 @@ outbound NAT share (services/object_adopt_service.py has the logic and the evide
|
||||
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
|
||||
|
||||
Like rule adoption this only creates central records; nothing is written to the firewall."""
|
||||
import json
|
||||
import uuid
|
||||
from typing import Callable
|
||||
|
||||
@@ -18,6 +19,7 @@ from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.adopt import AdoptAllRequest, AdoptAllResult, AdoptObjectRequest, UnmanagedObject
|
||||
from app.services import object_adopt_service
|
||||
from app.services.adopt_common import LINK_ERRORS, link_api_error
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import policy_lock
|
||||
@@ -44,11 +46,14 @@ def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Calla
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
created = await object_adopt_service.adopt(db, kind, host, payload.uuid)
|
||||
created = await object_adopt_service.adopt(db, kind, host, payload.uuid, link_to=payload.link_to)
|
||||
except object_adopt_service.NotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
|
||||
except object_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
||||
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons),
|
||||
reason_details=json.dumps(exc.details, ensure_ascii=False)) from exc
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
@@ -65,7 +70,9 @@ def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Calla
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
result = await object_adopt_service.adopt_all(db, kind, host)
|
||||
result = await object_adopt_service.adopt_all(db, kind, host, link_existing=payload.link_existing)
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
|
||||
@@ -205,6 +205,8 @@ async def remove_previous_target(alias_id: uuid.UUID, host_id: uuid.UUID, approv
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in alias.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_alias_item(deployment.host, deployment.opnsense_alias_uuid)
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from sqlalchemy import select, func, or_
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models.audit import AuditEvent
|
||||
from app.services.audit_service import ADMIN_ONLY_DETAILS
|
||||
|
||||
router = APIRouter(prefix="/audit", tags=["audit"])
|
||||
|
||||
@router.get("")
|
||||
async def history(kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||
async def history(request: Request, kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||
status: str | None = None, action: str | None = None, actor: str | None = None,
|
||||
before: datetime | None = None, since: datetime | None = None,
|
||||
q: str = Query(default="", max_length=200), offset: int = Query(default=0, ge=0),
|
||||
@@ -24,5 +25,17 @@ async def history(kind: str | None = None, host_id: str | None = None, entity_id
|
||||
clauses.append(or_(*(c.ilike('%'+q+'%') for c in [AuditEvent.entity_name, AuditEvent.host_name, AuditEvent.actor, AuditEvent.message])))
|
||||
total = await db.scalar(select(func.count()).select_from(AuditEvent).where(*clauses))
|
||||
rows = (await db.scalars(select(AuditEvent).where(*clauses).order_by(AuditEvent.created_at.desc(), AuditEvent.id.desc()).offset(offset).limit(limit))).all()
|
||||
return {"total": total, "offset": offset, "limit": limit,
|
||||
"items": [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]}
|
||||
items = [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]
|
||||
if not full_view(request):
|
||||
for item in items:
|
||||
if isinstance(item.get("details"), dict) and ADMIN_ONLY_DETAILS in item["details"]:
|
||||
item["details"] = {**{k: v for k, v in item["details"].items() if k != ADMIN_ONLY_DETAILS},
|
||||
ADMIN_ONLY_DETAILS: "redacted"}
|
||||
return {"total": total, "offset": offset, "limit": limit, "items": items}
|
||||
|
||||
|
||||
def full_view(request: Request | None) -> bool:
|
||||
"""An admin account, and with a token only one of scope admin."""
|
||||
user = getattr(getattr(request, "state", None), "user", None)
|
||||
token = getattr(getattr(request, "state", None), "api_token", None)
|
||||
return getattr(user, "role", None) == "admin" and (token is None or token.scope == "admin")
|
||||
@@ -15,7 +15,31 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
||||
|
||||
# Plain name: the session cookie of HTTP-only (lab) deployments. On HTTPS deployments it is NOT accepted any
|
||||
# more (1.29): it is only still deleted (login, logout) so that a pre-1.27 cookie does not linger.
|
||||
COOKIE_NAME = "opnexus-session"
|
||||
# HTTPS deployments: the __Host- prefix makes the browser refuse any cookie of
|
||||
# this name that is not Secure, has a Domain attribute or a Path other than
|
||||
# "/". A page on another port of the same host (the GUI gateway slots, any
|
||||
# other service on that machine) therefore cannot toss a narrower-path cookie
|
||||
# that shadows the real session, and cannot overwrite it from script (HttpOnly).
|
||||
SECURE_COOKIE_NAME = "__Host-opnexus-session"
|
||||
|
||||
|
||||
def session_cookie_name() -> str:
|
||||
return SECURE_COOKIE_NAME if settings.cookie_secure else COOKIE_NAME
|
||||
|
||||
|
||||
def session_token(request: Request) -> str:
|
||||
"""The session token the browser sent: the __Host- cookie on HTTPS deployments, the plain one otherwise.
|
||||
Never the other name (1.29): a plain-name cookie can be planted by any plain-HTTP or sibling-subdomain page and
|
||||
must not be able to carry a session on an HTTPS deployment (the __Host- prefix does not separate PORTS, see
|
||||
BACKLOG S8)."""
|
||||
return request.cookies.get(session_cookie_name(), "")
|
||||
|
||||
|
||||
def has_session_cookie(request: Request) -> bool:
|
||||
return bool(request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME))
|
||||
|
||||
|
||||
def client_ip(request: Request) -> str:
|
||||
@@ -32,7 +56,7 @@ def client_ip(request: Request) -> str:
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, max_age: int | None = None) -> None:
|
||||
response.set_cookie(
|
||||
key=COOKIE_NAME,
|
||||
key=session_cookie_name(),
|
||||
value=token,
|
||||
max_age=max_age or settings.session_lifetime,
|
||||
httponly=True,
|
||||
@@ -40,10 +64,15 @@ def _set_session_cookie(response: Response, token: str, max_age: int | None = No
|
||||
secure=settings.cookie_secure,
|
||||
path="/",
|
||||
)
|
||||
if settings.cookie_secure:
|
||||
# A pre-1.27 plain-name cookie must not linger next to the new one.
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||
|
||||
|
||||
def _clear_session_cookie(response: Response) -> None:
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/")
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=settings.cookie_secure, httponly=True, samesite="lax")
|
||||
if settings.cookie_secure:
|
||||
response.delete_cookie(key=SECURE_COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||
|
||||
|
||||
def _check_account_budget(user: AdminUser, action: str) -> None:
|
||||
@@ -71,7 +100,7 @@ async def resolve_user(request: Request, db: AsyncSession) -> AdminUser:
|
||||
user = await db.get(AdminUser, resolved.id)
|
||||
if user is not None:
|
||||
return user
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
user = await auth_service.resolve_session(db, token)
|
||||
if user is None:
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.NICHT_AUTHENTIFIZIERT)
|
||||
@@ -229,7 +258,7 @@ async def totp_enable(
|
||||
user.totp_last_step = step
|
||||
user.totp_recovery_hashes = hashes
|
||||
# Other sessions were established without the second factor — end them.
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
|
||||
await db.commit()
|
||||
logger.info("Two-factor enabled for '%s'", user.username)
|
||||
return {"enabled": True, "recovery_codes": plain}
|
||||
@@ -254,7 +283,7 @@ async def totp_disable(
|
||||
user.totp_enabled = False
|
||||
user.totp_recovery_hashes = None
|
||||
user.totp_last_step = 0
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
|
||||
await db.commit()
|
||||
logger.info("Two-factor disabled for '%s'", user.username)
|
||||
return {"enabled": False}
|
||||
@@ -262,14 +291,17 @@ async def totp_disable(
|
||||
|
||||
@router.post("/logout")
|
||||
async def logout(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
await auth_service.revoke_session(db, token)
|
||||
legacy = request.cookies.get(COOKIE_NAME, "")
|
||||
if settings.cookie_secure and legacy and legacy != token:
|
||||
await auth_service.revoke_session(db, legacy) # an old plain-name cookie ends with the logout too
|
||||
_clear_session_cookie(response)
|
||||
return {"authenticated": False}
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
async def me(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async def me(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||
raw = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
|
||||
if raw is not None:
|
||||
resolved = await api_token_service.resolve(db, raw, client_ip(request))
|
||||
@@ -278,7 +310,7 @@ async def me(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
user, api_token = resolved
|
||||
return {"authenticated": True, "username": user.username, "role": user.role,
|
||||
"via": "token", "scope": api_token.scope, "token_name": api_token.name}
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
user = await auth_service.resolve_session(db, token)
|
||||
if user is None:
|
||||
return {"authenticated": False}
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
|
||||
|
||||
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
|
||||
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
|
||||
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
|
||||
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, parse
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
router = APIRouter(prefix="/config", tags=["config"])
|
||||
|
||||
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
|
||||
# and read before parsing so a huge body never reaches the JSON parser.
|
||||
MAX_REQUEST_BYTES = 4 * MAX_BYTES
|
||||
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
|
||||
# open to read-only accounts and read tokens.
|
||||
_busy = asyncio.Semaphore(1)
|
||||
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
|
||||
MAX_WAITING = 4
|
||||
_waiting = 0
|
||||
|
||||
|
||||
class _Slot:
|
||||
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
|
||||
|
||||
async def __aenter__(self):
|
||||
global _waiting
|
||||
if _waiting >= MAX_WAITING:
|
||||
raise api_error(429, ErrorCode.CONFIG_BUSY)
|
||||
_waiting += 1
|
||||
return self
|
||||
|
||||
async def run(self, coro_fn):
|
||||
async with _busy:
|
||||
return await coro_fn()
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
global _waiting
|
||||
_waiting -= 1
|
||||
|
||||
|
||||
def _encode(text: str) -> bytes:
|
||||
try:
|
||||
return text.encode("utf-8")
|
||||
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
class ImportRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
yaml: str
|
||||
prune: bool = False
|
||||
|
||||
|
||||
class ApplyRequest(ImportRequest):
|
||||
plan_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
def _config_error(exc: ConfigError):
|
||||
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
|
||||
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
|
||||
|
||||
|
||||
async def _read_body(request: Request, model: type[BaseModel]):
|
||||
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks, size = [], 0
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks.append(chunk)
|
||||
try:
|
||||
return model.model_validate(json.loads(b"".join(chunks)))
|
||||
except (ValueError, ValidationError):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
@router.get("/export")
|
||||
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
|
||||
wanted = None
|
||||
if kinds:
|
||||
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
|
||||
text = await export_yaml(db, wanted)
|
||||
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
|
||||
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
|
||||
|
||||
|
||||
@router.post("/import/plan")
|
||||
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ImportRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
return await planner.build_plan(db, parsed, body.prune)
|
||||
except ConfigError as exc:
|
||||
raise _config_error(exc) from None
|
||||
finally:
|
||||
await db.rollback() # never leave anything in the session: plan is read-only
|
||||
|
||||
result = await slot.run(work)
|
||||
return result.public()
|
||||
|
||||
|
||||
@router.post("/import/apply")
|
||||
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ApplyRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
|
||||
except ConfigError as exc:
|
||||
await db.rollback()
|
||||
raise _config_error(exc) from None
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
return parsed, result
|
||||
|
||||
parsed, result = await slot.run(work)
|
||||
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
|
||||
@@ -49,7 +49,7 @@ from app.models.wireguard_tunnel import WireguardTunnel
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service
|
||||
from app.services import gateway_health_service, notification_settings_service
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.monitor_checks import STALE_AFTER
|
||||
|
||||
@@ -207,34 +207,43 @@ async def collect_pending_actions(db: AsyncSession) -> list[PendingAction]:
|
||||
return pending_actions
|
||||
|
||||
|
||||
async def _gateways_by_host(db: AsyncSession, hosts) -> dict:
|
||||
"""Gateway state from each host's newest resource sample; hosts without a fresh sample are left out."""
|
||||
result: dict = {}
|
||||
async def _sample_readings_by_host(db: AsyncSession, hosts, muted: set[str]) -> tuple[dict, dict]:
|
||||
"""(gateways, missing API rights) from each host's newest resource sample; hosts without a fresh sample are left out."""
|
||||
gateways: dict = {}
|
||||
denied: dict = {}
|
||||
now = datetime.now(timezone.utc)
|
||||
for host in hosts:
|
||||
row = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json())
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json(), HostResourceSample.raw["denied"].as_json())
|
||||
.where(HostResourceSample.host_id == host.id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).first()
|
||||
if row is not None and row[1] is not None and now - row[0] <= STALE_AFTER:
|
||||
result[host.id] = gateway_health_service.from_compact(row[1])
|
||||
return result
|
||||
if row is None or now - row[0] > STALE_AFTER:
|
||||
continue
|
||||
if row[1] is not None:
|
||||
gateways[host.id] = [
|
||||
g.model_copy(update={"alert_enabled": f"{host.id}:{g.name}" not in muted})
|
||||
for g in gateway_health_service.from_compact(row[1])
|
||||
]
|
||||
if isinstance(row[2], list):
|
||||
denied[host.id] = [str(x) for x in row[2]]
|
||||
return gateways, denied
|
||||
|
||||
|
||||
@router.get("", response_model=DashboardSummary)
|
||||
async def summary(db: AsyncSession = Depends(get_db)):
|
||||
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
||||
statuses = await get_statuses(hosts)
|
||||
gateways = await _gateways_by_host(db, hosts)
|
||||
muted = set((await notification_settings_service.get_settings(db)).gateway_alert_muted or [])
|
||||
gateways, denied = await _sample_readings_by_host(db, hosts, muted)
|
||||
host_summaries = [
|
||||
DashboardHost(
|
||||
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
|
||||
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
|
||||
gateways=gateways.get(host.id),
|
||||
gateways=gateways.get(host.id), api_rights_missing=denied.get(host.id),
|
||||
)
|
||||
for host in hosts
|
||||
]
|
||||
|
||||
@@ -1,17 +1,26 @@
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.gateway import GatewayHealth
|
||||
from app.services import gateway_health_service
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.schemas.gateway import ApiRightsState, GatewayAlertState, GatewayAlertUpdate, GatewayHealth
|
||||
from app.services import audit_service, gateway_health_service, notification_settings_service
|
||||
from app.services.monitor_checks import STALE_AFTER
|
||||
from app.services.opnsense_client import OpnsenseAPIError, OpnsenseForbiddenError
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
|
||||
|
||||
# OPNsense gateway names are letters, digits and underscores (WAN_GW, WAN_DHCP6); a little more is allowed.
|
||||
_GATEWAY_NAME = re.compile(r"[A-Za-z0-9_.-]{1,64}")
|
||||
MAX_MUTED_GATEWAYS = 200
|
||||
|
||||
|
||||
@router.get("", response_model=list[GatewayHealth])
|
||||
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
@@ -20,6 +29,71 @@ async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
return await gateway_health_service.get_gateway_health(host)
|
||||
gateways = await gateway_health_service.get_gateway_health(host)
|
||||
except OpnsenseForbiddenError as exc:
|
||||
# The API key may not read the gateway state: say so instead of a generic upstream failure.
|
||||
raise api_error(403, ErrorCode.API_RIGHTS_MISSING) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
muted = set((await notification_settings_service.get_settings(db)).gateway_alert_muted or [])
|
||||
return [g.model_copy(update={"alert_enabled": f"{host.id}:{g.name}" not in muted}) for g in gateways]
|
||||
|
||||
|
||||
@router.get("/api-rights", response_model=ApiRightsState)
|
||||
async def api_rights(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
"""Side readings the newest FRESH resource sample could not take because the API key lacks the right
|
||||
(null = no fresh sample, so nothing is known). Stored data only: no call to the firewall."""
|
||||
if await db.get(OpnsenseHost, host_id) is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
row = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["denied"].as_json())
|
||||
.where(HostResourceSample.host_id == host_id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).first()
|
||||
if row is None or datetime.now(timezone.utc) - row[0] > STALE_AFTER or not isinstance(row[1], list):
|
||||
return ApiRightsState(missing=None)
|
||||
return ApiRightsState(missing=[str(x) for x in row[1]])
|
||||
|
||||
|
||||
@router.put("/{name}/alert", response_model=GatewayAlertState)
|
||||
async def set_gateway_alert(host_id: uuid.UUID, name: str, payload: GatewayAlertUpdate, db: AsyncSession = Depends(get_db)):
|
||||
"""Switch the gateway-down alert on or off for ONE gateway (admins only, see api_token_service)."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
if not _GATEWAY_NAME.fullmatch(name):
|
||||
raise api_error(422, ErrorCode.GATEWAY_NAME_INVALID)
|
||||
if not payload.enabled:
|
||||
# Only a gateway the firewall really reported can be muted ("wan_gw" vs "WAN_GW" would silently do nothing).
|
||||
# Without a usable sample nothing is known, so the request goes through.
|
||||
known = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.raw["gateways"].as_json())
|
||||
.where(HostResourceSample.host_id == host_id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).scalar()
|
||||
if isinstance(known, list) and name not in {g.get("name") for g in known if isinstance(g, dict)}:
|
||||
raise api_error(404, ErrorCode.GATEWAY_UNKNOWN)
|
||||
row = await notification_settings_service.get_settings(db)
|
||||
await db.refresh(row, with_for_update=True)
|
||||
muted = list(row.gateway_alert_muted or [])
|
||||
key = f"{host.id}:{name}"
|
||||
if payload.enabled:
|
||||
muted = [m for m in muted if m != key]
|
||||
elif key not in muted:
|
||||
if len(muted) >= MAX_MUTED_GATEWAYS:
|
||||
raise api_error(422, ErrorCode.GATEWAY_MUTE_LIMIT, limit=str(MAX_MUTED_GATEWAYS))
|
||||
muted.append(key)
|
||||
row.gateway_alert_muted = muted # a new list, so the JSON column notices the change
|
||||
await db.commit()
|
||||
# The request middleware logs the call itself; this entry says what it did (only the validated name goes in).
|
||||
await audit_service.persist(
|
||||
kind="host", action="gateway_alert_on" if payload.enabled else "gateway_alert_off", entity_id=str(host.id),
|
||||
host_id=str(host.id), host_name=host.name, status="accepted", message=f"Gateway {name}",
|
||||
)
|
||||
return GatewayAlertState(name=name, alert_enabled=payload.enabled)
|
||||
@@ -1,6 +1,15 @@
|
||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it)."""
|
||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it).
|
||||
|
||||
The rule-cleanup settings (1.31) live here too: how many rules one cleanup may touch, how long a rule must stay
|
||||
switched off before it may be deleted, the management sources (addresses the firewalls are managed from; no rule
|
||||
that could match them is ever cleaned up) and extra management ports (besides the API port, 443, 80 and 22). Only an admin writes any of it (middleware: ``/api/settings/...`` is
|
||||
admin-only for operators and tokens below scope admin; checked again here) and every change is audited with the
|
||||
old and the new values."""
|
||||
import ipaddress
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
@@ -8,12 +17,83 @@ from app.errors import ErrorCode, api_error
|
||||
from app.models.governance import GovernanceSettings
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.services import audit_service
|
||||
|
||||
router = APIRouter(prefix="/settings/governance", tags=["governance"])
|
||||
|
||||
MAX_CLEANUP_RULES = 25 # same ceiling as services/rule_cleanup_service.MAX_RULES_CEILING
|
||||
MAX_DELETE_MIN_DAYS = 365
|
||||
MAX_MANAGEMENT_SOURCES = 20
|
||||
MAX_MANAGEMENT_PORTS = 20
|
||||
|
||||
|
||||
class GovernanceRead(BaseModel):
|
||||
operators_may_deploy: bool
|
||||
cleanup_max_rules: int = 10
|
||||
cleanup_delete_min_days: int = 7
|
||||
# The list itself is shown to admins only; everybody else sees just whether it is set.
|
||||
cleanup_management_sources: list[str] = Field(default_factory=list)
|
||||
cleanup_management_sources_configured: bool = False
|
||||
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
|
||||
cleanup_management_ports: list[int] = Field(default_factory=list)
|
||||
cleanup_management_ports_configured: bool = False
|
||||
|
||||
|
||||
class GovernanceUpdate(BaseModel):
|
||||
operators_may_deploy: bool | None = None
|
||||
cleanup_max_rules: int | None = None
|
||||
cleanup_delete_min_days: int | None = None
|
||||
cleanup_management_sources: list[str] | None = None
|
||||
cleanup_management_ports: list[Any] | None = None # validated in clean_management_ports (no bool coercion)
|
||||
|
||||
|
||||
def _bad(field: str, reason: str):
|
||||
return api_error(422, ErrorCode.BEREINIGUNG_EINSTELLUNG_UNGUELTIG, field=field, reason=reason)
|
||||
|
||||
|
||||
def clean_management_sources(values: list[str]) -> list[str]:
|
||||
"""IP addresses and CIDR networks only (no host names: they could resolve differently later), canonical form,
|
||||
without duplicates; not the whole address space, which would protect every rule."""
|
||||
if len(values) > MAX_MANAGEMENT_SOURCES:
|
||||
raise _bad("cleanup_management_sources", "too_many")
|
||||
cleaned: list[str] = []
|
||||
for raw in values:
|
||||
text = str(raw).strip()
|
||||
try:
|
||||
net = ipaddress.ip_network(text, strict=False)
|
||||
except ValueError:
|
||||
raise _bad("cleanup_management_sources", "not_an_address") from None
|
||||
if net.prefixlen == 0:
|
||||
raise _bad("cleanup_management_sources", "everything")
|
||||
entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)
|
||||
if entry not in cleaned:
|
||||
cleaned.append(entry)
|
||||
return cleaned
|
||||
|
||||
|
||||
def clean_management_ports(values) -> list[int]:
|
||||
"""TCP ports 1..65535, at most MAX_MANAGEMENT_PORTS, without duplicates, sorted."""
|
||||
if len(values) > MAX_MANAGEMENT_PORTS:
|
||||
raise _bad("cleanup_management_ports", "too_many")
|
||||
ports: set[int] = set()
|
||||
for raw in values:
|
||||
text = str(raw).strip()
|
||||
if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:
|
||||
raise _bad("cleanup_management_ports", "not_a_port")
|
||||
ports.add(int(text))
|
||||
return sorted(ports)
|
||||
|
||||
|
||||
def _read(row: GovernanceSettings, user: AdminUser) -> GovernanceRead:
|
||||
sources = list(row.cleanup_management_sources or [])
|
||||
ports = list(row.cleanup_management_ports or [])
|
||||
return GovernanceRead(
|
||||
operators_may_deploy=row.operators_may_deploy, cleanup_max_rules=row.cleanup_max_rules or 10,
|
||||
cleanup_delete_min_days=row.cleanup_delete_min_days or 7,
|
||||
cleanup_management_sources=sources if user.role == "admin" else [],
|
||||
cleanup_management_sources_configured=bool(sources),
|
||||
cleanup_management_ports=ports if user.role == "admin" else [],
|
||||
cleanup_management_ports_configured=bool(ports))
|
||||
|
||||
|
||||
async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
@@ -28,14 +108,33 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
@router.get("", response_model=GovernanceRead)
|
||||
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
|
||||
return await get_settings(db)
|
||||
return _read(await get_settings(db), user)
|
||||
|
||||
|
||||
@router.put("", response_model=GovernanceRead)
|
||||
async def update(payload: GovernanceRead, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
row = await get_settings(db)
|
||||
row.operators_may_deploy = payload.operators_may_deploy
|
||||
changes: dict[str, dict] = {}
|
||||
if payload.cleanup_max_rules is not None:
|
||||
if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:
|
||||
raise _bad("cleanup_max_rules", f"1..{MAX_CLEANUP_RULES}")
|
||||
if payload.cleanup_delete_min_days is not None:
|
||||
if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:
|
||||
raise _bad("cleanup_delete_min_days", f"1..{MAX_DELETE_MIN_DAYS}")
|
||||
sources = clean_management_sources(payload.cleanup_management_sources) if payload.cleanup_management_sources is not None else None
|
||||
ports = clean_management_ports(payload.cleanup_management_ports) if payload.cleanup_management_ports is not None else None
|
||||
for field, value in (("cleanup_max_rules", payload.cleanup_max_rules), ("cleanup_delete_min_days", payload.cleanup_delete_min_days),
|
||||
("cleanup_management_sources", sources), ("cleanup_management_ports", ports)):
|
||||
if value is not None and value != getattr(row, field):
|
||||
changes[field] = {"old": getattr(row, field), "new": value}
|
||||
setattr(row, field, value)
|
||||
if payload.operators_may_deploy is not None:
|
||||
row.operators_may_deploy = payload.operators_may_deploy
|
||||
if changes:
|
||||
db.add(audit_service.make_event(
|
||||
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
|
||||
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
|
||||
await db.commit()
|
||||
return row
|
||||
return _read(row, user)
|
||||
@@ -0,0 +1,190 @@
|
||||
"""Firewall GUI access through the gui-gateway container (1.27).
|
||||
|
||||
Public API (session cookie, admin only, never API tokens):
|
||||
* POST /api/hosts/{id}/gui-ticket — one-time ticket + the slot URL the browser opens (noopener).
|
||||
* GET/PUT /api/settings/gui-access — switch, idle/absolute timeout, number of slots.
|
||||
|
||||
Internal API for the gateway only (NOT under /api, so the proxy never forwards it; shared secret
|
||||
GUI_GATEWAY_TOKEN in the X-Onx-Gateway-Token header, compared in constant time):
|
||||
* POST /internal/gui/redeem, /internal/gui/revalidate, /internal/gui/end.
|
||||
The answers never contain firewall credentials: only the firewall origin, its pinned (public) CA
|
||||
certificate, the overlay proxy address and the timeouts.
|
||||
"""
|
||||
import hashlib
|
||||
import hmac
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Request, status
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import SECURE_COOKIE_NAME, current_user
|
||||
from app.services import api_token_service, gui_ticket_service as svc
|
||||
|
||||
router = APIRouter(tags=["gui-access"])
|
||||
internal_router = APIRouter(prefix="/internal/gui", include_in_schema=False)
|
||||
|
||||
|
||||
def _require_admin_session(request: Request, user: AdminUser) -> None:
|
||||
if getattr(request.state, "api_token", None) is not None or api_token_service.bearer_from_header(
|
||||
request.headers.get("authorization", "")
|
||||
) is not None:
|
||||
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.GUI_SESSION_REQUIRED)
|
||||
if user.role != "admin":
|
||||
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.ADMIN_REQUIRED)
|
||||
|
||||
|
||||
class GuiTicketRead(BaseModel):
|
||||
url: str
|
||||
slot: int
|
||||
expires_in: int
|
||||
|
||||
|
||||
@router.post("/hosts/{host_id}/gui-ticket", response_model=GuiTicketRead)
|
||||
async def create_ticket(host_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db),
|
||||
user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
if not svc.gateway_configured():
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED)
|
||||
session_cookie = request.cookies.get(SECURE_COOKIE_NAME, "")
|
||||
if not settings.cookie_secure or not session_cookie:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_REQUIRES_HTTPS)
|
||||
row = await svc.get_settings(db)
|
||||
if not row.enabled:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_ACCESS_DISABLED)
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(status.HTTP_404_NOT_FOUND, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
refusal = svc.host_refusal(host)
|
||||
if refusal:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_HOST_NOT_ELIGIBLE, reason=refusal)
|
||||
try:
|
||||
return await svc.issue_ticket(db, user=user, session_token=session_cookie, host=host,
|
||||
request_host=request.headers.get("host", ""))
|
||||
except svc.GuiAccessError as exc:
|
||||
raise api_error(exc.status, exc.code, **exc.params)
|
||||
|
||||
|
||||
class GuiAccessSettingsRead(BaseModel):
|
||||
enabled: bool
|
||||
idle_timeout_minutes: int
|
||||
absolute_timeout_minutes: int
|
||||
slot_count: int
|
||||
available: bool
|
||||
unavailable_reasons: list[str]
|
||||
slots: list[int]
|
||||
updated_by: str | None
|
||||
updated_at: datetime | None
|
||||
|
||||
|
||||
class GuiAccessSettingsUpdate(BaseModel):
|
||||
enabled: bool
|
||||
idle_timeout_minutes: int = Field(ge=1, le=240)
|
||||
absolute_timeout_minutes: int = Field(ge=5, le=720)
|
||||
slot_count: int = Field(ge=1, le=svc.MAX_SLOTS)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _idle_within_absolute(self):
|
||||
if self.idle_timeout_minutes > self.absolute_timeout_minutes:
|
||||
raise ValueError("idle_timeout_minutes must not exceed absolute_timeout_minutes")
|
||||
return self
|
||||
|
||||
|
||||
def _settings_read(row) -> dict:
|
||||
reasons = svc.unavailable_reasons()
|
||||
return {"enabled": row.enabled, "idle_timeout_minutes": row.idle_timeout_minutes,
|
||||
"absolute_timeout_minutes": row.absolute_timeout_minutes, "slot_count": row.slot_count,
|
||||
"available": not reasons, "unavailable_reasons": reasons, "slots": svc.active_slots(row),
|
||||
"updated_by": row.updated_by, "updated_at": row.updated_at}
|
||||
|
||||
|
||||
@router.get("/settings/gui-access", response_model=GuiAccessSettingsRead)
|
||||
async def read_settings(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
row = await svc.get_settings(db)
|
||||
await db.commit()
|
||||
return _settings_read(row)
|
||||
|
||||
|
||||
@router.put("/settings/gui-access", response_model=GuiAccessSettingsRead)
|
||||
async def update_settings(payload: GuiAccessSettingsUpdate, request: Request, db: AsyncSession = Depends(get_db),
|
||||
user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
if payload.enabled and svc.unavailable_reasons():
|
||||
# Never switchable on without HTTPS cookies and a configured gateway.
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED,
|
||||
reasons=", ".join(svc.unavailable_reasons()))
|
||||
row = await svc.get_settings(db)
|
||||
row.enabled = payload.enabled
|
||||
row.idle_timeout_minutes = payload.idle_timeout_minutes
|
||||
row.absolute_timeout_minutes = payload.absolute_timeout_minutes
|
||||
row.slot_count = payload.slot_count
|
||||
row.updated_by = user.username
|
||||
row.updated_at = datetime.now(timezone.utc)
|
||||
if not payload.enabled:
|
||||
await svc.end_all(db, "disabled")
|
||||
await db.commit()
|
||||
return _settings_read(row)
|
||||
|
||||
|
||||
# ── internal: gateway only ────────────────────────────────────────────────
|
||||
|
||||
def _gateway_auth(x_onx_gateway_token: str = Header(default="")) -> None:
|
||||
expected = settings.gui_gateway_token
|
||||
if len(expected) < svc.MIN_GATEWAY_TOKEN_LENGTH:
|
||||
raise HTTPException(status_code=404)
|
||||
# Fixed-length digests: the comparison time does not depend on the presented value.
|
||||
if not hmac.compare_digest(hashlib.sha256(x_onx_gateway_token.encode()).digest(),
|
||||
hashlib.sha256(expected.encode()).digest()):
|
||||
raise HTTPException(status_code=401)
|
||||
|
||||
|
||||
_HEX64 = r"^[0-9a-f]{64}$"
|
||||
|
||||
|
||||
class RedeemRequest(BaseModel):
|
||||
ticket_hash: str = Field(pattern=_HEX64)
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
slot: int = Field(ge=1, le=65535)
|
||||
client_ip: str | None = Field(default=None, max_length=45)
|
||||
|
||||
|
||||
class RevalidateRequest(BaseModel):
|
||||
gui_session_id: uuid.UUID
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
# Fingerprint of the target as handed out at redeem time (optional: an older gateway image omits it).
|
||||
target_fp: str | None = Field(default=None, pattern=_HEX64)
|
||||
|
||||
|
||||
class EndRequest(BaseModel):
|
||||
gui_session_id: uuid.UUID
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
reason: Literal["idle", "absolute", "logout", "revoked", "superseded", "shutdown"]
|
||||
|
||||
|
||||
@internal_router.post("/redeem", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_redeem(payload: RedeemRequest, db: AsyncSession = Depends(get_db)):
|
||||
result = await svc.redeem(db, ticket_hash=payload.ticket_hash, session_hash=payload.session_hash,
|
||||
slot=payload.slot, client_ip=payload.client_ip)
|
||||
if not result["ok"]:
|
||||
raise HTTPException(status_code=403, detail={"ok": False, "reason": result["reason"]})
|
||||
return result
|
||||
|
||||
|
||||
@internal_router.post("/revalidate", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_revalidate(payload: RevalidateRequest, db: AsyncSession = Depends(get_db)):
|
||||
return await svc.revalidate(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
|
||||
target_fp=payload.target_fp)
|
||||
|
||||
|
||||
@internal_router.post("/end", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_end(payload: EndRequest, db: AsyncSession = Depends(get_db)):
|
||||
return {"ok": await svc.end(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
|
||||
reason=payload.reason)}
|
||||
@@ -147,6 +147,11 @@ async def get_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _get_host_or_404(db, host_id)
|
||||
if (payload.base_url is not None or payload.enabled is not None or payload.clear_cluster_name
|
||||
or payload.cluster_name is not None):
|
||||
# An open rule cleanup (1.31) reverts through this address and this cluster pairing: not while it is open.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
|
||||
if payload.name is not None:
|
||||
existing = await db.scalar(
|
||||
@@ -188,6 +193,9 @@ async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
# Both deployment tables FK to opnsense_hosts with ondelete=CASCADE, so
|
||||
# deleting the host would silently drop OPNexus's only record of what
|
||||
# it pushed there — the rule/alias itself stays live on the actual
|
||||
@@ -226,6 +234,12 @@ async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
draft_certs = (await db.scalars(select(ManagedCertificate.id).where(ManagedCertificate.host_id == host_id))).all()
|
||||
draft_accounts = (await db.scalars(select(AcmeAccount.id).where(AcmeAccount.host_id == host_id))).all()
|
||||
await db.delete(host)
|
||||
# Gateway alert switches of this host would otherwise stay in the settings forever (and count against the limit).
|
||||
from app.services import notification_settings_service
|
||||
settings_row = await notification_settings_service.get_settings(db)
|
||||
prefix = f"{host_id}:"
|
||||
if any(str(m).startswith(prefix) for m in settings_row.gateway_alert_muted or []):
|
||||
settings_row.gateway_alert_muted = [m for m in settings_row.gateway_alert_muted if not str(m).startswith(prefix)]
|
||||
await db.commit()
|
||||
opnsense_secret_store.delete_secret(host_id)
|
||||
for cert_id in draft_certs:
|
||||
|
||||
@@ -193,6 +193,8 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_child_uuid:
|
||||
|
||||
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
|
||||
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
|
||||
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
|
||||
|
||||
class Preview(BaseModel):
|
||||
action: Literal['reboot','update','major_update']
|
||||
action: Literal['reboot','update','major_update','firmware_rollback']
|
||||
|
||||
class Execute(BaseModel):
|
||||
preview_id: uuid.UUID
|
||||
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
|
||||
def read_operation(op):
|
||||
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
|
||||
|
||||
async def snapshot_or_502(db, host):
|
||||
try: return await inspect_host(db,host)
|
||||
async def snapshot_or_502(db, host, action=None):
|
||||
try:
|
||||
state=await inspect_host(db,host)
|
||||
if action=='firmware_rollback':
|
||||
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
|
||||
rollback=await snapshot_service.rollback_state(host)
|
||||
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
|
||||
state['rollback']=rollback
|
||||
state['actions']['firmware_rollback']=state['actions']['reboot']+(
|
||||
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
|
||||
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
|
||||
return state
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
|
||||
|
||||
async def _changes_since(db, host, target):
|
||||
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
|
||||
if not target:
|
||||
return []
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return []
|
||||
out=[]
|
||||
for kind,model in _DEPLOYMENT_MODELS():
|
||||
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
|
||||
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
|
||||
return sorted(out,key=lambda r:r['deployed_at'])[:50]
|
||||
|
||||
|
||||
def _point_time(target):
|
||||
try:
|
||||
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _DEPLOYMENT_MODELS():
|
||||
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
|
||||
PortForwardDeployment, RuleDeployment)
|
||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
|
||||
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
|
||||
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
|
||||
|
||||
|
||||
async def _mark_rolled_back(db, host, target):
|
||||
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
|
||||
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
|
||||
really writes, and the UI shows the object as not in sync."""
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return 0
|
||||
count=0
|
||||
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
|
||||
for _kind,model in _DEPLOYMENT_MODELS():
|
||||
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
|
||||
row.status='error'; row.message=note[:500]; row.last_payload=None
|
||||
if hasattr(model,'revert_deadline'): row.revert_deadline=None
|
||||
count+=1
|
||||
return count
|
||||
|
||||
async def carp_snapshot_or_502(db, host):
|
||||
try: return await inspect_carp_switch(db,host)
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host=await host_or_404(db,host_id)
|
||||
state=await snapshot_or_502(db,host)
|
||||
state=await snapshot_or_502(db,host,payload.action)
|
||||
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
|
||||
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
|
||||
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
|
||||
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
|
||||
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
|
||||
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||
state=await snapshot_or_502(db,host)
|
||||
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
state=await snapshot_or_502(db,host,op.action)
|
||||
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
|
||||
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
|
||||
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
|
||||
# replaying a reboot/update automatically could interrupt recovery.
|
||||
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
if op.action in ('update','major_update'):
|
||||
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
|
||||
try:
|
||||
point=await snapshot_service.create_rollback_point(host,state.get('version'))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
op.snapshot={**op.snapshot,'rollback_point':point}
|
||||
await db.commit()
|
||||
if op.action=='firmware_rollback':
|
||||
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
|
||||
# Fail closed, like the backup before an update.
|
||||
try:
|
||||
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
rollback=state['rollback']
|
||||
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
|
||||
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
|
||||
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
|
||||
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
|
||||
else:
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
|
||||
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
|
||||
try:
|
||||
result=await action_fn(host)
|
||||
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
op.status='submitted';op.message=done
|
||||
if op.action=='firmware_rollback':
|
||||
marked=await _mark_rolled_back(db,host,rollback['target'])
|
||||
if marked:
|
||||
op.message+=f' {marked} Deployment(s) als unklar markiert.'
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
|
||||
result={}
|
||||
|
||||
@@ -113,6 +113,23 @@ TOOLS = [
|
||||
# The per-rule trace is for the UI; the verdict, deciding rule and reasons are what matters here.
|
||||
"drop": ("trace",),
|
||||
},
|
||||
{
|
||||
"name": "rule_cleanup_candidates",
|
||||
"description": "Cleanup candidates of one firewall: rules that are unused (no hit for N days), shadowed or "
|
||||
"duplicate (an earlier quick rule already decides everything they match), or disabled for a "
|
||||
"long time. Each with its confidence (sure/probable/unknown), the last observed hit and "
|
||||
"whether it could be removed (not for automatic, legacy or OPNexus-managed rules). Reads only.",
|
||||
"inputSchema": {
|
||||
"type": "object", "required": ["host_id"],
|
||||
"properties": {
|
||||
"host_id": {"type": "string", "description": "Firewall id from list_hosts"},
|
||||
"unused_days": {"type": "integer", "minimum": 1, "maximum": 3650, "default": 30,
|
||||
"description": "observation window for unused / disabled rules"},
|
||||
},
|
||||
},
|
||||
"path": "/api/hosts/{host_id}/rule-analysis/cleanup",
|
||||
"query": ("unused_days",),
|
||||
},
|
||||
{
|
||||
"name": "api_get",
|
||||
"description": "Read any other OPNexus API endpoint (GET only), e.g. /api/rules, /api/aliases, /api/logs. "
|
||||
|
||||
@@ -22,6 +22,28 @@ logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["oidc"])
|
||||
CALLBACK_PATH = "/api/auth/oidc/callback"
|
||||
STATE_COOKIE = "opnexus-oidc-state"
|
||||
# HTTPS deployments (1.29): __Host- (Secure, no Domain, Path=/) so that no sibling SUBDOMAIN or plain-HTTP page can
|
||||
# plant a state cookie of its own (login CSRF with the attacker's own authorisation). Known limit: cookies are not
|
||||
# separated by PORT, so script in a firewall GUI shown on a gateway slot port of the same host could still set it
|
||||
# (BACKLOG S8, accepted: needs a compromised firewall GUI and a mapped IdP account; the real fix is one hostname
|
||||
# per slot). The browser only accepts the prefix with Path=/, so the cookie travels with every request for its
|
||||
# 10 minutes; plain-HTTP labs keep the narrow path.
|
||||
SECURE_STATE_COOKIE = "__Host-opnexus-oidc-state"
|
||||
STATE_COOKIE_PATH = "/api/auth/oidc"
|
||||
|
||||
|
||||
def state_cookie_name() -> str:
|
||||
return SECURE_STATE_COOKIE if settings.cookie_secure else STATE_COOKIE
|
||||
|
||||
|
||||
def _state_cookie_path() -> str:
|
||||
return "/" if settings.cookie_secure else STATE_COOKIE_PATH
|
||||
|
||||
|
||||
def _clear_state_cookie(response) -> None:
|
||||
response.delete_cookie(state_cookie_name(), path=_state_cookie_path(), secure=settings.cookie_secure, httponly=True, samesite="lax")
|
||||
if settings.cookie_secure: # a plain-name cookie of an earlier version
|
||||
response.delete_cookie(STATE_COOKIE, path=STATE_COOKIE_PATH, secure=True, httponly=True, samesite="lax")
|
||||
# SSO sessions are short: the provider's group membership is only re-checked at login, so a removed person
|
||||
# must not keep a 30-day session.
|
||||
SSO_SESSION_SECONDS = 12 * 3600
|
||||
@@ -29,7 +51,7 @@ SSO_SESSION_SECONDS = 12 * 3600
|
||||
|
||||
def _login_error(code: str) -> RedirectResponse:
|
||||
response = RedirectResponse(f"/login?sso_error={code}", status_code=302)
|
||||
response.delete_cookie(STATE_COOKIE, path="/api/auth/oidc")
|
||||
_clear_state_cookie(response)
|
||||
return response
|
||||
|
||||
|
||||
@@ -63,7 +85,8 @@ async def start(request: Request, origin: str, db: AsyncSession = Depends(get_db
|
||||
response = RedirectResponse(url, status_code=302)
|
||||
# Binds the attempt to THIS browser: a callback URL handed to somebody else (login CSRF) or a captured
|
||||
# code+state pair is useless without the cookie.
|
||||
response.set_cookie(STATE_COOKIE, state, max_age=600, httponly=True, samesite="lax", secure=settings.cookie_secure, path="/api/auth/oidc")
|
||||
response.set_cookie(state_cookie_name(), state, max_age=600, httponly=True, samesite="lax", secure=settings.cookie_secure,
|
||||
path=_state_cookie_path())
|
||||
return response
|
||||
|
||||
|
||||
@@ -78,7 +101,9 @@ async def callback(request: Request, code: str = "", state: str = "", error: str
|
||||
try:
|
||||
if error or not code or not state:
|
||||
raise oidc_service.OidcError("provider", "Anbieter meldet " + (_printable(error) if error else "keinen Code"))
|
||||
if not secrets.compare_digest(request.cookies.get(STATE_COOKIE, ""), state):
|
||||
# Only the cookie name of this deployment counts (on HTTPS never the plain one); no cookie never matches.
|
||||
bound = request.cookies.get(state_cookie_name(), "")
|
||||
if not bound or not secrets.compare_digest(bound.encode(), state.encode()):
|
||||
raise oidc_service.OidcError("state", "Der Anmeldeversuch gehört nicht zu diesem Browser")
|
||||
user = await oidc_service.complete(db, cfg, code, state)
|
||||
except oidc_service.OidcError as exc:
|
||||
@@ -92,7 +117,7 @@ async def callback(request: Request, code: str = "", state: str = "", error: str
|
||||
)
|
||||
response = RedirectResponse("/dashboard", status_code=302)
|
||||
_set_session_cookie(response, token, max_age=SSO_SESSION_SECONDS)
|
||||
response.delete_cookie(STATE_COOKIE, path="/api/auth/oidc")
|
||||
_clear_state_cookie(response)
|
||||
logger.info("SSO-Anmeldung von '%s' (%s) aus %s", user.username, user.role, ip)
|
||||
return response
|
||||
|
||||
|
||||
@@ -50,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
@@ -231,6 +232,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_one_to_one_rule(deployment.host, deployment.opnsense_one_to_one_uuid)
|
||||
|
||||
@@ -195,6 +195,8 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_instance_uuid:
|
||||
|
||||
@@ -51,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
@@ -238,6 +239,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_outbound_nat_rule(deployment.host, deployment.opnsense_outbound_nat_uuid)
|
||||
|
||||
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
hierarchy_group_id=rule.hierarchy_group_id,
|
||||
rulebase=rule.rulebase,
|
||||
sequence=rule.sequence,
|
||||
@@ -330,6 +331,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
try:
|
||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_policy_hierarchy_uuid)
|
||||
await opnsense_client.apply(deployment.host)
|
||||
|
||||
@@ -50,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
ipprotocol=rule.ipprotocol,
|
||||
@@ -233,6 +234,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_port_forward_rule(deployment.host, deployment.opnsense_port_forward_uuid)
|
||||
|
||||
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -20,7 +20,7 @@ from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||
@@ -36,6 +36,19 @@ class WaveIn(BaseModel):
|
||||
require_approval: bool = False
|
||||
|
||||
|
||||
class ProbeIn(BaseModel):
|
||||
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
|
||||
type: Literal["ping", "tcp"]
|
||||
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
|
||||
port: int | None = Field(default=None, ge=1, le=65535)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _port_only_for_tcp(self):
|
||||
if (self.type == "tcp") != (self.port is not None):
|
||||
raise ValueError("port_required_for_tcp_only")
|
||||
return self
|
||||
|
||||
|
||||
class PlanIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
description: str = Field(default="", max_length=2000)
|
||||
@@ -43,6 +56,7 @@ class PlanIn(BaseModel):
|
||||
window_id: uuid.UUID | None = None
|
||||
auto_start: bool = False
|
||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
|
||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||
|
||||
@@ -64,6 +78,7 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||
return {
|
||||
**rollout_service.definition(plan),
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"description": plan.description,
|
||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||
@@ -81,6 +96,15 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
||||
}
|
||||
|
||||
|
||||
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
for probe in probes:
|
||||
value = {"type": probe.type, "target": probe.target, "port": probe.port}
|
||||
if value not in out:
|
||||
out.append(value)
|
||||
return out
|
||||
|
||||
|
||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
@@ -99,6 +123,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||
plan.gate_probes = _unique_probes(payload.gate_probes)
|
||||
plan.compliance_mode = payload.compliance_mode
|
||||
plan.waves = [
|
||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||
@@ -106,7 +131,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
for i, w in enumerate(payload.waves)
|
||||
]
|
||||
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
|
||||
plan.activated_at = plan.activated_by = plan.activated_fingerprint = None
|
||||
plan.activated_at = plan.activated_by = plan.activated_by_id = plan.activated_fingerprint = None
|
||||
plan.auto_start_error = None
|
||||
|
||||
|
||||
@@ -180,6 +205,7 @@ async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request,
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
|
||||
_resolved, current = await _checked_token(db, plan, payload.preview_token)
|
||||
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
|
||||
plan.activated_by_id = request.state.user.id
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
|
||||
@@ -191,7 +217,7 @@ async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request,
|
||||
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
plan.activated_by = plan.activated_at = plan.activated_fingerprint = None
|
||||
plan.activated_by = plan.activated_by_id = plan.activated_at = plan.activated_fingerprint = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
@@ -306,7 +332,8 @@ async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db:
|
||||
if (other := await rollout_engine.active_run(db)) is not None:
|
||||
# One run at a time: two runs could otherwise touch the same cluster at once.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
|
||||
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username, trigger="manual")
|
||||
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username,
|
||||
started_by_id=request.state.user.id, trigger="manual")
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
|
||||
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
@@ -363,7 +390,7 @@ async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Dep
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
|
||||
previous = run.started_by
|
||||
# Whoever resumes takes over the responsibility (and the permission check) from here on.
|
||||
run.started_by = request.state.user.username
|
||||
run.started_by, run.started_by_id = request.state.user.username, request.state.user.id
|
||||
run.status = "resuming"
|
||||
run.pause_reason = run.pause_detail = None
|
||||
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
|
||||
@@ -378,7 +405,7 @@ async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = D
|
||||
raise _not_now(run)
|
||||
wave = rollout_engine.wave_row(run, run.current_wave)
|
||||
wave.approved_by, wave.status = request.state.user.username, "preflight"
|
||||
run.started_by = request.state.user.username
|
||||
run.started_by, run.started_by_id = request.state.user.username, request.state.user.id
|
||||
run.status = "preflight"
|
||||
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
|
||||
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
|
||||
|
||||
@@ -17,6 +17,8 @@ from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rule_history import LiveRuleState
|
||||
from app.services import rule_history_service
|
||||
from app.services.rule_analysis import service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -89,3 +91,38 @@ async def fleet_match(
|
||||
return await _run(host, q, refresh)
|
||||
|
||||
return list(await asyncio.gather(*(one(h) for h in hosts)))
|
||||
|
||||
|
||||
@router.get("/hosts/{host_id}/rule-analysis/cleanup")
|
||||
async def host_cleanup(
|
||||
host_id: uuid.UUID, unused_days: int = Query(30, ge=1, le=3650, description="observation window for 'unused' / 'disabled for a long time'"),
|
||||
refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Cleanup candidates (unused, shadowed, duplicate, disabled for a long time) with confidence and whether
|
||||
each rule could be offered for removal. Read-only; removal is not part of this endpoint."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
return await service.cleanup_candidates(db, host, unused_days=unused_days, refresh=refresh)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001 - an unreachable firewall is a 502, not a crash
|
||||
logger.info("cleanup candidates failed for %s: %s", host.name, exc)
|
||||
raise api_error(502, ErrorCode.ANALYSE_FIREWALL_NICHT_LESBAR) from exc
|
||||
|
||||
|
||||
@router.get("/hosts/{host_id}/live-rules/{rule_uuid}/history")
|
||||
async def live_rule_history(host_id: uuid.UUID, rule_uuid: str, limit: int = Query(100, ge=1, le=500),
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""What happened to one rule on one firewall: observed changes (also hand-made ones), saved revisions of
|
||||
rules OPNexus manages and the audit entries, newest first. Reads the database only."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
baseline = await db.get(LiveRuleState, (host_id, rule_history_service.BASELINE_KEY))
|
||||
return {
|
||||
"host_id": str(host.id), "rule_uuid": rule_uuid.lower()[:64],
|
||||
"watching_since": baseline.first_seen_at if baseline else None,
|
||||
"events": await rule_history_service.rule_timeline(db, host, rule_uuid, limit),
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): switch hand-made rules off, delete them later, restore them (DESIGN-1.31 §4).
|
||||
|
||||
POST /hosts/{id}/rule-cleanup/preview operator, admin prepare (writes nothing to a firewall)
|
||||
POST /hosts/{id}/rule-cleanup/{op}/cancel operator, admin the preview's own actor
|
||||
POST /hosts/{id}/rule-cleanup/{op}/deploy ADMIN, token scope admin
|
||||
POST /hosts/{id}/rule-cleanup/{op}/confirm operator, admin BROWSER SESSION ONLY, never a token
|
||||
POST /hosts/{id}/rule-cleanup/{op}/revert admin; operator only with operators_may_deploy
|
||||
POST /hosts/{id}/rule-cleanup/{op}/retry ADMIN, token scope admin
|
||||
POST /hosts/{id}/rule-cleanup/items/{item}/restore ADMIN, token scope admin
|
||||
GET /hosts/{id}/rule-cleanup, /{op} everybody signed in (read tokens too)
|
||||
|
||||
Who may do what is enforced twice: AuthMiddleware (services/api_token_service.py: _ADMIN_ONLY_CLEANUP,
|
||||
_SESSION_ONLY, _ROLLOUT_POST) and here, so a gap in one layer does not open the route. The audit trail is the
|
||||
service's (one ``rule_cleanup`` entry per step and rule) plus the middleware's request entry (kind ``host``,
|
||||
action = last path segment; ``/preview`` is not logged by it, the service writes its own entry).
|
||||
The services raise ``api_error``; nothing is translated here. MCP stays read-only (no tool, ``api_get`` is GET).
|
||||
"""
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser
|
||||
from app.routers import governance
|
||||
from app.routers.auth import resolve_user
|
||||
from app.schemas.rule_cleanup import CleanupOperationRead, CleanupPreviewRead, ConfirmName, PreviewRequest
|
||||
from app.services import rule_cleanup_service as service
|
||||
from app.services.rule_cleanup_service import ACK_TEXT_VERSION
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/rule-cleanup", tags=["rule-cleanup"])
|
||||
|
||||
|
||||
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
return host
|
||||
|
||||
|
||||
def _token(request: Request):
|
||||
return getattr(request.state, "api_token", None)
|
||||
|
||||
|
||||
def _writer(user: AdminUser, request: Request) -> None:
|
||||
"""Any write: not for a read-only account, not with a read-scope token."""
|
||||
if user.role == "read_only":
|
||||
raise api_error(403, ErrorCode.READ_ONLY_ACCESS)
|
||||
token = _token(request)
|
||||
if token is not None and token.scope == "read":
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
def _admin_only(user: AdminUser, request: Request) -> None:
|
||||
"""Deploy, retry, restore: an admin account, and with a token only one of scope admin."""
|
||||
_writer(user, request)
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
token = _token(request)
|
||||
if token is not None and token.scope != "admin":
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
async def _deployer(user: AdminUser, request: Request, db: AsyncSession) -> None:
|
||||
"""Revert: an admin, or an operator while the governance switch operators_may_deploy is on."""
|
||||
_writer(user, request)
|
||||
if user.role == "admin":
|
||||
return
|
||||
if user.role == "operator" and (await governance.get_settings(db)).operators_may_deploy:
|
||||
return
|
||||
raise api_error(403, ErrorCode.DEPLOY_NEEDS_ADMIN)
|
||||
|
||||
|
||||
def _session_only(user: AdminUser, request: Request) -> None:
|
||||
"""Confirm: "I still have access" is a human decision in a browser session, never a token's."""
|
||||
_writer(user, request)
|
||||
token = _token(request)
|
||||
if token is not None:
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
def _full_view(request: Request) -> bool:
|
||||
"""Admins (session or admin-scope token) see the management sources/ports and firewall addresses in a plan."""
|
||||
from app.routers.audit import full_view
|
||||
return full_view(request)
|
||||
|
||||
|
||||
async def _names(db: AsyncSession, operations: list[dict], request: Request) -> list[dict]:
|
||||
"""The firewall names the UI shows next to the ids; for non-admins without the security-relevant plan parts."""
|
||||
if not _full_view(request):
|
||||
operations = [service.redact(op) for op in operations]
|
||||
ids = {uuid.UUID(op["host_id"]) for op in operations} | {uuid.UUID(i["host_id"]) for op in operations for i in op["items"]}
|
||||
names = {h.id: h.name for h in (await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)))).all()} if ids else {}
|
||||
for op in operations:
|
||||
op["host_name"] = names.get(uuid.UUID(op["host_id"]))
|
||||
for item in op["items"]:
|
||||
item["host_name"] = names.get(uuid.UUID(item["host_id"]))
|
||||
return operations
|
||||
|
||||
|
||||
async def _one(db: AsyncSession, operation: dict, request: Request) -> dict:
|
||||
return (await _names(db, [operation], request))[0]
|
||||
|
||||
|
||||
@router.get("", response_model=list[CleanupOperationRead])
|
||||
async def list_operations(host_id: uuid.UUID, request: Request, limit: int = Query(50, ge=1, le=100),
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""The newest operations of this host (and its cluster peers) with their items; unexpired previews included."""
|
||||
host = await _host(db, host_id)
|
||||
return await _names(db, await service.list_operations(db, host, limit), request)
|
||||
|
||||
|
||||
@router.post("/preview", response_model=CleanupPreviewRead)
|
||||
async def preview(host_id: uuid.UUID, payload: PreviewRequest, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Recompute the plan from the live firewall(s) and store a 5-minute preview bound to the caller."""
|
||||
user = await resolve_user(request, db)
|
||||
_writer(user, request)
|
||||
host = await _host(db, host_id)
|
||||
kind, uuids, acknowledge = payload.resolved()
|
||||
if kind is None:
|
||||
raise api_error(422, ErrorCode.BEREINIGUNG_AUSWAHL_UNGUELTIG, field="kind")
|
||||
result = await service.preview(
|
||||
db, host, actor=user.username, actor_id=user.id, kind=kind, uuids=uuids, acknowledge=acknowledge,
|
||||
**({"unused_days": payload.unused_days} if payload.unused_days else {}),
|
||||
ack_text_version=payload.ack_text_version or ACK_TEXT_VERSION)
|
||||
result["operation"] = await _one(db, await service.get_operation(db, host, uuid.UUID(result["preview_id"])), request)
|
||||
return result if _full_view(request) else service.redact(result)
|
||||
|
||||
|
||||
@router.get("/{op_id}", response_model=CleanupOperationRead)
|
||||
async def get_operation(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.get_operation(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/cancel", response_model=CleanupOperationRead)
|
||||
async def cancel(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Drop an unused preview (only its own actor can)."""
|
||||
user = await resolve_user(request, db)
|
||||
_writer(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.cancel(db, host, op_id, actor=user.username), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/deploy", response_model=CleanupOperationRead)
|
||||
async def deploy(host_id: uuid.UUID, op_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""Execute a preview. Admin only; the live state must still give the fingerprint of the preview."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.deploy(db, host, op_id, actor=user.username, confirm_name=payload.confirm_name), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/confirm", response_model=CleanupOperationRead)
|
||||
async def confirm(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
""""I still have access": re-reads every node first. Browser session only, never with an API token."""
|
||||
user = await resolve_user(request, db)
|
||||
_session_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.confirm(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/revert", response_model=CleanupOperationRead)
|
||||
async def revert(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Take the change back now (the same code the sweeper runs when the window passes)."""
|
||||
user = await resolve_user(request, db)
|
||||
await _deployer(user, request, db)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.revert_now(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/retry", response_model=CleanupOperationRead)
|
||||
async def retry(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Revert again after revert_failed, or read the firewall again for a delete/restore with an unclear item."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.retry(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/items/{item_id}/restore", response_model=CleanupOperationRead)
|
||||
async def restore(host_id: uuid.UUID, item_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""Bring a deleted rule back, switched OFF, at its old position (a new UUID). The result is the restore run;
|
||||
``warnings`` can hold BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.restore_item(
|
||||
db, host, item_id, actor=user.username, actor_id=user.id, confirm_name=payload.confirm_name), request)
|
||||
@@ -1,3 +1,4 @@
|
||||
import json
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, Body
|
||||
@@ -18,6 +19,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.rule_deploy_service import confirm_rule, deploy_rule, undeploy_rule, revert_deployment
|
||||
from app.services import rule_adopt_service
|
||||
from app.services.adopt_common import LINK_ERRORS, link_api_error
|
||||
from app.schemas.rule import AdoptRequest, UnmanagedRule
|
||||
|
||||
router = APIRouter(prefix="/rules", tags=["rules"])
|
||||
@@ -50,6 +52,7 @@ def _to_read(
|
||||
return RuleRead(
|
||||
workflow_state=state,
|
||||
workflow_counts=counts, id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
direction=rule.direction,
|
||||
@@ -106,12 +109,19 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
|
||||
host = await db.get(OpnsenseHost, payload.host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
# Taking over a hand-made rule that a rule cleanup (1.31) is switching off/deleting would make it "managed"
|
||||
# under the cleanup's feet: not while one is open on this host or its cluster.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
try:
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid)
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
|
||||
except rule_adopt_service.RuleNotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.REGEL_NICHT_GEFUNDEN) from exc
|
||||
except rule_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
||||
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons),
|
||||
reason_details=json.dumps(exc.details, ensure_ascii=False)) from exc
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
@@ -271,6 +281,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
try:
|
||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_rule_uuid)
|
||||
await opnsense_client.apply(deployment.host)
|
||||
|
||||
@@ -207,6 +207,8 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_peer_uuid:
|
||||
|
||||
@@ -5,11 +5,31 @@ from pydantic import BaseModel, Field
|
||||
|
||||
class AdoptObjectRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link to this existing central object instead of creating a new one (services/adopt_link_service.py).
|
||||
# Declared before `uuid`: that field's name shadows the module inside the class body.
|
||||
link_to: uuid.UUID | None = None
|
||||
uuid: str = Field(min_length=1, max_length=64) # the object's UUID on the firewall
|
||||
|
||||
|
||||
class AdoptAllRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link entries that have exactly one exact central twin instead of creating a duplicate.
|
||||
link_existing: bool = False
|
||||
|
||||
|
||||
class LinkDiff(BaseModel):
|
||||
field: str # model field ("content", "destination_port", ...) or "variables" (a {{VARIABLE}} has no value on this host)
|
||||
central: str
|
||||
live: str
|
||||
|
||||
|
||||
class LinkTarget(BaseModel):
|
||||
"""An existing central object that a live object could be linked to (exact match) or is similar to."""
|
||||
|
||||
id: uuid.UUID
|
||||
name: str
|
||||
targets: list[str] # names of the hosts it is deployed to
|
||||
diff: list[LinkDiff] = Field(default_factory=list) # empty for an exact match
|
||||
|
||||
|
||||
class UnmanagedObject(BaseModel):
|
||||
@@ -28,12 +48,20 @@ class UnmanagedObject(BaseModel):
|
||||
kept: list[str] = Field(default_factory=list)
|
||||
# Facts for display, depend on the object type.
|
||||
info: dict[str, str] = Field(default_factory=dict)
|
||||
# Central objects whose content, resolved for this host, equals the live object exactly (first = preferred).
|
||||
matches: list[LinkTarget] = Field(default_factory=list)
|
||||
# Objects with the same name/description but different content: a hint with the differing fields, never linked.
|
||||
similar: list[LinkTarget] = Field(default_factory=list)
|
||||
linkable: bool = False # can be linked to `matches[0]` (or another match) on its own
|
||||
# What "adopt all" would do: "link" | "new" | "skip" (None only on the adopt-all result).
|
||||
bulk_action: str | None = None
|
||||
|
||||
|
||||
class AdoptedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
id: uuid.UUID
|
||||
linked: bool = False # linked to an existing central object instead of newly created
|
||||
|
||||
|
||||
class SkippedObject(BaseModel):
|
||||
|
||||
@@ -14,6 +14,9 @@ class DashboardHost(BaseModel):
|
||||
cluster_name: str | None
|
||||
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
|
||||
gateways: list[GatewayHealth] | None = None
|
||||
# Readings the newest fresh sample could not take because the API key lacks the right (gateways, services, ...);
|
||||
# None = no fresh sample (or one from before this existed), [] = nothing missing.
|
||||
api_rights_missing: list[str] | None = None
|
||||
|
||||
|
||||
class PendingAction(BaseModel):
|
||||
|
||||
@@ -17,3 +17,17 @@ class GatewayHealth(BaseModel):
|
||||
delay: float | None = None # milliseconds
|
||||
stddev: float | None = None # milliseconds
|
||||
monitor: str | None = None # the monitored address
|
||||
alert_enabled: bool = True # False: the gateway alert is switched off for this gateway (Settings, 1.29)
|
||||
|
||||
|
||||
class GatewayAlertUpdate(BaseModel):
|
||||
enabled: bool
|
||||
|
||||
|
||||
class GatewayAlertState(BaseModel):
|
||||
name: str
|
||||
alert_enabled: bool
|
||||
|
||||
|
||||
class ApiRightsState(BaseModel):
|
||||
missing: list[str] | None = None # names of the refused readings (gateways, services, swap, cpu); None = unknown
|
||||
@@ -16,7 +16,7 @@ ProblemType = Literal[
|
||||
"backup_stale",
|
||||
"backup_offsite_failed",
|
||||
"compliance_finding",
|
||||
"gateway_down", "resource_threshold", "service_down",
|
||||
"gateway_down", "resource_threshold", "service_down", "api_rights_missing",
|
||||
"rollout_paused", "rollout_start_failed",
|
||||
]
|
||||
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
||||
@@ -54,6 +54,8 @@ class NotificationSettingsRead(BaseModel):
|
||||
resource_threshold_enabled: bool = False
|
||||
service_down_enabled: bool = False
|
||||
rollout_problems_enabled: bool = True
|
||||
api_rights_enabled: bool = True
|
||||
gateway_alert_muted: list[str] = Field(default_factory=list)
|
||||
gateway_loss_threshold_pct: int = 20
|
||||
gateway_delay_threshold_ms: int = 500
|
||||
disk_threshold_pct: int = 90
|
||||
@@ -111,6 +113,7 @@ class NotificationSettingsUpdate(BaseModel):
|
||||
resource_threshold_enabled: bool | None = None
|
||||
service_down_enabled: bool | None = None
|
||||
rollout_problems_enabled: bool | None = None
|
||||
api_rights_enabled: bool | None = None
|
||||
# 0 switches that single threshold off.
|
||||
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
||||
|
||||
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
hierarchy_group_id: uuid.UUID
|
||||
rulebase: Literal["pre", "post"]
|
||||
sequence: int
|
||||
|
||||
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
ipprotocol: str
|
||||
|
||||
@@ -4,6 +4,8 @@ from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from app.schemas.adopt import LinkTarget
|
||||
|
||||
|
||||
class RuleCreate(BaseModel):
|
||||
description: str = Field(min_length=1, max_length=255)
|
||||
@@ -46,6 +48,7 @@ class RuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
direction: str
|
||||
@@ -78,6 +81,8 @@ class DeployApproval(BaseModel):
|
||||
|
||||
class AdoptRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link to this existing central rule instead of creating a new one (services/adopt_link_service.py).
|
||||
link_to: uuid.UUID | None = None
|
||||
rule_uuid: str = Field(min_length=1, max_length=64)
|
||||
|
||||
|
||||
@@ -98,3 +103,7 @@ class UnmanagedRule(BaseModel):
|
||||
adoptable: bool
|
||||
# Codes such as "negated_source" or "not_quick"; the UI turns them into text.
|
||||
reasons: list[str]
|
||||
# Central rules equal to this one (resolved for the host), same-description rules that differ, link possible.
|
||||
matches: list[LinkTarget] = Field(default_factory=list)
|
||||
similar: list[LinkTarget] = Field(default_factory=list)
|
||||
linkable: bool = False
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Request and response shapes of the rule-cleanup API (routers/rule_cleanup.py, DESIGN-1.31 §4).
|
||||
|
||||
Responses mirror services/rule_cleanup_service.read_operation. Texts are not part of the API: ``message`` holds an
|
||||
ErrorCode name (``errors.<CODE>`` in the UI, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR) or the firewall's
|
||||
own wording; ``warnings`` is a list of ErrorCode names.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
CleanupKind = Literal["disable", "enable", "delete"]
|
||||
# Longest list the API accepts at all; the effective limit is the setting cleanup_max_rules (ceiling 25).
|
||||
MAX_SELECTION = 100
|
||||
|
||||
|
||||
class PreviewItem(BaseModel):
|
||||
"""One rule of the selection. ``kind`` (alias ``mode``) must be the same for all items of one preview."""
|
||||
uuid: str = Field(min_length=1, max_length=64)
|
||||
kind: CleanupKind | None = None
|
||||
mode: CleanupKind | None = None
|
||||
# The per-rule acknowledgement for a "probable" rule (the UI's checkbox).
|
||||
ack: bool = False
|
||||
|
||||
|
||||
class PreviewRequest(BaseModel):
|
||||
"""``{kind, uuids, acknowledge}`` or ``{items:[{uuid, kind|mode, ack}]}`` (or both: they are merged)."""
|
||||
kind: CleanupKind | None = None
|
||||
uuids: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
items: list[PreviewItem] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
acknowledge: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
unused_days: int | None = Field(default=None, ge=1, le=3650)
|
||||
ack_text_version: str | None = Field(default=None, max_length=16)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _single_kind(self):
|
||||
kinds = {k for k in [self.kind, *[i.kind or i.mode for i in self.items]] if k}
|
||||
if len(kinds) > 1:
|
||||
raise ValueError("kind must be the same for every item")
|
||||
return self
|
||||
|
||||
def resolved(self) -> tuple[str | None, list[str], list[str]]:
|
||||
kind = self.kind or next((i.kind or i.mode for i in self.items if i.kind or i.mode), None)
|
||||
uuids = [*self.uuids, *[i.uuid for i in self.items]]
|
||||
acknowledge = [*self.acknowledge, *[i.uuid for i in self.items if i.ack]]
|
||||
return kind, list(dict.fromkeys(u.strip() for u in uuids)), list(dict.fromkeys(a.strip() for a in acknowledge))
|
||||
|
||||
|
||||
class ConfirmName(BaseModel):
|
||||
"""Deploy and restore: the exact host name, typed by a human."""
|
||||
confirm_name: str = Field(max_length=255)
|
||||
|
||||
|
||||
class CleanupItemRead(BaseModel):
|
||||
id: str
|
||||
host_id: str
|
||||
host_name: str | None = None
|
||||
rule_uuid: str
|
||||
seq: int
|
||||
carp_role: str | None = None
|
||||
# queued, disabling, disabled, confirmed, reverting, reverted, deleting, deleted, restoring, restored, failed,
|
||||
# unknown, revert_failed, skipped (never attempted: the run stopped earlier)
|
||||
state: str
|
||||
description: str | None = None
|
||||
categories: Any = None
|
||||
confidence: str | None = None
|
||||
cover_uuid: str | None = None
|
||||
prio_group: str | None = None
|
||||
anchor_next_uuid: str | None = None
|
||||
anchor_prev_uuid: str | None = None
|
||||
source_item_id: str | None = None
|
||||
restored_uuid: str | None = None
|
||||
changed_at: str | None = None
|
||||
message: str | None = None
|
||||
has_snapshot: bool = False
|
||||
# A deleted rule with a stored snapshot that has not been restored yet (POST items/{id}/restore, admin).
|
||||
restorable: bool = False
|
||||
|
||||
|
||||
class CleanupOperationRead(BaseModel):
|
||||
id: str
|
||||
host_id: str
|
||||
host_name: str | None = None
|
||||
cluster_name: str | None = None
|
||||
kind: str # disable | enable | delete | restore
|
||||
# preview, running, awaiting_confirm, confirmed, partial, apply_failed, reverting, reverted, revert_failed, done,
|
||||
# failed, cancelled, expired
|
||||
status: str
|
||||
actor: str
|
||||
created_at: str | None = None
|
||||
expires_at: str | None = None
|
||||
started_at: str | None = None
|
||||
heartbeat_at: str | None = None
|
||||
revert_deadline: str | None = None
|
||||
confirmed_at: str | None = None
|
||||
finished_at: str | None = None
|
||||
message: str | None = None
|
||||
options: dict[str, Any] = Field(default_factory=dict)
|
||||
# Only while status == "preview".
|
||||
plan: dict[str, Any] | None = None
|
||||
items: list[CleanupItemRead] = Field(default_factory=list)
|
||||
can_confirm: bool = False
|
||||
can_revert: bool = False
|
||||
can_retry: bool = False
|
||||
# ErrorCode names, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR (restore only).
|
||||
warnings: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class CleanupPreviewRead(BaseModel):
|
||||
preview_id: str
|
||||
expires_at: str | None = None
|
||||
kind: str
|
||||
plan: dict[str, Any]
|
||||
max_rules: int
|
||||
window_seconds: int
|
||||
operation: CleanupOperationRead
|
||||
@@ -7,9 +7,19 @@ how the proxy setup works for IP *and* name access) nor an explicitly allowed
|
||||
origin. Requests without either header (curl, scripts) are not browsers, so
|
||||
they cannot be riding on a victim's cookie and pass through to normal auth.
|
||||
|
||||
Sec-Fetch-Site (1.27): a page on another port of the same host (e.g. a
|
||||
firewall GUI behind the GUI gateway) is *same-site* to OPNexus, so the
|
||||
browser attaches the session cookie to its requests and SameSite=Lax does
|
||||
not help. Cookie-carrying /api requests that the browser itself marks as
|
||||
same-site or cross-site are therefore refused — unless the Origin is an
|
||||
explicitly configured frontend origin (development setup with the frontend
|
||||
on its own port). Clients that send no Sec-Fetch-Site header (curl,
|
||||
scripts, API tokens) are not browsers and stay unaffected.
|
||||
|
||||
SecurityHeadersMiddleware: API responses must never be cached by browsers or
|
||||
proxies and must not be sniffed or framed.
|
||||
"""
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
@@ -18,6 +28,7 @@ from starlette.responses import JSONResponse
|
||||
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode
|
||||
from app.routers.auth import has_session_cookie
|
||||
|
||||
_SAFE_METHODS = {"GET", "HEAD", "OPTIONS"}
|
||||
|
||||
@@ -35,8 +46,53 @@ def _origin_allowed(origin: str, host_header: str) -> bool:
|
||||
return f"{parsed.scheme}://{parsed.netloc}".lower() in {o.lower() for o in settings.allowed_origins}
|
||||
|
||||
|
||||
# The SSO provider sends the browser back here by a cross-site navigation;
|
||||
# the endpoint authenticates by its own state cookie and creates the session.
|
||||
# The exemption is only for exactly that: a top-level GET navigation (a script, image or fetch from another site
|
||||
# is still blocked, whatever it carries) — and the callback fails without the matching state cookie anyway.
|
||||
_SEC_FETCH_EXEMPT = {"/api/auth/oidc/callback"}
|
||||
|
||||
|
||||
def _is_top_level_navigation(request: Request) -> bool:
|
||||
if request.method != "GET":
|
||||
return False
|
||||
mode = request.headers.get("sec-fetch-mode", "").strip().lower()
|
||||
dest = request.headers.get("sec-fetch-dest", "").strip().lower()
|
||||
return mode in ("", "navigate") and dest in ("", "document")
|
||||
|
||||
|
||||
def _explicit_frontend_origin(origin: str) -> bool:
|
||||
origin = origin.rstrip("/").lower()
|
||||
if not origin or origin == "null":
|
||||
return False
|
||||
if settings.allowed_origins:
|
||||
return origin in {o.lower() for o in settings.allowed_origins}
|
||||
return re.fullmatch(settings.cors_origin_regex, origin) is not None
|
||||
|
||||
|
||||
def sec_fetch_blocked(request: Request) -> bool:
|
||||
path = request.url.path
|
||||
if not (path == "/api" or path.startswith("/api/")):
|
||||
return False
|
||||
if path in _SEC_FETCH_EXEMPT and _is_top_level_navigation(request):
|
||||
return False
|
||||
if not has_session_cookie(request):
|
||||
return False
|
||||
site = request.headers.get("sec-fetch-site", "").strip().lower()
|
||||
if site == "cross-site":
|
||||
return True
|
||||
if site == "same-site":
|
||||
return not _explicit_frontend_origin(request.headers.get("origin", ""))
|
||||
return False # same-origin, none (typed URL/bookmark) or no header (not a browser)
|
||||
|
||||
|
||||
class OriginCheckMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if sec_fetch_blocked(request):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": {"code": ErrorCode.CROSS_SITE_REQUEST_BLOCKED.value, "params": {}}},
|
||||
)
|
||||
if request.method not in _SAFE_METHODS:
|
||||
source = request.headers.get("origin") or request.headers.get("referer")
|
||||
if source and source != "null" and not _origin_allowed(source, request.headers.get("host", "")):
|
||||
|
||||
@@ -25,3 +25,39 @@ def on(value) -> bool:
|
||||
|
||||
def clean(value) -> str:
|
||||
return "" if value is None else str(value).strip()
|
||||
|
||||
|
||||
class LinkTargetNotFound(Exception):
|
||||
"""The central object chosen to link to was deleted in the meantime."""
|
||||
|
||||
|
||||
class LinkTargetChanged(Exception):
|
||||
"""The central object no longer equals what is live on the host. `diff` lists the differing fields."""
|
||||
|
||||
def __init__(self, diff: list[dict]):
|
||||
super().__init__(", ".join(d["field"] for d in diff))
|
||||
self.diff = diff
|
||||
|
||||
|
||||
class HostAlreadyHasObject(Exception):
|
||||
"""The central object is already deployed to / managed on this host under another UUID."""
|
||||
|
||||
|
||||
class LinkTargetBusy(Exception):
|
||||
"""The central object has an unconfirmed deployment or a failed rollback; linking waits until it is settled."""
|
||||
|
||||
|
||||
LINK_ERRORS = (LinkTargetNotFound, LinkTargetChanged, HostAlreadyHasObject, LinkTargetBusy)
|
||||
|
||||
|
||||
def link_api_error(exc: Exception):
|
||||
"""The HTTP error for a failed link (shared by the rule and the object endpoints)."""
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
if isinstance(exc, LinkTargetNotFound):
|
||||
return api_error(404, ErrorCode.ADOPT_LINK_TARGET_NOT_FOUND)
|
||||
if isinstance(exc, LinkTargetChanged):
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_TARGET_CHANGED, fields=", ".join(d["field"] for d in exc.diff))
|
||||
if isinstance(exc, HostAlreadyHasObject):
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_HOST_HAS_OBJECT)
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_BUSY)
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Links a hand-made object that was found on a firewall to a central object that already exists, instead
|
||||
of creating a second central object for it ("Mit vorhandenem zentralem Objekt verknüpfen").
|
||||
|
||||
Typical case: the same rule/alias/NAT entry was made by hand on several firewalls (or on both nodes of an HA
|
||||
pair). The first adoption creates the central object; every further firewall is LINKED to it: the host
|
||||
becomes another target of that object and gets a deployment row holding the live UUID (remote_uuid) and
|
||||
the payload OPNexus would send. Like adopting, linking writes NOTHING to the firewall. The live state is
|
||||
read fresh from the host at link time (the browser only names the central object), so the preview right
|
||||
afterwards says "unchanged" and a later rollout updates the one object on every host without a duplicate.
|
||||
|
||||
A central object is an EXACT match when every field OPNexus manages is equal after the object's
|
||||
{{VARIABLE}} placeholders are replaced with this host's values (variables_service.resolve). An object with
|
||||
the same identity (alias name / rule description) but different content is only shown as a hint with the
|
||||
differing fields and is NEVER linked automatically.
|
||||
|
||||
Callers hold policy_lock (same serialisation as adopting), which also makes a repeated link idempotent.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Callable
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.services import variables_service
|
||||
from app.services.adopt_common import HostAlreadyHasObject, LinkTargetBusy, LinkTargetChanged, LinkTargetNotFound
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import effective_hosts
|
||||
|
||||
LINK_MESSAGE = "Mit vorhandenem Objekt verknüpft"
|
||||
BUSY_STATUSES = ("pending", "revert_failed") # the central object waits for a confirmation / failed to roll back
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LinkSpec:
|
||||
model: type
|
||||
deployment: type
|
||||
fk: str # deployment column pointing at the central object
|
||||
uuid_attr: str # deployment column holding the OPNsense UUID (remote_uuid)
|
||||
identity: str # attribute naming the object: "name" (alias) or "description"
|
||||
payload: Callable[[Any, OpnsenseHost], dict]
|
||||
revision: Callable
|
||||
|
||||
|
||||
@dataclass
|
||||
class Match:
|
||||
obj: Any
|
||||
diff: list[dict] # empty = exact match
|
||||
busy: bool = False # the central object waits for a confirmation: linking is refused until that is settled
|
||||
|
||||
|
||||
def _norm(value) -> Any:
|
||||
if value is None:
|
||||
return ""
|
||||
return value if isinstance(value, bool) else str(value)
|
||||
|
||||
|
||||
def _show(value) -> str:
|
||||
return ("true" if value else "false") if isinstance(value, bool) else ("" if value is None else str(value))
|
||||
|
||||
|
||||
def diff_fields(obj, values: dict, host: OpnsenseHost) -> list[dict]:
|
||||
"""Fields in which the central object (resolved for `host`) differs from the live `values`."""
|
||||
raw = {key: getattr(obj, key) for key in values}
|
||||
try:
|
||||
central = variables_service.resolve(raw, host)
|
||||
except variables_service.MissingVariables as exc:
|
||||
# Cannot be compared (and could not be deployed to this host either): report, never match.
|
||||
return [{"field": "variables", "central": ", ".join(exc.missing[host.name]), "live": ""}]
|
||||
return [{"field": key, "central": _show(central[key]), "live": _show(values[key])}
|
||||
for key in values if _norm(central[key]) != _norm(values[key])]
|
||||
|
||||
|
||||
def _options(spec: LinkSpec):
|
||||
return [
|
||||
selectinload(spec.model.targets),
|
||||
selectinload(spec.model.groups).selectinload(DeviceGroup.hosts),
|
||||
selectinload(spec.model.deployments),
|
||||
]
|
||||
|
||||
|
||||
async def load_central(db: AsyncSession, spec: LinkSpec) -> list:
|
||||
objs = list((await db.scalars(select(spec.model).options(*_options(spec)))).unique().all())
|
||||
await hydrate_hierarchy_targets(db, objs)
|
||||
return objs
|
||||
|
||||
|
||||
def find(objs: list, spec: LinkSpec, host: OpnsenseHost, values: dict) -> tuple[list[Match], list[Match]]:
|
||||
"""(exact matches, same-identity objects that differ). Objects that already have a deployment on
|
||||
this host are not candidates: linking would collide with that deployment."""
|
||||
exact: list[Match] = []
|
||||
similar: list[Match] = []
|
||||
for obj in objs:
|
||||
if any(d.host_id == host.id for d in obj.deployments):
|
||||
continue
|
||||
if _norm(getattr(obj, spec.identity)) != _norm(values[spec.identity]):
|
||||
continue
|
||||
d = diff_fields(obj, values, host)
|
||||
busy = any(dep.status in BUSY_STATUSES for dep in obj.deployments)
|
||||
(similar if d else exact).append(Match(obj, d, busy))
|
||||
# prefer the object that already reaches the most hosts (the "real" one), then a stable order
|
||||
exact.sort(key=lambda m: (-len(effective_hosts(m.obj)), str(m.obj.id)))
|
||||
return exact, similar
|
||||
|
||||
|
||||
def describe(spec: LinkSpec, match: Match) -> dict:
|
||||
return {
|
||||
"id": match.obj.id,
|
||||
"name": getattr(match.obj, spec.identity),
|
||||
"targets": [h.name for h in effective_hosts(match.obj)],
|
||||
"diff": match.diff,
|
||||
}
|
||||
|
||||
|
||||
async def link(db: AsyncSession, spec: LinkSpec, host: OpnsenseHost, object_id, remote_uuid: str, values: dict):
|
||||
"""Links the live object `remote_uuid` (whose normalised fields are `values`, read fresh by the caller)
|
||||
to the central object `object_id`. Returns it; a repeated call for the same pair changes nothing."""
|
||||
obj = (await db.execute(
|
||||
select(spec.model).where(spec.model.id == object_id).options(*_options(spec))
|
||||
.execution_options(populate_existing=True)
|
||||
)).unique().scalar_one_or_none()
|
||||
if obj is None:
|
||||
raise LinkTargetNotFound(str(object_id))
|
||||
await hydrate_hierarchy_targets(db, [obj])
|
||||
|
||||
existing = next((d for d in obj.deployments if d.host_id == host.id), None)
|
||||
if existing is not None:
|
||||
if getattr(existing, spec.uuid_attr) == remote_uuid:
|
||||
return obj # already linked: nothing to do
|
||||
raise HostAlreadyHasObject(str(object_id))
|
||||
if any(d.status in BUSY_STATUSES for d in obj.deployments):
|
||||
raise LinkTargetBusy(str(object_id))
|
||||
changed = diff_fields(obj, values, host)
|
||||
if changed:
|
||||
raise LinkTargetChanged(changed)
|
||||
|
||||
if host.id not in {h.id for h in effective_hosts(obj)}:
|
||||
obj.targets.append(host)
|
||||
# an association-only change does not touch the row; stale edit forms must still be refused (see edit_rule)
|
||||
obj.updated_at = datetime.now(timezone.utc)
|
||||
await db.flush()
|
||||
spec.revision(db, obj, f"Mit {host.name} verknüpft")
|
||||
db.add(spec.deployment(**{
|
||||
spec.fk: obj.id, "host_id": host.id, "status": "success", "message": LINK_MESSAGE,
|
||||
spec.uuid_attr: remote_uuid, "last_payload": spec.payload(obj, host), "previous_payload": None,
|
||||
}))
|
||||
await db.flush()
|
||||
return obj
|
||||
@@ -130,6 +130,9 @@ async def sweep_expired_alias_deployments(db: AsyncSession) -> list[AliasDeploym
|
||||
|
||||
|
||||
async def undeploy_alias(db: AsyncSession, alias: FirewallAlias) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)
|
||||
failures = []
|
||||
result = await db.execute(select(AliasDeployment).where(AliasDeployment.alias_id == alias.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -38,19 +38,35 @@ MAX_TOKENS_PER_USER = 50
|
||||
_LAST_USED_GRANULARITY = timedelta(minutes=1)
|
||||
|
||||
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)")
|
||||
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
|
||||
# Rule cleanup (1.31): "I still have access" is a human decision, so confirming a cleanup is never possible with a
|
||||
# token, whatever its scope (a script must not be able to confirm blindly).
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$")
|
||||
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
||||
# notification settings the webhook URL/SMTP data): admin scope only.
|
||||
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
|
||||
_MCP_PATH = "/api/mcp"
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
|
||||
# it compares a posted file with the database and writes nothing.
|
||||
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
|
||||
_ADMIN_ONLY_WRITES = re.compile(
|
||||
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
||||
r"|^/api/logs/settings$"
|
||||
r"|^/api/hosts(/tls-probe|/[^/]+)?$"
|
||||
r"|^/api/hosts/[^/]+/gui-ticket$"
|
||||
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
|
||||
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
|
||||
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
|
||||
r"|^/api/config/import/apply$"
|
||||
)
|
||||
|
||||
|
||||
# Rule cleanup: executing a cleanup, retrying a failed revert/reconcile and restoring a deleted rule are admin-only.
|
||||
# operators_may_deploy defaults to True, so _ROLLOUT_POST alone cannot express "admin only"; this pattern is checked
|
||||
# for operator accounts (operator_denies) and for tokens below scope admin (denies). Not part of _ADMIN_ONLY_WRITES:
|
||||
# that one means "who may access OPNexus / where data goes". The router checks the same rule once more.
|
||||
_ADMIN_ONLY_CLEANUP = re.compile(r"^/api/hosts/[^/]+/rule-cleanup/([^/]+/(deploy|retry)|items/[^/]+/restore)/?$")
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
@@ -63,13 +79,13 @@ def denies(scope: str, method: str, path: str) -> bool:
|
||||
"""True if a token with this scope must not make this request."""
|
||||
if _SESSION_ONLY.match(path):
|
||||
return True
|
||||
if path == _MCP_PATH:
|
||||
if path in _READ_ONLY_POSTS:
|
||||
return False
|
||||
if method in _SAFE_METHODS:
|
||||
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
|
||||
if scope == "read":
|
||||
return True
|
||||
if scope == "deploy" and _ADMIN_ONLY_WRITES.match(path):
|
||||
if scope == "deploy" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -78,7 +94,8 @@ def operator_denies(method: str, path: str) -> bool:
|
||||
"""True if an `operator` account must not make this request: the writes that
|
||||
change who may access OPNexus or where it sends data. (Their own theme at
|
||||
PUT /api/settings stays open; account endpoints are not matched here.)"""
|
||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(_ADMIN_ONLY_WRITES.match(path))
|
||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(
|
||||
_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))
|
||||
|
||||
|
||||
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
|
||||
@@ -88,6 +105,9 @@ def operator_denies(method: str, path: str) -> bool:
|
||||
_ROLLOUT_POST = re.compile(
|
||||
r"/(deploy|push|sign|poll|register|reassert-ordering"
|
||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||
# Rule cleanup (1.31): switching rules off (/deploy matches above) and taking it back. Confirming and
|
||||
# previewing stay open, like confirm_rule.
|
||||
r"|rule-cleanup/[^/]+/(revert|retry)"
|
||||
r"|maintenance/(execute|carp-execute|schedule)"
|
||||
r"|network/changes"
|
||||
r"|services/[^/]+/(start|stop|restart)"
|
||||
|
||||
@@ -13,6 +13,9 @@ from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
|
||||
context = ContextVar("audit_context", default=None)
|
||||
# A key in ``details`` whose content only admins (session or admin-scope token) get to read (routers/audit.py):
|
||||
# firewall addresses and management sources of the rule cleanup.
|
||||
ADMIN_ONLY_DETAILS = "admin_only"
|
||||
|
||||
|
||||
# Column widths of audit_events. entity_id/host_id/action are taken from the
|
||||
|
||||
@@ -26,6 +26,7 @@ from app.models.host import OpnsenseHost
|
||||
from app.services import backup_store, rule_hit_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.rule_analysis import coverage
|
||||
|
||||
MAX_CONFIG_BYTES = 32 * 1024 * 1024
|
||||
BACKUP_MAX_AGE_DAYS = 7
|
||||
@@ -83,10 +84,15 @@ def _mvc_rule(rule: Element) -> dict:
|
||||
"sport": _text(rule, "source_port"),
|
||||
"dst": _text(rule, "destination_net") or "any", "dst_not": _text(rule, "destination_not") == "1",
|
||||
"dport": _text(rule, "destination_port"),
|
||||
"floating": False,
|
||||
"if_not": _text(rule, "interfacenot") == "1",
|
||||
"floating": False, # decided in extract_rules (needs the list of interfaces)
|
||||
"quick": _text(rule, "quick") != "0",
|
||||
"descr": _text(rule, "description") or "",
|
||||
"order": int(_text(rule, "sequence") or 0),
|
||||
"sched": _text(rule, "sched") or "", "tagged": _text(rule, "tagged") or "",
|
||||
"received_on": _text(rule, "received-on") or "",
|
||||
"icmptype": ",".join(filter(None, [_text(rule, "icmptype"), _text(rule, "icmp6type")])),
|
||||
"limited": any(_text(rule, tag) for tag in ("max-src-conn-rate", "max-pkt-rate-number", "overload", "tcpflags1")),
|
||||
}
|
||||
|
||||
|
||||
@@ -101,17 +107,29 @@ def _legacy_rule(rule: Element, index: int) -> dict:
|
||||
"ipprotocol": _text(rule, "ipprotocol") or "inet",
|
||||
"protocol": (_text(rule, "protocol") or "any").lower(),
|
||||
"src": src, "src_not": src_not, "sport": sport, "dst": dst, "dst_not": dst_not, "dport": dport,
|
||||
"if_not": rule.find("interfacenot") is not None,
|
||||
"floating": _text(rule, "floating") == "yes",
|
||||
"quick": _text(rule, "quick") != "0",
|
||||
# Legacy rules: an interface rule is always evaluated "quick"; only a floating rule has the option,
|
||||
# and it is off unless <quick> says yes.
|
||||
"quick": _text(rule, "floating") != "yes" or (_text(rule, "quick") or "").lower() in ("yes", "1"),
|
||||
"descr": _text(rule, "descr") or "",
|
||||
"order": index,
|
||||
"sched": _text(rule, "sched") or "", "tagged": _text(rule, "tagged") or "",
|
||||
"received_on": _text(rule, "received-on") or "", "icmptype": _text(rule, "icmptype") or "",
|
||||
"limited": any(_text(rule, tag) for tag in ("max-src-conn-rate", "max-pkt-rate-number", "overload", "tcpflags1")),
|
||||
}
|
||||
|
||||
|
||||
def extract_rules(root: Element) -> list[list[dict]]:
|
||||
"""Rule lists in evaluation order, one per source (MVC, legacy)."""
|
||||
mvc = sorted((_mvc_rule(r) for r in root.findall("OPNsense/Firewall/Filter/rules/rule")), key=lambda r: r["order"])
|
||||
mvc = [_mvc_rule(r) for r in root.findall("OPNsense/Firewall/Filter/rules/rule")]
|
||||
known = {node.tag for node in root.findall("interfaces/*")}
|
||||
for rule in mvc: # same rule as OPNsense: not exactly one interface (or "not") = floating
|
||||
rule["floating"] = len(rule["interfaces"]) != 1 or rule["if_not"]
|
||||
rule["group"] = not rule["floating"] and rule["interfaces"][0] not in known
|
||||
mvc.sort(key=coverage.config_order)
|
||||
legacy = [_legacy_rule(r, i) for i, r in enumerate(root.findall("filter/rule"))]
|
||||
legacy.sort(key=coverage.config_order)
|
||||
return [group for group in (mvc, legacy) if group]
|
||||
|
||||
|
||||
@@ -133,41 +151,22 @@ def _is_any_any(rule: dict) -> bool:
|
||||
|
||||
|
||||
|
||||
def _covers(a: dict, b: dict) -> bool:
|
||||
"""Every packet matched by rule b is already matched by the earlier rule a.
|
||||
Only exact-or-any comparisons, so it never claims coverage it cannot prove."""
|
||||
if not (a["enabled"] and b["enabled"]) or a["floating"] or b["floating"] or not a["quick"]:
|
||||
return False
|
||||
if not a["interfaces"] or not set(b["interfaces"]) <= set(a["interfaces"]):
|
||||
return False
|
||||
if a["direction"] != b["direction"] or a["ipprotocol"] != b["ipprotocol"]:
|
||||
return False
|
||||
if a["protocol"] != "any" and a["protocol"] != b["protocol"]:
|
||||
return False
|
||||
for side in ("src", "dst"):
|
||||
if a[side + "_not"] or b[side + "_not"]:
|
||||
if (a[side], a[side + "_not"]) != (b[side], b[side + "_not"]):
|
||||
return False
|
||||
elif a[side] != "any" and a[side] != b[side]:
|
||||
return False
|
||||
for port in ("sport", "dport"):
|
||||
if a[port] and a[port] != b[port]:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _label(rule: dict) -> str:
|
||||
return rule["descr"] or f"{rule['action']} {rule['src']} → {rule['dst']}"
|
||||
|
||||
|
||||
def analyse_rules(groups: list[list[dict]]) -> dict:
|
||||
"""Shadowed and duplicate rules. The coverage logic lives in rule_analysis/coverage.py (shared with the
|
||||
cleanup candidates); the rules of one list are compared in evaluation order."""
|
||||
shadowed, duplicates = [], []
|
||||
for rules in groups:
|
||||
for j, later in enumerate(rules):
|
||||
for earlier in rules[:j]:
|
||||
if _covers(earlier, later):
|
||||
(duplicates if earlier["action"] == later["action"] and _covers(later, earlier) else shadowed).append(_label(later))
|
||||
break
|
||||
ordered = sorted(rules, key=coverage.config_order)
|
||||
norm = [coverage.from_config_rule(rule) for rule in ordered]
|
||||
by_identity = {id(n): r for n, r in zip(norm, ordered)}
|
||||
for overlap in coverage.analyse(norm):
|
||||
if overlap.by.order == overlap.rule.order:
|
||||
continue # equal position in config.xml: the real order is unknown, claim nothing
|
||||
(duplicates if overlap.kind == "duplicate" else shadowed).append(_label(by_identity[id(overlap.rule)]))
|
||||
return {"shadowed": shadowed, "duplicates": duplicates}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,384 @@
|
||||
"""Tickets and gateway sessions for the firewall GUI access (1.27, gui-gateway container).
|
||||
|
||||
Flow (see docs in routers/gui_access.py):
|
||||
* An admin asks for a ticket (POST /api/hosts/{id}/gui-ticket). 32 random bytes, valid 30 s, single use;
|
||||
only its SHA-256 is stored, bound to the OPNexus session (same hash as sessions.token_hash), the user,
|
||||
the host and a slot (gateway port = its own browser origin).
|
||||
* The gateway redeems it through /internal/gui/redeem and gets what it needs to reach the firewall —
|
||||
base URL, pinned CA, overlay proxy, timeouts. Never the API key or secret.
|
||||
* Every 20 s the gateway re-checks the gateway session (/internal/gui/revalidate): OPNexus session still
|
||||
valid, user still admin, switch still on, host still eligible, absolute limit not reached.
|
||||
|
||||
A slot stays assigned to one firewall for as long as that browser session has the firewall open; another
|
||||
firewall gets another slot (another origin), so two firewall GUIs never share an origin at the same time.
|
||||
After the last gateway session of a slot ended, the slot goes to a DIFFERENT firewall only after SLOT_COOLDOWN
|
||||
(1.29, S1): a popup of a possibly compromised firewall GUI may still be open and its cookies belong to that
|
||||
origin. The same firewall may take the slot again at once.
|
||||
|
||||
The target the gateway talks to (origin, pinned CA, overlay proxy) is fingerprinted at redeem time; every
|
||||
re-check compares it with the host's current configuration and ends the session when it changed (S4).
|
||||
"""
|
||||
import hashlib
|
||||
import secrets
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import delete, func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode
|
||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser, Session
|
||||
from app.services import audit_service, firewall_url, overlay_service
|
||||
|
||||
TICKET_TTL = timedelta(seconds=30)
|
||||
# 20 s: a switch-off, logout or role change must end a gateway session within 30 s of the change, and
|
||||
# the last re-check can lie up to one interval before it (live test 2026-10-21: 30 s gave 31 s).
|
||||
REVALIDATE_SECONDS = 20
|
||||
# A gateway session nobody re-checked for this long is treated as ended (gateway crashed without telling us).
|
||||
SESSION_STALE_AFTER = timedelta(seconds=REVALIDATE_SECONDS * 3)
|
||||
MAX_SLOTS = 4
|
||||
# A slot freed by firewall A goes to firewall B no earlier than this long after A's session ended.
|
||||
SLOT_COOLDOWN = timedelta(minutes=5)
|
||||
# Open gateway sessions at once (the gateway container enforces the same number; test_gui_access checks they match).
|
||||
MAX_GATEWAY_SESSIONS = 64
|
||||
MIN_GATEWAY_TOKEN_LENGTH = 32
|
||||
AUDIT_KIND = "gui_access"
|
||||
|
||||
|
||||
@dataclass
|
||||
class GuiAccessError(Exception):
|
||||
status: int
|
||||
code: ErrorCode
|
||||
params: dict
|
||||
|
||||
|
||||
def hash_value(raw: str) -> str:
|
||||
return hashlib.sha256(raw.encode()).hexdigest()
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
# ── configuration ─────────────────────────────────────────────────────────
|
||||
|
||||
def slot_sites() -> dict[int, list[str]]:
|
||||
"""Slot port -> configured https origins with that port (OPNEXUS_GUI_SITES)."""
|
||||
sites: dict[int, list[str]] = {}
|
||||
for origin in settings.gui_site_origins:
|
||||
try:
|
||||
parts = urlsplit(origin)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
continue
|
||||
if parts.scheme != "https" or not parts.hostname or not port or parts.path not in ("", "/"):
|
||||
continue
|
||||
sites.setdefault(port, []).append(f"https://{parts.netloc}")
|
||||
return sites
|
||||
|
||||
|
||||
def gateway_configured() -> bool:
|
||||
return len(settings.gui_gateway_token) >= MIN_GATEWAY_TOKEN_LENGTH and bool(slot_sites())
|
||||
|
||||
|
||||
def unavailable_reasons() -> list[str]:
|
||||
reasons = []
|
||||
if not settings.cookie_secure:
|
||||
reasons.append("cookie_secure")
|
||||
if len(settings.gui_gateway_token) < MIN_GATEWAY_TOKEN_LENGTH:
|
||||
reasons.append("gateway_token")
|
||||
if not slot_sites():
|
||||
reasons.append("gui_sites")
|
||||
return reasons
|
||||
|
||||
|
||||
def active_slots(row: GuiAccessSettings) -> list[int]:
|
||||
return sorted(slot_sites())[: max(1, min(row.slot_count, MAX_SLOTS))]
|
||||
|
||||
|
||||
async def get_settings(db: AsyncSession) -> GuiAccessSettings:
|
||||
row = await db.get(GuiAccessSettings, 1)
|
||||
if row is None:
|
||||
row = GuiAccessSettings(id=1, enabled=False, idle_timeout_minutes=15, absolute_timeout_minutes=60, slot_count=MAX_SLOTS)
|
||||
db.add(row)
|
||||
await db.flush()
|
||||
return row
|
||||
|
||||
|
||||
def host_refusal(host: OpnsenseHost) -> str | None:
|
||||
"""Why this firewall must not be opened through the gateway (None = eligible)."""
|
||||
if not host.enabled:
|
||||
return "disabled"
|
||||
if urlsplit(host.base_url.strip()).scheme != "https" or firewall_url.check_base_url(host.base_url):
|
||||
return "https"
|
||||
if not host.verify_tls:
|
||||
return "tls"
|
||||
return None
|
||||
|
||||
|
||||
def firewall_origin(base_url: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
return f"https://{parts.netloc.rsplit('@', 1)[-1]}"
|
||||
|
||||
|
||||
def target_fingerprint(host: OpnsenseHost) -> str:
|
||||
"""What a gateway session talks to: origin, pinned CA and overlay route. Changes -> the session must end."""
|
||||
material = "\n".join([firewall_origin(host.base_url), host.tls_ca_pem or "", overlay_service.proxy_for(host.base_url) or "",
|
||||
"verify" if host.verify_tls else "noverify"])
|
||||
return hash_value(material)
|
||||
|
||||
|
||||
# ── audit ─────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _audit(actor: str, *, action: str, status: str, host: OpnsenseHost | None = None,
|
||||
host_id=None, entity_id=None, message: str | None = None, details: dict | None = None) -> None:
|
||||
token = audit_service.context.set({"actor": actor, "id": str(uuid.uuid4()), "action": action})
|
||||
try:
|
||||
await audit_service.persist(
|
||||
kind=AUDIT_KIND, action=action, status=status, entity_id=str(entity_id) if entity_id else None,
|
||||
host_id=str(host.id) if host is not None else (str(host_id) if host_id else None),
|
||||
host_name=host.name if host is not None else None, message=message, details=details or {},
|
||||
)
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
# ── tickets ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _redeem_origin(slot: int, request_host: str) -> str:
|
||||
candidates = slot_sites().get(slot, [])
|
||||
wanted = (urlsplit(f"//{request_host}").hostname or "").lower()
|
||||
for origin in candidates:
|
||||
if (urlsplit(origin).hostname or "").lower() == wanted:
|
||||
return origin
|
||||
return candidates[0]
|
||||
|
||||
|
||||
async def _pick_slot(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id) -> int:
|
||||
now = _now()
|
||||
slots = active_slots(row)
|
||||
absolute = timedelta(minutes=row.absolute_timeout_minutes)
|
||||
in_use: dict[int, uuid.UUID] = {}
|
||||
open_sessions = await db.scalars(select(GuiSession).where(
|
||||
GuiSession.session_hash == session_hash, GuiSession.ended_at.is_(None), GuiSession.started_at > now - absolute))
|
||||
for gs in open_sessions:
|
||||
in_use[gs.slot] = gs.host_id
|
||||
pending = await db.scalars(select(GuiTicket).where(
|
||||
GuiTicket.session_hash == session_hash, GuiTicket.redeemed_at.is_(None), GuiTicket.expires_at > now))
|
||||
for ticket in pending:
|
||||
in_use.setdefault(ticket.slot, ticket.host_id)
|
||||
for slot in slots:
|
||||
if in_use.get(slot) == host_id:
|
||||
return slot # this firewall is already open in this browser session: same origin again
|
||||
free = [s for s in slots if s not in in_use]
|
||||
if not free:
|
||||
raise GuiAccessError(409, ErrorCode.GUI_NO_FREE_SLOT, {"slots": str(len(slots))})
|
||||
# S1: a slot this user's browser may still have a window of ANOTHER firewall on stays closed to this
|
||||
# firewall — while a gateway session of the user is open there (any browser session) and for SLOT_COOLDOWN
|
||||
# after the last one ended. The same firewall may take it again at once.
|
||||
held = (await db.execute(
|
||||
select(GuiSession.slot, GuiSession.host_id, GuiSession.ended_at, GuiSession.last_checked_at).where(
|
||||
GuiSession.user_id == user_id,
|
||||
((GuiSession.ended_at.is_(None)) & ((GuiSession.started_at > now - absolute) | (GuiSession.last_checked_at > now - SLOT_COOLDOWN)))
|
||||
| (GuiSession.ended_at > now - SLOT_COOLDOWN)))).all()
|
||||
cooling: dict[int, datetime] = {} # slot -> when it may go to another firewall
|
||||
for slot, h_id, ended_at, checked_at in held:
|
||||
if h_id == host_id or slot not in free:
|
||||
continue
|
||||
if ended_at is None and checked_at is not None and now - checked_at > SESSION_STALE_AFTER:
|
||||
ended_at = checked_at # the gateway stopped re-checking it (crash, kill): it ended when it was last seen
|
||||
release = (ended_at + SLOT_COOLDOWN) if ended_at is not None else now + SLOT_COOLDOWN
|
||||
if release <= now:
|
||||
continue # cooled down long ago (a session that stopped answering a while back)
|
||||
cooling[slot] = max(cooling.get(slot, release), release)
|
||||
usable = [s for s in free if s not in cooling]
|
||||
if not usable:
|
||||
wait = min(cooling.values()) - now
|
||||
raise GuiAccessError(409, ErrorCode.GUI_SLOT_COOLING_DOWN,
|
||||
{"minutes": str(max(1, -(-int(wait.total_seconds()) // 60)))})
|
||||
free = usable
|
||||
# Prefer the slot this user had this firewall on before, then the slot the user used least recently
|
||||
# (an old tab of another firewall on a reused origin is the documented residual risk).
|
||||
history = (await db.execute(
|
||||
select(GuiSession.slot, GuiSession.host_id, func.max(GuiSession.started_at))
|
||||
.where(GuiSession.user_id == user_id).group_by(GuiSession.slot, GuiSession.host_id)
|
||||
)).all()
|
||||
for slot, h_id, _ in sorted(history, key=lambda r: r[2], reverse=True):
|
||||
if h_id == host_id and slot in free:
|
||||
return slot
|
||||
last_used = {}
|
||||
for slot, _, started in history:
|
||||
last_used[slot] = max(last_used.get(slot, started), started)
|
||||
epoch = datetime.min.replace(tzinfo=timezone.utc)
|
||||
return min(free, key=lambda s: (last_used.get(s, epoch), s))
|
||||
|
||||
|
||||
async def issue_ticket(db: AsyncSession, *, user: AdminUser, session_token: str, host: OpnsenseHost,
|
||||
request_host: str) -> dict:
|
||||
row = await get_settings(db)
|
||||
now = _now()
|
||||
await db.execute(delete(GuiTicket).where(GuiTicket.expires_at < now - timedelta(hours=1)))
|
||||
session_hash = hash_value(session_token)
|
||||
slot = await _pick_slot(db, row, session_hash, user.id, host.id)
|
||||
# Reopening a slot in the same browser session replaces its earlier gateway session, so that one does not count.
|
||||
open_count = await db.scalar(select(func.count()).select_from(GuiSession).where(
|
||||
GuiSession.ended_at.is_(None),
|
||||
GuiSession.started_at > now - timedelta(minutes=row.absolute_timeout_minutes),
|
||||
~((GuiSession.session_hash == session_hash) & (GuiSession.slot == slot))))
|
||||
if (open_count or 0) >= MAX_GATEWAY_SESSIONS:
|
||||
raise GuiAccessError(409, ErrorCode.GUI_TOO_MANY_SESSIONS, {"max": str(MAX_GATEWAY_SESSIONS)})
|
||||
raw = secrets.token_urlsafe(32)
|
||||
db.add(GuiTicket(ticket_hash=hash_value(raw), session_hash=session_hash, user_id=user.id, host_id=host.id,
|
||||
slot=slot, created_at=now, expires_at=now + TICKET_TTL))
|
||||
await db.commit()
|
||||
await _audit(user.username, action="ticket", status="accepted", host=host, details={"slot": slot})
|
||||
return {"url": f"{_redeem_origin(slot, request_host)}/__onx/redeem?t={raw}", "slot": slot,
|
||||
"expires_in": int(TICKET_TTL.total_seconds())}
|
||||
|
||||
|
||||
# ── gateway side (internal endpoints) ─────────────────────────────────────
|
||||
|
||||
async def _live_check(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id,
|
||||
slot: int) -> tuple[str | None, AdminUser | None, OpnsenseHost | None]:
|
||||
session = await db.scalar(select(Session).where(Session.token_hash == session_hash))
|
||||
if session is None or session.expires_at <= _now() or session.user_id != user_id:
|
||||
return "session", None, None
|
||||
user = await db.get(AdminUser, user_id)
|
||||
if user is None or user.role != "admin":
|
||||
return "role", user, None
|
||||
if not row.enabled:
|
||||
return "disabled", user, None
|
||||
if not settings.cookie_secure or not gateway_configured():
|
||||
return "config", user, None
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None or host_refusal(host):
|
||||
return "host", user, host
|
||||
if slot not in active_slots(row):
|
||||
return "slot", user, host
|
||||
return None, user, host
|
||||
|
||||
|
||||
def _timing(row: GuiAccessSettings, started_at: datetime) -> dict:
|
||||
deadline = started_at + timedelta(minutes=row.absolute_timeout_minutes)
|
||||
return {"idle_timeout_seconds": row.idle_timeout_minutes * 60,
|
||||
"absolute_deadline": int(deadline.timestamp()),
|
||||
"revalidate_seconds": REVALIDATE_SECONDS}
|
||||
|
||||
|
||||
async def redeem(db: AsyncSession, *, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
|
||||
"""{'ok': True, ...upstream data...} or {'ok': False, 'reason': ...}. Any attempt burns the ticket."""
|
||||
now = _now()
|
||||
ticket = await db.scalar(select(GuiTicket).where(GuiTicket.ticket_hash == ticket_hash).with_for_update())
|
||||
if ticket is None:
|
||||
await db.rollback()
|
||||
return {"ok": False, "reason": "unknown"}
|
||||
replayed = ticket.redeemed_at is not None
|
||||
ticket.redeemed_at = ticket.redeemed_at or now
|
||||
row = await get_settings(db)
|
||||
reason = None
|
||||
if replayed:
|
||||
reason = "replayed"
|
||||
elif ticket.expires_at <= now:
|
||||
reason = "expired"
|
||||
elif not secrets.compare_digest(ticket.session_hash, session_hash):
|
||||
reason = "session"
|
||||
elif ticket.slot != slot:
|
||||
reason = "slot"
|
||||
user = host = None
|
||||
if reason is None:
|
||||
reason, user, host = await _live_check(db, row, ticket.session_hash, ticket.user_id, ticket.host_id, slot)
|
||||
if user is None:
|
||||
user = await db.get(AdminUser, ticket.user_id)
|
||||
if host is None:
|
||||
host = await db.get(OpnsenseHost, ticket.host_id)
|
||||
actor = user.username if user is not None else "unknown"
|
||||
if reason is not None:
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
|
||||
message=reason, details={"slot": slot, "client_ip": client_ip, "reason": reason})
|
||||
return {"ok": False, "reason": reason}
|
||||
# The same browser session reopening this slot replaces the earlier gateway session there.
|
||||
superseded = (await db.scalars(select(GuiSession).where(
|
||||
GuiSession.session_hash == session_hash, GuiSession.slot == slot, GuiSession.ended_at.is_(None)))).all()
|
||||
if any(old.host_id != host.id for old in superseded):
|
||||
# Two quick tickets both saw the slot free: the second must not take over another firewall's origin
|
||||
# without the cooling-down time (S1).
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
|
||||
message="slot", details={"slot": slot, "client_ip": client_ip, "reason": "slot"})
|
||||
return {"ok": False, "reason": "slot"}
|
||||
for old in superseded:
|
||||
old.ended_at, old.end_reason = now, "superseded"
|
||||
gs = GuiSession(session_hash=session_hash, user_id=user.id, host_id=host.id, slot=slot,
|
||||
client_ip=(client_ip or None) and client_ip[:45], started_at=now, last_checked_at=now)
|
||||
db.add(gs)
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="accepted", host=host, entity_id=gs.id,
|
||||
details={"slot": slot, "client_ip": client_ip})
|
||||
return {
|
||||
"ok": True, "gui_session_id": str(gs.id), "host_id": str(host.id), "host_name": host.name,
|
||||
"upstream_origin": firewall_origin(host.base_url),
|
||||
"ca_pem": host.tls_ca_pem or None, # public certificate (pin); None = system trust store
|
||||
"proxy_url": overlay_service.proxy_for(host.base_url),
|
||||
"target_fp": target_fingerprint(host),
|
||||
**_timing(row, now),
|
||||
}
|
||||
|
||||
|
||||
async def _finish(db: AsyncSession, gs: GuiSession, reason: str) -> None:
|
||||
now = _now()
|
||||
if gs.ended_at is None:
|
||||
gs.ended_at, gs.end_reason = now, reason[:32]
|
||||
await db.commit()
|
||||
user = await db.get(AdminUser, gs.user_id)
|
||||
host = await db.get(OpnsenseHost, gs.host_id)
|
||||
await _audit(user.username if user else "unknown", action="end", status="accepted", host=host,
|
||||
host_id=gs.host_id, entity_id=gs.id, message=reason,
|
||||
details={"slot": gs.slot, "client_ip": gs.client_ip, "reason": reason,
|
||||
"duration_seconds": int((gs.ended_at - gs.started_at).total_seconds())})
|
||||
|
||||
|
||||
async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str,
|
||||
target_fp: str | None = None) -> dict:
|
||||
gs = await db.get(GuiSession, gui_session_id)
|
||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||
return {"ok": False, "reason": "unknown"}
|
||||
if gs.ended_at is not None:
|
||||
return {"ok": False, "reason": gs.end_reason or "ended"}
|
||||
row = await get_settings(db)
|
||||
if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):
|
||||
await _finish(db, gs, "absolute")
|
||||
return {"ok": False, "reason": "absolute"}
|
||||
reason, _, host = await _live_check(db, row, gs.session_hash, gs.user_id, gs.host_id, gs.slot)
|
||||
# S4: the gateway presents the fingerprint it got at redeem time. Firewall address, pinned certificate or
|
||||
# overlay route changed since -> the session (and the upstream connection it holds) ends now.
|
||||
if reason is None and target_fp is not None and host is not None \
|
||||
and not secrets.compare_digest(target_fp, target_fingerprint(host)):
|
||||
reason = "target"
|
||||
if reason is not None:
|
||||
await _finish(db, gs, reason)
|
||||
return {"ok": False, "reason": reason}
|
||||
gs.last_checked_at = _now()
|
||||
await db.commit()
|
||||
return {"ok": True, **_timing(row, gs.started_at)}
|
||||
|
||||
|
||||
async def end(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str, reason: str) -> bool:
|
||||
gs = await db.get(GuiSession, gui_session_id)
|
||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||
return False
|
||||
if gs.ended_at is None:
|
||||
await _finish(db, gs, reason)
|
||||
return True
|
||||
|
||||
|
||||
async def end_all(db: AsyncSession, reason: str) -> int:
|
||||
"""Switch turned off: mark every open gateway session ended (the gateway drops them on its next check)."""
|
||||
result = await db.execute(update(GuiSession).where(GuiSession.ended_at.is_(None))
|
||||
.values(ended_at=_now(), end_reason=reason[:32]))
|
||||
return result.rowcount or 0
|
||||
@@ -34,7 +34,27 @@ async def peers(db, host):
|
||||
return [host]
|
||||
|
||||
|
||||
async def ensure_no_maintenance(db, hosts):
|
||||
async def ensure_no_open_cleanup(db, hosts):
|
||||
"""A rule cleanup (1.31) that is running, waiting for its confirmation or unresolved owns the host and its
|
||||
cluster peers: nothing else may write to the firewall until it is confirmed, reverted or settled."""
|
||||
from app.services.rule_cleanup_service import has_open_cleanup
|
||||
ids = set()
|
||||
for host in hosts:
|
||||
ids.update(h.id for h in await peers(db, host))
|
||||
if ids and await has_open_cleanup(db, ids):
|
||||
raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)
|
||||
|
||||
|
||||
async def ensure_no_open_cleanup_on(db, model, condition):
|
||||
"""ensure_no_open_cleanup for every host that has a deployment row of ``model`` matching ``condition`` (the
|
||||
undeploy paths: deleting a central object removes it from all its firewalls)."""
|
||||
ids = set((await db.scalars(select(model.host_id).where(condition))).all())
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, i) for i in ids] if h is not None]
|
||||
await ensure_no_open_cleanup(db, hosts)
|
||||
|
||||
|
||||
async def ensure_no_maintenance(db, hosts, *, cleanup=True):
|
||||
"""cleanup=False: only the cleanup service itself, which reports an open cleanup with its own error code."""
|
||||
ids = set()
|
||||
for host in hosts:
|
||||
ids.update(h.id for h in await peers(db, host))
|
||||
@@ -42,6 +62,8 @@ async def ensure_no_maintenance(db, hosts):
|
||||
raise api_error(409, ErrorCode.OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY)
|
||||
if ids and await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||
raise api_error(409, ErrorCode.OPEN_RESTORE_BLOCKS_DEPLOY)
|
||||
if cleanup and ids:
|
||||
await ensure_no_open_cleanup(db, hosts)
|
||||
|
||||
|
||||
async def _shared_change_blockers(db, fleet):
|
||||
@@ -56,6 +78,9 @@ async def _shared_change_blockers(db, fleet):
|
||||
blocked.append("Eine frühere Wartungsaktion im Host/Cluster muss zuerst geprüft und abgeschlossen werden.")
|
||||
if await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(fleet_ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||
blocked.append("Eine Wiederherstellung im Host/Cluster ist offen und muss zuerst bestätigt oder abgebrochen werden.")
|
||||
from app.services.rule_cleanup_service import has_open_cleanup
|
||||
if await has_open_cleanup(db, fleet_ids):
|
||||
blocked.append("Eine Regel-Bereinigung im Host/Cluster ist offen und muss zuerst bestätigt oder zurückgenommen werden.")
|
||||
return blocked
|
||||
|
||||
|
||||
|
||||
@@ -303,6 +303,9 @@ async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||
for this entity to gate on) — deletes Child/Local/Remote before their
|
||||
parent Connection (the ModelRelationField dependency direction), then
|
||||
the PreSharedKey (independent of the other 4), then reconfigures."""
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, IpsecTunnelDeployment, IpsecTunnelDeployment.tunnel_id == tunnel.id)
|
||||
failures = []
|
||||
result = await db.execute(select(IpsecTunnelDeployment).where(IpsecTunnelDeployment.tunnel_id == tunnel.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -21,6 +21,7 @@ from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.models.rollout import RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.routers.dashboard import collect_pending_actions
|
||||
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
||||
@@ -105,6 +106,7 @@ async def render(db: AsyncSession) -> str:
|
||||
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
|
||||
out.family("opnexus_host_api_rights_missing", "1 if the firewall refused a side reading (read=gateways, services, swap, cpu) because the API key lacks the right (last sample); absent otherwise.")
|
||||
configured = (await notification_settings_service.get_settings(db)).service_watchlist
|
||||
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
|
||||
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
|
||||
@@ -136,6 +138,8 @@ async def render(db: AsyncSession) -> str:
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, round(value * factor, 6), **gw_labels)
|
||||
for read in sorted(str(r) for r in raw.get("denied") or []):
|
||||
out.sample("opnexus_host_api_rights_missing", 1, **labels, read=read)
|
||||
for key, running in sorted((raw.get("services") or {}).items()):
|
||||
if key.split("/")[0] in watchlist:
|
||||
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
|
||||
@@ -176,4 +180,8 @@ async def render(db: AsyncSession) -> str:
|
||||
pending = await db.scalar(select(func.count()).select_from(ScheduledMaintenance).where(ScheduledMaintenance.status == "pending"))
|
||||
out.sample("opnexus_scheduled_maintenance_pending", pending or 0)
|
||||
|
||||
out.family("opnexus_rollout_runs", "Fleet rollout runs by status (waiting_window, wave_running, paused, succeeded, ...).")
|
||||
for status, count in sorted((await db.execute(select(RolloutRun.status, func.count()).group_by(RolloutRun.status))).all()):
|
||||
out.sample("opnexus_rollout_runs", count, status=status)
|
||||
|
||||
return out.text()
|
||||
@@ -97,6 +97,7 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
|
||||
muted = set(cfg.gateway_alert_muted or [])
|
||||
open_keys = await _open_keys(db, "gateway_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
@@ -110,6 +111,8 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
]
|
||||
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
|
||||
for name in sorted(names):
|
||||
if f"{host.id}:{name}" in muted:
|
||||
continue # alert switched off for this gateway: not judged, an open problem of it resolves
|
||||
flags: list[bool | None] = []
|
||||
for p in parsed:
|
||||
if p is None:
|
||||
@@ -131,6 +134,58 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
return problems
|
||||
|
||||
|
||||
# ── API rights ──────────────────────────────────────────────────────────────
|
||||
|
||||
# What the sampler reads on the side (resource_monitor_service) -> (ACL entry of the OPNsense user/group that
|
||||
# grants it, endpoint; the sample's "ok" list proves a reading went through). ACL names: opnsense/core
|
||||
# src/opnsense/mvc/app/models/OPNsense/Core/ACL/ACL.xml (page-system-gateways: api/routes/gateway/status;
|
||||
# page-status-services: api/core/service/*; page-system-login-logout "Lobby: Dashboard": api/diagnostics/cpu_usage/*
|
||||
# and api/diagnostics/system/system_swap).
|
||||
API_READS: dict[str, tuple[str, str]] = {
|
||||
"gateways": ("System: Gateways", "/api/routes/gateway/status"),
|
||||
"services": ("Status: Services", "/api/core/service/search"),
|
||||
"swap": ("Lobby: Dashboard", "/api/diagnostics/system/systemSwap"),
|
||||
"cpu": ("Lobby: Dashboard", "/api/diagnostics/cpu_usage/getCPUType"),
|
||||
}
|
||||
|
||||
|
||||
async def check_api_rights_missing(db: AsyncSession) -> dict[str, dict]:
|
||||
"""The API key of a firewall may be refused (403) on the side readings the alerts rely on (gateway state,
|
||||
service state): those alerts can then never fire, and nobody would notice. Only a refusal counts; a timeout
|
||||
or an error stays "unknown"."""
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
open_keys = await _open_keys(db, "api_rights_missing")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "denied", "ok")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
names = {n for s in samples for n in (s["denied"] or []) if n in API_READS} | (was_open & set(API_READS))
|
||||
for name in sorted(names):
|
||||
acl, path = API_READS[name]
|
||||
flags: list[bool | None] = []
|
||||
for s in samples:
|
||||
if s["denied"] is None:
|
||||
flags.append(None) # a sample from before this check existed
|
||||
elif name in s["denied"]:
|
||||
flags.append(True)
|
||||
else:
|
||||
# Only a reading that went through proves the right is back; a timeout or a sample from
|
||||
# before "ok" existed stays unknown (the problem neither opens nor closes on it).
|
||||
flags.append(False if s["ok"] is not None and name in s["ok"] else None)
|
||||
if verdict(flags, confirm, name in was_open):
|
||||
problems[f"{host.id}:{name}"] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "lesezugriff": name, "endpunkt": path,
|
||||
"fehlendes_recht": acl,
|
||||
"hinweis": f"Dem API-Benutzer fehlt das Recht, {path} zu lesen (OPNsense-Berechtigung \"{acl}\"). "
|
||||
"Alarme, die darauf beruhen, können nicht auslösen.",
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Resources ───────────────────────────────────────────────────────────────
|
||||
|
||||
def disk_groups(devices) -> dict[str, tuple[str, float]]:
|
||||
@@ -183,7 +238,9 @@ async def check_resource_threshold(db: AsyncSession) -> dict[str, dict]:
|
||||
|
||||
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
|
||||
if verdict(flags, confirm, key in was_open):
|
||||
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": f"{value}{unit}", "schwelle": f"{threshold}{unit}"}
|
||||
# An open alert stays open while the reading is unavailable: then there is no number to show (never "None %").
|
||||
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": None if value is None else f"{value}{unit}",
|
||||
"schwelle": f"{threshold}{unit}"}
|
||||
|
||||
latest = samples[0]
|
||||
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
|
||||
|
||||
@@ -224,14 +224,14 @@ async def send(channel: NotificationChannel, problem: NotificationProblem, trans
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
resp = await client.request(method, url, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
if ok_key and not (resp.json() or {}).get(ok_key):
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
|
||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
||||
reason = url_guard.failure_reason(exc)
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
|
||||
except ValueError:
|
||||
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
|
||||
|
||||
@@ -201,6 +201,62 @@ async def check_revert_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"host_name": host.name if host else None,
|
||||
"message": deployment.message,
|
||||
}
|
||||
problems.update(await _cleanup_problems(db))
|
||||
return problems
|
||||
|
||||
|
||||
# Rule cleanup (1.31) states that need a human: the automatic revert failed, or a run ended half-way.
|
||||
_CLEANUP_STILL_CHANGED = ("revert_failed", "unknown", "disabling", "disabled", "failed")
|
||||
|
||||
|
||||
async def _cleanup_problems(db: AsyncSession) -> dict[str, dict]:
|
||||
"""Rule cleanups whose automatic revert failed (per rule that could not be switched back) and runs stuck in
|
||||
``partial``/``apply_failed`` (a decision is needed before the window ends), plus delete/restore runs with an
|
||||
unresolved item. Same ``revert_failed`` alert as the deployments. Keys: ``rule_cleanup:<item id>`` for one rule,
|
||||
``rule_cleanup_op:<operation id>`` for a whole run, so one open problem is one message."""
|
||||
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
|
||||
|
||||
problems: dict[str, dict] = {}
|
||||
operations = (await db.scalars(select(RuleCleanupOperation).where(
|
||||
RuleCleanupOperation.status.in_(("revert_failed", "partial", "apply_failed", "failed"))))).all()
|
||||
for op in operations:
|
||||
items = list((await db.scalars(select(RuleCleanupItem).where(RuleCleanupItem.operation_id == op.id)
|
||||
.order_by(RuleCleanupItem.seq))).all())
|
||||
host = await db.get(OpnsenseHost, op.host_id)
|
||||
base = {
|
||||
"deployment_type": "rule_cleanup", "deployment_id": str(op.id), "host_id": str(op.host_id),
|
||||
"host_name": host.name if host else None, "operation": op.kind, "operation_status": op.status,
|
||||
}
|
||||
if op.status == "revert_failed":
|
||||
stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]
|
||||
for item in stuck:
|
||||
member = await db.get(OpnsenseHost, item.host_id)
|
||||
problems[f"rule_cleanup:{item.id}"] = {
|
||||
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||
"message": item.message or op.message,
|
||||
"hinweis": "Bereinigung: Die Regel konnte nicht automatisch zurückgesetzt werden und ist möglicherweise "
|
||||
"noch geändert. Auf der Firewall prüfen und in OPNexus „Erneut zurücknehmen“ wählen.",
|
||||
}
|
||||
if not stuck:
|
||||
problems[f"rule_cleanup_op:{op.id}"] = {**base, "entity_id": str(op.id), "message": op.message,
|
||||
"hinweis": "Bereinigung: Das automatische Zurücknehmen ist fehlgeschlagen. Zustand der Firewall prüfen."}
|
||||
elif op.status in ("partial", "apply_failed"):
|
||||
problems[f"rule_cleanup_op:{op.id}"] = {
|
||||
**base, "entity_id": str(op.id), "message": op.message,
|
||||
"regeln": ", ".join(i.rule_uuid for i in items if i.state in ("failed", "unknown")) or None,
|
||||
"hinweis": "Bereinigung nur teilweise ausgeführt (oder das Anwenden ist fehlgeschlagen). Bestätigen ist gesperrt; "
|
||||
"jetzt zurücknehmen oder die automatische Rücknahme am Fristende abwarten.",
|
||||
}
|
||||
elif op.kind in ("delete", "restore"):
|
||||
for item in (i for i in items if i.state == "unknown"):
|
||||
member = await db.get(OpnsenseHost, item.host_id)
|
||||
problems[f"rule_cleanup:{item.id}"] = {
|
||||
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||
"message": item.message or op.message,
|
||||
"hinweis": "Bereinigung: Der Ausgang beim Löschen/Wiederherstellen ist unklar. Mit „Erneut prüfen“ nachlesen.",
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
|
||||
@@ -40,6 +40,7 @@ _LABELS = {
|
||||
"gateway_down": "Gateway gestört",
|
||||
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
||||
"service_down": "Dienst ausgefallen",
|
||||
"api_rights_missing": "API-Rechte fehlen",
|
||||
"rollout_paused": "Rollout angehalten",
|
||||
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
||||
"test": "Testbenachrichtigung",
|
||||
@@ -96,14 +97,14 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
|
||||
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
||||
try:
|
||||
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
with url_guard.muted():
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which often is the credential itself
|
||||
# (token in path/query) — keep it out of messages and logs.
|
||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
||||
reason = url_guard.failure_reason(exc)
|
||||
raise NotificationDeliveryError(f"Webhook-Zustellung fehlgeschlagen ({reason})") from None
|
||||
|
||||
|
||||
|
||||
@@ -3,9 +3,11 @@ against previously open problems, and delivers on transitions. The single
|
||||
testable entry point — called directly by main.py's background loop AND
|
||||
directly by tests, same pattern as rule_deploy_service.sweep_expired_deployments.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||
@@ -37,6 +39,7 @@ _CHECKS = [
|
||||
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
|
||||
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
||||
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
||||
("api_rights_missing", "api_rights_enabled", monitor_checks.check_api_rights_missing),
|
||||
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
||||
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
||||
]
|
||||
@@ -46,44 +49,95 @@ _CHECKS = [
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
|
||||
|
||||
|
||||
# Delivery limits (1.29): a hanging channel must neither delay the others nor hold up the check loop.
|
||||
DELIVERY_TIMEOUT_SECONDS = 20.0 # one target (webhook, e-mail, one channel) for one message
|
||||
DELIVERY_CYCLE_BUDGET_SECONDS = 90.0 # everything one check cycle sends, in total
|
||||
DELIVERY_PARALLEL = 10 # targets served at the same time within a cycle
|
||||
|
||||
|
||||
async def _attempt(label: str, make_call, gate: asyncio.Semaphore | None = None, started: list | None = None) -> None:
|
||||
"""One delivery target: its own timeout, its own failure — never an exception for the caller. Texts logged
|
||||
here come from NotificationDeliveryError (names the channel and the HTTP status, never the URL or token)."""
|
||||
try:
|
||||
if gate is not None:
|
||||
await gate.acquire()
|
||||
if started is not None:
|
||||
started.append(label) # the target was really tried (a cancelled wait for the gate does not count)
|
||||
try:
|
||||
await asyncio.wait_for(make_call(), DELIVERY_TIMEOUT_SECONDS)
|
||||
finally:
|
||||
if gate is not None:
|
||||
gate.release()
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("%s delivery failed", label)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning("%s delivery timed out after %.0f s", label, DELIVERY_TIMEOUT_SECONDS)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc: # a broken target must not take the others down
|
||||
logger.warning("%s delivery failed unexpectedly (%s)", label, type(exc).__name__)
|
||||
|
||||
|
||||
async def _deliver(
|
||||
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
|
||||
gate: asyncio.Semaphore | None = None,
|
||||
) -> None:
|
||||
"""Best-effort, not an audit trail: attempted once per transition and
|
||||
never retried, even on failure — avoids retry-storming a broken webhook
|
||||
URL or SMTP relay every poll cycle."""
|
||||
URL or SMTP relay every poll cycle. Webhook, e-mail and every channel are
|
||||
served in parallel, each with its own timeout (the order has no meaning)."""
|
||||
if transition == "resolved" and problem.problem_type in _NO_RESOLVED_NOTICE:
|
||||
problem.resolved_notified_at = _now()
|
||||
return
|
||||
where = f"{problem.problem_type}/{problem.dedup_key}"
|
||||
calls = []
|
||||
started: list = []
|
||||
if settings_row.webhook_enabled:
|
||||
try:
|
||||
await notification_delivery.send_webhook(settings_row, problem, transition)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("Webhook delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
||||
calls.append(_attempt(f"Webhook for {where}", lambda: notification_delivery.send_webhook(settings_row, problem, transition), gate, started))
|
||||
if settings_row.email_enabled:
|
||||
try:
|
||||
async def email():
|
||||
subject, body = notification_delivery.render_message(problem, transition)
|
||||
await notification_delivery.send_email(settings_row, subject, body)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("Email delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
||||
|
||||
calls.append(_attempt(f"Email for {where}", email, gate, started))
|
||||
for channel in channels:
|
||||
try:
|
||||
await notification_channels.send(channel, problem, transition)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
# The error text names the channel and the HTTP status, never the URL or token.
|
||||
logger.exception("Channel %s (%s) delivery failed for %s/%s", channel.name, channel.channel_type, problem.problem_type, problem.dedup_key)
|
||||
calls.append(_attempt(f"Channel {channel.name} ({channel.channel_type}) for {where}",
|
||||
lambda channel=channel: notification_channels.send(channel, problem, transition), gate, started))
|
||||
try:
|
||||
await asyncio.gather(*calls, return_exceptions=True)
|
||||
finally:
|
||||
# "Attempted" is what is recorded (also when a target failed or timed out). A message the cycle budget
|
||||
# cancelled before ANY target was tried stays unrecorded: an opened problem is sent again next cycle.
|
||||
if started or not calls:
|
||||
if transition == "resolved":
|
||||
problem.resolved_notified_at = _now()
|
||||
else:
|
||||
problem.last_notified_at = _now()
|
||||
|
||||
if transition == "resolved":
|
||||
problem.resolved_notified_at = _now()
|
||||
else:
|
||||
problem.last_notified_at = _now()
|
||||
|
||||
async def _deliver_all(settings_row: NotificationSettings, pending: list[tuple[NotificationProblem, str]],
|
||||
channels: list[NotificationChannel]) -> None:
|
||||
"""Everything this cycle found, in parallel, within DELIVERY_CYCLE_BUDGET_SECONDS in total."""
|
||||
if not pending:
|
||||
return
|
||||
gate = asyncio.Semaphore(DELIVERY_PARALLEL)
|
||||
tasks = [asyncio.ensure_future(_deliver(settings_row, problem, transition, channels, gate)) for problem, transition in pending]
|
||||
done, late = await asyncio.wait(tasks, timeout=DELIVERY_CYCLE_BUDGET_SECONDS)
|
||||
for task in late:
|
||||
task.cancel()
|
||||
if late:
|
||||
logger.warning("Notification delivery stopped after %.0f s: %d of %d messages were not completed",
|
||||
DELIVERY_CYCLE_BUDGET_SECONDS, len(late), len(tasks))
|
||||
await asyncio.gather(*late, return_exceptions=True) # let the cancelled ones record "attempted"
|
||||
for task in done:
|
||||
if not task.cancelled() and task.exception() is not None:
|
||||
logger.warning("Notification delivery failed unexpectedly (%s)", type(task.exception()).__name__)
|
||||
|
||||
|
||||
async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
settings_row = await notification_settings_service.get_settings(db)
|
||||
result = {"opened": 0, "resolved": 0}
|
||||
channels = await notification_channels.enabled_channels(db)
|
||||
pending: list[tuple[NotificationProblem, str]] = []
|
||||
|
||||
for problem_type, enabled_flag, check_fn in _CHECKS:
|
||||
if not getattr(settings_row, enabled_flag):
|
||||
@@ -99,12 +153,21 @@ async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
continue
|
||||
|
||||
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
|
||||
for problem in opened:
|
||||
await _deliver(settings_row, problem, "opened", channels)
|
||||
for problem in resolved:
|
||||
await _deliver(settings_row, problem, "resolved", channels)
|
||||
pending += [(problem, "opened") for problem in opened] + [(problem, "resolved") for problem in resolved]
|
||||
result["opened"] += len(opened)
|
||||
result["resolved"] += len(resolved)
|
||||
|
||||
# Open problems that no target was ever tried for (an earlier cycle ran out of time) are sent now.
|
||||
enabled_types = [t for t, flag, _fn in _CHECKS if getattr(settings_row, flag)]
|
||||
already = {id(problem) for problem, _transition in pending}
|
||||
undelivered = (await db.scalars(
|
||||
select(NotificationProblem).where(
|
||||
NotificationProblem.status == "open", NotificationProblem.last_notified_at.is_(None),
|
||||
NotificationProblem.problem_type.in_(enabled_types),
|
||||
)
|
||||
)).all() if enabled_types else []
|
||||
pending += [(problem, "opened") for problem in undelivered if id(problem) not in already]
|
||||
# Delivered once all checks ran (and in parallel): a slow target no longer holds up the checks after it.
|
||||
await _deliver_all(settings_row, pending, channels)
|
||||
await db.commit()
|
||||
return result
|
||||
@@ -26,7 +26,7 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
|
||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
|
||||
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
||||
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled", "api_rights_enabled",
|
||||
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
||||
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
||||
"service_watchlist",
|
||||
|
||||
@@ -10,6 +10,10 @@ the deployment's last_payload. So the preview directly afterwards says "unchange
|
||||
updates the same object via setItem/setRule (no duplicate), and the normal preview → push →
|
||||
confirm/auto-revert flow applies from the first change on.
|
||||
|
||||
An object that already exists centrally for another firewall is LINKED instead of adopted a second time
|
||||
(adopt_link_service.py: exact match after resolving the host's variables, never automatic for a mere
|
||||
resemblance).
|
||||
|
||||
Only objects OPNexus can describe without distortion are adopted. Anything the model cannot express
|
||||
is refused with machine-readable reasons (the UI translates them) instead of being imported in a way
|
||||
that would silently change its meaning on the next edit. Fields OPNexus does not manage (log flag of a
|
||||
@@ -72,6 +76,7 @@ from app.services import alias_deploy_service, alias_policy_service
|
||||
from app.services import one_to_one_deploy_service, one_to_one_policy_service
|
||||
from app.services import opnsense_client, outbound_nat_deploy_service, outbound_nat_policy_service
|
||||
from app.services import port_forward_deploy_service, port_forward_policy_service
|
||||
from app.services import adopt_link_service
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on
|
||||
from app.services.nat_live_service import split_rows
|
||||
from app.services.outbound_nat_mode import host_mode
|
||||
@@ -85,6 +90,7 @@ ALIAS_CONTENT_MAX = 8000 # schemas/alias.py AliasCreate.content
|
||||
# OPNsense's own aliases that are not user objects (Firewall/Alias.php: bogons & co. are `external`
|
||||
# rows created by the system, __<if>_network are `internal`).
|
||||
SYSTEM_ALIAS_NAMES = frozenset({"bogons", "bogonsv6", "sshlockout", "virusprot"})
|
||||
LINK_TOLERATED = frozenset({"name_exists", "references_unmanaged_alias"})
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -93,6 +99,19 @@ class Entry:
|
||||
reasons: list[str] = field(default_factory=list)
|
||||
details: dict[str, str] = field(default_factory=dict)
|
||||
bulk: bool = False # not adoptable alone yet, but adoptable in the same run (its references come first)
|
||||
matches: list = field(default_factory=list) # central objects equal to the live object (link candidates)
|
||||
similar: list = field(default_factory=list) # same name/description but different content (hint only)
|
||||
|
||||
@property
|
||||
def linkable(self) -> bool:
|
||||
"""An exact central twin exists and nothing but "the name is taken" / "its references are not
|
||||
managed here yet" speaks against it (both are moot when the twin is linked, its content is equal)."""
|
||||
return any(not m.busy for m in self.matches) and set(self.reasons) <= LINK_TOLERATED
|
||||
|
||||
@property
|
||||
def busy_twin(self) -> bool:
|
||||
"""The one exact twin waits for a confirmation: \"adopt all\" leaves this entry alone (neither link nor duplicate)."""
|
||||
return len(self.matches) == 1 and self.matches[0].busy
|
||||
|
||||
@property
|
||||
def uuid(self) -> str:
|
||||
@@ -117,6 +136,14 @@ class Kind:
|
||||
context: Callable[[AsyncSession, list[dict]], Awaitable[dict]] | None = None
|
||||
refs: Callable[[dict], set[str]] | None = None
|
||||
name: Callable[[dict], str] | None = None
|
||||
identity: str = "description" # attribute that names the object (the alias name for aliases)
|
||||
|
||||
@property
|
||||
def link_spec(self) -> adopt_link_service.LinkSpec:
|
||||
return adopt_link_service.LinkSpec(
|
||||
model=self.model, deployment=self.deployment, fk=self.fk, uuid_attr=self.uuid_attr,
|
||||
identity=self.identity, payload=self.payload, revision=self.revision,
|
||||
)
|
||||
|
||||
|
||||
# ── shared helpers ───────────────────────────────────────────────────────
|
||||
@@ -388,7 +415,7 @@ KINDS: dict[str, Kind] = {
|
||||
uuid_attr="opnsense_alias_uuid", fetch=_fetch_aliases, check=_alias_check,
|
||||
title=lambda row: clean(row.get("name")), info=_alias_info, build=_alias_build,
|
||||
payload=alias_deploy_service._build_payload, revision=alias_policy_service.revision,
|
||||
context=_alias_context, refs=_alias_refs, name=lambda row: clean(row.get("name")),
|
||||
context=_alias_context, refs=_alias_refs, name=lambda row: clean(row.get("name")), identity="name",
|
||||
),
|
||||
"port_forward": Kind(
|
||||
key="port_forward", model=PortForwardRule, deployment=PortForwardDeployment, fk="port_forward_id",
|
||||
@@ -462,12 +489,16 @@ async def _plan(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> tuple[list[
|
||||
rows = await kind.fetch(host)
|
||||
managed = await _managed_uuids(db, kind, host)
|
||||
ctx = await kind.context(db, rows) if kind.context else {}
|
||||
central = await adopt_link_service.load_central(db, kind.link_spec)
|
||||
entries = []
|
||||
for row in rows:
|
||||
if clean(row.get("uuid")) in managed:
|
||||
continue
|
||||
reasons, details = kind.check(row, ctx)
|
||||
entries.append(Entry(row, list(reasons), dict(details)))
|
||||
entry = Entry(row, list(reasons), dict(details))
|
||||
if reasons != ["system"]:
|
||||
entry.matches, entry.similar = adopt_link_service.find(central, kind.link_spec, host, kind.build(row))
|
||||
entries.append(entry)
|
||||
if kind.refs:
|
||||
order = _resolve_references(kind, entries, rows, managed, ctx)
|
||||
else:
|
||||
@@ -475,8 +506,31 @@ async def _plan(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> tuple[list[
|
||||
return entries, order
|
||||
|
||||
|
||||
def _describe(kind: Kind, entry: Entry) -> dict:
|
||||
def _bulk_actions(entries: list[Entry], order: list[Entry], link_existing: bool) -> dict[str, tuple[str, Any]]:
|
||||
"""What "adopt all" does with each entry: ("link", central object) when exactly one central twin exists
|
||||
(and no other entry of this run took it), ("new", None), or nothing. One function for the dialog's
|
||||
summary and for the real run, so they cannot disagree. Several twins are never guessed between."""
|
||||
actions: dict[str, tuple[str, Any]] = {}
|
||||
taken: set = set()
|
||||
if link_existing:
|
||||
for entry in entries:
|
||||
if entry.linkable and len(entry.matches) == 1 and entry.matches[0].obj.id not in taken:
|
||||
actions[entry.uuid] = ("link", entry.matches[0].obj)
|
||||
taken.add(entry.matches[0].obj.id)
|
||||
for entry in order:
|
||||
if link_existing and entry.busy_twin:
|
||||
continue
|
||||
actions.setdefault(entry.uuid, ("new", None))
|
||||
return actions
|
||||
|
||||
|
||||
def _describe(kind: Kind, entry: Entry, action: str | None = None) -> dict:
|
||||
spec = kind.link_spec
|
||||
return {
|
||||
"matches": [adopt_link_service.describe(spec, m) for m in entry.matches],
|
||||
"similar": [adopt_link_service.describe(spec, m) for m in entry.similar],
|
||||
"linkable": entry.linkable,
|
||||
"bulk_action": action,
|
||||
"uuid": entry.uuid,
|
||||
"title": kind.title(entry.row),
|
||||
"enabled": on(entry.row.get("enabled")) if "disabled" not in entry.row else not on(entry.row.get("disabled")),
|
||||
@@ -491,8 +545,9 @@ def _describe(kind: Kind, entry: Entry) -> dict:
|
||||
|
||||
async def list_unmanaged(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> list[dict]:
|
||||
kind = KINDS[kind_key]
|
||||
entries, _ = await _plan(db, kind, host)
|
||||
return [_describe(kind, e) for e in entries]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
actions = _bulk_actions(entries, order, True)
|
||||
return [_describe(kind, e, actions.get(e.uuid, ("skip", None))[0]) for e in entries]
|
||||
|
||||
|
||||
def _create(db: AsyncSession, kind: Kind, host: OpnsenseHost, entry: Entry):
|
||||
@@ -511,8 +566,17 @@ async def _record(db: AsyncSession, kind: Kind, host: OpnsenseHost, obj, remote_
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid: str):
|
||||
async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid: str, link_to=None):
|
||||
"""Takes over one object as a NEW central object, or - with `link_to` - links it to that existing
|
||||
central object (adopt_link_service). Either way the live state is read fresh here."""
|
||||
kind = KINDS[kind_key]
|
||||
if link_to is not None:
|
||||
existing = await db.scalar(select(kind.deployment).where(
|
||||
kind.deployment.host_id == host.id, getattr(kind.deployment, kind.uuid_attr) == remote_uuid))
|
||||
if existing is not None:
|
||||
if getattr(existing, kind.fk) == link_to:
|
||||
return await db.get(kind.model, link_to) # second click: already linked, nothing to do
|
||||
raise NotAdoptable(["already_managed"])
|
||||
entries, _ = await _plan(db, kind, host)
|
||||
entry = next((e for e in entries if e.uuid == remote_uuid), None)
|
||||
if entry is None:
|
||||
@@ -520,6 +584,11 @@ async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid
|
||||
if remote_uuid in await _managed_uuids(db, kind, host):
|
||||
raise NotAdoptable(["already_managed"])
|
||||
raise NotFoundOnHost(remote_uuid)
|
||||
if link_to is not None:
|
||||
if not set(entry.reasons) <= LINK_TOLERATED:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = await adopt_link_service.link(db, kind.link_spec, host, link_to, remote_uuid, kind.build(entry.row))
|
||||
return obj
|
||||
if entry.reasons:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = _create(db, kind, host, entry)
|
||||
@@ -527,19 +596,32 @@ async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid
|
||||
return obj
|
||||
|
||||
|
||||
async def adopt_all(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> dict:
|
||||
async def adopt_all(db: AsyncSession, kind_key: str, host: OpnsenseHost, link_existing: bool = False) -> dict:
|
||||
"""Adopts everything adoptable in one transaction (the caller commits), nested aliases in
|
||||
dependency order. System objects are not reported; everything else that stays behind is, with reasons."""
|
||||
dependency order. With `link_existing`, entries with exactly one exact central twin are linked to it
|
||||
instead (see _bulk_actions). System objects are not reported; everything else that stays behind is,
|
||||
with reasons."""
|
||||
kind = KINDS[kind_key]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
actions = _bulk_actions(entries, order, link_existing)
|
||||
adopted = []
|
||||
done: set[str] = set()
|
||||
for entry in entries: # linked first: their names are central already, so nested aliases can refer to them
|
||||
action, target = actions.get(entry.uuid, ("skip", None))
|
||||
if action == "link":
|
||||
obj = await adopt_link_service.link(db, kind.link_spec, host, target.id, entry.uuid, kind.build(entry.row))
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": True})
|
||||
done.add(entry.uuid)
|
||||
for entry in order:
|
||||
if actions.get(entry.uuid, ("skip", None))[0] != "new":
|
||||
continue
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, entry.uuid)
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id})
|
||||
done = {e.uuid for e in order}
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": False})
|
||||
done.add(entry.uuid)
|
||||
skipped = [
|
||||
{"uuid": e.uuid, "title": kind.title(e.row), "reasons": e.reasons, "reason_details": e.details}
|
||||
{"uuid": e.uuid, "title": kind.title(e.row),
|
||||
"reasons": e.reasons + (["twin_busy"] if link_existing and e.busy_twin else []), "reason_details": e.details}
|
||||
for e in entries if e.uuid not in done and e.reasons != ["system"]
|
||||
]
|
||||
return {"adopted": adopted, "skipped": skipped}
|
||||
@@ -25,7 +25,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.oidc import OidcLoginState, OidcSettings
|
||||
from app.models.user import AdminUser
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -77,7 +77,7 @@ async def _get_json(url: str, **kw) -> dict:
|
||||
if not url.startswith("https://"):
|
||||
raise OidcError("provider", "Nur https:// wird akzeptiert")
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10, follow_redirects=False) as client:
|
||||
async with url_guard.guarded_client(timeout=10, follow_redirects=False) as client:
|
||||
response = await client.get(url, **kw)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
@@ -168,7 +168,7 @@ async def _exchange(cfg: OidcSettings, doc: dict, code: str, state: OidcLoginSta
|
||||
else:
|
||||
data["client_secret"] = secret
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=15, follow_redirects=False) as client:
|
||||
async with url_guard.guarded_client(timeout=15, follow_redirects=False) as client:
|
||||
response = await client.post(doc["token_endpoint"], data=data, auth=auth)
|
||||
if response.status_code >= 400:
|
||||
raise OidcError("token", f"Token-Endpunkt antwortet mit HTTP {response.status_code}")
|
||||
|
||||
@@ -135,6 +135,9 @@ async def sweep_expired_one_to_one_deployments(db: AsyncSession) -> list[OneToOn
|
||||
|
||||
|
||||
async def undeploy_one_to_one(db: AsyncSession, rule: OneToOneRule) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, OneToOneDeployment, OneToOneDeployment.one_to_one_id == rule.id)
|
||||
failures = []
|
||||
result = await db.execute(select(OneToOneDeployment).where(OneToOneDeployment.one_to_one_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -371,6 +371,9 @@ async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None
|
||||
for this entity to gate on) — deletes the Instance before its
|
||||
referenced StaticKey (the ModelRelationField dependency direction),
|
||||
then reconfigures."""
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, OpenVpnInstanceDeployment, OpenVpnInstanceDeployment.instance_id == instance.id)
|
||||
failures = []
|
||||
result = await db.execute(select(OpenVpnInstanceDeployment).where(OpenVpnInstanceDeployment.instance_id == instance.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -10,6 +10,7 @@ Auth is HTTP Basic: username = API key, password = API secret.
|
||||
"""
|
||||
import logging
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
@@ -33,6 +34,17 @@ class OpnsenseAmbiguousError(OpnsenseAPIError):
|
||||
outcome of a write is unknown and must be reconciled, not retried blindly."""
|
||||
|
||||
|
||||
class OpnsenseNotFoundError(OpnsenseAPIError):
|
||||
"""The firewall answered, but there is no such object (an empty `getRule` answer: unknown UUID, or a rule that is
|
||||
not part of the rules model - legacy or automatic). A refusal, never an unknown outcome."""
|
||||
|
||||
|
||||
class OpnsenseForbiddenError(OpnsenseAPIError):
|
||||
"""The firewall answered 403: the API key lacks the privilege for this endpoint. A refusal, not an unknown
|
||||
outcome; callers that only read can tell "no rights" from "unreachable". 401 (wrong key or secret) stays a
|
||||
plain OpnsenseAPIError."""
|
||||
|
||||
|
||||
def _client(host: OpnsenseHost) -> httpx.AsyncClient:
|
||||
# Before the secret is even read: an http:// host must never receive credentials.
|
||||
reason = firewall_url.check_base_url(host.base_url)
|
||||
@@ -71,7 +83,8 @@ async def _do_request(host: OpnsenseHost, method: str, path: str, **kwargs) -> h
|
||||
except httpx.HTTPStatusError as exc:
|
||||
# The body may reflect submitted secrets: keep only a scrubbed, bounded excerpt.
|
||||
secret = opnsense_secret_store.read_secret(host.id)
|
||||
error = OpnsenseAmbiguousError if exc.response.status_code >= 500 else OpnsenseAPIError
|
||||
status = exc.response.status_code
|
||||
error = OpnsenseAmbiguousError if status >= 500 else OpnsenseForbiddenError if status == 403 else OpnsenseAPIError
|
||||
raise error(
|
||||
f"{exc.response.status_code} {path}: {redact_text(exc.response.text, host.api_key, secret)}"
|
||||
) from exc
|
||||
@@ -234,6 +247,51 @@ def _mutation_result(data: Any, operation: str, results: set[str]) -> dict:
|
||||
return data
|
||||
|
||||
|
||||
# -- Filter rules (FilterController -> ApiMutableModelControllerBase, rules.rule) --------------------------------
|
||||
# Evidence for add/set/get/del/toggle/moveRuleBefore, checked 2026-10-22 against BOTH
|
||||
# source opnsense/core tags 26.7.7, 26.1.9 and 25.7.11, src/opnsense/mvc/app/controllers/OPNsense/:
|
||||
# Firewall/Api/FilterController.php (the *RuleAction methods, moveRuleBeforeAction, ittrRules),
|
||||
# Firewall/Api/FilterBaseController.php (applyAction),
|
||||
# Base/ApiMutableModelControllerBase.php (addBase, setBase, getBase, delBase, toggleBase)
|
||||
# live OPNsense 26.7.7 on two lab test firewalls (scratch rules whose description began with
|
||||
# `opnexus-b4-probe`, deleted again and applied afterwards; nothing was left behind).
|
||||
# Same in all three source lines unless said otherwise; the live checks cover 26.7.7 only.
|
||||
#
|
||||
# addRule POST, body {"rule": {...}}. Answer {"result": "saved", "uuid": "<new>"}; a validation failure is HTTP 200
|
||||
# {"result": "failed", "validations": {"rule.<field>": "<text>"}} (so _mutation_result raises).
|
||||
# `sequence` is a FilterSequenceField (models/OPNsense/Firewall/Filter.xml: 1..999999, default 1;
|
||||
# `sort_order` and `prio_group` are volatile, derived). Live: without `sequence` the rule gets the
|
||||
# highest sequence of ALL rules + 100 (566, 666, 766, ...), so a copy always lands at the END of its
|
||||
# prio_group; an explicit `sequence` IS honoured (sequence "3" -> sort_order 400000.0000003) and is not
|
||||
# checked for uniqueness. A restore can therefore either send the old number or call moveRuleBefore.
|
||||
# setRule POST /{uuid}, body {"rule": {...}}; answer {"result": "saved"}. It is an upsert (a well-formed unknown
|
||||
# UUID creates the rule) - never call it for a rule that may be gone.
|
||||
# delRule POST /{uuid}. {"result": "deleted"} or {"result": "not found"} (unknown, legacy and automatic UUIDs all
|
||||
# answer "not found", HTTP 200, nothing changes). 26.1 and 26.7 accept a comma-separated list; 25.7 a
|
||||
# single UUID. Consequence: "not found" proves nothing about a rule being gone - re-read it.
|
||||
# getRule GET /{uuid} -> {"rule": {...}} (every field of the rules model, select fields as
|
||||
# {"<option>": {"value": "<label>", "selected": 0|1}}, `categories` a select dict - or [] when the rule has
|
||||
# none -; no `uuid` key; read-only: `sequence`, `sort_order`, `prio_group` and the nested `audit`).
|
||||
# Live: an unknown, legacy (hex label) or automatic UUID answers HTTP 200 with a bare `[]`. 25.7 and 26.1
|
||||
# also know `?fetchmode=copy` (pre-fills `sequence`); 26.7.7 no longer has it - not used here.
|
||||
# Without a UUID it returns the template of a new rule (see get_rule_template above).
|
||||
# toggleRule POST /{uuid}/{enabled} - `enabled` is "0" or "1" and EXPLICIT; without it the rule flips (never do that).
|
||||
# Answer {"result": "Enabled"|"Disabled", "changed": true|false}. The explicit form is idempotent: repeating
|
||||
# it answers the same result with "changed": false (live: 0,0 -> true,false; 1,1 -> true,false). Any other
|
||||
# value answers {"result": "failed", "changed": false}; an unknown or legacy UUID, a GET, or a model that
|
||||
# has no such node answers {"result": "failed"} without "changed" - all HTTP 200. It only edits the
|
||||
# configuration; nothing reaches pf before apply() (the rule keeps its sort_order, searchRule shows
|
||||
# enabled "0"/"1" at once). 26.1/26.7 take a list of UUIDs (explicit value only); 25.7 one UUID.
|
||||
# moveRuleBefore POST /{selected}/{target}. {"status": "ok"} (no "result" key). Both rules must be in the same
|
||||
# `prio_group` - the sort_order prefix - which is NOT the interface: every single-interface rule of every
|
||||
# interface shares 400000 (lan and wan rules can be moved before each other), floating rules (several or
|
||||
# no interface) are 200000, interface-group rules 3000xx. Live: unknown UUID, same UUID twice and
|
||||
# different groups all answer HTTP 500 {"errorMessage": "Unexpected error, check log for details",
|
||||
# "errorTitle": "Filter"}, a GET answers 200 {"status": "error"}. A 500 is Ambiguous in _do_request, so a
|
||||
# caller that wants to tell "refused" from "unknown" must read sort_order back. The move only touches
|
||||
# `sequence` of the moved rule (and, when the gap is too small, of the rules behind the target); legacy
|
||||
# rules are not in the model and cannot be an anchor.
|
||||
# apply POST. {"status": "OK\n\n"} after `filter reload skip_alias`. FilterBaseController has no savepoint here.
|
||||
async def add_rule(host: OpnsenseHost, payload: dict) -> dict:
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/addRule", json={"rule": payload}), "Regel anlegen", {"saved"})
|
||||
|
||||
@@ -246,18 +304,93 @@ async def del_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||
return _mutation_result(await _request(host, "POST", f"/firewall/filter/delRule/{rule_uuid}"), "Regel entfernen", {"deleted", "not found"})
|
||||
|
||||
|
||||
_RULE_REF = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||
|
||||
|
||||
def _rule_ref(rule_uuid: str) -> str:
|
||||
"""A rule reference ends up in the URL path: refuse anything that could leave it."""
|
||||
if not isinstance(rule_uuid, str) or not _RULE_REF.match(rule_uuid):
|
||||
raise OpnsenseAPIError("Ungültige Regel-UUID")
|
||||
return rule_uuid
|
||||
|
||||
|
||||
async def get_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||
"""The full model record of one rule (`getRule/{uuid}`, the inner `rule` dict, select fields still in
|
||||
OPNsense's option form - reduce them with rule_write_payload()). A rule that is not in the rules model (unknown,
|
||||
legacy hex label, automatic) is answered with `[]`: raises OpnsenseNotFoundError."""
|
||||
data = await _request(host, "GET", f"/firewall/filter/getRule/{_rule_ref(rule_uuid)}")
|
||||
rule = data.get("rule") if isinstance(data, dict) else None
|
||||
if isinstance(rule, dict) and rule:
|
||||
return rule
|
||||
if data == [] or data == {}:
|
||||
raise OpnsenseNotFoundError(f"Regel {rule_uuid} gibt es auf {host.name} nicht (oder sie ist keine bearbeitbare Regel)")
|
||||
raise OpnsenseAPIError(f"Regel lesen: Keine auswertbare Antwort von {host.name}")
|
||||
|
||||
|
||||
async def toggle_rule(host: OpnsenseHost, rule_uuid: str, enabled: bool) -> dict:
|
||||
"""Sets the rule's `enabled` flag to exactly `enabled` (`toggleRule/{uuid}/1|0`; never the flipping form) and
|
||||
returns OPNsense's answer, e.g. {"result": "Disabled", "changed": False}. Repeating it is harmless. Only the
|
||||
configuration changes; call apply() afterwards. A refusal (unknown or legacy UUID) is a plain OpnsenseAPIError,
|
||||
a lost answer an OpnsenseAmbiguousError - then read the rule back before doing anything else."""
|
||||
if not isinstance(enabled, bool):
|
||||
raise TypeError("toggle_rule: 'enabled' must be a bool (an implicit flip would not be idempotent)")
|
||||
data = await _request(host, "POST", f"/firewall/filter/toggleRule/{_rule_ref(rule_uuid)}/{1 if enabled else 0}")
|
||||
return _mutation_result(data, "Regel ein-/ausschalten", {"enabled" if enabled else "disabled"})
|
||||
|
||||
|
||||
# Fields of a getRule record that a write must not send: the UUID is the rule's identity, `sequence` is assigned
|
||||
# on creation (sort_order and prio_group are derived from it), `audit` is metadata OPNsense maintains itself.
|
||||
RULE_READONLY_FIELDS = frozenset({"uuid", "sequence", "sort_order", "prio_group", "audit"})
|
||||
|
||||
|
||||
def rule_write_payload(rule: dict) -> dict[str, str]:
|
||||
"""Turns a getRule record into the flat form addRule/setRule accept: select fields become their selected
|
||||
option(s), comma-separated (`lan,wan`; none selected -> ""), the empty-list form of an empty multi-select
|
||||
becomes "", read-only fields are left out. Live round trip on 26.7.7 (getRule -> this -> addRule -> getRule)
|
||||
gave an identical record except uuid/sequence/sort_order/audit for single-interface, multi-interface (floating),
|
||||
no-interface (floating), interface-group, ICMP and any-protocol rules with categories, aliases, port ranges,
|
||||
TCP flags, tags and state limits."""
|
||||
out: dict[str, str] = {}
|
||||
for key, value in rule.items():
|
||||
if key in RULE_READONLY_FIELDS:
|
||||
continue
|
||||
if isinstance(value, dict):
|
||||
if value and all(isinstance(o, dict) and "selected" in o for o in value.values()):
|
||||
out[key] = ",".join(k for k, o in value.items() if o.get("selected") and k != "")
|
||||
# any other nested dict is metadata and is not sent
|
||||
elif isinstance(value, list):
|
||||
out[key] = ",".join(str(v) for v in value)
|
||||
else:
|
||||
out[key] = "" if value is None else str(value)
|
||||
return out
|
||||
|
||||
|
||||
async def apply(host: OpnsenseHost) -> dict:
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
|
||||
|
||||
|
||||
async def get_hasync(host: OpnsenseHost) -> dict:
|
||||
"""System > High Availability settings, read-only: `core/hasync/get` (Core/Api/HasyncController extends
|
||||
ApiMutableModelControllerBase, model Core/Hasync.xml; identical in opnsense/core 25.7.11, 26.1.9 and 26.7.7).
|
||||
Answer {"hasync": {"pfsyncinterface": {"": {"value": "Disabled", ...}, "lan": {"value": "LAN", "selected": 1},
|
||||
...}, "pfsyncpeerip": "<address or empty>", "synchronizetoip": ..., "syncitems": {...}}} - `pfsyncinterface` is an
|
||||
InterfaceField (the selected option, "" = disabled), `pfsyncpeerip` a NetworkField without mask. Live on 26.7.7
|
||||
(both nodes of the lab cluster): pfsyncinterface `lan`, a peer address, synchronizetoip empty. Used by the rule
|
||||
cleanup to keep pfsync rules out of reach (rule_analysis/protection.py); it reads nothing secret (the sync
|
||||
password field comes back, but is never stored or logged by the caller)."""
|
||||
return await _request(host, "GET", "/core/hasync/get")
|
||||
|
||||
|
||||
async def move_rule_before(host: OpnsenseHost, selected_uuid: str, target_uuid: str) -> dict:
|
||||
"""`firewall/filter/moveRuleBefore/{selected_uuid}/{target_uuid}`
|
||||
(FilterController::moveRuleBeforeAction) — repositions `selected_uuid`
|
||||
to sit directly before `target_uuid` via gap-numbered `sequence`
|
||||
floats, scoped to rules sharing the same `prio_group` (interface/type).
|
||||
Verified against OPNsense core source: no "move to top/bottom"
|
||||
sentinel exists, and both UUIDs must resolve to existing rules on the
|
||||
SAME interface/type or the controller throws. On success it returns
|
||||
floats, scoped to rules sharing the same `prio_group` (the sort_order
|
||||
prefix, see the evidence block above - not the interface). Verified against OPNsense core source and live
|
||||
(26.7.7): no "move to top/bottom"
|
||||
sentinel exists, and both UUIDs must resolve to existing rules in the
|
||||
SAME prio_group or the controller throws (HTTP 500, which this client
|
||||
reports as OpnsenseAmbiguousError). On success it returns
|
||||
`{"status": "ok"}` (confirmed against FilterController.php directly —
|
||||
unlike addRule/setRule/delRule this has no "result" key at all), so
|
||||
this is dispatched through _mutation_result with an empty `results`
|
||||
@@ -435,6 +568,11 @@ async def get_pf_rule_statistics(host: OpnsenseHost) -> dict:
|
||||
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
|
||||
# wireguard) appear with their name as uuid and empty members.
|
||||
async def get_rule_stats(host: OpnsenseHost) -> dict:
|
||||
"""Per-label counters, already summed over every pf line of a rule. Used for the rule usage sampling
|
||||
(rule_hit_service). Live check 2026-10-22 on all three test firewalls (26.7, one with unmigrated legacy
|
||||
rules): HTTP 200, ``stats`` has exactly one key per label of ``pfStatistics/rules`` (30/30/25 labels,
|
||||
including the 32-hex label of the legacy user rule on the third), and each ``packets`` equals the sum over
|
||||
that label's pf lines (differences only by traffic between the two reads)."""
|
||||
return await _request(host, "GET", "/firewall/filter_util/ruleStats")
|
||||
|
||||
|
||||
@@ -1316,6 +1454,13 @@ async def create_snapshot(host: OpnsenseHost, name: str, note: str = "") -> None
|
||||
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
|
||||
|
||||
|
||||
async def activate_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||
"""Boot environment for every following start (bectl activate, not -t); takes effect at the next reboot."""
|
||||
data = await _request(host, "POST", f"/core/snapshots/activate/{uuid}")
|
||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||
raise OpnsenseAPIError(f"Snapshot aktivieren von Firewall nicht bestätigt: {redact_data(data)}")
|
||||
|
||||
|
||||
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
|
||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||
|
||||
@@ -147,6 +147,9 @@ async def sweep_expired_outbound_nat_deployments(db: AsyncSession) -> list[Outbo
|
||||
|
||||
|
||||
async def undeploy_outbound_nat(db: AsyncSession, rule: OutboundNatRule) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, OutboundNatDeployment, OutboundNatDeployment.outbound_nat_id == rule.id)
|
||||
failures = []
|
||||
result = await db.execute(select(OutboundNatDeployment).where(OutboundNatDeployment.outbound_nat_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -185,6 +185,9 @@ async def reassert_group_ordering(db: AsyncSession, group: PolicyHierarchyGroup)
|
||||
for h in g.hosts:
|
||||
hosts[h.id] = h
|
||||
errors: list[str] = []
|
||||
# moveRuleBefore changes rule positions: not while a rule cleanup (1.31) owns one of the firewalls.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [h for h in hosts.values() if h.enabled])
|
||||
for host in sorted(hosts.values(), key=lambda h: (h.name, str(h.id))):
|
||||
if not host.enabled:
|
||||
continue
|
||||
@@ -300,9 +303,13 @@ async def sweep_expired_hierarchy_deployments(db: AsyncSession) -> list[PolicyHi
|
||||
|
||||
|
||||
async def undeploy_hierarchy_rule(db: AsyncSession, rule: PolicyHierarchyRule) -> None:
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
failures = []
|
||||
result = await db.execute(select(PolicyHierarchyRuleDeployment).where(PolicyHierarchyRuleDeployment.policy_hierarchy_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
deployments = list(result.scalars().all())
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them.
|
||||
await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])
|
||||
for deployment in deployments:
|
||||
host = await db.get(OpnsenseHost, deployment.host_id)
|
||||
if host and deployment.opnsense_policy_hierarchy_uuid:
|
||||
try:
|
||||
|
||||
@@ -147,6 +147,9 @@ async def sweep_expired_port_forward_deployments(db: AsyncSession) -> list[PortF
|
||||
|
||||
|
||||
async def undeploy_port_forward(db: AsyncSession, rule: PortForwardRule) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, PortForwardDeployment, PortForwardDeployment.port_forward_id == rule.id)
|
||||
failures = []
|
||||
result = await db.execute(select(PortForwardDeployment).where(PortForwardDeployment.port_forward_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -17,7 +17,7 @@ from app.models.host import OpnsenseHost
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.opnsense_client import OpnsenseAPIError, OpnsenseForbiddenError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -91,10 +91,19 @@ def _services_compact(listing) -> dict[str, int]:
|
||||
return result
|
||||
|
||||
|
||||
async def _optional(operation):
|
||||
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing)."""
|
||||
async def _optional(operation, name: str, denied: list[str], ok: list[str]):
|
||||
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing).
|
||||
A refusal by the firewall (403: the API key lacks the privilege) is recorded in ``denied`` so the
|
||||
missing right becomes visible; a reading that went through is recorded in ``ok`` (the proof that a right is
|
||||
back, whatever the answer contains); any other failure stays "unknown" and is no finding about rights."""
|
||||
try:
|
||||
return await operation
|
||||
result = await operation
|
||||
ok.append(name)
|
||||
return result
|
||||
except OpnsenseForbiddenError:
|
||||
denied.append(name)
|
||||
logger.debug("Optionale Messung %s: API-Recht fehlt", name)
|
||||
return None
|
||||
except Exception as exc:
|
||||
logger.debug("Optionale Messung fehlgeschlagen: %s", type(exc).__name__)
|
||||
return None
|
||||
@@ -105,11 +114,13 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
system_time = await client.get_system_time(host)
|
||||
disk = await client.get_system_disk(host)
|
||||
interfaces = await client.get_interface_statistics(host)
|
||||
denied: list[str] = []
|
||||
ok: list[str] = []
|
||||
swap, cpu_type, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_system_swap(host)),
|
||||
_optional(client.get_cpu_type(host)),
|
||||
_optional(gateway_health_service.get_gateway_health(host)),
|
||||
_optional(client.search_services(host)),
|
||||
_optional(client.get_system_swap(host), "swap", denied, ok),
|
||||
_optional(client.get_cpu_type(host), "cpu", denied, ok),
|
||||
_optional(gateway_health_service.get_gateway_health(host), "gateways", denied, ok),
|
||||
_optional(client.search_services(host), "services", denied, ok),
|
||||
)
|
||||
db.add(
|
||||
HostResourceSample(
|
||||
@@ -125,6 +136,9 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
"cpu_threads": _cpu_threads(cpu_type) if cpu_type is not None else None,
|
||||
"gateways": gateway_health_service.compact(gateways) if gateways is not None else None,
|
||||
"services": _services_compact(services) if services is not None else None,
|
||||
# Readings the firewall refused (403) because the API key lacks the right; always a list.
|
||||
"denied": sorted(denied),
|
||||
"ok": sorted(ok),
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
@@ -139,9 +139,14 @@ def pause(run: RolloutRun, reason: str, **detail) -> None:
|
||||
_progress(run, paused_at=_iso(_now()))
|
||||
|
||||
|
||||
async def may_roll_out(db: AsyncSession, username: str) -> bool:
|
||||
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout."""
|
||||
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
|
||||
async def may_roll_out(db: AsyncSession, username: str, user_id=None) -> bool:
|
||||
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout.
|
||||
With the stable account id (1.29) the id decides: a deleted account that was re-created under the same name is
|
||||
someone else. Rows from before the id existed (and still without one) are matched by name as before."""
|
||||
if user_id is not None:
|
||||
user = await db.get(AdminUser, user_id)
|
||||
else:
|
||||
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
|
||||
if user is None:
|
||||
return False
|
||||
if user.role == "admin":
|
||||
@@ -184,17 +189,18 @@ async def active_run(db: AsyncSession) -> RolloutRun | None:
|
||||
|
||||
|
||||
async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
|
||||
occurrence: datetime | None = None) -> RolloutRun:
|
||||
occurrence: datetime | None = None, started_by_id=None) -> RolloutRun:
|
||||
"""Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
|
||||
now = _now()
|
||||
snapshot = {
|
||||
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
||||
"window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
||||
}
|
||||
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
||||
current_wave=0, started_by=started_by, trigger=trigger, window_occurrence=occurrence,
|
||||
current_wave=0, started_by=started_by, started_by_id=started_by_id, trigger=trigger, window_occurrence=occurrence,
|
||||
progress={}, created_at=now, step_at=now, revision=0)
|
||||
run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
|
||||
for w in resolved["waves"]]
|
||||
@@ -329,7 +335,7 @@ async def _sync_hosts(db: AsyncSession, run: RolloutRun) -> bool:
|
||||
# ── handlers: one per run status; each returns an apply-function or None ──────
|
||||
|
||||
async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if await may_roll_out(db, run.started_by):
|
||||
if await may_roll_out(db, run.started_by, run.started_by_id):
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
@@ -376,7 +382,8 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
||||
baselines.update((w.baseline or {}))
|
||||
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||
probes=run.plan_snapshot.get("gate_probes"))
|
||||
|
||||
async def apply(db, run):
|
||||
if gate["verdict"] == "fail":
|
||||
@@ -402,21 +409,30 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
||||
return apply
|
||||
|
||||
|
||||
# The core package of each OPNsense release type (community, development, business edition). Switching the type
|
||||
# installs another core package: live 2026-10-23, community -> devel offered `opnsense-devel` 27.1.a_483 as a NEW
|
||||
# package while product_latest still said 26.7.7 (the rollout expected 26.7.7 and stopped at the gate).
|
||||
CORE_PACKAGES = ("opnsense", "opnsense-devel", "opnsense-business")
|
||||
|
||||
|
||||
def target_of(status: dict, action: str) -> str | None:
|
||||
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
|
||||
|
||||
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
|
||||
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
|
||||
core package's new version comes first; product_latest is only a fallback."""
|
||||
core package's new version comes first (whichever release type it is, see CORE_PACKAGES, also when it
|
||||
arrives as a new package); product_latest is only a fallback."""
|
||||
if action == "major_update":
|
||||
return status.get("upgrade_major_version") or None
|
||||
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
|
||||
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
|
||||
if core and core.get("new") and core.get("new") != "N/A":
|
||||
return str(core["new"])
|
||||
for row in status.get("upgrade_packages") or []:
|
||||
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
|
||||
return str(row["new_version"])
|
||||
for name in CORE_PACKAGES:
|
||||
core = packages.get(name) if isinstance(packages.get(name), dict) else None
|
||||
if core and core.get("new") and core.get("new") != "N/A":
|
||||
return str(core["new"])
|
||||
for key, field in (("upgrade_packages", "new_version"), ("new_packages", "version")):
|
||||
for row in status.get(key) or []:
|
||||
if isinstance(row, dict) and row.get("name") in CORE_PACKAGES and row.get(field):
|
||||
return str(row[field])
|
||||
product = status.get("product") if isinstance(status.get("product"), dict) else {}
|
||||
return product.get("product_latest") or status.get("product_latest") or None
|
||||
|
||||
@@ -532,7 +548,7 @@ async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | Non
|
||||
versions = {}
|
||||
for rh_id in offers:
|
||||
host = usable[rh_id]
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host)
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host, run.plan_snapshot.get("gate_probes"))
|
||||
versions[rh_id] = baselines[str(host.id)].get("version")
|
||||
roles = await get_statuses([usable[i] for i in offers])
|
||||
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
||||
@@ -625,6 +641,7 @@ async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision |
|
||||
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
||||
gate = await rollout_health.evaluate(
|
||||
db, done_peers + [host], baselines, target_version=run.target_version,
|
||||
# No probes here: the peers' probes were checked in their own gate, and this runs on every tick.
|
||||
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
||||
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
||||
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
||||
@@ -713,7 +730,8 @@ async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||
probes=run.plan_snapshot.get("gate_probes"))
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, index)
|
||||
@@ -769,7 +787,7 @@ async def auto_start(db: AsyncSession) -> int:
|
||||
problem = _note("auto_start_update_only")
|
||||
elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
|
||||
problem = _note("plan_changed") # someone changed the plan, its groups or its window since activation
|
||||
elif not await may_roll_out(db, plan.activated_by or ""):
|
||||
elif not await may_roll_out(db, plan.activated_by or "", plan.activated_by_id):
|
||||
problem = _note("actor_invalid", user=plan.activated_by)
|
||||
elif resolved["errors"]:
|
||||
problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
|
||||
@@ -786,7 +804,8 @@ async def auto_start(db: AsyncSession) -> int:
|
||||
audit_service.context.reset(token)
|
||||
continue
|
||||
try:
|
||||
run = await create_run(db, plan, resolved, started_by=plan.activated_by, trigger="window", occurrence=occ.start)
|
||||
run = await create_run(db, plan, resolved, started_by=plan.activated_by, started_by_id=plan.activated_by_id,
|
||||
trigger="window", occurrence=occ.start)
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
|
||||
@@ -18,13 +18,19 @@ the configuration like the 6-hourly security check does):
|
||||
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
||||
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
||||
update may legitimately change defaults.
|
||||
* probes — optional per plan (1.32): ping/TCP probes run FROM the firewall (uplink, DNS, an internal
|
||||
server). A probe that worked in the baseline and fails now fails the gate; one that failed
|
||||
already before only warns. This is reachability from the firewall, not traffic through it.
|
||||
|
||||
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
||||
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
||||
"""
|
||||
import asyncio
|
||||
import uuid
|
||||
from collections import defaultdict
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -41,6 +47,8 @@ DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": Tru
|
||||
"compliance": False}
|
||||
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
||||
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
||||
MAX_PROBES = 10
|
||||
PROBE_PING_COUNT = 3
|
||||
|
||||
|
||||
def _r(result: str, code: str, **params) -> dict:
|
||||
@@ -64,7 +72,55 @@ async def _services(host) -> dict | None:
|
||||
return None if listing is None else _services_compact(listing)
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
def _probe_key(probe: dict) -> str:
|
||||
return f"{probe['type']} {probe['target']}" + (f":{probe['port']}" if probe.get("port") else "")
|
||||
|
||||
|
||||
async def run_probes(host: OpnsenseHost, probes: list[dict] | None) -> list[dict]:
|
||||
"""Every probe from the firewall, one after the other; a probe that cannot run counts as failed. Audited by
|
||||
diagnostics_service like a probe from the host page, as `system:rollout` when no request context exists."""
|
||||
from app.services import audit_service, diagnostics_service
|
||||
|
||||
if not probes:
|
||||
return []
|
||||
token = None
|
||||
if audit_service.context.get() is None:
|
||||
token = audit_service.context.set({"actor": "system:rollout", "id": str(uuid.uuid4()), "action": "probe"})
|
||||
try:
|
||||
async def one(probe: dict) -> dict:
|
||||
ipproto = "inet6" if ":" in probe["target"] else "inet"
|
||||
try:
|
||||
if probe["type"] == "ping":
|
||||
result = await diagnostics_service.ping(host, probe["target"], ipproto, PROBE_PING_COUNT, None, None)
|
||||
ok = int(result.get("received") or 0) > 0
|
||||
else:
|
||||
result = await diagnostics_service.portprobe(host, probe["target"], int(probe["port"]), ipproto, None)
|
||||
ok = bool(result.get("open"))
|
||||
except (HTTPException, client.OpnsenseAPIError):
|
||||
ok = False
|
||||
return {"key": _probe_key(probe), "ok": ok}
|
||||
|
||||
# Concurrently (diagnostics_service caps it at a few at a time per process), results in plan order.
|
||||
return list(await asyncio.gather(*(one(p) for p in probes[:MAX_PROBES])))
|
||||
finally:
|
||||
if token is not None:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _check_probes(before: list | None, now: list | None) -> dict:
|
||||
if now is None:
|
||||
return _r("fail", "unreadable")
|
||||
was = {p["key"]: p["ok"] for p in (before or [])}
|
||||
regressed = [p["key"] for p in now if not p["ok"] and was.get(p["key"], True)]
|
||||
still = [p["key"] for p in now if not p["ok"] and was.get(p["key"]) is False]
|
||||
if regressed:
|
||||
return _r("fail", "probe_regressed", probes=", ".join(regressed))
|
||||
if still:
|
||||
return _r("warn", "probe_was_failing", probes=", ".join(still))
|
||||
return _r("ok", "ok", count=len(now))
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost, probes: list[dict] | None = None) -> dict:
|
||||
info, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
||||
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
||||
@@ -73,6 +129,7 @@ async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
"gateways": gateways,
|
||||
"services": services,
|
||||
"compliance_score": score,
|
||||
"probes": await run_probes(host, probes) if probes else None,
|
||||
}
|
||||
|
||||
|
||||
@@ -137,7 +194,7 @@ async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dic
|
||||
|
||||
|
||||
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
||||
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
|
||||
checks: dict | None = None, compliance_mode: str = "warn", probes: list[dict] | None = None) -> dict:
|
||||
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
||||
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
||||
for name in ALWAYS_ON:
|
||||
@@ -190,6 +247,10 @@ async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict,
|
||||
if enabled["services"]:
|
||||
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
||||
|
||||
if probes:
|
||||
# A probe new since the baseline (plan edited mid-run) has nothing to compare with: treated as "worked before".
|
||||
row["probes"] = _check_probes(before.get("probes"), await run_probes(host, probes) if reachable else None)
|
||||
|
||||
blockers = await _shared_change_blockers(db, fleets[hid])
|
||||
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
||||
|
||||
|
||||
@@ -110,6 +110,8 @@ def definition(plan: RolloutPlan) -> dict:
|
||||
return {
|
||||
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
# Only when set: an activated plan from before 1.32 must keep its fingerprint (else its automatic start stops).
|
||||
**({"gate_probes": list(plan.gate_probes)} if plan.gate_probes else {}),
|
||||
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
||||
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.services import opnsense_client
|
||||
from app.services import adopt_link_service, opnsense_client
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on # noqa: F401 (NotAdoptable is re-exported for routers/rules.py)
|
||||
from app.services.policy_service import revision
|
||||
from app.services.rule_deploy_service import _build_payload
|
||||
@@ -64,6 +64,30 @@ def reasons_not_adoptable(row: dict, managed_uuids: set[str]) -> list[str]:
|
||||
return reasons
|
||||
|
||||
|
||||
def _values(row: dict) -> dict:
|
||||
"""The fields OPNexus manages, normalised exactly as adopting stores them (also what a link compares)."""
|
||||
return {
|
||||
"description": _clean(row.get("description")),
|
||||
"interface": _clean(row.get("interface")),
|
||||
"direction": _clean(row.get("direction")),
|
||||
"ipprotocol": _clean(row.get("ipprotocol")),
|
||||
"protocol": _clean(row.get("protocol")) or "any",
|
||||
"action": _clean(row.get("action")),
|
||||
"source_net": _clean(row.get("source_net")) or "any",
|
||||
"source_port": _clean(row.get("source_port")) or None,
|
||||
"destination_net": _clean(row.get("destination_net")) or "any",
|
||||
"destination_port": _clean(row.get("destination_port")) or None,
|
||||
"enabled": _on(row.get("enabled")),
|
||||
"log": _on(row.get("log")),
|
||||
}
|
||||
|
||||
|
||||
LINK_SPEC = adopt_link_service.LinkSpec(
|
||||
model=FirewallRule, deployment=RuleDeployment, fk="rule_id", uuid_attr="opnsense_rule_uuid",
|
||||
identity="description", payload=_build_payload, revision=revision,
|
||||
)
|
||||
|
||||
|
||||
async def _managed_uuids(db: AsyncSession, host: OpnsenseHost) -> set[str]:
|
||||
rows = await db.scalars(
|
||||
select(RuleDeployment.opnsense_rule_uuid).where(
|
||||
@@ -81,6 +105,7 @@ async def _live_rows(host: OpnsenseHost) -> list[dict]:
|
||||
|
||||
async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
|
||||
managed = await _managed_uuids(db, host)
|
||||
central = await adopt_link_service.load_central(db, LINK_SPEC)
|
||||
result = []
|
||||
for row in await _live_rows(host):
|
||||
if row.get("is_automatic"):
|
||||
@@ -88,7 +113,11 @@ async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
|
||||
reasons = reasons_not_adoptable(row, managed)
|
||||
if "already_managed" in reasons:
|
||||
continue
|
||||
matches, similar = adopt_link_service.find(central, LINK_SPEC, host, _values(row))
|
||||
result.append({
|
||||
"matches": [adopt_link_service.describe(LINK_SPEC, m) for m in matches],
|
||||
"similar": [adopt_link_service.describe(LINK_SPEC, m) for m in similar],
|
||||
"linkable": any(not m.busy for m in matches) and not reasons,
|
||||
"uuid": _clean(row.get("uuid")),
|
||||
"description": _clean(row.get("description")),
|
||||
"interface": _clean(row.get("interface")),
|
||||
@@ -108,7 +137,15 @@ async def list_unmanaged(db: AsyncSession, host: OpnsenseHost) -> list[dict]:
|
||||
return result
|
||||
|
||||
|
||||
async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str) -> FirewallRule:
|
||||
async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str, link_to=None) -> FirewallRule:
|
||||
"""Takes the rule over as a new central rule, or - with `link_to` - links it to that existing rule."""
|
||||
if link_to is not None:
|
||||
existing = await db.scalar(select(RuleDeployment).where(
|
||||
RuleDeployment.host_id == host.id, RuleDeployment.opnsense_rule_uuid == rule_uuid))
|
||||
if existing is not None:
|
||||
if existing.rule_id == link_to:
|
||||
return await db.get(FirewallRule, link_to) # second click: already linked
|
||||
raise NotAdoptable(["already_managed"])
|
||||
managed = await _managed_uuids(db, host)
|
||||
row = next((r for r in await _live_rows(host) if _clean(r.get("uuid")) == rule_uuid), None)
|
||||
if row is None:
|
||||
@@ -116,23 +153,10 @@ async def adopt(db: AsyncSession, host: OpnsenseHost, rule_uuid: str) -> Firewal
|
||||
reasons = reasons_not_adoptable(row, managed)
|
||||
if reasons:
|
||||
raise NotAdoptable(reasons)
|
||||
if link_to is not None:
|
||||
return await adopt_link_service.link(db, LINK_SPEC, host, link_to, rule_uuid, _values(row))
|
||||
|
||||
rule = FirewallRule(
|
||||
description=_clean(row.get("description")),
|
||||
interface=_clean(row.get("interface")),
|
||||
direction=_clean(row.get("direction")),
|
||||
ipprotocol=_clean(row.get("ipprotocol")),
|
||||
protocol=_clean(row.get("protocol")) or "any",
|
||||
action=_clean(row.get("action")),
|
||||
source_net=_clean(row.get("source_net")) or "any",
|
||||
source_port=_clean(row.get("source_port")) or None,
|
||||
destination_net=_clean(row.get("destination_net")) or "any",
|
||||
destination_port=_clean(row.get("destination_port")) or None,
|
||||
enabled=_on(row.get("enabled")),
|
||||
log=_on(row.get("log")),
|
||||
targets=[host],
|
||||
groups=[],
|
||||
)
|
||||
rule = FirewallRule(**_values(row), targets=[host], groups=[])
|
||||
db.add(rule)
|
||||
await db.flush()
|
||||
revision(db, rule, "Von Firewall übernommen")
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Cleanup candidates of one firewall: unused, shadowed, duplicate and long-disabled rules (read-only).
|
||||
|
||||
Nothing here changes a firewall. The list says *which rules are worth a look and how sure that is*; removing
|
||||
them is a separate step (not part of this module).
|
||||
|
||||
Categories
|
||||
* ``unused`` OPNexus has watched the rule's pf counters for at least N days and saw no hit
|
||||
(services/rule_hit_service.py; N is a parameter, default 30).
|
||||
* ``shadowed`` an earlier quick rule already decides everything the rule matches (rule_analysis/coverage.py).
|
||||
* ``duplicate`` like shadowed, with the same action and the rule also covering the earlier one.
|
||||
* ``disabled_long`` disabled, and OPNexus has seen it unchanged for N days or more (rule history).
|
||||
|
||||
Confidence (never higher than what is proven; what is not proven is never "sure")
|
||||
* ``sure`` the evidence is complete: unused = the rule is loaded in pf, has no schedule, the counters were
|
||||
sampled recently and over the whole window; shadowed/duplicate = literal comparison, order from
|
||||
``sort_order``, the covering rule is loaded; disabled_long = observed unchanged for N days.
|
||||
* ``probable`` relies on a guess or on static alias content, or pf's loaded ruleset is not available, or the rule
|
||||
has a schedule (a hit might happen at another time), or (cluster) some node has no usage data for
|
||||
the rule. In a cluster "unused" is judged over all nodes (rule_hit_service.merge_cluster_hits):
|
||||
the CARP backup sees almost no traffic, so its zero alone never makes a rule unused.
|
||||
* ``unknown`` there is a hint but no support: the rule is not loaded in pf (no counters can exist), or the
|
||||
counters are stale.
|
||||
|
||||
``removable`` says whether the rule could be offered for removal at all; the reason is a code:
|
||||
``automatic`` (generated by OPNsense), ``legacy`` (not yet migrated, migrate it in OPNsense first),
|
||||
``managed`` (OPNexus manages the rule: remove it through the normal flow of the rules page).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Iterable
|
||||
|
||||
from app.services import rule_hit_service
|
||||
from app.services.rule_analysis import coverage
|
||||
from app.services.rule_analysis.context import Context
|
||||
from app.services.rule_analysis.model import SECTION_ANTISPOOF, SECTION_INVALID, SECTION_NAT_AUTO, NormRule
|
||||
|
||||
CATEGORIES = ("unused", "shadowed", "duplicate", "disabled_long")
|
||||
DEFAULT_UNUSED_DAYS = 30
|
||||
MIN_DAYS, MAX_DAYS = 1, 3650
|
||||
|
||||
|
||||
def removal_block(rule: NormRule, managed: dict | None) -> str | None:
|
||||
"""Why the rule could not be offered for removal (None = it could)."""
|
||||
if rule.automatic or rule.section in (SECTION_ANTISPOOF, SECTION_NAT_AUTO):
|
||||
return "automatic"
|
||||
if rule.legacy:
|
||||
return "legacy"
|
||||
if managed is not None:
|
||||
return "managed"
|
||||
return None
|
||||
|
||||
|
||||
def unused_confidence(rule: NormRule, hit, now: datetime) -> str:
|
||||
"""Confidence of an 'unused' verdict (the caller already knows the window passed without a hit)."""
|
||||
if rule.loaded is False:
|
||||
return "unknown" # pf did not load it (schedule, missing interface, gateway down): no counter can exist
|
||||
if hit.last_sampled_at is None or now - hit.last_sampled_at > rule_hit_service.SAMPLE_FRESH:
|
||||
return "unknown" # stale counters say nothing about the present
|
||||
if rule.loaded is None or rule.sched:
|
||||
return "probable" # pf's ruleset not available / the rule only works at certain times
|
||||
if getattr(hit, "peer_missing", False):
|
||||
return "probable" # cluster: some node has no usage data for this rule (rule_hit_service.ClusterHit)
|
||||
return "sure"
|
||||
|
||||
|
||||
def _name(rule: NormRule) -> str:
|
||||
return rule.description or rule.label or rule.uuid or (f"@{rule.pf_numbers[0]}" if rule.pf_numbers else "")
|
||||
|
||||
|
||||
def build(rules: list[NormRule], ctx: Context, *, hits: dict, states: dict, managed: dict[str, dict],
|
||||
now: datetime, unused_days: int = DEFAULT_UNUSED_DAYS, order_guessed: bool = False) -> list[dict]:
|
||||
"""Candidate rows, in evaluation order. ``hits``: RuleHit per rule label, ``states``: LiveRuleState per label,
|
||||
``managed``: {label: {"kind", "id", "name"}} of the rules OPNexus deployed to this firewall."""
|
||||
unused_days = max(MIN_DAYS, min(MAX_DAYS, int(unused_days)))
|
||||
overlaps = {id(o.rule): o for o in coverage.analyse(rules, ctx, order_guessed=order_guessed)}
|
||||
rows: list[dict] = []
|
||||
for rule in rules:
|
||||
if rule.automatic or rule.section in (SECTION_ANTISPOOF, SECTION_INVALID, SECTION_NAT_AUTO):
|
||||
continue
|
||||
categories: list[dict] = []
|
||||
key = rule.uuid.lower()
|
||||
hit = hits.get(key)
|
||||
if hit is not None and rule_hit_service.classify(hit, now, unused_days) == "unused":
|
||||
category = {"kind": "unused", "confidence": unused_confidence(rule, hit, now), "days": unused_days}
|
||||
if getattr(hit, "peer_missing", False):
|
||||
category["notes"] = ["CLUSTER_PEER_USAGE_UNKNOWN"]
|
||||
categories.append(category)
|
||||
overlap = overlaps.get(id(rule))
|
||||
if overlap is not None:
|
||||
categories.append({
|
||||
"kind": overlap.kind, "confidence": overlap.confidence,
|
||||
"by": {"uuid": overlap.by.uuid, "description": _name(overlap.by), "section": overlap.by.section,
|
||||
"automatic": overlap.by.automatic},
|
||||
})
|
||||
state = states.get(key)
|
||||
if (not rule.enabled and state is not None and state.present
|
||||
and now - state.changed_at >= timedelta(days=unused_days)):
|
||||
categories.append({"kind": "disabled_long", "confidence": "sure", "days": unused_days,
|
||||
"since": state.changed_at})
|
||||
if not categories:
|
||||
continue
|
||||
block = removal_block(rule, managed.get(key))
|
||||
rows.append({
|
||||
**rule.summary(), "enabled": rule.enabled, "categories": categories,
|
||||
"hit_state": rule_hit_service.classify(hit, now, unused_days) if hit is not None else None,
|
||||
"last_hit_at": hit.last_hit_at if hit is not None else None,
|
||||
"watched_since": hit.first_seen_at if hit is not None else None,
|
||||
"removable": block is None, "not_removable_reason": block,
|
||||
"managed_by": managed.get(key),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def summarize(rows: Iterable[dict]) -> dict:
|
||||
counts = {kind: 0 for kind in CATEGORIES}
|
||||
for row in rows:
|
||||
for category in row["categories"]:
|
||||
counts[category["kind"]] += 1
|
||||
return counts
|
||||
@@ -0,0 +1,290 @@
|
||||
"""Does an earlier rule make a later one pointless? (shadowed / duplicate)
|
||||
|
||||
Pure functions on ``NormRule``; no I/O. Used by the security check (``compliance_service``, rules taken
|
||||
from config.xml) and by the cleanup candidates (rules taken from ``searchRule``).
|
||||
|
||||
pf evaluates all rules in order, the last matching rule wins unless a matching rule is ``quick``. So a later
|
||||
rule b can only be made dead by an earlier rule a when
|
||||
|
||||
* a is ``quick`` (a non-quick match is overridden by later rules and decides nothing by itself),
|
||||
* a is in force (enabled, not known to be unloaded) and has no condition we cannot compare
|
||||
(schedule, tags, ``received-on``, TCP flags, rate limits), and
|
||||
* every packet b matches is also matched by a: set coverage per field (interfaces, direction, address
|
||||
family, protocols, source/destination, ports, ICMP types).
|
||||
|
||||
The order is ``sort_order`` (floating 200000 before groups 300000+ before interfaces 400000, then the
|
||||
sequence); nothing is re-implemented here, ``NormRule.order`` already holds it.
|
||||
|
||||
Every comparison answers ``"sure"`` (literal values or identical names), ``"probable"`` (the result relies
|
||||
on the static content of aliases or on a guessed order) or ``None`` (not provable, so no claim is made).
|
||||
Whatever is not proven is never "sure": the weakest link decides.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
from app.services.rule_analysis.context import Context
|
||||
from app.services.rule_analysis.model import (
|
||||
SECTION_ANTISPOOF, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE, SECTION_INVALID, SECTION_NAT_AUTO,
|
||||
Endpoint, NormRule,
|
||||
)
|
||||
from app.services.rule_analysis.normalize import _af, _protos
|
||||
from app.services.rule_analysis.resolve import ADDRESS_TYPES, MAX_ALIAS_DEPTH, port_ranges
|
||||
|
||||
SURE, PROBABLE = "sure", "probable"
|
||||
_RANK = {None: 0, PROBABLE: 1, SURE: 2}
|
||||
|
||||
|
||||
def weakest(*values: str | None) -> str | None:
|
||||
"""The lowest confidence of the arguments (None beats everything: one unproven part spoils the claim)."""
|
||||
return min(values, key=lambda v: _RANK[v]) if values else SURE
|
||||
|
||||
|
||||
def _empty_context() -> Context:
|
||||
return Context(rows=[], pf=None, interfaces={}, groups=None, aliases={}, port_forwards=None)
|
||||
|
||||
|
||||
# ── Addresses ─────────────────────────────────────────────────────────────
|
||||
|
||||
Range = tuple[int, int, int] # (ip version, low, high)
|
||||
|
||||
|
||||
def _literal(token: str) -> Range | None:
|
||||
"""CIDR, single address or ``low-high`` range as an integer interval; None for names."""
|
||||
token = token.strip()
|
||||
try:
|
||||
if "-" in token and not token.startswith("-"):
|
||||
low, _, high = token.partition("-")
|
||||
lo, hi = ipaddress.ip_address(low.strip()), ipaddress.ip_address(high.strip())
|
||||
return (lo.version, int(lo), int(hi)) if lo.version == hi.version and lo <= hi else None
|
||||
net = ipaddress.ip_network(token, strict=False)
|
||||
except ValueError:
|
||||
return None
|
||||
return (net.version, int(net.network_address), int(net.broadcast_address))
|
||||
|
||||
|
||||
def _alias_ranges(name: str, ctx: Context, depth: int = 0, seen: tuple = ()) -> list[Range] | None:
|
||||
"""Static content of a host/network alias as intervals; None when anything about it is not static
|
||||
(tables, URLs, FQDNs, exclusions, unknown names, cycles)."""
|
||||
alias = ctx.aliases.get(name)
|
||||
if alias is None or name in seen or depth > MAX_ALIAS_DEPTH:
|
||||
return None
|
||||
if str(alias.get("type") or "").lower() not in ADDRESS_TYPES:
|
||||
return None
|
||||
if str(alias.get("enabled", "1")).strip().lower() in ("0", "false", "no", "off"):
|
||||
return None # a disabled alias is an empty table in pf: its configured content proves nothing
|
||||
result: list[Range] = []
|
||||
for entry in (e.strip() for e in re.split(r"[\n,]", str(alias.get("content") or "")) if e.strip()):
|
||||
if entry.startswith("!"):
|
||||
return None
|
||||
literal = _literal(entry)
|
||||
if literal is not None:
|
||||
result.append(literal)
|
||||
continue
|
||||
nested = _alias_ranges(entry, ctx, depth + 1, (*seen, name)) if entry in ctx.aliases else None
|
||||
if nested is None:
|
||||
return None
|
||||
result += nested
|
||||
return result or None
|
||||
|
||||
|
||||
def _ranges_of(token: str, ctx: Context) -> tuple[list[Range] | None, bool]:
|
||||
"""(intervals, came_from_alias)"""
|
||||
literal = _literal(token)
|
||||
if literal is not None:
|
||||
return [literal], False
|
||||
return _alias_ranges(token, ctx), True
|
||||
|
||||
|
||||
def _covered(inner: list[Range], outer: list[Range]) -> bool:
|
||||
return all(any(o[0] == i[0] and o[1] <= i[1] and i[2] <= o[2] for o in outer) for i in inner)
|
||||
|
||||
|
||||
def _item_covers(x: str, y: str, ctx: Context) -> str | None:
|
||||
"""Does the address item x contain everything the item y contains?"""
|
||||
x, y = x.strip(), y.strip()
|
||||
if x == y:
|
||||
return SURE
|
||||
outer, outer_alias = _ranges_of(x, ctx)
|
||||
inner, inner_alias = _ranges_of(y, ctx)
|
||||
if outer is None or inner is None or not _covered(inner, outer):
|
||||
return None
|
||||
return PROBABLE if (outer_alias or inner_alias) else SURE
|
||||
|
||||
|
||||
def endpoint_covers(a: Endpoint, b: Endpoint, ctx: Context) -> str | None:
|
||||
if a.is_any and not a.negated:
|
||||
return SURE
|
||||
if b.is_any and not b.negated:
|
||||
return None # a is narrower than "any"
|
||||
if a.negated != b.negated:
|
||||
return None
|
||||
if not a.negated: # every item of b lies inside some item of a
|
||||
parts = []
|
||||
for y in b.items:
|
||||
best = None
|
||||
for x in a.items:
|
||||
got = _item_covers(x, y, ctx)
|
||||
if _RANK[got] > _RANK[best]:
|
||||
best = got
|
||||
parts.append(best)
|
||||
return weakest(*parts)
|
||||
# "!A" covers "!B" when A lies inside B (outside of B is a subset of outside of A)
|
||||
parts = []
|
||||
for x in a.items:
|
||||
best = None
|
||||
for y in b.items:
|
||||
got = _item_covers(y, x, ctx)
|
||||
if _RANK[got] > _RANK[best]:
|
||||
best = got
|
||||
parts.append(best)
|
||||
return weakest(*parts)
|
||||
|
||||
|
||||
# ── Ports, interfaces, the rest ───────────────────────────────────────────
|
||||
|
||||
def _norm_ports(spec: str) -> str:
|
||||
return re.sub(r"\s+", "", spec or "").lower()
|
||||
|
||||
|
||||
def ports_cover(a: str, b: str, ctx: Context) -> str | None:
|
||||
if _norm_ports(a) in ("", "any"):
|
||||
return SURE
|
||||
if _norm_ports(b) in ("", "any"):
|
||||
return None
|
||||
if _norm_ports(a) == _norm_ports(b):
|
||||
return SURE
|
||||
outer, _ = port_ranges(a, ctx)
|
||||
inner, _ = port_ranges(b, ctx)
|
||||
if outer is None or inner is None:
|
||||
return None
|
||||
if not all(any(lo <= il and ih <= hi for lo, hi in outer) for il, ih in inner):
|
||||
return None
|
||||
alias_involved = any(part.strip() in ctx.aliases for spec in (a, b) for part in re.split(r"[\n,]", spec))
|
||||
return PROBABLE if alias_involved else SURE
|
||||
|
||||
|
||||
def _expand(names: set[str], ctx: Context) -> set[str]:
|
||||
result: set[str] = set()
|
||||
for name in names:
|
||||
result |= (ctx.groups[name] | {name}) if ctx.groups and name in ctx.groups else {name}
|
||||
return result
|
||||
|
||||
|
||||
def interfaces_cover(a: NormRule, b: NormRule, ctx: Context) -> str | None:
|
||||
if not a.interfaces and not a.if_not:
|
||||
return SURE # every interface
|
||||
if a.if_not or b.if_not:
|
||||
return SURE if (a.interfaces, a.if_not) == (b.interfaces, b.if_not) else None
|
||||
if not b.interfaces:
|
||||
return None # b applies everywhere, a only to some interfaces
|
||||
return SURE if _expand(b.interfaces, ctx) <= _expand(a.interfaces, ctx) else None
|
||||
|
||||
|
||||
def _conditions_ok(a: NormRule) -> bool:
|
||||
"""a must not depend on something we cannot compare: schedule, tags, received-on, TCP flags, limits."""
|
||||
return not (a.sched or a.tagged or a.received_on or a.extra_notes)
|
||||
|
||||
|
||||
def covers_fields(a: NormRule, b: NormRule, ctx: Context | None = None) -> str | None:
|
||||
"""Every packet rule b matches is already matched by rule a (order and enabled state not considered)."""
|
||||
ctx = ctx or _empty_context()
|
||||
if a.quick is not True or not _conditions_ok(a):
|
||||
return None
|
||||
if a.direction != "any" and a.direction != b.direction:
|
||||
return None
|
||||
if not b.af <= a.af:
|
||||
return None
|
||||
if a.protos is not None and (b.protos is None or not b.protos <= a.protos):
|
||||
return None
|
||||
if a.icmptypes is not None and (b.icmptypes is None or not b.icmptypes <= a.icmptypes):
|
||||
return None
|
||||
parts = [
|
||||
interfaces_cover(a, b, ctx),
|
||||
endpoint_covers(a.src, b.src, ctx), endpoint_covers(a.dst, b.dst, ctx),
|
||||
ports_cover(a.sport, b.sport, ctx), ports_cover(a.dport, b.dport, ctx),
|
||||
]
|
||||
return weakest(*parts)
|
||||
|
||||
|
||||
# ── Order-aware analysis ──────────────────────────────────────────────────
|
||||
|
||||
def _in_force(rule: NormRule) -> bool:
|
||||
return rule.enabled and rule.loaded is not False and rule.section not in (SECTION_INVALID, SECTION_NAT_AUTO)
|
||||
|
||||
|
||||
# normalize() notes after which the evaluation order is not trustworthy: findings stay at "probable" at most.
|
||||
ORDER_DOUBT = frozenset({"ORDER_GUESSED", "ORDER_MISMATCH"})
|
||||
|
||||
|
||||
def order_doubtful(notes) -> bool:
|
||||
return bool(ORDER_DOUBT & set(notes))
|
||||
|
||||
|
||||
@dataclass
|
||||
class Overlap:
|
||||
rule: NormRule # the later rule that is made pointless
|
||||
by: NormRule # the earlier rule that already decides everything the later one matches
|
||||
kind: str # "shadowed" | "duplicate"
|
||||
confidence: str # "sure" | "probable"
|
||||
|
||||
|
||||
def analyse(rules: list[NormRule], ctx: Context | None = None, *, order_guessed: bool = False) -> list[Overlap]:
|
||||
"""``rules`` in evaluation order. Each later rule is reported at most once, with the first earlier rule that
|
||||
covers it. Duplicate = same action and the later rule also covers the earlier one."""
|
||||
ctx = ctx or _empty_context()
|
||||
result: list[Overlap] = []
|
||||
for index, later in enumerate(rules):
|
||||
if not later.enabled or later.section in (SECTION_ANTISPOOF, SECTION_INVALID, SECTION_NAT_AUTO):
|
||||
continue
|
||||
if later.action is None:
|
||||
continue
|
||||
for earlier in rules[:index]:
|
||||
if not _in_force(earlier) or earlier.action is None:
|
||||
continue
|
||||
confidence = covers_fields(earlier, later, ctx)
|
||||
if confidence is None:
|
||||
continue
|
||||
# (Equal sort_order without pf line numbers always comes with loaded=None, i.e. "probable" here.)
|
||||
if order_guessed or not (earlier.order_known and later.order_known) or earlier.loaded is None:
|
||||
confidence = weakest(confidence, PROBABLE)
|
||||
duplicate = earlier.action == later.action and covers_fields(later, earlier, ctx) is not None
|
||||
if duplicate:
|
||||
confidence = weakest(confidence, covers_fields(later, earlier, ctx))
|
||||
result.append(Overlap(later, earlier, "duplicate" if duplicate else "shadowed", confidence or PROBABLE))
|
||||
break
|
||||
return result
|
||||
|
||||
|
||||
# ── Rules read from a config.xml (security check) ─────────────────────────
|
||||
|
||||
def config_order(rule: dict) -> tuple[int, int]:
|
||||
"""sort_order of a rule from config.xml in OPNsense's terms: floating 200000, interface group 300000,
|
||||
single interface 400000, each by sequence."""
|
||||
return (200000 if rule["floating"] else 300000 if rule.get("group") else 400000, rule["order"])
|
||||
|
||||
|
||||
def _config_endpoint(raw: str, negated: bool) -> Endpoint:
|
||||
raw = (raw or "").strip()
|
||||
items = [] if raw.lower() in ("", "any") else [i.strip() for i in raw.split(",") if i.strip()]
|
||||
return Endpoint(raw=raw, negated=negated, items=items)
|
||||
|
||||
|
||||
def from_config_rule(rule: dict) -> NormRule:
|
||||
"""Adapter for the rule dicts of compliance_service (MVC and legacy shapes already merged). Nothing is known
|
||||
about the loaded pf ruleset there, so ``loaded`` stays None (coverage then caps at "probable")."""
|
||||
order = config_order(rule)
|
||||
section = SECTION_FLOATING if rule["floating"] else SECTION_GROUP if rule.get("group") else SECTION_INTERFACE
|
||||
return NormRule(
|
||||
uuid="", label="", description=rule["descr"], order=order, order_known=True, section=section,
|
||||
enabled=rule["enabled"], action=rule["action"] if rule["action"] in ("pass", "block", "reject") else None,
|
||||
quick=bool(rule["quick"]), interfaces=set(rule["interfaces"]), if_not=bool(rule.get("if_not")),
|
||||
direction=rule["direction"] or "in", af=_af(rule["ipprotocol"]), protos=_protos(rule["protocol"]),
|
||||
src=_config_endpoint(rule["src"], rule["src_not"]), dst=_config_endpoint(rule["dst"], rule["dst_not"]),
|
||||
sport=rule["sport"] or "", dport=rule["dport"] or "",
|
||||
icmptypes={t.strip() for t in (rule.get("icmptype") or "").split(",") if t.strip()} or None,
|
||||
tagged=rule.get("tagged") or "", received_on=rule.get("received_on") or "", sched=rule.get("sched") or "",
|
||||
extra_notes=["RATE_LIMIT"] if rule.get("limited") else [],
|
||||
)
|
||||
@@ -117,6 +117,31 @@ def rule_fields(r: NormRule, q: Query, direction: str, iface: str, ctx: Context,
|
||||
return fields
|
||||
|
||||
|
||||
def match_rule(r: NormRule, q: Query, ctx: Context, *, iface: str | None = None,
|
||||
source: Outcome | None = None, overrides: dict[str, Outcome] | None = None) -> Outcome:
|
||||
"""Could the rule's own fields match the packet ``q`` arriving on ``iface``? Order, ``quick``, other rules and
|
||||
whether the rule is enabled or loaded play no part: this is the question "is this rule about such a packet at
|
||||
all", which the cleanup asks about the way the admin reaches the firewall (DESIGN-1.31 §3.2).
|
||||
|
||||
``iface`` defaults to ``q.iface`` and then to the interface guessed from the address; an interface that cannot
|
||||
be resolved is UNKNOWN. ``source`` replaces the source field (the caller judged it for a whole network, see
|
||||
rule_analysis/protection.py); ``overrides`` replaces further fields by name (e.g. ``interface`` for a packet
|
||||
arriving on a device that is not an assigned interface). Tags set by earlier rules are not known here, so ``tagged`` is UNKNOWN. Only
|
||||
``False`` is a proof that the rule does not match; ``None`` (unknown) must be treated like ``True``."""
|
||||
iface = iface or q.iface or guess_iface(q.src if q.direction == "in" else q.dst, ctx)
|
||||
if iface is None:
|
||||
return Outcome(None, ["IFACE_UNRESOLVED"])
|
||||
fields = rule_fields(r, q, q.direction, iface, ctx, set(), None)
|
||||
if source is not None:
|
||||
fields["source"] = source
|
||||
for name, outcome in (overrides or {}).items():
|
||||
if name in fields or name == "interface":
|
||||
fields[name] = outcome
|
||||
value = t_and(*(f.value for f in fields.values()))
|
||||
reasons = sorted({reason for f in fields.values() if f.value is None for reason in f.reasons}) if value is None else []
|
||||
return Outcome(value, reasons)
|
||||
|
||||
|
||||
def _step(r: NormRule, fields: dict[str, Outcome], result, effect: str) -> dict:
|
||||
return {
|
||||
"rule": r.summary(), "result": tri_str(result), "effect": effect,
|
||||
|
||||
@@ -13,6 +13,8 @@ NAT, 600000 = interface missing, never loaded).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from app.services.rule_analysis.context import Context
|
||||
from app.services.rule_analysis.model import (
|
||||
SECTION_ANTISPOOF, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE, SECTION_INVALID,
|
||||
@@ -169,6 +171,11 @@ def normalize_row(row: dict, ctx: Context) -> NormRule:
|
||||
return rule
|
||||
|
||||
|
||||
# Words in an unlabeled pf line that restrict it in ways antispoof_rule() does not model (protocol, ports, flags,
|
||||
# tags, user/group, received-on, ICMP types, OS): such a line must never be used as a covering rule.
|
||||
_UNMODELLED = re.compile(r"\b(proto|port|flags|tagged|received-on|user|group|icmp6?-type|os|tag|label)\b")
|
||||
|
||||
|
||||
def antispoof_rule(line: PfLine, order: tuple[int, int]) -> NormRule:
|
||||
"""An unlabeled pf line OPNsense generated itself (antispoof / own-address block)."""
|
||||
return NormRule(
|
||||
@@ -179,6 +186,7 @@ def antispoof_rule(line: PfLine, order: tuple[int, int]) -> NormRule:
|
||||
src=Endpoint(raw="" if line.src == "any" else line.src, items=[] if line.src == "any" else [line.src]),
|
||||
dst=Endpoint(raw="" if line.dst == "any" else line.dst, items=[] if line.dst == "any" else [line.dst]),
|
||||
sport="", dport="", automatic=True, loaded=True, pf_numbers=[line.number], pf_line=line,
|
||||
extra_notes=["UNMODELLED_CONDITION"] if _UNMODELLED.search(line.text) else [],
|
||||
)
|
||||
|
||||
|
||||
@@ -188,7 +196,9 @@ def normalize(ctx: Context) -> tuple[list[NormRule], list[str]]:
|
||||
rules = [normalize_row(row, ctx) for row in ctx.rows]
|
||||
if any(not r.order_known for r in rules):
|
||||
notes.append("ORDER_GUESSED")
|
||||
rules.sort(key=lambda r: (r.order, r.uuid))
|
||||
# Equal sort_order (e.g. two rules with the same sequence): what pf actually loaded decides, the uuid only as
|
||||
# a last resort (then coverage.analyse() lowers its confidence, it cannot know the real order).
|
||||
rules.sort(key=lambda r: (r.order, min(r.pf_numbers) if r.pf_numbers else 10**9, r.uuid))
|
||||
if ctx.pf is not None:
|
||||
loaded = [r for r in rules if r.loaded and r.pf_numbers]
|
||||
firsts = [min(r.pf_numbers) for r in loaded]
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""May a rule be switched off without touching the way the admin reaches the firewall? (DESIGN-1.31 §3.2/§3.3)
|
||||
|
||||
Pure functions on ``NormRule`` and the analysis ``Context``; no I/O.
|
||||
|
||||
Management access (§3.2). For every configured management source (IP or network), every address of the
|
||||
firewall (the address in ``host.base_url`` plus every interface address and VIP) and every management port
|
||||
(the API port of ``base_url``, 443, 80, 22) and every interface the packet could arrive on, the rule's own
|
||||
fields are compared with that packet (``engine.match_rule``). A rule is safe only when EVERY such comparison is
|
||||
``False``. ``True`` or unknown means the rule is about management traffic: it is protected.
|
||||
|
||||
Why that is enough without simulating "the ruleset afterwards": a rule that does not match a packet takes no part
|
||||
in its evaluation at all - it neither decides it nor tags it (pf only tags on a match) - so removing it cannot
|
||||
change what happens to that packet. This holds whatever the verdict for the packet is, even when the analysis
|
||||
of the whole ruleset is only "probable". The packet's ingress interface is a routing question OPNexus cannot prove
|
||||
(asymmetric routes, VPNs, failover), so every interface is tried: that only ever protects more.
|
||||
|
||||
A management source that is a network is judged for all of its addresses at once: up to 256 addresses are
|
||||
enumerated (exact, with every resolver rule of the analysis), larger networks are compared as address ranges
|
||||
with the rule's literal/alias/interface addresses; anything that cannot be resolved is unknown = protected.
|
||||
|
||||
Infrastructure (§3.3, reason ``infrastructure``): CARP (IP protocol 112) and pfsync (240) rules, every rule that
|
||||
could match CARP advertisements (to 224.0.0.18 / ff02::12 from the interface's own networks) on an interface with
|
||||
a CARP VIP, and every rule that could match pfsync traffic on the pfsync interface (System > High Availability,
|
||||
``core/hasync/get``; when that cannot be read on a cluster node, on every interface). The second leash of the
|
||||
design ("(self) with port 22/80/443/API" when §3.2 is not False) is part of §3.2 already: such a rule is
|
||||
protected there.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from app.services.rule_analysis import coverage, resolve
|
||||
from app.services.rule_analysis.context import BUILTIN_GROUP_DEVICES, Context
|
||||
from app.services.rule_analysis.engine import Query, match_rule
|
||||
from app.services.rule_analysis.model import Endpoint, IPAddr, IPNet, NormRule, Outcome, o_or, t_not
|
||||
|
||||
MANAGEMENT_PORTS = (443, 80, 22)
|
||||
# A management network up to this size is checked address by address (exact); larger ones as address ranges.
|
||||
ENUMERATE_LIMIT = 256
|
||||
CARP_PROTO, PFSYNC_PROTO = "112", "240"
|
||||
CARP_GROUPS = {4: ipaddress.ip_address("224.0.0.18"), 6: ipaddress.ip_address("ff02::12")}
|
||||
PFSYNC_GROUP = ipaddress.ip_address("224.0.0.240")
|
||||
_LOOPBACK = {ipaddress.ip_address("127.0.0.1"), ipaddress.ip_address("::1")}
|
||||
|
||||
|
||||
def parse_sources(values) -> list[IPNet]:
|
||||
"""The configured management sources as networks (an address becomes a /32 or /128). Invalid entries are
|
||||
dropped; the caller treats an empty result as "not configured" (the cleanup is then locked)."""
|
||||
nets: list[IPNet] = []
|
||||
for value in values or []:
|
||||
try:
|
||||
net = ipaddress.ip_network(str(value).strip(), strict=False)
|
||||
except ValueError:
|
||||
continue
|
||||
if net not in nets:
|
||||
nets.append(net)
|
||||
return nets
|
||||
|
||||
|
||||
def management_targets(base_url: str, ctx: Context, extra_ports=()) -> tuple[list[IPAddr], list[int]]:
|
||||
"""Firewall addresses and ports the management traffic goes to; ``extra_ports`` = the admin's
|
||||
``cleanup_management_ports`` (an SSH or GUI port moved away from the defaults)."""
|
||||
parts = urlsplit(base_url or "")
|
||||
addrs: list[IPAddr] = []
|
||||
try:
|
||||
addrs.append(ipaddress.ip_address((parts.hostname or "").strip("[]")))
|
||||
except ValueError:
|
||||
pass # a DNS name: the interface addresses below cover the address it resolves to on the firewall
|
||||
for addr in ctx.self_addrs:
|
||||
if addr not in _LOOPBACK and addr not in addrs:
|
||||
addrs.append(addr)
|
||||
try:
|
||||
api_port = parts.port or 443
|
||||
except ValueError:
|
||||
api_port = 443
|
||||
ports = [api_port]
|
||||
for port in (*MANAGEMENT_PORTS, *extra_ports):
|
||||
if isinstance(port, int) and 1 <= port <= 65535 and port not in ports:
|
||||
ports.append(port)
|
||||
return addrs, ports
|
||||
|
||||
|
||||
def _interfaces(ctx: Context) -> list[str]:
|
||||
return sorted(i for i in ctx.interfaces if i != "lo0")
|
||||
|
||||
|
||||
# A packet can also arrive on a device that is not an assigned interface: a WireGuard instance (wg0), an OpenVPN
|
||||
# server (ovpns1), IPsec (enc0). They have no identifier, so they are not in ``ctx.interfaces``, and the built-in
|
||||
# groups that stand for them (``wireguard``, ``openvpn``, ``enc0``) list no members in ``firewall/group/searchItem``
|
||||
# (live fixture tests/fixtures/rule_analysis_fw.json: members ""). One extra, virtual ingress is checked per query:
|
||||
UNASSIGNED = "(unassigned)"
|
||||
|
||||
|
||||
def unassigned_interface(rule: NormRule, ctx: Context) -> Outcome:
|
||||
"""Could the rule's interface field match a packet arriving on an unassigned device? An assigned interface or a
|
||||
user-defined group (its members are assigned interfaces) cannot; a built-in group or an unknown name might
|
||||
(members unknown = protected); no interface (floating "any") always does."""
|
||||
if not rule.interfaces:
|
||||
return Outcome(True)
|
||||
results = []
|
||||
for name in rule.interfaces:
|
||||
if name in ctx.interfaces:
|
||||
results.append(Outcome(False))
|
||||
elif name in BUILTIN_GROUP_DEVICES or ctx.groups is None or name not in ctx.groups:
|
||||
results.append(Outcome(None, ["IFACE_MEMBERS_UNKNOWN"]))
|
||||
else:
|
||||
results.append(Outcome(False))
|
||||
result = o_or(results)
|
||||
return Outcome(t_not(result.value), result.reasons) if rule.if_not else result
|
||||
|
||||
|
||||
def _ingress(rule: NormRule, ctx: Context) -> list[tuple[str, dict | None]]:
|
||||
"""(interface, field overrides) for every ingress a management packet could take."""
|
||||
virtual = {"interface": unassigned_interface(rule, ctx)}
|
||||
if rule.received_on:
|
||||
virtual["received_on"] = Outcome(None, ["RECEIVED_ON"])
|
||||
return [(i, None) for i in _interfaces(ctx)] + [(UNASSIGNED, virtual)]
|
||||
|
||||
|
||||
def _item_ranges(token: str, ctx: Context) -> list[coverage.Range] | None:
|
||||
"""Address ranges an endpoint item stands for; None when that is not static."""
|
||||
ranges, _alias = coverage._ranges_of(token, ctx)
|
||||
if ranges is not None:
|
||||
return ranges
|
||||
name = token.strip().lower()
|
||||
if name in ("(self)", "self", "this firewall"):
|
||||
return [(a.version, int(a), int(a)) for a in ctx.self_addrs]
|
||||
if name.endswith("ip") and name[:-2] in ctx.interfaces:
|
||||
info = ctx.interfaces[name[:-2]]
|
||||
return [(a.version, int(a), int(a)) for a in (*info.addrs, *info.vips)]
|
||||
if name.startswith("__") and name.endswith("_network"):
|
||||
name = name[2:-len("_network")]
|
||||
if name in ctx.interfaces:
|
||||
return [(n.version, int(n.network_address), int(n.broadcast_address)) for n in ctx.interfaces[name].nets]
|
||||
return None
|
||||
|
||||
|
||||
def _covers_all(ranges: list[coverage.Range], version: int, low: int, high: int) -> bool:
|
||||
cursor = low
|
||||
for _v, a, b in sorted(r for r in ranges if r[0] == version):
|
||||
if a > cursor:
|
||||
return False
|
||||
cursor = max(cursor, b + 1)
|
||||
if cursor > high:
|
||||
return True
|
||||
return cursor > high
|
||||
|
||||
|
||||
def source_outcome(endpoint: Endpoint, net: IPNet, ctx: Context) -> Outcome:
|
||||
"""Does the endpoint match at least one address of ``net``? (False is a proof for every address.)"""
|
||||
if endpoint.is_any:
|
||||
return Outcome(True)
|
||||
if net.num_addresses <= ENUMERATE_LIMIT:
|
||||
return o_or([resolve.endpoint_match(endpoint, ip, ctx) for ip in net])
|
||||
ranges: list[coverage.Range] = []
|
||||
for item in endpoint.items:
|
||||
got = _item_ranges(item, ctx)
|
||||
if got is None:
|
||||
return Outcome(None, ["SOURCE_NOT_RESOLVABLE"])
|
||||
ranges += got
|
||||
low, high = int(net.network_address), int(net.broadcast_address)
|
||||
if endpoint.negated:
|
||||
return Outcome(not _covers_all(ranges, net.version, low, high))
|
||||
return Outcome(any(v == net.version and a <= high and b >= low for v, a, b in ranges))
|
||||
|
||||
|
||||
def _query(src: IPAddr, dst: IPAddr, proto: str, label: str, dport: int | None) -> Query:
|
||||
return Query(src=src, dst=dst, proto=proto, proto_label=label, dport=dport, direction="in", egress="none")
|
||||
|
||||
|
||||
def management_check(rule: NormRule, ctx: Context, sources: list[IPNet], targets: list[IPAddr],
|
||||
ports: list[int]) -> dict:
|
||||
"""{"result": "false" | "true" | "unknown", "queries": n, "reasons": [...], "match": {...} | None}.
|
||||
Only "false" makes the rule removable."""
|
||||
reasons: set[str] = set()
|
||||
queries = 0
|
||||
unknown_at = None
|
||||
interfaces = _ingress(rule, ctx)
|
||||
for net in sources:
|
||||
src = source_outcome(rule.src, net, ctx)
|
||||
if src.value is False:
|
||||
dsts = [d for d in targets if d.version == net.version]
|
||||
queries += len(dsts) * len(ports) * len(interfaces)
|
||||
continue
|
||||
for dst in (d for d in targets if d.version == net.version):
|
||||
for port in ports:
|
||||
q = _query(net.network_address, dst, "6", "tcp", port)
|
||||
for iface, overrides in interfaces:
|
||||
queries += 1
|
||||
out = match_rule(rule, q, ctx, iface=iface, source=src, overrides=overrides)
|
||||
where = {"source": str(net), "destination": str(dst), "port": port, "interface": iface}
|
||||
if out.value is True:
|
||||
return {"result": "true", "queries": queries, "reasons": [], "match": where}
|
||||
if out.value is None:
|
||||
reasons.update(out.reasons)
|
||||
unknown_at = unknown_at or where
|
||||
if not queries:
|
||||
# No address of the firewall in the family of any source: nothing could be proven.
|
||||
return {"result": "unknown", "queries": 0, "reasons": ["NO_MANAGEMENT_QUERY"], "match": None}
|
||||
if unknown_at is not None:
|
||||
return {"result": "unknown", "queries": queries, "reasons": sorted(reasons), "match": unknown_at}
|
||||
return {"result": "false", "queries": queries, "reasons": [], "match": None}
|
||||
|
||||
|
||||
def _hasync_interface(hasync: dict | None) -> str | None:
|
||||
"""The pfsync interface from ``core/hasync/get`` ("" = HA sync disabled); None when not readable."""
|
||||
node = (hasync or {}).get("hasync") if isinstance(hasync, dict) else None
|
||||
if not isinstance(node, dict) or "pfsyncinterface" not in node:
|
||||
return None
|
||||
value = node.get("pfsyncinterface")
|
||||
if isinstance(value, dict):
|
||||
return next((k for k, o in value.items() if isinstance(o, dict) and str(o.get("selected")) == "1"), "")
|
||||
return str(value or "")
|
||||
|
||||
|
||||
def _hasync_peer(hasync: dict | None) -> IPAddr | None:
|
||||
node = (hasync or {}).get("hasync") if isinstance(hasync, dict) else None
|
||||
try:
|
||||
return ipaddress.ip_address(str((node or {}).get("pfsyncpeerip") or "").strip())
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def infrastructure_reason(rule: NormRule, ctx: Context, *, hasync: dict | None, clustered: bool) -> str | None:
|
||||
"""'infrastructure' when the rule could carry CARP or pfsync traffic, else None."""
|
||||
if rule.protos is not None and rule.protos & {CARP_PROTO, PFSYNC_PROTO}:
|
||||
return "infrastructure"
|
||||
for info in ctx.interfaces.values():
|
||||
if info.identifier == "lo0" or not info.vips:
|
||||
continue
|
||||
for net in info.nets:
|
||||
group = CARP_GROUPS.get(net.version)
|
||||
out = match_rule(rule, _query(net.network_address, group, CARP_PROTO, "carp", None), ctx,
|
||||
iface=info.identifier, source=source_outcome(rule.src, net, ctx))
|
||||
if out.value is not False:
|
||||
return "infrastructure"
|
||||
pfsync_if = _hasync_interface(hasync)
|
||||
if pfsync_if == "" or (pfsync_if is None and not clustered):
|
||||
return None # HA sync off (or a single firewall whose HA settings are not readable)
|
||||
peer = _hasync_peer(hasync)
|
||||
candidates = [ctx.interfaces[pfsync_if]] if pfsync_if in ctx.interfaces else \
|
||||
[i for i in ctx.interfaces.values() if i.identifier != "lo0"] # unknown: every interface
|
||||
for info in candidates:
|
||||
for net in info.nets:
|
||||
if peer is not None and peer.version == net.version:
|
||||
src = Outcome(resolve.endpoint_match(rule.src, peer, ctx).value)
|
||||
src_ip = peer
|
||||
else:
|
||||
src, src_ip = source_outcome(rule.src, net, ctx), net.network_address
|
||||
for dst in [*(a for a in info.addrs if a.version == net.version)] + ([PFSYNC_GROUP] if net.version == 4 else []):
|
||||
out = match_rule(rule, _query(src_ip, dst, PFSYNC_PROTO, "pfsync", None), ctx,
|
||||
iface=info.identifier, source=src)
|
||||
if out.value is not False:
|
||||
return "infrastructure"
|
||||
return None
|
||||
@@ -3,11 +3,25 @@ from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeployment
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.rule_hit import RuleHit
|
||||
from app.models.rule_history import LiveRuleState
|
||||
from app.services import rule_hit_service
|
||||
from app.services.rule_history_service import BASELINE_KEY
|
||||
from app.services.rule_analysis import cleanup, coverage
|
||||
from app.services.rule_analysis import context as ctx_mod
|
||||
from app.services.rule_analysis.engine import Query, evaluate
|
||||
from app.services.rule_analysis.normalize import normalize
|
||||
from app.services.rule_analysis.normalize import PROTO_NUMBERS, norm_proto
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -74,3 +88,75 @@ async def analyze_host(host: OpnsenseHost, q: Query, *, refresh: bool = False) -
|
||||
if any(ctx.refs.get(ip) is None for ip in (str(q.src), str(q.dst))):
|
||||
result["notes"] = sorted({*result["notes"], "ALIAS_REFERENCES_UNAVAILABLE"})
|
||||
return result
|
||||
|
||||
|
||||
async def _managed_rules(db: AsyncSession, host: OpnsenseHost) -> dict[str, dict]:
|
||||
"""{label on the firewall: what manages it} for every rule OPNexus deployed to this host."""
|
||||
managed: dict[str, dict] = {}
|
||||
for dep_uuid, rule_id, description in (await db.execute(
|
||||
select(RuleDeployment.opnsense_rule_uuid, FirewallRule.id, FirewallRule.description)
|
||||
.join(FirewallRule, FirewallRule.id == RuleDeployment.rule_id)
|
||||
.where(RuleDeployment.host_id == host.id, RuleDeployment.opnsense_rule_uuid.is_not(None)))).all():
|
||||
managed[dep_uuid.lower()] = {"kind": "rule", "id": str(rule_id), "name": description}
|
||||
for dep_uuid, rule_id, description in (await db.execute(
|
||||
select(PolicyHierarchyRuleDeployment.opnsense_policy_hierarchy_uuid, PolicyHierarchyRule.id, PolicyHierarchyRule.description)
|
||||
.join(PolicyHierarchyRule, PolicyHierarchyRule.id == PolicyHierarchyRuleDeployment.policy_hierarchy_id)
|
||||
.where(PolicyHierarchyRuleDeployment.host_id == host.id,
|
||||
PolicyHierarchyRuleDeployment.opnsense_policy_hierarchy_uuid.is_not(None)))).all():
|
||||
managed[dep_uuid.lower()] = {"kind": "policy_hierarchy", "id": str(rule_id), "name": description}
|
||||
return managed
|
||||
|
||||
|
||||
async def _hits(db: AsyncSession, host_id) -> dict:
|
||||
return {r.rule_uuid.lower(): r for r in (await db.scalars(select(RuleHit).where(RuleHit.host_id == host_id))).all()}
|
||||
|
||||
|
||||
async def cluster_hits(db: AsyncSession, host: OpnsenseHost) -> tuple[dict, dict]:
|
||||
"""(usage to judge "unused" with, the host's own hit rows). For a cluster member the usage is merged over all
|
||||
nodes (DESIGN-1.31 §3.4): "unused" only when it holds on every node."""
|
||||
own = await _hits(db, host.id)
|
||||
if not host.cluster_name:
|
||||
return own, own
|
||||
others = (await db.scalars(select(OpnsenseHost.id).where(
|
||||
OpnsenseHost.cluster_name == host.cluster_name, OpnsenseHost.id != host.id))).all()
|
||||
if not others:
|
||||
return own, own
|
||||
return rule_hit_service.merge_cluster_hits(own, [await _hits(db, other) for other in others]), own
|
||||
|
||||
|
||||
async def cleanup_view(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
|
||||
refresh: bool = False, now: datetime | None = None):
|
||||
"""(context, normalized rules, candidate payload, managed rules) - the cleanup service needs all of them."""
|
||||
now = now or datetime.now(timezone.utc)
|
||||
ctx = await ctx_mod.load_context(host, use_cache=not refresh)
|
||||
rules, notes = normalize(ctx)
|
||||
usage, hits = await cluster_hits(db, host)
|
||||
states = {s.rule_uuid: s for s in (await db.scalars(select(LiveRuleState).where(LiveRuleState.host_id == host.id))).all()}
|
||||
managed = await _managed_rules(db, host)
|
||||
rows = cleanup.build(rules, ctx, hits=usage, states=states, managed=managed, now=now,
|
||||
unused_days=unused_days, order_guessed=coverage.order_doubtful(notes))
|
||||
return ctx, rules, _payload(host, rules, rows, hits, states, notes, ctx, unused_days, now), managed
|
||||
|
||||
|
||||
async def cleanup_candidates(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
|
||||
refresh: bool = False, now: datetime | None = None) -> dict:
|
||||
"""Read-only. Raises when the firewall cannot be read (callers map that)."""
|
||||
return (await cleanup_view(db, host, unused_days=unused_days, refresh=refresh, now=now))[2]
|
||||
|
||||
|
||||
def _payload(host, rules, rows, hits, states, notes, ctx, unused_days, now) -> dict:
|
||||
all_notes = sorted({*notes, *ctx.notes})
|
||||
if host.cluster_name and any("CLUSTER_PEER_USAGE_UNKNOWN" in c.get("notes", ()) for r in rows for c in r["categories"]):
|
||||
all_notes = sorted({*all_notes, "CLUSTER_PEER_USAGE_UNKNOWN"})
|
||||
return {
|
||||
"host_id": str(host.id), "host_name": host.name, "cluster_name": host.cluster_name,
|
||||
"unused_days": max(cleanup.MIN_DAYS, min(cleanup.MAX_DAYS, int(unused_days))), "generated_at": now,
|
||||
"rules_total": sum(1 for r in rules if not r.automatic and r.section not in ("antispoof", "nat_auto", "invalid")),
|
||||
"candidates": rows, "counts": cleanup.summarize(rows), "notes": all_notes,
|
||||
"usage": {
|
||||
"tracked": len(hits),
|
||||
"collecting": sum(1 for h in hits.values() if rule_hit_service.classify(h, now, unused_days) == "collecting"),
|
||||
"watching_since": min((h.first_seen_at for h in hits.values()), default=None),
|
||||
},
|
||||
"history": {"watching_since": baseline.first_seen_at if (baseline := states.get(BASELINE_KEY)) else None},
|
||||
}
|
||||
Loaded 100 of 217 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user