Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d93b154660 | ||
|
|
cf9981c6eb | ||
|
|
4eea6424e7 | ||
|
|
a8ca0a14ef | ||
|
|
b020e38d0a | ||
|
|
abd15318a3 | ||
|
|
ec20940139 | ||
|
|
8128d14da4 | ||
|
|
95f297d3f2 | ||
|
|
ad17dcad25 | ||
|
|
5cb2ebe1c6 | ||
|
|
18d489693d | ||
|
|
a9f562eec6 | ||
|
|
a3274f64ae |
No files matched your search
@@ -100,3 +100,26 @@ TRUST_PROXY_HEADERS=false
|
||||
# Zum Aktivieren (Handbuch „Firewalls hinter NAT“): COMPOSE_PROFILES=overlay
|
||||
# und den UDP-Port aus dem Internet erreichbar machen.
|
||||
#OVERLAY_PORT=51820
|
||||
|
||||
# ── GUI-Fernzugriff auf verwaltete Firewalls (optional, Standard AUS) ──
|
||||
# Öffnet die Weboberfläche einer Firewall über OPNexus, auch hinter NAT
|
||||
# (Management-Tunnel), jeweils in einem eigenen Browser-Ursprung (Slot-Port).
|
||||
# Dreifach aus, bis alles hier gesetzt ist UND ein Admin den Schalter unter
|
||||
# Einstellungen einschaltet. Voraussetzungen: HTTPS über den Proxy und
|
||||
# COOKIE_SECURE=true. Siehe Handbuch, „Firewall-Oberfläche über OPNexus“.
|
||||
# 1. Profil aktivieren (mit anderen Profilen kommagetrennt, z. B. overlay,gui):
|
||||
# COMPOSE_PROFILES=gui
|
||||
# 2. Slot-Ursprünge: dieselbe Adresse wie in OPNEXUS_SITES, je Slot ein Port
|
||||
# aus GUI_PORTS. Darf NICHT in FRONTEND_ORIGIN stehen (Start wird sonst verweigert).
|
||||
# OPNEXUS_GUI_SITES=https://10.0.0.5:9441, https://10.0.0.5:9442, https://10.0.0.5:9443, https://10.0.0.5:9444
|
||||
# 3. Gemeinsames Geheimnis Backend <-> Gateway, mind. 32 Zeichen, z. B.
|
||||
# `openssl rand -hex 32`:
|
||||
# GUI_GATEWAY_TOKEN=
|
||||
# 4. Wo die Slot-Ports lauschen. Standard nur lokal (127.0.0.1); für den
|
||||
# Zugriff aus dem Netz die Adresse dieses Hosts eintragen (wie bei
|
||||
# PROXY_BIND_ADDRESS). GUI_PORTS ist ein Bereich, Host- und Container-Port
|
||||
# sind gleich. GUI_PORTS zum Einschalten SETZEN: ohne den Eintrag nimmt
|
||||
# Docker zufällige freie Ports (damit ein Update nie an belegten Ports wie
|
||||
# 9443 scheitert), und die Slot-Adressen oben wären nicht erreichbar.
|
||||
# GUI_BIND_ADDRESS=127.0.0.1
|
||||
# GUI_PORTS=9441-9444
|
||||
+319
-1
@@ -29,7 +29,325 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
|
||||
den Einstellungen.
|
||||
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Merlin …
|
||||
ansteht): Saker …
|
||||
|
||||
## [1.33.0] "Saker" — 2026-10-24
|
||||
|
||||
- **Aus dem Testlauf 1.31.0 behoben (T2 bis T6, B15):**
|
||||
- **Benutzerliste:** `GET /api/users` zeigte jedem angemeldeten Konto (auch Nur-Lese, auch per Token) alle Benutzernamen, Rollen, letzte Anmeldung und 2FA-Status.
|
||||
Jetzt sehen **nur Admins** die ganze Liste, alle anderen nur das eigene Konto.
|
||||
- **Hosts:** Dieselbe Adresse zweimal einzutragen wird abgelehnt (`HOST_URL_EXISTS`, auch beim Ändern der Adresse). Die Hosts-Seite und der Einrichtungsassistent
|
||||
prüfen die Verbindung beim Anlegen (Schlüssel abgelehnt / zu wenig Rechte / Zertifikat / keine Verbindung), speichern bei einem Fehler nichts und bieten
|
||||
„Trotzdem anlegen“. Über die API ist das `verify_connection: true` (Standard aus). **Löschen verlangt jetzt auch am Server den Hostnamen:**
|
||||
`DELETE /api/hosts/{id}?confirm_name=<Name>` (sonst 409); Skripte, die Hosts löschen, müssen das anpassen.
|
||||
- **Zertifikate:** `cert_type: v3_ca` wird nicht mehr angenommen (es entstand ein Zertifikat mit CA-Erweiterung, keine nutzbare CA); die Oberfläche bot es nie an.
|
||||
- **Traceroute:** Zwischenstationen ohne Antwort stehen als Zeile „keine Antwort“ in der Liste (vorher sprang die Zählung 1 → 3); ein Weg ganz ohne Antwort wird erklärt.
|
||||
- **Netzwerk:** Die Vorschau prüft Zahlenbereiche vorab (VLAN-Tag 1–4094, PCP 0–7, VHID 1–255, Basis 1–254, Skew 0–254, gegen das Modell der OPNsense gelesen): `422` mit dem
|
||||
Feldnamen statt `502` von der Firewall.
|
||||
- **Gruppen:** Eine Gruppe, die ein VPN-Objekt als Ziel hat, lässt sich nicht mehr löschen (die Verknüpfung fiel still weg).
|
||||
- Das Skript `scripts/verify_i18n.mjs` läuft jetzt auch gegen die HTTPS-Test-Instanz.
|
||||
- **Regel-Bereinigung: OPNexus selbst ist automatisch geschützt (B10).** Beim Planen fragt die Bereinigung jede Firewall, von welcher Adresse sie OPNexus kommen sieht
|
||||
(die eigene API-Verbindung in der Zustandstabelle), und schützt diese Adresse zusätzlich zu den Verwaltungsquellen. Vorher konnte eine Regel abgeschaltet werden, über
|
||||
die nur OPNexus die Firewall erreicht, wenn in den Quellen nur Arbeitsplätze standen; dann scheiterte auch das automatische Zurücksetzen. Die Vorschau zeigt die Adresse
|
||||
je Firewall (nur Admins) oder sagt, dass sie sich nicht ermitteln ließ (API-Recht „Diagnose: Zustände“, Management-Tunnel).
|
||||
- **Keine verlorenen Regeltreffer mehr (B11):** Jedes Anwenden setzt die Regelzähler der Firewall zurück. OPNexus liest sie jetzt unmittelbar vor jedem Anwenden, das es
|
||||
selbst auslöst (Regeln, NAT, Aliase, IPsec/OpenVPN/WireGuard, Netzwerk); vorher konnte eine Regel deshalb fälschlich als „ungenutzt“ gelten.
|
||||
- **Einstellungen › Regel-Bereinigung:** schlägt die eigene Browser-Adresse als Verwaltungsquelle vor (B12). Geprüft: OPNsense 26.7 hat kein „savepoint“ zum Zurücknehmen
|
||||
durch die Firewall selbst (Quellcode und live 404), das automatische Zurücksetzen bleibt bei OPNexus.
|
||||
|
||||
- **Installation unter Windows:** Neue Anleitung auf der Website (`/docs/windows/`, auch auf Englisch) und im Handbuch: Docker Desktop mit WSL 2, Zertifizierungsstelle importieren, Zugriff aus dem Netz (Windows-Firewall), Updates mit `opnexus` in WSL und Fehlerbehebung. Entwickelt und getestet wird weiterhin unter Linux; die Anleitung sagt das offen.
|
||||
- **Sicherheit und Stabilität (aus einer externen Prüfung):**
|
||||
- **Rückweg bleibt erhalten:** Wer nach „Konfiguration geschrieben, Anwenden fehlgeschlagen“ erneut ausrollte, überschrieb den ursprünglichen Rückweg; die automatische
|
||||
Rücknahme stellte dann den unbestätigten Stand wieder her. Jetzt bleibt der Rückweg der Stand vor dem ersten Versuch. Wartung, Regel-Bereinigung und
|
||||
„Konfiguration als Code“ behandeln eine solche Zeile wie eine offene Bestätigung. Gilt für Regeln, Aliase, Port-Forwards, 1:1-NAT, Outbound-NAT und Hierarchie-Regeln.
|
||||
- **Anmeldung und 2FA:** Eine Passwort-Anmeldung, die genau während der Aktivierung von 2FA lief, bekam eine Sitzung ohne zweiten Faktor. Jetzt wird sie mit `TOTP_REQUIRED` abgewiesen.
|
||||
- **Letzter Administrator:** Zwei Administratoren, die sich gleichzeitig herabstuften oder gegenseitig löschten, konnten alle Administratoren entfernen. Beides ist jetzt serialisiert.
|
||||
- **Operatoren:** Mit ausgeschaltetem „Operatoren dürfen ausrollen“ konnten sie noch Snapshots (Boot-Umgebungen) anlegen. Ein Test hält künftig jede schreibende Route
|
||||
fest, die keiner Rechte-Regel zugeordnet ist.
|
||||
- **Zertifikate und ACME:** Der Ziel-Host ist nach dem ersten Push fest (`409 ZIEL_HOST_NACH_AUSROLLEN_FEST`); vorher meldete ein Host-Wechsel Erfolg für einen Host, der nie
|
||||
etwas erhielt. Ein ACME-Konto mit Anfragen lässt sich nicht auf einen anderen Host verschieben (`ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET`).
|
||||
- **VPN-Ausrollen:** IPsec, OpenVPN und WireGuard sichern jede von der Firewall vergebene Kennung sofort; bei einem Abbruch entstehen beim nächsten Versuch keine Duplikate.
|
||||
Pro VPN läuft nur ein Ausrollen oder Entfernen zugleich (`409 VPN_DEPLOY_LAEUFT_BEREITS`).
|
||||
- **Konfiguration als Code:** Gültiges YAML mit falsch geformtem Abschnitt (z. B. `aliases: 42`) ergibt `422 CONFIG_INVALID` statt eines Serverfehlers.
|
||||
- **`.env`:** Die Anleitungen legen sie mit `install -m 600` an; `opnexus update` und `opnexus status` setzen eine zu offene Datei auf 0600. `opnexus rollback` liest seinen
|
||||
Update-Zustand nur noch als Text und führt ihn nie aus.
|
||||
- **Abhängigkeiten:** Next.js 15.5.28, PyJWT 2.15.1.
|
||||
|
||||
## [1.32.1] "Merlin" — 2026-10-23
|
||||
|
||||
- **Korrektur: Rollouts erkannten die Zielversion bei anderen Release-Typen falsch.** OPNexus las die neue Version nur am Kernpaket `opnsense`. Bei der
|
||||
Business-Edition (`opnsense-business`) und beim Wechsel des Release-Typs (z. B. auf `opnsense-devel`, dann als neues Paket angeboten) fiel es auf
|
||||
`product_latest` zurück; das Gate hielt den Rollout dann fälschlich mit „Version passt nicht“ an. Live gefunden und belegt (Rollout mit Cluster in einer Welle).
|
||||
- **Plan-Hinweis zu Alias-Namen ehrlich formuliert:** „kein zentraler Alias in OPNexus“ statt „kein Alias dieses Namens“ (von Hand auf der Firewall
|
||||
angelegte Aliase sind in Ordnung; der Planer fragt bewusst keine Firewall).
|
||||
|
||||
## [1.32.0] "Merlin" — 2026-10-23
|
||||
|
||||
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
|
||||
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
|
||||
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
|
||||
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
|
||||
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
|
||||
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
|
||||
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
|
||||
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
|
||||
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
|
||||
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
|
||||
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
|
||||
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
|
||||
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
|
||||
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
|
||||
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
|
||||
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
|
||||
|
||||
## [1.31.1] "Vulture" — 2026-10-22
|
||||
|
||||
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
|
||||
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
|
||||
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
|
||||
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
|
||||
|
||||
## [1.31.0] "Vulture" — 2026-10-22
|
||||
|
||||
- **Regel-Bereinigung: Entfernen, Bestätigen, Löschen, Wiederherstellen:** Im Tab **Bereinigung** (Regelwerk › Regel-Analyse) lassen sich Kandidaten jetzt auswählen
|
||||
(Checkbox, höchstens so viele Regeln je Lauf wie in den Einstellungen; „vermutlich“ nur mit einer ausdrücklichen Einzelbestätigung je Regel, „unbekannt“ nie).
|
||||
**Vorschau** zeigt je Firewall (im Cluster zuerst der Backup) die Regeln mit Befund, Sicherheit und Schutzprüfung; **Ausrollen** deaktiviert die Regeln und wendet an.
|
||||
Danach läuft ein Countdown: **Bestätigen** heißt „ich erreiche die Firewall noch“, ohne Bestätigung setzt OPNexus **automatisch zurück**; bis dahin geht auch
|
||||
**Jetzt zurücksetzen**. Bestätigte Regeln lassen sich nach der Wartezeit (einstellbar, Standard 7 Tage) **löschen**; vorher sichert OPNexus den Regelinhalt, und
|
||||
unter **Verlauf der Bereinigungen** kommt die Regel mit **Wiederherstellen** deaktiviert und möglichst an alter Position zurück (sonst mit Hinweis am Ende der Gruppe);
|
||||
**Wieder aktivieren** hat wieder ein Bestätigungsfenster. Wichtig: **Ausrollen, Wiederholen und Wiederherstellen darf nur ein Administrator**, eine Vorschau gehört
|
||||
dem, der sie erstellt hat (ein Admin öffnet selbst eine neue); **Bestätigen geht nur in der Browser-Sitzung**, nie per API-Token. **Die Bereinigung bleibt gesperrt,
|
||||
bis unter Einstellungen die Verwaltungsquellen** (IP-Adressen oder Netze, von denen aus verwaltet wird) **eingetragen sind**: Regeln, die diesen Zugang treffen
|
||||
könnten, deaktiviert OPNexus nie. Dort stehen auch die Obergrenze je Lauf, die Wartezeit bis zum Löschen und **weitere Verwaltungsports** (zusätzlich zu
|
||||
API-Port, 443, 80, 22). Tragen Sie die Adresse ein, mit der OPNexus selbst bei den Firewalls ankommt (im Cluster für jeden Knoten). Quellen, Ports und
|
||||
Firewall-Adressen sehen nur Administratoren. Der Hinweis „Entfernen folgt“ entfällt.
|
||||
- **Update: neuere CLI aus dem Zielrelease:** `opnexus update` (auch per Klick im Webinterface) holt sich nach dem Pull die CLI aus dem Updater-Image des
|
||||
Zielreleases und startet sich damit neu, damit Korrekturen am Update-Ablauf sofort greifen statt erst beim übernächsten Update. Das geschieht nur, wenn
|
||||
`cosign` und `cosign.pub` vorhanden sind und die Signatur des Image-**Digests** in diesem Lauf geprüft wurde (ausgeführt wird genau dieser Digest);
|
||||
sonst und bei jedem Fehler läuft das Update mit der vorhandenen CLI weiter. Abschalten: `OPNEXUS_NO_CLI_REFRESH=1`. Wirksam wird es ab dem Update
|
||||
*von* dieser Version.
|
||||
|
||||
## [1.30.0] "Owl" — 2026-10-22
|
||||
|
||||
- **Regel-Analyse: Bereinigung (nur Ansicht):** Auf **Regelwerk › Regel-Analyse** gibt es den Tab **Bereinigung** (`?tab=cleanup`). Er listet je Firewall die
|
||||
Regeln, die einen Blick wert sind: **ungenutzt** (OPNexus hat in 7/30/90/180 Tagen keinen Treffer gesehen), **verdeckt** (eine frühere Quick-Regel
|
||||
trifft schon alles, was sie trifft), **doppelt** und **lange deaktiviert**. Jeder Befund trägt eine Sicherheit (**sicher / vermutlich / unbekannt**, Tooltip mit
|
||||
dem Grund): „sicher“ gibt es nur mit vollständigen, frischen Belegen, bei einer Regel mit Zeitplan, ohne geladenen pf-Regelsatz oder bei Aliasen bleibt es
|
||||
höchstens „vermutlich“. Die Tabelle zeigt auch den letzten Treffer und ob die Regel entfernbar wäre (nicht: automatische Regeln, Regeln im Altformat, von OPNexus
|
||||
verwaltete Regeln – die entfernt man über die Regelseite). Es wird nur gelesen; **Entfernen folgt in einer späteren Version**. Die Sicherheitslage verlinkt ihre
|
||||
Befunde „ungenutzt/verdeckt/doppelt“ direkt dorthin; per API `GET /api/hosts/{id}/rule-analysis/cleanup?unused_days=30`, per MCP `rule_cleanup_candidates`.
|
||||
- **Änderungsverlauf je Regel:** OPNexus liest alle 15 Minuten die Regeln jeder Firewall und merkt sich, was sich geändert hat – auch bei Regeln, die nicht von OPNexus
|
||||
stammen: angelegt, geändert (mit Vorher/Nachher), aktiviert, deaktiviert, entfernt, jeweils mit der Herkunft „über OPNexus“ (passender Audit-Eintrag) oder „extern“.
|
||||
Auf der Live-Regel-Ansicht (Regelwerk › Live über alle Hosts) öffnet die Zeilenaktion **Verlauf** die Ereignisse zusammen mit den gespeicherten Fassungen und dem Audit der
|
||||
verwalteten Regel. Die erste Beobachtung speichert nur den Ausgangszustand; der Verlauf wird 365 Tage aufbewahrt (`GET /api/hosts/{id}/live-rules/{uuid}/history`).
|
||||
Eine Umsortierung (andere Reihenfolge oder Gruppe) gilt als „verschoben“, nicht als Änderung. Eine leere oder unvollständige Antwort der Firewall (ohne
|
||||
automatische Regeln) wird verworfen; verschwinden mehr als die Hälfte der Regeln auf einmal, zählt erst die zweite gleiche Beobachtung. Die Herkunft „über OPNexus“
|
||||
erklärt nur Audit-Einträge seit der vorherigen Beobachtung, und jeder Eintrag nur eine Änderung.
|
||||
Schemaänderungen 0027 und 0028 (Tabellen und ein Feld für den Beobachtungszeitraum) laufen beim Start.
|
||||
- **Regelnutzung genauer:** Die Zähler kommen jetzt aus `ruleStats` (von der Firewall schon je Regel summiert; ältere Versionen weiterhin über den pf-Regelsatz) und werden alle
|
||||
15 Minuten statt alle 6 Stunden gelesen. Regeln im Altformat (noch nicht migriert) werden ebenfalls erfasst. Das Beobachtungsfenster für „ungenutzt“ beginnt neu, wenn eine Regel eingeschaltet oder geändert wird, und nach einem pf-Neuladen mit langer Lücke bei den Zählern.
|
||||
- **Verdeckt/doppelt in der Sicherheitslage genauer:** Der Vergleich kennt jetzt die echte Reihenfolge (Floating vor Interface-Gruppen vor Interface-Regeln), Netze und
|
||||
Portbereiche als Mengen (z. B. 10.1.0.0/16 liegt in 10.0.0.0/8), `inet46` und `tcp/udp`, und übergeht Regeln mit Zeitplan, Markierung oder Limits. Es findet dadurch
|
||||
mehr verdeckte Regeln als bisher, nie weniger belegte.
|
||||
|
||||
## [1.29.0] "Caracara" — 2026-10-22
|
||||
|
||||
- **Fehlende API-Rechte werden sichtbar:** Darf der API-Schlüssel einer Firewall den Gateway- oder Dienststatus nicht lesen (OPNsense antwortet
|
||||
mit 403), fiel das bisher niemandem auf, und der standardmäßig aktive Gateway-Alarm konnte nie auslösen. Jetzt zeigen die Host-Seite
|
||||
(Abschnitt „Gateways“) und die Dashboard-Karte „Fleet-/CARP-Gesundheit“ ein Warn-Badge „API-Rechte fehlen: Gateways, Dienste“; der Tooltip nennt die
|
||||
OPNsense-Berechtigung, die beim Benutzer oder bei der Gruppe fehlt (System: Gateways, Status: Services, Lobby: Dashboard). Zusätzlich gibt es
|
||||
den neuen Alarm **API-Rechte fehlen** (standardmäßig an, abschaltbar unter Einstellungen › Benachrichtigungen, gleiche Entprellung wie die
|
||||
anderen Messungs-Alarme) und die Prometheus-Metrik `opnexus_host_api_rights_missing{host,read}`. Nur eine ausdrückliche Ablehnung zählt;
|
||||
Zeitüberschreitungen oder Serverfehler bleiben „unbekannt“.
|
||||
- **Gateway-Alarm pro Gateway abschaltbar:** In der Gateway-Liste der Host-Seite schaltet eine Glocke den Alarm für ein einzelnes Gateway aus
|
||||
(z. B. ruhendes Zweit-WAN oder tote IPv6-Gateways); ein „Alarm aus“-Badge zeigt es an. Ein offener Alarm dieses Gateways löst sich dabei
|
||||
auf. Nur Administratoren, die Änderung steht im Audit-Log. Das Dashboard zeigt bei mehr als acht Gateways „+ N weitere“.
|
||||
- **Firewall-Oberfläche (GUI-Zugriff) härter:** Ein frei gewordener Platz wird für eine **andere** Firewall erst 5 Minuten nach dem Ende der letzten
|
||||
Sitzung vergeben (Schutz, falls dort noch ein Tab offen ist, der die Cookies des Platzes teilt); dieselbe Firewall bekommt ihn sofort wieder.
|
||||
Sind alle Plätze belegt oder noch in der Abkühlzeit, nennt die Meldung den Grund. Ändern sich Adresse, festgesetztes Zertifikat oder Tunnelweg
|
||||
einer Firewall, während ihre Oberfläche offen ist, endet die Sitzung spätestens nach 20 Sekunden. Das Limit von 64 gleichzeitigen Sitzungen
|
||||
verdrängt keine laufende Sitzung mehr: eine neue wird abgelehnt (mit Meldung), bis eine endet. **Hinweis:** Das GUI-Gateway-Image muss zur
|
||||
Backend-Version passen (wie bisher beim Update beider Container).
|
||||
- **Anmeldung (Cookies) härter:** Das alte Sitzungs-Cookie `opnexus-session` wird bei HTTPS **nicht mehr akzeptiert**. **Alle, die sich seit der
|
||||
Umstellung auf 1.27 nicht neu angemeldet haben, müssen sich einmal neu anmelden**; Skripte mit rohem Sitzungs-Cookie brauchen den Namen
|
||||
`__Host-opnexus-session` (API-Tokens sind nicht betroffen, ohne HTTPS bleibt `opnexus-session` der normale Name). Beim Abmelden wird ein altes
|
||||
Cookie weiterhin gelöscht. Das Cookie, das die SSO-Anmeldung an den Browser bindet, heißt bei HTTPS jetzt `__Host-opnexus-oidc-state`; der
|
||||
Rückweg vom Anbieter wird nur noch als Seitenaufruf (Navigation) von einer fremden Seite zugelassen, und ohne dieses Cookie scheitert er immer.
|
||||
- **Webhook, Benachrichtigungskanäle und SSO-Anbieter: Schutz vor DNS-Rebinding:** Bisher prüfte OPNexus die Adresse, auf die ein Name zeigt, und
|
||||
der HTTP-Client löste den Namen danach noch einmal auf; ein Server unter Fremdkontrolle konnte beim zweiten Mal eine interne Adresse liefern. Jetzt
|
||||
verbindet OPNexus genau zu der geprüften Adresse (Host-Name, TLS-Zertifikatsprüfung und Anfrage bleiben beim Namen); zeigt eine Antwort auch nur
|
||||
teilweise auf Loopback, Link-Local oder einen Cloud-Metadaten-Dienst (auch AWS-IPv6 und Alibaba), kommt keine Verbindung zustande und die Zustellung meldet „Zieladresse nicht
|
||||
erlaubt“. Private Netze bleiben erlaubt (eigener ntfy, Gotify, Matrix). Proxy-Umgebungsvariablen werden für diese Aufrufe nicht mehr beachtet.
|
||||
- **Benachrichtigungen parallel zugestellt:** Webhook, E-Mail und alle Kanäle werden gleichzeitig bedient, jeder mit eigener Zeitgrenze (20 Sekunden),
|
||||
und ein Prüfdurchlauf sendet insgesamt höchstens 90 Sekunden. Ein hängender Kanal verzögert weder die anderen noch die nächsten Prüfungen; im Log
|
||||
steht dann „timed out“ (ohne Adresse oder Token). Wie bisher wird einmal je Ereignis zugestellt, ohne Wiederholung, **außer** der Durchlauf war so voll, dass
|
||||
für ein Ereignis noch gar kein Versuch stattfand: es wird im nächsten Durchlauf zugestellt. Die Zustellung beginnt jetzt nach allen Prüfungen
|
||||
eines Durchlaufs statt mitten darin.
|
||||
- **Rollouts: Fingerabdruck, verantwortliche Person, überlappende Termine:** Der Fingerabdruck eines aktivierten Plans umfasst jetzt auch die
|
||||
ausgelassenen und verschobenen Termine seines Fensters: **wer einen Termin auslässt oder verschiebt, muss einen automatischen Plan danach neu
|
||||
aktivieren** (ohne solche Ausnahmen bleiben bestehende Aktivierungen gültig; **Pläne, deren Fenster schon Ausnahmen haben, nach dem Update
|
||||
einmal neu aktivieren**, sonst meldet der Plan „Plan geändert“ und startet nicht automatisch). Die verantwortliche Person (Aktivierende, Startende, Freigebende)
|
||||
wird zusätzlich als Konto-ID gespeichert (Migration 0026, bestehende Einträge werden über den Namen zugeordnet): ein gelöschtes und unter gleichem
|
||||
Namen neu angelegtes Konto gilt nicht mehr als dieselbe Person, der Lauf pausiert bzw. der automatische Start unterbleibt mit „Person darf nicht mehr
|
||||
ausrollen“. Angezeigt wird weiter der Benutzername. Überlappen sich zwei Termine eines Wartungsfensters, zählt der zuletzt begonnene (bisher der
|
||||
früheste), sodass ein älterer, längst abgeschnittener Termin keinen neuen Start mehr blockiert.
|
||||
- **Kleinigkeiten:** Ressourcen-Alarme zeigen bei fehlendem Messwert keine Zahl mehr („None %“ entfällt). Der Dialog „Von Firewall übernehmen“ zeigt ohne aktive
|
||||
Firewall einen Hinweis statt dauerhaft „Lädt…“. Fehlermeldungen beim Übernehmen und Verknüpfen nennen die Gründe in Klartext (Deutsch/Englisch) statt
|
||||
roher Codes wie `name_exists`; dasselbe gilt für „GUI-Zugriff nicht eingerichtet“ und blockierte Netzwerkänderungen. Der Test-Knopf der
|
||||
Benachrichtigungskanäle ist für Nicht-Admins ausgeblendet (der Server verweigert den Test ohnehin). Der Tooltip der aktiven Plätze nennt keine
|
||||
Umgebungsvariable mehr. Eine Gateway-Sitzung, die der Gateway-Container nicht mehr bestätigt (Absturz), zählt für die Abkühlzeit der Plätze als
|
||||
beendet.
|
||||
- **Grafana-Dashboard erweitert:** neue Felder für Gateways (erreichbar, Paketverlust, Latenz), Swap, gestoppte Dienste, fehlende API-Rechte und
|
||||
Rollouts nach Status; dazu die Metrik `opnexus_rollout_runs{status}`.
|
||||
|
||||
## [1.28.0] "Sparrowhawk" — 2026-10-21
|
||||
|
||||
- **Übernehmen: mit vorhandenem zentralem Objekt verknüpfen:** Dieselbe von Hand angelegte Regel, derselbe Alias oder dieselbe NAT-Regel auf mehreren
|
||||
Firewalls (z. B. beide HA-Knoten) wird nicht mehr als zweites Objekt übernommen. Im Dialog „Von Firewall übernehmen“ erscheint bei exakter
|
||||
Übereinstimmung (nach Einsetzen der Host-Variablen, bei Aliasen inkl. Verweisen) die Aktion **Verknüpfen**: die Firewall wird weiteres Ziel des
|
||||
vorhandenen Objekts, die Vorschau zeigt „keine Änderung“, und der nächste Rollout aktualisiert alle Firewalls an einem Objekt. Auf der Firewall wird
|
||||
dabei nichts geändert. Unterscheidet sich ein gleichnamiges Objekt, zeigt der Dialog die abweichenden Felder; verknüpft wird dann nie. „Alle
|
||||
übernehmbaren“ fragt vorher nach („3 verknüpfen, 2 neu übernehmen, 1 nicht übernehmbar“). Aliase, die bisher wegen „Name existiert“ abgelehnt wurden,
|
||||
lassen sich so übernehmen.
|
||||
Wartet das vorhandene Objekt gerade auf eine Bestätigung, wird nicht verknüpft; „Alle übernehmbaren“ lässt den Eintrag dann aus. Verknüpfen
|
||||
macht ein gerade geöffnetes Bearbeiten-Formular des Objekts ungültig, damit die neue Firewall nicht versehentlich wieder aus den Zielen fällt.
|
||||
|
||||
- **Tooltips überarbeitet:** Der deaktivierte Button „Oberfläche öffnen“ nennt jetzt den Grund im Klartext samt nächstem Schritt (statt nur einer
|
||||
Variablen wie `GUI_GATEWAY_TOKEN`); dasselbe gilt für ausgeschaltete Funktion, deaktivierte Firewall, Nicht-HTTPS-Adresse und ausgeschaltete
|
||||
Zertifikatsprüfung. Texte durchgehend in Du-Form, kürzere VPN-Einleitungen, verständlichere Hinweise bei gesperrten Buttons (Zertifikate, Benutzer,
|
||||
Dienste). Icon-Buttons (Bearbeiten, Löschen, Schließen, Reihenfolge) zeigen beim Darüberfahren ihren Namen.
|
||||
|
||||
## [1.27.2] "Buzzard" — 2026-10-21
|
||||
|
||||
- **`opnexus update` prüft den Proxy vor dem Wechsel:** Nach dem Ziehen der Images validiert das Werkzeug die Caddy-Konfiguration des neuen
|
||||
Proxy-Images mit den Umgebungsvariablen *dieser* Installation. Ist sie ungültig, bricht das Update ab, bevor etwas gewechselt wurde.
|
||||
Der Fall aus 1.27.0 (Compose-Datei mit leerem `OPNEXUS_GUI_SITES`) wird dabei automatisch repariert: Das Werkzeug trägt den Platzhalter
|
||||
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` ein und sagt es. Installationen, die noch die Compose-Datei aus 1.27.0 haben,
|
||||
müssen also nur einmal `opnexus update` ausführen.
|
||||
- **`opnexus update` erkennt einen abstürzenden Proxy:** Ein Dienst ohne Healthcheck (der Proxy) gilt jetzt erst nach 10 Sekunden
|
||||
durchgehender Laufzeit als gesund. Bisher meldete das Update „Fertig“, obwohl der Proxy in einer Neustart-Schleife hing, und das
|
||||
automatische Zurückrollen griff nicht.
|
||||
|
||||
## [1.27.1] "Buzzard" — 2026-10-21
|
||||
|
||||
- **Fix (1.27.1): Proxy startete mit der neuen Compose-Datei nicht.** Mit der `docker-compose.yml` aus 1.27.0 gab Compose die Variable
|
||||
`OPNEXUS_GUI_SITES` als leeren Text an den Proxy weiter; Caddy nutzt seinen Standardwert aber nur bei *nicht gesetzter* Variable und
|
||||
brach mit „server block without any key“ ab, OPNexus war dann nicht mehr erreichbar. Die Compose-Datei setzt jetzt einen gültigen
|
||||
Platzhalter als Standard. Betroffen waren nur Installationen, die die neue Compose-Datei aus 1.27.0 übernommen hatten (ohne
|
||||
`OPNEXUS_GUI_SITES` in der `.env`); wer nur die Images aktualisiert hatte, war nicht betroffen. Sofort-Abhilfe ohne Update:
|
||||
`OPNEXUS_GUI_SITES=https://localhost:9441` in die `.env` eintragen und `docker compose up -d proxy` ausführen.
|
||||
|
||||
## [1.27.0] "Buzzard" — 2026-10-21
|
||||
|
||||
Vorgesehen als MINOR-Release **1.27.0** „Firewall-Oberfläche über OPNexus“.
|
||||
|
||||
- **Firewall-Oberfläche über OPNexus:** Auf der Host-Seite öffnet **Oberfläche öffnen** die Weboberfläche der Firewall
|
||||
in einem neuen Tab, auch bei Firewalls hinter NAT im Management-Tunnel. Nur für Admins; jede Firewall läuft in einer
|
||||
eigenen Browser-Origin (Platz = Port), Cookies bleiben auf dem Server, das Ticket gilt einmalig 30 Sekunden,
|
||||
Inaktivitäts- und Gesamtlimit sind einstellbar, Ticket, Start und Ende stehen im Audit (Art „GUI-Zugriff“).
|
||||
Standardmäßig **aus**: braucht das Compose-Profil `gui`, `OPNEXUS_GUI_SITES`, `GUI_PORTS`, `GUI_GATEWAY_TOKEN`,
|
||||
`COOKIE_SECURE=true` und den Schalter unter Einstellungen (Handbuch, „Firewall-Oberfläche über OPNexus“).
|
||||
Live getestet auf OPNsense 26.7.
|
||||
- **Einschränkungen:** kein WebSocket, keine automatische Anmeldung, höchstens 4 Plätze gleichzeitig. OPNexus-Adresse
|
||||
vorher in die Whitelist des Anmeldeschutzes (sshguard) der Firewall eintragen, sonst können Fehlanmeldungen die
|
||||
API-Zugriffe von OPNexus aussperren. Nach einem Neustart des wg-hub braucht der Tunnel 2 bis 4 Minuten.
|
||||
- **Hinweis für Betreiber mit eigenen Skripten:** Bei HTTPS heißt das Sitzungs-Cookie jetzt `__Host-opnexus-session`
|
||||
(bestehende Sitzungen werden ohne Abmelden übernommen), und Cookie-Anfragen an `/api` mit `Sec-Fetch-Site: same-site/cross-site`
|
||||
werden abgelehnt. Zugriffe mit API-Token sind nicht betroffen.
|
||||
|
||||
## [1.26.0] "Kite" — 2026-10-21
|
||||
|
||||
Geplant als MINOR-Release **1.26.0** (Regel-Analyse). Backend, MCP und Oberfläche.
|
||||
|
||||
- **Regel-Analyse „Welche Regel greift?“ (API):** `GET /api/hosts/{id}/rule-analysis/match` und
|
||||
`GET /api/rule-analysis/match` (ganze Flotte, Auswahl oder Gerätegruppe) beantworten für Quelle, Ziel, Protokoll
|
||||
und Port, ob eine Firewall das erste Paket einer neuen Verbindung durchlässt oder blockiert, welche Regel das
|
||||
entscheidet und warum (Prüfprotokoll Regel für Regel). Grundlage ist der tatsächlich geladene pf-Regelsatz
|
||||
inklusive automatischer Regeln, Floating- und Gruppenregeln, `quick`/letzter Treffer, Negation, verschachtelter
|
||||
Aliase (pf-Tabellentest der Firewall), Port-Aliase und Adressfamilien. Jede Antwort trägt eine ehrliche
|
||||
Konfidenz **sicher / vermutlich / unbekannt** mit Begründungscodes; was nicht belegbar ist (z. B. `tagged`,
|
||||
nicht auflösbare Aliase, geratenes Interface, NAT in der Nähe), wird nie als „sicher“ gemeldet. Nur lesend,
|
||||
auch mit `read`-Token nutzbar; es wird nichts gespeichert und nichts an der Firewall geändert.
|
||||
- **MCP:** neues Werkzeug `rule_match` für KI-Assistenten (gleiche Abfrage, ohne Prüfprotokoll).
|
||||
- **Seite „Regel-Analyse“ (Regelwerk):** Formular für Hosts oder Gerätegruppe, Quell-/Ziel-IP, Protokoll, Ports,
|
||||
Eingangs-Interface („automatisch“ oder Liste) und optional die Ausgangsseite. Die Abfrage steht in der Adresszeile
|
||||
(teilbar, wird beim Öffnen sofort ausgeführt). Je Firewall eine Karte mit Erlaubt/Blockiert/Abgewiesen, Sicherheit
|
||||
des Urteils, entscheidender Regel und Hinweisen; ein „unbekannt“-Ergebnis erscheint bewusst nicht als Urteil.
|
||||
Aufklappbares „Warum?“ zeigt jede geprüfte Regel mit ✓/✗/? je Feld, überstimmte Regeln durchgestrichen. Auch mit
|
||||
Nur-Lesen-Rolle nutzbar.
|
||||
|
||||
## [1.25.0] "Goshawk" — 2026-10-21
|
||||
|
||||
Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).
|
||||
|
||||
- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
|
||||
`ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
|
||||
bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
|
||||
gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
|
||||
folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
|
||||
Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
|
||||
(nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
|
||||
Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
|
||||
Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
|
||||
- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
|
||||
Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
|
||||
nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
|
||||
eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
|
||||
älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
|
||||
- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
|
||||
Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
|
||||
zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
|
||||
(nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
|
||||
der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
|
||||
- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
|
||||
vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
|
||||
erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
|
||||
sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
|
||||
- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
|
||||
- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
|
||||
gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
|
||||
laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
|
||||
sicher, wird aber angekündigt.
|
||||
- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
|
||||
26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
|
||||
Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
|
||||
Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
|
||||
übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
|
||||
Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
|
||||
Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.
|
||||
|
||||
## [1.24.0] "Merlin" — 2026-10-21
|
||||
|
||||
- **Alarme für Gateways, Ressourcen und Dienste:** Neue Ereignisse in Einstellungen › Benachrichtigungen.
|
||||
**Gateway gestört** (standardmäßig **an**) meldet ein Gateway, das OPNsense als „down“ führt oder dessen Paketverlust (ab 20 %) bzw.
|
||||
Latenz (ab 500 ms) über der Schwelle liegt; bewusst abgeschaltete und nicht überwachte Gateways bleiben still. **Ressourcen-Schwelle**
|
||||
(standardmäßig aus) prüft Festplatte (ab 90 %), Arbeitsspeicher (ab 95 %), Swap (ab 50 %) und CPU-Last (ab 2 pro Thread), **Dienst ausgefallen**
|
||||
(standardmäßig aus) eine einstellbare Liste (Unbound, Dnsmasq, DHCP, NTP, OpenVPN, WireGuard, IPsec …) und meldet nur Dienste, die zuvor
|
||||
liefen. Alle drei stützen sich auf die Minutenmessung und lösen erst aus, wenn die letzten 3 (Ressourcen: 5) Messungen übereinstimmen, und
|
||||
enden erst, wenn alle wieder in Ordnung sind: Einzelspitzen und flackernde Verbindungen schicken nichts. Der Gateway-Status (Verlust, Latenz,
|
||||
Abweichung) steht außerdem auf der Host-Seite, als Kachel auf dem Dashboard und als Prometheus-Metriken (`opnexus_host_gateway_*`,
|
||||
`opnexus_host_swap_used_percent`, `opnexus_host_service_running`). Die Messung speichert dafür zusätzlich Swap, CPU-Threads, Gateways
|
||||
und Dienste in der vorhandenen Messreihe (keine neue Tabelle).
|
||||
- **Weitere Benachrichtigungskanäle:** ntfy, Gotify, Telegram, Microsoft Teams (Workflows-/Incoming-Webhook) und Matrix, beliebig viele
|
||||
je Typ, mit „Testen“-Knopf pro Kanal. Zugangsdaten (Tokens, die Teams-Adresse) liegen wie das SMTP-Passwort in der Geheimnis-Ablage, nie
|
||||
in der Datenbank oder in Antworten, und tauchen nicht in Fehlermeldungen oder Logs auf. Webhook und neue Kanäle prüfen die Zieladresse:
|
||||
nur http(s), keine Zugangsdaten in der Adresse, keine lokalen/Link-Local-/Metadaten-Adressen; Weiterleitungen werden nicht verfolgt.
|
||||
Betrifft die bestehende Webhook-Adresse: `localhost`, Loopback, 169.254.x.x und Zugangsdaten in der Adresse (`https://user:pass@…`)
|
||||
werden jetzt abgelehnt. Eine solche gespeicherte Adresse wird nicht mehr beliefert (nur ein Log-Eintrag), und die
|
||||
Benachrichtigungs-Einstellungen lassen sich erst wieder speichern, wenn sie korrigiert oder geleert ist (die Meldung nennt den Grund).
|
||||
Neue Einstellungen kommen mit Migration 0020.
|
||||
|
||||
- **Bestand übernehmen für Aliase und NAT:** Auf den Seiten Aliase und NAT (Port Forward, 1:1, Outbound) gibt es jetzt **Von Firewall übernehmen**
|
||||
wie bei den Regeln. Der Dialog zeigt, was von Hand in OPNsense angelegt wurde, einzeln zum Übernehmen oder mit „Alle übernehmbaren“ in einem Schritt;
|
||||
verschachtelte Aliase werden in der richtigen Reihenfolge übernommen. Die Übernahme schreibt nichts auf die Firewall: Die Vorschau zeigt direkt
|
||||
danach „keine Änderung“, der nächste Rollout aktualisiert dasselbe Objekt (kein Duplikat). Nicht übernommen werden OPNsense-eigene Aliase,
|
||||
Typen, die OPNexus nicht kennt (z. B. URL, GeoIP), Aliase mit Verweis auf nicht übernommene Aliase sowie NAT-Regeln, die das Modell verfälschen würden
|
||||
(z. B. „Kein Redirect“, verknüpfte Filterregel, mehrere Schnittstellen, „Kein NAT“); die Oberfläche nennt den Grund je Objekt.
|
||||
Outbound-NAT lässt sich nur übernehmen, wenn der Modus der Firewall Hybrid oder Manuell ist. Einstellungen, die OPNexus nicht verwaltet (Protokollierung,
|
||||
Pool-Optionen, Tags, Kategorien), bleiben unverändert und werden angezeigt.
|
||||
|
||||
- **Images für ARM:** Alle Release-Images (backend, frontend, proxy, updater) gibt es jetzt für `linux/amd64` und `linux/arm64`
|
||||
(Raspberry Pi 4/5, ARM-Server, Apple-Silicon-Docker).
|
||||
|
||||
## [1.23.11] "Harrier" — 2026-10-20
|
||||
|
||||
|
||||
+12
-6
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||

|
||||
|
||||
> **Stand:** `1.23.11` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.33.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -16,14 +16,18 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
- **Sichere Änderungen** — jede Änderung läuft *Prüfen → Ausrollen → Bestätigen*; bleibt die Bestätigung im Zeitfenster aus, rollt OPNexus zurück.
|
||||
- **Hierarchische Policies** — Pre-/Post-Rulebases mit Vererbung über Gruppenbäume.
|
||||
- **VPN** — IPsec-Site-to-Site, OpenVPN und WireGuard, einmal definiert und ausgerollt.
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln lassen sich übernehmen.
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln, Aliase und NAT-Regeln lassen sich übernehmen.
|
||||
- **Zertifikate** — Trust-Store für alle Firewalls und Ausstellung per ACME / Let's Encrypt.
|
||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||
- **Update-Center** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
|
||||
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
|
||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
|
||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit); ein Administrator kann ausgewählte Regeln abschalten (Vorschau, dann Ausrollen mit automatischer Rücknahme ohne Bestätigung), später löschen und aus einer Sicherung des Regelinhalts wiederherstellen. Jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
|
||||
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
|
||||
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
|
||||
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
|
||||
@@ -44,7 +48,7 @@ Voraussetzung: Docker mit Compose-Plugin.
|
||||
```bash
|
||||
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
||||
cd opnexus
|
||||
cp .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen
|
||||
install -m 600 .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen; nur für dich lesbar
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
@@ -56,13 +60,15 @@ scripts/opnexus password
|
||||
|
||||
Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassistent**: eigenes Passwort setzen, optional Zwei-Faktor-Anmeldung einschalten, die erste Firewall verbinden (samt Anleitung für den API-Key in OPNsense). Er lässt sich überspringen und jederzeit unter System › Einrichtung wieder öffnen.
|
||||
|
||||
**Windows:** läuft in Docker Desktop mit WSL 2, siehe die [Installationsanleitung für Windows](https://opnexus.dev/docs/windows/). Entwickelt und getestet wird unter Linux; für den Dauerbetrieb ist ein Linux-Server oder eine Linux-VM besser.
|
||||
|
||||
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
|
||||
|
||||
**Fertige Images** (amd64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.23.11
|
||||
OPNEXUS_VERSION=1.33.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||

|
||||
|
||||
> **Status:** `1.23.11` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.33.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -16,14 +16,18 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
- **Safe changes** — every change goes *preview → deploy → confirm*; without your confirmation inside the time window OPNexus rolls it back.
|
||||
- **Hierarchical policies** — pre/post rulebases with inheritance across group trees.
|
||||
- **VPN** — IPsec site-to-site, OpenVPN and WireGuard, defined once and rolled out.
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules created by hand on a firewall can be adopted.
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules, aliases and NAT rules created by hand on a firewall can be adopted.
|
||||
- **Certificates** — trust store for all firewalls and ACME / Let's Encrypt issuance.
|
||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||
- **Update center** — firmware updates for the fleet, aware of HA order and maintenance mode.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
|
||||
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
|
||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
|
||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence); an administrator can switch selected rules off (preview, then roll-out with an automatic revert unless confirmed), delete them later and restore them from a backup of their content. Every rule has a change history, including changes made outside OPNexus.
|
||||
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
|
||||
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
|
||||
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
|
||||
@@ -46,7 +50,7 @@ Requires Docker with the Compose plugin.
|
||||
```bash
|
||||
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
||||
cd opnexus
|
||||
cp .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD)
|
||||
install -m 600 .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD); readable by you only
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
@@ -58,13 +62,15 @@ scripts/opnexus password
|
||||
|
||||
After the first sign-in a short **setup assistant** opens on its own: set your own password, optionally enable two-factor sign-in, connect your first firewall (including how to create the API key in OPNsense). You can skip it and reopen it any time under System › Setup.
|
||||
|
||||
**Windows:** run it in Docker Desktop with WSL 2 – see the [installation guide for Windows](https://opnexus.dev/en/docs/windows/). OPNexus is developed and tested on Linux; for permanent operation a Linux server or VM is the better choice.
|
||||
|
||||
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
|
||||
|
||||
**Prebuilt images** (amd64) are published to this Gitea's registry. In `.env`:
|
||||
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.23.11
|
||||
OPNEXUS_VERSION=1.33.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ from starlette.responses import JSONResponse
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.errors import ErrorCode
|
||||
from app.routers.auth import COOKIE_NAME
|
||||
from app.routers.auth import session_token
|
||||
from app.routers import governance
|
||||
from app.routers.auth import client_ip
|
||||
from app.services import api_token_service, auth_service
|
||||
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
"/api/settings",
|
||||
# Read-only JSON-RPC tools (see routers/mcp.py).
|
||||
"/api/mcp",
|
||||
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
|
||||
"/api/config/import/plan",
|
||||
# Self-scoped: a user managing their own second factor.
|
||||
"/api/auth/2fa/setup",
|
||||
"/api/auth/2fa/enable",
|
||||
@@ -56,9 +58,9 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
_SECURITY_KINDS = {
|
||||
"ipsec": "ipsec_tunnel", "openvpn": "openvpn_instance", "wireguard": "wireguard_tunnel",
|
||||
"certificates": "certificate", "acme": "acme", "notifications": "notification", "auth": "auth",
|
||||
"tokens": "api_token",
|
||||
"tokens": "api_token", "rollouts": "rollout",
|
||||
}
|
||||
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings"}
|
||||
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings", "plans", "runs"}
|
||||
|
||||
|
||||
def _is_uuid(value: str) -> bool:
|
||||
@@ -94,10 +96,14 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if request.url.path in _PUBLIC_PATHS:
|
||||
return await call_next(request)
|
||||
if request.url.path.startswith("/internal/gui/"):
|
||||
# GUI gateway only: never routed by the proxy, authenticated by the
|
||||
# shared gateway token inside routers/gui_access.py.
|
||||
return await call_next(request)
|
||||
|
||||
api_token = None
|
||||
bearer = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
if bearer is not None:
|
||||
# An API token wins over a cookie; a bad one never falls back to it.
|
||||
async with SessionLocal() as db:
|
||||
@@ -117,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
)
|
||||
if user is not None:
|
||||
parts = request.url.path.strip("/").split("/")
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup"}
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
|
||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||
@@ -127,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
if len(parts) > 2 and parts[1] == "settings":
|
||||
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
|
||||
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
|
||||
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
|
||||
entity = parts[2] if len(parts) > 2 and kind != "config" else None
|
||||
event = {"kind": kind, "action": action, "entity_id": entity,
|
||||
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
|
||||
"details": {"method": request.method, "path": request.url.path}}
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
|
||||
|
||||
if (
|
||||
user.role == "read_only"
|
||||
|
||||
@@ -6,6 +6,17 @@ class Settings(BaseSettings):
|
||||
def allowed_origins(self) -> list[str]:
|
||||
return [o.strip().rstrip("/") for o in self.frontend_origin.split(",") if o.strip()]
|
||||
|
||||
@property
|
||||
def gui_site_origins(self) -> list[str]:
|
||||
return [o.strip().rstrip("/") for o in self.opnexus_gui_sites.split(",") if o.strip()]
|
||||
|
||||
def gui_origin_conflicts(self) -> list[str]:
|
||||
"""Slot origins of the GUI gateway that are also listed in FRONTEND_ORIGIN.
|
||||
Such an origin would get CORS with credentials and pass the Sec-Fetch-Site
|
||||
exception — a firewall GUI could then read OPNexus' API. Startup is refused."""
|
||||
allowed = {o.lower() for o in self.allowed_origins}
|
||||
return [o for o in self.gui_site_origins if o.lower() in allowed]
|
||||
|
||||
database_url: str = (
|
||||
"postgresql+asyncpg://opnexus:opnexus@postgres:5432/opnexus"
|
||||
)
|
||||
@@ -92,6 +103,14 @@ class Settings(BaseSettings):
|
||||
# per-IP lockout entirely.
|
||||
trust_proxy_headers: bool = False
|
||||
|
||||
# ── GUI gateway (1.27, optional container "gui-gateway") ─────────
|
||||
# Slot origins the gateway is served on, comma-separated, e.g.
|
||||
# "https://192.0.2.10:9441,https://192.0.2.10:9442". Empty = feature unavailable.
|
||||
opnexus_gui_sites: str = ""
|
||||
# Shared secret between backend and gateway for /internal/gui/* (at least
|
||||
# 32 characters). Empty = the internal endpoints answer 404.
|
||||
gui_gateway_token: str = ""
|
||||
|
||||
# ── Deploy lockout protection ────────────────────────────────────
|
||||
deploy_confirm_window_seconds: int = 180
|
||||
|
||||
@@ -141,6 +160,13 @@ class Settings(BaseSettings):
|
||||
# wireguard_key_store.py.
|
||||
wireguard_key_secret_dir: str = "/data/wireguard-key-secrets"
|
||||
|
||||
# ── Rule observation (usage counters + change history of the live rules) ─
|
||||
# Two read-only GETs per enabled firewall per cycle (searchRule, ruleStats). The interval is an operator
|
||||
# setting (env RULE_OBSERVE_INTERVAL_SECONDS); it can not go below the floor, which keeps a typo from
|
||||
# polling every firewall in a tight loop.
|
||||
rule_observe_interval_seconds: int = 900
|
||||
rule_observe_min_interval_seconds: int = 60
|
||||
|
||||
# ── Resource monitoring ──────────────────────────────────────────────
|
||||
# Fixed interval, no admin-editable settings row/UI for this — v1
|
||||
# deliberately kept minimal (see resource_monitor_service.py).
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
|
||||
|
||||
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
|
||||
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
|
||||
"""
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
|
||||
|
||||
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
|
||||
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
|
||||
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
|
||||
|
||||
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
|
||||
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
|
||||
|
||||
ACTOR = "cli (Host-Konsole)"
|
||||
TOKEN_MARK = "# plan-token: "
|
||||
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
|
||||
|
||||
ERRORS_DE = {
|
||||
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
|
||||
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
|
||||
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
|
||||
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
|
||||
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
|
||||
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
|
||||
"yaml_empty": "Die Datei ist leer",
|
||||
"yaml_syntax": "YAML-Syntaxfehler",
|
||||
"duplicate_key": "Doppelter Schlüssel",
|
||||
"non_string_key": "Schlüssel muss Text sein",
|
||||
"too_deep": "Zu tief verschachtelt",
|
||||
"too_many_nodes": "Zu viele Elemente",
|
||||
"scalar_too_long": "Wert zu lang",
|
||||
"too_many_objects": "Zu viele Objekte",
|
||||
"too_many_entries": "Zu viele Einträge",
|
||||
"too_many_variables": "Zu viele Variablen für diesen Host",
|
||||
"too_large": "Datei zu groß",
|
||||
"bad_encoding": "Kein gültiges UTF-8",
|
||||
"not_a_mapping": "Die Datei muss ein Mapping sein",
|
||||
"missing_version": "opnexus_config: 1 fehlt",
|
||||
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
|
||||
"unknown_field": "Unbekanntes Feld",
|
||||
"missing_field": "Pflichtfeld fehlt",
|
||||
"bad_type": "Falscher Typ",
|
||||
"bad_value": "Ungültiger Wert",
|
||||
"bad_format": "Falsches Format",
|
||||
"too_short": "Zu kurz",
|
||||
"too_long": "Zu lang",
|
||||
"control_character": "Steuerzeichen nicht erlaubt",
|
||||
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
|
||||
"bad_variable_name": "Ungültiger Variablenname",
|
||||
"bad_variable_value": "Ungültiger Variablenwert",
|
||||
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
|
||||
"duplicate_entry": "Eintrag kommt doppelt vor",
|
||||
"unknown_host": "Unbekannter Host",
|
||||
"unknown_group": "Unbekannte Gerätegruppe",
|
||||
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
|
||||
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
|
||||
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
|
||||
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
|
||||
}
|
||||
REASONS_DE = {
|
||||
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
|
||||
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
|
||||
"prune_referenced": "wird noch verwendet",
|
||||
}
|
||||
WARNINGS_DE = {
|
||||
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
|
||||
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
|
||||
"unknown_alias_reference": "{value}: kein zentraler Alias in OPNexus (auf der Firewall von Hand angelegt? dann in Ordnung)",
|
||||
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
|
||||
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
|
||||
}
|
||||
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
|
||||
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
|
||||
|
||||
|
||||
def safe(text) -> str:
|
||||
"""Nothing from a foreign file reaches the terminal as a control sequence."""
|
||||
return _ANSI.sub("?", str(text))
|
||||
|
||||
|
||||
def _short(value) -> str:
|
||||
return safe(display(json.dumps(value, ensure_ascii=False), 60))
|
||||
|
||||
|
||||
def format_errors(errors: list[dict]) -> str:
|
||||
lines = []
|
||||
for e in errors:
|
||||
where = f"Zeile {e['line']}: " if e.get("line") else ""
|
||||
path = f"{safe(e['path'])}: " if e.get("path") else ""
|
||||
detail = ""
|
||||
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
|
||||
if params:
|
||||
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
|
||||
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_plan(public: dict) -> str:
|
||||
lines = []
|
||||
for item in public["items"]:
|
||||
action = item["action"]
|
||||
planned = item["planned_action"]
|
||||
name = f"{item['kind']} {safe(item['key'])}"
|
||||
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
|
||||
if action == "blocked":
|
||||
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
|
||||
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
|
||||
elif planned in LABEL:
|
||||
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
|
||||
else:
|
||||
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
|
||||
if change["field"]:
|
||||
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
|
||||
else:
|
||||
lines.append(f"~ {name}{suffix}")
|
||||
if item.get("implicit"):
|
||||
lines[-1] += " (Folge der Reihenfolge)"
|
||||
for warning in public["warnings"]:
|
||||
template = WARNINGS_DE.get(warning["code"])
|
||||
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
|
||||
text = template.format(**params) if template else warning["code"]
|
||||
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
|
||||
lines.append(f"? {where}{text}")
|
||||
s = public["summary"]
|
||||
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _read_stdin() -> bytes:
|
||||
data = sys.stdin.buffer.read(MAX_BYTES + 1)
|
||||
return data
|
||||
|
||||
|
||||
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
|
||||
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
|
||||
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
|
||||
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
|
||||
parser.add_argument("--kinds", default=None)
|
||||
parser.add_argument("--prune", action="store_true")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--plan-token", default=None)
|
||||
try:
|
||||
args = parser.parse_args(argv)
|
||||
except SystemExit:
|
||||
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
|
||||
if args.command == "schema":
|
||||
return 0, schema.json_schema_text(), ""
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.services import audit_service
|
||||
|
||||
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
if args.command == "export":
|
||||
wanted = None
|
||||
if args.kinds:
|
||||
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
|
||||
return 0, await export_yaml(db, wanted), ""
|
||||
|
||||
raw = stdin if stdin is not None else _read_stdin()
|
||||
try:
|
||||
parsed = parse(raw)
|
||||
if args.command == "plan":
|
||||
plan = await planner.build_plan(db, parsed, args.prune)
|
||||
await db.rollback()
|
||||
public = plan.public()
|
||||
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
|
||||
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
|
||||
s = public["summary"]
|
||||
if not plan.applicable:
|
||||
return 4, out, ""
|
||||
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
|
||||
|
||||
if not args.plan_token:
|
||||
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
|
||||
try:
|
||||
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
s = plan.summary()
|
||||
if args.json:
|
||||
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
|
||||
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
|
||||
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
|
||||
except ConfigError as exc:
|
||||
if args.json:
|
||||
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
|
||||
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
|
||||
finally:
|
||||
audit_service.context.reset(context)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
try:
|
||||
code, out, err = asyncio.run(run(argv))
|
||||
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
|
||||
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
|
||||
return 1
|
||||
sys.stdout.write(out)
|
||||
sys.stderr.write(err)
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,312 @@
|
||||
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
|
||||
|
||||
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
|
||||
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
|
||||
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
|
||||
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
|
||||
|
||||
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
|
||||
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
|
||||
Den Commit macht der Aufrufer.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import Parsed
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
|
||||
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.services import audit_service
|
||||
|
||||
NOTE = "Import (Konfiguration als Code)"
|
||||
AUDIT_KIND = {
|
||||
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
|
||||
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
|
||||
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
|
||||
}
|
||||
|
||||
|
||||
class PlanStale(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PlanBlocked(Exception):
|
||||
def __init__(self, plan: planner.Plan):
|
||||
self.plan = plan
|
||||
super().__init__("plan blocked")
|
||||
|
||||
|
||||
class SelfCheckFailed(Exception):
|
||||
def __init__(self, differences: list[str]):
|
||||
self.differences = differences
|
||||
super().__init__("; ".join(differences[:5]))
|
||||
|
||||
|
||||
def _revision_writer(kind: str):
|
||||
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
|
||||
|
||||
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
|
||||
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
|
||||
"""
|
||||
if kind == "rules":
|
||||
from app.services.policy_service import snapshot
|
||||
return RuleRevision, "rule_id", snapshot
|
||||
if kind == "aliases":
|
||||
from app.services.alias_policy_service import snapshot
|
||||
return AliasRevision, "alias_id", snapshot
|
||||
if kind == "port_forwards":
|
||||
from app.services.port_forward_policy_service import snapshot
|
||||
return PortForwardRevision, "port_forward_id", snapshot
|
||||
if kind == "one_to_one":
|
||||
from app.services.one_to_one_policy_service import snapshot
|
||||
return OneToOneRevision, "one_to_one_id", snapshot
|
||||
if kind == "outbound_nat":
|
||||
from app.services.outbound_nat_policy_service import snapshot
|
||||
return OutboundNatRevision, "outbound_nat_id", snapshot
|
||||
from app.services.policy_hierarchy_service import snapshot
|
||||
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
|
||||
|
||||
|
||||
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
|
||||
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
|
||||
SCALARS = {
|
||||
"aliases": ("name", "type", "description", "enabled"),
|
||||
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "enabled", "log"),
|
||||
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "target", "local_port", "natreflection", "enabled"),
|
||||
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
|
||||
"destination_not", "natreflection", "log", "enabled"),
|
||||
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
|
||||
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
|
||||
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
|
||||
}
|
||||
|
||||
|
||||
class _Writer:
|
||||
def __init__(self, db: AsyncSession, plan: planner.Plan):
|
||||
self.db = db
|
||||
self.plan = plan
|
||||
self.now = datetime.now(timezone.utc)
|
||||
self.events: list[dict] = []
|
||||
|
||||
async def load(self) -> None:
|
||||
db = self.db
|
||||
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
self.groups = {g.name: g for g in (await db.execute(
|
||||
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
|
||||
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
self.rows: dict[str, dict] = {}
|
||||
for kind, model in MODELS.items():
|
||||
options = [selectinload(model.deployments)]
|
||||
if kind != "policy_hierarchy_rules":
|
||||
options += [selectinload(model.targets), selectinload(model.groups)]
|
||||
rows = (await db.execute(select(model).options(*options))).scalars().all()
|
||||
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
|
||||
def items(self, kind: str, *actions: str) -> list[planner.Item]:
|
||||
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
|
||||
|
||||
def event(self, item: planner.Item, entity_id, name: str) -> None:
|
||||
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
|
||||
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
|
||||
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
|
||||
|
||||
# -- groups ----------------------------------------------------------------------------------------------------------
|
||||
async def hierarchy_groups(self) -> None:
|
||||
items = self.items("policy_hierarchy_groups", "create", "update")
|
||||
for item in items:
|
||||
if item.action == "create":
|
||||
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
|
||||
self.db.add(group)
|
||||
self.hierarchy[item.key] = group
|
||||
await self.db.flush()
|
||||
for item in items:
|
||||
group = self.hierarchy[item.key]
|
||||
group.description = item.after["description"]
|
||||
parent = item.after["parent"]
|
||||
group.parent_id = self.hierarchy[parent].id if parent else None
|
||||
self.event(item, group.id, group.name)
|
||||
await self.db.flush()
|
||||
|
||||
async def device_groups(self) -> None:
|
||||
for item in self.items("device_groups", "create", "update"):
|
||||
group = self.groups.get(item.key)
|
||||
if group is None:
|
||||
group = DeviceGroup(name=item.key)
|
||||
self.db.add(group)
|
||||
self.groups[item.key] = group
|
||||
group.description = item.after["description"]
|
||||
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
|
||||
await self.db.flush()
|
||||
self.event(item, group.id, group.name)
|
||||
|
||||
# -- objects ---------------------------------------------------------------------------------------------------------
|
||||
async def objects(self, kind: str) -> None:
|
||||
model = MODELS[kind]
|
||||
revision_model, fk, snapshot = _revision_writer(kind)
|
||||
for item in self.items(kind, "create", "update"):
|
||||
after = item.after
|
||||
row = self.rows[kind].get(item.key)
|
||||
if row is None:
|
||||
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
|
||||
if kind != "aliases":
|
||||
row.config_key = item.key
|
||||
self.db.add(row)
|
||||
self.rows[kind][item.key] = row
|
||||
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
|
||||
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
|
||||
for name in SCALARS[kind]:
|
||||
setattr(row, name, after[name])
|
||||
if kind == "aliases":
|
||||
row.content = "\n".join(after["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
row.hierarchy_group_id = self.hierarchy[after["group"]].id
|
||||
else:
|
||||
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
|
||||
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
|
||||
if "hierarchy_group" in after["targets"]:
|
||||
hierarchy = after["targets"]["hierarchy_group"]
|
||||
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
|
||||
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
|
||||
await self.db.flush()
|
||||
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
|
||||
async def hierarchy_positions(self) -> None:
|
||||
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
|
||||
if "policy_hierarchy_rules" not in self.plan.touched:
|
||||
return
|
||||
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
|
||||
for key, rank in self.plan.positions.items():
|
||||
row = self.rows["policy_hierarchy_rules"].get(key)
|
||||
if row is not None and key not in deleted and row.sequence != rank:
|
||||
row.sequence = rank
|
||||
await self.db.flush()
|
||||
|
||||
async def host_hierarchy(self) -> None:
|
||||
for host_name, group_name in self.plan.host_hierarchy.items():
|
||||
host = self.hosts[host_name]
|
||||
wanted = self.hierarchy[group_name].id if group_name else None
|
||||
if host.policy_hierarchy_group_id != wanted:
|
||||
host.policy_hierarchy_group_id = wanted
|
||||
await self.db.flush()
|
||||
|
||||
async def variables(self) -> None:
|
||||
if "variables" not in self.plan.touched:
|
||||
return
|
||||
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
|
||||
for item in self.items("variables", "create", "update"):
|
||||
host = self.hosts[item.after["host"]]
|
||||
row = existing.get((host.id, item.after["name"]))
|
||||
if row is None:
|
||||
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
|
||||
else:
|
||||
row.value = item.after["value"]
|
||||
self.event(item, None, item.key)
|
||||
for item in self.items("variables", "delete"):
|
||||
host = self.hosts[item.before["host"]]
|
||||
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
|
||||
self.event(item, None, item.key)
|
||||
await self.db.flush()
|
||||
|
||||
# -- deletions -------------------------------------------------------------------------------------------------------
|
||||
async def deletions(self) -> None:
|
||||
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
|
||||
for item in self.items(kind, "delete"):
|
||||
row = self.rows[kind][item.key]
|
||||
if row.deployments: # the planner blocks this; never undeploy from here
|
||||
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
for item in self.items("device_groups", "delete"):
|
||||
row = self.groups[item.key]
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
|
||||
by_id = {g.id: g for g in self.hierarchy.values()}
|
||||
|
||||
def depth(group) -> int:
|
||||
n, current = 0, group
|
||||
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
|
||||
n, current = n + 1, by_id[current.parent_id]
|
||||
return n
|
||||
|
||||
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
|
||||
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
|
||||
|
||||
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
|
||||
db.expire_all() # compare what is in the database, not what the session believes
|
||||
actual = await canonical(db, set(plan.touched))
|
||||
differences: list[str] = []
|
||||
for kind in plan.touched:
|
||||
if kind == "variables":
|
||||
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in actual["variables"].items() for n, v in values.items()}
|
||||
else:
|
||||
ident = planner.KIND_SPECS[kind].ident
|
||||
got = {item[ident]: item for item in actual[kind]}
|
||||
want = plan.expected[kind]
|
||||
for key in sorted(set(got) | set(want)):
|
||||
if got.get(key) != want.get(key):
|
||||
differences.append(f"{kind}:{key}")
|
||||
if "policy_hierarchy_rules" in plan.touched:
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
|
||||
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
|
||||
for key, rank in ranks.items():
|
||||
if plan.positions.get(key) != rank:
|
||||
differences.append(f"policy_hierarchy_rules:{key}:position")
|
||||
if differences:
|
||||
raise SelfCheckFailed(differences)
|
||||
|
||||
|
||||
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
|
||||
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
|
||||
ConfigError; on any exception the caller must roll back."""
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
await policy_lock(db)
|
||||
plan = await planner.build_plan(db, parsed, prune)
|
||||
if plan.token != token:
|
||||
raise PlanStale()
|
||||
if not plan.applicable:
|
||||
raise PlanBlocked(plan)
|
||||
writer = _Writer(db, plan)
|
||||
await writer.load()
|
||||
await writer.hierarchy_groups()
|
||||
await writer.device_groups()
|
||||
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
|
||||
if kind in plan.touched:
|
||||
await writer.objects(kind)
|
||||
await writer.hierarchy_positions()
|
||||
await writer.host_hierarchy()
|
||||
await writer.variables()
|
||||
await writer.deletions()
|
||||
await _self_check(db, plan)
|
||||
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
|
||||
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
|
||||
"summary": plan.summary()}))
|
||||
for values in writer.events:
|
||||
db.add(audit_service.make_event(**values))
|
||||
await db.flush()
|
||||
return plan
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
|
||||
|
||||
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
|
||||
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
|
||||
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
|
||||
"""
|
||||
import yaml
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.version import APP_VERSION
|
||||
|
||||
HEADER = (
|
||||
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
|
||||
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
|
||||
)
|
||||
|
||||
|
||||
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
|
||||
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
|
||||
fields = list(doc.model_fields)
|
||||
if set(values) != set(fields):
|
||||
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
|
||||
return {name: values[name] for name in fields}
|
||||
|
||||
|
||||
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
|
||||
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
|
||||
if "hierarchy_group" in doc.model_fields:
|
||||
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
|
||||
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
|
||||
values = dict(extra)
|
||||
for name in doc.model_fields:
|
||||
if name in values:
|
||||
continue
|
||||
values[name] = obj.config_key if name == "key" else getattr(obj, name)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
|
||||
"""Parents before children, siblings by name; the same rule the loader expects."""
|
||||
children: dict = {}
|
||||
for group in groups:
|
||||
children.setdefault(group.parent_id, []).append(group)
|
||||
ordered: list[PolicyHierarchyGroup] = []
|
||||
|
||||
def walk(parent_id) -> None:
|
||||
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
|
||||
ordered.append(group)
|
||||
walk(group.id)
|
||||
|
||||
walk(None)
|
||||
return ordered
|
||||
|
||||
|
||||
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
|
||||
"""{kind: content} for the requested kinds (default all), in file order."""
|
||||
wanted = set(schema.KINDS) if kinds is None else kinds
|
||||
out: dict[str, list | dict] = {}
|
||||
|
||||
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
|
||||
hierarchy_names = {g.id: g.name for g in all_groups}
|
||||
|
||||
if "device_groups" in wanted:
|
||||
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
|
||||
out["device_groups"] = [
|
||||
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
|
||||
for g in sorted(groups, key=lambda g: g.name)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_groups" in wanted:
|
||||
out["policy_hierarchy_groups"] = [
|
||||
_ordered(schema.HierarchyGroupDoc, {
|
||||
"name": g.name,
|
||||
"parent": hierarchy_names.get(g.parent_id),
|
||||
"description": g.description or "",
|
||||
"hosts": sorted(h.name for h in g.hosts),
|
||||
})
|
||||
for g in _hierarchy_order(list(all_groups))
|
||||
]
|
||||
|
||||
if "aliases" in wanted:
|
||||
aliases = (await db.execute(
|
||||
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
|
||||
)).scalars().all()
|
||||
out["aliases"] = [
|
||||
_ordered(schema.AliasDoc, {
|
||||
"name": a.name,
|
||||
"type": a.type,
|
||||
"description": a.description or "",
|
||||
"enabled": a.enabled,
|
||||
"entries": [line for line in (a.content or "").split("\n") if line],
|
||||
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
|
||||
})
|
||||
for a in sorted(aliases, key=lambda a: a.name)
|
||||
]
|
||||
|
||||
simple = (
|
||||
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
|
||||
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
|
||||
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
|
||||
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
|
||||
)
|
||||
for kind, model, doc, targets_doc in simple:
|
||||
if kind not in wanted:
|
||||
continue
|
||||
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
|
||||
out[kind] = [
|
||||
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
|
||||
for row in sorted(rows, key=lambda r: r.config_key)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_rules" in wanted:
|
||||
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
|
||||
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
|
||||
out["policy_hierarchy_rules"] = [
|
||||
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
|
||||
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
|
||||
]
|
||||
|
||||
if "variables" in wanted:
|
||||
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
|
||||
out["variables"] = {
|
||||
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
|
||||
for h in sorted(hosts, key=lambda h: h.name)
|
||||
if h.variables
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def dump_yaml(content: dict[str, list | dict]) -> str:
|
||||
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
|
||||
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
|
||||
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
|
||||
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
|
||||
|
||||
|
||||
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
|
||||
return dump_yaml(await canonical(db, kinds))
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
|
||||
|
||||
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
|
||||
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
|
||||
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
|
||||
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
|
||||
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
|
||||
|
||||
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
|
||||
"""
|
||||
import hashlib
|
||||
import re
|
||||
from typing import NamedTuple
|
||||
|
||||
import yaml
|
||||
from pydantic import ValidationError
|
||||
from yaml.composer import Composer
|
||||
from yaml.constructor import SafeConstructor
|
||||
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
|
||||
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
|
||||
from yaml.parser import Parser
|
||||
from yaml.reader import Reader
|
||||
from yaml.resolver import BaseResolver
|
||||
from yaml.scanner import Scanner
|
||||
|
||||
from app.config_code import schema
|
||||
|
||||
MAX_BYTES = 1024 * 1024
|
||||
MAX_DEPTH = 8
|
||||
MAX_NODES = 100_000
|
||||
MAX_SCALAR = schema.MAX_SCALAR
|
||||
MAX_OBJECTS_PER_KIND = 5000
|
||||
MAX_VARIABLE_HOSTS = 500
|
||||
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
|
||||
MAX_ALIAS_ENTRIES = 2000
|
||||
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
|
||||
|
||||
_STR = "tag:yaml.org,2002:str"
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
|
||||
|
||||
def __init__(self, code: str, errors: list[dict]):
|
||||
self.code = code
|
||||
self.errors = errors
|
||||
super().__init__(f"{code}: {errors[:3]}")
|
||||
|
||||
|
||||
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
|
||||
return {"code": code, "path": path, "line": line, "params": params}
|
||||
|
||||
|
||||
def display(value, limit: int = 80) -> str:
|
||||
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
|
||||
text = value if isinstance(value, str) else repr(value)
|
||||
text = text.encode("unicode_escape").decode("ascii")
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def _path(parts) -> str:
|
||||
out = ""
|
||||
for part in parts:
|
||||
if isinstance(part, int):
|
||||
out += f"[{part}]"
|
||||
else:
|
||||
out += ("." if out else "") + str(part)
|
||||
return out
|
||||
|
||||
|
||||
class _Rejected(yaml.YAMLError):
|
||||
def __init__(self, code: str, mark=None, path=(), **params):
|
||||
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
class StrictResolver(BaseResolver):
|
||||
yaml_implicit_resolvers: dict = {}
|
||||
|
||||
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
|
||||
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
|
||||
|
||||
|
||||
class StrictConstructor(SafeConstructor):
|
||||
yaml_constructors: dict = {}
|
||||
|
||||
|
||||
for _tag, _fn in (
|
||||
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
|
||||
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
|
||||
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
|
||||
(_STR, SafeConstructor.construct_yaml_str),
|
||||
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
|
||||
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
|
||||
):
|
||||
StrictConstructor.add_constructor(_tag, _fn)
|
||||
|
||||
|
||||
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
|
||||
def __init__(self, stream: str):
|
||||
Reader.__init__(self, stream)
|
||||
Scanner.__init__(self)
|
||||
Parser.__init__(self)
|
||||
Composer.__init__(self)
|
||||
StrictConstructor.__init__(self)
|
||||
StrictResolver.__init__(self)
|
||||
self._depth = 0
|
||||
self._nodes = 0
|
||||
self._path: list = []
|
||||
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
|
||||
|
||||
# -- document level ---------------------------------------------------------------------------------------------
|
||||
def get_single_node(self):
|
||||
self.get_event() # StreamStart
|
||||
if self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_empty")
|
||||
start = self.peek_event()
|
||||
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
|
||||
raise _Rejected("yaml_directive_forbidden", start.start_mark)
|
||||
node = self.compose_document()
|
||||
if not self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
|
||||
self.get_event()
|
||||
return node
|
||||
|
||||
# -- node level ---------------------------------------------------------------------------------------------------
|
||||
def compose_node(self, parent, index):
|
||||
event = self.peek_event()
|
||||
if isinstance(event, AliasEvent):
|
||||
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "anchor", None) is not None:
|
||||
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "tag", None) is not None:
|
||||
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
|
||||
self._nodes += 1
|
||||
if self._nodes > MAX_NODES:
|
||||
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
|
||||
|
||||
if isinstance(index, ScalarNode):
|
||||
segment = index.value
|
||||
elif isinstance(index, int) and not isinstance(index, bool):
|
||||
segment = index
|
||||
else:
|
||||
segment = None # a mapping key
|
||||
if segment is not None:
|
||||
self._path.append(segment)
|
||||
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
|
||||
self._depth += 1
|
||||
try:
|
||||
if self._depth > MAX_DEPTH:
|
||||
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
|
||||
return super().compose_node(parent, index)
|
||||
finally:
|
||||
self._depth -= 1
|
||||
if segment is not None:
|
||||
self._path.pop()
|
||||
|
||||
def compose_scalar_node(self, anchor):
|
||||
event = self.peek_event()
|
||||
if len(event.value) > MAX_SCALAR:
|
||||
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
|
||||
return super().compose_scalar_node(anchor)
|
||||
|
||||
def compose_mapping_node(self, anchor):
|
||||
node = super().compose_mapping_node(anchor)
|
||||
seen: set[str] = set()
|
||||
for key, _value in node.value:
|
||||
if not isinstance(key, ScalarNode) or key.tag != _STR:
|
||||
raise _Rejected("non_string_key", key.start_mark, self._path)
|
||||
if key.value == "<<":
|
||||
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
|
||||
if key.value in seen:
|
||||
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
|
||||
seen.add(key.value)
|
||||
return node
|
||||
|
||||
|
||||
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
|
||||
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
if len(raw) > MAX_BYTES:
|
||||
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
|
||||
try:
|
||||
text = raw.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
|
||||
if text.startswith(""):
|
||||
text = text[1:]
|
||||
loader = None
|
||||
try:
|
||||
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
|
||||
node = loader.get_single_node()
|
||||
data = loader.construct_document(node)
|
||||
except _Rejected as exc:
|
||||
line = exc.mark.line + 1 if exc.mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
|
||||
except yaml.MarkedYAMLError as exc:
|
||||
mark = exc.problem_mark or exc.context_mark
|
||||
line = mark.line + 1 if mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
|
||||
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
|
||||
finally:
|
||||
if loader is not None:
|
||||
loader.dispose()
|
||||
return data, loader.lines
|
||||
|
||||
|
||||
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
|
||||
for end in range(len(loc), 0, -1):
|
||||
if loc[:end] in lines:
|
||||
return lines[loc[:end]]
|
||||
return None
|
||||
|
||||
|
||||
_PYDANTIC_CODES = {
|
||||
"extra_forbidden": "unknown_field",
|
||||
"missing": "missing_field",
|
||||
"literal_error": "bad_value",
|
||||
"string_too_short": "too_short",
|
||||
"string_too_long": "too_long",
|
||||
"string_pattern_mismatch": "bad_format",
|
||||
}
|
||||
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
|
||||
|
||||
|
||||
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
|
||||
out = []
|
||||
for error in exc.errors(include_url=False):
|
||||
loc = tuple(error["loc"])
|
||||
code = _PYDANTIC_CODES.get(error["type"])
|
||||
if code is None and error["type"] == "value_error":
|
||||
message = str(error.get("ctx", {}).get("error", ""))
|
||||
code = message if message in _OWN_CODES else "bad_value"
|
||||
if code is None:
|
||||
code = "bad_type"
|
||||
params = {}
|
||||
if error["type"] == "extra_forbidden":
|
||||
params["field"] = display(loc[-1]) if loc else ""
|
||||
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
|
||||
params["value"] = display(error["input"])
|
||||
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
|
||||
return out
|
||||
|
||||
|
||||
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
|
||||
errors = []
|
||||
for kind in schema.KINDS:
|
||||
section = data.get(kind)
|
||||
if kind == "variables":
|
||||
if isinstance(section, dict):
|
||||
if len(section) > MAX_VARIABLE_HOSTS:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
|
||||
for host, values in section.items():
|
||||
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
|
||||
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
|
||||
max=MAX_VARIABLES_PER_HOST))
|
||||
continue
|
||||
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
|
||||
aliases = data.get("aliases")
|
||||
for i, alias in enumerate(aliases if isinstance(aliases, list) else []): # any other shape is for the schema check
|
||||
entries = alias.get("entries") if isinstance(alias, dict) else None
|
||||
if isinstance(entries, list):
|
||||
loc = ("aliases", i, "entries")
|
||||
if len(entries) > MAX_ALIAS_ENTRIES:
|
||||
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
|
||||
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
|
||||
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
|
||||
return errors
|
||||
|
||||
|
||||
class Parsed(NamedTuple):
|
||||
doc: schema.ConfigDoc
|
||||
sha256: str # of the raw bytes, shown in the plan and stored in the audit
|
||||
lines: dict[tuple, int] # path -> line, for messages of the planner
|
||||
|
||||
|
||||
def parse(raw: bytes | str) -> Parsed:
|
||||
"""Load, limit and validate a file. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
sha256 = hashlib.sha256(raw).hexdigest()
|
||||
data, lines = load_yaml(raw)
|
||||
if not isinstance(data, dict):
|
||||
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
|
||||
version = data.get("opnexus_config")
|
||||
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
|
||||
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
|
||||
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
|
||||
"opnexus_config", _line_for(lines, ("opnexus_config",)),
|
||||
**({"value": display(version)} if version is not None else {}))])
|
||||
errors = _check_counts(data, lines)
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors)
|
||||
try:
|
||||
doc = schema.ConfigDoc.model_validate(data)
|
||||
except ValidationError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
|
||||
return Parsed(doc, sha256, lines)
|
||||
@@ -0,0 +1,933 @@
|
||||
{
|
||||
"$defs": {
|
||||
"AliasDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"pattern": "^[A-Za-z0-9_]+$",
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "host",
|
||||
"enum": [
|
||||
"host",
|
||||
"network",
|
||||
"port",
|
||||
"networkgroup",
|
||||
"mac",
|
||||
"asn"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 255,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"entries": {
|
||||
"items": {
|
||||
"maxLength": 8000,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Entries",
|
||||
"type": "array"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "AliasDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"DeviceGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "DeviceGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"parent": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Parent"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "HierarchyGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyRuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"group": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Group",
|
||||
"type": "string"
|
||||
},
|
||||
"rulebase": {
|
||||
"default": "pre",
|
||||
"enum": [
|
||||
"pre",
|
||||
"post"
|
||||
],
|
||||
"title": "Rulebase",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"group",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "HierarchyRuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OneToOneDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "binat",
|
||||
"enum": [
|
||||
"binat",
|
||||
"nat"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"external": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "External",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"enable",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"external"
|
||||
],
|
||||
"title": "OneToOneDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OutboundNatDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"target": {
|
||||
"default": "",
|
||||
"maxLength": 128,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"target_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Target Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "OutboundNatDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"PortForwardDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "tcp",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"target": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"local_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Local Port"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"purenat",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"target"
|
||||
],
|
||||
"title": "PortForwardDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"RuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/Targets"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "RuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"Targets": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"title": "Targets",
|
||||
"type": "object"
|
||||
},
|
||||
"TargetsWithHierarchy": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
},
|
||||
"hierarchy_group": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Hierarchy Group"
|
||||
}
|
||||
},
|
||||
"title": "TargetsWithHierarchy",
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
|
||||
"properties": {
|
||||
"opnexus_config": {
|
||||
"const": 1,
|
||||
"title": "Opnexus Config",
|
||||
"type": "integer"
|
||||
},
|
||||
"device_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/DeviceGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Device Groups"
|
||||
},
|
||||
"policy_hierarchy_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Groups"
|
||||
},
|
||||
"aliases": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/AliasDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Aliases"
|
||||
},
|
||||
"rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/RuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Rules"
|
||||
},
|
||||
"port_forwards": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/PortForwardDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Port Forwards"
|
||||
},
|
||||
"one_to_one": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OneToOneDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "One To One"
|
||||
},
|
||||
"outbound_nat": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OutboundNatDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Outbound Nat"
|
||||
},
|
||||
"policy_hierarchy_rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyRuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Rules"
|
||||
},
|
||||
"variables": {
|
||||
"anyOf": [
|
||||
{
|
||||
"additionalProperties": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"propertyNames": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Variables"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"opnexus_config"
|
||||
],
|
||||
"title": "OPNexus-Konfiguration (Schema v1)",
|
||||
"type": "object",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
}
|
||||
@@ -0,0 +1,570 @@
|
||||
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
|
||||
|
||||
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
|
||||
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
|
||||
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
|
||||
`prune` gleicht die Art ab.
|
||||
|
||||
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
|
||||
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
|
||||
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
|
||||
|
||||
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
|
||||
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
|
||||
from app.database import Base
|
||||
from app.services.deploy_flow import is_unsettled
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.schemas.alias import AliasCreate
|
||||
from app.schemas.one_to_one import OneToOneCreate
|
||||
from app.schemas.outbound_nat import OutboundNatCreate
|
||||
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||
from app.schemas.port_forward import PortForwardCreate
|
||||
from app.schemas.rule import RuleCreate
|
||||
from app.services.variables_service import names_in
|
||||
|
||||
DEPLOYED = ("success", "pending", "revert_failed")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
section: str
|
||||
model: type | None
|
||||
ident: str # identity field in the file ("key"/"name")
|
||||
api: type | None = None # API create schema, run on every file object (same validators as the UI)
|
||||
deployable: bool = False
|
||||
|
||||
|
||||
KIND_SPECS = {
|
||||
"device_groups": Kind("device_groups", DeviceGroup, "name"),
|
||||
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
|
||||
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
|
||||
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
|
||||
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
|
||||
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
|
||||
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
|
||||
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
|
||||
"variables": Kind("variables", None, "key"),
|
||||
}
|
||||
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
|
||||
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
|
||||
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
|
||||
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
|
||||
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
|
||||
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
|
||||
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
|
||||
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
kind: str
|
||||
key: str
|
||||
action: str # create | update | delete | unchanged
|
||||
before: dict | None = None
|
||||
after: dict | None = None
|
||||
changes: list = field(default_factory=list)
|
||||
reasons: list = field(default_factory=list)
|
||||
deployed_hosts: int = 0
|
||||
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
|
||||
db_id: uuid.UUID | None = None
|
||||
|
||||
def public(self) -> dict:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"key": self.key,
|
||||
"action": "blocked" if self.reasons else self.action,
|
||||
"planned_action": self.action,
|
||||
"changes": self.changes,
|
||||
"deployed_hosts": self.deployed_hosts,
|
||||
"reasons": self.reasons,
|
||||
"implicit": self.implicit,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Plan:
|
||||
sha256: str
|
||||
prune: bool
|
||||
touched: list[str]
|
||||
items: list[Item]
|
||||
warnings: list[dict]
|
||||
token: str = ""
|
||||
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
|
||||
expected: dict = field(default_factory=dict)
|
||||
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
|
||||
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
|
||||
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
|
||||
|
||||
@property
|
||||
def applicable(self) -> bool:
|
||||
return not any(i.reasons for i in self.items)
|
||||
|
||||
def summary(self) -> dict:
|
||||
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||
for item in self.items:
|
||||
out["blocked" if item.reasons else item.action] += 1
|
||||
return out
|
||||
|
||||
def public(self) -> dict:
|
||||
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
|
||||
return {
|
||||
"token": self.token,
|
||||
"sha256": self.sha256,
|
||||
"prune": self.prune,
|
||||
"kinds": self.touched,
|
||||
"summary": self.summary(),
|
||||
"items": [i.public() for i in changed],
|
||||
"warnings": self.warnings,
|
||||
"applicable": self.applicable,
|
||||
}
|
||||
|
||||
|
||||
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
|
||||
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
|
||||
|
||||
|
||||
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
|
||||
section = getattr(doc, kind)
|
||||
if kind == "variables":
|
||||
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
|
||||
for host, values in section.items() for name, value in values.items()]
|
||||
return [item.model_dump(mode="python") for item in section]
|
||||
|
||||
|
||||
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
|
||||
def sorted_unique(values: list, where: tuple) -> list:
|
||||
seen = set()
|
||||
for i, value in enumerate(values):
|
||||
if value in seen:
|
||||
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
|
||||
seen.add(value)
|
||||
return sorted(seen)
|
||||
|
||||
item = dict(item)
|
||||
if "targets" in item:
|
||||
targets = dict(item["targets"])
|
||||
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
|
||||
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
|
||||
item["targets"] = targets
|
||||
if kind in ("device_groups", "policy_hierarchy_groups"):
|
||||
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
|
||||
return item
|
||||
|
||||
|
||||
def _changes(before: dict, after: dict) -> list[dict]:
|
||||
out = []
|
||||
for name in after:
|
||||
if name == "targets":
|
||||
for sub in after["targets"]:
|
||||
if before["targets"].get(sub) != after["targets"][sub]:
|
||||
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
|
||||
elif before.get(name) != after[name]:
|
||||
out.append({"field": name, "before": before.get(name), "after": after[name]})
|
||||
return out
|
||||
|
||||
|
||||
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
|
||||
buckets: dict = {}
|
||||
for row in rows:
|
||||
buckets.setdefault(bucket_of(row), []).append(row)
|
||||
out = {}
|
||||
for members in buckets.values():
|
||||
for rank, row in enumerate(sorted(members, key=order_of)):
|
||||
out[key_of(row)] = rank
|
||||
return out
|
||||
|
||||
|
||||
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
|
||||
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
|
||||
found: dict = {}
|
||||
if not ids:
|
||||
return found
|
||||
for table in Base.metadata.sorted_tables:
|
||||
if table.name in skip:
|
||||
continue
|
||||
for fk in table.foreign_keys:
|
||||
if fk.column.table.name != target_table:
|
||||
continue
|
||||
column = fk.parent
|
||||
rows = (await db.execute(
|
||||
select(column, func.count()).where(column.in_(ids)).group_by(column)
|
||||
)).all()
|
||||
for ref_id, _count in rows:
|
||||
found.setdefault(ref_id, []).append(table.name)
|
||||
return found
|
||||
|
||||
|
||||
async def _load(db: AsyncSession) -> dict:
|
||||
loaded: dict = {}
|
||||
for kind in TARGET_KINDS:
|
||||
model = KIND_SPECS[kind].model
|
||||
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
|
||||
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
|
||||
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
|
||||
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
|
||||
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
return loaded
|
||||
|
||||
|
||||
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
|
||||
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
|
||||
doc, lines = parsed.doc, parsed.lines
|
||||
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
|
||||
errors: list[dict] = []
|
||||
warnings: list[dict] = []
|
||||
|
||||
db_canon = await canonical(db)
|
||||
loaded = await _load(db)
|
||||
hosts = loaded["hosts"]
|
||||
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
|
||||
|
||||
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
|
||||
file_objects: dict[str, dict[str, dict]] = {}
|
||||
file_locs: dict[tuple, tuple] = {}
|
||||
for kind in touched:
|
||||
spec = KIND_SPECS[kind]
|
||||
objects: dict[str, dict] = {}
|
||||
for index, raw in enumerate(_file_items(doc, kind)):
|
||||
if kind == "variables":
|
||||
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
|
||||
else:
|
||||
ident, loc = raw[spec.ident], (kind, index)
|
||||
if ident in objects:
|
||||
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
|
||||
continue
|
||||
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
|
||||
file_locs[(kind, ident)] = loc
|
||||
file_objects[kind] = objects
|
||||
|
||||
def db_items(kind: str) -> dict[str, dict]:
|
||||
if kind == "variables":
|
||||
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in db_canon["variables"].items() for n, v in values.items()}
|
||||
ident = KIND_SPECS[kind].ident
|
||||
return {item[ident]: item for item in db_canon[kind]}
|
||||
|
||||
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
|
||||
|
||||
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
|
||||
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
|
||||
post: dict[str, dict[str, dict]] = {}
|
||||
for kind in schema.KINDS:
|
||||
if kind in file_objects:
|
||||
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
|
||||
if prune:
|
||||
deleted[kind] = set(remaining)
|
||||
remaining = {}
|
||||
post[kind] = {**remaining, **file_objects[kind]}
|
||||
else:
|
||||
post[kind] = dict(db_objects[kind])
|
||||
|
||||
host_names = set(hosts)
|
||||
group_names = set(post["device_groups"])
|
||||
hierarchy_names = set(post["policy_hierarchy_groups"])
|
||||
|
||||
# -- references in the file ------------------------------------------------------------------------------------------
|
||||
for kind in touched:
|
||||
for ident, item in file_objects[kind].items():
|
||||
loc = file_locs[(kind, ident)]
|
||||
if kind == "variables":
|
||||
if item["host"] not in host_names:
|
||||
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
|
||||
continue
|
||||
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
|
||||
for name in member_hosts:
|
||||
if name not in host_names:
|
||||
_err(errors, "unknown_host", loc, lines, host=display(name))
|
||||
for name in item.get("targets", {}).get("groups", []):
|
||||
if name not in group_names:
|
||||
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
|
||||
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
|
||||
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
|
||||
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
|
||||
if ref is not None and ref not in hierarchy_names:
|
||||
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
|
||||
if kind == "policy_hierarchy_rules":
|
||||
for name, value in item.items():
|
||||
if names_in(value):
|
||||
_err(errors, "placeholder_not_supported", loc + (name,), lines)
|
||||
spec = KIND_SPECS[kind]
|
||||
if spec.api is not None:
|
||||
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
|
||||
for name in member_hosts:
|
||||
host = hosts.get(name)
|
||||
if host is not None and not host.enabled:
|
||||
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
|
||||
|
||||
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
|
||||
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
|
||||
if "policy_hierarchy_groups" in file_objects:
|
||||
file_groups = file_objects["policy_hierarchy_groups"]
|
||||
for host_name, group in list(host_hierarchy.items()):
|
||||
if prune or group in file_groups:
|
||||
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
|
||||
host_hierarchy[host_name] = None
|
||||
listed: dict[str, str] = {}
|
||||
for name, group in file_groups.items():
|
||||
for host_name in group["hosts"]:
|
||||
if host_name in listed:
|
||||
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
|
||||
lines, host=display(host_name), first=display(listed[host_name]))
|
||||
listed[host_name] = name
|
||||
if host_name in host_hierarchy:
|
||||
host_hierarchy[host_name] = name
|
||||
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
|
||||
for start in parents:
|
||||
seen, current = set(), start
|
||||
while current is not None and current not in seen:
|
||||
seen.add(current)
|
||||
current = parents.get(current)
|
||||
if current is not None and current == start:
|
||||
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
|
||||
lines, group=display(start))
|
||||
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
|
||||
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
|
||||
|
||||
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
|
||||
per_host: dict[str, int] = {}
|
||||
for item in post["variables"].values():
|
||||
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
|
||||
for host_name, count in sorted(per_host.items()):
|
||||
if count > MAX_VARIABLES_PER_HOST:
|
||||
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
|
||||
max=MAX_VARIABLES_PER_HOST, count=count)
|
||||
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors[:100])
|
||||
|
||||
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
|
||||
db_rule_rows = loaded["policy_hierarchy_rules"]
|
||||
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
|
||||
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
|
||||
positions = dict(db_positions)
|
||||
if "policy_hierarchy_rules" in file_objects:
|
||||
buckets: dict[tuple, list[str]] = {}
|
||||
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
|
||||
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for key, item in post["policy_hierarchy_rules"].items():
|
||||
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
|
||||
buckets.setdefault((item["group"], item["rulebase"]), [])
|
||||
positions = {}
|
||||
extras: dict[tuple, list[str]] = {}
|
||||
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
|
||||
key=lambda k: db_positions.get(k, 0)):
|
||||
item = post["policy_hierarchy_rules"][key]
|
||||
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for bucket in set(buckets) | set(extras):
|
||||
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
|
||||
positions[key] = rank
|
||||
|
||||
# -- items -----------------------------------------------------------------------------------------------------------
|
||||
items: list[Item] = []
|
||||
for kind in touched:
|
||||
rows = loaded.get(kind, {})
|
||||
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
|
||||
row = rows.get(ident) if kind != "variables" else None
|
||||
before = db_objects[kind].get(ident)
|
||||
after = post[kind].get(ident) if ident not in deleted[kind] else None
|
||||
if before is None:
|
||||
item = Item(kind, ident, "create", None, after)
|
||||
elif after is None:
|
||||
item = Item(kind, ident, "delete", before, None)
|
||||
else:
|
||||
changes = _changes(before, after)
|
||||
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
|
||||
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
|
||||
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
|
||||
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
|
||||
if row is not None:
|
||||
item.db_id = row.id
|
||||
deployments = getattr(row, "deployments", None) or []
|
||||
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
|
||||
if item.action in ("update", "delete") and any(is_unsettled(d) for d in deployments):
|
||||
item.reasons.append("busy_deployment")
|
||||
if item.action == "delete" and deployments:
|
||||
item.reasons.append("prune_deployed")
|
||||
items.append(item)
|
||||
|
||||
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
|
||||
|
||||
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
|
||||
if redeploy:
|
||||
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
|
||||
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
|
||||
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
|
||||
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
|
||||
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
|
||||
warnings += _alias_reference_warnings(post, file_objects, file_locs)
|
||||
warnings += _variable_warnings(post, deleted)
|
||||
|
||||
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
|
||||
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
|
||||
file_objects=file_objects)
|
||||
plan.expected = {kind: post[kind] for kind in touched}
|
||||
plan.token = _token(plan, db_canon, db_positions, loaded)
|
||||
return plan
|
||||
|
||||
|
||||
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
|
||||
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
|
||||
out = set()
|
||||
if kind == "policy_hierarchy_rules":
|
||||
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
|
||||
if kind == "policy_hierarchy_groups":
|
||||
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
|
||||
return out
|
||||
|
||||
|
||||
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
|
||||
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
|
||||
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
|
||||
if kind == "aliases":
|
||||
values["content"] = "\n".join(item["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
values["hierarchy_group_id"] = uuid.UUID(int=0)
|
||||
try:
|
||||
cleaned = KIND_SPECS[kind].api.model_validate(values)
|
||||
except ValidationError as exc:
|
||||
for error in exc.errors(include_url=False):
|
||||
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
|
||||
return item
|
||||
out = dict(item)
|
||||
for name in item:
|
||||
if name in values and name != "content" and hasattr(cleaned, name):
|
||||
out[name] = getattr(cleaned, name)
|
||||
if kind == "aliases":
|
||||
out["entries"] = [e for e in cleaned.content.split("\n") if e]
|
||||
return out
|
||||
|
||||
|
||||
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
|
||||
by_key = {(i.kind, i.key): i for i in items}
|
||||
|
||||
# device groups: post-state objects of the import, plus every other table that points at the group
|
||||
if deleted["device_groups"]:
|
||||
users: dict[str, list[str]] = {}
|
||||
for kind in TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
for group in item["targets"]["groups"]:
|
||||
users.setdefault(group, []).append(f"{kind}:{ident}")
|
||||
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
|
||||
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
|
||||
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["device_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("device_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["policy_hierarchy_groups"]:
|
||||
users = {}
|
||||
for kind in HIERARCHY_TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
if item["targets"]["hierarchy_group"]:
|
||||
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
|
||||
for ident, item in post["policy_hierarchy_rules"].items():
|
||||
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
|
||||
for name, item in post["policy_hierarchy_groups"].items():
|
||||
if item["parent"]:
|
||||
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
|
||||
for host_name, group in host_hierarchy.items():
|
||||
if group:
|
||||
users.setdefault(group, []).append(f"host:{host_name}")
|
||||
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
|
||||
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
|
||||
"policy_hierarchy_rules", "opnsense_hosts"}
|
||||
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["policy_hierarchy_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["aliases"]:
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS:
|
||||
for item in post[kind].values():
|
||||
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
|
||||
for item in post["aliases"].values():
|
||||
used.update(item["entries"])
|
||||
for name in deleted["aliases"]:
|
||||
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
|
||||
by_key[("aliases", name)].reasons.append("prune_referenced")
|
||||
|
||||
|
||||
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
|
||||
aliases = set(post["aliases"])
|
||||
out = []
|
||||
for kind in RULE_KINDS:
|
||||
for ident, item in file_objects.get(kind, {}).items():
|
||||
for name in ALIAS_REF_FIELDS:
|
||||
value = item.get(name)
|
||||
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
|
||||
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
|
||||
"params": {"value": display(value)}})
|
||||
return out
|
||||
|
||||
|
||||
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
|
||||
if not deleted["variables"]:
|
||||
return []
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS + ("aliases",):
|
||||
for item in post[kind].values():
|
||||
for value in item.values():
|
||||
if isinstance(value, str):
|
||||
used |= names_in(value)
|
||||
elif isinstance(value, list):
|
||||
for entry in value:
|
||||
used |= names_in(entry)
|
||||
out = []
|
||||
for key in sorted(deleted["variables"]):
|
||||
host, name = key.split("/", 1)
|
||||
if name in used:
|
||||
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
|
||||
return out
|
||||
|
||||
|
||||
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
|
||||
deployments = {
|
||||
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
|
||||
for ident, row in loaded[kind].items()}
|
||||
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
|
||||
}
|
||||
material = {
|
||||
"file": plan.file_objects,
|
||||
"prune": plan.prune,
|
||||
"db": db_canon,
|
||||
"positions": db_positions,
|
||||
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
|
||||
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
|
||||
"deployments": deployments,
|
||||
"items": [i.public() for i in plan.items],
|
||||
"warnings": plan.warnings,
|
||||
}
|
||||
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
|
||||
@@ -0,0 +1,242 @@
|
||||
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
|
||||
|
||||
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
|
||||
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
|
||||
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
|
||||
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
|
||||
"""
|
||||
import re
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
|
||||
|
||||
from app.models.config_key import CONFIG_KEY_RE
|
||||
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
MAX_SCALAR = 8000
|
||||
|
||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
|
||||
|
||||
|
||||
def _clean(value: str) -> str:
|
||||
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
if value != value.strip():
|
||||
raise ValueError("surrounding_whitespace")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_value(value: str) -> str:
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_text(value: str) -> str:
|
||||
if _CONTROL_TEXT.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _port(value):
|
||||
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
|
||||
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
|
||||
|
||||
|
||||
Text = Annotated[str, AfterValidator(_clean)]
|
||||
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
|
||||
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
|
||||
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
|
||||
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
|
||||
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
|
||||
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
|
||||
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
|
||||
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
|
||||
Port = Annotated[PortText | None, BeforeValidator(_port)]
|
||||
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
|
||||
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
|
||||
|
||||
IpProtocol = Literal["inet", "inet6"]
|
||||
|
||||
|
||||
class _Doc(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
class Targets(_Doc):
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
groups: list[GroupName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class TargetsWithHierarchy(Targets):
|
||||
hierarchy_group: GroupName | None = None
|
||||
|
||||
|
||||
class DeviceGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class HierarchyGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
parent: GroupName | None = None
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AliasDoc(_Doc):
|
||||
name: AliasName
|
||||
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
|
||||
description: Description = ""
|
||||
enabled: bool = True
|
||||
entries: list[AliasEntry] = Field(default_factory=list)
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class RuleDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
log: bool = False
|
||||
targets: Targets = Field(default_factory=Targets)
|
||||
|
||||
|
||||
class PortForwardDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "tcp"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
target: Net
|
||||
local_port: Port = None
|
||||
natreflection: Literal["", "purenat", "disable"] = ""
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OneToOneDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
type: Literal["binat", "nat"] = "binat"
|
||||
external: Net
|
||||
source_net: Net = "any"
|
||||
destination_net: Net = "any"
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
natreflection: Literal["", "enable", "disable"] = ""
|
||||
log: bool = False
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OutboundNatDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
|
||||
target_port: Port = None
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class HierarchyRuleDoc(_Doc):
|
||||
key: Key
|
||||
group: GroupName
|
||||
rulebase: Literal["pre", "post"] = "pre"
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
def _variable_name(value: str) -> str:
|
||||
if not VARIABLE_NAME_RE.match(value):
|
||||
raise ValueError("bad_variable_name")
|
||||
return value
|
||||
|
||||
|
||||
def _variable_value(value: str) -> str:
|
||||
if not VARIABLE_VALUE_RE.match(value):
|
||||
raise ValueError("bad_variable_value")
|
||||
return value
|
||||
|
||||
|
||||
VariableName = Annotated[str, AfterValidator(_variable_name)]
|
||||
VariableValue = Annotated[str, AfterValidator(_variable_value)]
|
||||
|
||||
|
||||
class ConfigDoc(_Doc):
|
||||
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
|
||||
|
||||
opnexus_config: Literal[1]
|
||||
device_groups: list[DeviceGroupDoc] | None = None
|
||||
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
|
||||
aliases: list[AliasDoc] | None = None
|
||||
rules: list[RuleDoc] | None = None
|
||||
port_forwards: list[PortForwardDoc] | None = None
|
||||
one_to_one: list[OneToOneDoc] | None = None
|
||||
outbound_nat: list[OutboundNatDoc] | None = None
|
||||
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
|
||||
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
|
||||
|
||||
|
||||
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
|
||||
KINDS = (
|
||||
"device_groups",
|
||||
"policy_hierarchy_groups",
|
||||
"aliases",
|
||||
"rules",
|
||||
"port_forwards",
|
||||
"one_to_one",
|
||||
"outbound_nat",
|
||||
"policy_hierarchy_rules",
|
||||
"variables",
|
||||
)
|
||||
|
||||
|
||||
SCHEMA_FILE = "opnexus-config-v1.schema.json"
|
||||
|
||||
|
||||
def json_schema_text() -> str:
|
||||
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
|
||||
import json
|
||||
|
||||
schema = ConfigDoc.model_json_schema()
|
||||
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
|
||||
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
|
||||
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
|
||||
@@ -62,6 +62,7 @@ class ErrorCode(str, Enum):
|
||||
GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET"
|
||||
GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET"
|
||||
GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET"
|
||||
GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET"
|
||||
HOSTNAMEN_EXAKT_EINGEBEN = "HOSTNAMEN_EXAKT_EINGEBEN"
|
||||
HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN = "HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN"
|
||||
HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0 = "HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0"
|
||||
@@ -72,6 +73,8 @@ class ErrorCode(str, Enum):
|
||||
HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN = "HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN"
|
||||
HOST_IST_DEAKTIVIERT = "HOST_IST_DEAKTIVIERT"
|
||||
HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS = "HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS"
|
||||
HOST_URL_EXISTS = "HOST_URL_EXISTS"
|
||||
HOST_CONNECTION_FAILED = "HOST_CONNECTION_FAILED"
|
||||
HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B = "HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B"
|
||||
HOST_NICHT_GEFUNDEN = "HOST_NICHT_GEFUNDEN"
|
||||
HOST_REQUEST_FAILED = "HOST_REQUEST_FAILED"
|
||||
@@ -87,6 +90,15 @@ class ErrorCode(str, Enum):
|
||||
MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN = "MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN"
|
||||
NICHT_AUTHENTIFIZIERT = "NICHT_AUTHENTIFIZIERT"
|
||||
CROSS_ORIGIN_BLOCKED = "CROSS_ORIGIN_BLOCKED"
|
||||
CROSS_SITE_REQUEST_BLOCKED = "CROSS_SITE_REQUEST_BLOCKED"
|
||||
GUI_SESSION_REQUIRED = "GUI_SESSION_REQUIRED"
|
||||
GUI_GATEWAY_NOT_CONFIGURED = "GUI_GATEWAY_NOT_CONFIGURED"
|
||||
GUI_REQUIRES_HTTPS = "GUI_REQUIRES_HTTPS"
|
||||
GUI_ACCESS_DISABLED = "GUI_ACCESS_DISABLED"
|
||||
GUI_HOST_NOT_ELIGIBLE = "GUI_HOST_NOT_ELIGIBLE"
|
||||
GUI_NO_FREE_SLOT = "GUI_NO_FREE_SLOT"
|
||||
GUI_SLOT_COOLING_DOWN = "GUI_SLOT_COOLING_DOWN"
|
||||
GUI_TOO_MANY_SESSIONS = "GUI_TOO_MANY_SESSIONS"
|
||||
TLS_PROBE_FAILED = "TLS_PROBE_FAILED"
|
||||
TLS_PEM_INVALID = "TLS_PEM_INVALID"
|
||||
TLS_PIN_MISMATCH = "TLS_PIN_MISMATCH"
|
||||
@@ -116,6 +128,13 @@ class ErrorCode(str, Enum):
|
||||
NETWORK_FIELD_INVALID = "NETWORK_FIELD_INVALID"
|
||||
NETWORK_ITEM_NOT_FOUND = "NETWORK_ITEM_NOT_FOUND"
|
||||
RULE_NOT_ADOPTABLE = "RULE_NOT_ADOPTABLE"
|
||||
OBJECT_NOT_ADOPTABLE = "OBJECT_NOT_ADOPTABLE"
|
||||
ADOPT_OBJECT_NOT_FOUND = "ADOPT_OBJECT_NOT_FOUND"
|
||||
OUTBOUND_MODE_BLOCKS_ADOPT = "OUTBOUND_MODE_BLOCKS_ADOPT"
|
||||
ADOPT_LINK_TARGET_NOT_FOUND = "ADOPT_LINK_TARGET_NOT_FOUND"
|
||||
ADOPT_LINK_TARGET_CHANGED = "ADOPT_LINK_TARGET_CHANGED"
|
||||
ADOPT_LINK_HOST_HAS_OBJECT = "ADOPT_LINK_HOST_HAS_OBJECT"
|
||||
ADOPT_LINK_BUSY = "ADOPT_LINK_BUSY"
|
||||
REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0 = "REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0"
|
||||
RESTORE_VORGANG_NICHT_GEFUNDEN = "RESTORE_VORGANG_NICHT_GEFUNDEN"
|
||||
ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL = "ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL"
|
||||
@@ -131,6 +150,10 @@ class ErrorCode(str, Enum):
|
||||
UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN = "UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN"
|
||||
UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT = "UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT"
|
||||
UPSTREAM_REQUEST_FAILED = "UPSTREAM_REQUEST_FAILED"
|
||||
API_RIGHTS_MISSING = "API_RIGHTS_MISSING"
|
||||
GATEWAY_NAME_INVALID = "GATEWAY_NAME_INVALID"
|
||||
GATEWAY_UNKNOWN = "GATEWAY_UNKNOWN"
|
||||
GATEWAY_MUTE_LIMIT = "GATEWAY_MUTE_LIMIT"
|
||||
VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN = "VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN"
|
||||
VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN = "VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN"
|
||||
VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN = "VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN"
|
||||
@@ -151,6 +174,7 @@ class ErrorCode(str, Enum):
|
||||
OPEN_DEPLOYMENT_DRAFT_UNCHANGED = "OPEN_DEPLOYMENT_DRAFT_UNCHANGED"
|
||||
OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY = "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
||||
OPEN_RESTORE_BLOCKS_DEPLOY = "OPEN_RESTORE_BLOCKS_DEPLOY"
|
||||
OPEN_CLEANUP_BLOCKS_DEPLOY = "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
UPSTREAM_REJECTED = "UPSTREAM_REJECTED"
|
||||
INVALID_CHARACTERS = "INVALID_CHARACTERS"
|
||||
SERVICE_NOT_FOUND = "SERVICE_NOT_FOUND"
|
||||
@@ -162,6 +186,11 @@ class ErrorCode(str, Enum):
|
||||
VARIABLES_MISSING = "VARIABLES_MISSING"
|
||||
VARIABLE_INVALID = "VARIABLE_INVALID"
|
||||
OVERLAY_INVALID = "OVERLAY_INVALID"
|
||||
NOTIFICATION_CHANNEL_INVALID = "NOTIFICATION_CHANNEL_INVALID"
|
||||
NOTIFICATION_CHANNEL_NOT_FOUND = "NOTIFICATION_CHANNEL_NOT_FOUND"
|
||||
NOTIFICATION_CHANNEL_NAME_TAKEN = "NOTIFICATION_CHANNEL_NAME_TAKEN"
|
||||
NOTIFICATION_CHANNEL_TEST_FAILED = "NOTIFICATION_CHANNEL_TEST_FAILED"
|
||||
NOTIFICATION_WEBHOOK_URL_INVALID = "NOTIFICATION_WEBHOOK_URL_INVALID"
|
||||
OVERLAY_SUBNET_LOCKED = "OVERLAY_SUBNET_LOCKED"
|
||||
OVERLAY_NAME_TAKEN = "OVERLAY_NAME_TAKEN"
|
||||
OVERLAY_PEER_NOT_FOUND = "OVERLAY_PEER_NOT_FOUND"
|
||||
@@ -171,6 +200,66 @@ class ErrorCode(str, Enum):
|
||||
API_TOKEN_NOT_FOUND = "API_TOKEN_NOT_FOUND"
|
||||
API_TOKEN_SCOPE_FORBIDDEN = "API_TOKEN_SCOPE_FORBIDDEN"
|
||||
API_TOKEN_SCOPE_TOO_HIGH = "API_TOKEN_SCOPE_TOO_HIGH"
|
||||
WINDOW_NOT_FOUND = "WINDOW_NOT_FOUND"
|
||||
WINDOW_INVALID = "WINDOW_INVALID"
|
||||
WINDOW_NAME_TAKEN = "WINDOW_NAME_TAKEN"
|
||||
WINDOW_IN_USE = "WINDOW_IN_USE"
|
||||
WINDOW_EXCEPTION_INVALID = "WINDOW_EXCEPTION_INVALID"
|
||||
WINDOW_EXCEPTION_EXISTS = "WINDOW_EXCEPTION_EXISTS"
|
||||
WINDOW_EXCEPTION_NOT_FOUND = "WINDOW_EXCEPTION_NOT_FOUND"
|
||||
ROLLOUT_PLAN_NOT_FOUND = "ROLLOUT_PLAN_NOT_FOUND"
|
||||
ROLLOUT_INVALID = "ROLLOUT_INVALID"
|
||||
ROLLOUT_NAME_TAKEN = "ROLLOUT_NAME_TAKEN"
|
||||
ROLLOUT_PLAN_INVALID = "ROLLOUT_PLAN_INVALID"
|
||||
ROLLOUT_PREVIEW_STALE = "ROLLOUT_PREVIEW_STALE"
|
||||
ROLLOUT_PLAN_BUSY = "ROLLOUT_PLAN_BUSY"
|
||||
ROLLOUT_GROUP_IN_USE = "ROLLOUT_GROUP_IN_USE"
|
||||
ROLLOUT_RUN_NOT_FOUND = "ROLLOUT_RUN_NOT_FOUND"
|
||||
ROLLOUT_ALREADY_ACTIVE = "ROLLOUT_ALREADY_ACTIVE"
|
||||
ROLLOUT_CONFIRM_NAME = "ROLLOUT_CONFIRM_NAME"
|
||||
ROLLOUT_ACTION_NOT_ALLOWED = "ROLLOUT_ACTION_NOT_ALLOWED"
|
||||
ROLLOUT_HOST_NOT_IN_RUN = "ROLLOUT_HOST_NOT_IN_RUN"
|
||||
ROLLOUT_HOST_RUNNING = "ROLLOUT_HOST_RUNNING"
|
||||
ROLLOUT_HOST_FAILED = "ROLLOUT_HOST_FAILED"
|
||||
ANALYSE_ADRESSE_UNGUELTIG = "ANALYSE_ADRESSE_UNGUELTIG"
|
||||
ANALYSE_FAMILIEN_MISCHUNG = "ANALYSE_FAMILIEN_MISCHUNG"
|
||||
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
|
||||
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
|
||||
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
|
||||
# Rule cleanup (1.31, services/rule_cleanup_service.py)
|
||||
BEREINIGUNG_NICHT_GEFUNDEN = "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||
BEREINIGUNG_AUSWAHL_UNGUELTIG = "BEREINIGUNG_AUSWAHL_UNGUELTIG"
|
||||
BEREINIGUNG_VORSCHAU_VERALTET = "BEREINIGUNG_VORSCHAU_VERALTET"
|
||||
BEREINIGUNG_REGEL_NICHT_AENDERBAR = "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
|
||||
BEREINIGUNG_OFFENE_OPERATION = "BEREINIGUNG_OFFENE_OPERATION"
|
||||
BEREINIGUNG_ZU_VIELE_REGELN = "BEREINIGUNG_ZU_VIELE_REGELN"
|
||||
BEREINIGUNG_BESTAETIGUNG_FEHLT = "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN = "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||
BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG = "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG"
|
||||
BEREINIGUNG_FRIST_ABGELAUFEN = "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
BEREINIGUNG_NICHT_BESTAETIGBAR = "BEREINIGUNG_NICHT_BESTAETIGBAR"
|
||||
BEREINIGUNG_LIVESTAND_ABWEICHEND = "BEREINIGUNG_LIVESTAND_ABWEICHEND"
|
||||
BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR = "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
|
||||
BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT = "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT"
|
||||
BEREINIGUNG_SNAPSHOT_FEHLT = "BEREINIGUNG_SNAPSHOT_FEHLT"
|
||||
BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR = "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
|
||||
BEREINIGUNG_AKTION_NICHT_MOEGLICH = "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
BEREINIGUNG_NICHT_AUSGEFUEHRT = "BEREINIGUNG_NICHT_AUSGEFUEHRT"
|
||||
BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN = "BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN"
|
||||
BEREINIGUNG_AUSGANG_UNKLAR = "BEREINIGUNG_AUSGANG_UNKLAR"
|
||||
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
|
||||
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
|
||||
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
|
||||
CONFIG_INVALID = "CONFIG_INVALID"
|
||||
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
|
||||
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
|
||||
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
|
||||
CONFIG_BLOCKED = "CONFIG_BLOCKED"
|
||||
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
|
||||
CONFIG_BUSY = "CONFIG_BUSY"
|
||||
ZIEL_HOST_NACH_AUSROLLEN_FEST = "ZIEL_HOST_NACH_AUSROLLEN_FEST"
|
||||
ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET = "ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET"
|
||||
VPN_DEPLOY_LAEUFT_BEREITS = "VPN_DEPLOY_LAEUFT_BEREITS"
|
||||
|
||||
|
||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||
|
||||
@@ -0,0 +1,609 @@
|
||||
"""The ``gui-gateway`` container: firewall web interfaces in their own browser origin (1.27).
|
||||
|
||||
Runs from the backend image (``python -m app.gui_gateway``) behind the proxy, which terminates TLS for the
|
||||
slot ports and sets ``X-Onx-Slot`` (overwriting any client value). It never touches the database or the
|
||||
secret stores; it only talks to the backend's internal API (``/internal/gui/*``, shared GUI_GATEWAY_TOKEN)
|
||||
and, per gateway session, to exactly one firewall.
|
||||
|
||||
Security model (see the 1.27 design):
|
||||
* Every slot port is its own origin. The firewall GUI therefore never runs in the OPNexus origin.
|
||||
* Cookies never cross: the browser's ``Cookie`` header is not forwarded (it carries the OPNexus session),
|
||||
and the firewall's ``Set-Cookie`` never reaches the browser — a server-side jar per gateway session
|
||||
(memory only) holds them. The browser only gets ``__Host-onx-gw-<slot>``.
|
||||
* Each upstream request is built from scratch: method, path and query from the browser, a small header
|
||||
allow-list, the jar's cookies; target origin, pinned certificate and overlay proxy from the backend.
|
||||
Absolute-form targets, CONNECT/TRACE, ``Upgrade`` and service-worker script requests are refused.
|
||||
* ``Sec-Fetch-Site`` must be same-origin/none (same-site only for the redeem step), fail-closed;
|
||||
non-GET/HEAD requests also need the slot's own ``Origin``.
|
||||
* ``Location`` pointing anywhere but the firewall is dropped; responses get CSP, X-Frame-Options DENY,
|
||||
``Cache-Control: no-store``.
|
||||
* At most 3 failed GUI logins per firewall in 5 minutes (protects OPNexus' API access from the firewall's
|
||||
login lockout, which would see all attempts coming from OPNexus).
|
||||
* Idle and absolute limits; every 20 s the backend re-checks the session (OPNexus session, admin role,
|
||||
switch, host, and that the firewall's address, pinned certificate and overlay route are still the ones the
|
||||
session started with).
|
||||
* At most MAX_SESSIONS sessions at once: expired ones are cleared first, a new session is refused otherwise —
|
||||
a running session is never evicted by a newer one. Logs carry method, path (no query), status and duration only.
|
||||
|
||||
OPNsense behaviour this relies on (opnsense/core, src/www/authgui.inc, master as of 2026-10-21):
|
||||
* session_auth(): a login is ``isset($_POST['login']) && !empty($_POST['usernamefld']) &&
|
||||
!empty($_POST['passwordfld'])`` posted to the current URL (login form has no action). Success sends a
|
||||
``Location:`` header and exits (302); a failure re-renders the login form (200) with "Wrong username or
|
||||
password." and an auth_log entry.
|
||||
* check_security_http_referer_enforcement() only runs when HTTP_REFERER is set, and wants the firewall's
|
||||
own host name/IP — so the gateway sends no Referer (and no Origin) upstream at all.
|
||||
* check_security_dns_rebind() accepts an IP address or the configured host name as Host — the gateway
|
||||
sends the Host of the host record's base URL (httpx derives it from the URL).
|
||||
"""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
import ssl
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from http.cookies import CookieError, SimpleCookie
|
||||
from typing import Awaitable, Callable
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("gui_gateway")
|
||||
# httpx/httpcore log full request URLs (query strings) at INFO/DEBUG: never in this process.
|
||||
for _noisy in ("httpx", "httpcore"):
|
||||
logging.getLogger(_noisy).setLevel(logging.WARNING)
|
||||
|
||||
LISTEN_PORT = 8080
|
||||
COOKIE_PREFIX = "__Host-onx-gw-"
|
||||
OPNEXUS_SESSION_COOKIE = "__Host-opnexus-session"
|
||||
BODY_LIMIT = 64 * 1024 * 1024 # config restore uploads
|
||||
LOGIN_FAILURE_LIMIT = 3
|
||||
LOGIN_FAILURE_WINDOW = 300.0
|
||||
UPSTREAM_TIMEOUT = 60.0
|
||||
MAX_SESSIONS = 64
|
||||
|
||||
# Request headers copied from the browser; everything else (Cookie, Authorization, Origin, Referer,
|
||||
# X-Forwarded-*, hop-by-hop, Sec-*, ...) is dropped.
|
||||
_FORWARD_REQUEST_HEADERS = {
|
||||
"accept", "accept-language", "content-type", "user-agent", "x-requested-with", "x-csrftoken", "range",
|
||||
}
|
||||
# Response headers copied from the firewall. Set-Cookie goes to the jar, Location is rewritten.
|
||||
_FORWARD_RESPONSE_HEADERS = {
|
||||
"content-type", "content-length", "content-encoding", "content-disposition", "content-language",
|
||||
"content-range", "accept-ranges", "content-security-policy",
|
||||
}
|
||||
_GATEWAY_CSP = ("connect-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'self'; "
|
||||
"object-src 'none'; worker-src 'none'")
|
||||
_SECURITY_HEADERS = [
|
||||
(b"content-security-policy", _GATEWAY_CSP.encode()),
|
||||
(b"x-frame-options", b"DENY"),
|
||||
(b"x-content-type-options", b"nosniff"),
|
||||
(b"cache-control", b"no-store"),
|
||||
(b"pragma", b"no-cache"),
|
||||
(b"referrer-policy", b"same-origin"),
|
||||
(b"cross-origin-opener-policy", b"same-origin"),
|
||||
(b"cross-origin-resource-policy", b"same-origin"),
|
||||
(b"permissions-policy", b"camera=(), microphone=(), geolocation=(), payment=(), usb=()"),
|
||||
]
|
||||
|
||||
|
||||
def _sha(value: str) -> str:
|
||||
return hashlib.sha256(value.encode()).hexdigest()
|
||||
|
||||
|
||||
def _origin_of(url: str) -> str | None:
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
return None
|
||||
if parts.scheme not in ("http", "https") or not parts.hostname:
|
||||
return None
|
||||
default = 443 if parts.scheme == "https" else 80
|
||||
host = parts.hostname.lower()
|
||||
if ":" in host:
|
||||
host = f"[{host}]"
|
||||
return f"{parts.scheme}://{host}" + (f":{port}" if port and port != default else "")
|
||||
|
||||
|
||||
def slot_origins_from(sites: str) -> dict[int, set[str]]:
|
||||
"""OPNEXUS_GUI_SITES -> {port: {normalised origins}} (https only, explicit port)."""
|
||||
slots: dict[int, set[str]] = {}
|
||||
for raw in sites.split(","):
|
||||
raw = raw.strip().rstrip("/")
|
||||
try:
|
||||
parts = urlsplit(raw)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
continue
|
||||
if parts.scheme == "https" and parts.hostname and port:
|
||||
slots.setdefault(port, set()).add(_origin_of(raw))
|
||||
return slots
|
||||
|
||||
|
||||
@dataclass
|
||||
class GatewaySession:
|
||||
gui_session_id: str
|
||||
session_hash: str
|
||||
slot: int
|
||||
upstream_origin: str
|
||||
ca_pem: str | None
|
||||
proxy_url: str | None
|
||||
idle_timeout: float
|
||||
absolute_deadline: float
|
||||
revalidate_every: float
|
||||
target_fp: str | None = None
|
||||
last_activity: float = field(default_factory=time.monotonic)
|
||||
next_check: float = 0.0
|
||||
jar: dict[str, str] = field(default_factory=dict)
|
||||
client: httpx.AsyncClient | None = None
|
||||
|
||||
|
||||
class BackendClient:
|
||||
"""The backend's internal GUI API. Raises httpx.HTTPError when the backend is unreachable."""
|
||||
|
||||
def __init__(self, base_url: str, token: str, transport: httpx.AsyncBaseTransport | None = None):
|
||||
self._client = httpx.AsyncClient(base_url=base_url, headers={"X-Onx-Gateway-Token": token},
|
||||
timeout=10.0, transport=transport)
|
||||
|
||||
async def redeem(self, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
|
||||
resp = await self._client.post("/internal/gui/redeem", json={
|
||||
"ticket_hash": ticket_hash, "session_hash": session_hash, "slot": slot, "client_ip": client_ip})
|
||||
if resp.status_code == 200:
|
||||
return resp.json()
|
||||
if resp.status_code == 403:
|
||||
return {"ok": False, "reason": (resp.json().get("detail") or {}).get("reason", "denied")}
|
||||
resp.raise_for_status()
|
||||
return {"ok": False, "reason": f"http_{resp.status_code}"}
|
||||
|
||||
async def revalidate(self, gs: GatewaySession) -> dict:
|
||||
resp = await self._client.post("/internal/gui/revalidate",
|
||||
json={"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash,
|
||||
"target_fp": gs.target_fp})
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def end(self, gs: GatewaySession, reason: str) -> None:
|
||||
try:
|
||||
await self._client.post("/internal/gui/end", json={
|
||||
"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash, "reason": reason})
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Could not report the end of a GUI session (%s)", type(exc).__name__)
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._client.aclose()
|
||||
|
||||
|
||||
def build_upstream_client(gs: GatewaySession) -> httpx.AsyncClient:
|
||||
"""Same transport rules as opnsense_client._client: pinned certificate (or system CAs), overlay proxy."""
|
||||
verify: ssl.SSLContext | bool
|
||||
if gs.ca_pem:
|
||||
from app.services.tls_pin_service import pinned_context # no database layer behind this import
|
||||
verify = pinned_context(gs.ca_pem)
|
||||
else:
|
||||
verify = True
|
||||
return httpx.AsyncClient(verify=verify, proxy=gs.proxy_url, timeout=UPSTREAM_TIMEOUT,
|
||||
follow_redirects=False, trust_env=False)
|
||||
|
||||
|
||||
class LoginLimiter:
|
||||
"""At most LOGIN_FAILURE_LIMIT failed GUI logins per firewall within LOGIN_FAILURE_WINDOW seconds."""
|
||||
|
||||
def __init__(self, clock: Callable[[], float] = time.monotonic):
|
||||
self._failures: dict[str, deque] = {}
|
||||
self._clock = clock
|
||||
|
||||
def _recent(self, key: str) -> deque:
|
||||
now = self._clock()
|
||||
q = self._failures.setdefault(key, deque())
|
||||
while q and now - q[0] > LOGIN_FAILURE_WINDOW:
|
||||
q.popleft()
|
||||
return q
|
||||
|
||||
def blocked(self, key: str) -> bool:
|
||||
return len(self._recent(key)) >= LOGIN_FAILURE_LIMIT
|
||||
|
||||
def record_failure(self, key: str) -> None:
|
||||
self._recent(key).append(self._clock())
|
||||
|
||||
|
||||
def is_login_attempt(method: str, content_type: str, body: bytes) -> bool:
|
||||
if method != "POST":
|
||||
return False
|
||||
if content_type.lower().startswith("multipart/form-data"):
|
||||
# PHP fills $_POST from multipart bodies too, so authgui.inc accepts a login posted this way.
|
||||
return all(f'name="{field}"'.encode() in body for field in ("login", "usernamefld", "passwordfld"))
|
||||
if not content_type.lower().startswith("application/x-www-form-urlencoded"):
|
||||
return False
|
||||
try:
|
||||
form = parse_qs(body.decode("latin-1"), keep_blank_values=True)
|
||||
except ValueError:
|
||||
return False
|
||||
return "login" in form and bool(form.get("usernamefld", [""])[0]) and bool(form.get("passwordfld", [""])[0])
|
||||
|
||||
|
||||
def rewrite_location(location: str, upstream_origin: str) -> str | None:
|
||||
"""Relative target for the browser, or None when the redirect leaves the firewall."""
|
||||
location = location.strip()
|
||||
if not location or "\\" in location or any(ord(c) < 0x20 for c in location):
|
||||
return None
|
||||
if location.startswith("//"):
|
||||
location = "https:" + location
|
||||
parts = urlsplit(location)
|
||||
if parts.scheme or parts.netloc:
|
||||
if _origin_of(location) != _origin_of(upstream_origin):
|
||||
return None
|
||||
path = parts.path or "/"
|
||||
return path + (f"?{parts.query}" if parts.query else "") + (f"#{parts.fragment}" if parts.fragment else "")
|
||||
return location
|
||||
|
||||
|
||||
def _page(title: str, text_de: str, text_en: str) -> bytes:
|
||||
return (f"<!doctype html><html lang=\"de\"><head><meta charset=\"utf-8\"><title>{title}</title></head>"
|
||||
f"<body style=\"font-family:sans-serif;max-width:40rem;margin:3rem auto\"><h1>{title}</h1>"
|
||||
f"<p>{text_de}</p><p lang=\"en\">{text_en}</p></body></html>").encode()
|
||||
|
||||
|
||||
_SESSION_GONE = _page("OPNexus", "Diese Sitzung ist abgelaufen oder ungültig. Öffne die Oberfläche erneut aus OPNexus.",
|
||||
"This session has expired or is invalid. Open the interface again from OPNexus.")
|
||||
_FORBIDDEN = _page("OPNexus", "Diese Anfrage wurde vom GUI-Gateway abgelehnt.", "This request was refused by the GUI gateway.")
|
||||
_BAD_REDIRECT = _page("OPNexus", "Die Firewall wollte auf eine fremde Adresse umleiten; die Umleitung wurde verworfen.",
|
||||
"The firewall tried to redirect to a foreign address; the redirect was dropped.")
|
||||
_LOGIN_LOCKED = _page("OPNexus", "Zu viele fehlgeschlagene Anmeldungen an dieser Firewall. Bitte warte 5 Minuten.",
|
||||
"Too many failed sign-ins at this firewall. Please wait 5 minutes.")
|
||||
_UNAVAILABLE = _page("OPNexus", "Die Firewall oder OPNexus ist gerade nicht erreichbar.",
|
||||
"The firewall or OPNexus is not reachable right now.")
|
||||
_TOO_MANY = _page("OPNexus", "Es laufen schon zu viele Firewall-Oberflächen gleichzeitig. Versuche es später erneut.",
|
||||
"Too many firewall interfaces are open at the same time. Try again later.")
|
||||
_REDEEMED = (b"<!doctype html><html><head><meta charset=\"utf-8\"><meta http-equiv=\"refresh\" content=\"0;url=/\">"
|
||||
b"<title>OPNexus</title></head><body><a href=\"/\">→</a></body></html>")
|
||||
|
||||
|
||||
class GuiGateway:
|
||||
def __init__(self, *, backend: BackendClient, slot_origins: dict[int, set[str]],
|
||||
upstream_factory: Callable[[GatewaySession], httpx.AsyncClient] = build_upstream_client,
|
||||
clock: Callable[[], float] = time.monotonic, wall_clock: Callable[[], float] = time.time):
|
||||
self.backend = backend
|
||||
self.slot_origins = slot_origins
|
||||
self.upstream_factory = upstream_factory
|
||||
self.clock = clock
|
||||
self.wall_clock = wall_clock
|
||||
self.sessions: dict[str, GatewaySession] = {} # sha256(gateway cookie) -> session
|
||||
self.limiter = LoginLimiter(clock)
|
||||
self._loop_task: asyncio.Task | None = None
|
||||
|
||||
# ── ASGI ──────────────────────────────────────────────────────────────
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] == "lifespan":
|
||||
await self._lifespan(receive, send)
|
||||
return
|
||||
if scope["type"] != "http":
|
||||
# WebSocket and anything else: not supported (design: no WebSocket).
|
||||
if scope["type"] == "websocket":
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
return
|
||||
started = time.monotonic()
|
||||
status = await self._handle(scope, receive, send)
|
||||
# Method, path and status only: never the query string, headers or bodies.
|
||||
logger.info("%s %s %d %.0fms", scope.get("method", "?"), scope.get("path", "?")[:200], status,
|
||||
(time.monotonic() - started) * 1000)
|
||||
|
||||
async def _lifespan(self, receive, send):
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] == "lifespan.startup":
|
||||
self._loop_task = asyncio.create_task(self._revalidate_loop())
|
||||
await send({"type": "lifespan.startup.complete"})
|
||||
elif message["type"] == "lifespan.shutdown":
|
||||
if self._loop_task:
|
||||
self._loop_task.cancel()
|
||||
for key in list(self.sessions):
|
||||
await self._drop(key, "shutdown")
|
||||
await self.backend.aclose()
|
||||
await send({"type": "lifespan.shutdown.complete"})
|
||||
return
|
||||
|
||||
async def _revalidate_loop(self):
|
||||
while True:
|
||||
await asyncio.sleep(10)
|
||||
try:
|
||||
await self.sweep()
|
||||
except Exception: # never let the loop die
|
||||
logger.exception("GUI session sweep failed")
|
||||
|
||||
async def sweep(self) -> None:
|
||||
"""Idle/absolute limits and the periodic backend re-check for every gateway session."""
|
||||
for key, gs in list(self.sessions.items()):
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
elif self.clock() >= gs.next_check:
|
||||
await self._recheck(key, gs)
|
||||
|
||||
# ── helpers ───────────────────────────────────────────────────────────
|
||||
@staticmethod
|
||||
def _headers(scope) -> dict[str, str]:
|
||||
result: dict[str, str] = {}
|
||||
for k, v in scope.get("headers", []):
|
||||
name = k.decode("latin-1").lower()
|
||||
value = v.decode("latin-1")
|
||||
if name in result:
|
||||
value = f"{result[name]}{'; ' if name == 'cookie' else ', '}{value}"
|
||||
result[name] = value
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _cookies(header: str) -> dict[str, str]:
|
||||
cookies: dict[str, str] = {}
|
||||
for part in header.split(";"):
|
||||
name, sep, value = part.strip().partition("=")
|
||||
if sep and name and name not in cookies:
|
||||
cookies[name] = value
|
||||
return cookies
|
||||
|
||||
async def _respond(self, send, status: int, body: bytes = b"", headers: list | None = None,
|
||||
content_type: bytes = b"text/html; charset=utf-8") -> int:
|
||||
out = [(b"content-type", content_type), (b"content-length", str(len(body)).encode())]
|
||||
out += list(headers or []) + _SECURITY_HEADERS
|
||||
await send({"type": "http.response.start", "status": status, "headers": out})
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
return status
|
||||
|
||||
def _expired(self, gs: GatewaySession) -> str | None:
|
||||
if self.wall_clock() >= gs.absolute_deadline:
|
||||
return "absolute"
|
||||
if self.clock() - gs.last_activity >= gs.idle_timeout:
|
||||
return "idle"
|
||||
return None
|
||||
|
||||
async def _drop(self, key: str, reason: str, *, report: bool = True) -> None:
|
||||
gs = self.sessions.pop(key, None)
|
||||
if gs is None:
|
||||
return
|
||||
gs.jar.clear()
|
||||
if gs.client is not None:
|
||||
await gs.client.aclose()
|
||||
if report:
|
||||
await self.backend.end(gs, reason)
|
||||
|
||||
async def _recheck(self, key: str, gs: GatewaySession) -> bool | None:
|
||||
"""True = still valid, False = ended (dropped), None = backend unreachable (fail closed, keep for retry)."""
|
||||
try:
|
||||
result = await self.backend.revalidate(gs)
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("GUI session re-check failed (%s)", type(exc).__name__)
|
||||
return None
|
||||
if not result.get("ok"):
|
||||
await self._drop(key, str(result.get("reason", "revoked")), report=False) # backend already ended it
|
||||
return False
|
||||
gs.absolute_deadline = min(gs.absolute_deadline, float(result.get("absolute_deadline", gs.absolute_deadline)))
|
||||
gs.idle_timeout = float(result.get("idle_timeout_seconds", gs.idle_timeout))
|
||||
gs.next_check = self.clock() + gs.revalidate_every
|
||||
return True
|
||||
|
||||
async def _clear_expired(self) -> None:
|
||||
for key, gs in list(self.sessions.items()):
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
|
||||
def _has_room(self, browser_hash: str, slot: int) -> bool:
|
||||
"""Room for one more session; the one this redeem would replace (same browser session and slot) does not count."""
|
||||
replaced = sum(1 for g in self.sessions.values() if g.slot == slot and g.session_hash == browser_hash)
|
||||
return len(self.sessions) - replaced < MAX_SESSIONS
|
||||
|
||||
# ── request handling ──────────────────────────────────────────────────
|
||||
async def _handle(self, scope, receive, send) -> int:
|
||||
method = scope.get("method", "")
|
||||
path = scope.get("path", "")
|
||||
raw_path = scope.get("raw_path") or path.encode()
|
||||
headers = self._headers(scope)
|
||||
|
||||
if method in ("CONNECT", "TRACE"):
|
||||
return await self._respond(send, 501, _FORBIDDEN)
|
||||
# Origin-form only: "GET http://elsewhere/ HTTP/1.1" (absolute-form) or "//host" never reach upstream.
|
||||
if not path.startswith("/") or path.startswith("//") or not raw_path.startswith(b"/") \
|
||||
or raw_path.startswith(b"//") or "\\" in path:
|
||||
return await self._respond(send, 400, _FORBIDDEN)
|
||||
if "upgrade" in headers or "upgrade" in headers.get("connection", "").lower():
|
||||
return await self._respond(send, 400, _FORBIDDEN)
|
||||
if headers.get("service-worker", "").lower() == "script":
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
|
||||
try:
|
||||
slot = int(headers.get("x-onx-slot", ""))
|
||||
except ValueError:
|
||||
slot = 0
|
||||
own_origins = self.slot_origins.get(slot)
|
||||
if not own_origins:
|
||||
return await self._respond(send, 400, _FORBIDDEN) # not coming through the proxy's slot block
|
||||
|
||||
site = headers.get("sec-fetch-site", "").strip().lower()
|
||||
if path == "/__onx/redeem":
|
||||
if method != "GET" or site not in ("same-site", "same-origin", "none"):
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
return await self._redeem(scope, headers, slot, send)
|
||||
if site not in ("same-origin", "none"):
|
||||
return await self._respond(send, 403, _FORBIDDEN) # missing header = old/odd client: fail closed
|
||||
if method not in ("GET", "HEAD"):
|
||||
origin = headers.get("origin", "")
|
||||
if _origin_of(origin) not in own_origins:
|
||||
return await self._respond(send, 403, _FORBIDDEN)
|
||||
if path.startswith("/__onx/"):
|
||||
return await self._respond(send, 404, _FORBIDDEN)
|
||||
|
||||
cookies = self._cookies(headers.get("cookie", ""))
|
||||
token = cookies.get(f"{COOKIE_PREFIX}{slot}", "")
|
||||
key = _sha(token) if token else ""
|
||||
gs = self.sessions.get(key) if key else None
|
||||
if gs is None or gs.slot != slot:
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
reason = self._expired(gs)
|
||||
if reason:
|
||||
await self._drop(key, reason)
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
if self.clock() >= gs.next_check:
|
||||
state = await self._recheck(key, gs)
|
||||
if state is False:
|
||||
return await self._respond(send, 401, _SESSION_GONE)
|
||||
if state is None:
|
||||
return await self._respond(send, 503, _UNAVAILABLE)
|
||||
gs.last_activity = self.clock()
|
||||
return await self._proxy(scope, receive, send, headers, gs)
|
||||
|
||||
async def _redeem(self, scope, headers, slot: int, send) -> int:
|
||||
query = parse_qs(scope.get("query_string", b"").decode("latin-1"))
|
||||
ticket = (query.get("t") or [""])[0]
|
||||
browser_session = self._cookies(headers.get("cookie", "")).get(OPNEXUS_SESSION_COOKIE, "")
|
||||
if not ticket or len(ticket) > 128 or not browser_session:
|
||||
return await self._respond(send, 403, _SESSION_GONE)
|
||||
client_ip = (headers.get("x-forwarded-for", "").split(",")[0].strip() or None)
|
||||
# Room first, before the ticket is burned: expired sessions go, running ones are never evicted.
|
||||
await self._clear_expired()
|
||||
if not self._has_room(_sha(browser_session), slot):
|
||||
return await self._respond(send, 503, _TOO_MANY)
|
||||
try:
|
||||
result = await self.backend.redeem(_sha(ticket), _sha(browser_session), slot, client_ip and client_ip[:45])
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Ticket redeem failed (%s)", type(exc).__name__)
|
||||
return await self._respond(send, 503, _UNAVAILABLE)
|
||||
if not result.get("ok"):
|
||||
return await self._respond(send, 403, _SESSION_GONE)
|
||||
# One browser session, one gateway session per slot: an older one in this slot is replaced.
|
||||
for old_key, old in list(self.sessions.items()):
|
||||
if old.slot == slot and old.session_hash == _sha(browser_session):
|
||||
await self._drop(old_key, "superseded", report=False)
|
||||
value = secrets.token_urlsafe(32)
|
||||
gs = GatewaySession(
|
||||
gui_session_id=result["gui_session_id"], session_hash=_sha(browser_session), slot=slot,
|
||||
upstream_origin=result["upstream_origin"], ca_pem=result.get("ca_pem"), proxy_url=result.get("proxy_url"),
|
||||
idle_timeout=float(result["idle_timeout_seconds"]), absolute_deadline=float(result["absolute_deadline"]),
|
||||
revalidate_every=float(result.get("revalidate_seconds", 20)), target_fp=result.get("target_fp"),
|
||||
)
|
||||
if len(self.sessions) >= MAX_SESSIONS: # filled up while the backend was answering
|
||||
await self.backend.end(gs, "revoked")
|
||||
return await self._respond(send, 503, _TOO_MANY)
|
||||
gs.last_activity = self.clock()
|
||||
gs.next_check = gs.last_activity + gs.revalidate_every
|
||||
self.sessions[_sha(value)] = gs
|
||||
cookie = f"{COOKIE_PREFIX}{slot}={value}; Path=/; Secure; HttpOnly; SameSite=Strict"
|
||||
# 200 + meta refresh instead of a 303: the follow-up request then starts in this origin
|
||||
# (Sec-Fetch-Site: same-origin) and the ticket leaves the address bar.
|
||||
return await self._respond(send, 200, _REDEEMED, headers=[
|
||||
(b"set-cookie", cookie.encode()), (b"clear-site-data", b'"cache", "storage"'),
|
||||
])
|
||||
|
||||
async def _read_body(self, receive) -> bytes | None:
|
||||
chunks, size = [], 0
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] == "http.disconnect":
|
||||
return None
|
||||
chunk = message.get("body", b"")
|
||||
size += len(chunk)
|
||||
if size > BODY_LIMIT:
|
||||
return None
|
||||
chunks.append(chunk)
|
||||
if not message.get("more_body"):
|
||||
return b"".join(chunks)
|
||||
|
||||
def _store_cookies(self, gs: GatewaySession, response: httpx.Response) -> None:
|
||||
for header in response.headers.get_list("set-cookie"):
|
||||
jar = SimpleCookie()
|
||||
try:
|
||||
jar.load(header)
|
||||
except CookieError:
|
||||
continue
|
||||
for name, morsel in jar.items():
|
||||
expired = morsel["max-age"] in ("0",) or morsel.value in ("", "deleted")
|
||||
if not expired and morsel["expires"]:
|
||||
try:
|
||||
from email.utils import parsedate_to_datetime
|
||||
expired = parsedate_to_datetime(morsel["expires"]).timestamp() <= self.wall_clock()
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
if expired:
|
||||
gs.jar.pop(name, None)
|
||||
else:
|
||||
gs.jar[name] = morsel.value
|
||||
|
||||
async def _proxy(self, scope, receive, send, headers: dict, gs: GatewaySession) -> int:
|
||||
length = headers.get("content-length", "")
|
||||
if length.isdigit() and int(length) > BODY_LIMIT:
|
||||
return await self._respond(send, 413, _FORBIDDEN)
|
||||
body = await self._read_body(receive)
|
||||
if body is None:
|
||||
return await self._respond(send, 413, _FORBIDDEN)
|
||||
method = scope["method"]
|
||||
login = is_login_attempt(method, headers.get("content-type", ""), body)
|
||||
if login and self.limiter.blocked(gs.upstream_origin):
|
||||
return await self._respond(send, 429, _LOGIN_LOCKED)
|
||||
|
||||
out_headers = {k: v for k, v in headers.items() if k in _FORWARD_REQUEST_HEADERS}
|
||||
out_headers["accept-encoding"] = "identity"
|
||||
if gs.jar:
|
||||
out_headers["cookie"] = "; ".join(f"{k}={v}" for k, v in gs.jar.items())
|
||||
query = scope.get("query_string", b"").decode("latin-1")
|
||||
# The path exactly as the browser sent it (still percent-encoded): the decoded scope["path"] would
|
||||
# turn %3F/%23 into query/fragment, %2F into a separator and decode %25xx a second time upstream.
|
||||
raw_path = (scope.get("raw_path") or scope["path"].encode()).split(b"?", 1)[0].decode("latin-1")
|
||||
url = gs.upstream_origin + raw_path + (f"?{query}" if query else "")
|
||||
if gs.client is None:
|
||||
gs.client = self.upstream_factory(gs)
|
||||
try:
|
||||
request = gs.client.build_request(method, url, headers=out_headers, content=body)
|
||||
except httpx.InvalidURL:
|
||||
return await self._respond(send, 400, _FORBIDDEN) # e.g. control characters in the path
|
||||
try:
|
||||
response = await gs.client.send(request, stream=True)
|
||||
except (httpx.HTTPError, ssl.SSLError, OSError) as exc:
|
||||
logger.warning("Upstream request failed (%s)", type(exc).__name__)
|
||||
return await self._respond(send, 502, _UNAVAILABLE)
|
||||
try:
|
||||
self._store_cookies(gs, response)
|
||||
if login and not (300 <= response.status_code < 400):
|
||||
self.limiter.record_failure(gs.upstream_origin)
|
||||
out = []
|
||||
for name, value in response.headers.multi_items():
|
||||
lname = name.lower()
|
||||
if lname in _FORWARD_RESPONSE_HEADERS:
|
||||
out.append((lname.encode("latin-1"), value.encode("latin-1")))
|
||||
if "location" in response.headers:
|
||||
target = rewrite_location(response.headers["location"], gs.upstream_origin)
|
||||
if target is None:
|
||||
await response.aclose()
|
||||
return await self._respond(send, 502, _BAD_REDIRECT)
|
||||
out.append((b"location", target.encode("latin-1")))
|
||||
await send({"type": "http.response.start", "status": response.status_code, "headers": out + _SECURITY_HEADERS})
|
||||
async for chunk in response.aiter_raw():
|
||||
await send({"type": "http.response.body", "body": chunk, "more_body": True})
|
||||
await send({"type": "http.response.body", "body": b""})
|
||||
return response.status_code
|
||||
finally:
|
||||
await response.aclose()
|
||||
|
||||
|
||||
def create_app(transport: httpx.AsyncBaseTransport | None = None) -> GuiGateway:
|
||||
token = os.environ.get("GUI_GATEWAY_TOKEN", "")
|
||||
if len(token) < 32:
|
||||
raise SystemExit("GUI_GATEWAY_TOKEN (at least 32 characters) is required")
|
||||
slots = slot_origins_from(os.environ.get("OPNEXUS_GUI_SITES", ""))
|
||||
if not slots:
|
||||
raise SystemExit("OPNEXUS_GUI_SITES must list the https slot origins")
|
||||
backend = BackendClient(os.environ.get("GUI_BACKEND_URL", "http://backend:8000"), token, transport)
|
||||
return GuiGateway(backend=backend, slot_origins=slots)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import uvicorn
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
|
||||
# h11: an absolute-form target stays visible in scope["path"] (and is refused); access_log off because
|
||||
# uvicorn's access log would contain query strings (the redeem ticket).
|
||||
uvicorn.run(create_app(), host="0.0.0.0", port=LISTEN_PORT, http="h11", ws="none", access_log=False,
|
||||
server_header=False, date_header=False, proxy_headers=False, lifespan="on", log_config=None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+60
-1
@@ -46,15 +46,25 @@ from app.version import APP_VERSION
|
||||
from app.routers import dashboard as dashboard_router
|
||||
from app.routers import cluster as cluster_router
|
||||
from app.routers import services as services_router
|
||||
from app.routers import gateways as gateways_router
|
||||
from app.routers import resource_samples as resource_samples_router
|
||||
from app.services import resource_monitor_service
|
||||
from app.routers import scheduled_maintenance as scheduled_maintenance_router
|
||||
from app.routers import maintenance_windows as maintenance_windows_router
|
||||
from app.routers import rollouts as rollouts_router
|
||||
from app.routers import rule_analysis as rule_analysis_router
|
||||
from app.routers import rule_cleanup as rule_cleanup_router
|
||||
from app.routers import config_code as config_code_router
|
||||
from app.routers import gui_access as gui_access_router
|
||||
from app.services import scheduled_maintenance_service
|
||||
from app.services import rollout_engine
|
||||
from app.services import compliance_service
|
||||
from app.services import audit_export_service
|
||||
from app.services import overlay_service, variables_service
|
||||
from app.services import backup_schedule_service
|
||||
from app.services import carp_event_service, carp_maintenance_window_service
|
||||
from app.services import rule_history_service
|
||||
from app.services import rule_cleanup_service
|
||||
|
||||
# Without this, Python's root logger defaults to WARNING with no handler,
|
||||
# so every logger.info() call in the 5 background loops below (CARP role
|
||||
@@ -135,6 +145,13 @@ async def _deploy_revert_sweep_loop():
|
||||
logger.info("Auto-reverted %d unconfirmed policy-hierarchy deployment(s)", len(reverted))
|
||||
except Exception:
|
||||
logger.exception("Policy-hierarchy deploy-revert sweep failed")
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
reverted = await rule_cleanup_service.sweep_expired_cleanups(db)
|
||||
if reverted:
|
||||
logger.info("Regel-Bereinigung: %d Operation(en) automatisch zurückgenommen/abgeschlossen", len(reverted))
|
||||
except Exception:
|
||||
logger.exception("Rule-cleanup revert sweep failed")
|
||||
await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
|
||||
|
||||
|
||||
@@ -228,6 +245,9 @@ async def _resource_poll_loop():
|
||||
deleted = await resource_monitor_service.prune_old_samples(db, settings.resource_sample_retention_days)
|
||||
if deleted:
|
||||
logger.info("Resource-Sample-Retention: %d alte Einträge gelöscht", deleted)
|
||||
pruned = await rule_history_service.prune(db)
|
||||
if pruned:
|
||||
logger.info("Regelverlauf-Retention: %d alte Einträge gelöscht", pruned)
|
||||
except Exception:
|
||||
logger.exception("Resource-Poll fehlgeschlagen")
|
||||
await asyncio.sleep(settings.resource_poll_interval_seconds)
|
||||
@@ -254,6 +274,12 @@ async def _scheduled_maintenance_poll_loop():
|
||||
except Exception:
|
||||
logger.exception("Geplante-Wartung-Poll fehlgeschlagen")
|
||||
in_progress = 0
|
||||
# Firmware rollouts (1.25) only ever create schedules; the next tick of the lines above fires them.
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
in_progress += await rollout_engine.advance_runs(db)
|
||||
except Exception:
|
||||
logger.exception("Rollout-Poll fehlgeschlagen")
|
||||
# A handover run is stepped quickly (the partner should take over within a minute or two);
|
||||
# otherwise the relaxed cadence is plenty.
|
||||
await asyncio.sleep(_RUNNING_MAINTENANCE_POLL_SECONDS if in_progress else _SCHEDULED_MAINTENANCE_POLL_SECONDS)
|
||||
@@ -310,8 +336,31 @@ async def _backup_poll_loop():
|
||||
await asyncio.sleep(_BACKUP_POLL_SECONDS)
|
||||
|
||||
|
||||
_RULE_OBSERVE_FIRST_RUN_SECONDS = 150
|
||||
|
||||
|
||||
async def _rule_observe_loop():
|
||||
"""Ninth background loop: reads the rules and their counters of every enabled firewall (two GETs each),
|
||||
keeps the usage sampling current and records what changed since the last look (services/rule_history_service.py).
|
||||
Writes only to the database. The first look at a firewall only stores the starting point."""
|
||||
await asyncio.sleep(_RULE_OBSERVE_FIRST_RUN_SECONDS)
|
||||
while True:
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
result = await rule_history_service.observe_all(db)
|
||||
if result["events"] or result["failed"]:
|
||||
logger.info("Regelverlauf: %s", result)
|
||||
except Exception:
|
||||
logger.exception("Regelverlauf fehlgeschlagen")
|
||||
await asyncio.sleep(max(settings.rule_observe_interval_seconds, settings.rule_observe_min_interval_seconds))
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
conflicts = settings.gui_origin_conflicts()
|
||||
if conflicts:
|
||||
# A GUI-gateway slot must never be a trusted OPNexus origin (CORS with credentials).
|
||||
raise RuntimeError(f"OPNEXUS_GUI_SITES overlaps FRONTEND_ORIGIN: {', '.join(conflicts)} — refusing to start")
|
||||
await migrate.run()
|
||||
async with SessionLocal() as db:
|
||||
await auth_service.bootstrap_admin(db)
|
||||
@@ -335,6 +384,7 @@ async def lifespan(app: FastAPI):
|
||||
compliance_task = asyncio.create_task(_compliance_poll_loop())
|
||||
audit_export_task = asyncio.create_task(_audit_export_poll_loop())
|
||||
backup_task = asyncio.create_task(_backup_poll_loop())
|
||||
rule_observe_task = asyncio.create_task(_rule_observe_loop())
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
@@ -346,7 +396,8 @@ async def lifespan(app: FastAPI):
|
||||
compliance_task.cancel()
|
||||
audit_export_task.cancel()
|
||||
backup_task.cancel()
|
||||
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task):
|
||||
rule_observe_task.cancel()
|
||||
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task, rule_observe_task):
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
@@ -510,9 +561,17 @@ app.include_router(restore_router.router, prefix="/api")
|
||||
app.include_router(dashboard_router.router, prefix="/api")
|
||||
app.include_router(cluster_router.router, prefix="/api")
|
||||
app.include_router(services_router.router, prefix="/api")
|
||||
app.include_router(gateways_router.router, prefix="/api")
|
||||
app.include_router(network_router.router, prefix="/api")
|
||||
app.include_router(snapshots_router.router, prefix="/api")
|
||||
app.include_router(network_drift_router.router, prefix="/api")
|
||||
app.include_router(diagnostics_router.router, prefix="/api")
|
||||
app.include_router(resource_samples_router.router, prefix="/api")
|
||||
app.include_router(scheduled_maintenance_router.router, prefix="/api")
|
||||
app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||
app.include_router(rollouts_router.router, prefix="/api")
|
||||
app.include_router(rule_analysis_router.router, prefix="/api")
|
||||
app.include_router(rule_cleanup_router.router, prefix="/api")
|
||||
app.include_router(config_code_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.internal_router)
|
||||
@@ -93,3 +93,23 @@ __all__ += ["HostResourceSample"]
|
||||
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
__all__ += ["ScheduledMaintenance"]
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
__all__ += ["MaintenanceWindow", "MaintenanceWindowException"]
|
||||
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun, RolloutRunHost, RolloutRunWave, rollout_wave_groups
|
||||
__all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "RolloutRunWave", "rollout_wave_groups"]
|
||||
|
||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||
__all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
|
||||
|
||||
from app.models.rule_history import LiveRuleState, RuleChangeEvent
|
||||
__all__ += ["LiveRuleState", "RuleChangeEvent"]
|
||||
|
||||
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
|
||||
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
|
||||
|
||||
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
|
||||
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
|
||||
from app.models.host_variable import HostVariable
|
||||
__all__ += ["HostVariable"]
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
|
||||
|
||||
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
|
||||
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
|
||||
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
|
||||
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import String, event
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
|
||||
|
||||
|
||||
def config_key_column() -> Mapped[str]:
|
||||
return mapped_column(String(64), unique=True)
|
||||
|
||||
|
||||
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
|
||||
return f"{prefix}-{object_id.hex[:12]}"
|
||||
|
||||
|
||||
def register_config_key(model: type, prefix: str) -> None:
|
||||
"""Fills an empty `config_key` just before INSERT (and the id, which is
|
||||
otherwise only assigned while the INSERT is built)."""
|
||||
|
||||
@event.listens_for(model, "before_insert")
|
||||
def _fill(_mapper, _connection, target) -> None:
|
||||
if target.id is None:
|
||||
target.id = uuid.uuid4()
|
||||
if not target.config_key:
|
||||
target.config_key = default_config_key(prefix, target.id)
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Separation of duties (singleton row): may operators roll changes out themselves?"""
|
||||
from sqlalchemy import Boolean, Integer
|
||||
from sqlalchemy import JSON, Boolean, Integer
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
@@ -11,3 +11,14 @@ class GovernanceSettings(Base):
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
# False: an operator prepares changes (edit, preview) but only an admin may deploy them.
|
||||
operators_may_deploy: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
|
||||
# Rule cleanup (1.31). At most this many rules per cleanup operation (default 10, ceiling 25).
|
||||
cleanup_max_rules: Mapped[int] = mapped_column(Integer, nullable=False, default=10)
|
||||
# A rule must have been switched off (and confirmed) this many days before it may be deleted (minimum 1).
|
||||
cleanup_delete_min_days: Mapped[int] = mapped_column(Integer, nullable=False, default=7)
|
||||
# IPs/CIDRs the admin manages the firewalls from; no rule that could match them may be cleaned up. Empty list =
|
||||
# the cleanup is locked (fail closed). Only an admin may change it.
|
||||
cleanup_management_sources: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# Extra TCP ports the firewalls are managed on (besides the API port of base_url, 443, 80 and 22): a rule that
|
||||
# could match a management source on one of them is protected too. Only an admin may change it.
|
||||
cleanup_management_ports: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""GUI access to managed firewalls through the separate gui-gateway container (1.27).
|
||||
|
||||
Settings are a singleton row (id fixed at 1), off by default. Tickets and gateway
|
||||
sessions only ever store hashes: the raw ticket travels once through the browser,
|
||||
the OPNexus session is referenced by the same SHA-256 that `sessions.token_hash` holds.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
class GuiAccessSettings(Base):
|
||||
__tablename__ = "gui_access_settings"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
idle_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=15, server_default="15")
|
||||
absolute_timeout_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
|
||||
slot_count: Mapped[int] = mapped_column(Integer, nullable=False, default=4, server_default="4")
|
||||
updated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class GuiTicket(Base):
|
||||
"""One-time, 30-second ticket, bound to the OPNexus session, host and slot."""
|
||||
__tablename__ = "gui_tickets"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
ticket_hash: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
slot: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
redeemed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class GuiSession(Base):
|
||||
"""A redeemed ticket: one firewall GUI open in one slot of one browser session."""
|
||||
__tablename__ = "gui_sessions"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
session_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("admin_users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
slot: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
client_ip: Mapped[str | None] = mapped_column(String(45), nullable=True)
|
||||
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
last_checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
end_reason: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Recurring maintenance windows (1.25): "every Sunday 02:00-05:00 Europe/Berlin",
|
||||
"first Tuesday of the month 22:00 for 3 h". A window on its own changes nothing on any
|
||||
firewall; rollout plans (models/rollout.py) use it to decide WHEN they may start work on a host.
|
||||
|
||||
Times are wall-clock times in the window's own IANA time zone, so a window stays at 02:00
|
||||
local time across daylight-saving changes (see services/window_service.py for the rules).
|
||||
"""
|
||||
import uuid
|
||||
from datetime import date, datetime, time, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, Date, DateTime, ForeignKey, Integer, String, Time, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class MaintenanceWindow(Base):
|
||||
__tablename__ = "maintenance_windows"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(120), unique=True)
|
||||
timezone: Mapped[str] = mapped_column(String(64))
|
||||
recurrence: Mapped[str] = mapped_column(String(16)) # weekly | monthly
|
||||
# weekly: weekdays 0 (Monday) .. 6 (Sunday). JSON columns get a Python default only (Alembic gotcha, see 0007).
|
||||
weekdays: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# monthly: "nth_weekday" (month_nth 1-4 or -1 = last, month_weekday 0-6) or "day" (month_day 1-28)
|
||||
month_mode: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
month_nth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
month_weekday: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
month_day: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
start_time: Mapped[time] = mapped_column(Time)
|
||||
duration_minutes: Mapped[int] = mapped_column(Integer)
|
||||
# After this many minutes into the window no NEW host action starts (running ones finish).
|
||||
start_cutoff_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
created_by: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
|
||||
exceptions: Mapped[list["MaintenanceWindowException"]] = relationship(
|
||||
"MaintenanceWindowException", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by="MaintenanceWindowException.occurrence_date",
|
||||
)
|
||||
|
||||
|
||||
class MaintenanceWindowException(Base):
|
||||
"""One occurrence skipped or moved. Keyed by the occurrence's LOCAL date."""
|
||||
|
||||
__tablename__ = "maintenance_window_exceptions"
|
||||
__table_args__ = (UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
window_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
occurrence_date: Mapped[date] = mapped_column(Date)
|
||||
kind: Mapped[str] = mapped_column(String(8)) # skip | move
|
||||
moved_start: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
reason: Mapped[str] = mapped_column(String(300), default="")
|
||||
actor: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
@@ -6,7 +6,7 @@ OPNsense API secrets stay off the database entirely.
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Integer, String, UniqueConstraint
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Float, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -17,6 +17,13 @@ def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
DEFAULT_SERVICE_WATCHLIST = (
|
||||
"unbound", "dnsmasq", "dhcpd", "dhcpd6", "kea-dhcp4", "kea-dhcp6", "ntpd", "openvpn", "wireguard", "strongswan",
|
||||
)
|
||||
|
||||
CHANNEL_TYPES = ("ntfy", "gotify", "telegram", "teams", "matrix")
|
||||
|
||||
|
||||
class NotificationSettings(Base):
|
||||
"""Single row, id fixed at 1 — see notification_settings_service.get_settings
|
||||
for the get-or-create helper. The only admin-editable operational config
|
||||
@@ -55,6 +62,34 @@ class NotificationSettings(Base):
|
||||
# Off by default: switching it on for an existing fleet reports every long-standing finding at once.
|
||||
compliance_alerts_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
# Monitoring alerts (1.24). The gateway alert is on by default (a WAN outage is the one thing every
|
||||
# operator wants to hear about); resource thresholds and service checks are off, like compliance alerts:
|
||||
# switching them on for an existing fleet may report long-standing conditions all at once.
|
||||
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# An API key that cannot read gateway / service state makes those alerts silently blind (1.29): say so.
|
||||
api_rights_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
# Gateways whose alert is switched off, "<host_id>:<gateway name>" (1.29), e.g. an idle second WAN.
|
||||
gateway_alert_muted: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
|
||||
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
|
||||
# OPNexus threshold on the measured values (0 = off).
|
||||
gateway_loss_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=20, server_default="20")
|
||||
gateway_delay_threshold_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=500, server_default="500")
|
||||
# Resources (0 = that threshold is off). Load is per CPU thread.
|
||||
disk_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=90, server_default="90")
|
||||
memory_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=95, server_default="95")
|
||||
swap_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=50, server_default="50")
|
||||
load_per_cpu_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=2.0, server_default="2.0")
|
||||
# Debouncing: consecutive 60 s resource samples that must agree before a problem opens or closes.
|
||||
alert_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=3, server_default="3")
|
||||
resource_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5")
|
||||
# OPNsense service names that are reported when they stop (the API cannot tell a crash from a stop on purpose).
|
||||
# NULL = the default list (an old row never got one).
|
||||
service_watchlist: Mapped[list | None] = mapped_column(JSON, nullable=True, default=lambda: list(DEFAULT_SERVICE_WATCHLIST))
|
||||
|
||||
poll_interval_seconds: Mapped[int] = mapped_column(Integer, default=300)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
|
||||
@@ -85,3 +120,21 @@ class NotificationProblem(Base):
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
last_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
resolved_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class NotificationChannel(Base):
|
||||
"""An additional delivery channel (ntfy, Gotify, Telegram, Microsoft Teams, Matrix) next to the SMTP
|
||||
and generic-webhook settings above. Several channels of one type are allowed. ``config`` holds only
|
||||
non-secret parameters (server URL, topic, chat id, room id); the credential — token, or the whole
|
||||
webhook URL for Teams — lives in notification_secret_store under ``channel_<id>``, never here."""
|
||||
|
||||
__tablename__ = "notification_channels"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||||
channel_type: Mapped[str] = mapped_column(String(16))
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
secret_set: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.one_to_one_target import one_to_one_targets
|
||||
from app.models.policy import one_to_one_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
|
||||
__tablename__ = "one_to_one_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
|
||||
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
|
||||
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OneToOneRule, "nat1")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.outbound_nat_target import outbound_nat_targets
|
||||
from app.models.policy import outbound_nat_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
|
||||
__tablename__ = "outbound_nat_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
|
||||
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
|
||||
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OutboundNatRule, "out")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
|
||||
|
||||
class PolicyHierarchyRule(Base):
|
||||
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
|
||||
__tablename__ = "policy_hierarchy_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
|
||||
)
|
||||
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
|
||||
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
|
||||
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PolicyHierarchyRule, "ph")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.port_forward_target import port_forward_targets
|
||||
from app.models.policy import port_forward_groups
|
||||
|
||||
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
|
||||
__tablename__ = "port_forward_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
|
||||
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
|
||||
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
|
||||
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PortForwardRule, "pf")
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Firmware rollouts in waves (1.25): plans (what, in which order, when) and runs (what happened).
|
||||
|
||||
A plan's waves point at device groups ("rings": canary -> pilot -> prod). A run freezes the plan
|
||||
(``plan_snapshot``) and the resolved hosts (``RolloutRunHost``) when it starts, so later edits of
|
||||
the plan or of a group never change a run that is under way. The run never fires firmware actions
|
||||
itself: every host step is an ordinary ScheduledMaintenance row (origin="rollout",
|
||||
auto_verify=True) linked from ``RolloutRunHost.scheduled_maintenance_id`` — see
|
||||
services/rollout_service.py.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import (JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Table, Text,
|
||||
UniqueConstraint)
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
rollout_wave_groups = Table(
|
||||
"rollout_wave_groups", Base.metadata,
|
||||
Column("wave_id", PGUUID(as_uuid=True), ForeignKey("rollout_plan_waves.id", ondelete="CASCADE"), primary_key=True),
|
||||
Column("group_id", PGUUID(as_uuid=True), ForeignKey("device_groups.id", ondelete="CASCADE"), primary_key=True),
|
||||
)
|
||||
|
||||
|
||||
class RolloutPlan(Base):
|
||||
__tablename__ = "rollout_plans"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(120), unique=True)
|
||||
description: Mapped[str] = mapped_column(Text, default="")
|
||||
action: Mapped[str] = mapped_column(String(16)) # update | major_update
|
||||
window_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
# Start a run by itself at every occurrence of the window. Only for "update" and only while the plan
|
||||
# is activated AND unchanged since (activated_fingerprint).
|
||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
|
||||
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
|
||||
activated_by_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
|
||||
auto_start_error: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
created_by: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
|
||||
waves: Mapped[list["RolloutPlanWave"]] = relationship(
|
||||
"RolloutPlanWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by="RolloutPlanWave.position")
|
||||
|
||||
|
||||
class RolloutPlanWave(Base):
|
||||
__tablename__ = "rollout_plan_waves"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
plan_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="CASCADE"), index=True)
|
||||
position: Mapped[int] = mapped_column(Integer)
|
||||
label: Mapped[str] = mapped_column(String(60))
|
||||
soak_minutes: Mapped[int] = mapped_column(Integer, default=30)
|
||||
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Wait for a human "release" before this wave starts (not used for the first wave: starting is the release).
|
||||
require_approval: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
groups: Mapped[list["DeviceGroup"]] = relationship("DeviceGroup", secondary=rollout_wave_groups, lazy="selectin") # noqa: F821
|
||||
|
||||
|
||||
class RolloutRun(Base):
|
||||
__tablename__ = "rollout_runs"
|
||||
__table_args__ = (UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
plan_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
plan_name: Mapped[str] = mapped_column(String(120))
|
||||
plan_snapshot: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
action: Mapped[str] = mapped_column(String(16))
|
||||
# waiting_window | preflight | wave_running | soaking | gate | awaiting_approval | resuming | paused
|
||||
# | succeeded | cancelled
|
||||
status: Mapped[str] = mapped_column(String(20), index=True)
|
||||
paused_from: Mapped[str | None] = mapped_column(String(20), nullable=True)
|
||||
current_wave: Mapped[int] = mapped_column(Integer, default=0)
|
||||
target_version: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_by: Mapped[str] = mapped_column(String(150))
|
||||
started_by_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True) # see activated_by_id
|
||||
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
|
||||
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
|
||||
revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
progress: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
step_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
waves: Mapped[list["RolloutRunWave"]] = relationship(
|
||||
"RolloutRunWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin", order_by="RolloutRunWave.index")
|
||||
hosts: Mapped[list["RolloutRunHost"]] = relationship(
|
||||
"RolloutRunHost", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by=lambda: [RolloutRunHost.wave_index, RolloutRunHost.order])
|
||||
|
||||
|
||||
class RolloutRunWave(Base):
|
||||
__tablename__ = "rollout_run_waves"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
|
||||
index: Mapped[int] = mapped_column(Integer)
|
||||
label: Mapped[str] = mapped_column(String(60))
|
||||
# pending | awaiting_approval | preflight | running | soaking | gate | passed | skipped | cancelled
|
||||
status: Mapped[str] = mapped_column(String(20), default="pending")
|
||||
baseline: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
gate_result: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
soak_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
gate_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
approved_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class RolloutRunHost(Base):
|
||||
__tablename__ = "rollout_run_hosts"
|
||||
__table_args__ = (UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
|
||||
wave_index: Mapped[int] = mapped_column(Integer)
|
||||
host_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
host_name: Mapped[str] = mapped_column(String(120))
|
||||
cluster_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
||||
order: Mapped[int] = mapped_column(Integer, default=0)
|
||||
# queued | running | verifying | succeeded | skipped | failed | blocked | cancelled
|
||||
status: Mapped[str] = mapped_column(String(16), default="queued")
|
||||
scheduled_maintenance_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("scheduled_maintenance.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
version_before: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
version_after: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
# {"code": ..., "params": {...}} — translated in the UI (rollouts.hostNote.*)
|
||||
detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.rule_target import rule_targets
|
||||
from app.models.policy import rule_groups
|
||||
|
||||
@@ -18,6 +19,7 @@ class FirewallRule(Base):
|
||||
__tablename__ = "firewall_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
direction: Mapped[str] = mapped_column(String(8))
|
||||
@@ -44,3 +46,6 @@ class FirewallRule(Base):
|
||||
deployments: Mapped[list["RuleDeployment"]] = relationship(
|
||||
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(FirewallRule, "r")
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): switching rules off or deleting them through preview -> deploy -> confirm.
|
||||
|
||||
``RuleCleanupOperation`` is one cleanup run on a host (or a cluster pair, anchored at one host); its id doubles as
|
||||
the preview token. ``RuleCleanupItem`` is one rule on one node of that run. The columns only carry what the design
|
||||
(DESIGN-1.31) needs to resume, revert and restore after a crash; the logic lives in services/rule_cleanup_service.py.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import JSON, DateTime, ForeignKey, Index, Integer, String, Text, text
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
# kind: disable | enable (re-enable after a restore) | delete | restore
|
||||
OPERATION_KINDS = ("disable", "enable", "delete", "restore")
|
||||
# preview -> running -> awaiting_confirm -> confirmed; the rest are the side exits.
|
||||
OPERATION_STATUSES = (
|
||||
"preview", "running", "awaiting_confirm", "confirmed", "partial", "apply_failed", "reverting", "reverted",
|
||||
"revert_failed", "done", "failed", "cancelled", "expired",
|
||||
)
|
||||
ITEM_STATES = (
|
||||
"queued", "disabling", "disabled", "confirmed", "reverting", "reverted", "deleting", "deleted", "restoring",
|
||||
"restored", "failed", "unknown", "revert_failed",
|
||||
# never attempted (the run stopped at an earlier error); not an open state, so the rule is free again
|
||||
"skipped",
|
||||
)
|
||||
# A rule in one of these states is being worked on or its outcome is unresolved: no second item may claim it.
|
||||
OPEN_ITEM_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class RuleCleanupOperation(Base):
|
||||
__tablename__ = "rule_cleanup_operations"
|
||||
__table_args__ = (
|
||||
Index("ix_rule_cleanup_operations_host", "host_id", "created_at"),
|
||||
Index("ix_rule_cleanup_operations_status", "status"),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
cluster_name: Mapped[Optional[str]] = mapped_column(String(120), nullable=True)
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="preview")
|
||||
actor: Mapped[str] = mapped_column(String(150), nullable=False)
|
||||
# The stable account id next to the name (see migration 0026); no foreign key, a deleted account stays gone.
|
||||
actor_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
started_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Written at every checkpoint, so another process can tell an abandoned ``running`` operation from a live one.
|
||||
heartbeat_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Auto-revert deadline (disable/enable only); NULL once confirmed or for kinds without a window.
|
||||
revert_deadline: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
confirmed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||
# sha256 over everything the preview showed; the deploy recomputes it and refuses on any difference.
|
||||
preview_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# unused_days, acknowledged rules (with the text version), protected management sources.
|
||||
options: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
|
||||
|
||||
class RuleCleanupItem(Base):
|
||||
__tablename__ = "rule_cleanup_items"
|
||||
__table_args__ = (
|
||||
Index("ix_rule_cleanup_items_operation", "operation_id"),
|
||||
# Independent of any lock: one rule on one node can be in only one unfinished item at a time.
|
||||
Index(
|
||||
"uq_cleanup_rule_open", "host_id", "rule_uuid", unique=True,
|
||||
postgresql_where=text("state IN (" + ", ".join(f"'{s}'" for s in OPEN_ITEM_STATES) + ")"),
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
operation_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rule_cleanup_operations.id", ondelete="CASCADE"), nullable=False)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
seq: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
carp_role: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
state: Mapped[str] = mapped_column(String(16), nullable=False, default="queued")
|
||||
# What the preview showed.
|
||||
description: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
categories: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
confidence: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
row_before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
fp_before: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# shadowed/duplicate: the covering rule must stay unchanged until the deploy.
|
||||
cover_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
cover_fp: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# Flattened getRule, raw answer and firmware; written BEFORE a rule is deleted.
|
||||
snapshot: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
# Where the rule sat (neighbours in the same prio_group by sort_order), for the restore.
|
||||
anchor_next_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
anchor_prev_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
prio_group: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
# delete -> the confirmed disable item it builds on; restore -> the delete item it undoes. Plain id, no foreign key.
|
||||
source_item_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
restored_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
changed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Change history of the rules that are live on a firewall (all rules, also hand-made ones).
|
||||
|
||||
``live_rule_states`` is the last observed state per rule (a fingerprint plus the row itself), maintained by the
|
||||
``rule_observe`` loop; ``rule_change_events`` is what changed between two observations. See
|
||||
services/rule_history_service.py. Neither table holds anything the firewall itself would not show on its rule page.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Index, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
class LiveRuleState(Base):
|
||||
__tablename__ = "live_rule_states"
|
||||
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), primary_key=True)
|
||||
# The rule's UUID/label on the firewall; BASELINE_KEY marks "this host has been observed once".
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
row: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
# False = the rule is gone (kept as a marker so a returning UUID continues the numbering).
|
||||
present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
# Counts the observed changes of this rule; the event of a change gets the new number, which makes the
|
||||
# observation idempotent when two processes see the same change.
|
||||
version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
changed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
|
||||
|
||||
class RuleChangeEvent(Base):
|
||||
__tablename__ = "rule_change_events"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("host_id", "rule_uuid", "seq", name="uq_rule_change_event"),
|
||||
Index("ix_rule_change_events_rule", "host_id", "rule_uuid", "observed_at"),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
seq: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
observed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True)
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False) # added | changed | removed | enabled | disabled
|
||||
before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
after: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
origin: Mapped[str] = mapped_column(String(16), nullable=False) # opnexus | external
|
||||
actor: Mapped[Optional[str]] = mapped_column(String(150), nullable=True)
|
||||
# The audit entry that explains an OPNexus-made change; no foreign key, the audit trail outlives everything.
|
||||
audit_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
# Start of the time since the previous observation of the host: only audit entries from then on can explain
|
||||
# this change (a later reconcile must not reach back to an older deployment).
|
||||
window_start: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
@@ -42,7 +42,12 @@ class ScheduledMaintenance(Base):
|
||||
host_operation_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
# Opt-in: hand the traffic over to the CARP partner first, run the action, then switch back.
|
||||
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Only while status == "running" with a handover: handover_wait | job_wait | leave | leave_wait.
|
||||
# 1.25: check the result automatically (job_wait -> verify) and close the operation, instead of
|
||||
# leaving it "submitted" for a human. Every rollout step sets it; off for existing plans.
|
||||
auto_verify: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# manual | rollout (created by a rollout run, see models/rollout.py RolloutRunHost.scheduled_maintenance_id)
|
||||
origin: Mapped[str] = mapped_column(String(16), nullable=False, default="manual", server_default="manual")
|
||||
# Only while status == "running" with a handover or auto_verify: handover_wait | job_wait | verify | leave | leave_wait.
|
||||
# State lives here, not in memory, so a backend restart in the middle of a run resumes cleanly.
|
||||
phase: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
phase_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
"""The three "take over what exists on the firewall" endpoints that aliases, port forwards, 1:1 NAT and
|
||||
outbound NAT share (services/object_adopt_service.py has the logic and the evidence):
|
||||
|
||||
GET <prefix>/unmanaged?host_id=… candidates found live on the host, with reasons for the refused ones
|
||||
POST <prefix>/adopt take over one object {host_id, uuid}; returns the new central object
|
||||
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
|
||||
|
||||
Like rule adoption this only creates central records; nothing is written to the firewall."""
|
||||
import json
|
||||
import uuid
|
||||
from typing import Callable
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.adopt import AdoptAllRequest, AdoptAllResult, AdoptObjectRequest, UnmanagedObject
|
||||
from app.services import object_adopt_service
|
||||
from app.services.adopt_common import LINK_ERRORS, link_api_error
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
|
||||
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
return host
|
||||
|
||||
|
||||
def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Callable, to_read: Callable, read_model) -> None:
|
||||
@router.get("/unmanaged", response_model=list[UnmanagedObject])
|
||||
async def list_unmanaged(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
host = await _host(db, host_id)
|
||||
try:
|
||||
return await object_adopt_service.list_unmanaged(db, kind, host)
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
|
||||
@router.post("/adopt", response_model=read_model, status_code=201)
|
||||
async def adopt(payload: AdoptObjectRequest, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
created = await object_adopt_service.adopt(db, kind, host, payload.uuid, link_to=payload.link_to)
|
||||
except object_adopt_service.NotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
|
||||
except object_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons),
|
||||
reason_details=json.dumps(exc.details, ensure_ascii=False)) from exc
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
# The new object sits in the session without its relationships loaded; db.get() would return it as-is.
|
||||
created = (await db.execute(
|
||||
select(model).where(model.id == created.id).options(*load_options())
|
||||
.execution_options(populate_existing=True)
|
||||
)).scalar_one()
|
||||
await hydrate_hierarchy_targets(db, [created])
|
||||
return to_read(created, created.targets, created.deployments)
|
||||
|
||||
@router.post("/adopt-all", response_model=AdoptAllResult)
|
||||
async def adopt_all(payload: AdoptAllRequest, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
result = await object_adopt_service.adopt_all(db, kind, host, link_existing=payload.link_existing)
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
return result
|
||||
@@ -17,6 +17,7 @@ from app.schemas.alias import DeploymentResult, AliasCreate, AliasRead, AliasEdi
|
||||
from app.services.alias_deploy_service import confirm_alias, deploy_alias, undeploy_alias, revert_alias_deployment
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/aliases", tags=["aliases"])
|
||||
|
||||
@@ -70,6 +71,9 @@ def _to_read(alias: FirewallAlias, hosts: list[OpnsenseHost], deployments: list[
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="alias", model=FirewallAlias, load_options=_load_options, to_read=_to_read, read_model=AliasRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[AliasRead])
|
||||
async def list_aliases(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(FirewallAlias).options(*_load_options()))
|
||||
@@ -201,6 +205,8 @@ async def remove_previous_target(alias_id: uuid.UUID, host_id: uuid.UUID, approv
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in alias.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_alias_item(deployment.host, deployment.opnsense_alias_uuid)
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from sqlalchemy import select, func, or_
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models.audit import AuditEvent
|
||||
from app.services.audit_service import ADMIN_ONLY_DETAILS
|
||||
|
||||
router = APIRouter(prefix="/audit", tags=["audit"])
|
||||
|
||||
@router.get("")
|
||||
async def history(kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||
async def history(request: Request, kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||
status: str | None = None, action: str | None = None, actor: str | None = None,
|
||||
before: datetime | None = None, since: datetime | None = None,
|
||||
q: str = Query(default="", max_length=200), offset: int = Query(default=0, ge=0),
|
||||
@@ -24,5 +25,17 @@ async def history(kind: str | None = None, host_id: str | None = None, entity_id
|
||||
clauses.append(or_(*(c.ilike('%'+q+'%') for c in [AuditEvent.entity_name, AuditEvent.host_name, AuditEvent.actor, AuditEvent.message])))
|
||||
total = await db.scalar(select(func.count()).select_from(AuditEvent).where(*clauses))
|
||||
rows = (await db.scalars(select(AuditEvent).where(*clauses).order_by(AuditEvent.created_at.desc(), AuditEvent.id.desc()).offset(offset).limit(limit))).all()
|
||||
return {"total": total, "offset": offset, "limit": limit,
|
||||
"items": [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]}
|
||||
items = [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]
|
||||
if not full_view(request):
|
||||
for item in items:
|
||||
if isinstance(item.get("details"), dict) and ADMIN_ONLY_DETAILS in item["details"]:
|
||||
item["details"] = {**{k: v for k, v in item["details"].items() if k != ADMIN_ONLY_DETAILS},
|
||||
ADMIN_ONLY_DETAILS: "redacted"}
|
||||
return {"total": total, "offset": offset, "limit": limit, "items": items}
|
||||
|
||||
|
||||
def full_view(request: Request | None) -> bool:
|
||||
"""An admin account, and with a token only one of scope admin."""
|
||||
user = getattr(getattr(request, "state", None), "user", None)
|
||||
token = getattr(getattr(request, "state", None), "api_token", None)
|
||||
return getattr(user, "role", None) == "admin" and (token is None or token.scope == "admin")
|
||||
+46
-10
@@ -15,7 +15,31 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
||||
|
||||
# Plain name: the session cookie of HTTP-only (lab) deployments. On HTTPS deployments it is NOT accepted any
|
||||
# more (1.29): it is only still deleted (login, logout) so that a pre-1.27 cookie does not linger.
|
||||
COOKIE_NAME = "opnexus-session"
|
||||
# HTTPS deployments: the __Host- prefix makes the browser refuse any cookie of
|
||||
# this name that is not Secure, has a Domain attribute or a Path other than
|
||||
# "/". A page on another port of the same host (the GUI gateway slots, any
|
||||
# other service on that machine) therefore cannot toss a narrower-path cookie
|
||||
# that shadows the real session, and cannot overwrite it from script (HttpOnly).
|
||||
SECURE_COOKIE_NAME = "__Host-opnexus-session"
|
||||
|
||||
|
||||
def session_cookie_name() -> str:
|
||||
return SECURE_COOKIE_NAME if settings.cookie_secure else COOKIE_NAME
|
||||
|
||||
|
||||
def session_token(request: Request) -> str:
|
||||
"""The session token the browser sent: the __Host- cookie on HTTPS deployments, the plain one otherwise.
|
||||
Never the other name (1.29): a plain-name cookie can be planted by any plain-HTTP or sibling-subdomain page and
|
||||
must not be able to carry a session on an HTTPS deployment (the __Host- prefix does not separate PORTS, see
|
||||
BACKLOG S8)."""
|
||||
return request.cookies.get(session_cookie_name(), "")
|
||||
|
||||
|
||||
def has_session_cookie(request: Request) -> bool:
|
||||
return bool(request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME))
|
||||
|
||||
|
||||
def client_ip(request: Request) -> str:
|
||||
@@ -32,7 +56,7 @@ def client_ip(request: Request) -> str:
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, max_age: int | None = None) -> None:
|
||||
response.set_cookie(
|
||||
key=COOKIE_NAME,
|
||||
key=session_cookie_name(),
|
||||
value=token,
|
||||
max_age=max_age or settings.session_lifetime,
|
||||
httponly=True,
|
||||
@@ -40,10 +64,15 @@ def _set_session_cookie(response: Response, token: str, max_age: int | None = No
|
||||
secure=settings.cookie_secure,
|
||||
path="/",
|
||||
)
|
||||
if settings.cookie_secure:
|
||||
# A pre-1.27 plain-name cookie must not linger next to the new one.
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||
|
||||
|
||||
def _clear_session_cookie(response: Response) -> None:
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/")
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=settings.cookie_secure, httponly=True, samesite="lax")
|
||||
if settings.cookie_secure:
|
||||
response.delete_cookie(key=SECURE_COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||
|
||||
|
||||
def _check_account_budget(user: AdminUser, action: str) -> None:
|
||||
@@ -71,7 +100,7 @@ async def resolve_user(request: Request, db: AsyncSession) -> AdminUser:
|
||||
user = await db.get(AdminUser, resolved.id)
|
||||
if user is not None:
|
||||
return user
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
user = await auth_service.resolve_session(db, token)
|
||||
if user is None:
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.NICHT_AUTHENTIFIZIERT)
|
||||
@@ -110,7 +139,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
||||
|
||||
verified_hash = user.password_hash
|
||||
if user.totp_enabled:
|
||||
second_factor_checked = bool(user.totp_enabled) # the state this login is decided on; create_session re-checks it
|
||||
if second_factor_checked:
|
||||
# Only revealed after the password was correct, so it cannot be used
|
||||
# to find out which usernames exist or have 2FA.
|
||||
if not payload.totp_code:
|
||||
@@ -124,7 +154,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
|
||||
auth_service.throttle.reset(ip, payload.username)
|
||||
failed_since_last_login = auth_service.throttle.pop_failures_since_success(payload.username)
|
||||
token, expires_at = await auth_service.create_session(
|
||||
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash
|
||||
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash,
|
||||
verified_totp=second_factor_checked,
|
||||
)
|
||||
_set_session_cookie(response, token)
|
||||
logger.info("Successful login for '%s' from %s", user.username, ip)
|
||||
@@ -223,13 +254,15 @@ async def totp_enable(
|
||||
_account_failure(user, "2fa-enable")
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.TOTP_INVALID)
|
||||
_account_success(user, "2fa-enable")
|
||||
# Same row lock as create_session: a sign-in that decided "no second factor needed" is ordered before or after this.
|
||||
await db.execute(select(AdminUser.id).where(AdminUser.id == user.id).with_for_update())
|
||||
plain, hashes = totp_service.new_recovery_codes()
|
||||
totp_service.promote_pending(user.id)
|
||||
user.totp_enabled = True
|
||||
user.totp_last_step = step
|
||||
user.totp_recovery_hashes = hashes
|
||||
# Other sessions were established without the second factor — end them.
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
|
||||
await db.commit()
|
||||
logger.info("Two-factor enabled for '%s'", user.username)
|
||||
return {"enabled": True, "recovery_codes": plain}
|
||||
@@ -254,7 +287,7 @@ async def totp_disable(
|
||||
user.totp_enabled = False
|
||||
user.totp_recovery_hashes = None
|
||||
user.totp_last_step = 0
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=request.cookies.get(COOKIE_NAME, ""), commit=False)
|
||||
await auth_service.revoke_sessions(db, user.id, keep_token=session_token(request), commit=False)
|
||||
await db.commit()
|
||||
logger.info("Two-factor disabled for '%s'", user.username)
|
||||
return {"enabled": False}
|
||||
@@ -262,14 +295,17 @@ async def totp_disable(
|
||||
|
||||
@router.post("/logout")
|
||||
async def logout(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
await auth_service.revoke_session(db, token)
|
||||
legacy = request.cookies.get(COOKIE_NAME, "")
|
||||
if settings.cookie_secure and legacy and legacy != token:
|
||||
await auth_service.revoke_session(db, legacy) # an old plain-name cookie ends with the logout too
|
||||
_clear_session_cookie(response)
|
||||
return {"authenticated": False}
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
async def me(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async def me(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||
raw = api_token_service.bearer_from_header(request.headers.get("authorization", ""))
|
||||
if raw is not None:
|
||||
resolved = await api_token_service.resolve(db, raw, client_ip(request))
|
||||
@@ -278,7 +314,7 @@ async def me(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
user, api_token = resolved
|
||||
return {"authenticated": True, "username": user.username, "role": user.role,
|
||||
"via": "token", "scope": api_token.scope, "token_name": api_token.name}
|
||||
token = request.cookies.get(COOKIE_NAME, "")
|
||||
token = session_token(request)
|
||||
user = await auth_service.resolve_session(db, token)
|
||||
if user is None:
|
||||
return {"authenticated": False}
|
||||
|
||||
@@ -37,6 +37,13 @@ acme_accounts_router = APIRouter(prefix="/acme/accounts", tags=["acme"])
|
||||
acme_certificates_router = APIRouter(prefix="/acme/certificates", tags=["acme"])
|
||||
|
||||
|
||||
def _ensure_host_unchanged_once_deployed(obj, new_host_id: uuid.UUID) -> None:
|
||||
"""The firewall-local UUID, the stored payload and the success status all belong to the host the object was pushed
|
||||
to. Re-pointing it would report success for a host that never received it and orphan the object on the old one."""
|
||||
if new_host_id != obj.host_id and obj.opnsense_uuid:
|
||||
raise api_error(409, ErrorCode.ZIEL_HOST_NACH_AUSROLLEN_FEST)
|
||||
|
||||
|
||||
async def _get_host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
@@ -82,6 +89,7 @@ async def edit_managed_certificate(cert_id: uuid.UUID, payload: ManagedCertifica
|
||||
if cert.updated_at != payload.expected_updated_at:
|
||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||
await _get_host(db, payload.host_id)
|
||||
_ensure_host_unchanged_once_deployed(cert, payload.host_id)
|
||||
for field, value in payload.model_dump(exclude={"expected_updated_at", "private_key"}).items():
|
||||
setattr(cert, field, value)
|
||||
if payload.private_key:
|
||||
@@ -146,6 +154,11 @@ async def edit_acme_account(account_id: uuid.UUID, payload: AcmeAccountEdit, db:
|
||||
if account.updated_at != payload.expected_updated_at:
|
||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||
await _get_host(db, payload.host_id)
|
||||
_ensure_host_unchanged_once_deployed(account, payload.host_id)
|
||||
if payload.host_id != account.host_id and await db.scalar(
|
||||
select(AcmeCertificateRequest.id).where(AcmeCertificateRequest.account_id == account.id).limit(1)
|
||||
):
|
||||
raise api_error(409, ErrorCode.ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET)
|
||||
for field, value in payload.model_dump(exclude={"expected_updated_at", "eab_hmac"}).items():
|
||||
setattr(account, field, value)
|
||||
if payload.eab_hmac:
|
||||
@@ -236,6 +249,7 @@ async def edit_acme_certificate_request(request_id: uuid.UUID, payload: AcmeCert
|
||||
if req.updated_at != payload.expected_updated_at:
|
||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||
await _get_host(db, payload.host_id)
|
||||
_ensure_host_unchanged_once_deployed(req, payload.host_id)
|
||||
await _get_account(db, payload.account_id, payload.host_id)
|
||||
for field, value in payload.model_dump(exclude={"expected_updated_at"}).items():
|
||||
setattr(req, field, value)
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
|
||||
|
||||
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
|
||||
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
|
||||
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
|
||||
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, parse
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
router = APIRouter(prefix="/config", tags=["config"])
|
||||
|
||||
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
|
||||
# and read before parsing so a huge body never reaches the JSON parser.
|
||||
MAX_REQUEST_BYTES = 4 * MAX_BYTES
|
||||
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
|
||||
# open to read-only accounts and read tokens.
|
||||
_busy = asyncio.Semaphore(1)
|
||||
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
|
||||
MAX_WAITING = 4
|
||||
_waiting = 0
|
||||
|
||||
|
||||
class _Slot:
|
||||
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
|
||||
|
||||
async def __aenter__(self):
|
||||
global _waiting
|
||||
if _waiting >= MAX_WAITING:
|
||||
raise api_error(429, ErrorCode.CONFIG_BUSY)
|
||||
_waiting += 1
|
||||
return self
|
||||
|
||||
async def run(self, coro_fn):
|
||||
async with _busy:
|
||||
return await coro_fn()
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
global _waiting
|
||||
_waiting -= 1
|
||||
|
||||
|
||||
def _encode(text: str) -> bytes:
|
||||
try:
|
||||
return text.encode("utf-8")
|
||||
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
class ImportRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
yaml: str
|
||||
prune: bool = False
|
||||
|
||||
|
||||
class ApplyRequest(ImportRequest):
|
||||
plan_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
def _config_error(exc: ConfigError):
|
||||
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
|
||||
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
|
||||
|
||||
|
||||
async def _read_body(request: Request, model: type[BaseModel]):
|
||||
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks, size = [], 0
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks.append(chunk)
|
||||
try:
|
||||
return model.model_validate(json.loads(b"".join(chunks)))
|
||||
except (ValueError, ValidationError):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
@router.get("/export")
|
||||
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
|
||||
wanted = None
|
||||
if kinds:
|
||||
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
|
||||
text = await export_yaml(db, wanted)
|
||||
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
|
||||
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
|
||||
|
||||
|
||||
@router.post("/import/plan")
|
||||
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ImportRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
return await planner.build_plan(db, parsed, body.prune)
|
||||
except ConfigError as exc:
|
||||
raise _config_error(exc) from None
|
||||
finally:
|
||||
await db.rollback() # never leave anything in the session: plan is read-only
|
||||
|
||||
result = await slot.run(work)
|
||||
return result.public()
|
||||
|
||||
|
||||
@router.post("/import/apply")
|
||||
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ApplyRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
|
||||
except ConfigError as exc:
|
||||
await db.rollback()
|
||||
raise _config_error(exc) from None
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
return parsed, result
|
||||
|
||||
parsed, result = await slot.run(work)
|
||||
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
|
||||
@@ -18,6 +18,8 @@ locks are deliberately NOT included here (no fleet-wide list endpoint
|
||||
exists for those, only per-host, and it's a rarely-triggered flow) —
|
||||
a documented v1 gap, not an oversight.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -46,7 +48,10 @@ from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.wireguard_tunnel import WireguardTunnel
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service, notification_settings_service
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.monitor_checks import STALE_AFTER
|
||||
|
||||
router = APIRouter(prefix="/dashboard", tags=["dashboard"])
|
||||
|
||||
@@ -202,14 +207,43 @@ async def collect_pending_actions(db: AsyncSession) -> list[PendingAction]:
|
||||
return pending_actions
|
||||
|
||||
|
||||
async def _sample_readings_by_host(db: AsyncSession, hosts, muted: set[str]) -> tuple[dict, dict]:
|
||||
"""(gateways, missing API rights) from each host's newest resource sample; hosts without a fresh sample are left out."""
|
||||
gateways: dict = {}
|
||||
denied: dict = {}
|
||||
now = datetime.now(timezone.utc)
|
||||
for host in hosts:
|
||||
row = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json(), HostResourceSample.raw["denied"].as_json())
|
||||
.where(HostResourceSample.host_id == host.id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).first()
|
||||
if row is None or now - row[0] > STALE_AFTER:
|
||||
continue
|
||||
if row[1] is not None:
|
||||
gateways[host.id] = [
|
||||
g.model_copy(update={"alert_enabled": f"{host.id}:{g.name}" not in muted})
|
||||
for g in gateway_health_service.from_compact(row[1])
|
||||
]
|
||||
if isinstance(row[2], list):
|
||||
denied[host.id] = [str(x) for x in row[2]]
|
||||
return gateways, denied
|
||||
|
||||
|
||||
@router.get("", response_model=DashboardSummary)
|
||||
async def summary(db: AsyncSession = Depends(get_db)):
|
||||
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
||||
statuses = await get_statuses(hosts)
|
||||
muted = set((await notification_settings_service.get_settings(db)).gateway_alert_muted or [])
|
||||
gateways, denied = await _sample_readings_by_host(db, hosts, muted)
|
||||
host_summaries = [
|
||||
DashboardHost(
|
||||
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
|
||||
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
|
||||
gateways=gateways.get(host.id), api_rights_missing=denied.get(host.id),
|
||||
)
|
||||
for host in hosts
|
||||
]
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.schemas.gateway import ApiRightsState, GatewayAlertState, GatewayAlertUpdate, GatewayHealth
|
||||
from app.services import audit_service, gateway_health_service, notification_settings_service
|
||||
from app.services.monitor_checks import STALE_AFTER
|
||||
from app.services.opnsense_client import OpnsenseAPIError, OpnsenseForbiddenError
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
|
||||
|
||||
# OPNsense gateway names are letters, digits and underscores (WAN_GW, WAN_DHCP6); a little more is allowed.
|
||||
_GATEWAY_NAME = re.compile(r"[A-Za-z0-9_.-]{1,64}")
|
||||
MAX_MUTED_GATEWAYS = 200
|
||||
|
||||
|
||||
@router.get("", response_model=list[GatewayHealth])
|
||||
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
"""Live gateway state (loss, delay, deviation, OPNsense' verdict) of one firewall."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
gateways = await gateway_health_service.get_gateway_health(host)
|
||||
except OpnsenseForbiddenError as exc:
|
||||
# The API key may not read the gateway state: say so instead of a generic upstream failure.
|
||||
raise api_error(403, ErrorCode.API_RIGHTS_MISSING) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
muted = set((await notification_settings_service.get_settings(db)).gateway_alert_muted or [])
|
||||
return [g.model_copy(update={"alert_enabled": f"{host.id}:{g.name}" not in muted}) for g in gateways]
|
||||
|
||||
|
||||
@router.get("/api-rights", response_model=ApiRightsState)
|
||||
async def api_rights(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
"""Side readings the newest FRESH resource sample could not take because the API key lacks the right
|
||||
(null = no fresh sample, so nothing is known). Stored data only: no call to the firewall."""
|
||||
if await db.get(OpnsenseHost, host_id) is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
row = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["denied"].as_json())
|
||||
.where(HostResourceSample.host_id == host_id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).first()
|
||||
if row is None or datetime.now(timezone.utc) - row[0] > STALE_AFTER or not isinstance(row[1], list):
|
||||
return ApiRightsState(missing=None)
|
||||
return ApiRightsState(missing=[str(x) for x in row[1]])
|
||||
|
||||
|
||||
@router.put("/{name}/alert", response_model=GatewayAlertState)
|
||||
async def set_gateway_alert(host_id: uuid.UUID, name: str, payload: GatewayAlertUpdate, db: AsyncSession = Depends(get_db)):
|
||||
"""Switch the gateway-down alert on or off for ONE gateway (admins only, see api_token_service)."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
if not _GATEWAY_NAME.fullmatch(name):
|
||||
raise api_error(422, ErrorCode.GATEWAY_NAME_INVALID)
|
||||
if not payload.enabled:
|
||||
# Only a gateway the firewall really reported can be muted ("wan_gw" vs "WAN_GW" would silently do nothing).
|
||||
# Without a usable sample nothing is known, so the request goes through.
|
||||
known = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.raw["gateways"].as_json())
|
||||
.where(HostResourceSample.host_id == host_id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).scalar()
|
||||
if isinstance(known, list) and name not in {g.get("name") for g in known if isinstance(g, dict)}:
|
||||
raise api_error(404, ErrorCode.GATEWAY_UNKNOWN)
|
||||
row = await notification_settings_service.get_settings(db)
|
||||
await db.refresh(row, with_for_update=True)
|
||||
muted = list(row.gateway_alert_muted or [])
|
||||
key = f"{host.id}:{name}"
|
||||
if payload.enabled:
|
||||
muted = [m for m in muted if m != key]
|
||||
elif key not in muted:
|
||||
if len(muted) >= MAX_MUTED_GATEWAYS:
|
||||
raise api_error(422, ErrorCode.GATEWAY_MUTE_LIMIT, limit=str(MAX_MUTED_GATEWAYS))
|
||||
muted.append(key)
|
||||
row.gateway_alert_muted = muted # a new list, so the JSON column notices the change
|
||||
await db.commit()
|
||||
# The request middleware logs the call itself; this entry says what it did (only the validated name goes in).
|
||||
await audit_service.persist(
|
||||
kind="host", action="gateway_alert_on" if payload.enabled else "gateway_alert_off", entity_id=str(host.id),
|
||||
host_id=str(host.id), host_name=host.name, status="accepted", message=f"Gateway {name}",
|
||||
)
|
||||
return GatewayAlertState(name=name, alert_enabled=payload.enabled)
|
||||
@@ -1,19 +1,102 @@
|
||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it)."""
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it).
|
||||
|
||||
The rule-cleanup settings (1.31) live here too: how many rules one cleanup may touch, how long a rule must stay
|
||||
switched off before it may be deleted, the management sources (addresses the firewalls are managed from; no rule
|
||||
that could match them is ever cleaned up) and extra management ports (besides the API port, 443, 80 and 22). Only an admin writes any of it (middleware: ``/api/settings/...`` is
|
||||
admin-only for operators and tokens below scope admin; checked again here) and every change is audited with the
|
||||
old and the new values."""
|
||||
import ipaddress
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.governance import GovernanceSettings
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.routers.auth import client_ip, current_user
|
||||
from app.services import audit_service
|
||||
|
||||
router = APIRouter(prefix="/settings/governance", tags=["governance"])
|
||||
|
||||
MAX_CLEANUP_RULES = 25 # same ceiling as services/rule_cleanup_service.MAX_RULES_CEILING
|
||||
MAX_DELETE_MIN_DAYS = 365
|
||||
MAX_MANAGEMENT_SOURCES = 20
|
||||
MAX_MANAGEMENT_PORTS = 20
|
||||
|
||||
|
||||
class GovernanceRead(BaseModel):
|
||||
operators_may_deploy: bool
|
||||
cleanup_max_rules: int = 10
|
||||
cleanup_delete_min_days: int = 7
|
||||
# The list itself is shown to admins only; everybody else sees just whether it is set.
|
||||
cleanup_management_sources: list[str] = Field(default_factory=list)
|
||||
cleanup_management_sources_configured: bool = False
|
||||
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
|
||||
cleanup_management_ports: list[int] = Field(default_factory=list)
|
||||
cleanup_management_ports_configured: bool = False
|
||||
# The address this request came from, as a suggestion for the management sources (B12; admins only). Only a
|
||||
# suggestion: the firewalls may see the workstation under another address (NAT, VPN).
|
||||
your_address: str | None = None
|
||||
|
||||
|
||||
class GovernanceUpdate(BaseModel):
|
||||
operators_may_deploy: bool | None = None
|
||||
cleanup_max_rules: int | None = None
|
||||
cleanup_delete_min_days: int | None = None
|
||||
cleanup_management_sources: list[str] | None = None
|
||||
cleanup_management_ports: list[Any] | None = None # validated in clean_management_ports (no bool coercion)
|
||||
|
||||
|
||||
def _bad(field: str, reason: str):
|
||||
return api_error(422, ErrorCode.BEREINIGUNG_EINSTELLUNG_UNGUELTIG, field=field, reason=reason)
|
||||
|
||||
|
||||
def clean_management_sources(values: list[str]) -> list[str]:
|
||||
"""IP addresses and CIDR networks only (no host names: they could resolve differently later), canonical form,
|
||||
without duplicates; not the whole address space, which would protect every rule."""
|
||||
if len(values) > MAX_MANAGEMENT_SOURCES:
|
||||
raise _bad("cleanup_management_sources", "too_many")
|
||||
cleaned: list[str] = []
|
||||
for raw in values:
|
||||
text = str(raw).strip()
|
||||
try:
|
||||
net = ipaddress.ip_network(text, strict=False)
|
||||
except ValueError:
|
||||
raise _bad("cleanup_management_sources", "not_an_address") from None
|
||||
if net.prefixlen == 0:
|
||||
raise _bad("cleanup_management_sources", "everything")
|
||||
entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)
|
||||
if entry not in cleaned:
|
||||
cleaned.append(entry)
|
||||
return cleaned
|
||||
|
||||
|
||||
def clean_management_ports(values) -> list[int]:
|
||||
"""TCP ports 1..65535, at most MAX_MANAGEMENT_PORTS, without duplicates, sorted."""
|
||||
if len(values) > MAX_MANAGEMENT_PORTS:
|
||||
raise _bad("cleanup_management_ports", "too_many")
|
||||
ports: set[int] = set()
|
||||
for raw in values:
|
||||
text = str(raw).strip()
|
||||
if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:
|
||||
raise _bad("cleanup_management_ports", "not_a_port")
|
||||
ports.add(int(text))
|
||||
return sorted(ports)
|
||||
|
||||
|
||||
def _read(row: GovernanceSettings, user: AdminUser) -> GovernanceRead:
|
||||
sources = list(row.cleanup_management_sources or [])
|
||||
ports = list(row.cleanup_management_ports or [])
|
||||
return GovernanceRead(
|
||||
operators_may_deploy=row.operators_may_deploy, cleanup_max_rules=row.cleanup_max_rules or 10,
|
||||
cleanup_delete_min_days=row.cleanup_delete_min_days or 7,
|
||||
cleanup_management_sources=sources if user.role == "admin" else [],
|
||||
cleanup_management_sources_configured=bool(sources),
|
||||
cleanup_management_ports=ports if user.role == "admin" else [],
|
||||
cleanup_management_ports_configured=bool(ports))
|
||||
|
||||
|
||||
async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
@@ -25,17 +108,45 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
return row
|
||||
|
||||
|
||||
def _suggestion(request: Request, user: AdminUser) -> str | None:
|
||||
if user.role != "admin":
|
||||
return None
|
||||
try:
|
||||
return str(ipaddress.ip_address(client_ip(request)))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
@router.get("", response_model=GovernanceRead)
|
||||
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
async def read(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
|
||||
return await get_settings(db)
|
||||
return _read(await get_settings(db), user).model_copy(update={"your_address": _suggestion(request, user)})
|
||||
|
||||
|
||||
@router.put("", response_model=GovernanceRead)
|
||||
async def update(payload: GovernanceRead, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
async def update(payload: GovernanceUpdate, request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
row = await get_settings(db)
|
||||
row.operators_may_deploy = payload.operators_may_deploy
|
||||
changes: dict[str, dict] = {}
|
||||
if payload.cleanup_max_rules is not None:
|
||||
if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:
|
||||
raise _bad("cleanup_max_rules", f"1..{MAX_CLEANUP_RULES}")
|
||||
if payload.cleanup_delete_min_days is not None:
|
||||
if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:
|
||||
raise _bad("cleanup_delete_min_days", f"1..{MAX_DELETE_MIN_DAYS}")
|
||||
sources = clean_management_sources(payload.cleanup_management_sources) if payload.cleanup_management_sources is not None else None
|
||||
ports = clean_management_ports(payload.cleanup_management_ports) if payload.cleanup_management_ports is not None else None
|
||||
for field, value in (("cleanup_max_rules", payload.cleanup_max_rules), ("cleanup_delete_min_days", payload.cleanup_delete_min_days),
|
||||
("cleanup_management_sources", sources), ("cleanup_management_ports", ports)):
|
||||
if value is not None and value != getattr(row, field):
|
||||
changes[field] = {"old": getattr(row, field), "new": value}
|
||||
setattr(row, field, value)
|
||||
if payload.operators_may_deploy is not None:
|
||||
row.operators_may_deploy = payload.operators_may_deploy
|
||||
if changes:
|
||||
db.add(audit_service.make_event(
|
||||
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
|
||||
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
|
||||
await db.commit()
|
||||
return row
|
||||
return _read(row, user).model_copy(update={"your_address": _suggestion(request, user)})
|
||||
@@ -99,6 +99,15 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
|
||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
|
||||
# VPN objects target groups too; their link rows are ON DELETE CASCADE, so without this check the group would
|
||||
# vanish and the VPN would silently lose its targets (B15, found while designing the config import).
|
||||
from app.models.policy import ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups
|
||||
for assignment in (ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups):
|
||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET)
|
||||
from app.models.rollout import rollout_wave_groups
|
||||
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
|
||||
await db.delete(group)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
"""Firewall GUI access through the gui-gateway container (1.27).
|
||||
|
||||
Public API (session cookie, admin only, never API tokens):
|
||||
* POST /api/hosts/{id}/gui-ticket — one-time ticket + the slot URL the browser opens (noopener).
|
||||
* GET/PUT /api/settings/gui-access — switch, idle/absolute timeout, number of slots.
|
||||
|
||||
Internal API for the gateway only (NOT under /api, so the proxy never forwards it; shared secret
|
||||
GUI_GATEWAY_TOKEN in the X-Onx-Gateway-Token header, compared in constant time):
|
||||
* POST /internal/gui/redeem, /internal/gui/revalidate, /internal/gui/end.
|
||||
The answers never contain firewall credentials: only the firewall origin, its pinned (public) CA
|
||||
certificate, the overlay proxy address and the timeouts.
|
||||
"""
|
||||
import hashlib
|
||||
import hmac
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Request, status
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import SECURE_COOKIE_NAME, current_user
|
||||
from app.services import api_token_service, gui_ticket_service as svc
|
||||
|
||||
router = APIRouter(tags=["gui-access"])
|
||||
internal_router = APIRouter(prefix="/internal/gui", include_in_schema=False)
|
||||
|
||||
|
||||
def _require_admin_session(request: Request, user: AdminUser) -> None:
|
||||
if getattr(request.state, "api_token", None) is not None or api_token_service.bearer_from_header(
|
||||
request.headers.get("authorization", "")
|
||||
) is not None:
|
||||
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.GUI_SESSION_REQUIRED)
|
||||
if user.role != "admin":
|
||||
raise api_error(status.HTTP_403_FORBIDDEN, ErrorCode.ADMIN_REQUIRED)
|
||||
|
||||
|
||||
class GuiTicketRead(BaseModel):
|
||||
url: str
|
||||
slot: int
|
||||
expires_in: int
|
||||
|
||||
|
||||
@router.post("/hosts/{host_id}/gui-ticket", response_model=GuiTicketRead)
|
||||
async def create_ticket(host_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db),
|
||||
user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
if not svc.gateway_configured():
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED)
|
||||
session_cookie = request.cookies.get(SECURE_COOKIE_NAME, "")
|
||||
if not settings.cookie_secure or not session_cookie:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_REQUIRES_HTTPS)
|
||||
row = await svc.get_settings(db)
|
||||
if not row.enabled:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_ACCESS_DISABLED)
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(status.HTTP_404_NOT_FOUND, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
refusal = svc.host_refusal(host)
|
||||
if refusal:
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_HOST_NOT_ELIGIBLE, reason=refusal)
|
||||
try:
|
||||
return await svc.issue_ticket(db, user=user, session_token=session_cookie, host=host,
|
||||
request_host=request.headers.get("host", ""))
|
||||
except svc.GuiAccessError as exc:
|
||||
raise api_error(exc.status, exc.code, **exc.params)
|
||||
|
||||
|
||||
class GuiAccessSettingsRead(BaseModel):
|
||||
enabled: bool
|
||||
idle_timeout_minutes: int
|
||||
absolute_timeout_minutes: int
|
||||
slot_count: int
|
||||
available: bool
|
||||
unavailable_reasons: list[str]
|
||||
slots: list[int]
|
||||
updated_by: str | None
|
||||
updated_at: datetime | None
|
||||
|
||||
|
||||
class GuiAccessSettingsUpdate(BaseModel):
|
||||
enabled: bool
|
||||
idle_timeout_minutes: int = Field(ge=1, le=240)
|
||||
absolute_timeout_minutes: int = Field(ge=5, le=720)
|
||||
slot_count: int = Field(ge=1, le=svc.MAX_SLOTS)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _idle_within_absolute(self):
|
||||
if self.idle_timeout_minutes > self.absolute_timeout_minutes:
|
||||
raise ValueError("idle_timeout_minutes must not exceed absolute_timeout_minutes")
|
||||
return self
|
||||
|
||||
|
||||
def _settings_read(row) -> dict:
|
||||
reasons = svc.unavailable_reasons()
|
||||
return {"enabled": row.enabled, "idle_timeout_minutes": row.idle_timeout_minutes,
|
||||
"absolute_timeout_minutes": row.absolute_timeout_minutes, "slot_count": row.slot_count,
|
||||
"available": not reasons, "unavailable_reasons": reasons, "slots": svc.active_slots(row),
|
||||
"updated_by": row.updated_by, "updated_at": row.updated_at}
|
||||
|
||||
|
||||
@router.get("/settings/gui-access", response_model=GuiAccessSettingsRead)
|
||||
async def read_settings(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
row = await svc.get_settings(db)
|
||||
await db.commit()
|
||||
return _settings_read(row)
|
||||
|
||||
|
||||
@router.put("/settings/gui-access", response_model=GuiAccessSettingsRead)
|
||||
async def update_settings(payload: GuiAccessSettingsUpdate, request: Request, db: AsyncSession = Depends(get_db),
|
||||
user: AdminUser = Depends(current_user)):
|
||||
_require_admin_session(request, user)
|
||||
if payload.enabled and svc.unavailable_reasons():
|
||||
# Never switchable on without HTTPS cookies and a configured gateway.
|
||||
raise api_error(status.HTTP_409_CONFLICT, ErrorCode.GUI_GATEWAY_NOT_CONFIGURED,
|
||||
reasons=", ".join(svc.unavailable_reasons()))
|
||||
row = await svc.get_settings(db)
|
||||
row.enabled = payload.enabled
|
||||
row.idle_timeout_minutes = payload.idle_timeout_minutes
|
||||
row.absolute_timeout_minutes = payload.absolute_timeout_minutes
|
||||
row.slot_count = payload.slot_count
|
||||
row.updated_by = user.username
|
||||
row.updated_at = datetime.now(timezone.utc)
|
||||
if not payload.enabled:
|
||||
await svc.end_all(db, "disabled")
|
||||
await db.commit()
|
||||
return _settings_read(row)
|
||||
|
||||
|
||||
# ── internal: gateway only ────────────────────────────────────────────────
|
||||
|
||||
def _gateway_auth(x_onx_gateway_token: str = Header(default="")) -> None:
|
||||
expected = settings.gui_gateway_token
|
||||
if len(expected) < svc.MIN_GATEWAY_TOKEN_LENGTH:
|
||||
raise HTTPException(status_code=404)
|
||||
# Fixed-length digests: the comparison time does not depend on the presented value.
|
||||
if not hmac.compare_digest(hashlib.sha256(x_onx_gateway_token.encode()).digest(),
|
||||
hashlib.sha256(expected.encode()).digest()):
|
||||
raise HTTPException(status_code=401)
|
||||
|
||||
|
||||
_HEX64 = r"^[0-9a-f]{64}$"
|
||||
|
||||
|
||||
class RedeemRequest(BaseModel):
|
||||
ticket_hash: str = Field(pattern=_HEX64)
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
slot: int = Field(ge=1, le=65535)
|
||||
client_ip: str | None = Field(default=None, max_length=45)
|
||||
|
||||
|
||||
class RevalidateRequest(BaseModel):
|
||||
gui_session_id: uuid.UUID
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
# Fingerprint of the target as handed out at redeem time (optional: an older gateway image omits it).
|
||||
target_fp: str | None = Field(default=None, pattern=_HEX64)
|
||||
|
||||
|
||||
class EndRequest(BaseModel):
|
||||
gui_session_id: uuid.UUID
|
||||
session_hash: str = Field(pattern=_HEX64)
|
||||
reason: Literal["idle", "absolute", "logout", "revoked", "superseded", "shutdown"]
|
||||
|
||||
|
||||
@internal_router.post("/redeem", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_redeem(payload: RedeemRequest, db: AsyncSession = Depends(get_db)):
|
||||
result = await svc.redeem(db, ticket_hash=payload.ticket_hash, session_hash=payload.session_hash,
|
||||
slot=payload.slot, client_ip=payload.client_ip)
|
||||
if not result["ok"]:
|
||||
raise HTTPException(status_code=403, detail={"ok": False, "reason": result["reason"]})
|
||||
return result
|
||||
|
||||
|
||||
@internal_router.post("/revalidate", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_revalidate(payload: RevalidateRequest, db: AsyncSession = Depends(get_db)):
|
||||
return await svc.revalidate(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
|
||||
target_fp=payload.target_fp)
|
||||
|
||||
|
||||
@internal_router.post("/end", dependencies=[Depends(_gateway_auth)])
|
||||
async def internal_end(payload: EndRequest, db: AsyncSession = Depends(get_db)):
|
||||
return {"ok": await svc.end(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
|
||||
reason=payload.reason)}
|
||||
@@ -5,7 +5,7 @@ from app.models.port_forward_deployment import PortForwardDeployment
|
||||
from app.models.one_to_one_deployment import OneToOneDeployment
|
||||
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -107,11 +107,37 @@ async def tls_probe(payload: TlsProbeRequest):
|
||||
raise api_error(502, ErrorCode.TLS_PROBE_FAILED)
|
||||
|
||||
|
||||
def _same_url(a: str, b: str) -> bool:
|
||||
return a.strip().rstrip("/").lower() == b.strip().rstrip("/").lower()
|
||||
|
||||
|
||||
async def _ensure_url_free(db: AsyncSession, base_url: str, exclude_id: uuid.UUID | None = None) -> None:
|
||||
"""One firewall, one entry: the same address twice would double every rollout, alert and backup (T4)."""
|
||||
for other in (await db.scalars(select(OpnsenseHost))).all():
|
||||
if other.id != exclude_id and _same_url(other.base_url, base_url):
|
||||
raise api_error(409, ErrorCode.HOST_URL_EXISTS, host=other.name)
|
||||
|
||||
|
||||
def _connection_failure_reason(exc: Exception) -> str:
|
||||
"""A short machine code for the UI (hosts.connection.<code>): why the first call to the firewall failed."""
|
||||
text = str(exc)
|
||||
if isinstance(exc, opnsense_client.OpnsenseForbiddenError):
|
||||
return "forbidden"
|
||||
if text.startswith("401 ") or text.startswith("403 "):
|
||||
return "credentials" if text.startswith("401 ") else "forbidden"
|
||||
if "TLS-Zertifikat" in text or "CERTIFICATE_VERIFY_FAILED" in text:
|
||||
return "tls"
|
||||
if "Verbindung zu" in text:
|
||||
return "unreachable"
|
||||
return "other"
|
||||
|
||||
|
||||
@router.post("", response_model=HostRead, status_code=201)
|
||||
async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
||||
existing = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == payload.name))
|
||||
if existing:
|
||||
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
||||
await _ensure_url_free(db, payload.base_url)
|
||||
if payload.policy_hierarchy_group_id and not await db.get(PolicyHierarchyGroup, payload.policy_hierarchy_group_id):
|
||||
raise api_error(400, ErrorCode.POLICY_HIERARCHIE_GRUPPE_NICHT_GEFUNDEN)
|
||||
|
||||
@@ -130,6 +156,14 @@ async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
||||
db.add(host)
|
||||
await db.flush()
|
||||
opnsense_secret_store.save_secret(host.id, payload.api_secret)
|
||||
if payload.verify_connection:
|
||||
# The call that decides "online" in the host list: if it works here, the host shows up online right away.
|
||||
try:
|
||||
await opnsense_client.get_vip_status(host)
|
||||
except opnsense_client.OpnsenseAPIError as exc:
|
||||
await db.rollback()
|
||||
opnsense_secret_store.delete_secret(host.id)
|
||||
raise api_error(422, ErrorCode.HOST_CONNECTION_FAILED, reason=_connection_failure_reason(exc)) from exc
|
||||
await db.commit()
|
||||
|
||||
statuses = await get_statuses([host])
|
||||
@@ -147,6 +181,11 @@ async def get_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _get_host_or_404(db, host_id)
|
||||
if (payload.base_url is not None or payload.enabled is not None or payload.clear_cluster_name
|
||||
or payload.cluster_name is not None):
|
||||
# An open rule cleanup (1.31) reverts through this address and this cluster pairing: not while it is open.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
|
||||
if payload.name is not None:
|
||||
existing = await db.scalar(
|
||||
@@ -156,6 +195,7 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
|
||||
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
||||
host.name = payload.name
|
||||
if payload.base_url is not None:
|
||||
await _ensure_url_free(db, payload.base_url, exclude_id=host_id)
|
||||
host.base_url = payload.base_url
|
||||
if payload.verify_tls is not None:
|
||||
host.verify_tls = payload.verify_tls
|
||||
@@ -183,11 +223,18 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
|
||||
|
||||
|
||||
@router.delete("/{host_id}", status_code=204)
|
||||
async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def delete_host(host_id: uuid.UUID, confirm_name: str = Query(default="", max_length=120), db: AsyncSession = Depends(get_db)):
|
||||
"""Deleting a host drops its stored credentials at once, so the caller must name it (like every other destructive
|
||||
action in OPNexus), not only a click in the UI: `DELETE /api/hosts/{id}?confirm_name=<name>` (T4)."""
|
||||
await policy_lock(db)
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
if confirm_name != host.name:
|
||||
raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
# Both deployment tables FK to opnsense_hosts with ondelete=CASCADE, so
|
||||
# deleting the host would silently drop OPNexus's only record of what
|
||||
# it pushed there — the rule/alias itself stays live on the actual
|
||||
@@ -226,6 +273,12 @@ async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
draft_certs = (await db.scalars(select(ManagedCertificate.id).where(ManagedCertificate.host_id == host_id))).all()
|
||||
draft_accounts = (await db.scalars(select(AcmeAccount.id).where(AcmeAccount.host_id == host_id))).all()
|
||||
await db.delete(host)
|
||||
# Gateway alert switches of this host would otherwise stay in the settings forever (and count against the limit).
|
||||
from app.services import notification_settings_service
|
||||
settings_row = await notification_settings_service.get_settings(db)
|
||||
prefix = f"{host_id}:"
|
||||
if any(str(m).startswith(prefix) for m in settings_row.gateway_alert_muted or []):
|
||||
settings_row.gateway_alert_muted = [m for m in settings_row.gateway_alert_muted if not str(m).startswith(prefix)]
|
||||
await db.commit()
|
||||
opnsense_secret_store.delete_secret(host_id)
|
||||
for cert_id in draft_certs:
|
||||
|
||||
@@ -16,6 +16,7 @@ from app.services import ipsec_psk_store
|
||||
from app.services.ipsec_service import effective_hosts, preview, revision, snapshot, workflow, IPSEC_FIELDS
|
||||
from app.services.ipsec_deploy_service import deploy_tunnel, undeploy_tunnel
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_lease import resource_lease
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/ipsec/tunnels", tags=["ipsec"])
|
||||
@@ -192,24 +193,27 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_child_uuid:
|
||||
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
|
||||
if deployment.opnsense_local_uuid:
|
||||
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
|
||||
if deployment.opnsense_remote_uuid:
|
||||
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
|
||||
if deployment.opnsense_connection_uuid:
|
||||
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
|
||||
if deployment.opnsense_psk_uuid:
|
||||
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
|
||||
await opnsense_client.reconfigure_ipsec(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
async with resource_lease("ipsec", tunnel.id):
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_child_uuid:
|
||||
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
|
||||
if deployment.opnsense_local_uuid:
|
||||
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
|
||||
if deployment.opnsense_remote_uuid:
|
||||
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
|
||||
if deployment.opnsense_connection_uuid:
|
||||
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
|
||||
if deployment.opnsense_psk_uuid:
|
||||
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
|
||||
await opnsense_client.reconfigure_ipsec(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
|
||||
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
|
||||
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
|
||||
|
||||
class Preview(BaseModel):
|
||||
action: Literal['reboot','update','major_update']
|
||||
action: Literal['reboot','update','major_update','firmware_rollback']
|
||||
|
||||
class Execute(BaseModel):
|
||||
preview_id: uuid.UUID
|
||||
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
|
||||
def read_operation(op):
|
||||
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
|
||||
|
||||
async def snapshot_or_502(db, host):
|
||||
try: return await inspect_host(db,host)
|
||||
async def snapshot_or_502(db, host, action=None):
|
||||
try:
|
||||
state=await inspect_host(db,host)
|
||||
if action=='firmware_rollback':
|
||||
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
|
||||
rollback=await snapshot_service.rollback_state(host)
|
||||
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
|
||||
state['rollback']=rollback
|
||||
state['actions']['firmware_rollback']=state['actions']['reboot']+(
|
||||
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
|
||||
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
|
||||
return state
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
|
||||
|
||||
async def _changes_since(db, host, target):
|
||||
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
|
||||
if not target:
|
||||
return []
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return []
|
||||
out=[]
|
||||
for kind,model in _DEPLOYMENT_MODELS():
|
||||
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
|
||||
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
|
||||
return sorted(out,key=lambda r:r['deployed_at'])[:50]
|
||||
|
||||
|
||||
def _point_time(target):
|
||||
try:
|
||||
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _DEPLOYMENT_MODELS():
|
||||
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
|
||||
PortForwardDeployment, RuleDeployment)
|
||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
|
||||
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
|
||||
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
|
||||
|
||||
|
||||
async def _mark_rolled_back(db, host, target):
|
||||
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
|
||||
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
|
||||
really writes, and the UI shows the object as not in sync."""
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return 0
|
||||
count=0
|
||||
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
|
||||
for _kind,model in _DEPLOYMENT_MODELS():
|
||||
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
|
||||
row.status='error'; row.message=note[:500]; row.last_payload=None
|
||||
if hasattr(model,'revert_deadline'): row.revert_deadline=None
|
||||
count+=1
|
||||
return count
|
||||
|
||||
async def carp_snapshot_or_502(db, host):
|
||||
try: return await inspect_carp_switch(db,host)
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host=await host_or_404(db,host_id)
|
||||
state=await snapshot_or_502(db,host)
|
||||
state=await snapshot_or_502(db,host,payload.action)
|
||||
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
|
||||
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
|
||||
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
|
||||
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
|
||||
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
|
||||
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||
state=await snapshot_or_502(db,host)
|
||||
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
state=await snapshot_or_502(db,host,op.action)
|
||||
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
|
||||
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
|
||||
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
|
||||
# replaying a reboot/update automatically could interrupt recovery.
|
||||
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
if op.action in ('update','major_update'):
|
||||
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
|
||||
try:
|
||||
point=await snapshot_service.create_rollback_point(host,state.get('version'))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
op.snapshot={**op.snapshot,'rollback_point':point}
|
||||
await db.commit()
|
||||
if op.action=='firmware_rollback':
|
||||
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
|
||||
# Fail closed, like the backup before an update.
|
||||
try:
|
||||
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
rollback=state['rollback']
|
||||
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
|
||||
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
|
||||
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
|
||||
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
|
||||
else:
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
|
||||
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
|
||||
try:
|
||||
result=await action_fn(host)
|
||||
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
op.status='submitted';op.message=done
|
||||
if op.action=='firmware_rollback':
|
||||
marked=await _mark_rolled_back(db,host,rollback['target'])
|
||||
if marked:
|
||||
op.message+=f' {marked} Deployment(s) als unklar markiert.'
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
|
||||
result={}
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Recurring maintenance windows and their exceptions (1.25).
|
||||
|
||||
Every write here counts as a rollout action (api_token_service.is_rollout): a window decides WHEN
|
||||
unattended firmware actions may run, so an operator needs the same right as for deploying.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
from datetime import date, datetime, time, timedelta, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
from app.services import audit_service, window_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/maintenance-windows", tags=["maintenance-windows"])
|
||||
|
||||
_TIME = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
|
||||
|
||||
|
||||
class WindowIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
timezone: str = Field(min_length=1, max_length=64)
|
||||
recurrence: Literal["weekly", "monthly"]
|
||||
weekdays: list[int] = Field(default_factory=list, max_length=7)
|
||||
month_mode: Literal["nth_weekday", "day"] | None = None
|
||||
month_nth: int | None = None
|
||||
month_weekday: int | None = None
|
||||
month_day: int | None = None
|
||||
start_time: str
|
||||
duration_minutes: int = Field(ge=15, le=window_service.MAX_DURATION_MINUTES)
|
||||
start_cutoff_minutes: int = Field(default=60, ge=5, le=window_service.MAX_DURATION_MINUTES)
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
class ExceptionIn(BaseModel):
|
||||
occurrence_date: date
|
||||
kind: Literal["skip", "move"]
|
||||
moved_start: datetime | None = None
|
||||
reason: str = Field(default="", max_length=300)
|
||||
|
||||
|
||||
def _invalid(field: str):
|
||||
return api_error(422, ErrorCode.WINDOW_INVALID, field=field)
|
||||
|
||||
|
||||
def _validated(payload: WindowIn) -> dict:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
raise _invalid("name")
|
||||
if not window_service.valid_timezone(payload.timezone):
|
||||
raise _invalid("timezone")
|
||||
match = _TIME.match(payload.start_time)
|
||||
if not match:
|
||||
raise _invalid("start_time")
|
||||
values = {
|
||||
"name": name, "timezone": payload.timezone, "recurrence": payload.recurrence,
|
||||
"weekdays": [], "month_mode": None, "month_nth": None, "month_weekday": None, "month_day": None,
|
||||
"start_time": time(int(match.group(1)), int(match.group(2))),
|
||||
"duration_minutes": payload.duration_minutes,
|
||||
"start_cutoff_minutes": min(payload.start_cutoff_minutes, payload.duration_minutes),
|
||||
"enabled": payload.enabled,
|
||||
}
|
||||
if payload.recurrence == "weekly":
|
||||
days = sorted(set(payload.weekdays))
|
||||
if not days or any(d < 0 or d > 6 for d in days):
|
||||
raise _invalid("weekdays")
|
||||
values["weekdays"] = days
|
||||
elif payload.month_mode == "day":
|
||||
if payload.month_day is None or not 1 <= payload.month_day <= 28:
|
||||
raise _invalid("month_day")
|
||||
values.update(month_mode="day", month_day=payload.month_day)
|
||||
elif payload.month_mode == "nth_weekday":
|
||||
if payload.month_nth not in (1, 2, 3, 4, -1):
|
||||
raise _invalid("month_nth")
|
||||
if payload.month_weekday is None or not 0 <= payload.month_weekday <= 6:
|
||||
raise _invalid("month_weekday")
|
||||
values.update(month_mode="nth_weekday", month_nth=payload.month_nth, month_weekday=payload.month_weekday)
|
||||
else:
|
||||
raise _invalid("month_mode")
|
||||
return values
|
||||
|
||||
|
||||
async def _plans_using(db: AsyncSession, window_id) -> list[dict]:
|
||||
from app.models.rollout import RolloutPlan
|
||||
rows = (await db.scalars(select(RolloutPlan).where(RolloutPlan.window_id == window_id).order_by(RolloutPlan.name))).all()
|
||||
return [{"id": str(p.id), "name": p.name, "auto_start": p.auto_start, "active": p.activated_at is not None} for p in rows]
|
||||
|
||||
|
||||
def read_window(window: MaintenanceWindow, *, now: datetime | None = None, count: int = 5) -> dict:
|
||||
now = now or datetime.now(timezone.utc)
|
||||
return {
|
||||
"id": str(window.id), "name": window.name, "timezone": window.timezone, "recurrence": window.recurrence,
|
||||
"weekdays": window.weekdays or [], "month_mode": window.month_mode, "month_nth": window.month_nth,
|
||||
"month_weekday": window.month_weekday, "month_day": window.month_day,
|
||||
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
|
||||
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
|
||||
"created_by": window.created_by, "created_at": window.created_at,
|
||||
"open_now": window_service.current(window, now) is not None,
|
||||
"next": [o.as_dict() for o in window_service.occurrences(window, now, count, with_skipped=True)] if window.enabled else [],
|
||||
"exceptions": [
|
||||
{"id": str(e.id), "occurrence_date": e.occurrence_date.isoformat(), "kind": e.kind,
|
||||
"moved_start": e.moved_start, "reason": e.reason, "actor": e.actor, "created_at": e.created_at}
|
||||
for e in window.exceptions
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def _window_or_404(db: AsyncSession, window_id: uuid.UUID) -> MaintenanceWindow:
|
||||
window = await db.get(MaintenanceWindow, window_id)
|
||||
if window is None:
|
||||
raise api_error(404, ErrorCode.WINDOW_NOT_FOUND)
|
||||
return window
|
||||
|
||||
|
||||
async def _name_taken(db: AsyncSession, name: str, own_id=None) -> bool:
|
||||
other = await db.scalar(select(MaintenanceWindow.id).where(MaintenanceWindow.name == name))
|
||||
return other is not None and other != own_id
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_windows(db: AsyncSession = Depends(get_db)):
|
||||
rows = (await db.scalars(select(MaintenanceWindow).order_by(MaintenanceWindow.name))).all()
|
||||
result = []
|
||||
for w in rows:
|
||||
item = read_window(w)
|
||||
item["used_by"] = await _plans_using(db, w.id)
|
||||
result.append(item)
|
||||
return result
|
||||
|
||||
|
||||
@router.post("", status_code=201)
|
||||
async def create_window(payload: WindowIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
values = _validated(payload)
|
||||
if await _name_taken(db, values["name"]):
|
||||
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
|
||||
window = MaintenanceWindow(**values, created_by=request.state.user.username, created_at=datetime.now(timezone.utc))
|
||||
window.exceptions = []
|
||||
db.add(window)
|
||||
await db.commit()
|
||||
item = read_window(await _window_or_404(db, window.id))
|
||||
item["used_by"] = []
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{window_id}")
|
||||
async def get_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
item = read_window(await _window_or_404(db, window_id))
|
||||
item["used_by"] = await _plans_using(db, window_id)
|
||||
return item
|
||||
|
||||
|
||||
@router.put("/{window_id}")
|
||||
async def update_window(window_id: uuid.UUID, payload: WindowIn, db: AsyncSession = Depends(get_db)):
|
||||
"""Changing a window changes the fingerprint of every plan that uses it, so an activated
|
||||
automatic start stops until the plan is activated again (see rollout_service.fingerprint)."""
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
values = _validated(payload)
|
||||
if await _name_taken(db, values["name"], window.id):
|
||||
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
|
||||
for key, value in values.items():
|
||||
setattr(window, key, value)
|
||||
await db.commit()
|
||||
item = read_window(await _window_or_404(db, window_id))
|
||||
item["used_by"] = await _plans_using(db, window_id)
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{window_id}", status_code=204)
|
||||
async def delete_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
# A plan without a window means "no time limit": deleting its window must never turn it into one.
|
||||
if await _plans_using(db, window_id):
|
||||
raise api_error(409, ErrorCode.WINDOW_IN_USE)
|
||||
await db.delete(window)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.get("/{window_id}/occurrences")
|
||||
async def window_occurrences(window_id: uuid.UUID, count: int = Query(10, ge=1, le=50), db: AsyncSession = Depends(get_db)):
|
||||
window = await _window_or_404(db, window_id)
|
||||
return [o.as_dict() for o in window_service.occurrences(window, datetime.now(timezone.utc), count, with_skipped=True)]
|
||||
|
||||
|
||||
@router.post("/{window_id}/exceptions", status_code=201)
|
||||
async def add_exception(window_id: uuid.UUID, payload: ExceptionIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
now = datetime.now(timezone.utc)
|
||||
if not window_service.is_occurrence_date(window, payload.occurrence_date):
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
|
||||
# Only future occurrences (or the one open now) can be changed; history stays as it was.
|
||||
if window_service.local_start(window, payload.occurrence_date) + timedelta(minutes=window.duration_minutes) <= now:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
|
||||
moved = None
|
||||
if payload.kind == "move":
|
||||
if payload.moved_start is None:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
|
||||
moved = payload.moved_start if payload.moved_start.tzinfo else payload.moved_start.replace(tzinfo=timezone.utc)
|
||||
if moved <= now:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
|
||||
if any(e.occurrence_date == payload.occurrence_date for e in window.exceptions):
|
||||
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
|
||||
exc = MaintenanceWindowException(
|
||||
window_id=window.id, occurrence_date=payload.occurrence_date, kind=payload.kind, moved_start=moved,
|
||||
reason=payload.reason.strip(), actor=request.state.user.username, created_at=now,
|
||||
)
|
||||
db.add(exc)
|
||||
try:
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
|
||||
await audit_service.persist(
|
||||
kind="maintenance_window", action=f"exception_{payload.kind}", status="saved", entity_id=str(window.id),
|
||||
entity_name=window.name, details={"occurrence_date": payload.occurrence_date.isoformat(),
|
||||
"moved_start": moved.isoformat() if moved else None, "reason": exc.reason},
|
||||
)
|
||||
db.expire_all()
|
||||
return read_window(await _window_or_404(db, window_id))
|
||||
|
||||
|
||||
@router.delete("/{window_id}/exceptions/{exception_id}")
|
||||
async def delete_exception(window_id: uuid.UUID, exception_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
exc = await db.get(MaintenanceWindowException, exception_id)
|
||||
if exc is None or exc.window_id != window.id:
|
||||
raise api_error(404, ErrorCode.WINDOW_EXCEPTION_NOT_FOUND)
|
||||
await db.delete(exc)
|
||||
await db.commit()
|
||||
db.expire_all()
|
||||
return read_window(await _window_or_404(db, window_id))
|
||||
|
||||
@@ -88,6 +88,48 @@ TOOLS = [
|
||||
"path": "/api/search",
|
||||
"query": ("q",),
|
||||
},
|
||||
{
|
||||
"name": "rule_match",
|
||||
"description": "Which firewall rule decides a packet (first packet of a new connection), per firewall, "
|
||||
"with the verdict (pass/block/reject), the deciding rule and an honest confidence "
|
||||
"(sure/probable/unknown plus reason codes). Reads only; nothing is changed.",
|
||||
"inputSchema": {
|
||||
"type": "object", "required": ["src", "dst"],
|
||||
"properties": {
|
||||
"src": {"type": "string", "description": "source IP address"},
|
||||
"dst": {"type": "string", "description": "destination IP address"},
|
||||
"proto": {"type": "string", "description": "tcp, udp, icmp, ipv6-icmp or a protocol number", "default": "tcp"},
|
||||
"dport": {"type": "integer", "minimum": 0, "maximum": 65535},
|
||||
"sport": {"type": "integer", "minimum": 0, "maximum": 65535, "description": "default: any ephemeral port"},
|
||||
"icmptype": {"type": "string", "description": "e.g. echoreq or 8"},
|
||||
"iface": {"type": "string", "description": "ingress interface identifier (e.g. lan); default: guessed per firewall"},
|
||||
"egress": {"type": "string", "description": "also check the outgoing side: auto or an interface identifier"},
|
||||
"host_ids": {"type": "string", "description": "comma separated firewall ids (default: all)"},
|
||||
"group_id": {"type": "string", "description": "device group id instead of host_ids"},
|
||||
},
|
||||
},
|
||||
"path": "/api/rule-analysis/match",
|
||||
"query": ("src", "dst", "proto", "dport", "sport", "icmptype", "iface", "egress", "host_ids", "group_id"),
|
||||
# The per-rule trace is for the UI; the verdict, deciding rule and reasons are what matters here.
|
||||
"drop": ("trace",),
|
||||
},
|
||||
{
|
||||
"name": "rule_cleanup_candidates",
|
||||
"description": "Cleanup candidates of one firewall: rules that are unused (no hit for N days), shadowed or "
|
||||
"duplicate (an earlier quick rule already decides everything they match), or disabled for a "
|
||||
"long time. Each with its confidence (sure/probable/unknown), the last observed hit and "
|
||||
"whether it could be removed (not for automatic, legacy or OPNexus-managed rules). Reads only.",
|
||||
"inputSchema": {
|
||||
"type": "object", "required": ["host_id"],
|
||||
"properties": {
|
||||
"host_id": {"type": "string", "description": "Firewall id from list_hosts"},
|
||||
"unused_days": {"type": "integer", "minimum": 1, "maximum": 3650, "default": 30,
|
||||
"description": "observation window for unused / disabled rules"},
|
||||
},
|
||||
},
|
||||
"path": "/api/hosts/{host_id}/rule-analysis/cleanup",
|
||||
"query": ("unused_days",),
|
||||
},
|
||||
{
|
||||
"name": "api_get",
|
||||
"description": "Read any other OPNexus API endpoint (GET only), e.g. /api/rules, /api/aliases, /api/logs. "
|
||||
|
||||
@@ -1,21 +1,26 @@
|
||||
import uuid
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.models.notification import NotificationProblem
|
||||
from app.models.notification import NotificationChannel, NotificationProblem
|
||||
from app.schemas.notification import (
|
||||
NotificationChannelCreate,
|
||||
NotificationChannelRead,
|
||||
NotificationChannelUpdate,
|
||||
NotificationProblemRead,
|
||||
NotificationSettingsRead,
|
||||
NotificationSettingsUpdate,
|
||||
TestNotificationRequest,
|
||||
)
|
||||
from app.services import notification_delivery, notification_settings_service
|
||||
from app.services import notification_channels, notification_delivery, notification_secret_store, notification_settings_service, url_guard
|
||||
|
||||
router = APIRouter(prefix="/notifications", tags=["notifications"])
|
||||
|
||||
@@ -43,6 +48,11 @@ async def read_settings(db: AsyncSession = Depends(get_db), user: AdminUser = De
|
||||
|
||||
@router.put("/settings", response_model=NotificationSettingsRead)
|
||||
async def update_settings(payload: NotificationSettingsUpdate, db: AsyncSession = Depends(get_db)):
|
||||
if payload.webhook_url:
|
||||
try:
|
||||
payload.webhook_url = url_guard.validate_url(payload.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise api_error(422, ErrorCode.NOTIFICATION_WEBHOOK_URL_INVALID, reason=str(exc)) from None
|
||||
return await notification_settings_service.update_settings(db, payload)
|
||||
|
||||
|
||||
@@ -79,3 +89,103 @@ async def test_notification(payload: TestNotificationRequest, db: AsyncSession =
|
||||
except notification_delivery.NotificationDeliveryError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ── Additional channels (ntfy, Gotify, Telegram, Teams, Matrix) ────────────────
|
||||
|
||||
def _channel_read(row: NotificationChannel, user: AdminUser) -> NotificationChannelRead:
|
||||
read = NotificationChannelRead.model_validate(row)
|
||||
# An ntfy topic or a chat id is as good as a password for whoever knows it: only admins see the settings.
|
||||
return read if user.role == "admin" else read.model_copy(update={"config": {}})
|
||||
|
||||
|
||||
async def _get_channel(db: AsyncSession, channel_id: uuid.UUID) -> NotificationChannel:
|
||||
row = await db.get(NotificationChannel, channel_id)
|
||||
if row is None:
|
||||
raise api_error(404, ErrorCode.NOTIFICATION_CHANNEL_NOT_FOUND)
|
||||
return row
|
||||
|
||||
|
||||
def _invalid(exc: ValueError):
|
||||
return api_error(422, ErrorCode.NOTIFICATION_CHANNEL_INVALID, reason=str(exc))
|
||||
|
||||
|
||||
@router.get("/channels", response_model=list[NotificationChannelRead])
|
||||
async def list_channels(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
rows = await db.scalars(select(NotificationChannel).order_by(NotificationChannel.name))
|
||||
return [_channel_read(r, user) for r in rows.all()]
|
||||
|
||||
|
||||
@router.post("/channels", response_model=NotificationChannelRead, status_code=201)
|
||||
async def create_channel(payload: NotificationChannelCreate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
try:
|
||||
config = notification_channels.clean_config(payload.channel_type, payload.config)
|
||||
secret = notification_channels.clean_secret(payload.channel_type, payload.secret) if payload.secret else None
|
||||
except ValueError as exc:
|
||||
raise _invalid(exc) from None
|
||||
row = NotificationChannel(
|
||||
name=payload.name.strip(), channel_type=payload.channel_type, enabled=payload.enabled, config=config,
|
||||
secret_set=secret is not None,
|
||||
)
|
||||
row.id = uuid.uuid4()
|
||||
if secret is None and notification_channels.missing_requirements(row) == "Zugangsdaten fehlen":
|
||||
raise _invalid(ValueError("Zugangsdaten fehlen"))
|
||||
db.add(row)
|
||||
try:
|
||||
await db.flush()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
|
||||
if secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(row.id), secret)
|
||||
await db.commit()
|
||||
return _channel_read(row, user)
|
||||
|
||||
|
||||
@router.put("/channels/{channel_id}", response_model=NotificationChannelRead)
|
||||
async def update_channel(
|
||||
channel_id: uuid.UUID, payload: NotificationChannelUpdate,
|
||||
db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user),
|
||||
):
|
||||
row = await _get_channel(db, channel_id)
|
||||
try:
|
||||
if payload.config is not None:
|
||||
row.config = notification_channels.clean_config(row.channel_type, payload.config)
|
||||
new_secret = notification_channels.clean_secret(row.channel_type, payload.secret) if payload.secret else None
|
||||
except ValueError as exc:
|
||||
raise _invalid(exc) from None
|
||||
if payload.name is not None:
|
||||
row.name = payload.name.strip()
|
||||
if payload.enabled is not None:
|
||||
row.enabled = payload.enabled
|
||||
if payload.clear_secret:
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
|
||||
row.secret_set = False
|
||||
elif new_secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(row.id), new_secret)
|
||||
row.secret_set = True
|
||||
try:
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
|
||||
return _channel_read(row, user)
|
||||
|
||||
|
||||
@router.delete("/channels/{channel_id}", status_code=204)
|
||||
async def delete_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
row = await _get_channel(db, channel_id)
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
|
||||
await db.delete(row)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.post("/channels/{channel_id}/test")
|
||||
async def test_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
row = await _get_channel(db, channel_id)
|
||||
try:
|
||||
await notification_channels.send_test(row)
|
||||
except notification_delivery.NotificationDeliveryError as exc:
|
||||
# The text names the channel and the HTTP status; it never contains a URL or a token.
|
||||
raise api_error(502, ErrorCode.NOTIFICATION_CHANNEL_TEST_FAILED, reason=str(exc)) from None
|
||||
return {"success": True}
|
||||
@@ -22,6 +22,28 @@ logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["oidc"])
|
||||
CALLBACK_PATH = "/api/auth/oidc/callback"
|
||||
STATE_COOKIE = "opnexus-oidc-state"
|
||||
# HTTPS deployments (1.29): __Host- (Secure, no Domain, Path=/) so that no sibling SUBDOMAIN or plain-HTTP page can
|
||||
# plant a state cookie of its own (login CSRF with the attacker's own authorisation). Known limit: cookies are not
|
||||
# separated by PORT, so script in a firewall GUI shown on a gateway slot port of the same host could still set it
|
||||
# (BACKLOG S8, accepted: needs a compromised firewall GUI and a mapped IdP account; the real fix is one hostname
|
||||
# per slot). The browser only accepts the prefix with Path=/, so the cookie travels with every request for its
|
||||
# 10 minutes; plain-HTTP labs keep the narrow path.
|
||||
SECURE_STATE_COOKIE = "__Host-opnexus-oidc-state"
|
||||
STATE_COOKIE_PATH = "/api/auth/oidc"
|
||||
|
||||
|
||||
def state_cookie_name() -> str:
|
||||
return SECURE_STATE_COOKIE if settings.cookie_secure else STATE_COOKIE
|
||||
|
||||
|
||||
def _state_cookie_path() -> str:
|
||||
return "/" if settings.cookie_secure else STATE_COOKIE_PATH
|
||||
|
||||
|
||||
def _clear_state_cookie(response) -> None:
|
||||
response.delete_cookie(state_cookie_name(), path=_state_cookie_path(), secure=settings.cookie_secure, httponly=True, samesite="lax")
|
||||
if settings.cookie_secure: # a plain-name cookie of an earlier version
|
||||
response.delete_cookie(STATE_COOKIE, path=STATE_COOKIE_PATH, secure=True, httponly=True, samesite="lax")
|
||||
# SSO sessions are short: the provider's group membership is only re-checked at login, so a removed person
|
||||
# must not keep a 30-day session.
|
||||
SSO_SESSION_SECONDS = 12 * 3600
|
||||
@@ -29,7 +51,7 @@ SSO_SESSION_SECONDS = 12 * 3600
|
||||
|
||||
def _login_error(code: str) -> RedirectResponse:
|
||||
response = RedirectResponse(f"/login?sso_error={code}", status_code=302)
|
||||
response.delete_cookie(STATE_COOKIE, path="/api/auth/oidc")
|
||||
_clear_state_cookie(response)
|
||||
return response
|
||||
|
||||
|
||||
@@ -63,7 +85,8 @@ async def start(request: Request, origin: str, db: AsyncSession = Depends(get_db
|
||||
response = RedirectResponse(url, status_code=302)
|
||||
# Binds the attempt to THIS browser: a callback URL handed to somebody else (login CSRF) or a captured
|
||||
# code+state pair is useless without the cookie.
|
||||
response.set_cookie(STATE_COOKIE, state, max_age=600, httponly=True, samesite="lax", secure=settings.cookie_secure, path="/api/auth/oidc")
|
||||
response.set_cookie(state_cookie_name(), state, max_age=600, httponly=True, samesite="lax", secure=settings.cookie_secure,
|
||||
path=_state_cookie_path())
|
||||
return response
|
||||
|
||||
|
||||
@@ -78,7 +101,9 @@ async def callback(request: Request, code: str = "", state: str = "", error: str
|
||||
try:
|
||||
if error or not code or not state:
|
||||
raise oidc_service.OidcError("provider", "Anbieter meldet " + (_printable(error) if error else "keinen Code"))
|
||||
if not secrets.compare_digest(request.cookies.get(STATE_COOKIE, ""), state):
|
||||
# Only the cookie name of this deployment counts (on HTTPS never the plain one); no cookie never matches.
|
||||
bound = request.cookies.get(state_cookie_name(), "")
|
||||
if not bound or not secrets.compare_digest(bound.encode(), state.encode()):
|
||||
raise oidc_service.OidcError("state", "Der Anmeldeversuch gehört nicht zu diesem Browser")
|
||||
user = await oidc_service.complete(db, cfg, code, state)
|
||||
except oidc_service.OidcError as exc:
|
||||
@@ -92,7 +117,7 @@ async def callback(request: Request, code: str = "", state: str = "", error: str
|
||||
)
|
||||
response = RedirectResponse("/dashboard", status_code=302)
|
||||
_set_session_cookie(response, token, max_age=SSO_SESSION_SECONDS)
|
||||
response.delete_cookie(STATE_COOKIE, path="/api/auth/oidc")
|
||||
_clear_state_cookie(response)
|
||||
logger.info("SSO-Anmeldung von '%s' (%s) aus %s", user.username, user.role, ip)
|
||||
return response
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.one_to_one_deploy_service import confirm_one_to_one, deploy_one_to_one, undeploy_one_to_one, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/one-to-one", tags=["one-to-one"])
|
||||
|
||||
@@ -49,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
@@ -79,6 +81,9 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="one_to_one", model=OneToOneRule, load_options=_load_options, to_read=_to_read, read_model=OneToOneRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[OneToOneRead])
|
||||
async def list_one_to_one_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(OneToOneRule).options(*_load_options()))
|
||||
@@ -227,6 +232,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_one_to_one_rule(deployment.host, deployment.opnsense_one_to_one_uuid)
|
||||
|
||||
@@ -18,6 +18,7 @@ from app.services import openvpn_key_store
|
||||
from app.services.openvpn_service import effective_hosts, preview, revision, snapshot, workflow, OPENVPN_FIELDS
|
||||
from app.services.openvpn_deploy_service import deploy_instance, undeploy_instance
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_lease import resource_lease
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/openvpn/instances", tags=["openvpn"])
|
||||
@@ -194,18 +195,21 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
|
||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_instance_uuid:
|
||||
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
|
||||
if deployment.opnsense_statickey_uuid:
|
||||
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
|
||||
await opnsense_client.reconfigure_openvpn(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
async with resource_lease("openvpn", instance.id):
|
||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_instance_uuid:
|
||||
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
|
||||
if deployment.opnsense_statickey_uuid:
|
||||
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
|
||||
await opnsense_client.reconfigure_openvpn(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
@@ -21,6 +21,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.outbound_nat_deploy_service import confirm_outbound_nat, deploy_outbound_nat, undeploy_outbound_nat, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/outbound-nat", tags=["outbound-nat"])
|
||||
|
||||
@@ -50,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
@@ -82,6 +84,9 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="outbound_nat", model=OutboundNatRule, load_options=_load_options, to_read=_to_read, read_model=OutboundNatRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[OutboundNatRead])
|
||||
async def list_outbound_nat_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(OutboundNatRule).options(*_load_options()))
|
||||
@@ -234,6 +239,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_outbound_nat_rule(deployment.host, deployment.opnsense_outbound_nat_uuid)
|
||||
|
||||
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
hierarchy_group_id=rule.hierarchy_group_id,
|
||||
rulebase=rule.rulebase,
|
||||
sequence=rule.sequence,
|
||||
@@ -330,6 +331,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
try:
|
||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_policy_hierarchy_uuid)
|
||||
await opnsense_client.apply(deployment.host)
|
||||
|
||||
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.port_forward_deploy_service import confirm_port_forward, deploy_port_forward, undeploy_port_forward, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/port-forward", tags=["port-forward"])
|
||||
|
||||
@@ -49,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
ipprotocol=rule.ipprotocol,
|
||||
@@ -80,6 +82,9 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="port_forward", model=PortForwardRule, load_options=_load_options, to_read=_to_read, read_model=PortForwardRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[PortForwardRead])
|
||||
async def list_port_forward_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(PortForwardRule).options(*_load_options()))
|
||||
@@ -229,6 +234,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_port_forward_rule(deployment.host, deployment.opnsense_port_forward_uuid)
|
||||
|
||||
@@ -0,0 +1,459 @@
|
||||
"""Firmware rollouts in waves (1.25): plans and runs.
|
||||
|
||||
Roles (enforced in AuthMiddleware, see api_token_service.is_rollout): everybody signed in reads;
|
||||
operators prepare plans; starting/activating a plan and resuming/approving/skipping/aborting a run
|
||||
need the deploy right (admin, or operator while operators may deploy). Pausing is deliberately NOT
|
||||
a rollout action: anybody who may write at all may always brake.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||
|
||||
ACTIVE_RUN = rollout_engine.ACTIVE
|
||||
|
||||
|
||||
class WaveIn(BaseModel):
|
||||
label: str = Field(min_length=1, max_length=60)
|
||||
group_ids: list[uuid.UUID] = Field(min_length=1, max_length=20)
|
||||
soak_minutes: int = Field(default=30, ge=0, le=rollout_service.MAX_SOAK_MINUTES)
|
||||
carp_handover: bool = False
|
||||
require_approval: bool = False
|
||||
|
||||
|
||||
class ProbeIn(BaseModel):
|
||||
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
|
||||
type: Literal["ping", "tcp"]
|
||||
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
|
||||
port: int | None = Field(default=None, ge=1, le=65535)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _port_only_for_tcp(self):
|
||||
if (self.type == "tcp") != (self.port is not None):
|
||||
raise ValueError("port_required_for_tcp_only")
|
||||
return self
|
||||
|
||||
|
||||
class PlanIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
description: str = Field(default="", max_length=2000)
|
||||
action: Literal["update", "major_update"] = "update"
|
||||
window_id: uuid.UUID | None = None
|
||||
auto_start: bool = False
|
||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
|
||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||
|
||||
|
||||
class TokenIn(BaseModel):
|
||||
preview_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
async def _plan_or_404(db: AsyncSession, plan_id: uuid.UUID) -> RolloutPlan:
|
||||
plan = await rollout_service.load_plan(db, plan_id)
|
||||
if plan is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_PLAN_NOT_FOUND)
|
||||
return plan
|
||||
|
||||
|
||||
async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None = None) -> dict:
|
||||
resolved = resolved or await rollout_service.resolve(db, plan)
|
||||
active = plan.activated_at is not None
|
||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||
return {
|
||||
**rollout_service.definition(plan),
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"description": plan.description,
|
||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||
"group_ids": [str(g.id) for g in w.groups], "groups": [g.name for g in w.groups],
|
||||
"host_count": len(resolved["waves"][i]["hosts"]) if i < len(resolved["waves"]) else 0}
|
||||
for i, w in enumerate(plan.waves)],
|
||||
"active": active, "activated_by": plan.activated_by, "activated_at": plan.activated_at,
|
||||
# Activated but the plan, its groups or its window changed since: the automatic start is held back.
|
||||
"fingerprint_ok": (not active) or plan.activated_fingerprint == rollout_service.fingerprint(plan, resolved),
|
||||
"auto_start_error": plan.auto_start_error,
|
||||
"window": resolved["window"], "errors": resolved["errors"], "warnings": resolved["warnings"],
|
||||
"host_count": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
"last_run": {"id": str(last_run.id), "status": last_run.status, "created_at": last_run.created_at} if last_run else None,
|
||||
"created_by": plan.created_by, "created_at": plan.created_at, "updated_at": plan.updated_at,
|
||||
}
|
||||
|
||||
|
||||
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
for probe in probes:
|
||||
value = {"type": probe.type, "target": probe.target, "port": probe.port}
|
||||
if value not in out:
|
||||
out.append(value)
|
||||
return out
|
||||
|
||||
|
||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="name")
|
||||
other = await db.scalar(select(RolloutPlan.id).where(RolloutPlan.name == name))
|
||||
if other is not None and other != plan.id:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_NAME_TAKEN)
|
||||
if payload.window_id is not None and await db.get(MaintenanceWindow, payload.window_id) is None:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="window_id")
|
||||
if any(k not in rollout_health.CHECKS for k in payload.gate_checks):
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="gate_checks")
|
||||
wanted = {g for w in payload.waves for g in w.group_ids}
|
||||
groups = {g.id: g for g in (await db.scalars(select(DeviceGroup).where(DeviceGroup.id.in_(wanted)))).all()}
|
||||
if len(groups) != len(wanted):
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="group_ids")
|
||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||
plan.gate_probes = _unique_probes(payload.gate_probes)
|
||||
plan.compliance_mode = payload.compliance_mode
|
||||
plan.waves = [
|
||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||
require_approval=w.require_approval, groups=[groups[g] for g in dict.fromkeys(w.group_ids)])
|
||||
for i, w in enumerate(payload.waves)
|
||||
]
|
||||
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
|
||||
plan.activated_at = plan.activated_by = plan.activated_by_id = plan.activated_fingerprint = None
|
||||
plan.auto_start_error = None
|
||||
|
||||
|
||||
@router.get("/plans")
|
||||
async def list_plans(db: AsyncSession = Depends(get_db)):
|
||||
plans = (await db.scalars(select(RolloutPlan).order_by(RolloutPlan.name))).all()
|
||||
return [await read_plan(db, p) for p in plans]
|
||||
|
||||
|
||||
@router.post("/plans", status_code=201)
|
||||
async def create_plan(payload: PlanIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = RolloutPlan(created_by=request.state.user.username, created_at=datetime.now(timezone.utc), waves=[])
|
||||
await _apply(db, plan, payload)
|
||||
db.add(plan)
|
||||
await db.commit()
|
||||
return await read_plan(db, await _plan_or_404(db, plan.id))
|
||||
|
||||
|
||||
@router.get("/plans/{plan_id}")
|
||||
async def get_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.put("/plans/{plan_id}")
|
||||
async def update_plan(plan_id: uuid.UUID, payload: PlanIn, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
was_active = plan.activated_at is not None
|
||||
await _apply(db, plan, payload)
|
||||
await db.commit()
|
||||
if was_active:
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id),
|
||||
entity_name=plan.name, message="Plan geändert: automatischer Start deaktiviert")
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.delete("/plans/{plan_id}", status_code=204)
|
||||
async def delete_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.status.in_(ACTIVE_RUN)).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_PLAN_BUSY)
|
||||
await db.delete(plan)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/preview")
|
||||
async def preview_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
return {**resolved, "preview_token": rollout_service.fingerprint(plan, resolved),
|
||||
"can_start": not resolved["errors"], "plan": await read_plan(db, plan, resolved)}
|
||||
|
||||
|
||||
async def _checked_token(db: AsyncSession, plan: RolloutPlan, token: str) -> tuple[dict, str]:
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
current = rollout_service.fingerprint(plan, resolved)
|
||||
if token != current:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_PREVIEW_STALE)
|
||||
if resolved["errors"]:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_PLAN_INVALID, reason=resolved["errors"][0]["code"])
|
||||
return resolved, current
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/activate")
|
||||
async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if not plan.auto_start:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
|
||||
_resolved, current = await _checked_token(db, plan, payload.preview_token)
|
||||
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
|
||||
plan.activated_by_id = request.state.user.id
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
|
||||
entity_name=plan.name, details={"fingerprint": current})
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/deactivate")
|
||||
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
plan.activated_by = plan.activated_by_id = plan.activated_at = plan.activated_fingerprint = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
# ── runs ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
class StartIn(TokenIn):
|
||||
confirm_name: str = Field(min_length=1, max_length=120)
|
||||
acknowledge_unattended: Literal[True]
|
||||
|
||||
|
||||
class ReasonIn(BaseModel):
|
||||
reason: str = Field(default="", max_length=500)
|
||||
|
||||
|
||||
class SkipIn(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
reason: str = Field(min_length=3, max_length=500)
|
||||
|
||||
|
||||
class AbortIn(BaseModel):
|
||||
reason: str = Field(min_length=3, max_length=500)
|
||||
confirm_name: str = Field(min_length=1, max_length=120)
|
||||
|
||||
|
||||
def _read_run_summary(run: RolloutRun) -> dict:
|
||||
counts: dict[str, int] = {}
|
||||
for h in run.hosts:
|
||||
counts[h.status] = counts.get(h.status, 0) + 1
|
||||
return {
|
||||
"id": str(run.id), "plan_id": str(run.plan_id) if run.plan_id else None, "plan_name": run.plan_name,
|
||||
"action": run.action, "status": run.status, "paused_from": run.paused_from, "current_wave": run.current_wave,
|
||||
"wave_count": len(run.waves), "current_wave_label": next((w.label for w in run.waves if w.index == run.current_wave), None),
|
||||
"target_version": run.target_version, "pause_reason": run.pause_reason, "pause_detail": run.pause_detail,
|
||||
"started_by": run.started_by, "trigger": run.trigger, "window_occurrence": run.window_occurrence,
|
||||
"created_at": run.created_at, "finished_at": run.finished_at, "step_at": run.step_at, "host_counts": counts,
|
||||
"active": run.status in rollout_engine.ACTIVE,
|
||||
}
|
||||
|
||||
|
||||
async def read_run(db: AsyncSession, run: RolloutRun) -> dict:
|
||||
scheds = {}
|
||||
ids = [h.scheduled_maintenance_id for h in run.hosts if h.scheduled_maintenance_id]
|
||||
if ids:
|
||||
scheds = {s.id: s for s in (await db.scalars(select(ScheduledMaintenance).where(ScheduledMaintenance.id.in_(ids)))).all()}
|
||||
soak_until = next((w.soak_until for w in run.waves if w.index == run.current_wave), None)
|
||||
return {
|
||||
**_read_run_summary(run),
|
||||
"soak_until": soak_until, "progress": {k: v for k, v in (run.progress or {}).items()
|
||||
if k in ("gate_recheck_at", "cluster_recheck_at", "blocked_since", "resume_gate",
|
||||
"cluster_gate", "preflight_started", "resume_recheck_at")},
|
||||
"snapshot": run.plan_snapshot,
|
||||
"waves": [{"index": w.index, "label": w.label, "status": w.status, "gate_result": w.gate_result,
|
||||
"gate_attempts": w.gate_attempts, "soak_until": w.soak_until, "approved_by": w.approved_by,
|
||||
"started_at": w.started_at, "finished_at": w.finished_at,
|
||||
"spec": run.plan_snapshot["waves"][w.index] if w.index < len(run.plan_snapshot.get("waves", [])) else None}
|
||||
for w in run.waves],
|
||||
"hosts": [{"id": str(h.id), "host_id": str(h.host_id) if h.host_id else None, "host_name": h.host_name,
|
||||
"cluster_name": h.cluster_name, "wave_index": h.wave_index, "order": h.order, "status": h.status,
|
||||
"version_before": h.version_before, "version_after": h.version_after, "detail": h.detail,
|
||||
"started_at": h.started_at, "finished_at": h.finished_at,
|
||||
"schedule": ({"id": str(s.id), "status": s.status, "phase": s.phase, "carp_handover": s.carp_handover,
|
||||
"host_operation_id": str(s.host_operation_id) if s.host_operation_id else None}
|
||||
if (s := scheds.get(h.scheduled_maintenance_id)) else None)}
|
||||
for h in sorted(run.hosts, key=lambda h: (h.wave_index, h.order, h.host_name))],
|
||||
}
|
||||
|
||||
|
||||
async def _cancel_if_pending(db: AsyncSession, sched_id) -> bool:
|
||||
"""Cancel a host's schedule only if it has not fired. Locked: the poll loop may be firing it right now
|
||||
(fire_due holds the row), and a blind write would turn a running action into "cancelled"."""
|
||||
if sched_id is None:
|
||||
return False
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sched_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if sched is not None and sched.status == "pending":
|
||||
sched.status = "cancelled"
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _run_or_404(db: AsyncSession, run_id: uuid.UUID, *, lock: bool = False) -> RolloutRun:
|
||||
stmt = select(RolloutRun).where(RolloutRun.id == run_id).execution_options(populate_existing=True)
|
||||
if lock:
|
||||
stmt = stmt.with_for_update()
|
||||
run = await db.scalar(stmt)
|
||||
if run is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_RUN_NOT_FOUND)
|
||||
return run
|
||||
|
||||
|
||||
def _not_now(run: RolloutRun):
|
||||
return api_error(409, ErrorCode.ROLLOUT_ACTION_NOT_ALLOWED, status=run.status)
|
||||
|
||||
|
||||
async def _done(db: AsyncSession, run: RolloutRun, action: str, status: str, request: Request, **details) -> dict:
|
||||
run.revision += 1
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id), entity_name=run.plan_name,
|
||||
details={"wave": run.current_wave, **details})
|
||||
return await read_run(db, await _run_or_404(db, run.id))
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/start", status_code=201)
|
||||
async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if payload.confirm_name != plan.name:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
|
||||
resolved, _token = await _checked_token(db, plan, payload.preview_token)
|
||||
if (other := await rollout_engine.active_run(db)) is not None:
|
||||
# One run at a time: two runs could otherwise touch the same cluster at once.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
|
||||
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username,
|
||||
started_by_id=request.state.user.id, trigger="manual")
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
|
||||
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
"waves": [w["label"] for w in resolved["waves"]]})
|
||||
return await read_run(db, await _run_or_404(db, run.id))
|
||||
|
||||
|
||||
@router.get("/runs")
|
||||
async def list_runs(db: AsyncSession = Depends(get_db)):
|
||||
runs = (await db.scalars(select(RolloutRun).order_by(RolloutRun.created_at.desc()).limit(50))).all()
|
||||
return [_read_run_summary(r) for r in runs]
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def rollout_status(db: AsyncSession = Depends(get_db)):
|
||||
"""For /updates and the host pages: the active run and the hosts it still owns."""
|
||||
run = await rollout_engine.active_run(db)
|
||||
if run is None:
|
||||
return {"active": None, "host_ids": [], "current_host_ids": []}
|
||||
return {
|
||||
"active": _read_run_summary(run),
|
||||
"host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.status in ("queued", "blocked", "running", "verifying", "failed")],
|
||||
"current_host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.wave_index == run.current_wave
|
||||
and h.status in ("queued", "blocked", "running", "verifying", "failed")],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}")
|
||||
async def get_run(run_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
return await read_run(db, await _run_or_404(db, run_id))
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/pause")
|
||||
async def pause_run(run_id: uuid.UUID, payload: ReasonIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE or run.status == "paused":
|
||||
raise _not_now(run)
|
||||
# A brake that lets the next host fire anyway is no brake: a step not fired yet goes back to the queue.
|
||||
requeued = [h.host_name for h in run.hosts if await rollout_engine.requeue_if_pending(db, h)]
|
||||
rollout_engine.pause(run, "manual", by=request.state.user.username, text=payload.reason.strip()[:300])
|
||||
return await _done(db, run, "pause", "paused", request, reason="manual", text=payload.reason.strip()[:300], requeued=requeued)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/resume")
|
||||
async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status != "paused":
|
||||
raise _not_now(run)
|
||||
failed = [h.host_name for h in run.hosts if h.status == "failed"]
|
||||
if failed:
|
||||
# A failed host is never retried by itself: check it, then skip it explicitly.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
|
||||
previous = run.started_by
|
||||
# Whoever resumes takes over the responsibility (and the permission check) from here on.
|
||||
run.started_by, run.started_by_id = request.state.user.username, request.state.user.id
|
||||
run.status = "resuming"
|
||||
run.pause_reason = run.pause_detail = None
|
||||
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
|
||||
return await _done(db, run, "resume", "resuming", request, previous_owner=previous)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/approve")
|
||||
async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status != "awaiting_approval":
|
||||
raise _not_now(run)
|
||||
wave = rollout_engine.wave_row(run, run.current_wave)
|
||||
wave.approved_by, wave.status = request.state.user.username, "preflight"
|
||||
run.started_by, run.started_by_id = request.state.user.username, request.state.user.id
|
||||
run.status = "preflight"
|
||||
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
|
||||
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/skip-host")
|
||||
async def skip_host(run_id: uuid.UUID, payload: SkipIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE:
|
||||
raise _not_now(run)
|
||||
host = next((h for h in run.hosts if h.host_id == payload.host_id), None)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_HOST_NOT_IN_RUN)
|
||||
if host.status in ("running", "verifying"):
|
||||
if not await _cancel_if_pending(db, host.scheduled_maintenance_id):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
|
||||
# not fired yet: nothing happened on the firewall
|
||||
elif host.status not in ("queued", "blocked", "failed"):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
|
||||
host.status = "skipped"
|
||||
host.detail = {"code": "skipped_by", "params": {"user": request.state.user.username, "reason": payload.reason.strip()[:300]}}
|
||||
host.finished_at = datetime.now(timezone.utc)
|
||||
return await _done(db, run, "skip_host", "skipped", request, host=host.host_name, reason=payload.reason.strip()[:300])
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/abort")
|
||||
async def abort_run(run_id: uuid.UUID, payload: AbortIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE:
|
||||
raise _not_now(run)
|
||||
if payload.confirm_name != run.plan_name:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
|
||||
now = datetime.now(timezone.utc)
|
||||
still_running = []
|
||||
for host in run.hosts:
|
||||
if host.status in ("running", "verifying") and host.scheduled_maintenance_id:
|
||||
if await _cancel_if_pending(db, host.scheduled_maintenance_id):
|
||||
host.status, host.finished_at = "cancelled", now
|
||||
else:
|
||||
still_running.append(host.host_name) # an action already on the firewall is finished in a controlled way
|
||||
elif host.status in ("queued", "blocked"):
|
||||
host.status, host.finished_at = "cancelled", now
|
||||
for wave in run.waves:
|
||||
if wave.status not in ("passed", "skipped"):
|
||||
wave.status = "cancelled"
|
||||
run.status, run.finished_at = "cancelled", now
|
||||
run.pause_reason, run.pause_detail = "aborted", {"by": request.state.user.username, "reason": payload.reason.strip()[:300],
|
||||
"still_running": still_running}
|
||||
return await _done(db, run, "abort", "cancelled", request, reason=payload.reason.strip()[:300], still_running=still_running)
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Rule analysis: which rule decides a packet (read-only, GET so ``read`` tokens and MCP can use it).
|
||||
|
||||
Nothing is stored and nothing is written to a firewall; every request reads the rules, the
|
||||
loaded pf ruleset and the interfaces (cached for 60 s per firewall, ``refresh=true`` skips
|
||||
the cache) and evaluates them with services/rule_analysis/engine.py.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rule_history import LiveRuleState
|
||||
from app.services import rule_history_service
|
||||
from app.services.rule_analysis import service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["rule-analysis"])
|
||||
_PARALLEL = 5
|
||||
|
||||
|
||||
def _query(src: str, dst: str, proto: str, sport: str | None, dport: str | None, icmptype: str | None,
|
||||
iface: str | None, direction: str, egress: str | None):
|
||||
return service.build_query(src=src, dst=dst, proto=proto, sport=sport, dport=dport, icmptype=icmptype,
|
||||
iface=iface, direction=direction, egress=egress)
|
||||
|
||||
|
||||
async def _run(host: OpnsenseHost, q, refresh: bool) -> dict:
|
||||
head = {"host_id": str(host.id), "host_name": host.name, "cluster_name": host.cluster_name}
|
||||
try:
|
||||
return {**head, **await service.analyze_host(host, q, refresh=refresh), "error": None}
|
||||
except HTTPException as exc:
|
||||
return {**head, "error": exc.detail}
|
||||
except Exception as exc: # noqa: BLE001 - one unreachable firewall must not fail the fleet answer
|
||||
logger.info("rule analysis failed for %s: %s", host.name, exc)
|
||||
return {**head, "error": {"code": ErrorCode.ANALYSE_FIREWALL_NICHT_LESBAR.value, "params": {}}}
|
||||
|
||||
|
||||
@router.get("/hosts/{host_id}/rule-analysis/match")
|
||||
async def host_match(
|
||||
host_id: uuid.UUID, src: str, dst: str, proto: str = "tcp", sport: str | None = None, dport: str | None = None,
|
||||
icmptype: str | None = None, iface: str | None = None, direction: str = "in", egress: str | None = None,
|
||||
refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
q = _query(src, dst, proto, sport, dport, icmptype, iface, direction, egress)
|
||||
result = await _run(host, q, refresh)
|
||||
error = result.get("error")
|
||||
if error:
|
||||
status = 422 if error.get("code") == ErrorCode.ANALYSE_INTERFACE_UNBEKANNT.value else 502
|
||||
raise HTTPException(status_code=status, detail=error)
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/rule-analysis/match")
|
||||
async def fleet_match(
|
||||
src: str, dst: str, proto: str = "tcp", sport: str | None = None, dport: str | None = None,
|
||||
icmptype: str | None = None, iface: str | None = None, direction: str = "in", egress: str | None = None,
|
||||
host_ids: str | None = Query(None, description="comma separated host ids; default: all enabled hosts"),
|
||||
group_id: uuid.UUID | None = None, refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
q = _query(src, dst, proto, sport, dport, icmptype, iface, direction, egress)
|
||||
if group_id is not None:
|
||||
group = await db.scalar(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)).where(DeviceGroup.id == group_id))
|
||||
if group is None:
|
||||
raise api_error(404, ErrorCode.GRUPPE_NICHT_GEFUNDEN)
|
||||
hosts = sorted(group.hosts, key=lambda h: h.name)
|
||||
elif host_ids:
|
||||
try:
|
||||
ids = [uuid.UUID(part.strip()) for part in host_ids.split(",") if part.strip()]
|
||||
except ValueError:
|
||||
raise api_error(422, ErrorCode.ANALYSE_PARAMETER_UNGUELTIG, field="host_ids", value=host_ids[:64]) from None
|
||||
hosts = list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)).order_by(OpnsenseHost.name))).all())
|
||||
if len(hosts) != len(set(ids)):
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
else:
|
||||
hosts = list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)).order_by(OpnsenseHost.name))).all())
|
||||
gate = asyncio.Semaphore(_PARALLEL)
|
||||
|
||||
async def one(host):
|
||||
async with gate:
|
||||
return await _run(host, q, refresh)
|
||||
|
||||
return list(await asyncio.gather(*(one(h) for h in hosts)))
|
||||
|
||||
|
||||
@router.get("/hosts/{host_id}/rule-analysis/cleanup")
|
||||
async def host_cleanup(
|
||||
host_id: uuid.UUID, unused_days: int = Query(30, ge=1, le=3650, description="observation window for 'unused' / 'disabled for a long time'"),
|
||||
refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Cleanup candidates (unused, shadowed, duplicate, disabled for a long time) with confidence and whether
|
||||
each rule could be offered for removal. Read-only; removal is not part of this endpoint."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
return await service.cleanup_candidates(db, host, unused_days=unused_days, refresh=refresh)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001 - an unreachable firewall is a 502, not a crash
|
||||
logger.info("cleanup candidates failed for %s: %s", host.name, exc)
|
||||
raise api_error(502, ErrorCode.ANALYSE_FIREWALL_NICHT_LESBAR) from exc
|
||||
|
||||
|
||||
@router.get("/hosts/{host_id}/live-rules/{rule_uuid}/history")
|
||||
async def live_rule_history(host_id: uuid.UUID, rule_uuid: str, limit: int = Query(100, ge=1, le=500),
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""What happened to one rule on one firewall: observed changes (also hand-made ones), saved revisions of
|
||||
rules OPNexus manages and the audit entries, newest first. Reads the database only."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
baseline = await db.get(LiveRuleState, (host_id, rule_history_service.BASELINE_KEY))
|
||||
return {
|
||||
"host_id": str(host.id), "rule_uuid": rule_uuid.lower()[:64],
|
||||
"watching_since": baseline.first_seen_at if baseline else None,
|
||||
"events": await rule_history_service.rule_timeline(db, host, rule_uuid, limit),
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): switch hand-made rules off, delete them later, restore them (DESIGN-1.31 §4).
|
||||
|
||||
POST /hosts/{id}/rule-cleanup/preview operator, admin prepare (writes nothing to a firewall)
|
||||
POST /hosts/{id}/rule-cleanup/{op}/cancel operator, admin the preview's own actor
|
||||
POST /hosts/{id}/rule-cleanup/{op}/deploy ADMIN, token scope admin
|
||||
POST /hosts/{id}/rule-cleanup/{op}/confirm operator, admin BROWSER SESSION ONLY, never a token
|
||||
POST /hosts/{id}/rule-cleanup/{op}/revert admin; operator only with operators_may_deploy
|
||||
POST /hosts/{id}/rule-cleanup/{op}/retry ADMIN, token scope admin
|
||||
POST /hosts/{id}/rule-cleanup/items/{item}/restore ADMIN, token scope admin
|
||||
GET /hosts/{id}/rule-cleanup, /{op} everybody signed in (read tokens too)
|
||||
|
||||
Who may do what is enforced twice: AuthMiddleware (services/api_token_service.py: _ADMIN_ONLY_CLEANUP,
|
||||
_SESSION_ONLY, _ROLLOUT_POST) and here, so a gap in one layer does not open the route. The audit trail is the
|
||||
service's (one ``rule_cleanup`` entry per step and rule) plus the middleware's request entry (kind ``host``,
|
||||
action = last path segment; ``/preview`` is not logged by it, the service writes its own entry).
|
||||
The services raise ``api_error``; nothing is translated here. MCP stays read-only (no tool, ``api_get`` is GET).
|
||||
"""
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser
|
||||
from app.routers import governance
|
||||
from app.routers.auth import resolve_user
|
||||
from app.schemas.rule_cleanup import CleanupOperationRead, CleanupPreviewRead, ConfirmName, PreviewRequest
|
||||
from app.services import rule_cleanup_service as service
|
||||
from app.services.rule_cleanup_service import ACK_TEXT_VERSION
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/rule-cleanup", tags=["rule-cleanup"])
|
||||
|
||||
|
||||
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
return host
|
||||
|
||||
|
||||
def _token(request: Request):
|
||||
return getattr(request.state, "api_token", None)
|
||||
|
||||
|
||||
def _writer(user: AdminUser, request: Request) -> None:
|
||||
"""Any write: not for a read-only account, not with a read-scope token."""
|
||||
if user.role == "read_only":
|
||||
raise api_error(403, ErrorCode.READ_ONLY_ACCESS)
|
||||
token = _token(request)
|
||||
if token is not None and token.scope == "read":
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
def _admin_only(user: AdminUser, request: Request) -> None:
|
||||
"""Deploy, retry, restore: an admin account, and with a token only one of scope admin."""
|
||||
_writer(user, request)
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
token = _token(request)
|
||||
if token is not None and token.scope != "admin":
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
async def _deployer(user: AdminUser, request: Request, db: AsyncSession) -> None:
|
||||
"""Revert: an admin, or an operator while the governance switch operators_may_deploy is on."""
|
||||
_writer(user, request)
|
||||
if user.role == "admin":
|
||||
return
|
||||
if user.role == "operator" and (await governance.get_settings(db)).operators_may_deploy:
|
||||
return
|
||||
raise api_error(403, ErrorCode.DEPLOY_NEEDS_ADMIN)
|
||||
|
||||
|
||||
def _session_only(user: AdminUser, request: Request) -> None:
|
||||
"""Confirm: "I still have access" is a human decision in a browser session, never a token's."""
|
||||
_writer(user, request)
|
||||
token = _token(request)
|
||||
if token is not None:
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
def _full_view(request: Request) -> bool:
|
||||
"""Admins (session or admin-scope token) see the management sources/ports and firewall addresses in a plan."""
|
||||
from app.routers.audit import full_view
|
||||
return full_view(request)
|
||||
|
||||
|
||||
async def _names(db: AsyncSession, operations: list[dict], request: Request) -> list[dict]:
|
||||
"""The firewall names the UI shows next to the ids; for non-admins without the security-relevant plan parts."""
|
||||
if not _full_view(request):
|
||||
operations = [service.redact(op) for op in operations]
|
||||
ids = {uuid.UUID(op["host_id"]) for op in operations} | {uuid.UUID(i["host_id"]) for op in operations for i in op["items"]}
|
||||
names = {h.id: h.name for h in (await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)))).all()} if ids else {}
|
||||
for op in operations:
|
||||
op["host_name"] = names.get(uuid.UUID(op["host_id"]))
|
||||
for item in op["items"]:
|
||||
item["host_name"] = names.get(uuid.UUID(item["host_id"]))
|
||||
return operations
|
||||
|
||||
|
||||
async def _one(db: AsyncSession, operation: dict, request: Request) -> dict:
|
||||
return (await _names(db, [operation], request))[0]
|
||||
|
||||
|
||||
@router.get("", response_model=list[CleanupOperationRead])
|
||||
async def list_operations(host_id: uuid.UUID, request: Request, limit: int = Query(50, ge=1, le=100),
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""The newest operations of this host (and its cluster peers) with their items; unexpired previews included."""
|
||||
host = await _host(db, host_id)
|
||||
return await _names(db, await service.list_operations(db, host, limit), request)
|
||||
|
||||
|
||||
@router.post("/preview", response_model=CleanupPreviewRead)
|
||||
async def preview(host_id: uuid.UUID, payload: PreviewRequest, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Recompute the plan from the live firewall(s) and store a 5-minute preview bound to the caller."""
|
||||
user = await resolve_user(request, db)
|
||||
_writer(user, request)
|
||||
host = await _host(db, host_id)
|
||||
kind, uuids, acknowledge = payload.resolved()
|
||||
if kind is None:
|
||||
raise api_error(422, ErrorCode.BEREINIGUNG_AUSWAHL_UNGUELTIG, field="kind")
|
||||
result = await service.preview(
|
||||
db, host, actor=user.username, actor_id=user.id, kind=kind, uuids=uuids, acknowledge=acknowledge,
|
||||
**({"unused_days": payload.unused_days} if payload.unused_days else {}),
|
||||
ack_text_version=payload.ack_text_version or ACK_TEXT_VERSION)
|
||||
result["operation"] = await _one(db, await service.get_operation(db, host, uuid.UUID(result["preview_id"])), request)
|
||||
return result if _full_view(request) else service.redact(result)
|
||||
|
||||
|
||||
@router.get("/{op_id}", response_model=CleanupOperationRead)
|
||||
async def get_operation(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.get_operation(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/cancel", response_model=CleanupOperationRead)
|
||||
async def cancel(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Drop an unused preview (only its own actor can)."""
|
||||
user = await resolve_user(request, db)
|
||||
_writer(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.cancel(db, host, op_id, actor=user.username), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/deploy", response_model=CleanupOperationRead)
|
||||
async def deploy(host_id: uuid.UUID, op_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""Execute a preview. Admin only; the live state must still give the fingerprint of the preview."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.deploy(db, host, op_id, actor=user.username, confirm_name=payload.confirm_name), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/confirm", response_model=CleanupOperationRead)
|
||||
async def confirm(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
""""I still have access": re-reads every node first. Browser session only, never with an API token."""
|
||||
user = await resolve_user(request, db)
|
||||
_session_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.confirm(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/revert", response_model=CleanupOperationRead)
|
||||
async def revert(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Take the change back now (the same code the sweeper runs when the window passes)."""
|
||||
user = await resolve_user(request, db)
|
||||
await _deployer(user, request, db)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.revert_now(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/retry", response_model=CleanupOperationRead)
|
||||
async def retry(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Revert again after revert_failed, or read the firewall again for a delete/restore with an unclear item."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.retry(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/items/{item_id}/restore", response_model=CleanupOperationRead)
|
||||
async def restore(host_id: uuid.UUID, item_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""Bring a deleted rule back, switched OFF, at its old position (a new UUID). The result is the restore run;
|
||||
``warnings`` can hold BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.restore_item(
|
||||
db, host, item_id, actor=user.username, actor_id=user.id, confirm_name=payload.confirm_name), request)
|
||||
@@ -1,3 +1,4 @@
|
||||
import json
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, Body
|
||||
@@ -18,6 +19,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.rule_deploy_service import confirm_rule, deploy_rule, undeploy_rule, revert_deployment
|
||||
from app.services import rule_adopt_service
|
||||
from app.services.adopt_common import LINK_ERRORS, link_api_error
|
||||
from app.schemas.rule import AdoptRequest, UnmanagedRule
|
||||
|
||||
router = APIRouter(prefix="/rules", tags=["rules"])
|
||||
@@ -50,6 +52,7 @@ def _to_read(
|
||||
return RuleRead(
|
||||
workflow_state=state,
|
||||
workflow_counts=counts, id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
direction=rule.direction,
|
||||
@@ -106,12 +109,19 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
|
||||
host = await db.get(OpnsenseHost, payload.host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
# Taking over a hand-made rule that a rule cleanup (1.31) is switching off/deleting would make it "managed"
|
||||
# under the cleanup's feet: not while one is open on this host or its cluster.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
try:
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid)
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
|
||||
except rule_adopt_service.RuleNotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.REGEL_NICHT_GEFUNDEN) from exc
|
||||
except rule_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
||||
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons),
|
||||
reason_details=json.dumps(exc.details, ensure_ascii=False)) from exc
|
||||
except LINK_ERRORS as exc:
|
||||
raise link_api_error(exc) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
@@ -271,6 +281,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
try:
|
||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_rule_uuid)
|
||||
await opnsense_client.apply(deployment.host)
|
||||
|
||||
@@ -24,6 +24,8 @@ class Schedule(BaseModel):
|
||||
acknowledge_unattended: Literal[True]
|
||||
# Hand the traffic over to the CARP partner first and switch back afterwards (cluster nodes only).
|
||||
carp_handover: bool = False
|
||||
# Check the result automatically afterwards and close the operation (no manual "Wartung abschließen").
|
||||
auto_verify: bool = False
|
||||
|
||||
|
||||
def _read(sched: ScheduledMaintenance) -> dict:
|
||||
@@ -32,6 +34,8 @@ def _read(sched: ScheduledMaintenance) -> dict:
|
||||
"status": sched.status, "actor": sched.actor, "created_at": sched.created_at,
|
||||
"result": sched.result, "host_operation_id": sched.host_operation_id,
|
||||
"carp_handover": sched.carp_handover, "phase": sched.phase,
|
||||
"auto_verify": sched.auto_verify, "origin": sched.origin,
|
||||
"rollout_run_id": ((sched.progress or {}).get("rollout") or {}).get("run_id"),
|
||||
}
|
||||
|
||||
|
||||
@@ -80,6 +84,7 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
|
||||
sched = ScheduledMaintenance(
|
||||
host_id=host.id, host_name=host.name, action=payload.action, run_at=run_at,
|
||||
actor=request.state.user.username, created_at=now, carp_handover=payload.carp_handover,
|
||||
auto_verify=payload.auto_verify,
|
||||
)
|
||||
db.add(sched)
|
||||
await db.commit()
|
||||
@@ -90,7 +95,9 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
|
||||
async def cancel_scheduled(host_id: uuid.UUID, schedule_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
await _host_or_404(db, host_id)
|
||||
sched = await db.get(ScheduledMaintenance, schedule_id)
|
||||
# Locked and re-read: the poll loop may be firing this very row right now (it holds the lock meanwhile).
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == schedule_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if not sched or sched.host_id != host_id:
|
||||
raise api_error(404, ErrorCode.GEPLANTE_WARTUNG_NICHT_GEFUNDEN)
|
||||
if sched.status != "pending":
|
||||
|
||||
@@ -7,6 +7,7 @@ from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.services import rollout_engine
|
||||
from app.services import update_check_service, updater_client
|
||||
|
||||
router = APIRouter(prefix="/system", tags=["system"])
|
||||
@@ -99,6 +100,9 @@ async def updater_status(db: AsyncSession = Depends(get_db), user: AdminUser = D
|
||||
"can_update": False, "major_change": False,
|
||||
"job": None, "phases": [], "log": [], "services": {},
|
||||
"rollback_available": False, "rollback_loses_data": False, "rollback_version": None,
|
||||
# A restart in the middle of a firmware rollout is safe (its state is in the database), but the
|
||||
# UI warns: the run pauses its 15-second steps for as long as OPNexus is down.
|
||||
"active_rollouts": await rollout_engine.count_active(db),
|
||||
}
|
||||
try:
|
||||
updater = await updater_client.status()
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
@@ -21,10 +21,29 @@ async def _admin_count(db: AsyncSession, exclude_id: uuid.UUID | None = None) ->
|
||||
return await db.scalar(query)
|
||||
|
||||
|
||||
_ADMIN_LOCK_KEY = 728194032 # next to policy_lock's key; only admin role changes and deletions take it
|
||||
|
||||
|
||||
async def _lock_admins(db: AsyncSession, current: AdminUser) -> None:
|
||||
"""Serialize every change that can reduce the number of administrators and re-read the caller under the lock.
|
||||
|
||||
The "at least one administrator" check counts rows; two requests that each see the other's admin row would both
|
||||
pass it. Held until commit/rollback (transaction-scoped advisory lock). The caller may have been demoted or
|
||||
deleted by the request that ran first, so it must still be an admin to proceed."""
|
||||
await db.execute(text("SELECT pg_advisory_xact_lock(:key)"), {"key": _ADMIN_LOCK_KEY})
|
||||
fresh = await db.scalar(select(AdminUser).where(AdminUser.id == current.id).execution_options(populate_existing=True))
|
||||
if fresh is None or fresh.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
|
||||
|
||||
@router.get("", response_model=list[UserRead])
|
||||
async def list_users(db: AsyncSession = Depends(get_db)):
|
||||
rows = (await db.scalars(select(AdminUser).order_by(AdminUser.username))).all()
|
||||
return rows
|
||||
async def list_users(user: AdminUser = Depends(current_user), db: AsyncSession = Depends(get_db)):
|
||||
"""Admins see everyone. Every other role sees only its own account: the full list (names, roles, last login,
|
||||
2FA status) is a directory an attacker with any low-privilege account or token could use to pick targets."""
|
||||
query = select(AdminUser).order_by(AdminUser.username)
|
||||
if user.role != "admin":
|
||||
query = query.where(AdminUser.id == user.id)
|
||||
return (await db.scalars(query)).all()
|
||||
|
||||
|
||||
@router.post("", response_model=UserRead, status_code=201)
|
||||
@@ -43,8 +62,12 @@ async def create_user(payload: UserCreate, db: AsyncSession = Depends(get_db)):
|
||||
|
||||
|
||||
@router.patch("/{user_id}/role", response_model=UserRead)
|
||||
async def update_role(user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db)):
|
||||
target = await db.get(AdminUser, user_id)
|
||||
async def update_role(
|
||||
user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db),
|
||||
current: AdminUser = Depends(current_user),
|
||||
):
|
||||
await _lock_admins(db, current)
|
||||
target = await db.get(AdminUser, user_id, populate_existing=True)
|
||||
if target is None:
|
||||
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
||||
if target.auth_source == "oidc":
|
||||
@@ -96,13 +119,13 @@ async def delete_user(
|
||||
):
|
||||
if user_id == current.id:
|
||||
raise api_error(400, ErrorCode.DER_EIGENE_ACCOUNT_KANN_NICHT_GELOESCHT_WERDEN)
|
||||
target = await db.get(AdminUser, user_id)
|
||||
await _lock_admins(db, current)
|
||||
target = await db.get(AdminUser, user_id, populate_existing=True)
|
||||
if target is None:
|
||||
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
||||
# No separate "last admin" guard here: only an admin can reach this
|
||||
# endpoint (read_only is rejected earlier, in AuthMiddleware) and the
|
||||
# self-delete check above already guarantees target != current — so
|
||||
# `current` is always a still-existing admin afterward, meaning the
|
||||
# admin count can never reach zero this way.
|
||||
# Two admins deleting each other (or demoting/deleting in parallel) would each still see the other: the lock above
|
||||
# orders them, and the guard below counts what is left without the target.
|
||||
if target.role == "admin" and await _admin_count(db, exclude_id=target.id) == 0:
|
||||
raise api_error(409, ErrorCode.MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN)
|
||||
await db.delete(target)
|
||||
await db.commit()
|
||||
@@ -18,6 +18,7 @@ from app.services import wireguard_key_store
|
||||
from app.services.wireguard_service import effective_hosts, preview, revision, snapshot, workflow, WIREGUARD_FIELDS
|
||||
from app.services.wireguard_deploy_service import deploy_tunnel, undeploy_tunnel
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_lease import resource_lease
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/wireguard/tunnels", tags=["wireguard"])
|
||||
@@ -206,18 +207,21 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_peer_uuid:
|
||||
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
|
||||
if deployment.opnsense_server_uuid:
|
||||
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
|
||||
await opnsense_client.reconfigure_wireguard(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
async with resource_lease("wireguard", tunnel.id):
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_peer_uuid:
|
||||
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
|
||||
if deployment.opnsense_server_uuid:
|
||||
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
|
||||
await opnsense_client.reconfigure_wireguard(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
@@ -0,0 +1,76 @@
|
||||
import uuid
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AdoptObjectRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link to this existing central object instead of creating a new one (services/adopt_link_service.py).
|
||||
# Declared before `uuid`: that field's name shadows the module inside the class body.
|
||||
link_to: uuid.UUID | None = None
|
||||
uuid: str = Field(min_length=1, max_length=64) # the object's UUID on the firewall
|
||||
|
||||
|
||||
class AdoptAllRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link entries that have exactly one exact central twin instead of creating a duplicate.
|
||||
link_existing: bool = False
|
||||
|
||||
|
||||
class LinkDiff(BaseModel):
|
||||
field: str # model field ("content", "destination_port", ...) or "variables" (a {{VARIABLE}} has no value on this host)
|
||||
central: str
|
||||
live: str
|
||||
|
||||
|
||||
class LinkTarget(BaseModel):
|
||||
"""An existing central object that a live object could be linked to (exact match) or is similar to."""
|
||||
|
||||
id: uuid.UUID
|
||||
name: str
|
||||
targets: list[str] # names of the hosts it is deployed to
|
||||
diff: list[LinkDiff] = Field(default_factory=list) # empty for an exact match
|
||||
|
||||
|
||||
class UnmanagedObject(BaseModel):
|
||||
"""An alias / NAT rule found on a host that OPNexus does not manage yet (see object_adopt_service.py)."""
|
||||
|
||||
uuid: str
|
||||
title: str
|
||||
enabled: bool
|
||||
adoptable: bool # can be adopted on its own right now
|
||||
# Adoptable in the same "adopt all" run although not on its own (an alias whose referenced aliases come first).
|
||||
bulk_adoptable: bool
|
||||
# Codes such as "unsupported_type" or "no_target"; the UI turns them into text.
|
||||
reasons: list[str]
|
||||
reason_details: dict[str, str] = Field(default_factory=dict) # code -> extra text (alias names, type, ...)
|
||||
# Settings OPNexus does not manage that stay untouched on the firewall (log, pool options, tags, ...).
|
||||
kept: list[str] = Field(default_factory=list)
|
||||
# Facts for display, depend on the object type.
|
||||
info: dict[str, str] = Field(default_factory=dict)
|
||||
# Central objects whose content, resolved for this host, equals the live object exactly (first = preferred).
|
||||
matches: list[LinkTarget] = Field(default_factory=list)
|
||||
# Objects with the same name/description but different content: a hint with the differing fields, never linked.
|
||||
similar: list[LinkTarget] = Field(default_factory=list)
|
||||
linkable: bool = False # can be linked to `matches[0]` (or another match) on its own
|
||||
# What "adopt all" would do: "link" | "new" | "skip" (None only on the adopt-all result).
|
||||
bulk_action: str | None = None
|
||||
|
||||
|
||||
class AdoptedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
id: uuid.UUID
|
||||
linked: bool = False # linked to an existing central object instead of newly created
|
||||
|
||||
|
||||
class SkippedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
reasons: list[str]
|
||||
reason_details: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AdoptAllResult(BaseModel):
|
||||
adopted: list[AdoptedObject]
|
||||
skipped: list[SkippedObject]
|
||||
@@ -12,7 +12,9 @@ CertAction = Literal["import", "internal", "external", "sign_csr", "import_csr",
|
||||
# EC curves) — see ManagedCertificate's docstring.
|
||||
CertKeyType = Literal["2048", "3072", "4096", "prime256v1", "secp384r1", "secp521r1"]
|
||||
CertDigest = Literal["sha1", "sha224", "sha256", "sha384", "sha512"]
|
||||
CertType = Literal["usr_cert", "server_cert", "combined_server_client", "v3_ca"]
|
||||
# No "v3_ca": OPNsense would create a certificate with the CA extension, not a CA that can be used as `caref`
|
||||
# (found by the 1.31.0 test run, T3). Creating a CA is not offered; the UI never did.
|
||||
CertType = Literal["usr_cert", "server_cert", "combined_server_client"]
|
||||
|
||||
_DN_ACTIONS = {"internal", "external", "reissue"}
|
||||
_CRT_ACTIONS = {"import", "import_csr", "manual"}
|
||||
|
||||
@@ -3,6 +3,8 @@ from datetime import datetime
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.schemas.gateway import GatewayHealth
|
||||
|
||||
|
||||
class DashboardHost(BaseModel):
|
||||
id: uuid.UUID
|
||||
@@ -10,6 +12,11 @@ class DashboardHost(BaseModel):
|
||||
status: str
|
||||
carp_status: str | None
|
||||
cluster_name: str | None
|
||||
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
|
||||
gateways: list[GatewayHealth] | None = None
|
||||
# Readings the newest fresh sample could not take because the API key lacks the right (gateways, services, ...);
|
||||
# None = no fresh sample (or one from before this existed), [] = nothing missing.
|
||||
api_rights_missing: list[str] | None = None
|
||||
|
||||
|
||||
class PendingAction(BaseModel):
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
# online: all fine · degraded: OPNsense flags loss and/or latency above the gateway's own limits ·
|
||||
# offline: down · forced_down: switched off by the admin on purpose · unknown: nothing measured
|
||||
# (monitoring disabled) or the verdict is still pending.
|
||||
GatewayState = Literal["online", "degraded", "offline", "forced_down", "unknown"]
|
||||
|
||||
|
||||
class GatewayHealth(BaseModel):
|
||||
name: str
|
||||
address: str | None = None
|
||||
status: str # OPNsense' raw verdict: none | down | force_down | delay | loss | delay+loss | …
|
||||
state: GatewayState
|
||||
loss: float | None = None # percent
|
||||
delay: float | None = None # milliseconds
|
||||
stddev: float | None = None # milliseconds
|
||||
monitor: str | None = None # the monitored address
|
||||
alert_enabled: bool = True # False: the gateway alert is switched off for this gateway (Settings, 1.29)
|
||||
|
||||
|
||||
class GatewayAlertUpdate(BaseModel):
|
||||
enabled: bool
|
||||
|
||||
|
||||
class GatewayAlertState(BaseModel):
|
||||
name: str
|
||||
alert_enabled: bool
|
||||
|
||||
|
||||
class ApiRightsState(BaseModel):
|
||||
missing: list[str] | None = None # names of the refused readings (gateways, services, swap, cpu); None = unknown
|
||||
@@ -18,6 +18,9 @@ class HostCreate(BaseModel):
|
||||
enabled: bool = True
|
||||
cluster_name: str | None = Field(default=None, max_length=120)
|
||||
policy_hierarchy_group_id: uuid.UUID | None = None
|
||||
# Ask the firewall once with these credentials before saving (the UI and the setup assistant do). Off by default so
|
||||
# scripts can still register a firewall that is not reachable yet (behind NAT, tunnel not up).
|
||||
verify_connection: bool = False
|
||||
|
||||
@field_validator("base_url")
|
||||
@classmethod
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST
|
||||
|
||||
SmtpSecurity = Literal["none", "starttls", "ssl"]
|
||||
ProblemType = Literal[
|
||||
@@ -14,7 +16,10 @@ ProblemType = Literal[
|
||||
"backup_stale",
|
||||
"backup_offsite_failed",
|
||||
"compliance_finding",
|
||||
"gateway_down", "resource_threshold", "service_down", "api_rights_missing",
|
||||
"rollout_paused", "rollout_start_failed",
|
||||
]
|
||||
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
||||
|
||||
|
||||
class NotificationSettingsRead(BaseModel):
|
||||
@@ -45,10 +50,30 @@ class NotificationSettingsRead(BaseModel):
|
||||
scheduled_maintenance_failed_enabled: bool = True
|
||||
backup_failed_enabled: bool = True
|
||||
compliance_alerts_enabled: bool = False
|
||||
gateway_down_enabled: bool = True
|
||||
resource_threshold_enabled: bool = False
|
||||
service_down_enabled: bool = False
|
||||
rollout_problems_enabled: bool = True
|
||||
api_rights_enabled: bool = True
|
||||
gateway_alert_muted: list[str] = Field(default_factory=list)
|
||||
gateway_loss_threshold_pct: int = 20
|
||||
gateway_delay_threshold_ms: int = 500
|
||||
disk_threshold_pct: int = 90
|
||||
memory_threshold_pct: int = 95
|
||||
swap_threshold_pct: int = 50
|
||||
load_per_cpu_threshold: float = 2.0
|
||||
alert_confirm_samples: int = 3
|
||||
resource_confirm_samples: int = 5
|
||||
service_watchlist: list[str] = Field(default_factory=lambda: list(DEFAULT_SERVICE_WATCHLIST))
|
||||
|
||||
poll_interval_seconds: int
|
||||
updated_at: datetime
|
||||
|
||||
@field_validator("service_watchlist", mode="before")
|
||||
@classmethod
|
||||
def _default_watchlist(cls, value):
|
||||
return list(DEFAULT_SERVICE_WATCHLIST) if value is None else value
|
||||
|
||||
|
||||
class NotificationSettingsUpdate(BaseModel):
|
||||
"""Partial update — everything optional, only provided fields change.
|
||||
@@ -84,6 +109,35 @@ class NotificationSettingsUpdate(BaseModel):
|
||||
scheduled_maintenance_failed_enabled: bool | None = None
|
||||
backup_failed_enabled: bool | None = None
|
||||
compliance_alerts_enabled: bool | None = None
|
||||
gateway_down_enabled: bool | None = None
|
||||
resource_threshold_enabled: bool | None = None
|
||||
service_down_enabled: bool | None = None
|
||||
rollout_problems_enabled: bool | None = None
|
||||
api_rights_enabled: bool | None = None
|
||||
# 0 switches that single threshold off.
|
||||
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
||||
disk_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
memory_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
swap_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
load_per_cpu_threshold: float | None = Field(default=None, ge=0, le=1000)
|
||||
alert_confirm_samples: int | None = Field(default=None, ge=1, le=60)
|
||||
resource_confirm_samples: int | None = Field(default=None, ge=1, le=60)
|
||||
service_watchlist: list[str] | None = Field(default=None, max_length=100)
|
||||
|
||||
@field_validator("service_watchlist")
|
||||
@classmethod
|
||||
def _clean_watchlist(cls, value):
|
||||
if value is None:
|
||||
return None
|
||||
names = []
|
||||
for raw in value:
|
||||
name = str(raw).strip()
|
||||
if not re.fullmatch(r"[A-Za-z0-9._+-]{1,64}", name):
|
||||
raise ValueError("Dienstnamen bestehen aus Buchstaben, Ziffern und . _ + -")
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
poll_interval_seconds: int | None = Field(
|
||||
default=None, ge=settings.notification_poll_min_interval_seconds
|
||||
@@ -106,3 +160,55 @@ class NotificationProblemRead(BaseModel):
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
channel: Literal["webhook", "email"]
|
||||
|
||||
|
||||
class NotificationChannelRead(BaseModel):
|
||||
"""Never carries the credential; ``config`` is withheld from non-admins (an ntfy topic is a password)."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
name: str
|
||||
channel_type: ChannelType
|
||||
enabled: bool
|
||||
config: dict
|
||||
secret_set: bool
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
def _channel_name(value: str | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if not value.strip():
|
||||
raise ValueError("Name darf nicht leer sein")
|
||||
return value.strip()
|
||||
|
||||
|
||||
class NotificationChannelCreate(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=100)
|
||||
channel_type: ChannelType
|
||||
enabled: bool = True
|
||||
config: dict = Field(default_factory=dict)
|
||||
# Token, or for Teams the webhook address. Write-only.
|
||||
secret: str | None = Field(default=None, max_length=2100)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def _clean_name(cls, value):
|
||||
return _channel_name(value)
|
||||
|
||||
|
||||
class NotificationChannelUpdate(BaseModel):
|
||||
"""Partial: omitted fields stay. ``secret`` replaces the credential, ``clear_secret`` removes it."""
|
||||
|
||||
name: str | None = Field(default=None, min_length=1, max_length=100)
|
||||
enabled: bool | None = None
|
||||
config: dict | None = None
|
||||
secret: str | None = Field(default=None, max_length=2100)
|
||||
clear_secret: bool = False
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def _clean_name(cls, value):
|
||||
return _channel_name(value)
|
||||
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
hierarchy_group_id: uuid.UUID
|
||||
rulebase: Literal["pre", "post"]
|
||||
sequence: int
|
||||
|
||||
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
ipprotocol: str
|
||||
|
||||
@@ -4,6 +4,8 @@ from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from app.schemas.adopt import LinkTarget
|
||||
|
||||
|
||||
class RuleCreate(BaseModel):
|
||||
description: str = Field(min_length=1, max_length=255)
|
||||
@@ -46,6 +48,7 @@ class RuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
direction: str
|
||||
@@ -78,6 +81,8 @@ class DeployApproval(BaseModel):
|
||||
|
||||
class AdoptRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
# Link to this existing central rule instead of creating a new one (services/adopt_link_service.py).
|
||||
link_to: uuid.UUID | None = None
|
||||
rule_uuid: str = Field(min_length=1, max_length=64)
|
||||
|
||||
|
||||
@@ -98,3 +103,7 @@ class UnmanagedRule(BaseModel):
|
||||
adoptable: bool
|
||||
# Codes such as "negated_source" or "not_quick"; the UI turns them into text.
|
||||
reasons: list[str]
|
||||
# Central rules equal to this one (resolved for the host), same-description rules that differ, link possible.
|
||||
matches: list[LinkTarget] = Field(default_factory=list)
|
||||
similar: list[LinkTarget] = Field(default_factory=list)
|
||||
linkable: bool = False
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Request and response shapes of the rule-cleanup API (routers/rule_cleanup.py, DESIGN-1.31 §4).
|
||||
|
||||
Responses mirror services/rule_cleanup_service.read_operation. Texts are not part of the API: ``message`` holds an
|
||||
ErrorCode name (``errors.<CODE>`` in the UI, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR) or the firewall's
|
||||
own wording; ``warnings`` is a list of ErrorCode names.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
CleanupKind = Literal["disable", "enable", "delete"]
|
||||
# Longest list the API accepts at all; the effective limit is the setting cleanup_max_rules (ceiling 25).
|
||||
MAX_SELECTION = 100
|
||||
|
||||
|
||||
class PreviewItem(BaseModel):
|
||||
"""One rule of the selection. ``kind`` (alias ``mode``) must be the same for all items of one preview."""
|
||||
uuid: str = Field(min_length=1, max_length=64)
|
||||
kind: CleanupKind | None = None
|
||||
mode: CleanupKind | None = None
|
||||
# The per-rule acknowledgement for a "probable" rule (the UI's checkbox).
|
||||
ack: bool = False
|
||||
|
||||
|
||||
class PreviewRequest(BaseModel):
|
||||
"""``{kind, uuids, acknowledge}`` or ``{items:[{uuid, kind|mode, ack}]}`` (or both: they are merged)."""
|
||||
kind: CleanupKind | None = None
|
||||
uuids: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
items: list[PreviewItem] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
acknowledge: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
unused_days: int | None = Field(default=None, ge=1, le=3650)
|
||||
ack_text_version: str | None = Field(default=None, max_length=16)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _single_kind(self):
|
||||
kinds = {k for k in [self.kind, *[i.kind or i.mode for i in self.items]] if k}
|
||||
if len(kinds) > 1:
|
||||
raise ValueError("kind must be the same for every item")
|
||||
return self
|
||||
|
||||
def resolved(self) -> tuple[str | None, list[str], list[str]]:
|
||||
kind = self.kind or next((i.kind or i.mode for i in self.items if i.kind or i.mode), None)
|
||||
uuids = [*self.uuids, *[i.uuid for i in self.items]]
|
||||
acknowledge = [*self.acknowledge, *[i.uuid for i in self.items if i.ack]]
|
||||
return kind, list(dict.fromkeys(u.strip() for u in uuids)), list(dict.fromkeys(a.strip() for a in acknowledge))
|
||||
|
||||
|
||||
class ConfirmName(BaseModel):
|
||||
"""Deploy and restore: the exact host name, typed by a human."""
|
||||
confirm_name: str = Field(max_length=255)
|
||||
|
||||
|
||||
class CleanupItemRead(BaseModel):
|
||||
id: str
|
||||
host_id: str
|
||||
host_name: str | None = None
|
||||
rule_uuid: str
|
||||
seq: int
|
||||
carp_role: str | None = None
|
||||
# queued, disabling, disabled, confirmed, reverting, reverted, deleting, deleted, restoring, restored, failed,
|
||||
# unknown, revert_failed, skipped (never attempted: the run stopped earlier)
|
||||
state: str
|
||||
description: str | None = None
|
||||
categories: Any = None
|
||||
confidence: str | None = None
|
||||
cover_uuid: str | None = None
|
||||
prio_group: str | None = None
|
||||
anchor_next_uuid: str | None = None
|
||||
anchor_prev_uuid: str | None = None
|
||||
source_item_id: str | None = None
|
||||
restored_uuid: str | None = None
|
||||
changed_at: str | None = None
|
||||
message: str | None = None
|
||||
has_snapshot: bool = False
|
||||
# A deleted rule with a stored snapshot that has not been restored yet (POST items/{id}/restore, admin).
|
||||
restorable: bool = False
|
||||
|
||||
|
||||
class CleanupOperationRead(BaseModel):
|
||||
id: str
|
||||
host_id: str
|
||||
host_name: str | None = None
|
||||
cluster_name: str | None = None
|
||||
kind: str # disable | enable | delete | restore
|
||||
# preview, running, awaiting_confirm, confirmed, partial, apply_failed, reverting, reverted, revert_failed, done,
|
||||
# failed, cancelled, expired
|
||||
status: str
|
||||
actor: str
|
||||
created_at: str | None = None
|
||||
expires_at: str | None = None
|
||||
started_at: str | None = None
|
||||
heartbeat_at: str | None = None
|
||||
revert_deadline: str | None = None
|
||||
confirmed_at: str | None = None
|
||||
finished_at: str | None = None
|
||||
message: str | None = None
|
||||
options: dict[str, Any] = Field(default_factory=dict)
|
||||
# Only while status == "preview".
|
||||
plan: dict[str, Any] | None = None
|
||||
items: list[CleanupItemRead] = Field(default_factory=list)
|
||||
can_confirm: bool = False
|
||||
can_revert: bool = False
|
||||
can_retry: bool = False
|
||||
# ErrorCode names, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR (restore only).
|
||||
warnings: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class CleanupPreviewRead(BaseModel):
|
||||
preview_id: str
|
||||
expires_at: str | None = None
|
||||
kind: str
|
||||
plan: dict[str, Any]
|
||||
max_rules: int
|
||||
window_seconds: int
|
||||
operation: CleanupOperationRead
|
||||
@@ -7,9 +7,19 @@ how the proxy setup works for IP *and* name access) nor an explicitly allowed
|
||||
origin. Requests without either header (curl, scripts) are not browsers, so
|
||||
they cannot be riding on a victim's cookie and pass through to normal auth.
|
||||
|
||||
Sec-Fetch-Site (1.27): a page on another port of the same host (e.g. a
|
||||
firewall GUI behind the GUI gateway) is *same-site* to OPNexus, so the
|
||||
browser attaches the session cookie to its requests and SameSite=Lax does
|
||||
not help. Cookie-carrying /api requests that the browser itself marks as
|
||||
same-site or cross-site are therefore refused — unless the Origin is an
|
||||
explicitly configured frontend origin (development setup with the frontend
|
||||
on its own port). Clients that send no Sec-Fetch-Site header (curl,
|
||||
scripts, API tokens) are not browsers and stay unaffected.
|
||||
|
||||
SecurityHeadersMiddleware: API responses must never be cached by browsers or
|
||||
proxies and must not be sniffed or framed.
|
||||
"""
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
@@ -18,6 +28,7 @@ from starlette.responses import JSONResponse
|
||||
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode
|
||||
from app.routers.auth import has_session_cookie
|
||||
|
||||
_SAFE_METHODS = {"GET", "HEAD", "OPTIONS"}
|
||||
|
||||
@@ -35,8 +46,53 @@ def _origin_allowed(origin: str, host_header: str) -> bool:
|
||||
return f"{parsed.scheme}://{parsed.netloc}".lower() in {o.lower() for o in settings.allowed_origins}
|
||||
|
||||
|
||||
# The SSO provider sends the browser back here by a cross-site navigation;
|
||||
# the endpoint authenticates by its own state cookie and creates the session.
|
||||
# The exemption is only for exactly that: a top-level GET navigation (a script, image or fetch from another site
|
||||
# is still blocked, whatever it carries) — and the callback fails without the matching state cookie anyway.
|
||||
_SEC_FETCH_EXEMPT = {"/api/auth/oidc/callback"}
|
||||
|
||||
|
||||
def _is_top_level_navigation(request: Request) -> bool:
|
||||
if request.method != "GET":
|
||||
return False
|
||||
mode = request.headers.get("sec-fetch-mode", "").strip().lower()
|
||||
dest = request.headers.get("sec-fetch-dest", "").strip().lower()
|
||||
return mode in ("", "navigate") and dest in ("", "document")
|
||||
|
||||
|
||||
def _explicit_frontend_origin(origin: str) -> bool:
|
||||
origin = origin.rstrip("/").lower()
|
||||
if not origin or origin == "null":
|
||||
return False
|
||||
if settings.allowed_origins:
|
||||
return origin in {o.lower() for o in settings.allowed_origins}
|
||||
return re.fullmatch(settings.cors_origin_regex, origin) is not None
|
||||
|
||||
|
||||
def sec_fetch_blocked(request: Request) -> bool:
|
||||
path = request.url.path
|
||||
if not (path == "/api" or path.startswith("/api/")):
|
||||
return False
|
||||
if path in _SEC_FETCH_EXEMPT and _is_top_level_navigation(request):
|
||||
return False
|
||||
if not has_session_cookie(request):
|
||||
return False
|
||||
site = request.headers.get("sec-fetch-site", "").strip().lower()
|
||||
if site == "cross-site":
|
||||
return True
|
||||
if site == "same-site":
|
||||
return not _explicit_frontend_origin(request.headers.get("origin", ""))
|
||||
return False # same-origin, none (typed URL/bookmark) or no header (not a browser)
|
||||
|
||||
|
||||
class OriginCheckMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if sec_fetch_blocked(request):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": {"code": ErrorCode.CROSS_SITE_REQUEST_BLOCKED.value, "params": {}}},
|
||||
)
|
||||
if request.method not in _SAFE_METHODS:
|
||||
source = request.headers.get("origin") or request.headers.get("referer")
|
||||
if source and source != "null" and not _origin_allowed(source, request.headers.get("host", "")):
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Small pieces shared by every "take over what was made by hand on a firewall" feature
|
||||
(rule_adopt_service.py for filter rules, object_adopt_service.py for aliases and NAT)."""
|
||||
|
||||
|
||||
class NotAdoptable(Exception):
|
||||
"""The object exists on the host but cannot be described without distortion.
|
||||
|
||||
`reasons` are machine-readable codes (the UI translates them); `details` maps a code to
|
||||
extra text the translation can show (e.g. the alias names a reference points to)."""
|
||||
|
||||
def __init__(self, reasons: list[str], details: dict[str, str] | None = None):
|
||||
super().__init__(", ".join(reasons))
|
||||
self.reasons = reasons
|
||||
self.details = details or {}
|
||||
|
||||
|
||||
class NotFoundOnHost(Exception):
|
||||
"""The UUID is not (or no longer) present in the host's live configuration."""
|
||||
|
||||
|
||||
def on(value) -> bool:
|
||||
"""OPNsense booleans arrive as "1"/"0" strings, sometimes as real booleans."""
|
||||
return value in ("1", 1, True)
|
||||
|
||||
|
||||
def clean(value) -> str:
|
||||
return "" if value is None else str(value).strip()
|
||||
|
||||
|
||||
class LinkTargetNotFound(Exception):
|
||||
"""The central object chosen to link to was deleted in the meantime."""
|
||||
|
||||
|
||||
class LinkTargetChanged(Exception):
|
||||
"""The central object no longer equals what is live on the host. `diff` lists the differing fields."""
|
||||
|
||||
def __init__(self, diff: list[dict]):
|
||||
super().__init__(", ".join(d["field"] for d in diff))
|
||||
self.diff = diff
|
||||
|
||||
|
||||
class HostAlreadyHasObject(Exception):
|
||||
"""The central object is already deployed to / managed on this host under another UUID."""
|
||||
|
||||
|
||||
class LinkTargetBusy(Exception):
|
||||
"""The central object has an unconfirmed deployment or a failed rollback; linking waits until it is settled."""
|
||||
|
||||
|
||||
LINK_ERRORS = (LinkTargetNotFound, LinkTargetChanged, HostAlreadyHasObject, LinkTargetBusy)
|
||||
|
||||
|
||||
def link_api_error(exc: Exception):
|
||||
"""The HTTP error for a failed link (shared by the rule and the object endpoints)."""
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
if isinstance(exc, LinkTargetNotFound):
|
||||
return api_error(404, ErrorCode.ADOPT_LINK_TARGET_NOT_FOUND)
|
||||
if isinstance(exc, LinkTargetChanged):
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_TARGET_CHANGED, fields=", ".join(d["field"] for d in exc.diff))
|
||||
if isinstance(exc, HostAlreadyHasObject):
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_HOST_HAS_OBJECT)
|
||||
return api_error(409, ErrorCode.ADOPT_LINK_BUSY)
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Links a hand-made object that was found on a firewall to a central object that already exists, instead
|
||||
of creating a second central object for it ("Mit vorhandenem zentralem Objekt verknüpfen").
|
||||
|
||||
Typical case: the same rule/alias/NAT entry was made by hand on several firewalls (or on both nodes of an HA
|
||||
pair). The first adoption creates the central object; every further firewall is LINKED to it: the host
|
||||
becomes another target of that object and gets a deployment row holding the live UUID (remote_uuid) and
|
||||
the payload OPNexus would send. Like adopting, linking writes NOTHING to the firewall. The live state is
|
||||
read fresh from the host at link time (the browser only names the central object), so the preview right
|
||||
afterwards says "unchanged" and a later rollout updates the one object on every host without a duplicate.
|
||||
|
||||
A central object is an EXACT match when every field OPNexus manages is equal after the object's
|
||||
{{VARIABLE}} placeholders are replaced with this host's values (variables_service.resolve). An object with
|
||||
the same identity (alias name / rule description) but different content is only shown as a hint with the
|
||||
differing fields and is NEVER linked automatically.
|
||||
|
||||
Callers hold policy_lock (same serialisation as adopting), which also makes a repeated link idempotent.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Callable
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.services import variables_service
|
||||
from app.services.adopt_common import HostAlreadyHasObject, LinkTargetBusy, LinkTargetChanged, LinkTargetNotFound
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import effective_hosts
|
||||
|
||||
LINK_MESSAGE = "Mit vorhandenem Objekt verknüpft"
|
||||
BUSY_STATUSES = ("pending", "revert_failed") # the central object waits for a confirmation / failed to roll back
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LinkSpec:
|
||||
model: type
|
||||
deployment: type
|
||||
fk: str # deployment column pointing at the central object
|
||||
uuid_attr: str # deployment column holding the OPNsense UUID (remote_uuid)
|
||||
identity: str # attribute naming the object: "name" (alias) or "description"
|
||||
payload: Callable[[Any, OpnsenseHost], dict]
|
||||
revision: Callable
|
||||
|
||||
|
||||
@dataclass
|
||||
class Match:
|
||||
obj: Any
|
||||
diff: list[dict] # empty = exact match
|
||||
busy: bool = False # the central object waits for a confirmation: linking is refused until that is settled
|
||||
|
||||
|
||||
def _norm(value) -> Any:
|
||||
if value is None:
|
||||
return ""
|
||||
return value if isinstance(value, bool) else str(value)
|
||||
|
||||
|
||||
def _show(value) -> str:
|
||||
return ("true" if value else "false") if isinstance(value, bool) else ("" if value is None else str(value))
|
||||
|
||||
|
||||
def diff_fields(obj, values: dict, host: OpnsenseHost) -> list[dict]:
|
||||
"""Fields in which the central object (resolved for `host`) differs from the live `values`."""
|
||||
raw = {key: getattr(obj, key) for key in values}
|
||||
try:
|
||||
central = variables_service.resolve(raw, host)
|
||||
except variables_service.MissingVariables as exc:
|
||||
# Cannot be compared (and could not be deployed to this host either): report, never match.
|
||||
return [{"field": "variables", "central": ", ".join(exc.missing[host.name]), "live": ""}]
|
||||
return [{"field": key, "central": _show(central[key]), "live": _show(values[key])}
|
||||
for key in values if _norm(central[key]) != _norm(values[key])]
|
||||
|
||||
|
||||
def _options(spec: LinkSpec):
|
||||
return [
|
||||
selectinload(spec.model.targets),
|
||||
selectinload(spec.model.groups).selectinload(DeviceGroup.hosts),
|
||||
selectinload(spec.model.deployments),
|
||||
]
|
||||
|
||||
|
||||
async def load_central(db: AsyncSession, spec: LinkSpec) -> list:
|
||||
objs = list((await db.scalars(select(spec.model).options(*_options(spec)))).unique().all())
|
||||
await hydrate_hierarchy_targets(db, objs)
|
||||
return objs
|
||||
|
||||
|
||||
def find(objs: list, spec: LinkSpec, host: OpnsenseHost, values: dict) -> tuple[list[Match], list[Match]]:
|
||||
"""(exact matches, same-identity objects that differ). Objects that already have a deployment on
|
||||
this host are not candidates: linking would collide with that deployment."""
|
||||
exact: list[Match] = []
|
||||
similar: list[Match] = []
|
||||
for obj in objs:
|
||||
if any(d.host_id == host.id for d in obj.deployments):
|
||||
continue
|
||||
if _norm(getattr(obj, spec.identity)) != _norm(values[spec.identity]):
|
||||
continue
|
||||
d = diff_fields(obj, values, host)
|
||||
busy = any(dep.status in BUSY_STATUSES for dep in obj.deployments)
|
||||
(similar if d else exact).append(Match(obj, d, busy))
|
||||
# prefer the object that already reaches the most hosts (the "real" one), then a stable order
|
||||
exact.sort(key=lambda m: (-len(effective_hosts(m.obj)), str(m.obj.id)))
|
||||
return exact, similar
|
||||
|
||||
|
||||
def describe(spec: LinkSpec, match: Match) -> dict:
|
||||
return {
|
||||
"id": match.obj.id,
|
||||
"name": getattr(match.obj, spec.identity),
|
||||
"targets": [h.name for h in effective_hosts(match.obj)],
|
||||
"diff": match.diff,
|
||||
}
|
||||
|
||||
|
||||
async def link(db: AsyncSession, spec: LinkSpec, host: OpnsenseHost, object_id, remote_uuid: str, values: dict):
|
||||
"""Links the live object `remote_uuid` (whose normalised fields are `values`, read fresh by the caller)
|
||||
to the central object `object_id`. Returns it; a repeated call for the same pair changes nothing."""
|
||||
obj = (await db.execute(
|
||||
select(spec.model).where(spec.model.id == object_id).options(*_options(spec))
|
||||
.execution_options(populate_existing=True)
|
||||
)).unique().scalar_one_or_none()
|
||||
if obj is None:
|
||||
raise LinkTargetNotFound(str(object_id))
|
||||
await hydrate_hierarchy_targets(db, [obj])
|
||||
|
||||
existing = next((d for d in obj.deployments if d.host_id == host.id), None)
|
||||
if existing is not None:
|
||||
if getattr(existing, spec.uuid_attr) == remote_uuid:
|
||||
return obj # already linked: nothing to do
|
||||
raise HostAlreadyHasObject(str(object_id))
|
||||
if any(d.status in BUSY_STATUSES for d in obj.deployments):
|
||||
raise LinkTargetBusy(str(object_id))
|
||||
changed = diff_fields(obj, values, host)
|
||||
if changed:
|
||||
raise LinkTargetChanged(changed)
|
||||
|
||||
if host.id not in {h.id for h in effective_hosts(obj)}:
|
||||
obj.targets.append(host)
|
||||
# an association-only change does not touch the row; stale edit forms must still be refused (see edit_rule)
|
||||
obj.updated_at = datetime.now(timezone.utc)
|
||||
await db.flush()
|
||||
spec.revision(db, obj, f"Mit {host.name} verknüpft")
|
||||
db.add(spec.deployment(**{
|
||||
spec.fk: obj.id, "host_id": host.id, "status": "success", "message": LINK_MESSAGE,
|
||||
spec.uuid_attr: remote_uuid, "last_payload": spec.payload(obj, host), "previous_payload": None,
|
||||
}))
|
||||
await db.flush()
|
||||
return obj
|
||||
@@ -130,6 +130,9 @@ async def sweep_expired_alias_deployments(db: AsyncSession) -> list[AliasDeploym
|
||||
|
||||
|
||||
async def undeploy_alias(db: AsyncSession, alias: FirewallAlias) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)
|
||||
failures = []
|
||||
result = await db.execute(select(AliasDeployment).where(AliasDeployment.alias_id == alias.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -13,6 +13,7 @@ from app.models.alias_deployment import AliasDeployment
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.host import AliasOption
|
||||
from app.services import opnsense_client
|
||||
from app.services.deploy_flow import unsettled
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
|
||||
@@ -87,7 +88,7 @@ async def ensure_aliases_settled(db: AsyncSession, hosts: list[OpnsenseHost], va
|
||||
.where(
|
||||
FirewallAlias.name.in_(names),
|
||||
AliasDeployment.host_id.in_([host.id for host in hosts]),
|
||||
AliasDeployment.status.in_(("pending", "revert_failed")),
|
||||
unsettled(AliasDeployment),
|
||||
)
|
||||
.order_by(FirewallAlias.name, OpnsenseHost.name)
|
||||
)).all()
|
||||
|
||||
@@ -38,19 +38,35 @@ MAX_TOKENS_PER_USER = 50
|
||||
_LAST_USED_GRANULARITY = timedelta(minutes=1)
|
||||
|
||||
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)")
|
||||
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
|
||||
# Rule cleanup (1.31): "I still have access" is a human decision, so confirming a cleanup is never possible with a
|
||||
# token, whatever its scope (a script must not be able to confirm blindly).
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$")
|
||||
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
||||
# notification settings the webhook URL/SMTP data): admin scope only.
|
||||
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
|
||||
_MCP_PATH = "/api/mcp"
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
|
||||
# it compares a posted file with the database and writes nothing.
|
||||
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
|
||||
_ADMIN_ONLY_WRITES = re.compile(
|
||||
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
||||
r"|^/api/logs/settings$"
|
||||
r"|^/api/hosts(/tls-probe|/[^/]+)?$"
|
||||
r"|^/api/hosts/[^/]+/gui-ticket$"
|
||||
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
|
||||
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
|
||||
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
|
||||
r"|^/api/config/import/apply$"
|
||||
)
|
||||
|
||||
|
||||
# Rule cleanup: executing a cleanup, retrying a failed revert/reconcile and restoring a deleted rule are admin-only.
|
||||
# operators_may_deploy defaults to True, so _ROLLOUT_POST alone cannot express "admin only"; this pattern is checked
|
||||
# for operator accounts (operator_denies) and for tokens below scope admin (denies). Not part of _ADMIN_ONLY_WRITES:
|
||||
# that one means "who may access OPNexus / where data goes". The router checks the same rule once more.
|
||||
_ADMIN_ONLY_CLEANUP = re.compile(r"^/api/hosts/[^/]+/rule-cleanup/([^/]+/(deploy|retry)|items/[^/]+/restore)/?$")
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
@@ -63,13 +79,13 @@ def denies(scope: str, method: str, path: str) -> bool:
|
||||
"""True if a token with this scope must not make this request."""
|
||||
if _SESSION_ONLY.match(path):
|
||||
return True
|
||||
if path == _MCP_PATH:
|
||||
if path in _READ_ONLY_POSTS:
|
||||
return False
|
||||
if method in _SAFE_METHODS:
|
||||
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
|
||||
if scope == "read":
|
||||
return True
|
||||
if scope == "deploy" and _ADMIN_ONLY_WRITES.match(path):
|
||||
if scope == "deploy" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -78,19 +94,27 @@ def operator_denies(method: str, path: str) -> bool:
|
||||
"""True if an `operator` account must not make this request: the writes that
|
||||
change who may access OPNexus or where it sends data. (Their own theme at
|
||||
PUT /api/settings stays open; account endpoints are not matched here.)"""
|
||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(_ADMIN_ONLY_WRITES.match(path))
|
||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(
|
||||
_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))
|
||||
|
||||
|
||||
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
|
||||
# maintenance (reboot/update/CARP switch), scheduling it, applying network changes, pushing certificates and
|
||||
# ACME accounts, re-asserting the hierarchy order, controlling services, and DELETING a centrally managed object
|
||||
# or snapshot (deleting a rule removes it from every firewall it was deployed to).
|
||||
# or snapshot (deleting a rule removes it from every firewall it was deployed to), and creating a snapshot.
|
||||
_ROLLOUT_POST = re.compile(
|
||||
r"/(deploy|push|sign|poll|register|reassert-ordering"
|
||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||
# Rule cleanup (1.31): switching rules off (/deploy matches above) and taking it back. Confirming and
|
||||
# previewing stay open, like confirm_rule.
|
||||
r"|rule-cleanup/[^/]+/(revert|retry)"
|
||||
r"|maintenance/(execute|carp-execute|schedule)"
|
||||
r"|network/changes"
|
||||
r"|services/[^/]+/(start|stop|restart))$"
|
||||
r"|snapshots" # creating a boot environment (deleting one is in _ROLLOUT_DELETE)
|
||||
r"|services/[^/]+/(start|stop|restart)"
|
||||
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
|
||||
r"|rollouts/plans/[^/]+/(start|activate)"
|
||||
r"|rollouts/runs/[^/]+/(resume|approve|skip-host|abort))$"
|
||||
)
|
||||
_ROLLOUT_DELETE = re.compile(
|
||||
r"^/api/(rules|aliases|port-forward|one-to-one|outbound-nat|policy-hierarchy/rules|ipsec/tunnels|openvpn/instances"
|
||||
@@ -99,7 +123,13 @@ _ROLLOUT_DELETE = re.compile(
|
||||
)
|
||||
|
||||
|
||||
# Recurring maintenance windows decide WHEN unattended firmware actions run (1.25): every write counts.
|
||||
_ROLLOUT_ANY_WRITE = re.compile(r"^/api/maintenance-windows(/|$)")
|
||||
|
||||
|
||||
def is_rollout(method: str, path: str) -> bool:
|
||||
if method in ("POST", "PUT", "PATCH", "DELETE") and _ROLLOUT_ANY_WRITE.match(path):
|
||||
return True
|
||||
if method == "POST":
|
||||
return bool(_ROLLOUT_POST.search(path))
|
||||
return method == "DELETE" and bool(_ROLLOUT_DELETE.match(path))
|
||||
|
||||
@@ -13,6 +13,9 @@ from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
|
||||
context = ContextVar("audit_context", default=None)
|
||||
# A key in ``details`` whose content only admins (session or admin-scope token) get to read (routers/audit.py):
|
||||
# firewall addresses and management sources of the rule cleanup.
|
||||
ADMIN_ONLY_DETAILS = "admin_only"
|
||||
|
||||
|
||||
# Column widths of audit_events. entity_id/host_id/action are taken from the
|
||||
|
||||
@@ -236,6 +236,7 @@ async def create_session(
|
||||
user_agent: str | None = None,
|
||||
ip_address: str | None = None,
|
||||
verified_hash: str | None = None,
|
||||
verified_totp: bool | None = None,
|
||||
lifetime_seconds: int | None = None,
|
||||
) -> tuple[str, datetime]:
|
||||
"""Create a session, returning the raw token (to be set as a cookie) and its expiry.
|
||||
@@ -244,15 +245,22 @@ async def create_session(
|
||||
against. The user row is re-read under a row lock first: a password
|
||||
change/reset that committed while the (slow) password check was running
|
||||
changed the hash, and the login must not outlive that revocation.
|
||||
|
||||
``verified_totp`` is whether the caller checked a second factor. Two-factor
|
||||
enabled in the meantime (it also revoked every session) means this login
|
||||
was decided without it and must start over.
|
||||
"""
|
||||
user_id = user.id
|
||||
if verified_hash is not None:
|
||||
locked = await db.scalar(
|
||||
select(AdminUser.password_hash).where(AdminUser.id == user_id).with_for_update()
|
||||
)
|
||||
if locked != verified_hash:
|
||||
if verified_hash is not None or verified_totp is not None:
|
||||
locked_hash, locked_totp = (await db.execute(
|
||||
select(AdminUser.password_hash, AdminUser.totp_enabled).where(AdminUser.id == user_id).with_for_update()
|
||||
)).one()
|
||||
if verified_hash is not None and locked_hash != verified_hash:
|
||||
await db.rollback()
|
||||
raise api_error(401, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
||||
if locked_totp and not verified_totp:
|
||||
await db.rollback()
|
||||
raise api_error(401, ErrorCode.TOTP_REQUIRED)
|
||||
token = secrets.token_urlsafe(SESSION_TOKEN_BYTES)
|
||||
expires_at = _now() + timedelta(seconds=lifetime_seconds or settings.session_lifetime)
|
||||
session = Session(
|
||||
|
||||
@@ -26,6 +26,7 @@ from app.models.host import OpnsenseHost
|
||||
from app.services import backup_store, rule_hit_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.rule_analysis import coverage
|
||||
|
||||
MAX_CONFIG_BYTES = 32 * 1024 * 1024
|
||||
BACKUP_MAX_AGE_DAYS = 7
|
||||
@@ -83,10 +84,15 @@ def _mvc_rule(rule: Element) -> dict:
|
||||
"sport": _text(rule, "source_port"),
|
||||
"dst": _text(rule, "destination_net") or "any", "dst_not": _text(rule, "destination_not") == "1",
|
||||
"dport": _text(rule, "destination_port"),
|
||||
"floating": False,
|
||||
"if_not": _text(rule, "interfacenot") == "1",
|
||||
"floating": False, # decided in extract_rules (needs the list of interfaces)
|
||||
"quick": _text(rule, "quick") != "0",
|
||||
"descr": _text(rule, "description") or "",
|
||||
"order": int(_text(rule, "sequence") or 0),
|
||||
"sched": _text(rule, "sched") or "", "tagged": _text(rule, "tagged") or "",
|
||||
"received_on": _text(rule, "received-on") or "",
|
||||
"icmptype": ",".join(filter(None, [_text(rule, "icmptype"), _text(rule, "icmp6type")])),
|
||||
"limited": any(_text(rule, tag) for tag in ("max-src-conn-rate", "max-pkt-rate-number", "overload", "tcpflags1")),
|
||||
}
|
||||
|
||||
|
||||
@@ -101,17 +107,29 @@ def _legacy_rule(rule: Element, index: int) -> dict:
|
||||
"ipprotocol": _text(rule, "ipprotocol") or "inet",
|
||||
"protocol": (_text(rule, "protocol") or "any").lower(),
|
||||
"src": src, "src_not": src_not, "sport": sport, "dst": dst, "dst_not": dst_not, "dport": dport,
|
||||
"if_not": rule.find("interfacenot") is not None,
|
||||
"floating": _text(rule, "floating") == "yes",
|
||||
"quick": _text(rule, "quick") != "0",
|
||||
# Legacy rules: an interface rule is always evaluated "quick"; only a floating rule has the option,
|
||||
# and it is off unless <quick> says yes.
|
||||
"quick": _text(rule, "floating") != "yes" or (_text(rule, "quick") or "").lower() in ("yes", "1"),
|
||||
"descr": _text(rule, "descr") or "",
|
||||
"order": index,
|
||||
"sched": _text(rule, "sched") or "", "tagged": _text(rule, "tagged") or "",
|
||||
"received_on": _text(rule, "received-on") or "", "icmptype": _text(rule, "icmptype") or "",
|
||||
"limited": any(_text(rule, tag) for tag in ("max-src-conn-rate", "max-pkt-rate-number", "overload", "tcpflags1")),
|
||||
}
|
||||
|
||||
|
||||
def extract_rules(root: Element) -> list[list[dict]]:
|
||||
"""Rule lists in evaluation order, one per source (MVC, legacy)."""
|
||||
mvc = sorted((_mvc_rule(r) for r in root.findall("OPNsense/Firewall/Filter/rules/rule")), key=lambda r: r["order"])
|
||||
mvc = [_mvc_rule(r) for r in root.findall("OPNsense/Firewall/Filter/rules/rule")]
|
||||
known = {node.tag for node in root.findall("interfaces/*")}
|
||||
for rule in mvc: # same rule as OPNsense: not exactly one interface (or "not") = floating
|
||||
rule["floating"] = len(rule["interfaces"]) != 1 or rule["if_not"]
|
||||
rule["group"] = not rule["floating"] and rule["interfaces"][0] not in known
|
||||
mvc.sort(key=coverage.config_order)
|
||||
legacy = [_legacy_rule(r, i) for i, r in enumerate(root.findall("filter/rule"))]
|
||||
legacy.sort(key=coverage.config_order)
|
||||
return [group for group in (mvc, legacy) if group]
|
||||
|
||||
|
||||
@@ -133,41 +151,22 @@ def _is_any_any(rule: dict) -> bool:
|
||||
|
||||
|
||||
|
||||
def _covers(a: dict, b: dict) -> bool:
|
||||
"""Every packet matched by rule b is already matched by the earlier rule a.
|
||||
Only exact-or-any comparisons, so it never claims coverage it cannot prove."""
|
||||
if not (a["enabled"] and b["enabled"]) or a["floating"] or b["floating"] or not a["quick"]:
|
||||
return False
|
||||
if not a["interfaces"] or not set(b["interfaces"]) <= set(a["interfaces"]):
|
||||
return False
|
||||
if a["direction"] != b["direction"] or a["ipprotocol"] != b["ipprotocol"]:
|
||||
return False
|
||||
if a["protocol"] != "any" and a["protocol"] != b["protocol"]:
|
||||
return False
|
||||
for side in ("src", "dst"):
|
||||
if a[side + "_not"] or b[side + "_not"]:
|
||||
if (a[side], a[side + "_not"]) != (b[side], b[side + "_not"]):
|
||||
return False
|
||||
elif a[side] != "any" and a[side] != b[side]:
|
||||
return False
|
||||
for port in ("sport", "dport"):
|
||||
if a[port] and a[port] != b[port]:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _label(rule: dict) -> str:
|
||||
return rule["descr"] or f"{rule['action']} {rule['src']} → {rule['dst']}"
|
||||
|
||||
|
||||
def analyse_rules(groups: list[list[dict]]) -> dict:
|
||||
"""Shadowed and duplicate rules. The coverage logic lives in rule_analysis/coverage.py (shared with the
|
||||
cleanup candidates); the rules of one list are compared in evaluation order."""
|
||||
shadowed, duplicates = [], []
|
||||
for rules in groups:
|
||||
for j, later in enumerate(rules):
|
||||
for earlier in rules[:j]:
|
||||
if _covers(earlier, later):
|
||||
(duplicates if earlier["action"] == later["action"] and _covers(later, earlier) else shadowed).append(_label(later))
|
||||
break
|
||||
ordered = sorted(rules, key=coverage.config_order)
|
||||
norm = [coverage.from_config_rule(rule) for rule in ordered]
|
||||
by_identity = {id(n): r for n, r in zip(norm, ordered)}
|
||||
for overlap in coverage.analyse(norm):
|
||||
if overlap.by.order == overlap.rule.order:
|
||||
continue # equal position in config.xml: the real order is unknown, claim nothing
|
||||
(duplicates if overlap.kind == "duplicate" else shadowed).append(_label(by_identity[id(overlap.rule)]))
|
||||
return {"shadowed": shadowed, "duplicates": duplicates}
|
||||
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ and the sweeper away in the gap (pending/unknown rows are never re-deployed).
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
from sqlalchemy import and_, or_
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
@@ -46,7 +47,21 @@ def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
async def _checkpoint(db: AsyncSession) -> None:
|
||||
def is_unsettled(row) -> bool:
|
||||
"""A change that may still be live and whose rollback is not yet done: pending, outcome unknown, or an
|
||||
"error" row whose revert deadline is still armed (written but apply/ordering not confirmed)."""
|
||||
return row.status in ("pending", "revert_failed") or (row.status == "error" and row.revert_deadline is not None)
|
||||
|
||||
|
||||
def unsettled(model):
|
||||
"""``is_unsettled`` as a SQL condition for the deployment model."""
|
||||
return or_(
|
||||
model.status.in_(("pending", "revert_failed")),
|
||||
and_(model.status == "error", model.revert_deadline.isnot(None)),
|
||||
)
|
||||
|
||||
|
||||
async def checkpoint(db: AsyncSession) -> None:
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
await db.commit()
|
||||
@@ -67,12 +82,16 @@ async def push_to_host(
|
||||
) -> Any:
|
||||
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
|
||||
before = {name: getattr(existing, name) for name in _FIELDS} if existing else None
|
||||
# An armed "error" row (written, apply not confirmed) holds the UNCONFIRMED payload as last_payload: a retry must
|
||||
# keep the original rollback target instead of promoting that payload to it.
|
||||
armed = existing is not None and existing.status == "error" and existing.revert_deadline is not None
|
||||
rollback_target = existing.previous_payload if armed else prior_live_payload
|
||||
|
||||
# 1. durable intent
|
||||
if existing is not None:
|
||||
row = existing
|
||||
row.status, row.message = "pending", None
|
||||
row.previous_payload, row.last_payload = prior_live_payload, payload
|
||||
row.previous_payload, row.last_payload = rollback_target, payload
|
||||
row.revert_deadline, row.deployed_at = _now() + window, _now()
|
||||
else:
|
||||
row = make(
|
||||
@@ -83,7 +102,7 @@ async def push_to_host(
|
||||
# The intent is a technical safety record, not a state change worth a history
|
||||
# entry of its own (the request itself is already audited as "requested").
|
||||
row._skip_audit = True
|
||||
await _checkpoint(db)
|
||||
await checkpoint(db)
|
||||
row._skip_audit = False
|
||||
|
||||
wrote = False
|
||||
@@ -93,9 +112,9 @@ async def push_to_host(
|
||||
# 3. recoverable by UUID from here on
|
||||
setattr(row, uuid_attr, new_uuid)
|
||||
row.status, row.message = "pending", None
|
||||
row.previous_payload = prior_live_payload if existing is not None else None
|
||||
row.previous_payload = rollback_target if existing is not None else None
|
||||
row.last_payload, row.revert_deadline, row.deployed_at = payload, _now() + window, _now()
|
||||
await _checkpoint(db)
|
||||
await checkpoint(db)
|
||||
await apply()
|
||||
except OpnsenseAPIError as exc:
|
||||
ambiguous = isinstance(exc, OpnsenseAmbiguousError)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
"""An exclusive per-object lock for flows that make several remote calls and commit in between (VPN deploys).
|
||||
|
||||
``policy_lock`` is transaction-scoped and is released by every checkpoint commit; this one is a session-level advisory
|
||||
lock on a connection of its own, so it spans the commits of the request session. If the process dies the connection
|
||||
closes and PostgreSQL frees the lock. It never waits: a second deploy/removal of the same object is refused at once.
|
||||
"""
|
||||
import uuid
|
||||
import zlib
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.database import engine
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
# Written when the first row is committed, before any remote call: a crash or cancellation leaves this, and a new
|
||||
# deploy continues from the UUIDs recorded after it.
|
||||
VPN_INTERRUPTED = (
|
||||
"Ausrollen unterbrochen oder Ausgang unklar: Die Firewall hat möglicherweise schon Teile der Konfiguration "
|
||||
"erhalten. Erneutes Ausrollen setzt dort fort."
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def resource_lease(kind: str, resource_id: uuid.UUID):
|
||||
params = {"a": zlib.crc32(kind.encode()) & 0x7FFFFFFF, "b": resource_id.int & 0x7FFFFFFF}
|
||||
async with engine.connect() as conn:
|
||||
if not await conn.scalar(text("SELECT pg_try_advisory_lock(:a, :b)"), params):
|
||||
raise api_error(409, ErrorCode.VPN_DEPLOY_LAEUFT_BEREITS)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
try:
|
||||
await conn.execute(text("SELECT pg_advisory_unlock(:a, :b)"), params)
|
||||
except Exception:
|
||||
await conn.invalidate() # never hand a connection that may still hold the lock back to the pool
|
||||
except BaseException: # cancelled while unlocking
|
||||
await conn.invalidate()
|
||||
raise
|
||||
@@ -63,6 +63,17 @@ async def _guarded(host: OpnsenseHost, tool: str, target: str, params: dict, cal
|
||||
return result
|
||||
|
||||
|
||||
def _with_silent_hops(hops: list[dict]) -> list[dict]:
|
||||
"""The firewall leaves out hops that never answered (TTL jumps 1 -> 3, T5 of the 1.31.0 test run). Put them back as
|
||||
explicit rows so a gap reads as "no answer here", not as a missing hop. A route where nothing answered at all stays
|
||||
an empty list; the UI says so."""
|
||||
seen = {int(h["ttl"]): h for h in hops if h["ttl"].isdigit()}
|
||||
if not seen:
|
||||
return hops
|
||||
silent = {"address": "", "host": "", "as": "", "probes": "", "silent": True}
|
||||
return [seen.get(ttl) or {**silent, "ttl": str(ttl)} for ttl in range(1, max(seen) + 1)]
|
||||
|
||||
|
||||
async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: str, source: str | None) -> dict:
|
||||
target, source = clean_target(target), clean_source(source)
|
||||
params = {"target": target, "ipproto": ipproto, "protocol": protocol, "source": source}
|
||||
@@ -73,8 +84,9 @@ async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: st
|
||||
{"hostname": target, "ipproto": ipproto, "protocol": protocol, "source_address": source})
|
||||
resp = data.get("response") or {}
|
||||
hops = [{"ttl": str(r.get("ttl", "")), "address": str(r.get("address", "")), "host": str(r.get("host", "")),
|
||||
"as": str(r.get("AS", "")), "probes": str(r.get("probes", ""))} for r in resp.get("rows", []) if isinstance(r, dict)]
|
||||
return {"target": target, "notice": str(resp.get("notice", "")), "hops": hops}
|
||||
"as": str(r.get("AS", "")), "probes": str(r.get("probes", "")), "silent": False}
|
||||
for r in resp.get("rows", []) if isinstance(r, dict)]
|
||||
return {"target": target, "notice": str(resp.get("notice", "")), "hops": _with_silent_hops(hops)}
|
||||
return await _guarded(host, "traceroute", target, params, call)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Gateway health of a firewall (dpinger numbers, as OPNsense reports them).
|
||||
|
||||
``get_gateway_health(host)`` is the one reusable entry point: a live read of
|
||||
/routes/gateway/status, one :class:`GatewayHealth` per gateway. The notification
|
||||
engine, the host page and the metrics read the SAME numbers from the periodic
|
||||
resource sample (``compact`` / ``from_compact``), so a scrape or a page view never
|
||||
fans out to the firewalls; the 1.25 health gate can call ``get_gateway_health``
|
||||
directly. Parsing is lenient on purpose: "~" means "nothing measured", never zero.
|
||||
"""
|
||||
import re
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.gateway import GatewayHealth, GatewayState
|
||||
from app.services import opnsense_client as client
|
||||
|
||||
_NUMBER = re.compile(r"-?\d+(?:\.\d+)?")
|
||||
_DEGRADED = {"delay", "loss", "delay+loss"}
|
||||
|
||||
|
||||
def _number(value) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
match = _NUMBER.search(str(value))
|
||||
return float(match.group()) if match else None
|
||||
|
||||
|
||||
def _state(status: str, measured: bool) -> GatewayState:
|
||||
if status == "down":
|
||||
return "offline"
|
||||
if status == "force_down":
|
||||
return "forced_down"
|
||||
if status in _DEGRADED:
|
||||
return "degraded"
|
||||
if status == "none":
|
||||
# "none" is also what a gateway without monitoring reports; only numbers make it "online".
|
||||
return "online" if measured else "unknown"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def parse_gateway_status(payload) -> list[GatewayHealth]:
|
||||
items = payload.get("items") if isinstance(payload, dict) else None
|
||||
result: list[GatewayHealth] = []
|
||||
for item in items or []:
|
||||
if not isinstance(item, dict) or not item.get("name"):
|
||||
continue
|
||||
status = str(item.get("status") or "").lower()
|
||||
loss, delay, stddev = _number(item.get("loss")), _number(item.get("delay")), _number(item.get("stddev"))
|
||||
if loss is not None and loss >= 100:
|
||||
delay = stddev = None # dpinger prints 0.0 ms when nothing answers: no round trip, not a fast one
|
||||
measured = loss is not None or delay is not None
|
||||
address = item.get("address")
|
||||
monitor = item.get("monitor")
|
||||
result.append(GatewayHealth(
|
||||
name=str(item["name"]),
|
||||
address=None if address in (None, "", "~") else str(address),
|
||||
status=status or "unknown",
|
||||
state=_state(status, measured),
|
||||
loss=loss, delay=delay, stddev=stddev,
|
||||
monitor=None if monitor in (None, "", "~") else str(monitor),
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
async def get_gateway_health(host: OpnsenseHost) -> list[GatewayHealth]:
|
||||
"""Live gateway state of ``host``. Raises OpnsenseAPIError if the firewall cannot be asked."""
|
||||
return parse_gateway_status(await client.get_gateway_status(host))
|
||||
|
||||
|
||||
def compact(gateways: list[GatewayHealth]) -> list[dict]:
|
||||
"""What the resource sampler stores per sample (small, no addresses)."""
|
||||
return [
|
||||
{"name": g.name, "status": g.status, "state": g.state, "loss": g.loss, "delay": g.delay, "stddev": g.stddev}
|
||||
for g in gateways
|
||||
]
|
||||
|
||||
|
||||
def from_compact(rows) -> list[GatewayHealth]:
|
||||
return [
|
||||
GatewayHealth(
|
||||
name=r["name"], status=r.get("status") or "unknown", state=r.get("state") or "unknown",
|
||||
loss=r.get("loss"), delay=r.get("delay"), stddev=r.get("stddev"),
|
||||
)
|
||||
for r in rows or [] if isinstance(r, dict) and r.get("name")
|
||||
]
|
||||
|
||||
|
||||
def is_bad(gateway: GatewayHealth, loss_threshold_pct: float = 0, delay_threshold_ms: float = 0) -> bool:
|
||||
"""Alarm-worthy: down, or (when the thresholds are on) measured loss / latency beyond them.
|
||||
A gateway forced down by the admin and one that is not measured are not alarms."""
|
||||
if gateway.state == "offline":
|
||||
return True
|
||||
if loss_threshold_pct and gateway.loss is not None and gateway.loss >= loss_threshold_pct:
|
||||
return True
|
||||
if delay_threshold_ms and gateway.delay is not None and gateway.delay >= delay_threshold_ms:
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -0,0 +1,384 @@
|
||||
"""Tickets and gateway sessions for the firewall GUI access (1.27, gui-gateway container).
|
||||
|
||||
Flow (see docs in routers/gui_access.py):
|
||||
* An admin asks for a ticket (POST /api/hosts/{id}/gui-ticket). 32 random bytes, valid 30 s, single use;
|
||||
only its SHA-256 is stored, bound to the OPNexus session (same hash as sessions.token_hash), the user,
|
||||
the host and a slot (gateway port = its own browser origin).
|
||||
* The gateway redeems it through /internal/gui/redeem and gets what it needs to reach the firewall —
|
||||
base URL, pinned CA, overlay proxy, timeouts. Never the API key or secret.
|
||||
* Every 20 s the gateway re-checks the gateway session (/internal/gui/revalidate): OPNexus session still
|
||||
valid, user still admin, switch still on, host still eligible, absolute limit not reached.
|
||||
|
||||
A slot stays assigned to one firewall for as long as that browser session has the firewall open; another
|
||||
firewall gets another slot (another origin), so two firewall GUIs never share an origin at the same time.
|
||||
After the last gateway session of a slot ended, the slot goes to a DIFFERENT firewall only after SLOT_COOLDOWN
|
||||
(1.29, S1): a popup of a possibly compromised firewall GUI may still be open and its cookies belong to that
|
||||
origin. The same firewall may take the slot again at once.
|
||||
|
||||
The target the gateway talks to (origin, pinned CA, overlay proxy) is fingerprinted at redeem time; every
|
||||
re-check compares it with the host's current configuration and ends the session when it changed (S4).
|
||||
"""
|
||||
import hashlib
|
||||
import secrets
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import delete, func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode
|
||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser, Session
|
||||
from app.services import audit_service, firewall_url, overlay_service
|
||||
|
||||
TICKET_TTL = timedelta(seconds=30)
|
||||
# 20 s: a switch-off, logout or role change must end a gateway session within 30 s of the change, and
|
||||
# the last re-check can lie up to one interval before it (live test 2026-10-21: 30 s gave 31 s).
|
||||
REVALIDATE_SECONDS = 20
|
||||
# A gateway session nobody re-checked for this long is treated as ended (gateway crashed without telling us).
|
||||
SESSION_STALE_AFTER = timedelta(seconds=REVALIDATE_SECONDS * 3)
|
||||
MAX_SLOTS = 4
|
||||
# A slot freed by firewall A goes to firewall B no earlier than this long after A's session ended.
|
||||
SLOT_COOLDOWN = timedelta(minutes=5)
|
||||
# Open gateway sessions at once (the gateway container enforces the same number; test_gui_access checks they match).
|
||||
MAX_GATEWAY_SESSIONS = 64
|
||||
MIN_GATEWAY_TOKEN_LENGTH = 32
|
||||
AUDIT_KIND = "gui_access"
|
||||
|
||||
|
||||
@dataclass
|
||||
class GuiAccessError(Exception):
|
||||
status: int
|
||||
code: ErrorCode
|
||||
params: dict
|
||||
|
||||
|
||||
def hash_value(raw: str) -> str:
|
||||
return hashlib.sha256(raw.encode()).hexdigest()
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
# ── configuration ─────────────────────────────────────────────────────────
|
||||
|
||||
def slot_sites() -> dict[int, list[str]]:
|
||||
"""Slot port -> configured https origins with that port (OPNEXUS_GUI_SITES)."""
|
||||
sites: dict[int, list[str]] = {}
|
||||
for origin in settings.gui_site_origins:
|
||||
try:
|
||||
parts = urlsplit(origin)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
continue
|
||||
if parts.scheme != "https" or not parts.hostname or not port or parts.path not in ("", "/"):
|
||||
continue
|
||||
sites.setdefault(port, []).append(f"https://{parts.netloc}")
|
||||
return sites
|
||||
|
||||
|
||||
def gateway_configured() -> bool:
|
||||
return len(settings.gui_gateway_token) >= MIN_GATEWAY_TOKEN_LENGTH and bool(slot_sites())
|
||||
|
||||
|
||||
def unavailable_reasons() -> list[str]:
|
||||
reasons = []
|
||||
if not settings.cookie_secure:
|
||||
reasons.append("cookie_secure")
|
||||
if len(settings.gui_gateway_token) < MIN_GATEWAY_TOKEN_LENGTH:
|
||||
reasons.append("gateway_token")
|
||||
if not slot_sites():
|
||||
reasons.append("gui_sites")
|
||||
return reasons
|
||||
|
||||
|
||||
def active_slots(row: GuiAccessSettings) -> list[int]:
|
||||
return sorted(slot_sites())[: max(1, min(row.slot_count, MAX_SLOTS))]
|
||||
|
||||
|
||||
async def get_settings(db: AsyncSession) -> GuiAccessSettings:
|
||||
row = await db.get(GuiAccessSettings, 1)
|
||||
if row is None:
|
||||
row = GuiAccessSettings(id=1, enabled=False, idle_timeout_minutes=15, absolute_timeout_minutes=60, slot_count=MAX_SLOTS)
|
||||
db.add(row)
|
||||
await db.flush()
|
||||
return row
|
||||
|
||||
|
||||
def host_refusal(host: OpnsenseHost) -> str | None:
|
||||
"""Why this firewall must not be opened through the gateway (None = eligible)."""
|
||||
if not host.enabled:
|
||||
return "disabled"
|
||||
if urlsplit(host.base_url.strip()).scheme != "https" or firewall_url.check_base_url(host.base_url):
|
||||
return "https"
|
||||
if not host.verify_tls:
|
||||
return "tls"
|
||||
return None
|
||||
|
||||
|
||||
def firewall_origin(base_url: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
return f"https://{parts.netloc.rsplit('@', 1)[-1]}"
|
||||
|
||||
|
||||
def target_fingerprint(host: OpnsenseHost) -> str:
|
||||
"""What a gateway session talks to: origin, pinned CA and overlay route. Changes -> the session must end."""
|
||||
material = "\n".join([firewall_origin(host.base_url), host.tls_ca_pem or "", overlay_service.proxy_for(host.base_url) or "",
|
||||
"verify" if host.verify_tls else "noverify"])
|
||||
return hash_value(material)
|
||||
|
||||
|
||||
# ── audit ─────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _audit(actor: str, *, action: str, status: str, host: OpnsenseHost | None = None,
|
||||
host_id=None, entity_id=None, message: str | None = None, details: dict | None = None) -> None:
|
||||
token = audit_service.context.set({"actor": actor, "id": str(uuid.uuid4()), "action": action})
|
||||
try:
|
||||
await audit_service.persist(
|
||||
kind=AUDIT_KIND, action=action, status=status, entity_id=str(entity_id) if entity_id else None,
|
||||
host_id=str(host.id) if host is not None else (str(host_id) if host_id else None),
|
||||
host_name=host.name if host is not None else None, message=message, details=details or {},
|
||||
)
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
# ── tickets ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _redeem_origin(slot: int, request_host: str) -> str:
|
||||
candidates = slot_sites().get(slot, [])
|
||||
wanted = (urlsplit(f"//{request_host}").hostname or "").lower()
|
||||
for origin in candidates:
|
||||
if (urlsplit(origin).hostname or "").lower() == wanted:
|
||||
return origin
|
||||
return candidates[0]
|
||||
|
||||
|
||||
async def _pick_slot(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id) -> int:
|
||||
now = _now()
|
||||
slots = active_slots(row)
|
||||
absolute = timedelta(minutes=row.absolute_timeout_minutes)
|
||||
in_use: dict[int, uuid.UUID] = {}
|
||||
open_sessions = await db.scalars(select(GuiSession).where(
|
||||
GuiSession.session_hash == session_hash, GuiSession.ended_at.is_(None), GuiSession.started_at > now - absolute))
|
||||
for gs in open_sessions:
|
||||
in_use[gs.slot] = gs.host_id
|
||||
pending = await db.scalars(select(GuiTicket).where(
|
||||
GuiTicket.session_hash == session_hash, GuiTicket.redeemed_at.is_(None), GuiTicket.expires_at > now))
|
||||
for ticket in pending:
|
||||
in_use.setdefault(ticket.slot, ticket.host_id)
|
||||
for slot in slots:
|
||||
if in_use.get(slot) == host_id:
|
||||
return slot # this firewall is already open in this browser session: same origin again
|
||||
free = [s for s in slots if s not in in_use]
|
||||
if not free:
|
||||
raise GuiAccessError(409, ErrorCode.GUI_NO_FREE_SLOT, {"slots": str(len(slots))})
|
||||
# S1: a slot this user's browser may still have a window of ANOTHER firewall on stays closed to this
|
||||
# firewall — while a gateway session of the user is open there (any browser session) and for SLOT_COOLDOWN
|
||||
# after the last one ended. The same firewall may take it again at once.
|
||||
held = (await db.execute(
|
||||
select(GuiSession.slot, GuiSession.host_id, GuiSession.ended_at, GuiSession.last_checked_at).where(
|
||||
GuiSession.user_id == user_id,
|
||||
((GuiSession.ended_at.is_(None)) & ((GuiSession.started_at > now - absolute) | (GuiSession.last_checked_at > now - SLOT_COOLDOWN)))
|
||||
| (GuiSession.ended_at > now - SLOT_COOLDOWN)))).all()
|
||||
cooling: dict[int, datetime] = {} # slot -> when it may go to another firewall
|
||||
for slot, h_id, ended_at, checked_at in held:
|
||||
if h_id == host_id or slot not in free:
|
||||
continue
|
||||
if ended_at is None and checked_at is not None and now - checked_at > SESSION_STALE_AFTER:
|
||||
ended_at = checked_at # the gateway stopped re-checking it (crash, kill): it ended when it was last seen
|
||||
release = (ended_at + SLOT_COOLDOWN) if ended_at is not None else now + SLOT_COOLDOWN
|
||||
if release <= now:
|
||||
continue # cooled down long ago (a session that stopped answering a while back)
|
||||
cooling[slot] = max(cooling.get(slot, release), release)
|
||||
usable = [s for s in free if s not in cooling]
|
||||
if not usable:
|
||||
wait = min(cooling.values()) - now
|
||||
raise GuiAccessError(409, ErrorCode.GUI_SLOT_COOLING_DOWN,
|
||||
{"minutes": str(max(1, -(-int(wait.total_seconds()) // 60)))})
|
||||
free = usable
|
||||
# Prefer the slot this user had this firewall on before, then the slot the user used least recently
|
||||
# (an old tab of another firewall on a reused origin is the documented residual risk).
|
||||
history = (await db.execute(
|
||||
select(GuiSession.slot, GuiSession.host_id, func.max(GuiSession.started_at))
|
||||
.where(GuiSession.user_id == user_id).group_by(GuiSession.slot, GuiSession.host_id)
|
||||
)).all()
|
||||
for slot, h_id, _ in sorted(history, key=lambda r: r[2], reverse=True):
|
||||
if h_id == host_id and slot in free:
|
||||
return slot
|
||||
last_used = {}
|
||||
for slot, _, started in history:
|
||||
last_used[slot] = max(last_used.get(slot, started), started)
|
||||
epoch = datetime.min.replace(tzinfo=timezone.utc)
|
||||
return min(free, key=lambda s: (last_used.get(s, epoch), s))
|
||||
|
||||
|
||||
async def issue_ticket(db: AsyncSession, *, user: AdminUser, session_token: str, host: OpnsenseHost,
|
||||
request_host: str) -> dict:
|
||||
row = await get_settings(db)
|
||||
now = _now()
|
||||
await db.execute(delete(GuiTicket).where(GuiTicket.expires_at < now - timedelta(hours=1)))
|
||||
session_hash = hash_value(session_token)
|
||||
slot = await _pick_slot(db, row, session_hash, user.id, host.id)
|
||||
# Reopening a slot in the same browser session replaces its earlier gateway session, so that one does not count.
|
||||
open_count = await db.scalar(select(func.count()).select_from(GuiSession).where(
|
||||
GuiSession.ended_at.is_(None),
|
||||
GuiSession.started_at > now - timedelta(minutes=row.absolute_timeout_minutes),
|
||||
~((GuiSession.session_hash == session_hash) & (GuiSession.slot == slot))))
|
||||
if (open_count or 0) >= MAX_GATEWAY_SESSIONS:
|
||||
raise GuiAccessError(409, ErrorCode.GUI_TOO_MANY_SESSIONS, {"max": str(MAX_GATEWAY_SESSIONS)})
|
||||
raw = secrets.token_urlsafe(32)
|
||||
db.add(GuiTicket(ticket_hash=hash_value(raw), session_hash=session_hash, user_id=user.id, host_id=host.id,
|
||||
slot=slot, created_at=now, expires_at=now + TICKET_TTL))
|
||||
await db.commit()
|
||||
await _audit(user.username, action="ticket", status="accepted", host=host, details={"slot": slot})
|
||||
return {"url": f"{_redeem_origin(slot, request_host)}/__onx/redeem?t={raw}", "slot": slot,
|
||||
"expires_in": int(TICKET_TTL.total_seconds())}
|
||||
|
||||
|
||||
# ── gateway side (internal endpoints) ─────────────────────────────────────
|
||||
|
||||
async def _live_check(db: AsyncSession, row: GuiAccessSettings, session_hash: str, user_id, host_id,
|
||||
slot: int) -> tuple[str | None, AdminUser | None, OpnsenseHost | None]:
|
||||
session = await db.scalar(select(Session).where(Session.token_hash == session_hash))
|
||||
if session is None or session.expires_at <= _now() or session.user_id != user_id:
|
||||
return "session", None, None
|
||||
user = await db.get(AdminUser, user_id)
|
||||
if user is None or user.role != "admin":
|
||||
return "role", user, None
|
||||
if not row.enabled:
|
||||
return "disabled", user, None
|
||||
if not settings.cookie_secure or not gateway_configured():
|
||||
return "config", user, None
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None or host_refusal(host):
|
||||
return "host", user, host
|
||||
if slot not in active_slots(row):
|
||||
return "slot", user, host
|
||||
return None, user, host
|
||||
|
||||
|
||||
def _timing(row: GuiAccessSettings, started_at: datetime) -> dict:
|
||||
deadline = started_at + timedelta(minutes=row.absolute_timeout_minutes)
|
||||
return {"idle_timeout_seconds": row.idle_timeout_minutes * 60,
|
||||
"absolute_deadline": int(deadline.timestamp()),
|
||||
"revalidate_seconds": REVALIDATE_SECONDS}
|
||||
|
||||
|
||||
async def redeem(db: AsyncSession, *, ticket_hash: str, session_hash: str, slot: int, client_ip: str | None) -> dict:
|
||||
"""{'ok': True, ...upstream data...} or {'ok': False, 'reason': ...}. Any attempt burns the ticket."""
|
||||
now = _now()
|
||||
ticket = await db.scalar(select(GuiTicket).where(GuiTicket.ticket_hash == ticket_hash).with_for_update())
|
||||
if ticket is None:
|
||||
await db.rollback()
|
||||
return {"ok": False, "reason": "unknown"}
|
||||
replayed = ticket.redeemed_at is not None
|
||||
ticket.redeemed_at = ticket.redeemed_at or now
|
||||
row = await get_settings(db)
|
||||
reason = None
|
||||
if replayed:
|
||||
reason = "replayed"
|
||||
elif ticket.expires_at <= now:
|
||||
reason = "expired"
|
||||
elif not secrets.compare_digest(ticket.session_hash, session_hash):
|
||||
reason = "session"
|
||||
elif ticket.slot != slot:
|
||||
reason = "slot"
|
||||
user = host = None
|
||||
if reason is None:
|
||||
reason, user, host = await _live_check(db, row, ticket.session_hash, ticket.user_id, ticket.host_id, slot)
|
||||
if user is None:
|
||||
user = await db.get(AdminUser, ticket.user_id)
|
||||
if host is None:
|
||||
host = await db.get(OpnsenseHost, ticket.host_id)
|
||||
actor = user.username if user is not None else "unknown"
|
||||
if reason is not None:
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
|
||||
message=reason, details={"slot": slot, "client_ip": client_ip, "reason": reason})
|
||||
return {"ok": False, "reason": reason}
|
||||
# The same browser session reopening this slot replaces the earlier gateway session there.
|
||||
superseded = (await db.scalars(select(GuiSession).where(
|
||||
GuiSession.session_hash == session_hash, GuiSession.slot == slot, GuiSession.ended_at.is_(None)))).all()
|
||||
if any(old.host_id != host.id for old in superseded):
|
||||
# Two quick tickets both saw the slot free: the second must not take over another firewall's origin
|
||||
# without the cooling-down time (S1).
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
|
||||
message="slot", details={"slot": slot, "client_ip": client_ip, "reason": "slot"})
|
||||
return {"ok": False, "reason": "slot"}
|
||||
for old in superseded:
|
||||
old.ended_at, old.end_reason = now, "superseded"
|
||||
gs = GuiSession(session_hash=session_hash, user_id=user.id, host_id=host.id, slot=slot,
|
||||
client_ip=(client_ip or None) and client_ip[:45], started_at=now, last_checked_at=now)
|
||||
db.add(gs)
|
||||
await db.commit()
|
||||
await _audit(actor, action="redeem", status="accepted", host=host, entity_id=gs.id,
|
||||
details={"slot": slot, "client_ip": client_ip})
|
||||
return {
|
||||
"ok": True, "gui_session_id": str(gs.id), "host_id": str(host.id), "host_name": host.name,
|
||||
"upstream_origin": firewall_origin(host.base_url),
|
||||
"ca_pem": host.tls_ca_pem or None, # public certificate (pin); None = system trust store
|
||||
"proxy_url": overlay_service.proxy_for(host.base_url),
|
||||
"target_fp": target_fingerprint(host),
|
||||
**_timing(row, now),
|
||||
}
|
||||
|
||||
|
||||
async def _finish(db: AsyncSession, gs: GuiSession, reason: str) -> None:
|
||||
now = _now()
|
||||
if gs.ended_at is None:
|
||||
gs.ended_at, gs.end_reason = now, reason[:32]
|
||||
await db.commit()
|
||||
user = await db.get(AdminUser, gs.user_id)
|
||||
host = await db.get(OpnsenseHost, gs.host_id)
|
||||
await _audit(user.username if user else "unknown", action="end", status="accepted", host=host,
|
||||
host_id=gs.host_id, entity_id=gs.id, message=reason,
|
||||
details={"slot": gs.slot, "client_ip": gs.client_ip, "reason": reason,
|
||||
"duration_seconds": int((gs.ended_at - gs.started_at).total_seconds())})
|
||||
|
||||
|
||||
async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str,
|
||||
target_fp: str | None = None) -> dict:
|
||||
gs = await db.get(GuiSession, gui_session_id)
|
||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||
return {"ok": False, "reason": "unknown"}
|
||||
if gs.ended_at is not None:
|
||||
return {"ok": False, "reason": gs.end_reason or "ended"}
|
||||
row = await get_settings(db)
|
||||
if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):
|
||||
await _finish(db, gs, "absolute")
|
||||
return {"ok": False, "reason": "absolute"}
|
||||
reason, _, host = await _live_check(db, row, gs.session_hash, gs.user_id, gs.host_id, gs.slot)
|
||||
# S4: the gateway presents the fingerprint it got at redeem time. Firewall address, pinned certificate or
|
||||
# overlay route changed since -> the session (and the upstream connection it holds) ends now.
|
||||
if reason is None and target_fp is not None and host is not None \
|
||||
and not secrets.compare_digest(target_fp, target_fingerprint(host)):
|
||||
reason = "target"
|
||||
if reason is not None:
|
||||
await _finish(db, gs, reason)
|
||||
return {"ok": False, "reason": reason}
|
||||
gs.last_checked_at = _now()
|
||||
await db.commit()
|
||||
return {"ok": True, **_timing(row, gs.started_at)}
|
||||
|
||||
|
||||
async def end(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str, reason: str) -> bool:
|
||||
gs = await db.get(GuiSession, gui_session_id)
|
||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||
return False
|
||||
if gs.ended_at is None:
|
||||
await _finish(db, gs, reason)
|
||||
return True
|
||||
|
||||
|
||||
async def end_all(db: AsyncSession, reason: str) -> int:
|
||||
"""Switch turned off: mark every open gateway session ended (the gateway drops them on its next check)."""
|
||||
result = await db.execute(update(GuiSession).where(GuiSession.ended_at.is_(None))
|
||||
.values(ended_at=_now(), end_reason=reason[:32]))
|
||||
return result.rowcount or 0
|
||||
@@ -12,6 +12,7 @@ from app.models import OpnsenseHost, RuleDeployment, AliasDeployment, PortForwar
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.restore_operation import RestoreOperation
|
||||
from app.services import opnsense_client as client, backup_store, carp_maintenance_window_service
|
||||
from app.services.deploy_flow import unsettled
|
||||
from app.services.host_status_service import get_statuses
|
||||
|
||||
DEPLOYMENTS = (RuleDeployment, AliasDeployment, PortForwardDeployment, OneToOneDeployment, OutboundNatDeployment)
|
||||
@@ -34,7 +35,27 @@ async def peers(db, host):
|
||||
return [host]
|
||||
|
||||
|
||||
async def ensure_no_maintenance(db, hosts):
|
||||
async def ensure_no_open_cleanup(db, hosts):
|
||||
"""A rule cleanup (1.31) that is running, waiting for its confirmation or unresolved owns the host and its
|
||||
cluster peers: nothing else may write to the firewall until it is confirmed, reverted or settled."""
|
||||
from app.services.rule_cleanup_service import has_open_cleanup
|
||||
ids = set()
|
||||
for host in hosts:
|
||||
ids.update(h.id for h in await peers(db, host))
|
||||
if ids and await has_open_cleanup(db, ids):
|
||||
raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)
|
||||
|
||||
|
||||
async def ensure_no_open_cleanup_on(db, model, condition):
|
||||
"""ensure_no_open_cleanup for every host that has a deployment row of ``model`` matching ``condition`` (the
|
||||
undeploy paths: deleting a central object removes it from all its firewalls)."""
|
||||
ids = set((await db.scalars(select(model.host_id).where(condition))).all())
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, i) for i in ids] if h is not None]
|
||||
await ensure_no_open_cleanup(db, hosts)
|
||||
|
||||
|
||||
async def ensure_no_maintenance(db, hosts, *, cleanup=True):
|
||||
"""cleanup=False: only the cleanup service itself, which reports an open cleanup with its own error code."""
|
||||
ids = set()
|
||||
for host in hosts:
|
||||
ids.update(h.id for h in await peers(db, host))
|
||||
@@ -42,13 +63,15 @@ async def ensure_no_maintenance(db, hosts):
|
||||
raise api_error(409, ErrorCode.OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY)
|
||||
if ids and await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||
raise api_error(409, ErrorCode.OPEN_RESTORE_BLOCKS_DEPLOY)
|
||||
if cleanup and ids:
|
||||
await ensure_no_open_cleanup(db, hosts)
|
||||
|
||||
|
||||
async def _shared_change_blockers(db, fleet):
|
||||
blocked = []
|
||||
fleet_ids = [h.id for h in fleet]
|
||||
for model in DEPLOYMENTS:
|
||||
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), model.status.in_(("pending", "revert_failed"))).limit(1)):
|
||||
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), unsettled(model)).limit(1)):
|
||||
blocked.append("Offene Bestätigung oder fehlgeschlagener Rollback im Host/Cluster.")
|
||||
break
|
||||
operations = list((await db.scalars(select(HostOperation).where(HostOperation.host_id.in_(fleet_ids), HostOperation.status.in_(ACTIVE)))).all())
|
||||
@@ -56,6 +79,9 @@ async def _shared_change_blockers(db, fleet):
|
||||
blocked.append("Eine frühere Wartungsaktion im Host/Cluster muss zuerst geprüft und abgeschlossen werden.")
|
||||
if await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(fleet_ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||
blocked.append("Eine Wiederherstellung im Host/Cluster ist offen und muss zuerst bestätigt oder abgebrochen werden.")
|
||||
from app.services.rule_cleanup_service import has_open_cleanup
|
||||
if await has_open_cleanup(db, fleet_ids):
|
||||
blocked.append("Eine Regel-Bereinigung im Host/Cluster ist offen und muss zuerst bestätigt oder zurückgenommen werden.")
|
||||
return blocked
|
||||
|
||||
|
||||
|
||||
@@ -109,6 +109,8 @@ from app.models.ipsec_tunnel import IpsecTunnel
|
||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||
from app.services import opnsense_client, ipsec_psk_store
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_flow import checkpoint
|
||||
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
|
||||
|
||||
_VERSION_MAP = {"ike": "0", "ikev1": "1", "ikev2": "2"}
|
||||
|
||||
@@ -176,7 +178,7 @@ def build_payloads(tunnel: IpsecTunnel) -> dict:
|
||||
}
|
||||
|
||||
|
||||
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
|
||||
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
|
||||
"""Runs the 5 config-write calls in dependency order, persisting each
|
||||
UUID onto `deployment` as soon as it's known so a later retry (after a
|
||||
partial failure) updates rather than duplicates. Raises
|
||||
@@ -189,6 +191,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_connection_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_connection_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Connection-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
# `payloads` (built by build_payloads()) is also what gets persisted as
|
||||
# last_payload for no-op-redeploy comparison — the injected `connection`
|
||||
@@ -209,6 +212,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_local_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_local_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Local-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
if deployment.opnsense_remote_uuid:
|
||||
await opnsense_client.set_ipsec_remote(host, deployment.opnsense_remote_uuid, remote_wire)
|
||||
@@ -217,6 +221,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_remote_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_remote_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Remote-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
if deployment.opnsense_child_uuid:
|
||||
await opnsense_client.set_ipsec_child(host, deployment.opnsense_child_uuid, child_wire)
|
||||
@@ -225,6 +230,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_child_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_child_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Child-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
if deployment.opnsense_psk_uuid:
|
||||
await opnsense_client.set_ipsec_psk(host, deployment.opnsense_psk_uuid, payloads["psk"])
|
||||
@@ -233,6 +239,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_psk_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_psk_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine PSK-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
|
||||
async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseHost) -> IpsecTunnelDeployment:
|
||||
@@ -241,7 +248,7 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
existing = await db.scalar(
|
||||
select(IpsecTunnelDeployment).where(
|
||||
IpsecTunnelDeployment.tunnel_id == tunnel.id, IpsecTunnelDeployment.host_id == host.id
|
||||
)
|
||||
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
|
||||
)
|
||||
if existing and existing.status == "success" and existing.last_payload == comparable:
|
||||
# Nothing actually changed on this host — re-affirm without
|
||||
@@ -252,8 +259,15 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
|
||||
deployment = existing or IpsecTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
|
||||
|
||||
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
|
||||
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
|
||||
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
await checkpoint(db)
|
||||
|
||||
try:
|
||||
await _write_config(host, payloads, deployment)
|
||||
await _write_config(db, host, payloads, deployment)
|
||||
# Same trap as WireGuard: with the global IPsec switch off the config
|
||||
# is stored but nothing runs, and reconfigure still answers ok.
|
||||
await opnsense_client.ensure_ipsec_enabled(host)
|
||||
@@ -261,8 +275,6 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
try:
|
||||
@@ -277,32 +289,37 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
deployment.status = "success"
|
||||
deployment.message = None
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
|
||||
async def deploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel, hosts: list[OpnsenseHost]) -> list[IpsecTunnelDeployment]:
|
||||
from app.services.host_maintenance_service import ensure_no_maintenance
|
||||
await ensure_no_maintenance(db, hosts)
|
||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||
await db.commit()
|
||||
async with resource_lease("ipsec", tunnel.id):
|
||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||
await db.commit()
|
||||
return deployments
|
||||
|
||||
|
||||
async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||
async with resource_lease("ipsec", tunnel.id):
|
||||
await _undeploy_tunnel(db, tunnel)
|
||||
|
||||
|
||||
async def _undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||
"""Explicit action, not gated by anything (there is no lockout state
|
||||
for this entity to gate on) — deletes Child/Local/Remote before their
|
||||
parent Connection (the ModelRelationField dependency direction), then
|
||||
the PreSharedKey (independent of the other 4), then reconfigures."""
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, IpsecTunnelDeployment, IpsecTunnelDeployment.tunnel_id == tunnel.id)
|
||||
failures = []
|
||||
result = await db.execute(select(IpsecTunnelDeployment).where(IpsecTunnelDeployment.tunnel_id == tunnel.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -5,7 +5,7 @@ fan out live calls to every firewall (an offline host would stall it past the
|
||||
scraper's timeout and a 15 s scrape interval would hammer the fleet). The
|
||||
inputs are what OPNexus' own pollers already store:
|
||||
|
||||
* host reachability and resources — the 60 s resource sampler (a host that
|
||||
* host reachability, resources, gateway health and service state — the 60 s resource sampler (a host that
|
||||
produced no sample for ``STALE_AFTER`` is reported down),
|
||||
* CARP role — the latest observed transition (carp_events),
|
||||
* open problems — the notification engine's problem table,
|
||||
@@ -19,11 +19,12 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.models.carp_event import CarpEvent
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import NotificationProblem
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.models.rollout import RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.routers.dashboard import collect_pending_actions
|
||||
from app.services import compliance_service
|
||||
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
||||
from app.version import APP_VERSION
|
||||
|
||||
CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
|
||||
@@ -99,6 +100,15 @@ async def render(db: AsyncSession) -> str:
|
||||
out.family("opnexus_host_memory_used_percent", "Memory in use, percent (last sample).")
|
||||
out.family("opnexus_host_load1", "1-minute load average (last sample).")
|
||||
out.family("opnexus_host_disk_used_percent", "Fullest disk, percent (last sample).")
|
||||
out.family("opnexus_host_swap_used_percent", "Swap in use, percent (last sample; absent without swap).")
|
||||
out.family("opnexus_host_gateway_up", "1 if the gateway is not down (online, degraded or not monitored), 0 if OPNsense reports it down.")
|
||||
out.family("opnexus_host_gateway_loss_percent", "Packet loss to the gateway monitor address, percent (last sample).")
|
||||
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
|
||||
out.family("opnexus_host_api_rights_missing", "1 if the firewall refused a side reading (read=gateways, services, swap, cpu) because the API key lacks the right (last sample); absent otherwise.")
|
||||
configured = (await notification_settings_service.get_settings(db)).service_watchlist
|
||||
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
|
||||
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
|
||||
for host in hosts:
|
||||
labels = {"host": host.name, "cluster": host.cluster_name or ""}
|
||||
@@ -115,6 +125,24 @@ async def render(db: AsyncSession) -> str:
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, value, **labels)
|
||||
if sample is not None:
|
||||
raw = sample.raw or {}
|
||||
if raw.get("swap_used_pct") is not None:
|
||||
out.sample("opnexus_host_swap_used_percent", raw["swap_used_pct"], **labels)
|
||||
for gw in gateway_health_service.from_compact(raw.get("gateways")):
|
||||
gw_labels = {**labels, "gateway": gw.name}
|
||||
out.sample("opnexus_host_gateway_up", 0 if gw.state == "offline" else 1, **gw_labels)
|
||||
for name, value, factor in (
|
||||
("opnexus_host_gateway_loss_percent", gw.loss, 1), ("opnexus_host_gateway_delay_seconds", gw.delay, 0.001),
|
||||
("opnexus_host_gateway_stddev_seconds", gw.stddev, 0.001),
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, round(value * factor, 6), **gw_labels)
|
||||
for read in sorted(str(r) for r in raw.get("denied") or []):
|
||||
out.sample("opnexus_host_api_rights_missing", 1, **labels, read=read)
|
||||
for key, running in sorted((raw.get("services") or {}).items()):
|
||||
if key.split("/")[0] in watchlist:
|
||||
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
|
||||
if carp_state.get(host.id) in ("MASTER", "BACKUP"):
|
||||
out.sample("opnexus_host_carp_master", 1 if carp_state[host.id] == "MASTER" else 0, **labels)
|
||||
|
||||
@@ -152,4 +180,8 @@ async def render(db: AsyncSession) -> str:
|
||||
pending = await db.scalar(select(func.count()).select_from(ScheduledMaintenance).where(ScheduledMaintenance.status == "pending"))
|
||||
out.sample("opnexus_scheduled_maintenance_pending", pending or 0)
|
||||
|
||||
out.family("opnexus_rollout_runs", "Fleet rollout runs by status (waiting_window, wave_running, paused, succeeded, ...).")
|
||||
for status, count in sorted((await db.execute(select(RolloutRun.status, func.count()).group_by(RolloutRun.status))).all()):
|
||||
out.sample("opnexus_rollout_runs", count, status=status)
|
||||
|
||||
return out.text()
|
||||
@@ -0,0 +1,291 @@
|
||||
"""Monitoring alerts (1.24): gateway down, resource thresholds, stopped services.
|
||||
|
||||
Same contract as notification_checks.py (each check returns {dedup_key: context} of what is
|
||||
bad right now), with one addition: they never judge a single reading. The inputs are the
|
||||
60-second samples the resource sampler stores (resource_monitor_service, HostResourceSample),
|
||||
and a problem needs the last N samples to agree before it opens AND before it closes again:
|
||||
|
||||
* all N bad -> problem,
|
||||
* all N good -> no problem,
|
||||
* anything in between -> keep the current state (open stays open, closed stays closed).
|
||||
|
||||
So a single spike never alerts, a flapping link does not open and close the alert every few
|
||||
minutes, and a sample that could not be read ("unknown") neither raises nor clears anything.
|
||||
A host without a fresh sample is not judged at all (check_host_offline reports it); its open
|
||||
problems are kept.
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service, notification_settings_service
|
||||
from app.services.notification_dedup import Problems
|
||||
|
||||
# The sampler runs every 60 s; three missed rounds mean the host is not reachable (same limit as metrics_service).
|
||||
STALE_AFTER = timedelta(seconds=180)
|
||||
# How far back a service must have been seen running before its absence is reported (samples, ~2 h at 60 s).
|
||||
SERVICE_EVIDENCE_SAMPLES = 120
|
||||
_DISK_TYPES = {"ufs", "zfs"}
|
||||
|
||||
|
||||
def verdict(flags: list[bool | None], confirm: int, is_open: bool) -> bool:
|
||||
"""flags: newest first; True = bad, False = good, None = unknown."""
|
||||
window = flags[:confirm]
|
||||
if len(window) >= confirm:
|
||||
if all(f is True for f in window):
|
||||
return True
|
||||
if all(f is False for f in window):
|
||||
return False
|
||||
return is_open
|
||||
|
||||
|
||||
async def _enabled_hosts(db: AsyncSession) -> list[OpnsenseHost]:
|
||||
return list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)))).all())
|
||||
|
||||
|
||||
async def _open_keys(db: AsyncSession, problem_type: str) -> set[str]:
|
||||
rows = await db.scalars(
|
||||
select(NotificationProblem.dedup_key).where(
|
||||
NotificationProblem.problem_type == problem_type, NotificationProblem.status == "open"
|
||||
)
|
||||
)
|
||||
return set(rows.all())
|
||||
|
||||
|
||||
async def _recent(db: AsyncSession, host_id, limit: int, *raw_keys: str) -> list[dict]:
|
||||
"""The newest ``limit`` samples of a host, newest first, with just the JSON keys asked for."""
|
||||
columns = [HostResourceSample.sampled_at, HostResourceSample.mem_used_pct, HostResourceSample.load_avg_1m]
|
||||
columns += [HostResourceSample.raw[key].as_json() for key in raw_keys]
|
||||
rows = (
|
||||
await db.execute(
|
||||
select(*columns).where(HostResourceSample.host_id == host_id).order_by(HostResourceSample.sampled_at.desc()).limit(limit)
|
||||
)
|
||||
).all()
|
||||
return [
|
||||
{"sampled_at": r[0], "mem_used_pct": r[1], "load_avg_1m": r[2], **{k: r[3 + i] for i, k in enumerate(raw_keys)}}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def _fresh(samples: list[dict]) -> bool:
|
||||
return bool(samples) and datetime.now(timezone.utc) - samples[0]["sampled_at"] <= STALE_AFTER
|
||||
|
||||
|
||||
def _keys_of_host(open_keys: set[str], host: OpnsenseHost) -> set[str]:
|
||||
prefix = f"{host.id}:"
|
||||
return {k[len(prefix):] for k in open_keys if k.startswith(prefix)}
|
||||
|
||||
|
||||
# ── Gateways ────────────────────────────────────────────────────────────────
|
||||
|
||||
def _gateway_reason(g) -> str:
|
||||
if g.state == "offline":
|
||||
return "Gateway offline"
|
||||
parts = []
|
||||
if g.loss is not None:
|
||||
parts.append(f"Paketverlust {g.loss:g} %")
|
||||
if g.delay is not None:
|
||||
parts.append(f"Latenz {g.delay:g} ms")
|
||||
return ", ".join(parts) or g.status
|
||||
|
||||
|
||||
async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
|
||||
muted = set(cfg.gateway_alert_muted or [])
|
||||
open_keys = await _open_keys(db, "gateway_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "gateways")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
parsed = [
|
||||
None if s["gateways"] is None else {g.name: g for g in gateway_health_service.from_compact(s["gateways"])}
|
||||
for s in samples
|
||||
]
|
||||
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
|
||||
for name in sorted(names):
|
||||
if f"{host.id}:{name}" in muted:
|
||||
continue # alert switched off for this gateway: not judged, an open problem of it resolves
|
||||
flags: list[bool | None] = []
|
||||
for p in parsed:
|
||||
if p is None:
|
||||
flags.append(None)
|
||||
else:
|
||||
g = p.get(name) # a gateway that no longer exists is "fine": its problem resolves
|
||||
flags.append(False if g is None else gateway_health_service.is_bad(g, loss_limit, delay_limit))
|
||||
key = f"{host.id}:{name}"
|
||||
if not verdict(flags, confirm, key in open_keys):
|
||||
continue
|
||||
latest = next((p[name] for p in parsed if p and name in p), None)
|
||||
problems[key] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "gateway": name,
|
||||
"grund": _gateway_reason(latest) if latest else "keine aktuellen Messwerte",
|
||||
"status": latest.status if latest else None,
|
||||
"paketverlust_prozent": latest.loss if latest else None,
|
||||
"latenz_ms": latest.delay if latest else None,
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── API rights ──────────────────────────────────────────────────────────────
|
||||
|
||||
# What the sampler reads on the side (resource_monitor_service) -> (ACL entry of the OPNsense user/group that
|
||||
# grants it, endpoint; the sample's "ok" list proves a reading went through). ACL names: opnsense/core
|
||||
# src/opnsense/mvc/app/models/OPNsense/Core/ACL/ACL.xml (page-system-gateways: api/routes/gateway/status;
|
||||
# page-status-services: api/core/service/*; page-system-login-logout "Lobby: Dashboard": api/diagnostics/cpu_usage/*
|
||||
# and api/diagnostics/system/system_swap).
|
||||
API_READS: dict[str, tuple[str, str]] = {
|
||||
"gateways": ("System: Gateways", "/api/routes/gateway/status"),
|
||||
"services": ("Status: Services", "/api/core/service/search"),
|
||||
"swap": ("Lobby: Dashboard", "/api/diagnostics/system/systemSwap"),
|
||||
"cpu": ("Lobby: Dashboard", "/api/diagnostics/cpu_usage/getCPUType"),
|
||||
}
|
||||
|
||||
|
||||
async def check_api_rights_missing(db: AsyncSession) -> dict[str, dict]:
|
||||
"""The API key of a firewall may be refused (403) on the side readings the alerts rely on (gateway state,
|
||||
service state): those alerts can then never fire, and nobody would notice. Only a refusal counts; a timeout
|
||||
or an error stays "unknown"."""
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
open_keys = await _open_keys(db, "api_rights_missing")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "denied", "ok")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
names = {n for s in samples for n in (s["denied"] or []) if n in API_READS} | (was_open & set(API_READS))
|
||||
for name in sorted(names):
|
||||
acl, path = API_READS[name]
|
||||
flags: list[bool | None] = []
|
||||
for s in samples:
|
||||
if s["denied"] is None:
|
||||
flags.append(None) # a sample from before this check existed
|
||||
elif name in s["denied"]:
|
||||
flags.append(True)
|
||||
else:
|
||||
# Only a reading that went through proves the right is back; a timeout or a sample from
|
||||
# before "ok" existed stays unknown (the problem neither opens nor closes on it).
|
||||
flags.append(False if s["ok"] is not None and name in s["ok"] else None)
|
||||
if verdict(flags, confirm, name in was_open):
|
||||
problems[f"{host.id}:{name}"] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "lesezugriff": name, "endpunkt": path,
|
||||
"fehlendes_recht": acl,
|
||||
"hinweis": f"Dem API-Benutzer fehlt das Recht, {path} zu lesen (OPNsense-Berechtigung \"{acl}\"). "
|
||||
"Alarme, die darauf beruhen, können nicht auslösen.",
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Resources ───────────────────────────────────────────────────────────────
|
||||
|
||||
def disk_groups(devices) -> dict[str, tuple[str, float]]:
|
||||
"""{group: (label, used %)} — one entry per ZFS pool / UFS device, the fullest dataset of a pool wins."""
|
||||
groups: dict[str, tuple[str, float]] = {}
|
||||
for dev in devices or []:
|
||||
if not isinstance(dev, dict) or dev.get("type") not in _DISK_TYPES:
|
||||
continue
|
||||
try:
|
||||
pct = float(dev["used_pct"])
|
||||
except (KeyError, TypeError, ValueError):
|
||||
continue
|
||||
device = str(dev.get("device") or dev.get("mountpoint") or "?")
|
||||
group = device.split("/")[0] if dev["type"] == "zfs" else device
|
||||
label = group if dev["type"] == "zfs" else str(dev.get("mountpoint") or device)
|
||||
if group not in groups or pct > groups[group][1]:
|
||||
groups[group] = (label, pct)
|
||||
return groups
|
||||
|
||||
|
||||
async def check_resource_threshold(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.resource_confirm_samples)
|
||||
open_keys = await _open_keys(db, "resource_threshold")
|
||||
problems = Problems()
|
||||
|
||||
def over(value, limit) -> bool | None:
|
||||
if not limit or value is None:
|
||||
return None if value is None else False
|
||||
return value >= limit
|
||||
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "disk", "swap_used_pct", "cpu_threads")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
base = {"host_id": str(host.id), "host_name": host.name}
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
|
||||
# Disk: one problem per pool/device.
|
||||
disk = [None if s["disk"] is None else disk_groups(s["disk"]) for s in samples]
|
||||
group_names = {g for d in disk if d for g in d} | {k[5:] for k in was_open if k.startswith("disk:")}
|
||||
for group in sorted(group_names):
|
||||
flags = [None if d is None else (over(d[group][1], cfg.disk_threshold_pct) if group in d else False) for d in disk]
|
||||
if verdict(flags, confirm, f"disk:{group}" in was_open):
|
||||
label, pct = next((d[group] for d in disk if d and group in d), (group, None))
|
||||
problems[f"{host.id}:disk:{group}"] = {
|
||||
**base, "ressource": f"Festplatte {label}", "wert_prozent": pct, "schwelle_prozent": cfg.disk_threshold_pct,
|
||||
}
|
||||
|
||||
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
|
||||
if verdict(flags, confirm, key in was_open):
|
||||
# An open alert stays open while the reading is unavailable: then there is no number to show (never "None %").
|
||||
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": None if value is None else f"{value}{unit}",
|
||||
"schwelle": f"{threshold}{unit}"}
|
||||
|
||||
latest = samples[0]
|
||||
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
|
||||
latest["mem_used_pct"], cfg.memory_threshold_pct, " %")
|
||||
simple("swap", "Swap", [over(s["swap_used_pct"], cfg.swap_threshold_pct) for s in samples],
|
||||
latest["swap_used_pct"], cfg.swap_threshold_pct, " %")
|
||||
|
||||
def per_cpu(s):
|
||||
if s["load_avg_1m"] is None or not s["cpu_threads"]:
|
||||
return None # thread count unknown (older firmware): the load cannot be judged
|
||||
return s["load_avg_1m"] / s["cpu_threads"]
|
||||
|
||||
simple("cpu", "CPU-Last (pro Thread)", [over(per_cpu(s), cfg.load_per_cpu_threshold) for s in samples],
|
||||
None if per_cpu(latest) is None else round(per_cpu(latest), 2), cfg.load_per_cpu_threshold, "")
|
||||
return problems
|
||||
|
||||
|
||||
# ── Services ────────────────────────────────────────────────────────────────
|
||||
|
||||
async def check_service_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
watch = set(cfg.service_watchlist if cfg.service_watchlist is not None else DEFAULT_SERVICE_WATCHLIST)
|
||||
open_keys = await _open_keys(db, "service_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm + SERVICE_EVIDENCE_SAMPLES, "services")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
listings = [s["services"] for s in samples]
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
candidates = {k for k in (listings[0] or {}) if k.split("/")[0] in watch} | was_open
|
||||
for svc in sorted(candidates):
|
||||
if svc.split("/")[0] not in watch:
|
||||
continue # taken off the watchlist: the problem resolves
|
||||
flags = [None if sv is None else (sv[svc] == 0 if svc in sv else False) for sv in listings]
|
||||
bad = verdict(flags, confirm, svc in was_open)
|
||||
# Only a service that was seen running before counts: one that never ran (enabled but idle,
|
||||
# e.g. DHCPv6 without a prefix) or was stopped on purpose long ago is not a failure.
|
||||
if bad and svc not in was_open and not any(sv and sv.get(svc) == 1 for sv in listings[confirm:]):
|
||||
bad = False
|
||||
if bad:
|
||||
problems[f"{host.id}:{svc}"] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "dienst": svc.split("/")[0],
|
||||
"hinweis": "Dienst läuft nicht mehr; absichtlich gestoppt? Dann aus der Überwachungsliste nehmen.",
|
||||
}
|
||||
return problems
|
||||
@@ -87,6 +87,13 @@ OBJECT_COLUMNS: dict[str, list[str]] = {
|
||||
# These change how the network is wired; the others (DNS/DHCP data) cannot cut the connection to the firewall.
|
||||
WIRING_KINDS = {"vlan", "vip", "gateway", "route", "interface", "lagg", "bridge", "gre", "gif", "vxlan"}
|
||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||
# Integer ranges of the firewall's own model (Interfaces/Vlan.xml, Interfaces/Vip.xml, read 2026-10-23). Checked before the
|
||||
# preview so a wrong number is a 422 with the field name, not a 502 after the firewall refused it (T6 of the 1.31.0 test
|
||||
# run). Only fields whose range is stated in the model; an empty value is left to the firewall (e.g. vhid of a non-CARP VIP).
|
||||
RANGES: dict[tuple[str, str], tuple[int, int]] = {
|
||||
("vlan", "tag"): (1, 4094), ("vlan", "pcp"): (0, 7),
|
||||
("vip", "vhid"): (1, 255), ("vip", "advbase"): (1, 254), ("vip", "advskew"): (0, 254),
|
||||
}
|
||||
|
||||
|
||||
def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
||||
@@ -104,6 +111,9 @@ def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
||||
text = str(value).strip()
|
||||
if len(text) > 255 or _CONTROL.search(text):
|
||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
||||
bounds = RANGES.get((kind, key))
|
||||
if bounds and text and not (text.isdecimal() and bounds[0] <= int(text) <= bounds[1]):
|
||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
||||
clean[key] = text
|
||||
return clean
|
||||
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Extra notification channels: ntfy, Gotify, Telegram, Microsoft Teams, Matrix.
|
||||
|
||||
They sit beside SMTP and the generic webhook (notification_delivery.py) and reuse its message
|
||||
text. Each channel is a row in ``notification_channels`` (see models/notification.py); the
|
||||
credential never touches the database: it is stored through notification_secret_store as
|
||||
``channel_<id>`` (mode 0600 file) and read only at send time. Where the credential is part of the
|
||||
URL (a Telegram bot token, the whole Teams webhook URL) the URL is built per request, and the
|
||||
HTTP request log is muted while it is in flight (``url_guard.muted``), so neither the logs nor an
|
||||
error message ever contain it.
|
||||
|
||||
API formats, checked against the vendors' documentation:
|
||||
|
||||
* ntfy — JSON publish: POST <server>/ with {"topic", "title", "message", "priority", "tags"}
|
||||
(docs.ntfy.sh/publish, "Publish as JSON"); optional ``Authorization: Bearer <token>``.
|
||||
* Gotify — POST <server>/message, header ``X-Gotify-Key: <app token>`` (keeps the token out of
|
||||
the URL), JSON {"title", "message", "priority"} (gotify.net/docs/pushmsg).
|
||||
* Telegram — POST <api>/bot<token>/sendMessage, JSON {"chat_id", "text"}; text limit 4096
|
||||
characters; answers {"ok": true, ...} (core.telegram.org/bots/api#sendmessage).
|
||||
* Teams — POST to the (Workflows / incoming) webhook URL with a message envelope carrying an
|
||||
Adaptive Card (learn.microsoft.com ... add-incoming-webhook); the URL is the secret.
|
||||
* Matrix — PUT <homeserver>/_matrix/client/v3/rooms/<roomId>/send/m.room.message/<txnId> with
|
||||
``Authorization: Bearer <access token>``, JSON {"msgtype": "m.text", "body"}; a fresh
|
||||
transaction id per message (Matrix client-server API, "Sending events to a room").
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import CHANNEL_TYPES, NotificationChannel, NotificationProblem
|
||||
from app.services import notification_delivery, notification_secret_store, url_guard
|
||||
from app.services.notification_delivery import NotificationDeliveryError
|
||||
|
||||
TELEGRAM_LIMIT = 4096
|
||||
TEAMS_LIMIT = 20000
|
||||
_NTFY_TOPIC = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||
_MATRIX_ROOM = re.compile(r"^![^\s:]+:[^\s/]+$")
|
||||
|
||||
# type -> (required config keys, secret required?)
|
||||
_SPEC = {
|
||||
"ntfy": ({"topic"}, False),
|
||||
"gotify": ({"server_url"}, True),
|
||||
"telegram": ({"chat_id"}, True),
|
||||
"teams": (set(), True),
|
||||
"matrix": ({"homeserver_url", "room_id"}, True),
|
||||
}
|
||||
assert set(_SPEC) == set(CHANNEL_TYPES)
|
||||
|
||||
|
||||
def secret_name(channel_id) -> str:
|
||||
return f"channel_{channel_id}"
|
||||
|
||||
|
||||
def clean_config(channel_type: str, config: dict) -> dict:
|
||||
"""Validates and normalises the non-secret settings of a channel. Raises ValueError (German reason)."""
|
||||
if channel_type not in _SPEC:
|
||||
raise ValueError("Unbekannter Kanaltyp")
|
||||
config = dict(config or {})
|
||||
out: dict = {}
|
||||
|
||||
def url(key: str, default: str | None = None) -> str | None:
|
||||
raw = str(config.get(key) or "").strip() or default
|
||||
if raw is None:
|
||||
return None
|
||||
return url_guard.validate_url(raw).rstrip("/")
|
||||
|
||||
def integer(key: str, low: int, high: int, default: int) -> int:
|
||||
value = config.get(key)
|
||||
if value in (None, ""):
|
||||
return default
|
||||
try:
|
||||
number = int(value)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError(f"{key}: ganze Zahl erwartet") from None
|
||||
if not low <= number <= high:
|
||||
raise ValueError(f"{key}: Wert zwischen {low} und {high} erwartet")
|
||||
return number
|
||||
|
||||
if channel_type == "ntfy":
|
||||
out["server_url"] = url("server_url", "https://ntfy.sh")
|
||||
topic = str(config.get("topic") or "").strip()
|
||||
if not _NTFY_TOPIC.match(topic):
|
||||
raise ValueError("Thema: 1–64 Zeichen aus Buchstaben, Ziffern, _ und -")
|
||||
out["topic"] = topic
|
||||
out["priority"] = integer("priority", 1, 5, 4)
|
||||
elif channel_type == "gotify":
|
||||
out["server_url"] = url("server_url")
|
||||
if not out["server_url"]:
|
||||
raise ValueError("Serveradresse fehlt")
|
||||
out["priority"] = integer("priority", 0, 10, 5)
|
||||
elif channel_type == "telegram":
|
||||
chat_id = str(config.get("chat_id") or "").strip()
|
||||
if not re.match(r"^(-?\d{1,20}|@[A-Za-z0-9_]{4,64})$", chat_id):
|
||||
raise ValueError("Chat-ID: Zahl (z. B. -1001234567890) oder @Kanalname")
|
||||
out["chat_id"] = chat_id
|
||||
out["api_base"] = url("api_base", "https://api.telegram.org")
|
||||
elif channel_type == "matrix":
|
||||
out["homeserver_url"] = url("homeserver_url")
|
||||
if not out["homeserver_url"]:
|
||||
raise ValueError("Homeserver-Adresse fehlt")
|
||||
room = str(config.get("room_id") or "").strip()
|
||||
if not _MATRIX_ROOM.match(room):
|
||||
raise ValueError("Raum-ID: Form !abc123:example.org (nicht der Alias #raum:…)")
|
||||
out["room_id"] = room
|
||||
return out
|
||||
|
||||
|
||||
def clean_secret(channel_type: str, secret: str) -> str:
|
||||
value = (secret or "").strip()
|
||||
if not value:
|
||||
raise ValueError("Zugangsdaten fehlen")
|
||||
if channel_type == "teams":
|
||||
value = url_guard.validate_url(value, max_length=2000)
|
||||
if not value.startswith("https://"):
|
||||
raise ValueError("Die Teams-Webhook-Adresse muss mit https:// beginnen")
|
||||
elif any(c.isspace() for c in value) or len(value) > 500:
|
||||
raise ValueError("Token: keine Leerzeichen, höchstens 500 Zeichen")
|
||||
return value
|
||||
|
||||
|
||||
def missing_requirements(channel: NotificationChannel) -> str | None:
|
||||
required, needs_secret = _SPEC[channel.channel_type]
|
||||
if any(not channel.config.get(k) for k in required):
|
||||
return "Konfiguration unvollständig"
|
||||
if needs_secret and not channel.secret_set:
|
||||
return "Zugangsdaten fehlen"
|
||||
return None
|
||||
|
||||
|
||||
# ── Message formatting (one function per channel, pure — unit-testable) ───────────
|
||||
|
||||
def _parts(problem: NotificationProblem, transition: str) -> tuple[str, str]:
|
||||
"""(title, details) — the title repeated at the top of render_message's body is dropped."""
|
||||
subject, body = notification_delivery.render_message(problem, transition)
|
||||
details = body.split("\n", 2)[2] if body.count("\n") >= 2 else ""
|
||||
return subject, details.strip()
|
||||
|
||||
|
||||
def _clip(text: str, limit: int) -> str:
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def format_ntfy(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
resolved = transition == "resolved"
|
||||
return {
|
||||
"topic": config["topic"], "title": title, "message": details or title,
|
||||
"priority": 3 if resolved else int(config.get("priority", 4)),
|
||||
"tags": ["white_check_mark"] if resolved else ["warning"],
|
||||
}
|
||||
|
||||
|
||||
def format_gotify(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {"title": title, "message": details or title, "priority": 1 if transition == "resolved" else int(config.get("priority", 5))}
|
||||
|
||||
|
||||
def format_telegram(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {
|
||||
"chat_id": config["chat_id"], "text": _clip(f"{title}\n\n{details}".strip(), TELEGRAM_LIMIT),
|
||||
"disable_web_page_preview": True,
|
||||
}
|
||||
|
||||
|
||||
def format_teams(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
body: list[dict] = [{"type": "TextBlock", "text": title, "weight": "Bolder", "size": "Medium", "wrap": True}]
|
||||
if details:
|
||||
body.append({"type": "TextBlock", "text": _clip(details, TEAMS_LIMIT), "wrap": True})
|
||||
return {
|
||||
"type": "message",
|
||||
"attachments": [{
|
||||
"contentType": "application/vnd.microsoft.card.adaptive", "contentUrl": None,
|
||||
"content": {
|
||||
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||
"type": "AdaptiveCard", "version": "1.4", "body": body,
|
||||
},
|
||||
}],
|
||||
}
|
||||
|
||||
|
||||
def format_matrix(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {"msgtype": "m.text", "body": f"{title}\n\n{details}".strip()}
|
||||
|
||||
|
||||
# ── Sending ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _request_for(channel: NotificationChannel, secret: str, problem, transition: str) -> tuple[str, str, dict, dict, str | None]:
|
||||
"""(method, url, headers, json body, expected "ok" check key)."""
|
||||
cfg, kind = channel.config, channel.channel_type
|
||||
if kind == "ntfy":
|
||||
headers = {"Authorization": f"Bearer {secret}"} if secret else {}
|
||||
return "POST", cfg["server_url"].rstrip("/") + "/", headers, format_ntfy(cfg, problem, transition), None
|
||||
if kind == "gotify":
|
||||
return "POST", cfg["server_url"].rstrip("/") + "/message", {"X-Gotify-Key": secret}, format_gotify(cfg, problem, transition), None
|
||||
if kind == "telegram":
|
||||
return "POST", f"{cfg['api_base'].rstrip('/')}/bot{secret}/sendMessage", {}, format_telegram(cfg, problem, transition), "ok"
|
||||
if kind == "teams":
|
||||
return "POST", secret, {}, format_teams(cfg, problem, transition), None
|
||||
room = quote(cfg["room_id"], safe="")
|
||||
url = f"{cfg['homeserver_url'].rstrip('/')}/_matrix/client/v3/rooms/{room}/send/m.room.message/{uuid.uuid4().hex}"
|
||||
return "PUT", url, {"Authorization": f"Bearer {secret}"}, format_matrix(cfg, problem, transition), None
|
||||
|
||||
|
||||
async def send(channel: NotificationChannel, problem: NotificationProblem, transition: str) -> None:
|
||||
problem_missing = missing_requirements(channel)
|
||||
if problem_missing:
|
||||
raise NotificationDeliveryError(f"Kanal „{channel.name}“: {problem_missing}")
|
||||
secret = notification_secret_store.read_secret(secret_name(channel.id)) or ""
|
||||
if channel.secret_set and not secret:
|
||||
raise NotificationDeliveryError(f"Kanal „{channel.name}“: Zugangsdaten nicht lesbar – bitte neu setzen")
|
||||
method, url, headers, payload, ok_key = _request_for(channel, secret, problem, transition)
|
||||
label = f"Kanal „{channel.name}“ ({channel.channel_type})"
|
||||
with url_guard.muted():
|
||||
try:
|
||||
url_guard.validate_url(url, max_length=2100)
|
||||
await url_guard.ensure_resolved_safe(url)
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
||||
try:
|
||||
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
resp = await client.request(method, url, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
if ok_key and not (resp.json() or {}).get(ok_key):
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
|
||||
reason = url_guard.failure_reason(exc)
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
|
||||
except ValueError:
|
||||
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
|
||||
|
||||
|
||||
async def send_test(channel: NotificationChannel) -> None:
|
||||
fake = NotificationProblem(
|
||||
problem_type="test", dedup_key="test",
|
||||
context={"message": "Dies ist eine Testbenachrichtigung von OPNexus."},
|
||||
)
|
||||
await send(channel, fake, "opened")
|
||||
|
||||
|
||||
async def enabled_channels(db: AsyncSession) -> list[NotificationChannel]:
|
||||
rows = await db.scalars(select(NotificationChannel).where(NotificationChannel.enabled.is_(True)).order_by(NotificationChannel.name))
|
||||
return list(rows.all())
|
||||
@@ -201,6 +201,62 @@ async def check_revert_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"host_name": host.name if host else None,
|
||||
"message": deployment.message,
|
||||
}
|
||||
problems.update(await _cleanup_problems(db))
|
||||
return problems
|
||||
|
||||
|
||||
# Rule cleanup (1.31) states that need a human: the automatic revert failed, or a run ended half-way.
|
||||
_CLEANUP_STILL_CHANGED = ("revert_failed", "unknown", "disabling", "disabled", "failed")
|
||||
|
||||
|
||||
async def _cleanup_problems(db: AsyncSession) -> dict[str, dict]:
|
||||
"""Rule cleanups whose automatic revert failed (per rule that could not be switched back) and runs stuck in
|
||||
``partial``/``apply_failed`` (a decision is needed before the window ends), plus delete/restore runs with an
|
||||
unresolved item. Same ``revert_failed`` alert as the deployments. Keys: ``rule_cleanup:<item id>`` for one rule,
|
||||
``rule_cleanup_op:<operation id>`` for a whole run, so one open problem is one message."""
|
||||
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
|
||||
|
||||
problems: dict[str, dict] = {}
|
||||
operations = (await db.scalars(select(RuleCleanupOperation).where(
|
||||
RuleCleanupOperation.status.in_(("revert_failed", "partial", "apply_failed", "failed"))))).all()
|
||||
for op in operations:
|
||||
items = list((await db.scalars(select(RuleCleanupItem).where(RuleCleanupItem.operation_id == op.id)
|
||||
.order_by(RuleCleanupItem.seq))).all())
|
||||
host = await db.get(OpnsenseHost, op.host_id)
|
||||
base = {
|
||||
"deployment_type": "rule_cleanup", "deployment_id": str(op.id), "host_id": str(op.host_id),
|
||||
"host_name": host.name if host else None, "operation": op.kind, "operation_status": op.status,
|
||||
}
|
||||
if op.status == "revert_failed":
|
||||
stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]
|
||||
for item in stuck:
|
||||
member = await db.get(OpnsenseHost, item.host_id)
|
||||
problems[f"rule_cleanup:{item.id}"] = {
|
||||
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||
"message": item.message or op.message,
|
||||
"hinweis": "Bereinigung: Die Regel konnte nicht automatisch zurückgesetzt werden und ist möglicherweise "
|
||||
"noch geändert. Auf der Firewall prüfen und in OPNexus „Erneut zurücknehmen“ wählen.",
|
||||
}
|
||||
if not stuck:
|
||||
problems[f"rule_cleanup_op:{op.id}"] = {**base, "entity_id": str(op.id), "message": op.message,
|
||||
"hinweis": "Bereinigung: Das automatische Zurücknehmen ist fehlgeschlagen. Zustand der Firewall prüfen."}
|
||||
elif op.status in ("partial", "apply_failed"):
|
||||
problems[f"rule_cleanup_op:{op.id}"] = {
|
||||
**base, "entity_id": str(op.id), "message": op.message,
|
||||
"regeln": ", ".join(i.rule_uuid for i in items if i.state in ("failed", "unknown")) or None,
|
||||
"hinweis": "Bereinigung nur teilweise ausgeführt (oder das Anwenden ist fehlgeschlagen). Bestätigen ist gesperrt; "
|
||||
"jetzt zurücknehmen oder die automatische Rücknahme am Fristende abwarten.",
|
||||
}
|
||||
elif op.kind in ("delete", "restore"):
|
||||
for item in (i for i in items if i.state == "unknown"):
|
||||
member = await db.get(OpnsenseHost, item.host_id)
|
||||
problems[f"rule_cleanup:{item.id}"] = {
|
||||
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||
"message": item.message or op.message,
|
||||
"hinweis": "Bereinigung: Der Ausgang beim Löschen/Wiederherstellen ist unklar. Mit „Erneut prüfen“ nachlesen.",
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
@@ -363,3 +419,49 @@ async def check_compliance(db: AsyncSession) -> dict[str, dict]:
|
||||
"beispiele": ", ".join(params.get("examples", [])) or None,
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Rollouts (1.25) ─────────────────────────────────────────────────────────
|
||||
|
||||
def _short(detail) -> str | None:
|
||||
if not detail:
|
||||
return None
|
||||
parts = []
|
||||
for key, value in detail.items():
|
||||
if isinstance(value, list):
|
||||
value = ", ".join(str(v) for v in value[:8])
|
||||
parts.append(f"{key}: {value}")
|
||||
return "; ".join(parts)[:500]
|
||||
|
||||
|
||||
async def check_rollout_paused(db: AsyncSession) -> dict[str, dict]:
|
||||
"""Open while a rollout run stands still waiting for a human (gate, failed host, window gone...)."""
|
||||
from app.models.rollout import RolloutRun
|
||||
problems: dict[str, dict] = {}
|
||||
for run in (await db.scalars(select(RolloutRun).where(RolloutRun.status == "paused"))).all():
|
||||
wave = next((w.label for w in run.waves if w.index == run.current_wave), None)
|
||||
problems[str(run.id)] = {
|
||||
"run_id": str(run.id), "plan": run.plan_name, "grund": run.pause_reason, "welle": wave,
|
||||
"zielversion": run.target_version, "details": _short(run.pause_detail),
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
ROLLOUT_START_FAILURE_WINDOW = timedelta(days=3)
|
||||
|
||||
|
||||
async def check_rollout_start_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"""An automatic start at a window occurrence did not happen (event: closed without a 'resolved' notice)."""
|
||||
from app.models.rollout import RolloutPlan
|
||||
since = datetime.now(timezone.utc) - ROLLOUT_START_FAILURE_WINDOW
|
||||
problems: dict[str, dict] = {}
|
||||
for plan in (await db.scalars(select(RolloutPlan).where(RolloutPlan.auto_start_error.is_not(None)))).all():
|
||||
error = plan.auto_start_error or {}
|
||||
at = datetime.fromisoformat(error["at"]) if error.get("at") else None
|
||||
if at is None or at < since:
|
||||
continue
|
||||
problems[f"{plan.id}:{error.get('occurrence')}"] = {
|
||||
"plan_id": str(plan.id), "plan": plan.name, "grund": error.get("code"), "termin": error.get("occurrence"),
|
||||
"details": _short(error.get("params")),
|
||||
}
|
||||
return problems
|
||||
@@ -16,7 +16,7 @@ import httpx
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
|
||||
|
||||
class NotificationDeliveryError(Exception):
|
||||
@@ -37,6 +37,13 @@ _LABELS = {
|
||||
"backup_stale": "Kein aktuelles Backup",
|
||||
"backup_offsite_failed": "Offsite-Backup fehlgeschlagen",
|
||||
"compliance_finding": "Sicherheitsbefund",
|
||||
"gateway_down": "Gateway gestört",
|
||||
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
||||
"service_down": "Dienst ausgefallen",
|
||||
"api_rights_missing": "API-Rechte fehlen",
|
||||
"rollout_paused": "Rollout angehalten",
|
||||
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
||||
"test": "Testbenachrichtigung",
|
||||
}
|
||||
|
||||
|
||||
@@ -84,13 +91,20 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
|
||||
headers["X-OPNexus-Signature"] = f"sha256={signature}"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout) as client:
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
url_guard.validate_url(settings_row.webhook_url)
|
||||
await url_guard.ensure_resolved_safe(settings_row.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
||||
try:
|
||||
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
||||
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
with url_guard.muted():
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which often is the credential itself
|
||||
# (token in path/query) — keep it out of messages and logs.
|
||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
||||
reason = url_guard.failure_reason(exc)
|
||||
raise NotificationDeliveryError(f"Webhook-Zustellung fehlgeschlagen ({reason})") from None
|
||||
|
||||
|
||||
|
||||
@@ -3,13 +3,15 @@ against previously open problems, and delivers on transitions. The single
|
||||
testable entry point — called directly by main.py's background loop AND
|
||||
directly by tests, same pattern as rule_deploy_service.sweep_expired_deployments.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_checks, notification_dedup, notification_delivery
|
||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||
from app.services import monitor_checks, notification_channels, notification_checks, notification_dedup, notification_delivery
|
||||
from app.services import notification_settings_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -34,41 +36,108 @@ _CHECKS = [
|
||||
("backup_stale", "backup_failed_enabled", notification_checks.check_backup_stale),
|
||||
("backup_offsite_failed", "backup_failed_enabled", notification_checks.check_backup_offsite),
|
||||
("compliance_finding", "compliance_alerts_enabled", notification_checks.check_compliance),
|
||||
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
|
||||
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
||||
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
||||
("api_rights_missing", "api_rights_enabled", monitor_checks.check_api_rights_missing),
|
||||
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
||||
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
||||
]
|
||||
|
||||
# Event-like problems: when they drop out of the window nothing was "fixed", so no
|
||||
# "behoben" message goes out for them.
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed"}
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
|
||||
|
||||
|
||||
async def _deliver(settings_row: NotificationSettings, problem: NotificationProblem, transition: str) -> None:
|
||||
# Delivery limits (1.29): a hanging channel must neither delay the others nor hold up the check loop.
|
||||
DELIVERY_TIMEOUT_SECONDS = 20.0 # one target (webhook, e-mail, one channel) for one message
|
||||
DELIVERY_CYCLE_BUDGET_SECONDS = 90.0 # everything one check cycle sends, in total
|
||||
DELIVERY_PARALLEL = 10 # targets served at the same time within a cycle
|
||||
|
||||
|
||||
async def _attempt(label: str, make_call, gate: asyncio.Semaphore | None = None, started: list | None = None) -> None:
|
||||
"""One delivery target: its own timeout, its own failure — never an exception for the caller. Texts logged
|
||||
here come from NotificationDeliveryError (names the channel and the HTTP status, never the URL or token)."""
|
||||
try:
|
||||
if gate is not None:
|
||||
await gate.acquire()
|
||||
if started is not None:
|
||||
started.append(label) # the target was really tried (a cancelled wait for the gate does not count)
|
||||
try:
|
||||
await asyncio.wait_for(make_call(), DELIVERY_TIMEOUT_SECONDS)
|
||||
finally:
|
||||
if gate is not None:
|
||||
gate.release()
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("%s delivery failed", label)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning("%s delivery timed out after %.0f s", label, DELIVERY_TIMEOUT_SECONDS)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc: # a broken target must not take the others down
|
||||
logger.warning("%s delivery failed unexpectedly (%s)", label, type(exc).__name__)
|
||||
|
||||
|
||||
async def _deliver(
|
||||
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
|
||||
gate: asyncio.Semaphore | None = None,
|
||||
) -> None:
|
||||
"""Best-effort, not an audit trail: attempted once per transition and
|
||||
never retried, even on failure — avoids retry-storming a broken webhook
|
||||
URL or SMTP relay every poll cycle."""
|
||||
URL or SMTP relay every poll cycle. Webhook, e-mail and every channel are
|
||||
served in parallel, each with its own timeout (the order has no meaning)."""
|
||||
if transition == "resolved" and problem.problem_type in _NO_RESOLVED_NOTICE:
|
||||
problem.resolved_notified_at = _now()
|
||||
return
|
||||
where = f"{problem.problem_type}/{problem.dedup_key}"
|
||||
calls = []
|
||||
started: list = []
|
||||
if settings_row.webhook_enabled:
|
||||
try:
|
||||
await notification_delivery.send_webhook(settings_row, problem, transition)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("Webhook delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
||||
calls.append(_attempt(f"Webhook for {where}", lambda: notification_delivery.send_webhook(settings_row, problem, transition), gate, started))
|
||||
if settings_row.email_enabled:
|
||||
try:
|
||||
async def email():
|
||||
subject, body = notification_delivery.render_message(problem, transition)
|
||||
await notification_delivery.send_email(settings_row, subject, body)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("Email delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
||||
calls.append(_attempt(f"Email for {where}", email, gate, started))
|
||||
for channel in channels:
|
||||
calls.append(_attempt(f"Channel {channel.name} ({channel.channel_type}) for {where}",
|
||||
lambda channel=channel: notification_channels.send(channel, problem, transition), gate, started))
|
||||
try:
|
||||
await asyncio.gather(*calls, return_exceptions=True)
|
||||
finally:
|
||||
# "Attempted" is what is recorded (also when a target failed or timed out). A message the cycle budget
|
||||
# cancelled before ANY target was tried stays unrecorded: an opened problem is sent again next cycle.
|
||||
if started or not calls:
|
||||
if transition == "resolved":
|
||||
problem.resolved_notified_at = _now()
|
||||
else:
|
||||
problem.last_notified_at = _now()
|
||||
|
||||
if transition == "resolved":
|
||||
problem.resolved_notified_at = _now()
|
||||
else:
|
||||
problem.last_notified_at = _now()
|
||||
|
||||
async def _deliver_all(settings_row: NotificationSettings, pending: list[tuple[NotificationProblem, str]],
|
||||
channels: list[NotificationChannel]) -> None:
|
||||
"""Everything this cycle found, in parallel, within DELIVERY_CYCLE_BUDGET_SECONDS in total."""
|
||||
if not pending:
|
||||
return
|
||||
gate = asyncio.Semaphore(DELIVERY_PARALLEL)
|
||||
tasks = [asyncio.ensure_future(_deliver(settings_row, problem, transition, channels, gate)) for problem, transition in pending]
|
||||
done, late = await asyncio.wait(tasks, timeout=DELIVERY_CYCLE_BUDGET_SECONDS)
|
||||
for task in late:
|
||||
task.cancel()
|
||||
if late:
|
||||
logger.warning("Notification delivery stopped after %.0f s: %d of %d messages were not completed",
|
||||
DELIVERY_CYCLE_BUDGET_SECONDS, len(late), len(tasks))
|
||||
await asyncio.gather(*late, return_exceptions=True) # let the cancelled ones record "attempted"
|
||||
for task in done:
|
||||
if not task.cancelled() and task.exception() is not None:
|
||||
logger.warning("Notification delivery failed unexpectedly (%s)", type(task.exception()).__name__)
|
||||
|
||||
|
||||
async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
settings_row = await notification_settings_service.get_settings(db)
|
||||
result = {"opened": 0, "resolved": 0}
|
||||
channels = await notification_channels.enabled_channels(db)
|
||||
pending: list[tuple[NotificationProblem, str]] = []
|
||||
|
||||
for problem_type, enabled_flag, check_fn in _CHECKS:
|
||||
if not getattr(settings_row, enabled_flag):
|
||||
@@ -84,12 +153,21 @@ async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
continue
|
||||
|
||||
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
|
||||
for problem in opened:
|
||||
await _deliver(settings_row, problem, "opened")
|
||||
for problem in resolved:
|
||||
await _deliver(settings_row, problem, "resolved")
|
||||
pending += [(problem, "opened") for problem in opened] + [(problem, "resolved") for problem in resolved]
|
||||
result["opened"] += len(opened)
|
||||
result["resolved"] += len(resolved)
|
||||
|
||||
# Open problems that no target was ever tried for (an earlier cycle ran out of time) are sent now.
|
||||
enabled_types = [t for t, flag, _fn in _CHECKS if getattr(settings_row, flag)]
|
||||
already = {id(problem) for problem, _transition in pending}
|
||||
undelivered = (await db.scalars(
|
||||
select(NotificationProblem).where(
|
||||
NotificationProblem.status == "open", NotificationProblem.last_notified_at.is_(None),
|
||||
NotificationProblem.problem_type.in_(enabled_types),
|
||||
)
|
||||
)).all() if enabled_types else []
|
||||
pending += [(problem, "opened") for problem in undelivered if id(problem) not in already]
|
||||
# Delivered once all checks ran (and in parallel): a slow target no longer holds up the checks after it.
|
||||
await _deliver_all(settings_row, pending, channels)
|
||||
await db.commit()
|
||||
return result
|
||||
@@ -26,6 +26,10 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
|
||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
|
||||
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
||||
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled", "api_rights_enabled",
|
||||
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
||||
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
||||
"service_watchlist",
|
||||
"poll_interval_seconds",
|
||||
]
|
||||
for field in simple_fields:
|
||||
|
||||
@@ -0,0 +1,627 @@
|
||||
"""Takes over aliases, port forwards, 1:1 NAT and outbound NAT rules that were created by hand on a
|
||||
firewall, so they can be edited, versioned and deployed from OPNexus like any other object. This is
|
||||
the generalisation of rule_adopt_service.py (filter rules keep their own module: their model has
|
||||
more special cases and the code is relied upon).
|
||||
|
||||
Adoption itself writes NOTHING to the firewall. It records the object as it is right now (read fresh
|
||||
from the host at adoption time, never from what the browser sent) together with its OPNsense UUID
|
||||
(`remote_uuid` = the deployment's opnsense_*_uuid), and stores the payload OPNexus would send for it as
|
||||
the deployment's last_payload. So the preview directly afterwards says "unchanged", a later deploy
|
||||
updates the same object via setItem/setRule (no duplicate), and the normal preview → push →
|
||||
confirm/auto-revert flow applies from the first change on.
|
||||
|
||||
An object that already exists centrally for another firewall is LINKED instead of adopted a second time
|
||||
(adopt_link_service.py: exact match after resolving the host's variables, never automatic for a mere
|
||||
resemblance).
|
||||
|
||||
Only objects OPNexus can describe without distortion are adopted. Anything the model cannot express
|
||||
is refused with machine-readable reasons (the UI translates them) instead of being imported in a way
|
||||
that would silently change its meaning on the next edit. Fields OPNexus does not manage (log flag of a
|
||||
port forward, pool options, tags, categories, ...) are never sent, and OPNsense's set*/setItem only
|
||||
overwrites the posted fields (ApiMutableModelControllerBase::setBase -> setNodes), so they stay as
|
||||
they are; the candidate list shows them as `kept`.
|
||||
|
||||
Evidence (OPNsense core master, src/opnsense/mvc/app/…, read 2026-10-20, and live against OPNsense
|
||||
26.7 on test-fw-3 on the same day with hand-made objects of every case below):
|
||||
|
||||
* Aliases — controllers/OPNsense/Firewall/Api/AliasController.php::searchItemAction -> searchBase
|
||||
("aliases.alias"), model models/OPNsense/Firewall/Alias.xml. Row keys used: uuid, name, type
|
||||
("host", "network", "port", "networkgroup", "mac", "asn", plus the unsupported "url", "urltable",
|
||||
"urljson", "geoip", "dynipv6host", "authgroup", and the system types "internal"/"external"),
|
||||
enabled "1"/"0", content (entries separated by "\n", live-verified; for "authgroup" the controller
|
||||
rewrites it to group names), description, proto (only meaningful for asn/geoip). The built-in rows
|
||||
are bogons, bogonsv6, sshlockout, virusprot (type external) and one __<if>_network per interface
|
||||
(type internal); they are OPNsense's own and never adoptable. A "networkgroup" lists alias NAMES;
|
||||
host/network/port aliases may also contain alias names (nesting).
|
||||
* Port forward — controllers/OPNsense/Firewall/Api/DNatController.php::searchRuleAction merges the
|
||||
configured rules (getFlatNodes: dotted keys "source.network", "source.port", "source.not",
|
||||
"destination.network", "destination.port", "destination.not", plus descr, disabled, interface,
|
||||
ipprotocol, protocol, target, local-port, natreflection, nordr, pass, log, poolopts, tag, tagged,
|
||||
nosync, categories, sequence) with synthetic automatic rows (uuid "automatic_*"/"lockout_*",
|
||||
is_automatic true) that setRule/delRule cannot target. models/OPNsense/Firewall/DNat.xml: interface is
|
||||
Multiple=Y (comma separated), ipprotocol may be blank/inet/inet6/inet46, protocol blank = any,
|
||||
`nordr` ("no redirect") requires an empty target/local-port (live: "Target must be empty when No
|
||||
RDR is set"), `pass` ("" manual / "pass" / "rule") links an associated filter rule, natreflection is
|
||||
""/purenat/disable. OPNexus sends pass "" and not "0", so a rule using one of those would change
|
||||
meaning on its first edit and is refused.
|
||||
* 1:1 NAT — OneToOneController::searchRuleAction (flat keys), model Filter.xml onetoone.rule:
|
||||
enabled, log, interface, type (binat/nat), source_net/source_not, destination_net/destination_not,
|
||||
external, natreflection ""/enable/disable, description. All of these are managed by OPNexus.
|
||||
* Outbound NAT — SourceNatController::searchRuleAction returns the configured rules ONLY when the
|
||||
global mode (filter.settings.nat.snat_mode) is hybrid or advanced; in automatic/disabled mode the
|
||||
manual rules do not show up at all, so adopting is refused up front (outbound_nat_mode.py) instead
|
||||
of presenting an empty list. Model Filter.xml snatrules.rule: `nonat` rules ("do not NAT") are not
|
||||
expressible and refused; staticnatport, poolopts, log, tag, tagged, nosync, endpoint-independent
|
||||
are left alone.
|
||||
"""
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Awaitable, Callable, get_args
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.one_to_one_deployment import OneToOneDeployment
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.port_forward_deployment import PortForwardDeployment
|
||||
from app.schemas.alias import AliasType
|
||||
from app.services import alias_deploy_service, alias_policy_service
|
||||
from app.services import one_to_one_deploy_service, one_to_one_policy_service
|
||||
from app.services import opnsense_client, outbound_nat_deploy_service, outbound_nat_policy_service
|
||||
from app.services import port_forward_deploy_service, port_forward_policy_service
|
||||
from app.services import adopt_link_service
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on
|
||||
from app.services.nat_live_service import split_rows
|
||||
from app.services.outbound_nat_mode import host_mode
|
||||
|
||||
ADOPT_NOTE = "Von Firewall übernommen"
|
||||
ADOPT_MESSAGE = "Von der Firewall übernommen"
|
||||
|
||||
ALIAS_NAME = re.compile(r"^[A-Za-z0-9_]+$") # schemas/alias.py AliasCreate.name
|
||||
ALIAS_TYPES = frozenset(get_args(AliasType))
|
||||
ALIAS_CONTENT_MAX = 8000 # schemas/alias.py AliasCreate.content
|
||||
# OPNsense's own aliases that are not user objects (Firewall/Alias.php: bogons & co. are `external`
|
||||
# rows created by the system, __<if>_network are `internal`).
|
||||
SYSTEM_ALIAS_NAMES = frozenset({"bogons", "bogonsv6", "sshlockout", "virusprot"})
|
||||
LINK_TOLERATED = frozenset({"name_exists", "references_unmanaged_alias"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entry:
|
||||
row: dict
|
||||
reasons: list[str] = field(default_factory=list)
|
||||
details: dict[str, str] = field(default_factory=dict)
|
||||
bulk: bool = False # not adoptable alone yet, but adoptable in the same run (its references come first)
|
||||
matches: list = field(default_factory=list) # central objects equal to the live object (link candidates)
|
||||
similar: list = field(default_factory=list) # same name/description but different content (hint only)
|
||||
|
||||
@property
|
||||
def linkable(self) -> bool:
|
||||
"""An exact central twin exists and nothing but "the name is taken" / "its references are not
|
||||
managed here yet" speaks against it (both are moot when the twin is linked, its content is equal)."""
|
||||
return any(not m.busy for m in self.matches) and set(self.reasons) <= LINK_TOLERATED
|
||||
|
||||
@property
|
||||
def busy_twin(self) -> bool:
|
||||
"""The one exact twin waits for a confirmation: \"adopt all\" leaves this entry alone (neither link nor duplicate)."""
|
||||
return len(self.matches) == 1 and self.matches[0].busy
|
||||
|
||||
@property
|
||||
def uuid(self) -> str:
|
||||
return clean(self.row.get("uuid"))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
key: str
|
||||
model: type
|
||||
deployment: type
|
||||
fk: str # deployment column pointing at the central object
|
||||
uuid_attr: str # deployment column holding the OPNsense UUID (= remote_uuid)
|
||||
fetch: Callable[[OpnsenseHost], Awaitable[list[dict]]]
|
||||
check: Callable[[dict, dict], tuple[list[str], dict[str, str]]]
|
||||
title: Callable[[dict], str]
|
||||
info: Callable[[dict], dict[str, str]]
|
||||
build: Callable[[dict], dict]
|
||||
payload: Callable[[Any], dict]
|
||||
revision: Callable
|
||||
kept: Callable[[dict], list[str]] = lambda row: []
|
||||
context: Callable[[AsyncSession, list[dict]], Awaitable[dict]] | None = None
|
||||
refs: Callable[[dict], set[str]] | None = None
|
||||
name: Callable[[dict], str] | None = None
|
||||
identity: str = "description" # attribute that names the object (the alias name for aliases)
|
||||
|
||||
@property
|
||||
def link_spec(self) -> adopt_link_service.LinkSpec:
|
||||
return adopt_link_service.LinkSpec(
|
||||
model=self.model, deployment=self.deployment, fk=self.fk, uuid_attr=self.uuid_attr,
|
||||
identity=self.identity, payload=self.payload, revision=self.revision,
|
||||
)
|
||||
|
||||
|
||||
# ── shared helpers ───────────────────────────────────────────────────────
|
||||
|
||||
def _single_interface(row: dict) -> bool:
|
||||
interface = clean(row.get("interface"))
|
||||
return bool(interface) and "," not in interface
|
||||
|
||||
|
||||
def _too_long(kind_model: type, values: dict) -> list[str]:
|
||||
"""Columns are bounded; a live value that does not fit would fail (or be cut) on the first edit."""
|
||||
columns = kind_model.__table__.columns
|
||||
return [name for name, value in values.items()
|
||||
if name in columns and getattr(columns[name].type, "length", None)
|
||||
and isinstance(value, str) and len(value) > columns[name].type.length]
|
||||
|
||||
|
||||
def _port(value) -> str | None:
|
||||
return clean(value) or None
|
||||
|
||||
|
||||
def _kept(row: dict, keys: tuple[str, ...]) -> list[str]:
|
||||
return [key for key in keys if clean(row.get(key)) not in ("", "0")]
|
||||
|
||||
|
||||
def _fetch_nat(search) -> Callable[[OpnsenseHost], Awaitable[list[dict]]]:
|
||||
async def fetch(host: OpnsenseHost) -> list[dict]:
|
||||
data = await search(host)
|
||||
# split_rows drops automatic_*/lockout_* and is_automatic rows: they are OPNsense's own
|
||||
return [r for r in split_rows(data)["rules"] if clean(r.get("uuid"))]
|
||||
return fetch
|
||||
|
||||
|
||||
# ── aliases ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _alias_content(row: dict) -> str:
|
||||
# same normalisation as AliasCreate.clean_content
|
||||
return "\n".join(line.strip() for line in str(row.get("content") or "").splitlines() if line.strip())
|
||||
|
||||
|
||||
def _alias_is_system(row: dict) -> bool:
|
||||
name = clean(row.get("name"))
|
||||
return clean(row.get("type")) == "internal" or name.startswith("__") or name in SYSTEM_ALIAS_NAMES
|
||||
|
||||
|
||||
async def _fetch_aliases(host: OpnsenseHost) -> list[dict]:
|
||||
data = await opnsense_client.search_aliases(host)
|
||||
rows = data.get("rows", []) if isinstance(data, dict) else []
|
||||
return [r for r in rows if isinstance(r, dict) and clean(r.get("uuid")) and clean(r.get("name"))]
|
||||
|
||||
|
||||
async def _alias_context(db: AsyncSession, rows: list[dict]) -> dict:
|
||||
return {"central_names": set((await db.scalars(select(FirewallAlias.name))).all())}
|
||||
|
||||
|
||||
def _alias_refs(row: dict) -> set[str]:
|
||||
return {line.lstrip("!").strip() for line in _alias_content(row).splitlines()} - {""}
|
||||
|
||||
|
||||
def _alias_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
if _alias_is_system(row):
|
||||
return ["system"], {}
|
||||
reasons: list[str] = []
|
||||
details: dict[str, str] = {}
|
||||
name, type_ = clean(row.get("name")), clean(row.get("type"))
|
||||
if type_ not in ALIAS_TYPES:
|
||||
reasons.append("unsupported_type")
|
||||
details["unsupported_type"] = clean(row.get("%type")) or type_
|
||||
if not ALIAS_NAME.fullmatch(name) or len(name) > 64:
|
||||
reasons.append("name_invalid")
|
||||
if name in ctx["central_names"]:
|
||||
reasons.append("name_exists")
|
||||
if len(_alias_content(row)) > ALIAS_CONTENT_MAX:
|
||||
reasons.append("content_too_large")
|
||||
if type_ == "asn" and clean(row.get("proto")):
|
||||
reasons.append("proto_set") # OPNexus has no field for it: a deploy elsewhere would drop the restriction
|
||||
if _too_long(FirewallAlias, {"description": clean(row.get("description"))}):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, details
|
||||
|
||||
|
||||
def _alias_info(row: dict) -> dict[str, str]:
|
||||
lines = _alias_content(row).splitlines()
|
||||
return {
|
||||
"type": clean(row.get("type")),
|
||||
"type_label": clean(row.get("%type")) or clean(row.get("type")),
|
||||
"description": clean(row.get("description")),
|
||||
"content": "\n".join(lines[:6]) + ("\n…" if len(lines) > 6 else ""),
|
||||
"entries": str(len(lines)),
|
||||
}
|
||||
|
||||
|
||||
def _alias_build(row: dict) -> dict:
|
||||
return {
|
||||
"name": clean(row.get("name")),
|
||||
"type": clean(row.get("type")),
|
||||
"content": _alias_content(row),
|
||||
"description": clean(row.get("description")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
}
|
||||
|
||||
|
||||
# ── port forward ─────────────────────────────────────────────────────────
|
||||
|
||||
def _pf_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
|
||||
reasons.append("ipprotocol_unsupported") # blank / inet46 have no counterpart in OPNexus
|
||||
if not clean(row.get("protocol")):
|
||||
reasons.append("protocol_any")
|
||||
if on(row.get("nordr")):
|
||||
reasons.append("no_redirect")
|
||||
elif not clean(row.get("target")):
|
||||
reasons.append("no_target")
|
||||
if clean(row.get("pass")):
|
||||
reasons.append("filter_rule_linked") # OPNexus sends pass "" (manual) and would cut the link
|
||||
if on(row.get("source.not")):
|
||||
reasons.append("negated_source")
|
||||
if on(row.get("destination.not")):
|
||||
reasons.append("negated_destination")
|
||||
if clean(row.get("natreflection")) not in ("", "purenat", "disable"):
|
||||
reasons.append("natreflection_unsupported")
|
||||
if not clean(row.get("descr")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(PortForwardRule, _pf_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _pf_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("descr")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"ipprotocol": clean(row.get("ipprotocol")),
|
||||
"protocol": clean(row.get("protocol")),
|
||||
"source_net": clean(row.get("source.network")) or "any",
|
||||
"source_port": _port(row.get("source.port")),
|
||||
"destination_net": clean(row.get("destination.network")) or "any",
|
||||
"destination_port": _port(row.get("destination.port")),
|
||||
"target": clean(row.get("target")),
|
||||
"local_port": _port(row.get("local-port")),
|
||||
"natreflection": clean(row.get("natreflection")),
|
||||
"enabled": not on(row.get("disabled")),
|
||||
}
|
||||
|
||||
|
||||
def _pf_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"protocol": clean(row.get("%protocol")) or clean(row.get("protocol")) or "any",
|
||||
"source": clean(row.get("source.network")) or "any",
|
||||
"source_port": clean(row.get("source.port")),
|
||||
"destination": clean(row.get("destination.network")) or "any",
|
||||
"destination_port": clean(row.get("destination.port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": clean(row.get("local-port")),
|
||||
}
|
||||
|
||||
|
||||
# ── 1:1 NAT ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _o2o_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if clean(row.get("type")) not in ("binat", "nat"):
|
||||
reasons.append("type_unsupported")
|
||||
if clean(row.get("natreflection")) not in ("", "enable", "disable"):
|
||||
reasons.append("natreflection_unsupported")
|
||||
if not clean(row.get("source_net")) or not clean(row.get("external")):
|
||||
reasons.append("incomplete")
|
||||
if not clean(row.get("description")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(OneToOneRule, _o2o_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _o2o_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("description")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"source_net": clean(row.get("source_net")),
|
||||
"destination_net": clean(row.get("destination_net")) or "any",
|
||||
"source_not": on(row.get("source_not")),
|
||||
"destination_not": on(row.get("destination_not")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
"type": clean(row.get("type")),
|
||||
"external": clean(row.get("external")),
|
||||
"natreflection": clean(row.get("natreflection")),
|
||||
"log": on(row.get("log")),
|
||||
}
|
||||
|
||||
|
||||
def _o2o_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"type": clean(row.get("type")),
|
||||
"source": ("! " if on(row.get("source_not")) else "") + clean(row.get("source_net")),
|
||||
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
|
||||
"external": clean(row.get("external")),
|
||||
}
|
||||
|
||||
|
||||
# ── outbound NAT ─────────────────────────────────────────────────────────
|
||||
|
||||
async def _fetch_outbound(host: OpnsenseHost) -> list[dict]:
|
||||
# In automatic/disabled mode OPNsense does not list manual rules at all (see module docstring):
|
||||
# say so instead of showing an empty list.
|
||||
state = await host_mode(host)
|
||||
if not state["can_deploy"]:
|
||||
raise api_error(409, ErrorCode.OUTBOUND_MODE_BLOCKS_ADOPT, host=host.name, mode=state["mode"])
|
||||
return await _fetch_nat(opnsense_client.search_outbound_nat_rules)(host)
|
||||
|
||||
|
||||
def _snat_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if on(row.get("nonat")):
|
||||
reasons.append("no_nat_rule")
|
||||
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
|
||||
reasons.append("ipprotocol_unsupported")
|
||||
if not clean(row.get("protocol")):
|
||||
reasons.append("protocol_missing")
|
||||
if not clean(row.get("description")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(OutboundNatRule, _snat_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _snat_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("description")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"source_net": clean(row.get("source_net")) or "any",
|
||||
"destination_net": clean(row.get("destination_net")) or "any",
|
||||
"source_not": on(row.get("source_not")),
|
||||
"destination_not": on(row.get("destination_not")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
"ipprotocol": clean(row.get("ipprotocol")),
|
||||
"protocol": clean(row.get("protocol")),
|
||||
"source_port": _port(row.get("source_port")),
|
||||
"destination_port": _port(row.get("destination_port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": _port(row.get("target_port")),
|
||||
}
|
||||
|
||||
|
||||
def _snat_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"protocol": clean(row.get("protocol")) or "any",
|
||||
"source": ("! " if on(row.get("source_not")) else "") + (clean(row.get("source_net")) or "any"),
|
||||
"source_port": clean(row.get("source_port")),
|
||||
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
|
||||
"destination_port": clean(row.get("destination_port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": clean(row.get("target_port")),
|
||||
}
|
||||
|
||||
|
||||
KINDS: dict[str, Kind] = {
|
||||
"alias": Kind(
|
||||
key="alias", model=FirewallAlias, deployment=AliasDeployment, fk="alias_id",
|
||||
uuid_attr="opnsense_alias_uuid", fetch=_fetch_aliases, check=_alias_check,
|
||||
title=lambda row: clean(row.get("name")), info=_alias_info, build=_alias_build,
|
||||
payload=alias_deploy_service._build_payload, revision=alias_policy_service.revision,
|
||||
context=_alias_context, refs=_alias_refs, name=lambda row: clean(row.get("name")), identity="name",
|
||||
),
|
||||
"port_forward": Kind(
|
||||
key="port_forward", model=PortForwardRule, deployment=PortForwardDeployment, fk="port_forward_id",
|
||||
uuid_attr="opnsense_port_forward_uuid", fetch=_fetch_nat(opnsense_client.search_port_forward_rules),
|
||||
check=_pf_check, title=lambda row: clean(row.get("descr")), info=_pf_info, build=_pf_build,
|
||||
payload=port_forward_deploy_service._build_payload, revision=port_forward_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("log", "poolopts", "tag", "tagged", "nosync", "categories")),
|
||||
),
|
||||
"one_to_one": Kind(
|
||||
key="one_to_one", model=OneToOneRule, deployment=OneToOneDeployment, fk="one_to_one_id",
|
||||
uuid_attr="opnsense_one_to_one_uuid", fetch=_fetch_nat(opnsense_client.search_one_to_one_rules),
|
||||
check=_o2o_check, title=lambda row: clean(row.get("description")), info=_o2o_info, build=_o2o_build,
|
||||
payload=one_to_one_deploy_service._build_payload, revision=one_to_one_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("categories",)),
|
||||
),
|
||||
"outbound_nat": Kind(
|
||||
key="outbound_nat", model=OutboundNatRule, deployment=OutboundNatDeployment, fk="outbound_nat_id",
|
||||
uuid_attr="opnsense_outbound_nat_uuid", fetch=_fetch_outbound,
|
||||
check=_snat_check, title=lambda row: clean(row.get("description")), info=_snat_info, build=_snat_build,
|
||||
payload=outbound_nat_deploy_service._build_payload, revision=outbound_nat_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("log", "staticnatport", "poolopts", "poolopts_sourcehashkey", "tag", "tagged",
|
||||
"nosync", "endpoint-independent", "categories")),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ── engine ───────────────────────────────────────────────────────────────
|
||||
|
||||
async def _managed_uuids(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> set[str]:
|
||||
column = getattr(kind.deployment, kind.uuid_attr)
|
||||
rows = await db.scalars(select(column).where(kind.deployment.host_id == host.id, column.is_not(None)))
|
||||
return set(rows.all())
|
||||
|
||||
|
||||
def _resolve_references(kind: Kind, entries: list[Entry], rows: list[dict], managed: set[str], ctx: dict) -> list[Entry]:
|
||||
"""Aliases may refer to other aliases by name. A reference to a live alias OPNexus does not manage
|
||||
(and that is not OPNsense's own) would be dangling in the central model, so the referring alias
|
||||
waits until its target is managed. Returns the entries in an order in which they can be adopted
|
||||
in ONE run (dependencies first); entries whose references can never be satisfied (target is a
|
||||
url alias, a user-defined external alias, ...) keep the reason `references_unmanaged_alias`."""
|
||||
live = {kind.name(r): r for r in rows}
|
||||
available = set(ctx["central_names"])
|
||||
available |= {n for n, r in live.items() if _alias_is_system(r) or clean(r.get("uuid")) in managed}
|
||||
|
||||
candidates = [e for e in entries if not e.reasons]
|
||||
for entry in candidates:
|
||||
missing = {ref for ref in kind.refs(entry.row) if ref in live and ref not in available}
|
||||
if missing:
|
||||
entry.reasons.append("references_unmanaged_alias")
|
||||
entry.details["references_unmanaged_alias"] = ", ".join(sorted(missing))
|
||||
|
||||
order: list[Entry] = []
|
||||
ready = set(available)
|
||||
progress = True
|
||||
while progress:
|
||||
progress = False
|
||||
for entry in candidates:
|
||||
if entry in order:
|
||||
continue
|
||||
if all(ref not in live or ref in ready for ref in kind.refs(entry.row)):
|
||||
order.append(entry)
|
||||
ready.add(kind.name(entry.row))
|
||||
progress = True
|
||||
for entry in order:
|
||||
entry.bulk = "references_unmanaged_alias" in entry.reasons
|
||||
return order
|
||||
|
||||
|
||||
async def _plan(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> tuple[list[Entry], list[Entry]]:
|
||||
"""(all unmanaged entries with their reasons, the ones adoptable in one run in a safe order)."""
|
||||
rows = await kind.fetch(host)
|
||||
managed = await _managed_uuids(db, kind, host)
|
||||
ctx = await kind.context(db, rows) if kind.context else {}
|
||||
central = await adopt_link_service.load_central(db, kind.link_spec)
|
||||
entries = []
|
||||
for row in rows:
|
||||
if clean(row.get("uuid")) in managed:
|
||||
continue
|
||||
reasons, details = kind.check(row, ctx)
|
||||
entry = Entry(row, list(reasons), dict(details))
|
||||
if reasons != ["system"]:
|
||||
entry.matches, entry.similar = adopt_link_service.find(central, kind.link_spec, host, kind.build(row))
|
||||
entries.append(entry)
|
||||
if kind.refs:
|
||||
order = _resolve_references(kind, entries, rows, managed, ctx)
|
||||
else:
|
||||
order = [e for e in entries if not e.reasons]
|
||||
return entries, order
|
||||
|
||||
|
||||
def _bulk_actions(entries: list[Entry], order: list[Entry], link_existing: bool) -> dict[str, tuple[str, Any]]:
|
||||
"""What "adopt all" does with each entry: ("link", central object) when exactly one central twin exists
|
||||
(and no other entry of this run took it), ("new", None), or nothing. One function for the dialog's
|
||||
summary and for the real run, so they cannot disagree. Several twins are never guessed between."""
|
||||
actions: dict[str, tuple[str, Any]] = {}
|
||||
taken: set = set()
|
||||
if link_existing:
|
||||
for entry in entries:
|
||||
if entry.linkable and len(entry.matches) == 1 and entry.matches[0].obj.id not in taken:
|
||||
actions[entry.uuid] = ("link", entry.matches[0].obj)
|
||||
taken.add(entry.matches[0].obj.id)
|
||||
for entry in order:
|
||||
if link_existing and entry.busy_twin:
|
||||
continue
|
||||
actions.setdefault(entry.uuid, ("new", None))
|
||||
return actions
|
||||
|
||||
|
||||
def _describe(kind: Kind, entry: Entry, action: str | None = None) -> dict:
|
||||
spec = kind.link_spec
|
||||
return {
|
||||
"matches": [adopt_link_service.describe(spec, m) for m in entry.matches],
|
||||
"similar": [adopt_link_service.describe(spec, m) for m in entry.similar],
|
||||
"linkable": entry.linkable,
|
||||
"bulk_action": action,
|
||||
"uuid": entry.uuid,
|
||||
"title": kind.title(entry.row),
|
||||
"enabled": on(entry.row.get("enabled")) if "disabled" not in entry.row else not on(entry.row.get("disabled")),
|
||||
"adoptable": not entry.reasons,
|
||||
"bulk_adoptable": not entry.reasons or entry.bulk,
|
||||
"reasons": entry.reasons,
|
||||
"reason_details": entry.details,
|
||||
"kept": kind.kept(entry.row),
|
||||
"info": kind.info(entry.row),
|
||||
}
|
||||
|
||||
|
||||
async def list_unmanaged(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> list[dict]:
|
||||
kind = KINDS[kind_key]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
actions = _bulk_actions(entries, order, True)
|
||||
return [_describe(kind, e, actions.get(e.uuid, ("skip", None))[0]) for e in entries]
|
||||
|
||||
|
||||
def _create(db: AsyncSession, kind: Kind, host: OpnsenseHost, entry: Entry):
|
||||
obj = kind.model(**kind.build(entry.row), targets=[host], groups=[])
|
||||
db.add(obj)
|
||||
return obj
|
||||
|
||||
|
||||
async def _record(db: AsyncSession, kind: Kind, host: OpnsenseHost, obj, remote_uuid: str) -> None:
|
||||
await db.flush()
|
||||
kind.revision(db, obj, ADOPT_NOTE)
|
||||
db.add(kind.deployment(**{
|
||||
kind.fk: obj.id, "host_id": host.id, "status": "success", "message": ADOPT_MESSAGE,
|
||||
kind.uuid_attr: remote_uuid, "last_payload": kind.payload(obj), "previous_payload": None,
|
||||
}))
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid: str, link_to=None):
|
||||
"""Takes over one object as a NEW central object, or - with `link_to` - links it to that existing
|
||||
central object (adopt_link_service). Either way the live state is read fresh here."""
|
||||
kind = KINDS[kind_key]
|
||||
if link_to is not None:
|
||||
existing = await db.scalar(select(kind.deployment).where(
|
||||
kind.deployment.host_id == host.id, getattr(kind.deployment, kind.uuid_attr) == remote_uuid))
|
||||
if existing is not None:
|
||||
if getattr(existing, kind.fk) == link_to:
|
||||
return await db.get(kind.model, link_to) # second click: already linked, nothing to do
|
||||
raise NotAdoptable(["already_managed"])
|
||||
entries, _ = await _plan(db, kind, host)
|
||||
entry = next((e for e in entries if e.uuid == remote_uuid), None)
|
||||
if entry is None:
|
||||
# either gone from the host or already managed
|
||||
if remote_uuid in await _managed_uuids(db, kind, host):
|
||||
raise NotAdoptable(["already_managed"])
|
||||
raise NotFoundOnHost(remote_uuid)
|
||||
if link_to is not None:
|
||||
if not set(entry.reasons) <= LINK_TOLERATED:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = await adopt_link_service.link(db, kind.link_spec, host, link_to, remote_uuid, kind.build(entry.row))
|
||||
return obj
|
||||
if entry.reasons:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, remote_uuid)
|
||||
return obj
|
||||
|
||||
|
||||
async def adopt_all(db: AsyncSession, kind_key: str, host: OpnsenseHost, link_existing: bool = False) -> dict:
|
||||
"""Adopts everything adoptable in one transaction (the caller commits), nested aliases in
|
||||
dependency order. With `link_existing`, entries with exactly one exact central twin are linked to it
|
||||
instead (see _bulk_actions). System objects are not reported; everything else that stays behind is,
|
||||
with reasons."""
|
||||
kind = KINDS[kind_key]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
actions = _bulk_actions(entries, order, link_existing)
|
||||
adopted = []
|
||||
done: set[str] = set()
|
||||
for entry in entries: # linked first: their names are central already, so nested aliases can refer to them
|
||||
action, target = actions.get(entry.uuid, ("skip", None))
|
||||
if action == "link":
|
||||
obj = await adopt_link_service.link(db, kind.link_spec, host, target.id, entry.uuid, kind.build(entry.row))
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": True})
|
||||
done.add(entry.uuid)
|
||||
for entry in order:
|
||||
if actions.get(entry.uuid, ("skip", None))[0] != "new":
|
||||
continue
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, entry.uuid)
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id, "linked": False})
|
||||
done.add(entry.uuid)
|
||||
skipped = [
|
||||
{"uuid": e.uuid, "title": kind.title(e.row),
|
||||
"reasons": e.reasons + (["twin_busy"] if link_existing and e.busy_twin else []), "reason_details": e.details}
|
||||
for e in entries if e.uuid not in done and e.reasons != ["system"]
|
||||
]
|
||||
return {"adopted": adopted, "skipped": skipped}
|
||||
Loaded 100 of 281 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user