Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d93b154660 | ||
|
|
cf9981c6eb | ||
|
|
4eea6424e7 | ||
|
|
a8ca0a14ef | ||
|
|
b020e38d0a | ||
|
|
abd15318a3 | ||
|
|
ec20940139 |
No files matched your search
+170
-1
@@ -29,7 +29,176 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
|
|||||||
den Einstellungen.
|
den Einstellungen.
|
||||||
|
|
||||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||||
ansteht): Merlin …
|
ansteht): Saker …
|
||||||
|
|
||||||
|
## [1.33.0] "Saker" — 2026-10-24
|
||||||
|
|
||||||
|
- **Aus dem Testlauf 1.31.0 behoben (T2 bis T6, B15):**
|
||||||
|
- **Benutzerliste:** `GET /api/users` zeigte jedem angemeldeten Konto (auch Nur-Lese, auch per Token) alle Benutzernamen, Rollen, letzte Anmeldung und 2FA-Status.
|
||||||
|
Jetzt sehen **nur Admins** die ganze Liste, alle anderen nur das eigene Konto.
|
||||||
|
- **Hosts:** Dieselbe Adresse zweimal einzutragen wird abgelehnt (`HOST_URL_EXISTS`, auch beim Ändern der Adresse). Die Hosts-Seite und der Einrichtungsassistent
|
||||||
|
prüfen die Verbindung beim Anlegen (Schlüssel abgelehnt / zu wenig Rechte / Zertifikat / keine Verbindung), speichern bei einem Fehler nichts und bieten
|
||||||
|
„Trotzdem anlegen“. Über die API ist das `verify_connection: true` (Standard aus). **Löschen verlangt jetzt auch am Server den Hostnamen:**
|
||||||
|
`DELETE /api/hosts/{id}?confirm_name=<Name>` (sonst 409); Skripte, die Hosts löschen, müssen das anpassen.
|
||||||
|
- **Zertifikate:** `cert_type: v3_ca` wird nicht mehr angenommen (es entstand ein Zertifikat mit CA-Erweiterung, keine nutzbare CA); die Oberfläche bot es nie an.
|
||||||
|
- **Traceroute:** Zwischenstationen ohne Antwort stehen als Zeile „keine Antwort“ in der Liste (vorher sprang die Zählung 1 → 3); ein Weg ganz ohne Antwort wird erklärt.
|
||||||
|
- **Netzwerk:** Die Vorschau prüft Zahlenbereiche vorab (VLAN-Tag 1–4094, PCP 0–7, VHID 1–255, Basis 1–254, Skew 0–254, gegen das Modell der OPNsense gelesen): `422` mit dem
|
||||||
|
Feldnamen statt `502` von der Firewall.
|
||||||
|
- **Gruppen:** Eine Gruppe, die ein VPN-Objekt als Ziel hat, lässt sich nicht mehr löschen (die Verknüpfung fiel still weg).
|
||||||
|
- Das Skript `scripts/verify_i18n.mjs` läuft jetzt auch gegen die HTTPS-Test-Instanz.
|
||||||
|
- **Regel-Bereinigung: OPNexus selbst ist automatisch geschützt (B10).** Beim Planen fragt die Bereinigung jede Firewall, von welcher Adresse sie OPNexus kommen sieht
|
||||||
|
(die eigene API-Verbindung in der Zustandstabelle), und schützt diese Adresse zusätzlich zu den Verwaltungsquellen. Vorher konnte eine Regel abgeschaltet werden, über
|
||||||
|
die nur OPNexus die Firewall erreicht, wenn in den Quellen nur Arbeitsplätze standen; dann scheiterte auch das automatische Zurücksetzen. Die Vorschau zeigt die Adresse
|
||||||
|
je Firewall (nur Admins) oder sagt, dass sie sich nicht ermitteln ließ (API-Recht „Diagnose: Zustände“, Management-Tunnel).
|
||||||
|
- **Keine verlorenen Regeltreffer mehr (B11):** Jedes Anwenden setzt die Regelzähler der Firewall zurück. OPNexus liest sie jetzt unmittelbar vor jedem Anwenden, das es
|
||||||
|
selbst auslöst (Regeln, NAT, Aliase, IPsec/OpenVPN/WireGuard, Netzwerk); vorher konnte eine Regel deshalb fälschlich als „ungenutzt“ gelten.
|
||||||
|
- **Einstellungen › Regel-Bereinigung:** schlägt die eigene Browser-Adresse als Verwaltungsquelle vor (B12). Geprüft: OPNsense 26.7 hat kein „savepoint“ zum Zurücknehmen
|
||||||
|
durch die Firewall selbst (Quellcode und live 404), das automatische Zurücksetzen bleibt bei OPNexus.
|
||||||
|
|
||||||
|
- **Installation unter Windows:** Neue Anleitung auf der Website (`/docs/windows/`, auch auf Englisch) und im Handbuch: Docker Desktop mit WSL 2, Zertifizierungsstelle importieren, Zugriff aus dem Netz (Windows-Firewall), Updates mit `opnexus` in WSL und Fehlerbehebung. Entwickelt und getestet wird weiterhin unter Linux; die Anleitung sagt das offen.
|
||||||
|
- **Sicherheit und Stabilität (aus einer externen Prüfung):**
|
||||||
|
- **Rückweg bleibt erhalten:** Wer nach „Konfiguration geschrieben, Anwenden fehlgeschlagen“ erneut ausrollte, überschrieb den ursprünglichen Rückweg; die automatische
|
||||||
|
Rücknahme stellte dann den unbestätigten Stand wieder her. Jetzt bleibt der Rückweg der Stand vor dem ersten Versuch. Wartung, Regel-Bereinigung und
|
||||||
|
„Konfiguration als Code“ behandeln eine solche Zeile wie eine offene Bestätigung. Gilt für Regeln, Aliase, Port-Forwards, 1:1-NAT, Outbound-NAT und Hierarchie-Regeln.
|
||||||
|
- **Anmeldung und 2FA:** Eine Passwort-Anmeldung, die genau während der Aktivierung von 2FA lief, bekam eine Sitzung ohne zweiten Faktor. Jetzt wird sie mit `TOTP_REQUIRED` abgewiesen.
|
||||||
|
- **Letzter Administrator:** Zwei Administratoren, die sich gleichzeitig herabstuften oder gegenseitig löschten, konnten alle Administratoren entfernen. Beides ist jetzt serialisiert.
|
||||||
|
- **Operatoren:** Mit ausgeschaltetem „Operatoren dürfen ausrollen“ konnten sie noch Snapshots (Boot-Umgebungen) anlegen. Ein Test hält künftig jede schreibende Route
|
||||||
|
fest, die keiner Rechte-Regel zugeordnet ist.
|
||||||
|
- **Zertifikate und ACME:** Der Ziel-Host ist nach dem ersten Push fest (`409 ZIEL_HOST_NACH_AUSROLLEN_FEST`); vorher meldete ein Host-Wechsel Erfolg für einen Host, der nie
|
||||||
|
etwas erhielt. Ein ACME-Konto mit Anfragen lässt sich nicht auf einen anderen Host verschieben (`ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET`).
|
||||||
|
- **VPN-Ausrollen:** IPsec, OpenVPN und WireGuard sichern jede von der Firewall vergebene Kennung sofort; bei einem Abbruch entstehen beim nächsten Versuch keine Duplikate.
|
||||||
|
Pro VPN läuft nur ein Ausrollen oder Entfernen zugleich (`409 VPN_DEPLOY_LAEUFT_BEREITS`).
|
||||||
|
- **Konfiguration als Code:** Gültiges YAML mit falsch geformtem Abschnitt (z. B. `aliases: 42`) ergibt `422 CONFIG_INVALID` statt eines Serverfehlers.
|
||||||
|
- **`.env`:** Die Anleitungen legen sie mit `install -m 600` an; `opnexus update` und `opnexus status` setzen eine zu offene Datei auf 0600. `opnexus rollback` liest seinen
|
||||||
|
Update-Zustand nur noch als Text und führt ihn nie aus.
|
||||||
|
- **Abhängigkeiten:** Next.js 15.5.28, PyJWT 2.15.1.
|
||||||
|
|
||||||
|
## [1.32.1] "Merlin" — 2026-10-23
|
||||||
|
|
||||||
|
- **Korrektur: Rollouts erkannten die Zielversion bei anderen Release-Typen falsch.** OPNexus las die neue Version nur am Kernpaket `opnsense`. Bei der
|
||||||
|
Business-Edition (`opnsense-business`) und beim Wechsel des Release-Typs (z. B. auf `opnsense-devel`, dann als neues Paket angeboten) fiel es auf
|
||||||
|
`product_latest` zurück; das Gate hielt den Rollout dann fälschlich mit „Version passt nicht“ an. Live gefunden und belegt (Rollout mit Cluster in einer Welle).
|
||||||
|
- **Plan-Hinweis zu Alias-Namen ehrlich formuliert:** „kein zentraler Alias in OPNexus“ statt „kein Alias dieses Namens“ (von Hand auf der Firewall
|
||||||
|
angelegte Aliase sind in Ordnung; der Planer fragt bewusst keine Firewall).
|
||||||
|
|
||||||
|
## [1.32.0] "Merlin" — 2026-10-23
|
||||||
|
|
||||||
|
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
|
||||||
|
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
|
||||||
|
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
|
||||||
|
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
|
||||||
|
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
|
||||||
|
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
|
||||||
|
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
|
||||||
|
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
|
||||||
|
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
|
||||||
|
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
|
||||||
|
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
|
||||||
|
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
|
||||||
|
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
|
||||||
|
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
|
||||||
|
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
|
||||||
|
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
|
||||||
|
|
||||||
|
## [1.31.1] "Vulture" — 2026-10-22
|
||||||
|
|
||||||
|
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
|
||||||
|
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
|
||||||
|
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
|
||||||
|
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
|
||||||
|
|
||||||
|
## [1.31.0] "Vulture" — 2026-10-22
|
||||||
|
|
||||||
|
- **Regel-Bereinigung: Entfernen, Bestätigen, Löschen, Wiederherstellen:** Im Tab **Bereinigung** (Regelwerk › Regel-Analyse) lassen sich Kandidaten jetzt auswählen
|
||||||
|
(Checkbox, höchstens so viele Regeln je Lauf wie in den Einstellungen; „vermutlich“ nur mit einer ausdrücklichen Einzelbestätigung je Regel, „unbekannt“ nie).
|
||||||
|
**Vorschau** zeigt je Firewall (im Cluster zuerst der Backup) die Regeln mit Befund, Sicherheit und Schutzprüfung; **Ausrollen** deaktiviert die Regeln und wendet an.
|
||||||
|
Danach läuft ein Countdown: **Bestätigen** heißt „ich erreiche die Firewall noch“, ohne Bestätigung setzt OPNexus **automatisch zurück**; bis dahin geht auch
|
||||||
|
**Jetzt zurücksetzen**. Bestätigte Regeln lassen sich nach der Wartezeit (einstellbar, Standard 7 Tage) **löschen**; vorher sichert OPNexus den Regelinhalt, und
|
||||||
|
unter **Verlauf der Bereinigungen** kommt die Regel mit **Wiederherstellen** deaktiviert und möglichst an alter Position zurück (sonst mit Hinweis am Ende der Gruppe);
|
||||||
|
**Wieder aktivieren** hat wieder ein Bestätigungsfenster. Wichtig: **Ausrollen, Wiederholen und Wiederherstellen darf nur ein Administrator**, eine Vorschau gehört
|
||||||
|
dem, der sie erstellt hat (ein Admin öffnet selbst eine neue); **Bestätigen geht nur in der Browser-Sitzung**, nie per API-Token. **Die Bereinigung bleibt gesperrt,
|
||||||
|
bis unter Einstellungen die Verwaltungsquellen** (IP-Adressen oder Netze, von denen aus verwaltet wird) **eingetragen sind**: Regeln, die diesen Zugang treffen
|
||||||
|
könnten, deaktiviert OPNexus nie. Dort stehen auch die Obergrenze je Lauf, die Wartezeit bis zum Löschen und **weitere Verwaltungsports** (zusätzlich zu
|
||||||
|
API-Port, 443, 80, 22). Tragen Sie die Adresse ein, mit der OPNexus selbst bei den Firewalls ankommt (im Cluster für jeden Knoten). Quellen, Ports und
|
||||||
|
Firewall-Adressen sehen nur Administratoren. Der Hinweis „Entfernen folgt“ entfällt.
|
||||||
|
- **Update: neuere CLI aus dem Zielrelease:** `opnexus update` (auch per Klick im Webinterface) holt sich nach dem Pull die CLI aus dem Updater-Image des
|
||||||
|
Zielreleases und startet sich damit neu, damit Korrekturen am Update-Ablauf sofort greifen statt erst beim übernächsten Update. Das geschieht nur, wenn
|
||||||
|
`cosign` und `cosign.pub` vorhanden sind und die Signatur des Image-**Digests** in diesem Lauf geprüft wurde (ausgeführt wird genau dieser Digest);
|
||||||
|
sonst und bei jedem Fehler läuft das Update mit der vorhandenen CLI weiter. Abschalten: `OPNEXUS_NO_CLI_REFRESH=1`. Wirksam wird es ab dem Update
|
||||||
|
*von* dieser Version.
|
||||||
|
|
||||||
|
## [1.30.0] "Owl" — 2026-10-22
|
||||||
|
|
||||||
|
- **Regel-Analyse: Bereinigung (nur Ansicht):** Auf **Regelwerk › Regel-Analyse** gibt es den Tab **Bereinigung** (`?tab=cleanup`). Er listet je Firewall die
|
||||||
|
Regeln, die einen Blick wert sind: **ungenutzt** (OPNexus hat in 7/30/90/180 Tagen keinen Treffer gesehen), **verdeckt** (eine frühere Quick-Regel
|
||||||
|
trifft schon alles, was sie trifft), **doppelt** und **lange deaktiviert**. Jeder Befund trägt eine Sicherheit (**sicher / vermutlich / unbekannt**, Tooltip mit
|
||||||
|
dem Grund): „sicher“ gibt es nur mit vollständigen, frischen Belegen, bei einer Regel mit Zeitplan, ohne geladenen pf-Regelsatz oder bei Aliasen bleibt es
|
||||||
|
höchstens „vermutlich“. Die Tabelle zeigt auch den letzten Treffer und ob die Regel entfernbar wäre (nicht: automatische Regeln, Regeln im Altformat, von OPNexus
|
||||||
|
verwaltete Regeln – die entfernt man über die Regelseite). Es wird nur gelesen; **Entfernen folgt in einer späteren Version**. Die Sicherheitslage verlinkt ihre
|
||||||
|
Befunde „ungenutzt/verdeckt/doppelt“ direkt dorthin; per API `GET /api/hosts/{id}/rule-analysis/cleanup?unused_days=30`, per MCP `rule_cleanup_candidates`.
|
||||||
|
- **Änderungsverlauf je Regel:** OPNexus liest alle 15 Minuten die Regeln jeder Firewall und merkt sich, was sich geändert hat – auch bei Regeln, die nicht von OPNexus
|
||||||
|
stammen: angelegt, geändert (mit Vorher/Nachher), aktiviert, deaktiviert, entfernt, jeweils mit der Herkunft „über OPNexus“ (passender Audit-Eintrag) oder „extern“.
|
||||||
|
Auf der Live-Regel-Ansicht (Regelwerk › Live über alle Hosts) öffnet die Zeilenaktion **Verlauf** die Ereignisse zusammen mit den gespeicherten Fassungen und dem Audit der
|
||||||
|
verwalteten Regel. Die erste Beobachtung speichert nur den Ausgangszustand; der Verlauf wird 365 Tage aufbewahrt (`GET /api/hosts/{id}/live-rules/{uuid}/history`).
|
||||||
|
Eine Umsortierung (andere Reihenfolge oder Gruppe) gilt als „verschoben“, nicht als Änderung. Eine leere oder unvollständige Antwort der Firewall (ohne
|
||||||
|
automatische Regeln) wird verworfen; verschwinden mehr als die Hälfte der Regeln auf einmal, zählt erst die zweite gleiche Beobachtung. Die Herkunft „über OPNexus“
|
||||||
|
erklärt nur Audit-Einträge seit der vorherigen Beobachtung, und jeder Eintrag nur eine Änderung.
|
||||||
|
Schemaänderungen 0027 und 0028 (Tabellen und ein Feld für den Beobachtungszeitraum) laufen beim Start.
|
||||||
|
- **Regelnutzung genauer:** Die Zähler kommen jetzt aus `ruleStats` (von der Firewall schon je Regel summiert; ältere Versionen weiterhin über den pf-Regelsatz) und werden alle
|
||||||
|
15 Minuten statt alle 6 Stunden gelesen. Regeln im Altformat (noch nicht migriert) werden ebenfalls erfasst. Das Beobachtungsfenster für „ungenutzt“ beginnt neu, wenn eine Regel eingeschaltet oder geändert wird, und nach einem pf-Neuladen mit langer Lücke bei den Zählern.
|
||||||
|
- **Verdeckt/doppelt in der Sicherheitslage genauer:** Der Vergleich kennt jetzt die echte Reihenfolge (Floating vor Interface-Gruppen vor Interface-Regeln), Netze und
|
||||||
|
Portbereiche als Mengen (z. B. 10.1.0.0/16 liegt in 10.0.0.0/8), `inet46` und `tcp/udp`, und übergeht Regeln mit Zeitplan, Markierung oder Limits. Es findet dadurch
|
||||||
|
mehr verdeckte Regeln als bisher, nie weniger belegte.
|
||||||
|
|
||||||
|
## [1.29.0] "Caracara" — 2026-10-22
|
||||||
|
|
||||||
|
- **Fehlende API-Rechte werden sichtbar:** Darf der API-Schlüssel einer Firewall den Gateway- oder Dienststatus nicht lesen (OPNsense antwortet
|
||||||
|
mit 403), fiel das bisher niemandem auf, und der standardmäßig aktive Gateway-Alarm konnte nie auslösen. Jetzt zeigen die Host-Seite
|
||||||
|
(Abschnitt „Gateways“) und die Dashboard-Karte „Fleet-/CARP-Gesundheit“ ein Warn-Badge „API-Rechte fehlen: Gateways, Dienste“; der Tooltip nennt die
|
||||||
|
OPNsense-Berechtigung, die beim Benutzer oder bei der Gruppe fehlt (System: Gateways, Status: Services, Lobby: Dashboard). Zusätzlich gibt es
|
||||||
|
den neuen Alarm **API-Rechte fehlen** (standardmäßig an, abschaltbar unter Einstellungen › Benachrichtigungen, gleiche Entprellung wie die
|
||||||
|
anderen Messungs-Alarme) und die Prometheus-Metrik `opnexus_host_api_rights_missing{host,read}`. Nur eine ausdrückliche Ablehnung zählt;
|
||||||
|
Zeitüberschreitungen oder Serverfehler bleiben „unbekannt“.
|
||||||
|
- **Gateway-Alarm pro Gateway abschaltbar:** In der Gateway-Liste der Host-Seite schaltet eine Glocke den Alarm für ein einzelnes Gateway aus
|
||||||
|
(z. B. ruhendes Zweit-WAN oder tote IPv6-Gateways); ein „Alarm aus“-Badge zeigt es an. Ein offener Alarm dieses Gateways löst sich dabei
|
||||||
|
auf. Nur Administratoren, die Änderung steht im Audit-Log. Das Dashboard zeigt bei mehr als acht Gateways „+ N weitere“.
|
||||||
|
- **Firewall-Oberfläche (GUI-Zugriff) härter:** Ein frei gewordener Platz wird für eine **andere** Firewall erst 5 Minuten nach dem Ende der letzten
|
||||||
|
Sitzung vergeben (Schutz, falls dort noch ein Tab offen ist, der die Cookies des Platzes teilt); dieselbe Firewall bekommt ihn sofort wieder.
|
||||||
|
Sind alle Plätze belegt oder noch in der Abkühlzeit, nennt die Meldung den Grund. Ändern sich Adresse, festgesetztes Zertifikat oder Tunnelweg
|
||||||
|
einer Firewall, während ihre Oberfläche offen ist, endet die Sitzung spätestens nach 20 Sekunden. Das Limit von 64 gleichzeitigen Sitzungen
|
||||||
|
verdrängt keine laufende Sitzung mehr: eine neue wird abgelehnt (mit Meldung), bis eine endet. **Hinweis:** Das GUI-Gateway-Image muss zur
|
||||||
|
Backend-Version passen (wie bisher beim Update beider Container).
|
||||||
|
- **Anmeldung (Cookies) härter:** Das alte Sitzungs-Cookie `opnexus-session` wird bei HTTPS **nicht mehr akzeptiert**. **Alle, die sich seit der
|
||||||
|
Umstellung auf 1.27 nicht neu angemeldet haben, müssen sich einmal neu anmelden**; Skripte mit rohem Sitzungs-Cookie brauchen den Namen
|
||||||
|
`__Host-opnexus-session` (API-Tokens sind nicht betroffen, ohne HTTPS bleibt `opnexus-session` der normale Name). Beim Abmelden wird ein altes
|
||||||
|
Cookie weiterhin gelöscht. Das Cookie, das die SSO-Anmeldung an den Browser bindet, heißt bei HTTPS jetzt `__Host-opnexus-oidc-state`; der
|
||||||
|
Rückweg vom Anbieter wird nur noch als Seitenaufruf (Navigation) von einer fremden Seite zugelassen, und ohne dieses Cookie scheitert er immer.
|
||||||
|
- **Webhook, Benachrichtigungskanäle und SSO-Anbieter: Schutz vor DNS-Rebinding:** Bisher prüfte OPNexus die Adresse, auf die ein Name zeigt, und
|
||||||
|
der HTTP-Client löste den Namen danach noch einmal auf; ein Server unter Fremdkontrolle konnte beim zweiten Mal eine interne Adresse liefern. Jetzt
|
||||||
|
verbindet OPNexus genau zu der geprüften Adresse (Host-Name, TLS-Zertifikatsprüfung und Anfrage bleiben beim Namen); zeigt eine Antwort auch nur
|
||||||
|
teilweise auf Loopback, Link-Local oder einen Cloud-Metadaten-Dienst (auch AWS-IPv6 und Alibaba), kommt keine Verbindung zustande und die Zustellung meldet „Zieladresse nicht
|
||||||
|
erlaubt“. Private Netze bleiben erlaubt (eigener ntfy, Gotify, Matrix). Proxy-Umgebungsvariablen werden für diese Aufrufe nicht mehr beachtet.
|
||||||
|
- **Benachrichtigungen parallel zugestellt:** Webhook, E-Mail und alle Kanäle werden gleichzeitig bedient, jeder mit eigener Zeitgrenze (20 Sekunden),
|
||||||
|
und ein Prüfdurchlauf sendet insgesamt höchstens 90 Sekunden. Ein hängender Kanal verzögert weder die anderen noch die nächsten Prüfungen; im Log
|
||||||
|
steht dann „timed out“ (ohne Adresse oder Token). Wie bisher wird einmal je Ereignis zugestellt, ohne Wiederholung, **außer** der Durchlauf war so voll, dass
|
||||||
|
für ein Ereignis noch gar kein Versuch stattfand: es wird im nächsten Durchlauf zugestellt. Die Zustellung beginnt jetzt nach allen Prüfungen
|
||||||
|
eines Durchlaufs statt mitten darin.
|
||||||
|
- **Rollouts: Fingerabdruck, verantwortliche Person, überlappende Termine:** Der Fingerabdruck eines aktivierten Plans umfasst jetzt auch die
|
||||||
|
ausgelassenen und verschobenen Termine seines Fensters: **wer einen Termin auslässt oder verschiebt, muss einen automatischen Plan danach neu
|
||||||
|
aktivieren** (ohne solche Ausnahmen bleiben bestehende Aktivierungen gültig; **Pläne, deren Fenster schon Ausnahmen haben, nach dem Update
|
||||||
|
einmal neu aktivieren**, sonst meldet der Plan „Plan geändert“ und startet nicht automatisch). Die verantwortliche Person (Aktivierende, Startende, Freigebende)
|
||||||
|
wird zusätzlich als Konto-ID gespeichert (Migration 0026, bestehende Einträge werden über den Namen zugeordnet): ein gelöschtes und unter gleichem
|
||||||
|
Namen neu angelegtes Konto gilt nicht mehr als dieselbe Person, der Lauf pausiert bzw. der automatische Start unterbleibt mit „Person darf nicht mehr
|
||||||
|
ausrollen“. Angezeigt wird weiter der Benutzername. Überlappen sich zwei Termine eines Wartungsfensters, zählt der zuletzt begonnene (bisher der
|
||||||
|
früheste), sodass ein älterer, längst abgeschnittener Termin keinen neuen Start mehr blockiert.
|
||||||
|
- **Kleinigkeiten:** Ressourcen-Alarme zeigen bei fehlendem Messwert keine Zahl mehr („None %“ entfällt). Der Dialog „Von Firewall übernehmen“ zeigt ohne aktive
|
||||||
|
Firewall einen Hinweis statt dauerhaft „Lädt…“. Fehlermeldungen beim Übernehmen und Verknüpfen nennen die Gründe in Klartext (Deutsch/Englisch) statt
|
||||||
|
roher Codes wie `name_exists`; dasselbe gilt für „GUI-Zugriff nicht eingerichtet“ und blockierte Netzwerkänderungen. Der Test-Knopf der
|
||||||
|
Benachrichtigungskanäle ist für Nicht-Admins ausgeblendet (der Server verweigert den Test ohnehin). Der Tooltip der aktiven Plätze nennt keine
|
||||||
|
Umgebungsvariable mehr. Eine Gateway-Sitzung, die der Gateway-Container nicht mehr bestätigt (Absturz), zählt für die Abkühlzeit der Plätze als
|
||||||
|
beendet.
|
||||||
|
- **Grafana-Dashboard erweitert:** neue Felder für Gateways (erreichbar, Paketverlust, Latenz), Swap, gestoppte Dienste, fehlende API-Rechte und
|
||||||
|
Rollouts nach Status; dazu die Metrik `opnexus_rollout_runs{status}`.
|
||||||
|
|
||||||
## [1.28.0] "Sparrowhawk" — 2026-10-21
|
## [1.28.0] "Sparrowhawk" — 2026-10-21
|
||||||
|
|
||||||
|
|||||||
+8
-5
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
> **Stand:** `1.28.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
> **Stand:** `1.33.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||||
|
|
||||||
## Funktionen
|
## Funktionen
|
||||||
|
|
||||||
@@ -21,12 +21,13 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
|||||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||||
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
||||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
|
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
|
||||||
|
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
|
||||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||||
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
|
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
|
||||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend.
|
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit); ein Administrator kann ausgewählte Regeln abschalten (Vorschau, dann Ausrollen mit automatischer Rücknahme ohne Bestätigung), später löschen und aus einer Sicherung des Regelinhalts wiederherstellen. Jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
|
||||||
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
|
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
|
||||||
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
|
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
|
||||||
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
|
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
|
||||||
@@ -47,7 +48,7 @@ Voraussetzung: Docker mit Compose-Plugin.
|
|||||||
```bash
|
```bash
|
||||||
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
||||||
cd opnexus
|
cd opnexus
|
||||||
cp .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen
|
install -m 600 .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen; nur für dich lesbar
|
||||||
docker compose up -d --build
|
docker compose up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -59,13 +60,15 @@ scripts/opnexus password
|
|||||||
|
|
||||||
Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassistent**: eigenes Passwort setzen, optional Zwei-Faktor-Anmeldung einschalten, die erste Firewall verbinden (samt Anleitung für den API-Key in OPNsense). Er lässt sich überspringen und jederzeit unter System › Einrichtung wieder öffnen.
|
Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassistent**: eigenes Passwort setzen, optional Zwei-Faktor-Anmeldung einschalten, die erste Firewall verbinden (samt Anleitung für den API-Key in OPNsense). Er lässt sich überspringen und jederzeit unter System › Einrichtung wieder öffnen.
|
||||||
|
|
||||||
|
**Windows:** läuft in Docker Desktop mit WSL 2, siehe die [Installationsanleitung für Windows](https://opnexus.dev/docs/windows/). Entwickelt und getestet wird unter Linux; für den Dauerbetrieb ist ein Linux-Server oder eine Linux-VM besser.
|
||||||
|
|
||||||
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
|
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
|
||||||
|
|
||||||
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
|
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||||
OPNEXUS_VERSION=1.28.0
|
OPNEXUS_VERSION=1.33.0
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
> **Status:** `1.28.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
> **Status:** `1.33.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
@@ -21,12 +21,13 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
|||||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||||
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
||||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
|
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
|
||||||
|
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
|
||||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||||
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
|
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
|
||||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only.
|
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence); an administrator can switch selected rules off (preview, then roll-out with an automatic revert unless confirmed), delete them later and restore them from a backup of their content. Every rule has a change history, including changes made outside OPNexus.
|
||||||
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
|
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
|
||||||
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
|
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
|
||||||
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
|
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
|
||||||
@@ -49,7 +50,7 @@ Requires Docker with the Compose plugin.
|
|||||||
```bash
|
```bash
|
||||||
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
||||||
cd opnexus
|
cd opnexus
|
||||||
cp .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD)
|
install -m 600 .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD); readable by you only
|
||||||
docker compose up -d --build
|
docker compose up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -61,13 +62,15 @@ scripts/opnexus password
|
|||||||
|
|
||||||
After the first sign-in a short **setup assistant** opens on its own: set your own password, optionally enable two-factor sign-in, connect your first firewall (including how to create the API key in OPNsense). You can skip it and reopen it any time under System › Setup.
|
After the first sign-in a short **setup assistant** opens on its own: set your own password, optionally enable two-factor sign-in, connect your first firewall (including how to create the API key in OPNsense). You can skip it and reopen it any time under System › Setup.
|
||||||
|
|
||||||
|
**Windows:** run it in Docker Desktop with WSL 2 – see the [installation guide for Windows](https://opnexus.dev/en/docs/windows/). OPNexus is developed and tested on Linux; for permanent operation a Linux server or VM is the better choice.
|
||||||
|
|
||||||
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
|
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
|
||||||
|
|
||||||
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
|
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||||
OPNEXUS_VERSION=1.28.0
|
OPNEXUS_VERSION=1.33.0
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
|
|||||||
"/api/settings",
|
"/api/settings",
|
||||||
# Read-only JSON-RPC tools (see routers/mcp.py).
|
# Read-only JSON-RPC tools (see routers/mcp.py).
|
||||||
"/api/mcp",
|
"/api/mcp",
|
||||||
|
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
|
||||||
|
"/api/config/import/plan",
|
||||||
# Self-scoped: a user managing their own second factor.
|
# Self-scoped: a user managing their own second factor.
|
||||||
"/api/auth/2fa/setup",
|
"/api/auth/2fa/setup",
|
||||||
"/api/auth/2fa/enable",
|
"/api/auth/2fa/enable",
|
||||||
@@ -121,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
|||||||
)
|
)
|
||||||
if user is not None:
|
if user is not None:
|
||||||
parts = request.url.path.strip("/").split("/")
|
parts = request.url.path.strip("/").split("/")
|
||||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
|
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
|
||||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||||
@@ -131,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
|||||||
if len(parts) > 2 and parts[1] == "settings":
|
if len(parts) > 2 and parts[1] == "settings":
|
||||||
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
|
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
|
||||||
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
|
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
|
||||||
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
|
entity = parts[2] if len(parts) > 2 and kind != "config" else None
|
||||||
|
event = {"kind": kind, "action": action, "entity_id": entity,
|
||||||
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
|
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
|
||||||
"details": {"method": request.method, "path": request.url.path}}
|
"details": {"method": request.method, "path": request.url.path}}
|
||||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
|
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
|
||||||
|
|
||||||
if (
|
if (
|
||||||
user.role == "read_only"
|
user.role == "read_only"
|
||||||
|
|||||||
@@ -160,6 +160,13 @@ class Settings(BaseSettings):
|
|||||||
# wireguard_key_store.py.
|
# wireguard_key_store.py.
|
||||||
wireguard_key_secret_dir: str = "/data/wireguard-key-secrets"
|
wireguard_key_secret_dir: str = "/data/wireguard-key-secrets"
|
||||||
|
|
||||||
|
# ── Rule observation (usage counters + change history of the live rules) ─
|
||||||
|
# Two read-only GETs per enabled firewall per cycle (searchRule, ruleStats). The interval is an operator
|
||||||
|
# setting (env RULE_OBSERVE_INTERVAL_SECONDS); it can not go below the floor, which keeps a typo from
|
||||||
|
# polling every firewall in a tight loop.
|
||||||
|
rule_observe_interval_seconds: int = 900
|
||||||
|
rule_observe_min_interval_seconds: int = 60
|
||||||
|
|
||||||
# ── Resource monitoring ──────────────────────────────────────────────
|
# ── Resource monitoring ──────────────────────────────────────────────
|
||||||
# Fixed interval, no admin-editable settings row/UI for this — v1
|
# Fixed interval, no admin-editable settings row/UI for this — v1
|
||||||
# deliberately kept minimal (see resource_monitor_service.py).
|
# deliberately kept minimal (see resource_monitor_service.py).
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
|
||||||
|
|
||||||
|
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
|
||||||
|
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
|
||||||
|
"""
|
||||||
@@ -0,0 +1,228 @@
|
|||||||
|
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
|
||||||
|
|
||||||
|
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
|
||||||
|
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
|
||||||
|
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
|
||||||
|
|
||||||
|
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
|
||||||
|
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from app.config_code import apply as applier
|
||||||
|
from app.config_code import plan as planner
|
||||||
|
from app.config_code import schema
|
||||||
|
from app.config_code.export import export_yaml
|
||||||
|
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
|
||||||
|
|
||||||
|
ACTOR = "cli (Host-Konsole)"
|
||||||
|
TOKEN_MARK = "# plan-token: "
|
||||||
|
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
|
||||||
|
|
||||||
|
ERRORS_DE = {
|
||||||
|
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
|
||||||
|
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
|
||||||
|
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
|
||||||
|
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
|
||||||
|
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
|
||||||
|
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
|
||||||
|
"yaml_empty": "Die Datei ist leer",
|
||||||
|
"yaml_syntax": "YAML-Syntaxfehler",
|
||||||
|
"duplicate_key": "Doppelter Schlüssel",
|
||||||
|
"non_string_key": "Schlüssel muss Text sein",
|
||||||
|
"too_deep": "Zu tief verschachtelt",
|
||||||
|
"too_many_nodes": "Zu viele Elemente",
|
||||||
|
"scalar_too_long": "Wert zu lang",
|
||||||
|
"too_many_objects": "Zu viele Objekte",
|
||||||
|
"too_many_entries": "Zu viele Einträge",
|
||||||
|
"too_many_variables": "Zu viele Variablen für diesen Host",
|
||||||
|
"too_large": "Datei zu groß",
|
||||||
|
"bad_encoding": "Kein gültiges UTF-8",
|
||||||
|
"not_a_mapping": "Die Datei muss ein Mapping sein",
|
||||||
|
"missing_version": "opnexus_config: 1 fehlt",
|
||||||
|
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
|
||||||
|
"unknown_field": "Unbekanntes Feld",
|
||||||
|
"missing_field": "Pflichtfeld fehlt",
|
||||||
|
"bad_type": "Falscher Typ",
|
||||||
|
"bad_value": "Ungültiger Wert",
|
||||||
|
"bad_format": "Falsches Format",
|
||||||
|
"too_short": "Zu kurz",
|
||||||
|
"too_long": "Zu lang",
|
||||||
|
"control_character": "Steuerzeichen nicht erlaubt",
|
||||||
|
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
|
||||||
|
"bad_variable_name": "Ungültiger Variablenname",
|
||||||
|
"bad_variable_value": "Ungültiger Variablenwert",
|
||||||
|
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
|
||||||
|
"duplicate_entry": "Eintrag kommt doppelt vor",
|
||||||
|
"unknown_host": "Unbekannter Host",
|
||||||
|
"unknown_group": "Unbekannte Gerätegruppe",
|
||||||
|
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
|
||||||
|
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
|
||||||
|
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
|
||||||
|
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
|
||||||
|
}
|
||||||
|
REASONS_DE = {
|
||||||
|
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
|
||||||
|
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
|
||||||
|
"prune_referenced": "wird noch verwendet",
|
||||||
|
}
|
||||||
|
WARNINGS_DE = {
|
||||||
|
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
|
||||||
|
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
|
||||||
|
"unknown_alias_reference": "{value}: kein zentraler Alias in OPNexus (auf der Firewall von Hand angelegt? dann in Ordnung)",
|
||||||
|
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
|
||||||
|
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
|
||||||
|
}
|
||||||
|
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
|
||||||
|
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
|
||||||
|
|
||||||
|
|
||||||
|
def safe(text) -> str:
|
||||||
|
"""Nothing from a foreign file reaches the terminal as a control sequence."""
|
||||||
|
return _ANSI.sub("?", str(text))
|
||||||
|
|
||||||
|
|
||||||
|
def _short(value) -> str:
|
||||||
|
return safe(display(json.dumps(value, ensure_ascii=False), 60))
|
||||||
|
|
||||||
|
|
||||||
|
def format_errors(errors: list[dict]) -> str:
|
||||||
|
lines = []
|
||||||
|
for e in errors:
|
||||||
|
where = f"Zeile {e['line']}: " if e.get("line") else ""
|
||||||
|
path = f"{safe(e['path'])}: " if e.get("path") else ""
|
||||||
|
detail = ""
|
||||||
|
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
|
||||||
|
if params:
|
||||||
|
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
|
||||||
|
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def format_plan(public: dict) -> str:
|
||||||
|
lines = []
|
||||||
|
for item in public["items"]:
|
||||||
|
action = item["action"]
|
||||||
|
planned = item["planned_action"]
|
||||||
|
name = f"{item['kind']} {safe(item['key'])}"
|
||||||
|
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
|
||||||
|
if action == "blocked":
|
||||||
|
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
|
||||||
|
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
|
||||||
|
elif planned in LABEL:
|
||||||
|
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
|
||||||
|
else:
|
||||||
|
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
|
||||||
|
if change["field"]:
|
||||||
|
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
|
||||||
|
else:
|
||||||
|
lines.append(f"~ {name}{suffix}")
|
||||||
|
if item.get("implicit"):
|
||||||
|
lines[-1] += " (Folge der Reihenfolge)"
|
||||||
|
for warning in public["warnings"]:
|
||||||
|
template = WARNINGS_DE.get(warning["code"])
|
||||||
|
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
|
||||||
|
text = template.format(**params) if template else warning["code"]
|
||||||
|
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
|
||||||
|
lines.append(f"? {where}{text}")
|
||||||
|
s = public["summary"]
|
||||||
|
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def _read_stdin() -> bytes:
|
||||||
|
data = sys.stdin.buffer.read(MAX_BYTES + 1)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
|
||||||
|
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
|
||||||
|
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
|
||||||
|
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
|
||||||
|
parser.add_argument("--kinds", default=None)
|
||||||
|
parser.add_argument("--prune", action="store_true")
|
||||||
|
parser.add_argument("--json", action="store_true")
|
||||||
|
parser.add_argument("--plan-token", default=None)
|
||||||
|
try:
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
except SystemExit:
|
||||||
|
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
|
||||||
|
if args.command == "schema":
|
||||||
|
return 0, schema.json_schema_text(), ""
|
||||||
|
|
||||||
|
from app.database import SessionLocal
|
||||||
|
from app.services import audit_service
|
||||||
|
|
||||||
|
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
|
||||||
|
try:
|
||||||
|
async with SessionLocal() as db:
|
||||||
|
if args.command == "export":
|
||||||
|
wanted = None
|
||||||
|
if args.kinds:
|
||||||
|
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
|
||||||
|
if not wanted <= set(schema.KINDS):
|
||||||
|
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
|
||||||
|
return 0, await export_yaml(db, wanted), ""
|
||||||
|
|
||||||
|
raw = stdin if stdin is not None else _read_stdin()
|
||||||
|
try:
|
||||||
|
parsed = parse(raw)
|
||||||
|
if args.command == "plan":
|
||||||
|
plan = await planner.build_plan(db, parsed, args.prune)
|
||||||
|
await db.rollback()
|
||||||
|
public = plan.public()
|
||||||
|
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
|
||||||
|
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
|
||||||
|
s = public["summary"]
|
||||||
|
if not plan.applicable:
|
||||||
|
return 4, out, ""
|
||||||
|
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
|
||||||
|
|
||||||
|
if not args.plan_token:
|
||||||
|
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
|
||||||
|
try:
|
||||||
|
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
|
||||||
|
except applier.PlanStale:
|
||||||
|
await db.rollback()
|
||||||
|
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
|
||||||
|
except applier.PlanBlocked as exc:
|
||||||
|
await db.rollback()
|
||||||
|
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
|
||||||
|
except applier.SelfCheckFailed as exc:
|
||||||
|
await db.rollback()
|
||||||
|
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
|
||||||
|
except Exception:
|
||||||
|
await db.rollback()
|
||||||
|
raise
|
||||||
|
await db.commit()
|
||||||
|
s = plan.summary()
|
||||||
|
if args.json:
|
||||||
|
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
|
||||||
|
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
|
||||||
|
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
|
||||||
|
except ConfigError as exc:
|
||||||
|
if args.json:
|
||||||
|
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
|
||||||
|
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
|
||||||
|
finally:
|
||||||
|
audit_service.context.reset(context)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str]) -> int:
|
||||||
|
try:
|
||||||
|
code, out, err = asyncio.run(run(argv))
|
||||||
|
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
|
||||||
|
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
|
||||||
|
return 1
|
||||||
|
sys.stdout.write(out)
|
||||||
|
sys.stderr.write(err)
|
||||||
|
return code
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main(sys.argv[1:]))
|
||||||
@@ -0,0 +1,312 @@
|
|||||||
|
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
|
||||||
|
|
||||||
|
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
|
||||||
|
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
|
||||||
|
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
|
||||||
|
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
|
||||||
|
|
||||||
|
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
|
||||||
|
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
|
||||||
|
Den Commit macht der Aufrufer.
|
||||||
|
"""
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from sqlalchemy import delete, select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
|
||||||
|
from app.config_code import plan as planner
|
||||||
|
from app.config_code.export import canonical
|
||||||
|
from app.config_code.loader import Parsed
|
||||||
|
from app.models.alias import FirewallAlias
|
||||||
|
from app.models.host import OpnsenseHost
|
||||||
|
from app.models.host_variable import HostVariable
|
||||||
|
from app.models.one_to_one import OneToOneRule
|
||||||
|
from app.models.outbound_nat import OutboundNatRule
|
||||||
|
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
|
||||||
|
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
|
||||||
|
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||||
|
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||||
|
from app.models.port_forward import PortForwardRule
|
||||||
|
from app.models.rule import FirewallRule
|
||||||
|
from app.services import audit_service
|
||||||
|
|
||||||
|
NOTE = "Import (Konfiguration als Code)"
|
||||||
|
AUDIT_KIND = {
|
||||||
|
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
|
||||||
|
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
|
||||||
|
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class PlanStale(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class PlanBlocked(Exception):
|
||||||
|
def __init__(self, plan: planner.Plan):
|
||||||
|
self.plan = plan
|
||||||
|
super().__init__("plan blocked")
|
||||||
|
|
||||||
|
|
||||||
|
class SelfCheckFailed(Exception):
|
||||||
|
def __init__(self, differences: list[str]):
|
||||||
|
self.differences = differences
|
||||||
|
super().__init__("; ".join(differences[:5]))
|
||||||
|
|
||||||
|
|
||||||
|
def _revision_writer(kind: str):
|
||||||
|
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
|
||||||
|
|
||||||
|
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
|
||||||
|
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
|
||||||
|
"""
|
||||||
|
if kind == "rules":
|
||||||
|
from app.services.policy_service import snapshot
|
||||||
|
return RuleRevision, "rule_id", snapshot
|
||||||
|
if kind == "aliases":
|
||||||
|
from app.services.alias_policy_service import snapshot
|
||||||
|
return AliasRevision, "alias_id", snapshot
|
||||||
|
if kind == "port_forwards":
|
||||||
|
from app.services.port_forward_policy_service import snapshot
|
||||||
|
return PortForwardRevision, "port_forward_id", snapshot
|
||||||
|
if kind == "one_to_one":
|
||||||
|
from app.services.one_to_one_policy_service import snapshot
|
||||||
|
return OneToOneRevision, "one_to_one_id", snapshot
|
||||||
|
if kind == "outbound_nat":
|
||||||
|
from app.services.outbound_nat_policy_service import snapshot
|
||||||
|
return OutboundNatRevision, "outbound_nat_id", snapshot
|
||||||
|
from app.services.policy_hierarchy_service import snapshot
|
||||||
|
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
|
||||||
|
|
||||||
|
|
||||||
|
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
|
||||||
|
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
|
||||||
|
SCALARS = {
|
||||||
|
"aliases": ("name", "type", "description", "enabled"),
|
||||||
|
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
|
||||||
|
"destination_net", "destination_port", "enabled", "log"),
|
||||||
|
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||||
|
"destination_port", "target", "local_port", "natreflection", "enabled"),
|
||||||
|
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
|
||||||
|
"destination_not", "natreflection", "log", "enabled"),
|
||||||
|
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||||
|
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
|
||||||
|
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
|
||||||
|
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class _Writer:
|
||||||
|
def __init__(self, db: AsyncSession, plan: planner.Plan):
|
||||||
|
self.db = db
|
||||||
|
self.plan = plan
|
||||||
|
self.now = datetime.now(timezone.utc)
|
||||||
|
self.events: list[dict] = []
|
||||||
|
|
||||||
|
async def load(self) -> None:
|
||||||
|
db = self.db
|
||||||
|
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||||
|
self.groups = {g.name: g for g in (await db.execute(
|
||||||
|
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
|
||||||
|
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||||
|
self.rows: dict[str, dict] = {}
|
||||||
|
for kind, model in MODELS.items():
|
||||||
|
options = [selectinload(model.deployments)]
|
||||||
|
if kind != "policy_hierarchy_rules":
|
||||||
|
options += [selectinload(model.targets), selectinload(model.groups)]
|
||||||
|
rows = (await db.execute(select(model).options(*options))).scalars().all()
|
||||||
|
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||||
|
|
||||||
|
def items(self, kind: str, *actions: str) -> list[planner.Item]:
|
||||||
|
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
|
||||||
|
|
||||||
|
def event(self, item: planner.Item, entity_id, name: str) -> None:
|
||||||
|
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
|
||||||
|
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
|
||||||
|
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
|
||||||
|
|
||||||
|
# -- groups ----------------------------------------------------------------------------------------------------------
|
||||||
|
async def hierarchy_groups(self) -> None:
|
||||||
|
items = self.items("policy_hierarchy_groups", "create", "update")
|
||||||
|
for item in items:
|
||||||
|
if item.action == "create":
|
||||||
|
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
|
||||||
|
self.db.add(group)
|
||||||
|
self.hierarchy[item.key] = group
|
||||||
|
await self.db.flush()
|
||||||
|
for item in items:
|
||||||
|
group = self.hierarchy[item.key]
|
||||||
|
group.description = item.after["description"]
|
||||||
|
parent = item.after["parent"]
|
||||||
|
group.parent_id = self.hierarchy[parent].id if parent else None
|
||||||
|
self.event(item, group.id, group.name)
|
||||||
|
await self.db.flush()
|
||||||
|
|
||||||
|
async def device_groups(self) -> None:
|
||||||
|
for item in self.items("device_groups", "create", "update"):
|
||||||
|
group = self.groups.get(item.key)
|
||||||
|
if group is None:
|
||||||
|
group = DeviceGroup(name=item.key)
|
||||||
|
self.db.add(group)
|
||||||
|
self.groups[item.key] = group
|
||||||
|
group.description = item.after["description"]
|
||||||
|
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
|
||||||
|
await self.db.flush()
|
||||||
|
self.event(item, group.id, group.name)
|
||||||
|
|
||||||
|
# -- objects ---------------------------------------------------------------------------------------------------------
|
||||||
|
async def objects(self, kind: str) -> None:
|
||||||
|
model = MODELS[kind]
|
||||||
|
revision_model, fk, snapshot = _revision_writer(kind)
|
||||||
|
for item in self.items(kind, "create", "update"):
|
||||||
|
after = item.after
|
||||||
|
row = self.rows[kind].get(item.key)
|
||||||
|
if row is None:
|
||||||
|
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
|
||||||
|
if kind != "aliases":
|
||||||
|
row.config_key = item.key
|
||||||
|
self.db.add(row)
|
||||||
|
self.rows[kind][item.key] = row
|
||||||
|
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
|
||||||
|
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
|
||||||
|
for name in SCALARS[kind]:
|
||||||
|
setattr(row, name, after[name])
|
||||||
|
if kind == "aliases":
|
||||||
|
row.content = "\n".join(after["entries"])
|
||||||
|
if kind == "policy_hierarchy_rules":
|
||||||
|
row.hierarchy_group_id = self.hierarchy[after["group"]].id
|
||||||
|
else:
|
||||||
|
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
|
||||||
|
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
|
||||||
|
if "hierarchy_group" in after["targets"]:
|
||||||
|
hierarchy = after["targets"]["hierarchy_group"]
|
||||||
|
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
|
||||||
|
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
|
||||||
|
await self.db.flush()
|
||||||
|
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
|
||||||
|
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||||
|
|
||||||
|
async def hierarchy_positions(self) -> None:
|
||||||
|
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
|
||||||
|
if "policy_hierarchy_rules" not in self.plan.touched:
|
||||||
|
return
|
||||||
|
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
|
||||||
|
for key, rank in self.plan.positions.items():
|
||||||
|
row = self.rows["policy_hierarchy_rules"].get(key)
|
||||||
|
if row is not None and key not in deleted and row.sequence != rank:
|
||||||
|
row.sequence = rank
|
||||||
|
await self.db.flush()
|
||||||
|
|
||||||
|
async def host_hierarchy(self) -> None:
|
||||||
|
for host_name, group_name in self.plan.host_hierarchy.items():
|
||||||
|
host = self.hosts[host_name]
|
||||||
|
wanted = self.hierarchy[group_name].id if group_name else None
|
||||||
|
if host.policy_hierarchy_group_id != wanted:
|
||||||
|
host.policy_hierarchy_group_id = wanted
|
||||||
|
await self.db.flush()
|
||||||
|
|
||||||
|
async def variables(self) -> None:
|
||||||
|
if "variables" not in self.plan.touched:
|
||||||
|
return
|
||||||
|
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
|
||||||
|
for item in self.items("variables", "create", "update"):
|
||||||
|
host = self.hosts[item.after["host"]]
|
||||||
|
row = existing.get((host.id, item.after["name"]))
|
||||||
|
if row is None:
|
||||||
|
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
|
||||||
|
else:
|
||||||
|
row.value = item.after["value"]
|
||||||
|
self.event(item, None, item.key)
|
||||||
|
for item in self.items("variables", "delete"):
|
||||||
|
host = self.hosts[item.before["host"]]
|
||||||
|
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
|
||||||
|
self.event(item, None, item.key)
|
||||||
|
await self.db.flush()
|
||||||
|
|
||||||
|
# -- deletions -------------------------------------------------------------------------------------------------------
|
||||||
|
async def deletions(self) -> None:
|
||||||
|
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
|
||||||
|
for item in self.items(kind, "delete"):
|
||||||
|
row = self.rows[kind][item.key]
|
||||||
|
if row.deployments: # the planner blocks this; never undeploy from here
|
||||||
|
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
|
||||||
|
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||||
|
await self.db.delete(row)
|
||||||
|
await self.db.flush()
|
||||||
|
for item in self.items("device_groups", "delete"):
|
||||||
|
row = self.groups[item.key]
|
||||||
|
self.event(item, row.id, row.name)
|
||||||
|
await self.db.delete(row)
|
||||||
|
await self.db.flush()
|
||||||
|
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
|
||||||
|
by_id = {g.id: g for g in self.hierarchy.values()}
|
||||||
|
|
||||||
|
def depth(group) -> int:
|
||||||
|
n, current = 0, group
|
||||||
|
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
|
||||||
|
n, current = n + 1, by_id[current.parent_id]
|
||||||
|
return n
|
||||||
|
|
||||||
|
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
|
||||||
|
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
|
||||||
|
self.event(item, row.id, row.name)
|
||||||
|
await self.db.delete(row)
|
||||||
|
await self.db.flush()
|
||||||
|
|
||||||
|
|
||||||
|
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
|
||||||
|
db.expire_all() # compare what is in the database, not what the session believes
|
||||||
|
actual = await canonical(db, set(plan.touched))
|
||||||
|
differences: list[str] = []
|
||||||
|
for kind in plan.touched:
|
||||||
|
if kind == "variables":
|
||||||
|
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||||
|
for h, values in actual["variables"].items() for n, v in values.items()}
|
||||||
|
else:
|
||||||
|
ident = planner.KIND_SPECS[kind].ident
|
||||||
|
got = {item[ident]: item for item in actual[kind]}
|
||||||
|
want = plan.expected[kind]
|
||||||
|
for key in sorted(set(got) | set(want)):
|
||||||
|
if got.get(key) != want.get(key):
|
||||||
|
differences.append(f"{kind}:{key}")
|
||||||
|
if "policy_hierarchy_rules" in plan.touched:
|
||||||
|
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
|
||||||
|
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
|
||||||
|
for key, rank in ranks.items():
|
||||||
|
if plan.positions.get(key) != rank:
|
||||||
|
differences.append(f"policy_hierarchy_rules:{key}:position")
|
||||||
|
if differences:
|
||||||
|
raise SelfCheckFailed(differences)
|
||||||
|
|
||||||
|
|
||||||
|
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
|
||||||
|
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
|
||||||
|
ConfigError; on any exception the caller must roll back."""
|
||||||
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
|
await policy_lock(db)
|
||||||
|
plan = await planner.build_plan(db, parsed, prune)
|
||||||
|
if plan.token != token:
|
||||||
|
raise PlanStale()
|
||||||
|
if not plan.applicable:
|
||||||
|
raise PlanBlocked(plan)
|
||||||
|
writer = _Writer(db, plan)
|
||||||
|
await writer.load()
|
||||||
|
await writer.hierarchy_groups()
|
||||||
|
await writer.device_groups()
|
||||||
|
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
|
||||||
|
if kind in plan.touched:
|
||||||
|
await writer.objects(kind)
|
||||||
|
await writer.hierarchy_positions()
|
||||||
|
await writer.host_hierarchy()
|
||||||
|
await writer.variables()
|
||||||
|
await writer.deletions()
|
||||||
|
await _self_check(db, plan)
|
||||||
|
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
|
||||||
|
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
|
||||||
|
"summary": plan.summary()}))
|
||||||
|
for values in writer.events:
|
||||||
|
db.add(audit_service.make_event(**values))
|
||||||
|
await db.flush()
|
||||||
|
return plan
|
||||||
@@ -0,0 +1,154 @@
|
|||||||
|
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
|
||||||
|
|
||||||
|
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
|
||||||
|
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
|
||||||
|
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
|
||||||
|
"""
|
||||||
|
import yaml
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
|
||||||
|
from app.config_code import schema
|
||||||
|
from app.models.alias import FirewallAlias
|
||||||
|
from app.models.host import OpnsenseHost
|
||||||
|
from app.models.one_to_one import OneToOneRule
|
||||||
|
from app.models.outbound_nat import OutboundNatRule
|
||||||
|
from app.models.policy import DeviceGroup
|
||||||
|
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||||
|
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||||
|
from app.models.port_forward import PortForwardRule
|
||||||
|
from app.models.rule import FirewallRule
|
||||||
|
from app.version import APP_VERSION
|
||||||
|
|
||||||
|
HEADER = (
|
||||||
|
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
|
||||||
|
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
|
||||||
|
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
|
||||||
|
fields = list(doc.model_fields)
|
||||||
|
if set(values) != set(fields):
|
||||||
|
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
|
||||||
|
return {name: values[name] for name in fields}
|
||||||
|
|
||||||
|
|
||||||
|
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
|
||||||
|
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
|
||||||
|
if "hierarchy_group" in doc.model_fields:
|
||||||
|
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
|
||||||
|
return _ordered(doc, values)
|
||||||
|
|
||||||
|
|
||||||
|
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
|
||||||
|
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
|
||||||
|
values = dict(extra)
|
||||||
|
for name in doc.model_fields:
|
||||||
|
if name in values:
|
||||||
|
continue
|
||||||
|
values[name] = obj.config_key if name == "key" else getattr(obj, name)
|
||||||
|
return _ordered(doc, values)
|
||||||
|
|
||||||
|
|
||||||
|
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
|
||||||
|
"""Parents before children, siblings by name; the same rule the loader expects."""
|
||||||
|
children: dict = {}
|
||||||
|
for group in groups:
|
||||||
|
children.setdefault(group.parent_id, []).append(group)
|
||||||
|
ordered: list[PolicyHierarchyGroup] = []
|
||||||
|
|
||||||
|
def walk(parent_id) -> None:
|
||||||
|
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
|
||||||
|
ordered.append(group)
|
||||||
|
walk(group.id)
|
||||||
|
|
||||||
|
walk(None)
|
||||||
|
return ordered
|
||||||
|
|
||||||
|
|
||||||
|
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
|
||||||
|
"""{kind: content} for the requested kinds (default all), in file order."""
|
||||||
|
wanted = set(schema.KINDS) if kinds is None else kinds
|
||||||
|
out: dict[str, list | dict] = {}
|
||||||
|
|
||||||
|
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
|
||||||
|
hierarchy_names = {g.id: g.name for g in all_groups}
|
||||||
|
|
||||||
|
if "device_groups" in wanted:
|
||||||
|
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
|
||||||
|
out["device_groups"] = [
|
||||||
|
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
|
||||||
|
for g in sorted(groups, key=lambda g: g.name)
|
||||||
|
]
|
||||||
|
|
||||||
|
if "policy_hierarchy_groups" in wanted:
|
||||||
|
out["policy_hierarchy_groups"] = [
|
||||||
|
_ordered(schema.HierarchyGroupDoc, {
|
||||||
|
"name": g.name,
|
||||||
|
"parent": hierarchy_names.get(g.parent_id),
|
||||||
|
"description": g.description or "",
|
||||||
|
"hosts": sorted(h.name for h in g.hosts),
|
||||||
|
})
|
||||||
|
for g in _hierarchy_order(list(all_groups))
|
||||||
|
]
|
||||||
|
|
||||||
|
if "aliases" in wanted:
|
||||||
|
aliases = (await db.execute(
|
||||||
|
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
|
||||||
|
)).scalars().all()
|
||||||
|
out["aliases"] = [
|
||||||
|
_ordered(schema.AliasDoc, {
|
||||||
|
"name": a.name,
|
||||||
|
"type": a.type,
|
||||||
|
"description": a.description or "",
|
||||||
|
"enabled": a.enabled,
|
||||||
|
"entries": [line for line in (a.content or "").split("\n") if line],
|
||||||
|
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
|
||||||
|
})
|
||||||
|
for a in sorted(aliases, key=lambda a: a.name)
|
||||||
|
]
|
||||||
|
|
||||||
|
simple = (
|
||||||
|
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
|
||||||
|
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
|
||||||
|
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
|
||||||
|
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
|
||||||
|
)
|
||||||
|
for kind, model, doc, targets_doc in simple:
|
||||||
|
if kind not in wanted:
|
||||||
|
continue
|
||||||
|
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
|
||||||
|
out[kind] = [
|
||||||
|
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
|
||||||
|
for row in sorted(rows, key=lambda r: r.config_key)
|
||||||
|
]
|
||||||
|
|
||||||
|
if "policy_hierarchy_rules" in wanted:
|
||||||
|
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
|
||||||
|
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
|
||||||
|
out["policy_hierarchy_rules"] = [
|
||||||
|
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
|
||||||
|
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
|
||||||
|
]
|
||||||
|
|
||||||
|
if "variables" in wanted:
|
||||||
|
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
|
||||||
|
out["variables"] = {
|
||||||
|
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
|
||||||
|
for h in sorted(hosts, key=lambda h: h.name)
|
||||||
|
if h.variables
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def dump_yaml(content: dict[str, list | dict]) -> str:
|
||||||
|
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
|
||||||
|
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
|
||||||
|
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
|
||||||
|
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
|
||||||
|
|
||||||
|
|
||||||
|
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
|
||||||
|
return dump_yaml(await canonical(db, kinds))
|
||||||
@@ -0,0 +1,307 @@
|
|||||||
|
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
|
||||||
|
|
||||||
|
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
|
||||||
|
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
|
||||||
|
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
|
||||||
|
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
|
||||||
|
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
|
||||||
|
|
||||||
|
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import re
|
||||||
|
from typing import NamedTuple
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from pydantic import ValidationError
|
||||||
|
from yaml.composer import Composer
|
||||||
|
from yaml.constructor import SafeConstructor
|
||||||
|
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
|
||||||
|
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
|
||||||
|
from yaml.parser import Parser
|
||||||
|
from yaml.reader import Reader
|
||||||
|
from yaml.resolver import BaseResolver
|
||||||
|
from yaml.scanner import Scanner
|
||||||
|
|
||||||
|
from app.config_code import schema
|
||||||
|
|
||||||
|
MAX_BYTES = 1024 * 1024
|
||||||
|
MAX_DEPTH = 8
|
||||||
|
MAX_NODES = 100_000
|
||||||
|
MAX_SCALAR = schema.MAX_SCALAR
|
||||||
|
MAX_OBJECTS_PER_KIND = 5000
|
||||||
|
MAX_VARIABLE_HOSTS = 500
|
||||||
|
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
|
||||||
|
MAX_ALIAS_ENTRIES = 2000
|
||||||
|
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
|
||||||
|
|
||||||
|
_STR = "tag:yaml.org,2002:str"
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigError(Exception):
|
||||||
|
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
|
||||||
|
|
||||||
|
def __init__(self, code: str, errors: list[dict]):
|
||||||
|
self.code = code
|
||||||
|
self.errors = errors
|
||||||
|
super().__init__(f"{code}: {errors[:3]}")
|
||||||
|
|
||||||
|
|
||||||
|
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
|
||||||
|
return {"code": code, "path": path, "line": line, "params": params}
|
||||||
|
|
||||||
|
|
||||||
|
def display(value, limit: int = 80) -> str:
|
||||||
|
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
|
||||||
|
text = value if isinstance(value, str) else repr(value)
|
||||||
|
text = text.encode("unicode_escape").decode("ascii")
|
||||||
|
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||||
|
|
||||||
|
|
||||||
|
def _path(parts) -> str:
|
||||||
|
out = ""
|
||||||
|
for part in parts:
|
||||||
|
if isinstance(part, int):
|
||||||
|
out += f"[{part}]"
|
||||||
|
else:
|
||||||
|
out += ("." if out else "") + str(part)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class _Rejected(yaml.YAMLError):
|
||||||
|
def __init__(self, code: str, mark=None, path=(), **params):
|
||||||
|
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
|
||||||
|
super().__init__(code)
|
||||||
|
|
||||||
|
|
||||||
|
class StrictResolver(BaseResolver):
|
||||||
|
yaml_implicit_resolvers: dict = {}
|
||||||
|
|
||||||
|
|
||||||
|
StrictResolver.add_implicit_resolver(
|
||||||
|
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
|
||||||
|
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
|
||||||
|
StrictResolver.add_implicit_resolver(
|
||||||
|
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
|
||||||
|
StrictResolver.add_implicit_resolver(
|
||||||
|
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
|
||||||
|
|
||||||
|
|
||||||
|
class StrictConstructor(SafeConstructor):
|
||||||
|
yaml_constructors: dict = {}
|
||||||
|
|
||||||
|
|
||||||
|
for _tag, _fn in (
|
||||||
|
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
|
||||||
|
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
|
||||||
|
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
|
||||||
|
(_STR, SafeConstructor.construct_yaml_str),
|
||||||
|
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
|
||||||
|
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
|
||||||
|
):
|
||||||
|
StrictConstructor.add_constructor(_tag, _fn)
|
||||||
|
|
||||||
|
|
||||||
|
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
|
||||||
|
def __init__(self, stream: str):
|
||||||
|
Reader.__init__(self, stream)
|
||||||
|
Scanner.__init__(self)
|
||||||
|
Parser.__init__(self)
|
||||||
|
Composer.__init__(self)
|
||||||
|
StrictConstructor.__init__(self)
|
||||||
|
StrictResolver.__init__(self)
|
||||||
|
self._depth = 0
|
||||||
|
self._nodes = 0
|
||||||
|
self._path: list = []
|
||||||
|
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
|
||||||
|
|
||||||
|
# -- document level ---------------------------------------------------------------------------------------------
|
||||||
|
def get_single_node(self):
|
||||||
|
self.get_event() # StreamStart
|
||||||
|
if self.check_event(StreamEndEvent):
|
||||||
|
raise _Rejected("yaml_empty")
|
||||||
|
start = self.peek_event()
|
||||||
|
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
|
||||||
|
raise _Rejected("yaml_directive_forbidden", start.start_mark)
|
||||||
|
node = self.compose_document()
|
||||||
|
if not self.check_event(StreamEndEvent):
|
||||||
|
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
|
||||||
|
self.get_event()
|
||||||
|
return node
|
||||||
|
|
||||||
|
# -- node level ---------------------------------------------------------------------------------------------------
|
||||||
|
def compose_node(self, parent, index):
|
||||||
|
event = self.peek_event()
|
||||||
|
if isinstance(event, AliasEvent):
|
||||||
|
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
|
||||||
|
if getattr(event, "anchor", None) is not None:
|
||||||
|
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
|
||||||
|
if getattr(event, "tag", None) is not None:
|
||||||
|
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
|
||||||
|
self._nodes += 1
|
||||||
|
if self._nodes > MAX_NODES:
|
||||||
|
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
|
||||||
|
|
||||||
|
if isinstance(index, ScalarNode):
|
||||||
|
segment = index.value
|
||||||
|
elif isinstance(index, int) and not isinstance(index, bool):
|
||||||
|
segment = index
|
||||||
|
else:
|
||||||
|
segment = None # a mapping key
|
||||||
|
if segment is not None:
|
||||||
|
self._path.append(segment)
|
||||||
|
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
|
||||||
|
self._depth += 1
|
||||||
|
try:
|
||||||
|
if self._depth > MAX_DEPTH:
|
||||||
|
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
|
||||||
|
return super().compose_node(parent, index)
|
||||||
|
finally:
|
||||||
|
self._depth -= 1
|
||||||
|
if segment is not None:
|
||||||
|
self._path.pop()
|
||||||
|
|
||||||
|
def compose_scalar_node(self, anchor):
|
||||||
|
event = self.peek_event()
|
||||||
|
if len(event.value) > MAX_SCALAR:
|
||||||
|
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
|
||||||
|
return super().compose_scalar_node(anchor)
|
||||||
|
|
||||||
|
def compose_mapping_node(self, anchor):
|
||||||
|
node = super().compose_mapping_node(anchor)
|
||||||
|
seen: set[str] = set()
|
||||||
|
for key, _value in node.value:
|
||||||
|
if not isinstance(key, ScalarNode) or key.tag != _STR:
|
||||||
|
raise _Rejected("non_string_key", key.start_mark, self._path)
|
||||||
|
if key.value == "<<":
|
||||||
|
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
|
||||||
|
if key.value in seen:
|
||||||
|
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
|
||||||
|
seen.add(key.value)
|
||||||
|
return node
|
||||||
|
|
||||||
|
|
||||||
|
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
|
||||||
|
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
|
||||||
|
if isinstance(raw, str):
|
||||||
|
raw = raw.encode("utf-8")
|
||||||
|
if len(raw) > MAX_BYTES:
|
||||||
|
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
|
||||||
|
try:
|
||||||
|
text = raw.decode("utf-8", errors="strict")
|
||||||
|
except UnicodeDecodeError as exc:
|
||||||
|
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
|
||||||
|
if text.startswith(""):
|
||||||
|
text = text[1:]
|
||||||
|
loader = None
|
||||||
|
try:
|
||||||
|
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
|
||||||
|
node = loader.get_single_node()
|
||||||
|
data = loader.construct_document(node)
|
||||||
|
except _Rejected as exc:
|
||||||
|
line = exc.mark.line + 1 if exc.mark is not None else None
|
||||||
|
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
|
||||||
|
except yaml.MarkedYAMLError as exc:
|
||||||
|
mark = exc.problem_mark or exc.context_mark
|
||||||
|
line = mark.line + 1 if mark is not None else None
|
||||||
|
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
|
||||||
|
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
|
||||||
|
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
|
||||||
|
finally:
|
||||||
|
if loader is not None:
|
||||||
|
loader.dispose()
|
||||||
|
return data, loader.lines
|
||||||
|
|
||||||
|
|
||||||
|
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
|
||||||
|
for end in range(len(loc), 0, -1):
|
||||||
|
if loc[:end] in lines:
|
||||||
|
return lines[loc[:end]]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
_PYDANTIC_CODES = {
|
||||||
|
"extra_forbidden": "unknown_field",
|
||||||
|
"missing": "missing_field",
|
||||||
|
"literal_error": "bad_value",
|
||||||
|
"string_too_short": "too_short",
|
||||||
|
"string_too_long": "too_long",
|
||||||
|
"string_pattern_mismatch": "bad_format",
|
||||||
|
}
|
||||||
|
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
|
||||||
|
|
||||||
|
|
||||||
|
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
|
||||||
|
out = []
|
||||||
|
for error in exc.errors(include_url=False):
|
||||||
|
loc = tuple(error["loc"])
|
||||||
|
code = _PYDANTIC_CODES.get(error["type"])
|
||||||
|
if code is None and error["type"] == "value_error":
|
||||||
|
message = str(error.get("ctx", {}).get("error", ""))
|
||||||
|
code = message if message in _OWN_CODES else "bad_value"
|
||||||
|
if code is None:
|
||||||
|
code = "bad_type"
|
||||||
|
params = {}
|
||||||
|
if error["type"] == "extra_forbidden":
|
||||||
|
params["field"] = display(loc[-1]) if loc else ""
|
||||||
|
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
|
||||||
|
params["value"] = display(error["input"])
|
||||||
|
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
|
||||||
|
errors = []
|
||||||
|
for kind in schema.KINDS:
|
||||||
|
section = data.get(kind)
|
||||||
|
if kind == "variables":
|
||||||
|
if isinstance(section, dict):
|
||||||
|
if len(section) > MAX_VARIABLE_HOSTS:
|
||||||
|
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
|
||||||
|
for host, values in section.items():
|
||||||
|
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
|
||||||
|
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
|
||||||
|
max=MAX_VARIABLES_PER_HOST))
|
||||||
|
continue
|
||||||
|
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
|
||||||
|
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
|
||||||
|
aliases = data.get("aliases")
|
||||||
|
for i, alias in enumerate(aliases if isinstance(aliases, list) else []): # any other shape is for the schema check
|
||||||
|
entries = alias.get("entries") if isinstance(alias, dict) else None
|
||||||
|
if isinstance(entries, list):
|
||||||
|
loc = ("aliases", i, "entries")
|
||||||
|
if len(entries) > MAX_ALIAS_ENTRIES:
|
||||||
|
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
|
||||||
|
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
|
||||||
|
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
|
||||||
|
return errors
|
||||||
|
|
||||||
|
|
||||||
|
class Parsed(NamedTuple):
|
||||||
|
doc: schema.ConfigDoc
|
||||||
|
sha256: str # of the raw bytes, shown in the plan and stored in the audit
|
||||||
|
lines: dict[tuple, int] # path -> line, for messages of the planner
|
||||||
|
|
||||||
|
|
||||||
|
def parse(raw: bytes | str) -> Parsed:
|
||||||
|
"""Load, limit and validate a file. Raises ConfigError."""
|
||||||
|
if isinstance(raw, str):
|
||||||
|
raw = raw.encode("utf-8")
|
||||||
|
sha256 = hashlib.sha256(raw).hexdigest()
|
||||||
|
data, lines = load_yaml(raw)
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
|
||||||
|
version = data.get("opnexus_config")
|
||||||
|
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
|
||||||
|
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
|
||||||
|
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
|
||||||
|
"opnexus_config", _line_for(lines, ("opnexus_config",)),
|
||||||
|
**({"value": display(version)} if version is not None else {}))])
|
||||||
|
errors = _check_counts(data, lines)
|
||||||
|
if errors:
|
||||||
|
raise ConfigError("CONFIG_INVALID", errors)
|
||||||
|
try:
|
||||||
|
doc = schema.ConfigDoc.model_validate(data)
|
||||||
|
except ValidationError as exc:
|
||||||
|
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
|
||||||
|
return Parsed(doc, sha256, lines)
|
||||||
@@ -0,0 +1,933 @@
|
|||||||
|
{
|
||||||
|
"$defs": {
|
||||||
|
"AliasDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"name": {
|
||||||
|
"maxLength": 64,
|
||||||
|
"minLength": 1,
|
||||||
|
"pattern": "^[A-Za-z0-9_]+$",
|
||||||
|
"title": "Name",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"type": {
|
||||||
|
"default": "host",
|
||||||
|
"enum": [
|
||||||
|
"host",
|
||||||
|
"network",
|
||||||
|
"port",
|
||||||
|
"networkgroup",
|
||||||
|
"mac",
|
||||||
|
"asn"
|
||||||
|
],
|
||||||
|
"title": "Type",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"default": "",
|
||||||
|
"maxLength": 255,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"enabled": {
|
||||||
|
"default": true,
|
||||||
|
"title": "Enabled",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"entries": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 8000,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Entries",
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
"targets": {
|
||||||
|
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"name"
|
||||||
|
],
|
||||||
|
"title": "AliasDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"DeviceGroupDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"name": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Name",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"default": "",
|
||||||
|
"maxLength": 2000,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"hosts": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Hosts",
|
||||||
|
"type": "array"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"name"
|
||||||
|
],
|
||||||
|
"title": "DeviceGroupDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"HierarchyGroupDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"name": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Name",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"parent": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Parent"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"default": "",
|
||||||
|
"maxLength": 2000,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"hosts": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Hosts",
|
||||||
|
"type": "array"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"name"
|
||||||
|
],
|
||||||
|
"title": "HierarchyGroupDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"HierarchyRuleDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"key": {
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||||
|
"title": "Key",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"group": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Group",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"rulebase": {
|
||||||
|
"default": "pre",
|
||||||
|
"enum": [
|
||||||
|
"pre",
|
||||||
|
"post"
|
||||||
|
],
|
||||||
|
"title": "Rulebase",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"maxLength": 255,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"interface": {
|
||||||
|
"maxLength": 64,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Interface",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"direction": {
|
||||||
|
"default": "in",
|
||||||
|
"enum": [
|
||||||
|
"in",
|
||||||
|
"out"
|
||||||
|
],
|
||||||
|
"title": "Direction",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"ipprotocol": {
|
||||||
|
"default": "inet",
|
||||||
|
"enum": [
|
||||||
|
"inet",
|
||||||
|
"inet6"
|
||||||
|
],
|
||||||
|
"title": "Ipprotocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"protocol": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 16,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Protocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"action": {
|
||||||
|
"default": "pass",
|
||||||
|
"enum": [
|
||||||
|
"pass",
|
||||||
|
"block",
|
||||||
|
"reject"
|
||||||
|
],
|
||||||
|
"title": "Action",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Source Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Source Port"
|
||||||
|
},
|
||||||
|
"destination_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Destination Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"destination_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Destination Port"
|
||||||
|
},
|
||||||
|
"enabled": {
|
||||||
|
"default": true,
|
||||||
|
"title": "Enabled",
|
||||||
|
"type": "boolean"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"key",
|
||||||
|
"group",
|
||||||
|
"description",
|
||||||
|
"interface"
|
||||||
|
],
|
||||||
|
"title": "HierarchyRuleDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"OneToOneDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"key": {
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||||
|
"title": "Key",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"maxLength": 255,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"interface": {
|
||||||
|
"maxLength": 64,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Interface",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"type": {
|
||||||
|
"default": "binat",
|
||||||
|
"enum": [
|
||||||
|
"binat",
|
||||||
|
"nat"
|
||||||
|
],
|
||||||
|
"title": "Type",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"external": {
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "External",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Source Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"destination_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Destination Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_not": {
|
||||||
|
"default": false,
|
||||||
|
"title": "Source Not",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"destination_not": {
|
||||||
|
"default": false,
|
||||||
|
"title": "Destination Not",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"natreflection": {
|
||||||
|
"default": "",
|
||||||
|
"enum": [
|
||||||
|
"",
|
||||||
|
"enable",
|
||||||
|
"disable"
|
||||||
|
],
|
||||||
|
"title": "Natreflection",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"log": {
|
||||||
|
"default": false,
|
||||||
|
"title": "Log",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"enabled": {
|
||||||
|
"default": true,
|
||||||
|
"title": "Enabled",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"targets": {
|
||||||
|
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"key",
|
||||||
|
"description",
|
||||||
|
"interface",
|
||||||
|
"external"
|
||||||
|
],
|
||||||
|
"title": "OneToOneDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"OutboundNatDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"key": {
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||||
|
"title": "Key",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"maxLength": 255,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"interface": {
|
||||||
|
"maxLength": 64,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Interface",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"ipprotocol": {
|
||||||
|
"default": "inet",
|
||||||
|
"enum": [
|
||||||
|
"inet",
|
||||||
|
"inet6"
|
||||||
|
],
|
||||||
|
"title": "Ipprotocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"protocol": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 16,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Protocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Source Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Source Port"
|
||||||
|
},
|
||||||
|
"destination_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Destination Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"destination_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Destination Port"
|
||||||
|
},
|
||||||
|
"source_not": {
|
||||||
|
"default": false,
|
||||||
|
"title": "Source Not",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"destination_not": {
|
||||||
|
"default": false,
|
||||||
|
"title": "Destination Not",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"target": {
|
||||||
|
"default": "",
|
||||||
|
"maxLength": 128,
|
||||||
|
"title": "Target",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"target_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Target Port"
|
||||||
|
},
|
||||||
|
"enabled": {
|
||||||
|
"default": true,
|
||||||
|
"title": "Enabled",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"targets": {
|
||||||
|
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"key",
|
||||||
|
"description",
|
||||||
|
"interface"
|
||||||
|
],
|
||||||
|
"title": "OutboundNatDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"PortForwardDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"key": {
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||||
|
"title": "Key",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"maxLength": 255,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"interface": {
|
||||||
|
"maxLength": 64,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Interface",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"ipprotocol": {
|
||||||
|
"default": "inet",
|
||||||
|
"enum": [
|
||||||
|
"inet",
|
||||||
|
"inet6"
|
||||||
|
],
|
||||||
|
"title": "Ipprotocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"protocol": {
|
||||||
|
"default": "tcp",
|
||||||
|
"maxLength": 16,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Protocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Source Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Source Port"
|
||||||
|
},
|
||||||
|
"destination_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Destination Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"destination_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Destination Port"
|
||||||
|
},
|
||||||
|
"target": {
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Target",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"local_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Local Port"
|
||||||
|
},
|
||||||
|
"natreflection": {
|
||||||
|
"default": "",
|
||||||
|
"enum": [
|
||||||
|
"",
|
||||||
|
"purenat",
|
||||||
|
"disable"
|
||||||
|
],
|
||||||
|
"title": "Natreflection",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"enabled": {
|
||||||
|
"default": true,
|
||||||
|
"title": "Enabled",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"targets": {
|
||||||
|
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"key",
|
||||||
|
"description",
|
||||||
|
"interface",
|
||||||
|
"target"
|
||||||
|
],
|
||||||
|
"title": "PortForwardDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"RuleDoc": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"key": {
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||||
|
"title": "Key",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"maxLength": 255,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Description",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"interface": {
|
||||||
|
"maxLength": 64,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Interface",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"direction": {
|
||||||
|
"default": "in",
|
||||||
|
"enum": [
|
||||||
|
"in",
|
||||||
|
"out"
|
||||||
|
],
|
||||||
|
"title": "Direction",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"ipprotocol": {
|
||||||
|
"default": "inet",
|
||||||
|
"enum": [
|
||||||
|
"inet",
|
||||||
|
"inet6"
|
||||||
|
],
|
||||||
|
"title": "Ipprotocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"protocol": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 16,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Protocol",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"action": {
|
||||||
|
"default": "pass",
|
||||||
|
"enum": [
|
||||||
|
"pass",
|
||||||
|
"block",
|
||||||
|
"reject"
|
||||||
|
],
|
||||||
|
"title": "Action",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Source Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"source_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Source Port"
|
||||||
|
},
|
||||||
|
"destination_net": {
|
||||||
|
"default": "any",
|
||||||
|
"maxLength": 128,
|
||||||
|
"minLength": 1,
|
||||||
|
"title": "Destination Net",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"destination_port": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 32,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Destination Port"
|
||||||
|
},
|
||||||
|
"enabled": {
|
||||||
|
"default": true,
|
||||||
|
"title": "Enabled",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"log": {
|
||||||
|
"default": false,
|
||||||
|
"title": "Log",
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"targets": {
|
||||||
|
"$ref": "#/$defs/Targets"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"key",
|
||||||
|
"description",
|
||||||
|
"interface"
|
||||||
|
],
|
||||||
|
"title": "RuleDoc",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"Targets": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"hosts": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Hosts",
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
"groups": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Groups",
|
||||||
|
"type": "array"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"title": "Targets",
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"TargetsWithHierarchy": {
|
||||||
|
"additionalProperties": false,
|
||||||
|
"properties": {
|
||||||
|
"hosts": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Hosts",
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
"groups": {
|
||||||
|
"items": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"title": "Groups",
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
"hierarchy_group": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1,
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Hierarchy Group"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"title": "TargetsWithHierarchy",
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additionalProperties": false,
|
||||||
|
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
|
||||||
|
"properties": {
|
||||||
|
"opnexus_config": {
|
||||||
|
"const": 1,
|
||||||
|
"title": "Opnexus Config",
|
||||||
|
"type": "integer"
|
||||||
|
},
|
||||||
|
"device_groups": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/DeviceGroupDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Device Groups"
|
||||||
|
},
|
||||||
|
"policy_hierarchy_groups": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/HierarchyGroupDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Policy Hierarchy Groups"
|
||||||
|
},
|
||||||
|
"aliases": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/AliasDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Aliases"
|
||||||
|
},
|
||||||
|
"rules": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/RuleDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Rules"
|
||||||
|
},
|
||||||
|
"port_forwards": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/PortForwardDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Port Forwards"
|
||||||
|
},
|
||||||
|
"one_to_one": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/OneToOneDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "One To One"
|
||||||
|
},
|
||||||
|
"outbound_nat": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/OutboundNatDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Outbound Nat"
|
||||||
|
},
|
||||||
|
"policy_hierarchy_rules": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/$defs/HierarchyRuleDoc"
|
||||||
|
},
|
||||||
|
"type": "array"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Policy Hierarchy Rules"
|
||||||
|
},
|
||||||
|
"variables": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"additionalProperties": {
|
||||||
|
"additionalProperties": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"propertyNames": {
|
||||||
|
"maxLength": 120,
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "null"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default": null,
|
||||||
|
"title": "Variables"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"opnexus_config"
|
||||||
|
],
|
||||||
|
"title": "OPNexus-Konfiguration (Schema v1)",
|
||||||
|
"type": "object",
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||||
|
}
|
||||||
@@ -0,0 +1,570 @@
|
|||||||
|
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
|
||||||
|
|
||||||
|
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
|
||||||
|
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
|
||||||
|
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
|
||||||
|
`prune` gleicht die Art ab.
|
||||||
|
|
||||||
|
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
|
||||||
|
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
|
||||||
|
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
|
||||||
|
|
||||||
|
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
|
||||||
|
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import uuid
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
from pydantic import ValidationError
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
|
||||||
|
from app.config_code import schema
|
||||||
|
from app.config_code.export import canonical
|
||||||
|
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
|
||||||
|
from app.database import Base
|
||||||
|
from app.services.deploy_flow import is_unsettled
|
||||||
|
from app.models.alias import FirewallAlias
|
||||||
|
from app.models.host import OpnsenseHost
|
||||||
|
from app.models.one_to_one import OneToOneRule
|
||||||
|
from app.models.outbound_nat import OutboundNatRule
|
||||||
|
from app.models.policy import DeviceGroup
|
||||||
|
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||||
|
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||||
|
from app.models.port_forward import PortForwardRule
|
||||||
|
from app.models.rule import FirewallRule
|
||||||
|
from app.schemas.alias import AliasCreate
|
||||||
|
from app.schemas.one_to_one import OneToOneCreate
|
||||||
|
from app.schemas.outbound_nat import OutboundNatCreate
|
||||||
|
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||||
|
from app.schemas.port_forward import PortForwardCreate
|
||||||
|
from app.schemas.rule import RuleCreate
|
||||||
|
from app.services.variables_service import names_in
|
||||||
|
|
||||||
|
DEPLOYED = ("success", "pending", "revert_failed")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Kind:
|
||||||
|
section: str
|
||||||
|
model: type | None
|
||||||
|
ident: str # identity field in the file ("key"/"name")
|
||||||
|
api: type | None = None # API create schema, run on every file object (same validators as the UI)
|
||||||
|
deployable: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
KIND_SPECS = {
|
||||||
|
"device_groups": Kind("device_groups", DeviceGroup, "name"),
|
||||||
|
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
|
||||||
|
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
|
||||||
|
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
|
||||||
|
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
|
||||||
|
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
|
||||||
|
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
|
||||||
|
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
|
||||||
|
"variables": Kind("variables", None, "key"),
|
||||||
|
}
|
||||||
|
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
|
||||||
|
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
|
||||||
|
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
|
||||||
|
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
|
||||||
|
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
|
||||||
|
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
|
||||||
|
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
|
||||||
|
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Item:
|
||||||
|
kind: str
|
||||||
|
key: str
|
||||||
|
action: str # create | update | delete | unchanged
|
||||||
|
before: dict | None = None
|
||||||
|
after: dict | None = None
|
||||||
|
changes: list = field(default_factory=list)
|
||||||
|
reasons: list = field(default_factory=list)
|
||||||
|
deployed_hosts: int = 0
|
||||||
|
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
|
||||||
|
db_id: uuid.UUID | None = None
|
||||||
|
|
||||||
|
def public(self) -> dict:
|
||||||
|
return {
|
||||||
|
"kind": self.kind,
|
||||||
|
"key": self.key,
|
||||||
|
"action": "blocked" if self.reasons else self.action,
|
||||||
|
"planned_action": self.action,
|
||||||
|
"changes": self.changes,
|
||||||
|
"deployed_hosts": self.deployed_hosts,
|
||||||
|
"reasons": self.reasons,
|
||||||
|
"implicit": self.implicit,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Plan:
|
||||||
|
sha256: str
|
||||||
|
prune: bool
|
||||||
|
touched: list[str]
|
||||||
|
items: list[Item]
|
||||||
|
warnings: list[dict]
|
||||||
|
token: str = ""
|
||||||
|
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
|
||||||
|
expected: dict = field(default_factory=dict)
|
||||||
|
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
|
||||||
|
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
|
||||||
|
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicable(self) -> bool:
|
||||||
|
return not any(i.reasons for i in self.items)
|
||||||
|
|
||||||
|
def summary(self) -> dict:
|
||||||
|
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||||
|
for item in self.items:
|
||||||
|
out["blocked" if item.reasons else item.action] += 1
|
||||||
|
return out
|
||||||
|
|
||||||
|
def public(self) -> dict:
|
||||||
|
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
|
||||||
|
return {
|
||||||
|
"token": self.token,
|
||||||
|
"sha256": self.sha256,
|
||||||
|
"prune": self.prune,
|
||||||
|
"kinds": self.touched,
|
||||||
|
"summary": self.summary(),
|
||||||
|
"items": [i.public() for i in changed],
|
||||||
|
"warnings": self.warnings,
|
||||||
|
"applicable": self.applicable,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
|
||||||
|
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
|
||||||
|
|
||||||
|
|
||||||
|
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
|
||||||
|
section = getattr(doc, kind)
|
||||||
|
if kind == "variables":
|
||||||
|
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
|
||||||
|
for host, values in section.items() for name, value in values.items()]
|
||||||
|
return [item.model_dump(mode="python") for item in section]
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||||
|
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
|
||||||
|
def sorted_unique(values: list, where: tuple) -> list:
|
||||||
|
seen = set()
|
||||||
|
for i, value in enumerate(values):
|
||||||
|
if value in seen:
|
||||||
|
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
|
||||||
|
seen.add(value)
|
||||||
|
return sorted(seen)
|
||||||
|
|
||||||
|
item = dict(item)
|
||||||
|
if "targets" in item:
|
||||||
|
targets = dict(item["targets"])
|
||||||
|
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
|
||||||
|
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
|
||||||
|
item["targets"] = targets
|
||||||
|
if kind in ("device_groups", "policy_hierarchy_groups"):
|
||||||
|
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def _changes(before: dict, after: dict) -> list[dict]:
|
||||||
|
out = []
|
||||||
|
for name in after:
|
||||||
|
if name == "targets":
|
||||||
|
for sub in after["targets"]:
|
||||||
|
if before["targets"].get(sub) != after["targets"][sub]:
|
||||||
|
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
|
||||||
|
elif before.get(name) != after[name]:
|
||||||
|
out.append({"field": name, "before": before.get(name), "after": after[name]})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
|
||||||
|
buckets: dict = {}
|
||||||
|
for row in rows:
|
||||||
|
buckets.setdefault(bucket_of(row), []).append(row)
|
||||||
|
out = {}
|
||||||
|
for members in buckets.values():
|
||||||
|
for rank, row in enumerate(sorted(members, key=order_of)):
|
||||||
|
out[key_of(row)] = rank
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
|
||||||
|
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
|
||||||
|
found: dict = {}
|
||||||
|
if not ids:
|
||||||
|
return found
|
||||||
|
for table in Base.metadata.sorted_tables:
|
||||||
|
if table.name in skip:
|
||||||
|
continue
|
||||||
|
for fk in table.foreign_keys:
|
||||||
|
if fk.column.table.name != target_table:
|
||||||
|
continue
|
||||||
|
column = fk.parent
|
||||||
|
rows = (await db.execute(
|
||||||
|
select(column, func.count()).where(column.in_(ids)).group_by(column)
|
||||||
|
)).all()
|
||||||
|
for ref_id, _count in rows:
|
||||||
|
found.setdefault(ref_id, []).append(table.name)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
async def _load(db: AsyncSession) -> dict:
|
||||||
|
loaded: dict = {}
|
||||||
|
for kind in TARGET_KINDS:
|
||||||
|
model = KIND_SPECS[kind].model
|
||||||
|
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
|
||||||
|
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||||
|
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
|
||||||
|
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
|
||||||
|
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
|
||||||
|
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||||
|
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||||
|
return loaded
|
||||||
|
|
||||||
|
|
||||||
|
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
|
||||||
|
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
|
||||||
|
doc, lines = parsed.doc, parsed.lines
|
||||||
|
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
|
||||||
|
errors: list[dict] = []
|
||||||
|
warnings: list[dict] = []
|
||||||
|
|
||||||
|
db_canon = await canonical(db)
|
||||||
|
loaded = await _load(db)
|
||||||
|
hosts = loaded["hosts"]
|
||||||
|
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
|
||||||
|
|
||||||
|
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
|
||||||
|
file_objects: dict[str, dict[str, dict]] = {}
|
||||||
|
file_locs: dict[tuple, tuple] = {}
|
||||||
|
for kind in touched:
|
||||||
|
spec = KIND_SPECS[kind]
|
||||||
|
objects: dict[str, dict] = {}
|
||||||
|
for index, raw in enumerate(_file_items(doc, kind)):
|
||||||
|
if kind == "variables":
|
||||||
|
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
|
||||||
|
else:
|
||||||
|
ident, loc = raw[spec.ident], (kind, index)
|
||||||
|
if ident in objects:
|
||||||
|
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
|
||||||
|
continue
|
||||||
|
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
|
||||||
|
file_locs[(kind, ident)] = loc
|
||||||
|
file_objects[kind] = objects
|
||||||
|
|
||||||
|
def db_items(kind: str) -> dict[str, dict]:
|
||||||
|
if kind == "variables":
|
||||||
|
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||||
|
for h, values in db_canon["variables"].items() for n, v in values.items()}
|
||||||
|
ident = KIND_SPECS[kind].ident
|
||||||
|
return {item[ident]: item for item in db_canon[kind]}
|
||||||
|
|
||||||
|
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
|
||||||
|
|
||||||
|
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
|
||||||
|
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
|
||||||
|
post: dict[str, dict[str, dict]] = {}
|
||||||
|
for kind in schema.KINDS:
|
||||||
|
if kind in file_objects:
|
||||||
|
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
|
||||||
|
if prune:
|
||||||
|
deleted[kind] = set(remaining)
|
||||||
|
remaining = {}
|
||||||
|
post[kind] = {**remaining, **file_objects[kind]}
|
||||||
|
else:
|
||||||
|
post[kind] = dict(db_objects[kind])
|
||||||
|
|
||||||
|
host_names = set(hosts)
|
||||||
|
group_names = set(post["device_groups"])
|
||||||
|
hierarchy_names = set(post["policy_hierarchy_groups"])
|
||||||
|
|
||||||
|
# -- references in the file ------------------------------------------------------------------------------------------
|
||||||
|
for kind in touched:
|
||||||
|
for ident, item in file_objects[kind].items():
|
||||||
|
loc = file_locs[(kind, ident)]
|
||||||
|
if kind == "variables":
|
||||||
|
if item["host"] not in host_names:
|
||||||
|
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
|
||||||
|
continue
|
||||||
|
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
|
||||||
|
for name in member_hosts:
|
||||||
|
if name not in host_names:
|
||||||
|
_err(errors, "unknown_host", loc, lines, host=display(name))
|
||||||
|
for name in item.get("targets", {}).get("groups", []):
|
||||||
|
if name not in group_names:
|
||||||
|
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
|
||||||
|
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
|
||||||
|
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
|
||||||
|
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
|
||||||
|
if ref is not None and ref not in hierarchy_names:
|
||||||
|
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
|
||||||
|
if kind == "policy_hierarchy_rules":
|
||||||
|
for name, value in item.items():
|
||||||
|
if names_in(value):
|
||||||
|
_err(errors, "placeholder_not_supported", loc + (name,), lines)
|
||||||
|
spec = KIND_SPECS[kind]
|
||||||
|
if spec.api is not None:
|
||||||
|
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
|
||||||
|
for name in member_hosts:
|
||||||
|
host = hosts.get(name)
|
||||||
|
if host is not None and not host.enabled:
|
||||||
|
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
|
||||||
|
|
||||||
|
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
|
||||||
|
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
|
||||||
|
if "policy_hierarchy_groups" in file_objects:
|
||||||
|
file_groups = file_objects["policy_hierarchy_groups"]
|
||||||
|
for host_name, group in list(host_hierarchy.items()):
|
||||||
|
if prune or group in file_groups:
|
||||||
|
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
|
||||||
|
host_hierarchy[host_name] = None
|
||||||
|
listed: dict[str, str] = {}
|
||||||
|
for name, group in file_groups.items():
|
||||||
|
for host_name in group["hosts"]:
|
||||||
|
if host_name in listed:
|
||||||
|
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
|
||||||
|
lines, host=display(host_name), first=display(listed[host_name]))
|
||||||
|
listed[host_name] = name
|
||||||
|
if host_name in host_hierarchy:
|
||||||
|
host_hierarchy[host_name] = name
|
||||||
|
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
|
||||||
|
for start in parents:
|
||||||
|
seen, current = set(), start
|
||||||
|
while current is not None and current not in seen:
|
||||||
|
seen.add(current)
|
||||||
|
current = parents.get(current)
|
||||||
|
if current is not None and current == start:
|
||||||
|
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
|
||||||
|
lines, group=display(start))
|
||||||
|
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
|
||||||
|
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
|
||||||
|
|
||||||
|
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
|
||||||
|
per_host: dict[str, int] = {}
|
||||||
|
for item in post["variables"].values():
|
||||||
|
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
|
||||||
|
for host_name, count in sorted(per_host.items()):
|
||||||
|
if count > MAX_VARIABLES_PER_HOST:
|
||||||
|
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
|
||||||
|
max=MAX_VARIABLES_PER_HOST, count=count)
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
raise ConfigError("CONFIG_INVALID", errors[:100])
|
||||||
|
|
||||||
|
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
|
||||||
|
db_rule_rows = loaded["policy_hierarchy_rules"]
|
||||||
|
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
|
||||||
|
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
|
||||||
|
positions = dict(db_positions)
|
||||||
|
if "policy_hierarchy_rules" in file_objects:
|
||||||
|
buckets: dict[tuple, list[str]] = {}
|
||||||
|
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
|
||||||
|
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||||
|
for key, item in post["policy_hierarchy_rules"].items():
|
||||||
|
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
|
||||||
|
buckets.setdefault((item["group"], item["rulebase"]), [])
|
||||||
|
positions = {}
|
||||||
|
extras: dict[tuple, list[str]] = {}
|
||||||
|
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
|
||||||
|
key=lambda k: db_positions.get(k, 0)):
|
||||||
|
item = post["policy_hierarchy_rules"][key]
|
||||||
|
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||||
|
for bucket in set(buckets) | set(extras):
|
||||||
|
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
|
||||||
|
positions[key] = rank
|
||||||
|
|
||||||
|
# -- items -----------------------------------------------------------------------------------------------------------
|
||||||
|
items: list[Item] = []
|
||||||
|
for kind in touched:
|
||||||
|
rows = loaded.get(kind, {})
|
||||||
|
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
|
||||||
|
row = rows.get(ident) if kind != "variables" else None
|
||||||
|
before = db_objects[kind].get(ident)
|
||||||
|
after = post[kind].get(ident) if ident not in deleted[kind] else None
|
||||||
|
if before is None:
|
||||||
|
item = Item(kind, ident, "create", None, after)
|
||||||
|
elif after is None:
|
||||||
|
item = Item(kind, ident, "delete", before, None)
|
||||||
|
else:
|
||||||
|
changes = _changes(before, after)
|
||||||
|
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
|
||||||
|
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
|
||||||
|
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
|
||||||
|
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
|
||||||
|
if row is not None:
|
||||||
|
item.db_id = row.id
|
||||||
|
deployments = getattr(row, "deployments", None) or []
|
||||||
|
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
|
||||||
|
if item.action in ("update", "delete") and any(is_unsettled(d) for d in deployments):
|
||||||
|
item.reasons.append("busy_deployment")
|
||||||
|
if item.action == "delete" and deployments:
|
||||||
|
item.reasons.append("prune_deployed")
|
||||||
|
items.append(item)
|
||||||
|
|
||||||
|
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
|
||||||
|
|
||||||
|
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
|
||||||
|
if redeploy:
|
||||||
|
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
|
||||||
|
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
|
||||||
|
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
|
||||||
|
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
|
||||||
|
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
|
||||||
|
warnings += _alias_reference_warnings(post, file_objects, file_locs)
|
||||||
|
warnings += _variable_warnings(post, deleted)
|
||||||
|
|
||||||
|
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
|
||||||
|
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
|
||||||
|
file_objects=file_objects)
|
||||||
|
plan.expected = {kind: post[kind] for kind in touched}
|
||||||
|
plan.token = _token(plan, db_canon, db_positions, loaded)
|
||||||
|
return plan
|
||||||
|
|
||||||
|
|
||||||
|
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
|
||||||
|
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
|
||||||
|
out = set()
|
||||||
|
if kind == "policy_hierarchy_rules":
|
||||||
|
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
|
||||||
|
if kind == "policy_hierarchy_groups":
|
||||||
|
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||||
|
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
|
||||||
|
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
|
||||||
|
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
|
||||||
|
if kind == "aliases":
|
||||||
|
values["content"] = "\n".join(item["entries"])
|
||||||
|
if kind == "policy_hierarchy_rules":
|
||||||
|
values["hierarchy_group_id"] = uuid.UUID(int=0)
|
||||||
|
try:
|
||||||
|
cleaned = KIND_SPECS[kind].api.model_validate(values)
|
||||||
|
except ValidationError as exc:
|
||||||
|
for error in exc.errors(include_url=False):
|
||||||
|
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
|
||||||
|
return item
|
||||||
|
out = dict(item)
|
||||||
|
for name in item:
|
||||||
|
if name in values and name != "content" and hasattr(cleaned, name):
|
||||||
|
out[name] = getattr(cleaned, name)
|
||||||
|
if kind == "aliases":
|
||||||
|
out["entries"] = [e for e in cleaned.content.split("\n") if e]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
|
||||||
|
by_key = {(i.kind, i.key): i for i in items}
|
||||||
|
|
||||||
|
# device groups: post-state objects of the import, plus every other table that points at the group
|
||||||
|
if deleted["device_groups"]:
|
||||||
|
users: dict[str, list[str]] = {}
|
||||||
|
for kind in TARGET_KINDS:
|
||||||
|
for ident, item in post[kind].items():
|
||||||
|
for group in item["targets"]["groups"]:
|
||||||
|
users.setdefault(group, []).append(f"{kind}:{ident}")
|
||||||
|
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
|
||||||
|
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
|
||||||
|
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
|
||||||
|
users.setdefault(ids[ref_id], []).extend(tables)
|
||||||
|
for name in deleted["device_groups"]:
|
||||||
|
if users.get(name):
|
||||||
|
by_key[("device_groups", name)].reasons.append("prune_referenced")
|
||||||
|
|
||||||
|
if deleted["policy_hierarchy_groups"]:
|
||||||
|
users = {}
|
||||||
|
for kind in HIERARCHY_TARGET_KINDS:
|
||||||
|
for ident, item in post[kind].items():
|
||||||
|
if item["targets"]["hierarchy_group"]:
|
||||||
|
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
|
||||||
|
for ident, item in post["policy_hierarchy_rules"].items():
|
||||||
|
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
|
||||||
|
for name, item in post["policy_hierarchy_groups"].items():
|
||||||
|
if item["parent"]:
|
||||||
|
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
|
||||||
|
for host_name, group in host_hierarchy.items():
|
||||||
|
if group:
|
||||||
|
users.setdefault(group, []).append(f"host:{host_name}")
|
||||||
|
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
|
||||||
|
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
|
||||||
|
"policy_hierarchy_rules", "opnsense_hosts"}
|
||||||
|
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
|
||||||
|
users.setdefault(ids[ref_id], []).extend(tables)
|
||||||
|
for name in deleted["policy_hierarchy_groups"]:
|
||||||
|
if users.get(name):
|
||||||
|
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
|
||||||
|
|
||||||
|
if deleted["aliases"]:
|
||||||
|
used: set[str] = set()
|
||||||
|
for kind in RULE_KINDS:
|
||||||
|
for item in post[kind].values():
|
||||||
|
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
|
||||||
|
for item in post["aliases"].values():
|
||||||
|
used.update(item["entries"])
|
||||||
|
for name in deleted["aliases"]:
|
||||||
|
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
|
||||||
|
by_key[("aliases", name)].reasons.append("prune_referenced")
|
||||||
|
|
||||||
|
|
||||||
|
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
|
||||||
|
aliases = set(post["aliases"])
|
||||||
|
out = []
|
||||||
|
for kind in RULE_KINDS:
|
||||||
|
for ident, item in file_objects.get(kind, {}).items():
|
||||||
|
for name in ALIAS_REF_FIELDS:
|
||||||
|
value = item.get(name)
|
||||||
|
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
|
||||||
|
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
|
||||||
|
"params": {"value": display(value)}})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
|
||||||
|
if not deleted["variables"]:
|
||||||
|
return []
|
||||||
|
used: set[str] = set()
|
||||||
|
for kind in RULE_KINDS + ("aliases",):
|
||||||
|
for item in post[kind].values():
|
||||||
|
for value in item.values():
|
||||||
|
if isinstance(value, str):
|
||||||
|
used |= names_in(value)
|
||||||
|
elif isinstance(value, list):
|
||||||
|
for entry in value:
|
||||||
|
used |= names_in(entry)
|
||||||
|
out = []
|
||||||
|
for key in sorted(deleted["variables"]):
|
||||||
|
host, name = key.split("/", 1)
|
||||||
|
if name in used:
|
||||||
|
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
|
||||||
|
deployments = {
|
||||||
|
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
|
||||||
|
for ident, row in loaded[kind].items()}
|
||||||
|
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
|
||||||
|
}
|
||||||
|
material = {
|
||||||
|
"file": plan.file_objects,
|
||||||
|
"prune": plan.prune,
|
||||||
|
"db": db_canon,
|
||||||
|
"positions": db_positions,
|
||||||
|
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
|
||||||
|
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
|
||||||
|
"deployments": deployments,
|
||||||
|
"items": [i.public() for i in plan.items],
|
||||||
|
"warnings": plan.warnings,
|
||||||
|
}
|
||||||
|
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
|
||||||
@@ -0,0 +1,242 @@
|
|||||||
|
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
|
||||||
|
|
||||||
|
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
|
||||||
|
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
|
||||||
|
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
|
||||||
|
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from typing import Annotated, Literal
|
||||||
|
|
||||||
|
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
|
||||||
|
|
||||||
|
from app.models.config_key import CONFIG_KEY_RE
|
||||||
|
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
|
||||||
|
|
||||||
|
SCHEMA_VERSION = 1
|
||||||
|
MAX_SCALAR = 8000
|
||||||
|
|
||||||
|
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||||
|
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(value: str) -> str:
|
||||||
|
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
|
||||||
|
if _CONTROL.search(value):
|
||||||
|
raise ValueError("control_character")
|
||||||
|
if value != value.strip():
|
||||||
|
raise ValueError("surrounding_whitespace")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_value(value: str) -> str:
|
||||||
|
if _CONTROL.search(value):
|
||||||
|
raise ValueError("control_character")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_text(value: str) -> str:
|
||||||
|
if _CONTROL_TEXT.search(value):
|
||||||
|
raise ValueError("control_character")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _port(value):
|
||||||
|
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
|
||||||
|
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
|
||||||
|
|
||||||
|
|
||||||
|
Text = Annotated[str, AfterValidator(_clean)]
|
||||||
|
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
|
||||||
|
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
|
||||||
|
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
|
||||||
|
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
|
||||||
|
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
|
||||||
|
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
|
||||||
|
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
|
||||||
|
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
|
||||||
|
Port = Annotated[PortText | None, BeforeValidator(_port)]
|
||||||
|
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||||
|
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||||
|
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
|
||||||
|
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
|
||||||
|
|
||||||
|
IpProtocol = Literal["inet", "inet6"]
|
||||||
|
|
||||||
|
|
||||||
|
class _Doc(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid", strict=True)
|
||||||
|
|
||||||
|
|
||||||
|
class Targets(_Doc):
|
||||||
|
hosts: list[HostName] = Field(default_factory=list)
|
||||||
|
groups: list[GroupName] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class TargetsWithHierarchy(Targets):
|
||||||
|
hierarchy_group: GroupName | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class DeviceGroupDoc(_Doc):
|
||||||
|
name: GroupName
|
||||||
|
description: GroupDescription = ""
|
||||||
|
hosts: list[HostName] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class HierarchyGroupDoc(_Doc):
|
||||||
|
name: GroupName
|
||||||
|
parent: GroupName | None = None
|
||||||
|
description: GroupDescription = ""
|
||||||
|
hosts: list[HostName] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class AliasDoc(_Doc):
|
||||||
|
name: AliasName
|
||||||
|
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
|
||||||
|
description: Description = ""
|
||||||
|
enabled: bool = True
|
||||||
|
entries: list[AliasEntry] = Field(default_factory=list)
|
||||||
|
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||||
|
|
||||||
|
|
||||||
|
class RuleDoc(_Doc):
|
||||||
|
key: Key
|
||||||
|
description: RuleDescription
|
||||||
|
interface: Interface
|
||||||
|
direction: Literal["in", "out"] = "in"
|
||||||
|
ipprotocol: IpProtocol = "inet"
|
||||||
|
protocol: Protocol = "any"
|
||||||
|
action: Literal["pass", "block", "reject"] = "pass"
|
||||||
|
source_net: Net = "any"
|
||||||
|
source_port: Port = None
|
||||||
|
destination_net: Net = "any"
|
||||||
|
destination_port: Port = None
|
||||||
|
enabled: bool = True
|
||||||
|
log: bool = False
|
||||||
|
targets: Targets = Field(default_factory=Targets)
|
||||||
|
|
||||||
|
|
||||||
|
class PortForwardDoc(_Doc):
|
||||||
|
key: Key
|
||||||
|
description: RuleDescription
|
||||||
|
interface: Interface
|
||||||
|
ipprotocol: IpProtocol = "inet"
|
||||||
|
protocol: Protocol = "tcp"
|
||||||
|
source_net: Net = "any"
|
||||||
|
source_port: Port = None
|
||||||
|
destination_net: Net = "any"
|
||||||
|
destination_port: Port = None
|
||||||
|
target: Net
|
||||||
|
local_port: Port = None
|
||||||
|
natreflection: Literal["", "purenat", "disable"] = ""
|
||||||
|
enabled: bool = True
|
||||||
|
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||||
|
|
||||||
|
|
||||||
|
class OneToOneDoc(_Doc):
|
||||||
|
key: Key
|
||||||
|
description: RuleDescription
|
||||||
|
interface: Interface
|
||||||
|
type: Literal["binat", "nat"] = "binat"
|
||||||
|
external: Net
|
||||||
|
source_net: Net = "any"
|
||||||
|
destination_net: Net = "any"
|
||||||
|
source_not: bool = False
|
||||||
|
destination_not: bool = False
|
||||||
|
natreflection: Literal["", "enable", "disable"] = ""
|
||||||
|
log: bool = False
|
||||||
|
enabled: bool = True
|
||||||
|
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||||
|
|
||||||
|
|
||||||
|
class OutboundNatDoc(_Doc):
|
||||||
|
key: Key
|
||||||
|
description: RuleDescription
|
||||||
|
interface: Interface
|
||||||
|
ipprotocol: IpProtocol = "inet"
|
||||||
|
protocol: Protocol = "any"
|
||||||
|
source_net: Net = "any"
|
||||||
|
source_port: Port = None
|
||||||
|
destination_net: Net = "any"
|
||||||
|
destination_port: Port = None
|
||||||
|
source_not: bool = False
|
||||||
|
destination_not: bool = False
|
||||||
|
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
|
||||||
|
target_port: Port = None
|
||||||
|
enabled: bool = True
|
||||||
|
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||||
|
|
||||||
|
|
||||||
|
class HierarchyRuleDoc(_Doc):
|
||||||
|
key: Key
|
||||||
|
group: GroupName
|
||||||
|
rulebase: Literal["pre", "post"] = "pre"
|
||||||
|
description: RuleDescription
|
||||||
|
interface: Interface
|
||||||
|
direction: Literal["in", "out"] = "in"
|
||||||
|
ipprotocol: IpProtocol = "inet"
|
||||||
|
protocol: Protocol = "any"
|
||||||
|
action: Literal["pass", "block", "reject"] = "pass"
|
||||||
|
source_net: Net = "any"
|
||||||
|
source_port: Port = None
|
||||||
|
destination_net: Net = "any"
|
||||||
|
destination_port: Port = None
|
||||||
|
enabled: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
def _variable_name(value: str) -> str:
|
||||||
|
if not VARIABLE_NAME_RE.match(value):
|
||||||
|
raise ValueError("bad_variable_name")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _variable_value(value: str) -> str:
|
||||||
|
if not VARIABLE_VALUE_RE.match(value):
|
||||||
|
raise ValueError("bad_variable_value")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
VariableName = Annotated[str, AfterValidator(_variable_name)]
|
||||||
|
VariableValue = Annotated[str, AfterValidator(_variable_value)]
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigDoc(_Doc):
|
||||||
|
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
|
||||||
|
|
||||||
|
opnexus_config: Literal[1]
|
||||||
|
device_groups: list[DeviceGroupDoc] | None = None
|
||||||
|
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
|
||||||
|
aliases: list[AliasDoc] | None = None
|
||||||
|
rules: list[RuleDoc] | None = None
|
||||||
|
port_forwards: list[PortForwardDoc] | None = None
|
||||||
|
one_to_one: list[OneToOneDoc] | None = None
|
||||||
|
outbound_nat: list[OutboundNatDoc] | None = None
|
||||||
|
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
|
||||||
|
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
|
||||||
|
KINDS = (
|
||||||
|
"device_groups",
|
||||||
|
"policy_hierarchy_groups",
|
||||||
|
"aliases",
|
||||||
|
"rules",
|
||||||
|
"port_forwards",
|
||||||
|
"one_to_one",
|
||||||
|
"outbound_nat",
|
||||||
|
"policy_hierarchy_rules",
|
||||||
|
"variables",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
SCHEMA_FILE = "opnexus-config-v1.schema.json"
|
||||||
|
|
||||||
|
|
||||||
|
def json_schema_text() -> str:
|
||||||
|
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
schema = ConfigDoc.model_json_schema()
|
||||||
|
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
|
||||||
|
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
|
||||||
|
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
|
||||||
@@ -62,6 +62,7 @@ class ErrorCode(str, Enum):
|
|||||||
GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET"
|
GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET"
|
||||||
GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET"
|
GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET"
|
||||||
GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET"
|
GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET"
|
||||||
|
GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET"
|
||||||
HOSTNAMEN_EXAKT_EINGEBEN = "HOSTNAMEN_EXAKT_EINGEBEN"
|
HOSTNAMEN_EXAKT_EINGEBEN = "HOSTNAMEN_EXAKT_EINGEBEN"
|
||||||
HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN = "HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN"
|
HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN = "HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN"
|
||||||
HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0 = "HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0"
|
HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0 = "HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0"
|
||||||
@@ -72,6 +73,8 @@ class ErrorCode(str, Enum):
|
|||||||
HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN = "HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN"
|
HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN = "HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN"
|
||||||
HOST_IST_DEAKTIVIERT = "HOST_IST_DEAKTIVIERT"
|
HOST_IST_DEAKTIVIERT = "HOST_IST_DEAKTIVIERT"
|
||||||
HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS = "HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS"
|
HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS = "HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS"
|
||||||
|
HOST_URL_EXISTS = "HOST_URL_EXISTS"
|
||||||
|
HOST_CONNECTION_FAILED = "HOST_CONNECTION_FAILED"
|
||||||
HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B = "HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B"
|
HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B = "HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B"
|
||||||
HOST_NICHT_GEFUNDEN = "HOST_NICHT_GEFUNDEN"
|
HOST_NICHT_GEFUNDEN = "HOST_NICHT_GEFUNDEN"
|
||||||
HOST_REQUEST_FAILED = "HOST_REQUEST_FAILED"
|
HOST_REQUEST_FAILED = "HOST_REQUEST_FAILED"
|
||||||
@@ -94,6 +97,8 @@ class ErrorCode(str, Enum):
|
|||||||
GUI_ACCESS_DISABLED = "GUI_ACCESS_DISABLED"
|
GUI_ACCESS_DISABLED = "GUI_ACCESS_DISABLED"
|
||||||
GUI_HOST_NOT_ELIGIBLE = "GUI_HOST_NOT_ELIGIBLE"
|
GUI_HOST_NOT_ELIGIBLE = "GUI_HOST_NOT_ELIGIBLE"
|
||||||
GUI_NO_FREE_SLOT = "GUI_NO_FREE_SLOT"
|
GUI_NO_FREE_SLOT = "GUI_NO_FREE_SLOT"
|
||||||
|
GUI_SLOT_COOLING_DOWN = "GUI_SLOT_COOLING_DOWN"
|
||||||
|
GUI_TOO_MANY_SESSIONS = "GUI_TOO_MANY_SESSIONS"
|
||||||
TLS_PROBE_FAILED = "TLS_PROBE_FAILED"
|
TLS_PROBE_FAILED = "TLS_PROBE_FAILED"
|
||||||
TLS_PEM_INVALID = "TLS_PEM_INVALID"
|
TLS_PEM_INVALID = "TLS_PEM_INVALID"
|
||||||
TLS_PIN_MISMATCH = "TLS_PIN_MISMATCH"
|
TLS_PIN_MISMATCH = "TLS_PIN_MISMATCH"
|
||||||
@@ -145,6 +150,10 @@ class ErrorCode(str, Enum):
|
|||||||
UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN = "UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN"
|
UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN = "UEBERGEORDNETE_GRUPPE_NICHT_GEFUNDEN"
|
||||||
UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT = "UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT"
|
UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT = "UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT"
|
||||||
UPSTREAM_REQUEST_FAILED = "UPSTREAM_REQUEST_FAILED"
|
UPSTREAM_REQUEST_FAILED = "UPSTREAM_REQUEST_FAILED"
|
||||||
|
API_RIGHTS_MISSING = "API_RIGHTS_MISSING"
|
||||||
|
GATEWAY_NAME_INVALID = "GATEWAY_NAME_INVALID"
|
||||||
|
GATEWAY_UNKNOWN = "GATEWAY_UNKNOWN"
|
||||||
|
GATEWAY_MUTE_LIMIT = "GATEWAY_MUTE_LIMIT"
|
||||||
VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN = "VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN"
|
VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN = "VERKNUEPFTER_ACME_ACCOUNT_NICHT_GEFUNDEN"
|
||||||
VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN = "VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN"
|
VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN = "VORGANG_KANN_NICHT_MEHR_ABGEBROCHEN_WERDEN"
|
||||||
VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN = "VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN"
|
VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN = "VORSCHAU_ABGELAUFEN_BITTE_ERNEUT_OEFFNEN"
|
||||||
@@ -165,6 +174,7 @@ class ErrorCode(str, Enum):
|
|||||||
OPEN_DEPLOYMENT_DRAFT_UNCHANGED = "OPEN_DEPLOYMENT_DRAFT_UNCHANGED"
|
OPEN_DEPLOYMENT_DRAFT_UNCHANGED = "OPEN_DEPLOYMENT_DRAFT_UNCHANGED"
|
||||||
OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY = "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY = "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
||||||
OPEN_RESTORE_BLOCKS_DEPLOY = "OPEN_RESTORE_BLOCKS_DEPLOY"
|
OPEN_RESTORE_BLOCKS_DEPLOY = "OPEN_RESTORE_BLOCKS_DEPLOY"
|
||||||
|
OPEN_CLEANUP_BLOCKS_DEPLOY = "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||||
UPSTREAM_REJECTED = "UPSTREAM_REJECTED"
|
UPSTREAM_REJECTED = "UPSTREAM_REJECTED"
|
||||||
INVALID_CHARACTERS = "INVALID_CHARACTERS"
|
INVALID_CHARACTERS = "INVALID_CHARACTERS"
|
||||||
SERVICE_NOT_FOUND = "SERVICE_NOT_FOUND"
|
SERVICE_NOT_FOUND = "SERVICE_NOT_FOUND"
|
||||||
@@ -216,6 +226,40 @@ class ErrorCode(str, Enum):
|
|||||||
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
|
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
|
||||||
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
|
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
|
||||||
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
|
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
|
||||||
|
# Rule cleanup (1.31, services/rule_cleanup_service.py)
|
||||||
|
BEREINIGUNG_NICHT_GEFUNDEN = "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||||
|
BEREINIGUNG_AUSWAHL_UNGUELTIG = "BEREINIGUNG_AUSWAHL_UNGUELTIG"
|
||||||
|
BEREINIGUNG_VORSCHAU_VERALTET = "BEREINIGUNG_VORSCHAU_VERALTET"
|
||||||
|
BEREINIGUNG_REGEL_NICHT_AENDERBAR = "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
|
||||||
|
BEREINIGUNG_OFFENE_OPERATION = "BEREINIGUNG_OFFENE_OPERATION"
|
||||||
|
BEREINIGUNG_ZU_VIELE_REGELN = "BEREINIGUNG_ZU_VIELE_REGELN"
|
||||||
|
BEREINIGUNG_BESTAETIGUNG_FEHLT = "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||||
|
BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN = "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||||
|
BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG = "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG"
|
||||||
|
BEREINIGUNG_FRIST_ABGELAUFEN = "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||||
|
BEREINIGUNG_NICHT_BESTAETIGBAR = "BEREINIGUNG_NICHT_BESTAETIGBAR"
|
||||||
|
BEREINIGUNG_LIVESTAND_ABWEICHEND = "BEREINIGUNG_LIVESTAND_ABWEICHEND"
|
||||||
|
BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR = "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
|
||||||
|
BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT = "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT"
|
||||||
|
BEREINIGUNG_SNAPSHOT_FEHLT = "BEREINIGUNG_SNAPSHOT_FEHLT"
|
||||||
|
BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR = "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
|
||||||
|
BEREINIGUNG_AKTION_NICHT_MOEGLICH = "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||||
|
BEREINIGUNG_NICHT_AUSGEFUEHRT = "BEREINIGUNG_NICHT_AUSGEFUEHRT"
|
||||||
|
BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN = "BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN"
|
||||||
|
BEREINIGUNG_AUSGANG_UNKLAR = "BEREINIGUNG_AUSGANG_UNKLAR"
|
||||||
|
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
|
||||||
|
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
|
||||||
|
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
|
||||||
|
CONFIG_INVALID = "CONFIG_INVALID"
|
||||||
|
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
|
||||||
|
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
|
||||||
|
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
|
||||||
|
CONFIG_BLOCKED = "CONFIG_BLOCKED"
|
||||||
|
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
|
||||||
|
CONFIG_BUSY = "CONFIG_BUSY"
|
||||||
|
ZIEL_HOST_NACH_AUSROLLEN_FEST = "ZIEL_HOST_NACH_AUSROLLEN_FEST"
|
||||||
|
ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET = "ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET"
|
||||||
|
VPN_DEPLOY_LAEUFT_BEREITS = "VPN_DEPLOY_LAEUFT_BEREITS"
|
||||||
|
|
||||||
|
|
||||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||||
|
|||||||
@@ -20,7 +20,10 @@ Security model (see the 1.27 design):
|
|||||||
* At most 3 failed GUI logins per firewall in 5 minutes (protects OPNexus' API access from the firewall's
|
* At most 3 failed GUI logins per firewall in 5 minutes (protects OPNexus' API access from the firewall's
|
||||||
login lockout, which would see all attempts coming from OPNexus).
|
login lockout, which would see all attempts coming from OPNexus).
|
||||||
* Idle and absolute limits; every 20 s the backend re-checks the session (OPNexus session, admin role,
|
* Idle and absolute limits; every 20 s the backend re-checks the session (OPNexus session, admin role,
|
||||||
switch, host). Logs carry method, path (no query), status and duration only.
|
switch, host, and that the firewall's address, pinned certificate and overlay route are still the ones the
|
||||||
|
session started with).
|
||||||
|
* At most MAX_SESSIONS sessions at once: expired ones are cleared first, a new session is refused otherwise —
|
||||||
|
a running session is never evicted by a newer one. Logs carry method, path (no query), status and duration only.
|
||||||
|
|
||||||
OPNsense behaviour this relies on (opnsense/core, src/www/authgui.inc, master as of 2026-10-21):
|
OPNsense behaviour this relies on (opnsense/core, src/www/authgui.inc, master as of 2026-10-21):
|
||||||
* session_auth(): a login is ``isset($_POST['login']) && !empty($_POST['usernamefld']) &&
|
* session_auth(): a login is ``isset($_POST['login']) && !empty($_POST['usernamefld']) &&
|
||||||
@@ -131,6 +134,7 @@ class GatewaySession:
|
|||||||
idle_timeout: float
|
idle_timeout: float
|
||||||
absolute_deadline: float
|
absolute_deadline: float
|
||||||
revalidate_every: float
|
revalidate_every: float
|
||||||
|
target_fp: str | None = None
|
||||||
last_activity: float = field(default_factory=time.monotonic)
|
last_activity: float = field(default_factory=time.monotonic)
|
||||||
next_check: float = 0.0
|
next_check: float = 0.0
|
||||||
jar: dict[str, str] = field(default_factory=dict)
|
jar: dict[str, str] = field(default_factory=dict)
|
||||||
@@ -156,7 +160,8 @@ class BackendClient:
|
|||||||
|
|
||||||
async def revalidate(self, gs: GatewaySession) -> dict:
|
async def revalidate(self, gs: GatewaySession) -> dict:
|
||||||
resp = await self._client.post("/internal/gui/revalidate",
|
resp = await self._client.post("/internal/gui/revalidate",
|
||||||
json={"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash})
|
json={"gui_session_id": gs.gui_session_id, "session_hash": gs.session_hash,
|
||||||
|
"target_fp": gs.target_fp})
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
return resp.json()
|
return resp.json()
|
||||||
|
|
||||||
@@ -250,6 +255,8 @@ _LOGIN_LOCKED = _page("OPNexus", "Zu viele fehlgeschlagene Anmeldungen an dieser
|
|||||||
"Too many failed sign-ins at this firewall. Please wait 5 minutes.")
|
"Too many failed sign-ins at this firewall. Please wait 5 minutes.")
|
||||||
_UNAVAILABLE = _page("OPNexus", "Die Firewall oder OPNexus ist gerade nicht erreichbar.",
|
_UNAVAILABLE = _page("OPNexus", "Die Firewall oder OPNexus ist gerade nicht erreichbar.",
|
||||||
"The firewall or OPNexus is not reachable right now.")
|
"The firewall or OPNexus is not reachable right now.")
|
||||||
|
_TOO_MANY = _page("OPNexus", "Es laufen schon zu viele Firewall-Oberflächen gleichzeitig. Versuche es später erneut.",
|
||||||
|
"Too many firewall interfaces are open at the same time. Try again later.")
|
||||||
_REDEEMED = (b"<!doctype html><html><head><meta charset=\"utf-8\"><meta http-equiv=\"refresh\" content=\"0;url=/\">"
|
_REDEEMED = (b"<!doctype html><html><head><meta charset=\"utf-8\"><meta http-equiv=\"refresh\" content=\"0;url=/\">"
|
||||||
b"<title>OPNexus</title></head><body><a href=\"/\">→</a></body></html>")
|
b"<title>OPNexus</title></head><body><a href=\"/\">→</a></body></html>")
|
||||||
|
|
||||||
@@ -376,6 +383,17 @@ class GuiGateway:
|
|||||||
gs.next_check = self.clock() + gs.revalidate_every
|
gs.next_check = self.clock() + gs.revalidate_every
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
async def _clear_expired(self) -> None:
|
||||||
|
for key, gs in list(self.sessions.items()):
|
||||||
|
reason = self._expired(gs)
|
||||||
|
if reason:
|
||||||
|
await self._drop(key, reason)
|
||||||
|
|
||||||
|
def _has_room(self, browser_hash: str, slot: int) -> bool:
|
||||||
|
"""Room for one more session; the one this redeem would replace (same browser session and slot) does not count."""
|
||||||
|
replaced = sum(1 for g in self.sessions.values() if g.slot == slot and g.session_hash == browser_hash)
|
||||||
|
return len(self.sessions) - replaced < MAX_SESSIONS
|
||||||
|
|
||||||
# ── request handling ──────────────────────────────────────────────────
|
# ── request handling ──────────────────────────────────────────────────
|
||||||
async def _handle(self, scope, receive, send) -> int:
|
async def _handle(self, scope, receive, send) -> int:
|
||||||
method = scope.get("method", "")
|
method = scope.get("method", "")
|
||||||
@@ -442,6 +460,10 @@ class GuiGateway:
|
|||||||
if not ticket or len(ticket) > 128 or not browser_session:
|
if not ticket or len(ticket) > 128 or not browser_session:
|
||||||
return await self._respond(send, 403, _SESSION_GONE)
|
return await self._respond(send, 403, _SESSION_GONE)
|
||||||
client_ip = (headers.get("x-forwarded-for", "").split(",")[0].strip() or None)
|
client_ip = (headers.get("x-forwarded-for", "").split(",")[0].strip() or None)
|
||||||
|
# Room first, before the ticket is burned: expired sessions go, running ones are never evicted.
|
||||||
|
await self._clear_expired()
|
||||||
|
if not self._has_room(_sha(browser_session), slot):
|
||||||
|
return await self._respond(send, 503, _TOO_MANY)
|
||||||
try:
|
try:
|
||||||
result = await self.backend.redeem(_sha(ticket), _sha(browser_session), slot, client_ip and client_ip[:45])
|
result = await self.backend.redeem(_sha(ticket), _sha(browser_session), slot, client_ip and client_ip[:45])
|
||||||
except httpx.HTTPError as exc:
|
except httpx.HTTPError as exc:
|
||||||
@@ -453,16 +475,16 @@ class GuiGateway:
|
|||||||
for old_key, old in list(self.sessions.items()):
|
for old_key, old in list(self.sessions.items()):
|
||||||
if old.slot == slot and old.session_hash == _sha(browser_session):
|
if old.slot == slot and old.session_hash == _sha(browser_session):
|
||||||
await self._drop(old_key, "superseded", report=False)
|
await self._drop(old_key, "superseded", report=False)
|
||||||
if len(self.sessions) >= MAX_SESSIONS:
|
|
||||||
oldest = min(self.sessions, key=lambda k: self.sessions[k].last_activity)
|
|
||||||
await self._drop(oldest, "superseded")
|
|
||||||
value = secrets.token_urlsafe(32)
|
value = secrets.token_urlsafe(32)
|
||||||
gs = GatewaySession(
|
gs = GatewaySession(
|
||||||
gui_session_id=result["gui_session_id"], session_hash=_sha(browser_session), slot=slot,
|
gui_session_id=result["gui_session_id"], session_hash=_sha(browser_session), slot=slot,
|
||||||
upstream_origin=result["upstream_origin"], ca_pem=result.get("ca_pem"), proxy_url=result.get("proxy_url"),
|
upstream_origin=result["upstream_origin"], ca_pem=result.get("ca_pem"), proxy_url=result.get("proxy_url"),
|
||||||
idle_timeout=float(result["idle_timeout_seconds"]), absolute_deadline=float(result["absolute_deadline"]),
|
idle_timeout=float(result["idle_timeout_seconds"]), absolute_deadline=float(result["absolute_deadline"]),
|
||||||
revalidate_every=float(result.get("revalidate_seconds", 20)),
|
revalidate_every=float(result.get("revalidate_seconds", 20)), target_fp=result.get("target_fp"),
|
||||||
)
|
)
|
||||||
|
if len(self.sessions) >= MAX_SESSIONS: # filled up while the backend was answering
|
||||||
|
await self.backend.end(gs, "revoked")
|
||||||
|
return await self._respond(send, 503, _TOO_MANY)
|
||||||
gs.last_activity = self.clock()
|
gs.last_activity = self.clock()
|
||||||
gs.next_check = gs.last_activity + gs.revalidate_every
|
gs.next_check = gs.last_activity + gs.revalidate_every
|
||||||
self.sessions[_sha(value)] = gs
|
self.sessions[_sha(value)] = gs
|
||||||
|
|||||||
+38
-1
@@ -53,6 +53,8 @@ from app.routers import scheduled_maintenance as scheduled_maintenance_router
|
|||||||
from app.routers import maintenance_windows as maintenance_windows_router
|
from app.routers import maintenance_windows as maintenance_windows_router
|
||||||
from app.routers import rollouts as rollouts_router
|
from app.routers import rollouts as rollouts_router
|
||||||
from app.routers import rule_analysis as rule_analysis_router
|
from app.routers import rule_analysis as rule_analysis_router
|
||||||
|
from app.routers import rule_cleanup as rule_cleanup_router
|
||||||
|
from app.routers import config_code as config_code_router
|
||||||
from app.routers import gui_access as gui_access_router
|
from app.routers import gui_access as gui_access_router
|
||||||
from app.services import scheduled_maintenance_service
|
from app.services import scheduled_maintenance_service
|
||||||
from app.services import rollout_engine
|
from app.services import rollout_engine
|
||||||
@@ -61,6 +63,8 @@ from app.services import audit_export_service
|
|||||||
from app.services import overlay_service, variables_service
|
from app.services import overlay_service, variables_service
|
||||||
from app.services import backup_schedule_service
|
from app.services import backup_schedule_service
|
||||||
from app.services import carp_event_service, carp_maintenance_window_service
|
from app.services import carp_event_service, carp_maintenance_window_service
|
||||||
|
from app.services import rule_history_service
|
||||||
|
from app.services import rule_cleanup_service
|
||||||
|
|
||||||
# Without this, Python's root logger defaults to WARNING with no handler,
|
# Without this, Python's root logger defaults to WARNING with no handler,
|
||||||
# so every logger.info() call in the 5 background loops below (CARP role
|
# so every logger.info() call in the 5 background loops below (CARP role
|
||||||
@@ -141,6 +145,13 @@ async def _deploy_revert_sweep_loop():
|
|||||||
logger.info("Auto-reverted %d unconfirmed policy-hierarchy deployment(s)", len(reverted))
|
logger.info("Auto-reverted %d unconfirmed policy-hierarchy deployment(s)", len(reverted))
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Policy-hierarchy deploy-revert sweep failed")
|
logger.exception("Policy-hierarchy deploy-revert sweep failed")
|
||||||
|
try:
|
||||||
|
async with SessionLocal() as db:
|
||||||
|
reverted = await rule_cleanup_service.sweep_expired_cleanups(db)
|
||||||
|
if reverted:
|
||||||
|
logger.info("Regel-Bereinigung: %d Operation(en) automatisch zurückgenommen/abgeschlossen", len(reverted))
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Rule-cleanup revert sweep failed")
|
||||||
await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
|
await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
@@ -234,6 +245,9 @@ async def _resource_poll_loop():
|
|||||||
deleted = await resource_monitor_service.prune_old_samples(db, settings.resource_sample_retention_days)
|
deleted = await resource_monitor_service.prune_old_samples(db, settings.resource_sample_retention_days)
|
||||||
if deleted:
|
if deleted:
|
||||||
logger.info("Resource-Sample-Retention: %d alte Einträge gelöscht", deleted)
|
logger.info("Resource-Sample-Retention: %d alte Einträge gelöscht", deleted)
|
||||||
|
pruned = await rule_history_service.prune(db)
|
||||||
|
if pruned:
|
||||||
|
logger.info("Regelverlauf-Retention: %d alte Einträge gelöscht", pruned)
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Resource-Poll fehlgeschlagen")
|
logger.exception("Resource-Poll fehlgeschlagen")
|
||||||
await asyncio.sleep(settings.resource_poll_interval_seconds)
|
await asyncio.sleep(settings.resource_poll_interval_seconds)
|
||||||
@@ -322,6 +336,25 @@ async def _backup_poll_loop():
|
|||||||
await asyncio.sleep(_BACKUP_POLL_SECONDS)
|
await asyncio.sleep(_BACKUP_POLL_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
_RULE_OBSERVE_FIRST_RUN_SECONDS = 150
|
||||||
|
|
||||||
|
|
||||||
|
async def _rule_observe_loop():
|
||||||
|
"""Ninth background loop: reads the rules and their counters of every enabled firewall (two GETs each),
|
||||||
|
keeps the usage sampling current and records what changed since the last look (services/rule_history_service.py).
|
||||||
|
Writes only to the database. The first look at a firewall only stores the starting point."""
|
||||||
|
await asyncio.sleep(_RULE_OBSERVE_FIRST_RUN_SECONDS)
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
async with SessionLocal() as db:
|
||||||
|
result = await rule_history_service.observe_all(db)
|
||||||
|
if result["events"] or result["failed"]:
|
||||||
|
logger.info("Regelverlauf: %s", result)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Regelverlauf fehlgeschlagen")
|
||||||
|
await asyncio.sleep(max(settings.rule_observe_interval_seconds, settings.rule_observe_min_interval_seconds))
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def lifespan(app: FastAPI):
|
async def lifespan(app: FastAPI):
|
||||||
conflicts = settings.gui_origin_conflicts()
|
conflicts = settings.gui_origin_conflicts()
|
||||||
@@ -351,6 +384,7 @@ async def lifespan(app: FastAPI):
|
|||||||
compliance_task = asyncio.create_task(_compliance_poll_loop())
|
compliance_task = asyncio.create_task(_compliance_poll_loop())
|
||||||
audit_export_task = asyncio.create_task(_audit_export_poll_loop())
|
audit_export_task = asyncio.create_task(_audit_export_poll_loop())
|
||||||
backup_task = asyncio.create_task(_backup_poll_loop())
|
backup_task = asyncio.create_task(_backup_poll_loop())
|
||||||
|
rule_observe_task = asyncio.create_task(_rule_observe_loop())
|
||||||
try:
|
try:
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
@@ -362,7 +396,8 @@ async def lifespan(app: FastAPI):
|
|||||||
compliance_task.cancel()
|
compliance_task.cancel()
|
||||||
audit_export_task.cancel()
|
audit_export_task.cancel()
|
||||||
backup_task.cancel()
|
backup_task.cancel()
|
||||||
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task):
|
rule_observe_task.cancel()
|
||||||
|
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task, rule_observe_task):
|
||||||
try:
|
try:
|
||||||
await task
|
await task
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
@@ -536,5 +571,7 @@ app.include_router(scheduled_maintenance_router.router, prefix="/api")
|
|||||||
app.include_router(maintenance_windows_router.router, prefix="/api")
|
app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||||
app.include_router(rollouts_router.router, prefix="/api")
|
app.include_router(rollouts_router.router, prefix="/api")
|
||||||
app.include_router(rule_analysis_router.router, prefix="/api")
|
app.include_router(rule_analysis_router.router, prefix="/api")
|
||||||
|
app.include_router(rule_cleanup_router.router, prefix="/api")
|
||||||
|
app.include_router(config_code_router.router, prefix="/api")
|
||||||
app.include_router(gui_access_router.router, prefix="/api")
|
app.include_router(gui_access_router.router, prefix="/api")
|
||||||
app.include_router(gui_access_router.internal_router)
|
app.include_router(gui_access_router.internal_router)
|
||||||
@@ -102,3 +102,14 @@ __all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "R
|
|||||||
|
|
||||||
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
from app.models.gui_access import GuiAccessSettings, GuiSession, GuiTicket
|
||||||
__all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
|
__all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
|
||||||
|
|
||||||
|
from app.models.rule_history import LiveRuleState, RuleChangeEvent
|
||||||
|
__all__ += ["LiveRuleState", "RuleChangeEvent"]
|
||||||
|
|
||||||
|
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
|
||||||
|
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
|
||||||
|
|
||||||
|
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
|
||||||
|
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
|
||||||
|
from app.models.host_variable import HostVariable
|
||||||
|
__all__ += ["HostVariable"]
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
|
||||||
|
|
||||||
|
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
|
||||||
|
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
|
||||||
|
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
|
||||||
|
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from sqlalchemy import String, event
|
||||||
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
|
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
|
||||||
|
|
||||||
|
|
||||||
|
def config_key_column() -> Mapped[str]:
|
||||||
|
return mapped_column(String(64), unique=True)
|
||||||
|
|
||||||
|
|
||||||
|
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
|
||||||
|
return f"{prefix}-{object_id.hex[:12]}"
|
||||||
|
|
||||||
|
|
||||||
|
def register_config_key(model: type, prefix: str) -> None:
|
||||||
|
"""Fills an empty `config_key` just before INSERT (and the id, which is
|
||||||
|
otherwise only assigned while the INSERT is built)."""
|
||||||
|
|
||||||
|
@event.listens_for(model, "before_insert")
|
||||||
|
def _fill(_mapper, _connection, target) -> None:
|
||||||
|
if target.id is None:
|
||||||
|
target.id = uuid.uuid4()
|
||||||
|
if not target.config_key:
|
||||||
|
target.config_key = default_config_key(prefix, target.id)
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
"""Separation of duties (singleton row): may operators roll changes out themselves?"""
|
"""Separation of duties (singleton row): may operators roll changes out themselves?"""
|
||||||
from sqlalchemy import Boolean, Integer
|
from sqlalchemy import JSON, Boolean, Integer
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
from app.database import Base
|
from app.database import Base
|
||||||
@@ -11,3 +11,14 @@ class GovernanceSettings(Base):
|
|||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
# False: an operator prepares changes (edit, preview) but only an admin may deploy them.
|
# False: an operator prepares changes (edit, preview) but only an admin may deploy them.
|
||||||
operators_may_deploy: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
operators_may_deploy: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||||
|
|
||||||
|
# Rule cleanup (1.31). At most this many rules per cleanup operation (default 10, ceiling 25).
|
||||||
|
cleanup_max_rules: Mapped[int] = mapped_column(Integer, nullable=False, default=10)
|
||||||
|
# A rule must have been switched off (and confirmed) this many days before it may be deleted (minimum 1).
|
||||||
|
cleanup_delete_min_days: Mapped[int] = mapped_column(Integer, nullable=False, default=7)
|
||||||
|
# IPs/CIDRs the admin manages the firewalls from; no rule that could match them may be cleaned up. Empty list =
|
||||||
|
# the cleanup is locked (fail closed). Only an admin may change it.
|
||||||
|
cleanup_management_sources: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||||
|
# Extra TCP ports the firewalls are managed on (besides the API port of base_url, 443, 80 and 22): a rule that
|
||||||
|
# could match a management source on one of them is protected too. Only an admin may change it.
|
||||||
|
cleanup_management_ports: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||||
@@ -68,6 +68,10 @@ class NotificationSettings(Base):
|
|||||||
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||||
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||||
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||||
|
# An API key that cannot read gateway / service state makes those alerts silently blind (1.29): say so.
|
||||||
|
api_rights_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||||
|
# Gateways whose alert is switched off, "<host_id>:<gateway name>" (1.29), e.g. an idle second WAN.
|
||||||
|
gateway_alert_muted: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||||
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
|
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
|
||||||
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||||
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
|
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
|||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from app.database import Base
|
from app.database import Base
|
||||||
|
from app.models.config_key import config_key_column, register_config_key
|
||||||
from app.models.one_to_one_target import one_to_one_targets
|
from app.models.one_to_one_target import one_to_one_targets
|
||||||
from app.models.policy import one_to_one_groups
|
from app.models.policy import one_to_one_groups
|
||||||
|
|
||||||
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
|
|||||||
__tablename__ = "one_to_one_rules"
|
__tablename__ = "one_to_one_rules"
|
||||||
|
|
||||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
config_key: Mapped[str] = config_key_column()
|
||||||
description: Mapped[str] = mapped_column(String(255))
|
description: Mapped[str] = mapped_column(String(255))
|
||||||
interface: Mapped[str] = mapped_column(String(64))
|
interface: Mapped[str] = mapped_column(String(64))
|
||||||
source_net: Mapped[str] = mapped_column(String(128))
|
source_net: Mapped[str] = mapped_column(String(128))
|
||||||
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
|
|||||||
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
|
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
|
||||||
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
|
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
register_config_key(OneToOneRule, "nat1")
|
||||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
|||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from app.database import Base
|
from app.database import Base
|
||||||
|
from app.models.config_key import config_key_column, register_config_key
|
||||||
from app.models.outbound_nat_target import outbound_nat_targets
|
from app.models.outbound_nat_target import outbound_nat_targets
|
||||||
from app.models.policy import outbound_nat_groups
|
from app.models.policy import outbound_nat_groups
|
||||||
|
|
||||||
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
|
|||||||
__tablename__ = "outbound_nat_rules"
|
__tablename__ = "outbound_nat_rules"
|
||||||
|
|
||||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
config_key: Mapped[str] = config_key_column()
|
||||||
description: Mapped[str] = mapped_column(String(255))
|
description: Mapped[str] = mapped_column(String(255))
|
||||||
interface: Mapped[str] = mapped_column(String(64))
|
interface: Mapped[str] = mapped_column(String(64))
|
||||||
source_net: Mapped[str] = mapped_column(String(128))
|
source_net: Mapped[str] = mapped_column(String(128))
|
||||||
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
|
|||||||
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
|
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
|
||||||
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
|
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
register_config_key(OutboundNatRule, "out")
|
||||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
|||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from app.database import Base
|
from app.database import Base
|
||||||
|
from app.models.config_key import config_key_column, register_config_key
|
||||||
|
|
||||||
|
|
||||||
class PolicyHierarchyRule(Base):
|
class PolicyHierarchyRule(Base):
|
||||||
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
|
|||||||
__tablename__ = "policy_hierarchy_rules"
|
__tablename__ = "policy_hierarchy_rules"
|
||||||
|
|
||||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
config_key: Mapped[str] = config_key_column()
|
||||||
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
|
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
|
||||||
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
|
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
|
||||||
)
|
)
|
||||||
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
|
|||||||
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
|
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
|
||||||
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
register_config_key(PolicyHierarchyRule, "ph")
|
||||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
|||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from app.database import Base
|
from app.database import Base
|
||||||
|
from app.models.config_key import config_key_column, register_config_key
|
||||||
from app.models.port_forward_target import port_forward_targets
|
from app.models.port_forward_target import port_forward_targets
|
||||||
from app.models.policy import port_forward_groups
|
from app.models.policy import port_forward_groups
|
||||||
|
|
||||||
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
|
|||||||
__tablename__ = "port_forward_rules"
|
__tablename__ = "port_forward_rules"
|
||||||
|
|
||||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
config_key: Mapped[str] = config_key_column()
|
||||||
description: Mapped[str] = mapped_column(String(255))
|
description: Mapped[str] = mapped_column(String(255))
|
||||||
interface: Mapped[str] = mapped_column(String(64))
|
interface: Mapped[str] = mapped_column(String(64))
|
||||||
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
|
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
|
||||||
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
|
|||||||
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
|
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
|
||||||
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
|
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
register_config_key(PortForwardRule, "pf")
|
||||||
@@ -42,8 +42,12 @@ class RolloutPlan(Base):
|
|||||||
# is activated AND unchanged since (activated_fingerprint).
|
# is activated AND unchanged since (activated_fingerprint).
|
||||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||||
|
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
|
||||||
|
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||||
|
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
|
||||||
|
activated_by_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||||
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||||
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
|
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
|
||||||
@@ -91,6 +95,7 @@ class RolloutRun(Base):
|
|||||||
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||||
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||||
started_by: Mapped[str] = mapped_column(String(150))
|
started_by: Mapped[str] = mapped_column(String(150))
|
||||||
|
started_by_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True) # see activated_by_id
|
||||||
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
|
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
|
||||||
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
|
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
|||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from app.database import Base
|
from app.database import Base
|
||||||
|
from app.models.config_key import config_key_column, register_config_key
|
||||||
from app.models.rule_target import rule_targets
|
from app.models.rule_target import rule_targets
|
||||||
from app.models.policy import rule_groups
|
from app.models.policy import rule_groups
|
||||||
|
|
||||||
@@ -18,6 +19,7 @@ class FirewallRule(Base):
|
|||||||
__tablename__ = "firewall_rules"
|
__tablename__ = "firewall_rules"
|
||||||
|
|
||||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
config_key: Mapped[str] = config_key_column()
|
||||||
description: Mapped[str] = mapped_column(String(255))
|
description: Mapped[str] = mapped_column(String(255))
|
||||||
interface: Mapped[str] = mapped_column(String(64))
|
interface: Mapped[str] = mapped_column(String(64))
|
||||||
direction: Mapped[str] = mapped_column(String(8))
|
direction: Mapped[str] = mapped_column(String(8))
|
||||||
@@ -44,3 +46,6 @@ class FirewallRule(Base):
|
|||||||
deployments: Mapped[list["RuleDeployment"]] = relationship(
|
deployments: Mapped[list["RuleDeployment"]] = relationship(
|
||||||
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
register_config_key(FirewallRule, "r")
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
"""Rule cleanup (1.31, B4 part 2): switching rules off or deleting them through preview -> deploy -> confirm.
|
||||||
|
|
||||||
|
``RuleCleanupOperation`` is one cleanup run on a host (or a cluster pair, anchored at one host); its id doubles as
|
||||||
|
the preview token. ``RuleCleanupItem`` is one rule on one node of that run. The columns only carry what the design
|
||||||
|
(DESIGN-1.31) needs to resume, revert and restore after a crash; the logic lives in services/rule_cleanup_service.py.
|
||||||
|
"""
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from sqlalchemy import JSON, DateTime, ForeignKey, Index, Integer, String, Text, text
|
||||||
|
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||||
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
|
from app.database import Base
|
||||||
|
|
||||||
|
# kind: disable | enable (re-enable after a restore) | delete | restore
|
||||||
|
OPERATION_KINDS = ("disable", "enable", "delete", "restore")
|
||||||
|
# preview -> running -> awaiting_confirm -> confirmed; the rest are the side exits.
|
||||||
|
OPERATION_STATUSES = (
|
||||||
|
"preview", "running", "awaiting_confirm", "confirmed", "partial", "apply_failed", "reverting", "reverted",
|
||||||
|
"revert_failed", "done", "failed", "cancelled", "expired",
|
||||||
|
)
|
||||||
|
ITEM_STATES = (
|
||||||
|
"queued", "disabling", "disabled", "confirmed", "reverting", "reverted", "deleting", "deleted", "restoring",
|
||||||
|
"restored", "failed", "unknown", "revert_failed",
|
||||||
|
# never attempted (the run stopped at an earlier error); not an open state, so the rule is free again
|
||||||
|
"skipped",
|
||||||
|
)
|
||||||
|
# A rule in one of these states is being worked on or its outcome is unresolved: no second item may claim it.
|
||||||
|
OPEN_ITEM_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
|
||||||
|
|
||||||
|
|
||||||
|
def _now() -> datetime:
|
||||||
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
class RuleCleanupOperation(Base):
|
||||||
|
__tablename__ = "rule_cleanup_operations"
|
||||||
|
__table_args__ = (
|
||||||
|
Index("ix_rule_cleanup_operations_host", "host_id", "created_at"),
|
||||||
|
Index("ix_rule_cleanup_operations_status", "status"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||||
|
cluster_name: Mapped[Optional[str]] = mapped_column(String(120), nullable=True)
|
||||||
|
kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||||
|
status: Mapped[str] = mapped_column(String(20), nullable=False, default="preview")
|
||||||
|
actor: Mapped[str] = mapped_column(String(150), nullable=False)
|
||||||
|
# The stable account id next to the name (see migration 0026); no foreign key, a deleted account stays gone.
|
||||||
|
actor_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now)
|
||||||
|
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
|
started_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
|
# Written at every checkpoint, so another process can tell an abandoned ``running`` operation from a live one.
|
||||||
|
heartbeat_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
|
# Auto-revert deadline (disable/enable only); NULL once confirmed or for kinds without a window.
|
||||||
|
revert_deadline: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
|
confirmed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
|
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
|
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||||
|
# sha256 over everything the preview showed; the deploy recomputes it and refuses on any difference.
|
||||||
|
preview_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
# unused_days, acknowledged rules (with the text version), protected management sources.
|
||||||
|
options: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class RuleCleanupItem(Base):
|
||||||
|
__tablename__ = "rule_cleanup_items"
|
||||||
|
__table_args__ = (
|
||||||
|
Index("ix_rule_cleanup_items_operation", "operation_id"),
|
||||||
|
# Independent of any lock: one rule on one node can be in only one unfinished item at a time.
|
||||||
|
Index(
|
||||||
|
"uq_cleanup_rule_open", "host_id", "rule_uuid", unique=True,
|
||||||
|
postgresql_where=text("state IN (" + ", ".join(f"'{s}'" for s in OPEN_ITEM_STATES) + ")"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
operation_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rule_cleanup_operations.id", ondelete="CASCADE"), nullable=False)
|
||||||
|
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||||
|
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
seq: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||||
|
carp_role: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||||
|
state: Mapped[str] = mapped_column(String(16), nullable=False, default="queued")
|
||||||
|
# What the preview showed.
|
||||||
|
description: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||||
|
categories: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||||
|
confidence: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||||
|
row_before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||||
|
fp_before: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||||
|
# shadowed/duplicate: the covering rule must stay unchanged until the deploy.
|
||||||
|
cover_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||||
|
cover_fp: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||||
|
# Flattened getRule, raw answer and firmware; written BEFORE a rule is deleted.
|
||||||
|
snapshot: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||||
|
# Where the rule sat (neighbours in the same prio_group by sort_order), for the restore.
|
||||||
|
anchor_next_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||||
|
anchor_prev_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||||
|
prio_group: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||||
|
# delete -> the confirmed disable item it builds on; restore -> the delete item it undoes. Plain id, no foreign key.
|
||||||
|
source_item_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||||
|
restored_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||||
|
changed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
|
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
"""Change history of the rules that are live on a firewall (all rules, also hand-made ones).
|
||||||
|
|
||||||
|
``live_rule_states`` is the last observed state per rule (a fingerprint plus the row itself), maintained by the
|
||||||
|
``rule_observe`` loop; ``rule_change_events`` is what changed between two observations. See
|
||||||
|
services/rule_history_service.py. Neither table holds anything the firewall itself would not show on its rule page.
|
||||||
|
"""
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Index, Integer, String, UniqueConstraint
|
||||||
|
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||||
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
|
from app.database import Base
|
||||||
|
|
||||||
|
|
||||||
|
class LiveRuleState(Base):
|
||||||
|
__tablename__ = "live_rule_states"
|
||||||
|
|
||||||
|
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||||
|
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), primary_key=True)
|
||||||
|
# The rule's UUID/label on the firewall; BASELINE_KEY marks "this host has been observed once".
|
||||||
|
rule_uuid: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||||
|
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
row: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||||
|
# False = the rule is gone (kept as a marker so a returning UUID continues the numbering).
|
||||||
|
present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||||
|
# Counts the observed changes of this rule; the event of a change gets the new number, which makes the
|
||||||
|
# observation idempotent when two processes see the same change.
|
||||||
|
version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||||
|
first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
|
changed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
|
last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
|
|
||||||
|
|
||||||
|
class RuleChangeEvent(Base):
|
||||||
|
__tablename__ = "rule_change_events"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint("host_id", "rule_uuid", "seq", name="uq_rule_change_event"),
|
||||||
|
Index("ix_rule_change_events_rule", "host_id", "rule_uuid", "observed_at"),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||||
|
host_id: Mapped[uuid.UUID] = mapped_column(
|
||||||
|
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||||
|
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
seq: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
|
observed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True)
|
||||||
|
kind: Mapped[str] = mapped_column(String(16), nullable=False) # added | changed | removed | enabled | disabled
|
||||||
|
before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||||
|
after: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||||
|
origin: Mapped[str] = mapped_column(String(16), nullable=False) # opnexus | external
|
||||||
|
actor: Mapped[Optional[str]] = mapped_column(String(150), nullable=True)
|
||||||
|
# The audit entry that explains an OPNexus-made change; no foreign key, the audit trail outlives everything.
|
||||||
|
audit_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||||
|
# Start of the time since the previous observation of the host: only audit entries from then on can explain
|
||||||
|
# this change (a later reconcile must not reach back to an older deployment).
|
||||||
|
window_start: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
@@ -6,6 +6,7 @@ outbound NAT share (services/object_adopt_service.py has the logic and the evide
|
|||||||
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
|
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
|
||||||
|
|
||||||
Like rule adoption this only creates central records; nothing is written to the firewall."""
|
Like rule adoption this only creates central records; nothing is written to the firewall."""
|
||||||
|
import json
|
||||||
import uuid
|
import uuid
|
||||||
from typing import Callable
|
from typing import Callable
|
||||||
|
|
||||||
@@ -49,7 +50,8 @@ def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Calla
|
|||||||
except object_adopt_service.NotFoundOnHost as exc:
|
except object_adopt_service.NotFoundOnHost as exc:
|
||||||
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
|
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
|
||||||
except object_adopt_service.NotAdoptable as exc:
|
except object_adopt_service.NotAdoptable as exc:
|
||||||
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons),
|
||||||
|
reason_details=json.dumps(exc.details, ensure_ascii=False)) from exc
|
||||||
except LINK_ERRORS as exc:
|
except LINK_ERRORS as exc:
|
||||||
raise link_api_error(exc) from exc
|
raise link_api_error(exc) from exc
|
||||||
except OpnsenseAPIError as exc:
|
except OpnsenseAPIError as exc:
|
||||||
|
|||||||
@@ -205,6 +205,8 @@ async def remove_previous_target(alias_id: uuid.UUID, host_id: uuid.UUID, approv
|
|||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||||
deployment = next(d for d in alias.deployments if d.host_id == host_id)
|
deployment = next(d for d in alias.deployments if d.host_id == host_id)
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
try:
|
try:
|
||||||
await opnsense_client.del_alias_item(deployment.host, deployment.opnsense_alias_uuid)
|
await opnsense_client.del_alias_item(deployment.host, deployment.opnsense_alias_uuid)
|
||||||
|
|||||||
@@ -1,14 +1,15 @@
|
|||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from fastapi import APIRouter, Depends, Query
|
from fastapi import APIRouter, Depends, Query, Request
|
||||||
from sqlalchemy import select, func, or_
|
from sqlalchemy import select, func, or_
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
from app.database import get_db
|
from app.database import get_db
|
||||||
from app.models.audit import AuditEvent
|
from app.models.audit import AuditEvent
|
||||||
|
from app.services.audit_service import ADMIN_ONLY_DETAILS
|
||||||
|
|
||||||
router = APIRouter(prefix="/audit", tags=["audit"])
|
router = APIRouter(prefix="/audit", tags=["audit"])
|
||||||
|
|
||||||
@router.get("")
|
@router.get("")
|
||||||
async def history(kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
async def history(request: Request, kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||||
status: str | None = None, action: str | None = None, actor: str | None = None,
|
status: str | None = None, action: str | None = None, actor: str | None = None,
|
||||||
before: datetime | None = None, since: datetime | None = None,
|
before: datetime | None = None, since: datetime | None = None,
|
||||||
q: str = Query(default="", max_length=200), offset: int = Query(default=0, ge=0),
|
q: str = Query(default="", max_length=200), offset: int = Query(default=0, ge=0),
|
||||||
@@ -24,5 +25,17 @@ async def history(kind: str | None = None, host_id: str | None = None, entity_id
|
|||||||
clauses.append(or_(*(c.ilike('%'+q+'%') for c in [AuditEvent.entity_name, AuditEvent.host_name, AuditEvent.actor, AuditEvent.message])))
|
clauses.append(or_(*(c.ilike('%'+q+'%') for c in [AuditEvent.entity_name, AuditEvent.host_name, AuditEvent.actor, AuditEvent.message])))
|
||||||
total = await db.scalar(select(func.count()).select_from(AuditEvent).where(*clauses))
|
total = await db.scalar(select(func.count()).select_from(AuditEvent).where(*clauses))
|
||||||
rows = (await db.scalars(select(AuditEvent).where(*clauses).order_by(AuditEvent.created_at.desc(), AuditEvent.id.desc()).offset(offset).limit(limit))).all()
|
rows = (await db.scalars(select(AuditEvent).where(*clauses).order_by(AuditEvent.created_at.desc(), AuditEvent.id.desc()).offset(offset).limit(limit))).all()
|
||||||
return {"total": total, "offset": offset, "limit": limit,
|
items = [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]
|
||||||
"items": [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]}
|
if not full_view(request):
|
||||||
|
for item in items:
|
||||||
|
if isinstance(item.get("details"), dict) and ADMIN_ONLY_DETAILS in item["details"]:
|
||||||
|
item["details"] = {**{k: v for k, v in item["details"].items() if k != ADMIN_ONLY_DETAILS},
|
||||||
|
ADMIN_ONLY_DETAILS: "redacted"}
|
||||||
|
return {"total": total, "offset": offset, "limit": limit, "items": items}
|
||||||
|
|
||||||
|
|
||||||
|
def full_view(request: Request | None) -> bool:
|
||||||
|
"""An admin account, and with a token only one of scope admin."""
|
||||||
|
user = getattr(getattr(request, "state", None), "user", None)
|
||||||
|
token = getattr(getattr(request, "state", None), "api_token", None)
|
||||||
|
return getattr(user, "role", None) == "admin" and (token is None or token.scope == "admin")
|
||||||
+17
-19
@@ -15,8 +15,8 @@ logger = logging.getLogger(__name__)
|
|||||||
|
|
||||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
||||||
|
|
||||||
# Plain name: used on HTTP-only (lab) deployments and accepted for a while on
|
# Plain name: the session cookie of HTTP-only (lab) deployments. On HTTPS deployments it is NOT accepted any
|
||||||
# HTTPS deployments so that sessions issued before 1.27 survive the upgrade.
|
# more (1.29): it is only still deleted (login, logout) so that a pre-1.27 cookie does not linger.
|
||||||
COOKIE_NAME = "opnexus-session"
|
COOKIE_NAME = "opnexus-session"
|
||||||
# HTTPS deployments: the __Host- prefix makes the browser refuse any cookie of
|
# HTTPS deployments: the __Host- prefix makes the browser refuse any cookie of
|
||||||
# this name that is not Secure, has a Domain attribute or a Path other than
|
# this name that is not Secure, has a Domain attribute or a Path other than
|
||||||
@@ -31,16 +31,11 @@ def session_cookie_name() -> str:
|
|||||||
|
|
||||||
|
|
||||||
def session_token(request: Request) -> str:
|
def session_token(request: Request) -> str:
|
||||||
"""The session token the browser sent. On HTTPS the __Host- cookie always
|
"""The session token the browser sent: the __Host- cookie on HTTPS deployments, the plain one otherwise.
|
||||||
wins; the plain name is only a migration fallback when the browser has no
|
Never the other name (1.29): a plain-name cookie can be planted by any plain-HTTP or sibling-subdomain page and
|
||||||
__Host- cookie yet (it is replaced on the next /api/auth/me)."""
|
must not be able to carry a session on an HTTPS deployment (the __Host- prefix does not separate PORTS, see
|
||||||
if settings.cookie_secure:
|
BACKLOG S8)."""
|
||||||
return request.cookies.get(SECURE_COOKIE_NAME) or request.cookies.get(COOKIE_NAME, "")
|
return request.cookies.get(session_cookie_name(), "")
|
||||||
return request.cookies.get(COOKIE_NAME, "")
|
|
||||||
|
|
||||||
|
|
||||||
def uses_legacy_cookie(request: Request) -> bool:
|
|
||||||
return settings.cookie_secure and not request.cookies.get(SECURE_COOKIE_NAME) and bool(request.cookies.get(COOKIE_NAME))
|
|
||||||
|
|
||||||
|
|
||||||
def has_session_cookie(request: Request) -> bool:
|
def has_session_cookie(request: Request) -> bool:
|
||||||
@@ -70,7 +65,7 @@ def _set_session_cookie(response: Response, token: str, max_age: int | None = No
|
|||||||
path="/",
|
path="/",
|
||||||
)
|
)
|
||||||
if settings.cookie_secure:
|
if settings.cookie_secure:
|
||||||
# Migration: a pre-1.27 plain-name cookie must not linger next to the new one.
|
# A pre-1.27 plain-name cookie must not linger next to the new one.
|
||||||
response.delete_cookie(key=COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
response.delete_cookie(key=COOKIE_NAME, path="/", secure=True, httponly=True, samesite="lax")
|
||||||
|
|
||||||
|
|
||||||
@@ -144,7 +139,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
|
|||||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
||||||
|
|
||||||
verified_hash = user.password_hash
|
verified_hash = user.password_hash
|
||||||
if user.totp_enabled:
|
second_factor_checked = bool(user.totp_enabled) # the state this login is decided on; create_session re-checks it
|
||||||
|
if second_factor_checked:
|
||||||
# Only revealed after the password was correct, so it cannot be used
|
# Only revealed after the password was correct, so it cannot be used
|
||||||
# to find out which usernames exist or have 2FA.
|
# to find out which usernames exist or have 2FA.
|
||||||
if not payload.totp_code:
|
if not payload.totp_code:
|
||||||
@@ -158,7 +154,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
|
|||||||
auth_service.throttle.reset(ip, payload.username)
|
auth_service.throttle.reset(ip, payload.username)
|
||||||
failed_since_last_login = auth_service.throttle.pop_failures_since_success(payload.username)
|
failed_since_last_login = auth_service.throttle.pop_failures_since_success(payload.username)
|
||||||
token, expires_at = await auth_service.create_session(
|
token, expires_at = await auth_service.create_session(
|
||||||
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash
|
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash,
|
||||||
|
verified_totp=second_factor_checked,
|
||||||
)
|
)
|
||||||
_set_session_cookie(response, token)
|
_set_session_cookie(response, token)
|
||||||
logger.info("Successful login for '%s' from %s", user.username, ip)
|
logger.info("Successful login for '%s' from %s", user.username, ip)
|
||||||
@@ -257,6 +254,8 @@ async def totp_enable(
|
|||||||
_account_failure(user, "2fa-enable")
|
_account_failure(user, "2fa-enable")
|
||||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.TOTP_INVALID)
|
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.TOTP_INVALID)
|
||||||
_account_success(user, "2fa-enable")
|
_account_success(user, "2fa-enable")
|
||||||
|
# Same row lock as create_session: a sign-in that decided "no second factor needed" is ordered before or after this.
|
||||||
|
await db.execute(select(AdminUser.id).where(AdminUser.id == user.id).with_for_update())
|
||||||
plain, hashes = totp_service.new_recovery_codes()
|
plain, hashes = totp_service.new_recovery_codes()
|
||||||
totp_service.promote_pending(user.id)
|
totp_service.promote_pending(user.id)
|
||||||
user.totp_enabled = True
|
user.totp_enabled = True
|
||||||
@@ -298,6 +297,9 @@ async def totp_disable(
|
|||||||
async def logout(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
async def logout(request: Request, response: Response, db: AsyncSession = Depends(get_db)):
|
||||||
token = session_token(request)
|
token = session_token(request)
|
||||||
await auth_service.revoke_session(db, token)
|
await auth_service.revoke_session(db, token)
|
||||||
|
legacy = request.cookies.get(COOKIE_NAME, "")
|
||||||
|
if settings.cookie_secure and legacy and legacy != token:
|
||||||
|
await auth_service.revoke_session(db, legacy) # an old plain-name cookie ends with the logout too
|
||||||
_clear_session_cookie(response)
|
_clear_session_cookie(response)
|
||||||
return {"authenticated": False}
|
return {"authenticated": False}
|
||||||
|
|
||||||
@@ -316,10 +318,6 @@ async def me(request: Request, response: Response, db: AsyncSession = Depends(ge
|
|||||||
user = await auth_service.resolve_session(db, token)
|
user = await auth_service.resolve_session(db, token)
|
||||||
if user is None:
|
if user is None:
|
||||||
return {"authenticated": False}
|
return {"authenticated": False}
|
||||||
if uses_legacy_cookie(request):
|
|
||||||
# Soft migration (1.27): the same server-side session moves to the
|
|
||||||
# __Host- cookie and the plain-name cookie is deleted — nobody is logged out.
|
|
||||||
_set_session_cookie(response, token)
|
|
||||||
return {"authenticated": True, "username": user.username, "role": user.role}
|
return {"authenticated": True, "username": user.username, "role": user.role}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -37,6 +37,13 @@ acme_accounts_router = APIRouter(prefix="/acme/accounts", tags=["acme"])
|
|||||||
acme_certificates_router = APIRouter(prefix="/acme/certificates", tags=["acme"])
|
acme_certificates_router = APIRouter(prefix="/acme/certificates", tags=["acme"])
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_host_unchanged_once_deployed(obj, new_host_id: uuid.UUID) -> None:
|
||||||
|
"""The firewall-local UUID, the stored payload and the success status all belong to the host the object was pushed
|
||||||
|
to. Re-pointing it would report success for a host that never received it and orphan the object on the old one."""
|
||||||
|
if new_host_id != obj.host_id and obj.opnsense_uuid:
|
||||||
|
raise api_error(409, ErrorCode.ZIEL_HOST_NACH_AUSROLLEN_FEST)
|
||||||
|
|
||||||
|
|
||||||
async def _get_host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
async def _get_host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||||
host = await db.get(OpnsenseHost, host_id)
|
host = await db.get(OpnsenseHost, host_id)
|
||||||
if not host:
|
if not host:
|
||||||
@@ -82,6 +89,7 @@ async def edit_managed_certificate(cert_id: uuid.UUID, payload: ManagedCertifica
|
|||||||
if cert.updated_at != payload.expected_updated_at:
|
if cert.updated_at != payload.expected_updated_at:
|
||||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||||
await _get_host(db, payload.host_id)
|
await _get_host(db, payload.host_id)
|
||||||
|
_ensure_host_unchanged_once_deployed(cert, payload.host_id)
|
||||||
for field, value in payload.model_dump(exclude={"expected_updated_at", "private_key"}).items():
|
for field, value in payload.model_dump(exclude={"expected_updated_at", "private_key"}).items():
|
||||||
setattr(cert, field, value)
|
setattr(cert, field, value)
|
||||||
if payload.private_key:
|
if payload.private_key:
|
||||||
@@ -146,6 +154,11 @@ async def edit_acme_account(account_id: uuid.UUID, payload: AcmeAccountEdit, db:
|
|||||||
if account.updated_at != payload.expected_updated_at:
|
if account.updated_at != payload.expected_updated_at:
|
||||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||||
await _get_host(db, payload.host_id)
|
await _get_host(db, payload.host_id)
|
||||||
|
_ensure_host_unchanged_once_deployed(account, payload.host_id)
|
||||||
|
if payload.host_id != account.host_id and await db.scalar(
|
||||||
|
select(AcmeCertificateRequest.id).where(AcmeCertificateRequest.account_id == account.id).limit(1)
|
||||||
|
):
|
||||||
|
raise api_error(409, ErrorCode.ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET)
|
||||||
for field, value in payload.model_dump(exclude={"expected_updated_at", "eab_hmac"}).items():
|
for field, value in payload.model_dump(exclude={"expected_updated_at", "eab_hmac"}).items():
|
||||||
setattr(account, field, value)
|
setattr(account, field, value)
|
||||||
if payload.eab_hmac:
|
if payload.eab_hmac:
|
||||||
@@ -236,6 +249,7 @@ async def edit_acme_certificate_request(request_id: uuid.UUID, payload: AcmeCert
|
|||||||
if req.updated_at != payload.expected_updated_at:
|
if req.updated_at != payload.expected_updated_at:
|
||||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||||
await _get_host(db, payload.host_id)
|
await _get_host(db, payload.host_id)
|
||||||
|
_ensure_host_unchanged_once_deployed(req, payload.host_id)
|
||||||
await _get_account(db, payload.account_id, payload.host_id)
|
await _get_account(db, payload.account_id, payload.host_id)
|
||||||
for field, value in payload.model_dump(exclude={"expected_updated_at"}).items():
|
for field, value in payload.model_dump(exclude={"expected_updated_at"}).items():
|
||||||
setattr(req, field, value)
|
setattr(req, field, value)
|
||||||
|
|||||||
@@ -0,0 +1,154 @@
|
|||||||
|
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
|
||||||
|
|
||||||
|
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
|
||||||
|
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
|
||||||
|
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
|
||||||
|
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, Query, Request
|
||||||
|
from fastapi.concurrency import run_in_threadpool
|
||||||
|
from fastapi.responses import Response
|
||||||
|
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from app.config_code import apply as applier
|
||||||
|
from app.config_code import plan as planner
|
||||||
|
from app.config_code import schema
|
||||||
|
from app.config_code.export import export_yaml
|
||||||
|
from app.config_code.loader import MAX_BYTES, ConfigError, parse
|
||||||
|
from app.database import get_db
|
||||||
|
from app.errors import ErrorCode, api_error
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/config", tags=["config"])
|
||||||
|
|
||||||
|
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
|
||||||
|
# and read before parsing so a huge body never reaches the JSON parser.
|
||||||
|
MAX_REQUEST_BYTES = 4 * MAX_BYTES
|
||||||
|
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
|
||||||
|
# open to read-only accounts and read tokens.
|
||||||
|
_busy = asyncio.Semaphore(1)
|
||||||
|
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
|
||||||
|
MAX_WAITING = 4
|
||||||
|
_waiting = 0
|
||||||
|
|
||||||
|
|
||||||
|
class _Slot:
|
||||||
|
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
|
||||||
|
|
||||||
|
async def __aenter__(self):
|
||||||
|
global _waiting
|
||||||
|
if _waiting >= MAX_WAITING:
|
||||||
|
raise api_error(429, ErrorCode.CONFIG_BUSY)
|
||||||
|
_waiting += 1
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def run(self, coro_fn):
|
||||||
|
async with _busy:
|
||||||
|
return await coro_fn()
|
||||||
|
|
||||||
|
async def __aexit__(self, *exc):
|
||||||
|
global _waiting
|
||||||
|
_waiting -= 1
|
||||||
|
|
||||||
|
|
||||||
|
def _encode(text: str) -> bytes:
|
||||||
|
try:
|
||||||
|
return text.encode("utf-8")
|
||||||
|
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
|
||||||
|
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
|
||||||
|
|
||||||
|
|
||||||
|
class ImportRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
yaml: str
|
||||||
|
prune: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class ApplyRequest(ImportRequest):
|
||||||
|
plan_token: str = Field(min_length=64, max_length=64)
|
||||||
|
|
||||||
|
|
||||||
|
def _config_error(exc: ConfigError):
|
||||||
|
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
|
||||||
|
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
|
||||||
|
|
||||||
|
|
||||||
|
async def _read_body(request: Request, model: type[BaseModel]):
|
||||||
|
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
|
||||||
|
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||||
|
chunks, size = [], 0
|
||||||
|
async for chunk in request.stream():
|
||||||
|
size += len(chunk)
|
||||||
|
if size > MAX_REQUEST_BYTES:
|
||||||
|
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||||
|
chunks.append(chunk)
|
||||||
|
try:
|
||||||
|
return model.model_validate(json.loads(b"".join(chunks)))
|
||||||
|
except (ValueError, ValidationError):
|
||||||
|
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/export")
|
||||||
|
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
|
||||||
|
wanted = None
|
||||||
|
if kinds:
|
||||||
|
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
|
||||||
|
if not wanted <= set(schema.KINDS):
|
||||||
|
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
|
||||||
|
text = await export_yaml(db, wanted)
|
||||||
|
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
|
||||||
|
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/import/plan")
|
||||||
|
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
async with _Slot() as slot:
|
||||||
|
body = await _read_body(request, ImportRequest)
|
||||||
|
raw = _encode(body.yaml)
|
||||||
|
|
||||||
|
async def work():
|
||||||
|
try:
|
||||||
|
parsed = await run_in_threadpool(parse, raw)
|
||||||
|
return await planner.build_plan(db, parsed, body.prune)
|
||||||
|
except ConfigError as exc:
|
||||||
|
raise _config_error(exc) from None
|
||||||
|
finally:
|
||||||
|
await db.rollback() # never leave anything in the session: plan is read-only
|
||||||
|
|
||||||
|
result = await slot.run(work)
|
||||||
|
return result.public()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/import/apply")
|
||||||
|
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
async with _Slot() as slot:
|
||||||
|
body = await _read_body(request, ApplyRequest)
|
||||||
|
raw = _encode(body.yaml)
|
||||||
|
|
||||||
|
async def work():
|
||||||
|
try:
|
||||||
|
parsed = await run_in_threadpool(parse, raw)
|
||||||
|
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
|
||||||
|
except ConfigError as exc:
|
||||||
|
await db.rollback()
|
||||||
|
raise _config_error(exc) from None
|
||||||
|
except applier.PlanStale:
|
||||||
|
await db.rollback()
|
||||||
|
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
|
||||||
|
except applier.PlanBlocked as exc:
|
||||||
|
await db.rollback()
|
||||||
|
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
|
||||||
|
except applier.SelfCheckFailed as exc:
|
||||||
|
await db.rollback()
|
||||||
|
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
|
||||||
|
except Exception:
|
||||||
|
await db.rollback()
|
||||||
|
raise
|
||||||
|
await db.commit()
|
||||||
|
return parsed, result
|
||||||
|
|
||||||
|
parsed, result = await slot.run(work)
|
||||||
|
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
|
||||||
@@ -49,7 +49,7 @@ from app.models.wireguard_tunnel import WireguardTunnel
|
|||||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||||
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
|
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
|
||||||
from app.models.resource_sample import HostResourceSample
|
from app.models.resource_sample import HostResourceSample
|
||||||
from app.services import gateway_health_service
|
from app.services import gateway_health_service, notification_settings_service
|
||||||
from app.services.host_status_service import get_statuses
|
from app.services.host_status_service import get_statuses
|
||||||
from app.services.monitor_checks import STALE_AFTER
|
from app.services.monitor_checks import STALE_AFTER
|
||||||
|
|
||||||
@@ -207,34 +207,43 @@ async def collect_pending_actions(db: AsyncSession) -> list[PendingAction]:
|
|||||||
return pending_actions
|
return pending_actions
|
||||||
|
|
||||||
|
|
||||||
async def _gateways_by_host(db: AsyncSession, hosts) -> dict:
|
async def _sample_readings_by_host(db: AsyncSession, hosts, muted: set[str]) -> tuple[dict, dict]:
|
||||||
"""Gateway state from each host's newest resource sample; hosts without a fresh sample are left out."""
|
"""(gateways, missing API rights) from each host's newest resource sample; hosts without a fresh sample are left out."""
|
||||||
result: dict = {}
|
gateways: dict = {}
|
||||||
|
denied: dict = {}
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
for host in hosts:
|
for host in hosts:
|
||||||
row = (
|
row = (
|
||||||
await db.execute(
|
await db.execute(
|
||||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json())
|
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json(), HostResourceSample.raw["denied"].as_json())
|
||||||
.where(HostResourceSample.host_id == host.id)
|
.where(HostResourceSample.host_id == host.id)
|
||||||
.order_by(HostResourceSample.sampled_at.desc())
|
.order_by(HostResourceSample.sampled_at.desc())
|
||||||
.limit(1)
|
.limit(1)
|
||||||
)
|
)
|
||||||
).first()
|
).first()
|
||||||
if row is not None and row[1] is not None and now - row[0] <= STALE_AFTER:
|
if row is None or now - row[0] > STALE_AFTER:
|
||||||
result[host.id] = gateway_health_service.from_compact(row[1])
|
continue
|
||||||
return result
|
if row[1] is not None:
|
||||||
|
gateways[host.id] = [
|
||||||
|
g.model_copy(update={"alert_enabled": f"{host.id}:{g.name}" not in muted})
|
||||||
|
for g in gateway_health_service.from_compact(row[1])
|
||||||
|
]
|
||||||
|
if isinstance(row[2], list):
|
||||||
|
denied[host.id] = [str(x) for x in row[2]]
|
||||||
|
return gateways, denied
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=DashboardSummary)
|
@router.get("", response_model=DashboardSummary)
|
||||||
async def summary(db: AsyncSession = Depends(get_db)):
|
async def summary(db: AsyncSession = Depends(get_db)):
|
||||||
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
||||||
statuses = await get_statuses(hosts)
|
statuses = await get_statuses(hosts)
|
||||||
gateways = await _gateways_by_host(db, hosts)
|
muted = set((await notification_settings_service.get_settings(db)).gateway_alert_muted or [])
|
||||||
|
gateways, denied = await _sample_readings_by_host(db, hosts, muted)
|
||||||
host_summaries = [
|
host_summaries = [
|
||||||
DashboardHost(
|
DashboardHost(
|
||||||
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
|
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
|
||||||
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
|
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
|
||||||
gateways=gateways.get(host.id),
|
gateways=gateways.get(host.id), api_rights_missing=denied.get(host.id),
|
||||||
)
|
)
|
||||||
for host in hosts
|
for host in hosts
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -1,17 +1,26 @@
|
|||||||
|
import re
|
||||||
import uuid
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends
|
from fastapi import APIRouter, Depends
|
||||||
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.database import get_db
|
from app.database import get_db
|
||||||
from app.errors import ErrorCode, api_error
|
from app.errors import ErrorCode, api_error
|
||||||
from app.models.host import OpnsenseHost
|
from app.models.host import OpnsenseHost
|
||||||
from app.schemas.gateway import GatewayHealth
|
from app.models.resource_sample import HostResourceSample
|
||||||
from app.services import gateway_health_service
|
from app.schemas.gateway import ApiRightsState, GatewayAlertState, GatewayAlertUpdate, GatewayHealth
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services import audit_service, gateway_health_service, notification_settings_service
|
||||||
|
from app.services.monitor_checks import STALE_AFTER
|
||||||
|
from app.services.opnsense_client import OpnsenseAPIError, OpnsenseForbiddenError
|
||||||
|
|
||||||
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
|
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
|
||||||
|
|
||||||
|
# OPNsense gateway names are letters, digits and underscores (WAN_GW, WAN_DHCP6); a little more is allowed.
|
||||||
|
_GATEWAY_NAME = re.compile(r"[A-Za-z0-9_.-]{1,64}")
|
||||||
|
MAX_MUTED_GATEWAYS = 200
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=list[GatewayHealth])
|
@router.get("", response_model=list[GatewayHealth])
|
||||||
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||||
@@ -20,6 +29,71 @@ async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
|||||||
if host is None:
|
if host is None:
|
||||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
try:
|
try:
|
||||||
return await gateway_health_service.get_gateway_health(host)
|
gateways = await gateway_health_service.get_gateway_health(host)
|
||||||
|
except OpnsenseForbiddenError as exc:
|
||||||
|
# The API key may not read the gateway state: say so instead of a generic upstream failure.
|
||||||
|
raise api_error(403, ErrorCode.API_RIGHTS_MISSING) from exc
|
||||||
except OpnsenseAPIError as exc:
|
except OpnsenseAPIError as exc:
|
||||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||||
|
muted = set((await notification_settings_service.get_settings(db)).gateway_alert_muted or [])
|
||||||
|
return [g.model_copy(update={"alert_enabled": f"{host.id}:{g.name}" not in muted}) for g in gateways]
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/api-rights", response_model=ApiRightsState)
|
||||||
|
async def api_rights(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Side readings the newest FRESH resource sample could not take because the API key lacks the right
|
||||||
|
(null = no fresh sample, so nothing is known). Stored data only: no call to the firewall."""
|
||||||
|
if await db.get(OpnsenseHost, host_id) is None:
|
||||||
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
row = (
|
||||||
|
await db.execute(
|
||||||
|
select(HostResourceSample.sampled_at, HostResourceSample.raw["denied"].as_json())
|
||||||
|
.where(HostResourceSample.host_id == host_id)
|
||||||
|
.order_by(HostResourceSample.sampled_at.desc())
|
||||||
|
.limit(1)
|
||||||
|
)
|
||||||
|
).first()
|
||||||
|
if row is None or datetime.now(timezone.utc) - row[0] > STALE_AFTER or not isinstance(row[1], list):
|
||||||
|
return ApiRightsState(missing=None)
|
||||||
|
return ApiRightsState(missing=[str(x) for x in row[1]])
|
||||||
|
|
||||||
|
|
||||||
|
@router.put("/{name}/alert", response_model=GatewayAlertState)
|
||||||
|
async def set_gateway_alert(host_id: uuid.UUID, name: str, payload: GatewayAlertUpdate, db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Switch the gateway-down alert on or off for ONE gateway (admins only, see api_token_service)."""
|
||||||
|
host = await db.get(OpnsenseHost, host_id)
|
||||||
|
if host is None:
|
||||||
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
if not _GATEWAY_NAME.fullmatch(name):
|
||||||
|
raise api_error(422, ErrorCode.GATEWAY_NAME_INVALID)
|
||||||
|
if not payload.enabled:
|
||||||
|
# Only a gateway the firewall really reported can be muted ("wan_gw" vs "WAN_GW" would silently do nothing).
|
||||||
|
# Without a usable sample nothing is known, so the request goes through.
|
||||||
|
known = (
|
||||||
|
await db.execute(
|
||||||
|
select(HostResourceSample.raw["gateways"].as_json())
|
||||||
|
.where(HostResourceSample.host_id == host_id)
|
||||||
|
.order_by(HostResourceSample.sampled_at.desc())
|
||||||
|
.limit(1)
|
||||||
|
)
|
||||||
|
).scalar()
|
||||||
|
if isinstance(known, list) and name not in {g.get("name") for g in known if isinstance(g, dict)}:
|
||||||
|
raise api_error(404, ErrorCode.GATEWAY_UNKNOWN)
|
||||||
|
row = await notification_settings_service.get_settings(db)
|
||||||
|
await db.refresh(row, with_for_update=True)
|
||||||
|
muted = list(row.gateway_alert_muted or [])
|
||||||
|
key = f"{host.id}:{name}"
|
||||||
|
if payload.enabled:
|
||||||
|
muted = [m for m in muted if m != key]
|
||||||
|
elif key not in muted:
|
||||||
|
if len(muted) >= MAX_MUTED_GATEWAYS:
|
||||||
|
raise api_error(422, ErrorCode.GATEWAY_MUTE_LIMIT, limit=str(MAX_MUTED_GATEWAYS))
|
||||||
|
muted.append(key)
|
||||||
|
row.gateway_alert_muted = muted # a new list, so the JSON column notices the change
|
||||||
|
await db.commit()
|
||||||
|
# The request middleware logs the call itself; this entry says what it did (only the validated name goes in).
|
||||||
|
await audit_service.persist(
|
||||||
|
kind="host", action="gateway_alert_on" if payload.enabled else "gateway_alert_off", entity_id=str(host.id),
|
||||||
|
host_id=str(host.id), host_name=host.name, status="accepted", message=f"Gateway {name}",
|
||||||
|
)
|
||||||
|
return GatewayAlertState(name=name, alert_enabled=payload.enabled)
|
||||||
@@ -1,19 +1,102 @@
|
|||||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it)."""
|
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it).
|
||||||
from fastapi import APIRouter, Depends
|
|
||||||
from pydantic import BaseModel
|
The rule-cleanup settings (1.31) live here too: how many rules one cleanup may touch, how long a rule must stay
|
||||||
|
switched off before it may be deleted, the management sources (addresses the firewalls are managed from; no rule
|
||||||
|
that could match them is ever cleaned up) and extra management ports (besides the API port, 443, 80 and 22). Only an admin writes any of it (middleware: ``/api/settings/...`` is
|
||||||
|
admin-only for operators and tokens below scope admin; checked again here) and every change is audited with the
|
||||||
|
old and the new values."""
|
||||||
|
import ipaddress
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, Request
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.database import get_db
|
from app.database import get_db
|
||||||
from app.errors import ErrorCode, api_error
|
from app.errors import ErrorCode, api_error
|
||||||
from app.models.governance import GovernanceSettings
|
from app.models.governance import GovernanceSettings
|
||||||
from app.models.user import AdminUser
|
from app.models.user import AdminUser
|
||||||
from app.routers.auth import current_user
|
from app.routers.auth import client_ip, current_user
|
||||||
|
from app.services import audit_service
|
||||||
|
|
||||||
router = APIRouter(prefix="/settings/governance", tags=["governance"])
|
router = APIRouter(prefix="/settings/governance", tags=["governance"])
|
||||||
|
|
||||||
|
MAX_CLEANUP_RULES = 25 # same ceiling as services/rule_cleanup_service.MAX_RULES_CEILING
|
||||||
|
MAX_DELETE_MIN_DAYS = 365
|
||||||
|
MAX_MANAGEMENT_SOURCES = 20
|
||||||
|
MAX_MANAGEMENT_PORTS = 20
|
||||||
|
|
||||||
|
|
||||||
class GovernanceRead(BaseModel):
|
class GovernanceRead(BaseModel):
|
||||||
operators_may_deploy: bool
|
operators_may_deploy: bool
|
||||||
|
cleanup_max_rules: int = 10
|
||||||
|
cleanup_delete_min_days: int = 7
|
||||||
|
# The list itself is shown to admins only; everybody else sees just whether it is set.
|
||||||
|
cleanup_management_sources: list[str] = Field(default_factory=list)
|
||||||
|
cleanup_management_sources_configured: bool = False
|
||||||
|
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
|
||||||
|
cleanup_management_ports: list[int] = Field(default_factory=list)
|
||||||
|
cleanup_management_ports_configured: bool = False
|
||||||
|
# The address this request came from, as a suggestion for the management sources (B12; admins only). Only a
|
||||||
|
# suggestion: the firewalls may see the workstation under another address (NAT, VPN).
|
||||||
|
your_address: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class GovernanceUpdate(BaseModel):
|
||||||
|
operators_may_deploy: bool | None = None
|
||||||
|
cleanup_max_rules: int | None = None
|
||||||
|
cleanup_delete_min_days: int | None = None
|
||||||
|
cleanup_management_sources: list[str] | None = None
|
||||||
|
cleanup_management_ports: list[Any] | None = None # validated in clean_management_ports (no bool coercion)
|
||||||
|
|
||||||
|
|
||||||
|
def _bad(field: str, reason: str):
|
||||||
|
return api_error(422, ErrorCode.BEREINIGUNG_EINSTELLUNG_UNGUELTIG, field=field, reason=reason)
|
||||||
|
|
||||||
|
|
||||||
|
def clean_management_sources(values: list[str]) -> list[str]:
|
||||||
|
"""IP addresses and CIDR networks only (no host names: they could resolve differently later), canonical form,
|
||||||
|
without duplicates; not the whole address space, which would protect every rule."""
|
||||||
|
if len(values) > MAX_MANAGEMENT_SOURCES:
|
||||||
|
raise _bad("cleanup_management_sources", "too_many")
|
||||||
|
cleaned: list[str] = []
|
||||||
|
for raw in values:
|
||||||
|
text = str(raw).strip()
|
||||||
|
try:
|
||||||
|
net = ipaddress.ip_network(text, strict=False)
|
||||||
|
except ValueError:
|
||||||
|
raise _bad("cleanup_management_sources", "not_an_address") from None
|
||||||
|
if net.prefixlen == 0:
|
||||||
|
raise _bad("cleanup_management_sources", "everything")
|
||||||
|
entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)
|
||||||
|
if entry not in cleaned:
|
||||||
|
cleaned.append(entry)
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
def clean_management_ports(values) -> list[int]:
|
||||||
|
"""TCP ports 1..65535, at most MAX_MANAGEMENT_PORTS, without duplicates, sorted."""
|
||||||
|
if len(values) > MAX_MANAGEMENT_PORTS:
|
||||||
|
raise _bad("cleanup_management_ports", "too_many")
|
||||||
|
ports: set[int] = set()
|
||||||
|
for raw in values:
|
||||||
|
text = str(raw).strip()
|
||||||
|
if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:
|
||||||
|
raise _bad("cleanup_management_ports", "not_a_port")
|
||||||
|
ports.add(int(text))
|
||||||
|
return sorted(ports)
|
||||||
|
|
||||||
|
|
||||||
|
def _read(row: GovernanceSettings, user: AdminUser) -> GovernanceRead:
|
||||||
|
sources = list(row.cleanup_management_sources or [])
|
||||||
|
ports = list(row.cleanup_management_ports or [])
|
||||||
|
return GovernanceRead(
|
||||||
|
operators_may_deploy=row.operators_may_deploy, cleanup_max_rules=row.cleanup_max_rules or 10,
|
||||||
|
cleanup_delete_min_days=row.cleanup_delete_min_days or 7,
|
||||||
|
cleanup_management_sources=sources if user.role == "admin" else [],
|
||||||
|
cleanup_management_sources_configured=bool(sources),
|
||||||
|
cleanup_management_ports=ports if user.role == "admin" else [],
|
||||||
|
cleanup_management_ports_configured=bool(ports))
|
||||||
|
|
||||||
|
|
||||||
async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||||
@@ -25,17 +108,45 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
|||||||
return row
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _suggestion(request: Request, user: AdminUser) -> str | None:
|
||||||
|
if user.role != "admin":
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return str(ipaddress.ip_address(client_ip(request)))
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=GovernanceRead)
|
@router.get("", response_model=GovernanceRead)
|
||||||
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
async def read(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||||
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
|
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
|
||||||
return await get_settings(db)
|
return _read(await get_settings(db), user).model_copy(update={"your_address": _suggestion(request, user)})
|
||||||
|
|
||||||
|
|
||||||
@router.put("", response_model=GovernanceRead)
|
@router.put("", response_model=GovernanceRead)
|
||||||
async def update(payload: GovernanceRead, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
async def update(payload: GovernanceUpdate, request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||||
if user.role != "admin":
|
if user.role != "admin":
|
||||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||||
row = await get_settings(db)
|
row = await get_settings(db)
|
||||||
row.operators_may_deploy = payload.operators_may_deploy
|
changes: dict[str, dict] = {}
|
||||||
|
if payload.cleanup_max_rules is not None:
|
||||||
|
if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:
|
||||||
|
raise _bad("cleanup_max_rules", f"1..{MAX_CLEANUP_RULES}")
|
||||||
|
if payload.cleanup_delete_min_days is not None:
|
||||||
|
if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:
|
||||||
|
raise _bad("cleanup_delete_min_days", f"1..{MAX_DELETE_MIN_DAYS}")
|
||||||
|
sources = clean_management_sources(payload.cleanup_management_sources) if payload.cleanup_management_sources is not None else None
|
||||||
|
ports = clean_management_ports(payload.cleanup_management_ports) if payload.cleanup_management_ports is not None else None
|
||||||
|
for field, value in (("cleanup_max_rules", payload.cleanup_max_rules), ("cleanup_delete_min_days", payload.cleanup_delete_min_days),
|
||||||
|
("cleanup_management_sources", sources), ("cleanup_management_ports", ports)):
|
||||||
|
if value is not None and value != getattr(row, field):
|
||||||
|
changes[field] = {"old": getattr(row, field), "new": value}
|
||||||
|
setattr(row, field, value)
|
||||||
|
if payload.operators_may_deploy is not None:
|
||||||
|
row.operators_may_deploy = payload.operators_may_deploy
|
||||||
|
if changes:
|
||||||
|
db.add(audit_service.make_event(
|
||||||
|
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
|
||||||
|
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
|
||||||
await db.commit()
|
await db.commit()
|
||||||
return row
|
return _read(row, user).model_copy(update={"your_address": _suggestion(request, user)})
|
||||||
@@ -99,6 +99,12 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
|||||||
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
|
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
|
||||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
|
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
|
||||||
|
# VPN objects target groups too; their link rows are ON DELETE CASCADE, so without this check the group would
|
||||||
|
# vanish and the VPN would silently lose its targets (B15, found while designing the config import).
|
||||||
|
from app.models.policy import ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups
|
||||||
|
for assignment in (ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups):
|
||||||
|
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||||
|
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET)
|
||||||
from app.models.rollout import rollout_wave_groups
|
from app.models.rollout import rollout_wave_groups
|
||||||
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
|
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
|
||||||
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
|
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
|
||||||
|
|||||||
@@ -159,9 +159,13 @@ class RedeemRequest(BaseModel):
|
|||||||
class RevalidateRequest(BaseModel):
|
class RevalidateRequest(BaseModel):
|
||||||
gui_session_id: uuid.UUID
|
gui_session_id: uuid.UUID
|
||||||
session_hash: str = Field(pattern=_HEX64)
|
session_hash: str = Field(pattern=_HEX64)
|
||||||
|
# Fingerprint of the target as handed out at redeem time (optional: an older gateway image omits it).
|
||||||
|
target_fp: str | None = Field(default=None, pattern=_HEX64)
|
||||||
|
|
||||||
|
|
||||||
class EndRequest(RevalidateRequest):
|
class EndRequest(BaseModel):
|
||||||
|
gui_session_id: uuid.UUID
|
||||||
|
session_hash: str = Field(pattern=_HEX64)
|
||||||
reason: Literal["idle", "absolute", "logout", "revoked", "superseded", "shutdown"]
|
reason: Literal["idle", "absolute", "logout", "revoked", "superseded", "shutdown"]
|
||||||
|
|
||||||
|
|
||||||
@@ -176,7 +180,8 @@ async def internal_redeem(payload: RedeemRequest, db: AsyncSession = Depends(get
|
|||||||
|
|
||||||
@internal_router.post("/revalidate", dependencies=[Depends(_gateway_auth)])
|
@internal_router.post("/revalidate", dependencies=[Depends(_gateway_auth)])
|
||||||
async def internal_revalidate(payload: RevalidateRequest, db: AsyncSession = Depends(get_db)):
|
async def internal_revalidate(payload: RevalidateRequest, db: AsyncSession = Depends(get_db)):
|
||||||
return await svc.revalidate(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash)
|
return await svc.revalidate(db, gui_session_id=payload.gui_session_id, session_hash=payload.session_hash,
|
||||||
|
target_fp=payload.target_fp)
|
||||||
|
|
||||||
|
|
||||||
@internal_router.post("/end", dependencies=[Depends(_gateway_auth)])
|
@internal_router.post("/end", dependencies=[Depends(_gateway_auth)])
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ from app.models.port_forward_deployment import PortForwardDeployment
|
|||||||
from app.models.one_to_one_deployment import OneToOneDeployment
|
from app.models.one_to_one_deployment import OneToOneDeployment
|
||||||
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
@@ -107,11 +107,37 @@ async def tls_probe(payload: TlsProbeRequest):
|
|||||||
raise api_error(502, ErrorCode.TLS_PROBE_FAILED)
|
raise api_error(502, ErrorCode.TLS_PROBE_FAILED)
|
||||||
|
|
||||||
|
|
||||||
|
def _same_url(a: str, b: str) -> bool:
|
||||||
|
return a.strip().rstrip("/").lower() == b.strip().rstrip("/").lower()
|
||||||
|
|
||||||
|
|
||||||
|
async def _ensure_url_free(db: AsyncSession, base_url: str, exclude_id: uuid.UUID | None = None) -> None:
|
||||||
|
"""One firewall, one entry: the same address twice would double every rollout, alert and backup (T4)."""
|
||||||
|
for other in (await db.scalars(select(OpnsenseHost))).all():
|
||||||
|
if other.id != exclude_id and _same_url(other.base_url, base_url):
|
||||||
|
raise api_error(409, ErrorCode.HOST_URL_EXISTS, host=other.name)
|
||||||
|
|
||||||
|
|
||||||
|
def _connection_failure_reason(exc: Exception) -> str:
|
||||||
|
"""A short machine code for the UI (hosts.connection.<code>): why the first call to the firewall failed."""
|
||||||
|
text = str(exc)
|
||||||
|
if isinstance(exc, opnsense_client.OpnsenseForbiddenError):
|
||||||
|
return "forbidden"
|
||||||
|
if text.startswith("401 ") or text.startswith("403 "):
|
||||||
|
return "credentials" if text.startswith("401 ") else "forbidden"
|
||||||
|
if "TLS-Zertifikat" in text or "CERTIFICATE_VERIFY_FAILED" in text:
|
||||||
|
return "tls"
|
||||||
|
if "Verbindung zu" in text:
|
||||||
|
return "unreachable"
|
||||||
|
return "other"
|
||||||
|
|
||||||
|
|
||||||
@router.post("", response_model=HostRead, status_code=201)
|
@router.post("", response_model=HostRead, status_code=201)
|
||||||
async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
||||||
existing = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == payload.name))
|
existing = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == payload.name))
|
||||||
if existing:
|
if existing:
|
||||||
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
||||||
|
await _ensure_url_free(db, payload.base_url)
|
||||||
if payload.policy_hierarchy_group_id and not await db.get(PolicyHierarchyGroup, payload.policy_hierarchy_group_id):
|
if payload.policy_hierarchy_group_id and not await db.get(PolicyHierarchyGroup, payload.policy_hierarchy_group_id):
|
||||||
raise api_error(400, ErrorCode.POLICY_HIERARCHIE_GRUPPE_NICHT_GEFUNDEN)
|
raise api_error(400, ErrorCode.POLICY_HIERARCHIE_GRUPPE_NICHT_GEFUNDEN)
|
||||||
|
|
||||||
@@ -130,6 +156,14 @@ async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
|||||||
db.add(host)
|
db.add(host)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
opnsense_secret_store.save_secret(host.id, payload.api_secret)
|
opnsense_secret_store.save_secret(host.id, payload.api_secret)
|
||||||
|
if payload.verify_connection:
|
||||||
|
# The call that decides "online" in the host list: if it works here, the host shows up online right away.
|
||||||
|
try:
|
||||||
|
await opnsense_client.get_vip_status(host)
|
||||||
|
except opnsense_client.OpnsenseAPIError as exc:
|
||||||
|
await db.rollback()
|
||||||
|
opnsense_secret_store.delete_secret(host.id)
|
||||||
|
raise api_error(422, ErrorCode.HOST_CONNECTION_FAILED, reason=_connection_failure_reason(exc)) from exc
|
||||||
await db.commit()
|
await db.commit()
|
||||||
|
|
||||||
statuses = await get_statuses([host])
|
statuses = await get_statuses([host])
|
||||||
@@ -147,6 +181,11 @@ async def get_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
|||||||
async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession = Depends(get_db)):
|
async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession = Depends(get_db)):
|
||||||
await policy_lock(db)
|
await policy_lock(db)
|
||||||
host = await _get_host_or_404(db, host_id)
|
host = await _get_host_or_404(db, host_id)
|
||||||
|
if (payload.base_url is not None or payload.enabled is not None or payload.clear_cluster_name
|
||||||
|
or payload.cluster_name is not None):
|
||||||
|
# An open rule cleanup (1.31) reverts through this address and this cluster pairing: not while it is open.
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [host])
|
||||||
|
|
||||||
if payload.name is not None:
|
if payload.name is not None:
|
||||||
existing = await db.scalar(
|
existing = await db.scalar(
|
||||||
@@ -156,6 +195,7 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
|
|||||||
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
||||||
host.name = payload.name
|
host.name = payload.name
|
||||||
if payload.base_url is not None:
|
if payload.base_url is not None:
|
||||||
|
await _ensure_url_free(db, payload.base_url, exclude_id=host_id)
|
||||||
host.base_url = payload.base_url
|
host.base_url = payload.base_url
|
||||||
if payload.verify_tls is not None:
|
if payload.verify_tls is not None:
|
||||||
host.verify_tls = payload.verify_tls
|
host.verify_tls = payload.verify_tls
|
||||||
@@ -183,11 +223,18 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
|
|||||||
|
|
||||||
|
|
||||||
@router.delete("/{host_id}", status_code=204)
|
@router.delete("/{host_id}", status_code=204)
|
||||||
async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
async def delete_host(host_id: uuid.UUID, confirm_name: str = Query(default="", max_length=120), db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Deleting a host drops its stored credentials at once, so the caller must name it (like every other destructive
|
||||||
|
action in OPNexus), not only a click in the UI: `DELETE /api/hosts/{id}?confirm_name=<name>` (T4)."""
|
||||||
await policy_lock(db)
|
await policy_lock(db)
|
||||||
host = await db.get(OpnsenseHost, host_id)
|
host = await db.get(OpnsenseHost, host_id)
|
||||||
if not host:
|
if not host:
|
||||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
if confirm_name != host.name:
|
||||||
|
raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||||
|
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [host])
|
||||||
# Both deployment tables FK to opnsense_hosts with ondelete=CASCADE, so
|
# Both deployment tables FK to opnsense_hosts with ondelete=CASCADE, so
|
||||||
# deleting the host would silently drop OPNexus's only record of what
|
# deleting the host would silently drop OPNexus's only record of what
|
||||||
# it pushed there — the rule/alias itself stays live on the actual
|
# it pushed there — the rule/alias itself stays live on the actual
|
||||||
@@ -226,6 +273,12 @@ async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
|||||||
draft_certs = (await db.scalars(select(ManagedCertificate.id).where(ManagedCertificate.host_id == host_id))).all()
|
draft_certs = (await db.scalars(select(ManagedCertificate.id).where(ManagedCertificate.host_id == host_id))).all()
|
||||||
draft_accounts = (await db.scalars(select(AcmeAccount.id).where(AcmeAccount.host_id == host_id))).all()
|
draft_accounts = (await db.scalars(select(AcmeAccount.id).where(AcmeAccount.host_id == host_id))).all()
|
||||||
await db.delete(host)
|
await db.delete(host)
|
||||||
|
# Gateway alert switches of this host would otherwise stay in the settings forever (and count against the limit).
|
||||||
|
from app.services import notification_settings_service
|
||||||
|
settings_row = await notification_settings_service.get_settings(db)
|
||||||
|
prefix = f"{host_id}:"
|
||||||
|
if any(str(m).startswith(prefix) for m in settings_row.gateway_alert_muted or []):
|
||||||
|
settings_row.gateway_alert_muted = [m for m in settings_row.gateway_alert_muted if not str(m).startswith(prefix)]
|
||||||
await db.commit()
|
await db.commit()
|
||||||
opnsense_secret_store.delete_secret(host_id)
|
opnsense_secret_store.delete_secret(host_id)
|
||||||
for cert_id in draft_certs:
|
for cert_id in draft_certs:
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ from app.services import ipsec_psk_store
|
|||||||
from app.services.ipsec_service import effective_hosts, preview, revision, snapshot, workflow, IPSEC_FIELDS
|
from app.services.ipsec_service import effective_hosts, preview, revision, snapshot, workflow, IPSEC_FIELDS
|
||||||
from app.services.ipsec_deploy_service import deploy_tunnel, undeploy_tunnel
|
from app.services.ipsec_deploy_service import deploy_tunnel, undeploy_tunnel
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
from app.services.deploy_lease import resource_lease
|
||||||
from app.services.policy_service import policy_lock
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
router = APIRouter(prefix="/ipsec/tunnels", tags=["ipsec"])
|
router = APIRouter(prefix="/ipsec/tunnels", tags=["ipsec"])
|
||||||
@@ -192,24 +193,27 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
|||||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
async with resource_lease("ipsec", tunnel.id):
|
||||||
from app.services import opnsense_client
|
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||||
try:
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
if deployment.opnsense_child_uuid:
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
|
from app.services import opnsense_client
|
||||||
if deployment.opnsense_local_uuid:
|
try:
|
||||||
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
|
if deployment.opnsense_child_uuid:
|
||||||
if deployment.opnsense_remote_uuid:
|
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
|
||||||
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
|
if deployment.opnsense_local_uuid:
|
||||||
if deployment.opnsense_connection_uuid:
|
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
|
||||||
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
|
if deployment.opnsense_remote_uuid:
|
||||||
if deployment.opnsense_psk_uuid:
|
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
|
||||||
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
|
if deployment.opnsense_connection_uuid:
|
||||||
await opnsense_client.reconfigure_ipsec(deployment.host)
|
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
|
||||||
except OpnsenseAPIError as exc:
|
if deployment.opnsense_psk_uuid:
|
||||||
deployment.status = "error"
|
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
|
||||||
deployment.message = str(exc)
|
await opnsense_client.reconfigure_ipsec(deployment.host)
|
||||||
|
except OpnsenseAPIError as exc:
|
||||||
|
deployment.status = "error"
|
||||||
|
deployment.message = str(exc)
|
||||||
|
await db.commit()
|
||||||
|
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||||
|
await db.delete(deployment)
|
||||||
await db.commit()
|
await db.commit()
|
||||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
|
||||||
await db.delete(deployment)
|
|
||||||
await db.commit()
|
|
||||||
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
|
|||||||
from app.database import get_db
|
from app.database import get_db
|
||||||
from app.models import OpnsenseHost
|
from app.models import OpnsenseHost
|
||||||
from app.models.host_operation import HostOperation
|
from app.models.host_operation import HostOperation
|
||||||
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
|
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
|
||||||
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
|
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
|
||||||
from app.services.policy_service import policy_lock
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
|
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
|
||||||
|
|
||||||
class Preview(BaseModel):
|
class Preview(BaseModel):
|
||||||
action: Literal['reboot','update','major_update']
|
action: Literal['reboot','update','major_update','firmware_rollback']
|
||||||
|
|
||||||
class Execute(BaseModel):
|
class Execute(BaseModel):
|
||||||
preview_id: uuid.UUID
|
preview_id: uuid.UUID
|
||||||
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
|
|||||||
def read_operation(op):
|
def read_operation(op):
|
||||||
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
|
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
|
||||||
|
|
||||||
async def snapshot_or_502(db, host):
|
async def snapshot_or_502(db, host, action=None):
|
||||||
try: return await inspect_host(db,host)
|
try:
|
||||||
|
state=await inspect_host(db,host)
|
||||||
|
if action=='firmware_rollback':
|
||||||
|
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
|
||||||
|
rollback=await snapshot_service.rollback_state(host)
|
||||||
|
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
|
||||||
|
state['rollback']=rollback
|
||||||
|
state['actions']['firmware_rollback']=state['actions']['reboot']+(
|
||||||
|
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
|
||||||
|
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
|
||||||
|
return state
|
||||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||||
|
|
||||||
|
|
||||||
|
async def _changes_since(db, host, target):
|
||||||
|
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
|
||||||
|
if not target:
|
||||||
|
return []
|
||||||
|
since=_point_time(target)
|
||||||
|
if since is None:
|
||||||
|
return []
|
||||||
|
out=[]
|
||||||
|
for kind,model in _DEPLOYMENT_MODELS():
|
||||||
|
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
|
||||||
|
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
|
||||||
|
return sorted(out,key=lambda r:r['deployed_at'])[:50]
|
||||||
|
|
||||||
|
|
||||||
|
def _point_time(target):
|
||||||
|
try:
|
||||||
|
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
|
||||||
|
except (KeyError, TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _DEPLOYMENT_MODELS():
|
||||||
|
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
|
||||||
|
PortForwardDeployment, RuleDeployment)
|
||||||
|
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||||
|
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||||
|
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||||
|
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
|
||||||
|
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
|
||||||
|
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
|
||||||
|
|
||||||
|
|
||||||
|
async def _mark_rolled_back(db, host, target):
|
||||||
|
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
|
||||||
|
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
|
||||||
|
really writes, and the UI shows the object as not in sync."""
|
||||||
|
since=_point_time(target)
|
||||||
|
if since is None:
|
||||||
|
return 0
|
||||||
|
count=0
|
||||||
|
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
|
||||||
|
for _kind,model in _DEPLOYMENT_MODELS():
|
||||||
|
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
|
||||||
|
row.status='error'; row.message=note[:500]; row.last_payload=None
|
||||||
|
if hasattr(model,'revert_deadline'): row.revert_deadline=None
|
||||||
|
count+=1
|
||||||
|
return count
|
||||||
|
|
||||||
async def carp_snapshot_or_502(db, host):
|
async def carp_snapshot_or_502(db, host):
|
||||||
try: return await inspect_carp_switch(db,host)
|
try: return await inspect_carp_switch(db,host)
|
||||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||||
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
|||||||
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
|
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
await policy_lock(db)
|
await policy_lock(db)
|
||||||
host=await host_or_404(db,host_id)
|
host=await host_or_404(db,host_id)
|
||||||
state=await snapshot_or_502(db,host)
|
state=await snapshot_or_502(db,host,payload.action)
|
||||||
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
|
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
|
||||||
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
|
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
|
||||||
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
|
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
|
||||||
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
|
|||||||
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||||
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
|
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
|
||||||
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||||
state=await snapshot_or_502(db,host)
|
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||||
|
state=await snapshot_or_502(db,host,op.action)
|
||||||
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
|
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
|
||||||
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
|
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
|
||||||
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
|
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
|
||||||
# replaying a reboot/update automatically could interrupt recovery.
|
# replaying a reboot/update automatically could interrupt recovery.
|
||||||
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
|
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
|
||||||
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
|
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
|
||||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
if op.action in ('update','major_update'):
|
||||||
|
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
|
||||||
|
try:
|
||||||
|
point=await snapshot_service.create_rollback_point(host,state.get('version'))
|
||||||
|
except client.OpnsenseAPIError as exc:
|
||||||
|
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
|
||||||
|
await db.commit()
|
||||||
|
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||||
|
return read_operation(op)
|
||||||
|
op.snapshot={**op.snapshot,'rollback_point':point}
|
||||||
|
await db.commit()
|
||||||
|
if op.action=='firmware_rollback':
|
||||||
|
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
|
||||||
|
# Fail closed, like the backup before an update.
|
||||||
|
try:
|
||||||
|
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||||
|
except client.OpnsenseAPIError as exc:
|
||||||
|
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
|
||||||
|
await db.commit()
|
||||||
|
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||||
|
return read_operation(op)
|
||||||
|
rollback=state['rollback']
|
||||||
|
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
|
||||||
|
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
|
||||||
|
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
|
||||||
|
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
|
||||||
|
else:
|
||||||
|
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||||
|
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||||
|
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
|
||||||
|
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
|
||||||
try:
|
try:
|
||||||
result=await action_fn(host)
|
result=await action_fn(host)
|
||||||
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
op.status='submitted';op.message=done
|
||||||
|
if op.action=='firmware_rollback':
|
||||||
|
marked=await _mark_rolled_back(db,host,rollback['target'])
|
||||||
|
if marked:
|
||||||
|
op.message+=f' {marked} Deployment(s) als unklar markiert.'
|
||||||
except client.OpnsenseAPIError as exc:
|
except client.OpnsenseAPIError as exc:
|
||||||
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
|
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
|
||||||
result={}
|
result={}
|
||||||
|
|||||||
@@ -113,6 +113,23 @@ TOOLS = [
|
|||||||
# The per-rule trace is for the UI; the verdict, deciding rule and reasons are what matters here.
|
# The per-rule trace is for the UI; the verdict, deciding rule and reasons are what matters here.
|
||||||
"drop": ("trace",),
|
"drop": ("trace",),
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "rule_cleanup_candidates",
|
||||||
|
"description": "Cleanup candidates of one firewall: rules that are unused (no hit for N days), shadowed or "
|
||||||
|
"duplicate (an earlier quick rule already decides everything they match), or disabled for a "
|
||||||
|
"long time. Each with its confidence (sure/probable/unknown), the last observed hit and "
|
||||||
|
"whether it could be removed (not for automatic, legacy or OPNexus-managed rules). Reads only.",
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object", "required": ["host_id"],
|
||||||
|
"properties": {
|
||||||
|
"host_id": {"type": "string", "description": "Firewall id from list_hosts"},
|
||||||
|
"unused_days": {"type": "integer", "minimum": 1, "maximum": 3650, "default": 30,
|
||||||
|
"description": "observation window for unused / disabled rules"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"path": "/api/hosts/{host_id}/rule-analysis/cleanup",
|
||||||
|
"query": ("unused_days",),
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"name": "api_get",
|
"name": "api_get",
|
||||||
"description": "Read any other OPNexus API endpoint (GET only), e.g. /api/rules, /api/aliases, /api/logs. "
|
"description": "Read any other OPNexus API endpoint (GET only), e.g. /api/rules, /api/aliases, /api/logs. "
|
||||||
|
|||||||
@@ -22,6 +22,28 @@ logger = logging.getLogger(__name__)
|
|||||||
router = APIRouter(tags=["oidc"])
|
router = APIRouter(tags=["oidc"])
|
||||||
CALLBACK_PATH = "/api/auth/oidc/callback"
|
CALLBACK_PATH = "/api/auth/oidc/callback"
|
||||||
STATE_COOKIE = "opnexus-oidc-state"
|
STATE_COOKIE = "opnexus-oidc-state"
|
||||||
|
# HTTPS deployments (1.29): __Host- (Secure, no Domain, Path=/) so that no sibling SUBDOMAIN or plain-HTTP page can
|
||||||
|
# plant a state cookie of its own (login CSRF with the attacker's own authorisation). Known limit: cookies are not
|
||||||
|
# separated by PORT, so script in a firewall GUI shown on a gateway slot port of the same host could still set it
|
||||||
|
# (BACKLOG S8, accepted: needs a compromised firewall GUI and a mapped IdP account; the real fix is one hostname
|
||||||
|
# per slot). The browser only accepts the prefix with Path=/, so the cookie travels with every request for its
|
||||||
|
# 10 minutes; plain-HTTP labs keep the narrow path.
|
||||||
|
SECURE_STATE_COOKIE = "__Host-opnexus-oidc-state"
|
||||||
|
STATE_COOKIE_PATH = "/api/auth/oidc"
|
||||||
|
|
||||||
|
|
||||||
|
def state_cookie_name() -> str:
|
||||||
|
return SECURE_STATE_COOKIE if settings.cookie_secure else STATE_COOKIE
|
||||||
|
|
||||||
|
|
||||||
|
def _state_cookie_path() -> str:
|
||||||
|
return "/" if settings.cookie_secure else STATE_COOKIE_PATH
|
||||||
|
|
||||||
|
|
||||||
|
def _clear_state_cookie(response) -> None:
|
||||||
|
response.delete_cookie(state_cookie_name(), path=_state_cookie_path(), secure=settings.cookie_secure, httponly=True, samesite="lax")
|
||||||
|
if settings.cookie_secure: # a plain-name cookie of an earlier version
|
||||||
|
response.delete_cookie(STATE_COOKIE, path=STATE_COOKIE_PATH, secure=True, httponly=True, samesite="lax")
|
||||||
# SSO sessions are short: the provider's group membership is only re-checked at login, so a removed person
|
# SSO sessions are short: the provider's group membership is only re-checked at login, so a removed person
|
||||||
# must not keep a 30-day session.
|
# must not keep a 30-day session.
|
||||||
SSO_SESSION_SECONDS = 12 * 3600
|
SSO_SESSION_SECONDS = 12 * 3600
|
||||||
@@ -29,7 +51,7 @@ SSO_SESSION_SECONDS = 12 * 3600
|
|||||||
|
|
||||||
def _login_error(code: str) -> RedirectResponse:
|
def _login_error(code: str) -> RedirectResponse:
|
||||||
response = RedirectResponse(f"/login?sso_error={code}", status_code=302)
|
response = RedirectResponse(f"/login?sso_error={code}", status_code=302)
|
||||||
response.delete_cookie(STATE_COOKIE, path="/api/auth/oidc")
|
_clear_state_cookie(response)
|
||||||
return response
|
return response
|
||||||
|
|
||||||
|
|
||||||
@@ -63,7 +85,8 @@ async def start(request: Request, origin: str, db: AsyncSession = Depends(get_db
|
|||||||
response = RedirectResponse(url, status_code=302)
|
response = RedirectResponse(url, status_code=302)
|
||||||
# Binds the attempt to THIS browser: a callback URL handed to somebody else (login CSRF) or a captured
|
# Binds the attempt to THIS browser: a callback URL handed to somebody else (login CSRF) or a captured
|
||||||
# code+state pair is useless without the cookie.
|
# code+state pair is useless without the cookie.
|
||||||
response.set_cookie(STATE_COOKIE, state, max_age=600, httponly=True, samesite="lax", secure=settings.cookie_secure, path="/api/auth/oidc")
|
response.set_cookie(state_cookie_name(), state, max_age=600, httponly=True, samesite="lax", secure=settings.cookie_secure,
|
||||||
|
path=_state_cookie_path())
|
||||||
return response
|
return response
|
||||||
|
|
||||||
|
|
||||||
@@ -78,7 +101,9 @@ async def callback(request: Request, code: str = "", state: str = "", error: str
|
|||||||
try:
|
try:
|
||||||
if error or not code or not state:
|
if error or not code or not state:
|
||||||
raise oidc_service.OidcError("provider", "Anbieter meldet " + (_printable(error) if error else "keinen Code"))
|
raise oidc_service.OidcError("provider", "Anbieter meldet " + (_printable(error) if error else "keinen Code"))
|
||||||
if not secrets.compare_digest(request.cookies.get(STATE_COOKIE, ""), state):
|
# Only the cookie name of this deployment counts (on HTTPS never the plain one); no cookie never matches.
|
||||||
|
bound = request.cookies.get(state_cookie_name(), "")
|
||||||
|
if not bound or not secrets.compare_digest(bound.encode(), state.encode()):
|
||||||
raise oidc_service.OidcError("state", "Der Anmeldeversuch gehört nicht zu diesem Browser")
|
raise oidc_service.OidcError("state", "Der Anmeldeversuch gehört nicht zu diesem Browser")
|
||||||
user = await oidc_service.complete(db, cfg, code, state)
|
user = await oidc_service.complete(db, cfg, code, state)
|
||||||
except oidc_service.OidcError as exc:
|
except oidc_service.OidcError as exc:
|
||||||
@@ -92,7 +117,7 @@ async def callback(request: Request, code: str = "", state: str = "", error: str
|
|||||||
)
|
)
|
||||||
response = RedirectResponse("/dashboard", status_code=302)
|
response = RedirectResponse("/dashboard", status_code=302)
|
||||||
_set_session_cookie(response, token, max_age=SSO_SESSION_SECONDS)
|
_set_session_cookie(response, token, max_age=SSO_SESSION_SECONDS)
|
||||||
response.delete_cookie(STATE_COOKIE, path="/api/auth/oidc")
|
_clear_state_cookie(response)
|
||||||
logger.info("SSO-Anmeldung von '%s' (%s) aus %s", user.username, user.role, ip)
|
logger.info("SSO-Anmeldung von '%s' (%s) aus %s", user.username, user.role, ip)
|
||||||
return response
|
return response
|
||||||
|
|
||||||
|
|||||||
@@ -50,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
|||||||
workflow_state=state,
|
workflow_state=state,
|
||||||
workflow_counts=counts,
|
workflow_counts=counts,
|
||||||
id=rule.id,
|
id=rule.id,
|
||||||
|
config_key=rule.config_key,
|
||||||
description=rule.description,
|
description=rule.description,
|
||||||
interface=rule.interface,
|
interface=rule.interface,
|
||||||
source_net=rule.source_net,
|
source_net=rule.source_net,
|
||||||
@@ -231,6 +232,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
|||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
try:
|
try:
|
||||||
await opnsense_client.del_one_to_one_rule(deployment.host, deployment.opnsense_one_to_one_uuid)
|
await opnsense_client.del_one_to_one_rule(deployment.host, deployment.opnsense_one_to_one_uuid)
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ from app.services import openvpn_key_store
|
|||||||
from app.services.openvpn_service import effective_hosts, preview, revision, snapshot, workflow, OPENVPN_FIELDS
|
from app.services.openvpn_service import effective_hosts, preview, revision, snapshot, workflow, OPENVPN_FIELDS
|
||||||
from app.services.openvpn_deploy_service import deploy_instance, undeploy_instance
|
from app.services.openvpn_deploy_service import deploy_instance, undeploy_instance
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
from app.services.deploy_lease import resource_lease
|
||||||
from app.services.policy_service import policy_lock
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
router = APIRouter(prefix="/openvpn/instances", tags=["openvpn"])
|
router = APIRouter(prefix="/openvpn/instances", tags=["openvpn"])
|
||||||
@@ -194,18 +195,21 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
|
|||||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
async with resource_lease("openvpn", instance.id):
|
||||||
from app.services import opnsense_client
|
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||||
try:
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
if deployment.opnsense_instance_uuid:
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
|
from app.services import opnsense_client
|
||||||
if deployment.opnsense_statickey_uuid:
|
try:
|
||||||
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
|
if deployment.opnsense_instance_uuid:
|
||||||
await opnsense_client.reconfigure_openvpn(deployment.host)
|
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
|
||||||
except OpnsenseAPIError as exc:
|
if deployment.opnsense_statickey_uuid:
|
||||||
deployment.status = "error"
|
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
|
||||||
deployment.message = str(exc)
|
await opnsense_client.reconfigure_openvpn(deployment.host)
|
||||||
|
except OpnsenseAPIError as exc:
|
||||||
|
deployment.status = "error"
|
||||||
|
deployment.message = str(exc)
|
||||||
|
await db.commit()
|
||||||
|
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||||
|
await db.delete(deployment)
|
||||||
await db.commit()
|
await db.commit()
|
||||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
|
||||||
await db.delete(deployment)
|
|
||||||
await db.commit()
|
|
||||||
@@ -51,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
|||||||
workflow_state=state,
|
workflow_state=state,
|
||||||
workflow_counts=counts,
|
workflow_counts=counts,
|
||||||
id=rule.id,
|
id=rule.id,
|
||||||
|
config_key=rule.config_key,
|
||||||
description=rule.description,
|
description=rule.description,
|
||||||
interface=rule.interface,
|
interface=rule.interface,
|
||||||
source_net=rule.source_net,
|
source_net=rule.source_net,
|
||||||
@@ -238,6 +239,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
|||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
try:
|
try:
|
||||||
await opnsense_client.del_outbound_nat_rule(deployment.host, deployment.opnsense_outbound_nat_uuid)
|
await opnsense_client.del_outbound_nat_rule(deployment.host, deployment.opnsense_outbound_nat_uuid)
|
||||||
|
|||||||
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
|
|||||||
workflow_state=state,
|
workflow_state=state,
|
||||||
workflow_counts=counts,
|
workflow_counts=counts,
|
||||||
id=rule.id,
|
id=rule.id,
|
||||||
|
config_key=rule.config_key,
|
||||||
hierarchy_group_id=rule.hierarchy_group_id,
|
hierarchy_group_id=rule.hierarchy_group_id,
|
||||||
rulebase=rule.rulebase,
|
rulebase=rule.rulebase,
|
||||||
sequence=rule.sequence,
|
sequence=rule.sequence,
|
||||||
@@ -330,6 +331,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
|||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
try:
|
try:
|
||||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_policy_hierarchy_uuid)
|
await opnsense_client.del_rule(deployment.host, deployment.opnsense_policy_hierarchy_uuid)
|
||||||
await opnsense_client.apply(deployment.host)
|
await opnsense_client.apply(deployment.host)
|
||||||
|
|||||||
@@ -50,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
|||||||
workflow_state=state,
|
workflow_state=state,
|
||||||
workflow_counts=counts,
|
workflow_counts=counts,
|
||||||
id=rule.id,
|
id=rule.id,
|
||||||
|
config_key=rule.config_key,
|
||||||
description=rule.description,
|
description=rule.description,
|
||||||
interface=rule.interface,
|
interface=rule.interface,
|
||||||
ipprotocol=rule.ipprotocol,
|
ipprotocol=rule.ipprotocol,
|
||||||
@@ -233,6 +234,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
|||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
try:
|
try:
|
||||||
await opnsense_client.del_port_forward_rule(deployment.host, deployment.opnsense_port_forward_uuid)
|
await opnsense_client.del_port_forward_rule(deployment.host, deployment.opnsense_port_forward_uuid)
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
|
|||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, Request
|
from fastapi import APIRouter, Depends, Request
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field, model_validator
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
@@ -20,7 +20,7 @@ from app.models.maintenance_window import MaintenanceWindow
|
|||||||
from app.models.policy import DeviceGroup
|
from app.models.policy import DeviceGroup
|
||||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||||
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
|
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
|
||||||
from app.services.policy_service import policy_lock
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||||
@@ -36,6 +36,19 @@ class WaveIn(BaseModel):
|
|||||||
require_approval: bool = False
|
require_approval: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class ProbeIn(BaseModel):
|
||||||
|
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
|
||||||
|
type: Literal["ping", "tcp"]
|
||||||
|
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
|
||||||
|
port: int | None = Field(default=None, ge=1, le=65535)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def _port_only_for_tcp(self):
|
||||||
|
if (self.type == "tcp") != (self.port is not None):
|
||||||
|
raise ValueError("port_required_for_tcp_only")
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
class PlanIn(BaseModel):
|
class PlanIn(BaseModel):
|
||||||
name: str = Field(min_length=1, max_length=120)
|
name: str = Field(min_length=1, max_length=120)
|
||||||
description: str = Field(default="", max_length=2000)
|
description: str = Field(default="", max_length=2000)
|
||||||
@@ -43,6 +56,7 @@ class PlanIn(BaseModel):
|
|||||||
window_id: uuid.UUID | None = None
|
window_id: uuid.UUID | None = None
|
||||||
auto_start: bool = False
|
auto_start: bool = False
|
||||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||||
|
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
|
||||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||||
|
|
||||||
@@ -64,6 +78,7 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
|||||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||||
return {
|
return {
|
||||||
**rollout_service.definition(plan),
|
**rollout_service.definition(plan),
|
||||||
|
"gate_probes": list(plan.gate_probes or []),
|
||||||
"description": plan.description,
|
"description": plan.description,
|
||||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||||
@@ -81,6 +96,15 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
|
||||||
|
out: list[dict] = []
|
||||||
|
for probe in probes:
|
||||||
|
value = {"type": probe.type, "target": probe.target, "port": probe.port}
|
||||||
|
if value not in out:
|
||||||
|
out.append(value)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||||
name = payload.name.strip()
|
name = payload.name.strip()
|
||||||
if not name:
|
if not name:
|
||||||
@@ -99,6 +123,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
|||||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||||
|
plan.gate_probes = _unique_probes(payload.gate_probes)
|
||||||
plan.compliance_mode = payload.compliance_mode
|
plan.compliance_mode = payload.compliance_mode
|
||||||
plan.waves = [
|
plan.waves = [
|
||||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||||
@@ -106,7 +131,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
|||||||
for i, w in enumerate(payload.waves)
|
for i, w in enumerate(payload.waves)
|
||||||
]
|
]
|
||||||
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
|
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
|
||||||
plan.activated_at = plan.activated_by = plan.activated_fingerprint = None
|
plan.activated_at = plan.activated_by = plan.activated_by_id = plan.activated_fingerprint = None
|
||||||
plan.auto_start_error = None
|
plan.auto_start_error = None
|
||||||
|
|
||||||
|
|
||||||
@@ -180,6 +205,7 @@ async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request,
|
|||||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
|
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
|
||||||
_resolved, current = await _checked_token(db, plan, payload.preview_token)
|
_resolved, current = await _checked_token(db, plan, payload.preview_token)
|
||||||
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
|
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
|
||||||
|
plan.activated_by_id = request.state.user.id
|
||||||
plan.auto_start_error = None
|
plan.auto_start_error = None
|
||||||
await db.commit()
|
await db.commit()
|
||||||
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
|
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
|
||||||
@@ -191,7 +217,7 @@ async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request,
|
|||||||
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||||
await policy_lock(db)
|
await policy_lock(db)
|
||||||
plan = await _plan_or_404(db, plan_id)
|
plan = await _plan_or_404(db, plan_id)
|
||||||
plan.activated_by = plan.activated_at = plan.activated_fingerprint = None
|
plan.activated_by = plan.activated_by_id = plan.activated_at = plan.activated_fingerprint = None
|
||||||
await db.commit()
|
await db.commit()
|
||||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
|
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
|
||||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||||
@@ -306,7 +332,8 @@ async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db:
|
|||||||
if (other := await rollout_engine.active_run(db)) is not None:
|
if (other := await rollout_engine.active_run(db)) is not None:
|
||||||
# One run at a time: two runs could otherwise touch the same cluster at once.
|
# One run at a time: two runs could otherwise touch the same cluster at once.
|
||||||
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
|
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
|
||||||
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username, trigger="manual")
|
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username,
|
||||||
|
started_by_id=request.state.user.id, trigger="manual")
|
||||||
await db.commit()
|
await db.commit()
|
||||||
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
|
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
|
||||||
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
|
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||||
@@ -363,7 +390,7 @@ async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Dep
|
|||||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
|
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
|
||||||
previous = run.started_by
|
previous = run.started_by
|
||||||
# Whoever resumes takes over the responsibility (and the permission check) from here on.
|
# Whoever resumes takes over the responsibility (and the permission check) from here on.
|
||||||
run.started_by = request.state.user.username
|
run.started_by, run.started_by_id = request.state.user.username, request.state.user.id
|
||||||
run.status = "resuming"
|
run.status = "resuming"
|
||||||
run.pause_reason = run.pause_detail = None
|
run.pause_reason = run.pause_detail = None
|
||||||
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
|
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
|
||||||
@@ -378,7 +405,7 @@ async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = D
|
|||||||
raise _not_now(run)
|
raise _not_now(run)
|
||||||
wave = rollout_engine.wave_row(run, run.current_wave)
|
wave = rollout_engine.wave_row(run, run.current_wave)
|
||||||
wave.approved_by, wave.status = request.state.user.username, "preflight"
|
wave.approved_by, wave.status = request.state.user.username, "preflight"
|
||||||
run.started_by = request.state.user.username
|
run.started_by, run.started_by_id = request.state.user.username, request.state.user.id
|
||||||
run.status = "preflight"
|
run.status = "preflight"
|
||||||
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
|
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
|
||||||
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
|
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
|
||||||
|
|||||||
@@ -17,6 +17,8 @@ from app.database import get_db
|
|||||||
from app.errors import ErrorCode, api_error
|
from app.errors import ErrorCode, api_error
|
||||||
from app.models.host import OpnsenseHost
|
from app.models.host import OpnsenseHost
|
||||||
from app.models.policy import DeviceGroup
|
from app.models.policy import DeviceGroup
|
||||||
|
from app.models.rule_history import LiveRuleState
|
||||||
|
from app.services import rule_history_service
|
||||||
from app.services.rule_analysis import service
|
from app.services.rule_analysis import service
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -89,3 +91,38 @@ async def fleet_match(
|
|||||||
return await _run(host, q, refresh)
|
return await _run(host, q, refresh)
|
||||||
|
|
||||||
return list(await asyncio.gather(*(one(h) for h in hosts)))
|
return list(await asyncio.gather(*(one(h) for h in hosts)))
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/hosts/{host_id}/rule-analysis/cleanup")
|
||||||
|
async def host_cleanup(
|
||||||
|
host_id: uuid.UUID, unused_days: int = Query(30, ge=1, le=3650, description="observation window for 'unused' / 'disabled for a long time'"),
|
||||||
|
refresh: bool = False, db: AsyncSession = Depends(get_db),
|
||||||
|
):
|
||||||
|
"""Cleanup candidates (unused, shadowed, duplicate, disabled for a long time) with confidence and whether
|
||||||
|
each rule could be offered for removal. Read-only; removal is not part of this endpoint."""
|
||||||
|
host = await db.get(OpnsenseHost, host_id)
|
||||||
|
if host is None:
|
||||||
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
try:
|
||||||
|
return await service.cleanup_candidates(db, host, unused_days=unused_days, refresh=refresh)
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as exc: # noqa: BLE001 - an unreachable firewall is a 502, not a crash
|
||||||
|
logger.info("cleanup candidates failed for %s: %s", host.name, exc)
|
||||||
|
raise api_error(502, ErrorCode.ANALYSE_FIREWALL_NICHT_LESBAR) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/hosts/{host_id}/live-rules/{rule_uuid}/history")
|
||||||
|
async def live_rule_history(host_id: uuid.UUID, rule_uuid: str, limit: int = Query(100, ge=1, le=500),
|
||||||
|
db: AsyncSession = Depends(get_db)):
|
||||||
|
"""What happened to one rule on one firewall: observed changes (also hand-made ones), saved revisions of
|
||||||
|
rules OPNexus manages and the audit entries, newest first. Reads the database only."""
|
||||||
|
host = await db.get(OpnsenseHost, host_id)
|
||||||
|
if host is None:
|
||||||
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
baseline = await db.get(LiveRuleState, (host_id, rule_history_service.BASELINE_KEY))
|
||||||
|
return {
|
||||||
|
"host_id": str(host.id), "rule_uuid": rule_uuid.lower()[:64],
|
||||||
|
"watching_since": baseline.first_seen_at if baseline else None,
|
||||||
|
"events": await rule_history_service.rule_timeline(db, host, rule_uuid, limit),
|
||||||
|
}
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
"""Rule cleanup (1.31, B4 part 2): switch hand-made rules off, delete them later, restore them (DESIGN-1.31 §4).
|
||||||
|
|
||||||
|
POST /hosts/{id}/rule-cleanup/preview operator, admin prepare (writes nothing to a firewall)
|
||||||
|
POST /hosts/{id}/rule-cleanup/{op}/cancel operator, admin the preview's own actor
|
||||||
|
POST /hosts/{id}/rule-cleanup/{op}/deploy ADMIN, token scope admin
|
||||||
|
POST /hosts/{id}/rule-cleanup/{op}/confirm operator, admin BROWSER SESSION ONLY, never a token
|
||||||
|
POST /hosts/{id}/rule-cleanup/{op}/revert admin; operator only with operators_may_deploy
|
||||||
|
POST /hosts/{id}/rule-cleanup/{op}/retry ADMIN, token scope admin
|
||||||
|
POST /hosts/{id}/rule-cleanup/items/{item}/restore ADMIN, token scope admin
|
||||||
|
GET /hosts/{id}/rule-cleanup, /{op} everybody signed in (read tokens too)
|
||||||
|
|
||||||
|
Who may do what is enforced twice: AuthMiddleware (services/api_token_service.py: _ADMIN_ONLY_CLEANUP,
|
||||||
|
_SESSION_ONLY, _ROLLOUT_POST) and here, so a gap in one layer does not open the route. The audit trail is the
|
||||||
|
service's (one ``rule_cleanup`` entry per step and rule) plus the middleware's request entry (kind ``host``,
|
||||||
|
action = last path segment; ``/preview`` is not logged by it, the service writes its own entry).
|
||||||
|
The services raise ``api_error``; nothing is translated here. MCP stays read-only (no tool, ``api_get`` is GET).
|
||||||
|
"""
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, Query, Request
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from app.database import get_db
|
||||||
|
from app.errors import ErrorCode, api_error
|
||||||
|
from app.models.host import OpnsenseHost
|
||||||
|
from app.models.user import AdminUser
|
||||||
|
from app.routers import governance
|
||||||
|
from app.routers.auth import resolve_user
|
||||||
|
from app.schemas.rule_cleanup import CleanupOperationRead, CleanupPreviewRead, ConfirmName, PreviewRequest
|
||||||
|
from app.services import rule_cleanup_service as service
|
||||||
|
from app.services.rule_cleanup_service import ACK_TEXT_VERSION
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/hosts/{host_id}/rule-cleanup", tags=["rule-cleanup"])
|
||||||
|
|
||||||
|
|
||||||
|
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||||
|
host = await db.get(OpnsenseHost, host_id)
|
||||||
|
if host is None:
|
||||||
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
return host
|
||||||
|
|
||||||
|
|
||||||
|
def _token(request: Request):
|
||||||
|
return getattr(request.state, "api_token", None)
|
||||||
|
|
||||||
|
|
||||||
|
def _writer(user: AdminUser, request: Request) -> None:
|
||||||
|
"""Any write: not for a read-only account, not with a read-scope token."""
|
||||||
|
if user.role == "read_only":
|
||||||
|
raise api_error(403, ErrorCode.READ_ONLY_ACCESS)
|
||||||
|
token = _token(request)
|
||||||
|
if token is not None and token.scope == "read":
|
||||||
|
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||||
|
|
||||||
|
|
||||||
|
def _admin_only(user: AdminUser, request: Request) -> None:
|
||||||
|
"""Deploy, retry, restore: an admin account, and with a token only one of scope admin."""
|
||||||
|
_writer(user, request)
|
||||||
|
if user.role != "admin":
|
||||||
|
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||||
|
token = _token(request)
|
||||||
|
if token is not None and token.scope != "admin":
|
||||||
|
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||||
|
|
||||||
|
|
||||||
|
async def _deployer(user: AdminUser, request: Request, db: AsyncSession) -> None:
|
||||||
|
"""Revert: an admin, or an operator while the governance switch operators_may_deploy is on."""
|
||||||
|
_writer(user, request)
|
||||||
|
if user.role == "admin":
|
||||||
|
return
|
||||||
|
if user.role == "operator" and (await governance.get_settings(db)).operators_may_deploy:
|
||||||
|
return
|
||||||
|
raise api_error(403, ErrorCode.DEPLOY_NEEDS_ADMIN)
|
||||||
|
|
||||||
|
|
||||||
|
def _session_only(user: AdminUser, request: Request) -> None:
|
||||||
|
"""Confirm: "I still have access" is a human decision in a browser session, never a token's."""
|
||||||
|
_writer(user, request)
|
||||||
|
token = _token(request)
|
||||||
|
if token is not None:
|
||||||
|
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||||
|
|
||||||
|
|
||||||
|
def _full_view(request: Request) -> bool:
|
||||||
|
"""Admins (session or admin-scope token) see the management sources/ports and firewall addresses in a plan."""
|
||||||
|
from app.routers.audit import full_view
|
||||||
|
return full_view(request)
|
||||||
|
|
||||||
|
|
||||||
|
async def _names(db: AsyncSession, operations: list[dict], request: Request) -> list[dict]:
|
||||||
|
"""The firewall names the UI shows next to the ids; for non-admins without the security-relevant plan parts."""
|
||||||
|
if not _full_view(request):
|
||||||
|
operations = [service.redact(op) for op in operations]
|
||||||
|
ids = {uuid.UUID(op["host_id"]) for op in operations} | {uuid.UUID(i["host_id"]) for op in operations for i in op["items"]}
|
||||||
|
names = {h.id: h.name for h in (await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)))).all()} if ids else {}
|
||||||
|
for op in operations:
|
||||||
|
op["host_name"] = names.get(uuid.UUID(op["host_id"]))
|
||||||
|
for item in op["items"]:
|
||||||
|
item["host_name"] = names.get(uuid.UUID(item["host_id"]))
|
||||||
|
return operations
|
||||||
|
|
||||||
|
|
||||||
|
async def _one(db: AsyncSession, operation: dict, request: Request) -> dict:
|
||||||
|
return (await _names(db, [operation], request))[0]
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=list[CleanupOperationRead])
|
||||||
|
async def list_operations(host_id: uuid.UUID, request: Request, limit: int = Query(50, ge=1, le=100),
|
||||||
|
db: AsyncSession = Depends(get_db)):
|
||||||
|
"""The newest operations of this host (and its cluster peers) with their items; unexpired previews included."""
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _names(db, await service.list_operations(db, host, limit), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/preview", response_model=CleanupPreviewRead)
|
||||||
|
async def preview(host_id: uuid.UUID, payload: PreviewRequest, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Recompute the plan from the live firewall(s) and store a 5-minute preview bound to the caller."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
_writer(user, request)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
kind, uuids, acknowledge = payload.resolved()
|
||||||
|
if kind is None:
|
||||||
|
raise api_error(422, ErrorCode.BEREINIGUNG_AUSWAHL_UNGUELTIG, field="kind")
|
||||||
|
result = await service.preview(
|
||||||
|
db, host, actor=user.username, actor_id=user.id, kind=kind, uuids=uuids, acknowledge=acknowledge,
|
||||||
|
**({"unused_days": payload.unused_days} if payload.unused_days else {}),
|
||||||
|
ack_text_version=payload.ack_text_version or ACK_TEXT_VERSION)
|
||||||
|
result["operation"] = await _one(db, await service.get_operation(db, host, uuid.UUID(result["preview_id"])), request)
|
||||||
|
return result if _full_view(request) else service.redact(result)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{op_id}", response_model=CleanupOperationRead)
|
||||||
|
async def get_operation(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.get_operation(db, host, op_id), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{op_id}/cancel", response_model=CleanupOperationRead)
|
||||||
|
async def cancel(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Drop an unused preview (only its own actor can)."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
_writer(user, request)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.cancel(db, host, op_id, actor=user.username), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{op_id}/deploy", response_model=CleanupOperationRead)
|
||||||
|
async def deploy(host_id: uuid.UUID, op_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||||
|
db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Execute a preview. Admin only; the live state must still give the fingerprint of the preview."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
_admin_only(user, request)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.deploy(db, host, op_id, actor=user.username, confirm_name=payload.confirm_name), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{op_id}/confirm", response_model=CleanupOperationRead)
|
||||||
|
async def confirm(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
""""I still have access": re-reads every node first. Browser session only, never with an API token."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
_session_only(user, request)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.confirm(db, host, op_id), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{op_id}/revert", response_model=CleanupOperationRead)
|
||||||
|
async def revert(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Take the change back now (the same code the sweeper runs when the window passes)."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
await _deployer(user, request, db)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.revert_now(db, host, op_id), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{op_id}/retry", response_model=CleanupOperationRead)
|
||||||
|
async def retry(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Revert again after revert_failed, or read the firewall again for a delete/restore with an unclear item."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
_admin_only(user, request)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.retry(db, host, op_id), request)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/items/{item_id}/restore", response_model=CleanupOperationRead)
|
||||||
|
async def restore(host_id: uuid.UUID, item_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||||
|
db: AsyncSession = Depends(get_db)):
|
||||||
|
"""Bring a deleted rule back, switched OFF, at its old position (a new UUID). The result is the restore run;
|
||||||
|
``warnings`` can hold BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR."""
|
||||||
|
user = await resolve_user(request, db)
|
||||||
|
_admin_only(user, request)
|
||||||
|
host = await _host(db, host_id)
|
||||||
|
return await _one(db, await service.restore_item(
|
||||||
|
db, host, item_id, actor=user.username, actor_id=user.id, confirm_name=payload.confirm_name), request)
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import json
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, Body
|
from fastapi import APIRouter, Depends, Body
|
||||||
@@ -51,6 +52,7 @@ def _to_read(
|
|||||||
return RuleRead(
|
return RuleRead(
|
||||||
workflow_state=state,
|
workflow_state=state,
|
||||||
workflow_counts=counts, id=rule.id,
|
workflow_counts=counts, id=rule.id,
|
||||||
|
config_key=rule.config_key,
|
||||||
description=rule.description,
|
description=rule.description,
|
||||||
interface=rule.interface,
|
interface=rule.interface,
|
||||||
direction=rule.direction,
|
direction=rule.direction,
|
||||||
@@ -107,12 +109,17 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
|
|||||||
host = await db.get(OpnsenseHost, payload.host_id)
|
host = await db.get(OpnsenseHost, payload.host_id)
|
||||||
if not host:
|
if not host:
|
||||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||||
|
# Taking over a hand-made rule that a rule cleanup (1.31) is switching off/deleting would make it "managed"
|
||||||
|
# under the cleanup's feet: not while one is open on this host or its cluster.
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [host])
|
||||||
try:
|
try:
|
||||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
|
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
|
||||||
except rule_adopt_service.RuleNotFoundOnHost as exc:
|
except rule_adopt_service.RuleNotFoundOnHost as exc:
|
||||||
raise api_error(404, ErrorCode.REGEL_NICHT_GEFUNDEN) from exc
|
raise api_error(404, ErrorCode.REGEL_NICHT_GEFUNDEN) from exc
|
||||||
except rule_adopt_service.NotAdoptable as exc:
|
except rule_adopt_service.NotAdoptable as exc:
|
||||||
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
raise api_error(409, ErrorCode.RULE_NOT_ADOPTABLE, reasons=", ".join(exc.reasons),
|
||||||
|
reason_details=json.dumps(exc.details, ensure_ascii=False)) from exc
|
||||||
except LINK_ERRORS as exc:
|
except LINK_ERRORS as exc:
|
||||||
raise link_api_error(exc) from exc
|
raise link_api_error(exc) from exc
|
||||||
except OpnsenseAPIError as exc:
|
except OpnsenseAPIError as exc:
|
||||||
@@ -274,6 +281,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
|||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
try:
|
try:
|
||||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_rule_uuid)
|
await opnsense_client.del_rule(deployment.host, deployment.opnsense_rule_uuid)
|
||||||
await opnsense_client.apply(deployment.host)
|
await opnsense_client.apply(deployment.host)
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends
|
from fastapi import APIRouter, Depends
|
||||||
from sqlalchemy import func, select
|
from sqlalchemy import func, select, text
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.errors import ErrorCode, api_error
|
from app.errors import ErrorCode, api_error
|
||||||
@@ -21,10 +21,29 @@ async def _admin_count(db: AsyncSession, exclude_id: uuid.UUID | None = None) ->
|
|||||||
return await db.scalar(query)
|
return await db.scalar(query)
|
||||||
|
|
||||||
|
|
||||||
|
_ADMIN_LOCK_KEY = 728194032 # next to policy_lock's key; only admin role changes and deletions take it
|
||||||
|
|
||||||
|
|
||||||
|
async def _lock_admins(db: AsyncSession, current: AdminUser) -> None:
|
||||||
|
"""Serialize every change that can reduce the number of administrators and re-read the caller under the lock.
|
||||||
|
|
||||||
|
The "at least one administrator" check counts rows; two requests that each see the other's admin row would both
|
||||||
|
pass it. Held until commit/rollback (transaction-scoped advisory lock). The caller may have been demoted or
|
||||||
|
deleted by the request that ran first, so it must still be an admin to proceed."""
|
||||||
|
await db.execute(text("SELECT pg_advisory_xact_lock(:key)"), {"key": _ADMIN_LOCK_KEY})
|
||||||
|
fresh = await db.scalar(select(AdminUser).where(AdminUser.id == current.id).execution_options(populate_existing=True))
|
||||||
|
if fresh is None or fresh.role != "admin":
|
||||||
|
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=list[UserRead])
|
@router.get("", response_model=list[UserRead])
|
||||||
async def list_users(db: AsyncSession = Depends(get_db)):
|
async def list_users(user: AdminUser = Depends(current_user), db: AsyncSession = Depends(get_db)):
|
||||||
rows = (await db.scalars(select(AdminUser).order_by(AdminUser.username))).all()
|
"""Admins see everyone. Every other role sees only its own account: the full list (names, roles, last login,
|
||||||
return rows
|
2FA status) is a directory an attacker with any low-privilege account or token could use to pick targets."""
|
||||||
|
query = select(AdminUser).order_by(AdminUser.username)
|
||||||
|
if user.role != "admin":
|
||||||
|
query = query.where(AdminUser.id == user.id)
|
||||||
|
return (await db.scalars(query)).all()
|
||||||
|
|
||||||
|
|
||||||
@router.post("", response_model=UserRead, status_code=201)
|
@router.post("", response_model=UserRead, status_code=201)
|
||||||
@@ -43,8 +62,12 @@ async def create_user(payload: UserCreate, db: AsyncSession = Depends(get_db)):
|
|||||||
|
|
||||||
|
|
||||||
@router.patch("/{user_id}/role", response_model=UserRead)
|
@router.patch("/{user_id}/role", response_model=UserRead)
|
||||||
async def update_role(user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db)):
|
async def update_role(
|
||||||
target = await db.get(AdminUser, user_id)
|
user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db),
|
||||||
|
current: AdminUser = Depends(current_user),
|
||||||
|
):
|
||||||
|
await _lock_admins(db, current)
|
||||||
|
target = await db.get(AdminUser, user_id, populate_existing=True)
|
||||||
if target is None:
|
if target is None:
|
||||||
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
||||||
if target.auth_source == "oidc":
|
if target.auth_source == "oidc":
|
||||||
@@ -96,13 +119,13 @@ async def delete_user(
|
|||||||
):
|
):
|
||||||
if user_id == current.id:
|
if user_id == current.id:
|
||||||
raise api_error(400, ErrorCode.DER_EIGENE_ACCOUNT_KANN_NICHT_GELOESCHT_WERDEN)
|
raise api_error(400, ErrorCode.DER_EIGENE_ACCOUNT_KANN_NICHT_GELOESCHT_WERDEN)
|
||||||
target = await db.get(AdminUser, user_id)
|
await _lock_admins(db, current)
|
||||||
|
target = await db.get(AdminUser, user_id, populate_existing=True)
|
||||||
if target is None:
|
if target is None:
|
||||||
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
||||||
# No separate "last admin" guard here: only an admin can reach this
|
# Two admins deleting each other (or demoting/deleting in parallel) would each still see the other: the lock above
|
||||||
# endpoint (read_only is rejected earlier, in AuthMiddleware) and the
|
# orders them, and the guard below counts what is left without the target.
|
||||||
# self-delete check above already guarantees target != current — so
|
if target.role == "admin" and await _admin_count(db, exclude_id=target.id) == 0:
|
||||||
# `current` is always a still-existing admin afterward, meaning the
|
raise api_error(409, ErrorCode.MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN)
|
||||||
# admin count can never reach zero this way.
|
|
||||||
await db.delete(target)
|
await db.delete(target)
|
||||||
await db.commit()
|
await db.commit()
|
||||||
@@ -18,6 +18,7 @@ from app.services import wireguard_key_store
|
|||||||
from app.services.wireguard_service import effective_hosts, preview, revision, snapshot, workflow, WIREGUARD_FIELDS
|
from app.services.wireguard_service import effective_hosts, preview, revision, snapshot, workflow, WIREGUARD_FIELDS
|
||||||
from app.services.wireguard_deploy_service import deploy_tunnel, undeploy_tunnel
|
from app.services.wireguard_deploy_service import deploy_tunnel, undeploy_tunnel
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
from app.services.deploy_lease import resource_lease
|
||||||
from app.services.policy_service import policy_lock
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
router = APIRouter(prefix="/wireguard/tunnels", tags=["wireguard"])
|
router = APIRouter(prefix="/wireguard/tunnels", tags=["wireguard"])
|
||||||
@@ -206,18 +207,21 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
|||||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||||
if not entry["can_remove"]:
|
if not entry["can_remove"]:
|
||||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
async with resource_lease("wireguard", tunnel.id):
|
||||||
from app.services import opnsense_client
|
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||||
try:
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
if deployment.opnsense_peer_uuid:
|
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||||
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
|
from app.services import opnsense_client
|
||||||
if deployment.opnsense_server_uuid:
|
try:
|
||||||
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
|
if deployment.opnsense_peer_uuid:
|
||||||
await opnsense_client.reconfigure_wireguard(deployment.host)
|
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
|
||||||
except OpnsenseAPIError as exc:
|
if deployment.opnsense_server_uuid:
|
||||||
deployment.status = "error"
|
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
|
||||||
deployment.message = str(exc)
|
await opnsense_client.reconfigure_wireguard(deployment.host)
|
||||||
|
except OpnsenseAPIError as exc:
|
||||||
|
deployment.status = "error"
|
||||||
|
deployment.message = str(exc)
|
||||||
|
await db.commit()
|
||||||
|
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||||
|
await db.delete(deployment)
|
||||||
await db.commit()
|
await db.commit()
|
||||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
|
||||||
await db.delete(deployment)
|
|
||||||
await db.commit()
|
|
||||||
@@ -12,7 +12,9 @@ CertAction = Literal["import", "internal", "external", "sign_csr", "import_csr",
|
|||||||
# EC curves) — see ManagedCertificate's docstring.
|
# EC curves) — see ManagedCertificate's docstring.
|
||||||
CertKeyType = Literal["2048", "3072", "4096", "prime256v1", "secp384r1", "secp521r1"]
|
CertKeyType = Literal["2048", "3072", "4096", "prime256v1", "secp384r1", "secp521r1"]
|
||||||
CertDigest = Literal["sha1", "sha224", "sha256", "sha384", "sha512"]
|
CertDigest = Literal["sha1", "sha224", "sha256", "sha384", "sha512"]
|
||||||
CertType = Literal["usr_cert", "server_cert", "combined_server_client", "v3_ca"]
|
# No "v3_ca": OPNsense would create a certificate with the CA extension, not a CA that can be used as `caref`
|
||||||
|
# (found by the 1.31.0 test run, T3). Creating a CA is not offered; the UI never did.
|
||||||
|
CertType = Literal["usr_cert", "server_cert", "combined_server_client"]
|
||||||
|
|
||||||
_DN_ACTIONS = {"internal", "external", "reissue"}
|
_DN_ACTIONS = {"internal", "external", "reissue"}
|
||||||
_CRT_ACTIONS = {"import", "import_csr", "manual"}
|
_CRT_ACTIONS = {"import", "import_csr", "manual"}
|
||||||
|
|||||||
@@ -14,6 +14,9 @@ class DashboardHost(BaseModel):
|
|||||||
cluster_name: str | None
|
cluster_name: str | None
|
||||||
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
|
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
|
||||||
gateways: list[GatewayHealth] | None = None
|
gateways: list[GatewayHealth] | None = None
|
||||||
|
# Readings the newest fresh sample could not take because the API key lacks the right (gateways, services, ...);
|
||||||
|
# None = no fresh sample (or one from before this existed), [] = nothing missing.
|
||||||
|
api_rights_missing: list[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
class PendingAction(BaseModel):
|
class PendingAction(BaseModel):
|
||||||
|
|||||||
@@ -17,3 +17,17 @@ class GatewayHealth(BaseModel):
|
|||||||
delay: float | None = None # milliseconds
|
delay: float | None = None # milliseconds
|
||||||
stddev: float | None = None # milliseconds
|
stddev: float | None = None # milliseconds
|
||||||
monitor: str | None = None # the monitored address
|
monitor: str | None = None # the monitored address
|
||||||
|
alert_enabled: bool = True # False: the gateway alert is switched off for this gateway (Settings, 1.29)
|
||||||
|
|
||||||
|
|
||||||
|
class GatewayAlertUpdate(BaseModel):
|
||||||
|
enabled: bool
|
||||||
|
|
||||||
|
|
||||||
|
class GatewayAlertState(BaseModel):
|
||||||
|
name: str
|
||||||
|
alert_enabled: bool
|
||||||
|
|
||||||
|
|
||||||
|
class ApiRightsState(BaseModel):
|
||||||
|
missing: list[str] | None = None # names of the refused readings (gateways, services, swap, cpu); None = unknown
|
||||||
@@ -18,6 +18,9 @@ class HostCreate(BaseModel):
|
|||||||
enabled: bool = True
|
enabled: bool = True
|
||||||
cluster_name: str | None = Field(default=None, max_length=120)
|
cluster_name: str | None = Field(default=None, max_length=120)
|
||||||
policy_hierarchy_group_id: uuid.UUID | None = None
|
policy_hierarchy_group_id: uuid.UUID | None = None
|
||||||
|
# Ask the firewall once with these credentials before saving (the UI and the setup assistant do). Off by default so
|
||||||
|
# scripts can still register a firewall that is not reachable yet (behind NAT, tunnel not up).
|
||||||
|
verify_connection: bool = False
|
||||||
|
|
||||||
@field_validator("base_url")
|
@field_validator("base_url")
|
||||||
@classmethod
|
@classmethod
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ ProblemType = Literal[
|
|||||||
"backup_stale",
|
"backup_stale",
|
||||||
"backup_offsite_failed",
|
"backup_offsite_failed",
|
||||||
"compliance_finding",
|
"compliance_finding",
|
||||||
"gateway_down", "resource_threshold", "service_down",
|
"gateway_down", "resource_threshold", "service_down", "api_rights_missing",
|
||||||
"rollout_paused", "rollout_start_failed",
|
"rollout_paused", "rollout_start_failed",
|
||||||
]
|
]
|
||||||
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
||||||
@@ -54,6 +54,8 @@ class NotificationSettingsRead(BaseModel):
|
|||||||
resource_threshold_enabled: bool = False
|
resource_threshold_enabled: bool = False
|
||||||
service_down_enabled: bool = False
|
service_down_enabled: bool = False
|
||||||
rollout_problems_enabled: bool = True
|
rollout_problems_enabled: bool = True
|
||||||
|
api_rights_enabled: bool = True
|
||||||
|
gateway_alert_muted: list[str] = Field(default_factory=list)
|
||||||
gateway_loss_threshold_pct: int = 20
|
gateway_loss_threshold_pct: int = 20
|
||||||
gateway_delay_threshold_ms: int = 500
|
gateway_delay_threshold_ms: int = 500
|
||||||
disk_threshold_pct: int = 90
|
disk_threshold_pct: int = 90
|
||||||
@@ -111,6 +113,7 @@ class NotificationSettingsUpdate(BaseModel):
|
|||||||
resource_threshold_enabled: bool | None = None
|
resource_threshold_enabled: bool | None = None
|
||||||
service_down_enabled: bool | None = None
|
service_down_enabled: bool | None = None
|
||||||
rollout_problems_enabled: bool | None = None
|
rollout_problems_enabled: bool | None = None
|
||||||
|
api_rights_enabled: bool | None = None
|
||||||
# 0 switches that single threshold off.
|
# 0 switches that single threshold off.
|
||||||
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||||
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
|
|||||||
model_config = ConfigDict(from_attributes=True)
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
id: uuid.UUID
|
id: uuid.UUID
|
||||||
|
config_key: str
|
||||||
description: str
|
description: str
|
||||||
interface: str
|
interface: str
|
||||||
source_net: str
|
source_net: str
|
||||||
|
|||||||
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
|
|||||||
model_config = ConfigDict(from_attributes=True)
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
id: uuid.UUID
|
id: uuid.UUID
|
||||||
|
config_key: str
|
||||||
description: str
|
description: str
|
||||||
interface: str
|
interface: str
|
||||||
source_net: str
|
source_net: str
|
||||||
|
|||||||
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
|
|||||||
model_config = ConfigDict(from_attributes=True)
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
id: uuid.UUID
|
id: uuid.UUID
|
||||||
|
config_key: str
|
||||||
hierarchy_group_id: uuid.UUID
|
hierarchy_group_id: uuid.UUID
|
||||||
rulebase: Literal["pre", "post"]
|
rulebase: Literal["pre", "post"]
|
||||||
sequence: int
|
sequence: int
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
|
|||||||
model_config = ConfigDict(from_attributes=True)
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
id: uuid.UUID
|
id: uuid.UUID
|
||||||
|
config_key: str
|
||||||
description: str
|
description: str
|
||||||
interface: str
|
interface: str
|
||||||
ipprotocol: str
|
ipprotocol: str
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ class RuleRead(BaseModel):
|
|||||||
model_config = ConfigDict(from_attributes=True)
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
id: uuid.UUID
|
id: uuid.UUID
|
||||||
|
config_key: str
|
||||||
description: str
|
description: str
|
||||||
interface: str
|
interface: str
|
||||||
direction: str
|
direction: str
|
||||||
|
|||||||
@@ -0,0 +1,117 @@
|
|||||||
|
"""Request and response shapes of the rule-cleanup API (routers/rule_cleanup.py, DESIGN-1.31 §4).
|
||||||
|
|
||||||
|
Responses mirror services/rule_cleanup_service.read_operation. Texts are not part of the API: ``message`` holds an
|
||||||
|
ErrorCode name (``errors.<CODE>`` in the UI, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR) or the firewall's
|
||||||
|
own wording; ``warnings`` is a list of ErrorCode names.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field, model_validator
|
||||||
|
|
||||||
|
CleanupKind = Literal["disable", "enable", "delete"]
|
||||||
|
# Longest list the API accepts at all; the effective limit is the setting cleanup_max_rules (ceiling 25).
|
||||||
|
MAX_SELECTION = 100
|
||||||
|
|
||||||
|
|
||||||
|
class PreviewItem(BaseModel):
|
||||||
|
"""One rule of the selection. ``kind`` (alias ``mode``) must be the same for all items of one preview."""
|
||||||
|
uuid: str = Field(min_length=1, max_length=64)
|
||||||
|
kind: CleanupKind | None = None
|
||||||
|
mode: CleanupKind | None = None
|
||||||
|
# The per-rule acknowledgement for a "probable" rule (the UI's checkbox).
|
||||||
|
ack: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class PreviewRequest(BaseModel):
|
||||||
|
"""``{kind, uuids, acknowledge}`` or ``{items:[{uuid, kind|mode, ack}]}`` (or both: they are merged)."""
|
||||||
|
kind: CleanupKind | None = None
|
||||||
|
uuids: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||||
|
items: list[PreviewItem] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||||
|
acknowledge: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||||
|
unused_days: int | None = Field(default=None, ge=1, le=3650)
|
||||||
|
ack_text_version: str | None = Field(default=None, max_length=16)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def _single_kind(self):
|
||||||
|
kinds = {k for k in [self.kind, *[i.kind or i.mode for i in self.items]] if k}
|
||||||
|
if len(kinds) > 1:
|
||||||
|
raise ValueError("kind must be the same for every item")
|
||||||
|
return self
|
||||||
|
|
||||||
|
def resolved(self) -> tuple[str | None, list[str], list[str]]:
|
||||||
|
kind = self.kind or next((i.kind or i.mode for i in self.items if i.kind or i.mode), None)
|
||||||
|
uuids = [*self.uuids, *[i.uuid for i in self.items]]
|
||||||
|
acknowledge = [*self.acknowledge, *[i.uuid for i in self.items if i.ack]]
|
||||||
|
return kind, list(dict.fromkeys(u.strip() for u in uuids)), list(dict.fromkeys(a.strip() for a in acknowledge))
|
||||||
|
|
||||||
|
|
||||||
|
class ConfirmName(BaseModel):
|
||||||
|
"""Deploy and restore: the exact host name, typed by a human."""
|
||||||
|
confirm_name: str = Field(max_length=255)
|
||||||
|
|
||||||
|
|
||||||
|
class CleanupItemRead(BaseModel):
|
||||||
|
id: str
|
||||||
|
host_id: str
|
||||||
|
host_name: str | None = None
|
||||||
|
rule_uuid: str
|
||||||
|
seq: int
|
||||||
|
carp_role: str | None = None
|
||||||
|
# queued, disabling, disabled, confirmed, reverting, reverted, deleting, deleted, restoring, restored, failed,
|
||||||
|
# unknown, revert_failed, skipped (never attempted: the run stopped earlier)
|
||||||
|
state: str
|
||||||
|
description: str | None = None
|
||||||
|
categories: Any = None
|
||||||
|
confidence: str | None = None
|
||||||
|
cover_uuid: str | None = None
|
||||||
|
prio_group: str | None = None
|
||||||
|
anchor_next_uuid: str | None = None
|
||||||
|
anchor_prev_uuid: str | None = None
|
||||||
|
source_item_id: str | None = None
|
||||||
|
restored_uuid: str | None = None
|
||||||
|
changed_at: str | None = None
|
||||||
|
message: str | None = None
|
||||||
|
has_snapshot: bool = False
|
||||||
|
# A deleted rule with a stored snapshot that has not been restored yet (POST items/{id}/restore, admin).
|
||||||
|
restorable: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class CleanupOperationRead(BaseModel):
|
||||||
|
id: str
|
||||||
|
host_id: str
|
||||||
|
host_name: str | None = None
|
||||||
|
cluster_name: str | None = None
|
||||||
|
kind: str # disable | enable | delete | restore
|
||||||
|
# preview, running, awaiting_confirm, confirmed, partial, apply_failed, reverting, reverted, revert_failed, done,
|
||||||
|
# failed, cancelled, expired
|
||||||
|
status: str
|
||||||
|
actor: str
|
||||||
|
created_at: str | None = None
|
||||||
|
expires_at: str | None = None
|
||||||
|
started_at: str | None = None
|
||||||
|
heartbeat_at: str | None = None
|
||||||
|
revert_deadline: str | None = None
|
||||||
|
confirmed_at: str | None = None
|
||||||
|
finished_at: str | None = None
|
||||||
|
message: str | None = None
|
||||||
|
options: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
# Only while status == "preview".
|
||||||
|
plan: dict[str, Any] | None = None
|
||||||
|
items: list[CleanupItemRead] = Field(default_factory=list)
|
||||||
|
can_confirm: bool = False
|
||||||
|
can_revert: bool = False
|
||||||
|
can_retry: bool = False
|
||||||
|
# ErrorCode names, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR (restore only).
|
||||||
|
warnings: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class CleanupPreviewRead(BaseModel):
|
||||||
|
preview_id: str
|
||||||
|
expires_at: str | None = None
|
||||||
|
kind: str
|
||||||
|
plan: dict[str, Any]
|
||||||
|
max_rules: int
|
||||||
|
window_seconds: int
|
||||||
|
operation: CleanupOperationRead
|
||||||
@@ -48,9 +48,19 @@ def _origin_allowed(origin: str, host_header: str) -> bool:
|
|||||||
|
|
||||||
# The SSO provider sends the browser back here by a cross-site navigation;
|
# The SSO provider sends the browser back here by a cross-site navigation;
|
||||||
# the endpoint authenticates by its own state cookie and creates the session.
|
# the endpoint authenticates by its own state cookie and creates the session.
|
||||||
|
# The exemption is only for exactly that: a top-level GET navigation (a script, image or fetch from another site
|
||||||
|
# is still blocked, whatever it carries) — and the callback fails without the matching state cookie anyway.
|
||||||
_SEC_FETCH_EXEMPT = {"/api/auth/oidc/callback"}
|
_SEC_FETCH_EXEMPT = {"/api/auth/oidc/callback"}
|
||||||
|
|
||||||
|
|
||||||
|
def _is_top_level_navigation(request: Request) -> bool:
|
||||||
|
if request.method != "GET":
|
||||||
|
return False
|
||||||
|
mode = request.headers.get("sec-fetch-mode", "").strip().lower()
|
||||||
|
dest = request.headers.get("sec-fetch-dest", "").strip().lower()
|
||||||
|
return mode in ("", "navigate") and dest in ("", "document")
|
||||||
|
|
||||||
|
|
||||||
def _explicit_frontend_origin(origin: str) -> bool:
|
def _explicit_frontend_origin(origin: str) -> bool:
|
||||||
origin = origin.rstrip("/").lower()
|
origin = origin.rstrip("/").lower()
|
||||||
if not origin or origin == "null":
|
if not origin or origin == "null":
|
||||||
@@ -62,7 +72,9 @@ def _explicit_frontend_origin(origin: str) -> bool:
|
|||||||
|
|
||||||
def sec_fetch_blocked(request: Request) -> bool:
|
def sec_fetch_blocked(request: Request) -> bool:
|
||||||
path = request.url.path
|
path = request.url.path
|
||||||
if not (path == "/api" or path.startswith("/api/")) or path in _SEC_FETCH_EXEMPT:
|
if not (path == "/api" or path.startswith("/api/")):
|
||||||
|
return False
|
||||||
|
if path in _SEC_FETCH_EXEMPT and _is_top_level_navigation(request):
|
||||||
return False
|
return False
|
||||||
if not has_session_cookie(request):
|
if not has_session_cookie(request):
|
||||||
return False
|
return False
|
||||||
|
|||||||
@@ -130,6 +130,9 @@ async def sweep_expired_alias_deployments(db: AsyncSession) -> list[AliasDeploym
|
|||||||
|
|
||||||
|
|
||||||
async def undeploy_alias(db: AsyncSession, alias: FirewallAlias) -> None:
|
async def undeploy_alias(db: AsyncSession, alias: FirewallAlias) -> None:
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||||
|
await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(AliasDeployment).where(AliasDeployment.alias_id == alias.id))
|
result = await db.execute(select(AliasDeployment).where(AliasDeployment.alias_id == alias.id))
|
||||||
for deployment in result.scalars().all():
|
for deployment in result.scalars().all():
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ from app.models.alias_deployment import AliasDeployment
|
|||||||
from app.models.host import OpnsenseHost
|
from app.models.host import OpnsenseHost
|
||||||
from app.schemas.host import AliasOption
|
from app.schemas.host import AliasOption
|
||||||
from app.services import opnsense_client
|
from app.services import opnsense_client
|
||||||
|
from app.services.deploy_flow import unsettled
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
|
||||||
|
|
||||||
@@ -87,7 +88,7 @@ async def ensure_aliases_settled(db: AsyncSession, hosts: list[OpnsenseHost], va
|
|||||||
.where(
|
.where(
|
||||||
FirewallAlias.name.in_(names),
|
FirewallAlias.name.in_(names),
|
||||||
AliasDeployment.host_id.in_([host.id for host in hosts]),
|
AliasDeployment.host_id.in_([host.id for host in hosts]),
|
||||||
AliasDeployment.status.in_(("pending", "revert_failed")),
|
unsettled(AliasDeployment),
|
||||||
)
|
)
|
||||||
.order_by(FirewallAlias.name, OpnsenseHost.name)
|
.order_by(FirewallAlias.name, OpnsenseHost.name)
|
||||||
)).all()
|
)).all()
|
||||||
|
|||||||
@@ -39,20 +39,34 @@ _LAST_USED_GRANULARITY = timedelta(minutes=1)
|
|||||||
|
|
||||||
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
|
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
|
||||||
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
|
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
|
||||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$")
|
# Rule cleanup (1.31): "I still have access" is a human decision, so confirming a cleanup is never possible with a
|
||||||
|
# token, whatever its scope (a script must not be able to confirm blindly).
|
||||||
|
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$")
|
||||||
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
||||||
# notification settings the webhook URL/SMTP data): admin scope only.
|
# notification settings the webhook URL/SMTP data): admin scope only.
|
||||||
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
||||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
|
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
|
||||||
_MCP_PATH = "/api/mcp"
|
# it compares a posted file with the database and writes nothing.
|
||||||
|
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
|
||||||
_ADMIN_ONLY_WRITES = re.compile(
|
_ADMIN_ONLY_WRITES = re.compile(
|
||||||
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
||||||
r"|^/api/logs/settings$"
|
r"|^/api/logs/settings$"
|
||||||
r"|^/api/hosts(/tls-probe|/[^/]+)?$"
|
r"|^/api/hosts(/tls-probe|/[^/]+)?$"
|
||||||
r"|^/api/hosts/[^/]+/gui-ticket$"
|
r"|^/api/hosts/[^/]+/gui-ticket$"
|
||||||
|
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
|
||||||
|
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
|
||||||
|
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
|
||||||
|
r"|^/api/config/import/apply$"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Rule cleanup: executing a cleanup, retrying a failed revert/reconcile and restoring a deleted rule are admin-only.
|
||||||
|
# operators_may_deploy defaults to True, so _ROLLOUT_POST alone cannot express "admin only"; this pattern is checked
|
||||||
|
# for operator accounts (operator_denies) and for tokens below scope admin (denies). Not part of _ADMIN_ONLY_WRITES:
|
||||||
|
# that one means "who may access OPNexus / where data goes". The router checks the same rule once more.
|
||||||
|
_ADMIN_ONLY_CLEANUP = re.compile(r"^/api/hosts/[^/]+/rule-cleanup/([^/]+/(deploy|retry)|items/[^/]+/restore)/?$")
|
||||||
|
|
||||||
|
|
||||||
def _now() -> datetime:
|
def _now() -> datetime:
|
||||||
return datetime.now(timezone.utc)
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
@@ -65,13 +79,13 @@ def denies(scope: str, method: str, path: str) -> bool:
|
|||||||
"""True if a token with this scope must not make this request."""
|
"""True if a token with this scope must not make this request."""
|
||||||
if _SESSION_ONLY.match(path):
|
if _SESSION_ONLY.match(path):
|
||||||
return True
|
return True
|
||||||
if path == _MCP_PATH:
|
if path in _READ_ONLY_POSTS:
|
||||||
return False
|
return False
|
||||||
if method in _SAFE_METHODS:
|
if method in _SAFE_METHODS:
|
||||||
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
|
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
|
||||||
if scope == "read":
|
if scope == "read":
|
||||||
return True
|
return True
|
||||||
if scope == "deploy" and _ADMIN_ONLY_WRITES.match(path):
|
if scope == "deploy" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):
|
||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
@@ -80,18 +94,23 @@ def operator_denies(method: str, path: str) -> bool:
|
|||||||
"""True if an `operator` account must not make this request: the writes that
|
"""True if an `operator` account must not make this request: the writes that
|
||||||
change who may access OPNexus or where it sends data. (Their own theme at
|
change who may access OPNexus or where it sends data. (Their own theme at
|
||||||
PUT /api/settings stays open; account endpoints are not matched here.)"""
|
PUT /api/settings stays open; account endpoints are not matched here.)"""
|
||||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(_ADMIN_ONLY_WRITES.match(path))
|
return method not in _SAFE_METHODS and path != "/api/settings" and bool(
|
||||||
|
_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))
|
||||||
|
|
||||||
|
|
||||||
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
|
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
|
||||||
# maintenance (reboot/update/CARP switch), scheduling it, applying network changes, pushing certificates and
|
# maintenance (reboot/update/CARP switch), scheduling it, applying network changes, pushing certificates and
|
||||||
# ACME accounts, re-asserting the hierarchy order, controlling services, and DELETING a centrally managed object
|
# ACME accounts, re-asserting the hierarchy order, controlling services, and DELETING a centrally managed object
|
||||||
# or snapshot (deleting a rule removes it from every firewall it was deployed to).
|
# or snapshot (deleting a rule removes it from every firewall it was deployed to), and creating a snapshot.
|
||||||
_ROLLOUT_POST = re.compile(
|
_ROLLOUT_POST = re.compile(
|
||||||
r"/(deploy|push|sign|poll|register|reassert-ordering"
|
r"/(deploy|push|sign|poll|register|reassert-ordering"
|
||||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||||
|
# Rule cleanup (1.31): switching rules off (/deploy matches above) and taking it back. Confirming and
|
||||||
|
# previewing stay open, like confirm_rule.
|
||||||
|
r"|rule-cleanup/[^/]+/(revert|retry)"
|
||||||
r"|maintenance/(execute|carp-execute|schedule)"
|
r"|maintenance/(execute|carp-execute|schedule)"
|
||||||
r"|network/changes"
|
r"|network/changes"
|
||||||
|
r"|snapshots" # creating a boot environment (deleting one is in _ROLLOUT_DELETE)
|
||||||
r"|services/[^/]+/(start|stop|restart)"
|
r"|services/[^/]+/(start|stop|restart)"
|
||||||
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
|
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
|
||||||
r"|rollouts/plans/[^/]+/(start|activate)"
|
r"|rollouts/plans/[^/]+/(start|activate)"
|
||||||
|
|||||||
@@ -13,6 +13,9 @@ from app.database import SessionLocal
|
|||||||
from app.models.audit import AuditEvent
|
from app.models.audit import AuditEvent
|
||||||
|
|
||||||
context = ContextVar("audit_context", default=None)
|
context = ContextVar("audit_context", default=None)
|
||||||
|
# A key in ``details`` whose content only admins (session or admin-scope token) get to read (routers/audit.py):
|
||||||
|
# firewall addresses and management sources of the rule cleanup.
|
||||||
|
ADMIN_ONLY_DETAILS = "admin_only"
|
||||||
|
|
||||||
|
|
||||||
# Column widths of audit_events. entity_id/host_id/action are taken from the
|
# Column widths of audit_events. entity_id/host_id/action are taken from the
|
||||||
|
|||||||
@@ -236,6 +236,7 @@ async def create_session(
|
|||||||
user_agent: str | None = None,
|
user_agent: str | None = None,
|
||||||
ip_address: str | None = None,
|
ip_address: str | None = None,
|
||||||
verified_hash: str | None = None,
|
verified_hash: str | None = None,
|
||||||
|
verified_totp: bool | None = None,
|
||||||
lifetime_seconds: int | None = None,
|
lifetime_seconds: int | None = None,
|
||||||
) -> tuple[str, datetime]:
|
) -> tuple[str, datetime]:
|
||||||
"""Create a session, returning the raw token (to be set as a cookie) and its expiry.
|
"""Create a session, returning the raw token (to be set as a cookie) and its expiry.
|
||||||
@@ -244,15 +245,22 @@ async def create_session(
|
|||||||
against. The user row is re-read under a row lock first: a password
|
against. The user row is re-read under a row lock first: a password
|
||||||
change/reset that committed while the (slow) password check was running
|
change/reset that committed while the (slow) password check was running
|
||||||
changed the hash, and the login must not outlive that revocation.
|
changed the hash, and the login must not outlive that revocation.
|
||||||
|
|
||||||
|
``verified_totp`` is whether the caller checked a second factor. Two-factor
|
||||||
|
enabled in the meantime (it also revoked every session) means this login
|
||||||
|
was decided without it and must start over.
|
||||||
"""
|
"""
|
||||||
user_id = user.id
|
user_id = user.id
|
||||||
if verified_hash is not None:
|
if verified_hash is not None or verified_totp is not None:
|
||||||
locked = await db.scalar(
|
locked_hash, locked_totp = (await db.execute(
|
||||||
select(AdminUser.password_hash).where(AdminUser.id == user_id).with_for_update()
|
select(AdminUser.password_hash, AdminUser.totp_enabled).where(AdminUser.id == user_id).with_for_update()
|
||||||
)
|
)).one()
|
||||||
if locked != verified_hash:
|
if verified_hash is not None and locked_hash != verified_hash:
|
||||||
await db.rollback()
|
await db.rollback()
|
||||||
raise api_error(401, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
raise api_error(401, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
||||||
|
if locked_totp and not verified_totp:
|
||||||
|
await db.rollback()
|
||||||
|
raise api_error(401, ErrorCode.TOTP_REQUIRED)
|
||||||
token = secrets.token_urlsafe(SESSION_TOKEN_BYTES)
|
token = secrets.token_urlsafe(SESSION_TOKEN_BYTES)
|
||||||
expires_at = _now() + timedelta(seconds=lifetime_seconds or settings.session_lifetime)
|
expires_at = _now() + timedelta(seconds=lifetime_seconds or settings.session_lifetime)
|
||||||
session = Session(
|
session = Session(
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ from app.models.host import OpnsenseHost
|
|||||||
from app.services import backup_store, rule_hit_service
|
from app.services import backup_store, rule_hit_service
|
||||||
from app.services import opnsense_client as client
|
from app.services import opnsense_client as client
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
from app.services.rule_analysis import coverage
|
||||||
|
|
||||||
MAX_CONFIG_BYTES = 32 * 1024 * 1024
|
MAX_CONFIG_BYTES = 32 * 1024 * 1024
|
||||||
BACKUP_MAX_AGE_DAYS = 7
|
BACKUP_MAX_AGE_DAYS = 7
|
||||||
@@ -83,10 +84,15 @@ def _mvc_rule(rule: Element) -> dict:
|
|||||||
"sport": _text(rule, "source_port"),
|
"sport": _text(rule, "source_port"),
|
||||||
"dst": _text(rule, "destination_net") or "any", "dst_not": _text(rule, "destination_not") == "1",
|
"dst": _text(rule, "destination_net") or "any", "dst_not": _text(rule, "destination_not") == "1",
|
||||||
"dport": _text(rule, "destination_port"),
|
"dport": _text(rule, "destination_port"),
|
||||||
"floating": False,
|
"if_not": _text(rule, "interfacenot") == "1",
|
||||||
|
"floating": False, # decided in extract_rules (needs the list of interfaces)
|
||||||
"quick": _text(rule, "quick") != "0",
|
"quick": _text(rule, "quick") != "0",
|
||||||
"descr": _text(rule, "description") or "",
|
"descr": _text(rule, "description") or "",
|
||||||
"order": int(_text(rule, "sequence") or 0),
|
"order": int(_text(rule, "sequence") or 0),
|
||||||
|
"sched": _text(rule, "sched") or "", "tagged": _text(rule, "tagged") or "",
|
||||||
|
"received_on": _text(rule, "received-on") or "",
|
||||||
|
"icmptype": ",".join(filter(None, [_text(rule, "icmptype"), _text(rule, "icmp6type")])),
|
||||||
|
"limited": any(_text(rule, tag) for tag in ("max-src-conn-rate", "max-pkt-rate-number", "overload", "tcpflags1")),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -101,17 +107,29 @@ def _legacy_rule(rule: Element, index: int) -> dict:
|
|||||||
"ipprotocol": _text(rule, "ipprotocol") or "inet",
|
"ipprotocol": _text(rule, "ipprotocol") or "inet",
|
||||||
"protocol": (_text(rule, "protocol") or "any").lower(),
|
"protocol": (_text(rule, "protocol") or "any").lower(),
|
||||||
"src": src, "src_not": src_not, "sport": sport, "dst": dst, "dst_not": dst_not, "dport": dport,
|
"src": src, "src_not": src_not, "sport": sport, "dst": dst, "dst_not": dst_not, "dport": dport,
|
||||||
|
"if_not": rule.find("interfacenot") is not None,
|
||||||
"floating": _text(rule, "floating") == "yes",
|
"floating": _text(rule, "floating") == "yes",
|
||||||
"quick": _text(rule, "quick") != "0",
|
# Legacy rules: an interface rule is always evaluated "quick"; only a floating rule has the option,
|
||||||
|
# and it is off unless <quick> says yes.
|
||||||
|
"quick": _text(rule, "floating") != "yes" or (_text(rule, "quick") or "").lower() in ("yes", "1"),
|
||||||
"descr": _text(rule, "descr") or "",
|
"descr": _text(rule, "descr") or "",
|
||||||
"order": index,
|
"order": index,
|
||||||
|
"sched": _text(rule, "sched") or "", "tagged": _text(rule, "tagged") or "",
|
||||||
|
"received_on": _text(rule, "received-on") or "", "icmptype": _text(rule, "icmptype") or "",
|
||||||
|
"limited": any(_text(rule, tag) for tag in ("max-src-conn-rate", "max-pkt-rate-number", "overload", "tcpflags1")),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def extract_rules(root: Element) -> list[list[dict]]:
|
def extract_rules(root: Element) -> list[list[dict]]:
|
||||||
"""Rule lists in evaluation order, one per source (MVC, legacy)."""
|
"""Rule lists in evaluation order, one per source (MVC, legacy)."""
|
||||||
mvc = sorted((_mvc_rule(r) for r in root.findall("OPNsense/Firewall/Filter/rules/rule")), key=lambda r: r["order"])
|
mvc = [_mvc_rule(r) for r in root.findall("OPNsense/Firewall/Filter/rules/rule")]
|
||||||
|
known = {node.tag for node in root.findall("interfaces/*")}
|
||||||
|
for rule in mvc: # same rule as OPNsense: not exactly one interface (or "not") = floating
|
||||||
|
rule["floating"] = len(rule["interfaces"]) != 1 or rule["if_not"]
|
||||||
|
rule["group"] = not rule["floating"] and rule["interfaces"][0] not in known
|
||||||
|
mvc.sort(key=coverage.config_order)
|
||||||
legacy = [_legacy_rule(r, i) for i, r in enumerate(root.findall("filter/rule"))]
|
legacy = [_legacy_rule(r, i) for i, r in enumerate(root.findall("filter/rule"))]
|
||||||
|
legacy.sort(key=coverage.config_order)
|
||||||
return [group for group in (mvc, legacy) if group]
|
return [group for group in (mvc, legacy) if group]
|
||||||
|
|
||||||
|
|
||||||
@@ -133,41 +151,22 @@ def _is_any_any(rule: dict) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
def _covers(a: dict, b: dict) -> bool:
|
|
||||||
"""Every packet matched by rule b is already matched by the earlier rule a.
|
|
||||||
Only exact-or-any comparisons, so it never claims coverage it cannot prove."""
|
|
||||||
if not (a["enabled"] and b["enabled"]) or a["floating"] or b["floating"] or not a["quick"]:
|
|
||||||
return False
|
|
||||||
if not a["interfaces"] or not set(b["interfaces"]) <= set(a["interfaces"]):
|
|
||||||
return False
|
|
||||||
if a["direction"] != b["direction"] or a["ipprotocol"] != b["ipprotocol"]:
|
|
||||||
return False
|
|
||||||
if a["protocol"] != "any" and a["protocol"] != b["protocol"]:
|
|
||||||
return False
|
|
||||||
for side in ("src", "dst"):
|
|
||||||
if a[side + "_not"] or b[side + "_not"]:
|
|
||||||
if (a[side], a[side + "_not"]) != (b[side], b[side + "_not"]):
|
|
||||||
return False
|
|
||||||
elif a[side] != "any" and a[side] != b[side]:
|
|
||||||
return False
|
|
||||||
for port in ("sport", "dport"):
|
|
||||||
if a[port] and a[port] != b[port]:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _label(rule: dict) -> str:
|
def _label(rule: dict) -> str:
|
||||||
return rule["descr"] or f"{rule['action']} {rule['src']} → {rule['dst']}"
|
return rule["descr"] or f"{rule['action']} {rule['src']} → {rule['dst']}"
|
||||||
|
|
||||||
|
|
||||||
def analyse_rules(groups: list[list[dict]]) -> dict:
|
def analyse_rules(groups: list[list[dict]]) -> dict:
|
||||||
|
"""Shadowed and duplicate rules. The coverage logic lives in rule_analysis/coverage.py (shared with the
|
||||||
|
cleanup candidates); the rules of one list are compared in evaluation order."""
|
||||||
shadowed, duplicates = [], []
|
shadowed, duplicates = [], []
|
||||||
for rules in groups:
|
for rules in groups:
|
||||||
for j, later in enumerate(rules):
|
ordered = sorted(rules, key=coverage.config_order)
|
||||||
for earlier in rules[:j]:
|
norm = [coverage.from_config_rule(rule) for rule in ordered]
|
||||||
if _covers(earlier, later):
|
by_identity = {id(n): r for n, r in zip(norm, ordered)}
|
||||||
(duplicates if earlier["action"] == later["action"] and _covers(later, earlier) else shadowed).append(_label(later))
|
for overlap in coverage.analyse(norm):
|
||||||
break
|
if overlap.by.order == overlap.rule.order:
|
||||||
|
continue # equal position in config.xml: the real order is unknown, claim nothing
|
||||||
|
(duplicates if overlap.kind == "duplicate" else shadowed).append(_label(by_identity[id(overlap.rule)]))
|
||||||
return {"shadowed": shadowed, "duplicates": duplicates}
|
return {"shadowed": shadowed, "duplicates": duplicates}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ and the sweeper away in the gap (pending/unknown rows are never re-deployed).
|
|||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
from typing import Any, Awaitable, Callable
|
from typing import Any, Awaitable, Callable
|
||||||
|
|
||||||
|
from sqlalchemy import and_, or_
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.config import settings
|
from app.config import settings
|
||||||
@@ -46,7 +47,21 @@ def _now() -> datetime:
|
|||||||
return datetime.now(timezone.utc)
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
async def _checkpoint(db: AsyncSession) -> None:
|
def is_unsettled(row) -> bool:
|
||||||
|
"""A change that may still be live and whose rollback is not yet done: pending, outcome unknown, or an
|
||||||
|
"error" row whose revert deadline is still armed (written but apply/ordering not confirmed)."""
|
||||||
|
return row.status in ("pending", "revert_failed") or (row.status == "error" and row.revert_deadline is not None)
|
||||||
|
|
||||||
|
|
||||||
|
def unsettled(model):
|
||||||
|
"""``is_unsettled`` as a SQL condition for the deployment model."""
|
||||||
|
return or_(
|
||||||
|
model.status.in_(("pending", "revert_failed")),
|
||||||
|
and_(model.status == "error", model.revert_deadline.isnot(None)),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def checkpoint(db: AsyncSession) -> None:
|
||||||
from app.services.policy_service import policy_lock
|
from app.services.policy_service import policy_lock
|
||||||
|
|
||||||
await db.commit()
|
await db.commit()
|
||||||
@@ -67,12 +82,16 @@ async def push_to_host(
|
|||||||
) -> Any:
|
) -> Any:
|
||||||
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
|
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
|
||||||
before = {name: getattr(existing, name) for name in _FIELDS} if existing else None
|
before = {name: getattr(existing, name) for name in _FIELDS} if existing else None
|
||||||
|
# An armed "error" row (written, apply not confirmed) holds the UNCONFIRMED payload as last_payload: a retry must
|
||||||
|
# keep the original rollback target instead of promoting that payload to it.
|
||||||
|
armed = existing is not None and existing.status == "error" and existing.revert_deadline is not None
|
||||||
|
rollback_target = existing.previous_payload if armed else prior_live_payload
|
||||||
|
|
||||||
# 1. durable intent
|
# 1. durable intent
|
||||||
if existing is not None:
|
if existing is not None:
|
||||||
row = existing
|
row = existing
|
||||||
row.status, row.message = "pending", None
|
row.status, row.message = "pending", None
|
||||||
row.previous_payload, row.last_payload = prior_live_payload, payload
|
row.previous_payload, row.last_payload = rollback_target, payload
|
||||||
row.revert_deadline, row.deployed_at = _now() + window, _now()
|
row.revert_deadline, row.deployed_at = _now() + window, _now()
|
||||||
else:
|
else:
|
||||||
row = make(
|
row = make(
|
||||||
@@ -83,7 +102,7 @@ async def push_to_host(
|
|||||||
# The intent is a technical safety record, not a state change worth a history
|
# The intent is a technical safety record, not a state change worth a history
|
||||||
# entry of its own (the request itself is already audited as "requested").
|
# entry of its own (the request itself is already audited as "requested").
|
||||||
row._skip_audit = True
|
row._skip_audit = True
|
||||||
await _checkpoint(db)
|
await checkpoint(db)
|
||||||
row._skip_audit = False
|
row._skip_audit = False
|
||||||
|
|
||||||
wrote = False
|
wrote = False
|
||||||
@@ -93,9 +112,9 @@ async def push_to_host(
|
|||||||
# 3. recoverable by UUID from here on
|
# 3. recoverable by UUID from here on
|
||||||
setattr(row, uuid_attr, new_uuid)
|
setattr(row, uuid_attr, new_uuid)
|
||||||
row.status, row.message = "pending", None
|
row.status, row.message = "pending", None
|
||||||
row.previous_payload = prior_live_payload if existing is not None else None
|
row.previous_payload = rollback_target if existing is not None else None
|
||||||
row.last_payload, row.revert_deadline, row.deployed_at = payload, _now() + window, _now()
|
row.last_payload, row.revert_deadline, row.deployed_at = payload, _now() + window, _now()
|
||||||
await _checkpoint(db)
|
await checkpoint(db)
|
||||||
await apply()
|
await apply()
|
||||||
except OpnsenseAPIError as exc:
|
except OpnsenseAPIError as exc:
|
||||||
ambiguous = isinstance(exc, OpnsenseAmbiguousError)
|
ambiguous = isinstance(exc, OpnsenseAmbiguousError)
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"""An exclusive per-object lock for flows that make several remote calls and commit in between (VPN deploys).
|
||||||
|
|
||||||
|
``policy_lock`` is transaction-scoped and is released by every checkpoint commit; this one is a session-level advisory
|
||||||
|
lock on a connection of its own, so it spans the commits of the request session. If the process dies the connection
|
||||||
|
closes and PostgreSQL frees the lock. It never waits: a second deploy/removal of the same object is refused at once.
|
||||||
|
"""
|
||||||
|
import uuid
|
||||||
|
import zlib
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
from app.database import engine
|
||||||
|
from app.errors import ErrorCode, api_error
|
||||||
|
|
||||||
|
# Written when the first row is committed, before any remote call: a crash or cancellation leaves this, and a new
|
||||||
|
# deploy continues from the UUIDs recorded after it.
|
||||||
|
VPN_INTERRUPTED = (
|
||||||
|
"Ausrollen unterbrochen oder Ausgang unklar: Die Firewall hat möglicherweise schon Teile der Konfiguration "
|
||||||
|
"erhalten. Erneutes Ausrollen setzt dort fort."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def resource_lease(kind: str, resource_id: uuid.UUID):
|
||||||
|
params = {"a": zlib.crc32(kind.encode()) & 0x7FFFFFFF, "b": resource_id.int & 0x7FFFFFFF}
|
||||||
|
async with engine.connect() as conn:
|
||||||
|
if not await conn.scalar(text("SELECT pg_try_advisory_lock(:a, :b)"), params):
|
||||||
|
raise api_error(409, ErrorCode.VPN_DEPLOY_LAEUFT_BEREITS)
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
await conn.execute(text("SELECT pg_advisory_unlock(:a, :b)"), params)
|
||||||
|
except Exception:
|
||||||
|
await conn.invalidate() # never hand a connection that may still hold the lock back to the pool
|
||||||
|
except BaseException: # cancelled while unlocking
|
||||||
|
await conn.invalidate()
|
||||||
|
raise
|
||||||
@@ -63,6 +63,17 @@ async def _guarded(host: OpnsenseHost, tool: str, target: str, params: dict, cal
|
|||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _with_silent_hops(hops: list[dict]) -> list[dict]:
|
||||||
|
"""The firewall leaves out hops that never answered (TTL jumps 1 -> 3, T5 of the 1.31.0 test run). Put them back as
|
||||||
|
explicit rows so a gap reads as "no answer here", not as a missing hop. A route where nothing answered at all stays
|
||||||
|
an empty list; the UI says so."""
|
||||||
|
seen = {int(h["ttl"]): h for h in hops if h["ttl"].isdigit()}
|
||||||
|
if not seen:
|
||||||
|
return hops
|
||||||
|
silent = {"address": "", "host": "", "as": "", "probes": "", "silent": True}
|
||||||
|
return [seen.get(ttl) or {**silent, "ttl": str(ttl)} for ttl in range(1, max(seen) + 1)]
|
||||||
|
|
||||||
|
|
||||||
async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: str, source: str | None) -> dict:
|
async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: str, source: str | None) -> dict:
|
||||||
target, source = clean_target(target), clean_source(source)
|
target, source = clean_target(target), clean_source(source)
|
||||||
params = {"target": target, "ipproto": ipproto, "protocol": protocol, "source": source}
|
params = {"target": target, "ipproto": ipproto, "protocol": protocol, "source": source}
|
||||||
@@ -73,8 +84,9 @@ async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: st
|
|||||||
{"hostname": target, "ipproto": ipproto, "protocol": protocol, "source_address": source})
|
{"hostname": target, "ipproto": ipproto, "protocol": protocol, "source_address": source})
|
||||||
resp = data.get("response") or {}
|
resp = data.get("response") or {}
|
||||||
hops = [{"ttl": str(r.get("ttl", "")), "address": str(r.get("address", "")), "host": str(r.get("host", "")),
|
hops = [{"ttl": str(r.get("ttl", "")), "address": str(r.get("address", "")), "host": str(r.get("host", "")),
|
||||||
"as": str(r.get("AS", "")), "probes": str(r.get("probes", ""))} for r in resp.get("rows", []) if isinstance(r, dict)]
|
"as": str(r.get("AS", "")), "probes": str(r.get("probes", "")), "silent": False}
|
||||||
return {"target": target, "notice": str(resp.get("notice", "")), "hops": hops}
|
for r in resp.get("rows", []) if isinstance(r, dict)]
|
||||||
|
return {"target": target, "notice": str(resp.get("notice", "")), "hops": _with_silent_hops(hops)}
|
||||||
return await _guarded(host, "traceroute", target, params, call)
|
return await _guarded(host, "traceroute", target, params, call)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,12 @@ Flow (see docs in routers/gui_access.py):
|
|||||||
|
|
||||||
A slot stays assigned to one firewall for as long as that browser session has the firewall open; another
|
A slot stays assigned to one firewall for as long as that browser session has the firewall open; another
|
||||||
firewall gets another slot (another origin), so two firewall GUIs never share an origin at the same time.
|
firewall gets another slot (another origin), so two firewall GUIs never share an origin at the same time.
|
||||||
|
After the last gateway session of a slot ended, the slot goes to a DIFFERENT firewall only after SLOT_COOLDOWN
|
||||||
|
(1.29, S1): a popup of a possibly compromised firewall GUI may still be open and its cookies belong to that
|
||||||
|
origin. The same firewall may take the slot again at once.
|
||||||
|
|
||||||
|
The target the gateway talks to (origin, pinned CA, overlay proxy) is fingerprinted at redeem time; every
|
||||||
|
re-check compares it with the host's current configuration and ends the session when it changed (S4).
|
||||||
"""
|
"""
|
||||||
import hashlib
|
import hashlib
|
||||||
import secrets
|
import secrets
|
||||||
@@ -33,7 +39,13 @@ TICKET_TTL = timedelta(seconds=30)
|
|||||||
# 20 s: a switch-off, logout or role change must end a gateway session within 30 s of the change, and
|
# 20 s: a switch-off, logout or role change must end a gateway session within 30 s of the change, and
|
||||||
# the last re-check can lie up to one interval before it (live test 2026-10-21: 30 s gave 31 s).
|
# the last re-check can lie up to one interval before it (live test 2026-10-21: 30 s gave 31 s).
|
||||||
REVALIDATE_SECONDS = 20
|
REVALIDATE_SECONDS = 20
|
||||||
|
# A gateway session nobody re-checked for this long is treated as ended (gateway crashed without telling us).
|
||||||
|
SESSION_STALE_AFTER = timedelta(seconds=REVALIDATE_SECONDS * 3)
|
||||||
MAX_SLOTS = 4
|
MAX_SLOTS = 4
|
||||||
|
# A slot freed by firewall A goes to firewall B no earlier than this long after A's session ended.
|
||||||
|
SLOT_COOLDOWN = timedelta(minutes=5)
|
||||||
|
# Open gateway sessions at once (the gateway container enforces the same number; test_gui_access checks they match).
|
||||||
|
MAX_GATEWAY_SESSIONS = 64
|
||||||
MIN_GATEWAY_TOKEN_LENGTH = 32
|
MIN_GATEWAY_TOKEN_LENGTH = 32
|
||||||
AUDIT_KIND = "gui_access"
|
AUDIT_KIND = "gui_access"
|
||||||
|
|
||||||
@@ -114,6 +126,13 @@ def firewall_origin(base_url: str) -> str:
|
|||||||
return f"https://{parts.netloc.rsplit('@', 1)[-1]}"
|
return f"https://{parts.netloc.rsplit('@', 1)[-1]}"
|
||||||
|
|
||||||
|
|
||||||
|
def target_fingerprint(host: OpnsenseHost) -> str:
|
||||||
|
"""What a gateway session talks to: origin, pinned CA and overlay route. Changes -> the session must end."""
|
||||||
|
material = "\n".join([firewall_origin(host.base_url), host.tls_ca_pem or "", overlay_service.proxy_for(host.base_url) or "",
|
||||||
|
"verify" if host.verify_tls else "noverify"])
|
||||||
|
return hash_value(material)
|
||||||
|
|
||||||
|
|
||||||
# ── audit ─────────────────────────────────────────────────────────────────
|
# ── audit ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
async def _audit(actor: str, *, action: str, status: str, host: OpnsenseHost | None = None,
|
async def _audit(actor: str, *, action: str, status: str, host: OpnsenseHost | None = None,
|
||||||
@@ -159,6 +178,30 @@ async def _pick_slot(db: AsyncSession, row: GuiAccessSettings, session_hash: str
|
|||||||
free = [s for s in slots if s not in in_use]
|
free = [s for s in slots if s not in in_use]
|
||||||
if not free:
|
if not free:
|
||||||
raise GuiAccessError(409, ErrorCode.GUI_NO_FREE_SLOT, {"slots": str(len(slots))})
|
raise GuiAccessError(409, ErrorCode.GUI_NO_FREE_SLOT, {"slots": str(len(slots))})
|
||||||
|
# S1: a slot this user's browser may still have a window of ANOTHER firewall on stays closed to this
|
||||||
|
# firewall — while a gateway session of the user is open there (any browser session) and for SLOT_COOLDOWN
|
||||||
|
# after the last one ended. The same firewall may take it again at once.
|
||||||
|
held = (await db.execute(
|
||||||
|
select(GuiSession.slot, GuiSession.host_id, GuiSession.ended_at, GuiSession.last_checked_at).where(
|
||||||
|
GuiSession.user_id == user_id,
|
||||||
|
((GuiSession.ended_at.is_(None)) & ((GuiSession.started_at > now - absolute) | (GuiSession.last_checked_at > now - SLOT_COOLDOWN)))
|
||||||
|
| (GuiSession.ended_at > now - SLOT_COOLDOWN)))).all()
|
||||||
|
cooling: dict[int, datetime] = {} # slot -> when it may go to another firewall
|
||||||
|
for slot, h_id, ended_at, checked_at in held:
|
||||||
|
if h_id == host_id or slot not in free:
|
||||||
|
continue
|
||||||
|
if ended_at is None and checked_at is not None and now - checked_at > SESSION_STALE_AFTER:
|
||||||
|
ended_at = checked_at # the gateway stopped re-checking it (crash, kill): it ended when it was last seen
|
||||||
|
release = (ended_at + SLOT_COOLDOWN) if ended_at is not None else now + SLOT_COOLDOWN
|
||||||
|
if release <= now:
|
||||||
|
continue # cooled down long ago (a session that stopped answering a while back)
|
||||||
|
cooling[slot] = max(cooling.get(slot, release), release)
|
||||||
|
usable = [s for s in free if s not in cooling]
|
||||||
|
if not usable:
|
||||||
|
wait = min(cooling.values()) - now
|
||||||
|
raise GuiAccessError(409, ErrorCode.GUI_SLOT_COOLING_DOWN,
|
||||||
|
{"minutes": str(max(1, -(-int(wait.total_seconds()) // 60)))})
|
||||||
|
free = usable
|
||||||
# Prefer the slot this user had this firewall on before, then the slot the user used least recently
|
# Prefer the slot this user had this firewall on before, then the slot the user used least recently
|
||||||
# (an old tab of another firewall on a reused origin is the documented residual risk).
|
# (an old tab of another firewall on a reused origin is the documented residual risk).
|
||||||
history = (await db.execute(
|
history = (await db.execute(
|
||||||
@@ -182,6 +225,13 @@ async def issue_ticket(db: AsyncSession, *, user: AdminUser, session_token: str,
|
|||||||
await db.execute(delete(GuiTicket).where(GuiTicket.expires_at < now - timedelta(hours=1)))
|
await db.execute(delete(GuiTicket).where(GuiTicket.expires_at < now - timedelta(hours=1)))
|
||||||
session_hash = hash_value(session_token)
|
session_hash = hash_value(session_token)
|
||||||
slot = await _pick_slot(db, row, session_hash, user.id, host.id)
|
slot = await _pick_slot(db, row, session_hash, user.id, host.id)
|
||||||
|
# Reopening a slot in the same browser session replaces its earlier gateway session, so that one does not count.
|
||||||
|
open_count = await db.scalar(select(func.count()).select_from(GuiSession).where(
|
||||||
|
GuiSession.ended_at.is_(None),
|
||||||
|
GuiSession.started_at > now - timedelta(minutes=row.absolute_timeout_minutes),
|
||||||
|
~((GuiSession.session_hash == session_hash) & (GuiSession.slot == slot))))
|
||||||
|
if (open_count or 0) >= MAX_GATEWAY_SESSIONS:
|
||||||
|
raise GuiAccessError(409, ErrorCode.GUI_TOO_MANY_SESSIONS, {"max": str(MAX_GATEWAY_SESSIONS)})
|
||||||
raw = secrets.token_urlsafe(32)
|
raw = secrets.token_urlsafe(32)
|
||||||
db.add(GuiTicket(ticket_hash=hash_value(raw), session_hash=session_hash, user_id=user.id, host_id=host.id,
|
db.add(GuiTicket(ticket_hash=hash_value(raw), session_hash=session_hash, user_id=user.id, host_id=host.id,
|
||||||
slot=slot, created_at=now, expires_at=now + TICKET_TTL))
|
slot=slot, created_at=now, expires_at=now + TICKET_TTL))
|
||||||
@@ -255,6 +305,13 @@ async def redeem(db: AsyncSession, *, ticket_hash: str, session_hash: str, slot:
|
|||||||
# The same browser session reopening this slot replaces the earlier gateway session there.
|
# The same browser session reopening this slot replaces the earlier gateway session there.
|
||||||
superseded = (await db.scalars(select(GuiSession).where(
|
superseded = (await db.scalars(select(GuiSession).where(
|
||||||
GuiSession.session_hash == session_hash, GuiSession.slot == slot, GuiSession.ended_at.is_(None)))).all()
|
GuiSession.session_hash == session_hash, GuiSession.slot == slot, GuiSession.ended_at.is_(None)))).all()
|
||||||
|
if any(old.host_id != host.id for old in superseded):
|
||||||
|
# Two quick tickets both saw the slot free: the second must not take over another firewall's origin
|
||||||
|
# without the cooling-down time (S1).
|
||||||
|
await db.commit()
|
||||||
|
await _audit(actor, action="redeem", status="denied", host=host, host_id=ticket.host_id,
|
||||||
|
message="slot", details={"slot": slot, "client_ip": client_ip, "reason": "slot"})
|
||||||
|
return {"ok": False, "reason": "slot"}
|
||||||
for old in superseded:
|
for old in superseded:
|
||||||
old.ended_at, old.end_reason = now, "superseded"
|
old.ended_at, old.end_reason = now, "superseded"
|
||||||
gs = GuiSession(session_hash=session_hash, user_id=user.id, host_id=host.id, slot=slot,
|
gs = GuiSession(session_hash=session_hash, user_id=user.id, host_id=host.id, slot=slot,
|
||||||
@@ -268,6 +325,7 @@ async def redeem(db: AsyncSession, *, ticket_hash: str, session_hash: str, slot:
|
|||||||
"upstream_origin": firewall_origin(host.base_url),
|
"upstream_origin": firewall_origin(host.base_url),
|
||||||
"ca_pem": host.tls_ca_pem or None, # public certificate (pin); None = system trust store
|
"ca_pem": host.tls_ca_pem or None, # public certificate (pin); None = system trust store
|
||||||
"proxy_url": overlay_service.proxy_for(host.base_url),
|
"proxy_url": overlay_service.proxy_for(host.base_url),
|
||||||
|
"target_fp": target_fingerprint(host),
|
||||||
**_timing(row, now),
|
**_timing(row, now),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -285,7 +343,8 @@ async def _finish(db: AsyncSession, gs: GuiSession, reason: str) -> None:
|
|||||||
"duration_seconds": int((gs.ended_at - gs.started_at).total_seconds())})
|
"duration_seconds": int((gs.ended_at - gs.started_at).total_seconds())})
|
||||||
|
|
||||||
|
|
||||||
async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str) -> dict:
|
async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_hash: str,
|
||||||
|
target_fp: str | None = None) -> dict:
|
||||||
gs = await db.get(GuiSession, gui_session_id)
|
gs = await db.get(GuiSession, gui_session_id)
|
||||||
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
if gs is None or not secrets.compare_digest(gs.session_hash, session_hash):
|
||||||
return {"ok": False, "reason": "unknown"}
|
return {"ok": False, "reason": "unknown"}
|
||||||
@@ -295,7 +354,12 @@ async def revalidate(db: AsyncSession, *, gui_session_id: uuid.UUID, session_has
|
|||||||
if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):
|
if _now() >= gs.started_at + timedelta(minutes=row.absolute_timeout_minutes):
|
||||||
await _finish(db, gs, "absolute")
|
await _finish(db, gs, "absolute")
|
||||||
return {"ok": False, "reason": "absolute"}
|
return {"ok": False, "reason": "absolute"}
|
||||||
reason, _, _ = await _live_check(db, row, gs.session_hash, gs.user_id, gs.host_id, gs.slot)
|
reason, _, host = await _live_check(db, row, gs.session_hash, gs.user_id, gs.host_id, gs.slot)
|
||||||
|
# S4: the gateway presents the fingerprint it got at redeem time. Firewall address, pinned certificate or
|
||||||
|
# overlay route changed since -> the session (and the upstream connection it holds) ends now.
|
||||||
|
if reason is None and target_fp is not None and host is not None \
|
||||||
|
and not secrets.compare_digest(target_fp, target_fingerprint(host)):
|
||||||
|
reason = "target"
|
||||||
if reason is not None:
|
if reason is not None:
|
||||||
await _finish(db, gs, reason)
|
await _finish(db, gs, reason)
|
||||||
return {"ok": False, "reason": reason}
|
return {"ok": False, "reason": reason}
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ from app.models import OpnsenseHost, RuleDeployment, AliasDeployment, PortForwar
|
|||||||
from app.models.host_operation import HostOperation
|
from app.models.host_operation import HostOperation
|
||||||
from app.models.restore_operation import RestoreOperation
|
from app.models.restore_operation import RestoreOperation
|
||||||
from app.services import opnsense_client as client, backup_store, carp_maintenance_window_service
|
from app.services import opnsense_client as client, backup_store, carp_maintenance_window_service
|
||||||
|
from app.services.deploy_flow import unsettled
|
||||||
from app.services.host_status_service import get_statuses
|
from app.services.host_status_service import get_statuses
|
||||||
|
|
||||||
DEPLOYMENTS = (RuleDeployment, AliasDeployment, PortForwardDeployment, OneToOneDeployment, OutboundNatDeployment)
|
DEPLOYMENTS = (RuleDeployment, AliasDeployment, PortForwardDeployment, OneToOneDeployment, OutboundNatDeployment)
|
||||||
@@ -34,7 +35,27 @@ async def peers(db, host):
|
|||||||
return [host]
|
return [host]
|
||||||
|
|
||||||
|
|
||||||
async def ensure_no_maintenance(db, hosts):
|
async def ensure_no_open_cleanup(db, hosts):
|
||||||
|
"""A rule cleanup (1.31) that is running, waiting for its confirmation or unresolved owns the host and its
|
||||||
|
cluster peers: nothing else may write to the firewall until it is confirmed, reverted or settled."""
|
||||||
|
from app.services.rule_cleanup_service import has_open_cleanup
|
||||||
|
ids = set()
|
||||||
|
for host in hosts:
|
||||||
|
ids.update(h.id for h in await peers(db, host))
|
||||||
|
if ids and await has_open_cleanup(db, ids):
|
||||||
|
raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)
|
||||||
|
|
||||||
|
|
||||||
|
async def ensure_no_open_cleanup_on(db, model, condition):
|
||||||
|
"""ensure_no_open_cleanup for every host that has a deployment row of ``model`` matching ``condition`` (the
|
||||||
|
undeploy paths: deleting a central object removes it from all its firewalls)."""
|
||||||
|
ids = set((await db.scalars(select(model.host_id).where(condition))).all())
|
||||||
|
hosts = [h for h in [await db.get(OpnsenseHost, i) for i in ids] if h is not None]
|
||||||
|
await ensure_no_open_cleanup(db, hosts)
|
||||||
|
|
||||||
|
|
||||||
|
async def ensure_no_maintenance(db, hosts, *, cleanup=True):
|
||||||
|
"""cleanup=False: only the cleanup service itself, which reports an open cleanup with its own error code."""
|
||||||
ids = set()
|
ids = set()
|
||||||
for host in hosts:
|
for host in hosts:
|
||||||
ids.update(h.id for h in await peers(db, host))
|
ids.update(h.id for h in await peers(db, host))
|
||||||
@@ -42,13 +63,15 @@ async def ensure_no_maintenance(db, hosts):
|
|||||||
raise api_error(409, ErrorCode.OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY)
|
raise api_error(409, ErrorCode.OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY)
|
||||||
if ids and await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
if ids and await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||||
raise api_error(409, ErrorCode.OPEN_RESTORE_BLOCKS_DEPLOY)
|
raise api_error(409, ErrorCode.OPEN_RESTORE_BLOCKS_DEPLOY)
|
||||||
|
if cleanup and ids:
|
||||||
|
await ensure_no_open_cleanup(db, hosts)
|
||||||
|
|
||||||
|
|
||||||
async def _shared_change_blockers(db, fleet):
|
async def _shared_change_blockers(db, fleet):
|
||||||
blocked = []
|
blocked = []
|
||||||
fleet_ids = [h.id for h in fleet]
|
fleet_ids = [h.id for h in fleet]
|
||||||
for model in DEPLOYMENTS:
|
for model in DEPLOYMENTS:
|
||||||
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), model.status.in_(("pending", "revert_failed"))).limit(1)):
|
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), unsettled(model)).limit(1)):
|
||||||
blocked.append("Offene Bestätigung oder fehlgeschlagener Rollback im Host/Cluster.")
|
blocked.append("Offene Bestätigung oder fehlgeschlagener Rollback im Host/Cluster.")
|
||||||
break
|
break
|
||||||
operations = list((await db.scalars(select(HostOperation).where(HostOperation.host_id.in_(fleet_ids), HostOperation.status.in_(ACTIVE)))).all())
|
operations = list((await db.scalars(select(HostOperation).where(HostOperation.host_id.in_(fleet_ids), HostOperation.status.in_(ACTIVE)))).all())
|
||||||
@@ -56,6 +79,9 @@ async def _shared_change_blockers(db, fleet):
|
|||||||
blocked.append("Eine frühere Wartungsaktion im Host/Cluster muss zuerst geprüft und abgeschlossen werden.")
|
blocked.append("Eine frühere Wartungsaktion im Host/Cluster muss zuerst geprüft und abgeschlossen werden.")
|
||||||
if await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(fleet_ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
if await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(fleet_ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||||
blocked.append("Eine Wiederherstellung im Host/Cluster ist offen und muss zuerst bestätigt oder abgebrochen werden.")
|
blocked.append("Eine Wiederherstellung im Host/Cluster ist offen und muss zuerst bestätigt oder abgebrochen werden.")
|
||||||
|
from app.services.rule_cleanup_service import has_open_cleanup
|
||||||
|
if await has_open_cleanup(db, fleet_ids):
|
||||||
|
blocked.append("Eine Regel-Bereinigung im Host/Cluster ist offen und muss zuerst bestätigt oder zurückgenommen werden.")
|
||||||
return blocked
|
return blocked
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -109,6 +109,8 @@ from app.models.ipsec_tunnel import IpsecTunnel
|
|||||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||||
from app.services import opnsense_client, ipsec_psk_store
|
from app.services import opnsense_client, ipsec_psk_store
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
from app.services.deploy_flow import checkpoint
|
||||||
|
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
|
||||||
|
|
||||||
_VERSION_MAP = {"ike": "0", "ikev1": "1", "ikev2": "2"}
|
_VERSION_MAP = {"ike": "0", "ikev1": "1", "ikev2": "2"}
|
||||||
|
|
||||||
@@ -176,7 +178,7 @@ def build_payloads(tunnel: IpsecTunnel) -> dict:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
|
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
|
||||||
"""Runs the 5 config-write calls in dependency order, persisting each
|
"""Runs the 5 config-write calls in dependency order, persisting each
|
||||||
UUID onto `deployment` as soon as it's known so a later retry (after a
|
UUID onto `deployment` as soon as it's known so a later retry (after a
|
||||||
partial failure) updates rather than duplicates. Raises
|
partial failure) updates rather than duplicates. Raises
|
||||||
@@ -189,6 +191,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
|||||||
deployment.opnsense_connection_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_connection_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_connection_uuid:
|
if not deployment.opnsense_connection_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine Connection-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine Connection-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
|
|
||||||
# `payloads` (built by build_payloads()) is also what gets persisted as
|
# `payloads` (built by build_payloads()) is also what gets persisted as
|
||||||
# last_payload for no-op-redeploy comparison — the injected `connection`
|
# last_payload for no-op-redeploy comparison — the injected `connection`
|
||||||
@@ -209,6 +212,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
|||||||
deployment.opnsense_local_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_local_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_local_uuid:
|
if not deployment.opnsense_local_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine Local-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine Local-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
|
|
||||||
if deployment.opnsense_remote_uuid:
|
if deployment.opnsense_remote_uuid:
|
||||||
await opnsense_client.set_ipsec_remote(host, deployment.opnsense_remote_uuid, remote_wire)
|
await opnsense_client.set_ipsec_remote(host, deployment.opnsense_remote_uuid, remote_wire)
|
||||||
@@ -217,6 +221,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
|||||||
deployment.opnsense_remote_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_remote_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_remote_uuid:
|
if not deployment.opnsense_remote_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine Remote-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine Remote-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
|
|
||||||
if deployment.opnsense_child_uuid:
|
if deployment.opnsense_child_uuid:
|
||||||
await opnsense_client.set_ipsec_child(host, deployment.opnsense_child_uuid, child_wire)
|
await opnsense_client.set_ipsec_child(host, deployment.opnsense_child_uuid, child_wire)
|
||||||
@@ -225,6 +230,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
|||||||
deployment.opnsense_child_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_child_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_child_uuid:
|
if not deployment.opnsense_child_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine Child-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine Child-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
|
|
||||||
if deployment.opnsense_psk_uuid:
|
if deployment.opnsense_psk_uuid:
|
||||||
await opnsense_client.set_ipsec_psk(host, deployment.opnsense_psk_uuid, payloads["psk"])
|
await opnsense_client.set_ipsec_psk(host, deployment.opnsense_psk_uuid, payloads["psk"])
|
||||||
@@ -233,6 +239,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
|||||||
deployment.opnsense_psk_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_psk_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_psk_uuid:
|
if not deployment.opnsense_psk_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine PSK-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine PSK-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
|
|
||||||
|
|
||||||
async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseHost) -> IpsecTunnelDeployment:
|
async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseHost) -> IpsecTunnelDeployment:
|
||||||
@@ -241,7 +248,7 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
|||||||
existing = await db.scalar(
|
existing = await db.scalar(
|
||||||
select(IpsecTunnelDeployment).where(
|
select(IpsecTunnelDeployment).where(
|
||||||
IpsecTunnelDeployment.tunnel_id == tunnel.id, IpsecTunnelDeployment.host_id == host.id
|
IpsecTunnelDeployment.tunnel_id == tunnel.id, IpsecTunnelDeployment.host_id == host.id
|
||||||
)
|
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
|
||||||
)
|
)
|
||||||
if existing and existing.status == "success" and existing.last_payload == comparable:
|
if existing and existing.status == "success" and existing.last_payload == comparable:
|
||||||
# Nothing actually changed on this host — re-affirm without
|
# Nothing actually changed on this host — re-affirm without
|
||||||
@@ -252,8 +259,15 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
|||||||
|
|
||||||
deployment = existing or IpsecTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
|
deployment = existing or IpsecTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
|
||||||
|
|
||||||
|
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
|
||||||
|
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
|
||||||
|
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
|
||||||
|
if not existing:
|
||||||
|
db.add(deployment)
|
||||||
|
await checkpoint(db)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
await _write_config(host, payloads, deployment)
|
await _write_config(db, host, payloads, deployment)
|
||||||
# Same trap as WireGuard: with the global IPsec switch off the config
|
# Same trap as WireGuard: with the global IPsec switch off the config
|
||||||
# is stored but nothing runs, and reconfigure still answers ok.
|
# is stored but nothing runs, and reconfigure still answers ok.
|
||||||
await opnsense_client.ensure_ipsec_enabled(host)
|
await opnsense_client.ensure_ipsec_enabled(host)
|
||||||
@@ -261,8 +275,6 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
|||||||
deployment.status = "error"
|
deployment.status = "error"
|
||||||
deployment.message = str(exc)
|
deployment.message = str(exc)
|
||||||
deployment.deployed_at = _now()
|
deployment.deployed_at = _now()
|
||||||
if not existing:
|
|
||||||
db.add(deployment)
|
|
||||||
return deployment
|
return deployment
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -277,32 +289,37 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
|||||||
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
||||||
deployment.last_payload = comparable
|
deployment.last_payload = comparable
|
||||||
deployment.deployed_at = _now()
|
deployment.deployed_at = _now()
|
||||||
if not existing:
|
|
||||||
db.add(deployment)
|
|
||||||
return deployment
|
return deployment
|
||||||
|
|
||||||
deployment.status = "success"
|
deployment.status = "success"
|
||||||
deployment.message = None
|
deployment.message = None
|
||||||
deployment.last_payload = comparable
|
deployment.last_payload = comparable
|
||||||
deployment.deployed_at = _now()
|
deployment.deployed_at = _now()
|
||||||
if not existing:
|
|
||||||
db.add(deployment)
|
|
||||||
return deployment
|
return deployment
|
||||||
|
|
||||||
|
|
||||||
async def deploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel, hosts: list[OpnsenseHost]) -> list[IpsecTunnelDeployment]:
|
async def deploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel, hosts: list[OpnsenseHost]) -> list[IpsecTunnelDeployment]:
|
||||||
from app.services.host_maintenance_service import ensure_no_maintenance
|
from app.services.host_maintenance_service import ensure_no_maintenance
|
||||||
await ensure_no_maintenance(db, hosts)
|
await ensure_no_maintenance(db, hosts)
|
||||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
async with resource_lease("ipsec", tunnel.id):
|
||||||
await db.commit()
|
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||||
|
await db.commit()
|
||||||
return deployments
|
return deployments
|
||||||
|
|
||||||
|
|
||||||
async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||||
|
async with resource_lease("ipsec", tunnel.id):
|
||||||
|
await _undeploy_tunnel(db, tunnel)
|
||||||
|
|
||||||
|
|
||||||
|
async def _undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||||
"""Explicit action, not gated by anything (there is no lockout state
|
"""Explicit action, not gated by anything (there is no lockout state
|
||||||
for this entity to gate on) — deletes Child/Local/Remote before their
|
for this entity to gate on) — deletes Child/Local/Remote before their
|
||||||
parent Connection (the ModelRelationField dependency direction), then
|
parent Connection (the ModelRelationField dependency direction), then
|
||||||
the PreSharedKey (independent of the other 4), then reconfigures."""
|
the PreSharedKey (independent of the other 4), then reconfigures."""
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||||
|
await ensure_no_open_cleanup_on(db, IpsecTunnelDeployment, IpsecTunnelDeployment.tunnel_id == tunnel.id)
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(IpsecTunnelDeployment).where(IpsecTunnelDeployment.tunnel_id == tunnel.id))
|
result = await db.execute(select(IpsecTunnelDeployment).where(IpsecTunnelDeployment.tunnel_id == tunnel.id))
|
||||||
for deployment in result.scalars().all():
|
for deployment in result.scalars().all():
|
||||||
|
|||||||
@@ -21,6 +21,7 @@ from app.models.compliance_result import ComplianceResult
|
|||||||
from app.models.host import OpnsenseHost
|
from app.models.host import OpnsenseHost
|
||||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||||
from app.models.resource_sample import HostResourceSample
|
from app.models.resource_sample import HostResourceSample
|
||||||
|
from app.models.rollout import RolloutRun
|
||||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||||
from app.routers.dashboard import collect_pending_actions
|
from app.routers.dashboard import collect_pending_actions
|
||||||
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
||||||
@@ -105,6 +106,7 @@ async def render(db: AsyncSession) -> str:
|
|||||||
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
|
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
|
||||||
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
|
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
|
||||||
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
|
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
|
||||||
|
out.family("opnexus_host_api_rights_missing", "1 if the firewall refused a side reading (read=gateways, services, swap, cpu) because the API key lacks the right (last sample); absent otherwise.")
|
||||||
configured = (await notification_settings_service.get_settings(db)).service_watchlist
|
configured = (await notification_settings_service.get_settings(db)).service_watchlist
|
||||||
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
|
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
|
||||||
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
|
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
|
||||||
@@ -136,6 +138,8 @@ async def render(db: AsyncSession) -> str:
|
|||||||
):
|
):
|
||||||
if value is not None:
|
if value is not None:
|
||||||
out.sample(name, round(value * factor, 6), **gw_labels)
|
out.sample(name, round(value * factor, 6), **gw_labels)
|
||||||
|
for read in sorted(str(r) for r in raw.get("denied") or []):
|
||||||
|
out.sample("opnexus_host_api_rights_missing", 1, **labels, read=read)
|
||||||
for key, running in sorted((raw.get("services") or {}).items()):
|
for key, running in sorted((raw.get("services") or {}).items()):
|
||||||
if key.split("/")[0] in watchlist:
|
if key.split("/")[0] in watchlist:
|
||||||
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
|
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
|
||||||
@@ -176,4 +180,8 @@ async def render(db: AsyncSession) -> str:
|
|||||||
pending = await db.scalar(select(func.count()).select_from(ScheduledMaintenance).where(ScheduledMaintenance.status == "pending"))
|
pending = await db.scalar(select(func.count()).select_from(ScheduledMaintenance).where(ScheduledMaintenance.status == "pending"))
|
||||||
out.sample("opnexus_scheduled_maintenance_pending", pending or 0)
|
out.sample("opnexus_scheduled_maintenance_pending", pending or 0)
|
||||||
|
|
||||||
|
out.family("opnexus_rollout_runs", "Fleet rollout runs by status (waiting_window, wave_running, paused, succeeded, ...).")
|
||||||
|
for status, count in sorted((await db.execute(select(RolloutRun.status, func.count()).group_by(RolloutRun.status))).all()):
|
||||||
|
out.sample("opnexus_rollout_runs", count, status=status)
|
||||||
|
|
||||||
return out.text()
|
return out.text()
|
||||||
@@ -97,6 +97,7 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
|||||||
cfg = await notification_settings_service.get_settings(db)
|
cfg = await notification_settings_service.get_settings(db)
|
||||||
confirm = max(1, cfg.alert_confirm_samples)
|
confirm = max(1, cfg.alert_confirm_samples)
|
||||||
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
|
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
|
||||||
|
muted = set(cfg.gateway_alert_muted or [])
|
||||||
open_keys = await _open_keys(db, "gateway_down")
|
open_keys = await _open_keys(db, "gateway_down")
|
||||||
problems = Problems()
|
problems = Problems()
|
||||||
for host in await _enabled_hosts(db):
|
for host in await _enabled_hosts(db):
|
||||||
@@ -110,6 +111,8 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
|||||||
]
|
]
|
||||||
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
|
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
|
||||||
for name in sorted(names):
|
for name in sorted(names):
|
||||||
|
if f"{host.id}:{name}" in muted:
|
||||||
|
continue # alert switched off for this gateway: not judged, an open problem of it resolves
|
||||||
flags: list[bool | None] = []
|
flags: list[bool | None] = []
|
||||||
for p in parsed:
|
for p in parsed:
|
||||||
if p is None:
|
if p is None:
|
||||||
@@ -131,6 +134,58 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
|||||||
return problems
|
return problems
|
||||||
|
|
||||||
|
|
||||||
|
# ── API rights ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
# What the sampler reads on the side (resource_monitor_service) -> (ACL entry of the OPNsense user/group that
|
||||||
|
# grants it, endpoint; the sample's "ok" list proves a reading went through). ACL names: opnsense/core
|
||||||
|
# src/opnsense/mvc/app/models/OPNsense/Core/ACL/ACL.xml (page-system-gateways: api/routes/gateway/status;
|
||||||
|
# page-status-services: api/core/service/*; page-system-login-logout "Lobby: Dashboard": api/diagnostics/cpu_usage/*
|
||||||
|
# and api/diagnostics/system/system_swap).
|
||||||
|
API_READS: dict[str, tuple[str, str]] = {
|
||||||
|
"gateways": ("System: Gateways", "/api/routes/gateway/status"),
|
||||||
|
"services": ("Status: Services", "/api/core/service/search"),
|
||||||
|
"swap": ("Lobby: Dashboard", "/api/diagnostics/system/systemSwap"),
|
||||||
|
"cpu": ("Lobby: Dashboard", "/api/diagnostics/cpu_usage/getCPUType"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def check_api_rights_missing(db: AsyncSession) -> dict[str, dict]:
|
||||||
|
"""The API key of a firewall may be refused (403) on the side readings the alerts rely on (gateway state,
|
||||||
|
service state): those alerts can then never fire, and nobody would notice. Only a refusal counts; a timeout
|
||||||
|
or an error stays "unknown"."""
|
||||||
|
cfg = await notification_settings_service.get_settings(db)
|
||||||
|
confirm = max(1, cfg.alert_confirm_samples)
|
||||||
|
open_keys = await _open_keys(db, "api_rights_missing")
|
||||||
|
problems = Problems()
|
||||||
|
for host in await _enabled_hosts(db):
|
||||||
|
samples = await _recent(db, host.id, confirm, "denied", "ok")
|
||||||
|
if not _fresh(samples):
|
||||||
|
problems.preserve_prefixes.add(f"{host.id}:")
|
||||||
|
continue
|
||||||
|
was_open = _keys_of_host(open_keys, host)
|
||||||
|
names = {n for s in samples for n in (s["denied"] or []) if n in API_READS} | (was_open & set(API_READS))
|
||||||
|
for name in sorted(names):
|
||||||
|
acl, path = API_READS[name]
|
||||||
|
flags: list[bool | None] = []
|
||||||
|
for s in samples:
|
||||||
|
if s["denied"] is None:
|
||||||
|
flags.append(None) # a sample from before this check existed
|
||||||
|
elif name in s["denied"]:
|
||||||
|
flags.append(True)
|
||||||
|
else:
|
||||||
|
# Only a reading that went through proves the right is back; a timeout or a sample from
|
||||||
|
# before "ok" existed stays unknown (the problem neither opens nor closes on it).
|
||||||
|
flags.append(False if s["ok"] is not None and name in s["ok"] else None)
|
||||||
|
if verdict(flags, confirm, name in was_open):
|
||||||
|
problems[f"{host.id}:{name}"] = {
|
||||||
|
"host_id": str(host.id), "host_name": host.name, "lesezugriff": name, "endpunkt": path,
|
||||||
|
"fehlendes_recht": acl,
|
||||||
|
"hinweis": f"Dem API-Benutzer fehlt das Recht, {path} zu lesen (OPNsense-Berechtigung \"{acl}\"). "
|
||||||
|
"Alarme, die darauf beruhen, können nicht auslösen.",
|
||||||
|
}
|
||||||
|
return problems
|
||||||
|
|
||||||
|
|
||||||
# ── Resources ───────────────────────────────────────────────────────────────
|
# ── Resources ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def disk_groups(devices) -> dict[str, tuple[str, float]]:
|
def disk_groups(devices) -> dict[str, tuple[str, float]]:
|
||||||
@@ -183,7 +238,9 @@ async def check_resource_threshold(db: AsyncSession) -> dict[str, dict]:
|
|||||||
|
|
||||||
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
|
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
|
||||||
if verdict(flags, confirm, key in was_open):
|
if verdict(flags, confirm, key in was_open):
|
||||||
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": f"{value}{unit}", "schwelle": f"{threshold}{unit}"}
|
# An open alert stays open while the reading is unavailable: then there is no number to show (never "None %").
|
||||||
|
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": None if value is None else f"{value}{unit}",
|
||||||
|
"schwelle": f"{threshold}{unit}"}
|
||||||
|
|
||||||
latest = samples[0]
|
latest = samples[0]
|
||||||
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
|
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
|
||||||
|
|||||||
@@ -87,6 +87,13 @@ OBJECT_COLUMNS: dict[str, list[str]] = {
|
|||||||
# These change how the network is wired; the others (DNS/DHCP data) cannot cut the connection to the firewall.
|
# These change how the network is wired; the others (DNS/DHCP data) cannot cut the connection to the firewall.
|
||||||
WIRING_KINDS = {"vlan", "vip", "gateway", "route", "interface", "lagg", "bridge", "gre", "gif", "vxlan"}
|
WIRING_KINDS = {"vlan", "vip", "gateway", "route", "interface", "lagg", "bridge", "gre", "gif", "vxlan"}
|
||||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||||
|
# Integer ranges of the firewall's own model (Interfaces/Vlan.xml, Interfaces/Vip.xml, read 2026-10-23). Checked before the
|
||||||
|
# preview so a wrong number is a 422 with the field name, not a 502 after the firewall refused it (T6 of the 1.31.0 test
|
||||||
|
# run). Only fields whose range is stated in the model; an empty value is left to the firewall (e.g. vhid of a non-CARP VIP).
|
||||||
|
RANGES: dict[tuple[str, str], tuple[int, int]] = {
|
||||||
|
("vlan", "tag"): (1, 4094), ("vlan", "pcp"): (0, 7),
|
||||||
|
("vip", "vhid"): (1, 255), ("vip", "advbase"): (1, 254), ("vip", "advskew"): (0, 254),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
||||||
@@ -104,6 +111,9 @@ def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
|||||||
text = str(value).strip()
|
text = str(value).strip()
|
||||||
if len(text) > 255 or _CONTROL.search(text):
|
if len(text) > 255 or _CONTROL.search(text):
|
||||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
||||||
|
bounds = RANGES.get((kind, key))
|
||||||
|
if bounds and text and not (text.isdecimal() and bounds[0] <= int(text) <= bounds[1]):
|
||||||
|
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
||||||
clean[key] = text
|
clean[key] = text
|
||||||
return clean
|
return clean
|
||||||
|
|
||||||
|
|||||||
@@ -224,14 +224,14 @@ async def send(channel: NotificationChannel, problem: NotificationProblem, trans
|
|||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
||||||
try:
|
try:
|
||||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||||
resp = await client.request(method, url, json=payload, headers=headers)
|
resp = await client.request(method, url, json=payload, headers=headers)
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
if ok_key and not (resp.json() or {}).get(ok_key):
|
if ok_key and not (resp.json() or {}).get(ok_key):
|
||||||
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
|
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
|
||||||
except httpx.HTTPError as exc:
|
except httpx.HTTPError as exc:
|
||||||
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
|
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
|
||||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
reason = url_guard.failure_reason(exc)
|
||||||
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
|
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
|
||||||
except ValueError:
|
except ValueError:
|
||||||
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
|
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
|
||||||
|
|||||||
@@ -201,6 +201,62 @@ async def check_revert_failed(db: AsyncSession) -> dict[str, dict]:
|
|||||||
"host_name": host.name if host else None,
|
"host_name": host.name if host else None,
|
||||||
"message": deployment.message,
|
"message": deployment.message,
|
||||||
}
|
}
|
||||||
|
problems.update(await _cleanup_problems(db))
|
||||||
|
return problems
|
||||||
|
|
||||||
|
|
||||||
|
# Rule cleanup (1.31) states that need a human: the automatic revert failed, or a run ended half-way.
|
||||||
|
_CLEANUP_STILL_CHANGED = ("revert_failed", "unknown", "disabling", "disabled", "failed")
|
||||||
|
|
||||||
|
|
||||||
|
async def _cleanup_problems(db: AsyncSession) -> dict[str, dict]:
|
||||||
|
"""Rule cleanups whose automatic revert failed (per rule that could not be switched back) and runs stuck in
|
||||||
|
``partial``/``apply_failed`` (a decision is needed before the window ends), plus delete/restore runs with an
|
||||||
|
unresolved item. Same ``revert_failed`` alert as the deployments. Keys: ``rule_cleanup:<item id>`` for one rule,
|
||||||
|
``rule_cleanup_op:<operation id>`` for a whole run, so one open problem is one message."""
|
||||||
|
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
|
||||||
|
|
||||||
|
problems: dict[str, dict] = {}
|
||||||
|
operations = (await db.scalars(select(RuleCleanupOperation).where(
|
||||||
|
RuleCleanupOperation.status.in_(("revert_failed", "partial", "apply_failed", "failed"))))).all()
|
||||||
|
for op in operations:
|
||||||
|
items = list((await db.scalars(select(RuleCleanupItem).where(RuleCleanupItem.operation_id == op.id)
|
||||||
|
.order_by(RuleCleanupItem.seq))).all())
|
||||||
|
host = await db.get(OpnsenseHost, op.host_id)
|
||||||
|
base = {
|
||||||
|
"deployment_type": "rule_cleanup", "deployment_id": str(op.id), "host_id": str(op.host_id),
|
||||||
|
"host_name": host.name if host else None, "operation": op.kind, "operation_status": op.status,
|
||||||
|
}
|
||||||
|
if op.status == "revert_failed":
|
||||||
|
stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]
|
||||||
|
for item in stuck:
|
||||||
|
member = await db.get(OpnsenseHost, item.host_id)
|
||||||
|
problems[f"rule_cleanup:{item.id}"] = {
|
||||||
|
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||||
|
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||||
|
"message": item.message or op.message,
|
||||||
|
"hinweis": "Bereinigung: Die Regel konnte nicht automatisch zurückgesetzt werden und ist möglicherweise "
|
||||||
|
"noch geändert. Auf der Firewall prüfen und in OPNexus „Erneut zurücknehmen“ wählen.",
|
||||||
|
}
|
||||||
|
if not stuck:
|
||||||
|
problems[f"rule_cleanup_op:{op.id}"] = {**base, "entity_id": str(op.id), "message": op.message,
|
||||||
|
"hinweis": "Bereinigung: Das automatische Zurücknehmen ist fehlgeschlagen. Zustand der Firewall prüfen."}
|
||||||
|
elif op.status in ("partial", "apply_failed"):
|
||||||
|
problems[f"rule_cleanup_op:{op.id}"] = {
|
||||||
|
**base, "entity_id": str(op.id), "message": op.message,
|
||||||
|
"regeln": ", ".join(i.rule_uuid for i in items if i.state in ("failed", "unknown")) or None,
|
||||||
|
"hinweis": "Bereinigung nur teilweise ausgeführt (oder das Anwenden ist fehlgeschlagen). Bestätigen ist gesperrt; "
|
||||||
|
"jetzt zurücknehmen oder die automatische Rücknahme am Fristende abwarten.",
|
||||||
|
}
|
||||||
|
elif op.kind in ("delete", "restore"):
|
||||||
|
for item in (i for i in items if i.state == "unknown"):
|
||||||
|
member = await db.get(OpnsenseHost, item.host_id)
|
||||||
|
problems[f"rule_cleanup:{item.id}"] = {
|
||||||
|
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||||
|
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||||
|
"message": item.message or op.message,
|
||||||
|
"hinweis": "Bereinigung: Der Ausgang beim Löschen/Wiederherstellen ist unklar. Mit „Erneut prüfen“ nachlesen.",
|
||||||
|
}
|
||||||
return problems
|
return problems
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -40,6 +40,7 @@ _LABELS = {
|
|||||||
"gateway_down": "Gateway gestört",
|
"gateway_down": "Gateway gestört",
|
||||||
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
||||||
"service_down": "Dienst ausgefallen",
|
"service_down": "Dienst ausgefallen",
|
||||||
|
"api_rights_missing": "API-Rechte fehlen",
|
||||||
"rollout_paused": "Rollout angehalten",
|
"rollout_paused": "Rollout angehalten",
|
||||||
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
||||||
"test": "Testbenachrichtigung",
|
"test": "Testbenachrichtigung",
|
||||||
@@ -96,14 +97,14 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
|
|||||||
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
||||||
try:
|
try:
|
||||||
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
||||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||||
with url_guard.muted():
|
with url_guard.muted():
|
||||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
except httpx.HTTPError as exc:
|
except httpx.HTTPError as exc:
|
||||||
# str(exc) names the full URL, which often is the credential itself
|
# str(exc) names the full URL, which often is the credential itself
|
||||||
# (token in path/query) — keep it out of messages and logs.
|
# (token in path/query) — keep it out of messages and logs.
|
||||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
reason = url_guard.failure_reason(exc)
|
||||||
raise NotificationDeliveryError(f"Webhook-Zustellung fehlgeschlagen ({reason})") from None
|
raise NotificationDeliveryError(f"Webhook-Zustellung fehlgeschlagen ({reason})") from None
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -3,9 +3,11 @@ against previously open problems, and delivers on transitions. The single
|
|||||||
testable entry point — called directly by main.py's background loop AND
|
testable entry point — called directly by main.py's background loop AND
|
||||||
directly by tests, same pattern as rule_deploy_service.sweep_expired_deployments.
|
directly by tests, same pattern as rule_deploy_service.sweep_expired_deployments.
|
||||||
"""
|
"""
|
||||||
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||||
@@ -37,6 +39,7 @@ _CHECKS = [
|
|||||||
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
|
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
|
||||||
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
||||||
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
||||||
|
("api_rights_missing", "api_rights_enabled", monitor_checks.check_api_rights_missing),
|
||||||
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
||||||
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
||||||
]
|
]
|
||||||
@@ -46,44 +49,95 @@ _CHECKS = [
|
|||||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
|
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
|
||||||
|
|
||||||
|
|
||||||
|
# Delivery limits (1.29): a hanging channel must neither delay the others nor hold up the check loop.
|
||||||
|
DELIVERY_TIMEOUT_SECONDS = 20.0 # one target (webhook, e-mail, one channel) for one message
|
||||||
|
DELIVERY_CYCLE_BUDGET_SECONDS = 90.0 # everything one check cycle sends, in total
|
||||||
|
DELIVERY_PARALLEL = 10 # targets served at the same time within a cycle
|
||||||
|
|
||||||
|
|
||||||
|
async def _attempt(label: str, make_call, gate: asyncio.Semaphore | None = None, started: list | None = None) -> None:
|
||||||
|
"""One delivery target: its own timeout, its own failure — never an exception for the caller. Texts logged
|
||||||
|
here come from NotificationDeliveryError (names the channel and the HTTP status, never the URL or token)."""
|
||||||
|
try:
|
||||||
|
if gate is not None:
|
||||||
|
await gate.acquire()
|
||||||
|
if started is not None:
|
||||||
|
started.append(label) # the target was really tried (a cancelled wait for the gate does not count)
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(make_call(), DELIVERY_TIMEOUT_SECONDS)
|
||||||
|
finally:
|
||||||
|
if gate is not None:
|
||||||
|
gate.release()
|
||||||
|
except notification_delivery.NotificationDeliveryError:
|
||||||
|
logger.exception("%s delivery failed", label)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
logger.warning("%s delivery timed out after %.0f s", label, DELIVERY_TIMEOUT_SECONDS)
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
raise
|
||||||
|
except Exception as exc: # a broken target must not take the others down
|
||||||
|
logger.warning("%s delivery failed unexpectedly (%s)", label, type(exc).__name__)
|
||||||
|
|
||||||
|
|
||||||
async def _deliver(
|
async def _deliver(
|
||||||
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
|
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
|
||||||
|
gate: asyncio.Semaphore | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Best-effort, not an audit trail: attempted once per transition and
|
"""Best-effort, not an audit trail: attempted once per transition and
|
||||||
never retried, even on failure — avoids retry-storming a broken webhook
|
never retried, even on failure — avoids retry-storming a broken webhook
|
||||||
URL or SMTP relay every poll cycle."""
|
URL or SMTP relay every poll cycle. Webhook, e-mail and every channel are
|
||||||
|
served in parallel, each with its own timeout (the order has no meaning)."""
|
||||||
if transition == "resolved" and problem.problem_type in _NO_RESOLVED_NOTICE:
|
if transition == "resolved" and problem.problem_type in _NO_RESOLVED_NOTICE:
|
||||||
problem.resolved_notified_at = _now()
|
problem.resolved_notified_at = _now()
|
||||||
return
|
return
|
||||||
|
where = f"{problem.problem_type}/{problem.dedup_key}"
|
||||||
|
calls = []
|
||||||
|
started: list = []
|
||||||
if settings_row.webhook_enabled:
|
if settings_row.webhook_enabled:
|
||||||
try:
|
calls.append(_attempt(f"Webhook for {where}", lambda: notification_delivery.send_webhook(settings_row, problem, transition), gate, started))
|
||||||
await notification_delivery.send_webhook(settings_row, problem, transition)
|
|
||||||
except notification_delivery.NotificationDeliveryError:
|
|
||||||
logger.exception("Webhook delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
|
||||||
if settings_row.email_enabled:
|
if settings_row.email_enabled:
|
||||||
try:
|
async def email():
|
||||||
subject, body = notification_delivery.render_message(problem, transition)
|
subject, body = notification_delivery.render_message(problem, transition)
|
||||||
await notification_delivery.send_email(settings_row, subject, body)
|
await notification_delivery.send_email(settings_row, subject, body)
|
||||||
except notification_delivery.NotificationDeliveryError:
|
calls.append(_attempt(f"Email for {where}", email, gate, started))
|
||||||
logger.exception("Email delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
|
||||||
|
|
||||||
for channel in channels:
|
for channel in channels:
|
||||||
try:
|
calls.append(_attempt(f"Channel {channel.name} ({channel.channel_type}) for {where}",
|
||||||
await notification_channels.send(channel, problem, transition)
|
lambda channel=channel: notification_channels.send(channel, problem, transition), gate, started))
|
||||||
except notification_delivery.NotificationDeliveryError:
|
try:
|
||||||
# The error text names the channel and the HTTP status, never the URL or token.
|
await asyncio.gather(*calls, return_exceptions=True)
|
||||||
logger.exception("Channel %s (%s) delivery failed for %s/%s", channel.name, channel.channel_type, problem.problem_type, problem.dedup_key)
|
finally:
|
||||||
|
# "Attempted" is what is recorded (also when a target failed or timed out). A message the cycle budget
|
||||||
|
# cancelled before ANY target was tried stays unrecorded: an opened problem is sent again next cycle.
|
||||||
|
if started or not calls:
|
||||||
|
if transition == "resolved":
|
||||||
|
problem.resolved_notified_at = _now()
|
||||||
|
else:
|
||||||
|
problem.last_notified_at = _now()
|
||||||
|
|
||||||
if transition == "resolved":
|
|
||||||
problem.resolved_notified_at = _now()
|
async def _deliver_all(settings_row: NotificationSettings, pending: list[tuple[NotificationProblem, str]],
|
||||||
else:
|
channels: list[NotificationChannel]) -> None:
|
||||||
problem.last_notified_at = _now()
|
"""Everything this cycle found, in parallel, within DELIVERY_CYCLE_BUDGET_SECONDS in total."""
|
||||||
|
if not pending:
|
||||||
|
return
|
||||||
|
gate = asyncio.Semaphore(DELIVERY_PARALLEL)
|
||||||
|
tasks = [asyncio.ensure_future(_deliver(settings_row, problem, transition, channels, gate)) for problem, transition in pending]
|
||||||
|
done, late = await asyncio.wait(tasks, timeout=DELIVERY_CYCLE_BUDGET_SECONDS)
|
||||||
|
for task in late:
|
||||||
|
task.cancel()
|
||||||
|
if late:
|
||||||
|
logger.warning("Notification delivery stopped after %.0f s: %d of %d messages were not completed",
|
||||||
|
DELIVERY_CYCLE_BUDGET_SECONDS, len(late), len(tasks))
|
||||||
|
await asyncio.gather(*late, return_exceptions=True) # let the cancelled ones record "attempted"
|
||||||
|
for task in done:
|
||||||
|
if not task.cancelled() and task.exception() is not None:
|
||||||
|
logger.warning("Notification delivery failed unexpectedly (%s)", type(task.exception()).__name__)
|
||||||
|
|
||||||
|
|
||||||
async def run_notification_checks(db: AsyncSession) -> dict:
|
async def run_notification_checks(db: AsyncSession) -> dict:
|
||||||
settings_row = await notification_settings_service.get_settings(db)
|
settings_row = await notification_settings_service.get_settings(db)
|
||||||
result = {"opened": 0, "resolved": 0}
|
result = {"opened": 0, "resolved": 0}
|
||||||
channels = await notification_channels.enabled_channels(db)
|
channels = await notification_channels.enabled_channels(db)
|
||||||
|
pending: list[tuple[NotificationProblem, str]] = []
|
||||||
|
|
||||||
for problem_type, enabled_flag, check_fn in _CHECKS:
|
for problem_type, enabled_flag, check_fn in _CHECKS:
|
||||||
if not getattr(settings_row, enabled_flag):
|
if not getattr(settings_row, enabled_flag):
|
||||||
@@ -99,12 +153,21 @@ async def run_notification_checks(db: AsyncSession) -> dict:
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
|
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
|
||||||
for problem in opened:
|
pending += [(problem, "opened") for problem in opened] + [(problem, "resolved") for problem in resolved]
|
||||||
await _deliver(settings_row, problem, "opened", channels)
|
|
||||||
for problem in resolved:
|
|
||||||
await _deliver(settings_row, problem, "resolved", channels)
|
|
||||||
result["opened"] += len(opened)
|
result["opened"] += len(opened)
|
||||||
result["resolved"] += len(resolved)
|
result["resolved"] += len(resolved)
|
||||||
|
|
||||||
|
# Open problems that no target was ever tried for (an earlier cycle ran out of time) are sent now.
|
||||||
|
enabled_types = [t for t, flag, _fn in _CHECKS if getattr(settings_row, flag)]
|
||||||
|
already = {id(problem) for problem, _transition in pending}
|
||||||
|
undelivered = (await db.scalars(
|
||||||
|
select(NotificationProblem).where(
|
||||||
|
NotificationProblem.status == "open", NotificationProblem.last_notified_at.is_(None),
|
||||||
|
NotificationProblem.problem_type.in_(enabled_types),
|
||||||
|
)
|
||||||
|
)).all() if enabled_types else []
|
||||||
|
pending += [(problem, "opened") for problem in undelivered if id(problem) not in already]
|
||||||
|
# Delivered once all checks ran (and in parallel): a slow target no longer holds up the checks after it.
|
||||||
|
await _deliver_all(settings_row, pending, channels)
|
||||||
await db.commit()
|
await db.commit()
|
||||||
return result
|
return result
|
||||||
@@ -26,7 +26,7 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
|
|||||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
|
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
|
||||||
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
||||||
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled",
|
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled", "api_rights_enabled",
|
||||||
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
||||||
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
||||||
"service_watchlist",
|
"service_watchlist",
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||||||
|
|
||||||
from app.models.oidc import OidcLoginState, OidcSettings
|
from app.models.oidc import OidcLoginState, OidcSettings
|
||||||
from app.models.user import AdminUser
|
from app.models.user import AdminUser
|
||||||
from app.services import notification_secret_store
|
from app.services import notification_secret_store, url_guard
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -77,7 +77,7 @@ async def _get_json(url: str, **kw) -> dict:
|
|||||||
if not url.startswith("https://"):
|
if not url.startswith("https://"):
|
||||||
raise OidcError("provider", "Nur https:// wird akzeptiert")
|
raise OidcError("provider", "Nur https:// wird akzeptiert")
|
||||||
try:
|
try:
|
||||||
async with httpx.AsyncClient(timeout=10, follow_redirects=False) as client:
|
async with url_guard.guarded_client(timeout=10, follow_redirects=False) as client:
|
||||||
response = await client.get(url, **kw)
|
response = await client.get(url, **kw)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
data = response.json()
|
data = response.json()
|
||||||
@@ -168,7 +168,7 @@ async def _exchange(cfg: OidcSettings, doc: dict, code: str, state: OidcLoginSta
|
|||||||
else:
|
else:
|
||||||
data["client_secret"] = secret
|
data["client_secret"] = secret
|
||||||
try:
|
try:
|
||||||
async with httpx.AsyncClient(timeout=15, follow_redirects=False) as client:
|
async with url_guard.guarded_client(timeout=15, follow_redirects=False) as client:
|
||||||
response = await client.post(doc["token_endpoint"], data=data, auth=auth)
|
response = await client.post(doc["token_endpoint"], data=data, auth=auth)
|
||||||
if response.status_code >= 400:
|
if response.status_code >= 400:
|
||||||
raise OidcError("token", f"Token-Endpunkt antwortet mit HTTP {response.status_code}")
|
raise OidcError("token", f"Token-Endpunkt antwortet mit HTTP {response.status_code}")
|
||||||
|
|||||||
@@ -135,6 +135,9 @@ async def sweep_expired_one_to_one_deployments(db: AsyncSession) -> list[OneToOn
|
|||||||
|
|
||||||
|
|
||||||
async def undeploy_one_to_one(db: AsyncSession, rule: OneToOneRule) -> None:
|
async def undeploy_one_to_one(db: AsyncSession, rule: OneToOneRule) -> None:
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||||
|
await ensure_no_open_cleanup_on(db, OneToOneDeployment, OneToOneDeployment.one_to_one_id == rule.id)
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(OneToOneDeployment).where(OneToOneDeployment.one_to_one_id == rule.id))
|
result = await db.execute(select(OneToOneDeployment).where(OneToOneDeployment.one_to_one_id == rule.id))
|
||||||
for deployment in result.scalars().all():
|
for deployment in result.scalars().all():
|
||||||
|
|||||||
@@ -165,6 +165,8 @@ from app.models.openvpn_instance import OpenVpnInstance
|
|||||||
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||||
from app.services import opnsense_client, openvpn_key_store
|
from app.services import opnsense_client, openvpn_key_store
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError
|
||||||
|
from app.services.deploy_flow import checkpoint
|
||||||
|
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
|
||||||
|
|
||||||
|
|
||||||
def _now() -> datetime:
|
def _now() -> datetime:
|
||||||
@@ -252,7 +254,7 @@ async def _next_vpnid(host: OpnsenseHost) -> str:
|
|||||||
return str(max(used, default=0) + 1)
|
return str(max(used, default=0) + 1)
|
||||||
|
|
||||||
|
|
||||||
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnInstanceDeployment) -> None:
|
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: OpenVpnInstanceDeployment) -> None:
|
||||||
"""Writes the StaticKey (if enabled) then the Instance, in that
|
"""Writes the StaticKey (if enabled) then the Instance, in that
|
||||||
dependency order, persisting each UUID onto `deployment` as soon as
|
dependency order, persisting each UUID onto `deployment` as soon as
|
||||||
it's known so a later retry (after a partial failure) updates rather
|
it's known so a later retry (after a partial failure) updates rather
|
||||||
@@ -267,11 +269,13 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnI
|
|||||||
deployment.opnsense_statickey_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_statickey_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_statickey_uuid:
|
if not deployment.opnsense_statickey_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine StaticKey-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine StaticKey-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
elif deployment.opnsense_statickey_uuid:
|
elif deployment.opnsense_statickey_uuid:
|
||||||
# tls_key hardening was switched off after previously being on —
|
# tls_key hardening was switched off after previously being on —
|
||||||
# remove the now-orphaned StaticKey rather than leaving it behind.
|
# remove the now-orphaned StaticKey rather than leaving it behind.
|
||||||
await opnsense_client.del_openvpn_static_key(host, deployment.opnsense_statickey_uuid)
|
await opnsense_client.del_openvpn_static_key(host, deployment.opnsense_statickey_uuid)
|
||||||
deployment.opnsense_statickey_uuid = None
|
deployment.opnsense_statickey_uuid = None
|
||||||
|
await checkpoint(db)
|
||||||
|
|
||||||
# Overlay the deploy-time-known tls_key UUID (or "" to explicitly
|
# Overlay the deploy-time-known tls_key UUID (or "" to explicitly
|
||||||
# clear a previously-set relation — see module docstring) onto a
|
# clear a previously-set relation — see module docstring) onto a
|
||||||
@@ -292,13 +296,14 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnI
|
|||||||
deployment.opnsense_instance_uuid = result.get("uuid") if isinstance(result, dict) else None
|
deployment.opnsense_instance_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||||
if not deployment.opnsense_instance_uuid:
|
if not deployment.opnsense_instance_uuid:
|
||||||
raise OpnsenseAPIError("Firewall hat keine Instance-UUID zurückgegeben; Stand manuell prüfen")
|
raise OpnsenseAPIError("Firewall hat keine Instance-UUID zurückgegeben; Stand manuell prüfen")
|
||||||
|
await checkpoint(db) # the UUID is durable before the next remote call
|
||||||
|
|
||||||
|
|
||||||
async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: OpnsenseHost) -> OpenVpnInstanceDeployment:
|
async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: OpnsenseHost) -> OpenVpnInstanceDeployment:
|
||||||
existing = await db.scalar(
|
existing = await db.scalar(
|
||||||
select(OpenVpnInstanceDeployment).where(
|
select(OpenVpnInstanceDeployment).where(
|
||||||
OpenVpnInstanceDeployment.instance_id == instance.id, OpenVpnInstanceDeployment.host_id == host.id
|
OpenVpnInstanceDeployment.instance_id == instance.id, OpenVpnInstanceDeployment.host_id == host.id
|
||||||
)
|
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
payloads = build_payloads(instance, host.id)
|
payloads = build_payloads(instance, host.id)
|
||||||
@@ -323,14 +328,19 @@ async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: Opn
|
|||||||
|
|
||||||
deployment = existing or OpenVpnInstanceDeployment(instance_id=instance.id, host_id=host.id, status="error")
|
deployment = existing or OpenVpnInstanceDeployment(instance_id=instance.id, host_id=host.id, status="error")
|
||||||
|
|
||||||
|
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
|
||||||
|
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
|
||||||
|
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
|
||||||
|
if not existing:
|
||||||
|
db.add(deployment)
|
||||||
|
await checkpoint(db)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
await _write_config(host, payloads, deployment)
|
await _write_config(db, host, payloads, deployment)
|
||||||
except OpnsenseAPIError as exc:
|
except OpnsenseAPIError as exc:
|
||||||
deployment.status = "error"
|
deployment.status = "error"
|
||||||
deployment.message = str(exc)
|
deployment.message = str(exc)
|
||||||
deployment.deployed_at = _now()
|
deployment.deployed_at = _now()
|
||||||
if not existing:
|
|
||||||
db.add(deployment)
|
|
||||||
return deployment
|
return deployment
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -345,32 +355,37 @@ async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: Opn
|
|||||||
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
||||||
deployment.last_payload = comparable
|
deployment.last_payload = comparable
|
||||||
deployment.deployed_at = _now()
|
deployment.deployed_at = _now()
|
||||||
if not existing:
|
|
||||||
db.add(deployment)
|
|
||||||
return deployment
|
return deployment
|
||||||
|
|
||||||
deployment.status = "success"
|
deployment.status = "success"
|
||||||
deployment.message = None
|
deployment.message = None
|
||||||
deployment.last_payload = comparable
|
deployment.last_payload = comparable
|
||||||
deployment.deployed_at = _now()
|
deployment.deployed_at = _now()
|
||||||
if not existing:
|
|
||||||
db.add(deployment)
|
|
||||||
return deployment
|
return deployment
|
||||||
|
|
||||||
|
|
||||||
async def deploy_instance(db: AsyncSession, instance: OpenVpnInstance, hosts: list[OpnsenseHost]) -> list[OpenVpnInstanceDeployment]:
|
async def deploy_instance(db: AsyncSession, instance: OpenVpnInstance, hosts: list[OpnsenseHost]) -> list[OpenVpnInstanceDeployment]:
|
||||||
from app.services.host_maintenance_service import ensure_no_maintenance
|
from app.services.host_maintenance_service import ensure_no_maintenance
|
||||||
await ensure_no_maintenance(db, hosts)
|
await ensure_no_maintenance(db, hosts)
|
||||||
deployments = [await _deploy_to_host(db, instance, host) for host in hosts]
|
async with resource_lease("openvpn", instance.id):
|
||||||
await db.commit()
|
deployments = [await _deploy_to_host(db, instance, host) for host in hosts]
|
||||||
|
await db.commit()
|
||||||
return deployments
|
return deployments
|
||||||
|
|
||||||
|
|
||||||
async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
|
async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
|
||||||
|
async with resource_lease("openvpn", instance.id):
|
||||||
|
await _undeploy_instance(db, instance)
|
||||||
|
|
||||||
|
|
||||||
|
async def _undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
|
||||||
"""Explicit action, not gated by anything (there is no lockout state
|
"""Explicit action, not gated by anything (there is no lockout state
|
||||||
for this entity to gate on) — deletes the Instance before its
|
for this entity to gate on) — deletes the Instance before its
|
||||||
referenced StaticKey (the ModelRelationField dependency direction),
|
referenced StaticKey (the ModelRelationField dependency direction),
|
||||||
then reconfigures."""
|
then reconfigures."""
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||||
|
await ensure_no_open_cleanup_on(db, OpenVpnInstanceDeployment, OpenVpnInstanceDeployment.instance_id == instance.id)
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(OpenVpnInstanceDeployment).where(OpenVpnInstanceDeployment.instance_id == instance.id))
|
result = await db.execute(select(OpenVpnInstanceDeployment).where(OpenVpnInstanceDeployment.instance_id == instance.id))
|
||||||
for deployment in result.scalars().all():
|
for deployment in result.scalars().all():
|
||||||
|
|||||||
@@ -8,8 +8,12 @@ required.
|
|||||||
|
|
||||||
Auth is HTTP Basic: username = API key, password = API secret.
|
Auth is HTTP Basic: username = API key, password = API secret.
|
||||||
"""
|
"""
|
||||||
|
import ipaddress
|
||||||
import logging
|
import logging
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from contextvars import ContextVar
|
||||||
import json
|
import json
|
||||||
|
import re
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
@@ -33,6 +37,17 @@ class OpnsenseAmbiguousError(OpnsenseAPIError):
|
|||||||
outcome of a write is unknown and must be reconciled, not retried blindly."""
|
outcome of a write is unknown and must be reconciled, not retried blindly."""
|
||||||
|
|
||||||
|
|
||||||
|
class OpnsenseNotFoundError(OpnsenseAPIError):
|
||||||
|
"""The firewall answered, but there is no such object (an empty `getRule` answer: unknown UUID, or a rule that is
|
||||||
|
not part of the rules model - legacy or automatic). A refusal, never an unknown outcome."""
|
||||||
|
|
||||||
|
|
||||||
|
class OpnsenseForbiddenError(OpnsenseAPIError):
|
||||||
|
"""The firewall answered 403: the API key lacks the privilege for this endpoint. A refusal, not an unknown
|
||||||
|
outcome; callers that only read can tell "no rights" from "unreachable". 401 (wrong key or secret) stays a
|
||||||
|
plain OpnsenseAPIError."""
|
||||||
|
|
||||||
|
|
||||||
def _client(host: OpnsenseHost) -> httpx.AsyncClient:
|
def _client(host: OpnsenseHost) -> httpx.AsyncClient:
|
||||||
# Before the secret is even read: an http:// host must never receive credentials.
|
# Before the secret is even read: an http:// host must never receive credentials.
|
||||||
reason = firewall_url.check_base_url(host.base_url)
|
reason = firewall_url.check_base_url(host.base_url)
|
||||||
@@ -51,6 +66,34 @@ def _client(host: OpnsenseHost) -> httpx.AsyncClient:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Rule counters before a reload (B11) ────────────────────────────────────
|
||||||
|
# Every request below that makes OPNsense reload pf (filter/NAT apply; alias and IPsec reconfigure run
|
||||||
|
# `filter reload skip_alias` too, verified in core stable/26.7; OpenVPN/WireGuard/network may through interface
|
||||||
|
# registration) zeroes every rule counter. Whatever was hit since the last observation would be lost and a rule
|
||||||
|
# could later look "unused". So the counters are read right before, through rule_hit_service. A writer that has
|
||||||
|
# just sampled itself (the rule cleanup) switches this off for its own call with ``counters_already_sampled()``.
|
||||||
|
_sampled_already: ContextVar[bool] = ContextVar("opnexus_counters_sampled", default=False)
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def counters_already_sampled():
|
||||||
|
token = _sampled_already.set(True)
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
_sampled_already.reset(token)
|
||||||
|
|
||||||
|
|
||||||
|
async def _before_reload(host: OpnsenseHost) -> None:
|
||||||
|
if _sampled_already.get():
|
||||||
|
return
|
||||||
|
from app.services.rule_hit_service import sample_before_reload # needs the database; this module does not
|
||||||
|
try:
|
||||||
|
await sample_before_reload(host)
|
||||||
|
except Exception as exc: # noqa: BLE001 - a failed sample must never block or fail the write itself
|
||||||
|
logger.info("Zählerprobe vor dem Anwenden auf %s fehlgeschlagen: %s", host.name, exc)
|
||||||
|
|
||||||
|
|
||||||
def _refuse_unresolved_placeholders(method: str, path: str, kwargs: dict) -> None:
|
def _refuse_unresolved_placeholders(method: str, path: str, kwargs: dict) -> None:
|
||||||
"""Last line of defence for site variables: a payload that still contains a {{NAME}} placeholder was never
|
"""Last line of defence for site variables: a payload that still contains a {{NAME}} placeholder was never
|
||||||
resolved for this host and must not reach a firewall."""
|
resolved for this host and must not reach a firewall."""
|
||||||
@@ -71,7 +114,8 @@ async def _do_request(host: OpnsenseHost, method: str, path: str, **kwargs) -> h
|
|||||||
except httpx.HTTPStatusError as exc:
|
except httpx.HTTPStatusError as exc:
|
||||||
# The body may reflect submitted secrets: keep only a scrubbed, bounded excerpt.
|
# The body may reflect submitted secrets: keep only a scrubbed, bounded excerpt.
|
||||||
secret = opnsense_secret_store.read_secret(host.id)
|
secret = opnsense_secret_store.read_secret(host.id)
|
||||||
error = OpnsenseAmbiguousError if exc.response.status_code >= 500 else OpnsenseAPIError
|
status = exc.response.status_code
|
||||||
|
error = OpnsenseAmbiguousError if status >= 500 else OpnsenseForbiddenError if status == 403 else OpnsenseAPIError
|
||||||
raise error(
|
raise error(
|
||||||
f"{exc.response.status_code} {path}: {redact_text(exc.response.text, host.api_key, secret)}"
|
f"{exc.response.status_code} {path}: {redact_text(exc.response.text, host.api_key, secret)}"
|
||||||
) from exc
|
) from exc
|
||||||
@@ -208,6 +252,7 @@ async def del_alias_item(host: OpnsenseHost, alias_uuid: str) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
async def reconfigure_aliases(host: OpnsenseHost) -> dict:
|
async def reconfigure_aliases(host: OpnsenseHost) -> dict:
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/firewall/alias/reconfigure"), "Aliase anwenden", set())
|
return _mutation_result(await _request(host, "POST", "/firewall/alias/reconfigure"), "Aliase anwenden", set())
|
||||||
|
|
||||||
|
|
||||||
@@ -234,6 +279,51 @@ def _mutation_result(data: Any, operation: str, results: set[str]) -> dict:
|
|||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
# -- Filter rules (FilterController -> ApiMutableModelControllerBase, rules.rule) --------------------------------
|
||||||
|
# Evidence for add/set/get/del/toggle/moveRuleBefore, checked 2026-10-22 against BOTH
|
||||||
|
# source opnsense/core tags 26.7.7, 26.1.9 and 25.7.11, src/opnsense/mvc/app/controllers/OPNsense/:
|
||||||
|
# Firewall/Api/FilterController.php (the *RuleAction methods, moveRuleBeforeAction, ittrRules),
|
||||||
|
# Firewall/Api/FilterBaseController.php (applyAction),
|
||||||
|
# Base/ApiMutableModelControllerBase.php (addBase, setBase, getBase, delBase, toggleBase)
|
||||||
|
# live OPNsense 26.7.7 on two lab test firewalls (scratch rules whose description began with
|
||||||
|
# `opnexus-b4-probe`, deleted again and applied afterwards; nothing was left behind).
|
||||||
|
# Same in all three source lines unless said otherwise; the live checks cover 26.7.7 only.
|
||||||
|
#
|
||||||
|
# addRule POST, body {"rule": {...}}. Answer {"result": "saved", "uuid": "<new>"}; a validation failure is HTTP 200
|
||||||
|
# {"result": "failed", "validations": {"rule.<field>": "<text>"}} (so _mutation_result raises).
|
||||||
|
# `sequence` is a FilterSequenceField (models/OPNsense/Firewall/Filter.xml: 1..999999, default 1;
|
||||||
|
# `sort_order` and `prio_group` are volatile, derived). Live: without `sequence` the rule gets the
|
||||||
|
# highest sequence of ALL rules + 100 (566, 666, 766, ...), so a copy always lands at the END of its
|
||||||
|
# prio_group; an explicit `sequence` IS honoured (sequence "3" -> sort_order 400000.0000003) and is not
|
||||||
|
# checked for uniqueness. A restore can therefore either send the old number or call moveRuleBefore.
|
||||||
|
# setRule POST /{uuid}, body {"rule": {...}}; answer {"result": "saved"}. It is an upsert (a well-formed unknown
|
||||||
|
# UUID creates the rule) - never call it for a rule that may be gone.
|
||||||
|
# delRule POST /{uuid}. {"result": "deleted"} or {"result": "not found"} (unknown, legacy and automatic UUIDs all
|
||||||
|
# answer "not found", HTTP 200, nothing changes). 26.1 and 26.7 accept a comma-separated list; 25.7 a
|
||||||
|
# single UUID. Consequence: "not found" proves nothing about a rule being gone - re-read it.
|
||||||
|
# getRule GET /{uuid} -> {"rule": {...}} (every field of the rules model, select fields as
|
||||||
|
# {"<option>": {"value": "<label>", "selected": 0|1}}, `categories` a select dict - or [] when the rule has
|
||||||
|
# none -; no `uuid` key; read-only: `sequence`, `sort_order`, `prio_group` and the nested `audit`).
|
||||||
|
# Live: an unknown, legacy (hex label) or automatic UUID answers HTTP 200 with a bare `[]`. 25.7 and 26.1
|
||||||
|
# also know `?fetchmode=copy` (pre-fills `sequence`); 26.7.7 no longer has it - not used here.
|
||||||
|
# Without a UUID it returns the template of a new rule (see get_rule_template above).
|
||||||
|
# toggleRule POST /{uuid}/{enabled} - `enabled` is "0" or "1" and EXPLICIT; without it the rule flips (never do that).
|
||||||
|
# Answer {"result": "Enabled"|"Disabled", "changed": true|false}. The explicit form is idempotent: repeating
|
||||||
|
# it answers the same result with "changed": false (live: 0,0 -> true,false; 1,1 -> true,false). Any other
|
||||||
|
# value answers {"result": "failed", "changed": false}; an unknown or legacy UUID, a GET, or a model that
|
||||||
|
# has no such node answers {"result": "failed"} without "changed" - all HTTP 200. It only edits the
|
||||||
|
# configuration; nothing reaches pf before apply() (the rule keeps its sort_order, searchRule shows
|
||||||
|
# enabled "0"/"1" at once). 26.1/26.7 take a list of UUIDs (explicit value only); 25.7 one UUID.
|
||||||
|
# moveRuleBefore POST /{selected}/{target}. {"status": "ok"} (no "result" key). Both rules must be in the same
|
||||||
|
# `prio_group` - the sort_order prefix - which is NOT the interface: every single-interface rule of every
|
||||||
|
# interface shares 400000 (lan and wan rules can be moved before each other), floating rules (several or
|
||||||
|
# no interface) are 200000, interface-group rules 3000xx. Live: unknown UUID, same UUID twice and
|
||||||
|
# different groups all answer HTTP 500 {"errorMessage": "Unexpected error, check log for details",
|
||||||
|
# "errorTitle": "Filter"}, a GET answers 200 {"status": "error"}. A 500 is Ambiguous in _do_request, so a
|
||||||
|
# caller that wants to tell "refused" from "unknown" must read sort_order back. The move only touches
|
||||||
|
# `sequence` of the moved rule (and, when the gap is too small, of the rules behind the target); legacy
|
||||||
|
# rules are not in the model and cannot be an anchor.
|
||||||
|
# apply POST. {"status": "OK\n\n"} after `filter reload skip_alias`. FilterBaseController has no savepoint here.
|
||||||
async def add_rule(host: OpnsenseHost, payload: dict) -> dict:
|
async def add_rule(host: OpnsenseHost, payload: dict) -> dict:
|
||||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/addRule", json={"rule": payload}), "Regel anlegen", {"saved"})
|
return _mutation_result(await _request(host, "POST", "/firewall/filter/addRule", json={"rule": payload}), "Regel anlegen", {"saved"})
|
||||||
|
|
||||||
@@ -246,18 +336,94 @@ async def del_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
|||||||
return _mutation_result(await _request(host, "POST", f"/firewall/filter/delRule/{rule_uuid}"), "Regel entfernen", {"deleted", "not found"})
|
return _mutation_result(await _request(host, "POST", f"/firewall/filter/delRule/{rule_uuid}"), "Regel entfernen", {"deleted", "not found"})
|
||||||
|
|
||||||
|
|
||||||
|
_RULE_REF = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||||
|
|
||||||
|
|
||||||
|
def _rule_ref(rule_uuid: str) -> str:
|
||||||
|
"""A rule reference ends up in the URL path: refuse anything that could leave it."""
|
||||||
|
if not isinstance(rule_uuid, str) or not _RULE_REF.match(rule_uuid):
|
||||||
|
raise OpnsenseAPIError("Ungültige Regel-UUID")
|
||||||
|
return rule_uuid
|
||||||
|
|
||||||
|
|
||||||
|
async def get_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||||
|
"""The full model record of one rule (`getRule/{uuid}`, the inner `rule` dict, select fields still in
|
||||||
|
OPNsense's option form - reduce them with rule_write_payload()). A rule that is not in the rules model (unknown,
|
||||||
|
legacy hex label, automatic) is answered with `[]`: raises OpnsenseNotFoundError."""
|
||||||
|
data = await _request(host, "GET", f"/firewall/filter/getRule/{_rule_ref(rule_uuid)}")
|
||||||
|
rule = data.get("rule") if isinstance(data, dict) else None
|
||||||
|
if isinstance(rule, dict) and rule:
|
||||||
|
return rule
|
||||||
|
if data == [] or data == {}:
|
||||||
|
raise OpnsenseNotFoundError(f"Regel {rule_uuid} gibt es auf {host.name} nicht (oder sie ist keine bearbeitbare Regel)")
|
||||||
|
raise OpnsenseAPIError(f"Regel lesen: Keine auswertbare Antwort von {host.name}")
|
||||||
|
|
||||||
|
|
||||||
|
async def toggle_rule(host: OpnsenseHost, rule_uuid: str, enabled: bool) -> dict:
|
||||||
|
"""Sets the rule's `enabled` flag to exactly `enabled` (`toggleRule/{uuid}/1|0`; never the flipping form) and
|
||||||
|
returns OPNsense's answer, e.g. {"result": "Disabled", "changed": False}. Repeating it is harmless. Only the
|
||||||
|
configuration changes; call apply() afterwards. A refusal (unknown or legacy UUID) is a plain OpnsenseAPIError,
|
||||||
|
a lost answer an OpnsenseAmbiguousError - then read the rule back before doing anything else."""
|
||||||
|
if not isinstance(enabled, bool):
|
||||||
|
raise TypeError("toggle_rule: 'enabled' must be a bool (an implicit flip would not be idempotent)")
|
||||||
|
data = await _request(host, "POST", f"/firewall/filter/toggleRule/{_rule_ref(rule_uuid)}/{1 if enabled else 0}")
|
||||||
|
return _mutation_result(data, "Regel ein-/ausschalten", {"enabled" if enabled else "disabled"})
|
||||||
|
|
||||||
|
|
||||||
|
# Fields of a getRule record that a write must not send: the UUID is the rule's identity, `sequence` is assigned
|
||||||
|
# on creation (sort_order and prio_group are derived from it), `audit` is metadata OPNsense maintains itself.
|
||||||
|
RULE_READONLY_FIELDS = frozenset({"uuid", "sequence", "sort_order", "prio_group", "audit"})
|
||||||
|
|
||||||
|
|
||||||
|
def rule_write_payload(rule: dict) -> dict[str, str]:
|
||||||
|
"""Turns a getRule record into the flat form addRule/setRule accept: select fields become their selected
|
||||||
|
option(s), comma-separated (`lan,wan`; none selected -> ""), the empty-list form of an empty multi-select
|
||||||
|
becomes "", read-only fields are left out. Live round trip on 26.7.7 (getRule -> this -> addRule -> getRule)
|
||||||
|
gave an identical record except uuid/sequence/sort_order/audit for single-interface, multi-interface (floating),
|
||||||
|
no-interface (floating), interface-group, ICMP and any-protocol rules with categories, aliases, port ranges,
|
||||||
|
TCP flags, tags and state limits."""
|
||||||
|
out: dict[str, str] = {}
|
||||||
|
for key, value in rule.items():
|
||||||
|
if key in RULE_READONLY_FIELDS:
|
||||||
|
continue
|
||||||
|
if isinstance(value, dict):
|
||||||
|
if value and all(isinstance(o, dict) and "selected" in o for o in value.values()):
|
||||||
|
out[key] = ",".join(k for k, o in value.items() if o.get("selected") and k != "")
|
||||||
|
# any other nested dict is metadata and is not sent
|
||||||
|
elif isinstance(value, list):
|
||||||
|
out[key] = ",".join(str(v) for v in value)
|
||||||
|
else:
|
||||||
|
out[key] = "" if value is None else str(value)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
async def apply(host: OpnsenseHost) -> dict:
|
async def apply(host: OpnsenseHost) -> dict:
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
|
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
|
||||||
|
|
||||||
|
|
||||||
|
async def get_hasync(host: OpnsenseHost) -> dict:
|
||||||
|
"""System > High Availability settings, read-only: `core/hasync/get` (Core/Api/HasyncController extends
|
||||||
|
ApiMutableModelControllerBase, model Core/Hasync.xml; identical in opnsense/core 25.7.11, 26.1.9 and 26.7.7).
|
||||||
|
Answer {"hasync": {"pfsyncinterface": {"": {"value": "Disabled", ...}, "lan": {"value": "LAN", "selected": 1},
|
||||||
|
...}, "pfsyncpeerip": "<address or empty>", "synchronizetoip": ..., "syncitems": {...}}} - `pfsyncinterface` is an
|
||||||
|
InterfaceField (the selected option, "" = disabled), `pfsyncpeerip` a NetworkField without mask. Live on 26.7.7
|
||||||
|
(both nodes of the lab cluster): pfsyncinterface `lan`, a peer address, synchronizetoip empty. Used by the rule
|
||||||
|
cleanup to keep pfsync rules out of reach (rule_analysis/protection.py); it reads nothing secret (the sync
|
||||||
|
password field comes back, but is never stored or logged by the caller)."""
|
||||||
|
return await _request(host, "GET", "/core/hasync/get")
|
||||||
|
|
||||||
|
|
||||||
async def move_rule_before(host: OpnsenseHost, selected_uuid: str, target_uuid: str) -> dict:
|
async def move_rule_before(host: OpnsenseHost, selected_uuid: str, target_uuid: str) -> dict:
|
||||||
"""`firewall/filter/moveRuleBefore/{selected_uuid}/{target_uuid}`
|
"""`firewall/filter/moveRuleBefore/{selected_uuid}/{target_uuid}`
|
||||||
(FilterController::moveRuleBeforeAction) — repositions `selected_uuid`
|
(FilterController::moveRuleBeforeAction) — repositions `selected_uuid`
|
||||||
to sit directly before `target_uuid` via gap-numbered `sequence`
|
to sit directly before `target_uuid` via gap-numbered `sequence`
|
||||||
floats, scoped to rules sharing the same `prio_group` (interface/type).
|
floats, scoped to rules sharing the same `prio_group` (the sort_order
|
||||||
Verified against OPNsense core source: no "move to top/bottom"
|
prefix, see the evidence block above - not the interface). Verified against OPNsense core source and live
|
||||||
sentinel exists, and both UUIDs must resolve to existing rules on the
|
(26.7.7): no "move to top/bottom"
|
||||||
SAME interface/type or the controller throws. On success it returns
|
sentinel exists, and both UUIDs must resolve to existing rules in the
|
||||||
|
SAME prio_group or the controller throws (HTTP 500, which this client
|
||||||
|
reports as OpnsenseAmbiguousError). On success it returns
|
||||||
`{"status": "ok"}` (confirmed against FilterController.php directly —
|
`{"status": "ok"}` (confirmed against FilterController.php directly —
|
||||||
unlike addRule/setRule/delRule this has no "result" key at all), so
|
unlike addRule/setRule/delRule this has no "result" key at all), so
|
||||||
this is dispatched through _mutation_result with an empty `results`
|
this is dispatched through _mutation_result with an empty `results`
|
||||||
@@ -338,6 +504,7 @@ async def apply_port_forward(host: OpnsenseHost) -> dict:
|
|||||||
apply (verified in core source: neither DNatController nor
|
apply (verified in core source: neither DNatController nor
|
||||||
FilterController override applyAction). Called via its own semantically
|
FilterController override applyAction). Called via its own semantically
|
||||||
named endpoint anyway, matching what OPNsense's own UI does."""
|
named endpoint anyway, matching what OPNsense's own UI does."""
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/firewall/d_nat/apply"), "Änderungen anwenden", set())
|
return _mutation_result(await _request(host, "POST", "/firewall/d_nat/apply"), "Änderungen anwenden", set())
|
||||||
|
|
||||||
|
|
||||||
@@ -377,6 +544,7 @@ async def del_one_to_one_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
|||||||
|
|
||||||
async def apply_one_to_one(host: OpnsenseHost) -> dict:
|
async def apply_one_to_one(host: OpnsenseHost) -> dict:
|
||||||
"""Apply through the type-specific controller, using FilterBaseController reload."""
|
"""Apply through the type-specific controller, using FilterBaseController reload."""
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/firewall/one_to_one/apply"), "Änderungen anwenden", set())
|
return _mutation_result(await _request(host, "POST", "/firewall/one_to_one/apply"), "Änderungen anwenden", set())
|
||||||
|
|
||||||
|
|
||||||
@@ -399,6 +567,7 @@ async def del_outbound_nat_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
|||||||
|
|
||||||
async def apply_outbound_nat(host: OpnsenseHost) -> dict:
|
async def apply_outbound_nat(host: OpnsenseHost) -> dict:
|
||||||
"""Apply through the type-specific controller, using FilterBaseController reload."""
|
"""Apply through the type-specific controller, using FilterBaseController reload."""
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/firewall/source_nat/apply"), "Änderungen anwenden", set())
|
return _mutation_result(await _request(host, "POST", "/firewall/source_nat/apply"), "Änderungen anwenden", set())
|
||||||
|
|
||||||
|
|
||||||
@@ -434,7 +603,65 @@ async def get_pf_rule_statistics(host: OpnsenseHost) -> dict:
|
|||||||
# - Firewall/Api/GroupController::searchItemAction -> searchBase("ifgroupentry"): rows with
|
# - Firewall/Api/GroupController::searchItemAction -> searchBase("ifgroupentry"): rows with
|
||||||
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
|
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
|
||||||
# wireguard) appear with their name as uuid and empty members.
|
# wireguard) appear with their name as uuid and empty members.
|
||||||
|
def _stream_addr(response: httpx.Response, which: str) -> tuple[str, int] | None:
|
||||||
|
stream = response.extensions.get("network_stream")
|
||||||
|
try:
|
||||||
|
addr = stream.get_extra_info(which) if stream is not None else None
|
||||||
|
except Exception: # noqa: BLE001 - transport without that information
|
||||||
|
return None
|
||||||
|
return (str(addr[0]), int(addr[1])) if isinstance(addr, (tuple, list)) and len(addr) >= 2 else None
|
||||||
|
|
||||||
|
|
||||||
|
async def own_source_address(host: OpnsenseHost) -> str | None:
|
||||||
|
"""The address this firewall sees OPNexus's API requests come from (B10): behind NAT or the management tunnel it is
|
||||||
|
not what OPNexus knows about itself, so the firewall is asked. Two requests on one keep-alive connection: the first
|
||||||
|
tells the firewall address and the local port of this connection, the second lists pf's states for that address
|
||||||
|
(`diagnostics/firewall/query_states`, POST, read-only; Diagnostics/Api/FirewallController, core stable/26.7) and the
|
||||||
|
state whose source port is our local port is this very connection. Live 2026-10-23 on test-fw-1/3: one ESTABLISHED
|
||||||
|
state, src 192.0.2.10 = the OPNexus server, port preserved by Docker's NAT.
|
||||||
|
|
||||||
|
None when it cannot be told for sure: through the overlay's CONNECT proxy (the local port belongs to the proxy
|
||||||
|
connection), a NAT that rewrote the port, or an API key without the states privilege (that raises
|
||||||
|
OpnsenseAPIError)."""
|
||||||
|
async with _client(host) as http:
|
||||||
|
try:
|
||||||
|
first = await http.get("/diagnostics/firewall/pf_states")
|
||||||
|
first.raise_for_status()
|
||||||
|
server = _stream_addr(first, "server_addr")
|
||||||
|
if server is None:
|
||||||
|
return None
|
||||||
|
second = await http.post("/diagnostics/firewall/query_states",
|
||||||
|
json={"searchPhrase": server[0], "rowCount": 500, "current": 1})
|
||||||
|
second.raise_for_status()
|
||||||
|
local, server = _stream_addr(second, "client_addr"), _stream_addr(second, "server_addr")
|
||||||
|
rows = second.json().get("rows") or []
|
||||||
|
except (httpx.HTTPError, ValueError, AttributeError) as exc:
|
||||||
|
raise OpnsenseAPIError(f"Zustandstabelle von {host.name} nicht lesbar: {str(exc)[:200]}") from exc
|
||||||
|
if local is None or server is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def same_ip(a, b) -> bool:
|
||||||
|
try:
|
||||||
|
return ipaddress.ip_address(str(a)) == ipaddress.ip_address(str(b))
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
for row in rows if isinstance(rows, list) else []:
|
||||||
|
if (isinstance(row, dict) and same_ip(row.get("dst_addr"), server[0]) and str(row.get("dst_port")) == str(server[1])
|
||||||
|
and str(row.get("src_port")) == str(local[1]) and str(row.get("state", "")).startswith("ESTABLISHED")):
|
||||||
|
try:
|
||||||
|
return str(ipaddress.ip_address(str(row.get("src_addr"))))
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
async def get_rule_stats(host: OpnsenseHost) -> dict:
|
async def get_rule_stats(host: OpnsenseHost) -> dict:
|
||||||
|
"""Per-label counters, already summed over every pf line of a rule. Used for the rule usage sampling
|
||||||
|
(rule_hit_service). Live check 2026-10-22 on all three test firewalls (26.7, one with unmigrated legacy
|
||||||
|
rules): HTTP 200, ``stats`` has exactly one key per label of ``pfStatistics/rules`` (30/30/25 labels,
|
||||||
|
including the 32-hex label of the legacy user rule on the third), and each ``packets`` equals the sum over
|
||||||
|
that label's pf lines (differences only by traffic between the two reads)."""
|
||||||
return await _request(host, "GET", "/firewall/filter_util/ruleStats")
|
return await _request(host, "GET", "/firewall/filter_util/ruleStats")
|
||||||
|
|
||||||
|
|
||||||
@@ -663,6 +890,7 @@ async def reconfigure_ipsec(host: OpnsenseHost) -> dict:
|
|||||||
Deliberately its own call, separate from the 5 config-write calls
|
Deliberately its own call, separate from the 5 config-write calls
|
||||||
above — see ipsec_deploy_service.py for why this app never bundles it
|
above — see ipsec_deploy_service.py for why this app never bundles it
|
||||||
silently."""
|
silently."""
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/ipsec/service/reconfigure"), "IPsec-Dienst neu laden", set())
|
return _mutation_result(await _request(host, "POST", "/ipsec/service/reconfigure"), "IPsec-Dienst neu laden", set())
|
||||||
|
|
||||||
|
|
||||||
@@ -760,6 +988,7 @@ async def reconfigure_openvpn(host: OpnsenseHost) -> dict:
|
|||||||
section comment above for why this is NOT assumed symmetric with
|
section comment above for why this is NOT assumed symmetric with
|
||||||
IPsec's `{"status": "ok"}`), dispatched via _mutation_result with the
|
IPsec's `{"status": "ok"}`), dispatched via _mutation_result with the
|
||||||
"result"-based branch (results={"ok"})."""
|
"result"-based branch (results={"ok"})."""
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/openvpn/service/reconfigure"), "OpenVPN-Dienst neu laden", {"ok"})
|
return _mutation_result(await _request(host, "POST", "/openvpn/service/reconfigure"), "OpenVPN-Dienst neu laden", {"ok"})
|
||||||
|
|
||||||
|
|
||||||
@@ -936,6 +1165,7 @@ async def reconfigure_wireguard(host: OpnsenseHost) -> dict:
|
|||||||
IPsec's `{"status": "ok"}`, and matches OpenVPN's shape instead),
|
IPsec's `{"status": "ok"}`, and matches OpenVPN's shape instead),
|
||||||
dispatched via _mutation_result with the "result"-based branch
|
dispatched via _mutation_result with the "result"-based branch
|
||||||
(results={"ok"})."""
|
(results={"ok"})."""
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(await _request(host, "POST", "/wireguard/service/reconfigure"), "WireGuard-Dienst neu laden", {"ok"})
|
return _mutation_result(await _request(host, "POST", "/wireguard/service/reconfigure"), "WireGuard-Dienst neu laden", {"ok"})
|
||||||
|
|
||||||
|
|
||||||
@@ -1256,6 +1486,7 @@ async def write_network_item(host: OpnsenseHost, kind: str, action: str, uuid: s
|
|||||||
|
|
||||||
|
|
||||||
async def reconfigure_network(host: OpnsenseHost, kind: str) -> dict:
|
async def reconfigure_network(host: OpnsenseHost, kind: str) -> dict:
|
||||||
|
await _before_reload(host)
|
||||||
return _mutation_result(
|
return _mutation_result(
|
||||||
await _request(host, "POST", NETWORK_ITEM_APIS[kind]["reconfigure"]), "Änderung anwenden", set())
|
await _request(host, "POST", NETWORK_ITEM_APIS[kind]["reconfigure"]), "Änderung anwenden", set())
|
||||||
|
|
||||||
@@ -1316,6 +1547,13 @@ async def create_snapshot(host: OpnsenseHost, name: str, note: str = "") -> None
|
|||||||
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
|
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
|
||||||
|
|
||||||
|
|
||||||
|
async def activate_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||||
|
"""Boot environment for every following start (bectl activate, not -t); takes effect at the next reboot."""
|
||||||
|
data = await _request(host, "POST", f"/core/snapshots/activate/{uuid}")
|
||||||
|
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||||
|
raise OpnsenseAPIError(f"Snapshot aktivieren von Firewall nicht bestätigt: {redact_data(data)}")
|
||||||
|
|
||||||
|
|
||||||
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||||
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
|
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
|
||||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||||
|
|||||||
@@ -147,6 +147,9 @@ async def sweep_expired_outbound_nat_deployments(db: AsyncSession) -> list[Outbo
|
|||||||
|
|
||||||
|
|
||||||
async def undeploy_outbound_nat(db: AsyncSession, rule: OutboundNatRule) -> None:
|
async def undeploy_outbound_nat(db: AsyncSession, rule: OutboundNatRule) -> None:
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||||
|
await ensure_no_open_cleanup_on(db, OutboundNatDeployment, OutboundNatDeployment.outbound_nat_id == rule.id)
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(OutboundNatDeployment).where(OutboundNatDeployment.outbound_nat_id == rule.id))
|
result = await db.execute(select(OutboundNatDeployment).where(OutboundNatDeployment.outbound_nat_id == rule.id))
|
||||||
for deployment in result.scalars().all():
|
for deployment in result.scalars().all():
|
||||||
|
|||||||
@@ -185,6 +185,9 @@ async def reassert_group_ordering(db: AsyncSession, group: PolicyHierarchyGroup)
|
|||||||
for h in g.hosts:
|
for h in g.hosts:
|
||||||
hosts[h.id] = h
|
hosts[h.id] = h
|
||||||
errors: list[str] = []
|
errors: list[str] = []
|
||||||
|
# moveRuleBefore changes rule positions: not while a rule cleanup (1.31) owns one of the firewalls.
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
|
await ensure_no_open_cleanup(db, [h for h in hosts.values() if h.enabled])
|
||||||
for host in sorted(hosts.values(), key=lambda h: (h.name, str(h.id))):
|
for host in sorted(hosts.values(), key=lambda h: (h.name, str(h.id))):
|
||||||
if not host.enabled:
|
if not host.enabled:
|
||||||
continue
|
continue
|
||||||
@@ -300,9 +303,13 @@ async def sweep_expired_hierarchy_deployments(db: AsyncSession) -> list[PolicyHi
|
|||||||
|
|
||||||
|
|
||||||
async def undeploy_hierarchy_rule(db: AsyncSession, rule: PolicyHierarchyRule) -> None:
|
async def undeploy_hierarchy_rule(db: AsyncSession, rule: PolicyHierarchyRule) -> None:
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(PolicyHierarchyRuleDeployment).where(PolicyHierarchyRuleDeployment.policy_hierarchy_id == rule.id))
|
result = await db.execute(select(PolicyHierarchyRuleDeployment).where(PolicyHierarchyRuleDeployment.policy_hierarchy_id == rule.id))
|
||||||
for deployment in result.scalars().all():
|
deployments = list(result.scalars().all())
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them.
|
||||||
|
await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])
|
||||||
|
for deployment in deployments:
|
||||||
host = await db.get(OpnsenseHost, deployment.host_id)
|
host = await db.get(OpnsenseHost, deployment.host_id)
|
||||||
if host and deployment.opnsense_policy_hierarchy_uuid:
|
if host and deployment.opnsense_policy_hierarchy_uuid:
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -147,6 +147,9 @@ async def sweep_expired_port_forward_deployments(db: AsyncSession) -> list[PortF
|
|||||||
|
|
||||||
|
|
||||||
async def undeploy_port_forward(db: AsyncSession, rule: PortForwardRule) -> None:
|
async def undeploy_port_forward(db: AsyncSession, rule: PortForwardRule) -> None:
|
||||||
|
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||||
|
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||||
|
await ensure_no_open_cleanup_on(db, PortForwardDeployment, PortForwardDeployment.port_forward_id == rule.id)
|
||||||
failures = []
|
failures = []
|
||||||
result = await db.execute(select(PortForwardDeployment).where(PortForwardDeployment.port_forward_id == rule.id))
|
result = await db.execute(select(PortForwardDeployment).where(PortForwardDeployment.port_forward_id == rule.id))
|
||||||
for deployment in result.scalars().all():
|
for deployment in result.scalars().all():
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ from app.models.host import OpnsenseHost
|
|||||||
from app.models.resource_sample import HostResourceSample
|
from app.models.resource_sample import HostResourceSample
|
||||||
from app.services import gateway_health_service
|
from app.services import gateway_health_service
|
||||||
from app.services import opnsense_client as client
|
from app.services import opnsense_client as client
|
||||||
from app.services.opnsense_client import OpnsenseAPIError
|
from app.services.opnsense_client import OpnsenseAPIError, OpnsenseForbiddenError
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -91,10 +91,19 @@ def _services_compact(listing) -> dict[str, int]:
|
|||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
async def _optional(operation):
|
async def _optional(operation, name: str, denied: list[str], ok: list[str]):
|
||||||
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing)."""
|
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing).
|
||||||
|
A refusal by the firewall (403: the API key lacks the privilege) is recorded in ``denied`` so the
|
||||||
|
missing right becomes visible; a reading that went through is recorded in ``ok`` (the proof that a right is
|
||||||
|
back, whatever the answer contains); any other failure stays "unknown" and is no finding about rights."""
|
||||||
try:
|
try:
|
||||||
return await operation
|
result = await operation
|
||||||
|
ok.append(name)
|
||||||
|
return result
|
||||||
|
except OpnsenseForbiddenError:
|
||||||
|
denied.append(name)
|
||||||
|
logger.debug("Optionale Messung %s: API-Recht fehlt", name)
|
||||||
|
return None
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.debug("Optionale Messung fehlgeschlagen: %s", type(exc).__name__)
|
logger.debug("Optionale Messung fehlgeschlagen: %s", type(exc).__name__)
|
||||||
return None
|
return None
|
||||||
@@ -105,11 +114,13 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
|||||||
system_time = await client.get_system_time(host)
|
system_time = await client.get_system_time(host)
|
||||||
disk = await client.get_system_disk(host)
|
disk = await client.get_system_disk(host)
|
||||||
interfaces = await client.get_interface_statistics(host)
|
interfaces = await client.get_interface_statistics(host)
|
||||||
|
denied: list[str] = []
|
||||||
|
ok: list[str] = []
|
||||||
swap, cpu_type, gateways, services = await asyncio.gather(
|
swap, cpu_type, gateways, services = await asyncio.gather(
|
||||||
_optional(client.get_system_swap(host)),
|
_optional(client.get_system_swap(host), "swap", denied, ok),
|
||||||
_optional(client.get_cpu_type(host)),
|
_optional(client.get_cpu_type(host), "cpu", denied, ok),
|
||||||
_optional(gateway_health_service.get_gateway_health(host)),
|
_optional(gateway_health_service.get_gateway_health(host), "gateways", denied, ok),
|
||||||
_optional(client.search_services(host)),
|
_optional(client.search_services(host), "services", denied, ok),
|
||||||
)
|
)
|
||||||
db.add(
|
db.add(
|
||||||
HostResourceSample(
|
HostResourceSample(
|
||||||
@@ -125,6 +136,9 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
|||||||
"cpu_threads": _cpu_threads(cpu_type) if cpu_type is not None else None,
|
"cpu_threads": _cpu_threads(cpu_type) if cpu_type is not None else None,
|
||||||
"gateways": gateway_health_service.compact(gateways) if gateways is not None else None,
|
"gateways": gateway_health_service.compact(gateways) if gateways is not None else None,
|
||||||
"services": _services_compact(services) if services is not None else None,
|
"services": _services_compact(services) if services is not None else None,
|
||||||
|
# Readings the firewall refused (403) because the API key lacks the right; always a list.
|
||||||
|
"denied": sorted(denied),
|
||||||
|
"ok": sorted(ok),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -139,9 +139,14 @@ def pause(run: RolloutRun, reason: str, **detail) -> None:
|
|||||||
_progress(run, paused_at=_iso(_now()))
|
_progress(run, paused_at=_iso(_now()))
|
||||||
|
|
||||||
|
|
||||||
async def may_roll_out(db: AsyncSession, username: str) -> bool:
|
async def may_roll_out(db: AsyncSession, username: str, user_id=None) -> bool:
|
||||||
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout."""
|
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout.
|
||||||
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
|
With the stable account id (1.29) the id decides: a deleted account that was re-created under the same name is
|
||||||
|
someone else. Rows from before the id existed (and still without one) are matched by name as before."""
|
||||||
|
if user_id is not None:
|
||||||
|
user = await db.get(AdminUser, user_id)
|
||||||
|
else:
|
||||||
|
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
|
||||||
if user is None:
|
if user is None:
|
||||||
return False
|
return False
|
||||||
if user.role == "admin":
|
if user.role == "admin":
|
||||||
@@ -184,17 +189,18 @@ async def active_run(db: AsyncSession) -> RolloutRun | None:
|
|||||||
|
|
||||||
|
|
||||||
async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
|
async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
|
||||||
occurrence: datetime | None = None) -> RolloutRun:
|
occurrence: datetime | None = None, started_by_id=None) -> RolloutRun:
|
||||||
"""Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
|
"""Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
|
||||||
now = _now()
|
now = _now()
|
||||||
snapshot = {
|
snapshot = {
|
||||||
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
||||||
"window_id": str(plan.window_id) if plan.window_id else None,
|
"window_id": str(plan.window_id) if plan.window_id else None,
|
||||||
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||||
|
"gate_probes": list(plan.gate_probes or []),
|
||||||
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
||||||
}
|
}
|
||||||
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
||||||
current_wave=0, started_by=started_by, trigger=trigger, window_occurrence=occurrence,
|
current_wave=0, started_by=started_by, started_by_id=started_by_id, trigger=trigger, window_occurrence=occurrence,
|
||||||
progress={}, created_at=now, step_at=now, revision=0)
|
progress={}, created_at=now, step_at=now, revision=0)
|
||||||
run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
|
run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
|
||||||
for w in resolved["waves"]]
|
for w in resolved["waves"]]
|
||||||
@@ -329,7 +335,7 @@ async def _sync_hosts(db: AsyncSession, run: RolloutRun) -> bool:
|
|||||||
# ── handlers: one per run status; each returns an apply-function or None ──────
|
# ── handlers: one per run status; each returns an apply-function or None ──────
|
||||||
|
|
||||||
async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||||
if await may_roll_out(db, run.started_by):
|
if await may_roll_out(db, run.started_by, run.started_by_id):
|
||||||
return None
|
return None
|
||||||
|
|
||||||
async def apply(db, run):
|
async def apply(db, run):
|
||||||
@@ -376,7 +382,8 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
|||||||
baselines.update((w.baseline or {}))
|
baselines.update((w.baseline or {}))
|
||||||
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
||||||
checks=run.plan_snapshot.get("gate_checks"),
|
checks=run.plan_snapshot.get("gate_checks"),
|
||||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||||
|
probes=run.plan_snapshot.get("gate_probes"))
|
||||||
|
|
||||||
async def apply(db, run):
|
async def apply(db, run):
|
||||||
if gate["verdict"] == "fail":
|
if gate["verdict"] == "fail":
|
||||||
@@ -402,21 +409,30 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
|||||||
return apply
|
return apply
|
||||||
|
|
||||||
|
|
||||||
|
# The core package of each OPNsense release type (community, development, business edition). Switching the type
|
||||||
|
# installs another core package: live 2026-10-23, community -> devel offered `opnsense-devel` 27.1.a_483 as a NEW
|
||||||
|
# package while product_latest still said 26.7.7 (the rollout expected 26.7.7 and stopped at the gate).
|
||||||
|
CORE_PACKAGES = ("opnsense", "opnsense-devel", "opnsense-business")
|
||||||
|
|
||||||
|
|
||||||
def target_of(status: dict, action: str) -> str | None:
|
def target_of(status: dict, action: str) -> str | None:
|
||||||
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
|
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
|
||||||
|
|
||||||
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
|
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
|
||||||
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
|
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
|
||||||
core package's new version comes first; product_latest is only a fallback."""
|
core package's new version comes first (whichever release type it is, see CORE_PACKAGES, also when it
|
||||||
|
arrives as a new package); product_latest is only a fallback."""
|
||||||
if action == "major_update":
|
if action == "major_update":
|
||||||
return status.get("upgrade_major_version") or None
|
return status.get("upgrade_major_version") or None
|
||||||
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
|
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
|
||||||
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
|
for name in CORE_PACKAGES:
|
||||||
if core and core.get("new") and core.get("new") != "N/A":
|
core = packages.get(name) if isinstance(packages.get(name), dict) else None
|
||||||
return str(core["new"])
|
if core and core.get("new") and core.get("new") != "N/A":
|
||||||
for row in status.get("upgrade_packages") or []:
|
return str(core["new"])
|
||||||
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
|
for key, field in (("upgrade_packages", "new_version"), ("new_packages", "version")):
|
||||||
return str(row["new_version"])
|
for row in status.get(key) or []:
|
||||||
|
if isinstance(row, dict) and row.get("name") in CORE_PACKAGES and row.get(field):
|
||||||
|
return str(row[field])
|
||||||
product = status.get("product") if isinstance(status.get("product"), dict) else {}
|
product = status.get("product") if isinstance(status.get("product"), dict) else {}
|
||||||
return product.get("product_latest") or status.get("product_latest") or None
|
return product.get("product_latest") or status.get("product_latest") or None
|
||||||
|
|
||||||
@@ -532,7 +548,7 @@ async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | Non
|
|||||||
versions = {}
|
versions = {}
|
||||||
for rh_id in offers:
|
for rh_id in offers:
|
||||||
host = usable[rh_id]
|
host = usable[rh_id]
|
||||||
baselines[str(host.id)] = await rollout_health.baseline(db, host)
|
baselines[str(host.id)] = await rollout_health.baseline(db, host, run.plan_snapshot.get("gate_probes"))
|
||||||
versions[rh_id] = baselines[str(host.id)].get("version")
|
versions[rh_id] = baselines[str(host.id)].get("version")
|
||||||
roles = await get_statuses([usable[i] for i in offers])
|
roles = await get_statuses([usable[i] for i in offers])
|
||||||
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
||||||
@@ -625,6 +641,7 @@ async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision |
|
|||||||
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
||||||
gate = await rollout_health.evaluate(
|
gate = await rollout_health.evaluate(
|
||||||
db, done_peers + [host], baselines, target_version=run.target_version,
|
db, done_peers + [host], baselines, target_version=run.target_version,
|
||||||
|
# No probes here: the peers' probes were checked in their own gate, and this runs on every tick.
|
||||||
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
||||||
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
||||||
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
||||||
@@ -713,7 +730,8 @@ async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
|||||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||||
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
||||||
checks=run.plan_snapshot.get("gate_checks"),
|
checks=run.plan_snapshot.get("gate_checks"),
|
||||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||||
|
probes=run.plan_snapshot.get("gate_probes"))
|
||||||
|
|
||||||
async def apply(db, run):
|
async def apply(db, run):
|
||||||
w = wave_row(run, index)
|
w = wave_row(run, index)
|
||||||
@@ -769,7 +787,7 @@ async def auto_start(db: AsyncSession) -> int:
|
|||||||
problem = _note("auto_start_update_only")
|
problem = _note("auto_start_update_only")
|
||||||
elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
|
elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
|
||||||
problem = _note("plan_changed") # someone changed the plan, its groups or its window since activation
|
problem = _note("plan_changed") # someone changed the plan, its groups or its window since activation
|
||||||
elif not await may_roll_out(db, plan.activated_by or ""):
|
elif not await may_roll_out(db, plan.activated_by or "", plan.activated_by_id):
|
||||||
problem = _note("actor_invalid", user=plan.activated_by)
|
problem = _note("actor_invalid", user=plan.activated_by)
|
||||||
elif resolved["errors"]:
|
elif resolved["errors"]:
|
||||||
problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
|
problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
|
||||||
@@ -786,7 +804,8 @@ async def auto_start(db: AsyncSession) -> int:
|
|||||||
audit_service.context.reset(token)
|
audit_service.context.reset(token)
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
run = await create_run(db, plan, resolved, started_by=plan.activated_by, trigger="window", occurrence=occ.start)
|
run = await create_run(db, plan, resolved, started_by=plan.activated_by, started_by_id=plan.activated_by_id,
|
||||||
|
trigger="window", occurrence=occ.start)
|
||||||
plan.auto_start_error = None
|
plan.auto_start_error = None
|
||||||
await db.commit()
|
await db.commit()
|
||||||
except IntegrityError:
|
except IntegrityError:
|
||||||
|
|||||||
@@ -18,13 +18,19 @@ the configuration like the 6-hourly security check does):
|
|||||||
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
||||||
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
||||||
update may legitimately change defaults.
|
update may legitimately change defaults.
|
||||||
|
* probes — optional per plan (1.32): ping/TCP probes run FROM the firewall (uplink, DNS, an internal
|
||||||
|
server). A probe that worked in the baseline and fails now fails the gate; one that failed
|
||||||
|
already before only warns. This is reachability from the firewall, not traffic through it.
|
||||||
|
|
||||||
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
||||||
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
||||||
"""
|
"""
|
||||||
import asyncio
|
import asyncio
|
||||||
|
import uuid
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
|
|
||||||
|
from fastapi import HTTPException
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
@@ -41,6 +47,8 @@ DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": Tru
|
|||||||
"compliance": False}
|
"compliance": False}
|
||||||
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
||||||
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
||||||
|
MAX_PROBES = 10
|
||||||
|
PROBE_PING_COUNT = 3
|
||||||
|
|
||||||
|
|
||||||
def _r(result: str, code: str, **params) -> dict:
|
def _r(result: str, code: str, **params) -> dict:
|
||||||
@@ -64,7 +72,55 @@ async def _services(host) -> dict | None:
|
|||||||
return None if listing is None else _services_compact(listing)
|
return None if listing is None else _services_compact(listing)
|
||||||
|
|
||||||
|
|
||||||
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
def _probe_key(probe: dict) -> str:
|
||||||
|
return f"{probe['type']} {probe['target']}" + (f":{probe['port']}" if probe.get("port") else "")
|
||||||
|
|
||||||
|
|
||||||
|
async def run_probes(host: OpnsenseHost, probes: list[dict] | None) -> list[dict]:
|
||||||
|
"""Every probe from the firewall, one after the other; a probe that cannot run counts as failed. Audited by
|
||||||
|
diagnostics_service like a probe from the host page, as `system:rollout` when no request context exists."""
|
||||||
|
from app.services import audit_service, diagnostics_service
|
||||||
|
|
||||||
|
if not probes:
|
||||||
|
return []
|
||||||
|
token = None
|
||||||
|
if audit_service.context.get() is None:
|
||||||
|
token = audit_service.context.set({"actor": "system:rollout", "id": str(uuid.uuid4()), "action": "probe"})
|
||||||
|
try:
|
||||||
|
async def one(probe: dict) -> dict:
|
||||||
|
ipproto = "inet6" if ":" in probe["target"] else "inet"
|
||||||
|
try:
|
||||||
|
if probe["type"] == "ping":
|
||||||
|
result = await diagnostics_service.ping(host, probe["target"], ipproto, PROBE_PING_COUNT, None, None)
|
||||||
|
ok = int(result.get("received") or 0) > 0
|
||||||
|
else:
|
||||||
|
result = await diagnostics_service.portprobe(host, probe["target"], int(probe["port"]), ipproto, None)
|
||||||
|
ok = bool(result.get("open"))
|
||||||
|
except (HTTPException, client.OpnsenseAPIError):
|
||||||
|
ok = False
|
||||||
|
return {"key": _probe_key(probe), "ok": ok}
|
||||||
|
|
||||||
|
# Concurrently (diagnostics_service caps it at a few at a time per process), results in plan order.
|
||||||
|
return list(await asyncio.gather(*(one(p) for p in probes[:MAX_PROBES])))
|
||||||
|
finally:
|
||||||
|
if token is not None:
|
||||||
|
audit_service.context.reset(token)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_probes(before: list | None, now: list | None) -> dict:
|
||||||
|
if now is None:
|
||||||
|
return _r("fail", "unreadable")
|
||||||
|
was = {p["key"]: p["ok"] for p in (before or [])}
|
||||||
|
regressed = [p["key"] for p in now if not p["ok"] and was.get(p["key"], True)]
|
||||||
|
still = [p["key"] for p in now if not p["ok"] and was.get(p["key"]) is False]
|
||||||
|
if regressed:
|
||||||
|
return _r("fail", "probe_regressed", probes=", ".join(regressed))
|
||||||
|
if still:
|
||||||
|
return _r("warn", "probe_was_failing", probes=", ".join(still))
|
||||||
|
return _r("ok", "ok", count=len(now))
|
||||||
|
|
||||||
|
|
||||||
|
async def baseline(db: AsyncSession, host: OpnsenseHost, probes: list[dict] | None = None) -> dict:
|
||||||
info, gateways, services = await asyncio.gather(
|
info, gateways, services = await asyncio.gather(
|
||||||
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
||||||
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
||||||
@@ -73,6 +129,7 @@ async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
|||||||
"gateways": gateways,
|
"gateways": gateways,
|
||||||
"services": services,
|
"services": services,
|
||||||
"compliance_score": score,
|
"compliance_score": score,
|
||||||
|
"probes": await run_probes(host, probes) if probes else None,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -137,7 +194,7 @@ async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dic
|
|||||||
|
|
||||||
|
|
||||||
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
||||||
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
|
checks: dict | None = None, compliance_mode: str = "warn", probes: list[dict] | None = None) -> dict:
|
||||||
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
||||||
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
||||||
for name in ALWAYS_ON:
|
for name in ALWAYS_ON:
|
||||||
@@ -190,6 +247,10 @@ async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict,
|
|||||||
if enabled["services"]:
|
if enabled["services"]:
|
||||||
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
||||||
|
|
||||||
|
if probes:
|
||||||
|
# A probe new since the baseline (plan edited mid-run) has nothing to compare with: treated as "worked before".
|
||||||
|
row["probes"] = _check_probes(before.get("probes"), await run_probes(host, probes) if reachable else None)
|
||||||
|
|
||||||
blockers = await _shared_change_blockers(db, fleets[hid])
|
blockers = await _shared_change_blockers(db, fleets[hid])
|
||||||
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
||||||
|
|
||||||
|
|||||||
@@ -110,6 +110,8 @@ def definition(plan: RolloutPlan) -> dict:
|
|||||||
return {
|
return {
|
||||||
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
||||||
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||||
|
# Only when set: an activated plan from before 1.32 must keep its fingerprint (else its automatic start stops).
|
||||||
|
**({"gate_probes": list(plan.gate_probes)} if plan.gate_probes else {}),
|
||||||
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
||||||
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,120 @@
|
|||||||
|
"""Cleanup candidates of one firewall: unused, shadowed, duplicate and long-disabled rules (read-only).
|
||||||
|
|
||||||
|
Nothing here changes a firewall. The list says *which rules are worth a look and how sure that is*; removing
|
||||||
|
them is a separate step (not part of this module).
|
||||||
|
|
||||||
|
Categories
|
||||||
|
* ``unused`` OPNexus has watched the rule's pf counters for at least N days and saw no hit
|
||||||
|
(services/rule_hit_service.py; N is a parameter, default 30).
|
||||||
|
* ``shadowed`` an earlier quick rule already decides everything the rule matches (rule_analysis/coverage.py).
|
||||||
|
* ``duplicate`` like shadowed, with the same action and the rule also covering the earlier one.
|
||||||
|
* ``disabled_long`` disabled, and OPNexus has seen it unchanged for N days or more (rule history).
|
||||||
|
|
||||||
|
Confidence (never higher than what is proven; what is not proven is never "sure")
|
||||||
|
* ``sure`` the evidence is complete: unused = the rule is loaded in pf, has no schedule, the counters were
|
||||||
|
sampled recently and over the whole window; shadowed/duplicate = literal comparison, order from
|
||||||
|
``sort_order``, the covering rule is loaded; disabled_long = observed unchanged for N days.
|
||||||
|
* ``probable`` relies on a guess or on static alias content, or pf's loaded ruleset is not available, or the rule
|
||||||
|
has a schedule (a hit might happen at another time), or (cluster) some node has no usage data for
|
||||||
|
the rule. In a cluster "unused" is judged over all nodes (rule_hit_service.merge_cluster_hits):
|
||||||
|
the CARP backup sees almost no traffic, so its zero alone never makes a rule unused.
|
||||||
|
* ``unknown`` there is a hint but no support: the rule is not loaded in pf (no counters can exist), or the
|
||||||
|
counters are stale.
|
||||||
|
|
||||||
|
``removable`` says whether the rule could be offered for removal at all; the reason is a code:
|
||||||
|
``automatic`` (generated by OPNsense), ``legacy`` (not yet migrated, migrate it in OPNsense first),
|
||||||
|
``managed`` (OPNexus manages the rule: remove it through the normal flow of the rules page).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
from app.services import rule_hit_service
|
||||||
|
from app.services.rule_analysis import coverage
|
||||||
|
from app.services.rule_analysis.context import Context
|
||||||
|
from app.services.rule_analysis.model import SECTION_ANTISPOOF, SECTION_INVALID, SECTION_NAT_AUTO, NormRule
|
||||||
|
|
||||||
|
CATEGORIES = ("unused", "shadowed", "duplicate", "disabled_long")
|
||||||
|
DEFAULT_UNUSED_DAYS = 30
|
||||||
|
MIN_DAYS, MAX_DAYS = 1, 3650
|
||||||
|
|
||||||
|
|
||||||
|
def removal_block(rule: NormRule, managed: dict | None) -> str | None:
|
||||||
|
"""Why the rule could not be offered for removal (None = it could)."""
|
||||||
|
if rule.automatic or rule.section in (SECTION_ANTISPOOF, SECTION_NAT_AUTO):
|
||||||
|
return "automatic"
|
||||||
|
if rule.legacy:
|
||||||
|
return "legacy"
|
||||||
|
if managed is not None:
|
||||||
|
return "managed"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def unused_confidence(rule: NormRule, hit, now: datetime) -> str:
|
||||||
|
"""Confidence of an 'unused' verdict (the caller already knows the window passed without a hit)."""
|
||||||
|
if rule.loaded is False:
|
||||||
|
return "unknown" # pf did not load it (schedule, missing interface, gateway down): no counter can exist
|
||||||
|
if hit.last_sampled_at is None or now - hit.last_sampled_at > rule_hit_service.SAMPLE_FRESH:
|
||||||
|
return "unknown" # stale counters say nothing about the present
|
||||||
|
if rule.loaded is None or rule.sched:
|
||||||
|
return "probable" # pf's ruleset not available / the rule only works at certain times
|
||||||
|
if getattr(hit, "peer_missing", False):
|
||||||
|
return "probable" # cluster: some node has no usage data for this rule (rule_hit_service.ClusterHit)
|
||||||
|
return "sure"
|
||||||
|
|
||||||
|
|
||||||
|
def _name(rule: NormRule) -> str:
|
||||||
|
return rule.description or rule.label or rule.uuid or (f"@{rule.pf_numbers[0]}" if rule.pf_numbers else "")
|
||||||
|
|
||||||
|
|
||||||
|
def build(rules: list[NormRule], ctx: Context, *, hits: dict, states: dict, managed: dict[str, dict],
|
||||||
|
now: datetime, unused_days: int = DEFAULT_UNUSED_DAYS, order_guessed: bool = False) -> list[dict]:
|
||||||
|
"""Candidate rows, in evaluation order. ``hits``: RuleHit per rule label, ``states``: LiveRuleState per label,
|
||||||
|
``managed``: {label: {"kind", "id", "name"}} of the rules OPNexus deployed to this firewall."""
|
||||||
|
unused_days = max(MIN_DAYS, min(MAX_DAYS, int(unused_days)))
|
||||||
|
overlaps = {id(o.rule): o for o in coverage.analyse(rules, ctx, order_guessed=order_guessed)}
|
||||||
|
rows: list[dict] = []
|
||||||
|
for rule in rules:
|
||||||
|
if rule.automatic or rule.section in (SECTION_ANTISPOOF, SECTION_INVALID, SECTION_NAT_AUTO):
|
||||||
|
continue
|
||||||
|
categories: list[dict] = []
|
||||||
|
key = rule.uuid.lower()
|
||||||
|
hit = hits.get(key)
|
||||||
|
if hit is not None and rule_hit_service.classify(hit, now, unused_days) == "unused":
|
||||||
|
category = {"kind": "unused", "confidence": unused_confidence(rule, hit, now), "days": unused_days}
|
||||||
|
if getattr(hit, "peer_missing", False):
|
||||||
|
category["notes"] = ["CLUSTER_PEER_USAGE_UNKNOWN"]
|
||||||
|
categories.append(category)
|
||||||
|
overlap = overlaps.get(id(rule))
|
||||||
|
if overlap is not None:
|
||||||
|
categories.append({
|
||||||
|
"kind": overlap.kind, "confidence": overlap.confidence,
|
||||||
|
"by": {"uuid": overlap.by.uuid, "description": _name(overlap.by), "section": overlap.by.section,
|
||||||
|
"automatic": overlap.by.automatic},
|
||||||
|
})
|
||||||
|
state = states.get(key)
|
||||||
|
if (not rule.enabled and state is not None and state.present
|
||||||
|
and now - state.changed_at >= timedelta(days=unused_days)):
|
||||||
|
categories.append({"kind": "disabled_long", "confidence": "sure", "days": unused_days,
|
||||||
|
"since": state.changed_at})
|
||||||
|
if not categories:
|
||||||
|
continue
|
||||||
|
block = removal_block(rule, managed.get(key))
|
||||||
|
rows.append({
|
||||||
|
**rule.summary(), "enabled": rule.enabled, "categories": categories,
|
||||||
|
"hit_state": rule_hit_service.classify(hit, now, unused_days) if hit is not None else None,
|
||||||
|
"last_hit_at": hit.last_hit_at if hit is not None else None,
|
||||||
|
"watched_since": hit.first_seen_at if hit is not None else None,
|
||||||
|
"removable": block is None, "not_removable_reason": block,
|
||||||
|
"managed_by": managed.get(key),
|
||||||
|
})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def summarize(rows: Iterable[dict]) -> dict:
|
||||||
|
counts = {kind: 0 for kind in CATEGORIES}
|
||||||
|
for row in rows:
|
||||||
|
for category in row["categories"]:
|
||||||
|
counts[category["kind"]] += 1
|
||||||
|
return counts
|
||||||
@@ -0,0 +1,290 @@
|
|||||||
|
"""Does an earlier rule make a later one pointless? (shadowed / duplicate)
|
||||||
|
|
||||||
|
Pure functions on ``NormRule``; no I/O. Used by the security check (``compliance_service``, rules taken
|
||||||
|
from config.xml) and by the cleanup candidates (rules taken from ``searchRule``).
|
||||||
|
|
||||||
|
pf evaluates all rules in order, the last matching rule wins unless a matching rule is ``quick``. So a later
|
||||||
|
rule b can only be made dead by an earlier rule a when
|
||||||
|
|
||||||
|
* a is ``quick`` (a non-quick match is overridden by later rules and decides nothing by itself),
|
||||||
|
* a is in force (enabled, not known to be unloaded) and has no condition we cannot compare
|
||||||
|
(schedule, tags, ``received-on``, TCP flags, rate limits), and
|
||||||
|
* every packet b matches is also matched by a: set coverage per field (interfaces, direction, address
|
||||||
|
family, protocols, source/destination, ports, ICMP types).
|
||||||
|
|
||||||
|
The order is ``sort_order`` (floating 200000 before groups 300000+ before interfaces 400000, then the
|
||||||
|
sequence); nothing is re-implemented here, ``NormRule.order`` already holds it.
|
||||||
|
|
||||||
|
Every comparison answers ``"sure"`` (literal values or identical names), ``"probable"`` (the result relies
|
||||||
|
on the static content of aliases or on a guessed order) or ``None`` (not provable, so no claim is made).
|
||||||
|
Whatever is not proven is never "sure": the weakest link decides.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from app.services.rule_analysis.context import Context
|
||||||
|
from app.services.rule_analysis.model import (
|
||||||
|
SECTION_ANTISPOOF, SECTION_FLOATING, SECTION_GROUP, SECTION_INTERFACE, SECTION_INVALID, SECTION_NAT_AUTO,
|
||||||
|
Endpoint, NormRule,
|
||||||
|
)
|
||||||
|
from app.services.rule_analysis.normalize import _af, _protos
|
||||||
|
from app.services.rule_analysis.resolve import ADDRESS_TYPES, MAX_ALIAS_DEPTH, port_ranges
|
||||||
|
|
||||||
|
SURE, PROBABLE = "sure", "probable"
|
||||||
|
_RANK = {None: 0, PROBABLE: 1, SURE: 2}
|
||||||
|
|
||||||
|
|
||||||
|
def weakest(*values: str | None) -> str | None:
|
||||||
|
"""The lowest confidence of the arguments (None beats everything: one unproven part spoils the claim)."""
|
||||||
|
return min(values, key=lambda v: _RANK[v]) if values else SURE
|
||||||
|
|
||||||
|
|
||||||
|
def _empty_context() -> Context:
|
||||||
|
return Context(rows=[], pf=None, interfaces={}, groups=None, aliases={}, port_forwards=None)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Addresses ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
Range = tuple[int, int, int] # (ip version, low, high)
|
||||||
|
|
||||||
|
|
||||||
|
def _literal(token: str) -> Range | None:
|
||||||
|
"""CIDR, single address or ``low-high`` range as an integer interval; None for names."""
|
||||||
|
token = token.strip()
|
||||||
|
try:
|
||||||
|
if "-" in token and not token.startswith("-"):
|
||||||
|
low, _, high = token.partition("-")
|
||||||
|
lo, hi = ipaddress.ip_address(low.strip()), ipaddress.ip_address(high.strip())
|
||||||
|
return (lo.version, int(lo), int(hi)) if lo.version == hi.version and lo <= hi else None
|
||||||
|
net = ipaddress.ip_network(token, strict=False)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
return (net.version, int(net.network_address), int(net.broadcast_address))
|
||||||
|
|
||||||
|
|
||||||
|
def _alias_ranges(name: str, ctx: Context, depth: int = 0, seen: tuple = ()) -> list[Range] | None:
|
||||||
|
"""Static content of a host/network alias as intervals; None when anything about it is not static
|
||||||
|
(tables, URLs, FQDNs, exclusions, unknown names, cycles)."""
|
||||||
|
alias = ctx.aliases.get(name)
|
||||||
|
if alias is None or name in seen or depth > MAX_ALIAS_DEPTH:
|
||||||
|
return None
|
||||||
|
if str(alias.get("type") or "").lower() not in ADDRESS_TYPES:
|
||||||
|
return None
|
||||||
|
if str(alias.get("enabled", "1")).strip().lower() in ("0", "false", "no", "off"):
|
||||||
|
return None # a disabled alias is an empty table in pf: its configured content proves nothing
|
||||||
|
result: list[Range] = []
|
||||||
|
for entry in (e.strip() for e in re.split(r"[\n,]", str(alias.get("content") or "")) if e.strip()):
|
||||||
|
if entry.startswith("!"):
|
||||||
|
return None
|
||||||
|
literal = _literal(entry)
|
||||||
|
if literal is not None:
|
||||||
|
result.append(literal)
|
||||||
|
continue
|
||||||
|
nested = _alias_ranges(entry, ctx, depth + 1, (*seen, name)) if entry in ctx.aliases else None
|
||||||
|
if nested is None:
|
||||||
|
return None
|
||||||
|
result += nested
|
||||||
|
return result or None
|
||||||
|
|
||||||
|
|
||||||
|
def _ranges_of(token: str, ctx: Context) -> tuple[list[Range] | None, bool]:
|
||||||
|
"""(intervals, came_from_alias)"""
|
||||||
|
literal = _literal(token)
|
||||||
|
if literal is not None:
|
||||||
|
return [literal], False
|
||||||
|
return _alias_ranges(token, ctx), True
|
||||||
|
|
||||||
|
|
||||||
|
def _covered(inner: list[Range], outer: list[Range]) -> bool:
|
||||||
|
return all(any(o[0] == i[0] and o[1] <= i[1] and i[2] <= o[2] for o in outer) for i in inner)
|
||||||
|
|
||||||
|
|
||||||
|
def _item_covers(x: str, y: str, ctx: Context) -> str | None:
|
||||||
|
"""Does the address item x contain everything the item y contains?"""
|
||||||
|
x, y = x.strip(), y.strip()
|
||||||
|
if x == y:
|
||||||
|
return SURE
|
||||||
|
outer, outer_alias = _ranges_of(x, ctx)
|
||||||
|
inner, inner_alias = _ranges_of(y, ctx)
|
||||||
|
if outer is None or inner is None or not _covered(inner, outer):
|
||||||
|
return None
|
||||||
|
return PROBABLE if (outer_alias or inner_alias) else SURE
|
||||||
|
|
||||||
|
|
||||||
|
def endpoint_covers(a: Endpoint, b: Endpoint, ctx: Context) -> str | None:
|
||||||
|
if a.is_any and not a.negated:
|
||||||
|
return SURE
|
||||||
|
if b.is_any and not b.negated:
|
||||||
|
return None # a is narrower than "any"
|
||||||
|
if a.negated != b.negated:
|
||||||
|
return None
|
||||||
|
if not a.negated: # every item of b lies inside some item of a
|
||||||
|
parts = []
|
||||||
|
for y in b.items:
|
||||||
|
best = None
|
||||||
|
for x in a.items:
|
||||||
|
got = _item_covers(x, y, ctx)
|
||||||
|
if _RANK[got] > _RANK[best]:
|
||||||
|
best = got
|
||||||
|
parts.append(best)
|
||||||
|
return weakest(*parts)
|
||||||
|
# "!A" covers "!B" when A lies inside B (outside of B is a subset of outside of A)
|
||||||
|
parts = []
|
||||||
|
for x in a.items:
|
||||||
|
best = None
|
||||||
|
for y in b.items:
|
||||||
|
got = _item_covers(y, x, ctx)
|
||||||
|
if _RANK[got] > _RANK[best]:
|
||||||
|
best = got
|
||||||
|
parts.append(best)
|
||||||
|
return weakest(*parts)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Ports, interfaces, the rest ───────────────────────────────────────────
|
||||||
|
|
||||||
|
def _norm_ports(spec: str) -> str:
|
||||||
|
return re.sub(r"\s+", "", spec or "").lower()
|
||||||
|
|
||||||
|
|
||||||
|
def ports_cover(a: str, b: str, ctx: Context) -> str | None:
|
||||||
|
if _norm_ports(a) in ("", "any"):
|
||||||
|
return SURE
|
||||||
|
if _norm_ports(b) in ("", "any"):
|
||||||
|
return None
|
||||||
|
if _norm_ports(a) == _norm_ports(b):
|
||||||
|
return SURE
|
||||||
|
outer, _ = port_ranges(a, ctx)
|
||||||
|
inner, _ = port_ranges(b, ctx)
|
||||||
|
if outer is None or inner is None:
|
||||||
|
return None
|
||||||
|
if not all(any(lo <= il and ih <= hi for lo, hi in outer) for il, ih in inner):
|
||||||
|
return None
|
||||||
|
alias_involved = any(part.strip() in ctx.aliases for spec in (a, b) for part in re.split(r"[\n,]", spec))
|
||||||
|
return PROBABLE if alias_involved else SURE
|
||||||
|
|
||||||
|
|
||||||
|
def _expand(names: set[str], ctx: Context) -> set[str]:
|
||||||
|
result: set[str] = set()
|
||||||
|
for name in names:
|
||||||
|
result |= (ctx.groups[name] | {name}) if ctx.groups and name in ctx.groups else {name}
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def interfaces_cover(a: NormRule, b: NormRule, ctx: Context) -> str | None:
|
||||||
|
if not a.interfaces and not a.if_not:
|
||||||
|
return SURE # every interface
|
||||||
|
if a.if_not or b.if_not:
|
||||||
|
return SURE if (a.interfaces, a.if_not) == (b.interfaces, b.if_not) else None
|
||||||
|
if not b.interfaces:
|
||||||
|
return None # b applies everywhere, a only to some interfaces
|
||||||
|
return SURE if _expand(b.interfaces, ctx) <= _expand(a.interfaces, ctx) else None
|
||||||
|
|
||||||
|
|
||||||
|
def _conditions_ok(a: NormRule) -> bool:
|
||||||
|
"""a must not depend on something we cannot compare: schedule, tags, received-on, TCP flags, limits."""
|
||||||
|
return not (a.sched or a.tagged or a.received_on or a.extra_notes)
|
||||||
|
|
||||||
|
|
||||||
|
def covers_fields(a: NormRule, b: NormRule, ctx: Context | None = None) -> str | None:
|
||||||
|
"""Every packet rule b matches is already matched by rule a (order and enabled state not considered)."""
|
||||||
|
ctx = ctx or _empty_context()
|
||||||
|
if a.quick is not True or not _conditions_ok(a):
|
||||||
|
return None
|
||||||
|
if a.direction != "any" and a.direction != b.direction:
|
||||||
|
return None
|
||||||
|
if not b.af <= a.af:
|
||||||
|
return None
|
||||||
|
if a.protos is not None and (b.protos is None or not b.protos <= a.protos):
|
||||||
|
return None
|
||||||
|
if a.icmptypes is not None and (b.icmptypes is None or not b.icmptypes <= a.icmptypes):
|
||||||
|
return None
|
||||||
|
parts = [
|
||||||
|
interfaces_cover(a, b, ctx),
|
||||||
|
endpoint_covers(a.src, b.src, ctx), endpoint_covers(a.dst, b.dst, ctx),
|
||||||
|
ports_cover(a.sport, b.sport, ctx), ports_cover(a.dport, b.dport, ctx),
|
||||||
|
]
|
||||||
|
return weakest(*parts)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Order-aware analysis ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def _in_force(rule: NormRule) -> bool:
|
||||||
|
return rule.enabled and rule.loaded is not False and rule.section not in (SECTION_INVALID, SECTION_NAT_AUTO)
|
||||||
|
|
||||||
|
|
||||||
|
# normalize() notes after which the evaluation order is not trustworthy: findings stay at "probable" at most.
|
||||||
|
ORDER_DOUBT = frozenset({"ORDER_GUESSED", "ORDER_MISMATCH"})
|
||||||
|
|
||||||
|
|
||||||
|
def order_doubtful(notes) -> bool:
|
||||||
|
return bool(ORDER_DOUBT & set(notes))
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Overlap:
|
||||||
|
rule: NormRule # the later rule that is made pointless
|
||||||
|
by: NormRule # the earlier rule that already decides everything the later one matches
|
||||||
|
kind: str # "shadowed" | "duplicate"
|
||||||
|
confidence: str # "sure" | "probable"
|
||||||
|
|
||||||
|
|
||||||
|
def analyse(rules: list[NormRule], ctx: Context | None = None, *, order_guessed: bool = False) -> list[Overlap]:
|
||||||
|
"""``rules`` in evaluation order. Each later rule is reported at most once, with the first earlier rule that
|
||||||
|
covers it. Duplicate = same action and the later rule also covers the earlier one."""
|
||||||
|
ctx = ctx or _empty_context()
|
||||||
|
result: list[Overlap] = []
|
||||||
|
for index, later in enumerate(rules):
|
||||||
|
if not later.enabled or later.section in (SECTION_ANTISPOOF, SECTION_INVALID, SECTION_NAT_AUTO):
|
||||||
|
continue
|
||||||
|
if later.action is None:
|
||||||
|
continue
|
||||||
|
for earlier in rules[:index]:
|
||||||
|
if not _in_force(earlier) or earlier.action is None:
|
||||||
|
continue
|
||||||
|
confidence = covers_fields(earlier, later, ctx)
|
||||||
|
if confidence is None:
|
||||||
|
continue
|
||||||
|
# (Equal sort_order without pf line numbers always comes with loaded=None, i.e. "probable" here.)
|
||||||
|
if order_guessed or not (earlier.order_known and later.order_known) or earlier.loaded is None:
|
||||||
|
confidence = weakest(confidence, PROBABLE)
|
||||||
|
duplicate = earlier.action == later.action and covers_fields(later, earlier, ctx) is not None
|
||||||
|
if duplicate:
|
||||||
|
confidence = weakest(confidence, covers_fields(later, earlier, ctx))
|
||||||
|
result.append(Overlap(later, earlier, "duplicate" if duplicate else "shadowed", confidence or PROBABLE))
|
||||||
|
break
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# ── Rules read from a config.xml (security check) ─────────────────────────
|
||||||
|
|
||||||
|
def config_order(rule: dict) -> tuple[int, int]:
|
||||||
|
"""sort_order of a rule from config.xml in OPNsense's terms: floating 200000, interface group 300000,
|
||||||
|
single interface 400000, each by sequence."""
|
||||||
|
return (200000 if rule["floating"] else 300000 if rule.get("group") else 400000, rule["order"])
|
||||||
|
|
||||||
|
|
||||||
|
def _config_endpoint(raw: str, negated: bool) -> Endpoint:
|
||||||
|
raw = (raw or "").strip()
|
||||||
|
items = [] if raw.lower() in ("", "any") else [i.strip() for i in raw.split(",") if i.strip()]
|
||||||
|
return Endpoint(raw=raw, negated=negated, items=items)
|
||||||
|
|
||||||
|
|
||||||
|
def from_config_rule(rule: dict) -> NormRule:
|
||||||
|
"""Adapter for the rule dicts of compliance_service (MVC and legacy shapes already merged). Nothing is known
|
||||||
|
about the loaded pf ruleset there, so ``loaded`` stays None (coverage then caps at "probable")."""
|
||||||
|
order = config_order(rule)
|
||||||
|
section = SECTION_FLOATING if rule["floating"] else SECTION_GROUP if rule.get("group") else SECTION_INTERFACE
|
||||||
|
return NormRule(
|
||||||
|
uuid="", label="", description=rule["descr"], order=order, order_known=True, section=section,
|
||||||
|
enabled=rule["enabled"], action=rule["action"] if rule["action"] in ("pass", "block", "reject") else None,
|
||||||
|
quick=bool(rule["quick"]), interfaces=set(rule["interfaces"]), if_not=bool(rule.get("if_not")),
|
||||||
|
direction=rule["direction"] or "in", af=_af(rule["ipprotocol"]), protos=_protos(rule["protocol"]),
|
||||||
|
src=_config_endpoint(rule["src"], rule["src_not"]), dst=_config_endpoint(rule["dst"], rule["dst_not"]),
|
||||||
|
sport=rule["sport"] or "", dport=rule["dport"] or "",
|
||||||
|
icmptypes={t.strip() for t in (rule.get("icmptype") or "").split(",") if t.strip()} or None,
|
||||||
|
tagged=rule.get("tagged") or "", received_on=rule.get("received_on") or "", sched=rule.get("sched") or "",
|
||||||
|
extra_notes=["RATE_LIMIT"] if rule.get("limited") else [],
|
||||||
|
)
|
||||||
Loaded 100 of 221 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user