Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d93b154660 | ||
|
|
cf9981c6eb | ||
|
|
4eea6424e7 | ||
|
|
a8ca0a14ef |
No files matched your search
+77
-1
@@ -29,7 +29,83 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
|
||||
den Einstellungen.
|
||||
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Merlin …
|
||||
ansteht): Saker …
|
||||
|
||||
## [1.33.0] "Saker" — 2026-10-24
|
||||
|
||||
- **Aus dem Testlauf 1.31.0 behoben (T2 bis T6, B15):**
|
||||
- **Benutzerliste:** `GET /api/users` zeigte jedem angemeldeten Konto (auch Nur-Lese, auch per Token) alle Benutzernamen, Rollen, letzte Anmeldung und 2FA-Status.
|
||||
Jetzt sehen **nur Admins** die ganze Liste, alle anderen nur das eigene Konto.
|
||||
- **Hosts:** Dieselbe Adresse zweimal einzutragen wird abgelehnt (`HOST_URL_EXISTS`, auch beim Ändern der Adresse). Die Hosts-Seite und der Einrichtungsassistent
|
||||
prüfen die Verbindung beim Anlegen (Schlüssel abgelehnt / zu wenig Rechte / Zertifikat / keine Verbindung), speichern bei einem Fehler nichts und bieten
|
||||
„Trotzdem anlegen“. Über die API ist das `verify_connection: true` (Standard aus). **Löschen verlangt jetzt auch am Server den Hostnamen:**
|
||||
`DELETE /api/hosts/{id}?confirm_name=<Name>` (sonst 409); Skripte, die Hosts löschen, müssen das anpassen.
|
||||
- **Zertifikate:** `cert_type: v3_ca` wird nicht mehr angenommen (es entstand ein Zertifikat mit CA-Erweiterung, keine nutzbare CA); die Oberfläche bot es nie an.
|
||||
- **Traceroute:** Zwischenstationen ohne Antwort stehen als Zeile „keine Antwort“ in der Liste (vorher sprang die Zählung 1 → 3); ein Weg ganz ohne Antwort wird erklärt.
|
||||
- **Netzwerk:** Die Vorschau prüft Zahlenbereiche vorab (VLAN-Tag 1–4094, PCP 0–7, VHID 1–255, Basis 1–254, Skew 0–254, gegen das Modell der OPNsense gelesen): `422` mit dem
|
||||
Feldnamen statt `502` von der Firewall.
|
||||
- **Gruppen:** Eine Gruppe, die ein VPN-Objekt als Ziel hat, lässt sich nicht mehr löschen (die Verknüpfung fiel still weg).
|
||||
- Das Skript `scripts/verify_i18n.mjs` läuft jetzt auch gegen die HTTPS-Test-Instanz.
|
||||
- **Regel-Bereinigung: OPNexus selbst ist automatisch geschützt (B10).** Beim Planen fragt die Bereinigung jede Firewall, von welcher Adresse sie OPNexus kommen sieht
|
||||
(die eigene API-Verbindung in der Zustandstabelle), und schützt diese Adresse zusätzlich zu den Verwaltungsquellen. Vorher konnte eine Regel abgeschaltet werden, über
|
||||
die nur OPNexus die Firewall erreicht, wenn in den Quellen nur Arbeitsplätze standen; dann scheiterte auch das automatische Zurücksetzen. Die Vorschau zeigt die Adresse
|
||||
je Firewall (nur Admins) oder sagt, dass sie sich nicht ermitteln ließ (API-Recht „Diagnose: Zustände“, Management-Tunnel).
|
||||
- **Keine verlorenen Regeltreffer mehr (B11):** Jedes Anwenden setzt die Regelzähler der Firewall zurück. OPNexus liest sie jetzt unmittelbar vor jedem Anwenden, das es
|
||||
selbst auslöst (Regeln, NAT, Aliase, IPsec/OpenVPN/WireGuard, Netzwerk); vorher konnte eine Regel deshalb fälschlich als „ungenutzt“ gelten.
|
||||
- **Einstellungen › Regel-Bereinigung:** schlägt die eigene Browser-Adresse als Verwaltungsquelle vor (B12). Geprüft: OPNsense 26.7 hat kein „savepoint“ zum Zurücknehmen
|
||||
durch die Firewall selbst (Quellcode und live 404), das automatische Zurücksetzen bleibt bei OPNexus.
|
||||
|
||||
- **Installation unter Windows:** Neue Anleitung auf der Website (`/docs/windows/`, auch auf Englisch) und im Handbuch: Docker Desktop mit WSL 2, Zertifizierungsstelle importieren, Zugriff aus dem Netz (Windows-Firewall), Updates mit `opnexus` in WSL und Fehlerbehebung. Entwickelt und getestet wird weiterhin unter Linux; die Anleitung sagt das offen.
|
||||
- **Sicherheit und Stabilität (aus einer externen Prüfung):**
|
||||
- **Rückweg bleibt erhalten:** Wer nach „Konfiguration geschrieben, Anwenden fehlgeschlagen“ erneut ausrollte, überschrieb den ursprünglichen Rückweg; die automatische
|
||||
Rücknahme stellte dann den unbestätigten Stand wieder her. Jetzt bleibt der Rückweg der Stand vor dem ersten Versuch. Wartung, Regel-Bereinigung und
|
||||
„Konfiguration als Code“ behandeln eine solche Zeile wie eine offene Bestätigung. Gilt für Regeln, Aliase, Port-Forwards, 1:1-NAT, Outbound-NAT und Hierarchie-Regeln.
|
||||
- **Anmeldung und 2FA:** Eine Passwort-Anmeldung, die genau während der Aktivierung von 2FA lief, bekam eine Sitzung ohne zweiten Faktor. Jetzt wird sie mit `TOTP_REQUIRED` abgewiesen.
|
||||
- **Letzter Administrator:** Zwei Administratoren, die sich gleichzeitig herabstuften oder gegenseitig löschten, konnten alle Administratoren entfernen. Beides ist jetzt serialisiert.
|
||||
- **Operatoren:** Mit ausgeschaltetem „Operatoren dürfen ausrollen“ konnten sie noch Snapshots (Boot-Umgebungen) anlegen. Ein Test hält künftig jede schreibende Route
|
||||
fest, die keiner Rechte-Regel zugeordnet ist.
|
||||
- **Zertifikate und ACME:** Der Ziel-Host ist nach dem ersten Push fest (`409 ZIEL_HOST_NACH_AUSROLLEN_FEST`); vorher meldete ein Host-Wechsel Erfolg für einen Host, der nie
|
||||
etwas erhielt. Ein ACME-Konto mit Anfragen lässt sich nicht auf einen anderen Host verschieben (`ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET`).
|
||||
- **VPN-Ausrollen:** IPsec, OpenVPN und WireGuard sichern jede von der Firewall vergebene Kennung sofort; bei einem Abbruch entstehen beim nächsten Versuch keine Duplikate.
|
||||
Pro VPN läuft nur ein Ausrollen oder Entfernen zugleich (`409 VPN_DEPLOY_LAEUFT_BEREITS`).
|
||||
- **Konfiguration als Code:** Gültiges YAML mit falsch geformtem Abschnitt (z. B. `aliases: 42`) ergibt `422 CONFIG_INVALID` statt eines Serverfehlers.
|
||||
- **`.env`:** Die Anleitungen legen sie mit `install -m 600` an; `opnexus update` und `opnexus status` setzen eine zu offene Datei auf 0600. `opnexus rollback` liest seinen
|
||||
Update-Zustand nur noch als Text und führt ihn nie aus.
|
||||
- **Abhängigkeiten:** Next.js 15.5.28, PyJWT 2.15.1.
|
||||
|
||||
## [1.32.1] "Merlin" — 2026-10-23
|
||||
|
||||
- **Korrektur: Rollouts erkannten die Zielversion bei anderen Release-Typen falsch.** OPNexus las die neue Version nur am Kernpaket `opnsense`. Bei der
|
||||
Business-Edition (`opnsense-business`) und beim Wechsel des Release-Typs (z. B. auf `opnsense-devel`, dann als neues Paket angeboten) fiel es auf
|
||||
`product_latest` zurück; das Gate hielt den Rollout dann fälschlich mit „Version passt nicht“ an. Live gefunden und belegt (Rollout mit Cluster in einer Welle).
|
||||
- **Plan-Hinweis zu Alias-Namen ehrlich formuliert:** „kein zentraler Alias in OPNexus“ statt „kein Alias dieses Namens“ (von Hand auf der Firewall
|
||||
angelegte Aliase sind in Ordnung; der Planer fragt bewusst keine Firewall).
|
||||
|
||||
## [1.32.0] "Merlin" — 2026-10-23
|
||||
|
||||
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
|
||||
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
|
||||
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
|
||||
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
|
||||
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
|
||||
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
|
||||
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
|
||||
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
|
||||
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
|
||||
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
|
||||
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
|
||||
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
|
||||
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
|
||||
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
|
||||
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
|
||||
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
|
||||
|
||||
## [1.31.1] "Vulture" — 2026-10-22
|
||||
|
||||
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
|
||||
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
|
||||
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
|
||||
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
|
||||
|
||||
## [1.31.0] "Vulture" — 2026-10-22
|
||||
|
||||
|
||||
+7
-4
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||

|
||||
|
||||
> **Stand:** `1.31.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.33.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -21,7 +21,8 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
|
||||
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
|
||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||
@@ -47,7 +48,7 @@ Voraussetzung: Docker mit Compose-Plugin.
|
||||
```bash
|
||||
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
||||
cd opnexus
|
||||
cp .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen
|
||||
install -m 600 .env.example .env # POSTGRES_PASSWORD (und ADMIN_PASSWORD) setzen; nur für dich lesbar
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
@@ -59,13 +60,15 @@ scripts/opnexus password
|
||||
|
||||
Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassistent**: eigenes Passwort setzen, optional Zwei-Faktor-Anmeldung einschalten, die erste Firewall verbinden (samt Anleitung für den API-Key in OPNsense). Er lässt sich überspringen und jederzeit unter System › Einrichtung wieder öffnen.
|
||||
|
||||
**Windows:** läuft in Docker Desktop mit WSL 2, siehe die [Installationsanleitung für Windows](https://opnexus.dev/docs/windows/). Entwickelt und getestet wird unter Linux; für den Dauerbetrieb ist ein Linux-Server oder eine Linux-VM besser.
|
||||
|
||||
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
|
||||
|
||||
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.31.0
|
||||
OPNEXUS_VERSION=1.33.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||

|
||||
|
||||
> **Status:** `1.31.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.33.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -21,7 +21,8 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
|
||||
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
|
||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||
@@ -49,7 +50,7 @@ Requires Docker with the Compose plugin.
|
||||
```bash
|
||||
git clone https://git.opnexus.dev/opnexus/opnexus.git
|
||||
cd opnexus
|
||||
cp .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD)
|
||||
install -m 600 .env.example .env # set POSTGRES_PASSWORD (and ADMIN_PASSWORD); readable by you only
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
@@ -61,13 +62,15 @@ scripts/opnexus password
|
||||
|
||||
After the first sign-in a short **setup assistant** opens on its own: set your own password, optionally enable two-factor sign-in, connect your first firewall (including how to create the API key in OPNsense). You can skip it and reopen it any time under System › Setup.
|
||||
|
||||
**Windows:** run it in Docker Desktop with WSL 2 – see the [installation guide for Windows](https://opnexus.dev/en/docs/windows/). OPNexus is developed and tested on Linux; for permanent operation a Linux server or VM is the better choice.
|
||||
|
||||
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
|
||||
|
||||
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.31.0
|
||||
OPNEXUS_VERSION=1.33.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
"/api/settings",
|
||||
# Read-only JSON-RPC tools (see routers/mcp.py).
|
||||
"/api/mcp",
|
||||
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
|
||||
"/api/config/import/plan",
|
||||
# Self-scoped: a user managing their own second factor.
|
||||
"/api/auth/2fa/setup",
|
||||
"/api/auth/2fa/enable",
|
||||
@@ -121,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
)
|
||||
if user is not None:
|
||||
parts = request.url.path.strip("/").split("/")
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
|
||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||
@@ -131,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
if len(parts) > 2 and parts[1] == "settings":
|
||||
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
|
||||
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
|
||||
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
|
||||
entity = parts[2] if len(parts) > 2 and kind != "config" else None
|
||||
event = {"kind": kind, "action": action, "entity_id": entity,
|
||||
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
|
||||
"details": {"method": request.method, "path": request.url.path}}
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
|
||||
|
||||
if (
|
||||
user.role == "read_only"
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
|
||||
|
||||
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
|
||||
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
|
||||
"""
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
|
||||
|
||||
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
|
||||
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
|
||||
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
|
||||
|
||||
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
|
||||
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
|
||||
|
||||
ACTOR = "cli (Host-Konsole)"
|
||||
TOKEN_MARK = "# plan-token: "
|
||||
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
|
||||
|
||||
ERRORS_DE = {
|
||||
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
|
||||
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
|
||||
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
|
||||
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
|
||||
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
|
||||
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
|
||||
"yaml_empty": "Die Datei ist leer",
|
||||
"yaml_syntax": "YAML-Syntaxfehler",
|
||||
"duplicate_key": "Doppelter Schlüssel",
|
||||
"non_string_key": "Schlüssel muss Text sein",
|
||||
"too_deep": "Zu tief verschachtelt",
|
||||
"too_many_nodes": "Zu viele Elemente",
|
||||
"scalar_too_long": "Wert zu lang",
|
||||
"too_many_objects": "Zu viele Objekte",
|
||||
"too_many_entries": "Zu viele Einträge",
|
||||
"too_many_variables": "Zu viele Variablen für diesen Host",
|
||||
"too_large": "Datei zu groß",
|
||||
"bad_encoding": "Kein gültiges UTF-8",
|
||||
"not_a_mapping": "Die Datei muss ein Mapping sein",
|
||||
"missing_version": "opnexus_config: 1 fehlt",
|
||||
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
|
||||
"unknown_field": "Unbekanntes Feld",
|
||||
"missing_field": "Pflichtfeld fehlt",
|
||||
"bad_type": "Falscher Typ",
|
||||
"bad_value": "Ungültiger Wert",
|
||||
"bad_format": "Falsches Format",
|
||||
"too_short": "Zu kurz",
|
||||
"too_long": "Zu lang",
|
||||
"control_character": "Steuerzeichen nicht erlaubt",
|
||||
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
|
||||
"bad_variable_name": "Ungültiger Variablenname",
|
||||
"bad_variable_value": "Ungültiger Variablenwert",
|
||||
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
|
||||
"duplicate_entry": "Eintrag kommt doppelt vor",
|
||||
"unknown_host": "Unbekannter Host",
|
||||
"unknown_group": "Unbekannte Gerätegruppe",
|
||||
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
|
||||
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
|
||||
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
|
||||
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
|
||||
}
|
||||
REASONS_DE = {
|
||||
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
|
||||
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
|
||||
"prune_referenced": "wird noch verwendet",
|
||||
}
|
||||
WARNINGS_DE = {
|
||||
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
|
||||
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
|
||||
"unknown_alias_reference": "{value}: kein zentraler Alias in OPNexus (auf der Firewall von Hand angelegt? dann in Ordnung)",
|
||||
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
|
||||
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
|
||||
}
|
||||
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
|
||||
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
|
||||
|
||||
|
||||
def safe(text) -> str:
|
||||
"""Nothing from a foreign file reaches the terminal as a control sequence."""
|
||||
return _ANSI.sub("?", str(text))
|
||||
|
||||
|
||||
def _short(value) -> str:
|
||||
return safe(display(json.dumps(value, ensure_ascii=False), 60))
|
||||
|
||||
|
||||
def format_errors(errors: list[dict]) -> str:
|
||||
lines = []
|
||||
for e in errors:
|
||||
where = f"Zeile {e['line']}: " if e.get("line") else ""
|
||||
path = f"{safe(e['path'])}: " if e.get("path") else ""
|
||||
detail = ""
|
||||
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
|
||||
if params:
|
||||
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
|
||||
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_plan(public: dict) -> str:
|
||||
lines = []
|
||||
for item in public["items"]:
|
||||
action = item["action"]
|
||||
planned = item["planned_action"]
|
||||
name = f"{item['kind']} {safe(item['key'])}"
|
||||
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
|
||||
if action == "blocked":
|
||||
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
|
||||
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
|
||||
elif planned in LABEL:
|
||||
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
|
||||
else:
|
||||
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
|
||||
if change["field"]:
|
||||
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
|
||||
else:
|
||||
lines.append(f"~ {name}{suffix}")
|
||||
if item.get("implicit"):
|
||||
lines[-1] += " (Folge der Reihenfolge)"
|
||||
for warning in public["warnings"]:
|
||||
template = WARNINGS_DE.get(warning["code"])
|
||||
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
|
||||
text = template.format(**params) if template else warning["code"]
|
||||
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
|
||||
lines.append(f"? {where}{text}")
|
||||
s = public["summary"]
|
||||
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _read_stdin() -> bytes:
|
||||
data = sys.stdin.buffer.read(MAX_BYTES + 1)
|
||||
return data
|
||||
|
||||
|
||||
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
|
||||
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
|
||||
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
|
||||
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
|
||||
parser.add_argument("--kinds", default=None)
|
||||
parser.add_argument("--prune", action="store_true")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--plan-token", default=None)
|
||||
try:
|
||||
args = parser.parse_args(argv)
|
||||
except SystemExit:
|
||||
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
|
||||
if args.command == "schema":
|
||||
return 0, schema.json_schema_text(), ""
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.services import audit_service
|
||||
|
||||
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
if args.command == "export":
|
||||
wanted = None
|
||||
if args.kinds:
|
||||
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
|
||||
return 0, await export_yaml(db, wanted), ""
|
||||
|
||||
raw = stdin if stdin is not None else _read_stdin()
|
||||
try:
|
||||
parsed = parse(raw)
|
||||
if args.command == "plan":
|
||||
plan = await planner.build_plan(db, parsed, args.prune)
|
||||
await db.rollback()
|
||||
public = plan.public()
|
||||
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
|
||||
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
|
||||
s = public["summary"]
|
||||
if not plan.applicable:
|
||||
return 4, out, ""
|
||||
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
|
||||
|
||||
if not args.plan_token:
|
||||
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
|
||||
try:
|
||||
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
s = plan.summary()
|
||||
if args.json:
|
||||
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
|
||||
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
|
||||
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
|
||||
except ConfigError as exc:
|
||||
if args.json:
|
||||
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
|
||||
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
|
||||
finally:
|
||||
audit_service.context.reset(context)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
try:
|
||||
code, out, err = asyncio.run(run(argv))
|
||||
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
|
||||
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
|
||||
return 1
|
||||
sys.stdout.write(out)
|
||||
sys.stderr.write(err)
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,312 @@
|
||||
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
|
||||
|
||||
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
|
||||
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
|
||||
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
|
||||
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
|
||||
|
||||
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
|
||||
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
|
||||
Den Commit macht der Aufrufer.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import Parsed
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
|
||||
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.services import audit_service
|
||||
|
||||
NOTE = "Import (Konfiguration als Code)"
|
||||
AUDIT_KIND = {
|
||||
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
|
||||
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
|
||||
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
|
||||
}
|
||||
|
||||
|
||||
class PlanStale(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PlanBlocked(Exception):
|
||||
def __init__(self, plan: planner.Plan):
|
||||
self.plan = plan
|
||||
super().__init__("plan blocked")
|
||||
|
||||
|
||||
class SelfCheckFailed(Exception):
|
||||
def __init__(self, differences: list[str]):
|
||||
self.differences = differences
|
||||
super().__init__("; ".join(differences[:5]))
|
||||
|
||||
|
||||
def _revision_writer(kind: str):
|
||||
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
|
||||
|
||||
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
|
||||
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
|
||||
"""
|
||||
if kind == "rules":
|
||||
from app.services.policy_service import snapshot
|
||||
return RuleRevision, "rule_id", snapshot
|
||||
if kind == "aliases":
|
||||
from app.services.alias_policy_service import snapshot
|
||||
return AliasRevision, "alias_id", snapshot
|
||||
if kind == "port_forwards":
|
||||
from app.services.port_forward_policy_service import snapshot
|
||||
return PortForwardRevision, "port_forward_id", snapshot
|
||||
if kind == "one_to_one":
|
||||
from app.services.one_to_one_policy_service import snapshot
|
||||
return OneToOneRevision, "one_to_one_id", snapshot
|
||||
if kind == "outbound_nat":
|
||||
from app.services.outbound_nat_policy_service import snapshot
|
||||
return OutboundNatRevision, "outbound_nat_id", snapshot
|
||||
from app.services.policy_hierarchy_service import snapshot
|
||||
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
|
||||
|
||||
|
||||
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
|
||||
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
|
||||
SCALARS = {
|
||||
"aliases": ("name", "type", "description", "enabled"),
|
||||
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "enabled", "log"),
|
||||
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "target", "local_port", "natreflection", "enabled"),
|
||||
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
|
||||
"destination_not", "natreflection", "log", "enabled"),
|
||||
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
|
||||
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
|
||||
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
|
||||
}
|
||||
|
||||
|
||||
class _Writer:
|
||||
def __init__(self, db: AsyncSession, plan: planner.Plan):
|
||||
self.db = db
|
||||
self.plan = plan
|
||||
self.now = datetime.now(timezone.utc)
|
||||
self.events: list[dict] = []
|
||||
|
||||
async def load(self) -> None:
|
||||
db = self.db
|
||||
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
self.groups = {g.name: g for g in (await db.execute(
|
||||
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
|
||||
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
self.rows: dict[str, dict] = {}
|
||||
for kind, model in MODELS.items():
|
||||
options = [selectinload(model.deployments)]
|
||||
if kind != "policy_hierarchy_rules":
|
||||
options += [selectinload(model.targets), selectinload(model.groups)]
|
||||
rows = (await db.execute(select(model).options(*options))).scalars().all()
|
||||
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
|
||||
def items(self, kind: str, *actions: str) -> list[planner.Item]:
|
||||
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
|
||||
|
||||
def event(self, item: planner.Item, entity_id, name: str) -> None:
|
||||
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
|
||||
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
|
||||
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
|
||||
|
||||
# -- groups ----------------------------------------------------------------------------------------------------------
|
||||
async def hierarchy_groups(self) -> None:
|
||||
items = self.items("policy_hierarchy_groups", "create", "update")
|
||||
for item in items:
|
||||
if item.action == "create":
|
||||
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
|
||||
self.db.add(group)
|
||||
self.hierarchy[item.key] = group
|
||||
await self.db.flush()
|
||||
for item in items:
|
||||
group = self.hierarchy[item.key]
|
||||
group.description = item.after["description"]
|
||||
parent = item.after["parent"]
|
||||
group.parent_id = self.hierarchy[parent].id if parent else None
|
||||
self.event(item, group.id, group.name)
|
||||
await self.db.flush()
|
||||
|
||||
async def device_groups(self) -> None:
|
||||
for item in self.items("device_groups", "create", "update"):
|
||||
group = self.groups.get(item.key)
|
||||
if group is None:
|
||||
group = DeviceGroup(name=item.key)
|
||||
self.db.add(group)
|
||||
self.groups[item.key] = group
|
||||
group.description = item.after["description"]
|
||||
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
|
||||
await self.db.flush()
|
||||
self.event(item, group.id, group.name)
|
||||
|
||||
# -- objects ---------------------------------------------------------------------------------------------------------
|
||||
async def objects(self, kind: str) -> None:
|
||||
model = MODELS[kind]
|
||||
revision_model, fk, snapshot = _revision_writer(kind)
|
||||
for item in self.items(kind, "create", "update"):
|
||||
after = item.after
|
||||
row = self.rows[kind].get(item.key)
|
||||
if row is None:
|
||||
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
|
||||
if kind != "aliases":
|
||||
row.config_key = item.key
|
||||
self.db.add(row)
|
||||
self.rows[kind][item.key] = row
|
||||
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
|
||||
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
|
||||
for name in SCALARS[kind]:
|
||||
setattr(row, name, after[name])
|
||||
if kind == "aliases":
|
||||
row.content = "\n".join(after["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
row.hierarchy_group_id = self.hierarchy[after["group"]].id
|
||||
else:
|
||||
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
|
||||
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
|
||||
if "hierarchy_group" in after["targets"]:
|
||||
hierarchy = after["targets"]["hierarchy_group"]
|
||||
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
|
||||
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
|
||||
await self.db.flush()
|
||||
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
|
||||
async def hierarchy_positions(self) -> None:
|
||||
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
|
||||
if "policy_hierarchy_rules" not in self.plan.touched:
|
||||
return
|
||||
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
|
||||
for key, rank in self.plan.positions.items():
|
||||
row = self.rows["policy_hierarchy_rules"].get(key)
|
||||
if row is not None and key not in deleted and row.sequence != rank:
|
||||
row.sequence = rank
|
||||
await self.db.flush()
|
||||
|
||||
async def host_hierarchy(self) -> None:
|
||||
for host_name, group_name in self.plan.host_hierarchy.items():
|
||||
host = self.hosts[host_name]
|
||||
wanted = self.hierarchy[group_name].id if group_name else None
|
||||
if host.policy_hierarchy_group_id != wanted:
|
||||
host.policy_hierarchy_group_id = wanted
|
||||
await self.db.flush()
|
||||
|
||||
async def variables(self) -> None:
|
||||
if "variables" not in self.plan.touched:
|
||||
return
|
||||
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
|
||||
for item in self.items("variables", "create", "update"):
|
||||
host = self.hosts[item.after["host"]]
|
||||
row = existing.get((host.id, item.after["name"]))
|
||||
if row is None:
|
||||
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
|
||||
else:
|
||||
row.value = item.after["value"]
|
||||
self.event(item, None, item.key)
|
||||
for item in self.items("variables", "delete"):
|
||||
host = self.hosts[item.before["host"]]
|
||||
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
|
||||
self.event(item, None, item.key)
|
||||
await self.db.flush()
|
||||
|
||||
# -- deletions -------------------------------------------------------------------------------------------------------
|
||||
async def deletions(self) -> None:
|
||||
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
|
||||
for item in self.items(kind, "delete"):
|
||||
row = self.rows[kind][item.key]
|
||||
if row.deployments: # the planner blocks this; never undeploy from here
|
||||
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
for item in self.items("device_groups", "delete"):
|
||||
row = self.groups[item.key]
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
|
||||
by_id = {g.id: g for g in self.hierarchy.values()}
|
||||
|
||||
def depth(group) -> int:
|
||||
n, current = 0, group
|
||||
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
|
||||
n, current = n + 1, by_id[current.parent_id]
|
||||
return n
|
||||
|
||||
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
|
||||
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
|
||||
|
||||
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
|
||||
db.expire_all() # compare what is in the database, not what the session believes
|
||||
actual = await canonical(db, set(plan.touched))
|
||||
differences: list[str] = []
|
||||
for kind in plan.touched:
|
||||
if kind == "variables":
|
||||
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in actual["variables"].items() for n, v in values.items()}
|
||||
else:
|
||||
ident = planner.KIND_SPECS[kind].ident
|
||||
got = {item[ident]: item for item in actual[kind]}
|
||||
want = plan.expected[kind]
|
||||
for key in sorted(set(got) | set(want)):
|
||||
if got.get(key) != want.get(key):
|
||||
differences.append(f"{kind}:{key}")
|
||||
if "policy_hierarchy_rules" in plan.touched:
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
|
||||
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
|
||||
for key, rank in ranks.items():
|
||||
if plan.positions.get(key) != rank:
|
||||
differences.append(f"policy_hierarchy_rules:{key}:position")
|
||||
if differences:
|
||||
raise SelfCheckFailed(differences)
|
||||
|
||||
|
||||
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
|
||||
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
|
||||
ConfigError; on any exception the caller must roll back."""
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
await policy_lock(db)
|
||||
plan = await planner.build_plan(db, parsed, prune)
|
||||
if plan.token != token:
|
||||
raise PlanStale()
|
||||
if not plan.applicable:
|
||||
raise PlanBlocked(plan)
|
||||
writer = _Writer(db, plan)
|
||||
await writer.load()
|
||||
await writer.hierarchy_groups()
|
||||
await writer.device_groups()
|
||||
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
|
||||
if kind in plan.touched:
|
||||
await writer.objects(kind)
|
||||
await writer.hierarchy_positions()
|
||||
await writer.host_hierarchy()
|
||||
await writer.variables()
|
||||
await writer.deletions()
|
||||
await _self_check(db, plan)
|
||||
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
|
||||
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
|
||||
"summary": plan.summary()}))
|
||||
for values in writer.events:
|
||||
db.add(audit_service.make_event(**values))
|
||||
await db.flush()
|
||||
return plan
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
|
||||
|
||||
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
|
||||
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
|
||||
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
|
||||
"""
|
||||
import yaml
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.version import APP_VERSION
|
||||
|
||||
HEADER = (
|
||||
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
|
||||
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
|
||||
)
|
||||
|
||||
|
||||
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
|
||||
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
|
||||
fields = list(doc.model_fields)
|
||||
if set(values) != set(fields):
|
||||
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
|
||||
return {name: values[name] for name in fields}
|
||||
|
||||
|
||||
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
|
||||
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
|
||||
if "hierarchy_group" in doc.model_fields:
|
||||
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
|
||||
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
|
||||
values = dict(extra)
|
||||
for name in doc.model_fields:
|
||||
if name in values:
|
||||
continue
|
||||
values[name] = obj.config_key if name == "key" else getattr(obj, name)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
|
||||
"""Parents before children, siblings by name; the same rule the loader expects."""
|
||||
children: dict = {}
|
||||
for group in groups:
|
||||
children.setdefault(group.parent_id, []).append(group)
|
||||
ordered: list[PolicyHierarchyGroup] = []
|
||||
|
||||
def walk(parent_id) -> None:
|
||||
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
|
||||
ordered.append(group)
|
||||
walk(group.id)
|
||||
|
||||
walk(None)
|
||||
return ordered
|
||||
|
||||
|
||||
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
|
||||
"""{kind: content} for the requested kinds (default all), in file order."""
|
||||
wanted = set(schema.KINDS) if kinds is None else kinds
|
||||
out: dict[str, list | dict] = {}
|
||||
|
||||
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
|
||||
hierarchy_names = {g.id: g.name for g in all_groups}
|
||||
|
||||
if "device_groups" in wanted:
|
||||
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
|
||||
out["device_groups"] = [
|
||||
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
|
||||
for g in sorted(groups, key=lambda g: g.name)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_groups" in wanted:
|
||||
out["policy_hierarchy_groups"] = [
|
||||
_ordered(schema.HierarchyGroupDoc, {
|
||||
"name": g.name,
|
||||
"parent": hierarchy_names.get(g.parent_id),
|
||||
"description": g.description or "",
|
||||
"hosts": sorted(h.name for h in g.hosts),
|
||||
})
|
||||
for g in _hierarchy_order(list(all_groups))
|
||||
]
|
||||
|
||||
if "aliases" in wanted:
|
||||
aliases = (await db.execute(
|
||||
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
|
||||
)).scalars().all()
|
||||
out["aliases"] = [
|
||||
_ordered(schema.AliasDoc, {
|
||||
"name": a.name,
|
||||
"type": a.type,
|
||||
"description": a.description or "",
|
||||
"enabled": a.enabled,
|
||||
"entries": [line for line in (a.content or "").split("\n") if line],
|
||||
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
|
||||
})
|
||||
for a in sorted(aliases, key=lambda a: a.name)
|
||||
]
|
||||
|
||||
simple = (
|
||||
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
|
||||
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
|
||||
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
|
||||
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
|
||||
)
|
||||
for kind, model, doc, targets_doc in simple:
|
||||
if kind not in wanted:
|
||||
continue
|
||||
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
|
||||
out[kind] = [
|
||||
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
|
||||
for row in sorted(rows, key=lambda r: r.config_key)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_rules" in wanted:
|
||||
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
|
||||
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
|
||||
out["policy_hierarchy_rules"] = [
|
||||
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
|
||||
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
|
||||
]
|
||||
|
||||
if "variables" in wanted:
|
||||
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
|
||||
out["variables"] = {
|
||||
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
|
||||
for h in sorted(hosts, key=lambda h: h.name)
|
||||
if h.variables
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def dump_yaml(content: dict[str, list | dict]) -> str:
|
||||
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
|
||||
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
|
||||
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
|
||||
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
|
||||
|
||||
|
||||
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
|
||||
return dump_yaml(await canonical(db, kinds))
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
|
||||
|
||||
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
|
||||
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
|
||||
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
|
||||
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
|
||||
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
|
||||
|
||||
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
|
||||
"""
|
||||
import hashlib
|
||||
import re
|
||||
from typing import NamedTuple
|
||||
|
||||
import yaml
|
||||
from pydantic import ValidationError
|
||||
from yaml.composer import Composer
|
||||
from yaml.constructor import SafeConstructor
|
||||
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
|
||||
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
|
||||
from yaml.parser import Parser
|
||||
from yaml.reader import Reader
|
||||
from yaml.resolver import BaseResolver
|
||||
from yaml.scanner import Scanner
|
||||
|
||||
from app.config_code import schema
|
||||
|
||||
MAX_BYTES = 1024 * 1024
|
||||
MAX_DEPTH = 8
|
||||
MAX_NODES = 100_000
|
||||
MAX_SCALAR = schema.MAX_SCALAR
|
||||
MAX_OBJECTS_PER_KIND = 5000
|
||||
MAX_VARIABLE_HOSTS = 500
|
||||
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
|
||||
MAX_ALIAS_ENTRIES = 2000
|
||||
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
|
||||
|
||||
_STR = "tag:yaml.org,2002:str"
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
|
||||
|
||||
def __init__(self, code: str, errors: list[dict]):
|
||||
self.code = code
|
||||
self.errors = errors
|
||||
super().__init__(f"{code}: {errors[:3]}")
|
||||
|
||||
|
||||
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
|
||||
return {"code": code, "path": path, "line": line, "params": params}
|
||||
|
||||
|
||||
def display(value, limit: int = 80) -> str:
|
||||
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
|
||||
text = value if isinstance(value, str) else repr(value)
|
||||
text = text.encode("unicode_escape").decode("ascii")
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def _path(parts) -> str:
|
||||
out = ""
|
||||
for part in parts:
|
||||
if isinstance(part, int):
|
||||
out += f"[{part}]"
|
||||
else:
|
||||
out += ("." if out else "") + str(part)
|
||||
return out
|
||||
|
||||
|
||||
class _Rejected(yaml.YAMLError):
|
||||
def __init__(self, code: str, mark=None, path=(), **params):
|
||||
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
class StrictResolver(BaseResolver):
|
||||
yaml_implicit_resolvers: dict = {}
|
||||
|
||||
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
|
||||
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
|
||||
|
||||
|
||||
class StrictConstructor(SafeConstructor):
|
||||
yaml_constructors: dict = {}
|
||||
|
||||
|
||||
for _tag, _fn in (
|
||||
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
|
||||
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
|
||||
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
|
||||
(_STR, SafeConstructor.construct_yaml_str),
|
||||
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
|
||||
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
|
||||
):
|
||||
StrictConstructor.add_constructor(_tag, _fn)
|
||||
|
||||
|
||||
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
|
||||
def __init__(self, stream: str):
|
||||
Reader.__init__(self, stream)
|
||||
Scanner.__init__(self)
|
||||
Parser.__init__(self)
|
||||
Composer.__init__(self)
|
||||
StrictConstructor.__init__(self)
|
||||
StrictResolver.__init__(self)
|
||||
self._depth = 0
|
||||
self._nodes = 0
|
||||
self._path: list = []
|
||||
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
|
||||
|
||||
# -- document level ---------------------------------------------------------------------------------------------
|
||||
def get_single_node(self):
|
||||
self.get_event() # StreamStart
|
||||
if self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_empty")
|
||||
start = self.peek_event()
|
||||
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
|
||||
raise _Rejected("yaml_directive_forbidden", start.start_mark)
|
||||
node = self.compose_document()
|
||||
if not self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
|
||||
self.get_event()
|
||||
return node
|
||||
|
||||
# -- node level ---------------------------------------------------------------------------------------------------
|
||||
def compose_node(self, parent, index):
|
||||
event = self.peek_event()
|
||||
if isinstance(event, AliasEvent):
|
||||
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "anchor", None) is not None:
|
||||
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "tag", None) is not None:
|
||||
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
|
||||
self._nodes += 1
|
||||
if self._nodes > MAX_NODES:
|
||||
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
|
||||
|
||||
if isinstance(index, ScalarNode):
|
||||
segment = index.value
|
||||
elif isinstance(index, int) and not isinstance(index, bool):
|
||||
segment = index
|
||||
else:
|
||||
segment = None # a mapping key
|
||||
if segment is not None:
|
||||
self._path.append(segment)
|
||||
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
|
||||
self._depth += 1
|
||||
try:
|
||||
if self._depth > MAX_DEPTH:
|
||||
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
|
||||
return super().compose_node(parent, index)
|
||||
finally:
|
||||
self._depth -= 1
|
||||
if segment is not None:
|
||||
self._path.pop()
|
||||
|
||||
def compose_scalar_node(self, anchor):
|
||||
event = self.peek_event()
|
||||
if len(event.value) > MAX_SCALAR:
|
||||
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
|
||||
return super().compose_scalar_node(anchor)
|
||||
|
||||
def compose_mapping_node(self, anchor):
|
||||
node = super().compose_mapping_node(anchor)
|
||||
seen: set[str] = set()
|
||||
for key, _value in node.value:
|
||||
if not isinstance(key, ScalarNode) or key.tag != _STR:
|
||||
raise _Rejected("non_string_key", key.start_mark, self._path)
|
||||
if key.value == "<<":
|
||||
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
|
||||
if key.value in seen:
|
||||
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
|
||||
seen.add(key.value)
|
||||
return node
|
||||
|
||||
|
||||
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
|
||||
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
if len(raw) > MAX_BYTES:
|
||||
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
|
||||
try:
|
||||
text = raw.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
|
||||
if text.startswith(""):
|
||||
text = text[1:]
|
||||
loader = None
|
||||
try:
|
||||
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
|
||||
node = loader.get_single_node()
|
||||
data = loader.construct_document(node)
|
||||
except _Rejected as exc:
|
||||
line = exc.mark.line + 1 if exc.mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
|
||||
except yaml.MarkedYAMLError as exc:
|
||||
mark = exc.problem_mark or exc.context_mark
|
||||
line = mark.line + 1 if mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
|
||||
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
|
||||
finally:
|
||||
if loader is not None:
|
||||
loader.dispose()
|
||||
return data, loader.lines
|
||||
|
||||
|
||||
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
|
||||
for end in range(len(loc), 0, -1):
|
||||
if loc[:end] in lines:
|
||||
return lines[loc[:end]]
|
||||
return None
|
||||
|
||||
|
||||
_PYDANTIC_CODES = {
|
||||
"extra_forbidden": "unknown_field",
|
||||
"missing": "missing_field",
|
||||
"literal_error": "bad_value",
|
||||
"string_too_short": "too_short",
|
||||
"string_too_long": "too_long",
|
||||
"string_pattern_mismatch": "bad_format",
|
||||
}
|
||||
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
|
||||
|
||||
|
||||
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
|
||||
out = []
|
||||
for error in exc.errors(include_url=False):
|
||||
loc = tuple(error["loc"])
|
||||
code = _PYDANTIC_CODES.get(error["type"])
|
||||
if code is None and error["type"] == "value_error":
|
||||
message = str(error.get("ctx", {}).get("error", ""))
|
||||
code = message if message in _OWN_CODES else "bad_value"
|
||||
if code is None:
|
||||
code = "bad_type"
|
||||
params = {}
|
||||
if error["type"] == "extra_forbidden":
|
||||
params["field"] = display(loc[-1]) if loc else ""
|
||||
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
|
||||
params["value"] = display(error["input"])
|
||||
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
|
||||
return out
|
||||
|
||||
|
||||
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
|
||||
errors = []
|
||||
for kind in schema.KINDS:
|
||||
section = data.get(kind)
|
||||
if kind == "variables":
|
||||
if isinstance(section, dict):
|
||||
if len(section) > MAX_VARIABLE_HOSTS:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
|
||||
for host, values in section.items():
|
||||
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
|
||||
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
|
||||
max=MAX_VARIABLES_PER_HOST))
|
||||
continue
|
||||
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
|
||||
aliases = data.get("aliases")
|
||||
for i, alias in enumerate(aliases if isinstance(aliases, list) else []): # any other shape is for the schema check
|
||||
entries = alias.get("entries") if isinstance(alias, dict) else None
|
||||
if isinstance(entries, list):
|
||||
loc = ("aliases", i, "entries")
|
||||
if len(entries) > MAX_ALIAS_ENTRIES:
|
||||
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
|
||||
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
|
||||
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
|
||||
return errors
|
||||
|
||||
|
||||
class Parsed(NamedTuple):
|
||||
doc: schema.ConfigDoc
|
||||
sha256: str # of the raw bytes, shown in the plan and stored in the audit
|
||||
lines: dict[tuple, int] # path -> line, for messages of the planner
|
||||
|
||||
|
||||
def parse(raw: bytes | str) -> Parsed:
|
||||
"""Load, limit and validate a file. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
sha256 = hashlib.sha256(raw).hexdigest()
|
||||
data, lines = load_yaml(raw)
|
||||
if not isinstance(data, dict):
|
||||
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
|
||||
version = data.get("opnexus_config")
|
||||
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
|
||||
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
|
||||
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
|
||||
"opnexus_config", _line_for(lines, ("opnexus_config",)),
|
||||
**({"value": display(version)} if version is not None else {}))])
|
||||
errors = _check_counts(data, lines)
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors)
|
||||
try:
|
||||
doc = schema.ConfigDoc.model_validate(data)
|
||||
except ValidationError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
|
||||
return Parsed(doc, sha256, lines)
|
||||
@@ -0,0 +1,933 @@
|
||||
{
|
||||
"$defs": {
|
||||
"AliasDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"pattern": "^[A-Za-z0-9_]+$",
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "host",
|
||||
"enum": [
|
||||
"host",
|
||||
"network",
|
||||
"port",
|
||||
"networkgroup",
|
||||
"mac",
|
||||
"asn"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 255,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"entries": {
|
||||
"items": {
|
||||
"maxLength": 8000,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Entries",
|
||||
"type": "array"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "AliasDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"DeviceGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "DeviceGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"parent": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Parent"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "HierarchyGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyRuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"group": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Group",
|
||||
"type": "string"
|
||||
},
|
||||
"rulebase": {
|
||||
"default": "pre",
|
||||
"enum": [
|
||||
"pre",
|
||||
"post"
|
||||
],
|
||||
"title": "Rulebase",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"group",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "HierarchyRuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OneToOneDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "binat",
|
||||
"enum": [
|
||||
"binat",
|
||||
"nat"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"external": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "External",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"enable",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"external"
|
||||
],
|
||||
"title": "OneToOneDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OutboundNatDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"target": {
|
||||
"default": "",
|
||||
"maxLength": 128,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"target_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Target Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "OutboundNatDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"PortForwardDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "tcp",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"target": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"local_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Local Port"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"purenat",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"target"
|
||||
],
|
||||
"title": "PortForwardDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"RuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/Targets"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "RuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"Targets": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"title": "Targets",
|
||||
"type": "object"
|
||||
},
|
||||
"TargetsWithHierarchy": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
},
|
||||
"hierarchy_group": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Hierarchy Group"
|
||||
}
|
||||
},
|
||||
"title": "TargetsWithHierarchy",
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
|
||||
"properties": {
|
||||
"opnexus_config": {
|
||||
"const": 1,
|
||||
"title": "Opnexus Config",
|
||||
"type": "integer"
|
||||
},
|
||||
"device_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/DeviceGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Device Groups"
|
||||
},
|
||||
"policy_hierarchy_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Groups"
|
||||
},
|
||||
"aliases": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/AliasDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Aliases"
|
||||
},
|
||||
"rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/RuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Rules"
|
||||
},
|
||||
"port_forwards": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/PortForwardDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Port Forwards"
|
||||
},
|
||||
"one_to_one": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OneToOneDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "One To One"
|
||||
},
|
||||
"outbound_nat": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OutboundNatDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Outbound Nat"
|
||||
},
|
||||
"policy_hierarchy_rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyRuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Rules"
|
||||
},
|
||||
"variables": {
|
||||
"anyOf": [
|
||||
{
|
||||
"additionalProperties": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"propertyNames": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Variables"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"opnexus_config"
|
||||
],
|
||||
"title": "OPNexus-Konfiguration (Schema v1)",
|
||||
"type": "object",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
}
|
||||
@@ -0,0 +1,570 @@
|
||||
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
|
||||
|
||||
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
|
||||
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
|
||||
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
|
||||
`prune` gleicht die Art ab.
|
||||
|
||||
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
|
||||
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
|
||||
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
|
||||
|
||||
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
|
||||
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
|
||||
from app.database import Base
|
||||
from app.services.deploy_flow import is_unsettled
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.schemas.alias import AliasCreate
|
||||
from app.schemas.one_to_one import OneToOneCreate
|
||||
from app.schemas.outbound_nat import OutboundNatCreate
|
||||
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||
from app.schemas.port_forward import PortForwardCreate
|
||||
from app.schemas.rule import RuleCreate
|
||||
from app.services.variables_service import names_in
|
||||
|
||||
DEPLOYED = ("success", "pending", "revert_failed")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
section: str
|
||||
model: type | None
|
||||
ident: str # identity field in the file ("key"/"name")
|
||||
api: type | None = None # API create schema, run on every file object (same validators as the UI)
|
||||
deployable: bool = False
|
||||
|
||||
|
||||
KIND_SPECS = {
|
||||
"device_groups": Kind("device_groups", DeviceGroup, "name"),
|
||||
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
|
||||
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
|
||||
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
|
||||
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
|
||||
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
|
||||
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
|
||||
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
|
||||
"variables": Kind("variables", None, "key"),
|
||||
}
|
||||
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
|
||||
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
|
||||
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
|
||||
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
|
||||
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
|
||||
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
|
||||
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
|
||||
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
kind: str
|
||||
key: str
|
||||
action: str # create | update | delete | unchanged
|
||||
before: dict | None = None
|
||||
after: dict | None = None
|
||||
changes: list = field(default_factory=list)
|
||||
reasons: list = field(default_factory=list)
|
||||
deployed_hosts: int = 0
|
||||
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
|
||||
db_id: uuid.UUID | None = None
|
||||
|
||||
def public(self) -> dict:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"key": self.key,
|
||||
"action": "blocked" if self.reasons else self.action,
|
||||
"planned_action": self.action,
|
||||
"changes": self.changes,
|
||||
"deployed_hosts": self.deployed_hosts,
|
||||
"reasons": self.reasons,
|
||||
"implicit": self.implicit,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Plan:
|
||||
sha256: str
|
||||
prune: bool
|
||||
touched: list[str]
|
||||
items: list[Item]
|
||||
warnings: list[dict]
|
||||
token: str = ""
|
||||
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
|
||||
expected: dict = field(default_factory=dict)
|
||||
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
|
||||
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
|
||||
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
|
||||
|
||||
@property
|
||||
def applicable(self) -> bool:
|
||||
return not any(i.reasons for i in self.items)
|
||||
|
||||
def summary(self) -> dict:
|
||||
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||
for item in self.items:
|
||||
out["blocked" if item.reasons else item.action] += 1
|
||||
return out
|
||||
|
||||
def public(self) -> dict:
|
||||
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
|
||||
return {
|
||||
"token": self.token,
|
||||
"sha256": self.sha256,
|
||||
"prune": self.prune,
|
||||
"kinds": self.touched,
|
||||
"summary": self.summary(),
|
||||
"items": [i.public() for i in changed],
|
||||
"warnings": self.warnings,
|
||||
"applicable": self.applicable,
|
||||
}
|
||||
|
||||
|
||||
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
|
||||
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
|
||||
|
||||
|
||||
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
|
||||
section = getattr(doc, kind)
|
||||
if kind == "variables":
|
||||
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
|
||||
for host, values in section.items() for name, value in values.items()]
|
||||
return [item.model_dump(mode="python") for item in section]
|
||||
|
||||
|
||||
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
|
||||
def sorted_unique(values: list, where: tuple) -> list:
|
||||
seen = set()
|
||||
for i, value in enumerate(values):
|
||||
if value in seen:
|
||||
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
|
||||
seen.add(value)
|
||||
return sorted(seen)
|
||||
|
||||
item = dict(item)
|
||||
if "targets" in item:
|
||||
targets = dict(item["targets"])
|
||||
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
|
||||
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
|
||||
item["targets"] = targets
|
||||
if kind in ("device_groups", "policy_hierarchy_groups"):
|
||||
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
|
||||
return item
|
||||
|
||||
|
||||
def _changes(before: dict, after: dict) -> list[dict]:
|
||||
out = []
|
||||
for name in after:
|
||||
if name == "targets":
|
||||
for sub in after["targets"]:
|
||||
if before["targets"].get(sub) != after["targets"][sub]:
|
||||
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
|
||||
elif before.get(name) != after[name]:
|
||||
out.append({"field": name, "before": before.get(name), "after": after[name]})
|
||||
return out
|
||||
|
||||
|
||||
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
|
||||
buckets: dict = {}
|
||||
for row in rows:
|
||||
buckets.setdefault(bucket_of(row), []).append(row)
|
||||
out = {}
|
||||
for members in buckets.values():
|
||||
for rank, row in enumerate(sorted(members, key=order_of)):
|
||||
out[key_of(row)] = rank
|
||||
return out
|
||||
|
||||
|
||||
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
|
||||
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
|
||||
found: dict = {}
|
||||
if not ids:
|
||||
return found
|
||||
for table in Base.metadata.sorted_tables:
|
||||
if table.name in skip:
|
||||
continue
|
||||
for fk in table.foreign_keys:
|
||||
if fk.column.table.name != target_table:
|
||||
continue
|
||||
column = fk.parent
|
||||
rows = (await db.execute(
|
||||
select(column, func.count()).where(column.in_(ids)).group_by(column)
|
||||
)).all()
|
||||
for ref_id, _count in rows:
|
||||
found.setdefault(ref_id, []).append(table.name)
|
||||
return found
|
||||
|
||||
|
||||
async def _load(db: AsyncSession) -> dict:
|
||||
loaded: dict = {}
|
||||
for kind in TARGET_KINDS:
|
||||
model = KIND_SPECS[kind].model
|
||||
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
|
||||
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
|
||||
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
|
||||
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
|
||||
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
return loaded
|
||||
|
||||
|
||||
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
|
||||
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
|
||||
doc, lines = parsed.doc, parsed.lines
|
||||
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
|
||||
errors: list[dict] = []
|
||||
warnings: list[dict] = []
|
||||
|
||||
db_canon = await canonical(db)
|
||||
loaded = await _load(db)
|
||||
hosts = loaded["hosts"]
|
||||
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
|
||||
|
||||
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
|
||||
file_objects: dict[str, dict[str, dict]] = {}
|
||||
file_locs: dict[tuple, tuple] = {}
|
||||
for kind in touched:
|
||||
spec = KIND_SPECS[kind]
|
||||
objects: dict[str, dict] = {}
|
||||
for index, raw in enumerate(_file_items(doc, kind)):
|
||||
if kind == "variables":
|
||||
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
|
||||
else:
|
||||
ident, loc = raw[spec.ident], (kind, index)
|
||||
if ident in objects:
|
||||
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
|
||||
continue
|
||||
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
|
||||
file_locs[(kind, ident)] = loc
|
||||
file_objects[kind] = objects
|
||||
|
||||
def db_items(kind: str) -> dict[str, dict]:
|
||||
if kind == "variables":
|
||||
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in db_canon["variables"].items() for n, v in values.items()}
|
||||
ident = KIND_SPECS[kind].ident
|
||||
return {item[ident]: item for item in db_canon[kind]}
|
||||
|
||||
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
|
||||
|
||||
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
|
||||
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
|
||||
post: dict[str, dict[str, dict]] = {}
|
||||
for kind in schema.KINDS:
|
||||
if kind in file_objects:
|
||||
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
|
||||
if prune:
|
||||
deleted[kind] = set(remaining)
|
||||
remaining = {}
|
||||
post[kind] = {**remaining, **file_objects[kind]}
|
||||
else:
|
||||
post[kind] = dict(db_objects[kind])
|
||||
|
||||
host_names = set(hosts)
|
||||
group_names = set(post["device_groups"])
|
||||
hierarchy_names = set(post["policy_hierarchy_groups"])
|
||||
|
||||
# -- references in the file ------------------------------------------------------------------------------------------
|
||||
for kind in touched:
|
||||
for ident, item in file_objects[kind].items():
|
||||
loc = file_locs[(kind, ident)]
|
||||
if kind == "variables":
|
||||
if item["host"] not in host_names:
|
||||
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
|
||||
continue
|
||||
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
|
||||
for name in member_hosts:
|
||||
if name not in host_names:
|
||||
_err(errors, "unknown_host", loc, lines, host=display(name))
|
||||
for name in item.get("targets", {}).get("groups", []):
|
||||
if name not in group_names:
|
||||
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
|
||||
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
|
||||
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
|
||||
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
|
||||
if ref is not None and ref not in hierarchy_names:
|
||||
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
|
||||
if kind == "policy_hierarchy_rules":
|
||||
for name, value in item.items():
|
||||
if names_in(value):
|
||||
_err(errors, "placeholder_not_supported", loc + (name,), lines)
|
||||
spec = KIND_SPECS[kind]
|
||||
if spec.api is not None:
|
||||
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
|
||||
for name in member_hosts:
|
||||
host = hosts.get(name)
|
||||
if host is not None and not host.enabled:
|
||||
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
|
||||
|
||||
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
|
||||
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
|
||||
if "policy_hierarchy_groups" in file_objects:
|
||||
file_groups = file_objects["policy_hierarchy_groups"]
|
||||
for host_name, group in list(host_hierarchy.items()):
|
||||
if prune or group in file_groups:
|
||||
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
|
||||
host_hierarchy[host_name] = None
|
||||
listed: dict[str, str] = {}
|
||||
for name, group in file_groups.items():
|
||||
for host_name in group["hosts"]:
|
||||
if host_name in listed:
|
||||
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
|
||||
lines, host=display(host_name), first=display(listed[host_name]))
|
||||
listed[host_name] = name
|
||||
if host_name in host_hierarchy:
|
||||
host_hierarchy[host_name] = name
|
||||
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
|
||||
for start in parents:
|
||||
seen, current = set(), start
|
||||
while current is not None and current not in seen:
|
||||
seen.add(current)
|
||||
current = parents.get(current)
|
||||
if current is not None and current == start:
|
||||
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
|
||||
lines, group=display(start))
|
||||
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
|
||||
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
|
||||
|
||||
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
|
||||
per_host: dict[str, int] = {}
|
||||
for item in post["variables"].values():
|
||||
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
|
||||
for host_name, count in sorted(per_host.items()):
|
||||
if count > MAX_VARIABLES_PER_HOST:
|
||||
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
|
||||
max=MAX_VARIABLES_PER_HOST, count=count)
|
||||
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors[:100])
|
||||
|
||||
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
|
||||
db_rule_rows = loaded["policy_hierarchy_rules"]
|
||||
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
|
||||
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
|
||||
positions = dict(db_positions)
|
||||
if "policy_hierarchy_rules" in file_objects:
|
||||
buckets: dict[tuple, list[str]] = {}
|
||||
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
|
||||
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for key, item in post["policy_hierarchy_rules"].items():
|
||||
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
|
||||
buckets.setdefault((item["group"], item["rulebase"]), [])
|
||||
positions = {}
|
||||
extras: dict[tuple, list[str]] = {}
|
||||
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
|
||||
key=lambda k: db_positions.get(k, 0)):
|
||||
item = post["policy_hierarchy_rules"][key]
|
||||
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for bucket in set(buckets) | set(extras):
|
||||
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
|
||||
positions[key] = rank
|
||||
|
||||
# -- items -----------------------------------------------------------------------------------------------------------
|
||||
items: list[Item] = []
|
||||
for kind in touched:
|
||||
rows = loaded.get(kind, {})
|
||||
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
|
||||
row = rows.get(ident) if kind != "variables" else None
|
||||
before = db_objects[kind].get(ident)
|
||||
after = post[kind].get(ident) if ident not in deleted[kind] else None
|
||||
if before is None:
|
||||
item = Item(kind, ident, "create", None, after)
|
||||
elif after is None:
|
||||
item = Item(kind, ident, "delete", before, None)
|
||||
else:
|
||||
changes = _changes(before, after)
|
||||
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
|
||||
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
|
||||
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
|
||||
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
|
||||
if row is not None:
|
||||
item.db_id = row.id
|
||||
deployments = getattr(row, "deployments", None) or []
|
||||
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
|
||||
if item.action in ("update", "delete") and any(is_unsettled(d) for d in deployments):
|
||||
item.reasons.append("busy_deployment")
|
||||
if item.action == "delete" and deployments:
|
||||
item.reasons.append("prune_deployed")
|
||||
items.append(item)
|
||||
|
||||
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
|
||||
|
||||
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
|
||||
if redeploy:
|
||||
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
|
||||
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
|
||||
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
|
||||
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
|
||||
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
|
||||
warnings += _alias_reference_warnings(post, file_objects, file_locs)
|
||||
warnings += _variable_warnings(post, deleted)
|
||||
|
||||
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
|
||||
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
|
||||
file_objects=file_objects)
|
||||
plan.expected = {kind: post[kind] for kind in touched}
|
||||
plan.token = _token(plan, db_canon, db_positions, loaded)
|
||||
return plan
|
||||
|
||||
|
||||
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
|
||||
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
|
||||
out = set()
|
||||
if kind == "policy_hierarchy_rules":
|
||||
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
|
||||
if kind == "policy_hierarchy_groups":
|
||||
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
|
||||
return out
|
||||
|
||||
|
||||
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
|
||||
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
|
||||
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
|
||||
if kind == "aliases":
|
||||
values["content"] = "\n".join(item["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
values["hierarchy_group_id"] = uuid.UUID(int=0)
|
||||
try:
|
||||
cleaned = KIND_SPECS[kind].api.model_validate(values)
|
||||
except ValidationError as exc:
|
||||
for error in exc.errors(include_url=False):
|
||||
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
|
||||
return item
|
||||
out = dict(item)
|
||||
for name in item:
|
||||
if name in values and name != "content" and hasattr(cleaned, name):
|
||||
out[name] = getattr(cleaned, name)
|
||||
if kind == "aliases":
|
||||
out["entries"] = [e for e in cleaned.content.split("\n") if e]
|
||||
return out
|
||||
|
||||
|
||||
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
|
||||
by_key = {(i.kind, i.key): i for i in items}
|
||||
|
||||
# device groups: post-state objects of the import, plus every other table that points at the group
|
||||
if deleted["device_groups"]:
|
||||
users: dict[str, list[str]] = {}
|
||||
for kind in TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
for group in item["targets"]["groups"]:
|
||||
users.setdefault(group, []).append(f"{kind}:{ident}")
|
||||
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
|
||||
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
|
||||
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["device_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("device_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["policy_hierarchy_groups"]:
|
||||
users = {}
|
||||
for kind in HIERARCHY_TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
if item["targets"]["hierarchy_group"]:
|
||||
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
|
||||
for ident, item in post["policy_hierarchy_rules"].items():
|
||||
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
|
||||
for name, item in post["policy_hierarchy_groups"].items():
|
||||
if item["parent"]:
|
||||
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
|
||||
for host_name, group in host_hierarchy.items():
|
||||
if group:
|
||||
users.setdefault(group, []).append(f"host:{host_name}")
|
||||
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
|
||||
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
|
||||
"policy_hierarchy_rules", "opnsense_hosts"}
|
||||
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["policy_hierarchy_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["aliases"]:
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS:
|
||||
for item in post[kind].values():
|
||||
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
|
||||
for item in post["aliases"].values():
|
||||
used.update(item["entries"])
|
||||
for name in deleted["aliases"]:
|
||||
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
|
||||
by_key[("aliases", name)].reasons.append("prune_referenced")
|
||||
|
||||
|
||||
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
|
||||
aliases = set(post["aliases"])
|
||||
out = []
|
||||
for kind in RULE_KINDS:
|
||||
for ident, item in file_objects.get(kind, {}).items():
|
||||
for name in ALIAS_REF_FIELDS:
|
||||
value = item.get(name)
|
||||
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
|
||||
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
|
||||
"params": {"value": display(value)}})
|
||||
return out
|
||||
|
||||
|
||||
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
|
||||
if not deleted["variables"]:
|
||||
return []
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS + ("aliases",):
|
||||
for item in post[kind].values():
|
||||
for value in item.values():
|
||||
if isinstance(value, str):
|
||||
used |= names_in(value)
|
||||
elif isinstance(value, list):
|
||||
for entry in value:
|
||||
used |= names_in(entry)
|
||||
out = []
|
||||
for key in sorted(deleted["variables"]):
|
||||
host, name = key.split("/", 1)
|
||||
if name in used:
|
||||
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
|
||||
return out
|
||||
|
||||
|
||||
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
|
||||
deployments = {
|
||||
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
|
||||
for ident, row in loaded[kind].items()}
|
||||
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
|
||||
}
|
||||
material = {
|
||||
"file": plan.file_objects,
|
||||
"prune": plan.prune,
|
||||
"db": db_canon,
|
||||
"positions": db_positions,
|
||||
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
|
||||
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
|
||||
"deployments": deployments,
|
||||
"items": [i.public() for i in plan.items],
|
||||
"warnings": plan.warnings,
|
||||
}
|
||||
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
|
||||
@@ -0,0 +1,242 @@
|
||||
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
|
||||
|
||||
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
|
||||
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
|
||||
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
|
||||
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
|
||||
"""
|
||||
import re
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
|
||||
|
||||
from app.models.config_key import CONFIG_KEY_RE
|
||||
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
MAX_SCALAR = 8000
|
||||
|
||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
|
||||
|
||||
|
||||
def _clean(value: str) -> str:
|
||||
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
if value != value.strip():
|
||||
raise ValueError("surrounding_whitespace")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_value(value: str) -> str:
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_text(value: str) -> str:
|
||||
if _CONTROL_TEXT.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _port(value):
|
||||
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
|
||||
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
|
||||
|
||||
|
||||
Text = Annotated[str, AfterValidator(_clean)]
|
||||
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
|
||||
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
|
||||
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
|
||||
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
|
||||
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
|
||||
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
|
||||
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
|
||||
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
|
||||
Port = Annotated[PortText | None, BeforeValidator(_port)]
|
||||
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
|
||||
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
|
||||
|
||||
IpProtocol = Literal["inet", "inet6"]
|
||||
|
||||
|
||||
class _Doc(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
class Targets(_Doc):
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
groups: list[GroupName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class TargetsWithHierarchy(Targets):
|
||||
hierarchy_group: GroupName | None = None
|
||||
|
||||
|
||||
class DeviceGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class HierarchyGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
parent: GroupName | None = None
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AliasDoc(_Doc):
|
||||
name: AliasName
|
||||
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
|
||||
description: Description = ""
|
||||
enabled: bool = True
|
||||
entries: list[AliasEntry] = Field(default_factory=list)
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class RuleDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
log: bool = False
|
||||
targets: Targets = Field(default_factory=Targets)
|
||||
|
||||
|
||||
class PortForwardDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "tcp"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
target: Net
|
||||
local_port: Port = None
|
||||
natreflection: Literal["", "purenat", "disable"] = ""
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OneToOneDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
type: Literal["binat", "nat"] = "binat"
|
||||
external: Net
|
||||
source_net: Net = "any"
|
||||
destination_net: Net = "any"
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
natreflection: Literal["", "enable", "disable"] = ""
|
||||
log: bool = False
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OutboundNatDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
|
||||
target_port: Port = None
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class HierarchyRuleDoc(_Doc):
|
||||
key: Key
|
||||
group: GroupName
|
||||
rulebase: Literal["pre", "post"] = "pre"
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
def _variable_name(value: str) -> str:
|
||||
if not VARIABLE_NAME_RE.match(value):
|
||||
raise ValueError("bad_variable_name")
|
||||
return value
|
||||
|
||||
|
||||
def _variable_value(value: str) -> str:
|
||||
if not VARIABLE_VALUE_RE.match(value):
|
||||
raise ValueError("bad_variable_value")
|
||||
return value
|
||||
|
||||
|
||||
VariableName = Annotated[str, AfterValidator(_variable_name)]
|
||||
VariableValue = Annotated[str, AfterValidator(_variable_value)]
|
||||
|
||||
|
||||
class ConfigDoc(_Doc):
|
||||
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
|
||||
|
||||
opnexus_config: Literal[1]
|
||||
device_groups: list[DeviceGroupDoc] | None = None
|
||||
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
|
||||
aliases: list[AliasDoc] | None = None
|
||||
rules: list[RuleDoc] | None = None
|
||||
port_forwards: list[PortForwardDoc] | None = None
|
||||
one_to_one: list[OneToOneDoc] | None = None
|
||||
outbound_nat: list[OutboundNatDoc] | None = None
|
||||
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
|
||||
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
|
||||
|
||||
|
||||
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
|
||||
KINDS = (
|
||||
"device_groups",
|
||||
"policy_hierarchy_groups",
|
||||
"aliases",
|
||||
"rules",
|
||||
"port_forwards",
|
||||
"one_to_one",
|
||||
"outbound_nat",
|
||||
"policy_hierarchy_rules",
|
||||
"variables",
|
||||
)
|
||||
|
||||
|
||||
SCHEMA_FILE = "opnexus-config-v1.schema.json"
|
||||
|
||||
|
||||
def json_schema_text() -> str:
|
||||
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
|
||||
import json
|
||||
|
||||
schema = ConfigDoc.model_json_schema()
|
||||
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
|
||||
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
|
||||
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
|
||||
@@ -62,6 +62,7 @@ class ErrorCode(str, Enum):
|
||||
GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_ALIASEN_VERWENDET"
|
||||
GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET"
|
||||
GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_REGELN_VERWENDET"
|
||||
GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET = "GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET"
|
||||
HOSTNAMEN_EXAKT_EINGEBEN = "HOSTNAMEN_EXAKT_EINGEBEN"
|
||||
HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN = "HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN"
|
||||
HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0 = "HOST_HAT_EINE_OFFENE_WARTUNGSAKTION_ZUERST_PRUEFEN_UND_8CC5FBE0"
|
||||
@@ -72,6 +73,8 @@ class ErrorCode(str, Enum):
|
||||
HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN = "HOST_HAT_NOCH_VPN_ODER_ZERTIFIKATS_OBJEKTE_DIESE_ZUERST_ENTFERNEN"
|
||||
HOST_IST_DEAKTIVIERT = "HOST_IST_DEAKTIVIERT"
|
||||
HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS = "HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS"
|
||||
HOST_URL_EXISTS = "HOST_URL_EXISTS"
|
||||
HOST_CONNECTION_FAILED = "HOST_CONNECTION_FAILED"
|
||||
HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B = "HOST_MUSS_ERREICHBAR_SEIN_UND_FIRMWARE_AUFTRAG_BEENDET_BCE1390B"
|
||||
HOST_NICHT_GEFUNDEN = "HOST_NICHT_GEFUNDEN"
|
||||
HOST_REQUEST_FAILED = "HOST_REQUEST_FAILED"
|
||||
@@ -247,6 +250,16 @@ class ErrorCode(str, Enum):
|
||||
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
|
||||
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
|
||||
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
|
||||
CONFIG_INVALID = "CONFIG_INVALID"
|
||||
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
|
||||
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
|
||||
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
|
||||
CONFIG_BLOCKED = "CONFIG_BLOCKED"
|
||||
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
|
||||
CONFIG_BUSY = "CONFIG_BUSY"
|
||||
ZIEL_HOST_NACH_AUSROLLEN_FEST = "ZIEL_HOST_NACH_AUSROLLEN_FEST"
|
||||
ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET = "ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET"
|
||||
VPN_DEPLOY_LAEUFT_BEREITS = "VPN_DEPLOY_LAEUFT_BEREITS"
|
||||
|
||||
|
||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||
|
||||
@@ -54,6 +54,7 @@ from app.routers import maintenance_windows as maintenance_windows_router
|
||||
from app.routers import rollouts as rollouts_router
|
||||
from app.routers import rule_analysis as rule_analysis_router
|
||||
from app.routers import rule_cleanup as rule_cleanup_router
|
||||
from app.routers import config_code as config_code_router
|
||||
from app.routers import gui_access as gui_access_router
|
||||
from app.services import scheduled_maintenance_service
|
||||
from app.services import rollout_engine
|
||||
@@ -571,5 +572,6 @@ app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||
app.include_router(rollouts_router.router, prefix="/api")
|
||||
app.include_router(rule_analysis_router.router, prefix="/api")
|
||||
app.include_router(rule_cleanup_router.router, prefix="/api")
|
||||
app.include_router(config_code_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.internal_router)
|
||||
@@ -108,3 +108,8 @@ __all__ += ["LiveRuleState", "RuleChangeEvent"]
|
||||
|
||||
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
|
||||
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
|
||||
|
||||
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
|
||||
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
|
||||
from app.models.host_variable import HostVariable
|
||||
__all__ += ["HostVariable"]
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
|
||||
|
||||
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
|
||||
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
|
||||
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
|
||||
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import String, event
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
|
||||
|
||||
|
||||
def config_key_column() -> Mapped[str]:
|
||||
return mapped_column(String(64), unique=True)
|
||||
|
||||
|
||||
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
|
||||
return f"{prefix}-{object_id.hex[:12]}"
|
||||
|
||||
|
||||
def register_config_key(model: type, prefix: str) -> None:
|
||||
"""Fills an empty `config_key` just before INSERT (and the id, which is
|
||||
otherwise only assigned while the INSERT is built)."""
|
||||
|
||||
@event.listens_for(model, "before_insert")
|
||||
def _fill(_mapper, _connection, target) -> None:
|
||||
if target.id is None:
|
||||
target.id = uuid.uuid4()
|
||||
if not target.config_key:
|
||||
target.config_key = default_config_key(prefix, target.id)
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.one_to_one_target import one_to_one_targets
|
||||
from app.models.policy import one_to_one_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
|
||||
__tablename__ = "one_to_one_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
|
||||
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
|
||||
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OneToOneRule, "nat1")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.outbound_nat_target import outbound_nat_targets
|
||||
from app.models.policy import outbound_nat_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
|
||||
__tablename__ = "outbound_nat_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
|
||||
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
|
||||
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OutboundNatRule, "out")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
|
||||
|
||||
class PolicyHierarchyRule(Base):
|
||||
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
|
||||
__tablename__ = "policy_hierarchy_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
|
||||
)
|
||||
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
|
||||
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
|
||||
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PolicyHierarchyRule, "ph")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.port_forward_target import port_forward_targets
|
||||
from app.models.policy import port_forward_groups
|
||||
|
||||
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
|
||||
__tablename__ = "port_forward_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
|
||||
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
|
||||
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
|
||||
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PortForwardRule, "pf")
|
||||
@@ -42,6 +42,8 @@ class RolloutPlan(Base):
|
||||
# is activated AND unchanged since (activated_fingerprint).
|
||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
|
||||
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
|
||||
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.rule_target import rule_targets
|
||||
from app.models.policy import rule_groups
|
||||
|
||||
@@ -18,6 +19,7 @@ class FirewallRule(Base):
|
||||
__tablename__ = "firewall_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
direction: Mapped[str] = mapped_column(String(8))
|
||||
@@ -44,3 +46,6 @@ class FirewallRule(Base):
|
||||
deployments: Mapped[list["RuleDeployment"]] = relationship(
|
||||
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(FirewallRule, "r")
|
||||
@@ -139,7 +139,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
||||
|
||||
verified_hash = user.password_hash
|
||||
if user.totp_enabled:
|
||||
second_factor_checked = bool(user.totp_enabled) # the state this login is decided on; create_session re-checks it
|
||||
if second_factor_checked:
|
||||
# Only revealed after the password was correct, so it cannot be used
|
||||
# to find out which usernames exist or have 2FA.
|
||||
if not payload.totp_code:
|
||||
@@ -153,7 +154,8 @@ async def login(payload: LoginRequest, request: Request, response: Response, db:
|
||||
auth_service.throttle.reset(ip, payload.username)
|
||||
failed_since_last_login = auth_service.throttle.pop_failures_since_success(payload.username)
|
||||
token, expires_at = await auth_service.create_session(
|
||||
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash
|
||||
db, user, user_agent=request.headers.get("User-Agent"), ip_address=ip, verified_hash=verified_hash,
|
||||
verified_totp=second_factor_checked,
|
||||
)
|
||||
_set_session_cookie(response, token)
|
||||
logger.info("Successful login for '%s' from %s", user.username, ip)
|
||||
@@ -252,6 +254,8 @@ async def totp_enable(
|
||||
_account_failure(user, "2fa-enable")
|
||||
raise api_error(status.HTTP_401_UNAUTHORIZED, ErrorCode.TOTP_INVALID)
|
||||
_account_success(user, "2fa-enable")
|
||||
# Same row lock as create_session: a sign-in that decided "no second factor needed" is ordered before or after this.
|
||||
await db.execute(select(AdminUser.id).where(AdminUser.id == user.id).with_for_update())
|
||||
plain, hashes = totp_service.new_recovery_codes()
|
||||
totp_service.promote_pending(user.id)
|
||||
user.totp_enabled = True
|
||||
|
||||
@@ -37,6 +37,13 @@ acme_accounts_router = APIRouter(prefix="/acme/accounts", tags=["acme"])
|
||||
acme_certificates_router = APIRouter(prefix="/acme/certificates", tags=["acme"])
|
||||
|
||||
|
||||
def _ensure_host_unchanged_once_deployed(obj, new_host_id: uuid.UUID) -> None:
|
||||
"""The firewall-local UUID, the stored payload and the success status all belong to the host the object was pushed
|
||||
to. Re-pointing it would report success for a host that never received it and orphan the object on the old one."""
|
||||
if new_host_id != obj.host_id and obj.opnsense_uuid:
|
||||
raise api_error(409, ErrorCode.ZIEL_HOST_NACH_AUSROLLEN_FEST)
|
||||
|
||||
|
||||
async def _get_host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
@@ -82,6 +89,7 @@ async def edit_managed_certificate(cert_id: uuid.UUID, payload: ManagedCertifica
|
||||
if cert.updated_at != payload.expected_updated_at:
|
||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||
await _get_host(db, payload.host_id)
|
||||
_ensure_host_unchanged_once_deployed(cert, payload.host_id)
|
||||
for field, value in payload.model_dump(exclude={"expected_updated_at", "private_key"}).items():
|
||||
setattr(cert, field, value)
|
||||
if payload.private_key:
|
||||
@@ -146,6 +154,11 @@ async def edit_acme_account(account_id: uuid.UUID, payload: AcmeAccountEdit, db:
|
||||
if account.updated_at != payload.expected_updated_at:
|
||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||
await _get_host(db, payload.host_id)
|
||||
_ensure_host_unchanged_once_deployed(account, payload.host_id)
|
||||
if payload.host_id != account.host_id and await db.scalar(
|
||||
select(AcmeCertificateRequest.id).where(AcmeCertificateRequest.account_id == account.id).limit(1)
|
||||
):
|
||||
raise api_error(409, ErrorCode.ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET)
|
||||
for field, value in payload.model_dump(exclude={"expected_updated_at", "eab_hmac"}).items():
|
||||
setattr(account, field, value)
|
||||
if payload.eab_hmac:
|
||||
@@ -236,6 +249,7 @@ async def edit_acme_certificate_request(request_id: uuid.UUID, payload: AcmeCert
|
||||
if req.updated_at != payload.expected_updated_at:
|
||||
raise api_error(409, ErrorCode.EINTRAG_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_B5A5CE87)
|
||||
await _get_host(db, payload.host_id)
|
||||
_ensure_host_unchanged_once_deployed(req, payload.host_id)
|
||||
await _get_account(db, payload.account_id, payload.host_id)
|
||||
for field, value in payload.model_dump(exclude={"expected_updated_at"}).items():
|
||||
setattr(req, field, value)
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
|
||||
|
||||
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
|
||||
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
|
||||
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
|
||||
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, parse
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
router = APIRouter(prefix="/config", tags=["config"])
|
||||
|
||||
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
|
||||
# and read before parsing so a huge body never reaches the JSON parser.
|
||||
MAX_REQUEST_BYTES = 4 * MAX_BYTES
|
||||
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
|
||||
# open to read-only accounts and read tokens.
|
||||
_busy = asyncio.Semaphore(1)
|
||||
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
|
||||
MAX_WAITING = 4
|
||||
_waiting = 0
|
||||
|
||||
|
||||
class _Slot:
|
||||
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
|
||||
|
||||
async def __aenter__(self):
|
||||
global _waiting
|
||||
if _waiting >= MAX_WAITING:
|
||||
raise api_error(429, ErrorCode.CONFIG_BUSY)
|
||||
_waiting += 1
|
||||
return self
|
||||
|
||||
async def run(self, coro_fn):
|
||||
async with _busy:
|
||||
return await coro_fn()
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
global _waiting
|
||||
_waiting -= 1
|
||||
|
||||
|
||||
def _encode(text: str) -> bytes:
|
||||
try:
|
||||
return text.encode("utf-8")
|
||||
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
class ImportRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
yaml: str
|
||||
prune: bool = False
|
||||
|
||||
|
||||
class ApplyRequest(ImportRequest):
|
||||
plan_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
def _config_error(exc: ConfigError):
|
||||
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
|
||||
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
|
||||
|
||||
|
||||
async def _read_body(request: Request, model: type[BaseModel]):
|
||||
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks, size = [], 0
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks.append(chunk)
|
||||
try:
|
||||
return model.model_validate(json.loads(b"".join(chunks)))
|
||||
except (ValueError, ValidationError):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
@router.get("/export")
|
||||
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
|
||||
wanted = None
|
||||
if kinds:
|
||||
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
|
||||
text = await export_yaml(db, wanted)
|
||||
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
|
||||
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
|
||||
|
||||
|
||||
@router.post("/import/plan")
|
||||
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ImportRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
return await planner.build_plan(db, parsed, body.prune)
|
||||
except ConfigError as exc:
|
||||
raise _config_error(exc) from None
|
||||
finally:
|
||||
await db.rollback() # never leave anything in the session: plan is read-only
|
||||
|
||||
result = await slot.run(work)
|
||||
return result.public()
|
||||
|
||||
|
||||
@router.post("/import/apply")
|
||||
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ApplyRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
|
||||
except ConfigError as exc:
|
||||
await db.rollback()
|
||||
raise _config_error(exc) from None
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
return parsed, result
|
||||
|
||||
parsed, result = await slot.run(work)
|
||||
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
|
||||
@@ -8,7 +8,7 @@ old and the new values."""
|
||||
import ipaddress
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -16,7 +16,7 @@ from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.governance import GovernanceSettings
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.routers.auth import client_ip, current_user
|
||||
from app.services import audit_service
|
||||
|
||||
router = APIRouter(prefix="/settings/governance", tags=["governance"])
|
||||
@@ -37,6 +37,9 @@ class GovernanceRead(BaseModel):
|
||||
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
|
||||
cleanup_management_ports: list[int] = Field(default_factory=list)
|
||||
cleanup_management_ports_configured: bool = False
|
||||
# The address this request came from, as a suggestion for the management sources (B12; admins only). Only a
|
||||
# suggestion: the firewalls may see the workstation under another address (NAT, VPN).
|
||||
your_address: str | None = None
|
||||
|
||||
|
||||
class GovernanceUpdate(BaseModel):
|
||||
@@ -105,14 +108,23 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
return row
|
||||
|
||||
|
||||
def _suggestion(request: Request, user: AdminUser) -> str | None:
|
||||
if user.role != "admin":
|
||||
return None
|
||||
try:
|
||||
return str(ipaddress.ip_address(client_ip(request)))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
@router.get("", response_model=GovernanceRead)
|
||||
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
async def read(request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
|
||||
return _read(await get_settings(db), user)
|
||||
return _read(await get_settings(db), user).model_copy(update={"your_address": _suggestion(request, user)})
|
||||
|
||||
|
||||
@router.put("", response_model=GovernanceRead)
|
||||
async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
async def update(payload: GovernanceUpdate, request: Request, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
row = await get_settings(db)
|
||||
@@ -137,4 +149,4 @@ async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db),
|
||||
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
|
||||
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
|
||||
await db.commit()
|
||||
return _read(row, user)
|
||||
return _read(row, user).model_copy(update={"your_address": _suggestion(request, user)})
|
||||
@@ -99,6 +99,12 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
|
||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
|
||||
# VPN objects target groups too; their link rows are ON DELETE CASCADE, so without this check the group would
|
||||
# vanish and the VPN would silently lose its targets (B15, found while designing the config import).
|
||||
from app.models.policy import ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups
|
||||
for assignment in (ipsec_tunnel_groups, openvpn_instance_groups, wireguard_tunnel_groups):
|
||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET)
|
||||
from app.models.rollout import rollout_wave_groups
|
||||
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
|
||||
|
||||
@@ -5,7 +5,7 @@ from app.models.port_forward_deployment import PortForwardDeployment
|
||||
from app.models.one_to_one_deployment import OneToOneDeployment
|
||||
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -107,11 +107,37 @@ async def tls_probe(payload: TlsProbeRequest):
|
||||
raise api_error(502, ErrorCode.TLS_PROBE_FAILED)
|
||||
|
||||
|
||||
def _same_url(a: str, b: str) -> bool:
|
||||
return a.strip().rstrip("/").lower() == b.strip().rstrip("/").lower()
|
||||
|
||||
|
||||
async def _ensure_url_free(db: AsyncSession, base_url: str, exclude_id: uuid.UUID | None = None) -> None:
|
||||
"""One firewall, one entry: the same address twice would double every rollout, alert and backup (T4)."""
|
||||
for other in (await db.scalars(select(OpnsenseHost))).all():
|
||||
if other.id != exclude_id and _same_url(other.base_url, base_url):
|
||||
raise api_error(409, ErrorCode.HOST_URL_EXISTS, host=other.name)
|
||||
|
||||
|
||||
def _connection_failure_reason(exc: Exception) -> str:
|
||||
"""A short machine code for the UI (hosts.connection.<code>): why the first call to the firewall failed."""
|
||||
text = str(exc)
|
||||
if isinstance(exc, opnsense_client.OpnsenseForbiddenError):
|
||||
return "forbidden"
|
||||
if text.startswith("401 ") or text.startswith("403 "):
|
||||
return "credentials" if text.startswith("401 ") else "forbidden"
|
||||
if "TLS-Zertifikat" in text or "CERTIFICATE_VERIFY_FAILED" in text:
|
||||
return "tls"
|
||||
if "Verbindung zu" in text:
|
||||
return "unreachable"
|
||||
return "other"
|
||||
|
||||
|
||||
@router.post("", response_model=HostRead, status_code=201)
|
||||
async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
||||
existing = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == payload.name))
|
||||
if existing:
|
||||
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
||||
await _ensure_url_free(db, payload.base_url)
|
||||
if payload.policy_hierarchy_group_id and not await db.get(PolicyHierarchyGroup, payload.policy_hierarchy_group_id):
|
||||
raise api_error(400, ErrorCode.POLICY_HIERARCHIE_GRUPPE_NICHT_GEFUNDEN)
|
||||
|
||||
@@ -130,6 +156,14 @@ async def create_host(payload: HostCreate, db: AsyncSession = Depends(get_db)):
|
||||
db.add(host)
|
||||
await db.flush()
|
||||
opnsense_secret_store.save_secret(host.id, payload.api_secret)
|
||||
if payload.verify_connection:
|
||||
# The call that decides "online" in the host list: if it works here, the host shows up online right away.
|
||||
try:
|
||||
await opnsense_client.get_vip_status(host)
|
||||
except opnsense_client.OpnsenseAPIError as exc:
|
||||
await db.rollback()
|
||||
opnsense_secret_store.delete_secret(host.id)
|
||||
raise api_error(422, ErrorCode.HOST_CONNECTION_FAILED, reason=_connection_failure_reason(exc)) from exc
|
||||
await db.commit()
|
||||
|
||||
statuses = await get_statuses([host])
|
||||
@@ -161,6 +195,7 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
|
||||
raise api_error(409, ErrorCode.HOST_MIT_DIESEM_NAMEN_EXISTIERT_BEREITS)
|
||||
host.name = payload.name
|
||||
if payload.base_url is not None:
|
||||
await _ensure_url_free(db, payload.base_url, exclude_id=host_id)
|
||||
host.base_url = payload.base_url
|
||||
if payload.verify_tls is not None:
|
||||
host.verify_tls = payload.verify_tls
|
||||
@@ -188,11 +223,15 @@ async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession
|
||||
|
||||
|
||||
@router.delete("/{host_id}", status_code=204)
|
||||
async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def delete_host(host_id: uuid.UUID, confirm_name: str = Query(default="", max_length=120), db: AsyncSession = Depends(get_db)):
|
||||
"""Deleting a host drops its stored credentials at once, so the caller must name it (like every other destructive
|
||||
action in OPNexus), not only a click in the UI: `DELETE /api/hosts/{id}?confirm_name=<name>` (T4)."""
|
||||
await policy_lock(db)
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
if confirm_name != host.name:
|
||||
raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
|
||||
@@ -16,6 +16,7 @@ from app.services import ipsec_psk_store
|
||||
from app.services.ipsec_service import effective_hosts, preview, revision, snapshot, workflow, IPSEC_FIELDS
|
||||
from app.services.ipsec_deploy_service import deploy_tunnel, undeploy_tunnel
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_lease import resource_lease
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/ipsec/tunnels", tags=["ipsec"])
|
||||
@@ -192,26 +193,27 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_child_uuid:
|
||||
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
|
||||
if deployment.opnsense_local_uuid:
|
||||
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
|
||||
if deployment.opnsense_remote_uuid:
|
||||
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
|
||||
if deployment.opnsense_connection_uuid:
|
||||
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
|
||||
if deployment.opnsense_psk_uuid:
|
||||
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
|
||||
await opnsense_client.reconfigure_ipsec(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
async with resource_lease("ipsec", tunnel.id):
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_child_uuid:
|
||||
await opnsense_client.del_ipsec_child(deployment.host, deployment.opnsense_child_uuid)
|
||||
if deployment.opnsense_local_uuid:
|
||||
await opnsense_client.del_ipsec_local(deployment.host, deployment.opnsense_local_uuid)
|
||||
if deployment.opnsense_remote_uuid:
|
||||
await opnsense_client.del_ipsec_remote(deployment.host, deployment.opnsense_remote_uuid)
|
||||
if deployment.opnsense_connection_uuid:
|
||||
await opnsense_client.del_ipsec_connection(deployment.host, deployment.opnsense_connection_uuid)
|
||||
if deployment.opnsense_psk_uuid:
|
||||
await opnsense_client.del_ipsec_psk(deployment.host, deployment.opnsense_psk_uuid)
|
||||
await opnsense_client.reconfigure_ipsec(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
|
||||
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
|
||||
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
|
||||
|
||||
class Preview(BaseModel):
|
||||
action: Literal['reboot','update','major_update']
|
||||
action: Literal['reboot','update','major_update','firmware_rollback']
|
||||
|
||||
class Execute(BaseModel):
|
||||
preview_id: uuid.UUID
|
||||
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
|
||||
def read_operation(op):
|
||||
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
|
||||
|
||||
async def snapshot_or_502(db, host):
|
||||
try: return await inspect_host(db,host)
|
||||
async def snapshot_or_502(db, host, action=None):
|
||||
try:
|
||||
state=await inspect_host(db,host)
|
||||
if action=='firmware_rollback':
|
||||
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
|
||||
rollback=await snapshot_service.rollback_state(host)
|
||||
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
|
||||
state['rollback']=rollback
|
||||
state['actions']['firmware_rollback']=state['actions']['reboot']+(
|
||||
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
|
||||
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
|
||||
return state
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
|
||||
|
||||
async def _changes_since(db, host, target):
|
||||
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
|
||||
if not target:
|
||||
return []
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return []
|
||||
out=[]
|
||||
for kind,model in _DEPLOYMENT_MODELS():
|
||||
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
|
||||
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
|
||||
return sorted(out,key=lambda r:r['deployed_at'])[:50]
|
||||
|
||||
|
||||
def _point_time(target):
|
||||
try:
|
||||
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _DEPLOYMENT_MODELS():
|
||||
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
|
||||
PortForwardDeployment, RuleDeployment)
|
||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
|
||||
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
|
||||
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
|
||||
|
||||
|
||||
async def _mark_rolled_back(db, host, target):
|
||||
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
|
||||
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
|
||||
really writes, and the UI shows the object as not in sync."""
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return 0
|
||||
count=0
|
||||
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
|
||||
for _kind,model in _DEPLOYMENT_MODELS():
|
||||
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
|
||||
row.status='error'; row.message=note[:500]; row.last_payload=None
|
||||
if hasattr(model,'revert_deadline'): row.revert_deadline=None
|
||||
count+=1
|
||||
return count
|
||||
|
||||
async def carp_snapshot_or_502(db, host):
|
||||
try: return await inspect_carp_switch(db,host)
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host=await host_or_404(db,host_id)
|
||||
state=await snapshot_or_502(db,host)
|
||||
state=await snapshot_or_502(db,host,payload.action)
|
||||
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
|
||||
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
|
||||
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
|
||||
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
|
||||
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
|
||||
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||
state=await snapshot_or_502(db,host)
|
||||
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
state=await snapshot_or_502(db,host,op.action)
|
||||
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
|
||||
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
|
||||
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
|
||||
# replaying a reboot/update automatically could interrupt recovery.
|
||||
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
if op.action in ('update','major_update'):
|
||||
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
|
||||
try:
|
||||
point=await snapshot_service.create_rollback_point(host,state.get('version'))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
op.snapshot={**op.snapshot,'rollback_point':point}
|
||||
await db.commit()
|
||||
if op.action=='firmware_rollback':
|
||||
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
|
||||
# Fail closed, like the backup before an update.
|
||||
try:
|
||||
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
rollback=state['rollback']
|
||||
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
|
||||
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
|
||||
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
|
||||
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
|
||||
else:
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
|
||||
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
|
||||
try:
|
||||
result=await action_fn(host)
|
||||
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
op.status='submitted';op.message=done
|
||||
if op.action=='firmware_rollback':
|
||||
marked=await _mark_rolled_back(db,host,rollback['target'])
|
||||
if marked:
|
||||
op.message+=f' {marked} Deployment(s) als unklar markiert.'
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
|
||||
result={}
|
||||
|
||||
@@ -50,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
|
||||
@@ -18,6 +18,7 @@ from app.services import openvpn_key_store
|
||||
from app.services.openvpn_service import effective_hosts, preview, revision, snapshot, workflow, OPENVPN_FIELDS
|
||||
from app.services.openvpn_deploy_service import deploy_instance, undeploy_instance
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_lease import resource_lease
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/openvpn/instances", tags=["openvpn"])
|
||||
@@ -194,20 +195,21 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
|
||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_instance_uuid:
|
||||
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
|
||||
if deployment.opnsense_statickey_uuid:
|
||||
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
|
||||
await opnsense_client.reconfigure_openvpn(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
async with resource_lease("openvpn", instance.id):
|
||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_instance_uuid:
|
||||
await opnsense_client.del_openvpn_instance(deployment.host, deployment.opnsense_instance_uuid)
|
||||
if deployment.opnsense_statickey_uuid:
|
||||
await opnsense_client.del_openvpn_static_key(deployment.host, deployment.opnsense_statickey_uuid)
|
||||
await opnsense_client.reconfigure_openvpn(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
@@ -51,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
|
||||
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
hierarchy_group_id=rule.hierarchy_group_id,
|
||||
rulebase=rule.rulebase,
|
||||
sequence=rule.sequence,
|
||||
|
||||
@@ -50,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
ipprotocol=rule.ipprotocol,
|
||||
|
||||
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -20,7 +20,7 @@ from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||
@@ -36,6 +36,19 @@ class WaveIn(BaseModel):
|
||||
require_approval: bool = False
|
||||
|
||||
|
||||
class ProbeIn(BaseModel):
|
||||
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
|
||||
type: Literal["ping", "tcp"]
|
||||
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
|
||||
port: int | None = Field(default=None, ge=1, le=65535)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _port_only_for_tcp(self):
|
||||
if (self.type == "tcp") != (self.port is not None):
|
||||
raise ValueError("port_required_for_tcp_only")
|
||||
return self
|
||||
|
||||
|
||||
class PlanIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
description: str = Field(default="", max_length=2000)
|
||||
@@ -43,6 +56,7 @@ class PlanIn(BaseModel):
|
||||
window_id: uuid.UUID | None = None
|
||||
auto_start: bool = False
|
||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
|
||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||
|
||||
@@ -64,6 +78,7 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||
return {
|
||||
**rollout_service.definition(plan),
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"description": plan.description,
|
||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||
@@ -81,6 +96,15 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
||||
}
|
||||
|
||||
|
||||
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
for probe in probes:
|
||||
value = {"type": probe.type, "target": probe.target, "port": probe.port}
|
||||
if value not in out:
|
||||
out.append(value)
|
||||
return out
|
||||
|
||||
|
||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
@@ -99,6 +123,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||
plan.gate_probes = _unique_probes(payload.gate_probes)
|
||||
plan.compliance_mode = payload.compliance_mode
|
||||
plan.waves = [
|
||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||
|
||||
@@ -52,6 +52,7 @@ def _to_read(
|
||||
return RuleRead(
|
||||
workflow_state=state,
|
||||
workflow_counts=counts, id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
direction=rule.direction,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
@@ -21,10 +21,29 @@ async def _admin_count(db: AsyncSession, exclude_id: uuid.UUID | None = None) ->
|
||||
return await db.scalar(query)
|
||||
|
||||
|
||||
_ADMIN_LOCK_KEY = 728194032 # next to policy_lock's key; only admin role changes and deletions take it
|
||||
|
||||
|
||||
async def _lock_admins(db: AsyncSession, current: AdminUser) -> None:
|
||||
"""Serialize every change that can reduce the number of administrators and re-read the caller under the lock.
|
||||
|
||||
The "at least one administrator" check counts rows; two requests that each see the other's admin row would both
|
||||
pass it. Held until commit/rollback (transaction-scoped advisory lock). The caller may have been demoted or
|
||||
deleted by the request that ran first, so it must still be an admin to proceed."""
|
||||
await db.execute(text("SELECT pg_advisory_xact_lock(:key)"), {"key": _ADMIN_LOCK_KEY})
|
||||
fresh = await db.scalar(select(AdminUser).where(AdminUser.id == current.id).execution_options(populate_existing=True))
|
||||
if fresh is None or fresh.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
|
||||
|
||||
@router.get("", response_model=list[UserRead])
|
||||
async def list_users(db: AsyncSession = Depends(get_db)):
|
||||
rows = (await db.scalars(select(AdminUser).order_by(AdminUser.username))).all()
|
||||
return rows
|
||||
async def list_users(user: AdminUser = Depends(current_user), db: AsyncSession = Depends(get_db)):
|
||||
"""Admins see everyone. Every other role sees only its own account: the full list (names, roles, last login,
|
||||
2FA status) is a directory an attacker with any low-privilege account or token could use to pick targets."""
|
||||
query = select(AdminUser).order_by(AdminUser.username)
|
||||
if user.role != "admin":
|
||||
query = query.where(AdminUser.id == user.id)
|
||||
return (await db.scalars(query)).all()
|
||||
|
||||
|
||||
@router.post("", response_model=UserRead, status_code=201)
|
||||
@@ -43,8 +62,12 @@ async def create_user(payload: UserCreate, db: AsyncSession = Depends(get_db)):
|
||||
|
||||
|
||||
@router.patch("/{user_id}/role", response_model=UserRead)
|
||||
async def update_role(user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db)):
|
||||
target = await db.get(AdminUser, user_id)
|
||||
async def update_role(
|
||||
user_id: uuid.UUID, payload: UserRoleUpdate, db: AsyncSession = Depends(get_db),
|
||||
current: AdminUser = Depends(current_user),
|
||||
):
|
||||
await _lock_admins(db, current)
|
||||
target = await db.get(AdminUser, user_id, populate_existing=True)
|
||||
if target is None:
|
||||
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
||||
if target.auth_source == "oidc":
|
||||
@@ -96,13 +119,13 @@ async def delete_user(
|
||||
):
|
||||
if user_id == current.id:
|
||||
raise api_error(400, ErrorCode.DER_EIGENE_ACCOUNT_KANN_NICHT_GELOESCHT_WERDEN)
|
||||
target = await db.get(AdminUser, user_id)
|
||||
await _lock_admins(db, current)
|
||||
target = await db.get(AdminUser, user_id, populate_existing=True)
|
||||
if target is None:
|
||||
raise api_error(404, ErrorCode.BENUTZER_NICHT_GEFUNDEN)
|
||||
# No separate "last admin" guard here: only an admin can reach this
|
||||
# endpoint (read_only is rejected earlier, in AuthMiddleware) and the
|
||||
# self-delete check above already guarantees target != current — so
|
||||
# `current` is always a still-existing admin afterward, meaning the
|
||||
# admin count can never reach zero this way.
|
||||
# Two admins deleting each other (or demoting/deleting in parallel) would each still see the other: the lock above
|
||||
# orders them, and the guard below counts what is left without the target.
|
||||
if target.role == "admin" and await _admin_count(db, exclude_id=target.id) == 0:
|
||||
raise api_error(409, ErrorCode.MINDESTENS_EIN_ADMIN_ACCOUNT_MUSS_BESTEHEN_BLEIBEN)
|
||||
await db.delete(target)
|
||||
await db.commit()
|
||||
@@ -18,6 +18,7 @@ from app.services import wireguard_key_store
|
||||
from app.services.wireguard_service import effective_hosts, preview, revision, snapshot, workflow, WIREGUARD_FIELDS
|
||||
from app.services.wireguard_deploy_service import deploy_tunnel, undeploy_tunnel
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_lease import resource_lease
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/wireguard/tunnels", tags=["wireguard"])
|
||||
@@ -206,20 +207,21 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
raise api_error(404, ErrorCode.DEPLOYMENT_NICHT_GEFUNDEN)
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_peer_uuid:
|
||||
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
|
||||
if deployment.opnsense_server_uuid:
|
||||
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
|
||||
await opnsense_client.reconfigure_wireguard(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
async with resource_lease("wireguard", tunnel.id):
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_peer_uuid:
|
||||
await opnsense_client.del_wireguard_peer(deployment.host, deployment.opnsense_peer_uuid)
|
||||
if deployment.opnsense_server_uuid:
|
||||
await opnsense_client.del_wireguard_server(deployment.host, deployment.opnsense_server_uuid)
|
||||
await opnsense_client.reconfigure_wireguard(deployment.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
raise api_error(502, ErrorCode.ENTFERNEN_FEHLGESCHLAGEN_DIE_ZUORDNUNG_BLEIBT_FUER_EINEN_NEUEN_4CF90E4C) from exc
|
||||
await db.delete(deployment)
|
||||
await db.commit()
|
||||
@@ -12,7 +12,9 @@ CertAction = Literal["import", "internal", "external", "sign_csr", "import_csr",
|
||||
# EC curves) — see ManagedCertificate's docstring.
|
||||
CertKeyType = Literal["2048", "3072", "4096", "prime256v1", "secp384r1", "secp521r1"]
|
||||
CertDigest = Literal["sha1", "sha224", "sha256", "sha384", "sha512"]
|
||||
CertType = Literal["usr_cert", "server_cert", "combined_server_client", "v3_ca"]
|
||||
# No "v3_ca": OPNsense would create a certificate with the CA extension, not a CA that can be used as `caref`
|
||||
# (found by the 1.31.0 test run, T3). Creating a CA is not offered; the UI never did.
|
||||
CertType = Literal["usr_cert", "server_cert", "combined_server_client"]
|
||||
|
||||
_DN_ACTIONS = {"internal", "external", "reissue"}
|
||||
_CRT_ACTIONS = {"import", "import_csr", "manual"}
|
||||
|
||||
@@ -18,6 +18,9 @@ class HostCreate(BaseModel):
|
||||
enabled: bool = True
|
||||
cluster_name: str | None = Field(default=None, max_length=120)
|
||||
policy_hierarchy_group_id: uuid.UUID | None = None
|
||||
# Ask the firewall once with these credentials before saving (the UI and the setup assistant do). Off by default so
|
||||
# scripts can still register a firewall that is not reachable yet (behind NAT, tunnel not up).
|
||||
verify_connection: bool = False
|
||||
|
||||
@field_validator("base_url")
|
||||
@classmethod
|
||||
|
||||
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
hierarchy_group_id: uuid.UUID
|
||||
rulebase: Literal["pre", "post"]
|
||||
sequence: int
|
||||
|
||||
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
ipprotocol: str
|
||||
|
||||
@@ -48,6 +48,7 @@ class RuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
direction: str
|
||||
|
||||
@@ -13,6 +13,7 @@ from app.models.alias_deployment import AliasDeployment
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.host import AliasOption
|
||||
from app.services import opnsense_client
|
||||
from app.services.deploy_flow import unsettled
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
|
||||
@@ -87,7 +88,7 @@ async def ensure_aliases_settled(db: AsyncSession, hosts: list[OpnsenseHost], va
|
||||
.where(
|
||||
FirewallAlias.name.in_(names),
|
||||
AliasDeployment.host_id.in_([host.id for host in hosts]),
|
||||
AliasDeployment.status.in_(("pending", "revert_failed")),
|
||||
unsettled(AliasDeployment),
|
||||
)
|
||||
.order_by(FirewallAlias.name, OpnsenseHost.name)
|
||||
)).all()
|
||||
|
||||
@@ -45,8 +45,9 @@ _SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-tick
|
||||
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
||||
# notification settings the webhook URL/SMTP data): admin scope only.
|
||||
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
|
||||
_MCP_PATH = "/api/mcp"
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
|
||||
# it compares a posted file with the database and writes nothing.
|
||||
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
|
||||
_ADMIN_ONLY_WRITES = re.compile(
|
||||
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
||||
r"|^/api/logs/settings$"
|
||||
@@ -54,6 +55,8 @@ _ADMIN_ONLY_WRITES = re.compile(
|
||||
r"|^/api/hosts/[^/]+/gui-ticket$"
|
||||
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
|
||||
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
|
||||
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
|
||||
r"|^/api/config/import/apply$"
|
||||
)
|
||||
|
||||
|
||||
@@ -76,7 +79,7 @@ def denies(scope: str, method: str, path: str) -> bool:
|
||||
"""True if a token with this scope must not make this request."""
|
||||
if _SESSION_ONLY.match(path):
|
||||
return True
|
||||
if path == _MCP_PATH:
|
||||
if path in _READ_ONLY_POSTS:
|
||||
return False
|
||||
if method in _SAFE_METHODS:
|
||||
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
|
||||
@@ -98,7 +101,7 @@ def operator_denies(method: str, path: str) -> bool:
|
||||
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
|
||||
# maintenance (reboot/update/CARP switch), scheduling it, applying network changes, pushing certificates and
|
||||
# ACME accounts, re-asserting the hierarchy order, controlling services, and DELETING a centrally managed object
|
||||
# or snapshot (deleting a rule removes it from every firewall it was deployed to).
|
||||
# or snapshot (deleting a rule removes it from every firewall it was deployed to), and creating a snapshot.
|
||||
_ROLLOUT_POST = re.compile(
|
||||
r"/(deploy|push|sign|poll|register|reassert-ordering"
|
||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||
@@ -107,6 +110,7 @@ _ROLLOUT_POST = re.compile(
|
||||
r"|rule-cleanup/[^/]+/(revert|retry)"
|
||||
r"|maintenance/(execute|carp-execute|schedule)"
|
||||
r"|network/changes"
|
||||
r"|snapshots" # creating a boot environment (deleting one is in _ROLLOUT_DELETE)
|
||||
r"|services/[^/]+/(start|stop|restart)"
|
||||
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
|
||||
r"|rollouts/plans/[^/]+/(start|activate)"
|
||||
|
||||
@@ -236,6 +236,7 @@ async def create_session(
|
||||
user_agent: str | None = None,
|
||||
ip_address: str | None = None,
|
||||
verified_hash: str | None = None,
|
||||
verified_totp: bool | None = None,
|
||||
lifetime_seconds: int | None = None,
|
||||
) -> tuple[str, datetime]:
|
||||
"""Create a session, returning the raw token (to be set as a cookie) and its expiry.
|
||||
@@ -244,15 +245,22 @@ async def create_session(
|
||||
against. The user row is re-read under a row lock first: a password
|
||||
change/reset that committed while the (slow) password check was running
|
||||
changed the hash, and the login must not outlive that revocation.
|
||||
|
||||
``verified_totp`` is whether the caller checked a second factor. Two-factor
|
||||
enabled in the meantime (it also revoked every session) means this login
|
||||
was decided without it and must start over.
|
||||
"""
|
||||
user_id = user.id
|
||||
if verified_hash is not None:
|
||||
locked = await db.scalar(
|
||||
select(AdminUser.password_hash).where(AdminUser.id == user_id).with_for_update()
|
||||
)
|
||||
if locked != verified_hash:
|
||||
if verified_hash is not None or verified_totp is not None:
|
||||
locked_hash, locked_totp = (await db.execute(
|
||||
select(AdminUser.password_hash, AdminUser.totp_enabled).where(AdminUser.id == user_id).with_for_update()
|
||||
)).one()
|
||||
if verified_hash is not None and locked_hash != verified_hash:
|
||||
await db.rollback()
|
||||
raise api_error(401, ErrorCode.UNGUELTIGER_BENUTZERNAME_ODER_PASSWORT)
|
||||
if locked_totp and not verified_totp:
|
||||
await db.rollback()
|
||||
raise api_error(401, ErrorCode.TOTP_REQUIRED)
|
||||
token = secrets.token_urlsafe(SESSION_TOKEN_BYTES)
|
||||
expires_at = _now() + timedelta(seconds=lifetime_seconds or settings.session_lifetime)
|
||||
session = Session(
|
||||
|
||||
@@ -29,6 +29,7 @@ and the sweeper away in the gap (pending/unknown rows are never re-deployed).
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
from sqlalchemy import and_, or_
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
@@ -46,7 +47,21 @@ def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
async def _checkpoint(db: AsyncSession) -> None:
|
||||
def is_unsettled(row) -> bool:
|
||||
"""A change that may still be live and whose rollback is not yet done: pending, outcome unknown, or an
|
||||
"error" row whose revert deadline is still armed (written but apply/ordering not confirmed)."""
|
||||
return row.status in ("pending", "revert_failed") or (row.status == "error" and row.revert_deadline is not None)
|
||||
|
||||
|
||||
def unsettled(model):
|
||||
"""``is_unsettled`` as a SQL condition for the deployment model."""
|
||||
return or_(
|
||||
model.status.in_(("pending", "revert_failed")),
|
||||
and_(model.status == "error", model.revert_deadline.isnot(None)),
|
||||
)
|
||||
|
||||
|
||||
async def checkpoint(db: AsyncSession) -> None:
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
await db.commit()
|
||||
@@ -67,12 +82,16 @@ async def push_to_host(
|
||||
) -> Any:
|
||||
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
|
||||
before = {name: getattr(existing, name) for name in _FIELDS} if existing else None
|
||||
# An armed "error" row (written, apply not confirmed) holds the UNCONFIRMED payload as last_payload: a retry must
|
||||
# keep the original rollback target instead of promoting that payload to it.
|
||||
armed = existing is not None and existing.status == "error" and existing.revert_deadline is not None
|
||||
rollback_target = existing.previous_payload if armed else prior_live_payload
|
||||
|
||||
# 1. durable intent
|
||||
if existing is not None:
|
||||
row = existing
|
||||
row.status, row.message = "pending", None
|
||||
row.previous_payload, row.last_payload = prior_live_payload, payload
|
||||
row.previous_payload, row.last_payload = rollback_target, payload
|
||||
row.revert_deadline, row.deployed_at = _now() + window, _now()
|
||||
else:
|
||||
row = make(
|
||||
@@ -83,7 +102,7 @@ async def push_to_host(
|
||||
# The intent is a technical safety record, not a state change worth a history
|
||||
# entry of its own (the request itself is already audited as "requested").
|
||||
row._skip_audit = True
|
||||
await _checkpoint(db)
|
||||
await checkpoint(db)
|
||||
row._skip_audit = False
|
||||
|
||||
wrote = False
|
||||
@@ -93,9 +112,9 @@ async def push_to_host(
|
||||
# 3. recoverable by UUID from here on
|
||||
setattr(row, uuid_attr, new_uuid)
|
||||
row.status, row.message = "pending", None
|
||||
row.previous_payload = prior_live_payload if existing is not None else None
|
||||
row.previous_payload = rollback_target if existing is not None else None
|
||||
row.last_payload, row.revert_deadline, row.deployed_at = payload, _now() + window, _now()
|
||||
await _checkpoint(db)
|
||||
await checkpoint(db)
|
||||
await apply()
|
||||
except OpnsenseAPIError as exc:
|
||||
ambiguous = isinstance(exc, OpnsenseAmbiguousError)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
"""An exclusive per-object lock for flows that make several remote calls and commit in between (VPN deploys).
|
||||
|
||||
``policy_lock`` is transaction-scoped and is released by every checkpoint commit; this one is a session-level advisory
|
||||
lock on a connection of its own, so it spans the commits of the request session. If the process dies the connection
|
||||
closes and PostgreSQL frees the lock. It never waits: a second deploy/removal of the same object is refused at once.
|
||||
"""
|
||||
import uuid
|
||||
import zlib
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.database import engine
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
# Written when the first row is committed, before any remote call: a crash or cancellation leaves this, and a new
|
||||
# deploy continues from the UUIDs recorded after it.
|
||||
VPN_INTERRUPTED = (
|
||||
"Ausrollen unterbrochen oder Ausgang unklar: Die Firewall hat möglicherweise schon Teile der Konfiguration "
|
||||
"erhalten. Erneutes Ausrollen setzt dort fort."
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def resource_lease(kind: str, resource_id: uuid.UUID):
|
||||
params = {"a": zlib.crc32(kind.encode()) & 0x7FFFFFFF, "b": resource_id.int & 0x7FFFFFFF}
|
||||
async with engine.connect() as conn:
|
||||
if not await conn.scalar(text("SELECT pg_try_advisory_lock(:a, :b)"), params):
|
||||
raise api_error(409, ErrorCode.VPN_DEPLOY_LAEUFT_BEREITS)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
try:
|
||||
await conn.execute(text("SELECT pg_advisory_unlock(:a, :b)"), params)
|
||||
except Exception:
|
||||
await conn.invalidate() # never hand a connection that may still hold the lock back to the pool
|
||||
except BaseException: # cancelled while unlocking
|
||||
await conn.invalidate()
|
||||
raise
|
||||
@@ -63,6 +63,17 @@ async def _guarded(host: OpnsenseHost, tool: str, target: str, params: dict, cal
|
||||
return result
|
||||
|
||||
|
||||
def _with_silent_hops(hops: list[dict]) -> list[dict]:
|
||||
"""The firewall leaves out hops that never answered (TTL jumps 1 -> 3, T5 of the 1.31.0 test run). Put them back as
|
||||
explicit rows so a gap reads as "no answer here", not as a missing hop. A route where nothing answered at all stays
|
||||
an empty list; the UI says so."""
|
||||
seen = {int(h["ttl"]): h for h in hops if h["ttl"].isdigit()}
|
||||
if not seen:
|
||||
return hops
|
||||
silent = {"address": "", "host": "", "as": "", "probes": "", "silent": True}
|
||||
return [seen.get(ttl) or {**silent, "ttl": str(ttl)} for ttl in range(1, max(seen) + 1)]
|
||||
|
||||
|
||||
async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: str, source: str | None) -> dict:
|
||||
target, source = clean_target(target), clean_source(source)
|
||||
params = {"target": target, "ipproto": ipproto, "protocol": protocol, "source": source}
|
||||
@@ -73,8 +84,9 @@ async def traceroute(host: OpnsenseHost, target: str, ipproto: str, protocol: st
|
||||
{"hostname": target, "ipproto": ipproto, "protocol": protocol, "source_address": source})
|
||||
resp = data.get("response") or {}
|
||||
hops = [{"ttl": str(r.get("ttl", "")), "address": str(r.get("address", "")), "host": str(r.get("host", "")),
|
||||
"as": str(r.get("AS", "")), "probes": str(r.get("probes", ""))} for r in resp.get("rows", []) if isinstance(r, dict)]
|
||||
return {"target": target, "notice": str(resp.get("notice", "")), "hops": hops}
|
||||
"as": str(r.get("AS", "")), "probes": str(r.get("probes", "")), "silent": False}
|
||||
for r in resp.get("rows", []) if isinstance(r, dict)]
|
||||
return {"target": target, "notice": str(resp.get("notice", "")), "hops": _with_silent_hops(hops)}
|
||||
return await _guarded(host, "traceroute", target, params, call)
|
||||
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ from app.models import OpnsenseHost, RuleDeployment, AliasDeployment, PortForwar
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.restore_operation import RestoreOperation
|
||||
from app.services import opnsense_client as client, backup_store, carp_maintenance_window_service
|
||||
from app.services.deploy_flow import unsettled
|
||||
from app.services.host_status_service import get_statuses
|
||||
|
||||
DEPLOYMENTS = (RuleDeployment, AliasDeployment, PortForwardDeployment, OneToOneDeployment, OutboundNatDeployment)
|
||||
@@ -70,7 +71,7 @@ async def _shared_change_blockers(db, fleet):
|
||||
blocked = []
|
||||
fleet_ids = [h.id for h in fleet]
|
||||
for model in DEPLOYMENTS:
|
||||
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), model.status.in_(("pending", "revert_failed"))).limit(1)):
|
||||
if await db.scalar(select(model.id).where(model.host_id.in_(fleet_ids), unsettled(model)).limit(1)):
|
||||
blocked.append("Offene Bestätigung oder fehlgeschlagener Rollback im Host/Cluster.")
|
||||
break
|
||||
operations = list((await db.scalars(select(HostOperation).where(HostOperation.host_id.in_(fleet_ids), HostOperation.status.in_(ACTIVE)))).all())
|
||||
|
||||
@@ -109,6 +109,8 @@ from app.models.ipsec_tunnel import IpsecTunnel
|
||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||
from app.services import opnsense_client, ipsec_psk_store
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_flow import checkpoint
|
||||
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
|
||||
|
||||
_VERSION_MAP = {"ike": "0", "ikev1": "1", "ikev2": "2"}
|
||||
|
||||
@@ -176,7 +178,7 @@ def build_payloads(tunnel: IpsecTunnel) -> dict:
|
||||
}
|
||||
|
||||
|
||||
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
|
||||
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: IpsecTunnelDeployment) -> None:
|
||||
"""Runs the 5 config-write calls in dependency order, persisting each
|
||||
UUID onto `deployment` as soon as it's known so a later retry (after a
|
||||
partial failure) updates rather than duplicates. Raises
|
||||
@@ -189,6 +191,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_connection_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_connection_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Connection-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
# `payloads` (built by build_payloads()) is also what gets persisted as
|
||||
# last_payload for no-op-redeploy comparison — the injected `connection`
|
||||
@@ -209,6 +212,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_local_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_local_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Local-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
if deployment.opnsense_remote_uuid:
|
||||
await opnsense_client.set_ipsec_remote(host, deployment.opnsense_remote_uuid, remote_wire)
|
||||
@@ -217,6 +221,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_remote_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_remote_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Remote-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
if deployment.opnsense_child_uuid:
|
||||
await opnsense_client.set_ipsec_child(host, deployment.opnsense_child_uuid, child_wire)
|
||||
@@ -225,6 +230,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_child_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_child_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Child-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
if deployment.opnsense_psk_uuid:
|
||||
await opnsense_client.set_ipsec_psk(host, deployment.opnsense_psk_uuid, payloads["psk"])
|
||||
@@ -233,6 +239,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: IpsecTun
|
||||
deployment.opnsense_psk_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_psk_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine PSK-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
|
||||
async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseHost) -> IpsecTunnelDeployment:
|
||||
@@ -241,7 +248,7 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
existing = await db.scalar(
|
||||
select(IpsecTunnelDeployment).where(
|
||||
IpsecTunnelDeployment.tunnel_id == tunnel.id, IpsecTunnelDeployment.host_id == host.id
|
||||
)
|
||||
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
|
||||
)
|
||||
if existing and existing.status == "success" and existing.last_payload == comparable:
|
||||
# Nothing actually changed on this host — re-affirm without
|
||||
@@ -252,8 +259,15 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
|
||||
deployment = existing or IpsecTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
|
||||
|
||||
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
|
||||
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
|
||||
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
await checkpoint(db)
|
||||
|
||||
try:
|
||||
await _write_config(host, payloads, deployment)
|
||||
await _write_config(db, host, payloads, deployment)
|
||||
# Same trap as WireGuard: with the global IPsec switch off the config
|
||||
# is stored but nothing runs, and reconfigure still answers ok.
|
||||
await opnsense_client.ensure_ipsec_enabled(host)
|
||||
@@ -261,8 +275,6 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
try:
|
||||
@@ -277,28 +289,30 @@ async def _deploy_to_host(db: AsyncSession, tunnel: IpsecTunnel, host: OpnsenseH
|
||||
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
deployment.status = "success"
|
||||
deployment.message = None
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
|
||||
async def deploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel, hosts: list[OpnsenseHost]) -> list[IpsecTunnelDeployment]:
|
||||
from app.services.host_maintenance_service import ensure_no_maintenance
|
||||
await ensure_no_maintenance(db, hosts)
|
||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||
await db.commit()
|
||||
async with resource_lease("ipsec", tunnel.id):
|
||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||
await db.commit()
|
||||
return deployments
|
||||
|
||||
|
||||
async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||
async with resource_lease("ipsec", tunnel.id):
|
||||
await _undeploy_tunnel(db, tunnel)
|
||||
|
||||
|
||||
async def _undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||
"""Explicit action, not gated by anything (there is no lockout state
|
||||
for this entity to gate on) — deletes Child/Local/Remote before their
|
||||
parent Connection (the ModelRelationField dependency direction), then
|
||||
|
||||
@@ -87,6 +87,13 @@ OBJECT_COLUMNS: dict[str, list[str]] = {
|
||||
# These change how the network is wired; the others (DNS/DHCP data) cannot cut the connection to the firewall.
|
||||
WIRING_KINDS = {"vlan", "vip", "gateway", "route", "interface", "lagg", "bridge", "gre", "gif", "vxlan"}
|
||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||
# Integer ranges of the firewall's own model (Interfaces/Vlan.xml, Interfaces/Vip.xml, read 2026-10-23). Checked before the
|
||||
# preview so a wrong number is a 422 with the field name, not a 502 after the firewall refused it (T6 of the 1.31.0 test
|
||||
# run). Only fields whose range is stated in the model; an empty value is left to the firewall (e.g. vhid of a non-CARP VIP).
|
||||
RANGES: dict[tuple[str, str], tuple[int, int]] = {
|
||||
("vlan", "tag"): (1, 4094), ("vlan", "pcp"): (0, 7),
|
||||
("vip", "vhid"): (1, 255), ("vip", "advbase"): (1, 254), ("vip", "advskew"): (0, 254),
|
||||
}
|
||||
|
||||
|
||||
def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
||||
@@ -104,6 +111,9 @@ def validate_fields(kind: str, fields: dict) -> dict[str, str]:
|
||||
text = str(value).strip()
|
||||
if len(text) > 255 or _CONTROL.search(text):
|
||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
||||
bounds = RANGES.get((kind, key))
|
||||
if bounds and text and not (text.isdecimal() and bounds[0] <= int(text) <= bounds[1]):
|
||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field=key)
|
||||
clean[key] = text
|
||||
return clean
|
||||
|
||||
|
||||
@@ -165,6 +165,8 @@ from app.models.openvpn_instance import OpenVpnInstance
|
||||
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||
from app.services import opnsense_client, openvpn_key_store
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_flow import checkpoint
|
||||
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
@@ -252,7 +254,7 @@ async def _next_vpnid(host: OpnsenseHost) -> str:
|
||||
return str(max(used, default=0) + 1)
|
||||
|
||||
|
||||
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnInstanceDeployment) -> None:
|
||||
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: OpenVpnInstanceDeployment) -> None:
|
||||
"""Writes the StaticKey (if enabled) then the Instance, in that
|
||||
dependency order, persisting each UUID onto `deployment` as soon as
|
||||
it's known so a later retry (after a partial failure) updates rather
|
||||
@@ -267,11 +269,13 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnI
|
||||
deployment.opnsense_statickey_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_statickey_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine StaticKey-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
elif deployment.opnsense_statickey_uuid:
|
||||
# tls_key hardening was switched off after previously being on —
|
||||
# remove the now-orphaned StaticKey rather than leaving it behind.
|
||||
await opnsense_client.del_openvpn_static_key(host, deployment.opnsense_statickey_uuid)
|
||||
deployment.opnsense_statickey_uuid = None
|
||||
await checkpoint(db)
|
||||
|
||||
# Overlay the deploy-time-known tls_key UUID (or "" to explicitly
|
||||
# clear a previously-set relation — see module docstring) onto a
|
||||
@@ -292,13 +296,14 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: OpenVpnI
|
||||
deployment.opnsense_instance_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_instance_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Instance-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
|
||||
async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: OpnsenseHost) -> OpenVpnInstanceDeployment:
|
||||
existing = await db.scalar(
|
||||
select(OpenVpnInstanceDeployment).where(
|
||||
OpenVpnInstanceDeployment.instance_id == instance.id, OpenVpnInstanceDeployment.host_id == host.id
|
||||
)
|
||||
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
|
||||
)
|
||||
try:
|
||||
payloads = build_payloads(instance, host.id)
|
||||
@@ -323,14 +328,19 @@ async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: Opn
|
||||
|
||||
deployment = existing or OpenVpnInstanceDeployment(instance_id=instance.id, host_id=host.id, status="error")
|
||||
|
||||
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
|
||||
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
|
||||
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
await checkpoint(db)
|
||||
|
||||
try:
|
||||
await _write_config(host, payloads, deployment)
|
||||
await _write_config(db, host, payloads, deployment)
|
||||
except OpnsenseAPIError as exc:
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
try:
|
||||
@@ -345,28 +355,30 @@ async def _deploy_to_host(db: AsyncSession, instance: OpenVpnInstance, host: Opn
|
||||
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
deployment.status = "success"
|
||||
deployment.message = None
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
|
||||
async def deploy_instance(db: AsyncSession, instance: OpenVpnInstance, hosts: list[OpnsenseHost]) -> list[OpenVpnInstanceDeployment]:
|
||||
from app.services.host_maintenance_service import ensure_no_maintenance
|
||||
await ensure_no_maintenance(db, hosts)
|
||||
deployments = [await _deploy_to_host(db, instance, host) for host in hosts]
|
||||
await db.commit()
|
||||
async with resource_lease("openvpn", instance.id):
|
||||
deployments = [await _deploy_to_host(db, instance, host) for host in hosts]
|
||||
await db.commit()
|
||||
return deployments
|
||||
|
||||
|
||||
async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
|
||||
async with resource_lease("openvpn", instance.id):
|
||||
await _undeploy_instance(db, instance)
|
||||
|
||||
|
||||
async def _undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None:
|
||||
"""Explicit action, not gated by anything (there is no lockout state
|
||||
for this entity to gate on) — deletes the Instance before its
|
||||
referenced StaticKey (the ModelRelationField dependency direction),
|
||||
|
||||
@@ -8,7 +8,10 @@ required.
|
||||
|
||||
Auth is HTTP Basic: username = API key, password = API secret.
|
||||
"""
|
||||
import ipaddress
|
||||
import logging
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
@@ -63,6 +66,34 @@ def _client(host: OpnsenseHost) -> httpx.AsyncClient:
|
||||
)
|
||||
|
||||
|
||||
# ── Rule counters before a reload (B11) ────────────────────────────────────
|
||||
# Every request below that makes OPNsense reload pf (filter/NAT apply; alias and IPsec reconfigure run
|
||||
# `filter reload skip_alias` too, verified in core stable/26.7; OpenVPN/WireGuard/network may through interface
|
||||
# registration) zeroes every rule counter. Whatever was hit since the last observation would be lost and a rule
|
||||
# could later look "unused". So the counters are read right before, through rule_hit_service. A writer that has
|
||||
# just sampled itself (the rule cleanup) switches this off for its own call with ``counters_already_sampled()``.
|
||||
_sampled_already: ContextVar[bool] = ContextVar("opnexus_counters_sampled", default=False)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def counters_already_sampled():
|
||||
token = _sampled_already.set(True)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_sampled_already.reset(token)
|
||||
|
||||
|
||||
async def _before_reload(host: OpnsenseHost) -> None:
|
||||
if _sampled_already.get():
|
||||
return
|
||||
from app.services.rule_hit_service import sample_before_reload # needs the database; this module does not
|
||||
try:
|
||||
await sample_before_reload(host)
|
||||
except Exception as exc: # noqa: BLE001 - a failed sample must never block or fail the write itself
|
||||
logger.info("Zählerprobe vor dem Anwenden auf %s fehlgeschlagen: %s", host.name, exc)
|
||||
|
||||
|
||||
def _refuse_unresolved_placeholders(method: str, path: str, kwargs: dict) -> None:
|
||||
"""Last line of defence for site variables: a payload that still contains a {{NAME}} placeholder was never
|
||||
resolved for this host and must not reach a firewall."""
|
||||
@@ -221,6 +252,7 @@ async def del_alias_item(host: OpnsenseHost, alias_uuid: str) -> dict:
|
||||
|
||||
|
||||
async def reconfigure_aliases(host: OpnsenseHost) -> dict:
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/alias/reconfigure"), "Aliase anwenden", set())
|
||||
|
||||
|
||||
@@ -366,6 +398,7 @@ def rule_write_payload(rule: dict) -> dict[str, str]:
|
||||
|
||||
|
||||
async def apply(host: OpnsenseHost) -> dict:
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
|
||||
|
||||
|
||||
@@ -471,6 +504,7 @@ async def apply_port_forward(host: OpnsenseHost) -> dict:
|
||||
apply (verified in core source: neither DNatController nor
|
||||
FilterController override applyAction). Called via its own semantically
|
||||
named endpoint anyway, matching what OPNsense's own UI does."""
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/d_nat/apply"), "Änderungen anwenden", set())
|
||||
|
||||
|
||||
@@ -510,6 +544,7 @@ async def del_one_to_one_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||
|
||||
async def apply_one_to_one(host: OpnsenseHost) -> dict:
|
||||
"""Apply through the type-specific controller, using FilterBaseController reload."""
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/one_to_one/apply"), "Änderungen anwenden", set())
|
||||
|
||||
|
||||
@@ -532,6 +567,7 @@ async def del_outbound_nat_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||
|
||||
async def apply_outbound_nat(host: OpnsenseHost) -> dict:
|
||||
"""Apply through the type-specific controller, using FilterBaseController reload."""
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/source_nat/apply"), "Änderungen anwenden", set())
|
||||
|
||||
|
||||
@@ -567,6 +603,59 @@ async def get_pf_rule_statistics(host: OpnsenseHost) -> dict:
|
||||
# - Firewall/Api/GroupController::searchItemAction -> searchBase("ifgroupentry"): rows with
|
||||
# ifname, members (comma separated identifiers), sequence; the built-in groups (enc0, openvpn,
|
||||
# wireguard) appear with their name as uuid and empty members.
|
||||
def _stream_addr(response: httpx.Response, which: str) -> tuple[str, int] | None:
|
||||
stream = response.extensions.get("network_stream")
|
||||
try:
|
||||
addr = stream.get_extra_info(which) if stream is not None else None
|
||||
except Exception: # noqa: BLE001 - transport without that information
|
||||
return None
|
||||
return (str(addr[0]), int(addr[1])) if isinstance(addr, (tuple, list)) and len(addr) >= 2 else None
|
||||
|
||||
|
||||
async def own_source_address(host: OpnsenseHost) -> str | None:
|
||||
"""The address this firewall sees OPNexus's API requests come from (B10): behind NAT or the management tunnel it is
|
||||
not what OPNexus knows about itself, so the firewall is asked. Two requests on one keep-alive connection: the first
|
||||
tells the firewall address and the local port of this connection, the second lists pf's states for that address
|
||||
(`diagnostics/firewall/query_states`, POST, read-only; Diagnostics/Api/FirewallController, core stable/26.7) and the
|
||||
state whose source port is our local port is this very connection. Live 2026-10-23 on test-fw-1/3: one ESTABLISHED
|
||||
state, src 192.0.2.10 = the OPNexus server, port preserved by Docker's NAT.
|
||||
|
||||
None when it cannot be told for sure: through the overlay's CONNECT proxy (the local port belongs to the proxy
|
||||
connection), a NAT that rewrote the port, or an API key without the states privilege (that raises
|
||||
OpnsenseAPIError)."""
|
||||
async with _client(host) as http:
|
||||
try:
|
||||
first = await http.get("/diagnostics/firewall/pf_states")
|
||||
first.raise_for_status()
|
||||
server = _stream_addr(first, "server_addr")
|
||||
if server is None:
|
||||
return None
|
||||
second = await http.post("/diagnostics/firewall/query_states",
|
||||
json={"searchPhrase": server[0], "rowCount": 500, "current": 1})
|
||||
second.raise_for_status()
|
||||
local, server = _stream_addr(second, "client_addr"), _stream_addr(second, "server_addr")
|
||||
rows = second.json().get("rows") or []
|
||||
except (httpx.HTTPError, ValueError, AttributeError) as exc:
|
||||
raise OpnsenseAPIError(f"Zustandstabelle von {host.name} nicht lesbar: {str(exc)[:200]}") from exc
|
||||
if local is None or server is None:
|
||||
return None
|
||||
|
||||
def same_ip(a, b) -> bool:
|
||||
try:
|
||||
return ipaddress.ip_address(str(a)) == ipaddress.ip_address(str(b))
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
for row in rows if isinstance(rows, list) else []:
|
||||
if (isinstance(row, dict) and same_ip(row.get("dst_addr"), server[0]) and str(row.get("dst_port")) == str(server[1])
|
||||
and str(row.get("src_port")) == str(local[1]) and str(row.get("state", "")).startswith("ESTABLISHED")):
|
||||
try:
|
||||
return str(ipaddress.ip_address(str(row.get("src_addr"))))
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
async def get_rule_stats(host: OpnsenseHost) -> dict:
|
||||
"""Per-label counters, already summed over every pf line of a rule. Used for the rule usage sampling
|
||||
(rule_hit_service). Live check 2026-10-22 on all three test firewalls (26.7, one with unmigrated legacy
|
||||
@@ -801,6 +890,7 @@ async def reconfigure_ipsec(host: OpnsenseHost) -> dict:
|
||||
Deliberately its own call, separate from the 5 config-write calls
|
||||
above — see ipsec_deploy_service.py for why this app never bundles it
|
||||
silently."""
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/ipsec/service/reconfigure"), "IPsec-Dienst neu laden", set())
|
||||
|
||||
|
||||
@@ -898,6 +988,7 @@ async def reconfigure_openvpn(host: OpnsenseHost) -> dict:
|
||||
section comment above for why this is NOT assumed symmetric with
|
||||
IPsec's `{"status": "ok"}`), dispatched via _mutation_result with the
|
||||
"result"-based branch (results={"ok"})."""
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/openvpn/service/reconfigure"), "OpenVPN-Dienst neu laden", {"ok"})
|
||||
|
||||
|
||||
@@ -1074,6 +1165,7 @@ async def reconfigure_wireguard(host: OpnsenseHost) -> dict:
|
||||
IPsec's `{"status": "ok"}`, and matches OpenVPN's shape instead),
|
||||
dispatched via _mutation_result with the "result"-based branch
|
||||
(results={"ok"})."""
|
||||
await _before_reload(host)
|
||||
return _mutation_result(await _request(host, "POST", "/wireguard/service/reconfigure"), "WireGuard-Dienst neu laden", {"ok"})
|
||||
|
||||
|
||||
@@ -1394,6 +1486,7 @@ async def write_network_item(host: OpnsenseHost, kind: str, action: str, uuid: s
|
||||
|
||||
|
||||
async def reconfigure_network(host: OpnsenseHost, kind: str) -> dict:
|
||||
await _before_reload(host)
|
||||
return _mutation_result(
|
||||
await _request(host, "POST", NETWORK_ITEM_APIS[kind]["reconfigure"]), "Änderung anwenden", set())
|
||||
|
||||
@@ -1454,6 +1547,13 @@ async def create_snapshot(host: OpnsenseHost, name: str, note: str = "") -> None
|
||||
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
|
||||
|
||||
|
||||
async def activate_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||
"""Boot environment for every following start (bectl activate, not -t); takes effect at the next reboot."""
|
||||
data = await _request(host, "POST", f"/core/snapshots/activate/{uuid}")
|
||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||
raise OpnsenseAPIError(f"Snapshot aktivieren von Firewall nicht bestätigt: {redact_data(data)}")
|
||||
|
||||
|
||||
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
|
||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||
|
||||
@@ -196,6 +196,7 @@ async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, sta
|
||||
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
||||
"window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
||||
}
|
||||
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
||||
@@ -381,7 +382,8 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
||||
baselines.update((w.baseline or {}))
|
||||
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||
probes=run.plan_snapshot.get("gate_probes"))
|
||||
|
||||
async def apply(db, run):
|
||||
if gate["verdict"] == "fail":
|
||||
@@ -407,21 +409,30 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
||||
return apply
|
||||
|
||||
|
||||
# The core package of each OPNsense release type (community, development, business edition). Switching the type
|
||||
# installs another core package: live 2026-10-23, community -> devel offered `opnsense-devel` 27.1.a_483 as a NEW
|
||||
# package while product_latest still said 26.7.7 (the rollout expected 26.7.7 and stopped at the gate).
|
||||
CORE_PACKAGES = ("opnsense", "opnsense-devel", "opnsense-business")
|
||||
|
||||
|
||||
def target_of(status: dict, action: str) -> str | None:
|
||||
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
|
||||
|
||||
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
|
||||
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
|
||||
core package's new version comes first; product_latest is only a fallback."""
|
||||
core package's new version comes first (whichever release type it is, see CORE_PACKAGES, also when it
|
||||
arrives as a new package); product_latest is only a fallback."""
|
||||
if action == "major_update":
|
||||
return status.get("upgrade_major_version") or None
|
||||
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
|
||||
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
|
||||
if core and core.get("new") and core.get("new") != "N/A":
|
||||
return str(core["new"])
|
||||
for row in status.get("upgrade_packages") or []:
|
||||
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
|
||||
return str(row["new_version"])
|
||||
for name in CORE_PACKAGES:
|
||||
core = packages.get(name) if isinstance(packages.get(name), dict) else None
|
||||
if core and core.get("new") and core.get("new") != "N/A":
|
||||
return str(core["new"])
|
||||
for key, field in (("upgrade_packages", "new_version"), ("new_packages", "version")):
|
||||
for row in status.get(key) or []:
|
||||
if isinstance(row, dict) and row.get("name") in CORE_PACKAGES and row.get(field):
|
||||
return str(row[field])
|
||||
product = status.get("product") if isinstance(status.get("product"), dict) else {}
|
||||
return product.get("product_latest") or status.get("product_latest") or None
|
||||
|
||||
@@ -537,7 +548,7 @@ async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | Non
|
||||
versions = {}
|
||||
for rh_id in offers:
|
||||
host = usable[rh_id]
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host)
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host, run.plan_snapshot.get("gate_probes"))
|
||||
versions[rh_id] = baselines[str(host.id)].get("version")
|
||||
roles = await get_statuses([usable[i] for i in offers])
|
||||
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
||||
@@ -630,6 +641,7 @@ async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision |
|
||||
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
||||
gate = await rollout_health.evaluate(
|
||||
db, done_peers + [host], baselines, target_version=run.target_version,
|
||||
# No probes here: the peers' probes were checked in their own gate, and this runs on every tick.
|
||||
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
||||
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
||||
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
||||
@@ -718,7 +730,8 @@ async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||
probes=run.plan_snapshot.get("gate_probes"))
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, index)
|
||||
|
||||
@@ -18,13 +18,19 @@ the configuration like the 6-hourly security check does):
|
||||
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
||||
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
||||
update may legitimately change defaults.
|
||||
* probes — optional per plan (1.32): ping/TCP probes run FROM the firewall (uplink, DNS, an internal
|
||||
server). A probe that worked in the baseline and fails now fails the gate; one that failed
|
||||
already before only warns. This is reachability from the firewall, not traffic through it.
|
||||
|
||||
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
||||
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
||||
"""
|
||||
import asyncio
|
||||
import uuid
|
||||
from collections import defaultdict
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -41,6 +47,8 @@ DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": Tru
|
||||
"compliance": False}
|
||||
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
||||
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
||||
MAX_PROBES = 10
|
||||
PROBE_PING_COUNT = 3
|
||||
|
||||
|
||||
def _r(result: str, code: str, **params) -> dict:
|
||||
@@ -64,7 +72,55 @@ async def _services(host) -> dict | None:
|
||||
return None if listing is None else _services_compact(listing)
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
def _probe_key(probe: dict) -> str:
|
||||
return f"{probe['type']} {probe['target']}" + (f":{probe['port']}" if probe.get("port") else "")
|
||||
|
||||
|
||||
async def run_probes(host: OpnsenseHost, probes: list[dict] | None) -> list[dict]:
|
||||
"""Every probe from the firewall, one after the other; a probe that cannot run counts as failed. Audited by
|
||||
diagnostics_service like a probe from the host page, as `system:rollout` when no request context exists."""
|
||||
from app.services import audit_service, diagnostics_service
|
||||
|
||||
if not probes:
|
||||
return []
|
||||
token = None
|
||||
if audit_service.context.get() is None:
|
||||
token = audit_service.context.set({"actor": "system:rollout", "id": str(uuid.uuid4()), "action": "probe"})
|
||||
try:
|
||||
async def one(probe: dict) -> dict:
|
||||
ipproto = "inet6" if ":" in probe["target"] else "inet"
|
||||
try:
|
||||
if probe["type"] == "ping":
|
||||
result = await diagnostics_service.ping(host, probe["target"], ipproto, PROBE_PING_COUNT, None, None)
|
||||
ok = int(result.get("received") or 0) > 0
|
||||
else:
|
||||
result = await diagnostics_service.portprobe(host, probe["target"], int(probe["port"]), ipproto, None)
|
||||
ok = bool(result.get("open"))
|
||||
except (HTTPException, client.OpnsenseAPIError):
|
||||
ok = False
|
||||
return {"key": _probe_key(probe), "ok": ok}
|
||||
|
||||
# Concurrently (diagnostics_service caps it at a few at a time per process), results in plan order.
|
||||
return list(await asyncio.gather(*(one(p) for p in probes[:MAX_PROBES])))
|
||||
finally:
|
||||
if token is not None:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _check_probes(before: list | None, now: list | None) -> dict:
|
||||
if now is None:
|
||||
return _r("fail", "unreadable")
|
||||
was = {p["key"]: p["ok"] for p in (before or [])}
|
||||
regressed = [p["key"] for p in now if not p["ok"] and was.get(p["key"], True)]
|
||||
still = [p["key"] for p in now if not p["ok"] and was.get(p["key"]) is False]
|
||||
if regressed:
|
||||
return _r("fail", "probe_regressed", probes=", ".join(regressed))
|
||||
if still:
|
||||
return _r("warn", "probe_was_failing", probes=", ".join(still))
|
||||
return _r("ok", "ok", count=len(now))
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost, probes: list[dict] | None = None) -> dict:
|
||||
info, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
||||
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
||||
@@ -73,6 +129,7 @@ async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
"gateways": gateways,
|
||||
"services": services,
|
||||
"compliance_score": score,
|
||||
"probes": await run_probes(host, probes) if probes else None,
|
||||
}
|
||||
|
||||
|
||||
@@ -137,7 +194,7 @@ async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dic
|
||||
|
||||
|
||||
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
||||
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
|
||||
checks: dict | None = None, compliance_mode: str = "warn", probes: list[dict] | None = None) -> dict:
|
||||
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
||||
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
||||
for name in ALWAYS_ON:
|
||||
@@ -190,6 +247,10 @@ async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict,
|
||||
if enabled["services"]:
|
||||
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
||||
|
||||
if probes:
|
||||
# A probe new since the baseline (plan edited mid-run) has nothing to compare with: treated as "worked before".
|
||||
row["probes"] = _check_probes(before.get("probes"), await run_probes(host, probes) if reachable else None)
|
||||
|
||||
blockers = await _shared_change_blockers(db, fleets[hid])
|
||||
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
||||
|
||||
|
||||
@@ -110,6 +110,8 @@ def definition(plan: RolloutPlan) -> dict:
|
||||
return {
|
||||
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
# Only when set: an activated plan from before 1.32 must keep its fingerprint (else its automatic start stops).
|
||||
**({"gate_probes": list(plan.gate_probes)} if plan.gate_probes else {}),
|
||||
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
||||
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
||||
}
|
||||
|
||||
@@ -75,6 +75,7 @@ Item and operation ``message`` hold an ErrorCode name (``errors.<CODE>`` in the
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
@@ -96,6 +97,7 @@ from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeplo
|
||||
from app.models.rule_cleanup import OPEN_ITEM_STATES, RuleCleanupItem, RuleCleanupOperation
|
||||
from app.models.rule_history import LiveRuleState, RuleChangeEvent
|
||||
from app.services import audit_service
|
||||
from app.services.deploy_flow import unsettled
|
||||
from app.services import opnsense_client as client
|
||||
from app.services import rule_history_service as rhs
|
||||
from app.services import rule_hit_service
|
||||
@@ -236,7 +238,7 @@ async def _ensure_free(db: AsyncSession, host: OpnsenseHost, fleet: list[Opnsens
|
||||
await ensure_no_maintenance(db, [host], cleanup=False)
|
||||
ids = [h.id for h in fleet]
|
||||
for model in (*DEPLOYMENTS, PolicyHierarchyRuleDeployment):
|
||||
if await db.scalar(select(model.id).where(model.host_id.in_(ids), model.status.in_(("pending", "revert_failed"))).limit(1)):
|
||||
if await db.scalar(select(model.id).where(model.host_id.in_(ids), unsettled(model)).limit(1)):
|
||||
raise api_error(409, ErrorCode.BEREINIGUNG_OFFENE_OPERATION, reason="deployment")
|
||||
if await has_open_cleanup(db, ids, exclude_op=exclude_op):
|
||||
raise api_error(409, ErrorCode.BEREINIGUNG_OFFENE_OPERATION, reason="cleanup")
|
||||
@@ -255,6 +257,7 @@ class Node:
|
||||
managed: dict[str, dict]
|
||||
hasync: dict | None
|
||||
states: dict = field(default_factory=dict)
|
||||
own_source: str | None = None # where this firewall sees OPNexus's API requests come from (B10)
|
||||
|
||||
|
||||
def _hasync_brief(data) -> dict | None:
|
||||
@@ -312,12 +315,17 @@ async def _load_nodes(db: AsyncSession, host: OpnsenseHost, *, unused_days: int,
|
||||
except OpnsenseAPIError:
|
||||
hasync = None # protection.infrastructure_reason then assumes the worst on a cluster node
|
||||
states = {s.rule_uuid: s for s in (await db.scalars(select(LiveRuleState).where(LiveRuleState.host_id == member.id))).all()}
|
||||
try:
|
||||
own_source = await client.own_source_address(member)
|
||||
except OpnsenseAPIError as exc:
|
||||
own_source = None # shown as "unknown" in the preview; the configured sources still protect
|
||||
logger.info("Bereinigung: OPNexus-Adresse auf %s nicht ermittelbar: %s", member.name, exc)
|
||||
nodes.append(Node(
|
||||
host=member, role=roles.get(member.id), ctx=ctx,
|
||||
rules={r.uuid.lower(): r for r in rules if r.uuid and not r.uuid.startswith("pf@")},
|
||||
rows={str(r.get("uuid")).lower(): r for r in ctx.rows if r.get("uuid")},
|
||||
candidates={str(c["uuid"]).lower(): c for c in view["candidates"] if c.get("uuid")},
|
||||
managed=managed, hasync=hasync, states=states,
|
||||
managed=managed, hasync=hasync, states=states, own_source=own_source,
|
||||
))
|
||||
return _write_order(nodes)
|
||||
|
||||
@@ -394,6 +402,18 @@ async def _uuid_map(db: AsyncSession, nodes: list[Node], kind: str, uuids: list[
|
||||
return mapping
|
||||
|
||||
|
||||
def _with_own_source(sources: list, own: str | None) -> tuple[list, dict]:
|
||||
"""The configured management sources plus OPNexus's own address on this firewall (B10): a rule that only an
|
||||
admin's workstation was listed for could otherwise cut OPNexus off, and then the automatic revert cannot reach the
|
||||
firewall any more. Unknown address: the configured sources alone, and the preview says so."""
|
||||
if not own:
|
||||
return sources, {"address": None, "status": "unknown"}
|
||||
addr = ipaddress.ip_address(own)
|
||||
if any(addr.version == net.version and addr in net for net in sources):
|
||||
return sources, {"address": own, "status": "configured"}
|
||||
return [*sources, ipaddress.ip_network(own)], {"address": own, "status": "added"}
|
||||
|
||||
|
||||
async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[str], unused_days: int,
|
||||
acknowledge: set[str], limits: Limits, now: datetime) -> dict:
|
||||
"""What the operation would do, recomputed from the live state, or a refusal (api_error)."""
|
||||
@@ -408,6 +428,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
|
||||
missing_ack: list[str] = []
|
||||
for node in nodes:
|
||||
targets, ports = protection.management_targets(node.host.base_url, node.ctx, limits.ports)
|
||||
node_sources, opnexus_source = _with_own_source(sources, node.own_source)
|
||||
node_uuids = [mapping[u][node.host.id] for u in uuids]
|
||||
rows = node.rows
|
||||
covers = {c.get("by", {}).get("uuid", "").lower() for u in node_uuids
|
||||
@@ -452,7 +473,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
|
||||
if infra:
|
||||
refuse(infra)
|
||||
continue
|
||||
check = protection.management_check(rule, node.ctx, sources, targets, ports)
|
||||
check = protection.management_check(rule, node.ctx, node_sources, targets, ports)
|
||||
item["protection"] = check
|
||||
if check["result"] != "false":
|
||||
refuse("management")
|
||||
@@ -487,7 +508,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
|
||||
if not eligible:
|
||||
refuse("not_candidate")
|
||||
continue
|
||||
item["protection"] = protection.management_check(rule, node.ctx, sources, targets, ports)
|
||||
item["protection"] = protection.management_check(rule, node.ctx, node_sources, targets, ports)
|
||||
# Switching a block/reject rule back ON can cut off the management path; the auto-revert then only
|
||||
# works if OPNexus itself still reaches the firewall. Unless the rule is proven not to touch
|
||||
# management traffic: an explicit acknowledgement per rule (review finding 7).
|
||||
@@ -506,7 +527,7 @@ async def _plan(db: AsyncSession, nodes: list[Node], *, kind: str, uuids: list[s
|
||||
continue
|
||||
items.append(item)
|
||||
plan_nodes.append({"host_id": str(node.host.id), "host_name": node.host.name, "base_url": node.host.base_url,
|
||||
"role": node.role, "items": items})
|
||||
"role": node.role, "opnexus_source": opnexus_source, "items": items})
|
||||
if problems:
|
||||
first = problems[0]
|
||||
if first.get("code") == "wait":
|
||||
@@ -677,7 +698,8 @@ def redact_plan(plan: dict | None) -> dict | None:
|
||||
return plan
|
||||
out = {k: v for k, v in plan.items() if k not in ("sources", "ports")}
|
||||
out["sources_configured"] = bool(plan.get("sources"))
|
||||
out["nodes"] = [{**{k: v for k, v in n.items() if k != "base_url"},
|
||||
out["nodes"] = [{**{k: v for k, v in n.items() if k not in ("base_url", "opnexus_source")},
|
||||
"opnexus_source": {"status": (n.get("opnexus_source") or {}).get("status")},
|
||||
"items": [{**i, "protection": _brief_protection(i.get("protection"))} for i in n.get("items", [])]}
|
||||
for n in plan.get("nodes", [])]
|
||||
return out
|
||||
@@ -963,7 +985,8 @@ async def _run_switch(db: AsyncSession, op: RuleCleanupOperation, nodes: list[No
|
||||
# Also after an error: what is switched in the configuration must not wait for a foreign apply.
|
||||
await _sample(db, node.host, strict=False)
|
||||
try:
|
||||
await client.apply(node.host)
|
||||
with client.counters_already_sampled(): # _sample just ran: no second probe
|
||||
await client.apply(node.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
apply_failed = True
|
||||
op.message = f"{ErrorCode.BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN.value}: {node.host.name}: {exc}"[:1000]
|
||||
@@ -1110,7 +1133,8 @@ async def _revert(db: AsyncSession, op: RuleCleanupOperation, *, allowed: tuple,
|
||||
if touched and member is not None:
|
||||
await _sample(db, member, strict=False)
|
||||
try:
|
||||
await client.apply(member)
|
||||
with client.counters_already_sampled(): # _sample just ran: no second probe
|
||||
await client.apply(member)
|
||||
except OpnsenseAPIError as exc:
|
||||
for item in node_items:
|
||||
if item.state == "reverted":
|
||||
@@ -1322,7 +1346,8 @@ async def _run_delete(db: AsyncSession, op: RuleCleanupOperation, nodes: list[No
|
||||
# and leaves no "pending changes" in the GUI. It still zeroes the counters: sample first.
|
||||
await _sample(db, node.host, strict=False)
|
||||
try:
|
||||
await client.apply(node.host)
|
||||
with client.counters_already_sampled(): # _sample just ran: no second probe
|
||||
await client.apply(node.host)
|
||||
except OpnsenseAPIError as exc:
|
||||
op.message = f"{ErrorCode.BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN.value}: {node.host.name}: {exc}"[:1000]
|
||||
stop = True
|
||||
@@ -1467,7 +1492,8 @@ async def restore_item(db: AsyncSession, host: OpnsenseHost, item_id, *, actor:
|
||||
warnings.append(f"{member.name}:{item.rule_uuid}")
|
||||
await _sample(db, member, strict=False)
|
||||
try:
|
||||
await client.apply(member)
|
||||
with client.counters_already_sampled(): # _sample just ran: no second probe
|
||||
await client.apply(member)
|
||||
except OpnsenseAPIError as exc: # the rule is disabled: pf does not change; the GUI shows pending changes
|
||||
op.message = f"{ErrorCode.BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN.value}: {member.name}: {exc}"[:1000]
|
||||
item.state, item.changed_at = "restored", _now()
|
||||
|
||||
@@ -142,6 +142,15 @@ async def apply_sample(db: AsyncSession, host: OpnsenseHost, counters: dict[str,
|
||||
return len(rules)
|
||||
|
||||
|
||||
async def sample_before_reload(host: OpnsenseHost) -> None:
|
||||
"""Called by opnsense_client right before any request that makes the firewall reload pf (B11): read the counters
|
||||
while they still hold what was hit since the last sample. Own session, so the writer's transaction stays as it is."""
|
||||
from app.database import SessionLocal
|
||||
|
||||
async with SessionLocal() as db:
|
||||
await sample_host(db, host)
|
||||
|
||||
|
||||
async def restart_window(db: AsyncSession, host_id, rule_uuids, now: datetime | None = None) -> int:
|
||||
"""The rules were edited, enabled or created: what OPNexus saw before does not describe them any more, so the
|
||||
observation window for "unused" starts over (an earlier last hit is kept)."""
|
||||
|
||||
@@ -40,7 +40,7 @@ from app.models.host_operation import HostOperation
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from contextlib import contextmanager
|
||||
|
||||
from app.services import audit_service, backup_store, opnsense_client as client
|
||||
from app.services import audit_service, backup_store, opnsense_client as client, snapshot_service
|
||||
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
|
||||
@@ -202,6 +202,17 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
await db.commit()
|
||||
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
|
||||
return
|
||||
if sched.action in _BACKUP_BEFORE:
|
||||
# Way back (1.32, B5): a boot environment of the running system, after every other check passed.
|
||||
# Fail closed like the backup: no rollback point on a ZFS firewall, no update.
|
||||
try:
|
||||
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": f"Rückweg-Punkt (Boot-Environment) vor dem Update nicht angelegt: {str(exc)[:300]}"}
|
||||
await db.commit()
|
||||
logger.warning("Geplante Wartung %s: Rückweg-Punkt fehlgeschlagen", sched.id)
|
||||
return
|
||||
|
||||
boot_before = await _boot_time(host) if sched.auto_verify else None
|
||||
op = HostOperation(
|
||||
@@ -302,6 +313,9 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||
state = await inspect_host(db, host, skip_role_checks=True)
|
||||
blocked = list(dict.fromkeys(blocked + state["actions"][sched.action]))
|
||||
if not blocked and sched.action in _BACKUP_BEFORE:
|
||||
# Way back (1.32, B5) while nothing has been toggled yet; a failure ends here like a failed backup.
|
||||
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
await fail(f"Vorbereitung der CARP-Übergabe fehlgeschlagen, nichts wurde verändert: {str(exc)[:300]}")
|
||||
return
|
||||
|
||||
@@ -1,17 +1,22 @@
|
||||
"""ZFS snapshots (boot environments) of one firewall: list, create, delete.
|
||||
|
||||
Snapshots are a safety net before bigger changes, not a rollback button:
|
||||
activating one needs a reboot and discards everything since, so that action is
|
||||
deliberately NOT offered here. Deleting refuses the active/next-boot snapshot
|
||||
Snapshots are a safety net before bigger changes. Activating an arbitrary one is
|
||||
deliberately NOT offered (it needs a reboot and discards everything since); the
|
||||
only way back is the guided firmware rollback to a point OPNexus itself created
|
||||
right before an update (see "Rollback points" below, maintenance action
|
||||
`firmware_rollback`). Deleting refuses the active/next-boot snapshot
|
||||
(the firewall refuses too; we say so up front). Every step is audited.
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import audit_service, opnsense_client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$")
|
||||
|
||||
|
||||
@@ -23,20 +28,121 @@ def _row(row: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
async def list_snapshots(host: OpnsenseHost) -> dict:
|
||||
async def supported(host: OpnsenseHost) -> bool:
|
||||
try:
|
||||
supported = (await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported")
|
||||
return bool((await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported"))
|
||||
except OpnsenseAPIError as exc:
|
||||
if "404" in str(exc):
|
||||
return {"supported": False, "rows": []}
|
||||
if "404" in str(exc): # OPNsense without the snapshots API
|
||||
return False
|
||||
raise
|
||||
if not supported:
|
||||
|
||||
|
||||
async def list_snapshots(host: OpnsenseHost) -> dict:
|
||||
if not await supported(host):
|
||||
return {"supported": False, "rows": []}
|
||||
return {"supported": True, "rows": [_row(r) for r in await opnsense_client.list_snapshots(host)]}
|
||||
|
||||
|
||||
# ── Rollback points (1.32, B5) ──────────────────────────────────────────────
|
||||
# Before every firmware update/upgrade OPNexus creates a boot environment of the running system. Going back to it
|
||||
# (activate + reboot) restores the old firmware AND the old configuration of that moment: everything changed on the
|
||||
# firewall since is gone. Only these points are offered for going back, never boot environments made by hand.
|
||||
ROLLBACK_PREFIX = "opnexus-pre-"
|
||||
KEEP_ROLLBACK_POINTS = 2
|
||||
_POINT = re.compile(r"^opnexus-pre-(?P<version>[A-Za-z0-9._-]+)-(?P<stamp>\d{8}-\d{6})$")
|
||||
|
||||
|
||||
def _is_active(row: dict) -> bool:
|
||||
# N = active now, R = active after reboot.
|
||||
return str(row.get("active", "")).strip() not in ("", "-")
|
||||
|
||||
|
||||
def rollback_points(rows: list[dict]) -> list[dict]:
|
||||
"""OPNexus rollback points, newest first: {uuid, name, version, stamp, active, created}."""
|
||||
out = []
|
||||
for row in rows:
|
||||
match = _POINT.match(str(row.get("name", "")))
|
||||
if match:
|
||||
out.append({"uuid": str(row.get("uuid", "")), "name": match.group(0), "version": match.group("version"),
|
||||
"stamp": match.group("stamp"), "active": str(row.get("active", "")).strip(),
|
||||
"created": str(row.get("created_str", ""))})
|
||||
return sorted(out, key=lambda p: p["stamp"], reverse=True)
|
||||
|
||||
|
||||
async def create_rollback_point(host: OpnsenseHost, version: str | None) -> dict | None:
|
||||
"""Boot environment right before a firmware action. None when the firewall has no ZFS snapshots (UFS install):
|
||||
the update may still run, without a way back. Raises OpnsenseAPIError when creating it fails (callers abort the
|
||||
update: fail closed). Older OPNexus points beyond KEEP_ROLLBACK_POINTS are removed afterwards."""
|
||||
if not await supported(host):
|
||||
return None
|
||||
clean_version = re.sub(r"[^A-Za-z0-9._-]", "-", version or "unbekannt")[:30] or "unbekannt"
|
||||
name = f"{ROLLBACK_PREFIX}{clean_version}-{datetime.now(timezone.utc):%Y%m%d-%H%M%S}"
|
||||
audit = {"kind": "snapshot", "action": "snapshot.rollback_point", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
|
||||
await audit_service.persist(**audit, status="started", message=None, details={"version": version})
|
||||
try:
|
||||
await opnsense_client.create_snapshot(host, name, f"OPNexus: Rückweg-Punkt vor dem Firmware-Update von {version}")
|
||||
rows = await opnsense_client.list_snapshots(host)
|
||||
except OpnsenseAPIError as exc:
|
||||
await audit_service.persist(**audit, status="rejected", message=str(exc)[:500], details={})
|
||||
raise
|
||||
point = next((p for p in rollback_points(rows) if p["name"] == name), None)
|
||||
if point is None:
|
||||
await audit_service.persist(**audit, status="unverified", message=None, details={})
|
||||
raise OpnsenseAPIError(f"Rückweg-Punkt {name} nach dem Anlegen nicht in der Liste")
|
||||
await audit_service.persist(**audit, status="applied", message=None, details={})
|
||||
for old in [p for p in rollback_points(rows) if not _is_active(p)][KEEP_ROLLBACK_POINTS:]:
|
||||
try: # housekeeping only: a point that cannot be removed stays, the update goes on
|
||||
await opnsense_client.delete_snapshot(host, old["uuid"])
|
||||
await audit_service.persist(kind="snapshot", action="snapshot.del", entity_name=old["name"], host_id=str(host.id),
|
||||
host_name=host.name, status="applied", message="Älterer Rückweg-Punkt aufgeräumt", details={})
|
||||
except OpnsenseAPIError as exc:
|
||||
logger.warning("Rückweg-Punkt %s auf %s nicht gelöscht: %s", old["name"], host.name, exc)
|
||||
return {"uuid": point["uuid"], "name": point["name"], "version": version}
|
||||
|
||||
|
||||
async def rollback_state(host: OpnsenseHost) -> dict:
|
||||
"""What going back would do: the newest OPNexus point that is not running now, and the boot environment that is."""
|
||||
if not await supported(host):
|
||||
return {"supported": False, "target": None, "current": None}
|
||||
rows = await opnsense_client.list_snapshots(host)
|
||||
current = next((r for r in rows if "N" in str(r.get("active", ""))), None) # running now (R alone = next start)
|
||||
target = next((p for p in rollback_points(rows) if "N" not in p["active"]), None)
|
||||
return {"supported": True, "target": target,
|
||||
"current": {"uuid": str(current.get("uuid", "")), "name": str(current.get("name", ""))} if current else None}
|
||||
|
||||
|
||||
async def activate_and_reboot(host: OpnsenseHost, rollback: dict) -> dict:
|
||||
"""Activate the rollback point, check it is marked for the next start, reboot. If the reboot cannot be handed over,
|
||||
the previous boot environment is activated again: an activated old one must never wait for the next unplanned
|
||||
reboot."""
|
||||
target, current = rollback["target"], rollback["current"]
|
||||
|
||||
async def undo() -> None:
|
||||
if current:
|
||||
try:
|
||||
await opnsense_client.activate_snapshot(host, current["uuid"])
|
||||
except OpnsenseAPIError:
|
||||
logger.warning("Rückweg auf %s: vorheriges Boot-Environment nicht reaktiviert", host.name)
|
||||
|
||||
await opnsense_client.activate_snapshot(host, target["uuid"])
|
||||
try:
|
||||
rows = await opnsense_client.list_snapshots(host)
|
||||
except OpnsenseAPIError:
|
||||
await undo() # cannot confirm the mark: never leave it waiting for an unplanned reboot
|
||||
raise
|
||||
marked = next((r for r in rows if str(r.get("uuid", "")) == target["uuid"]), None)
|
||||
if marked is None or "R" not in str(marked.get("active", "")):
|
||||
await undo()
|
||||
raise OpnsenseAPIError(f"Boot-Environment {target['name']} ist nicht für den nächsten Start vorgemerkt")
|
||||
try:
|
||||
return await opnsense_client.reboot_host(host)
|
||||
except OpnsenseAPIError:
|
||||
await undo()
|
||||
raise
|
||||
|
||||
|
||||
async def create(host: OpnsenseHost, name: str, note: str) -> dict:
|
||||
if not NAME.match(name):
|
||||
if not NAME.match(name) or name.startswith(ROLLBACK_PREFIX): # that prefix marks OPNexus' own rollback points
|
||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field="name")
|
||||
audit = {"kind": "snapshot", "action": "snapshot.add", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
|
||||
await audit_service.persist(**audit, status="started", message=None, details={"note": note[:200]})
|
||||
|
||||
@@ -211,6 +211,8 @@ from app.models.wireguard_tunnel import WireguardTunnel
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
from app.services import opnsense_client, wireguard_key_store
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.deploy_flow import checkpoint
|
||||
from app.services.deploy_lease import VPN_INTERRUPTED, resource_lease
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
@@ -274,7 +276,7 @@ def build_payloads(tunnel: WireguardTunnel) -> dict:
|
||||
}
|
||||
|
||||
|
||||
async def _write_config(host: OpnsenseHost, payloads: dict, deployment: WireguardTunnelDeployment) -> None:
|
||||
async def _write_config(db: AsyncSession, host: OpnsenseHost, payloads: dict, deployment: WireguardTunnelDeployment) -> None:
|
||||
"""Writes the Server then the Peer, in that dependency order (the
|
||||
Peer's payload needs the Server's UUID for the `servers` field — see
|
||||
module docstring), persisting each UUID onto `deployment` as soon as
|
||||
@@ -288,6 +290,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: Wireguar
|
||||
deployment.opnsense_server_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_server_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Server-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
# Overlay the deploy-time-known Server UUID onto a shallow copy, never
|
||||
# mutating payloads["peer"] itself — the same reasoning ipsec_deploy_
|
||||
@@ -303,6 +306,7 @@ async def _write_config(host: OpnsenseHost, payloads: dict, deployment: Wireguar
|
||||
deployment.opnsense_peer_uuid = result.get("uuid") if isinstance(result, dict) else None
|
||||
if not deployment.opnsense_peer_uuid:
|
||||
raise OpnsenseAPIError("Firewall hat keine Peer-UUID zurückgegeben; Stand manuell prüfen")
|
||||
await checkpoint(db) # the UUID is durable before the next remote call
|
||||
|
||||
|
||||
async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: OpnsenseHost) -> WireguardTunnelDeployment:
|
||||
@@ -311,7 +315,7 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
|
||||
existing = await db.scalar(
|
||||
select(WireguardTunnelDeployment).where(
|
||||
WireguardTunnelDeployment.tunnel_id == tunnel.id, WireguardTunnelDeployment.host_id == host.id
|
||||
)
|
||||
).execution_options(populate_existing=True) # fresh: another request may have just changed the row
|
||||
)
|
||||
if existing and existing.status == "success" and existing.last_payload == comparable:
|
||||
# Nothing actually changed on this host — re-affirm without
|
||||
@@ -322,8 +326,15 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
|
||||
|
||||
deployment = existing or WireguardTunnelDeployment(tunnel_id=tunnel.id, host_id=host.id, status="error")
|
||||
|
||||
# Durable intent BEFORE the first remote call: from here on a crash or cancellation leaves a row, and every UUID the
|
||||
# firewall hands out is committed at once (see _write_config), so a retry continues instead of duplicating.
|
||||
deployment.status, deployment.message, deployment.deployed_at = "error", VPN_INTERRUPTED, _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
await checkpoint(db)
|
||||
|
||||
try:
|
||||
await _write_config(host, payloads, deployment)
|
||||
await _write_config(db, host, payloads, deployment)
|
||||
# Without the global switch the config is stored but no interface
|
||||
# ever comes up (live-verified), so a "success" would be a lie.
|
||||
await opnsense_client.ensure_wireguard_enabled(host)
|
||||
@@ -331,8 +342,6 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
|
||||
deployment.status = "error"
|
||||
deployment.message = str(exc)
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
try:
|
||||
@@ -347,28 +356,30 @@ async def _deploy_to_host(db: AsyncSession, tunnel: WireguardTunnel, host: Opnse
|
||||
deployment.message = "Konfiguration wurde geschrieben, Dienst-Reload aber nicht bestätigt — Erreichbarkeit manuell prüfen"
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
deployment.status = "success"
|
||||
deployment.message = None
|
||||
deployment.last_payload = comparable
|
||||
deployment.deployed_at = _now()
|
||||
if not existing:
|
||||
db.add(deployment)
|
||||
return deployment
|
||||
|
||||
|
||||
async def deploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel, hosts: list[OpnsenseHost]) -> list[WireguardTunnelDeployment]:
|
||||
from app.services.host_maintenance_service import ensure_no_maintenance
|
||||
await ensure_no_maintenance(db, hosts)
|
||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||
await db.commit()
|
||||
async with resource_lease("wireguard", tunnel.id):
|
||||
deployments = [await _deploy_to_host(db, tunnel, host) for host in hosts]
|
||||
await db.commit()
|
||||
return deployments
|
||||
|
||||
|
||||
async def undeploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel) -> None:
|
||||
async with resource_lease("wireguard", tunnel.id):
|
||||
await _undeploy_tunnel(db, tunnel)
|
||||
|
||||
|
||||
async def _undeploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel) -> None:
|
||||
"""Explicit action, not gated by anything (there is no lockout state
|
||||
for this entity to gate on) — deletes the Peer before the Server (the
|
||||
dependency direction; deleting the Peer first also auto-unlinks it
|
||||
|
||||
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
|
||||
Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
"""
|
||||
|
||||
APP_VERSION = "1.31.0"
|
||||
APP_CODENAME = "Vulture"
|
||||
APP_VERSION = "1.33.0"
|
||||
APP_CODENAME = "Saker"
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Stabile Schlüssel (config_key) für Regeln und NAT-Objekte (1.32, Konfiguration als Code).
|
||||
|
||||
Revision ID: 0030
|
||||
Revises: 0029
|
||||
Create Date: 2026-10-23
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0030"
|
||||
down_revision = "0029"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
_TABLES = (
|
||||
("firewall_rules", "r"),
|
||||
("port_forward_rules", "pf"),
|
||||
("one_to_one_rules", "nat1"),
|
||||
("outbound_nat_rules", "out"),
|
||||
("policy_hierarchy_rules", "ph"),
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# IF NOT EXISTS / DO-Block: ein Altbestand ohne Alembic wird zuerst per create_all() angehoben und hat Spalte samt
|
||||
# Constraint dann schon.
|
||||
for table, prefix in _TABLES:
|
||||
op.execute(f"ALTER TABLE {table} ADD COLUMN IF NOT EXISTS config_key varchar(64)")
|
||||
op.execute(
|
||||
f"UPDATE {table} SET config_key = '{prefix}-' || substr(replace(id::text, '-', ''), 1, 12) "
|
||||
"WHERE config_key IS NULL"
|
||||
)
|
||||
op.execute(f"ALTER TABLE {table} ALTER COLUMN config_key SET NOT NULL")
|
||||
op.execute(
|
||||
f"""DO $$ BEGIN
|
||||
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '{table}_config_key_key') THEN
|
||||
ALTER TABLE {table} ADD CONSTRAINT {table}_config_key_key UNIQUE (config_key);
|
||||
END IF;
|
||||
END $$"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for table, _prefix in _TABLES:
|
||||
op.execute(f"ALTER TABLE {table} DROP CONSTRAINT IF EXISTS {table}_config_key_key")
|
||||
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS config_key")
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Rollout-Gate: Erreichbarkeitsproben von der Firewall aus (1.32, B5).
|
||||
|
||||
Revision ID: 0031
|
||||
Revises: 0030
|
||||
Create Date: 2026-10-23
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0031"
|
||||
down_revision = "0030"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# IF NOT EXISTS: a pre-Alembic install is bridged with create_all() first, which may already have the column.
|
||||
op.execute("ALTER TABLE rollout_plans ADD COLUMN IF NOT EXISTS gate_probes json")
|
||||
op.execute("UPDATE rollout_plans SET gate_probes = '[]'::json WHERE gate_probes IS NULL")
|
||||
op.execute("ALTER TABLE rollout_plans ALTER COLUMN gate_probes SET NOT NULL")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE rollout_plans DROP COLUMN IF EXISTS gate_probes")
|
||||
@@ -12,5 +12,6 @@ pytest-asyncio==1.4.0
|
||||
respx==0.23.1
|
||||
cryptography==50.0.2
|
||||
alembic==1.20.0
|
||||
PyJWT==2.10.1
|
||||
PyJWT==2.15.1
|
||||
tzdata==2025.2
|
||||
PyYAML==6.0.3
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Seed data for the configuration-as-code tests: every object kind, hierarchy, placeholders, variables."""
|
||||
from app.database import SessionLocal
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
|
||||
|
||||
async def seed_everything() -> dict:
|
||||
"""Two hosts, a device group, a hierarchy (Organisation > Nord), and one object of every kind."""
|
||||
async with SessionLocal() as db:
|
||||
fw1 = OpnsenseHost(name="fw-filiale-1", base_url="https://secret-host.example:8443", api_key="AKIA-SECRET-KEY")
|
||||
fw2 = OpnsenseHost(name="fw-filiale-2", base_url="https://other.example", api_key="k2")
|
||||
group = DeviceGroup(name="Filialen", description="alle Filialen", hosts=[fw1, fw2])
|
||||
org = PolicyHierarchyGroup(name="Organisation", description="")
|
||||
db.add_all([fw1, fw2, group, org])
|
||||
await db.flush()
|
||||
nord = PolicyHierarchyGroup(name="Nord", description="Region Nord", parent_id=org.id)
|
||||
db.add(nord)
|
||||
await db.flush()
|
||||
fw1.policy_hierarchy_group_id = nord.id
|
||||
db.add_all([
|
||||
HostVariable(host_id=fw1.id, name="LAN_NET", value="10.20.30.0/24"),
|
||||
HostVariable(host_id=fw1.id, name="SITE", value="Filiale Nord"),
|
||||
FirewallAlias(name="WEB_SERVERS", type="host", description="DMZ", enabled=True,
|
||||
content="10.20.30.10\n203.0.113.20", groups=[group], policy_hierarchy_group_id=None),
|
||||
FirewallRule(config_key="allow-dns-lan", description="DNS aus {{SITE}}", interface="lan", direction="in",
|
||||
source_net="{{LAN_NET}}", destination_net="WEB_SERVERS", destination_port="53", protocol="TCP/UDP",
|
||||
groups=[group], targets=[]),
|
||||
FirewallRule(config_key="block-all", description="alles sperren", interface="wan", direction="in", action="block",
|
||||
log=True, targets=[fw2], groups=[]),
|
||||
PortForwardRule(config_key="pf-web", description="Web DMZ", interface="wan", destination_net="wanip",
|
||||
destination_port="443", target="10.20.30.10", local_port="443", targets=[fw1], groups=[]),
|
||||
OneToOneRule(config_key="nat1-mail", description="Mail", interface="wan", external="203.0.113.25",
|
||||
source_net="10.20.30.25", destination_net="any", groups=[group], targets=[]),
|
||||
OutboundNatRule(config_key="out-lan", description="LAN raus", interface="wan", protocol="any",
|
||||
source_net="{{LAN_NET}}", destination_net="any", target="wanip", groups=[group], targets=[],
|
||||
policy_hierarchy_group_id=None),
|
||||
PolicyHierarchyRule(config_key="ph-block-bogons", hierarchy_group_id=org.id, rulebase="pre", sequence=0,
|
||||
description="Bogons", interface="wan", direction="in", action="block", source_net="bogons"),
|
||||
PolicyHierarchyRule(config_key="ph-allow-mgmt", hierarchy_group_id=org.id, rulebase="pre", sequence=1,
|
||||
description="Mgmt", interface="lan", direction="in", action="pass"),
|
||||
PolicyHierarchyRule(config_key="ph-post-deny", hierarchy_group_id=nord.id, rulebase="post", sequence=0,
|
||||
description="Standard sperren", interface="lan", direction="in", action="block"),
|
||||
])
|
||||
await db.commit()
|
||||
return {"fw1": "fw-filiale-1", "fw2": "fw-filiale-2"}
|
||||
@@ -7,3 +7,16 @@ async def deploy_approved(client, deploy_url: str):
|
||||
preview = await client.get(deploy_url.rsplit("/deploy", 1)[0] + "/preview")
|
||||
assert preview.status_code == 200, preview.text
|
||||
return await client.post(deploy_url, json={"preview_token": preview.json()["token"]})
|
||||
|
||||
|
||||
async def delete_host(client, host_id):
|
||||
"""DELETE /api/hosts/{id} with the confirmation the API now requires (T4): the host's own name, read from the DB
|
||||
so the helper does not touch the firewall mocks."""
|
||||
import uuid as _uuid
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.host import OpnsenseHost
|
||||
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, _uuid.UUID(str(host_id)))
|
||||
return await client.delete(f"/api/hosts/{host_id}", params={"confirm_name": host.name if host else ""})
|
||||
@@ -161,6 +161,12 @@ class FakeFirewall:
|
||||
return httpx.Response(200, json={"rows": [{"name": "tbl", "type": "urltable", "content": "https://203.0.113.9/list"}]})
|
||||
if method == "GET" and path == "/firewall/d_nat/searchRule":
|
||||
return httpx.Response(200, json={"rows": []})
|
||||
# B10: OPNexus asks for its own source address. respx has no real socket, so the state cannot be matched to
|
||||
# this connection: the address stays unknown unless a test sets ``own_source`` (then the fake answers it).
|
||||
if method == "GET" and path == "/diagnostics/firewall/pf_states":
|
||||
return httpx.Response(200, json={"current": 5, "limit": "100000"})
|
||||
if method == "POST" and path == "/diagnostics/firewall/query_states":
|
||||
return httpx.Response(200, json={"rows": [], "rowCount": 0, "total": 0, "current": 1})
|
||||
if method == "GET" and path == "/core/hasync/get":
|
||||
sel = {"": {"value": "Disabled", "selected": 1 if not self.pfsync else 0},
|
||||
"lan": {"value": "LAN", "selected": 1 if self.pfsync == "lan" else 0}}
|
||||
|
||||
@@ -326,3 +326,30 @@ async def test_poll_certificate_status_updates_remote_fields(client, respx_mock)
|
||||
assert body["remote_status_code"] == 200
|
||||
assert body["remote_status_last_update"] is not None
|
||||
assert body["remote_last_update"] is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_acme_target_host_is_fixed_once_deployed_or_used_by_requests(client, respx_mock):
|
||||
"""Audit AUD-03: the same re-pointing problem for ACME accounts and certificate requests."""
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
a = await _create_host(client, "acme-a", "https://acme-a.test")
|
||||
b = await _create_host(client, "acme-b", "https://acme-b.test")
|
||||
acct = (await client.post("/api/acme/accounts", json=_account_payload(a["id"]))).json()
|
||||
free = await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(b["id"], expected_updated_at=acct["updated_at"]))
|
||||
assert free.status_code == 200 and free.json()["host_id"] == b["id"] # draft, no requests: free to move
|
||||
acct = (await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(a["id"], expected_updated_at=free.json()["updated_at"]))).json()
|
||||
|
||||
req = (await client.post("/api/acme/certificates", json=_cert_request_payload(a["id"], acct["id"]))).json()
|
||||
used = await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(b["id"], expected_updated_at=acct["updated_at"]))
|
||||
assert used.status_code == 409 and used.json()["detail"]["code"] == "ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET"
|
||||
|
||||
respx_mock.post(url__regex=r".*/acmeclient/accounts/add$").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "acct-uuid-1"}))
|
||||
pushed_account = (await client.post(f"/api/acme/accounts/{acct['id']}/push")).json()
|
||||
fixed = await client.put(f"/api/acme/accounts/{acct['id']}", json=_account_payload(b["id"], expected_updated_at=pushed_account["updated_at"]))
|
||||
assert fixed.status_code == 409 and fixed.json()["detail"]["code"] == "ZIEL_HOST_NACH_AUSROLLEN_FEST"
|
||||
|
||||
respx_mock.post(url__regex=r".*/acmeclient/certificates/add$").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "req-uuid-1"}))
|
||||
pushed_request = (await client.post(f"/api/acme/certificates/{req['id']}/push")).json()
|
||||
assert pushed_request["status"] == "success"
|
||||
moved = await client.put(f"/api/acme/certificates/{req['id']}", json=_cert_request_payload(b["id"], acct["id"], expected_updated_at=pushed_request["updated_at"]))
|
||||
assert moved.status_code == 409 and moved.json()["detail"]["code"] == "ZIEL_HOST_NACH_AUSROLLEN_FEST"
|
||||
@@ -18,6 +18,7 @@ from tests.test_monitoring_alerts import (
|
||||
)
|
||||
from tests.test_notifications import _create_host
|
||||
from tests.test_resource_monitor import _mock_resource_endpoints, _mock_vip
|
||||
from tests.helpers import delete_host
|
||||
|
||||
|
||||
async def _host_row(host_id: str) -> OpnsenseHost:
|
||||
@@ -413,5 +414,5 @@ async def test_deleting_a_host_removes_its_gateway_mutes(client):
|
||||
gone = await _create_host(client, "fw-gone", "https://fwgone.test")
|
||||
for h in (keep, gone):
|
||||
assert (await client.put(f"/api/hosts/{h['id']}/gateways/WAN_GW/alert", json={"enabled": False})).status_code == 200
|
||||
assert (await client.delete(f"/api/hosts/{gone['id']}")).status_code == 204
|
||||
assert (await delete_host(client, gone['id'])).status_code == 204
|
||||
assert (await client.get("/api/notifications/settings")).json()["gateway_alert_muted"] == [f"{keep['id']}:WAN_GW"]
|
||||
@@ -7,6 +7,7 @@ from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from tests.test_rules import _create_host
|
||||
from tests.test_port_forward import _pf_payload, _mock_pf_push
|
||||
from tests.helpers import delete_host
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audit_retains_payload_actor_after_rule_and_host_deletion(client,respx_mock):
|
||||
@@ -18,7 +19,7 @@ async def test_audit_retains_payload_actor_after_rule_and_host_deletion(client,r
|
||||
assert (await client.post(f"/api/port-forward/{r['id']}/confirm")).status_code==200
|
||||
respx_mock.post('https://fw1.test/api/firewall/d_nat/delRule/pf-uuid-1').respond(200,json={'result':'deleted'})
|
||||
assert (await client.delete(f"/api/port-forward/{r['id']}")).status_code==204
|
||||
assert (await client.delete(f"/api/hosts/{h['id']}")).status_code==204
|
||||
assert (await delete_host(client, h['id'])).status_code==204
|
||||
data=(await client.get('/api/audit',params={'entity_id':r['id'],'limit':200})).json()
|
||||
state=[e for e in data['items'] if e['host_id']==h['id']]
|
||||
assert {e['status'] for e in state} == {'pending','success','removed'}
|
||||
|
||||
@@ -166,3 +166,35 @@ async def test_delete_in_use_certificate_returns_stable_error_code(client, respx
|
||||
assert detail["code"] == "UPSTREAM_REJECTED"
|
||||
assert "Item in use by" in detail["params"]["reason"]
|
||||
assert "roadwarrior" in detail["params"]["reason"]
|
||||
|
||||
|
||||
async def test_a_ca_cannot_be_requested_through_the_managed_certificate_api(client, respx_mock):
|
||||
"""T3 (1.31.0 test run): cert_type=v3_ca created a certificate with the CA extension, not a usable CA."""
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = await _create_host(client, "fw-ca", "https://fw-ca.test")
|
||||
resp = await client.post("/api/certificates/managed", json=_cert_payload(host["id"], cert_type="v3_ca"))
|
||||
assert resp.status_code == 422
|
||||
for allowed in ("usr_cert", "server_cert", "combined_server_client"):
|
||||
assert (await client.post("/api/certificates/managed", json=_cert_payload(host["id"], cert_type=allowed))).status_code == 201
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_target_host_of_a_deployed_certificate_is_fixed_but_free_before_the_first_push(client, respx_mock):
|
||||
"""Audit AUD-03: re-pointing a deployed certificate reported success for a host that never received it."""
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
a = await _create_host(client, "fix-a", "https://fix-a.test")
|
||||
b = await _create_host(client, "fix-b", "https://fix-b.test")
|
||||
cert = (await client.post("/api/certificates/managed", json=_cert_payload(a["id"]))).json()
|
||||
moved = await client.put(f"/api/certificates/managed/{cert['id']}", json=_cert_payload(b["id"], expected_updated_at=cert["updated_at"]))
|
||||
assert moved.status_code == 200 and moved.json()["host_id"] == b["id"] # never pushed: still free to move
|
||||
|
||||
add = respx_mock.post("https://fix-b.test/api/trust/cert/add").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "uuid-on-b"}))
|
||||
pushed = (await client.post(f"/api/certificates/managed/{cert['id']}/push")).json()
|
||||
assert pushed["status"] == "success" and add.call_count == 1
|
||||
|
||||
calls = len(respx_mock.calls)
|
||||
refused = await client.put(f"/api/certificates/managed/{cert['id']}", json=_cert_payload(a["id"], expected_updated_at=pushed["updated_at"]))
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ZIEL_HOST_NACH_AUSROLLEN_FEST"
|
||||
assert len(respx_mock.calls) == calls # nothing was sent anywhere
|
||||
same_host = await client.put(f"/api/certificates/managed/{cert['id']}", json=_cert_payload(b["id"], description="renamed", expected_updated_at=pushed["updated_at"]))
|
||||
assert same_host.status_code == 200 # everything else stays editable
|
||||
@@ -0,0 +1,176 @@
|
||||
"""HTTP API of configuration as code (1.32 P7): rights matrix, audit, errors, limits."""
|
||||
import pytest
|
||||
import yaml
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.main import app
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.rule import FirewallRule
|
||||
from app.services import api_token_service
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
PW = "config-code-password-1"
|
||||
|
||||
|
||||
async def _login(username: str, password: str) -> AsyncClient:
|
||||
ac = AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
|
||||
resp = await ac.post("/api/auth/login", json={"username": username, "password": password})
|
||||
assert resp.status_code == 200, resp.text
|
||||
return ac
|
||||
|
||||
|
||||
def _bearer(token: str) -> AsyncClient:
|
||||
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test", headers={"Authorization": f"Bearer {token}"})
|
||||
|
||||
|
||||
async def _token(client, scope: str) -> str:
|
||||
resp = await client.post("/api/tokens", json={"name": f"t-{scope}", "scope": scope, "current_password": settings.admin_password})
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()["token"]
|
||||
|
||||
|
||||
async def _changed_file(client) -> str:
|
||||
text = (await client.get("/api/config/export")).text
|
||||
data = yaml.safe_load(text)
|
||||
data["rules"][0]["description"] = "über die API"
|
||||
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scope,method,path,denied", [
|
||||
("read", "GET", "/api/config/export", False),
|
||||
("read", "POST", "/api/config/import/plan", False),
|
||||
("read", "POST", "/api/config/import/apply", True),
|
||||
("deploy", "POST", "/api/config/import/plan", False),
|
||||
("deploy", "POST", "/api/config/import/apply", True),
|
||||
("admin", "POST", "/api/config/import/apply", False),
|
||||
# the read-only exception is exactly this path, not its neighbours
|
||||
("read", "POST", "/api/config/import/plan/x", True),
|
||||
("read", "POST", "/api/config/import", True),
|
||||
])
|
||||
def test_token_scope_matrix(scope, method, path, denied):
|
||||
assert api_token_service.denies(scope, method, path) is denied
|
||||
|
||||
|
||||
def test_operators_may_plan_but_not_apply():
|
||||
assert not api_token_service.operator_denies("POST", "/api/config/import/plan")
|
||||
assert api_token_service.operator_denies("POST", "/api/config/import/apply")
|
||||
|
||||
|
||||
async def test_export_plan_apply_as_admin_with_audit(client):
|
||||
await seed_everything()
|
||||
resp = await client.get("/api/config/export")
|
||||
assert resp.status_code == 200 and resp.headers["content-type"].startswith("text/yaml")
|
||||
assert "attachment" in resp.headers["content-disposition"] and resp.headers["cache-control"] == "no-store"
|
||||
text = await _changed_file(client)
|
||||
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
|
||||
assert plan["summary"]["update"] == 1 and plan["applicable"] and len(plan["token"]) == 64
|
||||
assert [i["key"] for i in plan["items"]] == ["allow-dns-lan"] # unchanged objects are not listed
|
||||
resp = await client.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan["token"]})
|
||||
assert resp.status_code == 200 and resp.json()["summary"]["update"] == 1
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.execute(select(AuditEvent))).scalars().all()
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
|
||||
assert rule.description == "über die API"
|
||||
requests = [e for e in events if e.kind == "config" and e.action == "apply"]
|
||||
assert [e.status for e in requests] == ["requested", "accepted"] and requests[0].entity_id is None
|
||||
assert not any(e.action == "plan" for e in events) # planning is like a preview: not audited
|
||||
assert all("über die API" not in str(e.details) for e in events if e.kind == "config")
|
||||
assert any(e.kind == "rule" and e.action == "import_update" and e.actor == settings.admin_username for e in events)
|
||||
|
||||
|
||||
async def test_stale_blocked_invalid_and_too_large(client):
|
||||
await seed_everything()
|
||||
text = await _changed_file(client)
|
||||
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
|
||||
resp = await client.post("/api/config/import/apply", json={"yaml": text + "\n# anders\n", "plan_token": "0" * 64})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_PLAN_STALE"
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "a: &x 1\n"})
|
||||
detail = resp.json()["detail"]
|
||||
assert resp.status_code == 422 and detail["code"] == "CONFIG_INVALID" and detail["params"]["errors"][0]["code"] == "yaml_anchor_forbidden"
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "#" * (1024 * 1024 + 1)})
|
||||
assert resp.status_code == 413 and resp.json()["detail"]["code"] == "CONFIG_TOO_LARGE"
|
||||
resp = await client.post("/api/config/import/plan", content=b"x" * (4 * 1024 * 1024 + 1), headers={"content-type": "application/json"})
|
||||
assert resp.status_code == 413
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 9\n"})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "CONFIG_UNSUPPORTED_VERSION"
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n", "extra": 1})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["params"]["errors"][0]["code"] == "bad_request"
|
||||
resp = await client.get("/api/config/export?kinds=rules,secrets")
|
||||
assert resp.status_code == 422
|
||||
blocked = "opnexus_config: 1\naliases: []\n"
|
||||
plan = (await client.post("/api/config/import/plan", json={"yaml": blocked, "prune": True})).json()
|
||||
assert not plan["applicable"] and plan["items"][0]["action"] == "blocked"
|
||||
resp = await client.post("/api/config/import/apply", json={"yaml": blocked, "prune": True, "plan_token": plan["token"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_BLOCKED"
|
||||
|
||||
|
||||
async def test_read_only_account_can_export_and_plan_but_not_apply(client):
|
||||
await seed_everything()
|
||||
assert (await client.post("/api/users", json={"username": "cfg-reader", "password": PW, "role": "read_only"})).status_code == 201
|
||||
text = await _changed_file(client)
|
||||
reader = await _login("cfg-reader", PW)
|
||||
try:
|
||||
assert (await reader.get("/api/config/export")).status_code == 200
|
||||
plan = await reader.post("/api/config/import/plan", json={"yaml": text})
|
||||
assert plan.status_code == 200
|
||||
resp = await reader.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "READ_ONLY_ACCESS"
|
||||
finally:
|
||||
await reader.aclose()
|
||||
|
||||
|
||||
async def test_operator_can_plan_but_not_apply(client):
|
||||
await seed_everything()
|
||||
assert (await client.post("/api/users", json={"username": "cfg-operator", "password": PW, "role": "operator"})).status_code == 201
|
||||
text = await _changed_file(client)
|
||||
op = await _login("cfg-operator", PW)
|
||||
try:
|
||||
plan = await op.post("/api/config/import/plan", json={"yaml": text})
|
||||
assert plan.status_code == 200
|
||||
resp = await op.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "ADMIN_REQUIRED"
|
||||
finally:
|
||||
await op.aclose()
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "config"))).scalars().all()
|
||||
assert [e.status for e in events] == ["denied"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scope,plan_status,apply_status", [("read", 200, 403), ("deploy", 200, 403), ("admin", 200, 200)])
|
||||
async def test_tokens(client, scope, plan_status, apply_status):
|
||||
await seed_everything()
|
||||
text = await _changed_file(client)
|
||||
async with _bearer(await _token(client, scope)) as api:
|
||||
assert (await api.get("/api/config/export")).status_code == 200
|
||||
plan = await api.post("/api/config/import/plan", json={"yaml": text})
|
||||
assert plan.status_code == plan_status
|
||||
resp = await api.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
|
||||
assert resp.status_code == apply_status, resp.text
|
||||
|
||||
|
||||
async def test_group_changes_are_audited_now(client):
|
||||
resp = await client.post("/api/groups", json={"name": "Neu", "description": "", "host_ids": []})
|
||||
assert resp.status_code == 201
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "group"))).scalars().all()
|
||||
assert [e.status for e in events] == ["requested", "accepted"]
|
||||
|
||||
|
||||
async def test_too_many_waiting_plans_are_turned_away(client, monkeypatch):
|
||||
from app.routers import config_code
|
||||
|
||||
monkeypatch.setattr(config_code, "_waiting", config_code.MAX_WAITING)
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})
|
||||
assert resp.status_code == 429 and resp.json()["detail"]["code"] == "CONFIG_BUSY"
|
||||
monkeypatch.setattr(config_code, "_waiting", 0)
|
||||
assert (await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})).status_code == 200
|
||||
|
||||
|
||||
async def test_a_wrong_shaped_section_is_answered_with_422_not_a_server_error(client):
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\naliases: 42\n"})
|
||||
assert resp.status_code == 422, resp.text
|
||||
detail = resp.json()["detail"]
|
||||
assert detail["code"] == "CONFIG_INVALID" and detail["params"]["errors"]
|
||||
@@ -0,0 +1,434 @@
|
||||
"""Planner and apply (1.32 P5/P6): round trip, partial files, prune protection, conflicts, atomicity, no firewall."""
|
||||
import ast
|
||||
import copy
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import respx
|
||||
import yaml
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import ConfigError, parse
|
||||
from app.database import SessionLocal
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup, RuleRevision
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
CENTRAL = (PolicyHierarchyRule, OutboundNatRule, OneToOneRule, PortForwardRule, FirewallRule, FirewallAlias, DeviceGroup)
|
||||
|
||||
|
||||
async def export() -> str:
|
||||
async with SessionLocal() as db:
|
||||
return await export_yaml(db)
|
||||
|
||||
|
||||
async def plan(text: str, prune: bool = False) -> planner.Plan:
|
||||
async with SessionLocal() as db:
|
||||
return await planner.build_plan(db, parse(text), prune)
|
||||
|
||||
|
||||
async def apply(text: str, prune: bool = False, token: str | None = None) -> planner.Plan:
|
||||
token = token or (await plan(text, prune)).token
|
||||
async with SessionLocal() as db:
|
||||
try:
|
||||
result = await applier.apply_config(db, parse(text), prune, token)
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def edit(text: str, fn) -> str:
|
||||
data = yaml.safe_load(text)
|
||||
fn(data)
|
||||
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
def by_key(p: planner.Plan) -> dict:
|
||||
return {(i.kind, i.key): i for i in p.items}
|
||||
|
||||
|
||||
async def wipe_central_objects() -> None:
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostVariable.__table__.delete())
|
||||
for model in CENTRAL:
|
||||
await db.execute(model.__table__.delete())
|
||||
await db.execute(PolicyHierarchyGroup.__table__.update().values(parent_id=None))
|
||||
await db.execute(OpnsenseHost.__table__.update().values(policy_hierarchy_group_id=None))
|
||||
await db.execute(PolicyHierarchyGroup.__table__.delete())
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def payloads() -> dict:
|
||||
"""What every object would push to every host: the ROADMAP criterion "identical preview" after a round trip."""
|
||||
from app.services.alias_deploy_service import _build_payload as alias_payload
|
||||
from app.services.one_to_one_deploy_service import _build_payload as one_payload
|
||||
from app.services.outbound_nat_deploy_service import _build_payload as out_payload
|
||||
from app.services.port_forward_deploy_service import _build_payload as pf_payload
|
||||
from app.services.rule_deploy_service import _build_payload as rule_payload
|
||||
|
||||
out = {}
|
||||
async with SessionLocal() as db:
|
||||
hosts = (await db.execute(select(OpnsenseHost))).scalars().all()
|
||||
for model, builder, ident in ((FirewallRule, rule_payload, "config_key"), (PortForwardRule, pf_payload, "config_key"),
|
||||
(OneToOneRule, one_payload, "config_key"), (OutboundNatRule, out_payload, "config_key"),
|
||||
(FirewallAlias, alias_payload, "name")):
|
||||
for row in (await db.execute(select(model))).scalars().all():
|
||||
for host in hosts:
|
||||
try:
|
||||
out[(model.__name__, getattr(row, ident), host.name)] = builder(row, host)
|
||||
except Exception as exc: # a missing variable is part of the result too
|
||||
out[(model.__name__, getattr(row, ident), host.name)] = type(exc).__name__
|
||||
return out
|
||||
|
||||
|
||||
# -- round trip ---------------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_export_then_plan_shows_no_changes_and_apply_is_a_no_op():
|
||||
await seed_everything()
|
||||
text = await export()
|
||||
p = await plan(text)
|
||||
assert p.applicable and p.summary()["create"] == p.summary()["update"] == p.summary()["delete"] == 0
|
||||
await apply(text, prune=True)
|
||||
assert await export() == text
|
||||
|
||||
|
||||
async def test_import_into_an_instance_without_central_objects_restores_everything_identically():
|
||||
await seed_everything()
|
||||
text, before = await export(), await payloads()
|
||||
await wipe_central_objects()
|
||||
p = await plan(text, prune=True)
|
||||
assert p.applicable and p.summary()["create"] > 10
|
||||
await apply(text, prune=True)
|
||||
assert await export() == text
|
||||
assert await payloads() == before
|
||||
assert (await plan(text, prune=True)).summary()["create"] == 0
|
||||
|
||||
|
||||
async def test_apply_writes_revisions_and_audit_events_without_the_file_content():
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="geändert"))
|
||||
await apply(text)
|
||||
async with SessionLocal() as db:
|
||||
notes = (await db.execute(select(RuleRevision.note))).scalars().all()
|
||||
events = (await db.execute(select(AuditEvent))).scalars().all()
|
||||
assert "Import (Konfiguration als Code)" in notes and "Vor erster Bearbeitung" in notes
|
||||
summary = next(e for e in events if e.kind == "config")
|
||||
assert summary.action == "import" and summary.details["summary"]["update"] == 1
|
||||
assert any(e.kind == "rule" and e.action == "import_update" and e.details["fields"] == ["description"] for e in events)
|
||||
assert all("geändert" not in str(e.details) for e in events if e.kind == "config")
|
||||
|
||||
|
||||
# -- partial files and prune ---------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_a_missing_section_leaves_the_kind_alone_and_an_empty_one_only_deletes_with_prune():
|
||||
await seed_everything()
|
||||
only_aliases = "opnexus_config: 1\naliases: []\n"
|
||||
p = await plan(only_aliases)
|
||||
assert p.touched == ["aliases"] and p.summary() == {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||
await apply(only_aliases)
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
|
||||
p = await plan(only_aliases, prune=True)
|
||||
assert by_key(p)[("aliases", "WEB_SERVERS")].reasons == ["prune_referenced"] # a rule still names it
|
||||
rules_without_alias = edit(await export(), lambda d: d["rules"][0].update(destination_net="any"))
|
||||
await apply(rules_without_alias)
|
||||
await apply(only_aliases, prune=True)
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 0
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
|
||||
|
||||
|
||||
async def test_prune_never_deletes_a_deployed_object_and_nothing_changes():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="reverted", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
text = edit(await export(), lambda d: d.update(rules=[r for r in d["rules"] if r["key"] != "block-all"]))
|
||||
p = await plan(text, prune=True)
|
||||
assert by_key(p)[("rules", "block-all")].reasons == ["prune_deployed"] and not p.applicable
|
||||
with pytest.raises(applier.PlanBlocked):
|
||||
await apply(text, prune=True, token=p.token)
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
|
||||
|
||||
|
||||
async def test_an_object_with_an_open_deployment_cannot_be_changed():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="pending", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
text = edit(await export(), lambda d: d["rules"][1].update(description="neu"))
|
||||
item = by_key(await plan(text))[("rules", "block-all")]
|
||||
assert item.reasons == ["busy_deployment"] and item.deployed_hosts == 1
|
||||
unchanged = await plan(await export())
|
||||
assert unchanged.applicable # an unchanged busy object does not block
|
||||
|
||||
|
||||
async def test_pruning_a_group_still_used_by_a_rollout_wave_is_blocked():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
|
||||
rollout = RolloutPlan(name="r", action="update", created_by="t")
|
||||
db.add(rollout)
|
||||
await db.flush()
|
||||
wave = RolloutPlanWave(plan_id=rollout.id, position=0, label="w", groups=[group])
|
||||
db.add(wave)
|
||||
await db.commit()
|
||||
data = yaml.safe_load(await export())
|
||||
for section in ("aliases", "rules", "one_to_one", "outbound_nat"):
|
||||
for obj in data[section]:
|
||||
obj["targets"]["groups"] = []
|
||||
data["device_groups"] = []
|
||||
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("device_groups", "Filialen")]
|
||||
assert item.reasons == ["prune_referenced"]
|
||||
|
||||
|
||||
async def test_pruning_a_hierarchy_group_is_blocked_while_hosts_rules_or_children_remain():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await export())
|
||||
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Organisation"]
|
||||
with pytest.raises(ConfigError) as info: # Nord still names Organisation as parent
|
||||
await plan(yaml.safe_dump(data), prune=True)
|
||||
assert info.value.errors[0]["code"] == "unknown_hierarchy_group"
|
||||
data = yaml.safe_load(await export())
|
||||
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Nord"]
|
||||
text = yaml.safe_dump(data)
|
||||
with pytest.raises(ConfigError): # the hierarchy rule ph-post-deny still sits in Nord
|
||||
await plan(text, prune=True)
|
||||
data["policy_hierarchy_rules"] = [r for r in data["policy_hierarchy_rules"] if r["group"] != "Nord"]
|
||||
p = await plan(yaml.safe_dump(data), prune=True)
|
||||
assert p.applicable # fw-filiale-1 drops out of the hierarchy with its group
|
||||
await apply(yaml.safe_dump(data), prune=True)
|
||||
async with SessionLocal() as db:
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
|
||||
assert host.policy_hierarchy_group_id is None
|
||||
data = yaml.safe_load(await export())
|
||||
data.pop("policy_hierarchy_rules")
|
||||
data["policy_hierarchy_groups"] = []
|
||||
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("policy_hierarchy_groups", "Organisation")]
|
||||
assert item.reasons == ["prune_referenced"] # its hierarchy rules stay in the database
|
||||
|
||||
|
||||
# -- validation against the database ------------------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("change,code", [
|
||||
(lambda d: d["rules"][0]["targets"]["hosts"].append("nope"), "unknown_host"),
|
||||
(lambda d: d["rules"][0]["targets"]["groups"].append("nope"), "unknown_group"),
|
||||
(lambda d: d["rules"].append(copy.deepcopy(d["rules"][0])), "duplicate_object_key"),
|
||||
(lambda d: d["rules"][0]["targets"]["groups"].append("Filialen"), "duplicate_entry"),
|
||||
(lambda d: d["policy_hierarchy_rules"][0].update(description="{{SITE}}"), "placeholder_not_supported"),
|
||||
(lambda d: d["policy_hierarchy_groups"][0].update(parent="Nord"), "hierarchy_cycle"),
|
||||
(lambda d: d["policy_hierarchy_groups"][0]["hosts"].append("fw-filiale-1"), "host_in_two_hierarchy_groups"),
|
||||
(lambda d: d["variables"].update({"nope": {"X": "1"}}), "unknown_host"),
|
||||
(lambda d: d["aliases"][0]["targets"].update(hierarchy_group="nope"), "unknown_hierarchy_group"),
|
||||
])
|
||||
async def test_file_errors_are_reported_with_a_path(change, code):
|
||||
await seed_everything()
|
||||
with pytest.raises(ConfigError) as info:
|
||||
await plan(edit(await export(), change))
|
||||
assert code in [e["code"] for e in info.value.errors]
|
||||
assert all(e["path"] for e in info.value.errors)
|
||||
|
||||
|
||||
async def test_warnings_for_unknown_aliases_removed_variables_and_redeploys():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="success", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
|
||||
def change(d):
|
||||
d["rules"][0].update(destination_net="MISSING_ALIAS", source_net="{{LAN_NET}}")
|
||||
d["rules"][1].update(source_net="lan", destination_net="wanip")
|
||||
d["variables"] = {"fw-filiale-1": {"SITE": "x"}}
|
||||
|
||||
p = await plan(edit(await export(), change), prune=True)
|
||||
codes = {w["code"]: w for w in p.warnings}
|
||||
assert codes["unknown_alias_reference"]["params"]["value"] == "MISSING_ALIAS"
|
||||
assert codes["variable_removed_in_use"]["params"]["name"] == "LAN_NET"
|
||||
assert codes["redeploy_needed"]["params"]["count"] == 1
|
||||
assert [w for w in p.warnings if w["code"] == "unknown_alias_reference"] == [codes["unknown_alias_reference"]]
|
||||
|
||||
|
||||
# -- hierarchy order -----------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_hierarchy_rule_order_is_the_order_in_the_file():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await export())
|
||||
rules = data["policy_hierarchy_rules"]
|
||||
rules[0], rules[1] = rules[1], rules[0] # both in Organisation/pre
|
||||
text = yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
p = await plan(text)
|
||||
changed = {i.key: i.changes for i in p.items if i.action == "update"}
|
||||
assert set(changed) == {"ph-block-bogons", "ph-allow-mgmt"} and changed["ph-allow-mgmt"][0]["field"] == "position"
|
||||
await apply(text)
|
||||
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]] == ["ph-allow-mgmt", "ph-block-bogons", "ph-post-deny"]
|
||||
assert (await plan(text)).summary()["update"] == 0
|
||||
|
||||
|
||||
async def test_new_hierarchy_rules_go_before_rules_the_file_does_not_name():
|
||||
await seed_everything()
|
||||
text = ("opnexus_config: 1\npolicy_hierarchy_rules:\n"
|
||||
"- {key: ph-new, group: Organisation, rulebase: pre, description: neu, interface: lan}\n")
|
||||
p = await plan(text)
|
||||
assert {i.key: i.action for i in p.items} == {"ph-new": "create", "ph-block-bogons": "update", "ph-allow-mgmt": "update"}
|
||||
assert all(i.implicit for i in p.items if i.key != "ph-new")
|
||||
await apply(text)
|
||||
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]][:3] == ["ph-new", "ph-block-bogons", "ph-allow-mgmt"]
|
||||
|
||||
|
||||
# -- variables -----------------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_variables_are_merged_per_name_and_only_removed_with_prune():
|
||||
await seed_everything()
|
||||
text = "opnexus_config: 1\nvariables:\n fw-filiale-2:\n LAN_NET: 10.9.0.0/24\n fw-filiale-1:\n SITE: Neu\n"
|
||||
await apply(text)
|
||||
data = yaml.safe_load(await export())["variables"]
|
||||
assert data == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
|
||||
await apply(text, prune=True)
|
||||
assert yaml.safe_load(await export())["variables"] == {"fw-filiale-1": {"SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
|
||||
|
||||
|
||||
# -- conflicts, atomicity, self-check -----------------------------------------------------------------------------------
|
||||
|
||||
async def test_a_change_between_plan_and_apply_makes_the_plan_stale():
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="aus der Datei"))
|
||||
token = (await plan(text)).token
|
||||
async with SessionLocal() as db:
|
||||
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen").options(selectinload(DeviceGroup.hosts)))
|
||||
group.hosts = group.hosts[:1] # only a group member: no updated_at anywhere
|
||||
await db.commit()
|
||||
with pytest.raises(applier.PlanStale):
|
||||
await apply(text, token=token)
|
||||
async with SessionLocal() as db:
|
||||
db.add(FirewallRule(description="neu in der UI", interface="lan", direction="in"))
|
||||
await db.commit()
|
||||
token = (await plan(text)).token
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.description == "neu in der UI"))
|
||||
rule.description = "noch einmal geändert"
|
||||
await db.commit()
|
||||
with pytest.raises(applier.PlanStale):
|
||||
await apply(text, token=token)
|
||||
|
||||
|
||||
async def test_a_failure_in_the_middle_leaves_the_database_unchanged(monkeypatch):
|
||||
await seed_everything()
|
||||
before = await export()
|
||||
text = edit(before, lambda d: [r.update(description=f"neu {i}") for i, r in enumerate(d["rules"])] and
|
||||
d["aliases"][0].update(description="neu"))
|
||||
original = applier._Writer.objects
|
||||
|
||||
async def broken(self, kind):
|
||||
await original(self, kind)
|
||||
if kind == "rules":
|
||||
raise RuntimeError("injected")
|
||||
|
||||
monkeypatch.setattr(applier._Writer, "objects", broken)
|
||||
with pytest.raises(RuntimeError):
|
||||
await apply(text)
|
||||
assert await export() == before
|
||||
|
||||
|
||||
async def test_the_self_check_catches_a_writer_that_drops_a_field(monkeypatch):
|
||||
await seed_everything()
|
||||
before = await export()
|
||||
text = edit(before, lambda d: d["rules"][0].update(log=True))
|
||||
monkeypatch.setitem(applier.SCALARS, "rules", tuple(f for f in applier.SCALARS["rules"] if f != "log"))
|
||||
with pytest.raises(applier.SelfCheckFailed) as info:
|
||||
await apply(text)
|
||||
assert info.value.differences == ["rules:allow-dns-lan"]
|
||||
assert await export() == before
|
||||
|
||||
|
||||
async def test_the_writer_covers_every_field_of_the_schema():
|
||||
from app.config_code import schema
|
||||
|
||||
docs = {"aliases": schema.AliasDoc, "rules": schema.RuleDoc, "port_forwards": schema.PortForwardDoc,
|
||||
"one_to_one": schema.OneToOneDoc, "outbound_nat": schema.OutboundNatDoc, "policy_hierarchy_rules": schema.HierarchyRuleDoc}
|
||||
computed = {"key", "targets", "entries", "group"}
|
||||
for kind, doc in docs.items():
|
||||
assert set(applier.SCALARS[kind]) == set(doc.model_fields) - computed, kind
|
||||
|
||||
|
||||
# -- never a firewall ----------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_plan_and_apply_never_talk_to_a_firewall():
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="x"))
|
||||
with respx.mock(assert_all_called=False) as mock: # no routes: any HTTP request fails the test
|
||||
await apply(text, prune=True)
|
||||
await wipe_central_objects()
|
||||
await apply(text, prune=True)
|
||||
assert mock.calls.call_count == 0
|
||||
|
||||
|
||||
def test_the_config_code_package_does_not_import_a_firewall_client():
|
||||
forbidden = ("opnsense_client", "_deploy_service", "rollout_engine", "host_maintenance_service")
|
||||
for path in Path(planner.__file__).parent.glob("*.py"):
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.Import, ast.ImportFrom)):
|
||||
names = [a.name for a in node.names] + ([node.module] if isinstance(node, ast.ImportFrom) and node.module else [])
|
||||
assert not any(f in n for n in names for f in forbidden), f"{path.name}: {names}"
|
||||
|
||||
|
||||
async def test_two_concurrent_applies_of_the_same_plan_serialize_and_the_second_is_stale():
|
||||
import asyncio
|
||||
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="parallel"))
|
||||
token = (await plan(text)).token
|
||||
results = await asyncio.gather(apply(text, token=token), apply(text, token=token), return_exceptions=True)
|
||||
assert sorted(type(r).__name__ for r in results) == ["Plan", "PlanStale"]
|
||||
|
||||
|
||||
async def test_variables_merged_beyond_the_per_host_limit_are_refused():
|
||||
"""The host page allows 200 variables per host; merging a file into existing values must not get past that."""
|
||||
await seed_everything() # fw-filiale-1 has 2 already
|
||||
values = "".join(f" V{i}: x\n" for i in range(199))
|
||||
with pytest.raises(ConfigError) as info:
|
||||
await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values)
|
||||
assert [e["code"] for e in info.value.errors] == ["too_many_variables"]
|
||||
assert (await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values, prune=True)).applicable
|
||||
|
||||
|
||||
async def test_what_the_ui_can_store_survives_the_round_trip_and_padding_is_cleaned_like_the_ui_does():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
|
||||
group.description = "zwei\nZeilen " # GroupWrite does not strip descriptions
|
||||
alias = await db.scalar(select(FirewallAlias).where(FirewallAlias.name == "WEB_SERVERS"))
|
||||
alias.description = " eingerückt"
|
||||
await db.commit()
|
||||
text = await export()
|
||||
assert (await plan(text)).summary()["update"] == 0
|
||||
padded = edit(text, lambda d: d["rules"][0].update(description=" DNS aus {{SITE}} ", interface=" lan "))
|
||||
p = await plan(padded)
|
||||
assert p.summary()["update"] == 0 # RuleCreate strips these fields, so this is the same rule
|
||||
with_entries = edit(text, lambda d: d["aliases"][0].update(entries=[" 10.20.30.10 ", "203.0.113.20"]))
|
||||
assert (await plan(with_entries)).summary()["update"] == 0
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Host console of configuration as code (1.32 P8): exit codes, text diff, actor, no control sequences."""
|
||||
import copy
|
||||
import json
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config_code.__main__ import TOKEN_MARK, format_errors, run
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.rule import FirewallRule
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
|
||||
async def cli(*argv, stdin: str | bytes | None = None):
|
||||
data = stdin.encode() if isinstance(stdin, str) else stdin
|
||||
return await run(list(argv), data)
|
||||
|
||||
|
||||
def token_of(out: str) -> str:
|
||||
return next(line[len(TOKEN_MARK):] for line in out.splitlines() if line.startswith(TOKEN_MARK))
|
||||
|
||||
|
||||
async def export_text() -> str:
|
||||
code, out, _ = await cli("export")
|
||||
assert code == 0
|
||||
return out
|
||||
|
||||
|
||||
def edit(text: str, fn) -> str:
|
||||
data = yaml.safe_load(text)
|
||||
fn(data)
|
||||
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
async def test_export_plan_without_changes_exits_0_and_a_change_exits_2_with_a_readable_diff():
|
||||
await seed_everything()
|
||||
text = await export_text()
|
||||
code, out, _ = await cli("plan", stdin=text)
|
||||
assert code == 0 and "Plan: 0 neu, 0 geändert, 0 entfernt" in out and token_of(out)
|
||||
def change(d):
|
||||
d["rules"][0]["destination_port"] = 853
|
||||
d["rules"].append({**copy.deepcopy(d["rules"][0]), "key": "new-rule"}) # no shared objects: YAML would write anchors
|
||||
|
||||
changed = edit(text, change)
|
||||
code, out, _ = await cli("plan", stdin=changed)
|
||||
assert code == 2
|
||||
assert '+ rules new-rule (neu)' in out
|
||||
assert '~ rules allow-dns-lan destination_port: "53" → "853"' in out
|
||||
assert out.rstrip().splitlines()[-2].startswith("Plan: 1 neu, 1 geändert, 0 entfernt")
|
||||
|
||||
|
||||
async def test_json_output_is_the_api_answer():
|
||||
await seed_everything()
|
||||
code, out, _ = await cli("plan", "--json", stdin=await export_text())
|
||||
body = json.loads(out)
|
||||
assert code == 0 and body["applicable"] and len(body["token"]) == 64 and body["summary"]["create"] == 0
|
||||
|
||||
|
||||
async def test_apply_needs_the_token_of_the_plan_and_writes_audit_as_the_console():
|
||||
await seed_everything()
|
||||
changed = edit(await export_text(), lambda d: d["rules"][0].update(description="per CLI"))
|
||||
code, _, err = await cli("apply", stdin=changed)
|
||||
assert code == 1 and "--plan-token" in err
|
||||
code, out, _ = await cli("plan", stdin=changed)
|
||||
token = token_of(out)
|
||||
code, out, err = await cli("apply", "--plan-token", "0" * 64, stdin=changed)
|
||||
assert code == 4 and "neu planen" in err
|
||||
code, out, err = await cli("apply", "--plan-token", token, stdin=changed)
|
||||
assert code == 0 and "Übernommen: 0 neu, 1 geändert, 0 entfernt" in out and "nichts auf Firewalls geschrieben" in out
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
|
||||
events = (await db.execute(select(AuditEvent))).scalars().all()
|
||||
assert rule.description == "per CLI"
|
||||
assert events and {e.actor for e in events} == {"cli (Host-Konsole)"}
|
||||
|
||||
|
||||
async def test_invalid_files_exit_3_and_name_line_and_path_without_control_characters():
|
||||
await seed_everything()
|
||||
bad = "opnexus_config: 1\nrules:\n- key: r-1\n description: \"x\\e[31mred\"\n interface: lan\n nope: 1\n"
|
||||
code, out, err = await cli("plan", stdin=bad)
|
||||
assert code == 3 and not out
|
||||
assert "Zeile 6: rules[0].nope: Unbekanntes Feld (field=nope)" in err and "\x1b" not in err
|
||||
code, out, _ = await cli("plan", "--json", stdin="a: &x 1\n")
|
||||
assert code == 3 and json.loads(out)["errors"][0]["code"] == "yaml_anchor_forbidden"
|
||||
assert "\x1b" not in format_errors([{"code": "x", "path": "p\x1b[2J", "line": 1, "params": {"value": "v\x1b[0m"}}])
|
||||
|
||||
|
||||
async def test_blocked_plans_exit_4_and_apply_refuses_them():
|
||||
await seed_everything()
|
||||
only_aliases = "opnexus_config: 1\naliases: []\n"
|
||||
code, out, _ = await cli("plan", "--prune", stdin=only_aliases)
|
||||
assert code == 4 and "! aliases WEB_SERVERS blockiert (entfernen): wird noch verwendet" in out
|
||||
code, out, err = await cli("apply", "--prune", "--plan-token", token_of(out), stdin=only_aliases)
|
||||
assert code == 4 and "blockierte Objekte" in err
|
||||
|
||||
|
||||
async def test_kinds_schema_and_usage():
|
||||
await seed_everything()
|
||||
code, out, _ = await cli("export", "--kinds", "aliases")
|
||||
assert code == 0 and list(yaml.safe_load(out)) == ["opnexus_config", "aliases"]
|
||||
assert (await cli("export", "--kinds", "nope"))[0] == 1
|
||||
code, out, _ = await cli("schema")
|
||||
assert code == 0 and json.loads(out)["title"].startswith("OPNexus-Konfiguration")
|
||||
assert (await cli("frobnicate"))[0] == 1
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Export (1.32 P3): deterministic, complete, schema-conform, and free of anything secret."""
|
||||
import re
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import canonical, dump_yaml, export_yaml
|
||||
from app.database import SessionLocal
|
||||
from app.models.rule import FirewallRule
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
GOLDEN = '''\
|
||||
# OPNexus-Konfiguration (Schema v1), exportiert von OPNexus {version}.
|
||||
# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run
|
||||
opnexus_config: 1
|
||||
device_groups:
|
||||
- name: Filialen
|
||||
description: alle Filialen
|
||||
hosts:
|
||||
- fw-filiale-1
|
||||
- fw-filiale-2
|
||||
policy_hierarchy_groups:
|
||||
- name: Organisation
|
||||
parent: null
|
||||
description: ''
|
||||
hosts: []
|
||||
- name: Nord
|
||||
parent: Organisation
|
||||
description: Region Nord
|
||||
hosts:
|
||||
- fw-filiale-1
|
||||
aliases:
|
||||
- name: WEB_SERVERS
|
||||
type: host
|
||||
description: DMZ
|
||||
enabled: true
|
||||
entries:
|
||||
- 10.20.30.10
|
||||
- 203.0.113.20
|
||||
targets:
|
||||
hosts: []
|
||||
groups:
|
||||
- Filialen
|
||||
hierarchy_group: null
|
||||
'''
|
||||
|
||||
|
||||
async def _export(**kw) -> str:
|
||||
async with SessionLocal() as db:
|
||||
return await export_yaml(db, **kw)
|
||||
|
||||
|
||||
async def test_export_starts_with_the_fixed_header_and_the_expected_sections():
|
||||
from app.version import APP_VERSION
|
||||
|
||||
await seed_everything()
|
||||
text = await _export()
|
||||
assert text.startswith(GOLDEN.format(version=APP_VERSION))
|
||||
assert [k for k in yaml.safe_load(text) if k != "opnexus_config"] == list(schema.KINDS)
|
||||
assert text.endswith("\n") and "\r" not in text
|
||||
|
||||
|
||||
async def test_export_is_deterministic_and_validates_against_the_schema():
|
||||
await seed_everything()
|
||||
first, second = await _export(), await _export()
|
||||
assert first == second
|
||||
doc = schema.ConfigDoc.model_validate(yaml.safe_load(first)) # strict: unknown fields, wrong types, control chars
|
||||
assert [r.key for r in doc.rules] == ["allow-dns-lan", "block-all"]
|
||||
assert doc.variables == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Filiale Nord"}}
|
||||
|
||||
|
||||
async def test_hierarchy_rules_are_ordered_by_group_rulebase_and_sequence_and_keep_placeholders_unresolved():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await _export())
|
||||
assert [r["key"] for r in data["policy_hierarchy_rules"]] == ["ph-block-bogons", "ph-allow-mgmt", "ph-post-deny"]
|
||||
rule = next(r for r in data["rules"] if r["key"] == "allow-dns-lan")
|
||||
assert rule["source_net"] == "{{LAN_NET}}" and rule["destination_port"] == "53"
|
||||
|
||||
|
||||
async def test_ports_are_written_as_strings_so_they_survive_the_round_trip():
|
||||
await seed_everything()
|
||||
text = await _export()
|
||||
assert re.search(r"destination_port: '53'", text)
|
||||
assert schema.ConfigDoc.model_validate(yaml.safe_load(text)).rules[0].destination_port == "53"
|
||||
|
||||
|
||||
async def test_export_never_contains_secrets_or_connection_data():
|
||||
await seed_everything()
|
||||
text = (await _export()).lower()
|
||||
for forbidden in ("akia-secret-key", "secret-host.example", "https://", "api_key", "api_secret", "private key", "opx_", "password"):
|
||||
assert forbidden not in text
|
||||
|
||||
|
||||
async def test_kinds_limits_the_sections():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await _export(kinds={"aliases", "rules"}))
|
||||
assert list(data) == ["opnexus_config", "aliases", "rules"]
|
||||
|
||||
|
||||
async def test_new_objects_get_a_key_and_an_empty_database_exports_empty_sections():
|
||||
async with SessionLocal() as db:
|
||||
rule = FirewallRule(description="x", interface="lan", direction="in")
|
||||
db.add(rule)
|
||||
await db.commit()
|
||||
assert re.fullmatch(r"r-[0-9a-f]{12}", rule.config_key) and rule.config_key == f"r-{rule.id.hex[:12]}"
|
||||
data = yaml.safe_load(await _export(kinds={"aliases", "variables"}))
|
||||
assert data["aliases"] == [] and data["variables"] == {}
|
||||
|
||||
|
||||
async def test_exporter_field_lists_match_the_schema():
|
||||
"""A field added to the schema but not to the exporter (or vice versa) must fail loudly, not export silently wrong."""
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
content = await canonical(db)
|
||||
for kind, doc in (("rules", schema.RuleDoc), ("port_forwards", schema.PortForwardDoc), ("one_to_one", schema.OneToOneDoc),
|
||||
("outbound_nat", schema.OutboundNatDoc), ("policy_hierarchy_rules", schema.HierarchyRuleDoc),
|
||||
("aliases", schema.AliasDoc), ("device_groups", schema.DeviceGroupDoc),
|
||||
("policy_hierarchy_groups", schema.HierarchyGroupDoc)):
|
||||
assert all(list(item) == list(doc.model_fields) for item in content[kind]), kind
|
||||
assert dump_yaml(content) == await _export()
|
||||
|
||||
|
||||
async def test_objects_are_sorted_by_key_not_by_insertion_order():
|
||||
async with SessionLocal() as db:
|
||||
for key in ("zz-last", "aa-first", "mm-middle"):
|
||||
db.add(FirewallRule(config_key=key, description=key, interface="lan", direction="in"))
|
||||
await db.commit()
|
||||
assert [r["key"] for r in yaml.safe_load(await _export())["rules"]] == ["aa-first", "mm-middle", "zz-last"]
|
||||
@@ -0,0 +1,184 @@
|
||||
"""Strict YAML loader (1.32 P2): everything a hostile file could try is refused with a stable code."""
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config_code import loader
|
||||
from app.config_code.loader import ConfigError, load_yaml, parse
|
||||
|
||||
HEAD = "opnexus_config: 1\n"
|
||||
|
||||
|
||||
def _code(raw) -> tuple[str, str]:
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(raw)
|
||||
return info.value.code, info.value.errors[0]["code"]
|
||||
|
||||
|
||||
def _codes(raw) -> list[str]:
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(raw)
|
||||
return [e["code"] for e in info.value.errors]
|
||||
|
||||
|
||||
RULE = "rules:\n- key: r-1\n description: d\n interface: lan\n"
|
||||
|
||||
|
||||
def test_a_plain_file_loads():
|
||||
doc, sha, _lines = parse(HEAD + RULE)
|
||||
assert doc.rules[0].key == "r-1" and len(sha) == 64
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,code", [
|
||||
# Billion laughs and every other use of anchors/aliases
|
||||
("a: &x [1, 2]\nb: *x\n", "yaml_anchor_forbidden"),
|
||||
("a: *x\n", "yaml_alias_forbidden"),
|
||||
('lol1: &l1 ["x","x"]\nlol2: &l2 [*l1,*l1]\nlol3: [*l2,*l2]\n', "yaml_anchor_forbidden"),
|
||||
# merge keys
|
||||
("a: {b: 1}\nc:\n <<: {d: 1}\n", "yaml_merge_forbidden"),
|
||||
# explicit tags of any kind
|
||||
("a: !!python/object/apply:os.system ['id']\n", "yaml_tag_forbidden"),
|
||||
("a: !!python/name:os.system\n", "yaml_tag_forbidden"),
|
||||
("a: !!binary aGVsbG8=\n", "yaml_tag_forbidden"),
|
||||
("a: !!set {x: null}\n", "yaml_tag_forbidden"),
|
||||
("a: !!str 1\n", "yaml_tag_forbidden"),
|
||||
("a: ! x\n", "yaml_tag_forbidden"),
|
||||
("a: !local x\n", "yaml_tag_forbidden"),
|
||||
("!!map {a: 1}\n", "yaml_tag_forbidden"),
|
||||
# directives and multiple documents
|
||||
("%YAML 1.1\n---\na: 1\n", "yaml_directive_forbidden"),
|
||||
("%TAG !e! tag:example.com,2000:\n---\na: 1\n", "yaml_directive_forbidden"),
|
||||
("a: 1\n---\nb: 2\n", "yaml_multiple_documents"),
|
||||
("", "yaml_empty"),
|
||||
("# only a comment\n", "yaml_empty"),
|
||||
# keys
|
||||
("a: 1\na: 2\n", "duplicate_key"),
|
||||
("rules:\n- key: x\n key: y\n", "duplicate_key"),
|
||||
("1: a\n", "non_string_key"),
|
||||
("true: a\n", "non_string_key"),
|
||||
("? [a]\n: b\n", "non_string_key"),
|
||||
("null: a\n", "non_string_key"),
|
||||
# structure limits
|
||||
("a: " + "[" * 20 + "]" * 20 + "\n", "too_deep"),
|
||||
("a: x\0y\n", "yaml_syntax"),
|
||||
("a: 'x\x1b[31m'\n", "yaml_syntax"),
|
||||
("a: [1, 2\n", "yaml_syntax"),
|
||||
("a:\n\t- b\n", "yaml_syntax"),
|
||||
])
|
||||
def test_hostile_yaml_is_refused(raw, code):
|
||||
assert _code(raw) == ("CONFIG_INVALID", code)
|
||||
|
||||
|
||||
def test_errors_name_line_and_path():
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + RULE + " targets:\n hosts: [a]\n hosts: [b]\n")
|
||||
error = info.value.errors[0]
|
||||
assert error["code"] == "duplicate_key" and error["line"] == 8 and error["path"] == "rules[0].targets"
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + RULE + " destination_port: [1]\n secret_ref: x\n")
|
||||
by_code = {e["code"]: e for e in info.value.errors}
|
||||
assert by_code["unknown_field"]["path"] == "rules[0].secret_ref" and by_code["unknown_field"]["line"] == 7
|
||||
assert by_code["unknown_field"]["params"] == {"field": "secret_ref"}
|
||||
assert by_code["bad_type"]["path"].startswith("rules[0].destination_port") and by_code["bad_type"]["line"] == 6
|
||||
|
||||
|
||||
def test_yaml_1_1_surprises_stay_text_or_are_type_errors():
|
||||
data, _ = load_yaml("a: on\nb: yes\nc: 1.5\nd: 2026-10-22\ne: 0755\nf: 1:30\ng: 0x1F\nh: .inf\ni: Off\nj: 1e3\n")
|
||||
assert data == {"a": "on", "b": "yes", "c": "1.5", "d": "2026-10-22", "e": "0755", "f": "1:30", "g": "0x1F",
|
||||
"h": ".inf", "i": "Off", "j": "1e3"}
|
||||
data, _ = load_yaml("a: true\nb: False\nc: null\nd: ~\ne:\nf: 53\ng: -1\nh: 0\n")
|
||||
assert data == {"a": True, "b": False, "c": None, "d": None, "e": None, "f": 53, "g": -1, "h": 0}
|
||||
assert "enabled" in _field_error(RULE + " enabled: yes\n") # strict bool: "yes" is not true
|
||||
|
||||
|
||||
def _field_error(body: str) -> str:
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + body)
|
||||
return " ".join(e["path"] for e in info.value.errors)
|
||||
|
||||
|
||||
def test_huge_integers_stay_text_and_do_not_hit_pythons_int_limit():
|
||||
data, _ = load_yaml("a: " + "9" * 5000 + "\n")
|
||||
assert data["a"] == "9" * 5000
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,code", [
|
||||
(b"a: \xff\xfe\n", "bad_encoding"),
|
||||
(b"a: \xc3\x28\n", "bad_encoding"),
|
||||
])
|
||||
def test_invalid_utf8_is_refused(raw, code):
|
||||
assert _code(raw) == ("CONFIG_INVALID", code)
|
||||
|
||||
|
||||
def test_a_bom_is_accepted():
|
||||
doc = parse("\ufeff" + HEAD).doc
|
||||
assert doc.rules is None
|
||||
|
||||
|
||||
def test_size_limits():
|
||||
assert _code(b"#" * (loader.MAX_BYTES + 1)) == ("CONFIG_TOO_LARGE", "too_large")
|
||||
assert _code("a: " + "x" * (loader.MAX_SCALAR + 1) + "\n") == ("CONFIG_INVALID", "scalar_too_long")
|
||||
many = "a: [" + ",".join(["1"] * (loader.MAX_NODES + 1)) + "]\n"
|
||||
assert _code(many) == ("CONFIG_INVALID", "too_many_nodes")
|
||||
|
||||
|
||||
def test_object_and_entry_limits():
|
||||
rules = "rules:\n" + "".join(f"- {{key: r-{i}, description: d, interface: lan}}\n" for i in range(loader.MAX_OBJECTS_PER_KIND + 1))
|
||||
assert "too_many_objects" in _codes(HEAD + rules)
|
||||
entries = "aliases:\n- name: A\n entries: [" + ",".join(f"10.0.{i // 250}.{i % 250}" for i in range(loader.MAX_ALIAS_ENTRIES + 1)) + "]\n"
|
||||
assert "too_many_entries" in _codes(HEAD + entries)
|
||||
long_entries = "aliases:\n- name: A\n entries: [" + ",".join(["x" * 999] * 9) + "]\n"
|
||||
assert "too_long" in _codes(HEAD + long_entries)
|
||||
variables = "variables:\n fw:\n" + "".join(f" V{i}: x\n" for i in range(loader.MAX_VARIABLES_PER_HOST + 1))
|
||||
assert "too_many_objects" in _codes(HEAD + variables)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,code,top", [
|
||||
("- a\n", "not_a_mapping", "CONFIG_INVALID"),
|
||||
("rules: []\n", "missing_version", "CONFIG_INVALID"),
|
||||
("opnexus_config: '1'\n", "missing_version", "CONFIG_INVALID"),
|
||||
("opnexus_config: true\n", "missing_version", "CONFIG_INVALID"),
|
||||
("opnexus_config: 2\n", "unsupported_version", "CONFIG_UNSUPPORTED_VERSION"),
|
||||
])
|
||||
def test_version_and_shape(raw, code, top):
|
||||
assert _code(raw) == (top, code)
|
||||
|
||||
|
||||
def test_error_values_are_shortened_and_escaped():
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + RULE + " direction: \"" + "\\u001b[2J" + "x" * 200 + "\"\n")
|
||||
value = info.value.errors[0]["params"]["value"]
|
||||
assert len(value) <= 80 and "\x1b" not in value and value.startswith("\\x1b")
|
||||
|
||||
|
||||
def test_one_mebibyte_parses_in_reasonable_time():
|
||||
"""Pure-Python PyYAML; design risk (4). A full-size file of realistic rules must stay well under the request timeout."""
|
||||
line = "- {key: r-%05d, description: Regel %05d mit etwas Text, interface: lan, source_net: 10.0.0.0/24, destination_port: '443'}\n"
|
||||
body = HEAD + "rules:\n"
|
||||
i = 0
|
||||
while len(body) + len(line) < loader.MAX_BYTES - 200 and i < loader.MAX_OBJECTS_PER_KIND:
|
||||
body += line % (i, i)
|
||||
i += 1
|
||||
body += "aliases:\n- name: BIG\n entries:\n" + "".join(f" - 10.{n // 65536}.{n // 256 % 256}.{n % 256}\n" for n in range(loader.MAX_ALIAS_ENTRIES))
|
||||
raw = body.encode()[: loader.MAX_BYTES]
|
||||
started = time.monotonic()
|
||||
try:
|
||||
parse(raw)
|
||||
except ConfigError:
|
||||
pass # truncation may cut the last line; the parse time is what counts
|
||||
elapsed = time.monotonic() - started
|
||||
print(f"parse {len(raw)} bytes: {elapsed:.2f}s")
|
||||
assert elapsed < 8
|
||||
|
||||
|
||||
WRONG_SHAPES = ["42", '"text"', "true", "{a: 1}", "[1, 2]", "[[1]]", "[{entries: 5}]", "[{entries: [1, [2]]}]", "null"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shape", WRONG_SHAPES)
|
||||
@pytest.mark.parametrize("kind", __import__("app.config_code.schema", fromlist=["KINDS"]).KINDS)
|
||||
def test_every_section_with_the_wrong_shape_is_a_config_error_never_another_exception(kind, shape):
|
||||
"""Valid YAML of the wrong form used to end in a TypeError before the schema check (audit AUD-07)."""
|
||||
try:
|
||||
parse(f"{HEAD}{kind}: {shape}\n")
|
||||
except ConfigError:
|
||||
pass # the expected, orderly refusal (null parses as an empty section)
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Schema v1 (1.32 P3): strict, equal to the API limits, JSON Schema file up to date."""
|
||||
import typing
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.config_code import schema
|
||||
from app.schemas.alias import AliasCreate
|
||||
from app.schemas.one_to_one import OneToOneCreate
|
||||
from app.schemas.outbound_nat import OutboundNatCreate
|
||||
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||
from app.schemas.port_forward import PortForwardCreate
|
||||
from app.schemas.rule import RuleCreate
|
||||
|
||||
# (file model, API model, fields that exist only on one side)
|
||||
PAIRS = [
|
||||
(schema.RuleDoc, RuleCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids"}),
|
||||
(schema.PortForwardDoc, PortForwardCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
(schema.OneToOneDoc, OneToOneCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
(schema.OutboundNatDoc, OutboundNatCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
(schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, {"key", "group"}, {"hierarchy_group_id"}),
|
||||
(schema.AliasDoc, AliasCreate, {"entries", "targets"}, {"content", "target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("doc,api,only_doc,only_api", PAIRS, ids=[p[0].__name__ for p in PAIRS])
|
||||
def test_file_model_has_the_same_fields_defaults_and_limits_as_the_api_model(doc, api, only_doc, only_api):
|
||||
assert set(doc.model_fields) - only_doc == set(api.model_fields) - only_api
|
||||
for name in set(doc.model_fields) - only_doc:
|
||||
a, b = doc.model_fields[name], api.model_fields[name]
|
||||
assert a.default == b.default, name
|
||||
assert _limits(a) == _limits(b), name
|
||||
|
||||
|
||||
def _limits(field) -> dict:
|
||||
"""Length/pattern limits of a field, also when they sit on the str branch of `str | None`."""
|
||||
metas = list(field.metadata)
|
||||
for arg in typing.get_args(field.annotation):
|
||||
metas += list(getattr(arg, "__metadata__", ()))
|
||||
metas = [m for meta in metas for m in (meta.metadata if hasattr(meta, "metadata") else [meta])] # unwrap Field(...)
|
||||
out = {}
|
||||
for meta in metas:
|
||||
for attr in ("min_length", "max_length", "pattern"):
|
||||
if getattr(meta, attr, None) is not None:
|
||||
out[attr] = getattr(meta, attr)
|
||||
return out
|
||||
|
||||
|
||||
def test_enumerations_match_the_api():
|
||||
for doc, api, name in ((schema.RuleDoc, RuleCreate, "action"), (schema.PortForwardDoc, PortForwardCreate, "natreflection"),
|
||||
(schema.OneToOneDoc, OneToOneCreate, "natreflection"), (schema.OneToOneDoc, OneToOneCreate, "type"),
|
||||
(schema.AliasDoc, AliasCreate, "type"), (schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, "rulebase")):
|
||||
assert doc.model_fields[name].annotation == api.model_fields[name].annotation, (doc.__name__, name)
|
||||
|
||||
|
||||
def _rule(**over):
|
||||
return {"key": "r-1", "description": "d", "interface": "lan", **over}
|
||||
|
||||
|
||||
def test_unknown_fields_wrong_types_and_secret_ref_are_refused():
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(secret_ref="x"))
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(enabled="yes")) # strict: no truthy strings
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(source_port=True)) # bool is never a port
|
||||
with pytest.raises(ValidationError):
|
||||
schema.ConfigDoc.model_validate({"opnexus_config": 2})
|
||||
|
||||
|
||||
def test_ports_accept_numbers_and_ranges_and_become_strings():
|
||||
assert schema.RuleDoc.model_validate(_rule(destination_port=53)).destination_port == "53"
|
||||
assert schema.RuleDoc.model_validate(_rule(destination_port="8000-8100")).destination_port == "8000-8100"
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(destination_port="1" * 33))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["a\x00b", "a\x1b[31m", "a\x7fb", "a\rb", "a\x0bb"])
|
||||
def test_control_characters_are_refused(bad):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(description=bad))
|
||||
with pytest.raises(ValidationError):
|
||||
schema.AliasDoc.model_validate({"name": "A", "entries": [bad]})
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(interface=bad))
|
||||
|
||||
|
||||
def test_descriptions_may_have_several_lines_and_padding_but_names_and_keys_not():
|
||||
assert schema.RuleDoc.model_validate(_rule(description="a\n\tb ")).description == "a\n\tb "
|
||||
assert schema.DeviceGroupDoc.model_validate({"name": "G", "description": " zwei\nZeilen "}).description == " zwei\nZeilen "
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(interface="a\nb")) # a newline only in free text
|
||||
for bad in (" G", "G "):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.DeviceGroupDoc.model_validate({"name": bad})
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(targets={"hosts": [" fw"]}))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", ["", "Upper", "-lead", "a b", "x" * 65, "ü"])
|
||||
def test_keys_must_be_lowercase_slugs(key):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(key=key))
|
||||
|
||||
|
||||
def test_variable_names_and_values_use_the_same_rules_as_the_host_page():
|
||||
ok = {"opnexus_config": 1, "variables": {"fw": {"LAN_NET": "10.0.0.0/24"}}}
|
||||
assert schema.ConfigDoc.model_validate(ok).variables == {"fw": {"LAN_NET": "10.0.0.0/24"}}
|
||||
for bad in ({"lan": "x"}, {"LAN": "{{X}}"}, {"LAN": "a\nb"}, {"LAN": ""}):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.ConfigDoc.model_validate({"opnexus_config": 1, "variables": {"fw": bad}})
|
||||
|
||||
|
||||
def test_a_missing_section_is_not_an_empty_section():
|
||||
doc = schema.ConfigDoc.model_validate({"opnexus_config": 1, "aliases": []})
|
||||
assert doc.aliases == [] and doc.rules is None
|
||||
|
||||
|
||||
def test_checked_in_json_schema_is_current():
|
||||
path = Path(schema.__file__).with_name(schema.SCHEMA_FILE)
|
||||
assert path.read_text(encoding="utf-8") == schema.json_schema_text(), (
|
||||
"Schema geändert: `docker compose run --rm --no-deps -v \"$PWD/backend:/app\" backend python -m app.config_code schema "
|
||||
f"> backend/app/config_code/{schema.SCHEMA_FILE}`"
|
||||
)
|
||||
@@ -196,3 +196,125 @@ async def test_sweep_marks_revert_failed_when_host_unreachable(client, respx_moc
|
||||
still_there = (await db.execute(select(RuleDeployment))).scalar_one_or_none()
|
||||
assert still_there is not None
|
||||
assert still_there.status == "revert_failed"
|
||||
|
||||
|
||||
# ── A retry after "written, apply failed" must keep the ORIGINAL rollback target (audit 2026-10-23) ──
|
||||
|
||||
def _patch_rule_calls(monkeypatch, *, apply_results):
|
||||
from unittest.mock import AsyncMock
|
||||
from app.services import opnsense_client
|
||||
calls = {
|
||||
"add": AsyncMock(return_value={"uuid": "rule-uuid-1"}),
|
||||
"set": AsyncMock(return_value={}),
|
||||
"delete": AsyncMock(return_value={}),
|
||||
"apply": AsyncMock(side_effect=apply_results),
|
||||
}
|
||||
monkeypatch.setattr(opnsense_client, "add_rule", calls["add"])
|
||||
monkeypatch.setattr(opnsense_client, "set_rule", calls["set"])
|
||||
monkeypatch.setattr(opnsense_client, "del_rule", calls["delete"])
|
||||
monkeypatch.setattr(opnsense_client, "apply", calls["apply"])
|
||||
return calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_after_failed_apply_still_removes_a_first_installation(client, respx_mock, monkeypatch):
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
rule_id = (await client.post("/api/rules", json=_rule_payload([host["id"]]))).json()["id"]
|
||||
calls = _patch_rule_calls(monkeypatch, apply_results=[OpnsenseAPIError("apply failed"), {}, {}])
|
||||
|
||||
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
dep = (await db.execute(select(RuleDeployment))).scalar_one()
|
||||
assert dep.status == "error" and dep.revert_deadline is not None and dep.previous_payload is None
|
||||
|
||||
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
dep = (await db.execute(select(RuleDeployment))).scalar_one()
|
||||
assert dep.status == "pending"
|
||||
assert dep.previous_payload is None # the rollback is still "remove the rule"
|
||||
dep.revert_deadline = datetime.now(timezone.utc) - timedelta(seconds=1)
|
||||
await db.commit()
|
||||
await sweep_expired_deployments(db)
|
||||
assert (await db.execute(select(RuleDeployment))).scalar_one_or_none() is None
|
||||
|
||||
assert calls["delete"].call_count == 1
|
||||
assert calls["delete"].call_args.args[1] == "rule-uuid-1"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_after_failed_apply_restores_the_confirmed_payload_of_an_existing_rule(client, respx_mock, monkeypatch):
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
rule_id = (await client.post("/api/rules", json=_rule_payload([host["id"]]))).json()["id"]
|
||||
confirmed = {"description": "confirmed before", "action": "pass"}
|
||||
async with SessionLocal() as db:
|
||||
db.add(RuleDeployment(
|
||||
rule_id=uuid.UUID(rule_id), host_id=uuid.UUID(host["id"]), status="success",
|
||||
opnsense_rule_uuid="existing-uuid", last_payload=confirmed,
|
||||
))
|
||||
await db.commit()
|
||||
calls = _patch_rule_calls(monkeypatch, apply_results=[OpnsenseAPIError("apply failed"), {}, {}])
|
||||
|
||||
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
|
||||
assert (await client.post(f"/api/rules/{rule_id}/deploy")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
dep = (await db.execute(select(RuleDeployment))).scalar_one()
|
||||
assert dep.status == "pending"
|
||||
assert dep.previous_payload == confirmed
|
||||
assert dep.last_payload != confirmed
|
||||
dep.revert_deadline = datetime.now(timezone.utc) - timedelta(seconds=1)
|
||||
await db.commit()
|
||||
await sweep_expired_deployments(db)
|
||||
|
||||
assert calls["delete"].call_count == 0
|
||||
assert calls["set"].call_args.args[2] == confirmed # the last set_rule is the rollback to the confirmed payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_armed_error_row_blocks_cleanup_and_maintenance_like_a_pending_one(client, respx_mock, monkeypatch):
|
||||
from app.models.rule_deployment import RuleDeployment as Dep
|
||||
from app.services.deploy_flow import is_unsettled, unsettled
|
||||
now = datetime.now(timezone.utc)
|
||||
assert is_unsettled(Dep(status="error", revert_deadline=now + timedelta(minutes=1)))
|
||||
assert not is_unsettled(Dep(status="error", revert_deadline=None))
|
||||
assert not is_unsettled(Dep(status="success", revert_deadline=None))
|
||||
assert is_unsettled(Dep(status="revert_failed", revert_deadline=None))
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
rule_id = (await client.post("/api/rules", json=_rule_payload([host["id"]]))).json()["id"]
|
||||
async with SessionLocal() as db:
|
||||
db.add(Dep(rule_id=uuid.UUID(rule_id), host_id=uuid.UUID(host["id"]), status="error",
|
||||
opnsense_rule_uuid="u", last_payload={"a": 1}, revert_deadline=now + timedelta(minutes=1)))
|
||||
await db.commit()
|
||||
assert await db.scalar(select(Dep.id).where(unsettled(Dep)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_retry_after_failed_reconfigure_still_removes_a_first_installation(client, respx_mock, monkeypatch):
|
||||
from unittest.mock import AsyncMock
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
from app.services import opnsense_client
|
||||
from app.services.alias_deploy_service import sweep_expired_alias_deployments
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from tests.test_aliases import _alias_payload
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
alias_id = (await client.post("/api/aliases", json=_alias_payload([host["id"]]))).json()["id"]
|
||||
delete = AsyncMock(return_value={})
|
||||
monkeypatch.setattr(opnsense_client, "add_alias_item", AsyncMock(return_value={"uuid": "alias-uuid-1"}))
|
||||
monkeypatch.setattr(opnsense_client, "set_alias_item", AsyncMock(return_value={}))
|
||||
monkeypatch.setattr(opnsense_client, "del_alias_item", delete)
|
||||
monkeypatch.setattr(opnsense_client, "reconfigure_aliases", AsyncMock(side_effect=[OpnsenseAPIError("reconfigure failed"), {}, {}]))
|
||||
|
||||
assert (await client.post(f"/api/aliases/{alias_id}/deploy")).status_code == 200
|
||||
assert (await client.post(f"/api/aliases/{alias_id}/deploy")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
dep = (await db.execute(select(AliasDeployment))).scalar_one()
|
||||
assert dep.previous_payload is None
|
||||
dep.revert_deadline = datetime.now(timezone.utc) - timedelta(seconds=1)
|
||||
await db.commit()
|
||||
await sweep_expired_alias_deployments(db)
|
||||
assert delete.call_count == 1
|
||||
@@ -42,7 +42,7 @@ async def test_traceroute_is_parsed_and_audited(client, respx_mock):
|
||||
{"address": "10.0.0.1", "ttl": "1", "AS": "AS0", "host": "gw.lan", "probes": "0.4 ms"}]}}))
|
||||
resp = await client.post(f"/api/hosts/{host_id}/diagnostics/traceroute", json={"target": "example.com", "protocol": "icmp"})
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["hops"] == [{"ttl": "1", "address": "10.0.0.1", "host": "gw.lan", "as": "AS0", "probes": "0.4 ms"}]
|
||||
assert resp.json()["hops"] == [{"ttl": "1", "address": "10.0.0.1", "host": "gw.lan", "as": "AS0", "probes": "0.4 ms", "silent": False}]
|
||||
assert json.loads(sent.calls.last.request.content)["traceroute"]["settings"]["hostname"] == "example.com"
|
||||
events = await _events(host_id)
|
||||
assert [(e.action, e.status, e.entity_name) for e in events] == [("diag.traceroute", "ok", "example.com")]
|
||||
@@ -141,3 +141,17 @@ async def test_only_admins_can_run_probes(anon_client, client, respx_mock):
|
||||
assert not probe.called
|
||||
finally:
|
||||
await client.delete(f"/api/users/{created.json()['id']}")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hops_that_never_answered_come_back_as_explicit_silent_rows(client, respx_mock):
|
||||
"""T5: the firewall skips them (TTL 1 -> 3); an all-silent route is an empty list the UI explains."""
|
||||
host_id = await _host(client, respx_mock, "fw-diag-silent")
|
||||
route = respx_mock.post(_url("/diagnostics/traceroute/set"))
|
||||
route.mock(return_value=httpx.Response(200, json={"result": "ok", "response": {"notice": "n", "rows": [
|
||||
{"address": "10.0.0.1", "ttl": "1", "AS": "", "host": "", "probes": "0.4 ms"},
|
||||
{"address": "192.0.2.9", "ttl": "3", "AS": "", "host": "", "probes": "9 ms"}]}}))
|
||||
hops = (await client.post(f"/api/hosts/{host_id}/diagnostics/traceroute", json={"target": "example.com"})).json()["hops"]
|
||||
assert [(h["ttl"], h["address"], h["silent"]) for h in hops] == [("1", "10.0.0.1", False), ("2", "", True), ("3", "192.0.2.9", False)]
|
||||
route.mock(return_value=httpx.Response(200, json={"result": "ok", "response": {"notice": "n", "rows": []}}))
|
||||
assert (await client.post(f"/api/hosts/{host_id}/diagnostics/traceroute", json={"target": "example.com"})).json()["hops"] == []
|
||||
@@ -0,0 +1,266 @@
|
||||
"""Way back to the previous firmware (1.32, B5): rollback point before every update, guided rollback."""
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import backup_store, scheduled_maintenance_service
|
||||
from tests.test_maintenance import execute, preview
|
||||
from tests.test_rules import _create_host
|
||||
from tests.test_scheduled_maintenance import _due_update
|
||||
|
||||
BASE = "https://fw1.test"
|
||||
|
||||
|
||||
class Snapshots:
|
||||
"""A firewall with ZFS boot environments, as Core/Api/SnapshotsController answers."""
|
||||
|
||||
def __init__(self, mock, *, version="26.7.6", status="update", fail_add=False, hide_new=False, fail_reboot=False,
|
||||
mark_on_activate=True):
|
||||
self.rows = [{"uuid": "be-default", "name": "default", "active": "NR", "created_str": "2026-10-01 10:00"}]
|
||||
self.fail_add, self.hide_new, self.fail_reboot, self.mark_on_activate = fail_add, hide_new, fail_reboot, mark_on_activate
|
||||
self.activated: list[str] = []
|
||||
mock.get(f"{BASE}/api/diagnostics/interface/getVipStatus").respond(200, json={"rows": []})
|
||||
mock.get(f"{BASE}/api/core/firmware/info").respond(200, json={"product_version": version})
|
||||
mock.get(f"{BASE}/api/core/firmware/status").respond(200, json={"status": status, "all_packages": {}, "status_msg": "ok"})
|
||||
mock.get(f"{BASE}/api/core/firmware/running").respond(200, json={"status": "ready"})
|
||||
mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": True})
|
||||
self.fail_list = False
|
||||
mock.get(f"{BASE}/api/core/snapshots/search").mock(side_effect=self._search)
|
||||
self.add = mock.post(f"{BASE}/api/core/snapshots/add").mock(side_effect=self._add)
|
||||
self.delete = mock.post(url__regex=rf"{BASE}/api/core/snapshots/del/.*").mock(side_effect=self._del)
|
||||
self.activate = mock.post(url__regex=rf"{BASE}/api/core/snapshots/activate/.*").mock(side_effect=self._activate)
|
||||
self.update = mock.post(f"{BASE}/api/core/firmware/update").respond(200, json={"status": "ok"})
|
||||
self.upgrade = mock.post(f"{BASE}/api/core/firmware/upgrade").respond(200, json={"status": "ok"})
|
||||
self.reboot = mock.post(f"{BASE}/api/core/system/reboot").mock(side_effect=self._reboot)
|
||||
self.download = mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
|
||||
|
||||
def _search(self, request):
|
||||
if self.fail_list:
|
||||
return httpx.Response(502, json={})
|
||||
return httpx.Response(200, json={"rows": self.rows})
|
||||
|
||||
def point(self, name: str, active: str = "-") -> None:
|
||||
self.rows.append({"uuid": f"be-{name}", "name": name, "active": active, "created_str": "x"})
|
||||
|
||||
def _add(self, request):
|
||||
if self.fail_add:
|
||||
return httpx.Response(200, json={"status": "failed", "validations": {"snapshot.name": "nope"}})
|
||||
import json
|
||||
name = json.loads(request.content)["name"]
|
||||
if not self.hide_new:
|
||||
self.point(name)
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
def _del(self, request):
|
||||
uid = str(request.url).rsplit("/", 1)[-1]
|
||||
self.rows = [r for r in self.rows if r["uuid"] != uid]
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
def _activate(self, request):
|
||||
uid = str(request.url).rsplit("/", 1)[-1]
|
||||
self.activated.append(uid)
|
||||
if getattr(self, "fail_list_after_activate", False) and len(self.activated) == 1:
|
||||
self.fail_list = True
|
||||
if self.mark_on_activate:
|
||||
for r in self.rows:
|
||||
r["active"] = ("N" if "N" in r["active"] else "") + ("R" if r["uuid"] == uid else "")
|
||||
r["active"] = r["active"] or "-"
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
def _reboot(self, request):
|
||||
if self.fail_reboot:
|
||||
return httpx.Response(500, json={})
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
|
||||
async def _host_with_backup(client):
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
backup_store.save_snapshot(uuid.UUID(h["id"]), b"<opnsense/>")
|
||||
return h
|
||||
|
||||
|
||||
async def _op(op_id) -> HostOperation:
|
||||
async with SessionLocal() as db:
|
||||
return await db.get(HostOperation, uuid.UUID(str(op_id)))
|
||||
|
||||
|
||||
async def test_a_manual_update_creates_a_rollback_point_first_and_keeps_only_two(client, respx_mock):
|
||||
fw = Snapshots(respx_mock)
|
||||
fw.point("opnexus-pre-26.7.4-20261001-100000")
|
||||
fw.point("opnexus-pre-26.7.5-20261010-100000")
|
||||
fw.point("handmade") # never touched
|
||||
h = await _host_with_backup(client)
|
||||
p = await preview(client, h, "update")
|
||||
r = await execute(client, h, p.json())
|
||||
assert r.json()["status"] == "submitted" and fw.update.call_count == 1
|
||||
names = [row["name"] for row in fw.rows]
|
||||
new = next(n for n in names if n.startswith("opnexus-pre-26.7.6-"))
|
||||
assert "opnexus-pre-26.7.4-20261001-100000" not in names # the oldest of three is cleaned up
|
||||
assert {"default", "handmade", "opnexus-pre-26.7.5-20261010-100000", new} == set(names)
|
||||
op = await _op(r.json()["id"])
|
||||
assert op.snapshot["rollback_point"]["name"] == new and op.snapshot["rollback_point"]["version"] == "26.7.6"
|
||||
assert fw.add.call_count == 1
|
||||
order = [str(c.request.url) for c in respx_mock.calls if c.request.method == "POST"]
|
||||
assert order.index(f"{BASE}/api/core/snapshots/add") < order.index(f"{BASE}/api/core/firmware/update")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("problem", ["fail_add", "hide_new"])
|
||||
async def test_no_rollback_point_no_update(client, respx_mock, problem):
|
||||
fw = Snapshots(respx_mock, **{problem: True})
|
||||
h = await _host_with_backup(client)
|
||||
p = await preview(client, h, "update")
|
||||
r = await execute(client, h, p.json())
|
||||
assert r.json()["status"] == "cancelled" and "Update nicht gestartet" in r.json()["message"]
|
||||
assert fw.update.call_count == 0
|
||||
|
||||
|
||||
async def test_without_zfs_the_update_runs_and_says_there_is_no_way_back(client, respx_mock):
|
||||
fw = Snapshots(respx_mock)
|
||||
respx_mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": False})
|
||||
h = await _host_with_backup(client)
|
||||
r = await execute(client, h, (await preview(client, h, "update")).json())
|
||||
assert r.json()["status"] == "submitted" and "Kein Rückweg-Punkt" in r.json()["message"]
|
||||
assert fw.add.call_count == 0 and fw.update.call_count == 1
|
||||
|
||||
|
||||
async def test_a_scheduled_update_fails_closed_without_a_rollback_point(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, fail_add=True)
|
||||
respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
sched_id = await _due_update(h)
|
||||
async with SessionLocal() as db:
|
||||
await scheduled_maintenance_service.fire_due(db)
|
||||
async with SessionLocal() as db:
|
||||
sched = await db.get(ScheduledMaintenance, sched_id)
|
||||
assert sched.status == "failed" and "Rückweg-Punkt" in sched.result["error"]
|
||||
assert fw.update.call_count == 0
|
||||
|
||||
|
||||
async def test_rollback_needs_a_point_from_opnexus(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("handmade")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await preview(client, h, "firmware_rollback")
|
||||
assert r.status_code == 409 and "Kein Rückweg-Punkt" in r.json()["detail"]["params"]["reason"]
|
||||
assert (await client.post(f"/api/hosts/{h['id']}/snapshots", json={"name": "opnexus-pre-1-20260101-000000"})).status_code == 422
|
||||
|
||||
|
||||
async def test_rollback_activates_the_newest_point_reboots_and_lists_what_is_lost(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, version="26.7.7", status="none")
|
||||
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
|
||||
fw.point(f"opnexus-pre-26.7.5-{(stamp - timedelta(days=5)):%Y%m%d-%H%M%S}")
|
||||
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
async with SessionLocal() as db:
|
||||
rule = FirewallRule(description="nachher", interface="lan", direction="in")
|
||||
db.add(rule)
|
||||
await db.flush()
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
p = await preview(client, h, "firmware_rollback")
|
||||
assert p.status_code == 200, p.text
|
||||
rollback = p.json()["state"]["rollback"]
|
||||
assert rollback["target"]["version"] == "26.7.6" and rollback["current"]["name"] == "default"
|
||||
assert [c["kind"] for c in rollback["changes_since"]] == ["rule"]
|
||||
r = await execute(client, h, p.json())
|
||||
assert r.json()["status"] == "submitted" and "erwartet 26.7.6" in r.json()["message"]
|
||||
assert fw.activated == [rollback["target"]["uuid"]] and fw.reboot.call_count == 1
|
||||
|
||||
|
||||
async def test_rollback_that_is_not_marked_for_the_next_start_never_reboots(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none", mark_on_activate=False)
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
|
||||
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
|
||||
|
||||
|
||||
async def test_a_failed_reboot_reactivates_the_current_boot_environment(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none", fail_reboot=True)
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "unknown"
|
||||
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
|
||||
assert next(row for row in fw.rows if row["uuid"] == "be-default")["active"].endswith("R")
|
||||
|
||||
|
||||
async def test_rollback_on_a_carp_master_is_refused_like_a_reboot(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
|
||||
r = await preview(client, h, "firmware_rollback")
|
||||
assert r.status_code == 409 and fw.activate.call_count == 0
|
||||
|
||||
|
||||
async def test_the_rollback_endpoint_needs_the_deploy_right_like_every_maintenance_execute():
|
||||
from app.services import api_token_service
|
||||
|
||||
assert api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/execute")
|
||||
assert not api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/preview")
|
||||
|
||||
|
||||
async def test_rollback_saves_the_current_configuration_first_and_refuses_without_it(client, respx_mock):
|
||||
"""Going back discards every change since the point on the firewall: OPNexus keeps a copy of the config first."""
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
download = respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense>now</opnsense>")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "submitted" and download.call_count == 1
|
||||
assert backup_store.list_snapshots(uuid.UUID(h["id"])) # stored locally
|
||||
order = [str(c.request.url) for c in respx_mock.calls]
|
||||
assert order.index(f"{BASE}/api/core/backup/download/this") < order.index(f"{BASE}/api/core/snapshots/activate/be-opnexus-pre-26.7.6-20261020-100000")
|
||||
|
||||
|
||||
async def test_rollback_without_a_configuration_copy_does_not_start(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
respx_mock.get(f"{BASE}/api/core/backup/download/this").mock(side_effect=httpx.ConnectError("down"))
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "cancelled" and fw.activate.call_count == 0 and fw.reboot.call_count == 0
|
||||
|
||||
|
||||
async def test_an_unreadable_list_after_activating_switches_back_and_never_reboots(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.fail_list_after_activate = True
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
|
||||
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
|
||||
|
||||
|
||||
async def test_after_a_rollback_newer_deployments_are_marked_so_the_next_deploy_really_writes(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
|
||||
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
async with SessionLocal() as db:
|
||||
old, new = FirewallRule(description="alt", interface="lan", direction="in"), FirewallRule(description="neu", interface="lan", direction="in")
|
||||
db.add_all([old, new])
|
||||
await db.flush()
|
||||
db.add_all([
|
||||
RuleDeployment(rule_id=old.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u1",
|
||||
last_payload={"x": 1}, deployed_at=stamp - timedelta(days=1)),
|
||||
RuleDeployment(rule_id=new.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u2",
|
||||
last_payload={"x": 2}, deployed_at=stamp + timedelta(minutes=30)),
|
||||
])
|
||||
await db.commit()
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "submitted" and "1 Deployment(s) als unklar markiert" in r.json()["message"]
|
||||
async with SessionLocal() as db:
|
||||
rows = {d.opnsense_rule_uuid: d for d in (await db.scalars(select(RuleDeployment))).all()}
|
||||
assert rows["u1"].status == "success" and rows["u1"].last_payload == {"x": 1} # older than the point: untouched
|
||||
assert rows["u2"].status == "error" and rows["u2"].last_payload is None and "Firmware-Rückweg" in rows["u2"].message
|
||||
@@ -218,3 +218,18 @@ async def test_rejected_draft_can_be_corrected_then_pushed(client, respx_mock):
|
||||
respx_mock.post('https://fw1.test/api/firewall/filter/apply').mock(return_value=httpx.Response(200, json={'status':'OK\n'}))
|
||||
pushed = await client.post(f"/api/rules/{r['id']}/deploy")
|
||||
assert pushed.json()['deployments'][0]['status'] == 'pending'
|
||||
|
||||
|
||||
async def test_a_group_that_targets_a_vpn_object_cannot_be_deleted(client, respx_mock):
|
||||
"""B15: the VPN link tables cascade, so the delete used to succeed and the VPN silently lost its targets."""
|
||||
from tests.test_ipsec import _tunnel_payload
|
||||
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = await _create_host(client, "fw-vpn", "https://fw-vpn.test")
|
||||
target = await group(client, [host["id"]], name="VPN-Ziel")
|
||||
resp = await client.post("/api/ipsec/tunnels", json=_tunnel_payload([], target_group_ids=[target["id"]]))
|
||||
assert resp.status_code == 201, resp.text
|
||||
refused = await client.delete(f"/api/groups/{target['id']}")
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "GRUPPE_WIRD_NOCH_VON_VPN_VERWENDET"
|
||||
assert (await client.delete(f"/api/ipsec/tunnels/{resp.json()['id']}")).status_code == 204
|
||||
assert (await client.delete(f"/api/groups/{target['id']}")).status_code == 204
|
||||
@@ -5,6 +5,7 @@ import pytest
|
||||
|
||||
from tests.test_rules import _rule_payload
|
||||
from tests.test_aliases import _alias_payload, _mock_alias_push
|
||||
from tests.helpers import delete_host
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -18,12 +19,12 @@ async def test_delete_host_with_active_rule_deployment_rejected(client, respx_mo
|
||||
await client.post(f"/api/rules/{rule['id']}/deploy")
|
||||
await client.post(f"/api/rules/{rule['id']}/confirm")
|
||||
|
||||
resp = await client.delete(f"/api/hosts/{host['id']}")
|
||||
resp = await delete_host(client, host['id'])
|
||||
assert resp.status_code == 409, resp.text
|
||||
|
||||
respx_mock.post("https://fw-k.test/api/firewall/filter/delRule/r1").mock(return_value=httpx.Response(200, json={"result": "deleted"}))
|
||||
await client.delete(f"/api/rules/{rule['id']}") # undeploys + removes the rule first
|
||||
assert (await client.delete(f"/api/hosts/{host['id']}")).status_code == 204
|
||||
assert (await delete_host(client, host['id'])).status_code == 204
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -36,12 +37,12 @@ async def test_delete_host_with_active_alias_deployment_rejected(client, respx_m
|
||||
await client.post(f"/api/aliases/{alias['id']}/deploy")
|
||||
await client.post(f"/api/aliases/{alias['id']}/confirm")
|
||||
|
||||
resp = await client.delete(f"/api/hosts/{host['id']}")
|
||||
resp = await delete_host(client, host['id'])
|
||||
assert resp.status_code == 409, resp.text
|
||||
|
||||
respx_mock.post("https://fw-l.test/api/firewall/alias/delItem/alias-uuid-1").mock(return_value=httpx.Response(200, json={"result": "deleted"}))
|
||||
await client.delete(f"/api/aliases/{alias['id']}")
|
||||
assert (await client.delete(f"/api/hosts/{host['id']}")).status_code == 204
|
||||
assert (await delete_host(client, host['id'])).status_code == 204
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -72,7 +73,7 @@ async def test_create_list_delete_host(client, respx_mock):
|
||||
names = [h["name"] for h in resp.json()]
|
||||
assert "fw1" in names
|
||||
|
||||
resp = await client.delete(f"/api/hosts/{host_id}")
|
||||
resp = await delete_host(client, host_id)
|
||||
assert resp.status_code == 204
|
||||
|
||||
resp = await client.get("/api/hosts")
|
||||
@@ -587,3 +588,52 @@ async def test_host_address_can_be_changed_after_the_firewall_moved(client, resp
|
||||
for bad in ("ftp://x", "http://10.9.9.3", "https://user:[email protected]", "https://10.9.9.3/?a=b"):
|
||||
assert (await client.patch(f"/api/hosts/{host['id']}", json={"base_url": bad})).status_code == 422, bad
|
||||
assert next(h for h in (await client.get("/api/hosts")).json() if h["id"] == host["id"])["base_url"] == "https://10.9.9.2"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_second_host_with_the_same_address_is_refused(client, respx_mock):
|
||||
"""T4: the same firewall twice would double every rollout, alert and backup."""
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
first = await client.post("/api/hosts", json={"name": "dup-a", "base_url": "https://dup.test", "api_key": "k", "api_secret": "s"})
|
||||
assert first.status_code == 201
|
||||
for url in ("https://dup.test", "https://DUP.test/", "https://dup.test//"):
|
||||
again = await client.post("/api/hosts", json={"name": "dup-b", "base_url": url, "api_key": "k", "api_secret": "s"})
|
||||
assert again.status_code == 409, url
|
||||
assert again.json()["detail"] == {"code": "HOST_URL_EXISTS", "params": {"host": "dup-a"}}
|
||||
other = (await client.post("/api/hosts", json={"name": "dup-c", "base_url": "https://dup-c.test", "api_key": "k", "api_secret": "s"})).json()
|
||||
moved = await client.patch(f"/api/hosts/{other['id']}", json={"base_url": "https://dup.test"})
|
||||
assert moved.status_code == 409 and moved.json()["detail"]["code"] == "HOST_URL_EXISTS"
|
||||
assert (await client.patch(f"/api/hosts/{other['id']}", json={"base_url": "https://dup-c.test/"})).status_code == 200 # itself
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("failure,reason", [
|
||||
(httpx.Response(401, json={"status": 401}), "credentials"),
|
||||
(httpx.Response(403, json={"status": 403}), "forbidden"),
|
||||
(httpx.ConnectError("no route"), "unreachable"),
|
||||
(httpx.Response(200, json={"rows": []}), None),
|
||||
])
|
||||
async def test_verify_connection_asks_the_firewall_once_and_keeps_nothing_on_failure(client, respx_mock, failure, reason):
|
||||
route = respx_mock.get(url__regex=r".*/getVipStatus")
|
||||
route.mock(side_effect=failure) if isinstance(failure, Exception) else route.mock(return_value=failure)
|
||||
body = {"name": f"vc-{reason}", "base_url": f"https://vc-{reason}.test", "api_key": "k", "api_secret": "s", "verify_connection": True}
|
||||
resp = await client.post("/api/hosts", json=body)
|
||||
names = [h["name"] for h in (await client.get("/api/hosts")).json()]
|
||||
if reason is None:
|
||||
assert resp.status_code == 201 and body["name"] in names
|
||||
else:
|
||||
assert resp.status_code == 422 and resp.json()["detail"] == {"code": "HOST_CONNECTION_FAILED", "params": {"reason": reason}}
|
||||
assert body["name"] not in names # nothing saved, so a corrected retry is not blocked by a leftover
|
||||
assert (await client.post("/api/hosts", json={**body, "verify_connection": False})).status_code == 201 # "create anyway"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_deleting_a_host_needs_its_name_on_the_server_side_too(client, respx_mock):
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
host = (await client.post("/api/hosts", json={"name": "del-me", "base_url": "https://del-me.test", "api_key": "k", "api_secret": "s"})).json()
|
||||
for params in ({}, {"confirm_name": ""}, {"confirm_name": "DEL-ME"}, {"confirm_name": "other"}):
|
||||
refused = await client.delete(f"/api/hosts/{host['id']}", params=params)
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "HOSTNAMEN_EXAKT_EINGEBEN", params
|
||||
assert (await client.get(f"/api/hosts/{host['id']}")).status_code == 200
|
||||
assert (await client.delete(f"/api/hosts/{host['id']}", params={"confirm_name": "del-me"})).status_code == 204
|
||||
assert (await client.delete(f"/api/hosts/{host['id']}", params={"confirm_name": "del-me"})).status_code == 404
|
||||
@@ -14,6 +14,7 @@ def mock_reads(mock,base='https://fw1.test',carp=None,status='none',running='rea
|
||||
mock.get(f'{base}/api/core/firmware/info').respond(200,json={'product_version':version})
|
||||
mock.get(f'{base}/api/core/firmware/status').respond(200,json={'status':status,'all_packages':{},'status_msg':'cached result'})
|
||||
mock.get(f'{base}/api/core/firmware/running').respond(200,json={'status':running})
|
||||
mock.get(f'{base}/api/core/snapshots/isSupported').respond(200,json={'supported':False}) # UFS: no rollback point (1.32)
|
||||
|
||||
async def preview(client,h,action='reboot'):
|
||||
return await client.post(f"/api/hosts/{h['id']}/maintenance/preview",json={'action':action})
|
||||
|
||||
@@ -450,4 +450,49 @@ async def test_0029_creates_the_rule_cleanup_tables_and_settings_and_is_idempote
|
||||
assert not {"cleanup_max_rules", "cleanup_delete_min_days", "cleanup_management_sources", "cleanup_management_ports"} & cols
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head") # second run: nothing to do
|
||||
assert await _version(scratch) == _head() == "0029" and await _schema_diff(scratch) == []
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
|
||||
|
||||
async def test_0030_backfills_config_keys_and_is_idempotent(scratch):
|
||||
"""1.31 -> 1.32: every rule/NAT table gets a unique, readable key derived from the UUID; the downgrade drops it."""
|
||||
await _alembic(scratch, "upgrade", "0029")
|
||||
rule_id = "3f2a9c1d-0000-4000-8000-000000000001"
|
||||
async with scratch.begin() as conn:
|
||||
await conn.execute(text(
|
||||
"INSERT INTO firewall_rules (id, description, interface, direction, ipprotocol, protocol, action, "
|
||||
"source_net, destination_net, enabled, log, created_at, updated_at) VALUES "
|
||||
f"('{rule_id}', 'r', 'lan', 'in', 'inet', 'any', 'pass', 'any', 'any', true, false, now(), now())"))
|
||||
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
async with scratch.connect() as conn:
|
||||
key = (await conn.execute(text("SELECT config_key FROM firewall_rules"))).scalar_one()
|
||||
nullable = (await conn.execute(text(
|
||||
"SELECT is_nullable FROM information_schema.columns "
|
||||
"WHERE table_name = 'port_forward_rules' AND column_name = 'config_key'"))).scalar_one()
|
||||
assert key == "r-3f2a9c1d0000" and nullable == "NO"
|
||||
|
||||
await _alembic(scratch, "downgrade", "0029")
|
||||
async with scratch.connect() as conn:
|
||||
cols = set((await conn.execute(text(
|
||||
"SELECT column_name FROM information_schema.columns WHERE table_name = 'firewall_rules'"))).scalars().all())
|
||||
assert "config_key" not in cols
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
|
||||
|
||||
async def test_0031_adds_gate_probes_to_existing_plans_and_is_idempotent(scratch):
|
||||
await _alembic(scratch, "upgrade", "0030")
|
||||
async with scratch.begin() as conn:
|
||||
await conn.execute(text(
|
||||
"INSERT INTO rollout_plans (id, name, description, action, auto_start, gate_checks, compliance_mode, created_by, created_at, updated_at) "
|
||||
"VALUES (gen_random_uuid(), 'p', '', 'update', false, '{}', 'warn', 'admin', now(), now())"))
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
async with scratch.connect() as conn:
|
||||
assert (await conn.execute(text("SELECT gate_probes::text FROM rollout_plans"))).scalar_one() == "[]"
|
||||
await _alembic(scratch, "downgrade", "0030")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
@@ -0,0 +1,16 @@
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def test_importing_models_alone_configures_all_mappers():
|
||||
"""`python -m app.reset_password` / `app.reset_2fa` import only part of the app: the mapper registry must still be complete."""
|
||||
code = "import app.models; from sqlalchemy.orm import configure_mappers; configure_mappers()"
|
||||
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
|
||||
assert result.returncode == 0, result.stderr[-800:]
|
||||
|
||||
|
||||
def test_reset_scripts_import_cleanly_and_configure_mappers():
|
||||
for module in ("app.reset_password", "app.reset_2fa"):
|
||||
code = f"import {module}; from sqlalchemy.orm import configure_mappers; configure_mappers()"
|
||||
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
|
||||
assert result.returncode == 0, (module, result.stderr[-800:])
|
||||
@@ -1,6 +1,7 @@
|
||||
import httpx
|
||||
import pytest
|
||||
from tests.test_rules import _create_host
|
||||
from tests.helpers import delete_host
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize('kind,remote,fields',[('port-forward','d_nat',{'target':'192.0.2.20'}),('one-to-one','one_to_one',{'external':'192.0.2.20'}),('outbound-nat','source_nat',{'target':'192.0.2.20'})])
|
||||
@@ -14,7 +15,7 @@ async def test_nat_groups_and_deletion_guards(client,respx_mock,outbound_mode_hy
|
||||
respx_mock.post(f'https://fw1.test/api/firewall/{remote}/addRule').respond(200,json={'result':'saved','uuid':'nat1'})
|
||||
respx_mock.post(f'https://fw1.test/api/firewall/{remote}/apply').respond(200,json={'status':'ok'})
|
||||
assert (await client.post(f"/api/{kind}/{r['id']}/deploy")).status_code==200
|
||||
assert (await client.delete('/api/hosts/'+h['id'])).status_code==409
|
||||
assert (await delete_host(client, h['id'])).status_code==409
|
||||
impact=(await client.post(f"/api/groups/{g['id']}/preview",json={'name':'NAT group','description':'','host_ids':[]})).json()
|
||||
assert impact['nat'][0]['retained_host_ids']==[h['id']]
|
||||
assert impact['nat'][0]['kind']==kind.replace('-','_')
|
||||
@@ -25,7 +26,7 @@ async def test_nat_groups_and_deletion_guards(client,respx_mock,outbound_mode_hy
|
||||
respx_mock.post(f'https://fw1.test/api/firewall/{remote}/delRule/nat1').respond(200,json={'result':'deleted'})
|
||||
p=(await client.get(f"/api/{kind}/{r['id']}/preview")).json()
|
||||
assert (await client.post(f"/api/{kind}/{r['id']}/deployments/{h['id']}/remove",json={'preview_token':p['token']})).status_code==204
|
||||
assert (await client.delete('/api/hosts/'+h['id'])).status_code==204
|
||||
assert (await delete_host(client, h['id'])).status_code==204
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pf_disabled_polarity_and_readonly_automatic_rows(client,respx_mock):
|
||||
|
||||
@@ -600,3 +600,25 @@ async def test_vxlan_and_gif_are_listed_and_changed_like_the_other_devices(clien
|
||||
done = await _execute(client, host_id, plan["token"], **change)
|
||||
assert done.status_code == 200 and done.json()["verified"] is True, done.text
|
||||
assert _body(add.calls.last) == {"gif": change["fields"]}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("kind,field,bad,good", [
|
||||
("vlan", "tag", ["0", "4095", "-1", "abc", "20.5", "99999"], ["1", "20", "4094"]),
|
||||
("vlan", "pcp", ["8", "-1"], ["0", "7"]),
|
||||
("vip", "vhid", ["0", "256"], ["1", "255", ""]),
|
||||
("vip", "advbase", ["0", "255"], ["1", "254"]),
|
||||
("vip", "advskew", ["255", "-5"], ["0", "254"]),
|
||||
])
|
||||
async def test_numbers_outside_the_firewalls_range_are_refused_in_the_preview(client, respx_mock, kind, field, bad, good):
|
||||
"""T6: a VLAN tag of 4095 used to come back from the firewall as 502 UPSTREAM_REJECTED; now 422 with the field name,
|
||||
before the firewall is asked anything (ranges from Interfaces/Vlan.xml and Vip.xml)."""
|
||||
host_id = await _setup(client, respx_mock, f"fw-r-{kind}-{field}")
|
||||
base_fields = {"vlan": {"if": "vtnet1", "tag": "20"}, "vip": {"interface": "lan", "mode": "carp", "network": "192.0.2.1/24"}}[kind]
|
||||
for value in bad:
|
||||
resp = await _preview(client, host_id, kind=kind, action="add", fields={**base_fields, field: value})
|
||||
assert resp.status_code == 422, (field, value)
|
||||
assert resp.json()["detail"] == {"code": "NETWORK_FIELD_INVALID", "params": {"field": field}}
|
||||
for value in good:
|
||||
resp = await _preview(client, host_id, kind=kind, action="add", fields={**base_fields, field: value})
|
||||
assert resp.status_code == 200, (field, value, resp.text)
|
||||
@@ -2,6 +2,7 @@
|
||||
confused login could succeed."""
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import time
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
@@ -72,6 +73,15 @@ def _claims(sent_nonce, **over):
|
||||
return {k: v for k, v in base.items() if v is not None}
|
||||
|
||||
|
||||
def _forged_hs256(claims, secret: str, kid="k1"):
|
||||
"""What an attacker sends: PyJWT >= 2.12 itself refuses to sign with a JWK-looking secret, so build the token by hand."""
|
||||
def b64(raw: bytes) -> bytes:
|
||||
return base64.urlsafe_b64encode(raw).rstrip(b"=")
|
||||
signing_input = b64(json.dumps({"alg": "HS256", "typ": "JWT", "kid": kid}).encode()) + b"." + b64(json.dumps(claims).encode())
|
||||
signature = hmac.new(secret.encode(), signing_input, hashlib.sha256).digest()
|
||||
return (signing_input + b"." + b64(signature)).decode()
|
||||
|
||||
|
||||
def _id_token(claims, key=KEY, alg="RS256", kid="k1"):
|
||||
return jwt.encode(claims, key, algorithm=alg, headers={"kid": kid})
|
||||
|
||||
@@ -221,7 +231,7 @@ async def test_signature_attacks_are_rejected(client, anon_client, respx_mock):
|
||||
if spec is None:
|
||||
token = jwt.encode(claims, None, algorithm="none", headers={"kid": "k1"})
|
||||
elif spec == "hs":
|
||||
token = jwt.encode(claims, json.dumps(JWK), algorithm="HS256", headers={"kid": "k1"})
|
||||
token = _forged_hs256(claims, json.dumps(JWK))
|
||||
else:
|
||||
token = _id_token(claims, **spec)
|
||||
resp, _ = await _callback(anon_client, respx_mock, query, token_response={"id_token": token})
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
"""The operator role: rolls out and operates, but cannot change access or destinations."""
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
@@ -42,7 +43,8 @@ async def test_operator_keeps_self_service(client):
|
||||
assert (await op.put("/api/settings", json={"theme": "dark"})).status_code == 200
|
||||
resp = await op.post("/api/auth/change-password", json={"current_password": PW, "new_password": "another-password-1"})
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert (await op.get("/api/users")).status_code == 200
|
||||
listed = await op.get("/api/users") # self-service: only the own account, never the directory (T2)
|
||||
assert listed.status_code == 200 and [u["username"] for u in listed.json()] == ["selfserve"]
|
||||
finally:
|
||||
await op.aclose()
|
||||
|
||||
@@ -136,6 +138,8 @@ async def test_demoting_an_admin_to_operator_takes_effect_at_once(client):
|
||||
("POST", "/api/rules/abc/confirm", False),
|
||||
("GET", "/api/rules/abc/deploy", False),
|
||||
("POST", "/api/hosts/abc/network/changes/preview", False),
|
||||
("POST", "/api/hosts/abc/snapshots", True),
|
||||
("GET", "/api/hosts/abc/snapshots", False),
|
||||
])
|
||||
def test_what_counts_as_a_rollout(method, path, rollout):
|
||||
from app.services import api_token_service
|
||||
@@ -206,3 +210,26 @@ async def test_channel_test_button_is_admin_only_on_the_server_too(client):
|
||||
await op.aclose()
|
||||
if ro is not None:
|
||||
await ro.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_creating_a_snapshot_is_a_rollout_and_follows_the_separation_of_duties(client, respx_mock):
|
||||
"""A boot environment is created ON the firewall: with deploying admin-only an operator may not do it (AUD-05)."""
|
||||
from tests.test_snapshots import BASE, ROW, _host
|
||||
host_id = await _host(client, respx_mock, "fw-oper-snap")
|
||||
new = {"uuid": "s9", "name": "by_operator", "active": "-", "mountpoint": "-", "size": "0", "created_str": "now"}
|
||||
respx_mock.get(f"{BASE}/api/core/snapshots/isSupported").mock(return_value=httpx.Response(200, json={"supported": True}))
|
||||
respx_mock.get(f"{BASE}/api/core/snapshots/search").mock(return_value=httpx.Response(200, json={"rows": [ROW, new]}))
|
||||
add = respx_mock.post(f"{BASE}/api/core/snapshots/add").mock(return_value=httpx.Response(200, json="OK"))
|
||||
op = await _operator(client, "snapop")
|
||||
try:
|
||||
assert (await op.post(f"/api/hosts/{host_id}/snapshots", json={"name": "by_operator"})).status_code == 201
|
||||
assert add.call_count == 1
|
||||
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": False})).status_code == 200
|
||||
resp = await op.post(f"/api/hosts/{host_id}/snapshots", json={"name": "by_operator"})
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
|
||||
assert add.call_count == 1 # nothing was sent to the firewall
|
||||
assert (await client.post(f"/api/hosts/{host_id}/snapshots", json={"name": "by_operator"})).status_code == 201 # an admin may
|
||||
finally:
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
await op.aclose()
|
||||
@@ -1,4 +1,6 @@
|
||||
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.test_maintenance_windows import _WEEKLY
|
||||
@@ -145,3 +147,27 @@ async def test_role_matrix_for_plans(client):
|
||||
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
|
||||
finally:
|
||||
await ro.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gate_probes_are_validated_stored_and_frozen_into_the_fingerprint(client):
|
||||
"""1.32 (B5): probes per plan; a plan without probes keeps the fingerprint of earlier versions."""
|
||||
from app.database import SessionLocal
|
||||
from app.models.rollout import RolloutPlan
|
||||
from app.services import rollout_service
|
||||
|
||||
_hosts, groups = await _fleet(client)
|
||||
plain = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
|
||||
assert plain["gate_probes"] == []
|
||||
async with SessionLocal() as db:
|
||||
assert "gate_probes" not in rollout_service.definition(await db.get(RolloutPlan, uuid.UUID(plain["id"])))
|
||||
probes = [{"type": "ping", "target": "1.1.1.1"}, {"type": "tcp", "target": "intranet.example", "port": 443},
|
||||
{"type": "ping", "target": "1.1.1.1"}]
|
||||
resp = await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=probes))
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["gate_probes"] == [{"type": "ping", "target": "1.1.1.1", "port": None},
|
||||
{"type": "tcp", "target": "intranet.example", "port": 443}]
|
||||
for bad in ([{"type": "tcp", "target": "x"}], [{"type": "ping", "target": "x", "port": 1}],
|
||||
[{"type": "ping", "target": "-rf"}], [{"type": "ping", "target": "a b"}], [{"type": "udp", "target": "x"}],
|
||||
[{"type": "ping", "target": f"h{i}"} for i in range(11)]):
|
||||
assert (await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=bad))).status_code == 422, bad
|
||||
@@ -31,6 +31,7 @@ class Fw:
|
||||
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
|
||||
r, b = fleet.mock, self.base
|
||||
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
|
||||
r.get(f"{b}/api/core/snapshots/isSupported").mock(side_effect=lambda q: self._ok({"supported": False}))
|
||||
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
|
||||
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
|
||||
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
|
||||
@@ -620,6 +621,19 @@ def test_target_version_comes_from_the_core_package_not_product_latest():
|
||||
assert engine.target_of({"upgrade_major_version": "27.1"}, "major_update") == "27.1"
|
||||
|
||||
|
||||
def test_target_version_of_other_release_types_and_a_switched_core_package():
|
||||
# Shape seen live on test-fw-1 (2026-10-23) after switching the release type community -> development.
|
||||
switch = {"status": "update", "product": {"product_id": "opnsense", "product_version": "26.7.7", "product_latest": "26.7.7"},
|
||||
"all_packages": {"opnsense-devel": {"new": "27.1.a_483", "reason": "new", "name": "opnsense-devel", "old": "N/A"}},
|
||||
"upgrade_packages": [], "new_packages": [{"name": "opnsense-devel", "repository": "OPNsense", "version": "27.1.a_483"}]}
|
||||
assert engine.target_of(switch, "update") == "27.1.a_483"
|
||||
assert engine.target_of({**switch, "all_packages": {}}, "update") == "27.1.a_483" # from new_packages
|
||||
business = {"status": "update", "product": {"product_latest": "26.10"},
|
||||
"all_packages": {"opnsense-business": {"old": "26.10", "new": "26.10.1", "reason": "upgrade"}}}
|
||||
assert engine.target_of(business, "update") == "26.10.1"
|
||||
assert engine.target_of({"all_packages": {"unrelated": {"new": "9"}}, "product": {"product_latest": "26.7.7"}}, "update") == "26.7.7"
|
||||
|
||||
|
||||
# ── 1.29 (S6): window exceptions in the fingerprint, stable account id ────────
|
||||
|
||||
async def _activate(client, plan):
|
||||
@@ -711,3 +725,28 @@ async def test_rows_without_an_account_id_still_work_by_name(client, respx_mock,
|
||||
assert await engine.may_roll_out(db, "admin") is True
|
||||
assert await engine.may_roll_out(db, "nobody-here") is False
|
||||
assert await engine.may_roll_out(db, "admin", uuid.uuid4()) is False # a given id that does not exist is final
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_probe_that_breaks_after_the_update_stops_the_rollout(client, respx_mock, clock, monkeypatch):
|
||||
"""1.32 (B5): the plan's probes are frozen into the run, taken in the baseline and compared in the gate."""
|
||||
from app.services import diagnostics_service
|
||||
|
||||
reachable = {"ok": True}
|
||||
calls = []
|
||||
|
||||
async def portprobe(host, target, port, ipproto, source):
|
||||
calls.append(host.name)
|
||||
return {"open": reachable["ok"] or host.name != "c1"}
|
||||
|
||||
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, gate_probes=[{"type": "tcp", "target": "10.0.0.5", "port": 443}])
|
||||
run = await _start(client, plan)
|
||||
assert (await _run(run["id"])).plan_snapshot["gate_probes"] == [{"type": "tcp", "target": "10.0.0.5", "port": 443}]
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
assert "c1" in calls # baseline
|
||||
reachable["ok"] = False # the update broke the path to the server
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "gate_failed"
|
||||
assert paused.waves[0].gate_result["hosts"][str(_host(paused, "c1").host_id)]["probes"]["code"] == "probe_regressed"
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
@@ -193,3 +193,47 @@ async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeyp
|
||||
def datetime_now():
|
||||
from datetime import datetime, timezone
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_probes_from_the_firewall_compare_against_the_baseline(client, respx_mock, monkeypatch):
|
||||
"""1.32 (B5): reachability probes. Worked before and fails now = fail; failed already before = warn."""
|
||||
from app.services import audit_service, diagnostics_service
|
||||
|
||||
reachable = {"ping 1.1.1.1": True, "tcp 10.0.0.5:443": True, "ping 192.0.2.1": False}
|
||||
actors = []
|
||||
|
||||
async def ping(host, target, ipproto, count, packetsize, source):
|
||||
actors.append((audit_service.context.get() or {}).get("actor"))
|
||||
return {"received": count if reachable[f"ping {target}"] else 0}
|
||||
|
||||
async def portprobe(host, target, port, ipproto, source):
|
||||
if target == "broken":
|
||||
raise diagnostics_service.OpnsenseAPIError("job failed")
|
||||
return {"open": reachable[f"tcp {target}:{port}"]}
|
||||
|
||||
monkeypatch.setattr(diagnostics_service, "ping", ping)
|
||||
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
|
||||
Fw(respx_mock, A)
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
probes = [{"type": "ping", "target": "1.1.1.1", "port": None}, {"type": "tcp", "target": "10.0.0.5", "port": 443},
|
||||
{"type": "ping", "target": "192.0.2.1", "port": None}]
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, uuid.UUID(a["id"]))
|
||||
base = {a["id"]: await rh.baseline(db, host, probes)}
|
||||
assert [p["ok"] for p in base[a["id"]]["probes"]] == [True, True, False] and set(actors) == {"system:rollout"}
|
||||
gate, _ = await _eval([a["id"]], base, probes=probes)
|
||||
assert gate["hosts"][a["id"]]["probes"] == {"result": "warn", "code": "probe_was_failing", "params": {"probes": "ping 192.0.2.1"}}
|
||||
reachable["tcp 10.0.0.5:443"] = False
|
||||
gate, _ = await _eval([a["id"]], base, probes=probes)
|
||||
assert gate["verdict"] == "fail"
|
||||
assert gate["hosts"][a["id"]]["probes"] == {"result": "fail", "code": "probe_regressed", "params": {"probes": "tcp 10.0.0.5:443"}}
|
||||
broken = probes + [{"type": "tcp", "target": "broken", "port": 22}] # cannot run = failed; new since the baseline = worked before
|
||||
gate, _ = await _eval([a["id"]], base, probes=broken)
|
||||
assert "tcp broken:22" in gate["hosts"][a["id"]]["probes"]["params"]["probes"]
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert "probes" not in gate["hosts"][a["id"]] # a plan without probes runs none
|
||||
fw_down = Fw(respx_mock, A)
|
||||
fw_down.up = False
|
||||
gate, _ = await _eval([a["id"]], base, probes=probes)
|
||||
assert gate["hosts"][a["id"]]["probes"]["code"] == "unreadable"
|
||||
@@ -2,6 +2,7 @@
|
||||
fake firewall (tests/rule_cleanup_fake.py) and the test database. Numbers refer to DESIGN-1.31 §6. Roles and API
|
||||
tokens are checked in the router (package 6), not here."""
|
||||
import asyncio
|
||||
import json
|
||||
import uuid as uuid_lib
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace
|
||||
@@ -992,3 +993,70 @@ async def test_an_outdated_acknowledgement_text_is_refused(env): # review findi
|
||||
assert env.fw.calls == [] # refused before anything is read
|
||||
ok = await preview(env.host_id, [R_BLOCK], acknowledge=[R_BLOCK], ack_text_version=svc.ACK_TEXT_VERSION)
|
||||
assert ok["plan"]["acknowledged"] == [R_BLOCK]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opnexus_own_address_is_protected_even_when_only_workstations_are_configured(env, monkeypatch):
|
||||
"""B10: the management sources may list only the admins' workstations. A rule that is the only way OPNexus itself
|
||||
reaches the firewall would then look harmless, and after switching it off the automatic revert could not reach the
|
||||
firewall any more. The address the firewall sees OPNexus come from is protected as well."""
|
||||
from app.services import opnsense_client
|
||||
from app.services.rule_cleanup_service import redact
|
||||
|
||||
custom = "aaaaaaaa-0000-4000-8000-000000000920"
|
||||
env.fw.rules.append(rule(custom, 920, interface="wan", src="198.51.100.0/24", dport="443", description="opnexus server net"))
|
||||
await _seed_hits(env.host_id, [custom])
|
||||
ok = await preview(env.host_id, [custom]) # address unknown (respx has no socket): configured sources only
|
||||
assert ok["plan"]["nodes"][0]["opnexus_source"] == {"address": None, "status": "unknown"}
|
||||
|
||||
async def seen_from(_host):
|
||||
return "198.51.100.7"
|
||||
|
||||
monkeypatch.setattr(opnsense_client, "own_source_address", seen_from)
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[custom])
|
||||
assert got["code"] == "BEREINIGUNG_REGEL_NICHT_AENDERBAR" and got["params"]["reason"] == "management"
|
||||
assert env.fw.writes() == []
|
||||
|
||||
async def configured(_host):
|
||||
return MGMT_SOURCE
|
||||
|
||||
monkeypatch.setattr(opnsense_client, "own_source_address", configured)
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
node = p["plan"]["nodes"][0]
|
||||
assert node["opnexus_source"] == {"address": MGMT_SOURCE, "status": "configured"}
|
||||
hidden = redact(p)["plan"]["nodes"][0] # an address is security-relevant like the sources: admins only
|
||||
assert hidden["opnexus_source"] == {"status": "configured"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_own_source_address_matches_exactly_this_connection(monkeypatch, respx_mock):
|
||||
"""The state whose source port is the local port of the request; nothing else (other clients on 443, old states)."""
|
||||
import httpx as _httpx
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import opnsense_client, opnsense_secret_store
|
||||
|
||||
host = OpnsenseHost(id=uuid_lib.uuid4(), name="fw-src", base_url="https://192.0.2.1", api_key="k", verify_tls=False)
|
||||
opnsense_secret_store.save_secret(host.id, "s")
|
||||
respx_mock.get("https://192.0.2.1/api/diagnostics/firewall/pf_states").mock(return_value=_httpx.Response(200, json={"current": 3}))
|
||||
states = respx_mock.post("https://192.0.2.1/api/diagnostics/firewall/query_states")
|
||||
rows = [
|
||||
{"src_addr": "203.0.113.50", "src_port": "40000", "dst_addr": "192.0.2.1", "dst_port": "443", "state": "ESTABLISHED:ESTABLISHED"}, # a browser
|
||||
{"src_addr": "192.0.2.10", "src_port": "55555", "dst_addr": "192.0.2.1", "dst_port": "443", "state": "TIME_WAIT:TIME_WAIT"}, # old
|
||||
{"src_addr": "192.0.2.10", "src_port": "33160", "dst_addr": "192.0.2.1", "dst_port": "443", "state": "ESTABLISHED:ESTABLISHED"}, # us
|
||||
]
|
||||
states.mock(return_value=_httpx.Response(200, json={"rows": rows}))
|
||||
addrs = {"server_addr": ("192.0.2.1", 443), "client_addr": ("172.22.0.5", 33160)}
|
||||
monkeypatch.setattr(opnsense_client, "_stream_addr", lambda _resp, which: addrs.get(which))
|
||||
assert await opnsense_client.own_source_address(host) == "192.0.2.10"
|
||||
assert json.loads(states.calls.last.request.content)["searchPhrase"] == "192.0.2.1"
|
||||
|
||||
addrs["client_addr"] = ("172.22.0.5", 40001) # NAT rewrote the port: no guessing
|
||||
assert await opnsense_client.own_source_address(host) is None
|
||||
addrs["server_addr"] = None # through the overlay's CONNECT proxy
|
||||
assert await opnsense_client.own_source_address(host) is None
|
||||
states.mock(return_value=_httpx.Response(403, json={"message": "forbidden"}))
|
||||
addrs["server_addr"] = ("192.0.2.1", 443)
|
||||
with pytest.raises(opnsense_client.OpnsenseAPIError):
|
||||
await opnsense_client.own_source_address(host)
|
||||
opnsense_secret_store.delete_secret(host.id)
|
||||
@@ -24,6 +24,7 @@ from app.routers import rule_cleanup as router_mod
|
||||
from app.schemas.rule_cleanup import CleanupItemRead, CleanupOperationRead
|
||||
from app.services import api_token_service, notification_checks
|
||||
from app.services import rule_cleanup_service as svc
|
||||
from tests.helpers import delete_host
|
||||
from tests.rule_cleanup_fake import MGMT_SOURCE, R_ALIAS, R_BASE, R_BLOCK, R_DUP, R_MGMT, R_TAIL, R_UNUSED
|
||||
from tests.test_api_tokens import _bearer, _login, _make
|
||||
from tests.test_rule_cleanup import ( # noqa: F401 (fixtures are used by name)
|
||||
@@ -583,7 +584,7 @@ async def test_the_block_covers_the_cluster_and_the_maintenance_inspection(clust
|
||||
await ensure_no_maintenance(db, [a], cleanup=False) # the cleanup service's own check does not trip over itself
|
||||
# deleting or re-addressing either node is refused while it is open
|
||||
for host_id in (cluster.a, cluster.b):
|
||||
r = await client.delete(f"/api/hosts/{host_id}")
|
||||
r = await delete_host(client, host_id)
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", host_id
|
||||
op = (await client.get(_base(cluster.a))).json()[0]
|
||||
assert (await client.post(f"{_base(cluster.a)}/{op['id']}/revert")).status_code == 200
|
||||
@@ -596,7 +597,7 @@ async def test_the_block_covers_the_cluster_and_the_maintenance_inspection(clust
|
||||
async def test_host_delete_and_readdressing_are_refused_while_a_cleanup_is_open(client, env):
|
||||
hid = env.host_id
|
||||
await _open(client, env)
|
||||
gone = await client.delete(f"/api/hosts/{hid}")
|
||||
gone = await delete_host(client, hid)
|
||||
assert gone.status_code == 409 and gone.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
for body in ({"base_url": "https://203.0.113.99"}, {"enabled": False}, {"cluster_name": "ha-x"}, {"clear_cluster_name": True}):
|
||||
r = await client.patch(f"/api/hosts/{hid}", json=body)
|
||||
@@ -607,7 +608,7 @@ async def test_host_delete_and_readdressing_are_refused_while_a_cleanup_is_open(
|
||||
assert host.base_url == FW1 and host.enabled and host.name == "fw-a2"
|
||||
op = (await client.get(_base(env))).json()[0]
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/revert")).status_code == 200
|
||||
assert (await client.delete(f"/api/hosts/{hid}")).status_code == 204 # settled: the host may go (its runs go with it)
|
||||
assert (await delete_host(client, hid)).status_code == 204 # settled: the host may go (its runs go with it)
|
||||
async with SessionLocal() as db:
|
||||
assert (await db.scalars(select(RuleCleanupOperation))).all() == []
|
||||
|
||||
@@ -865,11 +866,12 @@ async def test_sources_and_firewall_addresses_are_for_admins_only(client, env, w
|
||||
|
||||
|
||||
# Review finding 9: undeploying a central object and removing a former target write to the firewall too.
|
||||
# (The VPN undeploy_* are thin wrappers that take the per-object lease; the work and the check live in _undeploy_*.)
|
||||
_UNDEPLOY_SITES = [
|
||||
("app.services.alias_deploy_service", "undeploy_alias"), ("app.services.openvpn_deploy_service", "undeploy_instance"),
|
||||
("app.services.wireguard_deploy_service", "undeploy_tunnel"), ("app.services.port_forward_deploy_service", "undeploy_port_forward"),
|
||||
("app.services.alias_deploy_service", "undeploy_alias"), ("app.services.openvpn_deploy_service", "_undeploy_instance"),
|
||||
("app.services.wireguard_deploy_service", "_undeploy_tunnel"), ("app.services.port_forward_deploy_service", "undeploy_port_forward"),
|
||||
("app.services.outbound_nat_deploy_service", "undeploy_outbound_nat"), ("app.services.one_to_one_deploy_service", "undeploy_one_to_one"),
|
||||
("app.services.ipsec_deploy_service", "undeploy_tunnel"),
|
||||
("app.services.ipsec_deploy_service", "_undeploy_tunnel"),
|
||||
] + [(f"app.routers.{r}", "remove_previous_target") for r in
|
||||
("aliases", "port_forward", "one_to_one", "outbound_nat", "wireguard", "openvpn", "ipsec")]
|
||||
|
||||
@@ -908,3 +910,19 @@ async def test_alias_delete_and_target_removal_wait_for_the_cleanup(client, env)
|
||||
r = await client.post(f"/api/aliases/{former.json()['id']}/deployments/{env.host_id}/remove", json={"preview_token": preview["token"]})
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
|
||||
assert env.fw.writes() == writes
|
||||
|
||||
|
||||
async def test_admins_get_their_own_address_as_a_source_suggestion(client):
|
||||
"""B12: the settings suggest the address the request came from; only to admins (an address is security-relevant
|
||||
like the sources themselves)."""
|
||||
from app.main import app
|
||||
|
||||
got = (await client.get("/api/settings/governance")).json()
|
||||
assert got["your_address"] == "127.0.0.1" # ASGI test client
|
||||
saved = (await client.put("/api/settings/governance", json={"cleanup_management_sources": ["203.0.113.10"]})).json()
|
||||
assert saved["your_address"] == "127.0.0.1"
|
||||
assert (await client.post("/api/users", json={"username": "b12-op", "password": "password-123456", "role": "operator"})).status_code == 201
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as op:
|
||||
await op.post("/api/auth/login", json={"username": "b12-op", "password": "password-123456"})
|
||||
seen = (await op.get("/api/settings/governance")).json()
|
||||
assert seen["your_address"] is None and seen["cleanup_management_sources"] == []
|
||||
@@ -625,3 +625,57 @@ def test_firmware_without_the_automatic_flag_is_not_rejected_as_incomplete():
|
||||
assert not rule_hit_service.looks_complete({"rows": []})
|
||||
assert not rule_hit_service.looks_complete({"rows": [{"uuid": "u1", "is_automatic": "0"}]}) # flagged firmware, no automatic rule: partial
|
||||
assert rule_hit_service.looks_complete({"rows": [{"uuid": "u1", "is_automatic": "0"}, {"uuid": "a", "is_automatic": "1"}]})
|
||||
|
||||
|
||||
# ── B11: counters are read right before every apply that reloads pf ───────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_every_reloading_apply_samples_the_counters_first_and_a_failed_sample_never_blocks(client, respx_mock):
|
||||
"""`apply` (and NAT/alias/VPN/network reconfigure) zeroes all rule counters; hits since the last sample would be
|
||||
lost. Each of these client calls now reads ruleStats + searchRule right before its own POST."""
|
||||
from app.services import opnsense_client
|
||||
|
||||
host_id = await host_of(client, respx_mock, url=FW)
|
||||
stats = respx_mock.get(f"{FW}/api/firewall/filter_util/ruleStats").mock(return_value=httpx.Response(200, json={
|
||||
"status": "ok", "stats": {R1: {"packets": 42, "bytes": 4200, "evaluations": 50, "pf_rules": 1}}}))
|
||||
respx_mock.get(f"{FW}/api/firewall/filter/searchRule").mock(return_value=httpx.Response(200, json=answer(rule(R1))))
|
||||
reloads = {
|
||||
"apply": "/firewall/filter/apply", "apply_port_forward": "/firewall/d_nat/apply",
|
||||
"apply_one_to_one": "/firewall/one_to_one/apply", "apply_outbound_nat": "/firewall/source_nat/apply",
|
||||
"reconfigure_aliases": "/firewall/alias/reconfigure", "reconfigure_ipsec": "/ipsec/service/reconfigure",
|
||||
"reconfigure_openvpn": "/openvpn/service/reconfigure", "reconfigure_wireguard": "/wireguard/service/reconfigure",
|
||||
}
|
||||
for path in reloads.values():
|
||||
respx_mock.post(f"{FW}/api{path}").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
for name, path in reloads.items():
|
||||
before = len(respx_mock.calls)
|
||||
await getattr(opnsense_client, name)(host)
|
||||
order = [c.request.url.path for c in list(respx_mock.calls)[before:]]
|
||||
assert order[-1] == f"/api{path}", name
|
||||
assert "/api/firewall/filter_util/ruleStats" in order[:-1] and "/api/firewall/filter/searchRule" in order[:-1], (name, order)
|
||||
async with SessionLocal() as db:
|
||||
row = (await db.scalars(select(RuleHit).where(RuleHit.host_id == host_id))).one()
|
||||
assert row.rule_uuid == R1 and row.last_packets == 42 and row.last_hit_at is not None
|
||||
|
||||
stats.mock(return_value=httpx.Response(500, json={})) # the probe fails: the write goes on anyway
|
||||
respx_mock.get(f"{FW}/api/diagnostics/firewall/pfStatistics/rules").mock(return_value=httpx.Response(500, json={}))
|
||||
assert (await opnsense_client.apply(host))["status"] == "ok"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_writer_that_just_sampled_itself_does_not_probe_twice(client, respx_mock):
|
||||
from app.services import opnsense_client
|
||||
|
||||
host_id = await host_of(client, respx_mock, url=FW)
|
||||
stats = respx_mock.get(f"{FW}/api/firewall/filter_util/ruleStats").mock(return_value=httpx.Response(200, json={"status": "ok", "stats": {}}))
|
||||
respx_mock.get(f"{FW}/api/firewall/filter/searchRule").mock(return_value=httpx.Response(200, json=answer()))
|
||||
respx_mock.post(f"{FW}/api/firewall/filter/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
with opnsense_client.counters_already_sampled():
|
||||
await opnsense_client.apply(host)
|
||||
assert stats.call_count == 0
|
||||
await opnsense_client.apply(host)
|
||||
assert stats.call_count == 1
|
||||
Loaded 100 of 130 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user