Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4eea6424e7 | ||
|
|
a8ca0a14ef | ||
|
|
b020e38d0a |
No files matched your search
+47
-1
@@ -29,7 +29,53 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
|
||||
den Einstellungen.
|
||||
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Merlin …
|
||||
ansteht): Saker …
|
||||
|
||||
## [1.32.0] "Merlin" — 2026-10-23
|
||||
|
||||
- **Konfiguration als Code:** Unter **System › Konfiguration als Code** (und per `opnexus export` / `opnexus apply`) lassen sich Gerätegruppen, Hierarchiegruppen,
|
||||
Aliase, Filterregeln, Port-Forwards, 1:1-NAT, Outbound-NAT, Hierarchie-Regeln und Standort-Variablen als **eine YAML-Datei exportieren** (sortiert, ohne Zeitstempel,
|
||||
ohne Zugänge oder Schlüssel, für Git und Reviews) und wieder **importieren**: erst ein **Plan** (neu, geändert, entfernt, unverändert, blockiert, mit Hinweisen), dann
|
||||
das **Übernehmen** durch einen Administrator in einer Transaktion mit dem Token genau dieses Plans. **Der Import ändert nur die Entwürfe in OPNexus, nie eine Firewall**;
|
||||
ausgerollt wird wie immer über Vorschau → Ausrollen → Bestätigen. Ausgerollte oder noch verwendete Objekte entfernt er nie (Prune blockiert den Plan). Die Datei wird
|
||||
nur als Daten gelesen (keine Anker, Tags, Merge-Schlüssel, Mehrfach-Dokumente; 1 MiB, strenges Schema mit Zeile und Pfad bei Fehlern). Plan und Export gehen auch mit „Lesen“-Token
|
||||
und Nur-Lese-Konten, `opnexus apply --dry-run` liefert Exit-Codes für CI (0 / 2 / 3 / 4). Regeln und NAT-Objekte tragen dafür einen stabilen `key` (Migration, wird für
|
||||
Bestand automatisch vergeben). Gruppenänderungen stehen jetzt im Audit.
|
||||
- **Rückweg zur vorherigen Firmware:** Vor jedem Firmware-Update und Major-Upgrade (manuell, geplant, Rollout, CARP-Übergabe) legt OPNexus auf der Firewall ein
|
||||
**ZFS-Boot-Environment** `opnexus-pre-<Version>-<Zeit>` als Rückweg-Punkt an (ohne den Punkt startet das Update nicht; ohne ZFS mit Hinweis). Neue Aktion **Zur vorherigen Firmware**
|
||||
(Host-Wartung): Vorschau, Hostname, Downtime-Bestätigung, vorher lokale Sicherung der Konfiguration, dann Aktivieren, Prüfung und Neustart (scheitert ein Schritt, wird das
|
||||
vorherige Boot-Environment wieder eingeschaltet). **Der Rückweg setzt auch die Konfiguration zurück** (live belegt): Ausrollvorgänge seit dem Punkt werden als unklar markiert,
|
||||
damit das nächste Ausrollen wirklich schreibt. Ein Rollout macht nie von selbst einen Rückweg.
|
||||
- **Rollout-Gate: Erreichbarkeitsproben.** Bis zu zehn Ping-/TCP-Proben je Plan, von der Firewall selbst ausgeführt, vor der Welle und nach der Beobachtungszeit:
|
||||
vorher ok und jetzt nicht mehr hält den Rollout an, schon vorher gestört ist nur ein Hinweis. (Erreichbarkeit von der Firewall aus, nicht der Verkehr durch sie hindurch.)
|
||||
- Migrationen 0030 (`config_key`) und 0031 (`gate_probes`).
|
||||
|
||||
## [1.31.1] "Vulture" — 2026-10-22
|
||||
|
||||
- **Korrektur: `opnexus password` und `reset_2fa` stürzten ab.** Seit den Standort-Variablen (1.22) brach das Zurücksetzen des Admin-Passworts
|
||||
(`./opnexus password`) bzw. der Zwei-Faktor-Anmeldung (`docker compose exec backend python -m app.reset_2fa <benutzer>`) mit einem
|
||||
SQLAlchemy-Fehler ab, weil ein Datenmodell nicht registriert war. Der Weg bei vergessenem Passwort oder verlorenem 2FA-Gerät funktioniert wieder;
|
||||
ein Regressionstest sichert ab, dass jedes Hilfsskript alle Modelle kennt.
|
||||
|
||||
## [1.31.0] "Vulture" — 2026-10-22
|
||||
|
||||
- **Regel-Bereinigung: Entfernen, Bestätigen, Löschen, Wiederherstellen:** Im Tab **Bereinigung** (Regelwerk › Regel-Analyse) lassen sich Kandidaten jetzt auswählen
|
||||
(Checkbox, höchstens so viele Regeln je Lauf wie in den Einstellungen; „vermutlich“ nur mit einer ausdrücklichen Einzelbestätigung je Regel, „unbekannt“ nie).
|
||||
**Vorschau** zeigt je Firewall (im Cluster zuerst der Backup) die Regeln mit Befund, Sicherheit und Schutzprüfung; **Ausrollen** deaktiviert die Regeln und wendet an.
|
||||
Danach läuft ein Countdown: **Bestätigen** heißt „ich erreiche die Firewall noch“, ohne Bestätigung setzt OPNexus **automatisch zurück**; bis dahin geht auch
|
||||
**Jetzt zurücksetzen**. Bestätigte Regeln lassen sich nach der Wartezeit (einstellbar, Standard 7 Tage) **löschen**; vorher sichert OPNexus den Regelinhalt, und
|
||||
unter **Verlauf der Bereinigungen** kommt die Regel mit **Wiederherstellen** deaktiviert und möglichst an alter Position zurück (sonst mit Hinweis am Ende der Gruppe);
|
||||
**Wieder aktivieren** hat wieder ein Bestätigungsfenster. Wichtig: **Ausrollen, Wiederholen und Wiederherstellen darf nur ein Administrator**, eine Vorschau gehört
|
||||
dem, der sie erstellt hat (ein Admin öffnet selbst eine neue); **Bestätigen geht nur in der Browser-Sitzung**, nie per API-Token. **Die Bereinigung bleibt gesperrt,
|
||||
bis unter Einstellungen die Verwaltungsquellen** (IP-Adressen oder Netze, von denen aus verwaltet wird) **eingetragen sind**: Regeln, die diesen Zugang treffen
|
||||
könnten, deaktiviert OPNexus nie. Dort stehen auch die Obergrenze je Lauf, die Wartezeit bis zum Löschen und **weitere Verwaltungsports** (zusätzlich zu
|
||||
API-Port, 443, 80, 22). Tragen Sie die Adresse ein, mit der OPNexus selbst bei den Firewalls ankommt (im Cluster für jeden Knoten). Quellen, Ports und
|
||||
Firewall-Adressen sehen nur Administratoren. Der Hinweis „Entfernen folgt“ entfällt.
|
||||
- **Update: neuere CLI aus dem Zielrelease:** `opnexus update` (auch per Klick im Webinterface) holt sich nach dem Pull die CLI aus dem Updater-Image des
|
||||
Zielreleases und startet sich damit neu, damit Korrekturen am Update-Ablauf sofort greifen statt erst beim übernächsten Update. Das geschieht nur, wenn
|
||||
`cosign` und `cosign.pub` vorhanden sind und die Signatur des Image-**Digests** in diesem Lauf geprüft wurde (ausgeführt wird genau dieser Digest);
|
||||
sonst und bei jedem Fehler läuft das Update mit der vorhandenen CLI weiter. Abschalten: `OPNEXUS_NO_CLI_REFRESH=1`. Wirksam wird es ab dem Update
|
||||
*von* dieser Version.
|
||||
|
||||
## [1.30.0] "Owl" — 2026-10-22
|
||||
|
||||
|
||||
+5
-4
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||

|
||||
|
||||
> **Stand:** `1.30.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.32.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -21,12 +21,13 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||
- **Firewall-Oberfläche über OPNexus** — die Weboberfläche einer Firewall in einem eigenen Browser-Tab über OPNexus öffnen, auch hinter NAT; nur für Admins, Einmal-Ticket, eigene Origin, Zeitlimits und Audit (standardmäßig aus).
|
||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen (auf Wunsch mit Erreichbarkeitsproben, die die Firewall selbst ausführt) und automatischem Halt bei jeder Auffälligkeit. Vor jedem Update legt OPNexus ein ZFS-Boot-Environment als Rückweg-Punkt an; „Zur vorherigen Firmware“ stellt es geführt wieder her.
|
||||
- **Konfiguration als Code** — Regeln, Aliase, NAT, Gruppen, Policy-Hierarchie und Standort-Variablen als eine YAML-Datei exportieren (für Git und Review) und wieder importieren: erst ein Plan (neu / geändert / entfernt / blockiert), dann das Übernehmen in einer Transaktion, das nur die Entwürfe in OPNexus ändert (nie eine Firewall). `opnexus export` / `opnexus apply --dry-run` für Konsole und CI.
|
||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||
- **Sicherheitslage** — ein Wert von 0 bis 100 je Firewall aus Härtungs- und Regel-Hygiene-Prüfungen (offene WAN-Regeln, schwaches SSH/SNMP, verdeckte Regeln, alte Backups …), nur lesend.
|
||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit), und jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
|
||||
- **Regel-Analyse** — „Welche Regel trifft?“ für Quelle, Ziel, Protokoll und Port: die entscheidende Regel je Firewall (oder Flotte/Gruppe) mit ehrlicher Sicherheit (sicher / vermutlich / unbekannt) und Prüfprotokoll Regel für Regel, nur lesend. Eine Bereinigungsliste nennt ungenutzte, verdeckte, doppelte und lange deaktivierte Regeln (jeweils mit Sicherheit); ein Administrator kann ausgewählte Regeln abschalten (Vorschau, dann Ausrollen mit automatischer Rücknahme ohne Bestätigung), später löschen und aus einer Sicherung des Regelinhalts wiederherstellen. Jede Regel hat einen Änderungsverlauf – auch für Änderungen außerhalb von OPNexus.
|
||||
- **Backups & Wiederherstellung** — geplante Konfigurationssicherungen mit Aufbewahrung, verschlüsselte Offsite-Kopie auf S3-kompatiblen Speicher, Änderungsansicht zwischen Sicherungen und eine vollständige verschlüsselte Sicherung von OPNexus selbst (`opnexus backup --full`).
|
||||
- **Single Sign-on** — Anmeldung über deinen Identitätsanbieter (OIDC); die Rolle kommt bei jeder Anmeldung aus dessen Gruppen.
|
||||
- **Zugang** — HTTPS mit gepinnten Firewall-Zertifikaten, optionale Zwei-Faktor-Anmeldung, Rollen (Admin / Operator / Nur-Lesen), Tastaturbedienung, Dark Mode, Oberfläche auf Deutsch und Englisch.
|
||||
@@ -65,7 +66,7 @@ Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelie
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.30.0
|
||||
OPNEXUS_VERSION=1.32.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||

|
||||
|
||||
> **Status:** `1.30.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.32.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -21,12 +21,13 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||
- **Firewall interface through OPNexus** — open a firewall's own web interface in a separate browser tab through OPNexus, even behind NAT; admins only, one-time ticket, isolated origin, time limits and audit (off by default).
|
||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves (including optional reachability probes run from the firewall) and an automatic stop on anything unusual. Before every update OPNexus creates a ZFS boot environment as a rollback point, and a guided "back to previous firmware" restores it.
|
||||
- **Configuration as code** — export rules, aliases, NAT, groups, the policy hierarchy and site variables as one YAML file for Git and review, and import it again: a plan first (new / changed / removed / blocked), then a one-transaction apply that only changes OPNexus' drafts (never a firewall). `opnexus export` / `opnexus apply --dry-run` for the console and CI.
|
||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||
- **Security posture** — a 0–100 score per firewall from hardening and rule-hygiene checks (open WAN rules, weak SSH/SNMP, shadowed rules, stale backups …), read-only.
|
||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence) and every rule has a change history, including changes made outside OPNexus.
|
||||
- **Rule analysis** — "Which rule matches?" for a source, destination, protocol and port: the deciding rule per firewall (or fleet/group) with an honest confidence (sure / probable / unknown) and a rule-by-rule trace, read-only. A cleanup list names unused, shadowed, duplicate and long-disabled rules (each with its confidence); an administrator can switch selected rules off (preview, then roll-out with an automatic revert unless confirmed), delete them later and restore them from a backup of their content. Every rule has a change history, including changes made outside OPNexus.
|
||||
- **Backups & recovery** — scheduled config backups with retention, an encrypted offsite copy to S3-compatible storage, a redacted change view between backups, and a full encrypted backup/restore of OPNexus itself (`opnexus backup --full`).
|
||||
- **Single sign-on** — sign in with your identity provider (OIDC); roles come from its groups at every login.
|
||||
- **Access** — HTTPS with pinned firewall certificates, optional two-factor sign-in, roles (admin / operator / read-only), keyboard operation, dark mode, German and English UI.
|
||||
@@ -67,7 +68,7 @@ For access from other machines and for production use with TLS (bundled Caddy pr
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.30.0
|
||||
OPNEXUS_VERSION=1.32.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -43,6 +43,8 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
"/api/settings",
|
||||
# Read-only JSON-RPC tools (see routers/mcp.py).
|
||||
"/api/mcp",
|
||||
# Configuration as code: the plan only compares a file with the database (routers/config_code.py).
|
||||
"/api/config/import/plan",
|
||||
# Self-scoped: a user managing their own second factor.
|
||||
"/api/auth/2fa/setup",
|
||||
"/api/auth/2fa/enable",
|
||||
@@ -121,7 +123,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
)
|
||||
if user is not None:
|
||||
parts = request.url.path.strip("/").split("/")
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window", "groups": "group", "config": "config"}
|
||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||
@@ -131,10 +133,11 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
if len(parts) > 2 and parts[1] == "settings":
|
||||
kind = "settings" # where OPNexus stores/sends things (own prefs at /api/settings stay unaudited)
|
||||
action = ("delete" if request.method == "DELETE" else parts[-1] if len(parts) > 3 else "edit" if request.method in ("PUT", "PATCH") else "create")
|
||||
event = {"kind": kind, "action": action, "entity_id": parts[2] if len(parts) > 2 else None,
|
||||
entity = parts[2] if len(parts) > 2 and kind != "config" else None
|
||||
event = {"kind": kind, "action": action, "entity_id": entity,
|
||||
"host_id": parts[2] if kind == "host" and len(parts) > 2 else None,
|
||||
"details": {"method": request.method, "path": request.url.path}}
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions"))
|
||||
tracked = kind and writing and not request.url.path.endswith(("/preview", "/carp-preview", "/revisions", "/import/plan"))
|
||||
|
||||
if (
|
||||
user.role == "read_only"
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export und -Import der zentralen Objekte.
|
||||
|
||||
Leitsatz: Der Import ändert nur die OPNexus-Datenbank (Entwürfe). Auf Firewalls kommt weiterhin nur etwas über
|
||||
Vorschau, Ausrollen und Bestätigen. Entwurf: docs/schema/opnexus-config-v1.schema.json, Handbuch „Konfiguration als Code“.
|
||||
"""
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Host-Konsole: `python -m app.config_code {export|plan|apply|schema}`.
|
||||
|
||||
Normalerweise über `opnexus export` und `opnexus apply` auf dem Docker-Host aufgerufen. Wer die Konsole des Hosts
|
||||
bedient, ist ohnehin allmächtig, deshalb braucht das CLI kein Token; der Akteur im Audit ist `cli (Host-Konsole)`.
|
||||
Die Datei kommt über stdin (das Backend öffnet nie eine Datei), das Ergebnis geht nach stdout.
|
||||
|
||||
Exit-Codes (für CI): 0 angewendet bzw. keine Unterschiede, 2 Unterschiede (nur `plan`), 3 Datei ungültig,
|
||||
4 blockiert/veraltet/Selbstprüfung, 1 sonstiger Fehler.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, display, parse
|
||||
|
||||
ACTOR = "cli (Host-Konsole)"
|
||||
TOKEN_MARK = "# plan-token: "
|
||||
_ANSI = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]")
|
||||
|
||||
ERRORS_DE = {
|
||||
"yaml_alias_forbidden": "Aliase (*name) sind nicht erlaubt",
|
||||
"yaml_anchor_forbidden": "Anker (&name) sind nicht erlaubt",
|
||||
"yaml_merge_forbidden": "Merge-Schlüssel (<<) sind nicht erlaubt",
|
||||
"yaml_tag_forbidden": "Explizite Tags (!, !!) sind nicht erlaubt",
|
||||
"yaml_directive_forbidden": "Direktiven (%YAML, %TAG) sind nicht erlaubt",
|
||||
"yaml_multiple_documents": "Mehr als ein YAML-Dokument",
|
||||
"yaml_empty": "Die Datei ist leer",
|
||||
"yaml_syntax": "YAML-Syntaxfehler",
|
||||
"duplicate_key": "Doppelter Schlüssel",
|
||||
"non_string_key": "Schlüssel muss Text sein",
|
||||
"too_deep": "Zu tief verschachtelt",
|
||||
"too_many_nodes": "Zu viele Elemente",
|
||||
"scalar_too_long": "Wert zu lang",
|
||||
"too_many_objects": "Zu viele Objekte",
|
||||
"too_many_entries": "Zu viele Einträge",
|
||||
"too_many_variables": "Zu viele Variablen für diesen Host",
|
||||
"too_large": "Datei zu groß",
|
||||
"bad_encoding": "Kein gültiges UTF-8",
|
||||
"not_a_mapping": "Die Datei muss ein Mapping sein",
|
||||
"missing_version": "opnexus_config: 1 fehlt",
|
||||
"unsupported_version": "Diese Version des Dateiformats ist unbekannt",
|
||||
"unknown_field": "Unbekanntes Feld",
|
||||
"missing_field": "Pflichtfeld fehlt",
|
||||
"bad_type": "Falscher Typ",
|
||||
"bad_value": "Ungültiger Wert",
|
||||
"bad_format": "Falsches Format",
|
||||
"too_short": "Zu kurz",
|
||||
"too_long": "Zu lang",
|
||||
"control_character": "Steuerzeichen nicht erlaubt",
|
||||
"surrounding_whitespace": "Leerzeichen am Rand nicht erlaubt",
|
||||
"bad_variable_name": "Ungültiger Variablenname",
|
||||
"bad_variable_value": "Ungültiger Variablenwert",
|
||||
"duplicate_object_key": "Schlüssel kommt zweimal in der Datei vor",
|
||||
"duplicate_entry": "Eintrag kommt doppelt vor",
|
||||
"unknown_host": "Unbekannter Host",
|
||||
"unknown_group": "Unbekannte Gerätegruppe",
|
||||
"unknown_hierarchy_group": "Unbekannte Hierarchiegruppe",
|
||||
"hierarchy_cycle": "Zyklus im Hierarchiebaum",
|
||||
"host_in_two_hierarchy_groups": "Host in zwei Hierarchiegruppen",
|
||||
"placeholder_not_supported": "Platzhalter {{…}} sind in Hierarchie-Regeln nicht möglich",
|
||||
}
|
||||
REASONS_DE = {
|
||||
"busy_deployment": "offenes Deployment (erst bestätigen oder zurückrollen)",
|
||||
"prune_deployed": "war ausgerollt (zuerst in der Oberfläche entfernen)",
|
||||
"prune_referenced": "wird noch verwendet",
|
||||
}
|
||||
WARNINGS_DE = {
|
||||
"redeploy_needed": "{count} geänderte Objekte sind ausgerollt: danach über die Vorschau neu ausrollen",
|
||||
"targets_without_link": "{count} neue Objekte haben Ziele, aber keine Verknüpfung zu Objekten auf den Firewalls (Duplikate beim Ausrollen möglich)",
|
||||
"unknown_alias_reference": "{value}: kein Alias dieses Namens",
|
||||
"variable_removed_in_use": "Variable {name} von {host} wird entfernt, ein Objekt nutzt sie noch",
|
||||
"target_host_disabled": "Ziel-Host {host} ist deaktiviert",
|
||||
}
|
||||
SYMBOL = {"create": "+", "update": "~", "delete": "-", "blocked": "!"}
|
||||
LABEL = {"create": "(neu)", "delete": "(entfernen)"}
|
||||
|
||||
|
||||
def safe(text) -> str:
|
||||
"""Nothing from a foreign file reaches the terminal as a control sequence."""
|
||||
return _ANSI.sub("?", str(text))
|
||||
|
||||
|
||||
def _short(value) -> str:
|
||||
return safe(display(json.dumps(value, ensure_ascii=False), 60))
|
||||
|
||||
|
||||
def format_errors(errors: list[dict]) -> str:
|
||||
lines = []
|
||||
for e in errors:
|
||||
where = f"Zeile {e['line']}: " if e.get("line") else ""
|
||||
path = f"{safe(e['path'])}: " if e.get("path") else ""
|
||||
detail = ""
|
||||
params = {k: v for k, v in (e.get("params") or {}).items() if k in ("value", "field", "key", "host", "group", "max", "probes")}
|
||||
if params:
|
||||
detail = " (" + ", ".join(f"{k}={safe(v)}" for k, v in params.items()) + ")"
|
||||
lines.append(f" {where}{path}{ERRORS_DE.get(e['code'], e['code'])}{detail}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_plan(public: dict) -> str:
|
||||
lines = []
|
||||
for item in public["items"]:
|
||||
action = item["action"]
|
||||
planned = item["planned_action"]
|
||||
name = f"{item['kind']} {safe(item['key'])}"
|
||||
suffix = "" if item["deployed_hosts"] == 0 else f" [auf {item['deployed_hosts']} Firewall(s) ausgerollt]"
|
||||
if action == "blocked":
|
||||
reasons = "; ".join(REASONS_DE.get(r, r) for r in item["reasons"])
|
||||
lines.append(f"! {name} blockiert {LABEL.get(planned, '(ändern)')}: {reasons}")
|
||||
elif planned in LABEL:
|
||||
lines.append(f"{SYMBOL[planned]} {name} {LABEL[planned]}{suffix}")
|
||||
else:
|
||||
for change in item["changes"] or [{"field": "", "before": None, "after": None}]:
|
||||
if change["field"]:
|
||||
lines.append(f"~ {name} {safe(change['field'])}: {_short(change['before'])} → {_short(change['after'])}{suffix}")
|
||||
else:
|
||||
lines.append(f"~ {name}{suffix}")
|
||||
if item.get("implicit"):
|
||||
lines[-1] += " (Folge der Reihenfolge)"
|
||||
for warning in public["warnings"]:
|
||||
template = WARNINGS_DE.get(warning["code"])
|
||||
params = {k: safe(v) for k, v in (warning.get("params") or {}).items()}
|
||||
text = template.format(**params) if template else warning["code"]
|
||||
where = f"{safe(warning['path'])}: " if warning.get("path") else ""
|
||||
lines.append(f"? {where}{text}")
|
||||
s = public["summary"]
|
||||
lines.append(f"Plan: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt, {s['unchanged']} unverändert, {s['blocked']} blockiert")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _read_stdin() -> bytes:
|
||||
data = sys.stdin.buffer.read(MAX_BYTES + 1)
|
||||
return data
|
||||
|
||||
|
||||
async def run(argv: list[str], stdin: bytes | None = None) -> tuple[int, str, str]:
|
||||
"""(exit code, stdout, stderr). Separate from main() so tests can call it inside their event loop."""
|
||||
parser = argparse.ArgumentParser(prog="python -m app.config_code", add_help=False)
|
||||
parser.add_argument("command", choices=["export", "plan", "apply", "schema"])
|
||||
parser.add_argument("--kinds", default=None)
|
||||
parser.add_argument("--prune", action="store_true")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--plan-token", default=None)
|
||||
try:
|
||||
args = parser.parse_args(argv)
|
||||
except SystemExit:
|
||||
return 1, "", "Aufruf: python -m app.config_code {export|plan|apply|schema}\n"
|
||||
if args.command == "schema":
|
||||
return 0, schema.json_schema_text(), ""
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.services import audit_service
|
||||
|
||||
context = audit_service.context.set({"actor": ACTOR, "id": str(uuid.uuid4()), "action": args.command})
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
if args.command == "export":
|
||||
wanted = None
|
||||
if args.kinds:
|
||||
wanted = {k.strip() for k in args.kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
return 1, "", f"Unbekannte Art in --kinds. Möglich: {', '.join(schema.KINDS)}\n"
|
||||
return 0, await export_yaml(db, wanted), ""
|
||||
|
||||
raw = stdin if stdin is not None else _read_stdin()
|
||||
try:
|
||||
parsed = parse(raw)
|
||||
if args.command == "plan":
|
||||
plan = await planner.build_plan(db, parsed, args.prune)
|
||||
await db.rollback()
|
||||
public = plan.public()
|
||||
out = json.dumps(public, ensure_ascii=False, indent=2) + "\n" if args.json else (
|
||||
format_plan(public) + f"\n{TOKEN_MARK}{plan.token}\n")
|
||||
s = public["summary"]
|
||||
if not plan.applicable:
|
||||
return 4, out, ""
|
||||
return (2 if s["create"] + s["update"] + s["delete"] else 0), out, ""
|
||||
|
||||
if not args.plan_token:
|
||||
return 1, "", "apply braucht --plan-token (opnexus apply zeigt erst den Plan und fragt dann).\n"
|
||||
try:
|
||||
plan = await applier.apply_config(db, parsed, args.prune, args.plan_token)
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
return 4, "", "Seit dem Plan hat sich etwas geändert. Nichts wurde übernommen. Bitte neu planen.\n"
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
return 4, format_plan(exc.plan.public()) + "\n", "Der Plan enthält blockierte Objekte. Nichts wurde übernommen.\n"
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
return 4, "", f"Selbstprüfung fehlgeschlagen, zurückgenommen: {', '.join(exc.differences[:10])}\n"
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
s = plan.summary()
|
||||
if args.json:
|
||||
return 0, json.dumps({"summary": s, "sha256": parsed.sha256}, ensure_ascii=False) + "\n", ""
|
||||
return 0, (f"Übernommen: {s['create']} neu, {s['update']} geändert, {s['delete']} entfernt. "
|
||||
"Es wurde nichts auf Firewalls geschrieben: die Entwürfe jetzt über Vorschau → Ausrollen → Bestätigen ausrollen.\n"), ""
|
||||
except ConfigError as exc:
|
||||
if args.json:
|
||||
return 3, json.dumps({"code": exc.code, "errors": exc.errors}, ensure_ascii=False, indent=2) + "\n", ""
|
||||
return 3, "", f"Die Datei ist ungültig:\n{format_errors(exc.errors)}\n"
|
||||
finally:
|
||||
audit_service.context.reset(context)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
try:
|
||||
code, out, err = asyncio.run(run(argv))
|
||||
except Exception as exc: # noqa: BLE001 the host CLI shows one line, details are in the container log
|
||||
sys.stderr.write(f"Fehler: {safe(exc)[:300]}\n")
|
||||
return 1
|
||||
sys.stdout.write(out)
|
||||
sys.stderr.write(err)
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,312 @@
|
||||
"""Apply: einen geprüften Plan in einer Transaktion in die Datenbank schreiben. Nie auf eine Firewall.
|
||||
|
||||
Ablauf: `policy_lock`, Plan neu berechnen, Token vergleichen (sonst PlanStale), blockierte Pläne ablehnen, schreiben
|
||||
(Hierarchiegruppen → Gerätegruppen → Aliase → Regeln → NAT → Hierarchie-Regeln → Host-Zuordnung → Variablen,
|
||||
Löschungen umgekehrt), Revisionen und Audit wie in der UI, dann Selbstprüfung: die Datenbank muss danach genau dem
|
||||
erwarteten Zustand des Plans entsprechen, sonst SelfCheckFailed (der Aufrufer rollt zurück).
|
||||
|
||||
Dieses Modul importiert bewusst weder `opnsense_client` noch einen `*_deploy_service` (test_config_code_apply prüft das).
|
||||
Es legt keine Deployment-Zeile an und ruft kein `undeploy_*`: gelöscht werden nur Objekte ohne jedes Deployment.
|
||||
Den Commit macht der Aufrufer.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import Parsed
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import (AliasRevision, DeviceGroup, OneToOneRevision, OutboundNatRevision,
|
||||
PolicyHierarchyRuleRevision, PortForwardRevision, RuleRevision)
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.services import audit_service
|
||||
|
||||
NOTE = "Import (Konfiguration als Code)"
|
||||
AUDIT_KIND = {
|
||||
"device_groups": "group", "policy_hierarchy_groups": "policy_hierarchy_group", "aliases": "alias", "rules": "rule",
|
||||
"port_forwards": "port_forward", "one_to_one": "one_to_one", "outbound_nat": "outbound_nat",
|
||||
"policy_hierarchy_rules": "policy_hierarchy", "variables": "variable",
|
||||
}
|
||||
|
||||
|
||||
class PlanStale(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PlanBlocked(Exception):
|
||||
def __init__(self, plan: planner.Plan):
|
||||
self.plan = plan
|
||||
super().__init__("plan blocked")
|
||||
|
||||
|
||||
class SelfCheckFailed(Exception):
|
||||
def __init__(self, differences: list[str]):
|
||||
self.differences = differences
|
||||
super().__init__("; ".join(differences[:5]))
|
||||
|
||||
|
||||
def _revision_writer(kind: str):
|
||||
"""(revision model, foreign key, snapshot function) as the UI uses them, imported lazily per kind.
|
||||
|
||||
The *_policy_service modules import their deploy services (for previews); importing them only here keeps the
|
||||
import of this module free of any firewall client, and nothing here calls into them except `snapshot`.
|
||||
"""
|
||||
if kind == "rules":
|
||||
from app.services.policy_service import snapshot
|
||||
return RuleRevision, "rule_id", snapshot
|
||||
if kind == "aliases":
|
||||
from app.services.alias_policy_service import snapshot
|
||||
return AliasRevision, "alias_id", snapshot
|
||||
if kind == "port_forwards":
|
||||
from app.services.port_forward_policy_service import snapshot
|
||||
return PortForwardRevision, "port_forward_id", snapshot
|
||||
if kind == "one_to_one":
|
||||
from app.services.one_to_one_policy_service import snapshot
|
||||
return OneToOneRevision, "one_to_one_id", snapshot
|
||||
if kind == "outbound_nat":
|
||||
from app.services.outbound_nat_policy_service import snapshot
|
||||
return OutboundNatRevision, "outbound_nat_id", snapshot
|
||||
from app.services.policy_hierarchy_service import snapshot
|
||||
return PolicyHierarchyRuleRevision, "policy_hierarchy_id", snapshot
|
||||
|
||||
|
||||
MODELS = {"aliases": FirewallAlias, "rules": FirewallRule, "port_forwards": PortForwardRule, "one_to_one": OneToOneRule,
|
||||
"outbound_nat": OutboundNatRule, "policy_hierarchy_rules": PolicyHierarchyRule}
|
||||
SCALARS = {
|
||||
"aliases": ("name", "type", "description", "enabled"),
|
||||
"rules": ("description", "interface", "direction", "ipprotocol", "protocol", "action", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "enabled", "log"),
|
||||
"port_forwards": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "target", "local_port", "natreflection", "enabled"),
|
||||
"one_to_one": ("description", "interface", "type", "external", "source_net", "destination_net", "source_not",
|
||||
"destination_not", "natreflection", "log", "enabled"),
|
||||
"outbound_nat": ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port", "destination_net",
|
||||
"destination_port", "source_not", "destination_not", "target", "target_port", "enabled"),
|
||||
"policy_hierarchy_rules": ("rulebase", "description", "interface", "direction", "ipprotocol", "protocol", "action",
|
||||
"source_net", "source_port", "destination_net", "destination_port", "enabled"),
|
||||
}
|
||||
|
||||
|
||||
class _Writer:
|
||||
def __init__(self, db: AsyncSession, plan: planner.Plan):
|
||||
self.db = db
|
||||
self.plan = plan
|
||||
self.now = datetime.now(timezone.utc)
|
||||
self.events: list[dict] = []
|
||||
|
||||
async def load(self) -> None:
|
||||
db = self.db
|
||||
self.hosts = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
self.groups = {g.name: g for g in (await db.execute(
|
||||
select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()}
|
||||
self.hierarchy = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
self.rows: dict[str, dict] = {}
|
||||
for kind, model in MODELS.items():
|
||||
options = [selectinload(model.deployments)]
|
||||
if kind != "policy_hierarchy_rules":
|
||||
options += [selectinload(model.targets), selectinload(model.groups)]
|
||||
rows = (await db.execute(select(model).options(*options))).scalars().all()
|
||||
self.rows[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
|
||||
def items(self, kind: str, *actions: str) -> list[planner.Item]:
|
||||
return [i for i in self.plan.items if i.kind == kind and i.action in actions]
|
||||
|
||||
def event(self, item: planner.Item, entity_id, name: str) -> None:
|
||||
self.events.append({"kind": AUDIT_KIND[item.kind], "action": f"import_{item.action}", "status": "success",
|
||||
"entity_id": str(entity_id) if entity_id else None, "entity_name": name,
|
||||
"details": {"key": item.key, "fields": [c["field"] for c in item.changes]}})
|
||||
|
||||
# -- groups ----------------------------------------------------------------------------------------------------------
|
||||
async def hierarchy_groups(self) -> None:
|
||||
items = self.items("policy_hierarchy_groups", "create", "update")
|
||||
for item in items:
|
||||
if item.action == "create":
|
||||
group = PolicyHierarchyGroup(name=item.key, description=item.after["description"])
|
||||
self.db.add(group)
|
||||
self.hierarchy[item.key] = group
|
||||
await self.db.flush()
|
||||
for item in items:
|
||||
group = self.hierarchy[item.key]
|
||||
group.description = item.after["description"]
|
||||
parent = item.after["parent"]
|
||||
group.parent_id = self.hierarchy[parent].id if parent else None
|
||||
self.event(item, group.id, group.name)
|
||||
await self.db.flush()
|
||||
|
||||
async def device_groups(self) -> None:
|
||||
for item in self.items("device_groups", "create", "update"):
|
||||
group = self.groups.get(item.key)
|
||||
if group is None:
|
||||
group = DeviceGroup(name=item.key)
|
||||
self.db.add(group)
|
||||
self.groups[item.key] = group
|
||||
group.description = item.after["description"]
|
||||
group.hosts = [self.hosts[n] for n in item.after["hosts"]]
|
||||
await self.db.flush()
|
||||
self.event(item, group.id, group.name)
|
||||
|
||||
# -- objects ---------------------------------------------------------------------------------------------------------
|
||||
async def objects(self, kind: str) -> None:
|
||||
model = MODELS[kind]
|
||||
revision_model, fk, snapshot = _revision_writer(kind)
|
||||
for item in self.items(kind, "create", "update"):
|
||||
after = item.after
|
||||
row = self.rows[kind].get(item.key)
|
||||
if row is None:
|
||||
row = model(targets=[], groups=[]) if kind != "policy_hierarchy_rules" else model()
|
||||
if kind != "aliases":
|
||||
row.config_key = item.key
|
||||
self.db.add(row)
|
||||
self.rows[kind][item.key] = row
|
||||
elif not await self.db.scalar(select(revision_model.id).where(getattr(revision_model, fk) == row.id).limit(1)):
|
||||
self.db.add(revision_model(**{fk: row.id}, note="Vor erster Bearbeitung", snapshot=snapshot(row)))
|
||||
for name in SCALARS[kind]:
|
||||
setattr(row, name, after[name])
|
||||
if kind == "aliases":
|
||||
row.content = "\n".join(after["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
row.hierarchy_group_id = self.hierarchy[after["group"]].id
|
||||
else:
|
||||
row.targets = [self.hosts[n] for n in after["targets"]["hosts"]]
|
||||
row.groups = [self.groups[n] for n in after["targets"]["groups"]]
|
||||
if "hierarchy_group" in after["targets"]:
|
||||
hierarchy = after["targets"]["hierarchy_group"]
|
||||
row.policy_hierarchy_group_id = self.hierarchy[hierarchy].id if hierarchy else None
|
||||
row.updated_at = self.now # also for target-only changes: invalidates optimistic edits open in the UI
|
||||
await self.db.flush()
|
||||
self.db.add(revision_model(**{fk: row.id}, note=NOTE, snapshot=snapshot(row)))
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
|
||||
async def hierarchy_positions(self) -> None:
|
||||
"""sequence = rank for every rule in a list the file touches (contiguous 0..n-1, same order as the plan)."""
|
||||
if "policy_hierarchy_rules" not in self.plan.touched:
|
||||
return
|
||||
deleted = {i.key for i in self.items("policy_hierarchy_rules", "delete")}
|
||||
for key, rank in self.plan.positions.items():
|
||||
row = self.rows["policy_hierarchy_rules"].get(key)
|
||||
if row is not None and key not in deleted and row.sequence != rank:
|
||||
row.sequence = rank
|
||||
await self.db.flush()
|
||||
|
||||
async def host_hierarchy(self) -> None:
|
||||
for host_name, group_name in self.plan.host_hierarchy.items():
|
||||
host = self.hosts[host_name]
|
||||
wanted = self.hierarchy[group_name].id if group_name else None
|
||||
if host.policy_hierarchy_group_id != wanted:
|
||||
host.policy_hierarchy_group_id = wanted
|
||||
await self.db.flush()
|
||||
|
||||
async def variables(self) -> None:
|
||||
if "variables" not in self.plan.touched:
|
||||
return
|
||||
existing = {(v.host_id, v.name): v for v in (await self.db.execute(select(HostVariable))).scalars().all()}
|
||||
for item in self.items("variables", "create", "update"):
|
||||
host = self.hosts[item.after["host"]]
|
||||
row = existing.get((host.id, item.after["name"]))
|
||||
if row is None:
|
||||
self.db.add(HostVariable(host_id=host.id, name=item.after["name"], value=item.after["value"]))
|
||||
else:
|
||||
row.value = item.after["value"]
|
||||
self.event(item, None, item.key)
|
||||
for item in self.items("variables", "delete"):
|
||||
host = self.hosts[item.before["host"]]
|
||||
await self.db.execute(delete(HostVariable).where(HostVariable.host_id == host.id, HostVariable.name == item.before["name"]))
|
||||
self.event(item, None, item.key)
|
||||
await self.db.flush()
|
||||
|
||||
# -- deletions -------------------------------------------------------------------------------------------------------
|
||||
async def deletions(self) -> None:
|
||||
for kind in ("policy_hierarchy_rules", "outbound_nat", "one_to_one", "port_forwards", "rules", "aliases"):
|
||||
for item in self.items(kind, "delete"):
|
||||
row = self.rows[kind][item.key]
|
||||
if row.deployments: # the planner blocks this; never undeploy from here
|
||||
raise RuntimeError(f"{kind} {item.key}: delete with deployments")
|
||||
self.event(item, row.id, row.name if kind == "aliases" else row.description)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
for item in self.items("device_groups", "delete"):
|
||||
row = self.groups[item.key]
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
doomed = [self.hierarchy[i.key] for i in self.items("policy_hierarchy_groups", "delete")]
|
||||
by_id = {g.id: g for g in self.hierarchy.values()}
|
||||
|
||||
def depth(group) -> int:
|
||||
n, current = 0, group
|
||||
while current.parent_id is not None and current.parent_id in by_id and n < 1000:
|
||||
n, current = n + 1, by_id[current.parent_id]
|
||||
return n
|
||||
|
||||
for row in sorted(doomed, key=depth, reverse=True): # children before parents (parent_id is RESTRICT)
|
||||
item = next(i for i in self.items("policy_hierarchy_groups", "delete") if i.key == row.name)
|
||||
self.event(item, row.id, row.name)
|
||||
await self.db.delete(row)
|
||||
await self.db.flush()
|
||||
|
||||
|
||||
async def _self_check(db: AsyncSession, plan: planner.Plan) -> None:
|
||||
db.expire_all() # compare what is in the database, not what the session believes
|
||||
actual = await canonical(db, set(plan.touched))
|
||||
differences: list[str] = []
|
||||
for kind in plan.touched:
|
||||
if kind == "variables":
|
||||
got = {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in actual["variables"].items() for n, v in values.items()}
|
||||
else:
|
||||
ident = planner.KIND_SPECS[kind].ident
|
||||
got = {item[ident]: item for item in actual[kind]}
|
||||
want = plan.expected[kind]
|
||||
for key in sorted(set(got) | set(want)):
|
||||
if got.get(key) != want.get(key):
|
||||
differences.append(f"{kind}:{key}")
|
||||
if "policy_hierarchy_rules" in plan.touched:
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.group)))).scalars().all()
|
||||
ranks = planner._ranks(rows, lambda r: r.config_key, lambda r: (r.group.name, r.rulebase), lambda r: (r.sequence, str(r.id)))
|
||||
for key, rank in ranks.items():
|
||||
if plan.positions.get(key) != rank:
|
||||
differences.append(f"policy_hierarchy_rules:{key}:position")
|
||||
if differences:
|
||||
raise SelfCheckFailed(differences)
|
||||
|
||||
|
||||
async def apply_config(db: AsyncSession, parsed: Parsed, prune: bool, token: str) -> planner.Plan:
|
||||
"""Writes the plan into the session (flushed, not committed). Raises PlanStale, PlanBlocked, SelfCheckFailed or
|
||||
ConfigError; on any exception the caller must roll back."""
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
await policy_lock(db)
|
||||
plan = await planner.build_plan(db, parsed, prune)
|
||||
if plan.token != token:
|
||||
raise PlanStale()
|
||||
if not plan.applicable:
|
||||
raise PlanBlocked(plan)
|
||||
writer = _Writer(db, plan)
|
||||
await writer.load()
|
||||
await writer.hierarchy_groups()
|
||||
await writer.device_groups()
|
||||
for kind in ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules"):
|
||||
if kind in plan.touched:
|
||||
await writer.objects(kind)
|
||||
await writer.hierarchy_positions()
|
||||
await writer.host_hierarchy()
|
||||
await writer.variables()
|
||||
await writer.deletions()
|
||||
await _self_check(db, plan)
|
||||
db.add(audit_service.make_event(kind="config", action="import", status="success", entity_name=parsed.sha256[:16],
|
||||
details={"sha256": parsed.sha256, "prune": prune, "kinds": plan.touched,
|
||||
"summary": plan.summary()}))
|
||||
for values in writer.events:
|
||||
db.add(audit_service.make_event(**values))
|
||||
await db.flush()
|
||||
return plan
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Kanonische Darstellung der zentralen Objekte und deren YAML-Text.
|
||||
|
||||
`canonical(db)` liefert dieselbe Form, die der Lader aus einer Datei erzeugt (Feldnamen und -folge des Schemas). Der
|
||||
Planer vergleicht beide; der Export schreibt sie nur aus. Deterministisch: feste Abschnittsfolge, Objekte nach Schlüssel,
|
||||
Ziel- und Mitgliederlisten sortiert, alle Felder immer (auch Defaults und `null`), kein Zeitstempel.
|
||||
"""
|
||||
import yaml
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.version import APP_VERSION
|
||||
|
||||
HEADER = (
|
||||
"# OPNexus-Konfiguration (Schema v{schema}), exportiert von OPNexus {version}.\n"
|
||||
"# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run\n"
|
||||
)
|
||||
|
||||
|
||||
def _ordered(doc: type[schema._Doc], values: dict) -> dict:
|
||||
"""Values in the field order of the schema model; a missing or extra field is a bug in the exporter."""
|
||||
fields = list(doc.model_fields)
|
||||
if set(values) != set(fields):
|
||||
raise RuntimeError(f"{doc.__name__}: exporter fields differ from schema ({sorted(set(values) ^ set(fields))})")
|
||||
return {name: values[name] for name in fields}
|
||||
|
||||
|
||||
def _targets(doc: type[schema._Doc], obj, hierarchy_names: dict) -> dict:
|
||||
values = {"hosts": sorted(h.name for h in obj.targets), "groups": sorted(g.name for g in obj.groups)}
|
||||
if "hierarchy_group" in doc.model_fields:
|
||||
values["hierarchy_group"] = hierarchy_names.get(obj.policy_hierarchy_group_id)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _flat(doc: type[schema._Doc], obj, **extra) -> dict:
|
||||
"""Scalar columns of `obj` for every schema field that is not computed here (`key` comes from `config_key`)."""
|
||||
values = dict(extra)
|
||||
for name in doc.model_fields:
|
||||
if name in values:
|
||||
continue
|
||||
values[name] = obj.config_key if name == "key" else getattr(obj, name)
|
||||
return _ordered(doc, values)
|
||||
|
||||
|
||||
def _hierarchy_order(groups: list[PolicyHierarchyGroup]) -> list[PolicyHierarchyGroup]:
|
||||
"""Parents before children, siblings by name; the same rule the loader expects."""
|
||||
children: dict = {}
|
||||
for group in groups:
|
||||
children.setdefault(group.parent_id, []).append(group)
|
||||
ordered: list[PolicyHierarchyGroup] = []
|
||||
|
||||
def walk(parent_id) -> None:
|
||||
for group in sorted(children.get(parent_id, []), key=lambda g: g.name):
|
||||
ordered.append(group)
|
||||
walk(group.id)
|
||||
|
||||
walk(None)
|
||||
return ordered
|
||||
|
||||
|
||||
async def canonical(db: AsyncSession, kinds: set[str] | None = None) -> dict[str, list | dict]:
|
||||
"""{kind: content} for the requested kinds (default all), in file order."""
|
||||
wanted = set(schema.KINDS) if kinds is None else kinds
|
||||
out: dict[str, list | dict] = {}
|
||||
|
||||
all_groups = (await db.execute(select(PolicyHierarchyGroup).options(selectinload(PolicyHierarchyGroup.hosts)))).scalars().all()
|
||||
hierarchy_names = {g.id: g.name for g in all_groups}
|
||||
|
||||
if "device_groups" in wanted:
|
||||
groups = (await db.execute(select(DeviceGroup).options(selectinload(DeviceGroup.hosts)))).scalars().all()
|
||||
out["device_groups"] = [
|
||||
_ordered(schema.DeviceGroupDoc, {"name": g.name, "description": g.description or "", "hosts": sorted(h.name for h in g.hosts)})
|
||||
for g in sorted(groups, key=lambda g: g.name)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_groups" in wanted:
|
||||
out["policy_hierarchy_groups"] = [
|
||||
_ordered(schema.HierarchyGroupDoc, {
|
||||
"name": g.name,
|
||||
"parent": hierarchy_names.get(g.parent_id),
|
||||
"description": g.description or "",
|
||||
"hosts": sorted(h.name for h in g.hosts),
|
||||
})
|
||||
for g in _hierarchy_order(list(all_groups))
|
||||
]
|
||||
|
||||
if "aliases" in wanted:
|
||||
aliases = (await db.execute(
|
||||
select(FirewallAlias).options(selectinload(FirewallAlias.targets), selectinload(FirewallAlias.groups))
|
||||
)).scalars().all()
|
||||
out["aliases"] = [
|
||||
_ordered(schema.AliasDoc, {
|
||||
"name": a.name,
|
||||
"type": a.type,
|
||||
"description": a.description or "",
|
||||
"enabled": a.enabled,
|
||||
"entries": [line for line in (a.content or "").split("\n") if line],
|
||||
"targets": _targets(schema.TargetsWithHierarchy, a, hierarchy_names),
|
||||
})
|
||||
for a in sorted(aliases, key=lambda a: a.name)
|
||||
]
|
||||
|
||||
simple = (
|
||||
("rules", FirewallRule, schema.RuleDoc, schema.Targets),
|
||||
("port_forwards", PortForwardRule, schema.PortForwardDoc, schema.TargetsWithHierarchy),
|
||||
("one_to_one", OneToOneRule, schema.OneToOneDoc, schema.TargetsWithHierarchy),
|
||||
("outbound_nat", OutboundNatRule, schema.OutboundNatDoc, schema.TargetsWithHierarchy),
|
||||
)
|
||||
for kind, model, doc, targets_doc in simple:
|
||||
if kind not in wanted:
|
||||
continue
|
||||
rows = (await db.execute(select(model).options(selectinload(model.targets), selectinload(model.groups)))).scalars().all()
|
||||
out[kind] = [
|
||||
_flat(doc, row, targets=_targets(targets_doc, row, hierarchy_names))
|
||||
for row in sorted(rows, key=lambda r: r.config_key)
|
||||
]
|
||||
|
||||
if "policy_hierarchy_rules" in wanted:
|
||||
rules = (await db.execute(select(PolicyHierarchyRule))).scalars().all()
|
||||
group_order = {g.id: i for i, g in enumerate(_hierarchy_order(list(all_groups)))}
|
||||
out["policy_hierarchy_rules"] = [
|
||||
_flat(schema.HierarchyRuleDoc, r, group=hierarchy_names[r.hierarchy_group_id])
|
||||
for r in sorted(rules, key=lambda r: (group_order[r.hierarchy_group_id], r.rulebase != "pre", r.sequence, str(r.id))) # deploy order
|
||||
]
|
||||
|
||||
if "variables" in wanted:
|
||||
hosts = (await db.execute(select(OpnsenseHost).options(selectinload(OpnsenseHost.variables)))).scalars().all()
|
||||
out["variables"] = {
|
||||
h.name: {v.name: v.value for v in sorted(h.variables, key=lambda v: v.name)}
|
||||
for h in sorted(hosts, key=lambda h: h.name)
|
||||
if h.variables
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def dump_yaml(content: dict[str, list | dict]) -> str:
|
||||
"""The file text: fixed header, `opnexus_config: 1`, then the sections in schema order."""
|
||||
document = {"opnexus_config": schema.SCHEMA_VERSION, **{kind: content[kind] for kind in schema.KINDS if kind in content}}
|
||||
body = yaml.safe_dump(document, sort_keys=False, default_flow_style=False, allow_unicode=True, width=4096)
|
||||
return HEADER.format(schema=schema.SCHEMA_VERSION, version=APP_VERSION) + body
|
||||
|
||||
|
||||
async def export_yaml(db: AsyncSession, kinds: set[str] | None = None) -> str:
|
||||
return dump_yaml(await canonical(db, kinds))
|
||||
@@ -0,0 +1,306 @@
|
||||
"""Strikter YAML-Lader für Konfigurationsdateien (Sicherheitsfläche: die Datei kommt von außen).
|
||||
|
||||
Erlaubt ist nur das, was der Exporter selbst schreibt: Mappings, Listen und Skalare. Verboten und mit eigenem Code
|
||||
gemeldet: Anker/Aliase (Billion Laughs), Merge-Keys, explizite Tags (`!!python/...`, `!!binary`, `!`), Direktiven,
|
||||
mehrere Dokumente, doppelte oder nicht-textuelle Mapping-Schlüssel. Implizite Typen nur wie im YAML-1.2-Core-Schema
|
||||
eingeschränkt: `true/false`, `null/~`/leer und Dezimal-Ganzzahlen; `on`, `yes`, `1.5`, `2026-10-22`, `0755`, `1:30` bleiben
|
||||
Text. Danach prüft das Schema (schema.ConfigDoc) Felder, Typen, Längen und Steuerzeichen.
|
||||
|
||||
Reines Python (`yaml.SafeLoader`, nicht `CSafeLoader`): nur dort greifen die Hooks unten.
|
||||
"""
|
||||
import hashlib
|
||||
import re
|
||||
from typing import NamedTuple
|
||||
|
||||
import yaml
|
||||
from pydantic import ValidationError
|
||||
from yaml.composer import Composer
|
||||
from yaml.constructor import SafeConstructor
|
||||
from yaml.events import AliasEvent, DocumentStartEvent, StreamEndEvent
|
||||
from yaml.nodes import MappingNode, ScalarNode, SequenceNode
|
||||
from yaml.parser import Parser
|
||||
from yaml.reader import Reader
|
||||
from yaml.resolver import BaseResolver
|
||||
from yaml.scanner import Scanner
|
||||
|
||||
from app.config_code import schema
|
||||
|
||||
MAX_BYTES = 1024 * 1024
|
||||
MAX_DEPTH = 8
|
||||
MAX_NODES = 100_000
|
||||
MAX_SCALAR = schema.MAX_SCALAR
|
||||
MAX_OBJECTS_PER_KIND = 5000
|
||||
MAX_VARIABLE_HOSTS = 500
|
||||
MAX_VARIABLES_PER_HOST = 200 # wie routers/variables.py
|
||||
MAX_ALIAS_ENTRIES = 2000
|
||||
MAX_ALIAS_CONTENT = 8000 # wie schemas/alias.py `content`
|
||||
|
||||
_STR = "tag:yaml.org,2002:str"
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
"""The file is not acceptable. `errors` is a list of {code, path, line, params}; `code` is the overall reason."""
|
||||
|
||||
def __init__(self, code: str, errors: list[dict]):
|
||||
self.code = code
|
||||
self.errors = errors
|
||||
super().__init__(f"{code}: {errors[:3]}")
|
||||
|
||||
|
||||
def _err(code: str, path: str = "", line: int | None = None, **params) -> dict:
|
||||
return {"code": code, "path": path, "line": line, "params": params}
|
||||
|
||||
|
||||
def display(value, limit: int = 80) -> str:
|
||||
"""A value for an error message: shortened and with every control character escaped (no ANSI from foreign files)."""
|
||||
text = value if isinstance(value, str) else repr(value)
|
||||
text = text.encode("unicode_escape").decode("ascii")
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def _path(parts) -> str:
|
||||
out = ""
|
||||
for part in parts:
|
||||
if isinstance(part, int):
|
||||
out += f"[{part}]"
|
||||
else:
|
||||
out += ("." if out else "") + str(part)
|
||||
return out
|
||||
|
||||
|
||||
class _Rejected(yaml.YAMLError):
|
||||
def __init__(self, code: str, mark=None, path=(), **params):
|
||||
self.code, self.mark, self.path, self.params = code, mark, tuple(path), params
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
class StrictResolver(BaseResolver):
|
||||
yaml_implicit_resolvers: dict = {}
|
||||
|
||||
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:bool", re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), list("tTfF"))
|
||||
# Dezimal ohne führende Null und höchstens 18 Stellen: int() großer Zahlen kostet Rechenzeit (und Python begrenzt die Länge).
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:int", re.compile(r"^(?:0|-?[1-9][0-9]{0,17})$"), list("-0123456789"))
|
||||
StrictResolver.add_implicit_resolver(
|
||||
"tag:yaml.org,2002:null", re.compile(r"^(?:~|null|Null|NULL|)$"), ["~", "n", "N", ""])
|
||||
|
||||
|
||||
class StrictConstructor(SafeConstructor):
|
||||
yaml_constructors: dict = {}
|
||||
|
||||
|
||||
for _tag, _fn in (
|
||||
("tag:yaml.org,2002:null", SafeConstructor.construct_yaml_null),
|
||||
("tag:yaml.org,2002:bool", SafeConstructor.construct_yaml_bool),
|
||||
("tag:yaml.org,2002:int", SafeConstructor.construct_yaml_int),
|
||||
(_STR, SafeConstructor.construct_yaml_str),
|
||||
("tag:yaml.org,2002:seq", SafeConstructor.construct_yaml_seq),
|
||||
("tag:yaml.org,2002:map", SafeConstructor.construct_yaml_map),
|
||||
):
|
||||
StrictConstructor.add_constructor(_tag, _fn)
|
||||
|
||||
|
||||
class StrictLoader(Reader, Scanner, Parser, Composer, StrictConstructor, StrictResolver):
|
||||
def __init__(self, stream: str):
|
||||
Reader.__init__(self, stream)
|
||||
Scanner.__init__(self)
|
||||
Parser.__init__(self)
|
||||
Composer.__init__(self)
|
||||
StrictConstructor.__init__(self)
|
||||
StrictResolver.__init__(self)
|
||||
self._depth = 0
|
||||
self._nodes = 0
|
||||
self._path: list = []
|
||||
self.lines: dict[tuple, int] = {} # path -> 1-based line, for messages from the schema check
|
||||
|
||||
# -- document level ---------------------------------------------------------------------------------------------
|
||||
def get_single_node(self):
|
||||
self.get_event() # StreamStart
|
||||
if self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_empty")
|
||||
start = self.peek_event()
|
||||
if isinstance(start, DocumentStartEvent) and (start.version is not None or start.tags):
|
||||
raise _Rejected("yaml_directive_forbidden", start.start_mark)
|
||||
node = self.compose_document()
|
||||
if not self.check_event(StreamEndEvent):
|
||||
raise _Rejected("yaml_multiple_documents", self.peek_event().start_mark)
|
||||
self.get_event()
|
||||
return node
|
||||
|
||||
# -- node level ---------------------------------------------------------------------------------------------------
|
||||
def compose_node(self, parent, index):
|
||||
event = self.peek_event()
|
||||
if isinstance(event, AliasEvent):
|
||||
raise _Rejected("yaml_alias_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "anchor", None) is not None:
|
||||
raise _Rejected("yaml_anchor_forbidden", event.start_mark, self._path)
|
||||
if getattr(event, "tag", None) is not None:
|
||||
raise _Rejected("yaml_tag_forbidden", event.start_mark, self._path, tag=display(event.tag))
|
||||
self._nodes += 1
|
||||
if self._nodes > MAX_NODES:
|
||||
raise _Rejected("too_many_nodes", event.start_mark, self._path, max=MAX_NODES)
|
||||
|
||||
if isinstance(index, ScalarNode):
|
||||
segment = index.value
|
||||
elif isinstance(index, int) and not isinstance(index, bool):
|
||||
segment = index
|
||||
else:
|
||||
segment = None # a mapping key
|
||||
if segment is not None:
|
||||
self._path.append(segment)
|
||||
self.lines.setdefault(tuple(self._path), event.start_mark.line + 1)
|
||||
self._depth += 1
|
||||
try:
|
||||
if self._depth > MAX_DEPTH:
|
||||
raise _Rejected("too_deep", event.start_mark, self._path, max=MAX_DEPTH)
|
||||
return super().compose_node(parent, index)
|
||||
finally:
|
||||
self._depth -= 1
|
||||
if segment is not None:
|
||||
self._path.pop()
|
||||
|
||||
def compose_scalar_node(self, anchor):
|
||||
event = self.peek_event()
|
||||
if len(event.value) > MAX_SCALAR:
|
||||
raise _Rejected("scalar_too_long", event.start_mark, self._path, max=MAX_SCALAR)
|
||||
return super().compose_scalar_node(anchor)
|
||||
|
||||
def compose_mapping_node(self, anchor):
|
||||
node = super().compose_mapping_node(anchor)
|
||||
seen: set[str] = set()
|
||||
for key, _value in node.value:
|
||||
if not isinstance(key, ScalarNode) or key.tag != _STR:
|
||||
raise _Rejected("non_string_key", key.start_mark, self._path)
|
||||
if key.value == "<<":
|
||||
raise _Rejected("yaml_merge_forbidden", key.start_mark, self._path)
|
||||
if key.value in seen:
|
||||
raise _Rejected("duplicate_key", key.start_mark, self._path, key=display(key.value))
|
||||
seen.add(key.value)
|
||||
return node
|
||||
|
||||
|
||||
def load_yaml(raw: bytes | str) -> tuple[object, dict[tuple, int]]:
|
||||
"""Raw bytes -> plain Python data (dict/list/str/int/bool/None) plus a path->line map. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
if len(raw) > MAX_BYTES:
|
||||
raise ConfigError("CONFIG_TOO_LARGE", [_err("too_large", max=MAX_BYTES)])
|
||||
try:
|
||||
text = raw.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", [_err("bad_encoding", offset=exc.start)]) from None
|
||||
if text.startswith(""):
|
||||
text = text[1:]
|
||||
loader = None
|
||||
try:
|
||||
loader = StrictLoader(text) # the Reader already rejects unprintable characters here
|
||||
node = loader.get_single_node()
|
||||
data = loader.construct_document(node)
|
||||
except _Rejected as exc:
|
||||
line = exc.mark.line + 1 if exc.mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err(exc.code, _path(exc.path), line, **exc.params)]) from None
|
||||
except yaml.MarkedYAMLError as exc:
|
||||
mark = exc.problem_mark or exc.context_mark
|
||||
line = mark.line + 1 if mark is not None else None
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", "", line, problem=display(exc.problem or ""))]) from None
|
||||
except yaml.YAMLError as exc: # e.g. ReaderError for unprintable characters
|
||||
raise ConfigError("CONFIG_INVALID", [_err("yaml_syntax", problem=display(str(exc)))]) from None
|
||||
finally:
|
||||
if loader is not None:
|
||||
loader.dispose()
|
||||
return data, loader.lines
|
||||
|
||||
|
||||
def _line_for(lines: dict[tuple, int], loc: tuple) -> int | None:
|
||||
for end in range(len(loc), 0, -1):
|
||||
if loc[:end] in lines:
|
||||
return lines[loc[:end]]
|
||||
return None
|
||||
|
||||
|
||||
_PYDANTIC_CODES = {
|
||||
"extra_forbidden": "unknown_field",
|
||||
"missing": "missing_field",
|
||||
"literal_error": "bad_value",
|
||||
"string_too_short": "too_short",
|
||||
"string_too_long": "too_long",
|
||||
"string_pattern_mismatch": "bad_format",
|
||||
}
|
||||
_OWN_CODES = {"control_character", "surrounding_whitespace", "bad_variable_name", "bad_variable_value"}
|
||||
|
||||
|
||||
def _schema_errors(exc: ValidationError, lines: dict[tuple, int]) -> list[dict]:
|
||||
out = []
|
||||
for error in exc.errors(include_url=False):
|
||||
loc = tuple(error["loc"])
|
||||
code = _PYDANTIC_CODES.get(error["type"])
|
||||
if code is None and error["type"] == "value_error":
|
||||
message = str(error.get("ctx", {}).get("error", ""))
|
||||
code = message if message in _OWN_CODES else "bad_value"
|
||||
if code is None:
|
||||
code = "bad_type"
|
||||
params = {}
|
||||
if error["type"] == "extra_forbidden":
|
||||
params["field"] = display(loc[-1]) if loc else ""
|
||||
elif code not in ("missing_field",) and "input" in error and not isinstance(error["input"], (dict, list)):
|
||||
params["value"] = display(error["input"])
|
||||
out.append(_err(code, _path(loc), _line_for(lines, loc), **params))
|
||||
return out
|
||||
|
||||
|
||||
def _check_counts(data: dict, lines: dict[tuple, int]) -> list[dict]:
|
||||
errors = []
|
||||
for kind in schema.KINDS:
|
||||
section = data.get(kind)
|
||||
if kind == "variables":
|
||||
if isinstance(section, dict):
|
||||
if len(section) > MAX_VARIABLE_HOSTS:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_VARIABLE_HOSTS))
|
||||
for host, values in section.items():
|
||||
if isinstance(values, dict) and len(values) > MAX_VARIABLES_PER_HOST:
|
||||
errors.append(_err("too_many_objects", _path((kind, host)), _line_for(lines, (kind, host)),
|
||||
max=MAX_VARIABLES_PER_HOST))
|
||||
continue
|
||||
if isinstance(section, list) and len(section) > MAX_OBJECTS_PER_KIND:
|
||||
errors.append(_err("too_many_objects", kind, _line_for(lines, (kind,)), max=MAX_OBJECTS_PER_KIND))
|
||||
for i, alias in enumerate(data.get("aliases") or []):
|
||||
entries = alias.get("entries") if isinstance(alias, dict) else None
|
||||
if isinstance(entries, list):
|
||||
loc = ("aliases", i, "entries")
|
||||
if len(entries) > MAX_ALIAS_ENTRIES:
|
||||
errors.append(_err("too_many_entries", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_ENTRIES))
|
||||
elif len("\n".join(str(e) for e in entries)) > MAX_ALIAS_CONTENT:
|
||||
errors.append(_err("too_long", _path(loc), _line_for(lines, loc), max=MAX_ALIAS_CONTENT))
|
||||
return errors
|
||||
|
||||
|
||||
class Parsed(NamedTuple):
|
||||
doc: schema.ConfigDoc
|
||||
sha256: str # of the raw bytes, shown in the plan and stored in the audit
|
||||
lines: dict[tuple, int] # path -> line, for messages of the planner
|
||||
|
||||
|
||||
def parse(raw: bytes | str) -> Parsed:
|
||||
"""Load, limit and validate a file. Raises ConfigError."""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("utf-8")
|
||||
sha256 = hashlib.sha256(raw).hexdigest()
|
||||
data, lines = load_yaml(raw)
|
||||
if not isinstance(data, dict):
|
||||
raise ConfigError("CONFIG_INVALID", [_err("not_a_mapping", line=1)])
|
||||
version = data.get("opnexus_config")
|
||||
if type(version) is not int or version != schema.SCHEMA_VERSION: # `true == 1` in Python
|
||||
code = "CONFIG_UNSUPPORTED_VERSION" if isinstance(version, int) and not isinstance(version, bool) else "CONFIG_INVALID"
|
||||
raise ConfigError(code, [_err("unsupported_version" if code != "CONFIG_INVALID" else "missing_version",
|
||||
"opnexus_config", _line_for(lines, ("opnexus_config",)),
|
||||
**({"value": display(version)} if version is not None else {}))])
|
||||
errors = _check_counts(data, lines)
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors)
|
||||
try:
|
||||
doc = schema.ConfigDoc.model_validate(data)
|
||||
except ValidationError as exc:
|
||||
raise ConfigError("CONFIG_INVALID", _schema_errors(exc, lines)[:100]) from None
|
||||
return Parsed(doc, sha256, lines)
|
||||
@@ -0,0 +1,933 @@
|
||||
{
|
||||
"$defs": {
|
||||
"AliasDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"pattern": "^[A-Za-z0-9_]+$",
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "host",
|
||||
"enum": [
|
||||
"host",
|
||||
"network",
|
||||
"port",
|
||||
"networkgroup",
|
||||
"mac",
|
||||
"asn"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 255,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"entries": {
|
||||
"items": {
|
||||
"maxLength": 8000,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Entries",
|
||||
"type": "array"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "AliasDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"DeviceGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "DeviceGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyGroupDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Name",
|
||||
"type": "string"
|
||||
},
|
||||
"parent": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Parent"
|
||||
},
|
||||
"description": {
|
||||
"default": "",
|
||||
"maxLength": 2000,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name"
|
||||
],
|
||||
"title": "HierarchyGroupDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"HierarchyRuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"group": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"title": "Group",
|
||||
"type": "string"
|
||||
},
|
||||
"rulebase": {
|
||||
"default": "pre",
|
||||
"enum": [
|
||||
"pre",
|
||||
"post"
|
||||
],
|
||||
"title": "Rulebase",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"group",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "HierarchyRuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OneToOneDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"default": "binat",
|
||||
"enum": [
|
||||
"binat",
|
||||
"nat"
|
||||
],
|
||||
"title": "Type",
|
||||
"type": "string"
|
||||
},
|
||||
"external": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "External",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"enable",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"external"
|
||||
],
|
||||
"title": "OneToOneDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"OutboundNatDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"source_not": {
|
||||
"default": false,
|
||||
"title": "Source Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"destination_not": {
|
||||
"default": false,
|
||||
"title": "Destination Not",
|
||||
"type": "boolean"
|
||||
},
|
||||
"target": {
|
||||
"default": "",
|
||||
"maxLength": 128,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"target_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Target Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "OutboundNatDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"PortForwardDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "tcp",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"target": {
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Target",
|
||||
"type": "string"
|
||||
},
|
||||
"local_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Local Port"
|
||||
},
|
||||
"natreflection": {
|
||||
"default": "",
|
||||
"enum": [
|
||||
"",
|
||||
"purenat",
|
||||
"disable"
|
||||
],
|
||||
"title": "Natreflection",
|
||||
"type": "string"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/TargetsWithHierarchy"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface",
|
||||
"target"
|
||||
],
|
||||
"title": "PortForwardDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"RuleDoc": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"key": {
|
||||
"pattern": "^[a-z0-9][a-z0-9._-]{0,63}$",
|
||||
"title": "Key",
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"maxLength": 255,
|
||||
"minLength": 1,
|
||||
"title": "Description",
|
||||
"type": "string"
|
||||
},
|
||||
"interface": {
|
||||
"maxLength": 64,
|
||||
"minLength": 1,
|
||||
"title": "Interface",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"default": "in",
|
||||
"enum": [
|
||||
"in",
|
||||
"out"
|
||||
],
|
||||
"title": "Direction",
|
||||
"type": "string"
|
||||
},
|
||||
"ipprotocol": {
|
||||
"default": "inet",
|
||||
"enum": [
|
||||
"inet",
|
||||
"inet6"
|
||||
],
|
||||
"title": "Ipprotocol",
|
||||
"type": "string"
|
||||
},
|
||||
"protocol": {
|
||||
"default": "any",
|
||||
"maxLength": 16,
|
||||
"minLength": 1,
|
||||
"title": "Protocol",
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"default": "pass",
|
||||
"enum": [
|
||||
"pass",
|
||||
"block",
|
||||
"reject"
|
||||
],
|
||||
"title": "Action",
|
||||
"type": "string"
|
||||
},
|
||||
"source_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Source Net",
|
||||
"type": "string"
|
||||
},
|
||||
"source_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Source Port"
|
||||
},
|
||||
"destination_net": {
|
||||
"default": "any",
|
||||
"maxLength": 128,
|
||||
"minLength": 1,
|
||||
"title": "Destination Net",
|
||||
"type": "string"
|
||||
},
|
||||
"destination_port": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 32,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Destination Port"
|
||||
},
|
||||
"enabled": {
|
||||
"default": true,
|
||||
"title": "Enabled",
|
||||
"type": "boolean"
|
||||
},
|
||||
"log": {
|
||||
"default": false,
|
||||
"title": "Log",
|
||||
"type": "boolean"
|
||||
},
|
||||
"targets": {
|
||||
"$ref": "#/$defs/Targets"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"key",
|
||||
"description",
|
||||
"interface"
|
||||
],
|
||||
"title": "RuleDoc",
|
||||
"type": "object"
|
||||
},
|
||||
"Targets": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"title": "Targets",
|
||||
"type": "object"
|
||||
},
|
||||
"TargetsWithHierarchy": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"hosts": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Hosts",
|
||||
"type": "array"
|
||||
},
|
||||
"groups": {
|
||||
"items": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Groups",
|
||||
"type": "array"
|
||||
},
|
||||
"hierarchy_group": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 120,
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Hierarchy Group"
|
||||
}
|
||||
},
|
||||
"title": "TargetsWithHierarchy",
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"description": "Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand.",
|
||||
"properties": {
|
||||
"opnexus_config": {
|
||||
"const": 1,
|
||||
"title": "Opnexus Config",
|
||||
"type": "integer"
|
||||
},
|
||||
"device_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/DeviceGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Device Groups"
|
||||
},
|
||||
"policy_hierarchy_groups": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyGroupDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Groups"
|
||||
},
|
||||
"aliases": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/AliasDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Aliases"
|
||||
},
|
||||
"rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/RuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Rules"
|
||||
},
|
||||
"port_forwards": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/PortForwardDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Port Forwards"
|
||||
},
|
||||
"one_to_one": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OneToOneDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "One To One"
|
||||
},
|
||||
"outbound_nat": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/OutboundNatDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Outbound Nat"
|
||||
},
|
||||
"policy_hierarchy_rules": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"$ref": "#/$defs/HierarchyRuleDoc"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Policy Hierarchy Rules"
|
||||
},
|
||||
"variables": {
|
||||
"anyOf": [
|
||||
{
|
||||
"additionalProperties": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"propertyNames": {
|
||||
"maxLength": 120,
|
||||
"minLength": 1
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Variables"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"opnexus_config"
|
||||
],
|
||||
"title": "OPNexus-Konfiguration (Schema v1)",
|
||||
"type": "object",
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
}
|
||||
@@ -0,0 +1,570 @@
|
||||
"""Planer: Datei gegen Datenbank, ohne etwas zu schreiben und ohne Firewall-Kontakt.
|
||||
|
||||
Der Plan ist ein reiner Inhaltsvergleich der kanonischen Formen (export.canonical und die Datei) je Art und Schlüssel:
|
||||
`create`, `update` (Feldänderungen vorher/nachher), `delete` (nur mit `prune`), `unchanged`, und Gründe, die ein Objekt
|
||||
blockieren. Ein Abschnitt, der in der Datei fehlt, lässt die Art unberührt; ein vorhandener, auch leerer Abschnitt plus
|
||||
`prune` gleicht die Art ab.
|
||||
|
||||
Fehler in der Datei (unbekannter Host, Zyklus, doppelter Schlüssel, Platzhalter in Hierarchie-Regeln …) sind
|
||||
ConfigError (422). Zustände in der Datenbank, die das Übernehmen verhindern (offene Deployments, Prune von
|
||||
ausgerollten oder noch referenzierten Objekten), sind `blocked`: der Plan wird gezeigt, ist aber nicht anwendbar.
|
||||
|
||||
Das Plan-Token ist ein SHA-256 über Datei, `prune` und alles in der Datenbank, wovon das Ergebnis abhängt. Apply
|
||||
rechnet den Plan unter `policy_lock` neu und vergleicht das Token (`CONFIG_PLAN_STALE`).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import canonical
|
||||
from app.config_code.loader import MAX_VARIABLES_PER_HOST, ConfigError, Parsed, _line_for, _path, display
|
||||
from app.database import Base
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
from app.schemas.alias import AliasCreate
|
||||
from app.schemas.one_to_one import OneToOneCreate
|
||||
from app.schemas.outbound_nat import OutboundNatCreate
|
||||
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||
from app.schemas.port_forward import PortForwardCreate
|
||||
from app.schemas.rule import RuleCreate
|
||||
from app.services.variables_service import names_in
|
||||
|
||||
BUSY = ("pending", "revert_failed")
|
||||
DEPLOYED = ("success", "pending", "revert_failed")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
section: str
|
||||
model: type | None
|
||||
ident: str # identity field in the file ("key"/"name")
|
||||
api: type | None = None # API create schema, run on every file object (same validators as the UI)
|
||||
deployable: bool = False
|
||||
|
||||
|
||||
KIND_SPECS = {
|
||||
"device_groups": Kind("device_groups", DeviceGroup, "name"),
|
||||
"policy_hierarchy_groups": Kind("policy_hierarchy_groups", PolicyHierarchyGroup, "name"),
|
||||
"aliases": Kind("aliases", FirewallAlias, "name", AliasCreate, True),
|
||||
"rules": Kind("rules", FirewallRule, "key", RuleCreate, True),
|
||||
"port_forwards": Kind("port_forwards", PortForwardRule, "key", PortForwardCreate, True),
|
||||
"one_to_one": Kind("one_to_one", OneToOneRule, "key", OneToOneCreate, True),
|
||||
"outbound_nat": Kind("outbound_nat", OutboundNatRule, "key", OutboundNatCreate, True),
|
||||
"policy_hierarchy_rules": Kind("policy_hierarchy_rules", PolicyHierarchyRule, "key", PolicyHierarchyRuleCreate, True),
|
||||
"variables": Kind("variables", None, "key"),
|
||||
}
|
||||
TARGET_KINDS = ("aliases", "rules", "port_forwards", "one_to_one", "outbound_nat")
|
||||
HIERARCHY_TARGET_KINDS = ("aliases", "port_forwards", "one_to_one", "outbound_nat")
|
||||
RULE_KINDS = ("rules", "port_forwards", "one_to_one", "outbound_nat", "policy_hierarchy_rules")
|
||||
# Fields whose value may name an alias (OPNsense resolves aliases by name in these).
|
||||
ALIAS_REF_FIELDS = ("source_net", "destination_net", "source_port", "destination_port", "target", "local_port", "target_port", "external")
|
||||
# Values in those fields that are not aliases: keywords, interface names/addresses, OPNsense's built-in aliases.
|
||||
_NOT_ALIAS = re.compile(r"^(?:any|\(self\)|(?:lan|wan|lo0|opt\d+)(?:ip|net)?|wanip|lanip|bogons|bogonsv6|sshlockout|virusprot|__.*)$", re.I)
|
||||
_ALIAS_LIKE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
kind: str
|
||||
key: str
|
||||
action: str # create | update | delete | unchanged
|
||||
before: dict | None = None
|
||||
after: dict | None = None
|
||||
changes: list = field(default_factory=list)
|
||||
reasons: list = field(default_factory=list)
|
||||
deployed_hosts: int = 0
|
||||
implicit: bool = False # not in the file, changed only as a consequence (position, hierarchy members)
|
||||
db_id: uuid.UUID | None = None
|
||||
|
||||
def public(self) -> dict:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"key": self.key,
|
||||
"action": "blocked" if self.reasons else self.action,
|
||||
"planned_action": self.action,
|
||||
"changes": self.changes,
|
||||
"deployed_hosts": self.deployed_hosts,
|
||||
"reasons": self.reasons,
|
||||
"implicit": self.implicit,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Plan:
|
||||
sha256: str
|
||||
prune: bool
|
||||
touched: list[str]
|
||||
items: list[Item]
|
||||
warnings: list[dict]
|
||||
token: str = ""
|
||||
# For apply and its self-check: the complete expected state of every touched kind, keyed by identity.
|
||||
expected: dict = field(default_factory=dict)
|
||||
positions: dict = field(default_factory=dict) # hierarchy rule key -> rank in its (group, rulebase)
|
||||
host_hierarchy: dict = field(default_factory=dict) # host name -> hierarchy group name (only when that section is touched)
|
||||
file_objects: dict = field(default_factory=dict) # kind -> {identity: file dict}
|
||||
|
||||
@property
|
||||
def applicable(self) -> bool:
|
||||
return not any(i.reasons for i in self.items)
|
||||
|
||||
def summary(self) -> dict:
|
||||
out = {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||
for item in self.items:
|
||||
out["blocked" if item.reasons else item.action] += 1
|
||||
return out
|
||||
|
||||
def public(self) -> dict:
|
||||
changed = [i for i in self.items if i.action != "unchanged" or i.reasons]
|
||||
return {
|
||||
"token": self.token,
|
||||
"sha256": self.sha256,
|
||||
"prune": self.prune,
|
||||
"kinds": self.touched,
|
||||
"summary": self.summary(),
|
||||
"items": [i.public() for i in changed],
|
||||
"warnings": self.warnings,
|
||||
"applicable": self.applicable,
|
||||
}
|
||||
|
||||
|
||||
def _err(errors: list, code: str, loc: tuple, lines: dict, **params) -> None:
|
||||
errors.append({"code": code, "path": _path(loc), "line": _line_for(lines, loc), "params": params})
|
||||
|
||||
|
||||
def _file_items(doc: schema.ConfigDoc, kind: str) -> list[dict]:
|
||||
section = getattr(doc, kind)
|
||||
if kind == "variables":
|
||||
return [{"key": f"{host}/{name}", "host": host, "name": name, "value": value}
|
||||
for host, values in section.items() for name, value in values.items()]
|
||||
return [item.model_dump(mode="python") for item in section]
|
||||
|
||||
|
||||
def _normalize(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""Sorted target/member lists like the exporter writes them; duplicates are an error, not silently merged."""
|
||||
def sorted_unique(values: list, where: tuple) -> list:
|
||||
seen = set()
|
||||
for i, value in enumerate(values):
|
||||
if value in seen:
|
||||
_err(errors, "duplicate_entry", where + (i,), lines, value=display(value))
|
||||
seen.add(value)
|
||||
return sorted(seen)
|
||||
|
||||
item = dict(item)
|
||||
if "targets" in item:
|
||||
targets = dict(item["targets"])
|
||||
targets["hosts"] = sorted_unique(targets["hosts"], loc + ("targets", "hosts"))
|
||||
targets["groups"] = sorted_unique(targets["groups"], loc + ("targets", "groups"))
|
||||
item["targets"] = targets
|
||||
if kind in ("device_groups", "policy_hierarchy_groups"):
|
||||
item["hosts"] = sorted_unique(item["hosts"], loc + ("hosts",))
|
||||
return item
|
||||
|
||||
|
||||
def _changes(before: dict, after: dict) -> list[dict]:
|
||||
out = []
|
||||
for name in after:
|
||||
if name == "targets":
|
||||
for sub in after["targets"]:
|
||||
if before["targets"].get(sub) != after["targets"][sub]:
|
||||
out.append({"field": f"targets.{sub}", "before": before["targets"].get(sub), "after": after["targets"][sub]})
|
||||
elif before.get(name) != after[name]:
|
||||
out.append({"field": name, "before": before.get(name), "after": after[name]})
|
||||
return out
|
||||
|
||||
|
||||
def _ranks(rows, key_of, bucket_of, order_of) -> dict:
|
||||
buckets: dict = {}
|
||||
for row in rows:
|
||||
buckets.setdefault(bucket_of(row), []).append(row)
|
||||
out = {}
|
||||
for members in buckets.values():
|
||||
for rank, row in enumerate(sorted(members, key=order_of)):
|
||||
out[key_of(row)] = rank
|
||||
return out
|
||||
|
||||
|
||||
async def _generic_references(db: AsyncSession, target_table: str, skip: set[str], ids: list[uuid.UUID]) -> dict:
|
||||
"""{id: [table, ...]} for rows in tables outside the import that point at `ids` (VPN, rollout waves, future tables)."""
|
||||
found: dict = {}
|
||||
if not ids:
|
||||
return found
|
||||
for table in Base.metadata.sorted_tables:
|
||||
if table.name in skip:
|
||||
continue
|
||||
for fk in table.foreign_keys:
|
||||
if fk.column.table.name != target_table:
|
||||
continue
|
||||
column = fk.parent
|
||||
rows = (await db.execute(
|
||||
select(column, func.count()).where(column.in_(ids)).group_by(column)
|
||||
)).all()
|
||||
for ref_id, _count in rows:
|
||||
found.setdefault(ref_id, []).append(table.name)
|
||||
return found
|
||||
|
||||
|
||||
async def _load(db: AsyncSession) -> dict:
|
||||
loaded: dict = {}
|
||||
for kind in TARGET_KINDS:
|
||||
model = KIND_SPECS[kind].model
|
||||
rows = (await db.execute(select(model).options(selectinload(model.deployments)))).scalars().all()
|
||||
loaded[kind] = {(r.name if kind == "aliases" else r.config_key): r for r in rows}
|
||||
rows = (await db.execute(select(PolicyHierarchyRule).options(selectinload(PolicyHierarchyRule.deployments)))).scalars().all()
|
||||
loaded["policy_hierarchy_rules"] = {r.config_key: r for r in rows}
|
||||
loaded["device_groups"] = {g.name: g for g in (await db.execute(select(DeviceGroup))).scalars().all()}
|
||||
loaded["policy_hierarchy_groups"] = {g.name: g for g in (await db.execute(select(PolicyHierarchyGroup))).scalars().all()}
|
||||
loaded["hosts"] = {h.name: h for h in (await db.execute(select(OpnsenseHost))).scalars().all()}
|
||||
return loaded
|
||||
|
||||
|
||||
async def build_plan(db: AsyncSession, parsed: Parsed, prune: bool) -> Plan:
|
||||
"""The plan for `parsed` against the current database. Raises ConfigError for problems in the file itself."""
|
||||
doc, lines = parsed.doc, parsed.lines
|
||||
touched = [k for k in schema.KINDS if getattr(doc, k) is not None]
|
||||
errors: list[dict] = []
|
||||
warnings: list[dict] = []
|
||||
|
||||
db_canon = await canonical(db)
|
||||
loaded = await _load(db)
|
||||
hosts = loaded["hosts"]
|
||||
hierarchy_by_id = {g.id: g.name for g in loaded["policy_hierarchy_groups"].values()}
|
||||
|
||||
# -- file objects: identity, duplicates, normalization -------------------------------------------------------------
|
||||
file_objects: dict[str, dict[str, dict]] = {}
|
||||
file_locs: dict[tuple, tuple] = {}
|
||||
for kind in touched:
|
||||
spec = KIND_SPECS[kind]
|
||||
objects: dict[str, dict] = {}
|
||||
for index, raw in enumerate(_file_items(doc, kind)):
|
||||
if kind == "variables":
|
||||
ident, loc = raw["key"], ("variables", raw["host"], raw["name"])
|
||||
else:
|
||||
ident, loc = raw[spec.ident], (kind, index)
|
||||
if ident in objects:
|
||||
_err(errors, "duplicate_object_key", loc + ((spec.ident,) if kind != "variables" else ()), lines, key=display(ident))
|
||||
continue
|
||||
objects[ident] = raw if kind == "variables" else _normalize(kind, raw, loc, errors, lines)
|
||||
file_locs[(kind, ident)] = loc
|
||||
file_objects[kind] = objects
|
||||
|
||||
def db_items(kind: str) -> dict[str, dict]:
|
||||
if kind == "variables":
|
||||
return {f"{h}/{n}": {"key": f"{h}/{n}", "host": h, "name": n, "value": v}
|
||||
for h, values in db_canon["variables"].items() for n, v in values.items()}
|
||||
ident = KIND_SPECS[kind].ident
|
||||
return {item[ident]: item for item in db_canon[kind]}
|
||||
|
||||
db_objects = {kind: db_items(kind) for kind in schema.KINDS}
|
||||
|
||||
# -- post-import state of every kind (file objects + what stays in the database) -----------------------------------
|
||||
deleted: dict[str, set[str]] = {kind: set() for kind in schema.KINDS}
|
||||
post: dict[str, dict[str, dict]] = {}
|
||||
for kind in schema.KINDS:
|
||||
if kind in file_objects:
|
||||
remaining = {k: v for k, v in db_objects[kind].items() if k not in file_objects[kind]}
|
||||
if prune:
|
||||
deleted[kind] = set(remaining)
|
||||
remaining = {}
|
||||
post[kind] = {**remaining, **file_objects[kind]}
|
||||
else:
|
||||
post[kind] = dict(db_objects[kind])
|
||||
|
||||
host_names = set(hosts)
|
||||
group_names = set(post["device_groups"])
|
||||
hierarchy_names = set(post["policy_hierarchy_groups"])
|
||||
|
||||
# -- references in the file ------------------------------------------------------------------------------------------
|
||||
for kind in touched:
|
||||
for ident, item in file_objects[kind].items():
|
||||
loc = file_locs[(kind, ident)]
|
||||
if kind == "variables":
|
||||
if item["host"] not in host_names:
|
||||
_err(errors, "unknown_host", ("variables", item["host"]), lines, host=display(item["host"]))
|
||||
continue
|
||||
member_hosts = item.get("hosts", []) + item.get("targets", {}).get("hosts", [])
|
||||
for name in member_hosts:
|
||||
if name not in host_names:
|
||||
_err(errors, "unknown_host", loc, lines, host=display(name))
|
||||
for name in item.get("targets", {}).get("groups", []):
|
||||
if name not in group_names:
|
||||
_err(errors, "unknown_group", loc + ("targets", "groups"), lines, group=display(name))
|
||||
hierarchy_ref = item.get("targets", {}).get("hierarchy_group") if kind in HIERARCHY_TARGET_KINDS else None
|
||||
for ref, where in ((hierarchy_ref, ("targets", "hierarchy_group")), (item.get("parent"), ("parent",)),
|
||||
(item.get("group") if kind == "policy_hierarchy_rules" else None, ("group",))):
|
||||
if ref is not None and ref not in hierarchy_names:
|
||||
_err(errors, "unknown_hierarchy_group", loc + where, lines, group=display(ref))
|
||||
if kind == "policy_hierarchy_rules":
|
||||
for name, value in item.items():
|
||||
if names_in(value):
|
||||
_err(errors, "placeholder_not_supported", loc + (name,), lines)
|
||||
spec = KIND_SPECS[kind]
|
||||
if spec.api is not None:
|
||||
item = file_objects[kind][ident] = post[kind][ident] = _check_api_model(kind, item, loc, errors, lines)
|
||||
for name in member_hosts:
|
||||
host = hosts.get(name)
|
||||
if host is not None and not host.enabled:
|
||||
warnings.append({"code": "target_host_disabled", "path": _path(loc), "params": {"host": display(name)}})
|
||||
|
||||
# -- hierarchy tree and host membership ------------------------------------------------------------------------------
|
||||
host_hierarchy: dict[str, str | None] = {h.name: hierarchy_by_id.get(h.policy_hierarchy_group_id) for h in hosts.values()}
|
||||
if "policy_hierarchy_groups" in file_objects:
|
||||
file_groups = file_objects["policy_hierarchy_groups"]
|
||||
for host_name, group in list(host_hierarchy.items()):
|
||||
if prune or group in file_groups:
|
||||
# The file lists the complete membership of its groups; with prune it is the whole hierarchy.
|
||||
host_hierarchy[host_name] = None
|
||||
listed: dict[str, str] = {}
|
||||
for name, group in file_groups.items():
|
||||
for host_name in group["hosts"]:
|
||||
if host_name in listed:
|
||||
_err(errors, "host_in_two_hierarchy_groups", file_locs[("policy_hierarchy_groups", name)] + ("hosts",),
|
||||
lines, host=display(host_name), first=display(listed[host_name]))
|
||||
listed[host_name] = name
|
||||
if host_name in host_hierarchy:
|
||||
host_hierarchy[host_name] = name
|
||||
parents = {name: g["parent"] for name, g in post["policy_hierarchy_groups"].items()}
|
||||
for start in parents:
|
||||
seen, current = set(), start
|
||||
while current is not None and current not in seen:
|
||||
seen.add(current)
|
||||
current = parents.get(current)
|
||||
if current is not None and current == start:
|
||||
_err(errors, "hierarchy_cycle", file_locs.get(("policy_hierarchy_groups", start), ("policy_hierarchy_groups",)),
|
||||
lines, group=display(start))
|
||||
for name, group in post["policy_hierarchy_groups"].items(): # membership as it will be after the import
|
||||
post["policy_hierarchy_groups"][name] = {**group, "hosts": sorted(h for h, g in host_hierarchy.items() if g == name)}
|
||||
|
||||
if "variables" in file_objects: # the host page allows MAX_VARIABLES_PER_HOST; a merged import must not get past it
|
||||
per_host: dict[str, int] = {}
|
||||
for item in post["variables"].values():
|
||||
per_host[item["host"]] = per_host.get(item["host"], 0) + 1
|
||||
for host_name, count in sorted(per_host.items()):
|
||||
if count > MAX_VARIABLES_PER_HOST:
|
||||
_err(errors, "too_many_variables", ("variables", host_name), lines, host=display(host_name),
|
||||
max=MAX_VARIABLES_PER_HOST, count=count)
|
||||
|
||||
if errors:
|
||||
raise ConfigError("CONFIG_INVALID", errors[:100])
|
||||
|
||||
# -- hierarchy rule positions ----------------------------------------------------------------------------------------
|
||||
db_rule_rows = loaded["policy_hierarchy_rules"]
|
||||
db_positions = _ranks(db_rule_rows.values(), lambda r: r.config_key,
|
||||
lambda r: (hierarchy_by_id[r.hierarchy_group_id], r.rulebase), lambda r: (r.sequence, str(r.id))) # = deploy order
|
||||
positions = dict(db_positions)
|
||||
if "policy_hierarchy_rules" in file_objects:
|
||||
buckets: dict[tuple, list[str]] = {}
|
||||
for key, item in file_objects["policy_hierarchy_rules"].items(): # file order = order in the file
|
||||
buckets.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for key, item in post["policy_hierarchy_rules"].items():
|
||||
if key not in file_objects["policy_hierarchy_rules"]: # stays in the database: after the file's rules
|
||||
buckets.setdefault((item["group"], item["rulebase"]), [])
|
||||
positions = {}
|
||||
extras: dict[tuple, list[str]] = {}
|
||||
for key in sorted((k for k in post["policy_hierarchy_rules"] if k not in file_objects["policy_hierarchy_rules"]),
|
||||
key=lambda k: db_positions.get(k, 0)):
|
||||
item = post["policy_hierarchy_rules"][key]
|
||||
extras.setdefault((item["group"], item["rulebase"]), []).append(key)
|
||||
for bucket in set(buckets) | set(extras):
|
||||
for rank, key in enumerate(buckets.get(bucket, []) + extras.get(bucket, [])):
|
||||
positions[key] = rank
|
||||
|
||||
# -- items -----------------------------------------------------------------------------------------------------------
|
||||
items: list[Item] = []
|
||||
for kind in touched:
|
||||
rows = loaded.get(kind, {})
|
||||
for ident in sorted(set(file_objects[kind]) | deleted[kind] | _implicit(kind, post, db_objects, positions, db_positions)):
|
||||
row = rows.get(ident) if kind != "variables" else None
|
||||
before = db_objects[kind].get(ident)
|
||||
after = post[kind].get(ident) if ident not in deleted[kind] else None
|
||||
if before is None:
|
||||
item = Item(kind, ident, "create", None, after)
|
||||
elif after is None:
|
||||
item = Item(kind, ident, "delete", before, None)
|
||||
else:
|
||||
changes = _changes(before, after)
|
||||
if kind == "policy_hierarchy_rules" and db_positions.get(ident) != positions.get(ident):
|
||||
changes.append({"field": "position", "before": db_positions.get(ident), "after": positions.get(ident)})
|
||||
item = Item(kind, ident, "update" if changes else "unchanged", before, after, changes)
|
||||
item.implicit = ident not in file_objects[kind] and ident not in deleted[kind]
|
||||
if row is not None:
|
||||
item.db_id = row.id
|
||||
deployments = getattr(row, "deployments", None) or []
|
||||
item.deployed_hosts = sum(1 for d in deployments if d.status in DEPLOYED)
|
||||
if item.action in ("update", "delete") and any(d.status in BUSY for d in deployments):
|
||||
item.reasons.append("busy_deployment")
|
||||
if item.action == "delete" and deployments:
|
||||
item.reasons.append("prune_deployed")
|
||||
items.append(item)
|
||||
|
||||
await _prune_references(db, items, post, loaded, deleted, host_hierarchy)
|
||||
|
||||
redeploy = sum(1 for i in items if i.action == "update" and i.deployed_hosts)
|
||||
if redeploy:
|
||||
warnings.append({"code": "redeploy_needed", "path": "", "params": {"count": redeploy}})
|
||||
creates_with_targets = sum(1 for i in items if i.action == "create" and i.kind in TARGET_KINDS
|
||||
and (i.after["targets"]["hosts"] or i.after["targets"]["groups"] or i.after["targets"].get("hierarchy_group")))
|
||||
if creates_with_targets and not any(loaded[k] for k in TARGET_KINDS + ("policy_hierarchy_rules",)):
|
||||
warnings.append({"code": "targets_without_link", "path": "", "params": {"count": creates_with_targets}})
|
||||
warnings += _alias_reference_warnings(post, file_objects, file_locs)
|
||||
warnings += _variable_warnings(post, deleted)
|
||||
|
||||
plan = Plan(parsed.sha256, prune, touched, items, warnings, positions=positions,
|
||||
host_hierarchy=host_hierarchy if "policy_hierarchy_groups" in file_objects else {},
|
||||
file_objects=file_objects)
|
||||
plan.expected = {kind: post[kind] for kind in touched}
|
||||
plan.token = _token(plan, db_canon, db_positions, loaded)
|
||||
return plan
|
||||
|
||||
|
||||
def _implicit(kind: str, post: dict, db_objects: dict, positions: dict, db_positions: dict) -> set[str]:
|
||||
"""Objects the file does not name but whose stored state still changes (rank in a reordered list, group members)."""
|
||||
out = set()
|
||||
if kind == "policy_hierarchy_rules":
|
||||
out = {k for k in post[kind] if k in db_objects[kind] and positions.get(k) != db_positions.get(k)}
|
||||
if kind == "policy_hierarchy_groups":
|
||||
out = {k for k, v in post[kind].items() if k in db_objects[kind] and db_objects[kind][k] != v}
|
||||
return out
|
||||
|
||||
|
||||
def _check_api_model(kind: str, item: dict, loc: tuple, errors: list, lines: dict) -> dict:
|
||||
"""The real API schema on the same values (minus references), so a file can never store what the UI refuses.
|
||||
Returns the item with the values as the API would store them (stripped fields, cleaned alias entries)."""
|
||||
values = {k: v for k, v in item.items() if k not in ("key", "targets", "group", "entries")}
|
||||
if kind == "aliases":
|
||||
values["content"] = "\n".join(item["entries"])
|
||||
if kind == "policy_hierarchy_rules":
|
||||
values["hierarchy_group_id"] = uuid.UUID(int=0)
|
||||
try:
|
||||
cleaned = KIND_SPECS[kind].api.model_validate(values)
|
||||
except ValidationError as exc:
|
||||
for error in exc.errors(include_url=False):
|
||||
_err(errors, "bad_value", loc + tuple(error["loc"]), lines)
|
||||
return item
|
||||
out = dict(item)
|
||||
for name in item:
|
||||
if name in values and name != "content" and hasattr(cleaned, name):
|
||||
out[name] = getattr(cleaned, name)
|
||||
if kind == "aliases":
|
||||
out["entries"] = [e for e in cleaned.content.split("\n") if e]
|
||||
return out
|
||||
|
||||
|
||||
async def _prune_references(db, items: list[Item], post: dict, loaded: dict, deleted: dict, host_hierarchy: dict) -> None:
|
||||
by_key = {(i.kind, i.key): i for i in items}
|
||||
|
||||
# device groups: post-state objects of the import, plus every other table that points at the group
|
||||
if deleted["device_groups"]:
|
||||
users: dict[str, list[str]] = {}
|
||||
for kind in TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
for group in item["targets"]["groups"]:
|
||||
users.setdefault(group, []).append(f"{kind}:{ident}")
|
||||
ids = {loaded["device_groups"][n].id: n for n in deleted["device_groups"]}
|
||||
skip = {"device_group_members", "rule_groups", "alias_groups", "port_forward_groups", "one_to_one_groups", "outbound_nat_groups"}
|
||||
for ref_id, tables in (await _generic_references(db, "device_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["device_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("device_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["policy_hierarchy_groups"]:
|
||||
users = {}
|
||||
for kind in HIERARCHY_TARGET_KINDS:
|
||||
for ident, item in post[kind].items():
|
||||
if item["targets"]["hierarchy_group"]:
|
||||
users.setdefault(item["targets"]["hierarchy_group"], []).append(f"{kind}:{ident}")
|
||||
for ident, item in post["policy_hierarchy_rules"].items():
|
||||
users.setdefault(item["group"], []).append(f"policy_hierarchy_rules:{ident}")
|
||||
for name, item in post["policy_hierarchy_groups"].items():
|
||||
if item["parent"]:
|
||||
users.setdefault(item["parent"], []).append(f"policy_hierarchy_groups:{name}")
|
||||
for host_name, group in host_hierarchy.items():
|
||||
if group:
|
||||
users.setdefault(group, []).append(f"host:{host_name}")
|
||||
ids = {loaded["policy_hierarchy_groups"][n].id: n for n in deleted["policy_hierarchy_groups"]}
|
||||
skip = {"policy_hierarchy_groups", "firewall_aliases", "port_forward_rules", "one_to_one_rules", "outbound_nat_rules",
|
||||
"policy_hierarchy_rules", "opnsense_hosts"}
|
||||
for ref_id, tables in (await _generic_references(db, "policy_hierarchy_groups", skip, list(ids))).items():
|
||||
users.setdefault(ids[ref_id], []).extend(tables)
|
||||
for name in deleted["policy_hierarchy_groups"]:
|
||||
if users.get(name):
|
||||
by_key[("policy_hierarchy_groups", name)].reasons.append("prune_referenced")
|
||||
|
||||
if deleted["aliases"]:
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS:
|
||||
for item in post[kind].values():
|
||||
used.update(str(item[f]) for f in ALIAS_REF_FIELDS if item.get(f) is not None)
|
||||
for item in post["aliases"].values():
|
||||
used.update(item["entries"])
|
||||
for name in deleted["aliases"]:
|
||||
if name in used and "prune_referenced" not in by_key[("aliases", name)].reasons:
|
||||
by_key[("aliases", name)].reasons.append("prune_referenced")
|
||||
|
||||
|
||||
def _alias_reference_warnings(post: dict, file_objects: dict, file_locs: dict) -> list[dict]:
|
||||
aliases = set(post["aliases"])
|
||||
out = []
|
||||
for kind in RULE_KINDS:
|
||||
for ident, item in file_objects.get(kind, {}).items():
|
||||
for name in ALIAS_REF_FIELDS:
|
||||
value = item.get(name)
|
||||
if isinstance(value, str) and _ALIAS_LIKE.match(value) and not _NOT_ALIAS.match(value) and value not in aliases:
|
||||
out.append({"code": "unknown_alias_reference", "path": _path(file_locs[(kind, ident)] + (name,)),
|
||||
"params": {"value": display(value)}})
|
||||
return out
|
||||
|
||||
|
||||
def _variable_warnings(post: dict, deleted: dict) -> list[dict]:
|
||||
if not deleted["variables"]:
|
||||
return []
|
||||
used: set[str] = set()
|
||||
for kind in RULE_KINDS + ("aliases",):
|
||||
for item in post[kind].values():
|
||||
for value in item.values():
|
||||
if isinstance(value, str):
|
||||
used |= names_in(value)
|
||||
elif isinstance(value, list):
|
||||
for entry in value:
|
||||
used |= names_in(entry)
|
||||
out = []
|
||||
for key in sorted(deleted["variables"]):
|
||||
host, name = key.split("/", 1)
|
||||
if name in used:
|
||||
out.append({"code": "variable_removed_in_use", "path": f"variables.{host}", "params": {"host": display(host), "name": name}})
|
||||
return out
|
||||
|
||||
|
||||
def _token(plan: Plan, db_canon: dict, db_positions: dict, loaded: dict) -> str:
|
||||
deployments = {
|
||||
kind: {ident: sorted((str(d.host_id), d.status) for d in (row.deployments or []))
|
||||
for ident, row in loaded[kind].items()}
|
||||
for kind in TARGET_KINDS + ("policy_hierarchy_rules",)
|
||||
}
|
||||
material = {
|
||||
"file": plan.file_objects,
|
||||
"prune": plan.prune,
|
||||
"db": db_canon,
|
||||
"positions": db_positions,
|
||||
"hosts": sorted((h.name, str(h.id), h.enabled, str(h.policy_hierarchy_group_id)) for h in loaded["hosts"].values()),
|
||||
"ids": {k: sorted((n, str(o.id)) for n, o in loaded[k].items()) for k in ("device_groups", "policy_hierarchy_groups")},
|
||||
"deployments": deployments,
|
||||
"items": [i.public() for i in plan.items],
|
||||
"warnings": plan.warnings,
|
||||
}
|
||||
return hashlib.sha256(json.dumps(material, sort_keys=True, default=str, ensure_ascii=False).encode()).hexdigest()
|
||||
@@ -0,0 +1,242 @@
|
||||
"""Schema v1 der Konfigurationsdatei. Strikt: unbekannte Felder, falsche Typen und Steuerzeichen sind Fehler.
|
||||
|
||||
Die Grenzen entsprechen den API-Schemas der jeweiligen Objekte (schemas/rule.py usw.); `tests/test_config_code_schema.py`
|
||||
hält beide gleich. Steuerzeichen sind überall verboten, außer Tab und Zeilenumbruch in Beschreibungen (die UI erlaubt
|
||||
mehrzeilige Beschreibungen). Namen und Schlüssel (die Identität eines Objekts) dürfen keine umgebenden Leerzeichen haben;
|
||||
alle anderen Werte säubert der Planer mit demselben API-Schema wie die UI (plan._check_api_model).
|
||||
"""
|
||||
import re
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import AfterValidator, BaseModel, BeforeValidator, ConfigDict, Field, StringConstraints
|
||||
|
||||
from app.models.config_key import CONFIG_KEY_RE
|
||||
from app.services.variables_service import NAME_RE as VARIABLE_NAME_RE, VALUE_RE as VARIABLE_VALUE_RE
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
MAX_SCALAR = 8000
|
||||
|
||||
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
|
||||
_CONTROL_TEXT = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") # tab and newline allowed
|
||||
|
||||
|
||||
def _clean(value: str) -> str:
|
||||
"""Names and keys: they identify an object, so no control characters and no surrounding whitespace."""
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
if value != value.strip():
|
||||
raise ValueError("surrounding_whitespace")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_value(value: str) -> str:
|
||||
if _CONTROL.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _clean_text(value: str) -> str:
|
||||
if _CONTROL_TEXT.search(value):
|
||||
raise ValueError("control_character")
|
||||
return value
|
||||
|
||||
|
||||
def _port(value):
|
||||
# YAML liest `53` als Zahl; Ports sind in OPNsense Text (auch Bereiche wie "8000-8100"). bool bleibt ein Typfehler.
|
||||
return str(value) if isinstance(value, int) and not isinstance(value, bool) else value
|
||||
|
||||
|
||||
Text = Annotated[str, AfterValidator(_clean)]
|
||||
Key = Annotated[str, StringConstraints(pattern=CONFIG_KEY_RE.pattern), AfterValidator(_clean)]
|
||||
Description = Annotated[str, Field(max_length=255), AfterValidator(_clean_text)]
|
||||
RuleDescription = Annotated[str, Field(min_length=1, max_length=255), AfterValidator(_clean_text)]
|
||||
GroupDescription = Annotated[str, Field(max_length=2000), AfterValidator(_clean_text)]
|
||||
Interface = Annotated[str, Field(min_length=1, max_length=64), AfterValidator(_clean_value)]
|
||||
Protocol = Annotated[str, Field(min_length=1, max_length=16), AfterValidator(_clean_value)]
|
||||
Net = Annotated[str, Field(min_length=1, max_length=128), AfterValidator(_clean_value)]
|
||||
PortText = Annotated[str, Field(max_length=32), AfterValidator(_clean_value)]
|
||||
Port = Annotated[PortText | None, BeforeValidator(_port)]
|
||||
GroupName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
HostName = Annotated[str, Field(min_length=1, max_length=120), AfterValidator(_clean)]
|
||||
AliasName = Annotated[str, Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_]+$")]
|
||||
AliasEntry = Annotated[str, Field(min_length=1, max_length=MAX_SCALAR), AfterValidator(_clean_value)]
|
||||
|
||||
IpProtocol = Literal["inet", "inet6"]
|
||||
|
||||
|
||||
class _Doc(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
class Targets(_Doc):
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
groups: list[GroupName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class TargetsWithHierarchy(Targets):
|
||||
hierarchy_group: GroupName | None = None
|
||||
|
||||
|
||||
class DeviceGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class HierarchyGroupDoc(_Doc):
|
||||
name: GroupName
|
||||
parent: GroupName | None = None
|
||||
description: GroupDescription = ""
|
||||
hosts: list[HostName] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AliasDoc(_Doc):
|
||||
name: AliasName
|
||||
type: Literal["host", "network", "port", "networkgroup", "mac", "asn"] = "host"
|
||||
description: Description = ""
|
||||
enabled: bool = True
|
||||
entries: list[AliasEntry] = Field(default_factory=list)
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class RuleDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
log: bool = False
|
||||
targets: Targets = Field(default_factory=Targets)
|
||||
|
||||
|
||||
class PortForwardDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "tcp"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
target: Net
|
||||
local_port: Port = None
|
||||
natreflection: Literal["", "purenat", "disable"] = ""
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OneToOneDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
type: Literal["binat", "nat"] = "binat"
|
||||
external: Net
|
||||
source_net: Net = "any"
|
||||
destination_net: Net = "any"
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
natreflection: Literal["", "enable", "disable"] = ""
|
||||
log: bool = False
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class OutboundNatDoc(_Doc):
|
||||
key: Key
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
source_not: bool = False
|
||||
destination_not: bool = False
|
||||
target: Annotated[str, Field(max_length=128), AfterValidator(_clean_value)] = ""
|
||||
target_port: Port = None
|
||||
enabled: bool = True
|
||||
targets: TargetsWithHierarchy = Field(default_factory=TargetsWithHierarchy)
|
||||
|
||||
|
||||
class HierarchyRuleDoc(_Doc):
|
||||
key: Key
|
||||
group: GroupName
|
||||
rulebase: Literal["pre", "post"] = "pre"
|
||||
description: RuleDescription
|
||||
interface: Interface
|
||||
direction: Literal["in", "out"] = "in"
|
||||
ipprotocol: IpProtocol = "inet"
|
||||
protocol: Protocol = "any"
|
||||
action: Literal["pass", "block", "reject"] = "pass"
|
||||
source_net: Net = "any"
|
||||
source_port: Port = None
|
||||
destination_net: Net = "any"
|
||||
destination_port: Port = None
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
def _variable_name(value: str) -> str:
|
||||
if not VARIABLE_NAME_RE.match(value):
|
||||
raise ValueError("bad_variable_name")
|
||||
return value
|
||||
|
||||
|
||||
def _variable_value(value: str) -> str:
|
||||
if not VARIABLE_VALUE_RE.match(value):
|
||||
raise ValueError("bad_variable_value")
|
||||
return value
|
||||
|
||||
|
||||
VariableName = Annotated[str, AfterValidator(_variable_name)]
|
||||
VariableValue = Annotated[str, AfterValidator(_variable_value)]
|
||||
|
||||
|
||||
class ConfigDoc(_Doc):
|
||||
"""Eine Datei. Ein fehlender Abschnitt lässt die Art unberührt (None); ein leerer Abschnitt ist ein leerer Sollzustand."""
|
||||
|
||||
opnexus_config: Literal[1]
|
||||
device_groups: list[DeviceGroupDoc] | None = None
|
||||
policy_hierarchy_groups: list[HierarchyGroupDoc] | None = None
|
||||
aliases: list[AliasDoc] | None = None
|
||||
rules: list[RuleDoc] | None = None
|
||||
port_forwards: list[PortForwardDoc] | None = None
|
||||
one_to_one: list[OneToOneDoc] | None = None
|
||||
outbound_nat: list[OutboundNatDoc] | None = None
|
||||
policy_hierarchy_rules: list[HierarchyRuleDoc] | None = None
|
||||
variables: dict[HostName, dict[VariableName, VariableValue]] | None = None
|
||||
|
||||
|
||||
# Abschnittsfolge der Datei = Reihenfolge, in der Export und Plan die Arten behandeln.
|
||||
KINDS = (
|
||||
"device_groups",
|
||||
"policy_hierarchy_groups",
|
||||
"aliases",
|
||||
"rules",
|
||||
"port_forwards",
|
||||
"one_to_one",
|
||||
"outbound_nat",
|
||||
"policy_hierarchy_rules",
|
||||
"variables",
|
||||
)
|
||||
|
||||
|
||||
SCHEMA_FILE = "opnexus-config-v1.schema.json"
|
||||
|
||||
|
||||
def json_schema_text() -> str:
|
||||
"""JSON Schema of the file for editors and CI; checked in next to this module and compared by a test."""
|
||||
import json
|
||||
|
||||
schema = ConfigDoc.model_json_schema()
|
||||
schema["$schema"] = "https://json-schema.org/draft/2020-12/schema"
|
||||
schema["title"] = "OPNexus-Konfiguration (Schema v1)"
|
||||
return json.dumps(schema, indent=2, ensure_ascii=False, sort_keys=False) + "\n"
|
||||
@@ -171,6 +171,7 @@ class ErrorCode(str, Enum):
|
||||
OPEN_DEPLOYMENT_DRAFT_UNCHANGED = "OPEN_DEPLOYMENT_DRAFT_UNCHANGED"
|
||||
OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY = "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
||||
OPEN_RESTORE_BLOCKS_DEPLOY = "OPEN_RESTORE_BLOCKS_DEPLOY"
|
||||
OPEN_CLEANUP_BLOCKS_DEPLOY = "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
UPSTREAM_REJECTED = "UPSTREAM_REJECTED"
|
||||
INVALID_CHARACTERS = "INVALID_CHARACTERS"
|
||||
SERVICE_NOT_FOUND = "SERVICE_NOT_FOUND"
|
||||
@@ -222,6 +223,37 @@ class ErrorCode(str, Enum):
|
||||
ANALYSE_INTERFACE_UNBEKANNT = "ANALYSE_INTERFACE_UNBEKANNT"
|
||||
ANALYSE_PARAMETER_UNGUELTIG = "ANALYSE_PARAMETER_UNGUELTIG"
|
||||
ANALYSE_FIREWALL_NICHT_LESBAR = "ANALYSE_FIREWALL_NICHT_LESBAR"
|
||||
# Rule cleanup (1.31, services/rule_cleanup_service.py)
|
||||
BEREINIGUNG_NICHT_GEFUNDEN = "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||
BEREINIGUNG_AUSWAHL_UNGUELTIG = "BEREINIGUNG_AUSWAHL_UNGUELTIG"
|
||||
BEREINIGUNG_VORSCHAU_VERALTET = "BEREINIGUNG_VORSCHAU_VERALTET"
|
||||
BEREINIGUNG_REGEL_NICHT_AENDERBAR = "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
|
||||
BEREINIGUNG_OFFENE_OPERATION = "BEREINIGUNG_OFFENE_OPERATION"
|
||||
BEREINIGUNG_ZU_VIELE_REGELN = "BEREINIGUNG_ZU_VIELE_REGELN"
|
||||
BEREINIGUNG_BESTAETIGUNG_FEHLT = "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN = "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||
BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG = "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG"
|
||||
BEREINIGUNG_FRIST_ABGELAUFEN = "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
BEREINIGUNG_NICHT_BESTAETIGBAR = "BEREINIGUNG_NICHT_BESTAETIGBAR"
|
||||
BEREINIGUNG_LIVESTAND_ABWEICHEND = "BEREINIGUNG_LIVESTAND_ABWEICHEND"
|
||||
BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR = "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
|
||||
BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT = "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT"
|
||||
BEREINIGUNG_SNAPSHOT_FEHLT = "BEREINIGUNG_SNAPSHOT_FEHLT"
|
||||
BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR = "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
|
||||
BEREINIGUNG_AKTION_NICHT_MOEGLICH = "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
BEREINIGUNG_NICHT_AUSGEFUEHRT = "BEREINIGUNG_NICHT_AUSGEFUEHRT"
|
||||
BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN = "BEREINIGUNG_ANWENDEN_FEHLGESCHLAGEN"
|
||||
BEREINIGUNG_AUSGANG_UNKLAR = "BEREINIGUNG_AUSGANG_UNKLAR"
|
||||
BEREINIGUNG_ABGEBROCHEN_VERWAIST = "BEREINIGUNG_ABGEBROCHEN_VERWAIST"
|
||||
BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET = "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET"
|
||||
BEREINIGUNG_EINSTELLUNG_UNGUELTIG = "BEREINIGUNG_EINSTELLUNG_UNGUELTIG"
|
||||
CONFIG_INVALID = "CONFIG_INVALID"
|
||||
CONFIG_TOO_LARGE = "CONFIG_TOO_LARGE"
|
||||
CONFIG_UNSUPPORTED_VERSION = "CONFIG_UNSUPPORTED_VERSION"
|
||||
CONFIG_PLAN_STALE = "CONFIG_PLAN_STALE"
|
||||
CONFIG_BLOCKED = "CONFIG_BLOCKED"
|
||||
CONFIG_APPLY_SELFCHECK = "CONFIG_APPLY_SELFCHECK"
|
||||
CONFIG_BUSY = "CONFIG_BUSY"
|
||||
|
||||
|
||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||
|
||||
@@ -53,6 +53,8 @@ from app.routers import scheduled_maintenance as scheduled_maintenance_router
|
||||
from app.routers import maintenance_windows as maintenance_windows_router
|
||||
from app.routers import rollouts as rollouts_router
|
||||
from app.routers import rule_analysis as rule_analysis_router
|
||||
from app.routers import rule_cleanup as rule_cleanup_router
|
||||
from app.routers import config_code as config_code_router
|
||||
from app.routers import gui_access as gui_access_router
|
||||
from app.services import scheduled_maintenance_service
|
||||
from app.services import rollout_engine
|
||||
@@ -62,6 +64,7 @@ from app.services import overlay_service, variables_service
|
||||
from app.services import backup_schedule_service
|
||||
from app.services import carp_event_service, carp_maintenance_window_service
|
||||
from app.services import rule_history_service
|
||||
from app.services import rule_cleanup_service
|
||||
|
||||
# Without this, Python's root logger defaults to WARNING with no handler,
|
||||
# so every logger.info() call in the 5 background loops below (CARP role
|
||||
@@ -142,6 +145,13 @@ async def _deploy_revert_sweep_loop():
|
||||
logger.info("Auto-reverted %d unconfirmed policy-hierarchy deployment(s)", len(reverted))
|
||||
except Exception:
|
||||
logger.exception("Policy-hierarchy deploy-revert sweep failed")
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
reverted = await rule_cleanup_service.sweep_expired_cleanups(db)
|
||||
if reverted:
|
||||
logger.info("Regel-Bereinigung: %d Operation(en) automatisch zurückgenommen/abgeschlossen", len(reverted))
|
||||
except Exception:
|
||||
logger.exception("Rule-cleanup revert sweep failed")
|
||||
await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
|
||||
|
||||
|
||||
@@ -561,5 +571,7 @@ app.include_router(scheduled_maintenance_router.router, prefix="/api")
|
||||
app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||
app.include_router(rollouts_router.router, prefix="/api")
|
||||
app.include_router(rule_analysis_router.router, prefix="/api")
|
||||
app.include_router(rule_cleanup_router.router, prefix="/api")
|
||||
app.include_router(config_code_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.router, prefix="/api")
|
||||
app.include_router(gui_access_router.internal_router)
|
||||
@@ -105,3 +105,11 @@ __all__ += ["GuiAccessSettings", "GuiSession", "GuiTicket"]
|
||||
|
||||
from app.models.rule_history import LiveRuleState, RuleChangeEvent
|
||||
__all__ += ["LiveRuleState", "RuleChangeEvent"]
|
||||
|
||||
from app.models.rule_cleanup import RuleCleanupOperation, RuleCleanupItem
|
||||
__all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
|
||||
|
||||
# Every mapped class must be registered here: a script that imports only app.models (python -m app.reset_password,
|
||||
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
|
||||
from app.models.host_variable import HostVariable
|
||||
__all__ += ["HostVariable"]
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Stabile Schlüssel für „Konfiguration als Code“ (1.32).
|
||||
|
||||
Regeln und NAT-Objekte haben außer der UUID keinen natürlichen Schlüssel, die
|
||||
Beschreibung ist nicht eindeutig. `config_key` ist die Identität in der
|
||||
YAML-Datei: lesbar, vom Betreiber vergebbar, je Tabelle eindeutig. Bestand und
|
||||
neue Objekte ohne eigenen Schlüssel bekommen `<präfix>-<12 Hex der UUID>`.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import String, event
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
CONFIG_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
|
||||
|
||||
|
||||
def config_key_column() -> Mapped[str]:
|
||||
return mapped_column(String(64), unique=True)
|
||||
|
||||
|
||||
def default_config_key(prefix: str, object_id: uuid.UUID) -> str:
|
||||
return f"{prefix}-{object_id.hex[:12]}"
|
||||
|
||||
|
||||
def register_config_key(model: type, prefix: str) -> None:
|
||||
"""Fills an empty `config_key` just before INSERT (and the id, which is
|
||||
otherwise only assigned while the INSERT is built)."""
|
||||
|
||||
@event.listens_for(model, "before_insert")
|
||||
def _fill(_mapper, _connection, target) -> None:
|
||||
if target.id is None:
|
||||
target.id = uuid.uuid4()
|
||||
if not target.config_key:
|
||||
target.config_key = default_config_key(prefix, target.id)
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Separation of duties (singleton row): may operators roll changes out themselves?"""
|
||||
from sqlalchemy import Boolean, Integer
|
||||
from sqlalchemy import JSON, Boolean, Integer
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
@@ -11,3 +11,14 @@ class GovernanceSettings(Base):
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
# False: an operator prepares changes (edit, preview) but only an admin may deploy them.
|
||||
operators_may_deploy: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
|
||||
# Rule cleanup (1.31). At most this many rules per cleanup operation (default 10, ceiling 25).
|
||||
cleanup_max_rules: Mapped[int] = mapped_column(Integer, nullable=False, default=10)
|
||||
# A rule must have been switched off (and confirmed) this many days before it may be deleted (minimum 1).
|
||||
cleanup_delete_min_days: Mapped[int] = mapped_column(Integer, nullable=False, default=7)
|
||||
# IPs/CIDRs the admin manages the firewalls from; no rule that could match them may be cleaned up. Empty list =
|
||||
# the cleanup is locked (fail closed). Only an admin may change it.
|
||||
cleanup_management_sources: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# Extra TCP ports the firewalls are managed on (besides the API port of base_url, 443, 80 and 22): a rule that
|
||||
# could match a management source on one of them is protected too. Only an admin may change it.
|
||||
cleanup_management_ports: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.one_to_one_target import one_to_one_targets
|
||||
from app.models.policy import one_to_one_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OneToOneRule(Base):
|
||||
__tablename__ = "one_to_one_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -42,3 +44,6 @@ class OneToOneRule(Base):
|
||||
deployments: Mapped[list["OneToOneDeployment"]] = relationship(
|
||||
"OneToOneDeployment", back_populates="one_to_one", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OneToOneRule, "nat1")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.outbound_nat_target import outbound_nat_targets
|
||||
from app.models.policy import outbound_nat_groups
|
||||
|
||||
@@ -14,6 +15,7 @@ class OutboundNatRule(Base):
|
||||
__tablename__ = "outbound_nat_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
source_net: Mapped[str] = mapped_column(String(128))
|
||||
@@ -44,3 +46,6 @@ class OutboundNatRule(Base):
|
||||
deployments: Mapped[list["OutboundNatDeployment"]] = relationship(
|
||||
"OutboundNatDeployment", back_populates="outbound_nat", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(OutboundNatRule, "out")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
|
||||
|
||||
class PolicyHierarchyRule(Base):
|
||||
@@ -33,6 +34,7 @@ class PolicyHierarchyRule(Base):
|
||||
__tablename__ = "policy_hierarchy_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
hierarchy_group_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("policy_hierarchy_groups.id", ondelete="CASCADE")
|
||||
)
|
||||
@@ -62,3 +64,6 @@ class PolicyHierarchyRule(Base):
|
||||
deployments: Mapped[list["PolicyHierarchyRuleDeployment"]] = relationship(
|
||||
"PolicyHierarchyRuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PolicyHierarchyRule, "ph")
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.port_forward_target import port_forward_targets
|
||||
from app.models.policy import port_forward_groups
|
||||
|
||||
@@ -20,6 +21,7 @@ class PortForwardRule(Base):
|
||||
__tablename__ = "port_forward_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
ipprotocol: Mapped[str] = mapped_column(String(8), default="inet")
|
||||
@@ -49,3 +51,6 @@ class PortForwardRule(Base):
|
||||
deployments: Mapped[list["PortForwardDeployment"]] = relationship(
|
||||
"PortForwardDeployment", back_populates="port_forward", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(PortForwardRule, "pf")
|
||||
@@ -42,6 +42,8 @@ class RolloutPlan(Base):
|
||||
# is activated AND unchanged since (activated_fingerprint).
|
||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
# Reachability probes run FROM the firewall in the health gate (1.32): [{type: ping|tcp, target, port}].
|
||||
gate_probes: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
# Stable id of that account (1.29, no foreign key on purpose: a deleted account stays gone, a new one with the same name is someone else).
|
||||
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
from app.models.config_key import config_key_column, register_config_key
|
||||
from app.models.rule_target import rule_targets
|
||||
from app.models.policy import rule_groups
|
||||
|
||||
@@ -18,6 +19,7 @@ class FirewallRule(Base):
|
||||
__tablename__ = "firewall_rules"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
config_key: Mapped[str] = config_key_column()
|
||||
description: Mapped[str] = mapped_column(String(255))
|
||||
interface: Mapped[str] = mapped_column(String(64))
|
||||
direction: Mapped[str] = mapped_column(String(8))
|
||||
@@ -44,3 +46,6 @@ class FirewallRule(Base):
|
||||
deployments: Mapped[list["RuleDeployment"]] = relationship(
|
||||
"RuleDeployment", back_populates="rule", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
register_config_key(FirewallRule, "r")
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): switching rules off or deleting them through preview -> deploy -> confirm.
|
||||
|
||||
``RuleCleanupOperation`` is one cleanup run on a host (or a cluster pair, anchored at one host); its id doubles as
|
||||
the preview token. ``RuleCleanupItem`` is one rule on one node of that run. The columns only carry what the design
|
||||
(DESIGN-1.31) needs to resume, revert and restore after a crash; the logic lives in services/rule_cleanup_service.py.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import JSON, DateTime, ForeignKey, Index, Integer, String, Text, text
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
# kind: disable | enable (re-enable after a restore) | delete | restore
|
||||
OPERATION_KINDS = ("disable", "enable", "delete", "restore")
|
||||
# preview -> running -> awaiting_confirm -> confirmed; the rest are the side exits.
|
||||
OPERATION_STATUSES = (
|
||||
"preview", "running", "awaiting_confirm", "confirmed", "partial", "apply_failed", "reverting", "reverted",
|
||||
"revert_failed", "done", "failed", "cancelled", "expired",
|
||||
)
|
||||
ITEM_STATES = (
|
||||
"queued", "disabling", "disabled", "confirmed", "reverting", "reverted", "deleting", "deleted", "restoring",
|
||||
"restored", "failed", "unknown", "revert_failed",
|
||||
# never attempted (the run stopped at an earlier error); not an open state, so the rule is free again
|
||||
"skipped",
|
||||
)
|
||||
# A rule in one of these states is being worked on or its outcome is unresolved: no second item may claim it.
|
||||
OPEN_ITEM_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class RuleCleanupOperation(Base):
|
||||
__tablename__ = "rule_cleanup_operations"
|
||||
__table_args__ = (
|
||||
Index("ix_rule_cleanup_operations_host", "host_id", "created_at"),
|
||||
Index("ix_rule_cleanup_operations_status", "status"),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
cluster_name: Mapped[Optional[str]] = mapped_column(String(120), nullable=True)
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="preview")
|
||||
actor: Mapped[str] = mapped_column(String(150), nullable=False)
|
||||
# The stable account id next to the name (see migration 0026); no foreign key, a deleted account stays gone.
|
||||
actor_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
started_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Written at every checkpoint, so another process can tell an abandoned ``running`` operation from a live one.
|
||||
heartbeat_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Auto-revert deadline (disable/enable only); NULL once confirmed or for kinds without a window.
|
||||
revert_deadline: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
confirmed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||
# sha256 over everything the preview showed; the deploy recomputes it and refuses on any difference.
|
||||
preview_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# unused_days, acknowledged rules (with the text version), protected management sources.
|
||||
options: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
|
||||
|
||||
class RuleCleanupItem(Base):
|
||||
__tablename__ = "rule_cleanup_items"
|
||||
__table_args__ = (
|
||||
Index("ix_rule_cleanup_items_operation", "operation_id"),
|
||||
# Independent of any lock: one rule on one node can be in only one unfinished item at a time.
|
||||
Index(
|
||||
"uq_cleanup_rule_open", "host_id", "rule_uuid", unique=True,
|
||||
postgresql_where=text("state IN (" + ", ".join(f"'{s}'" for s in OPEN_ITEM_STATES) + ")"),
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
operation_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rule_cleanup_operations.id", ondelete="CASCADE"), nullable=False)
|
||||
host_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="CASCADE"), nullable=False)
|
||||
rule_uuid: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
seq: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
carp_role: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
state: Mapped[str] = mapped_column(String(16), nullable=False, default="queued")
|
||||
# What the preview showed.
|
||||
description: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
categories: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
confidence: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
row_before: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
fp_before: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# shadowed/duplicate: the covering rule must stay unchanged until the deploy.
|
||||
cover_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
cover_fp: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# Flattened getRule, raw answer and firmware; written BEFORE a rule is deleted.
|
||||
snapshot: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
# Where the rule sat (neighbours in the same prio_group by sort_order), for the restore.
|
||||
anchor_next_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
anchor_prev_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
prio_group: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
# delete -> the confirmed disable item it builds on; restore -> the delete item it undoes. Plain id, no foreign key.
|
||||
source_item_id: Mapped[Optional[uuid.UUID]] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
restored_uuid: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
changed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
message: Mapped[Optional[str]] = mapped_column(String(1000), nullable=True)
|
||||
@@ -205,6 +205,8 @@ async def remove_previous_target(alias_id: uuid.UUID, host_id: uuid.UUID, approv
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in alias.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_alias_item(deployment.host, deployment.opnsense_alias_uuid)
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from sqlalchemy import select, func, or_
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models.audit import AuditEvent
|
||||
from app.services.audit_service import ADMIN_ONLY_DETAILS
|
||||
|
||||
router = APIRouter(prefix="/audit", tags=["audit"])
|
||||
|
||||
@router.get("")
|
||||
async def history(kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||
async def history(request: Request, kind: str | None = None, host_id: str | None = None, entity_id: str | None = None,
|
||||
status: str | None = None, action: str | None = None, actor: str | None = None,
|
||||
before: datetime | None = None, since: datetime | None = None,
|
||||
q: str = Query(default="", max_length=200), offset: int = Query(default=0, ge=0),
|
||||
@@ -24,5 +25,17 @@ async def history(kind: str | None = None, host_id: str | None = None, entity_id
|
||||
clauses.append(or_(*(c.ilike('%'+q+'%') for c in [AuditEvent.entity_name, AuditEvent.host_name, AuditEvent.actor, AuditEvent.message])))
|
||||
total = await db.scalar(select(func.count()).select_from(AuditEvent).where(*clauses))
|
||||
rows = (await db.scalars(select(AuditEvent).where(*clauses).order_by(AuditEvent.created_at.desc(), AuditEvent.id.desc()).offset(offset).limit(limit))).all()
|
||||
return {"total": total, "offset": offset, "limit": limit,
|
||||
"items": [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]}
|
||||
items = [{c.name: getattr(row, c.name) for c in AuditEvent.__table__.columns} for row in rows]
|
||||
if not full_view(request):
|
||||
for item in items:
|
||||
if isinstance(item.get("details"), dict) and ADMIN_ONLY_DETAILS in item["details"]:
|
||||
item["details"] = {**{k: v for k, v in item["details"].items() if k != ADMIN_ONLY_DETAILS},
|
||||
ADMIN_ONLY_DETAILS: "redacted"}
|
||||
return {"total": total, "offset": offset, "limit": limit, "items": items}
|
||||
|
||||
|
||||
def full_view(request: Request | None) -> bool:
|
||||
"""An admin account, and with a token only one of scope admin."""
|
||||
user = getattr(getattr(request, "state", None), "user", None)
|
||||
token = getattr(getattr(request, "state", None), "api_token", None)
|
||||
return getattr(user, "role", None) == "admin" and (token is None or token.scope == "admin")
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Konfiguration als Code (1.32): YAML-Export, Plan und Übernahme. Schreibt nur in die OPNexus-Datenbank.
|
||||
|
||||
Rechte (durchgesetzt in AuthMiddleware/api_token_service, hier nicht noch einmal):
|
||||
* Export: jede angemeldete Rolle, Token-Scope `read` genügt (nichts, was die GET-Endpunkte nicht schon zeigen).
|
||||
* Plan: jede Rolle inkl. Nur-Lese und `read`-Token; schreibt nichts (wie eine Vorschau, nicht im Audit).
|
||||
* Übernehmen: nur Admins (Sitzung oder `admin`-Token): Massenänderung inkl. Prune und Host-Hierarchie-Zuordnung.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import MAX_BYTES, ConfigError, parse
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
|
||||
router = APIRouter(prefix="/config", tags=["config"])
|
||||
|
||||
# JSON escapes can grow the YAML text (a quote becomes \", a control character \u001b): room for that, but bounded,
|
||||
# and read before parsing so a huge body never reaches the JSON parser.
|
||||
MAX_REQUEST_BYTES = 4 * MAX_BYTES
|
||||
# One plan/apply at a time per process: the pure-Python YAML parser costs seconds for a full-size file, and plan is
|
||||
# open to read-only accounts and read tokens.
|
||||
_busy = asyncio.Semaphore(1)
|
||||
# ... and a short queue: every waiting request holds its body (up to MAX_REQUEST_BYTES) in memory.
|
||||
MAX_WAITING = 4
|
||||
_waiting = 0
|
||||
|
||||
|
||||
class _Slot:
|
||||
"""Admission before the body is read (a waiting request holds its body in memory), then one at a time."""
|
||||
|
||||
async def __aenter__(self):
|
||||
global _waiting
|
||||
if _waiting >= MAX_WAITING:
|
||||
raise api_error(429, ErrorCode.CONFIG_BUSY)
|
||||
_waiting += 1
|
||||
return self
|
||||
|
||||
async def run(self, coro_fn):
|
||||
async with _busy:
|
||||
return await coro_fn()
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
global _waiting
|
||||
_waiting -= 1
|
||||
|
||||
|
||||
def _encode(text: str) -> bytes:
|
||||
try:
|
||||
return text.encode("utf-8")
|
||||
except UnicodeEncodeError: # a lone surrogate from a JSON \ud800 escape
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_encoding", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
class ImportRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
yaml: str
|
||||
prune: bool = False
|
||||
|
||||
|
||||
class ApplyRequest(ImportRequest):
|
||||
plan_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
def _config_error(exc: ConfigError):
|
||||
status = 413 if exc.code == "CONFIG_TOO_LARGE" else 422
|
||||
return api_error(status, ErrorCode(exc.code), errors=exc.errors)
|
||||
|
||||
|
||||
async def _read_body(request: Request, model: type[BaseModel]):
|
||||
if int(request.headers.get("content-length") or 0) > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks, size = [], 0
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_REQUEST_BYTES:
|
||||
raise api_error(413, ErrorCode.CONFIG_TOO_LARGE, errors=[{"code": "too_large", "path": "", "line": None, "params": {"max": MAX_BYTES}}])
|
||||
chunks.append(chunk)
|
||||
try:
|
||||
return model.model_validate(json.loads(b"".join(chunks)))
|
||||
except (ValueError, ValidationError):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "bad_request", "path": "", "line": None, "params": {}}]) from None
|
||||
|
||||
|
||||
@router.get("/export")
|
||||
async def export(kinds: str | None = Query(default=None, max_length=300), db: AsyncSession = Depends(get_db)):
|
||||
wanted = None
|
||||
if kinds:
|
||||
wanted = {k.strip() for k in kinds.split(",") if k.strip()}
|
||||
if not wanted <= set(schema.KINDS):
|
||||
raise api_error(422, ErrorCode.CONFIG_INVALID, errors=[{"code": "unknown_kind", "path": "kinds", "line": None, "params": {}}])
|
||||
text = await export_yaml(db, wanted)
|
||||
return Response(content=text.encode("utf-8"), media_type="text/yaml; charset=utf-8",
|
||||
headers={"Content-Disposition": 'attachment; filename="opnexus-config.yaml"', "Cache-Control": "no-store"})
|
||||
|
||||
|
||||
@router.post("/import/plan")
|
||||
async def plan(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ImportRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
return await planner.build_plan(db, parsed, body.prune)
|
||||
except ConfigError as exc:
|
||||
raise _config_error(exc) from None
|
||||
finally:
|
||||
await db.rollback() # never leave anything in the session: plan is read-only
|
||||
|
||||
result = await slot.run(work)
|
||||
return result.public()
|
||||
|
||||
|
||||
@router.post("/import/apply")
|
||||
async def apply(request: Request, db: AsyncSession = Depends(get_db)):
|
||||
async with _Slot() as slot:
|
||||
body = await _read_body(request, ApplyRequest)
|
||||
raw = _encode(body.yaml)
|
||||
|
||||
async def work():
|
||||
try:
|
||||
parsed = await run_in_threadpool(parse, raw)
|
||||
result = await applier.apply_config(db, parsed, body.prune, body.plan_token)
|
||||
except ConfigError as exc:
|
||||
await db.rollback()
|
||||
raise _config_error(exc) from None
|
||||
except applier.PlanStale:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_PLAN_STALE) from None
|
||||
except applier.PlanBlocked as exc:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.CONFIG_BLOCKED, summary=exc.plan.summary()) from None
|
||||
except applier.SelfCheckFailed as exc:
|
||||
await db.rollback()
|
||||
raise api_error(500, ErrorCode.CONFIG_APPLY_SELFCHECK, objects=exc.differences[:20]) from None
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
return parsed, result
|
||||
|
||||
parsed, result = await slot.run(work)
|
||||
return {"summary": result.summary(), "sha256": parsed.sha256, "kinds": result.touched}
|
||||
@@ -1,6 +1,15 @@
|
||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it)."""
|
||||
"""Separation of duties: with ``operators_may_deploy`` off, only admins roll changes out (AuthMiddleware enforces it).
|
||||
|
||||
The rule-cleanup settings (1.31) live here too: how many rules one cleanup may touch, how long a rule must stay
|
||||
switched off before it may be deleted, the management sources (addresses the firewalls are managed from; no rule
|
||||
that could match them is ever cleaned up) and extra management ports (besides the API port, 443, 80 and 22). Only an admin writes any of it (middleware: ``/api/settings/...`` is
|
||||
admin-only for operators and tokens below scope admin; checked again here) and every change is audited with the
|
||||
old and the new values."""
|
||||
import ipaddress
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
@@ -8,12 +17,83 @@ from app.errors import ErrorCode, api_error
|
||||
from app.models.governance import GovernanceSettings
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.services import audit_service
|
||||
|
||||
router = APIRouter(prefix="/settings/governance", tags=["governance"])
|
||||
|
||||
MAX_CLEANUP_RULES = 25 # same ceiling as services/rule_cleanup_service.MAX_RULES_CEILING
|
||||
MAX_DELETE_MIN_DAYS = 365
|
||||
MAX_MANAGEMENT_SOURCES = 20
|
||||
MAX_MANAGEMENT_PORTS = 20
|
||||
|
||||
|
||||
class GovernanceRead(BaseModel):
|
||||
operators_may_deploy: bool
|
||||
cleanup_max_rules: int = 10
|
||||
cleanup_delete_min_days: int = 7
|
||||
# The list itself is shown to admins only; everybody else sees just whether it is set.
|
||||
cleanup_management_sources: list[str] = Field(default_factory=list)
|
||||
cleanup_management_sources_configured: bool = False
|
||||
# Extra management ports (besides API port, 443, 80, 22): also admins only, others see the flag.
|
||||
cleanup_management_ports: list[int] = Field(default_factory=list)
|
||||
cleanup_management_ports_configured: bool = False
|
||||
|
||||
|
||||
class GovernanceUpdate(BaseModel):
|
||||
operators_may_deploy: bool | None = None
|
||||
cleanup_max_rules: int | None = None
|
||||
cleanup_delete_min_days: int | None = None
|
||||
cleanup_management_sources: list[str] | None = None
|
||||
cleanup_management_ports: list[Any] | None = None # validated in clean_management_ports (no bool coercion)
|
||||
|
||||
|
||||
def _bad(field: str, reason: str):
|
||||
return api_error(422, ErrorCode.BEREINIGUNG_EINSTELLUNG_UNGUELTIG, field=field, reason=reason)
|
||||
|
||||
|
||||
def clean_management_sources(values: list[str]) -> list[str]:
|
||||
"""IP addresses and CIDR networks only (no host names: they could resolve differently later), canonical form,
|
||||
without duplicates; not the whole address space, which would protect every rule."""
|
||||
if len(values) > MAX_MANAGEMENT_SOURCES:
|
||||
raise _bad("cleanup_management_sources", "too_many")
|
||||
cleaned: list[str] = []
|
||||
for raw in values:
|
||||
text = str(raw).strip()
|
||||
try:
|
||||
net = ipaddress.ip_network(text, strict=False)
|
||||
except ValueError:
|
||||
raise _bad("cleanup_management_sources", "not_an_address") from None
|
||||
if net.prefixlen == 0:
|
||||
raise _bad("cleanup_management_sources", "everything")
|
||||
entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)
|
||||
if entry not in cleaned:
|
||||
cleaned.append(entry)
|
||||
return cleaned
|
||||
|
||||
|
||||
def clean_management_ports(values) -> list[int]:
|
||||
"""TCP ports 1..65535, at most MAX_MANAGEMENT_PORTS, without duplicates, sorted."""
|
||||
if len(values) > MAX_MANAGEMENT_PORTS:
|
||||
raise _bad("cleanup_management_ports", "too_many")
|
||||
ports: set[int] = set()
|
||||
for raw in values:
|
||||
text = str(raw).strip()
|
||||
if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:
|
||||
raise _bad("cleanup_management_ports", "not_a_port")
|
||||
ports.add(int(text))
|
||||
return sorted(ports)
|
||||
|
||||
|
||||
def _read(row: GovernanceSettings, user: AdminUser) -> GovernanceRead:
|
||||
sources = list(row.cleanup_management_sources or [])
|
||||
ports = list(row.cleanup_management_ports or [])
|
||||
return GovernanceRead(
|
||||
operators_may_deploy=row.operators_may_deploy, cleanup_max_rules=row.cleanup_max_rules or 10,
|
||||
cleanup_delete_min_days=row.cleanup_delete_min_days or 7,
|
||||
cleanup_management_sources=sources if user.role == "admin" else [],
|
||||
cleanup_management_sources_configured=bool(sources),
|
||||
cleanup_management_ports=ports if user.role == "admin" else [],
|
||||
cleanup_management_ports_configured=bool(ports))
|
||||
|
||||
|
||||
async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
@@ -28,14 +108,33 @@ async def get_settings(db: AsyncSession) -> GovernanceSettings:
|
||||
@router.get("", response_model=GovernanceRead)
|
||||
async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
"""Readable by everybody signed in (the UI explains why a deploy was refused)."""
|
||||
return await get_settings(db)
|
||||
return _read(await get_settings(db), user)
|
||||
|
||||
|
||||
@router.put("", response_model=GovernanceRead)
|
||||
async def update(payload: GovernanceRead, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
async def update(payload: GovernanceUpdate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
row = await get_settings(db)
|
||||
row.operators_may_deploy = payload.operators_may_deploy
|
||||
changes: dict[str, dict] = {}
|
||||
if payload.cleanup_max_rules is not None:
|
||||
if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:
|
||||
raise _bad("cleanup_max_rules", f"1..{MAX_CLEANUP_RULES}")
|
||||
if payload.cleanup_delete_min_days is not None:
|
||||
if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:
|
||||
raise _bad("cleanup_delete_min_days", f"1..{MAX_DELETE_MIN_DAYS}")
|
||||
sources = clean_management_sources(payload.cleanup_management_sources) if payload.cleanup_management_sources is not None else None
|
||||
ports = clean_management_ports(payload.cleanup_management_ports) if payload.cleanup_management_ports is not None else None
|
||||
for field, value in (("cleanup_max_rules", payload.cleanup_max_rules), ("cleanup_delete_min_days", payload.cleanup_delete_min_days),
|
||||
("cleanup_management_sources", sources), ("cleanup_management_ports", ports)):
|
||||
if value is not None and value != getattr(row, field):
|
||||
changes[field] = {"old": getattr(row, field), "new": value}
|
||||
setattr(row, field, value)
|
||||
if payload.operators_may_deploy is not None:
|
||||
row.operators_may_deploy = payload.operators_may_deploy
|
||||
if changes:
|
||||
db.add(audit_service.make_event(
|
||||
kind="settings", action="cleanup_settings", status="accepted", entity_id="governance",
|
||||
message="Einstellungen der Regel-Bereinigung geändert", details={"changes": changes}))
|
||||
await db.commit()
|
||||
return row
|
||||
return _read(row, user)
|
||||
@@ -147,6 +147,11 @@ async def get_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def update_host(host_id: uuid.UUID, payload: HostUpdate, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _get_host_or_404(db, host_id)
|
||||
if (payload.base_url is not None or payload.enabled is not None or payload.clear_cluster_name
|
||||
or payload.cluster_name is not None):
|
||||
# An open rule cleanup (1.31) reverts through this address and this cluster pairing: not while it is open.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
|
||||
if payload.name is not None:
|
||||
existing = await db.scalar(
|
||||
@@ -188,6 +193,9 @@ async def delete_host(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
# An open rule cleanup (1.31) holds a revert deadline and snapshots for this host (and its cluster peers).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
# Both deployment tables FK to opnsense_hosts with ondelete=CASCADE, so
|
||||
# deleting the host would silently drop OPNexus's only record of what
|
||||
# it pushed there — the rule/alias itself stays live on the actual
|
||||
|
||||
@@ -193,6 +193,8 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_child_uuid:
|
||||
|
||||
@@ -9,14 +9,14 @@ from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.services import opnsense_client as client, audit_service, carp_maintenance_window_service
|
||||
from app.services import opnsense_client as client, audit_service, backup_store, carp_maintenance_window_service, snapshot_service
|
||||
from app.services.host_maintenance_service import inspect_host, inspect_carp_switch, fingerprint, peers, ACTIVE
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/maintenance", tags=["host-maintenance"])
|
||||
|
||||
class Preview(BaseModel):
|
||||
action: Literal['reboot','update','major_update']
|
||||
action: Literal['reboot','update','major_update','firmware_rollback']
|
||||
|
||||
class Execute(BaseModel):
|
||||
preview_id: uuid.UUID
|
||||
@@ -41,10 +41,69 @@ async def host_or_404(db, host_id):
|
||||
def read_operation(op):
|
||||
return {"id":op.id,"host_id":op.host_id,"action":op.action,"status":op.status,"created_at":op.created_at,"expires_at":op.expires_at,"actor":op.actor,"message":op.message}
|
||||
|
||||
async def snapshot_or_502(db, host):
|
||||
try: return await inspect_host(db,host)
|
||||
async def snapshot_or_502(db, host, action=None):
|
||||
try:
|
||||
state=await inspect_host(db,host)
|
||||
if action=='firmware_rollback':
|
||||
# Read only for this action, so every other preview/status keeps its exact set of firewall calls.
|
||||
rollback=await snapshot_service.rollback_state(host)
|
||||
rollback['changes_since']=await _changes_since(db,host,rollback.get('target'))
|
||||
state['rollback']=rollback
|
||||
state['actions']['firmware_rollback']=state['actions']['reboot']+(
|
||||
[] if rollback['supported'] else ['Die Firewall hat keine ZFS-Snapshots (Boot-Environments).'])+(
|
||||
[] if rollback.get('target') else ['Kein Rückweg-Punkt vorhanden: OPNexus legt ihn vor jedem Firmware-Update an.'])
|
||||
return state
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
|
||||
|
||||
async def _changes_since(db, host, target):
|
||||
"""OPNexus' own pushes to this host after the rollback point: going back removes them from the firewall."""
|
||||
if not target:
|
||||
return []
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return []
|
||||
out=[]
|
||||
for kind,model in _DEPLOYMENT_MODELS():
|
||||
rows=(await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all()
|
||||
out+=[{'kind':kind,'status':r.status,'deployed_at':r.deployed_at.isoformat()} for r in rows]
|
||||
return sorted(out,key=lambda r:r['deployed_at'])[:50]
|
||||
|
||||
|
||||
def _point_time(target):
|
||||
try:
|
||||
return datetime.strptime(target['stamp'],'%Y%m%d-%H%M%S').replace(tzinfo=timezone.utc)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _DEPLOYMENT_MODELS():
|
||||
from app.models import (AliasDeployment, OneToOneDeployment, OutboundNatDeployment, PolicyHierarchyRuleDeployment,
|
||||
PortForwardDeployment, RuleDeployment)
|
||||
from app.models.ipsec_tunnel_deployment import IpsecTunnelDeployment
|
||||
from app.models.openvpn_instance_deployment import OpenVpnInstanceDeployment
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
return (('rule',RuleDeployment),('alias',AliasDeployment),('port_forward',PortForwardDeployment),
|
||||
('one_to_one',OneToOneDeployment),('outbound_nat',OutboundNatDeployment),('policy_hierarchy',PolicyHierarchyRuleDeployment),
|
||||
('ipsec_tunnel',IpsecTunnelDeployment),('openvpn_instance',OpenVpnInstanceDeployment),('wireguard_tunnel',WireguardTunnelDeployment))
|
||||
|
||||
|
||||
async def _mark_rolled_back(db, host, target):
|
||||
"""After going back, what OPNexus pushed since the point is no longer on the firewall. A redeploy with an unchanged
|
||||
payload only re-affirms (writes nothing), so the remembered payload is cleared and the row marked: the next deploy
|
||||
really writes, and the UI shows the object as not in sync."""
|
||||
since=_point_time(target)
|
||||
if since is None:
|
||||
return 0
|
||||
count=0
|
||||
note=f"Firmware-Rückweg auf {target['name']}: Stand auf der Firewall unklar. Live-Stand prüfen und neu ausrollen."
|
||||
for _kind,model in _DEPLOYMENT_MODELS():
|
||||
for row in (await db.scalars(select(model).where(model.host_id==host.id, model.deployed_at>since))).all():
|
||||
row.status='error'; row.message=note[:500]; row.last_payload=None
|
||||
if hasattr(model,'revert_deadline'): row.revert_deadline=None
|
||||
count+=1
|
||||
return count
|
||||
|
||||
async def carp_snapshot_or_502(db, host):
|
||||
try: return await inspect_carp_switch(db,host)
|
||||
except client.OpnsenseAPIError as exc: raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
@@ -62,7 +121,7 @@ async def status(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
async def preview(host_id: uuid.UUID, payload: Preview, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host=await host_or_404(db,host_id)
|
||||
state=await snapshot_or_502(db,host)
|
||||
state=await snapshot_or_502(db,host,payload.action)
|
||||
if state['actions'][payload.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][payload.action]))
|
||||
op=HostOperation(host_id=host.id,host_name=host.name,actor=request.state.user.username,action=payload.action,
|
||||
expires_at=datetime.now(timezone.utc)+timedelta(minutes=5),fingerprint=fingerprint(state),snapshot=state)
|
||||
@@ -77,17 +136,52 @@ async def execute(host_id: uuid.UUID, payload: Execute, request: Request, db: As
|
||||
if not op or op.host_id!=host_id or op.actor!=request.state.user.username: raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
if op.status!='preview' or op.expires_at <= datetime.now(timezone.utc): raise api_error(409, ErrorCode.VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET)
|
||||
if payload.confirm_name != host.name: raise api_error(409, ErrorCode.HOSTNAMEN_EXAKT_EINGEBEN)
|
||||
state=await snapshot_or_502(db,host)
|
||||
if op.action not in ('reboot','update','major_update','firmware_rollback'): raise api_error(404, ErrorCode.WARTUNGSVORSCHAU_NICHT_GEFUNDEN)
|
||||
state=await snapshot_or_502(db,host,op.action)
|
||||
if state['actions'][op.action]: raise api_error(409, ErrorCode.ACTION_NOT_ALLOWED, reason=' '.join(state['actions'][op.action]))
|
||||
if fingerprint(state)!=op.fingerprint: raise api_error(409, ErrorCode.HOSTZUSTAND_WURDE_VERAENDERT_VORSCHAU_ERNEUT_OEFFNEN)
|
||||
# Consume durably BEFORE the remote request. A timeout is ambiguous, and
|
||||
# replaying a reboot/update automatically could interrupt recovery.
|
||||
op.status='starting'; op.started_at=datetime.now(timezone.utc); await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='requested',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),details={'snapshot':state})
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
if op.action in ('update','major_update'):
|
||||
# Way back first (1.32, B5): a boot environment of the running system. Fail closed: no point, no update.
|
||||
try:
|
||||
point=await snapshot_service.create_rollback_point(host,state.get('version'))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Rückweg-Punkt (Boot-Environment) nicht angelegt: '+str(exc)[:500]+'. Update nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
op.snapshot={**op.snapshot,'rollback_point':point}
|
||||
await db.commit()
|
||||
if op.action=='firmware_rollback':
|
||||
# Going back discards every change on the firewall since the point: keep a copy of the current config first.
|
||||
# Fail closed, like the backup before an update.
|
||||
try:
|
||||
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='cancelled';op.message='Sicherung der aktuellen Konfiguration fehlgeschlagen: '+str(exc)[:500]+'. Rückweg nicht gestartet.'
|
||||
await db.commit()
|
||||
await audit_service.persist(kind='host',action=op.action,status='cancelled',host_id=str(host.id),host_name=host.name,entity_id=str(op.id),message=op.message,details={})
|
||||
return read_operation(op)
|
||||
rollback=state['rollback']
|
||||
action_fn=lambda h: snapshot_service.activate_and_reboot(h,rollback)
|
||||
done=(f"Boot-Environment {rollback['target']['name']} aktiviert, Neustart übergeben. Nach dem Start Version prüfen "
|
||||
f"(erwartet {rollback['target']['version']}) und Wartung abschließen. Alles, was seit dem Rückweg-Punkt auf der "
|
||||
"Firewall geändert wurde, ist danach nicht mehr vorhanden: zentrale Objekte über die Vorschau neu ausrollen.")
|
||||
else:
|
||||
action_fn={'reboot':client.reboot_host,'update':client.update_firmware,'major_update':client.upgrade_firmware}[op.action]
|
||||
done='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
if op.action in ('update','major_update') and not (op.snapshot or {}).get('rollback_point'):
|
||||
done+=' Kein Rückweg-Punkt: die Firewall hat keine ZFS-Snapshots.'
|
||||
try:
|
||||
result=await action_fn(host)
|
||||
op.status='submitted';op.message='An Firewall übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen.'
|
||||
op.status='submitted';op.message=done
|
||||
if op.action=='firmware_rollback':
|
||||
marked=await _mark_rolled_back(db,host,rollback['target'])
|
||||
if marked:
|
||||
op.message+=f' {marked} Deployment(s) als unklar markiert.'
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status='unknown';op.message='Ausgang unklar: '+str(exc)[:800]+'. Nicht automatisch wiederholen.'
|
||||
result={}
|
||||
|
||||
@@ -50,6 +50,7 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
@@ -231,6 +232,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_one_to_one_rule(deployment.host, deployment.opnsense_one_to_one_uuid)
|
||||
|
||||
@@ -195,6 +195,8 @@ async def remove_previous_target(instance_id: uuid.UUID, host_id: uuid.UUID, app
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in instance.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_instance_uuid:
|
||||
|
||||
@@ -51,6 +51,7 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
source_net=rule.source_net,
|
||||
@@ -238,6 +239,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_outbound_nat_rule(deployment.host, deployment.opnsense_outbound_nat_uuid)
|
||||
|
||||
@@ -75,6 +75,7 @@ def _to_read(rule: PolicyHierarchyRule, all_groups: list[PolicyHierarchyGroup])
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
hierarchy_group_id=rule.hierarchy_group_id,
|
||||
rulebase=rule.rulebase,
|
||||
sequence=rule.sequence,
|
||||
@@ -330,6 +331,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
try:
|
||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_policy_hierarchy_uuid)
|
||||
await opnsense_client.apply(deployment.host)
|
||||
|
||||
@@ -50,6 +50,7 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
workflow_state=state,
|
||||
workflow_counts=counts,
|
||||
id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
ipprotocol=rule.ipprotocol,
|
||||
@@ -233,6 +234,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
await opnsense_client.del_port_forward_rule(deployment.host, deployment.opnsense_port_forward_uuid)
|
||||
|
||||
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -20,7 +20,7 @@ from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services import audit_service, diagnostics_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||
@@ -36,6 +36,19 @@ class WaveIn(BaseModel):
|
||||
require_approval: bool = False
|
||||
|
||||
|
||||
class ProbeIn(BaseModel):
|
||||
"""A reachability probe run FROM the firewall in the health gate (same target rules as the host diagnostics)."""
|
||||
type: Literal["ping", "tcp"]
|
||||
target: str = Field(min_length=1, max_length=253, pattern=diagnostics_service.TARGET.pattern)
|
||||
port: int | None = Field(default=None, ge=1, le=65535)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _port_only_for_tcp(self):
|
||||
if (self.type == "tcp") != (self.port is not None):
|
||||
raise ValueError("port_required_for_tcp_only")
|
||||
return self
|
||||
|
||||
|
||||
class PlanIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
description: str = Field(default="", max_length=2000)
|
||||
@@ -43,6 +56,7 @@ class PlanIn(BaseModel):
|
||||
window_id: uuid.UUID | None = None
|
||||
auto_start: bool = False
|
||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||
gate_probes: list[ProbeIn] = Field(default_factory=list, max_length=rollout_health.MAX_PROBES)
|
||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||
|
||||
@@ -64,6 +78,7 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||
return {
|
||||
**rollout_service.definition(plan),
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"description": plan.description,
|
||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||
@@ -81,6 +96,15 @@ async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None =
|
||||
}
|
||||
|
||||
|
||||
def _unique_probes(probes: list[ProbeIn]) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
for probe in probes:
|
||||
value = {"type": probe.type, "target": probe.target, "port": probe.port}
|
||||
if value not in out:
|
||||
out.append(value)
|
||||
return out
|
||||
|
||||
|
||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
@@ -99,6 +123,7 @@ async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||
plan.gate_probes = _unique_probes(payload.gate_probes)
|
||||
plan.compliance_mode = payload.compliance_mode
|
||||
plan.waves = [
|
||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): switch hand-made rules off, delete them later, restore them (DESIGN-1.31 §4).
|
||||
|
||||
POST /hosts/{id}/rule-cleanup/preview operator, admin prepare (writes nothing to a firewall)
|
||||
POST /hosts/{id}/rule-cleanup/{op}/cancel operator, admin the preview's own actor
|
||||
POST /hosts/{id}/rule-cleanup/{op}/deploy ADMIN, token scope admin
|
||||
POST /hosts/{id}/rule-cleanup/{op}/confirm operator, admin BROWSER SESSION ONLY, never a token
|
||||
POST /hosts/{id}/rule-cleanup/{op}/revert admin; operator only with operators_may_deploy
|
||||
POST /hosts/{id}/rule-cleanup/{op}/retry ADMIN, token scope admin
|
||||
POST /hosts/{id}/rule-cleanup/items/{item}/restore ADMIN, token scope admin
|
||||
GET /hosts/{id}/rule-cleanup, /{op} everybody signed in (read tokens too)
|
||||
|
||||
Who may do what is enforced twice: AuthMiddleware (services/api_token_service.py: _ADMIN_ONLY_CLEANUP,
|
||||
_SESSION_ONLY, _ROLLOUT_POST) and here, so a gap in one layer does not open the route. The audit trail is the
|
||||
service's (one ``rule_cleanup`` entry per step and rule) plus the middleware's request entry (kind ``host``,
|
||||
action = last path segment; ``/preview`` is not logged by it, the service writes its own entry).
|
||||
The services raise ``api_error``; nothing is translated here. MCP stays read-only (no tool, ``api_get`` is GET).
|
||||
"""
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.user import AdminUser
|
||||
from app.routers import governance
|
||||
from app.routers.auth import resolve_user
|
||||
from app.schemas.rule_cleanup import CleanupOperationRead, CleanupPreviewRead, ConfirmName, PreviewRequest
|
||||
from app.services import rule_cleanup_service as service
|
||||
from app.services.rule_cleanup_service import ACK_TEXT_VERSION
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/rule-cleanup", tags=["rule-cleanup"])
|
||||
|
||||
|
||||
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
return host
|
||||
|
||||
|
||||
def _token(request: Request):
|
||||
return getattr(request.state, "api_token", None)
|
||||
|
||||
|
||||
def _writer(user: AdminUser, request: Request) -> None:
|
||||
"""Any write: not for a read-only account, not with a read-scope token."""
|
||||
if user.role == "read_only":
|
||||
raise api_error(403, ErrorCode.READ_ONLY_ACCESS)
|
||||
token = _token(request)
|
||||
if token is not None and token.scope == "read":
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
def _admin_only(user: AdminUser, request: Request) -> None:
|
||||
"""Deploy, retry, restore: an admin account, and with a token only one of scope admin."""
|
||||
_writer(user, request)
|
||||
if user.role != "admin":
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
token = _token(request)
|
||||
if token is not None and token.scope != "admin":
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
async def _deployer(user: AdminUser, request: Request, db: AsyncSession) -> None:
|
||||
"""Revert: an admin, or an operator while the governance switch operators_may_deploy is on."""
|
||||
_writer(user, request)
|
||||
if user.role == "admin":
|
||||
return
|
||||
if user.role == "operator" and (await governance.get_settings(db)).operators_may_deploy:
|
||||
return
|
||||
raise api_error(403, ErrorCode.DEPLOY_NEEDS_ADMIN)
|
||||
|
||||
|
||||
def _session_only(user: AdminUser, request: Request) -> None:
|
||||
"""Confirm: "I still have access" is a human decision in a browser session, never a token's."""
|
||||
_writer(user, request)
|
||||
token = _token(request)
|
||||
if token is not None:
|
||||
raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)
|
||||
|
||||
|
||||
def _full_view(request: Request) -> bool:
|
||||
"""Admins (session or admin-scope token) see the management sources/ports and firewall addresses in a plan."""
|
||||
from app.routers.audit import full_view
|
||||
return full_view(request)
|
||||
|
||||
|
||||
async def _names(db: AsyncSession, operations: list[dict], request: Request) -> list[dict]:
|
||||
"""The firewall names the UI shows next to the ids; for non-admins without the security-relevant plan parts."""
|
||||
if not _full_view(request):
|
||||
operations = [service.redact(op) for op in operations]
|
||||
ids = {uuid.UUID(op["host_id"]) for op in operations} | {uuid.UUID(i["host_id"]) for op in operations for i in op["items"]}
|
||||
names = {h.id: h.name for h in (await db.scalars(select(OpnsenseHost).where(OpnsenseHost.id.in_(ids)))).all()} if ids else {}
|
||||
for op in operations:
|
||||
op["host_name"] = names.get(uuid.UUID(op["host_id"]))
|
||||
for item in op["items"]:
|
||||
item["host_name"] = names.get(uuid.UUID(item["host_id"]))
|
||||
return operations
|
||||
|
||||
|
||||
async def _one(db: AsyncSession, operation: dict, request: Request) -> dict:
|
||||
return (await _names(db, [operation], request))[0]
|
||||
|
||||
|
||||
@router.get("", response_model=list[CleanupOperationRead])
|
||||
async def list_operations(host_id: uuid.UUID, request: Request, limit: int = Query(50, ge=1, le=100),
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""The newest operations of this host (and its cluster peers) with their items; unexpired previews included."""
|
||||
host = await _host(db, host_id)
|
||||
return await _names(db, await service.list_operations(db, host, limit), request)
|
||||
|
||||
|
||||
@router.post("/preview", response_model=CleanupPreviewRead)
|
||||
async def preview(host_id: uuid.UUID, payload: PreviewRequest, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Recompute the plan from the live firewall(s) and store a 5-minute preview bound to the caller."""
|
||||
user = await resolve_user(request, db)
|
||||
_writer(user, request)
|
||||
host = await _host(db, host_id)
|
||||
kind, uuids, acknowledge = payload.resolved()
|
||||
if kind is None:
|
||||
raise api_error(422, ErrorCode.BEREINIGUNG_AUSWAHL_UNGUELTIG, field="kind")
|
||||
result = await service.preview(
|
||||
db, host, actor=user.username, actor_id=user.id, kind=kind, uuids=uuids, acknowledge=acknowledge,
|
||||
**({"unused_days": payload.unused_days} if payload.unused_days else {}),
|
||||
ack_text_version=payload.ack_text_version or ACK_TEXT_VERSION)
|
||||
result["operation"] = await _one(db, await service.get_operation(db, host, uuid.UUID(result["preview_id"])), request)
|
||||
return result if _full_view(request) else service.redact(result)
|
||||
|
||||
|
||||
@router.get("/{op_id}", response_model=CleanupOperationRead)
|
||||
async def get_operation(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.get_operation(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/cancel", response_model=CleanupOperationRead)
|
||||
async def cancel(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Drop an unused preview (only its own actor can)."""
|
||||
user = await resolve_user(request, db)
|
||||
_writer(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.cancel(db, host, op_id, actor=user.username), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/deploy", response_model=CleanupOperationRead)
|
||||
async def deploy(host_id: uuid.UUID, op_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""Execute a preview. Admin only; the live state must still give the fingerprint of the preview."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.deploy(db, host, op_id, actor=user.username, confirm_name=payload.confirm_name), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/confirm", response_model=CleanupOperationRead)
|
||||
async def confirm(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
""""I still have access": re-reads every node first. Browser session only, never with an API token."""
|
||||
user = await resolve_user(request, db)
|
||||
_session_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.confirm(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/revert", response_model=CleanupOperationRead)
|
||||
async def revert(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Take the change back now (the same code the sweeper runs when the window passes)."""
|
||||
user = await resolve_user(request, db)
|
||||
await _deployer(user, request, db)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.revert_now(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/{op_id}/retry", response_model=CleanupOperationRead)
|
||||
async def retry(host_id: uuid.UUID, op_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
"""Revert again after revert_failed, or read the firewall again for a delete/restore with an unclear item."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.retry(db, host, op_id), request)
|
||||
|
||||
|
||||
@router.post("/items/{item_id}/restore", response_model=CleanupOperationRead)
|
||||
async def restore(host_id: uuid.UUID, item_id: uuid.UUID, payload: ConfirmName, request: Request,
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
"""Bring a deleted rule back, switched OFF, at its old position (a new UUID). The result is the restore run;
|
||||
``warnings`` can hold BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR."""
|
||||
user = await resolve_user(request, db)
|
||||
_admin_only(user, request)
|
||||
host = await _host(db, host_id)
|
||||
return await _one(db, await service.restore_item(
|
||||
db, host, item_id, actor=user.username, actor_id=user.id, confirm_name=payload.confirm_name), request)
|
||||
@@ -52,6 +52,7 @@ def _to_read(
|
||||
return RuleRead(
|
||||
workflow_state=state,
|
||||
workflow_counts=counts, id=rule.id,
|
||||
config_key=rule.config_key,
|
||||
description=rule.description,
|
||||
interface=rule.interface,
|
||||
direction=rule.direction,
|
||||
@@ -108,6 +109,10 @@ async def adopt_rule(payload: AdoptRequest, db: AsyncSession = Depends(get_db)):
|
||||
host = await db.get(OpnsenseHost, payload.host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
# Taking over a hand-made rule that a rule cleanup (1.31) is switching off/deleting would make it "managed"
|
||||
# under the cleanup's feet: not while one is open on this host or its cluster.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [host])
|
||||
try:
|
||||
rule = await rule_adopt_service.adopt(db, host, payload.rule_uuid, link_to=payload.link_to)
|
||||
except rule_adopt_service.RuleNotFoundOnHost as exc:
|
||||
@@ -276,6 +281,8 @@ async def remove_previous_target(rule_id: uuid.UUID, host_id: uuid.UUID, approva
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_OHNE_OFFENEN_ROLLBACK_KOENNEN_06DE0ADB)
|
||||
deployment = next(d for d in rule.deployments if d.host_id == host_id)
|
||||
from app.services import opnsense_client
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
try:
|
||||
await opnsense_client.del_rule(deployment.host, deployment.opnsense_rule_uuid)
|
||||
await opnsense_client.apply(deployment.host)
|
||||
|
||||
@@ -207,6 +207,8 @@ async def remove_previous_target(tunnel_id: uuid.UUID, host_id: uuid.UUID, appro
|
||||
if not entry["can_remove"]:
|
||||
raise api_error(409, ErrorCode.NUR_FRUEHERE_AKTIVE_ZIELE_KOENNEN_BEREINIGT_WERDEN)
|
||||
deployment = next(d for d in tunnel.deployments if d.host_id == host_id)
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes
|
||||
from app.services import opnsense_client
|
||||
try:
|
||||
if deployment.opnsense_peer_uuid:
|
||||
|
||||
@@ -35,6 +35,7 @@ class OneToOneRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -37,6 +37,7 @@ class OutboundNatRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
source_net: str
|
||||
|
||||
@@ -62,6 +62,7 @@ class PolicyHierarchyRuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
hierarchy_group_id: uuid.UUID
|
||||
rulebase: Literal["pre", "post"]
|
||||
sequence: int
|
||||
|
||||
@@ -36,6 +36,7 @@ class PortForwardRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
ipprotocol: str
|
||||
|
||||
@@ -48,6 +48,7 @@ class RuleRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
config_key: str
|
||||
description: str
|
||||
interface: str
|
||||
direction: str
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Request and response shapes of the rule-cleanup API (routers/rule_cleanup.py, DESIGN-1.31 §4).
|
||||
|
||||
Responses mirror services/rule_cleanup_service.read_operation. Texts are not part of the API: ``message`` holds an
|
||||
ErrorCode name (``errors.<CODE>`` in the UI, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR) or the firewall's
|
||||
own wording; ``warnings`` is a list of ErrorCode names.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
CleanupKind = Literal["disable", "enable", "delete"]
|
||||
# Longest list the API accepts at all; the effective limit is the setting cleanup_max_rules (ceiling 25).
|
||||
MAX_SELECTION = 100
|
||||
|
||||
|
||||
class PreviewItem(BaseModel):
|
||||
"""One rule of the selection. ``kind`` (alias ``mode``) must be the same for all items of one preview."""
|
||||
uuid: str = Field(min_length=1, max_length=64)
|
||||
kind: CleanupKind | None = None
|
||||
mode: CleanupKind | None = None
|
||||
# The per-rule acknowledgement for a "probable" rule (the UI's checkbox).
|
||||
ack: bool = False
|
||||
|
||||
|
||||
class PreviewRequest(BaseModel):
|
||||
"""``{kind, uuids, acknowledge}`` or ``{items:[{uuid, kind|mode, ack}]}`` (or both: they are merged)."""
|
||||
kind: CleanupKind | None = None
|
||||
uuids: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
items: list[PreviewItem] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
acknowledge: list[str] = Field(default_factory=list, max_length=MAX_SELECTION)
|
||||
unused_days: int | None = Field(default=None, ge=1, le=3650)
|
||||
ack_text_version: str | None = Field(default=None, max_length=16)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _single_kind(self):
|
||||
kinds = {k for k in [self.kind, *[i.kind or i.mode for i in self.items]] if k}
|
||||
if len(kinds) > 1:
|
||||
raise ValueError("kind must be the same for every item")
|
||||
return self
|
||||
|
||||
def resolved(self) -> tuple[str | None, list[str], list[str]]:
|
||||
kind = self.kind or next((i.kind or i.mode for i in self.items if i.kind or i.mode), None)
|
||||
uuids = [*self.uuids, *[i.uuid for i in self.items]]
|
||||
acknowledge = [*self.acknowledge, *[i.uuid for i in self.items if i.ack]]
|
||||
return kind, list(dict.fromkeys(u.strip() for u in uuids)), list(dict.fromkeys(a.strip() for a in acknowledge))
|
||||
|
||||
|
||||
class ConfirmName(BaseModel):
|
||||
"""Deploy and restore: the exact host name, typed by a human."""
|
||||
confirm_name: str = Field(max_length=255)
|
||||
|
||||
|
||||
class CleanupItemRead(BaseModel):
|
||||
id: str
|
||||
host_id: str
|
||||
host_name: str | None = None
|
||||
rule_uuid: str
|
||||
seq: int
|
||||
carp_role: str | None = None
|
||||
# queued, disabling, disabled, confirmed, reverting, reverted, deleting, deleted, restoring, restored, failed,
|
||||
# unknown, revert_failed, skipped (never attempted: the run stopped earlier)
|
||||
state: str
|
||||
description: str | None = None
|
||||
categories: Any = None
|
||||
confidence: str | None = None
|
||||
cover_uuid: str | None = None
|
||||
prio_group: str | None = None
|
||||
anchor_next_uuid: str | None = None
|
||||
anchor_prev_uuid: str | None = None
|
||||
source_item_id: str | None = None
|
||||
restored_uuid: str | None = None
|
||||
changed_at: str | None = None
|
||||
message: str | None = None
|
||||
has_snapshot: bool = False
|
||||
# A deleted rule with a stored snapshot that has not been restored yet (POST items/{id}/restore, admin).
|
||||
restorable: bool = False
|
||||
|
||||
|
||||
class CleanupOperationRead(BaseModel):
|
||||
id: str
|
||||
host_id: str
|
||||
host_name: str | None = None
|
||||
cluster_name: str | None = None
|
||||
kind: str # disable | enable | delete | restore
|
||||
# preview, running, awaiting_confirm, confirmed, partial, apply_failed, reverting, reverted, revert_failed, done,
|
||||
# failed, cancelled, expired
|
||||
status: str
|
||||
actor: str
|
||||
created_at: str | None = None
|
||||
expires_at: str | None = None
|
||||
started_at: str | None = None
|
||||
heartbeat_at: str | None = None
|
||||
revert_deadline: str | None = None
|
||||
confirmed_at: str | None = None
|
||||
finished_at: str | None = None
|
||||
message: str | None = None
|
||||
options: dict[str, Any] = Field(default_factory=dict)
|
||||
# Only while status == "preview".
|
||||
plan: dict[str, Any] | None = None
|
||||
items: list[CleanupItemRead] = Field(default_factory=list)
|
||||
can_confirm: bool = False
|
||||
can_revert: bool = False
|
||||
can_retry: bool = False
|
||||
# ErrorCode names, e.g. BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR (restore only).
|
||||
warnings: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class CleanupPreviewRead(BaseModel):
|
||||
preview_id: str
|
||||
expires_at: str | None = None
|
||||
kind: str
|
||||
plan: dict[str, Any]
|
||||
max_rules: int
|
||||
window_seconds: int
|
||||
operation: CleanupOperationRead
|
||||
@@ -130,6 +130,9 @@ async def sweep_expired_alias_deployments(db: AsyncSession) -> list[AliasDeploym
|
||||
|
||||
|
||||
async def undeploy_alias(db: AsyncSession, alias: FirewallAlias) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)
|
||||
failures = []
|
||||
result = await db.execute(select(AliasDeployment).where(AliasDeployment.alias_id == alias.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -39,12 +39,15 @@ _LAST_USED_GRANULARITY = timedelta(minutes=1)
|
||||
|
||||
_SAFE_METHODS = ("GET", "HEAD", "OPTIONS")
|
||||
# GUI tickets open a firewall's web interface in the browser: an interactive admin session only.
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$")
|
||||
# Rule cleanup (1.31): "I still have access" is a human decision, so confirming a cleanup is never possible with a
|
||||
# token, whatever its scope (a script must not be able to confirm blindly).
|
||||
_SESSION_ONLY = re.compile(r"^/api/(tokens|auth)(/|$)|^/api/hosts/[^/]+/gui-ticket$|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$")
|
||||
# Reads that expose secrets (config.xml holds password hashes and VPN keys, the
|
||||
# notification settings the webhook URL/SMTP data): admin scope only.
|
||||
_ADMIN_SCOPE_READS = re.compile(r"^/api/hosts/[^/]+/backups/[^/]+$|^/api/notifications(/|$)|^/api/settings/|^/api/overlay/peers/[^/]+/instructions$")
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read.
|
||||
_MCP_PATH = "/api/mcp"
|
||||
# The MCP endpoint is POST (JSON-RPC) but its tools only ever read; so does the configuration-as-code plan (1.32):
|
||||
# it compares a posted file with the database and writes nothing.
|
||||
_READ_ONLY_POSTS = ("/api/mcp", "/api/config/import/plan")
|
||||
_ADMIN_ONLY_WRITES = re.compile(
|
||||
r"^/api/(users|settings|system|notifications|overlay)(/|$)"
|
||||
r"|^/api/logs/settings$"
|
||||
@@ -52,9 +55,18 @@ _ADMIN_ONLY_WRITES = re.compile(
|
||||
r"|^/api/hosts/[^/]+/gui-ticket$"
|
||||
# The per-gateway alert switch is a notification setting (same rights as PUT /api/notifications/settings).
|
||||
r"|^/api/hosts/[^/]+/gateways/[^/]+/alert$"
|
||||
# Configuration as code (1.32): a bulk change incl. prune and host hierarchy membership (host writes are admin-only).
|
||||
r"|^/api/config/import/apply$"
|
||||
)
|
||||
|
||||
|
||||
# Rule cleanup: executing a cleanup, retrying a failed revert/reconcile and restoring a deleted rule are admin-only.
|
||||
# operators_may_deploy defaults to True, so _ROLLOUT_POST alone cannot express "admin only"; this pattern is checked
|
||||
# for operator accounts (operator_denies) and for tokens below scope admin (denies). Not part of _ADMIN_ONLY_WRITES:
|
||||
# that one means "who may access OPNexus / where data goes". The router checks the same rule once more.
|
||||
_ADMIN_ONLY_CLEANUP = re.compile(r"^/api/hosts/[^/]+/rule-cleanup/([^/]+/(deploy|retry)|items/[^/]+/restore)/?$")
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
@@ -67,13 +79,13 @@ def denies(scope: str, method: str, path: str) -> bool:
|
||||
"""True if a token with this scope must not make this request."""
|
||||
if _SESSION_ONLY.match(path):
|
||||
return True
|
||||
if path == _MCP_PATH:
|
||||
if path in _READ_ONLY_POSTS:
|
||||
return False
|
||||
if method in _SAFE_METHODS:
|
||||
return scope != "admin" and bool(_ADMIN_SCOPE_READS.match(path))
|
||||
if scope == "read":
|
||||
return True
|
||||
if scope == "deploy" and _ADMIN_ONLY_WRITES.match(path):
|
||||
if scope == "deploy" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -82,7 +94,8 @@ def operator_denies(method: str, path: str) -> bool:
|
||||
"""True if an `operator` account must not make this request: the writes that
|
||||
change who may access OPNexus or where it sends data. (Their own theme at
|
||||
PUT /api/settings stays open; account endpoints are not matched here.)"""
|
||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(_ADMIN_ONLY_WRITES.match(path))
|
||||
return method not in _SAFE_METHODS and path != "/api/settings" and bool(
|
||||
_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))
|
||||
|
||||
|
||||
# Writes that put a change onto the fleet (as opposed to preparing one): deploy, remove/rollback a deployment,
|
||||
@@ -92,6 +105,9 @@ def operator_denies(method: str, path: str) -> bool:
|
||||
_ROLLOUT_POST = re.compile(
|
||||
r"/(deploy|push|sign|poll|register|reassert-ordering"
|
||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||
# Rule cleanup (1.31): switching rules off (/deploy matches above) and taking it back. Confirming and
|
||||
# previewing stay open, like confirm_rule.
|
||||
r"|rule-cleanup/[^/]+/(revert|retry)"
|
||||
r"|maintenance/(execute|carp-execute|schedule)"
|
||||
r"|network/changes"
|
||||
r"|services/[^/]+/(start|stop|restart)"
|
||||
|
||||
@@ -13,6 +13,9 @@ from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
|
||||
context = ContextVar("audit_context", default=None)
|
||||
# A key in ``details`` whose content only admins (session or admin-scope token) get to read (routers/audit.py):
|
||||
# firewall addresses and management sources of the rule cleanup.
|
||||
ADMIN_ONLY_DETAILS = "admin_only"
|
||||
|
||||
|
||||
# Column widths of audit_events. entity_id/host_id/action are taken from the
|
||||
|
||||
@@ -34,7 +34,27 @@ async def peers(db, host):
|
||||
return [host]
|
||||
|
||||
|
||||
async def ensure_no_maintenance(db, hosts):
|
||||
async def ensure_no_open_cleanup(db, hosts):
|
||||
"""A rule cleanup (1.31) that is running, waiting for its confirmation or unresolved owns the host and its
|
||||
cluster peers: nothing else may write to the firewall until it is confirmed, reverted or settled."""
|
||||
from app.services.rule_cleanup_service import has_open_cleanup
|
||||
ids = set()
|
||||
for host in hosts:
|
||||
ids.update(h.id for h in await peers(db, host))
|
||||
if ids and await has_open_cleanup(db, ids):
|
||||
raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)
|
||||
|
||||
|
||||
async def ensure_no_open_cleanup_on(db, model, condition):
|
||||
"""ensure_no_open_cleanup for every host that has a deployment row of ``model`` matching ``condition`` (the
|
||||
undeploy paths: deleting a central object removes it from all its firewalls)."""
|
||||
ids = set((await db.scalars(select(model.host_id).where(condition))).all())
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, i) for i in ids] if h is not None]
|
||||
await ensure_no_open_cleanup(db, hosts)
|
||||
|
||||
|
||||
async def ensure_no_maintenance(db, hosts, *, cleanup=True):
|
||||
"""cleanup=False: only the cleanup service itself, which reports an open cleanup with its own error code."""
|
||||
ids = set()
|
||||
for host in hosts:
|
||||
ids.update(h.id for h in await peers(db, host))
|
||||
@@ -42,6 +62,8 @@ async def ensure_no_maintenance(db, hosts):
|
||||
raise api_error(409, ErrorCode.OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY)
|
||||
if ids and await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||
raise api_error(409, ErrorCode.OPEN_RESTORE_BLOCKS_DEPLOY)
|
||||
if cleanup and ids:
|
||||
await ensure_no_open_cleanup(db, hosts)
|
||||
|
||||
|
||||
async def _shared_change_blockers(db, fleet):
|
||||
@@ -56,6 +78,9 @@ async def _shared_change_blockers(db, fleet):
|
||||
blocked.append("Eine frühere Wartungsaktion im Host/Cluster muss zuerst geprüft und abgeschlossen werden.")
|
||||
if await db.scalar(select(RestoreOperation.id).where(RestoreOperation.host_id.in_(fleet_ids), RestoreOperation.status.in_(RESTORE_OPEN)).limit(1)):
|
||||
blocked.append("Eine Wiederherstellung im Host/Cluster ist offen und muss zuerst bestätigt oder abgebrochen werden.")
|
||||
from app.services.rule_cleanup_service import has_open_cleanup
|
||||
if await has_open_cleanup(db, fleet_ids):
|
||||
blocked.append("Eine Regel-Bereinigung im Host/Cluster ist offen und muss zuerst bestätigt oder zurückgenommen werden.")
|
||||
return blocked
|
||||
|
||||
|
||||
|
||||
@@ -303,6 +303,9 @@ async def undeploy_tunnel(db: AsyncSession, tunnel: IpsecTunnel) -> None:
|
||||
for this entity to gate on) — deletes Child/Local/Remote before their
|
||||
parent Connection (the ModelRelationField dependency direction), then
|
||||
the PreSharedKey (independent of the other 4), then reconfigures."""
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, IpsecTunnelDeployment, IpsecTunnelDeployment.tunnel_id == tunnel.id)
|
||||
failures = []
|
||||
result = await db.execute(select(IpsecTunnelDeployment).where(IpsecTunnelDeployment.tunnel_id == tunnel.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -201,6 +201,62 @@ async def check_revert_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"host_name": host.name if host else None,
|
||||
"message": deployment.message,
|
||||
}
|
||||
problems.update(await _cleanup_problems(db))
|
||||
return problems
|
||||
|
||||
|
||||
# Rule cleanup (1.31) states that need a human: the automatic revert failed, or a run ended half-way.
|
||||
_CLEANUP_STILL_CHANGED = ("revert_failed", "unknown", "disabling", "disabled", "failed")
|
||||
|
||||
|
||||
async def _cleanup_problems(db: AsyncSession) -> dict[str, dict]:
|
||||
"""Rule cleanups whose automatic revert failed (per rule that could not be switched back) and runs stuck in
|
||||
``partial``/``apply_failed`` (a decision is needed before the window ends), plus delete/restore runs with an
|
||||
unresolved item. Same ``revert_failed`` alert as the deployments. Keys: ``rule_cleanup:<item id>`` for one rule,
|
||||
``rule_cleanup_op:<operation id>`` for a whole run, so one open problem is one message."""
|
||||
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
|
||||
|
||||
problems: dict[str, dict] = {}
|
||||
operations = (await db.scalars(select(RuleCleanupOperation).where(
|
||||
RuleCleanupOperation.status.in_(("revert_failed", "partial", "apply_failed", "failed"))))).all()
|
||||
for op in operations:
|
||||
items = list((await db.scalars(select(RuleCleanupItem).where(RuleCleanupItem.operation_id == op.id)
|
||||
.order_by(RuleCleanupItem.seq))).all())
|
||||
host = await db.get(OpnsenseHost, op.host_id)
|
||||
base = {
|
||||
"deployment_type": "rule_cleanup", "deployment_id": str(op.id), "host_id": str(op.host_id),
|
||||
"host_name": host.name if host else None, "operation": op.kind, "operation_status": op.status,
|
||||
}
|
||||
if op.status == "revert_failed":
|
||||
stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]
|
||||
for item in stuck:
|
||||
member = await db.get(OpnsenseHost, item.host_id)
|
||||
problems[f"rule_cleanup:{item.id}"] = {
|
||||
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||
"message": item.message or op.message,
|
||||
"hinweis": "Bereinigung: Die Regel konnte nicht automatisch zurückgesetzt werden und ist möglicherweise "
|
||||
"noch geändert. Auf der Firewall prüfen und in OPNexus „Erneut zurücknehmen“ wählen.",
|
||||
}
|
||||
if not stuck:
|
||||
problems[f"rule_cleanup_op:{op.id}"] = {**base, "entity_id": str(op.id), "message": op.message,
|
||||
"hinweis": "Bereinigung: Das automatische Zurücknehmen ist fehlgeschlagen. Zustand der Firewall prüfen."}
|
||||
elif op.status in ("partial", "apply_failed"):
|
||||
problems[f"rule_cleanup_op:{op.id}"] = {
|
||||
**base, "entity_id": str(op.id), "message": op.message,
|
||||
"regeln": ", ".join(i.rule_uuid for i in items if i.state in ("failed", "unknown")) or None,
|
||||
"hinweis": "Bereinigung nur teilweise ausgeführt (oder das Anwenden ist fehlgeschlagen). Bestätigen ist gesperrt; "
|
||||
"jetzt zurücknehmen oder die automatische Rücknahme am Fristende abwarten.",
|
||||
}
|
||||
elif op.kind in ("delete", "restore"):
|
||||
for item in (i for i in items if i.state == "unknown"):
|
||||
member = await db.get(OpnsenseHost, item.host_id)
|
||||
problems[f"rule_cleanup:{item.id}"] = {
|
||||
**base, "entity_id": item.rule_uuid, "host_id": str(item.host_id),
|
||||
"host_name": member.name if member else None, "rule": item.description, "item_state": item.state,
|
||||
"message": item.message or op.message,
|
||||
"hinweis": "Bereinigung: Der Ausgang beim Löschen/Wiederherstellen ist unklar. Mit „Erneut prüfen“ nachlesen.",
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
|
||||
@@ -135,6 +135,9 @@ async def sweep_expired_one_to_one_deployments(db: AsyncSession) -> list[OneToOn
|
||||
|
||||
|
||||
async def undeploy_one_to_one(db: AsyncSession, rule: OneToOneRule) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, OneToOneDeployment, OneToOneDeployment.one_to_one_id == rule.id)
|
||||
failures = []
|
||||
result = await db.execute(select(OneToOneDeployment).where(OneToOneDeployment.one_to_one_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -371,6 +371,9 @@ async def undeploy_instance(db: AsyncSession, instance: OpenVpnInstance) -> None
|
||||
for this entity to gate on) — deletes the Instance before its
|
||||
referenced StaticKey (the ModelRelationField dependency direction),
|
||||
then reconfigures."""
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, OpenVpnInstanceDeployment, OpenVpnInstanceDeployment.instance_id == instance.id)
|
||||
failures = []
|
||||
result = await db.execute(select(OpenVpnInstanceDeployment).where(OpenVpnInstanceDeployment.instance_id == instance.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -10,6 +10,7 @@ Auth is HTTP Basic: username = API key, password = API secret.
|
||||
"""
|
||||
import logging
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
@@ -33,6 +34,11 @@ class OpnsenseAmbiguousError(OpnsenseAPIError):
|
||||
outcome of a write is unknown and must be reconciled, not retried blindly."""
|
||||
|
||||
|
||||
class OpnsenseNotFoundError(OpnsenseAPIError):
|
||||
"""The firewall answered, but there is no such object (an empty `getRule` answer: unknown UUID, or a rule that is
|
||||
not part of the rules model - legacy or automatic). A refusal, never an unknown outcome."""
|
||||
|
||||
|
||||
class OpnsenseForbiddenError(OpnsenseAPIError):
|
||||
"""The firewall answered 403: the API key lacks the privilege for this endpoint. A refusal, not an unknown
|
||||
outcome; callers that only read can tell "no rights" from "unreachable". 401 (wrong key or secret) stays a
|
||||
@@ -241,6 +247,51 @@ def _mutation_result(data: Any, operation: str, results: set[str]) -> dict:
|
||||
return data
|
||||
|
||||
|
||||
# -- Filter rules (FilterController -> ApiMutableModelControllerBase, rules.rule) --------------------------------
|
||||
# Evidence for add/set/get/del/toggle/moveRuleBefore, checked 2026-10-22 against BOTH
|
||||
# source opnsense/core tags 26.7.7, 26.1.9 and 25.7.11, src/opnsense/mvc/app/controllers/OPNsense/:
|
||||
# Firewall/Api/FilterController.php (the *RuleAction methods, moveRuleBeforeAction, ittrRules),
|
||||
# Firewall/Api/FilterBaseController.php (applyAction),
|
||||
# Base/ApiMutableModelControllerBase.php (addBase, setBase, getBase, delBase, toggleBase)
|
||||
# live OPNsense 26.7.7 on two lab test firewalls (scratch rules whose description began with
|
||||
# `opnexus-b4-probe`, deleted again and applied afterwards; nothing was left behind).
|
||||
# Same in all three source lines unless said otherwise; the live checks cover 26.7.7 only.
|
||||
#
|
||||
# addRule POST, body {"rule": {...}}. Answer {"result": "saved", "uuid": "<new>"}; a validation failure is HTTP 200
|
||||
# {"result": "failed", "validations": {"rule.<field>": "<text>"}} (so _mutation_result raises).
|
||||
# `sequence` is a FilterSequenceField (models/OPNsense/Firewall/Filter.xml: 1..999999, default 1;
|
||||
# `sort_order` and `prio_group` are volatile, derived). Live: without `sequence` the rule gets the
|
||||
# highest sequence of ALL rules + 100 (566, 666, 766, ...), so a copy always lands at the END of its
|
||||
# prio_group; an explicit `sequence` IS honoured (sequence "3" -> sort_order 400000.0000003) and is not
|
||||
# checked for uniqueness. A restore can therefore either send the old number or call moveRuleBefore.
|
||||
# setRule POST /{uuid}, body {"rule": {...}}; answer {"result": "saved"}. It is an upsert (a well-formed unknown
|
||||
# UUID creates the rule) - never call it for a rule that may be gone.
|
||||
# delRule POST /{uuid}. {"result": "deleted"} or {"result": "not found"} (unknown, legacy and automatic UUIDs all
|
||||
# answer "not found", HTTP 200, nothing changes). 26.1 and 26.7 accept a comma-separated list; 25.7 a
|
||||
# single UUID. Consequence: "not found" proves nothing about a rule being gone - re-read it.
|
||||
# getRule GET /{uuid} -> {"rule": {...}} (every field of the rules model, select fields as
|
||||
# {"<option>": {"value": "<label>", "selected": 0|1}}, `categories` a select dict - or [] when the rule has
|
||||
# none -; no `uuid` key; read-only: `sequence`, `sort_order`, `prio_group` and the nested `audit`).
|
||||
# Live: an unknown, legacy (hex label) or automatic UUID answers HTTP 200 with a bare `[]`. 25.7 and 26.1
|
||||
# also know `?fetchmode=copy` (pre-fills `sequence`); 26.7.7 no longer has it - not used here.
|
||||
# Without a UUID it returns the template of a new rule (see get_rule_template above).
|
||||
# toggleRule POST /{uuid}/{enabled} - `enabled` is "0" or "1" and EXPLICIT; without it the rule flips (never do that).
|
||||
# Answer {"result": "Enabled"|"Disabled", "changed": true|false}. The explicit form is idempotent: repeating
|
||||
# it answers the same result with "changed": false (live: 0,0 -> true,false; 1,1 -> true,false). Any other
|
||||
# value answers {"result": "failed", "changed": false}; an unknown or legacy UUID, a GET, or a model that
|
||||
# has no such node answers {"result": "failed"} without "changed" - all HTTP 200. It only edits the
|
||||
# configuration; nothing reaches pf before apply() (the rule keeps its sort_order, searchRule shows
|
||||
# enabled "0"/"1" at once). 26.1/26.7 take a list of UUIDs (explicit value only); 25.7 one UUID.
|
||||
# moveRuleBefore POST /{selected}/{target}. {"status": "ok"} (no "result" key). Both rules must be in the same
|
||||
# `prio_group` - the sort_order prefix - which is NOT the interface: every single-interface rule of every
|
||||
# interface shares 400000 (lan and wan rules can be moved before each other), floating rules (several or
|
||||
# no interface) are 200000, interface-group rules 3000xx. Live: unknown UUID, same UUID twice and
|
||||
# different groups all answer HTTP 500 {"errorMessage": "Unexpected error, check log for details",
|
||||
# "errorTitle": "Filter"}, a GET answers 200 {"status": "error"}. A 500 is Ambiguous in _do_request, so a
|
||||
# caller that wants to tell "refused" from "unknown" must read sort_order back. The move only touches
|
||||
# `sequence` of the moved rule (and, when the gap is too small, of the rules behind the target); legacy
|
||||
# rules are not in the model and cannot be an anchor.
|
||||
# apply POST. {"status": "OK\n\n"} after `filter reload skip_alias`. FilterBaseController has no savepoint here.
|
||||
async def add_rule(host: OpnsenseHost, payload: dict) -> dict:
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/addRule", json={"rule": payload}), "Regel anlegen", {"saved"})
|
||||
|
||||
@@ -253,18 +304,93 @@ async def del_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||
return _mutation_result(await _request(host, "POST", f"/firewall/filter/delRule/{rule_uuid}"), "Regel entfernen", {"deleted", "not found"})
|
||||
|
||||
|
||||
_RULE_REF = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||
|
||||
|
||||
def _rule_ref(rule_uuid: str) -> str:
|
||||
"""A rule reference ends up in the URL path: refuse anything that could leave it."""
|
||||
if not isinstance(rule_uuid, str) or not _RULE_REF.match(rule_uuid):
|
||||
raise OpnsenseAPIError("Ungültige Regel-UUID")
|
||||
return rule_uuid
|
||||
|
||||
|
||||
async def get_rule(host: OpnsenseHost, rule_uuid: str) -> dict:
|
||||
"""The full model record of one rule (`getRule/{uuid}`, the inner `rule` dict, select fields still in
|
||||
OPNsense's option form - reduce them with rule_write_payload()). A rule that is not in the rules model (unknown,
|
||||
legacy hex label, automatic) is answered with `[]`: raises OpnsenseNotFoundError."""
|
||||
data = await _request(host, "GET", f"/firewall/filter/getRule/{_rule_ref(rule_uuid)}")
|
||||
rule = data.get("rule") if isinstance(data, dict) else None
|
||||
if isinstance(rule, dict) and rule:
|
||||
return rule
|
||||
if data == [] or data == {}:
|
||||
raise OpnsenseNotFoundError(f"Regel {rule_uuid} gibt es auf {host.name} nicht (oder sie ist keine bearbeitbare Regel)")
|
||||
raise OpnsenseAPIError(f"Regel lesen: Keine auswertbare Antwort von {host.name}")
|
||||
|
||||
|
||||
async def toggle_rule(host: OpnsenseHost, rule_uuid: str, enabled: bool) -> dict:
|
||||
"""Sets the rule's `enabled` flag to exactly `enabled` (`toggleRule/{uuid}/1|0`; never the flipping form) and
|
||||
returns OPNsense's answer, e.g. {"result": "Disabled", "changed": False}. Repeating it is harmless. Only the
|
||||
configuration changes; call apply() afterwards. A refusal (unknown or legacy UUID) is a plain OpnsenseAPIError,
|
||||
a lost answer an OpnsenseAmbiguousError - then read the rule back before doing anything else."""
|
||||
if not isinstance(enabled, bool):
|
||||
raise TypeError("toggle_rule: 'enabled' must be a bool (an implicit flip would not be idempotent)")
|
||||
data = await _request(host, "POST", f"/firewall/filter/toggleRule/{_rule_ref(rule_uuid)}/{1 if enabled else 0}")
|
||||
return _mutation_result(data, "Regel ein-/ausschalten", {"enabled" if enabled else "disabled"})
|
||||
|
||||
|
||||
# Fields of a getRule record that a write must not send: the UUID is the rule's identity, `sequence` is assigned
|
||||
# on creation (sort_order and prio_group are derived from it), `audit` is metadata OPNsense maintains itself.
|
||||
RULE_READONLY_FIELDS = frozenset({"uuid", "sequence", "sort_order", "prio_group", "audit"})
|
||||
|
||||
|
||||
def rule_write_payload(rule: dict) -> dict[str, str]:
|
||||
"""Turns a getRule record into the flat form addRule/setRule accept: select fields become their selected
|
||||
option(s), comma-separated (`lan,wan`; none selected -> ""), the empty-list form of an empty multi-select
|
||||
becomes "", read-only fields are left out. Live round trip on 26.7.7 (getRule -> this -> addRule -> getRule)
|
||||
gave an identical record except uuid/sequence/sort_order/audit for single-interface, multi-interface (floating),
|
||||
no-interface (floating), interface-group, ICMP and any-protocol rules with categories, aliases, port ranges,
|
||||
TCP flags, tags and state limits."""
|
||||
out: dict[str, str] = {}
|
||||
for key, value in rule.items():
|
||||
if key in RULE_READONLY_FIELDS:
|
||||
continue
|
||||
if isinstance(value, dict):
|
||||
if value and all(isinstance(o, dict) and "selected" in o for o in value.values()):
|
||||
out[key] = ",".join(k for k, o in value.items() if o.get("selected") and k != "")
|
||||
# any other nested dict is metadata and is not sent
|
||||
elif isinstance(value, list):
|
||||
out[key] = ",".join(str(v) for v in value)
|
||||
else:
|
||||
out[key] = "" if value is None else str(value)
|
||||
return out
|
||||
|
||||
|
||||
async def apply(host: OpnsenseHost) -> dict:
|
||||
return _mutation_result(await _request(host, "POST", "/firewall/filter/apply"), "Änderungen anwenden", set())
|
||||
|
||||
|
||||
async def get_hasync(host: OpnsenseHost) -> dict:
|
||||
"""System > High Availability settings, read-only: `core/hasync/get` (Core/Api/HasyncController extends
|
||||
ApiMutableModelControllerBase, model Core/Hasync.xml; identical in opnsense/core 25.7.11, 26.1.9 and 26.7.7).
|
||||
Answer {"hasync": {"pfsyncinterface": {"": {"value": "Disabled", ...}, "lan": {"value": "LAN", "selected": 1},
|
||||
...}, "pfsyncpeerip": "<address or empty>", "synchronizetoip": ..., "syncitems": {...}}} - `pfsyncinterface` is an
|
||||
InterfaceField (the selected option, "" = disabled), `pfsyncpeerip` a NetworkField without mask. Live on 26.7.7
|
||||
(both nodes of the lab cluster): pfsyncinterface `lan`, a peer address, synchronizetoip empty. Used by the rule
|
||||
cleanup to keep pfsync rules out of reach (rule_analysis/protection.py); it reads nothing secret (the sync
|
||||
password field comes back, but is never stored or logged by the caller)."""
|
||||
return await _request(host, "GET", "/core/hasync/get")
|
||||
|
||||
|
||||
async def move_rule_before(host: OpnsenseHost, selected_uuid: str, target_uuid: str) -> dict:
|
||||
"""`firewall/filter/moveRuleBefore/{selected_uuid}/{target_uuid}`
|
||||
(FilterController::moveRuleBeforeAction) — repositions `selected_uuid`
|
||||
to sit directly before `target_uuid` via gap-numbered `sequence`
|
||||
floats, scoped to rules sharing the same `prio_group` (interface/type).
|
||||
Verified against OPNsense core source: no "move to top/bottom"
|
||||
sentinel exists, and both UUIDs must resolve to existing rules on the
|
||||
SAME interface/type or the controller throws. On success it returns
|
||||
floats, scoped to rules sharing the same `prio_group` (the sort_order
|
||||
prefix, see the evidence block above - not the interface). Verified against OPNsense core source and live
|
||||
(26.7.7): no "move to top/bottom"
|
||||
sentinel exists, and both UUIDs must resolve to existing rules in the
|
||||
SAME prio_group or the controller throws (HTTP 500, which this client
|
||||
reports as OpnsenseAmbiguousError). On success it returns
|
||||
`{"status": "ok"}` (confirmed against FilterController.php directly —
|
||||
unlike addRule/setRule/delRule this has no "result" key at all), so
|
||||
this is dispatched through _mutation_result with an empty `results`
|
||||
@@ -1328,6 +1454,13 @@ async def create_snapshot(host: OpnsenseHost, name: str, note: str = "") -> None
|
||||
raise OpnsenseAPIError(f"Snapshot anlegen von Firewall nicht bestätigt: {redact_data(data)}")
|
||||
|
||||
|
||||
async def activate_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||
"""Boot environment for every following start (bectl activate, not -t); takes effect at the next reboot."""
|
||||
data = await _request(host, "POST", f"/core/snapshots/activate/{uuid}")
|
||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||
raise OpnsenseAPIError(f"Snapshot aktivieren von Firewall nicht bestätigt: {redact_data(data)}")
|
||||
|
||||
|
||||
async def delete_snapshot(host: OpnsenseHost, uuid: str) -> None:
|
||||
data = await _request(host, "POST", f"/core/snapshots/del/{uuid}")
|
||||
if isinstance(data, dict) and (data.get("validations") or str(data.get("status", "")).lower() == "failed"):
|
||||
|
||||
@@ -147,6 +147,9 @@ async def sweep_expired_outbound_nat_deployments(db: AsyncSession) -> list[Outbo
|
||||
|
||||
|
||||
async def undeploy_outbound_nat(db: AsyncSession, rule: OutboundNatRule) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, OutboundNatDeployment, OutboundNatDeployment.outbound_nat_id == rule.id)
|
||||
failures = []
|
||||
result = await db.execute(select(OutboundNatDeployment).where(OutboundNatDeployment.outbound_nat_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -185,6 +185,9 @@ async def reassert_group_ordering(db: AsyncSession, group: PolicyHierarchyGroup)
|
||||
for h in g.hosts:
|
||||
hosts[h.id] = h
|
||||
errors: list[str] = []
|
||||
# moveRuleBefore changes rule positions: not while a rule cleanup (1.31) owns one of the firewalls.
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
await ensure_no_open_cleanup(db, [h for h in hosts.values() if h.enabled])
|
||||
for host in sorted(hosts.values(), key=lambda h: (h.name, str(h.id))):
|
||||
if not host.enabled:
|
||||
continue
|
||||
@@ -300,9 +303,13 @@ async def sweep_expired_hierarchy_deployments(db: AsyncSession) -> list[PolicyHi
|
||||
|
||||
|
||||
async def undeploy_hierarchy_rule(db: AsyncSession, rule: PolicyHierarchyRule) -> None:
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
failures = []
|
||||
result = await db.execute(select(PolicyHierarchyRuleDeployment).where(PolicyHierarchyRuleDeployment.policy_hierarchy_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
deployments = list(result.scalars().all())
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them.
|
||||
await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])
|
||||
for deployment in deployments:
|
||||
host = await db.get(OpnsenseHost, deployment.host_id)
|
||||
if host and deployment.opnsense_policy_hierarchy_uuid:
|
||||
try:
|
||||
|
||||
@@ -147,6 +147,9 @@ async def sweep_expired_port_forward_deployments(db: AsyncSession) -> list[PortF
|
||||
|
||||
|
||||
async def undeploy_port_forward(db: AsyncSession, rule: PortForwardRule) -> None:
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, PortForwardDeployment, PortForwardDeployment.port_forward_id == rule.id)
|
||||
failures = []
|
||||
result = await db.execute(select(PortForwardDeployment).where(PortForwardDeployment.port_forward_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -196,6 +196,7 @@ async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, sta
|
||||
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
||||
"window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"gate_probes": list(plan.gate_probes or []),
|
||||
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
||||
}
|
||||
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
||||
@@ -381,7 +382,8 @@ async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None
|
||||
baselines.update((w.baseline or {}))
|
||||
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||
probes=run.plan_snapshot.get("gate_probes"))
|
||||
|
||||
async def apply(db, run):
|
||||
if gate["verdict"] == "fail":
|
||||
@@ -537,7 +539,7 @@ async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | Non
|
||||
versions = {}
|
||||
for rh_id in offers:
|
||||
host = usable[rh_id]
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host)
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host, run.plan_snapshot.get("gate_probes"))
|
||||
versions[rh_id] = baselines[str(host.id)].get("version")
|
||||
roles = await get_statuses([usable[i] for i in offers])
|
||||
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
||||
@@ -630,6 +632,7 @@ async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision |
|
||||
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
||||
gate = await rollout_health.evaluate(
|
||||
db, done_peers + [host], baselines, target_version=run.target_version,
|
||||
# No probes here: the peers' probes were checked in their own gate, and this runs on every tick.
|
||||
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
||||
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
||||
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
||||
@@ -718,7 +721,8 @@ async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"),
|
||||
probes=run.plan_snapshot.get("gate_probes"))
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, index)
|
||||
|
||||
@@ -18,13 +18,19 @@ the configuration like the 6-hourly security check does):
|
||||
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
||||
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
||||
update may legitimately change defaults.
|
||||
* probes — optional per plan (1.32): ping/TCP probes run FROM the firewall (uplink, DNS, an internal
|
||||
server). A probe that worked in the baseline and fails now fails the gate; one that failed
|
||||
already before only warns. This is reachability from the firewall, not traffic through it.
|
||||
|
||||
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
||||
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
||||
"""
|
||||
import asyncio
|
||||
import uuid
|
||||
from collections import defaultdict
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -41,6 +47,8 @@ DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": Tru
|
||||
"compliance": False}
|
||||
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
||||
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
||||
MAX_PROBES = 10
|
||||
PROBE_PING_COUNT = 3
|
||||
|
||||
|
||||
def _r(result: str, code: str, **params) -> dict:
|
||||
@@ -64,7 +72,55 @@ async def _services(host) -> dict | None:
|
||||
return None if listing is None else _services_compact(listing)
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
def _probe_key(probe: dict) -> str:
|
||||
return f"{probe['type']} {probe['target']}" + (f":{probe['port']}" if probe.get("port") else "")
|
||||
|
||||
|
||||
async def run_probes(host: OpnsenseHost, probes: list[dict] | None) -> list[dict]:
|
||||
"""Every probe from the firewall, one after the other; a probe that cannot run counts as failed. Audited by
|
||||
diagnostics_service like a probe from the host page, as `system:rollout` when no request context exists."""
|
||||
from app.services import audit_service, diagnostics_service
|
||||
|
||||
if not probes:
|
||||
return []
|
||||
token = None
|
||||
if audit_service.context.get() is None:
|
||||
token = audit_service.context.set({"actor": "system:rollout", "id": str(uuid.uuid4()), "action": "probe"})
|
||||
try:
|
||||
async def one(probe: dict) -> dict:
|
||||
ipproto = "inet6" if ":" in probe["target"] else "inet"
|
||||
try:
|
||||
if probe["type"] == "ping":
|
||||
result = await diagnostics_service.ping(host, probe["target"], ipproto, PROBE_PING_COUNT, None, None)
|
||||
ok = int(result.get("received") or 0) > 0
|
||||
else:
|
||||
result = await diagnostics_service.portprobe(host, probe["target"], int(probe["port"]), ipproto, None)
|
||||
ok = bool(result.get("open"))
|
||||
except (HTTPException, client.OpnsenseAPIError):
|
||||
ok = False
|
||||
return {"key": _probe_key(probe), "ok": ok}
|
||||
|
||||
# Concurrently (diagnostics_service caps it at a few at a time per process), results in plan order.
|
||||
return list(await asyncio.gather(*(one(p) for p in probes[:MAX_PROBES])))
|
||||
finally:
|
||||
if token is not None:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _check_probes(before: list | None, now: list | None) -> dict:
|
||||
if now is None:
|
||||
return _r("fail", "unreadable")
|
||||
was = {p["key"]: p["ok"] for p in (before or [])}
|
||||
regressed = [p["key"] for p in now if not p["ok"] and was.get(p["key"], True)]
|
||||
still = [p["key"] for p in now if not p["ok"] and was.get(p["key"]) is False]
|
||||
if regressed:
|
||||
return _r("fail", "probe_regressed", probes=", ".join(regressed))
|
||||
if still:
|
||||
return _r("warn", "probe_was_failing", probes=", ".join(still))
|
||||
return _r("ok", "ok", count=len(now))
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost, probes: list[dict] | None = None) -> dict:
|
||||
info, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
||||
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
||||
@@ -73,6 +129,7 @@ async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
"gateways": gateways,
|
||||
"services": services,
|
||||
"compliance_score": score,
|
||||
"probes": await run_probes(host, probes) if probes else None,
|
||||
}
|
||||
|
||||
|
||||
@@ -137,7 +194,7 @@ async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dic
|
||||
|
||||
|
||||
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
||||
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
|
||||
checks: dict | None = None, compliance_mode: str = "warn", probes: list[dict] | None = None) -> dict:
|
||||
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
||||
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
||||
for name in ALWAYS_ON:
|
||||
@@ -190,6 +247,10 @@ async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict,
|
||||
if enabled["services"]:
|
||||
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
||||
|
||||
if probes:
|
||||
# A probe new since the baseline (plan edited mid-run) has nothing to compare with: treated as "worked before".
|
||||
row["probes"] = _check_probes(before.get("probes"), await run_probes(host, probes) if reachable else None)
|
||||
|
||||
blockers = await _shared_change_blockers(db, fleets[hid])
|
||||
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
||||
|
||||
|
||||
@@ -110,6 +110,8 @@ def definition(plan: RolloutPlan) -> dict:
|
||||
return {
|
||||
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
# Only when set: an activated plan from before 1.32 must keep its fingerprint (else its automatic start stops).
|
||||
**({"gate_probes": list(plan.gate_probes)} if plan.gate_probes else {}),
|
||||
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
||||
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
||||
}
|
||||
|
||||
@@ -15,7 +15,9 @@ Confidence (never higher than what is proven; what is not proven is never "sure"
|
||||
sampled recently and over the whole window; shadowed/duplicate = literal comparison, order from
|
||||
``sort_order``, the covering rule is loaded; disabled_long = observed unchanged for N days.
|
||||
* ``probable`` relies on a guess or on static alias content, or pf's loaded ruleset is not available, or the rule
|
||||
has a schedule (a hit might happen at another time).
|
||||
has a schedule (a hit might happen at another time), or (cluster) some node has no usage data for
|
||||
the rule. In a cluster "unused" is judged over all nodes (rule_hit_service.merge_cluster_hits):
|
||||
the CARP backup sees almost no traffic, so its zero alone never makes a rule unused.
|
||||
* ``unknown`` there is a hint but no support: the rule is not loaded in pf (no counters can exist), or the
|
||||
counters are stale.
|
||||
|
||||
@@ -57,6 +59,8 @@ def unused_confidence(rule: NormRule, hit, now: datetime) -> str:
|
||||
return "unknown" # stale counters say nothing about the present
|
||||
if rule.loaded is None or rule.sched:
|
||||
return "probable" # pf's ruleset not available / the rule only works at certain times
|
||||
if getattr(hit, "peer_missing", False):
|
||||
return "probable" # cluster: some node has no usage data for this rule (rule_hit_service.ClusterHit)
|
||||
return "sure"
|
||||
|
||||
|
||||
@@ -78,7 +82,10 @@ def build(rules: list[NormRule], ctx: Context, *, hits: dict, states: dict, mana
|
||||
key = rule.uuid.lower()
|
||||
hit = hits.get(key)
|
||||
if hit is not None and rule_hit_service.classify(hit, now, unused_days) == "unused":
|
||||
categories.append({"kind": "unused", "confidence": unused_confidence(rule, hit, now), "days": unused_days})
|
||||
category = {"kind": "unused", "confidence": unused_confidence(rule, hit, now), "days": unused_days}
|
||||
if getattr(hit, "peer_missing", False):
|
||||
category["notes"] = ["CLUSTER_PEER_USAGE_UNKNOWN"]
|
||||
categories.append(category)
|
||||
overlap = overlaps.get(id(rule))
|
||||
if overlap is not None:
|
||||
categories.append({
|
||||
|
||||
@@ -117,6 +117,31 @@ def rule_fields(r: NormRule, q: Query, direction: str, iface: str, ctx: Context,
|
||||
return fields
|
||||
|
||||
|
||||
def match_rule(r: NormRule, q: Query, ctx: Context, *, iface: str | None = None,
|
||||
source: Outcome | None = None, overrides: dict[str, Outcome] | None = None) -> Outcome:
|
||||
"""Could the rule's own fields match the packet ``q`` arriving on ``iface``? Order, ``quick``, other rules and
|
||||
whether the rule is enabled or loaded play no part: this is the question "is this rule about such a packet at
|
||||
all", which the cleanup asks about the way the admin reaches the firewall (DESIGN-1.31 §3.2).
|
||||
|
||||
``iface`` defaults to ``q.iface`` and then to the interface guessed from the address; an interface that cannot
|
||||
be resolved is UNKNOWN. ``source`` replaces the source field (the caller judged it for a whole network, see
|
||||
rule_analysis/protection.py); ``overrides`` replaces further fields by name (e.g. ``interface`` for a packet
|
||||
arriving on a device that is not an assigned interface). Tags set by earlier rules are not known here, so ``tagged`` is UNKNOWN. Only
|
||||
``False`` is a proof that the rule does not match; ``None`` (unknown) must be treated like ``True``."""
|
||||
iface = iface or q.iface or guess_iface(q.src if q.direction == "in" else q.dst, ctx)
|
||||
if iface is None:
|
||||
return Outcome(None, ["IFACE_UNRESOLVED"])
|
||||
fields = rule_fields(r, q, q.direction, iface, ctx, set(), None)
|
||||
if source is not None:
|
||||
fields["source"] = source
|
||||
for name, outcome in (overrides or {}).items():
|
||||
if name in fields or name == "interface":
|
||||
fields[name] = outcome
|
||||
value = t_and(*(f.value for f in fields.values()))
|
||||
reasons = sorted({reason for f in fields.values() if f.value is None for reason in f.reasons}) if value is None else []
|
||||
return Outcome(value, reasons)
|
||||
|
||||
|
||||
def _step(r: NormRule, fields: dict[str, Outcome], result, effect: str) -> dict:
|
||||
return {
|
||||
"rule": r.summary(), "result": tri_str(result), "effect": effect,
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""May a rule be switched off without touching the way the admin reaches the firewall? (DESIGN-1.31 §3.2/§3.3)
|
||||
|
||||
Pure functions on ``NormRule`` and the analysis ``Context``; no I/O.
|
||||
|
||||
Management access (§3.2). For every configured management source (IP or network), every address of the
|
||||
firewall (the address in ``host.base_url`` plus every interface address and VIP) and every management port
|
||||
(the API port of ``base_url``, 443, 80, 22) and every interface the packet could arrive on, the rule's own
|
||||
fields are compared with that packet (``engine.match_rule``). A rule is safe only when EVERY such comparison is
|
||||
``False``. ``True`` or unknown means the rule is about management traffic: it is protected.
|
||||
|
||||
Why that is enough without simulating "the ruleset afterwards": a rule that does not match a packet takes no part
|
||||
in its evaluation at all - it neither decides it nor tags it (pf only tags on a match) - so removing it cannot
|
||||
change what happens to that packet. This holds whatever the verdict for the packet is, even when the analysis
|
||||
of the whole ruleset is only "probable". The packet's ingress interface is a routing question OPNexus cannot prove
|
||||
(asymmetric routes, VPNs, failover), so every interface is tried: that only ever protects more.
|
||||
|
||||
A management source that is a network is judged for all of its addresses at once: up to 256 addresses are
|
||||
enumerated (exact, with every resolver rule of the analysis), larger networks are compared as address ranges
|
||||
with the rule's literal/alias/interface addresses; anything that cannot be resolved is unknown = protected.
|
||||
|
||||
Infrastructure (§3.3, reason ``infrastructure``): CARP (IP protocol 112) and pfsync (240) rules, every rule that
|
||||
could match CARP advertisements (to 224.0.0.18 / ff02::12 from the interface's own networks) on an interface with
|
||||
a CARP VIP, and every rule that could match pfsync traffic on the pfsync interface (System > High Availability,
|
||||
``core/hasync/get``; when that cannot be read on a cluster node, on every interface). The second leash of the
|
||||
design ("(self) with port 22/80/443/API" when §3.2 is not False) is part of §3.2 already: such a rule is
|
||||
protected there.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from app.services.rule_analysis import coverage, resolve
|
||||
from app.services.rule_analysis.context import BUILTIN_GROUP_DEVICES, Context
|
||||
from app.services.rule_analysis.engine import Query, match_rule
|
||||
from app.services.rule_analysis.model import Endpoint, IPAddr, IPNet, NormRule, Outcome, o_or, t_not
|
||||
|
||||
MANAGEMENT_PORTS = (443, 80, 22)
|
||||
# A management network up to this size is checked address by address (exact); larger ones as address ranges.
|
||||
ENUMERATE_LIMIT = 256
|
||||
CARP_PROTO, PFSYNC_PROTO = "112", "240"
|
||||
CARP_GROUPS = {4: ipaddress.ip_address("224.0.0.18"), 6: ipaddress.ip_address("ff02::12")}
|
||||
PFSYNC_GROUP = ipaddress.ip_address("224.0.0.240")
|
||||
_LOOPBACK = {ipaddress.ip_address("127.0.0.1"), ipaddress.ip_address("::1")}
|
||||
|
||||
|
||||
def parse_sources(values) -> list[IPNet]:
|
||||
"""The configured management sources as networks (an address becomes a /32 or /128). Invalid entries are
|
||||
dropped; the caller treats an empty result as "not configured" (the cleanup is then locked)."""
|
||||
nets: list[IPNet] = []
|
||||
for value in values or []:
|
||||
try:
|
||||
net = ipaddress.ip_network(str(value).strip(), strict=False)
|
||||
except ValueError:
|
||||
continue
|
||||
if net not in nets:
|
||||
nets.append(net)
|
||||
return nets
|
||||
|
||||
|
||||
def management_targets(base_url: str, ctx: Context, extra_ports=()) -> tuple[list[IPAddr], list[int]]:
|
||||
"""Firewall addresses and ports the management traffic goes to; ``extra_ports`` = the admin's
|
||||
``cleanup_management_ports`` (an SSH or GUI port moved away from the defaults)."""
|
||||
parts = urlsplit(base_url or "")
|
||||
addrs: list[IPAddr] = []
|
||||
try:
|
||||
addrs.append(ipaddress.ip_address((parts.hostname or "").strip("[]")))
|
||||
except ValueError:
|
||||
pass # a DNS name: the interface addresses below cover the address it resolves to on the firewall
|
||||
for addr in ctx.self_addrs:
|
||||
if addr not in _LOOPBACK and addr not in addrs:
|
||||
addrs.append(addr)
|
||||
try:
|
||||
api_port = parts.port or 443
|
||||
except ValueError:
|
||||
api_port = 443
|
||||
ports = [api_port]
|
||||
for port in (*MANAGEMENT_PORTS, *extra_ports):
|
||||
if isinstance(port, int) and 1 <= port <= 65535 and port not in ports:
|
||||
ports.append(port)
|
||||
return addrs, ports
|
||||
|
||||
|
||||
def _interfaces(ctx: Context) -> list[str]:
|
||||
return sorted(i for i in ctx.interfaces if i != "lo0")
|
||||
|
||||
|
||||
# A packet can also arrive on a device that is not an assigned interface: a WireGuard instance (wg0), an OpenVPN
|
||||
# server (ovpns1), IPsec (enc0). They have no identifier, so they are not in ``ctx.interfaces``, and the built-in
|
||||
# groups that stand for them (``wireguard``, ``openvpn``, ``enc0``) list no members in ``firewall/group/searchItem``
|
||||
# (live fixture tests/fixtures/rule_analysis_fw.json: members ""). One extra, virtual ingress is checked per query:
|
||||
UNASSIGNED = "(unassigned)"
|
||||
|
||||
|
||||
def unassigned_interface(rule: NormRule, ctx: Context) -> Outcome:
|
||||
"""Could the rule's interface field match a packet arriving on an unassigned device? An assigned interface or a
|
||||
user-defined group (its members are assigned interfaces) cannot; a built-in group or an unknown name might
|
||||
(members unknown = protected); no interface (floating "any") always does."""
|
||||
if not rule.interfaces:
|
||||
return Outcome(True)
|
||||
results = []
|
||||
for name in rule.interfaces:
|
||||
if name in ctx.interfaces:
|
||||
results.append(Outcome(False))
|
||||
elif name in BUILTIN_GROUP_DEVICES or ctx.groups is None or name not in ctx.groups:
|
||||
results.append(Outcome(None, ["IFACE_MEMBERS_UNKNOWN"]))
|
||||
else:
|
||||
results.append(Outcome(False))
|
||||
result = o_or(results)
|
||||
return Outcome(t_not(result.value), result.reasons) if rule.if_not else result
|
||||
|
||||
|
||||
def _ingress(rule: NormRule, ctx: Context) -> list[tuple[str, dict | None]]:
|
||||
"""(interface, field overrides) for every ingress a management packet could take."""
|
||||
virtual = {"interface": unassigned_interface(rule, ctx)}
|
||||
if rule.received_on:
|
||||
virtual["received_on"] = Outcome(None, ["RECEIVED_ON"])
|
||||
return [(i, None) for i in _interfaces(ctx)] + [(UNASSIGNED, virtual)]
|
||||
|
||||
|
||||
def _item_ranges(token: str, ctx: Context) -> list[coverage.Range] | None:
|
||||
"""Address ranges an endpoint item stands for; None when that is not static."""
|
||||
ranges, _alias = coverage._ranges_of(token, ctx)
|
||||
if ranges is not None:
|
||||
return ranges
|
||||
name = token.strip().lower()
|
||||
if name in ("(self)", "self", "this firewall"):
|
||||
return [(a.version, int(a), int(a)) for a in ctx.self_addrs]
|
||||
if name.endswith("ip") and name[:-2] in ctx.interfaces:
|
||||
info = ctx.interfaces[name[:-2]]
|
||||
return [(a.version, int(a), int(a)) for a in (*info.addrs, *info.vips)]
|
||||
if name.startswith("__") and name.endswith("_network"):
|
||||
name = name[2:-len("_network")]
|
||||
if name in ctx.interfaces:
|
||||
return [(n.version, int(n.network_address), int(n.broadcast_address)) for n in ctx.interfaces[name].nets]
|
||||
return None
|
||||
|
||||
|
||||
def _covers_all(ranges: list[coverage.Range], version: int, low: int, high: int) -> bool:
|
||||
cursor = low
|
||||
for _v, a, b in sorted(r for r in ranges if r[0] == version):
|
||||
if a > cursor:
|
||||
return False
|
||||
cursor = max(cursor, b + 1)
|
||||
if cursor > high:
|
||||
return True
|
||||
return cursor > high
|
||||
|
||||
|
||||
def source_outcome(endpoint: Endpoint, net: IPNet, ctx: Context) -> Outcome:
|
||||
"""Does the endpoint match at least one address of ``net``? (False is a proof for every address.)"""
|
||||
if endpoint.is_any:
|
||||
return Outcome(True)
|
||||
if net.num_addresses <= ENUMERATE_LIMIT:
|
||||
return o_or([resolve.endpoint_match(endpoint, ip, ctx) for ip in net])
|
||||
ranges: list[coverage.Range] = []
|
||||
for item in endpoint.items:
|
||||
got = _item_ranges(item, ctx)
|
||||
if got is None:
|
||||
return Outcome(None, ["SOURCE_NOT_RESOLVABLE"])
|
||||
ranges += got
|
||||
low, high = int(net.network_address), int(net.broadcast_address)
|
||||
if endpoint.negated:
|
||||
return Outcome(not _covers_all(ranges, net.version, low, high))
|
||||
return Outcome(any(v == net.version and a <= high and b >= low for v, a, b in ranges))
|
||||
|
||||
|
||||
def _query(src: IPAddr, dst: IPAddr, proto: str, label: str, dport: int | None) -> Query:
|
||||
return Query(src=src, dst=dst, proto=proto, proto_label=label, dport=dport, direction="in", egress="none")
|
||||
|
||||
|
||||
def management_check(rule: NormRule, ctx: Context, sources: list[IPNet], targets: list[IPAddr],
|
||||
ports: list[int]) -> dict:
|
||||
"""{"result": "false" | "true" | "unknown", "queries": n, "reasons": [...], "match": {...} | None}.
|
||||
Only "false" makes the rule removable."""
|
||||
reasons: set[str] = set()
|
||||
queries = 0
|
||||
unknown_at = None
|
||||
interfaces = _ingress(rule, ctx)
|
||||
for net in sources:
|
||||
src = source_outcome(rule.src, net, ctx)
|
||||
if src.value is False:
|
||||
dsts = [d for d in targets if d.version == net.version]
|
||||
queries += len(dsts) * len(ports) * len(interfaces)
|
||||
continue
|
||||
for dst in (d for d in targets if d.version == net.version):
|
||||
for port in ports:
|
||||
q = _query(net.network_address, dst, "6", "tcp", port)
|
||||
for iface, overrides in interfaces:
|
||||
queries += 1
|
||||
out = match_rule(rule, q, ctx, iface=iface, source=src, overrides=overrides)
|
||||
where = {"source": str(net), "destination": str(dst), "port": port, "interface": iface}
|
||||
if out.value is True:
|
||||
return {"result": "true", "queries": queries, "reasons": [], "match": where}
|
||||
if out.value is None:
|
||||
reasons.update(out.reasons)
|
||||
unknown_at = unknown_at or where
|
||||
if not queries:
|
||||
# No address of the firewall in the family of any source: nothing could be proven.
|
||||
return {"result": "unknown", "queries": 0, "reasons": ["NO_MANAGEMENT_QUERY"], "match": None}
|
||||
if unknown_at is not None:
|
||||
return {"result": "unknown", "queries": queries, "reasons": sorted(reasons), "match": unknown_at}
|
||||
return {"result": "false", "queries": queries, "reasons": [], "match": None}
|
||||
|
||||
|
||||
def _hasync_interface(hasync: dict | None) -> str | None:
|
||||
"""The pfsync interface from ``core/hasync/get`` ("" = HA sync disabled); None when not readable."""
|
||||
node = (hasync or {}).get("hasync") if isinstance(hasync, dict) else None
|
||||
if not isinstance(node, dict) or "pfsyncinterface" not in node:
|
||||
return None
|
||||
value = node.get("pfsyncinterface")
|
||||
if isinstance(value, dict):
|
||||
return next((k for k, o in value.items() if isinstance(o, dict) and str(o.get("selected")) == "1"), "")
|
||||
return str(value or "")
|
||||
|
||||
|
||||
def _hasync_peer(hasync: dict | None) -> IPAddr | None:
|
||||
node = (hasync or {}).get("hasync") if isinstance(hasync, dict) else None
|
||||
try:
|
||||
return ipaddress.ip_address(str((node or {}).get("pfsyncpeerip") or "").strip())
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def infrastructure_reason(rule: NormRule, ctx: Context, *, hasync: dict | None, clustered: bool) -> str | None:
|
||||
"""'infrastructure' when the rule could carry CARP or pfsync traffic, else None."""
|
||||
if rule.protos is not None and rule.protos & {CARP_PROTO, PFSYNC_PROTO}:
|
||||
return "infrastructure"
|
||||
for info in ctx.interfaces.values():
|
||||
if info.identifier == "lo0" or not info.vips:
|
||||
continue
|
||||
for net in info.nets:
|
||||
group = CARP_GROUPS.get(net.version)
|
||||
out = match_rule(rule, _query(net.network_address, group, CARP_PROTO, "carp", None), ctx,
|
||||
iface=info.identifier, source=source_outcome(rule.src, net, ctx))
|
||||
if out.value is not False:
|
||||
return "infrastructure"
|
||||
pfsync_if = _hasync_interface(hasync)
|
||||
if pfsync_if == "" or (pfsync_if is None and not clustered):
|
||||
return None # HA sync off (or a single firewall whose HA settings are not readable)
|
||||
peer = _hasync_peer(hasync)
|
||||
candidates = [ctx.interfaces[pfsync_if]] if pfsync_if in ctx.interfaces else \
|
||||
[i for i in ctx.interfaces.values() if i.identifier != "lo0"] # unknown: every interface
|
||||
for info in candidates:
|
||||
for net in info.nets:
|
||||
if peer is not None and peer.version == net.version:
|
||||
src = Outcome(resolve.endpoint_match(rule.src, peer, ctx).value)
|
||||
src_ip = peer
|
||||
else:
|
||||
src, src_ip = source_outcome(rule.src, net, ctx), net.network_address
|
||||
for dst in [*(a for a in info.addrs if a.version == net.version)] + ([PFSYNC_GROUP] if net.version == 4 else []):
|
||||
out = match_rule(rule, _query(src_ip, dst, PFSYNC_PROTO, "pfsync", None), ctx,
|
||||
iface=info.identifier, source=src)
|
||||
if out.value is not False:
|
||||
return "infrastructure"
|
||||
return None
|
||||
@@ -107,17 +107,47 @@ async def _managed_rules(db: AsyncSession, host: OpnsenseHost) -> dict[str, dict
|
||||
return managed
|
||||
|
||||
|
||||
async def cleanup_candidates(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
|
||||
refresh: bool = False, now: datetime | None = None) -> dict:
|
||||
"""Read-only. Raises when the firewall cannot be read (callers map that)."""
|
||||
async def _hits(db: AsyncSession, host_id) -> dict:
|
||||
return {r.rule_uuid.lower(): r for r in (await db.scalars(select(RuleHit).where(RuleHit.host_id == host_id))).all()}
|
||||
|
||||
|
||||
async def cluster_hits(db: AsyncSession, host: OpnsenseHost) -> tuple[dict, dict]:
|
||||
"""(usage to judge "unused" with, the host's own hit rows). For a cluster member the usage is merged over all
|
||||
nodes (DESIGN-1.31 §3.4): "unused" only when it holds on every node."""
|
||||
own = await _hits(db, host.id)
|
||||
if not host.cluster_name:
|
||||
return own, own
|
||||
others = (await db.scalars(select(OpnsenseHost.id).where(
|
||||
OpnsenseHost.cluster_name == host.cluster_name, OpnsenseHost.id != host.id))).all()
|
||||
if not others:
|
||||
return own, own
|
||||
return rule_hit_service.merge_cluster_hits(own, [await _hits(db, other) for other in others]), own
|
||||
|
||||
|
||||
async def cleanup_view(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
|
||||
refresh: bool = False, now: datetime | None = None):
|
||||
"""(context, normalized rules, candidate payload, managed rules) - the cleanup service needs all of them."""
|
||||
now = now or datetime.now(timezone.utc)
|
||||
ctx = await ctx_mod.load_context(host, use_cache=not refresh)
|
||||
rules, notes = normalize(ctx)
|
||||
hits = {r.rule_uuid.lower(): r for r in (await db.scalars(select(RuleHit).where(RuleHit.host_id == host.id))).all()}
|
||||
usage, hits = await cluster_hits(db, host)
|
||||
states = {s.rule_uuid: s for s in (await db.scalars(select(LiveRuleState).where(LiveRuleState.host_id == host.id))).all()}
|
||||
rows = cleanup.build(rules, ctx, hits=hits, states=states, managed=await _managed_rules(db, host), now=now,
|
||||
managed = await _managed_rules(db, host)
|
||||
rows = cleanup.build(rules, ctx, hits=usage, states=states, managed=managed, now=now,
|
||||
unused_days=unused_days, order_guessed=coverage.order_doubtful(notes))
|
||||
return ctx, rules, _payload(host, rules, rows, hits, states, notes, ctx, unused_days, now), managed
|
||||
|
||||
|
||||
async def cleanup_candidates(db: AsyncSession, host: OpnsenseHost, *, unused_days: int = cleanup.DEFAULT_UNUSED_DAYS,
|
||||
refresh: bool = False, now: datetime | None = None) -> dict:
|
||||
"""Read-only. Raises when the firewall cannot be read (callers map that)."""
|
||||
return (await cleanup_view(db, host, unused_days=unused_days, refresh=refresh, now=now))[2]
|
||||
|
||||
|
||||
def _payload(host, rules, rows, hits, states, notes, ctx, unused_days, now) -> dict:
|
||||
all_notes = sorted({*notes, *ctx.notes})
|
||||
if host.cluster_name and any("CLUSTER_PEER_USAGE_UNKNOWN" in c.get("notes", ()) for r in rows for c in r["categories"]):
|
||||
all_notes = sorted({*all_notes, "CLUSTER_PEER_USAGE_UNKNOWN"})
|
||||
return {
|
||||
"host_id": str(host.id), "host_name": host.name, "cluster_name": host.cluster_name,
|
||||
"unused_days": max(cleanup.MIN_DAYS, min(cleanup.MAX_DAYS, int(unused_days))), "generated_at": now,
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -155,9 +155,13 @@ async def sweep_expired_deployments(db: AsyncSession) -> list[RuleDeployment]:
|
||||
|
||||
|
||||
async def undeploy_rule(db: AsyncSession, rule: FirewallRule) -> None:
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup
|
||||
failures = []
|
||||
result = await db.execute(select(RuleDeployment).where(RuleDeployment.rule_id == rule.id))
|
||||
for deployment in result.scalars().all():
|
||||
deployments = list(result.scalars().all())
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them.
|
||||
await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])
|
||||
for deployment in deployments:
|
||||
host = await db.get(OpnsenseHost, deployment.host_id)
|
||||
if host and deployment.opnsense_rule_uuid:
|
||||
try:
|
||||
|
||||
@@ -40,10 +40,12 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
BASELINE_KEY = "__baseline__"
|
||||
RETENTION_DAYS = 365
|
||||
# Audit kinds whose events carry the UUID of a rule on a firewall in details["remote_uuid"].
|
||||
AUDIT_KINDS = ("rule", "policy_hierarchy")
|
||||
# Audit statuses that mean the firewall's rule was (or was meant to be) touched.
|
||||
AUDIT_STATUSES_NO_CHANGE = ("requested", "interrupted", "error")
|
||||
# Audit kinds whose events carry the UUID of a rule on a firewall in details["remote_uuid"]. "rule_cleanup": the
|
||||
# cleanup assistant (rule_cleanup_service) switching a hand-made rule off/on, deleting or restoring it.
|
||||
AUDIT_KINDS = ("rule", "policy_hierarchy", "rule_cleanup")
|
||||
# Audit statuses that mean the firewall's rule was NOT changed by that event (so it cannot explain a change).
|
||||
# "confirmed": the cleanup's confirmation only re-reads the firewall.
|
||||
AUDIT_STATUSES_NO_CHANGE = ("requested", "interrupted", "error", "confirmed")
|
||||
# The audit entry is written right after the firewall changed; allow for the observer running just before it.
|
||||
AUDIT_SLACK = timedelta(minutes=2)
|
||||
RECHECK_WINDOW = timedelta(minutes=45)
|
||||
@@ -274,8 +276,11 @@ async def prune(db: AsyncSession, retention_days: int = RETENTION_DAYS) -> int:
|
||||
events = await db.execute(delete(RuleChangeEvent).where(RuleChangeEvent.observed_at < cutoff))
|
||||
gone = await db.execute(delete(LiveRuleState).where(
|
||||
LiveRuleState.present.is_(False), LiveRuleState.rule_uuid != BASELINE_KEY, LiveRuleState.changed_at < cutoff))
|
||||
# Finished cleanup operations (with the snapshots of deleted rules) are kept as long as the history.
|
||||
from app.services.rule_cleanup_service import prune_operations
|
||||
cleanups = await prune_operations(db, cutoff)
|
||||
await db.commit()
|
||||
return (events.rowcount or 0) + (gone.rowcount or 0)
|
||||
return (events.rowcount or 0) + (gone.rowcount or 0) + cleanups
|
||||
|
||||
|
||||
# ── Reading the history of one rule ───────────────────────────────────────
|
||||
|
||||
@@ -9,7 +9,9 @@ releases without that endpoint fall back to parsing ``diagnostics/firewall/pfSta
|
||||
* A counter that grew since the last sample is a hit "now" (at most one sampling interval late).
|
||||
* A counter that is *lower* than before means pf reloaded its ruleset (an apply or a reboot), which
|
||||
zeroes everything. Whatever it shows above zero was hit since the reload, so that also counts as a hit now;
|
||||
zero tells nothing, the old ``last_hit_at`` stays.
|
||||
zero tells nothing, the old ``last_hit_at`` stays. (Live, 2026-10-22, OPNsense 26.7.7: ``firewall/filter/apply``
|
||||
WITHOUT any pending change already zeroes every ``ruleStats`` counter - 16228 -> 0 packets on the busiest rule -
|
||||
so every apply is such a reload; writers sample right before their own apply, see rule_cleanup_service.)
|
||||
* A rule is called unused only after OPNexus has watched it for ``UNUSED_AFTER_DAYS`` without a hit.
|
||||
A new rule, or a brand-new installation, is "still collecting data" instead of a false alarm.
|
||||
|
||||
@@ -20,6 +22,7 @@ never tracked.
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete, select, update
|
||||
@@ -165,6 +168,41 @@ async def sample_all(db: AsyncSession) -> int:
|
||||
return sampled
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClusterHit:
|
||||
"""The usage of one rule over all nodes of a cluster (DESIGN-1.31 §3.4). Has the attributes ``classify`` and
|
||||
``cleanup.unused_confidence`` read from a ``RuleHit``. Traffic goes through the CARP master, a backup sees
|
||||
almost nothing: a rule is only unused when it is unused everywhere, so the newest hit of any node counts, the
|
||||
window starts at the latest start of any node and the counters are only as fresh as the oldest node's."""
|
||||
rule_uuid: str
|
||||
enabled: bool
|
||||
first_seen_at: datetime
|
||||
last_hit_at: datetime | None
|
||||
last_sampled_at: datetime
|
||||
# The rule is not tracked on some node (missing there, or the node was never sampled): nothing is known
|
||||
# about its traffic there, so "unused" is never better than "probable".
|
||||
peer_missing: bool = False
|
||||
|
||||
|
||||
def merge_cluster_hits(own: dict[str, RuleHit], peers: list[dict[str, RuleHit]]) -> dict[str, ClusterHit]:
|
||||
"""Per rule over the union of all nodes' rows (a rule this node has no row for is still judged, as
|
||||
``peer_missing``: what this node saw of it is unknown)."""
|
||||
merged: dict[str, ClusterHit] = {}
|
||||
for key in {*own, *(k for p in peers for k in p)}:
|
||||
rows = [own.get(key), *(p.get(key) for p in peers)]
|
||||
present = [r for r in rows if r is not None]
|
||||
hits = [r.last_hit_at for r in present if r.last_hit_at is not None]
|
||||
mine = own.get(key) or present[0]
|
||||
merged[key] = ClusterHit(
|
||||
rule_uuid=mine.rule_uuid, enabled=mine.enabled,
|
||||
first_seen_at=max(r.first_seen_at for r in present),
|
||||
last_hit_at=max(hits) if hits else None,
|
||||
last_sampled_at=min(r.last_sampled_at for r in present),
|
||||
peer_missing=len(present) < len(rows),
|
||||
)
|
||||
return merged
|
||||
|
||||
|
||||
def classify(row: RuleHit, now: datetime, days: int = UNUSED_AFTER_DAYS) -> str:
|
||||
"""'used' | 'unused' | 'collecting' | 'disabled'; ``days`` is the observation window."""
|
||||
if not row.enabled:
|
||||
|
||||
@@ -40,7 +40,7 @@ from app.models.host_operation import HostOperation
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from contextlib import contextmanager
|
||||
|
||||
from app.services import audit_service, backup_store, opnsense_client as client
|
||||
from app.services import audit_service, backup_store, opnsense_client as client, snapshot_service
|
||||
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
|
||||
@@ -202,6 +202,17 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
await db.commit()
|
||||
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
|
||||
return
|
||||
if sched.action in _BACKUP_BEFORE:
|
||||
# Way back (1.32, B5): a boot environment of the running system, after every other check passed.
|
||||
# Fail closed like the backup: no rollback point on a ZFS firewall, no update.
|
||||
try:
|
||||
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": f"Rückweg-Punkt (Boot-Environment) vor dem Update nicht angelegt: {str(exc)[:300]}"}
|
||||
await db.commit()
|
||||
logger.warning("Geplante Wartung %s: Rückweg-Punkt fehlgeschlagen", sched.id)
|
||||
return
|
||||
|
||||
boot_before = await _boot_time(host) if sched.auto_verify else None
|
||||
op = HostOperation(
|
||||
@@ -302,6 +313,9 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
backup_store.save_snapshot(host.id, await client.download_backup(host))
|
||||
state = await inspect_host(db, host, skip_role_checks=True)
|
||||
blocked = list(dict.fromkeys(blocked + state["actions"][sched.action]))
|
||||
if not blocked and sched.action in _BACKUP_BEFORE:
|
||||
# Way back (1.32, B5) while nothing has been toggled yet; a failure ends here like a failed backup.
|
||||
state["rollback_point"] = await snapshot_service.create_rollback_point(host, state.get("version"))
|
||||
except client.OpnsenseAPIError as exc:
|
||||
await fail(f"Vorbereitung der CARP-Übergabe fehlgeschlagen, nichts wurde verändert: {str(exc)[:300]}")
|
||||
return
|
||||
|
||||
@@ -1,17 +1,22 @@
|
||||
"""ZFS snapshots (boot environments) of one firewall: list, create, delete.
|
||||
|
||||
Snapshots are a safety net before bigger changes, not a rollback button:
|
||||
activating one needs a reboot and discards everything since, so that action is
|
||||
deliberately NOT offered here. Deleting refuses the active/next-boot snapshot
|
||||
Snapshots are a safety net before bigger changes. Activating an arbitrary one is
|
||||
deliberately NOT offered (it needs a reboot and discards everything since); the
|
||||
only way back is the guided firmware rollback to a point OPNexus itself created
|
||||
right before an update (see "Rollback points" below, maintenance action
|
||||
`firmware_rollback`). Deleting refuses the active/next-boot snapshot
|
||||
(the firewall refuses too; we say so up front). Every step is audited.
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import audit_service, opnsense_client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$")
|
||||
|
||||
|
||||
@@ -23,20 +28,121 @@ def _row(row: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
async def list_snapshots(host: OpnsenseHost) -> dict:
|
||||
async def supported(host: OpnsenseHost) -> bool:
|
||||
try:
|
||||
supported = (await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported")
|
||||
return bool((await opnsense_client._request(host, "GET", "/core/snapshots/isSupported")).get("supported"))
|
||||
except OpnsenseAPIError as exc:
|
||||
if "404" in str(exc):
|
||||
return {"supported": False, "rows": []}
|
||||
if "404" in str(exc): # OPNsense without the snapshots API
|
||||
return False
|
||||
raise
|
||||
if not supported:
|
||||
|
||||
|
||||
async def list_snapshots(host: OpnsenseHost) -> dict:
|
||||
if not await supported(host):
|
||||
return {"supported": False, "rows": []}
|
||||
return {"supported": True, "rows": [_row(r) for r in await opnsense_client.list_snapshots(host)]}
|
||||
|
||||
|
||||
# ── Rollback points (1.32, B5) ──────────────────────────────────────────────
|
||||
# Before every firmware update/upgrade OPNexus creates a boot environment of the running system. Going back to it
|
||||
# (activate + reboot) restores the old firmware AND the old configuration of that moment: everything changed on the
|
||||
# firewall since is gone. Only these points are offered for going back, never boot environments made by hand.
|
||||
ROLLBACK_PREFIX = "opnexus-pre-"
|
||||
KEEP_ROLLBACK_POINTS = 2
|
||||
_POINT = re.compile(r"^opnexus-pre-(?P<version>[A-Za-z0-9._-]+)-(?P<stamp>\d{8}-\d{6})$")
|
||||
|
||||
|
||||
def _is_active(row: dict) -> bool:
|
||||
# N = active now, R = active after reboot.
|
||||
return str(row.get("active", "")).strip() not in ("", "-")
|
||||
|
||||
|
||||
def rollback_points(rows: list[dict]) -> list[dict]:
|
||||
"""OPNexus rollback points, newest first: {uuid, name, version, stamp, active, created}."""
|
||||
out = []
|
||||
for row in rows:
|
||||
match = _POINT.match(str(row.get("name", "")))
|
||||
if match:
|
||||
out.append({"uuid": str(row.get("uuid", "")), "name": match.group(0), "version": match.group("version"),
|
||||
"stamp": match.group("stamp"), "active": str(row.get("active", "")).strip(),
|
||||
"created": str(row.get("created_str", ""))})
|
||||
return sorted(out, key=lambda p: p["stamp"], reverse=True)
|
||||
|
||||
|
||||
async def create_rollback_point(host: OpnsenseHost, version: str | None) -> dict | None:
|
||||
"""Boot environment right before a firmware action. None when the firewall has no ZFS snapshots (UFS install):
|
||||
the update may still run, without a way back. Raises OpnsenseAPIError when creating it fails (callers abort the
|
||||
update: fail closed). Older OPNexus points beyond KEEP_ROLLBACK_POINTS are removed afterwards."""
|
||||
if not await supported(host):
|
||||
return None
|
||||
clean_version = re.sub(r"[^A-Za-z0-9._-]", "-", version or "unbekannt")[:30] or "unbekannt"
|
||||
name = f"{ROLLBACK_PREFIX}{clean_version}-{datetime.now(timezone.utc):%Y%m%d-%H%M%S}"
|
||||
audit = {"kind": "snapshot", "action": "snapshot.rollback_point", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
|
||||
await audit_service.persist(**audit, status="started", message=None, details={"version": version})
|
||||
try:
|
||||
await opnsense_client.create_snapshot(host, name, f"OPNexus: Rückweg-Punkt vor dem Firmware-Update von {version}")
|
||||
rows = await opnsense_client.list_snapshots(host)
|
||||
except OpnsenseAPIError as exc:
|
||||
await audit_service.persist(**audit, status="rejected", message=str(exc)[:500], details={})
|
||||
raise
|
||||
point = next((p for p in rollback_points(rows) if p["name"] == name), None)
|
||||
if point is None:
|
||||
await audit_service.persist(**audit, status="unverified", message=None, details={})
|
||||
raise OpnsenseAPIError(f"Rückweg-Punkt {name} nach dem Anlegen nicht in der Liste")
|
||||
await audit_service.persist(**audit, status="applied", message=None, details={})
|
||||
for old in [p for p in rollback_points(rows) if not _is_active(p)][KEEP_ROLLBACK_POINTS:]:
|
||||
try: # housekeeping only: a point that cannot be removed stays, the update goes on
|
||||
await opnsense_client.delete_snapshot(host, old["uuid"])
|
||||
await audit_service.persist(kind="snapshot", action="snapshot.del", entity_name=old["name"], host_id=str(host.id),
|
||||
host_name=host.name, status="applied", message="Älterer Rückweg-Punkt aufgeräumt", details={})
|
||||
except OpnsenseAPIError as exc:
|
||||
logger.warning("Rückweg-Punkt %s auf %s nicht gelöscht: %s", old["name"], host.name, exc)
|
||||
return {"uuid": point["uuid"], "name": point["name"], "version": version}
|
||||
|
||||
|
||||
async def rollback_state(host: OpnsenseHost) -> dict:
|
||||
"""What going back would do: the newest OPNexus point that is not running now, and the boot environment that is."""
|
||||
if not await supported(host):
|
||||
return {"supported": False, "target": None, "current": None}
|
||||
rows = await opnsense_client.list_snapshots(host)
|
||||
current = next((r for r in rows if "N" in str(r.get("active", ""))), None) # running now (R alone = next start)
|
||||
target = next((p for p in rollback_points(rows) if "N" not in p["active"]), None)
|
||||
return {"supported": True, "target": target,
|
||||
"current": {"uuid": str(current.get("uuid", "")), "name": str(current.get("name", ""))} if current else None}
|
||||
|
||||
|
||||
async def activate_and_reboot(host: OpnsenseHost, rollback: dict) -> dict:
|
||||
"""Activate the rollback point, check it is marked for the next start, reboot. If the reboot cannot be handed over,
|
||||
the previous boot environment is activated again: an activated old one must never wait for the next unplanned
|
||||
reboot."""
|
||||
target, current = rollback["target"], rollback["current"]
|
||||
|
||||
async def undo() -> None:
|
||||
if current:
|
||||
try:
|
||||
await opnsense_client.activate_snapshot(host, current["uuid"])
|
||||
except OpnsenseAPIError:
|
||||
logger.warning("Rückweg auf %s: vorheriges Boot-Environment nicht reaktiviert", host.name)
|
||||
|
||||
await opnsense_client.activate_snapshot(host, target["uuid"])
|
||||
try:
|
||||
rows = await opnsense_client.list_snapshots(host)
|
||||
except OpnsenseAPIError:
|
||||
await undo() # cannot confirm the mark: never leave it waiting for an unplanned reboot
|
||||
raise
|
||||
marked = next((r for r in rows if str(r.get("uuid", "")) == target["uuid"]), None)
|
||||
if marked is None or "R" not in str(marked.get("active", "")):
|
||||
await undo()
|
||||
raise OpnsenseAPIError(f"Boot-Environment {target['name']} ist nicht für den nächsten Start vorgemerkt")
|
||||
try:
|
||||
return await opnsense_client.reboot_host(host)
|
||||
except OpnsenseAPIError:
|
||||
await undo()
|
||||
raise
|
||||
|
||||
|
||||
async def create(host: OpnsenseHost, name: str, note: str) -> dict:
|
||||
if not NAME.match(name):
|
||||
if not NAME.match(name) or name.startswith(ROLLBACK_PREFIX): # that prefix marks OPNexus' own rollback points
|
||||
raise api_error(422, ErrorCode.NETWORK_FIELD_INVALID, field="name")
|
||||
audit = {"kind": "snapshot", "action": "snapshot.add", "entity_name": name, "host_id": str(host.id), "host_name": host.name}
|
||||
await audit_service.persist(**audit, status="started", message=None, details={"note": note[:200]})
|
||||
|
||||
@@ -374,6 +374,9 @@ async def undeploy_tunnel(db: AsyncSession, tunnel: WireguardTunnel) -> None:
|
||||
dependency direction; deleting the Peer first also auto-unlinks it
|
||||
from the Server's `peers` list server-side, per delClientAction's own
|
||||
behavior — see module docstring), then reconfigures."""
|
||||
# Deleting writes to the firewalls: not while a rule cleanup (1.31) owns one of them (or its cluster).
|
||||
from app.services.host_maintenance_service import ensure_no_open_cleanup_on
|
||||
await ensure_no_open_cleanup_on(db, WireguardTunnelDeployment, WireguardTunnelDeployment.tunnel_id == tunnel.id)
|
||||
failures = []
|
||||
result = await db.execute(select(WireguardTunnelDeployment).where(WireguardTunnelDeployment.tunnel_id == tunnel.id))
|
||||
for deployment in result.scalars().all():
|
||||
|
||||
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
|
||||
Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
"""
|
||||
|
||||
APP_VERSION = "1.30.0"
|
||||
APP_CODENAME = "Owl"
|
||||
APP_VERSION = "1.32.0"
|
||||
APP_CODENAME = "Merlin"
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Regel-Bereinigung (1.31, B4 Teil 2): Operationen, Einträge, Einstellungen.
|
||||
|
||||
Revision ID: 0029
|
||||
Revises: 0028
|
||||
Create Date: 2026-10-22
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0029"
|
||||
down_revision = "0028"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
UUID = postgresql.UUID(as_uuid=True)
|
||||
TS = sa.DateTime(timezone=True)
|
||||
|
||||
# Must equal OPEN_ITEM_STATES in app/models/rule_cleanup.py (the schema comparison of the tests notices a drift).
|
||||
_OPEN_STATES = ("queued", "disabling", "disabled", "reverting", "deleting", "restoring", "unknown", "revert_failed")
|
||||
|
||||
_SETTINGS = [
|
||||
("cleanup_max_rules", "integer NOT NULL DEFAULT 10"),
|
||||
("cleanup_delete_min_days", "integer NOT NULL DEFAULT 7"),
|
||||
("cleanup_management_sources", "json NOT NULL DEFAULT '[]'::json"),
|
||||
("cleanup_management_ports", "json NOT NULL DEFAULT '[]'::json"),
|
||||
]
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which may already have made all of it.
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
if not inspector.has_table("rule_cleanup_operations"):
|
||||
op.create_table(
|
||||
"rule_cleanup_operations",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("host_id", UUID, nullable=False),
|
||||
sa.Column("cluster_name", sa.String(120), nullable=True),
|
||||
sa.Column("kind", sa.String(16), nullable=False),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("actor", sa.String(150), nullable=False),
|
||||
sa.Column("actor_id", UUID, nullable=True),
|
||||
sa.Column("created_at", TS, nullable=False),
|
||||
sa.Column("expires_at", TS, nullable=False),
|
||||
sa.Column("started_at", TS, nullable=True),
|
||||
sa.Column("heartbeat_at", TS, nullable=True),
|
||||
sa.Column("revert_deadline", TS, nullable=True),
|
||||
sa.Column("confirmed_at", TS, nullable=True),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.Column("message", sa.String(1000), nullable=True),
|
||||
sa.Column("preview_fingerprint", sa.String(64), nullable=False),
|
||||
sa.Column("options", sa.JSON(), nullable=False),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_rule_cleanup_operations_host", "rule_cleanup_operations", ["host_id", "created_at"])
|
||||
op.create_index("ix_rule_cleanup_operations_status", "rule_cleanup_operations", ["status"])
|
||||
if not inspector.has_table("rule_cleanup_items"):
|
||||
op.create_table(
|
||||
"rule_cleanup_items",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("operation_id", UUID, nullable=False),
|
||||
sa.Column("host_id", UUID, nullable=False),
|
||||
sa.Column("rule_uuid", sa.String(64), nullable=False),
|
||||
sa.Column("seq", sa.Integer(), nullable=False),
|
||||
sa.Column("carp_role", sa.String(16), nullable=True),
|
||||
sa.Column("state", sa.String(16), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=False),
|
||||
sa.Column("categories", sa.JSON(), nullable=False),
|
||||
sa.Column("confidence", sa.String(16), nullable=True),
|
||||
sa.Column("row_before", sa.JSON(), nullable=True),
|
||||
sa.Column("fp_before", sa.String(64), nullable=True),
|
||||
sa.Column("cover_uuid", sa.String(64), nullable=True),
|
||||
sa.Column("cover_fp", sa.String(64), nullable=True),
|
||||
sa.Column("snapshot", sa.JSON(), nullable=True),
|
||||
sa.Column("anchor_next_uuid", sa.String(64), nullable=True),
|
||||
sa.Column("anchor_prev_uuid", sa.String(64), nullable=True),
|
||||
sa.Column("prio_group", sa.String(32), nullable=True),
|
||||
sa.Column("source_item_id", UUID, nullable=True),
|
||||
sa.Column("restored_uuid", sa.String(64), nullable=True),
|
||||
sa.Column("changed_at", TS, nullable=True),
|
||||
sa.Column("message", sa.String(1000), nullable=True),
|
||||
sa.ForeignKeyConstraint(["operation_id"], ["rule_cleanup_operations.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_rule_cleanup_items_operation", "rule_cleanup_items", ["operation_id"])
|
||||
op.create_index(
|
||||
"uq_cleanup_rule_open", "rule_cleanup_items", ["host_id", "rule_uuid"], unique=True,
|
||||
postgresql_where=sa.text("state IN (" + ", ".join(f"'{s}'" for s in _OPEN_STATES) + ")"),
|
||||
)
|
||||
for name, ddl in _SETTINGS:
|
||||
op.execute(f"ALTER TABLE governance_settings ADD COLUMN IF NOT EXISTS {name} {ddl}")
|
||||
# The default only fills the existing row; the model supplies it for new ones (and the schema comparison
|
||||
# cannot compare a JSON server default).
|
||||
op.execute(f"ALTER TABLE governance_settings ALTER COLUMN {name} DROP DEFAULT")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for name, _ddl in reversed(_SETTINGS):
|
||||
op.execute(f"ALTER TABLE governance_settings DROP COLUMN IF EXISTS {name}")
|
||||
op.execute("DROP TABLE IF EXISTS rule_cleanup_items")
|
||||
op.execute("DROP TABLE IF EXISTS rule_cleanup_operations")
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Stabile Schlüssel (config_key) für Regeln und NAT-Objekte (1.32, Konfiguration als Code).
|
||||
|
||||
Revision ID: 0030
|
||||
Revises: 0029
|
||||
Create Date: 2026-10-23
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0030"
|
||||
down_revision = "0029"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
_TABLES = (
|
||||
("firewall_rules", "r"),
|
||||
("port_forward_rules", "pf"),
|
||||
("one_to_one_rules", "nat1"),
|
||||
("outbound_nat_rules", "out"),
|
||||
("policy_hierarchy_rules", "ph"),
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# IF NOT EXISTS / DO-Block: ein Altbestand ohne Alembic wird zuerst per create_all() angehoben und hat Spalte samt
|
||||
# Constraint dann schon.
|
||||
for table, prefix in _TABLES:
|
||||
op.execute(f"ALTER TABLE {table} ADD COLUMN IF NOT EXISTS config_key varchar(64)")
|
||||
op.execute(
|
||||
f"UPDATE {table} SET config_key = '{prefix}-' || substr(replace(id::text, '-', ''), 1, 12) "
|
||||
"WHERE config_key IS NULL"
|
||||
)
|
||||
op.execute(f"ALTER TABLE {table} ALTER COLUMN config_key SET NOT NULL")
|
||||
op.execute(
|
||||
f"""DO $$ BEGIN
|
||||
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '{table}_config_key_key') THEN
|
||||
ALTER TABLE {table} ADD CONSTRAINT {table}_config_key_key UNIQUE (config_key);
|
||||
END IF;
|
||||
END $$"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for table, _prefix in _TABLES:
|
||||
op.execute(f"ALTER TABLE {table} DROP CONSTRAINT IF EXISTS {table}_config_key_key")
|
||||
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS config_key")
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Rollout-Gate: Erreichbarkeitsproben von der Firewall aus (1.32, B5).
|
||||
|
||||
Revision ID: 0031
|
||||
Revises: 0030
|
||||
Create Date: 2026-10-23
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0031"
|
||||
down_revision = "0030"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# IF NOT EXISTS: a pre-Alembic install is bridged with create_all() first, which may already have the column.
|
||||
op.execute("ALTER TABLE rollout_plans ADD COLUMN IF NOT EXISTS gate_probes json")
|
||||
op.execute("UPDATE rollout_plans SET gate_probes = '[]'::json WHERE gate_probes IS NULL")
|
||||
op.execute("ALTER TABLE rollout_plans ALTER COLUMN gate_probes SET NOT NULL")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE rollout_plans DROP COLUMN IF EXISTS gate_probes")
|
||||
@@ -14,3 +14,4 @@ cryptography==50.0.2
|
||||
alembic==1.20.0
|
||||
PyJWT==2.10.1
|
||||
tzdata==2025.2
|
||||
PyYAML==6.0.3
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Seed data for the configuration-as-code tests: every object kind, hierarchy, placeholders, variables."""
|
||||
from app.database import SessionLocal
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rule import FirewallRule
|
||||
|
||||
|
||||
async def seed_everything() -> dict:
|
||||
"""Two hosts, a device group, a hierarchy (Organisation > Nord), and one object of every kind."""
|
||||
async with SessionLocal() as db:
|
||||
fw1 = OpnsenseHost(name="fw-filiale-1", base_url="https://secret-host.example:8443", api_key="AKIA-SECRET-KEY")
|
||||
fw2 = OpnsenseHost(name="fw-filiale-2", base_url="https://other.example", api_key="k2")
|
||||
group = DeviceGroup(name="Filialen", description="alle Filialen", hosts=[fw1, fw2])
|
||||
org = PolicyHierarchyGroup(name="Organisation", description="")
|
||||
db.add_all([fw1, fw2, group, org])
|
||||
await db.flush()
|
||||
nord = PolicyHierarchyGroup(name="Nord", description="Region Nord", parent_id=org.id)
|
||||
db.add(nord)
|
||||
await db.flush()
|
||||
fw1.policy_hierarchy_group_id = nord.id
|
||||
db.add_all([
|
||||
HostVariable(host_id=fw1.id, name="LAN_NET", value="10.20.30.0/24"),
|
||||
HostVariable(host_id=fw1.id, name="SITE", value="Filiale Nord"),
|
||||
FirewallAlias(name="WEB_SERVERS", type="host", description="DMZ", enabled=True,
|
||||
content="10.20.30.10\n203.0.113.20", groups=[group], policy_hierarchy_group_id=None),
|
||||
FirewallRule(config_key="allow-dns-lan", description="DNS aus {{SITE}}", interface="lan", direction="in",
|
||||
source_net="{{LAN_NET}}", destination_net="WEB_SERVERS", destination_port="53", protocol="TCP/UDP",
|
||||
groups=[group], targets=[]),
|
||||
FirewallRule(config_key="block-all", description="alles sperren", interface="wan", direction="in", action="block",
|
||||
log=True, targets=[fw2], groups=[]),
|
||||
PortForwardRule(config_key="pf-web", description="Web DMZ", interface="wan", destination_net="wanip",
|
||||
destination_port="443", target="10.20.30.10", local_port="443", targets=[fw1], groups=[]),
|
||||
OneToOneRule(config_key="nat1-mail", description="Mail", interface="wan", external="203.0.113.25",
|
||||
source_net="10.20.30.25", destination_net="any", groups=[group], targets=[]),
|
||||
OutboundNatRule(config_key="out-lan", description="LAN raus", interface="wan", protocol="any",
|
||||
source_net="{{LAN_NET}}", destination_net="any", target="wanip", groups=[group], targets=[],
|
||||
policy_hierarchy_group_id=None),
|
||||
PolicyHierarchyRule(config_key="ph-block-bogons", hierarchy_group_id=org.id, rulebase="pre", sequence=0,
|
||||
description="Bogons", interface="wan", direction="in", action="block", source_net="bogons"),
|
||||
PolicyHierarchyRule(config_key="ph-allow-mgmt", hierarchy_group_id=org.id, rulebase="pre", sequence=1,
|
||||
description="Mgmt", interface="lan", direction="in", action="pass"),
|
||||
PolicyHierarchyRule(config_key="ph-post-deny", hierarchy_group_id=nord.id, rulebase="post", sequence=0,
|
||||
description="Standard sperren", interface="lan", direction="in", action="block"),
|
||||
])
|
||||
await db.commit()
|
||||
return {"fw1": "fw-filiale-1", "fw2": "fw-filiale-2"}
|
||||
@@ -6,6 +6,7 @@ Each mutation: {"file": ..., "old": ..., "new": ..., "tests": "pytest selection"
|
||||
The file is patched, the selected tests must FAIL, then the file is restored.
|
||||
"""
|
||||
import json
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
@@ -23,7 +24,7 @@ def main(spec_path: str) -> int:
|
||||
continue
|
||||
try:
|
||||
path.write_text(original.replace(m["old"], m["new"]))
|
||||
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", *m["tests"].split()],
|
||||
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", *shlex.split(m["tests"])],
|
||||
capture_output=True, text=True)
|
||||
finally:
|
||||
path.write_text(original)
|
||||
|
||||
@@ -0,0 +1,380 @@
|
||||
[
|
||||
{
|
||||
"name": "fingerprint not compared at deploy",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if plan_fingerprint(plan) != op.preview_fingerprint:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_deploy_refuses_a_stale_preview_without_writing"
|
||||
},
|
||||
{
|
||||
"name": "management UNKNOWN treated as FALSE (service)",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if check[\"result\"] != \"false\":",
|
||||
"new": " if check[\"result\"] == \"true\":",
|
||||
"tests": "tests/test_rule_cleanup.py::test_rules_that_must_not_be_switched_off_are_refused"
|
||||
},
|
||||
{
|
||||
"name": "management UNKNOWN treated as FALSE (protection)",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " return {\"result\": \"unknown\", \"queries\": queries, \"reasons\": sorted(reasons), \"match\": unknown_at}",
|
||||
"new": " return {\"result\": \"false\", \"queries\": queries, \"reasons\": [], \"match\": None}",
|
||||
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_rules_that_must_not_be_switched_off_are_refused"
|
||||
},
|
||||
{
|
||||
"name": "network source: ranges ignored",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " return Outcome(any(v == net.version and a <= high and b >= low for v, a, b in ranges))",
|
||||
"new": " return Outcome(False)",
|
||||
"tests": "tests/test_rule_cleanup_protection.py"
|
||||
},
|
||||
{
|
||||
"name": "CARP/pfsync protocol not excluded",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " if rule.protos is not None and rule.protos & {CARP_PROTO, PFSYNC_PROTO}:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_protection.py"
|
||||
},
|
||||
{
|
||||
"name": "pfsync interface not checked",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " if pfsync_if == \"\" or (pfsync_if is None and not clustered):",
|
||||
"new": " if True:",
|
||||
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_cluster_pfsync_interface_rules_are_infrastructure"
|
||||
},
|
||||
{
|
||||
"name": "no management sources not locked",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if not sources:\n raise api_error(409, ErrorCode.BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN)\n clustered",
|
||||
"new": " if False:\n raise api_error(409, ErrorCode.BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN)\n clustered",
|
||||
"tests": "tests/test_rule_cleanup.py::test_without_management_sources_everything_is_locked"
|
||||
},
|
||||
{
|
||||
"name": "unused sure below 30 days",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if unused_days < MIN_SURE_UNUSED_DAYS:",
|
||||
"new": " if unused_days < 7:",
|
||||
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_short_window_and_external_change_cap_at_probable"
|
||||
},
|
||||
{
|
||||
"name": "unused block rule not capped",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if rule.action in (\"block\", \"reject\"):",
|
||||
"new": " if rule.action in (\"reject\",):",
|
||||
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_probable_needs_an_explicit_acknowledgement_per_rule"
|
||||
},
|
||||
{
|
||||
"name": "external change not capped",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if external and best is not None:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_protection.py tests/test_rule_cleanup.py::test_short_window_and_external_change_cap_at_probable"
|
||||
},
|
||||
{
|
||||
"name": "probable without acknowledgement",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if level == \"probable\" and given not in acknowledge:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_probable_needs_an_explicit_acknowledgement_per_rule"
|
||||
},
|
||||
{
|
||||
"name": "unknown confidence selectable",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if level == \"unknown\":\n refuse(\"confidence\")",
|
||||
"new": " if False:\n refuse(\"confidence\")",
|
||||
"tests": "tests/test_rule_cleanup.py::test_managed_legacy_and_unknown_confidence_are_refused"
|
||||
},
|
||||
{
|
||||
"name": "cluster usage merge removed",
|
||||
"file": "app/services/rule_analysis/service.py",
|
||||
"old": " return rule_hit_service.merge_cluster_hits(own, [await _hits(db, other) for other in others]), own",
|
||||
"new": " return own, own",
|
||||
"tests": "tests/test_rule_cleanup.py::test_cluster_usage_is_judged_over_all_nodes"
|
||||
},
|
||||
{
|
||||
"name": "write order master first",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " return sorted(nodes, key=lambda n: (n.role == \"MASTER\", n.host.name))",
|
||||
"new": " return sorted(nodes, key=lambda n: (n.role != \"MASTER\", n.host.name))",
|
||||
"tests": "tests/test_rule_cleanup.py::test_cluster_writes_backup_first_and_reverts_master_first"
|
||||
},
|
||||
{
|
||||
"name": "revert order backup first",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " for item in sorted(items, key=lambda i: (i.carp_role != \"MASTER\", str(i.host_id))):",
|
||||
"new": " for item in sorted(items, key=lambda i: (i.carp_role == \"MASTER\", str(i.host_id))):",
|
||||
"tests": "tests/test_rule_cleanup.py::test_cluster_writes_backup_first_and_reverts_master_first"
|
||||
},
|
||||
{
|
||||
"name": "no apply after a partial failure",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if any(i.state in (\"disabled\", \"unknown\") for i in node_items):\n # Also after an error",
|
||||
"new": " if not stop and any(i.state in (\"disabled\", \"unknown\") for i in node_items):\n # Also after an error",
|
||||
"tests": "tests/test_rule_cleanup.py::test_refused_toggle_stops_writing_but_applies_what_is_switched"
|
||||
},
|
||||
{
|
||||
"name": "write continues after a refusal",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " item.state, item.message = \"failed\", str(exc)[:1000]\n _audit(db, op, item, node.host, action=op.kind, status=\"error\", message=item.message)\n stop = True",
|
||||
"new": " item.state, item.message = \"failed\", str(exc)[:1000]\n _audit(db, op, item, node.host, action=op.kind, status=\"error\", message=item.message)",
|
||||
"tests": "tests/test_rule_cleanup.py::test_refused_toggle_stops_writing_but_applies_what_is_switched"
|
||||
},
|
||||
{
|
||||
"name": "unclear toggle counted as done",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " item.state = \"disabled\" if seen is target else (\"failed\" if seen is not None else \"unknown\")",
|
||||
"new": " item.state = \"disabled\"",
|
||||
"tests": "tests/test_rule_cleanup.py::test_unclear_toggle_is_settled_by_reading"
|
||||
},
|
||||
{
|
||||
"name": "confirm with unreachable firewall",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " except OpnsenseAPIError as exc:\n raise api_error(502, ErrorCode.BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR, host=member.name) from exc\n if not rule_hit_service.looks_complete(search):",
|
||||
"new": " except OpnsenseAPIError as exc:\n continue\n if not rule_hit_service.looks_complete(search):",
|
||||
"tests": "tests/test_rule_cleanup.py::test_confirm_reads_the_firewall_and_refuses_when_unreachable_or_different"
|
||||
},
|
||||
{
|
||||
"name": "confirm ignores re-enabled rule",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if row is None or _truthy(row.get(\"enabled\", \"1\")) is not target \\\n or _content_fp",
|
||||
"new": " if row is None \\\n or _content_fp",
|
||||
"tests": "tests/test_rule_cleanup.py::test_confirm_reads_the_firewall_and_refuses_when_unreachable_or_different"
|
||||
},
|
||||
{
|
||||
"name": "confirm after the deadline",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " RuleCleanupOperation.revert_deadline > now).values(\n status=\"confirmed\"",
|
||||
"new": " RuleCleanupOperation.revert_deadline > now - timedelta(days=1)).values(\n status=\"confirmed\"",
|
||||
"tests": "tests/test_rule_cleanup.py::test_deadline_passing_while_confirm_reads_the_firewall_refuses tests/test_rule_cleanup.py::test_confirm_and_sweeper_at_the_deadline_decide_once"
|
||||
},
|
||||
{
|
||||
"name": "sweeper does not revert expired",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " or_(and_(RuleCleanupOperation.status.in_(SWEEP_STATUSES), RuleCleanupOperation.revert_deadline <= now),",
|
||||
"new": " or_(and_(RuleCleanupOperation.status.in_(SWEEP_STATUSES), RuleCleanupOperation.revert_deadline <= now - timedelta(days=1)),",
|
||||
"tests": "tests/test_rule_cleanup.py::test_confirm_after_the_deadline_is_refused_and_the_sweeper_reverts tests/test_rule_cleanup.py::test_failed_apply_keeps_the_deadline_and_the_sweeper_reverts"
|
||||
},
|
||||
{
|
||||
"name": "sweeper retries revert_failed",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": "SWEEP_STATUSES = (\"awaiting_confirm\", \"partial\", \"apply_failed\")",
|
||||
"new": "SWEEP_STATUSES = (\"awaiting_confirm\", \"partial\", \"apply_failed\", \"revert_failed\")",
|
||||
"tests": "tests/test_rule_cleanup.py::test_revert_now_and_failed_revert_is_not_retried_automatically"
|
||||
},
|
||||
{
|
||||
"name": "heartbeat not checked",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if not _stale(op, len(items), nodes, now):\n # alive",
|
||||
"new": " if False:\n # alive",
|
||||
"tests": "tests/test_rule_cleanup.py::test_sweeper_skips_a_live_run_and_takes_over_an_abandoned_one"
|
||||
},
|
||||
{
|
||||
"name": "delete before the waiting time",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if now < due:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_delete_needs_the_waiting_time_an_unchanged_rule_and_no_external_change"
|
||||
},
|
||||
{
|
||||
"name": "delete of a changed rule",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if _content_fp(row, False) != _content_fp(item.row_before or {}, False):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_delete_needs_the_waiting_time_an_unchanged_rule_and_no_external_change"
|
||||
},
|
||||
{
|
||||
"name": "snapshot not committed before delRule",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " item.state, item.changed_at = \"deleting\", _now()\n await _commit(db) # heartbeat + snapshot + intent, durable before delRule\n",
|
||||
"new": " item.state, item.changed_at = \"deleting\", _now()\n",
|
||||
"tests": "tests/test_rule_cleanup.py::test_delete_commits_the_snapshot_before_delrule"
|
||||
},
|
||||
{
|
||||
"name": "unclear delete counted as deleted",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " return \"failed\" # still there",
|
||||
"new": " return \"deleted\"",
|
||||
"tests": "tests/test_rule_cleanup.py::test_unclear_delete_is_settled_with_getrule"
|
||||
},
|
||||
{
|
||||
"name": "restore anchor ignored",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " await client.move_rule_before(member, new_uuid, anchor)",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup.py::test_restore_adds_disabled_at_the_anchor_with_a_new_uuid"
|
||||
},
|
||||
{
|
||||
"name": "restore position not verified",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " return i > 0 and position[i - 1] == new_uuid.lower()",
|
||||
"new": " return True",
|
||||
"tests": "tests/test_rule_cleanup.py::test_restore_with_a_refused_move_is_verified_by_reading_and_only_warns"
|
||||
},
|
||||
{
|
||||
"name": "R1 unassigned ingress not checked",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " return [(i, None) for i in _interfaces(ctx)] + [(UNASSIGNED, virtual)]",
|
||||
"new": " return [(i, None) for i in _interfaces(ctx)]",
|
||||
"tests": "tests/test_rule_cleanup_protection.py"
|
||||
},
|
||||
{
|
||||
"name": "R1 builtin group members treated as known",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " elif name in BUILTIN_GROUP_DEVICES or ctx.groups is None or name not in ctx.groups:",
|
||||
"new": " elif ctx.groups is None or name not in ctx.groups:",
|
||||
"tests": "tests/test_rule_cleanup_protection.py"
|
||||
},
|
||||
{
|
||||
"name": "R2 sweeper takes over a live run past its deadline",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if not _stale(op, len(items), nodes, now):",
|
||||
"new": " if not (op.revert_deadline is not None and op.revert_deadline <= now) and not _stale(op, len(items), nodes, now):",
|
||||
"tests": "tests/test_rule_cleanup.py::test_sweeper_leaves_a_live_run_alone_even_past_its_deadline"
|
||||
},
|
||||
{
|
||||
"name": "R2 heartbeat before a write not conditional",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " RuleCleanupOperation.id == op.id, RuleCleanupOperation.status == expect).values(heartbeat_at=now)",
|
||||
"new": " RuleCleanupOperation.id == op.id).values(heartbeat_at=now)",
|
||||
"tests": "tests/test_rule_cleanup.py::test_a_run_taken_over_by_another_backend_writes_nothing_more"
|
||||
},
|
||||
{
|
||||
"name": "R3 extra management ports ignored",
|
||||
"file": "app/services/rule_analysis/protection.py",
|
||||
"old": " for port in (*MANAGEMENT_PORTS, *extra_ports):",
|
||||
"new": " for port in MANAGEMENT_PORTS:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_extra_management_ports_protect_rules_on_them"
|
||||
},
|
||||
{
|
||||
"name": "R3 management ports shown to non-admins",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " cleanup_management_ports=ports if user.role == \"admin\" else [],",
|
||||
"new": " cleanup_management_ports=ports,",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_management_ports_are_validated_normalised_audited_and_hidden"
|
||||
},
|
||||
{
|
||||
"name": "R3 port range not validated",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if isinstance(raw, bool) or not text.isdigit() or not 1 <= int(text) <= 65535:",
|
||||
"new": " if not text.isdigit():",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_management_ports_are_validated_normalised_audited_and_hidden"
|
||||
},
|
||||
{
|
||||
"name": "R4 any host may act on any operation",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if op.host_id != host.id and not (cluster and host.cluster_name and op.host_id in {h.id for h in await _fleet(db, host)}):",
|
||||
"new": " if op.host_id != host.id and not cluster:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_cluster_peer_may_confirm_and_read_but_strangers_may_not"
|
||||
},
|
||||
{
|
||||
"name": "R4 peers refused",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if op.host_id != host.id and not (cluster and host.cluster_name and op.host_id in {h.id for h in await _fleet(db, host)}):",
|
||||
"new": " if op.host_id != host.id:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_cluster_peer_may_confirm_and_read_but_strangers_may_not"
|
||||
},
|
||||
{
|
||||
"name": "R5 half-deleted node not skipped",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if clustered and kind == \"delete\" and await _already_deleted(db, node.host.id, u):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_finished"
|
||||
},
|
||||
{
|
||||
"name": "R5 failed delete counts as basis",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if not (o.kind == \"delete\" and i.state in (\"failed\", \"skipped\"))]",
|
||||
"new": " ]",
|
||||
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_finished"
|
||||
},
|
||||
{
|
||||
"name": "R5 nothing left still planned",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if not any(i[\"given\"] == given for n in plan_nodes for i in n[\"items\"]):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_finished"
|
||||
},
|
||||
{
|
||||
"name": "R5 restore needs all siblings deleted",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if s.state == \"deleted\" and not s.restored_uuid]",
|
||||
"new": " ]\n if any(s.state != \"deleted\" or s.restored_uuid for s in siblings):\n raise api_error(409, ErrorCode.BEREINIGUNG_AKTION_NICHT_MOEGLICH, status=\"partial\")",
|
||||
"tests": "tests/test_rule_cleanup.py::test_half_done_cluster_delete_can_be_restored_where_it_was_deleted"
|
||||
},
|
||||
{
|
||||
"name": "R7 enabling a block rule without acknowledgement",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if rule.action != \"pass\" and item[\"protection\"][\"result\"] != \"false\":",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_switching_a_block_rule_back_on_needs_an_acknowledgement_unless_proven_harmless"
|
||||
},
|
||||
{
|
||||
"name": "R7 unknown protection counted as harmless",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if rule.action != \"pass\" and item[\"protection\"][\"result\"] != \"false\":",
|
||||
"new": " if rule.action != \"pass\" and item[\"protection\"][\"result\"] == \"true\":",
|
||||
"tests": "tests/test_rule_cleanup.py::test_switching_a_block_rule_back_on_needs_an_acknowledgement_unless_proven_harmless"
|
||||
},
|
||||
{
|
||||
"name": "R8 router does not redact for non-admins",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " if not _full_view(request):\n operations = [service.redact(op) for op in operations]",
|
||||
"new": " if False:\n operations = [service.redact(op) for op in operations]",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
|
||||
},
|
||||
{
|
||||
"name": "R8 preview answer not redacted",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " return result if _full_view(request) else service.redact(result)",
|
||||
"new": " return result",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
|
||||
},
|
||||
{
|
||||
"name": "R8 audit admin_only not stripped",
|
||||
"file": "app/routers/audit.py",
|
||||
"old": " if not full_view(request):",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
|
||||
},
|
||||
{
|
||||
"name": "R8 read token of an admin sees everything",
|
||||
"file": "app/routers/audit.py",
|
||||
"old": " return getattr(user, \"role\", None) == \"admin\" and (token is None or token.scope == \"admin\")",
|
||||
"new": " return getattr(user, \"role\", None) == \"admin\"",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
|
||||
},
|
||||
{
|
||||
"name": "R8 base_url kept in plan",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " out[\"nodes\"] = [{**{k: v for k, v in n.items() if k != \"base_url\"},",
|
||||
"new": " out[\"nodes\"] = [{**{k: v for k, v in n.items()},",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
|
||||
},
|
||||
{
|
||||
"name": "R8 protection match kept",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " return {k: v for k, v in check.items() if k != \"match\"}",
|
||||
"new": " return check",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_sources_and_firewall_addresses_are_for_admins_only tests/test_rule_cleanup_protection.py::test_redacted_plan_names_no_source_port_or_firewall_address"
|
||||
},
|
||||
{
|
||||
"name": "R10 stored restored_uuid ignored",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if item.restored_uuid:\n try:\n await client.get_rule(member, item.restored_uuid)",
|
||||
"new": " if False:\n try:\n await client.get_rule(member, item.restored_uuid)",
|
||||
"tests": "tests/test_rule_cleanup.py::test_unclear_restore_uses_the_stored_uuid_before_searching_by_content"
|
||||
},
|
||||
{
|
||||
"name": "R10 twins resolved to the first",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if len(found) == 1:\n return \"found\", found[0]",
|
||||
"new": " if found:\n return \"found\", found[0]",
|
||||
"tests": "tests/test_rule_cleanup.py::test_unclear_restore_with_twins_and_no_stored_uuid_stays_unknown"
|
||||
},
|
||||
{
|
||||
"name": "R11 any acknowledgement text version accepted",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " if str(ack_text_version) != ACK_TEXT_VERSION:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup.py::test_an_outdated_acknowledgement_text_is_refused"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,373 @@
|
||||
[
|
||||
{
|
||||
"name": "middleware: confirm not session-only",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": "|^/api/hosts/[^/]+/rule-cleanup/[^/]+/confirm/?$\")",
|
||||
"new": "\")",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "middleware: retry not admin-only",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": "(deploy|retry)|items",
|
||||
"new": "(deploy)|items",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "middleware: restore not admin-only",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": "|items/[^/]+/restore)/?$\")",
|
||||
"new": ")/?$\")",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "middleware: deploy-scope token passes admin-only cleanup routes",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": "if scope == \"deploy\" and (_ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path)):",
|
||||
"new": "if scope == \"deploy\" and (_ADMIN_ONLY_WRITES.match(path)):",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "middleware: operator passes admin-only cleanup routes",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": " _ADMIN_ONLY_WRITES.match(path) or _ADMIN_ONLY_CLEANUP.match(path))",
|
||||
"new": " _ADMIN_ONLY_WRITES.match(path))",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "middleware: revert not a rollout (operators_may_deploy ignored)",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": "r\"|rule-cleanup/[^/]+/(revert|retry)\"",
|
||||
"new": "r\"|rule-cleanup/[^/]+/(retry)\"",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "middleware: retry not a rollout",
|
||||
"file": "app/services/api_token_service.py",
|
||||
"old": "r\"|rule-cleanup/[^/]+/(revert|retry)\"",
|
||||
"new": "r\"|rule-cleanup/[^/]+/(revert)\"",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: deploy only needs a writer",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " _admin_only(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.deploy(",
|
||||
"new": " _writer(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.deploy(",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: retry only needs a writer",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " _admin_only(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.retry(",
|
||||
"new": " _writer(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.retry(",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: restore only needs a writer",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " _admin_only(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.restore_item(",
|
||||
"new": " _writer(user, request)\n host = await _host(db, host_id)\n return await _one(db, await service.restore_item(",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: confirm only needs a writer (tokens may confirm)",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " _session_only(user, request)\n",
|
||||
"new": " _writer(user, request)\n",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: revert needs no deployer",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " await _deployer(user, request, db)\n",
|
||||
"new": " _writer(user, request)\n",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: admin role not checked",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " if user.role != \"admin\":\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n token = _token(request)",
|
||||
"new": " if False:\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n token = _token(request)",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: admin token scope not checked",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " if token is not None and token.scope != \"admin\":",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: session-only token check removed",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " if token is not None:\n raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)",
|
||||
"new": " if False:\n raise api_error(403, ErrorCode.API_TOKEN_SCOPE_FORBIDDEN, scope=token.scope)",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: operator revert ignores operators_may_deploy",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": "if user.role == \"operator\" and (await governance.get_settings(db)).operators_may_deploy:",
|
||||
"new": "if user.role == \"operator\":",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: read-only account may write",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " if user.role == \"read_only\":\n raise api_error(403, ErrorCode.READ_ONLY_ACCESS)",
|
||||
"new": " if False:\n raise api_error(403, ErrorCode.READ_ONLY_ACCESS)",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: read-scope token may write",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " if token is not None and token.scope == \"read\":",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: preview needs no writer",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": " user = await resolve_user(request, db)\n _writer(user, request)\n host = await _host(db, host_id)\n kind, uuids",
|
||||
"new": " user = await resolve_user(request, db)\n host = await _host(db, host_id)\n kind, uuids",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'every_route_for_every_caller or pattern_table or governance_settings_are_admin'"
|
||||
},
|
||||
{
|
||||
"name": "router: actor is not the account name",
|
||||
"file": "app/routers/rule_cleanup.py",
|
||||
"old": "actor=user.username, actor_id=user.id, kind=kind",
|
||||
"new": "actor='x', actor_id=user.id, kind=kind",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'preview_belongs_to_its_actor or shapes_and_audit'"
|
||||
},
|
||||
{
|
||||
"name": "settings: any role may write",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if user.role != \"admin\":\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n row = await get_settings(db)\n changes",
|
||||
"new": " if False:\n raise api_error(403, ErrorCode.ADMIN_REQUIRED)\n row = await get_settings(db)\n changes",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k only_an_admin_writes"
|
||||
},
|
||||
{
|
||||
"name": "settings: whole address space accepted",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if net.prefixlen == 0:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: host names accepted",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " raise _bad(\"cleanup_management_sources\", \"not_an_address\") from None",
|
||||
"new": " continue",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: unlimited list",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if len(values) > MAX_MANAGEMENT_SOURCES:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: sources not canonical",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " entry = str(net.network_address) if net.prefixlen == net.max_prefixlen else str(net)",
|
||||
"new": " entry = text",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: sources shown to everybody",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": "cleanup_management_sources=sources if user.role == \"admin\" else [],",
|
||||
"new": "cleanup_management_sources=sources,",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: change not audited",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if changes:\n db.add(audit_service.make_event(",
|
||||
"new": " if False:\n db.add(audit_service.make_event(",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: max rules unbounded",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if not 1 <= payload.cleanup_max_rules <= MAX_CLEANUP_RULES:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "settings: min days unbounded",
|
||||
"file": "app/routers/governance.py",
|
||||
"old": " if not 1 <= payload.cleanup_delete_min_days <= MAX_DELETE_MIN_DAYS:",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'cleanup_settings or management_sources or only_see_whether'"
|
||||
},
|
||||
{
|
||||
"name": "lock: ensure_no_maintenance ignores a cleanup",
|
||||
"file": "app/services/host_maintenance_service.py",
|
||||
"old": " if cleanup and ids:\n await ensure_no_open_cleanup(db, hosts)",
|
||||
"new": " if False:\n await ensure_no_open_cleanup(db, hosts)",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: ensure_no_open_cleanup never raises",
|
||||
"file": "app/services/host_maintenance_service.py",
|
||||
"old": " if ids and await has_open_cleanup(db, ids):\n raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)",
|
||||
"new": " if False:\n raise api_error(409, ErrorCode.OPEN_CLEANUP_BLOCKS_DEPLOY)",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: maintenance inspection ignores a cleanup",
|
||||
"file": "app/services/host_maintenance_service.py",
|
||||
"old": " if await has_open_cleanup(db, fleet_ids):\n blocked.append(",
|
||||
"new": " if False:\n blocked.append(",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: host delete ignores a cleanup",
|
||||
"file": "app/routers/hosts.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [host])\n # Both deployment tables FK",
|
||||
"new": " # Both deployment tables FK",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: host re-addressing ignores a cleanup",
|
||||
"file": "app/routers/hosts.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [host])\n\n if payload.name is not None:",
|
||||
"new": " pass\n\n if payload.name is not None:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: rule deletion ignores a cleanup",
|
||||
"file": "app/services/rule_deploy_service.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: hierarchy rule deletion ignores a cleanup",
|
||||
"file": "app/services/policy_hierarchy_deploy_service.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [h for h in [await db.get(OpnsenseHost, d.host_id) for d in deployments] if h])",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: hierarchy reordering ignores a cleanup",
|
||||
"file": "app/services/policy_hierarchy_deploy_service.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [h for h in hosts.values() if h.enabled])",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: removing a rule target ignores a cleanup",
|
||||
"file": "app/routers/rules.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: removing a hierarchy target ignores a cleanup",
|
||||
"file": "app/routers/policy_hierarchy.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: adoption ignores a cleanup",
|
||||
"file": "app/routers/rules.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [host])\n try:\n rule = await rule_adopt_service.adopt(",
|
||||
"new": " try:\n rule = await rule_adopt_service.adopt(",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or covers_the_cluster or host_delete_and_readdressing or rule_deletion_and_removal or hierarchy_rule_deletion or former_hierarchy or adopting_a_rule or confirmed_cleanup_blocks_nothing'"
|
||||
},
|
||||
{
|
||||
"name": "lock: cleanup service trips over itself",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": "await ensure_no_maintenance(db, [host], cleanup=False)",
|
||||
"new": "await ensure_no_maintenance(db, [host])",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'open_cleanup_blocks_other or preview_input_forms or shapes_and_audit or other_open_work'"
|
||||
},
|
||||
{
|
||||
"name": "service: lost revert claim hits an expired row",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": "\n await db.refresh(op) # _revert rolled back when it lost the claim: the status is expired\n raise api_error(409, ErrorCode.BEREINIGUNG_AKTION_NICHT_MOEGLICH, status=op.status)\n await db.commit()",
|
||||
"new": "\n raise api_error(409, ErrorCode.BEREINIGUNG_AKTION_NICHT_MOEGLICH, status=op.status)\n await db.commit()",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k revert_is_for_admins"
|
||||
},
|
||||
{
|
||||
"name": "service: lost retry claim hits an expired row",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": "\n await db.refresh(op) # _revert rolled back when it lost the claim: the status is expired",
|
||||
"new": "",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k retry_after_a_failed"
|
||||
},
|
||||
{
|
||||
"name": "notify: failed revert not reported",
|
||||
"file": "app/services/notification_checks.py",
|
||||
"old": " if op.status == \"revert_failed\":",
|
||||
"new": " if False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
|
||||
},
|
||||
{
|
||||
"name": "notify: partial run not reported",
|
||||
"file": "app/services/notification_checks.py",
|
||||
"old": " elif op.status in (\"partial\", \"apply_failed\"):",
|
||||
"new": " elif False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
|
||||
},
|
||||
{
|
||||
"name": "notify: unclear delete not reported",
|
||||
"file": "app/services/notification_checks.py",
|
||||
"old": " elif op.kind in (\"delete\", \"restore\"):",
|
||||
"new": " elif False:",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
|
||||
},
|
||||
{
|
||||
"name": "notify: every item of a failed revert is reported",
|
||||
"file": "app/services/notification_checks.py",
|
||||
"old": " stuck = [i for i in items if i.state in _CLEANUP_STILL_CHANGED]",
|
||||
"new": " stuck = list(items)",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
|
||||
},
|
||||
{
|
||||
"name": "notify: cleanup problems not part of the revert alert",
|
||||
"file": "app/services/notification_checks.py",
|
||||
"old": " problems.update(await _cleanup_problems(db))",
|
||||
"new": " pass",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k 'failed_reverts_and_unclear or unclear_delete_is_reported'"
|
||||
},
|
||||
{
|
||||
"name": "preview writes no own audit entry",
|
||||
"file": "app/services/rule_cleanup_service.py",
|
||||
"old": " db.add(audit_service.make_event(\n kind=\"rule_cleanup\", action=\"preview\"",
|
||||
"new": " (lambda *a, **k: None)(audit_service.make_event(\n kind=\"rule_cleanup\", action=\"preview\"",
|
||||
"tests": "tests/test_rule_cleanup_api.py -k shapes_and_audit"
|
||||
},
|
||||
{
|
||||
"name": "R9 alias undeploy without cleanup lock",
|
||||
"file": "app/services/alias_deploy_service.py",
|
||||
"old": " await ensure_no_open_cleanup_on(db, AliasDeployment, AliasDeployment.alias_id == alias.id)\n",
|
||||
"new": "",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_alias_delete_and_target_removal_wait_for_the_cleanup"
|
||||
},
|
||||
{
|
||||
"name": "R9 alias target removal without cleanup lock",
|
||||
"file": "app/routers/aliases.py",
|
||||
"old": " await ensure_no_open_cleanup(db, [deployment.host]) # a rule cleanup (1.31) owns the firewall: no other writes\n",
|
||||
"new": "",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_alias_delete_and_target_removal_wait_for_the_cleanup"
|
||||
},
|
||||
{
|
||||
"name": "R9 wireguard undeploy without cleanup lock",
|
||||
"file": "app/services/wireguard_deploy_service.py",
|
||||
"old": " await ensure_no_open_cleanup_on(db, WireguardTunnelDeployment, WireguardTunnelDeployment.tunnel_id == tunnel.id)\n",
|
||||
"new": "",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_every_undeploy_and_target_removal_checks_for_an_open_cleanup_first"
|
||||
},
|
||||
{
|
||||
"name": "R9 helper checks nothing",
|
||||
"file": "app/services/host_maintenance_service.py",
|
||||
"old": " ids = set((await db.scalars(select(model.host_id).where(condition))).all())",
|
||||
"new": " ids = set()",
|
||||
"tests": "tests/test_rule_cleanup_api.py::test_alias_delete_and_target_removal_wait_for_the_cleanup"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,241 @@
|
||||
"""A small stateful OPNsense for the rule-cleanup tests (respx side effects).
|
||||
|
||||
It keeps its rules in memory and answers searchRule, getRule, toggleRule, delRule, addRule, moveRuleBefore, apply,
|
||||
ruleStats, pfStatistics (generated from the enabled rules, like pf after an apply), interfacesInfo, groups, aliases,
|
||||
port forwards, getVipStatus and core/hasync/get in the shapes seen live (OPNsense 26.7, see opnsense_client.py).
|
||||
``fail`` injects errors per call: {("toggle", uuid): httpx.ReadTimeout | httpx.Response | callable}.
|
||||
Addresses: documentation/private ranges only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import re
|
||||
import uuid as uuid_lib
|
||||
|
||||
import httpx
|
||||
|
||||
R_BASE = "aaaaaaaa-0000-4000-8000-000000000100"
|
||||
R_DUP = "aaaaaaaa-0000-4000-8000-000000000200"
|
||||
R_UNUSED = "aaaaaaaa-0000-4000-8000-000000000300"
|
||||
R_BLOCK = "aaaaaaaa-0000-4000-8000-000000000400"
|
||||
R_MGMT = "aaaaaaaa-0000-4000-8000-000000000500"
|
||||
R_CARP = "aaaaaaaa-0000-4000-8000-000000000600"
|
||||
R_ALIAS = "aaaaaaaa-0000-4000-8000-000000000700"
|
||||
R_TAIL = "aaaaaaaa-0000-4000-8000-000000000800"
|
||||
R_LEGACY = "0123456789abcdef0123456789abcdef"
|
||||
AUTO = "02f4bab031b57d1e30553ce08e0ec131"
|
||||
MGMT_SOURCE = "203.0.113.10"
|
||||
DEVICES = {"wan": "vtnet0", "lan": "vtnet1"}
|
||||
|
||||
|
||||
def rule(uuid, seq, *, action="pass", interface="lan", proto="TCP", src="any", dst="any", dport="", enabled="1",
|
||||
description=None, **extra) -> dict:
|
||||
row = {
|
||||
"uuid": uuid, "sort_order": f"400000.{seq:07d}", "prio_group": "400000", "sequence": str(seq),
|
||||
"enabled": enabled, "action": action, "quick": "1", "interface": interface, "interfacenot": "0",
|
||||
"direction": "in", "ipprotocol": "inet", "protocol": proto, "source_net": src, "source_not": "0",
|
||||
"source_port": "", "destination_net": dst, "destination_not": "0", "destination_port": dport,
|
||||
"gateway": "", "sched": "", "statetype": "keep", "tagged": "", "log": "0", "categories": "",
|
||||
"description": description or f"cleanup-{seq}",
|
||||
}
|
||||
row.update(extra)
|
||||
return row
|
||||
|
||||
|
||||
def default_rules() -> list[dict]:
|
||||
return [
|
||||
{"uuid": AUTO, "sort_order": "000001.1000001", "legacy": True, "is_automatic": True, "enabled": "1",
|
||||
"action": "block", "quick": False, "direction": "in", "ipprotocol": "inet46", "protocol": "",
|
||||
"source_net": "", "source_port": "", "destination_net": "", "destination_port": "",
|
||||
"description": "Default deny / state violation rule"},
|
||||
rule(R_BASE, 100, src="10.20.30.0/24", dport="9000", description="base 9000"),
|
||||
rule(R_DUP, 200, src="10.20.30.0/24", dport="9000", description="dup 9000"),
|
||||
rule(R_UNUSED, 300, src="10.20.30.0/24", dport="8081", description="unused 8081"),
|
||||
rule(R_BLOCK, 400, action="block", proto="UDP", dport="9999", description="block 9999"),
|
||||
rule(R_MGMT, 500, interface="wan", src="203.0.113.0/24", dport="443", description="mgmt 443"),
|
||||
rule(R_CARP, 600, proto="CARP", description="carp"),
|
||||
rule(R_ALIAS, 700, interface="wan", src="tbl", dport="443", description="alias 443"),
|
||||
rule(R_TAIL, 800, src="10.20.30.7", dport="7000", description="tail 7000"),
|
||||
]
|
||||
|
||||
|
||||
class FakeFirewall:
|
||||
def __init__(self, base: str, *, role: str | None = None, rules: list[dict] | None = None,
|
||||
pfsync: str = "", peer: str = "", lan_addr: str = "10.20.30.2", wan_addr: str = "203.0.113.57",
|
||||
log: list | None = None):
|
||||
self.base = base
|
||||
self.log = log # shared across firewalls: the order of calls over a cluster
|
||||
self.role = role
|
||||
self.rules: list[dict] = copy.deepcopy(rules if rules is not None else default_rules())
|
||||
self.pfsync, self.peer = pfsync, peer
|
||||
self.lan_addr, self.wan_addr = lan_addr, wan_addr
|
||||
self.fail: dict = {}
|
||||
self.calls: list[tuple[str, str]] = []
|
||||
self.applied = 0
|
||||
self.down = False
|
||||
self.on_call = None # callable(kind, uuid) run before a write is answered (e.g. look at the database)
|
||||
|
||||
# ── state helpers ──
|
||||
def get(self, uuid):
|
||||
return next((r for r in self.rules if r["uuid"].lower() == uuid.lower()), None)
|
||||
|
||||
def _inject(self, key):
|
||||
effect = self.fail.get(key)
|
||||
if effect is None:
|
||||
return None
|
||||
if isinstance(effect, list):
|
||||
effect = effect.pop(0) if effect else None
|
||||
if effect is None:
|
||||
return None
|
||||
if isinstance(effect, type) and issubclass(effect, Exception):
|
||||
raise effect("injected")
|
||||
if isinstance(effect, Exception):
|
||||
raise effect
|
||||
if callable(effect) and not isinstance(effect, httpx.Response):
|
||||
return effect()
|
||||
return effect
|
||||
|
||||
def pf_lines(self) -> dict:
|
||||
lines = {"@0 scrub in all fragment reassemble": {}}
|
||||
n = 1
|
||||
for r in sorted(self.rules, key=lambda r: r["sort_order"]):
|
||||
if str(r.get("enabled")) != "1":
|
||||
continue
|
||||
if r.get("is_automatic"):
|
||||
lines[f'@{n} block drop in log inet all label "{r["uuid"]}"'] = {"evaluations": 0}
|
||||
else:
|
||||
words = ["block" if r["action"] == "block" else "pass", "in", "quick", f"on {DEVICES.get(r['interface'], r['interface'])}", "inet"]
|
||||
if r["protocol"] not in ("", "any"):
|
||||
words.append(f"proto {r['protocol'].lower()}")
|
||||
words.append(f'all label "{r["uuid"]}"')
|
||||
lines[f"@{n} " + " ".join(words)] = {"evaluations": 0, "packets": 0}
|
||||
n += 1
|
||||
return {"rules": {"filter rules": lines}}
|
||||
|
||||
def model(self, row: dict) -> dict:
|
||||
"""getRule shape: select fields as option dicts, no uuid key."""
|
||||
out = {k: v for k, v in row.items() if k not in ("uuid", "is_automatic", "legacy")}
|
||||
out["action"] = {a: {"value": a, "selected": 1 if row["action"] == a else 0} for a in ("pass", "block", "reject")}
|
||||
out["interface"] = {i: {"value": i.upper(), "selected": 1 if i in str(row["interface"]).split(",") else 0} for i in ("lan", "wan")}
|
||||
out["categories"] = []
|
||||
out["audit"] = {"created": "x", "updated": "y"}
|
||||
return out
|
||||
|
||||
# ── respx wiring ──
|
||||
def mount(self, respx_mock):
|
||||
respx_mock.route(url__startswith=self.base).mock(side_effect=self.handle)
|
||||
return self
|
||||
|
||||
def handle(self, request: httpx.Request) -> httpx.Response:
|
||||
path = request.url.path.removeprefix("/api")
|
||||
method = request.method
|
||||
self.calls.append((method, path))
|
||||
if self.log is not None:
|
||||
self.log.append((self.base, method, path))
|
||||
if self.down:
|
||||
raise httpx.ConnectError("down")
|
||||
if method == "GET" and path == "/diagnostics/interface/getVipStatus":
|
||||
rows = [] if self.role is None else [{"interface": "LAN", "vhid": "10", "status": self.role}]
|
||||
return httpx.Response(200, json={"rows": rows, "carp": {"allow": "1", "maintenancemode": False}})
|
||||
if method == "GET" and path == "/firewall/filter/searchRule":
|
||||
got = self._inject(("search", ""))
|
||||
if got is not None:
|
||||
return got
|
||||
return httpx.Response(200, json={"rows": copy.deepcopy(sorted(self.rules, key=lambda r: r["sort_order"]))})
|
||||
if method == "GET" and path == "/diagnostics/firewall/pfStatistics/rules":
|
||||
return httpx.Response(200, json=self.pf_lines())
|
||||
if method == "GET" and path == "/firewall/filter_util/ruleStats":
|
||||
return httpx.Response(200, json={"status": "ok", "stats": {}})
|
||||
if method == "GET" and path == "/interfaces/overview/interfacesInfo":
|
||||
return httpx.Response(200, json={"rows": [
|
||||
{"identifier": "wan", "device": "vtnet0", "description": "WAN", "ipv4": [{"ipaddr": f"{self.wan_addr}/24"}],
|
||||
"ipv6": [], "gateways": ["203.0.113.1"], "routes": ["default"]},
|
||||
{"identifier": "lan", "device": "vtnet1", "description": "LAN",
|
||||
"ipv4": [{"ipaddr": f"{self.lan_addr}/24"}, {"ipaddr": "10.20.30.1/24", "vhid": "10"}],
|
||||
"ipv6": [], "gateways": [], "routes": ["10.20.30.0/24"]},
|
||||
{"identifier": "lo0", "device": "lo0", "description": "Loopback", "ipv4": [{"ipaddr": "127.0.0.1/8"}], "ipv6": []},
|
||||
]})
|
||||
if method == "GET" and path == "/firewall/group/searchItem":
|
||||
return httpx.Response(200, json={"rows": []})
|
||||
if method == "GET" and path == "/firewall/alias/searchItem":
|
||||
return httpx.Response(200, json={"rows": [{"name": "tbl", "type": "urltable", "content": "https://203.0.113.9/list"}]})
|
||||
if method == "GET" and path == "/firewall/d_nat/searchRule":
|
||||
return httpx.Response(200, json={"rows": []})
|
||||
if method == "GET" and path == "/core/hasync/get":
|
||||
sel = {"": {"value": "Disabled", "selected": 1 if not self.pfsync else 0},
|
||||
"lan": {"value": "LAN", "selected": 1 if self.pfsync == "lan" else 0}}
|
||||
return httpx.Response(200, json={"hasync": {"pfsyncinterface": sel, "pfsyncpeerip": self.peer, "password": "x"}})
|
||||
m = re.fullmatch(r"/firewall/filter/getRule/([^/]+)", path)
|
||||
if method == "GET" and m:
|
||||
got = self._inject(("get", m.group(1)))
|
||||
if got is not None:
|
||||
return got
|
||||
row = self.get(m.group(1))
|
||||
if row is None or row.get("is_automatic") or row.get("legacy"):
|
||||
return httpx.Response(200, json=[])
|
||||
return httpx.Response(200, json={"rule": self.model(row)})
|
||||
m = re.fullmatch(r"/firewall/filter/toggleRule/([^/]+)/([01])", path)
|
||||
if method == "POST" and m:
|
||||
if self.on_call:
|
||||
self.on_call("toggle", m.group(1))
|
||||
row = self.get(m.group(1))
|
||||
if row is not None and not row.get("legacy"):
|
||||
pending = ("toggle", m.group(1))
|
||||
if pending in self.fail and self.fail.get(("toggle_applies", m.group(1))):
|
||||
row["enabled"] = m.group(2) # applied on the firewall, but the answer is lost
|
||||
got = self._inject(pending)
|
||||
if got is not None:
|
||||
return got
|
||||
changed = row["enabled"] != m.group(2)
|
||||
row["enabled"] = m.group(2)
|
||||
return httpx.Response(200, json={"result": "Enabled" if m.group(2) == "1" else "Disabled", "changed": changed})
|
||||
return httpx.Response(200, json={"result": "failed"})
|
||||
m = re.fullmatch(r"/firewall/filter/delRule/([^/]+)", path)
|
||||
if method == "POST" and m:
|
||||
if self.on_call:
|
||||
self.on_call("delete", m.group(1))
|
||||
row = self.get(m.group(1))
|
||||
applies = self.fail.get(("delete_applies", m.group(1)))
|
||||
if applies and row is not None:
|
||||
self.rules.remove(row)
|
||||
got = self._inject(("delete", m.group(1)))
|
||||
if got is not None:
|
||||
return got
|
||||
if row is None or row.get("legacy"):
|
||||
return httpx.Response(200, json={"result": "not found"})
|
||||
self.rules.remove(row)
|
||||
return httpx.Response(200, json={"result": "deleted"})
|
||||
if method == "POST" and path == "/firewall/filter/addRule":
|
||||
got = self._inject(("add", ""))
|
||||
if got is not None:
|
||||
return got
|
||||
import json as _json
|
||||
body = _json.loads(request.content)["rule"]
|
||||
new = str(uuid_lib.uuid4())
|
||||
top = max(int(float(r.get("sequence") or 0)) for r in self.rules if not r.get("is_automatic"))
|
||||
row = {**body, "uuid": new, "sequence": str(top + 100), "sort_order": f"400000.{top + 100:07d}", "prio_group": "400000"}
|
||||
self.rules.append(row)
|
||||
return httpx.Response(200, json={"result": "saved", "uuid": new})
|
||||
m = re.fullmatch(r"/firewall/filter/moveRuleBefore/([^/]+)/([^/]+)", path)
|
||||
if method == "POST" and m:
|
||||
got = self._inject(("move", m.group(1))) or self._inject(("move", "*"))
|
||||
if got is not None:
|
||||
return got
|
||||
moving, target = self.get(m.group(1)), self.get(m.group(2))
|
||||
if moving is None or target is None or moving is target:
|
||||
return httpx.Response(500, json={"errorMessage": "Unexpected error, check log for details"})
|
||||
seq = int(target["sequence"]) - 1
|
||||
moving["sequence"], moving["sort_order"] = str(seq), f"400000.{seq:07d}"
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
if method == "POST" and path == "/firewall/filter/apply":
|
||||
if self.on_call:
|
||||
self.on_call("apply", "")
|
||||
got = self._inject(("apply", ""))
|
||||
if got is not None:
|
||||
return got
|
||||
self.applied += 1
|
||||
return httpx.Response(200, json={"status": "OK\n\n"})
|
||||
return httpx.Response(404, json={"message": f"unexpected {method} {path}"})
|
||||
|
||||
def writes(self) -> list[tuple[str, str]]:
|
||||
return [(m, p) for m, p in self.calls if m == "POST"]
|
||||
@@ -0,0 +1,169 @@
|
||||
"""HTTP API of configuration as code (1.32 P7): rights matrix, audit, errors, limits."""
|
||||
import pytest
|
||||
import yaml
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.main import app
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.rule import FirewallRule
|
||||
from app.services import api_token_service
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
PW = "config-code-password-1"
|
||||
|
||||
|
||||
async def _login(username: str, password: str) -> AsyncClient:
|
||||
ac = AsyncClient(transport=ASGITransport(app=app), base_url="http://test")
|
||||
resp = await ac.post("/api/auth/login", json={"username": username, "password": password})
|
||||
assert resp.status_code == 200, resp.text
|
||||
return ac
|
||||
|
||||
|
||||
def _bearer(token: str) -> AsyncClient:
|
||||
return AsyncClient(transport=ASGITransport(app=app), base_url="http://test", headers={"Authorization": f"Bearer {token}"})
|
||||
|
||||
|
||||
async def _token(client, scope: str) -> str:
|
||||
resp = await client.post("/api/tokens", json={"name": f"t-{scope}", "scope": scope, "current_password": settings.admin_password})
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()["token"]
|
||||
|
||||
|
||||
async def _changed_file(client) -> str:
|
||||
text = (await client.get("/api/config/export")).text
|
||||
data = yaml.safe_load(text)
|
||||
data["rules"][0]["description"] = "über die API"
|
||||
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scope,method,path,denied", [
|
||||
("read", "GET", "/api/config/export", False),
|
||||
("read", "POST", "/api/config/import/plan", False),
|
||||
("read", "POST", "/api/config/import/apply", True),
|
||||
("deploy", "POST", "/api/config/import/plan", False),
|
||||
("deploy", "POST", "/api/config/import/apply", True),
|
||||
("admin", "POST", "/api/config/import/apply", False),
|
||||
# the read-only exception is exactly this path, not its neighbours
|
||||
("read", "POST", "/api/config/import/plan/x", True),
|
||||
("read", "POST", "/api/config/import", True),
|
||||
])
|
||||
def test_token_scope_matrix(scope, method, path, denied):
|
||||
assert api_token_service.denies(scope, method, path) is denied
|
||||
|
||||
|
||||
def test_operators_may_plan_but_not_apply():
|
||||
assert not api_token_service.operator_denies("POST", "/api/config/import/plan")
|
||||
assert api_token_service.operator_denies("POST", "/api/config/import/apply")
|
||||
|
||||
|
||||
async def test_export_plan_apply_as_admin_with_audit(client):
|
||||
await seed_everything()
|
||||
resp = await client.get("/api/config/export")
|
||||
assert resp.status_code == 200 and resp.headers["content-type"].startswith("text/yaml")
|
||||
assert "attachment" in resp.headers["content-disposition"] and resp.headers["cache-control"] == "no-store"
|
||||
text = await _changed_file(client)
|
||||
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
|
||||
assert plan["summary"]["update"] == 1 and plan["applicable"] and len(plan["token"]) == 64
|
||||
assert [i["key"] for i in plan["items"]] == ["allow-dns-lan"] # unchanged objects are not listed
|
||||
resp = await client.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan["token"]})
|
||||
assert resp.status_code == 200 and resp.json()["summary"]["update"] == 1
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.execute(select(AuditEvent))).scalars().all()
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
|
||||
assert rule.description == "über die API"
|
||||
requests = [e for e in events if e.kind == "config" and e.action == "apply"]
|
||||
assert [e.status for e in requests] == ["requested", "accepted"] and requests[0].entity_id is None
|
||||
assert not any(e.action == "plan" for e in events) # planning is like a preview: not audited
|
||||
assert all("über die API" not in str(e.details) for e in events if e.kind == "config")
|
||||
assert any(e.kind == "rule" and e.action == "import_update" and e.actor == settings.admin_username for e in events)
|
||||
|
||||
|
||||
async def test_stale_blocked_invalid_and_too_large(client):
|
||||
await seed_everything()
|
||||
text = await _changed_file(client)
|
||||
plan = (await client.post("/api/config/import/plan", json={"yaml": text})).json()
|
||||
resp = await client.post("/api/config/import/apply", json={"yaml": text + "\n# anders\n", "plan_token": "0" * 64})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_PLAN_STALE"
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "a: &x 1\n"})
|
||||
detail = resp.json()["detail"]
|
||||
assert resp.status_code == 422 and detail["code"] == "CONFIG_INVALID" and detail["params"]["errors"][0]["code"] == "yaml_anchor_forbidden"
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "#" * (1024 * 1024 + 1)})
|
||||
assert resp.status_code == 413 and resp.json()["detail"]["code"] == "CONFIG_TOO_LARGE"
|
||||
resp = await client.post("/api/config/import/plan", content=b"x" * (4 * 1024 * 1024 + 1), headers={"content-type": "application/json"})
|
||||
assert resp.status_code == 413
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 9\n"})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "CONFIG_UNSUPPORTED_VERSION"
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n", "extra": 1})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["params"]["errors"][0]["code"] == "bad_request"
|
||||
resp = await client.get("/api/config/export?kinds=rules,secrets")
|
||||
assert resp.status_code == 422
|
||||
blocked = "opnexus_config: 1\naliases: []\n"
|
||||
plan = (await client.post("/api/config/import/plan", json={"yaml": blocked, "prune": True})).json()
|
||||
assert not plan["applicable"] and plan["items"][0]["action"] == "blocked"
|
||||
resp = await client.post("/api/config/import/apply", json={"yaml": blocked, "prune": True, "plan_token": plan["token"]})
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "CONFIG_BLOCKED"
|
||||
|
||||
|
||||
async def test_read_only_account_can_export_and_plan_but_not_apply(client):
|
||||
await seed_everything()
|
||||
assert (await client.post("/api/users", json={"username": "cfg-reader", "password": PW, "role": "read_only"})).status_code == 201
|
||||
text = await _changed_file(client)
|
||||
reader = await _login("cfg-reader", PW)
|
||||
try:
|
||||
assert (await reader.get("/api/config/export")).status_code == 200
|
||||
plan = await reader.post("/api/config/import/plan", json={"yaml": text})
|
||||
assert plan.status_code == 200
|
||||
resp = await reader.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "READ_ONLY_ACCESS"
|
||||
finally:
|
||||
await reader.aclose()
|
||||
|
||||
|
||||
async def test_operator_can_plan_but_not_apply(client):
|
||||
await seed_everything()
|
||||
assert (await client.post("/api/users", json={"username": "cfg-operator", "password": PW, "role": "operator"})).status_code == 201
|
||||
text = await _changed_file(client)
|
||||
op = await _login("cfg-operator", PW)
|
||||
try:
|
||||
plan = await op.post("/api/config/import/plan", json={"yaml": text})
|
||||
assert plan.status_code == 200
|
||||
resp = await op.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "ADMIN_REQUIRED"
|
||||
finally:
|
||||
await op.aclose()
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "config"))).scalars().all()
|
||||
assert [e.status for e in events] == ["denied"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scope,plan_status,apply_status", [("read", 200, 403), ("deploy", 200, 403), ("admin", 200, 200)])
|
||||
async def test_tokens(client, scope, plan_status, apply_status):
|
||||
await seed_everything()
|
||||
text = await _changed_file(client)
|
||||
async with _bearer(await _token(client, scope)) as api:
|
||||
assert (await api.get("/api/config/export")).status_code == 200
|
||||
plan = await api.post("/api/config/import/plan", json={"yaml": text})
|
||||
assert plan.status_code == plan_status
|
||||
resp = await api.post("/api/config/import/apply", json={"yaml": text, "plan_token": plan.json()["token"]})
|
||||
assert resp.status_code == apply_status, resp.text
|
||||
|
||||
|
||||
async def test_group_changes_are_audited_now(client):
|
||||
resp = await client.post("/api/groups", json={"name": "Neu", "description": "", "host_ids": []})
|
||||
assert resp.status_code == 201
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.execute(select(AuditEvent).where(AuditEvent.kind == "group"))).scalars().all()
|
||||
assert [e.status for e in events] == ["requested", "accepted"]
|
||||
|
||||
|
||||
async def test_too_many_waiting_plans_are_turned_away(client, monkeypatch):
|
||||
from app.routers import config_code
|
||||
|
||||
monkeypatch.setattr(config_code, "_waiting", config_code.MAX_WAITING)
|
||||
resp = await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})
|
||||
assert resp.status_code == 429 and resp.json()["detail"]["code"] == "CONFIG_BUSY"
|
||||
monkeypatch.setattr(config_code, "_waiting", 0)
|
||||
assert (await client.post("/api/config/import/plan", json={"yaml": "opnexus_config: 1\n"})).status_code == 200
|
||||
@@ -0,0 +1,434 @@
|
||||
"""Planner and apply (1.32 P5/P6): round trip, partial files, prune protection, conflicts, atomicity, no firewall."""
|
||||
import ast
|
||||
import copy
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import respx
|
||||
import yaml
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.config_code import apply as applier
|
||||
from app.config_code import plan as planner
|
||||
from app.config_code.export import export_yaml
|
||||
from app.config_code.loader import ConfigError, parse
|
||||
from app.database import SessionLocal
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_variable import HostVariable
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.policy import DeviceGroup, RuleRevision
|
||||
from app.models.policy_hierarchy_group import PolicyHierarchyGroup
|
||||
from app.models.policy_hierarchy_rule import PolicyHierarchyRule
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
CENTRAL = (PolicyHierarchyRule, OutboundNatRule, OneToOneRule, PortForwardRule, FirewallRule, FirewallAlias, DeviceGroup)
|
||||
|
||||
|
||||
async def export() -> str:
|
||||
async with SessionLocal() as db:
|
||||
return await export_yaml(db)
|
||||
|
||||
|
||||
async def plan(text: str, prune: bool = False) -> planner.Plan:
|
||||
async with SessionLocal() as db:
|
||||
return await planner.build_plan(db, parse(text), prune)
|
||||
|
||||
|
||||
async def apply(text: str, prune: bool = False, token: str | None = None) -> planner.Plan:
|
||||
token = token or (await plan(text, prune)).token
|
||||
async with SessionLocal() as db:
|
||||
try:
|
||||
result = await applier.apply_config(db, parse(text), prune, token)
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
raise
|
||||
await db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def edit(text: str, fn) -> str:
|
||||
data = yaml.safe_load(text)
|
||||
fn(data)
|
||||
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
def by_key(p: planner.Plan) -> dict:
|
||||
return {(i.kind, i.key): i for i in p.items}
|
||||
|
||||
|
||||
async def wipe_central_objects() -> None:
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostVariable.__table__.delete())
|
||||
for model in CENTRAL:
|
||||
await db.execute(model.__table__.delete())
|
||||
await db.execute(PolicyHierarchyGroup.__table__.update().values(parent_id=None))
|
||||
await db.execute(OpnsenseHost.__table__.update().values(policy_hierarchy_group_id=None))
|
||||
await db.execute(PolicyHierarchyGroup.__table__.delete())
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def payloads() -> dict:
|
||||
"""What every object would push to every host: the ROADMAP criterion "identical preview" after a round trip."""
|
||||
from app.services.alias_deploy_service import _build_payload as alias_payload
|
||||
from app.services.one_to_one_deploy_service import _build_payload as one_payload
|
||||
from app.services.outbound_nat_deploy_service import _build_payload as out_payload
|
||||
from app.services.port_forward_deploy_service import _build_payload as pf_payload
|
||||
from app.services.rule_deploy_service import _build_payload as rule_payload
|
||||
|
||||
out = {}
|
||||
async with SessionLocal() as db:
|
||||
hosts = (await db.execute(select(OpnsenseHost))).scalars().all()
|
||||
for model, builder, ident in ((FirewallRule, rule_payload, "config_key"), (PortForwardRule, pf_payload, "config_key"),
|
||||
(OneToOneRule, one_payload, "config_key"), (OutboundNatRule, out_payload, "config_key"),
|
||||
(FirewallAlias, alias_payload, "name")):
|
||||
for row in (await db.execute(select(model))).scalars().all():
|
||||
for host in hosts:
|
||||
try:
|
||||
out[(model.__name__, getattr(row, ident), host.name)] = builder(row, host)
|
||||
except Exception as exc: # a missing variable is part of the result too
|
||||
out[(model.__name__, getattr(row, ident), host.name)] = type(exc).__name__
|
||||
return out
|
||||
|
||||
|
||||
# -- round trip ---------------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_export_then_plan_shows_no_changes_and_apply_is_a_no_op():
|
||||
await seed_everything()
|
||||
text = await export()
|
||||
p = await plan(text)
|
||||
assert p.applicable and p.summary()["create"] == p.summary()["update"] == p.summary()["delete"] == 0
|
||||
await apply(text, prune=True)
|
||||
assert await export() == text
|
||||
|
||||
|
||||
async def test_import_into_an_instance_without_central_objects_restores_everything_identically():
|
||||
await seed_everything()
|
||||
text, before = await export(), await payloads()
|
||||
await wipe_central_objects()
|
||||
p = await plan(text, prune=True)
|
||||
assert p.applicable and p.summary()["create"] > 10
|
||||
await apply(text, prune=True)
|
||||
assert await export() == text
|
||||
assert await payloads() == before
|
||||
assert (await plan(text, prune=True)).summary()["create"] == 0
|
||||
|
||||
|
||||
async def test_apply_writes_revisions_and_audit_events_without_the_file_content():
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="geändert"))
|
||||
await apply(text)
|
||||
async with SessionLocal() as db:
|
||||
notes = (await db.execute(select(RuleRevision.note))).scalars().all()
|
||||
events = (await db.execute(select(AuditEvent))).scalars().all()
|
||||
assert "Import (Konfiguration als Code)" in notes and "Vor erster Bearbeitung" in notes
|
||||
summary = next(e for e in events if e.kind == "config")
|
||||
assert summary.action == "import" and summary.details["summary"]["update"] == 1
|
||||
assert any(e.kind == "rule" and e.action == "import_update" and e.details["fields"] == ["description"] for e in events)
|
||||
assert all("geändert" not in str(e.details) for e in events if e.kind == "config")
|
||||
|
||||
|
||||
# -- partial files and prune ---------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_a_missing_section_leaves_the_kind_alone_and_an_empty_one_only_deletes_with_prune():
|
||||
await seed_everything()
|
||||
only_aliases = "opnexus_config: 1\naliases: []\n"
|
||||
p = await plan(only_aliases)
|
||||
assert p.touched == ["aliases"] and p.summary() == {"create": 0, "update": 0, "delete": 0, "unchanged": 0, "blocked": 0}
|
||||
await apply(only_aliases)
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 1
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
|
||||
p = await plan(only_aliases, prune=True)
|
||||
assert by_key(p)[("aliases", "WEB_SERVERS")].reasons == ["prune_referenced"] # a rule still names it
|
||||
rules_without_alias = edit(await export(), lambda d: d["rules"][0].update(destination_net="any"))
|
||||
await apply(rules_without_alias)
|
||||
await apply(only_aliases, prune=True)
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallAlias)) == 0
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
|
||||
|
||||
|
||||
async def test_prune_never_deletes_a_deployed_object_and_nothing_changes():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="reverted", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
text = edit(await export(), lambda d: d.update(rules=[r for r in d["rules"] if r["key"] != "block-all"]))
|
||||
p = await plan(text, prune=True)
|
||||
assert by_key(p)[("rules", "block-all")].reasons == ["prune_deployed"] and not p.applicable
|
||||
with pytest.raises(applier.PlanBlocked):
|
||||
await apply(text, prune=True, token=p.token)
|
||||
async with SessionLocal() as db:
|
||||
assert await db.scalar(select(func.count()).select_from(FirewallRule)) == 2
|
||||
|
||||
|
||||
async def test_an_object_with_an_open_deployment_cannot_be_changed():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "block-all"))
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-2"))
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="pending", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
text = edit(await export(), lambda d: d["rules"][1].update(description="neu"))
|
||||
item = by_key(await plan(text))[("rules", "block-all")]
|
||||
assert item.reasons == ["busy_deployment"] and item.deployed_hosts == 1
|
||||
unchanged = await plan(await export())
|
||||
assert unchanged.applicable # an unchanged busy object does not block
|
||||
|
||||
|
||||
async def test_pruning_a_group_still_used_by_a_rollout_wave_is_blocked():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
|
||||
rollout = RolloutPlan(name="r", action="update", created_by="t")
|
||||
db.add(rollout)
|
||||
await db.flush()
|
||||
wave = RolloutPlanWave(plan_id=rollout.id, position=0, label="w", groups=[group])
|
||||
db.add(wave)
|
||||
await db.commit()
|
||||
data = yaml.safe_load(await export())
|
||||
for section in ("aliases", "rules", "one_to_one", "outbound_nat"):
|
||||
for obj in data[section]:
|
||||
obj["targets"]["groups"] = []
|
||||
data["device_groups"] = []
|
||||
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("device_groups", "Filialen")]
|
||||
assert item.reasons == ["prune_referenced"]
|
||||
|
||||
|
||||
async def test_pruning_a_hierarchy_group_is_blocked_while_hosts_rules_or_children_remain():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await export())
|
||||
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Organisation"]
|
||||
with pytest.raises(ConfigError) as info: # Nord still names Organisation as parent
|
||||
await plan(yaml.safe_dump(data), prune=True)
|
||||
assert info.value.errors[0]["code"] == "unknown_hierarchy_group"
|
||||
data = yaml.safe_load(await export())
|
||||
data["policy_hierarchy_groups"] = [g for g in data["policy_hierarchy_groups"] if g["name"] != "Nord"]
|
||||
text = yaml.safe_dump(data)
|
||||
with pytest.raises(ConfigError): # the hierarchy rule ph-post-deny still sits in Nord
|
||||
await plan(text, prune=True)
|
||||
data["policy_hierarchy_rules"] = [r for r in data["policy_hierarchy_rules"] if r["group"] != "Nord"]
|
||||
p = await plan(yaml.safe_dump(data), prune=True)
|
||||
assert p.applicable # fw-filiale-1 drops out of the hierarchy with its group
|
||||
await apply(yaml.safe_dump(data), prune=True)
|
||||
async with SessionLocal() as db:
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
|
||||
assert host.policy_hierarchy_group_id is None
|
||||
data = yaml.safe_load(await export())
|
||||
data.pop("policy_hierarchy_rules")
|
||||
data["policy_hierarchy_groups"] = []
|
||||
item = by_key(await plan(yaml.safe_dump(data), prune=True))[("policy_hierarchy_groups", "Organisation")]
|
||||
assert item.reasons == ["prune_referenced"] # its hierarchy rules stay in the database
|
||||
|
||||
|
||||
# -- validation against the database ------------------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("change,code", [
|
||||
(lambda d: d["rules"][0]["targets"]["hosts"].append("nope"), "unknown_host"),
|
||||
(lambda d: d["rules"][0]["targets"]["groups"].append("nope"), "unknown_group"),
|
||||
(lambda d: d["rules"].append(copy.deepcopy(d["rules"][0])), "duplicate_object_key"),
|
||||
(lambda d: d["rules"][0]["targets"]["groups"].append("Filialen"), "duplicate_entry"),
|
||||
(lambda d: d["policy_hierarchy_rules"][0].update(description="{{SITE}}"), "placeholder_not_supported"),
|
||||
(lambda d: d["policy_hierarchy_groups"][0].update(parent="Nord"), "hierarchy_cycle"),
|
||||
(lambda d: d["policy_hierarchy_groups"][0]["hosts"].append("fw-filiale-1"), "host_in_two_hierarchy_groups"),
|
||||
(lambda d: d["variables"].update({"nope": {"X": "1"}}), "unknown_host"),
|
||||
(lambda d: d["aliases"][0]["targets"].update(hierarchy_group="nope"), "unknown_hierarchy_group"),
|
||||
])
|
||||
async def test_file_errors_are_reported_with_a_path(change, code):
|
||||
await seed_everything()
|
||||
with pytest.raises(ConfigError) as info:
|
||||
await plan(edit(await export(), change))
|
||||
assert code in [e["code"] for e in info.value.errors]
|
||||
assert all(e["path"] for e in info.value.errors)
|
||||
|
||||
|
||||
async def test_warnings_for_unknown_aliases_removed_variables_and_redeploys():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
|
||||
host = await db.scalar(select(OpnsenseHost).where(OpnsenseHost.name == "fw-filiale-1"))
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=host.id, status="success", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
|
||||
def change(d):
|
||||
d["rules"][0].update(destination_net="MISSING_ALIAS", source_net="{{LAN_NET}}")
|
||||
d["rules"][1].update(source_net="lan", destination_net="wanip")
|
||||
d["variables"] = {"fw-filiale-1": {"SITE": "x"}}
|
||||
|
||||
p = await plan(edit(await export(), change), prune=True)
|
||||
codes = {w["code"]: w for w in p.warnings}
|
||||
assert codes["unknown_alias_reference"]["params"]["value"] == "MISSING_ALIAS"
|
||||
assert codes["variable_removed_in_use"]["params"]["name"] == "LAN_NET"
|
||||
assert codes["redeploy_needed"]["params"]["count"] == 1
|
||||
assert [w for w in p.warnings if w["code"] == "unknown_alias_reference"] == [codes["unknown_alias_reference"]]
|
||||
|
||||
|
||||
# -- hierarchy order -----------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_hierarchy_rule_order_is_the_order_in_the_file():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await export())
|
||||
rules = data["policy_hierarchy_rules"]
|
||||
rules[0], rules[1] = rules[1], rules[0] # both in Organisation/pre
|
||||
text = yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
p = await plan(text)
|
||||
changed = {i.key: i.changes for i in p.items if i.action == "update"}
|
||||
assert set(changed) == {"ph-block-bogons", "ph-allow-mgmt"} and changed["ph-allow-mgmt"][0]["field"] == "position"
|
||||
await apply(text)
|
||||
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]] == ["ph-allow-mgmt", "ph-block-bogons", "ph-post-deny"]
|
||||
assert (await plan(text)).summary()["update"] == 0
|
||||
|
||||
|
||||
async def test_new_hierarchy_rules_go_before_rules_the_file_does_not_name():
|
||||
await seed_everything()
|
||||
text = ("opnexus_config: 1\npolicy_hierarchy_rules:\n"
|
||||
"- {key: ph-new, group: Organisation, rulebase: pre, description: neu, interface: lan}\n")
|
||||
p = await plan(text)
|
||||
assert {i.key: i.action for i in p.items} == {"ph-new": "create", "ph-block-bogons": "update", "ph-allow-mgmt": "update"}
|
||||
assert all(i.implicit for i in p.items if i.key != "ph-new")
|
||||
await apply(text)
|
||||
assert [r["key"] for r in yaml.safe_load(await export())["policy_hierarchy_rules"]][:3] == ["ph-new", "ph-block-bogons", "ph-allow-mgmt"]
|
||||
|
||||
|
||||
# -- variables -----------------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_variables_are_merged_per_name_and_only_removed_with_prune():
|
||||
await seed_everything()
|
||||
text = "opnexus_config: 1\nvariables:\n fw-filiale-2:\n LAN_NET: 10.9.0.0/24\n fw-filiale-1:\n SITE: Neu\n"
|
||||
await apply(text)
|
||||
data = yaml.safe_load(await export())["variables"]
|
||||
assert data == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
|
||||
await apply(text, prune=True)
|
||||
assert yaml.safe_load(await export())["variables"] == {"fw-filiale-1": {"SITE": "Neu"}, "fw-filiale-2": {"LAN_NET": "10.9.0.0/24"}}
|
||||
|
||||
|
||||
# -- conflicts, atomicity, self-check -----------------------------------------------------------------------------------
|
||||
|
||||
async def test_a_change_between_plan_and_apply_makes_the_plan_stale():
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="aus der Datei"))
|
||||
token = (await plan(text)).token
|
||||
async with SessionLocal() as db:
|
||||
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen").options(selectinload(DeviceGroup.hosts)))
|
||||
group.hosts = group.hosts[:1] # only a group member: no updated_at anywhere
|
||||
await db.commit()
|
||||
with pytest.raises(applier.PlanStale):
|
||||
await apply(text, token=token)
|
||||
async with SessionLocal() as db:
|
||||
db.add(FirewallRule(description="neu in der UI", interface="lan", direction="in"))
|
||||
await db.commit()
|
||||
token = (await plan(text)).token
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.description == "neu in der UI"))
|
||||
rule.description = "noch einmal geändert"
|
||||
await db.commit()
|
||||
with pytest.raises(applier.PlanStale):
|
||||
await apply(text, token=token)
|
||||
|
||||
|
||||
async def test_a_failure_in_the_middle_leaves_the_database_unchanged(monkeypatch):
|
||||
await seed_everything()
|
||||
before = await export()
|
||||
text = edit(before, lambda d: [r.update(description=f"neu {i}") for i, r in enumerate(d["rules"])] and
|
||||
d["aliases"][0].update(description="neu"))
|
||||
original = applier._Writer.objects
|
||||
|
||||
async def broken(self, kind):
|
||||
await original(self, kind)
|
||||
if kind == "rules":
|
||||
raise RuntimeError("injected")
|
||||
|
||||
monkeypatch.setattr(applier._Writer, "objects", broken)
|
||||
with pytest.raises(RuntimeError):
|
||||
await apply(text)
|
||||
assert await export() == before
|
||||
|
||||
|
||||
async def test_the_self_check_catches_a_writer_that_drops_a_field(monkeypatch):
|
||||
await seed_everything()
|
||||
before = await export()
|
||||
text = edit(before, lambda d: d["rules"][0].update(log=True))
|
||||
monkeypatch.setitem(applier.SCALARS, "rules", tuple(f for f in applier.SCALARS["rules"] if f != "log"))
|
||||
with pytest.raises(applier.SelfCheckFailed) as info:
|
||||
await apply(text)
|
||||
assert info.value.differences == ["rules:allow-dns-lan"]
|
||||
assert await export() == before
|
||||
|
||||
|
||||
async def test_the_writer_covers_every_field_of_the_schema():
|
||||
from app.config_code import schema
|
||||
|
||||
docs = {"aliases": schema.AliasDoc, "rules": schema.RuleDoc, "port_forwards": schema.PortForwardDoc,
|
||||
"one_to_one": schema.OneToOneDoc, "outbound_nat": schema.OutboundNatDoc, "policy_hierarchy_rules": schema.HierarchyRuleDoc}
|
||||
computed = {"key", "targets", "entries", "group"}
|
||||
for kind, doc in docs.items():
|
||||
assert set(applier.SCALARS[kind]) == set(doc.model_fields) - computed, kind
|
||||
|
||||
|
||||
# -- never a firewall ----------------------------------------------------------------------------------------------------
|
||||
|
||||
async def test_plan_and_apply_never_talk_to_a_firewall():
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="x"))
|
||||
with respx.mock(assert_all_called=False) as mock: # no routes: any HTTP request fails the test
|
||||
await apply(text, prune=True)
|
||||
await wipe_central_objects()
|
||||
await apply(text, prune=True)
|
||||
assert mock.calls.call_count == 0
|
||||
|
||||
|
||||
def test_the_config_code_package_does_not_import_a_firewall_client():
|
||||
forbidden = ("opnsense_client", "_deploy_service", "rollout_engine", "host_maintenance_service")
|
||||
for path in Path(planner.__file__).parent.glob("*.py"):
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.Import, ast.ImportFrom)):
|
||||
names = [a.name for a in node.names] + ([node.module] if isinstance(node, ast.ImportFrom) and node.module else [])
|
||||
assert not any(f in n for n in names for f in forbidden), f"{path.name}: {names}"
|
||||
|
||||
|
||||
async def test_two_concurrent_applies_of_the_same_plan_serialize_and_the_second_is_stale():
|
||||
import asyncio
|
||||
|
||||
await seed_everything()
|
||||
text = edit(await export(), lambda d: d["rules"][0].update(description="parallel"))
|
||||
token = (await plan(text)).token
|
||||
results = await asyncio.gather(apply(text, token=token), apply(text, token=token), return_exceptions=True)
|
||||
assert sorted(type(r).__name__ for r in results) == ["Plan", "PlanStale"]
|
||||
|
||||
|
||||
async def test_variables_merged_beyond_the_per_host_limit_are_refused():
|
||||
"""The host page allows 200 variables per host; merging a file into existing values must not get past that."""
|
||||
await seed_everything() # fw-filiale-1 has 2 already
|
||||
values = "".join(f" V{i}: x\n" for i in range(199))
|
||||
with pytest.raises(ConfigError) as info:
|
||||
await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values)
|
||||
assert [e["code"] for e in info.value.errors] == ["too_many_variables"]
|
||||
assert (await plan("opnexus_config: 1\nvariables:\n fw-filiale-1:\n" + values, prune=True)).applicable
|
||||
|
||||
|
||||
async def test_what_the_ui_can_store_survives_the_round_trip_and_padding_is_cleaned_like_the_ui_does():
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
group = await db.scalar(select(DeviceGroup).where(DeviceGroup.name == "Filialen"))
|
||||
group.description = "zwei\nZeilen " # GroupWrite does not strip descriptions
|
||||
alias = await db.scalar(select(FirewallAlias).where(FirewallAlias.name == "WEB_SERVERS"))
|
||||
alias.description = " eingerückt"
|
||||
await db.commit()
|
||||
text = await export()
|
||||
assert (await plan(text)).summary()["update"] == 0
|
||||
padded = edit(text, lambda d: d["rules"][0].update(description=" DNS aus {{SITE}} ", interface=" lan "))
|
||||
p = await plan(padded)
|
||||
assert p.summary()["update"] == 0 # RuleCreate strips these fields, so this is the same rule
|
||||
with_entries = edit(text, lambda d: d["aliases"][0].update(entries=[" 10.20.30.10 ", "203.0.113.20"]))
|
||||
assert (await plan(with_entries)).summary()["update"] == 0
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Host console of configuration as code (1.32 P8): exit codes, text diff, actor, no control sequences."""
|
||||
import copy
|
||||
import json
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config_code.__main__ import TOKEN_MARK, format_errors, run
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.rule import FirewallRule
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
|
||||
async def cli(*argv, stdin: str | bytes | None = None):
|
||||
data = stdin.encode() if isinstance(stdin, str) else stdin
|
||||
return await run(list(argv), data)
|
||||
|
||||
|
||||
def token_of(out: str) -> str:
|
||||
return next(line[len(TOKEN_MARK):] for line in out.splitlines() if line.startswith(TOKEN_MARK))
|
||||
|
||||
|
||||
async def export_text() -> str:
|
||||
code, out, _ = await cli("export")
|
||||
assert code == 0
|
||||
return out
|
||||
|
||||
|
||||
def edit(text: str, fn) -> str:
|
||||
data = yaml.safe_load(text)
|
||||
fn(data)
|
||||
return yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
async def test_export_plan_without_changes_exits_0_and_a_change_exits_2_with_a_readable_diff():
|
||||
await seed_everything()
|
||||
text = await export_text()
|
||||
code, out, _ = await cli("plan", stdin=text)
|
||||
assert code == 0 and "Plan: 0 neu, 0 geändert, 0 entfernt" in out and token_of(out)
|
||||
def change(d):
|
||||
d["rules"][0]["destination_port"] = 853
|
||||
d["rules"].append({**copy.deepcopy(d["rules"][0]), "key": "new-rule"}) # no shared objects: YAML would write anchors
|
||||
|
||||
changed = edit(text, change)
|
||||
code, out, _ = await cli("plan", stdin=changed)
|
||||
assert code == 2
|
||||
assert '+ rules new-rule (neu)' in out
|
||||
assert '~ rules allow-dns-lan destination_port: "53" → "853"' in out
|
||||
assert out.rstrip().splitlines()[-2].startswith("Plan: 1 neu, 1 geändert, 0 entfernt")
|
||||
|
||||
|
||||
async def test_json_output_is_the_api_answer():
|
||||
await seed_everything()
|
||||
code, out, _ = await cli("plan", "--json", stdin=await export_text())
|
||||
body = json.loads(out)
|
||||
assert code == 0 and body["applicable"] and len(body["token"]) == 64 and body["summary"]["create"] == 0
|
||||
|
||||
|
||||
async def test_apply_needs_the_token_of_the_plan_and_writes_audit_as_the_console():
|
||||
await seed_everything()
|
||||
changed = edit(await export_text(), lambda d: d["rules"][0].update(description="per CLI"))
|
||||
code, _, err = await cli("apply", stdin=changed)
|
||||
assert code == 1 and "--plan-token" in err
|
||||
code, out, _ = await cli("plan", stdin=changed)
|
||||
token = token_of(out)
|
||||
code, out, err = await cli("apply", "--plan-token", "0" * 64, stdin=changed)
|
||||
assert code == 4 and "neu planen" in err
|
||||
code, out, err = await cli("apply", "--plan-token", token, stdin=changed)
|
||||
assert code == 0 and "Übernommen: 0 neu, 1 geändert, 0 entfernt" in out and "nichts auf Firewalls geschrieben" in out
|
||||
async with SessionLocal() as db:
|
||||
rule = await db.scalar(select(FirewallRule).where(FirewallRule.config_key == "allow-dns-lan"))
|
||||
events = (await db.execute(select(AuditEvent))).scalars().all()
|
||||
assert rule.description == "per CLI"
|
||||
assert events and {e.actor for e in events} == {"cli (Host-Konsole)"}
|
||||
|
||||
|
||||
async def test_invalid_files_exit_3_and_name_line_and_path_without_control_characters():
|
||||
await seed_everything()
|
||||
bad = "opnexus_config: 1\nrules:\n- key: r-1\n description: \"x\\e[31mred\"\n interface: lan\n nope: 1\n"
|
||||
code, out, err = await cli("plan", stdin=bad)
|
||||
assert code == 3 and not out
|
||||
assert "Zeile 6: rules[0].nope: Unbekanntes Feld (field=nope)" in err and "\x1b" not in err
|
||||
code, out, _ = await cli("plan", "--json", stdin="a: &x 1\n")
|
||||
assert code == 3 and json.loads(out)["errors"][0]["code"] == "yaml_anchor_forbidden"
|
||||
assert "\x1b" not in format_errors([{"code": "x", "path": "p\x1b[2J", "line": 1, "params": {"value": "v\x1b[0m"}}])
|
||||
|
||||
|
||||
async def test_blocked_plans_exit_4_and_apply_refuses_them():
|
||||
await seed_everything()
|
||||
only_aliases = "opnexus_config: 1\naliases: []\n"
|
||||
code, out, _ = await cli("plan", "--prune", stdin=only_aliases)
|
||||
assert code == 4 and "! aliases WEB_SERVERS blockiert (entfernen): wird noch verwendet" in out
|
||||
code, out, err = await cli("apply", "--prune", "--plan-token", token_of(out), stdin=only_aliases)
|
||||
assert code == 4 and "blockierte Objekte" in err
|
||||
|
||||
|
||||
async def test_kinds_schema_and_usage():
|
||||
await seed_everything()
|
||||
code, out, _ = await cli("export", "--kinds", "aliases")
|
||||
assert code == 0 and list(yaml.safe_load(out)) == ["opnexus_config", "aliases"]
|
||||
assert (await cli("export", "--kinds", "nope"))[0] == 1
|
||||
code, out, _ = await cli("schema")
|
||||
assert code == 0 and json.loads(out)["title"].startswith("OPNexus-Konfiguration")
|
||||
assert (await cli("frobnicate"))[0] == 1
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Export (1.32 P3): deterministic, complete, schema-conform, and free of anything secret."""
|
||||
import re
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from app.config_code import schema
|
||||
from app.config_code.export import canonical, dump_yaml, export_yaml
|
||||
from app.database import SessionLocal
|
||||
from app.models.rule import FirewallRule
|
||||
from tests.config_code_seed import seed_everything
|
||||
|
||||
GOLDEN = '''\
|
||||
# OPNexus-Konfiguration (Schema v1), exportiert von OPNexus {version}.
|
||||
# Enthält keine Zugänge zu Firewalls, Benutzer, Tokens oder Schlüssel. Prüfen: opnexus apply DATEI --dry-run
|
||||
opnexus_config: 1
|
||||
device_groups:
|
||||
- name: Filialen
|
||||
description: alle Filialen
|
||||
hosts:
|
||||
- fw-filiale-1
|
||||
- fw-filiale-2
|
||||
policy_hierarchy_groups:
|
||||
- name: Organisation
|
||||
parent: null
|
||||
description: ''
|
||||
hosts: []
|
||||
- name: Nord
|
||||
parent: Organisation
|
||||
description: Region Nord
|
||||
hosts:
|
||||
- fw-filiale-1
|
||||
aliases:
|
||||
- name: WEB_SERVERS
|
||||
type: host
|
||||
description: DMZ
|
||||
enabled: true
|
||||
entries:
|
||||
- 10.20.30.10
|
||||
- 203.0.113.20
|
||||
targets:
|
||||
hosts: []
|
||||
groups:
|
||||
- Filialen
|
||||
hierarchy_group: null
|
||||
'''
|
||||
|
||||
|
||||
async def _export(**kw) -> str:
|
||||
async with SessionLocal() as db:
|
||||
return await export_yaml(db, **kw)
|
||||
|
||||
|
||||
async def test_export_starts_with_the_fixed_header_and_the_expected_sections():
|
||||
from app.version import APP_VERSION
|
||||
|
||||
await seed_everything()
|
||||
text = await _export()
|
||||
assert text.startswith(GOLDEN.format(version=APP_VERSION))
|
||||
assert [k for k in yaml.safe_load(text) if k != "opnexus_config"] == list(schema.KINDS)
|
||||
assert text.endswith("\n") and "\r" not in text
|
||||
|
||||
|
||||
async def test_export_is_deterministic_and_validates_against_the_schema():
|
||||
await seed_everything()
|
||||
first, second = await _export(), await _export()
|
||||
assert first == second
|
||||
doc = schema.ConfigDoc.model_validate(yaml.safe_load(first)) # strict: unknown fields, wrong types, control chars
|
||||
assert [r.key for r in doc.rules] == ["allow-dns-lan", "block-all"]
|
||||
assert doc.variables == {"fw-filiale-1": {"LAN_NET": "10.20.30.0/24", "SITE": "Filiale Nord"}}
|
||||
|
||||
|
||||
async def test_hierarchy_rules_are_ordered_by_group_rulebase_and_sequence_and_keep_placeholders_unresolved():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await _export())
|
||||
assert [r["key"] for r in data["policy_hierarchy_rules"]] == ["ph-block-bogons", "ph-allow-mgmt", "ph-post-deny"]
|
||||
rule = next(r for r in data["rules"] if r["key"] == "allow-dns-lan")
|
||||
assert rule["source_net"] == "{{LAN_NET}}" and rule["destination_port"] == "53"
|
||||
|
||||
|
||||
async def test_ports_are_written_as_strings_so_they_survive_the_round_trip():
|
||||
await seed_everything()
|
||||
text = await _export()
|
||||
assert re.search(r"destination_port: '53'", text)
|
||||
assert schema.ConfigDoc.model_validate(yaml.safe_load(text)).rules[0].destination_port == "53"
|
||||
|
||||
|
||||
async def test_export_never_contains_secrets_or_connection_data():
|
||||
await seed_everything()
|
||||
text = (await _export()).lower()
|
||||
for forbidden in ("akia-secret-key", "secret-host.example", "https://", "api_key", "api_secret", "private key", "opx_", "password"):
|
||||
assert forbidden not in text
|
||||
|
||||
|
||||
async def test_kinds_limits_the_sections():
|
||||
await seed_everything()
|
||||
data = yaml.safe_load(await _export(kinds={"aliases", "rules"}))
|
||||
assert list(data) == ["opnexus_config", "aliases", "rules"]
|
||||
|
||||
|
||||
async def test_new_objects_get_a_key_and_an_empty_database_exports_empty_sections():
|
||||
async with SessionLocal() as db:
|
||||
rule = FirewallRule(description="x", interface="lan", direction="in")
|
||||
db.add(rule)
|
||||
await db.commit()
|
||||
assert re.fullmatch(r"r-[0-9a-f]{12}", rule.config_key) and rule.config_key == f"r-{rule.id.hex[:12]}"
|
||||
data = yaml.safe_load(await _export(kinds={"aliases", "variables"}))
|
||||
assert data["aliases"] == [] and data["variables"] == {}
|
||||
|
||||
|
||||
async def test_exporter_field_lists_match_the_schema():
|
||||
"""A field added to the schema but not to the exporter (or vice versa) must fail loudly, not export silently wrong."""
|
||||
await seed_everything()
|
||||
async with SessionLocal() as db:
|
||||
content = await canonical(db)
|
||||
for kind, doc in (("rules", schema.RuleDoc), ("port_forwards", schema.PortForwardDoc), ("one_to_one", schema.OneToOneDoc),
|
||||
("outbound_nat", schema.OutboundNatDoc), ("policy_hierarchy_rules", schema.HierarchyRuleDoc),
|
||||
("aliases", schema.AliasDoc), ("device_groups", schema.DeviceGroupDoc),
|
||||
("policy_hierarchy_groups", schema.HierarchyGroupDoc)):
|
||||
assert all(list(item) == list(doc.model_fields) for item in content[kind]), kind
|
||||
assert dump_yaml(content) == await _export()
|
||||
|
||||
|
||||
async def test_objects_are_sorted_by_key_not_by_insertion_order():
|
||||
async with SessionLocal() as db:
|
||||
for key in ("zz-last", "aa-first", "mm-middle"):
|
||||
db.add(FirewallRule(config_key=key, description=key, interface="lan", direction="in"))
|
||||
await db.commit()
|
||||
assert [r["key"] for r in yaml.safe_load(await _export())["rules"]] == ["aa-first", "mm-middle", "zz-last"]
|
||||
@@ -0,0 +1,171 @@
|
||||
"""Strict YAML loader (1.32 P2): everything a hostile file could try is refused with a stable code."""
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config_code import loader
|
||||
from app.config_code.loader import ConfigError, load_yaml, parse
|
||||
|
||||
HEAD = "opnexus_config: 1\n"
|
||||
|
||||
|
||||
def _code(raw) -> tuple[str, str]:
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(raw)
|
||||
return info.value.code, info.value.errors[0]["code"]
|
||||
|
||||
|
||||
def _codes(raw) -> list[str]:
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(raw)
|
||||
return [e["code"] for e in info.value.errors]
|
||||
|
||||
|
||||
RULE = "rules:\n- key: r-1\n description: d\n interface: lan\n"
|
||||
|
||||
|
||||
def test_a_plain_file_loads():
|
||||
doc, sha, _lines = parse(HEAD + RULE)
|
||||
assert doc.rules[0].key == "r-1" and len(sha) == 64
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,code", [
|
||||
# Billion laughs and every other use of anchors/aliases
|
||||
("a: &x [1, 2]\nb: *x\n", "yaml_anchor_forbidden"),
|
||||
("a: *x\n", "yaml_alias_forbidden"),
|
||||
('lol1: &l1 ["x","x"]\nlol2: &l2 [*l1,*l1]\nlol3: [*l2,*l2]\n', "yaml_anchor_forbidden"),
|
||||
# merge keys
|
||||
("a: {b: 1}\nc:\n <<: {d: 1}\n", "yaml_merge_forbidden"),
|
||||
# explicit tags of any kind
|
||||
("a: !!python/object/apply:os.system ['id']\n", "yaml_tag_forbidden"),
|
||||
("a: !!python/name:os.system\n", "yaml_tag_forbidden"),
|
||||
("a: !!binary aGVsbG8=\n", "yaml_tag_forbidden"),
|
||||
("a: !!set {x: null}\n", "yaml_tag_forbidden"),
|
||||
("a: !!str 1\n", "yaml_tag_forbidden"),
|
||||
("a: ! x\n", "yaml_tag_forbidden"),
|
||||
("a: !local x\n", "yaml_tag_forbidden"),
|
||||
("!!map {a: 1}\n", "yaml_tag_forbidden"),
|
||||
# directives and multiple documents
|
||||
("%YAML 1.1\n---\na: 1\n", "yaml_directive_forbidden"),
|
||||
("%TAG !e! tag:example.com,2000:\n---\na: 1\n", "yaml_directive_forbidden"),
|
||||
("a: 1\n---\nb: 2\n", "yaml_multiple_documents"),
|
||||
("", "yaml_empty"),
|
||||
("# only a comment\n", "yaml_empty"),
|
||||
# keys
|
||||
("a: 1\na: 2\n", "duplicate_key"),
|
||||
("rules:\n- key: x\n key: y\n", "duplicate_key"),
|
||||
("1: a\n", "non_string_key"),
|
||||
("true: a\n", "non_string_key"),
|
||||
("? [a]\n: b\n", "non_string_key"),
|
||||
("null: a\n", "non_string_key"),
|
||||
# structure limits
|
||||
("a: " + "[" * 20 + "]" * 20 + "\n", "too_deep"),
|
||||
("a: x\0y\n", "yaml_syntax"),
|
||||
("a: 'x\x1b[31m'\n", "yaml_syntax"),
|
||||
("a: [1, 2\n", "yaml_syntax"),
|
||||
("a:\n\t- b\n", "yaml_syntax"),
|
||||
])
|
||||
def test_hostile_yaml_is_refused(raw, code):
|
||||
assert _code(raw) == ("CONFIG_INVALID", code)
|
||||
|
||||
|
||||
def test_errors_name_line_and_path():
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + RULE + " targets:\n hosts: [a]\n hosts: [b]\n")
|
||||
error = info.value.errors[0]
|
||||
assert error["code"] == "duplicate_key" and error["line"] == 8 and error["path"] == "rules[0].targets"
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + RULE + " destination_port: [1]\n secret_ref: x\n")
|
||||
by_code = {e["code"]: e for e in info.value.errors}
|
||||
assert by_code["unknown_field"]["path"] == "rules[0].secret_ref" and by_code["unknown_field"]["line"] == 7
|
||||
assert by_code["unknown_field"]["params"] == {"field": "secret_ref"}
|
||||
assert by_code["bad_type"]["path"].startswith("rules[0].destination_port") and by_code["bad_type"]["line"] == 6
|
||||
|
||||
|
||||
def test_yaml_1_1_surprises_stay_text_or_are_type_errors():
|
||||
data, _ = load_yaml("a: on\nb: yes\nc: 1.5\nd: 2026-10-22\ne: 0755\nf: 1:30\ng: 0x1F\nh: .inf\ni: Off\nj: 1e3\n")
|
||||
assert data == {"a": "on", "b": "yes", "c": "1.5", "d": "2026-10-22", "e": "0755", "f": "1:30", "g": "0x1F",
|
||||
"h": ".inf", "i": "Off", "j": "1e3"}
|
||||
data, _ = load_yaml("a: true\nb: False\nc: null\nd: ~\ne:\nf: 53\ng: -1\nh: 0\n")
|
||||
assert data == {"a": True, "b": False, "c": None, "d": None, "e": None, "f": 53, "g": -1, "h": 0}
|
||||
assert "enabled" in _field_error(RULE + " enabled: yes\n") # strict bool: "yes" is not true
|
||||
|
||||
|
||||
def _field_error(body: str) -> str:
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + body)
|
||||
return " ".join(e["path"] for e in info.value.errors)
|
||||
|
||||
|
||||
def test_huge_integers_stay_text_and_do_not_hit_pythons_int_limit():
|
||||
data, _ = load_yaml("a: " + "9" * 5000 + "\n")
|
||||
assert data["a"] == "9" * 5000
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,code", [
|
||||
(b"a: \xff\xfe\n", "bad_encoding"),
|
||||
(b"a: \xc3\x28\n", "bad_encoding"),
|
||||
])
|
||||
def test_invalid_utf8_is_refused(raw, code):
|
||||
assert _code(raw) == ("CONFIG_INVALID", code)
|
||||
|
||||
|
||||
def test_a_bom_is_accepted():
|
||||
doc = parse("\ufeff" + HEAD).doc
|
||||
assert doc.rules is None
|
||||
|
||||
|
||||
def test_size_limits():
|
||||
assert _code(b"#" * (loader.MAX_BYTES + 1)) == ("CONFIG_TOO_LARGE", "too_large")
|
||||
assert _code("a: " + "x" * (loader.MAX_SCALAR + 1) + "\n") == ("CONFIG_INVALID", "scalar_too_long")
|
||||
many = "a: [" + ",".join(["1"] * (loader.MAX_NODES + 1)) + "]\n"
|
||||
assert _code(many) == ("CONFIG_INVALID", "too_many_nodes")
|
||||
|
||||
|
||||
def test_object_and_entry_limits():
|
||||
rules = "rules:\n" + "".join(f"- {{key: r-{i}, description: d, interface: lan}}\n" for i in range(loader.MAX_OBJECTS_PER_KIND + 1))
|
||||
assert "too_many_objects" in _codes(HEAD + rules)
|
||||
entries = "aliases:\n- name: A\n entries: [" + ",".join(f"10.0.{i // 250}.{i % 250}" for i in range(loader.MAX_ALIAS_ENTRIES + 1)) + "]\n"
|
||||
assert "too_many_entries" in _codes(HEAD + entries)
|
||||
long_entries = "aliases:\n- name: A\n entries: [" + ",".join(["x" * 999] * 9) + "]\n"
|
||||
assert "too_long" in _codes(HEAD + long_entries)
|
||||
variables = "variables:\n fw:\n" + "".join(f" V{i}: x\n" for i in range(loader.MAX_VARIABLES_PER_HOST + 1))
|
||||
assert "too_many_objects" in _codes(HEAD + variables)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,code,top", [
|
||||
("- a\n", "not_a_mapping", "CONFIG_INVALID"),
|
||||
("rules: []\n", "missing_version", "CONFIG_INVALID"),
|
||||
("opnexus_config: '1'\n", "missing_version", "CONFIG_INVALID"),
|
||||
("opnexus_config: true\n", "missing_version", "CONFIG_INVALID"),
|
||||
("opnexus_config: 2\n", "unsupported_version", "CONFIG_UNSUPPORTED_VERSION"),
|
||||
])
|
||||
def test_version_and_shape(raw, code, top):
|
||||
assert _code(raw) == (top, code)
|
||||
|
||||
|
||||
def test_error_values_are_shortened_and_escaped():
|
||||
with pytest.raises(ConfigError) as info:
|
||||
parse(HEAD + RULE + " direction: \"" + "\\u001b[2J" + "x" * 200 + "\"\n")
|
||||
value = info.value.errors[0]["params"]["value"]
|
||||
assert len(value) <= 80 and "\x1b" not in value and value.startswith("\\x1b")
|
||||
|
||||
|
||||
def test_one_mebibyte_parses_in_reasonable_time():
|
||||
"""Pure-Python PyYAML; design risk (4). A full-size file of realistic rules must stay well under the request timeout."""
|
||||
line = "- {key: r-%05d, description: Regel %05d mit etwas Text, interface: lan, source_net: 10.0.0.0/24, destination_port: '443'}\n"
|
||||
body = HEAD + "rules:\n"
|
||||
i = 0
|
||||
while len(body) + len(line) < loader.MAX_BYTES - 200 and i < loader.MAX_OBJECTS_PER_KIND:
|
||||
body += line % (i, i)
|
||||
i += 1
|
||||
body += "aliases:\n- name: BIG\n entries:\n" + "".join(f" - 10.{n // 65536}.{n // 256 % 256}.{n % 256}\n" for n in range(loader.MAX_ALIAS_ENTRIES))
|
||||
raw = body.encode()[: loader.MAX_BYTES]
|
||||
started = time.monotonic()
|
||||
try:
|
||||
parse(raw)
|
||||
except ConfigError:
|
||||
pass # truncation may cut the last line; the parse time is what counts
|
||||
elapsed = time.monotonic() - started
|
||||
print(f"parse {len(raw)} bytes: {elapsed:.2f}s")
|
||||
assert elapsed < 8
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Schema v1 (1.32 P3): strict, equal to the API limits, JSON Schema file up to date."""
|
||||
import typing
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.config_code import schema
|
||||
from app.schemas.alias import AliasCreate
|
||||
from app.schemas.one_to_one import OneToOneCreate
|
||||
from app.schemas.outbound_nat import OutboundNatCreate
|
||||
from app.schemas.policy_hierarchy import PolicyHierarchyRuleCreate
|
||||
from app.schemas.port_forward import PortForwardCreate
|
||||
from app.schemas.rule import RuleCreate
|
||||
|
||||
# (file model, API model, fields that exist only on one side)
|
||||
PAIRS = [
|
||||
(schema.RuleDoc, RuleCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids"}),
|
||||
(schema.PortForwardDoc, PortForwardCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
(schema.OneToOneDoc, OneToOneCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
(schema.OutboundNatDoc, OutboundNatCreate, {"key", "targets"}, {"target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
(schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, {"key", "group"}, {"hierarchy_group_id"}),
|
||||
(schema.AliasDoc, AliasCreate, {"entries", "targets"}, {"content", "target_host_ids", "target_group_ids", "policy_hierarchy_group_id"}),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("doc,api,only_doc,only_api", PAIRS, ids=[p[0].__name__ for p in PAIRS])
|
||||
def test_file_model_has_the_same_fields_defaults_and_limits_as_the_api_model(doc, api, only_doc, only_api):
|
||||
assert set(doc.model_fields) - only_doc == set(api.model_fields) - only_api
|
||||
for name in set(doc.model_fields) - only_doc:
|
||||
a, b = doc.model_fields[name], api.model_fields[name]
|
||||
assert a.default == b.default, name
|
||||
assert _limits(a) == _limits(b), name
|
||||
|
||||
|
||||
def _limits(field) -> dict:
|
||||
"""Length/pattern limits of a field, also when they sit on the str branch of `str | None`."""
|
||||
metas = list(field.metadata)
|
||||
for arg in typing.get_args(field.annotation):
|
||||
metas += list(getattr(arg, "__metadata__", ()))
|
||||
metas = [m for meta in metas for m in (meta.metadata if hasattr(meta, "metadata") else [meta])] # unwrap Field(...)
|
||||
out = {}
|
||||
for meta in metas:
|
||||
for attr in ("min_length", "max_length", "pattern"):
|
||||
if getattr(meta, attr, None) is not None:
|
||||
out[attr] = getattr(meta, attr)
|
||||
return out
|
||||
|
||||
|
||||
def test_enumerations_match_the_api():
|
||||
for doc, api, name in ((schema.RuleDoc, RuleCreate, "action"), (schema.PortForwardDoc, PortForwardCreate, "natreflection"),
|
||||
(schema.OneToOneDoc, OneToOneCreate, "natreflection"), (schema.OneToOneDoc, OneToOneCreate, "type"),
|
||||
(schema.AliasDoc, AliasCreate, "type"), (schema.HierarchyRuleDoc, PolicyHierarchyRuleCreate, "rulebase")):
|
||||
assert doc.model_fields[name].annotation == api.model_fields[name].annotation, (doc.__name__, name)
|
||||
|
||||
|
||||
def _rule(**over):
|
||||
return {"key": "r-1", "description": "d", "interface": "lan", **over}
|
||||
|
||||
|
||||
def test_unknown_fields_wrong_types_and_secret_ref_are_refused():
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(secret_ref="x"))
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(enabled="yes")) # strict: no truthy strings
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(source_port=True)) # bool is never a port
|
||||
with pytest.raises(ValidationError):
|
||||
schema.ConfigDoc.model_validate({"opnexus_config": 2})
|
||||
|
||||
|
||||
def test_ports_accept_numbers_and_ranges_and_become_strings():
|
||||
assert schema.RuleDoc.model_validate(_rule(destination_port=53)).destination_port == "53"
|
||||
assert schema.RuleDoc.model_validate(_rule(destination_port="8000-8100")).destination_port == "8000-8100"
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(destination_port="1" * 33))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["a\x00b", "a\x1b[31m", "a\x7fb", "a\rb", "a\x0bb"])
|
||||
def test_control_characters_are_refused(bad):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(description=bad))
|
||||
with pytest.raises(ValidationError):
|
||||
schema.AliasDoc.model_validate({"name": "A", "entries": [bad]})
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(interface=bad))
|
||||
|
||||
|
||||
def test_descriptions_may_have_several_lines_and_padding_but_names_and_keys_not():
|
||||
assert schema.RuleDoc.model_validate(_rule(description="a\n\tb ")).description == "a\n\tb "
|
||||
assert schema.DeviceGroupDoc.model_validate({"name": "G", "description": " zwei\nZeilen "}).description == " zwei\nZeilen "
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(interface="a\nb")) # a newline only in free text
|
||||
for bad in (" G", "G "):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.DeviceGroupDoc.model_validate({"name": bad})
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(targets={"hosts": [" fw"]}))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", ["", "Upper", "-lead", "a b", "x" * 65, "ü"])
|
||||
def test_keys_must_be_lowercase_slugs(key):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.RuleDoc.model_validate(_rule(key=key))
|
||||
|
||||
|
||||
def test_variable_names_and_values_use_the_same_rules_as_the_host_page():
|
||||
ok = {"opnexus_config": 1, "variables": {"fw": {"LAN_NET": "10.0.0.0/24"}}}
|
||||
assert schema.ConfigDoc.model_validate(ok).variables == {"fw": {"LAN_NET": "10.0.0.0/24"}}
|
||||
for bad in ({"lan": "x"}, {"LAN": "{{X}}"}, {"LAN": "a\nb"}, {"LAN": ""}):
|
||||
with pytest.raises(ValidationError):
|
||||
schema.ConfigDoc.model_validate({"opnexus_config": 1, "variables": {"fw": bad}})
|
||||
|
||||
|
||||
def test_a_missing_section_is_not_an_empty_section():
|
||||
doc = schema.ConfigDoc.model_validate({"opnexus_config": 1, "aliases": []})
|
||||
assert doc.aliases == [] and doc.rules is None
|
||||
|
||||
|
||||
def test_checked_in_json_schema_is_current():
|
||||
path = Path(schema.__file__).with_name(schema.SCHEMA_FILE)
|
||||
assert path.read_text(encoding="utf-8") == schema.json_schema_text(), (
|
||||
"Schema geändert: `docker compose run --rm --no-deps -v \"$PWD/backend:/app\" backend python -m app.config_code schema "
|
||||
f"> backend/app/config_code/{schema.SCHEMA_FILE}`"
|
||||
)
|
||||
@@ -0,0 +1,266 @@
|
||||
"""Way back to the previous firmware (1.32, B5): rollback point before every update, guided rollback."""
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import backup_store, scheduled_maintenance_service
|
||||
from tests.test_maintenance import execute, preview
|
||||
from tests.test_rules import _create_host
|
||||
from tests.test_scheduled_maintenance import _due_update
|
||||
|
||||
BASE = "https://fw1.test"
|
||||
|
||||
|
||||
class Snapshots:
|
||||
"""A firewall with ZFS boot environments, as Core/Api/SnapshotsController answers."""
|
||||
|
||||
def __init__(self, mock, *, version="26.7.6", status="update", fail_add=False, hide_new=False, fail_reboot=False,
|
||||
mark_on_activate=True):
|
||||
self.rows = [{"uuid": "be-default", "name": "default", "active": "NR", "created_str": "2026-10-01 10:00"}]
|
||||
self.fail_add, self.hide_new, self.fail_reboot, self.mark_on_activate = fail_add, hide_new, fail_reboot, mark_on_activate
|
||||
self.activated: list[str] = []
|
||||
mock.get(f"{BASE}/api/diagnostics/interface/getVipStatus").respond(200, json={"rows": []})
|
||||
mock.get(f"{BASE}/api/core/firmware/info").respond(200, json={"product_version": version})
|
||||
mock.get(f"{BASE}/api/core/firmware/status").respond(200, json={"status": status, "all_packages": {}, "status_msg": "ok"})
|
||||
mock.get(f"{BASE}/api/core/firmware/running").respond(200, json={"status": "ready"})
|
||||
mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": True})
|
||||
self.fail_list = False
|
||||
mock.get(f"{BASE}/api/core/snapshots/search").mock(side_effect=self._search)
|
||||
self.add = mock.post(f"{BASE}/api/core/snapshots/add").mock(side_effect=self._add)
|
||||
self.delete = mock.post(url__regex=rf"{BASE}/api/core/snapshots/del/.*").mock(side_effect=self._del)
|
||||
self.activate = mock.post(url__regex=rf"{BASE}/api/core/snapshots/activate/.*").mock(side_effect=self._activate)
|
||||
self.update = mock.post(f"{BASE}/api/core/firmware/update").respond(200, json={"status": "ok"})
|
||||
self.upgrade = mock.post(f"{BASE}/api/core/firmware/upgrade").respond(200, json={"status": "ok"})
|
||||
self.reboot = mock.post(f"{BASE}/api/core/system/reboot").mock(side_effect=self._reboot)
|
||||
self.download = mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
|
||||
|
||||
def _search(self, request):
|
||||
if self.fail_list:
|
||||
return httpx.Response(502, json={})
|
||||
return httpx.Response(200, json={"rows": self.rows})
|
||||
|
||||
def point(self, name: str, active: str = "-") -> None:
|
||||
self.rows.append({"uuid": f"be-{name}", "name": name, "active": active, "created_str": "x"})
|
||||
|
||||
def _add(self, request):
|
||||
if self.fail_add:
|
||||
return httpx.Response(200, json={"status": "failed", "validations": {"snapshot.name": "nope"}})
|
||||
import json
|
||||
name = json.loads(request.content)["name"]
|
||||
if not self.hide_new:
|
||||
self.point(name)
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
def _del(self, request):
|
||||
uid = str(request.url).rsplit("/", 1)[-1]
|
||||
self.rows = [r for r in self.rows if r["uuid"] != uid]
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
def _activate(self, request):
|
||||
uid = str(request.url).rsplit("/", 1)[-1]
|
||||
self.activated.append(uid)
|
||||
if getattr(self, "fail_list_after_activate", False) and len(self.activated) == 1:
|
||||
self.fail_list = True
|
||||
if self.mark_on_activate:
|
||||
for r in self.rows:
|
||||
r["active"] = ("N" if "N" in r["active"] else "") + ("R" if r["uuid"] == uid else "")
|
||||
r["active"] = r["active"] or "-"
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
def _reboot(self, request):
|
||||
if self.fail_reboot:
|
||||
return httpx.Response(500, json={})
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
|
||||
async def _host_with_backup(client):
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
backup_store.save_snapshot(uuid.UUID(h["id"]), b"<opnsense/>")
|
||||
return h
|
||||
|
||||
|
||||
async def _op(op_id) -> HostOperation:
|
||||
async with SessionLocal() as db:
|
||||
return await db.get(HostOperation, uuid.UUID(str(op_id)))
|
||||
|
||||
|
||||
async def test_a_manual_update_creates_a_rollback_point_first_and_keeps_only_two(client, respx_mock):
|
||||
fw = Snapshots(respx_mock)
|
||||
fw.point("opnexus-pre-26.7.4-20261001-100000")
|
||||
fw.point("opnexus-pre-26.7.5-20261010-100000")
|
||||
fw.point("handmade") # never touched
|
||||
h = await _host_with_backup(client)
|
||||
p = await preview(client, h, "update")
|
||||
r = await execute(client, h, p.json())
|
||||
assert r.json()["status"] == "submitted" and fw.update.call_count == 1
|
||||
names = [row["name"] for row in fw.rows]
|
||||
new = next(n for n in names if n.startswith("opnexus-pre-26.7.6-"))
|
||||
assert "opnexus-pre-26.7.4-20261001-100000" not in names # the oldest of three is cleaned up
|
||||
assert {"default", "handmade", "opnexus-pre-26.7.5-20261010-100000", new} == set(names)
|
||||
op = await _op(r.json()["id"])
|
||||
assert op.snapshot["rollback_point"]["name"] == new and op.snapshot["rollback_point"]["version"] == "26.7.6"
|
||||
assert fw.add.call_count == 1
|
||||
order = [str(c.request.url) for c in respx_mock.calls if c.request.method == "POST"]
|
||||
assert order.index(f"{BASE}/api/core/snapshots/add") < order.index(f"{BASE}/api/core/firmware/update")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("problem", ["fail_add", "hide_new"])
|
||||
async def test_no_rollback_point_no_update(client, respx_mock, problem):
|
||||
fw = Snapshots(respx_mock, **{problem: True})
|
||||
h = await _host_with_backup(client)
|
||||
p = await preview(client, h, "update")
|
||||
r = await execute(client, h, p.json())
|
||||
assert r.json()["status"] == "cancelled" and "Update nicht gestartet" in r.json()["message"]
|
||||
assert fw.update.call_count == 0
|
||||
|
||||
|
||||
async def test_without_zfs_the_update_runs_and_says_there_is_no_way_back(client, respx_mock):
|
||||
fw = Snapshots(respx_mock)
|
||||
respx_mock.get(f"{BASE}/api/core/snapshots/isSupported").respond(200, json={"supported": False})
|
||||
h = await _host_with_backup(client)
|
||||
r = await execute(client, h, (await preview(client, h, "update")).json())
|
||||
assert r.json()["status"] == "submitted" and "Kein Rückweg-Punkt" in r.json()["message"]
|
||||
assert fw.add.call_count == 0 and fw.update.call_count == 1
|
||||
|
||||
|
||||
async def test_a_scheduled_update_fails_closed_without_a_rollback_point(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, fail_add=True)
|
||||
respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
sched_id = await _due_update(h)
|
||||
async with SessionLocal() as db:
|
||||
await scheduled_maintenance_service.fire_due(db)
|
||||
async with SessionLocal() as db:
|
||||
sched = await db.get(ScheduledMaintenance, sched_id)
|
||||
assert sched.status == "failed" and "Rückweg-Punkt" in sched.result["error"]
|
||||
assert fw.update.call_count == 0
|
||||
|
||||
|
||||
async def test_rollback_needs_a_point_from_opnexus(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("handmade")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await preview(client, h, "firmware_rollback")
|
||||
assert r.status_code == 409 and "Kein Rückweg-Punkt" in r.json()["detail"]["params"]["reason"]
|
||||
assert (await client.post(f"/api/hosts/{h['id']}/snapshots", json={"name": "opnexus-pre-1-20260101-000000"})).status_code == 422
|
||||
|
||||
|
||||
async def test_rollback_activates_the_newest_point_reboots_and_lists_what_is_lost(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, version="26.7.7", status="none")
|
||||
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
|
||||
fw.point(f"opnexus-pre-26.7.5-{(stamp - timedelta(days=5)):%Y%m%d-%H%M%S}")
|
||||
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
async with SessionLocal() as db:
|
||||
rule = FirewallRule(description="nachher", interface="lan", direction="in")
|
||||
db.add(rule)
|
||||
await db.flush()
|
||||
db.add(RuleDeployment(rule_id=rule.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u"))
|
||||
await db.commit()
|
||||
p = await preview(client, h, "firmware_rollback")
|
||||
assert p.status_code == 200, p.text
|
||||
rollback = p.json()["state"]["rollback"]
|
||||
assert rollback["target"]["version"] == "26.7.6" and rollback["current"]["name"] == "default"
|
||||
assert [c["kind"] for c in rollback["changes_since"]] == ["rule"]
|
||||
r = await execute(client, h, p.json())
|
||||
assert r.json()["status"] == "submitted" and "erwartet 26.7.6" in r.json()["message"]
|
||||
assert fw.activated == [rollback["target"]["uuid"]] and fw.reboot.call_count == 1
|
||||
|
||||
|
||||
async def test_rollback_that_is_not_marked_for_the_next_start_never_reboots(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none", mark_on_activate=False)
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
|
||||
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
|
||||
|
||||
|
||||
async def test_a_failed_reboot_reactivates_the_current_boot_environment(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none", fail_reboot=True)
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "unknown"
|
||||
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
|
||||
assert next(row for row in fw.rows if row["uuid"] == "be-default")["active"].endswith("R")
|
||||
|
||||
|
||||
async def test_rollback_on_a_carp_master_is_refused_like_a_reboot(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
|
||||
r = await preview(client, h, "firmware_rollback")
|
||||
assert r.status_code == 409 and fw.activate.call_count == 0
|
||||
|
||||
|
||||
async def test_the_rollback_endpoint_needs_the_deploy_right_like_every_maintenance_execute():
|
||||
from app.services import api_token_service
|
||||
|
||||
assert api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/execute")
|
||||
assert not api_token_service.is_rollout("POST", "/api/hosts/x/maintenance/preview")
|
||||
|
||||
|
||||
async def test_rollback_saves_the_current_configuration_first_and_refuses_without_it(client, respx_mock):
|
||||
"""Going back discards every change since the point on the firewall: OPNexus keeps a copy of the config first."""
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
download = respx_mock.get(f"{BASE}/api/core/backup/download/this").respond(200, content=b"<opnsense>now</opnsense>")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "submitted" and download.call_count == 1
|
||||
assert backup_store.list_snapshots(uuid.UUID(h["id"])) # stored locally
|
||||
order = [str(c.request.url) for c in respx_mock.calls]
|
||||
assert order.index(f"{BASE}/api/core/backup/download/this") < order.index(f"{BASE}/api/core/snapshots/activate/be-opnexus-pre-26.7.6-20261020-100000")
|
||||
|
||||
|
||||
async def test_rollback_without_a_configuration_copy_does_not_start(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
respx_mock.get(f"{BASE}/api/core/backup/download/this").mock(side_effect=httpx.ConnectError("down"))
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "cancelled" and fw.activate.call_count == 0 and fw.reboot.call_count == 0
|
||||
|
||||
|
||||
async def test_an_unreadable_list_after_activating_switches_back_and_never_reboots(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
fw.fail_list_after_activate = True
|
||||
fw.point("opnexus-pre-26.7.6-20261020-100000")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "unknown" and fw.reboot.call_count == 0
|
||||
assert fw.activated == ["be-opnexus-pre-26.7.6-20261020-100000", "be-default"]
|
||||
|
||||
|
||||
async def test_after_a_rollback_newer_deployments_are_marked_so_the_next_deploy_really_writes(client, respx_mock):
|
||||
fw = Snapshots(respx_mock, status="none")
|
||||
stamp = datetime.now(timezone.utc) - timedelta(hours=2)
|
||||
fw.point(f"opnexus-pre-26.7.6-{stamp:%Y%m%d-%H%M%S}")
|
||||
h = await _create_host(client, "fw1", BASE)
|
||||
async with SessionLocal() as db:
|
||||
old, new = FirewallRule(description="alt", interface="lan", direction="in"), FirewallRule(description="neu", interface="lan", direction="in")
|
||||
db.add_all([old, new])
|
||||
await db.flush()
|
||||
db.add_all([
|
||||
RuleDeployment(rule_id=old.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u1",
|
||||
last_payload={"x": 1}, deployed_at=stamp - timedelta(days=1)),
|
||||
RuleDeployment(rule_id=new.id, host_id=uuid.UUID(h["id"]), status="success", opnsense_rule_uuid="u2",
|
||||
last_payload={"x": 2}, deployed_at=stamp + timedelta(minutes=30)),
|
||||
])
|
||||
await db.commit()
|
||||
r = await execute(client, h, (await preview(client, h, "firmware_rollback")).json())
|
||||
assert r.json()["status"] == "submitted" and "1 Deployment(s) als unklar markiert" in r.json()["message"]
|
||||
async with SessionLocal() as db:
|
||||
rows = {d.opnsense_rule_uuid: d for d in (await db.scalars(select(RuleDeployment))).all()}
|
||||
assert rows["u1"].status == "success" and rows["u1"].last_payload == {"x": 1} # older than the point: untouched
|
||||
assert rows["u2"].status == "error" and rows["u2"].last_payload is None and "Firmware-Rückweg" in rows["u2"].message
|
||||
@@ -14,6 +14,7 @@ def mock_reads(mock,base='https://fw1.test',carp=None,status='none',running='rea
|
||||
mock.get(f'{base}/api/core/firmware/info').respond(200,json={'product_version':version})
|
||||
mock.get(f'{base}/api/core/firmware/status').respond(200,json={'status':status,'all_packages':{},'status_msg':'cached result'})
|
||||
mock.get(f'{base}/api/core/firmware/running').respond(200,json={'status':running})
|
||||
mock.get(f'{base}/api/core/snapshots/isSupported').respond(200,json={'supported':False}) # UFS: no rollback point (1.32)
|
||||
|
||||
async def preview(client,h,action='reboot'):
|
||||
return await client.post(f"/api/hosts/{h['id']}/maintenance/preview",json={'action':action})
|
||||
|
||||
@@ -378,7 +378,7 @@ async def test_0028_adds_the_event_window_and_is_idempotent(scratch):
|
||||
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'rule_change_events'"))).scalars().all())
|
||||
assert "window_start" not in cols
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() == "0028" and await _schema_diff(scratch) == []
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
await _alembic(scratch, "downgrade", "0027")
|
||||
async with scratch.connect() as conn:
|
||||
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'rule_change_events'"))).scalars().all())
|
||||
@@ -386,3 +386,113 @@ async def test_0028_adds_the_event_window_and_is_idempotent(scratch):
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
|
||||
|
||||
async def test_0029_creates_the_rule_cleanup_tables_and_settings_and_is_idempotent(scratch):
|
||||
"""1.30 -> 1.31: two tables with a partial unique index (one unfinished item per rule and node) and three
|
||||
columns on governance_settings; the downgrade removes all of it again."""
|
||||
await _alembic(scratch, "upgrade", "0028")
|
||||
async with scratch.begin() as conn:
|
||||
await conn.execute(text("INSERT INTO governance_settings (id, operators_may_deploy) VALUES (1, true)"))
|
||||
tables = set((await conn.execute(text("SELECT tablename FROM pg_tables WHERE schemaname = 'public'"))).scalars().all())
|
||||
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'governance_settings'"))).scalars().all())
|
||||
assert "rule_cleanup_operations" not in tables and "rule_cleanup_items" not in tables
|
||||
assert "cleanup_max_rules" not in cols
|
||||
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
async with scratch.connect() as conn: # the existing settings row got the defaults of the design
|
||||
row = (await conn.execute(text(
|
||||
"SELECT cleanup_max_rules, cleanup_delete_min_days, cleanup_management_sources, cleanup_management_ports "
|
||||
"FROM governance_settings"))).one()
|
||||
assert tuple(row) == (10, 7, [], [])
|
||||
|
||||
host, op_id = "11111111-1111-1111-1111-111111111111", "22222222-2222-2222-2222-222222222222"
|
||||
async with scratch.begin() as conn:
|
||||
await conn.execute(text(
|
||||
"INSERT INTO opnsense_hosts (id, name, base_url, api_key, verify_tls, enabled, created_at) "
|
||||
"VALUES (:h, 'fw', 'https://fw.test', 'k', true, true, now())"), {"h": host})
|
||||
await conn.execute(text(
|
||||
"INSERT INTO rule_cleanup_operations (id, host_id, kind, status, actor, created_at, expires_at, preview_fingerprint, options) "
|
||||
"VALUES (:o, :h, 'disable', 'preview', 'admin', now(), now(), 'f', '{}')"), {"o": op_id, "h": host})
|
||||
|
||||
async def item(conn, rule: str, state: str):
|
||||
await conn.execute(text(
|
||||
"INSERT INTO rule_cleanup_items (id, operation_id, host_id, rule_uuid, seq, state, description, categories) "
|
||||
f"VALUES (gen_random_uuid(), :o, :h, '{rule}', 0, '{state}', 'r', '[]')"), {"o": op_id, "h": host})
|
||||
|
||||
async with scratch.begin() as conn:
|
||||
await item(conn, "r1", "disabled")
|
||||
await item(conn, "r2", "disabled") # another rule is fine
|
||||
await item(conn, "r1", "confirmed") # finished items do not count
|
||||
await item(conn, "r1", "reverted")
|
||||
for blocked in ("queued", "unknown", "revert_failed", "deleting"): # r1 is still open as "disabled"
|
||||
with pytest.raises(DBAPIError):
|
||||
async with scratch.begin() as conn:
|
||||
await item(conn, "r1", blocked)
|
||||
async with scratch.begin() as conn: # on another node the same UUID is a different rule
|
||||
await conn.execute(text(
|
||||
"INSERT INTO opnsense_hosts (id, name, base_url, api_key, verify_tls, enabled, created_at) "
|
||||
"VALUES ('33333333-3333-3333-3333-333333333333', 'fw2', 'https://fw2.test', 'k', true, true, now())"))
|
||||
await conn.execute(text(
|
||||
"INSERT INTO rule_cleanup_items (id, operation_id, host_id, rule_uuid, seq, state, description, categories) "
|
||||
"VALUES (gen_random_uuid(), :o, '33333333-3333-3333-3333-333333333333', 'r1', 0, 'disabled', 'r', '[]')"), {"o": op_id})
|
||||
async with scratch.begin() as conn: # the host takes its operations and their items with it
|
||||
await conn.execute(text("DELETE FROM opnsense_hosts WHERE id = :h"), {"h": host})
|
||||
assert (await conn.execute(text("SELECT count(*) FROM rule_cleanup_operations"))).scalar() == 0
|
||||
assert (await conn.execute(text("SELECT count(*) FROM rule_cleanup_items"))).scalar() == 0
|
||||
|
||||
await _alembic(scratch, "downgrade", "0028")
|
||||
async with scratch.connect() as conn:
|
||||
tables = set((await conn.execute(text("SELECT tablename FROM pg_tables WHERE schemaname = 'public'"))).scalars().all())
|
||||
cols = set((await conn.execute(text("SELECT column_name FROM information_schema.columns WHERE table_name = 'governance_settings'"))).scalars().all())
|
||||
assert "rule_cleanup_operations" not in tables and "rule_cleanup_items" not in tables
|
||||
assert not {"cleanup_max_rules", "cleanup_delete_min_days", "cleanup_management_sources", "cleanup_management_ports"} & cols
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head") # second run: nothing to do
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
|
||||
|
||||
async def test_0030_backfills_config_keys_and_is_idempotent(scratch):
|
||||
"""1.31 -> 1.32: every rule/NAT table gets a unique, readable key derived from the UUID; the downgrade drops it."""
|
||||
await _alembic(scratch, "upgrade", "0029")
|
||||
rule_id = "3f2a9c1d-0000-4000-8000-000000000001"
|
||||
async with scratch.begin() as conn:
|
||||
await conn.execute(text(
|
||||
"INSERT INTO firewall_rules (id, description, interface, direction, ipprotocol, protocol, action, "
|
||||
"source_net, destination_net, enabled, log, created_at, updated_at) VALUES "
|
||||
f"('{rule_id}', 'r', 'lan', 'in', 'inet', 'any', 'pass', 'any', 'any', true, false, now(), now())"))
|
||||
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
async with scratch.connect() as conn:
|
||||
key = (await conn.execute(text("SELECT config_key FROM firewall_rules"))).scalar_one()
|
||||
nullable = (await conn.execute(text(
|
||||
"SELECT is_nullable FROM information_schema.columns "
|
||||
"WHERE table_name = 'port_forward_rules' AND column_name = 'config_key'"))).scalar_one()
|
||||
assert key == "r-3f2a9c1d0000" and nullable == "NO"
|
||||
|
||||
await _alembic(scratch, "downgrade", "0029")
|
||||
async with scratch.connect() as conn:
|
||||
cols = set((await conn.execute(text(
|
||||
"SELECT column_name FROM information_schema.columns WHERE table_name = 'firewall_rules'"))).scalars().all())
|
||||
assert "config_key" not in cols
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
|
||||
|
||||
async def test_0031_adds_gate_probes_to_existing_plans_and_is_idempotent(scratch):
|
||||
await _alembic(scratch, "upgrade", "0030")
|
||||
async with scratch.begin() as conn:
|
||||
await conn.execute(text(
|
||||
"INSERT INTO rollout_plans (id, name, description, action, auto_start, gate_checks, compliance_mode, created_by, created_at, updated_at) "
|
||||
"VALUES (gen_random_uuid(), 'p', '', 'update', false, '{}', 'warn', 'admin', now(), now())"))
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
async with scratch.connect() as conn:
|
||||
assert (await conn.execute(text("SELECT gate_probes::text FROM rollout_plans"))).scalar_one() == "[]"
|
||||
await _alembic(scratch, "downgrade", "0030")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head() and await _schema_diff(scratch) == []
|
||||
@@ -0,0 +1,16 @@
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def test_importing_models_alone_configures_all_mappers():
|
||||
"""`python -m app.reset_password` / `app.reset_2fa` import only part of the app: the mapper registry must still be complete."""
|
||||
code = "import app.models; from sqlalchemy.orm import configure_mappers; configure_mappers()"
|
||||
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
|
||||
assert result.returncode == 0, result.stderr[-800:]
|
||||
|
||||
|
||||
def test_reset_scripts_import_cleanly_and_configure_mappers():
|
||||
for module in ("app.reset_password", "app.reset_2fa"):
|
||||
code = f"import {module}; from sqlalchemy.orm import configure_mappers; configure_mappers()"
|
||||
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60)
|
||||
assert result.returncode == 0, (module, result.stderr[-800:])
|
||||
@@ -146,9 +146,9 @@ def test_what_counts_as_a_rollout(method, path, rollout):
|
||||
async def test_with_separation_of_duties_operators_prepare_but_only_admins_deploy(client):
|
||||
op = await _operator(client, "dutyop")
|
||||
try:
|
||||
assert (await op.get("/api/settings/governance")).json() == {"operators_may_deploy": True}
|
||||
assert (await op.get("/api/settings/governance")).json()["operators_may_deploy"] is True
|
||||
assert (await op.put("/api/settings/governance", json={"operators_may_deploy": False})).status_code == 403
|
||||
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": False})).json() == {"operators_may_deploy": False}
|
||||
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": False})).json()["operators_may_deploy"] is False
|
||||
# Preparing still works ...
|
||||
assert (await op.post("/api/groups", json={"name": "prepared"})).status_code == 201
|
||||
# ... rolling out does not, whatever the target (the 403 comes before any lookup).
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
"""Client calls for filter rules that the cleanup assistant (1.31) builds on: toggleRule, getRule, the
|
||||
getRule -> addRule payload, and the request shape / failure classification of add/del/move/apply.
|
||||
|
||||
The JSON fixtures are shaped like live answers of OPNsense 26.7.7 (checked 2026-10-22 on a test firewall) with
|
||||
documentation addresses and made-up UUIDs. Evidence for each endpoint: comment block in services/opnsense_client.py.
|
||||
"""
|
||||
import json
|
||||
import uuid
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import opnsense_client as c
|
||||
|
||||
BASE = "https://fw.test/api/firewall/filter"
|
||||
RULE = "5c6f3bfa-1718-4216-81c8-04a75527960f"
|
||||
OTHER = "74190f58-f35f-4402-8794-5410e99ea364"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def host(monkeypatch):
|
||||
monkeypatch.setattr(c.opnsense_secret_store, "read_secret", lambda host_id: "secret")
|
||||
return OpnsenseHost(id=uuid.uuid4(), name="fw-a", base_url="https://fw.test", api_key="key", verify_tls=False)
|
||||
|
||||
|
||||
def _opts(selected: list[str], options: dict[str, str]) -> dict:
|
||||
return {k: {"value": v, "selected": 1 if k in selected else 0} for k, v in options.items()}
|
||||
|
||||
|
||||
def live_rule() -> dict:
|
||||
"""A `getRule/<uuid>` record: select fields as option dicts, `categories` an option dict, read-only extras."""
|
||||
return {
|
||||
"enabled": "1", "sequence": "566", "sort_order": "200000.0000566", "prio_group": "200000", "quick": "1",
|
||||
"action": _opts(["pass"], {"pass": "Pass", "block": "Block", "reject": "Reject"}),
|
||||
"interface": _opts(["lan", "wan"], {"lan": "LAN", "wan": "WAN", "wireguard": "WireGuard (Group)"}),
|
||||
"direction": _opts(["in"], {"in": "In", "out": "Out", "any": "Both"}),
|
||||
"protocol": _opts(["TCP"], {"any": "any", "TCP": "TCP", "UDP": "UDP"}),
|
||||
"gateway": _opts([""], {"": "None", "WAN_GW": "WAN_GW - 203.0.113.1"}),
|
||||
"categories": _opts(["c1", "c2"], {"c1": "one", "c2": "two", "c3": "three"}),
|
||||
"tcpflags1": _opts([], {"syn": "SYN", "ack": "ACK"}),
|
||||
"source_net": "bogons", "destination_net": "203.0.113.9", "destination_port": "8080-8443", "source_port": "",
|
||||
"description": "opnexus-b4-probe", "statetimeout": None, "log": 1,
|
||||
"audit": {"created": {"username": "", "time": "", "description": ""}, "userdata": {"note": ""}},
|
||||
}
|
||||
|
||||
|
||||
def _body(route) -> dict:
|
||||
return json.loads(route.calls.last.request.content)
|
||||
|
||||
|
||||
# ── toggle_rule ────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("enabled, tail, answer", [(False, "0", "Disabled"), (True, "1", "Enabled")])
|
||||
async def test_toggle_rule_sends_the_explicit_value_and_returns_the_answer(host, respx_mock, enabled, tail, answer):
|
||||
route = respx_mock.post(f"{BASE}/toggleRule/{RULE}/{tail}").mock(
|
||||
return_value=httpx.Response(200, json={"result": answer, "changed": True}))
|
||||
assert await c.toggle_rule(host, RULE, enabled) == {"result": answer, "changed": True}
|
||||
assert route.call_count == 1 and not route.calls.last.request.content # never the flipping form, no body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_toggle_rule_is_idempotent_an_unchanged_answer_is_success(host, respx_mock):
|
||||
respx_mock.post(f"{BASE}/toggleRule/{RULE}/0").mock(
|
||||
return_value=httpx.Response(200, json={"result": "Disabled", "changed": False}))
|
||||
assert (await c.toggle_rule(host, RULE, False))["changed"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("bad", [0, 1, "0", "1", None, "false"])
|
||||
async def test_toggle_rule_refuses_anything_but_a_bool_without_a_request(host, respx_mock, bad):
|
||||
route = respx_mock.post(url__regex=r".*/toggleRule/.*").mock(return_value=httpx.Response(200, json={}))
|
||||
with pytest.raises(TypeError):
|
||||
await c.toggle_rule(host, RULE, bad)
|
||||
assert route.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_toggle_rule_answer_for_the_other_state_is_not_accepted(host, respx_mock):
|
||||
respx_mock.post(f"{BASE}/toggleRule/{RULE}/0").mock(return_value=httpx.Response(200, json={"result": "Enabled", "changed": True}))
|
||||
with pytest.raises(c.OpnsenseAPIError) as info:
|
||||
await c.toggle_rule(host, RULE, False)
|
||||
assert not isinstance(info.value, c.OpnsenseAmbiguousError)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("answer", [{"result": "failed"}, {"result": "failed", "changed": False}])
|
||||
async def test_toggle_rule_unknown_uuid_is_a_plain_refusal(host, respx_mock, answer):
|
||||
"""Live: an unknown or legacy UUID answers HTTP 200 {"result": "failed"} (no "changed" key)."""
|
||||
respx_mock.post(f"{BASE}/toggleRule/{RULE}/0").mock(return_value=httpx.Response(200, json=answer))
|
||||
with pytest.raises(c.OpnsenseAPIError) as info:
|
||||
await c.toggle_rule(host, RULE, False)
|
||||
assert type(info.value) is c.OpnsenseAPIError
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_toggle_rule_failure_classes(host, respx_mock):
|
||||
route = respx_mock.post(f"{BASE}/toggleRule/{RULE}/0")
|
||||
route.mock(return_value=httpx.Response(403, json={"status": 403, "message": "Forbidden"}))
|
||||
with pytest.raises(c.OpnsenseForbiddenError):
|
||||
await c.toggle_rule(host, RULE, False)
|
||||
route.mock(return_value=httpx.Response(500, text="boom"))
|
||||
with pytest.raises(c.OpnsenseAmbiguousError): # the change may have been saved
|
||||
await c.toggle_rule(host, RULE, False)
|
||||
route.mock(side_effect=httpx.ReadTimeout("slow"))
|
||||
with pytest.raises(c.OpnsenseAmbiguousError):
|
||||
await c.toggle_rule(host, RULE, False)
|
||||
route.mock(side_effect=httpx.ConnectError("down")) # never sent: a plain error, not an unknown outcome
|
||||
with pytest.raises(c.OpnsenseAPIError) as info:
|
||||
await c.toggle_rule(host, RULE, False)
|
||||
assert not isinstance(info.value, c.OpnsenseAmbiguousError)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("bad", ["../x", "a/b", "a?b", "", "x" * 65, "a b"])
|
||||
async def test_rule_references_cannot_leave_the_url_path(host, respx_mock, bad):
|
||||
route = respx_mock.route().mock(return_value=httpx.Response(200, json={}))
|
||||
with pytest.raises(c.OpnsenseAPIError):
|
||||
await c.toggle_rule(host, bad, False)
|
||||
with pytest.raises(c.OpnsenseAPIError):
|
||||
await c.get_rule(host, bad)
|
||||
assert route.call_count == 0
|
||||
|
||||
|
||||
# ── get_rule ───────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_rule_returns_the_inner_record(host, respx_mock):
|
||||
route = respx_mock.get(f"{BASE}/getRule/{RULE}").mock(return_value=httpx.Response(200, json={"rule": live_rule()}))
|
||||
rule = await c.get_rule(host, RULE)
|
||||
assert route.call_count == 1 and rule["description"] == "opnexus-b4-probe" and rule["interface"]["lan"]["selected"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("answer", [[], {}])
|
||||
async def test_get_rule_empty_answer_means_not_found(host, respx_mock, answer):
|
||||
"""Live: unknown, legacy (hex label) and automatic UUIDs answer HTTP 200 with a bare `[]`."""
|
||||
respx_mock.get(f"{BASE}/getRule/{RULE}").mock(return_value=httpx.Response(200, json=answer))
|
||||
with pytest.raises(c.OpnsenseNotFoundError):
|
||||
await c.get_rule(host, RULE)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_rule_failure_classes(host, respx_mock):
|
||||
route = respx_mock.get(f"{BASE}/getRule/{RULE}")
|
||||
route.mock(return_value=httpx.Response(200, json={"rule": "nonsense"}))
|
||||
with pytest.raises(c.OpnsenseAPIError) as info:
|
||||
await c.get_rule(host, RULE)
|
||||
assert not isinstance(info.value, c.OpnsenseNotFoundError) # an unreadable answer is not proof that the rule is gone
|
||||
route.mock(return_value=httpx.Response(403, text="no"))
|
||||
with pytest.raises(c.OpnsenseForbiddenError):
|
||||
await c.get_rule(host, RULE)
|
||||
route.mock(return_value=httpx.Response(502, text="bad gateway"))
|
||||
with pytest.raises(c.OpnsenseAmbiguousError):
|
||||
await c.get_rule(host, RULE)
|
||||
|
||||
|
||||
# ── rule_write_payload (getRule -> addRule) ───────────────────────────────
|
||||
|
||||
def test_rule_write_payload_reduces_selects_and_drops_read_only_fields():
|
||||
payload = c.rule_write_payload(live_rule())
|
||||
assert payload["interface"] == "lan,wan" # multi-interface: comma separated
|
||||
assert payload["categories"] == "c1,c2"
|
||||
assert payload["action"] == "pass" and payload["protocol"] == "TCP" and payload["direction"] == "in"
|
||||
assert payload["gateway"] == "" # the "None" option has the key ""
|
||||
assert payload["tcpflags1"] == "" # nothing selected
|
||||
assert payload["statetimeout"] == "" and payload["log"] == "1" and payload["source_port"] == ""
|
||||
assert payload["enabled"] == "1" and payload["destination_port"] == "8080-8443"
|
||||
assert not set(payload) & {"uuid", "sequence", "sort_order", "prio_group", "audit"}
|
||||
assert all(isinstance(v, str) for v in payload.values())
|
||||
|
||||
|
||||
def test_rule_write_payload_empty_category_list_becomes_an_empty_string():
|
||||
"""Live: a rule without categories answers `"categories": []` (PHP's empty array), not an option dict."""
|
||||
rule = live_rule() | {"categories": []}
|
||||
assert c.rule_write_payload(rule)["categories"] == ""
|
||||
|
||||
|
||||
def test_rule_write_payload_does_not_mutate_its_input():
|
||||
rule = live_rule()
|
||||
before = json.dumps(rule, sort_keys=True)
|
||||
c.rule_write_payload(rule)
|
||||
assert json.dumps(rule, sort_keys=True) == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_copy_is_added_with_exactly_the_payload_and_stays_disabled(host, respx_mock):
|
||||
"""The restore path: getRule -> payload -> addRule(enabled=0). The body must carry the payload unchanged."""
|
||||
respx_mock.get(f"{BASE}/getRule/{RULE}").mock(return_value=httpx.Response(200, json={"rule": live_rule()}))
|
||||
add = respx_mock.post(f"{BASE}/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": OTHER}))
|
||||
payload = c.rule_write_payload(await c.get_rule(host, RULE)) | {"enabled": "0"}
|
||||
assert (await c.add_rule(host, payload))["uuid"] == OTHER
|
||||
sent = _body(add)["rule"]
|
||||
assert sent == payload and sent["enabled"] == "0" and "sequence" not in sent and "uuid" not in sent
|
||||
|
||||
|
||||
# ── add / set / del / move / apply: request shape and classification ─────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_rule_validation_failure_is_a_plain_refusal(host, respx_mock):
|
||||
respx_mock.post(f"{BASE}/addRule").mock(return_value=httpx.Response(
|
||||
200, json={"result": "failed", "validations": {"rule.destination_port": "Please specify a valid portnumber"}}))
|
||||
with pytest.raises(c.OpnsenseAPIError) as info:
|
||||
await c.add_rule(host, {"destination_port": "1,2"})
|
||||
assert type(info.value) is c.OpnsenseAPIError and "destination_port" in str(info.value)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("answer", ["deleted", "not found"])
|
||||
async def test_del_rule_accepts_deleted_and_not_found(host, respx_mock, answer):
|
||||
route = respx_mock.post(f"{BASE}/delRule/{RULE}").mock(return_value=httpx.Response(200, json={"result": answer}))
|
||||
assert (await c.del_rule(host, RULE))["result"] == answer and route.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_del_rule_failed_answer_and_5xx(host, respx_mock):
|
||||
route = respx_mock.post(f"{BASE}/delRule/{RULE}")
|
||||
route.mock(return_value=httpx.Response(200, json={"result": "failed"}))
|
||||
with pytest.raises(c.OpnsenseAPIError):
|
||||
await c.del_rule(host, RULE)
|
||||
route.mock(return_value=httpx.Response(500, text="x"))
|
||||
with pytest.raises(c.OpnsenseAmbiguousError):
|
||||
await c.del_rule(host, RULE)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_rule_before_url_and_answer(host, respx_mock):
|
||||
route = respx_mock.post(f"{BASE}/moveRuleBefore/{RULE}/{OTHER}").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert await c.move_rule_before(host, RULE, OTHER) == {"status": "ok"} and route.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_rule_before_refusal_is_http_500_and_therefore_ambiguous(host, respx_mock):
|
||||
"""Live: unknown UUID, the same UUID twice and rules of different prio_groups all answer HTTP 500
|
||||
{"errorMessage": "Unexpected error, check log for details"}. Callers must read the position back."""
|
||||
respx_mock.post(f"{BASE}/moveRuleBefore/{RULE}/{OTHER}").mock(return_value=httpx.Response(
|
||||
500, json={"errorMessage": "Unexpected error, check log for details", "errorTitle": "Filter", "errorLevel": "warning"}))
|
||||
with pytest.raises(c.OpnsenseAmbiguousError):
|
||||
await c.move_rule_before(host, RULE, OTHER)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_url_and_answer(host, respx_mock):
|
||||
route = respx_mock.post(f"{BASE}/apply").mock(return_value=httpx.Response(200, json={"status": "OK\n\n"}))
|
||||
assert (await c.apply(host))["status"].strip() == "OK" and route.call_count == 1
|
||||
@@ -1,4 +1,6 @@
|
||||
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.test_maintenance_windows import _WEEKLY
|
||||
@@ -145,3 +147,27 @@ async def test_role_matrix_for_plans(client):
|
||||
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
|
||||
finally:
|
||||
await ro.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gate_probes_are_validated_stored_and_frozen_into_the_fingerprint(client):
|
||||
"""1.32 (B5): probes per plan; a plan without probes keeps the fingerprint of earlier versions."""
|
||||
from app.database import SessionLocal
|
||||
from app.models.rollout import RolloutPlan
|
||||
from app.services import rollout_service
|
||||
|
||||
_hosts, groups = await _fleet(client)
|
||||
plain = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
|
||||
assert plain["gate_probes"] == []
|
||||
async with SessionLocal() as db:
|
||||
assert "gate_probes" not in rollout_service.definition(await db.get(RolloutPlan, uuid.UUID(plain["id"])))
|
||||
probes = [{"type": "ping", "target": "1.1.1.1"}, {"type": "tcp", "target": "intranet.example", "port": 443},
|
||||
{"type": "ping", "target": "1.1.1.1"}]
|
||||
resp = await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=probes))
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["gate_probes"] == [{"type": "ping", "target": "1.1.1.1", "port": None},
|
||||
{"type": "tcp", "target": "intranet.example", "port": 443}]
|
||||
for bad in ([{"type": "tcp", "target": "x"}], [{"type": "ping", "target": "x", "port": 1}],
|
||||
[{"type": "ping", "target": "-rf"}], [{"type": "ping", "target": "a b"}], [{"type": "udp", "target": "x"}],
|
||||
[{"type": "ping", "target": f"h{i}"} for i in range(11)]):
|
||||
assert (await client.put(f"/api/rollouts/plans/{plain['id']}", json=_plan_body(groups, gate_probes=bad))).status_code == 422, bad
|
||||
@@ -31,6 +31,7 @@ class Fw:
|
||||
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
|
||||
r, b = fleet.mock, self.base
|
||||
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
|
||||
r.get(f"{b}/api/core/snapshots/isSupported").mock(side_effect=lambda q: self._ok({"supported": False}))
|
||||
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
|
||||
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
|
||||
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
|
||||
@@ -711,3 +712,28 @@ async def test_rows_without_an_account_id_still_work_by_name(client, respx_mock,
|
||||
assert await engine.may_roll_out(db, "admin") is True
|
||||
assert await engine.may_roll_out(db, "nobody-here") is False
|
||||
assert await engine.may_roll_out(db, "admin", uuid.uuid4()) is False # a given id that does not exist is final
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_probe_that_breaks_after_the_update_stops_the_rollout(client, respx_mock, clock, monkeypatch):
|
||||
"""1.32 (B5): the plan's probes are frozen into the run, taken in the baseline and compared in the gate."""
|
||||
from app.services import diagnostics_service
|
||||
|
||||
reachable = {"ok": True}
|
||||
calls = []
|
||||
|
||||
async def portprobe(host, target, port, ipproto, source):
|
||||
calls.append(host.name)
|
||||
return {"open": reachable["ok"] or host.name != "c1"}
|
||||
|
||||
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, gate_probes=[{"type": "tcp", "target": "10.0.0.5", "port": 443}])
|
||||
run = await _start(client, plan)
|
||||
assert (await _run(run["id"])).plan_snapshot["gate_probes"] == [{"type": "tcp", "target": "10.0.0.5", "port": 443}]
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
assert "c1" in calls # baseline
|
||||
reachable["ok"] = False # the update broke the path to the server
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "gate_failed"
|
||||
assert paused.waves[0].gate_result["hosts"][str(_host(paused, "c1").host_id)]["probes"]["code"] == "probe_regressed"
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
@@ -193,3 +193,47 @@ async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeyp
|
||||
def datetime_now():
|
||||
from datetime import datetime, timezone
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_probes_from_the_firewall_compare_against_the_baseline(client, respx_mock, monkeypatch):
|
||||
"""1.32 (B5): reachability probes. Worked before and fails now = fail; failed already before = warn."""
|
||||
from app.services import audit_service, diagnostics_service
|
||||
|
||||
reachable = {"ping 1.1.1.1": True, "tcp 10.0.0.5:443": True, "ping 192.0.2.1": False}
|
||||
actors = []
|
||||
|
||||
async def ping(host, target, ipproto, count, packetsize, source):
|
||||
actors.append((audit_service.context.get() or {}).get("actor"))
|
||||
return {"received": count if reachable[f"ping {target}"] else 0}
|
||||
|
||||
async def portprobe(host, target, port, ipproto, source):
|
||||
if target == "broken":
|
||||
raise diagnostics_service.OpnsenseAPIError("job failed")
|
||||
return {"open": reachable[f"tcp {target}:{port}"]}
|
||||
|
||||
monkeypatch.setattr(diagnostics_service, "ping", ping)
|
||||
monkeypatch.setattr(diagnostics_service, "portprobe", portprobe)
|
||||
Fw(respx_mock, A)
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
probes = [{"type": "ping", "target": "1.1.1.1", "port": None}, {"type": "tcp", "target": "10.0.0.5", "port": 443},
|
||||
{"type": "ping", "target": "192.0.2.1", "port": None}]
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, uuid.UUID(a["id"]))
|
||||
base = {a["id"]: await rh.baseline(db, host, probes)}
|
||||
assert [p["ok"] for p in base[a["id"]]["probes"]] == [True, True, False] and set(actors) == {"system:rollout"}
|
||||
gate, _ = await _eval([a["id"]], base, probes=probes)
|
||||
assert gate["hosts"][a["id"]]["probes"] == {"result": "warn", "code": "probe_was_failing", "params": {"probes": "ping 192.0.2.1"}}
|
||||
reachable["tcp 10.0.0.5:443"] = False
|
||||
gate, _ = await _eval([a["id"]], base, probes=probes)
|
||||
assert gate["verdict"] == "fail"
|
||||
assert gate["hosts"][a["id"]]["probes"] == {"result": "fail", "code": "probe_regressed", "params": {"probes": "tcp 10.0.0.5:443"}}
|
||||
broken = probes + [{"type": "tcp", "target": "broken", "port": 22}] # cannot run = failed; new since the baseline = worked before
|
||||
gate, _ = await _eval([a["id"]], base, probes=broken)
|
||||
assert "tcp broken:22" in gate["hosts"][a["id"]]["probes"]["params"]["probes"]
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert "probes" not in gate["hosts"][a["id"]] # a plan without probes runs none
|
||||
fw_down = Fw(respx_mock, A)
|
||||
fw_down.up = False
|
||||
gate, _ = await _eval([a["id"]], base, probes=probes)
|
||||
assert gate["hosts"][a["id"]]["probes"]["code"] == "unreadable"
|
||||
@@ -0,0 +1,994 @@
|
||||
"""Rule cleanup (1.31, B4 part 2): preview, deploy, confirm, revert/sweeper, delete and restore against a stateful
|
||||
fake firewall (tests/rule_cleanup_fake.py) and the test database. Numbers refer to DESIGN-1.31 §6. Roles and API
|
||||
tokens are checked in the router (package 6), not here."""
|
||||
import asyncio
|
||||
import uuid as uuid_lib
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.governance import GovernanceSettings
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
|
||||
from app.models.rule_hit import RuleHit
|
||||
from app.models.rule_history import LiveRuleState, RuleChangeEvent
|
||||
from app.services import rule_cleanup_service as svc
|
||||
from app.services import rule_history_service as rhs
|
||||
from app.services.rule_analysis import context as ctx_mod
|
||||
from tests.rule_cleanup_fake import (
|
||||
AUTO, MGMT_SOURCE, R_ALIAS, R_BASE, R_BLOCK, R_CARP, R_DUP, R_LEGACY, R_MGMT, R_TAIL, R_UNUSED, FakeFirewall,
|
||||
default_rules, rule,
|
||||
)
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
FW1, FW2 = "https://203.0.113.57", "https://203.0.113.58"
|
||||
UNUSED_SEEDED = (R_UNUSED, R_BLOCK, R_MGMT, R_CARP, R_ALIAS, R_TAIL)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_cache():
|
||||
ctx_mod.clear_cache()
|
||||
yield
|
||||
ctx_mod.clear_cache()
|
||||
|
||||
|
||||
def _now():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
async def _settings(sources=(MGMT_SOURCE,), max_rules=10, min_days=7, ports=()):
|
||||
async with SessionLocal() as db:
|
||||
row = await db.get(GovernanceSettings, 1)
|
||||
if row is None:
|
||||
row = GovernanceSettings(id=1, operators_may_deploy=True)
|
||||
db.add(row)
|
||||
row.cleanup_management_sources = list(sources)
|
||||
row.cleanup_management_ports = list(ports)
|
||||
row.cleanup_max_rules, row.cleanup_delete_min_days = max_rules, min_days
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _seed_hits(host_id, uuids, *, days=100, hit_days=None, sampled_minutes=10):
|
||||
async with SessionLocal() as db:
|
||||
now = _now()
|
||||
for u in uuids:
|
||||
db.add(RuleHit(host_id=host_id, rule_uuid=u.lower(), description=u, enabled=True,
|
||||
first_seen_at=now - timedelta(days=days), last_packets=0, last_bytes=0,
|
||||
last_sampled_at=now - timedelta(minutes=sampled_minutes),
|
||||
last_hit_at=None if hit_days is None else now - timedelta(days=hit_days)))
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def call(host_id, fn, *args, **kw):
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
return await fn(db, host, *args, **kw)
|
||||
|
||||
|
||||
async def refused(host_id, fn, *args, **kw) -> dict:
|
||||
with pytest.raises(HTTPException) as info:
|
||||
await call(host_id, fn, *args, **kw)
|
||||
return {"status": info.value.status_code, **info.value.detail}
|
||||
|
||||
|
||||
async def preview(host_id, uuids, actor="admin", **kw):
|
||||
kw.setdefault("kind", "disable")
|
||||
return await call(host_id, svc.preview, actor=actor, uuids=uuids, **kw)
|
||||
|
||||
|
||||
async def deploy(host_id, preview_id, actor="admin", name=None):
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
return await svc.deploy(db, host, uuid_lib.UUID(preview_id), actor=actor, confirm_name=name or host.name)
|
||||
|
||||
|
||||
async def _items(op_id):
|
||||
async with SessionLocal() as db:
|
||||
return {i.rule_uuid: i for i in (await db.scalars(select(RuleCleanupItem).where(
|
||||
RuleCleanupItem.operation_id == uuid_lib.UUID(str(op_id))))).all()}
|
||||
|
||||
|
||||
async def _op(op_id):
|
||||
async with SessionLocal() as db:
|
||||
return await db.get(RuleCleanupOperation, uuid_lib.UUID(str(op_id)))
|
||||
|
||||
|
||||
async def _backdate(op_id, **fields):
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleCleanupOperation).where(RuleCleanupOperation.id == uuid_lib.UUID(str(op_id))).values(**fields))
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _sweep():
|
||||
async with SessionLocal() as db:
|
||||
return await svc.sweep_expired_cleanups(db)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def env(client, respx_mock):
|
||||
log: list = []
|
||||
fw = FakeFirewall(FW1, log=log).mount(respx_mock)
|
||||
host_id = uuid_lib.UUID((await _create_host(client, "fw-a", FW1))["id"])
|
||||
await _settings()
|
||||
await _seed_hits(host_id, UNUSED_SEEDED)
|
||||
fw.calls.clear()
|
||||
log.clear()
|
||||
return SimpleNamespace(fw=fw, host_id=host_id, log=log)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def cluster(client, respx_mock):
|
||||
log: list = []
|
||||
backup = FakeFirewall(FW2, role="BACKUP", pfsync="lan", peer="10.20.30.2", lan_addr="10.20.30.3",
|
||||
wan_addr="203.0.113.58", log=log).mount(respx_mock)
|
||||
master = FakeFirewall(FW1, role="MASTER", pfsync="lan", peer="10.20.30.3", log=log).mount(respx_mock)
|
||||
a = uuid_lib.UUID((await _create_host(client, "fw-a", FW1))["id"])
|
||||
b = uuid_lib.UUID((await _create_host(client, "fw-b", FW2))["id"])
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(OpnsenseHost).where(OpnsenseHost.id.in_([a, b])).values(cluster_name="ha-test"))
|
||||
await db.commit()
|
||||
await _settings()
|
||||
await _seed_hits(a, UNUSED_SEEDED)
|
||||
await _seed_hits(b, UNUSED_SEEDED)
|
||||
for f in (master, backup):
|
||||
f.calls.clear()
|
||||
log.clear()
|
||||
return SimpleNamespace(master=master, backup=backup, a=a, b=b, log=log)
|
||||
|
||||
|
||||
def _writes(log):
|
||||
return [(base, path) for base, method, path in log if method == "POST"]
|
||||
|
||||
|
||||
# ── Preview ───────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_recomputes_from_the_live_firewall_and_stores_a_bound_token(env):
|
||||
result = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
plan = result["plan"]
|
||||
items = {i["uuid"]: i for i in plan["nodes"][0]["items"]}
|
||||
assert set(items) == {R_UNUSED, R_DUP}
|
||||
assert items[R_UNUSED]["level"] == "sure" and items[R_UNUSED]["protection"]["result"] == "false"
|
||||
assert items[R_DUP]["cover_uuid"] == R_BASE and items[R_DUP]["cover_fp"]
|
||||
assert env.fw.writes() == [] # nothing is written by a preview
|
||||
assert ("GET", "/firewall/filter/searchRule") in env.fw.calls # read live, not from a cache or the request
|
||||
op = await _op(result["preview_id"])
|
||||
assert op.status == "preview" and op.actor == "admin" and len(op.preview_fingerprint) == 64
|
||||
assert timedelta(minutes=4) < op.expires_at - _now() <= timedelta(minutes=5)
|
||||
async with SessionLocal() as db:
|
||||
audit = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rule_cleanup"))).all()
|
||||
assert [(a.action, a.status) for a in audit] == [("preview", "requested")]
|
||||
assert "remote_uuid" not in audit[0].details # a preview never appears as a change in the rule history
|
||||
# a second preview of the same actor replaces the first
|
||||
again = await preview(env.host_id, [R_UNUSED])
|
||||
assert (await _op(result["preview_id"])).status == "cancelled" and (await _op(again["preview_id"])).status == "preview"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_without_management_sources_everything_is_locked(env):
|
||||
await _settings(sources=())
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||
await _settings(sources=("not-an-address",))
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("uuid,reason", [
|
||||
(R_MGMT, "management"), # allows the management source to the firewall's 443
|
||||
(R_ALIAS, "management"), # source is a URL table: match unknown = protected (case 18)
|
||||
(R_CARP, "infrastructure"), # CARP protocol
|
||||
(R_BASE, "not_candidate"), # nothing says it is removable
|
||||
])
|
||||
async def test_rules_that_must_not_be_switched_off_are_refused(env, uuid, reason):
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[uuid], acknowledge=[uuid])
|
||||
assert got["status"] == 409 and got["code"] == "BEREINIGUNG_REGEL_NICHT_AENDERBAR"
|
||||
assert got["params"]["reason"] == reason and got["params"]["uuid"] == uuid
|
||||
assert env.fw.writes() == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_managed_legacy_and_unknown_confidence_are_refused(env):
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
env.fw.rules.append(rule(R_LEGACY, 900, src="10.20.30.0/24", dport="6000", legacy=True))
|
||||
await _seed_hits(env.host_id, [R_LEGACY])
|
||||
async with SessionLocal() as db:
|
||||
managed = FirewallRule(description="Managed", interface="lan", action="pass", direction="in", protocol="tcp",
|
||||
source_net="any", destination_net="any", destination_port="8081")
|
||||
db.add(managed)
|
||||
await db.flush()
|
||||
db.add(RuleDeployment(rule_id=managed.id, host_id=env.host_id, status="success", opnsense_rule_uuid=R_UNUSED))
|
||||
# stale counters (sampled two days ago): "unused" is unknown, never selectable
|
||||
await db.execute(update(RuleHit).where(RuleHit.rule_uuid == R_TAIL).values(last_sampled_at=_now() - timedelta(days=2)))
|
||||
await db.commit()
|
||||
for uuid, reason in ((R_UNUSED, "managed"), (R_LEGACY, "legacy"), (R_TAIL, "confidence")):
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[uuid], acknowledge=[uuid])
|
||||
assert got["params"]["reason"] == reason, (uuid, got)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_probable_needs_an_explicit_acknowledgement_per_rule(env):
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_BLOCK, R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT" and got["params"]["uuids"] == R_BLOCK
|
||||
ok = await preview(env.host_id, [R_BLOCK, R_UNUSED], acknowledge=[R_BLOCK], ack_text_version="1")
|
||||
item = next(i for i in ok["plan"]["nodes"][0]["items"] if i["uuid"] == R_BLOCK)
|
||||
assert item["level"] == "probable" and item["needs_ack"] and "UNUSED_BLOCK_RULE" in item["notes"]
|
||||
op = await _op(ok["preview_id"])
|
||||
assert op.options["acknowledged"] == [{"uuid": R_BLOCK, "text_version": "1"}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_short_window_and_external_change_cap_at_probable(env):
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED], unused_days=14)
|
||||
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
async with SessionLocal() as db:
|
||||
db.add(RuleChangeEvent(host_id=env.host_id, rule_uuid=R_UNUSED, seq=1, observed_at=_now() - timedelta(days=2),
|
||||
kind="changed", before={}, after={}, origin="external"))
|
||||
await db.commit()
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
ok = await preview(env.host_id, [R_UNUSED], acknowledge=[R_UNUSED])
|
||||
assert "EXTERNAL_CHANGE" in ok["plan"]["nodes"][0]["items"][0]["notes"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_limit_and_selection_validation(env):
|
||||
await _settings(max_rules=2)
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED, R_DUP, R_TAIL])
|
||||
assert got["status"] == 422 and got["code"] == "BEREINIGUNG_ZU_VIELE_REGELN" and got["params"]["max"] == 2
|
||||
for uuids in ([], ["../apply"]):
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=uuids)
|
||||
assert got["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG"
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="wipe", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_maintenance_open_deployments_and_open_cleanups_block(env):
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
async with SessionLocal() as db:
|
||||
op = HostOperation(host_id=env.host_id, host_name="fw-a", actor="admin", action="reboot", status="submitted",
|
||||
expires_at=_now(), fingerprint="x", snapshot={})
|
||||
db.add(op)
|
||||
await db.commit()
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(HostOperation).values(status="done"))
|
||||
managed = FirewallRule(description="x", interface="lan", action="pass", direction="in", protocol="tcp",
|
||||
source_net="any", destination_net="any")
|
||||
db.add(managed)
|
||||
await db.flush()
|
||||
dep = RuleDeployment(rule_id=managed.id, host_id=env.host_id, status="pending", opnsense_rule_uuid=R_TAIL)
|
||||
db.add(dep)
|
||||
await db.commit()
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_OFFENE_OPERATION" and got["params"]["reason"] == "deployment"
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleDeployment).values(status="success"))
|
||||
await db.commit()
|
||||
first = await preview(env.host_id, [R_UNUSED])
|
||||
await deploy(env.host_id, first["preview_id"])
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_DUP])
|
||||
assert got["code"] == "BEREINIGUNG_OFFENE_OPERATION" and got["params"]["reason"] == "cleanup"
|
||||
async with SessionLocal() as db:
|
||||
assert await svc.has_open_cleanup(db, [env.host_id])
|
||||
|
||||
|
||||
# ── Deploy ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_deploy_switches_off_samples_before_apply_and_waits_for_confirmation(env):
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
env.fw.calls.clear()
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
assert result["status"] == "awaiting_confirm" and result["can_confirm"]
|
||||
writes = env.fw.writes()
|
||||
assert sorted(writes[:2]) == sorted([("POST", f"/firewall/filter/toggleRule/{R_DUP}/0"),
|
||||
("POST", f"/firewall/filter/toggleRule/{R_UNUSED}/0")])
|
||||
assert writes[2:] == [("POST", "/firewall/filter/apply")] # one apply per node, after the toggles
|
||||
apply_at = env.fw.calls.index(("POST", "/firewall/filter/apply"))
|
||||
last_toggle = max(i for i, c in enumerate(env.fw.calls) if "toggleRule" in c[1])
|
||||
assert ("GET", "/firewall/filter_util/ruleStats") in env.fw.calls[last_toggle:apply_at] # sampled before the apply
|
||||
assert env.fw.get(R_UNUSED)["enabled"] == "0" and env.fw.get(R_DUP)["enabled"] == "0"
|
||||
op = await _op(p["preview_id"])
|
||||
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
|
||||
assert window - timedelta(seconds=10) < op.revert_deadline - _now() <= window
|
||||
items = await _items(p["preview_id"])
|
||||
assert {i.state for i in items.values()} == {"disabled"}
|
||||
assert items[R_UNUSED].fp_before and items[R_UNUSED].row_before["enabled"] == "1"
|
||||
async with SessionLocal() as db:
|
||||
audits = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rule_cleanup", AuditEvent.action == "disable"))).all()
|
||||
assert {(a.details["remote_uuid"], a.status) for a in audits} == {(R_UNUSED, "accepted"), (R_DUP, "accepted")}
|
||||
assert all(a.host_id == str(env.host_id) and a.details["preview_fingerprint"] == op.preview_fingerprint for a in audits)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_history_shows_the_cleanup_as_opnexus(env):
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, env.host_id)
|
||||
await rhs.observe_rows(db, host, {"rows": env.fw.rules}) # baseline
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, env.host_id)
|
||||
await rhs.observe_rows(db, host, {"rows": env.fw.rules})
|
||||
event = (await db.scalars(select(RuleChangeEvent).where(RuleChangeEvent.rule_uuid == R_UNUSED))).one()
|
||||
assert event.kind == "disabled" and event.origin == "opnexus" and event.audit_id is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("change", ["edited", "moved", "cover_edited", "hit"])
|
||||
async def test_deploy_refuses_a_stale_preview_without_writing(env, change):
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
if change == "edited":
|
||||
env.fw.get(R_UNUSED)["destination_port"] = "8082"
|
||||
elif change == "moved":
|
||||
env.fw.get(R_UNUSED)["sort_order"] = "400000.0000301"
|
||||
elif change == "cover_edited":
|
||||
env.fw.get(R_BASE)["description"] = "base changed"
|
||||
else: # a hit since the preview: the fresh sample at deploy sees the counter grow
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleHit).where(RuleHit.rule_uuid == R_UNUSED).values(last_hit_at=_now()))
|
||||
await db.commit()
|
||||
env.fw.calls.clear()
|
||||
got = await refused(env.host_id, svc.deploy, uuid_lib.UUID(p["preview_id"]), actor="admin", confirm_name="fw-a")
|
||||
expected = R_DUP if change == "cover_edited" else R_UNUSED
|
||||
if change == "hit":
|
||||
assert got["code"] == "BEREINIGUNG_REGEL_NICHT_AENDERBAR" and got["params"]["uuid"] == R_UNUSED
|
||||
else:
|
||||
assert got["code"] == "BEREINIGUNG_VORSCHAU_VERALTET" and got["params"]["uuids"] == expected
|
||||
assert env.fw.writes() == []
|
||||
assert (await _op(p["preview_id"])).status == "expired"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_token_is_single_use_bound_to_actor_and_name(env):
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
pid = uuid_lib.UUID(p["preview_id"])
|
||||
got = await refused(env.host_id, svc.deploy, pid, actor="someone-else", confirm_name="fw-a")
|
||||
assert got["status"] == 404
|
||||
got = await refused(env.host_id, svc.deploy, pid, actor="admin", confirm_name="FW-A")
|
||||
assert got["code"] == "HOSTNAMEN_EXAKT_EINGEBEN"
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
got = await refused(env.host_id, svc.deploy, pid, actor="admin", confirm_name="fw-a")
|
||||
assert got["code"] == "VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET"
|
||||
# an expired preview
|
||||
await call(env.host_id, svc.revert_now, pid)
|
||||
q = await preview(env.host_id, [R_DUP])
|
||||
await _backdate(q["preview_id"], expires_at=_now() - timedelta(seconds=1))
|
||||
got = await refused(env.host_id, svc.deploy, uuid_lib.UUID(q["preview_id"]), actor="admin", confirm_name="fw-a")
|
||||
assert got["code"] == "VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refused_toggle_stops_writing_but_applies_what_is_switched(env):
|
||||
uuids = [R_DUP, R_UNUSED, R_BLOCK, R_TAIL]
|
||||
p = await preview(env.host_id, uuids, acknowledge=[R_BLOCK])
|
||||
env.fw.fail[("toggle", R_BLOCK)] = httpx.Response(200, json={"result": "failed"})
|
||||
env.fw.calls.clear()
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
states = {i.rule_uuid: i.state for i in (await _items(p["preview_id"])).values()}
|
||||
assert states == {R_DUP: "disabled", R_UNUSED: "disabled", R_BLOCK: "failed", R_TAIL: "queued"}
|
||||
assert ("POST", f"/firewall/filter/toggleRule/{R_TAIL}/0") not in env.fw.calls # nothing after the error
|
||||
assert env.fw.writes()[-1] == ("POST", "/firewall/filter/apply") # but the switched ones are applied
|
||||
assert result["status"] == "partial" and not result["can_confirm"] and result["can_revert"]
|
||||
assert (await _op(p["preview_id"])).revert_deadline > _now()
|
||||
got = await refused(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
assert got["code"] == "BEREINIGUNG_NICHT_BESTAETIGBAR"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("applied,readable,state", [(True, True, "disabled"), (False, True, "failed"), (False, False, "unknown")])
|
||||
async def test_unclear_toggle_is_settled_by_reading(env, applied, readable, state):
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
items_order = [i["uuid"] for i in p["plan"]["nodes"][0]["items"]]
|
||||
first = items_order[0]
|
||||
env.fw.fail[("toggle", first)] = httpx.ReadTimeout
|
||||
env.fw.fail[("toggle_applies", first)] = applied
|
||||
if not readable:
|
||||
env.fw.fail[("get", first)] = httpx.ConnectError
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
items = await _items(p["preview_id"])
|
||||
assert items[first].state == state
|
||||
# no further write after an unclear answer (never attempted: "skipped" once nothing changed at all)
|
||||
assert items[items_order[1]].state == ("skipped" if state == "failed" else "queued")
|
||||
expect_apply = state in ("disabled", "unknown")
|
||||
assert (("POST", "/firewall/filter/apply") in env.fw.calls) is expect_apply
|
||||
assert result["status"] == {"disabled": "partial", "failed": "failed", "unknown": "partial"}[state]
|
||||
assert not result["can_confirm"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_apply_keeps_the_deadline_and_the_sweeper_reverts(env):
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
env.fw.fail[("apply", "")] = [httpx.Response(500, json={}), None]
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
assert result["status"] == "apply_failed" and result["revert_deadline"]
|
||||
await _sweep()
|
||||
assert (await _op(p["preview_id"])).status == "apply_failed" # deadline not reached yet
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
|
||||
env.fw.calls.clear()
|
||||
await _sweep()
|
||||
assert (await _op(p["preview_id"])).status == "reverted"
|
||||
assert env.fw.writes() == [("POST", f"/firewall/filter/toggleRule/{R_UNUSED}/1"), ("POST", "/firewall/filter/apply")]
|
||||
assert env.fw.get(R_UNUSED)["enabled"] == "1"
|
||||
|
||||
|
||||
# ── Confirm ───────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_confirm_reads_the_firewall_and_refuses_when_unreachable_or_different(env):
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
pid = uuid_lib.UUID(p["preview_id"])
|
||||
env.fw.down = True
|
||||
got = await refused(env.host_id, svc.confirm, pid)
|
||||
assert got["status"] == 502 and got["code"] == "BEREINIGUNG_FIREWALL_NICHT_ERREICHBAR"
|
||||
assert (await _op(pid)).status == "awaiting_confirm" # stays open: the sweeper reverts it
|
||||
env.fw.down = False
|
||||
env.fw.get(R_DUP)["enabled"] = "1" # someone switched it back on in the GUI
|
||||
got = await refused(env.host_id, svc.confirm, pid)
|
||||
assert got["code"] == "BEREINIGUNG_LIVESTAND_ABWEICHEND" and got["params"]["uuids"] == R_DUP
|
||||
env.fw.get(R_DUP)["enabled"] = "0"
|
||||
env.fw.calls.clear()
|
||||
result = await call(env.host_id, svc.confirm, pid)
|
||||
assert result["status"] == "confirmed" and {i["state"] for i in result["items"]} == {"confirmed"}
|
||||
assert env.fw.writes() == [] # confirming only reads
|
||||
op = await _op(pid)
|
||||
assert op.revert_deadline is None and op.confirmed_at is not None
|
||||
assert await _sweep() == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_confirm_after_the_deadline_is_refused_and_the_sweeper_reverts(env):
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
|
||||
got = await refused(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
assert got["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
assert len(await _sweep()) == 1
|
||||
assert env.fw.get(R_UNUSED)["enabled"] == "1"
|
||||
assert {i.state for i in (await _items(p["preview_id"])).values()} == {"reverted"}
|
||||
async with SessionLocal() as db:
|
||||
audit = (await db.scalars(select(AuditEvent).where(AuditEvent.action == "revert"))).one()
|
||||
assert audit.actor == "system:rollback" and audit.details["remote_uuid"] == R_UNUSED
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revert_now_and_failed_revert_is_not_retried_automatically(env):
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
env.fw.fail[("toggle", R_DUP)] = httpx.Response(200, json={"result": "failed"})
|
||||
result = await call(env.host_id, svc.revert_now, uuid_lib.UUID(p["preview_id"]))
|
||||
assert result["status"] == "revert_failed" and result["can_retry"]
|
||||
states = {i["rule_uuid"]: i["state"] for i in result["items"]}
|
||||
assert states == {R_UNUSED: "reverted", R_DUP: "revert_failed"} # the others are still reverted
|
||||
assert env.fw.get(R_UNUSED)["enabled"] == "1" and env.fw.writes()[-1] == ("POST", "/firewall/filter/apply")
|
||||
env.fw.calls.clear()
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(minutes=5))
|
||||
await _sweep()
|
||||
assert env.fw.writes() == [] # never retried by the sweeper
|
||||
del env.fw.fail[("toggle", R_DUP)]
|
||||
result = await call(env.host_id, svc.retry, uuid_lib.UUID(p["preview_id"]))
|
||||
assert result["status"] == "reverted" and env.fw.get(R_DUP)["enabled"] == "1"
|
||||
|
||||
|
||||
# ── Sweeper and two backends ──────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweeper_skips_a_live_run_and_takes_over_an_abandoned_one(env):
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
# simulate a backend that died mid-run: running, item half-way
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleCleanupOperation).values(status="running", heartbeat_at=_now(),
|
||||
revert_deadline=_now() + timedelta(minutes=10)))
|
||||
await db.execute(update(RuleCleanupItem).values(state="disabling"))
|
||||
await db.commit()
|
||||
env.fw.calls.clear()
|
||||
assert await _sweep() == [] # (b) fresh heartbeat: someone is still working on it
|
||||
assert env.fw.writes() == []
|
||||
await _backdate(p["preview_id"], heartbeat_at=_now() - timedelta(hours=1))
|
||||
assert await _sweep() == [p["preview_id"]] # (c) stale heartbeat: taken over and reverted
|
||||
assert env.fw.writes() == [("POST", f"/firewall/filter/toggleRule/{R_UNUSED}/1"), ("POST", "/firewall/filter/apply")]
|
||||
assert (await _op(p["preview_id"])).status == "reverted"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_two_deploys_of_one_preview_write_once(env):
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
outcomes = await asyncio.gather(deploy(env.host_id, p["preview_id"]), deploy(env.host_id, p["preview_id"]),
|
||||
return_exceptions=True)
|
||||
errors = [o for o in outcomes if isinstance(o, HTTPException)]
|
||||
assert len(errors) == 1 and errors[0].detail["code"] == "VORSCHAU_ABGELAUFEN_ODER_BEREITS_VERWENDET"
|
||||
assert sum(1 for c in env.fw.calls if "toggleRule" in c[1]) == 2 # (a) each rule exactly once
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_second_operation_on_the_same_rule_is_refused(env):
|
||||
a = await preview(env.host_id, [R_UNUSED], actor="admin")
|
||||
b = await preview(env.host_id, [R_UNUSED], actor="admin2")
|
||||
await deploy(env.host_id, a["preview_id"])
|
||||
got = await refused(env.host_id, svc.deploy, uuid_lib.UUID(b["preview_id"]), actor="admin2", confirm_name="fw-a")
|
||||
assert got["code"] == "BEREINIGUNG_OFFENE_OPERATION" # (d)
|
||||
# and the partial unique index holds even without the service's checks
|
||||
async with SessionLocal() as db:
|
||||
op = await db.get(RuleCleanupOperation, uuid_lib.UUID(b["preview_id"]))
|
||||
db.add(RuleCleanupItem(operation_id=op.id, host_id=env.host_id, rule_uuid=R_UNUSED, state="queued", description=""))
|
||||
with pytest.raises(Exception):
|
||||
await db.commit()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_confirm_and_sweeper_at_the_deadline_decide_once(env):
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() + timedelta(milliseconds=300))
|
||||
|
||||
async def late_confirm():
|
||||
await asyncio.sleep(0.4)
|
||||
return await call(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
|
||||
async def late_sweep():
|
||||
await asyncio.sleep(0.4)
|
||||
return await _sweep()
|
||||
|
||||
outcomes = await asyncio.gather(late_confirm(), late_sweep(), return_exceptions=True)
|
||||
op = await _op(p["preview_id"])
|
||||
assert op.status == "reverted" # (e) the deadline passed: confirm refused, exactly one revert
|
||||
assert isinstance(outcomes[0], HTTPException) and outcomes[0].detail["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
assert sum(1 for c in env.fw.calls if c[1].endswith(f"{R_UNUSED}/1")) == 1
|
||||
|
||||
|
||||
# ── Cluster ───────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_writes_backup_first_and_reverts_master_first(cluster):
|
||||
p = await preview(cluster.a, [R_UNUSED])
|
||||
assert [n["role"] for n in p["plan"]["nodes"]] == ["BACKUP", "MASTER"]
|
||||
await deploy(cluster.a, p["preview_id"])
|
||||
writes = _writes(cluster.log)
|
||||
assert writes == [(FW2, f"/firewall/filter/toggleRule/{R_UNUSED}/0"), (FW2, "/firewall/filter/apply"),
|
||||
(FW1, f"/firewall/filter/toggleRule/{R_UNUSED}/0"), (FW1, "/firewall/filter/apply")]
|
||||
items = await _items(p["preview_id"])
|
||||
assert len(items) == 1 # (same rule_uuid on both nodes: keyed by uuid here)
|
||||
async with SessionLocal() as db:
|
||||
roles = {i.host_id: i.carp_role for i in (await db.scalars(select(RuleCleanupItem))).all()}
|
||||
assert roles == {cluster.a: "MASTER", cluster.b: "BACKUP"}
|
||||
cluster.log.clear()
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
|
||||
await _sweep()
|
||||
assert _writes(cluster.log) == [(FW1, f"/firewall/filter/toggleRule/{R_UNUSED}/1"), (FW1, "/firewall/filter/apply"),
|
||||
(FW2, f"/firewall/filter/toggleRule/{R_UNUSED}/1"), (FW2, "/firewall/filter/apply")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_master_failure_leaves_backup_switched_and_unconfirmable(cluster):
|
||||
p = await preview(cluster.a, [R_UNUSED])
|
||||
cluster.master.fail[("toggle", R_UNUSED)] = httpx.Response(200, json={"result": "failed"})
|
||||
result = await deploy(cluster.a, p["preview_id"])
|
||||
assert result["status"] == "partial" and not result["can_confirm"]
|
||||
assert cluster.backup.get(R_UNUSED)["enabled"] == "0" and cluster.master.get(R_UNUSED)["enabled"] == "1"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_refuses_missing_uuid_offline_peer_and_two_masters(cluster):
|
||||
cluster.backup.rules = [r for r in cluster.backup.rules if r["uuid"] != R_UNUSED]
|
||||
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG" and got["params"]["reason"] == "uuid_missing"
|
||||
cluster.backup.rules = default_rules()
|
||||
cluster.backup.role = "MASTER"
|
||||
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_CLUSTER_UNVOLLSTAENDIG" and got["params"]["reason"] == "roles"
|
||||
cluster.backup.role = "BACKUP"
|
||||
cluster.backup.down = True
|
||||
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["status"] == 502 and got["params"]["host"] == "fw-b"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_usage_is_judged_over_all_nodes(cluster):
|
||||
# case 16: the backup saw no hit, the master did: not unused
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleHit).where(RuleHit.host_id == cluster.a, RuleHit.rule_uuid == R_UNUSED)
|
||||
.values(last_hit_at=_now() - timedelta(days=1)))
|
||||
await db.commit()
|
||||
got = await refused(cluster.b, svc.preview, actor="admin", kind="disable", uuids=[R_UNUSED])
|
||||
assert got["params"]["reason"] == "not_candidate"
|
||||
# no usage data on one node: at best probable
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(RuleHit.__table__.delete().where(RuleHit.host_id == cluster.a, RuleHit.rule_uuid == R_TAIL))
|
||||
await db.commit()
|
||||
got = await refused(cluster.b, svc.preview, actor="admin", kind="disable", uuids=[R_TAIL])
|
||||
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_pfsync_interface_rules_are_infrastructure(cluster):
|
||||
# "any" from the LAN net on the pfsync interface (LAN) would also carry pfsync
|
||||
for fw in (cluster.master, cluster.backup):
|
||||
fw.rules.append(rule("aaaaaaaa-0000-4000-8000-000000000900", 900, proto="any", src="10.20.30.0/24", description="lan any"))
|
||||
await _seed_hits(cluster.a, ["aaaaaaaa-0000-4000-8000-000000000900"])
|
||||
await _seed_hits(cluster.b, ["aaaaaaaa-0000-4000-8000-000000000900"])
|
||||
got = await refused(cluster.a, svc.preview, actor="admin", kind="disable", uuids=["aaaaaaaa-0000-4000-8000-000000000900"])
|
||||
assert got["params"]["reason"] == "infrastructure"
|
||||
|
||||
|
||||
# ── Delete and restore ────────────────────────────────────────────────────
|
||||
|
||||
async def _confirmed_disable(host_id, uuids, *, days_ago=8):
|
||||
p = await preview(host_id, uuids)
|
||||
await deploy(host_id, p["preview_id"])
|
||||
await call(host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
await _backdate(p["preview_id"], confirmed_at=_now() - timedelta(days=days_ago))
|
||||
return p["preview_id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_needs_the_waiting_time_an_unchanged_rule_and_no_external_change(env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED], days_ago=2)
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_WARTEZEIT_NICHT_ERREICHT" and got["params"]["days"] == 5
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleCleanupOperation).values(confirmed_at=_now() - timedelta(days=8)))
|
||||
await db.commit()
|
||||
ok = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
assert ok["plan"]["nodes"][0]["items"][0]["basis"]["type"] == "confirmed"
|
||||
env.fw.get(R_UNUSED)["destination_port"] = "8099"
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_VORSCHAU_VERALTET"
|
||||
env.fw.get(R_UNUSED)["destination_port"] = "8081"
|
||||
async with SessionLocal() as db:
|
||||
db.add(RuleChangeEvent(host_id=env.host_id, rule_uuid=R_UNUSED, seq=5, observed_at=_now() - timedelta(days=1),
|
||||
kind="changed", before={}, after={}, origin="external"))
|
||||
await db.commit()
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED])
|
||||
assert got["code"] == "BEREINIGUNG_VORSCHAU_VERALTET"
|
||||
env.fw.get(R_DUP)["enabled"] = "1"
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="delete", uuids=[R_DUP])
|
||||
assert got["params"]["reason"] == "not_candidate" # an enabled rule is never deleted
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_commits_the_snapshot_before_delrule(env, monkeypatch):
|
||||
from app.services import opnsense_client as client_mod
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
seen: list = []
|
||||
real_del = client_mod.del_rule
|
||||
|
||||
async def spy(host, uuid): # what another session sees in the database at the moment delRule goes out
|
||||
async with SessionLocal() as db:
|
||||
item = (await db.scalars(select(RuleCleanupItem).where(
|
||||
RuleCleanupItem.operation_id == uuid_lib.UUID(p["preview_id"])))).one()
|
||||
seen.append((item.state, bool(item.snapshot), item.anchor_next_uuid))
|
||||
return await real_del(host, uuid)
|
||||
monkeypatch.setattr(client_mod, "del_rule", spy)
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
assert seen == [("deleting", True, R_BLOCK)] # durable before the remote delete
|
||||
assert result["status"] == "done"
|
||||
item = next(iter((await _items(p["preview_id"])).values()))
|
||||
assert item.state == "deleted" and item.snapshot["rule"]["destination_port"] == "8081"
|
||||
assert item.snapshot["rule"]["action"] == "pass" and "uuid" not in item.snapshot["rule"]
|
||||
assert item.prio_group == "400000" and item.anchor_prev_uuid == R_DUP
|
||||
assert env.fw.get(R_UNUSED) is None and env.fw.writes()[-1] == ("POST", "/firewall/filter/apply")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("applied,state", [(True, "deleted"), (False, "failed")])
|
||||
async def test_unclear_delete_is_settled_with_getrule(env, applied, state):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
env.fw.fail[("delete", R_UNUSED)] = httpx.ReadTimeout
|
||||
env.fw.fail[("delete_applies", R_UNUSED)] = applied
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
assert next(iter((await _items(p["preview_id"])).values())).state == state
|
||||
assert result["status"] == ("done" if state == "deleted" else "failed")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restore_adds_disabled_at_the_anchor_with_a_new_uuid(env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
deleted = next(iter((await _items(p["preview_id"])).values()))
|
||||
env.fw.calls.clear()
|
||||
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert result["status"] == "done" and result["warnings"] == []
|
||||
item = result["items"][0]
|
||||
new = item["restored_uuid"]
|
||||
assert new and new != R_UNUSED and item["state"] == "restored"
|
||||
restored = env.fw.get(new)
|
||||
assert restored["enabled"] == "0" and restored["destination_port"] == "8081"
|
||||
assert ("POST", f"/firewall/filter/moveRuleBefore/{new}/{R_BLOCK}") in env.fw.calls
|
||||
order = [r["uuid"] for r in sorted(env.fw.rules, key=lambda r: r["sort_order"]) if not r.get("is_automatic")]
|
||||
assert order.index(new) == order.index(R_BLOCK) - 1 and order.index(new) == order.index(R_DUP) + 1
|
||||
async with SessionLocal() as db:
|
||||
audit = (await db.scalars(select(AuditEvent).where(AuditEvent.action == "restore"))).one()
|
||||
source = await db.get(RuleCleanupItem, deleted.id)
|
||||
assert audit.details["remote_uuid"] == new and audit.details["restored_from"] == R_UNUSED
|
||||
assert source.restored_uuid == new
|
||||
got = await refused(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert got["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH" # restored once only
|
||||
# re-enabling the restored rule is an operation with a window
|
||||
e = await preview(env.host_id, [new], kind="enable")
|
||||
env.fw.calls.clear()
|
||||
r = await deploy(env.host_id, e["preview_id"])
|
||||
assert r["status"] == "awaiting_confirm" and ("POST", f"/firewall/filter/toggleRule/{new}/1") in env.fw.calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restore_without_anchor_warns_and_keeps_the_rule(env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
deleted = next(iter((await _items(p["preview_id"])).values()))
|
||||
env.fw.rules = [r for r in env.fw.rules if r["uuid"] != R_BLOCK] # the anchor is gone meanwhile
|
||||
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert result["status"] == "done" and result["warnings"] == ["BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"]
|
||||
assert result["items"][0]["message"] == "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
|
||||
assert env.fw.get(result["items"][0]["restored_uuid"])["enabled"] == "0"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restore_refused_by_the_firewall_is_failed_with_its_message(env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
deleted = next(iter((await _items(p["preview_id"])).values()))
|
||||
env.fw.fail[("add", "")] = httpx.Response(200, json={"result": "failed", "validations": {"rule.source_net": "alias missing"}})
|
||||
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert result["status"] == "failed" and "alias missing" in result["items"][0]["message"]
|
||||
async with SessionLocal() as db:
|
||||
assert (await db.get(RuleCleanupItem, deleted.id)).restored_uuid is None # can be tried again
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_delete_restore_and_reenable_follow_the_restore_link(cluster):
|
||||
p = await preview(cluster.a, [R_UNUSED])
|
||||
await deploy(cluster.a, p["preview_id"])
|
||||
await call(cluster.a, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
await _backdate(p["preview_id"], confirmed_at=_now() - timedelta(days=8))
|
||||
d = await preview(cluster.a, [R_UNUSED], kind="delete")
|
||||
cluster.log.clear()
|
||||
assert (await deploy(cluster.a, d["preview_id"]))["status"] == "done"
|
||||
assert [w for w in _writes(cluster.log) if "delRule" in w[1]] == [(FW2, f"/firewall/filter/delRule/{R_UNUSED}"),
|
||||
(FW1, f"/firewall/filter/delRule/{R_UNUSED}")]
|
||||
async with SessionLocal() as db:
|
||||
deleted = (await db.scalars(select(RuleCleanupItem).where(
|
||||
RuleCleanupItem.operation_id == uuid_lib.UUID(d["preview_id"]), RuleCleanupItem.host_id == cluster.a))).one()
|
||||
restored = await call(cluster.a, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert restored["status"] == "done" and len(restored["items"]) == 2 # both nodes, backup first
|
||||
new = {i["host_id"]: i["restored_uuid"] for i in restored["items"]}
|
||||
assert new[str(cluster.a)] != new[str(cluster.b)] # addRule: a new UUID on every node
|
||||
assert cluster.backup.get(new[str(cluster.b)])["enabled"] == "0" and cluster.master.get(new[str(cluster.a)])["enabled"] == "0"
|
||||
# re-enable through the master's UUID: the backup's copy is found through the restore operation
|
||||
e = await preview(cluster.a, [new[str(cluster.a)]], kind="enable")
|
||||
assert {n["host_name"]: n["items"][0]["uuid"] for n in e["plan"]["nodes"]} == {"fw-a": new[str(cluster.a)], "fw-b": new[str(cluster.b)]}
|
||||
cluster.log.clear()
|
||||
r = await deploy(cluster.a, e["preview_id"])
|
||||
assert r["status"] == "awaiting_confirm"
|
||||
assert [w for w in _writes(cluster.log) if "toggleRule" in w[1]] == [
|
||||
(FW2, f"/firewall/filter/toggleRule/{new[str(cluster.b)]}/1"), (FW1, f"/firewall/filter/toggleRule/{new[str(cluster.a)]}/1")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restore_with_a_refused_move_is_verified_by_reading_and_only_warns(env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
deleted = next(iter((await _items(p["preview_id"])).values()))
|
||||
# moveRuleBefore answers HTTP 500 (an Ambiguous error for the client, P3): the position is read back
|
||||
env.fw.fail[("move", "*")] = httpx.Response(500, json={"errorMessage": "Unexpected error, check log for details"})
|
||||
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert result["status"] == "done" and result["warnings"] == ["BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_deadline_passing_while_confirm_reads_the_firewall_refuses(env):
|
||||
import time
|
||||
p = await preview(env.host_id, [R_UNUSED])
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() + timedelta(milliseconds=300))
|
||||
env.fw.fail[("search", "")] = lambda: time.sleep(0.6) # a slow firewall: the window closes during the read
|
||||
got = await refused(env.host_id, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
assert got["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
assert (await _op(p["preview_id"])).status == "awaiting_confirm" # left to the sweeper
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweeper_leaves_a_live_run_alone_even_past_its_deadline(env, monkeypatch): # review finding 2
|
||||
from app.services import opnsense_client as client_mod
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
real = client_mod.toggle_rule
|
||||
swept: list = []
|
||||
|
||||
async def toggle(host, uuid, enabled): # a second backend's sweeper starts while the first toggle is on the wire
|
||||
if not swept:
|
||||
await _backdate(p["preview_id"], revert_deadline=_now() - timedelta(seconds=1))
|
||||
swept.append(asyncio.create_task(_sweep())) # it gets the lock in the deploy's next checkpoint gap
|
||||
await asyncio.sleep(0.05)
|
||||
return await real(host, uuid, enabled)
|
||||
monkeypatch.setattr(client_mod, "toggle_rule", toggle)
|
||||
result = await deploy(env.host_id, p["preview_id"])
|
||||
assert await swept[0] == [] # fresh heartbeat: not taken over although the deadline passed
|
||||
assert result["status"] == "awaiting_confirm"
|
||||
assert {i.state for i in (await _items(p["preview_id"])).values()} == {"disabled"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_run_taken_over_by_another_backend_writes_nothing_more(env, monkeypatch): # review finding 2
|
||||
from app.services import opnsense_client as client_mod
|
||||
p = await preview(env.host_id, [R_UNUSED, R_DUP])
|
||||
real = client_mod.toggle_rule
|
||||
calls: list = []
|
||||
|
||||
async def toggle(host, uuid, enabled):
|
||||
calls.append(uuid)
|
||||
if len(calls) == 1: # another backend took the operation over while this toggle was on the wire
|
||||
await _backdate(p["preview_id"], status="reverting")
|
||||
return await real(host, uuid, enabled)
|
||||
monkeypatch.setattr(client_mod, "toggle_rule", toggle)
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
assert len(calls) == 1 # the conditional heartbeat before the second toggle found the operation gone
|
||||
assert (await _op(p["preview_id"])).status == "reverting" # left to the new owner, not overwritten
|
||||
states = sorted(i.state for i in (await _items(p["preview_id"])).values())
|
||||
assert states == ["disabled", "queued"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extra_management_ports_protect_rules_on_them(env): # review finding 3
|
||||
custom = "aaaaaaaa-0000-4000-8000-000000000910"
|
||||
env.fw.rules.append(rule(custom, 910, interface="wan", src="203.0.113.0/24", dport="8443", description="gui 8443"))
|
||||
await _seed_hits(env.host_id, [custom])
|
||||
ok = await preview(env.host_id, [custom]) # 8443 is not one of the fixed ports: not about management
|
||||
assert ok["plan"]["ports"] == []
|
||||
await _settings(ports=(8443,))
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[custom])
|
||||
assert got["params"]["reason"] == "management"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_peer_may_confirm_and_read_but_strangers_may_not(cluster, client, respx_mock): # review finding 4
|
||||
p = await preview(cluster.a, [R_UNUSED])
|
||||
await deploy(cluster.a, p["preview_id"])
|
||||
pid = uuid_lib.UUID(p["preview_id"])
|
||||
assert (await call(cluster.b, svc.get_operation, pid))["id"] == p["preview_id"]
|
||||
FakeFirewall("https://203.0.113.59").mount(respx_mock)
|
||||
other = uuid_lib.UUID((await _create_host(client, "fw-c", "https://203.0.113.59"))["id"])
|
||||
for fn in (svc.get_operation, svc.confirm, svc.revert_now, svc.retry):
|
||||
assert (await refused(other, fn, pid))["status"] == 404
|
||||
got = await refused(cluster.b, svc.cancel, pid, actor="admin") # previews/deploys stay bound to the anchor
|
||||
assert got["status"] == 404
|
||||
result = await call(cluster.b, svc.confirm, pid)
|
||||
assert result["status"] == "confirmed"
|
||||
|
||||
|
||||
async def _half_deleted_cluster(cluster):
|
||||
p = await preview(cluster.a, [R_UNUSED])
|
||||
await deploy(cluster.a, p["preview_id"])
|
||||
await call(cluster.a, svc.confirm, uuid_lib.UUID(p["preview_id"]))
|
||||
await _backdate(p["preview_id"], confirmed_at=_now() - timedelta(days=8))
|
||||
d = await preview(cluster.a, [R_UNUSED], kind="delete")
|
||||
cluster.master.fail[("delete", R_UNUSED)] = httpx.Response(200, json={"result": "failed"})
|
||||
result = await deploy(cluster.a, d["preview_id"])
|
||||
assert result["status"] == "failed"
|
||||
assert cluster.backup.get(R_UNUSED) is None and cluster.master.get(R_UNUSED) is not None
|
||||
del cluster.master.fail[("delete", R_UNUSED)]
|
||||
return d["preview_id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_half_done_cluster_delete_can_be_finished(cluster): # review finding 5
|
||||
await _half_deleted_cluster(cluster)
|
||||
again = await preview(cluster.a, [R_UNUSED], kind="delete")
|
||||
assert [(n["host_name"], len(n["items"])) for n in again["plan"]["nodes"]] == [("fw-b", 0), ("fw-a", 1)]
|
||||
cluster.log.clear()
|
||||
assert (await deploy(cluster.a, again["preview_id"]))["status"] == "done"
|
||||
assert [w for w in _writes(cluster.log) if "delRule" in w[1]] == [(FW1, f"/firewall/filter/delRule/{R_UNUSED}")]
|
||||
assert cluster.master.get(R_UNUSED) is None
|
||||
got = await refused(cluster.a, svc.preview, actor="admin", kind="delete", uuids=[R_UNUSED]) # nothing left
|
||||
assert got["params"]["reason"] == "not_candidate"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_half_done_cluster_delete_can_be_restored_where_it_was_deleted(cluster): # review finding 5
|
||||
delete_op = await _half_deleted_cluster(cluster)
|
||||
async with SessionLocal() as db:
|
||||
deleted = (await db.scalars(select(RuleCleanupItem).where(
|
||||
RuleCleanupItem.operation_id == uuid_lib.UUID(delete_op), RuleCleanupItem.state == "deleted"))).one()
|
||||
result = await call(cluster.a, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert result["status"] == "done" and [i["host_id"] for i in result["items"]] == [str(cluster.b)]
|
||||
assert cluster.backup.get(result["items"][0]["restored_uuid"])["enabled"] == "0"
|
||||
assert cluster.master.get(R_UNUSED)["enabled"] == "0" # the master's copy was never deleted: untouched
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_switching_a_block_rule_back_on_needs_an_acknowledgement_unless_proven_harmless(env): # review finding 7
|
||||
lock_out = "aaaaaaaa-0000-4000-8000-000000000920"
|
||||
env.fw.rules.append(rule(lock_out, 920, action="block", interface="wan", src="203.0.113.0/24", dport="443",
|
||||
enabled="0", description="block mgmt"))
|
||||
unknown = "aaaaaaaa-0000-4000-8000-000000000940" # source is a URL table: match unknown
|
||||
env.fw.rules.append(rule(unknown, 940, action="block", interface="wan", src="tbl", dport="443", enabled="0",
|
||||
description="block table"))
|
||||
harmless = "aaaaaaaa-0000-4000-8000-000000000930"
|
||||
env.fw.rules.append(rule(harmless, 930, action="block", proto="UDP", dport="5353", enabled="0", description="block mdns"))
|
||||
async with SessionLocal() as db: # both were switched off by a confirmed cleanup earlier
|
||||
op = RuleCleanupOperation(host_id=env.host_id, kind="disable", status="confirmed", actor="admin",
|
||||
expires_at=_now(), preview_fingerprint="x", options={}, confirmed_at=_now())
|
||||
db.add(op)
|
||||
await db.flush()
|
||||
for u in (lock_out, harmless, unknown):
|
||||
db.add(RuleCleanupItem(operation_id=op.id, host_id=env.host_id, rule_uuid=u, state="confirmed", description=u))
|
||||
await db.commit()
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="enable", uuids=[lock_out, harmless, unknown])
|
||||
assert got["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT" and got["params"]["uuids"] == ", ".join(sorted([lock_out, unknown]))
|
||||
ok = await preview(env.host_id, [lock_out, harmless, unknown], kind="enable", acknowledge=[lock_out, unknown])
|
||||
items = {i["uuid"]: i for i in ok["plan"]["nodes"][0]["items"]}
|
||||
assert items[lock_out]["needs_ack"] and items[lock_out]["notes"] == ["ENABLE_MAY_BLOCK_MANAGEMENT"]
|
||||
assert not items[harmless]["needs_ack"] and items[harmless]["protection"]["result"] == "false"
|
||||
|
||||
|
||||
async def _unknown_restore(env):
|
||||
"""A restore whose addRule answer was lost: the item stays unknown."""
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
p = await preview(env.host_id, [R_UNUSED], kind="delete")
|
||||
await deploy(env.host_id, p["preview_id"])
|
||||
deleted = next(iter((await _items(p["preview_id"])).values()))
|
||||
snapshot_row = dict(deleted.snapshot["row"])
|
||||
env.fw.fail[("add", "")] = httpx.ReadTimeout
|
||||
result = await call(env.host_id, svc.restore_item, deleted.id, actor="admin", confirm_name="fw-a")
|
||||
assert result["items"][0]["state"] == "unknown"
|
||||
del env.fw.fail[("add", "")]
|
||||
return result, snapshot_row
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unclear_restore_uses_the_stored_uuid_before_searching_by_content(env): # review finding 10
|
||||
result, row = await _unknown_restore(env)
|
||||
# on the firewall: the real copy and a content-identical twin someone made by hand
|
||||
real, twin = "cccccccc-0000-4000-8000-000000000001", "cccccccc-0000-4000-8000-000000000002"
|
||||
for u, seq in ((real, 950), (twin, 960)):
|
||||
env.fw.rules.append({**row, "uuid": u, "sequence": str(seq), "sort_order": f"400000.{seq:07d}"})
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleCleanupItem).where(RuleCleanupItem.id == uuid_lib.UUID(result["items"][0]["id"]))
|
||||
.values(restored_uuid=real))
|
||||
await db.commit()
|
||||
done = await call(env.host_id, svc.retry, uuid_lib.UUID(result["id"]))
|
||||
assert done["items"][0]["state"] == "restored" and done["items"][0]["restored_uuid"] == real
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unclear_restore_with_twins_and_no_stored_uuid_stays_unknown(env): # review finding 10
|
||||
result, row = await _unknown_restore(env)
|
||||
for u, seq in (("cccccccc-0000-4000-8000-000000000003", 950), ("cccccccc-0000-4000-8000-000000000004", 960)):
|
||||
env.fw.rules.append({**row, "uuid": u, "sequence": str(seq), "sort_order": f"400000.{seq:07d}"})
|
||||
done = await call(env.host_id, svc.retry, uuid_lib.UUID(result["id"]))
|
||||
assert done["items"][0]["state"] == "unknown" and done["items"][0]["message"] == "BEREINIGUNG_AUSGANG_UNKLAR"
|
||||
env.fw.rules = [r for r in env.fw.rules if not r["uuid"].startswith("cccccccc")]
|
||||
done = await call(env.host_id, svc.retry, uuid_lib.UUID(result["id"])) # nothing there: it was never created
|
||||
assert done["items"][0]["state"] == "failed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_outdated_acknowledgement_text_is_refused(env): # review finding 11
|
||||
for version in ("0", "2", ""):
|
||||
got = await refused(env.host_id, svc.preview, actor="admin", kind="disable", uuids=[R_BLOCK],
|
||||
acknowledge=[R_BLOCK], ack_text_version=version)
|
||||
assert got["status"] == 409 and got["code"] == "BEREINIGUNG_BESTAETIGUNGSTEXT_VERALTET", version
|
||||
assert env.fw.calls == [] # refused before anything is read
|
||||
ok = await preview(env.host_id, [R_BLOCK], acknowledge=[R_BLOCK], ack_text_version=svc.ACK_TEXT_VERSION)
|
||||
assert ok["plan"]["acknowledged"] == [R_BLOCK]
|
||||
@@ -0,0 +1,910 @@
|
||||
"""Rule cleanup API (1.31, package 6): router, rights, locks in both directions, notification, settings.
|
||||
|
||||
Who may do what is enforced by TWO layers (AuthMiddleware via services/api_token_service.py, and the router itself).
|
||||
The role matrix therefore runs three times: with both layers, with the middleware's rights checks switched off (the
|
||||
router alone must refuse) and with the router's checks switched off (the middleware alone must refuse). A mutation
|
||||
of either layer is then caught by one of the variants (tests/mutations_cleanup_1_31_api.json).
|
||||
Addresses: documentation/private ranges only."""
|
||||
import uuid as uuid_lib
|
||||
from datetime import timedelta
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app import auth_middleware
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.rule_cleanup import RuleCleanupItem, RuleCleanupOperation
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.routers import rule_cleanup as router_mod
|
||||
from app.schemas.rule_cleanup import CleanupItemRead, CleanupOperationRead
|
||||
from app.services import api_token_service, notification_checks
|
||||
from app.services import rule_cleanup_service as svc
|
||||
from tests.rule_cleanup_fake import MGMT_SOURCE, R_ALIAS, R_BASE, R_BLOCK, R_DUP, R_MGMT, R_TAIL, R_UNUSED
|
||||
from tests.test_api_tokens import _bearer, _login, _make
|
||||
from tests.test_rule_cleanup import ( # noqa: F401 (fixtures are used by name)
|
||||
FW1, FW2, _backdate, _confirmed_disable, _fresh_cache, _items, _now, _op, _settings, cluster, env, refused,
|
||||
)
|
||||
|
||||
PW = "cleanup-api-test-pw-1"
|
||||
NOHOST = "00000000-0000-0000-0000-0000000000aa"
|
||||
NOID = "00000000-0000-0000-0000-0000000000bb"
|
||||
|
||||
|
||||
# ── Principals ────────────────────────────────────────────────────────────
|
||||
|
||||
async def _user(client, name: str, role: str):
|
||||
resp = await client.post("/api/users", json={"username": name, "password": PW, "role": role})
|
||||
assert resp.status_code in (201, 409), resp.text # the account survives between tests
|
||||
return await _login(name, PW)
|
||||
|
||||
|
||||
async def _token_of(session, name: str, scope: str):
|
||||
resp = await session.post("/api/tokens", json={"name": name, "scope": scope, "current_password": PW})
|
||||
assert resp.status_code == 201, resp.text
|
||||
return _bearer(resp.json()["token"])
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def who(client):
|
||||
"""Every kind of caller: sessions of an admin, an operator and a read-only account, and tokens of each scope
|
||||
(an admin's read/deploy/admin, an operator's deploy, a read-only account's read)."""
|
||||
operator = await _user(client, "cl-operator", "operator")
|
||||
read_only = await _user(client, "cl-readonly", "read_only")
|
||||
callers = {
|
||||
"admin": client, "operator": operator, "read_only": read_only,
|
||||
"tok_read": _bearer((await _make(client, "read", name="cl-read"))["token"]),
|
||||
"tok_deploy": _bearer((await _make(client, "deploy", name="cl-deploy"))["token"]),
|
||||
"tok_admin": _bearer((await _make(client, "admin", name="cl-admin"))["token"]),
|
||||
"tok_op_deploy": await _token_of(operator, "cl-op-deploy", "deploy"),
|
||||
"tok_ro_read": await _token_of(read_only, "cl-ro-read", "read"),
|
||||
}
|
||||
yield callers
|
||||
for name, session in callers.items():
|
||||
if name != "admin":
|
||||
await session.aclose()
|
||||
|
||||
|
||||
async def _governance(client, may_deploy: bool):
|
||||
assert (await client.put("/api/settings/governance", json={"operators_may_deploy": may_deploy})).status_code == 200
|
||||
|
||||
|
||||
# ── The role matrix ───────────────────────────────────────────────────────
|
||||
|
||||
ROUTES = {
|
||||
"list": ("GET", "", None),
|
||||
"get": ("GET", "/{op}", None),
|
||||
"preview": ("POST", "/preview", {"kind": "disable", "uuids": ["a"]}),
|
||||
"cancel": ("POST", "/{op}/cancel", None),
|
||||
"deploy": ("POST", "/{op}/deploy", {"confirm_name": "x"}),
|
||||
"confirm": ("POST", "/{op}/confirm", None),
|
||||
"revert": ("POST", "/{op}/revert", None),
|
||||
"retry": ("POST", "/{op}/retry", None),
|
||||
"restore": ("POST", "/items/{op}/restore", {"confirm_name": "x"}),
|
||||
}
|
||||
READ_ONLY_CODES = {"READ_ONLY_ACCESS", "API_TOKEN_SCOPE_FORBIDDEN"}
|
||||
|
||||
|
||||
def verdict(route: str, caller: str, may_deploy: bool):
|
||||
"""None: the caller passes the gate (the unknown host then answers 404). Otherwise the acceptable 403 codes."""
|
||||
if route in ("list", "get"):
|
||||
return None
|
||||
if caller in ("read_only", "tok_ro_read"):
|
||||
return READ_ONLY_CODES
|
||||
if caller == "tok_read":
|
||||
return {"API_TOKEN_SCOPE_FORBIDDEN"}
|
||||
if route in ("preview", "cancel"):
|
||||
return None
|
||||
if route == "confirm": # a human in a browser session; no token, whatever its scope
|
||||
return None if caller in ("admin", "operator") else {"API_TOKEN_SCOPE_FORBIDDEN"}
|
||||
if route in ("deploy", "retry", "restore"): # admin only, even though operators_may_deploy is on
|
||||
if caller in ("admin", "tok_admin"):
|
||||
return None
|
||||
return {"admin": None, "operator": {"ADMIN_REQUIRED"}, "tok_deploy": {"API_TOKEN_SCOPE_FORBIDDEN"},
|
||||
"tok_op_deploy": {"API_TOKEN_SCOPE_FORBIDDEN", "ADMIN_REQUIRED"}}[caller]
|
||||
assert route == "revert" # an admin, or an operator while operators_may_deploy is on
|
||||
if caller in ("admin", "tok_deploy", "tok_admin"):
|
||||
return None
|
||||
return None if may_deploy else {"DEPLOY_NEEDS_ADMIN"}
|
||||
|
||||
|
||||
async def _gate(session, route: str):
|
||||
method, path, body = ROUTES[route]
|
||||
url = f"/api/hosts/{NOHOST}/rule-cleanup" + path.format(op=NOID)
|
||||
resp = await session.request(method, url, **({"json": body} if body is not None else {}))
|
||||
code = (resp.json().get("detail") or {}).get("code") if resp.headers.get("content-type", "").startswith("application/json") else None
|
||||
return resp.status_code, code
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("may_deploy", [True, False])
|
||||
@pytest.mark.parametrize("variant", ["both", "router_only", "middleware_only"])
|
||||
async def test_every_route_for_every_caller(client, who, monkeypatch, variant, may_deploy):
|
||||
await _governance(client, may_deploy)
|
||||
try:
|
||||
if variant == "router_only": # switch the middleware's rights checks off: the router must still refuse
|
||||
monkeypatch.setattr(api_token_service, "denies", lambda *a, **k: False)
|
||||
monkeypatch.setattr(api_token_service, "operator_denies", lambda *a, **k: False)
|
||||
monkeypatch.setattr(api_token_service, "is_rollout", lambda *a, **k: False)
|
||||
|
||||
class _Everything: # the read-only gate of the middleware lets this request through
|
||||
def __contains__(self, _):
|
||||
return True
|
||||
monkeypatch.setattr(auth_middleware, "_READ_ONLY_ALLOWED_WRITES", _Everything())
|
||||
if variant == "middleware_only": # switch the router's checks off: the middleware must still refuse
|
||||
async def _nothing_async(*a, **k):
|
||||
return None
|
||||
for name in ("_writer", "_admin_only", "_session_only"):
|
||||
monkeypatch.setattr(router_mod, name, lambda *a, **k: None)
|
||||
monkeypatch.setattr(router_mod, "_deployer", _nothing_async)
|
||||
problems = []
|
||||
for route in ROUTES:
|
||||
for caller, session in who.items():
|
||||
want = verdict(route, caller, may_deploy)
|
||||
got = await _gate(session, route)
|
||||
ok = got == (404, "HOST_NICHT_GEFUNDEN") if want is None else (got[0] == 403 and got[1] in want)
|
||||
if not ok:
|
||||
problems.append((route, caller, got, want))
|
||||
assert not problems, problems
|
||||
finally:
|
||||
await _governance(client, True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method,path,session_only,admin_only,rollout", [
|
||||
("POST", "/api/hosts/h/rule-cleanup/preview", False, False, False),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/cancel", False, False, False),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/confirm", True, False, False),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/confirm/", True, False, False),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/deploy", False, True, True),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/deploy/", False, True, False),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/revert", False, False, True),
|
||||
("POST", "/api/hosts/h/rule-cleanup/o/retry", False, True, True),
|
||||
("POST", "/api/hosts/h/rule-cleanup/items/i/restore", False, True, False),
|
||||
("GET", "/api/hosts/h/rule-cleanup", False, False, False),
|
||||
("GET", "/api/hosts/h/rule-cleanup/o", False, False, False),
|
||||
])
|
||||
def test_pattern_table(method, path, session_only, admin_only, rollout):
|
||||
denied_for_read = method == "POST"
|
||||
assert api_token_service.denies("read", method, path) is denied_for_read
|
||||
# a token below scope admin: admin-only routes and confirm are closed
|
||||
assert api_token_service.denies("deploy", method, path) is (session_only or admin_only)
|
||||
# an admin-scope token reaches everything except what only a session may do
|
||||
assert api_token_service.denies("admin", method, path) is session_only
|
||||
assert api_token_service.operator_denies(method, path) is admin_only
|
||||
assert api_token_service.is_rollout(method, path) is rollout
|
||||
|
||||
|
||||
def test_governance_settings_are_admin_writes_for_operators_and_deploy_tokens():
|
||||
assert api_token_service.operator_denies("PUT", "/api/settings/governance") is True
|
||||
assert api_token_service.denies("deploy", "PUT", "/api/settings/governance") is True
|
||||
assert api_token_service.denies("admin", "PUT", "/api/settings/governance") is False
|
||||
|
||||
|
||||
# ── A full run over the API: shapes, audit, input forms ──────────────────
|
||||
|
||||
def _base(env_or_id) -> str:
|
||||
return f"/api/hosts/{getattr(env_or_id, 'host_id', env_or_id)}/rule-cleanup"
|
||||
|
||||
|
||||
async def _audit(**filters):
|
||||
async with SessionLocal() as db:
|
||||
rows = (await db.scalars(select(AuditEvent).where(*[getattr(AuditEvent, k) == v for k, v in filters.items()])
|
||||
.order_by(AuditEvent.created_at))).all()
|
||||
return rows
|
||||
|
||||
|
||||
async def _open(session, env, uuids=(R_UNUSED, R_DUP), name="fw-a"):
|
||||
"""Preview + deploy over the API; the run waits for its confirmation."""
|
||||
pv = await session.post(_base(env) + "/preview", json={"kind": "disable", "uuids": list(uuids)})
|
||||
assert pv.status_code == 200, pv.text
|
||||
op = await session.post(f"{_base(env)}/{pv.json()['preview_id']}/deploy", json={"confirm_name": name})
|
||||
assert op.status_code == 200, op.text
|
||||
return op.json()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_deploy_confirm_over_the_api_with_shapes_and_audit(client, env):
|
||||
pv = await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED, R_DUP]})
|
||||
assert pv.status_code == 200, pv.text
|
||||
body = pv.json()
|
||||
assert {"preview_id", "expires_at", "kind", "plan", "max_rules", "window_seconds", "operation"} <= set(body)
|
||||
assert body["max_rules"] == 10 and body["kind"] == "disable" and body["window_seconds"] > 0
|
||||
assert {i["uuid"] for i in body["plan"]["nodes"][0]["items"]} == {R_UNUSED, R_DUP}
|
||||
operation = body["operation"]
|
||||
assert set(operation) == set(CleanupOperationRead.model_fields) # the UI gets exactly the documented fields
|
||||
assert operation["status"] == "preview" and operation["plan"] and operation["items"] == []
|
||||
assert operation["actor"] == "admin" and operation["host_name"] == "fw-a"
|
||||
# The middleware does not log /preview; the service writes the one entry (and it is no history change).
|
||||
assert [(a.action, a.status) for a in await _audit(kind="rule_cleanup")] == [("preview", "requested")]
|
||||
assert await _audit(kind="host", action="preview") == []
|
||||
assert env.fw.writes() == []
|
||||
|
||||
pid = body["preview_id"]
|
||||
wrong = await client.post(f"{_base(env)}/{pid}/deploy", json={"confirm_name": "fw-x"})
|
||||
assert wrong.status_code == 409 and wrong.json()["detail"]["code"] == "HOSTNAMEN_EXAKT_EINGEBEN"
|
||||
deployed = await client.post(f"{_base(env)}/{pid}/deploy", json={"confirm_name": "fw-a"})
|
||||
assert deployed.status_code == 200, deployed.text
|
||||
op = deployed.json()
|
||||
assert set(op) == set(CleanupOperationRead.model_fields) and all(set(i) == set(CleanupItemRead.model_fields) for i in op["items"])
|
||||
assert op["status"] == "awaiting_confirm" and op["revert_deadline"] and op["plan"] is None
|
||||
assert (op["can_confirm"], op["can_revert"], op["can_retry"]) == (True, True, False)
|
||||
assert {(i["rule_uuid"], i["state"], i["host_name"]) for i in op["items"]} == {(R_UNUSED, "disabled", "fw-a"), (R_DUP, "disabled", "fw-a")}
|
||||
assert env.fw.get(R_UNUSED)["enabled"] == "0"
|
||||
# Audit: one request pair from the middleware (kind host), one entry per rule from the service - no duplicates.
|
||||
# the wrong-name attempt and the real one: a request entry and an outcome each, from the middleware only
|
||||
assert sorted(a.status for a in await _audit(kind="host", action="deploy")) == ["accepted", "rejected", "requested", "requested"]
|
||||
assert [a.status for a in await _audit(kind="rule_cleanup", action="disable")] == ["accepted", "accepted"]
|
||||
assert await _audit(kind="rule_cleanup", action="deploy") == [] and await _audit(kind="host", action="disable") == []
|
||||
|
||||
listing = (await client.get(_base(env))).json()
|
||||
assert [o["id"] for o in listing] == [pid] and listing[0]["status"] == "awaiting_confirm"
|
||||
one = (await client.get(f"{_base(env)}/{pid}")).json()
|
||||
assert one == listing[0]
|
||||
confirmed = await client.post(f"{_base(env)}/{pid}/confirm")
|
||||
assert confirmed.status_code == 200, confirmed.text
|
||||
c = confirmed.json()
|
||||
assert c["status"] == "confirmed" and c["revert_deadline"] is None and {i["state"] for i in c["items"]} == {"confirmed"}
|
||||
assert (c["can_confirm"], c["can_revert"], c["can_retry"]) == (False, False, False)
|
||||
assert [a.status for a in await _audit(kind="rule_cleanup", action="confirm")] == ["confirmed", "confirmed"]
|
||||
assert sorted(a.status for a in await _audit(kind="host", action="confirm")) == ["accepted", "requested"]
|
||||
again = await client.post(f"{_base(env)}/{pid}/confirm")
|
||||
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_FRIST_ABGELAUFEN"
|
||||
missing = await client.get(f"{_base(env)}/{NOID}")
|
||||
assert missing.status_code == 404 and missing.json()["detail"]["code"] == "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||
assert (await client.get(f"/api/hosts/{NOHOST}/rule-cleanup")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_input_forms_and_validation(client, env):
|
||||
url = _base(env) + "/preview"
|
||||
# items[] with per-rule kind/mode and acknowledgement; a "probable" rule needs the acknowledgement
|
||||
plain = {"items": [{"uuid": R_UNUSED, "mode": "disable"}, {"uuid": R_BLOCK, "kind": "disable"}]}
|
||||
got = await client.post(url, json=plain)
|
||||
assert got.status_code == 409 and got.json()["detail"]["code"] == "BEREINIGUNG_BESTAETIGUNG_FEHLT"
|
||||
assert R_BLOCK in got.json()["detail"]["params"]["uuids"]
|
||||
ack = {"items": [{"uuid": R_UNUSED, "mode": "disable"}, {"uuid": R_BLOCK, "kind": "disable", "ack": True}]}
|
||||
done = await client.post(url, json=ack)
|
||||
assert done.status_code == 200, done.text
|
||||
assert {i["uuid"] for i in done.json()["plan"]["nodes"][0]["items"]} == {R_UNUSED, R_BLOCK}
|
||||
assert done.json()["plan"]["acknowledged"] == [R_BLOCK]
|
||||
# the old form: kind + uuids + acknowledge
|
||||
old = await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED, R_BLOCK], "acknowledge": [R_BLOCK]})
|
||||
assert old.status_code == 200 and (await _op(done.json()["preview_id"])).status == "cancelled" # one preview per actor
|
||||
# bad input
|
||||
for bad in ({"uuids": [R_UNUSED]}, {"items": []}):
|
||||
r = await client.post(url, json=bad)
|
||||
assert r.status_code == 422 and r.json()["detail"]["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG", bad
|
||||
for bad in ({"kind": "disable", "uuids": []}, {"kind": "disable", "uuids": ["../x"]}):
|
||||
r = await client.post(url, json=bad)
|
||||
assert r.status_code == 422 and r.json()["detail"]["code"] == "BEREINIGUNG_AUSWAHL_UNGUELTIG", bad
|
||||
mixed = {"items": [{"uuid": R_UNUSED, "kind": "disable"}, {"uuid": R_DUP, "kind": "delete"}]}
|
||||
assert (await client.post(url, json=mixed)).status_code == 422
|
||||
assert (await client.post(url, json={"kind": "restore", "uuids": [R_UNUSED]})).status_code == 422
|
||||
assert (await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED], "unused_days": 0})).status_code == 422
|
||||
assert (await client.post(f"/api/hosts/{NOHOST}/rule-cleanup/preview", json={"kind": "disable", "uuids": [R_UNUSED]})).status_code == 404
|
||||
# the settings reach the service: the limit and the fail-closed lock without management sources
|
||||
assert (await client.put("/api/settings/governance", json={"cleanup_max_rules": 1})).status_code == 200
|
||||
many = await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED, R_DUP]})
|
||||
assert many.status_code == 422 and many.json()["detail"]["code"] == "BEREINIGUNG_ZU_VIELE_REGELN" and many.json()["detail"]["params"]["max"] == 1
|
||||
assert (await client.put("/api/settings/governance", json={"cleanup_management_sources": []})).status_code == 200
|
||||
locked = await client.post(url, json={"kind": "disable", "uuids": [R_UNUSED]})
|
||||
assert locked.status_code == 409 and locked.json()["detail"]["code"] == "BEREINIGUNG_VERWALTUNGSQUELLEN_FEHLEN"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_preview_belongs_to_its_actor_and_can_be_cancelled(client, env, who):
|
||||
operator = who["operator"]
|
||||
mine = (await operator.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})).json()
|
||||
assert mine["operation"]["actor"] == "cl-operator"
|
||||
# the admin neither sees it as his own nor deploys it; the operator cannot deploy at all
|
||||
stranger = await client.post(f"{_base(env)}/{mine['preview_id']}/deploy", json={"confirm_name": "fw-a"})
|
||||
assert stranger.status_code == 404 and stranger.json()["detail"]["code"] == "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||
refused_op = await operator.post(f"{_base(env)}/{mine['preview_id']}/deploy", json={"confirm_name": "fw-a"})
|
||||
assert refused_op.status_code == 403 and refused_op.json()["detail"]["code"] == "ADMIN_REQUIRED"
|
||||
assert (await client.post(f"{_base(env)}/{mine['preview_id']}/cancel")).status_code == 404 # not the admin's
|
||||
cancelled = await operator.post(f"{_base(env)}/{mine['preview_id']}/cancel")
|
||||
assert cancelled.status_code == 200 and cancelled.json()["status"] == "cancelled"
|
||||
# the admin's refused attempt (not his preview) and the operator's real cancel: a request entry and an outcome each
|
||||
assert sorted(a.status for a in await _audit(kind="host", action="cancel")) == ["accepted", "rejected", "requested", "requested"]
|
||||
again = await operator.post(f"{_base(env)}/{mine['preview_id']}/cancel")
|
||||
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
gone = await client.post(f"{_base(env)}/{mine['preview_id']}/deploy", json={"confirm_name": "fw-a"})
|
||||
assert gone.status_code == 404 # (not the admin's preview)
|
||||
assert env.fw.writes() == []
|
||||
# an operator prepares, the admin prepares his own and executes; the operator may confirm afterwards
|
||||
own = (await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})).json()
|
||||
deployed = await client.post(f"{_base(env)}/{own['preview_id']}/deploy", json={"confirm_name": "fw-a"})
|
||||
assert deployed.status_code == 200
|
||||
confirmed = await operator.post(f"{_base(env)}/{own['preview_id']}/confirm")
|
||||
assert confirmed.status_code == 200 and confirmed.json()["status"] == "confirmed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_confirm_never_works_with_a_token_and_changes_nothing(client, env, who):
|
||||
op = await _open(client, env)
|
||||
for name in ("tok_admin", "tok_deploy", "tok_op_deploy"):
|
||||
r = await who[name].post(f"{_base(env)}/{op['id']}/confirm")
|
||||
assert r.status_code == 403 and r.json()["detail"]["code"] == "API_TOKEN_SCOPE_FORBIDDEN", name
|
||||
state = await _op(op["id"])
|
||||
assert state.status == "awaiting_confirm" and state.revert_deadline is not None
|
||||
assert {i.state for i in (await _items(op["id"])).values()} == {"disabled"}
|
||||
# the same token may read the run, and the browser session confirms
|
||||
assert (await who["tok_read"].get(f"{_base(env)}/{op['id']}")).json()["can_confirm"] is True
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/confirm")).json()["status"] == "confirmed"
|
||||
denied = await _audit(kind="host", action="confirm", status="rejected")
|
||||
assert not denied # the middleware refused before the router: no request entry for a refused token confirm
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revert_is_for_admins_and_for_operators_only_while_they_may_deploy(client, env, who):
|
||||
op = await _open(client, env)
|
||||
url = f"{_base(env)}/{op['id']}/revert"
|
||||
await _governance(client, False)
|
||||
try:
|
||||
assert (await who["operator"].post(url)).status_code == 403
|
||||
assert (await who["tok_op_deploy"].post(url)).status_code == 403
|
||||
assert (await who["read_only"].post(url)).status_code == 403
|
||||
assert (await _op(op["id"])).status == "awaiting_confirm" and env.fw.get(R_UNUSED)["enabled"] == "0"
|
||||
finally:
|
||||
await _governance(client, True)
|
||||
reverted = await who["operator"].post(url)
|
||||
assert reverted.status_code == 200, reverted.text
|
||||
r = reverted.json()
|
||||
assert r["status"] == "reverted" and {i["state"] for i in r["items"]} == {"reverted"} and not r["can_revert"]
|
||||
assert env.fw.get(R_UNUSED)["enabled"] == "1" and env.fw.get(R_DUP)["enabled"] == "1"
|
||||
again = await client.post(url)
|
||||
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_after_a_failed_revert_is_admin_only(client, env, who):
|
||||
op = await _open(client, env)
|
||||
env.fw.fail[("toggle", R_DUP)] = httpx.Response(200, json={"result": "failed"})
|
||||
failed = await client.post(f"{_base(env)}/{op['id']}/revert")
|
||||
assert failed.status_code == 200 and failed.json()["status"] == "revert_failed" and failed.json()["can_retry"] is True
|
||||
del env.fw.fail[("toggle", R_DUP)]
|
||||
url = f"{_base(env)}/{op['id']}/retry"
|
||||
for name, code in (("operator", "ADMIN_REQUIRED"), ("tok_deploy", "API_TOKEN_SCOPE_FORBIDDEN"), ("read_only", "READ_ONLY_ACCESS")):
|
||||
r = await who[name].post(url)
|
||||
assert r.status_code == 403 and r.json()["detail"]["code"] == code, name
|
||||
assert (await _op(op["id"])).status == "revert_failed"
|
||||
done = await who["tok_admin"].post(url)
|
||||
assert done.status_code == 200 and done.json()["status"] == "reverted" and env.fw.get(R_DUP)["enabled"] == "1"
|
||||
nothing = await client.post(url)
|
||||
assert nothing.status_code == 409 and nothing.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_and_restore_over_the_api(client, env, who):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
pv = (await client.post(_base(env) + "/preview", json={"kind": "delete", "uuids": [R_UNUSED]})).json()
|
||||
assert pv["operation"]["kind"] == "delete"
|
||||
done = await client.post(f"{_base(env)}/{pv['preview_id']}/deploy", json={"confirm_name": "fw-a"})
|
||||
assert done.status_code == 200, done.text
|
||||
op = done.json()
|
||||
assert op["status"] == "done" and op["kind"] == "delete" and (op["can_confirm"], op["can_revert"], op["can_retry"]) == (False,) * 3
|
||||
item = op["items"][0]
|
||||
assert item["state"] == "deleted" and item["restorable"] is True and item["has_snapshot"] is True
|
||||
assert env.fw.get(R_UNUSED) is None
|
||||
assert [a.status for a in await _audit(kind="rule_cleanup", action="delete")] == ["accepted"]
|
||||
url = f"{_base(env)}/items/{item['id']}/restore"
|
||||
for name, code in (("operator", "ADMIN_REQUIRED"), ("tok_deploy", "API_TOKEN_SCOPE_FORBIDDEN"), ("read_only", "READ_ONLY_ACCESS")):
|
||||
r = await who[name].post(url, json={"confirm_name": "fw-a"})
|
||||
assert r.status_code == 403 and r.json()["detail"]["code"] == code, name
|
||||
wrong = await client.post(url, json={"confirm_name": "nope"})
|
||||
assert wrong.status_code == 409 and wrong.json()["detail"]["code"] == "HOSTNAMEN_EXAKT_EINGEBEN"
|
||||
assert env.fw.get(R_UNUSED) is None
|
||||
restored = await who["tok_admin"].post(url, json={"confirm_name": "fw-a"})
|
||||
assert restored.status_code == 200, restored.text
|
||||
r = restored.json()
|
||||
assert r["kind"] == "restore" and r["status"] == "done" and r["warnings"] == []
|
||||
new = r["items"][0]["restored_uuid"]
|
||||
assert new and r["items"][0]["state"] == "restored" and env.fw.get(new)["enabled"] == "0"
|
||||
listed = {o["id"]: o for o in (await client.get(_base(env))).json()}
|
||||
source = next(i for i in listed[op["id"]]["items"] if i["id"] == item["id"])
|
||||
assert source["restorable"] is False and source["restored_uuid"] == new
|
||||
assert [a.status for a in await _audit(kind="rule_cleanup", action="restore")] == ["accepted"]
|
||||
again = await client.post(url, json={"confirm_name": "fw-a"})
|
||||
assert again.status_code == 409 and again.json()["detail"]["code"] == "BEREINIGUNG_AKTION_NICHT_MOEGLICH"
|
||||
nothing = await client.post(f"{_base(env)}/{op['id']}/retry")
|
||||
assert nothing.status_code == 409 # nothing unclear in a finished delete
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restore_reports_a_lost_position_as_a_warning(client, env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
pv = (await client.post(_base(env) + "/preview", json={"kind": "delete", "uuids": [R_UNUSED]})).json()
|
||||
op = (await client.post(f"{_base(env)}/{pv['preview_id']}/deploy", json={"confirm_name": "fw-a"})).json()
|
||||
env.fw.rules = [r for r in env.fw.rules if r["uuid"] != R_BLOCK] # the anchor is gone
|
||||
r = await client.post(f"{_base(env)}/items/{op['items'][0]['id']}/restore", json={"confirm_name": "fw-a"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["warnings"] == ["BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"]
|
||||
assert r.json()["items"][0]["message"] == "BEREINIGUNG_POSITION_NICHT_WIEDERHERSTELLBAR"
|
||||
missing = await client.post(f"{_base(env)}/items/{NOID}/restore", json={"confirm_name": "fw-a"})
|
||||
assert missing.status_code == 404 and missing.json()["detail"]["code"] == "BEREINIGUNG_NICHT_GEFUNDEN"
|
||||
|
||||
|
||||
# ── Settings ──────────────────────────────────────────────────────────────
|
||||
|
||||
GOV = "/api/settings/governance"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_management_sources_are_validated_normalised_and_audited(client):
|
||||
put = await client.put(GOV, json={"cleanup_management_sources": [" 203.0.113.10 ", "10.20.30.5/24", "203.0.113.10", "2001:db8::1"],
|
||||
"cleanup_max_rules": 25, "cleanup_delete_min_days": 30})
|
||||
assert put.status_code == 200, put.text
|
||||
got = put.json()
|
||||
assert got["cleanup_management_sources"] == ["203.0.113.10", "10.20.30.0/24", "2001:db8::1"]
|
||||
assert got["cleanup_management_sources_configured"] is True and got["cleanup_max_rules"] == 25 and got["cleanup_delete_min_days"] == 30
|
||||
assert (await client.get(GOV)).json() == got
|
||||
audit = await _audit(kind="settings", action="cleanup_settings")
|
||||
assert len(audit) == 1 and audit[0].actor == "admin"
|
||||
changes = audit[0].details["changes"]
|
||||
assert changes["cleanup_management_sources"]["new"] == got["cleanup_management_sources"]
|
||||
assert changes["cleanup_max_rules"] == {"old": 10, "new": 25}
|
||||
# an unchanged value writes no further entry; a partial update leaves the rest alone
|
||||
assert (await client.put(GOV, json={"cleanup_max_rules": 25, "operators_may_deploy": True})).json()["cleanup_delete_min_days"] == 30
|
||||
assert len(await _audit(kind="settings", action="cleanup_settings")) == 1
|
||||
bad_sources = [["fw.example.test"], ["203.0.113.999"], ["0.0.0.0/0"], ["::/0"], ["203.0.113.1", ""], [f"10.20.30.{i}" for i in range(21)], ["10.20.30.0/33"]]
|
||||
for sources in bad_sources:
|
||||
r = await client.put(GOV, json={"cleanup_management_sources": sources})
|
||||
assert r.status_code == 422 and r.json()["detail"]["code"] == "BEREINIGUNG_EINSTELLUNG_UNGUELTIG", sources
|
||||
assert r.json()["detail"]["params"]["field"] == "cleanup_management_sources"
|
||||
for field, value in (("cleanup_max_rules", 0), ("cleanup_max_rules", 26), ("cleanup_delete_min_days", 0), ("cleanup_delete_min_days", 366)):
|
||||
r = await client.put(GOV, json={field: value})
|
||||
assert r.status_code == 422 and r.json()["detail"]["params"]["field"] == field, (field, value)
|
||||
assert (await client.get(GOV)).json() == got # nothing of the refused requests was stored
|
||||
assert len(await _audit(kind="settings", action="cleanup_settings")) == 1
|
||||
assert (await client.put(GOV, json={"cleanup_management_sources": [f"10.20.30.{i}" for i in range(20)]})).status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_management_ports_are_validated_normalised_audited_and_hidden(client, who): # review finding 3
|
||||
put = await client.put(GOV, json={"cleanup_management_ports": [8443, "2222", 8443]})
|
||||
assert put.status_code == 200, put.text
|
||||
assert put.json()["cleanup_management_ports"] == [2222, 8443] and put.json()["cleanup_management_ports_configured"] is True
|
||||
audit = await _audit(kind="settings", action="cleanup_settings")
|
||||
assert audit[-1].details["changes"]["cleanup_management_ports"] == {"old": [], "new": [2222, 8443]}
|
||||
for ports in ([0], [65536], ["ssh"], [True], [-1], list(range(1, 22))):
|
||||
r = await client.put(GOV, json={"cleanup_management_ports": ports})
|
||||
assert r.status_code == 422 and r.json()["detail"]["params"]["field"] == "cleanup_management_ports", ports
|
||||
assert (await client.get(GOV)).json()["cleanup_management_ports"] == [2222, 8443]
|
||||
for name in ("operator", "read_only"):
|
||||
seen = (await who[name].get(GOV)).json()
|
||||
assert seen["cleanup_management_ports"] == [] and seen["cleanup_management_ports_configured"] is True, name
|
||||
r = await who[name].put(GOV, json={"cleanup_management_ports": [1]})
|
||||
assert r.status_code == 403, name
|
||||
assert (await client.put(GOV, json={"cleanup_management_ports": []})).json()["cleanup_management_ports_configured"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("variant", ["both", "router_only"])
|
||||
async def test_only_an_admin_writes_the_cleanup_settings(client, who, monkeypatch, variant):
|
||||
await client.put(GOV, json={"cleanup_management_sources": ["203.0.113.10"]})
|
||||
if variant == "router_only":
|
||||
monkeypatch.setattr(api_token_service, "denies", lambda *a, **k: False)
|
||||
monkeypatch.setattr(api_token_service, "operator_denies", lambda *a, **k: False)
|
||||
# router_only: an admin-owned deploy/read token IS an admin account: only the token layer (switched off) stops it
|
||||
attempts = {"operator": {"ADMIN_REQUIRED"}, "tok_op_deploy": {"API_TOKEN_SCOPE_FORBIDDEN", "ADMIN_REQUIRED"},
|
||||
"read_only": {"READ_ONLY_ACCESS", "ADMIN_REQUIRED"}}
|
||||
if variant == "both":
|
||||
attempts.update({"tok_deploy": {"API_TOKEN_SCOPE_FORBIDDEN"}, "tok_read": {"API_TOKEN_SCOPE_FORBIDDEN"}})
|
||||
for name, codes in attempts.items():
|
||||
r = await who[name].put(GOV, json={"cleanup_management_sources": ["198.51.100.7"], "cleanup_max_rules": 3})
|
||||
assert r.status_code == 403 and r.json()["detail"]["code"] in codes, (name, r.text)
|
||||
assert (await client.get(GOV)).json()["cleanup_management_sources"] == ["203.0.113.10"]
|
||||
assert (await client.get(GOV)).json()["cleanup_max_rules"] == 10
|
||||
# an admin-scope token acts as the admin and the audit names the token
|
||||
ok = await who["tok_admin"].put(GOV, json={"cleanup_management_sources": ["198.51.100.7"]})
|
||||
assert ok.status_code == 200, ok.text
|
||||
assert [a.actor for a in await _audit(kind="settings", action="cleanup_settings")][-1] == "admin (API: cl-admin)"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_others_only_see_whether_management_sources_are_set(client, who):
|
||||
await client.put(GOV, json={"cleanup_management_sources": ["203.0.113.10"]})
|
||||
assert (await client.get(GOV)).json()["cleanup_management_sources"] == ["203.0.113.10"]
|
||||
for name in ("operator", "read_only"):
|
||||
seen = (await who[name].get(GOV)).json()
|
||||
assert seen["cleanup_management_sources"] == [] and seen["cleanup_management_sources_configured"] is True, name
|
||||
assert seen["cleanup_max_rules"] == 10 and seen["operators_may_deploy"] is True
|
||||
|
||||
|
||||
# ── Locks in both directions ──────────────────────────────────────────────
|
||||
|
||||
async def _central_rule(client, env, **extra):
|
||||
resp = await client.post("/api/rules", json={
|
||||
"description": "managed ssh", "interface": "lan", "direction": "in", "protocol": "tcp", "action": "pass",
|
||||
"source_net": "any", "destination_net": "lan", "destination_port": "22",
|
||||
"target_host_ids": [str(env.host_id)], **extra})
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
async def _deployed_row(rule_id, host_id, firewall_uuid, status="success"):
|
||||
async with SessionLocal() as db:
|
||||
dep = RuleDeployment(rule_id=uuid_lib.UUID(rule_id), host_id=host_id, status=status, opnsense_rule_uuid=firewall_uuid)
|
||||
db.add(dep)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_open_cleanup_blocks_other_writes_to_the_firewall(client, env):
|
||||
rule = await _central_rule(client, env)
|
||||
alias = await client.post("/api/aliases", json={"name": "cl_alias", "type": "host", "content": "203.0.113.5",
|
||||
"target_host_ids": [str(env.host_id)]})
|
||||
assert alias.status_code == 201, alias.text
|
||||
hid = env.host_id
|
||||
await _open(client, env)
|
||||
writes = list(env.fw.writes())
|
||||
blocked = [
|
||||
("POST", f"/api/rules/{rule['id']}/deploy", None),
|
||||
("POST", f"/api/aliases/{alias.json()['id']}/deploy", None),
|
||||
("POST", f"/api/hosts/{hid}/snapshots", {"name": "x"}),
|
||||
("POST", f"/api/hosts/{hid}/services/dhcpd/restart", None),
|
||||
("POST", f"/api/hosts/{hid}/maintenance/check-updates", None),
|
||||
("POST", f"/api/hosts/{hid}/network/changes", {"kind": "vlan", "action": "add", "fields": {}, "preview_token": "0" * 64}),
|
||||
("DELETE", f"/api/hosts/{hid}/snapshots/abc", None),
|
||||
]
|
||||
for method, path, body in blocked:
|
||||
r = await client.request(method, path, **({"json": body} if body is not None else {}))
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", (path, r.text)
|
||||
assert env.fw.writes() == writes # nothing reached the firewall
|
||||
# the cleanup's own follow-up steps are not blocked by itself, and a second cleanup is refused with its own code
|
||||
second = await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_BLOCK], "acknowledge": [R_BLOCK]})
|
||||
assert second.status_code == 409 and second.json()["detail"]["code"] == "BEREINIGUNG_OFFENE_OPERATION"
|
||||
assert second.json()["detail"]["params"]["reason"] == "cleanup"
|
||||
op = (await client.get(_base(env))).json()[0]
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/revert")).status_code == 200
|
||||
again = await client.post(f"/api/hosts/{hid}/maintenance/check-updates")
|
||||
assert again.status_code != 409 or again.json()["detail"]["code"] != "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_block_covers_the_cluster_and_the_maintenance_inspection(cluster, client):
|
||||
from app.services.host_maintenance_service import _shared_change_blockers, ensure_no_maintenance, ensure_no_open_cleanup, peers
|
||||
pv = await client.post(_base(cluster.a) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})
|
||||
assert pv.status_code == 200, pv.text
|
||||
assert (await client.post(f"{_base(cluster.a)}/{pv.json()['preview_id']}/deploy", json={"confirm_name": "fw-a"})).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
a, b = await db.get(OpnsenseHost, cluster.a), await db.get(OpnsenseHost, cluster.b)
|
||||
for host in (a, b): # the partner node is just as busy
|
||||
for call in (ensure_no_maintenance, ensure_no_open_cleanup):
|
||||
with pytest.raises(Exception) as info:
|
||||
await call(db, [host])
|
||||
assert info.value.detail["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
blockers = await _shared_change_blockers(db, await peers(db, b))
|
||||
assert any("Bereinigung" in text for text in blockers)
|
||||
await ensure_no_maintenance(db, [a], cleanup=False) # the cleanup service's own check does not trip over itself
|
||||
# deleting or re-addressing either node is refused while it is open
|
||||
for host_id in (cluster.a, cluster.b):
|
||||
r = await client.delete(f"/api/hosts/{host_id}")
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", host_id
|
||||
op = (await client.get(_base(cluster.a))).json()[0]
|
||||
assert (await client.post(f"{_base(cluster.a)}/{op['id']}/revert")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
assert await _shared_change_blockers(db, await peers(db, await db.get(OpnsenseHost, cluster.b))) == []
|
||||
await ensure_no_maintenance(db, [await db.get(OpnsenseHost, cluster.b)])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_host_delete_and_readdressing_are_refused_while_a_cleanup_is_open(client, env):
|
||||
hid = env.host_id
|
||||
await _open(client, env)
|
||||
gone = await client.delete(f"/api/hosts/{hid}")
|
||||
assert gone.status_code == 409 and gone.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
for body in ({"base_url": "https://203.0.113.99"}, {"enabled": False}, {"cluster_name": "ha-x"}, {"clear_cluster_name": True}):
|
||||
r = await client.patch(f"/api/hosts/{hid}", json=body)
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", body
|
||||
assert (await client.patch(f"/api/hosts/{hid}", json={"name": "fw-a2"})).status_code == 200 # a rename is harmless
|
||||
async with SessionLocal() as db:
|
||||
host = await db.get(OpnsenseHost, hid)
|
||||
assert host.base_url == FW1 and host.enabled and host.name == "fw-a2"
|
||||
op = (await client.get(_base(env))).json()[0]
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/revert")).status_code == 200
|
||||
assert (await client.delete(f"/api/hosts/{hid}")).status_code == 204 # settled: the host may go (its runs go with it)
|
||||
async with SessionLocal() as db:
|
||||
assert (await db.scalars(select(RuleCleanupOperation))).all() == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_confirmed_cleanup_blocks_nothing(client, env):
|
||||
op = await _open(client, env)
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/confirm")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
assert not await svc.has_open_cleanup(db, [env.host_id])
|
||||
r = await client.post(f"/api/hosts/{env.host_id}/maintenance/check-updates")
|
||||
assert r.status_code != 409 or r.json()["detail"]["code"] != "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_deletion_and_removal_of_targets_wait_for_the_cleanup(client, env):
|
||||
rule = await _central_rule(client, env, target_host_ids=[])
|
||||
await _deployed_row(rule["id"], env.host_id, R_TAIL)
|
||||
token = (await client.get(f"/api/rules/{rule['id']}/preview")).json()["token"]
|
||||
await _open(client, env)
|
||||
env.fw.calls.clear()
|
||||
deleted = await client.delete(f"/api/rules/{rule['id']}")
|
||||
assert deleted.status_code == 409 and deleted.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY"
|
||||
removed = await client.post(f"/api/rules/{rule['id']}/deployments/{env.host_id}/remove", json={"preview_token": token})
|
||||
assert removed.status_code == 409 and removed.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", removed.text
|
||||
assert env.fw.writes() == [] and env.fw.get(R_TAIL) is not None
|
||||
op = (await client.get(_base(env))).json()[0]
|
||||
await client.post(f"{_base(env)}/{op['id']}/revert")
|
||||
assert (await client.delete(f"/api/rules/{rule['id']}")).status_code == 204 # the same call works afterwards
|
||||
assert env.fw.get(R_TAIL) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hierarchy_rule_deletion_and_reordering_wait_for_the_cleanup(client, env):
|
||||
from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeployment
|
||||
from tests.test_policy_hierarchy import _create_group, _hrule_payload
|
||||
group = await _create_group(client, "Cleanup-Root")
|
||||
assert (await client.patch(f"/api/hosts/{env.host_id}", json={"policy_hierarchy_group_id": group["id"]})).status_code == 200
|
||||
rule = (await client.post("/api/policy-hierarchy/rules", json=_hrule_payload(group["id"], description="guard"))).json()
|
||||
async with SessionLocal() as db:
|
||||
db.add(PolicyHierarchyRuleDeployment(policy_hierarchy_id=uuid_lib.UUID(rule["id"]), host_id=env.host_id, status="success",
|
||||
opnsense_policy_hierarchy_uuid=R_TAIL))
|
||||
await db.commit()
|
||||
await _open(client, env)
|
||||
env.fw.calls.clear()
|
||||
deleted = await client.delete(f"/api/policy-hierarchy/rules/{rule['id']}")
|
||||
assert deleted.status_code == 409 and deleted.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", deleted.text
|
||||
reorder = await client.post(f"/api/policy-hierarchy/groups/{group['id']}/reassert-ordering")
|
||||
assert reorder.status_code == 409 and reorder.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", reorder.text
|
||||
assert env.fw.writes() == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_removing_a_former_hierarchy_target_waits_for_the_cleanup(client, env):
|
||||
from app.models.policy_hierarchy_rule_deployment import PolicyHierarchyRuleDeployment
|
||||
from tests.test_policy_hierarchy import _create_group, _hrule_payload
|
||||
group = await _create_group(client, "Cleanup-Former")
|
||||
rule = (await client.post("/api/policy-hierarchy/rules", json=_hrule_payload(group["id"], description="former"))).json()
|
||||
async with SessionLocal() as db: # deployed on the host, which is not in the group (any more): "retained"
|
||||
db.add(PolicyHierarchyRuleDeployment(policy_hierarchy_id=uuid_lib.UUID(rule["id"]), host_id=env.host_id, status="success",
|
||||
opnsense_policy_hierarchy_uuid=R_TAIL))
|
||||
await db.commit()
|
||||
plan = (await client.get(f"/api/policy-hierarchy/rules/{rule['id']}/preview")).json()
|
||||
assert [h["can_remove"] for h in plan["hosts"]] == [True]
|
||||
await _open(client, env)
|
||||
env.fw.calls.clear()
|
||||
r = await client.post(f"/api/policy-hierarchy/rules/{rule['id']}/deployments/{env.host_id}/remove", json={"preview_token": plan["token"]})
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
|
||||
assert env.fw.writes() == [] and env.fw.get(R_TAIL) is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopting_a_rule_waits_for_the_cleanup(client, env):
|
||||
await _open(client, env)
|
||||
r = await client.post("/api/rules/adopt", json={"host_id": str(env.host_id), "rule_uuid": R_BASE})
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_other_open_work_blocks_a_new_cleanup_over_the_api(client, env):
|
||||
url = _base(env) + "/preview"
|
||||
body = {"kind": "disable", "uuids": [R_UNUSED]}
|
||||
async with SessionLocal() as db:
|
||||
db.add(HostOperation(host_id=env.host_id, host_name="fw-a", actor="admin", action="reboot", status="submitted",
|
||||
expires_at=_now(), fingerprint="x", snapshot={}))
|
||||
await db.commit()
|
||||
r = await client.post(url, json=body)
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_HOST_MAINTENANCE_BLOCKS_DEPLOY"
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(HostOperation).values(status="done"))
|
||||
await db.commit()
|
||||
rule = await _central_rule(client, env)
|
||||
await _deployed_row(rule["id"], env.host_id, R_TAIL, status="pending")
|
||||
r = await client.post(url, json=body)
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "BEREINIGUNG_OFFENE_OPERATION" and r.json()["detail"]["params"]["reason"] == "deployment"
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleDeployment).values(status="success"))
|
||||
await db.commit()
|
||||
assert (await client.post(url, json=body)).status_code == 200
|
||||
assert env.fw.writes() == []
|
||||
|
||||
|
||||
# ── Notification ──────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_reverts_and_unclear_runs_are_reported_with_stable_keys(client, env):
|
||||
op = await _open(client, env)
|
||||
async with SessionLocal() as db:
|
||||
assert not [k for k in await notification_checks.check_revert_failed(db) if k.startswith("rule_cleanup")]
|
||||
# a run that ended half-way (decision needed): one problem for the run
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(update(RuleCleanupOperation).where(RuleCleanupOperation.id == uuid_lib.UUID(op["id"])).values(status="partial"))
|
||||
await db.commit()
|
||||
problems = await notification_checks.check_revert_failed(db)
|
||||
key = f"rule_cleanup_op:{op['id']}"
|
||||
assert set(problems) == {key} and problems[key]["deployment_type"] == "rule_cleanup"
|
||||
assert problems[key]["host_name"] == "fw-a" and "teilweise" in problems[key]["hinweis"]
|
||||
# the automatic revert failed: one problem per rule that is still changed (the key is the item, not the run)
|
||||
env.fw.fail[("toggle", R_DUP)] = httpx.Response(200, json={"result": "failed"})
|
||||
await client.post(f"{_base(env)}/{op['id']}/revert")
|
||||
items = await _items(op["id"])
|
||||
assert (await _op(op["id"])).status == "revert_failed"
|
||||
async with SessionLocal() as db:
|
||||
first = await notification_checks.check_revert_failed(db)
|
||||
second = await notification_checks.check_revert_failed(db)
|
||||
stuck = f"rule_cleanup:{items[R_DUP].id}"
|
||||
assert set(first) == {stuck} and set(second) == set(first) # stable: the same problem is not reported twice
|
||||
assert first[stuck]["entity_id"] == R_DUP and first[stuck]["host_name"] == "fw-a" and "Erneut zurücknehmen" in first[stuck]["hinweis"]
|
||||
assert first[stuck]["operation_status"] == "revert_failed" and first[stuck]["item_state"] == "revert_failed"
|
||||
# solved: gone
|
||||
del env.fw.fail[("toggle", R_DUP)]
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/retry")).json()["status"] == "reverted"
|
||||
async with SessionLocal() as db:
|
||||
assert not [k for k in await notification_checks.check_revert_failed(db) if k.startswith("rule_cleanup")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_unclear_delete_is_reported_until_it_is_read_again(client, env):
|
||||
await _confirmed_disable(env.host_id, [R_UNUSED])
|
||||
pv = (await client.post(_base(env) + "/preview", json={"kind": "delete", "uuids": [R_UNUSED]})).json()
|
||||
env.fw.fail[("delete", R_UNUSED)] = httpx.ReadTimeout
|
||||
|
||||
def _then_unreadable(kind, rule_uuid): # the firewall stops answering once delRule went out
|
||||
if kind == "delete":
|
||||
env.fw.fail[("get", R_UNUSED)] = httpx.ReadTimeout
|
||||
env.fw.on_call = _then_unreadable
|
||||
op = (await client.post(f"{_base(env)}/{pv['preview_id']}/deploy", json={"confirm_name": "fw-a"})).json()
|
||||
assert op["status"] == "failed" and op["items"][0]["state"] == "unknown" and op["can_retry"] is True
|
||||
async with SessionLocal() as db:
|
||||
problems = await notification_checks.check_revert_failed(db)
|
||||
key = f"rule_cleanup:{op['items'][0]['id']}"
|
||||
assert set(problems) == {key} and problems[key]["operation"] == "delete" and "unklar" in problems[key]["hinweis"]
|
||||
env.fw.on_call = None
|
||||
del env.fw.fail[("get", R_UNUSED)]
|
||||
del env.fw.fail[("delete", R_UNUSED)]
|
||||
assert (await client.post(f"{_base(env)}/{op['id']}/retry")).status_code == 200
|
||||
async with SessionLocal() as db:
|
||||
assert not [k for k in await notification_checks.check_revert_failed(db) if k.startswith("rule_cleanup")]
|
||||
|
||||
|
||||
# ── MCP and the schema ────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mcp_stays_read_only_for_the_cleanup(client, env, who):
|
||||
from app.routers.mcp import TOOLS
|
||||
names = {t["name"] for t in TOOLS}
|
||||
assert not [n for n in names if "cleanup" in n and n != "rule_cleanup_candidates"]
|
||||
assert all(t.get("path", "").startswith("/api/") for t in TOOLS if "path" in t)
|
||||
op = await _open(client, env)
|
||||
|
||||
def call(path):
|
||||
return {"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "api_get", "arguments": {"path": path}}}
|
||||
|
||||
async with _bearer((await _make(client, "admin", name="mcp-admin"))["token"]) as api:
|
||||
for tail in ("/preview", f"/{op['id']}/confirm", f"/{op['id']}/revert", f"/{op['id']}/deploy", f"/{op['id']}/retry"):
|
||||
result = (await api.post("/api/mcp", json=call(f"/api/hosts/{env.host_id}/rule-cleanup{tail}"))).json()["result"]
|
||||
assert result["isError"] is True, tail # a GET on a POST route: refused, nothing runs
|
||||
listing = (await api.post("/api/mcp", json=call(f"/api/hosts/{env.host_id}/rule-cleanup"))).json()["result"]
|
||||
assert listing["isError"] is False and op["id"] in listing["content"][0]["text"]
|
||||
assert (await _op(op["id"])).status == "awaiting_confirm"
|
||||
# a plain POST of a tool name that would write does not exist
|
||||
async with _bearer((await _make(client, "admin", name="mcp-admin2"))["token"]) as api:
|
||||
r = (await api.post("/api/mcp", json={"jsonrpc": "2.0", "id": 2, "method": "tools/call",
|
||||
"params": {"name": "rule_cleanup_deploy", "arguments": {}}})).json()
|
||||
assert "error" in r or r["result"]["isError"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openapi_knows_the_routes_and_their_models(client):
|
||||
schema = (await client.get("/api/openapi.json")).json()
|
||||
base = "/api/hosts/{host_id}/rule-cleanup"
|
||||
paths = schema["paths"]
|
||||
assert set(paths[base]) == {"get"} and set(paths[base + "/preview"]) == {"post"} and set(paths[base + "/{op_id}"]) == {"get"}
|
||||
for tail in ("cancel", "deploy", "confirm", "revert", "retry"):
|
||||
assert set(paths[f"{base}/{{op_id}}/{tail}"]) == {"post"}, tail
|
||||
assert set(paths[base + "/items/{item_id}/restore"]) == {"post"}
|
||||
assert {"CleanupOperationRead", "CleanupItemRead", "CleanupPreviewRead", "PreviewRequest", "ConfirmName"} <= set(schema["components"]["schemas"])
|
||||
ops = schema["components"]["schemas"]["CleanupOperationRead"]["properties"]
|
||||
assert {"can_confirm", "can_revert", "can_retry", "plan", "warnings", "items", "revert_deadline", "message"} <= set(ops)
|
||||
assert {"restorable", "state", "host_name", "has_snapshot"} <= set(schema["components"]["schemas"]["CleanupItemRead"]["properties"])
|
||||
|
||||
|
||||
def test_every_new_error_code_has_a_text_in_both_languages():
|
||||
import json
|
||||
import pathlib
|
||||
from app.errors import ErrorCode
|
||||
root = pathlib.Path(__file__).resolve().parents[2] / "frontend" / "lib" / "i18n"
|
||||
if not root.is_dir():
|
||||
pytest.skip("the frontend is not part of this image (scripts/check_i18n.py checks the same from the repo root)")
|
||||
for lang in ("de", "en"):
|
||||
errors = json.loads((root / f"{lang}.json").read_text(encoding="utf-8"))["errors"]
|
||||
for code in ErrorCode:
|
||||
if code.name.startswith("BEREINIGUNG_") or code.name == "OPEN_CLEANUP_BLOCKS_DEPLOY":
|
||||
assert errors.get(code.value), (lang, code.value)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_refused_operator_attempt_is_audited_and_writes_nothing(client, env, who):
|
||||
op = await _open(client, env)
|
||||
r = await who["operator"].post(f"{_base(env)}/{op['id']}/retry")
|
||||
assert r.status_code == 403
|
||||
denied = await _audit(kind="host", action="retry", status="denied")
|
||||
assert [a.actor for a in denied] == ["cl-operator"] and denied[0].host_id == str(env.host_id)
|
||||
assert (await _op(op["id"])).status == "awaiting_confirm"
|
||||
|
||||
|
||||
def _leaks(value) -> bool:
|
||||
"""Anything that names a management source or a firewall's base URL."""
|
||||
import json as _json
|
||||
text = _json.dumps(value)
|
||||
return MGMT_SOURCE in text or "203.0.113.57" in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sources_and_firewall_addresses_are_for_admins_only(client, env, who): # review finding 8
|
||||
mine = await client.post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_UNUSED]})
|
||||
assert mine.json()["plan"]["sources"] == [f"{MGMT_SOURCE}/32"] and mine.json()["plan"]["nodes"][0]["base_url"]
|
||||
theirs = await who["operator"].post(_base(env) + "/preview", json={"kind": "disable", "uuids": [R_DUP]})
|
||||
assert theirs.status_code == 200, theirs.text
|
||||
body = theirs.json()
|
||||
assert "sources" not in body["plan"] and body["plan"]["sources_configured"] is True and not _leaks(body)
|
||||
assert body["plan"]["nodes"][0]["items"][0]["protection"]["result"] == "false"
|
||||
for name in ("operator", "read_only", "tok_read", "tok_deploy", "tok_ro_read"):
|
||||
listed = await who[name].get(_base(env))
|
||||
assert listed.status_code == 200 and listed.json() and not _leaks(listed.json()), name
|
||||
one = await who[name].get(f"{_base(env)}/{mine.json()['preview_id']}")
|
||||
assert one.status_code == 200 and not _leaks(one.json()), name
|
||||
audit = await who[name].get("/api/audit", params={"kind": "rule_cleanup"})
|
||||
assert audit.status_code == 200 and audit.json()["items"] and not _leaks(audit.json()), name
|
||||
assert {i["details"]["admin_only"] for i in audit.json()["items"]} == {"redacted"}, name
|
||||
for name in ("admin", "tok_admin"):
|
||||
assert _leaks((await who[name].get(_base(env))).json()), name
|
||||
audit = (await who[name].get("/api/audit", params={"kind": "rule_cleanup"})).json()
|
||||
assert all(isinstance(i["details"]["admin_only"], dict) for i in audit["items"]), name
|
||||
|
||||
|
||||
# Review finding 9: undeploying a central object and removing a former target write to the firewall too.
|
||||
_UNDEPLOY_SITES = [
|
||||
("app.services.alias_deploy_service", "undeploy_alias"), ("app.services.openvpn_deploy_service", "undeploy_instance"),
|
||||
("app.services.wireguard_deploy_service", "undeploy_tunnel"), ("app.services.port_forward_deploy_service", "undeploy_port_forward"),
|
||||
("app.services.outbound_nat_deploy_service", "undeploy_outbound_nat"), ("app.services.one_to_one_deploy_service", "undeploy_one_to_one"),
|
||||
("app.services.ipsec_deploy_service", "undeploy_tunnel"),
|
||||
] + [(f"app.routers.{r}", "remove_previous_target") for r in
|
||||
("aliases", "port_forward", "one_to_one", "outbound_nat", "wireguard", "openvpn", "ipsec")]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("module,name", _UNDEPLOY_SITES)
|
||||
def test_every_undeploy_and_target_removal_checks_for_an_open_cleanup_first(module, name):
|
||||
import importlib
|
||||
import inspect
|
||||
source = inspect.getsource(getattr(importlib.import_module(module), name))
|
||||
lock = min(i for i in (source.find("ensure_no_open_cleanup_on("), source.find("ensure_no_open_cleanup(")) if i >= 0)
|
||||
first_write = source.find("opnsense_client.")
|
||||
first_write = min(i for i in (first_write, source.find("client.del_"), source.find("_client.del")) if i >= 0)
|
||||
assert 0 <= lock < first_write, (module, name)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_delete_and_target_removal_wait_for_the_cleanup(client, env):
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
deployed = await client.post("/api/aliases", json={"name": "cl_gone", "type": "host", "content": "203.0.113.5",
|
||||
"target_host_ids": [str(env.host_id)]})
|
||||
former = await client.post("/api/aliases", json={"name": "cl_former", "type": "host", "content": "203.0.113.6",
|
||||
"target_host_ids": []})
|
||||
assert deployed.status_code == 201 and former.status_code == 201, (deployed.text, former.text)
|
||||
async with SessionLocal() as db:
|
||||
for alias in (deployed.json(), former.json()):
|
||||
db.add(AliasDeployment(alias_id=uuid_lib.UUID(alias["id"]), host_id=env.host_id, status="success",
|
||||
opnsense_alias_uuid="aaaaaaaa-1111-4111-8111-111111111111", last_payload={"name": alias["name"]}))
|
||||
await db.commit()
|
||||
preview = (await client.get(f"/api/aliases/{former.json()['id']}/preview")).json()
|
||||
assert next(h for h in preview["hosts"] if h["host_id"] == str(env.host_id))["can_remove"], preview
|
||||
await _open(client, env)
|
||||
writes = list(env.fw.writes())
|
||||
r = await client.delete(f"/api/aliases/{deployed.json()['id']}")
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
|
||||
preview = (await client.get(f"/api/aliases/{former.json()['id']}/preview")).json()
|
||||
r = await client.post(f"/api/aliases/{former.json()['id']}/deployments/{env.host_id}/remove", json={"preview_token": preview["token"]})
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "OPEN_CLEANUP_BLOCKS_DEPLOY", r.text
|
||||
assert env.fw.writes() == writes
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Pure parts of the rule cleanup (1.31): engine.match_rule, the management-access protection, the infrastructure
|
||||
exclusion, the confidence caps and the cluster usage merge. No database, no firewall."""
|
||||
import ipaddress
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from app.services import rule_hit_service
|
||||
from app.services.rule_analysis import cleanup, protection
|
||||
from app.services.rule_analysis.context import Context, IfInfo
|
||||
from app.services.rule_analysis.engine import Query, match_rule
|
||||
from app.services.rule_analysis.model import Endpoint, NormRule
|
||||
from app.services.rule_cleanup_service import selectable_level
|
||||
|
||||
NOW = datetime(2026, 10, 22, 12, 0, tzinfo=timezone.utc)
|
||||
ip = ipaddress.ip_address
|
||||
net = ipaddress.ip_network
|
||||
|
||||
|
||||
def ctx(**kw) -> Context:
|
||||
interfaces = {
|
||||
"wan": IfInfo("wan", "vtnet0", "WAN", nets=[net("203.0.113.0/24")], addrs=[ip("203.0.113.57")], has_gateway=True),
|
||||
"lan": IfInfo("lan", "vtnet1", "LAN", nets=[net("10.20.30.0/24")], addrs=[ip("10.20.30.2")], vips=[ip("10.20.30.1")]),
|
||||
}
|
||||
return Context(rows=[], pf=None, interfaces=interfaces, groups={}, aliases=kw.get("aliases", {}), port_forwards=[])
|
||||
|
||||
|
||||
def nrule(*, src="", dst="", dport="", protos=frozenset({"6"}), interfaces=frozenset({"wan"}), action="pass", negated=False,
|
||||
tagged="", uuid="u1") -> NormRule:
|
||||
items = [i for i in src.split(",") if i]
|
||||
return NormRule(uuid=uuid, label=uuid, description=uuid, order=(400000, 1), order_known=True, section="interface",
|
||||
enabled=True, action=action, quick=True, interfaces=set(interfaces), if_not=False, direction="in",
|
||||
af={4}, protos=None if protos is None else set(protos), src=Endpoint(raw=src, negated=negated, items=items),
|
||||
dst=Endpoint(raw=dst, items=[i for i in dst.split(",") if i]), sport="", dport=dport, tagged=tagged)
|
||||
|
||||
|
||||
SOURCES = [net("203.0.113.10/32")]
|
||||
|
||||
|
||||
def check(rule, c=None, sources=SOURCES):
|
||||
c = c or ctx()
|
||||
targets, ports = protection.management_targets("https://203.0.113.57:4443", c)
|
||||
return protection.management_check(rule, c, sources, targets, ports)
|
||||
|
||||
|
||||
def test_targets_are_the_api_address_and_port_plus_every_firewall_address():
|
||||
targets, ports = protection.management_targets("https://203.0.113.57:4443", ctx())
|
||||
assert targets[0] == ip("203.0.113.57") and ip("10.20.30.1") in targets and ip("127.0.0.1") not in targets
|
||||
assert ports == [4443, 443, 80, 22]
|
||||
|
||||
|
||||
def test_a_rule_for_the_management_path_is_protected():
|
||||
assert check(nrule(src="203.0.113.0/24", dport="443"))["result"] == "true"
|
||||
assert check(nrule(src="", dport="4443"))["result"] == "true" # any source, API port
|
||||
assert check(nrule(src="203.0.113.10", dport="", protos=None))["result"] == "true"
|
||||
|
||||
|
||||
def test_only_a_proven_non_match_is_removable():
|
||||
assert check(nrule(src="10.20.30.0/24", dport="443"))["result"] == "false" # other source
|
||||
assert check(nrule(src="203.0.113.0/24", dport="8081"))["result"] == "false" # other port
|
||||
assert check(nrule(src="203.0.113.0/24", dport="443", protos={"17"}))["result"] == "false" # udp
|
||||
|
||||
|
||||
def test_unknown_is_protected(): # case 18
|
||||
unknown = check(nrule(src="tbl", dport="443"), ctx(aliases={"tbl": {"name": "tbl", "type": "urltable"}}))
|
||||
assert unknown["result"] == "unknown" and unknown["match"]["port"] in (4443, 443)
|
||||
tagged = check(nrule(src="", dport="443", tagged="mark"))
|
||||
assert tagged["result"] == "unknown" and "TAGGED" in tagged["reasons"]
|
||||
q = Query(src=ip("203.0.113.10"), dst=ip("203.0.113.57"), proto="6", proto_label="tcp", dport=443)
|
||||
assert match_rule(nrule(interfaces={"nosuchgroup"}), q, ctx(), iface="wan").value is None
|
||||
|
||||
|
||||
def test_network_sources_are_judged_for_every_address():
|
||||
small = [net("203.0.113.0/28")]
|
||||
assert check(nrule(src="203.0.113.14", dport="443"), sources=small)["result"] == "true" # one address inside
|
||||
assert check(nrule(src="203.0.113.20", dport="443"), sources=small)["result"] == "false"
|
||||
big = [net("10.0.0.0/8")]
|
||||
assert check(nrule(src="10.20.30.0/24", dport="443"), sources=big)["result"] == "true" # ranges overlap
|
||||
assert check(nrule(src="192.0.2.0/24", dport="443"), sources=big)["result"] == "false"
|
||||
assert check(nrule(src="lan", dport="443"), sources=big)["result"] == "true" # interface network
|
||||
assert check(nrule(src="10.0.0.0/8", dport="443", negated=True), sources=big)["result"] == "false" # !whole net
|
||||
assert check(nrule(src="tbl", dport="443"), ctx(aliases={"tbl": {"type": "urltable"}}), sources=big)["result"] == "unknown"
|
||||
|
||||
|
||||
def test_no_query_possible_is_protected():
|
||||
got = check(nrule(src="", dport="443"), sources=[net("2001:db8::/64")]) # no IPv6 address on the firewall
|
||||
assert got["result"] == "unknown" and got["reasons"] == ["NO_MANAGEMENT_QUERY"]
|
||||
|
||||
|
||||
def test_carp_and_pfsync_rules_are_infrastructure():
|
||||
c = ctx()
|
||||
assert protection.infrastructure_reason(nrule(protos={"112"}), c, hasync=None, clustered=False) == "infrastructure"
|
||||
assert protection.infrastructure_reason(nrule(protos={"240"}), c, hasync=None, clustered=False) == "infrastructure"
|
||||
lan_any = nrule(src="10.20.30.0/24", protos=None, interfaces={"lan"})
|
||||
assert protection.infrastructure_reason(lan_any, c, hasync=None, clustered=False) == "infrastructure" # CARP adverts
|
||||
lan_tcp = nrule(src="10.20.30.0/24", dport="8081", interfaces={"lan"})
|
||||
assert protection.infrastructure_reason(lan_tcp, c, hasync=None, clustered=True) is None
|
||||
wan_any = nrule(src="", protos=None, interfaces={"wan"})
|
||||
off = {"hasync": {"pfsyncinterface": {"": {"selected": 1}, "wan": {"selected": 0}}, "pfsyncpeerip": ""}}
|
||||
on = {"hasync": {"pfsyncinterface": {"": {"selected": 0}, "wan": {"selected": 1}}, "pfsyncpeerip": "203.0.113.58"}}
|
||||
assert protection.infrastructure_reason(wan_any, c, hasync=off, clustered=True) is None
|
||||
assert protection.infrastructure_reason(wan_any, c, hasync=on, clustered=True) == "infrastructure"
|
||||
# HA settings not readable on a cluster node: every interface counts as a possible pfsync interface
|
||||
assert protection.infrastructure_reason(wan_any, c, hasync=None, clustered=True) == "infrastructure"
|
||||
|
||||
|
||||
def test_confidence_caps():
|
||||
rule = nrule()
|
||||
unused = {"categories": [{"kind": "unused", "confidence": "sure"}]}
|
||||
assert selectable_level(unused, rule, unused_days=30, external=False)[0] == "sure"
|
||||
assert selectable_level(unused, rule, unused_days=29, external=False)[0] == "probable"
|
||||
assert selectable_level(unused, nrule(action="block"), unused_days=90, external=False)[0] == "probable"
|
||||
assert selectable_level(unused, nrule(action="reject"), unused_days=90, external=False)[0] == "probable"
|
||||
assert selectable_level(unused, rule, unused_days=90, external=True)[0] == "probable"
|
||||
both = {"categories": [{"kind": "unused", "confidence": "unknown"}, {"kind": "duplicate", "confidence": "sure"}]}
|
||||
assert selectable_level(both, nrule(action="block"), unused_days=7, external=False)[0] == "sure" # best category
|
||||
assert selectable_level({"categories": [{"kind": "shadowed", "confidence": "unknown"}]}, rule, unused_days=30, external=False)[0] == "unknown"
|
||||
assert selectable_level({"categories": [{"kind": "disabled_long", "confidence": "sure"}]}, rule, unused_days=30, external=False)[0] is None
|
||||
|
||||
|
||||
class Hit:
|
||||
def __init__(self, uuid, first_days, hit_days=None, sampled_minutes=10, enabled=True):
|
||||
self.rule_uuid, self.enabled = uuid, enabled
|
||||
self.first_seen_at = NOW - timedelta(days=first_days)
|
||||
self.last_hit_at = None if hit_days is None else NOW - timedelta(days=hit_days)
|
||||
self.last_sampled_at = NOW - timedelta(minutes=sampled_minutes)
|
||||
|
||||
|
||||
def test_cluster_usage_merge(): # case 16
|
||||
master = {"r": Hit("r", 100, hit_days=1)}
|
||||
backup = {"r": Hit("r", 100)}
|
||||
merged = rule_hit_service.merge_cluster_hits(backup, [master])
|
||||
assert rule_hit_service.classify(merged["r"], NOW, 30) == "used"
|
||||
both_idle = rule_hit_service.merge_cluster_hits({"r": Hit("r", 100)}, [{"r": Hit("r", 20)}])
|
||||
assert rule_hit_service.classify(both_idle["r"], NOW, 30) == "collecting" # window = latest start
|
||||
stale = rule_hit_service.merge_cluster_hits({"r": Hit("r", 100)}, [{"r": Hit("r", 100, sampled_minutes=60 * 48)}])
|
||||
assert stale["r"].last_sampled_at == NOW - timedelta(hours=48) # as fresh as the oldest node
|
||||
missing = rule_hit_service.merge_cluster_hits({"r": Hit("r", 100)}, [{}])
|
||||
assert missing["r"].peer_missing and rule_hit_service.classify(missing["r"], NOW, 30) == "unused"
|
||||
rule = nrule()
|
||||
rule.loaded = True
|
||||
assert cleanup.unused_confidence(rule, missing["r"], NOW) == "probable"
|
||||
assert cleanup.unused_confidence(rule, both_idle["r"], NOW) == "sure"
|
||||
|
||||
|
||||
def _live_rule(**kw):
|
||||
"""A rule normalized inside the live fixture (tests/fixtures/rule_analysis_fw.json: built-in groups wireguard,
|
||||
openvpn, enc0 with members "" and unassigned devices without identifier)."""
|
||||
from app.services.rule_analysis.normalize import normalize
|
||||
from tests.test_rule_analysis import build, mvc
|
||||
row = mvc("bbbbbbbb-0000-4000-8000-000000000001", seq=900, proto="tcp", **kw)
|
||||
c = build([row])
|
||||
rules, _ = normalize(c)
|
||||
return next(r for r in rules if r.uuid == row["uuid"]), c
|
||||
|
||||
|
||||
def test_rules_on_builtin_groups_without_known_members_are_protected(): # review finding 1
|
||||
for group in ("wireguard", "openvpn", "enc0"):
|
||||
rule, c = _live_rule(interface=group, src="203.0.113.10", dst="(self)", dport="443")
|
||||
got = check(rule, c)
|
||||
assert got["result"] == "unknown" and "IFACE_MEMBERS_UNKNOWN" in got["reasons"], (group, got)
|
||||
assert got["match"]["interface"] == protection.UNASSIGNED
|
||||
# the virtual ingress does not make every rule protected
|
||||
rule, c = _live_rule(interface="wireguard", src="203.0.113.10", dst="(self)", dport="8081")
|
||||
assert check(rule, c)["result"] == "false"
|
||||
rule, c = _live_rule(interface="lan", src="10.20.30.0/24", dst="(self)", dport="443")
|
||||
assert check(rule, c)["result"] == "false"
|
||||
# floating "any interface" and "not lan" also match a packet on wg0
|
||||
rule, c = _live_rule(interface="", src="203.0.113.10", dst="(self)", dport="22")
|
||||
assert check(rule, c)["result"] == "true"
|
||||
rule, c = _live_rule(interface="lan", interfacenot="1", src="203.0.113.10", dst="(self)", dport="22")
|
||||
assert check(rule, c)["result"] == "true"
|
||||
|
||||
|
||||
def test_unassigned_interface_outcome():
|
||||
c = ctx()
|
||||
c.groups = {"mygroup": {"lan"}, "wireguard": set()}
|
||||
assert protection.unassigned_interface(nrule(interfaces={"lan"}), c).value is False
|
||||
assert protection.unassigned_interface(nrule(interfaces={"mygroup"}), c).value is False
|
||||
assert protection.unassigned_interface(nrule(interfaces={"wireguard"}), c).value is None
|
||||
assert protection.unassigned_interface(nrule(interfaces={"nosuch"}), c).value is None
|
||||
assert protection.unassigned_interface(nrule(interfaces=set()), c).value is True
|
||||
|
||||
|
||||
def test_redacted_plan_names_no_source_port_or_firewall_address(): # review finding 8
|
||||
from app.services.rule_cleanup_service import redact
|
||||
plan = {"kind": "enable", "sources": ["203.0.113.10/32"], "ports": [2222], "nodes": [
|
||||
{"host_id": "h", "base_url": "https://203.0.113.57", "items": [
|
||||
{"uuid": "u", "protection": {"result": "true", "queries": 3, "reasons": [],
|
||||
"match": {"source": "203.0.113.10/32", "destination": "203.0.113.57"}}}]}]}
|
||||
out = redact({"plan": plan, "operation": {"plan": plan}})
|
||||
for p in (out["plan"], out["operation"]["plan"]):
|
||||
assert "203.0.113" not in str(p) and "2222" not in str(p)
|
||||
assert p["sources_configured"] is True and p["nodes"][0]["items"][0]["protection"]["result"] == "true"
|
||||
assert plan["sources"] and plan["nodes"][0]["base_url"] # the stored plan itself is untouched
|
||||
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user