Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Julian Boldt
|
a9f562eec6 | ||
|
Julian Boldt
|
a3274f64ae | ||
|
Julian Boldt
|
9ba6357193 |
No files matched your search
@@ -31,6 +31,85 @@ den Einstellungen.
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Merlin …
|
||||
|
||||
## [1.25.0] "Goshawk" — 2026-10-21
|
||||
|
||||
Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).
|
||||
|
||||
- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
|
||||
`ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
|
||||
bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
|
||||
gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
|
||||
folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
|
||||
Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
|
||||
(nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
|
||||
Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
|
||||
Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
|
||||
- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
|
||||
Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
|
||||
nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
|
||||
eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
|
||||
älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
|
||||
- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
|
||||
Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
|
||||
zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
|
||||
(nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
|
||||
der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
|
||||
- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
|
||||
vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
|
||||
erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
|
||||
sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
|
||||
- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
|
||||
- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
|
||||
gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
|
||||
laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
|
||||
sicher, wird aber angekündigt.
|
||||
- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
|
||||
26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
|
||||
Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
|
||||
Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
|
||||
übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
|
||||
Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
|
||||
Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.
|
||||
|
||||
## [1.24.0] "Merlin" — 2026-10-21
|
||||
|
||||
- **Alarme für Gateways, Ressourcen und Dienste:** Neue Ereignisse in Einstellungen › Benachrichtigungen.
|
||||
**Gateway gestört** (standardmäßig **an**) meldet ein Gateway, das OPNsense als „down“ führt oder dessen Paketverlust (ab 20 %) bzw.
|
||||
Latenz (ab 500 ms) über der Schwelle liegt; bewusst abgeschaltete und nicht überwachte Gateways bleiben still. **Ressourcen-Schwelle**
|
||||
(standardmäßig aus) prüft Festplatte (ab 90 %), Arbeitsspeicher (ab 95 %), Swap (ab 50 %) und CPU-Last (ab 2 pro Thread), **Dienst ausgefallen**
|
||||
(standardmäßig aus) eine einstellbare Liste (Unbound, Dnsmasq, DHCP, NTP, OpenVPN, WireGuard, IPsec …) und meldet nur Dienste, die zuvor
|
||||
liefen. Alle drei stützen sich auf die Minutenmessung und lösen erst aus, wenn die letzten 3 (Ressourcen: 5) Messungen übereinstimmen, und
|
||||
enden erst, wenn alle wieder in Ordnung sind: Einzelspitzen und flackernde Verbindungen schicken nichts. Der Gateway-Status (Verlust, Latenz,
|
||||
Abweichung) steht außerdem auf der Host-Seite, als Kachel auf dem Dashboard und als Prometheus-Metriken (`opnexus_host_gateway_*`,
|
||||
`opnexus_host_swap_used_percent`, `opnexus_host_service_running`). Die Messung speichert dafür zusätzlich Swap, CPU-Threads, Gateways
|
||||
und Dienste in der vorhandenen Messreihe (keine neue Tabelle).
|
||||
- **Weitere Benachrichtigungskanäle:** ntfy, Gotify, Telegram, Microsoft Teams (Workflows-/Incoming-Webhook) und Matrix, beliebig viele
|
||||
je Typ, mit „Testen“-Knopf pro Kanal. Zugangsdaten (Tokens, die Teams-Adresse) liegen wie das SMTP-Passwort in der Geheimnis-Ablage, nie
|
||||
in der Datenbank oder in Antworten, und tauchen nicht in Fehlermeldungen oder Logs auf. Webhook und neue Kanäle prüfen die Zieladresse:
|
||||
nur http(s), keine Zugangsdaten in der Adresse, keine lokalen/Link-Local-/Metadaten-Adressen; Weiterleitungen werden nicht verfolgt.
|
||||
Betrifft die bestehende Webhook-Adresse: `localhost`, Loopback, 169.254.x.x und Zugangsdaten in der Adresse (`https://user:pass@…`)
|
||||
werden jetzt abgelehnt. Eine solche gespeicherte Adresse wird nicht mehr beliefert (nur ein Log-Eintrag), und die
|
||||
Benachrichtigungs-Einstellungen lassen sich erst wieder speichern, wenn sie korrigiert oder geleert ist (die Meldung nennt den Grund).
|
||||
Neue Einstellungen kommen mit Migration 0020.
|
||||
|
||||
- **Bestand übernehmen für Aliase und NAT:** Auf den Seiten Aliase und NAT (Port Forward, 1:1, Outbound) gibt es jetzt **Von Firewall übernehmen**
|
||||
wie bei den Regeln. Der Dialog zeigt, was von Hand in OPNsense angelegt wurde, einzeln zum Übernehmen oder mit „Alle übernehmbaren“ in einem Schritt;
|
||||
verschachtelte Aliase werden in der richtigen Reihenfolge übernommen. Die Übernahme schreibt nichts auf die Firewall: Die Vorschau zeigt direkt
|
||||
danach „keine Änderung“, der nächste Rollout aktualisiert dasselbe Objekt (kein Duplikat). Nicht übernommen werden OPNsense-eigene Aliase,
|
||||
Typen, die OPNexus nicht kennt (z. B. URL, GeoIP), Aliase mit Verweis auf nicht übernommene Aliase sowie NAT-Regeln, die das Modell verfälschen würden
|
||||
(z. B. „Kein Redirect“, verknüpfte Filterregel, mehrere Schnittstellen, „Kein NAT“); die Oberfläche nennt den Grund je Objekt.
|
||||
Outbound-NAT lässt sich nur übernehmen, wenn der Modus der Firewall Hybrid oder Manuell ist. Einstellungen, die OPNexus nicht verwaltet (Protokollierung,
|
||||
Pool-Optionen, Tags, Kategorien), bleiben unverändert und werden angezeigt.
|
||||
|
||||
- **Images für ARM:** Alle Release-Images (backend, frontend, proxy, updater) gibt es jetzt für `linux/amd64` und `linux/arm64`
|
||||
(Raspberry Pi 4/5, ARM-Server, Apple-Silicon-Docker).
|
||||
|
||||
## [1.23.11] "Harrier" — 2026-10-20
|
||||
|
||||
- **Updates per Klick prüfen jetzt ebenfalls die Signatur:** Der Updater-Container enthält `cosign` (feste Version, Prüfsumme beim Bau
|
||||
geprüft) und verifiziert die neuen Images vor dem Wechsel, wenn `cosign.pub` im Projektverzeichnis liegt. Das Image wird dadurch größer
|
||||
(rund 266 statt 123 MB).
|
||||
|
||||
## [1.23.10] "Harrier" — 2026-10-20
|
||||
|
||||
- **Signierte Release-Images:** Die Images (backend, frontend, proxy, updater) werden mit cosign signiert, beginnend mit 1.23.9. Der öffentliche
|
||||
|
||||
+6
-5
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||
|
||||
|
||||
> **Stand:** `1.23.10` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.25.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -16,10 +16,11 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
- **Sichere Änderungen** — jede Änderung läuft *Prüfen → Ausrollen → Bestätigen*; bleibt die Bestätigung im Zeitfenster aus, rollt OPNexus zurück.
|
||||
- **Hierarchische Policies** — Pre-/Post-Rulebases mit Vererbung über Gruppenbäume.
|
||||
- **VPN** — IPsec-Site-to-Site, OpenVPN und WireGuard, einmal definiert und ausgerollt.
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln lassen sich übernehmen.
|
||||
- **Netzwerk** — Schnittstellen, VLANs, virtuelle IPs, Gateways, Routen, LAGG/Bridge/GRE, DNS und DHCP je Firewall, dazu Tunables, Cron, Syslog, Snapshots und Diagnose (Ping, Traceroute, Port-Prüfung, DNS), ein Abgleich der Cluster-Knoten, mit Diff-Vorschau, Backup vor jeder Änderung und Schutz davor, die eigene Verbindung zu kappen; von Hand angelegte Regeln, Aliase und NAT-Regeln lassen sich übernehmen.
|
||||
- **Zertifikate** — Trust-Store für alle Firewalls und Ausstellung per ACME / Let's Encrypt.
|
||||
- **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
|
||||
- **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
|
||||
- **Update-Center** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus.
|
||||
- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
|
||||
- **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
|
||||
- **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
|
||||
- **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
|
||||
@@ -58,11 +59,11 @@ Nach der ersten Anmeldung öffnet sich von selbst ein kurzer **Einrichtungsassis
|
||||
|
||||
Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelieferter Caddy-Proxy) siehe das [Handbuch](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls).
|
||||
|
||||
**Fertige Images** (amd64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
**Fertige Images** (amd64 und arm64) liegen in der Registry dieses Gitea. In der `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.23.10
|
||||
OPNEXUS_VERSION=1.25.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||
|
||||
|
||||
> **Status:** `1.23.10` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.25.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -16,10 +16,11 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
- **Safe changes** — every change goes *preview → deploy → confirm*; without your confirmation inside the time window OPNexus rolls it back.
|
||||
- **Hierarchical policies** — pre/post rulebases with inheritance across group trees.
|
||||
- **VPN** — IPsec site-to-site, OpenVPN and WireGuard, defined once and rolled out.
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules created by hand on a firewall can be adopted.
|
||||
- **Network** — interfaces, VLANs, virtual IPs, gateways, routes, LAGG/bridge/GRE, DNS and DHCP per firewall, plus tunables, cron, syslog, snapshots and diagnostics (ping, traceroute, port check, DNS), a comparison of cluster nodes, with a diff preview, a backup before every change and a guard against cutting your own connection; rules, aliases and NAT rules created by hand on a firewall can be adopted.
|
||||
- **Certificates** — trust store for all firewalls and ACME / Let's Encrypt issuance.
|
||||
- **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
|
||||
- **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
|
||||
- **Update center** — firmware updates for the fleet, aware of HA order and maintenance mode.
|
||||
- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
|
||||
- **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
|
||||
- **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
|
||||
- **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
|
||||
@@ -60,11 +61,11 @@ After the first sign-in a short **setup assistant** opens on its own: set your o
|
||||
|
||||
For access from other machines and for production use with TLS (bundled Caddy proxy), see the [handbook](docs/HANDBUCH.md#zugriff-aus-dem-netz-und-tls) (German).
|
||||
|
||||
**Prebuilt images** (amd64) are published to this Gitea's registry. In `.env`:
|
||||
**Prebuilt images** (amd64 and arm64) are published to this Gitea's registry. In `.env`:
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.23.10
|
||||
OPNEXUS_VERSION=1.25.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -56,9 +56,9 @@ _READ_ONLY_ALLOWED_WRITES = {
|
||||
_SECURITY_KINDS = {
|
||||
"ipsec": "ipsec_tunnel", "openvpn": "openvpn_instance", "wireguard": "wireguard_tunnel",
|
||||
"certificates": "certificate", "acme": "acme", "notifications": "notification", "auth": "auth",
|
||||
"tokens": "api_token",
|
||||
"tokens": "api_token", "rollouts": "rollout",
|
||||
}
|
||||
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings"}
|
||||
_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings", "plans", "runs"}
|
||||
|
||||
|
||||
def _is_uuid(value: str) -> bool:
|
||||
@@ -117,7 +117,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
|
||||
)
|
||||
if user is not None:
|
||||
parts = request.url.path.strip("/").split("/")
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup"}
|
||||
kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
|
||||
writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
|
||||
if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
|
||||
event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}
|
||||
|
||||
@@ -116,6 +116,9 @@ class ErrorCode(str, Enum):
|
||||
NETWORK_FIELD_INVALID = "NETWORK_FIELD_INVALID"
|
||||
NETWORK_ITEM_NOT_FOUND = "NETWORK_ITEM_NOT_FOUND"
|
||||
RULE_NOT_ADOPTABLE = "RULE_NOT_ADOPTABLE"
|
||||
OBJECT_NOT_ADOPTABLE = "OBJECT_NOT_ADOPTABLE"
|
||||
ADOPT_OBJECT_NOT_FOUND = "ADOPT_OBJECT_NOT_FOUND"
|
||||
OUTBOUND_MODE_BLOCKS_ADOPT = "OUTBOUND_MODE_BLOCKS_ADOPT"
|
||||
REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0 = "REGEL_WURDE_INZWISCHEN_GEAENDERT_NEU_LADEN_UND_AENDERUNGEN_BF51B6B0"
|
||||
RESTORE_VORGANG_NICHT_GEFUNDEN = "RESTORE_VORGANG_NICHT_GEFUNDEN"
|
||||
ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL = "ROLLBACK_ERNEUT_FEHLGESCHLAGEN_PRUEFE_DIE_ERREICHBARKEIT_DER_FIREWALL"
|
||||
@@ -162,6 +165,11 @@ class ErrorCode(str, Enum):
|
||||
VARIABLES_MISSING = "VARIABLES_MISSING"
|
||||
VARIABLE_INVALID = "VARIABLE_INVALID"
|
||||
OVERLAY_INVALID = "OVERLAY_INVALID"
|
||||
NOTIFICATION_CHANNEL_INVALID = "NOTIFICATION_CHANNEL_INVALID"
|
||||
NOTIFICATION_CHANNEL_NOT_FOUND = "NOTIFICATION_CHANNEL_NOT_FOUND"
|
||||
NOTIFICATION_CHANNEL_NAME_TAKEN = "NOTIFICATION_CHANNEL_NAME_TAKEN"
|
||||
NOTIFICATION_CHANNEL_TEST_FAILED = "NOTIFICATION_CHANNEL_TEST_FAILED"
|
||||
NOTIFICATION_WEBHOOK_URL_INVALID = "NOTIFICATION_WEBHOOK_URL_INVALID"
|
||||
OVERLAY_SUBNET_LOCKED = "OVERLAY_SUBNET_LOCKED"
|
||||
OVERLAY_NAME_TAKEN = "OVERLAY_NAME_TAKEN"
|
||||
OVERLAY_PEER_NOT_FOUND = "OVERLAY_PEER_NOT_FOUND"
|
||||
@@ -171,6 +179,27 @@ class ErrorCode(str, Enum):
|
||||
API_TOKEN_NOT_FOUND = "API_TOKEN_NOT_FOUND"
|
||||
API_TOKEN_SCOPE_FORBIDDEN = "API_TOKEN_SCOPE_FORBIDDEN"
|
||||
API_TOKEN_SCOPE_TOO_HIGH = "API_TOKEN_SCOPE_TOO_HIGH"
|
||||
WINDOW_NOT_FOUND = "WINDOW_NOT_FOUND"
|
||||
WINDOW_INVALID = "WINDOW_INVALID"
|
||||
WINDOW_NAME_TAKEN = "WINDOW_NAME_TAKEN"
|
||||
WINDOW_IN_USE = "WINDOW_IN_USE"
|
||||
WINDOW_EXCEPTION_INVALID = "WINDOW_EXCEPTION_INVALID"
|
||||
WINDOW_EXCEPTION_EXISTS = "WINDOW_EXCEPTION_EXISTS"
|
||||
WINDOW_EXCEPTION_NOT_FOUND = "WINDOW_EXCEPTION_NOT_FOUND"
|
||||
ROLLOUT_PLAN_NOT_FOUND = "ROLLOUT_PLAN_NOT_FOUND"
|
||||
ROLLOUT_INVALID = "ROLLOUT_INVALID"
|
||||
ROLLOUT_NAME_TAKEN = "ROLLOUT_NAME_TAKEN"
|
||||
ROLLOUT_PLAN_INVALID = "ROLLOUT_PLAN_INVALID"
|
||||
ROLLOUT_PREVIEW_STALE = "ROLLOUT_PREVIEW_STALE"
|
||||
ROLLOUT_PLAN_BUSY = "ROLLOUT_PLAN_BUSY"
|
||||
ROLLOUT_GROUP_IN_USE = "ROLLOUT_GROUP_IN_USE"
|
||||
ROLLOUT_RUN_NOT_FOUND = "ROLLOUT_RUN_NOT_FOUND"
|
||||
ROLLOUT_ALREADY_ACTIVE = "ROLLOUT_ALREADY_ACTIVE"
|
||||
ROLLOUT_CONFIRM_NAME = "ROLLOUT_CONFIRM_NAME"
|
||||
ROLLOUT_ACTION_NOT_ALLOWED = "ROLLOUT_ACTION_NOT_ALLOWED"
|
||||
ROLLOUT_HOST_NOT_IN_RUN = "ROLLOUT_HOST_NOT_IN_RUN"
|
||||
ROLLOUT_HOST_RUNNING = "ROLLOUT_HOST_RUNNING"
|
||||
ROLLOUT_HOST_FAILED = "ROLLOUT_HOST_FAILED"
|
||||
|
||||
|
||||
def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:
|
||||
|
||||
@@ -46,10 +46,14 @@ from app.version import APP_VERSION
|
||||
from app.routers import dashboard as dashboard_router
|
||||
from app.routers import cluster as cluster_router
|
||||
from app.routers import services as services_router
|
||||
from app.routers import gateways as gateways_router
|
||||
from app.routers import resource_samples as resource_samples_router
|
||||
from app.services import resource_monitor_service
|
||||
from app.routers import scheduled_maintenance as scheduled_maintenance_router
|
||||
from app.routers import maintenance_windows as maintenance_windows_router
|
||||
from app.routers import rollouts as rollouts_router
|
||||
from app.services import scheduled_maintenance_service
|
||||
from app.services import rollout_engine
|
||||
from app.services import compliance_service
|
||||
from app.services import audit_export_service
|
||||
from app.services import overlay_service, variables_service
|
||||
@@ -254,6 +258,12 @@ async def _scheduled_maintenance_poll_loop():
|
||||
except Exception:
|
||||
logger.exception("Geplante-Wartung-Poll fehlgeschlagen")
|
||||
in_progress = 0
|
||||
# Firmware rollouts (1.25) only ever create schedules; the next tick of the lines above fires them.
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
in_progress += await rollout_engine.advance_runs(db)
|
||||
except Exception:
|
||||
logger.exception("Rollout-Poll fehlgeschlagen")
|
||||
# A handover run is stepped quickly (the partner should take over within a minute or two);
|
||||
# otherwise the relaxed cadence is plenty.
|
||||
await asyncio.sleep(_RUNNING_MAINTENANCE_POLL_SECONDS if in_progress else _SCHEDULED_MAINTENANCE_POLL_SECONDS)
|
||||
@@ -510,9 +520,12 @@ app.include_router(restore_router.router, prefix="/api")
|
||||
app.include_router(dashboard_router.router, prefix="/api")
|
||||
app.include_router(cluster_router.router, prefix="/api")
|
||||
app.include_router(services_router.router, prefix="/api")
|
||||
app.include_router(gateways_router.router, prefix="/api")
|
||||
app.include_router(network_router.router, prefix="/api")
|
||||
app.include_router(snapshots_router.router, prefix="/api")
|
||||
app.include_router(network_drift_router.router, prefix="/api")
|
||||
app.include_router(diagnostics_router.router, prefix="/api")
|
||||
app.include_router(resource_samples_router.router, prefix="/api")
|
||||
app.include_router(scheduled_maintenance_router.router, prefix="/api")
|
||||
app.include_router(maintenance_windows_router.router, prefix="/api")
|
||||
app.include_router(rollouts_router.router, prefix="/api")
|
||||
@@ -93,3 +93,9 @@ __all__ += ["HostResourceSample"]
|
||||
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
__all__ += ["ScheduledMaintenance"]
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
__all__ += ["MaintenanceWindow", "MaintenanceWindowException"]
|
||||
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun, RolloutRunHost, RolloutRunWave, rollout_wave_groups
|
||||
__all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "RolloutRunWave", "rollout_wave_groups"]
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Recurring maintenance windows (1.25): "every Sunday 02:00-05:00 Europe/Berlin",
|
||||
"first Tuesday of the month 22:00 for 3 h". A window on its own changes nothing on any
|
||||
firewall; rollout plans (models/rollout.py) use it to decide WHEN they may start work on a host.
|
||||
|
||||
Times are wall-clock times in the window's own IANA time zone, so a window stays at 02:00
|
||||
local time across daylight-saving changes (see services/window_service.py for the rules).
|
||||
"""
|
||||
import uuid
|
||||
from datetime import date, datetime, time, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, Date, DateTime, ForeignKey, Integer, String, Time, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class MaintenanceWindow(Base):
|
||||
__tablename__ = "maintenance_windows"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(120), unique=True)
|
||||
timezone: Mapped[str] = mapped_column(String(64))
|
||||
recurrence: Mapped[str] = mapped_column(String(16)) # weekly | monthly
|
||||
# weekly: weekdays 0 (Monday) .. 6 (Sunday). JSON columns get a Python default only (Alembic gotcha, see 0007).
|
||||
weekdays: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
# monthly: "nth_weekday" (month_nth 1-4 or -1 = last, month_weekday 0-6) or "day" (month_day 1-28)
|
||||
month_mode: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
month_nth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
month_weekday: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
month_day: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
start_time: Mapped[time] = mapped_column(Time)
|
||||
duration_minutes: Mapped[int] = mapped_column(Integer)
|
||||
# After this many minutes into the window no NEW host action starts (running ones finish).
|
||||
start_cutoff_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
created_by: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
|
||||
exceptions: Mapped[list["MaintenanceWindowException"]] = relationship(
|
||||
"MaintenanceWindowException", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by="MaintenanceWindowException.occurrence_date",
|
||||
)
|
||||
|
||||
|
||||
class MaintenanceWindowException(Base):
|
||||
"""One occurrence skipped or moved. Keyed by the occurrence's LOCAL date."""
|
||||
|
||||
__tablename__ = "maintenance_window_exceptions"
|
||||
__table_args__ = (UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
window_id: Mapped[uuid.UUID] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
occurrence_date: Mapped[date] = mapped_column(Date)
|
||||
kind: Mapped[str] = mapped_column(String(8)) # skip | move
|
||||
moved_start: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
reason: Mapped[str] = mapped_column(String(300), default="")
|
||||
actor: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
@@ -6,7 +6,7 @@ OPNsense API secrets stay off the database entirely.
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Integer, String, UniqueConstraint
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Float, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -17,6 +17,13 @@ def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
DEFAULT_SERVICE_WATCHLIST = (
|
||||
"unbound", "dnsmasq", "dhcpd", "dhcpd6", "kea-dhcp4", "kea-dhcp6", "ntpd", "openvpn", "wireguard", "strongswan",
|
||||
)
|
||||
|
||||
CHANNEL_TYPES = ("ntfy", "gotify", "telegram", "teams", "matrix")
|
||||
|
||||
|
||||
class NotificationSettings(Base):
|
||||
"""Single row, id fixed at 1 — see notification_settings_service.get_settings
|
||||
for the get-or-create helper. The only admin-editable operational config
|
||||
@@ -55,6 +62,30 @@ class NotificationSettings(Base):
|
||||
# Off by default: switching it on for an existing fleet reports every long-standing finding at once.
|
||||
compliance_alerts_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
# Monitoring alerts (1.24). The gateway alert is on by default (a WAN outage is the one thing every
|
||||
# operator wants to hear about); resource thresholds and service checks are off, like compliance alerts:
|
||||
# switching them on for an existing fleet may report long-standing conditions all at once.
|
||||
gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
|
||||
rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
# Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
|
||||
# OPNexus threshold on the measured values (0 = off).
|
||||
gateway_loss_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=20, server_default="20")
|
||||
gateway_delay_threshold_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=500, server_default="500")
|
||||
# Resources (0 = that threshold is off). Load is per CPU thread.
|
||||
disk_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=90, server_default="90")
|
||||
memory_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=95, server_default="95")
|
||||
swap_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=50, server_default="50")
|
||||
load_per_cpu_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=2.0, server_default="2.0")
|
||||
# Debouncing: consecutive 60 s resource samples that must agree before a problem opens or closes.
|
||||
alert_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=3, server_default="3")
|
||||
resource_confirm_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5")
|
||||
# OPNsense service names that are reported when they stop (the API cannot tell a crash from a stop on purpose).
|
||||
# NULL = the default list (an old row never got one).
|
||||
service_watchlist: Mapped[list | None] = mapped_column(JSON, nullable=True, default=lambda: list(DEFAULT_SERVICE_WATCHLIST))
|
||||
|
||||
poll_interval_seconds: Mapped[int] = mapped_column(Integer, default=300)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
|
||||
@@ -85,3 +116,21 @@ class NotificationProblem(Base):
|
||||
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
last_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
resolved_notified_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class NotificationChannel(Base):
|
||||
"""An additional delivery channel (ntfy, Gotify, Telegram, Microsoft Teams, Matrix) next to the SMTP
|
||||
and generic-webhook settings above. Several channels of one type are allowed. ``config`` holds only
|
||||
non-secret parameters (server URL, topic, chat id, room id); the credential — token, or the whole
|
||||
webhook URL for Teams — lives in notification_secret_store under ``channel_<id>``, never here."""
|
||||
|
||||
__tablename__ = "notification_channels"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||||
channel_type: Mapped[str] = mapped_column(String(16))
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
secret_set: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Firmware rollouts in waves (1.25): plans (what, in which order, when) and runs (what happened).
|
||||
|
||||
A plan's waves point at device groups ("rings": canary -> pilot -> prod). A run freezes the plan
|
||||
(``plan_snapshot``) and the resolved hosts (``RolloutRunHost``) when it starts, so later edits of
|
||||
the plan or of a group never change a run that is under way. The run never fires firmware actions
|
||||
itself: every host step is an ordinary ScheduledMaintenance row (origin="rollout",
|
||||
auto_verify=True) linked from ``RolloutRunHost.scheduled_maintenance_id`` — see
|
||||
services/rollout_service.py.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import (JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Table, Text,
|
||||
UniqueConstraint)
|
||||
from sqlalchemy.dialects.postgresql import UUID as PGUUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
rollout_wave_groups = Table(
|
||||
"rollout_wave_groups", Base.metadata,
|
||||
Column("wave_id", PGUUID(as_uuid=True), ForeignKey("rollout_plan_waves.id", ondelete="CASCADE"), primary_key=True),
|
||||
Column("group_id", PGUUID(as_uuid=True), ForeignKey("device_groups.id", ondelete="CASCADE"), primary_key=True),
|
||||
)
|
||||
|
||||
|
||||
class RolloutPlan(Base):
|
||||
__tablename__ = "rollout_plans"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
name: Mapped[str] = mapped_column(String(120), unique=True)
|
||||
description: Mapped[str] = mapped_column(Text, default="")
|
||||
action: Mapped[str] = mapped_column(String(16)) # update | major_update
|
||||
window_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
# Start a run by itself at every occurrence of the window. Only for "update" and only while the plan
|
||||
# is activated AND unchanged since (activated_fingerprint).
|
||||
auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
|
||||
activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
# Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
|
||||
auto_start_error: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
created_by: Mapped[str] = mapped_column(String(150))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
|
||||
|
||||
waves: Mapped[list["RolloutPlanWave"]] = relationship(
|
||||
"RolloutPlanWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by="RolloutPlanWave.position")
|
||||
|
||||
|
||||
class RolloutPlanWave(Base):
|
||||
__tablename__ = "rollout_plan_waves"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
plan_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="CASCADE"), index=True)
|
||||
position: Mapped[int] = mapped_column(Integer)
|
||||
label: Mapped[str] = mapped_column(String(60))
|
||||
soak_minutes: Mapped[int] = mapped_column(Integer, default=30)
|
||||
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Wait for a human "release" before this wave starts (not used for the first wave: starting is the release).
|
||||
require_approval: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
groups: Mapped[list["DeviceGroup"]] = relationship("DeviceGroup", secondary=rollout_wave_groups, lazy="selectin") # noqa: F821
|
||||
|
||||
|
||||
class RolloutRun(Base):
|
||||
__tablename__ = "rollout_runs"
|
||||
__table_args__ = (UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
plan_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
plan_name: Mapped[str] = mapped_column(String(120))
|
||||
plan_snapshot: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
action: Mapped[str] = mapped_column(String(16))
|
||||
# waiting_window | preflight | wave_running | soaking | gate | awaiting_approval | resuming | paused
|
||||
# | succeeded | cancelled
|
||||
status: Mapped[str] = mapped_column(String(20), index=True)
|
||||
paused_from: Mapped[str | None] = mapped_column(String(20), nullable=True)
|
||||
current_wave: Mapped[int] = mapped_column(Integer, default=0)
|
||||
target_version: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_by: Mapped[str] = mapped_column(String(150))
|
||||
trigger: Mapped[str] = mapped_column(String(8)) # manual | window
|
||||
window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
# Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
|
||||
revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
progress: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
step_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
waves: Mapped[list["RolloutRunWave"]] = relationship(
|
||||
"RolloutRunWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin", order_by="RolloutRunWave.index")
|
||||
hosts: Mapped[list["RolloutRunHost"]] = relationship(
|
||||
"RolloutRunHost", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
|
||||
order_by=lambda: [RolloutRunHost.wave_index, RolloutRunHost.order])
|
||||
|
||||
|
||||
class RolloutRunWave(Base):
|
||||
__tablename__ = "rollout_run_waves"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
|
||||
index: Mapped[int] = mapped_column(Integer)
|
||||
label: Mapped[str] = mapped_column(String(60))
|
||||
# pending | awaiting_approval | preflight | running | soaking | gate | passed | skipped | cancelled
|
||||
status: Mapped[str] = mapped_column(String(20), default="pending")
|
||||
baseline: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
gate_result: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
soak_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
gate_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
approved_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class RolloutRunHost(Base):
|
||||
__tablename__ = "rollout_run_hosts"
|
||||
__table_args__ = (UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),)
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
|
||||
wave_index: Mapped[int] = mapped_column(Integer)
|
||||
host_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
host_name: Mapped[str] = mapped_column(String(120))
|
||||
cluster_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
||||
order: Mapped[int] = mapped_column(Integer, default=0)
|
||||
# queued | running | verifying | succeeded | skipped | failed | blocked | cancelled
|
||||
status: Mapped[str] = mapped_column(String(16), default="queued")
|
||||
scheduled_maintenance_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
PGUUID(as_uuid=True), ForeignKey("scheduled_maintenance.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
version_before: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
version_after: Mapped[str | None] = mapped_column(String(40), nullable=True)
|
||||
# {"code": ..., "params": {...}} — translated in the UI (rollouts.hostNote.*)
|
||||
detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
@@ -42,7 +42,12 @@ class ScheduledMaintenance(Base):
|
||||
host_operation_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
|
||||
# Opt-in: hand the traffic over to the CARP partner first, run the action, then switch back.
|
||||
carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# Only while status == "running" with a handover: handover_wait | job_wait | leave | leave_wait.
|
||||
# 1.25: check the result automatically (job_wait -> verify) and close the operation, instead of
|
||||
# leaving it "submitted" for a human. Every rollout step sets it; off for existing plans.
|
||||
auto_verify: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
# manual | rollout (created by a rollout run, see models/rollout.py RolloutRunHost.scheduled_maintenance_id)
|
||||
origin: Mapped[str] = mapped_column(String(16), nullable=False, default="manual", server_default="manual")
|
||||
# Only while status == "running" with a handover or auto_verify: handover_wait | job_wait | verify | leave | leave_wait.
|
||||
# State lives here, not in memory, so a backend restart in the middle of a run resumes cleanly.
|
||||
phase: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
phase_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
"""The three "take over what exists on the firewall" endpoints that aliases, port forwards, 1:1 NAT and
|
||||
outbound NAT share (services/object_adopt_service.py has the logic and the evidence):
|
||||
|
||||
GET <prefix>/unmanaged?host_id=… candidates found live on the host, with reasons for the refused ones
|
||||
POST <prefix>/adopt take over one object {host_id, uuid}; returns the new central object
|
||||
POST <prefix>/adopt-all take over everything adoptable {host_id}; returns adopted + skipped
|
||||
|
||||
Like rule adoption this only creates central records; nothing is written to the firewall."""
|
||||
import uuid
|
||||
from typing import Callable
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.adopt import AdoptAllRequest, AdoptAllResult, AdoptObjectRequest, UnmanagedObject
|
||||
from app.services import object_adopt_service
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
|
||||
async def _host(db: AsyncSession, host_id: uuid.UUID) -> OpnsenseHost:
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if not host:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
return host
|
||||
|
||||
|
||||
def add_adopt_routes(router: APIRouter, *, kind: str, model, load_options: Callable, to_read: Callable, read_model) -> None:
|
||||
@router.get("/unmanaged", response_model=list[UnmanagedObject])
|
||||
async def list_unmanaged(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
host = await _host(db, host_id)
|
||||
try:
|
||||
return await object_adopt_service.list_unmanaged(db, kind, host)
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
|
||||
@router.post("/adopt", response_model=read_model, status_code=201)
|
||||
async def adopt(payload: AdoptObjectRequest, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
created = await object_adopt_service.adopt(db, kind, host, payload.uuid)
|
||||
except object_adopt_service.NotFoundOnHost as exc:
|
||||
raise api_error(404, ErrorCode.ADOPT_OBJECT_NOT_FOUND) from exc
|
||||
except object_adopt_service.NotAdoptable as exc:
|
||||
raise api_error(409, ErrorCode.OBJECT_NOT_ADOPTABLE, reasons=", ".join(exc.reasons)) from exc
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
# The new object sits in the session without its relationships loaded; db.get() would return it as-is.
|
||||
created = (await db.execute(
|
||||
select(model).where(model.id == created.id).options(*load_options())
|
||||
.execution_options(populate_existing=True)
|
||||
)).scalar_one()
|
||||
await hydrate_hierarchy_targets(db, [created])
|
||||
return to_read(created, created.targets, created.deployments)
|
||||
|
||||
@router.post("/adopt-all", response_model=AdoptAllResult)
|
||||
async def adopt_all(payload: AdoptAllRequest, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
host = await _host(db, payload.host_id)
|
||||
try:
|
||||
result = await object_adopt_service.adopt_all(db, kind, host)
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.HOST_REQUEST_FAILED, host=host.name) from exc
|
||||
await db.commit()
|
||||
return result
|
||||
@@ -17,6 +17,7 @@ from app.schemas.alias import DeploymentResult, AliasCreate, AliasRead, AliasEdi
|
||||
from app.services.alias_deploy_service import confirm_alias, deploy_alias, undeploy_alias, revert_alias_deployment
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/aliases", tags=["aliases"])
|
||||
|
||||
@@ -70,6 +71,9 @@ def _to_read(alias: FirewallAlias, hosts: list[OpnsenseHost], deployments: list[
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="alias", model=FirewallAlias, load_options=_load_options, to_read=_to_read, read_model=AliasRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[AliasRead])
|
||||
async def list_aliases(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(FirewallAlias).options(*_load_options()))
|
||||
|
||||
@@ -18,6 +18,8 @@ locks are deliberately NOT included here (no fleet-wide list endpoint
|
||||
exists for those, only per-host, and it's a rarely-triggered flow) —
|
||||
a documented v1 gap, not an oversight.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -46,7 +48,10 @@ from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.wireguard_tunnel import WireguardTunnel
|
||||
from app.models.wireguard_tunnel_deployment import WireguardTunnelDeployment
|
||||
from app.schemas.dashboard import ActivityItem, DashboardHost, DashboardSummary, PendingAction
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.monitor_checks import STALE_AFTER
|
||||
|
||||
router = APIRouter(prefix="/dashboard", tags=["dashboard"])
|
||||
|
||||
@@ -202,14 +207,34 @@ async def collect_pending_actions(db: AsyncSession) -> list[PendingAction]:
|
||||
return pending_actions
|
||||
|
||||
|
||||
async def _gateways_by_host(db: AsyncSession, hosts) -> dict:
|
||||
"""Gateway state from each host's newest resource sample; hosts without a fresh sample are left out."""
|
||||
result: dict = {}
|
||||
now = datetime.now(timezone.utc)
|
||||
for host in hosts:
|
||||
row = (
|
||||
await db.execute(
|
||||
select(HostResourceSample.sampled_at, HostResourceSample.raw["gateways"].as_json())
|
||||
.where(HostResourceSample.host_id == host.id)
|
||||
.order_by(HostResourceSample.sampled_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
).first()
|
||||
if row is not None and row[1] is not None and now - row[0] <= STALE_AFTER:
|
||||
result[host.id] = gateway_health_service.from_compact(row[1])
|
||||
return result
|
||||
|
||||
|
||||
@router.get("", response_model=DashboardSummary)
|
||||
async def summary(db: AsyncSession = Depends(get_db)):
|
||||
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
||||
statuses = await get_statuses(hosts)
|
||||
gateways = await _gateways_by_host(db, hosts)
|
||||
host_summaries = [
|
||||
DashboardHost(
|
||||
id=host.id, name=host.name, status=statuses.get(host.id, {}).get("status", "offline"),
|
||||
carp_status=statuses.get(host.id, {}).get("carp_status"), cluster_name=host.cluster_name,
|
||||
gateways=gateways.get(host.id),
|
||||
)
|
||||
for host in hosts
|
||||
]
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.gateway import GatewayHealth
|
||||
from app.services import gateway_health_service
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
router = APIRouter(prefix="/hosts/{host_id}/gateways", tags=["host-gateways"])
|
||||
|
||||
|
||||
@router.get("", response_model=list[GatewayHealth])
|
||||
async def list_gateways(host_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
"""Live gateway state (loss, delay, deviation, OPNsense' verdict) of one firewall."""
|
||||
host = await db.get(OpnsenseHost, host_id)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.HOST_NICHT_GEFUNDEN)
|
||||
try:
|
||||
return await gateway_health_service.get_gateway_health(host)
|
||||
except OpnsenseAPIError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
@@ -99,6 +99,9 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
|
||||
if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
|
||||
from app.models.rollout import rollout_wave_groups
|
||||
if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
|
||||
await db.delete(group)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Recurring maintenance windows and their exceptions (1.25).
|
||||
|
||||
Every write here counts as a rollout action (api_token_service.is_rollout): a window decides WHEN
|
||||
unattended firmware actions may run, so an operator needs the same right as for deploying.
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
from datetime import date, datetime, time, timedelta, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
from app.services import audit_service, window_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/maintenance-windows", tags=["maintenance-windows"])
|
||||
|
||||
_TIME = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
|
||||
|
||||
|
||||
class WindowIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
timezone: str = Field(min_length=1, max_length=64)
|
||||
recurrence: Literal["weekly", "monthly"]
|
||||
weekdays: list[int] = Field(default_factory=list, max_length=7)
|
||||
month_mode: Literal["nth_weekday", "day"] | None = None
|
||||
month_nth: int | None = None
|
||||
month_weekday: int | None = None
|
||||
month_day: int | None = None
|
||||
start_time: str
|
||||
duration_minutes: int = Field(ge=15, le=window_service.MAX_DURATION_MINUTES)
|
||||
start_cutoff_minutes: int = Field(default=60, ge=5, le=window_service.MAX_DURATION_MINUTES)
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
class ExceptionIn(BaseModel):
|
||||
occurrence_date: date
|
||||
kind: Literal["skip", "move"]
|
||||
moved_start: datetime | None = None
|
||||
reason: str = Field(default="", max_length=300)
|
||||
|
||||
|
||||
def _invalid(field: str):
|
||||
return api_error(422, ErrorCode.WINDOW_INVALID, field=field)
|
||||
|
||||
|
||||
def _validated(payload: WindowIn) -> dict:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
raise _invalid("name")
|
||||
if not window_service.valid_timezone(payload.timezone):
|
||||
raise _invalid("timezone")
|
||||
match = _TIME.match(payload.start_time)
|
||||
if not match:
|
||||
raise _invalid("start_time")
|
||||
values = {
|
||||
"name": name, "timezone": payload.timezone, "recurrence": payload.recurrence,
|
||||
"weekdays": [], "month_mode": None, "month_nth": None, "month_weekday": None, "month_day": None,
|
||||
"start_time": time(int(match.group(1)), int(match.group(2))),
|
||||
"duration_minutes": payload.duration_minutes,
|
||||
"start_cutoff_minutes": min(payload.start_cutoff_minutes, payload.duration_minutes),
|
||||
"enabled": payload.enabled,
|
||||
}
|
||||
if payload.recurrence == "weekly":
|
||||
days = sorted(set(payload.weekdays))
|
||||
if not days or any(d < 0 or d > 6 for d in days):
|
||||
raise _invalid("weekdays")
|
||||
values["weekdays"] = days
|
||||
elif payload.month_mode == "day":
|
||||
if payload.month_day is None or not 1 <= payload.month_day <= 28:
|
||||
raise _invalid("month_day")
|
||||
values.update(month_mode="day", month_day=payload.month_day)
|
||||
elif payload.month_mode == "nth_weekday":
|
||||
if payload.month_nth not in (1, 2, 3, 4, -1):
|
||||
raise _invalid("month_nth")
|
||||
if payload.month_weekday is None or not 0 <= payload.month_weekday <= 6:
|
||||
raise _invalid("month_weekday")
|
||||
values.update(month_mode="nth_weekday", month_nth=payload.month_nth, month_weekday=payload.month_weekday)
|
||||
else:
|
||||
raise _invalid("month_mode")
|
||||
return values
|
||||
|
||||
|
||||
async def _plans_using(db: AsyncSession, window_id) -> list[dict]:
|
||||
from app.models.rollout import RolloutPlan
|
||||
rows = (await db.scalars(select(RolloutPlan).where(RolloutPlan.window_id == window_id).order_by(RolloutPlan.name))).all()
|
||||
return [{"id": str(p.id), "name": p.name, "auto_start": p.auto_start, "active": p.activated_at is not None} for p in rows]
|
||||
|
||||
|
||||
def read_window(window: MaintenanceWindow, *, now: datetime | None = None, count: int = 5) -> dict:
|
||||
now = now or datetime.now(timezone.utc)
|
||||
return {
|
||||
"id": str(window.id), "name": window.name, "timezone": window.timezone, "recurrence": window.recurrence,
|
||||
"weekdays": window.weekdays or [], "month_mode": window.month_mode, "month_nth": window.month_nth,
|
||||
"month_weekday": window.month_weekday, "month_day": window.month_day,
|
||||
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
|
||||
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
|
||||
"created_by": window.created_by, "created_at": window.created_at,
|
||||
"open_now": window_service.current(window, now) is not None,
|
||||
"next": [o.as_dict() for o in window_service.occurrences(window, now, count, with_skipped=True)] if window.enabled else [],
|
||||
"exceptions": [
|
||||
{"id": str(e.id), "occurrence_date": e.occurrence_date.isoformat(), "kind": e.kind,
|
||||
"moved_start": e.moved_start, "reason": e.reason, "actor": e.actor, "created_at": e.created_at}
|
||||
for e in window.exceptions
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def _window_or_404(db: AsyncSession, window_id: uuid.UUID) -> MaintenanceWindow:
|
||||
window = await db.get(MaintenanceWindow, window_id)
|
||||
if window is None:
|
||||
raise api_error(404, ErrorCode.WINDOW_NOT_FOUND)
|
||||
return window
|
||||
|
||||
|
||||
async def _name_taken(db: AsyncSession, name: str, own_id=None) -> bool:
|
||||
other = await db.scalar(select(MaintenanceWindow.id).where(MaintenanceWindow.name == name))
|
||||
return other is not None and other != own_id
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_windows(db: AsyncSession = Depends(get_db)):
|
||||
rows = (await db.scalars(select(MaintenanceWindow).order_by(MaintenanceWindow.name))).all()
|
||||
result = []
|
||||
for w in rows:
|
||||
item = read_window(w)
|
||||
item["used_by"] = await _plans_using(db, w.id)
|
||||
result.append(item)
|
||||
return result
|
||||
|
||||
|
||||
@router.post("", status_code=201)
|
||||
async def create_window(payload: WindowIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
values = _validated(payload)
|
||||
if await _name_taken(db, values["name"]):
|
||||
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
|
||||
window = MaintenanceWindow(**values, created_by=request.state.user.username, created_at=datetime.now(timezone.utc))
|
||||
window.exceptions = []
|
||||
db.add(window)
|
||||
await db.commit()
|
||||
item = read_window(await _window_or_404(db, window.id))
|
||||
item["used_by"] = []
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{window_id}")
|
||||
async def get_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
item = read_window(await _window_or_404(db, window_id))
|
||||
item["used_by"] = await _plans_using(db, window_id)
|
||||
return item
|
||||
|
||||
|
||||
@router.put("/{window_id}")
|
||||
async def update_window(window_id: uuid.UUID, payload: WindowIn, db: AsyncSession = Depends(get_db)):
|
||||
"""Changing a window changes the fingerprint of every plan that uses it, so an activated
|
||||
automatic start stops until the plan is activated again (see rollout_service.fingerprint)."""
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
values = _validated(payload)
|
||||
if await _name_taken(db, values["name"], window.id):
|
||||
raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
|
||||
for key, value in values.items():
|
||||
setattr(window, key, value)
|
||||
await db.commit()
|
||||
item = read_window(await _window_or_404(db, window_id))
|
||||
item["used_by"] = await _plans_using(db, window_id)
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{window_id}", status_code=204)
|
||||
async def delete_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
# A plan without a window means "no time limit": deleting its window must never turn it into one.
|
||||
if await _plans_using(db, window_id):
|
||||
raise api_error(409, ErrorCode.WINDOW_IN_USE)
|
||||
await db.delete(window)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.get("/{window_id}/occurrences")
|
||||
async def window_occurrences(window_id: uuid.UUID, count: int = Query(10, ge=1, le=50), db: AsyncSession = Depends(get_db)):
|
||||
window = await _window_or_404(db, window_id)
|
||||
return [o.as_dict() for o in window_service.occurrences(window, datetime.now(timezone.utc), count, with_skipped=True)]
|
||||
|
||||
|
||||
@router.post("/{window_id}/exceptions", status_code=201)
|
||||
async def add_exception(window_id: uuid.UUID, payload: ExceptionIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
now = datetime.now(timezone.utc)
|
||||
if not window_service.is_occurrence_date(window, payload.occurrence_date):
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
|
||||
# Only future occurrences (or the one open now) can be changed; history stays as it was.
|
||||
if window_service.local_start(window, payload.occurrence_date) + timedelta(minutes=window.duration_minutes) <= now:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
|
||||
moved = None
|
||||
if payload.kind == "move":
|
||||
if payload.moved_start is None:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
|
||||
moved = payload.moved_start if payload.moved_start.tzinfo else payload.moved_start.replace(tzinfo=timezone.utc)
|
||||
if moved <= now:
|
||||
raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
|
||||
if any(e.occurrence_date == payload.occurrence_date for e in window.exceptions):
|
||||
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
|
||||
exc = MaintenanceWindowException(
|
||||
window_id=window.id, occurrence_date=payload.occurrence_date, kind=payload.kind, moved_start=moved,
|
||||
reason=payload.reason.strip(), actor=request.state.user.username, created_at=now,
|
||||
)
|
||||
db.add(exc)
|
||||
try:
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
|
||||
await audit_service.persist(
|
||||
kind="maintenance_window", action=f"exception_{payload.kind}", status="saved", entity_id=str(window.id),
|
||||
entity_name=window.name, details={"occurrence_date": payload.occurrence_date.isoformat(),
|
||||
"moved_start": moved.isoformat() if moved else None, "reason": exc.reason},
|
||||
)
|
||||
db.expire_all()
|
||||
return read_window(await _window_or_404(db, window_id))
|
||||
|
||||
|
||||
@router.delete("/{window_id}/exceptions/{exception_id}")
|
||||
async def delete_exception(window_id: uuid.UUID, exception_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
window = await _window_or_404(db, window_id)
|
||||
exc = await db.get(MaintenanceWindowException, exception_id)
|
||||
if exc is None or exc.window_id != window.id:
|
||||
raise api_error(404, ErrorCode.WINDOW_EXCEPTION_NOT_FOUND)
|
||||
await db.delete(exc)
|
||||
await db.commit()
|
||||
db.expire_all()
|
||||
return read_window(await _window_or_404(db, window_id))
|
||||
|
||||
@@ -1,21 +1,26 @@
|
||||
import uuid
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.database import get_db
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.models.notification import NotificationProblem
|
||||
from app.models.notification import NotificationChannel, NotificationProblem
|
||||
from app.schemas.notification import (
|
||||
NotificationChannelCreate,
|
||||
NotificationChannelRead,
|
||||
NotificationChannelUpdate,
|
||||
NotificationProblemRead,
|
||||
NotificationSettingsRead,
|
||||
NotificationSettingsUpdate,
|
||||
TestNotificationRequest,
|
||||
)
|
||||
from app.services import notification_delivery, notification_settings_service
|
||||
from app.services import notification_channels, notification_delivery, notification_secret_store, notification_settings_service, url_guard
|
||||
|
||||
router = APIRouter(prefix="/notifications", tags=["notifications"])
|
||||
|
||||
@@ -43,6 +48,11 @@ async def read_settings(db: AsyncSession = Depends(get_db), user: AdminUser = De
|
||||
|
||||
@router.put("/settings", response_model=NotificationSettingsRead)
|
||||
async def update_settings(payload: NotificationSettingsUpdate, db: AsyncSession = Depends(get_db)):
|
||||
if payload.webhook_url:
|
||||
try:
|
||||
payload.webhook_url = url_guard.validate_url(payload.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise api_error(422, ErrorCode.NOTIFICATION_WEBHOOK_URL_INVALID, reason=str(exc)) from None
|
||||
return await notification_settings_service.update_settings(db, payload)
|
||||
|
||||
|
||||
@@ -79,3 +89,103 @@ async def test_notification(payload: TestNotificationRequest, db: AsyncSession =
|
||||
except notification_delivery.NotificationDeliveryError as exc:
|
||||
raise api_error(502, ErrorCode.UPSTREAM_REQUEST_FAILED) from exc
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ── Additional channels (ntfy, Gotify, Telegram, Teams, Matrix) ────────────────
|
||||
|
||||
def _channel_read(row: NotificationChannel, user: AdminUser) -> NotificationChannelRead:
|
||||
read = NotificationChannelRead.model_validate(row)
|
||||
# An ntfy topic or a chat id is as good as a password for whoever knows it: only admins see the settings.
|
||||
return read if user.role == "admin" else read.model_copy(update={"config": {}})
|
||||
|
||||
|
||||
async def _get_channel(db: AsyncSession, channel_id: uuid.UUID) -> NotificationChannel:
|
||||
row = await db.get(NotificationChannel, channel_id)
|
||||
if row is None:
|
||||
raise api_error(404, ErrorCode.NOTIFICATION_CHANNEL_NOT_FOUND)
|
||||
return row
|
||||
|
||||
|
||||
def _invalid(exc: ValueError):
|
||||
return api_error(422, ErrorCode.NOTIFICATION_CHANNEL_INVALID, reason=str(exc))
|
||||
|
||||
|
||||
@router.get("/channels", response_model=list[NotificationChannelRead])
|
||||
async def list_channels(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
rows = await db.scalars(select(NotificationChannel).order_by(NotificationChannel.name))
|
||||
return [_channel_read(r, user) for r in rows.all()]
|
||||
|
||||
|
||||
@router.post("/channels", response_model=NotificationChannelRead, status_code=201)
|
||||
async def create_channel(payload: NotificationChannelCreate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
try:
|
||||
config = notification_channels.clean_config(payload.channel_type, payload.config)
|
||||
secret = notification_channels.clean_secret(payload.channel_type, payload.secret) if payload.secret else None
|
||||
except ValueError as exc:
|
||||
raise _invalid(exc) from None
|
||||
row = NotificationChannel(
|
||||
name=payload.name.strip(), channel_type=payload.channel_type, enabled=payload.enabled, config=config,
|
||||
secret_set=secret is not None,
|
||||
)
|
||||
row.id = uuid.uuid4()
|
||||
if secret is None and notification_channels.missing_requirements(row) == "Zugangsdaten fehlen":
|
||||
raise _invalid(ValueError("Zugangsdaten fehlen"))
|
||||
db.add(row)
|
||||
try:
|
||||
await db.flush()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
|
||||
if secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(row.id), secret)
|
||||
await db.commit()
|
||||
return _channel_read(row, user)
|
||||
|
||||
|
||||
@router.put("/channels/{channel_id}", response_model=NotificationChannelRead)
|
||||
async def update_channel(
|
||||
channel_id: uuid.UUID, payload: NotificationChannelUpdate,
|
||||
db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user),
|
||||
):
|
||||
row = await _get_channel(db, channel_id)
|
||||
try:
|
||||
if payload.config is not None:
|
||||
row.config = notification_channels.clean_config(row.channel_type, payload.config)
|
||||
new_secret = notification_channels.clean_secret(row.channel_type, payload.secret) if payload.secret else None
|
||||
except ValueError as exc:
|
||||
raise _invalid(exc) from None
|
||||
if payload.name is not None:
|
||||
row.name = payload.name.strip()
|
||||
if payload.enabled is not None:
|
||||
row.enabled = payload.enabled
|
||||
if payload.clear_secret:
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
|
||||
row.secret_set = False
|
||||
elif new_secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(row.id), new_secret)
|
||||
row.secret_set = True
|
||||
try:
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback()
|
||||
raise api_error(409, ErrorCode.NOTIFICATION_CHANNEL_NAME_TAKEN) from None
|
||||
return _channel_read(row, user)
|
||||
|
||||
|
||||
@router.delete("/channels/{channel_id}", status_code=204)
|
||||
async def delete_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
row = await _get_channel(db, channel_id)
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(row.id))
|
||||
await db.delete(row)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.post("/channels/{channel_id}/test")
|
||||
async def test_channel(channel_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
row = await _get_channel(db, channel_id)
|
||||
try:
|
||||
await notification_channels.send_test(row)
|
||||
except notification_delivery.NotificationDeliveryError as exc:
|
||||
# The text names the channel and the HTTP status; it never contains a URL or a token.
|
||||
raise api_error(502, ErrorCode.NOTIFICATION_CHANNEL_TEST_FAILED, reason=str(exc)) from None
|
||||
return {"success": True}
|
||||
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.one_to_one_deploy_service import confirm_one_to_one, deploy_one_to_one, undeploy_one_to_one, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/one-to-one", tags=["one-to-one"])
|
||||
|
||||
@@ -79,6 +80,9 @@ def _to_read(rule: OneToOneRule, hosts: list[OpnsenseHost], deployments: list[On
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="one_to_one", model=OneToOneRule, load_options=_load_options, to_read=_to_read, read_model=OneToOneRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[OneToOneRead])
|
||||
async def list_one_to_one_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(OneToOneRule).options(*_load_options()))
|
||||
|
||||
@@ -21,6 +21,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.outbound_nat_deploy_service import confirm_outbound_nat, deploy_outbound_nat, undeploy_outbound_nat, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/outbound-nat", tags=["outbound-nat"])
|
||||
|
||||
@@ -82,6 +83,9 @@ def _to_read(rule: OutboundNatRule, hosts: list[OpnsenseHost], deployments: list
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="outbound_nat", model=OutboundNatRule, load_options=_load_options, to_read=_to_read, read_model=OutboundNatRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[OutboundNatRead])
|
||||
async def list_outbound_nat_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(OutboundNatRule).options(*_load_options()))
|
||||
|
||||
@@ -20,6 +20,7 @@ from app.services.interface_service import check_interface_consistency
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
from app.services.port_forward_deploy_service import confirm_port_forward, deploy_port_forward, undeploy_port_forward, revert_deployment
|
||||
from app.services.policy_hierarchy_target_service import hydrate_hierarchy_targets, validate_hierarchy_group
|
||||
from app.routers.adopt_routes import add_adopt_routes
|
||||
|
||||
router = APIRouter(prefix="/port-forward", tags=["port-forward"])
|
||||
|
||||
@@ -80,6 +81,9 @@ def _to_read(rule: PortForwardRule, hosts: list[OpnsenseHost], deployments: list
|
||||
)
|
||||
|
||||
|
||||
add_adopt_routes(router, kind="port_forward", model=PortForwardRule, load_options=_load_options, to_read=_to_read, read_model=PortForwardRead)
|
||||
|
||||
|
||||
@router.get("", response_model=list[PortForwardRead])
|
||||
async def list_port_forward_rules(db: AsyncSession = Depends(get_db)):
|
||||
result = await db.execute(select(PortForwardRule).options(*_load_options()))
|
||||
|
||||
@@ -0,0 +1,432 @@
|
||||
"""Firmware rollouts in waves (1.25): plans and runs.
|
||||
|
||||
Roles (enforced in AuthMiddleware, see api_token_service.is_rollout): everybody signed in reads;
|
||||
operators prepare plans; starting/activating a plan and resuming/approving/skipping/aborting a run
|
||||
need the deploy right (admin, or operator while operators may deploy). Pausing is deliberately NOT
|
||||
a rollout action: anybody who may write at all may always brake.
|
||||
"""
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import audit_service, rollout_engine, rollout_health, rollout_service
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
router = APIRouter(prefix="/rollouts", tags=["rollouts"])
|
||||
|
||||
ACTIVE_RUN = rollout_engine.ACTIVE
|
||||
|
||||
|
||||
class WaveIn(BaseModel):
|
||||
label: str = Field(min_length=1, max_length=60)
|
||||
group_ids: list[uuid.UUID] = Field(min_length=1, max_length=20)
|
||||
soak_minutes: int = Field(default=30, ge=0, le=rollout_service.MAX_SOAK_MINUTES)
|
||||
carp_handover: bool = False
|
||||
require_approval: bool = False
|
||||
|
||||
|
||||
class PlanIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=120)
|
||||
description: str = Field(default="", max_length=2000)
|
||||
action: Literal["update", "major_update"] = "update"
|
||||
window_id: uuid.UUID | None = None
|
||||
auto_start: bool = False
|
||||
gate_checks: dict[str, bool] = Field(default_factory=dict)
|
||||
compliance_mode: Literal["warn", "fail"] = "warn"
|
||||
waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
|
||||
|
||||
|
||||
class TokenIn(BaseModel):
|
||||
preview_token: str = Field(min_length=64, max_length=64)
|
||||
|
||||
|
||||
async def _plan_or_404(db: AsyncSession, plan_id: uuid.UUID) -> RolloutPlan:
|
||||
plan = await rollout_service.load_plan(db, plan_id)
|
||||
if plan is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_PLAN_NOT_FOUND)
|
||||
return plan
|
||||
|
||||
|
||||
async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None = None) -> dict:
|
||||
resolved = resolved or await rollout_service.resolve(db, plan)
|
||||
active = plan.activated_at is not None
|
||||
last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
|
||||
return {
|
||||
**rollout_service.definition(plan),
|
||||
"description": plan.description,
|
||||
"waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
|
||||
"carp_handover": w.carp_handover, "require_approval": w.require_approval,
|
||||
"group_ids": [str(g.id) for g in w.groups], "groups": [g.name for g in w.groups],
|
||||
"host_count": len(resolved["waves"][i]["hosts"]) if i < len(resolved["waves"]) else 0}
|
||||
for i, w in enumerate(plan.waves)],
|
||||
"active": active, "activated_by": plan.activated_by, "activated_at": plan.activated_at,
|
||||
# Activated but the plan, its groups or its window changed since: the automatic start is held back.
|
||||
"fingerprint_ok": (not active) or plan.activated_fingerprint == rollout_service.fingerprint(plan, resolved),
|
||||
"auto_start_error": plan.auto_start_error,
|
||||
"window": resolved["window"], "errors": resolved["errors"], "warnings": resolved["warnings"],
|
||||
"host_count": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
"last_run": {"id": str(last_run.id), "status": last_run.status, "created_at": last_run.created_at} if last_run else None,
|
||||
"created_by": plan.created_by, "created_at": plan.created_at, "updated_at": plan.updated_at,
|
||||
}
|
||||
|
||||
|
||||
async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
|
||||
name = payload.name.strip()
|
||||
if not name:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="name")
|
||||
other = await db.scalar(select(RolloutPlan.id).where(RolloutPlan.name == name))
|
||||
if other is not None and other != plan.id:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_NAME_TAKEN)
|
||||
if payload.window_id is not None and await db.get(MaintenanceWindow, payload.window_id) is None:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="window_id")
|
||||
if any(k not in rollout_health.CHECKS for k in payload.gate_checks):
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="gate_checks")
|
||||
wanted = {g for w in payload.waves for g in w.group_ids}
|
||||
groups = {g.id: g for g in (await db.scalars(select(DeviceGroup).where(DeviceGroup.id.in_(wanted)))).all()}
|
||||
if len(groups) != len(wanted):
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="group_ids")
|
||||
plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
|
||||
plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
|
||||
plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
|
||||
plan.compliance_mode = payload.compliance_mode
|
||||
plan.waves = [
|
||||
RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
|
||||
require_approval=w.require_approval, groups=[groups[g] for g in dict.fromkeys(w.group_ids)])
|
||||
for i, w in enumerate(payload.waves)
|
||||
]
|
||||
# Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
|
||||
plan.activated_at = plan.activated_by = plan.activated_fingerprint = None
|
||||
plan.auto_start_error = None
|
||||
|
||||
|
||||
@router.get("/plans")
|
||||
async def list_plans(db: AsyncSession = Depends(get_db)):
|
||||
plans = (await db.scalars(select(RolloutPlan).order_by(RolloutPlan.name))).all()
|
||||
return [await read_plan(db, p) for p in plans]
|
||||
|
||||
|
||||
@router.post("/plans", status_code=201)
|
||||
async def create_plan(payload: PlanIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = RolloutPlan(created_by=request.state.user.username, created_at=datetime.now(timezone.utc), waves=[])
|
||||
await _apply(db, plan, payload)
|
||||
db.add(plan)
|
||||
await db.commit()
|
||||
return await read_plan(db, await _plan_or_404(db, plan.id))
|
||||
|
||||
|
||||
@router.get("/plans/{plan_id}")
|
||||
async def get_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.put("/plans/{plan_id}")
|
||||
async def update_plan(plan_id: uuid.UUID, payload: PlanIn, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
was_active = plan.activated_at is not None
|
||||
await _apply(db, plan, payload)
|
||||
await db.commit()
|
||||
if was_active:
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id),
|
||||
entity_name=plan.name, message="Plan geändert: automatischer Start deaktiviert")
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.delete("/plans/{plan_id}", status_code=204)
|
||||
async def delete_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.status.in_(ACTIVE_RUN)).limit(1)):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_PLAN_BUSY)
|
||||
await db.delete(plan)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/preview")
|
||||
async def preview_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
return {**resolved, "preview_token": rollout_service.fingerprint(plan, resolved),
|
||||
"can_start": not resolved["errors"], "plan": await read_plan(db, plan, resolved)}
|
||||
|
||||
|
||||
async def _checked_token(db: AsyncSession, plan: RolloutPlan, token: str) -> tuple[dict, str]:
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
current = rollout_service.fingerprint(plan, resolved)
|
||||
if token != current:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_PREVIEW_STALE)
|
||||
if resolved["errors"]:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_PLAN_INVALID, reason=resolved["errors"][0]["code"])
|
||||
return resolved, current
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/activate")
|
||||
async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if not plan.auto_start:
|
||||
raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
|
||||
_resolved, current = await _checked_token(db, plan, payload.preview_token)
|
||||
plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
|
||||
entity_name=plan.name, details={"fingerprint": current})
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/deactivate")
|
||||
async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
plan.activated_by = plan.activated_at = plan.activated_fingerprint = None
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
|
||||
return await read_plan(db, await _plan_or_404(db, plan_id))
|
||||
|
||||
|
||||
# ── runs ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
class StartIn(TokenIn):
|
||||
confirm_name: str = Field(min_length=1, max_length=120)
|
||||
acknowledge_unattended: Literal[True]
|
||||
|
||||
|
||||
class ReasonIn(BaseModel):
|
||||
reason: str = Field(default="", max_length=500)
|
||||
|
||||
|
||||
class SkipIn(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
reason: str = Field(min_length=3, max_length=500)
|
||||
|
||||
|
||||
class AbortIn(BaseModel):
|
||||
reason: str = Field(min_length=3, max_length=500)
|
||||
confirm_name: str = Field(min_length=1, max_length=120)
|
||||
|
||||
|
||||
def _read_run_summary(run: RolloutRun) -> dict:
|
||||
counts: dict[str, int] = {}
|
||||
for h in run.hosts:
|
||||
counts[h.status] = counts.get(h.status, 0) + 1
|
||||
return {
|
||||
"id": str(run.id), "plan_id": str(run.plan_id) if run.plan_id else None, "plan_name": run.plan_name,
|
||||
"action": run.action, "status": run.status, "paused_from": run.paused_from, "current_wave": run.current_wave,
|
||||
"wave_count": len(run.waves), "current_wave_label": next((w.label for w in run.waves if w.index == run.current_wave), None),
|
||||
"target_version": run.target_version, "pause_reason": run.pause_reason, "pause_detail": run.pause_detail,
|
||||
"started_by": run.started_by, "trigger": run.trigger, "window_occurrence": run.window_occurrence,
|
||||
"created_at": run.created_at, "finished_at": run.finished_at, "step_at": run.step_at, "host_counts": counts,
|
||||
"active": run.status in rollout_engine.ACTIVE,
|
||||
}
|
||||
|
||||
|
||||
async def read_run(db: AsyncSession, run: RolloutRun) -> dict:
|
||||
scheds = {}
|
||||
ids = [h.scheduled_maintenance_id for h in run.hosts if h.scheduled_maintenance_id]
|
||||
if ids:
|
||||
scheds = {s.id: s for s in (await db.scalars(select(ScheduledMaintenance).where(ScheduledMaintenance.id.in_(ids)))).all()}
|
||||
soak_until = next((w.soak_until for w in run.waves if w.index == run.current_wave), None)
|
||||
return {
|
||||
**_read_run_summary(run),
|
||||
"soak_until": soak_until, "progress": {k: v for k, v in (run.progress or {}).items()
|
||||
if k in ("gate_recheck_at", "cluster_recheck_at", "blocked_since", "resume_gate",
|
||||
"cluster_gate", "preflight_started", "resume_recheck_at")},
|
||||
"snapshot": run.plan_snapshot,
|
||||
"waves": [{"index": w.index, "label": w.label, "status": w.status, "gate_result": w.gate_result,
|
||||
"gate_attempts": w.gate_attempts, "soak_until": w.soak_until, "approved_by": w.approved_by,
|
||||
"started_at": w.started_at, "finished_at": w.finished_at,
|
||||
"spec": run.plan_snapshot["waves"][w.index] if w.index < len(run.plan_snapshot.get("waves", [])) else None}
|
||||
for w in run.waves],
|
||||
"hosts": [{"id": str(h.id), "host_id": str(h.host_id) if h.host_id else None, "host_name": h.host_name,
|
||||
"cluster_name": h.cluster_name, "wave_index": h.wave_index, "order": h.order, "status": h.status,
|
||||
"version_before": h.version_before, "version_after": h.version_after, "detail": h.detail,
|
||||
"started_at": h.started_at, "finished_at": h.finished_at,
|
||||
"schedule": ({"id": str(s.id), "status": s.status, "phase": s.phase, "carp_handover": s.carp_handover,
|
||||
"host_operation_id": str(s.host_operation_id) if s.host_operation_id else None}
|
||||
if (s := scheds.get(h.scheduled_maintenance_id)) else None)}
|
||||
for h in sorted(run.hosts, key=lambda h: (h.wave_index, h.order, h.host_name))],
|
||||
}
|
||||
|
||||
|
||||
async def _cancel_if_pending(db: AsyncSession, sched_id) -> bool:
|
||||
"""Cancel a host's schedule only if it has not fired. Locked: the poll loop may be firing it right now
|
||||
(fire_due holds the row), and a blind write would turn a running action into "cancelled"."""
|
||||
if sched_id is None:
|
||||
return False
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sched_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if sched is not None and sched.status == "pending":
|
||||
sched.status = "cancelled"
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _run_or_404(db: AsyncSession, run_id: uuid.UUID, *, lock: bool = False) -> RolloutRun:
|
||||
stmt = select(RolloutRun).where(RolloutRun.id == run_id).execution_options(populate_existing=True)
|
||||
if lock:
|
||||
stmt = stmt.with_for_update()
|
||||
run = await db.scalar(stmt)
|
||||
if run is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_RUN_NOT_FOUND)
|
||||
return run
|
||||
|
||||
|
||||
def _not_now(run: RolloutRun):
|
||||
return api_error(409, ErrorCode.ROLLOUT_ACTION_NOT_ALLOWED, status=run.status)
|
||||
|
||||
|
||||
async def _done(db: AsyncSession, run: RolloutRun, action: str, status: str, request: Request, **details) -> dict:
|
||||
run.revision += 1
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id), entity_name=run.plan_name,
|
||||
details={"wave": run.current_wave, **details})
|
||||
return await read_run(db, await _run_or_404(db, run.id))
|
||||
|
||||
|
||||
@router.post("/plans/{plan_id}/start", status_code=201)
|
||||
async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
plan = await _plan_or_404(db, plan_id)
|
||||
if payload.confirm_name != plan.name:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
|
||||
resolved, _token = await _checked_token(db, plan, payload.preview_token)
|
||||
if (other := await rollout_engine.active_run(db)) is not None:
|
||||
# One run at a time: two runs could otherwise touch the same cluster at once.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
|
||||
run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username, trigger="manual")
|
||||
await db.commit()
|
||||
await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
|
||||
details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
|
||||
"waves": [w["label"] for w in resolved["waves"]]})
|
||||
return await read_run(db, await _run_or_404(db, run.id))
|
||||
|
||||
|
||||
@router.get("/runs")
|
||||
async def list_runs(db: AsyncSession = Depends(get_db)):
|
||||
runs = (await db.scalars(select(RolloutRun).order_by(RolloutRun.created_at.desc()).limit(50))).all()
|
||||
return [_read_run_summary(r) for r in runs]
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def rollout_status(db: AsyncSession = Depends(get_db)):
|
||||
"""For /updates and the host pages: the active run and the hosts it still owns."""
|
||||
run = await rollout_engine.active_run(db)
|
||||
if run is None:
|
||||
return {"active": None, "host_ids": [], "current_host_ids": []}
|
||||
return {
|
||||
"active": _read_run_summary(run),
|
||||
"host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.status in ("queued", "blocked", "running", "verifying", "failed")],
|
||||
"current_host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.wave_index == run.current_wave
|
||||
and h.status in ("queued", "blocked", "running", "verifying", "failed")],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}")
|
||||
async def get_run(run_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
return await read_run(db, await _run_or_404(db, run_id))
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/pause")
|
||||
async def pause_run(run_id: uuid.UUID, payload: ReasonIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE or run.status == "paused":
|
||||
raise _not_now(run)
|
||||
# A brake that lets the next host fire anyway is no brake: a step not fired yet goes back to the queue.
|
||||
requeued = [h.host_name for h in run.hosts if await rollout_engine.requeue_if_pending(db, h)]
|
||||
rollout_engine.pause(run, "manual", by=request.state.user.username, text=payload.reason.strip()[:300])
|
||||
return await _done(db, run, "pause", "paused", request, reason="manual", text=payload.reason.strip()[:300], requeued=requeued)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/resume")
|
||||
async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status != "paused":
|
||||
raise _not_now(run)
|
||||
failed = [h.host_name for h in run.hosts if h.status == "failed"]
|
||||
if failed:
|
||||
# A failed host is never retried by itself: check it, then skip it explicitly.
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
|
||||
previous = run.started_by
|
||||
# Whoever resumes takes over the responsibility (and the permission check) from here on.
|
||||
run.started_by = request.state.user.username
|
||||
run.status = "resuming"
|
||||
run.pause_reason = run.pause_detail = None
|
||||
run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
|
||||
return await _done(db, run, "resume", "resuming", request, previous_owner=previous)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/approve")
|
||||
async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status != "awaiting_approval":
|
||||
raise _not_now(run)
|
||||
wave = rollout_engine.wave_row(run, run.current_wave)
|
||||
wave.approved_by, wave.status = request.state.user.username, "preflight"
|
||||
run.started_by = request.state.user.username
|
||||
run.status = "preflight"
|
||||
run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
|
||||
return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/skip-host")
|
||||
async def skip_host(run_id: uuid.UUID, payload: SkipIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE:
|
||||
raise _not_now(run)
|
||||
host = next((h for h in run.hosts if h.host_id == payload.host_id), None)
|
||||
if host is None:
|
||||
raise api_error(404, ErrorCode.ROLLOUT_HOST_NOT_IN_RUN)
|
||||
if host.status in ("running", "verifying"):
|
||||
if not await _cancel_if_pending(db, host.scheduled_maintenance_id):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
|
||||
# not fired yet: nothing happened on the firewall
|
||||
elif host.status not in ("queued", "blocked", "failed"):
|
||||
raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
|
||||
host.status = "skipped"
|
||||
host.detail = {"code": "skipped_by", "params": {"user": request.state.user.username, "reason": payload.reason.strip()[:300]}}
|
||||
host.finished_at = datetime.now(timezone.utc)
|
||||
return await _done(db, run, "skip_host", "skipped", request, host=host.host_name, reason=payload.reason.strip()[:300])
|
||||
|
||||
|
||||
@router.post("/runs/{run_id}/abort")
|
||||
async def abort_run(run_id: uuid.UUID, payload: AbortIn, request: Request, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
run = await _run_or_404(db, run_id, lock=True)
|
||||
if run.status not in rollout_engine.ACTIVE:
|
||||
raise _not_now(run)
|
||||
if payload.confirm_name != run.plan_name:
|
||||
raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
|
||||
now = datetime.now(timezone.utc)
|
||||
still_running = []
|
||||
for host in run.hosts:
|
||||
if host.status in ("running", "verifying") and host.scheduled_maintenance_id:
|
||||
if await _cancel_if_pending(db, host.scheduled_maintenance_id):
|
||||
host.status, host.finished_at = "cancelled", now
|
||||
else:
|
||||
still_running.append(host.host_name) # an action already on the firewall is finished in a controlled way
|
||||
elif host.status in ("queued", "blocked"):
|
||||
host.status, host.finished_at = "cancelled", now
|
||||
for wave in run.waves:
|
||||
if wave.status not in ("passed", "skipped"):
|
||||
wave.status = "cancelled"
|
||||
run.status, run.finished_at = "cancelled", now
|
||||
run.pause_reason, run.pause_detail = "aborted", {"by": request.state.user.username, "reason": payload.reason.strip()[:300],
|
||||
"still_running": still_running}
|
||||
return await _done(db, run, "abort", "cancelled", request, reason=payload.reason.strip()[:300], still_running=still_running)
|
||||
@@ -24,6 +24,8 @@ class Schedule(BaseModel):
|
||||
acknowledge_unattended: Literal[True]
|
||||
# Hand the traffic over to the CARP partner first and switch back afterwards (cluster nodes only).
|
||||
carp_handover: bool = False
|
||||
# Check the result automatically afterwards and close the operation (no manual "Wartung abschließen").
|
||||
auto_verify: bool = False
|
||||
|
||||
|
||||
def _read(sched: ScheduledMaintenance) -> dict:
|
||||
@@ -32,6 +34,8 @@ def _read(sched: ScheduledMaintenance) -> dict:
|
||||
"status": sched.status, "actor": sched.actor, "created_at": sched.created_at,
|
||||
"result": sched.result, "host_operation_id": sched.host_operation_id,
|
||||
"carp_handover": sched.carp_handover, "phase": sched.phase,
|
||||
"auto_verify": sched.auto_verify, "origin": sched.origin,
|
||||
"rollout_run_id": ((sched.progress or {}).get("rollout") or {}).get("run_id"),
|
||||
}
|
||||
|
||||
|
||||
@@ -80,6 +84,7 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
|
||||
sched = ScheduledMaintenance(
|
||||
host_id=host.id, host_name=host.name, action=payload.action, run_at=run_at,
|
||||
actor=request.state.user.username, created_at=now, carp_handover=payload.carp_handover,
|
||||
auto_verify=payload.auto_verify,
|
||||
)
|
||||
db.add(sched)
|
||||
await db.commit()
|
||||
@@ -90,7 +95,9 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
|
||||
async def cancel_scheduled(host_id: uuid.UUID, schedule_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
|
||||
await policy_lock(db)
|
||||
await _host_or_404(db, host_id)
|
||||
sched = await db.get(ScheduledMaintenance, schedule_id)
|
||||
# Locked and re-read: the poll loop may be firing this very row right now (it holds the lock meanwhile).
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == schedule_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if not sched or sched.host_id != host_id:
|
||||
raise api_error(404, ErrorCode.GEPLANTE_WARTUNG_NICHT_GEFUNDEN)
|
||||
if sched.status != "pending":
|
||||
|
||||
@@ -7,6 +7,7 @@ from app.database import get_db
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.services import rollout_engine
|
||||
from app.services import update_check_service, updater_client
|
||||
|
||||
router = APIRouter(prefix="/system", tags=["system"])
|
||||
@@ -99,6 +100,9 @@ async def updater_status(db: AsyncSession = Depends(get_db), user: AdminUser = D
|
||||
"can_update": False, "major_change": False,
|
||||
"job": None, "phases": [], "log": [], "services": {},
|
||||
"rollback_available": False, "rollback_loses_data": False, "rollback_version": None,
|
||||
# A restart in the middle of a firmware rollout is safe (its state is in the database), but the
|
||||
# UI warns: the run pauses its 15-second steps for as long as OPNexus is down.
|
||||
"active_rollouts": await rollout_engine.count_active(db),
|
||||
}
|
||||
try:
|
||||
updater = await updater_client.status()
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import uuid
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AdoptObjectRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
uuid: str = Field(min_length=1, max_length=64) # the object's UUID on the firewall
|
||||
|
||||
|
||||
class AdoptAllRequest(BaseModel):
|
||||
host_id: uuid.UUID
|
||||
|
||||
|
||||
class UnmanagedObject(BaseModel):
|
||||
"""An alias / NAT rule found on a host that OPNexus does not manage yet (see object_adopt_service.py)."""
|
||||
|
||||
uuid: str
|
||||
title: str
|
||||
enabled: bool
|
||||
adoptable: bool # can be adopted on its own right now
|
||||
# Adoptable in the same "adopt all" run although not on its own (an alias whose referenced aliases come first).
|
||||
bulk_adoptable: bool
|
||||
# Codes such as "unsupported_type" or "no_target"; the UI turns them into text.
|
||||
reasons: list[str]
|
||||
reason_details: dict[str, str] = Field(default_factory=dict) # code -> extra text (alias names, type, ...)
|
||||
# Settings OPNexus does not manage that stay untouched on the firewall (log, pool options, tags, ...).
|
||||
kept: list[str] = Field(default_factory=list)
|
||||
# Facts for display, depend on the object type.
|
||||
info: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AdoptedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
id: uuid.UUID
|
||||
|
||||
|
||||
class SkippedObject(BaseModel):
|
||||
uuid: str
|
||||
title: str
|
||||
reasons: list[str]
|
||||
reason_details: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AdoptAllResult(BaseModel):
|
||||
adopted: list[AdoptedObject]
|
||||
skipped: list[SkippedObject]
|
||||
@@ -3,6 +3,8 @@ from datetime import datetime
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.schemas.gateway import GatewayHealth
|
||||
|
||||
|
||||
class DashboardHost(BaseModel):
|
||||
id: uuid.UUID
|
||||
@@ -10,6 +12,8 @@ class DashboardHost(BaseModel):
|
||||
status: str
|
||||
carp_status: str | None
|
||||
cluster_name: str | None
|
||||
# From the last resource sample (no live call); None when there is no sample from the last 3 minutes.
|
||||
gateways: list[GatewayHealth] | None = None
|
||||
|
||||
|
||||
class PendingAction(BaseModel):
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
# online: all fine · degraded: OPNsense flags loss and/or latency above the gateway's own limits ·
|
||||
# offline: down · forced_down: switched off by the admin on purpose · unknown: nothing measured
|
||||
# (monitoring disabled) or the verdict is still pending.
|
||||
GatewayState = Literal["online", "degraded", "offline", "forced_down", "unknown"]
|
||||
|
||||
|
||||
class GatewayHealth(BaseModel):
|
||||
name: str
|
||||
address: str | None = None
|
||||
status: str # OPNsense' raw verdict: none | down | force_down | delay | loss | delay+loss | …
|
||||
state: GatewayState
|
||||
loss: float | None = None # percent
|
||||
delay: float | None = None # milliseconds
|
||||
stddev: float | None = None # milliseconds
|
||||
monitor: str | None = None # the monitored address
|
||||
@@ -1,10 +1,12 @@
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST
|
||||
|
||||
SmtpSecurity = Literal["none", "starttls", "ssl"]
|
||||
ProblemType = Literal[
|
||||
@@ -14,7 +16,10 @@ ProblemType = Literal[
|
||||
"backup_stale",
|
||||
"backup_offsite_failed",
|
||||
"compliance_finding",
|
||||
"gateway_down", "resource_threshold", "service_down",
|
||||
"rollout_paused", "rollout_start_failed",
|
||||
]
|
||||
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
||||
|
||||
|
||||
class NotificationSettingsRead(BaseModel):
|
||||
@@ -45,10 +50,28 @@ class NotificationSettingsRead(BaseModel):
|
||||
scheduled_maintenance_failed_enabled: bool = True
|
||||
backup_failed_enabled: bool = True
|
||||
compliance_alerts_enabled: bool = False
|
||||
gateway_down_enabled: bool = True
|
||||
resource_threshold_enabled: bool = False
|
||||
service_down_enabled: bool = False
|
||||
rollout_problems_enabled: bool = True
|
||||
gateway_loss_threshold_pct: int = 20
|
||||
gateway_delay_threshold_ms: int = 500
|
||||
disk_threshold_pct: int = 90
|
||||
memory_threshold_pct: int = 95
|
||||
swap_threshold_pct: int = 50
|
||||
load_per_cpu_threshold: float = 2.0
|
||||
alert_confirm_samples: int = 3
|
||||
resource_confirm_samples: int = 5
|
||||
service_watchlist: list[str] = Field(default_factory=lambda: list(DEFAULT_SERVICE_WATCHLIST))
|
||||
|
||||
poll_interval_seconds: int
|
||||
updated_at: datetime
|
||||
|
||||
@field_validator("service_watchlist", mode="before")
|
||||
@classmethod
|
||||
def _default_watchlist(cls, value):
|
||||
return list(DEFAULT_SERVICE_WATCHLIST) if value is None else value
|
||||
|
||||
|
||||
class NotificationSettingsUpdate(BaseModel):
|
||||
"""Partial update — everything optional, only provided fields change.
|
||||
@@ -84,6 +107,34 @@ class NotificationSettingsUpdate(BaseModel):
|
||||
scheduled_maintenance_failed_enabled: bool | None = None
|
||||
backup_failed_enabled: bool | None = None
|
||||
compliance_alerts_enabled: bool | None = None
|
||||
gateway_down_enabled: bool | None = None
|
||||
resource_threshold_enabled: bool | None = None
|
||||
service_down_enabled: bool | None = None
|
||||
rollout_problems_enabled: bool | None = None
|
||||
# 0 switches that single threshold off.
|
||||
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
||||
disk_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
memory_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
swap_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
load_per_cpu_threshold: float | None = Field(default=None, ge=0, le=1000)
|
||||
alert_confirm_samples: int | None = Field(default=None, ge=1, le=60)
|
||||
resource_confirm_samples: int | None = Field(default=None, ge=1, le=60)
|
||||
service_watchlist: list[str] | None = Field(default=None, max_length=100)
|
||||
|
||||
@field_validator("service_watchlist")
|
||||
@classmethod
|
||||
def _clean_watchlist(cls, value):
|
||||
if value is None:
|
||||
return None
|
||||
names = []
|
||||
for raw in value:
|
||||
name = str(raw).strip()
|
||||
if not re.fullmatch(r"[A-Za-z0-9._+-]{1,64}", name):
|
||||
raise ValueError("Dienstnamen bestehen aus Buchstaben, Ziffern und . _ + -")
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
poll_interval_seconds: int | None = Field(
|
||||
default=None, ge=settings.notification_poll_min_interval_seconds
|
||||
@@ -106,3 +157,55 @@ class NotificationProblemRead(BaseModel):
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
channel: Literal["webhook", "email"]
|
||||
|
||||
|
||||
class NotificationChannelRead(BaseModel):
|
||||
"""Never carries the credential; ``config`` is withheld from non-admins (an ntfy topic is a password)."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: uuid.UUID
|
||||
name: str
|
||||
channel_type: ChannelType
|
||||
enabled: bool
|
||||
config: dict
|
||||
secret_set: bool
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
def _channel_name(value: str | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if not value.strip():
|
||||
raise ValueError("Name darf nicht leer sein")
|
||||
return value.strip()
|
||||
|
||||
|
||||
class NotificationChannelCreate(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=100)
|
||||
channel_type: ChannelType
|
||||
enabled: bool = True
|
||||
config: dict = Field(default_factory=dict)
|
||||
# Token, or for Teams the webhook address. Write-only.
|
||||
secret: str | None = Field(default=None, max_length=2100)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def _clean_name(cls, value):
|
||||
return _channel_name(value)
|
||||
|
||||
|
||||
class NotificationChannelUpdate(BaseModel):
|
||||
"""Partial: omitted fields stay. ``secret`` replaces the credential, ``clear_secret`` removes it."""
|
||||
|
||||
name: str | None = Field(default=None, min_length=1, max_length=100)
|
||||
enabled: bool | None = None
|
||||
config: dict | None = None
|
||||
secret: str | None = Field(default=None, max_length=2100)
|
||||
clear_secret: bool = False
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def _clean_name(cls, value):
|
||||
return _channel_name(value)
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Small pieces shared by every "take over what was made by hand on a firewall" feature
|
||||
(rule_adopt_service.py for filter rules, object_adopt_service.py for aliases and NAT)."""
|
||||
|
||||
|
||||
class NotAdoptable(Exception):
|
||||
"""The object exists on the host but cannot be described without distortion.
|
||||
|
||||
`reasons` are machine-readable codes (the UI translates them); `details` maps a code to
|
||||
extra text the translation can show (e.g. the alias names a reference points to)."""
|
||||
|
||||
def __init__(self, reasons: list[str], details: dict[str, str] | None = None):
|
||||
super().__init__(", ".join(reasons))
|
||||
self.reasons = reasons
|
||||
self.details = details or {}
|
||||
|
||||
|
||||
class NotFoundOnHost(Exception):
|
||||
"""The UUID is not (or no longer) present in the host's live configuration."""
|
||||
|
||||
|
||||
def on(value) -> bool:
|
||||
"""OPNsense booleans arrive as "1"/"0" strings, sometimes as real booleans."""
|
||||
return value in ("1", 1, True)
|
||||
|
||||
|
||||
def clean(value) -> str:
|
||||
return "" if value is None else str(value).strip()
|
||||
@@ -90,7 +90,10 @@ _ROLLOUT_POST = re.compile(
|
||||
r"|deployments/[^/]+/(remove|retry-rollback)"
|
||||
r"|maintenance/(execute|carp-execute|schedule)"
|
||||
r"|network/changes"
|
||||
r"|services/[^/]+/(start|stop|restart))$"
|
||||
r"|services/[^/]+/(start|stop|restart)"
|
||||
# Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
|
||||
r"|rollouts/plans/[^/]+/(start|activate)"
|
||||
r"|rollouts/runs/[^/]+/(resume|approve|skip-host|abort))$"
|
||||
)
|
||||
_ROLLOUT_DELETE = re.compile(
|
||||
r"^/api/(rules|aliases|port-forward|one-to-one|outbound-nat|policy-hierarchy/rules|ipsec/tunnels|openvpn/instances"
|
||||
@@ -99,7 +102,13 @@ _ROLLOUT_DELETE = re.compile(
|
||||
)
|
||||
|
||||
|
||||
# Recurring maintenance windows decide WHEN unattended firmware actions run (1.25): every write counts.
|
||||
_ROLLOUT_ANY_WRITE = re.compile(r"^/api/maintenance-windows(/|$)")
|
||||
|
||||
|
||||
def is_rollout(method: str, path: str) -> bool:
|
||||
if method in ("POST", "PUT", "PATCH", "DELETE") and _ROLLOUT_ANY_WRITE.match(path):
|
||||
return True
|
||||
if method == "POST":
|
||||
return bool(_ROLLOUT_POST.search(path))
|
||||
return method == "DELETE" and bool(_ROLLOUT_DELETE.match(path))
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Gateway health of a firewall (dpinger numbers, as OPNsense reports them).
|
||||
|
||||
``get_gateway_health(host)`` is the one reusable entry point: a live read of
|
||||
/routes/gateway/status, one :class:`GatewayHealth` per gateway. The notification
|
||||
engine, the host page and the metrics read the SAME numbers from the periodic
|
||||
resource sample (``compact`` / ``from_compact``), so a scrape or a page view never
|
||||
fans out to the firewalls; the 1.25 health gate can call ``get_gateway_health``
|
||||
directly. Parsing is lenient on purpose: "~" means "nothing measured", never zero.
|
||||
"""
|
||||
import re
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.schemas.gateway import GatewayHealth, GatewayState
|
||||
from app.services import opnsense_client as client
|
||||
|
||||
_NUMBER = re.compile(r"-?\d+(?:\.\d+)?")
|
||||
_DEGRADED = {"delay", "loss", "delay+loss"}
|
||||
|
||||
|
||||
def _number(value) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
match = _NUMBER.search(str(value))
|
||||
return float(match.group()) if match else None
|
||||
|
||||
|
||||
def _state(status: str, measured: bool) -> GatewayState:
|
||||
if status == "down":
|
||||
return "offline"
|
||||
if status == "force_down":
|
||||
return "forced_down"
|
||||
if status in _DEGRADED:
|
||||
return "degraded"
|
||||
if status == "none":
|
||||
# "none" is also what a gateway without monitoring reports; only numbers make it "online".
|
||||
return "online" if measured else "unknown"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def parse_gateway_status(payload) -> list[GatewayHealth]:
|
||||
items = payload.get("items") if isinstance(payload, dict) else None
|
||||
result: list[GatewayHealth] = []
|
||||
for item in items or []:
|
||||
if not isinstance(item, dict) or not item.get("name"):
|
||||
continue
|
||||
status = str(item.get("status") or "").lower()
|
||||
loss, delay, stddev = _number(item.get("loss")), _number(item.get("delay")), _number(item.get("stddev"))
|
||||
if loss is not None and loss >= 100:
|
||||
delay = stddev = None # dpinger prints 0.0 ms when nothing answers: no round trip, not a fast one
|
||||
measured = loss is not None or delay is not None
|
||||
address = item.get("address")
|
||||
monitor = item.get("monitor")
|
||||
result.append(GatewayHealth(
|
||||
name=str(item["name"]),
|
||||
address=None if address in (None, "", "~") else str(address),
|
||||
status=status or "unknown",
|
||||
state=_state(status, measured),
|
||||
loss=loss, delay=delay, stddev=stddev,
|
||||
monitor=None if monitor in (None, "", "~") else str(monitor),
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
async def get_gateway_health(host: OpnsenseHost) -> list[GatewayHealth]:
|
||||
"""Live gateway state of ``host``. Raises OpnsenseAPIError if the firewall cannot be asked."""
|
||||
return parse_gateway_status(await client.get_gateway_status(host))
|
||||
|
||||
|
||||
def compact(gateways: list[GatewayHealth]) -> list[dict]:
|
||||
"""What the resource sampler stores per sample (small, no addresses)."""
|
||||
return [
|
||||
{"name": g.name, "status": g.status, "state": g.state, "loss": g.loss, "delay": g.delay, "stddev": g.stddev}
|
||||
for g in gateways
|
||||
]
|
||||
|
||||
|
||||
def from_compact(rows) -> list[GatewayHealth]:
|
||||
return [
|
||||
GatewayHealth(
|
||||
name=r["name"], status=r.get("status") or "unknown", state=r.get("state") or "unknown",
|
||||
loss=r.get("loss"), delay=r.get("delay"), stddev=r.get("stddev"),
|
||||
)
|
||||
for r in rows or [] if isinstance(r, dict) and r.get("name")
|
||||
]
|
||||
|
||||
|
||||
def is_bad(gateway: GatewayHealth, loss_threshold_pct: float = 0, delay_threshold_ms: float = 0) -> bool:
|
||||
"""Alarm-worthy: down, or (when the thresholds are on) measured loss / latency beyond them.
|
||||
A gateway forced down by the admin and one that is not measured are not alarms."""
|
||||
if gateway.state == "offline":
|
||||
return True
|
||||
if loss_threshold_pct and gateway.loss is not None and gateway.loss >= loss_threshold_pct:
|
||||
return True
|
||||
if delay_threshold_ms and gateway.delay is not None and gateway.delay >= delay_threshold_ms:
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -5,7 +5,7 @@ fan out live calls to every firewall (an offline host would stall it past the
|
||||
scraper's timeout and a 15 s scrape interval would hammer the fleet). The
|
||||
inputs are what OPNexus' own pollers already store:
|
||||
|
||||
* host reachability and resources — the 60 s resource sampler (a host that
|
||||
* host reachability, resources, gateway health and service state — the 60 s resource sampler (a host that
|
||||
produced no sample for ``STALE_AFTER`` is reported down),
|
||||
* CARP role — the latest observed transition (carp_events),
|
||||
* open problems — the notification engine's problem table,
|
||||
@@ -19,11 +19,11 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.models.carp_event import CarpEvent
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import NotificationProblem
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.routers.dashboard import collect_pending_actions
|
||||
from app.services import compliance_service
|
||||
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
||||
from app.version import APP_VERSION
|
||||
|
||||
CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
|
||||
@@ -99,6 +99,14 @@ async def render(db: AsyncSession) -> str:
|
||||
out.family("opnexus_host_memory_used_percent", "Memory in use, percent (last sample).")
|
||||
out.family("opnexus_host_load1", "1-minute load average (last sample).")
|
||||
out.family("opnexus_host_disk_used_percent", "Fullest disk, percent (last sample).")
|
||||
out.family("opnexus_host_swap_used_percent", "Swap in use, percent (last sample; absent without swap).")
|
||||
out.family("opnexus_host_gateway_up", "1 if the gateway is not down (online, degraded or not monitored), 0 if OPNsense reports it down.")
|
||||
out.family("opnexus_host_gateway_loss_percent", "Packet loss to the gateway monitor address, percent (last sample).")
|
||||
out.family("opnexus_host_gateway_delay_seconds", "Round-trip time to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_gateway_stddev_seconds", "Round-trip time deviation to the gateway monitor address (last sample).")
|
||||
out.family("opnexus_host_service_running", "1 if a watched service (Settings > Notifications) runs, 0 if it is stopped (last sample).")
|
||||
configured = (await notification_settings_service.get_settings(db)).service_watchlist
|
||||
watchlist = set(DEFAULT_SERVICE_WATCHLIST if configured is None else configured)
|
||||
out.family("opnexus_host_carp_master", "1 if the host's last observed CARP role is MASTER, 0 if BACKUP; absent if unknown.")
|
||||
for host in hosts:
|
||||
labels = {"host": host.name, "cluster": host.cluster_name or ""}
|
||||
@@ -115,6 +123,22 @@ async def render(db: AsyncSession) -> str:
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, value, **labels)
|
||||
if sample is not None:
|
||||
raw = sample.raw or {}
|
||||
if raw.get("swap_used_pct") is not None:
|
||||
out.sample("opnexus_host_swap_used_percent", raw["swap_used_pct"], **labels)
|
||||
for gw in gateway_health_service.from_compact(raw.get("gateways")):
|
||||
gw_labels = {**labels, "gateway": gw.name}
|
||||
out.sample("opnexus_host_gateway_up", 0 if gw.state == "offline" else 1, **gw_labels)
|
||||
for name, value, factor in (
|
||||
("opnexus_host_gateway_loss_percent", gw.loss, 1), ("opnexus_host_gateway_delay_seconds", gw.delay, 0.001),
|
||||
("opnexus_host_gateway_stddev_seconds", gw.stddev, 0.001),
|
||||
):
|
||||
if value is not None:
|
||||
out.sample(name, round(value * factor, 6), **gw_labels)
|
||||
for key, running in sorted((raw.get("services") or {}).items()):
|
||||
if key.split("/")[0] in watchlist:
|
||||
out.sample("opnexus_host_service_running", running, **labels, service=key.split("/")[0], instance=key.split("/", 1)[1] if "/" in key else None)
|
||||
if carp_state.get(host.id) in ("MASTER", "BACKUP"):
|
||||
out.sample("opnexus_host_carp_master", 1 if carp_state[host.id] == "MASTER" else 0, **labels)
|
||||
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
"""Monitoring alerts (1.24): gateway down, resource thresholds, stopped services.
|
||||
|
||||
Same contract as notification_checks.py (each check returns {dedup_key: context} of what is
|
||||
bad right now), with one addition: they never judge a single reading. The inputs are the
|
||||
60-second samples the resource sampler stores (resource_monitor_service, HostResourceSample),
|
||||
and a problem needs the last N samples to agree before it opens AND before it closes again:
|
||||
|
||||
* all N bad -> problem,
|
||||
* all N good -> no problem,
|
||||
* anything in between -> keep the current state (open stays open, closed stays closed).
|
||||
|
||||
So a single spike never alerts, a flapping link does not open and close the alert every few
|
||||
minutes, and a sample that could not be read ("unknown") neither raises nor clears anything.
|
||||
A host without a fresh sample is not judged at all (check_host_offline reports it); its open
|
||||
problems are kept.
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.notification import DEFAULT_SERVICE_WATCHLIST, NotificationProblem
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service, notification_settings_service
|
||||
from app.services.notification_dedup import Problems
|
||||
|
||||
# The sampler runs every 60 s; three missed rounds mean the host is not reachable (same limit as metrics_service).
|
||||
STALE_AFTER = timedelta(seconds=180)
|
||||
# How far back a service must have been seen running before its absence is reported (samples, ~2 h at 60 s).
|
||||
SERVICE_EVIDENCE_SAMPLES = 120
|
||||
_DISK_TYPES = {"ufs", "zfs"}
|
||||
|
||||
|
||||
def verdict(flags: list[bool | None], confirm: int, is_open: bool) -> bool:
|
||||
"""flags: newest first; True = bad, False = good, None = unknown."""
|
||||
window = flags[:confirm]
|
||||
if len(window) >= confirm:
|
||||
if all(f is True for f in window):
|
||||
return True
|
||||
if all(f is False for f in window):
|
||||
return False
|
||||
return is_open
|
||||
|
||||
|
||||
async def _enabled_hosts(db: AsyncSession) -> list[OpnsenseHost]:
|
||||
return list((await db.scalars(select(OpnsenseHost).where(OpnsenseHost.enabled.is_(True)))).all())
|
||||
|
||||
|
||||
async def _open_keys(db: AsyncSession, problem_type: str) -> set[str]:
|
||||
rows = await db.scalars(
|
||||
select(NotificationProblem.dedup_key).where(
|
||||
NotificationProblem.problem_type == problem_type, NotificationProblem.status == "open"
|
||||
)
|
||||
)
|
||||
return set(rows.all())
|
||||
|
||||
|
||||
async def _recent(db: AsyncSession, host_id, limit: int, *raw_keys: str) -> list[dict]:
|
||||
"""The newest ``limit`` samples of a host, newest first, with just the JSON keys asked for."""
|
||||
columns = [HostResourceSample.sampled_at, HostResourceSample.mem_used_pct, HostResourceSample.load_avg_1m]
|
||||
columns += [HostResourceSample.raw[key].as_json() for key in raw_keys]
|
||||
rows = (
|
||||
await db.execute(
|
||||
select(*columns).where(HostResourceSample.host_id == host_id).order_by(HostResourceSample.sampled_at.desc()).limit(limit)
|
||||
)
|
||||
).all()
|
||||
return [
|
||||
{"sampled_at": r[0], "mem_used_pct": r[1], "load_avg_1m": r[2], **{k: r[3 + i] for i, k in enumerate(raw_keys)}}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def _fresh(samples: list[dict]) -> bool:
|
||||
return bool(samples) and datetime.now(timezone.utc) - samples[0]["sampled_at"] <= STALE_AFTER
|
||||
|
||||
|
||||
def _keys_of_host(open_keys: set[str], host: OpnsenseHost) -> set[str]:
|
||||
prefix = f"{host.id}:"
|
||||
return {k[len(prefix):] for k in open_keys if k.startswith(prefix)}
|
||||
|
||||
|
||||
# ── Gateways ────────────────────────────────────────────────────────────────
|
||||
|
||||
def _gateway_reason(g) -> str:
|
||||
if g.state == "offline":
|
||||
return "Gateway offline"
|
||||
parts = []
|
||||
if g.loss is not None:
|
||||
parts.append(f"Paketverlust {g.loss:g} %")
|
||||
if g.delay is not None:
|
||||
parts.append(f"Latenz {g.delay:g} ms")
|
||||
return ", ".join(parts) or g.status
|
||||
|
||||
|
||||
async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
loss_limit, delay_limit = cfg.gateway_loss_threshold_pct, cfg.gateway_delay_threshold_ms
|
||||
open_keys = await _open_keys(db, "gateway_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "gateways")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
parsed = [
|
||||
None if s["gateways"] is None else {g.name: g for g in gateway_health_service.from_compact(s["gateways"])}
|
||||
for s in samples
|
||||
]
|
||||
names = {name for p in parsed if p for name in p} | _keys_of_host(open_keys, host)
|
||||
for name in sorted(names):
|
||||
flags: list[bool | None] = []
|
||||
for p in parsed:
|
||||
if p is None:
|
||||
flags.append(None)
|
||||
else:
|
||||
g = p.get(name) # a gateway that no longer exists is "fine": its problem resolves
|
||||
flags.append(False if g is None else gateway_health_service.is_bad(g, loss_limit, delay_limit))
|
||||
key = f"{host.id}:{name}"
|
||||
if not verdict(flags, confirm, key in open_keys):
|
||||
continue
|
||||
latest = next((p[name] for p in parsed if p and name in p), None)
|
||||
problems[key] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "gateway": name,
|
||||
"grund": _gateway_reason(latest) if latest else "keine aktuellen Messwerte",
|
||||
"status": latest.status if latest else None,
|
||||
"paketverlust_prozent": latest.loss if latest else None,
|
||||
"latenz_ms": latest.delay if latest else None,
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Resources ───────────────────────────────────────────────────────────────
|
||||
|
||||
def disk_groups(devices) -> dict[str, tuple[str, float]]:
|
||||
"""{group: (label, used %)} — one entry per ZFS pool / UFS device, the fullest dataset of a pool wins."""
|
||||
groups: dict[str, tuple[str, float]] = {}
|
||||
for dev in devices or []:
|
||||
if not isinstance(dev, dict) or dev.get("type") not in _DISK_TYPES:
|
||||
continue
|
||||
try:
|
||||
pct = float(dev["used_pct"])
|
||||
except (KeyError, TypeError, ValueError):
|
||||
continue
|
||||
device = str(dev.get("device") or dev.get("mountpoint") or "?")
|
||||
group = device.split("/")[0] if dev["type"] == "zfs" else device
|
||||
label = group if dev["type"] == "zfs" else str(dev.get("mountpoint") or device)
|
||||
if group not in groups or pct > groups[group][1]:
|
||||
groups[group] = (label, pct)
|
||||
return groups
|
||||
|
||||
|
||||
async def check_resource_threshold(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.resource_confirm_samples)
|
||||
open_keys = await _open_keys(db, "resource_threshold")
|
||||
problems = Problems()
|
||||
|
||||
def over(value, limit) -> bool | None:
|
||||
if not limit or value is None:
|
||||
return None if value is None else False
|
||||
return value >= limit
|
||||
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm, "disk", "swap_used_pct", "cpu_threads")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
base = {"host_id": str(host.id), "host_name": host.name}
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
|
||||
# Disk: one problem per pool/device.
|
||||
disk = [None if s["disk"] is None else disk_groups(s["disk"]) for s in samples]
|
||||
group_names = {g for d in disk if d for g in d} | {k[5:] for k in was_open if k.startswith("disk:")}
|
||||
for group in sorted(group_names):
|
||||
flags = [None if d is None else (over(d[group][1], cfg.disk_threshold_pct) if group in d else False) for d in disk]
|
||||
if verdict(flags, confirm, f"disk:{group}" in was_open):
|
||||
label, pct = next((d[group] for d in disk if d and group in d), (group, None))
|
||||
problems[f"{host.id}:disk:{group}"] = {
|
||||
**base, "ressource": f"Festplatte {label}", "wert_prozent": pct, "schwelle_prozent": cfg.disk_threshold_pct,
|
||||
}
|
||||
|
||||
def simple(key: str, label: str, flags: list, value, threshold, unit: str) -> None:
|
||||
if verdict(flags, confirm, key in was_open):
|
||||
problems[f"{host.id}:{key}"] = {**base, "ressource": label, "wert": f"{value}{unit}", "schwelle": f"{threshold}{unit}"}
|
||||
|
||||
latest = samples[0]
|
||||
simple("memory", "Arbeitsspeicher", [over(s["mem_used_pct"], cfg.memory_threshold_pct) for s in samples],
|
||||
latest["mem_used_pct"], cfg.memory_threshold_pct, " %")
|
||||
simple("swap", "Swap", [over(s["swap_used_pct"], cfg.swap_threshold_pct) for s in samples],
|
||||
latest["swap_used_pct"], cfg.swap_threshold_pct, " %")
|
||||
|
||||
def per_cpu(s):
|
||||
if s["load_avg_1m"] is None or not s["cpu_threads"]:
|
||||
return None # thread count unknown (older firmware): the load cannot be judged
|
||||
return s["load_avg_1m"] / s["cpu_threads"]
|
||||
|
||||
simple("cpu", "CPU-Last (pro Thread)", [over(per_cpu(s), cfg.load_per_cpu_threshold) for s in samples],
|
||||
None if per_cpu(latest) is None else round(per_cpu(latest), 2), cfg.load_per_cpu_threshold, "")
|
||||
return problems
|
||||
|
||||
|
||||
# ── Services ────────────────────────────────────────────────────────────────
|
||||
|
||||
async def check_service_down(db: AsyncSession) -> dict[str, dict]:
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
confirm = max(1, cfg.alert_confirm_samples)
|
||||
watch = set(cfg.service_watchlist if cfg.service_watchlist is not None else DEFAULT_SERVICE_WATCHLIST)
|
||||
open_keys = await _open_keys(db, "service_down")
|
||||
problems = Problems()
|
||||
for host in await _enabled_hosts(db):
|
||||
samples = await _recent(db, host.id, confirm + SERVICE_EVIDENCE_SAMPLES, "services")
|
||||
if not _fresh(samples):
|
||||
problems.preserve_prefixes.add(f"{host.id}:")
|
||||
continue
|
||||
listings = [s["services"] for s in samples]
|
||||
was_open = _keys_of_host(open_keys, host)
|
||||
candidates = {k for k in (listings[0] or {}) if k.split("/")[0] in watch} | was_open
|
||||
for svc in sorted(candidates):
|
||||
if svc.split("/")[0] not in watch:
|
||||
continue # taken off the watchlist: the problem resolves
|
||||
flags = [None if sv is None else (sv[svc] == 0 if svc in sv else False) for sv in listings]
|
||||
bad = verdict(flags, confirm, svc in was_open)
|
||||
# Only a service that was seen running before counts: one that never ran (enabled but idle,
|
||||
# e.g. DHCPv6 without a prefix) or was stopped on purpose long ago is not a failure.
|
||||
if bad and svc not in was_open and not any(sv and sv.get(svc) == 1 for sv in listings[confirm:]):
|
||||
bad = False
|
||||
if bad:
|
||||
problems[f"{host.id}:{svc}"] = {
|
||||
"host_id": str(host.id), "host_name": host.name, "dienst": svc.split("/")[0],
|
||||
"hinweis": "Dienst läuft nicht mehr; absichtlich gestoppt? Dann aus der Überwachungsliste nehmen.",
|
||||
}
|
||||
return problems
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Extra notification channels: ntfy, Gotify, Telegram, Microsoft Teams, Matrix.
|
||||
|
||||
They sit beside SMTP and the generic webhook (notification_delivery.py) and reuse its message
|
||||
text. Each channel is a row in ``notification_channels`` (see models/notification.py); the
|
||||
credential never touches the database: it is stored through notification_secret_store as
|
||||
``channel_<id>`` (mode 0600 file) and read only at send time. Where the credential is part of the
|
||||
URL (a Telegram bot token, the whole Teams webhook URL) the URL is built per request, and the
|
||||
HTTP request log is muted while it is in flight (``url_guard.muted``), so neither the logs nor an
|
||||
error message ever contain it.
|
||||
|
||||
API formats, checked against the vendors' documentation:
|
||||
|
||||
* ntfy — JSON publish: POST <server>/ with {"topic", "title", "message", "priority", "tags"}
|
||||
(docs.ntfy.sh/publish, "Publish as JSON"); optional ``Authorization: Bearer <token>``.
|
||||
* Gotify — POST <server>/message, header ``X-Gotify-Key: <app token>`` (keeps the token out of
|
||||
the URL), JSON {"title", "message", "priority"} (gotify.net/docs/pushmsg).
|
||||
* Telegram — POST <api>/bot<token>/sendMessage, JSON {"chat_id", "text"}; text limit 4096
|
||||
characters; answers {"ok": true, ...} (core.telegram.org/bots/api#sendmessage).
|
||||
* Teams — POST to the (Workflows / incoming) webhook URL with a message envelope carrying an
|
||||
Adaptive Card (learn.microsoft.com ... add-incoming-webhook); the URL is the secret.
|
||||
* Matrix — PUT <homeserver>/_matrix/client/v3/rooms/<roomId>/send/m.room.message/<txnId> with
|
||||
``Authorization: Bearer <access token>``, JSON {"msgtype": "m.text", "body"}; a fresh
|
||||
transaction id per message (Matrix client-server API, "Sending events to a room").
|
||||
"""
|
||||
import re
|
||||
import uuid
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import CHANNEL_TYPES, NotificationChannel, NotificationProblem
|
||||
from app.services import notification_delivery, notification_secret_store, url_guard
|
||||
from app.services.notification_delivery import NotificationDeliveryError
|
||||
|
||||
TELEGRAM_LIMIT = 4096
|
||||
TEAMS_LIMIT = 20000
|
||||
_NTFY_TOPIC = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||
_MATRIX_ROOM = re.compile(r"^![^\s:]+:[^\s/]+$")
|
||||
|
||||
# type -> (required config keys, secret required?)
|
||||
_SPEC = {
|
||||
"ntfy": ({"topic"}, False),
|
||||
"gotify": ({"server_url"}, True),
|
||||
"telegram": ({"chat_id"}, True),
|
||||
"teams": (set(), True),
|
||||
"matrix": ({"homeserver_url", "room_id"}, True),
|
||||
}
|
||||
assert set(_SPEC) == set(CHANNEL_TYPES)
|
||||
|
||||
|
||||
def secret_name(channel_id) -> str:
|
||||
return f"channel_{channel_id}"
|
||||
|
||||
|
||||
def clean_config(channel_type: str, config: dict) -> dict:
|
||||
"""Validates and normalises the non-secret settings of a channel. Raises ValueError (German reason)."""
|
||||
if channel_type not in _SPEC:
|
||||
raise ValueError("Unbekannter Kanaltyp")
|
||||
config = dict(config or {})
|
||||
out: dict = {}
|
||||
|
||||
def url(key: str, default: str | None = None) -> str | None:
|
||||
raw = str(config.get(key) or "").strip() or default
|
||||
if raw is None:
|
||||
return None
|
||||
return url_guard.validate_url(raw).rstrip("/")
|
||||
|
||||
def integer(key: str, low: int, high: int, default: int) -> int:
|
||||
value = config.get(key)
|
||||
if value in (None, ""):
|
||||
return default
|
||||
try:
|
||||
number = int(value)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError(f"{key}: ganze Zahl erwartet") from None
|
||||
if not low <= number <= high:
|
||||
raise ValueError(f"{key}: Wert zwischen {low} und {high} erwartet")
|
||||
return number
|
||||
|
||||
if channel_type == "ntfy":
|
||||
out["server_url"] = url("server_url", "https://ntfy.sh")
|
||||
topic = str(config.get("topic") or "").strip()
|
||||
if not _NTFY_TOPIC.match(topic):
|
||||
raise ValueError("Thema: 1–64 Zeichen aus Buchstaben, Ziffern, _ und -")
|
||||
out["topic"] = topic
|
||||
out["priority"] = integer("priority", 1, 5, 4)
|
||||
elif channel_type == "gotify":
|
||||
out["server_url"] = url("server_url")
|
||||
if not out["server_url"]:
|
||||
raise ValueError("Serveradresse fehlt")
|
||||
out["priority"] = integer("priority", 0, 10, 5)
|
||||
elif channel_type == "telegram":
|
||||
chat_id = str(config.get("chat_id") or "").strip()
|
||||
if not re.match(r"^(-?\d{1,20}|@[A-Za-z0-9_]{4,64})$", chat_id):
|
||||
raise ValueError("Chat-ID: Zahl (z. B. -1001234567890) oder @Kanalname")
|
||||
out["chat_id"] = chat_id
|
||||
out["api_base"] = url("api_base", "https://api.telegram.org")
|
||||
elif channel_type == "matrix":
|
||||
out["homeserver_url"] = url("homeserver_url")
|
||||
if not out["homeserver_url"]:
|
||||
raise ValueError("Homeserver-Adresse fehlt")
|
||||
room = str(config.get("room_id") or "").strip()
|
||||
if not _MATRIX_ROOM.match(room):
|
||||
raise ValueError("Raum-ID: Form !abc123:example.org (nicht der Alias #raum:…)")
|
||||
out["room_id"] = room
|
||||
return out
|
||||
|
||||
|
||||
def clean_secret(channel_type: str, secret: str) -> str:
|
||||
value = (secret or "").strip()
|
||||
if not value:
|
||||
raise ValueError("Zugangsdaten fehlen")
|
||||
if channel_type == "teams":
|
||||
value = url_guard.validate_url(value, max_length=2000)
|
||||
if not value.startswith("https://"):
|
||||
raise ValueError("Die Teams-Webhook-Adresse muss mit https:// beginnen")
|
||||
elif any(c.isspace() for c in value) or len(value) > 500:
|
||||
raise ValueError("Token: keine Leerzeichen, höchstens 500 Zeichen")
|
||||
return value
|
||||
|
||||
|
||||
def missing_requirements(channel: NotificationChannel) -> str | None:
|
||||
required, needs_secret = _SPEC[channel.channel_type]
|
||||
if any(not channel.config.get(k) for k in required):
|
||||
return "Konfiguration unvollständig"
|
||||
if needs_secret and not channel.secret_set:
|
||||
return "Zugangsdaten fehlen"
|
||||
return None
|
||||
|
||||
|
||||
# ── Message formatting (one function per channel, pure — unit-testable) ───────────
|
||||
|
||||
def _parts(problem: NotificationProblem, transition: str) -> tuple[str, str]:
|
||||
"""(title, details) — the title repeated at the top of render_message's body is dropped."""
|
||||
subject, body = notification_delivery.render_message(problem, transition)
|
||||
details = body.split("\n", 2)[2] if body.count("\n") >= 2 else ""
|
||||
return subject, details.strip()
|
||||
|
||||
|
||||
def _clip(text: str, limit: int) -> str:
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def format_ntfy(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
resolved = transition == "resolved"
|
||||
return {
|
||||
"topic": config["topic"], "title": title, "message": details or title,
|
||||
"priority": 3 if resolved else int(config.get("priority", 4)),
|
||||
"tags": ["white_check_mark"] if resolved else ["warning"],
|
||||
}
|
||||
|
||||
|
||||
def format_gotify(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {"title": title, "message": details or title, "priority": 1 if transition == "resolved" else int(config.get("priority", 5))}
|
||||
|
||||
|
||||
def format_telegram(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {
|
||||
"chat_id": config["chat_id"], "text": _clip(f"{title}\n\n{details}".strip(), TELEGRAM_LIMIT),
|
||||
"disable_web_page_preview": True,
|
||||
}
|
||||
|
||||
|
||||
def format_teams(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
body: list[dict] = [{"type": "TextBlock", "text": title, "weight": "Bolder", "size": "Medium", "wrap": True}]
|
||||
if details:
|
||||
body.append({"type": "TextBlock", "text": _clip(details, TEAMS_LIMIT), "wrap": True})
|
||||
return {
|
||||
"type": "message",
|
||||
"attachments": [{
|
||||
"contentType": "application/vnd.microsoft.card.adaptive", "contentUrl": None,
|
||||
"content": {
|
||||
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||
"type": "AdaptiveCard", "version": "1.4", "body": body,
|
||||
},
|
||||
}],
|
||||
}
|
||||
|
||||
|
||||
def format_matrix(config: dict, problem, transition: str) -> dict:
|
||||
title, details = _parts(problem, transition)
|
||||
return {"msgtype": "m.text", "body": f"{title}\n\n{details}".strip()}
|
||||
|
||||
|
||||
# ── Sending ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _request_for(channel: NotificationChannel, secret: str, problem, transition: str) -> tuple[str, str, dict, dict, str | None]:
|
||||
"""(method, url, headers, json body, expected "ok" check key)."""
|
||||
cfg, kind = channel.config, channel.channel_type
|
||||
if kind == "ntfy":
|
||||
headers = {"Authorization": f"Bearer {secret}"} if secret else {}
|
||||
return "POST", cfg["server_url"].rstrip("/") + "/", headers, format_ntfy(cfg, problem, transition), None
|
||||
if kind == "gotify":
|
||||
return "POST", cfg["server_url"].rstrip("/") + "/message", {"X-Gotify-Key": secret}, format_gotify(cfg, problem, transition), None
|
||||
if kind == "telegram":
|
||||
return "POST", f"{cfg['api_base'].rstrip('/')}/bot{secret}/sendMessage", {}, format_telegram(cfg, problem, transition), "ok"
|
||||
if kind == "teams":
|
||||
return "POST", secret, {}, format_teams(cfg, problem, transition), None
|
||||
room = quote(cfg["room_id"], safe="")
|
||||
url = f"{cfg['homeserver_url'].rstrip('/')}/_matrix/client/v3/rooms/{room}/send/m.room.message/{uuid.uuid4().hex}"
|
||||
return "PUT", url, {"Authorization": f"Bearer {secret}"}, format_matrix(cfg, problem, transition), None
|
||||
|
||||
|
||||
async def send(channel: NotificationChannel, problem: NotificationProblem, transition: str) -> None:
|
||||
problem_missing = missing_requirements(channel)
|
||||
if problem_missing:
|
||||
raise NotificationDeliveryError(f"Kanal „{channel.name}“: {problem_missing}")
|
||||
secret = notification_secret_store.read_secret(secret_name(channel.id)) or ""
|
||||
if channel.secret_set and not secret:
|
||||
raise NotificationDeliveryError(f"Kanal „{channel.name}“: Zugangsdaten nicht lesbar – bitte neu setzen")
|
||||
method, url, headers, payload, ok_key = _request_for(channel, secret, problem, transition)
|
||||
label = f"Kanal „{channel.name}“ ({channel.channel_type})"
|
||||
with url_guard.muted():
|
||||
try:
|
||||
url_guard.validate_url(url, max_length=2100)
|
||||
await url_guard.ensure_resolved_safe(url)
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
resp = await client.request(method, url, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
if ok_key and not (resp.json() or {}).get(ok_key):
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung abgelehnt")
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which for Telegram/Teams is the credential itself.
|
||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
||||
raise NotificationDeliveryError(f"{label}: Zustellung fehlgeschlagen ({reason})") from None
|
||||
except ValueError:
|
||||
raise NotificationDeliveryError(f"{label}: Antwort nicht lesbar") from None
|
||||
|
||||
|
||||
async def send_test(channel: NotificationChannel) -> None:
|
||||
fake = NotificationProblem(
|
||||
problem_type="test", dedup_key="test",
|
||||
context={"message": "Dies ist eine Testbenachrichtigung von OPNexus."},
|
||||
)
|
||||
await send(channel, fake, "opened")
|
||||
|
||||
|
||||
async def enabled_channels(db: AsyncSession) -> list[NotificationChannel]:
|
||||
rows = await db.scalars(select(NotificationChannel).where(NotificationChannel.enabled.is_(True)).order_by(NotificationChannel.name))
|
||||
return list(rows.all())
|
||||
@@ -363,3 +363,49 @@ async def check_compliance(db: AsyncSession) -> dict[str, dict]:
|
||||
"beispiele": ", ".join(params.get("examples", [])) or None,
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
# ── Rollouts (1.25) ─────────────────────────────────────────────────────────
|
||||
|
||||
def _short(detail) -> str | None:
|
||||
if not detail:
|
||||
return None
|
||||
parts = []
|
||||
for key, value in detail.items():
|
||||
if isinstance(value, list):
|
||||
value = ", ".join(str(v) for v in value[:8])
|
||||
parts.append(f"{key}: {value}")
|
||||
return "; ".join(parts)[:500]
|
||||
|
||||
|
||||
async def check_rollout_paused(db: AsyncSession) -> dict[str, dict]:
|
||||
"""Open while a rollout run stands still waiting for a human (gate, failed host, window gone...)."""
|
||||
from app.models.rollout import RolloutRun
|
||||
problems: dict[str, dict] = {}
|
||||
for run in (await db.scalars(select(RolloutRun).where(RolloutRun.status == "paused"))).all():
|
||||
wave = next((w.label for w in run.waves if w.index == run.current_wave), None)
|
||||
problems[str(run.id)] = {
|
||||
"run_id": str(run.id), "plan": run.plan_name, "grund": run.pause_reason, "welle": wave,
|
||||
"zielversion": run.target_version, "details": _short(run.pause_detail),
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
ROLLOUT_START_FAILURE_WINDOW = timedelta(days=3)
|
||||
|
||||
|
||||
async def check_rollout_start_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"""An automatic start at a window occurrence did not happen (event: closed without a 'resolved' notice)."""
|
||||
from app.models.rollout import RolloutPlan
|
||||
since = datetime.now(timezone.utc) - ROLLOUT_START_FAILURE_WINDOW
|
||||
problems: dict[str, dict] = {}
|
||||
for plan in (await db.scalars(select(RolloutPlan).where(RolloutPlan.auto_start_error.is_not(None)))).all():
|
||||
error = plan.auto_start_error or {}
|
||||
at = datetime.fromisoformat(error["at"]) if error.get("at") else None
|
||||
if at is None or at < since:
|
||||
continue
|
||||
problems[f"{plan.id}:{error.get('occurrence')}"] = {
|
||||
"plan_id": str(plan.id), "plan": plan.name, "grund": error.get("code"), "termin": error.get("occurrence"),
|
||||
"details": _short(error.get("params")),
|
||||
}
|
||||
return problems
|
||||
@@ -16,7 +16,7 @@ import httpx
|
||||
|
||||
from app.config import settings
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
|
||||
|
||||
class NotificationDeliveryError(Exception):
|
||||
@@ -37,6 +37,12 @@ _LABELS = {
|
||||
"backup_stale": "Kein aktuelles Backup",
|
||||
"backup_offsite_failed": "Offsite-Backup fehlgeschlagen",
|
||||
"compliance_finding": "Sicherheitsbefund",
|
||||
"gateway_down": "Gateway gestört",
|
||||
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
||||
"service_down": "Dienst ausgefallen",
|
||||
"rollout_paused": "Rollout angehalten",
|
||||
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
||||
"test": "Testbenachrichtigung",
|
||||
}
|
||||
|
||||
|
||||
@@ -84,9 +90,16 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
|
||||
headers["X-OPNexus-Signature"] = f"sha256={signature}"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout) as client:
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
url_guard.validate_url(settings_row.webhook_url)
|
||||
await url_guard.ensure_resolved_safe(settings_row.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
||||
try:
|
||||
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
with url_guard.muted():
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
# str(exc) names the full URL, which often is the credential itself
|
||||
# (token in path/query) — keep it out of messages and logs.
|
||||
|
||||
@@ -8,8 +8,8 @@ from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_checks, notification_dedup, notification_delivery
|
||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||
from app.services import monitor_checks, notification_channels, notification_checks, notification_dedup, notification_delivery
|
||||
from app.services import notification_settings_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -34,14 +34,21 @@ _CHECKS = [
|
||||
("backup_stale", "backup_failed_enabled", notification_checks.check_backup_stale),
|
||||
("backup_offsite_failed", "backup_failed_enabled", notification_checks.check_backup_offsite),
|
||||
("compliance_finding", "compliance_alerts_enabled", notification_checks.check_compliance),
|
||||
("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
|
||||
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
||||
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
||||
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
||||
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
||||
]
|
||||
|
||||
# Event-like problems: when they drop out of the window nothing was "fixed", so no
|
||||
# "behoben" message goes out for them.
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed"}
|
||||
_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
|
||||
|
||||
|
||||
async def _deliver(settings_row: NotificationSettings, problem: NotificationProblem, transition: str) -> None:
|
||||
async def _deliver(
|
||||
settings_row: NotificationSettings, problem: NotificationProblem, transition: str, channels: list[NotificationChannel] = (),
|
||||
) -> None:
|
||||
"""Best-effort, not an audit trail: attempted once per transition and
|
||||
never retried, even on failure — avoids retry-storming a broken webhook
|
||||
URL or SMTP relay every poll cycle."""
|
||||
@@ -60,6 +67,13 @@ async def _deliver(settings_row: NotificationSettings, problem: NotificationProb
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
logger.exception("Email delivery failed for %s/%s", problem.problem_type, problem.dedup_key)
|
||||
|
||||
for channel in channels:
|
||||
try:
|
||||
await notification_channels.send(channel, problem, transition)
|
||||
except notification_delivery.NotificationDeliveryError:
|
||||
# The error text names the channel and the HTTP status, never the URL or token.
|
||||
logger.exception("Channel %s (%s) delivery failed for %s/%s", channel.name, channel.channel_type, problem.problem_type, problem.dedup_key)
|
||||
|
||||
if transition == "resolved":
|
||||
problem.resolved_notified_at = _now()
|
||||
else:
|
||||
@@ -69,6 +83,7 @@ async def _deliver(settings_row: NotificationSettings, problem: NotificationProb
|
||||
async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
settings_row = await notification_settings_service.get_settings(db)
|
||||
result = {"opened": 0, "resolved": 0}
|
||||
channels = await notification_channels.enabled_channels(db)
|
||||
|
||||
for problem_type, enabled_flag, check_fn in _CHECKS:
|
||||
if not getattr(settings_row, enabled_flag):
|
||||
@@ -85,9 +100,9 @@ async def run_notification_checks(db: AsyncSession) -> dict:
|
||||
|
||||
opened, resolved = await notification_dedup.diff_and_update(db, problem_type, current)
|
||||
for problem in opened:
|
||||
await _deliver(settings_row, problem, "opened")
|
||||
await _deliver(settings_row, problem, "opened", channels)
|
||||
for problem in resolved:
|
||||
await _deliver(settings_row, problem, "resolved")
|
||||
await _deliver(settings_row, problem, "resolved", channels)
|
||||
result["opened"] += len(opened)
|
||||
result["resolved"] += len(resolved)
|
||||
|
||||
|
||||
@@ -26,6 +26,10 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
|
||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
|
||||
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
||||
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled",
|
||||
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
||||
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
||||
"service_watchlist",
|
||||
"poll_interval_seconds",
|
||||
]
|
||||
for field in simple_fields:
|
||||
|
||||
@@ -0,0 +1,545 @@
|
||||
"""Takes over aliases, port forwards, 1:1 NAT and outbound NAT rules that were created by hand on a
|
||||
firewall, so they can be edited, versioned and deployed from OPNexus like any other object. This is
|
||||
the generalisation of rule_adopt_service.py (filter rules keep their own module: their model has
|
||||
more special cases and the code is relied upon).
|
||||
|
||||
Adoption itself writes NOTHING to the firewall. It records the object as it is right now (read fresh
|
||||
from the host at adoption time, never from what the browser sent) together with its OPNsense UUID
|
||||
(`remote_uuid` = the deployment's opnsense_*_uuid), and stores the payload OPNexus would send for it as
|
||||
the deployment's last_payload. So the preview directly afterwards says "unchanged", a later deploy
|
||||
updates the same object via setItem/setRule (no duplicate), and the normal preview → push →
|
||||
confirm/auto-revert flow applies from the first change on.
|
||||
|
||||
Only objects OPNexus can describe without distortion are adopted. Anything the model cannot express
|
||||
is refused with machine-readable reasons (the UI translates them) instead of being imported in a way
|
||||
that would silently change its meaning on the next edit. Fields OPNexus does not manage (log flag of a
|
||||
port forward, pool options, tags, categories, ...) are never sent, and OPNsense's set*/setItem only
|
||||
overwrites the posted fields (ApiMutableModelControllerBase::setBase -> setNodes), so they stay as
|
||||
they are; the candidate list shows them as `kept`.
|
||||
|
||||
Evidence (OPNsense core master, src/opnsense/mvc/app/…, read 2026-10-20, and live against OPNsense
|
||||
26.7 on test-fw-3 on the same day with hand-made objects of every case below):
|
||||
|
||||
* Aliases — controllers/OPNsense/Firewall/Api/AliasController.php::searchItemAction -> searchBase
|
||||
("aliases.alias"), model models/OPNsense/Firewall/Alias.xml. Row keys used: uuid, name, type
|
||||
("host", "network", "port", "networkgroup", "mac", "asn", plus the unsupported "url", "urltable",
|
||||
"urljson", "geoip", "dynipv6host", "authgroup", and the system types "internal"/"external"),
|
||||
enabled "1"/"0", content (entries separated by "\n", live-verified; for "authgroup" the controller
|
||||
rewrites it to group names), description, proto (only meaningful for asn/geoip). The built-in rows
|
||||
are bogons, bogonsv6, sshlockout, virusprot (type external) and one __<if>_network per interface
|
||||
(type internal); they are OPNsense's own and never adoptable. A "networkgroup" lists alias NAMES;
|
||||
host/network/port aliases may also contain alias names (nesting).
|
||||
* Port forward — controllers/OPNsense/Firewall/Api/DNatController.php::searchRuleAction merges the
|
||||
configured rules (getFlatNodes: dotted keys "source.network", "source.port", "source.not",
|
||||
"destination.network", "destination.port", "destination.not", plus descr, disabled, interface,
|
||||
ipprotocol, protocol, target, local-port, natreflection, nordr, pass, log, poolopts, tag, tagged,
|
||||
nosync, categories, sequence) with synthetic automatic rows (uuid "automatic_*"/"lockout_*",
|
||||
is_automatic true) that setRule/delRule cannot target. models/OPNsense/Firewall/DNat.xml: interface is
|
||||
Multiple=Y (comma separated), ipprotocol may be blank/inet/inet6/inet46, protocol blank = any,
|
||||
`nordr` ("no redirect") requires an empty target/local-port (live: "Target must be empty when No
|
||||
RDR is set"), `pass` ("" manual / "pass" / "rule") links an associated filter rule, natreflection is
|
||||
""/purenat/disable. OPNexus sends pass "" and not "0", so a rule using one of those would change
|
||||
meaning on its first edit and is refused.
|
||||
* 1:1 NAT — OneToOneController::searchRuleAction (flat keys), model Filter.xml onetoone.rule:
|
||||
enabled, log, interface, type (binat/nat), source_net/source_not, destination_net/destination_not,
|
||||
external, natreflection ""/enable/disable, description. All of these are managed by OPNexus.
|
||||
* Outbound NAT — SourceNatController::searchRuleAction returns the configured rules ONLY when the
|
||||
global mode (filter.settings.nat.snat_mode) is hybrid or advanced; in automatic/disabled mode the
|
||||
manual rules do not show up at all, so adopting is refused up front (outbound_nat_mode.py) instead
|
||||
of presenting an empty list. Model Filter.xml snatrules.rule: `nonat` rules ("do not NAT") are not
|
||||
expressible and refused; staticnatport, poolopts, log, tag, tagged, nosync, endpoint-independent
|
||||
are left alone.
|
||||
"""
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Awaitable, Callable, get_args
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.alias import FirewallAlias
|
||||
from app.models.alias_deployment import AliasDeployment
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.one_to_one import OneToOneRule
|
||||
from app.models.one_to_one_deployment import OneToOneDeployment
|
||||
from app.models.outbound_nat import OutboundNatRule
|
||||
from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||
from app.models.port_forward import PortForwardRule
|
||||
from app.models.port_forward_deployment import PortForwardDeployment
|
||||
from app.schemas.alias import AliasType
|
||||
from app.services import alias_deploy_service, alias_policy_service
|
||||
from app.services import one_to_one_deploy_service, one_to_one_policy_service
|
||||
from app.services import opnsense_client, outbound_nat_deploy_service, outbound_nat_policy_service
|
||||
from app.services import port_forward_deploy_service, port_forward_policy_service
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on
|
||||
from app.services.nat_live_service import split_rows
|
||||
from app.services.outbound_nat_mode import host_mode
|
||||
|
||||
ADOPT_NOTE = "Von Firewall übernommen"
|
||||
ADOPT_MESSAGE = "Von der Firewall übernommen"
|
||||
|
||||
ALIAS_NAME = re.compile(r"^[A-Za-z0-9_]+$") # schemas/alias.py AliasCreate.name
|
||||
ALIAS_TYPES = frozenset(get_args(AliasType))
|
||||
ALIAS_CONTENT_MAX = 8000 # schemas/alias.py AliasCreate.content
|
||||
# OPNsense's own aliases that are not user objects (Firewall/Alias.php: bogons & co. are `external`
|
||||
# rows created by the system, __<if>_network are `internal`).
|
||||
SYSTEM_ALIAS_NAMES = frozenset({"bogons", "bogonsv6", "sshlockout", "virusprot"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entry:
|
||||
row: dict
|
||||
reasons: list[str] = field(default_factory=list)
|
||||
details: dict[str, str] = field(default_factory=dict)
|
||||
bulk: bool = False # not adoptable alone yet, but adoptable in the same run (its references come first)
|
||||
|
||||
@property
|
||||
def uuid(self) -> str:
|
||||
return clean(self.row.get("uuid"))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Kind:
|
||||
key: str
|
||||
model: type
|
||||
deployment: type
|
||||
fk: str # deployment column pointing at the central object
|
||||
uuid_attr: str # deployment column holding the OPNsense UUID (= remote_uuid)
|
||||
fetch: Callable[[OpnsenseHost], Awaitable[list[dict]]]
|
||||
check: Callable[[dict, dict], tuple[list[str], dict[str, str]]]
|
||||
title: Callable[[dict], str]
|
||||
info: Callable[[dict], dict[str, str]]
|
||||
build: Callable[[dict], dict]
|
||||
payload: Callable[[Any], dict]
|
||||
revision: Callable
|
||||
kept: Callable[[dict], list[str]] = lambda row: []
|
||||
context: Callable[[AsyncSession, list[dict]], Awaitable[dict]] | None = None
|
||||
refs: Callable[[dict], set[str]] | None = None
|
||||
name: Callable[[dict], str] | None = None
|
||||
|
||||
|
||||
# ── shared helpers ───────────────────────────────────────────────────────
|
||||
|
||||
def _single_interface(row: dict) -> bool:
|
||||
interface = clean(row.get("interface"))
|
||||
return bool(interface) and "," not in interface
|
||||
|
||||
|
||||
def _too_long(kind_model: type, values: dict) -> list[str]:
|
||||
"""Columns are bounded; a live value that does not fit would fail (or be cut) on the first edit."""
|
||||
columns = kind_model.__table__.columns
|
||||
return [name for name, value in values.items()
|
||||
if name in columns and getattr(columns[name].type, "length", None)
|
||||
and isinstance(value, str) and len(value) > columns[name].type.length]
|
||||
|
||||
|
||||
def _port(value) -> str | None:
|
||||
return clean(value) or None
|
||||
|
||||
|
||||
def _kept(row: dict, keys: tuple[str, ...]) -> list[str]:
|
||||
return [key for key in keys if clean(row.get(key)) not in ("", "0")]
|
||||
|
||||
|
||||
def _fetch_nat(search) -> Callable[[OpnsenseHost], Awaitable[list[dict]]]:
|
||||
async def fetch(host: OpnsenseHost) -> list[dict]:
|
||||
data = await search(host)
|
||||
# split_rows drops automatic_*/lockout_* and is_automatic rows: they are OPNsense's own
|
||||
return [r for r in split_rows(data)["rules"] if clean(r.get("uuid"))]
|
||||
return fetch
|
||||
|
||||
|
||||
# ── aliases ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _alias_content(row: dict) -> str:
|
||||
# same normalisation as AliasCreate.clean_content
|
||||
return "\n".join(line.strip() for line in str(row.get("content") or "").splitlines() if line.strip())
|
||||
|
||||
|
||||
def _alias_is_system(row: dict) -> bool:
|
||||
name = clean(row.get("name"))
|
||||
return clean(row.get("type")) == "internal" or name.startswith("__") or name in SYSTEM_ALIAS_NAMES
|
||||
|
||||
|
||||
async def _fetch_aliases(host: OpnsenseHost) -> list[dict]:
|
||||
data = await opnsense_client.search_aliases(host)
|
||||
rows = data.get("rows", []) if isinstance(data, dict) else []
|
||||
return [r for r in rows if isinstance(r, dict) and clean(r.get("uuid")) and clean(r.get("name"))]
|
||||
|
||||
|
||||
async def _alias_context(db: AsyncSession, rows: list[dict]) -> dict:
|
||||
return {"central_names": set((await db.scalars(select(FirewallAlias.name))).all())}
|
||||
|
||||
|
||||
def _alias_refs(row: dict) -> set[str]:
|
||||
return {line.lstrip("!").strip() for line in _alias_content(row).splitlines()} - {""}
|
||||
|
||||
|
||||
def _alias_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
if _alias_is_system(row):
|
||||
return ["system"], {}
|
||||
reasons: list[str] = []
|
||||
details: dict[str, str] = {}
|
||||
name, type_ = clean(row.get("name")), clean(row.get("type"))
|
||||
if type_ not in ALIAS_TYPES:
|
||||
reasons.append("unsupported_type")
|
||||
details["unsupported_type"] = clean(row.get("%type")) or type_
|
||||
if not ALIAS_NAME.fullmatch(name) or len(name) > 64:
|
||||
reasons.append("name_invalid")
|
||||
if name in ctx["central_names"]:
|
||||
reasons.append("name_exists")
|
||||
if len(_alias_content(row)) > ALIAS_CONTENT_MAX:
|
||||
reasons.append("content_too_large")
|
||||
if type_ == "asn" and clean(row.get("proto")):
|
||||
reasons.append("proto_set") # OPNexus has no field for it: a deploy elsewhere would drop the restriction
|
||||
if _too_long(FirewallAlias, {"description": clean(row.get("description"))}):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, details
|
||||
|
||||
|
||||
def _alias_info(row: dict) -> dict[str, str]:
|
||||
lines = _alias_content(row).splitlines()
|
||||
return {
|
||||
"type": clean(row.get("type")),
|
||||
"type_label": clean(row.get("%type")) or clean(row.get("type")),
|
||||
"description": clean(row.get("description")),
|
||||
"content": "\n".join(lines[:6]) + ("\n…" if len(lines) > 6 else ""),
|
||||
"entries": str(len(lines)),
|
||||
}
|
||||
|
||||
|
||||
def _alias_build(row: dict) -> dict:
|
||||
return {
|
||||
"name": clean(row.get("name")),
|
||||
"type": clean(row.get("type")),
|
||||
"content": _alias_content(row),
|
||||
"description": clean(row.get("description")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
}
|
||||
|
||||
|
||||
# ── port forward ─────────────────────────────────────────────────────────
|
||||
|
||||
def _pf_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
|
||||
reasons.append("ipprotocol_unsupported") # blank / inet46 have no counterpart in OPNexus
|
||||
if not clean(row.get("protocol")):
|
||||
reasons.append("protocol_any")
|
||||
if on(row.get("nordr")):
|
||||
reasons.append("no_redirect")
|
||||
elif not clean(row.get("target")):
|
||||
reasons.append("no_target")
|
||||
if clean(row.get("pass")):
|
||||
reasons.append("filter_rule_linked") # OPNexus sends pass "" (manual) and would cut the link
|
||||
if on(row.get("source.not")):
|
||||
reasons.append("negated_source")
|
||||
if on(row.get("destination.not")):
|
||||
reasons.append("negated_destination")
|
||||
if clean(row.get("natreflection")) not in ("", "purenat", "disable"):
|
||||
reasons.append("natreflection_unsupported")
|
||||
if not clean(row.get("descr")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(PortForwardRule, _pf_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _pf_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("descr")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"ipprotocol": clean(row.get("ipprotocol")),
|
||||
"protocol": clean(row.get("protocol")),
|
||||
"source_net": clean(row.get("source.network")) or "any",
|
||||
"source_port": _port(row.get("source.port")),
|
||||
"destination_net": clean(row.get("destination.network")) or "any",
|
||||
"destination_port": _port(row.get("destination.port")),
|
||||
"target": clean(row.get("target")),
|
||||
"local_port": _port(row.get("local-port")),
|
||||
"natreflection": clean(row.get("natreflection")),
|
||||
"enabled": not on(row.get("disabled")),
|
||||
}
|
||||
|
||||
|
||||
def _pf_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"protocol": clean(row.get("%protocol")) or clean(row.get("protocol")) or "any",
|
||||
"source": clean(row.get("source.network")) or "any",
|
||||
"source_port": clean(row.get("source.port")),
|
||||
"destination": clean(row.get("destination.network")) or "any",
|
||||
"destination_port": clean(row.get("destination.port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": clean(row.get("local-port")),
|
||||
}
|
||||
|
||||
|
||||
# ── 1:1 NAT ──────────────────────────────────────────────────────────────
|
||||
|
||||
def _o2o_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if clean(row.get("type")) not in ("binat", "nat"):
|
||||
reasons.append("type_unsupported")
|
||||
if clean(row.get("natreflection")) not in ("", "enable", "disable"):
|
||||
reasons.append("natreflection_unsupported")
|
||||
if not clean(row.get("source_net")) or not clean(row.get("external")):
|
||||
reasons.append("incomplete")
|
||||
if not clean(row.get("description")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(OneToOneRule, _o2o_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _o2o_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("description")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"source_net": clean(row.get("source_net")),
|
||||
"destination_net": clean(row.get("destination_net")) or "any",
|
||||
"source_not": on(row.get("source_not")),
|
||||
"destination_not": on(row.get("destination_not")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
"type": clean(row.get("type")),
|
||||
"external": clean(row.get("external")),
|
||||
"natreflection": clean(row.get("natreflection")),
|
||||
"log": on(row.get("log")),
|
||||
}
|
||||
|
||||
|
||||
def _o2o_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"type": clean(row.get("type")),
|
||||
"source": ("! " if on(row.get("source_not")) else "") + clean(row.get("source_net")),
|
||||
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
|
||||
"external": clean(row.get("external")),
|
||||
}
|
||||
|
||||
|
||||
# ── outbound NAT ─────────────────────────────────────────────────────────
|
||||
|
||||
async def _fetch_outbound(host: OpnsenseHost) -> list[dict]:
|
||||
# In automatic/disabled mode OPNsense does not list manual rules at all (see module docstring):
|
||||
# say so instead of showing an empty list.
|
||||
state = await host_mode(host)
|
||||
if not state["can_deploy"]:
|
||||
raise api_error(409, ErrorCode.OUTBOUND_MODE_BLOCKS_ADOPT, host=host.name, mode=state["mode"])
|
||||
return await _fetch_nat(opnsense_client.search_outbound_nat_rules)(host)
|
||||
|
||||
|
||||
def _snat_check(row: dict, ctx: dict) -> tuple[list[str], dict[str, str]]:
|
||||
reasons = []
|
||||
if not _single_interface(row):
|
||||
reasons.append("interface_not_single")
|
||||
if on(row.get("nonat")):
|
||||
reasons.append("no_nat_rule")
|
||||
if clean(row.get("ipprotocol")) not in ("inet", "inet6"):
|
||||
reasons.append("ipprotocol_unsupported")
|
||||
if not clean(row.get("protocol")):
|
||||
reasons.append("protocol_missing")
|
||||
if not clean(row.get("description")):
|
||||
reasons.append("no_description")
|
||||
if _too_long(OutboundNatRule, _snat_build(row)):
|
||||
reasons.append("value_too_long")
|
||||
return reasons, {}
|
||||
|
||||
|
||||
def _snat_build(row: dict) -> dict:
|
||||
return {
|
||||
"description": clean(row.get("description")),
|
||||
"interface": clean(row.get("interface")),
|
||||
"source_net": clean(row.get("source_net")) or "any",
|
||||
"destination_net": clean(row.get("destination_net")) or "any",
|
||||
"source_not": on(row.get("source_not")),
|
||||
"destination_not": on(row.get("destination_not")),
|
||||
"enabled": on(row.get("enabled")),
|
||||
"ipprotocol": clean(row.get("ipprotocol")),
|
||||
"protocol": clean(row.get("protocol")),
|
||||
"source_port": _port(row.get("source_port")),
|
||||
"destination_port": _port(row.get("destination_port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": _port(row.get("target_port")),
|
||||
}
|
||||
|
||||
|
||||
def _snat_info(row: dict) -> dict[str, str]:
|
||||
return {
|
||||
"interface_label": clean(row.get("%interface")) or clean(row.get("interface")),
|
||||
"protocol": clean(row.get("protocol")) or "any",
|
||||
"source": ("! " if on(row.get("source_not")) else "") + (clean(row.get("source_net")) or "any"),
|
||||
"source_port": clean(row.get("source_port")),
|
||||
"destination": ("! " if on(row.get("destination_not")) else "") + (clean(row.get("destination_net")) or "any"),
|
||||
"destination_port": clean(row.get("destination_port")),
|
||||
"target": clean(row.get("target")),
|
||||
"target_port": clean(row.get("target_port")),
|
||||
}
|
||||
|
||||
|
||||
KINDS: dict[str, Kind] = {
|
||||
"alias": Kind(
|
||||
key="alias", model=FirewallAlias, deployment=AliasDeployment, fk="alias_id",
|
||||
uuid_attr="opnsense_alias_uuid", fetch=_fetch_aliases, check=_alias_check,
|
||||
title=lambda row: clean(row.get("name")), info=_alias_info, build=_alias_build,
|
||||
payload=alias_deploy_service._build_payload, revision=alias_policy_service.revision,
|
||||
context=_alias_context, refs=_alias_refs, name=lambda row: clean(row.get("name")),
|
||||
),
|
||||
"port_forward": Kind(
|
||||
key="port_forward", model=PortForwardRule, deployment=PortForwardDeployment, fk="port_forward_id",
|
||||
uuid_attr="opnsense_port_forward_uuid", fetch=_fetch_nat(opnsense_client.search_port_forward_rules),
|
||||
check=_pf_check, title=lambda row: clean(row.get("descr")), info=_pf_info, build=_pf_build,
|
||||
payload=port_forward_deploy_service._build_payload, revision=port_forward_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("log", "poolopts", "tag", "tagged", "nosync", "categories")),
|
||||
),
|
||||
"one_to_one": Kind(
|
||||
key="one_to_one", model=OneToOneRule, deployment=OneToOneDeployment, fk="one_to_one_id",
|
||||
uuid_attr="opnsense_one_to_one_uuid", fetch=_fetch_nat(opnsense_client.search_one_to_one_rules),
|
||||
check=_o2o_check, title=lambda row: clean(row.get("description")), info=_o2o_info, build=_o2o_build,
|
||||
payload=one_to_one_deploy_service._build_payload, revision=one_to_one_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("categories",)),
|
||||
),
|
||||
"outbound_nat": Kind(
|
||||
key="outbound_nat", model=OutboundNatRule, deployment=OutboundNatDeployment, fk="outbound_nat_id",
|
||||
uuid_attr="opnsense_outbound_nat_uuid", fetch=_fetch_outbound,
|
||||
check=_snat_check, title=lambda row: clean(row.get("description")), info=_snat_info, build=_snat_build,
|
||||
payload=outbound_nat_deploy_service._build_payload, revision=outbound_nat_policy_service.revision,
|
||||
kept=lambda row: _kept(row, ("log", "staticnatport", "poolopts", "poolopts_sourcehashkey", "tag", "tagged",
|
||||
"nosync", "endpoint-independent", "categories")),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ── engine ───────────────────────────────────────────────────────────────
|
||||
|
||||
async def _managed_uuids(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> set[str]:
|
||||
column = getattr(kind.deployment, kind.uuid_attr)
|
||||
rows = await db.scalars(select(column).where(kind.deployment.host_id == host.id, column.is_not(None)))
|
||||
return set(rows.all())
|
||||
|
||||
|
||||
def _resolve_references(kind: Kind, entries: list[Entry], rows: list[dict], managed: set[str], ctx: dict) -> list[Entry]:
|
||||
"""Aliases may refer to other aliases by name. A reference to a live alias OPNexus does not manage
|
||||
(and that is not OPNsense's own) would be dangling in the central model, so the referring alias
|
||||
waits until its target is managed. Returns the entries in an order in which they can be adopted
|
||||
in ONE run (dependencies first); entries whose references can never be satisfied (target is a
|
||||
url alias, a user-defined external alias, ...) keep the reason `references_unmanaged_alias`."""
|
||||
live = {kind.name(r): r for r in rows}
|
||||
available = set(ctx["central_names"])
|
||||
available |= {n for n, r in live.items() if _alias_is_system(r) or clean(r.get("uuid")) in managed}
|
||||
|
||||
candidates = [e for e in entries if not e.reasons]
|
||||
for entry in candidates:
|
||||
missing = {ref for ref in kind.refs(entry.row) if ref in live and ref not in available}
|
||||
if missing:
|
||||
entry.reasons.append("references_unmanaged_alias")
|
||||
entry.details["references_unmanaged_alias"] = ", ".join(sorted(missing))
|
||||
|
||||
order: list[Entry] = []
|
||||
ready = set(available)
|
||||
progress = True
|
||||
while progress:
|
||||
progress = False
|
||||
for entry in candidates:
|
||||
if entry in order:
|
||||
continue
|
||||
if all(ref not in live or ref in ready for ref in kind.refs(entry.row)):
|
||||
order.append(entry)
|
||||
ready.add(kind.name(entry.row))
|
||||
progress = True
|
||||
for entry in order:
|
||||
entry.bulk = "references_unmanaged_alias" in entry.reasons
|
||||
return order
|
||||
|
||||
|
||||
async def _plan(db: AsyncSession, kind: Kind, host: OpnsenseHost) -> tuple[list[Entry], list[Entry]]:
|
||||
"""(all unmanaged entries with their reasons, the ones adoptable in one run in a safe order)."""
|
||||
rows = await kind.fetch(host)
|
||||
managed = await _managed_uuids(db, kind, host)
|
||||
ctx = await kind.context(db, rows) if kind.context else {}
|
||||
entries = []
|
||||
for row in rows:
|
||||
if clean(row.get("uuid")) in managed:
|
||||
continue
|
||||
reasons, details = kind.check(row, ctx)
|
||||
entries.append(Entry(row, list(reasons), dict(details)))
|
||||
if kind.refs:
|
||||
order = _resolve_references(kind, entries, rows, managed, ctx)
|
||||
else:
|
||||
order = [e for e in entries if not e.reasons]
|
||||
return entries, order
|
||||
|
||||
|
||||
def _describe(kind: Kind, entry: Entry) -> dict:
|
||||
return {
|
||||
"uuid": entry.uuid,
|
||||
"title": kind.title(entry.row),
|
||||
"enabled": on(entry.row.get("enabled")) if "disabled" not in entry.row else not on(entry.row.get("disabled")),
|
||||
"adoptable": not entry.reasons,
|
||||
"bulk_adoptable": not entry.reasons or entry.bulk,
|
||||
"reasons": entry.reasons,
|
||||
"reason_details": entry.details,
|
||||
"kept": kind.kept(entry.row),
|
||||
"info": kind.info(entry.row),
|
||||
}
|
||||
|
||||
|
||||
async def list_unmanaged(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> list[dict]:
|
||||
kind = KINDS[kind_key]
|
||||
entries, _ = await _plan(db, kind, host)
|
||||
return [_describe(kind, e) for e in entries]
|
||||
|
||||
|
||||
def _create(db: AsyncSession, kind: Kind, host: OpnsenseHost, entry: Entry):
|
||||
obj = kind.model(**kind.build(entry.row), targets=[host], groups=[])
|
||||
db.add(obj)
|
||||
return obj
|
||||
|
||||
|
||||
async def _record(db: AsyncSession, kind: Kind, host: OpnsenseHost, obj, remote_uuid: str) -> None:
|
||||
await db.flush()
|
||||
kind.revision(db, obj, ADOPT_NOTE)
|
||||
db.add(kind.deployment(**{
|
||||
kind.fk: obj.id, "host_id": host.id, "status": "success", "message": ADOPT_MESSAGE,
|
||||
kind.uuid_attr: remote_uuid, "last_payload": kind.payload(obj), "previous_payload": None,
|
||||
}))
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def adopt(db: AsyncSession, kind_key: str, host: OpnsenseHost, remote_uuid: str):
|
||||
kind = KINDS[kind_key]
|
||||
entries, _ = await _plan(db, kind, host)
|
||||
entry = next((e for e in entries if e.uuid == remote_uuid), None)
|
||||
if entry is None:
|
||||
# either gone from the host or already managed
|
||||
if remote_uuid in await _managed_uuids(db, kind, host):
|
||||
raise NotAdoptable(["already_managed"])
|
||||
raise NotFoundOnHost(remote_uuid)
|
||||
if entry.reasons:
|
||||
raise NotAdoptable(entry.reasons, entry.details)
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, remote_uuid)
|
||||
return obj
|
||||
|
||||
|
||||
async def adopt_all(db: AsyncSession, kind_key: str, host: OpnsenseHost) -> dict:
|
||||
"""Adopts everything adoptable in one transaction (the caller commits), nested aliases in
|
||||
dependency order. System objects are not reported; everything else that stays behind is, with reasons."""
|
||||
kind = KINDS[kind_key]
|
||||
entries, order = await _plan(db, kind, host)
|
||||
adopted = []
|
||||
for entry in order:
|
||||
obj = _create(db, kind, host, entry)
|
||||
await _record(db, kind, host, obj, entry.uuid)
|
||||
adopted.append({"uuid": entry.uuid, "title": kind.title(entry.row), "id": obj.id})
|
||||
done = {e.uuid for e in order}
|
||||
skipped = [
|
||||
{"uuid": e.uuid, "title": kind.title(e.row), "reasons": e.reasons, "reason_details": e.details}
|
||||
for e in entries if e.uuid not in done and e.reasons != ["system"]
|
||||
]
|
||||
return {"adopted": adopted, "skipped": skipped}
|
||||
@@ -145,6 +145,34 @@ async def get_interface_statistics(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/diagnostics/interface/getInterfaceStatistics")
|
||||
|
||||
|
||||
# Swap and CPU count, for the resource-threshold alert (1.24). Verified against core source
|
||||
# (Diagnostics/Api/SystemController::systemSwapAction -> configd "system show swapinfo") and live
|
||||
# (read-only GET, test-fw-3, OPNsense 26.7):
|
||||
# systemSwap -> {"swap": [{"device": "/dev/da0p3", "total": "8388608", "used": "0"}]} (KiB, strings;
|
||||
# an empty list when the firewall has no swap)
|
||||
# cpu_usage/getCPUType -> ["QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)"]
|
||||
async def get_system_swap(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/diagnostics/system/systemSwap")
|
||||
|
||||
|
||||
async def get_cpu_type(host: OpnsenseHost) -> Any:
|
||||
return await _request(host, "GET", "/diagnostics/cpu_usage/getCPUType")
|
||||
|
||||
|
||||
# Gateway health (read-only). Routes\Api\GatewayController::statusAction runs configd "interface gateways status"
|
||||
# -> scripts/routes/gateway_status.php, which merges the configured gateways with dpinger's live numbers
|
||||
# (src/etc/inc/plugins.inc.d/dpinger.inc, dpinger_status()). Live-verified on test-fw-3 (26.7):
|
||||
# {"items": [{"name": "WAN_GW", "address": "203.0.113.1", "status": "none", "loss": "~", "delay": "~",
|
||||
# "stddev": "~", "monitor": "~", "status_translated": "Online"}, ...], "status": "ok"}
|
||||
# "status" is the machine-readable verdict: none (fine), down, force_down (switched off by the admin), delay,
|
||||
# loss, delay+loss (thresholds of the gateway itself), anything else = pending. loss/delay/stddev are "12.3 %" /
|
||||
# "4.5 ms" strings, or "~" while nothing is measured (e.g. monitoring disabled on that gateway). "status" is
|
||||
# "failed" with no items when the firewall has no gateways. searchGateway carries the same numbers but with a
|
||||
# translated status text ("Online"), so it is not used here.
|
||||
async def get_gateway_status(host: OpnsenseHost) -> dict:
|
||||
return await _request(host, "GET", "/routes/gateway/status")
|
||||
|
||||
|
||||
async def toggle_carp_maintenance(host: OpnsenseHost) -> dict:
|
||||
"""Toggle persistent CARP maintenance mode.
|
||||
|
||||
|
||||
@@ -5,7 +5,9 @@ poll_all_hosts. v1 deliberately has no admin-editable poll interval (see
|
||||
config.py) — just a fixed constant, to keep this feature's first cut
|
||||
small.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
@@ -13,6 +15,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import gateway_health_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.opnsense_client import OpnsenseAPIError
|
||||
|
||||
@@ -55,11 +58,59 @@ def _disk_used_pct(disk: dict) -> float | None:
|
||||
return None
|
||||
|
||||
|
||||
def _swap_used_pct(swap) -> float | None:
|
||||
"""systemSwap: {"swap": [{"device", "total", "used"}]} in KiB; None when there is no swap at all."""
|
||||
try:
|
||||
rows = swap.get("swap") or []
|
||||
total = sum(int(r["total"]) for r in rows)
|
||||
used = sum(int(r["used"]) for r in rows)
|
||||
except (AttributeError, KeyError, TypeError, ValueError):
|
||||
return None
|
||||
return round(used / total * 100, 1) if total > 0 else None
|
||||
|
||||
|
||||
_CPU_THREADS = re.compile(r"(\d+)\s+threads?", re.IGNORECASE)
|
||||
_CPU_CORES = re.compile(r"(\d+)\s+cores?", re.IGNORECASE)
|
||||
|
||||
|
||||
def _cpu_threads(cpu_type) -> int | None:
|
||||
"""getCPUType: ["<model> (6 cores, 6 threads)"]."""
|
||||
text = " ".join(str(x) for x in cpu_type) if isinstance(cpu_type, list) else str(cpu_type or "")
|
||||
match = _CPU_THREADS.search(text) or _CPU_CORES.search(text)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def _services_compact(listing) -> dict[str, int]:
|
||||
"""{"unbound": 1, "openvpn/<uuid>": 0}: 1 = running. Only services an admin can stop (not 'locked')."""
|
||||
result: dict[str, int] = {}
|
||||
for row in (listing.get("rows") if isinstance(listing, dict) else None) or []:
|
||||
if not isinstance(row, dict) or not row.get("name") or str(row.get("locked")) in ("1", "True", "true"):
|
||||
continue
|
||||
name, id_ = str(row["name"]), str(row.get("id") or row["name"])
|
||||
result[name if id_ == name else f"{name}/{id_}"] = 1 if str(row.get("running")) in ("1", "True", "true") else 0
|
||||
return result
|
||||
|
||||
|
||||
async def _optional(operation):
|
||||
"""The extra readings must never cost the base sample (older firmware, an endpoint that is missing)."""
|
||||
try:
|
||||
return await operation
|
||||
except Exception as exc:
|
||||
logger.debug("Optionale Messung fehlgeschlagen: %s", type(exc).__name__)
|
||||
return None
|
||||
|
||||
|
||||
async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
resources = await client.get_system_resources(host)
|
||||
system_time = await client.get_system_time(host)
|
||||
disk = await client.get_system_disk(host)
|
||||
interfaces = await client.get_interface_statistics(host)
|
||||
swap, cpu_type, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_system_swap(host)),
|
||||
_optional(client.get_cpu_type(host)),
|
||||
_optional(gateway_health_service.get_gateway_health(host)),
|
||||
_optional(client.search_services(host)),
|
||||
)
|
||||
db.add(
|
||||
HostResourceSample(
|
||||
host_id=host.id,
|
||||
@@ -67,7 +118,14 @@ async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
mem_used_pct=_mem_used_pct(resources),
|
||||
load_avg_1m=_load_avg_1m(system_time),
|
||||
disk_used_pct=_disk_used_pct(disk),
|
||||
raw={"memory": resources.get("memory"), "disk": disk.get("devices"), "interfaces": interfaces.get("statistics")},
|
||||
raw={
|
||||
"memory": resources.get("memory"), "disk": disk.get("devices"), "interfaces": interfaces.get("statistics"),
|
||||
# None = not readable this round (the alert checks treat that as "unknown", not as "fine").
|
||||
"swap_used_pct": _swap_used_pct(swap) if swap is not None else None,
|
||||
"cpu_threads": _cpu_threads(cpu_type) if cpu_type is not None else None,
|
||||
"gateways": gateway_health_service.compact(gateways) if gateways is not None else None,
|
||||
"services": _services_compact(services) if services is not None else None,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,805 @@
|
||||
"""The rollout run engine (1.25): moves runs forward, one small step per poll tick.
|
||||
|
||||
Ground rules (an unattended engine that triggers firmware updates is the riskiest thing in OPNexus):
|
||||
|
||||
* It never calls a firmware action itself. Each host step is an ordinary ScheduledMaintenance
|
||||
(run_at=now, origin="rollout", auto_verify=True) — locks, backup before the update, CARP handover,
|
||||
audit and restart safety all come from that existing path.
|
||||
* Never twice: a host gets a schedule only while its ``scheduled_maintenance_id`` is empty, written
|
||||
in the same transaction as the schedule itself, under the run's row lock.
|
||||
* Never in parallel: one host at a time per run, and only one active run at all; nodes of a cluster
|
||||
go BACKUP first, the MASTER only with the wave's CARP handover, and before every node of an
|
||||
already touched cluster the cluster is checked again.
|
||||
* When in doubt, stop: anything unexpected pauses the run (``pause_reason``) and notifies; nothing
|
||||
is retried on its own. A paused run only continues when a human resumes it, and resuming first
|
||||
checks the health of everything done so far.
|
||||
* Every step re-checks that the person responsible for the run may still roll out.
|
||||
* Hard limits: waves/hosts (rollout_service), soak time, preflight/host timeouts, run age.
|
||||
|
||||
Concurrency: a tick (1) does DB-only bookkeeping under the run's row lock, (2) decides with the
|
||||
lock released (remote reads may take seconds), and (3) applies the decision only if the run's
|
||||
``revision`` is still the one it decided on — a pause/abort in between, or a second backend, wins.
|
||||
"""
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Awaitable, Callable
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.rollout import RolloutPlan, RolloutRun, RolloutRunHost, RolloutRunWave
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.models.user import AdminUser
|
||||
from app.services import audit_service, rollout_health, rollout_service, window_service
|
||||
from app.services import opnsense_client as client
|
||||
from app.services.host_maintenance_service import ensure_no_maintenance, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.policy_service import policy_lock
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ACTIVE = ("waiting_window", "preflight", "wave_running", "soaking", "gate", "awaiting_approval", "resuming", "paused")
|
||||
TERMINAL = ("succeeded", "cancelled")
|
||||
# Statuses in which the poll loop should tick quickly.
|
||||
BUSY = ("preflight", "wave_running", "gate", "resuming")
|
||||
HOST_OPEN = ("running", "verifying")
|
||||
|
||||
PREFLIGHT_CHECK_DELAY = timedelta(seconds=20) # after asking for an update check, before reading the result
|
||||
PREFLIGHT_TIMEOUT = timedelta(minutes=10)
|
||||
GATE_RECHECK = timedelta(minutes=2) # a failed gate is looked at once more (slow start after a reboot)
|
||||
BLOCKED_PATIENCE = timedelta(minutes=15) # someone else's maintenance in the cluster
|
||||
HOST_START_TIMEOUT = timedelta(minutes=10) # the schedule never fired
|
||||
HOST_MAX_DURATION = timedelta(hours=4) # backstop above the schedule's own timeouts
|
||||
MAX_RUN_AGE = timedelta(days=14)
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
return value.isoformat() if value else None
|
||||
|
||||
|
||||
def _ts(value: str | None) -> datetime | None:
|
||||
return datetime.fromisoformat(value) if value else None
|
||||
|
||||
|
||||
def _note(code: str, **params) -> dict:
|
||||
return {"code": code, "params": params}
|
||||
|
||||
|
||||
# ── small helpers on a loaded run ─────────────────────────────────────────────
|
||||
|
||||
def wave_row(run: RolloutRun, index: int) -> RolloutRunWave:
|
||||
return next(w for w in run.waves if w.index == index)
|
||||
|
||||
|
||||
def wave_spec(run: RolloutRun, index: int) -> dict:
|
||||
return run.plan_snapshot["waves"][index]
|
||||
|
||||
|
||||
def hosts_of(run: RolloutRun, index: int) -> list[RolloutRunHost]:
|
||||
return sorted((h for h in run.hosts if h.wave_index == index), key=lambda h: (h.order, h.host_name))
|
||||
|
||||
|
||||
def _named(run: RolloutRun, failed: list[str]) -> list[str]:
|
||||
""""<host-id>:<check>" -> "<host name>:<check>" for pause details people read."""
|
||||
names = {str(h.host_id): h.host_name for h in run.hosts if h.host_id}
|
||||
return [f"{names.get(item.split(':', 1)[0], item.split(':', 1)[0])}:{item.split(':', 1)[1]}" for item in failed]
|
||||
|
||||
|
||||
def _progress(run: RolloutRun, **changes) -> None:
|
||||
run.progress = {**(run.progress or {}), **changes}
|
||||
|
||||
|
||||
async def _audit(run: RolloutRun, action: str, status: str, message: str | None = None, **details) -> None:
|
||||
token = audit_service.context.set({"actor": f"{run.started_by} (Rollout)"[:150], "id": str(run.id)})
|
||||
try:
|
||||
await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id),
|
||||
entity_name=run.plan_name, message=message,
|
||||
details={"wave": run.current_wave, **details})
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _requeue(host: RolloutRunHost, code: str) -> None:
|
||||
"""Back to the queue: only for a host whose schedule provably never fired (cancelled under its row lock,
|
||||
or refused by fire_due as overdue). Clearing the schedule id lets a resumed run plan it once more."""
|
||||
host.status, host.scheduled_maintenance_id, host.started_at, host.detail = "queued", None, None, _note(code)
|
||||
|
||||
|
||||
async def requeue_if_pending(db: AsyncSession, host: RolloutRunHost) -> bool:
|
||||
"""Cancel a host's schedule that has not fired yet and put the host back in the queue. Locked like the
|
||||
router's abort: fire_due holds the row while firing, and a running action is never touched."""
|
||||
if host.status not in HOST_OPEN or host.scheduled_maintenance_id is None:
|
||||
return False
|
||||
sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == host.scheduled_maintenance_id)
|
||||
.with_for_update().execution_options(populate_existing=True))
|
||||
if sched is None or sched.status != "pending":
|
||||
return False
|
||||
sched.status = "cancelled"
|
||||
sched.result = {"error": "Rollout angehalten, nicht ausgeführt", "not_fired": True}
|
||||
_requeue(host, "not_fired")
|
||||
return True
|
||||
|
||||
|
||||
def pause(run: RolloutRun, reason: str, **detail) -> None:
|
||||
if run.status != "paused":
|
||||
run.paused_from = run.status
|
||||
run.status = "paused"
|
||||
run.pause_reason = reason
|
||||
run.pause_detail = detail
|
||||
_progress(run, paused_at=_iso(_now()))
|
||||
|
||||
|
||||
async def may_roll_out(db: AsyncSession, username: str) -> bool:
|
||||
"""Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout."""
|
||||
user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
|
||||
if user is None:
|
||||
return False
|
||||
if user.role == "admin":
|
||||
return True
|
||||
if user.role == "operator":
|
||||
from app.routers import governance
|
||||
return (await governance.get_settings(db)).operators_may_deploy
|
||||
return False
|
||||
|
||||
|
||||
async def _window(db: AsyncSession, run: RolloutRun) -> tuple[bool, MaintenanceWindow | None]:
|
||||
"""(window configured, window). Configured but gone means: stop, never 'no time limit'."""
|
||||
window_id = run.plan_snapshot.get("window_id")
|
||||
if not window_id:
|
||||
return False, None
|
||||
return True, await db.get(MaintenanceWindow, window_id)
|
||||
|
||||
|
||||
async def _may_start_now(db: AsyncSession, run: RolloutRun) -> str | None:
|
||||
"""None if new host work may start now, else the reason ('window_closed' / 'window_missing')."""
|
||||
configured, window = await _window(db, run)
|
||||
if not configured:
|
||||
return None
|
||||
if window is None:
|
||||
return "window_missing"
|
||||
return None if window_service.startable(window, _now()) else "window_closed"
|
||||
|
||||
|
||||
async def _lock(db: AsyncSession, run_id, revision: int | None = None) -> RolloutRun | None:
|
||||
stmt = select(RolloutRun).where(RolloutRun.id == run_id)
|
||||
if revision is not None:
|
||||
stmt = stmt.where(RolloutRun.revision == revision)
|
||||
return await db.scalar(stmt.with_for_update().execution_options(populate_existing=True))
|
||||
|
||||
|
||||
# ── creating a run ────────────────────────────────────────────────────────────
|
||||
|
||||
async def active_run(db: AsyncSession) -> RolloutRun | None:
|
||||
return await db.scalar(select(RolloutRun).where(RolloutRun.status.in_(ACTIVE)).order_by(RolloutRun.created_at).limit(1))
|
||||
|
||||
|
||||
async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
|
||||
occurrence: datetime | None = None) -> RolloutRun:
|
||||
"""Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
|
||||
now = _now()
|
||||
snapshot = {
|
||||
"plan_id": str(plan.id), "name": plan.name, "action": plan.action,
|
||||
"window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
|
||||
}
|
||||
run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
|
||||
current_wave=0, started_by=started_by, trigger=trigger, window_occurrence=occurrence,
|
||||
progress={}, created_at=now, step_at=now, revision=0)
|
||||
run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
|
||||
for w in resolved["waves"]]
|
||||
run.hosts = [RolloutRunHost(wave_index=w["index"], host_id=uuid.UUID(h["id"]), host_name=h["name"], cluster_name=h["cluster_name"],
|
||||
order=i, status="queued")
|
||||
for w in resolved["waves"] for i, h in enumerate(w["hosts"])]
|
||||
db.add(run)
|
||||
await db.flush()
|
||||
return run
|
||||
|
||||
|
||||
# ── the tick ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def advance_runs(db: AsyncSession) -> int:
|
||||
"""One step for every run. Returns how many runs want a quick next tick."""
|
||||
try:
|
||||
await auto_start(db)
|
||||
except Exception:
|
||||
logger.exception("Rollout: automatischer Start fehlgeschlagen")
|
||||
await db.rollback()
|
||||
ids = list((await db.scalars(select(RolloutRun.id).where(
|
||||
(RolloutRun.status.in_(ACTIVE)) | RolloutRun.hosts.any(RolloutRunHost.status.in_(HOST_OPEN))))).all())
|
||||
await db.commit()
|
||||
busy = 0
|
||||
for run_id in ids:
|
||||
try:
|
||||
await tick(run_id)
|
||||
except Exception as exc:
|
||||
logger.exception("Rollout %s: Schritt fehlgeschlagen", run_id)
|
||||
async with SessionLocal() as s:
|
||||
run = await _lock(s, run_id)
|
||||
if run is not None and run.status in ACTIVE and run.status != "paused":
|
||||
pause(run, "internal_error", error=type(exc).__name__)
|
||||
run.revision += 1
|
||||
await s.commit()
|
||||
await _audit(run, "pause", "paused", "Interner Fehler: Rollout angehalten", reason="internal_error")
|
||||
async with SessionLocal() as s:
|
||||
status = await s.scalar(select(RolloutRun.status).where(RolloutRun.id == run_id))
|
||||
busy += status in BUSY
|
||||
return busy
|
||||
|
||||
|
||||
async def tick(run_id) -> None:
|
||||
async with SessionLocal() as db:
|
||||
# (1) bookkeeping under the lock: what did the schedules do?
|
||||
run = await _lock(db, run_id)
|
||||
if run is None:
|
||||
return
|
||||
changed = await _sync_hosts(db, run)
|
||||
if (run.status in ACTIVE and run.status != "paused" and _now() - run.created_at > MAX_RUN_AGE
|
||||
and not any(h.status in HOST_OPEN for h in run.hosts)):
|
||||
pause(run, "run_too_old", days=MAX_RUN_AGE.days)
|
||||
changed = True
|
||||
if changed:
|
||||
run.revision += 1
|
||||
await db.commit()
|
||||
if changed and run.status == "paused" and run.pause_reason in ("host_failed", "host_timeout", "host_not_started", "run_too_old"):
|
||||
await _audit(run, "pause", "paused", reason=run.pause_reason, detail=run.pause_detail)
|
||||
if run.status not in ACTIVE or run.status in ("paused", "awaiting_approval"):
|
||||
return
|
||||
revision = run.revision
|
||||
# (2) decide, without holding the lock
|
||||
decision = await _HANDLERS[run.status](db, run)
|
||||
await db.commit()
|
||||
if decision is None:
|
||||
return
|
||||
# (3) apply only on the revision it was decided on
|
||||
run = await _lock(db, run_id, revision)
|
||||
if run is None:
|
||||
await db.commit()
|
||||
return
|
||||
events = await decision(db, run)
|
||||
run.revision += 1
|
||||
run.step_at = _now()
|
||||
await db.commit()
|
||||
for event in events or []:
|
||||
await _audit(run, *event[:2], **(event[2] if len(event) > 2 else {}))
|
||||
|
||||
|
||||
Decision = Callable[[AsyncSession, RolloutRun], Awaitable[list | None]]
|
||||
|
||||
|
||||
async def _sync_hosts(db: AsyncSession, run: RolloutRun) -> bool:
|
||||
"""Carry the outcome of each host's schedule over to the run. Returns True if anything changed."""
|
||||
changed = False
|
||||
now = _now()
|
||||
for host in run.hosts:
|
||||
if host.status not in HOST_OPEN:
|
||||
continue
|
||||
sched = await db.get(ScheduledMaintenance, host.scheduled_maintenance_id, populate_existing=True) \
|
||||
if host.scheduled_maintenance_id else None
|
||||
problem = None
|
||||
if sched is None:
|
||||
host.status, host.detail, host.finished_at = "failed", _note("schedule_missing"), now
|
||||
problem = ("host_failed", {"host": host.host_name, "error": "schedule_missing"})
|
||||
elif sched.status == "pending":
|
||||
if host.started_at and now - host.started_at > HOST_START_TIMEOUT:
|
||||
# Never leave it pending behind a paused run: it would fire whenever the poll loop comes back.
|
||||
if await requeue_if_pending(db, host):
|
||||
problem = ("host_not_started", {"host": host.host_name})
|
||||
changed = True
|
||||
elif (sched.result or {}).get("not_fired"):
|
||||
_requeue(host, "not_fired") # fire_due refused it as overdue: nothing happened on the firewall
|
||||
problem = ("host_not_started", {"host": host.host_name})
|
||||
changed = True
|
||||
elif sched.status == "running":
|
||||
want = "verifying" if sched.phase == "verify" else "running"
|
||||
if host.status != want:
|
||||
host.status = want
|
||||
changed = True
|
||||
if host.started_at and now - host.started_at > HOST_MAX_DURATION:
|
||||
problem = ("host_timeout", {"host": host.host_name})
|
||||
elif sched.status == "succeeded":
|
||||
result = sched.result or {}
|
||||
host.status, host.finished_at = "succeeded", now
|
||||
host.version_after = result.get("version_after")
|
||||
host.detail = _note("verified", version=result.get("version_after"), rebooted=bool(result.get("rebooted")))
|
||||
changed = True
|
||||
else: # failed / cancelled
|
||||
error = (sched.result or {}).get("error") or sched.status
|
||||
host.status, host.finished_at = "failed", now
|
||||
host.detail = _note("host_failed", error=error[:300])
|
||||
problem = ("host_failed", {"host": host.host_name, "error": error[:300]})
|
||||
if problem and run.status in ACTIVE and run.status != "paused":
|
||||
pause(run, problem[0], **problem[1])
|
||||
changed = True
|
||||
elif problem and host.status == "failed":
|
||||
changed = True
|
||||
return changed
|
||||
|
||||
|
||||
# ── handlers: one per run status; each returns an apply-function or None ──────
|
||||
|
||||
async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if await may_roll_out(db, run.started_by):
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
pause(run, "actor_invalid", user=run.started_by)
|
||||
return [("pause", "paused", {"reason": "actor_invalid"})]
|
||||
return apply
|
||||
|
||||
|
||||
def _to_waiting(reason: str) -> Decision:
|
||||
async def apply(db, run):
|
||||
if reason == "window_missing":
|
||||
pause(run, "window_missing")
|
||||
return [("pause", "paused", {"reason": reason})]
|
||||
run.paused_from = run.status
|
||||
run.status = "waiting_window"
|
||||
return [("waiting_window", "waiting")]
|
||||
return apply
|
||||
|
||||
|
||||
async def _handle_waiting_window(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
reason = await _may_start_now(db, run)
|
||||
if reason == "window_missing":
|
||||
return _to_waiting(reason)
|
||||
if reason:
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
_progress(run, resume_recheck_at=None, resume_attempts=0)
|
||||
run.status = "resuming" # the gate of everything done so far is checked again first
|
||||
return [("window_open", "resuming")]
|
||||
return apply
|
||||
|
||||
|
||||
async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
recheck = _ts((run.progress or {}).get("resume_recheck_at"))
|
||||
if recheck and _now() < recheck:
|
||||
return None
|
||||
done = [h for h in run.hosts if h.status == "succeeded" and h.host_id]
|
||||
gate = {"hosts": {}, "verdict": "ok"}
|
||||
if done:
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
baselines = {}
|
||||
for w in run.waves:
|
||||
baselines.update((w.baseline or {}))
|
||||
gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
|
||||
async def apply(db, run):
|
||||
if gate["verdict"] == "fail":
|
||||
attempts = (run.progress or {}).get("resume_attempts", 0) + 1
|
||||
_progress(run, resume_gate=gate, resume_attempts=attempts)
|
||||
if attempts < 2:
|
||||
_progress(run, resume_recheck_at=_iso(_now() + GATE_RECHECK))
|
||||
return [("resume_gate", "recheck", {"failed": rollout_health.failed_checks(gate)})]
|
||||
run.paused_from = run.paused_from or "preflight"
|
||||
_progress(run, resume_recheck_at=None)
|
||||
run.status = "paused"
|
||||
run.pause_reason, run.pause_detail = "gate_failed", {"failed": _named(run, rollout_health.failed_checks(gate)), "on_resume": True}
|
||||
return [("resume_gate", "failed", {"failed": rollout_health.failed_checks(gate)})]
|
||||
_progress(run, resume_gate=gate, resume_recheck_at=None, resume_attempts=0)
|
||||
target = run.paused_from if run.paused_from in ("preflight", "wave_running", "soaking", "gate") else "preflight"
|
||||
if target == "gate":
|
||||
wave_row(run, run.current_wave).gate_attempts = 0
|
||||
_progress(run, gate_recheck_at=None)
|
||||
if target == "preflight":
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None)
|
||||
run.status, run.paused_from, run.pause_reason, run.pause_detail = target, None, None, None
|
||||
return [("resume_gate", "passed", {"continue_with": target})]
|
||||
return apply
|
||||
|
||||
|
||||
def target_of(status: dict, action: str) -> str | None:
|
||||
"""The version a firmware action will install, from core/firmware/status (read after a fresh check).
|
||||
|
||||
Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
|
||||
``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
|
||||
core package's new version comes first; product_latest is only a fallback."""
|
||||
if action == "major_update":
|
||||
return status.get("upgrade_major_version") or None
|
||||
packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
|
||||
core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
|
||||
if core and core.get("new") and core.get("new") != "N/A":
|
||||
return str(core["new"])
|
||||
for row in status.get("upgrade_packages") or []:
|
||||
if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
|
||||
return str(row["new_version"])
|
||||
product = status.get("product") if isinstance(status.get("product"), dict) else {}
|
||||
return product.get("product_latest") or status.get("product_latest") or None
|
||||
|
||||
|
||||
async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if problem := await _actor_problem(db, run):
|
||||
return problem
|
||||
reason = await _may_start_now(db, run)
|
||||
if reason:
|
||||
return _to_waiting(reason)
|
||||
index = run.current_wave
|
||||
prog = run.progress or {}
|
||||
started = _ts(prog.get("preflight_started")) or _now()
|
||||
requested = _ts(prog.get("preflight_requested_at"))
|
||||
candidates = [h for h in hosts_of(run, index) if h.status in ("queued", "blocked")]
|
||||
now = _now()
|
||||
|
||||
# Which hosts are usable at all?
|
||||
usable: dict = {}
|
||||
skips: dict = {}
|
||||
for rh in candidates:
|
||||
host = await db.get(OpnsenseHost, rh.host_id) if rh.host_id else None
|
||||
if host is None:
|
||||
skips[rh.id] = _note("host_deleted")
|
||||
elif not host.enabled:
|
||||
skips[rh.id] = _note("host_disabled")
|
||||
else:
|
||||
usable[rh.id] = host
|
||||
|
||||
if requested is None:
|
||||
# Step 1: ask every usable host to look for updates (a firmware *check*, changes nothing installed).
|
||||
asked = []
|
||||
for rh_id, host in usable.items():
|
||||
try:
|
||||
await ensure_no_maintenance(db, [host])
|
||||
if (await client.get_firmware_running(host)).get("status") == "ready":
|
||||
await client.check_firmware_updates(host)
|
||||
asked.append(str(rh_id))
|
||||
except (client.OpnsenseAPIError, HTTPException):
|
||||
pass # judged in step 2 (and blocked after the timeout)
|
||||
|
||||
async def apply(db, run):
|
||||
_progress(run, preflight_started=_iso(started), preflight_requested_at=_iso(_now()), preflight_asked=asked)
|
||||
w = wave_row(run, index)
|
||||
w.status = "preflight"
|
||||
w.started_at = w.started_at or _now()
|
||||
return [("preflight", "started", {"hosts": len(usable)})]
|
||||
return apply
|
||||
if now - requested < PREFLIGHT_CHECK_DELAY:
|
||||
return None
|
||||
|
||||
# Step 2: read what each host offers.
|
||||
offers: dict = {}
|
||||
waiting: dict = {}
|
||||
for rh_id, host in usable.items():
|
||||
try:
|
||||
await ensure_no_maintenance(db, [host])
|
||||
except HTTPException:
|
||||
waiting[rh_id] = _note("locked")
|
||||
continue
|
||||
try:
|
||||
running = (await client.get_firmware_running(host)).get("status")
|
||||
if running != "ready":
|
||||
waiting[rh_id] = _note("firmware_busy")
|
||||
continue
|
||||
status = await client.get_firmware_status(host)
|
||||
except client.OpnsenseAPIError:
|
||||
waiting[rh_id] = _note("offline")
|
||||
continue
|
||||
fw = status.get("status")
|
||||
wanted = "upgrade" if run.action == "major_update" else "update"
|
||||
if fw == wanted:
|
||||
offers[rh_id] = target_of(status, run.action)
|
||||
elif fw in ("none", "update", "upgrade"):
|
||||
skips[rh_id] = _note("no_update_offered", status=fw)
|
||||
else:
|
||||
waiting[rh_id] = _note("firmware_status", status=str(fw)[:40])
|
||||
|
||||
if waiting:
|
||||
if now - started < PREFLIGHT_TIMEOUT:
|
||||
return None
|
||||
|
||||
async def apply_blocked(db, run):
|
||||
names = []
|
||||
for rh in hosts_of(run, index):
|
||||
if rh.id in waiting:
|
||||
rh.status, rh.detail = "blocked", waiting[rh.id]
|
||||
names.append(rh.host_name)
|
||||
pause(run, "preflight_blocked", hosts=", ".join(names))
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None)
|
||||
return [("pause", "paused", {"reason": "preflight_blocked", "hosts": names})]
|
||||
return apply_blocked
|
||||
|
||||
targets = {t for t in offers.values()}
|
||||
expected = run.target_version
|
||||
mismatch = None
|
||||
if None in targets:
|
||||
mismatch = {"offered": "?"}
|
||||
elif expected is None and len(targets) > 1:
|
||||
mismatch = {"offered": ", ".join(sorted(targets))}
|
||||
elif expected is not None and targets - {expected}:
|
||||
mismatch = {"expected": expected, "offered": ", ".join(sorted(targets - {expected}))}
|
||||
if mismatch:
|
||||
async def apply_mismatch(db, run):
|
||||
# OPNsense always installs the newest offered version; a later ring must not get one the earlier ring never ran.
|
||||
pause(run, "target_version_changed", **mismatch)
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None)
|
||||
return [("pause", "paused", {"reason": "target_version_changed", **mismatch})]
|
||||
return apply_mismatch
|
||||
|
||||
# Baselines and order (BACKUP before MASTER inside a cluster).
|
||||
baselines = {}
|
||||
versions = {}
|
||||
for rh_id in offers:
|
||||
host = usable[rh_id]
|
||||
baselines[str(host.id)] = await rollout_health.baseline(db, host)
|
||||
versions[rh_id] = baselines[str(host.id)].get("version")
|
||||
roles = await get_statuses([usable[i] for i in offers])
|
||||
rank = {"BACKUP": 0, None: 1, "MASTER": 2}
|
||||
|
||||
def order_key(rh_id):
|
||||
host = usable[rh_id]
|
||||
role = roles.get(host.id, {}).get("carp_status") if host.cluster_name else None
|
||||
return (1 if host.cluster_name else 0, host.cluster_name or "", rank.get(role, 1), host.name)
|
||||
ordered = sorted(offers, key=order_key)
|
||||
target = next(iter(targets)) if targets else expected
|
||||
|
||||
async def apply_ready(db, run):
|
||||
for rh in hosts_of(run, index):
|
||||
if rh.id in skips:
|
||||
rh.status, rh.detail = "skipped", skips[rh.id]
|
||||
elif rh.id in offers:
|
||||
rh.status, rh.detail = "queued", None
|
||||
rh.order = ordered.index(rh.id)
|
||||
rh.version_before = versions.get(rh.id)
|
||||
w = wave_row(run, index)
|
||||
w.baseline = baselines
|
||||
if run.target_version is None and target:
|
||||
run.target_version = target
|
||||
_progress(run, preflight_started=None, preflight_requested_at=None, blocked_since=None, cluster_recheck_at=None)
|
||||
if not offers:
|
||||
w.status, w.finished_at = "skipped", _now()
|
||||
events = [("wave", "skipped", {"wave": index})]
|
||||
events += _next_wave(run)
|
||||
return events
|
||||
w.status = "running"
|
||||
run.status = "wave_running"
|
||||
return [("wave", "started", {"wave": index, "target_version": run.target_version,
|
||||
"hosts": [usable[i].name for i in ordered], "skipped": len(skips)})]
|
||||
return apply_ready
|
||||
|
||||
|
||||
def _next_wave(run: RolloutRun) -> list:
|
||||
index = run.current_wave + 1
|
||||
if index >= len(run.waves):
|
||||
run.status, run.finished_at = "succeeded", _now()
|
||||
return [("finish", "succeeded")]
|
||||
run.current_wave = index
|
||||
w = wave_row(run, index)
|
||||
if wave_spec(run, index).get("require_approval"):
|
||||
w.status = run.status = "awaiting_approval"
|
||||
return [("wave", "awaiting_approval", {"wave": index})]
|
||||
w.status, run.status = "preflight", "preflight"
|
||||
return []
|
||||
|
||||
|
||||
async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
if any(h.status in HOST_OPEN for h in run.hosts):
|
||||
return None # strictly one host at a time
|
||||
index = run.current_wave
|
||||
queued = [h for h in hosts_of(run, index) if h.status == "queued"]
|
||||
if not queued:
|
||||
soak = int(wave_spec(run, index).get("soak_minutes") or 0)
|
||||
|
||||
async def apply_soak(db, run):
|
||||
w = wave_row(run, index)
|
||||
w.status, w.soak_until = "soaking", _now() + timedelta(minutes=soak)
|
||||
run.status = "soaking"
|
||||
return [("wave", "soaking", {"wave": index, "minutes": soak})]
|
||||
return apply_soak
|
||||
if problem := await _actor_problem(db, run):
|
||||
return problem
|
||||
reason = await _may_start_now(db, run)
|
||||
if reason:
|
||||
return _to_waiting(reason)
|
||||
nxt = queued[0]
|
||||
host = await db.get(OpnsenseHost, nxt.host_id) if nxt.host_id else None
|
||||
if host is None or not host.enabled:
|
||||
async def apply_skip(db, run):
|
||||
rh = next(h for h in run.hosts if h.id == nxt.id)
|
||||
rh.status, rh.detail = "skipped", _note("host_deleted" if host is None else "host_disabled")
|
||||
return [("host", "skipped", {"host": nxt.host_name})]
|
||||
return apply_skip
|
||||
spec = wave_spec(run, index)
|
||||
prog = run.progress or {}
|
||||
|
||||
if host.cluster_name:
|
||||
fleet = await peers(db, host)
|
||||
statuses = await get_statuses(fleet)
|
||||
done_ids = {h.host_id for h in run.hosts if h.status == "succeeded"}
|
||||
done_peers = [p for p in fleet if p.id in done_ids and p.id != host.id]
|
||||
baselines = {}
|
||||
for w in run.waves:
|
||||
baselines.update(w.baseline or {})
|
||||
# The cluster as a whole must be healthy before a node is touched; nodes done earlier in this run
|
||||
# must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
|
||||
gate = await rollout_health.evaluate(
|
||||
db, done_peers + [host], baselines, target_version=run.target_version,
|
||||
checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
|
||||
# The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
|
||||
failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
|
||||
if r["result"] == "fail" and not (hid == str(host.id) and c in ("version", "gateways", "services"))]
|
||||
if failed:
|
||||
recheck = _ts(prog.get("cluster_recheck_at"))
|
||||
if recheck is None:
|
||||
async def apply_recheck(db, run):
|
||||
_progress(run, cluster_recheck_at=_iso(_now() + GATE_RECHECK), cluster_gate=gate)
|
||||
return [("cluster_gate", "recheck", {"host": host.name, "failed": failed})]
|
||||
return apply_recheck
|
||||
if _now() < recheck:
|
||||
return None
|
||||
|
||||
async def apply_cluster_fail(db, run):
|
||||
pause(run, "cluster_gate_failed", cluster=host.cluster_name, host=host.name, failed=_named(run, failed))
|
||||
_progress(run, cluster_recheck_at=None, cluster_gate=gate)
|
||||
return [("cluster_gate", "failed", {"host": host.name, "failed": failed})]
|
||||
return apply_cluster_fail
|
||||
if statuses.get(host.id, {}).get("carp_status") == "MASTER" and not spec.get("carp_handover"):
|
||||
async def apply_master(db, run):
|
||||
pause(run, "master_needs_handover", host=host.name, cluster=host.cluster_name)
|
||||
return [("pause", "paused", {"reason": "master_needs_handover", "host": host.name})]
|
||||
return apply_master
|
||||
|
||||
try:
|
||||
await ensure_no_maintenance(db, [host])
|
||||
except HTTPException:
|
||||
since = _ts(prog.get("blocked_since"))
|
||||
if since is None:
|
||||
async def apply_blocked_start(db, run):
|
||||
_progress(run, blocked_since=_iso(_now()))
|
||||
return [("host", "waiting_for_lock", {"host": host.name})]
|
||||
return apply_blocked_start
|
||||
if _now() - since < BLOCKED_PATIENCE:
|
||||
return None
|
||||
|
||||
async def apply_blocked(db, run):
|
||||
pause(run, "blocked_by_maintenance", host=host.name)
|
||||
_progress(run, blocked_since=None)
|
||||
return [("pause", "paused", {"reason": "blocked_by_maintenance", "host": host.name})]
|
||||
return apply_blocked
|
||||
|
||||
async def apply_start(db, run):
|
||||
await policy_lock(db)
|
||||
rh = next(h for h in run.hosts if h.id == nxt.id)
|
||||
# Idempotency guard: a host that already has a schedule never gets a second one.
|
||||
if rh.status != "queued" or rh.scheduled_maintenance_id is not None:
|
||||
return []
|
||||
now = _now()
|
||||
sched = ScheduledMaintenance(
|
||||
host_id=host.id, host_name=host.name, action=run.action, run_at=now, status="pending",
|
||||
actor=run.started_by, created_at=now, carp_handover=bool(spec.get("carp_handover") and host.cluster_name),
|
||||
auto_verify=True, origin="rollout",
|
||||
progress={"rollout": {"run_id": str(run.id), "wave": index, "plan": run.plan_name}},
|
||||
)
|
||||
db.add(sched)
|
||||
await db.flush()
|
||||
rh.scheduled_maintenance_id, rh.status, rh.started_at = sched.id, "running", now
|
||||
_progress(run, blocked_since=None, cluster_recheck_at=None)
|
||||
return [("host", "started", {"host": host.name, "schedule_id": str(sched.id), "handover": sched.carp_handover})]
|
||||
return apply_start
|
||||
|
||||
|
||||
async def _handle_soaking(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
w = wave_row(run, run.current_wave)
|
||||
if w.soak_until and _now() < w.soak_until:
|
||||
return None
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, run.current_wave)
|
||||
w.status, w.gate_attempts = "gate", 0
|
||||
run.status = "gate"
|
||||
_progress(run, gate_recheck_at=None)
|
||||
return []
|
||||
return apply
|
||||
|
||||
|
||||
async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
|
||||
recheck = _ts((run.progress or {}).get("gate_recheck_at"))
|
||||
if recheck and _now() < recheck:
|
||||
return None
|
||||
index = run.current_wave
|
||||
w = wave_row(run, index)
|
||||
done = [h for h in hosts_of(run, index) if h.status == "succeeded" and h.host_id]
|
||||
hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
|
||||
gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
|
||||
checks=run.plan_snapshot.get("gate_checks"),
|
||||
compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
|
||||
|
||||
async def apply(db, run):
|
||||
w = wave_row(run, index)
|
||||
w.gate_result = gate
|
||||
w.gate_attempts = (w.gate_attempts or 0) + 1
|
||||
failed = rollout_health.failed_checks(gate)
|
||||
events = [("gate", gate["verdict"], {"wave": index, "attempt": w.gate_attempts, "failed": failed, "result": gate})]
|
||||
if gate["verdict"] == "fail":
|
||||
if w.gate_attempts < 2:
|
||||
_progress(run, gate_recheck_at=_iso(_now() + GATE_RECHECK))
|
||||
return events
|
||||
_progress(run, gate_recheck_at=None)
|
||||
pause(run, "gate_failed", wave=w.label, failed=_named(run, failed))
|
||||
return events + [("pause", "paused", {"reason": "gate_failed"})]
|
||||
w.status, w.finished_at = "passed", _now()
|
||||
_progress(run, gate_recheck_at=None)
|
||||
return events + _next_wave(run)
|
||||
return apply
|
||||
|
||||
|
||||
_HANDLERS = {
|
||||
"waiting_window": _handle_waiting_window,
|
||||
"resuming": _handle_resuming,
|
||||
"preflight": _handle_preflight,
|
||||
"wave_running": _handle_wave_running,
|
||||
"soaking": _handle_soaking,
|
||||
"gate": _handle_gate,
|
||||
}
|
||||
|
||||
|
||||
# ── automatic start at a window occurrence ────────────────────────────────────
|
||||
|
||||
async def auto_start(db: AsyncSession) -> int:
|
||||
started = 0
|
||||
plans = (await db.scalars(select(RolloutPlan).where(
|
||||
RolloutPlan.auto_start.is_(True), RolloutPlan.activated_at.is_not(None), RolloutPlan.window_id.is_not(None)))).all()
|
||||
for plan in plans:
|
||||
window = await db.get(MaintenanceWindow, plan.window_id)
|
||||
occ = window_service.startable(window, _now()) if window else None
|
||||
if occ is None:
|
||||
continue
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
|
||||
continue
|
||||
if (plan.auto_start_error or {}).get("occurrence") == _iso(occ.start):
|
||||
continue # already reported for this occurrence
|
||||
await policy_lock(db)
|
||||
plan = await rollout_service.load_plan(db, plan.id)
|
||||
if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
|
||||
continue # another backend started it while this one waited for the lock (not a refusal)
|
||||
resolved = await rollout_service.resolve(db, plan)
|
||||
problem = None
|
||||
if plan.action != "update":
|
||||
problem = _note("auto_start_update_only")
|
||||
elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
|
||||
problem = _note("plan_changed") # someone changed the plan, its groups or its window since activation
|
||||
elif not await may_roll_out(db, plan.activated_by or ""):
|
||||
problem = _note("actor_invalid", user=plan.activated_by)
|
||||
elif resolved["errors"]:
|
||||
problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
|
||||
elif (other := await active_run(db)) is not None:
|
||||
problem = _note("other_run_active", plan=other.plan_name)
|
||||
if problem:
|
||||
plan.auto_start_error = {"occurrence": _iso(occ.start), "at": _iso(_now()), **problem}
|
||||
await db.commit()
|
||||
token = audit_service.context.set({"actor": "system:rollout", "id": str(plan.id)})
|
||||
try:
|
||||
await audit_service.persist(kind="rollout", action="auto_start", status="refused", entity_id=str(plan.id),
|
||||
entity_name=plan.name, details=plan.auto_start_error)
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
continue
|
||||
try:
|
||||
run = await create_run(db, plan, resolved, started_by=plan.activated_by, trigger="window", occurrence=occ.start)
|
||||
plan.auto_start_error = None
|
||||
await db.commit()
|
||||
except IntegrityError:
|
||||
await db.rollback() # another backend started it for this occurrence
|
||||
continue
|
||||
started += 1
|
||||
await _audit(run, "start", "started", trigger="window", occurrence=_iso(occ.start),
|
||||
hosts=sum(len(w["hosts"]) for w in resolved["waves"]))
|
||||
await db.commit()
|
||||
return started
|
||||
|
||||
|
||||
# ── human actions (called by the router under policy_lock) ────────────────────
|
||||
|
||||
async def count_active(db: AsyncSession) -> int:
|
||||
return await db.scalar(select(func.count()).select_from(RolloutRun).where(RolloutRun.status.in_(ACTIVE))) or 0
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Health gate of a firmware rollout (1.25): is a host, after its update, as healthy as before?
|
||||
|
||||
``baseline(db, host)`` is taken right before a wave starts, ``evaluate(...)`` after the soak time
|
||||
(and before each node of an already touched cluster, and on resume). Every check of every host
|
||||
yields ``{"result": ok|warn|fail|skip, "code": <machine code>, "params": {...}}``; the UI translates
|
||||
the code (rollouts.gate.code.*), so no German prose travels through here.
|
||||
|
||||
What the checks look at (all read-only on the firewall, except that the compliance check downloads
|
||||
the configuration like the 6-hourly security check does):
|
||||
|
||||
* online — enabled, the API answers, firmware job idle (``running == ready``).
|
||||
* version — the installed version is the rollout's target version.
|
||||
* carp — per cluster: every node online, exactly one MASTER per VHID, no node in CARP
|
||||
maintenance mode, no CARP auto-revert timer running.
|
||||
* gateways — gateways that were up in the baseline are up again; a gateway that was already bad
|
||||
before only warns (1.24 ``get_gateway_health``).
|
||||
* services — services that ran in the baseline run again.
|
||||
* locks — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
|
||||
* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
|
||||
update may legitimately change defaults.
|
||||
|
||||
A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
|
||||
unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
|
||||
"""
|
||||
import asyncio
|
||||
from collections import defaultdict
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.carp_maintenance_window import CarpMaintenanceWindow
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import compliance_service, gateway_health_service, opnsense_client as client
|
||||
from app.services.host_maintenance_service import _shared_change_blockers, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
from app.services.resource_monitor_service import _services_compact
|
||||
|
||||
CHECKS = ("online", "version", "carp", "gateways", "services", "locks", "compliance")
|
||||
DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": True, "services": True, "locks": True,
|
||||
"compliance": False}
|
||||
ALWAYS_ON = ("online", "version", "locks") # a rollout without these would be blind
|
||||
_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
|
||||
|
||||
|
||||
def _r(result: str, code: str, **params) -> dict:
|
||||
return {"result": result, "code": code, "params": params}
|
||||
|
||||
|
||||
async def _optional(coro):
|
||||
try:
|
||||
return await coro
|
||||
except client.OpnsenseAPIError:
|
||||
return None
|
||||
|
||||
|
||||
async def _gateways(host) -> dict | None:
|
||||
gws = await _optional(gateway_health_service.get_gateway_health(host))
|
||||
return None if gws is None else {g.name: g.state for g in gws}
|
||||
|
||||
|
||||
async def _services(host) -> dict | None:
|
||||
listing = await _optional(client.search_services(host))
|
||||
return None if listing is None else _services_compact(listing)
|
||||
|
||||
|
||||
async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
|
||||
info, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_firmware_info(host)), _gateways(host), _services(host))
|
||||
score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
|
||||
return {
|
||||
"version": (info or {}).get("product_version"),
|
||||
"gateways": gateways,
|
||||
"services": services,
|
||||
"compliance_score": score,
|
||||
}
|
||||
|
||||
|
||||
def _check_gateways(before: dict | None, now: dict | None) -> dict:
|
||||
if now is None:
|
||||
return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
|
||||
if before is None:
|
||||
bad = sorted(n for n, s in now.items() if s == "offline")
|
||||
return _r("warn", "gateway_down", names=", ".join(bad)) if bad else _r("ok", "no_baseline")
|
||||
down = sorted(n for n, s in before.items() if s in ("online", "unknown") and now.get(n) == "offline")
|
||||
gone = sorted(n for n in before if n not in now)
|
||||
degraded = sorted(n for n, s in now.items() if s == "degraded" and before.get(n) != "degraded")
|
||||
still_bad = sorted(n for n, s in before.items() if s in ("offline", "degraded") and now.get(n) in ("offline", "degraded"))
|
||||
# "unknown" (not monitored) going to "offline" is a real change too: monitoring just started and reports down.
|
||||
if down:
|
||||
return _r("fail", "gateway_down", names=", ".join(down))
|
||||
if gone:
|
||||
return _r("warn", "gateway_missing", names=", ".join(gone))
|
||||
if degraded:
|
||||
return _r("warn", "gateway_degraded", names=", ".join(degraded))
|
||||
if still_bad:
|
||||
return _r("warn", "gateway_bad_before", names=", ".join(still_bad))
|
||||
return _r("ok", "ok")
|
||||
|
||||
|
||||
def _check_services(before: dict | None, now: dict | None) -> dict:
|
||||
if now is None:
|
||||
return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
|
||||
if before is None:
|
||||
return _r("ok", "no_baseline")
|
||||
stopped = sorted(name.split("/")[0] for name, running in before.items() if running == 1 and now.get(name) != 1)
|
||||
if stopped:
|
||||
return _r("fail", "service_stopped", names=", ".join(dict.fromkeys(stopped)))
|
||||
return _r("ok", "ok")
|
||||
|
||||
|
||||
async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dict) -> dict:
|
||||
offline = sorted(h.name for h in fleet if h.enabled and statuses.get(h.id, {}).get("status") != "online")
|
||||
if offline:
|
||||
return _r("fail", "node_offline", names=", ".join(offline))
|
||||
online = [h for h in fleet if statuses.get(h.id, {}).get("status") == "online"]
|
||||
demoted = sorted(h.name for h in online if statuses[h.id].get("carp_maintenance_mode"))
|
||||
if demoted:
|
||||
return _r("fail", "maintenance_mode", names=", ".join(demoted))
|
||||
timers = (await db.scalars(select(CarpMaintenanceWindow.host_name).where(
|
||||
CarpMaintenanceWindow.host_id.in_([h.id for h in fleet]), CarpMaintenanceWindow.status == "active"))).all()
|
||||
if timers:
|
||||
return _r("fail", "auto_revert_active", names=", ".join(sorted(timers)))
|
||||
masters: dict[str, int] = defaultdict(int)
|
||||
broken: set[str] = set()
|
||||
for h in online:
|
||||
for vhid, state in (statuses[h.id].get("carp_vips") or {}).items():
|
||||
masters[vhid] += state == "MASTER"
|
||||
if state not in ("MASTER", "BACKUP"):
|
||||
broken.add(vhid)
|
||||
wrong = sorted(set(v for v, n in masters.items() if n != 1) | broken, key=lambda v: (len(v), v))
|
||||
if wrong:
|
||||
return _r("fail", "vhid_roles", vhids=", ".join(wrong))
|
||||
if not masters and not any(statuses[h.id].get("carp_status") == "MASTER" for h in online):
|
||||
return _r("fail", "no_master")
|
||||
return _r("ok", "ok")
|
||||
|
||||
|
||||
async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
|
||||
checks: dict | None = None, compliance_mode: str = "warn") -> dict:
|
||||
"""{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
|
||||
enabled = {**DEFAULT_CHECKS, **(checks or {})}
|
||||
for name in ALWAYS_ON:
|
||||
enabled[name] = True
|
||||
results: dict[str, dict] = {}
|
||||
fleets: dict[str, list[OpnsenseHost]] = {}
|
||||
for host in hosts:
|
||||
fleets[str(host.id)] = await peers(db, host) if host.cluster_name else [host]
|
||||
every = {h.id: h for fleet in fleets.values() for h in fleet}
|
||||
statuses = await get_statuses(list(every.values()))
|
||||
carp_cache: dict[str, dict] = {}
|
||||
for host in hosts:
|
||||
hid = str(host.id)
|
||||
before = baselines.get(hid) or {}
|
||||
row: dict[str, dict] = {}
|
||||
st = statuses.get(host.id, {})
|
||||
running = await _optional(client.get_firmware_running(host)) if st.get("status") == "online" else None
|
||||
if not host.enabled:
|
||||
row["online"] = _r("fail", "disabled")
|
||||
elif st.get("status") != "online":
|
||||
row["online"] = _r("fail", "offline")
|
||||
elif (running or {}).get("status") != "ready":
|
||||
row["online"] = _r("fail", "firmware_busy")
|
||||
else:
|
||||
row["online"] = _r("ok", "ok")
|
||||
reachable = row["online"]["result"] == "ok"
|
||||
|
||||
if enabled["version"]:
|
||||
info = await _optional(client.get_firmware_info(host)) if reachable else None
|
||||
version = (info or {}).get("product_version")
|
||||
if not target_version:
|
||||
row["version"] = _r("skip", "no_target")
|
||||
elif version is None:
|
||||
row["version"] = _r("fail", "unreadable")
|
||||
elif version != target_version:
|
||||
row["version"] = _r("fail", "version_mismatch", expected=target_version, actual=version)
|
||||
else:
|
||||
row["version"] = _r("ok", "ok", actual=version)
|
||||
|
||||
if enabled["carp"]:
|
||||
if not host.cluster_name:
|
||||
row["carp"] = _r("skip", "no_cluster")
|
||||
else:
|
||||
if host.cluster_name not in carp_cache:
|
||||
carp_cache[host.cluster_name] = await _check_carp(db, fleets[hid], statuses)
|
||||
row["carp"] = carp_cache[host.cluster_name]
|
||||
|
||||
if enabled["gateways"]:
|
||||
row["gateways"] = _check_gateways(before.get("gateways"), await _gateways(host) if reachable else None)
|
||||
if enabled["services"]:
|
||||
row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
|
||||
|
||||
blockers = await _shared_change_blockers(db, fleets[hid])
|
||||
row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
|
||||
|
||||
if enabled["compliance"]:
|
||||
base_score = before.get("compliance_score")
|
||||
if base_score is None:
|
||||
row["compliance"] = _r("skip", "no_baseline")
|
||||
elif not reachable:
|
||||
row["compliance"] = _r("fail", "unreadable")
|
||||
else:
|
||||
fresh = await compliance_service.check_host(db, host)
|
||||
if fresh.score is None:
|
||||
row["compliance"] = _r("warn", "unreadable")
|
||||
elif fresh.score < base_score:
|
||||
row["compliance"] = _r("fail" if compliance_mode == "fail" else "warn", "score_lower",
|
||||
before=base_score, after=fresh.score)
|
||||
else:
|
||||
row["compliance"] = _r("ok", "ok", before=base_score, after=fresh.score)
|
||||
results[hid] = row
|
||||
worst = max((_RANK[c["result"]] for row in results.values() for c in row.values()), default=0)
|
||||
return {"hosts": results, "verdict": ("ok", "warn", "fail")[worst]}
|
||||
|
||||
|
||||
def failed_checks(gate: dict) -> list[str]:
|
||||
"""Flat "host-id:check" list of everything that failed (for notifications and the audit trail)."""
|
||||
return [f"{hid}:{name}" for hid, row in (gate.get("hosts") or {}).items() for name, c in row.items() if c["result"] == "fail"]
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Rollout plans: resolving waves to hosts, validation, preview token / activation fingerprint.
|
||||
|
||||
The run engine (rollout_engine.py) builds on ``resolve``: what a preview shows is exactly what a run
|
||||
freezes when it starts.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
from app.models.policy import DeviceGroup
|
||||
from app.models.rollout import RolloutPlan
|
||||
from app.services import rollout_health, window_service
|
||||
|
||||
# Hard limits: an unattended engine that fires firmware updates must stay small and explainable.
|
||||
MAX_WAVES = 10
|
||||
MAX_HOSTS = 100
|
||||
MAX_SOAK_MINUTES = 24 * 60
|
||||
|
||||
|
||||
def _issue(code: str, **params) -> dict:
|
||||
return {"code": code, "params": params}
|
||||
|
||||
|
||||
async def resolve(db: AsyncSession, plan: RolloutPlan) -> dict:
|
||||
"""Waves with their hosts (a host in several waves belongs to the earliest), errors that block a
|
||||
start/activation, and warnings."""
|
||||
errors: list[dict] = []
|
||||
warnings: list[dict] = []
|
||||
seen: dict = {}
|
||||
duplicates: set[str] = set()
|
||||
waves = []
|
||||
group_ids = {g.id for w in plan.waves for g in w.groups}
|
||||
groups = {g.id: g for g in (await db.scalars(
|
||||
select(DeviceGroup).where(DeviceGroup.id.in_(group_ids)).options(selectinload(DeviceGroup.hosts)))).all()} if group_ids else {}
|
||||
for index, wave in enumerate(plan.waves):
|
||||
hosts: list[OpnsenseHost] = []
|
||||
for group in sorted(wave.groups, key=lambda g: g.name):
|
||||
for host in sorted(groups[group.id].hosts, key=lambda h: h.name):
|
||||
if host.id in seen:
|
||||
if seen[host.id] != index:
|
||||
duplicates.add(host.name)
|
||||
continue
|
||||
seen[host.id] = index
|
||||
hosts.append(host)
|
||||
if not hosts:
|
||||
warnings.append(_issue("empty_wave", wave=wave.label))
|
||||
clusters = sorted({h.cluster_name for h in hosts if h.cluster_name})
|
||||
if clusters and not wave.carp_handover:
|
||||
warnings.append(_issue("master_needs_handover", wave=wave.label, clusters=", ".join(clusters)))
|
||||
waves.append({
|
||||
"index": index, "label": wave.label, "soak_minutes": wave.soak_minutes, "carp_handover": wave.carp_handover,
|
||||
"require_approval": wave.require_approval, "groups": sorted(g.name for g in wave.groups),
|
||||
"hosts": [{"id": str(h.id), "name": h.name, "cluster_name": h.cluster_name, "enabled": h.enabled} for h in hosts],
|
||||
})
|
||||
if duplicates:
|
||||
warnings.append(_issue("duplicate_hosts", names=", ".join(sorted(duplicates))))
|
||||
all_hosts = [h for w in waves for h in w["hosts"]]
|
||||
disabled = sorted(h["name"] for h in all_hosts if not h["enabled"])
|
||||
if disabled:
|
||||
warnings.append(_issue("disabled_hosts", names=", ".join(disabled)))
|
||||
if not plan.waves:
|
||||
errors.append(_issue("no_waves"))
|
||||
elif not all_hosts:
|
||||
errors.append(_issue("no_hosts"))
|
||||
if len(plan.waves) > MAX_WAVES:
|
||||
errors.append(_issue("too_many_waves", limit=MAX_WAVES))
|
||||
if len(all_hosts) > MAX_HOSTS:
|
||||
errors.append(_issue("too_many_hosts", limit=MAX_HOSTS))
|
||||
|
||||
# Clusters: where are their nodes?
|
||||
cluster_names = sorted({h["cluster_name"] for h in all_hosts if h["cluster_name"]})
|
||||
for cluster in cluster_names:
|
||||
members = (await db.scalars(select(OpnsenseHost.id).where(OpnsenseHost.cluster_name == cluster))).all()
|
||||
in_plan = {h["id"]: w["index"] for w in waves for h in w["hosts"] if h["cluster_name"] == cluster}
|
||||
partial = len(in_plan) < len(members)
|
||||
split = len(set(in_plan.values())) > 1
|
||||
if plan.action == "major_update" and (partial or split):
|
||||
# Never two major releases side by side in one cluster longer than one wave.
|
||||
errors.append(_issue("cluster_split", cluster=cluster))
|
||||
elif partial:
|
||||
warnings.append(_issue("cluster_partial", cluster=cluster))
|
||||
elif split:
|
||||
warnings.append(_issue("mixed_versions", cluster=cluster, wave=waves[max(in_plan.values())]["label"]))
|
||||
|
||||
window = await db.get(MaintenanceWindow, plan.window_id) if plan.window_id else None
|
||||
if plan.auto_start and plan.action != "update":
|
||||
errors.append(_issue("auto_start_update_only"))
|
||||
if plan.auto_start and window is None:
|
||||
errors.append(_issue("auto_start_needs_window"))
|
||||
if window is None:
|
||||
warnings.append(_issue("no_window"))
|
||||
elif not window.enabled:
|
||||
warnings.append(_issue("window_disabled"))
|
||||
now = datetime.now(timezone.utc)
|
||||
return {
|
||||
"waves": waves, "errors": errors, "warnings": warnings,
|
||||
"window": {"id": str(window.id), "name": window.name, "definition": window_service.definition(window),
|
||||
"next": [o.as_dict() for o in window_service.occurrences(window, now, 3)] if window.enabled else []}
|
||||
if window else None,
|
||||
}
|
||||
|
||||
|
||||
def definition(plan: RolloutPlan) -> dict:
|
||||
return {
|
||||
"id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
|
||||
"auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
|
||||
"waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
|
||||
"require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
|
||||
}
|
||||
|
||||
|
||||
def effective_checks(checks: dict | None) -> dict:
|
||||
merged = {**rollout_health.DEFAULT_CHECKS, **{k: bool(v) for k, v in (checks or {}).items() if k in rollout_health.CHECKS}}
|
||||
for name in rollout_health.ALWAYS_ON:
|
||||
merged[name] = True
|
||||
return merged
|
||||
|
||||
|
||||
def fingerprint(plan: RolloutPlan, resolved: dict) -> str:
|
||||
"""Covers the plan, every resolved host (id, name, cluster, enabled) and the window definition: a change
|
||||
to any of them invalidates a preview and stops an activated automatic start."""
|
||||
state = {
|
||||
"plan": definition(plan),
|
||||
"hosts": [[w["index"], h["id"], h["name"], h["cluster_name"], h["enabled"]] for w in resolved["waves"] for h in w["hosts"]],
|
||||
"window": (resolved.get("window") or {}).get("definition"),
|
||||
}
|
||||
return hashlib.sha256(json.dumps(state, sort_keys=True, default=str).encode()).hexdigest()
|
||||
|
||||
|
||||
async def load_plan(db: AsyncSession, plan_id) -> RolloutPlan | None:
|
||||
return await db.scalar(
|
||||
select(RolloutPlan).where(RolloutPlan.id == plan_id).execution_options(populate_existing=True))
|
||||
@@ -20,26 +20,17 @@ from app.models.host import OpnsenseHost
|
||||
from app.models.rule import FirewallRule
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.services import opnsense_client
|
||||
from app.services.adopt_common import NotAdoptable, NotFoundOnHost, clean, on # noqa: F401 (NotAdoptable is re-exported for routers/rules.py)
|
||||
from app.services.policy_service import revision
|
||||
from app.services.rule_deploy_service import _build_payload
|
||||
|
||||
|
||||
class RuleNotFoundOnHost(Exception):
|
||||
class RuleNotFoundOnHost(NotFoundOnHost):
|
||||
pass
|
||||
|
||||
|
||||
class NotAdoptable(Exception):
|
||||
def __init__(self, reasons: list[str]):
|
||||
super().__init__(", ".join(reasons))
|
||||
self.reasons = reasons
|
||||
|
||||
|
||||
def _on(value) -> bool:
|
||||
return value in ("1", 1, True)
|
||||
|
||||
|
||||
def _clean(value) -> str:
|
||||
return "" if value is None else str(value).strip()
|
||||
_on = on
|
||||
_clean = clean
|
||||
|
||||
|
||||
def reasons_not_adoptable(row: dict, managed_uuids: set[str]) -> list[str]:
|
||||
|
||||
@@ -15,18 +15,31 @@ the UI) but is NOT wired into the NotificationSettings webhook/email
|
||||
system — that system is built around persistent NotificationProblem rows
|
||||
with their own admin-configurable toggles, and adding a new problem kind
|
||||
for this is a bigger change than this feature's own scope. Flagged here,
|
||||
not silently dropped, in case it's wanted later.
|
||||
not silently dropped, in case it's wanted later. (Later: the failure is reported by
|
||||
notification_checks.check_scheduled_maintenance_failed since 1.17.)
|
||||
|
||||
auto_verify (1.25, set for every rollout step): instead of leaving the operation "submitted"
|
||||
for a human, the run waits for the job (phase job_wait) and then checks the result itself
|
||||
(phase verify: firmware idle for a while, a reboot really happened when one was due, version
|
||||
readable). Only then the operation is closed as "acknowledged"; anything unclear ends as
|
||||
"unknown" (stays locked, notification). Without auto_verify nothing changes.
|
||||
|
||||
Concurrency: due rows and running rows are each taken with SELECT ... FOR UPDATE SKIP LOCKED,
|
||||
one row per transaction, so a second backend (briefly possible during a web update) can never
|
||||
fire or step the same row at the same time.
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import select, true
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from contextlib import contextmanager
|
||||
|
||||
from app.services import audit_service, backup_store, opnsense_client as client
|
||||
from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
|
||||
from app.services.host_status_service import get_statuses
|
||||
@@ -51,40 +64,93 @@ HANDOVER_TIMEOUT = timedelta(minutes=3)
|
||||
LEAVE_TIMEOUT = timedelta(minutes=5)
|
||||
JOB_GRACE = timedelta(minutes=5) # no sign of a job after this long: treat a package update as already done
|
||||
JOB_TIMEOUT = {"reboot": timedelta(minutes=20), "update": timedelta(minutes=60), "major_update": timedelta(minutes=150)}
|
||||
# auto_verify: the firmware must report "ready" for this long in a row (a reboot at the end of an update
|
||||
# can follow a short "ready" moment), and the whole check gives up after VERIFY_TIMEOUT.
|
||||
VERIFY_SETTLE = timedelta(seconds=60)
|
||||
VERIFY_TIMEOUT = timedelta(minutes=20)
|
||||
# A rollout step is created with run_at=now and fires on the next poll. Found much later (OPNexus was down,
|
||||
# e.g. during an update), it no longer belongs to the decision that created it: the window may have closed
|
||||
# and the run may have been paused meanwhile. It is then not fired; the rollout engine re-queues the host.
|
||||
ROLLOUT_FIRE_GRACE = timedelta(minutes=5)
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _as_actor(sched: ScheduledMaintenance):
|
||||
"""Audit events of an unattended run name the person who planned it (not the rollback worker)."""
|
||||
token = audit_service.context.set({"actor": f"{sched.actor} (geplant)"[:150], "id": str(sched.id)})
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
audit_service.context.reset(token)
|
||||
|
||||
|
||||
def _rollout_details(sched: ScheduledMaintenance) -> dict:
|
||||
rollout = (sched.progress or {}).get("rollout")
|
||||
return {"rollout": rollout} if rollout else {}
|
||||
|
||||
|
||||
async def fire_due(db: AsyncSession) -> dict:
|
||||
due = list(
|
||||
(
|
||||
await db.scalars(
|
||||
select(ScheduledMaintenance).where(
|
||||
ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now()
|
||||
)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
fired = 0
|
||||
blocked = 0
|
||||
for sched in due:
|
||||
try:
|
||||
await _fire_one(db, sched)
|
||||
except Exception:
|
||||
logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
|
||||
due = fired = blocked = 0
|
||||
seen: set = set()
|
||||
while True:
|
||||
# One row per transaction, locked: a second backend skips it instead of firing it again.
|
||||
sched = await db.scalar(
|
||||
select(ScheduledMaintenance)
|
||||
.where(ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now(),
|
||||
ScheduledMaintenance.id.not_in(seen) if seen else true())
|
||||
.order_by(ScheduledMaintenance.run_at)
|
||||
.limit(1)
|
||||
.with_for_update(skip_locked=True)
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if sched is None:
|
||||
await db.commit()
|
||||
break
|
||||
seen.add(sched.id)
|
||||
due += 1
|
||||
with _as_actor(sched):
|
||||
try:
|
||||
await _fire_one(db, sched)
|
||||
except Exception:
|
||||
logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
|
||||
await db.rollback()
|
||||
sched = await db.get(ScheduledMaintenance, sched.id, populate_existing=True)
|
||||
if sched.status in ("pending", "running") and sched.phase is None:
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
|
||||
await db.commit()
|
||||
await db.commit() # never keep the row lock past this row
|
||||
if sched.status == "failed":
|
||||
blocked += 1
|
||||
else:
|
||||
fired += 1
|
||||
return {"due": len(due), "fired": fired, "blocked_or_failed": blocked}
|
||||
return {"due": due, "fired": fired, "blocked_or_failed": blocked}
|
||||
|
||||
|
||||
def _reboot_expected(action: str, state: dict) -> bool:
|
||||
return action in ("reboot", "major_update") or bool(state.get("reboot_required"))
|
||||
|
||||
|
||||
async def _boot_time(host: OpnsenseHost) -> str | None:
|
||||
"""systemTime's "boottime" (Diagnostics SystemController::systemTimeAction, read-only, already used by the
|
||||
resource sampler). A different value afterwards proves a reboot even if no poll caught the host offline."""
|
||||
try:
|
||||
return (await client.get_system_time(host)).get("boottime") or None
|
||||
except client.OpnsenseAPIError:
|
||||
return None
|
||||
|
||||
|
||||
async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
if sched.origin == "rollout" and _now() - sched.run_at > ROLLOUT_FIRE_GRACE:
|
||||
sched.status = "failed"
|
||||
sched.result = {"error": "Rollout-Schritt nicht rechtzeitig gestartet, nichts ausgeführt", "not_fired": True}
|
||||
await db.commit()
|
||||
logger.warning("Geplante Wartung %s: Rollout-Schritt überfällig, nicht ausgeführt", sched.id)
|
||||
return
|
||||
host = await db.get(OpnsenseHost, sched.host_id)
|
||||
if not host or not host.enabled:
|
||||
sched.status = "failed"
|
||||
@@ -137,6 +203,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
|
||||
return
|
||||
|
||||
boot_before = await _boot_time(host) if sched.auto_verify else None
|
||||
op = HostOperation(
|
||||
host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
|
||||
expires_at=_now(), fingerprint="", snapshot=state, status="starting", started_at=_now(),
|
||||
@@ -147,14 +214,23 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
await audit_service.persist(
|
||||
kind="host", action=sched.action, status="requested",
|
||||
host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state},
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state, **_rollout_details(sched)},
|
||||
)
|
||||
try:
|
||||
result = await _ACTION_FN[sched.action](host)
|
||||
op.status = "submitted"
|
||||
op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
|
||||
sched.status = "succeeded"
|
||||
sched.result = {"response": result}
|
||||
if sched.auto_verify:
|
||||
# Not done yet: the poll loop waits for the job and checks the result (job_wait -> verify).
|
||||
op.message = "Geplant übergeben. Abschluss wird automatisch geprüft."
|
||||
sched.phase, sched.phase_since = "job_wait", _now()
|
||||
_progress(sched, handover=False, job_started_at=_now().isoformat(), seen_down=False, seen_running=False,
|
||||
version_before=state.get("version"), boot_before=boot_before,
|
||||
reboot_expected=_reboot_expected(sched.action, state))
|
||||
sched.result = {"response": result}
|
||||
else:
|
||||
op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
|
||||
sched.status = "succeeded"
|
||||
sched.result = {"response": result}
|
||||
except client.OpnsenseAPIError as exc:
|
||||
op.status = "unknown"
|
||||
op.message = "Ausgang unklar: " + str(exc)[:800] + ". Nicht automatisch wiederholen."
|
||||
@@ -166,7 +242,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
await audit_service.persist(
|
||||
kind="host", action=sched.action, status=op.status,
|
||||
host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
|
||||
message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result},
|
||||
message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result, **_rollout_details(sched)},
|
||||
)
|
||||
|
||||
|
||||
@@ -182,21 +258,24 @@ async def _audit(host, sched, op, action, status, message=None, **details):
|
||||
await audit_service.persist(
|
||||
kind="host", action=action, status=status, host_id=str(host.id), host_name=host.name,
|
||||
entity_id=str(op.id) if op else None, message=message,
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "handover": True, **details},
|
||||
details={"scheduled": True, "schedule_id": str(sched.id), "handover": bool((sched.progress or {}).get("handover")),
|
||||
**_rollout_details(sched), **details},
|
||||
)
|
||||
|
||||
|
||||
async def _end(db, sched, host, op, *, ok, message, op_status=None, **result):
|
||||
handover = bool((sched.progress or {}).get("handover"))
|
||||
sched.status = "succeeded" if ok else "failed"
|
||||
sched.phase = None
|
||||
sched.result = {"handover": True, **result} if ok else {"error": message[:900], **result}
|
||||
sched.result = ({"handover": handover, "auto_verified": bool(sched.auto_verify), **result} if ok
|
||||
else {"error": message[:900], **result})
|
||||
if op is not None:
|
||||
op.status = op_status or ("acknowledged" if ok else "unknown")
|
||||
op.message = message[:1000]
|
||||
await db.commit()
|
||||
await _audit(host, sched, op, sched.action, "completed" if ok else (op_status or "unknown"), message)
|
||||
if not ok:
|
||||
logger.warning("Geplante Wartung %s (CARP-Übergabe) beendet: %s", sched.id, message)
|
||||
logger.warning("Geplante Wartung %s beendet: %s", sched.id, message)
|
||||
|
||||
|
||||
async def _toggle(host, expected: str) -> None:
|
||||
@@ -230,6 +309,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
await fail(" ".join(blocked))
|
||||
return
|
||||
|
||||
boot_before = await _boot_time(host) if sched.auto_verify else None
|
||||
now = _now()
|
||||
op = HostOperation(
|
||||
host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
|
||||
@@ -237,6 +317,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
)
|
||||
db.add(op)
|
||||
sched.status = "running"
|
||||
_progress(sched, handover=True, boot_before=boot_before, reboot_expected=_reboot_expected(sched.action, state))
|
||||
await db.commit()
|
||||
sched.host_operation_id = op.id
|
||||
await _audit(host, sched, op, "carp_enter", "requested", snapshot=state)
|
||||
@@ -255,32 +336,56 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
|
||||
|
||||
|
||||
async def advance_running(db: AsyncSession) -> int:
|
||||
"""One step for every handover run in progress. Returns how many are still running."""
|
||||
rows = list((await db.scalars(
|
||||
select(ScheduledMaintenance).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
|
||||
"""One step for every run in progress (handover and/or auto_verify). Returns how many are still running."""
|
||||
ids = list((await db.scalars(
|
||||
select(ScheduledMaintenance.id).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
|
||||
)).all())
|
||||
for sched in rows:
|
||||
host = await db.get(OpnsenseHost, sched.host_id)
|
||||
op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
|
||||
try:
|
||||
if host is None:
|
||||
raise RuntimeError("Host nicht mehr vorhanden")
|
||||
if op is not None and op.status not in ACTIVE:
|
||||
# A human closed or cancelled the operation: stop here and touch nothing more.
|
||||
sched.status, sched.phase = "failed", None
|
||||
sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt. CARP-Rolle von {host.name} prüfen."}
|
||||
await db.commit()
|
||||
continue
|
||||
await _STEPS[sched.phase](db, sched, host, op)
|
||||
except Exception as exc: # a stuck run must not loop forever or take the poll loop down
|
||||
logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
|
||||
if host is not None:
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Unerwarteter Fehler während der CARP-Übergabe ({type(exc).__name__}). "
|
||||
f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen.")
|
||||
return sum(1 for s in rows if s.status == "running")
|
||||
await db.commit()
|
||||
still_running = 0
|
||||
for sched_id in ids:
|
||||
# Locked for the whole step: a second backend skips the row instead of toggling or submitting twice.
|
||||
sched = await db.scalar(
|
||||
select(ScheduledMaintenance)
|
||||
.where(ScheduledMaintenance.id == sched_id, ScheduledMaintenance.status == "running",
|
||||
ScheduledMaintenance.phase.is_not(None))
|
||||
.with_for_update(skip_locked=True)
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if sched is None:
|
||||
await db.commit()
|
||||
still_running += 1 if await db.scalar(select(ScheduledMaintenance.status).where(ScheduledMaintenance.id == sched_id)) == "running" else 0
|
||||
await db.commit()
|
||||
continue
|
||||
with _as_actor(sched):
|
||||
await _advance_one(db, sched)
|
||||
await db.commit()
|
||||
if sched.status == "running":
|
||||
still_running += 1
|
||||
return still_running
|
||||
|
||||
|
||||
async def _advance_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
|
||||
host = await db.get(OpnsenseHost, sched.host_id)
|
||||
op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
|
||||
try:
|
||||
if host is None:
|
||||
raise RuntimeError("Host nicht mehr vorhanden")
|
||||
if op is not None and op.status not in ACTIVE:
|
||||
# A human closed or cancelled the operation: stop here and touch nothing more.
|
||||
hint = f" CARP-Rolle von {host.name} prüfen." if (sched.progress or {}).get("handover") else ""
|
||||
sched.status, sched.phase = "failed", None
|
||||
sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt.{hint}"}
|
||||
await db.commit()
|
||||
return
|
||||
await _STEPS[sched.phase](db, sched, host, op)
|
||||
except Exception as exc: # a stuck run must not loop forever or take the poll loop down
|
||||
logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
|
||||
if host is not None:
|
||||
hint = (f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen."
|
||||
if (sched.progress or {}).get("handover") else f"Zustand von {host.name} prüfen.")
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Unerwarteter Fehler während der geplanten Wartung ({type(exc).__name__}). {hint}")
|
||||
|
||||
def _since(sched: ScheduledMaintenance) -> timedelta:
|
||||
return _now() - (sched.phase_since or _now())
|
||||
|
||||
@@ -352,17 +457,75 @@ async def _step_job_wait(db, sched, host, op):
|
||||
else:
|
||||
done = bool(prog.get("seen_down") or prog.get("seen_running")) or elapsed >= JOB_GRACE
|
||||
if done:
|
||||
sched.phase, sched.phase_since = "leave", _now()
|
||||
sched.phase = "verify" if sched.auto_verify else "leave"
|
||||
sched.phase_since = _now()
|
||||
await db.commit()
|
||||
return
|
||||
if elapsed >= JOB_TIMEOUT[sched.action]:
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
|
||||
f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
|
||||
if prog.get("handover"):
|
||||
text = (f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
|
||||
f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
|
||||
else:
|
||||
text = f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. Bitte {host.name} prüfen."
|
||||
await _end(db, sched, host, op, ok=False, message=text)
|
||||
return
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _step_verify(db, sched, host, op):
|
||||
"""auto_verify: is the job really over and the host healthy? Read-only, never repeats the action."""
|
||||
prog = dict(sched.progress or {})
|
||||
handover = bool(prog.get("handover"))
|
||||
|
||||
async def wait_or_give_up(why: str):
|
||||
if _since(sched) >= VERIFY_TIMEOUT:
|
||||
tail = (f" {host.name} bleibt im CARP-Wartungsmodus, der Partner trägt den Verkehr." if handover else "")
|
||||
await _end(db, sched, host, op, ok=False,
|
||||
message=f"Automatische Prüfung nach „{sched.action}“ nicht bestanden: {why}.{tail} "
|
||||
f"Bitte {host.name} prüfen und die Wartung von Hand abschließen.")
|
||||
return
|
||||
_progress(sched, waiting_for=why)
|
||||
await db.commit()
|
||||
|
||||
try:
|
||||
running = (await client.get_firmware_running(host)).get("status")
|
||||
except client.OpnsenseAPIError:
|
||||
running = None
|
||||
_progress(sched, seen_down=True)
|
||||
if running != "ready":
|
||||
_progress(sched, ready_since=None)
|
||||
await wait_or_give_up("Firmware-Auftrag läuft noch oder Host nicht erreichbar")
|
||||
return
|
||||
if not prog.get("ready_since"):
|
||||
_progress(sched, ready_since=_now().isoformat())
|
||||
await db.commit()
|
||||
return
|
||||
if _now() - datetime.fromisoformat(prog["ready_since"]) < VERIFY_SETTLE:
|
||||
await db.commit()
|
||||
return
|
||||
boot_now = await _boot_time(host)
|
||||
rebooted = bool(prog.get("seen_down")) or bool(prog.get("boot_before") and boot_now and boot_now != prog["boot_before"])
|
||||
if prog.get("reboot_expected") and not rebooted:
|
||||
await wait_or_give_up("der erwartete Neustart wurde nicht beobachtet")
|
||||
return
|
||||
try:
|
||||
version = (await client.get_firmware_info(host)).get("product_version")
|
||||
except client.OpnsenseAPIError:
|
||||
version = None
|
||||
online = (await get_statuses([host])).get(host.id, {}).get("status") == "online"
|
||||
if not version or not online:
|
||||
await wait_or_give_up("Host nicht vollständig erreichbar (Version/Status nicht lesbar)")
|
||||
return
|
||||
_progress(sched, version_after=version, boot_after=boot_now, rebooted=rebooted, waiting_for=None)
|
||||
if handover:
|
||||
sched.phase, sched.phase_since = "leave", _now() # only now take the traffic back
|
||||
await db.commit()
|
||||
return
|
||||
await _end(db, sched, host, op, ok=True,
|
||||
message=f"Automatisch geprüft: {host.name} erreichbar, Firmware-Auftrag beendet, Version {version}.",
|
||||
version_before=prog.get("version_before"), version_after=version, rebooted=rebooted)
|
||||
|
||||
|
||||
async def _step_leave(db, sched, host, op):
|
||||
me, _partner = await _roles(db, host)
|
||||
if me.get("status") != "online":
|
||||
@@ -397,6 +560,8 @@ async def _step_leave_wait(db, sched, host, op):
|
||||
except client.OpnsenseAPIError:
|
||||
pass
|
||||
extra["version_before"] = (sched.progress or {}).get("version_before")
|
||||
if sched.auto_verify:
|
||||
extra["rebooted"] = bool((sched.progress or {}).get("rebooted")) # what verify saw before leaving
|
||||
text = "Automatisch abgeschlossen: Übergabe, Auftrag und Zurückschalten bestätigt."
|
||||
if extra.get("reboot_required"):
|
||||
text += " Ein Neustart ist noch erforderlich."
|
||||
@@ -410,6 +575,7 @@ async def _step_leave_wait(db, sched, host, op):
|
||||
_STEPS = {
|
||||
"handover_wait": _step_handover_wait,
|
||||
"job_wait": _step_job_wait,
|
||||
"verify": _step_verify,
|
||||
"leave": _step_leave,
|
||||
"leave_wait": _step_leave_wait,
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Destination check for URLs OPNexus is told to call (notification channels, webhook).
|
||||
|
||||
Admins legitimately point these at servers on their own network (a self-hosted ntfy, Gotify or
|
||||
Matrix), so private ranges are allowed. What is refused is what cannot be a notification target:
|
||||
anything but http(s), credentials inside the URL, and addresses that only ever mean "this machine"
|
||||
or "the cloud metadata service" (loopback, link-local such as 169.254.169.254, unspecified,
|
||||
multicast). Redirects are never followed by the callers, so a public host cannot bounce a request
|
||||
to such an address. The hostname is resolved at send time and every address is checked (a name that
|
||||
does not resolve is let through; the request then fails on its own).
|
||||
"""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import contextvars
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
# httpx logs every request line ("HTTP Request: POST https://api.telegram.org/bot<token>/sendMessage"). For
|
||||
# targets whose URL is the credential that line must not reach the logs: callers wrap the request in muted().
|
||||
_muted = contextvars.ContextVar("url_guard_muted", default=False)
|
||||
|
||||
|
||||
class _MuteFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
return not _muted.get()
|
||||
|
||||
|
||||
for _name in ("httpx", "httpcore"):
|
||||
logging.getLogger(_name).addFilter(_MuteFilter())
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def muted():
|
||||
token = _muted.set(True)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_muted.reset(token)
|
||||
|
||||
|
||||
def _forbidden(ip: ipaddress._BaseAddress) -> bool:
|
||||
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
|
||||
ip = ip.ipv4_mapped
|
||||
return ip.is_loopback or ip.is_link_local or ip.is_unspecified or ip.is_multicast
|
||||
|
||||
|
||||
def validate_url(url: str, *, max_length: int = 500) -> str:
|
||||
"""Returns the cleaned URL or raises ValueError with a short reason (shown to the admin)."""
|
||||
value = (url or "").strip()
|
||||
if not value or len(value) > max_length or any(c.isspace() for c in value):
|
||||
raise ValueError("Die Adresse ist leer, zu lang oder enthält Leerzeichen")
|
||||
try:
|
||||
parts = urlsplit(value)
|
||||
host = parts.hostname
|
||||
parts.port # noqa: B018 - raises ValueError for an invalid port
|
||||
except ValueError:
|
||||
raise ValueError("Die Adresse ist ungültig") from None
|
||||
if parts.scheme not in ("http", "https") or not host:
|
||||
raise ValueError("Die Adresse muss mit http:// oder https:// beginnen")
|
||||
if parts.username or parts.password:
|
||||
raise ValueError("Zugangsdaten gehören nicht in die Adresse")
|
||||
try:
|
||||
literal = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
literal = None
|
||||
if literal is not None and _forbidden(literal):
|
||||
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
|
||||
if literal is None and host.lower() in ("localhost", "localhost.localdomain"):
|
||||
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
|
||||
return value
|
||||
|
||||
|
||||
async def ensure_resolved_safe(url: str) -> None:
|
||||
"""Send-time check against DNS answers pointing at forbidden addresses. Raises ValueError."""
|
||||
host = urlsplit(url).hostname
|
||||
if not host:
|
||||
raise ValueError("Die Adresse ist ungültig")
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
return # a literal was already checked by validate_url
|
||||
except ValueError:
|
||||
pass
|
||||
try:
|
||||
infos = await asyncio.get_running_loop().getaddrinfo(host, None, type=socket.SOCK_STREAM)
|
||||
except OSError:
|
||||
return
|
||||
for info in infos:
|
||||
try:
|
||||
ip = ipaddress.ip_address(info[4][0].split("%")[0])
|
||||
except ValueError:
|
||||
continue
|
||||
if _forbidden(ip):
|
||||
raise ValueError("Der Name zeigt auf eine nicht erlaubte Adresse (lokal, Link-Local oder Metadaten-Dienst)")
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Occurrences of recurring maintenance windows.
|
||||
|
||||
Rules (deliberately simple and written down, because a firmware rollout depends on them):
|
||||
|
||||
* A window is a wall-clock start time in its own IANA time zone plus a duration. "Sunday 02:00
|
||||
Europe/Berlin" stays 02:00 local time in summer and in winter.
|
||||
* Daylight saving: a local start time that does not exist on that day (spring forward, e.g. 02:30 on
|
||||
the last Sunday of March in Europe/Berlin) moves FORWARD by the gap (03:30). A local time that
|
||||
exists twice (fall back) means the FIRST of the two (fold=0, still summer time). Both follow from
|
||||
Python's zoneinfo with fold=0 and are covered by tests.
|
||||
* The duration is real elapsed time (a 3 h window over the fall-back night still lasts 3 h).
|
||||
* Exceptions are keyed by the occurrence's local date: "skip" drops it, "move" replaces its start
|
||||
with an absolute instant (same duration and cut-off).
|
||||
* New host actions may only START within the first ``start_cutoff_minutes`` of an occurrence
|
||||
(``startable``); actions already running finish even after the window has closed.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError, available_timezones
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow
|
||||
|
||||
# How far ahead occurrences are searched. A monthly "5th ..." does not exist; the longest real gap
|
||||
# between two occurrences is a month, so two years is plenty and bounds the work per call.
|
||||
_SEARCH_DAYS = 2 * 366
|
||||
MAX_DURATION_MINUTES = 24 * 60
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Occurrence:
|
||||
local_date: date
|
||||
start: datetime # UTC
|
||||
end: datetime # UTC
|
||||
cutoff: datetime # UTC: no new host action after this instant
|
||||
moved: bool = False
|
||||
skipped: bool = False
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {"local_date": self.local_date.isoformat(), "start": self.start.isoformat(), "end": self.end.isoformat(),
|
||||
"cutoff": self.cutoff.isoformat(), "moved": self.moved, "skipped": self.skipped}
|
||||
|
||||
|
||||
def valid_timezone(name: str) -> bool:
|
||||
if not name or name not in available_timezones():
|
||||
return False
|
||||
try:
|
||||
ZoneInfo(name)
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _matches(window: MaintenanceWindow, day: date) -> bool:
|
||||
if window.recurrence == "weekly":
|
||||
return day.weekday() in (window.weekdays or [])
|
||||
if window.recurrence == "monthly":
|
||||
if window.month_mode == "day":
|
||||
return day.day == window.month_day
|
||||
if window.month_mode == "nth_weekday":
|
||||
if day.weekday() != window.month_weekday:
|
||||
return False
|
||||
if window.month_nth == -1:
|
||||
return (day + timedelta(days=7)).month != day.month
|
||||
return (day.day - 1) // 7 + 1 == window.month_nth
|
||||
return False
|
||||
|
||||
|
||||
def local_start(window: MaintenanceWindow, day: date) -> datetime:
|
||||
"""The UTC instant of the window's start on local date ``day`` (DST rules see module docstring)."""
|
||||
tz = ZoneInfo(window.timezone)
|
||||
wall = datetime.combine(day, window.start_time, tzinfo=tz).replace(fold=0)
|
||||
return wall.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def _occurrence(window: MaintenanceWindow, day: date, start: datetime, *, moved: bool = False, skipped: bool = False) -> Occurrence:
|
||||
duration = timedelta(minutes=window.duration_minutes)
|
||||
cutoff = timedelta(minutes=min(window.start_cutoff_minutes, window.duration_minutes))
|
||||
return Occurrence(local_date=day, start=start, end=start + duration, cutoff=start + cutoff, moved=moved, skipped=skipped)
|
||||
|
||||
|
||||
def occurrences(window: MaintenanceWindow, after: datetime, n: int = 10, *, with_skipped: bool = False) -> list[Occurrence]:
|
||||
"""The next ``n`` occurrences whose END lies after ``after`` (so one that is open right now is first).
|
||||
``with_skipped`` lists skipped ones too (flagged, for the UI); the engine never sees them."""
|
||||
if n <= 0:
|
||||
return []
|
||||
after = after.astimezone(timezone.utc)
|
||||
tz = ZoneInfo(window.timezone)
|
||||
exceptions = {e.occurrence_date: e for e in (window.exceptions or [])}
|
||||
duration = timedelta(minutes=window.duration_minutes)
|
||||
# Start one day early: an occurrence that began yesterday (local) may still be open.
|
||||
day = (after - duration).astimezone(tz).date() - timedelta(days=1)
|
||||
found: list[Occurrence] = []
|
||||
# Moved occurrences can land anywhere, so they are collected separately and merged at the end.
|
||||
for exc in exceptions.values():
|
||||
if exc.kind == "move" and exc.moved_start is not None and _matches(window, exc.occurrence_date):
|
||||
occ = _occurrence(window, exc.occurrence_date, exc.moved_start.astimezone(timezone.utc), moved=True)
|
||||
if occ.end > after:
|
||||
found.append(occ)
|
||||
regular = 0
|
||||
for _ in range(_SEARCH_DAYS):
|
||||
if regular >= n:
|
||||
break
|
||||
if _matches(window, day):
|
||||
exc = exceptions.get(day)
|
||||
if exc is None or (with_skipped and exc.kind == "skip"):
|
||||
occ = _occurrence(window, day, local_start(window, day), skipped=exc is not None)
|
||||
if occ.end > after:
|
||||
found.append(occ)
|
||||
regular += 1
|
||||
day += timedelta(days=1)
|
||||
found.sort(key=lambda o: o.start)
|
||||
return found[:n]
|
||||
|
||||
|
||||
def current(window: MaintenanceWindow, now: datetime) -> Occurrence | None:
|
||||
"""The occurrence open at ``now`` (start <= now < end), if any."""
|
||||
if not window.enabled:
|
||||
return None
|
||||
for occ in occurrences(window, now, 3):
|
||||
if occ.start <= now < occ.end:
|
||||
return occ
|
||||
return None
|
||||
|
||||
|
||||
def startable(window: MaintenanceWindow | None, now: datetime) -> Occurrence | None:
|
||||
"""The occurrence in which a NEW host action may start right now (start <= now < cutoff)."""
|
||||
if window is None or not window.enabled:
|
||||
return None
|
||||
occ = current(window, now)
|
||||
if occ is not None and occ.start <= now < occ.cutoff:
|
||||
return occ
|
||||
return None
|
||||
|
||||
|
||||
def is_occurrence_date(window: MaintenanceWindow, day: date) -> bool:
|
||||
return _matches(window, day)
|
||||
|
||||
|
||||
def definition(window: MaintenanceWindow) -> dict:
|
||||
"""What a rollout plan's fingerprint covers: changing any of this re-requires activation."""
|
||||
return {
|
||||
"id": str(window.id), "timezone": window.timezone, "recurrence": window.recurrence,
|
||||
"weekdays": sorted(window.weekdays or []), "month_mode": window.month_mode, "month_nth": window.month_nth,
|
||||
"month_weekday": window.month_weekday, "month_day": window.month_day,
|
||||
"start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
|
||||
"start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
|
||||
}
|
||||
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
|
||||
Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
"""
|
||||
|
||||
APP_VERSION = "1.23.10"
|
||||
APP_CODENAME = "Harrier"
|
||||
APP_VERSION = "1.25.0"
|
||||
APP_CODENAME = "Goshawk"
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Alarme für Gateways, Ressourcen und Dienste; zusätzliche Benachrichtigungskanäle.
|
||||
|
||||
Revision ID: 0020
|
||||
Revises: 0019
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0020"
|
||||
down_revision = "0019"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
_COLUMNS = [
|
||||
("gateway_down_enabled", "boolean NOT NULL DEFAULT true"),
|
||||
("resource_threshold_enabled", "boolean NOT NULL DEFAULT false"),
|
||||
("service_down_enabled", "boolean NOT NULL DEFAULT false"),
|
||||
("gateway_loss_threshold_pct", "integer NOT NULL DEFAULT 20"),
|
||||
("gateway_delay_threshold_ms", "integer NOT NULL DEFAULT 500"),
|
||||
("disk_threshold_pct", "integer NOT NULL DEFAULT 90"),
|
||||
("memory_threshold_pct", "integer NOT NULL DEFAULT 95"),
|
||||
("swap_threshold_pct", "integer NOT NULL DEFAULT 50"),
|
||||
("load_per_cpu_threshold", "double precision NOT NULL DEFAULT 2.0"),
|
||||
("alert_confirm_samples", "integer NOT NULL DEFAULT 3"),
|
||||
("resource_confirm_samples", "integer NOT NULL DEFAULT 5"),
|
||||
("service_watchlist", "json"),
|
||||
]
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for name, ddl in _COLUMNS:
|
||||
op.execute(f"ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS {name} {ddl}")
|
||||
if sa.inspect(op.get_bind()).has_table("notification_channels"):
|
||||
return
|
||||
op.create_table(
|
||||
"notification_channels",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("name", sa.String(length=100), nullable=False),
|
||||
sa.Column("channel_type", sa.String(length=16), nullable=False),
|
||||
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
|
||||
sa.Column("config", sa.JSON(), nullable=False),
|
||||
sa.Column("secret_set", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DROP TABLE IF EXISTS notification_channels")
|
||||
for name, _ddl in reversed(_COLUMNS):
|
||||
op.execute(f"ALTER TABLE notification_settings DROP COLUMN IF EXISTS {name}")
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Wiederkehrende Wartungsfenster mit Ausnahmen (1.25).
|
||||
|
||||
Revision ID: 0021
|
||||
Revises: 0020
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0021"
|
||||
down_revision = "0020"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
|
||||
if sa.inspect(op.get_bind()).has_table("maintenance_windows"):
|
||||
return
|
||||
op.create_table(
|
||||
"maintenance_windows",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("name", sa.String(length=120), nullable=False),
|
||||
sa.Column("timezone", sa.String(length=64), nullable=False),
|
||||
sa.Column("recurrence", sa.String(length=16), nullable=False),
|
||||
sa.Column("weekdays", sa.JSON(), nullable=False),
|
||||
sa.Column("month_mode", sa.String(length=16), nullable=True),
|
||||
sa.Column("month_nth", sa.Integer(), nullable=True),
|
||||
sa.Column("month_weekday", sa.Integer(), nullable=True),
|
||||
sa.Column("month_day", sa.Integer(), nullable=True),
|
||||
sa.Column("start_time", sa.Time(), nullable=False),
|
||||
sa.Column("duration_minutes", sa.Integer(), nullable=False),
|
||||
sa.Column("start_cutoff_minutes", sa.Integer(), server_default="60", nullable=False),
|
||||
sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
|
||||
sa.Column("created_by", sa.String(length=150), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
)
|
||||
op.create_table(
|
||||
"maintenance_window_exceptions",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("window_id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("occurrence_date", sa.Date(), nullable=False),
|
||||
sa.Column("kind", sa.String(length=8), nullable=False),
|
||||
sa.Column("moved_start", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("reason", sa.String(length=300), nullable=False),
|
||||
sa.Column("actor", sa.String(length=150), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),
|
||||
)
|
||||
op.create_index("ix_maintenance_window_exceptions_window_id", "maintenance_window_exceptions", ["window_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DROP TABLE IF EXISTS maintenance_window_exceptions")
|
||||
op.execute("DROP TABLE IF EXISTS maintenance_windows")
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Geplante Wartung: automatische Prüfung nach dem Auftrag (auto_verify) und Herkunft (manuell/Rollout).
|
||||
|
||||
Revision ID: 0022
|
||||
Revises: 0021
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0022"
|
||||
down_revision = "0021"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS auto_verify boolean NOT NULL DEFAULT false")
|
||||
op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS origin varchar(16) NOT NULL DEFAULT 'manual'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS origin")
|
||||
op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS auto_verify")
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Firmware-Rollouts in Wellen: Pläne, Läufe, Hosts je Lauf; Benachrichtigung bei Rollout-Problemen (1.25).
|
||||
|
||||
Revision ID: 0023
|
||||
Revises: 0022
|
||||
Create Date: 2026-10-21
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0023"
|
||||
down_revision = "0022"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
UUID = postgresql.UUID(as_uuid=True)
|
||||
TS = sa.DateTime(timezone=True)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute("ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS rollout_problems_enabled boolean NOT NULL DEFAULT true")
|
||||
# A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
|
||||
if sa.inspect(op.get_bind()).has_table("rollout_plans"):
|
||||
return
|
||||
op.create_table(
|
||||
"rollout_plans",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("name", sa.String(120), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=False),
|
||||
sa.Column("action", sa.String(16), nullable=False),
|
||||
sa.Column("window_id", UUID, nullable=True),
|
||||
sa.Column("auto_start", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("gate_checks", sa.JSON(), nullable=False),
|
||||
sa.Column("compliance_mode", sa.String(8), server_default="warn", nullable=False),
|
||||
sa.Column("activated_by", sa.String(150), nullable=True),
|
||||
sa.Column("activated_at", TS, nullable=True),
|
||||
sa.Column("activated_fingerprint", sa.String(64), nullable=True),
|
||||
sa.Column("auto_start_error", sa.JSON(), nullable=True),
|
||||
sa.Column("created_by", sa.String(150), nullable=False),
|
||||
sa.Column("created_at", TS, nullable=False),
|
||||
sa.Column("updated_at", TS, nullable=False),
|
||||
sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
)
|
||||
op.create_index("ix_rollout_plans_window_id", "rollout_plans", ["window_id"])
|
||||
op.create_table(
|
||||
"rollout_plan_waves",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("plan_id", UUID, nullable=False),
|
||||
sa.Column("position", sa.Integer(), nullable=False),
|
||||
sa.Column("label", sa.String(60), nullable=False),
|
||||
sa.Column("soak_minutes", sa.Integer(), nullable=False),
|
||||
sa.Column("carp_handover", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.Column("require_approval", sa.Boolean(), server_default="false", nullable=False),
|
||||
sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_rollout_plan_waves_plan_id", "rollout_plan_waves", ["plan_id"])
|
||||
op.create_table(
|
||||
"rollout_wave_groups",
|
||||
sa.Column("wave_id", UUID, nullable=False),
|
||||
sa.Column("group_id", UUID, nullable=False),
|
||||
sa.ForeignKeyConstraint(["wave_id"], ["rollout_plan_waves.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["group_id"], ["device_groups.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("wave_id", "group_id"),
|
||||
)
|
||||
op.create_table(
|
||||
"rollout_runs",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("plan_id", UUID, nullable=True),
|
||||
sa.Column("plan_name", sa.String(120), nullable=False),
|
||||
sa.Column("plan_snapshot", sa.JSON(), nullable=False),
|
||||
sa.Column("action", sa.String(16), nullable=False),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("paused_from", sa.String(20), nullable=True),
|
||||
sa.Column("current_wave", sa.Integer(), nullable=False),
|
||||
sa.Column("target_version", sa.String(40), nullable=True),
|
||||
sa.Column("pause_reason", sa.String(40), nullable=True),
|
||||
sa.Column("pause_detail", sa.JSON(), nullable=True),
|
||||
sa.Column("started_by", sa.String(150), nullable=False),
|
||||
sa.Column("trigger", sa.String(8), nullable=False),
|
||||
sa.Column("window_occurrence", TS, nullable=True),
|
||||
sa.Column("revision", sa.Integer(), server_default="0", nullable=False),
|
||||
sa.Column("progress", sa.JSON(), nullable=False),
|
||||
sa.Column("step_at", TS, nullable=True),
|
||||
sa.Column("created_at", TS, nullable=False),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),
|
||||
)
|
||||
op.create_index("ix_rollout_runs_plan_id", "rollout_runs", ["plan_id"])
|
||||
op.create_index("ix_rollout_runs_status", "rollout_runs", ["status"])
|
||||
op.create_table(
|
||||
"rollout_run_waves",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("run_id", UUID, nullable=False),
|
||||
sa.Column("index", sa.Integer(), nullable=False),
|
||||
sa.Column("label", sa.String(60), nullable=False),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("baseline", sa.JSON(), nullable=True),
|
||||
sa.Column("gate_result", sa.JSON(), nullable=True),
|
||||
sa.Column("soak_until", TS, nullable=True),
|
||||
sa.Column("gate_attempts", sa.Integer(), server_default="0", nullable=False),
|
||||
sa.Column("approved_by", sa.String(150), nullable=True),
|
||||
sa.Column("started_at", TS, nullable=True),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_rollout_run_waves_run_id", "rollout_run_waves", ["run_id"])
|
||||
op.create_table(
|
||||
"rollout_run_hosts",
|
||||
sa.Column("id", UUID, nullable=False),
|
||||
sa.Column("run_id", UUID, nullable=False),
|
||||
sa.Column("wave_index", sa.Integer(), nullable=False),
|
||||
sa.Column("host_id", UUID, nullable=True),
|
||||
sa.Column("host_name", sa.String(120), nullable=False),
|
||||
sa.Column("cluster_name", sa.String(120), nullable=True),
|
||||
sa.Column("order", sa.Integer(), nullable=False),
|
||||
sa.Column("status", sa.String(16), nullable=False),
|
||||
sa.Column("scheduled_maintenance_id", UUID, nullable=True),
|
||||
sa.Column("version_before", sa.String(40), nullable=True),
|
||||
sa.Column("version_after", sa.String(40), nullable=True),
|
||||
sa.Column("detail", sa.JSON(), nullable=True),
|
||||
sa.Column("started_at", TS, nullable=True),
|
||||
sa.Column("finished_at", TS, nullable=True),
|
||||
sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="SET NULL"),
|
||||
sa.ForeignKeyConstraint(["scheduled_maintenance_id"], ["scheduled_maintenance.id"], ondelete="SET NULL"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),
|
||||
)
|
||||
op.create_index("ix_rollout_run_hosts_run_id", "rollout_run_hosts", ["run_id"])
|
||||
op.create_index("ix_rollout_run_hosts_host_id", "rollout_run_hosts", ["host_id"])
|
||||
op.create_index("ix_rollout_run_hosts_scheduled_maintenance_id", "rollout_run_hosts", ["scheduled_maintenance_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for table in ("rollout_run_hosts", "rollout_run_waves", "rollout_runs", "rollout_wave_groups", "rollout_plan_waves", "rollout_plans"):
|
||||
op.execute(f"DROP TABLE IF EXISTS {table}")
|
||||
op.execute("ALTER TABLE notification_settings DROP COLUMN IF EXISTS rollout_problems_enabled")
|
||||
@@ -13,3 +13,4 @@ respx==0.23.1
|
||||
cryptography==50.0.2
|
||||
alembic==1.20.0
|
||||
PyJWT==2.10.1
|
||||
tzdata==2025.2
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Recurring maintenance windows: occurrence maths (weekly, monthly, DST, exceptions, cut-off)."""
|
||||
from datetime import date, datetime, time, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
|
||||
from app.services import window_service as ws
|
||||
|
||||
UTC = timezone.utc
|
||||
|
||||
|
||||
def _w(**kw) -> MaintenanceWindow:
|
||||
base = dict(name="w", timezone="Europe/Berlin", recurrence="weekly", weekdays=[6], start_time=time(2, 0),
|
||||
duration_minutes=180, start_cutoff_minutes=60, enabled=True, created_by="admin")
|
||||
base.update(kw)
|
||||
w = MaintenanceWindow(**base)
|
||||
w.exceptions = kw.get("exceptions", [])
|
||||
return w
|
||||
|
||||
|
||||
def _starts(w, after, n=4, **kw):
|
||||
return [o.start for o in ws.occurrences(w, after, n, **kw)]
|
||||
|
||||
|
||||
def test_weekly_sunday_in_summer_and_winter_keeps_local_time():
|
||||
w = _w()
|
||||
# 2026-10-18 is a Sunday (CEST, +2), 2026-11-01 a Sunday (CET, +1)
|
||||
starts = _starts(w, datetime(2026, 10, 12, tzinfo=UTC), 4)
|
||||
assert starts[0] == datetime(2026, 10, 18, 0, 0, tzinfo=UTC)
|
||||
assert starts[2] == datetime(2026, 11, 1, 1, 0, tzinfo=UTC)
|
||||
assert all(s.astimezone(ws.ZoneInfo("Europe/Berlin")).hour == 2 for s in starts)
|
||||
|
||||
|
||||
def test_weekly_several_days_in_order():
|
||||
w = _w(weekdays=[1, 3], timezone="UTC", start_time=time(22, 0), duration_minutes=60)
|
||||
starts = _starts(w, datetime(2026, 10, 19, tzinfo=UTC), 3) # Monday
|
||||
assert starts == [datetime(2026, 10, 20, 22, tzinfo=UTC), datetime(2026, 10, 22, 22, tzinfo=UTC),
|
||||
datetime(2026, 10, 27, 22, tzinfo=UTC)]
|
||||
|
||||
|
||||
def test_monthly_nth_weekday_first_and_last():
|
||||
first_tue = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=1, month_weekday=1, timezone="UTC")
|
||||
assert [s.date() for s in _starts(first_tue, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
|
||||
date(2026, 10, 6), date(2026, 11, 3), date(2026, 12, 1)]
|
||||
last_sun = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=-1, month_weekday=6, timezone="UTC")
|
||||
assert [s.date() for s in _starts(last_sun, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
|
||||
date(2026, 10, 25), date(2026, 11, 29), date(2026, 12, 27)]
|
||||
|
||||
|
||||
def test_monthly_fixed_day():
|
||||
w = _w(recurrence="monthly", weekdays=[], month_mode="day", month_day=15, timezone="UTC")
|
||||
assert [s.date() for s in _starts(w, datetime(2026, 10, 16, tzinfo=UTC), 2)] == [date(2026, 11, 15), date(2026, 12, 15)]
|
||||
|
||||
|
||||
def test_spring_forward_missing_time_moves_forward():
|
||||
# 2026-03-29 (last Sunday of March): 02:00-03:00 local does not exist in Europe/Berlin.
|
||||
w = _w(start_time=time(2, 30))
|
||||
occ = ws.occurrences(w, datetime(2026, 3, 28, tzinfo=UTC), 1)[0]
|
||||
assert occ.local_date == date(2026, 3, 29)
|
||||
assert occ.start == datetime(2026, 3, 29, 1, 30, tzinfo=UTC) # = 03:30 CEST
|
||||
assert occ.start.astimezone(ws.ZoneInfo("Europe/Berlin")).strftime("%H:%M") == "03:30"
|
||||
|
||||
|
||||
def test_fall_back_ambiguous_time_uses_the_first_and_duration_is_real_time():
|
||||
# 2026-10-25 (last Sunday of October): 02:00-03:00 local happens twice.
|
||||
w = _w(start_time=time(2, 30), duration_minutes=120)
|
||||
occ = ws.occurrences(w, datetime(2026, 10, 24, tzinfo=UTC), 1)[0]
|
||||
assert occ.start == datetime(2026, 10, 25, 0, 30, tzinfo=UTC) # 02:30 CEST (first pass)
|
||||
assert occ.end - occ.start == timedelta(hours=2)
|
||||
|
||||
|
||||
def test_skip_and_move_exceptions():
|
||||
moved = datetime(2026, 10, 31, 20, 0, tzinfo=UTC) # Saturday evening instead of Sunday night
|
||||
w = _w(exceptions=[
|
||||
MaintenanceWindowException(occurrence_date=date(2026, 10, 25), kind="skip", reason="x", actor="a"),
|
||||
MaintenanceWindowException(occurrence_date=date(2026, 11, 1), kind="move", moved_start=moved, reason="x", actor="a"),
|
||||
])
|
||||
occs = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3)
|
||||
assert [o.local_date for o in occs] == [date(2026, 11, 1), date(2026, 11, 8), date(2026, 11, 15)]
|
||||
assert occs[0].moved and occs[0].start == moved and occs[0].end == moved + timedelta(minutes=180)
|
||||
listed = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3, with_skipped=True)
|
||||
assert [(o.local_date, o.skipped, o.moved) for o in listed] == [
|
||||
(date(2026, 10, 25), True, False), (date(2026, 11, 1), False, True), (date(2026, 11, 8), False, False)]
|
||||
|
||||
|
||||
def test_open_window_is_current_and_cutoff_limits_new_starts():
|
||||
w = _w(timezone="UTC", weekdays=[6], start_time=time(2, 0), duration_minutes=180, start_cutoff_minutes=60)
|
||||
start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
|
||||
assert ws.current(w, start - timedelta(seconds=1)) is None
|
||||
assert ws.current(w, start).start == start
|
||||
assert ws.startable(w, start + timedelta(minutes=59)) is not None
|
||||
assert ws.startable(w, start + timedelta(minutes=60)) is None # past the cut-off: nothing new starts
|
||||
assert ws.current(w, start + timedelta(minutes=179)) is not None # still open, running work may finish
|
||||
assert ws.current(w, start + timedelta(minutes=180)) is None
|
||||
# an occurrence that is open right now is the first one listed
|
||||
assert ws.occurrences(w, start + timedelta(minutes=30), 1)[0].start == start
|
||||
|
||||
|
||||
def test_cutoff_never_exceeds_duration_and_disabled_window_is_never_open():
|
||||
w = _w(timezone="UTC", duration_minutes=30, start_cutoff_minutes=90)
|
||||
start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
|
||||
assert ws.occurrences(w, start, 1)[0].cutoff == start + timedelta(minutes=30)
|
||||
w.enabled = False
|
||||
assert ws.current(w, start) is None and ws.startable(w, start) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,ok", [("Europe/Berlin", True), ("UTC", True), ("Mars/Olympus", False), ("", False), ("../etc", False)])
|
||||
def test_timezone_validation(name, ok):
|
||||
assert ws.valid_timezone(name) is ok
|
||||
|
||||
|
||||
# ── API ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
from sqlalchemy import select # noqa: E402
|
||||
|
||||
from app.database import SessionLocal # noqa: E402
|
||||
from app.models.audit import AuditEvent # noqa: E402
|
||||
|
||||
_WEEKLY = {"name": "Sonntagnacht", "timezone": "Europe/Berlin", "recurrence": "weekly", "weekdays": [6],
|
||||
"start_time": "02:00", "duration_minutes": 180, "start_cutoff_minutes": 60}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_window_crud_and_next_occurrences(client):
|
||||
resp = await client.post("/api/maintenance-windows", json=_WEEKLY)
|
||||
assert resp.status_code == 201, resp.text
|
||||
w = resp.json()
|
||||
assert w["start_time"] == "02:00" and len(w["next"]) == 5 and w["exceptions"] == [] and w["used_by"] == []
|
||||
assert (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()["detail"]["code"] == "WINDOW_NAME_TAKEN"
|
||||
upd = await client.put(f"/api/maintenance-windows/{w['id']}", json={**_WEEKLY, "start_time": "03:15", "start_cutoff_minutes": 999})
|
||||
assert upd.status_code == 200 and upd.json()["start_time"] == "03:15"
|
||||
assert upd.json()["start_cutoff_minutes"] == 180 # never longer than the window itself
|
||||
occ = await client.get(f"/api/maintenance-windows/{w['id']}/occurrences?count=12")
|
||||
assert occ.status_code == 200 and len(occ.json()) == 12
|
||||
assert (await client.delete(f"/api/maintenance-windows/{w['id']}")).status_code == 204
|
||||
assert (await client.get(f"/api/maintenance-windows/{w['id']}")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change,field", [
|
||||
({"timezone": "Mars/Base"}, "timezone"),
|
||||
({"start_time": "25:00"}, "start_time"),
|
||||
({"weekdays": []}, "weekdays"),
|
||||
({"weekdays": [7]}, "weekdays"),
|
||||
({"recurrence": "monthly", "month_mode": "day", "month_day": 31}, "month_day"),
|
||||
({"recurrence": "monthly", "month_mode": "nth_weekday", "month_nth": 5, "month_weekday": 1}, "month_nth"),
|
||||
({"recurrence": "monthly"}, "month_mode"),
|
||||
])
|
||||
@pytest.mark.asyncio
|
||||
async def test_window_validation(client, change, field):
|
||||
resp = await client.post("/api/maintenance-windows", json={**_WEEKLY, **change})
|
||||
assert resp.status_code == 422 and resp.json()["detail"] == {"code": "WINDOW_INVALID", "params": {"field": field}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_exceptions_skip_move_and_audit(client):
|
||||
w = (await client.post("/api/maintenance-windows", json={**_WEEKLY, "timezone": "UTC"})).json()
|
||||
first = date.fromisoformat(w["next"][0]["local_date"])
|
||||
second = date.fromisoformat(w["next"][1]["local_date"])
|
||||
bad = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first + timedelta(days=1)).isoformat(), "kind": "skip"})
|
||||
assert bad.status_code == 422 and bad.json()["detail"]["params"]["field"] == "occurrence_date"
|
||||
past = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first - timedelta(days=14)).isoformat(), "kind": "skip"})
|
||||
assert past.status_code == 422
|
||||
skip = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip", "reason": "Feiertag"})
|
||||
assert skip.status_code == 201, skip.text
|
||||
assert skip.json()["next"][0]["skipped"] is True
|
||||
dup = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip"})
|
||||
assert dup.status_code == 409 and dup.json()["detail"]["code"] == "WINDOW_EXCEPTION_EXISTS"
|
||||
no_target = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move"})
|
||||
assert no_target.status_code == 422 and no_target.json()["detail"]["params"]["field"] == "moved_start"
|
||||
target = datetime.combine(second, time(20, 0), tzinfo=UTC) - timedelta(days=1)
|
||||
moved = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move", "moved_start": target.isoformat()})
|
||||
assert moved.status_code == 201
|
||||
regular = [o for o in moved.json()["next"] if not o["skipped"]]
|
||||
assert regular[0]["moved"] is True and datetime.fromisoformat(regular[0]["start"]) == target
|
||||
eid = moved.json()["exceptions"][1]["id"]
|
||||
after = await client.delete(f"/api/maintenance-windows/{w['id']}/exceptions/{eid}")
|
||||
assert after.status_code == 200 and len(after.json()["exceptions"]) == 1
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "maintenance_window"))).all()
|
||||
assert any(e.action == "exception_skip" and e.details["reason"] == "Feiertag" for e in events)
|
||||
assert any(e.status == "accepted" and e.action == "delete" for e in events)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_window_writes_follow_the_deploy_right(client):
|
||||
from tests.test_operator_role import _operator
|
||||
op = await _operator(client, "winop")
|
||||
try:
|
||||
made = await op.post("/api/maintenance-windows", json={**_WEEKLY, "name": "op-window"})
|
||||
assert made.status_code == 201 # operators may prepare windows while they may deploy
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
for method, path, body in [
|
||||
("POST", "/api/maintenance-windows", {**_WEEKLY, "name": "x"}),
|
||||
("PUT", f"/api/maintenance-windows/{made.json()['id']}", _WEEKLY),
|
||||
("DELETE", f"/api/maintenance-windows/{made.json()['id']}", None),
|
||||
("POST", f"/api/maintenance-windows/{made.json()['id']}/exceptions", {"occurrence_date": "2030-01-06", "kind": "skip"}),
|
||||
]:
|
||||
resp = await op.request(method, path, json=body)
|
||||
assert resp.status_code == 403 and resp.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN", path
|
||||
assert (await op.get("/api/maintenance-windows")).status_code == 200
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
@@ -206,3 +206,53 @@ async def test_fresh_database_needs_no_backup(scratch, monkeypatch):
|
||||
await migrate.run() # würde ohne Backup-Ziel scheitern, wenn eins verlangt würde
|
||||
from pathlib import Path
|
||||
assert not Path(settings.migration_backup_dir).exists()
|
||||
|
||||
|
||||
async def _alembic(engine, action: str, revision: str) -> None:
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(lambda sync_conn: migrate._run_alembic(sync_conn, action, revision))
|
||||
|
||||
|
||||
async def test_0020_upgrade_keeps_existing_settings_and_downgrade_is_clean(scratch):
|
||||
"""1.23 -> 1.24 with a real settings row: the new alert columns get their defaults, the stored webhook
|
||||
and flags stay, the channel table appears; a downgrade removes exactly that again and re-upgrading works."""
|
||||
await _alembic(scratch, "upgrade", "0019")
|
||||
async with scratch.begin() as conn:
|
||||
# Every NOT NULL column of the 0019 table gets a value, so the row is what a real 1.23 install has.
|
||||
required = (await conn.execute(text(
|
||||
"SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'notification_settings' "
|
||||
"AND is_nullable = 'NO' AND column_default IS NULL"
|
||||
))).all()
|
||||
by_type = {"boolean": "true", "integer": "300", "json": "'[]'", "timestamp with time zone": "now()"}
|
||||
values = {name: by_type.get(kind, "'x'") for name, kind in required}
|
||||
values.update({"id": "1", "webhook_enabled": "true", "webhook_url": "'https://hooks.example/x'"})
|
||||
await conn.execute(text(
|
||||
f"INSERT INTO notification_settings ({', '.join(values)}) VALUES ({', '.join(values.values())})"
|
||||
))
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
async with scratch.connect() as conn:
|
||||
row = (await conn.execute(text(
|
||||
"SELECT webhook_enabled, webhook_url, gateway_down_enabled, resource_threshold_enabled, service_down_enabled, "
|
||||
"disk_threshold_pct, load_per_cpu_threshold, alert_confirm_samples, resource_confirm_samples, service_watchlist "
|
||||
"FROM notification_settings"
|
||||
))).one()
|
||||
assert tuple(row) == (True, "https://hooks.example/x", True, False, False, 90, 2.0, 3, 5, None)
|
||||
await conn.execute(text(
|
||||
"INSERT INTO notification_channels (id, name, channel_type, config, created_at, updated_at) "
|
||||
"VALUES (gen_random_uuid(), 'n', 'ntfy', '{\"topic\": \"t\"}', now(), now())"
|
||||
))
|
||||
await conn.commit()
|
||||
assert await _schema_diff(scratch) == []
|
||||
|
||||
await _alembic(scratch, "downgrade", "0019")
|
||||
async with scratch.connect() as conn:
|
||||
assert (await conn.execute(text("SELECT to_regclass('notification_channels')"))).scalar() is None
|
||||
columns = set((await conn.execute(text(
|
||||
"SELECT column_name FROM information_schema.columns WHERE table_name = 'notification_settings'"
|
||||
))).scalars().all())
|
||||
assert "gateway_down_enabled" not in columns and "service_watchlist" not in columns
|
||||
assert (await conn.execute(text("SELECT webhook_url FROM notification_settings"))).scalar() == "https://hooks.example/x"
|
||||
|
||||
await _alembic(scratch, "upgrade", "head")
|
||||
assert await _version(scratch) == _head()
|
||||
assert await _schema_diff(scratch) == []
|
||||
@@ -0,0 +1,717 @@
|
||||
"""Monitoring alerts (1.24): gateway health, resource thresholds, stopped services, extra channels."""
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.notification import NotificationChannel, NotificationProblem, NotificationSettings
|
||||
from app.models.resource_sample import HostResourceSample
|
||||
from app.services import (
|
||||
gateway_health_service, monitor_checks, notification_channels, notification_secret_store, resource_monitor_service,
|
||||
url_guard,
|
||||
)
|
||||
from app.services.notification_delivery import NotificationDeliveryError
|
||||
from app.services.notification_service import run_notification_checks
|
||||
from tests.test_notifications import _create_host
|
||||
from tests.test_resource_monitor import _mock_resource_endpoints, _mock_vip
|
||||
|
||||
# What routes/gateway/status returns on a real 26.7 firewall (test-fw-3, live), plus a degraded and a dead gateway.
|
||||
STATUS_PAYLOAD = {"status": "ok", "items": [
|
||||
{"name": "WAN_GW", "address": "203.0.113.1", "status": "none", "loss": "0.0 %", "delay": "1.9 ms", "stddev": "0.3 ms", "monitor": "203.0.113.1"},
|
||||
{"name": "LTE_GW", "address": "192.0.2.9", "status": "delay+loss", "loss": "12.5 %", "delay": "310.2 ms", "stddev": "40.0 ms", "monitor": "192.0.2.9"},
|
||||
{"name": "DOWN_GW", "address": "192.0.2.1", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~", "monitor": "192.0.2.1"},
|
||||
{"name": "WAN_DHCP6", "address": "~", "status": "none", "loss": "~", "delay": "~", "stddev": "~", "monitor": "~"},
|
||||
{"name": "OFF_GW", "address": "192.0.2.5", "status": "force_down", "loss": "~", "delay": "~", "stddev": "~", "monitor": "~"},
|
||||
]}
|
||||
|
||||
|
||||
# ── Gateway parsing ──────────────────────────────────────────────────────────
|
||||
|
||||
def test_parse_gateway_status_maps_states_and_numbers():
|
||||
by_name = {g.name: g for g in gateway_health_service.parse_gateway_status(STATUS_PAYLOAD)}
|
||||
assert by_name["WAN_GW"].state == "online" and by_name["WAN_GW"].loss == 0.0 and by_name["WAN_GW"].delay == 1.9
|
||||
assert by_name["WAN_GW"].stddev == 0.3
|
||||
assert by_name["LTE_GW"].state == "degraded" and by_name["LTE_GW"].loss == 12.5
|
||||
assert by_name["DOWN_GW"].state == "offline" and by_name["DOWN_GW"].delay is None
|
||||
# nothing measured -> unknown, never "online" and never 0 ms
|
||||
assert by_name["WAN_DHCP6"].state == "unknown" and by_name["WAN_DHCP6"].loss is None and by_name["WAN_DHCP6"].address is None
|
||||
assert by_name["OFF_GW"].state == "forced_down"
|
||||
|
||||
|
||||
def test_dead_gateway_has_no_round_trip_time():
|
||||
# dpinger reports "100.0 %" loss together with "0.0 ms" when nothing answers: 0 ms must not look fast
|
||||
payload = {"items": [{"name": "GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms", "monitor": "192.0.2.1"}]}
|
||||
gw = gateway_health_service.parse_gateway_status(payload)[0]
|
||||
assert gw.state == "offline" and gw.loss == 100.0 and gw.delay is None and gw.stddev is None
|
||||
|
||||
|
||||
def test_parse_gateway_status_tolerates_failed_answer():
|
||||
assert gateway_health_service.parse_gateway_status({"items": [], "status": "failed"}) == []
|
||||
assert gateway_health_service.parse_gateway_status(None) == []
|
||||
|
||||
|
||||
def test_is_bad_thresholds():
|
||||
by_name = {g.name: g for g in gateway_health_service.parse_gateway_status(STATUS_PAYLOAD)}
|
||||
assert gateway_health_service.is_bad(by_name["DOWN_GW"])
|
||||
assert not gateway_health_service.is_bad(by_name["OFF_GW"]) # switched off on purpose
|
||||
assert not gateway_health_service.is_bad(by_name["WAN_DHCP6"])
|
||||
assert not gateway_health_service.is_bad(by_name["LTE_GW"]) # degraded by OPNsense' limits only, own thresholds off
|
||||
assert gateway_health_service.is_bad(by_name["LTE_GW"], loss_threshold_pct=10)
|
||||
assert gateway_health_service.is_bad(by_name["LTE_GW"], delay_threshold_ms=300)
|
||||
assert not gateway_health_service.is_bad(by_name["WAN_GW"], loss_threshold_pct=10, delay_threshold_ms=300)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_gateway_health_reads_live(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(200, json=STATUS_PAYLOAD)
|
||||
resp = await client.get(f"/api/hosts/{host['id']}/gateways")
|
||||
assert resp.status_code == 200, resp.text
|
||||
rows = {g["name"]: g for g in resp.json()}
|
||||
assert rows["DOWN_GW"]["state"] == "offline" and rows["WAN_GW"]["delay"] == 1.9
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(500)
|
||||
assert (await client.get(f"/api/hosts/{host['id']}/gateways")).status_code == 502
|
||||
|
||||
|
||||
# ── Sampler stores the extra readings ────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sampler_stores_gateways_services_swap_and_cpu(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
await _create_host(client, "fw1", "https://fw1.test")
|
||||
_mock_resource_endpoints(respx_mock)
|
||||
respx_mock.get("https://fw1.test/api/diagnostics/system/systemSwap").respond(200, json={"swap": [{"device": "/dev/da0p3", "total": "8388608", "used": "2097152"}]})
|
||||
respx_mock.get("https://fw1.test/api/diagnostics/cpu_usage/getCPUType").respond(200, json=["QEMU Virtual CPU version 2.5+ (6 cores, 6 threads)"])
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").respond(200, json=STATUS_PAYLOAD)
|
||||
respx_mock.get("https://fw1.test/api/core/service/search").respond(200, json={"rows": [
|
||||
{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
|
||||
{"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0},
|
||||
{"id": "pf", "name": "pf", "locked": 1, "running": 1},
|
||||
{"id": "3f2a", "name": "openvpn", "locked": 0, "running": 0},
|
||||
]})
|
||||
async with SessionLocal() as db:
|
||||
await resource_monitor_service.poll_all_hosts(db)
|
||||
raw = (await db.scalars(select(HostResourceSample))).one().raw
|
||||
assert raw["swap_used_pct"] == 25.0 and raw["cpu_threads"] == 6
|
||||
assert {g["name"]: g["state"] for g in raw["gateways"]}["DOWN_GW"] == "offline"
|
||||
assert raw["services"] == {"unbound": 1, "dhcpd6": 0, "openvpn/3f2a": 0} # locked services are not recorded
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sampler_survives_missing_optional_endpoints(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
await _create_host(client, "fw1", "https://fw1.test")
|
||||
_mock_resource_endpoints(respx_mock) # nothing for swap/cpu/gateways/services -> unmocked routes fail
|
||||
async with SessionLocal() as db:
|
||||
result = await resource_monitor_service.poll_all_hosts(db)
|
||||
row = (await db.scalars(select(HostResourceSample))).one()
|
||||
assert result["sampled"] == 1 and row.mem_used_pct == 25.0
|
||||
assert row.raw["gateways"] is None and row.raw["services"] is None and row.raw["swap_used_pct"] is None
|
||||
|
||||
|
||||
# ── Debouncing ───────────────────────────────────────────────────────────────
|
||||
|
||||
def test_verdict_needs_agreement_and_keeps_state_otherwise():
|
||||
v = monitor_checks.verdict
|
||||
assert v([True, True, True], 3, False) is True
|
||||
assert v([True, True, False], 3, False) is False # no single spike / short blip
|
||||
assert v([True, True], 3, False) is False # not enough history yet
|
||||
assert v([False, False, False], 3, True) is False # needs the full window to close
|
||||
assert v([False, True, False], 3, True) is True # flapping keeps an open alert open
|
||||
assert v([True, None, True], 3, False) is False # unreadable sample is no evidence
|
||||
assert v([None, None, None], 3, True) is True # ... and does not clear either
|
||||
assert v([True], 1, False) is True
|
||||
|
||||
|
||||
async def _add_samples(host_id, rows: list[dict], *, age_minutes: int = 0) -> None:
|
||||
"""rows: newest first; each {mem, load, disk, raw...}. One sample per minute, the newest ``age_minutes`` old."""
|
||||
now = datetime.now(timezone.utc)
|
||||
async with SessionLocal() as db:
|
||||
for i, row in enumerate(rows):
|
||||
raw = {k: v for k, v in row.items() if k not in ("mem", "load")}
|
||||
db.add(HostResourceSample(
|
||||
host_id=host_id, sampled_at=now - timedelta(minutes=age_minutes + i),
|
||||
mem_used_pct=row.get("mem"), load_avg_1m=row.get("load"), disk_used_pct=None, raw=raw,
|
||||
))
|
||||
await db.commit()
|
||||
|
||||
|
||||
def _gw(state="online", loss=0.0, delay=2.0, name="WAN_GW", status=None):
|
||||
return {"name": name, "status": status or {"offline": "down", "online": "none"}.get(state, state), "state": state,
|
||||
"loss": loss, "delay": delay, "stddev": 0.1}
|
||||
|
||||
|
||||
async def _set(**fields):
|
||||
async with SessionLocal() as db:
|
||||
row = await db.get(NotificationSettings, 1) or NotificationSettings(id=1)
|
||||
for k, v in fields.items():
|
||||
setattr(row, k, v)
|
||||
db.add(row)
|
||||
await db.commit()
|
||||
|
||||
|
||||
_ALL_FLAGS = [
|
||||
"host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled", "revert_failed_enabled", "maintenance_unknown_enabled",
|
||||
"carp_health_enabled", "nat_drift_enabled", "network_drift_enabled", "carp_auto_revert_failed_enabled",
|
||||
"scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled",
|
||||
]
|
||||
|
||||
|
||||
async def _only(*enabled: str):
|
||||
"""Run run_notification_checks with just these checks on (the mocked firewalls are unreachable, which would
|
||||
otherwise also open host_offline problems)."""
|
||||
await _set(**{flag: flag in enabled for flag in _ALL_FLAGS})
|
||||
|
||||
|
||||
async def _open_count(db, problem_type: str | None = None) -> int:
|
||||
query = select(NotificationProblem).where(NotificationProblem.status == "open")
|
||||
if problem_type:
|
||||
query = query.where(NotificationProblem.problem_type == problem_type)
|
||||
return len((await db.scalars(query)).all())
|
||||
|
||||
|
||||
async def _gateway_problems():
|
||||
async with SessionLocal() as db:
|
||||
return await monitor_checks.check_gateway_down(db)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_down_needs_consecutive_samples(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"gateways": [_gw("offline", loss=100.0, delay=None)]}, {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}])
|
||||
assert dict(await _gateway_problems()) == {} # only two bad samples in a row
|
||||
|
||||
await _add_samples(hid, [{"gateways": [_gw("offline", 100.0, None)]}] * 3, age_minutes=-3) # three newer bad ones
|
||||
problems = await _gateway_problems()
|
||||
assert list(problems) == [f"{hid}:WAN_GW"]
|
||||
ctx = problems[f"{hid}:WAN_GW"]
|
||||
assert ctx["host_name"] == "fw1" and ctx["gateway"] == "WAN_GW" and ctx["grund"] == "Gateway offline"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_alert_resolves_only_after_full_recovery_and_ignores_unknown(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
bad, good = {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}
|
||||
await _only("gateway_down_enabled")
|
||||
await _add_samples(hid, [bad, bad, bad, good, good])
|
||||
async with SessionLocal() as db:
|
||||
await run_notification_checks(db)
|
||||
assert await _open_count(db) == 1
|
||||
|
||||
# one good sample after the outage, then a relapse: still open (flapping link must not re-notify)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [bad, good, bad, bad, good])
|
||||
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"]
|
||||
# a sample that could not read the gateways does not clear it either
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"gateways": None}] * 3)
|
||||
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"]
|
||||
# three good ones close it
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [good, good, good])
|
||||
assert dict(await _gateway_problems()) == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_thresholds_and_forced_down(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
lossy = {"gateways": [_gw("degraded", loss=35.0, delay=20.0, status="loss"), _gw("forced_down", None, None, name="OFF", status="force_down")]}
|
||||
await _add_samples(hid, [lossy] * 3)
|
||||
assert list(await _gateway_problems()) == [f"{hid}:WAN_GW"] # 35 % >= 20 %; the switched-off gateway is not reported
|
||||
await _set(gateway_loss_threshold_pct=0)
|
||||
assert dict(await _gateway_problems()) == {} # own loss threshold off; OPNsense says "degraded", not "down"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stale_samples_are_not_judged_and_keep_open_problems(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
bad = {"gateways": [_gw("offline", 100.0, None)]}
|
||||
await _only("gateway_down_enabled")
|
||||
await _add_samples(hid, [bad] * 3)
|
||||
async with SessionLocal() as db:
|
||||
await run_notification_checks(db)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"gateways": [_gw()]}] * 3, age_minutes=30) # the host stopped answering 30 min ago
|
||||
async with SessionLocal() as db:
|
||||
result = await run_notification_checks(db)
|
||||
assert result["resolved"] == 0
|
||||
assert await _open_count(db) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_alert_is_delivered_once_and_resolution_follows(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _only("gateway_down_enabled")
|
||||
await _set(webhook_enabled=True, webhook_url="https://hooks.test/n")
|
||||
hook = respx_mock.post("https://hooks.test/n").mock(return_value=httpx.Response(200))
|
||||
bad, good = {"gateways": [_gw("offline", 100.0, None)]}, {"gateways": [_gw()]}
|
||||
await _add_samples(hid, [bad] * 3)
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 0
|
||||
assert hook.call_count == 1 and b"Gateway" in hook.calls.last.request.content
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [good] * 3)
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["resolved"] == 1
|
||||
assert hook.call_count == 2 and b"behoben" in hook.calls.last.request.content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_check_is_on_by_default_and_the_others_are_off(client):
|
||||
data = (await client.get("/api/notifications/settings")).json()
|
||||
assert data["gateway_down_enabled"] is True
|
||||
assert data["resource_threshold_enabled"] is False and data["service_down_enabled"] is False
|
||||
assert data["service_watchlist"][:2] == ["unbound", "dnsmasq"]
|
||||
|
||||
|
||||
# ── Resource thresholds ──────────────────────────────────────────────────────────
|
||||
|
||||
DISK_OK = [{"device": "zroot/ROOT/default", "type": "zfs", "used_pct": 40, "mountpoint": "/"}, {"device": "devfs", "type": "devfs", "used_pct": 100, "mountpoint": "/dev"}]
|
||||
DISK_FULL = [{"device": "zroot/ROOT/default", "type": "zfs", "used_pct": 93, "mountpoint": "/"},
|
||||
{"device": "zroot/var/log", "type": "zfs", "used_pct": 91, "mountpoint": "/var/log"},
|
||||
{"device": "devfs", "type": "devfs", "used_pct": 100, "mountpoint": "/dev"}]
|
||||
|
||||
|
||||
async def _resource_problems():
|
||||
async with SessionLocal() as db:
|
||||
return await monitor_checks.check_resource_threshold(db)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_disk_threshold_one_alert_per_pool_and_ignores_devfs(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"disk": DISK_FULL, "mem": 10, "load": 0.1}] * 5)
|
||||
problems = await _resource_problems()
|
||||
assert list(problems) == [f"{hid}:disk:zroot"]
|
||||
assert problems[f"{hid}:disk:zroot"]["wert_prozent"] == 93 and problems[f"{hid}:disk:zroot"]["schwelle_prozent"] == 90
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_memory_spike_does_not_alert_but_sustained_use_does(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
calm = {"disk": DISK_OK, "mem": 40, "load": 0.1}
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 99, "load": 0.1}, calm, calm, calm, calm])
|
||||
assert dict(await _resource_problems()) == {}
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 96, "load": 0.1}] * 5)
|
||||
assert list(await _resource_problems()) == [f"{hid}:memory"]
|
||||
await _set(memory_threshold_pct=0) # 0 = off
|
||||
assert dict(await _resource_problems()) == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_swap_and_cpu_per_thread(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
# load 6 on 2 threads = 3 per thread (>= 2.0); load 6 on 8 threads would be fine
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 6.0, "swap_used_pct": 70.0, "cpu_threads": 2}] * 5)
|
||||
assert sorted(await _resource_problems()) == sorted([f"{hid}:cpu", f"{hid}:swap"])
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 6.0, "swap_used_pct": None, "cpu_threads": 8}] * 5)
|
||||
assert dict(await _resource_problems()) == {}
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
# unknown thread count: the load cannot be judged, so it is not reported
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 50.0, "swap_used_pct": None, "cpu_threads": None}] * 5)
|
||||
assert dict(await _resource_problems()) == {}
|
||||
|
||||
|
||||
# ── Services ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _service_problems():
|
||||
async with SessionLocal() as db:
|
||||
return await monitor_checks.check_service_down(db)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_down_only_after_it_was_seen_running(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
up = {"services": {"unbound": 1, "dhcpd6": 0, "cron": 0}}
|
||||
down = {"services": {"unbound": 0, "dhcpd6": 0, "cron": 0}}
|
||||
await _add_samples(hid, [down, down, down, up, up])
|
||||
problems = await _service_problems()
|
||||
# unbound stopped; dhcpd6 never ran; cron is not on the watchlist
|
||||
assert list(problems) == [f"{hid}:unbound"]
|
||||
assert problems[f"{hid}:unbound"]["dienst"] == "unbound"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_that_never_ran_or_stopped_long_ago_is_not_reported(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"services": {"unbound": 0}}] * 6)
|
||||
assert dict(await _service_problems()) == {}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_alert_resolves_when_it_runs_again_or_leaves_the_watchlist(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
up, down = {"services": {"unbound": 1}}, {"services": {"unbound": 0}}
|
||||
await _only("service_down_enabled")
|
||||
await _add_samples(hid, [down, down, down, up])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1
|
||||
await _set(service_watchlist=["ntpd"])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["resolved"] == 1
|
||||
await _set(service_watchlist=["unbound"])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1 # reopens (the row is reused)
|
||||
async with SessionLocal() as db:
|
||||
await db.execute(HostResourceSample.__table__.delete())
|
||||
await db.commit()
|
||||
await _add_samples(hid, [up, up, up, down])
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["resolved"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_service_watchlist_validation(client):
|
||||
ok = await client.put("/api/notifications/settings", json={"service_watchlist": [" unbound ", "openvpn", "unbound"]})
|
||||
assert ok.status_code == 200 and ok.json()["service_watchlist"] == ["unbound", "openvpn"]
|
||||
assert (await client.put("/api/notifications/settings", json={"service_watchlist": ["bad name!"]})).status_code == 422
|
||||
assert (await client.put("/api/notifications/settings", json={"disk_threshold_pct": 101})).status_code == 422
|
||||
assert (await client.put("/api/notifications/settings", json={"alert_confirm_samples": 0})).status_code == 422
|
||||
|
||||
|
||||
# ── Metrics and dashboard ─────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_metrics_and_dashboard_show_gateways_and_services(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test", cluster_name=None)
|
||||
hid = uuid.UUID(host["id"])
|
||||
await _add_samples(hid, [{"disk": DISK_OK, "mem": 10, "load": 0.1, "swap_used_pct": 5.0,
|
||||
"gateways": [_gw(), _gw("offline", 100.0, None, name="DOWN_GW")],
|
||||
"services": {"unbound": 1, "openvpn/3f2a": 0, "cron": 0}}])
|
||||
text = (await client.get("/api/metrics")).text
|
||||
assert 'opnexus_host_gateway_up{host="fw1",cluster="",gateway="WAN_GW"} 1' in text
|
||||
assert 'opnexus_host_gateway_up{host="fw1",cluster="",gateway="DOWN_GW"} 0' in text
|
||||
assert 'opnexus_host_gateway_delay_seconds{host="fw1",cluster="",gateway="WAN_GW"} 0.002' in text
|
||||
assert 'opnexus_host_gateway_loss_percent{host="fw1",cluster="",gateway="DOWN_GW"} 100' in text
|
||||
assert 'opnexus_host_swap_used_percent{host="fw1",cluster=""} 5' in text
|
||||
assert 'opnexus_host_service_running{host="fw1",cluster="",service="unbound"} 1' in text
|
||||
assert 'service="openvpn",instance="3f2a"} 0' in text
|
||||
assert 'service="cron"' not in text # not on the watchlist
|
||||
|
||||
summary = (await client.get("/api/dashboard")).json()
|
||||
gateways = {g["name"]: g for g in summary["hosts"][0]["gateways"]}
|
||||
assert gateways["DOWN_GW"]["state"] == "offline"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashboard_has_no_gateways_without_a_fresh_sample(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
await _add_samples(uuid.UUID(host["id"]), [{"gateways": [_gw()]}], age_minutes=20)
|
||||
assert (await client.get("/api/dashboard")).json()["hosts"][0]["gateways"] is None
|
||||
|
||||
|
||||
# ── URL guard ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("url", [
|
||||
"http://169.254.169.254/latest/meta-data", "http://127.0.0.1:8000/x", "http://localhost/x", "http://[::1]/x",
|
||||
"http://0.0.0.0/x", "http://[::ffff:169.254.169.254]/x", "ftp://ntfy.example/x", "file:///etc/passwd",
|
||||
"https://user:[email protected]/", "https://", "not a url", "http://224.0.0.1/x", "http://[fe80::1]/x",
|
||||
])
|
||||
def test_url_guard_refuses(url):
|
||||
with pytest.raises(ValueError):
|
||||
url_guard.validate_url(url)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url", ["https://ntfy.sh", "http://10.20.30.5:8080/path", "https://gotify.lan.example/", "http://192.168.1.10"])
|
||||
def test_url_guard_allows_private_and_public_targets(url):
|
||||
assert url_guard.validate_url(url) == url
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_url_guard_refuses_names_resolving_to_metadata_address(monkeypatch):
|
||||
async def fake(self, host, port, **kw):
|
||||
return [(2, 1, 6, "", ("169.254.169.254", 0))]
|
||||
monkeypatch.setattr("asyncio.base_events.BaseEventLoop.getaddrinfo", fake)
|
||||
with pytest.raises(ValueError):
|
||||
await url_guard.ensure_resolved_safe("https://innocent.example/hook")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_webhook_url_is_validated_when_saved_and_sent(client, respx_mock):
|
||||
resp = await client.put("/api/notifications/settings", json={"webhook_url": "http://169.254.169.254/x"})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "NOTIFICATION_WEBHOOK_URL_INVALID"
|
||||
row = NotificationSettings(webhook_url="http://127.0.0.1:9/x")
|
||||
from app.services import notification_delivery
|
||||
with pytest.raises(NotificationDeliveryError):
|
||||
await notification_delivery.send_webhook(row, NotificationProblem(problem_type="test", dedup_key="t", context={}), "opened")
|
||||
|
||||
|
||||
# ── Channels: formats ────────────────────────────────────────────────────────────
|
||||
|
||||
PROBLEM = NotificationProblem(problem_type="gateway_down", dedup_key="k", context={"host_name": "fw1", "gateway": "WAN_GW", "grund": "Gateway offline"})
|
||||
|
||||
|
||||
def _channel(kind: str, config: dict, secret: str | None, name: str = "c1") -> NotificationChannel:
|
||||
ch = NotificationChannel(id=uuid.uuid4(), name=name, channel_type=kind, enabled=True, config=config, secret_set=secret is not None)
|
||||
if secret is not None:
|
||||
notification_secret_store.save_secret(notification_channels.secret_name(ch.id), secret)
|
||||
return ch
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ntfy_posts_json_to_root_with_bearer(respx_mock):
|
||||
ch = _channel("ntfy", {"server_url": "https://ntfy.test", "topic": "alerts", "priority": 5}, "tk_secret")
|
||||
route = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200, json={"id": "x"}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
req = route.calls.last.request
|
||||
assert req.headers["authorization"] == "Bearer tk_secret"
|
||||
body = httpx.Response(200, content=req.content).json()
|
||||
assert body["topic"] == "alerts" and body["priority"] == 5 and body["tags"] == ["warning"]
|
||||
assert body["title"] == "OPNexus: Gateway gestört" and "gateway: WAN_GW" in body["message"]
|
||||
await notification_channels.send(ch, PROBLEM, "resolved")
|
||||
resolved = httpx.Response(200, content=route.calls.last.request.content).json()
|
||||
assert resolved["priority"] == 3 and resolved["title"].endswith("behoben") and resolved["tags"] == ["white_check_mark"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ntfy_without_token_sends_no_auth_header(respx_mock):
|
||||
ch = _channel("ntfy", {"server_url": "https://ntfy.test", "topic": "alerts"}, None)
|
||||
route = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
assert "authorization" not in route.calls.last.request.headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gotify_uses_key_header_not_query(respx_mock):
|
||||
ch = _channel("gotify", {"server_url": "https://gotify.test/", "priority": 8}, "AppTok123")
|
||||
route = respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(200, json={"id": 1}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
req = route.calls.last.request
|
||||
assert req.headers["x-gotify-key"] == "AppTok123" and "AppTok123" not in str(req.url)
|
||||
body = httpx.Response(200, content=req.content).json()
|
||||
assert body["priority"] == 8 and body["title"] == "OPNexus: Gateway gestört"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_telegram_sendmessage_and_ok_flag(respx_mock):
|
||||
ch = _channel("telegram", {"chat_id": "-1001234", "api_base": "https://api.telegram.test"}, "123456:ABC-token")
|
||||
route = respx_mock.post("https://api.telegram.test/bot123456:ABC-token/sendMessage").mock(
|
||||
return_value=httpx.Response(200, json={"ok": True, "result": {}}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
body = httpx.Response(200, content=route.calls.last.request.content).json()
|
||||
assert body["chat_id"] == "-1001234" and body["text"].startswith("OPNexus: Gateway gestört\n\n") and len(body["text"]) <= 4096
|
||||
route.mock(return_value=httpx.Response(200, json={"ok": False, "description": "chat not found"}))
|
||||
with pytest.raises(NotificationDeliveryError):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
|
||||
|
||||
def test_telegram_text_is_clipped_to_the_api_limit():
|
||||
big = NotificationProblem(problem_type="gateway_down", dedup_key="k", context={"x": "y" * 9000})
|
||||
text = notification_channels.format_telegram({"chat_id": "1"}, big, "opened")["text"]
|
||||
assert len(text) == 4096 and text.endswith("…")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_teams_adaptive_card_envelope(respx_mock):
|
||||
url = "https://prod-12.westeurope.logic.azure.test/workflows/abc/triggers/manual/paths/invoke?sig=SECRETSIG"
|
||||
ch = _channel("teams", {}, url)
|
||||
route = respx_mock.post(url.split("?")[0]).mock(return_value=httpx.Response(202))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
body = httpx.Response(200, content=route.calls.last.request.content).json()
|
||||
assert body["type"] == "message"
|
||||
att = body["attachments"][0]
|
||||
assert att["contentType"] == "application/vnd.microsoft.card.adaptive" and att["content"]["type"] == "AdaptiveCard"
|
||||
assert att["content"]["body"][0]["text"] == "OPNexus: Gateway gestört"
|
||||
assert "sig=SECRETSIG" in str(route.calls.last.request.url)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_matrix_put_with_transaction_id_and_encoded_room(respx_mock):
|
||||
ch = _channel("matrix", {"homeserver_url": "https://matrix.test", "room_id": "!abc123:matrix.test"}, "syt_token")
|
||||
route = respx_mock.put(url__regex=r"^https://matrix\.test/_matrix/client/v3/rooms/%21abc123%3Amatrix\.test/send/m\.room\.message/[0-9a-f]{32}$").mock(
|
||||
return_value=httpx.Response(200, json={"event_id": "$e"}))
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
first, second = (c.request for c in route.calls)
|
||||
assert first.headers["authorization"] == "Bearer syt_token"
|
||||
assert first.url.path != second.url.path # a new transaction id per message
|
||||
body = httpx.Response(200, content=first.content).json()
|
||||
assert body["msgtype"] == "m.text" and body["body"].startswith("OPNexus: Gateway gestört")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failures_and_logs_never_contain_the_credential(respx_mock, caplog):
|
||||
token = "123456:SECRET-BOT-TOKEN"
|
||||
ch = _channel("telegram", {"chat_id": "42", "api_base": "https://api.telegram.test"}, token)
|
||||
respx_mock.post(f"https://api.telegram.test/bot{token}/sendMessage").mock(return_value=httpx.Response(401, json={"ok": False}))
|
||||
with caplog.at_level(logging.DEBUG):
|
||||
with pytest.raises(NotificationDeliveryError) as exc_info:
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
respx_mock.post(f"https://api.telegram.test/bot{token}/sendMessage").mock(side_effect=httpx.ConnectError("boom"))
|
||||
with pytest.raises(NotificationDeliveryError) as exc_info2:
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
assert "HTTP 401" in str(exc_info.value) and "ConnectError" in str(exc_info2.value)
|
||||
for text in (str(exc_info.value), str(exc_info2.value), caplog.text):
|
||||
assert "SECRET-BOT-TOKEN" not in text
|
||||
assert exc_info.value.__cause__ is None and exc_info.value.__suppress_context__
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_or_unreadable_secret_fails_closed(respx_mock):
|
||||
ch = _channel("gotify", {"server_url": "https://gotify.test"}, "tok")
|
||||
notification_secret_store.delete_secret(notification_channels.secret_name(ch.id))
|
||||
with pytest.raises(NotificationDeliveryError, match="nicht lesbar"):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
ch.secret_set = False
|
||||
with pytest.raises(NotificationDeliveryError, match="fehlen"):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redirects_are_not_followed(respx_mock):
|
||||
ch = _channel("gotify", {"server_url": "https://gotify.test"}, "tok")
|
||||
respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(302, headers={"location": "http://169.254.169.254/"}))
|
||||
with pytest.raises(NotificationDeliveryError, match="HTTP 302"):
|
||||
await notification_channels.send(ch, PROBLEM, "opened")
|
||||
|
||||
|
||||
# ── Channels: API ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_crud_keeps_the_secret_out_of_database_and_responses(client, respx_mock):
|
||||
resp = await client.post("/api/notifications/channels", json={
|
||||
"name": "Telegram Ops", "channel_type": "telegram", "config": {"chat_id": "-100123"}, "secret": "111:TOKEN-VALUE",
|
||||
})
|
||||
assert resp.status_code == 201, resp.text
|
||||
created = resp.json()
|
||||
assert created["secret_set"] is True and "TOKEN-VALUE" not in resp.text
|
||||
assert created["config"] == {"chat_id": "-100123", "api_base": "https://api.telegram.org"}
|
||||
cid = created["id"]
|
||||
assert notification_secret_store.read_secret(f"channel_{cid}") == "111:TOKEN-VALUE"
|
||||
async with SessionLocal() as db:
|
||||
row = await db.get(NotificationChannel, uuid.UUID(cid))
|
||||
assert "TOKEN-VALUE" not in str(row.config) and "TOKEN-VALUE" not in repr(row.__dict__)
|
||||
assert "TOKEN-VALUE" not in (await client.get("/api/notifications/channels")).text
|
||||
|
||||
# test button
|
||||
route = respx_mock.post("https://api.telegram.org/bot111:TOKEN-VALUE/sendMessage").mock(return_value=httpx.Response(200, json={"ok": True}))
|
||||
assert (await client.post(f"/api/notifications/channels/{cid}/test")).json() == {"success": True}
|
||||
assert b"Testbenachrichtigung" in route.calls.last.request.content
|
||||
route.mock(return_value=httpx.Response(500))
|
||||
failed = await client.post(f"/api/notifications/channels/{cid}/test")
|
||||
assert failed.status_code == 502 and failed.json()["detail"]["code"] == "NOTIFICATION_CHANNEL_TEST_FAILED"
|
||||
assert "TOKEN-VALUE" not in failed.text
|
||||
|
||||
# update: rename, disable, rotate secret, clear secret
|
||||
upd = await client.put(f"/api/notifications/channels/{cid}", json={"name": "Telegram", "enabled": False, "secret": "222:NEW"})
|
||||
assert upd.status_code == 200 and upd.json()["enabled"] is False
|
||||
assert notification_secret_store.read_secret(f"channel_{cid}") == "222:NEW"
|
||||
assert (await client.put(f"/api/notifications/channels/{cid}", json={"clear_secret": True})).json()["secret_set"] is False
|
||||
assert notification_secret_store.read_secret(f"channel_{cid}") is None
|
||||
|
||||
assert (await client.delete(f"/api/notifications/channels/{cid}")).status_code == 204
|
||||
assert (await client.get("/api/notifications/channels")).json() == []
|
||||
assert (await client.delete(f"/api/notifications/channels/{cid}")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [
|
||||
{"name": "a", "channel_type": "ntfy", "config": {"topic": "bad topic!"}},
|
||||
{"name": "a", "channel_type": "ntfy", "config": {"topic": "ok", "server_url": "http://169.254.169.254"}},
|
||||
{"name": "a", "channel_type": "ntfy", "config": {"topic": "ok", "priority": 9}},
|
||||
{"name": "a", "channel_type": "gotify", "config": {"server_url": "https://g.test"}}, # token missing
|
||||
{"name": "a", "channel_type": "gotify", "config": {}, "secret": "t"}, # server missing
|
||||
{"name": "a", "channel_type": "telegram", "config": {"chat_id": "abc"}, "secret": "t"},
|
||||
{"name": "a", "channel_type": "teams", "config": {}, "secret": "http://insecure.test/hook"},
|
||||
{"name": "a", "channel_type": "teams", "config": {}},
|
||||
{"name": "a", "channel_type": "matrix", "config": {"homeserver_url": "https://m.test", "room_id": "#alias:m.test"}, "secret": "t"},
|
||||
{"name": "a", "channel_type": "carrier-pigeon", "config": {}},
|
||||
{"name": " ", "channel_type": "ntfy", "config": {"topic": "ok"}}, # name only blanks
|
||||
])
|
||||
async def test_channel_validation(client, payload):
|
||||
resp = await client.post("/api/notifications/channels", json=payload)
|
||||
assert resp.status_code == 422, resp.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_names_are_unique_and_ntfy_needs_no_secret(client):
|
||||
body = {"name": "Phone", "channel_type": "ntfy", "config": {"topic": "opnexus-alerts"}}
|
||||
first = await client.post("/api/notifications/channels", json=body)
|
||||
assert first.status_code == 201 and first.json()["secret_set"] is False
|
||||
assert first.json()["config"]["server_url"] == "https://ntfy.sh" and first.json()["config"]["priority"] == 4
|
||||
dup = await client.post("/api/notifications/channels", json=body)
|
||||
assert dup.status_code == 409 and dup.json()["detail"]["code"] == "NOTIFICATION_CHANNEL_NAME_TAKEN"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_operator_cannot_change_channels_and_does_not_see_their_settings(client):
|
||||
from tests.test_operator_role import PW, _operator
|
||||
created = await client.post("/api/notifications/channels", json={"name": "Phone", "channel_type": "ntfy", "config": {"topic": "secret-topic"}})
|
||||
op = await _operator(client, "chan-op")
|
||||
try:
|
||||
for method, path, body in [
|
||||
("POST", "/api/notifications/channels", {"name": "x", "channel_type": "ntfy", "config": {"topic": "t"}}),
|
||||
("PUT", f"/api/notifications/channels/{created.json()['id']}", {"enabled": False}),
|
||||
("DELETE", f"/api/notifications/channels/{created.json()['id']}", None),
|
||||
("POST", f"/api/notifications/channels/{created.json()['id']}/test", {}),
|
||||
]:
|
||||
resp = await op.request(method, path, json=body) if body is not None else await op.request(method, path)
|
||||
assert resp.status_code == 403, (method, path, resp.text)
|
||||
listing = await op.get("/api/notifications/channels")
|
||||
assert listing.status_code == 200 and listing.json()[0]["config"] == {} and "secret-topic" not in listing.text
|
||||
finally:
|
||||
await op.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alerts_reach_every_enabled_channel_and_one_failure_does_not_block_the_rest(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
async with SessionLocal() as db:
|
||||
for kind, cfg, secret, name in [
|
||||
("gotify", {"server_url": "https://gotify.test"}, "g-tok", "A gotify"),
|
||||
("ntfy", {"server_url": "https://ntfy.test", "topic": "t"}, None, "B ntfy"),
|
||||
]:
|
||||
ch = _channel(kind, cfg, secret, name)
|
||||
db.add(ch)
|
||||
off = _channel("ntfy", {"server_url": "https://off.test", "topic": "t"}, None, "C disabled")
|
||||
off.enabled = False
|
||||
db.add(off)
|
||||
await db.commit()
|
||||
await _only("gateway_down_enabled")
|
||||
gotify = respx_mock.post("https://gotify.test/message").mock(return_value=httpx.Response(500))
|
||||
ntfy = respx_mock.post("https://ntfy.test/").mock(return_value=httpx.Response(200))
|
||||
await _add_samples(hid, [{"gateways": [_gw("offline", 100.0, None)]}] * 3)
|
||||
async with SessionLocal() as db:
|
||||
assert (await run_notification_checks(db))["opened"] == 1
|
||||
assert gotify.call_count == 1 and ntfy.call_count == 1
|
||||
assert "off.test" not in str([c.request.url for c in respx_mock.calls])
|
||||
@@ -0,0 +1,452 @@
|
||||
"""Taking over aliases, port forwards, 1:1 NAT and outbound NAT that were created by hand on a firewall.
|
||||
|
||||
The live rows below have the shape recorded from OPNsense 26.7 (test-fw-3, 2026-10-20), see the module
|
||||
docstring of app/services/object_adopt_service.py."""
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from tests.helpers import deploy_approved
|
||||
from tests.test_api_tokens import _login
|
||||
|
||||
BASE = "https://fw-adopt-obj.test"
|
||||
API = f"{BASE}/api/firewall"
|
||||
|
||||
|
||||
async def _host(client, respx_mock, name="fw-obj"):
|
||||
respx_mock.get(url__regex=r".*/getVipStatus").mock(return_value=httpx.Response(200, json={"rows": []}))
|
||||
r = await client.post("/api/hosts", json={"name": name, "base_url": BASE, "api_key": "k", "api_secret": "s"})
|
||||
assert r.status_code == 201, r.text
|
||||
return r.json()["id"]
|
||||
|
||||
|
||||
def _search(respx_mock, path, rows):
|
||||
return respx_mock.get(f"{API}/{path}").mock(return_value=httpx.Response(200, json={"rows": rows, "total": len(rows)}))
|
||||
|
||||
|
||||
def _mode(respx_mock, mode="hybrid"):
|
||||
respx_mock.get(f"{API}/source_nat/get").mock(return_value=httpx.Response(
|
||||
200, json={"filter": {"general": {"snat_mode": {m: {"selected": 1 if m == mode else 0} for m in
|
||||
("automatic", "hybrid", "advanced", "disabled")}}}}))
|
||||
|
||||
|
||||
def _writes(respx_mock):
|
||||
return [c for c in respx_mock.calls if c.request.method != "GET"]
|
||||
|
||||
|
||||
# ── live rows (shapes captured from OPNsense 26.7) ──────────────────────
|
||||
|
||||
def alias_row(name, type_="host", content="10.9.9.1\n10.9.9.2", **over):
|
||||
row = {"uuid": f"uuid-{name}", "enabled": "1", "name": name, "type": type_, "%type": type_.title(),
|
||||
"content": content, "description": f"{name} desc", "proto": "", "interface": "", "counters": "0",
|
||||
"updatefreq": "", "categories": ""}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
SYSTEM_ALIASES = [
|
||||
alias_row("bogons", "external", "", description="bogon networks (internal)"),
|
||||
alias_row("sshlockout", "external", "", description="abuse lockout table (internal)"),
|
||||
alias_row("__lan_network", "internal", "", description="lan net"),
|
||||
]
|
||||
|
||||
|
||||
def pf_row(**over):
|
||||
row = {"uuid": "pf-1", "ipprotocol": "inet", "protocol": "tcp", "%protocol": "TCP", "disabled": "0", "nordr": "0",
|
||||
"interface": "wan", "%interface": "WAN", "source.network": "any", "source.port": "", "source.not": "0",
|
||||
"destination.network": "wanip", "destination.port": "8443", "destination.not": "0", "target": "192.168.1.50",
|
||||
"local-port": "443", "natreflection": "", "descr": "Web", "pass": "", "log": "0", "poolopts": "", "tag": "",
|
||||
"tagged": "", "nosync": "0", "categories": "", "is_automatic": False}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
def o2o_row(**over):
|
||||
row = {"uuid": "o2o-1", "enabled": "1", "interface": "wan", "%interface": "WAN", "type": "binat",
|
||||
"source_net": "192.168.1.10", "source_not": "0", "destination_net": "any", "destination_not": "0",
|
||||
"external": "203.0.113.201", "natreflection": "", "log": "0", "description": "Mail 1:1", "categories": ""}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
def snat_row(**over):
|
||||
row = {"uuid": "snat-1", "enabled": "1", "interface": "wan", "%interface": "WAN", "ipprotocol": "inet",
|
||||
"protocol": "any", "source_net": "192.168.1.0/24", "source_not": "0", "source_port": "",
|
||||
"destination_net": "any", "destination_not": "0", "destination_port": "", "target": "", "target_port": "",
|
||||
"nonat": "0", "nosync": "0", "staticnatport": "0", "log": "0", "tag": "", "tagged": "",
|
||||
"description": "LAN out", "categories": ""}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
AUTOMATIC_NAT = [
|
||||
{"uuid": "lockout_0", "interface": "lan", "descr": "Anti-Lockout Rule", "is_automatic": True},
|
||||
{"uuid": "automatic_wan", "interface": "wan", "description": "Auto created rule", "is_automatic": True},
|
||||
]
|
||||
|
||||
# kind -> (url prefix, search path, row factory, id field of the preview)
|
||||
NAT_KINDS = {
|
||||
"port-forward": ("d_nat/searchRule", pf_row, "Web"),
|
||||
"one-to-one": ("one_to_one/searchRule", o2o_row, "Mail 1:1"),
|
||||
"outbound-nat": ("source_nat/searchRule", snat_row, "LAN out"),
|
||||
}
|
||||
|
||||
|
||||
# ── aliases ─────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_candidates_mark_system_unsupported_and_nested(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
|
||||
alias_row("ok_hosts"),
|
||||
alias_row("ok_nets", "network", "10.0.0.0/8\n192.168.0.0/16"),
|
||||
alias_row("ok_ports", "port", "8080\n9000:9010"),
|
||||
alias_row("ok_group", "networkgroup", "ok_nets\n__lan_network"),
|
||||
alias_row("a_url", "url", "https://example.org/list.txt", updatefreq="1"),
|
||||
alias_row("geo", "geoip", "DE", proto="IPv4"),
|
||||
alias_row("uses_url", "host", "a_url"),
|
||||
alias_row("uses_hosts", "host", "ok_hosts\n10.9.9.3"),
|
||||
alias_row("asn_v4", "asn", "64496", proto="IPv4"),
|
||||
alias_row("bad name", "host", "10.1.1.1"),
|
||||
alias_row("off", "host", "10.9.9.9", enabled="0"),
|
||||
])
|
||||
resp = await client.get(f"/api/aliases/unmanaged?host_id={host_id}")
|
||||
assert resp.status_code == 200, resp.text
|
||||
by = {r["title"]: r for r in resp.json()}
|
||||
|
||||
for name in ("bogons", "sshlockout", "__lan_network"):
|
||||
assert by[name]["reasons"] == ["system"] and not by[name]["adoptable"]
|
||||
for name in ("ok_hosts", "ok_nets", "ok_ports", "off"):
|
||||
assert by[name]["adoptable"] is True and by[name]["reasons"] == []
|
||||
assert by["off"]["enabled"] is False
|
||||
# __lan_network is OPNsense's own and fine; ok_nets is a live alias that is not managed yet
|
||||
assert by["ok_group"]["reasons"] == ["references_unmanaged_alias"] and by["ok_group"]["bulk_adoptable"] is True
|
||||
assert by["ok_group"]["reason_details"]["references_unmanaged_alias"] == "ok_nets"
|
||||
assert by["a_url"]["reasons"] == ["unsupported_type"] and by["geo"]["reasons"] == ["unsupported_type"]
|
||||
assert by["a_url"]["reason_details"]["unsupported_type"]
|
||||
assert by["uses_url"]["reasons"] == ["references_unmanaged_alias"]
|
||||
assert by["uses_url"]["reason_details"]["references_unmanaged_alias"] == "a_url"
|
||||
assert by["uses_url"]["bulk_adoptable"] is False # its target can never be adopted
|
||||
assert by["asn_v4"]["reasons"] == ["proto_set"]
|
||||
assert by["bad name"]["reasons"] == ["name_invalid"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_nested_alias_waits_for_its_target_but_bulk_orders_them(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", [
|
||||
alias_row("outer", "host", "inner\n10.0.0.9"),
|
||||
alias_row("inner", "host", "base"),
|
||||
alias_row("base", "host", "10.0.0.1"),
|
||||
])
|
||||
by = {r["title"]: r for r in (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()}
|
||||
assert by["base"]["adoptable"] is True
|
||||
for name, target in (("inner", "base"), ("outer", "inner")):
|
||||
assert by[name]["adoptable"] is False and by[name]["bulk_adoptable"] is True
|
||||
assert by[name]["reason_details"]["references_unmanaged_alias"] == target
|
||||
# alone it is refused ...
|
||||
refused = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-outer"})
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "OBJECT_NOT_ADOPTABLE"
|
||||
assert "references_unmanaged_alias" in refused.json()["detail"]["params"]["reasons"]
|
||||
# ... after its target is managed it works
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-base"})).status_code == 201
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-inner"})).status_code == 201
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-outer"})).status_code == 201
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_alias_records_it_without_writing_and_preview_is_unchanged(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
|
||||
alias_row("web_servers", "host", "10.0.0.1\n 10.0.0.2 \n\nexample.org", description="Web tier"),
|
||||
])
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})
|
||||
assert resp.status_code == 201, resp.text
|
||||
alias = resp.json()
|
||||
assert (alias["name"], alias["type"], alias["description"], alias["enabled"]) == ("web_servers", "host", "Web tier", True)
|
||||
assert alias["content"] == "10.0.0.1\n10.0.0.2\nexample.org"
|
||||
assert alias["target_host_ids"] == [host_id]
|
||||
assert [d["status"] for d in alias["deployments"]] == ["success"]
|
||||
assert alias["workflow_state"] == "synced"
|
||||
assert not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
|
||||
|
||||
preview = (await client.get(f"/api/aliases/{alias['id']}/preview")).json()
|
||||
assert [(h["action"], h["changes"]) for h in preview["hosts"]] == [("unchanged", [])]
|
||||
assert preview["can_deploy"] is False
|
||||
# no longer offered; adopting again is refused
|
||||
assert [r["title"] for r in (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()
|
||||
if r["reasons"] != ["system"]] == []
|
||||
again = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})
|
||||
assert again.status_code == 409 and "already_managed" in again.json()["detail"]["params"]["reasons"]
|
||||
|
||||
# audit: the deployment record (kind alias, remote_uuid) is written with the adoption
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "alias"))).all()
|
||||
assert any(e.entity_name == "web_servers" and e.details["remote_uuid"] == "uuid-web_servers"
|
||||
and e.message == "Von der Firewall übernommen" for e in events)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_edit_after_alias_adopt_updates_the_same_remote_object(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", [alias_row("web_servers", "host", "10.0.0.1")])
|
||||
alias = (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-web_servers"})).json()
|
||||
edit = {"name": "web_servers", "type": "host", "content": "10.0.0.1\n10.0.0.2", "description": "web_servers desc",
|
||||
"enabled": True, "target_host_ids": [host_id], "target_group_ids": [], "expected_updated_at": alias["updated_at"]}
|
||||
assert (await client.put(f"/api/aliases/{alias['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(f"{API}/alias/addItem").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
setter = respx_mock.post(f"{API}/alias/setItem/uuid-web_servers").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(f"{API}/alias/reconfigure").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
resp = await deploy_approved(client, f"/api/aliases/{alias['id']}/deploy")
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert setter.called and not add.called # no duplicate
|
||||
assert json.loads(setter.calls.last.request.content)["alias"]["content"] == "10.0.0.1\n10.0.0.2"
|
||||
confirm = await client.post(f"/api/aliases/{alias['id']}/confirm")
|
||||
assert [d["status"] for d in confirm.json()["deployments"]] == ["success"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alias_name_that_exists_centrally_is_refused(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
assert (await client.post("/api/aliases", json={"name": "dup", "type": "host", "content": "1.1.1.1"})).status_code == 201
|
||||
_search(respx_mock, "alias/searchItem", [alias_row("dup", "host", "2.2.2.2")])
|
||||
row = (await client.get(f"/api/aliases/unmanaged?host_id={host_id}")).json()[0]
|
||||
assert row["reasons"] == ["name_exists"]
|
||||
assert (await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "uuid-dup"})).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_all_aliases_orders_dependencies_and_reports_skipped(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", SYSTEM_ALIASES + [
|
||||
alias_row("outer", "host", "inner"),
|
||||
alias_row("inner", "host", "10.0.0.1"),
|
||||
alias_row("a_url", "url", "https://example.org/x"),
|
||||
alias_row("uses_url", "host", "a_url"),
|
||||
alias_row("plain", "port", "443"),
|
||||
])
|
||||
resp = await client.post("/api/aliases/adopt-all", json={"host_id": host_id})
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert sorted(a["title"] for a in body["adopted"]) == ["inner", "outer", "plain"]
|
||||
skipped = {s["title"]: s for s in body["skipped"]}
|
||||
assert set(skipped) == {"a_url", "uses_url"} # system aliases are not reported
|
||||
assert skipped["a_url"]["reasons"] == ["unsupported_type"]
|
||||
assert skipped["uses_url"]["reasons"] == ["references_unmanaged_alias"]
|
||||
aliases = {a["name"]: a for a in (await client.get("/api/aliases")).json()}
|
||||
assert set(aliases) == {"inner", "outer", "plain"}
|
||||
assert all([d["status"] for d in a["deployments"]] == ["success"] and a["workflow_state"] == "synced" for a in aliases.values())
|
||||
# a second run finds nothing left to take over
|
||||
again = (await client.post("/api/aliases/adopt-all", json={"host_id": host_id})).json()
|
||||
assert again["adopted"] == [] and {s["title"] for s in again["skipped"]} == {"a_url", "uses_url"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_adopt_unknown_alias_is_404(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "alias/searchItem", [alias_row("x")])
|
||||
resp = await client.post("/api/aliases/adopt", json={"host_id": host_id, "uuid": "nope"})
|
||||
assert resp.status_code == 404 and resp.json()["detail"]["code"] == "ADOPT_OBJECT_NOT_FOUND"
|
||||
|
||||
|
||||
# ── NAT ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automatic_nat_rows_are_never_offered(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock)
|
||||
for prefix, (path, factory, _) in NAT_KINDS.items():
|
||||
_search(respx_mock, path, [factory(), *AUTOMATIC_NAT])
|
||||
rows = (await client.get(f"/api/{prefix}/unmanaged?host_id={host_id}")).json()
|
||||
assert [r["uuid"] for r in rows] == [factory()["uuid"]], prefix
|
||||
assert rows[0]["adoptable"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_port_forward_refusals(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
cases = {
|
||||
"multi": (dict(interface="wan,lan"), "interface_not_single"),
|
||||
"v46": (dict(ipprotocol="inet46"), "ipprotocol_unsupported"),
|
||||
"anyproto": (dict(protocol=""), "protocol_any"),
|
||||
"nordr": (dict(nordr="1", target="", **{"local-port": ""}), "no_redirect"),
|
||||
"notarget": (dict(target=""), "no_target"),
|
||||
"linked": (dict(**{"pass": "pass"}), "filter_rule_linked"),
|
||||
"srcnot": (dict(**{"source.not": "1"}), "negated_source"),
|
||||
"dstnot": (dict(**{"destination.not": "1"}), "negated_destination"),
|
||||
"refl": (dict(natreflection="enable"), "natreflection_unsupported"),
|
||||
"nodesc": (dict(descr=""), "no_description"),
|
||||
"long": (dict(target="x" * 200), "value_too_long"),
|
||||
}
|
||||
_search(respx_mock, "d_nat/searchRule", [pf_row(**{"uuid": k, "descr": f"rule {k}", **over})
|
||||
for k, (over, _) in cases.items()] + [pf_row(uuid="fine")])
|
||||
rows = {r["uuid"]: r for r in (await client.get(f"/api/port-forward/unmanaged?host_id={host_id}")).json()}
|
||||
assert rows["fine"]["adoptable"] is True
|
||||
for key, (_, reason) in cases.items():
|
||||
assert reason in rows[key]["reasons"] and not rows[key]["adoptable"], key
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_port_forward_adopt_maps_fields_and_keeps_unmanaged_ones(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "d_nat/searchRule", [
|
||||
pf_row(disabled="1", natreflection="purenat", log="1", poolopts="random", tag="tg1", **{"source.network": "10.0.0.0/8"}),
|
||||
pf_row(uuid="pf-2", descr="Alias based", target="web_servers", **{"destination.port": "web_ports", "local-port": ""}),
|
||||
*AUTOMATIC_NAT,
|
||||
])
|
||||
candidates = {r["uuid"]: r for r in (await client.get(f"/api/port-forward/unmanaged?host_id={host_id}")).json()}
|
||||
assert sorted(candidates["pf-1"]["kept"]) == ["log", "poolopts", "tag"]
|
||||
resp = await client.post("/api/port-forward/adopt", json={"host_id": host_id, "uuid": "pf-1"})
|
||||
assert resp.status_code == 201, resp.text
|
||||
rule = resp.json()
|
||||
assert (rule["description"], rule["interface"], rule["ipprotocol"], rule["protocol"]) == ("Web", "wan", "inet", "tcp")
|
||||
assert (rule["source_net"], rule["destination_net"], rule["destination_port"]) == ("10.0.0.0/8", "wanip", "8443")
|
||||
assert (rule["target"], rule["local_port"], rule["natreflection"], rule["enabled"]) == ("192.168.1.50", "443", "purenat", False)
|
||||
assert rule["workflow_state"] == "synced" and [d["status"] for d in rule["deployments"]] == ["success"]
|
||||
preview = (await client.get(f"/api/port-forward/{rule['id']}/preview")).json()
|
||||
assert [h["action"] for h in preview["hosts"]] == ["unchanged"] and preview["can_deploy"] is False
|
||||
assert not _writes(respx_mock) or not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
|
||||
|
||||
# edit -> deploy updates the same remote rule and never sends the fields it does not manage
|
||||
edit = {k: rule[k] for k in ("description", "interface", "ipprotocol", "protocol", "source_net", "source_port",
|
||||
"destination_net", "destination_port", "target", "local_port", "natreflection", "enabled")}
|
||||
edit.update(target="192.168.1.51", target_host_ids=[host_id], target_group_ids=[], expected_updated_at=rule["updated_at"])
|
||||
assert (await client.put(f"/api/port-forward/{rule['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(f"{API}/d_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
setter = respx_mock.post(f"{API}/d_nat/setRule/pf-1").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(f"{API}/d_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert (await deploy_approved(client, f"/api/port-forward/{rule['id']}/deploy")).status_code == 200
|
||||
assert setter.called and not add.called
|
||||
sent = json.loads(setter.calls.last.request.content)["rule"]
|
||||
assert sent["target"] == "192.168.1.51" and sent["disabled"] == "1"
|
||||
assert not {"log", "poolopts", "tag", "tagged", "nosync", "categories", "nordr"} & set(sent)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_one_to_one_adopt_and_refusals(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_search(respx_mock, "one_to_one/searchRule", [
|
||||
o2o_row(),
|
||||
o2o_row(uuid="o2", description="NAT off", type="nat", enabled="0", natreflection="enable", log="1",
|
||||
source_not="1", destination_net="10.0.0.0/8", destination_not="1"),
|
||||
o2o_row(uuid="o3", interface="wan,lan", description="multi"),
|
||||
o2o_row(uuid="o4", description=""),
|
||||
o2o_row(uuid="o5", description="weird", type="other"),
|
||||
o2o_row(uuid="o6", description="refl", natreflection="purenat"),
|
||||
*AUTOMATIC_NAT,
|
||||
])
|
||||
rows = {r["uuid"]: r for r in (await client.get(f"/api/one-to-one/unmanaged?host_id={host_id}")).json()}
|
||||
assert rows["o2o-1"]["adoptable"] and rows["o2"]["adoptable"]
|
||||
assert rows["o3"]["reasons"] == ["interface_not_single"] and rows["o4"]["reasons"] == ["no_description"]
|
||||
assert rows["o5"]["reasons"] == ["type_unsupported"] and rows["o6"]["reasons"] == ["natreflection_unsupported"]
|
||||
|
||||
rule = (await client.post("/api/one-to-one/adopt", json={"host_id": host_id, "uuid": "o2"})).json()
|
||||
assert (rule["type"], rule["enabled"], rule["natreflection"], rule["log"], rule["source_not"], rule["destination_not"]) == (
|
||||
"nat", False, "enable", True, True, True)
|
||||
assert rule["destination_net"] == "10.0.0.0/8" and rule["external"] == "203.0.113.201"
|
||||
preview = (await client.get(f"/api/one-to-one/{rule['id']}/preview")).json()
|
||||
assert [h["action"] for h in preview["hosts"]] == ["unchanged"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_outbound_nat_adopt_and_refusals(client, respx_mock):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock, "hybrid")
|
||||
_search(respx_mock, "source_nat/searchRule", [
|
||||
snat_row(),
|
||||
snat_row(uuid="s2", description="to pool", protocol="TCP", destination_port="443", target="203.0.113.205",
|
||||
target_port="1024", staticnatport="1", log="1", enabled="0", destination_not="1", destination_net="10.0.0.0/8"),
|
||||
snat_row(uuid="s3", description="no nat", nonat="1"),
|
||||
snat_row(uuid="s4", description="v46", ipprotocol="inet46"),
|
||||
snat_row(uuid="s5", description=""),
|
||||
*AUTOMATIC_NAT,
|
||||
])
|
||||
rows = {r["uuid"]: r for r in (await client.get(f"/api/outbound-nat/unmanaged?host_id={host_id}")).json()}
|
||||
assert rows["snat-1"]["adoptable"] and rows["s2"]["adoptable"]
|
||||
assert sorted(rows["s2"]["kept"]) == ["log", "staticnatport"]
|
||||
assert rows["s3"]["reasons"] == ["no_nat_rule"] and rows["s4"]["reasons"] == ["ipprotocol_unsupported"]
|
||||
assert rows["s5"]["reasons"] == ["no_description"]
|
||||
|
||||
rule = (await client.post("/api/outbound-nat/adopt", json={"host_id": host_id, "uuid": "s2"})).json()
|
||||
assert (rule["protocol"], rule["destination_port"], rule["target"], rule["target_port"], rule["enabled"],
|
||||
rule["destination_not"]) == ("TCP", "443", "203.0.113.205", "1024", False, True)
|
||||
assert rule["source_port"] is None
|
||||
preview = (await client.get(f"/api/outbound-nat/{rule['id']}/preview")).json()
|
||||
assert [h["action"] for h in preview["hosts"]] == ["unchanged"]
|
||||
|
||||
# edit -> deploy: setRule on the same uuid, nonat/log/staticnatport are not part of the payload
|
||||
edit = {k: rule[k] for k in ("description", "interface", "source_net", "destination_net", "source_not", "destination_not",
|
||||
"enabled", "ipprotocol", "protocol", "source_port", "destination_port", "target", "target_port")}
|
||||
edit.update(enabled=True, target_host_ids=[host_id], target_group_ids=[], expected_updated_at=rule["updated_at"])
|
||||
assert (await client.put(f"/api/outbound-nat/{rule['id']}", json=edit)).status_code == 200
|
||||
add = respx_mock.post(f"{API}/source_nat/addRule").mock(return_value=httpx.Response(200, json={"result": "saved", "uuid": "dup"}))
|
||||
setter = respx_mock.post(f"{API}/source_nat/setRule/s2").mock(return_value=httpx.Response(200, json={"result": "saved"}))
|
||||
respx_mock.post(f"{API}/source_nat/apply").mock(return_value=httpx.Response(200, json={"status": "ok"}))
|
||||
assert (await deploy_approved(client, f"/api/outbound-nat/{rule['id']}/deploy")).status_code == 200
|
||||
assert setter.called and not add.called
|
||||
sent = json.loads(setter.calls.last.request.content)["rule"]
|
||||
assert sent["enabled"] == "1" and not {"nonat", "log", "staticnatport", "nosync", "tag"} & set(sent)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("mode", ["automatic", "disabled"])
|
||||
async def test_outbound_nat_adoption_needs_a_manual_capable_mode(client, respx_mock, mode):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock, mode)
|
||||
search = _search(respx_mock, "source_nat/searchRule", [snat_row()])
|
||||
for call in (client.get(f"/api/outbound-nat/unmanaged?host_id={host_id}"),
|
||||
client.post("/api/outbound-nat/adopt", json={"host_id": host_id, "uuid": "snat-1"}),
|
||||
client.post("/api/outbound-nat/adopt-all", json={"host_id": host_id})):
|
||||
resp = await call
|
||||
assert resp.status_code == 409 and resp.json()["detail"]["code"] == "OUTBOUND_MODE_BLOCKS_ADOPT", resp.text
|
||||
assert resp.json()["detail"]["params"]["mode"] == mode
|
||||
assert not search.called
|
||||
assert (await client.get("/api/outbound-nat")).json() == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("prefix", list(NAT_KINDS))
|
||||
async def test_adopt_all_nat_takes_what_is_adoptable_and_is_idempotent(client, respx_mock, prefix):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock)
|
||||
path, factory, title = NAT_KINDS[prefix]
|
||||
bad = {"descr": ""} if prefix == "port-forward" else {"description": ""}
|
||||
_search(respx_mock, path, [factory(), factory(uuid="second", **({"descr": "Second"} if prefix == "port-forward" else {"description": "Second"})),
|
||||
factory(uuid="bad", **bad), *AUTOMATIC_NAT])
|
||||
result = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": host_id})).json()
|
||||
assert sorted(a["title"] for a in result["adopted"]) == sorted([title, "Second"])
|
||||
assert [(s["uuid"], s["reasons"]) for s in result["skipped"]] == [("bad", ["no_description"])]
|
||||
listed = (await client.get(f"/api/{prefix}")).json()
|
||||
assert len(listed) == 2 and all(r["workflow_state"] == "synced" for r in listed)
|
||||
again = (await client.post(f"/api/{prefix}/adopt-all", json={"host_id": host_id})).json()
|
||||
assert again["adopted"] == [] and len((await client.get(f"/api/{prefix}")).json()) == 2
|
||||
assert not any("/firewall/" in str(c.request.url) for c in _writes(respx_mock))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("prefix", ["aliases", *NAT_KINDS])
|
||||
async def test_roles_and_unknown_host(client, respx_mock, prefix):
|
||||
host_id = await _host(client, respx_mock)
|
||||
_mode(respx_mock)
|
||||
path = "alias/searchItem" if prefix == "aliases" else NAT_KINDS[prefix][0]
|
||||
row = alias_row("a1") if prefix == "aliases" else NAT_KINDS[prefix][1]()
|
||||
_search(respx_mock, path, [row])
|
||||
unknown = "00000000-0000-0000-0000-000000000000"
|
||||
assert (await client.get(f"/api/{prefix}/unmanaged?host_id={unknown}")).status_code == 404
|
||||
assert (await client.post(f"/api/{prefix}/adopt", json={"host_id": unknown, "uuid": "x"})).status_code == 404
|
||||
|
||||
# read-only users cannot adopt (it creates central objects), operators can (it writes nothing to a firewall)
|
||||
for role, expected in (("read_only", 403), ("operator", 201)):
|
||||
user = await client.post("/api/users", json={"username": f"u-{role}-{prefix}", "password": "adopt-test-password-1", "role": role})
|
||||
assert user.status_code == 201, user.text
|
||||
session = await _login(f"u-{role}-{prefix}", "adopt-test-password-1")
|
||||
try:
|
||||
assert (await session.get(f"/api/{prefix}/unmanaged?host_id={host_id}")).status_code == 200
|
||||
resp = await session.post(f"/api/{prefix}/adopt", json={"host_id": host_id, "uuid": row["uuid"]})
|
||||
assert resp.status_code == expected, (role, resp.text)
|
||||
finally:
|
||||
await session.aclose()
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
|
||||
import pytest
|
||||
|
||||
from tests.test_maintenance_windows import _WEEKLY
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
|
||||
async def _fleet(client):
|
||||
"""canary: c1 · pilot: p1 + ha-b (BACKUP) · prod: ha-a; ha-a/ha-b form cluster "ha"."""
|
||||
hosts = {}
|
||||
for name in ("c1", "p1", "ha-a", "ha-b"):
|
||||
hosts[name] = await _create_host(client, name, f"https://{name}.test")
|
||||
for name in ("ha-a", "ha-b"):
|
||||
await client.patch(f"/api/hosts/{hosts[name]['id']}", json={"cluster_name": "ha"})
|
||||
groups = {}
|
||||
for gname, members in (("ring-canary", ["c1"]), ("ring-pilot", ["p1", "ha-b"]), ("ring-prod", ["ha-a", "c1"])):
|
||||
resp = await client.post("/api/groups", json={"name": gname, "host_ids": [hosts[m]["id"] for m in members]})
|
||||
assert resp.status_code == 201, resp.text
|
||||
groups[gname] = resp.json()["id"]
|
||||
return hosts, groups
|
||||
|
||||
|
||||
def _plan_body(groups, **kw):
|
||||
body = {"name": "Ringe", "action": "update", "waves": [
|
||||
{"label": "canary", "group_ids": [groups["ring-canary"]], "soak_minutes": 10},
|
||||
{"label": "pilot", "group_ids": [groups["ring-pilot"]], "soak_minutes": 10},
|
||||
{"label": "prod", "group_ids": [groups["ring-prod"]], "soak_minutes": 0, "carp_handover": True, "require_approval": True},
|
||||
]}
|
||||
body.update(kw)
|
||||
return body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_plan_crud_and_preview_resolves_rings(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
resp = await client.post("/api/rollouts/plans", json=_plan_body(groups))
|
||||
assert resp.status_code == 201, resp.text
|
||||
plan = resp.json()
|
||||
assert [w["label"] for w in plan["waves"]] == ["canary", "pilot", "prod"] and plan["host_count"] == 4
|
||||
prev = await client.post(f"/api/rollouts/plans/{plan['id']}/preview")
|
||||
assert prev.status_code == 200
|
||||
body = prev.json()
|
||||
assert [[h["name"] for h in w["hosts"]] for w in body["waves"]] == [["c1"], ["ha-b", "p1"], ["ha-a"]] # c1 only once
|
||||
codes = {w["code"] for w in body["warnings"]}
|
||||
assert {"duplicate_hosts", "mixed_versions", "master_needs_handover", "no_window"} <= codes
|
||||
assert body["errors"] == [] and body["can_start"] and len(body["preview_token"]) == 64
|
||||
assert (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()["detail"]["code"] == "ROLLOUT_NAME_TAKEN"
|
||||
assert (await client.delete(f"/api/groups/{groups['ring-canary']}")).json()["detail"]["code"] == "ROLLOUT_GROUP_IN_USE"
|
||||
assert (await client.delete(f"/api/rollouts/plans/{plan['id']}")).status_code == 204
|
||||
assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_major_update_must_keep_a_cluster_in_one_wave(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, action="major_update"))).json()
|
||||
assert {e["code"] for e in plan["errors"]} == {"cluster_split"}
|
||||
prev = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()
|
||||
assert prev["can_start"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_activation_needs_auto_start_a_window_and_the_current_token(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
|
||||
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == "auto_start"
|
||||
|
||||
plan = (await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"]))).json()
|
||||
stale = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert stale.status_code == 409 and stale.json()["detail"]["code"] == "ROLLOUT_PREVIEW_STALE"
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
active = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert active.status_code == 200 and active.json()["active"] and active.json()["fingerprint_ok"]
|
||||
assert active.json()["activated_by"] == "admin"
|
||||
|
||||
# A group change behind the plan's back: still "active", but the fingerprint no longer matches.
|
||||
await client.put(f"/api/groups/{groups['ring-canary']}", json={"name": "ring-canary", "host_ids": []})
|
||||
assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).json()["fingerprint_ok"] is False
|
||||
# A window that a plan uses cannot be deleted (the plan would silently lose its time limit).
|
||||
assert (await client.delete(f"/api/maintenance-windows/{window['id']}")).json()["detail"]["code"] == "WINDOW_IN_USE"
|
||||
|
||||
# Any edit disarms it.
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
edited = await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"], name="Ringe 2"))
|
||||
assert edited.json()["active"] is False
|
||||
off = await client.post(f"/api/rollouts/plans/{plan['id']}/deactivate")
|
||||
assert off.status_code == 200 and off.json()["active"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_auto_start_only_for_minor_updates_with_a_window(client):
|
||||
_hosts, groups = await _fleet(client)
|
||||
plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True))).json()
|
||||
assert "auto_start_needs_window" in {e["code"] for e in plan["errors"]}
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "ROLLOUT_PLAN_INVALID"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change,field", [
|
||||
({"gate_checks": {"nonsense": True}}, "gate_checks"),
|
||||
({"window_id": "00000000-0000-0000-0000-000000000000"}, "window_id"),
|
||||
])
|
||||
@pytest.mark.asyncio
|
||||
async def test_plan_validation(client, change, field):
|
||||
_hosts, groups = await _fleet(client)
|
||||
resp = await client.post("/api/rollouts/plans", json=_plan_body(groups, **change))
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == field
|
||||
bad_group = _plan_body(groups)
|
||||
bad_group["waves"][0]["group_ids"] = ["00000000-0000-0000-0000-000000000000"]
|
||||
assert (await client.post("/api/rollouts/plans", json=bad_group)).json()["detail"]["params"]["field"] == "group_ids"
|
||||
too_many = _plan_body(groups, waves=[{"label": str(i), "group_ids": [groups["ring-canary"]]} for i in range(11)])
|
||||
assert (await client.post("/api/rollouts/plans", json=too_many)).status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_role_matrix_for_plans(client):
|
||||
from tests.test_operator_role import _operator
|
||||
_hosts, groups = await _fleet(client)
|
||||
window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
|
||||
op = await _operator(client, "rollop")
|
||||
try:
|
||||
made = await op.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True, window_id=window["id"]))
|
||||
assert made.status_code == 201 # preparing is fine
|
||||
pid = made.json()["id"]
|
||||
token = (await op.post(f"/api/rollouts/plans/{pid}/preview")).json()["preview_token"]
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
denied = await op.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})
|
||||
assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
|
||||
assert (await op.post(f"/api/rollouts/plans/{pid}/deactivate")).status_code == 200 # braking is always allowed
|
||||
assert (await client.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})).status_code == 200
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
from tests.test_api_tokens import _login
|
||||
assert (await client.post("/api/users", json={"username": "rollro", "password": "readonly-pass-1", "role": "read_only"})).status_code == 201
|
||||
ro = await _login("rollro", "readonly-pass-1")
|
||||
try:
|
||||
assert (await ro.get("/api/rollouts/plans")).status_code == 200
|
||||
for path in ("/api/rollouts/plans", f"/api/rollouts/plans/{pid}/deactivate"):
|
||||
assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
|
||||
finally:
|
||||
await ro.aclose()
|
||||
@@ -0,0 +1,620 @@
|
||||
"""The rollout engine end to end against a fake fleet: c1 (stand-alone), ha-a (MASTER) + ha-b (BACKUP)
|
||||
in cluster "ha". A fake clock drives timeouts, soak and re-checks; every tick runs exactly what the
|
||||
poll loop runs (fire due schedules, step running ones, step the rollout)."""
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.models.rollout import RolloutPlan, RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import rollout_engine as engine
|
||||
from app.services import scheduled_maintenance_service as svc
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
|
||||
class Fw:
|
||||
def __init__(self, fleet, name, *, role=None, version="26.7.6", latest="26.7.7"):
|
||||
self.fleet, self.name, self.base = fleet, name, f"https://{name}.test"
|
||||
self.role, self.maint = role, False
|
||||
self.version, self.latest, self.offer = version, latest, "update"
|
||||
self.job, self.job_ticks, self.boot, self.up = "ready", 0, "boot-1", True
|
||||
self.update_error = False
|
||||
self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
|
||||
{"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
|
||||
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
|
||||
r, b = fleet.mock, self.base
|
||||
self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
|
||||
self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
|
||||
self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
|
||||
r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
|
||||
r.get(f"{b}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
|
||||
r.get(f"{b}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
|
||||
r.get(f"{b}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
|
||||
r.get(f"{b}/api/core/firmware/status").mock(side_effect=self._status)
|
||||
r.get(f"{b}/api/diagnostics/system/systemTime").mock(side_effect=lambda q: self._ok({"boottime": self.boot}))
|
||||
r.get(f"{b}/api/routes/gateway/status").mock(side_effect=lambda q: self._ok({"items": self.gateways, "status": "ok"}))
|
||||
r.get(f"{b}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
|
||||
|
||||
@property
|
||||
def down(self):
|
||||
return self.job == "down" or not self.up
|
||||
|
||||
def _ok(self, data):
|
||||
if self.down:
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, json=data)
|
||||
|
||||
def _raw(self, content):
|
||||
if self.down:
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, content=content)
|
||||
|
||||
def _status(self, request):
|
||||
if self.offer == "update":
|
||||
return self._ok({"status": "update", "status_msg": "x", "status_reboot": "1", "all_packages": {},
|
||||
"product": {"product_latest": self.latest}})
|
||||
return self._ok({"status": self.offer, "status_msg": "x", "status_reboot": "0", "all_packages": {}, "product": {}})
|
||||
|
||||
def _update(self, request):
|
||||
if self.update_error:
|
||||
return httpx.Response(500, json={"status": "error"})
|
||||
self.job, self.job_ticks = "running", 2
|
||||
return self._ok({"status": "ok"})
|
||||
|
||||
def _vip(self, request):
|
||||
rows = [{"vhid": "1", "status": self.role}] if self.role else []
|
||||
return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "240" if self.maint else "0",
|
||||
"status_msg": ""}})
|
||||
|
||||
def _toggle(self, request):
|
||||
self.maint = not self.maint
|
||||
partner = self.fleet.partner(self)
|
||||
if self.maint:
|
||||
self.role, partner.role = "BACKUP", "MASTER"
|
||||
else:
|
||||
self.role, partner.role = "MASTER", "BACKUP" # preempt: takes MASTER back
|
||||
return self._ok({"status": "ok", "action": "enter_maintenance" if self.maint else "leave_maintenance"})
|
||||
|
||||
def progress(self):
|
||||
if self.job == "running":
|
||||
self.job_ticks -= 1
|
||||
if self.job_ticks <= 0:
|
||||
self.job, self.job_ticks = "down", 2
|
||||
elif self.job == "down":
|
||||
self.job_ticks -= 1
|
||||
if self.job_ticks <= 0:
|
||||
self.job, self.version, self.offer, self.boot = "ready", self.latest, "none", self.boot + "+"
|
||||
|
||||
|
||||
class Fleet:
|
||||
def __init__(self, respx_mock):
|
||||
self.mock = respx_mock
|
||||
self.fw = {"c1": Fw(self, "c1"), "ha-a": Fw(self, "ha-a", role="MASTER"), "ha-b": Fw(self, "ha-b", role="BACKUP")}
|
||||
|
||||
def partner(self, fw):
|
||||
return self.fw["ha-b"] if fw.name == "ha-a" else self.fw["ha-a"]
|
||||
|
||||
def progress(self):
|
||||
for fw in self.fw.values():
|
||||
fw.progress()
|
||||
|
||||
|
||||
class Clock:
|
||||
def __init__(self):
|
||||
self.now = datetime.now(timezone.utc)
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clock(monkeypatch):
|
||||
c = Clock()
|
||||
monkeypatch.setattr(engine, "_now", c)
|
||||
monkeypatch.setattr(svc, "_now", c)
|
||||
return c
|
||||
|
||||
|
||||
async def _setup(client, respx_mock, *, waves=None, **plan_kw):
|
||||
fleet = Fleet(respx_mock)
|
||||
hosts = {n: await _create_host(client, n, f"https://{n}.test") for n in ("c1", "ha-a", "ha-b")}
|
||||
for n in ("ha-a", "ha-b"):
|
||||
await client.patch(f"/api/hosts/{hosts[n]['id']}", json={"cluster_name": "ha"})
|
||||
groups = {}
|
||||
for g, members in (("canary", ["c1"]), ("pilot", ["ha-b"]), ("prod", ["ha-a"]), ("cluster", ["ha-a", "ha-b"])):
|
||||
groups[g] = (await client.post("/api/groups", json={"name": g, "host_ids": [hosts[m]["id"] for m in members]})).json()["id"]
|
||||
waves = waves or [
|
||||
{"label": "canary", "group_ids": [groups["canary"]], "soak_minutes": 5},
|
||||
{"label": "pilot", "group_ids": [groups["pilot"]], "soak_minutes": 5},
|
||||
{"label": "prod", "group_ids": [groups["prod"]], "soak_minutes": 0, "carp_handover": True},
|
||||
]
|
||||
waves = [{**w, "group_ids": [groups[g] if g in groups else g for g in w["group_ids"]]} for w in waves]
|
||||
plan = (await client.post("/api/rollouts/plans", json={"name": "ringe", "action": "update", "waves": waves, **plan_kw})).json()
|
||||
return fleet, hosts, groups, plan
|
||||
|
||||
|
||||
async def _start(client, plan, api=None):
|
||||
api = api or client
|
||||
token = (await api.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
resp = await api.post(f"/api/rollouts/plans/{plan['id']}/start",
|
||||
json={"preview_token": token, "confirm_name": plan["name"], "acknowledge_unattended": True})
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
async def _tick(fleet, clock, seconds=30):
|
||||
fleet.progress()
|
||||
async with SessionLocal() as db:
|
||||
await svc.fire_due(db)
|
||||
async with SessionLocal() as db:
|
||||
await svc.advance_running(db)
|
||||
async with SessionLocal() as db:
|
||||
await engine.advance_runs(db)
|
||||
clock.now += timedelta(seconds=seconds)
|
||||
|
||||
|
||||
async def _run(run_id):
|
||||
async with SessionLocal() as db:
|
||||
return await db.scalar(select(RolloutRun).where(RolloutRun.id == uuid.UUID(str(run_id))).execution_options(populate_existing=True))
|
||||
|
||||
|
||||
async def _until(fleet, clock, run_id, pred, max_ticks=400, check=None):
|
||||
for _ in range(max_ticks):
|
||||
run = await _run(run_id)
|
||||
if pred(run):
|
||||
return run
|
||||
if check:
|
||||
await check()
|
||||
await _tick(fleet, clock)
|
||||
raise AssertionError(f"not reached: status={run.status} reason={run.pause_reason} detail={run.pause_detail}")
|
||||
|
||||
|
||||
def _host(run, name):
|
||||
return next(h for h in run.hosts if h.host_name == name)
|
||||
|
||||
|
||||
async def _scheds():
|
||||
async with SessionLocal() as db:
|
||||
return list((await db.scalars(select(ScheduledMaintenance).order_by(ScheduledMaintenance.created_at))).all())
|
||||
|
||||
|
||||
async def _no_two_at_once():
|
||||
active = [s for s in await _scheds() if s.status in ("pending", "running")]
|
||||
assert len(active) <= 1, [(s.host_name, s.status, s.phase) for s in active]
|
||||
|
||||
|
||||
# ── happy path ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_three_rings_run_through_and_the_master_hands_over(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert done.target_version == "26.7.7"
|
||||
assert [(h.host_name, h.status, h.version_before, h.version_after) for h in sorted(done.hosts, key=lambda h: h.wave_index)] == [
|
||||
("c1", "succeeded", "26.7.6", "26.7.7"), ("ha-b", "succeeded", "26.7.6", "26.7.7"), ("ha-a", "succeeded", "26.7.6", "26.7.7")]
|
||||
assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
|
||||
# The MASTER went through the CARP handover and is MASTER again; the BACKUP never needed one.
|
||||
assert fleet.fw["ha-a"].toggles.call_count == 2 and fleet.fw["ha-b"].toggles.call_count == 0
|
||||
assert (fleet.fw["ha-a"].role, fleet.fw["ha-b"].role) == ("MASTER", "BACKUP")
|
||||
scheds = await _scheds()
|
||||
assert [s.host_name for s in scheds] == ["c1", "ha-b", "ha-a"]
|
||||
assert all(s.origin == "rollout" and s.auto_verify and s.status == "succeeded" for s in scheds)
|
||||
async with SessionLocal() as db:
|
||||
ops = (await db.scalars(select(HostOperation))).all()
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rollout"))).all()
|
||||
assert {o.status for o in ops} == {"acknowledged"} # nothing left locked
|
||||
assert sum(1 for e in events if e.action == "gate" and e.status == "ok") == 3
|
||||
assert any(e.action == "finish" and e.status == "succeeded" for e in events)
|
||||
host_events = [e for e in (await _audit_host()) if (e.details or {}).get("rollout")]
|
||||
assert host_events and all(e.details["rollout"]["run_id"] == run["id"] for e in host_events)
|
||||
|
||||
|
||||
async def _audit_host():
|
||||
async with SessionLocal() as db:
|
||||
return (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
|
||||
|
||||
|
||||
# ── the gate stops the rollout ─────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gate_failure_in_the_canary_ring_stops_everything_after_it(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
fleet.fw["c1"].services[1]["running"] = 0 # ntpd did not come back
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "gate_failed" and paused.paused_from == "gate"
|
||||
assert paused.waves[0].gate_attempts == 2 # looked at twice (slow start), then stopped
|
||||
assert paused.waves[0].gate_result["hosts"][_host(paused, "c1").host_id.__str__()]["services"]["code"] == "service_stopped"
|
||||
for _ in range(20): # time passes, nothing moves
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
|
||||
assert [s.host_name for s in await _scheds()] == ["c1"]
|
||||
# Notification while paused.
|
||||
from app.services import notification_checks
|
||||
async with SessionLocal() as db:
|
||||
problems = await notification_checks.check_rollout_paused(db)
|
||||
assert problems[run["id"]]["grund"] == "gate_failed"
|
||||
|
||||
# Fixed by hand, resumed: the gate is checked again first, then it goes on.
|
||||
fleet.fw["c1"].services[1]["running"] = 1
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
|
||||
assert resp.status_code == 200 and resp.json()["status"] == "resuming"
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert fleet.fw["ha-a"].update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resume_rechecks_and_pauses_again_if_still_broken(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
fleet.fw["c1"].gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100 %", "delay": "~", "stddev": "~"}
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
|
||||
again = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert again.pause_reason == "gate_failed" and again.pause_detail.get("on_resume") is True
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
|
||||
|
||||
# ── never twice, never in parallel ─────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_restart_in_the_middle_of_job_wait_submits_exactly_once(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
|
||||
async def in_job_wait(r):
|
||||
return any(s.host_name == "ha-b" and s.phase == "job_wait" for s in await _scheds())
|
||||
for _ in range(300):
|
||||
if await in_job_wait(None):
|
||||
break
|
||||
await _tick(fleet, clock)
|
||||
# "Restart": nothing survives but the database (each tick opens fresh sessions anyway); two backends
|
||||
# even step the same run at the very same moment.
|
||||
await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(3)))
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded"
|
||||
assert fleet.fw["ha-b"].update.call_count == 1 and len([s for s in await _scheds() if s.host_name == "ha-b"]) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_two_backends_deciding_at_once_create_one_schedule(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "wave_running")
|
||||
await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(4)))
|
||||
assert len(await _scheds()) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_cluster_in_one_wave_goes_backup_first_and_never_both_at_once(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
|
||||
run = await _start(client, plan)
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert [s.host_name for s in await _scheds()] == ["ha-b", "ha-a"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_master_without_handover_pauses_before_anything_happens_to_it(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": False}])
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "master_needs_handover" and paused.pause_detail["host"] == "ha-a"
|
||||
assert fleet.fw["ha-b"].update.call_count == 1 and fleet.fw["ha-a"].update.call_count == 0
|
||||
assert fleet.fw["ha-a"].toggles.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cluster_gate_before_the_second_node(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
|
||||
run = await _start(client, plan)
|
||||
|
||||
async def break_backup_after_update():
|
||||
if fleet.fw["ha-b"].version == "26.7.7":
|
||||
fleet.fw["ha-b"].services[0]["running"] = 0 # unbound dead on the freshly updated node
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused", check=break_backup_after_update)
|
||||
assert paused.pause_reason == "cluster_gate_failed" and paused.pause_detail["host"] == "ha-a"
|
||||
assert fleet.fw["ha-a"].update.call_count == 0 and fleet.fw["ha-a"].toggles.call_count == 0
|
||||
|
||||
|
||||
# ── stop when in doubt ─────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_different_target_version_in_a_later_ring_pauses(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["ha-b"].latest = "26.7.8" # a newer release appeared after the canary ran
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "target_version_changed"
|
||||
assert paused.pause_detail == {"expected": "26.7.7", "offered": "26.7.8"}
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hosts_without_an_update_are_skipped(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["ha-b"].offer = "none"
|
||||
run = await _start(client, plan)
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded" and _host(done, "ha-b").status == "skipped"
|
||||
assert _host(done, "ha-b").detail["code"] == "no_update_offered" and done.waves[1].status == "skipped"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_host_pauses_and_resume_needs_an_explicit_skip(client, respx_mock, clock):
|
||||
fleet, hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["c1"].update_error = True
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "host_failed" and _host(paused, "c1").status == "failed"
|
||||
refused = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
|
||||
assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ROLLOUT_HOST_FAILED"
|
||||
skip = await client.post(f"/api/rollouts/runs/{run['id']}/skip-host", json={"host_id": hosts["c1"]["id"], "reason": "geprüft, später"})
|
||||
assert skip.status_code == 200
|
||||
# The failed operation stays "unknown" (locks c1, notifies) until a human closes it; it is not retried.
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert fleet.fw["c1"].update.call_count == 1 and _host(done, "c1").detail["code"] == "skipped_by"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unreachable_host_in_preflight_is_blocked_then_paused(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
fleet.fw["c1"].up = False
|
||||
run = await _start(client, plan)
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "preflight_blocked" and _host(paused, "c1").status == "blocked"
|
||||
assert clock.now - datetime.fromisoformat(paused.progress["paused_at"]) < timedelta(minutes=1)
|
||||
fleet.fw["c1"].up = True
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_someone_elses_open_maintenance_blocks_then_pauses(client, respx_mock, clock):
|
||||
fleet, hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: r.current_wave == 1 and r.status == "wave_running")
|
||||
async with SessionLocal() as db:
|
||||
db.add(HostOperation(host_id=uuid.UUID(hosts["ha-a"]["id"]), host_name="ha-a", actor="someone", action="reboot",
|
||||
status="submitted", expires_at=clock.now, fingerprint="", snapshot={}))
|
||||
await db.commit()
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason in ("blocked_by_maintenance", "cluster_gate_failed")
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_actor_who_lost_the_right_pauses_the_run(client, respx_mock, clock):
|
||||
from tests.test_operator_role import _operator
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
op = await _operator(client, "rollengine-op")
|
||||
try:
|
||||
run = await _start(client, plan, api=op)
|
||||
assert run["started_by"] == "rollengine-op"
|
||||
await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
|
||||
assert paused.pause_reason == "actor_invalid" and fleet.fw["ha-b"].update.call_count == 0
|
||||
# An admin takes over by resuming.
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).json()["started_by"] == "admin"
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded"
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
|
||||
|
||||
# ── windows, approval, skip, abort, automatic start ───────────────────────────
|
||||
|
||||
async def _window_now(client, clock, *, cutoff=60, duration=120):
|
||||
local = clock.now
|
||||
body = {"name": f"jetzt-{uuid.uuid4().hex[:6]}", "timezone": "UTC", "recurrence": "weekly", "weekdays": [local.weekday()],
|
||||
"start_time": (local - timedelta(minutes=1)).strftime("%H:%M"), "duration_minutes": duration, "start_cutoff_minutes": cutoff}
|
||||
if (local - timedelta(minutes=1)).date() != local.date():
|
||||
pytest.skip("too close to midnight for a same-day window")
|
||||
resp = await client.post("/api/maintenance-windows", json=body)
|
||||
assert resp.status_code == 201, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_closed_window_waits_and_the_next_occurrence_resumes(client, respx_mock, clock):
|
||||
window = await _window_now(client, clock, cutoff=15)
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"])
|
||||
run = await _start(client, plan)
|
||||
waiting = await _until(fleet, clock, run["id"], lambda r: r.status in ("waiting_window", "paused", "succeeded"))
|
||||
assert waiting.status == "waiting_window" and waiting.current_wave >= 1
|
||||
started = {s.host_name for s in await _scheds()}
|
||||
assert "c1" in started and "ha-a" not in started
|
||||
clock.now += timedelta(days=7) - timedelta(minutes=40) # one week later, a few minutes into the next occurrence
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_approval_holds_the_wave(client, respx_mock, clock):
|
||||
fleet, _hosts, groups, plan = await _setup(client, respx_mock, waves=[
|
||||
{"label": "canary", "group_ids": ["canary"], "soak_minutes": 1},
|
||||
{"label": "rest", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True, "require_approval": True}])
|
||||
run = await _start(client, plan)
|
||||
held = await _until(fleet, clock, run["id"], lambda r: r.status == "awaiting_approval")
|
||||
for _ in range(10):
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw["ha-b"].update.call_count == 0
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/approve")
|
||||
assert resp.status_code == 200 and resp.json()["waves"][1]["approved_by"] == "admin"
|
||||
done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
|
||||
assert done.status == "succeeded" and held.current_wave == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_abort_cancels_what_has_not_started_and_lets_the_running_host_finish(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await _until(fleet, clock, run["id"], lambda r: any(h.status == "running" for h in r.hosts))
|
||||
bad = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "falsch"})
|
||||
assert bad.status_code == 409 and bad.json()["detail"]["code"] == "ROLLOUT_CONFIRM_NAME"
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "ringe"})
|
||||
assert resp.status_code == 200 and resp.json()["status"] == "cancelled"
|
||||
for _ in range(40):
|
||||
await _tick(fleet, clock)
|
||||
run_now = await _run(run["id"])
|
||||
assert run_now.status == "cancelled"
|
||||
assert {h.host_name: h.status for h in run_now.hosts} in (
|
||||
{"c1": "succeeded", "ha-b": "cancelled", "ha-a": "cancelled"}, {"c1": "cancelled", "ha-b": "cancelled", "ha-a": "cancelled"})
|
||||
assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
|
||||
assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pause_by_hand_and_operator_may_always_brake(client, respx_mock, clock):
|
||||
from tests.test_operator_role import _operator
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": False})
|
||||
op = await _operator(client, "brake-op")
|
||||
try:
|
||||
resp = await op.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "Bauchgefühl"})
|
||||
assert resp.status_code == 200 and resp.json()["pause_reason"] == "manual"
|
||||
denied = await op.post(f"/api/rollouts/runs/{run['id']}/resume")
|
||||
assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
|
||||
finally:
|
||||
await op.aclose()
|
||||
await client.put("/api/settings/governance", json={"operators_may_deploy": True})
|
||||
for _ in range(10):
|
||||
await _tick(fleet, clock)
|
||||
assert sum(fw.update.call_count for fw in fleet.fw.values()) == 0
|
||||
second = await client.post(f"/api/rollouts/plans/{plan['id']}/start", json={
|
||||
"preview_token": (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"],
|
||||
"confirm_name": "ringe", "acknowledge_unattended": True})
|
||||
assert second.status_code == 409 and second.json()["detail"]["code"] == "ROLLOUT_ALREADY_ACTIVE"
|
||||
|
||||
|
||||
async def _until_step_pending(fleet, clock, run_id):
|
||||
"""Right after the engine created a host's schedule (fire_due has not seen it yet)."""
|
||||
run = await _until(fleet, clock, run_id, lambda r: any(h.status == "running" for h in r.hosts))
|
||||
name = next(h.host_name for h in run.hosts if h.status == "running")
|
||||
assert [s.status for s in await _scheds()] == ["pending"]
|
||||
return name
|
||||
|
||||
|
||||
async def _resume_to_the_end(client, fleet, clock, run_id):
|
||||
assert (await client.post(f"/api/rollouts/runs/{run_id}/resume")).status_code == 200
|
||||
done = await _until(fleet, clock, run_id, lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
|
||||
assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
|
||||
assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pause_while_a_step_is_pending_really_brakes_and_resume_runs_it_once(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
name = await _until_step_pending(fleet, clock, run["id"])
|
||||
resp = await client.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "halt"})
|
||||
assert resp.status_code == 200
|
||||
for _ in range(10):
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw[name].update.call_count == 0 # the pending step did not fire behind the pause
|
||||
paused = await _run(run["id"])
|
||||
assert paused.status == "paused" and _host(paused, name).status == "queued"
|
||||
assert _host(paused, name).scheduled_maintenance_id is None
|
||||
assert [s.status for s in await _scheds()] == ["cancelled"]
|
||||
await _resume_to_the_end(client, fleet, clock, run["id"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_step_found_long_overdue_after_an_outage_is_not_fired(client, respx_mock, clock):
|
||||
fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
|
||||
run = await _start(client, plan)
|
||||
name = await _until_step_pending(fleet, clock, run["id"])
|
||||
clock.now += timedelta(minutes=30) # OPNexus was down (e.g. a web update); the window may be over by now
|
||||
await _tick(fleet, clock)
|
||||
assert fleet.fw[name].update.call_count == 0
|
||||
paused = await _run(run["id"])
|
||||
assert (paused.status, paused.pause_reason) == ("paused", "host_not_started")
|
||||
assert _host(paused, name).status == "queued" and _host(paused, name).detail["code"] == "not_fired"
|
||||
(sched,) = await _scheds()
|
||||
assert sched.status == "failed" and sched.result["not_fired"]
|
||||
await _resume_to_the_end(client, fleet, clock, run["id"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automatic_start_once_per_occurrence_and_only_while_unchanged(client, respx_mock, clock):
|
||||
window = await _window_now(client, clock)
|
||||
fleet, _hosts, groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
|
||||
for _ in range(3):
|
||||
async with SessionLocal() as db:
|
||||
await engine.auto_start(db)
|
||||
async with SessionLocal() as db:
|
||||
runs = (await db.scalars(select(RolloutRun))).all()
|
||||
assert len(runs) == 1 and runs[0].trigger == "window" and runs[0].started_by == "admin"
|
||||
assert runs[0].window_occurrence is not None
|
||||
|
||||
# Next week, but somebody added a host to the canary group meanwhile: no start, a notification instead.
|
||||
async with SessionLocal() as db:
|
||||
(await db.get(RolloutRun, runs[0].id)).status = "cancelled"
|
||||
await db.commit()
|
||||
await client.put(f"/api/groups/{groups['canary']}", json={"name": "canary", "host_ids": [_hosts["c1"]["id"], _hosts["ha-a"]["id"]]})
|
||||
clock.now += timedelta(days=7)
|
||||
async with SessionLocal() as db:
|
||||
await engine.auto_start(db)
|
||||
async with SessionLocal() as db:
|
||||
assert len((await db.scalars(select(RolloutRun))).all()) == 1
|
||||
stored = await db.get(RolloutPlan, uuid.UUID(plan["id"]))
|
||||
assert stored.auto_start_error["code"] == "plan_changed"
|
||||
from app.services import notification_checks
|
||||
async with SessionLocal() as db:
|
||||
problems = await notification_checks.check_rollout_start_failed(db)
|
||||
assert list(problems.values())[0]["grund"] == "plan_changed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_second_backend_waiting_for_the_lock_reports_no_refusal(client, respx_mock, clock, monkeypatch):
|
||||
window = await _window_now(client, clock)
|
||||
_fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
|
||||
token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
|
||||
assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
|
||||
real_lock = engine.policy_lock
|
||||
|
||||
async def racing_lock(db):
|
||||
monkeypatch.setattr(engine, "policy_lock", real_lock)
|
||||
async with SessionLocal() as other: # backend A starts the run while B waits for the lock
|
||||
assert await engine.auto_start(other) == 1
|
||||
await real_lock(db)
|
||||
monkeypatch.setattr(engine, "policy_lock", racing_lock)
|
||||
async with SessionLocal() as db:
|
||||
assert await engine.auto_start(db) == 0
|
||||
async with SessionLocal() as db:
|
||||
assert len((await db.scalars(select(RolloutRun))).all()) == 1
|
||||
assert (await db.get(RolloutPlan, uuid.UUID(plan["id"]))).auto_start_error is None
|
||||
|
||||
|
||||
def test_target_version_comes_from_the_core_package_not_product_latest():
|
||||
# Shape seen live on test-fw-3 (26.7.1_1): product_latest lagged behind what would really be installed.
|
||||
status = {"status": "update", "product": {"product_latest": "26.7.2"},
|
||||
"all_packages": {"opnsense": {"old": "26.7.1_1", "new": "26.7.7", "reason": "upgrade"}},
|
||||
"upgrade_packages": [{"name": "opnsense", "current_version": "26.7.1_1", "new_version": "26.7.7"}]}
|
||||
assert engine.target_of(status, "update") == "26.7.7"
|
||||
assert engine.target_of({**status, "all_packages": {}}, "update") == "26.7.7"
|
||||
assert engine.target_of({"product": {"product_latest": "26.7.7"}}, "update") == "26.7.7"
|
||||
assert engine.target_of({"upgrade_major_version": "27.1"}, "major_update") == "27.1"
|
||||
@@ -0,0 +1,195 @@
|
||||
"""Health gate checks of the rollout engine, one by one, against a fake firewall pair."""
|
||||
import uuid
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models.carp_maintenance_window import CarpMaintenanceWindow
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.models.host_operation import HostOperation
|
||||
from app.services import rollout_health as rh
|
||||
from tests.test_rules import _create_host
|
||||
|
||||
A, B = "https://gw-a.test", "https://gw-b.test"
|
||||
|
||||
|
||||
class Fw:
|
||||
def __init__(self, respx_mock, base, *, role="MASTER", version="26.7.7"):
|
||||
self.base, self.role, self.version, self.up, self.job, self.maint = base, role, version, True, "ready", False
|
||||
self.vips = None
|
||||
self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1.0 ms", "stddev": "0.1 ms"}]
|
||||
self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
|
||||
{"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
|
||||
self.gw_error = False
|
||||
r = respx_mock
|
||||
r.get(f"{base}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
|
||||
r.get(f"{base}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
|
||||
r.get(f"{base}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
|
||||
r.get(f"{base}/api/routes/gateway/status").mock(side_effect=self._gw)
|
||||
r.get(f"{base}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
|
||||
|
||||
def _ok(self, data):
|
||||
if not self.up:
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, json=data)
|
||||
|
||||
def _gw(self, request):
|
||||
if self.gw_error:
|
||||
return httpx.Response(403, json={"message": "forbidden"})
|
||||
return self._ok({"items": self.gateways, "status": "ok"})
|
||||
|
||||
def _vip(self, request):
|
||||
rows = [{"vhid": v, "status": s} for v, s in (self.vips or {"1": self.role}).items()]
|
||||
return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "0", "status_msg": ""}})
|
||||
|
||||
|
||||
async def _hosts(client, cluster=True):
|
||||
a = await _create_host(client, "gw-a", A)
|
||||
if not cluster:
|
||||
return a, None
|
||||
b = await _create_host(client, "gw-b", B)
|
||||
if cluster:
|
||||
for h in (a, b):
|
||||
await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
|
||||
return a, b
|
||||
|
||||
|
||||
async def _eval(host_ids, baselines=None, target="26.7.7", **kw):
|
||||
async with SessionLocal() as db:
|
||||
hosts = [await db.get(OpnsenseHost, uuid.UUID(h)) for h in host_ids]
|
||||
if baselines is None:
|
||||
baselines = {str(h.id): await rh.baseline(db, h) for h in hosts}
|
||||
return await rh.evaluate(db, hosts, baselines, target_version=target, **kw), baselines
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_healthy_cluster_passes_every_check(client, respx_mock):
|
||||
Fw(respx_mock, A)
|
||||
Fw(respx_mock, B, role="BACKUP")
|
||||
a, _ = await _hosts(client)
|
||||
gate, base = await _eval([a["id"]])
|
||||
assert gate["verdict"] == "ok", gate
|
||||
row = gate["hosts"][a["id"]]
|
||||
assert {k: v["result"] for k, v in row.items()} == {
|
||||
"online": "ok", "version": "ok", "carp": "ok", "gateways": "ok", "services": "ok", "locks": "ok"}
|
||||
assert base[a["id"]]["gateways"] == {"WAN_GW": "online"} and base[a["id"]]["services"]["ntpd"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_offline_or_busy_host_and_wrong_version_fail(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
fa.version = "26.7.6"
|
||||
gate, base = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["version"] == {"result": "fail", "code": "version_mismatch",
|
||||
"params": {"expected": "26.7.7", "actual": "26.7.6"}}
|
||||
fa.job = "busy"
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["online"]["code"] == "firmware_busy"
|
||||
fa.up = False
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["online"]["code"] == "offline" and gate["verdict"] == "fail"
|
||||
assert gate["hosts"][a["id"]]["gateways"]["result"] == "fail" # readable before, not now
|
||||
assert rh.failed_checks(gate)[0].startswith(a["id"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_carp_split_brain_maintenance_mode_and_timer_fail(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
fb = Fw(respx_mock, B, role="BACKUP")
|
||||
a, b = await _hosts(client)
|
||||
fb.role = "MASTER" # both MASTER on VHID 1
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "vhid_roles", "params": {"vhids": "1"}}
|
||||
fb.role = "BACKUP"
|
||||
fa.vips, fb.vips = {"1": "MASTER", "2": "MASTER"}, {"1": "BACKUP", "2": "MASTER"} # partial failover on VHID 2
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"]["params"] == {"vhids": "2"}
|
||||
fa.vips = fb.vips = None
|
||||
fb.maint = True
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"]["code"] == "maintenance_mode"
|
||||
fb.maint = False
|
||||
async with SessionLocal() as db:
|
||||
db.add(CarpMaintenanceWindow(host_id=uuid.UUID(b["id"]), host_name="gw-b", status="active"))
|
||||
await db.commit()
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"]["code"] == "auto_revert_active"
|
||||
fb.up = False
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "node_offline", "params": {"names": "gw-b"}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateways_compare_against_the_baseline(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
fa.gateways.append({"name": "OLD_GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms"})
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
_gate, base = await _eval([a["id"]])
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["code"] == "gateway_bad_before" # only warns
|
||||
fa.gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~"}
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["gateways"] == {"result": "fail", "code": "gateway_down", "params": {"names": "WAN_GW"}}
|
||||
fa.gateways[0] = {"name": "WAN_GW", "status": "loss", "loss": "30 %", "delay": "5 ms", "stddev": "1 ms"}
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "warn"
|
||||
fa.gw_error = True
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "fail"
|
||||
gate, _ = await _eval([a["id"]], {a["id"]: {**base[a["id"]], "gateways": None}})
|
||||
assert gate["hosts"][a["id"]]["gateways"]["result"] == "skip" # no permission before either: nothing to compare
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_services_that_ran_before_must_run_again(client, respx_mock):
|
||||
fa = Fw(respx_mock, A)
|
||||
fa.services.append({"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0})
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
_gate, base = await _eval([a["id"]])
|
||||
fa.services[1] = {"id": "ntpd", "name": "ntpd", "locked": 0, "running": 0}
|
||||
gate, _ = await _eval([a["id"]], base)
|
||||
assert gate["hosts"][a["id"]]["services"] == {"result": "fail", "code": "service_stopped", "params": {"names": "ntpd"}}
|
||||
fa.services[1]["running"] = 1
|
||||
assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["services"]["result"] == "ok" # dhcpd6 never ran: fine
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_open_operation_in_the_cluster_is_a_lock(client, respx_mock):
|
||||
Fw(respx_mock, A)
|
||||
Fw(respx_mock, B, role="BACKUP")
|
||||
a, b = await _hosts(client)
|
||||
from datetime import datetime, timezone
|
||||
async with SessionLocal() as db:
|
||||
db.add(HostOperation(host_id=uuid.UUID(b["id"]), host_name="gw-b", actor="x", action="reboot", status="submitted",
|
||||
expires_at=datetime.now(timezone.utc), fingerprint="", snapshot={}))
|
||||
await db.commit()
|
||||
gate, _ = await _eval([a["id"]])
|
||||
assert gate["hosts"][a["id"]]["locks"] == {"result": "fail", "code": "open_locks", "params": {"count": 1}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode,expected", [("warn", "warn"), ("fail", "fail")])
|
||||
@pytest.mark.asyncio
|
||||
async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeypatch, mode, expected):
|
||||
Fw(respx_mock, A)
|
||||
a, _ = await _hosts(client, cluster=False)
|
||||
async with SessionLocal() as db:
|
||||
db.add(ComplianceResult(host_id=uuid.UUID(a["id"]), checked_at=datetime_now(), score=80, findings=[]))
|
||||
await db.commit()
|
||||
|
||||
async def fake_check(db, host):
|
||||
row = await db.get(ComplianceResult, (await db.scalar(__import__("sqlalchemy").select(ComplianceResult.id))))
|
||||
row.score = 70
|
||||
await db.commit()
|
||||
return row
|
||||
monkeypatch.setattr(rh.compliance_service, "check_host", fake_check)
|
||||
gate, _ = await _eval([a["id"]], checks={"compliance": True}, compliance_mode=mode)
|
||||
assert gate["hosts"][a["id"]]["compliance"] == {"result": expected, "code": "score_lower", "params": {"before": 80, "after": 70}}
|
||||
assert gate["verdict"] == expected
|
||||
# the gate cannot be switched off where it matters
|
||||
gate, _ = await _eval([a["id"]], checks={"online": False, "version": False, "locks": False})
|
||||
assert {"online", "version", "locks"} <= set(gate["hosts"][a["id"]])
|
||||
|
||||
|
||||
def datetime_now():
|
||||
from datetime import datetime, timezone
|
||||
return datetime.now(timezone.utc)
|
||||
@@ -0,0 +1,306 @@
|
||||
"""auto_verify (1.25): an unattended action checks its own result and closes the operation,
|
||||
so the host/cluster is not left locked until a human clicks "Wartung abschließen"."""
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models import HostOperation
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.services import scheduled_maintenance_service as svc
|
||||
from tests.test_rules import _create_host
|
||||
from tests.test_scheduled_handover import FakeCluster, _setup
|
||||
|
||||
H = "https://fw-solo.test"
|
||||
|
||||
|
||||
class FakeHost:
|
||||
"""One stand-alone firewall: job = ready | running | down; boot = what systemTime reports."""
|
||||
|
||||
def __init__(self, respx_mock, *, firmware="update", reboot_flag="1", version="26.7.5"):
|
||||
self.job, self.boot, self.version = "ready", "Mon Oct 5 10:00:00 UTC 2026", version
|
||||
self.update = respx_mock.post(f"{H}/api/core/firmware/update").respond(200, json={"status": "ok"})
|
||||
self.reboot = respx_mock.post(f"{H}/api/core/system/reboot").respond(200, json={"status": "ok"})
|
||||
respx_mock.get(f"{H}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
|
||||
respx_mock.get(f"{H}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
|
||||
respx_mock.get(f"{H}/api/core/firmware/info").mock(side_effect=lambda r: self._up(json={"product_version": self.version}))
|
||||
respx_mock.get(f"{H}/api/core/firmware/status").respond(
|
||||
200, json={"status": firmware, "all_packages": {}, "status_msg": "x", "status_reboot": reboot_flag})
|
||||
respx_mock.get(f"{H}/api/core/firmware/running").mock(side_effect=lambda r: self._up(json={"status": self.job}))
|
||||
respx_mock.get(f"{H}/api/diagnostics/system/systemTime").mock(side_effect=lambda r: self._up(json={"boottime": self.boot}))
|
||||
|
||||
def _up(self, **kw):
|
||||
if self.job == "down":
|
||||
raise httpx.ConnectError("rebooting")
|
||||
return httpx.Response(200, **kw)
|
||||
|
||||
def _vip(self, request):
|
||||
return self._up(json={"rows": [], "carp": {"allow": "1", "maintenancemode": False, "demotion": "0", "status_msg": ""}})
|
||||
|
||||
def reboot_now(self, new_boot="Wed Oct 21 03:00:00 UTC 2026", version=None):
|
||||
self.boot = new_boot
|
||||
if version:
|
||||
self.version = version
|
||||
|
||||
|
||||
async def _plan(host, action="update", *, auto_verify=True, handover=False):
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(
|
||||
host_id=uuid.UUID(host["id"]), host_name=host["name"], action=action, carp_handover=handover,
|
||||
auto_verify=auto_verify, run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin",
|
||||
created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
return s.id
|
||||
|
||||
|
||||
async def _fire():
|
||||
async with SessionLocal() as db:
|
||||
return await svc.fire_due(db)
|
||||
|
||||
|
||||
async def _advance():
|
||||
async with SessionLocal() as db:
|
||||
return await svc.advance_running(db)
|
||||
|
||||
|
||||
async def _get(sid):
|
||||
async with SessionLocal() as db:
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
op = await db.get(HostOperation, s.host_operation_id) if s.host_operation_id else None
|
||||
return s, op
|
||||
|
||||
|
||||
async def _shift(sid, *, phase=None, ready=None):
|
||||
"""Pretend the phase began / the firmware turned ready that long ago."""
|
||||
async with SessionLocal() as db:
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
if phase is not None:
|
||||
s.phase_since = datetime.now(timezone.utc) - phase
|
||||
if ready is not None:
|
||||
s.progress = {**s.progress, "ready_since": (datetime.now(timezone.utc) - ready).isoformat()}
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _settle(sid):
|
||||
"""verify needs the firmware idle for VERIFY_SETTLE: one tick to notice, then pretend it has been a while."""
|
||||
await _advance()
|
||||
await _shift(sid, ready=svc.VERIFY_SETTLE + timedelta(seconds=1))
|
||||
await _advance()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_is_verified_and_the_operation_closed(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
s, op = await _get(sid)
|
||||
assert (s.status, s.phase, op.status, fw.update.call_count) == ("running", "job_wait", "submitted", 1)
|
||||
assert s.progress["reboot_expected"] is True and s.progress["boot_before"] == fw.boot
|
||||
|
||||
fw.job = "running"
|
||||
await _advance()
|
||||
fw.job = "down"
|
||||
await _advance()
|
||||
fw.job = "ready"
|
||||
fw.reboot_now(version="26.7.7")
|
||||
await _advance()
|
||||
assert (await _get(sid))[0].phase == "verify"
|
||||
await _settle(sid)
|
||||
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "succeeded" and s.phase is None
|
||||
assert s.result["auto_verified"] is True and s.result["version_after"] == "26.7.7" and s.result["version_before"] == "26.7.5"
|
||||
assert op.status == "acknowledged" and "Automatisch geprüft" in op.message
|
||||
assert fw.update.call_count == 1
|
||||
# the host is free again: a new preview is not blocked by a forgotten operation
|
||||
resp = await client.get(f"/api/hosts/{host['id']}/maintenance")
|
||||
assert not any("frühere Wartungsaktion" in b for b in resp.json()["state"]["blocked"])
|
||||
async with SessionLocal() as db:
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
|
||||
assert {e.actor for e in events if (e.details or {}).get("scheduled")} == {"admin (geplant)"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_boot_time_proves_a_reboot_the_poll_never_saw(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
await _shift(sid, phase=timedelta(0))
|
||||
fw.reboot_now() # rebooted between two polls: never seen offline
|
||||
async with SessionLocal() as db: # the job grace has passed
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
|
||||
await db.commit()
|
||||
await _advance()
|
||||
assert (await _get(sid))[0].phase == "verify"
|
||||
await _settle(sid)
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "succeeded" and s.result["rebooted"] is True and op.status == "acknowledged"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_expected_reboot_never_happened_ends_unknown_after_timeout(client, respx_mock):
|
||||
FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
async with SessionLocal() as db:
|
||||
s = await db.get(ScheduledMaintenance, sid)
|
||||
s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
|
||||
await db.commit()
|
||||
await _advance()
|
||||
await _settle(sid)
|
||||
s, op = await _get(sid)
|
||||
assert (s.status, s.phase, op.status) == ("running", "verify", "submitted") # waiting, not done
|
||||
assert "Neustart" in s.progress["waiting_for"]
|
||||
await _shift(sid, phase=svc.VERIFY_TIMEOUT)
|
||||
await _advance()
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "failed" and op.status == "unknown" and "nicht bestanden" in s.result["error"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ready_must_hold_a_while_a_short_ready_before_the_reboot_does_not_count(client, respx_mock):
|
||||
fw = FakeHost(respx_mock, reboot_flag="0")
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
fw.job = "running"
|
||||
await _advance()
|
||||
fw.job = "ready"
|
||||
await _advance() # job_wait -> verify
|
||||
await _advance() # ready noticed
|
||||
s, _ = await _get(sid)
|
||||
assert s.phase == "verify" and s.progress["ready_since"]
|
||||
fw.job = "running" # busy again (e.g. the reboot at the end of the update)
|
||||
await _advance()
|
||||
assert (await _get(sid))[0].progress["ready_since"] is None
|
||||
fw.job = "ready"
|
||||
await _settle(sid)
|
||||
assert (await _get(sid))[0].status == "succeeded"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_without_auto_verify_nothing_changes(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host, auto_verify=False)
|
||||
await _fire()
|
||||
s, op = await _get(sid)
|
||||
assert (s.status, s.phase, op.status, fw.update.call_count) == ("succeeded", None, "submitted", 1)
|
||||
assert await _advance() == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_human_closing_the_operation_stops_the_check(client, respx_mock):
|
||||
FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
await _fire()
|
||||
s, op = await _get(sid)
|
||||
async with SessionLocal() as db:
|
||||
(await db.get(HostOperation, op.id)).status = "cancelled"
|
||||
await db.commit()
|
||||
await _advance()
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "failed" and "von Hand beendet" in s.result["error"] and op.status == "cancelled"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handover_verifies_before_taking_the_traffic_back(client, respx_mock):
|
||||
cluster = FakeCluster(respx_mock)
|
||||
cluster.boot = "before"
|
||||
|
||||
def system_time(request):
|
||||
if cluster.job == "down":
|
||||
raise httpx.ConnectError("down")
|
||||
return httpx.Response(200, json={"boottime": cluster.boot})
|
||||
respx_mock.get("https://fw-a.test/api/diagnostics/system/systemTime").mock(side_effect=system_time)
|
||||
a, _b = await _setup(client)
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True, auto_verify=True,
|
||||
run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
sid = s.id
|
||||
await _fire()
|
||||
await _advance() # partner MASTER -> job submitted
|
||||
cluster.job = "down"
|
||||
await _advance()
|
||||
cluster.job = "ready"
|
||||
cluster.boot = "after"
|
||||
await _advance()
|
||||
s, _ = await _get(sid)
|
||||
assert s.phase == "verify" and cluster.toggles.call_count == 1 # still demoted while checking
|
||||
await _settle(sid)
|
||||
assert (await _get(sid))[0].phase == "leave" and cluster.toggles.call_count == 1
|
||||
await _advance() # leave
|
||||
await _advance() # leave_wait -> done
|
||||
s, op = await _get(sid)
|
||||
assert s.status == "succeeded" and s.result["handover"] is True and s.result["auto_verified"] is True
|
||||
assert s.result["rebooted"] is True # found live: the handover path dropped what verify had seen
|
||||
assert op.status == "acknowledged" and cluster.toggles.call_count == 2 and cluster.update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_due_row_locked_by_another_backend_is_not_fired_twice(client, respx_mock):
|
||||
fw = FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
sid = await _plan(host)
|
||||
async with SessionLocal() as other: # another backend is firing this very row right now
|
||||
await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
|
||||
# Without the row lock this would try to update the locked row and hang: bounded, so it fails instead.
|
||||
result = await asyncio.wait_for(_fire(), timeout=10)
|
||||
assert result["due"] == 0 and fw.update.call_count == 0
|
||||
await other.rollback()
|
||||
await _fire()
|
||||
assert fw.update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_running_row_locked_by_another_backend_is_not_stepped_twice(client, respx_mock):
|
||||
cluster = FakeCluster(respx_mock)
|
||||
a, _b = await _setup(client)
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True,
|
||||
run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
sid = s.id
|
||||
await _fire()
|
||||
async with SessionLocal() as other:
|
||||
await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
|
||||
assert await asyncio.wait_for(_advance(), timeout=10) == 1 # still counted as running, but not touched
|
||||
assert cluster.update.call_count == 0
|
||||
await other.rollback()
|
||||
await _advance()
|
||||
assert cluster.update.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancel_waits_for_a_firing_row_and_never_overwrites_running(client, respx_mock):
|
||||
FakeHost(respx_mock)
|
||||
host = await _create_host(client, "fw-solo", H)
|
||||
async with SessionLocal() as db:
|
||||
s = ScheduledMaintenance(host_id=uuid.UUID(host["id"]), host_name="fw-solo", action="update", auto_verify=True,
|
||||
run_at=datetime.now(timezone.utc) + timedelta(hours=1), actor="admin", created_at=datetime.now(timezone.utc))
|
||||
db.add(s)
|
||||
await db.commit()
|
||||
sid = s.id
|
||||
async with SessionLocal() as firing: # the poll loop has the row and is about to mark it running
|
||||
row = await firing.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
|
||||
cancel = asyncio.create_task(client.delete(f"/api/hosts/{host['id']}/maintenance/schedule/{sid}"))
|
||||
await asyncio.sleep(0.5)
|
||||
row.status = "running"
|
||||
await firing.commit()
|
||||
resp = await asyncio.wait_for(cancel, timeout=10)
|
||||
assert resp.status_code == 409
|
||||
assert (await _get(sid))[0].status == "running"
|
||||
@@ -47,6 +47,7 @@ async def test_status_reports_update_and_can_update(client, respx_mock):
|
||||
body = (await client.get("/api/system/updater")).json()
|
||||
assert body["available"] is True and body["can_update"] is True
|
||||
assert body["latest"] == _bump(minor=1) and body["major_change"] is False
|
||||
assert body["active_rollouts"] == 0 # the UI warns before restarting OPNexus in the middle of a rollout
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
+158
-6
@@ -95,7 +95,8 @@ dem Tag.
|
||||
|
||||
`opnexus update` prüft die Signaturen selbst, bevor es etwas wechselt, sobald `cosign`
|
||||
installiert ist und `cosign.pub` neben der `docker-compose.yml` liegt (`cosign.pub` liegt im Bundle
|
||||
bereits dabei, `cosign` selbst musst du installieren). Schlägt die Prüfung fehl, bricht das
|
||||
bereits dabei, `cosign` selbst musst du installieren). Der Updater-Container für Updates per Klick
|
||||
bringt `cosign` mit und prüft deshalb ebenfalls, solange `cosign.pub` im Projektverzeichnis liegt. Schlägt die Prüfung fehl, bricht das
|
||||
Update ab und es wurde nichts verändert. Fehlt `cosign`, sagt `opnexus update` das und
|
||||
macht ohne Prüfung weiter.
|
||||
|
||||
@@ -487,6 +488,37 @@ Beschreibung und Systemregeln. Die Oberfläche nennt den Grund je Regel.
|
||||
> strikt „erster Treffer gewinnt“ aus. Die Reihenfolge ist dadurch
|
||||
> sicherheitsrelevant, siehe [hierarchische Policies](#gerätegruppen-und-hierarchische-policies).
|
||||
|
||||
### Aliase und NAT von der Firewall übernehmen
|
||||
|
||||
Wie die Regeln lassen sich auch **Aliase** und **NAT-Regeln** (Port Forward, 1:1, Outbound) übernehmen, die von Hand
|
||||
in OPNsense entstanden sind: auf der jeweiligen Seite **Von Firewall übernehmen**, Firewall wählen, dann entweder ein
|
||||
Objekt mit **Übernehmen** oder alle auf einmal mit **Alle übernehmbaren übernehmen**. So kommt eine bestehende Firewall
|
||||
ohne Abtippen unter zentrale Verwaltung.
|
||||
|
||||
Die Übernahme **schreibt nichts auf die Firewall**. OPNexus liest das Objekt frisch von der Firewall, legt es zentral an
|
||||
(Ziel: genau diese Firewall) und merkt sich die UUID aus OPNsense. Deshalb zeigt die Vorschau direkt danach „keine Änderung“,
|
||||
und der nächste Rollout nach einer Änderung aktualisiert dasselbe Objekt, statt ein zweites anzulegen. Die Übernahme steht im
|
||||
Audit-Log; Operatoren dürfen übernehmen (sie legt nur zentrale Einträge an), Nur-Lese-Benutzer nicht.
|
||||
|
||||
Nicht übernommen wird, was das Modell verfälschen würde. Die Oberfläche nennt den Grund je Objekt:
|
||||
|
||||
| Objekt | Nicht übernehmbar, weil … |
|
||||
|---|---|
|
||||
| Aliase | OPNsense-eigen (`bogons`, `sshlockout`, `__lan_network` …); Typ unbekannt für OPNexus (URL, URL-Tabelle, GeoIP, dynamische IPv6-Hosts, OpenVPN-Gruppe, „Extern“); der Name existiert in OPNexus schon; Name mit Sonderzeichen; BGP-ASN mit IPv4/IPv6-Einschränkung; Verweis auf einen Alias, der nicht übernommen ist |
|
||||
| Port Forward | mehrere oder keine Schnittstelle; IP-Version weder IPv4 noch IPv6; Protokoll „beliebig“; „Kein Redirect“; kein Ziel; verknüpfte Filterregel; invertierte Quelle oder Ziel; NAT-Reflection-Wert unbekannt; Beschreibung fehlt |
|
||||
| 1:1-NAT | mehrere oder keine Schnittstelle; unbekannter Typ oder NAT-Reflection-Wert; Pflichtangaben oder Beschreibung fehlen |
|
||||
| Outbound-NAT | mehrere oder keine Schnittstelle; „Kein NAT“-Regel (`nonat`); IP-Version weder IPv4 noch IPv6; Beschreibung fehlt |
|
||||
|
||||
Automatisch erzeugte NAT-Regeln (Anti-Lockout, „Auto created rule“) erscheinen nie in der Liste. **Verschachtelte Aliase:**
|
||||
Ein Alias, der einen anderen, noch nicht übernommenen Alias enthält, wartet auf diesen; „Alle übernehmbaren übernehmen“
|
||||
übernimmt beide in der richtigen Reihenfolge, einzeln zuerst den enthaltenen Alias. Verweist er auf einen Alias, der nie
|
||||
übernommen werden kann (z. B. vom Typ URL), bleibt er mit diesem Grund zurück. **Outbound-NAT** lässt sich nur übernehmen,
|
||||
wenn der Outbound-Modus der Firewall Hybrid oder Manuell ist; im Modus Automatisch zeigt OPNsense manuelle Regeln gar nicht an,
|
||||
OPNexus verweist dann darauf und ändert den Modus nicht.
|
||||
|
||||
Einstellungen, die OPNexus nicht verwaltet (Protokollierung und Pool-Optionen beim Port Forward, Tags, Kategorien, statischer
|
||||
NAT-Port), werden nie überschrieben und im Dialog als „Bleibt auf der Firewall unverändert“ angezeigt.
|
||||
|
||||
---
|
||||
|
||||
## Gerätegruppen und hierarchische Policies
|
||||
@@ -790,7 +822,15 @@ frische Backup erneut. Eine blockierte Aktion wird als „fehlgeschlagen“ mark
|
||||
und **nicht** wiederholt. Sie steht im Status der geplanten Wartung und meldet sich
|
||||
zusätzlich per Benachrichtigung („Geplante Wartung nicht ausgeführt“, einzeln
|
||||
schaltbar), damit man es am nächsten Morgen nicht erst beim Nachsehen merkt.
|
||||
Wiederkehrende Zeitpläne gibt es nicht.
|
||||
Wiederkehrende Termine und ganze Flotten laufen über [Rollouts](#rollouts-in-wellen).
|
||||
|
||||
**Ergebnis automatisch prüfen und abschließen** (im Dialog vorausgewählt): Nach dem Auftrag
|
||||
wartet OPNexus, bis der Firmware-Auftrag beendet ist, und prüft dann selbst: Firmware
|
||||
mindestens eine Minute am Stück bereit, ein erwarteter Neustart hat wirklich stattgefunden
|
||||
(Host offline gesehen oder neue Boot-Zeit), Version lesbar. Erst dann wird die Wartungsaktion
|
||||
als abgeschlossen markiert und sperrt Host und Cluster nicht mehr. Gelingt das nicht innerhalb
|
||||
von 20 Minuten, bleibt sie „unklar“ (gesperrt, mit Benachrichtigung). Ohne die Option bleibt
|
||||
sie wie bisher „übergeben“, bis jemand sie von Hand abschließt.
|
||||
|
||||
In einem Cluster läuft unbeaufsichtigte Wartung standardmäßig **nur auf dem
|
||||
CARP-BACKUP**. Ist der Host beim Einplanen gerade MASTER, lehnt OPNexus das sofort mit
|
||||
@@ -817,7 +857,72 @@ der Plan steht auf „fehlgeschlagen“, die Wartungsaktion auf „unklar“, un
|
||||
Benachrichtigung. Der CARP-Umschalter wird nie blind wiederholt. Beendest du die
|
||||
Wartungsaktion von Hand, stoppt der Ablauf und fasst nichts mehr an. Bei Erfolg schließt
|
||||
OPNexus die Wartungsaktion selbst ab. Sie läuft auch einen Neustart des OPNexus-Backends
|
||||
mitten im Ablauf weiter (der Zustand steht in der Datenbank).
|
||||
mitten im Ablauf weiter (der Zustand steht in der Datenbank). Mit „Ergebnis automatisch
|
||||
prüfen“ kommt die Prüfung **vor** dem Verlassen des Wartungsmodus: Der Host bekommt den
|
||||
Verkehr erst zurück, wenn er nachweislich fertig ist.
|
||||
|
||||
### Rollouts in Wellen
|
||||
|
||||
**System › Rollouts** verteilt ein Firmware-Update (stabiles Update oder Major-Upgrade) in
|
||||
Wellen über die Flotte. Die Wellen bestehen aus [Gerätegruppen](#gerätegruppen), die man
|
||||
als Ringe anlegt, zum Beispiel `ring-canary` (eine unwichtige Firewall), `ring-pilot`
|
||||
(einige) und `ring-prod` (der Rest). Steht ein Host in mehreren Wellen, gilt die früheste.
|
||||
|
||||
**Ablauf je Welle:**
|
||||
|
||||
1. **Vorprüfung:** Update-Suche auf den Hosts der Welle, dann Bestand aufnehmen (Version,
|
||||
Gateways, laufende Dienste, Sicherheits-Score). Hosts ohne passendes Update werden
|
||||
übersprungen. Die Version, die in der ersten Welle installiert wird, ist die
|
||||
**Zielversion**; bietet ein Host einer späteren Welle eine andere an, hält der Rollout an
|
||||
(OPNsense installiert immer die neueste, sonst bekäme „prod“ eine Version, die im
|
||||
Canary-Ring nie lief).
|
||||
2. **Hosts nacheinander:** Jeder Host läuft als [geplante Wartung](#geplante-wartung) mit
|
||||
automatischer Prüfung (Backup vor dem Update, Sperren, Audit). Immer nur ein Host,
|
||||
immer nur ein Rollout gleichzeitig. Im Cluster zuerst der BACKUP, dann der MASTER;
|
||||
der MASTER nur, wenn die Welle die CARP-Übergabe erlaubt, sonst hält der Rollout an.
|
||||
Vor dem zweiten Knoten eines Clusters werden der Cluster (alle Knoten erreichbar,
|
||||
genau ein MASTER je VHID, kein Wartungsmodus) und der schon aktualisierte Knoten
|
||||
erneut geprüft.
|
||||
3. **Beobachtungszeit** (je Welle einstellbar), dann die **Gesundheitsprüfung** der
|
||||
aktualisierten Hosts: erreichbar und Firmware bereit, Zielversion installiert, CARP
|
||||
gesund, Gateways und Dienste, die vorher liefen, laufen wieder (vorher schon gestörte
|
||||
nur als Hinweis), keine offenen Sperren, optional Sicherheits-Score nicht gesunken
|
||||
(Hinweis oder Halt). Ist etwas rot, prüft OPNexus zwei Minuten später noch einmal
|
||||
(langsamer Start nach dem Neustart) und hält dann an.
|
||||
4. **Freigabe** (optional je Welle): Die Welle beginnt erst, wenn jemand auf „Welle
|
||||
freigeben“ klickt.
|
||||
|
||||
**Angehalten** wird bei jeder Unklarheit, nie wird etwas von selbst wiederholt: Gate rot,
|
||||
Host fehlgeschlagen oder ungewöhnlich lange unterwegs, Host in der Vorprüfung nicht bereit,
|
||||
abweichende Zielversion, MASTER ohne Übergabe, eine fremde offene Wartung im Cluster länger
|
||||
als 15 Minuten, die verantwortliche Person darf nicht mehr ausrollen, das Fenster wurde
|
||||
gelöscht, der Lauf ist älter als 14 Tage. Es kommt die Benachrichtigung „Rollout
|
||||
angehalten“. **Fortsetzen** prüft zuerst alle bisher aktualisierten Hosts erneut; wer
|
||||
fortsetzt, ist ab dann verantwortlich. Ein fehlgeschlagener Host muss vorher ausdrücklich
|
||||
**übersprungen** werden (seine Wartungsaktion bleibt „unklar“, bis du sie auf der
|
||||
Host-Seite abschließt). **Abbrechen** verwirft alles, was noch nicht begonnen hat; eine
|
||||
Aktion, die schon auf einer Firewall läuft, wird geführt zu Ende gebracht. **Anhalten**
|
||||
darf jeder mit Schreibrecht, alles andere braucht das Recht zum Ausrollen.
|
||||
|
||||
**Wartungsfenster** (Reiter „Wartungsfenster“): wöchentlich an bestimmten Tagen oder
|
||||
monatlich (erster…vierter oder letzter Wochentag, oder ein fester Tag 1–28), Beginn in einer
|
||||
Zeitzone, Dauer und Abschneidezeit: Nach den ersten N Minuten beginnt kein neuer Host mehr,
|
||||
laufende Updates laufen zu Ende, und der Rollout wartet auf den nächsten Termin (dann wird
|
||||
zuerst das Bisherige erneut geprüft). Sommerzeit: Eine Uhrzeit, die es an dem Tag nicht gibt,
|
||||
rückt vor (02:30 → 03:30); eine doppelte gilt beim ersten Mal. Einzelne Termine lassen sich
|
||||
auslassen oder verschieben. Ein Fenster, das ein Plan nutzt, lässt sich nicht löschen.
|
||||
|
||||
**Automatischer Start:** Ein Plan mit Fenster und stabilem Update kann zu jedem Termin von
|
||||
selbst starten. Dazu muss ihn jemand mit Ausroll-Recht nach der Vorschau **aktivieren**.
|
||||
OPNexus merkt sich dabei einen Fingerabdruck von Plan, aufgelösten Hosts und Fenster; ändert
|
||||
sich davon etwas (auch die Mitglieder einer Gruppe), startet nichts, und es kommt „Rollout
|
||||
nicht automatisch gestartet“. Jede Bearbeitung des Plans deaktiviert ihn. So kann niemand
|
||||
die Ziele eines aktivierten Plans unbemerkt ändern.
|
||||
|
||||
Das **Update Center** zeigt den laufenden Rollout als Banner; Hosts, die er noch bearbeitet,
|
||||
sind dort für manuelle Firmware-Aktionen gesperrt. Ein Neustart oder Update von OPNexus
|
||||
während eines Rollouts ist sicher (der Zustand steht in der Datenbank), hält ihn aber für
|
||||
diese Zeit an.
|
||||
|
||||
### Dienste und Systemauslastung
|
||||
|
||||
@@ -943,18 +1048,58 @@ Einstellungen › **Benachrichtigungen**. Kanäle:
|
||||
Absender und Empfängern. Zertifikat und Hostname werden geprüft; ein Mailserver
|
||||
mit privater CA braucht `SMTP_CA_FILE`. Anmeldung ohne Verschlüsselung bleibt
|
||||
gesperrt. Mit **testen** prüfst du jeden Kanal.
|
||||
- **Weitere Kanäle** (Abschnitt „Weitere Kanäle“, beliebig viele je Typ, jeweils mit **Testen**):
|
||||
|
||||
| Typ | Angaben | Zugangsdaten |
|
||||
|---|---|---|
|
||||
| **ntfy** | Serveradresse (Standard `https://ntfy.sh`), Thema, Priorität 1–5 | Zugriffstoken, optional |
|
||||
| **Gotify** | Serveradresse, Priorität 0–10 | App-Token |
|
||||
| **Telegram** | Chat-ID (Zahl oder `@kanal`), optional eine eigene API-Adresse | Bot-Token (von @BotFather) |
|
||||
| **Microsoft Teams** | – | Webhook-Adresse eines Workflows („Beim Empfang einer Webhook-Anfrage in einem Kanal posten“) oder eines Incoming-Webhooks |
|
||||
| **Matrix** | Homeserver-Adresse, Raum-ID (`!abc:example.org`, nicht der Alias) | Zugriffstoken eines Benutzers, der im Raum ist |
|
||||
|
||||
Die Zugangsdaten liegen wie das SMTP-Passwort in der Geheimnis-Ablage des Servers, nie in der
|
||||
Datenbank, werden nie wieder angezeigt (nur „gesetzt“) und stehen nicht in Fehlermeldungen oder Logs.
|
||||
Weil bei Telegram und Teams die Adresse selbst das Geheimnis ist, wird sie wie ein Token behandelt.
|
||||
Ziele müssen `http(s)` sein, ohne Zugangsdaten in der Adresse; lokale Adressen (`localhost`, Loopback,
|
||||
Link-Local, Metadaten-Dienst) werden abgelehnt, Ziele im eigenen Netz sind erlaubt. Weiterleitungen werden
|
||||
nicht verfolgt. Ein ausgefallener Kanal hält die anderen nicht auf; zugestellt wird einmal je Ereignis.
|
||||
|
||||
Ereignisse (jeweils einzeln schaltbar): Host nicht erreichbar, Zertifikat läuft ab,
|
||||
Regel-Drift im Cluster, NAT-Drift im Cluster, Netzwerk-Abweichung im Cluster,
|
||||
fehlgeschlagener Rollback, Wartungsergebnis unklar, CARP-Zustand kritisch
|
||||
(Split-Brain/kein Master), CARP-Auto-Revert fehlgeschlagen, geplante Wartung nicht
|
||||
ausgeführt. Geprüft wird im eingestellten Intervall (Standard 300 s). Ein Problem
|
||||
ausgeführt, Gateway gestört, Ressourcen-Schwelle überschritten, Dienst ausgefallen
|
||||
(die letzten drei sind unten beschrieben). Geprüft wird im eingestellten Intervall (Standard 300 s). Ein Problem
|
||||
meldet sich einmal und löst sich selbst auf, sobald es behoben ist; die
|
||||
Dashboard-Karte „Warnungen“ zeigt die offenen. Ausnahme „geplante Wartung nicht
|
||||
ausgeführt“: Sie bleibt drei Tage nach dem geplanten Zeitpunkt offen und schließt dann
|
||||
ohne „behoben“-Meldung, weil sich ein verpasster Termin nicht von selbst behebt. Der
|
||||
Text der Benachrichtigungen ist nicht übersetzt.
|
||||
|
||||
#### Gateway-, Ressourcen- und Dienst-Alarme
|
||||
|
||||
Diese drei Ereignisse lesen die **Minutenmessung** (dieselbe wie die Auslastungs-Diagramme), fragen die
|
||||
Firewalls also nicht zusätzlich ab. Damit Einzelspitzen und flackernde Verbindungen nichts melden,
|
||||
öffnet ein Alarm erst, wenn die letzten *N* Messungen alle schlecht sind, und schließt erst, wenn die letzten
|
||||
*N* alle wieder gut sind; dazwischen bleibt der Zustand, wie er ist. *N* ist einstellbar (Standard 3 für
|
||||
Gateways und Dienste, 5 für Ressourcen; die Messung läuft jede Minute). Meldet eine Firewall gerade nicht,
|
||||
beurteilt OPNexus sie nicht (das meldet „Host nicht erreichbar“) und lässt offene Alarme stehen.
|
||||
|
||||
- **Gateway gestört** – **standardmäßig an**, weil ein WAN-Ausfall das ist, was jeder wissen will. Ein Alarm je Gateway, wenn OPNsense es als
|
||||
*down* führt, oder wenn der Paketverlust (Standard ab 20 %) oder die Latenz (Standard ab 500 ms) darüber liegt (0 schaltet die jeweilige Schwelle ab).
|
||||
OPNsense selbst entscheidet über „down“ nach den Grenzen im Gateway (Einstellung „Gateway-Überwachung“). Von Hand abgeschaltete
|
||||
Gateways („Gateway deaktivieren/erzwungen aus“) und Gateways ohne Überwachung (kein Monitor-IP, Werte „~“) melden nichts.
|
||||
Der Status steht auch auf der Host-Seite (Abschnitt „Gateways“, live gelesen), als Kachel auf dem Dashboard und in Prometheus.
|
||||
- **Ressourcen-Schwelle** – standardmäßig **aus**: Auf einer bestehenden Flotte würde das Einschalten sonst gleich alles melden, was schon länger voll
|
||||
ist. Schwellen: Festplatte ab 90 % (je ZFS-Pool bzw. UFS-Gerät ein Alarm), Arbeitsspeicher ab 95 %, Swap ab 50 %, CPU-Last ab 2,0 je Thread
|
||||
(1-Minuten-Last geteilt durch die Zahl der CPU-Threads). Jede Schwelle lässt sich mit 0 abschalten.
|
||||
- **Dienst ausgefallen** – standardmäßig **aus**. OPNsense liefert nur „läuft / läuft nicht“ und „gesperrt“, aber nicht, ob jemand einen Dienst
|
||||
absichtlich gestoppt hat. Deshalb gilt eine **Liste überwachter Dienste** (Standard: unbound, dnsmasq, dhcpd, dhcpd6, kea-dhcp4, kea-dhcp6, ntpd,
|
||||
openvpn, wireguard, strongswan), und gemeldet wird nur ein Dienst, der in den letzten rund zwei Stunden lief und jetzt nicht mehr. Ein Dienst, der
|
||||
nie lief (etwa DHCPv6 ohne Präfix), bleibt still; einen absichtlich gestoppten Dienst nimmst du aus der Liste. Eine Firewall listet nur Dienste, die
|
||||
in ihrer Konfiguration aktiviert sind, ein nicht eingerichteter Dienst taucht also gar nicht auf.
|
||||
|
||||
### Sicherheitslage
|
||||
|
||||
Unter **Monitoring › Sicherheitslage** prüft OPNexus jede Firewall auf typische
|
||||
@@ -1055,6 +1200,10 @@ scrape_configs:
|
||||
| `opnexus_host_enabled`, `…_last_sample_timestamp_seconds` | Überwachung an; Zeit der letzten Messung |
|
||||
| `opnexus_host_memory_used_percent`, `opnexus_host_load1`, `opnexus_host_disk_used_percent` | Letzte Messung |
|
||||
| `opnexus_host_carp_master` | 1 = MASTER, 0 = BACKUP (fehlt, wenn unbekannt) |
|
||||
| `opnexus_host_swap_used_percent` | Swap-Belegung der letzten Messung (fehlt ohne Swap) |
|
||||
| `opnexus_host_gateway_up{gateway}` | 1, solange OPNsense das Gateway nicht als down führt |
|
||||
| `opnexus_host_gateway_loss_percent`, `…_delay_seconds`, `…_stddev_seconds` `{gateway}` | Paketverlust, Laufzeit und Abweichung zum Monitor (fehlen ohne Messung) |
|
||||
| `opnexus_host_service_running{service,instance}` | 1 = läuft, nur für Dienste aus der Überwachungsliste |
|
||||
| `opnexus_host_compliance_score`, `opnexus_host_compliance_findings{severity}` | Sicherheitslage |
|
||||
| `opnexus_pending_actions{kind,status}` | Offene Aktionen wie auf dem Dashboard |
|
||||
| `opnexus_problems_open{type}` | Offene Probleme der Benachrichtigungen |
|
||||
@@ -1349,8 +1498,11 @@ nach einem Neustart erhalten. Alternativ legt ihn OPNexus unter
|
||||
OPNsense, keine Ausführung durch OPNexus.
|
||||
- **Logs:** kein Live-Streaming (Polling im Sekundenbereich), die Freitextsuche ist
|
||||
eine einfache Teilstring-Suche ohne Index.
|
||||
- **Geplante Wartung:** nur einmalige Zeitpunkte, ohne Benachrichtigung bei
|
||||
Fehlschlag.
|
||||
- **Geplante Wartung und Rollouts:** Rollouts kennen keine Zielversion „von außen“
|
||||
(OPNsense installiert immer die neueste angebotene) und keinen automatischen Rückweg zur
|
||||
alten Version. Major-Upgrade-Rollouts sind nur gegen eine nachgestellte Firewall getestet.
|
||||
Die Gesundheitsprüfung sieht die Firewall aus Sicht der Verwaltung; ob der Nutzverkehr
|
||||
durchgeht, prüft sie nicht.
|
||||
- **Dienste:** keine Auswahlliste, alle Dienste sind steuerbar.
|
||||
- **VPN:** Umfang je Verfahren siehe [VPN](#vpn-ipsec-openvpn-wireguard).
|
||||
- **Zertifikate:** kein Push auf mehrere Hosts, keine Verwaltung von DNS-Providern
|
||||
|
||||
+9
-2
@@ -1,9 +1,11 @@
|
||||
FROM node:20-alpine AS deps
|
||||
# deps und builder laufen immer auf der Build-Plattform (nativ schnell, `next build` unter QEMU-Emulation
|
||||
# hängt); das Ergebnis (.next/standalone) ist reines JavaScript und gilt für jede Zielarchitektur.
|
||||
FROM --platform=$BUILDPLATFORM node:20-alpine AS deps
|
||||
WORKDIR /app
|
||||
COPY package.json package-lock.json ./
|
||||
RUN npm ci --no-audit --no-fund
|
||||
|
||||
FROM node:20-alpine AS builder
|
||||
FROM --platform=$BUILDPLATFORM node:20-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
@@ -11,10 +13,15 @@ ENV NEXT_TELEMETRY_DISABLED=1
|
||||
RUN npm run build
|
||||
|
||||
FROM node:20-alpine AS runner
|
||||
ARG TARGETARCH
|
||||
ARG BUILDARCH
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production
|
||||
COPY --from=builder /app/.next/standalone ./
|
||||
COPY --from=builder /app/.next/static ./.next/static
|
||||
# Beim Cross-Build liegen die nativen sharp-Binärdateien der Build-Architektur im Standalone-Ordner; die App nutzt
|
||||
# next/image nicht, sie wären nur totes Gewicht.
|
||||
RUN if [ "$TARGETARCH" != "$BUILDARCH" ]; then rm -rf node_modules/@img node_modules/sharp; fi
|
||||
|
||||
EXPOSE 3000
|
||||
CMD ["node", "server.js"]
|
||||
@@ -18,11 +18,13 @@ import {
|
||||
Tags,
|
||||
Trash2,
|
||||
XCircle,
|
||||
Download,
|
||||
} from "lucide-react";
|
||||
import { WorkflowBadge, workflowStates } from "@/components/WorkflowBadge";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { AliasReview } from "@/components/AliasReview";
|
||||
import { LiveAliasesDrift } from "@/components/LiveAliasesDrift";
|
||||
import { AdoptObjectsModal } from "@/components/AdoptObjectsModal";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
@@ -70,6 +72,7 @@ function AliasesContent() {
|
||||
const typeLabel = (type: string) => (aliasTypes.includes(type as PolicyAliasType) ? t(`aliases.type.${type}`) : type);
|
||||
const typeHint = (type: PolicyAliasType) => t(`aliases.hint.${type}`);
|
||||
const readOnly = useReadOnly();
|
||||
const [showAdopt, setShowAdopt] = useState(false);
|
||||
const toast = useToast();
|
||||
const router = useRouter();
|
||||
const params = useSearchParams();
|
||||
@@ -242,13 +245,21 @@ function AliasesContent() {
|
||||
</Button>
|
||||
)}
|
||||
{activeTab === "managed" && (
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("aliases.new")}
|
||||
</Button>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button variant="outline" onClick={() => setShowAdopt(true)} disabled={readOnly}>
|
||||
<Download size={16} />
|
||||
{t("adopt.button")}
|
||||
</Button>
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("aliases.new")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{showAdopt && <AdoptObjectsModal kind="aliases" hosts={hosts} onClose={() => setShowAdopt(false)} onAdopted={load} />}
|
||||
|
||||
<Tabs
|
||||
className="mb-6"
|
||||
items={[
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
Activity,
|
||||
CheckCircle2,
|
||||
ClipboardList,
|
||||
Globe,
|
||||
Layers,
|
||||
Network,
|
||||
RefreshCw,
|
||||
@@ -21,6 +22,7 @@ import Link from "next/link";
|
||||
import { useRouter } from "next/navigation";
|
||||
import { useEffect, useState } from "react";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { GatewayStateBadge, gatewayNumbers } from "@/components/GatewayStatus";
|
||||
import { labelFor, type Language } from "@/lib/i18n";
|
||||
|
||||
const PENDING_KIND_LABELS: Record<string, string> = {
|
||||
@@ -110,7 +112,7 @@ function certProblemLabel(context: Record<string, unknown>, t: (key: string, par
|
||||
}
|
||||
|
||||
export default function DashboardPage() {
|
||||
const { language, t, errorMessage } = useLanguage();
|
||||
const { language, t, ts, errorMessage } = useLanguage();
|
||||
const router = useRouter();
|
||||
const [summary, setSummary] = useState<DashboardSummary | null>(null);
|
||||
const [problems, setProblems] = useState<NotificationProblem[] | null>(null);
|
||||
@@ -180,7 +182,11 @@ export default function DashboardPage() {
|
||||
};
|
||||
}, [errorMessage, router]);
|
||||
|
||||
const healthProblems = (problems ?? []).filter((p) => ["host_offline", "cert_expiry", "rule_drift"].includes(p.problem_type));
|
||||
const healthProblems = (problems ?? []).filter((p) => ["host_offline", "cert_expiry", "rule_drift", "gateway_down", "resource_threshold", "service_down"].includes(p.problem_type));
|
||||
const gatewayOrder: Record<string, number> = { offline: 0, degraded: 1, unknown: 2, forced_down: 3, online: 4 };
|
||||
const gatewayRows = (summary?.hosts ?? [])
|
||||
.flatMap((host) => (host.gateways ?? []).map((gateway) => ({ host, gateway })))
|
||||
.sort((a, b) => gatewayOrder[a.gateway.state] - gatewayOrder[b.gateway.state]);
|
||||
const securityWarningsLoading = problems === null || firmwareWarnings === null;
|
||||
const securityWarningCount = healthProblems.length + (firmwareWarnings?.length ?? 0);
|
||||
|
||||
@@ -245,6 +251,28 @@ export default function DashboardPage() {
|
||||
)}
|
||||
</SectionCard>
|
||||
|
||||
<SectionCard title={t("dashboard.gateways")} icon={Globe}>
|
||||
{summary === null ? (
|
||||
<div className="space-y-2">{[0, 1].map((i) => <Skeleton key={i} className="h-10" />)}</div>
|
||||
) : gatewayRows.length === 0 ? (
|
||||
<EmptyState icon={Globe} title={t("dashboard.noGateways")} description={t("dashboard.noGatewaysDescription")} />
|
||||
) : (
|
||||
<ul className="divide-y divide-slate-100">
|
||||
{gatewayRows.slice(0, 8).map(({ host, gateway }) => (
|
||||
<li key={`${host.id}-${gateway.name}`} className="flex flex-wrap items-center justify-between gap-2 py-2 text-sm">
|
||||
<Link href={`/hosts/${host.id}`} className="min-w-0 truncate font-medium text-sky-700 hover:underline">
|
||||
{host.name} · {gateway.name}
|
||||
</Link>
|
||||
<span className="flex flex-wrap items-center gap-2">
|
||||
<span className="text-xs text-slate-500">{gatewayNumbers(gateway, t)}</span>
|
||||
<GatewayStateBadge state={gateway.state} />
|
||||
</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</SectionCard>
|
||||
|
||||
<SectionCard title={t("dashboard.securityPosture")} icon={ShieldAlert}>
|
||||
{posture === null ? (
|
||||
<Skeleton className="h-10" />
|
||||
@@ -305,6 +333,14 @@ export default function DashboardPage() {
|
||||
<Badge tone="danger" className="shrink-0">{t("dashboard.hostOffline")}</Badge>
|
||||
</Link>
|
||||
)}
|
||||
{(problem.problem_type === "gateway_down" || problem.problem_type === "resource_threshold" || problem.problem_type === "service_down") && (
|
||||
<Link href={`/hosts/${String(problem.context.host_id)}`} className="flex items-center justify-between gap-3 hover:underline">
|
||||
<span className="min-w-0 truncate font-medium text-slate-900">
|
||||
{String(problem.context.host_name)}: {ts(String(problem.context.gateway ?? problem.context.ressource ?? problem.context.dienst))}
|
||||
</span>
|
||||
<Badge tone="danger" className="shrink-0">{t(`dashboard.alert.${problem.problem_type}`)}</Badge>
|
||||
</Link>
|
||||
)}
|
||||
{problem.problem_type === "cert_expiry" && (
|
||||
<Link href="/certificates" className="flex items-center justify-between gap-3 hover:underline">
|
||||
<span className="min-w-0 truncate font-medium text-slate-900">
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import { HostMaintenance } from "@/components/HostMaintenance";
|
||||
import { HostResourceUsage } from "@/components/HostResourceUsage";
|
||||
import { HostGateways } from "@/components/GatewayStatus";
|
||||
import { HostServices } from "@/components/HostServices";
|
||||
import { RestoreWizard } from "@/components/RestoreWizard";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
@@ -154,6 +155,8 @@ export default function HostDetailPage() {
|
||||
|
||||
<HostResourceUsage host={host} />
|
||||
|
||||
<HostGateways host={host} />
|
||||
|
||||
<HostMaintenance host={host} />
|
||||
|
||||
<h2 className="mb-3 text-lg font-semibold text-slate-900">{t("hostDetail.managedRules")}</h2>
|
||||
|
||||
@@ -0,0 +1,361 @@
|
||||
"use client";
|
||||
|
||||
import { MaintenanceWindowEditor } from "@/components/MaintenanceWindowEditor";
|
||||
import { RolloutPlanEditor } from "@/components/RolloutPlanEditor";
|
||||
import { RolloutStatusBanner } from "@/components/RolloutStatusBanner";
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
import { Input } from "@/components/ui/Field";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
import { Skeleton } from "@/components/ui/Skeleton";
|
||||
import { Tabs } from "@/components/ui/Tabs";
|
||||
import { useToast } from "@/components/ui/Toast";
|
||||
import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, DeviceGroup, MaintenanceWindow, Occurrence, RolloutPlan, RolloutPreview, RolloutRunSummary } from "@/lib/api";
|
||||
import { codeText, RUN_TONE, windowSummary } from "@/lib/rollouts";
|
||||
import { useReadOnly } from "@/lib/useReadOnly";
|
||||
import { AlertTriangle, CalendarClock, ListOrdered, Pencil, Play, Plus, Power, PowerOff, Trash2, Waves } from "lucide-react";
|
||||
import Link from "next/link";
|
||||
import { useRouter } from "next/navigation";
|
||||
import { useEffect, useState } from "react";
|
||||
|
||||
type Tab = "runs" | "plans" | "windows";
|
||||
|
||||
export default function RolloutsPage() {
|
||||
const { t, errorMessage, locale } = useLanguage();
|
||||
const toast = useToast();
|
||||
const router = useRouter();
|
||||
const readOnly = useReadOnly();
|
||||
const [tab, setTab] = useState<Tab>("runs");
|
||||
const [runs, setRuns] = useState<RolloutRunSummary[] | null>(null);
|
||||
const [plans, setPlans] = useState<RolloutPlan[] | null>(null);
|
||||
const [windows, setWindows] = useState<MaintenanceWindow[] | null>(null);
|
||||
const [groups, setGroups] = useState<DeviceGroup[]>([]);
|
||||
const [error, setError] = useState("");
|
||||
const [editPlan, setEditPlan] = useState<RolloutPlan | "new" | null>(null);
|
||||
const [editWindow, setEditWindow] = useState<MaintenanceWindow | "new" | null>(null);
|
||||
const [preview, setPreview] = useState<{ mode: "start" | "activate"; data: RolloutPreview } | null>(null);
|
||||
const [move, setMove] = useState<{ window: MaintenanceWindow; occ: Occurrence; to: string } | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const wanted = new URLSearchParams(window.location.search).get("tab");
|
||||
if (wanted === "plans" || wanted === "windows") setTab(wanted);
|
||||
}, []);
|
||||
|
||||
async function load() {
|
||||
try {
|
||||
const [r, p, w, g] = await Promise.all([api.listRolloutRuns(), api.listRolloutPlans(), api.listMaintenanceWindows(), api.listGroups()]);
|
||||
setRuns(r);
|
||||
setPlans(p);
|
||||
setWindows(w);
|
||||
setGroups(g);
|
||||
setError("");
|
||||
} catch (e) {
|
||||
setError(errorMessage(e, "rollouts.loadFailed"));
|
||||
}
|
||||
}
|
||||
useEffect(() => {
|
||||
load();
|
||||
const timer = setInterval(() => { if (document.visibilityState === "visible") api.listRolloutRuns().then(setRuns).catch(() => {}); }, 15000);
|
||||
return () => clearInterval(timer);
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, []);
|
||||
|
||||
function changeTab(next: string) {
|
||||
setTab(next as Tab);
|
||||
window.history.replaceState(null, "", next === "runs" ? "/rollouts" : `/rollouts?tab=${next}`);
|
||||
}
|
||||
|
||||
async function act(fn: () => Promise<unknown>, okKey: string) {
|
||||
try {
|
||||
await fn();
|
||||
toast.show(t(okKey), "success");
|
||||
await load();
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "rollouts.actionFailed"), "danger");
|
||||
}
|
||||
}
|
||||
|
||||
async function openPreview(plan: RolloutPlan, mode: "start" | "activate") {
|
||||
try {
|
||||
setPreview({ mode, data: await api.previewRolloutPlan(plan.id) });
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "rollouts.actionFailed"), "danger");
|
||||
}
|
||||
}
|
||||
|
||||
const when = (value: string | null | undefined) => (value ? new Date(value).toLocaleString(locale) : "");
|
||||
|
||||
return (
|
||||
<div>
|
||||
<div className="mb-4 flex flex-wrap items-center justify-between gap-2">
|
||||
<h1 className="flex items-center gap-2 text-2xl font-semibold text-slate-900">
|
||||
{t("rollouts.title")}
|
||||
<InfoTip text={t("rollouts.intro")} size={18} />
|
||||
</h1>
|
||||
{!readOnly && tab === "plans" && <Button onClick={() => setEditPlan("new")}><Plus size={16} />{t("rollouts.plan.new")}</Button>}
|
||||
{!readOnly && tab === "windows" && <Button onClick={() => setEditWindow("new")}><Plus size={16} />{t("mwindows.new")}</Button>}
|
||||
</div>
|
||||
<RolloutStatusBanner />
|
||||
<Tabs className="mb-4" active={tab} onChange={changeTab} items={[
|
||||
{ key: "runs", label: t("rollouts.tab.runs") },
|
||||
{ key: "plans", label: t("rollouts.tab.plans") },
|
||||
{ key: "windows", label: t("rollouts.tab.windows") },
|
||||
]} />
|
||||
{error && <p role="alert" className="mb-4 text-sm text-red-600">{error}</p>}
|
||||
|
||||
{tab === "runs" && (runs === null ? <Skeleton className="h-32" /> : runs.length === 0 ? (
|
||||
<EmptyState icon={Waves} title={t("rollouts.noRuns")} description={t("rollouts.noRunsHint")} />
|
||||
) : (
|
||||
<Card className="overflow-x-auto">
|
||||
<table className="w-full text-left text-sm">
|
||||
<thead className="border-b border-slate-200 bg-slate-50 text-xs uppercase tracking-wide text-slate-500">
|
||||
<tr>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.plan")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.status")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.wave")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.target")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.hosts")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.started")}</th>
|
||||
<th className="px-4 py-3 font-medium"><span className="sr-only">{t("common.actions")}</span></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{runs.map((r) => (
|
||||
<tr key={r.id} className="border-b border-slate-100 last:border-0">
|
||||
<td className="px-4 py-3 font-medium text-slate-900">{r.plan_name}<div className="text-xs font-normal text-slate-500">{t(`rollouts.action.${r.action}`)}</div></td>
|
||||
<td className="px-4 py-3"><Badge tone={RUN_TONE[r.status]}>{t(`rollouts.run.status.${r.status}`)}</Badge></td>
|
||||
<td className="px-4 py-3 whitespace-nowrap">{t("rollouts.waveOf", { current: Math.min(r.current_wave + 1, r.wave_count), total: r.wave_count })} {r.current_wave_label && <span className="text-slate-500">({r.current_wave_label})</span>}</td>
|
||||
<td className="px-4 py-3">{r.target_version ?? "–"}</td>
|
||||
<td className="px-4 py-3 whitespace-nowrap text-xs">
|
||||
{Object.entries(r.host_counts).map(([s, n]) => <span key={s} className="mr-2">{t(`rollouts.host.status.${s}`)}: {n}</span>)}
|
||||
</td>
|
||||
<td className="px-4 py-3 text-xs text-slate-600">{when(r.created_at)}<div>{t("rollouts.by", { user: r.started_by })} · {t(`rollouts.trigger.${r.trigger}`)}</div></td>
|
||||
<td className="px-4 py-3 text-right"><Button variant="outline" href={`/rollouts/runs/${r.id}`}>{t("rollouts.details")}</Button></td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</Card>
|
||||
))}
|
||||
|
||||
{tab === "plans" && (plans === null ? <Skeleton className="h-32" /> : plans.length === 0 ? (
|
||||
<EmptyState icon={ListOrdered} title={t("rollouts.plan.none")} description={t("rollouts.plan.noneHint")}
|
||||
action={!readOnly && <Button onClick={() => setEditPlan("new")}><Plus size={16} />{t("rollouts.plan.new")}</Button>} />
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
{plans.map((p) => (
|
||||
<Card key={p.id} className="p-4">
|
||||
<div className="flex flex-wrap items-start justify-between gap-3">
|
||||
<div className="min-w-0">
|
||||
<h2 className="flex flex-wrap items-center gap-2 font-semibold text-slate-900">
|
||||
{p.name}
|
||||
<Badge tone={p.action === "major_update" ? "accent" : "info"}>{t(`rollouts.action.${p.action}`)}</Badge>
|
||||
{p.auto_start && (p.active ? (p.fingerprint_ok
|
||||
? <Badge tone="success">{t("rollouts.plan.active")}</Badge>
|
||||
: <Tooltip focusable content={t("rollouts.plan.changedTip")}><Badge tone="warning">{t("rollouts.plan.changed")}</Badge></Tooltip>)
|
||||
: <Badge tone="neutral">{t("rollouts.plan.inactive")}</Badge>)}
|
||||
{p.errors.length > 0 && <Tooltip focusable content={<span className="block space-y-1">{p.errors.map((e, i) => <span key={i} className="block">{codeText(t, "rollouts.issue", e)}</span>)}</span>}><Badge tone="danger">{t("rollouts.plan.errors", { count: p.errors.length })}</Badge></Tooltip>}
|
||||
{p.warnings.length > 0 && <Tooltip focusable content={<span className="block space-y-1">{p.warnings.map((e, i) => <span key={i} className="block">{codeText(t, "rollouts.issue", e)}</span>)}</span>}><Badge tone="warning">{t("rollouts.plan.warnings", { count: p.warnings.length })}</Badge></Tooltip>}
|
||||
</h2>
|
||||
{p.description && <p className="mt-1 text-sm text-slate-600">{p.description}</p>}
|
||||
<ol className="mt-2 flex flex-wrap items-center gap-1 text-sm">
|
||||
{p.waves.map((w, i) => (
|
||||
<li key={w.id} className="flex items-center gap-1">
|
||||
{i > 0 && <span aria-hidden className="text-slate-400">→</span>}
|
||||
<span className="rounded-lg border border-slate-200 px-2 py-0.5">
|
||||
<strong>{w.label}</strong> <span className="text-slate-500">{w.groups.join(", ")} · {t("rollouts.plan.hostsCount", { count: w.host_count })}</span>
|
||||
</span>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
<p className="mt-2 text-xs text-slate-500">
|
||||
{p.window ? <>{t("rollouts.plan.window")}: {p.window.name}{p.window.next[0] && <> · {t("rollouts.plan.nextOccurrence", { time: when(p.window.next[0].start) })}</>}</> : t("rollouts.plan.noWindow")}
|
||||
{p.active && p.activated_by && <> · {t("rollouts.plan.activatedBy", { user: p.activated_by })}</>}
|
||||
</p>
|
||||
{p.auto_start_error && (
|
||||
<p className="mt-1 flex items-center gap-1 text-xs text-flame-700"><AlertTriangle size={13} />
|
||||
{t("rollouts.plan.lastAutoStartError", { time: when(p.auto_start_error.occurrence), reason: codeText(t, "rollouts.issue", { code: p.auto_start_error.code, params: p.auto_start_error.params ?? {} }) })}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
{!readOnly && (
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button onClick={() => openPreview(p, "start")}><Play size={14} />{t("rollouts.plan.previewStart")}</Button>
|
||||
{p.auto_start && (p.active
|
||||
? <Button variant="outline" onClick={() => act(() => api.deactivateRolloutPlan(p.id), "rollouts.plan.deactivated")}><PowerOff size={14} />{t("rollouts.plan.deactivate")}</Button>
|
||||
: <Button variant="outline" onClick={() => openPreview(p, "activate")}><Power size={14} />{t("rollouts.plan.activate")}</Button>)}
|
||||
{p.auto_start && p.active && !p.fingerprint_ok && <Button variant="outline" onClick={() => openPreview(p, "activate")}><Power size={14} />{t("rollouts.plan.activate")}</Button>}
|
||||
<Button variant="outline" aria-label={t("rollouts.plan.edit")} onClick={() => setEditPlan(p)}><Pencil size={14} /></Button>
|
||||
<Button variant="dangerSubtle" aria-label={t("common.delete")} onClick={() => confirm(t("rollouts.plan.deleteConfirm", { name: p.name })) && act(() => api.deleteRolloutPlan(p.id), "rollouts.plan.deleted")}><Trash2 size={14} /></Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</Card>
|
||||
))}
|
||||
</div>
|
||||
))}
|
||||
|
||||
{tab === "windows" && (windows === null ? <Skeleton className="h-32" /> : windows.length === 0 ? (
|
||||
<EmptyState icon={CalendarClock} title={t("mwindows.none")} description={t("mwindows.noneHint")}
|
||||
action={!readOnly && <Button onClick={() => setEditWindow("new")}><Plus size={16} />{t("mwindows.new")}</Button>} />
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
{windows.map((w) => (
|
||||
<Card key={w.id} className="p-4">
|
||||
<div className="flex flex-wrap items-start justify-between gap-3">
|
||||
<div className="min-w-0">
|
||||
<h2 className="flex flex-wrap items-center gap-2 font-semibold text-slate-900">
|
||||
{w.name}
|
||||
{w.open_now && <Badge tone="success">{t("mwindows.openNow")}</Badge>}
|
||||
{!w.enabled && <Badge tone="neutral">{t("mwindows.off")}</Badge>}
|
||||
</h2>
|
||||
<p className="mt-1 text-sm text-slate-600">{windowSummary(t, w)} · {t("mwindows.cutoffSummary", { minutes: w.start_cutoff_minutes })}</p>
|
||||
{w.used_by.length > 0 && <p className="mt-1 text-xs text-slate-500">{t("mwindows.usedBy", { plans: w.used_by.map((p) => p.name).join(", ") })}</p>}
|
||||
</div>
|
||||
{!readOnly && (
|
||||
<div className="flex gap-2">
|
||||
<Button variant="outline" aria-label={t("mwindows.edit")} onClick={() => setEditWindow(w)}><Pencil size={14} /></Button>
|
||||
<Button variant="dangerSubtle" aria-label={t("common.delete")} onClick={() => confirm(t("mwindows.deleteConfirm", { name: w.name })) && act(() => api.deleteMaintenanceWindow(w.id), "mwindows.deleted")}><Trash2 size={14} /></Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
{w.next.length > 0 && (
|
||||
<div className="mt-3">
|
||||
<h3 className="mb-1 text-xs font-semibold uppercase tracking-wide text-slate-500">{t("mwindows.next")}</h3>
|
||||
<ul className="space-y-1 text-sm">
|
||||
{w.next.map((o) => {
|
||||
const exc = w.exceptions.find((e) => e.occurrence_date === o.local_date);
|
||||
return (
|
||||
<li key={`${o.local_date}-${o.start}`} className="flex flex-wrap items-center gap-2">
|
||||
<span className={o.skipped ? "text-slate-400 line-through" : ""}>{when(o.start)} – {new Date(o.end).toLocaleTimeString(locale, { hour: "2-digit", minute: "2-digit" })}</span>
|
||||
{o.skipped && <Badge tone="neutral">{t("mwindows.skipped")}</Badge>}
|
||||
{o.moved && <Badge tone="info">{t("mwindows.moved")}</Badge>}
|
||||
{exc?.reason && <span className="text-xs text-slate-500">{exc.reason}</span>}
|
||||
{!readOnly && !exc && (
|
||||
<span className="flex gap-1">
|
||||
<Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => act(() => api.addWindowException(w.id, { occurrence_date: o.local_date, kind: "skip" }), "mwindows.exceptionSaved")}>{t("mwindows.skip")}</Button>
|
||||
<Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => setMove({ window: w, occ: o, to: "" })}>{t("mwindows.move")}</Button>
|
||||
</span>
|
||||
)}
|
||||
{!readOnly && exc && <Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => act(() => api.deleteWindowException(w.id, exc.id), "mwindows.exceptionSaved")}>{t("mwindows.removeException")}</Button>}
|
||||
</li>
|
||||
);
|
||||
})}
|
||||
</ul>
|
||||
</div>
|
||||
)}
|
||||
</Card>
|
||||
))}
|
||||
</div>
|
||||
))}
|
||||
|
||||
{editPlan && (
|
||||
<RolloutPlanEditor plan={editPlan === "new" ? null : editPlan} groups={groups} windows={windows ?? []} onClose={() => setEditPlan(null)}
|
||||
onSaved={async () => { setEditPlan(null); toast.show(t("rollouts.plan.saved"), "success"); await load(); }} />
|
||||
)}
|
||||
{editWindow && (
|
||||
<MaintenanceWindowEditor window={editWindow === "new" ? null : editWindow} onClose={() => setEditWindow(null)}
|
||||
onSaved={async () => { setEditWindow(null); toast.show(t("mwindows.saved"), "success"); await load(); }} />
|
||||
)}
|
||||
{move && (
|
||||
<Modal title={`${t("mwindows.move")}: ${when(move.occ.start)}`} onClose={() => setMove(null)}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<label className="block"><span className="mb-1 block font-medium text-slate-700">{t("mwindows.moveTo")}</span>
|
||||
<Input type="datetime-local" value={move.to} onChange={(e) => setMove({ ...move, to: e.target.value })} /></label>
|
||||
<div className="flex justify-end gap-2">
|
||||
<Button variant="ghost" onClick={() => setMove(null)}>{t("common.cancel")}</Button>
|
||||
<Button disabled={!move.to} onClick={async () => {
|
||||
const m = move;
|
||||
setMove(null);
|
||||
await act(() => api.addWindowException(m.window.id, { occurrence_date: m.occ.local_date, kind: "move", moved_start: new Date(m.to).toISOString() }), "mwindows.exceptionSaved");
|
||||
}}>{t("rollouts.plan.save")}</Button>
|
||||
</div>
|
||||
</div>
|
||||
</Modal>
|
||||
)}
|
||||
{preview && <PreviewModal preview={preview} onClose={() => setPreview(null)} onDone={async (runId) => {
|
||||
setPreview(null);
|
||||
if (runId) router.push(`/rollouts/runs/${runId}`);
|
||||
else await load();
|
||||
}} />}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function PreviewModal({ preview, onClose, onDone }: { preview: { mode: "start" | "activate"; data: RolloutPreview }; onClose: () => void; onDone: (runId?: string) => void }) {
|
||||
const { t, errorMessage, locale } = useLanguage();
|
||||
const toast = useToast();
|
||||
const { data, mode } = preview;
|
||||
const [name, setName] = useState("");
|
||||
const [ack, setAck] = useState(false);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const plan = data.plan;
|
||||
|
||||
async function go() {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
if (mode === "start") {
|
||||
const run = await api.startRolloutPlan(plan.id, data.preview_token, name);
|
||||
toast.show(t("rollouts.plan.started"), "success");
|
||||
onDone(run.id);
|
||||
} else {
|
||||
await api.activateRolloutPlan(plan.id, data.preview_token);
|
||||
toast.show(t("rollouts.plan.activated"), "success");
|
||||
onDone();
|
||||
}
|
||||
} catch (e) {
|
||||
setError(errorMessage(e, "rollouts.actionFailed"));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal title={t("rollouts.plan.previewTitle", { name: plan.name })} onClose={() => !busy && onClose()}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<p><Badge tone={plan.action === "major_update" ? "accent" : "info"}>{t(`rollouts.action.${plan.action}`)}</Badge>
|
||||
{data.window && <span className="ml-2 text-slate-600">{t("rollouts.plan.window")}: {data.window.name}{data.window.next[0] && <> · {t("rollouts.plan.nextOccurrence", { time: new Date(data.window.next[0].start).toLocaleString(locale) })}</>}</span>}</p>
|
||||
<ol className="space-y-2">
|
||||
{data.waves.map((w) => (
|
||||
<li key={w.index} className="rounded-lg border border-slate-200 p-3">
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<strong>{w.index + 1}. {w.label}</strong>
|
||||
<span className="text-xs text-slate-500">{t("rollouts.detail.soak", { minutes: w.soak_minutes })}</span>
|
||||
{w.carp_handover && <Badge tone="info">{t("rollouts.detail.handover")}</Badge>}
|
||||
{w.require_approval && w.index > 0 && <Badge tone="accent">{t("rollouts.detail.approval")}</Badge>}
|
||||
</div>
|
||||
<p className="mt-1 text-slate-600">
|
||||
{w.hosts.length === 0 ? t("rollouts.plan.emptyWave") : w.hosts.map((h) => `${h.name}${h.cluster_name ? ` (${h.cluster_name})` : ""}${h.enabled ? "" : ` – ${t("rollouts.plan.disabledHost")}`}`).join(", ")}
|
||||
</p>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
{data.errors.length > 0 && <ul role="alert" className="space-y-1 rounded-lg bg-red-50 p-3 text-red-700">{data.errors.map((e, i) => <li key={i}>{codeText(t, "rollouts.issue", e)}</li>)}</ul>}
|
||||
{data.warnings.length > 0 && <ul className="space-y-1 rounded-lg bg-flame-50 p-3 text-flame-800">{data.warnings.map((e, i) => <li key={i} className="flex gap-1"><AlertTriangle size={14} className="mt-0.5 shrink-0" />{codeText(t, "rollouts.issue", e)}</li>)}</ul>}
|
||||
{data.can_start ? (
|
||||
<>
|
||||
{mode === "start" && (
|
||||
<label className="block"><span className="mb-1 block">{t("rollouts.actions.confirmPlan")}</span>
|
||||
<Input autoComplete="off" spellCheck={false} aria-label={t("rollouts.actions.confirmPlan")} value={name} placeholder={plan.name} onChange={(e) => setName(e.target.value)} /></label>
|
||||
)}
|
||||
<label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={ack} onChange={(e) => setAck(e.target.checked)} />{t("rollouts.plan.unattended")}</label>
|
||||
{error && <p role="alert" className="text-red-600">{error}</p>}
|
||||
<div className="flex justify-end gap-2">
|
||||
<Button variant="ghost" onClick={onClose} disabled={busy}>{t("common.cancel")}</Button>
|
||||
<Button variant={mode === "start" ? "danger" : "primary"} loading={busy} disabled={!ack || (mode === "start" && name !== plan.name)} onClick={go}>
|
||||
{mode === "start" ? t("rollouts.plan.start") : t("rollouts.plan.activate")}
|
||||
</Button>
|
||||
</div>
|
||||
</>
|
||||
) : <p className="text-red-700">{t("rollouts.plan.cannotStart")}</p>}
|
||||
{data.can_start || <div className="flex justify-end"><Link href="#" onClick={(e) => { e.preventDefault(); onClose(); }} className="text-brand-600">{t("common.close")}</Link></div>}
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,262 @@
|
||||
"use client";
|
||||
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
import { Input } from "@/components/ui/Field";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
import { Skeleton } from "@/components/ui/Skeleton";
|
||||
import { useToast } from "@/components/ui/Toast";
|
||||
import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, RolloutGate, RolloutRunDetail } from "@/lib/api";
|
||||
import { CHECKS, codeText, HOST_TONE, isActiveRun, pauseDetailText, pauseText, RESULT_TONE, RUN_TONE, WAVE_TONE } from "@/lib/rollouts";
|
||||
import { useReadOnly } from "@/lib/useReadOnly";
|
||||
import { AlertTriangle, ArrowLeft, CheckCheck, Pause, Play, SkipForward, Square, Waves } from "lucide-react";
|
||||
import Link from "next/link";
|
||||
import { useParams } from "next/navigation";
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
|
||||
type HostRow = RolloutRunDetail["hosts"][number];
|
||||
|
||||
function GateMatrix({ gate, run, title }: { gate: RolloutGate; run: RolloutRunDetail; title: string }) {
|
||||
const { t } = useLanguage();
|
||||
const names = Object.fromEntries(run.hosts.filter((h) => h.host_id).map((h) => [h.host_id as string, h.host_name]));
|
||||
const checks = CHECKS.filter((c) => Object.values(gate.hosts).some((row) => row[c]));
|
||||
return (
|
||||
<div className="overflow-x-auto">
|
||||
<div className="mb-2 flex items-center gap-2 text-sm font-medium text-slate-700">
|
||||
{title}
|
||||
<Badge tone={gate.verdict === "ok" ? "success" : gate.verdict === "warn" ? "warning" : "danger"}>{t(`rollouts.gate.verdict.${gate.verdict}`)}</Badge>
|
||||
</div>
|
||||
<table className="w-full text-left text-sm">
|
||||
<thead className="border-b border-slate-200 text-xs uppercase tracking-wide text-slate-500">
|
||||
<tr>
|
||||
<th className="px-3 py-2 font-medium">{t("rollouts.col.host")}</th>
|
||||
{checks.map((c) => <th key={c} className="px-3 py-2 font-medium">{t(`rollouts.gate.check.${c}`)}</th>)}
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{Object.entries(gate.hosts).map(([hid, row]) => (
|
||||
<tr key={hid} className="border-b border-slate-100 last:border-0">
|
||||
<td className="px-3 py-2 font-medium">{names[hid] ?? hid.slice(0, 8)}</td>
|
||||
{checks.map((c) => {
|
||||
const r = row[c];
|
||||
return (
|
||||
<td key={c} className="px-3 py-2">
|
||||
{r ? (
|
||||
<Tooltip focusable content={codeText(t, "rollouts.gate.code", r)}>
|
||||
<Badge tone={RESULT_TONE[r.result]}>{t(`rollouts.gate.result.${r.result}`)}</Badge>
|
||||
</Tooltip>
|
||||
) : <span className="text-slate-400">–</span>}
|
||||
</td>
|
||||
);
|
||||
})}
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function RolloutRunPage() {
|
||||
const { t, ts, errorMessage, locale } = useLanguage();
|
||||
const { id } = useParams<{ id: string }>();
|
||||
const toast = useToast();
|
||||
const readOnly = useReadOnly();
|
||||
const [run, setRun] = useState<RolloutRunDetail | null>(null);
|
||||
const [error, setError] = useState("");
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [dialog, setDialog] = useState<{ kind: "pause" } | { kind: "abort" } | { kind: "skip"; host: HostRow } | null>(null);
|
||||
const [reason, setReason] = useState("");
|
||||
const [confirmName, setConfirmName] = useState("");
|
||||
|
||||
const load = useCallback(async () => {
|
||||
try {
|
||||
setRun(await api.getRolloutRun(id));
|
||||
setError("");
|
||||
} catch (e) {
|
||||
setError(errorMessage(e, "rollouts.loadFailed"));
|
||||
}
|
||||
}, [id, errorMessage]);
|
||||
|
||||
useEffect(() => {
|
||||
load();
|
||||
}, [load]);
|
||||
useEffect(() => {
|
||||
if (!run || !isActiveRun(run.status)) return;
|
||||
const timer = setInterval(load, 15000);
|
||||
return () => clearInterval(timer);
|
||||
}, [run?.status, load]); // eslint-disable-line react-hooks/exhaustive-deps
|
||||
|
||||
async function act(fn: () => Promise<RolloutRunDetail>, okKey: string) {
|
||||
setBusy(true);
|
||||
try {
|
||||
setRun(await fn());
|
||||
toast.show(t(okKey), "success");
|
||||
setDialog(null);
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "rollouts.actionFailed"), "danger");
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
const when = (value: string | null | undefined) => (value ? new Date(value).toLocaleString(locale) : "");
|
||||
const time = (value: string | null | undefined) => (value ? new Date(value).toLocaleTimeString(locale, { hour: "2-digit", minute: "2-digit" }) : "");
|
||||
|
||||
if (error && !run) {
|
||||
return <EmptyState icon={Waves} title={t("rollouts.detail.notFound")} description={error} action={<Button href="/rollouts" variant="outline">{t("rollouts.detail.back")}</Button>} />;
|
||||
}
|
||||
if (!run) return <Skeleton className="h-64" />;
|
||||
|
||||
const active = isActiveRun(run.status);
|
||||
const current = run.waves.find((w) => w.index === run.current_wave);
|
||||
const recheck = run.progress.gate_recheck_at || run.progress.cluster_recheck_at || run.progress.resume_recheck_at;
|
||||
return (
|
||||
<div>
|
||||
<Link href="/rollouts" className="mb-3 inline-flex items-center gap-1 text-sm text-slate-600 hover:text-slate-900"><ArrowLeft size={14} />{t("rollouts.detail.back")}</Link>
|
||||
<div className="mb-4 flex flex-wrap items-center justify-between gap-3">
|
||||
<h1 className="flex flex-wrap items-center gap-2 text-2xl font-semibold text-slate-900">
|
||||
{run.plan_name}
|
||||
<Badge tone={RUN_TONE[run.status]}>{t(`rollouts.run.status.${run.status}`)}</Badge>
|
||||
</h1>
|
||||
{!readOnly && active && (
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{run.status !== "paused" && <Button variant="outline" disabled={busy} onClick={() => { setReason(""); setDialog({ kind: "pause" }); }}><Pause size={14} />{t("rollouts.actions.pause")}</Button>}
|
||||
{run.status === "paused" && <Button disabled={busy} onClick={() => act(() => api.resumeRolloutRun(run.id), "rollouts.actions.resumed")}><Play size={14} />{t("rollouts.actions.resume")}</Button>}
|
||||
{run.status === "awaiting_approval" && <Button disabled={busy} onClick={() => act(() => api.approveRolloutWave(run.id), "rollouts.actions.approved")}><CheckCheck size={14} />{t("rollouts.actions.approve")}</Button>}
|
||||
<Button variant="dangerSubtle" disabled={busy} onClick={() => { setReason(""); setConfirmName(""); setDialog({ kind: "abort" }); }}><Square size={14} />{t("rollouts.actions.abort")}</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="mb-4 grid gap-2 text-sm text-slate-700 sm:grid-cols-2 lg:grid-cols-4">
|
||||
<p>{t("rollouts.col.target")}: <strong>{run.target_version ?? "–"}</strong></p>
|
||||
<p>{t(`rollouts.action.${run.action}`)} · {t(`rollouts.trigger.${run.trigger}`)}</p>
|
||||
<p>{t("rollouts.col.started")}: {when(run.created_at)}</p>
|
||||
<p>{t("rollouts.detail.owner")}: {run.started_by}</p>
|
||||
</div>
|
||||
|
||||
{run.status === "paused" && (
|
||||
<div role="alert" className="mb-4 rounded-xl border border-flame-200 bg-flame-50 p-4 text-sm text-flame-800">
|
||||
<p className="flex items-center gap-2 font-semibold"><AlertTriangle size={16} />{pauseText(t, run.pause_reason)}
|
||||
<InfoTip text={t("rollouts.pauseHint")} size={14} /></p>
|
||||
{run.pause_detail && <p className="mt-1 break-words">{pauseDetailText(t, run.pause_detail)}</p>}
|
||||
</div>
|
||||
)}
|
||||
{run.status === "cancelled" && run.pause_detail && Array.isArray(run.pause_detail.still_running) && run.pause_detail.still_running.length > 0 && (
|
||||
<p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.stillRunning", { hosts: (run.pause_detail.still_running as string[]).join(", ") })}</p>
|
||||
)}
|
||||
{run.status === "waiting_window" && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.waitingWindow")}</p>}
|
||||
{run.status === "awaiting_approval" && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.awaiting")}</p>}
|
||||
{(run.status === "soaking" && run.soak_until) && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.soakUntil", { time: time(run.soak_until) })}</p>}
|
||||
{["gate", "wave_running", "resuming"].includes(run.status) && recheck && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.recheckAt", { time: time(recheck) })}</p>}
|
||||
{run.status === "wave_running" && run.progress.blocked_since && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.blockedSince", { time: time(run.progress.blocked_since) })}</p>}
|
||||
|
||||
<h2 className="mb-2 text-sm font-semibold uppercase tracking-wide text-slate-500">{t("rollouts.detail.waves")}</h2>
|
||||
<ol className="mb-6 flex flex-wrap gap-2">
|
||||
{run.waves.map((w) => (
|
||||
<li key={w.index} aria-current={w.index === run.current_wave && active ? "step" : undefined}
|
||||
className={`min-w-[9rem] flex-1 rounded-xl border p-3 text-sm ${w.index === run.current_wave && active ? "border-sky-300" : "border-slate-200"}`}>
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<strong>{w.index + 1}. {w.label}</strong>
|
||||
<Badge tone={WAVE_TONE[w.status] ?? "neutral"}>{t(`rollouts.wave.status.${w.status}`)}</Badge>
|
||||
</div>
|
||||
{w.spec && <p className="mt-1 text-xs text-slate-500">{w.spec.groups.join(", ")} · {t("rollouts.detail.soak", { minutes: w.spec.soak_minutes })}
|
||||
{w.spec.carp_handover && <> · {t("rollouts.detail.handover")}</>}{w.spec.require_approval && w.index > 0 && <> · {t("rollouts.detail.approval")}</>}</p>}
|
||||
{w.approved_by && <p className="mt-1 text-xs text-slate-500">{t("rollouts.detail.approvedBy", { user: w.approved_by })}</p>}
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
|
||||
<h2 className="mb-2 text-sm font-semibold uppercase tracking-wide text-slate-500">{t("rollouts.detail.hosts")}</h2>
|
||||
<Card className="mb-6 overflow-x-auto">
|
||||
<table className="w-full text-left text-sm">
|
||||
<thead className="border-b border-slate-200 bg-slate-50 text-xs uppercase tracking-wide text-slate-500">
|
||||
<tr>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.host")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.wave")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.status")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.version")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.step")}</th>
|
||||
<th className="px-4 py-3 font-medium">{t("rollouts.col.note")}</th>
|
||||
<th className="px-4 py-3 font-medium"><span className="sr-only">{t("common.actions")}</span></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{run.hosts.map((h) => (
|
||||
<tr key={h.id} className="border-b border-slate-100 last:border-0">
|
||||
<td className="px-4 py-3 font-medium">{h.host_id ? <Link className="text-brand-600 hover:underline" href={`/hosts/${h.host_id}`}>{h.host_name}</Link> : h.host_name}
|
||||
{h.cluster_name && <div className="text-xs font-normal text-slate-500">{h.cluster_name}</div>}</td>
|
||||
<td className="px-4 py-3">{run.waves.find((w) => w.index === h.wave_index)?.label}</td>
|
||||
<td className="px-4 py-3"><Badge tone={HOST_TONE[h.status]}>{t(`rollouts.host.status.${h.status}`)}</Badge></td>
|
||||
<td className="px-4 py-3 whitespace-nowrap">{h.version_before ?? "–"}{h.version_after && <> → <strong>{h.version_after}</strong></>}</td>
|
||||
<td className="px-4 py-3 text-xs">{h.schedule?.phase ? t(`maint.phase.${h.schedule.phase}`) : h.schedule ? t(`maint.scheduleStatus.${h.schedule.status === "succeeded" ? "completed" : h.schedule.status}`) : ""}
|
||||
{h.schedule?.carp_handover && <div><Badge tone="info">{t("maint.handoverBadge")}</Badge></div>}</td>
|
||||
<td className="max-w-xs px-4 py-3 text-xs text-slate-600 break-words">{h.detail ? ts(codeText(t, "rollouts.hostNote", h.detail)) : ""}</td>
|
||||
<td className="px-4 py-3 text-right">
|
||||
{!readOnly && active && h.host_id && ["queued", "blocked", "failed"].includes(h.status) && (
|
||||
<Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => { setReason(""); setDialog({ kind: "skip", host: h }); }}><SkipForward size={13} />{t("rollouts.actions.skip")}</Button>
|
||||
)}
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</Card>
|
||||
|
||||
<h2 className="mb-2 flex items-center gap-1 text-sm font-semibold uppercase tracking-wide text-slate-500">{t("rollouts.gate.title")}<InfoTip text={t("rollouts.gate.hint")} size={13} /></h2>
|
||||
<div className="space-y-4">
|
||||
{run.progress.resume_gate && Object.keys(run.progress.resume_gate.hosts).length > 0 && (
|
||||
<Card className="p-4"><GateMatrix gate={run.progress.resume_gate} run={run} title={t("rollouts.gate.resume")} /></Card>
|
||||
)}
|
||||
{active && run.progress.cluster_gate && (
|
||||
<Card className="p-4"><GateMatrix gate={run.progress.cluster_gate} run={run} title={t("rollouts.gate.cluster")} /></Card>
|
||||
)}
|
||||
{[...run.waves].reverse().filter((w) => w.gate_result).map((w) => (
|
||||
<Card key={w.index} className="p-4">
|
||||
<GateMatrix gate={w.gate_result as RolloutGate} run={run} title={`${w.index + 1}. ${w.label} · ${t("rollouts.detail.attempt", { n: w.gate_attempts })}`} />
|
||||
</Card>
|
||||
))}
|
||||
{!run.waves.some((w) => w.gate_result) && !run.progress.resume_gate && <p className="text-sm text-slate-500">{t("rollouts.detail.noGate")}{current ? ` (${current.label})` : ""}</p>}
|
||||
</div>
|
||||
|
||||
{dialog?.kind === "pause" && (
|
||||
<Modal title={t("rollouts.actions.pause")} onClose={() => !busy && setDialog(null)}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<label className="block"><span className="mb-1 block">{t("rollouts.actions.pauseReason")}</span>
|
||||
<Input value={reason} maxLength={500} onChange={(e) => setReason(e.target.value)} /></label>
|
||||
<div className="flex justify-end"><Button loading={busy} onClick={() => act(() => api.pauseRolloutRun(run.id, reason), "rollouts.actions.paused")}><Pause size={14} />{t("rollouts.actions.pause")}</Button></div>
|
||||
</div>
|
||||
</Modal>
|
||||
)}
|
||||
{dialog?.kind === "skip" && (
|
||||
<Modal title={t("rollouts.actions.skipTitle", { host: dialog.host.host_name })} onClose={() => !busy && setDialog(null)}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<p>{t("rollouts.actions.skipHint")}</p>
|
||||
<label className="block"><span className="mb-1 block">{t("rollouts.actions.reason")}</span>
|
||||
<Input value={reason} maxLength={500} onChange={(e) => setReason(e.target.value)} /></label>
|
||||
<div className="flex justify-end"><Button loading={busy} disabled={reason.trim().length < 3}
|
||||
onClick={() => act(() => api.skipRolloutHost(run.id, dialog.host.host_id as string, reason), "rollouts.actions.skipped")}>{t("rollouts.actions.skip")}</Button></div>
|
||||
</div>
|
||||
</Modal>
|
||||
)}
|
||||
{dialog?.kind === "abort" && (
|
||||
<Modal title={t("rollouts.actions.abortTitle")} onClose={() => !busy && setDialog(null)}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<p className="flex items-start gap-2 text-flame-800"><AlertTriangle size={16} className="mt-0.5 shrink-0" />{t("rollouts.actions.abortHint")}</p>
|
||||
<label className="block"><span className="mb-1 block">{t("rollouts.actions.reason")}</span>
|
||||
<Input value={reason} maxLength={500} onChange={(e) => setReason(e.target.value)} /></label>
|
||||
<label className="block"><span className="mb-1 block">{t("rollouts.actions.confirmPlan")}</span>
|
||||
<Input autoComplete="off" spellCheck={false} value={confirmName} placeholder={run.plan_name} onChange={(e) => setConfirmName(e.target.value)} /></label>
|
||||
<div className="flex justify-end"><Button variant="danger" loading={busy} disabled={reason.trim().length < 3 || confirmName !== run.plan_name}
|
||||
onClick={() => act(() => api.abortRolloutRun(run.id, reason, confirmName), "rollouts.actions.aborted")}>{t("rollouts.actions.abort")}</Button></div>
|
||||
</div>
|
||||
</Modal>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
"use client";
|
||||
import { TwoFactorSection } from "@/components/TwoFactorSection";
|
||||
import { NotificationChannelsSection } from "@/components/NotificationChannelsSection";
|
||||
import { OpnexusUpdateSection } from "@/components/OpnexusUpdateSection";
|
||||
import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
|
||||
import { useEffect, useState } from "react";
|
||||
@@ -7,7 +8,7 @@ import { api, AppSettings, LogSettings, NotificationSettings, SmtpSecurity, Them
|
||||
import { useTheme } from "@/components/ThemeProvider";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Input, Select } from "@/components/ui/Field";
|
||||
import { FieldGroup, Input, Select } from "@/components/ui/Field";
|
||||
import { useNotAdmin, useReadOnly } from "@/lib/useReadOnly";
|
||||
import { OidcSection } from "@/components/OidcSection";
|
||||
import { BackupSettingsSection } from "@/components/BackupSettingsSection";
|
||||
@@ -66,6 +67,19 @@ function NotificationsSection() {
|
||||
const [scheduledFailed, setScheduledFailed] = useState(true);
|
||||
const [backupFailed, setBackupFailed] = useState(true);
|
||||
const [complianceAlerts, setComplianceAlerts] = useState(false);
|
||||
const [gatewayDown, setGatewayDown] = useState(true);
|
||||
const [resourceThreshold, setResourceThreshold] = useState(false);
|
||||
const [serviceDown, setServiceDown] = useState(false);
|
||||
const [rolloutProblems, setRolloutProblems] = useState(true);
|
||||
const [gatewayLoss, setGatewayLoss] = useState(20);
|
||||
const [gatewayDelay, setGatewayDelay] = useState(500);
|
||||
const [diskLimit, setDiskLimit] = useState(90);
|
||||
const [memoryLimit, setMemoryLimit] = useState(95);
|
||||
const [swapLimit, setSwapLimit] = useState(50);
|
||||
const [loadLimit, setLoadLimit] = useState(2);
|
||||
const [confirmSamples, setConfirmSamples] = useState(3);
|
||||
const [resourceSamples, setResourceSamples] = useState(5);
|
||||
const [watchlist, setWatchlist] = useState("");
|
||||
const [pollInterval, setPollInterval] = useState(300);
|
||||
|
||||
function applyFromServer(data: NotificationSettings) {
|
||||
@@ -91,6 +105,19 @@ function NotificationsSection() {
|
||||
setScheduledFailed(data.scheduled_maintenance_failed_enabled);
|
||||
setBackupFailed(data.backup_failed_enabled);
|
||||
setComplianceAlerts(data.compliance_alerts_enabled);
|
||||
setGatewayDown(data.gateway_down_enabled);
|
||||
setResourceThreshold(data.resource_threshold_enabled);
|
||||
setServiceDown(data.service_down_enabled);
|
||||
setRolloutProblems(data.rollout_problems_enabled ?? true);
|
||||
setGatewayLoss(data.gateway_loss_threshold_pct);
|
||||
setGatewayDelay(data.gateway_delay_threshold_ms);
|
||||
setDiskLimit(data.disk_threshold_pct);
|
||||
setMemoryLimit(data.memory_threshold_pct);
|
||||
setSwapLimit(data.swap_threshold_pct);
|
||||
setLoadLimit(data.load_per_cpu_threshold);
|
||||
setConfirmSamples(data.alert_confirm_samples);
|
||||
setResourceSamples(data.resource_confirm_samples);
|
||||
setWatchlist(data.service_watchlist.join(", "));
|
||||
setPollInterval(data.poll_interval_seconds);
|
||||
setWebhookSecret("");
|
||||
setClearWebhookSecret(false);
|
||||
@@ -134,6 +161,19 @@ function NotificationsSection() {
|
||||
scheduled_maintenance_failed_enabled: scheduledFailed,
|
||||
backup_failed_enabled: backupFailed,
|
||||
compliance_alerts_enabled: complianceAlerts,
|
||||
gateway_down_enabled: gatewayDown,
|
||||
resource_threshold_enabled: resourceThreshold,
|
||||
service_down_enabled: serviceDown,
|
||||
rollout_problems_enabled: rolloutProblems,
|
||||
gateway_loss_threshold_pct: gatewayLoss,
|
||||
gateway_delay_threshold_ms: gatewayDelay,
|
||||
disk_threshold_pct: diskLimit,
|
||||
memory_threshold_pct: memoryLimit,
|
||||
swap_threshold_pct: swapLimit,
|
||||
load_per_cpu_threshold: loadLimit,
|
||||
alert_confirm_samples: confirmSamples,
|
||||
resource_confirm_samples: resourceSamples,
|
||||
service_watchlist: watchlist.split(/[\s,]+/).filter(Boolean),
|
||||
poll_interval_seconds: pollInterval,
|
||||
});
|
||||
applyFromServer(data);
|
||||
@@ -268,6 +308,10 @@ function NotificationsSection() {
|
||||
[t("settings.notify.event.scheduledFailed"), scheduledFailed, setScheduledFailed],
|
||||
[t("settings.notify.event.backupFailed"), backupFailed, setBackupFailed],
|
||||
[t("settings.notify.event.complianceAlerts"), complianceAlerts, setComplianceAlerts],
|
||||
[t("settings.notify.event.gatewayDown"), gatewayDown, setGatewayDown],
|
||||
[t("settings.notify.event.resourceThreshold"), resourceThreshold, setResourceThreshold],
|
||||
[t("settings.notify.event.serviceDown"), serviceDown, setServiceDown],
|
||||
[t("settings.notify.event.rolloutProblems"), rolloutProblems, setRolloutProblems],
|
||||
].map(([label, value, setter]) => (
|
||||
<label key={label as string} className="flex items-center gap-2 text-sm">
|
||||
<input type="checkbox" checked={value as boolean} onChange={(e) => (setter as (v: boolean) => void)(e.target.checked)} />
|
||||
@@ -281,9 +325,34 @@ function NotificationsSection() {
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="space-y-3 rounded-lg border border-slate-100 p-4">
|
||||
<p className="flex items-center gap-2 text-sm font-medium">{t("settings.notify.monitoring")}<InfoTip text={t("settings.notify.monitoringHint")} /></p>
|
||||
<div className="grid gap-3 sm:grid-cols-2">
|
||||
{([
|
||||
["gatewayLoss", gatewayLoss, setGatewayLoss, 0, 100],
|
||||
["gatewayDelay", gatewayDelay, setGatewayDelay, 0, 60000],
|
||||
["diskLimit", diskLimit, setDiskLimit, 0, 100],
|
||||
["memoryLimit", memoryLimit, setMemoryLimit, 0, 100],
|
||||
["swapLimit", swapLimit, setSwapLimit, 0, 100],
|
||||
["loadLimit", loadLimit, setLoadLimit, 0, 1000],
|
||||
["confirmSamples", confirmSamples, setConfirmSamples, 1, 60],
|
||||
["resourceSamples", resourceSamples, setResourceSamples, 1, 60],
|
||||
] as [string, number, (v: number) => void, number, number][]).map(([key, value, setter, min, max]) => (
|
||||
<label key={key} className="block text-sm">
|
||||
<span className="mb-1 block font-medium text-slate-700">{t(`settings.notify.limit.${key}`)}</span>
|
||||
<Input type="number" min={min} max={max} step={key === "loadLimit" ? 0.5 : 1} value={value} onChange={(e) => setter(Number(e.target.value))} />
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
<FieldGroup label={t("settings.notify.limit.watchlist")} hint={t("settings.notify.limit.watchlistHint")}>
|
||||
<Input value={watchlist} onChange={(e) => setWatchlist(e.target.value)} placeholder="unbound, dnsmasq, ntpd" />
|
||||
</FieldGroup>
|
||||
</div>
|
||||
|
||||
<div className="flex justify-end border-t border-slate-100 pt-4"><Button type="submit" disabled={saving || !ns}>{t("common.save")}</Button></div>
|
||||
</fieldset>
|
||||
</form>
|
||||
<NotificationChannelsSection />
|
||||
{error && <p role="alert" className="rounded-lg bg-red-50 p-3 text-sm text-red-700">{error}</p>}
|
||||
{notice && <p role="status" className="text-sm text-emerald-700">{notice}</p>}
|
||||
</section>
|
||||
|
||||
@@ -17,7 +17,9 @@ import {
|
||||
MaintenancePreview,
|
||||
MaintenanceRead,
|
||||
MaintenanceState,
|
||||
RolloutStatus,
|
||||
} from "@/lib/api";
|
||||
import { RolloutStatusBanner } from "@/components/RolloutStatusBanner";
|
||||
import { useReadOnly } from "@/lib/useReadOnly";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import type { TranslationParams } from "@/lib/i18n";
|
||||
@@ -132,6 +134,7 @@ export default function UpdatesPage() {
|
||||
const [cancelReason, setCancelReason] = useState("");
|
||||
const [submitting, setSubmitting] = useState(false);
|
||||
const [log, setLog] = useState<{ host: Host; text: string } | null>(null);
|
||||
const [rollout, setRollout] = useState<RolloutStatus | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
mounted.current = true;
|
||||
@@ -341,6 +344,7 @@ export default function UpdatesPage() {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<RolloutStatusBanner onStatus={setRollout} />
|
||||
{pageError && <p role="alert" className="mb-4 rounded-xl bg-red-50 px-4 py-3 text-sm text-red-700">{pageError}</p>}
|
||||
|
||||
<div className="mb-6 grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
|
||||
@@ -373,6 +377,9 @@ export default function UpdatesPage() {
|
||||
const carpWarning = state ? carpMaintenanceWarning(host, state, t) : null;
|
||||
const canCheck = host.enabled && state?.running === "ready" && !activeOperation;
|
||||
const isExpanded = expanded.has(host.id);
|
||||
// A host the active rollout still owns: no manual firmware action next to it (the rollout does it or skips it).
|
||||
const inRollout = Boolean(rollout?.host_ids.includes(host.id));
|
||||
const rolloutLock = inRollout ? [<span key="rollout">{t("rollouts.inRolloutTip")}</span>] : [];
|
||||
return (
|
||||
<Card key={host.id} className="overflow-hidden">
|
||||
<div className="p-4 sm:p-5">
|
||||
@@ -384,18 +391,19 @@ export default function UpdatesPage() {
|
||||
{!host.enabled && <Badge tone="neutral">{t("updates.disabled")}</Badge>}
|
||||
{(state?.carp_status ?? host.carp_status) && <Badge tone="neutral">CARP {state?.carp_status ?? host.carp_status}</Badge>}
|
||||
{carpWarning && <Tooltip content={carpWarning} focusable><Badge tone="warning">{t("updates.carpMode")}</Badge></Tooltip>}
|
||||
{inRollout && <Tooltip content={t("rollouts.inRolloutTip")} focusable><Badge tone="accent">{t("rollouts.inRollout")}</Badge></Tooltip>}
|
||||
{state?.reboot_required && <Tooltip content={t("updates.tipRebootRequired")} focusable><Badge tone="warning"><AlertTriangle size={12} />{t("updates.rebootRequired")}</Badge></Tooltip>}
|
||||
</div>
|
||||
<p className="mt-1 truncate text-xs text-slate-400">{host.base_url}{host.cluster_name ? t("updates.clusterSuffix", { name: host.cluster_name }) : ""}</p>
|
||||
</div>
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<Tooltip content={t("updates.tipCheck")} focusable={!canCheck}><Button variant="secondary" disabled={readOnly || !canCheck || detail?.checking || submitting || bulkChecking} onClick={() => void checkHost(host)}>{detail?.checking ? <Spinner size={13} /> : <RefreshCw size={14} />}{t("updates.check")}</Button></Tooltip>
|
||||
{state && <CarpSwitchControl host={host} state={state} disabled={readOnly || Boolean(activeOperation) || submitting || detail?.checking} onChanged={() => loadOne(host)} />}
|
||||
<BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : blockerReasons(visibleBlockers)}>
|
||||
<Button disabled={readOnly || blockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "update")}>{blockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareUpdate")}</Button>
|
||||
{state && <CarpSwitchControl host={host} state={state} disabled={readOnly || inRollout || Boolean(activeOperation) || submitting || detail?.checking} onChanged={() => loadOne(host)} />}
|
||||
<BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : [...rolloutLock, ...blockerReasons(visibleBlockers)]}>
|
||||
<Button disabled={readOnly || inRollout || blockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "update")}>{blockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareUpdate")}</Button>
|
||||
</BlockedTip>
|
||||
<BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : blockerReasons(visibleMajorBlockers)}>
|
||||
<Button variant="danger" disabled={readOnly || majorBlockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "major_update")}>{majorBlockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareMajor")}</Button>
|
||||
<BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : [...rolloutLock, ...blockerReasons(visibleMajorBlockers)]}>
|
||||
<Button variant="danger" disabled={readOnly || inRollout || majorBlockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "major_update")}>{majorBlockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareMajor")}</Button>
|
||||
</BlockedTip>
|
||||
{/* Not nested under `state &&`: a host can be offline/unreachable
|
||||
precisely because a maintenance action is stuck — the operator
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
"use client";
|
||||
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
import { Select } from "@/components/ui/Field";
|
||||
import { useToast } from "@/components/ui/Toast";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { AdoptAllResult, AdoptKind, api, Host, UnmanagedObject } from "@/lib/api";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
|
||||
interface Props {
|
||||
kind: AdoptKind;
|
||||
hosts: Host[];
|
||||
onClose: () => void;
|
||||
onAdopted: () => void;
|
||||
}
|
||||
|
||||
const TITLE_KEY: Record<AdoptKind, string> = {
|
||||
aliases: "adopt.title.aliases",
|
||||
"port-forward": "adopt.title.portForward",
|
||||
"one-to-one": "adopt.title.oneToOne",
|
||||
"outbound-nat": "adopt.title.outboundNat",
|
||||
};
|
||||
|
||||
// The alias types OPNexus manages (the others only show OPNsense's own English label).
|
||||
const ALIAS_TYPES = ["host", "network", "port", "networkgroup", "mac", "asn"];
|
||||
|
||||
function endpoint(net: string | undefined, port: string | undefined): string {
|
||||
const base = net || "any";
|
||||
return port ? `${base}:${port}` : base;
|
||||
}
|
||||
|
||||
function isSystem(item: UnmanagedObject): boolean {
|
||||
return item.reasons.length === 1 && item.reasons[0] === "system";
|
||||
}
|
||||
|
||||
/**
|
||||
* Lists the aliases / NAT rules that exist on a firewall but are not managed by OPNexus yet and lets the
|
||||
* operator take them over, one by one or all adoptable ones at once. Taking over only records the object —
|
||||
* nothing is written to the firewall until it is changed and rolled out the normal way (same idea as
|
||||
* AdoptRulesModal for filter rules).
|
||||
*/
|
||||
export function AdoptObjectsModal({ kind, hosts, onClose, onAdopted }: Props) {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const toast = useToast();
|
||||
const enabledHosts = hosts.filter((h) => h.enabled);
|
||||
// Derived rather than initialised once: the host list may still be loading when the dialog opens.
|
||||
const [chosenHostId, setHostId] = useState("");
|
||||
const hostId = chosenHostId || enabledHosts[0]?.id || "";
|
||||
const [items, setItems] = useState<UnmanagedObject[] | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [busy, setBusy] = useState<string | null>(null);
|
||||
const [result, setResult] = useState<AdoptAllResult | null>(null);
|
||||
|
||||
function load(id: string) {
|
||||
setItems(null);
|
||||
setError(null);
|
||||
api
|
||||
.listUnmanagedObjects(kind, id)
|
||||
.then(setItems)
|
||||
.catch((e) => {
|
||||
setItems([]);
|
||||
setError(errorMessage(e, "adopt.loadFailed"));
|
||||
});
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
if (hostId) {
|
||||
setResult(null);
|
||||
load(hostId);
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [hostId]);
|
||||
|
||||
const shown = useMemo(
|
||||
() => (items ?? []).filter((i) => !isSystem(i)).sort((a, b) => Number(b.adoptable) - Number(a.adoptable) || Number(b.bulk_adoptable) - Number(a.bulk_adoptable)),
|
||||
[items]
|
||||
);
|
||||
const systemCount = (items ?? []).filter(isSystem).length;
|
||||
const bulkCount = shown.filter((i) => i.bulk_adoptable).length;
|
||||
|
||||
function reasonText(code: string, details: Record<string, string>): string {
|
||||
return t(`adopt.reasons.${code}`, { detail: details[code] ?? "" });
|
||||
}
|
||||
|
||||
function describe(item: UnmanagedObject): string {
|
||||
const i = item.info;
|
||||
switch (kind) {
|
||||
case "aliases":
|
||||
return [ALIAS_TYPES.includes(i.type) ? t(`aliases.type.${i.type}`) : i.type_label, t("adopt.entries", { count: i.entries })].filter(Boolean).join(" · ");
|
||||
case "port-forward":
|
||||
return `${i.interface_label} · ${i.protocol} ${endpoint(i.source, i.source_port)} → ${endpoint(i.destination, i.destination_port)} ⇒ ${endpoint(i.target, i.target_port)}`;
|
||||
case "one-to-one":
|
||||
return `${i.interface_label} · ${i.type.toUpperCase()} · ${i.source} ⇄ ${i.external}${i.destination && i.destination !== "any" ? ` · ${t("adopt.towards", { destination: i.destination })}` : ""}`;
|
||||
case "outbound-nat":
|
||||
return `${i.interface_label} · ${i.protocol} ${endpoint(i.source, i.source_port)} → ${endpoint(i.destination, i.destination_port)} ⇒ ${i.target ? endpoint(i.target, i.target_port) : t("adopt.interfaceAddress")}`;
|
||||
}
|
||||
}
|
||||
|
||||
async function adopt(item: UnmanagedObject) {
|
||||
setBusy(item.uuid);
|
||||
try {
|
||||
await api.adoptObject(kind, hostId, item.uuid);
|
||||
toast.show(t("adopt.adopted", { name: item.title }), "success");
|
||||
onAdopted();
|
||||
load(hostId);
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "adopt.failed"), "danger");
|
||||
} finally {
|
||||
setBusy(null);
|
||||
}
|
||||
}
|
||||
|
||||
async function adoptAll() {
|
||||
setBusy("all");
|
||||
try {
|
||||
const done = await api.adoptAllObjects(kind, hostId);
|
||||
setResult(done);
|
||||
toast.show(t("adopt.allDone", { count: done.adopted.length, skipped: done.skipped.length }), done.adopted.length ? "success" : "info");
|
||||
onAdopted();
|
||||
load(hostId);
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "adopt.failed"), "danger");
|
||||
} finally {
|
||||
setBusy(null);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal title={t(TITLE_KEY[kind])} onClose={onClose}>
|
||||
<div className="max-h-[70vh] space-y-4 overflow-y-auto pr-1">
|
||||
<p className="text-sm text-slate-600">{t("adopt.intro")}</p>
|
||||
<div className="flex flex-wrap items-center justify-between gap-3">
|
||||
<label className="flex items-center gap-2 text-sm text-slate-600">
|
||||
{t("adopt.host")}
|
||||
<Select className="w-auto" value={hostId} onChange={(e) => setHostId(e.target.value)}>
|
||||
{enabledHosts.map((h) => (
|
||||
<option key={h.id} value={h.id}>
|
||||
{h.name}
|
||||
</option>
|
||||
))}
|
||||
</Select>
|
||||
</label>
|
||||
{bulkCount > 0 && (
|
||||
<Button onClick={adoptAll} disabled={busy !== null} loading={busy === "all"}>
|
||||
{t("adopt.all", { count: bulkCount })}
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
{error && <p role="alert" className="text-sm text-red-600">{error}</p>}
|
||||
{items === null && <p className="text-sm text-slate-500">{t("common.loading")}</p>}
|
||||
{items !== null && shown.length === 0 && !error && <p className="text-sm text-slate-500">{t("adopt.none")}</p>}
|
||||
{result && (
|
||||
<div role="status" className="rounded-lg border border-slate-200 bg-slate-50 p-3 text-sm text-slate-700">
|
||||
<p className="font-medium">{t("adopt.allDone", { count: result.adopted.length, skipped: result.skipped.length })}</p>
|
||||
{result.skipped.length > 0 && (
|
||||
<ul className="mt-2 space-y-1 text-xs text-slate-500">
|
||||
{result.skipped.map((s) => (
|
||||
<li key={s.uuid}>
|
||||
<span className="font-medium text-slate-700">{s.title}</span>: {s.reasons.map((code) => reasonText(code, s.reason_details)).join(" · ")}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
<ul className="space-y-2">
|
||||
{shown.map((item) => (
|
||||
<li key={item.uuid} className="rounded-lg border border-slate-200 p-3">
|
||||
<div className="flex flex-wrap items-center justify-between gap-2">
|
||||
<div className="min-w-0">
|
||||
<p className="truncate font-medium text-slate-900">
|
||||
{item.title}
|
||||
{!item.enabled && <span className="ml-2 text-xs font-normal text-slate-500">{t("common.statusDisabled")}</span>}
|
||||
</p>
|
||||
<p className="break-words text-xs text-slate-500">{describe(item)}</p>
|
||||
{kind === "aliases" && item.info.content && (
|
||||
<p className="mt-1 whitespace-pre-line break-all font-mono text-xs text-slate-500">{item.info.content}</p>
|
||||
)}
|
||||
{kind === "aliases" && item.info.description && <p className="mt-1 text-xs text-slate-500">{item.info.description}</p>}
|
||||
</div>
|
||||
{item.adoptable ? (
|
||||
<Button variant="outline" onClick={() => adopt(item)} disabled={busy !== null} loading={busy === item.uuid}>
|
||||
{t("adopt.action")}
|
||||
</Button>
|
||||
) : (
|
||||
<Badge tone={item.bulk_adoptable ? "info" : "neutral"}>
|
||||
{item.bulk_adoptable ? t("adopt.bulkOnly") : t("adopt.notPossible")}
|
||||
</Badge>
|
||||
)}
|
||||
</div>
|
||||
{!item.adoptable && (
|
||||
<p className="mt-2 text-xs text-slate-500">{item.reasons.map((code) => reasonText(code, item.reason_details)).join(" · ")}</p>
|
||||
)}
|
||||
{item.kept.length > 0 && (
|
||||
<p className="mt-2 text-xs text-slate-500">
|
||||
{t("adopt.kept", { fields: item.kept.map((field) => t(`adopt.keptFields.${field}`)).join(", ") })}
|
||||
</p>
|
||||
)}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
{systemCount > 0 && <p className="text-xs text-slate-500">{t("adopt.systemNote", { count: systemCount })}</p>}
|
||||
<div className="flex justify-end">
|
||||
<Button type="button" variant="ghost" onClick={onClose}>
|
||||
{t("common.close")}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
"use client";
|
||||
|
||||
import { Badge, BadgeTone } from "@/components/ui/Badge";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
import { Skeleton } from "@/components/ui/Skeleton";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, GatewayHealth, GatewayState, Host } from "@/lib/api";
|
||||
import { Globe, RefreshCw } from "lucide-react";
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
|
||||
const TONE: Record<GatewayState, BadgeTone> = {
|
||||
online: "success", degraded: "warning", offline: "danger", forced_down: "neutral", unknown: "neutral",
|
||||
};
|
||||
|
||||
export function GatewayStateBadge({ state }: { state: GatewayState }) {
|
||||
const { t } = useLanguage();
|
||||
return <Badge tone={TONE[state]}>{t(`gateway.state.${state}`)}</Badge>;
|
||||
}
|
||||
|
||||
export function gatewayNumbers(g: GatewayHealth, t: (key: string, params?: Record<string, string | number>) => string): string {
|
||||
if (g.loss === null && g.delay === null) return t("gateway.notMeasured");
|
||||
const parts: string[] = [];
|
||||
if (g.loss !== null) parts.push(t("gateway.loss", { value: g.loss }));
|
||||
if (g.delay !== null) parts.push(t("gateway.delay", { value: g.delay }));
|
||||
if (g.stddev !== null) parts.push(t("gateway.stddev", { value: g.stddev }));
|
||||
return parts.join(" · ");
|
||||
}
|
||||
|
||||
/** Live gateway state of one firewall (host detail page). */
|
||||
export function HostGateways({ host }: { host: Host }) {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const [gateways, setGateways] = useState<GatewayHealth[] | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
const load = useCallback(() => {
|
||||
setError(null);
|
||||
api.hostGateways(host.id).then(setGateways).catch((e) => { setGateways([]); setError(errorMessage(e, "gateway.loadFailed")); });
|
||||
}, [host.id, errorMessage]);
|
||||
|
||||
useEffect(() => { load(); }, [load]);
|
||||
|
||||
return (
|
||||
<>
|
||||
<div className="mb-3 mt-8 flex items-center justify-between gap-3">
|
||||
<h2 className="text-lg font-semibold text-slate-900">{t("gateway.title")}</h2>
|
||||
<Button type="button" variant="ghost" onClick={load} aria-label={t("common.refresh")}><RefreshCw size={14} />{t("common.refresh")}</Button>
|
||||
</div>
|
||||
<Card className="mb-8 p-5">
|
||||
{error && <p role="alert" className="text-sm text-red-600">{error}</p>}
|
||||
{!error && gateways === null && <Skeleton className="h-12 w-full" />}
|
||||
{!error && gateways !== null && gateways.length === 0 && <EmptyState icon={Globe} title={t("gateway.none")} />}
|
||||
{gateways !== null && gateways.length > 0 && (
|
||||
<ul className="divide-y divide-slate-100">
|
||||
{gateways.map((g) => (
|
||||
<li key={g.name} className="flex flex-wrap items-center justify-between gap-2 py-2 text-sm">
|
||||
<span className="min-w-0">
|
||||
<span className="font-medium text-slate-900">{g.name}</span>
|
||||
{g.address && <span className="text-slate-500"> · {g.address}</span>}
|
||||
</span>
|
||||
<span className="flex flex-wrap items-center gap-3">
|
||||
<span className="text-xs text-slate-500">{gatewayNumbers(g, t)}</span>
|
||||
<GatewayStateBadge state={g.state} />
|
||||
</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</Card>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -3,6 +3,7 @@ import { AlertTriangle, Clock3 } from "lucide-react";
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
|
||||
import { useEffect, useState } from "react";
|
||||
import Link from "next/link";
|
||||
import { api, Host, MaintenanceRead, MaintenancePreview, HostOperation, MaintenanceState, ScheduledMaintenanceItem } from "@/lib/api";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
@@ -21,11 +22,11 @@ export function HostMaintenance({host}: {host:Host}) {
|
||||
const [name,setName]=useState('');const [ack,setAck]=useState(false);const [log,setLog]=useState('');
|
||||
const [cancelOp,setCancelOp]=useState<HostOperation|null>(null);const [reason,setReason]=useState('');
|
||||
const [scheduled,setScheduled]=useState<ScheduledMaintenanceItem[]>([]);
|
||||
const [schedule,setSchedule]=useState<{action:'reboot'|'update'|'major_update';runAt:string;ack:boolean;handover:boolean}|null>(null);
|
||||
const [schedule,setSchedule]=useState<{action:'reboot'|'update'|'major_update';runAt:string;ack:boolean;handover:boolean;autoVerify:boolean}|null>(null);
|
||||
async function load(){setError('');try{setData(await api.hostMaintenance(host.id));}catch(e){setError(errorMessage(e,'maint.loadFailed'));}}
|
||||
async function loadScheduled(){try{setScheduled(await api.listScheduledMaintenance(host.id));}catch{/* non-fatal, section just stays empty */}}
|
||||
useEffect(()=>{if(open){load();loadScheduled();}},[open,host.id]);
|
||||
async function createSchedule(){if(!schedule)return;setBusy(true);setError('');try{await api.createScheduledMaintenance(host.id,schedule.action,new Date(schedule.runAt).toISOString(),schedule.handover);setSchedule(null);await loadScheduled();}catch(e){setError(errorMessage(e,'maint.scheduleFailed'));}finally{setBusy(false);}}
|
||||
async function createSchedule(){if(!schedule)return;setBusy(true);setError('');try{await api.createScheduledMaintenance(host.id,schedule.action,new Date(schedule.runAt).toISOString(),schedule.handover,schedule.autoVerify);setSchedule(null);await loadScheduled();}catch(e){setError(errorMessage(e,'maint.scheduleFailed'));}finally{setBusy(false);}}
|
||||
async function cancelSchedule(id:string){setBusy(true);setError('');try{await api.cancelScheduledMaintenance(host.id,id);await loadScheduled();}catch(e){setError(errorMessage(e,'maint.cancelFailed'));}finally{setBusy(false);}}
|
||||
useEffect(()=>{if(!open || !data?.operations.some(o=>['starting','submitted','unknown'].includes(o.status)))return;const timer=setInterval(()=>{load();loadScheduled();},15000);return()=>clearInterval(timer);},[open,host.id,data?.operations.map(o=>o.status).join(',')]);
|
||||
async function preview(action:'reboot'|'update'|'major_update'){setBusy(true);setError('');try{setPlan(await api.previewMaintenance(host.id,action));setName('');setAck(false);}catch(e){setError(errorMessage(e,'maint.previewFailed'));}finally{setBusy(false);}}
|
||||
@@ -34,10 +35,10 @@ export function HostMaintenance({host}: {host:Host}) {
|
||||
async function cancelOperation(){if(!cancelOp)return;setBusy(true);setError('');try{await api.cancelMaintenance(host.id,cancelOp.id,name,reason);setCancelOp(null);await load();}catch(e){setError(errorMessage(e,'maint.cancelFailed'));}finally{setBusy(false);}}
|
||||
async function check(){setBusy(true);setError('');try{await api.checkHostUpdates(host.id);setNotice(t('maint.checkRequested'));await load();}catch(e){setError(errorMessage(e,'maint.checkFailed'));}finally{setBusy(false);}}
|
||||
const masterBlocked=!!data?.state?.cluster_name && data.state.carp_status==='MASTER' && !schedule?.handover;
|
||||
return <section className="my-6 rounded-xl border border-slate-200 bg-white p-5"><div className="flex flex-wrap items-center justify-between gap-2"><h2 className="flex items-center gap-2 font-semibold">{t('maint.title')}<InfoTip text={t('maint.intro')} /></h2><Button variant="ghost" onClick={()=>setOpen(v=>!v)}>{open?t('maint.close'):t('maint.open')}</Button></div>{open && <div className="mt-4 space-y-4">{error && <p role="alert" className="text-sm text-red-600">{error}</p>}{notice && <p role="status" className="text-sm text-sky-700">{notice}</p>}{!data && !error && <p>{t('common.loading')}</p>}{data?.error && <p role="alert" className="text-sm text-red-600">{ts(data.error)}</p>}{data?.state && <><div className="grid gap-2 text-sm sm:grid-cols-2"><p>{t('maint.installed',{version:''})}<strong>{data.state.version}</strong></p><p>{t('maint.firmwareJob',{state:data.state.running})}</p><p className="sm:col-span-2">{ts(data.state.firmware_message)}</p><p>{t('maint.lastCheck',{value:data.state.last_check || t('maint.unknown')})}</p><p>{t('maint.backup',{value:data.state.backup?new Date(data.state.backup.created_at).toLocaleString(locale):t('maint.noBackup')})}</p></div><div className="flex flex-wrap items-center gap-2"><Tooltip content={t('updates.tipCheck')}><Button variant="secondary" disabled={busy || data.state.running!=='ready' || !host.enabled || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onClick={check}>{t('maint.checkUpdates')}</Button></Tooltip><CarpSwitchControl host={host} state={data.state} disabled={busy || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onChanged={load}/><Button variant="danger" disabled={busy || data.state.actions.reboot.length>0} onClick={()=>preview('reboot')}>{t('maint.previewReboot')}</Button><Tooltip content={t('updates.tipPrepare')}><Button disabled={busy || data.state.actions.update.length>0} onClick={()=>preview('update')}>{t('maint.previewUpdate')}</Button></Tooltip><Tooltip content={t('updates.tipPrepareMajor')}><Button variant="danger" disabled={busy || data.state.actions.major_update.length>0} onClick={()=>preview('major_update')}>{t('maint.previewMajor')}</Button></Tooltip><Button variant="ghost" disabled={busy || !host.enabled} onClick={()=>setSchedule({action:'reboot',runAt:'',ack:false,handover:false})}>{t('maint.scheduleLater')}</Button><Button variant="ghost" className="ml-auto" disabled={busy} onClick={load}>{t('maint.refreshStatus')}</Button></div>{readableReasons(data.state,host,t,ts).length>0 && <Tooltip focusable placement="bottom" content={<span className="block space-y-1">{readableReasons(data.state,host,t,ts).map(reason=><span key={reason} className="block">{reason}</span>)}</span>}><Badge tone="warning">{t('maint.hints')} ({readableReasons(data.state,host,t,ts).length})</Badge></Tooltip>}{scheduled.length>0 && <div className="space-y-2"><h3 className="text-sm font-semibold text-slate-700">{t('maint.scheduled')}</h3>{scheduled.map(s=><div key={s.id} className="flex flex-wrap items-center justify-between gap-2 rounded-lg border border-slate-200 p-3 text-sm"><div><strong>{labels[s.action]}</strong> · {new Date(s.run_at).toLocaleString(locale)} · {t(`maint.scheduleStatus.${s.status==='succeeded' && s.result?.handover ? 'completed' : s.status}`)}{s.status==='running' && s.phase && <span> ({t(`maint.phase.${s.phase}`)})</span>}{s.carp_handover && <Badge tone="info" className="ml-2">{t('maint.handoverBadge')}</Badge>}{s.result?.error && <span className="text-red-600"> — {ts(s.result.error)}</span>}</div>{s.status==='pending' && <Button variant="ghost" disabled={busy} onClick={()=>cancelSchedule(s.id)}>{t('common.cancel')}</Button>}</div>)}</div>}</>}{data?.operations && data.operations.length>0 && <h3 className="text-sm font-semibold text-slate-700">{t('maint.history')}</h3>}{data?.operations.map(op=><div key={op.id} className="rounded-lg border border-slate-200 p-3 text-sm"><strong>{labels[op.action]} · {['acknowledged','cancelled','unknown','starting'].includes(op.status)?t(`maint.opStatus.${op.status}`):t('maint.opStatus.submitted')}</strong><p className="text-slate-500">{new Date(op.created_at).toLocaleString(locale)} · {op.actor}</p><p className="mt-1">{ts(op.message ?? "")}</p>{['starting','submitted','unknown'].includes(op.status) && <div className="mt-2 flex flex-wrap gap-2"><Button variant="secondary" disabled={busy} onClick={()=>{setFinish(op);setName('');setAck(false);}}>{t('maint.confirmFinish')}</Button><Button variant="danger" disabled={busy} onClick={()=>{setCancelOp(op);setName('');setReason('');}}>{t('maint.cancelOperation')}</Button></div>}</div>)}<Button variant="ghost" disabled={busy} onClick={async()=>{try{const p=await api.hostUpdateProgress(host.id);setLog(p.log || p.status);}catch(e){setError(errorMessage(e,'maint.logFailed'));}}}>{t('maint.readLog')}</Button>{log && <pre className="max-h-72 overflow-auto whitespace-pre-wrap break-all rounded-lg bg-slate-50 p-3 tLine truncated
|
||||
return <section className="my-6 rounded-xl border border-slate-200 bg-white p-5"><div className="flex flex-wrap items-center justify-between gap-2"><h2 className="flex items-center gap-2 font-semibold">{t('maint.title')}<InfoTip text={t('maint.intro')} /></h2><Button variant="ghost" onClick={()=>setOpen(v=>!v)}>{open?t('maint.close'):t('maint.open')}</Button></div>{open && <div className="mt-4 space-y-4">{error && <p role="alert" className="text-sm text-red-600">{error}</p>}{notice && <p role="status" className="text-sm text-sky-700">{notice}</p>}{!data && !error && <p>{t('common.loading')}</p>}{data?.error && <p role="alert" className="text-sm text-red-600">{ts(data.error)}</p>}{data?.state && <><div className="grid gap-2 text-sm sm:grid-cols-2"><p>{t('maint.installed',{version:''})}<strong>{data.state.version}</strong></p><p>{t('maint.firmwareJob',{state:data.state.running})}</p><p className="sm:col-span-2">{ts(data.state.firmware_message)}</p><p>{t('maint.lastCheck',{value:data.state.last_check || t('maint.unknown')})}</p><p>{t('maint.backup',{value:data.state.backup?new Date(data.state.backup.created_at).toLocaleString(locale):t('maint.noBackup')})}</p></div><div className="flex flex-wrap items-center gap-2"><Tooltip content={t('updates.tipCheck')}><Button variant="secondary" disabled={busy || data.state.running!=='ready' || !host.enabled || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onClick={check}>{t('maint.checkUpdates')}</Button></Tooltip><CarpSwitchControl host={host} state={data.state} disabled={busy || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onChanged={load}/><Button variant="danger" disabled={busy || data.state.actions.reboot.length>0} onClick={()=>preview('reboot')}>{t('maint.previewReboot')}</Button><Tooltip content={t('updates.tipPrepare')}><Button disabled={busy || data.state.actions.update.length>0} onClick={()=>preview('update')}>{t('maint.previewUpdate')}</Button></Tooltip><Tooltip content={t('updates.tipPrepareMajor')}><Button variant="danger" disabled={busy || data.state.actions.major_update.length>0} onClick={()=>preview('major_update')}>{t('maint.previewMajor')}</Button></Tooltip><Button variant="ghost" disabled={busy || !host.enabled} onClick={()=>setSchedule({action:'reboot',runAt:'',ack:false,handover:false,autoVerify:true})}>{t('maint.scheduleLater')}</Button><Button variant="ghost" className="ml-auto" disabled={busy} onClick={load}>{t('maint.refreshStatus')}</Button></div>{readableReasons(data.state,host,t,ts).length>0 && <Tooltip focusable placement="bottom" content={<span className="block space-y-1">{readableReasons(data.state,host,t,ts).map(reason=><span key={reason} className="block">{reason}</span>)}</span>}><Badge tone="warning">{t('maint.hints')} ({readableReasons(data.state,host,t,ts).length})</Badge></Tooltip>}{scheduled.length>0 && <div className="space-y-2"><h3 className="text-sm font-semibold text-slate-700">{t('maint.scheduled')}</h3>{scheduled.map(s=><div key={s.id} className="flex flex-wrap items-center justify-between gap-2 rounded-lg border border-slate-200 p-3 text-sm"><div><strong>{labels[s.action]}</strong> · {new Date(s.run_at).toLocaleString(locale)} · {t(`maint.scheduleStatus.${s.status==='succeeded' && (s.result?.handover || s.result?.auto_verified) ? 'completed' : s.status}`)}{s.status==='running' && s.phase && <span> ({t(`maint.phase.${s.phase}`)})</span>}{s.carp_handover && <Badge tone="info" className="ml-2">{t('maint.handoverBadge')}</Badge>}{s.origin==='rollout' && <Badge tone="accent" className="ml-2">{t('maint.rolloutBadge')}</Badge>}{s.rollout_run_id && <Link href={`/rollouts/runs/${s.rollout_run_id}`} className="ml-2 text-brand-600 hover:underline">{t('maint.openRollout')}</Link>}{s.result?.error && <span className="text-red-600"> — {ts(s.result.error)}</span>}</div>{s.status==='pending' && <Button variant="ghost" disabled={busy} onClick={()=>cancelSchedule(s.id)}>{t('common.cancel')}</Button>}</div>)}</div>}</>}{data?.operations && data.operations.length>0 && <h3 className="text-sm font-semibold text-slate-700">{t('maint.history')}</h3>}{data?.operations.map(op=><div key={op.id} className="rounded-lg border border-slate-200 p-3 text-sm"><strong>{labels[op.action]} · {['acknowledged','cancelled','unknown','starting'].includes(op.status)?t(`maint.opStatus.${op.status}`):t('maint.opStatus.submitted')}</strong><p className="text-slate-500">{new Date(op.created_at).toLocaleString(locale)} · {op.actor}</p><p className="mt-1">{ts(op.message ?? "")}</p>{['starting','submitted','unknown'].includes(op.status) && <div className="mt-2 flex flex-wrap gap-2"><Button variant="secondary" disabled={busy} onClick={()=>{setFinish(op);setName('');setAck(false);}}>{t('maint.confirmFinish')}</Button><Button variant="danger" disabled={busy} onClick={()=>{setCancelOp(op);setName('');setReason('');}}>{t('maint.cancelOperation')}</Button></div>}</div>)}<Button variant="ghost" disabled={Line truncated
|
||||
{plan && <Modal title={t('maint.confirmTitle',{action:labels[plan.action]})} onClose={()=>{if(!busy)setPlan(null);}}><div className="space-y-4 text-sm"><p><strong>{plan.state.host_name}</strong> · {plan.state.base_url} · {plan.state.version}</p>{plan.action==='major_update' && <p className="flex items-center gap-2 font-medium text-red-700"><AlertTriangle size={16}/>{t('updates.majorWarnShort')}<InfoTip text={t('updates.majorWarnTip')}/></p>}<p className="flex items-center gap-2 font-medium text-flame-800"><Clock3 size={16}/>{t('updates.downtimeShort')}<InfoTip text={t('maint.downtimeNote',{reboot:plan.state.reboot_required?t('maint.andReboot'):''})}/></p><p className="text-xs text-slate-500">{t('updates.validUntilShort',{time:new Date(plan.expires_at).toLocaleTimeString(locale)})}</p><label className="block"><span className="flex items-center gap-1">{t('maint.confirmHostname')}<InfoTip text={t('updates.tipConfirmName')} size={13}/></span><Input className="mt-1" autoFocus autoComplete="off" spellCheck={false} aria-label={t('maint.confirmHostnameAria')} value={name} placeholder={host.name} onChange={e=>setName(e.target.value)} onKeyDown={e=>{if(e.key==='Enter'&&!busy&&name===host.name&&ack)void execute();}}/></label><label className="flex items-start gap-2"><input type="checkbox" checked={ack} onChange={e=>setAck(e.target.checked)}/>{t('maint.windowReady')}</label><Tooltip focusable={busy||name!==host.name||!ack} content={name!==host.name?t('updates.confirmRunDisabledName'):!ack?t('updates.confirmRunDisabledAck'):null}><Button variant="danger" disabled={busy || name!==host.name || !ack} onClick={execute}>{busy?t('maint.submitting'):t('maint.executeNow',{action:labels[plan.action]})}</Button></Tooltip></div></Modal>}
|
||||
{finish && <Modal title={finish.action.startsWith('carp_')?t('maint.finishCarpTitle'):t('maint.finishTitle')} onClose={()=>{if(!busy)setFinish(null);}}><div className="space-y-4 text-sm"><p className="flex items-start gap-1">{finish.action.startsWith('carp_')?t('maint.finishCarpCheck'):<>{t('maint.finishCheckPre')}<strong>{host.name}</strong>. </>}<InfoTip text={t('updates.finishNoteTip')} size={13} className="mt-0.5"/></p><Input autoFocus autoComplete="off" spellCheck={false} aria-label={t('maint.finishHostnameAria')} value={name} onChange={e=>setName(e.target.value)} placeholder={host.name}/><label className="flex gap-2"><input type="checkbox" checked={ack} onChange={e=>setAck(e.target.checked)}/>{t('maint.finishChecked')}</label>{error && <p className="text-red-600">{error}</p>}<Tooltip focusable={busy||name!==host.name||!ack} content={name!==host.name?t('updates.confirmRunDisabledName'):!ack?t('updates.confirmRunDisabledAck'):null}><Button disabled={busy || name!==host.name || !ack} onClick={closeOperation}>{t('maint.saveFinish')}</Button></Tooltip></div></Modal>}
|
||||
{cancelOp && <Modal title={t('maint.cancelOperation')} onClose={()=>{if(!busy)setCancelOp(null);}}><div className="space-y-4 text-sm"><p className="flex items-center gap-2 font-medium text-flame-800"><AlertTriangle size={16}/>{t('updates.cancelWarnShort')}<InfoTip text={t('updates.cancelWarnTip')}/></p><Input autoFocus autoComplete="off" spellCheck={false} aria-label={t('maint.cancelHostnameAria')} value={name} onChange={e=>setName(e.target.value)} placeholder={host.name}/><label className="block"><span className="flex items-center gap-1">{t('maint.reasonLabel')}<InfoTip text={t('updates.reasonMinHint')} size={13}/></span><textarea className="mt-1 w-full rounded-lg border border-slate-300 p-2" rows={2} maxLength={500} value={reason} onChange={e=>setReason(e.target.value)} placeholder={t('maint.reasonPlaceholder')}/></label>{error && <p className="text-red-600">{error}</p>}<Button variant="danger" disabled={busy || name!==host.name || reason.trim().length<3} onClick={cancelOperation}>{t('maint.closeWithoutConfirm')}</Button></div></Modal>}
|
||||
{schedule && <Modal title={t('maint.scheduleTitle')} onClose={()=>{if(!busy)setSchedule(null);}}><div className="space-y-4 text-sm"><label className="block"><span className="block">{t('maint.actionLabel')}</span><select className="mt-1 w-full rounded-lg border border-slate-300 p-2" value={schedule.action} onChange={e=>setSchedule({...schedule,action:e.target.value as typeof schedule.action})}><option value="reboot">{t('maint.action.reboot')}</option><option value="update">{t('maint.action.update')}</option><option value="major_update">{t('maint.action.major_update')}</option></select></label><label className="block"><span className="block">{t('maint.timeLabel')}</span><Input className="mt-1" aria-label={t('maint.timeLabel')} type="datetime-local" value={schedule.runAt} onChange={e=>setSchedule({...schedule,runAt:e.target.value})}/></label>{host.cluster_name && <label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={schedule.handover} onChange={e=>setSchedule({...schedule,handover:e.target.checked})}/><span>{t('maint.handoverOption')}<InfoTip text={t('maint.handoverHint')}/></span></label>}<p className="flex items-center gap-2 font-medium text-flame-800"><AlertTriangle size={16}/>{t('maint.unattendedShort')}<InfoTip text={t('maint.unattendedWarning')}/></p><label className="flex items-start gap-2"><input type="checkbox" checked={schedule.ack} onChange={e=>setSchedule({...schedule,ack:e.target.checked})}/>{t('maint.unattendedAck')}</label>{masterBlocked && <p role="alert" className="flex items-start gap-2 rounded-lg bg-red-50 p-3 text-red-700"><AlertTriangle size={16} className="mt-0.5 shrink-0"/>{t('maint.scheduleMasterBlocked')}</p>}{error && <p className="text-red-600">{error}</p>}<Button variant="danger" disabled={busy || masterBlocked || !schedule.runAt || !schedule.ack} onClick={createSchedule}>{busy?t('maint.scheduling'):t('maint.scheduleIt')}</Button></div></Modal>}
|
||||
{schedule && <Modal title={t('maint.scheduleTitle')} onClose={()=>{if(!busy)setSchedule(null);}}><div className="space-y-4 text-sm"><label className="block"><span className="block">{t('maint.actionLabel')}</span><select className="mt-1 w-full rounded-lg border border-slate-300 p-2" value={schedule.action} onChange={e=>setSchedule({...schedule,action:e.target.value as typeof schedule.action})}><option value="reboot">{t('maint.action.reboot')}</option><option value="update">{t('maint.action.update')}</option><option value="major_update">{t('maint.action.major_update')}</option></select></label><label className="block"><span className="block">{t('maint.timeLabel')}</span><Input className="mt-1" aria-label={t('maint.timeLabel')} type="datetime-local" value={schedule.runAt} onChange={e=>setSchedule({...schedule,runAt:e.target.value})}/></label>{host.cluster_name && <label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={schedule.handover} onChange={e=>setSchedule({...schedule,handover:e.target.checked})}/><span>{t('maint.handoverOption')}<InfoTip text={t('maint.handoverHint')}/></span></label>}<label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={schedule.autoVerify} onChange={e=>setSchedule({...schedule,autoVerify:e.target.checked})}/><span>{t('maint.autoVerifyOption')}<InfoTip text={t('maint.autoVerifyHint')}/></span></label><p className="flex items-center gap-2 font-medium text-flame-800"><AlertTriangle size={16}/>{t('maint.unattendedShort')}<InfoTip text={t('maint.unattendedWarning')}/></p><label className="flex items-start gap-2"><input type="checkbox" checked={schedule.ack} onChange={e=>setSchedule({...schedule,ack:e.target.checked})}/>{t('maint.unattendedAck')}</label>{masterBlocked && <p role="alert" className="flex items-start gap-2 rounded-lg bg-red-50 p-3 text-red-700"><AlertTriangle size={16} className="mt-0.5 shrink-0"/>{t('maint.scheduleMasterBlocked')}</p>}{error && <p className="text-red-600">{error}</p>}<Button variant="danger" disabled={busy || masterBlocked || !schedule.runAt || !schedule.ack} onClick={createSchedule}>{busy?t('maint.scheduling'):t('maint.scheduleIt')}</Button></div></Modal>}
|
||||
</section>;
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
"use client";
|
||||
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { FieldGroup, Input, Select } from "@/components/ui/Field";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
import { InfoTip } from "@/components/ui/Tooltip";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, MaintenanceWindow, MaintenanceWindowWrite } from "@/lib/api";
|
||||
import { useMemo, useState } from "react";
|
||||
|
||||
const EMPTY: MaintenanceWindowWrite = {
|
||||
name: "", timezone: "Europe/Berlin", recurrence: "weekly", weekdays: [6], month_mode: null, month_nth: null,
|
||||
month_weekday: null, month_day: null, start_time: "02:00", duration_minutes: 180, start_cutoff_minutes: 60, enabled: true,
|
||||
};
|
||||
|
||||
function timeZones(): string[] {
|
||||
try {
|
||||
const intl = Intl as unknown as { supportedValuesOf?: (key: string) => string[] };
|
||||
const zones = intl.supportedValuesOf?.("timeZone") ?? [];
|
||||
return zones.includes("UTC") ? zones : ["UTC", ...zones];
|
||||
} catch {
|
||||
return ["UTC", "Europe/Berlin"];
|
||||
}
|
||||
}
|
||||
|
||||
export function MaintenanceWindowEditor({ window, onClose, onSaved }: {
|
||||
window: MaintenanceWindow | null; onClose: () => void; onSaved: (w: MaintenanceWindow) => void;
|
||||
}) {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const [form, setForm] = useState<MaintenanceWindowWrite>(() => (window ? {
|
||||
name: window.name, timezone: window.timezone, recurrence: window.recurrence, weekdays: window.weekdays,
|
||||
month_mode: window.month_mode, month_nth: window.month_nth, month_weekday: window.month_weekday, month_day: window.month_day,
|
||||
start_time: window.start_time, duration_minutes: window.duration_minutes, start_cutoff_minutes: window.start_cutoff_minutes,
|
||||
enabled: window.enabled,
|
||||
} : EMPTY));
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const zones = useMemo(timeZones, []);
|
||||
const set = (patch: Partial<MaintenanceWindowWrite>) => setForm((f) => ({ ...f, ...patch }));
|
||||
|
||||
async function save() {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
const body: MaintenanceWindowWrite = form.recurrence === "weekly"
|
||||
? { ...form, month_mode: null, month_nth: null, month_weekday: null, month_day: null }
|
||||
: form.month_mode === "day"
|
||||
? { ...form, weekdays: [], month_nth: null, month_weekday: null }
|
||||
: { ...form, weekdays: [], month_mode: "nth_weekday", month_nth: form.month_nth ?? 1, month_weekday: form.month_weekday ?? 6, month_day: null };
|
||||
try {
|
||||
const saved = window ? await api.updateMaintenanceWindow(window.id, body) : await api.createMaintenanceWindow(body);
|
||||
onSaved(saved);
|
||||
} catch (e) {
|
||||
setError(errorMessage(e, "common.saveFailed"));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal title={window ? t("mwindows.edit") : t("mwindows.new")} onClose={() => !busy && onClose()}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<FieldGroup label={t("mwindows.name")}>
|
||||
<Input value={form.name} maxLength={120} onChange={(e) => set({ name: e.target.value })} />
|
||||
</FieldGroup>
|
||||
<div className="grid gap-4 sm:grid-cols-2">
|
||||
<FieldGroup label={t("mwindows.timezone")}>
|
||||
<Select value={form.timezone} onChange={(e) => set({ timezone: e.target.value })}>
|
||||
{zones.map((z) => <option key={z} value={z}>{z}</option>)}
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
<FieldGroup label={t("mwindows.recurrence")}>
|
||||
<Select value={form.recurrence} onChange={(e) => set({ recurrence: e.target.value as "weekly" | "monthly",
|
||||
month_mode: e.target.value === "monthly" ? (form.month_mode ?? "nth_weekday") : null })}>
|
||||
<option value="weekly">{t("mwindows.weekly")}</option>
|
||||
<option value="monthly">{t("mwindows.monthly")}</option>
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
</div>
|
||||
{form.recurrence === "weekly" ? (
|
||||
<fieldset>
|
||||
<legend className="mb-1 font-medium text-slate-700">{t("mwindows.weekdays")}</legend>
|
||||
<div className="flex flex-wrap gap-3">
|
||||
{[0, 1, 2, 3, 4, 5, 6].map((d) => (
|
||||
<label key={d} className="flex items-center gap-1">
|
||||
<input type="checkbox" checked={form.weekdays.includes(d)}
|
||||
onChange={(e) => set({ weekdays: e.target.checked ? [...form.weekdays, d].sort() : form.weekdays.filter((x) => x !== d) })} />
|
||||
{t(`mwindows.dayName.${d}`)}
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
</fieldset>
|
||||
) : (
|
||||
<div className="grid gap-4 sm:grid-cols-3">
|
||||
<FieldGroup label={t("mwindows.monthMode")}>
|
||||
<Select value={form.month_mode ?? "nth_weekday"} onChange={(e) => set({ month_mode: e.target.value as "nth_weekday" | "day" })}>
|
||||
<option value="nth_weekday">{t("mwindows.nthWeekday")}</option>
|
||||
<option value="day">{t("mwindows.dayOfMonth")}</option>
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
{form.month_mode === "day" ? (
|
||||
<FieldGroup label={t("mwindows.day")}>
|
||||
<Input type="number" min={1} max={28} value={form.month_day ?? 1} onChange={(e) => set({ month_day: Number(e.target.value) })} />
|
||||
</FieldGroup>
|
||||
) : (
|
||||
<>
|
||||
<FieldGroup label={t("mwindows.nthWeekday")}>
|
||||
<Select value={String(form.month_nth ?? 1)} onChange={(e) => set({ month_nth: Number(e.target.value) })}>
|
||||
{[1, 2, 3, 4, -1].map((n) => <option key={n} value={n}>{t(`mwindows.nth.${n}`)}</option>)}
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
<FieldGroup label={t("mwindows.weekdays")}>
|
||||
<Select value={String(form.month_weekday ?? 6)} onChange={(e) => set({ month_weekday: Number(e.target.value) })}>
|
||||
{[0, 1, 2, 3, 4, 5, 6].map((d) => <option key={d} value={d}>{t(`mwindows.dayLong.${d}`)}</option>)}
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
<div className="grid gap-4 sm:grid-cols-3">
|
||||
<FieldGroup label={t("mwindows.startTime")}>
|
||||
<Input type="time" value={form.start_time} onChange={(e) => set({ start_time: e.target.value })} />
|
||||
</FieldGroup>
|
||||
<FieldGroup label={t("mwindows.duration")}>
|
||||
<Input type="number" min={15} max={1440} value={form.duration_minutes} onChange={(e) => set({ duration_minutes: Number(e.target.value) })} />
|
||||
</FieldGroup>
|
||||
<label className="block">
|
||||
<span className="mb-1 flex items-center gap-1 font-medium text-slate-700">{t("mwindows.cutoff")}<InfoTip text={t("mwindows.cutoffHint")} size={13} /></span>
|
||||
<Input type="number" min={5} max={1440} value={form.start_cutoff_minutes} onChange={(e) => set({ start_cutoff_minutes: Number(e.target.value) })} />
|
||||
</label>
|
||||
</div>
|
||||
<p className="flex items-center gap-1 text-xs text-slate-500">{t("mwindows.dstHint")}</p>
|
||||
<label className="flex items-center gap-2">
|
||||
<input type="checkbox" checked={form.enabled} onChange={(e) => set({ enabled: e.target.checked })} />
|
||||
{t("mwindows.enabled")}
|
||||
</label>
|
||||
{error && <p role="alert" className="text-red-600">{error}</p>}
|
||||
<div className="flex justify-end gap-2">
|
||||
<Button variant="ghost" onClick={onClose} disabled={busy}>{t("common.cancel")}</Button>
|
||||
<Button onClick={save} loading={busy} disabled={!form.name.trim()}>{t("rollouts.plan.save")}</Button>
|
||||
</div>
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
@@ -35,6 +35,7 @@ import {
|
||||
Tags,
|
||||
Users,
|
||||
Waypoints,
|
||||
Waves,
|
||||
} from "lucide-react";
|
||||
import Link from "next/link";
|
||||
import { usePathname, useRouter } from "next/navigation";
|
||||
@@ -138,6 +139,12 @@ export const navGroups: NavGroup[] = [
|
||||
icon: RefreshCw,
|
||||
keywords: ["Firmware", "OPNsense-Updates", "Pakete", "Wartung", "Neustart erforderlich", "Versionen"],
|
||||
},
|
||||
{
|
||||
href: "/rollouts",
|
||||
label: "Rollouts",
|
||||
icon: Waves,
|
||||
keywords: ["Update-Ringe", "Canary", "Wellen", "Rollout", "Wartungsfenster", "Firmware-Rollout", "Pilot", "Health-Gate", "Ringe"],
|
||||
},
|
||||
{
|
||||
href: "/settings",
|
||||
label: "Einstellungen",
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
"use client";
|
||||
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
import { Input, Select } from "@/components/ui/Field";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
import { Skeleton } from "@/components/ui/Skeleton";
|
||||
import { useToast } from "@/components/ui/Toast";
|
||||
import { InfoTip } from "@/components/ui/Tooltip";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, ChannelType, NotificationChannel } from "@/lib/api";
|
||||
import { useNotAdmin } from "@/lib/useReadOnly";
|
||||
import { BellRing } from "lucide-react";
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
|
||||
const TYPES: ChannelType[] = ["ntfy", "gotify", "telegram", "teams", "matrix"];
|
||||
|
||||
interface FieldSpec { key: string; kind: "text" | "number"; placeholder?: string; min?: number; max?: number }
|
||||
|
||||
// Non-secret settings per channel type; the credential is a separate write-only field.
|
||||
const FIELDS: Record<ChannelType, FieldSpec[]> = {
|
||||
ntfy: [{ key: "server_url", kind: "text", placeholder: "https://ntfy.sh" }, { key: "topic", kind: "text" }, { key: "priority", kind: "number", min: 1, max: 5 }],
|
||||
gotify: [{ key: "server_url", kind: "text", placeholder: "https://gotify.example.org" }, { key: "priority", kind: "number", min: 0, max: 10 }],
|
||||
telegram: [{ key: "chat_id", kind: "text", placeholder: "-1001234567890" }, { key: "api_base", kind: "text", placeholder: "https://api.telegram.org" }],
|
||||
teams: [],
|
||||
matrix: [{ key: "homeserver_url", kind: "text", placeholder: "https://matrix.example.org" }, { key: "room_id", kind: "text", placeholder: "!abc123:example.org" }],
|
||||
};
|
||||
|
||||
const DEFAULTS: Record<ChannelType, Record<string, string | number>> = {
|
||||
ntfy: { server_url: "https://ntfy.sh", topic: "", priority: 4 },
|
||||
gotify: { server_url: "", priority: 5 },
|
||||
telegram: { chat_id: "" },
|
||||
teams: {},
|
||||
matrix: { homeserver_url: "", room_id: "" },
|
||||
};
|
||||
|
||||
function ChannelForm({ channel, onClose, onSaved }: { channel: NotificationChannel | null; onClose: () => void; onSaved: () => void }) {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const toast = useToast();
|
||||
const [type, setType] = useState<ChannelType>(channel?.channel_type ?? "ntfy");
|
||||
const [name, setName] = useState(channel?.name ?? "");
|
||||
const [enabled, setEnabled] = useState(channel?.enabled ?? true);
|
||||
const [config, setConfig] = useState<Record<string, string | number>>(channel ? { ...channel.config } : { ...DEFAULTS.ntfy });
|
||||
const [secret, setSecret] = useState("");
|
||||
const [clearSecret, setClearSecret] = useState(false);
|
||||
const [saving, setSaving] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const secretRequired = type !== "ntfy";
|
||||
|
||||
function changeType(next: ChannelType) {
|
||||
setType(next);
|
||||
setConfig({ ...DEFAULTS[next] });
|
||||
}
|
||||
|
||||
async function submit(e: React.FormEvent) {
|
||||
e.preventDefault();
|
||||
setSaving(true);
|
||||
setError("");
|
||||
try {
|
||||
if (channel) {
|
||||
await api.updateNotificationChannel(channel.id, { name, enabled, config, ...(secret ? { secret } : {}), clear_secret: clearSecret });
|
||||
} else {
|
||||
await api.createNotificationChannel({ name, channel_type: type, enabled, config, ...(secret ? { secret } : {}) });
|
||||
}
|
||||
toast.show(t("settings.channels.saved"), "success");
|
||||
onSaved();
|
||||
onClose();
|
||||
} catch (err) {
|
||||
setError(errorMessage(err, "common.saveFailed"));
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal title={channel ? t("settings.channels.edit") : t("settings.channels.add")} onClose={onClose}>
|
||||
<form onSubmit={submit} className="min-h-0 space-y-3 overflow-y-auto pr-1">
|
||||
{!channel && (
|
||||
<label className="block text-sm">
|
||||
<span className="mb-1 block font-medium text-slate-700">{t("settings.channels.type")}</span>
|
||||
<Select value={type} onChange={(e) => changeType(e.target.value as ChannelType)}>
|
||||
{TYPES.map((v) => <option key={v} value={v}>{t(`settings.channels.types.${v}`)}</option>)}
|
||||
</Select>
|
||||
</label>
|
||||
)}
|
||||
<label className="block text-sm">
|
||||
<span className="mb-1 block font-medium text-slate-700">{t("settings.channels.name")}</span>
|
||||
<Input value={name} maxLength={100} required onChange={(e) => setName(e.target.value)} />
|
||||
</label>
|
||||
{FIELDS[type].map((f) => (
|
||||
<label key={f.key} className="block text-sm">
|
||||
<span className="mb-1 block font-medium text-slate-700">{t(`settings.channels.field.${f.key}`)}</span>
|
||||
<Input
|
||||
type={f.kind}
|
||||
min={f.min}
|
||||
max={f.max}
|
||||
placeholder={f.placeholder}
|
||||
value={config[f.key] ?? ""}
|
||||
onChange={(e) => setConfig({ ...config, [f.key]: f.kind === "number" ? Number(e.target.value) : e.target.value })}
|
||||
/>
|
||||
</label>
|
||||
))}
|
||||
<label className="block text-sm">
|
||||
<span className="mb-1 block font-medium text-slate-700">
|
||||
{t(`settings.channels.secret.${type}`)} {channel?.secret_set && !clearSecret ? t("settings.notify.isSet") : t("settings.notify.notSet")}
|
||||
</span>
|
||||
<Input
|
||||
type="password"
|
||||
autoComplete="off"
|
||||
value={secret}
|
||||
required={secretRequired && !channel?.secret_set}
|
||||
onChange={(e) => { setSecret(e.target.value); setClearSecret(false); }}
|
||||
placeholder={channel?.secret_set ? t("settings.notify.keepUnchanged") : secretRequired ? "" : t("settings.notify.optional")}
|
||||
/>
|
||||
</label>
|
||||
{channel?.secret_set && !secretRequired && (
|
||||
<label className="flex items-center gap-2 text-sm text-slate-500">
|
||||
<input type="checkbox" checked={clearSecret} onChange={(e) => { setClearSecret(e.target.checked); if (e.target.checked) setSecret(""); }} />
|
||||
{t("settings.channels.removeSecret")}
|
||||
</label>
|
||||
)}
|
||||
<label className="flex items-center gap-2 text-sm">
|
||||
<input type="checkbox" checked={enabled} onChange={(e) => setEnabled(e.target.checked)} />
|
||||
{t("settings.channels.enabled")}
|
||||
</label>
|
||||
{error && <p role="alert" className="rounded-lg bg-red-50 p-3 text-sm text-red-700">{error}</p>}
|
||||
<div className="flex justify-end gap-2 border-t border-slate-100 pt-3">
|
||||
<Button type="button" variant="ghost" onClick={onClose}>{t("common.cancel")}</Button>
|
||||
<Button type="submit" loading={saving}>{t("common.save")}</Button>
|
||||
</div>
|
||||
</form>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
|
||||
export function NotificationChannelsSection() {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const toast = useToast();
|
||||
const readOnly = useNotAdmin();
|
||||
const [channels, setChannels] = useState<NotificationChannel[] | null>(null);
|
||||
const [editing, setEditing] = useState<NotificationChannel | "new" | null>(null);
|
||||
const [busy, setBusy] = useState<string | null>(null);
|
||||
|
||||
const load = useCallback(() => {
|
||||
api.notificationChannels().then(setChannels).catch((e) => { setChannels([]); toast.show(errorMessage(e, "settings.channels.loadFailed"), "danger"); });
|
||||
}, [errorMessage, toast]);
|
||||
|
||||
useEffect(() => { load(); }, [load]);
|
||||
|
||||
async function test(c: NotificationChannel) {
|
||||
setBusy(c.id);
|
||||
try {
|
||||
await api.testNotificationChannel(c.id);
|
||||
toast.show(t("settings.channels.testSent", { name: c.name }), "success");
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "settings.notify.testFailed"), "danger");
|
||||
} finally {
|
||||
setBusy(null);
|
||||
}
|
||||
}
|
||||
|
||||
async function toggle(c: NotificationChannel) {
|
||||
setBusy(c.id);
|
||||
try {
|
||||
await api.updateNotificationChannel(c.id, { enabled: !c.enabled });
|
||||
load();
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "common.saveFailed"), "danger");
|
||||
} finally {
|
||||
setBusy(null);
|
||||
}
|
||||
}
|
||||
|
||||
async function remove(c: NotificationChannel) {
|
||||
if (!confirm(t("settings.channels.confirmDelete", { name: c.name }))) return;
|
||||
setBusy(c.id);
|
||||
try {
|
||||
await api.deleteNotificationChannel(c.id);
|
||||
toast.show(t("settings.channels.deleted", { name: c.name }), "success");
|
||||
load();
|
||||
} catch (e) {
|
||||
toast.show(errorMessage(e, "common.deleteFailed"), "danger");
|
||||
} finally {
|
||||
setBusy(null);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-3 rounded-lg border border-slate-100 p-4">
|
||||
<div className="flex flex-wrap items-center justify-between gap-2">
|
||||
<h3 className="flex items-center gap-2 text-sm font-medium">{t("settings.channels.title")}<InfoTip text={t("settings.channels.hint")} /></h3>
|
||||
{!readOnly && <Button type="button" variant="outline" onClick={() => setEditing("new")}>{t("settings.channels.add")}</Button>}
|
||||
</div>
|
||||
{channels === null ? (
|
||||
<Skeleton className="h-12" />
|
||||
) : channels.length === 0 ? (
|
||||
<EmptyState icon={BellRing} title={t("settings.channels.empty")} />
|
||||
) : (
|
||||
<ul className="divide-y divide-slate-100">
|
||||
{channels.map((c) => (
|
||||
<li key={c.id} className="flex flex-wrap items-center justify-between gap-2 py-2 text-sm">
|
||||
<span className="flex min-w-0 items-center gap-2">
|
||||
<span className="truncate font-medium text-slate-900">{c.name}</span>
|
||||
<Badge tone="info">{t(`settings.channels.types.${c.channel_type}`)}</Badge>
|
||||
{!c.enabled && <Badge tone="neutral">{t("settings.channels.disabled")}</Badge>}
|
||||
</span>
|
||||
<span className="flex flex-wrap items-center gap-1">
|
||||
<Button type="button" variant="ghost" disabled={busy === c.id} onClick={() => test(c)}>{t("settings.channels.test")}</Button>
|
||||
{!readOnly && (
|
||||
<>
|
||||
<Button type="button" variant="ghost" disabled={busy === c.id} onClick={() => toggle(c)}>{c.enabled ? t("settings.channels.disable") : t("settings.channels.enable")}</Button>
|
||||
<Button type="button" variant="ghost" disabled={busy === c.id} onClick={() => setEditing(c)}>{t("settings.channels.editAction")}</Button>
|
||||
<Button type="button" variant="ghost" disabled={busy === c.id} onClick={() => remove(c)}>{t("settings.channels.deleteAction")}</Button>
|
||||
</>
|
||||
)}
|
||||
</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
{editing !== null && <ChannelForm channel={editing === "new" ? null : editing} onClose={() => setEditing(null)} onSaved={load} />}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import { useRouter, useSearchParams } from "next/navigation";
|
||||
import { WorkflowBadge, workflowStates } from "@/components/WorkflowBadge";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { OneToOneReview } from "@/components/OneToOneReview";
|
||||
import { AdoptObjectsModal } from "@/components/AdoptObjectsModal";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
@@ -39,6 +40,7 @@ import {
|
||||
ShieldCheck,
|
||||
Trash2,
|
||||
XCircle,
|
||||
Download,
|
||||
} from "lucide-react";
|
||||
import { Suspense, useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useReadOnly } from "@/lib/useReadOnly";
|
||||
@@ -115,6 +117,7 @@ export default function OneToOnePanel() {
|
||||
function OneToOneContent() {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const readOnly = useReadOnly();
|
||||
const [showAdopt, setShowAdopt] = useState(false);
|
||||
const toast = useToast();
|
||||
const router = useRouter();
|
||||
const params = useSearchParams();
|
||||
@@ -391,13 +394,21 @@ function OneToOneContent() {
|
||||
<div className="mb-6 flex flex-wrap items-center justify-between gap-3">
|
||||
<h1 className="text-2xl font-semibold text-slate-900 flex items-center gap-2">{selectedGroup ? t("rules.groupTitle", { name: selectedGroup.name }) : '1:1 NAT'}<InfoTip text={t("rules.subtitle")} size={18} /></h1>
|
||||
{activeTab === "managed" && (
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("rules.new")}
|
||||
</Button>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button variant="outline" onClick={() => setShowAdopt(true)} disabled={readOnly}>
|
||||
<Download size={16} />
|
||||
{t("adopt.button")}
|
||||
</Button>
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("rules.new")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{showAdopt && <AdoptObjectsModal kind="one-to-one" hosts={hosts} onClose={() => setShowAdopt(false)} onAdopted={load} />}
|
||||
|
||||
<Tabs
|
||||
className="mb-6"
|
||||
items={[
|
||||
|
||||
@@ -285,6 +285,9 @@ export function OpnexusUpdateSection() {
|
||||
</details>
|
||||
)}
|
||||
|
||||
{status.can_update && (status.active_rollouts ?? 0) > 0 && (
|
||||
<p role="status" className="rounded-lg bg-flame-50 p-3 text-sm text-flame-800">{t("rollouts.updaterWarning")}</p>
|
||||
)}
|
||||
{status.can_update && status.latest && !running && (
|
||||
<Button onClick={() => setConfirm("update")} disabled={busy}>
|
||||
{t("settings.update.button", { latest: status.latest })}
|
||||
|
||||
@@ -7,6 +7,7 @@ import { useRouter, useSearchParams } from "next/navigation";
|
||||
import { WorkflowBadge, workflowStates } from "@/components/WorkflowBadge";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { OutboundNatReview } from "@/components/OutboundNatReview";
|
||||
import { AdoptObjectsModal } from "@/components/AdoptObjectsModal";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
@@ -40,6 +41,7 @@ import {
|
||||
ShieldCheck,
|
||||
Trash2,
|
||||
XCircle,
|
||||
Download,
|
||||
} from "lucide-react";
|
||||
import { Suspense, useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useReadOnly } from "@/lib/useReadOnly";
|
||||
@@ -118,6 +120,7 @@ export default function OutboundNatPanel() {
|
||||
function OutboundNatContent() {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const readOnly = useReadOnly();
|
||||
const [showAdopt, setShowAdopt] = useState(false);
|
||||
const toast = useToast();
|
||||
const router = useRouter();
|
||||
const params = useSearchParams();
|
||||
@@ -399,14 +402,22 @@ function OutboundNatContent() {
|
||||
<div className="mb-6 flex flex-wrap items-center justify-between gap-3">
|
||||
<h1 className="text-2xl font-semibold text-slate-900 flex items-center gap-2">{selectedGroup ? t("rules.groupTitle", { name: selectedGroup.name }) : "Outbound NAT"}<InfoTip text={t("rules.subtitle")} size={18} /></h1>
|
||||
{activeTab === "managed" && (
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("rules.new")}
|
||||
</Button>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button variant="outline" onClick={() => setShowAdopt(true)} disabled={readOnly}>
|
||||
<Download size={16} />
|
||||
{t("adopt.button")}
|
||||
</Button>
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("rules.new")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<OutboundModes hosts={hosts} />
|
||||
{showAdopt && <AdoptObjectsModal kind="outbound-nat" hosts={hosts} onClose={() => setShowAdopt(false)} onAdopted={load} />}
|
||||
|
||||
<Tabs
|
||||
className="mb-6"
|
||||
items={[
|
||||
|
||||
@@ -6,6 +6,7 @@ import { useRouter, useSearchParams } from "next/navigation";
|
||||
import { WorkflowBadge, workflowStates } from "@/components/WorkflowBadge";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { PortForwardReview } from "@/components/PortForwardReview";
|
||||
import { AdoptObjectsModal } from "@/components/AdoptObjectsModal";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Card } from "@/components/ui/Card";
|
||||
import { EmptyState } from "@/components/ui/EmptyState";
|
||||
@@ -39,6 +40,7 @@ import {
|
||||
ShieldCheck,
|
||||
Trash2,
|
||||
XCircle,
|
||||
Download,
|
||||
} from "lucide-react";
|
||||
import { Suspense, useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useReadOnly } from "@/lib/useReadOnly";
|
||||
@@ -116,6 +118,7 @@ export default function PortForwardPanel() {
|
||||
function PortForwardContent() {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const readOnly = useReadOnly();
|
||||
const [showAdopt, setShowAdopt] = useState(false);
|
||||
const toast = useToast();
|
||||
const router = useRouter();
|
||||
const params = useSearchParams();
|
||||
@@ -397,13 +400,21 @@ function PortForwardContent() {
|
||||
<div className="mb-6 flex flex-wrap items-center justify-between gap-3">
|
||||
<h1 className="text-2xl font-semibold text-slate-900 flex items-center gap-2">{selectedGroup ? t("rules.groupTitle", { name: selectedGroup.name }) : "Port Forward"}<InfoTip text={t("rules.subtitle")} size={18} /></h1>
|
||||
{activeTab === "managed" && (
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("rules.new")}
|
||||
</Button>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button variant="outline" onClick={() => setShowAdopt(true)} disabled={readOnly}>
|
||||
<Download size={16} />
|
||||
{t("adopt.button")}
|
||||
</Button>
|
||||
<Button disabled={readOnly} onClick={() => openEditor(null)}>
|
||||
<Plus size={16} />
|
||||
{t("rules.new")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{showAdopt && <AdoptObjectsModal kind="port-forward" hosts={hosts} onClose={() => setShowAdopt(false)} onAdopted={load} />}
|
||||
|
||||
<Tabs
|
||||
className="mb-6"
|
||||
items={[
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
"use client";
|
||||
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { FieldGroup, Input, Select } from "@/components/ui/Field";
|
||||
import { Modal } from "@/components/ui/Modal";
|
||||
import { InfoTip } from "@/components/ui/Tooltip";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, DeviceGroup, MaintenanceWindow, RolloutCheck, RolloutPlan, RolloutPlanWrite, RolloutWaveWrite } from "@/lib/api";
|
||||
import { ArrowDown, ArrowUp, Plus, Trash2 } from "lucide-react";
|
||||
import { useState } from "react";
|
||||
|
||||
const OPTIONAL_CHECKS: RolloutCheck[] = ["carp", "gateways", "services", "compliance"];
|
||||
const DEFAULT_CHECKS: Partial<Record<RolloutCheck, boolean>> = { carp: true, gateways: true, services: true, compliance: false };
|
||||
|
||||
function newWave(index: number): RolloutWaveWrite {
|
||||
return { label: ["canary", "pilot", "prod"][index] ?? `${index + 1}`, group_ids: [], soak_minutes: 30, carp_handover: index > 0, require_approval: false };
|
||||
}
|
||||
|
||||
export function RolloutPlanEditor({ plan, groups, windows, onClose, onSaved }: {
|
||||
plan: RolloutPlan | null; groups: DeviceGroup[]; windows: MaintenanceWindow[]; onClose: () => void; onSaved: (p: RolloutPlan) => void;
|
||||
}) {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const [form, setForm] = useState<RolloutPlanWrite>(() => plan ? {
|
||||
name: plan.name, description: plan.description, action: plan.action, window_id: plan.window_id, auto_start: plan.auto_start,
|
||||
gate_checks: { ...DEFAULT_CHECKS, ...plan.gate_checks }, compliance_mode: plan.compliance_mode,
|
||||
waves: plan.waves.map((w) => ({ label: w.label, group_ids: w.group_ids, soak_minutes: w.soak_minutes, carp_handover: w.carp_handover, require_approval: w.require_approval })),
|
||||
} : { name: "", description: "", action: "update", window_id: null, auto_start: false, gate_checks: DEFAULT_CHECKS, compliance_mode: "warn", waves: [newWave(0)] });
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const set = (patch: Partial<RolloutPlanWrite>) => setForm((f) => ({ ...f, ...patch }));
|
||||
const setWave = (i: number, patch: Partial<RolloutWaveWrite>) => set({ waves: form.waves.map((w, j) => (j === i ? { ...w, ...patch } : w)) });
|
||||
const moveWave = (i: number, dir: -1 | 1) => {
|
||||
const waves = [...form.waves];
|
||||
[waves[i], waves[i + dir]] = [waves[i + dir], waves[i]];
|
||||
set({ waves });
|
||||
};
|
||||
const autoStartPossible = form.action === "update" && !!form.window_id;
|
||||
|
||||
async function save() {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
const body = { ...form, auto_start: autoStartPossible && form.auto_start };
|
||||
try {
|
||||
onSaved(plan ? await api.updateRolloutPlan(plan.id, body) : await api.createRolloutPlan(body));
|
||||
} catch (e) {
|
||||
setError(errorMessage(e, "common.saveFailed"));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
const valid = form.name.trim() && form.waves.length > 0 && form.waves.every((w) => w.label.trim() && w.group_ids.length > 0);
|
||||
return (
|
||||
<Modal title={plan ? t("rollouts.plan.edit") : t("rollouts.plan.new")} onClose={() => !busy && onClose()}>
|
||||
<div className="space-y-4 text-sm">
|
||||
<div className="grid gap-4 sm:grid-cols-2">
|
||||
<FieldGroup label={t("rollouts.plan.name")}>
|
||||
<Input value={form.name} maxLength={120} onChange={(e) => set({ name: e.target.value })} />
|
||||
</FieldGroup>
|
||||
<FieldGroup label={t("rollouts.plan.action")}>
|
||||
<Select value={form.action} onChange={(e) => set({ action: e.target.value as RolloutPlanWrite["action"] })}>
|
||||
<option value="update">{t("rollouts.action.update")}</option>
|
||||
<option value="major_update">{t("rollouts.action.major_update")}</option>
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
</div>
|
||||
<FieldGroup label={t("rollouts.plan.description")}>
|
||||
<Input value={form.description} maxLength={2000} onChange={(e) => set({ description: e.target.value })} />
|
||||
</FieldGroup>
|
||||
<FieldGroup label={t("rollouts.plan.window")}>
|
||||
<Select value={form.window_id ?? ""} onChange={(e) => set({ window_id: e.target.value || null })}>
|
||||
<option value="">{t("rollouts.plan.noWindow")}</option>
|
||||
{windows.map((w) => <option key={w.id} value={w.id}>{w.name}</option>)}
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
<label className={`flex items-start gap-2 ${autoStartPossible ? "" : "opacity-60"}`}>
|
||||
<input type="checkbox" className="mt-1" disabled={!autoStartPossible} checked={autoStartPossible && form.auto_start}
|
||||
onChange={(e) => set({ auto_start: e.target.checked })} />
|
||||
<span>{t("rollouts.plan.autoStart")} <InfoTip text={t("rollouts.plan.autoStartHint")} size={13} /></span>
|
||||
</label>
|
||||
|
||||
<fieldset className="space-y-3">
|
||||
<legend className="mb-1 font-medium text-slate-700">{t("rollouts.plan.waves")}</legend>
|
||||
{form.waves.map((wave, i) => (
|
||||
<div key={i} className="space-y-3 rounded-lg border border-slate-200 p-3">
|
||||
<div className="flex flex-wrap items-end gap-2">
|
||||
<span className="mb-2 inline-flex h-6 w-6 items-center justify-center rounded-full bg-slate-100 text-xs font-semibold text-slate-700">{i + 1}</span>
|
||||
<div className="min-w-[8rem] flex-1">
|
||||
<FieldGroup label={t("rollouts.plan.waveLabel")}>
|
||||
<Input value={wave.label} maxLength={60} onChange={(e) => setWave(i, { label: e.target.value })} />
|
||||
</FieldGroup>
|
||||
</div>
|
||||
<div className="w-32">
|
||||
<FieldGroup label={t("rollouts.plan.soak")} hint={t("rollouts.plan.soakHint")}>
|
||||
<Input type="number" min={0} max={1440} value={wave.soak_minutes} onChange={(e) => setWave(i, { soak_minutes: Number(e.target.value) })} />
|
||||
</FieldGroup>
|
||||
</div>
|
||||
<div className="mb-0.5 flex gap-1">
|
||||
<Button variant="ghost" aria-label={t("rollouts.plan.moveUp")} disabled={i === 0} onClick={() => moveWave(i, -1)}><ArrowUp size={14} /></Button>
|
||||
<Button variant="ghost" aria-label={t("rollouts.plan.moveDown")} disabled={i === form.waves.length - 1} onClick={() => moveWave(i, 1)}><ArrowDown size={14} /></Button>
|
||||
<Button variant="dangerSubtle" aria-label={t("rollouts.plan.removeWave")} disabled={form.waves.length === 1}
|
||||
onClick={() => set({ waves: form.waves.filter((_, j) => j !== i) })}><Trash2 size={14} /></Button>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<p className="mb-1 font-medium text-slate-700">{t("rollouts.plan.groups")}</p>
|
||||
<div className="flex flex-wrap gap-x-4 gap-y-1">
|
||||
{groups.map((g) => (
|
||||
<label key={g.id} className="flex items-center gap-1">
|
||||
<input type="checkbox" checked={wave.group_ids.includes(g.id)}
|
||||
onChange={(e) => setWave(i, { group_ids: e.target.checked ? [...wave.group_ids, g.id] : wave.group_ids.filter((x) => x !== g.id) })} />
|
||||
{g.name} <span className="text-xs text-slate-500">({g.host_ids.length})</span>
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex flex-wrap gap-x-6 gap-y-1">
|
||||
<label className="flex items-center gap-2">
|
||||
<input type="checkbox" checked={wave.carp_handover} onChange={(e) => setWave(i, { carp_handover: e.target.checked })} />
|
||||
{t("rollouts.plan.handover")}
|
||||
</label>
|
||||
{i > 0 && (
|
||||
<label className="flex items-center gap-2">
|
||||
<input type="checkbox" checked={wave.require_approval} onChange={(e) => setWave(i, { require_approval: e.target.checked })} />
|
||||
{t("rollouts.plan.approval")}
|
||||
</label>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
<Button variant="outline" disabled={form.waves.length >= 10} onClick={() => set({ waves: [...form.waves, newWave(form.waves.length)] })}>
|
||||
<Plus size={14} />{t("rollouts.plan.addWave")}
|
||||
</Button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend className="mb-1 flex items-center gap-1 font-medium text-slate-700">{t("rollouts.plan.checks")}<InfoTip text={t("rollouts.plan.checksHint")} size={13} /></legend>
|
||||
<div className="flex flex-wrap gap-x-5 gap-y-1">
|
||||
{OPTIONAL_CHECKS.map((c) => (
|
||||
<label key={c} className="flex items-center gap-2">
|
||||
<input type="checkbox" checked={!!form.gate_checks[c]} onChange={(e) => set({ gate_checks: { ...form.gate_checks, [c]: e.target.checked } })} />
|
||||
{t(`rollouts.gate.check.${c}`)}
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
{form.gate_checks.compliance && (
|
||||
<div className="mt-2 max-w-xs">
|
||||
<FieldGroup label={t("rollouts.plan.complianceMode")}>
|
||||
<Select value={form.compliance_mode} onChange={(e) => set({ compliance_mode: e.target.value as "warn" | "fail" })}>
|
||||
<option value="warn">{t("rollouts.plan.complianceWarn")}</option>
|
||||
<option value="fail">{t("rollouts.plan.complianceFail")}</option>
|
||||
</Select>
|
||||
</FieldGroup>
|
||||
</div>
|
||||
)}
|
||||
</fieldset>
|
||||
{error && <p role="alert" className="text-red-600">{error}</p>}
|
||||
<div className="flex justify-end gap-2">
|
||||
<Button variant="ghost" onClick={onClose} disabled={busy}>{t("common.cancel")}</Button>
|
||||
<Button onClick={save} loading={busy} disabled={!valid}>{t("rollouts.plan.save")}</Button>
|
||||
</div>
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
"use client";
|
||||
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, RolloutStatus } from "@/lib/api";
|
||||
import { pauseText, RUN_TONE } from "@/lib/rollouts";
|
||||
import { Waves } from "lucide-react";
|
||||
import Link from "next/link";
|
||||
import { useEffect, useState } from "react";
|
||||
|
||||
/** "Rollout X – wave 2/3" strip for /updates and /rollouts. Polls while a run is active. */
|
||||
export function RolloutStatusBanner({ onStatus }: { onStatus?: (status: RolloutStatus) => void }) {
|
||||
const { t } = useLanguage();
|
||||
const [status, setStatus] = useState<RolloutStatus | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
let alive = true;
|
||||
async function load() {
|
||||
try {
|
||||
const next = await api.rolloutStatus();
|
||||
if (!alive) return;
|
||||
setStatus(next);
|
||||
onStatus?.(next);
|
||||
} catch {
|
||||
/* the banner is optional */
|
||||
}
|
||||
}
|
||||
load();
|
||||
const timer = setInterval(load, 15000);
|
||||
return () => {
|
||||
alive = false;
|
||||
clearInterval(timer);
|
||||
};
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, []);
|
||||
|
||||
const run = status?.active;
|
||||
if (!run) return null;
|
||||
return (
|
||||
<div role="status" className="mb-4 flex flex-wrap items-center gap-2 rounded-xl border border-sky-200 bg-brand-50 px-4 py-3 text-sm text-slate-800">
|
||||
<Waves size={16} className="shrink-0 text-brand-600" aria-hidden />
|
||||
<span className="min-w-0 flex-1">
|
||||
{t("rollouts.banner.active", {
|
||||
plan: run.plan_name, wave: `${run.current_wave + 1}/${run.wave_count}`, label: run.current_wave_label ?? "",
|
||||
status: t(`rollouts.run.status.${run.status}`),
|
||||
})}
|
||||
{run.status === "paused" && run.pause_reason && <span className="text-slate-600"> — {pauseText(t, run.pause_reason)}</span>}
|
||||
</span>
|
||||
<Badge tone={RUN_TONE[run.status]}>{t(`rollouts.run.status.${run.status}`)}</Badge>
|
||||
<Link href={`/rollouts/runs/${run.id}`} className="font-medium text-brand-600 underline-offset-2 hover:underline">
|
||||
{t("rollouts.banner.open")}
|
||||
</Link>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
+159
-4
@@ -601,6 +601,28 @@ export interface UnmanagedRule {
|
||||
reasons: string[];
|
||||
}
|
||||
|
||||
export type AdoptKind = "aliases" | "port-forward" | "one-to-one" | "outbound-nat";
|
||||
|
||||
/** An alias / NAT rule found live on a host that OPNexus does not manage yet. */
|
||||
export interface UnmanagedObject {
|
||||
uuid: string;
|
||||
title: string;
|
||||
enabled: boolean;
|
||||
adoptable: boolean;
|
||||
/** Not adoptable alone yet, but in the same "adopt all" run (an alias whose referenced aliases come first). */
|
||||
bulk_adoptable: boolean;
|
||||
reasons: string[];
|
||||
reason_details: Record<string, string>;
|
||||
/** Settings OPNexus does not manage that stay untouched on the firewall. */
|
||||
kept: string[];
|
||||
info: Record<string, string>;
|
||||
}
|
||||
|
||||
export interface AdoptAllResult {
|
||||
adopted: { uuid: string; title: string; id: string }[];
|
||||
skipped: { uuid: string; title: string; reasons: string[]; reason_details: Record<string, string> }[];
|
||||
}
|
||||
|
||||
export type PolicyAliasType = "host" | "network" | "port" | "networkgroup" | "mac" | "asn";
|
||||
|
||||
export interface PolicyAlias {
|
||||
@@ -1251,6 +1273,7 @@ export interface UpdaterStatus {
|
||||
rollback_available: boolean;
|
||||
rollback_loses_data: boolean;
|
||||
rollback_version: string | null;
|
||||
active_rollouts?: number;
|
||||
}
|
||||
|
||||
export const api = {
|
||||
@@ -1264,6 +1287,12 @@ export const api = {
|
||||
getAuditExport: () => request<AuditExportSettings>("/settings/audit-export"),
|
||||
updateAuditExport: (data: Partial<AuditExportUpdate>) => request<AuditExportSettings>("/settings/audit-export", { method: "PUT", body: JSON.stringify(data) }),
|
||||
testAuditExport: () => request<{ success: boolean }>("/settings/audit-export/test", { method: "POST" }),
|
||||
notificationChannels: () => request<NotificationChannel[]>("/notifications/channels"),
|
||||
createNotificationChannel: (data: NotificationChannelCreate) => request<NotificationChannel>("/notifications/channels", { method: "POST", body: JSON.stringify(data) }),
|
||||
updateNotificationChannel: (id: string, data: NotificationChannelUpdate) => request<NotificationChannel>(`/notifications/channels/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
||||
deleteNotificationChannel: (id: string) => request<void>(`/notifications/channels/${id}`, { method: "DELETE" }),
|
||||
testNotificationChannel: (id: string) => request<{ success: boolean }>(`/notifications/channels/${id}/test`, { method: "POST" }),
|
||||
hostGateways: (id: string) => request<GatewayHealth[]>(`/hosts/${id}/gateways`),
|
||||
testNotification: (channel: "webhook" | "email") => request<{ success: boolean }>("/notifications/test", { method: "POST", body: JSON.stringify({ channel }) }),
|
||||
notificationProblems: (includeResolved = false) => request<NotificationProblem[]>(`/notifications/problems?include_resolved=${includeResolved}`),
|
||||
carpEvents: (clusterName?: string) => request<CarpEvent[]>(`/cluster/carp-events${clusterName ? `?cluster_name=${encodeURIComponent(clusterName)}` : ""}`),
|
||||
@@ -1283,8 +1312,32 @@ export const api = {
|
||||
stopService: (id:string, name:string) => request<{result:string}>(`/hosts/${id}/services/${encodeURIComponent(name)}/stop`,{method:"POST"}),
|
||||
restartService: (id:string, name:string) => request<{result:string}>(`/hosts/${id}/services/${encodeURIComponent(name)}/restart`,{method:"POST"}),
|
||||
listScheduledMaintenance: (id:string) => request<ScheduledMaintenanceItem[]>(`/hosts/${id}/maintenance/schedule`),
|
||||
createScheduledMaintenance: (id:string, action:"reboot"|"update"|"major_update", run_at:string, carp_handover=false) => request<ScheduledMaintenanceItem>(`/hosts/${id}/maintenance/schedule`,{method:"POST",body:JSON.stringify({action,run_at,acknowledge_unattended:true,carp_handover})}),
|
||||
createScheduledMaintenance: (id:string, action:"reboot"|"update"|"major_update", run_at:string, carp_handover=false, auto_verify=false) => request<ScheduledMaintenanceItem>(`/hosts/${id}/maintenance/schedule`,{method:"POST",body:JSON.stringify({action,run_at,acknowledge_unattended:true,carp_handover,auto_verify})}),
|
||||
cancelScheduledMaintenance: (id:string, scheduleId:string) => request<ScheduledMaintenanceItem>(`/hosts/${id}/maintenance/schedule/${scheduleId}`,{method:"DELETE"}),
|
||||
// Maintenance windows and firmware rollouts (1.25)
|
||||
listMaintenanceWindows: () => request<MaintenanceWindow[]>("/maintenance-windows"),
|
||||
createMaintenanceWindow: (data: MaintenanceWindowWrite) => request<MaintenanceWindow>("/maintenance-windows", { method: "POST", body: JSON.stringify(data) }),
|
||||
updateMaintenanceWindow: (id: string, data: MaintenanceWindowWrite) => request<MaintenanceWindow>(`/maintenance-windows/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
||||
deleteMaintenanceWindow: (id: string) => request<void>(`/maintenance-windows/${id}`, { method: "DELETE" }),
|
||||
addWindowException: (id: string, data: { occurrence_date: string; kind: "skip" | "move"; moved_start?: string; reason?: string }) => request<MaintenanceWindow>(`/maintenance-windows/${id}/exceptions`, { method: "POST", body: JSON.stringify(data) }),
|
||||
deleteWindowException: (id: string, exceptionId: string) => request<MaintenanceWindow>(`/maintenance-windows/${id}/exceptions/${exceptionId}`, { method: "DELETE" }),
|
||||
listRolloutPlans: () => request<RolloutPlan[]>("/rollouts/plans"),
|
||||
getRolloutPlan: (id: string) => request<RolloutPlan>(`/rollouts/plans/${id}`),
|
||||
createRolloutPlan: (data: RolloutPlanWrite) => request<RolloutPlan>("/rollouts/plans", { method: "POST", body: JSON.stringify(data) }),
|
||||
updateRolloutPlan: (id: string, data: RolloutPlanWrite) => request<RolloutPlan>(`/rollouts/plans/${id}`, { method: "PUT", body: JSON.stringify(data) }),
|
||||
deleteRolloutPlan: (id: string) => request<void>(`/rollouts/plans/${id}`, { method: "DELETE" }),
|
||||
previewRolloutPlan: (id: string) => request<RolloutPreview>(`/rollouts/plans/${id}/preview`, { method: "POST" }),
|
||||
activateRolloutPlan: (id: string, preview_token: string) => request<RolloutPlan>(`/rollouts/plans/${id}/activate`, { method: "POST", body: JSON.stringify({ preview_token }) }),
|
||||
deactivateRolloutPlan: (id: string) => request<RolloutPlan>(`/rollouts/plans/${id}/deactivate`, { method: "POST" }),
|
||||
startRolloutPlan: (id: string, preview_token: string, confirm_name: string) => request<RolloutRunDetail>(`/rollouts/plans/${id}/start`, { method: "POST", body: JSON.stringify({ preview_token, confirm_name, acknowledge_unattended: true }) }),
|
||||
listRolloutRuns: () => request<RolloutRunSummary[]>("/rollouts/runs"),
|
||||
rolloutStatus: () => request<RolloutStatus>("/rollouts/status"),
|
||||
getRolloutRun: (id: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}`),
|
||||
pauseRolloutRun: (id: string, reason: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/pause`, { method: "POST", body: JSON.stringify({ reason }) }),
|
||||
resumeRolloutRun: (id: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/resume`, { method: "POST" }),
|
||||
approveRolloutWave: (id: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/approve`, { method: "POST" }),
|
||||
skipRolloutHost: (id: string, host_id: string, reason: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/skip-host`, { method: "POST", body: JSON.stringify({ host_id, reason }) }),
|
||||
abortRolloutRun: (id: string, reason: string, confirm_name: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/abort`, { method: "POST", body: JSON.stringify({ reason, confirm_name }) }),
|
||||
audit: (filters: Record<string,string | number> = {}) => request<AuditPage>(`/audit?${new URLSearchParams(Object.entries(filters).filter(([,v])=>v!=="").map(([k,v])=>[k,String(v)]))}`),
|
||||
dashboard: () => request<DashboardSummary>("/dashboard"),
|
||||
versionCheck: () => request<VersionCheck>("/system/version-check"),
|
||||
@@ -1396,6 +1449,9 @@ export const api = {
|
||||
listRules: () => request<Rule[]>("/rules"),
|
||||
createRule: (data: RuleCreate) => request<Rule>("/rules", { method: "POST", body: JSON.stringify(data) }),
|
||||
listUnmanagedRules: (hostId: string) => request<UnmanagedRule[]>(`/rules/unmanaged?host_id=${hostId}`),
|
||||
listUnmanagedObjects: (kind: AdoptKind, hostId: string) => request<UnmanagedObject[]>(`/${kind}/unmanaged?host_id=${hostId}`),
|
||||
adoptObject: (kind: AdoptKind, hostId: string, uuid: string) => request<{ id: string }>(`/${kind}/adopt`, { method: "POST", body: JSON.stringify({ host_id: hostId, uuid }) }),
|
||||
adoptAllObjects: (kind: AdoptKind, hostId: string) => request<AdoptAllResult>(`/${kind}/adopt-all`, { method: "POST", body: JSON.stringify({ host_id: hostId }) }),
|
||||
adoptRule: (hostId: string, ruleUuid: string) => request<Rule>("/rules/adopt", { method: "POST", body: JSON.stringify({ host_id: hostId, rule_uuid: ruleUuid }) }),
|
||||
updateRule: (id: string, data: RuleCreate, expectedUpdatedAt: string) => request<Rule>(`/rules/${id}`, { method: "PUT", body: JSON.stringify({ ...data, expected_updated_at: expectedUpdatedAt }) }),
|
||||
previewRule: (id: string) => request<RulePreview>(`/rules/${id}/preview`),
|
||||
@@ -1678,7 +1734,9 @@ export interface OutboundMode { host_id: string; host_name: string; mode: string
|
||||
export interface AuditEvent { id: string; created_at: string; correlation_id: string; actor: string; kind: string; action: string; status: string; entity_id: string | null; entity_name: string | null; host_id: string | null; host_name: string | null; message: string | null; details: Record<string,unknown>; }
|
||||
export interface AuditPage { total: number; offset: number; limit: number; items: AuditEvent[]; }
|
||||
|
||||
export interface DashboardHost { id: string; name: string; status: HostStatus; carp_status: string | null; cluster_name: string | null; }
|
||||
export type GatewayState = "online" | "degraded" | "offline" | "forced_down" | "unknown";
|
||||
export interface GatewayHealth { name: string; address: string | null; status: string; state: GatewayState; loss: number | null; delay: number | null; stddev: number | null; monitor: string | null; }
|
||||
export interface DashboardHost { id: string; name: string; status: HostStatus; carp_status: string | null; cluster_name: string | null; gateways: GatewayHealth[] | null; }
|
||||
export interface PendingAction { kind: string; status: string; entity_name: string; host_name: string; message: string | null; created_at: string; link: string; }
|
||||
export interface ActivityItem { id: string; created_at: string; actor: string; kind: string; action: string; status: string; entity_name: string | null; host_name: string | null; message: string | null; }
|
||||
export interface DashboardSummary { hosts: DashboardHost[]; pending_actions: PendingAction[]; recent_activity: ActivityItem[]; }
|
||||
@@ -1713,7 +1771,62 @@ export interface MaintenanceState {
|
||||
export interface HostOperation { id:string; host_id:string; action:"reboot"|"update"|"major_update"|"carp_enter"|"carp_leave"; status:string; created_at:string; expires_at:string; actor:string; message:string|null; }
|
||||
export interface OpnsenseService { id:string; name:string; description:string; locked:boolean|number; running:boolean|number; }
|
||||
export interface HostResourceSample { id:string; sampled_at:string; mem_used_pct:number|null; load_avg_1m:number|null; disk_used_pct:number|null; raw:Record<string,unknown>; }
|
||||
export interface ScheduledMaintenanceItem { id:string; host_id:string; action:"reboot"|"update"|"major_update"; run_at:string; status:"pending"|"running"|"succeeded"|"failed"|"cancelled"; actor:string; created_at:string; result:{error?:string;response?:unknown;handover?:boolean;version_before?:string|null;version_after?:string|null;reboot_required?:boolean}; host_operation_id:string|null; carp_handover:boolean; phase:"handover_wait"|"job_wait"|"leave"|"leave_wait"|null; }
|
||||
export interface ScheduledMaintenanceItem { id:string; host_id:string; action:"reboot"|"update"|"major_update"; run_at:string; status:"pending"|"running"|"succeeded"|"failed"|"cancelled"; actor:string; created_at:string; result:{error?:string;response?:unknown;handover?:boolean;auto_verified?:boolean;version_before?:string|null;version_after?:string|null;reboot_required?:boolean}; host_operation_id:string|null; carp_handover:boolean; phase:"handover_wait"|"job_wait"|"verify"|"leave"|"leave_wait"|null; auto_verify?:boolean; origin?:"manual"|"rollout"; rollout_run_id?:string|null; }
|
||||
|
||||
// ── Maintenance windows and firmware rollouts (1.25) ──
|
||||
export interface Occurrence { local_date: string; start: string; end: string; cutoff: string; moved: boolean; skipped: boolean; }
|
||||
export interface MaintenanceWindowWrite {
|
||||
name: string; timezone: string; recurrence: "weekly" | "monthly"; weekdays: number[];
|
||||
month_mode: "nth_weekday" | "day" | null; month_nth: number | null; month_weekday: number | null; month_day: number | null;
|
||||
start_time: string; duration_minutes: number; start_cutoff_minutes: number; enabled: boolean;
|
||||
}
|
||||
export interface MaintenanceWindow extends MaintenanceWindowWrite {
|
||||
id: string; created_by: string; created_at: string; open_now: boolean; next: Occurrence[];
|
||||
exceptions: { id: string; occurrence_date: string; kind: "skip" | "move"; moved_start: string | null; reason: string; actor: string; created_at: string }[];
|
||||
used_by: { id: string; name: string; auto_start: boolean; active: boolean }[];
|
||||
}
|
||||
export type RolloutCheck = "online" | "version" | "carp" | "gateways" | "services" | "locks" | "compliance";
|
||||
export interface RolloutIssue { code: string; params: Record<string, string | number | boolean | null>; }
|
||||
export interface RolloutWaveWrite { label: string; group_ids: string[]; soak_minutes: number; carp_handover: boolean; require_approval: boolean; }
|
||||
export interface RolloutPlanWrite {
|
||||
name: string; description: string; action: "update" | "major_update"; window_id: string | null; auto_start: boolean;
|
||||
gate_checks: Partial<Record<RolloutCheck, boolean>>; compliance_mode: "warn" | "fail"; waves: RolloutWaveWrite[];
|
||||
}
|
||||
export interface RolloutPlan extends Omit<RolloutPlanWrite, "waves"> {
|
||||
id: string;
|
||||
waves: (RolloutWaveWrite & { id: string; position: number; groups: string[]; host_count: number })[];
|
||||
active: boolean; activated_by: string | null; activated_at: string | null; fingerprint_ok: boolean;
|
||||
auto_start_error: { occurrence: string; at: string; code: string; params?: Record<string, string> } | null;
|
||||
window: { id: string; name: string; next: Occurrence[] } | null; errors: RolloutIssue[]; warnings: RolloutIssue[];
|
||||
host_count: number; last_run: { id: string; status: RolloutRunStatus; created_at: string } | null;
|
||||
created_by: string; created_at: string; updated_at: string;
|
||||
}
|
||||
export interface RolloutPreview {
|
||||
waves: { index: number; label: string; soak_minutes: number; carp_handover: boolean; require_approval: boolean; groups: string[];
|
||||
hosts: { id: string; name: string; cluster_name: string | null; enabled: boolean }[] }[];
|
||||
errors: RolloutIssue[]; warnings: RolloutIssue[]; window: RolloutPlan["window"]; preview_token: string; can_start: boolean; plan: RolloutPlan;
|
||||
}
|
||||
export type RolloutRunStatus = "waiting_window" | "preflight" | "wave_running" | "soaking" | "gate" | "awaiting_approval" | "resuming" | "paused" | "succeeded" | "cancelled";
|
||||
export type RolloutHostStatus = "queued" | "running" | "verifying" | "succeeded" | "skipped" | "failed" | "blocked" | "cancelled";
|
||||
export interface RolloutCheckResult { result: "ok" | "warn" | "fail" | "skip"; code: string; params: Record<string, string | number | null>; }
|
||||
export interface RolloutGate { hosts: Record<string, Partial<Record<RolloutCheck, RolloutCheckResult>>>; verdict: "ok" | "warn" | "fail"; }
|
||||
export interface RolloutRunSummary {
|
||||
id: string; plan_id: string | null; plan_name: string; action: "update" | "major_update"; status: RolloutRunStatus; paused_from: string | null;
|
||||
current_wave: number; wave_count: number; current_wave_label: string | null; target_version: string | null;
|
||||
pause_reason: string | null; pause_detail: Record<string, unknown> | null; started_by: string; trigger: "manual" | "window";
|
||||
window_occurrence: string | null; created_at: string; finished_at: string | null; step_at: string | null;
|
||||
host_counts: Partial<Record<RolloutHostStatus, number>>; active: boolean;
|
||||
}
|
||||
export interface RolloutRunDetail extends RolloutRunSummary {
|
||||
soak_until: string | null; progress: { gate_recheck_at?: string | null; cluster_recheck_at?: string | null; blocked_since?: string | null; resume_gate?: RolloutGate; cluster_gate?: RolloutGate; resume_recheck_at?: string | null };
|
||||
waves: { index: number; label: string; status: string; gate_result: RolloutGate | null; gate_attempts: number; soak_until: string | null;
|
||||
approved_by: string | null; started_at: string | null; finished_at: string | null;
|
||||
spec: { label: string; soak_minutes: number; carp_handover: boolean; require_approval: boolean; groups: string[] } | null }[];
|
||||
hosts: { id: string; host_id: string | null; host_name: string; cluster_name: string | null; wave_index: number; order: number; status: RolloutHostStatus;
|
||||
version_before: string | null; version_after: string | null; detail: RolloutIssue | null; started_at: string | null; finished_at: string | null;
|
||||
schedule: { id: string; status: string; phase: string | null; carp_handover: boolean; host_operation_id: string | null } | null }[];
|
||||
}
|
||||
export interface RolloutStatus { active: RolloutRunSummary | null; host_ids: string[]; current_host_ids: string[]; }
|
||||
export interface MaintenanceRead {state:MaintenanceState|null;error:string|null;operations:HostOperation[];}
|
||||
export interface MaintenancePreview {preview_id:string;expires_at:string;action:"reboot"|"update"|"major_update";state:MaintenanceState;}
|
||||
export interface CarpSwitchState {
|
||||
@@ -1761,10 +1874,37 @@ export interface NotificationSettings {
|
||||
scheduled_maintenance_failed_enabled: boolean;
|
||||
backup_failed_enabled: boolean;
|
||||
compliance_alerts_enabled: boolean;
|
||||
gateway_down_enabled: boolean;
|
||||
resource_threshold_enabled: boolean;
|
||||
service_down_enabled: boolean;
|
||||
rollout_problems_enabled: boolean;
|
||||
gateway_loss_threshold_pct: number;
|
||||
gateway_delay_threshold_ms: number;
|
||||
disk_threshold_pct: number;
|
||||
memory_threshold_pct: number;
|
||||
swap_threshold_pct: number;
|
||||
load_per_cpu_threshold: number;
|
||||
alert_confirm_samples: number;
|
||||
resource_confirm_samples: number;
|
||||
service_watchlist: string[];
|
||||
poll_interval_seconds: number;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export type ChannelType = "ntfy" | "gotify" | "telegram" | "teams" | "matrix";
|
||||
export interface NotificationChannel {
|
||||
id: string;
|
||||
name: string;
|
||||
channel_type: ChannelType;
|
||||
enabled: boolean;
|
||||
config: Record<string, string | number>;
|
||||
secret_set: boolean;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
export interface NotificationChannelCreate { name: string; channel_type: ChannelType; enabled: boolean; config: Record<string, string | number>; secret?: string; }
|
||||
export interface NotificationChannelUpdate { name?: string; enabled?: boolean; config?: Record<string, string | number>; secret?: string; clear_secret?: boolean; }
|
||||
|
||||
export interface NotificationSettingsUpdate {
|
||||
webhook_enabled?: boolean;
|
||||
webhook_url?: string;
|
||||
@@ -1791,6 +1931,19 @@ export interface NotificationSettingsUpdate {
|
||||
scheduled_maintenance_failed_enabled?: boolean;
|
||||
backup_failed_enabled?: boolean;
|
||||
compliance_alerts_enabled?: boolean;
|
||||
gateway_down_enabled?: boolean;
|
||||
resource_threshold_enabled?: boolean;
|
||||
service_down_enabled?: boolean;
|
||||
rollout_problems_enabled?: boolean;
|
||||
gateway_loss_threshold_pct?: number;
|
||||
gateway_delay_threshold_ms?: number;
|
||||
disk_threshold_pct?: number;
|
||||
memory_threshold_pct?: number;
|
||||
swap_threshold_pct?: number;
|
||||
load_per_cpu_threshold?: number;
|
||||
alert_confirm_samples?: number;
|
||||
resource_confirm_samples?: number;
|
||||
service_watchlist?: string[];
|
||||
poll_interval_seconds?: number;
|
||||
}
|
||||
|
||||
@@ -1817,7 +1970,9 @@ export type ProblemType =
|
||||
| "scheduled_maintenance_failed"
|
||||
| "backup_stale"
|
||||
| "backup_offsite_failed"
|
||||
| "compliance_finding";
|
||||
| "compliance_finding"
|
||||
| "gateway_down" | "resource_threshold" | "service_down"
|
||||
| "rollout_paused" | "rollout_start_failed";
|
||||
|
||||
export interface CarpEvent {
|
||||
id: string;
|
||||
|
||||
+560
-8
@@ -15,7 +15,8 @@
|
||||
"saving": "Speichern…",
|
||||
"deleteFailed": "Löschen fehlgeschlagen",
|
||||
"description": "Beschreibung",
|
||||
"versionLabel": "OPNexus {version} „{codename}“"
|
||||
"versionLabel": "OPNexus {version} „{codename}“",
|
||||
"delete": "Löschen"
|
||||
},
|
||||
"settings": {
|
||||
"language": {
|
||||
@@ -66,10 +67,28 @@
|
||||
"scheduledFailed": "Geplante Wartung nicht ausgeführt",
|
||||
"backupFailed": "Kein aktuelles Backup / Offsite-Kopie fehlgeschlagen",
|
||||
"complianceAlerts": "Kritischer oder hoher Sicherheitsbefund (standardmäßig aus)",
|
||||
"networkDrift": "Netzwerk-Abweichung im Cluster"
|
||||
"networkDrift": "Netzwerk-Abweichung im Cluster",
|
||||
"gatewayDown": "Gateway gestört (offline, Paketverlust oder Latenz über der Schwelle)",
|
||||
"resourceThreshold": "Ressourcen-Schwelle überschritten (Festplatte, Speicher, Swap, CPU-Last; standardmäßig aus)",
|
||||
"serviceDown": "Überwachter Dienst ausgefallen (standardmäßig aus)",
|
||||
"rolloutProblems": "Rollout angehalten oder nicht automatisch gestartet"
|
||||
},
|
||||
"pollInterval": "Prüfintervall (Sekunden)",
|
||||
"pollHint": "Mindestens 60s, um die Firewalls nicht unnötig zu belasten."
|
||||
"pollHint": "Mindestens 60s, um die Firewalls nicht unnötig zu belasten.",
|
||||
"monitoring": "Schwellen und Entprellung",
|
||||
"monitoringHint": "Alarme für Gateways, Ressourcen und Dienste stützen sich auf die Messungen, die OPNexus jede Minute aufzeichnet. Ein Alarm entsteht erst, wenn die letzten Messungen alle schlecht sind, und endet erst, wenn die letzten Messungen alle wieder gut sind – Einzelspitzen und flackernde Verbindungen lösen keine Meldungen aus. 0 schaltet eine einzelne Schwelle ab.",
|
||||
"limit": {
|
||||
"gatewayLoss": "Gateway: Paketverlust ab (%)",
|
||||
"gatewayDelay": "Gateway: Latenz ab (ms)",
|
||||
"diskLimit": "Festplatte ab (% belegt)",
|
||||
"memoryLimit": "Arbeitsspeicher ab (% belegt)",
|
||||
"swapLimit": "Swap ab (% belegt)",
|
||||
"loadLimit": "CPU-Last ab (Last je Thread)",
|
||||
"confirmSamples": "Gateways und Dienste: Messungen in Folge",
|
||||
"resourceSamples": "Ressourcen: Messungen in Folge",
|
||||
"watchlist": "Überwachte Dienste",
|
||||
"watchlistHint": "OPNsense-Dienstnamen, kommagetrennt. OPNexus meldet einen Dienst nur, wenn er zuvor lief und jetzt nicht mehr. Absichtlich gestoppte Dienste hier entfernen."
|
||||
}
|
||||
},
|
||||
"logs": {
|
||||
"saved": "Log-Ingest-Einstellungen gespeichert.",
|
||||
@@ -180,6 +199,51 @@
|
||||
"title": "Einrichtungsassistent",
|
||||
"intro": "Führt Schritt für Schritt durch Passwort, Zwei-Faktor-Anmeldung und das Verbinden einer Firewall.",
|
||||
"open": "Assistent öffnen"
|
||||
},
|
||||
"channels": {
|
||||
"title": "Weitere Kanäle",
|
||||
"hint": "ntfy, Gotify, Telegram, Microsoft Teams und Matrix erhalten dieselben Meldungen wie Webhook und E-Mail. Zugangsdaten werden getrennt von der Datenbank gespeichert und nie angezeigt.",
|
||||
"add": "Kanal hinzufügen",
|
||||
"edit": "Kanal bearbeiten",
|
||||
"type": "Typ",
|
||||
"name": "Name",
|
||||
"enabled": "Aktiv",
|
||||
"disabled": "Aus",
|
||||
"enable": "Einschalten",
|
||||
"disable": "Ausschalten",
|
||||
"test": "Testen",
|
||||
"editAction": "Bearbeiten",
|
||||
"deleteAction": "Löschen",
|
||||
"empty": "Noch keine weiteren Kanäle eingerichtet",
|
||||
"saved": "Kanal gespeichert.",
|
||||
"deleted": "Kanal „{name}“ gelöscht.",
|
||||
"confirmDelete": "Kanal „{name}“ wirklich löschen?",
|
||||
"testSent": "Testnachricht über „{name}“ gesendet.",
|
||||
"loadFailed": "Kanäle konnten nicht geladen werden",
|
||||
"removeSecret": "Zugriffstoken entfernen",
|
||||
"types": {
|
||||
"ntfy": "ntfy",
|
||||
"gotify": "Gotify",
|
||||
"telegram": "Telegram",
|
||||
"teams": "Microsoft Teams",
|
||||
"matrix": "Matrix"
|
||||
},
|
||||
"field": {
|
||||
"server_url": "Serveradresse",
|
||||
"topic": "Thema",
|
||||
"priority": "Priorität",
|
||||
"chat_id": "Chat-ID",
|
||||
"api_base": "API-Adresse (optional)",
|
||||
"homeserver_url": "Homeserver-Adresse",
|
||||
"room_id": "Raum-ID"
|
||||
},
|
||||
"secret": {
|
||||
"ntfy": "Zugriffstoken (optional)",
|
||||
"gotify": "App-Token",
|
||||
"telegram": "Bot-Token",
|
||||
"teams": "Webhook-Adresse des Workflows",
|
||||
"matrix": "Zugriffstoken"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nav": {
|
||||
@@ -235,7 +299,8 @@
|
||||
"hostsystem-syslog": "Syslog",
|
||||
"hostsystem-snapshots": "Snapshots",
|
||||
"hostsystem-diagnostics": "Diagnose",
|
||||
"setup": "Einrichtung"
|
||||
"setup": "Einrichtung",
|
||||
"rollouts": "Rollouts"
|
||||
}
|
||||
},
|
||||
"dashboard": {
|
||||
@@ -297,7 +362,15 @@
|
||||
"majorAvailable": "Major-Upgrade verfügbar",
|
||||
"stableAvailable": "Stabiles Update verfügbar",
|
||||
"updateBadge": "Update",
|
||||
"loadFailed": "Dashboard konnte nicht geladen werden"
|
||||
"loadFailed": "Dashboard konnte nicht geladen werden",
|
||||
"gateways": "Gateways",
|
||||
"noGateways": "Keine Gateway-Daten",
|
||||
"noGatewaysDescription": "Sobald die Messungen der Firewalls vorliegen (jede Minute), erscheinen hier Status, Verlust und Latenz.",
|
||||
"alert": {
|
||||
"gateway_down": "Gateway",
|
||||
"resource_threshold": "Ressource",
|
||||
"service_down": "Dienst"
|
||||
}
|
||||
},
|
||||
"hosts": {
|
||||
"add": "Host hinzufügen",
|
||||
@@ -435,6 +508,67 @@
|
||||
},
|
||||
"logPackets": "Protokollieren"
|
||||
},
|
||||
"adopt": {
|
||||
"button": "Von Firewall übernehmen",
|
||||
"title": {
|
||||
"aliases": "Aliase von der Firewall übernehmen",
|
||||
"portForward": "Port-Forwards von der Firewall übernehmen",
|
||||
"oneToOne": "1:1-NAT-Regeln von der Firewall übernehmen",
|
||||
"outboundNat": "Outbound-NAT-Regeln von der Firewall übernehmen"
|
||||
},
|
||||
"intro": "Hier stehen Objekte, die direkt in OPNsense angelegt wurden. Nach der Übernahme verwaltest du sie in OPNexus. Die Übernahme selbst ändert nichts auf der Firewall.",
|
||||
"host": "Firewall",
|
||||
"none": "Nichts Übernehmbares gefunden. Alles ist bereits verwaltet.",
|
||||
"action": "Übernehmen",
|
||||
"all": "Alle übernehmbaren übernehmen ({count})",
|
||||
"adopted": "„{name}“ übernommen.",
|
||||
"allDone": "{count} übernommen, {skipped} übersprungen.",
|
||||
"failed": "Übernahme fehlgeschlagen",
|
||||
"loadFailed": "Objekte konnten nicht geladen werden.",
|
||||
"notPossible": "Nicht übernehmbar",
|
||||
"bulkOnly": "Nur mit „Alle übernehmbaren“",
|
||||
"entries": "Einträge: {count}",
|
||||
"towards": "Ziel {destination}",
|
||||
"interfaceAddress": "Adresse der Schnittstelle",
|
||||
"systemNote": "{count} OPNsense-eigene Aliase (z. B. bogons, Schnittstellen-Netze) werden nie übernommen.",
|
||||
"kept": "Bleibt auf der Firewall unverändert: {fields}",
|
||||
"keptFields": {
|
||||
"log": "Protokollieren",
|
||||
"poolopts": "Pool-Optionen",
|
||||
"poolopts_sourcehashkey": "Source-Hash-Schlüssel",
|
||||
"tag": "Tag",
|
||||
"tagged": "Tagged",
|
||||
"nosync": "Nicht synchronisieren",
|
||||
"categories": "Kategorien",
|
||||
"staticnatport": "Statischer NAT-Port",
|
||||
"endpoint-independent": "Endpoint-Independent"
|
||||
},
|
||||
"reasons": {
|
||||
"system": "OPNsense-eigenes Objekt",
|
||||
"already_managed": "Wird bereits verwaltet",
|
||||
"unsupported_type": "Typ „{detail}“ unterstützt OPNexus nicht",
|
||||
"name_invalid": "Der Name enthält Zeichen, die OPNexus nicht zulässt (nur Buchstaben, Ziffern, _)",
|
||||
"name_exists": "In OPNexus gibt es schon einen Alias mit diesem Namen",
|
||||
"content_too_large": "Inhalt zu groß (über 8000 Zeichen)",
|
||||
"proto_set": "Hat eine IPv4/IPv6-Einschränkung, die OPNexus nicht abbildet",
|
||||
"value_too_long": "Ein Wert ist länger, als OPNexus speichert",
|
||||
"references_unmanaged_alias": "Verweist auf nicht übernommene Aliase: {detail}",
|
||||
"interface_not_single": "Mehrere oder keine Schnittstelle",
|
||||
"ipprotocol_unsupported": "IP-Version nicht abbildbar (nur IPv4 oder IPv6)",
|
||||
"protocol_any": "Protokoll „beliebig“ wird nicht unterstützt",
|
||||
"protocol_missing": "Protokoll fehlt",
|
||||
"no_redirect": "„Kein Redirect“-Regel (No RDR)",
|
||||
"no_target": "Kein Ziel gesetzt",
|
||||
"filter_rule_linked": "Legt eine zugehörige Filterregel an oder ist mit einer verknüpft",
|
||||
"negated_source": "Quelle ist invertiert (NOT)",
|
||||
"negated_destination": "Ziel ist invertiert (NOT)",
|
||||
"natreflection_unsupported": "NAT-Reflection-Wert wird nicht unterstützt",
|
||||
"type_unsupported": "Regeltyp wird nicht unterstützt",
|
||||
"incomplete": "Pflichtangaben fehlen",
|
||||
"no_nat_rule": "„Kein NAT“-Regel (nonat) wird nicht unterstützt",
|
||||
"no_description": "Beschreibung fehlt – bitte in OPNsense setzen"
|
||||
}
|
||||
},
|
||||
"login": {
|
||||
"prompt": "Bitte anmelden",
|
||||
"username": "Benutzername",
|
||||
@@ -596,6 +730,9 @@
|
||||
"TOTP_NOT_ENABLED": "Zwei-Faktor-Anmeldung ist nicht aktiv",
|
||||
"TOTP_SETUP_MISSING": "Es wurde keine Einrichtung gestartet",
|
||||
"RULE_NOT_ADOPTABLE": "Diese Regel kann nicht übernommen werden ({reasons})",
|
||||
"OBJECT_NOT_ADOPTABLE": "Dieses Objekt kann nicht übernommen werden ({reasons})",
|
||||
"ADOPT_OBJECT_NOT_FOUND": "Das Objekt wurde auf der Firewall nicht gefunden. Liste neu laden.",
|
||||
"OUTBOUND_MODE_BLOCKS_ADOPT": "{host}: Im Outbound-NAT-Modus „{mode}“ zeigt OPNsense keine manuellen Regeln an. Modus in OPNsense selbst auf Hybrid/Manuell umstellen, dann lassen sie sich übernehmen. OPNexus ändert diesen Modus nicht.",
|
||||
"NETWORK_ACK_REQUIRED": "Diese Änderung kann die Verbindung zur Firewall trennen und muss ausdrücklich bestätigt werden ({reasons})",
|
||||
"NETWORK_APPLY_UNCERTAIN": "Die Änderung wurde geschrieben, das Anwenden auf „{host}“ ist aber fehlgeschlagen oder unklar. Stand auf der Firewall prüfen; Backup: {backup_id}",
|
||||
"NETWORK_BACKUP_FAILED": "Backup von „{host}“ vor der Änderung fehlgeschlagen – es wurde nichts geändert",
|
||||
@@ -624,7 +761,33 @@
|
||||
"OFFSITE_TEST_FAILED": "Der Test des Offsite-Ziels ist fehlgeschlagen: {reason}",
|
||||
"API_TOKEN_NOT_FOUND": "API-Token nicht gefunden",
|
||||
"API_TOKEN_SCOPE_FORBIDDEN": "Dieses API-Token (Bereich „{scope}“) darf diese Aktion nicht ausführen.",
|
||||
"API_TOKEN_SCOPE_TOO_HIGH": "Mit deiner Rolle ist dieser Token-Bereich nicht möglich (Nur-Lesen: „Lesen“, Operator: bis „Ausrollen“)."
|
||||
"API_TOKEN_SCOPE_TOO_HIGH": "Mit deiner Rolle ist dieser Token-Bereich nicht möglich (Nur-Lesen: „Lesen“, Operator: bis „Ausrollen“).",
|
||||
"NOTIFICATION_CHANNEL_INVALID": "Ungültige Angabe: {reason}",
|
||||
"NOTIFICATION_CHANNEL_NOT_FOUND": "Der Benachrichtigungskanal wurde nicht gefunden.",
|
||||
"NOTIFICATION_CHANNEL_NAME_TAKEN": "Ein Kanal mit diesem Namen existiert bereits.",
|
||||
"NOTIFICATION_CHANNEL_TEST_FAILED": "Testnachricht nicht zugestellt: {reason}",
|
||||
"NOTIFICATION_WEBHOOK_URL_INVALID": "Ungültige Webhook-Adresse: {reason}",
|
||||
"WINDOW_NOT_FOUND": "Wartungsfenster nicht gefunden.",
|
||||
"WINDOW_INVALID": "Wartungsfenster ungültig (Feld: {field}).",
|
||||
"WINDOW_NAME_TAKEN": "Ein Wartungsfenster mit diesem Namen gibt es bereits.",
|
||||
"WINDOW_IN_USE": "Das Fenster wird von einem Rollout-Plan verwendet. Erst den Plan ändern oder löschen.",
|
||||
"WINDOW_EXCEPTION_INVALID": "Ausnahme ungültig (Feld: {field}). Nur künftige Termine des Fensters lassen sich auslassen oder verschieben.",
|
||||
"WINDOW_EXCEPTION_EXISTS": "Für diesen Termin gibt es schon eine Ausnahme.",
|
||||
"WINDOW_EXCEPTION_NOT_FOUND": "Ausnahme nicht gefunden.",
|
||||
"ROLLOUT_PLAN_NOT_FOUND": "Rollout-Plan nicht gefunden.",
|
||||
"ROLLOUT_INVALID": "Rollout-Plan ungültig (Feld: {field}).",
|
||||
"ROLLOUT_NAME_TAKEN": "Ein Rollout-Plan mit diesem Namen gibt es bereits.",
|
||||
"ROLLOUT_PLAN_INVALID": "Der Plan kann so nicht starten: {reason}. Vorschau prüfen.",
|
||||
"ROLLOUT_PREVIEW_STALE": "Plan, Gruppen oder Fenster haben sich seit der Vorschau geändert. Vorschau neu laden.",
|
||||
"ROLLOUT_PLAN_BUSY": "Für diesen Plan läuft gerade ein Rollout. Erst abschließen oder abbrechen.",
|
||||
"ROLLOUT_GROUP_IN_USE": "Die Gruppe ist eine Welle eines Rollout-Plans. Erst den Plan ändern.",
|
||||
"ROLLOUT_RUN_NOT_FOUND": "Rollout-Lauf nicht gefunden.",
|
||||
"ROLLOUT_ALREADY_ACTIVE": "Es läuft bereits ein Rollout ({plan}). Es läuft immer nur einer gleichzeitig.",
|
||||
"ROLLOUT_CONFIRM_NAME": "Zur Bestätigung den Namen exakt eingeben.",
|
||||
"ROLLOUT_ACTION_NOT_ALLOWED": "Im Zustand „{status}“ nicht möglich.",
|
||||
"ROLLOUT_HOST_NOT_IN_RUN": "Dieser Host gehört nicht zu diesem Rollout.",
|
||||
"ROLLOUT_HOST_RUNNING": "Auf {host} läuft die Aktion bereits; sie wird geführt zu Ende gebracht und kann nicht übersprungen werden.",
|
||||
"ROLLOUT_HOST_FAILED": "Fehlgeschlagen: {hosts}. Erst prüfen und den Host überspringen, dann fortsetzen."
|
||||
},
|
||||
"workflow": {
|
||||
"attention": "Fehler prüfen",
|
||||
@@ -962,7 +1125,18 @@
|
||||
"handoverOption": "Vorher automatisch auf den Partner umschalten und danach zurückschalten (CARP)",
|
||||
"handoverHint": "Ablauf: Wartungsmodus auf diesem Host, Partner wird MASTER (wird geprüft), dann läuft die Aktion, nach Rückkehr des Hosts wird der Wartungsmodus wieder verlassen. Bei Problemen bleibt der Partner MASTER und es kommt eine Benachrichtigung. Beim Zurückschalten kann der Host je nach CARP-Einstellung die MASTER-Rolle zurückholen (kurzer zweiter Wechsel).",
|
||||
"handoverBadge": "mit CARP-Umschaltung",
|
||||
"phase": {"handover_wait": "wartet auf Übernahme durch den Partner", "job_wait": "Auftrag läuft", "leave": "schaltet zurück", "leave_wait": "prüft den Normalbetrieb"}
|
||||
"phase": {
|
||||
"handover_wait": "wartet auf Übernahme durch den Partner",
|
||||
"job_wait": "Auftrag läuft",
|
||||
"leave": "schaltet zurück",
|
||||
"leave_wait": "prüft den Normalbetrieb",
|
||||
"verify": "prüft das Ergebnis"
|
||||
},
|
||||
"autoVerifyOption": "Ergebnis automatisch prüfen und abschließen",
|
||||
"autoVerifyHint": "Nach dem Auftrag prüft OPNexus selbst: Firmware wieder bereit, erwarteter Neustart erfolgt, Version lesbar. Dann ist die Wartung abgeschlossen und sperrt den Host nicht mehr. Ist etwas unklar, bleibt sie offen (Ausgang unklar) und du wirst benachrichtigt.",
|
||||
"rolloutBadge": "Rollout",
|
||||
"openRollout": "Zum Rollout",
|
||||
"autoVerifiedBadge": "automatisch geprüft"
|
||||
},
|
||||
"restore": {
|
||||
"handoffWarning": "OPNexus führt die Wiederherstellung NICHT selbst aus. Nach der Übergabe musst du die Sicherung manuell im OPNsense-Webinterface importieren. Solange dieser Vorgang offen ist, sind Regel-/Alias-/NAT-Pushes und Host-Wartung für diesen Host gesperrt.",
|
||||
@@ -1944,7 +2118,7 @@
|
||||
"schedule": "Eingeplant",
|
||||
"major_update": "Major-Upgrade",
|
||||
"changes": "Änderungen",
|
||||
"download": "Download",
|
||||
"download": "Download",
|
||||
"push": "Übertragen",
|
||||
"register": "Registrieren",
|
||||
"sign": "Signieren",
|
||||
@@ -2686,5 +2860,383 @@
|
||||
"backupHelp": "Sichert die Datenbank (Regeln, Hosts, Verlauf). API-Secrets und Schlüssel liegen in eigenen Docker-Volumes und sind nicht im Dump. Details im Handbuch.",
|
||||
"docs": "Dokumentation zum Kommandozeilen-Werkzeug"
|
||||
}
|
||||
},
|
||||
"gateway": {
|
||||
"title": "Gateways",
|
||||
"none": "Diese Firewall meldet keine Gateways.",
|
||||
"loadFailed": "Gateway-Status konnte nicht geladen werden",
|
||||
"notMeasured": "nicht überwacht",
|
||||
"loss": "Verlust {value} %",
|
||||
"delay": "{value} ms",
|
||||
"stddev": "± {value} ms",
|
||||
"state": {
|
||||
"online": "Online",
|
||||
"degraded": "Beeinträchtigt",
|
||||
"offline": "Offline",
|
||||
"forced_down": "Abgeschaltet",
|
||||
"unknown": "Unbekannt"
|
||||
}
|
||||
},
|
||||
"rollouts": {
|
||||
"title": "Rollouts",
|
||||
"intro": "Firmware-Updates in Wellen: erst eine kleine Gruppe (Canary), dann Pilot, dann der Rest. Zwischen den Wellen wartet OPNexus eine Beobachtungszeit ab und prüft die Gesundheit der aktualisierten Firewalls. Bei jeder Auffälligkeit hält der Rollout an und meldet sich. Jeder Host läuft über die normale geplante Wartung (Backup, CARP-Übergabe, Audit).",
|
||||
"tab": {
|
||||
"runs": "Läufe",
|
||||
"plans": "Pläne",
|
||||
"windows": "Wartungsfenster"
|
||||
},
|
||||
"loadFailed": "Rollouts konnten nicht geladen werden",
|
||||
"actionFailed": "Aktion fehlgeschlagen",
|
||||
"noRuns": "Noch keine Rollouts",
|
||||
"noRunsHint": "Lege unter „Pläne“ Wellen aus Gerätegruppen an und starte einen Plan.",
|
||||
"col": {
|
||||
"plan": "Plan",
|
||||
"status": "Status",
|
||||
"wave": "Welle",
|
||||
"target": "Zielversion",
|
||||
"started": "Gestartet",
|
||||
"hosts": "Hosts",
|
||||
"host": "Host",
|
||||
"version": "Version",
|
||||
"step": "Schritt",
|
||||
"note": "Hinweis"
|
||||
},
|
||||
"details": "Details",
|
||||
"waveOf": "{current}/{total}",
|
||||
"by": "von {user}",
|
||||
"trigger": {
|
||||
"manual": "von Hand",
|
||||
"window": "automatisch im Fenster"
|
||||
},
|
||||
"action": {
|
||||
"update": "Stabiles Update",
|
||||
"major_update": "Major-Upgrade"
|
||||
},
|
||||
"run": {
|
||||
"status": {
|
||||
"waiting_window": "wartet auf Wartungsfenster",
|
||||
"preflight": "Vorprüfung",
|
||||
"wave_running": "Welle läuft",
|
||||
"soaking": "Beobachtungszeit",
|
||||
"gate": "Gesundheitsprüfung",
|
||||
"awaiting_approval": "wartet auf Freigabe",
|
||||
"resuming": "prüft vor dem Fortsetzen",
|
||||
"paused": "angehalten",
|
||||
"succeeded": "abgeschlossen",
|
||||
"cancelled": "abgebrochen"
|
||||
}
|
||||
},
|
||||
"wave": {
|
||||
"status": {
|
||||
"pending": "offen",
|
||||
"awaiting_approval": "wartet auf Freigabe",
|
||||
"preflight": "Vorprüfung",
|
||||
"running": "läuft",
|
||||
"soaking": "Beobachtung",
|
||||
"gate": "Prüfung",
|
||||
"passed": "bestanden",
|
||||
"skipped": "nichts zu tun",
|
||||
"cancelled": "abgebrochen"
|
||||
}
|
||||
},
|
||||
"host": {
|
||||
"status": {
|
||||
"queued": "wartet",
|
||||
"running": "läuft",
|
||||
"verifying": "wird geprüft",
|
||||
"succeeded": "fertig",
|
||||
"skipped": "übersprungen",
|
||||
"failed": "fehlgeschlagen",
|
||||
"blocked": "blockiert",
|
||||
"cancelled": "abgebrochen"
|
||||
}
|
||||
},
|
||||
"hostNote": {
|
||||
"verified": "geprüft: Version {version}",
|
||||
"host_failed": "Fehler: {error}",
|
||||
"schedule_missing": "Geplante Wartung fehlt",
|
||||
"host_deleted": "Host gelöscht",
|
||||
"host_disabled": "Host deaktiviert",
|
||||
"no_update_offered": "kein passendes Update angeboten (Status: {status})",
|
||||
"locked": "andere Wartung offen",
|
||||
"firmware_busy": "Firmware-Auftrag läuft",
|
||||
"offline": "nicht erreichbar",
|
||||
"firmware_status": "Firmware-Status unklar ({status})",
|
||||
"not_fired": "nicht gestartet, wieder eingereiht (auf der Firewall ist nichts passiert)",
|
||||
"skipped_by": "übersprungen von {user}: {reason}"
|
||||
},
|
||||
"pauseReason": {
|
||||
"manual": "von Hand angehalten",
|
||||
"gate_failed": "Gesundheitsprüfung nicht bestanden",
|
||||
"cluster_gate_failed": "Cluster nicht gesund – nächster Knoten wartet",
|
||||
"host_failed": "Update eines Hosts fehlgeschlagen",
|
||||
"host_timeout": "Host braucht ungewöhnlich lange",
|
||||
"host_not_started": "Geplante Wartung startete nicht",
|
||||
"preflight_blocked": "Vorprüfung: Hosts nicht bereit",
|
||||
"target_version_changed": "Angebotene Version weicht von der Zielversion ab",
|
||||
"master_needs_handover": "CARP-MASTER braucht die Übergabe (in der Welle aus)",
|
||||
"blocked_by_maintenance": "Andere Wartung im Cluster offen",
|
||||
"actor_invalid": "Verantwortliche Person darf nicht mehr ausrollen",
|
||||
"window_missing": "Wartungsfenster gelöscht",
|
||||
"run_too_old": "Lauf älter als 14 Tage",
|
||||
"internal_error": "Interner Fehler – angehalten",
|
||||
"aborted": "abgebrochen"
|
||||
},
|
||||
"pauseHint": "Ursache prüfen und beheben, dann „Fortsetzen“: OPNexus prüft zuerst alle bereits aktualisierten Hosts erneut. Ein fehlgeschlagener Host muss vorher ausdrücklich übersprungen werden.",
|
||||
"gate": {
|
||||
"title": "Gesundheitsprüfung",
|
||||
"hint": "Vergleich mit dem Zustand vor der Welle. Rot hält den Rollout an (nach einer Nachprüfung 2 Minuten später), Gelb ist nur ein Hinweis.",
|
||||
"verdict": {
|
||||
"ok": "bestanden",
|
||||
"warn": "bestanden mit Hinweisen",
|
||||
"fail": "nicht bestanden"
|
||||
},
|
||||
"resume": "Prüfung vor dem Fortsetzen",
|
||||
"cluster": "Cluster-Prüfung vor dem nächsten Knoten",
|
||||
"result": {
|
||||
"ok": "ok",
|
||||
"warn": "Hinweis",
|
||||
"fail": "Fehler",
|
||||
"skip": "–"
|
||||
},
|
||||
"check": {
|
||||
"online": "Erreichbar",
|
||||
"version": "Version",
|
||||
"carp": "CARP",
|
||||
"gateways": "Gateways",
|
||||
"services": "Dienste",
|
||||
"locks": "Sperren",
|
||||
"compliance": "Sicherheitslage"
|
||||
},
|
||||
"code": {
|
||||
"ok": "in Ordnung",
|
||||
"disabled": "Host deaktiviert",
|
||||
"offline": "nicht erreichbar",
|
||||
"firmware_busy": "Firmware-Auftrag läuft noch",
|
||||
"no_target": "keine Zielversion bekannt",
|
||||
"unreadable": "nicht lesbar (vorher schon)",
|
||||
"unreadable_both": "nicht lesbar (API-Recht fehlt?) – nicht verglichen",
|
||||
"version_mismatch": "erwartet {expected}, installiert {actual}",
|
||||
"no_cluster": "kein Cluster",
|
||||
"node_offline": "Knoten nicht erreichbar: {names}",
|
||||
"maintenance_mode": "im CARP-Wartungsmodus: {names}",
|
||||
"auto_revert_active": "CARP-Zeitfenster aktiv: {names}",
|
||||
"vhid_roles": "VHID ohne genau einen MASTER: {vhids}",
|
||||
"no_master": "kein MASTER erkannt",
|
||||
"no_baseline": "kein Vergleichswert von vorher",
|
||||
"gateway_down": "Gateway ausgefallen: {names}",
|
||||
"gateway_missing": "Gateway fehlt: {names}",
|
||||
"gateway_degraded": "Gateway mit Verlust/Latenz: {names}",
|
||||
"gateway_bad_before": "schon vorher gestört: {names}",
|
||||
"service_stopped": "läuft nicht mehr: {names}",
|
||||
"open_locks": "{count} offene Sperre(n) im Host/Cluster",
|
||||
"score_lower": "Score {before} → {after}"
|
||||
}
|
||||
},
|
||||
"actions": {
|
||||
"pause": "Anhalten",
|
||||
"resume": "Fortsetzen",
|
||||
"approve": "Welle freigeben",
|
||||
"skip": "Überspringen",
|
||||
"abort": "Abbrechen",
|
||||
"pauseReason": "Grund (optional)",
|
||||
"skipTitle": "{host} überspringen",
|
||||
"skipHint": "Der Host bleibt auf seinem Stand und wird in diesem Rollout nicht mehr angefasst. Eine fehlgeschlagene Wartung bleibt offen, bis du sie auf der Host-Seite abschließt.",
|
||||
"reason": "Begründung",
|
||||
"abortTitle": "Rollout abbrechen",
|
||||
"abortHint": "Noch nicht begonnene Hosts werden nicht mehr aktualisiert. Eine Aktion, die schon auf einer Firewall läuft, wird geführt zu Ende gebracht (nicht abgewürgt).",
|
||||
"confirmPlan": "Zur Bestätigung den Plannamen eingeben",
|
||||
"paused": "Rollout angehalten",
|
||||
"resumed": "Wird geprüft und fortgesetzt",
|
||||
"approved": "Welle freigegeben",
|
||||
"skipped": "Host übersprungen",
|
||||
"aborted": "Rollout abgebrochen"
|
||||
},
|
||||
"detail": {
|
||||
"back": "Alle Rollouts",
|
||||
"notFound": "Rollout nicht gefunden",
|
||||
"soakUntil": "Beobachtung bis {time}",
|
||||
"recheckAt": "Nachprüfung um {time}",
|
||||
"blockedSince": "wartet seit {time} auf eine fremde Wartung (max. 15 min)",
|
||||
"waves": "Wellen",
|
||||
"hosts": "Hosts",
|
||||
"handover": "mit CARP-Übergabe",
|
||||
"approval": "Freigabe nötig",
|
||||
"approvedBy": "freigegeben von {user}",
|
||||
"soak": "{minutes} min Beobachtung",
|
||||
"noGate": "Noch keine Prüfung",
|
||||
"attempt": "Versuch {n}",
|
||||
"stillRunning": "läuft noch zu Ende: {hosts}",
|
||||
"waitingWindow": "Neue Hosts starten erst im nächsten Wartungsfenster.",
|
||||
"awaiting": "Die nächste Welle wartet auf Freigabe.",
|
||||
"owner": "verantwortlich"
|
||||
},
|
||||
"banner": {
|
||||
"active": "Rollout „{plan}“ – Welle {wave} ({label}): {status}",
|
||||
"open": "Ansehen"
|
||||
},
|
||||
"inRollout": "im Rollout",
|
||||
"inRolloutTip": "Dieser Host gehört zum laufenden Rollout. Manuelle Wartung ist hier gesperrt, bis der Rollout ihn erledigt oder übersprungen hat.",
|
||||
"updaterWarning": "Ein Firmware-Rollout ist aktiv. Ein Neustart von OPNexus ist sicher (der Zustand liegt in der Datenbank), hält den Rollout aber für die Dauer an.",
|
||||
"plan": {
|
||||
"new": "Neuer Plan",
|
||||
"edit": "Plan bearbeiten",
|
||||
"none": "Noch keine Pläne",
|
||||
"noneHint": "Lege Gerätegruppen als Ringe an (z. B. ring-canary, ring-pilot, ring-prod) und baue daraus Wellen.",
|
||||
"name": "Name",
|
||||
"description": "Beschreibung",
|
||||
"action": "Aktion",
|
||||
"window": "Wartungsfenster",
|
||||
"noWindow": "keins (Hosts starten sofort)",
|
||||
"autoStart": "Automatisch zu jedem Fenstertermin starten",
|
||||
"autoStartHint": "Nur für stabile Updates. Wirkt erst nach „Aktivieren“ und nur, solange Plan, Gruppen und Fenster unverändert sind; jede Änderung deaktiviert den Plan.",
|
||||
"waves": "Wellen",
|
||||
"addWave": "Welle hinzufügen",
|
||||
"waveLabel": "Bezeichnung",
|
||||
"groups": "Gerätegruppen",
|
||||
"soak": "Beobachtungszeit (min)",
|
||||
"soakHint": "So lange nach dem letzten Host der Welle warten, bevor die Gesundheit geprüft wird.",
|
||||
"handover": "CARP-Übergabe für MASTER-Knoten",
|
||||
"approval": "Vor dieser Welle auf Freigabe warten",
|
||||
"moveUp": "Welle nach oben",
|
||||
"moveDown": "Welle nach unten",
|
||||
"removeWave": "Welle entfernen",
|
||||
"checks": "Gesundheitsprüfung",
|
||||
"checksHint": "Erreichbarkeit, Version und Sperren werden immer geprüft.",
|
||||
"complianceMode": "Niedrigerer Sicherheits-Score",
|
||||
"complianceWarn": "nur Hinweis",
|
||||
"complianceFail": "hält an",
|
||||
"save": "Speichern",
|
||||
"saved": "Plan gespeichert",
|
||||
"deleted": "Plan gelöscht",
|
||||
"deleteConfirm": "Plan „{name}“ löschen? Laufhistorie bleibt erhalten.",
|
||||
"previewStart": "Vorschau & Start",
|
||||
"activate": "Aktivieren",
|
||||
"deactivate": "Deaktivieren",
|
||||
"active": "automatisch aktiv",
|
||||
"inactive": "automatisch aus",
|
||||
"changed": "geändert seit Aktivierung",
|
||||
"changedTip": "Plan, Gruppen oder Fenster haben sich seit der Aktivierung geändert. Der automatische Start ist angehalten, bis der Plan erneut aktiviert wird.",
|
||||
"activatedBy": "aktiviert von {user}",
|
||||
"lastAutoStartError": "Letzter automatischer Start ausgeblieben ({time}): {reason}",
|
||||
"hostsCount": "{count} Hosts",
|
||||
"errors": "{count} Fehler",
|
||||
"warnings": "{count} Hinweise",
|
||||
"nextOccurrence": "Nächster Termin: {time}",
|
||||
"previewTitle": "Vorschau: {name}",
|
||||
"unattended": "Ich weiß, dass die Firewalls dann ohne weitere Rückfrage aktualisiert und neu gestartet werden; der Rollout hält bei jeder Auffälligkeit selbst an.",
|
||||
"start": "Rollout starten",
|
||||
"started": "Rollout gestartet",
|
||||
"activated": "Automatischer Start aktiviert",
|
||||
"deactivated": "Automatischer Start deaktiviert",
|
||||
"emptyWave": "keine Hosts",
|
||||
"disabledHost": "deaktiviert",
|
||||
"cannotStart": "So nicht startbar – siehe Fehler oben."
|
||||
},
|
||||
"issue": {
|
||||
"no_waves": "Keine Wellen",
|
||||
"no_hosts": "Keine Hosts in den Gruppen",
|
||||
"too_many_waves": "Mehr als {limit} Wellen",
|
||||
"too_many_hosts": "Mehr als {limit} Hosts",
|
||||
"cluster_split": "Cluster {cluster}: Für ein Major-Upgrade müssen alle Knoten in derselben Welle liegen.",
|
||||
"cluster_partial": "Cluster {cluster}: Nicht alle Knoten sind im Plan.",
|
||||
"mixed_versions": "Cluster {cluster}: Knoten in verschiedenen Wellen – gemischte Versionen bis Welle {wave}.",
|
||||
"master_needs_handover": "Welle {wave} enthält Cluster ({clusters}) ohne CARP-Übergabe: beim MASTER hält der Rollout an.",
|
||||
"empty_wave": "Welle {wave} hat keine Hosts.",
|
||||
"duplicate_hosts": "In mehreren Wellen (die früheste gilt): {names}",
|
||||
"disabled_hosts": "Deaktiviert, werden übersprungen: {names}",
|
||||
"auto_start_update_only": "Automatischer Start nur für stabile Updates.",
|
||||
"auto_start_needs_window": "Automatischer Start braucht ein Wartungsfenster.",
|
||||
"no_window": "Kein Wartungsfenster: Hosts starten sofort nacheinander.",
|
||||
"window_disabled": "Das Wartungsfenster ist ausgeschaltet: nichts startet.",
|
||||
"plan_changed": "Plan, Gruppen oder Fenster seit der Aktivierung geändert",
|
||||
"actor_invalid": "{user} darf nicht mehr ausrollen",
|
||||
"plan_invalid": "Plan ungültig ({reason})",
|
||||
"other_run_active": "anderer Rollout läuft ({plan})"
|
||||
},
|
||||
"detailKey": {
|
||||
"wave": "Welle",
|
||||
"failed": "Nicht bestanden",
|
||||
"host": "Host",
|
||||
"hosts": "Hosts",
|
||||
"cluster": "Cluster",
|
||||
"expected": "Erwartet",
|
||||
"offered": "Angeboten",
|
||||
"user": "Benutzer",
|
||||
"by": "Von",
|
||||
"text": "Grund",
|
||||
"reason": "Grund",
|
||||
"error": "Fehler",
|
||||
"days": "Tage",
|
||||
"still_running": "Läuft noch"
|
||||
}
|
||||
},
|
||||
"mwindows": {
|
||||
"none": "Noch keine Wartungsfenster",
|
||||
"noneHint": "Ein Fenster legt fest, wann ein Rollout neue Hosts beginnen darf, z. B. sonntags 02:00–05:00.",
|
||||
"new": "Neues Fenster",
|
||||
"edit": "Fenster bearbeiten",
|
||||
"name": "Name",
|
||||
"timezone": "Zeitzone",
|
||||
"recurrence": "Wiederholung",
|
||||
"weekly": "wöchentlich",
|
||||
"monthly": "monatlich",
|
||||
"weekdays": "Wochentage",
|
||||
"monthMode": "Monatlich am",
|
||||
"nthWeekday": "n-ten Wochentag",
|
||||
"dayOfMonth": "Tag des Monats",
|
||||
"nth": {
|
||||
"1": "ersten",
|
||||
"2": "zweiten",
|
||||
"3": "dritten",
|
||||
"4": "vierten",
|
||||
"-1": "letzten"
|
||||
},
|
||||
"day": "Tag (1–28)",
|
||||
"startTime": "Beginn (Ortszeit)",
|
||||
"duration": "Dauer (min)",
|
||||
"cutoff": "Neue Hosts nur in den ersten … min",
|
||||
"cutoffHint": "Danach beginnt kein neuer Host mehr; laufende Updates laufen zu Ende. So passt ein Update samt Neustart sicher ins Fenster.",
|
||||
"enabled": "eingeschaltet",
|
||||
"dstHint": "Sommerzeit: Eine Uhrzeit, die es an dem Tag nicht gibt, rückt vor; eine doppelte gilt beim ersten Mal.",
|
||||
"saved": "Fenster gespeichert",
|
||||
"deleted": "Fenster gelöscht",
|
||||
"deleteConfirm": "Fenster „{name}“ löschen?",
|
||||
"next": "Nächste Termine",
|
||||
"openNow": "gerade offen",
|
||||
"off": "ausgeschaltet",
|
||||
"skipped": "ausgelassen",
|
||||
"moved": "verschoben",
|
||||
"skip": "Auslassen",
|
||||
"move": "Verschieben",
|
||||
"moveTo": "Neuer Beginn",
|
||||
"exceptions": "Ausnahmen",
|
||||
"removeException": "Ausnahme entfernen",
|
||||
"exceptionSaved": "Ausnahme gespeichert",
|
||||
"usedBy": "Genutzt von: {plans}",
|
||||
"summary": {
|
||||
"weekly": "{days}, {time} ({tz}), {duration} min",
|
||||
"nth": "am {nth} {day} im Monat, {time} ({tz}), {duration} min",
|
||||
"day": "am {day}. jedes Monats, {time} ({tz}), {duration} min"
|
||||
},
|
||||
"dayName": {
|
||||
"0": "Mo",
|
||||
"1": "Di",
|
||||
"2": "Mi",
|
||||
"3": "Do",
|
||||
"4": "Fr",
|
||||
"5": "Sa",
|
||||
"6": "So"
|
||||
},
|
||||
"dayLong": {
|
||||
"0": "Montag",
|
||||
"1": "Dienstag",
|
||||
"2": "Mittwoch",
|
||||
"3": "Donnerstag",
|
||||
"4": "Freitag",
|
||||
"5": "Samstag",
|
||||
"6": "Sonntag"
|
||||
},
|
||||
"cutoffSummary": "neue Hosts nur in den ersten {minutes} min"
|
||||
}
|
||||
}
|
||||
+560
-8
@@ -15,7 +15,8 @@
|
||||
"saving": "Saving…",
|
||||
"deleteFailed": "Deleting failed",
|
||||
"description": "Description",
|
||||
"versionLabel": "OPNexus {version} “{codename}”"
|
||||
"versionLabel": "OPNexus {version} “{codename}”",
|
||||
"delete": "Delete"
|
||||
},
|
||||
"settings": {
|
||||
"language": {
|
||||
@@ -66,10 +67,28 @@
|
||||
"scheduledFailed": "Scheduled maintenance did not run",
|
||||
"backupFailed": "No recent backup / offsite copy failed",
|
||||
"complianceAlerts": "Critical or high security finding (off by default)",
|
||||
"networkDrift": "Network differences in the cluster"
|
||||
"networkDrift": "Network differences in the cluster",
|
||||
"gatewayDown": "Gateway failing (offline, packet loss or latency above the limit)",
|
||||
"resourceThreshold": "Resource threshold exceeded (disk, memory, swap, CPU load; off by default)",
|
||||
"serviceDown": "Watched service stopped (off by default)",
|
||||
"rolloutProblems": "Rollout paused or not started automatically"
|
||||
},
|
||||
"pollInterval": "Check interval (seconds)",
|
||||
"pollHint": "At least 60s, so the firewalls are not loaded unnecessarily."
|
||||
"pollHint": "At least 60s, so the firewalls are not loaded unnecessarily.",
|
||||
"monitoring": "Thresholds and debouncing",
|
||||
"monitoringHint": "Alerts for gateways, resources and services use the measurements OPNexus records every minute. An alert opens only when the latest measurements are all bad and closes only when they are all good again, so single spikes and flapping links do not send messages. 0 switches a single threshold off.",
|
||||
"limit": {
|
||||
"gatewayLoss": "Gateway: packet loss from (%)",
|
||||
"gatewayDelay": "Gateway: latency from (ms)",
|
||||
"diskLimit": "Disk from (% used)",
|
||||
"memoryLimit": "Memory from (% used)",
|
||||
"swapLimit": "Swap from (% used)",
|
||||
"loadLimit": "CPU load from (load per thread)",
|
||||
"confirmSamples": "Gateways and services: consecutive measurements",
|
||||
"resourceSamples": "Resources: consecutive measurements",
|
||||
"watchlist": "Watched services",
|
||||
"watchlistHint": "OPNsense service names, comma-separated. OPNexus reports a service only if it ran before and has stopped now. Remove services you stopped on purpose."
|
||||
}
|
||||
},
|
||||
"logs": {
|
||||
"saved": "Log ingest settings saved.",
|
||||
@@ -180,6 +199,51 @@
|
||||
"title": "Setup assistant",
|
||||
"intro": "Walks you step by step through password, two-factor sign-in and connecting a firewall.",
|
||||
"open": "Open assistant"
|
||||
},
|
||||
"channels": {
|
||||
"title": "More channels",
|
||||
"hint": "ntfy, Gotify, Telegram, Microsoft Teams and Matrix receive the same messages as webhook and email. Credentials are stored apart from the database and are never shown.",
|
||||
"add": "Add channel",
|
||||
"edit": "Edit channel",
|
||||
"type": "Type",
|
||||
"name": "Name",
|
||||
"enabled": "Active",
|
||||
"disabled": "Off",
|
||||
"enable": "Turn on",
|
||||
"disable": "Turn off",
|
||||
"test": "Test",
|
||||
"editAction": "Edit",
|
||||
"deleteAction": "Delete",
|
||||
"empty": "No further channels set up yet",
|
||||
"saved": "Channel saved.",
|
||||
"deleted": "Channel “{name}” deleted.",
|
||||
"confirmDelete": "Really delete channel “{name}”?",
|
||||
"testSent": "Test message sent through “{name}”.",
|
||||
"loadFailed": "Channels could not be loaded",
|
||||
"removeSecret": "Remove access token",
|
||||
"types": {
|
||||
"ntfy": "ntfy",
|
||||
"gotify": "Gotify",
|
||||
"telegram": "Telegram",
|
||||
"teams": "Microsoft Teams",
|
||||
"matrix": "Matrix"
|
||||
},
|
||||
"field": {
|
||||
"server_url": "Server address",
|
||||
"topic": "Topic",
|
||||
"priority": "Priority",
|
||||
"chat_id": "Chat ID",
|
||||
"api_base": "API address (optional)",
|
||||
"homeserver_url": "Homeserver address",
|
||||
"room_id": "Room ID"
|
||||
},
|
||||
"secret": {
|
||||
"ntfy": "Access token (optional)",
|
||||
"gotify": "App token",
|
||||
"telegram": "Bot token",
|
||||
"teams": "Workflow webhook address",
|
||||
"matrix": "Access token"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nav": {
|
||||
@@ -235,7 +299,8 @@
|
||||
"hostsystem-syslog": "Syslog",
|
||||
"hostsystem-snapshots": "Snapshots",
|
||||
"hostsystem-diagnostics": "Diagnostics",
|
||||
"setup": "Setup"
|
||||
"setup": "Setup",
|
||||
"rollouts": "Rollouts"
|
||||
}
|
||||
},
|
||||
"dashboard": {
|
||||
@@ -297,7 +362,15 @@
|
||||
"majorAvailable": "Major upgrade available",
|
||||
"stableAvailable": "Stable update available",
|
||||
"updateBadge": "Update",
|
||||
"loadFailed": "Could not load dashboard"
|
||||
"loadFailed": "Could not load dashboard",
|
||||
"gateways": "Gateways",
|
||||
"noGateways": "No gateway data",
|
||||
"noGatewaysDescription": "Once the firewalls' measurements are in (every minute), status, loss and latency appear here.",
|
||||
"alert": {
|
||||
"gateway_down": "Gateway",
|
||||
"resource_threshold": "Resource",
|
||||
"service_down": "Service"
|
||||
}
|
||||
},
|
||||
"hosts": {
|
||||
"add": "Add host",
|
||||
@@ -435,6 +508,67 @@
|
||||
},
|
||||
"logPackets": "Log matching packets"
|
||||
},
|
||||
"adopt": {
|
||||
"button": "Adopt from firewall",
|
||||
"title": {
|
||||
"aliases": "Adopt aliases from the firewall",
|
||||
"portForward": "Adopt port forwards from the firewall",
|
||||
"oneToOne": "Adopt 1:1 NAT rules from the firewall",
|
||||
"outboundNat": "Adopt outbound NAT rules from the firewall"
|
||||
},
|
||||
"intro": "These objects were created directly in OPNsense. Once adopted you manage them in OPNexus. Adopting itself changes nothing on the firewall.",
|
||||
"host": "Firewall",
|
||||
"none": "Nothing adoptable found. Everything is already managed.",
|
||||
"action": "Adopt",
|
||||
"all": "Adopt all adoptable ({count})",
|
||||
"adopted": "Adopted “{name}”.",
|
||||
"allDone": "{count} adopted, {skipped} skipped.",
|
||||
"failed": "Adopting failed",
|
||||
"loadFailed": "Could not load objects.",
|
||||
"notPossible": "Cannot be adopted",
|
||||
"bulkOnly": "Only via “Adopt all”",
|
||||
"entries": "Entries: {count}",
|
||||
"towards": "to {destination}",
|
||||
"interfaceAddress": "interface address",
|
||||
"systemNote": "{count} OPNsense system aliases (e.g. bogons, interface networks) are never adopted.",
|
||||
"kept": "Stays unchanged on the firewall: {fields}",
|
||||
"keptFields": {
|
||||
"log": "Logging",
|
||||
"poolopts": "Pool options",
|
||||
"poolopts_sourcehashkey": "Source hash key",
|
||||
"tag": "Tag",
|
||||
"tagged": "Tagged",
|
||||
"nosync": "No sync",
|
||||
"categories": "Categories",
|
||||
"staticnatport": "Static NAT port",
|
||||
"endpoint-independent": "Endpoint-independent"
|
||||
},
|
||||
"reasons": {
|
||||
"system": "OPNsense system object",
|
||||
"already_managed": "Already managed",
|
||||
"unsupported_type": "OPNexus does not support type “{detail}”",
|
||||
"name_invalid": "The name contains characters OPNexus does not allow (letters, digits, _ only)",
|
||||
"name_exists": "OPNexus already has an alias with this name",
|
||||
"content_too_large": "Content too large (over 8000 characters)",
|
||||
"proto_set": "Has an IPv4/IPv6 restriction OPNexus cannot represent",
|
||||
"value_too_long": "A value is longer than OPNexus stores",
|
||||
"references_unmanaged_alias": "Refers to aliases that are not adopted: {detail}",
|
||||
"interface_not_single": "Several or no interface",
|
||||
"ipprotocol_unsupported": "IP version cannot be represented (IPv4 or IPv6 only)",
|
||||
"protocol_any": "Protocol “any” is not supported",
|
||||
"protocol_missing": "Protocol missing",
|
||||
"no_redirect": "“No redirect” rule (No RDR)",
|
||||
"no_target": "No target set",
|
||||
"filter_rule_linked": "Creates or is linked to an associated filter rule",
|
||||
"negated_source": "Source is inverted (NOT)",
|
||||
"negated_destination": "Destination is inverted (NOT)",
|
||||
"natreflection_unsupported": "NAT reflection value is not supported",
|
||||
"type_unsupported": "Rule type is not supported",
|
||||
"incomplete": "Required fields missing",
|
||||
"no_nat_rule": "“No NAT” rule (nonat) is not supported",
|
||||
"no_description": "Description missing – please set one in OPNsense"
|
||||
}
|
||||
},
|
||||
"login": {
|
||||
"prompt": "Please sign in",
|
||||
"username": "Username",
|
||||
@@ -596,6 +730,9 @@
|
||||
"TOTP_NOT_ENABLED": "Two-factor authentication is not enabled",
|
||||
"TOTP_SETUP_MISSING": "No setup has been started",
|
||||
"RULE_NOT_ADOPTABLE": "This rule cannot be adopted ({reasons})",
|
||||
"OBJECT_NOT_ADOPTABLE": "This object cannot be adopted ({reasons})",
|
||||
"ADOPT_OBJECT_NOT_FOUND": "The object was not found on the firewall. Reload the list.",
|
||||
"OUTBOUND_MODE_BLOCKS_ADOPT": "{host}: in outbound NAT mode “{mode}” OPNsense does not list manual rules. Switch the mode to Hybrid/Manual in OPNsense itself, then they can be adopted. OPNexus does not change this mode.",
|
||||
"NETWORK_ACK_REQUIRED": "This change can cut the connection to the firewall and must be confirmed explicitly ({reasons})",
|
||||
"NETWORK_APPLY_UNCERTAIN": "The change was written, but applying it on “{host}” failed or is unclear. Check the firewall; backup: {backup_id}",
|
||||
"NETWORK_BACKUP_FAILED": "Backup of “{host}” before the change failed – nothing was changed",
|
||||
@@ -624,7 +761,33 @@
|
||||
"OFFSITE_TEST_FAILED": "The offsite target test failed: {reason}",
|
||||
"API_TOKEN_NOT_FOUND": "API token not found",
|
||||
"API_TOKEN_SCOPE_FORBIDDEN": "This API token (scope “{scope}”) is not allowed to perform this action.",
|
||||
"API_TOKEN_SCOPE_TOO_HIGH": "Your role does not allow a token with this scope (read-only: read, operator: up to deploy)."
|
||||
"API_TOKEN_SCOPE_TOO_HIGH": "Your role does not allow a token with this scope (read-only: read, operator: up to deploy).",
|
||||
"NOTIFICATION_CHANNEL_INVALID": "Invalid value: {reason}",
|
||||
"NOTIFICATION_CHANNEL_NOT_FOUND": "The notification channel was not found.",
|
||||
"NOTIFICATION_CHANNEL_NAME_TAKEN": "A channel with this name already exists.",
|
||||
"NOTIFICATION_CHANNEL_TEST_FAILED": "Test message not delivered: {reason}",
|
||||
"NOTIFICATION_WEBHOOK_URL_INVALID": "Invalid webhook address: {reason}",
|
||||
"WINDOW_NOT_FOUND": "Maintenance window not found.",
|
||||
"WINDOW_INVALID": "Invalid maintenance window (field: {field}).",
|
||||
"WINDOW_NAME_TAKEN": "A maintenance window with this name already exists.",
|
||||
"WINDOW_IN_USE": "A rollout plan uses this window. Change or delete the plan first.",
|
||||
"WINDOW_EXCEPTION_INVALID": "Invalid exception (field: {field}). Only future occurrences of the window can be skipped or moved.",
|
||||
"WINDOW_EXCEPTION_EXISTS": "This occurrence already has an exception.",
|
||||
"WINDOW_EXCEPTION_NOT_FOUND": "Exception not found.",
|
||||
"ROLLOUT_PLAN_NOT_FOUND": "Rollout plan not found.",
|
||||
"ROLLOUT_INVALID": "Invalid rollout plan (field: {field}).",
|
||||
"ROLLOUT_NAME_TAKEN": "A rollout plan with this name already exists.",
|
||||
"ROLLOUT_PLAN_INVALID": "The plan cannot start like this: {reason}. Check the preview.",
|
||||
"ROLLOUT_PREVIEW_STALE": "The plan, its groups or its window changed since the preview. Reload the preview.",
|
||||
"ROLLOUT_PLAN_BUSY": "A rollout of this plan is in progress. Finish or abort it first.",
|
||||
"ROLLOUT_GROUP_IN_USE": "The group is a wave of a rollout plan. Change the plan first.",
|
||||
"ROLLOUT_RUN_NOT_FOUND": "Rollout run not found.",
|
||||
"ROLLOUT_ALREADY_ACTIVE": "A rollout is already in progress ({plan}). Only one runs at a time.",
|
||||
"ROLLOUT_CONFIRM_NAME": "Type the exact name to confirm.",
|
||||
"ROLLOUT_ACTION_NOT_ALLOWED": "Not possible while the run is “{status}”.",
|
||||
"ROLLOUT_HOST_NOT_IN_RUN": "This host is not part of this rollout.",
|
||||
"ROLLOUT_HOST_RUNNING": "The action on {host} is already running; it is finished in a controlled way and cannot be skipped.",
|
||||
"ROLLOUT_HOST_FAILED": "Failed: {hosts}. Check it and skip the host first, then resume."
|
||||
},
|
||||
"workflow": {
|
||||
"attention": "Check errors",
|
||||
@@ -962,7 +1125,18 @@
|
||||
"handoverOption": "Switch to the partner first and switch back afterwards (CARP)",
|
||||
"handoverHint": "Flow: maintenance mode on this host, the partner becomes MASTER (verified), then the action runs, and once the host is back the maintenance mode is left again. If anything goes wrong the partner stays MASTER and a notification is sent. When switching back, the host may take the MASTER role back depending on the CARP settings (a short second switchover).",
|
||||
"handoverBadge": "with CARP switchover",
|
||||
"phase": {"handover_wait": "waiting for the partner to take over", "job_wait": "job running", "leave": "switching back", "leave_wait": "verifying normal operation"}
|
||||
"phase": {
|
||||
"handover_wait": "waiting for the partner to take over",
|
||||
"job_wait": "job running",
|
||||
"leave": "switching back",
|
||||
"leave_wait": "verifying normal operation",
|
||||
"verify": "checking the result"
|
||||
},
|
||||
"autoVerifyOption": "Check the result automatically and close the operation",
|
||||
"autoVerifyHint": "After the job OPNexus checks by itself: firmware idle again, the expected reboot happened, version readable. Then the maintenance is closed and no longer locks the host. If anything is unclear it stays open (outcome unclear) and you are notified.",
|
||||
"rolloutBadge": "Rollout",
|
||||
"openRollout": "Open rollout",
|
||||
"autoVerifiedBadge": "checked automatically"
|
||||
},
|
||||
"restore": {
|
||||
"handoffWarning": "OPNexus does NOT perform the restore itself. After the handover you must import the backup manually in the OPNsense web interface. While this operation is open, rule/alias/NAT pushes and host maintenance are blocked for this host.",
|
||||
@@ -1944,7 +2118,7 @@
|
||||
"schedule": "Scheduled",
|
||||
"major_update": "Major upgrade",
|
||||
"changes": "Changes",
|
||||
"download": "Download",
|
||||
"download": "Download",
|
||||
"push": "Push",
|
||||
"register": "Register",
|
||||
"sign": "Sign",
|
||||
@@ -2686,5 +2860,383 @@
|
||||
"backupHelp": "Covers the database (rules, hosts, history). API secrets and keys live in their own Docker volumes and are not in the dump. See the manual for details.",
|
||||
"docs": "Documentation for the command-line tool"
|
||||
}
|
||||
},
|
||||
"gateway": {
|
||||
"title": "Gateways",
|
||||
"none": "This firewall reports no gateways.",
|
||||
"loadFailed": "Gateway status could not be loaded",
|
||||
"notMeasured": "not monitored",
|
||||
"loss": "Loss {value} %",
|
||||
"delay": "{value} ms",
|
||||
"stddev": "± {value} ms",
|
||||
"state": {
|
||||
"online": "Online",
|
||||
"degraded": "Degraded",
|
||||
"offline": "Offline",
|
||||
"forced_down": "Switched off",
|
||||
"unknown": "Unknown"
|
||||
}
|
||||
},
|
||||
"rollouts": {
|
||||
"title": "Rollouts",
|
||||
"intro": "Firmware updates in waves: a small group first (canary), then pilot, then the rest. Between waves OPNexus waits a soak time and checks the health of the updated firewalls. On anything unusual the rollout stops and notifies you. Each host goes through the normal scheduled maintenance (backup, CARP handover, audit).",
|
||||
"tab": {
|
||||
"runs": "Runs",
|
||||
"plans": "Plans",
|
||||
"windows": "Maintenance windows"
|
||||
},
|
||||
"loadFailed": "Could not load rollouts",
|
||||
"actionFailed": "Action failed",
|
||||
"noRuns": "No rollouts yet",
|
||||
"noRunsHint": "Create waves from device groups under “Plans” and start a plan.",
|
||||
"col": {
|
||||
"plan": "Plan",
|
||||
"status": "Status",
|
||||
"wave": "Wave",
|
||||
"target": "Target version",
|
||||
"started": "Started",
|
||||
"hosts": "Hosts",
|
||||
"host": "Host",
|
||||
"version": "Version",
|
||||
"step": "Step",
|
||||
"note": "Note"
|
||||
},
|
||||
"details": "Details",
|
||||
"waveOf": "{current}/{total}",
|
||||
"by": "by {user}",
|
||||
"trigger": {
|
||||
"manual": "manual",
|
||||
"window": "automatic in window"
|
||||
},
|
||||
"action": {
|
||||
"update": "Stable update",
|
||||
"major_update": "Major upgrade"
|
||||
},
|
||||
"run": {
|
||||
"status": {
|
||||
"waiting_window": "waiting for window",
|
||||
"preflight": "preflight",
|
||||
"wave_running": "wave running",
|
||||
"soaking": "soaking",
|
||||
"gate": "health gate",
|
||||
"awaiting_approval": "awaiting approval",
|
||||
"resuming": "checking before resuming",
|
||||
"paused": "paused",
|
||||
"succeeded": "completed",
|
||||
"cancelled": "aborted"
|
||||
}
|
||||
},
|
||||
"wave": {
|
||||
"status": {
|
||||
"pending": "pending",
|
||||
"awaiting_approval": "awaiting approval",
|
||||
"preflight": "preflight",
|
||||
"running": "running",
|
||||
"soaking": "soaking",
|
||||
"gate": "gate",
|
||||
"passed": "passed",
|
||||
"skipped": "nothing to do",
|
||||
"cancelled": "aborted"
|
||||
}
|
||||
},
|
||||
"host": {
|
||||
"status": {
|
||||
"queued": "queued",
|
||||
"running": "running",
|
||||
"verifying": "verifying",
|
||||
"succeeded": "done",
|
||||
"skipped": "skipped",
|
||||
"failed": "failed",
|
||||
"blocked": "blocked",
|
||||
"cancelled": "aborted"
|
||||
}
|
||||
},
|
||||
"hostNote": {
|
||||
"verified": "verified: version {version}",
|
||||
"host_failed": "Error: {error}",
|
||||
"schedule_missing": "Scheduled maintenance missing",
|
||||
"host_deleted": "Host deleted",
|
||||
"host_disabled": "Host disabled",
|
||||
"no_update_offered": "no matching update offered (status: {status})",
|
||||
"locked": "other maintenance open",
|
||||
"firmware_busy": "firmware job running",
|
||||
"offline": "unreachable",
|
||||
"firmware_status": "firmware status unclear ({status})",
|
||||
"not_fired": "not started, queued again (nothing happened on the firewall)",
|
||||
"skipped_by": "skipped by {user}: {reason}"
|
||||
},
|
||||
"pauseReason": {
|
||||
"manual": "paused by hand",
|
||||
"gate_failed": "health gate failed",
|
||||
"cluster_gate_failed": "cluster not healthy – next node waits",
|
||||
"host_failed": "a host update failed",
|
||||
"host_timeout": "host is taking unusually long",
|
||||
"host_not_started": "scheduled maintenance did not start",
|
||||
"preflight_blocked": "preflight: hosts not ready",
|
||||
"target_version_changed": "offered version differs from the target version",
|
||||
"master_needs_handover": "CARP MASTER needs the handover (off in this wave)",
|
||||
"blocked_by_maintenance": "other maintenance open in the cluster",
|
||||
"actor_invalid": "the responsible person may no longer roll out",
|
||||
"window_missing": "maintenance window deleted",
|
||||
"run_too_old": "run older than 14 days",
|
||||
"internal_error": "internal error – paused",
|
||||
"aborted": "aborted"
|
||||
},
|
||||
"pauseHint": "Check and fix the cause, then “Resume”: OPNexus first re-checks every host updated so far. A failed host must be skipped explicitly first.",
|
||||
"gate": {
|
||||
"title": "Health gate",
|
||||
"hint": "Compared with the state before the wave. Red stops the rollout (after one re-check 2 minutes later), yellow is only a hint.",
|
||||
"verdict": {
|
||||
"ok": "passed",
|
||||
"warn": "passed with hints",
|
||||
"fail": "failed"
|
||||
},
|
||||
"resume": "Check before resuming",
|
||||
"cluster": "Cluster check before the next node",
|
||||
"result": {
|
||||
"ok": "ok",
|
||||
"warn": "hint",
|
||||
"fail": "fail",
|
||||
"skip": "–"
|
||||
},
|
||||
"check": {
|
||||
"online": "Online",
|
||||
"version": "Version",
|
||||
"carp": "CARP",
|
||||
"gateways": "Gateways",
|
||||
"services": "Services",
|
||||
"locks": "Locks",
|
||||
"compliance": "Compliance"
|
||||
},
|
||||
"code": {
|
||||
"ok": "fine",
|
||||
"disabled": "host disabled",
|
||||
"offline": "unreachable",
|
||||
"firmware_busy": "firmware job still running",
|
||||
"no_target": "no target version known",
|
||||
"unreadable": "unreadable",
|
||||
"unreadable_both": "unreadable (API permission missing?) – not compared",
|
||||
"version_mismatch": "expected {expected}, installed {actual}",
|
||||
"no_cluster": "no cluster",
|
||||
"node_offline": "node unreachable: {names}",
|
||||
"maintenance_mode": "in CARP maintenance mode: {names}",
|
||||
"auto_revert_active": "CARP timer active: {names}",
|
||||
"vhid_roles": "VHID without exactly one MASTER: {vhids}",
|
||||
"no_master": "no MASTER found",
|
||||
"no_baseline": "no earlier value to compare",
|
||||
"gateway_down": "gateway down: {names}",
|
||||
"gateway_missing": "gateway missing: {names}",
|
||||
"gateway_degraded": "gateway with loss/latency: {names}",
|
||||
"gateway_bad_before": "already bad before: {names}",
|
||||
"service_stopped": "no longer running: {names}",
|
||||
"open_locks": "{count} open lock(s) in host/cluster",
|
||||
"score_lower": "score {before} → {after}"
|
||||
}
|
||||
},
|
||||
"actions": {
|
||||
"pause": "Pause",
|
||||
"resume": "Resume",
|
||||
"approve": "Release wave",
|
||||
"skip": "Skip",
|
||||
"abort": "Abort",
|
||||
"pauseReason": "Reason (optional)",
|
||||
"skipTitle": "Skip {host}",
|
||||
"skipHint": "The host stays as it is and is not touched again in this rollout. A failed maintenance stays open until you close it on the host page.",
|
||||
"reason": "Reason",
|
||||
"abortTitle": "Abort rollout",
|
||||
"abortHint": "Hosts that have not started are no longer updated. An action already running on a firewall is finished in a controlled way (not cut off).",
|
||||
"confirmPlan": "Type the plan name to confirm",
|
||||
"paused": "Rollout paused",
|
||||
"resumed": "Checking and resuming",
|
||||
"approved": "Wave released",
|
||||
"skipped": "Host skipped",
|
||||
"aborted": "Rollout aborted"
|
||||
},
|
||||
"detail": {
|
||||
"back": "All rollouts",
|
||||
"notFound": "Rollout not found",
|
||||
"soakUntil": "Soaking until {time}",
|
||||
"recheckAt": "Re-check at {time}",
|
||||
"blockedSince": "waiting since {time} for someone else's maintenance (max. 15 min)",
|
||||
"waves": "Waves",
|
||||
"hosts": "Hosts",
|
||||
"handover": "with CARP handover",
|
||||
"approval": "approval needed",
|
||||
"approvedBy": "released by {user}",
|
||||
"soak": "{minutes} min soak",
|
||||
"noGate": "No check yet",
|
||||
"attempt": "attempt {n}",
|
||||
"stillRunning": "still finishing: {hosts}",
|
||||
"waitingWindow": "New hosts only start in the next maintenance window.",
|
||||
"awaiting": "The next wave is waiting for release.",
|
||||
"owner": "responsible"
|
||||
},
|
||||
"banner": {
|
||||
"active": "Rollout “{plan}” – wave {wave} ({label}): {status}",
|
||||
"open": "View"
|
||||
},
|
||||
"inRollout": "in rollout",
|
||||
"inRolloutTip": "This host belongs to the running rollout. Manual maintenance is locked here until the rollout has done or skipped it.",
|
||||
"updaterWarning": "A firmware rollout is active. Restarting OPNexus is safe (its state is in the database) but holds the rollout for that time.",
|
||||
"plan": {
|
||||
"new": "New plan",
|
||||
"edit": "Edit plan",
|
||||
"none": "No plans yet",
|
||||
"noneHint": "Create device groups as rings (e.g. ring-canary, ring-pilot, ring-prod) and build waves from them.",
|
||||
"name": "Name",
|
||||
"description": "Description",
|
||||
"action": "Action",
|
||||
"window": "Maintenance window",
|
||||
"noWindow": "none (hosts start right away)",
|
||||
"autoStart": "Start automatically at every window occurrence",
|
||||
"autoStartHint": "Stable updates only. Takes effect after “Activate” and only while plan, groups and window are unchanged; any change deactivates the plan.",
|
||||
"waves": "Waves",
|
||||
"addWave": "Add wave",
|
||||
"waveLabel": "Label",
|
||||
"groups": "Device groups",
|
||||
"soak": "Soak time (min)",
|
||||
"soakHint": "How long to wait after the wave's last host before checking health.",
|
||||
"handover": "CARP handover for MASTER nodes",
|
||||
"approval": "Wait for release before this wave",
|
||||
"moveUp": "Move wave up",
|
||||
"moveDown": "Move wave down",
|
||||
"removeWave": "Remove wave",
|
||||
"checks": "Health gate",
|
||||
"checksHint": "Reachability, version and locks are always checked.",
|
||||
"complianceMode": "Lower security score",
|
||||
"complianceWarn": "hint only",
|
||||
"complianceFail": "stops",
|
||||
"save": "Save",
|
||||
"saved": "Plan saved",
|
||||
"deleted": "Plan deleted",
|
||||
"deleteConfirm": "Delete plan “{name}”? Run history is kept.",
|
||||
"previewStart": "Preview & start",
|
||||
"activate": "Activate",
|
||||
"deactivate": "Deactivate",
|
||||
"active": "automatic: on",
|
||||
"inactive": "automatic: off",
|
||||
"changed": "changed since activation",
|
||||
"changedTip": "The plan, its groups or its window changed since activation. The automatic start is held until the plan is activated again.",
|
||||
"activatedBy": "activated by {user}",
|
||||
"lastAutoStartError": "Last automatic start did not happen ({time}): {reason}",
|
||||
"hostsCount": "{count} hosts",
|
||||
"errors": "{count} errors",
|
||||
"warnings": "{count} hints",
|
||||
"nextOccurrence": "Next occurrence: {time}",
|
||||
"previewTitle": "Preview: {name}",
|
||||
"unattended": "I understand that the firewalls will then be updated and rebooted without further confirmation; the rollout stops by itself on anything unusual.",
|
||||
"start": "Start rollout",
|
||||
"started": "Rollout started",
|
||||
"activated": "Automatic start activated",
|
||||
"deactivated": "Automatic start deactivated",
|
||||
"emptyWave": "no hosts",
|
||||
"disabledHost": "disabled",
|
||||
"cannotStart": "Cannot start like this – see the errors above."
|
||||
},
|
||||
"issue": {
|
||||
"no_waves": "No waves",
|
||||
"no_hosts": "No hosts in the groups",
|
||||
"too_many_waves": "More than {limit} waves",
|
||||
"too_many_hosts": "More than {limit} hosts",
|
||||
"cluster_split": "Cluster {cluster}: for a major upgrade all nodes must be in the same wave.",
|
||||
"cluster_partial": "Cluster {cluster}: not all nodes are in the plan.",
|
||||
"mixed_versions": "Cluster {cluster}: nodes in different waves – mixed versions until wave {wave}.",
|
||||
"master_needs_handover": "Wave {wave} contains clusters ({clusters}) without CARP handover: the rollout stops at the MASTER.",
|
||||
"empty_wave": "Wave {wave} has no hosts.",
|
||||
"duplicate_hosts": "In several waves (the earliest counts): {names}",
|
||||
"disabled_hosts": "Disabled, will be skipped: {names}",
|
||||
"auto_start_update_only": "Automatic start only for stable updates.",
|
||||
"auto_start_needs_window": "Automatic start needs a maintenance window.",
|
||||
"no_window": "No maintenance window: hosts start one after the other right away.",
|
||||
"window_disabled": "The maintenance window is switched off: nothing starts.",
|
||||
"plan_changed": "plan, groups or window changed since activation",
|
||||
"actor_invalid": "{user} may no longer roll out",
|
||||
"plan_invalid": "plan invalid ({reason})",
|
||||
"other_run_active": "another rollout is running ({plan})"
|
||||
},
|
||||
"detailKey": {
|
||||
"wave": "Wave",
|
||||
"failed": "Failed",
|
||||
"host": "Host",
|
||||
"hosts": "Hosts",
|
||||
"cluster": "Cluster",
|
||||
"expected": "Expected",
|
||||
"offered": "Offered",
|
||||
"user": "User",
|
||||
"by": "By",
|
||||
"text": "Reason",
|
||||
"reason": "Reason",
|
||||
"error": "Error",
|
||||
"days": "Days",
|
||||
"still_running": "Still running"
|
||||
}
|
||||
},
|
||||
"mwindows": {
|
||||
"none": "No maintenance windows yet",
|
||||
"noneHint": "A window defines when a rollout may start new hosts, e.g. Sundays 02:00–05:00.",
|
||||
"new": "New window",
|
||||
"edit": "Edit window",
|
||||
"name": "Name",
|
||||
"timezone": "Time zone",
|
||||
"recurrence": "Recurrence",
|
||||
"weekly": "weekly",
|
||||
"monthly": "monthly",
|
||||
"weekdays": "Weekdays",
|
||||
"monthMode": "Monthly on",
|
||||
"nthWeekday": "nth weekday",
|
||||
"dayOfMonth": "day of month",
|
||||
"nth": {
|
||||
"1": "first",
|
||||
"2": "second",
|
||||
"3": "third",
|
||||
"4": "fourth",
|
||||
"-1": "last"
|
||||
},
|
||||
"day": "Day (1–28)",
|
||||
"startTime": "Start (local time)",
|
||||
"duration": "Duration (min)",
|
||||
"cutoff": "New hosts only in the first … min",
|
||||
"cutoffHint": "After that no new host starts; running updates finish. This way an update including reboot fits into the window.",
|
||||
"enabled": "enabled",
|
||||
"dstHint": "Daylight saving: a time that does not exist that day moves forward; a time that exists twice counts the first time.",
|
||||
"saved": "Window saved",
|
||||
"deleted": "Window deleted",
|
||||
"deleteConfirm": "Delete window “{name}”?",
|
||||
"next": "Next occurrences",
|
||||
"openNow": "open now",
|
||||
"off": "switched off",
|
||||
"skipped": "skipped",
|
||||
"moved": "moved",
|
||||
"skip": "Skip",
|
||||
"move": "Move",
|
||||
"moveTo": "New start",
|
||||
"exceptions": "Exceptions",
|
||||
"removeException": "Remove exception",
|
||||
"exceptionSaved": "Exception saved",
|
||||
"usedBy": "Used by: {plans}",
|
||||
"summary": {
|
||||
"weekly": "{days}, {time} ({tz}), {duration} min",
|
||||
"nth": "on the {nth} {day} of the month, {time} ({tz}), {duration} min",
|
||||
"day": "on day {day} of every month, {time} ({tz}), {duration} min"
|
||||
},
|
||||
"dayName": {
|
||||
"0": "Mon",
|
||||
"1": "Tue",
|
||||
"2": "Wed",
|
||||
"3": "Thu",
|
||||
"4": "Fri",
|
||||
"5": "Sat",
|
||||
"6": "Sun"
|
||||
},
|
||||
"dayLong": {
|
||||
"0": "Monday",
|
||||
"1": "Tuesday",
|
||||
"2": "Wednesday",
|
||||
"3": "Thursday",
|
||||
"4": "Friday",
|
||||
"5": "Saturday",
|
||||
"6": "Sunday"
|
||||
},
|
||||
"cutoffSummary": "new hosts only in the first {minutes} min"
|
||||
}
|
||||
}
|
||||
@@ -119,6 +119,8 @@ export const serverMessagesEn: Array<[string | RegExp, string]> = [
|
||||
[/^Manuell abgebrochen, kein bestätigter Abschluss\. Grund: (?<reason>[\s\S]*)$/, "Manually cancelled, no confirmed completion. Reason: {reason}"],
|
||||
["Unerwarteter Fehler beim Ausführen", "Unexpected error during execution"],
|
||||
["Host nicht gefunden oder deaktiviert", "Host not found or disabled"],
|
||||
["Rollout-Schritt nicht rechtzeitig gestartet, nichts ausgeführt", "Rollout step not started in time, nothing was run"],
|
||||
["Rollout angehalten, nicht ausgeführt", "Rollout paused, not run"],
|
||||
[/^Host nicht erreichbar: (?<detail>[\s\S]*)$/, "Host unreachable: {detail}"],
|
||||
[/^Automatisches Backup vor dem Update fehlgeschlagen: (?<detail>[\s\S]*)$/, "Automatic backup before the update failed: {detail}"],
|
||||
[/^Vorbereitung der CARP-Übergabe fehlgeschlagen, nichts wurde verändert: (?<detail>[\s\S]*)$/, "Preparing the CARP handover failed, nothing was changed: {detail}"],
|
||||
@@ -203,4 +205,34 @@ export const serverMessagesEn: Array<[string | RegExp, string]> = [
|
||||
["Für diese Aktion wird ein Zertifikat (PEM) benötigt", "A certificate (PEM) is required for this action"],
|
||||
["Ein privater Schlüssel kann nur bei Aktion 'import' hinterlegt werden", "A private key can only be stored for the “import” action"],
|
||||
["Für CA 'custom' wird eine Custom-CA-URL benötigt", "A custom CA URL is required for CA “custom”"],
|
||||
|
||||
// --- notification channels, webhook address check, monitoring alerts (1.24) ---
|
||||
["Die Adresse ist leer, zu lang oder enthält Leerzeichen", "The address is empty, too long or contains spaces"],
|
||||
["Die Adresse ist ungültig", "The address is invalid"],
|
||||
["Die Adresse muss mit http:// oder https:// beginnen", "The address must start with http:// or https://"],
|
||||
["Zugangsdaten gehören nicht in die Adresse", "Credentials do not belong in the address"],
|
||||
["Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt", "This address (local, link-local or metadata service) is not allowed as a target"],
|
||||
["Der Name zeigt auf eine nicht erlaubte Adresse (lokal, Link-Local oder Metadaten-Dienst)", "The name points to an address that is not allowed (local, link-local or metadata service)"],
|
||||
["Unbekannter Kanaltyp", "Unknown channel type"],
|
||||
[/^(?<key>\w+): ganze Zahl erwartet$/, "{key}: whole number expected"],
|
||||
[/^(?<key>\w+): Wert zwischen (?<low>-?\d+) und (?<high>-?\d+) erwartet$/, "{key}: value between {low} and {high} expected"],
|
||||
["Thema: 1–64 Zeichen aus Buchstaben, Ziffern, _ und -", "Topic: 1–64 characters (letters, digits, _ and -)"],
|
||||
["Serveradresse fehlt", "Server address missing"],
|
||||
["Chat-ID: Zahl (z. B. -1001234567890) oder @Kanalname", "Chat ID: a number (e.g. -1001234567890) or @channelname"],
|
||||
["Homeserver-Adresse fehlt", "Homeserver address missing"],
|
||||
["Raum-ID: Form !abc123:example.org (nicht der Alias #raum:…)", "Room ID: form !abc123:example.org (not the alias #room:…)"],
|
||||
["Zugangsdaten fehlen", "Credentials missing"],
|
||||
["Konfiguration unvollständig", "Configuration incomplete"],
|
||||
["Die Teams-Webhook-Adresse muss mit https:// beginnen", "The Teams webhook address must start with https://"],
|
||||
["Token: keine Leerzeichen, höchstens 500 Zeichen", "Token: no spaces, at most 500 characters"],
|
||||
["Zugangsdaten nicht lesbar – bitte neu setzen", "Credentials not readable – please set them again"],
|
||||
["Zustellung abgelehnt", "Delivery refused"],
|
||||
[/^Zustellung fehlgeschlagen \((?<reason>[^)]+)\)$/, "Delivery failed ({reason})"],
|
||||
["Antwort nicht lesbar", "Response not readable"],
|
||||
[/^Kanal „(?<name>[\s\S]+?)“ \((?<type>\w+)\): (?<reason>[\s\S]+)$/, "Channel “{name}” ({type}): {reason}"],
|
||||
[/^Kanal „(?<name>[\s\S]+?)“: (?<reason>[\s\S]+)$/, "Channel “{name}”: {reason}"],
|
||||
["Dienstnamen bestehen aus Buchstaben, Ziffern und . _ + -", "Service names consist of letters, digits and . _ + -"],
|
||||
["Arbeitsspeicher", "Memory"],
|
||||
["CPU-Last (pro Thread)", "CPU load (per thread)"],
|
||||
[/^Festplatte (?<disk>[\s\S]+)$/, "Disk {disk}"],
|
||||
];
|
||||
@@ -0,0 +1,78 @@
|
||||
// Shared display helpers for firmware rollouts (1.25): status tones and translated codes.
|
||||
import type { BadgeTone } from "@/components/ui/Badge";
|
||||
import type { TranslationParams } from "@/lib/i18n";
|
||||
import type { MaintenanceWindow, RolloutCheckResult, RolloutHostStatus, RolloutIssue, RolloutRunStatus } from "@/lib/api";
|
||||
|
||||
type Translate = (key: string, params?: TranslationParams) => string;
|
||||
|
||||
export const RUN_TONE: Record<RolloutRunStatus, BadgeTone> = {
|
||||
waiting_window: "neutral", preflight: "info", wave_running: "info", soaking: "info", gate: "info",
|
||||
awaiting_approval: "accent", resuming: "info", paused: "warning", succeeded: "success", cancelled: "neutral",
|
||||
};
|
||||
|
||||
export const HOST_TONE: Record<RolloutHostStatus, BadgeTone> = {
|
||||
queued: "neutral", running: "info", verifying: "info", succeeded: "success", skipped: "neutral",
|
||||
failed: "danger", blocked: "warning", cancelled: "neutral",
|
||||
};
|
||||
|
||||
export const WAVE_TONE: Record<string, BadgeTone> = {
|
||||
pending: "neutral", awaiting_approval: "accent", preflight: "info", running: "info", soaking: "info", gate: "info",
|
||||
passed: "success", skipped: "neutral", cancelled: "neutral",
|
||||
};
|
||||
|
||||
export const RESULT_TONE: Record<RolloutCheckResult["result"], BadgeTone> = { ok: "success", warn: "warning", fail: "danger", skip: "neutral" };
|
||||
|
||||
export const CHECKS = ["online", "version", "carp", "gateways", "services", "locks", "compliance"] as const;
|
||||
|
||||
function stringParams(params: Record<string, unknown> | null | undefined): TranslationParams {
|
||||
const out: TranslationParams = {};
|
||||
for (const [k, v] of Object.entries(params ?? {})) out[k] = v === null || v === undefined ? "" : typeof v === "number" ? v : String(v);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** A translated text for a {code, params} pair under `prefix`, with a readable fallback for unknown codes. */
|
||||
export function codeText(t: Translate, prefix: string, issue: RolloutIssue | RolloutCheckResult | null | undefined): string {
|
||||
if (!issue) return "";
|
||||
const key = `${prefix}.${issue.code}`;
|
||||
const text = t(key, stringParams(issue.params));
|
||||
return text === key ? issue.code : text;
|
||||
}
|
||||
|
||||
export function pauseText(t: Translate, reason: string | null): string {
|
||||
if (!reason) return "";
|
||||
const key = `rollouts.pauseReason.${reason}`;
|
||||
const text = t(key);
|
||||
return text === key ? reason : text;
|
||||
}
|
||||
|
||||
function failedItem(t: Translate, item: string): string {
|
||||
const [host, check] = item.split(":");
|
||||
const key = `rollouts.gate.check.${check}`;
|
||||
const label = t(key);
|
||||
return check ? `${host}: ${label === key ? check : label}` : item;
|
||||
}
|
||||
|
||||
/** Human-readable pause details: known keys translated, failed checks as "host: check". */
|
||||
export function pauseDetailText(t: Translate, detail: Record<string, unknown> | null | undefined): string {
|
||||
if (!detail) return "";
|
||||
return Object.entries(detail)
|
||||
.filter(([k, v]) => !["on_resume"].includes(k) && v !== null && v !== "" && !(Array.isArray(v) && v.length === 0))
|
||||
.map(([k, v]) => {
|
||||
const key = `rollouts.detailKey.${k}`;
|
||||
const label = t(key) === key ? k : t(key);
|
||||
const value = k === "failed" && Array.isArray(v) ? v.map((x) => failedItem(t, String(x))).join(", ") : Array.isArray(v) ? v.join(", ") : String(v);
|
||||
return `${label}: ${value}`;
|
||||
})
|
||||
.join(" · ");
|
||||
}
|
||||
|
||||
export function windowSummary(t: Translate, w: MaintenanceWindow): string {
|
||||
const base = { time: w.start_time, tz: w.timezone, duration: w.duration_minutes };
|
||||
if (w.recurrence === "weekly") {
|
||||
return t("mwindows.summary.weekly", { ...base, days: w.weekdays.map((d) => t(`mwindows.dayName.${d}`)).join(", ") });
|
||||
}
|
||||
if (w.month_mode === "day") return t("mwindows.summary.day", { ...base, day: w.month_day ?? "" });
|
||||
return t("mwindows.summary.nth", { ...base, nth: t(`mwindows.nth.${w.month_nth}`), day: t(`mwindows.dayLong.${w.month_weekday}`) });
|
||||
}
|
||||
|
||||
export const isActiveRun = (status: RolloutRunStatus) => !["succeeded", "cancelled"].includes(status);
|
||||
@@ -1,6 +1,6 @@
|
||||
// Manually bumped whenever a notable change ships — mirrors the dated
|
||||
// entries already kept in the project README/CHANGELOG.md, no build
|
||||
// tooling involved. Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
export const APP_VERSION = "1.23.10";
|
||||
export const APP_CODENAME = "Harrier";
|
||||
export const APP_VERSION = "1.25.0";
|
||||
export const APP_CODENAME = "Goshawk";
|
||||
export const APP_MAINTAINER = "Julian Boldt";
|
||||
@@ -9,6 +9,11 @@
|
||||
# scripts/build_release_images.sh v1.10.0 --push # bauen und pushen
|
||||
# scripts/build_release_images.sh v1.10.0 --push --multiarch # amd64+arm64 (braucht QEMU/buildx)
|
||||
#
|
||||
# Einmalig pro Build-Host für --multiarch (scripts/release_all.sh nutzt es immer):
|
||||
# docker run --privileged --rm tonistiigi/binfmt --install arm64 # QEMU-Emulation (bis zum Neustart)
|
||||
# docker buildx create --name opnexus-multiarch --driver docker-container --bootstrap
|
||||
# Der Builder lässt sich mit OPNEXUS_BUILDER ändern; fehlt er, bricht das Skript mit dieser Anleitung ab.
|
||||
#
|
||||
# Signieren (cosign): OPNEXUS_SIGN=1 COSIGN_KEY=/pfad/cosign.key [COSIGN_PASSWORD=…] — der öffentliche
|
||||
# Schlüssel liegt als cosign.pub im Repo; die Registry muss ihren Token-Realm per https melden.
|
||||
# REPO: Umgebungsvariable OPNEXUS_IMAGE_REPO (Standard git.opnexus.dev/opnexus).
|
||||
@@ -24,6 +29,17 @@ for a in "$@"; do case "$a" in --push) push=1;; --multiarch) multi=1;; *) echo "
|
||||
[ "$multi" = 0 ] || [ "$push" = 1 ] || { echo "--multiarch geht nur zusammen mit --push" >&2; exit 1; }
|
||||
git rev-parse -q --verify "refs/tags/$tag^{commit}" >/dev/null || { echo "Tag $tag existiert lokal nicht" >&2; exit 1; }
|
||||
repo="${OPNEXUS_IMAGE_REPO:-git.opnexus.dev/opnexus}"
|
||||
builder="${OPNEXUS_BUILDER:-opnexus-multiarch}"
|
||||
if [ "$multi" = 1 ]; then
|
||||
docker buildx inspect "$builder" >/dev/null 2>&1 \
|
||||
|| { echo "buildx-Builder '$builder' fehlt: docker buildx create --name $builder --driver docker-container --bootstrap" >&2; exit 1; }
|
||||
# Der Cross-Build füllt den Build-Cache um mehrere GB; ein volles Dateisystem legt auch die laufenden Dienste lahm.
|
||||
free_mb="$(df --output=avail -BM . | tail -1 | tr -dc 0-9)"
|
||||
[ "$free_mb" -ge 3500 ] || { echo "Nur ${free_mb} MB frei (für --multiarch werden ca. 3500 MB gebraucht): Platz schaffen (docker image prune, docker buildx prune)" >&2; exit 1; }
|
||||
# Emulation ist nach einem Neustart weg; ohne sie scheitert der arm64-Teil erst nach Minuten.
|
||||
docker buildx inspect "$builder" | grep -q 'linux/arm64' \
|
||||
|| { echo "arm64 nicht verfügbar: docker run --privileged --rm tonistiigi/binfmt --install arm64" >&2; exit 1; }
|
||||
fi
|
||||
version="${tag#v}"
|
||||
|
||||
# Aus dem Tag bauen, nicht aus dem Working Tree (keine uncommitteten Änderungen im Image).
|
||||
@@ -42,7 +58,8 @@ for svc in backend frontend proxy updater; do
|
||||
targs=(); for t in "${tags[@]}"; do targs+=(-t "$image:$t"); done
|
||||
echo "==> $image (${tags[*]})"
|
||||
if [ "$multi" = 1 ]; then
|
||||
docker buildx build --platform linux/amd64,linux/arm64 "${targs[@]}" --push "$src/$svc"
|
||||
docker buildx build --builder "$builder" --platform linux/amd64,linux/arm64 "${targs[@]}" --push "$src/$svc"
|
||||
docker buildx prune --builder "$builder" -af >/dev/null 2>&1 || true # Platz für den nächsten Dienst
|
||||
else
|
||||
docker build "${targs[@]}" "$src/$svc"
|
||||
if [ "$push" = 1 ]; then for t in "${tags[@]}"; do docker push "$image:$t"; done; fi
|
||||
|
||||
@@ -5,6 +5,20 @@ FROM python:3.12-alpine
|
||||
|
||||
RUN apk add --no-cache bash docker-cli docker-cli-compose
|
||||
|
||||
# cosign prüft vor einem Update die Signatur der neuen Images (cosign.pub liegt im gemounteten
|
||||
# Projektverzeichnis, siehe scripts/opnexus). Version und Prüfsummen sind fest eingetragen.
|
||||
ARG COSIGN_VERSION=v3.1.3
|
||||
ARG TARGETARCH
|
||||
RUN set -e; arch="${TARGETARCH:-amd64}"; \
|
||||
case "$arch" in \
|
||||
amd64) sha=4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71 ;; \
|
||||
arm64) sha=c5d324e091826b0d7a78eb16fef316450b4eb9aaec045611c08ba06f5e73220a ;; \
|
||||
*) echo "Architektur $arch nicht unterstützt" >&2; exit 1 ;; \
|
||||
esac; \
|
||||
python -c "import urllib.request; urllib.request.urlretrieve('https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-${arch}', '/usr/local/bin/cosign')"; \
|
||||
echo "$sha /usr/local/bin/cosign" | sha256sum -c -; \
|
||||
chmod 0755 /usr/local/bin/cosign
|
||||
|
||||
COPY opnexus /usr/local/bin/opnexus
|
||||
COPY server.py /app/server.py
|
||||
RUN chmod 0755 /usr/local/bin/opnexus
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Reference in new issue
Block a user