Release v1.25.0

Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).

- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
  `ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
  bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
  gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
  folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
  Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
  (nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
  Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
  Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
  Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
  nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
  eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
  älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
  Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
  zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
  (nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
  der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
  vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
  erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
  sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
  gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
  laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
  sicher, wird aber angekündigt.
- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
  26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
  Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
  Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
  übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
  Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
  Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.

CHANGELOG.md

@@ -31,6 +31,46 @@ den Einstellungen.
 Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
 ansteht): Merlin …
 
+## [1.25.0] "Goshawk" — 2026-10-21
+
+Geplant als MINOR-Release **1.25.0** (Flotten-Rollout).
+
+- **Rollouts in Wellen (System › Rollouts):** Firmware-Updates für die ganze Flotte in Ringen, z. B. erst
+  `ring-canary`, dann `ring-pilot`, dann `ring-prod` (Wellen aus Gerätegruppen). Jeder Host läuft über die
+  bestehende geplante Wartung (Backup vor dem Update, CARP-Übergabe für den MASTER, Audit), immer nur ein Host
+  gleichzeitig und nur ein Rollout zur selben Zeit; im Cluster zuerst der BACKUP, dann der MASTER. Nach jeder Welle
+  folgen eine Beobachtungszeit und eine **Gesundheitsprüfung** (erreichbar, Zielversion installiert, CARP je VHID,
+  Gateways und Dienste wie vorher, keine offenen Sperren, optional Sicherheits-Score). Rot hält den Rollout an
+  (nach einer Nachprüfung zwei Minuten später) und meldet sich. Anhalten, Fortsetzen (prüft zuerst alles bisher
+  Aktualisierte erneut), Welle freigeben, Host überspringen und Abbrechen auf der Lauf-Seite; die Seite zeigt
+  Wellen, Hosts (Version vorher → nachher, aktueller Schritt) und die Prüfmatrix.
+- **Sicherheitsnetz:** Weicht die angebotene Version in einem späteren Ring von der des ersten Rings ab, hält der
+  Rollout an (OPNsense installiert immer die neueste). Ein Host ohne Update wird übersprungen, ein fehlgeschlagener
+  nie von selbst wiederholt. Angehalten wird auch, wenn die verantwortliche Person das Recht zum Ausrollen verliert,
+  eine fremde Wartung im Cluster länger als 15 Minuten offen ist, der MASTER ohne Übergabe dran wäre oder ein Lauf
+  älter als 14 Tage ist. Höchstens 10 Wellen und 100 Hosts je Plan.
+- **Wiederkehrende Wartungsfenster:** wöchentlich oder monatlich (n-ter Wochentag, letzter, fester Tag) in einer
+  Zeitzone, mit Sommerzeit-Regel, Abschneidezeit („neue Hosts nur in den ersten N Minuten“) und einzelnen Terminen
+  zum Auslassen oder Verschieben. Ein Plan mit Fenster startet neue Hosts nur darin; mit **automatischem Start**
+  (nur stabile Updates) beginnt er zu jedem Termin von selbst, aber nur, solange Plan, Gruppen und Fenster seit
+  der Aktivierung unverändert sind. Jede Änderung deaktiviert ihn.
+- **Geplante Wartung prüft sich selbst:** Neue Option „Ergebnis automatisch prüfen und abschließen“ (im Dialog
+  vorausgewählt, bei Rollouts immer an). Nach dem Auftrag prüft OPNexus, ob die Firmware wieder bereit ist, der
+  erwartete Neustart stattfand und die Version lesbar ist, und schließt die Wartung dann selbst ab. Bisher blieb
+  sie auf „übergeben“ und sperrte Host und Cluster, bis jemand sie von Hand abschloss.
+- **Benachrichtigungen:** „Rollout angehalten“ und „Rollout nicht automatisch gestartet“ (ein Schalter, standardmäßig an).
+- **Rechte:** Rollout starten/aktivieren/fortsetzen/freigeben/überspringen/abbrechen und Wartungsfenster ändern
+  gelten als Ausrollen (Aufgabentrennung); **Anhalten** darf jeder mit Schreibrecht. Update Center: Hosts im
+  laufenden Rollout sind für manuelle Firmware-Aktionen gesperrt. Ein OPNexus-Update während eines Rollouts ist
+  sicher, wird aber angekündigt.
+- **Getestet:** live auf den Test-Firewalls (OPNsense 26.7): automatischer Start im Fenster, Canary-Update
+  26.7.1_1 → 26.7.7, absichtlich gestoppter Dienst → Gate rot → angehalten (zweiter Ring unberührt,
+  Benachrichtigung), Fortsetzen, Pilot-Update 26.7.6 → 26.7.7 auf dem damaligen MASTER mit CARP-Übergabe und einem
+  Neustart des OPNexus-Backends mitten im Auftrag (genau ein Update-Auftrag), Freigabe, Host ohne Update
+  übersprungen, Anhalten/Fortsetzen/Abbrechen von Hand, Termin auslassen. **Nur gemockt:** Major-Upgrade als
+  Rollout, ein Cluster in *einer* Welle, Cluster-Prüfung vor dem zweiten Knoten, abweichende Zielversion,
+  Zeitüberschreitungen, Rechteverlust, Fenster schließt mitten im Lauf; die Sommerzeit-Regeln sind Rechentests.
+
 ## [1.24.0] "Merlin" — 2026-10-21
 
 - **Alarme für Gateways, Ressourcen und Dienste:** Neue Ereignisse in Einstellungen › Benachrichtigungen.

README.de.md

@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
 
 ![Zentrale Firewall-Regeln](docs/screenshots/readme-rules-de.webp)
 
-> **Stand:** `1.24.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
+> **Stand:** `1.25.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
 
 ## Funktionen
 
@@ -20,7 +20,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
 - **Zertifikate** — Trust-Store für alle Firewalls und Ausstellung per ACME / Let's Encrypt.
 - **Alarme** — Gateway-/WAN-Ausfall (down, Paketverlust, Latenz), Schwellen für Platte, Speicher, Swap und CPU sowie ausgefallene Dienste; über mehrere Messungen entprellt, damit Einzelspitzen und flackernde Verbindungen schweigen. Zustellung per E-Mail, Webhook, ntfy, Gotify, Telegram, Microsoft Teams oder Matrix, jedes Ereignis einzeln schaltbar.
 - **Logs & Audit** — zentrale Log-Suche mit Korrelation, Dashboards und lückenloser Deploy-Verlauf.
-- **Update-Center** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus.
+- **Update-Center und Rollouts** — Firmware-Updates für die Flotte, mit Blick auf HA-Reihenfolge und Wartungsmodus; Rollouts in Wellen (Canary → Pilot → Prod) mit wiederkehrenden Wartungsfenstern, Gesundheitsprüfung zwischen den Wellen und automatischem Halt bei jeder Auffälligkeit.
 - **Einrichtungsassistent** — geführter Erststart nach der Installation: eigenes Passwort, Zwei-Faktor und die erste Firewall verbinden.
 - **Selbst-Update** — OPNexus aktualisiert sich auf Wunsch aus der eigenen Oberfläche, mit Fortschritt und automatischem Rollback (optional).
 - **Automatisierung** — persönliche API-Tokens (Lesen / Ausrollen / Admin), Prometheus-Metriken, ein nur lesender MCP-Endpunkt für KI-Assistenten und ein OpenAPI-Schema.
@@ -63,7 +63,7 @@ Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelie
 
 ```bash
 OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
-OPNEXUS_VERSION=1.24.0
+OPNEXUS_VERSION=1.25.0
 ```
 
 ```bash

README.md

@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
 
 ![Central firewall rules](docs/screenshots/readme-rules-en.webp)
 
-> **Status:** `1.24.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
+> **Status:** `1.25.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
 
 ## What it does
 
@@ -20,7 +20,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
 - **Certificates** — trust store for all firewalls and ACME / Let's Encrypt issuance.
 - **Alerts** — gateway/WAN outage (down, packet loss, latency), disk/memory/swap/CPU thresholds and stopped services, debounced over several measurements so spikes and flapping links stay quiet; delivered by email, webhook, ntfy, Gotify, Telegram, Microsoft Teams or Matrix, all switchable per event.
 - **Logs & audit** — central log search with correlation, dashboards and a complete deploy audit trail.
-- **Update center** — firmware updates for the fleet, aware of HA order and maintenance mode.
+- **Update center & rollouts** — firmware updates for the fleet, aware of HA order and maintenance mode; rollouts in waves (canary → pilot → prod) with recurring maintenance windows, a health gate between the waves and an automatic stop on anything unusual.
 - **Setup assistant** — a guided first run after installation: your own password, two-factor, and connecting the first firewall.
 - **Self-update** — OPNexus can update itself from its web interface, with progress and automatic rollback (opt-in).
 - **Automation** — personal API tokens (read / deploy / admin), Prometheus metrics, a read-only MCP endpoint for AI assistants and an OpenAPI schema.
@@ -65,7 +65,7 @@ For access from other machines and for production use with TLS (bundled Caddy pr
 
 ```bash
 OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
-OPNEXUS_VERSION=1.24.0
+OPNEXUS_VERSION=1.25.0
 ```
 
 ```bash

backend/app/auth_middleware.py

@@ -56,9 +56,9 @@ _READ_ONLY_ALLOWED_WRITES = {
 _SECURITY_KINDS = {
     "ipsec": "ipsec_tunnel", "openvpn": "openvpn_instance", "wireguard": "wireguard_tunnel",
     "certificates": "certificate", "acme": "acme", "notifications": "notification", "auth": "auth",
-    "tokens": "api_token",
+    "tokens": "api_token", "rollouts": "rollout",
 }
-_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings"}
+_COLLECTIONS = {"tunnels", "instances", "managed", "accounts", "certificates", "settings", "plans", "runs"}
 
 
 def _is_uuid(value: str) -> bool:
@@ -117,7 +117,7 @@ class AuthMiddleware(BaseHTTPMiddleware):
                 )
             if user is not None:
                 parts = request.url.path.strip("/").split("/")
-                kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup"}
+                kinds = {"rules": "rule", "aliases": "alias", "port-forward": "port_forward", "one-to-one": "one_to_one", "outbound-nat": "outbound_nat", "hosts": "host", "users": "user", "system": "system", "backups": "backup", "maintenance-windows": "maintenance_window"}
                 writing = request.method in ("POST", "PUT", "PATCH", "DELETE")
                 if len(parts) > 1 and parts[1] in _SECURITY_KINDS:
                     event = {**_security_event(parts, request.method), "details": {"method": request.method, "path": request.url.path}}

backend/app/errors.py

@@ -179,6 +179,27 @@ class ErrorCode(str, Enum):
     API_TOKEN_NOT_FOUND = "API_TOKEN_NOT_FOUND"
     API_TOKEN_SCOPE_FORBIDDEN = "API_TOKEN_SCOPE_FORBIDDEN"
     API_TOKEN_SCOPE_TOO_HIGH = "API_TOKEN_SCOPE_TOO_HIGH"
+    WINDOW_NOT_FOUND = "WINDOW_NOT_FOUND"
+    WINDOW_INVALID = "WINDOW_INVALID"
+    WINDOW_NAME_TAKEN = "WINDOW_NAME_TAKEN"
+    WINDOW_IN_USE = "WINDOW_IN_USE"
+    WINDOW_EXCEPTION_INVALID = "WINDOW_EXCEPTION_INVALID"
+    WINDOW_EXCEPTION_EXISTS = "WINDOW_EXCEPTION_EXISTS"
+    WINDOW_EXCEPTION_NOT_FOUND = "WINDOW_EXCEPTION_NOT_FOUND"
+    ROLLOUT_PLAN_NOT_FOUND = "ROLLOUT_PLAN_NOT_FOUND"
+    ROLLOUT_INVALID = "ROLLOUT_INVALID"
+    ROLLOUT_NAME_TAKEN = "ROLLOUT_NAME_TAKEN"
+    ROLLOUT_PLAN_INVALID = "ROLLOUT_PLAN_INVALID"
+    ROLLOUT_PREVIEW_STALE = "ROLLOUT_PREVIEW_STALE"
+    ROLLOUT_PLAN_BUSY = "ROLLOUT_PLAN_BUSY"
+    ROLLOUT_GROUP_IN_USE = "ROLLOUT_GROUP_IN_USE"
+    ROLLOUT_RUN_NOT_FOUND = "ROLLOUT_RUN_NOT_FOUND"
+    ROLLOUT_ALREADY_ACTIVE = "ROLLOUT_ALREADY_ACTIVE"
+    ROLLOUT_CONFIRM_NAME = "ROLLOUT_CONFIRM_NAME"
+    ROLLOUT_ACTION_NOT_ALLOWED = "ROLLOUT_ACTION_NOT_ALLOWED"
+    ROLLOUT_HOST_NOT_IN_RUN = "ROLLOUT_HOST_NOT_IN_RUN"
+    ROLLOUT_HOST_RUNNING = "ROLLOUT_HOST_RUNNING"
+    ROLLOUT_HOST_FAILED = "ROLLOUT_HOST_FAILED"
 
 
 def api_error(status_code: int, code: ErrorCode, **params: Any) -> HTTPException:

backend/app/main.py

@@ -50,7 +50,10 @@ from app.routers import gateways as gateways_router
 from app.routers import resource_samples as resource_samples_router
 from app.services import resource_monitor_service
 from app.routers import scheduled_maintenance as scheduled_maintenance_router
+from app.routers import maintenance_windows as maintenance_windows_router
+from app.routers import rollouts as rollouts_router
 from app.services import scheduled_maintenance_service
+from app.services import rollout_engine
 from app.services import compliance_service
 from app.services import audit_export_service
 from app.services import overlay_service, variables_service
@@ -255,6 +258,12 @@ async def _scheduled_maintenance_poll_loop():
         except Exception:
             logger.exception("Geplante-Wartung-Poll fehlgeschlagen")
             in_progress = 0
+        # Firmware rollouts (1.25) only ever create schedules; the next tick of the lines above fires them.
+        try:
+            async with SessionLocal() as db:
+                in_progress += await rollout_engine.advance_runs(db)
+        except Exception:
+            logger.exception("Rollout-Poll fehlgeschlagen")
         # A handover run is stepped quickly (the partner should take over within a minute or two);
         # otherwise the relaxed cadence is plenty.
         await asyncio.sleep(_RUNNING_MAINTENANCE_POLL_SECONDS if in_progress else _SCHEDULED_MAINTENANCE_POLL_SECONDS)
@@ -518,3 +527,5 @@ app.include_router(network_drift_router.router, prefix="/api")
 app.include_router(diagnostics_router.router, prefix="/api")
 app.include_router(resource_samples_router.router, prefix="/api")
 app.include_router(scheduled_maintenance_router.router, prefix="/api")
+app.include_router(maintenance_windows_router.router, prefix="/api")
+app.include_router(rollouts_router.router, prefix="/api")

backend/app/models/__init__.py

@@ -93,3 +93,9 @@ __all__ += ["HostResourceSample"]
 
 from app.models.scheduled_maintenance import ScheduledMaintenance
 __all__ += ["ScheduledMaintenance"]
+
+from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
+__all__ += ["MaintenanceWindow", "MaintenanceWindowException"]
+
+from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun, RolloutRunHost, RolloutRunWave, rollout_wave_groups
+__all__ += ["RolloutPlan", "RolloutPlanWave", "RolloutRun", "RolloutRunHost", "RolloutRunWave", "rollout_wave_groups"]

backend/app/models/maintenance_window.py

@@ -0,0 +1,65 @@
+"""Recurring maintenance windows (1.25): "every Sunday 02:00-05:00 Europe/Berlin",
+"first Tuesday of the month 22:00 for 3 h". A window on its own changes nothing on any
+firewall; rollout plans (models/rollout.py) use it to decide WHEN they may start work on a host.
+
+Times are wall-clock times in the window's own IANA time zone, so a window stays at 02:00
+local time across daylight-saving changes (see services/window_service.py for the rules).
+"""
+import uuid
+from datetime import date, datetime, time, timezone
+
+from sqlalchemy import JSON, Boolean, Date, DateTime, ForeignKey, Integer, String, Time, UniqueConstraint
+from sqlalchemy.dialects.postgresql import UUID as PGUUID
+from sqlalchemy.orm import Mapped, mapped_column, relationship
+
+from app.database import Base
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+class MaintenanceWindow(Base):
+    __tablename__ = "maintenance_windows"
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    name: Mapped[str] = mapped_column(String(120), unique=True)
+    timezone: Mapped[str] = mapped_column(String(64))
+    recurrence: Mapped[str] = mapped_column(String(16))  # weekly | monthly
+    # weekly: weekdays 0 (Monday) .. 6 (Sunday). JSON columns get a Python default only (Alembic gotcha, see 0007).
+    weekdays: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
+    # monthly: "nth_weekday" (month_nth 1-4 or -1 = last, month_weekday 0-6) or "day" (month_day 1-28)
+    month_mode: Mapped[str | None] = mapped_column(String(16), nullable=True)
+    month_nth: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    month_weekday: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    month_day: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    start_time: Mapped[time] = mapped_column(Time)
+    duration_minutes: Mapped[int] = mapped_column(Integer)
+    # After this many minutes into the window no NEW host action starts (running ones finish).
+    start_cutoff_minutes: Mapped[int] = mapped_column(Integer, nullable=False, default=60, server_default="60")
+    enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
+    created_by: Mapped[str] = mapped_column(String(150))
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
+
+    exceptions: Mapped[list["MaintenanceWindowException"]] = relationship(
+        "MaintenanceWindowException", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
+        order_by="MaintenanceWindowException.occurrence_date",
+    )
+
+
+class MaintenanceWindowException(Base):
+    """One occurrence skipped or moved. Keyed by the occurrence's LOCAL date."""
+
+    __tablename__ = "maintenance_window_exceptions"
+    __table_args__ = (UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),)
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    window_id: Mapped[uuid.UUID] = mapped_column(
+        PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="CASCADE"), index=True
+    )
+    occurrence_date: Mapped[date] = mapped_column(Date)
+    kind: Mapped[str] = mapped_column(String(8))  # skip | move
+    moved_start: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    reason: Mapped[str] = mapped_column(String(300), default="")
+    actor: Mapped[str] = mapped_column(String(150))
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)

backend/app/models/notification.py

@@ -68,6 +68,8 @@ class NotificationSettings(Base):
     gateway_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
     resource_threshold_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
     service_down_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
+    # Rollouts (1.25): a run paused (gate, failure, window...) or an automatic start did not happen.
+    rollout_problems_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
     # Gateway: OPNsense' own verdict (down / loss / delay) decides "offline"; these two add an
     # OPNexus threshold on the measured values (0 = off).
     gateway_loss_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=20, server_default="20")

backend/app/models/rollout.py

@@ -0,0 +1,149 @@
+"""Firmware rollouts in waves (1.25): plans (what, in which order, when) and runs (what happened).
+
+A plan's waves point at device groups ("rings": canary -> pilot -> prod). A run freezes the plan
+(``plan_snapshot``) and the resolved hosts (``RolloutRunHost``) when it starts, so later edits of
+the plan or of a group never change a run that is under way. The run never fires firmware actions
+itself: every host step is an ordinary ScheduledMaintenance row (origin="rollout",
+auto_verify=True) linked from ``RolloutRunHost.scheduled_maintenance_id`` — see
+services/rollout_service.py.
+"""
+import uuid
+from datetime import datetime, timezone
+
+from sqlalchemy import (JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Table, Text,
+                        UniqueConstraint)
+from sqlalchemy.dialects.postgresql import UUID as PGUUID
+from sqlalchemy.orm import Mapped, mapped_column, relationship
+
+from app.database import Base
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+rollout_wave_groups = Table(
+    "rollout_wave_groups", Base.metadata,
+    Column("wave_id", PGUUID(as_uuid=True), ForeignKey("rollout_plan_waves.id", ondelete="CASCADE"), primary_key=True),
+    Column("group_id", PGUUID(as_uuid=True), ForeignKey("device_groups.id", ondelete="CASCADE"), primary_key=True),
+)
+
+
+class RolloutPlan(Base):
+    __tablename__ = "rollout_plans"
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    name: Mapped[str] = mapped_column(String(120), unique=True)
+    description: Mapped[str] = mapped_column(Text, default="")
+    action: Mapped[str] = mapped_column(String(16))  # update | major_update
+    window_id: Mapped[uuid.UUID | None] = mapped_column(
+        PGUUID(as_uuid=True), ForeignKey("maintenance_windows.id", ondelete="SET NULL"), nullable=True, index=True)
+    # Start a run by itself at every occurrence of the window. Only for "update" and only while the plan
+    # is activated AND unchanged since (activated_fingerprint).
+    auto_start: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
+    gate_checks: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
+    compliance_mode: Mapped[str] = mapped_column(String(8), nullable=False, default="warn", server_default="warn")
+    activated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
+    activated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    activated_fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    # Why the last automatic start did not happen: {"occurrence", "code", "params", "at"} (notification source).
+    auto_start_error: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    created_by: Mapped[str] = mapped_column(String(150))
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+    waves: Mapped[list["RolloutPlanWave"]] = relationship(
+        "RolloutPlanWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
+        order_by="RolloutPlanWave.position")
+
+
+class RolloutPlanWave(Base):
+    __tablename__ = "rollout_plan_waves"
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    plan_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="CASCADE"), index=True)
+    position: Mapped[int] = mapped_column(Integer)
+    label: Mapped[str] = mapped_column(String(60))
+    soak_minutes: Mapped[int] = mapped_column(Integer, default=30)
+    carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
+    # Wait for a human "release" before this wave starts (not used for the first wave: starting is the release).
+    require_approval: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
+
+    groups: Mapped[list["DeviceGroup"]] = relationship("DeviceGroup", secondary=rollout_wave_groups, lazy="selectin")  # noqa: F821
+
+
+class RolloutRun(Base):
+    __tablename__ = "rollout_runs"
+    __table_args__ = (UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),)
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    plan_id: Mapped[uuid.UUID | None] = mapped_column(
+        PGUUID(as_uuid=True), ForeignKey("rollout_plans.id", ondelete="SET NULL"), nullable=True, index=True)
+    plan_name: Mapped[str] = mapped_column(String(120))
+    plan_snapshot: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
+    action: Mapped[str] = mapped_column(String(16))
+    # waiting_window | preflight | wave_running | soaking | gate | awaiting_approval | resuming | paused
+    # | succeeded | cancelled
+    status: Mapped[str] = mapped_column(String(20), index=True)
+    paused_from: Mapped[str | None] = mapped_column(String(20), nullable=True)
+    current_wave: Mapped[int] = mapped_column(Integer, default=0)
+    target_version: Mapped[str | None] = mapped_column(String(40), nullable=True)
+    pause_reason: Mapped[str | None] = mapped_column(String(40), nullable=True)
+    pause_detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    started_by: Mapped[str] = mapped_column(String(150))
+    trigger: Mapped[str] = mapped_column(String(8))  # manual | window
+    window_occurrence: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    # Optimistic lock: every state change bumps it; an engine step computed on an older revision is dropped.
+    revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    progress: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
+    step_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+
+    waves: Mapped[list["RolloutRunWave"]] = relationship(
+        "RolloutRunWave", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin", order_by="RolloutRunWave.index")
+    hosts: Mapped[list["RolloutRunHost"]] = relationship(
+        "RolloutRunHost", cascade="all, delete-orphan", passive_deletes=True, lazy="selectin",
+        order_by=lambda: [RolloutRunHost.wave_index, RolloutRunHost.order])
+
+
+class RolloutRunWave(Base):
+    __tablename__ = "rollout_run_waves"
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
+    index: Mapped[int] = mapped_column(Integer)
+    label: Mapped[str] = mapped_column(String(60))
+    # pending | awaiting_approval | preflight | running | soaking | gate | passed | skipped | cancelled
+    status: Mapped[str] = mapped_column(String(20), default="pending")
+    baseline: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    gate_result: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    soak_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    gate_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    approved_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
+    started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+
+
+class RolloutRunHost(Base):
+    __tablename__ = "rollout_run_hosts"
+    __table_args__ = (UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),)
+
+    id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    run_id: Mapped[uuid.UUID] = mapped_column(PGUUID(as_uuid=True), ForeignKey("rollout_runs.id", ondelete="CASCADE"), index=True)
+    wave_index: Mapped[int] = mapped_column(Integer)
+    host_id: Mapped[uuid.UUID | None] = mapped_column(
+        PGUUID(as_uuid=True), ForeignKey("opnsense_hosts.id", ondelete="SET NULL"), nullable=True, index=True)
+    host_name: Mapped[str] = mapped_column(String(120))
+    cluster_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
+    order: Mapped[int] = mapped_column(Integer, default=0)
+    # queued | running | verifying | succeeded | skipped | failed | blocked | cancelled
+    status: Mapped[str] = mapped_column(String(16), default="queued")
+    scheduled_maintenance_id: Mapped[uuid.UUID | None] = mapped_column(
+        PGUUID(as_uuid=True), ForeignKey("scheduled_maintenance.id", ondelete="SET NULL"), nullable=True, index=True)
+    version_before: Mapped[str | None] = mapped_column(String(40), nullable=True)
+    version_after: Mapped[str | None] = mapped_column(String(40), nullable=True)
+    # {"code": ..., "params": {...}} — translated in the UI (rollouts.hostNote.*)
+    detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)

backend/app/models/scheduled_maintenance.py

@@ -42,7 +42,12 @@ class ScheduledMaintenance(Base):
     host_operation_id: Mapped[uuid.UUID | None] = mapped_column(PGUUID(as_uuid=True), nullable=True)
     # Opt-in: hand the traffic over to the CARP partner first, run the action, then switch back.
     carp_handover: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
-    # Only while status == "running" with a handover: handover_wait | job_wait | leave | leave_wait.
+    # 1.25: check the result automatically (job_wait -> verify) and close the operation, instead of
+    # leaving it "submitted" for a human. Every rollout step sets it; off for existing plans.
+    auto_verify: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
+    # manual | rollout (created by a rollout run, see models/rollout.py RolloutRunHost.scheduled_maintenance_id)
+    origin: Mapped[str] = mapped_column(String(16), nullable=False, default="manual", server_default="manual")
+    # Only while status == "running" with a handover or auto_verify: handover_wait | job_wait | verify | leave | leave_wait.
     # State lives here, not in memory, so a backend restart in the middle of a run resumes cleanly.
     phase: Mapped[str | None] = mapped_column(String(16), nullable=True)
     phase_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)

backend/app/routers/groups.py

@@ -99,6 +99,9 @@ async def delete_group(group_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
     for assignment in (port_forward_groups, one_to_one_groups, outbound_nat_groups):
         if await db.scalar(select(assignment.c.group_id).where(assignment.c.group_id == group_id).limit(1)):
             raise api_error(409, ErrorCode.GRUPPE_WIRD_NOCH_VON_NAT_REGELN_VERWENDET)
+    from app.models.rollout import rollout_wave_groups
+    if await db.scalar(select(rollout_wave_groups.c.group_id).where(rollout_wave_groups.c.group_id == group_id).limit(1)):
+        raise api_error(409, ErrorCode.ROLLOUT_GROUP_IN_USE)
     await db.delete(group)
     await db.commit()
 

backend/app/routers/maintenance_windows.py

@@ -0,0 +1,244 @@
+"""Recurring maintenance windows and their exceptions (1.25).
+
+Every write here counts as a rollout action (api_token_service.is_rollout): a window decides WHEN
+unattended firmware actions may run, so an operator needs the same right as for deploying.
+"""
+import re
+import uuid
+from datetime import date, datetime, time, timedelta, timezone
+from typing import Literal
+
+from fastapi import APIRouter, Depends, Query, Request
+from pydantic import BaseModel, Field
+from sqlalchemy import select
+from sqlalchemy.exc import IntegrityError
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.database import get_db
+from app.errors import ErrorCode, api_error
+from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
+from app.services import audit_service, window_service
+from app.services.policy_service import policy_lock
+
+router = APIRouter(prefix="/maintenance-windows", tags=["maintenance-windows"])
+
+_TIME = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)$")
+
+
+class WindowIn(BaseModel):
+    name: str = Field(min_length=1, max_length=120)
+    timezone: str = Field(min_length=1, max_length=64)
+    recurrence: Literal["weekly", "monthly"]
+    weekdays: list[int] = Field(default_factory=list, max_length=7)
+    month_mode: Literal["nth_weekday", "day"] | None = None
+    month_nth: int | None = None
+    month_weekday: int | None = None
+    month_day: int | None = None
+    start_time: str
+    duration_minutes: int = Field(ge=15, le=window_service.MAX_DURATION_MINUTES)
+    start_cutoff_minutes: int = Field(default=60, ge=5, le=window_service.MAX_DURATION_MINUTES)
+    enabled: bool = True
+
+
+class ExceptionIn(BaseModel):
+    occurrence_date: date
+    kind: Literal["skip", "move"]
+    moved_start: datetime | None = None
+    reason: str = Field(default="", max_length=300)
+
+
+def _invalid(field: str):
+    return api_error(422, ErrorCode.WINDOW_INVALID, field=field)
+
+
+def _validated(payload: WindowIn) -> dict:
+    name = payload.name.strip()
+    if not name:
+        raise _invalid("name")
+    if not window_service.valid_timezone(payload.timezone):
+        raise _invalid("timezone")
+    match = _TIME.match(payload.start_time)
+    if not match:
+        raise _invalid("start_time")
+    values = {
+        "name": name, "timezone": payload.timezone, "recurrence": payload.recurrence,
+        "weekdays": [], "month_mode": None, "month_nth": None, "month_weekday": None, "month_day": None,
+        "start_time": time(int(match.group(1)), int(match.group(2))),
+        "duration_minutes": payload.duration_minutes,
+        "start_cutoff_minutes": min(payload.start_cutoff_minutes, payload.duration_minutes),
+        "enabled": payload.enabled,
+    }
+    if payload.recurrence == "weekly":
+        days = sorted(set(payload.weekdays))
+        if not days or any(d < 0 or d > 6 for d in days):
+            raise _invalid("weekdays")
+        values["weekdays"] = days
+    elif payload.month_mode == "day":
+        if payload.month_day is None or not 1 <= payload.month_day <= 28:
+            raise _invalid("month_day")
+        values.update(month_mode="day", month_day=payload.month_day)
+    elif payload.month_mode == "nth_weekday":
+        if payload.month_nth not in (1, 2, 3, 4, -1):
+            raise _invalid("month_nth")
+        if payload.month_weekday is None or not 0 <= payload.month_weekday <= 6:
+            raise _invalid("month_weekday")
+        values.update(month_mode="nth_weekday", month_nth=payload.month_nth, month_weekday=payload.month_weekday)
+    else:
+        raise _invalid("month_mode")
+    return values
+
+
+async def _plans_using(db: AsyncSession, window_id) -> list[dict]:
+    from app.models.rollout import RolloutPlan
+    rows = (await db.scalars(select(RolloutPlan).where(RolloutPlan.window_id == window_id).order_by(RolloutPlan.name))).all()
+    return [{"id": str(p.id), "name": p.name, "auto_start": p.auto_start, "active": p.activated_at is not None} for p in rows]
+
+
+def read_window(window: MaintenanceWindow, *, now: datetime | None = None, count: int = 5) -> dict:
+    now = now or datetime.now(timezone.utc)
+    return {
+        "id": str(window.id), "name": window.name, "timezone": window.timezone, "recurrence": window.recurrence,
+        "weekdays": window.weekdays or [], "month_mode": window.month_mode, "month_nth": window.month_nth,
+        "month_weekday": window.month_weekday, "month_day": window.month_day,
+        "start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
+        "start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
+        "created_by": window.created_by, "created_at": window.created_at,
+        "open_now": window_service.current(window, now) is not None,
+        "next": [o.as_dict() for o in window_service.occurrences(window, now, count, with_skipped=True)] if window.enabled else [],
+        "exceptions": [
+            {"id": str(e.id), "occurrence_date": e.occurrence_date.isoformat(), "kind": e.kind,
+             "moved_start": e.moved_start, "reason": e.reason, "actor": e.actor, "created_at": e.created_at}
+            for e in window.exceptions
+        ],
+    }
+
+
+async def _window_or_404(db: AsyncSession, window_id: uuid.UUID) -> MaintenanceWindow:
+    window = await db.get(MaintenanceWindow, window_id)
+    if window is None:
+        raise api_error(404, ErrorCode.WINDOW_NOT_FOUND)
+    return window
+
+
+async def _name_taken(db: AsyncSession, name: str, own_id=None) -> bool:
+    other = await db.scalar(select(MaintenanceWindow.id).where(MaintenanceWindow.name == name))
+    return other is not None and other != own_id
+
+
+@router.get("")
+async def list_windows(db: AsyncSession = Depends(get_db)):
+    rows = (await db.scalars(select(MaintenanceWindow).order_by(MaintenanceWindow.name))).all()
+    result = []
+    for w in rows:
+        item = read_window(w)
+        item["used_by"] = await _plans_using(db, w.id)
+        result.append(item)
+    return result
+
+
+@router.post("", status_code=201)
+async def create_window(payload: WindowIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    values = _validated(payload)
+    if await _name_taken(db, values["name"]):
+        raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
+    window = MaintenanceWindow(**values, created_by=request.state.user.username, created_at=datetime.now(timezone.utc))
+    window.exceptions = []
+    db.add(window)
+    await db.commit()
+    item = read_window(await _window_or_404(db, window.id))
+    item["used_by"] = []
+    return item
+
+
+@router.get("/{window_id}")
+async def get_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    item = read_window(await _window_or_404(db, window_id))
+    item["used_by"] = await _plans_using(db, window_id)
+    return item
+
+
+@router.put("/{window_id}")
+async def update_window(window_id: uuid.UUID, payload: WindowIn, db: AsyncSession = Depends(get_db)):
+    """Changing a window changes the fingerprint of every plan that uses it, so an activated
+    automatic start stops until the plan is activated again (see rollout_service.fingerprint)."""
+    await policy_lock(db)
+    window = await _window_or_404(db, window_id)
+    values = _validated(payload)
+    if await _name_taken(db, values["name"], window.id):
+        raise api_error(409, ErrorCode.WINDOW_NAME_TAKEN)
+    for key, value in values.items():
+        setattr(window, key, value)
+    await db.commit()
+    item = read_window(await _window_or_404(db, window_id))
+    item["used_by"] = await _plans_using(db, window_id)
+    return item
+
+
+@router.delete("/{window_id}", status_code=204)
+async def delete_window(window_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    window = await _window_or_404(db, window_id)
+    # A plan without a window means "no time limit": deleting its window must never turn it into one.
+    if await _plans_using(db, window_id):
+        raise api_error(409, ErrorCode.WINDOW_IN_USE)
+    await db.delete(window)
+    await db.commit()
+
+
+@router.get("/{window_id}/occurrences")
+async def window_occurrences(window_id: uuid.UUID, count: int = Query(10, ge=1, le=50), db: AsyncSession = Depends(get_db)):
+    window = await _window_or_404(db, window_id)
+    return [o.as_dict() for o in window_service.occurrences(window, datetime.now(timezone.utc), count, with_skipped=True)]
+
+
+@router.post("/{window_id}/exceptions", status_code=201)
+async def add_exception(window_id: uuid.UUID, payload: ExceptionIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    window = await _window_or_404(db, window_id)
+    now = datetime.now(timezone.utc)
+    if not window_service.is_occurrence_date(window, payload.occurrence_date):
+        raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
+    # Only future occurrences (or the one open now) can be changed; history stays as it was.
+    if window_service.local_start(window, payload.occurrence_date) + timedelta(minutes=window.duration_minutes) <= now:
+        raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="occurrence_date")
+    moved = None
+    if payload.kind == "move":
+        if payload.moved_start is None:
+            raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
+        moved = payload.moved_start if payload.moved_start.tzinfo else payload.moved_start.replace(tzinfo=timezone.utc)
+        if moved <= now:
+            raise api_error(422, ErrorCode.WINDOW_EXCEPTION_INVALID, field="moved_start")
+    if any(e.occurrence_date == payload.occurrence_date for e in window.exceptions):
+        raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
+    exc = MaintenanceWindowException(
+        window_id=window.id, occurrence_date=payload.occurrence_date, kind=payload.kind, moved_start=moved,
+        reason=payload.reason.strip(), actor=request.state.user.username, created_at=now,
+    )
+    db.add(exc)
+    try:
+        await db.commit()
+    except IntegrityError:
+        await db.rollback()
+        raise api_error(409, ErrorCode.WINDOW_EXCEPTION_EXISTS)
+    await audit_service.persist(
+        kind="maintenance_window", action=f"exception_{payload.kind}", status="saved", entity_id=str(window.id),
+        entity_name=window.name, details={"occurrence_date": payload.occurrence_date.isoformat(),
+                                          "moved_start": moved.isoformat() if moved else None, "reason": exc.reason},
+    )
+    db.expire_all()
+    return read_window(await _window_or_404(db, window_id))
+
+
+@router.delete("/{window_id}/exceptions/{exception_id}")
+async def delete_exception(window_id: uuid.UUID, exception_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    window = await _window_or_404(db, window_id)
+    exc = await db.get(MaintenanceWindowException, exception_id)
+    if exc is None or exc.window_id != window.id:
+        raise api_error(404, ErrorCode.WINDOW_EXCEPTION_NOT_FOUND)
+    await db.delete(exc)
+    await db.commit()
+    db.expire_all()
+    return read_window(await _window_or_404(db, window_id))
+

backend/app/routers/rollouts.py

@@ -0,0 +1,432 @@
+"""Firmware rollouts in waves (1.25): plans and runs.
+
+Roles (enforced in AuthMiddleware, see api_token_service.is_rollout): everybody signed in reads;
+operators prepare plans; starting/activating a plan and resuming/approving/skipping/aborting a run
+need the deploy right (admin, or operator while operators may deploy). Pausing is deliberately NOT
+a rollout action: anybody who may write at all may always brake.
+"""
+import uuid
+from datetime import datetime, timezone
+from typing import Literal
+
+from fastapi import APIRouter, Depends, Request
+from pydantic import BaseModel, Field
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.database import get_db
+from app.errors import ErrorCode, api_error
+from app.models.maintenance_window import MaintenanceWindow
+from app.models.policy import DeviceGroup
+from app.models.rollout import RolloutPlan, RolloutPlanWave, RolloutRun
+from app.models.scheduled_maintenance import ScheduledMaintenance
+from app.services import audit_service, rollout_engine, rollout_health, rollout_service
+from app.services.policy_service import policy_lock
+
+router = APIRouter(prefix="/rollouts", tags=["rollouts"])
+
+ACTIVE_RUN = rollout_engine.ACTIVE
+
+
+class WaveIn(BaseModel):
+    label: str = Field(min_length=1, max_length=60)
+    group_ids: list[uuid.UUID] = Field(min_length=1, max_length=20)
+    soak_minutes: int = Field(default=30, ge=0, le=rollout_service.MAX_SOAK_MINUTES)
+    carp_handover: bool = False
+    require_approval: bool = False
+
+
+class PlanIn(BaseModel):
+    name: str = Field(min_length=1, max_length=120)
+    description: str = Field(default="", max_length=2000)
+    action: Literal["update", "major_update"] = "update"
+    window_id: uuid.UUID | None = None
+    auto_start: bool = False
+    gate_checks: dict[str, bool] = Field(default_factory=dict)
+    compliance_mode: Literal["warn", "fail"] = "warn"
+    waves: list[WaveIn] = Field(min_length=1, max_length=rollout_service.MAX_WAVES)
+
+
+class TokenIn(BaseModel):
+    preview_token: str = Field(min_length=64, max_length=64)
+
+
+async def _plan_or_404(db: AsyncSession, plan_id: uuid.UUID) -> RolloutPlan:
+    plan = await rollout_service.load_plan(db, plan_id)
+    if plan is None:
+        raise api_error(404, ErrorCode.ROLLOUT_PLAN_NOT_FOUND)
+    return plan
+
+
+async def read_plan(db: AsyncSession, plan: RolloutPlan, resolved: dict | None = None) -> dict:
+    resolved = resolved or await rollout_service.resolve(db, plan)
+    active = plan.activated_at is not None
+    last_run = await db.scalar(select(RolloutRun).where(RolloutRun.plan_id == plan.id).order_by(RolloutRun.created_at.desc()).limit(1))
+    return {
+        **rollout_service.definition(plan),
+        "description": plan.description,
+        "waves": [{"id": str(w.id), "position": w.position, "label": w.label, "soak_minutes": w.soak_minutes,
+                   "carp_handover": w.carp_handover, "require_approval": w.require_approval,
+                   "group_ids": [str(g.id) for g in w.groups], "groups": [g.name for g in w.groups],
+                   "host_count": len(resolved["waves"][i]["hosts"]) if i < len(resolved["waves"]) else 0}
+                  for i, w in enumerate(plan.waves)],
+        "active": active, "activated_by": plan.activated_by, "activated_at": plan.activated_at,
+        # Activated but the plan, its groups or its window changed since: the automatic start is held back.
+        "fingerprint_ok": (not active) or plan.activated_fingerprint == rollout_service.fingerprint(plan, resolved),
+        "auto_start_error": plan.auto_start_error,
+        "window": resolved["window"], "errors": resolved["errors"], "warnings": resolved["warnings"],
+        "host_count": sum(len(w["hosts"]) for w in resolved["waves"]),
+        "last_run": {"id": str(last_run.id), "status": last_run.status, "created_at": last_run.created_at} if last_run else None,
+        "created_by": plan.created_by, "created_at": plan.created_at, "updated_at": plan.updated_at,
+    }
+
+
+async def _apply(db: AsyncSession, plan: RolloutPlan, payload: PlanIn) -> None:
+    name = payload.name.strip()
+    if not name:
+        raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="name")
+    other = await db.scalar(select(RolloutPlan.id).where(RolloutPlan.name == name))
+    if other is not None and other != plan.id:
+        raise api_error(409, ErrorCode.ROLLOUT_NAME_TAKEN)
+    if payload.window_id is not None and await db.get(MaintenanceWindow, payload.window_id) is None:
+        raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="window_id")
+    if any(k not in rollout_health.CHECKS for k in payload.gate_checks):
+        raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="gate_checks")
+    wanted = {g for w in payload.waves for g in w.group_ids}
+    groups = {g.id: g for g in (await db.scalars(select(DeviceGroup).where(DeviceGroup.id.in_(wanted)))).all()}
+    if len(groups) != len(wanted):
+        raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="group_ids")
+    plan.name, plan.description, plan.action = name, payload.description.strip(), payload.action
+    plan.window_id, plan.auto_start = payload.window_id, payload.auto_start
+    plan.gate_checks = {k: bool(v) for k, v in payload.gate_checks.items()}
+    plan.compliance_mode = payload.compliance_mode
+    plan.waves = [
+        RolloutPlanWave(position=i, label=w.label.strip() or f"{i + 1}", soak_minutes=w.soak_minutes, carp_handover=w.carp_handover,
+                        require_approval=w.require_approval, groups=[groups[g] for g in dict.fromkeys(w.group_ids)])
+        for i, w in enumerate(payload.waves)
+    ]
+    # Any edit disarms the automatic start: an operator cannot quietly change what an admin activated.
+    plan.activated_at = plan.activated_by = plan.activated_fingerprint = None
+    plan.auto_start_error = None
+
+
+@router.get("/plans")
+async def list_plans(db: AsyncSession = Depends(get_db)):
+    plans = (await db.scalars(select(RolloutPlan).order_by(RolloutPlan.name))).all()
+    return [await read_plan(db, p) for p in plans]
+
+
+@router.post("/plans", status_code=201)
+async def create_plan(payload: PlanIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    plan = RolloutPlan(created_by=request.state.user.username, created_at=datetime.now(timezone.utc), waves=[])
+    await _apply(db, plan, payload)
+    db.add(plan)
+    await db.commit()
+    return await read_plan(db, await _plan_or_404(db, plan.id))
+
+
+@router.get("/plans/{plan_id}")
+async def get_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    return await read_plan(db, await _plan_or_404(db, plan_id))
+
+
+@router.put("/plans/{plan_id}")
+async def update_plan(plan_id: uuid.UUID, payload: PlanIn, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    plan = await _plan_or_404(db, plan_id)
+    was_active = plan.activated_at is not None
+    await _apply(db, plan, payload)
+    await db.commit()
+    if was_active:
+        await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id),
+                                    entity_name=plan.name, message="Plan geändert: automatischer Start deaktiviert")
+    return await read_plan(db, await _plan_or_404(db, plan_id))
+
+
+@router.delete("/plans/{plan_id}", status_code=204)
+async def delete_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    plan = await _plan_or_404(db, plan_id)
+    if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.status.in_(ACTIVE_RUN)).limit(1)):
+        raise api_error(409, ErrorCode.ROLLOUT_PLAN_BUSY)
+    await db.delete(plan)
+    await db.commit()
+
+
+@router.post("/plans/{plan_id}/preview")
+async def preview_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    plan = await _plan_or_404(db, plan_id)
+    resolved = await rollout_service.resolve(db, plan)
+    return {**resolved, "preview_token": rollout_service.fingerprint(plan, resolved),
+            "can_start": not resolved["errors"], "plan": await read_plan(db, plan, resolved)}
+
+
+async def _checked_token(db: AsyncSession, plan: RolloutPlan, token: str) -> tuple[dict, str]:
+    resolved = await rollout_service.resolve(db, plan)
+    current = rollout_service.fingerprint(plan, resolved)
+    if token != current:
+        raise api_error(409, ErrorCode.ROLLOUT_PREVIEW_STALE)
+    if resolved["errors"]:
+        raise api_error(422, ErrorCode.ROLLOUT_PLAN_INVALID, reason=resolved["errors"][0]["code"])
+    return resolved, current
+
+
+@router.post("/plans/{plan_id}/activate")
+async def activate_plan(plan_id: uuid.UUID, payload: TokenIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    plan = await _plan_or_404(db, plan_id)
+    if not plan.auto_start:
+        raise api_error(422, ErrorCode.ROLLOUT_INVALID, field="auto_start")
+    _resolved, current = await _checked_token(db, plan, payload.preview_token)
+    plan.activated_by, plan.activated_at, plan.activated_fingerprint = request.state.user.username, datetime.now(timezone.utc), current
+    plan.auto_start_error = None
+    await db.commit()
+    await audit_service.persist(kind="rollout", action="activate", status="completed", entity_id=str(plan.id),
+                                entity_name=plan.name, details={"fingerprint": current})
+    return await read_plan(db, await _plan_or_404(db, plan_id))
+
+
+@router.post("/plans/{plan_id}/deactivate")
+async def deactivate_plan(plan_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    plan = await _plan_or_404(db, plan_id)
+    plan.activated_by = plan.activated_at = plan.activated_fingerprint = None
+    await db.commit()
+    await audit_service.persist(kind="rollout", action="deactivate", status="completed", entity_id=str(plan.id), entity_name=plan.name)
+    return await read_plan(db, await _plan_or_404(db, plan_id))
+
+
+# ── runs ─────────────────────────────────────────────────────────────────────
+
+class StartIn(TokenIn):
+    confirm_name: str = Field(min_length=1, max_length=120)
+    acknowledge_unattended: Literal[True]
+
+
+class ReasonIn(BaseModel):
+    reason: str = Field(default="", max_length=500)
+
+
+class SkipIn(BaseModel):
+    host_id: uuid.UUID
+    reason: str = Field(min_length=3, max_length=500)
+
+
+class AbortIn(BaseModel):
+    reason: str = Field(min_length=3, max_length=500)
+    confirm_name: str = Field(min_length=1, max_length=120)
+
+
+def _read_run_summary(run: RolloutRun) -> dict:
+    counts: dict[str, int] = {}
+    for h in run.hosts:
+        counts[h.status] = counts.get(h.status, 0) + 1
+    return {
+        "id": str(run.id), "plan_id": str(run.plan_id) if run.plan_id else None, "plan_name": run.plan_name,
+        "action": run.action, "status": run.status, "paused_from": run.paused_from, "current_wave": run.current_wave,
+        "wave_count": len(run.waves), "current_wave_label": next((w.label for w in run.waves if w.index == run.current_wave), None),
+        "target_version": run.target_version, "pause_reason": run.pause_reason, "pause_detail": run.pause_detail,
+        "started_by": run.started_by, "trigger": run.trigger, "window_occurrence": run.window_occurrence,
+        "created_at": run.created_at, "finished_at": run.finished_at, "step_at": run.step_at, "host_counts": counts,
+        "active": run.status in rollout_engine.ACTIVE,
+    }
+
+
+async def read_run(db: AsyncSession, run: RolloutRun) -> dict:
+    scheds = {}
+    ids = [h.scheduled_maintenance_id for h in run.hosts if h.scheduled_maintenance_id]
+    if ids:
+        scheds = {s.id: s for s in (await db.scalars(select(ScheduledMaintenance).where(ScheduledMaintenance.id.in_(ids)))).all()}
+    soak_until = next((w.soak_until for w in run.waves if w.index == run.current_wave), None)
+    return {
+        **_read_run_summary(run),
+        "soak_until": soak_until, "progress": {k: v for k, v in (run.progress or {}).items()
+                                               if k in ("gate_recheck_at", "cluster_recheck_at", "blocked_since", "resume_gate",
+                                                        "cluster_gate", "preflight_started", "resume_recheck_at")},
+        "snapshot": run.plan_snapshot,
+        "waves": [{"index": w.index, "label": w.label, "status": w.status, "gate_result": w.gate_result,
+                   "gate_attempts": w.gate_attempts, "soak_until": w.soak_until, "approved_by": w.approved_by,
+                   "started_at": w.started_at, "finished_at": w.finished_at,
+                   "spec": run.plan_snapshot["waves"][w.index] if w.index < len(run.plan_snapshot.get("waves", [])) else None}
+                  for w in run.waves],
+        "hosts": [{"id": str(h.id), "host_id": str(h.host_id) if h.host_id else None, "host_name": h.host_name,
+                   "cluster_name": h.cluster_name, "wave_index": h.wave_index, "order": h.order, "status": h.status,
+                   "version_before": h.version_before, "version_after": h.version_after, "detail": h.detail,
+                   "started_at": h.started_at, "finished_at": h.finished_at,
+                   "schedule": ({"id": str(s.id), "status": s.status, "phase": s.phase, "carp_handover": s.carp_handover,
+                                 "host_operation_id": str(s.host_operation_id) if s.host_operation_id else None}
+                                if (s := scheds.get(h.scheduled_maintenance_id)) else None)}
+                  for h in sorted(run.hosts, key=lambda h: (h.wave_index, h.order, h.host_name))],
+    }
+
+
+async def _cancel_if_pending(db: AsyncSession, sched_id) -> bool:
+    """Cancel a host's schedule only if it has not fired. Locked: the poll loop may be firing it right now
+    (fire_due holds the row), and a blind write would turn a running action into "cancelled"."""
+    if sched_id is None:
+        return False
+    sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sched_id)
+                            .with_for_update().execution_options(populate_existing=True))
+    if sched is not None and sched.status == "pending":
+        sched.status = "cancelled"
+        return True
+    return False
+
+
+async def _run_or_404(db: AsyncSession, run_id: uuid.UUID, *, lock: bool = False) -> RolloutRun:
+    stmt = select(RolloutRun).where(RolloutRun.id == run_id).execution_options(populate_existing=True)
+    if lock:
+        stmt = stmt.with_for_update()
+    run = await db.scalar(stmt)
+    if run is None:
+        raise api_error(404, ErrorCode.ROLLOUT_RUN_NOT_FOUND)
+    return run
+
+
+def _not_now(run: RolloutRun):
+    return api_error(409, ErrorCode.ROLLOUT_ACTION_NOT_ALLOWED, status=run.status)
+
+
+async def _done(db: AsyncSession, run: RolloutRun, action: str, status: str, request: Request, **details) -> dict:
+    run.revision += 1
+    await db.commit()
+    await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id), entity_name=run.plan_name,
+                                details={"wave": run.current_wave, **details})
+    return await read_run(db, await _run_or_404(db, run.id))
+
+
+@router.post("/plans/{plan_id}/start", status_code=201)
+async def start_plan(plan_id: uuid.UUID, payload: StartIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    plan = await _plan_or_404(db, plan_id)
+    if payload.confirm_name != plan.name:
+        raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
+    resolved, _token = await _checked_token(db, plan, payload.preview_token)
+    if (other := await rollout_engine.active_run(db)) is not None:
+        # One run at a time: two runs could otherwise touch the same cluster at once.
+        raise api_error(409, ErrorCode.ROLLOUT_ALREADY_ACTIVE, plan=other.plan_name)
+    run = await rollout_engine.create_run(db, plan, resolved, started_by=request.state.user.username, trigger="manual")
+    await db.commit()
+    await audit_service.persist(kind="rollout", action="start", status="started", entity_id=str(run.id), entity_name=plan.name,
+                                details={"trigger": "manual", "hosts": sum(len(w["hosts"]) for w in resolved["waves"]),
+                                         "waves": [w["label"] for w in resolved["waves"]]})
+    return await read_run(db, await _run_or_404(db, run.id))
+
+
+@router.get("/runs")
+async def list_runs(db: AsyncSession = Depends(get_db)):
+    runs = (await db.scalars(select(RolloutRun).order_by(RolloutRun.created_at.desc()).limit(50))).all()
+    return [_read_run_summary(r) for r in runs]
+
+
+@router.get("/status")
+async def rollout_status(db: AsyncSession = Depends(get_db)):
+    """For /updates and the host pages: the active run and the hosts it still owns."""
+    run = await rollout_engine.active_run(db)
+    if run is None:
+        return {"active": None, "host_ids": [], "current_host_ids": []}
+    return {
+        "active": _read_run_summary(run),
+        "host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.status in ("queued", "blocked", "running", "verifying", "failed")],
+        "current_host_ids": [str(h.host_id) for h in run.hosts if h.host_id and h.wave_index == run.current_wave
+                             and h.status in ("queued", "blocked", "running", "verifying", "failed")],
+    }
+
+
+@router.get("/runs/{run_id}")
+async def get_run(run_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
+    return await read_run(db, await _run_or_404(db, run_id))
+
+
+@router.post("/runs/{run_id}/pause")
+async def pause_run(run_id: uuid.UUID, payload: ReasonIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    run = await _run_or_404(db, run_id, lock=True)
+    if run.status not in rollout_engine.ACTIVE or run.status == "paused":
+        raise _not_now(run)
+    # A brake that lets the next host fire anyway is no brake: a step not fired yet goes back to the queue.
+    requeued = [h.host_name for h in run.hosts if await rollout_engine.requeue_if_pending(db, h)]
+    rollout_engine.pause(run, "manual", by=request.state.user.username, text=payload.reason.strip()[:300])
+    return await _done(db, run, "pause", "paused", request, reason="manual", text=payload.reason.strip()[:300], requeued=requeued)
+
+
+@router.post("/runs/{run_id}/resume")
+async def resume_run(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    run = await _run_or_404(db, run_id, lock=True)
+    if run.status != "paused":
+        raise _not_now(run)
+    failed = [h.host_name for h in run.hosts if h.status == "failed"]
+    if failed:
+        # A failed host is never retried by itself: check it, then skip it explicitly.
+        raise api_error(409, ErrorCode.ROLLOUT_HOST_FAILED, hosts=", ".join(failed))
+    previous = run.started_by
+    # Whoever resumes takes over the responsibility (and the permission check) from here on.
+    run.started_by = request.state.user.username
+    run.status = "resuming"
+    run.pause_reason = run.pause_detail = None
+    run.progress = {**(run.progress or {}), "resume_attempts": 0, "resume_recheck_at": None, "paused_at": None}
+    return await _done(db, run, "resume", "resuming", request, previous_owner=previous)
+
+
+@router.post("/runs/{run_id}/approve")
+async def approve_wave(run_id: uuid.UUID, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    run = await _run_or_404(db, run_id, lock=True)
+    if run.status != "awaiting_approval":
+        raise _not_now(run)
+    wave = rollout_engine.wave_row(run, run.current_wave)
+    wave.approved_by, wave.status = request.state.user.username, "preflight"
+    run.started_by = request.state.user.username
+    run.status = "preflight"
+    run.progress = {**(run.progress or {}), "preflight_started": None, "preflight_requested_at": None}
+    return await _done(db, run, "approve", "approved", request, wave_label=wave.label)
+
+
+@router.post("/runs/{run_id}/skip-host")
+async def skip_host(run_id: uuid.UUID, payload: SkipIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    run = await _run_or_404(db, run_id, lock=True)
+    if run.status not in rollout_engine.ACTIVE:
+        raise _not_now(run)
+    host = next((h for h in run.hosts if h.host_id == payload.host_id), None)
+    if host is None:
+        raise api_error(404, ErrorCode.ROLLOUT_HOST_NOT_IN_RUN)
+    if host.status in ("running", "verifying"):
+        if not await _cancel_if_pending(db, host.scheduled_maintenance_id):
+            raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
+        # not fired yet: nothing happened on the firewall
+    elif host.status not in ("queued", "blocked", "failed"):
+        raise api_error(409, ErrorCode.ROLLOUT_HOST_RUNNING, host=host.host_name)
+    host.status = "skipped"
+    host.detail = {"code": "skipped_by", "params": {"user": request.state.user.username, "reason": payload.reason.strip()[:300]}}
+    host.finished_at = datetime.now(timezone.utc)
+    return await _done(db, run, "skip_host", "skipped", request, host=host.host_name, reason=payload.reason.strip()[:300])
+
+
+@router.post("/runs/{run_id}/abort")
+async def abort_run(run_id: uuid.UUID, payload: AbortIn, request: Request, db: AsyncSession = Depends(get_db)):
+    await policy_lock(db)
+    run = await _run_or_404(db, run_id, lock=True)
+    if run.status not in rollout_engine.ACTIVE:
+        raise _not_now(run)
+    if payload.confirm_name != run.plan_name:
+        raise api_error(409, ErrorCode.ROLLOUT_CONFIRM_NAME)
+    now = datetime.now(timezone.utc)
+    still_running = []
+    for host in run.hosts:
+        if host.status in ("running", "verifying") and host.scheduled_maintenance_id:
+            if await _cancel_if_pending(db, host.scheduled_maintenance_id):
+                host.status, host.finished_at = "cancelled", now
+            else:
+                still_running.append(host.host_name)  # an action already on the firewall is finished in a controlled way
+        elif host.status in ("queued", "blocked"):
+            host.status, host.finished_at = "cancelled", now
+    for wave in run.waves:
+        if wave.status not in ("passed", "skipped"):
+            wave.status = "cancelled"
+    run.status, run.finished_at = "cancelled", now
+    run.pause_reason, run.pause_detail = "aborted", {"by": request.state.user.username, "reason": payload.reason.strip()[:300],
+                                                     "still_running": still_running}
+    return await _done(db, run, "abort", "cancelled", request, reason=payload.reason.strip()[:300], still_running=still_running)

backend/app/routers/scheduled_maintenance.py

@@ -24,6 +24,8 @@ class Schedule(BaseModel):
     acknowledge_unattended: Literal[True]
     # Hand the traffic over to the CARP partner first and switch back afterwards (cluster nodes only).
     carp_handover: bool = False
+    # Check the result automatically afterwards and close the operation (no manual "Wartung abschließen").
+    auto_verify: bool = False
 
 
 def _read(sched: ScheduledMaintenance) -> dict:
@@ -32,6 +34,8 @@ def _read(sched: ScheduledMaintenance) -> dict:
         "status": sched.status, "actor": sched.actor, "created_at": sched.created_at,
         "result": sched.result, "host_operation_id": sched.host_operation_id,
         "carp_handover": sched.carp_handover, "phase": sched.phase,
+        "auto_verify": sched.auto_verify, "origin": sched.origin,
+        "rollout_run_id": ((sched.progress or {}).get("rollout") or {}).get("run_id"),
     }
 
 
@@ -80,6 +84,7 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
     sched = ScheduledMaintenance(
         host_id=host.id, host_name=host.name, action=payload.action, run_at=run_at,
         actor=request.state.user.username, created_at=now, carp_handover=payload.carp_handover,
+        auto_verify=payload.auto_verify,
     )
     db.add(sched)
     await db.commit()
@@ -90,7 +95,9 @@ async def create_scheduled(host_id: uuid.UUID, payload: Schedule, request: Reque
 async def cancel_scheduled(host_id: uuid.UUID, schedule_id: uuid.UUID, db: AsyncSession = Depends(get_db)):
     await policy_lock(db)
     await _host_or_404(db, host_id)
-    sched = await db.get(ScheduledMaintenance, schedule_id)
+    # Locked and re-read: the poll loop may be firing this very row right now (it holds the lock meanwhile).
+    sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == schedule_id)
+                            .with_for_update().execution_options(populate_existing=True))
     if not sched or sched.host_id != host_id:
         raise api_error(404, ErrorCode.GEPLANTE_WARTUNG_NICHT_GEFUNDEN)
     if sched.status != "pending":

backend/app/routers/system.py

@@ -7,6 +7,7 @@ from app.database import get_db
 from app.errors import ErrorCode, api_error
 from app.models.user import AdminUser
 from app.routers.auth import current_user
+from app.services import rollout_engine
 from app.services import update_check_service, updater_client
 
 router = APIRouter(prefix="/system", tags=["system"])
@@ -99,6 +100,9 @@ async def updater_status(db: AsyncSession = Depends(get_db), user: AdminUser = D
         "can_update": False, "major_change": False,
         "job": None, "phases": [], "log": [], "services": {},
         "rollback_available": False, "rollback_loses_data": False, "rollback_version": None,
+        # A restart in the middle of a firmware rollout is safe (its state is in the database), but the
+        # UI warns: the run pauses its 15-second steps for as long as OPNexus is down.
+        "active_rollouts": await rollout_engine.count_active(db),
     }
     try:
         updater = await updater_client.status()

backend/app/schemas/notification.py

@@ -17,6 +17,7 @@ ProblemType = Literal[
     "backup_offsite_failed",
     "compliance_finding",
     "gateway_down", "resource_threshold", "service_down",
+    "rollout_paused", "rollout_start_failed",
 ]
 ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
 
@@ -52,6 +53,7 @@ class NotificationSettingsRead(BaseModel):
     gateway_down_enabled: bool = True
     resource_threshold_enabled: bool = False
     service_down_enabled: bool = False
+    rollout_problems_enabled: bool = True
     gateway_loss_threshold_pct: int = 20
     gateway_delay_threshold_ms: int = 500
     disk_threshold_pct: int = 90
@@ -108,6 +110,7 @@ class NotificationSettingsUpdate(BaseModel):
     gateway_down_enabled: bool | None = None
     resource_threshold_enabled: bool | None = None
     service_down_enabled: bool | None = None
+    rollout_problems_enabled: bool | None = None
     # 0 switches that single threshold off.
     gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
     gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)

backend/app/services/api_token_service.py

@@ -90,7 +90,10 @@ _ROLLOUT_POST = re.compile(
     r"|deployments/[^/]+/(remove|retry-rollback)"
     r"|maintenance/(execute|carp-execute|schedule)"
     r"|network/changes"
-    r"|services/[^/]+/(start|stop|restart))$"
+    r"|services/[^/]+/(start|stop|restart)"
+    # Firmware rollouts (1.25): starting/arming a plan and moving a run forward. Pausing is deliberately absent.
+    r"|rollouts/plans/[^/]+/(start|activate)"
+    r"|rollouts/runs/[^/]+/(resume|approve|skip-host|abort))$"
 )
 _ROLLOUT_DELETE = re.compile(
     r"^/api/(rules|aliases|port-forward|one-to-one|outbound-nat|policy-hierarchy/rules|ipsec/tunnels|openvpn/instances"
@@ -99,7 +102,13 @@ _ROLLOUT_DELETE = re.compile(
 )
 
 
+# Recurring maintenance windows decide WHEN unattended firmware actions run (1.25): every write counts.
+_ROLLOUT_ANY_WRITE = re.compile(r"^/api/maintenance-windows(/|$)")
+
+
 def is_rollout(method: str, path: str) -> bool:
+    if method in ("POST", "PUT", "PATCH", "DELETE") and _ROLLOUT_ANY_WRITE.match(path):
+        return True
     if method == "POST":
         return bool(_ROLLOUT_POST.search(path))
     return method == "DELETE" and bool(_ROLLOUT_DELETE.match(path))

backend/app/services/notification_checks.py

@@ -363,3 +363,49 @@ async def check_compliance(db: AsyncSession) -> dict[str, dict]:
                     "beispiele": ", ".join(params.get("examples", [])) or None,
                 }
     return problems
+
+
+# ── Rollouts (1.25) ─────────────────────────────────────────────────────────
+
+def _short(detail) -> str | None:
+    if not detail:
+        return None
+    parts = []
+    for key, value in detail.items():
+        if isinstance(value, list):
+            value = ", ".join(str(v) for v in value[:8])
+        parts.append(f"{key}: {value}")
+    return "; ".join(parts)[:500]
+
+
+async def check_rollout_paused(db: AsyncSession) -> dict[str, dict]:
+    """Open while a rollout run stands still waiting for a human (gate, failed host, window gone...)."""
+    from app.models.rollout import RolloutRun
+    problems: dict[str, dict] = {}
+    for run in (await db.scalars(select(RolloutRun).where(RolloutRun.status == "paused"))).all():
+        wave = next((w.label for w in run.waves if w.index == run.current_wave), None)
+        problems[str(run.id)] = {
+            "run_id": str(run.id), "plan": run.plan_name, "grund": run.pause_reason, "welle": wave,
+            "zielversion": run.target_version, "details": _short(run.pause_detail),
+        }
+    return problems
+
+
+ROLLOUT_START_FAILURE_WINDOW = timedelta(days=3)
+
+
+async def check_rollout_start_failed(db: AsyncSession) -> dict[str, dict]:
+    """An automatic start at a window occurrence did not happen (event: closed without a 'resolved' notice)."""
+    from app.models.rollout import RolloutPlan
+    since = datetime.now(timezone.utc) - ROLLOUT_START_FAILURE_WINDOW
+    problems: dict[str, dict] = {}
+    for plan in (await db.scalars(select(RolloutPlan).where(RolloutPlan.auto_start_error.is_not(None)))).all():
+        error = plan.auto_start_error or {}
+        at = datetime.fromisoformat(error["at"]) if error.get("at") else None
+        if at is None or at < since:
+            continue
+        problems[f"{plan.id}:{error.get('occurrence')}"] = {
+            "plan_id": str(plan.id), "plan": plan.name, "grund": error.get("code"), "termin": error.get("occurrence"),
+            "details": _short(error.get("params")),
+        }
+    return problems

backend/app/services/notification_delivery.py

@@ -40,6 +40,8 @@ _LABELS = {
     "gateway_down": "Gateway gestört",
     "resource_threshold": "Ressourcen-Schwelle überschritten",
     "service_down": "Dienst ausgefallen",
+    "rollout_paused": "Rollout angehalten",
+    "rollout_start_failed": "Rollout nicht automatisch gestartet",
     "test": "Testbenachrichtigung",
 }
 

backend/app/services/notification_service.py

@@ -37,11 +37,13 @@ _CHECKS = [
     ("gateway_down", "gateway_down_enabled", monitor_checks.check_gateway_down),
     ("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
     ("service_down", "service_down_enabled", monitor_checks.check_service_down),
+    ("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
+    ("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
 ]
 
 # Event-like problems: when they drop out of the window nothing was "fixed", so no
 # "behoben" message goes out for them.
-_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed"}
+_NO_RESOLVED_NOTICE = {"scheduled_maintenance_failed", "rollout_start_failed"}
 
 
 async def _deliver(

backend/app/services/notification_settings_service.py

@@ -26,7 +26,7 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
         "host_offline_enabled", "cert_expiring_enabled", "rule_drift_enabled",
         "revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
         "network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
-        "gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled",
+        "gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled",
         "gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
         "swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
         "service_watchlist",

backend/app/services/rollout_engine.py

@@ -0,0 +1,805 @@
+"""The rollout run engine (1.25): moves runs forward, one small step per poll tick.
+
+Ground rules (an unattended engine that triggers firmware updates is the riskiest thing in OPNexus):
+
+* It never calls a firmware action itself. Each host step is an ordinary ScheduledMaintenance
+  (run_at=now, origin="rollout", auto_verify=True) — locks, backup before the update, CARP handover,
+  audit and restart safety all come from that existing path.
+* Never twice: a host gets a schedule only while its ``scheduled_maintenance_id`` is empty, written
+  in the same transaction as the schedule itself, under the run's row lock.
+* Never in parallel: one host at a time per run, and only one active run at all; nodes of a cluster
+  go BACKUP first, the MASTER only with the wave's CARP handover, and before every node of an
+  already touched cluster the cluster is checked again.
+* When in doubt, stop: anything unexpected pauses the run (``pause_reason``) and notifies; nothing
+  is retried on its own. A paused run only continues when a human resumes it, and resuming first
+  checks the health of everything done so far.
+* Every step re-checks that the person responsible for the run may still roll out.
+* Hard limits: waves/hosts (rollout_service), soak time, preflight/host timeouts, run age.
+
+Concurrency: a tick (1) does DB-only bookkeeping under the run's row lock, (2) decides with the
+lock released (remote reads may take seconds), and (3) applies the decision only if the run's
+``revision`` is still the one it decided on — a pause/abort in between, or a second backend, wins.
+"""
+import logging
+import uuid
+from datetime import datetime, timedelta, timezone
+from typing import Awaitable, Callable
+
+from fastapi import HTTPException
+from sqlalchemy import func, select
+from sqlalchemy.exc import IntegrityError
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.database import SessionLocal
+from app.models.host import OpnsenseHost
+from app.models.maintenance_window import MaintenanceWindow
+from app.models.rollout import RolloutPlan, RolloutRun, RolloutRunHost, RolloutRunWave
+from app.models.scheduled_maintenance import ScheduledMaintenance
+from app.models.user import AdminUser
+from app.services import audit_service, rollout_health, rollout_service, window_service
+from app.services import opnsense_client as client
+from app.services.host_maintenance_service import ensure_no_maintenance, peers
+from app.services.host_status_service import get_statuses
+from app.services.policy_service import policy_lock
+
+logger = logging.getLogger(__name__)
+
+ACTIVE = ("waiting_window", "preflight", "wave_running", "soaking", "gate", "awaiting_approval", "resuming", "paused")
+TERMINAL = ("succeeded", "cancelled")
+# Statuses in which the poll loop should tick quickly.
+BUSY = ("preflight", "wave_running", "gate", "resuming")
+HOST_OPEN = ("running", "verifying")
+
+PREFLIGHT_CHECK_DELAY = timedelta(seconds=20)  # after asking for an update check, before reading the result
+PREFLIGHT_TIMEOUT = timedelta(minutes=10)
+GATE_RECHECK = timedelta(minutes=2)  # a failed gate is looked at once more (slow start after a reboot)
+BLOCKED_PATIENCE = timedelta(minutes=15)  # someone else's maintenance in the cluster
+HOST_START_TIMEOUT = timedelta(minutes=10)  # the schedule never fired
+HOST_MAX_DURATION = timedelta(hours=4)  # backstop above the schedule's own timeouts
+MAX_RUN_AGE = timedelta(days=14)
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _iso(value: datetime | None) -> str | None:
+    return value.isoformat() if value else None
+
+
+def _ts(value: str | None) -> datetime | None:
+    return datetime.fromisoformat(value) if value else None
+
+
+def _note(code: str, **params) -> dict:
+    return {"code": code, "params": params}
+
+
+# ── small helpers on a loaded run ─────────────────────────────────────────────
+
+def wave_row(run: RolloutRun, index: int) -> RolloutRunWave:
+    return next(w for w in run.waves if w.index == index)
+
+
+def wave_spec(run: RolloutRun, index: int) -> dict:
+    return run.plan_snapshot["waves"][index]
+
+
+def hosts_of(run: RolloutRun, index: int) -> list[RolloutRunHost]:
+    return sorted((h for h in run.hosts if h.wave_index == index), key=lambda h: (h.order, h.host_name))
+
+
+def _named(run: RolloutRun, failed: list[str]) -> list[str]:
+    """"<host-id>:<check>" -> "<host name>:<check>" for pause details people read."""
+    names = {str(h.host_id): h.host_name for h in run.hosts if h.host_id}
+    return [f"{names.get(item.split(':', 1)[0], item.split(':', 1)[0])}:{item.split(':', 1)[1]}" for item in failed]
+
+
+def _progress(run: RolloutRun, **changes) -> None:
+    run.progress = {**(run.progress or {}), **changes}
+
+
+async def _audit(run: RolloutRun, action: str, status: str, message: str | None = None, **details) -> None:
+    token = audit_service.context.set({"actor": f"{run.started_by} (Rollout)"[:150], "id": str(run.id)})
+    try:
+        await audit_service.persist(kind="rollout", action=action, status=status, entity_id=str(run.id),
+                                    entity_name=run.plan_name, message=message,
+                                    details={"wave": run.current_wave, **details})
+    finally:
+        audit_service.context.reset(token)
+
+
+def _requeue(host: RolloutRunHost, code: str) -> None:
+    """Back to the queue: only for a host whose schedule provably never fired (cancelled under its row lock,
+    or refused by fire_due as overdue). Clearing the schedule id lets a resumed run plan it once more."""
+    host.status, host.scheduled_maintenance_id, host.started_at, host.detail = "queued", None, None, _note(code)
+
+
+async def requeue_if_pending(db: AsyncSession, host: RolloutRunHost) -> bool:
+    """Cancel a host's schedule that has not fired yet and put the host back in the queue. Locked like the
+    router's abort: fire_due holds the row while firing, and a running action is never touched."""
+    if host.status not in HOST_OPEN or host.scheduled_maintenance_id is None:
+        return False
+    sched = await db.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == host.scheduled_maintenance_id)
+                            .with_for_update().execution_options(populate_existing=True))
+    if sched is None or sched.status != "pending":
+        return False
+    sched.status = "cancelled"
+    sched.result = {"error": "Rollout angehalten, nicht ausgeführt", "not_fired": True}
+    _requeue(host, "not_fired")
+    return True
+
+
+def pause(run: RolloutRun, reason: str, **detail) -> None:
+    if run.status != "paused":
+        run.paused_from = run.status
+    run.status = "paused"
+    run.pause_reason = reason
+    run.pause_detail = detail
+    _progress(run, paused_at=_iso(_now()))
+
+
+async def may_roll_out(db: AsyncSession, username: str) -> bool:
+    """Does this account (still) have the right to roll out? Same rule as AuthMiddleware's is_rollout."""
+    user = await db.scalar(select(AdminUser).where(AdminUser.username == username))
+    if user is None:
+        return False
+    if user.role == "admin":
+        return True
+    if user.role == "operator":
+        from app.routers import governance
+        return (await governance.get_settings(db)).operators_may_deploy
+    return False
+
+
+async def _window(db: AsyncSession, run: RolloutRun) -> tuple[bool, MaintenanceWindow | None]:
+    """(window configured, window). Configured but gone means: stop, never 'no time limit'."""
+    window_id = run.plan_snapshot.get("window_id")
+    if not window_id:
+        return False, None
+    return True, await db.get(MaintenanceWindow, window_id)
+
+
+async def _may_start_now(db: AsyncSession, run: RolloutRun) -> str | None:
+    """None if new host work may start now, else the reason ('window_closed' / 'window_missing')."""
+    configured, window = await _window(db, run)
+    if not configured:
+        return None
+    if window is None:
+        return "window_missing"
+    return None if window_service.startable(window, _now()) else "window_closed"
+
+
+async def _lock(db: AsyncSession, run_id, revision: int | None = None) -> RolloutRun | None:
+    stmt = select(RolloutRun).where(RolloutRun.id == run_id)
+    if revision is not None:
+        stmt = stmt.where(RolloutRun.revision == revision)
+    return await db.scalar(stmt.with_for_update().execution_options(populate_existing=True))
+
+
+# ── creating a run ────────────────────────────────────────────────────────────
+
+async def active_run(db: AsyncSession) -> RolloutRun | None:
+    return await db.scalar(select(RolloutRun).where(RolloutRun.status.in_(ACTIVE)).order_by(RolloutRun.created_at).limit(1))
+
+
+async def create_run(db: AsyncSession, plan: RolloutPlan, resolved: dict, *, started_by: str, trigger: str,
+                     occurrence: datetime | None = None) -> RolloutRun:
+    """Freeze the plan and its resolved hosts. The caller holds policy_lock and has checked the token."""
+    now = _now()
+    snapshot = {
+        "plan_id": str(plan.id), "name": plan.name, "action": plan.action,
+        "window_id": str(plan.window_id) if plan.window_id else None,
+        "gate_checks": rollout_service.effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
+        "waves": [{k: w[k] for k in ("label", "soak_minutes", "carp_handover", "require_approval", "groups")} for w in resolved["waves"]],
+    }
+    run = RolloutRun(plan_id=plan.id, plan_name=plan.name, plan_snapshot=snapshot, action=plan.action, status="preflight",
+                     current_wave=0, started_by=started_by, trigger=trigger, window_occurrence=occurrence,
+                     progress={}, created_at=now, step_at=now, revision=0)
+    run.waves = [RolloutRunWave(index=w["index"], label=w["label"], status="preflight" if w["index"] == 0 else "pending")
+                 for w in resolved["waves"]]
+    run.hosts = [RolloutRunHost(wave_index=w["index"], host_id=uuid.UUID(h["id"]), host_name=h["name"], cluster_name=h["cluster_name"],
+                                order=i, status="queued")
+                 for w in resolved["waves"] for i, h in enumerate(w["hosts"])]
+    db.add(run)
+    await db.flush()
+    return run
+
+
+# ── the tick ──────────────────────────────────────────────────────────────────
+
+async def advance_runs(db: AsyncSession) -> int:
+    """One step for every run. Returns how many runs want a quick next tick."""
+    try:
+        await auto_start(db)
+    except Exception:
+        logger.exception("Rollout: automatischer Start fehlgeschlagen")
+        await db.rollback()
+    ids = list((await db.scalars(select(RolloutRun.id).where(
+        (RolloutRun.status.in_(ACTIVE)) | RolloutRun.hosts.any(RolloutRunHost.status.in_(HOST_OPEN))))).all())
+    await db.commit()
+    busy = 0
+    for run_id in ids:
+        try:
+            await tick(run_id)
+        except Exception as exc:
+            logger.exception("Rollout %s: Schritt fehlgeschlagen", run_id)
+            async with SessionLocal() as s:
+                run = await _lock(s, run_id)
+                if run is not None and run.status in ACTIVE and run.status != "paused":
+                    pause(run, "internal_error", error=type(exc).__name__)
+                    run.revision += 1
+                    await s.commit()
+                    await _audit(run, "pause", "paused", "Interner Fehler: Rollout angehalten", reason="internal_error")
+        async with SessionLocal() as s:
+            status = await s.scalar(select(RolloutRun.status).where(RolloutRun.id == run_id))
+        busy += status in BUSY
+    return busy
+
+
+async def tick(run_id) -> None:
+    async with SessionLocal() as db:
+        # (1) bookkeeping under the lock: what did the schedules do?
+        run = await _lock(db, run_id)
+        if run is None:
+            return
+        changed = await _sync_hosts(db, run)
+        if (run.status in ACTIVE and run.status != "paused" and _now() - run.created_at > MAX_RUN_AGE
+                and not any(h.status in HOST_OPEN for h in run.hosts)):
+            pause(run, "run_too_old", days=MAX_RUN_AGE.days)
+            changed = True
+        if changed:
+            run.revision += 1
+        await db.commit()
+        if changed and run.status == "paused" and run.pause_reason in ("host_failed", "host_timeout", "host_not_started", "run_too_old"):
+            await _audit(run, "pause", "paused", reason=run.pause_reason, detail=run.pause_detail)
+        if run.status not in ACTIVE or run.status in ("paused", "awaiting_approval"):
+            return
+        revision = run.revision
+        # (2) decide, without holding the lock
+        decision = await _HANDLERS[run.status](db, run)
+        await db.commit()
+        if decision is None:
+            return
+        # (3) apply only on the revision it was decided on
+        run = await _lock(db, run_id, revision)
+        if run is None:
+            await db.commit()
+            return
+        events = await decision(db, run)
+        run.revision += 1
+        run.step_at = _now()
+        await db.commit()
+        for event in events or []:
+            await _audit(run, *event[:2], **(event[2] if len(event) > 2 else {}))
+
+
+Decision = Callable[[AsyncSession, RolloutRun], Awaitable[list | None]]
+
+
+async def _sync_hosts(db: AsyncSession, run: RolloutRun) -> bool:
+    """Carry the outcome of each host's schedule over to the run. Returns True if anything changed."""
+    changed = False
+    now = _now()
+    for host in run.hosts:
+        if host.status not in HOST_OPEN:
+            continue
+        sched = await db.get(ScheduledMaintenance, host.scheduled_maintenance_id, populate_existing=True) \
+            if host.scheduled_maintenance_id else None
+        problem = None
+        if sched is None:
+            host.status, host.detail, host.finished_at = "failed", _note("schedule_missing"), now
+            problem = ("host_failed", {"host": host.host_name, "error": "schedule_missing"})
+        elif sched.status == "pending":
+            if host.started_at and now - host.started_at > HOST_START_TIMEOUT:
+                # Never leave it pending behind a paused run: it would fire whenever the poll loop comes back.
+                if await requeue_if_pending(db, host):
+                    problem = ("host_not_started", {"host": host.host_name})
+                    changed = True
+        elif (sched.result or {}).get("not_fired"):
+            _requeue(host, "not_fired")  # fire_due refused it as overdue: nothing happened on the firewall
+            problem = ("host_not_started", {"host": host.host_name})
+            changed = True
+        elif sched.status == "running":
+            want = "verifying" if sched.phase == "verify" else "running"
+            if host.status != want:
+                host.status = want
+                changed = True
+            if host.started_at and now - host.started_at > HOST_MAX_DURATION:
+                problem = ("host_timeout", {"host": host.host_name})
+        elif sched.status == "succeeded":
+            result = sched.result or {}
+            host.status, host.finished_at = "succeeded", now
+            host.version_after = result.get("version_after")
+            host.detail = _note("verified", version=result.get("version_after"), rebooted=bool(result.get("rebooted")))
+            changed = True
+        else:  # failed / cancelled
+            error = (sched.result or {}).get("error") or sched.status
+            host.status, host.finished_at = "failed", now
+            host.detail = _note("host_failed", error=error[:300])
+            problem = ("host_failed", {"host": host.host_name, "error": error[:300]})
+        if problem and run.status in ACTIVE and run.status != "paused":
+            pause(run, problem[0], **problem[1])
+            changed = True
+        elif problem and host.status == "failed":
+            changed = True
+    return changed
+
+
+# ── handlers: one per run status; each returns an apply-function or None ──────
+
+async def _actor_problem(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    if await may_roll_out(db, run.started_by):
+        return None
+
+    async def apply(db, run):
+        pause(run, "actor_invalid", user=run.started_by)
+        return [("pause", "paused", {"reason": "actor_invalid"})]
+    return apply
+
+
+def _to_waiting(reason: str) -> Decision:
+    async def apply(db, run):
+        if reason == "window_missing":
+            pause(run, "window_missing")
+            return [("pause", "paused", {"reason": reason})]
+        run.paused_from = run.status
+        run.status = "waiting_window"
+        return [("waiting_window", "waiting")]
+    return apply
+
+
+async def _handle_waiting_window(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    reason = await _may_start_now(db, run)
+    if reason == "window_missing":
+        return _to_waiting(reason)
+    if reason:
+        return None
+
+    async def apply(db, run):
+        _progress(run, resume_recheck_at=None, resume_attempts=0)
+        run.status = "resuming"  # the gate of everything done so far is checked again first
+        return [("window_open", "resuming")]
+    return apply
+
+
+async def _handle_resuming(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    recheck = _ts((run.progress or {}).get("resume_recheck_at"))
+    if recheck and _now() < recheck:
+        return None
+    done = [h for h in run.hosts if h.status == "succeeded" and h.host_id]
+    gate = {"hosts": {}, "verdict": "ok"}
+    if done:
+        hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
+        baselines = {}
+        for w in run.waves:
+            baselines.update((w.baseline or {}))
+        gate = await rollout_health.evaluate(db, hosts, baselines, target_version=run.target_version,
+                                             checks=run.plan_snapshot.get("gate_checks"),
+                                             compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
+
+    async def apply(db, run):
+        if gate["verdict"] == "fail":
+            attempts = (run.progress or {}).get("resume_attempts", 0) + 1
+            _progress(run, resume_gate=gate, resume_attempts=attempts)
+            if attempts < 2:
+                _progress(run, resume_recheck_at=_iso(_now() + GATE_RECHECK))
+                return [("resume_gate", "recheck", {"failed": rollout_health.failed_checks(gate)})]
+            run.paused_from = run.paused_from or "preflight"
+            _progress(run, resume_recheck_at=None)
+            run.status = "paused"
+            run.pause_reason, run.pause_detail = "gate_failed", {"failed": _named(run, rollout_health.failed_checks(gate)), "on_resume": True}
+            return [("resume_gate", "failed", {"failed": rollout_health.failed_checks(gate)})]
+        _progress(run, resume_gate=gate, resume_recheck_at=None, resume_attempts=0)
+        target = run.paused_from if run.paused_from in ("preflight", "wave_running", "soaking", "gate") else "preflight"
+        if target == "gate":
+            wave_row(run, run.current_wave).gate_attempts = 0
+            _progress(run, gate_recheck_at=None)
+        if target == "preflight":
+            _progress(run, preflight_started=None, preflight_requested_at=None)
+        run.status, run.paused_from, run.pause_reason, run.pause_detail = target, None, None, None
+        return [("resume_gate", "passed", {"continue_with": target})]
+    return apply
+
+
+def target_of(status: dict, action: str) -> str | None:
+    """The version a firmware action will install, from core/firmware/status (read after a fresh check).
+
+    Live finding (test-fw-3, 26.7.1_1, 2026-10-21): ``product.product_latest`` said 26.7.2 while the
+    ``opnsense`` core package was offered as 26.7.7 — and the package is what gets installed. So the
+    core package's new version comes first; product_latest is only a fallback."""
+    if action == "major_update":
+        return status.get("upgrade_major_version") or None
+    packages = status.get("all_packages") if isinstance(status.get("all_packages"), dict) else {}
+    core = packages.get("opnsense") if isinstance(packages.get("opnsense"), dict) else None
+    if core and core.get("new") and core.get("new") != "N/A":
+        return str(core["new"])
+    for row in status.get("upgrade_packages") or []:
+        if isinstance(row, dict) and row.get("name") == "opnsense" and row.get("new_version"):
+            return str(row["new_version"])
+    product = status.get("product") if isinstance(status.get("product"), dict) else {}
+    return product.get("product_latest") or status.get("product_latest") or None
+
+
+async def _handle_preflight(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    if problem := await _actor_problem(db, run):
+        return problem
+    reason = await _may_start_now(db, run)
+    if reason:
+        return _to_waiting(reason)
+    index = run.current_wave
+    prog = run.progress or {}
+    started = _ts(prog.get("preflight_started")) or _now()
+    requested = _ts(prog.get("preflight_requested_at"))
+    candidates = [h for h in hosts_of(run, index) if h.status in ("queued", "blocked")]
+    now = _now()
+
+    # Which hosts are usable at all?
+    usable: dict = {}
+    skips: dict = {}
+    for rh in candidates:
+        host = await db.get(OpnsenseHost, rh.host_id) if rh.host_id else None
+        if host is None:
+            skips[rh.id] = _note("host_deleted")
+        elif not host.enabled:
+            skips[rh.id] = _note("host_disabled")
+        else:
+            usable[rh.id] = host
+
+    if requested is None:
+        # Step 1: ask every usable host to look for updates (a firmware *check*, changes nothing installed).
+        asked = []
+        for rh_id, host in usable.items():
+            try:
+                await ensure_no_maintenance(db, [host])
+                if (await client.get_firmware_running(host)).get("status") == "ready":
+                    await client.check_firmware_updates(host)
+                    asked.append(str(rh_id))
+            except (client.OpnsenseAPIError, HTTPException):
+                pass  # judged in step 2 (and blocked after the timeout)
+
+        async def apply(db, run):
+            _progress(run, preflight_started=_iso(started), preflight_requested_at=_iso(_now()), preflight_asked=asked)
+            w = wave_row(run, index)
+            w.status = "preflight"
+            w.started_at = w.started_at or _now()
+            return [("preflight", "started", {"hosts": len(usable)})]
+        return apply
+    if now - requested < PREFLIGHT_CHECK_DELAY:
+        return None
+
+    # Step 2: read what each host offers.
+    offers: dict = {}
+    waiting: dict = {}
+    for rh_id, host in usable.items():
+        try:
+            await ensure_no_maintenance(db, [host])
+        except HTTPException:
+            waiting[rh_id] = _note("locked")
+            continue
+        try:
+            running = (await client.get_firmware_running(host)).get("status")
+            if running != "ready":
+                waiting[rh_id] = _note("firmware_busy")
+                continue
+            status = await client.get_firmware_status(host)
+        except client.OpnsenseAPIError:
+            waiting[rh_id] = _note("offline")
+            continue
+        fw = status.get("status")
+        wanted = "upgrade" if run.action == "major_update" else "update"
+        if fw == wanted:
+            offers[rh_id] = target_of(status, run.action)
+        elif fw in ("none", "update", "upgrade"):
+            skips[rh_id] = _note("no_update_offered", status=fw)
+        else:
+            waiting[rh_id] = _note("firmware_status", status=str(fw)[:40])
+
+    if waiting:
+        if now - started < PREFLIGHT_TIMEOUT:
+            return None
+
+        async def apply_blocked(db, run):
+            names = []
+            for rh in hosts_of(run, index):
+                if rh.id in waiting:
+                    rh.status, rh.detail = "blocked", waiting[rh.id]
+                    names.append(rh.host_name)
+            pause(run, "preflight_blocked", hosts=", ".join(names))
+            _progress(run, preflight_started=None, preflight_requested_at=None)
+            return [("pause", "paused", {"reason": "preflight_blocked", "hosts": names})]
+        return apply_blocked
+
+    targets = {t for t in offers.values()}
+    expected = run.target_version
+    mismatch = None
+    if None in targets:
+        mismatch = {"offered": "?"}
+    elif expected is None and len(targets) > 1:
+        mismatch = {"offered": ", ".join(sorted(targets))}
+    elif expected is not None and targets - {expected}:
+        mismatch = {"expected": expected, "offered": ", ".join(sorted(targets - {expected}))}
+    if mismatch:
+        async def apply_mismatch(db, run):
+            # OPNsense always installs the newest offered version; a later ring must not get one the earlier ring never ran.
+            pause(run, "target_version_changed", **mismatch)
+            _progress(run, preflight_started=None, preflight_requested_at=None)
+            return [("pause", "paused", {"reason": "target_version_changed", **mismatch})]
+        return apply_mismatch
+
+    # Baselines and order (BACKUP before MASTER inside a cluster).
+    baselines = {}
+    versions = {}
+    for rh_id in offers:
+        host = usable[rh_id]
+        baselines[str(host.id)] = await rollout_health.baseline(db, host)
+        versions[rh_id] = baselines[str(host.id)].get("version")
+    roles = await get_statuses([usable[i] for i in offers])
+    rank = {"BACKUP": 0, None: 1, "MASTER": 2}
+
+    def order_key(rh_id):
+        host = usable[rh_id]
+        role = roles.get(host.id, {}).get("carp_status") if host.cluster_name else None
+        return (1 if host.cluster_name else 0, host.cluster_name or "", rank.get(role, 1), host.name)
+    ordered = sorted(offers, key=order_key)
+    target = next(iter(targets)) if targets else expected
+
+    async def apply_ready(db, run):
+        for rh in hosts_of(run, index):
+            if rh.id in skips:
+                rh.status, rh.detail = "skipped", skips[rh.id]
+            elif rh.id in offers:
+                rh.status, rh.detail = "queued", None
+                rh.order = ordered.index(rh.id)
+                rh.version_before = versions.get(rh.id)
+        w = wave_row(run, index)
+        w.baseline = baselines
+        if run.target_version is None and target:
+            run.target_version = target
+        _progress(run, preflight_started=None, preflight_requested_at=None, blocked_since=None, cluster_recheck_at=None)
+        if not offers:
+            w.status, w.finished_at = "skipped", _now()
+            events = [("wave", "skipped", {"wave": index})]
+            events += _next_wave(run)
+            return events
+        w.status = "running"
+        run.status = "wave_running"
+        return [("wave", "started", {"wave": index, "target_version": run.target_version,
+                                     "hosts": [usable[i].name for i in ordered], "skipped": len(skips)})]
+    return apply_ready
+
+
+def _next_wave(run: RolloutRun) -> list:
+    index = run.current_wave + 1
+    if index >= len(run.waves):
+        run.status, run.finished_at = "succeeded", _now()
+        return [("finish", "succeeded")]
+    run.current_wave = index
+    w = wave_row(run, index)
+    if wave_spec(run, index).get("require_approval"):
+        w.status = run.status = "awaiting_approval"
+        return [("wave", "awaiting_approval", {"wave": index})]
+    w.status, run.status = "preflight", "preflight"
+    return []
+
+
+async def _handle_wave_running(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    if any(h.status in HOST_OPEN for h in run.hosts):
+        return None  # strictly one host at a time
+    index = run.current_wave
+    queued = [h for h in hosts_of(run, index) if h.status == "queued"]
+    if not queued:
+        soak = int(wave_spec(run, index).get("soak_minutes") or 0)
+
+        async def apply_soak(db, run):
+            w = wave_row(run, index)
+            w.status, w.soak_until = "soaking", _now() + timedelta(minutes=soak)
+            run.status = "soaking"
+            return [("wave", "soaking", {"wave": index, "minutes": soak})]
+        return apply_soak
+    if problem := await _actor_problem(db, run):
+        return problem
+    reason = await _may_start_now(db, run)
+    if reason:
+        return _to_waiting(reason)
+    nxt = queued[0]
+    host = await db.get(OpnsenseHost, nxt.host_id) if nxt.host_id else None
+    if host is None or not host.enabled:
+        async def apply_skip(db, run):
+            rh = next(h for h in run.hosts if h.id == nxt.id)
+            rh.status, rh.detail = "skipped", _note("host_deleted" if host is None else "host_disabled")
+            return [("host", "skipped", {"host": nxt.host_name})]
+        return apply_skip
+    spec = wave_spec(run, index)
+    prog = run.progress or {}
+
+    if host.cluster_name:
+        fleet = await peers(db, host)
+        statuses = await get_statuses(fleet)
+        done_ids = {h.host_id for h in run.hosts if h.status == "succeeded"}
+        done_peers = [p for p in fleet if p.id in done_ids and p.id != host.id]
+        baselines = {}
+        for w in run.waves:
+            baselines.update(w.baseline or {})
+        # The cluster as a whole must be healthy before a node is touched; nodes done earlier in this run
+        # must pass their checks again (BACKUP updated -> now it may carry the traffic for the MASTER).
+        gate = await rollout_health.evaluate(
+            db, done_peers + [host], baselines, target_version=run.target_version,
+            checks={**(run.plan_snapshot.get("gate_checks") or {}), "carp": True, "compliance": False})
+        # The node itself is not updated yet: only its reachability, the cluster and the locks count for it.
+        failed = [f"{hid}:{c}" for hid, row in gate["hosts"].items() for c, r in row.items()
+                  if r["result"] == "fail" and not (hid == str(host.id) and c in ("version", "gateways", "services"))]
+        if failed:
+            recheck = _ts(prog.get("cluster_recheck_at"))
+            if recheck is None:
+                async def apply_recheck(db, run):
+                    _progress(run, cluster_recheck_at=_iso(_now() + GATE_RECHECK), cluster_gate=gate)
+                    return [("cluster_gate", "recheck", {"host": host.name, "failed": failed})]
+                return apply_recheck
+            if _now() < recheck:
+                return None
+
+            async def apply_cluster_fail(db, run):
+                pause(run, "cluster_gate_failed", cluster=host.cluster_name, host=host.name, failed=_named(run, failed))
+                _progress(run, cluster_recheck_at=None, cluster_gate=gate)
+                return [("cluster_gate", "failed", {"host": host.name, "failed": failed})]
+            return apply_cluster_fail
+        if statuses.get(host.id, {}).get("carp_status") == "MASTER" and not spec.get("carp_handover"):
+            async def apply_master(db, run):
+                pause(run, "master_needs_handover", host=host.name, cluster=host.cluster_name)
+                return [("pause", "paused", {"reason": "master_needs_handover", "host": host.name})]
+            return apply_master
+
+    try:
+        await ensure_no_maintenance(db, [host])
+    except HTTPException:
+        since = _ts(prog.get("blocked_since"))
+        if since is None:
+            async def apply_blocked_start(db, run):
+                _progress(run, blocked_since=_iso(_now()))
+                return [("host", "waiting_for_lock", {"host": host.name})]
+            return apply_blocked_start
+        if _now() - since < BLOCKED_PATIENCE:
+            return None
+
+        async def apply_blocked(db, run):
+            pause(run, "blocked_by_maintenance", host=host.name)
+            _progress(run, blocked_since=None)
+            return [("pause", "paused", {"reason": "blocked_by_maintenance", "host": host.name})]
+        return apply_blocked
+
+    async def apply_start(db, run):
+        await policy_lock(db)
+        rh = next(h for h in run.hosts if h.id == nxt.id)
+        # Idempotency guard: a host that already has a schedule never gets a second one.
+        if rh.status != "queued" or rh.scheduled_maintenance_id is not None:
+            return []
+        now = _now()
+        sched = ScheduledMaintenance(
+            host_id=host.id, host_name=host.name, action=run.action, run_at=now, status="pending",
+            actor=run.started_by, created_at=now, carp_handover=bool(spec.get("carp_handover") and host.cluster_name),
+            auto_verify=True, origin="rollout",
+            progress={"rollout": {"run_id": str(run.id), "wave": index, "plan": run.plan_name}},
+        )
+        db.add(sched)
+        await db.flush()
+        rh.scheduled_maintenance_id, rh.status, rh.started_at = sched.id, "running", now
+        _progress(run, blocked_since=None, cluster_recheck_at=None)
+        return [("host", "started", {"host": host.name, "schedule_id": str(sched.id), "handover": sched.carp_handover})]
+    return apply_start
+
+
+async def _handle_soaking(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    w = wave_row(run, run.current_wave)
+    if w.soak_until and _now() < w.soak_until:
+        return None
+
+    async def apply(db, run):
+        w = wave_row(run, run.current_wave)
+        w.status, w.gate_attempts = "gate", 0
+        run.status = "gate"
+        _progress(run, gate_recheck_at=None)
+        return []
+    return apply
+
+
+async def _handle_gate(db: AsyncSession, run: RolloutRun) -> Decision | None:
+    recheck = _ts((run.progress or {}).get("gate_recheck_at"))
+    if recheck and _now() < recheck:
+        return None
+    index = run.current_wave
+    w = wave_row(run, index)
+    done = [h for h in hosts_of(run, index) if h.status == "succeeded" and h.host_id]
+    hosts = [h for h in [await db.get(OpnsenseHost, d.host_id) for d in done] if h is not None]
+    gate = await rollout_health.evaluate(db, hosts, w.baseline or {}, target_version=run.target_version,
+                                         checks=run.plan_snapshot.get("gate_checks"),
+                                         compliance_mode=run.plan_snapshot.get("compliance_mode", "warn"))
+
+    async def apply(db, run):
+        w = wave_row(run, index)
+        w.gate_result = gate
+        w.gate_attempts = (w.gate_attempts or 0) + 1
+        failed = rollout_health.failed_checks(gate)
+        events = [("gate", gate["verdict"], {"wave": index, "attempt": w.gate_attempts, "failed": failed, "result": gate})]
+        if gate["verdict"] == "fail":
+            if w.gate_attempts < 2:
+                _progress(run, gate_recheck_at=_iso(_now() + GATE_RECHECK))
+                return events
+            _progress(run, gate_recheck_at=None)
+            pause(run, "gate_failed", wave=w.label, failed=_named(run, failed))
+            return events + [("pause", "paused", {"reason": "gate_failed"})]
+        w.status, w.finished_at = "passed", _now()
+        _progress(run, gate_recheck_at=None)
+        return events + _next_wave(run)
+    return apply
+
+
+_HANDLERS = {
+    "waiting_window": _handle_waiting_window,
+    "resuming": _handle_resuming,
+    "preflight": _handle_preflight,
+    "wave_running": _handle_wave_running,
+    "soaking": _handle_soaking,
+    "gate": _handle_gate,
+}
+
+
+# ── automatic start at a window occurrence ────────────────────────────────────
+
+async def auto_start(db: AsyncSession) -> int:
+    started = 0
+    plans = (await db.scalars(select(RolloutPlan).where(
+        RolloutPlan.auto_start.is_(True), RolloutPlan.activated_at.is_not(None), RolloutPlan.window_id.is_not(None)))).all()
+    for plan in plans:
+        window = await db.get(MaintenanceWindow, plan.window_id)
+        occ = window_service.startable(window, _now()) if window else None
+        if occ is None:
+            continue
+        if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
+            continue
+        if (plan.auto_start_error or {}).get("occurrence") == _iso(occ.start):
+            continue  # already reported for this occurrence
+        await policy_lock(db)
+        plan = await rollout_service.load_plan(db, plan.id)
+        if await db.scalar(select(RolloutRun.id).where(RolloutRun.plan_id == plan.id, RolloutRun.window_occurrence == occ.start)):
+            continue  # another backend started it while this one waited for the lock (not a refusal)
+        resolved = await rollout_service.resolve(db, plan)
+        problem = None
+        if plan.action != "update":
+            problem = _note("auto_start_update_only")
+        elif plan.activated_fingerprint != rollout_service.fingerprint(plan, resolved):
+            problem = _note("plan_changed")  # someone changed the plan, its groups or its window since activation
+        elif not await may_roll_out(db, plan.activated_by or ""):
+            problem = _note("actor_invalid", user=plan.activated_by)
+        elif resolved["errors"]:
+            problem = _note("plan_invalid", reason=resolved["errors"][0]["code"])
+        elif (other := await active_run(db)) is not None:
+            problem = _note("other_run_active", plan=other.plan_name)
+        if problem:
+            plan.auto_start_error = {"occurrence": _iso(occ.start), "at": _iso(_now()), **problem}
+            await db.commit()
+            token = audit_service.context.set({"actor": "system:rollout", "id": str(plan.id)})
+            try:
+                await audit_service.persist(kind="rollout", action="auto_start", status="refused", entity_id=str(plan.id),
+                                            entity_name=plan.name, details=plan.auto_start_error)
+            finally:
+                audit_service.context.reset(token)
+            continue
+        try:
+            run = await create_run(db, plan, resolved, started_by=plan.activated_by, trigger="window", occurrence=occ.start)
+            plan.auto_start_error = None
+            await db.commit()
+        except IntegrityError:
+            await db.rollback()  # another backend started it for this occurrence
+            continue
+        started += 1
+        await _audit(run, "start", "started", trigger="window", occurrence=_iso(occ.start),
+                     hosts=sum(len(w["hosts"]) for w in resolved["waves"]))
+    await db.commit()
+    return started
+
+
+# ── human actions (called by the router under policy_lock) ────────────────────
+
+async def count_active(db: AsyncSession) -> int:
+    return await db.scalar(select(func.count()).select_from(RolloutRun).where(RolloutRun.status.in_(ACTIVE))) or 0

backend/app/services/rollout_health.py

@@ -0,0 +1,218 @@
+"""Health gate of a firmware rollout (1.25): is a host, after its update, as healthy as before?
+
+``baseline(db, host)`` is taken right before a wave starts, ``evaluate(...)`` after the soak time
+(and before each node of an already touched cluster, and on resume). Every check of every host
+yields ``{"result": ok|warn|fail|skip, "code": <machine code>, "params": {...}}``; the UI translates
+the code (rollouts.gate.code.*), so no German prose travels through here.
+
+What the checks look at (all read-only on the firewall, except that the compliance check downloads
+the configuration like the 6-hourly security check does):
+
+* online     — enabled, the API answers, firmware job idle (``running == ready``).
+* version    — the installed version is the rollout's target version.
+* carp       — per cluster: every node online, exactly one MASTER per VHID, no node in CARP
+               maintenance mode, no CARP auto-revert timer running.
+* gateways   — gateways that were up in the baseline are up again; a gateway that was already bad
+               before only warns (1.24 ``get_gateway_health``).
+* services   — services that ran in the baseline run again.
+* locks      — no open maintenance operation, restore or unconfirmed deployment in the host/cluster.
+* compliance — security score not lower than before: warn (default) or fail. A weak signal: an
+               update may legitimately change defaults.
+
+A reading that cannot be taken is never "ok": unreadable now but readable in the baseline fails,
+unreadable in both is "skip" (nothing to compare; e.g. an API key without that permission).
+"""
+import asyncio
+from collections import defaultdict
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.carp_maintenance_window import CarpMaintenanceWindow
+from app.models.compliance_result import ComplianceResult
+from app.models.host import OpnsenseHost
+from app.services import compliance_service, gateway_health_service, opnsense_client as client
+from app.services.host_maintenance_service import _shared_change_blockers, peers
+from app.services.host_status_service import get_statuses
+from app.services.resource_monitor_service import _services_compact
+
+CHECKS = ("online", "version", "carp", "gateways", "services", "locks", "compliance")
+DEFAULT_CHECKS = {"online": True, "version": True, "carp": True, "gateways": True, "services": True, "locks": True,
+                  "compliance": False}
+ALWAYS_ON = ("online", "version", "locks")  # a rollout without these would be blind
+_RANK = {"skip": 0, "ok": 0, "warn": 1, "fail": 2}
+
+
+def _r(result: str, code: str, **params) -> dict:
+    return {"result": result, "code": code, "params": params}
+
+
+async def _optional(coro):
+    try:
+        return await coro
+    except client.OpnsenseAPIError:
+        return None
+
+
+async def _gateways(host) -> dict | None:
+    gws = await _optional(gateway_health_service.get_gateway_health(host))
+    return None if gws is None else {g.name: g.state for g in gws}
+
+
+async def _services(host) -> dict | None:
+    listing = await _optional(client.search_services(host))
+    return None if listing is None else _services_compact(listing)
+
+
+async def baseline(db: AsyncSession, host: OpnsenseHost) -> dict:
+    info, gateways, services = await asyncio.gather(
+        _optional(client.get_firmware_info(host)), _gateways(host), _services(host))
+    score = await db.scalar(select(ComplianceResult.score).where(ComplianceResult.host_id == host.id))
+    return {
+        "version": (info or {}).get("product_version"),
+        "gateways": gateways,
+        "services": services,
+        "compliance_score": score,
+    }
+
+
+def _check_gateways(before: dict | None, now: dict | None) -> dict:
+    if now is None:
+        return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
+    if before is None:
+        bad = sorted(n for n, s in now.items() if s == "offline")
+        return _r("warn", "gateway_down", names=", ".join(bad)) if bad else _r("ok", "no_baseline")
+    down = sorted(n for n, s in before.items() if s in ("online", "unknown") and now.get(n) == "offline")
+    gone = sorted(n for n in before if n not in now)
+    degraded = sorted(n for n, s in now.items() if s == "degraded" and before.get(n) != "degraded")
+    still_bad = sorted(n for n, s in before.items() if s in ("offline", "degraded") and now.get(n) in ("offline", "degraded"))
+    # "unknown" (not monitored) going to "offline" is a real change too: monitoring just started and reports down.
+    if down:
+        return _r("fail", "gateway_down", names=", ".join(down))
+    if gone:
+        return _r("warn", "gateway_missing", names=", ".join(gone))
+    if degraded:
+        return _r("warn", "gateway_degraded", names=", ".join(degraded))
+    if still_bad:
+        return _r("warn", "gateway_bad_before", names=", ".join(still_bad))
+    return _r("ok", "ok")
+
+
+def _check_services(before: dict | None, now: dict | None) -> dict:
+    if now is None:
+        return _r("skip", "unreadable_both") if before is None else _r("fail", "unreadable")
+    if before is None:
+        return _r("ok", "no_baseline")
+    stopped = sorted(name.split("/")[0] for name, running in before.items() if running == 1 and now.get(name) != 1)
+    if stopped:
+        return _r("fail", "service_stopped", names=", ".join(dict.fromkeys(stopped)))
+    return _r("ok", "ok")
+
+
+async def _check_carp(db: AsyncSession, fleet: list[OpnsenseHost], statuses: dict) -> dict:
+    offline = sorted(h.name for h in fleet if h.enabled and statuses.get(h.id, {}).get("status") != "online")
+    if offline:
+        return _r("fail", "node_offline", names=", ".join(offline))
+    online = [h for h in fleet if statuses.get(h.id, {}).get("status") == "online"]
+    demoted = sorted(h.name for h in online if statuses[h.id].get("carp_maintenance_mode"))
+    if demoted:
+        return _r("fail", "maintenance_mode", names=", ".join(demoted))
+    timers = (await db.scalars(select(CarpMaintenanceWindow.host_name).where(
+        CarpMaintenanceWindow.host_id.in_([h.id for h in fleet]), CarpMaintenanceWindow.status == "active"))).all()
+    if timers:
+        return _r("fail", "auto_revert_active", names=", ".join(sorted(timers)))
+    masters: dict[str, int] = defaultdict(int)
+    broken: set[str] = set()
+    for h in online:
+        for vhid, state in (statuses[h.id].get("carp_vips") or {}).items():
+            masters[vhid] += state == "MASTER"
+            if state not in ("MASTER", "BACKUP"):
+                broken.add(vhid)
+    wrong = sorted(set(v for v, n in masters.items() if n != 1) | broken, key=lambda v: (len(v), v))
+    if wrong:
+        return _r("fail", "vhid_roles", vhids=", ".join(wrong))
+    if not masters and not any(statuses[h.id].get("carp_status") == "MASTER" for h in online):
+        return _r("fail", "no_master")
+    return _r("ok", "ok")
+
+
+async def evaluate(db: AsyncSession, hosts: list[OpnsenseHost], baselines: dict, *, target_version: str | None,
+                   checks: dict | None = None, compliance_mode: str = "warn") -> dict:
+    """{"hosts": {host_id: {check: result}}, "verdict": ok|warn|fail}. ``baselines`` is keyed by str(host.id)."""
+    enabled = {**DEFAULT_CHECKS, **(checks or {})}
+    for name in ALWAYS_ON:
+        enabled[name] = True
+    results: dict[str, dict] = {}
+    fleets: dict[str, list[OpnsenseHost]] = {}
+    for host in hosts:
+        fleets[str(host.id)] = await peers(db, host) if host.cluster_name else [host]
+    every = {h.id: h for fleet in fleets.values() for h in fleet}
+    statuses = await get_statuses(list(every.values()))
+    carp_cache: dict[str, dict] = {}
+    for host in hosts:
+        hid = str(host.id)
+        before = baselines.get(hid) or {}
+        row: dict[str, dict] = {}
+        st = statuses.get(host.id, {})
+        running = await _optional(client.get_firmware_running(host)) if st.get("status") == "online" else None
+        if not host.enabled:
+            row["online"] = _r("fail", "disabled")
+        elif st.get("status") != "online":
+            row["online"] = _r("fail", "offline")
+        elif (running or {}).get("status") != "ready":
+            row["online"] = _r("fail", "firmware_busy")
+        else:
+            row["online"] = _r("ok", "ok")
+        reachable = row["online"]["result"] == "ok"
+
+        if enabled["version"]:
+            info = await _optional(client.get_firmware_info(host)) if reachable else None
+            version = (info or {}).get("product_version")
+            if not target_version:
+                row["version"] = _r("skip", "no_target")
+            elif version is None:
+                row["version"] = _r("fail", "unreadable")
+            elif version != target_version:
+                row["version"] = _r("fail", "version_mismatch", expected=target_version, actual=version)
+            else:
+                row["version"] = _r("ok", "ok", actual=version)
+
+        if enabled["carp"]:
+            if not host.cluster_name:
+                row["carp"] = _r("skip", "no_cluster")
+            else:
+                if host.cluster_name not in carp_cache:
+                    carp_cache[host.cluster_name] = await _check_carp(db, fleets[hid], statuses)
+                row["carp"] = carp_cache[host.cluster_name]
+
+        if enabled["gateways"]:
+            row["gateways"] = _check_gateways(before.get("gateways"), await _gateways(host) if reachable else None)
+        if enabled["services"]:
+            row["services"] = _check_services(before.get("services"), await _services(host) if reachable else None)
+
+        blockers = await _shared_change_blockers(db, fleets[hid])
+        row["locks"] = _r("fail", "open_locks", count=len(blockers)) if blockers else _r("ok", "ok")
+
+        if enabled["compliance"]:
+            base_score = before.get("compliance_score")
+            if base_score is None:
+                row["compliance"] = _r("skip", "no_baseline")
+            elif not reachable:
+                row["compliance"] = _r("fail", "unreadable")
+            else:
+                fresh = await compliance_service.check_host(db, host)
+                if fresh.score is None:
+                    row["compliance"] = _r("warn", "unreadable")
+                elif fresh.score < base_score:
+                    row["compliance"] = _r("fail" if compliance_mode == "fail" else "warn", "score_lower",
+                                           before=base_score, after=fresh.score)
+                else:
+                    row["compliance"] = _r("ok", "ok", before=base_score, after=fresh.score)
+        results[hid] = row
+    worst = max((_RANK[c["result"]] for row in results.values() for c in row.values()), default=0)
+    return {"hosts": results, "verdict": ("ok", "warn", "fail")[worst]}
+
+
+def failed_checks(gate: dict) -> list[str]:
+    """Flat "host-id:check" list of everything that failed (for notifications and the audit trail)."""
+    return [f"{hid}:{name}" for hid, row in (gate.get("hosts") or {}).items() for name, c in row.items() if c["result"] == "fail"]

backend/app/services/rollout_service.py

@@ -0,0 +1,138 @@
+"""Rollout plans: resolving waves to hosts, validation, preview token / activation fingerprint.
+
+The run engine (rollout_engine.py) builds on ``resolve``: what a preview shows is exactly what a run
+freezes when it starts.
+"""
+import hashlib
+import json
+from datetime import datetime, timezone
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+from sqlalchemy.orm import selectinload
+
+from app.models.host import OpnsenseHost
+from app.models.maintenance_window import MaintenanceWindow
+from app.models.policy import DeviceGroup
+from app.models.rollout import RolloutPlan
+from app.services import rollout_health, window_service
+
+# Hard limits: an unattended engine that fires firmware updates must stay small and explainable.
+MAX_WAVES = 10
+MAX_HOSTS = 100
+MAX_SOAK_MINUTES = 24 * 60
+
+
+def _issue(code: str, **params) -> dict:
+    return {"code": code, "params": params}
+
+
+async def resolve(db: AsyncSession, plan: RolloutPlan) -> dict:
+    """Waves with their hosts (a host in several waves belongs to the earliest), errors that block a
+    start/activation, and warnings."""
+    errors: list[dict] = []
+    warnings: list[dict] = []
+    seen: dict = {}
+    duplicates: set[str] = set()
+    waves = []
+    group_ids = {g.id for w in plan.waves for g in w.groups}
+    groups = {g.id: g for g in (await db.scalars(
+        select(DeviceGroup).where(DeviceGroup.id.in_(group_ids)).options(selectinload(DeviceGroup.hosts)))).all()} if group_ids else {}
+    for index, wave in enumerate(plan.waves):
+        hosts: list[OpnsenseHost] = []
+        for group in sorted(wave.groups, key=lambda g: g.name):
+            for host in sorted(groups[group.id].hosts, key=lambda h: h.name):
+                if host.id in seen:
+                    if seen[host.id] != index:
+                        duplicates.add(host.name)
+                    continue
+                seen[host.id] = index
+                hosts.append(host)
+        if not hosts:
+            warnings.append(_issue("empty_wave", wave=wave.label))
+        clusters = sorted({h.cluster_name for h in hosts if h.cluster_name})
+        if clusters and not wave.carp_handover:
+            warnings.append(_issue("master_needs_handover", wave=wave.label, clusters=", ".join(clusters)))
+        waves.append({
+            "index": index, "label": wave.label, "soak_minutes": wave.soak_minutes, "carp_handover": wave.carp_handover,
+            "require_approval": wave.require_approval, "groups": sorted(g.name for g in wave.groups),
+            "hosts": [{"id": str(h.id), "name": h.name, "cluster_name": h.cluster_name, "enabled": h.enabled} for h in hosts],
+        })
+    if duplicates:
+        warnings.append(_issue("duplicate_hosts", names=", ".join(sorted(duplicates))))
+    all_hosts = [h for w in waves for h in w["hosts"]]
+    disabled = sorted(h["name"] for h in all_hosts if not h["enabled"])
+    if disabled:
+        warnings.append(_issue("disabled_hosts", names=", ".join(disabled)))
+    if not plan.waves:
+        errors.append(_issue("no_waves"))
+    elif not all_hosts:
+        errors.append(_issue("no_hosts"))
+    if len(plan.waves) > MAX_WAVES:
+        errors.append(_issue("too_many_waves", limit=MAX_WAVES))
+    if len(all_hosts) > MAX_HOSTS:
+        errors.append(_issue("too_many_hosts", limit=MAX_HOSTS))
+
+    # Clusters: where are their nodes?
+    cluster_names = sorted({h["cluster_name"] for h in all_hosts if h["cluster_name"]})
+    for cluster in cluster_names:
+        members = (await db.scalars(select(OpnsenseHost.id).where(OpnsenseHost.cluster_name == cluster))).all()
+        in_plan = {h["id"]: w["index"] for w in waves for h in w["hosts"] if h["cluster_name"] == cluster}
+        partial = len(in_plan) < len(members)
+        split = len(set(in_plan.values())) > 1
+        if plan.action == "major_update" and (partial or split):
+            # Never two major releases side by side in one cluster longer than one wave.
+            errors.append(_issue("cluster_split", cluster=cluster))
+        elif partial:
+            warnings.append(_issue("cluster_partial", cluster=cluster))
+        elif split:
+            warnings.append(_issue("mixed_versions", cluster=cluster, wave=waves[max(in_plan.values())]["label"]))
+
+    window = await db.get(MaintenanceWindow, plan.window_id) if plan.window_id else None
+    if plan.auto_start and plan.action != "update":
+        errors.append(_issue("auto_start_update_only"))
+    if plan.auto_start and window is None:
+        errors.append(_issue("auto_start_needs_window"))
+    if window is None:
+        warnings.append(_issue("no_window"))
+    elif not window.enabled:
+        warnings.append(_issue("window_disabled"))
+    now = datetime.now(timezone.utc)
+    return {
+        "waves": waves, "errors": errors, "warnings": warnings,
+        "window": {"id": str(window.id), "name": window.name, "definition": window_service.definition(window),
+                   "next": [o.as_dict() for o in window_service.occurrences(window, now, 3)] if window.enabled else []}
+        if window else None,
+    }
+
+
+def definition(plan: RolloutPlan) -> dict:
+    return {
+        "id": str(plan.id), "name": plan.name, "action": plan.action, "window_id": str(plan.window_id) if plan.window_id else None,
+        "auto_start": plan.auto_start, "gate_checks": effective_checks(plan.gate_checks), "compliance_mode": plan.compliance_mode,
+        "waves": [{"position": w.position, "label": w.label, "soak_minutes": w.soak_minutes, "carp_handover": w.carp_handover,
+                   "require_approval": w.require_approval, "group_ids": sorted(str(g.id) for g in w.groups)} for w in plan.waves],
+    }
+
+
+def effective_checks(checks: dict | None) -> dict:
+    merged = {**rollout_health.DEFAULT_CHECKS, **{k: bool(v) for k, v in (checks or {}).items() if k in rollout_health.CHECKS}}
+    for name in rollout_health.ALWAYS_ON:
+        merged[name] = True
+    return merged
+
+
+def fingerprint(plan: RolloutPlan, resolved: dict) -> str:
+    """Covers the plan, every resolved host (id, name, cluster, enabled) and the window definition: a change
+    to any of them invalidates a preview and stops an activated automatic start."""
+    state = {
+        "plan": definition(plan),
+        "hosts": [[w["index"], h["id"], h["name"], h["cluster_name"], h["enabled"]] for w in resolved["waves"] for h in w["hosts"]],
+        "window": (resolved.get("window") or {}).get("definition"),
+    }
+    return hashlib.sha256(json.dumps(state, sort_keys=True, default=str).encode()).hexdigest()
+
+
+async def load_plan(db: AsyncSession, plan_id) -> RolloutPlan | None:
+    return await db.scalar(
+        select(RolloutPlan).where(RolloutPlan.id == plan_id).execution_options(populate_existing=True))

backend/app/services/scheduled_maintenance_service.py

@@ -15,18 +15,31 @@ the UI) but is NOT wired into the NotificationSettings webhook/email
 system — that system is built around persistent NotificationProblem rows
 with their own admin-configurable toggles, and adding a new problem kind
 for this is a bigger change than this feature's own scope. Flagged here,
-not silently dropped, in case it's wanted later.
+not silently dropped, in case it's wanted later. (Later: the failure is reported by
+notification_checks.check_scheduled_maintenance_failed since 1.17.)
+
+auto_verify (1.25, set for every rollout step): instead of leaving the operation "submitted"
+for a human, the run waits for the job (phase job_wait) and then checks the result itself
+(phase verify: firmware idle for a while, a reboot really happened when one was due, version
+readable). Only then the operation is closed as "acknowledged"; anything unclear ends as
+"unknown" (stays locked, notification). Without auto_verify nothing changes.
+
+Concurrency: due rows and running rows are each taken with SELECT ... FOR UPDATE SKIP LOCKED,
+one row per transaction, so a second backend (briefly possible during a web update) can never
+fire or step the same row at the same time.
 """
 import logging
 from datetime import datetime, timedelta, timezone
 
 from fastapi import HTTPException
-from sqlalchemy import select
+from sqlalchemy import select, true
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.models.host import OpnsenseHost
 from app.models.host_operation import HostOperation
 from app.models.scheduled_maintenance import ScheduledMaintenance
+from contextlib import contextmanager
+
 from app.services import audit_service, backup_store, opnsense_client as client
 from app.services.host_maintenance_service import ACTIVE, ensure_no_maintenance, inspect_carp_switch, inspect_host, peers
 from app.services.host_status_service import get_statuses
@@ -51,40 +64,93 @@ HANDOVER_TIMEOUT = timedelta(minutes=3)
 LEAVE_TIMEOUT = timedelta(minutes=5)
 JOB_GRACE = timedelta(minutes=5)  # no sign of a job after this long: treat a package update as already done
 JOB_TIMEOUT = {"reboot": timedelta(minutes=20), "update": timedelta(minutes=60), "major_update": timedelta(minutes=150)}
+# auto_verify: the firmware must report "ready" for this long in a row (a reboot at the end of an update
+# can follow a short "ready" moment), and the whole check gives up after VERIFY_TIMEOUT.
+VERIFY_SETTLE = timedelta(seconds=60)
+VERIFY_TIMEOUT = timedelta(minutes=20)
+# A rollout step is created with run_at=now and fires on the next poll. Found much later (OPNexus was down,
+# e.g. during an update), it no longer belongs to the decision that created it: the window may have closed
+# and the run may have been paused meanwhile. It is then not fired; the rollout engine re-queues the host.
+ROLLOUT_FIRE_GRACE = timedelta(minutes=5)
 
 
 def _now() -> datetime:
     return datetime.now(timezone.utc)
 
 
+@contextmanager
+def _as_actor(sched: ScheduledMaintenance):
+    """Audit events of an unattended run name the person who planned it (not the rollback worker)."""
+    token = audit_service.context.set({"actor": f"{sched.actor} (geplant)"[:150], "id": str(sched.id)})
+    try:
+        yield
+    finally:
+        audit_service.context.reset(token)
+
+
+def _rollout_details(sched: ScheduledMaintenance) -> dict:
+    rollout = (sched.progress or {}).get("rollout")
+    return {"rollout": rollout} if rollout else {}
+
+
 async def fire_due(db: AsyncSession) -> dict:
-    due = list(
-        (
-            await db.scalars(
-                select(ScheduledMaintenance).where(
-                    ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now()
-                )
-            )
-        ).all()
-    )
-    fired = 0
-    blocked = 0
-    for sched in due:
-        try:
-            await _fire_one(db, sched)
-        except Exception:
-            logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
-            sched.status = "failed"
-            sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
+    due = fired = blocked = 0
+    seen: set = set()
+    while True:
+        # One row per transaction, locked: a second backend skips it instead of firing it again.
+        sched = await db.scalar(
+            select(ScheduledMaintenance)
+            .where(ScheduledMaintenance.status == "pending", ScheduledMaintenance.run_at <= _now(),
+                   ScheduledMaintenance.id.not_in(seen) if seen else true())
+            .order_by(ScheduledMaintenance.run_at)
+            .limit(1)
+            .with_for_update(skip_locked=True)
+            .execution_options(populate_existing=True)
+        )
+        if sched is None:
             await db.commit()
+            break
+        seen.add(sched.id)
+        due += 1
+        with _as_actor(sched):
+            try:
+                await _fire_one(db, sched)
+            except Exception:
+                logger.exception("Geplante Wartung %s fehlgeschlagen", sched.id)
+                await db.rollback()
+                sched = await db.get(ScheduledMaintenance, sched.id, populate_existing=True)
+                if sched.status in ("pending", "running") and sched.phase is None:
+                    sched.status = "failed"
+                    sched.result = {"error": "Unerwarteter Fehler beim Ausführen"}
+                await db.commit()
+        await db.commit()  # never keep the row lock past this row
         if sched.status == "failed":
             blocked += 1
         else:
             fired += 1
-    return {"due": len(due), "fired": fired, "blocked_or_failed": blocked}
+    return {"due": due, "fired": fired, "blocked_or_failed": blocked}
+
+
+def _reboot_expected(action: str, state: dict) -> bool:
+    return action in ("reboot", "major_update") or bool(state.get("reboot_required"))
+
+
+async def _boot_time(host: OpnsenseHost) -> str | None:
+    """systemTime's "boottime" (Diagnostics SystemController::systemTimeAction, read-only, already used by the
+    resource sampler). A different value afterwards proves a reboot even if no poll caught the host offline."""
+    try:
+        return (await client.get_system_time(host)).get("boottime") or None
+    except client.OpnsenseAPIError:
+        return None
 
 
 async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
+    if sched.origin == "rollout" and _now() - sched.run_at > ROLLOUT_FIRE_GRACE:
+        sched.status = "failed"
+        sched.result = {"error": "Rollout-Schritt nicht rechtzeitig gestartet, nichts ausgeführt", "not_fired": True}
+        await db.commit()
+        logger.warning("Geplante Wartung %s: Rollout-Schritt überfällig, nicht ausgeführt", sched.id)
+        return
     host = await db.get(OpnsenseHost, sched.host_id)
     if not host or not host.enabled:
         sched.status = "failed"
@@ -137,6 +203,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
         logger.warning("Geplante Wartung %s blockiert: %s", sched.id, sched.result["error"])
         return
 
+    boot_before = await _boot_time(host) if sched.auto_verify else None
     op = HostOperation(
         host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
         expires_at=_now(), fingerprint="", snapshot=state, status="starting", started_at=_now(),
@@ -147,14 +214,23 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
     await audit_service.persist(
         kind="host", action=sched.action, status="requested",
         host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
-        details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state},
+        details={"scheduled": True, "schedule_id": str(sched.id), "snapshot": state, **_rollout_details(sched)},
     )
     try:
         result = await _ACTION_FN[sched.action](host)
         op.status = "submitted"
-        op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
-        sched.status = "succeeded"
-        sched.result = {"response": result}
+        if sched.auto_verify:
+            # Not done yet: the poll loop waits for the job and checks the result (job_wait -> verify).
+            op.message = "Geplant übergeben. Abschluss wird automatisch geprüft."
+            sched.phase, sched.phase_since = "job_wait", _now()
+            _progress(sched, handover=False, job_started_at=_now().isoformat(), seen_down=False, seen_running=False,
+                      version_before=state.get("version"), boot_before=boot_before,
+                      reboot_expected=_reboot_expected(sched.action, state))
+            sched.result = {"response": result}
+        else:
+            op.message = "Geplant übergeben. Abschluss und Erreichbarkeit prüfen; anschließend Wartung abschließen."
+            sched.status = "succeeded"
+            sched.result = {"response": result}
     except client.OpnsenseAPIError as exc:
         op.status = "unknown"
         op.message = "Ausgang unklar: " + str(exc)[:800] + ". Nicht automatisch wiederholen."
@@ -166,7 +242,7 @@ async def _fire_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
     await audit_service.persist(
         kind="host", action=sched.action, status=op.status,
         host_id=str(host.id), host_name=host.name, entity_id=str(op.id),
-        message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result},
+        message=op.message, details={"scheduled": True, "schedule_id": str(sched.id), "response": result, **_rollout_details(sched)},
     )
 
 
@@ -182,21 +258,24 @@ async def _audit(host, sched, op, action, status, message=None, **details):
     await audit_service.persist(
         kind="host", action=action, status=status, host_id=str(host.id), host_name=host.name,
         entity_id=str(op.id) if op else None, message=message,
-        details={"scheduled": True, "schedule_id": str(sched.id), "handover": True, **details},
+        details={"scheduled": True, "schedule_id": str(sched.id), "handover": bool((sched.progress or {}).get("handover")),
+                 **_rollout_details(sched), **details},
     )
 
 
 async def _end(db, sched, host, op, *, ok, message, op_status=None, **result):
+    handover = bool((sched.progress or {}).get("handover"))
     sched.status = "succeeded" if ok else "failed"
     sched.phase = None
-    sched.result = {"handover": True, **result} if ok else {"error": message[:900], **result}
+    sched.result = ({"handover": handover, "auto_verified": bool(sched.auto_verify), **result} if ok
+                    else {"error": message[:900], **result})
     if op is not None:
         op.status = op_status or ("acknowledged" if ok else "unknown")
         op.message = message[:1000]
     await db.commit()
     await _audit(host, sched, op, sched.action, "completed" if ok else (op_status or "unknown"), message)
     if not ok:
-        logger.warning("Geplante Wartung %s (CARP-Übergabe) beendet: %s", sched.id, message)
+        logger.warning("Geplante Wartung %s beendet: %s", sched.id, message)
 
 
 async def _toggle(host, expected: str) -> None:
@@ -230,6 +309,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
         await fail(" ".join(blocked))
         return
 
+    boot_before = await _boot_time(host) if sched.auto_verify else None
     now = _now()
     op = HostOperation(
         host_id=host.id, host_name=host.name, actor=sched.actor, action=sched.action,
@@ -237,6 +317,7 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
     )
     db.add(op)
     sched.status = "running"
+    _progress(sched, handover=True, boot_before=boot_before, reboot_expected=_reboot_expected(sched.action, state))
     await db.commit()
     sched.host_operation_id = op.id
     await _audit(host, sched, op, "carp_enter", "requested", snapshot=state)
@@ -255,32 +336,56 @@ async def _start_handover(db: AsyncSession, sched: ScheduledMaintenance, host: O
 
 
 async def advance_running(db: AsyncSession) -> int:
-    """One step for every handover run in progress. Returns how many are still running."""
-    rows = list((await db.scalars(
-        select(ScheduledMaintenance).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
+    """One step for every run in progress (handover and/or auto_verify). Returns how many are still running."""
+    ids = list((await db.scalars(
+        select(ScheduledMaintenance.id).where(ScheduledMaintenance.status == "running", ScheduledMaintenance.phase.is_not(None))
     )).all())
-    for sched in rows:
-        host = await db.get(OpnsenseHost, sched.host_id)
-        op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
-        try:
-            if host is None:
-                raise RuntimeError("Host nicht mehr vorhanden")
-            if op is not None and op.status not in ACTIVE:
-                # A human closed or cancelled the operation: stop here and touch nothing more.
-                sched.status, sched.phase = "failed", None
-                sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt. CARP-Rolle von {host.name} prüfen."}
-                await db.commit()
-                continue
-            await _STEPS[sched.phase](db, sched, host, op)
-        except Exception as exc:  # a stuck run must not loop forever or take the poll loop down
-            logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
-            if host is not None:
-                await _end(db, sched, host, op, ok=False,
-                           message=f"Unerwarteter Fehler während der CARP-Übergabe ({type(exc).__name__}). "
-                                   f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen.")
-    return sum(1 for s in rows if s.status == "running")
+    await db.commit()
+    still_running = 0
+    for sched_id in ids:
+        # Locked for the whole step: a second backend skips the row instead of toggling or submitting twice.
+        sched = await db.scalar(
+            select(ScheduledMaintenance)
+            .where(ScheduledMaintenance.id == sched_id, ScheduledMaintenance.status == "running",
+                   ScheduledMaintenance.phase.is_not(None))
+            .with_for_update(skip_locked=True)
+            .execution_options(populate_existing=True)
+        )
+        if sched is None:
+            await db.commit()
+            still_running += 1 if await db.scalar(select(ScheduledMaintenance.status).where(ScheduledMaintenance.id == sched_id)) == "running" else 0
+            await db.commit()
+            continue
+        with _as_actor(sched):
+            await _advance_one(db, sched)
+        await db.commit()
+        if sched.status == "running":
+            still_running += 1
+    return still_running
 
 
+async def _advance_one(db: AsyncSession, sched: ScheduledMaintenance) -> None:
+    host = await db.get(OpnsenseHost, sched.host_id)
+    op = await db.get(HostOperation, sched.host_operation_id) if sched.host_operation_id else None
+    try:
+        if host is None:
+            raise RuntimeError("Host nicht mehr vorhanden")
+        if op is not None and op.status not in ACTIVE:
+            # A human closed or cancelled the operation: stop here and touch nothing more.
+            hint = f" CARP-Rolle von {host.name} prüfen." if (sched.progress or {}).get("handover") else ""
+            sched.status, sched.phase = "failed", None
+            sched.result = {"error": f"Vorgang von Hand beendet, Ablauf gestoppt.{hint}"}
+            await db.commit()
+            return
+        await _STEPS[sched.phase](db, sched, host, op)
+    except Exception as exc:  # a stuck run must not loop forever or take the poll loop down
+        logger.exception("Geplante Wartung %s: Schritt %s fehlgeschlagen", sched.id, sched.phase)
+        if host is not None:
+            hint = (f"{host.name} kann im CARP-Wartungsmodus verblieben sein — Rollen prüfen."
+                    if (sched.progress or {}).get("handover") else f"Zustand von {host.name} prüfen.")
+            await _end(db, sched, host, op, ok=False,
+                       message=f"Unerwarteter Fehler während der geplanten Wartung ({type(exc).__name__}). {hint}")
+
 def _since(sched: ScheduledMaintenance) -> timedelta:
     return _now() - (sched.phase_since or _now())
 
@@ -352,17 +457,75 @@ async def _step_job_wait(db, sched, host, op):
         else:
             done = bool(prog.get("seen_down") or prog.get("seen_running")) or elapsed >= JOB_GRACE
         if done:
-            sched.phase, sched.phase_since = "leave", _now()
+            sched.phase = "verify" if sched.auto_verify else "leave"
+            sched.phase_since = _now()
             await db.commit()
             return
     if elapsed >= JOB_TIMEOUT[sched.action]:
-        await _end(db, sched, host, op, ok=False,
-                   message=f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
-                           f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
+        if prog.get("handover"):
+            text = (f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. {host.name} bleibt im "
+                    f"CARP-Wartungsmodus, der Partner trägt den Verkehr. Bitte prüfen und von Hand zurückschalten.")
+        else:
+            text = f"Auftrag „{sched.action}“ auf {host.name} nicht rechtzeitig abgeschlossen. Bitte {host.name} prüfen."
+        await _end(db, sched, host, op, ok=False, message=text)
         return
     await db.commit()
 
 
+async def _step_verify(db, sched, host, op):
+    """auto_verify: is the job really over and the host healthy? Read-only, never repeats the action."""
+    prog = dict(sched.progress or {})
+    handover = bool(prog.get("handover"))
+
+    async def wait_or_give_up(why: str):
+        if _since(sched) >= VERIFY_TIMEOUT:
+            tail = (f" {host.name} bleibt im CARP-Wartungsmodus, der Partner trägt den Verkehr." if handover else "")
+            await _end(db, sched, host, op, ok=False,
+                       message=f"Automatische Prüfung nach „{sched.action}“ nicht bestanden: {why}.{tail} "
+                               f"Bitte {host.name} prüfen und die Wartung von Hand abschließen.")
+            return
+        _progress(sched, waiting_for=why)
+        await db.commit()
+
+    try:
+        running = (await client.get_firmware_running(host)).get("status")
+    except client.OpnsenseAPIError:
+        running = None
+        _progress(sched, seen_down=True)
+    if running != "ready":
+        _progress(sched, ready_since=None)
+        await wait_or_give_up("Firmware-Auftrag läuft noch oder Host nicht erreichbar")
+        return
+    if not prog.get("ready_since"):
+        _progress(sched, ready_since=_now().isoformat())
+        await db.commit()
+        return
+    if _now() - datetime.fromisoformat(prog["ready_since"]) < VERIFY_SETTLE:
+        await db.commit()
+        return
+    boot_now = await _boot_time(host)
+    rebooted = bool(prog.get("seen_down")) or bool(prog.get("boot_before") and boot_now and boot_now != prog["boot_before"])
+    if prog.get("reboot_expected") and not rebooted:
+        await wait_or_give_up("der erwartete Neustart wurde nicht beobachtet")
+        return
+    try:
+        version = (await client.get_firmware_info(host)).get("product_version")
+    except client.OpnsenseAPIError:
+        version = None
+    online = (await get_statuses([host])).get(host.id, {}).get("status") == "online"
+    if not version or not online:
+        await wait_or_give_up("Host nicht vollständig erreichbar (Version/Status nicht lesbar)")
+        return
+    _progress(sched, version_after=version, boot_after=boot_now, rebooted=rebooted, waiting_for=None)
+    if handover:
+        sched.phase, sched.phase_since = "leave", _now()  # only now take the traffic back
+        await db.commit()
+        return
+    await _end(db, sched, host, op, ok=True,
+               message=f"Automatisch geprüft: {host.name} erreichbar, Firmware-Auftrag beendet, Version {version}.",
+               version_before=prog.get("version_before"), version_after=version, rebooted=rebooted)
+
+
 async def _step_leave(db, sched, host, op):
     me, _partner = await _roles(db, host)
     if me.get("status") != "online":
@@ -397,6 +560,8 @@ async def _step_leave_wait(db, sched, host, op):
         except client.OpnsenseAPIError:
             pass
         extra["version_before"] = (sched.progress or {}).get("version_before")
+        if sched.auto_verify:
+            extra["rebooted"] = bool((sched.progress or {}).get("rebooted"))  # what verify saw before leaving
         text = "Automatisch abgeschlossen: Übergabe, Auftrag und Zurückschalten bestätigt."
         if extra.get("reboot_required"):
             text += " Ein Neustart ist noch erforderlich."
@@ -410,6 +575,7 @@ async def _step_leave_wait(db, sched, host, op):
 _STEPS = {
     "handover_wait": _step_handover_wait,
     "job_wait": _step_job_wait,
+    "verify": _step_verify,
     "leave": _step_leave,
     "leave_wait": _step_leave_wait,
 }

backend/app/services/window_service.py

@@ -0,0 +1,147 @@
+"""Occurrences of recurring maintenance windows.
+
+Rules (deliberately simple and written down, because a firmware rollout depends on them):
+
+* A window is a wall-clock start time in its own IANA time zone plus a duration. "Sunday 02:00
+  Europe/Berlin" stays 02:00 local time in summer and in winter.
+* Daylight saving: a local start time that does not exist on that day (spring forward, e.g. 02:30 on
+  the last Sunday of March in Europe/Berlin) moves FORWARD by the gap (03:30). A local time that
+  exists twice (fall back) means the FIRST of the two (fold=0, still summer time). Both follow from
+  Python's zoneinfo with fold=0 and are covered by tests.
+* The duration is real elapsed time (a 3 h window over the fall-back night still lasts 3 h).
+* Exceptions are keyed by the occurrence's local date: "skip" drops it, "move" replaces its start
+  with an absolute instant (same duration and cut-off).
+* New host actions may only START within the first ``start_cutoff_minutes`` of an occurrence
+  (``startable``); actions already running finish even after the window has closed.
+"""
+from dataclasses import dataclass
+from datetime import date, datetime, timedelta, timezone
+from zoneinfo import ZoneInfo, ZoneInfoNotFoundError, available_timezones
+
+from app.models.maintenance_window import MaintenanceWindow
+
+# How far ahead occurrences are searched. A monthly "5th ..." does not exist; the longest real gap
+# between two occurrences is a month, so two years is plenty and bounds the work per call.
+_SEARCH_DAYS = 2 * 366
+MAX_DURATION_MINUTES = 24 * 60
+
+
+@dataclass(frozen=True)
+class Occurrence:
+    local_date: date
+    start: datetime  # UTC
+    end: datetime  # UTC
+    cutoff: datetime  # UTC: no new host action after this instant
+    moved: bool = False
+    skipped: bool = False
+
+    def as_dict(self) -> dict:
+        return {"local_date": self.local_date.isoformat(), "start": self.start.isoformat(), "end": self.end.isoformat(),
+                "cutoff": self.cutoff.isoformat(), "moved": self.moved, "skipped": self.skipped}
+
+
+def valid_timezone(name: str) -> bool:
+    if not name or name not in available_timezones():
+        return False
+    try:
+        ZoneInfo(name)
+    except (ZoneInfoNotFoundError, ValueError):
+        return False
+    return True
+
+
+def _matches(window: MaintenanceWindow, day: date) -> bool:
+    if window.recurrence == "weekly":
+        return day.weekday() in (window.weekdays or [])
+    if window.recurrence == "monthly":
+        if window.month_mode == "day":
+            return day.day == window.month_day
+        if window.month_mode == "nth_weekday":
+            if day.weekday() != window.month_weekday:
+                return False
+            if window.month_nth == -1:
+                return (day + timedelta(days=7)).month != day.month
+            return (day.day - 1) // 7 + 1 == window.month_nth
+    return False
+
+
+def local_start(window: MaintenanceWindow, day: date) -> datetime:
+    """The UTC instant of the window's start on local date ``day`` (DST rules see module docstring)."""
+    tz = ZoneInfo(window.timezone)
+    wall = datetime.combine(day, window.start_time, tzinfo=tz).replace(fold=0)
+    return wall.astimezone(timezone.utc)
+
+
+def _occurrence(window: MaintenanceWindow, day: date, start: datetime, *, moved: bool = False, skipped: bool = False) -> Occurrence:
+    duration = timedelta(minutes=window.duration_minutes)
+    cutoff = timedelta(minutes=min(window.start_cutoff_minutes, window.duration_minutes))
+    return Occurrence(local_date=day, start=start, end=start + duration, cutoff=start + cutoff, moved=moved, skipped=skipped)
+
+
+def occurrences(window: MaintenanceWindow, after: datetime, n: int = 10, *, with_skipped: bool = False) -> list[Occurrence]:
+    """The next ``n`` occurrences whose END lies after ``after`` (so one that is open right now is first).
+    ``with_skipped`` lists skipped ones too (flagged, for the UI); the engine never sees them."""
+    if n <= 0:
+        return []
+    after = after.astimezone(timezone.utc)
+    tz = ZoneInfo(window.timezone)
+    exceptions = {e.occurrence_date: e for e in (window.exceptions or [])}
+    duration = timedelta(minutes=window.duration_minutes)
+    # Start one day early: an occurrence that began yesterday (local) may still be open.
+    day = (after - duration).astimezone(tz).date() - timedelta(days=1)
+    found: list[Occurrence] = []
+    # Moved occurrences can land anywhere, so they are collected separately and merged at the end.
+    for exc in exceptions.values():
+        if exc.kind == "move" and exc.moved_start is not None and _matches(window, exc.occurrence_date):
+            occ = _occurrence(window, exc.occurrence_date, exc.moved_start.astimezone(timezone.utc), moved=True)
+            if occ.end > after:
+                found.append(occ)
+    regular = 0
+    for _ in range(_SEARCH_DAYS):
+        if regular >= n:
+            break
+        if _matches(window, day):
+            exc = exceptions.get(day)
+            if exc is None or (with_skipped and exc.kind == "skip"):
+                occ = _occurrence(window, day, local_start(window, day), skipped=exc is not None)
+                if occ.end > after:
+                    found.append(occ)
+                    regular += 1
+        day += timedelta(days=1)
+    found.sort(key=lambda o: o.start)
+    return found[:n]
+
+
+def current(window: MaintenanceWindow, now: datetime) -> Occurrence | None:
+    """The occurrence open at ``now`` (start <= now < end), if any."""
+    if not window.enabled:
+        return None
+    for occ in occurrences(window, now, 3):
+        if occ.start <= now < occ.end:
+            return occ
+    return None
+
+
+def startable(window: MaintenanceWindow | None, now: datetime) -> Occurrence | None:
+    """The occurrence in which a NEW host action may start right now (start <= now < cutoff)."""
+    if window is None or not window.enabled:
+        return None
+    occ = current(window, now)
+    if occ is not None and occ.start <= now < occ.cutoff:
+        return occ
+    return None
+
+
+def is_occurrence_date(window: MaintenanceWindow, day: date) -> bool:
+    return _matches(window, day)
+
+
+def definition(window: MaintenanceWindow) -> dict:
+    """What a rollout plan's fingerprint covers: changing any of this re-requires activation."""
+    return {
+        "id": str(window.id), "timezone": window.timezone, "recurrence": window.recurrence,
+        "weekdays": sorted(window.weekdays or []), "month_mode": window.month_mode, "month_nth": window.month_nth,
+        "month_weekday": window.month_weekday, "month_day": window.month_day,
+        "start_time": window.start_time.strftime("%H:%M"), "duration_minutes": window.duration_minutes,
+        "start_cutoff_minutes": window.start_cutoff_minutes, "enabled": window.enabled,
+    }

backend/app/version.py

@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
 Versioning scheme: see CHANGELOG.md ("Versionierung").
 """
 
-APP_VERSION = "1.24.0"
-APP_CODENAME = "Merlin"
+APP_VERSION = "1.25.0"
+APP_CODENAME = "Goshawk"

backend/migrations/versions/0021_maintenance_windows.py

@@ -0,0 +1,60 @@
+"""Wiederkehrende Wartungsfenster mit Ausnahmen (1.25).
+
+Revision ID: 0021
+Revises: 0020
+Create Date: 2026-10-21
+"""
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision = "0021"
+down_revision = "0020"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
+    if sa.inspect(op.get_bind()).has_table("maintenance_windows"):
+        return
+    op.create_table(
+        "maintenance_windows",
+        sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
+        sa.Column("name", sa.String(length=120), nullable=False),
+        sa.Column("timezone", sa.String(length=64), nullable=False),
+        sa.Column("recurrence", sa.String(length=16), nullable=False),
+        sa.Column("weekdays", sa.JSON(), nullable=False),
+        sa.Column("month_mode", sa.String(length=16), nullable=True),
+        sa.Column("month_nth", sa.Integer(), nullable=True),
+        sa.Column("month_weekday", sa.Integer(), nullable=True),
+        sa.Column("month_day", sa.Integer(), nullable=True),
+        sa.Column("start_time", sa.Time(), nullable=False),
+        sa.Column("duration_minutes", sa.Integer(), nullable=False),
+        sa.Column("start_cutoff_minutes", sa.Integer(), server_default="60", nullable=False),
+        sa.Column("enabled", sa.Boolean(), server_default="true", nullable=False),
+        sa.Column("created_by", sa.String(length=150), nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
+        sa.PrimaryKeyConstraint("id"),
+        sa.UniqueConstraint("name"),
+    )
+    op.create_table(
+        "maintenance_window_exceptions",
+        sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
+        sa.Column("window_id", postgresql.UUID(as_uuid=True), nullable=False),
+        sa.Column("occurrence_date", sa.Date(), nullable=False),
+        sa.Column("kind", sa.String(length=8), nullable=False),
+        sa.Column("moved_start", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("reason", sa.String(length=300), nullable=False),
+        sa.Column("actor", sa.String(length=150), nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
+        sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="CASCADE"),
+        sa.PrimaryKeyConstraint("id"),
+        sa.UniqueConstraint("window_id", "occurrence_date", name="uq_window_exception_date"),
+    )
+    op.create_index("ix_maintenance_window_exceptions_window_id", "maintenance_window_exceptions", ["window_id"])
+
+
+def downgrade() -> None:
+    op.execute("DROP TABLE IF EXISTS maintenance_window_exceptions")
+    op.execute("DROP TABLE IF EXISTS maintenance_windows")

backend/migrations/versions/0022_scheduled_auto_verify.py

@@ -0,0 +1,22 @@
+"""Geplante Wartung: automatische Prüfung nach dem Auftrag (auto_verify) und Herkunft (manuell/Rollout).
+
+Revision ID: 0022
+Revises: 0021
+Create Date: 2026-10-21
+"""
+from alembic import op
+
+revision = "0022"
+down_revision = "0021"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS auto_verify boolean NOT NULL DEFAULT false")
+    op.execute("ALTER TABLE scheduled_maintenance ADD COLUMN IF NOT EXISTS origin varchar(16) NOT NULL DEFAULT 'manual'")
+
+
+def downgrade() -> None:
+    op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS origin")
+    op.execute("ALTER TABLE scheduled_maintenance DROP COLUMN IF EXISTS auto_verify")

backend/migrations/versions/0023_rollouts.py

@@ -0,0 +1,143 @@
+"""Firmware-Rollouts in Wellen: Pläne, Läufe, Hosts je Lauf; Benachrichtigung bei Rollout-Problemen (1.25).
+
+Revision ID: 0023
+Revises: 0022
+Create Date: 2026-10-21
+"""
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision = "0023"
+down_revision = "0022"
+branch_labels = None
+depends_on = None
+
+UUID = postgresql.UUID(as_uuid=True)
+TS = sa.DateTime(timezone=True)
+
+
+def upgrade() -> None:
+    op.execute("ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS rollout_problems_enabled boolean NOT NULL DEFAULT true")
+    # A pre-Alembic install is bridged with create_all() first (app/migrate.py), which already made the tables.
+    if sa.inspect(op.get_bind()).has_table("rollout_plans"):
+        return
+    op.create_table(
+        "rollout_plans",
+        sa.Column("id", UUID, nullable=False),
+        sa.Column("name", sa.String(120), nullable=False),
+        sa.Column("description", sa.Text(), nullable=False),
+        sa.Column("action", sa.String(16), nullable=False),
+        sa.Column("window_id", UUID, nullable=True),
+        sa.Column("auto_start", sa.Boolean(), server_default="false", nullable=False),
+        sa.Column("gate_checks", sa.JSON(), nullable=False),
+        sa.Column("compliance_mode", sa.String(8), server_default="warn", nullable=False),
+        sa.Column("activated_by", sa.String(150), nullable=True),
+        sa.Column("activated_at", TS, nullable=True),
+        sa.Column("activated_fingerprint", sa.String(64), nullable=True),
+        sa.Column("auto_start_error", sa.JSON(), nullable=True),
+        sa.Column("created_by", sa.String(150), nullable=False),
+        sa.Column("created_at", TS, nullable=False),
+        sa.Column("updated_at", TS, nullable=False),
+        sa.ForeignKeyConstraint(["window_id"], ["maintenance_windows.id"], ondelete="SET NULL"),
+        sa.PrimaryKeyConstraint("id"),
+        sa.UniqueConstraint("name"),
+    )
+    op.create_index("ix_rollout_plans_window_id", "rollout_plans", ["window_id"])
+    op.create_table(
+        "rollout_plan_waves",
+        sa.Column("id", UUID, nullable=False),
+        sa.Column("plan_id", UUID, nullable=False),
+        sa.Column("position", sa.Integer(), nullable=False),
+        sa.Column("label", sa.String(60), nullable=False),
+        sa.Column("soak_minutes", sa.Integer(), nullable=False),
+        sa.Column("carp_handover", sa.Boolean(), server_default="false", nullable=False),
+        sa.Column("require_approval", sa.Boolean(), server_default="false", nullable=False),
+        sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="CASCADE"),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    op.create_index("ix_rollout_plan_waves_plan_id", "rollout_plan_waves", ["plan_id"])
+    op.create_table(
+        "rollout_wave_groups",
+        sa.Column("wave_id", UUID, nullable=False),
+        sa.Column("group_id", UUID, nullable=False),
+        sa.ForeignKeyConstraint(["wave_id"], ["rollout_plan_waves.id"], ondelete="CASCADE"),
+        sa.ForeignKeyConstraint(["group_id"], ["device_groups.id"], ondelete="CASCADE"),
+        sa.PrimaryKeyConstraint("wave_id", "group_id"),
+    )
+    op.create_table(
+        "rollout_runs",
+        sa.Column("id", UUID, nullable=False),
+        sa.Column("plan_id", UUID, nullable=True),
+        sa.Column("plan_name", sa.String(120), nullable=False),
+        sa.Column("plan_snapshot", sa.JSON(), nullable=False),
+        sa.Column("action", sa.String(16), nullable=False),
+        sa.Column("status", sa.String(20), nullable=False),
+        sa.Column("paused_from", sa.String(20), nullable=True),
+        sa.Column("current_wave", sa.Integer(), nullable=False),
+        sa.Column("target_version", sa.String(40), nullable=True),
+        sa.Column("pause_reason", sa.String(40), nullable=True),
+        sa.Column("pause_detail", sa.JSON(), nullable=True),
+        sa.Column("started_by", sa.String(150), nullable=False),
+        sa.Column("trigger", sa.String(8), nullable=False),
+        sa.Column("window_occurrence", TS, nullable=True),
+        sa.Column("revision", sa.Integer(), server_default="0", nullable=False),
+        sa.Column("progress", sa.JSON(), nullable=False),
+        sa.Column("step_at", TS, nullable=True),
+        sa.Column("created_at", TS, nullable=False),
+        sa.Column("finished_at", TS, nullable=True),
+        sa.ForeignKeyConstraint(["plan_id"], ["rollout_plans.id"], ondelete="SET NULL"),
+        sa.PrimaryKeyConstraint("id"),
+        sa.UniqueConstraint("plan_id", "window_occurrence", name="uq_rollout_run_occurrence"),
+    )
+    op.create_index("ix_rollout_runs_plan_id", "rollout_runs", ["plan_id"])
+    op.create_index("ix_rollout_runs_status", "rollout_runs", ["status"])
+    op.create_table(
+        "rollout_run_waves",
+        sa.Column("id", UUID, nullable=False),
+        sa.Column("run_id", UUID, nullable=False),
+        sa.Column("index", sa.Integer(), nullable=False),
+        sa.Column("label", sa.String(60), nullable=False),
+        sa.Column("status", sa.String(20), nullable=False),
+        sa.Column("baseline", sa.JSON(), nullable=True),
+        sa.Column("gate_result", sa.JSON(), nullable=True),
+        sa.Column("soak_until", TS, nullable=True),
+        sa.Column("gate_attempts", sa.Integer(), server_default="0", nullable=False),
+        sa.Column("approved_by", sa.String(150), nullable=True),
+        sa.Column("started_at", TS, nullable=True),
+        sa.Column("finished_at", TS, nullable=True),
+        sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    op.create_index("ix_rollout_run_waves_run_id", "rollout_run_waves", ["run_id"])
+    op.create_table(
+        "rollout_run_hosts",
+        sa.Column("id", UUID, nullable=False),
+        sa.Column("run_id", UUID, nullable=False),
+        sa.Column("wave_index", sa.Integer(), nullable=False),
+        sa.Column("host_id", UUID, nullable=True),
+        sa.Column("host_name", sa.String(120), nullable=False),
+        sa.Column("cluster_name", sa.String(120), nullable=True),
+        sa.Column("order", sa.Integer(), nullable=False),
+        sa.Column("status", sa.String(16), nullable=False),
+        sa.Column("scheduled_maintenance_id", UUID, nullable=True),
+        sa.Column("version_before", sa.String(40), nullable=True),
+        sa.Column("version_after", sa.String(40), nullable=True),
+        sa.Column("detail", sa.JSON(), nullable=True),
+        sa.Column("started_at", TS, nullable=True),
+        sa.Column("finished_at", TS, nullable=True),
+        sa.ForeignKeyConstraint(["run_id"], ["rollout_runs.id"], ondelete="CASCADE"),
+        sa.ForeignKeyConstraint(["host_id"], ["opnsense_hosts.id"], ondelete="SET NULL"),
+        sa.ForeignKeyConstraint(["scheduled_maintenance_id"], ["scheduled_maintenance.id"], ondelete="SET NULL"),
+        sa.PrimaryKeyConstraint("id"),
+        sa.UniqueConstraint("run_id", "host_id", name="uq_rollout_run_host"),
+    )
+    op.create_index("ix_rollout_run_hosts_run_id", "rollout_run_hosts", ["run_id"])
+    op.create_index("ix_rollout_run_hosts_host_id", "rollout_run_hosts", ["host_id"])
+    op.create_index("ix_rollout_run_hosts_scheduled_maintenance_id", "rollout_run_hosts", ["scheduled_maintenance_id"])
+
+
+def downgrade() -> None:
+    for table in ("rollout_run_hosts", "rollout_run_waves", "rollout_runs", "rollout_wave_groups", "rollout_plan_waves", "rollout_plans"):
+        op.execute(f"DROP TABLE IF EXISTS {table}")
+    op.execute("ALTER TABLE notification_settings DROP COLUMN IF EXISTS rollout_problems_enabled")

backend/requirements.txt

@@ -13,3 +13,4 @@ respx==0.23.1
 cryptography==50.0.2
 alembic==1.20.0
 PyJWT==2.10.1
+tzdata==2025.2

backend/tests/test_maintenance_windows.py

@@ -0,0 +1,203 @@
+"""Recurring maintenance windows: occurrence maths (weekly, monthly, DST, exceptions, cut-off)."""
+from datetime import date, datetime, time, timedelta, timezone
+
+import pytest
+
+from app.models.maintenance_window import MaintenanceWindow, MaintenanceWindowException
+from app.services import window_service as ws
+
+UTC = timezone.utc
+
+
+def _w(**kw) -> MaintenanceWindow:
+    base = dict(name="w", timezone="Europe/Berlin", recurrence="weekly", weekdays=[6], start_time=time(2, 0),
+                duration_minutes=180, start_cutoff_minutes=60, enabled=True, created_by="admin")
+    base.update(kw)
+    w = MaintenanceWindow(**base)
+    w.exceptions = kw.get("exceptions", [])
+    return w
+
+
+def _starts(w, after, n=4, **kw):
+    return [o.start for o in ws.occurrences(w, after, n, **kw)]
+
+
+def test_weekly_sunday_in_summer_and_winter_keeps_local_time():
+    w = _w()
+    # 2026-10-18 is a Sunday (CEST, +2), 2026-11-01 a Sunday (CET, +1)
+    starts = _starts(w, datetime(2026, 10, 12, tzinfo=UTC), 4)
+    assert starts[0] == datetime(2026, 10, 18, 0, 0, tzinfo=UTC)
+    assert starts[2] == datetime(2026, 11, 1, 1, 0, tzinfo=UTC)
+    assert all(s.astimezone(ws.ZoneInfo("Europe/Berlin")).hour == 2 for s in starts)
+
+
+def test_weekly_several_days_in_order():
+    w = _w(weekdays=[1, 3], timezone="UTC", start_time=time(22, 0), duration_minutes=60)
+    starts = _starts(w, datetime(2026, 10, 19, tzinfo=UTC), 3)  # Monday
+    assert starts == [datetime(2026, 10, 20, 22, tzinfo=UTC), datetime(2026, 10, 22, 22, tzinfo=UTC),
+                      datetime(2026, 10, 27, 22, tzinfo=UTC)]
+
+
+def test_monthly_nth_weekday_first_and_last():
+    first_tue = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=1, month_weekday=1, timezone="UTC")
+    assert [s.date() for s in _starts(first_tue, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
+        date(2026, 10, 6), date(2026, 11, 3), date(2026, 12, 1)]
+    last_sun = _w(recurrence="monthly", weekdays=[], month_mode="nth_weekday", month_nth=-1, month_weekday=6, timezone="UTC")
+    assert [s.date() for s in _starts(last_sun, datetime(2026, 10, 1, tzinfo=UTC), 3)] == [
+        date(2026, 10, 25), date(2026, 11, 29), date(2026, 12, 27)]
+
+
+def test_monthly_fixed_day():
+    w = _w(recurrence="monthly", weekdays=[], month_mode="day", month_day=15, timezone="UTC")
+    assert [s.date() for s in _starts(w, datetime(2026, 10, 16, tzinfo=UTC), 2)] == [date(2026, 11, 15), date(2026, 12, 15)]
+
+
+def test_spring_forward_missing_time_moves_forward():
+    # 2026-03-29 (last Sunday of March): 02:00-03:00 local does not exist in Europe/Berlin.
+    w = _w(start_time=time(2, 30))
+    occ = ws.occurrences(w, datetime(2026, 3, 28, tzinfo=UTC), 1)[0]
+    assert occ.local_date == date(2026, 3, 29)
+    assert occ.start == datetime(2026, 3, 29, 1, 30, tzinfo=UTC)  # = 03:30 CEST
+    assert occ.start.astimezone(ws.ZoneInfo("Europe/Berlin")).strftime("%H:%M") == "03:30"
+
+
+def test_fall_back_ambiguous_time_uses_the_first_and_duration_is_real_time():
+    # 2026-10-25 (last Sunday of October): 02:00-03:00 local happens twice.
+    w = _w(start_time=time(2, 30), duration_minutes=120)
+    occ = ws.occurrences(w, datetime(2026, 10, 24, tzinfo=UTC), 1)[0]
+    assert occ.start == datetime(2026, 10, 25, 0, 30, tzinfo=UTC)  # 02:30 CEST (first pass)
+    assert occ.end - occ.start == timedelta(hours=2)
+
+
+def test_skip_and_move_exceptions():
+    moved = datetime(2026, 10, 31, 20, 0, tzinfo=UTC)  # Saturday evening instead of Sunday night
+    w = _w(exceptions=[
+        MaintenanceWindowException(occurrence_date=date(2026, 10, 25), kind="skip", reason="x", actor="a"),
+        MaintenanceWindowException(occurrence_date=date(2026, 11, 1), kind="move", moved_start=moved, reason="x", actor="a"),
+    ])
+    occs = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3)
+    assert [o.local_date for o in occs] == [date(2026, 11, 1), date(2026, 11, 8), date(2026, 11, 15)]
+    assert occs[0].moved and occs[0].start == moved and occs[0].end == moved + timedelta(minutes=180)
+    listed = ws.occurrences(w, datetime(2026, 10, 19, tzinfo=UTC), 3, with_skipped=True)
+    assert [(o.local_date, o.skipped, o.moved) for o in listed] == [
+        (date(2026, 10, 25), True, False), (date(2026, 11, 1), False, True), (date(2026, 11, 8), False, False)]
+
+
+def test_open_window_is_current_and_cutoff_limits_new_starts():
+    w = _w(timezone="UTC", weekdays=[6], start_time=time(2, 0), duration_minutes=180, start_cutoff_minutes=60)
+    start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
+    assert ws.current(w, start - timedelta(seconds=1)) is None
+    assert ws.current(w, start).start == start
+    assert ws.startable(w, start + timedelta(minutes=59)) is not None
+    assert ws.startable(w, start + timedelta(minutes=60)) is None  # past the cut-off: nothing new starts
+    assert ws.current(w, start + timedelta(minutes=179)) is not None  # still open, running work may finish
+    assert ws.current(w, start + timedelta(minutes=180)) is None
+    # an occurrence that is open right now is the first one listed
+    assert ws.occurrences(w, start + timedelta(minutes=30), 1)[0].start == start
+
+
+def test_cutoff_never_exceeds_duration_and_disabled_window_is_never_open():
+    w = _w(timezone="UTC", duration_minutes=30, start_cutoff_minutes=90)
+    start = datetime(2026, 10, 18, 2, 0, tzinfo=UTC)
+    assert ws.occurrences(w, start, 1)[0].cutoff == start + timedelta(minutes=30)
+    w.enabled = False
+    assert ws.current(w, start) is None and ws.startable(w, start) is None
+
+
+@pytest.mark.parametrize("name,ok", [("Europe/Berlin", True), ("UTC", True), ("Mars/Olympus", False), ("", False), ("../etc", False)])
+def test_timezone_validation(name, ok):
+    assert ws.valid_timezone(name) is ok
+
+
+# ── API ─────────────────────────────────────────────────────────────────────
+
+from sqlalchemy import select  # noqa: E402
+
+from app.database import SessionLocal  # noqa: E402
+from app.models.audit import AuditEvent  # noqa: E402
+
+_WEEKLY = {"name": "Sonntagnacht", "timezone": "Europe/Berlin", "recurrence": "weekly", "weekdays": [6],
+           "start_time": "02:00", "duration_minutes": 180, "start_cutoff_minutes": 60}
+
+
+@pytest.mark.asyncio
+async def test_window_crud_and_next_occurrences(client):
+    resp = await client.post("/api/maintenance-windows", json=_WEEKLY)
+    assert resp.status_code == 201, resp.text
+    w = resp.json()
+    assert w["start_time"] == "02:00" and len(w["next"]) == 5 and w["exceptions"] == [] and w["used_by"] == []
+    assert (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()["detail"]["code"] == "WINDOW_NAME_TAKEN"
+    upd = await client.put(f"/api/maintenance-windows/{w['id']}", json={**_WEEKLY, "start_time": "03:15", "start_cutoff_minutes": 999})
+    assert upd.status_code == 200 and upd.json()["start_time"] == "03:15"
+    assert upd.json()["start_cutoff_minutes"] == 180  # never longer than the window itself
+    occ = await client.get(f"/api/maintenance-windows/{w['id']}/occurrences?count=12")
+    assert occ.status_code == 200 and len(occ.json()) == 12
+    assert (await client.delete(f"/api/maintenance-windows/{w['id']}")).status_code == 204
+    assert (await client.get(f"/api/maintenance-windows/{w['id']}")).status_code == 404
+
+
+@pytest.mark.parametrize("change,field", [
+    ({"timezone": "Mars/Base"}, "timezone"),
+    ({"start_time": "25:00"}, "start_time"),
+    ({"weekdays": []}, "weekdays"),
+    ({"weekdays": [7]}, "weekdays"),
+    ({"recurrence": "monthly", "month_mode": "day", "month_day": 31}, "month_day"),
+    ({"recurrence": "monthly", "month_mode": "nth_weekday", "month_nth": 5, "month_weekday": 1}, "month_nth"),
+    ({"recurrence": "monthly"}, "month_mode"),
+])
+@pytest.mark.asyncio
+async def test_window_validation(client, change, field):
+    resp = await client.post("/api/maintenance-windows", json={**_WEEKLY, **change})
+    assert resp.status_code == 422 and resp.json()["detail"] == {"code": "WINDOW_INVALID", "params": {"field": field}}
+
+
+@pytest.mark.asyncio
+async def test_exceptions_skip_move_and_audit(client):
+    w = (await client.post("/api/maintenance-windows", json={**_WEEKLY, "timezone": "UTC"})).json()
+    first = date.fromisoformat(w["next"][0]["local_date"])
+    second = date.fromisoformat(w["next"][1]["local_date"])
+    bad = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first + timedelta(days=1)).isoformat(), "kind": "skip"})
+    assert bad.status_code == 422 and bad.json()["detail"]["params"]["field"] == "occurrence_date"
+    past = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": (first - timedelta(days=14)).isoformat(), "kind": "skip"})
+    assert past.status_code == 422
+    skip = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip", "reason": "Feiertag"})
+    assert skip.status_code == 201, skip.text
+    assert skip.json()["next"][0]["skipped"] is True
+    dup = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": first.isoformat(), "kind": "skip"})
+    assert dup.status_code == 409 and dup.json()["detail"]["code"] == "WINDOW_EXCEPTION_EXISTS"
+    no_target = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move"})
+    assert no_target.status_code == 422 and no_target.json()["detail"]["params"]["field"] == "moved_start"
+    target = datetime.combine(second, time(20, 0), tzinfo=UTC) - timedelta(days=1)
+    moved = await client.post(f"/api/maintenance-windows/{w['id']}/exceptions", json={"occurrence_date": second.isoformat(), "kind": "move", "moved_start": target.isoformat()})
+    assert moved.status_code == 201
+    regular = [o for o in moved.json()["next"] if not o["skipped"]]
+    assert regular[0]["moved"] is True and datetime.fromisoformat(regular[0]["start"]) == target
+    eid = moved.json()["exceptions"][1]["id"]
+    after = await client.delete(f"/api/maintenance-windows/{w['id']}/exceptions/{eid}")
+    assert after.status_code == 200 and len(after.json()["exceptions"]) == 1
+    async with SessionLocal() as db:
+        events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "maintenance_window"))).all()
+    assert any(e.action == "exception_skip" and e.details["reason"] == "Feiertag" for e in events)
+    assert any(e.status == "accepted" and e.action == "delete" for e in events)
+
+
+@pytest.mark.asyncio
+async def test_window_writes_follow_the_deploy_right(client):
+    from tests.test_operator_role import _operator
+    op = await _operator(client, "winop")
+    try:
+        made = await op.post("/api/maintenance-windows", json={**_WEEKLY, "name": "op-window"})
+        assert made.status_code == 201  # operators may prepare windows while they may deploy
+        await client.put("/api/settings/governance", json={"operators_may_deploy": False})
+        for method, path, body in [
+            ("POST", "/api/maintenance-windows", {**_WEEKLY, "name": "x"}),
+            ("PUT", f"/api/maintenance-windows/{made.json()['id']}", _WEEKLY),
+            ("DELETE", f"/api/maintenance-windows/{made.json()['id']}", None),
+            ("POST", f"/api/maintenance-windows/{made.json()['id']}/exceptions", {"occurrence_date": "2030-01-06", "kind": "skip"}),
+        ]:
+            resp = await op.request(method, path, json=body)
+            assert resp.status_code == 403 and resp.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN", path
+        assert (await op.get("/api/maintenance-windows")).status_code == 200
+    finally:
+        await op.aclose()
+        await client.put("/api/settings/governance", json={"operators_may_deploy": True})

backend/tests/test_rollout_api.py

@@ -0,0 +1,147 @@
+"""Rollout plans: CRUD, wave resolution, preview token, activation, roles."""
+import pytest
+
+from tests.test_maintenance_windows import _WEEKLY
+from tests.test_rules import _create_host
+
+
+async def _fleet(client):
+    """canary: c1 · pilot: p1 + ha-b (BACKUP) · prod: ha-a; ha-a/ha-b form cluster "ha"."""
+    hosts = {}
+    for name in ("c1", "p1", "ha-a", "ha-b"):
+        hosts[name] = await _create_host(client, name, f"https://{name}.test")
+    for name in ("ha-a", "ha-b"):
+        await client.patch(f"/api/hosts/{hosts[name]['id']}", json={"cluster_name": "ha"})
+    groups = {}
+    for gname, members in (("ring-canary", ["c1"]), ("ring-pilot", ["p1", "ha-b"]), ("ring-prod", ["ha-a", "c1"])):
+        resp = await client.post("/api/groups", json={"name": gname, "host_ids": [hosts[m]["id"] for m in members]})
+        assert resp.status_code == 201, resp.text
+        groups[gname] = resp.json()["id"]
+    return hosts, groups
+
+
+def _plan_body(groups, **kw):
+    body = {"name": "Ringe", "action": "update", "waves": [
+        {"label": "canary", "group_ids": [groups["ring-canary"]], "soak_minutes": 10},
+        {"label": "pilot", "group_ids": [groups["ring-pilot"]], "soak_minutes": 10},
+        {"label": "prod", "group_ids": [groups["ring-prod"]], "soak_minutes": 0, "carp_handover": True, "require_approval": True},
+    ]}
+    body.update(kw)
+    return body
+
+
+@pytest.mark.asyncio
+async def test_plan_crud_and_preview_resolves_rings(client):
+    _hosts, groups = await _fleet(client)
+    resp = await client.post("/api/rollouts/plans", json=_plan_body(groups))
+    assert resp.status_code == 201, resp.text
+    plan = resp.json()
+    assert [w["label"] for w in plan["waves"]] == ["canary", "pilot", "prod"] and plan["host_count"] == 4
+    prev = await client.post(f"/api/rollouts/plans/{plan['id']}/preview")
+    assert prev.status_code == 200
+    body = prev.json()
+    assert [[h["name"] for h in w["hosts"]] for w in body["waves"]] == [["c1"], ["ha-b", "p1"], ["ha-a"]]  # c1 only once
+    codes = {w["code"] for w in body["warnings"]}
+    assert {"duplicate_hosts", "mixed_versions", "master_needs_handover", "no_window"} <= codes
+    assert body["errors"] == [] and body["can_start"] and len(body["preview_token"]) == 64
+    assert (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()["detail"]["code"] == "ROLLOUT_NAME_TAKEN"
+    assert (await client.delete(f"/api/groups/{groups['ring-canary']}")).json()["detail"]["code"] == "ROLLOUT_GROUP_IN_USE"
+    assert (await client.delete(f"/api/rollouts/plans/{plan['id']}")).status_code == 204
+    assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_major_update_must_keep_a_cluster_in_one_wave(client):
+    _hosts, groups = await _fleet(client)
+    plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, action="major_update"))).json()
+    assert {e["code"] for e in plan["errors"]} == {"cluster_split"}
+    prev = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()
+    assert prev["can_start"] is False
+
+
+@pytest.mark.asyncio
+async def test_activation_needs_auto_start_a_window_and_the_current_token(client):
+    _hosts, groups = await _fleet(client)
+    window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
+    plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups))).json()
+    token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
+    assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == "auto_start"
+
+    plan = (await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"]))).json()
+    stale = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
+    assert stale.status_code == 409 and stale.json()["detail"]["code"] == "ROLLOUT_PREVIEW_STALE"
+    token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    active = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
+    assert active.status_code == 200 and active.json()["active"] and active.json()["fingerprint_ok"]
+    assert active.json()["activated_by"] == "admin"
+
+    # A group change behind the plan's back: still "active", but the fingerprint no longer matches.
+    await client.put(f"/api/groups/{groups['ring-canary']}", json={"name": "ring-canary", "host_ids": []})
+    assert (await client.get(f"/api/rollouts/plans/{plan['id']}")).json()["fingerprint_ok"] is False
+    # A window that a plan uses cannot be deleted (the plan would silently lose its time limit).
+    assert (await client.delete(f"/api/maintenance-windows/{window['id']}")).json()["detail"]["code"] == "WINDOW_IN_USE"
+
+    # Any edit disarms it.
+    token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
+    edited = await client.put(f"/api/rollouts/plans/{plan['id']}", json=_plan_body(groups, auto_start=True, window_id=window["id"], name="Ringe 2"))
+    assert edited.json()["active"] is False
+    off = await client.post(f"/api/rollouts/plans/{plan['id']}/deactivate")
+    assert off.status_code == 200 and off.json()["active"] is False
+
+
+@pytest.mark.asyncio
+async def test_auto_start_only_for_minor_updates_with_a_window(client):
+    _hosts, groups = await _fleet(client)
+    plan = (await client.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True))).json()
+    assert "auto_start_needs_window" in {e["code"] for e in plan["errors"]}
+    token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    resp = await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})
+    assert resp.status_code == 422 and resp.json()["detail"]["code"] == "ROLLOUT_PLAN_INVALID"
+
+
+@pytest.mark.parametrize("change,field", [
+    ({"gate_checks": {"nonsense": True}}, "gate_checks"),
+    ({"window_id": "00000000-0000-0000-0000-000000000000"}, "window_id"),
+])
+@pytest.mark.asyncio
+async def test_plan_validation(client, change, field):
+    _hosts, groups = await _fleet(client)
+    resp = await client.post("/api/rollouts/plans", json=_plan_body(groups, **change))
+    assert resp.status_code == 422 and resp.json()["detail"]["params"]["field"] == field
+    bad_group = _plan_body(groups)
+    bad_group["waves"][0]["group_ids"] = ["00000000-0000-0000-0000-000000000000"]
+    assert (await client.post("/api/rollouts/plans", json=bad_group)).json()["detail"]["params"]["field"] == "group_ids"
+    too_many = _plan_body(groups, waves=[{"label": str(i), "group_ids": [groups["ring-canary"]]} for i in range(11)])
+    assert (await client.post("/api/rollouts/plans", json=too_many)).status_code == 422
+
+
+@pytest.mark.asyncio
+async def test_role_matrix_for_plans(client):
+    from tests.test_operator_role import _operator
+    _hosts, groups = await _fleet(client)
+    window = (await client.post("/api/maintenance-windows", json=_WEEKLY)).json()
+    op = await _operator(client, "rollop")
+    try:
+        made = await op.post("/api/rollouts/plans", json=_plan_body(groups, auto_start=True, window_id=window["id"]))
+        assert made.status_code == 201  # preparing is fine
+        pid = made.json()["id"]
+        token = (await op.post(f"/api/rollouts/plans/{pid}/preview")).json()["preview_token"]
+        await client.put("/api/settings/governance", json={"operators_may_deploy": False})
+        denied = await op.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})
+        assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
+        assert (await op.post(f"/api/rollouts/plans/{pid}/deactivate")).status_code == 200  # braking is always allowed
+        assert (await client.post(f"/api/rollouts/plans/{pid}/activate", json={"preview_token": token})).status_code == 200
+    finally:
+        await op.aclose()
+        await client.put("/api/settings/governance", json={"operators_may_deploy": True})
+    from tests.test_api_tokens import _login
+    assert (await client.post("/api/users", json={"username": "rollro", "password": "readonly-pass-1", "role": "read_only"})).status_code == 201
+    ro = await _login("rollro", "readonly-pass-1")
+    try:
+        assert (await ro.get("/api/rollouts/plans")).status_code == 200
+        for path in ("/api/rollouts/plans", f"/api/rollouts/plans/{pid}/deactivate"):
+            assert (await ro.post(path, json={})).json()["detail"]["code"] == "READ_ONLY_ACCESS"
+    finally:
+        await ro.aclose()

backend/tests/test_rollout_engine.py

@@ -0,0 +1,620 @@
+"""The rollout engine end to end against a fake fleet: c1 (stand-alone), ha-a (MASTER) + ha-b (BACKUP)
+in cluster "ha". A fake clock drives timeouts, soak and re-checks; every tick runs exactly what the
+poll loop runs (fire due schedules, step running ones, step the rollout)."""
+import asyncio
+import uuid
+from datetime import datetime, timedelta, timezone
+
+import httpx
+import pytest
+from sqlalchemy import select
+
+from app.database import SessionLocal
+from app.models.audit import AuditEvent
+from app.models.host_operation import HostOperation
+from app.models.rollout import RolloutPlan, RolloutRun
+from app.models.scheduled_maintenance import ScheduledMaintenance
+from app.services import rollout_engine as engine
+from app.services import scheduled_maintenance_service as svc
+from tests.test_rules import _create_host
+
+
+class Fw:
+    def __init__(self, fleet, name, *, role=None, version="26.7.6", latest="26.7.7"):
+        self.fleet, self.name, self.base = fleet, name, f"https://{name}.test"
+        self.role, self.maint = role, False
+        self.version, self.latest, self.offer = version, latest, "update"
+        self.job, self.job_ticks, self.boot, self.up = "ready", 0, "boot-1", True
+        self.update_error = False
+        self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
+                         {"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
+        self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1 ms", "stddev": "0.1 ms"}]
+        r, b = fleet.mock, self.base
+        self.update = r.post(f"{b}/api/core/firmware/update").mock(side_effect=self._update)
+        self.check = r.post(f"{b}/api/core/firmware/check").mock(side_effect=lambda q: self._ok({"status": "ok"}))
+        self.toggles = r.post(f"{b}/api/diagnostics/interface/carp_status/maintenance").mock(side_effect=self._toggle)
+        r.get(f"{b}/api/core/backup/download/this").mock(side_effect=lambda q: self._raw(b"<opnsense/>"))
+        r.get(f"{b}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
+        r.get(f"{b}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
+        r.get(f"{b}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
+        r.get(f"{b}/api/core/firmware/status").mock(side_effect=self._status)
+        r.get(f"{b}/api/diagnostics/system/systemTime").mock(side_effect=lambda q: self._ok({"boottime": self.boot}))
+        r.get(f"{b}/api/routes/gateway/status").mock(side_effect=lambda q: self._ok({"items": self.gateways, "status": "ok"}))
+        r.get(f"{b}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
+
+    @property
+    def down(self):
+        return self.job == "down" or not self.up
+
+    def _ok(self, data):
+        if self.down:
+            raise httpx.ConnectError("down")
+        return httpx.Response(200, json=data)
+
+    def _raw(self, content):
+        if self.down:
+            raise httpx.ConnectError("down")
+        return httpx.Response(200, content=content)
+
+    def _status(self, request):
+        if self.offer == "update":
+            return self._ok({"status": "update", "status_msg": "x", "status_reboot": "1", "all_packages": {},
+                             "product": {"product_latest": self.latest}})
+        return self._ok({"status": self.offer, "status_msg": "x", "status_reboot": "0", "all_packages": {}, "product": {}})
+
+    def _update(self, request):
+        if self.update_error:
+            return httpx.Response(500, json={"status": "error"})
+        self.job, self.job_ticks = "running", 2
+        return self._ok({"status": "ok"})
+
+    def _vip(self, request):
+        rows = [{"vhid": "1", "status": self.role}] if self.role else []
+        return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "240" if self.maint else "0",
+                                                "status_msg": ""}})
+
+    def _toggle(self, request):
+        self.maint = not self.maint
+        partner = self.fleet.partner(self)
+        if self.maint:
+            self.role, partner.role = "BACKUP", "MASTER"
+        else:
+            self.role, partner.role = "MASTER", "BACKUP"  # preempt: takes MASTER back
+        return self._ok({"status": "ok", "action": "enter_maintenance" if self.maint else "leave_maintenance"})
+
+    def progress(self):
+        if self.job == "running":
+            self.job_ticks -= 1
+            if self.job_ticks <= 0:
+                self.job, self.job_ticks = "down", 2
+        elif self.job == "down":
+            self.job_ticks -= 1
+            if self.job_ticks <= 0:
+                self.job, self.version, self.offer, self.boot = "ready", self.latest, "none", self.boot + "+"
+
+
+class Fleet:
+    def __init__(self, respx_mock):
+        self.mock = respx_mock
+        self.fw = {"c1": Fw(self, "c1"), "ha-a": Fw(self, "ha-a", role="MASTER"), "ha-b": Fw(self, "ha-b", role="BACKUP")}
+
+    def partner(self, fw):
+        return self.fw["ha-b"] if fw.name == "ha-a" else self.fw["ha-a"]
+
+    def progress(self):
+        for fw in self.fw.values():
+            fw.progress()
+
+
+class Clock:
+    def __init__(self):
+        self.now = datetime.now(timezone.utc)
+
+    def __call__(self):
+        return self.now
+
+
+@pytest.fixture
+def clock(monkeypatch):
+    c = Clock()
+    monkeypatch.setattr(engine, "_now", c)
+    monkeypatch.setattr(svc, "_now", c)
+    return c
+
+
+async def _setup(client, respx_mock, *, waves=None, **plan_kw):
+    fleet = Fleet(respx_mock)
+    hosts = {n: await _create_host(client, n, f"https://{n}.test") for n in ("c1", "ha-a", "ha-b")}
+    for n in ("ha-a", "ha-b"):
+        await client.patch(f"/api/hosts/{hosts[n]['id']}", json={"cluster_name": "ha"})
+    groups = {}
+    for g, members in (("canary", ["c1"]), ("pilot", ["ha-b"]), ("prod", ["ha-a"]), ("cluster", ["ha-a", "ha-b"])):
+        groups[g] = (await client.post("/api/groups", json={"name": g, "host_ids": [hosts[m]["id"] for m in members]})).json()["id"]
+    waves = waves or [
+        {"label": "canary", "group_ids": [groups["canary"]], "soak_minutes": 5},
+        {"label": "pilot", "group_ids": [groups["pilot"]], "soak_minutes": 5},
+        {"label": "prod", "group_ids": [groups["prod"]], "soak_minutes": 0, "carp_handover": True},
+    ]
+    waves = [{**w, "group_ids": [groups[g] if g in groups else g for g in w["group_ids"]]} for w in waves]
+    plan = (await client.post("/api/rollouts/plans", json={"name": "ringe", "action": "update", "waves": waves, **plan_kw})).json()
+    return fleet, hosts, groups, plan
+
+
+async def _start(client, plan, api=None):
+    api = api or client
+    token = (await api.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    resp = await api.post(f"/api/rollouts/plans/{plan['id']}/start",
+                          json={"preview_token": token, "confirm_name": plan["name"], "acknowledge_unattended": True})
+    assert resp.status_code == 201, resp.text
+    return resp.json()
+
+
+async def _tick(fleet, clock, seconds=30):
+    fleet.progress()
+    async with SessionLocal() as db:
+        await svc.fire_due(db)
+    async with SessionLocal() as db:
+        await svc.advance_running(db)
+    async with SessionLocal() as db:
+        await engine.advance_runs(db)
+    clock.now += timedelta(seconds=seconds)
+
+
+async def _run(run_id):
+    async with SessionLocal() as db:
+        return await db.scalar(select(RolloutRun).where(RolloutRun.id == uuid.UUID(str(run_id))).execution_options(populate_existing=True))
+
+
+async def _until(fleet, clock, run_id, pred, max_ticks=400, check=None):
+    for _ in range(max_ticks):
+        run = await _run(run_id)
+        if pred(run):
+            return run
+        if check:
+            await check()
+        await _tick(fleet, clock)
+    raise AssertionError(f"not reached: status={run.status} reason={run.pause_reason} detail={run.pause_detail}")
+
+
+def _host(run, name):
+    return next(h for h in run.hosts if h.host_name == name)
+
+
+async def _scheds():
+    async with SessionLocal() as db:
+        return list((await db.scalars(select(ScheduledMaintenance).order_by(ScheduledMaintenance.created_at))).all())
+
+
+async def _no_two_at_once():
+    active = [s for s in await _scheds() if s.status in ("pending", "running")]
+    assert len(active) <= 1, [(s.host_name, s.status, s.phase) for s in active]
+
+
+# ── happy path ────────────────────────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_three_rings_run_through_and_the_master_hands_over(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
+    assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
+    assert done.target_version == "26.7.7"
+    assert [(h.host_name, h.status, h.version_before, h.version_after) for h in sorted(done.hosts, key=lambda h: h.wave_index)] == [
+        ("c1", "succeeded", "26.7.6", "26.7.7"), ("ha-b", "succeeded", "26.7.6", "26.7.7"), ("ha-a", "succeeded", "26.7.6", "26.7.7")]
+    assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
+    # The MASTER went through the CARP handover and is MASTER again; the BACKUP never needed one.
+    assert fleet.fw["ha-a"].toggles.call_count == 2 and fleet.fw["ha-b"].toggles.call_count == 0
+    assert (fleet.fw["ha-a"].role, fleet.fw["ha-b"].role) == ("MASTER", "BACKUP")
+    scheds = await _scheds()
+    assert [s.host_name for s in scheds] == ["c1", "ha-b", "ha-a"]
+    assert all(s.origin == "rollout" and s.auto_verify and s.status == "succeeded" for s in scheds)
+    async with SessionLocal() as db:
+        ops = (await db.scalars(select(HostOperation))).all()
+        events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "rollout"))).all()
+    assert {o.status for o in ops} == {"acknowledged"}  # nothing left locked
+    assert sum(1 for e in events if e.action == "gate" and e.status == "ok") == 3
+    assert any(e.action == "finish" and e.status == "succeeded" for e in events)
+    host_events = [e for e in (await _audit_host()) if (e.details or {}).get("rollout")]
+    assert host_events and all(e.details["rollout"]["run_id"] == run["id"] for e in host_events)
+
+
+async def _audit_host():
+    async with SessionLocal() as db:
+        return (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
+
+
+# ── the gate stops the rollout ─────────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_gate_failure_in_the_canary_ring_stops_everything_after_it(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
+    fleet.fw["c1"].services[1]["running"] = 0  # ntpd did not come back
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert paused.pause_reason == "gate_failed" and paused.paused_from == "gate"
+    assert paused.waves[0].gate_attempts == 2  # looked at twice (slow start), then stopped
+    assert paused.waves[0].gate_result["hosts"][_host(paused, "c1").host_id.__str__()]["services"]["code"] == "service_stopped"
+    for _ in range(20):  # time passes, nothing moves
+        await _tick(fleet, clock)
+    assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
+    assert [s.host_name for s in await _scheds()] == ["c1"]
+    # Notification while paused.
+    from app.services import notification_checks
+    async with SessionLocal() as db:
+        problems = await notification_checks.check_rollout_paused(db)
+    assert problems[run["id"]]["grund"] == "gate_failed"
+
+    # Fixed by hand, resumed: the gate is checked again first, then it goes on.
+    fleet.fw["c1"].services[1]["running"] = 1
+    resp = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
+    assert resp.status_code == 200 and resp.json()["status"] == "resuming"
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
+    assert fleet.fw["ha-a"].update.call_count == 1
+
+
+@pytest.mark.asyncio
+async def test_resume_rechecks_and_pauses_again_if_still_broken(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
+    fleet.fw["c1"].gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100 %", "delay": "~", "stddev": "~"}
+    await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
+    again = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert again.pause_reason == "gate_failed" and again.pause_detail.get("on_resume") is True
+    assert fleet.fw["ha-b"].update.call_count == 0
+
+
+# ── never twice, never in parallel ─────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_restart_in_the_middle_of_job_wait_submits_exactly_once(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+
+    async def in_job_wait(r):
+        return any(s.host_name == "ha-b" and s.phase == "job_wait" for s in await _scheds())
+    for _ in range(300):
+        if await in_job_wait(None):
+            break
+        await _tick(fleet, clock)
+    # "Restart": nothing survives but the database (each tick opens fresh sessions anyway); two backends
+    # even step the same run at the very same moment.
+    await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(3)))
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded"
+    assert fleet.fw["ha-b"].update.call_count == 1 and len([s for s in await _scheds() if s.host_name == "ha-b"]) == 1
+
+
+@pytest.mark.asyncio
+async def test_two_backends_deciding_at_once_create_one_schedule(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    await _until(fleet, clock, run["id"], lambda r: r.status == "wave_running")
+    await asyncio.gather(*(engine.tick(uuid.UUID(run["id"])) for _ in range(4)))
+    assert len(await _scheds()) == 1
+
+
+@pytest.mark.asyncio
+async def test_a_cluster_in_one_wave_goes_backup_first_and_never_both_at_once(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
+        {"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
+    run = await _start(client, plan)
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
+    assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
+    assert [s.host_name for s in await _scheds()] == ["ha-b", "ha-a"]
+
+
+@pytest.mark.asyncio
+async def test_master_without_handover_pauses_before_anything_happens_to_it(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
+        {"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": False}])
+    run = await _start(client, plan)
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert paused.pause_reason == "master_needs_handover" and paused.pause_detail["host"] == "ha-a"
+    assert fleet.fw["ha-b"].update.call_count == 1 and fleet.fw["ha-a"].update.call_count == 0
+    assert fleet.fw["ha-a"].toggles.call_count == 0
+
+
+@pytest.mark.asyncio
+async def test_cluster_gate_before_the_second_node(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock, waves=[
+        {"label": "cluster", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True}])
+    run = await _start(client, plan)
+
+    async def break_backup_after_update():
+        if fleet.fw["ha-b"].version == "26.7.7":
+            fleet.fw["ha-b"].services[0]["running"] = 0  # unbound dead on the freshly updated node
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused", check=break_backup_after_update)
+    assert paused.pause_reason == "cluster_gate_failed" and paused.pause_detail["host"] == "ha-a"
+    assert fleet.fw["ha-a"].update.call_count == 0 and fleet.fw["ha-a"].toggles.call_count == 0
+
+
+# ── stop when in doubt ─────────────────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_a_different_target_version_in_a_later_ring_pauses(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    fleet.fw["ha-b"].latest = "26.7.8"  # a newer release appeared after the canary ran
+    run = await _start(client, plan)
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert paused.pause_reason == "target_version_changed"
+    assert paused.pause_detail == {"expected": "26.7.7", "offered": "26.7.8"}
+    assert fleet.fw["ha-b"].update.call_count == 0
+
+
+@pytest.mark.asyncio
+async def test_hosts_without_an_update_are_skipped(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    fleet.fw["ha-b"].offer = "none"
+    run = await _start(client, plan)
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded" and _host(done, "ha-b").status == "skipped"
+    assert _host(done, "ha-b").detail["code"] == "no_update_offered" and done.waves[1].status == "skipped"
+
+
+@pytest.mark.asyncio
+async def test_failed_host_pauses_and_resume_needs_an_explicit_skip(client, respx_mock, clock):
+    fleet, hosts, _groups, plan = await _setup(client, respx_mock)
+    fleet.fw["c1"].update_error = True
+    run = await _start(client, plan)
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert paused.pause_reason == "host_failed" and _host(paused, "c1").status == "failed"
+    refused = await client.post(f"/api/rollouts/runs/{run['id']}/resume")
+    assert refused.status_code == 409 and refused.json()["detail"]["code"] == "ROLLOUT_HOST_FAILED"
+    skip = await client.post(f"/api/rollouts/runs/{run['id']}/skip-host", json={"host_id": hosts["c1"]["id"], "reason": "geprüft, später"})
+    assert skip.status_code == 200
+    # The failed operation stays "unknown" (locks c1, notifies) until a human closes it; it is not retried.
+    assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
+    assert fleet.fw["c1"].update.call_count == 1 and _host(done, "c1").detail["code"] == "skipped_by"
+
+
+@pytest.mark.asyncio
+async def test_unreachable_host_in_preflight_is_blocked_then_paused(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    fleet.fw["c1"].up = False
+    run = await _start(client, plan)
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert paused.pause_reason == "preflight_blocked" and _host(paused, "c1").status == "blocked"
+    assert clock.now - datetime.fromisoformat(paused.progress["paused_at"]) < timedelta(minutes=1)
+    fleet.fw["c1"].up = True
+    assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 200
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded"
+
+
+@pytest.mark.asyncio
+async def test_someone_elses_open_maintenance_blocks_then_pauses(client, respx_mock, clock):
+    fleet, hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    await _until(fleet, clock, run["id"], lambda r: r.current_wave == 1 and r.status == "wave_running")
+    async with SessionLocal() as db:
+        db.add(HostOperation(host_id=uuid.UUID(hosts["ha-a"]["id"]), host_name="ha-a", actor="someone", action="reboot",
+                             status="submitted", expires_at=clock.now, fingerprint="", snapshot={}))
+        await db.commit()
+    paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+    assert paused.pause_reason in ("blocked_by_maintenance", "cluster_gate_failed")
+    assert fleet.fw["ha-b"].update.call_count == 0
+
+
+@pytest.mark.asyncio
+async def test_actor_who_lost_the_right_pauses_the_run(client, respx_mock, clock):
+    from tests.test_operator_role import _operator
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    op = await _operator(client, "rollengine-op")
+    try:
+        run = await _start(client, plan, api=op)
+        assert run["started_by"] == "rollengine-op"
+        await _until(fleet, clock, run["id"], lambda r: r.status == "soaking")
+        await client.put("/api/settings/governance", json={"operators_may_deploy": False})
+        paused = await _until(fleet, clock, run["id"], lambda r: r.status == "paused")
+        assert paused.pause_reason == "actor_invalid" and fleet.fw["ha-b"].update.call_count == 0
+        # An admin takes over by resuming.
+        assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).json()["started_by"] == "admin"
+        done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+        assert done.status == "succeeded"
+    finally:
+        await op.aclose()
+        await client.put("/api/settings/governance", json={"operators_may_deploy": True})
+
+
+# ── windows, approval, skip, abort, automatic start ───────────────────────────
+
+async def _window_now(client, clock, *, cutoff=60, duration=120):
+    local = clock.now
+    body = {"name": f"jetzt-{uuid.uuid4().hex[:6]}", "timezone": "UTC", "recurrence": "weekly", "weekdays": [local.weekday()],
+            "start_time": (local - timedelta(minutes=1)).strftime("%H:%M"), "duration_minutes": duration, "start_cutoff_minutes": cutoff}
+    if (local - timedelta(minutes=1)).date() != local.date():
+        pytest.skip("too close to midnight for a same-day window")
+    resp = await client.post("/api/maintenance-windows", json=body)
+    assert resp.status_code == 201, resp.text
+    return resp.json()
+
+
+@pytest.mark.asyncio
+async def test_closed_window_waits_and_the_next_occurrence_resumes(client, respx_mock, clock):
+    window = await _window_now(client, clock, cutoff=15)
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"])
+    run = await _start(client, plan)
+    waiting = await _until(fleet, clock, run["id"], lambda r: r.status in ("waiting_window", "paused", "succeeded"))
+    assert waiting.status == "waiting_window" and waiting.current_wave >= 1
+    started = {s.host_name for s in await _scheds()}
+    assert "c1" in started and "ha-a" not in started
+    clock.now += timedelta(days=7) - timedelta(minutes=40)  # one week later, a few minutes into the next occurrence
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
+
+
+@pytest.mark.asyncio
+async def test_approval_holds_the_wave(client, respx_mock, clock):
+    fleet, _hosts, groups, plan = await _setup(client, respx_mock, waves=[
+        {"label": "canary", "group_ids": ["canary"], "soak_minutes": 1},
+        {"label": "rest", "group_ids": ["cluster"], "soak_minutes": 1, "carp_handover": True, "require_approval": True}])
+    run = await _start(client, plan)
+    held = await _until(fleet, clock, run["id"], lambda r: r.status == "awaiting_approval")
+    for _ in range(10):
+        await _tick(fleet, clock)
+    assert fleet.fw["ha-b"].update.call_count == 0
+    resp = await client.post(f"/api/rollouts/runs/{run['id']}/approve")
+    assert resp.status_code == 200 and resp.json()["waves"][1]["approved_by"] == "admin"
+    done = await _until(fleet, clock, run["id"], lambda r: r.status in ("succeeded", "paused"))
+    assert done.status == "succeeded" and held.current_wave == 1
+
+
+@pytest.mark.asyncio
+async def test_abort_cancels_what_has_not_started_and_lets_the_running_host_finish(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    await _until(fleet, clock, run["id"], lambda r: any(h.status == "running" for h in r.hosts))
+    bad = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "falsch"})
+    assert bad.status_code == 409 and bad.json()["detail"]["code"] == "ROLLOUT_CONFIRM_NAME"
+    resp = await client.post(f"/api/rollouts/runs/{run['id']}/abort", json={"reason": "Notfall", "confirm_name": "ringe"})
+    assert resp.status_code == 200 and resp.json()["status"] == "cancelled"
+    for _ in range(40):
+        await _tick(fleet, clock)
+    run_now = await _run(run["id"])
+    assert run_now.status == "cancelled"
+    assert {h.host_name: h.status for h in run_now.hosts} in (
+        {"c1": "succeeded", "ha-b": "cancelled", "ha-a": "cancelled"}, {"c1": "cancelled", "ha-b": "cancelled", "ha-a": "cancelled"})
+    assert fleet.fw["ha-b"].update.call_count == 0 and fleet.fw["ha-a"].update.call_count == 0
+    assert (await client.post(f"/api/rollouts/runs/{run['id']}/resume")).status_code == 409
+
+
+@pytest.mark.asyncio
+async def test_pause_by_hand_and_operator_may_always_brake(client, respx_mock, clock):
+    from tests.test_operator_role import _operator
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    await client.put("/api/settings/governance", json={"operators_may_deploy": False})
+    op = await _operator(client, "brake-op")
+    try:
+        resp = await op.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "Bauchgefühl"})
+        assert resp.status_code == 200 and resp.json()["pause_reason"] == "manual"
+        denied = await op.post(f"/api/rollouts/runs/{run['id']}/resume")
+        assert denied.status_code == 403 and denied.json()["detail"]["code"] == "DEPLOY_NEEDS_ADMIN"
+    finally:
+        await op.aclose()
+        await client.put("/api/settings/governance", json={"operators_may_deploy": True})
+    for _ in range(10):
+        await _tick(fleet, clock)
+    assert sum(fw.update.call_count for fw in fleet.fw.values()) == 0
+    second = await client.post(f"/api/rollouts/plans/{plan['id']}/start", json={
+        "preview_token": (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"],
+        "confirm_name": "ringe", "acknowledge_unattended": True})
+    assert second.status_code == 409 and second.json()["detail"]["code"] == "ROLLOUT_ALREADY_ACTIVE"
+
+
+async def _until_step_pending(fleet, clock, run_id):
+    """Right after the engine created a host's schedule (fire_due has not seen it yet)."""
+    run = await _until(fleet, clock, run_id, lambda r: any(h.status == "running" for h in r.hosts))
+    name = next(h.host_name for h in run.hosts if h.status == "running")
+    assert [s.status for s in await _scheds()] == ["pending"]
+    return name
+
+
+async def _resume_to_the_end(client, fleet, clock, run_id):
+    assert (await client.post(f"/api/rollouts/runs/{run_id}/resume")).status_code == 200
+    done = await _until(fleet, clock, run_id, lambda r: r.status in ("succeeded", "paused"), check=_no_two_at_once)
+    assert done.status == "succeeded", (done.pause_reason, done.pause_detail)
+    assert all(fw.update.call_count == 1 for fw in fleet.fw.values())
+
+
+@pytest.mark.asyncio
+async def test_pause_while_a_step_is_pending_really_brakes_and_resume_runs_it_once(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    name = await _until_step_pending(fleet, clock, run["id"])
+    resp = await client.post(f"/api/rollouts/runs/{run['id']}/pause", json={"reason": "halt"})
+    assert resp.status_code == 200
+    for _ in range(10):
+        await _tick(fleet, clock)
+    assert fleet.fw[name].update.call_count == 0  # the pending step did not fire behind the pause
+    paused = await _run(run["id"])
+    assert paused.status == "paused" and _host(paused, name).status == "queued"
+    assert _host(paused, name).scheduled_maintenance_id is None
+    assert [s.status for s in await _scheds()] == ["cancelled"]
+    await _resume_to_the_end(client, fleet, clock, run["id"])
+
+
+@pytest.mark.asyncio
+async def test_a_step_found_long_overdue_after_an_outage_is_not_fired(client, respx_mock, clock):
+    fleet, _hosts, _groups, plan = await _setup(client, respx_mock)
+    run = await _start(client, plan)
+    name = await _until_step_pending(fleet, clock, run["id"])
+    clock.now += timedelta(minutes=30)  # OPNexus was down (e.g. a web update); the window may be over by now
+    await _tick(fleet, clock)
+    assert fleet.fw[name].update.call_count == 0
+    paused = await _run(run["id"])
+    assert (paused.status, paused.pause_reason) == ("paused", "host_not_started")
+    assert _host(paused, name).status == "queued" and _host(paused, name).detail["code"] == "not_fired"
+    (sched,) = await _scheds()
+    assert sched.status == "failed" and sched.result["not_fired"]
+    await _resume_to_the_end(client, fleet, clock, run["id"])
+
+
+@pytest.mark.asyncio
+async def test_automatic_start_once_per_occurrence_and_only_while_unchanged(client, respx_mock, clock):
+    window = await _window_now(client, clock)
+    fleet, _hosts, groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
+    token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
+    for _ in range(3):
+        async with SessionLocal() as db:
+            await engine.auto_start(db)
+    async with SessionLocal() as db:
+        runs = (await db.scalars(select(RolloutRun))).all()
+    assert len(runs) == 1 and runs[0].trigger == "window" and runs[0].started_by == "admin"
+    assert runs[0].window_occurrence is not None
+
+    # Next week, but somebody added a host to the canary group meanwhile: no start, a notification instead.
+    async with SessionLocal() as db:
+        (await db.get(RolloutRun, runs[0].id)).status = "cancelled"
+        await db.commit()
+    await client.put(f"/api/groups/{groups['canary']}", json={"name": "canary", "host_ids": [_hosts["c1"]["id"], _hosts["ha-a"]["id"]]})
+    clock.now += timedelta(days=7)
+    async with SessionLocal() as db:
+        await engine.auto_start(db)
+    async with SessionLocal() as db:
+        assert len((await db.scalars(select(RolloutRun))).all()) == 1
+        stored = await db.get(RolloutPlan, uuid.UUID(plan["id"]))
+    assert stored.auto_start_error["code"] == "plan_changed"
+    from app.services import notification_checks
+    async with SessionLocal() as db:
+        problems = await notification_checks.check_rollout_start_failed(db)
+    assert list(problems.values())[0]["grund"] == "plan_changed"
+
+
+@pytest.mark.asyncio
+async def test_a_second_backend_waiting_for_the_lock_reports_no_refusal(client, respx_mock, clock, monkeypatch):
+    window = await _window_now(client, clock)
+    _fleet, _hosts, _groups, plan = await _setup(client, respx_mock, window_id=window["id"], auto_start=True)
+    token = (await client.post(f"/api/rollouts/plans/{plan['id']}/preview")).json()["preview_token"]
+    assert (await client.post(f"/api/rollouts/plans/{plan['id']}/activate", json={"preview_token": token})).status_code == 200
+    real_lock = engine.policy_lock
+
+    async def racing_lock(db):
+        monkeypatch.setattr(engine, "policy_lock", real_lock)
+        async with SessionLocal() as other:  # backend A starts the run while B waits for the lock
+            assert await engine.auto_start(other) == 1
+        await real_lock(db)
+    monkeypatch.setattr(engine, "policy_lock", racing_lock)
+    async with SessionLocal() as db:
+        assert await engine.auto_start(db) == 0
+    async with SessionLocal() as db:
+        assert len((await db.scalars(select(RolloutRun))).all()) == 1
+        assert (await db.get(RolloutPlan, uuid.UUID(plan["id"]))).auto_start_error is None
+
+
+def test_target_version_comes_from_the_core_package_not_product_latest():
+    # Shape seen live on test-fw-3 (26.7.1_1): product_latest lagged behind what would really be installed.
+    status = {"status": "update", "product": {"product_latest": "26.7.2"},
+              "all_packages": {"opnsense": {"old": "26.7.1_1", "new": "26.7.7", "reason": "upgrade"}},
+              "upgrade_packages": [{"name": "opnsense", "current_version": "26.7.1_1", "new_version": "26.7.7"}]}
+    assert engine.target_of(status, "update") == "26.7.7"
+    assert engine.target_of({**status, "all_packages": {}}, "update") == "26.7.7"
+    assert engine.target_of({"product": {"product_latest": "26.7.7"}}, "update") == "26.7.7"
+    assert engine.target_of({"upgrade_major_version": "27.1"}, "major_update") == "27.1"

backend/tests/test_rollout_health.py

@@ -0,0 +1,195 @@
+"""Health gate checks of the rollout engine, one by one, against a fake firewall pair."""
+import uuid
+
+import httpx
+import pytest
+
+from app.database import SessionLocal
+from app.models.carp_maintenance_window import CarpMaintenanceWindow
+from app.models.compliance_result import ComplianceResult
+from app.models.host import OpnsenseHost
+from app.models.host_operation import HostOperation
+from app.services import rollout_health as rh
+from tests.test_rules import _create_host
+
+A, B = "https://gw-a.test", "https://gw-b.test"
+
+
+class Fw:
+    def __init__(self, respx_mock, base, *, role="MASTER", version="26.7.7"):
+        self.base, self.role, self.version, self.up, self.job, self.maint = base, role, version, True, "ready", False
+        self.vips = None
+        self.gateways = [{"name": "WAN_GW", "status": "none", "loss": "0.0 %", "delay": "1.0 ms", "stddev": "0.1 ms"}]
+        self.services = [{"id": "unbound", "name": "unbound", "locked": 0, "running": 1},
+                         {"id": "ntpd", "name": "ntpd", "locked": 0, "running": 1}]
+        self.gw_error = False
+        r = respx_mock
+        r.get(f"{base}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
+        r.get(f"{base}/api/core/firmware/running").mock(side_effect=lambda q: self._ok({"status": self.job}))
+        r.get(f"{base}/api/core/firmware/info").mock(side_effect=lambda q: self._ok({"product_version": self.version}))
+        r.get(f"{base}/api/routes/gateway/status").mock(side_effect=self._gw)
+        r.get(f"{base}/api/core/service/search").mock(side_effect=lambda q: self._ok({"rows": self.services}))
+
+    def _ok(self, data):
+        if not self.up:
+            raise httpx.ConnectError("down")
+        return httpx.Response(200, json=data)
+
+    def _gw(self, request):
+        if self.gw_error:
+            return httpx.Response(403, json={"message": "forbidden"})
+        return self._ok({"items": self.gateways, "status": "ok"})
+
+    def _vip(self, request):
+        rows = [{"vhid": v, "status": s} for v, s in (self.vips or {"1": self.role}).items()]
+        return self._ok({"rows": rows, "carp": {"allow": "1", "maintenancemode": self.maint, "demotion": "0", "status_msg": ""}})
+
+
+async def _hosts(client, cluster=True):
+    a = await _create_host(client, "gw-a", A)
+    if not cluster:
+        return a, None
+    b = await _create_host(client, "gw-b", B)
+    if cluster:
+        for h in (a, b):
+            await client.patch(f"/api/hosts/{h['id']}", json={"cluster_name": "ha"})
+    return a, b
+
+
+async def _eval(host_ids, baselines=None, target="26.7.7", **kw):
+    async with SessionLocal() as db:
+        hosts = [await db.get(OpnsenseHost, uuid.UUID(h)) for h in host_ids]
+        if baselines is None:
+            baselines = {str(h.id): await rh.baseline(db, h) for h in hosts}
+        return await rh.evaluate(db, hosts, baselines, target_version=target, **kw), baselines
+
+
+@pytest.mark.asyncio
+async def test_healthy_cluster_passes_every_check(client, respx_mock):
+    Fw(respx_mock, A)
+    Fw(respx_mock, B, role="BACKUP")
+    a, _ = await _hosts(client)
+    gate, base = await _eval([a["id"]])
+    assert gate["verdict"] == "ok", gate
+    row = gate["hosts"][a["id"]]
+    assert {k: v["result"] for k, v in row.items()} == {
+        "online": "ok", "version": "ok", "carp": "ok", "gateways": "ok", "services": "ok", "locks": "ok"}
+    assert base[a["id"]]["gateways"] == {"WAN_GW": "online"} and base[a["id"]]["services"]["ntpd"] == 1
+
+
+@pytest.mark.asyncio
+async def test_offline_or_busy_host_and_wrong_version_fail(client, respx_mock):
+    fa = Fw(respx_mock, A)
+    a, _ = await _hosts(client, cluster=False)
+    fa.version = "26.7.6"
+    gate, base = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["version"] == {"result": "fail", "code": "version_mismatch",
+                                                 "params": {"expected": "26.7.7", "actual": "26.7.6"}}
+    fa.job = "busy"
+    gate, _ = await _eval([a["id"]], base)
+    assert gate["hosts"][a["id"]]["online"]["code"] == "firmware_busy"
+    fa.up = False
+    gate, _ = await _eval([a["id"]], base)
+    assert gate["hosts"][a["id"]]["online"]["code"] == "offline" and gate["verdict"] == "fail"
+    assert gate["hosts"][a["id"]]["gateways"]["result"] == "fail"  # readable before, not now
+    assert rh.failed_checks(gate)[0].startswith(a["id"])
+
+
+@pytest.mark.asyncio
+async def test_carp_split_brain_maintenance_mode_and_timer_fail(client, respx_mock):
+    fa = Fw(respx_mock, A)
+    fb = Fw(respx_mock, B, role="BACKUP")
+    a, b = await _hosts(client)
+    fb.role = "MASTER"  # both MASTER on VHID 1
+    gate, _ = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "vhid_roles", "params": {"vhids": "1"}}
+    fb.role = "BACKUP"
+    fa.vips, fb.vips = {"1": "MASTER", "2": "MASTER"}, {"1": "BACKUP", "2": "MASTER"}  # partial failover on VHID 2
+    gate, _ = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["carp"]["params"] == {"vhids": "2"}
+    fa.vips = fb.vips = None
+    fb.maint = True
+    gate, _ = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["carp"]["code"] == "maintenance_mode"
+    fb.maint = False
+    async with SessionLocal() as db:
+        db.add(CarpMaintenanceWindow(host_id=uuid.UUID(b["id"]), host_name="gw-b", status="active"))
+        await db.commit()
+    gate, _ = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["carp"]["code"] == "auto_revert_active"
+    fb.up = False
+    gate, _ = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["carp"] == {"result": "fail", "code": "node_offline", "params": {"names": "gw-b"}}
+
+
+@pytest.mark.asyncio
+async def test_gateways_compare_against_the_baseline(client, respx_mock):
+    fa = Fw(respx_mock, A)
+    fa.gateways.append({"name": "OLD_GW", "status": "down", "loss": "100.0 %", "delay": "0.0 ms", "stddev": "0.0 ms"})
+    a, _ = await _hosts(client, cluster=False)
+    _gate, base = await _eval([a["id"]])
+    assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["code"] == "gateway_bad_before"  # only warns
+    fa.gateways[0] = {"name": "WAN_GW", "status": "down", "loss": "100.0 %", "delay": "~", "stddev": "~"}
+    gate, _ = await _eval([a["id"]], base)
+    assert gate["hosts"][a["id"]]["gateways"] == {"result": "fail", "code": "gateway_down", "params": {"names": "WAN_GW"}}
+    fa.gateways[0] = {"name": "WAN_GW", "status": "loss", "loss": "30 %", "delay": "5 ms", "stddev": "1 ms"}
+    assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "warn"
+    fa.gw_error = True
+    assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["gateways"]["result"] == "fail"
+    gate, _ = await _eval([a["id"]], {a["id"]: {**base[a["id"]], "gateways": None}})
+    assert gate["hosts"][a["id"]]["gateways"]["result"] == "skip"  # no permission before either: nothing to compare
+
+
+@pytest.mark.asyncio
+async def test_services_that_ran_before_must_run_again(client, respx_mock):
+    fa = Fw(respx_mock, A)
+    fa.services.append({"id": "dhcpd6", "name": "dhcpd6", "locked": 0, "running": 0})
+    a, _ = await _hosts(client, cluster=False)
+    _gate, base = await _eval([a["id"]])
+    fa.services[1] = {"id": "ntpd", "name": "ntpd", "locked": 0, "running": 0}
+    gate, _ = await _eval([a["id"]], base)
+    assert gate["hosts"][a["id"]]["services"] == {"result": "fail", "code": "service_stopped", "params": {"names": "ntpd"}}
+    fa.services[1]["running"] = 1
+    assert (await _eval([a["id"]], base))[0]["hosts"][a["id"]]["services"]["result"] == "ok"  # dhcpd6 never ran: fine
+
+
+@pytest.mark.asyncio
+async def test_open_operation_in_the_cluster_is_a_lock(client, respx_mock):
+    Fw(respx_mock, A)
+    Fw(respx_mock, B, role="BACKUP")
+    a, b = await _hosts(client)
+    from datetime import datetime, timezone
+    async with SessionLocal() as db:
+        db.add(HostOperation(host_id=uuid.UUID(b["id"]), host_name="gw-b", actor="x", action="reboot", status="submitted",
+                             expires_at=datetime.now(timezone.utc), fingerprint="", snapshot={}))
+        await db.commit()
+    gate, _ = await _eval([a["id"]])
+    assert gate["hosts"][a["id"]]["locks"] == {"result": "fail", "code": "open_locks", "params": {"count": 1}}
+
+
+@pytest.mark.parametrize("mode,expected", [("warn", "warn"), ("fail", "fail")])
+@pytest.mark.asyncio
+async def test_lower_compliance_score_warns_or_fails(client, respx_mock, monkeypatch, mode, expected):
+    Fw(respx_mock, A)
+    a, _ = await _hosts(client, cluster=False)
+    async with SessionLocal() as db:
+        db.add(ComplianceResult(host_id=uuid.UUID(a["id"]), checked_at=datetime_now(), score=80, findings=[]))
+        await db.commit()
+
+    async def fake_check(db, host):
+        row = await db.get(ComplianceResult, (await db.scalar(__import__("sqlalchemy").select(ComplianceResult.id))))
+        row.score = 70
+        await db.commit()
+        return row
+    monkeypatch.setattr(rh.compliance_service, "check_host", fake_check)
+    gate, _ = await _eval([a["id"]], checks={"compliance": True}, compliance_mode=mode)
+    assert gate["hosts"][a["id"]]["compliance"] == {"result": expected, "code": "score_lower", "params": {"before": 80, "after": 70}}
+    assert gate["verdict"] == expected
+    # the gate cannot be switched off where it matters
+    gate, _ = await _eval([a["id"]], checks={"online": False, "version": False, "locks": False})
+    assert {"online", "version", "locks"} <= set(gate["hosts"][a["id"]])
+
+
+def datetime_now():
+    from datetime import datetime, timezone
+    return datetime.now(timezone.utc)

backend/tests/test_scheduled_auto_verify.py

@@ -0,0 +1,306 @@
+"""auto_verify (1.25): an unattended action checks its own result and closes the operation,
+so the host/cluster is not left locked until a human clicks "Wartung abschließen"."""
+import asyncio
+import uuid
+from datetime import datetime, timedelta, timezone
+
+import httpx
+import pytest
+from sqlalchemy import select
+
+from app.database import SessionLocal
+from app.models import HostOperation
+from app.models.audit import AuditEvent
+from app.models.scheduled_maintenance import ScheduledMaintenance
+from app.services import scheduled_maintenance_service as svc
+from tests.test_rules import _create_host
+from tests.test_scheduled_handover import FakeCluster, _setup
+
+H = "https://fw-solo.test"
+
+
+class FakeHost:
+    """One stand-alone firewall: job = ready | running | down; boot = what systemTime reports."""
+
+    def __init__(self, respx_mock, *, firmware="update", reboot_flag="1", version="26.7.5"):
+        self.job, self.boot, self.version = "ready", "Mon Oct 5 10:00:00 UTC 2026", version
+        self.update = respx_mock.post(f"{H}/api/core/firmware/update").respond(200, json={"status": "ok"})
+        self.reboot = respx_mock.post(f"{H}/api/core/system/reboot").respond(200, json={"status": "ok"})
+        respx_mock.get(f"{H}/api/core/backup/download/this").respond(200, content=b"<opnsense/>")
+        respx_mock.get(f"{H}/api/diagnostics/interface/getVipStatus").mock(side_effect=self._vip)
+        respx_mock.get(f"{H}/api/core/firmware/info").mock(side_effect=lambda r: self._up(json={"product_version": self.version}))
+        respx_mock.get(f"{H}/api/core/firmware/status").respond(
+            200, json={"status": firmware, "all_packages": {}, "status_msg": "x", "status_reboot": reboot_flag})
+        respx_mock.get(f"{H}/api/core/firmware/running").mock(side_effect=lambda r: self._up(json={"status": self.job}))
+        respx_mock.get(f"{H}/api/diagnostics/system/systemTime").mock(side_effect=lambda r: self._up(json={"boottime": self.boot}))
+
+    def _up(self, **kw):
+        if self.job == "down":
+            raise httpx.ConnectError("rebooting")
+        return httpx.Response(200, **kw)
+
+    def _vip(self, request):
+        return self._up(json={"rows": [], "carp": {"allow": "1", "maintenancemode": False, "demotion": "0", "status_msg": ""}})
+
+    def reboot_now(self, new_boot="Wed Oct 21 03:00:00 UTC 2026", version=None):
+        self.boot = new_boot
+        if version:
+            self.version = version
+
+
+async def _plan(host, action="update", *, auto_verify=True, handover=False):
+    async with SessionLocal() as db:
+        s = ScheduledMaintenance(
+            host_id=uuid.UUID(host["id"]), host_name=host["name"], action=action, carp_handover=handover,
+            auto_verify=auto_verify, run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin",
+            created_at=datetime.now(timezone.utc))
+        db.add(s)
+        await db.commit()
+        return s.id
+
+
+async def _fire():
+    async with SessionLocal() as db:
+        return await svc.fire_due(db)
+
+
+async def _advance():
+    async with SessionLocal() as db:
+        return await svc.advance_running(db)
+
+
+async def _get(sid):
+    async with SessionLocal() as db:
+        s = await db.get(ScheduledMaintenance, sid)
+        op = await db.get(HostOperation, s.host_operation_id) if s.host_operation_id else None
+        return s, op
+
+
+async def _shift(sid, *, phase=None, ready=None):
+    """Pretend the phase began / the firmware turned ready that long ago."""
+    async with SessionLocal() as db:
+        s = await db.get(ScheduledMaintenance, sid)
+        if phase is not None:
+            s.phase_since = datetime.now(timezone.utc) - phase
+        if ready is not None:
+            s.progress = {**s.progress, "ready_since": (datetime.now(timezone.utc) - ready).isoformat()}
+        await db.commit()
+
+
+async def _settle(sid):
+    """verify needs the firmware idle for VERIFY_SETTLE: one tick to notice, then pretend it has been a while."""
+    await _advance()
+    await _shift(sid, ready=svc.VERIFY_SETTLE + timedelta(seconds=1))
+    await _advance()
+
+
+@pytest.mark.asyncio
+async def test_update_is_verified_and_the_operation_closed(client, respx_mock):
+    fw = FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host)
+    await _fire()
+    s, op = await _get(sid)
+    assert (s.status, s.phase, op.status, fw.update.call_count) == ("running", "job_wait", "submitted", 1)
+    assert s.progress["reboot_expected"] is True and s.progress["boot_before"] == fw.boot
+
+    fw.job = "running"
+    await _advance()
+    fw.job = "down"
+    await _advance()
+    fw.job = "ready"
+    fw.reboot_now(version="26.7.7")
+    await _advance()
+    assert (await _get(sid))[0].phase == "verify"
+    await _settle(sid)
+
+    s, op = await _get(sid)
+    assert s.status == "succeeded" and s.phase is None
+    assert s.result["auto_verified"] is True and s.result["version_after"] == "26.7.7" and s.result["version_before"] == "26.7.5"
+    assert op.status == "acknowledged" and "Automatisch geprüft" in op.message
+    assert fw.update.call_count == 1
+    # the host is free again: a new preview is not blocked by a forgotten operation
+    resp = await client.get(f"/api/hosts/{host['id']}/maintenance")
+    assert not any("frühere Wartungsaktion" in b for b in resp.json()["state"]["blocked"])
+    async with SessionLocal() as db:
+        events = (await db.scalars(select(AuditEvent).where(AuditEvent.kind == "host"))).all()
+    assert {e.actor for e in events if (e.details or {}).get("scheduled")} == {"admin (geplant)"}
+
+
+@pytest.mark.asyncio
+async def test_boot_time_proves_a_reboot_the_poll_never_saw(client, respx_mock):
+    fw = FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host)
+    await _fire()
+    await _shift(sid, phase=timedelta(0))
+    fw.reboot_now()  # rebooted between two polls: never seen offline
+    async with SessionLocal() as db:  # the job grace has passed
+        s = await db.get(ScheduledMaintenance, sid)
+        s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
+        await db.commit()
+    await _advance()
+    assert (await _get(sid))[0].phase == "verify"
+    await _settle(sid)
+    s, op = await _get(sid)
+    assert s.status == "succeeded" and s.result["rebooted"] is True and op.status == "acknowledged"
+
+
+@pytest.mark.asyncio
+async def test_expected_reboot_never_happened_ends_unknown_after_timeout(client, respx_mock):
+    FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host)
+    await _fire()
+    async with SessionLocal() as db:
+        s = await db.get(ScheduledMaintenance, sid)
+        s.progress = {**s.progress, "job_started_at": (datetime.now(timezone.utc) - svc.JOB_GRACE).isoformat()}
+        await db.commit()
+    await _advance()
+    await _settle(sid)
+    s, op = await _get(sid)
+    assert (s.status, s.phase, op.status) == ("running", "verify", "submitted")  # waiting, not done
+    assert "Neustart" in s.progress["waiting_for"]
+    await _shift(sid, phase=svc.VERIFY_TIMEOUT)
+    await _advance()
+    s, op = await _get(sid)
+    assert s.status == "failed" and op.status == "unknown" and "nicht bestanden" in s.result["error"]
+
+
+@pytest.mark.asyncio
+async def test_ready_must_hold_a_while_a_short_ready_before_the_reboot_does_not_count(client, respx_mock):
+    fw = FakeHost(respx_mock, reboot_flag="0")
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host)
+    await _fire()
+    fw.job = "running"
+    await _advance()
+    fw.job = "ready"
+    await _advance()  # job_wait -> verify
+    await _advance()  # ready noticed
+    s, _ = await _get(sid)
+    assert s.phase == "verify" and s.progress["ready_since"]
+    fw.job = "running"  # busy again (e.g. the reboot at the end of the update)
+    await _advance()
+    assert (await _get(sid))[0].progress["ready_since"] is None
+    fw.job = "ready"
+    await _settle(sid)
+    assert (await _get(sid))[0].status == "succeeded"
+
+
+@pytest.mark.asyncio
+async def test_without_auto_verify_nothing_changes(client, respx_mock):
+    fw = FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host, auto_verify=False)
+    await _fire()
+    s, op = await _get(sid)
+    assert (s.status, s.phase, op.status, fw.update.call_count) == ("succeeded", None, "submitted", 1)
+    assert await _advance() == 0
+
+
+@pytest.mark.asyncio
+async def test_human_closing_the_operation_stops_the_check(client, respx_mock):
+    FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host)
+    await _fire()
+    s, op = await _get(sid)
+    async with SessionLocal() as db:
+        (await db.get(HostOperation, op.id)).status = "cancelled"
+        await db.commit()
+    await _advance()
+    s, op = await _get(sid)
+    assert s.status == "failed" and "von Hand beendet" in s.result["error"] and op.status == "cancelled"
+
+
+@pytest.mark.asyncio
+async def test_handover_verifies_before_taking_the_traffic_back(client, respx_mock):
+    cluster = FakeCluster(respx_mock)
+    cluster.boot = "before"
+
+    def system_time(request):
+        if cluster.job == "down":
+            raise httpx.ConnectError("down")
+        return httpx.Response(200, json={"boottime": cluster.boot})
+    respx_mock.get("https://fw-a.test/api/diagnostics/system/systemTime").mock(side_effect=system_time)
+    a, _b = await _setup(client)
+    async with SessionLocal() as db:
+        s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True, auto_verify=True,
+                                 run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
+        db.add(s)
+        await db.commit()
+        sid = s.id
+    await _fire()
+    await _advance()  # partner MASTER -> job submitted
+    cluster.job = "down"
+    await _advance()
+    cluster.job = "ready"
+    cluster.boot = "after"
+    await _advance()
+    s, _ = await _get(sid)
+    assert s.phase == "verify" and cluster.toggles.call_count == 1  # still demoted while checking
+    await _settle(sid)
+    assert (await _get(sid))[0].phase == "leave" and cluster.toggles.call_count == 1
+    await _advance()  # leave
+    await _advance()  # leave_wait -> done
+    s, op = await _get(sid)
+    assert s.status == "succeeded" and s.result["handover"] is True and s.result["auto_verified"] is True
+    assert s.result["rebooted"] is True  # found live: the handover path dropped what verify had seen
+    assert op.status == "acknowledged" and cluster.toggles.call_count == 2 and cluster.update.call_count == 1
+
+
+@pytest.mark.asyncio
+async def test_a_due_row_locked_by_another_backend_is_not_fired_twice(client, respx_mock):
+    fw = FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    sid = await _plan(host)
+    async with SessionLocal() as other:  # another backend is firing this very row right now
+        await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
+        # Without the row lock this would try to update the locked row and hang: bounded, so it fails instead.
+        result = await asyncio.wait_for(_fire(), timeout=10)
+        assert result["due"] == 0 and fw.update.call_count == 0
+        await other.rollback()
+    await _fire()
+    assert fw.update.call_count == 1
+
+
+@pytest.mark.asyncio
+async def test_a_running_row_locked_by_another_backend_is_not_stepped_twice(client, respx_mock):
+    cluster = FakeCluster(respx_mock)
+    a, _b = await _setup(client)
+    async with SessionLocal() as db:
+        s = ScheduledMaintenance(host_id=uuid.UUID(a["id"]), host_name="fw-a", action="update", carp_handover=True,
+                                 run_at=datetime.now(timezone.utc) - timedelta(minutes=1), actor="admin", created_at=datetime.now(timezone.utc))
+        db.add(s)
+        await db.commit()
+        sid = s.id
+    await _fire()
+    async with SessionLocal() as other:
+        await other.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
+        assert await asyncio.wait_for(_advance(), timeout=10) == 1  # still counted as running, but not touched
+        assert cluster.update.call_count == 0
+        await other.rollback()
+    await _advance()
+    assert cluster.update.call_count == 1
+
+
+@pytest.mark.asyncio
+async def test_cancel_waits_for_a_firing_row_and_never_overwrites_running(client, respx_mock):
+    FakeHost(respx_mock)
+    host = await _create_host(client, "fw-solo", H)
+    async with SessionLocal() as db:
+        s = ScheduledMaintenance(host_id=uuid.UUID(host["id"]), host_name="fw-solo", action="update", auto_verify=True,
+                                 run_at=datetime.now(timezone.utc) + timedelta(hours=1), actor="admin", created_at=datetime.now(timezone.utc))
+        db.add(s)
+        await db.commit()
+        sid = s.id
+    async with SessionLocal() as firing:  # the poll loop has the row and is about to mark it running
+        row = await firing.scalar(select(ScheduledMaintenance).where(ScheduledMaintenance.id == sid).with_for_update())
+        cancel = asyncio.create_task(client.delete(f"/api/hosts/{host['id']}/maintenance/schedule/{sid}"))
+        await asyncio.sleep(0.5)
+        row.status = "running"
+        await firing.commit()
+    resp = await asyncio.wait_for(cancel, timeout=10)
+    assert resp.status_code == 409
+    assert (await _get(sid))[0].status == "running"

backend/tests/test_updater.py

@@ -47,6 +47,7 @@ async def test_status_reports_update_and_can_update(client, respx_mock):
     body = (await client.get("/api/system/updater")).json()
     assert body["available"] is True and body["can_update"] is True
     assert body["latest"] == _bump(minor=1) and body["major_change"] is False
+    assert body["active_rollouts"] == 0  # the UI warns before restarting OPNexus in the middle of a rollout
 
 
 @pytest.mark.asyncio

docs/HANDBUCH.md

@@ -822,7 +822,15 @@ frische Backup erneut. Eine blockierte Aktion wird als „fehlgeschlagen“ mark
 und **nicht** wiederholt. Sie steht im Status der geplanten Wartung und meldet sich
 zusätzlich per Benachrichtigung („Geplante Wartung nicht ausgeführt“, einzeln
 schaltbar), damit man es am nächsten Morgen nicht erst beim Nachsehen merkt.
-Wiederkehrende Zeitpläne gibt es nicht.
+Wiederkehrende Termine und ganze Flotten laufen über [Rollouts](#rollouts-in-wellen).
+
+**Ergebnis automatisch prüfen und abschließen** (im Dialog vorausgewählt): Nach dem Auftrag
+wartet OPNexus, bis der Firmware-Auftrag beendet ist, und prüft dann selbst: Firmware
+mindestens eine Minute am Stück bereit, ein erwarteter Neustart hat wirklich stattgefunden
+(Host offline gesehen oder neue Boot-Zeit), Version lesbar. Erst dann wird die Wartungsaktion
+als abgeschlossen markiert und sperrt Host und Cluster nicht mehr. Gelingt das nicht innerhalb
+von 20 Minuten, bleibt sie „unklar“ (gesperrt, mit Benachrichtigung). Ohne die Option bleibt
+sie wie bisher „übergeben“, bis jemand sie von Hand abschließt.
 
 In einem Cluster läuft unbeaufsichtigte Wartung standardmäßig **nur auf dem
 CARP-BACKUP**. Ist der Host beim Einplanen gerade MASTER, lehnt OPNexus das sofort mit
@@ -849,7 +857,72 @@ der Plan steht auf „fehlgeschlagen“, die Wartungsaktion auf „unklar“, un
 Benachrichtigung. Der CARP-Umschalter wird nie blind wiederholt. Beendest du die
 Wartungsaktion von Hand, stoppt der Ablauf und fasst nichts mehr an. Bei Erfolg schließt
 OPNexus die Wartungsaktion selbst ab. Sie läuft auch einen Neustart des OPNexus-Backends
-mitten im Ablauf weiter (der Zustand steht in der Datenbank).
+mitten im Ablauf weiter (der Zustand steht in der Datenbank). Mit „Ergebnis automatisch
+prüfen“ kommt die Prüfung **vor** dem Verlassen des Wartungsmodus: Der Host bekommt den
+Verkehr erst zurück, wenn er nachweislich fertig ist.
+
+### Rollouts in Wellen
+
+**System › Rollouts** verteilt ein Firmware-Update (stabiles Update oder Major-Upgrade) in
+Wellen über die Flotte. Die Wellen bestehen aus [Gerätegruppen](#gerätegruppen), die man
+als Ringe anlegt, zum Beispiel `ring-canary` (eine unwichtige Firewall), `ring-pilot`
+(einige) und `ring-prod` (der Rest). Steht ein Host in mehreren Wellen, gilt die früheste.
+
+**Ablauf je Welle:**
+
+1. **Vorprüfung:** Update-Suche auf den Hosts der Welle, dann Bestand aufnehmen (Version,
+   Gateways, laufende Dienste, Sicherheits-Score). Hosts ohne passendes Update werden
+   übersprungen. Die Version, die in der ersten Welle installiert wird, ist die
+   **Zielversion**; bietet ein Host einer späteren Welle eine andere an, hält der Rollout an
+   (OPNsense installiert immer die neueste, sonst bekäme „prod“ eine Version, die im
+   Canary-Ring nie lief).
+2. **Hosts nacheinander:** Jeder Host läuft als [geplante Wartung](#geplante-wartung) mit
+   automatischer Prüfung (Backup vor dem Update, Sperren, Audit). Immer nur ein Host,
+   immer nur ein Rollout gleichzeitig. Im Cluster zuerst der BACKUP, dann der MASTER;
+   der MASTER nur, wenn die Welle die CARP-Übergabe erlaubt, sonst hält der Rollout an.
+   Vor dem zweiten Knoten eines Clusters werden der Cluster (alle Knoten erreichbar,
+   genau ein MASTER je VHID, kein Wartungsmodus) und der schon aktualisierte Knoten
+   erneut geprüft.
+3. **Beobachtungszeit** (je Welle einstellbar), dann die **Gesundheitsprüfung** der
+   aktualisierten Hosts: erreichbar und Firmware bereit, Zielversion installiert, CARP
+   gesund, Gateways und Dienste, die vorher liefen, laufen wieder (vorher schon gestörte
+   nur als Hinweis), keine offenen Sperren, optional Sicherheits-Score nicht gesunken
+   (Hinweis oder Halt). Ist etwas rot, prüft OPNexus zwei Minuten später noch einmal
+   (langsamer Start nach dem Neustart) und hält dann an.
+4. **Freigabe** (optional je Welle): Die Welle beginnt erst, wenn jemand auf „Welle
+   freigeben“ klickt.
+
+**Angehalten** wird bei jeder Unklarheit, nie wird etwas von selbst wiederholt: Gate rot,
+Host fehlgeschlagen oder ungewöhnlich lange unterwegs, Host in der Vorprüfung nicht bereit,
+abweichende Zielversion, MASTER ohne Übergabe, eine fremde offene Wartung im Cluster länger
+als 15 Minuten, die verantwortliche Person darf nicht mehr ausrollen, das Fenster wurde
+gelöscht, der Lauf ist älter als 14 Tage. Es kommt die Benachrichtigung „Rollout
+angehalten“. **Fortsetzen** prüft zuerst alle bisher aktualisierten Hosts erneut; wer
+fortsetzt, ist ab dann verantwortlich. Ein fehlgeschlagener Host muss vorher ausdrücklich
+**übersprungen** werden (seine Wartungsaktion bleibt „unklar“, bis du sie auf der
+Host-Seite abschließt). **Abbrechen** verwirft alles, was noch nicht begonnen hat; eine
+Aktion, die schon auf einer Firewall läuft, wird geführt zu Ende gebracht. **Anhalten**
+darf jeder mit Schreibrecht, alles andere braucht das Recht zum Ausrollen.
+
+**Wartungsfenster** (Reiter „Wartungsfenster“): wöchentlich an bestimmten Tagen oder
+monatlich (erster…vierter oder letzter Wochentag, oder ein fester Tag 1–28), Beginn in einer
+Zeitzone, Dauer und Abschneidezeit: Nach den ersten N Minuten beginnt kein neuer Host mehr,
+laufende Updates laufen zu Ende, und der Rollout wartet auf den nächsten Termin (dann wird
+zuerst das Bisherige erneut geprüft). Sommerzeit: Eine Uhrzeit, die es an dem Tag nicht gibt,
+rückt vor (02:30 → 03:30); eine doppelte gilt beim ersten Mal. Einzelne Termine lassen sich
+auslassen oder verschieben. Ein Fenster, das ein Plan nutzt, lässt sich nicht löschen.
+
+**Automatischer Start:** Ein Plan mit Fenster und stabilem Update kann zu jedem Termin von
+selbst starten. Dazu muss ihn jemand mit Ausroll-Recht nach der Vorschau **aktivieren**.
+OPNexus merkt sich dabei einen Fingerabdruck von Plan, aufgelösten Hosts und Fenster; ändert
+sich davon etwas (auch die Mitglieder einer Gruppe), startet nichts, und es kommt „Rollout
+nicht automatisch gestartet“. Jede Bearbeitung des Plans deaktiviert ihn. So kann niemand
+die Ziele eines aktivierten Plans unbemerkt ändern.
+
+Das **Update Center** zeigt den laufenden Rollout als Banner; Hosts, die er noch bearbeitet,
+sind dort für manuelle Firmware-Aktionen gesperrt. Ein Neustart oder Update von OPNexus
+während eines Rollouts ist sicher (der Zustand steht in der Datenbank), hält ihn aber für
+diese Zeit an.
 
 ### Dienste und Systemauslastung
 
@@ -1425,8 +1498,11 @@ nach einem Neustart erhalten. Alternativ legt ihn OPNexus unter
   OPNsense, keine Ausführung durch OPNexus.
 - **Logs:** kein Live-Streaming (Polling im Sekundenbereich), die Freitextsuche ist
   eine einfache Teilstring-Suche ohne Index.
-- **Geplante Wartung:** nur einmalige Zeitpunkte, ohne Benachrichtigung bei
-  Fehlschlag.
+- **Geplante Wartung und Rollouts:** Rollouts kennen keine Zielversion „von außen“
+  (OPNsense installiert immer die neueste angebotene) und keinen automatischen Rückweg zur
+  alten Version. Major-Upgrade-Rollouts sind nur gegen eine nachgestellte Firewall getestet.
+  Die Gesundheitsprüfung sieht die Firewall aus Sicht der Verwaltung; ob der Nutzverkehr
+  durchgeht, prüft sie nicht.
 - **Dienste:** keine Auswahlliste, alle Dienste sind steuerbar.
 - **VPN:** Umfang je Verfahren siehe [VPN](#vpn-ipsec-openvpn-wireguard).
 - **Zertifikate:** kein Push auf mehrere Hosts, keine Verwaltung von DNS-Providern

frontend/app/rollouts/page.tsx

@@ -0,0 +1,361 @@
+"use client";
+
+import { MaintenanceWindowEditor } from "@/components/MaintenanceWindowEditor";
+import { RolloutPlanEditor } from "@/components/RolloutPlanEditor";
+import { RolloutStatusBanner } from "@/components/RolloutStatusBanner";
+import { Badge } from "@/components/ui/Badge";
+import { Button } from "@/components/ui/Button";
+import { Card } from "@/components/ui/Card";
+import { EmptyState } from "@/components/ui/EmptyState";
+import { Input } from "@/components/ui/Field";
+import { Modal } from "@/components/ui/Modal";
+import { Skeleton } from "@/components/ui/Skeleton";
+import { Tabs } from "@/components/ui/Tabs";
+import { useToast } from "@/components/ui/Toast";
+import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
+import { useLanguage } from "@/components/LanguageProvider";
+import { api, DeviceGroup, MaintenanceWindow, Occurrence, RolloutPlan, RolloutPreview, RolloutRunSummary } from "@/lib/api";
+import { codeText, RUN_TONE, windowSummary } from "@/lib/rollouts";
+import { useReadOnly } from "@/lib/useReadOnly";
+import { AlertTriangle, CalendarClock, ListOrdered, Pencil, Play, Plus, Power, PowerOff, Trash2, Waves } from "lucide-react";
+import Link from "next/link";
+import { useRouter } from "next/navigation";
+import { useEffect, useState } from "react";
+
+type Tab = "runs" | "plans" | "windows";
+
+export default function RolloutsPage() {
+  const { t, errorMessage, locale } = useLanguage();
+  const toast = useToast();
+  const router = useRouter();
+  const readOnly = useReadOnly();
+  const [tab, setTab] = useState<Tab>("runs");
+  const [runs, setRuns] = useState<RolloutRunSummary[] | null>(null);
+  const [plans, setPlans] = useState<RolloutPlan[] | null>(null);
+  const [windows, setWindows] = useState<MaintenanceWindow[] | null>(null);
+  const [groups, setGroups] = useState<DeviceGroup[]>([]);
+  const [error, setError] = useState("");
+  const [editPlan, setEditPlan] = useState<RolloutPlan | "new" | null>(null);
+  const [editWindow, setEditWindow] = useState<MaintenanceWindow | "new" | null>(null);
+  const [preview, setPreview] = useState<{ mode: "start" | "activate"; data: RolloutPreview } | null>(null);
+  const [move, setMove] = useState<{ window: MaintenanceWindow; occ: Occurrence; to: string } | null>(null);
+
+  useEffect(() => {
+    const wanted = new URLSearchParams(window.location.search).get("tab");
+    if (wanted === "plans" || wanted === "windows") setTab(wanted);
+  }, []);
+
+  async function load() {
+    try {
+      const [r, p, w, g] = await Promise.all([api.listRolloutRuns(), api.listRolloutPlans(), api.listMaintenanceWindows(), api.listGroups()]);
+      setRuns(r);
+      setPlans(p);
+      setWindows(w);
+      setGroups(g);
+      setError("");
+    } catch (e) {
+      setError(errorMessage(e, "rollouts.loadFailed"));
+    }
+  }
+  useEffect(() => {
+    load();
+    const timer = setInterval(() => { if (document.visibilityState === "visible") api.listRolloutRuns().then(setRuns).catch(() => {}); }, 15000);
+    return () => clearInterval(timer);
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+  }, []);
+
+  function changeTab(next: string) {
+    setTab(next as Tab);
+    window.history.replaceState(null, "", next === "runs" ? "/rollouts" : `/rollouts?tab=${next}`);
+  }
+
+  async function act(fn: () => Promise<unknown>, okKey: string) {
+    try {
+      await fn();
+      toast.show(t(okKey), "success");
+      await load();
+    } catch (e) {
+      toast.show(errorMessage(e, "rollouts.actionFailed"), "danger");
+    }
+  }
+
+  async function openPreview(plan: RolloutPlan, mode: "start" | "activate") {
+    try {
+      setPreview({ mode, data: await api.previewRolloutPlan(plan.id) });
+    } catch (e) {
+      toast.show(errorMessage(e, "rollouts.actionFailed"), "danger");
+    }
+  }
+
+  const when = (value: string | null | undefined) => (value ? new Date(value).toLocaleString(locale) : "");
+
+  return (
+    <div>
+      <div className="mb-4 flex flex-wrap items-center justify-between gap-2">
+        <h1 className="flex items-center gap-2 text-2xl font-semibold text-slate-900">
+          {t("rollouts.title")}
+          <InfoTip text={t("rollouts.intro")} size={18} />
+        </h1>
+        {!readOnly && tab === "plans" && <Button onClick={() => setEditPlan("new")}><Plus size={16} />{t("rollouts.plan.new")}</Button>}
+        {!readOnly && tab === "windows" && <Button onClick={() => setEditWindow("new")}><Plus size={16} />{t("mwindows.new")}</Button>}
+      </div>
+      <RolloutStatusBanner />
+      <Tabs className="mb-4" active={tab} onChange={changeTab} items={[
+        { key: "runs", label: t("rollouts.tab.runs") },
+        { key: "plans", label: t("rollouts.tab.plans") },
+        { key: "windows", label: t("rollouts.tab.windows") },
+      ]} />
+      {error && <p role="alert" className="mb-4 text-sm text-red-600">{error}</p>}
+
+      {tab === "runs" && (runs === null ? <Skeleton className="h-32" /> : runs.length === 0 ? (
+        <EmptyState icon={Waves} title={t("rollouts.noRuns")} description={t("rollouts.noRunsHint")} />
+      ) : (
+        <Card className="overflow-x-auto">
+          <table className="w-full text-left text-sm">
+            <thead className="border-b border-slate-200 bg-slate-50 text-xs uppercase tracking-wide text-slate-500">
+              <tr>
+                <th className="px-4 py-3 font-medium">{t("rollouts.col.plan")}</th>
+                <th className="px-4 py-3 font-medium">{t("rollouts.col.status")}</th>
+                <th className="px-4 py-3 font-medium">{t("rollouts.col.wave")}</th>
+                <th className="px-4 py-3 font-medium">{t("rollouts.col.target")}</th>
+                <th className="px-4 py-3 font-medium">{t("rollouts.col.hosts")}</th>
+                <th className="px-4 py-3 font-medium">{t("rollouts.col.started")}</th>
+                <th className="px-4 py-3 font-medium"><span className="sr-only">{t("common.actions")}</span></th>
+              </tr>
+            </thead>
+            <tbody>
+              {runs.map((r) => (
+                <tr key={r.id} className="border-b border-slate-100 last:border-0">
+                  <td className="px-4 py-3 font-medium text-slate-900">{r.plan_name}<div className="text-xs font-normal text-slate-500">{t(`rollouts.action.${r.action}`)}</div></td>
+                  <td className="px-4 py-3"><Badge tone={RUN_TONE[r.status]}>{t(`rollouts.run.status.${r.status}`)}</Badge></td>
+                  <td className="px-4 py-3 whitespace-nowrap">{t("rollouts.waveOf", { current: Math.min(r.current_wave + 1, r.wave_count), total: r.wave_count })} {r.current_wave_label && <span className="text-slate-500">({r.current_wave_label})</span>}</td>
+                  <td className="px-4 py-3">{r.target_version ?? "–"}</td>
+                  <td className="px-4 py-3 whitespace-nowrap text-xs">
+                    {Object.entries(r.host_counts).map(([s, n]) => <span key={s} className="mr-2">{t(`rollouts.host.status.${s}`)}: {n}</span>)}
+                  </td>
+                  <td className="px-4 py-3 text-xs text-slate-600">{when(r.created_at)}<div>{t("rollouts.by", { user: r.started_by })} · {t(`rollouts.trigger.${r.trigger}`)}</div></td>
+                  <td className="px-4 py-3 text-right"><Button variant="outline" href={`/rollouts/runs/${r.id}`}>{t("rollouts.details")}</Button></td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        </Card>
+      ))}
+
+      {tab === "plans" && (plans === null ? <Skeleton className="h-32" /> : plans.length === 0 ? (
+        <EmptyState icon={ListOrdered} title={t("rollouts.plan.none")} description={t("rollouts.plan.noneHint")}
+          action={!readOnly && <Button onClick={() => setEditPlan("new")}><Plus size={16} />{t("rollouts.plan.new")}</Button>} />
+      ) : (
+        <div className="space-y-3">
+          {plans.map((p) => (
+            <Card key={p.id} className="p-4">
+              <div className="flex flex-wrap items-start justify-between gap-3">
+                <div className="min-w-0">
+                  <h2 className="flex flex-wrap items-center gap-2 font-semibold text-slate-900">
+                    {p.name}
+                    <Badge tone={p.action === "major_update" ? "accent" : "info"}>{t(`rollouts.action.${p.action}`)}</Badge>
+                    {p.auto_start && (p.active ? (p.fingerprint_ok
+                      ? <Badge tone="success">{t("rollouts.plan.active")}</Badge>
+                      : <Tooltip focusable content={t("rollouts.plan.changedTip")}><Badge tone="warning">{t("rollouts.plan.changed")}</Badge></Tooltip>)
+                      : <Badge tone="neutral">{t("rollouts.plan.inactive")}</Badge>)}
+                    {p.errors.length > 0 && <Tooltip focusable content={<span className="block space-y-1">{p.errors.map((e, i) => <span key={i} className="block">{codeText(t, "rollouts.issue", e)}</span>)}</span>}><Badge tone="danger">{t("rollouts.plan.errors", { count: p.errors.length })}</Badge></Tooltip>}
+                    {p.warnings.length > 0 && <Tooltip focusable content={<span className="block space-y-1">{p.warnings.map((e, i) => <span key={i} className="block">{codeText(t, "rollouts.issue", e)}</span>)}</span>}><Badge tone="warning">{t("rollouts.plan.warnings", { count: p.warnings.length })}</Badge></Tooltip>}
+                  </h2>
+                  {p.description && <p className="mt-1 text-sm text-slate-600">{p.description}</p>}
+                  <ol className="mt-2 flex flex-wrap items-center gap-1 text-sm">
+                    {p.waves.map((w, i) => (
+                      <li key={w.id} className="flex items-center gap-1">
+                        {i > 0 && <span aria-hidden className="text-slate-400">→</span>}
+                        <span className="rounded-lg border border-slate-200 px-2 py-0.5">
+                          <strong>{w.label}</strong> <span className="text-slate-500">{w.groups.join(", ")} · {t("rollouts.plan.hostsCount", { count: w.host_count })}</span>
+                        </span>
+                      </li>
+                    ))}
+                  </ol>
+                  <p className="mt-2 text-xs text-slate-500">
+                    {p.window ? <>{t("rollouts.plan.window")}: {p.window.name}{p.window.next[0] && <> · {t("rollouts.plan.nextOccurrence", { time: when(p.window.next[0].start) })}</>}</> : t("rollouts.plan.noWindow")}
+                    {p.active && p.activated_by && <> · {t("rollouts.plan.activatedBy", { user: p.activated_by })}</>}
+                  </p>
+                  {p.auto_start_error && (
+                    <p className="mt-1 flex items-center gap-1 text-xs text-flame-700"><AlertTriangle size={13} />
+                      {t("rollouts.plan.lastAutoStartError", { time: when(p.auto_start_error.occurrence), reason: codeText(t, "rollouts.issue", { code: p.auto_start_error.code, params: p.auto_start_error.params ?? {} }) })}
+                    </p>
+                  )}
+                </div>
+                {!readOnly && (
+                  <div className="flex flex-wrap gap-2">
+                    <Button onClick={() => openPreview(p, "start")}><Play size={14} />{t("rollouts.plan.previewStart")}</Button>
+                    {p.auto_start && (p.active
+                      ? <Button variant="outline" onClick={() => act(() => api.deactivateRolloutPlan(p.id), "rollouts.plan.deactivated")}><PowerOff size={14} />{t("rollouts.plan.deactivate")}</Button>
+                      : <Button variant="outline" onClick={() => openPreview(p, "activate")}><Power size={14} />{t("rollouts.plan.activate")}</Button>)}
+                    {p.auto_start && p.active && !p.fingerprint_ok && <Button variant="outline" onClick={() => openPreview(p, "activate")}><Power size={14} />{t("rollouts.plan.activate")}</Button>}
+                    <Button variant="outline" aria-label={t("rollouts.plan.edit")} onClick={() => setEditPlan(p)}><Pencil size={14} /></Button>
+                    <Button variant="dangerSubtle" aria-label={t("common.delete")} onClick={() => confirm(t("rollouts.plan.deleteConfirm", { name: p.name })) && act(() => api.deleteRolloutPlan(p.id), "rollouts.plan.deleted")}><Trash2 size={14} /></Button>
+                  </div>
+                )}
+              </div>
+            </Card>
+          ))}
+        </div>
+      ))}
+
+      {tab === "windows" && (windows === null ? <Skeleton className="h-32" /> : windows.length === 0 ? (
+        <EmptyState icon={CalendarClock} title={t("mwindows.none")} description={t("mwindows.noneHint")}
+          action={!readOnly && <Button onClick={() => setEditWindow("new")}><Plus size={16} />{t("mwindows.new")}</Button>} />
+      ) : (
+        <div className="space-y-3">
+          {windows.map((w) => (
+            <Card key={w.id} className="p-4">
+              <div className="flex flex-wrap items-start justify-between gap-3">
+                <div className="min-w-0">
+                  <h2 className="flex flex-wrap items-center gap-2 font-semibold text-slate-900">
+                    {w.name}
+                    {w.open_now && <Badge tone="success">{t("mwindows.openNow")}</Badge>}
+                    {!w.enabled && <Badge tone="neutral">{t("mwindows.off")}</Badge>}
+                  </h2>
+                  <p className="mt-1 text-sm text-slate-600">{windowSummary(t, w)} · {t("mwindows.cutoffSummary", { minutes: w.start_cutoff_minutes })}</p>
+                  {w.used_by.length > 0 && <p className="mt-1 text-xs text-slate-500">{t("mwindows.usedBy", { plans: w.used_by.map((p) => p.name).join(", ") })}</p>}
+                </div>
+                {!readOnly && (
+                  <div className="flex gap-2">
+                    <Button variant="outline" aria-label={t("mwindows.edit")} onClick={() => setEditWindow(w)}><Pencil size={14} /></Button>
+                    <Button variant="dangerSubtle" aria-label={t("common.delete")} onClick={() => confirm(t("mwindows.deleteConfirm", { name: w.name })) && act(() => api.deleteMaintenanceWindow(w.id), "mwindows.deleted")}><Trash2 size={14} /></Button>
+                  </div>
+                )}
+              </div>
+              {w.next.length > 0 && (
+                <div className="mt-3">
+                  <h3 className="mb-1 text-xs font-semibold uppercase tracking-wide text-slate-500">{t("mwindows.next")}</h3>
+                  <ul className="space-y-1 text-sm">
+                    {w.next.map((o) => {
+                      const exc = w.exceptions.find((e) => e.occurrence_date === o.local_date);
+                      return (
+                        <li key={`${o.local_date}-${o.start}`} className="flex flex-wrap items-center gap-2">
+                          <span className={o.skipped ? "text-slate-400 line-through" : ""}>{when(o.start)} – {new Date(o.end).toLocaleTimeString(locale, { hour: "2-digit", minute: "2-digit" })}</span>
+                          {o.skipped && <Badge tone="neutral">{t("mwindows.skipped")}</Badge>}
+                          {o.moved && <Badge tone="info">{t("mwindows.moved")}</Badge>}
+                          {exc?.reason && <span className="text-xs text-slate-500">{exc.reason}</span>}
+                          {!readOnly && !exc && (
+                            <span className="flex gap-1">
+                              <Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => act(() => api.addWindowException(w.id, { occurrence_date: o.local_date, kind: "skip" }), "mwindows.exceptionSaved")}>{t("mwindows.skip")}</Button>
+                              <Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => setMove({ window: w, occ: o, to: "" })}>{t("mwindows.move")}</Button>
+                            </span>
+                          )}
+                          {!readOnly && exc && <Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => act(() => api.deleteWindowException(w.id, exc.id), "mwindows.exceptionSaved")}>{t("mwindows.removeException")}</Button>}
+                        </li>
+                      );
+                    })}
+                  </ul>
+                </div>
+              )}
+            </Card>
+          ))}
+        </div>
+      ))}
+
+      {editPlan && (
+        <RolloutPlanEditor plan={editPlan === "new" ? null : editPlan} groups={groups} windows={windows ?? []} onClose={() => setEditPlan(null)}
+          onSaved={async () => { setEditPlan(null); toast.show(t("rollouts.plan.saved"), "success"); await load(); }} />
+      )}
+      {editWindow && (
+        <MaintenanceWindowEditor window={editWindow === "new" ? null : editWindow} onClose={() => setEditWindow(null)}
+          onSaved={async () => { setEditWindow(null); toast.show(t("mwindows.saved"), "success"); await load(); }} />
+      )}
+      {move && (
+        <Modal title={`${t("mwindows.move")}: ${when(move.occ.start)}`} onClose={() => setMove(null)}>
+          <div className="space-y-4 text-sm">
+            <label className="block"><span className="mb-1 block font-medium text-slate-700">{t("mwindows.moveTo")}</span>
+              <Input type="datetime-local" value={move.to} onChange={(e) => setMove({ ...move, to: e.target.value })} /></label>
+            <div className="flex justify-end gap-2">
+              <Button variant="ghost" onClick={() => setMove(null)}>{t("common.cancel")}</Button>
+              <Button disabled={!move.to} onClick={async () => {
+                const m = move;
+                setMove(null);
+                await act(() => api.addWindowException(m.window.id, { occurrence_date: m.occ.local_date, kind: "move", moved_start: new Date(m.to).toISOString() }), "mwindows.exceptionSaved");
+              }}>{t("rollouts.plan.save")}</Button>
+            </div>
+          </div>
+        </Modal>
+      )}
+      {preview && <PreviewModal preview={preview} onClose={() => setPreview(null)} onDone={async (runId) => {
+        setPreview(null);
+        if (runId) router.push(`/rollouts/runs/${runId}`);
+        else await load();
+      }} />}
+    </div>
+  );
+}
+
+function PreviewModal({ preview, onClose, onDone }: { preview: { mode: "start" | "activate"; data: RolloutPreview }; onClose: () => void; onDone: (runId?: string) => void }) {
+  const { t, errorMessage, locale } = useLanguage();
+  const toast = useToast();
+  const { data, mode } = preview;
+  const [name, setName] = useState("");
+  const [ack, setAck] = useState(false);
+  const [busy, setBusy] = useState(false);
+  const [error, setError] = useState("");
+  const plan = data.plan;
+
+  async function go() {
+    setBusy(true);
+    setError("");
+    try {
+      if (mode === "start") {
+        const run = await api.startRolloutPlan(plan.id, data.preview_token, name);
+        toast.show(t("rollouts.plan.started"), "success");
+        onDone(run.id);
+      } else {
+        await api.activateRolloutPlan(plan.id, data.preview_token);
+        toast.show(t("rollouts.plan.activated"), "success");
+        onDone();
+      }
+    } catch (e) {
+      setError(errorMessage(e, "rollouts.actionFailed"));
+    } finally {
+      setBusy(false);
+    }
+  }
+
+  return (
+    <Modal title={t("rollouts.plan.previewTitle", { name: plan.name })} onClose={() => !busy && onClose()}>
+      <div className="space-y-4 text-sm">
+        <p><Badge tone={plan.action === "major_update" ? "accent" : "info"}>{t(`rollouts.action.${plan.action}`)}</Badge>
+          {data.window && <span className="ml-2 text-slate-600">{t("rollouts.plan.window")}: {data.window.name}{data.window.next[0] && <> · {t("rollouts.plan.nextOccurrence", { time: new Date(data.window.next[0].start).toLocaleString(locale) })}</>}</span>}</p>
+        <ol className="space-y-2">
+          {data.waves.map((w) => (
+            <li key={w.index} className="rounded-lg border border-slate-200 p-3">
+              <div className="flex flex-wrap items-center gap-2">
+                <strong>{w.index + 1}. {w.label}</strong>
+                <span className="text-xs text-slate-500">{t("rollouts.detail.soak", { minutes: w.soak_minutes })}</span>
+                {w.carp_handover && <Badge tone="info">{t("rollouts.detail.handover")}</Badge>}
+                {w.require_approval && w.index > 0 && <Badge tone="accent">{t("rollouts.detail.approval")}</Badge>}
+              </div>
+              <p className="mt-1 text-slate-600">
+                {w.hosts.length === 0 ? t("rollouts.plan.emptyWave") : w.hosts.map((h) => `${h.name}${h.cluster_name ? ` (${h.cluster_name})` : ""}${h.enabled ? "" : ` – ${t("rollouts.plan.disabledHost")}`}`).join(", ")}
+              </p>
+            </li>
+          ))}
+        </ol>
+        {data.errors.length > 0 && <ul role="alert" className="space-y-1 rounded-lg bg-red-50 p-3 text-red-700">{data.errors.map((e, i) => <li key={i}>{codeText(t, "rollouts.issue", e)}</li>)}</ul>}
+        {data.warnings.length > 0 && <ul className="space-y-1 rounded-lg bg-flame-50 p-3 text-flame-800">{data.warnings.map((e, i) => <li key={i} className="flex gap-1"><AlertTriangle size={14} className="mt-0.5 shrink-0" />{codeText(t, "rollouts.issue", e)}</li>)}</ul>}
+        {data.can_start ? (
+          <>
+            {mode === "start" && (
+              <label className="block"><span className="mb-1 block">{t("rollouts.actions.confirmPlan")}</span>
+                <Input autoComplete="off" spellCheck={false} aria-label={t("rollouts.actions.confirmPlan")} value={name} placeholder={plan.name} onChange={(e) => setName(e.target.value)} /></label>
+            )}
+            <label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={ack} onChange={(e) => setAck(e.target.checked)} />{t("rollouts.plan.unattended")}</label>
+            {error && <p role="alert" className="text-red-600">{error}</p>}
+            <div className="flex justify-end gap-2">
+              <Button variant="ghost" onClick={onClose} disabled={busy}>{t("common.cancel")}</Button>
+              <Button variant={mode === "start" ? "danger" : "primary"} loading={busy} disabled={!ack || (mode === "start" && name !== plan.name)} onClick={go}>
+                {mode === "start" ? t("rollouts.plan.start") : t("rollouts.plan.activate")}
+              </Button>
+            </div>
+          </>
+        ) : <p className="text-red-700">{t("rollouts.plan.cannotStart")}</p>}
+        {data.can_start || <div className="flex justify-end"><Link href="#" onClick={(e) => { e.preventDefault(); onClose(); }} className="text-brand-600">{t("common.close")}</Link></div>}
+      </div>
+    </Modal>
+  );
+}

frontend/app/rollouts/runs/[id]/page.tsx

@@ -0,0 +1,262 @@
+"use client";
+
+import { Badge } from "@/components/ui/Badge";
+import { Button } from "@/components/ui/Button";
+import { Card } from "@/components/ui/Card";
+import { EmptyState } from "@/components/ui/EmptyState";
+import { Input } from "@/components/ui/Field";
+import { Modal } from "@/components/ui/Modal";
+import { Skeleton } from "@/components/ui/Skeleton";
+import { useToast } from "@/components/ui/Toast";
+import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
+import { useLanguage } from "@/components/LanguageProvider";
+import { api, RolloutGate, RolloutRunDetail } from "@/lib/api";
+import { CHECKS, codeText, HOST_TONE, isActiveRun, pauseDetailText, pauseText, RESULT_TONE, RUN_TONE, WAVE_TONE } from "@/lib/rollouts";
+import { useReadOnly } from "@/lib/useReadOnly";
+import { AlertTriangle, ArrowLeft, CheckCheck, Pause, Play, SkipForward, Square, Waves } from "lucide-react";
+import Link from "next/link";
+import { useParams } from "next/navigation";
+import { useCallback, useEffect, useState } from "react";
+
+type HostRow = RolloutRunDetail["hosts"][number];
+
+function GateMatrix({ gate, run, title }: { gate: RolloutGate; run: RolloutRunDetail; title: string }) {
+  const { t } = useLanguage();
+  const names = Object.fromEntries(run.hosts.filter((h) => h.host_id).map((h) => [h.host_id as string, h.host_name]));
+  const checks = CHECKS.filter((c) => Object.values(gate.hosts).some((row) => row[c]));
+  return (
+    <div className="overflow-x-auto">
+      <div className="mb-2 flex items-center gap-2 text-sm font-medium text-slate-700">
+        {title}
+        <Badge tone={gate.verdict === "ok" ? "success" : gate.verdict === "warn" ? "warning" : "danger"}>{t(`rollouts.gate.verdict.${gate.verdict}`)}</Badge>
+      </div>
+      <table className="w-full text-left text-sm">
+        <thead className="border-b border-slate-200 text-xs uppercase tracking-wide text-slate-500">
+          <tr>
+            <th className="px-3 py-2 font-medium">{t("rollouts.col.host")}</th>
+            {checks.map((c) => <th key={c} className="px-3 py-2 font-medium">{t(`rollouts.gate.check.${c}`)}</th>)}
+          </tr>
+        </thead>
+        <tbody>
+          {Object.entries(gate.hosts).map(([hid, row]) => (
+            <tr key={hid} className="border-b border-slate-100 last:border-0">
+              <td className="px-3 py-2 font-medium">{names[hid] ?? hid.slice(0, 8)}</td>
+              {checks.map((c) => {
+                const r = row[c];
+                return (
+                  <td key={c} className="px-3 py-2">
+                    {r ? (
+                      <Tooltip focusable content={codeText(t, "rollouts.gate.code", r)}>
+                        <Badge tone={RESULT_TONE[r.result]}>{t(`rollouts.gate.result.${r.result}`)}</Badge>
+                      </Tooltip>
+                    ) : <span className="text-slate-400">–</span>}
+                  </td>
+                );
+              })}
+            </tr>
+          ))}
+        </tbody>
+      </table>
+    </div>
+  );
+}
+
+export default function RolloutRunPage() {
+  const { t, ts, errorMessage, locale } = useLanguage();
+  const { id } = useParams<{ id: string }>();
+  const toast = useToast();
+  const readOnly = useReadOnly();
+  const [run, setRun] = useState<RolloutRunDetail | null>(null);
+  const [error, setError] = useState("");
+  const [busy, setBusy] = useState(false);
+  const [dialog, setDialog] = useState<{ kind: "pause" } | { kind: "abort" } | { kind: "skip"; host: HostRow } | null>(null);
+  const [reason, setReason] = useState("");
+  const [confirmName, setConfirmName] = useState("");
+
+  const load = useCallback(async () => {
+    try {
+      setRun(await api.getRolloutRun(id));
+      setError("");
+    } catch (e) {
+      setError(errorMessage(e, "rollouts.loadFailed"));
+    }
+  }, [id, errorMessage]);
+
+  useEffect(() => {
+    load();
+  }, [load]);
+  useEffect(() => {
+    if (!run || !isActiveRun(run.status)) return;
+    const timer = setInterval(load, 15000);
+    return () => clearInterval(timer);
+  }, [run?.status, load]); // eslint-disable-line react-hooks/exhaustive-deps
+
+  async function act(fn: () => Promise<RolloutRunDetail>, okKey: string) {
+    setBusy(true);
+    try {
+      setRun(await fn());
+      toast.show(t(okKey), "success");
+      setDialog(null);
+    } catch (e) {
+      toast.show(errorMessage(e, "rollouts.actionFailed"), "danger");
+    } finally {
+      setBusy(false);
+    }
+  }
+
+  const when = (value: string | null | undefined) => (value ? new Date(value).toLocaleString(locale) : "");
+  const time = (value: string | null | undefined) => (value ? new Date(value).toLocaleTimeString(locale, { hour: "2-digit", minute: "2-digit" }) : "");
+
+  if (error && !run) {
+    return <EmptyState icon={Waves} title={t("rollouts.detail.notFound")} description={error} action={<Button href="/rollouts" variant="outline">{t("rollouts.detail.back")}</Button>} />;
+  }
+  if (!run) return <Skeleton className="h-64" />;
+
+  const active = isActiveRun(run.status);
+  const current = run.waves.find((w) => w.index === run.current_wave);
+  const recheck = run.progress.gate_recheck_at || run.progress.cluster_recheck_at || run.progress.resume_recheck_at;
+  return (
+    <div>
+      <Link href="/rollouts" className="mb-3 inline-flex items-center gap-1 text-sm text-slate-600 hover:text-slate-900"><ArrowLeft size={14} />{t("rollouts.detail.back")}</Link>
+      <div className="mb-4 flex flex-wrap items-center justify-between gap-3">
+        <h1 className="flex flex-wrap items-center gap-2 text-2xl font-semibold text-slate-900">
+          {run.plan_name}
+          <Badge tone={RUN_TONE[run.status]}>{t(`rollouts.run.status.${run.status}`)}</Badge>
+        </h1>
+        {!readOnly && active && (
+          <div className="flex flex-wrap gap-2">
+            {run.status !== "paused" && <Button variant="outline" disabled={busy} onClick={() => { setReason(""); setDialog({ kind: "pause" }); }}><Pause size={14} />{t("rollouts.actions.pause")}</Button>}
+            {run.status === "paused" && <Button disabled={busy} onClick={() => act(() => api.resumeRolloutRun(run.id), "rollouts.actions.resumed")}><Play size={14} />{t("rollouts.actions.resume")}</Button>}
+            {run.status === "awaiting_approval" && <Button disabled={busy} onClick={() => act(() => api.approveRolloutWave(run.id), "rollouts.actions.approved")}><CheckCheck size={14} />{t("rollouts.actions.approve")}</Button>}
+            <Button variant="dangerSubtle" disabled={busy} onClick={() => { setReason(""); setConfirmName(""); setDialog({ kind: "abort" }); }}><Square size={14} />{t("rollouts.actions.abort")}</Button>
+          </div>
+        )}
+      </div>
+
+      <div className="mb-4 grid gap-2 text-sm text-slate-700 sm:grid-cols-2 lg:grid-cols-4">
+        <p>{t("rollouts.col.target")}: <strong>{run.target_version ?? "–"}</strong></p>
+        <p>{t(`rollouts.action.${run.action}`)} · {t(`rollouts.trigger.${run.trigger}`)}</p>
+        <p>{t("rollouts.col.started")}: {when(run.created_at)}</p>
+        <p>{t("rollouts.detail.owner")}: {run.started_by}</p>
+      </div>
+
+      {run.status === "paused" && (
+        <div role="alert" className="mb-4 rounded-xl border border-flame-200 bg-flame-50 p-4 text-sm text-flame-800">
+          <p className="flex items-center gap-2 font-semibold"><AlertTriangle size={16} />{pauseText(t, run.pause_reason)}
+            <InfoTip text={t("rollouts.pauseHint")} size={14} /></p>
+          {run.pause_detail && <p className="mt-1 break-words">{pauseDetailText(t, run.pause_detail)}</p>}
+        </div>
+      )}
+      {run.status === "cancelled" && run.pause_detail && Array.isArray(run.pause_detail.still_running) && run.pause_detail.still_running.length > 0 && (
+        <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.stillRunning", { hosts: (run.pause_detail.still_running as string[]).join(", ") })}</p>
+      )}
+      {run.status === "waiting_window" && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.waitingWindow")}</p>}
+      {run.status === "awaiting_approval" && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.awaiting")}</p>}
+      {(run.status === "soaking" && run.soak_until) && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.soakUntil", { time: time(run.soak_until) })}</p>}
+      {["gate", "wave_running", "resuming"].includes(run.status) && recheck && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.recheckAt", { time: time(recheck) })}</p>}
+      {run.status === "wave_running" && run.progress.blocked_since && <p className="mb-4 text-sm text-slate-600">{t("rollouts.detail.blockedSince", { time: time(run.progress.blocked_since) })}</p>}
+
+      <h2 className="mb-2 text-sm font-semibold uppercase tracking-wide text-slate-500">{t("rollouts.detail.waves")}</h2>
+      <ol className="mb-6 flex flex-wrap gap-2">
+        {run.waves.map((w) => (
+          <li key={w.index} aria-current={w.index === run.current_wave && active ? "step" : undefined}
+            className={`min-w-[9rem] flex-1 rounded-xl border p-3 text-sm ${w.index === run.current_wave && active ? "border-sky-300" : "border-slate-200"}`}>
+            <div className="flex items-center justify-between gap-2">
+              <strong>{w.index + 1}. {w.label}</strong>
+              <Badge tone={WAVE_TONE[w.status] ?? "neutral"}>{t(`rollouts.wave.status.${w.status}`)}</Badge>
+            </div>
+            {w.spec && <p className="mt-1 text-xs text-slate-500">{w.spec.groups.join(", ")} · {t("rollouts.detail.soak", { minutes: w.spec.soak_minutes })}
+              {w.spec.carp_handover && <> · {t("rollouts.detail.handover")}</>}{w.spec.require_approval && w.index > 0 && <> · {t("rollouts.detail.approval")}</>}</p>}
+            {w.approved_by && <p className="mt-1 text-xs text-slate-500">{t("rollouts.detail.approvedBy", { user: w.approved_by })}</p>}
+          </li>
+        ))}
+      </ol>
+
+      <h2 className="mb-2 text-sm font-semibold uppercase tracking-wide text-slate-500">{t("rollouts.detail.hosts")}</h2>
+      <Card className="mb-6 overflow-x-auto">
+        <table className="w-full text-left text-sm">
+          <thead className="border-b border-slate-200 bg-slate-50 text-xs uppercase tracking-wide text-slate-500">
+            <tr>
+              <th className="px-4 py-3 font-medium">{t("rollouts.col.host")}</th>
+              <th className="px-4 py-3 font-medium">{t("rollouts.col.wave")}</th>
+              <th className="px-4 py-3 font-medium">{t("rollouts.col.status")}</th>
+              <th className="px-4 py-3 font-medium">{t("rollouts.col.version")}</th>
+              <th className="px-4 py-3 font-medium">{t("rollouts.col.step")}</th>
+              <th className="px-4 py-3 font-medium">{t("rollouts.col.note")}</th>
+              <th className="px-4 py-3 font-medium"><span className="sr-only">{t("common.actions")}</span></th>
+            </tr>
+          </thead>
+          <tbody>
+            {run.hosts.map((h) => (
+              <tr key={h.id} className="border-b border-slate-100 last:border-0">
+                <td className="px-4 py-3 font-medium">{h.host_id ? <Link className="text-brand-600 hover:underline" href={`/hosts/${h.host_id}`}>{h.host_name}</Link> : h.host_name}
+                  {h.cluster_name && <div className="text-xs font-normal text-slate-500">{h.cluster_name}</div>}</td>
+                <td className="px-4 py-3">{run.waves.find((w) => w.index === h.wave_index)?.label}</td>
+                <td className="px-4 py-3"><Badge tone={HOST_TONE[h.status]}>{t(`rollouts.host.status.${h.status}`)}</Badge></td>
+                <td className="px-4 py-3 whitespace-nowrap">{h.version_before ?? "–"}{h.version_after && <> → <strong>{h.version_after}</strong></>}</td>
+                <td className="px-4 py-3 text-xs">{h.schedule?.phase ? t(`maint.phase.${h.schedule.phase}`) : h.schedule ? t(`maint.scheduleStatus.${h.schedule.status === "succeeded" ? "completed" : h.schedule.status}`) : ""}
+                  {h.schedule?.carp_handover && <div><Badge tone="info">{t("maint.handoverBadge")}</Badge></div>}</td>
+                <td className="max-w-xs px-4 py-3 text-xs text-slate-600 break-words">{h.detail ? ts(codeText(t, "rollouts.hostNote", h.detail)) : ""}</td>
+                <td className="px-4 py-3 text-right">
+                  {!readOnly && active && h.host_id && ["queued", "blocked", "failed"].includes(h.status) && (
+                    <Button variant="ghost" className="px-2 py-1 text-xs" onClick={() => { setReason(""); setDialog({ kind: "skip", host: h }); }}><SkipForward size={13} />{t("rollouts.actions.skip")}</Button>
+                  )}
+                </td>
+              </tr>
+            ))}
+          </tbody>
+        </table>
+      </Card>
+
+      <h2 className="mb-2 flex items-center gap-1 text-sm font-semibold uppercase tracking-wide text-slate-500">{t("rollouts.gate.title")}<InfoTip text={t("rollouts.gate.hint")} size={13} /></h2>
+      <div className="space-y-4">
+        {run.progress.resume_gate && Object.keys(run.progress.resume_gate.hosts).length > 0 && (
+          <Card className="p-4"><GateMatrix gate={run.progress.resume_gate} run={run} title={t("rollouts.gate.resume")} /></Card>
+        )}
+        {active && run.progress.cluster_gate && (
+          <Card className="p-4"><GateMatrix gate={run.progress.cluster_gate} run={run} title={t("rollouts.gate.cluster")} /></Card>
+        )}
+        {[...run.waves].reverse().filter((w) => w.gate_result).map((w) => (
+          <Card key={w.index} className="p-4">
+            <GateMatrix gate={w.gate_result as RolloutGate} run={run} title={`${w.index + 1}. ${w.label} · ${t("rollouts.detail.attempt", { n: w.gate_attempts })}`} />
+          </Card>
+        ))}
+        {!run.waves.some((w) => w.gate_result) && !run.progress.resume_gate && <p className="text-sm text-slate-500">{t("rollouts.detail.noGate")}{current ? ` (${current.label})` : ""}</p>}
+      </div>
+
+      {dialog?.kind === "pause" && (
+        <Modal title={t("rollouts.actions.pause")} onClose={() => !busy && setDialog(null)}>
+          <div className="space-y-4 text-sm">
+            <label className="block"><span className="mb-1 block">{t("rollouts.actions.pauseReason")}</span>
+              <Input value={reason} maxLength={500} onChange={(e) => setReason(e.target.value)} /></label>
+            <div className="flex justify-end"><Button loading={busy} onClick={() => act(() => api.pauseRolloutRun(run.id, reason), "rollouts.actions.paused")}><Pause size={14} />{t("rollouts.actions.pause")}</Button></div>
+          </div>
+        </Modal>
+      )}
+      {dialog?.kind === "skip" && (
+        <Modal title={t("rollouts.actions.skipTitle", { host: dialog.host.host_name })} onClose={() => !busy && setDialog(null)}>
+          <div className="space-y-4 text-sm">
+            <p>{t("rollouts.actions.skipHint")}</p>
+            <label className="block"><span className="mb-1 block">{t("rollouts.actions.reason")}</span>
+              <Input value={reason} maxLength={500} onChange={(e) => setReason(e.target.value)} /></label>
+            <div className="flex justify-end"><Button loading={busy} disabled={reason.trim().length < 3}
+              onClick={() => act(() => api.skipRolloutHost(run.id, dialog.host.host_id as string, reason), "rollouts.actions.skipped")}>{t("rollouts.actions.skip")}</Button></div>
+          </div>
+        </Modal>
+      )}
+      {dialog?.kind === "abort" && (
+        <Modal title={t("rollouts.actions.abortTitle")} onClose={() => !busy && setDialog(null)}>
+          <div className="space-y-4 text-sm">
+            <p className="flex items-start gap-2 text-flame-800"><AlertTriangle size={16} className="mt-0.5 shrink-0" />{t("rollouts.actions.abortHint")}</p>
+            <label className="block"><span className="mb-1 block">{t("rollouts.actions.reason")}</span>
+              <Input value={reason} maxLength={500} onChange={(e) => setReason(e.target.value)} /></label>
+            <label className="block"><span className="mb-1 block">{t("rollouts.actions.confirmPlan")}</span>
+              <Input autoComplete="off" spellCheck={false} value={confirmName} placeholder={run.plan_name} onChange={(e) => setConfirmName(e.target.value)} /></label>
+            <div className="flex justify-end"><Button variant="danger" loading={busy} disabled={reason.trim().length < 3 || confirmName !== run.plan_name}
+              onClick={() => act(() => api.abortRolloutRun(run.id, reason, confirmName), "rollouts.actions.aborted")}>{t("rollouts.actions.abort")}</Button></div>
+          </div>
+        </Modal>
+      )}
+    </div>
+  );
+}

frontend/app/settings/page.tsx

@@ -70,6 +70,7 @@ function NotificationsSection() {
   const [gatewayDown, setGatewayDown] = useState(true);
   const [resourceThreshold, setResourceThreshold] = useState(false);
   const [serviceDown, setServiceDown] = useState(false);
+  const [rolloutProblems, setRolloutProblems] = useState(true);
   const [gatewayLoss, setGatewayLoss] = useState(20);
   const [gatewayDelay, setGatewayDelay] = useState(500);
   const [diskLimit, setDiskLimit] = useState(90);
@@ -107,6 +108,7 @@ function NotificationsSection() {
     setGatewayDown(data.gateway_down_enabled);
     setResourceThreshold(data.resource_threshold_enabled);
     setServiceDown(data.service_down_enabled);
+    setRolloutProblems(data.rollout_problems_enabled ?? true);
     setGatewayLoss(data.gateway_loss_threshold_pct);
     setGatewayDelay(data.gateway_delay_threshold_ms);
     setDiskLimit(data.disk_threshold_pct);
@@ -162,6 +164,7 @@ function NotificationsSection() {
         gateway_down_enabled: gatewayDown,
         resource_threshold_enabled: resourceThreshold,
         service_down_enabled: serviceDown,
+        rollout_problems_enabled: rolloutProblems,
         gateway_loss_threshold_pct: gatewayLoss,
         gateway_delay_threshold_ms: gatewayDelay,
         disk_threshold_pct: diskLimit,
@@ -308,6 +311,7 @@ function NotificationsSection() {
             [t("settings.notify.event.gatewayDown"), gatewayDown, setGatewayDown],
             [t("settings.notify.event.resourceThreshold"), resourceThreshold, setResourceThreshold],
             [t("settings.notify.event.serviceDown"), serviceDown, setServiceDown],
+            [t("settings.notify.event.rolloutProblems"), rolloutProblems, setRolloutProblems],
           ].map(([label, value, setter]) => (
             <label key={label as string} className="flex items-center gap-2 text-sm">
               <input type="checkbox" checked={value as boolean} onChange={(e) => (setter as (v: boolean) => void)(e.target.checked)} />

frontend/app/updates/page.tsx

@@ -17,7 +17,9 @@ import {
   MaintenancePreview,
   MaintenanceRead,
   MaintenanceState,
+  RolloutStatus,
 } from "@/lib/api";
+import { RolloutStatusBanner } from "@/components/RolloutStatusBanner";
 import { useReadOnly } from "@/lib/useReadOnly";
 import { useLanguage } from "@/components/LanguageProvider";
 import type { TranslationParams } from "@/lib/i18n";
@@ -132,6 +134,7 @@ export default function UpdatesPage() {
   const [cancelReason, setCancelReason] = useState("");
   const [submitting, setSubmitting] = useState(false);
   const [log, setLog] = useState<{ host: Host; text: string } | null>(null);
+  const [rollout, setRollout] = useState<RolloutStatus | null>(null);
 
   useEffect(() => {
     mounted.current = true;
@@ -341,6 +344,7 @@ export default function UpdatesPage() {
         </div>
       </div>
 
+      <RolloutStatusBanner onStatus={setRollout} />
       {pageError && <p role="alert" className="mb-4 rounded-xl bg-red-50 px-4 py-3 text-sm text-red-700">{pageError}</p>}
 
       <div className="mb-6 grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
@@ -373,6 +377,9 @@ export default function UpdatesPage() {
           const carpWarning = state ? carpMaintenanceWarning(host, state, t) : null;
           const canCheck = host.enabled && state?.running === "ready" && !activeOperation;
           const isExpanded = expanded.has(host.id);
+          // A host the active rollout still owns: no manual firmware action next to it (the rollout does it or skips it).
+          const inRollout = Boolean(rollout?.host_ids.includes(host.id));
+          const rolloutLock = inRollout ? [<span key="rollout">{t("rollouts.inRolloutTip")}</span>] : [];
           return (
             <Card key={host.id} className="overflow-hidden">
               <div className="p-4 sm:p-5">
@@ -384,18 +391,19 @@ export default function UpdatesPage() {
                       {!host.enabled && <Badge tone="neutral">{t("updates.disabled")}</Badge>}
                       {(state?.carp_status ?? host.carp_status) && <Badge tone="neutral">CARP {state?.carp_status ?? host.carp_status}</Badge>}
                       {carpWarning && <Tooltip content={carpWarning} focusable><Badge tone="warning">{t("updates.carpMode")}</Badge></Tooltip>}
+                      {inRollout && <Tooltip content={t("rollouts.inRolloutTip")} focusable><Badge tone="accent">{t("rollouts.inRollout")}</Badge></Tooltip>}
                       {state?.reboot_required && <Tooltip content={t("updates.tipRebootRequired")} focusable><Badge tone="warning"><AlertTriangle size={12} />{t("updates.rebootRequired")}</Badge></Tooltip>}
                     </div>
                     <p className="mt-1 truncate text-xs text-slate-400">{host.base_url}{host.cluster_name ? t("updates.clusterSuffix", { name: host.cluster_name }) : ""}</p>
                   </div>
                   <div className="flex flex-wrap items-center gap-2">
                     <Tooltip content={t("updates.tipCheck")} focusable={!canCheck}><Button variant="secondary" disabled={readOnly || !canCheck || detail?.checking || submitting || bulkChecking} onClick={() => void checkHost(host)}>{detail?.checking ? <Spinner size={13} /> : <RefreshCw size={14} />}{t("updates.check")}</Button></Tooltip>
-                    {state && <CarpSwitchControl host={host} state={state} disabled={readOnly || Boolean(activeOperation) || submitting || detail?.checking} onChanged={() => loadOne(host)} />}
-                    <BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : blockerReasons(visibleBlockers)}>
-                      <Button disabled={readOnly || blockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "update")}>{blockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareUpdate")}</Button>
+                    {state && <CarpSwitchControl host={host} state={state} disabled={readOnly || inRollout || Boolean(activeOperation) || submitting || detail?.checking} onChanged={() => loadOne(host)} />}
+                    <BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : [...rolloutLock, ...blockerReasons(visibleBlockers)]}>
+                      <Button disabled={readOnly || inRollout || blockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "update")}>{blockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareUpdate")}</Button>
                     </BlockedTip>
-                    <BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : blockerReasons(visibleMajorBlockers)}>
-                      <Button variant="danger" disabled={readOnly || majorBlockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "major_update")}>{majorBlockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareMajor")}</Button>
+                    <BlockedTip reasons={readOnly ? [t("updates.readOnlyNote")] : [...rolloutLock, ...blockerReasons(visibleMajorBlockers)]}>
+                      <Button variant="danger" disabled={readOnly || inRollout || majorBlockers.length > 0 || submitting || detail?.checking} onClick={() => void openPreview(host, "major_update")}>{majorBlockers.length > 0 && !readOnly && <Lock size={13} aria-label={t("updates.lockedPrefix")} />}{t("updates.prepareMajor")}</Button>
                     </BlockedTip>
                     {/* Not nested under `state &&`: a host can be offline/unreachable
                         precisely because a maintenance action is stuck — the operator

frontend/components/HostMaintenance.tsx

@@ -3,6 +3,7 @@ import { AlertTriangle, Clock3 } from "lucide-react";
 import { Badge } from "@/components/ui/Badge";
 import { InfoTip, Tooltip } from "@/components/ui/Tooltip";
 import { useEffect, useState } from "react";
+import Link from "next/link";
 import { api, Host, MaintenanceRead, MaintenancePreview, HostOperation, MaintenanceState, ScheduledMaintenanceItem } from "@/lib/api";
 import { Button } from "@/components/ui/Button";
 import { Modal } from "@/components/ui/Modal";
@@ -21,11 +22,11 @@ export function HostMaintenance({host}: {host:Host}) {
   const [name,setName]=useState('');const [ack,setAck]=useState(false);const [log,setLog]=useState('');
   const [cancelOp,setCancelOp]=useState<HostOperation|null>(null);const [reason,setReason]=useState('');
   const [scheduled,setScheduled]=useState<ScheduledMaintenanceItem[]>([]);
-  const [schedule,setSchedule]=useState<{action:'reboot'|'update'|'major_update';runAt:string;ack:boolean;handover:boolean}|null>(null);
+  const [schedule,setSchedule]=useState<{action:'reboot'|'update'|'major_update';runAt:string;ack:boolean;handover:boolean;autoVerify:boolean}|null>(null);
   async function load(){setError('');try{setData(await api.hostMaintenance(host.id));}catch(e){setError(errorMessage(e,'maint.loadFailed'));}}
   async function loadScheduled(){try{setScheduled(await api.listScheduledMaintenance(host.id));}catch{/* non-fatal, section just stays empty */}}
   useEffect(()=>{if(open){load();loadScheduled();}},[open,host.id]);
-  async function createSchedule(){if(!schedule)return;setBusy(true);setError('');try{await api.createScheduledMaintenance(host.id,schedule.action,new Date(schedule.runAt).toISOString(),schedule.handover);setSchedule(null);await loadScheduled();}catch(e){setError(errorMessage(e,'maint.scheduleFailed'));}finally{setBusy(false);}}
+  async function createSchedule(){if(!schedule)return;setBusy(true);setError('');try{await api.createScheduledMaintenance(host.id,schedule.action,new Date(schedule.runAt).toISOString(),schedule.handover,schedule.autoVerify);setSchedule(null);await loadScheduled();}catch(e){setError(errorMessage(e,'maint.scheduleFailed'));}finally{setBusy(false);}}
   async function cancelSchedule(id:string){setBusy(true);setError('');try{await api.cancelScheduledMaintenance(host.id,id);await loadScheduled();}catch(e){setError(errorMessage(e,'maint.cancelFailed'));}finally{setBusy(false);}}
   useEffect(()=>{if(!open || !data?.operations.some(o=>['starting','submitted','unknown'].includes(o.status)))return;const timer=setInterval(()=>{load();loadScheduled();},15000);return()=>clearInterval(timer);},[open,host.id,data?.operations.map(o=>o.status).join(',')]);
   async function preview(action:'reboot'|'update'|'major_update'){setBusy(true);setError('');try{setPlan(await api.previewMaintenance(host.id,action));setName('');setAck(false);}catch(e){setError(errorMessage(e,'maint.previewFailed'));}finally{setBusy(false);}}
@@ -34,10 +35,10 @@ export function HostMaintenance({host}: {host:Host}) {
   async function cancelOperation(){if(!cancelOp)return;setBusy(true);setError('');try{await api.cancelMaintenance(host.id,cancelOp.id,name,reason);setCancelOp(null);await load();}catch(e){setError(errorMessage(e,'maint.cancelFailed'));}finally{setBusy(false);}}
   async function check(){setBusy(true);setError('');try{await api.checkHostUpdates(host.id);setNotice(t('maint.checkRequested'));await load();}catch(e){setError(errorMessage(e,'maint.checkFailed'));}finally{setBusy(false);}}
   const masterBlocked=!!data?.state?.cluster_name && data.state.carp_status==='MASTER' && !schedule?.handover;
-  return <section className="my-6 rounded-xl border border-slate-200 bg-white p-5"><div className="flex flex-wrap items-center justify-between gap-2"><h2 className="flex items-center gap-2 font-semibold">{t('maint.title')}<InfoTip text={t('maint.intro')} /></h2><Button variant="ghost" onClick={()=>setOpen(v=>!v)}>{open?t('maint.close'):t('maint.open')}</Button></div>{open && <div className="mt-4 space-y-4">{error && <p role="alert" className="text-sm text-red-600">{error}</p>}{notice && <p role="status" className="text-sm text-sky-700">{notice}</p>}{!data && !error && <p>{t('common.loading')}</p>}{data?.error && <p role="alert" className="text-sm text-red-600">{ts(data.error)}</p>}{data?.state && <><div className="grid gap-2 text-sm sm:grid-cols-2"><p>{t('maint.installed',{version:''})}<strong>{data.state.version}</strong></p><p>{t('maint.firmwareJob',{state:data.state.running})}</p><p className="sm:col-span-2">{ts(data.state.firmware_message)}</p><p>{t('maint.lastCheck',{value:data.state.last_check || t('maint.unknown')})}</p><p>{t('maint.backup',{value:data.state.backup?new Date(data.state.backup.created_at).toLocaleString(locale):t('maint.noBackup')})}</p></div><div className="flex flex-wrap items-center gap-2"><Tooltip content={t('updates.tipCheck')}><Button variant="secondary" disabled={busy || data.state.running!=='ready' || !host.enabled || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onClick={check}>{t('maint.checkUpdates')}</Button></Tooltip><CarpSwitchControl host={host} state={data.state} disabled={busy || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onChanged={load}/><Button variant="danger" disabled={busy || data.state.actions.reboot.length>0} onClick={()=>preview('reboot')}>{t('maint.previewReboot')}</Button><Tooltip content={t('updates.tipPrepare')}><Button disabled={busy || data.state.actions.update.length>0} onClick={()=>preview('update')}>{t('maint.previewUpdate')}</Button></Tooltip><Tooltip content={t('updates.tipPrepareMajor')}><Button variant="danger" disabled={busy || data.state.actions.major_update.length>0} onClick={()=>preview('major_update')}>{t('maint.previewMajor')}</Button></Tooltip><Button variant="ghost" disabled={busy || !host.enabled} onClick={()=>setSchedule({action:'reboot',runAt:'',ack:false,handover:false})}>{t('maint.scheduleLater')}</Button><Button variant="ghost" className="ml-auto" disabled={busy} onClick={load}>{t('maint.refreshStatus')}</Button></div>{readableReasons(data.state,host,t,ts).length>0 && <Tooltip focusable placement="bottom" content={<span className="block space-y-1">{readableReasons(data.state,host,t,ts).map(reason=><span key={reason} className="block">{reason}</span>)}</span>}><Badge tone="warning">{t('maint.hints')} ({readableReasons(data.state,host,t,ts).length})</Badge></Tooltip>}{scheduled.length>0 && <div className="space-y-2"><h3 className="text-sm font-semibold text-slate-700">{t('maint.scheduled')}</h3>{scheduled.map(s=><div key={s.id} className="flex flex-wrap items-center justify-between gap-2 rounded-lg border border-slate-200 p-3 text-sm"><div><strong>{labels[s.action]}</strong> · {new Date(s.run_at).toLocaleString(locale)} · {t(`maint.scheduleStatus.${s.status==='succeeded' && s.result?.handover ? 'completed' : s.status}`)}{s.status==='running' && s.phase && <span> ({t(`maint.phase.${s.phase}`)})</span>}{s.carp_handover && <Badge tone="info" className="ml-2">{t('maint.handoverBadge')}</Badge>}{s.result?.error && <span className="text-red-600"> — {ts(s.result.error)}</span>}</div>{s.status==='pending' && <Button variant="ghost" disabled={busy} onClick={()=>cancelSchedule(s.id)}>{t('common.cancel')}</Button>}</div>)}</div>}</>}{data?.operations && data.operations.length>0 && <h3 className="text-sm font-semibold text-slate-700">{t('maint.history')}</h3>}{data?.operations.map(op=><div key={op.id} className="rounded-lg border border-slate-200 p-3 text-sm"><strong>{labels[op.action]} · {['acknowledged','cancelled','unknown','starting'].includes(op.status)?t(`maint.opStatus.${op.status}`):t('maint.opStatus.submitted')}</strong><p className="text-slate-500">{new Date(op.created_at).toLocaleString(locale)} · {op.actor}</p><p className="mt-1">{ts(op.message ?? "")}</p>{['starting','submitted','unknown'].includes(op.status) && <div className="mt-2 flex flex-wrap gap-2"><Button variant="secondary" disabled={busy} onClick={()=>{setFinish(op);setName('');setAck(false);}}>{t('maint.confirmFinish')}</Button><Button variant="danger" disabled={busy} onClick={()=>{setCancelOp(op);setName('');setReason('');}}>{t('maint.cancelOperation')}</Button></div>}</div>)}<Button variant="ghost" disabled={busy} onClick={async()=>{try{const p=await api.hostUpdateProgress(host.id);setLog(p.log || p.status);}catch(e){setError(errorMessage(e,'maint.logFailed'));}}}>{t('maint.readLog')}</Button>{log && <pre className="max-h-72 overflow-auto whitespace-pre-wrap break-all rounded-lg bg-slate-50 p-3 tLine truncated
+  return <section className="my-6 rounded-xl border border-slate-200 bg-white p-5"><div className="flex flex-wrap items-center justify-between gap-2"><h2 className="flex items-center gap-2 font-semibold">{t('maint.title')}<InfoTip text={t('maint.intro')} /></h2><Button variant="ghost" onClick={()=>setOpen(v=>!v)}>{open?t('maint.close'):t('maint.open')}</Button></div>{open && <div className="mt-4 space-y-4">{error && <p role="alert" className="text-sm text-red-600">{error}</p>}{notice && <p role="status" className="text-sm text-sky-700">{notice}</p>}{!data && !error && <p>{t('common.loading')}</p>}{data?.error && <p role="alert" className="text-sm text-red-600">{ts(data.error)}</p>}{data?.state && <><div className="grid gap-2 text-sm sm:grid-cols-2"><p>{t('maint.installed',{version:''})}<strong>{data.state.version}</strong></p><p>{t('maint.firmwareJob',{state:data.state.running})}</p><p className="sm:col-span-2">{ts(data.state.firmware_message)}</p><p>{t('maint.lastCheck',{value:data.state.last_check || t('maint.unknown')})}</p><p>{t('maint.backup',{value:data.state.backup?new Date(data.state.backup.created_at).toLocaleString(locale):t('maint.noBackup')})}</p></div><div className="flex flex-wrap items-center gap-2"><Tooltip content={t('updates.tipCheck')}><Button variant="secondary" disabled={busy || data.state.running!=='ready' || !host.enabled || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onClick={check}>{t('maint.checkUpdates')}</Button></Tooltip><CarpSwitchControl host={host} state={data.state} disabled={busy || data.operations.some(o=>['starting','submitted','unknown'].includes(o.status))} onChanged={load}/><Button variant="danger" disabled={busy || data.state.actions.reboot.length>0} onClick={()=>preview('reboot')}>{t('maint.previewReboot')}</Button><Tooltip content={t('updates.tipPrepare')}><Button disabled={busy || data.state.actions.update.length>0} onClick={()=>preview('update')}>{t('maint.previewUpdate')}</Button></Tooltip><Tooltip content={t('updates.tipPrepareMajor')}><Button variant="danger" disabled={busy || data.state.actions.major_update.length>0} onClick={()=>preview('major_update')}>{t('maint.previewMajor')}</Button></Tooltip><Button variant="ghost" disabled={busy || !host.enabled} onClick={()=>setSchedule({action:'reboot',runAt:'',ack:false,handover:false,autoVerify:true})}>{t('maint.scheduleLater')}</Button><Button variant="ghost" className="ml-auto" disabled={busy} onClick={load}>{t('maint.refreshStatus')}</Button></div>{readableReasons(data.state,host,t,ts).length>0 && <Tooltip focusable placement="bottom" content={<span className="block space-y-1">{readableReasons(data.state,host,t,ts).map(reason=><span key={reason} className="block">{reason}</span>)}</span>}><Badge tone="warning">{t('maint.hints')} ({readableReasons(data.state,host,t,ts).length})</Badge></Tooltip>}{scheduled.length>0 && <div className="space-y-2"><h3 className="text-sm font-semibold text-slate-700">{t('maint.scheduled')}</h3>{scheduled.map(s=><div key={s.id} className="flex flex-wrap items-center justify-between gap-2 rounded-lg border border-slate-200 p-3 text-sm"><div><strong>{labels[s.action]}</strong> · {new Date(s.run_at).toLocaleString(locale)} · {t(`maint.scheduleStatus.${s.status==='succeeded' && (s.result?.handover || s.result?.auto_verified) ? 'completed' : s.status}`)}{s.status==='running' && s.phase && <span> ({t(`maint.phase.${s.phase}`)})</span>}{s.carp_handover && <Badge tone="info" className="ml-2">{t('maint.handoverBadge')}</Badge>}{s.origin==='rollout' && <Badge tone="accent" className="ml-2">{t('maint.rolloutBadge')}</Badge>}{s.rollout_run_id && <Link href={`/rollouts/runs/${s.rollout_run_id}`} className="ml-2 text-brand-600 hover:underline">{t('maint.openRollout')}</Link>}{s.result?.error && <span className="text-red-600"> — {ts(s.result.error)}</span>}</div>{s.status==='pending' && <Button variant="ghost" disabled={busy} onClick={()=>cancelSchedule(s.id)}>{t('common.cancel')}</Button>}</div>)}</div>}</>}{data?.operations && data.operations.length>0 && <h3 className="text-sm font-semibold text-slate-700">{t('maint.history')}</h3>}{data?.operations.map(op=><div key={op.id} className="rounded-lg border border-slate-200 p-3 text-sm"><strong>{labels[op.action]} · {['acknowledged','cancelled','unknown','starting'].includes(op.status)?t(`maint.opStatus.${op.status}`):t('maint.opStatus.submitted')}</strong><p className="text-slate-500">{new Date(op.created_at).toLocaleString(locale)} · {op.actor}</p><p className="mt-1">{ts(op.message ?? "")}</p>{['starting','submitted','unknown'].includes(op.status) && <div className="mt-2 flex flex-wrap gap-2"><Button variant="secondary" disabled={busy} onClick={()=>{setFinish(op);setName('');setAck(false);}}>{t('maint.confirmFinish')}</Button><Button variant="danger" disabled={busy} onClick={()=>{setCancelOp(op);setName('');setReason('');}}>{t('maint.cancelOperation')}</Button></div>}</div>)}<Button variant="ghost" disabled={Line truncated
   {plan && <Modal title={t('maint.confirmTitle',{action:labels[plan.action]})} onClose={()=>{if(!busy)setPlan(null);}}><div className="space-y-4 text-sm"><p><strong>{plan.state.host_name}</strong> · {plan.state.base_url} · {plan.state.version}</p>{plan.action==='major_update' && <p className="flex items-center gap-2 font-medium text-red-700"><AlertTriangle size={16}/>{t('updates.majorWarnShort')}<InfoTip text={t('updates.majorWarnTip')}/></p>}<p className="flex items-center gap-2 font-medium text-flame-800"><Clock3 size={16}/>{t('updates.downtimeShort')}<InfoTip text={t('maint.downtimeNote',{reboot:plan.state.reboot_required?t('maint.andReboot'):''})}/></p><p className="text-xs text-slate-500">{t('updates.validUntilShort',{time:new Date(plan.expires_at).toLocaleTimeString(locale)})}</p><label className="block"><span className="flex items-center gap-1">{t('maint.confirmHostname')}<InfoTip text={t('updates.tipConfirmName')} size={13}/></span><Input className="mt-1" autoFocus autoComplete="off" spellCheck={false} aria-label={t('maint.confirmHostnameAria')} value={name} placeholder={host.name} onChange={e=>setName(e.target.value)} onKeyDown={e=>{if(e.key==='Enter'&&!busy&&name===host.name&&ack)void execute();}}/></label><label className="flex items-start gap-2"><input type="checkbox" checked={ack} onChange={e=>setAck(e.target.checked)}/>{t('maint.windowReady')}</label><Tooltip focusable={busy||name!==host.name||!ack} content={name!==host.name?t('updates.confirmRunDisabledName'):!ack?t('updates.confirmRunDisabledAck'):null}><Button variant="danger" disabled={busy || name!==host.name || !ack} onClick={execute}>{busy?t('maint.submitting'):t('maint.executeNow',{action:labels[plan.action]})}</Button></Tooltip></div></Modal>}
   {finish && <Modal title={finish.action.startsWith('carp_')?t('maint.finishCarpTitle'):t('maint.finishTitle')} onClose={()=>{if(!busy)setFinish(null);}}><div className="space-y-4 text-sm"><p className="flex items-start gap-1">{finish.action.startsWith('carp_')?t('maint.finishCarpCheck'):<>{t('maint.finishCheckPre')}<strong>{host.name}</strong>. </>}<InfoTip text={t('updates.finishNoteTip')} size={13} className="mt-0.5"/></p><Input autoFocus autoComplete="off" spellCheck={false} aria-label={t('maint.finishHostnameAria')} value={name} onChange={e=>setName(e.target.value)} placeholder={host.name}/><label className="flex gap-2"><input type="checkbox" checked={ack} onChange={e=>setAck(e.target.checked)}/>{t('maint.finishChecked')}</label>{error && <p className="text-red-600">{error}</p>}<Tooltip focusable={busy||name!==host.name||!ack} content={name!==host.name?t('updates.confirmRunDisabledName'):!ack?t('updates.confirmRunDisabledAck'):null}><Button disabled={busy || name!==host.name || !ack} onClick={closeOperation}>{t('maint.saveFinish')}</Button></Tooltip></div></Modal>}
   {cancelOp && <Modal title={t('maint.cancelOperation')} onClose={()=>{if(!busy)setCancelOp(null);}}><div className="space-y-4 text-sm"><p className="flex items-center gap-2 font-medium text-flame-800"><AlertTriangle size={16}/>{t('updates.cancelWarnShort')}<InfoTip text={t('updates.cancelWarnTip')}/></p><Input autoFocus autoComplete="off" spellCheck={false} aria-label={t('maint.cancelHostnameAria')} value={name} onChange={e=>setName(e.target.value)} placeholder={host.name}/><label className="block"><span className="flex items-center gap-1">{t('maint.reasonLabel')}<InfoTip text={t('updates.reasonMinHint')} size={13}/></span><textarea className="mt-1 w-full rounded-lg border border-slate-300 p-2" rows={2} maxLength={500} value={reason} onChange={e=>setReason(e.target.value)} placeholder={t('maint.reasonPlaceholder')}/></label>{error && <p className="text-red-600">{error}</p>}<Button variant="danger" disabled={busy || name!==host.name || reason.trim().length<3} onClick={cancelOperation}>{t('maint.closeWithoutConfirm')}</Button></div></Modal>}
-  {schedule && <Modal title={t('maint.scheduleTitle')} onClose={()=>{if(!busy)setSchedule(null);}}><div className="space-y-4 text-sm"><label className="block"><span className="block">{t('maint.actionLabel')}</span><select className="mt-1 w-full rounded-lg border border-slate-300 p-2" value={schedule.action} onChange={e=>setSchedule({...schedule,action:e.target.value as typeof schedule.action})}><option value="reboot">{t('maint.action.reboot')}</option><option value="update">{t('maint.action.update')}</option><option value="major_update">{t('maint.action.major_update')}</option></select></label><label className="block"><span className="block">{t('maint.timeLabel')}</span><Input className="mt-1"  aria-label={t('maint.timeLabel')} type="datetime-local" value={schedule.runAt} onChange={e=>setSchedule({...schedule,runAt:e.target.value})}/></label>{host.cluster_name && <label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={schedule.handover} onChange={e=>setSchedule({...schedule,handover:e.target.checked})}/><span>{t('maint.handoverOption')}<InfoTip text={t('maint.handoverHint')}/></span></label>}<p className="flex items-center gap-2 font-medium text-flame-800"><AlertTriangle size={16}/>{t('maint.unattendedShort')}<InfoTip text={t('maint.unattendedWarning')}/></p><label className="flex items-start gap-2"><input type="checkbox" checked={schedule.ack} onChange={e=>setSchedule({...schedule,ack:e.target.checked})}/>{t('maint.unattendedAck')}</label>{masterBlocked && <p role="alert" className="flex items-start gap-2 rounded-lg bg-red-50 p-3 text-red-700"><AlertTriangle size={16} className="mt-0.5 shrink-0"/>{t('maint.scheduleMasterBlocked')}</p>}{error && <p className="text-red-600">{error}</p>}<Button variant="danger" disabled={busy || masterBlocked || !schedule.runAt || !schedule.ack} onClick={createSchedule}>{busy?t('maint.scheduling'):t('maint.scheduleIt')}</Button></div></Modal>}
+  {schedule && <Modal title={t('maint.scheduleTitle')} onClose={()=>{if(!busy)setSchedule(null);}}><div className="space-y-4 text-sm"><label className="block"><span className="block">{t('maint.actionLabel')}</span><select className="mt-1 w-full rounded-lg border border-slate-300 p-2" value={schedule.action} onChange={e=>setSchedule({...schedule,action:e.target.value as typeof schedule.action})}><option value="reboot">{t('maint.action.reboot')}</option><option value="update">{t('maint.action.update')}</option><option value="major_update">{t('maint.action.major_update')}</option></select></label><label className="block"><span className="block">{t('maint.timeLabel')}</span><Input className="mt-1"  aria-label={t('maint.timeLabel')} type="datetime-local" value={schedule.runAt} onChange={e=>setSchedule({...schedule,runAt:e.target.value})}/></label>{host.cluster_name && <label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={schedule.handover} onChange={e=>setSchedule({...schedule,handover:e.target.checked})}/><span>{t('maint.handoverOption')}<InfoTip text={t('maint.handoverHint')}/></span></label>}<label className="flex items-start gap-2"><input type="checkbox" className="mt-1" checked={schedule.autoVerify} onChange={e=>setSchedule({...schedule,autoVerify:e.target.checked})}/><span>{t('maint.autoVerifyOption')}<InfoTip text={t('maint.autoVerifyHint')}/></span></label><p className="flex items-center gap-2 font-medium text-flame-800"><AlertTriangle size={16}/>{t('maint.unattendedShort')}<InfoTip text={t('maint.unattendedWarning')}/></p><label className="flex items-start gap-2"><input type="checkbox" checked={schedule.ack} onChange={e=>setSchedule({...schedule,ack:e.target.checked})}/>{t('maint.unattendedAck')}</label>{masterBlocked && <p role="alert" className="flex items-start gap-2 rounded-lg bg-red-50 p-3 text-red-700"><AlertTriangle size={16} className="mt-0.5 shrink-0"/>{t('maint.scheduleMasterBlocked')}</p>}{error && <p className="text-red-600">{error}</p>}<Button variant="danger" disabled={busy || masterBlocked || !schedule.runAt || !schedule.ack} onClick={createSchedule}>{busy?t('maint.scheduling'):t('maint.scheduleIt')}</Button></div></Modal>}
   </section>;
 }

frontend/components/MaintenanceWindowEditor.tsx

@@ -0,0 +1,145 @@
+"use client";
+
+import { Button } from "@/components/ui/Button";
+import { FieldGroup, Input, Select } from "@/components/ui/Field";
+import { Modal } from "@/components/ui/Modal";
+import { InfoTip } from "@/components/ui/Tooltip";
+import { useLanguage } from "@/components/LanguageProvider";
+import { api, MaintenanceWindow, MaintenanceWindowWrite } from "@/lib/api";
+import { useMemo, useState } from "react";
+
+const EMPTY: MaintenanceWindowWrite = {
+  name: "", timezone: "Europe/Berlin", recurrence: "weekly", weekdays: [6], month_mode: null, month_nth: null,
+  month_weekday: null, month_day: null, start_time: "02:00", duration_minutes: 180, start_cutoff_minutes: 60, enabled: true,
+};
+
+function timeZones(): string[] {
+  try {
+    const intl = Intl as unknown as { supportedValuesOf?: (key: string) => string[] };
+    const zones = intl.supportedValuesOf?.("timeZone") ?? [];
+    return zones.includes("UTC") ? zones : ["UTC", ...zones];
+  } catch {
+    return ["UTC", "Europe/Berlin"];
+  }
+}
+
+export function MaintenanceWindowEditor({ window, onClose, onSaved }: {
+  window: MaintenanceWindow | null; onClose: () => void; onSaved: (w: MaintenanceWindow) => void;
+}) {
+  const { t, errorMessage } = useLanguage();
+  const [form, setForm] = useState<MaintenanceWindowWrite>(() => (window ? {
+    name: window.name, timezone: window.timezone, recurrence: window.recurrence, weekdays: window.weekdays,
+    month_mode: window.month_mode, month_nth: window.month_nth, month_weekday: window.month_weekday, month_day: window.month_day,
+    start_time: window.start_time, duration_minutes: window.duration_minutes, start_cutoff_minutes: window.start_cutoff_minutes,
+    enabled: window.enabled,
+  } : EMPTY));
+  const [busy, setBusy] = useState(false);
+  const [error, setError] = useState("");
+  const zones = useMemo(timeZones, []);
+  const set = (patch: Partial<MaintenanceWindowWrite>) => setForm((f) => ({ ...f, ...patch }));
+
+  async function save() {
+    setBusy(true);
+    setError("");
+    const body: MaintenanceWindowWrite = form.recurrence === "weekly"
+      ? { ...form, month_mode: null, month_nth: null, month_weekday: null, month_day: null }
+      : form.month_mode === "day"
+        ? { ...form, weekdays: [], month_nth: null, month_weekday: null }
+        : { ...form, weekdays: [], month_mode: "nth_weekday", month_nth: form.month_nth ?? 1, month_weekday: form.month_weekday ?? 6, month_day: null };
+    try {
+      const saved = window ? await api.updateMaintenanceWindow(window.id, body) : await api.createMaintenanceWindow(body);
+      onSaved(saved);
+    } catch (e) {
+      setError(errorMessage(e, "common.saveFailed"));
+    } finally {
+      setBusy(false);
+    }
+  }
+
+  return (
+    <Modal title={window ? t("mwindows.edit") : t("mwindows.new")} onClose={() => !busy && onClose()}>
+      <div className="space-y-4 text-sm">
+        <FieldGroup label={t("mwindows.name")}>
+          <Input value={form.name} maxLength={120} onChange={(e) => set({ name: e.target.value })} />
+        </FieldGroup>
+        <div className="grid gap-4 sm:grid-cols-2">
+          <FieldGroup label={t("mwindows.timezone")}>
+            <Select value={form.timezone} onChange={(e) => set({ timezone: e.target.value })}>
+              {zones.map((z) => <option key={z} value={z}>{z}</option>)}
+            </Select>
+          </FieldGroup>
+          <FieldGroup label={t("mwindows.recurrence")}>
+            <Select value={form.recurrence} onChange={(e) => set({ recurrence: e.target.value as "weekly" | "monthly",
+              month_mode: e.target.value === "monthly" ? (form.month_mode ?? "nth_weekday") : null })}>
+              <option value="weekly">{t("mwindows.weekly")}</option>
+              <option value="monthly">{t("mwindows.monthly")}</option>
+            </Select>
+          </FieldGroup>
+        </div>
+        {form.recurrence === "weekly" ? (
+          <fieldset>
+            <legend className="mb-1 font-medium text-slate-700">{t("mwindows.weekdays")}</legend>
+            <div className="flex flex-wrap gap-3">
+              {[0, 1, 2, 3, 4, 5, 6].map((d) => (
+                <label key={d} className="flex items-center gap-1">
+                  <input type="checkbox" checked={form.weekdays.includes(d)}
+                    onChange={(e) => set({ weekdays: e.target.checked ? [...form.weekdays, d].sort() : form.weekdays.filter((x) => x !== d) })} />
+                  {t(`mwindows.dayName.${d}`)}
+                </label>
+              ))}
+            </div>
+          </fieldset>
+        ) : (
+          <div className="grid gap-4 sm:grid-cols-3">
+            <FieldGroup label={t("mwindows.monthMode")}>
+              <Select value={form.month_mode ?? "nth_weekday"} onChange={(e) => set({ month_mode: e.target.value as "nth_weekday" | "day" })}>
+                <option value="nth_weekday">{t("mwindows.nthWeekday")}</option>
+                <option value="day">{t("mwindows.dayOfMonth")}</option>
+              </Select>
+            </FieldGroup>
+            {form.month_mode === "day" ? (
+              <FieldGroup label={t("mwindows.day")}>
+                <Input type="number" min={1} max={28} value={form.month_day ?? 1} onChange={(e) => set({ month_day: Number(e.target.value) })} />
+              </FieldGroup>
+            ) : (
+              <>
+                <FieldGroup label={t("mwindows.nthWeekday")}>
+                  <Select value={String(form.month_nth ?? 1)} onChange={(e) => set({ month_nth: Number(e.target.value) })}>
+                    {[1, 2, 3, 4, -1].map((n) => <option key={n} value={n}>{t(`mwindows.nth.${n}`)}</option>)}
+                  </Select>
+                </FieldGroup>
+                <FieldGroup label={t("mwindows.weekdays")}>
+                  <Select value={String(form.month_weekday ?? 6)} onChange={(e) => set({ month_weekday: Number(e.target.value) })}>
+                    {[0, 1, 2, 3, 4, 5, 6].map((d) => <option key={d} value={d}>{t(`mwindows.dayLong.${d}`)}</option>)}
+                  </Select>
+                </FieldGroup>
+              </>
+            )}
+          </div>
+        )}
+        <div className="grid gap-4 sm:grid-cols-3">
+          <FieldGroup label={t("mwindows.startTime")}>
+            <Input type="time" value={form.start_time} onChange={(e) => set({ start_time: e.target.value })} />
+          </FieldGroup>
+          <FieldGroup label={t("mwindows.duration")}>
+            <Input type="number" min={15} max={1440} value={form.duration_minutes} onChange={(e) => set({ duration_minutes: Number(e.target.value) })} />
+          </FieldGroup>
+          <label className="block">
+            <span className="mb-1 flex items-center gap-1 font-medium text-slate-700">{t("mwindows.cutoff")}<InfoTip text={t("mwindows.cutoffHint")} size={13} /></span>
+            <Input type="number" min={5} max={1440} value={form.start_cutoff_minutes} onChange={(e) => set({ start_cutoff_minutes: Number(e.target.value) })} />
+          </label>
+        </div>
+        <p className="flex items-center gap-1 text-xs text-slate-500">{t("mwindows.dstHint")}</p>
+        <label className="flex items-center gap-2">
+          <input type="checkbox" checked={form.enabled} onChange={(e) => set({ enabled: e.target.checked })} />
+          {t("mwindows.enabled")}
+        </label>
+        {error && <p role="alert" className="text-red-600">{error}</p>}
+        <div className="flex justify-end gap-2">
+          <Button variant="ghost" onClick={onClose} disabled={busy}>{t("common.cancel")}</Button>
+          <Button onClick={save} loading={busy} disabled={!form.name.trim()}>{t("rollouts.plan.save")}</Button>
+        </div>
+      </div>
+    </Modal>
+  );
+}

frontend/components/Nav.tsx

@@ -35,6 +35,7 @@ import {
   Tags,
   Users,
   Waypoints,
+  Waves,
 } from "lucide-react";
 import Link from "next/link";
 import { usePathname, useRouter } from "next/navigation";
@@ -138,6 +139,12 @@ export const navGroups: NavGroup[] = [
         icon: RefreshCw,
         keywords: ["Firmware", "OPNsense-Updates", "Pakete", "Wartung", "Neustart erforderlich", "Versionen"],
       },
+      {
+        href: "/rollouts",
+        label: "Rollouts",
+        icon: Waves,
+        keywords: ["Update-Ringe", "Canary", "Wellen", "Rollout", "Wartungsfenster", "Firmware-Rollout", "Pilot", "Health-Gate", "Ringe"],
+      },
       {
         href: "/settings",
         label: "Einstellungen",

frontend/components/OpnexusUpdateSection.tsx

@@ -285,6 +285,9 @@ export function OpnexusUpdateSection() {
             </details>
           )}
 
+          {status.can_update && (status.active_rollouts ?? 0) > 0 && (
+            <p role="status" className="rounded-lg bg-flame-50 p-3 text-sm text-flame-800">{t("rollouts.updaterWarning")}</p>
+          )}
           {status.can_update && status.latest && !running && (
             <Button onClick={() => setConfirm("update")} disabled={busy}>
               {t("settings.update.button", { latest: status.latest })}

frontend/components/RolloutPlanEditor.tsx

@@ -0,0 +1,165 @@
+"use client";
+
+import { Button } from "@/components/ui/Button";
+import { FieldGroup, Input, Select } from "@/components/ui/Field";
+import { Modal } from "@/components/ui/Modal";
+import { InfoTip } from "@/components/ui/Tooltip";
+import { useLanguage } from "@/components/LanguageProvider";
+import { api, DeviceGroup, MaintenanceWindow, RolloutCheck, RolloutPlan, RolloutPlanWrite, RolloutWaveWrite } from "@/lib/api";
+import { ArrowDown, ArrowUp, Plus, Trash2 } from "lucide-react";
+import { useState } from "react";
+
+const OPTIONAL_CHECKS: RolloutCheck[] = ["carp", "gateways", "services", "compliance"];
+const DEFAULT_CHECKS: Partial<Record<RolloutCheck, boolean>> = { carp: true, gateways: true, services: true, compliance: false };
+
+function newWave(index: number): RolloutWaveWrite {
+  return { label: ["canary", "pilot", "prod"][index] ?? `${index + 1}`, group_ids: [], soak_minutes: 30, carp_handover: index > 0, require_approval: false };
+}
+
+export function RolloutPlanEditor({ plan, groups, windows, onClose, onSaved }: {
+  plan: RolloutPlan | null; groups: DeviceGroup[]; windows: MaintenanceWindow[]; onClose: () => void; onSaved: (p: RolloutPlan) => void;
+}) {
+  const { t, errorMessage } = useLanguage();
+  const [form, setForm] = useState<RolloutPlanWrite>(() => plan ? {
+    name: plan.name, description: plan.description, action: plan.action, window_id: plan.window_id, auto_start: plan.auto_start,
+    gate_checks: { ...DEFAULT_CHECKS, ...plan.gate_checks }, compliance_mode: plan.compliance_mode,
+    waves: plan.waves.map((w) => ({ label: w.label, group_ids: w.group_ids, soak_minutes: w.soak_minutes, carp_handover: w.carp_handover, require_approval: w.require_approval })),
+  } : { name: "", description: "", action: "update", window_id: null, auto_start: false, gate_checks: DEFAULT_CHECKS, compliance_mode: "warn", waves: [newWave(0)] });
+  const [busy, setBusy] = useState(false);
+  const [error, setError] = useState("");
+  const set = (patch: Partial<RolloutPlanWrite>) => setForm((f) => ({ ...f, ...patch }));
+  const setWave = (i: number, patch: Partial<RolloutWaveWrite>) => set({ waves: form.waves.map((w, j) => (j === i ? { ...w, ...patch } : w)) });
+  const moveWave = (i: number, dir: -1 | 1) => {
+    const waves = [...form.waves];
+    [waves[i], waves[i + dir]] = [waves[i + dir], waves[i]];
+    set({ waves });
+  };
+  const autoStartPossible = form.action === "update" && !!form.window_id;
+
+  async function save() {
+    setBusy(true);
+    setError("");
+    const body = { ...form, auto_start: autoStartPossible && form.auto_start };
+    try {
+      onSaved(plan ? await api.updateRolloutPlan(plan.id, body) : await api.createRolloutPlan(body));
+    } catch (e) {
+      setError(errorMessage(e, "common.saveFailed"));
+    } finally {
+      setBusy(false);
+    }
+  }
+
+  const valid = form.name.trim() && form.waves.length > 0 && form.waves.every((w) => w.label.trim() && w.group_ids.length > 0);
+  return (
+    <Modal title={plan ? t("rollouts.plan.edit") : t("rollouts.plan.new")} onClose={() => !busy && onClose()}>
+      <div className="space-y-4 text-sm">
+        <div className="grid gap-4 sm:grid-cols-2">
+          <FieldGroup label={t("rollouts.plan.name")}>
+            <Input value={form.name} maxLength={120} onChange={(e) => set({ name: e.target.value })} />
+          </FieldGroup>
+          <FieldGroup label={t("rollouts.plan.action")}>
+            <Select value={form.action} onChange={(e) => set({ action: e.target.value as RolloutPlanWrite["action"] })}>
+              <option value="update">{t("rollouts.action.update")}</option>
+              <option value="major_update">{t("rollouts.action.major_update")}</option>
+            </Select>
+          </FieldGroup>
+        </div>
+        <FieldGroup label={t("rollouts.plan.description")}>
+          <Input value={form.description} maxLength={2000} onChange={(e) => set({ description: e.target.value })} />
+        </FieldGroup>
+        <FieldGroup label={t("rollouts.plan.window")}>
+          <Select value={form.window_id ?? ""} onChange={(e) => set({ window_id: e.target.value || null })}>
+            <option value="">{t("rollouts.plan.noWindow")}</option>
+            {windows.map((w) => <option key={w.id} value={w.id}>{w.name}</option>)}
+          </Select>
+        </FieldGroup>
+        <label className={`flex items-start gap-2 ${autoStartPossible ? "" : "opacity-60"}`}>
+          <input type="checkbox" className="mt-1" disabled={!autoStartPossible} checked={autoStartPossible && form.auto_start}
+            onChange={(e) => set({ auto_start: e.target.checked })} />
+          <span>{t("rollouts.plan.autoStart")} <InfoTip text={t("rollouts.plan.autoStartHint")} size={13} /></span>
+        </label>
+
+        <fieldset className="space-y-3">
+          <legend className="mb-1 font-medium text-slate-700">{t("rollouts.plan.waves")}</legend>
+          {form.waves.map((wave, i) => (
+            <div key={i} className="space-y-3 rounded-lg border border-slate-200 p-3">
+              <div className="flex flex-wrap items-end gap-2">
+                <span className="mb-2 inline-flex h-6 w-6 items-center justify-center rounded-full bg-slate-100 text-xs font-semibold text-slate-700">{i + 1}</span>
+                <div className="min-w-[8rem] flex-1">
+                  <FieldGroup label={t("rollouts.plan.waveLabel")}>
+                    <Input value={wave.label} maxLength={60} onChange={(e) => setWave(i, { label: e.target.value })} />
+                  </FieldGroup>
+                </div>
+                <div className="w-32">
+                  <FieldGroup label={t("rollouts.plan.soak")} hint={t("rollouts.plan.soakHint")}>
+                    <Input type="number" min={0} max={1440} value={wave.soak_minutes} onChange={(e) => setWave(i, { soak_minutes: Number(e.target.value) })} />
+                  </FieldGroup>
+                </div>
+                <div className="mb-0.5 flex gap-1">
+                  <Button variant="ghost" aria-label={t("rollouts.plan.moveUp")} disabled={i === 0} onClick={() => moveWave(i, -1)}><ArrowUp size={14} /></Button>
+                  <Button variant="ghost" aria-label={t("rollouts.plan.moveDown")} disabled={i === form.waves.length - 1} onClick={() => moveWave(i, 1)}><ArrowDown size={14} /></Button>
+                  <Button variant="dangerSubtle" aria-label={t("rollouts.plan.removeWave")} disabled={form.waves.length === 1}
+                    onClick={() => set({ waves: form.waves.filter((_, j) => j !== i) })}><Trash2 size={14} /></Button>
+                </div>
+              </div>
+              <div>
+                <p className="mb-1 font-medium text-slate-700">{t("rollouts.plan.groups")}</p>
+                <div className="flex flex-wrap gap-x-4 gap-y-1">
+                  {groups.map((g) => (
+                    <label key={g.id} className="flex items-center gap-1">
+                      <input type="checkbox" checked={wave.group_ids.includes(g.id)}
+                        onChange={(e) => setWave(i, { group_ids: e.target.checked ? [...wave.group_ids, g.id] : wave.group_ids.filter((x) => x !== g.id) })} />
+                      {g.name} <span className="text-xs text-slate-500">({g.host_ids.length})</span>
+                    </label>
+                  ))}
+                </div>
+              </div>
+              <div className="flex flex-wrap gap-x-6 gap-y-1">
+                <label className="flex items-center gap-2">
+                  <input type="checkbox" checked={wave.carp_handover} onChange={(e) => setWave(i, { carp_handover: e.target.checked })} />
+                  {t("rollouts.plan.handover")}
+                </label>
+                {i > 0 && (
+                  <label className="flex items-center gap-2">
+                    <input type="checkbox" checked={wave.require_approval} onChange={(e) => setWave(i, { require_approval: e.target.checked })} />
+                    {t("rollouts.plan.approval")}
+                  </label>
+                )}
+              </div>
+            </div>
+          ))}
+          <Button variant="outline" disabled={form.waves.length >= 10} onClick={() => set({ waves: [...form.waves, newWave(form.waves.length)] })}>
+            <Plus size={14} />{t("rollouts.plan.addWave")}
+          </Button>
+        </fieldset>
+
+        <fieldset>
+          <legend className="mb-1 flex items-center gap-1 font-medium text-slate-700">{t("rollouts.plan.checks")}<InfoTip text={t("rollouts.plan.checksHint")} size={13} /></legend>
+          <div className="flex flex-wrap gap-x-5 gap-y-1">
+            {OPTIONAL_CHECKS.map((c) => (
+              <label key={c} className="flex items-center gap-2">
+                <input type="checkbox" checked={!!form.gate_checks[c]} onChange={(e) => set({ gate_checks: { ...form.gate_checks, [c]: e.target.checked } })} />
+                {t(`rollouts.gate.check.${c}`)}
+              </label>
+            ))}
+          </div>
+          {form.gate_checks.compliance && (
+            <div className="mt-2 max-w-xs">
+              <FieldGroup label={t("rollouts.plan.complianceMode")}>
+                <Select value={form.compliance_mode} onChange={(e) => set({ compliance_mode: e.target.value as "warn" | "fail" })}>
+                  <option value="warn">{t("rollouts.plan.complianceWarn")}</option>
+                  <option value="fail">{t("rollouts.plan.complianceFail")}</option>
+                </Select>
+              </FieldGroup>
+            </div>
+          )}
+        </fieldset>
+        {error && <p role="alert" className="text-red-600">{error}</p>}
+        <div className="flex justify-end gap-2">
+          <Button variant="ghost" onClick={onClose} disabled={busy}>{t("common.cancel")}</Button>
+          <Button onClick={save} loading={busy} disabled={!valid}>{t("rollouts.plan.save")}</Button>
+        </div>
+      </div>
+    </Modal>
+  );
+}

frontend/components/RolloutStatusBanner.tsx

@@ -0,0 +1,55 @@
+"use client";
+
+import { Badge } from "@/components/ui/Badge";
+import { useLanguage } from "@/components/LanguageProvider";
+import { api, RolloutStatus } from "@/lib/api";
+import { pauseText, RUN_TONE } from "@/lib/rollouts";
+import { Waves } from "lucide-react";
+import Link from "next/link";
+import { useEffect, useState } from "react";
+
+/** "Rollout X – wave 2/3" strip for /updates and /rollouts. Polls while a run is active. */
+export function RolloutStatusBanner({ onStatus }: { onStatus?: (status: RolloutStatus) => void }) {
+  const { t } = useLanguage();
+  const [status, setStatus] = useState<RolloutStatus | null>(null);
+
+  useEffect(() => {
+    let alive = true;
+    async function load() {
+      try {
+        const next = await api.rolloutStatus();
+        if (!alive) return;
+        setStatus(next);
+        onStatus?.(next);
+      } catch {
+        /* the banner is optional */
+      }
+    }
+    load();
+    const timer = setInterval(load, 15000);
+    return () => {
+      alive = false;
+      clearInterval(timer);
+    };
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+  }, []);
+
+  const run = status?.active;
+  if (!run) return null;
+  return (
+    <div role="status" className="mb-4 flex flex-wrap items-center gap-2 rounded-xl border border-sky-200 bg-brand-50 px-4 py-3 text-sm text-slate-800">
+      <Waves size={16} className="shrink-0 text-brand-600" aria-hidden />
+      <span className="min-w-0 flex-1">
+        {t("rollouts.banner.active", {
+          plan: run.plan_name, wave: `${run.current_wave + 1}/${run.wave_count}`, label: run.current_wave_label ?? "",
+          status: t(`rollouts.run.status.${run.status}`),
+        })}
+        {run.status === "paused" && run.pause_reason && <span className="text-slate-600"> — {pauseText(t, run.pause_reason)}</span>}
+      </span>
+      <Badge tone={RUN_TONE[run.status]}>{t(`rollouts.run.status.${run.status}`)}</Badge>
+      <Link href={`/rollouts/runs/${run.id}`} className="font-medium text-brand-600 underline-offset-2 hover:underline">
+        {t("rollouts.banner.open")}
+      </Link>
+    </div>
+  );
+}

frontend/lib/api.ts

@@ -1273,6 +1273,7 @@ export interface UpdaterStatus {
   rollback_available: boolean;
   rollback_loses_data: boolean;
   rollback_version: string | null;
+  active_rollouts?: number;
 }
 
 export const api = {
@@ -1311,8 +1312,32 @@ export const api = {
   stopService: (id:string, name:string) => request<{result:string}>(`/hosts/${id}/services/${encodeURIComponent(name)}/stop`,{method:"POST"}),
   restartService: (id:string, name:string) => request<{result:string}>(`/hosts/${id}/services/${encodeURIComponent(name)}/restart`,{method:"POST"}),
   listScheduledMaintenance: (id:string) => request<ScheduledMaintenanceItem[]>(`/hosts/${id}/maintenance/schedule`),
-  createScheduledMaintenance: (id:string, action:"reboot"|"update"|"major_update", run_at:string, carp_handover=false) => request<ScheduledMaintenanceItem>(`/hosts/${id}/maintenance/schedule`,{method:"POST",body:JSON.stringify({action,run_at,acknowledge_unattended:true,carp_handover})}),
+  createScheduledMaintenance: (id:string, action:"reboot"|"update"|"major_update", run_at:string, carp_handover=false, auto_verify=false) => request<ScheduledMaintenanceItem>(`/hosts/${id}/maintenance/schedule`,{method:"POST",body:JSON.stringify({action,run_at,acknowledge_unattended:true,carp_handover,auto_verify})}),
   cancelScheduledMaintenance: (id:string, scheduleId:string) => request<ScheduledMaintenanceItem>(`/hosts/${id}/maintenance/schedule/${scheduleId}`,{method:"DELETE"}),
+  // Maintenance windows and firmware rollouts (1.25)
+  listMaintenanceWindows: () => request<MaintenanceWindow[]>("/maintenance-windows"),
+  createMaintenanceWindow: (data: MaintenanceWindowWrite) => request<MaintenanceWindow>("/maintenance-windows", { method: "POST", body: JSON.stringify(data) }),
+  updateMaintenanceWindow: (id: string, data: MaintenanceWindowWrite) => request<MaintenanceWindow>(`/maintenance-windows/${id}`, { method: "PUT", body: JSON.stringify(data) }),
+  deleteMaintenanceWindow: (id: string) => request<void>(`/maintenance-windows/${id}`, { method: "DELETE" }),
+  addWindowException: (id: string, data: { occurrence_date: string; kind: "skip" | "move"; moved_start?: string; reason?: string }) => request<MaintenanceWindow>(`/maintenance-windows/${id}/exceptions`, { method: "POST", body: JSON.stringify(data) }),
+  deleteWindowException: (id: string, exceptionId: string) => request<MaintenanceWindow>(`/maintenance-windows/${id}/exceptions/${exceptionId}`, { method: "DELETE" }),
+  listRolloutPlans: () => request<RolloutPlan[]>("/rollouts/plans"),
+  getRolloutPlan: (id: string) => request<RolloutPlan>(`/rollouts/plans/${id}`),
+  createRolloutPlan: (data: RolloutPlanWrite) => request<RolloutPlan>("/rollouts/plans", { method: "POST", body: JSON.stringify(data) }),
+  updateRolloutPlan: (id: string, data: RolloutPlanWrite) => request<RolloutPlan>(`/rollouts/plans/${id}`, { method: "PUT", body: JSON.stringify(data) }),
+  deleteRolloutPlan: (id: string) => request<void>(`/rollouts/plans/${id}`, { method: "DELETE" }),
+  previewRolloutPlan: (id: string) => request<RolloutPreview>(`/rollouts/plans/${id}/preview`, { method: "POST" }),
+  activateRolloutPlan: (id: string, preview_token: string) => request<RolloutPlan>(`/rollouts/plans/${id}/activate`, { method: "POST", body: JSON.stringify({ preview_token }) }),
+  deactivateRolloutPlan: (id: string) => request<RolloutPlan>(`/rollouts/plans/${id}/deactivate`, { method: "POST" }),
+  startRolloutPlan: (id: string, preview_token: string, confirm_name: string) => request<RolloutRunDetail>(`/rollouts/plans/${id}/start`, { method: "POST", body: JSON.stringify({ preview_token, confirm_name, acknowledge_unattended: true }) }),
+  listRolloutRuns: () => request<RolloutRunSummary[]>("/rollouts/runs"),
+  rolloutStatus: () => request<RolloutStatus>("/rollouts/status"),
+  getRolloutRun: (id: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}`),
+  pauseRolloutRun: (id: string, reason: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/pause`, { method: "POST", body: JSON.stringify({ reason }) }),
+  resumeRolloutRun: (id: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/resume`, { method: "POST" }),
+  approveRolloutWave: (id: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/approve`, { method: "POST" }),
+  skipRolloutHost: (id: string, host_id: string, reason: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/skip-host`, { method: "POST", body: JSON.stringify({ host_id, reason }) }),
+  abortRolloutRun: (id: string, reason: string, confirm_name: string) => request<RolloutRunDetail>(`/rollouts/runs/${id}/abort`, { method: "POST", body: JSON.stringify({ reason, confirm_name }) }),
   audit: (filters: Record<string,string | number> = {}) => request<AuditPage>(`/audit?${new URLSearchParams(Object.entries(filters).filter(([,v])=>v!=="").map(([k,v])=>[k,String(v)]))}`),
   dashboard: () => request<DashboardSummary>("/dashboard"),
   versionCheck: () => request<VersionCheck>("/system/version-check"),
@@ -1746,7 +1771,62 @@ export interface MaintenanceState {
 export interface HostOperation { id:string; host_id:string; action:"reboot"|"update"|"major_update"|"carp_enter"|"carp_leave"; status:string; created_at:string; expires_at:string; actor:string; message:string|null; }
 export interface OpnsenseService { id:string; name:string; description:string; locked:boolean|number; running:boolean|number; }
 export interface HostResourceSample { id:string; sampled_at:string; mem_used_pct:number|null; load_avg_1m:number|null; disk_used_pct:number|null; raw:Record<string,unknown>; }
-export interface ScheduledMaintenanceItem { id:string; host_id:string; action:"reboot"|"update"|"major_update"; run_at:string; status:"pending"|"running"|"succeeded"|"failed"|"cancelled"; actor:string; created_at:string; result:{error?:string;response?:unknown;handover?:boolean;version_before?:string|null;version_after?:string|null;reboot_required?:boolean}; host_operation_id:string|null; carp_handover:boolean; phase:"handover_wait"|"job_wait"|"leave"|"leave_wait"|null; }
+export interface ScheduledMaintenanceItem { id:string; host_id:string; action:"reboot"|"update"|"major_update"; run_at:string; status:"pending"|"running"|"succeeded"|"failed"|"cancelled"; actor:string; created_at:string; result:{error?:string;response?:unknown;handover?:boolean;auto_verified?:boolean;version_before?:string|null;version_after?:string|null;reboot_required?:boolean}; host_operation_id:string|null; carp_handover:boolean; phase:"handover_wait"|"job_wait"|"verify"|"leave"|"leave_wait"|null; auto_verify?:boolean; origin?:"manual"|"rollout"; rollout_run_id?:string|null; }
+
+// ── Maintenance windows and firmware rollouts (1.25) ──
+export interface Occurrence { local_date: string; start: string; end: string; cutoff: string; moved: boolean; skipped: boolean; }
+export interface MaintenanceWindowWrite {
+  name: string; timezone: string; recurrence: "weekly" | "monthly"; weekdays: number[];
+  month_mode: "nth_weekday" | "day" | null; month_nth: number | null; month_weekday: number | null; month_day: number | null;
+  start_time: string; duration_minutes: number; start_cutoff_minutes: number; enabled: boolean;
+}
+export interface MaintenanceWindow extends MaintenanceWindowWrite {
+  id: string; created_by: string; created_at: string; open_now: boolean; next: Occurrence[];
+  exceptions: { id: string; occurrence_date: string; kind: "skip" | "move"; moved_start: string | null; reason: string; actor: string; created_at: string }[];
+  used_by: { id: string; name: string; auto_start: boolean; active: boolean }[];
+}
+export type RolloutCheck = "online" | "version" | "carp" | "gateways" | "services" | "locks" | "compliance";
+export interface RolloutIssue { code: string; params: Record<string, string | number | boolean | null>; }
+export interface RolloutWaveWrite { label: string; group_ids: string[]; soak_minutes: number; carp_handover: boolean; require_approval: boolean; }
+export interface RolloutPlanWrite {
+  name: string; description: string; action: "update" | "major_update"; window_id: string | null; auto_start: boolean;
+  gate_checks: Partial<Record<RolloutCheck, boolean>>; compliance_mode: "warn" | "fail"; waves: RolloutWaveWrite[];
+}
+export interface RolloutPlan extends Omit<RolloutPlanWrite, "waves"> {
+  id: string;
+  waves: (RolloutWaveWrite & { id: string; position: number; groups: string[]; host_count: number })[];
+  active: boolean; activated_by: string | null; activated_at: string | null; fingerprint_ok: boolean;
+  auto_start_error: { occurrence: string; at: string; code: string; params?: Record<string, string> } | null;
+  window: { id: string; name: string; next: Occurrence[] } | null; errors: RolloutIssue[]; warnings: RolloutIssue[];
+  host_count: number; last_run: { id: string; status: RolloutRunStatus; created_at: string } | null;
+  created_by: string; created_at: string; updated_at: string;
+}
+export interface RolloutPreview {
+  waves: { index: number; label: string; soak_minutes: number; carp_handover: boolean; require_approval: boolean; groups: string[];
+           hosts: { id: string; name: string; cluster_name: string | null; enabled: boolean }[] }[];
+  errors: RolloutIssue[]; warnings: RolloutIssue[]; window: RolloutPlan["window"]; preview_token: string; can_start: boolean; plan: RolloutPlan;
+}
+export type RolloutRunStatus = "waiting_window" | "preflight" | "wave_running" | "soaking" | "gate" | "awaiting_approval" | "resuming" | "paused" | "succeeded" | "cancelled";
+export type RolloutHostStatus = "queued" | "running" | "verifying" | "succeeded" | "skipped" | "failed" | "blocked" | "cancelled";
+export interface RolloutCheckResult { result: "ok" | "warn" | "fail" | "skip"; code: string; params: Record<string, string | number | null>; }
+export interface RolloutGate { hosts: Record<string, Partial<Record<RolloutCheck, RolloutCheckResult>>>; verdict: "ok" | "warn" | "fail"; }
+export interface RolloutRunSummary {
+  id: string; plan_id: string | null; plan_name: string; action: "update" | "major_update"; status: RolloutRunStatus; paused_from: string | null;
+  current_wave: number; wave_count: number; current_wave_label: string | null; target_version: string | null;
+  pause_reason: string | null; pause_detail: Record<string, unknown> | null; started_by: string; trigger: "manual" | "window";
+  window_occurrence: string | null; created_at: string; finished_at: string | null; step_at: string | null;
+  host_counts: Partial<Record<RolloutHostStatus, number>>; active: boolean;
+}
+export interface RolloutRunDetail extends RolloutRunSummary {
+  soak_until: string | null; progress: { gate_recheck_at?: string | null; cluster_recheck_at?: string | null; blocked_since?: string | null; resume_gate?: RolloutGate; cluster_gate?: RolloutGate; resume_recheck_at?: string | null };
+  waves: { index: number; label: string; status: string; gate_result: RolloutGate | null; gate_attempts: number; soak_until: string | null;
+           approved_by: string | null; started_at: string | null; finished_at: string | null;
+           spec: { label: string; soak_minutes: number; carp_handover: boolean; require_approval: boolean; groups: string[] } | null }[];
+  hosts: { id: string; host_id: string | null; host_name: string; cluster_name: string | null; wave_index: number; order: number; status: RolloutHostStatus;
+           version_before: string | null; version_after: string | null; detail: RolloutIssue | null; started_at: string | null; finished_at: string | null;
+           schedule: { id: string; status: string; phase: string | null; carp_handover: boolean; host_operation_id: string | null } | null }[];
+}
+export interface RolloutStatus { active: RolloutRunSummary | null; host_ids: string[]; current_host_ids: string[]; }
 export interface MaintenanceRead {state:MaintenanceState|null;error:string|null;operations:HostOperation[];}
 export interface MaintenancePreview {preview_id:string;expires_at:string;action:"reboot"|"update"|"major_update";state:MaintenanceState;}
 export interface CarpSwitchState {
@@ -1797,6 +1877,7 @@ export interface NotificationSettings {
   gateway_down_enabled: boolean;
   resource_threshold_enabled: boolean;
   service_down_enabled: boolean;
+  rollout_problems_enabled: boolean;
   gateway_loss_threshold_pct: number;
   gateway_delay_threshold_ms: number;
   disk_threshold_pct: number;
@@ -1853,6 +1934,7 @@ export interface NotificationSettingsUpdate {
   gateway_down_enabled?: boolean;
   resource_threshold_enabled?: boolean;
   service_down_enabled?: boolean;
+  rollout_problems_enabled?: boolean;
   gateway_loss_threshold_pct?: number;
   gateway_delay_threshold_ms?: number;
   disk_threshold_pct?: number;
@@ -1889,7 +1971,8 @@ export type ProblemType =
   | "backup_stale"
   | "backup_offsite_failed"
   | "compliance_finding"
-  | "gateway_down" | "resource_threshold" | "service_down";
+  | "gateway_down" | "resource_threshold" | "service_down"
+  | "rollout_paused" | "rollout_start_failed";
 
 export interface CarpEvent {
   id: string;

frontend/lib/i18n/de.json

@@ -15,7 +15,8 @@
     "saving": "Speichern…",
     "deleteFailed": "Löschen fehlgeschlagen",
     "description": "Beschreibung",
-    "versionLabel": "OPNexus {version} „{codename}“"
+    "versionLabel": "OPNexus {version} „{codename}“",
+    "delete": "Löschen"
   },
   "settings": {
     "language": {
@@ -69,7 +70,8 @@
         "networkDrift": "Netzwerk-Abweichung im Cluster",
         "gatewayDown": "Gateway gestört (offline, Paketverlust oder Latenz über der Schwelle)",
         "resourceThreshold": "Ressourcen-Schwelle überschritten (Festplatte, Speicher, Swap, CPU-Last; standardmäßig aus)",
-        "serviceDown": "Überwachter Dienst ausgefallen (standardmäßig aus)"
+        "serviceDown": "Überwachter Dienst ausgefallen (standardmäßig aus)",
+        "rolloutProblems": "Rollout angehalten oder nicht automatisch gestartet"
       },
       "pollInterval": "Prüfintervall (Sekunden)",
       "pollHint": "Mindestens 60s, um die Firewalls nicht unnötig zu belasten.",
@@ -297,7 +299,8 @@
       "hostsystem-syslog": "Syslog",
       "hostsystem-snapshots": "Snapshots",
       "hostsystem-diagnostics": "Diagnose",
-      "setup": "Einrichtung"
+      "setup": "Einrichtung",
+      "rollouts": "Rollouts"
     }
   },
   "dashboard": {
@@ -763,7 +766,28 @@
     "NOTIFICATION_CHANNEL_NOT_FOUND": "Der Benachrichtigungskanal wurde nicht gefunden.",
     "NOTIFICATION_CHANNEL_NAME_TAKEN": "Ein Kanal mit diesem Namen existiert bereits.",
     "NOTIFICATION_CHANNEL_TEST_FAILED": "Testnachricht nicht zugestellt: {reason}",
-    "NOTIFICATION_WEBHOOK_URL_INVALID": "Ungültige Webhook-Adresse: {reason}"
+    "NOTIFICATION_WEBHOOK_URL_INVALID": "Ungültige Webhook-Adresse: {reason}",
+    "WINDOW_NOT_FOUND": "Wartungsfenster nicht gefunden.",
+    "WINDOW_INVALID": "Wartungsfenster ungültig (Feld: {field}).",
+    "WINDOW_NAME_TAKEN": "Ein Wartungsfenster mit diesem Namen gibt es bereits.",
+    "WINDOW_IN_USE": "Das Fenster wird von einem Rollout-Plan verwendet. Erst den Plan ändern oder löschen.",
+    "WINDOW_EXCEPTION_INVALID": "Ausnahme ungültig (Feld: {field}). Nur künftige Termine des Fensters lassen sich auslassen oder verschieben.",
+    "WINDOW_EXCEPTION_EXISTS": "Für diesen Termin gibt es schon eine Ausnahme.",
+    "WINDOW_EXCEPTION_NOT_FOUND": "Ausnahme nicht gefunden.",
+    "ROLLOUT_PLAN_NOT_FOUND": "Rollout-Plan nicht gefunden.",
+    "ROLLOUT_INVALID": "Rollout-Plan ungültig (Feld: {field}).",
+    "ROLLOUT_NAME_TAKEN": "Ein Rollout-Plan mit diesem Namen gibt es bereits.",
+    "ROLLOUT_PLAN_INVALID": "Der Plan kann so nicht starten: {reason}. Vorschau prüfen.",
+    "ROLLOUT_PREVIEW_STALE": "Plan, Gruppen oder Fenster haben sich seit der Vorschau geändert. Vorschau neu laden.",
+    "ROLLOUT_PLAN_BUSY": "Für diesen Plan läuft gerade ein Rollout. Erst abschließen oder abbrechen.",
+    "ROLLOUT_GROUP_IN_USE": "Die Gruppe ist eine Welle eines Rollout-Plans. Erst den Plan ändern.",
+    "ROLLOUT_RUN_NOT_FOUND": "Rollout-Lauf nicht gefunden.",
+    "ROLLOUT_ALREADY_ACTIVE": "Es läuft bereits ein Rollout ({plan}). Es läuft immer nur einer gleichzeitig.",
+    "ROLLOUT_CONFIRM_NAME": "Zur Bestätigung den Namen exakt eingeben.",
+    "ROLLOUT_ACTION_NOT_ALLOWED": "Im Zustand „{status}“ nicht möglich.",
+    "ROLLOUT_HOST_NOT_IN_RUN": "Dieser Host gehört nicht zu diesem Rollout.",
+    "ROLLOUT_HOST_RUNNING": "Auf {host} läuft die Aktion bereits; sie wird geführt zu Ende gebracht und kann nicht übersprungen werden.",
+    "ROLLOUT_HOST_FAILED": "Fehlgeschlagen: {hosts}. Erst prüfen und den Host überspringen, dann fortsetzen."
   },
   "workflow": {
     "attention": "Fehler prüfen",
@@ -1105,8 +1129,14 @@
       "handover_wait": "wartet auf Übernahme durch den Partner",
       "job_wait": "Auftrag läuft",
       "leave": "schaltet zurück",
-      "leave_wait": "prüft den Normalbetrieb"
-    }
+      "leave_wait": "prüft den Normalbetrieb",
+      "verify": "prüft das Ergebnis"
+    },
+    "autoVerifyOption": "Ergebnis automatisch prüfen und abschließen",
+    "autoVerifyHint": "Nach dem Auftrag prüft OPNexus selbst: Firmware wieder bereit, erwarteter Neustart erfolgt, Version lesbar. Dann ist die Wartung abgeschlossen und sperrt den Host nicht mehr. Ist etwas unklar, bleibt sie offen (Ausgang unklar) und du wirst benachrichtigt.",
+    "rolloutBadge": "Rollout",
+    "openRollout": "Zum Rollout",
+    "autoVerifiedBadge": "automatisch geprüft"
   },
   "restore": {
     "handoffWarning": "OPNexus führt die Wiederherstellung NICHT selbst aus. Nach der Übergabe musst du die Sicherung manuell im OPNsense-Webinterface importieren. Solange dieser Vorgang offen ist, sind Regel-/Alias-/NAT-Pushes und Host-Wartung für diesen Host gesperrt.",
@@ -2846,5 +2876,367 @@
       "forced_down": "Abgeschaltet",
       "unknown": "Unbekannt"
     }
+  },
+  "rollouts": {
+    "title": "Rollouts",
+    "intro": "Firmware-Updates in Wellen: erst eine kleine Gruppe (Canary), dann Pilot, dann der Rest. Zwischen den Wellen wartet OPNexus eine Beobachtungszeit ab und prüft die Gesundheit der aktualisierten Firewalls. Bei jeder Auffälligkeit hält der Rollout an und meldet sich. Jeder Host läuft über die normale geplante Wartung (Backup, CARP-Übergabe, Audit).",
+    "tab": {
+      "runs": "Läufe",
+      "plans": "Pläne",
+      "windows": "Wartungsfenster"
+    },
+    "loadFailed": "Rollouts konnten nicht geladen werden",
+    "actionFailed": "Aktion fehlgeschlagen",
+    "noRuns": "Noch keine Rollouts",
+    "noRunsHint": "Lege unter „Pläne“ Wellen aus Gerätegruppen an und starte einen Plan.",
+    "col": {
+      "plan": "Plan",
+      "status": "Status",
+      "wave": "Welle",
+      "target": "Zielversion",
+      "started": "Gestartet",
+      "hosts": "Hosts",
+      "host": "Host",
+      "version": "Version",
+      "step": "Schritt",
+      "note": "Hinweis"
+    },
+    "details": "Details",
+    "waveOf": "{current}/{total}",
+    "by": "von {user}",
+    "trigger": {
+      "manual": "von Hand",
+      "window": "automatisch im Fenster"
+    },
+    "action": {
+      "update": "Stabiles Update",
+      "major_update": "Major-Upgrade"
+    },
+    "run": {
+      "status": {
+        "waiting_window": "wartet auf Wartungsfenster",
+        "preflight": "Vorprüfung",
+        "wave_running": "Welle läuft",
+        "soaking": "Beobachtungszeit",
+        "gate": "Gesundheitsprüfung",
+        "awaiting_approval": "wartet auf Freigabe",
+        "resuming": "prüft vor dem Fortsetzen",
+        "paused": "angehalten",
+        "succeeded": "abgeschlossen",
+        "cancelled": "abgebrochen"
+      }
+    },
+    "wave": {
+      "status": {
+        "pending": "offen",
+        "awaiting_approval": "wartet auf Freigabe",
+        "preflight": "Vorprüfung",
+        "running": "läuft",
+        "soaking": "Beobachtung",
+        "gate": "Prüfung",
+        "passed": "bestanden",
+        "skipped": "nichts zu tun",
+        "cancelled": "abgebrochen"
+      }
+    },
+    "host": {
+      "status": {
+        "queued": "wartet",
+        "running": "läuft",
+        "verifying": "wird geprüft",
+        "succeeded": "fertig",
+        "skipped": "übersprungen",
+        "failed": "fehlgeschlagen",
+        "blocked": "blockiert",
+        "cancelled": "abgebrochen"
+      }
+    },
+    "hostNote": {
+      "verified": "geprüft: Version {version}",
+      "host_failed": "Fehler: {error}",
+      "schedule_missing": "Geplante Wartung fehlt",
+      "host_deleted": "Host gelöscht",
+      "host_disabled": "Host deaktiviert",
+      "no_update_offered": "kein passendes Update angeboten (Status: {status})",
+      "locked": "andere Wartung offen",
+      "firmware_busy": "Firmware-Auftrag läuft",
+      "offline": "nicht erreichbar",
+      "firmware_status": "Firmware-Status unklar ({status})",
+      "not_fired": "nicht gestartet, wieder eingereiht (auf der Firewall ist nichts passiert)",
+      "skipped_by": "übersprungen von {user}: {reason}"
+    },
+    "pauseReason": {
+      "manual": "von Hand angehalten",
+      "gate_failed": "Gesundheitsprüfung nicht bestanden",
+      "cluster_gate_failed": "Cluster nicht gesund – nächster Knoten wartet",
+      "host_failed": "Update eines Hosts fehlgeschlagen",
+      "host_timeout": "Host braucht ungewöhnlich lange",
+      "host_not_started": "Geplante Wartung startete nicht",
+      "preflight_blocked": "Vorprüfung: Hosts nicht bereit",
+      "target_version_changed": "Angebotene Version weicht von der Zielversion ab",
+      "master_needs_handover": "CARP-MASTER braucht die Übergabe (in der Welle aus)",
+      "blocked_by_maintenance": "Andere Wartung im Cluster offen",
+      "actor_invalid": "Verantwortliche Person darf nicht mehr ausrollen",
+      "window_missing": "Wartungsfenster gelöscht",
+      "run_too_old": "Lauf älter als 14 Tage",
+      "internal_error": "Interner Fehler – angehalten",
+      "aborted": "abgebrochen"
+    },
+    "pauseHint": "Ursache prüfen und beheben, dann „Fortsetzen“: OPNexus prüft zuerst alle bereits aktualisierten Hosts erneut. Ein fehlgeschlagener Host muss vorher ausdrücklich übersprungen werden.",
+    "gate": {
+      "title": "Gesundheitsprüfung",
+      "hint": "Vergleich mit dem Zustand vor der Welle. Rot hält den Rollout an (nach einer Nachprüfung 2 Minuten später), Gelb ist nur ein Hinweis.",
+      "verdict": {
+        "ok": "bestanden",
+        "warn": "bestanden mit Hinweisen",
+        "fail": "nicht bestanden"
+      },
+      "resume": "Prüfung vor dem Fortsetzen",
+      "cluster": "Cluster-Prüfung vor dem nächsten Knoten",
+      "result": {
+        "ok": "ok",
+        "warn": "Hinweis",
+        "fail": "Fehler",
+        "skip": "–"
+      },
+      "check": {
+        "online": "Erreichbar",
+        "version": "Version",
+        "carp": "CARP",
+        "gateways": "Gateways",
+        "services": "Dienste",
+        "locks": "Sperren",
+        "compliance": "Sicherheitslage"
+      },
+      "code": {
+        "ok": "in Ordnung",
+        "disabled": "Host deaktiviert",
+        "offline": "nicht erreichbar",
+        "firmware_busy": "Firmware-Auftrag läuft noch",
+        "no_target": "keine Zielversion bekannt",
+        "unreadable": "nicht lesbar (vorher schon)",
+        "unreadable_both": "nicht lesbar (API-Recht fehlt?) – nicht verglichen",
+        "version_mismatch": "erwartet {expected}, installiert {actual}",
+        "no_cluster": "kein Cluster",
+        "node_offline": "Knoten nicht erreichbar: {names}",
+        "maintenance_mode": "im CARP-Wartungsmodus: {names}",
+        "auto_revert_active": "CARP-Zeitfenster aktiv: {names}",
+        "vhid_roles": "VHID ohne genau einen MASTER: {vhids}",
+        "no_master": "kein MASTER erkannt",
+        "no_baseline": "kein Vergleichswert von vorher",
+        "gateway_down": "Gateway ausgefallen: {names}",
+        "gateway_missing": "Gateway fehlt: {names}",
+        "gateway_degraded": "Gateway mit Verlust/Latenz: {names}",
+        "gateway_bad_before": "schon vorher gestört: {names}",
+        "service_stopped": "läuft nicht mehr: {names}",
+        "open_locks": "{count} offene Sperre(n) im Host/Cluster",
+        "score_lower": "Score {before} → {after}"
+      }
+    },
+    "actions": {
+      "pause": "Anhalten",
+      "resume": "Fortsetzen",
+      "approve": "Welle freigeben",
+      "skip": "Überspringen",
+      "abort": "Abbrechen",
+      "pauseReason": "Grund (optional)",
+      "skipTitle": "{host} überspringen",
+      "skipHint": "Der Host bleibt auf seinem Stand und wird in diesem Rollout nicht mehr angefasst. Eine fehlgeschlagene Wartung bleibt offen, bis du sie auf der Host-Seite abschließt.",
+      "reason": "Begründung",
+      "abortTitle": "Rollout abbrechen",
+      "abortHint": "Noch nicht begonnene Hosts werden nicht mehr aktualisiert. Eine Aktion, die schon auf einer Firewall läuft, wird geführt zu Ende gebracht (nicht abgewürgt).",
+      "confirmPlan": "Zur Bestätigung den Plannamen eingeben",
+      "paused": "Rollout angehalten",
+      "resumed": "Wird geprüft und fortgesetzt",
+      "approved": "Welle freigegeben",
+      "skipped": "Host übersprungen",
+      "aborted": "Rollout abgebrochen"
+    },
+    "detail": {
+      "back": "Alle Rollouts",
+      "notFound": "Rollout nicht gefunden",
+      "soakUntil": "Beobachtung bis {time}",
+      "recheckAt": "Nachprüfung um {time}",
+      "blockedSince": "wartet seit {time} auf eine fremde Wartung (max. 15 min)",
+      "waves": "Wellen",
+      "hosts": "Hosts",
+      "handover": "mit CARP-Übergabe",
+      "approval": "Freigabe nötig",
+      "approvedBy": "freigegeben von {user}",
+      "soak": "{minutes} min Beobachtung",
+      "noGate": "Noch keine Prüfung",
+      "attempt": "Versuch {n}",
+      "stillRunning": "läuft noch zu Ende: {hosts}",
+      "waitingWindow": "Neue Hosts starten erst im nächsten Wartungsfenster.",
+      "awaiting": "Die nächste Welle wartet auf Freigabe.",
+      "owner": "verantwortlich"
+    },
+    "banner": {
+      "active": "Rollout „{plan}“ – Welle {wave} ({label}): {status}",
+      "open": "Ansehen"
+    },
+    "inRollout": "im Rollout",
+    "inRolloutTip": "Dieser Host gehört zum laufenden Rollout. Manuelle Wartung ist hier gesperrt, bis der Rollout ihn erledigt oder übersprungen hat.",
+    "updaterWarning": "Ein Firmware-Rollout ist aktiv. Ein Neustart von OPNexus ist sicher (der Zustand liegt in der Datenbank), hält den Rollout aber für die Dauer an.",
+    "plan": {
+      "new": "Neuer Plan",
+      "edit": "Plan bearbeiten",
+      "none": "Noch keine Pläne",
+      "noneHint": "Lege Gerätegruppen als Ringe an (z. B. ring-canary, ring-pilot, ring-prod) und baue daraus Wellen.",
+      "name": "Name",
+      "description": "Beschreibung",
+      "action": "Aktion",
+      "window": "Wartungsfenster",
+      "noWindow": "keins (Hosts starten sofort)",
+      "autoStart": "Automatisch zu jedem Fenstertermin starten",
+      "autoStartHint": "Nur für stabile Updates. Wirkt erst nach „Aktivieren“ und nur, solange Plan, Gruppen und Fenster unverändert sind; jede Änderung deaktiviert den Plan.",
+      "waves": "Wellen",
+      "addWave": "Welle hinzufügen",
+      "waveLabel": "Bezeichnung",
+      "groups": "Gerätegruppen",
+      "soak": "Beobachtungszeit (min)",
+      "soakHint": "So lange nach dem letzten Host der Welle warten, bevor die Gesundheit geprüft wird.",
+      "handover": "CARP-Übergabe für MASTER-Knoten",
+      "approval": "Vor dieser Welle auf Freigabe warten",
+      "moveUp": "Welle nach oben",
+      "moveDown": "Welle nach unten",
+      "removeWave": "Welle entfernen",
+      "checks": "Gesundheitsprüfung",
+      "checksHint": "Erreichbarkeit, Version und Sperren werden immer geprüft.",
+      "complianceMode": "Niedrigerer Sicherheits-Score",
+      "complianceWarn": "nur Hinweis",
+      "complianceFail": "hält an",
+      "save": "Speichern",
+      "saved": "Plan gespeichert",
+      "deleted": "Plan gelöscht",
+      "deleteConfirm": "Plan „{name}“ löschen? Laufhistorie bleibt erhalten.",
+      "previewStart": "Vorschau & Start",
+      "activate": "Aktivieren",
+      "deactivate": "Deaktivieren",
+      "active": "automatisch aktiv",
+      "inactive": "automatisch aus",
+      "changed": "geändert seit Aktivierung",
+      "changedTip": "Plan, Gruppen oder Fenster haben sich seit der Aktivierung geändert. Der automatische Start ist angehalten, bis der Plan erneut aktiviert wird.",
+      "activatedBy": "aktiviert von {user}",
+      "lastAutoStartError": "Letzter automatischer Start ausgeblieben ({time}): {reason}",
+      "hostsCount": "{count} Hosts",
+      "errors": "{count} Fehler",
+      "warnings": "{count} Hinweise",
+      "nextOccurrence": "Nächster Termin: {time}",
+      "previewTitle": "Vorschau: {name}",
+      "unattended": "Ich weiß, dass die Firewalls dann ohne weitere Rückfrage aktualisiert und neu gestartet werden; der Rollout hält bei jeder Auffälligkeit selbst an.",
+      "start": "Rollout starten",
+      "started": "Rollout gestartet",
+      "activated": "Automatischer Start aktiviert",
+      "deactivated": "Automatischer Start deaktiviert",
+      "emptyWave": "keine Hosts",
+      "disabledHost": "deaktiviert",
+      "cannotStart": "So nicht startbar – siehe Fehler oben."
+    },
+    "issue": {
+      "no_waves": "Keine Wellen",
+      "no_hosts": "Keine Hosts in den Gruppen",
+      "too_many_waves": "Mehr als {limit} Wellen",
+      "too_many_hosts": "Mehr als {limit} Hosts",
+      "cluster_split": "Cluster {cluster}: Für ein Major-Upgrade müssen alle Knoten in derselben Welle liegen.",
+      "cluster_partial": "Cluster {cluster}: Nicht alle Knoten sind im Plan.",
+      "mixed_versions": "Cluster {cluster}: Knoten in verschiedenen Wellen – gemischte Versionen bis Welle {wave}.",
+      "master_needs_handover": "Welle {wave} enthält Cluster ({clusters}) ohne CARP-Übergabe: beim MASTER hält der Rollout an.",
+      "empty_wave": "Welle {wave} hat keine Hosts.",
+      "duplicate_hosts": "In mehreren Wellen (die früheste gilt): {names}",
+      "disabled_hosts": "Deaktiviert, werden übersprungen: {names}",
+      "auto_start_update_only": "Automatischer Start nur für stabile Updates.",
+      "auto_start_needs_window": "Automatischer Start braucht ein Wartungsfenster.",
+      "no_window": "Kein Wartungsfenster: Hosts starten sofort nacheinander.",
+      "window_disabled": "Das Wartungsfenster ist ausgeschaltet: nichts startet.",
+      "plan_changed": "Plan, Gruppen oder Fenster seit der Aktivierung geändert",
+      "actor_invalid": "{user} darf nicht mehr ausrollen",
+      "plan_invalid": "Plan ungültig ({reason})",
+      "other_run_active": "anderer Rollout läuft ({plan})"
+    },
+    "detailKey": {
+      "wave": "Welle",
+      "failed": "Nicht bestanden",
+      "host": "Host",
+      "hosts": "Hosts",
+      "cluster": "Cluster",
+      "expected": "Erwartet",
+      "offered": "Angeboten",
+      "user": "Benutzer",
+      "by": "Von",
+      "text": "Grund",
+      "reason": "Grund",
+      "error": "Fehler",
+      "days": "Tage",
+      "still_running": "Läuft noch"
+    }
+  },
+  "mwindows": {
+    "none": "Noch keine Wartungsfenster",
+    "noneHint": "Ein Fenster legt fest, wann ein Rollout neue Hosts beginnen darf, z. B. sonntags 02:00–05:00.",
+    "new": "Neues Fenster",
+    "edit": "Fenster bearbeiten",
+    "name": "Name",
+    "timezone": "Zeitzone",
+    "recurrence": "Wiederholung",
+    "weekly": "wöchentlich",
+    "monthly": "monatlich",
+    "weekdays": "Wochentage",
+    "monthMode": "Monatlich am",
+    "nthWeekday": "n-ten Wochentag",
+    "dayOfMonth": "Tag des Monats",
+    "nth": {
+      "1": "ersten",
+      "2": "zweiten",
+      "3": "dritten",
+      "4": "vierten",
+      "-1": "letzten"
+    },
+    "day": "Tag (1–28)",
+    "startTime": "Beginn (Ortszeit)",
+    "duration": "Dauer (min)",
+    "cutoff": "Neue Hosts nur in den ersten … min",
+    "cutoffHint": "Danach beginnt kein neuer Host mehr; laufende Updates laufen zu Ende. So passt ein Update samt Neustart sicher ins Fenster.",
+    "enabled": "eingeschaltet",
+    "dstHint": "Sommerzeit: Eine Uhrzeit, die es an dem Tag nicht gibt, rückt vor; eine doppelte gilt beim ersten Mal.",
+    "saved": "Fenster gespeichert",
+    "deleted": "Fenster gelöscht",
+    "deleteConfirm": "Fenster „{name}“ löschen?",
+    "next": "Nächste Termine",
+    "openNow": "gerade offen",
+    "off": "ausgeschaltet",
+    "skipped": "ausgelassen",
+    "moved": "verschoben",
+    "skip": "Auslassen",
+    "move": "Verschieben",
+    "moveTo": "Neuer Beginn",
+    "exceptions": "Ausnahmen",
+    "removeException": "Ausnahme entfernen",
+    "exceptionSaved": "Ausnahme gespeichert",
+    "usedBy": "Genutzt von: {plans}",
+    "summary": {
+      "weekly": "{days}, {time} ({tz}), {duration} min",
+      "nth": "am {nth} {day} im Monat, {time} ({tz}), {duration} min",
+      "day": "am {day}. jedes Monats, {time} ({tz}), {duration} min"
+    },
+    "dayName": {
+      "0": "Mo",
+      "1": "Di",
+      "2": "Mi",
+      "3": "Do",
+      "4": "Fr",
+      "5": "Sa",
+      "6": "So"
+    },
+    "dayLong": {
+      "0": "Montag",
+      "1": "Dienstag",
+      "2": "Mittwoch",
+      "3": "Donnerstag",
+      "4": "Freitag",
+      "5": "Samstag",
+      "6": "Sonntag"
+    },
+    "cutoffSummary": "neue Hosts nur in den ersten {minutes} min"
   }
 }

frontend/lib/i18n/en.json

@@ -15,7 +15,8 @@
     "saving": "Saving…",
     "deleteFailed": "Deleting failed",
     "description": "Description",
-    "versionLabel": "OPNexus {version} “{codename}”"
+    "versionLabel": "OPNexus {version} “{codename}”",
+    "delete": "Delete"
   },
   "settings": {
     "language": {
@@ -69,7 +70,8 @@
         "networkDrift": "Network differences in the cluster",
         "gatewayDown": "Gateway failing (offline, packet loss or latency above the limit)",
         "resourceThreshold": "Resource threshold exceeded (disk, memory, swap, CPU load; off by default)",
-        "serviceDown": "Watched service stopped (off by default)"
+        "serviceDown": "Watched service stopped (off by default)",
+        "rolloutProblems": "Rollout paused or not started automatically"
       },
       "pollInterval": "Check interval (seconds)",
       "pollHint": "At least 60s, so the firewalls are not loaded unnecessarily.",
@@ -297,7 +299,8 @@
       "hostsystem-syslog": "Syslog",
       "hostsystem-snapshots": "Snapshots",
       "hostsystem-diagnostics": "Diagnostics",
-      "setup": "Setup"
+      "setup": "Setup",
+      "rollouts": "Rollouts"
     }
   },
   "dashboard": {
@@ -763,7 +766,28 @@
     "NOTIFICATION_CHANNEL_NOT_FOUND": "The notification channel was not found.",
     "NOTIFICATION_CHANNEL_NAME_TAKEN": "A channel with this name already exists.",
     "NOTIFICATION_CHANNEL_TEST_FAILED": "Test message not delivered: {reason}",
-    "NOTIFICATION_WEBHOOK_URL_INVALID": "Invalid webhook address: {reason}"
+    "NOTIFICATION_WEBHOOK_URL_INVALID": "Invalid webhook address: {reason}",
+    "WINDOW_NOT_FOUND": "Maintenance window not found.",
+    "WINDOW_INVALID": "Invalid maintenance window (field: {field}).",
+    "WINDOW_NAME_TAKEN": "A maintenance window with this name already exists.",
+    "WINDOW_IN_USE": "A rollout plan uses this window. Change or delete the plan first.",
+    "WINDOW_EXCEPTION_INVALID": "Invalid exception (field: {field}). Only future occurrences of the window can be skipped or moved.",
+    "WINDOW_EXCEPTION_EXISTS": "This occurrence already has an exception.",
+    "WINDOW_EXCEPTION_NOT_FOUND": "Exception not found.",
+    "ROLLOUT_PLAN_NOT_FOUND": "Rollout plan not found.",
+    "ROLLOUT_INVALID": "Invalid rollout plan (field: {field}).",
+    "ROLLOUT_NAME_TAKEN": "A rollout plan with this name already exists.",
+    "ROLLOUT_PLAN_INVALID": "The plan cannot start like this: {reason}. Check the preview.",
+    "ROLLOUT_PREVIEW_STALE": "The plan, its groups or its window changed since the preview. Reload the preview.",
+    "ROLLOUT_PLAN_BUSY": "A rollout of this plan is in progress. Finish or abort it first.",
+    "ROLLOUT_GROUP_IN_USE": "The group is a wave of a rollout plan. Change the plan first.",
+    "ROLLOUT_RUN_NOT_FOUND": "Rollout run not found.",
+    "ROLLOUT_ALREADY_ACTIVE": "A rollout is already in progress ({plan}). Only one runs at a time.",
+    "ROLLOUT_CONFIRM_NAME": "Type the exact name to confirm.",
+    "ROLLOUT_ACTION_NOT_ALLOWED": "Not possible while the run is “{status}”.",
+    "ROLLOUT_HOST_NOT_IN_RUN": "This host is not part of this rollout.",
+    "ROLLOUT_HOST_RUNNING": "The action on {host} is already running; it is finished in a controlled way and cannot be skipped.",
+    "ROLLOUT_HOST_FAILED": "Failed: {hosts}. Check it and skip the host first, then resume."
   },
   "workflow": {
     "attention": "Check errors",
@@ -1105,8 +1129,14 @@
       "handover_wait": "waiting for the partner to take over",
       "job_wait": "job running",
       "leave": "switching back",
-      "leave_wait": "verifying normal operation"
-    }
+      "leave_wait": "verifying normal operation",
+      "verify": "checking the result"
+    },
+    "autoVerifyOption": "Check the result automatically and close the operation",
+    "autoVerifyHint": "After the job OPNexus checks by itself: firmware idle again, the expected reboot happened, version readable. Then the maintenance is closed and no longer locks the host. If anything is unclear it stays open (outcome unclear) and you are notified.",
+    "rolloutBadge": "Rollout",
+    "openRollout": "Open rollout",
+    "autoVerifiedBadge": "checked automatically"
   },
   "restore": {
     "handoffWarning": "OPNexus does NOT perform the restore itself. After the handover you must import the backup manually in the OPNsense web interface. While this operation is open, rule/alias/NAT pushes and host maintenance are blocked for this host.",
@@ -2846,5 +2876,367 @@
       "forced_down": "Switched off",
       "unknown": "Unknown"
     }
+  },
+  "rollouts": {
+    "title": "Rollouts",
+    "intro": "Firmware updates in waves: a small group first (canary), then pilot, then the rest. Between waves OPNexus waits a soak time and checks the health of the updated firewalls. On anything unusual the rollout stops and notifies you. Each host goes through the normal scheduled maintenance (backup, CARP handover, audit).",
+    "tab": {
+      "runs": "Runs",
+      "plans": "Plans",
+      "windows": "Maintenance windows"
+    },
+    "loadFailed": "Could not load rollouts",
+    "actionFailed": "Action failed",
+    "noRuns": "No rollouts yet",
+    "noRunsHint": "Create waves from device groups under “Plans” and start a plan.",
+    "col": {
+      "plan": "Plan",
+      "status": "Status",
+      "wave": "Wave",
+      "target": "Target version",
+      "started": "Started",
+      "hosts": "Hosts",
+      "host": "Host",
+      "version": "Version",
+      "step": "Step",
+      "note": "Note"
+    },
+    "details": "Details",
+    "waveOf": "{current}/{total}",
+    "by": "by {user}",
+    "trigger": {
+      "manual": "manual",
+      "window": "automatic in window"
+    },
+    "action": {
+      "update": "Stable update",
+      "major_update": "Major upgrade"
+    },
+    "run": {
+      "status": {
+        "waiting_window": "waiting for window",
+        "preflight": "preflight",
+        "wave_running": "wave running",
+        "soaking": "soaking",
+        "gate": "health gate",
+        "awaiting_approval": "awaiting approval",
+        "resuming": "checking before resuming",
+        "paused": "paused",
+        "succeeded": "completed",
+        "cancelled": "aborted"
+      }
+    },
+    "wave": {
+      "status": {
+        "pending": "pending",
+        "awaiting_approval": "awaiting approval",
+        "preflight": "preflight",
+        "running": "running",
+        "soaking": "soaking",
+        "gate": "gate",
+        "passed": "passed",
+        "skipped": "nothing to do",
+        "cancelled": "aborted"
+      }
+    },
+    "host": {
+      "status": {
+        "queued": "queued",
+        "running": "running",
+        "verifying": "verifying",
+        "succeeded": "done",
+        "skipped": "skipped",
+        "failed": "failed",
+        "blocked": "blocked",
+        "cancelled": "aborted"
+      }
+    },
+    "hostNote": {
+      "verified": "verified: version {version}",
+      "host_failed": "Error: {error}",
+      "schedule_missing": "Scheduled maintenance missing",
+      "host_deleted": "Host deleted",
+      "host_disabled": "Host disabled",
+      "no_update_offered": "no matching update offered (status: {status})",
+      "locked": "other maintenance open",
+      "firmware_busy": "firmware job running",
+      "offline": "unreachable",
+      "firmware_status": "firmware status unclear ({status})",
+      "not_fired": "not started, queued again (nothing happened on the firewall)",
+      "skipped_by": "skipped by {user}: {reason}"
+    },
+    "pauseReason": {
+      "manual": "paused by hand",
+      "gate_failed": "health gate failed",
+      "cluster_gate_failed": "cluster not healthy – next node waits",
+      "host_failed": "a host update failed",
+      "host_timeout": "host is taking unusually long",
+      "host_not_started": "scheduled maintenance did not start",
+      "preflight_blocked": "preflight: hosts not ready",
+      "target_version_changed": "offered version differs from the target version",
+      "master_needs_handover": "CARP MASTER needs the handover (off in this wave)",
+      "blocked_by_maintenance": "other maintenance open in the cluster",
+      "actor_invalid": "the responsible person may no longer roll out",
+      "window_missing": "maintenance window deleted",
+      "run_too_old": "run older than 14 days",
+      "internal_error": "internal error – paused",
+      "aborted": "aborted"
+    },
+    "pauseHint": "Check and fix the cause, then “Resume”: OPNexus first re-checks every host updated so far. A failed host must be skipped explicitly first.",
+    "gate": {
+      "title": "Health gate",
+      "hint": "Compared with the state before the wave. Red stops the rollout (after one re-check 2 minutes later), yellow is only a hint.",
+      "verdict": {
+        "ok": "passed",
+        "warn": "passed with hints",
+        "fail": "failed"
+      },
+      "resume": "Check before resuming",
+      "cluster": "Cluster check before the next node",
+      "result": {
+        "ok": "ok",
+        "warn": "hint",
+        "fail": "fail",
+        "skip": "–"
+      },
+      "check": {
+        "online": "Online",
+        "version": "Version",
+        "carp": "CARP",
+        "gateways": "Gateways",
+        "services": "Services",
+        "locks": "Locks",
+        "compliance": "Compliance"
+      },
+      "code": {
+        "ok": "fine",
+        "disabled": "host disabled",
+        "offline": "unreachable",
+        "firmware_busy": "firmware job still running",
+        "no_target": "no target version known",
+        "unreadable": "unreadable",
+        "unreadable_both": "unreadable (API permission missing?) – not compared",
+        "version_mismatch": "expected {expected}, installed {actual}",
+        "no_cluster": "no cluster",
+        "node_offline": "node unreachable: {names}",
+        "maintenance_mode": "in CARP maintenance mode: {names}",
+        "auto_revert_active": "CARP timer active: {names}",
+        "vhid_roles": "VHID without exactly one MASTER: {vhids}",
+        "no_master": "no MASTER found",
+        "no_baseline": "no earlier value to compare",
+        "gateway_down": "gateway down: {names}",
+        "gateway_missing": "gateway missing: {names}",
+        "gateway_degraded": "gateway with loss/latency: {names}",
+        "gateway_bad_before": "already bad before: {names}",
+        "service_stopped": "no longer running: {names}",
+        "open_locks": "{count} open lock(s) in host/cluster",
+        "score_lower": "score {before} → {after}"
+      }
+    },
+    "actions": {
+      "pause": "Pause",
+      "resume": "Resume",
+      "approve": "Release wave",
+      "skip": "Skip",
+      "abort": "Abort",
+      "pauseReason": "Reason (optional)",
+      "skipTitle": "Skip {host}",
+      "skipHint": "The host stays as it is and is not touched again in this rollout. A failed maintenance stays open until you close it on the host page.",
+      "reason": "Reason",
+      "abortTitle": "Abort rollout",
+      "abortHint": "Hosts that have not started are no longer updated. An action already running on a firewall is finished in a controlled way (not cut off).",
+      "confirmPlan": "Type the plan name to confirm",
+      "paused": "Rollout paused",
+      "resumed": "Checking and resuming",
+      "approved": "Wave released",
+      "skipped": "Host skipped",
+      "aborted": "Rollout aborted"
+    },
+    "detail": {
+      "back": "All rollouts",
+      "notFound": "Rollout not found",
+      "soakUntil": "Soaking until {time}",
+      "recheckAt": "Re-check at {time}",
+      "blockedSince": "waiting since {time} for someone else's maintenance (max. 15 min)",
+      "waves": "Waves",
+      "hosts": "Hosts",
+      "handover": "with CARP handover",
+      "approval": "approval needed",
+      "approvedBy": "released by {user}",
+      "soak": "{minutes} min soak",
+      "noGate": "No check yet",
+      "attempt": "attempt {n}",
+      "stillRunning": "still finishing: {hosts}",
+      "waitingWindow": "New hosts only start in the next maintenance window.",
+      "awaiting": "The next wave is waiting for release.",
+      "owner": "responsible"
+    },
+    "banner": {
+      "active": "Rollout “{plan}” – wave {wave} ({label}): {status}",
+      "open": "View"
+    },
+    "inRollout": "in rollout",
+    "inRolloutTip": "This host belongs to the running rollout. Manual maintenance is locked here until the rollout has done or skipped it.",
+    "updaterWarning": "A firmware rollout is active. Restarting OPNexus is safe (its state is in the database) but holds the rollout for that time.",
+    "plan": {
+      "new": "New plan",
+      "edit": "Edit plan",
+      "none": "No plans yet",
+      "noneHint": "Create device groups as rings (e.g. ring-canary, ring-pilot, ring-prod) and build waves from them.",
+      "name": "Name",
+      "description": "Description",
+      "action": "Action",
+      "window": "Maintenance window",
+      "noWindow": "none (hosts start right away)",
+      "autoStart": "Start automatically at every window occurrence",
+      "autoStartHint": "Stable updates only. Takes effect after “Activate” and only while plan, groups and window are unchanged; any change deactivates the plan.",
+      "waves": "Waves",
+      "addWave": "Add wave",
+      "waveLabel": "Label",
+      "groups": "Device groups",
+      "soak": "Soak time (min)",
+      "soakHint": "How long to wait after the wave's last host before checking health.",
+      "handover": "CARP handover for MASTER nodes",
+      "approval": "Wait for release before this wave",
+      "moveUp": "Move wave up",
+      "moveDown": "Move wave down",
+      "removeWave": "Remove wave",
+      "checks": "Health gate",
+      "checksHint": "Reachability, version and locks are always checked.",
+      "complianceMode": "Lower security score",
+      "complianceWarn": "hint only",
+      "complianceFail": "stops",
+      "save": "Save",
+      "saved": "Plan saved",
+      "deleted": "Plan deleted",
+      "deleteConfirm": "Delete plan “{name}”? Run history is kept.",
+      "previewStart": "Preview & start",
+      "activate": "Activate",
+      "deactivate": "Deactivate",
+      "active": "automatic: on",
+      "inactive": "automatic: off",
+      "changed": "changed since activation",
+      "changedTip": "The plan, its groups or its window changed since activation. The automatic start is held until the plan is activated again.",
+      "activatedBy": "activated by {user}",
+      "lastAutoStartError": "Last automatic start did not happen ({time}): {reason}",
+      "hostsCount": "{count} hosts",
+      "errors": "{count} errors",
+      "warnings": "{count} hints",
+      "nextOccurrence": "Next occurrence: {time}",
+      "previewTitle": "Preview: {name}",
+      "unattended": "I understand that the firewalls will then be updated and rebooted without further confirmation; the rollout stops by itself on anything unusual.",
+      "start": "Start rollout",
+      "started": "Rollout started",
+      "activated": "Automatic start activated",
+      "deactivated": "Automatic start deactivated",
+      "emptyWave": "no hosts",
+      "disabledHost": "disabled",
+      "cannotStart": "Cannot start like this – see the errors above."
+    },
+    "issue": {
+      "no_waves": "No waves",
+      "no_hosts": "No hosts in the groups",
+      "too_many_waves": "More than {limit} waves",
+      "too_many_hosts": "More than {limit} hosts",
+      "cluster_split": "Cluster {cluster}: for a major upgrade all nodes must be in the same wave.",
+      "cluster_partial": "Cluster {cluster}: not all nodes are in the plan.",
+      "mixed_versions": "Cluster {cluster}: nodes in different waves – mixed versions until wave {wave}.",
+      "master_needs_handover": "Wave {wave} contains clusters ({clusters}) without CARP handover: the rollout stops at the MASTER.",
+      "empty_wave": "Wave {wave} has no hosts.",
+      "duplicate_hosts": "In several waves (the earliest counts): {names}",
+      "disabled_hosts": "Disabled, will be skipped: {names}",
+      "auto_start_update_only": "Automatic start only for stable updates.",
+      "auto_start_needs_window": "Automatic start needs a maintenance window.",
+      "no_window": "No maintenance window: hosts start one after the other right away.",
+      "window_disabled": "The maintenance window is switched off: nothing starts.",
+      "plan_changed": "plan, groups or window changed since activation",
+      "actor_invalid": "{user} may no longer roll out",
+      "plan_invalid": "plan invalid ({reason})",
+      "other_run_active": "another rollout is running ({plan})"
+    },
+    "detailKey": {
+      "wave": "Wave",
+      "failed": "Failed",
+      "host": "Host",
+      "hosts": "Hosts",
+      "cluster": "Cluster",
+      "expected": "Expected",
+      "offered": "Offered",
+      "user": "User",
+      "by": "By",
+      "text": "Reason",
+      "reason": "Reason",
+      "error": "Error",
+      "days": "Days",
+      "still_running": "Still running"
+    }
+  },
+  "mwindows": {
+    "none": "No maintenance windows yet",
+    "noneHint": "A window defines when a rollout may start new hosts, e.g. Sundays 02:00–05:00.",
+    "new": "New window",
+    "edit": "Edit window",
+    "name": "Name",
+    "timezone": "Time zone",
+    "recurrence": "Recurrence",
+    "weekly": "weekly",
+    "monthly": "monthly",
+    "weekdays": "Weekdays",
+    "monthMode": "Monthly on",
+    "nthWeekday": "nth weekday",
+    "dayOfMonth": "day of month",
+    "nth": {
+      "1": "first",
+      "2": "second",
+      "3": "third",
+      "4": "fourth",
+      "-1": "last"
+    },
+    "day": "Day (1–28)",
+    "startTime": "Start (local time)",
+    "duration": "Duration (min)",
+    "cutoff": "New hosts only in the first … min",
+    "cutoffHint": "After that no new host starts; running updates finish. This way an update including reboot fits into the window.",
+    "enabled": "enabled",
+    "dstHint": "Daylight saving: a time that does not exist that day moves forward; a time that exists twice counts the first time.",
+    "saved": "Window saved",
+    "deleted": "Window deleted",
+    "deleteConfirm": "Delete window “{name}”?",
+    "next": "Next occurrences",
+    "openNow": "open now",
+    "off": "switched off",
+    "skipped": "skipped",
+    "moved": "moved",
+    "skip": "Skip",
+    "move": "Move",
+    "moveTo": "New start",
+    "exceptions": "Exceptions",
+    "removeException": "Remove exception",
+    "exceptionSaved": "Exception saved",
+    "usedBy": "Used by: {plans}",
+    "summary": {
+      "weekly": "{days}, {time} ({tz}), {duration} min",
+      "nth": "on the {nth} {day} of the month, {time} ({tz}), {duration} min",
+      "day": "on day {day} of every month, {time} ({tz}), {duration} min"
+    },
+    "dayName": {
+      "0": "Mon",
+      "1": "Tue",
+      "2": "Wed",
+      "3": "Thu",
+      "4": "Fri",
+      "5": "Sat",
+      "6": "Sun"
+    },
+    "dayLong": {
+      "0": "Monday",
+      "1": "Tuesday",
+      "2": "Wednesday",
+      "3": "Thursday",
+      "4": "Friday",
+      "5": "Saturday",
+      "6": "Sunday"
+    },
+    "cutoffSummary": "new hosts only in the first {minutes} min"
   }
 }

frontend/lib/i18n/serverMessages.ts

@@ -119,6 +119,8 @@ export const serverMessagesEn: Array<[string | RegExp, string]> = [
   [/^Manuell abgebrochen, kein bestätigter Abschluss\. Grund: (?<reason>[\s\S]*)$/, "Manually cancelled, no confirmed completion. Reason: {reason}"],
   ["Unerwarteter Fehler beim Ausführen", "Unexpected error during execution"],
   ["Host nicht gefunden oder deaktiviert", "Host not found or disabled"],
+  ["Rollout-Schritt nicht rechtzeitig gestartet, nichts ausgeführt", "Rollout step not started in time, nothing was run"],
+  ["Rollout angehalten, nicht ausgeführt", "Rollout paused, not run"],
   [/^Host nicht erreichbar: (?<detail>[\s\S]*)$/, "Host unreachable: {detail}"],
   [/^Automatisches Backup vor dem Update fehlgeschlagen: (?<detail>[\s\S]*)$/, "Automatic backup before the update failed: {detail}"],
   [/^Vorbereitung der CARP-Übergabe fehlgeschlagen, nichts wurde verändert: (?<detail>[\s\S]*)$/, "Preparing the CARP handover failed, nothing was changed: {detail}"],

frontend/lib/rollouts.ts

@@ -0,0 +1,78 @@
+// Shared display helpers for firmware rollouts (1.25): status tones and translated codes.
+import type { BadgeTone } from "@/components/ui/Badge";
+import type { TranslationParams } from "@/lib/i18n";
+import type { MaintenanceWindow, RolloutCheckResult, RolloutHostStatus, RolloutIssue, RolloutRunStatus } from "@/lib/api";
+
+type Translate = (key: string, params?: TranslationParams) => string;
+
+export const RUN_TONE: Record<RolloutRunStatus, BadgeTone> = {
+  waiting_window: "neutral", preflight: "info", wave_running: "info", soaking: "info", gate: "info",
+  awaiting_approval: "accent", resuming: "info", paused: "warning", succeeded: "success", cancelled: "neutral",
+};
+
+export const HOST_TONE: Record<RolloutHostStatus, BadgeTone> = {
+  queued: "neutral", running: "info", verifying: "info", succeeded: "success", skipped: "neutral",
+  failed: "danger", blocked: "warning", cancelled: "neutral",
+};
+
+export const WAVE_TONE: Record<string, BadgeTone> = {
+  pending: "neutral", awaiting_approval: "accent", preflight: "info", running: "info", soaking: "info", gate: "info",
+  passed: "success", skipped: "neutral", cancelled: "neutral",
+};
+
+export const RESULT_TONE: Record<RolloutCheckResult["result"], BadgeTone> = { ok: "success", warn: "warning", fail: "danger", skip: "neutral" };
+
+export const CHECKS = ["online", "version", "carp", "gateways", "services", "locks", "compliance"] as const;
+
+function stringParams(params: Record<string, unknown> | null | undefined): TranslationParams {
+  const out: TranslationParams = {};
+  for (const [k, v] of Object.entries(params ?? {})) out[k] = v === null || v === undefined ? "" : typeof v === "number" ? v : String(v);
+  return out;
+}
+
+/** A translated text for a {code, params} pair under `prefix`, with a readable fallback for unknown codes. */
+export function codeText(t: Translate, prefix: string, issue: RolloutIssue | RolloutCheckResult | null | undefined): string {
+  if (!issue) return "";
+  const key = `${prefix}.${issue.code}`;
+  const text = t(key, stringParams(issue.params));
+  return text === key ? issue.code : text;
+}
+
+export function pauseText(t: Translate, reason: string | null): string {
+  if (!reason) return "";
+  const key = `rollouts.pauseReason.${reason}`;
+  const text = t(key);
+  return text === key ? reason : text;
+}
+
+function failedItem(t: Translate, item: string): string {
+  const [host, check] = item.split(":");
+  const key = `rollouts.gate.check.${check}`;
+  const label = t(key);
+  return check ? `${host}: ${label === key ? check : label}` : item;
+}
+
+/** Human-readable pause details: known keys translated, failed checks as "host: check". */
+export function pauseDetailText(t: Translate, detail: Record<string, unknown> | null | undefined): string {
+  if (!detail) return "";
+  return Object.entries(detail)
+    .filter(([k, v]) => !["on_resume"].includes(k) && v !== null && v !== "" && !(Array.isArray(v) && v.length === 0))
+    .map(([k, v]) => {
+      const key = `rollouts.detailKey.${k}`;
+      const label = t(key) === key ? k : t(key);
+      const value = k === "failed" && Array.isArray(v) ? v.map((x) => failedItem(t, String(x))).join(", ") : Array.isArray(v) ? v.join(", ") : String(v);
+      return `${label}: ${value}`;
+    })
+    .join(" · ");
+}
+
+export function windowSummary(t: Translate, w: MaintenanceWindow): string {
+  const base = { time: w.start_time, tz: w.timezone, duration: w.duration_minutes };
+  if (w.recurrence === "weekly") {
+    return t("mwindows.summary.weekly", { ...base, days: w.weekdays.map((d) => t(`mwindows.dayName.${d}`)).join(", ") });
+  }
+  if (w.month_mode === "day") return t("mwindows.summary.day", { ...base, day: w.month_day ?? "" });
+  return t("mwindows.summary.nth", { ...base, nth: t(`mwindows.nth.${w.month_nth}`), day: t(`mwindows.dayLong.${w.month_weekday}`) });
+}
+
+export const isActiveRun = (status: RolloutRunStatus) => !["succeeded", "cancelled"].includes(status);

frontend/lib/version.ts

@@ -1,6 +1,6 @@
 // Manually bumped whenever a notable change ships — mirrors the dated
 // entries already kept in the project README/CHANGELOG.md, no build
 // tooling involved. Versioning scheme: see CHANGELOG.md ("Versionierung").
-export const APP_VERSION = "1.24.0";
-export const APP_CODENAME = "Merlin";
+export const APP_VERSION = "1.25.0";
+export const APP_CODENAME = "Goshawk";
 export const APP_MAINTAINER = "Julian Boldt";