Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b1a96c7563 |
No files matched your search
@@ -9,3 +9,4 @@ backend/.pytest_cache/
|
||||
.opnexus-update-state
|
||||
dist/
|
||||
.public-repo/
|
||||
updater/docker-compose.release.yml
|
||||
+46
-1
@@ -29,7 +29,52 @@ in der App auf der Login-Seite, im Footer jeder Seite, auf `/about` und in
|
||||
den Einstellungen.
|
||||
|
||||
Nächste Codenamen (der Reihe nach, wenn die jeweils nächste MINOR-Version
|
||||
ansteht): Saker …
|
||||
ansteht): wird beim nächsten MINOR-Release festgelegt (Saker ist mit 1.33.0 vergeben).
|
||||
|
||||
## [1.34.0] "Griffon" — 2026-10-24
|
||||
|
||||
- **Zum Update:** Das erste Update auf diese Version übernimmt auch die `docker-compose.yml` (siehe unten; braucht wie die
|
||||
CLI-Aktualisierung `cosign` und `cosign.pub`), migriert die Datenbank (Revision 0032; ein Rollback spielt die Sicherung davor
|
||||
zurück) und stellt das Backend auf einen unprivilegierten Benutzer um. Nichts davon braucht einen Handgriff.
|
||||
- **Nur ein Backend führt die Hintergrunddienste aus (HA, Stufe H0):** Laufen zwei Backends an derselben Datenbank, verschickten
|
||||
beide jeden Alarm und brachen Zyklen mit `UniqueViolation` ab. Jetzt führt nur der Prozess mit der Leader-Sperre in der
|
||||
Datenbank die neun Hintergrundschleifen aus; die anderen warten in Bereitschaft und übernehmen innerhalb einer halben Minute,
|
||||
wenn der Leader ausfällt. Verliert der Leader seine Datenbankverbindung, hält er alle Schleifen sofort an (geprüft alle 5 Sekunden
|
||||
und vor jedem Durchlauf). An einer Replica startet kein Leader. Rolle in `GET /api/health` und als Metrik `opnexus_leader`.
|
||||
Zwei gleichzeitige Erststarts legen das Admin-Konto nur einmal an.
|
||||
- **Umzug und Kaltreserve (HA, Stufe H1):** `opnexus backup --full --with-env --with-ca` nimmt zusätzlich die `.env` und die
|
||||
interne CA des Proxys mit, `opnexus restore DATEI --with-env --with-ca` spielt sie zurück (ohne Datenbank-Zugang, Version und
|
||||
Adressen des Zielservers; die CA bleibt dieselbe, Browser vertrauen weiter). Neu: `opnexus migrate-host BENUTZER@ZIEL` zieht
|
||||
eine Installation in einem Schritt auf einen anderen Server um (Vorprüfung, Quelle anhalten, Sicherung mit Zufalls-Passphrase,
|
||||
Restore, Kontrolle). Im Lab 90 Sekunden mit drei Firewalls. `opnexus status` zeigt das Alter der letzten Vollsicherung.
|
||||
Handbuch: „Umzug und Notfall“.
|
||||
- **Betriebseinstellungen im Webinterface:** Was bisher nur in der `.env` oder gar nur im Code stand, lässt sich unter
|
||||
Einstellungen › Betriebseinstellungen ändern: Sitzungsdauer und Anmeldesperren, Bestätigungsfrist beim Ausrollen, Zeitlimit für
|
||||
Firewall-Anfragen, Messintervalle und Aufbewahrung, Intervall der Versionsprüfung, Heartbeat-Adresse, CA des Mailservers (PEM
|
||||
einfügen statt Datei einbinden), Zeitlimit Mailversand. Mit Grenzen, Standardwert, Zurücksetzen und Audit-Eintrag. Ein Wert in der
|
||||
`.env` hat weiter Vorrang und wird als „vom Server“ angezeigt. Adressen, TLS und Datenbank-Zugang bleiben bewusst in der `.env`.
|
||||
- **Heartbeat für einen externen Überwacher:** Die Heartbeat-Adresse (nur `https://`) wird nach jedem erfolgreichen Prüfzyklus
|
||||
aufgerufen; bleibt der Aufruf aus, schlägt der Überwacher Alarm. Die Adresse wird nie angezeigt oder protokolliert.
|
||||
- **Updates bringen die Compose-Datei mit:** Bisher tauschte `opnexus update` nur die Images; Änderungen an der
|
||||
`docker-compose.yml` (neue Variablen, Härtung) kamen bei bestehenden Installationen nie an. Jetzt übernimmt das Update die Datei
|
||||
des Zielreleases aus dem signierten Updater-Image, sichert die alte als `docker-compose.yml.bak-<version>` und legt sie beim
|
||||
Rollback zurück. Eine hier angepasste Datei wird nicht überschrieben (neue Fassung als `docker-compose.yml.new`); eigene
|
||||
Anpassungen gehören in `docker-compose.override.yml`.
|
||||
- **Alarm „Platte des OPNexus-Servers fast voll“** (standardmäßig an, ab 90 %, einstellbar): eine volle Platte hält die Datenbank
|
||||
an und damit jeden anderen Alarm. Auch im Dashboard und als Metrik `opnexus_server_disk_used_percent`.
|
||||
- **Härtung aus einem externen Audit:**
|
||||
- **Kein root mehr in den Containern:** Backend (uid 10001) und Frontend (`node`) laufen unprivilegiert, ohne neue Rechte (`no-new-privileges`), das Frontend zusätzlich
|
||||
ohne Capabilities. Das Backend startet über `backend/scripts/start.sh` noch kurz als root, stellt die vorhandenen Datenvolumes einmalig um und gibt dann alle Rechte ab:
|
||||
ein Update braucht keinen Handgriff, ein Rückweg auf eine ältere Version bleibt möglich. Notausgang für Sonderfälle: `OPNEXUS_RUN_AS_ROOT=1` in der `.env`.
|
||||
`wg-hub` (braucht root für WireGuard) und `gui-gateway` (läuft schon als nobody) sind unverändert. Der Proxy (Caddy) und der optionale Updater (Docker-Socket) laufen weiter wie bisher.
|
||||
- **Vollsicherung konsistent:** Ändern sich Geheimnisse, während die Sicherung läuft (jemand legt gerade ein Objekt an oder löscht eines), wird sie bis zu dreimal wiederholt;
|
||||
gelingt das nicht, bricht sie mit einer klaren Meldung ab, statt ein unstimmiges Archiv zu schreiben.
|
||||
- **Zieladress-Schutz überall:** Audit-Export (Webhook und Syslog), Offsite-S3 und der Update-Feed verbinden nur noch zu Adressen, die die Prüfung bestanden haben
|
||||
(kein Loopback, Link-Local, Metadaten-Dienst; auch nicht über DNS-Rebinding oder Weiterleitungen). Adressen im eigenen Netz bleiben erlaubt. Der Update-Feed wird beim Lesen auf 1 MB begrenzt, nicht erst danach.
|
||||
- **Fehlende Basis-Rechte werden sichtbar:** Fehlen dem API-Schlüssel schon die Rechte für die Grundmessungen (Speicher, Last, Platte), schlug das ganze Sampling still fehl und der
|
||||
Rechte-Alarm konnte nie auslösen. Jetzt entsteht ein Befund „Grundwerte“ (Recht „Lobby: Dashboard“), der sich nach Korrektur von selbst schließt.
|
||||
- **Installation:** Die `INSTALL.txt` im Bundle und die Anleitung auf der Website legen die `.env` jetzt mit `install -m 600` an (nur für dich lesbar), statt mit `cp`.
|
||||
- **Oberfläche:** Die Aktionsknöpfe der Backup-Tabelle auf der Host-Seite werden auf schmalen Bildschirmen als Symbole gezeigt, statt abgeschnitten zu werden.
|
||||
|
||||
## [1.33.0] "Saker" — 2026-10-24
|
||||
|
||||
|
||||
+2
-2
@@ -7,7 +7,7 @@ Zentrale Verwaltung mehrerer [OPNsense](https://opnsense.org)-Firewalls: Regeln,
|
||||
|
||||

|
||||
|
||||
> **Stand:** `1.33.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
> **Stand:** `1.34.0` ist die aktuelle stabile Version. Entwickelt und getestet wurde gegen echte OPNsense-Instanzen; eine frische Installation aus dem Release-Bundle sowie Updates und Rollbacks sind durchgängig geprüft. Ecken und Kanten bitte melden.
|
||||
|
||||
## Funktionen
|
||||
|
||||
@@ -68,7 +68,7 @@ Für den Zugriff von anderen Rechnern und den Produktivbetrieb mit TLS (mitgelie
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.33.0
|
||||
OPNEXUS_VERSION=1.34.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
@@ -7,7 +7,7 @@ Central management for multiple [OPNsense](https://opnsense.org) firewalls: rule
|
||||
|
||||

|
||||
|
||||
> **Status:** `1.33.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
> **Status:** `1.34.0` is the current stable release. It has been developed and tested against real OPNsense instances; a clean install from the release bundle, updates and rollbacks have been verified end to end. Please report rough edges.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -70,7 +70,7 @@ For access from other machines and for production use with TLS (bundled Caddy pr
|
||||
|
||||
```bash
|
||||
OPNEXUS_IMAGE_REPO=git.opnexus.dev/opnexus
|
||||
OPNEXUS_VERSION=1.33.0
|
||||
OPNEXUS_VERSION=1.34.0
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
+10
-1
@@ -17,4 +17,13 @@ RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
# Unprivilegierter Benutzer. Der Container startet über scripts/start.sh noch als
|
||||
# root, stellt die Datenvolumes um und gibt die Rechte dann ab (Begründung dort).
|
||||
# wg-hub und gui-gateway überschreiben den Befehl und umgehen das Skript bewusst:
|
||||
# der Hub braucht root für die WireGuard-Schnittstelle, das Gateway läuft per
|
||||
# Compose als nobody.
|
||||
RUN groupadd --system --gid 10001 opnexus \
|
||||
&& useradd --system --uid 10001 --gid 10001 --home-dir /app --shell /usr/sbin/nologin opnexus \
|
||||
&& chmod +x /app/scripts/start.sh
|
||||
|
||||
CMD ["/app/scripts/start.sh", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
+10
-1
@@ -42,6 +42,13 @@ class Settings(BaseSettings):
|
||||
# Quellcode gebaut; dann gibt es keine Updates per Klick (nur der Hinweis in den Einstellungen).
|
||||
opnexus_image_repo: str = ""
|
||||
|
||||
# ── Heartbeat (app/services/heartbeat_service.py) ──────────────────
|
||||
# Optionale Adresse eines externen Überwachers (healthchecks-Stil, ntfy, Uptime Kuma Push …). OPNexus ruft sie
|
||||
# nach jedem erfolgreichen Prüfzyklus der Benachrichtigungen per GET auf: bleibt der Aufruf aus, ist OPNexus
|
||||
# ausgefallen oder kommt nicht mehr an seine Datenbank. Nur https://, die Adresse wird nie protokolliert.
|
||||
# Leer = aus.
|
||||
heartbeat_url: str = ""
|
||||
|
||||
# ── Updates aus dem Webinterface (app/services/updater_client.py) ─────
|
||||
# Optionaler Updater-Container (Compose-Profil "updater"). Das Backend hat
|
||||
# nie Zugriff auf den Docker-Socket; es reicht nur "aktualisieren" /
|
||||
@@ -182,7 +189,9 @@ class Settings(BaseSettings):
|
||||
# entity id — see acme_secret_store.py.
|
||||
acme_secret_dir: str = "/data/acme-secrets"
|
||||
|
||||
model_config = SettingsConfigDict(case_sensitive=False)
|
||||
# env_ignore_empty: Compose passes unset variables as "" ("${X:-}"); they must mean "not set", so the value from
|
||||
# the web interface (app/runtime_settings.py) applies and a number field does not fail to parse.
|
||||
model_config = SettingsConfigDict(case_sensitive=False, env_ignore_empty=True)
|
||||
|
||||
|
||||
settings = Settings()
|
||||
@@ -191,6 +191,7 @@ class ErrorCode(str, Enum):
|
||||
NOTIFICATION_CHANNEL_NAME_TAKEN = "NOTIFICATION_CHANNEL_NAME_TAKEN"
|
||||
NOTIFICATION_CHANNEL_TEST_FAILED = "NOTIFICATION_CHANNEL_TEST_FAILED"
|
||||
NOTIFICATION_WEBHOOK_URL_INVALID = "NOTIFICATION_WEBHOOK_URL_INVALID"
|
||||
SETTING_INVALID = "SETTING_INVALID"
|
||||
OVERLAY_SUBNET_LOCKED = "OVERLAY_SUBNET_LOCKED"
|
||||
OVERLAY_NAME_TAKEN = "OVERLAY_NAME_TAKEN"
|
||||
OVERLAY_PEER_NOT_FOUND = "OVERLAY_PEER_NOT_FOUND"
|
||||
|
||||
+119
-12
@@ -15,6 +15,10 @@ or directly (passphrase in FULL_BACKUP_PASSPHRASE):
|
||||
python -m app.full_backup create > file.opxfull
|
||||
python -m app.full_backup info < file.opxfull # manifest only
|
||||
python -m app.full_backup restore < file.opxfull
|
||||
|
||||
Files of the server itself (the .env and the proxy's internal CA) are not visible to the backend. The CLI hands them
|
||||
over as a tar on stdin (``create --host-files``); they are stored under ``host/`` and only ever handed back as a tar
|
||||
on stdout (``host-files``) for the CLI to place: ``restore`` never writes them.
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
@@ -87,13 +91,30 @@ def _head_revision() -> str:
|
||||
return ScriptDirectory.from_config(migrate._config()).get_current_head()
|
||||
|
||||
|
||||
def create(passphrase: str, *, dump=_dump_database, revision=None) -> bytes:
|
||||
if len(passphrase) < MIN_PASSPHRASE:
|
||||
raise FullBackupError(f"Die Passphrase braucht mindestens {MIN_PASSPHRASE} Zeichen")
|
||||
manifest = {
|
||||
"format": FORMAT, "app_version": APP_VERSION, "db_revision": revision or _head_revision(),
|
||||
"created_at": datetime.now(timezone.utc).isoformat(), "data": sorted(_DATA_DIRS),
|
||||
}
|
||||
# Die Geheimnis-Verzeichnisse ändern sich genau dann, wenn sich ein Objekt in der Datenbank ändert. Ein
|
||||
# Schreibschutz über Datenbank und Dateien gleichzeitig gibt es nicht, deshalb wird eine Änderung während
|
||||
# der Sicherung erkannt und die Sicherung wiederholt. (Die Verzeichnisse der Firewall-Backups und des Overlays
|
||||
# gehören nicht dazu: sie ändern sich laufend und eine neue Datei darin macht die Sicherung nicht unstimmig.)
|
||||
_STABLE_DIRS = tuple(name for name in _DATA_DIRS if name.endswith("secrets"))
|
||||
ATTEMPTS = 3
|
||||
|
||||
|
||||
def _fingerprint() -> frozenset:
|
||||
seen = set()
|
||||
for name in _STABLE_DIRS:
|
||||
root = _dir(name)
|
||||
if not root.is_dir():
|
||||
continue
|
||||
for path in root.rglob("*"):
|
||||
try:
|
||||
st = path.stat()
|
||||
except OSError: # vom Löschen überholt: zählt als Änderung beim nächsten Vergleich
|
||||
continue
|
||||
seen.add((name, str(path.relative_to(root)), st.st_mtime_ns, st.st_size))
|
||||
return frozenset(seen)
|
||||
|
||||
|
||||
def _pack(manifest: dict, dump, host_files: bytes | None = None) -> bytes:
|
||||
plain = io.BytesIO()
|
||||
with tempfile.TemporaryDirectory() as tmp, tarfile.open(fileobj=plain, mode="w:gz") as tar:
|
||||
raw = json.dumps(manifest, indent=1).encode()
|
||||
@@ -107,7 +128,69 @@ def create(passphrase: str, *, dump=_dump_database, revision=None) -> bytes:
|
||||
path = _dir(name)
|
||||
if path.is_dir():
|
||||
tar.add(path, arcname=f"data/{name}")
|
||||
return backup_crypto.encrypt(plain.getvalue(), passphrase)
|
||||
for member, content in _host_members(host_files):
|
||||
tar.addfile(member, io.BytesIO(content) if content is not None else None)
|
||||
return plain.getvalue()
|
||||
|
||||
|
||||
_HOST_TOP = ("env", "caddy-ca")
|
||||
_HOST_MAX_BYTES = 10_000_000
|
||||
|
||||
|
||||
def _host_members(host_files: bytes | None) -> list[tuple[tarfile.TarInfo, bytes | None]]:
|
||||
"""The regular files and directories of the CLI's tar, re-rooted under host/. Anything else (links, devices,
|
||||
absolute or escaping paths, unknown top-level names) is refused: this tar comes from outside the container."""
|
||||
if host_files is None:
|
||||
return []
|
||||
if len(host_files) > _HOST_MAX_BYTES:
|
||||
raise FullBackupError("Die Dateien des Servers sind zu groß")
|
||||
try:
|
||||
src = tarfile.open(fileobj=io.BytesIO(host_files), mode="r:")
|
||||
except tarfile.TarError:
|
||||
raise FullBackupError("Die Dateien des Servers sind kein gültiges tar") from None
|
||||
out = []
|
||||
with src:
|
||||
for member in src.getmembers():
|
||||
parts = [p for p in Path(member.name).parts if p not in (".", "")]
|
||||
if not parts:
|
||||
continue
|
||||
if member.name.startswith(("/", "\\")) or ".." in parts or parts[0] not in _HOST_TOP:
|
||||
raise FullBackupError(f"Unzulässiger Eintrag in den Dateien des Servers: {member.name!r}")
|
||||
if not (member.isfile() or member.isdir()):
|
||||
raise FullBackupError(f"Unzulässiger Eintrag in den Dateien des Servers: {member.name!r}")
|
||||
info = tarfile.TarInfo("host/" + "/".join(parts))
|
||||
info.mtime, info.mode = member.mtime, (0o600 if member.isfile() else 0o700)
|
||||
if member.isdir():
|
||||
info.type = tarfile.DIRTYPE
|
||||
out.append((info, None))
|
||||
else:
|
||||
content = src.extractfile(member).read()
|
||||
info.size = len(content)
|
||||
out.append((info, content))
|
||||
if not out:
|
||||
raise FullBackupError("Die Dateien des Servers sind leer")
|
||||
return out
|
||||
|
||||
|
||||
def create(passphrase: str, *, dump=_dump_database, revision=None, host_files: bytes | None = None) -> bytes:
|
||||
if len(passphrase) < MIN_PASSPHRASE:
|
||||
raise FullBackupError(f"Die Passphrase braucht mindestens {MIN_PASSPHRASE} Zeichen")
|
||||
manifest = {
|
||||
"format": FORMAT, "app_version": APP_VERSION, "db_revision": revision or _head_revision(),
|
||||
"created_at": datetime.now(timezone.utc).isoformat(), "data": sorted(_DATA_DIRS),
|
||||
}
|
||||
members = _host_members(host_files) # validated before the (slow) dump
|
||||
if members:
|
||||
manifest["host"] = sorted({m.name.split("/")[1] for m, _ in members})
|
||||
for _ in range(ATTEMPTS):
|
||||
before = _fingerprint()
|
||||
blob = _pack(manifest, dump, host_files)
|
||||
if _fingerprint() == before:
|
||||
return backup_crypto.encrypt(blob, passphrase)
|
||||
raise FullBackupError(
|
||||
"Während der Sicherung haben sich Geheimnisse geändert (jemand legt gerade Objekte an oder löscht sie). "
|
||||
"Die Sicherung wäre in sich unstimmig; bitte gleich noch einmal versuchen."
|
||||
)
|
||||
|
||||
|
||||
def _open(blob: bytes, passphrase: str) -> tarfile.TarFile:
|
||||
@@ -178,6 +261,25 @@ def restore(blob: bytes, passphrase: str, *, restore_db=_restore_database, head=
|
||||
return {**manifest, "files_restored": len(restored)}
|
||||
|
||||
|
||||
def host_files(blob: bytes, passphrase: str) -> bytes:
|
||||
"""The host/ part as a tar (paths without the host/ prefix) for the CLI; empty tar when the backup has none."""
|
||||
out = io.BytesIO()
|
||||
with _open(blob, passphrase) as tar, tarfile.open(fileobj=out, mode="w") as dst:
|
||||
_manifest(tar)
|
||||
for member in tar.getmembers():
|
||||
parts = Path(member.name).parts
|
||||
if len(parts) < 2 or parts[0] != "host" or ".." in parts or not (member.isfile() or member.isdir()):
|
||||
continue
|
||||
info = tarfile.TarInfo("/".join(parts[1:]))
|
||||
info.mtime, info.mode, info.type = member.mtime, member.mode, member.type
|
||||
if member.isfile():
|
||||
info.size = member.size
|
||||
dst.addfile(info, tar.extractfile(member))
|
||||
else:
|
||||
dst.addfile(info)
|
||||
return out.getvalue()
|
||||
|
||||
|
||||
def _older_or_equal(revision: str | None, head: str | None) -> bool:
|
||||
"""Revisions are zero-padded numbers ('0009'): a dump from an older schema is fine, the app migrates it."""
|
||||
try:
|
||||
@@ -188,13 +290,18 @@ def _older_or_equal(revision: str | None, head: str | None) -> bool:
|
||||
|
||||
def _main(argv: list[str]) -> int:
|
||||
passphrase = os.environ.get("FULL_BACKUP_PASSPHRASE", "")
|
||||
if len(argv) != 2 or argv[1] not in ("create", "info", "restore"):
|
||||
args = argv[1:]
|
||||
with_host = args[1:] == ["--host-files"] and args[0] == "create"
|
||||
if not (len(args) == 1 and args[0] in ("create", "info", "restore", "host-files")) and not with_host:
|
||||
print(__doc__, file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
if argv[1] == "create":
|
||||
sys.stdout.buffer.write(create(passphrase))
|
||||
elif argv[1] == "info":
|
||||
if args[0] == "create":
|
||||
host = sys.stdin.buffer.read(_HOST_MAX_BYTES + 1) if with_host else None
|
||||
sys.stdout.buffer.write(create(passphrase, host_files=host))
|
||||
elif args[0] == "host-files":
|
||||
sys.stdout.buffer.write(host_files(sys.stdin.buffer.read(), passphrase))
|
||||
elif args[0] == "info":
|
||||
print(json.dumps(info(sys.stdin.buffer.read(), passphrase), indent=1))
|
||||
else:
|
||||
print(json.dumps(restore(sys.stdin.buffer.read(), passphrase), indent=1))
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
"""Leader gate: only one backend process runs the background loops (phase H0 of docs/HA-PLAN.md).
|
||||
|
||||
Two backends against the same database are not hypothetical: a second stack during a host move, an update that
|
||||
briefly overlaps, more than one Uvicorn worker, or (later) a standby. Deploys and sweeps are already serialized by
|
||||
``policy_lock``, but the loops are not: in the lab two backends sent every notification twice and aborted cycles on
|
||||
``UniqueViolation``. So the loops run only in the process that holds a session-level advisory lock.
|
||||
|
||||
How it holds:
|
||||
- The lock lives on a connection of its own that is never returned to the pool. If the process dies, PostgreSQL
|
||||
closes the session and frees the lock.
|
||||
- The lock can also vanish while this process lives (PostgreSQL restart, network cut, the session terminated). The
|
||||
connection is therefore checked every ``check_interval`` seconds and before every loop iteration (``checkpoint``).
|
||||
On the first failure every loop is cancelled at once and the process applies for the lock again.
|
||||
- A process that just got the lock waits ``takeover_delay`` before starting its loops: longer than the previous
|
||||
holder needs to notice its loss (one check interval plus the check timeout). Without that wait both could run an
|
||||
iteration at the same moment right after a takeover.
|
||||
- Losing the lock cancels the loops mid-iteration, like a hard restart would; every loop is built to survive that
|
||||
(deadlines and phases live in the database, ambiguous firewall calls are reconciled). The check timeout is generous
|
||||
(5 s) so a merely sluggish database does not make a single installation drop its loops.
|
||||
- A database in recovery (a streaming replica) never yields a leader: advisory locks work there, the loops would
|
||||
not (they write). The process stays in standby until the database is promoted.
|
||||
|
||||
The API is served by every process regardless of its role; writes through the API are serialized by the database
|
||||
as before. ``role`` is shown by /api/health and the metrics.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.database import engine
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Next to policy_lock (728194031) and the admin-role lock (728194032).
|
||||
LOCK_KEY = 728194033
|
||||
|
||||
STARTING, STANDBY, TAKEOVER, LEADER, REPLICA = "starting", "standby", "takeover", "leader", "replica"
|
||||
|
||||
|
||||
class LeadershipLost(Exception):
|
||||
"""Raised by ``checkpoint`` in a loop of a process that is no longer the leader. Loops must not catch it."""
|
||||
|
||||
|
||||
LoopFactory = Callable[[], Awaitable[None]]
|
||||
|
||||
|
||||
class LeaderGate:
|
||||
def __init__(self, loops: dict[str, LoopFactory], *, check_interval: float = 5.0, check_timeout: float = 5.0,
|
||||
retry_interval: float = 15.0, takeover_delay: float | None = None, connect=engine.connect):
|
||||
self._loops = loops
|
||||
self.check_interval = check_interval
|
||||
self.check_timeout = check_timeout
|
||||
self.retry_interval = retry_interval
|
||||
self.takeover_delay = check_interval + check_timeout + 2.0 if takeover_delay is None else takeover_delay
|
||||
self._connect = connect
|
||||
self._conn = None
|
||||
self._conn_lock = asyncio.Lock() # one statement at a time on the lock connection
|
||||
self._tasks: dict[str, asyncio.Task] = {}
|
||||
self._lost = asyncio.Event()
|
||||
self._last_ok = 0.0
|
||||
self.role = STARTING
|
||||
|
||||
# ── lock connection ──────────────────────────────────────────────────
|
||||
|
||||
async def _close(self) -> None:
|
||||
conn, self._conn = self._conn, None
|
||||
if conn is None:
|
||||
return
|
||||
try:
|
||||
await asyncio.wait_for(conn.invalidate(), self.check_timeout) # never back to the pool: it may hold the lock
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await asyncio.wait_for(conn.close(), self.check_timeout)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _try_acquire(self) -> str:
|
||||
"""STANDBY (lock held elsewhere or database unreachable), REPLICA, or TAKEOVER (lock is ours)."""
|
||||
try:
|
||||
self._conn = await asyncio.wait_for(self._connect(), self.check_timeout)
|
||||
async with self._conn_lock:
|
||||
if await asyncio.wait_for(self._conn.scalar(text("SELECT pg_is_in_recovery()")), self.check_timeout):
|
||||
await self._close()
|
||||
return REPLICA
|
||||
got = await asyncio.wait_for(
|
||||
self._conn.scalar(text("SELECT pg_try_advisory_lock(:key)"), {"key": LOCK_KEY}), self.check_timeout)
|
||||
await self._conn.commit() # leave no transaction open on a connection that lives for days
|
||||
except Exception as exc:
|
||||
logger.warning("Leader-Sperre: Datenbank nicht erreichbar (%s)", type(exc).__name__)
|
||||
await self._close()
|
||||
return STANDBY
|
||||
if not got:
|
||||
await self._close()
|
||||
return STANDBY
|
||||
self._last_ok = asyncio.get_running_loop().time()
|
||||
return TAKEOVER
|
||||
|
||||
async def check(self) -> bool:
|
||||
"""True while the lock connection answers and the database is still a primary. A False is final for this
|
||||
tenure: the caller demotes."""
|
||||
if self._conn is None:
|
||||
return False
|
||||
try:
|
||||
async with self._conn_lock:
|
||||
recovery = await asyncio.wait_for(self._conn.scalar(text("SELECT pg_is_in_recovery()")),
|
||||
self.check_timeout)
|
||||
await self._conn.commit()
|
||||
except Exception as exc:
|
||||
logger.error("Leader-Sperre verloren: Verbindung zur Datenbank gestört (%s)", type(exc).__name__)
|
||||
self._lost.set()
|
||||
return False
|
||||
if recovery:
|
||||
logger.error("Leader-Sperre aufgegeben: die Datenbank ist jetzt eine Replica")
|
||||
self._lost.set()
|
||||
return False
|
||||
self._last_ok = asyncio.get_running_loop().time()
|
||||
return True
|
||||
|
||||
async def checkpoint(self) -> None:
|
||||
"""Called at the top of every loop iteration. Raises LeadershipLost unless this process is still the leader;
|
||||
a recent successful check (under a second old) is reused so a draining loop does not hammer the database."""
|
||||
if self.role != LEADER or self._lost.is_set():
|
||||
raise LeadershipLost()
|
||||
if asyncio.get_running_loop().time() - self._last_ok < 1.0:
|
||||
return
|
||||
if not await self.check():
|
||||
raise LeadershipLost()
|
||||
|
||||
# ── loops ────────────────────────────────────────────────────────────
|
||||
|
||||
def _start_loops(self) -> None:
|
||||
for name, factory in self._loops.items():
|
||||
self._tasks[name] = asyncio.create_task(factory(), name=f"loop:{name}")
|
||||
|
||||
async def _stop_loops(self) -> None:
|
||||
tasks, self._tasks = list(self._tasks.values()), {}
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
# return_exceptions collects the loops' own CancelledError; a cancellation of THIS task still propagates.
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
def _restart_crashed(self) -> None:
|
||||
for name, task in list(self._tasks.items()):
|
||||
if not task.done() or task.cancelled():
|
||||
continue
|
||||
exc = task.exception()
|
||||
if isinstance(exc, LeadershipLost):
|
||||
self._lost.set()
|
||||
continue
|
||||
logger.error("Hintergrundschleife %s beendet (%s), wird neu gestartet", name,
|
||||
type(exc).__name__ if exc else "ohne Fehler")
|
||||
self._tasks[name] = asyncio.create_task(self._loops[name](), name=f"loop:{name}")
|
||||
|
||||
# ── supervisor ───────────────────────────────────────────────────────
|
||||
|
||||
async def _wait_or_lost(self, seconds: float) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(self._lost.wait(), seconds)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
|
||||
async def _tenure(self) -> None:
|
||||
"""From the moment the lock is ours until it is lost."""
|
||||
self.role = TAKEOVER
|
||||
logger.info("Leader-Sperre erhalten; Hintergrunddienste starten in %.0f s", self.takeover_delay)
|
||||
deadline = asyncio.get_running_loop().time() + self.takeover_delay
|
||||
while (left := deadline - asyncio.get_running_loop().time()) > 0:
|
||||
await self._wait_or_lost(min(left, self.check_interval))
|
||||
if self._lost.is_set() or not await self.check():
|
||||
return
|
||||
self.role = LEADER
|
||||
self._start_loops()
|
||||
logger.info("Dieser Prozess führt die Hintergrunddienste aus (Leader)")
|
||||
while not self._lost.is_set():
|
||||
await self._wait_or_lost(self.check_interval)
|
||||
if self._lost.is_set() or not await self.check():
|
||||
return
|
||||
self._restart_crashed()
|
||||
|
||||
async def run(self) -> None:
|
||||
announced = None
|
||||
while True:
|
||||
self._lost.clear()
|
||||
state = await self._try_acquire()
|
||||
if state == TAKEOVER:
|
||||
try:
|
||||
await self._tenure()
|
||||
finally:
|
||||
# Order matters: no loop may outlive the lock.
|
||||
await self._stop_loops()
|
||||
await self._close()
|
||||
self.role = STANDBY
|
||||
logger.warning("Hintergrunddienste angehalten (Leader-Sperre verloren); bewerbe mich neu")
|
||||
announced = None
|
||||
else:
|
||||
self.role = state
|
||||
if announced != state:
|
||||
logger.info(
|
||||
"Bereitschaft: %s", "die Datenbank ist eine Replica" if state == REPLICA
|
||||
else "ein anderer Prozess führt die Hintergrunddienste aus oder die Datenbank fehlt")
|
||||
announced = state
|
||||
await asyncio.sleep(self.retry_interval)
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Shutdown: loops first, then the lock connection (closing it releases the lock)."""
|
||||
self._lost.set()
|
||||
await self._stop_loops()
|
||||
await self._close()
|
||||
self.role = STANDBY
|
||||
|
||||
|
||||
# Set by main.lifespan; read by /api/health and the metrics.
|
||||
gate: LeaderGate | None = None
|
||||
|
||||
|
||||
def current_role() -> str:
|
||||
return gate.role if gate is not None else STARTING
|
||||
|
||||
|
||||
async def checkpoint() -> None:
|
||||
"""For the loops in main.py. Without a gate (tests that drive a loop directly) there is nothing to check."""
|
||||
if gate is not None:
|
||||
await gate.checkpoint()
|
||||
+44
-29
@@ -10,6 +10,7 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
|
||||
from app import runtime_settings
|
||||
from app.errors import ErrorCode
|
||||
|
||||
from app import models # noqa: F401 - registers models with Base.metadata
|
||||
@@ -65,6 +66,8 @@ from app.services import backup_schedule_service
|
||||
from app.services import carp_event_service, carp_maintenance_window_service
|
||||
from app.services import rule_history_service
|
||||
from app.services import rule_cleanup_service
|
||||
from app.services import heartbeat_service
|
||||
from app import leader
|
||||
|
||||
# Without this, Python's root logger defaults to WARNING with no handler,
|
||||
# so every logger.info() call in the 5 background loops below (CARP role
|
||||
@@ -87,7 +90,7 @@ _SWEEP_INTERVAL_SECONDS = 10
|
||||
# work once the table is non-trivially sized.
|
||||
_LOG_PRUNE_EVERY_N_CYCLES = 20
|
||||
# Same rationale as _LOG_PRUNE_EVERY_N_CYCLES, but the resource-sampling
|
||||
# loop's own cadence (settings.resource_poll_interval_seconds, default 60s).
|
||||
# loop's own cadence (runtime_settings.value("resource_poll_interval_seconds"), default 60s).
|
||||
_RESOURCE_PRUNE_EVERY_N_CYCLES = 20
|
||||
|
||||
|
||||
@@ -96,6 +99,7 @@ async def _deploy_revert_sweep_loop():
|
||||
RuleDeployment whose confirm window passed without a human confirming
|
||||
(see rule_deploy_service.sweep_expired_deployments)."""
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
reverted = await sweep_expired_deployments(db)
|
||||
@@ -164,12 +168,15 @@ async def _notification_poll_loop():
|
||||
NotificationSettings.poll_interval_seconds, floored by
|
||||
settings.notification_poll_min_interval_seconds."""
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
interval = settings.notification_poll_min_interval_seconds
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
await notification_service.run_notification_checks(db)
|
||||
row = await notification_settings_service.get_settings(db)
|
||||
interval = max(row.poll_interval_seconds, settings.notification_poll_min_interval_seconds)
|
||||
# Only after a cycle that went through: the external watcher must hear nothing when the checks fail.
|
||||
await heartbeat_service.ping()
|
||||
except Exception:
|
||||
logger.exception("Notification poll failed")
|
||||
try:
|
||||
@@ -194,6 +201,7 @@ async def _log_ingest_poll_loop():
|
||||
every _LOG_PRUNE_EVERY_N_CYCLES cycles (see constant above)."""
|
||||
cycle = 0
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
interval = settings.log_poll_min_interval_seconds
|
||||
cycle += 1
|
||||
try:
|
||||
@@ -227,11 +235,12 @@ async def _log_ingest_poll_loop():
|
||||
|
||||
async def _resource_poll_loop():
|
||||
"""Fourth background loop, same shape as _log_ingest_poll_loop but with
|
||||
a fixed interval (settings.resource_poll_interval_seconds) — v1
|
||||
a fixed interval (runtime_settings.value("resource_poll_interval_seconds")) — v1
|
||||
deliberately has no admin-editable settings row for this, see
|
||||
resource_monitor_service.py's module docstring."""
|
||||
cycle = 0
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
cycle += 1
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
@@ -242,7 +251,7 @@ async def _resource_poll_loop():
|
||||
result["hosts_failed"], result["hosts_polled"],
|
||||
)
|
||||
if cycle % _RESOURCE_PRUNE_EVERY_N_CYCLES == 0:
|
||||
deleted = await resource_monitor_service.prune_old_samples(db, settings.resource_sample_retention_days)
|
||||
deleted = await resource_monitor_service.prune_old_samples(db, runtime_settings.value("resource_sample_retention_days"))
|
||||
if deleted:
|
||||
logger.info("Resource-Sample-Retention: %d alte Einträge gelöscht", deleted)
|
||||
pruned = await rule_history_service.prune(db)
|
||||
@@ -250,7 +259,7 @@ async def _resource_poll_loop():
|
||||
logger.info("Regelverlauf-Retention: %d alte Einträge gelöscht", pruned)
|
||||
except Exception:
|
||||
logger.exception("Resource-Poll fehlgeschlagen")
|
||||
await asyncio.sleep(settings.resource_poll_interval_seconds)
|
||||
await asyncio.sleep(runtime_settings.value("resource_poll_interval_seconds"))
|
||||
|
||||
|
||||
_SCHEDULED_MAINTENANCE_POLL_SECONDS = 60
|
||||
@@ -262,6 +271,7 @@ async def _scheduled_maintenance_poll_loop():
|
||||
A fixed 60s cadence is plenty — these are human-picked timestamps, not
|
||||
a sub-minute-precision cron."""
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
result = await scheduled_maintenance_service.fire_due(db)
|
||||
@@ -295,6 +305,7 @@ async def _compliance_poll_loop():
|
||||
is plenty; the first run waits a little so a restart is not a thundering herd."""
|
||||
await asyncio.sleep(_COMPLIANCE_FIRST_RUN_SECONDS)
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
await compliance_service.check_all(db)
|
||||
@@ -310,6 +321,7 @@ async def _audit_export_poll_loop():
|
||||
"""Seventh background loop: forwards new audit events (syslog / webhook) when configured.
|
||||
Cheap when off (one settings read); keeps draining while batches come back full."""
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
shipped = 0
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
@@ -326,6 +338,7 @@ async def _backup_poll_loop():
|
||||
"""Eighth background loop: takes due scheduled config backups, prunes, copies offsite."""
|
||||
await asyncio.sleep(90)
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
result = await backup_schedule_service.run_due(db)
|
||||
@@ -345,6 +358,7 @@ async def _rule_observe_loop():
|
||||
Writes only to the database. The first look at a firewall only stores the starting point."""
|
||||
await asyncio.sleep(_RULE_OBSERVE_FIRST_RUN_SECONDS)
|
||||
while True:
|
||||
await leader.checkpoint()
|
||||
try:
|
||||
async with SessionLocal() as db:
|
||||
result = await rule_history_service.observe_all(db)
|
||||
@@ -352,7 +366,7 @@ async def _rule_observe_loop():
|
||||
logger.info("Regelverlauf: %s", result)
|
||||
except Exception:
|
||||
logger.exception("Regelverlauf fehlgeschlagen")
|
||||
await asyncio.sleep(max(settings.rule_observe_interval_seconds, settings.rule_observe_min_interval_seconds))
|
||||
await asyncio.sleep(max(runtime_settings.value("rule_observe_interval_seconds"), settings.rule_observe_min_interval_seconds))
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -362,6 +376,7 @@ async def lifespan(app: FastAPI):
|
||||
# A GUI-gateway slot must never be a trusted OPNexus origin (CORS with credentials).
|
||||
raise RuntimeError(f"OPNEXUS_GUI_SITES overlaps FRONTEND_ORIGIN: {', '.join(conflicts)} — refusing to start")
|
||||
await migrate.run()
|
||||
await runtime_settings.refresh() # values from Settings › Betrieb; refreshed in every process (leader or not)
|
||||
async with SessionLocal() as db:
|
||||
await auth_service.bootstrap_admin(db)
|
||||
try:
|
||||
@@ -376,32 +391,30 @@ async def lifespan(app: FastAPI):
|
||||
if settings.allow_insecure_firewall_urls:
|
||||
logger.warning("ALLOW_INSECURE_FIREWALL_URLS is on: firewall API credentials may travel over plain HTTP.")
|
||||
|
||||
sweep_task = asyncio.create_task(_deploy_revert_sweep_loop())
|
||||
notification_task = asyncio.create_task(_notification_poll_loop())
|
||||
log_ingest_task = asyncio.create_task(_log_ingest_poll_loop())
|
||||
resource_task = asyncio.create_task(_resource_poll_loop())
|
||||
scheduled_maintenance_task = asyncio.create_task(_scheduled_maintenance_poll_loop())
|
||||
compliance_task = asyncio.create_task(_compliance_poll_loop())
|
||||
audit_export_task = asyncio.create_task(_audit_export_poll_loop())
|
||||
backup_task = asyncio.create_task(_backup_poll_loop())
|
||||
rule_observe_task = asyncio.create_task(_rule_observe_loop())
|
||||
# Only the process holding the leader lock runs these (app/leader.py); a second backend on the same database waits.
|
||||
leader.gate = leader.LeaderGate({
|
||||
"revert-sweep": _deploy_revert_sweep_loop,
|
||||
"notifications": _notification_poll_loop,
|
||||
"log-ingest": _log_ingest_poll_loop,
|
||||
"resources": _resource_poll_loop,
|
||||
"scheduled-maintenance": _scheduled_maintenance_poll_loop,
|
||||
"compliance": _compliance_poll_loop,
|
||||
"audit-export": _audit_export_poll_loop,
|
||||
"backups": _backup_poll_loop,
|
||||
"rule-observe": _rule_observe_loop,
|
||||
})
|
||||
gate_task = asyncio.create_task(leader.gate.run(), name="leader-gate")
|
||||
settings_task = asyncio.create_task(runtime_settings.refresh_loop(), name="runtime-settings")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
sweep_task.cancel()
|
||||
notification_task.cancel()
|
||||
log_ingest_task.cancel()
|
||||
resource_task.cancel()
|
||||
scheduled_maintenance_task.cancel()
|
||||
compliance_task.cancel()
|
||||
audit_export_task.cancel()
|
||||
backup_task.cancel()
|
||||
rule_observe_task.cancel()
|
||||
for task in (sweep_task, notification_task, log_ingest_task, resource_task, scheduled_maintenance_task, compliance_task, audit_export_task, backup_task, rule_observe_task):
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
settings_task.cancel()
|
||||
gate_task.cancel()
|
||||
try:
|
||||
await gate_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
await leader.gate.stop()
|
||||
|
||||
|
||||
API_DESCRIPTION = (
|
||||
@@ -517,7 +530,9 @@ app.include_router(port_forward.router, prefix="/api")
|
||||
|
||||
@app.get("/api/health")
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
# role: leader (runs the background loops), standby/takeover (another process does, or about to start),
|
||||
# replica (database in recovery), starting. The HTTP status stays 200 in every role: the API works in all of them.
|
||||
return {"status": "ok", "role": leader.current_role()}
|
||||
|
||||
app.include_router(one_to_one.router, prefix="/api")
|
||||
|
||||
|
||||
@@ -41,6 +41,7 @@ from sqlalchemy.engine import make_url
|
||||
from sqlalchemy.ext.asyncio import AsyncConnection
|
||||
|
||||
import app.models
|
||||
from app import runtime_settings
|
||||
from app import legacy_schema_patches
|
||||
from app.config import settings
|
||||
from app.database import Base, engine
|
||||
@@ -146,7 +147,7 @@ async def _ensure_pg_client_matches_server() -> None:
|
||||
|
||||
def _prune_backups(directory: Path) -> None:
|
||||
dumps = sorted(directory.glob(f"{_BACKUP_PREFIX}*.dump"), key=lambda p: p.stat().st_mtime)
|
||||
for old in dumps[: max(0, len(dumps) - settings.migration_backup_keep)]:
|
||||
for old in dumps[: max(0, len(dumps) - runtime_settings.value("migration_backup_keep"))]:
|
||||
old.unlink(missing_ok=True)
|
||||
|
||||
|
||||
@@ -208,6 +209,10 @@ async def _migrate() -> None:
|
||||
logger.info("Datenbankschema ist aktuell (Revision %s)", head)
|
||||
return
|
||||
|
||||
try: # the kept-backups count may be set in the web interface; an old schema without the table keeps the default
|
||||
await runtime_settings.refresh()
|
||||
except Exception:
|
||||
pass
|
||||
await _backup_before_migration(current, head)
|
||||
logger.info("Migriere Datenbankschema %s → %s", current, head)
|
||||
async with engine.begin() as conn:
|
||||
|
||||
@@ -113,3 +113,5 @@ __all__ += ["RuleCleanupOperation", "RuleCleanupItem"]
|
||||
# app.reset_2fa) otherwise fails at the first query because OpnsenseHost refers to classes nobody imported.
|
||||
from app.models.host_variable import HostVariable
|
||||
__all__ += ["HostVariable"]
|
||||
from app.models.runtime_setting import RuntimeSetting
|
||||
__all__ += ["RuntimeSetting"]
|
||||
@@ -78,6 +78,9 @@ class NotificationSettings(Base):
|
||||
# OPNexus threshold on the measured values (0 = off).
|
||||
gateway_loss_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=20, server_default="20")
|
||||
gateway_delay_threshold_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=500, server_default="500")
|
||||
# The disk OPNexus itself runs on (1.34): a full disk stops the database, and with it every alert. On by default.
|
||||
server_disk_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
server_disk_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=90, server_default="90")
|
||||
# Resources (0 = that threshold is off). Load is per CPU thread.
|
||||
disk_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=90, server_default="90")
|
||||
memory_threshold_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=95, server_default="95")
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Values saved in Settings › Betrieb (app/runtime_settings.py); one row per changed setting."""
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import JSON, DateTime, String, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
|
||||
|
||||
class RuntimeSetting(Base):
|
||||
__tablename__ = "runtime_settings"
|
||||
|
||||
key: Mapped[str] = mapped_column(String(100), primary_key=True)
|
||||
# Number or text in the unit of the spec; NULL for secrets (they live in the notification secret store).
|
||||
value: Mapped[object | None] = mapped_column(JSON, nullable=True)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now(),
|
||||
onupdate=func.now())
|
||||
updated_by: Mapped[str | None] = mapped_column(String(150), nullable=True)
|
||||
@@ -10,7 +10,7 @@ from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.schemas.audit_export import AuditExportRead, AuditExportUpdate
|
||||
from app.services import audit_export_service as svc
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
|
||||
router = APIRouter(prefix="/settings/audit-export", tags=["audit-export"])
|
||||
|
||||
@@ -29,6 +29,11 @@ async def read(db: AsyncSession = Depends(get_db), user: AdminUser = Depends(cur
|
||||
@router.put("", response_model=AuditExportRead)
|
||||
async def update(payload: AuditExportUpdate, db: AsyncSession = Depends(get_db), user: AdminUser = Depends(current_user)):
|
||||
_require_admin(user)
|
||||
if payload.webhook_url:
|
||||
try:
|
||||
payload.webhook_url = url_guard.validate_url(payload.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise api_error(422, ErrorCode.NOTIFICATION_WEBHOOK_URL_INVALID, reason=str(exc)) from None
|
||||
row = await svc.get_settings(db)
|
||||
was_active = svc.active(row)
|
||||
for field in ("syslog_enabled", "syslog_host", "syslog_port", "syslog_protocol", "webhook_enabled", "webhook_url"):
|
||||
|
||||
@@ -5,6 +5,7 @@ from pydantic import BaseModel, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import runtime_settings
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.config import settings
|
||||
from app.database import get_db
|
||||
@@ -58,7 +59,7 @@ def _set_session_cookie(response: Response, token: str, max_age: int | None = No
|
||||
response.set_cookie(
|
||||
key=session_cookie_name(),
|
||||
value=token,
|
||||
max_age=max_age or settings.session_lifetime,
|
||||
max_age=max_age or runtime_settings.value("session_lifetime_days"),
|
||||
httponly=True,
|
||||
samesite="lax",
|
||||
secure=settings.cookie_secure,
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
from typing import Literal
|
||||
from typing import Any, Literal
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.database import get_db
|
||||
from app.models.preferences import UserPreferences
|
||||
from app.models.user import AdminUser
|
||||
from app.routers.auth import current_user
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.services import audit_service
|
||||
|
||||
router=APIRouter(prefix='/settings',tags=['settings'])
|
||||
|
||||
@@ -22,8 +25,8 @@ async def read(user: AdminUser=Depends(current_user),db:AsyncSession=Depends(get
|
||||
return {'username':user.username,'theme':preferences.theme if preferences else 'system',
|
||||
'language':preferences.language if preferences else 'de',
|
||||
'setup_done':preferences.setup_done if preferences else False,
|
||||
'deploy_confirm_window_seconds':settings.deploy_confirm_window_seconds,
|
||||
'session_lifetime_seconds':settings.session_lifetime}
|
||||
'deploy_confirm_window_seconds':runtime_settings.value("deploy_confirm_window_seconds"),
|
||||
'session_lifetime_seconds':runtime_settings.value("session_lifetime_days")}
|
||||
|
||||
@router.put('')
|
||||
async def update(payload:Preferences,user:AdminUser=Depends(current_user),db:AsyncSession=Depends(get_db)):
|
||||
@@ -35,5 +38,45 @@ async def update(payload:Preferences,user:AdminUser=Depends(current_user),db:Asy
|
||||
await db.execute(insert(UserPreferences).values(user_id=user.id,theme=theme,language=language,setup_done=setup_done).on_conflict_do_update(index_elements=['user_id'],set_={'theme':theme,'language':language,'setup_done':setup_done}))
|
||||
await db.commit()
|
||||
return {'username':username,'theme':theme,'language':language,'setup_done':setup_done,
|
||||
'deploy_confirm_window_seconds':settings.deploy_confirm_window_seconds,
|
||||
'session_lifetime_seconds':settings.session_lifetime}
|
||||
'deploy_confirm_window_seconds':runtime_settings.value("deploy_confirm_window_seconds"),
|
||||
'session_lifetime_seconds':runtime_settings.value("session_lifetime_days")}
|
||||
|
||||
|
||||
|
||||
# ── Settings › Betrieb: values that used to need the .env (app/runtime_settings.py) ──────────────────────────
|
||||
|
||||
class SystemSettingsUpdate(BaseModel):
|
||||
# key -> new value; null (or "") resets the setting to its default.
|
||||
values: dict[str, Any] = Field(max_length=50)
|
||||
|
||||
|
||||
@router.get('/system')
|
||||
async def read_system(user: AdminUser = Depends(current_user)):
|
||||
return {'settings': runtime_settings.describe()}
|
||||
|
||||
|
||||
@router.put('/system')
|
||||
async def update_system(payload: SystemSettingsUpdate, user: AdminUser = Depends(current_user),
|
||||
db: AsyncSession = Depends(get_db)):
|
||||
if user.role != 'admin':
|
||||
raise api_error(403, ErrorCode.ADMIN_REQUIRED)
|
||||
before = {item['key']: item for item in runtime_settings.describe()}
|
||||
changes = {}
|
||||
for key, new in payload.values.items():
|
||||
spec = runtime_settings.BY_KEY.get(key)
|
||||
if spec is None:
|
||||
continue # rejected by save() below with a proper error
|
||||
if spec.kind in ('secret_url', 'pem'): # never put the secret (or a whole certificate) into the audit trail
|
||||
changes[key] = {'old': 'gesetzt' if before[key].get('is_set') else 'leer', 'new': 'leer' if not new else 'gesetzt'}
|
||||
else:
|
||||
changes[key] = {'old': before[key]['value'], 'new': new if new not in ('', None) else before[key]['default']}
|
||||
try:
|
||||
if changes:
|
||||
db.add(audit_service.make_event(
|
||||
kind='settings', action='system_settings', status='accepted', entity_id='system',
|
||||
message='Betriebseinstellungen geändert', details={'changes': changes}))
|
||||
await runtime_settings.save(db, payload.values, user.username)
|
||||
except runtime_settings.SettingError as exc:
|
||||
await db.rollback()
|
||||
raise api_error(422, ErrorCode.SETTING_INVALID, setting=exc.key, reason=exc.reason) from None
|
||||
return {'settings': runtime_settings.describe()}
|
||||
@@ -0,0 +1,259 @@
|
||||
"""Settings an admin changes in the web interface instead of the .env (Settings › Betrieb).
|
||||
|
||||
One registry for every tunable that is safe to change while OPNexus runs. The value in effect is, in this order:
|
||||
1. the environment variable, if the spec names one and it is set to a non-empty value ("vom Server festgelegt": a
|
||||
scripted installation stays in control, the UI shows the value read-only);
|
||||
2. the value saved in the web interface (table ``runtime_settings``; secrets in the notification secret store);
|
||||
3. the built-in default (the attribute of ``app.config.settings``).
|
||||
|
||||
Reads are synchronous (``value(key)``) from a cache that every backend process refreshes every few seconds
|
||||
(``refresh_loop``), so a leader and a standby agree within that time and hot paths (login, every firewall call) do
|
||||
not touch the database. A save refreshes the cache of the saving process at once.
|
||||
|
||||
What deliberately stays in the .env: addresses and TLS of OPNexus itself (a wrong value there locks everyone out of
|
||||
the very interface that would fix it), the database credentials, the first admin account, the container switches
|
||||
(Compose profiles) and the escape hatches (ALLOW_INSECURE_*, OPNEXUS_RUN_AS_ROOT).
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import ssl
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.models.runtime_setting import RuntimeSetting
|
||||
from app.services import notification_secret_store, url_guard
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
REFRESH_SECONDS = 15
|
||||
_MAX_PEM = 65536
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Spec:
|
||||
key: str
|
||||
group: str # security | operation | monitoring | integrations (UI sections)
|
||||
kind: str # int | float | secret_url | pem
|
||||
attr: str # attribute of app.config.settings holding the default (and the env value, via pydantic)
|
||||
unit: str = "" # shown next to the field: seconds | minutes | hours | days | count
|
||||
minimum: float | None = None
|
||||
maximum: float | None = None
|
||||
scale: int = 1 # stored/shown in `unit`; the code reads value * scale (days -> seconds)
|
||||
env: str | None = None
|
||||
|
||||
|
||||
SPECS: list[Spec] = [
|
||||
# Security
|
||||
Spec("session_lifetime_days", "security", "int", "session_lifetime", "days", 1, 365, scale=86400),
|
||||
Spec("login_max_attempts", "security", "int", "login_max_attempts", "count", 3, 50),
|
||||
Spec("login_ip_max_attempts", "security", "int", "login_ip_max_attempts", "count", 5, 500),
|
||||
Spec("login_window_seconds", "security", "int", "login_window_seconds", "seconds", 60, 3600),
|
||||
Spec("login_lockout_seconds", "security", "int", "login_lockout_seconds", "seconds", 60, 86400),
|
||||
# Operation
|
||||
Spec("deploy_confirm_window_seconds", "operation", "int", "deploy_confirm_window_seconds", "seconds", 60, 1800),
|
||||
Spec("opnsense_timeout", "operation", "float", "opnsense_timeout", "seconds", 5, 120),
|
||||
Spec("migration_backup_keep", "operation", "int", "migration_backup_keep", "count", 1, 50),
|
||||
# Monitoring
|
||||
Spec("resource_poll_interval_seconds", "monitoring", "int", "resource_poll_interval_seconds", "seconds", 30, 3600),
|
||||
Spec("resource_sample_retention_days", "monitoring", "int", "resource_sample_retention_days", "days", 1, 365),
|
||||
Spec("rule_observe_interval_seconds", "monitoring", "int", "rule_observe_interval_seconds", "seconds", 60, 86400),
|
||||
# No env lock: Compose passes UPDATE_CHECK_INTERVAL_HOURS=12 when unset (it must: the backend before 1.34 cannot
|
||||
# parse an empty value, and a rollback runs that backend with the new compose file), so the variable cannot tell
|
||||
# "set by the admin" from "default". It stays the default; the web interface may override it.
|
||||
Spec("update_check_interval_hours", "monitoring", "int", "update_check_interval_hours", "hours", 1, 168),
|
||||
# Integrations
|
||||
Spec("heartbeat_url", "integrations", "secret_url", "heartbeat_url", env="HEARTBEAT_URL"),
|
||||
Spec("smtp_ca_pem", "integrations", "pem", "smtp_ca_file", env="SMTP_CA_FILE"),
|
||||
Spec("notification_smtp_timeout_seconds", "integrations", "float", "notification_smtp_timeout_seconds", "seconds",
|
||||
5, 120),
|
||||
]
|
||||
BY_KEY = {s.key: s for s in SPECS}
|
||||
|
||||
# Saved values (numbers as stored, in the spec's unit; secrets and PEM as text). Missing key = not saved.
|
||||
_cache: dict[str, Any] = {}
|
||||
|
||||
|
||||
class SettingError(ValueError):
|
||||
def __init__(self, key: str, reason: str):
|
||||
super().__init__(reason)
|
||||
self.key = key
|
||||
self.reason = reason
|
||||
|
||||
|
||||
def _secret_name(key: str) -> str:
|
||||
return f"runtime-{key}"
|
||||
|
||||
|
||||
def from_server(spec: Spec) -> bool:
|
||||
if spec.env and os.environ.get(spec.env, "").strip():
|
||||
return True
|
||||
# Text settings default to "": a non-empty value can only come from the server (env, parsed by pydantic).
|
||||
return spec.kind in ("secret_url", "pem") and bool(str(getattr(settings, spec.attr) or "").strip())
|
||||
|
||||
|
||||
def _default(spec: Spec) -> Any:
|
||||
raw = getattr(settings, spec.attr)
|
||||
if spec.kind in ("int", "float") and spec.scale != 1:
|
||||
return type(raw)(raw / spec.scale) if spec.kind == "float" else int(raw // spec.scale)
|
||||
return raw
|
||||
|
||||
|
||||
def shown(spec: Spec) -> tuple[Any, str]:
|
||||
"""(value in the spec's unit, source): source is server | ui | default."""
|
||||
if from_server(spec):
|
||||
return _default(spec), "server"
|
||||
if spec.key in _cache:
|
||||
return _cache[spec.key], "ui"
|
||||
return _default(spec), "default"
|
||||
|
||||
|
||||
def value(key: str) -> Any:
|
||||
"""The value the code uses (numbers multiplied by the spec's scale)."""
|
||||
spec = BY_KEY[key]
|
||||
if from_server(spec) or spec.key not in _cache:
|
||||
return getattr(settings, spec.attr) # tests monkeypatch settings; the env is already parsed into it
|
||||
raw = _cache[spec.key]
|
||||
if spec.kind == "int":
|
||||
return int(raw) * spec.scale
|
||||
if spec.kind == "float":
|
||||
return float(raw) * spec.scale
|
||||
return raw
|
||||
|
||||
|
||||
# ── validation ───────────────────────────────────────────────────────────
|
||||
|
||||
def validate(spec: Spec, raw: Any) -> Any:
|
||||
if spec.kind in ("int", "float"):
|
||||
if isinstance(raw, bool) or not isinstance(raw, (int, float)):
|
||||
raise SettingError(spec.key, "Bitte eine Zahl angeben")
|
||||
if spec.kind == "int" and int(raw) != raw:
|
||||
raise SettingError(spec.key, "Bitte eine ganze Zahl angeben")
|
||||
number = int(raw) if spec.kind == "int" else float(raw)
|
||||
if (spec.minimum is not None and number < spec.minimum) or (spec.maximum is not None and number > spec.maximum):
|
||||
raise SettingError(spec.key, f"Erlaubt ist {spec.minimum:g} bis {spec.maximum:g}")
|
||||
return number
|
||||
if not isinstance(raw, str):
|
||||
raise SettingError(spec.key, "Bitte Text angeben")
|
||||
text = raw.strip()
|
||||
if spec.kind == "secret_url":
|
||||
try:
|
||||
url = url_guard.validate_url(text, max_length=2000)
|
||||
except ValueError as exc:
|
||||
raise SettingError(spec.key, str(exc)) from None
|
||||
if not url.startswith("https://"):
|
||||
raise SettingError(spec.key, "Die Adresse muss mit https:// beginnen")
|
||||
return url
|
||||
if spec.kind == "pem":
|
||||
if len(text) > _MAX_PEM or "-----BEGIN CERTIFICATE-----" not in text:
|
||||
raise SettingError(spec.key, "Bitte ein CA-Zertifikat im PEM-Format einfügen")
|
||||
try:
|
||||
ssl.create_default_context(cadata=text)
|
||||
except (ssl.SSLError, ValueError):
|
||||
raise SettingError(spec.key, "Das Zertifikat ist nicht lesbar") from None
|
||||
return text
|
||||
raise SettingError(spec.key, "Unbekannte Einstellung")
|
||||
|
||||
|
||||
# ── storage ──────────────────────────────────────────────────────────────
|
||||
|
||||
async def load(db: AsyncSession) -> dict[str, Any]:
|
||||
rows = (await db.scalars(select(RuntimeSetting))).all()
|
||||
fresh: dict[str, Any] = {}
|
||||
for row in rows:
|
||||
spec = BY_KEY.get(row.key)
|
||||
if spec is None:
|
||||
continue # a setting a newer version knew about; keep the row, ignore it
|
||||
if spec.kind == "secret_url":
|
||||
secret = notification_secret_store.read_secret(_secret_name(spec.key))
|
||||
if secret:
|
||||
fresh[spec.key] = secret
|
||||
elif row.value is not None:
|
||||
fresh[spec.key] = row.value
|
||||
return fresh
|
||||
|
||||
|
||||
async def refresh(db: AsyncSession | None = None) -> None:
|
||||
global _cache
|
||||
if db is not None:
|
||||
_cache = await load(db)
|
||||
return
|
||||
async with SessionLocal() as session:
|
||||
_cache = await load(session)
|
||||
|
||||
|
||||
async def refresh_loop() -> None:
|
||||
while True:
|
||||
await asyncio.sleep(REFRESH_SECONDS)
|
||||
try:
|
||||
await refresh()
|
||||
except Exception as exc:
|
||||
logger.warning("Einstellungen konnten nicht neu gelesen werden (%s)", type(exc).__name__)
|
||||
|
||||
|
||||
async def save(db: AsyncSession, values: dict[str, Any], actor: str) -> list[str]:
|
||||
"""Validates everything first, then writes; None resets a setting to its default. Returns the changed keys."""
|
||||
clean: dict[str, Any] = {}
|
||||
for key, raw in values.items():
|
||||
spec = BY_KEY.get(key)
|
||||
if spec is None:
|
||||
raise SettingError(key, "Unbekannte Einstellung")
|
||||
if from_server(spec):
|
||||
raise SettingError(key, f"Wird vom Server festgelegt ({spec.env} in der .env)")
|
||||
clean[key] = None if raw is None or (isinstance(raw, str) and not raw.strip()) else validate(spec, raw)
|
||||
for key, val in clean.items():
|
||||
spec = BY_KEY[key]
|
||||
if val is None:
|
||||
await db.execute(delete(RuntimeSetting).where(RuntimeSetting.key == key))
|
||||
if spec.kind == "secret_url":
|
||||
notification_secret_store.delete_secret(_secret_name(key))
|
||||
continue
|
||||
stored = val
|
||||
if spec.kind == "secret_url":
|
||||
notification_secret_store.save_secret(_secret_name(key), val)
|
||||
stored = None # the database only knows that it is set
|
||||
await db.execute(
|
||||
insert(RuntimeSetting).values(key=key, value=stored, updated_by=actor)
|
||||
.on_conflict_do_update(index_elements=["key"], set_={"value": stored, "updated_by": actor})
|
||||
)
|
||||
await db.commit()
|
||||
await refresh(db)
|
||||
return sorted(clean)
|
||||
|
||||
|
||||
def describe() -> list[dict]:
|
||||
out = []
|
||||
for spec in SPECS:
|
||||
val, source = shown(spec)
|
||||
item = {"key": spec.key, "group": spec.group, "kind": spec.kind, "unit": spec.unit,
|
||||
"min": spec.minimum, "max": spec.maximum, "default": None, "source": source,
|
||||
"env": spec.env if source == "server" else None}
|
||||
if spec.kind == "secret_url":
|
||||
item["value"] = None
|
||||
item["is_set"] = bool(val)
|
||||
elif spec.kind == "pem":
|
||||
item["value"] = val if source != "server" else None # an env value is a path, not the certificate
|
||||
item["is_set"] = bool(val)
|
||||
else:
|
||||
item["value"] = val
|
||||
item["default"] = _default(spec)
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
# ── consumers ────────────────────────────────────────────────────────────
|
||||
|
||||
def smtp_ssl_context() -> ssl.SSLContext:
|
||||
"""Trust for the mail server (and the TLS syslog of the audit export): the system store plus the private CA from
|
||||
SMTP_CA_FILE or the certificate pasted in the web interface."""
|
||||
spec = BY_KEY["smtp_ca_pem"]
|
||||
if from_server(spec):
|
||||
return ssl.create_default_context(cafile=settings.smtp_ca_file)
|
||||
pem = _cache.get(spec.key)
|
||||
return ssl.create_default_context(cadata=pem) if pem else ssl.create_default_context()
|
||||
@@ -16,7 +16,7 @@ ProblemType = Literal[
|
||||
"backup_stale",
|
||||
"backup_offsite_failed",
|
||||
"compliance_finding",
|
||||
"gateway_down", "resource_threshold", "service_down", "api_rights_missing",
|
||||
"gateway_down", "resource_threshold", "service_down", "api_rights_missing", "server_disk",
|
||||
"rollout_paused", "rollout_start_failed",
|
||||
]
|
||||
ChannelType = Literal["ntfy", "gotify", "telegram", "teams", "matrix"]
|
||||
@@ -55,6 +55,8 @@ class NotificationSettingsRead(BaseModel):
|
||||
service_down_enabled: bool = False
|
||||
rollout_problems_enabled: bool = True
|
||||
api_rights_enabled: bool = True
|
||||
server_disk_enabled: bool = True
|
||||
server_disk_threshold_pct: int = 90
|
||||
gateway_alert_muted: list[str] = Field(default_factory=list)
|
||||
gateway_loss_threshold_pct: int = 20
|
||||
gateway_delay_threshold_ms: int = 500
|
||||
@@ -114,6 +116,8 @@ class NotificationSettingsUpdate(BaseModel):
|
||||
service_down_enabled: bool | None = None
|
||||
rollout_problems_enabled: bool | None = None
|
||||
api_rights_enabled: bool | None = None
|
||||
server_disk_enabled: bool | None = None
|
||||
server_disk_threshold_pct: int | None = Field(default=None, ge=50, le=99)
|
||||
# 0 switches that single threshold off.
|
||||
gateway_loss_threshold_pct: int | None = Field(default=None, ge=0, le=100)
|
||||
gateway_delay_threshold_ms: int | None = Field(default=None, ge=0, le=60000)
|
||||
|
||||
@@ -23,10 +23,11 @@ import httpx
|
||||
from sqlalchemy import and_, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.audit_export import AuditExportSettings
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
from app.version import APP_VERSION
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -75,17 +76,28 @@ def _frame_tcp(lines: list[bytes]) -> bytes:
|
||||
return b"".join(str(len(line)).encode() + b" " + line for line in lines)
|
||||
|
||||
|
||||
def _send_syslog_sync(row: AuditExportSettings, lines: list[bytes]) -> None:
|
||||
def _send_syslog_sync(row: AuditExportSettings, lines: list[bytes], addresses: list[str]) -> None:
|
||||
"""``addresses`` are the already checked addresses of ``syslog_host``: connect to those, never to the name."""
|
||||
host, port = row.syslog_host, row.syslog_port
|
||||
if row.syslog_protocol == "udp":
|
||||
with socket.socket(socket.AF_INET6 if ":" in (host or "") else socket.AF_INET, socket.SOCK_DGRAM) as sock:
|
||||
address = addresses[0]
|
||||
with socket.socket(socket.AF_INET6 if ":" in address else socket.AF_INET, socket.SOCK_DGRAM) as sock:
|
||||
for line in lines:
|
||||
sock.sendto(line[:8000], (host, port))
|
||||
sock.sendto(line[:8000], (address, port))
|
||||
return
|
||||
with socket.create_connection((host, port), timeout=settings.opnsense_timeout) as raw:
|
||||
last: OSError | None = None
|
||||
for address in addresses:
|
||||
try:
|
||||
raw = socket.create_connection((address, port), timeout=runtime_settings.value("opnsense_timeout"))
|
||||
break
|
||||
except OSError as exc:
|
||||
last = exc
|
||||
else:
|
||||
raise last or OSError("keine Adresse")
|
||||
with raw:
|
||||
if row.syslog_protocol == "tls":
|
||||
context = ssl.create_default_context(cafile=settings.smtp_ca_file or None)
|
||||
with context.wrap_socket(raw, server_hostname=host) as tls:
|
||||
context = runtime_settings.smtp_ssl_context()
|
||||
with context.wrap_socket(raw, server_hostname=host) as tls: # Zertifikat gegen den Namen, nicht die IP
|
||||
tls.sendall(_frame_tcp(lines))
|
||||
else:
|
||||
raw.sendall(_frame_tcp(lines))
|
||||
@@ -95,7 +107,13 @@ async def send_syslog(row: AuditExportSettings, events: list[dict]) -> None:
|
||||
if not row.syslog_host:
|
||||
raise ExportError("Kein Syslog-Ziel konfiguriert")
|
||||
try:
|
||||
await asyncio.to_thread(_send_syslog_sync, row, [syslog_line(e) for e in events])
|
||||
addresses = await url_guard.checked_addresses(row.syslog_host, row.syslog_port)
|
||||
except ValueError as exc:
|
||||
raise ExportError(f"Syslog-Ziel nicht erlaubt: {exc}") from None
|
||||
except OSError as exc:
|
||||
raise ExportError(f"Syslog-Zustellung fehlgeschlagen ({type(exc).__name__})") from None
|
||||
try:
|
||||
await asyncio.to_thread(_send_syslog_sync, row, [syslog_line(e) for e in events], addresses)
|
||||
except (OSError, ssl.SSLError) as exc:
|
||||
raise ExportError(f"Syslog-Zustellung fehlgeschlagen ({type(exc).__name__})") from None
|
||||
|
||||
@@ -110,13 +128,18 @@ async def send_webhook(row: AuditExportSettings, events: list[dict]) -> None:
|
||||
if not secret:
|
||||
raise ExportError("Signaturschlüssel ist konfiguriert, aber nicht lesbar – bitte neu setzen")
|
||||
headers["X-OPNexus-Signature"] = "sha256=" + hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest()
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=settings.opnsense_timeout) as client:
|
||||
(await client.post(row.webhook_url, content=raw, headers=headers)).raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
# The URL often is the credential: never put it in a message.
|
||||
reason = f"HTTP {exc.response.status_code}" if isinstance(exc, httpx.HTTPStatusError) else type(exc).__name__
|
||||
raise ExportError(f"Webhook-Zustellung fehlgeschlagen ({reason})") from None
|
||||
# Die URL ist oft selbst das Zugangsdatum: das Anfrage-Log von httpx bleibt währenddessen stumm.
|
||||
with url_guard.muted():
|
||||
try:
|
||||
url_guard.validate_url(row.webhook_url)
|
||||
await url_guard.ensure_resolved_safe(row.webhook_url)
|
||||
except ValueError as exc:
|
||||
raise ExportError(f"Webhook-Ziel nicht erlaubt: {exc}") from None
|
||||
try:
|
||||
async with url_guard.guarded_client(timeout=runtime_settings.value("opnsense_timeout"), follow_redirects=False) as client:
|
||||
(await client.post(row.webhook_url, content=raw, headers=headers)).raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
raise ExportError(f"Webhook-Zustellung fehlgeschlagen ({url_guard.failure_reason(exc)})") from None
|
||||
|
||||
|
||||
def active(row: AuditExportSettings) -> bool:
|
||||
|
||||
@@ -30,9 +30,10 @@ from datetime import datetime, timedelta, timezone
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerifyMismatchError
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy import delete, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.user import AdminUser, Session
|
||||
@@ -47,6 +48,9 @@ _DUMMY_HASH = _ph.hash(secrets.token_urlsafe(32))
|
||||
|
||||
SESSION_TOKEN_BYTES = 32
|
||||
|
||||
# Serializes the first-start account creation of several backends (policy_lock and leader.py use 728194031-33).
|
||||
_BOOTSTRAP_LOCK_KEY = 728194034
|
||||
|
||||
|
||||
# ── Password hashing ─────────────────────────────────────────────────────
|
||||
|
||||
@@ -158,7 +162,7 @@ class _LoginThrottle:
|
||||
if not force and now - self._last_purge < self._PURGE_EVERY_SECONDS and len(self._records) < self.MAX_RECORDS:
|
||||
return
|
||||
self._last_purge = now
|
||||
window = settings.login_window_seconds
|
||||
window = runtime_settings.value("login_window_seconds")
|
||||
for key in [
|
||||
k for k, r in self._records.items()
|
||||
if r.locked_until <= now and all(now - t >= window for t in r.failures)
|
||||
@@ -179,11 +183,11 @@ class _LoginThrottle:
|
||||
def _record_key(self, key: str, max_attempts: int) -> None:
|
||||
now = time.monotonic()
|
||||
rec = self._records.setdefault(key, _AttemptRecord())
|
||||
window = settings.login_window_seconds
|
||||
window = runtime_settings.value("login_window_seconds")
|
||||
rec.failures = [t for t in rec.failures if now - t < window]
|
||||
rec.failures.append(now)
|
||||
if len(rec.failures) >= max_attempts:
|
||||
rec.locked_until = now + settings.login_lockout_seconds
|
||||
rec.locked_until = now + runtime_settings.value("login_lockout_seconds")
|
||||
rec.failures.clear()
|
||||
|
||||
def retry_after(self, ip: str, username: str) -> int:
|
||||
@@ -196,8 +200,8 @@ class _LoginThrottle:
|
||||
|
||||
def record_failure(self, ip: str, username: str) -> None:
|
||||
self._purge(time.monotonic())
|
||||
self._record_key(self._user_key(ip, username), settings.login_max_attempts)
|
||||
self._record_key(self._ip_key(ip), settings.login_ip_max_attempts)
|
||||
self._record_key(self._user_key(ip, username), runtime_settings.value("login_max_attempts"))
|
||||
self._record_key(self._ip_key(ip), runtime_settings.value("login_ip_max_attempts"))
|
||||
key = username.lower()
|
||||
count, _ = self._failures_since_success.get(key, (0, 0.0))
|
||||
self._failures_since_success[key] = (count + 1, time.monotonic())
|
||||
@@ -262,7 +266,7 @@ async def create_session(
|
||||
await db.rollback()
|
||||
raise api_error(401, ErrorCode.TOTP_REQUIRED)
|
||||
token = secrets.token_urlsafe(SESSION_TOKEN_BYTES)
|
||||
expires_at = _now() + timedelta(seconds=lifetime_seconds or settings.session_lifetime)
|
||||
expires_at = _now() + timedelta(seconds=lifetime_seconds or runtime_settings.value("session_lifetime_days"))
|
||||
session = Session(
|
||||
user_id=user_id,
|
||||
token_hash=_hash_token(token),
|
||||
@@ -352,9 +356,12 @@ async def finalize_login(db: AsyncSession, user: AdminUser, ip: str) -> None:
|
||||
# ── Bootstrap ─────────────────────────────────────────────────────────
|
||||
|
||||
async def bootstrap_admin(db: AsyncSession) -> None:
|
||||
"""Ensure the admin account exists."""
|
||||
"""Ensure the admin account exists. Two backends starting on an empty database at once (HA, a second stack) are
|
||||
serialized here; the second one then sees the account of the first."""
|
||||
await db.execute(text("SELECT pg_advisory_xact_lock(:key)"), {"key": _BOOTSTRAP_LOCK_KEY})
|
||||
existing = await db.execute(select(AdminUser.id).limit(1))
|
||||
if existing.first() is not None:
|
||||
await db.commit() # releases the lock
|
||||
return
|
||||
|
||||
password = settings.admin_password.strip()
|
||||
|
||||
@@ -32,6 +32,7 @@ from typing import Any, Awaitable, Callable
|
||||
from sqlalchemy import and_, or_
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.services.opnsense_client import OpnsenseAmbiguousError, OpnsenseAPIError
|
||||
|
||||
@@ -80,7 +81,7 @@ async def push_to_host(
|
||||
write: Callable[[str | None], Awaitable[str]],
|
||||
apply: Callable[[], Awaitable[Any]],
|
||||
) -> Any:
|
||||
window = timedelta(seconds=settings.deploy_confirm_window_seconds)
|
||||
window = timedelta(seconds=runtime_settings.value("deploy_confirm_window_seconds"))
|
||||
before = {name: getattr(existing, name) for name in _FIELDS} if existing else None
|
||||
# An armed "error" row (written, apply not confirmed) holds the UNCONFIRMED payload as last_payload: a retry must
|
||||
# keep the original rollback target instead of promoting that payload to it.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Heartbeat to an external watcher: who watches the watcher (docs/HA-PLAN.md, 2.7).
|
||||
|
||||
If OPNexus stops, every alert it would send stops with it. With HEARTBEAT_URL set, the leader's notification loop
|
||||
calls that URL after each successful check cycle; a watcher outside OPNexus (healthchecks.io style, ntfy, Uptime
|
||||
Kuma push monitor) raises the alarm when the calls stop. Only the leader sends (the loop runs only there), and only
|
||||
after a cycle that reached the database and finished its checks: a backend that is up but cut off from PostgreSQL
|
||||
must not report "fine".
|
||||
|
||||
The URL usually carries a secret (the check id), so it is validated like a notification target, called through the
|
||||
guarded client without redirects and never written to the log.
|
||||
"""
|
||||
import logging
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.services import url_guard
|
||||
from app.version import APP_VERSION
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TIMEOUT_SECONDS = 10.0
|
||||
_warned = False
|
||||
|
||||
|
||||
def configured_url() -> str | None:
|
||||
"""The validated URL, or None (off or invalid; an invalid one is reported once)."""
|
||||
global _warned
|
||||
raw = runtime_settings.value("heartbeat_url").strip()
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
url = url_guard.validate_url(raw, max_length=2000)
|
||||
if urlsplit(url).scheme != "https":
|
||||
raise ValueError("Die Adresse muss mit https:// beginnen")
|
||||
except ValueError as exc:
|
||||
if not _warned:
|
||||
logger.warning("HEARTBEAT_URL wird ignoriert: %s", exc)
|
||||
_warned = True
|
||||
return None
|
||||
return url
|
||||
|
||||
|
||||
async def ping() -> bool:
|
||||
url = configured_url()
|
||||
if url is None:
|
||||
return False
|
||||
with url_guard.muted():
|
||||
try:
|
||||
await url_guard.ensure_resolved_safe(url)
|
||||
async with url_guard.guarded_client(timeout=_TIMEOUT_SECONDS, follow_redirects=False) as client:
|
||||
(await client.get(url, headers={"User-Agent": f"OPNexus/{APP_VERSION}"})).raise_for_status()
|
||||
except ValueError as exc:
|
||||
logger.warning("Heartbeat nicht gesendet: %s", exc)
|
||||
return False
|
||||
except httpx.HTTPError as exc:
|
||||
logger.warning("Heartbeat fehlgeschlagen (%s)", url_guard.failure_reason(exc))
|
||||
return False
|
||||
return True
|
||||
@@ -16,6 +16,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import leader
|
||||
from app.models.carp_event import CarpEvent
|
||||
from app.models.compliance_result import ComplianceResult
|
||||
from app.models.host import OpnsenseHost
|
||||
@@ -24,7 +25,7 @@ from app.models.resource_sample import HostResourceSample
|
||||
from app.models.rollout import RolloutRun
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.routers.dashboard import collect_pending_actions
|
||||
from app.services import compliance_service, gateway_health_service, notification_settings_service
|
||||
from app.services import compliance_service, gateway_health_service, notification_checks, notification_settings_service
|
||||
from app.version import APP_VERSION
|
||||
|
||||
CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
|
||||
@@ -59,6 +60,12 @@ async def render(db: AsyncSession) -> str:
|
||||
|
||||
out.family("opnexus_info", "OPNexus version (always 1).")
|
||||
out.sample("opnexus_info", 1, version=APP_VERSION)
|
||||
out.family("opnexus_leader", "1 if this backend process runs the background loops (leader lock held), else 0.")
|
||||
out.sample("opnexus_leader", 1 if leader.current_role() == leader.LEADER else 0, role=leader.current_role())
|
||||
disk = notification_checks.server_disk_usage()
|
||||
if disk is not None:
|
||||
out.family("opnexus_server_disk_used_percent", "Used share of the disk holding OPNexus' Docker volumes (this backend's view).")
|
||||
out.sample("opnexus_server_disk_used_percent", round(disk[0], 1))
|
||||
|
||||
hosts = (await db.scalars(select(OpnsenseHost).order_by(OpnsenseHost.name))).all()
|
||||
latest_sample = (
|
||||
|
||||
@@ -142,6 +142,8 @@ async def check_gateway_down(db: AsyncSession) -> dict[str, dict]:
|
||||
# page-status-services: api/core/service/*; page-system-login-logout "Lobby: Dashboard": api/diagnostics/cpu_usage/*
|
||||
# and api/diagnostics/system/system_swap).
|
||||
API_READS: dict[str, tuple[str, str]] = {
|
||||
# Die Basis-Messungen selbst (Speicher, Last, Platte, Schnittstellen): ohne sie gibt es keinen einzigen Messwert.
|
||||
"base": ("Lobby: Dashboard", "/api/diagnostics/system/systemResources"),
|
||||
"gateways": ("System: Gateways", "/api/routes/gateway/status"),
|
||||
"services": ("Status: Services", "/api/core/service/search"),
|
||||
"swap": ("Lobby: Dashboard", "/api/diagnostics/system/systemSwap"),
|
||||
|
||||
@@ -30,6 +30,7 @@ import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.models.notification import CHANNEL_TYPES, NotificationChannel, NotificationProblem
|
||||
from app.services import notification_delivery, notification_secret_store, url_guard
|
||||
@@ -224,7 +225,7 @@ async def send(channel: NotificationChannel, problem: NotificationProblem, trans
|
||||
except ValueError as exc:
|
||||
raise NotificationDeliveryError(f"{label}: {exc}") from None
|
||||
try:
|
||||
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
async with url_guard.guarded_client(timeout=runtime_settings.value("opnsense_timeout"), follow_redirects=False) as client:
|
||||
resp = await client.request(method, url, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
if ok_key and not (resp.json() or {}).get(ok_key):
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
whatever's currently bad. Pure detection — no delivery, no state tracking
|
||||
here (see notification_dedup.py for the open/resolved diffing that turns
|
||||
these snapshots into actual notifications)."""
|
||||
import shutil
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
@@ -18,6 +19,7 @@ from app.models.outbound_nat_deployment import OutboundNatDeployment
|
||||
from app.models.port_forward_deployment import PortForwardDeployment
|
||||
from app.models.rule_deployment import RuleDeployment
|
||||
from app.models.scheduled_maintenance import ScheduledMaintenance
|
||||
from app.config import settings
|
||||
from app.services import backup_store, nat_drift_service, network_drift_service, rule_drift_service
|
||||
from app.services.notification_dedup import Problems
|
||||
from app.services.host_status_service import get_statuses
|
||||
@@ -465,3 +467,36 @@ async def check_rollout_start_failed(db: AsyncSession) -> dict[str, dict]:
|
||||
"details": _short(error.get("params")),
|
||||
}
|
||||
return problems
|
||||
|
||||
|
||||
def server_disk_usage() -> tuple[float, int, int] | None:
|
||||
"""(used %, used bytes, total bytes) of the file system that holds OPNexus' Docker volumes, measured on the
|
||||
firewall-backup volume the backend mounts (in a standard installation the database lives on the same disk)."""
|
||||
try:
|
||||
usage = shutil.disk_usage(settings.opnsense_backup_dir)
|
||||
except OSError:
|
||||
return None
|
||||
if not usage.total:
|
||||
return None
|
||||
return usage.used * 100.0 / usage.total, usage.used, usage.total
|
||||
|
||||
|
||||
async def check_server_disk(db: AsyncSession) -> dict[str, dict]:
|
||||
"""The disk OPNexus itself runs on. A full disk stops PostgreSQL and with it every other alert, so this one
|
||||
warns while there is still room. Only the leader checks (the loop runs only there)."""
|
||||
from app.services import notification_settings_service
|
||||
cfg = await notification_settings_service.get_settings(db)
|
||||
measured = server_disk_usage()
|
||||
if measured is None:
|
||||
unknown = Problems()
|
||||
unknown.preserve_prefixes.add("server") # not measurable this round: leave an open problem as it is
|
||||
return unknown
|
||||
pct, used, total = measured
|
||||
if pct < cfg.server_disk_threshold_pct:
|
||||
return {}
|
||||
return {"server": {
|
||||
"belegt_prozent": round(pct, 1), "schwelle_prozent": cfg.server_disk_threshold_pct,
|
||||
"frei_gb": round((total - used) / 1e9, 1), "gesamt_gb": round(total / 1e9, 1),
|
||||
"hinweis": "Platz schaffen (z. B. alte Images mit 'docker image prune', Log-Obergrenze unter Einstellungen › Logs senken) "
|
||||
"oder die Platte vergrößern. Läuft sie voll, bleibt die Datenbank stehen.",
|
||||
}}
|
||||
@@ -14,6 +14,7 @@ from email.message import EmailMessage
|
||||
|
||||
import httpx
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.models.notification import NotificationProblem, NotificationSettings
|
||||
from app.services import notification_secret_store, url_guard
|
||||
@@ -41,6 +42,7 @@ _LABELS = {
|
||||
"resource_threshold": "Ressourcen-Schwelle überschritten",
|
||||
"service_down": "Dienst ausgefallen",
|
||||
"api_rights_missing": "API-Rechte fehlen",
|
||||
"server_disk": "Platte des OPNexus-Servers fast voll",
|
||||
"rollout_paused": "Rollout angehalten",
|
||||
"rollout_start_failed": "Rollout nicht automatisch gestartet",
|
||||
"test": "Testbenachrichtigung",
|
||||
@@ -97,7 +99,7 @@ async def send_webhook(settings_row: NotificationSettings, problem: Notification
|
||||
raise NotificationDeliveryError(f"Webhook-Adresse nicht erlaubt: {exc}") from None
|
||||
try:
|
||||
# follow_redirects stays off: a redirect must not move the request to an address the check above never saw.
|
||||
async with url_guard.guarded_client(timeout=settings.opnsense_timeout, follow_redirects=False) as client:
|
||||
async with url_guard.guarded_client(timeout=runtime_settings.value("opnsense_timeout"), follow_redirects=False) as client:
|
||||
with url_guard.muted():
|
||||
resp = await client.post(settings_row.webhook_url, content=raw, headers=headers)
|
||||
resp.raise_for_status()
|
||||
@@ -112,7 +114,7 @@ def _tls_context() -> ssl.SSLContext:
|
||||
"""Verifying context (certificate chain AND hostname). smtplib's own default
|
||||
when no context is passed is an *unverifying* one — the connection would be
|
||||
encrypted but could be impersonated."""
|
||||
return ssl.create_default_context(cafile=settings.smtp_ca_file or None)
|
||||
return runtime_settings.smtp_ssl_context()
|
||||
|
||||
|
||||
def _send_email_sync(settings_row: NotificationSettings, subject: str, body: str) -> None:
|
||||
@@ -130,7 +132,7 @@ def _send_email_sync(settings_row: NotificationSettings, subject: str, body: str
|
||||
message["To"] = ", ".join(settings_row.smtp_to_addresses)
|
||||
message.set_content(body)
|
||||
|
||||
timeout = settings.notification_smtp_timeout_seconds
|
||||
timeout = runtime_settings.value("notification_smtp_timeout_seconds")
|
||||
if settings_row.smtp_security == "ssl":
|
||||
client = smtplib.SMTP_SSL(
|
||||
settings_row.smtp_host, settings_row.smtp_port, timeout=timeout, context=_tls_context()
|
||||
|
||||
@@ -40,6 +40,7 @@ _CHECKS = [
|
||||
("resource_threshold", "resource_threshold_enabled", monitor_checks.check_resource_threshold),
|
||||
("service_down", "service_down_enabled", monitor_checks.check_service_down),
|
||||
("api_rights_missing", "api_rights_enabled", monitor_checks.check_api_rights_missing),
|
||||
("server_disk", "server_disk_enabled", notification_checks.check_server_disk),
|
||||
("rollout_paused", "rollout_problems_enabled", notification_checks.check_rollout_paused),
|
||||
("rollout_start_failed", "rollout_problems_enabled", notification_checks.check_rollout_start_failed),
|
||||
]
|
||||
|
||||
@@ -27,6 +27,7 @@ async def update_settings(db: AsyncSession, payload: NotificationSettingsUpdate)
|
||||
"revert_failed_enabled", "maintenance_unknown_enabled", "carp_health_enabled", "nat_drift_enabled",
|
||||
"network_drift_enabled", "carp_auto_revert_failed_enabled", "scheduled_maintenance_failed_enabled", "backup_failed_enabled", "compliance_alerts_enabled",
|
||||
"gateway_down_enabled", "resource_threshold_enabled", "service_down_enabled", "rollout_problems_enabled", "api_rights_enabled",
|
||||
"server_disk_enabled", "server_disk_threshold_pct",
|
||||
"gateway_loss_threshold_pct", "gateway_delay_threshold_ms", "disk_threshold_pct", "memory_threshold_pct",
|
||||
"swap_threshold_pct", "load_per_cpu_threshold", "alert_confirm_samples", "resource_confirm_samples",
|
||||
"service_watchlist",
|
||||
|
||||
@@ -18,6 +18,7 @@ from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.models.host import OpnsenseHost
|
||||
from app.services import firewall_url, opnsense_secret_store, overlay_service, tls_pin_service
|
||||
@@ -60,7 +61,7 @@ def _client(host: OpnsenseHost) -> httpx.AsyncClient:
|
||||
base_url=host.base_url.rstrip("/") + "/api",
|
||||
auth=(host.api_key, secret),
|
||||
verify=tls_pin_service.verify_argument(host.verify_tls, host.tls_ca_pem),
|
||||
timeout=settings.opnsense_timeout,
|
||||
timeout=runtime_settings.value("opnsense_timeout"),
|
||||
# Firewalls reached through the management overlay (WireGuard hub) go via the hub's CONNECT proxy.
|
||||
proxy=overlay_service.proxy_for(host.base_url),
|
||||
)
|
||||
|
||||
@@ -110,12 +110,22 @@ async def _optional(operation, name: str, denied: list[str], ok: list[str]):
|
||||
|
||||
|
||||
async def sample_host(db: AsyncSession, host: OpnsenseHost) -> None:
|
||||
resources = await client.get_system_resources(host)
|
||||
system_time = await client.get_system_time(host)
|
||||
disk = await client.get_system_disk(host)
|
||||
interfaces = await client.get_interface_statistics(host)
|
||||
denied: list[str] = []
|
||||
ok: list[str] = []
|
||||
try:
|
||||
resources = await client.get_system_resources(host)
|
||||
system_time = await client.get_system_time(host)
|
||||
disk = await client.get_system_disk(host)
|
||||
interfaces = await client.get_interface_statistics(host)
|
||||
ok.append("base")
|
||||
except OpnsenseForbiddenError:
|
||||
# Dem API-Schlüssel fehlen schon die Basis-Rechte (Dashboard): ohne Messwerte gäbe es nie einen Messpunkt und
|
||||
# der Rechte-Alarm könnte nicht auslösen. Ein leerer Punkt trägt den Befund ("base" in denied); die
|
||||
# Schwellenwert-Alarme sehen darin "unbekannt", nicht "in Ordnung".
|
||||
db.add(HostResourceSample(host_id=host.id, sampled_at=_now(), mem_used_pct=None, load_avg_1m=None,
|
||||
disk_used_pct=None, raw={"denied": ["base"], "ok": []}))
|
||||
logger.warning("Resource-Sampling für Host %s: dem API-Schlüssel fehlen die Basis-Rechte (403)", host.name)
|
||||
return
|
||||
swap, cpu_type, gateways, services = await asyncio.gather(
|
||||
_optional(client.get_system_swap(host), "swap", denied, ok),
|
||||
_optional(client.get_cpu_type(host), "cpu", denied, ok),
|
||||
|
||||
@@ -89,6 +89,7 @@ from sqlalchemy.orm.attributes import set_committed_value
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.errors import ErrorCode, api_error
|
||||
from app.models.governance import GovernanceSettings
|
||||
@@ -146,13 +147,13 @@ async def _commit(db: AsyncSession) -> None:
|
||||
|
||||
|
||||
def _window() -> timedelta:
|
||||
return timedelta(seconds=settings.deploy_confirm_window_seconds)
|
||||
return timedelta(seconds=runtime_settings.value("deploy_confirm_window_seconds"))
|
||||
|
||||
|
||||
def _budget(items: int, nodes: int) -> timedelta:
|
||||
"""Upper bound for the remote calls of one run: a toggle and a read-back per item, a sample, an apply and a
|
||||
read per node, each at most one client timeout."""
|
||||
return timedelta(seconds=(2 * items + 3 * nodes) * settings.opnsense_timeout)
|
||||
return timedelta(seconds=(2 * items + 3 * nodes) * runtime_settings.value("opnsense_timeout"))
|
||||
|
||||
|
||||
def _truthy(value) -> bool:
|
||||
@@ -805,7 +806,7 @@ async def preview(db: AsyncSession, host: OpnsenseHost, *, actor: str, actor_id=
|
||||
audit_service.ADMIN_ONLY_DETAILS: {"plan": plan}})))
|
||||
await db.commit()
|
||||
return {"preview_id": str(op.id), "expires_at": _iso(op.expires_at), "kind": kind, "plan": plan,
|
||||
"max_rules": limits.max_rules, "window_seconds": settings.deploy_confirm_window_seconds}
|
||||
"max_rules": limits.max_rules, "window_seconds": runtime_settings.value("deploy_confirm_window_seconds")}
|
||||
|
||||
|
||||
async def cancel(db: AsyncSession, host: OpnsenseHost, op_id, *, actor: str) -> dict:
|
||||
|
||||
@@ -13,6 +13,7 @@ from urllib.parse import quote, urlsplit
|
||||
import httpx
|
||||
|
||||
from app.config import settings
|
||||
from app.services import url_guard
|
||||
|
||||
|
||||
class S3Error(Exception):
|
||||
@@ -58,6 +59,10 @@ class S3Client:
|
||||
if parts.scheme == "http" and not settings.allow_insecure_offsite:
|
||||
# Credentials and (encrypted, but still) backups must not travel in the clear.
|
||||
raise S3Error("Der Endpunkt muss https:// sein")
|
||||
try:
|
||||
url_guard.validate_url(endpoint)
|
||||
except ValueError as exc: # z. B. localhost, Link-Local, Metadaten-Dienst
|
||||
raise S3Error(f"Ungültiger Endpunkt: {exc}") from None
|
||||
# httpx leaves a scheme's default port out of the Host header, so the signed host must not carry it either.
|
||||
port = parts.port
|
||||
default = {"https": 443, "http": 80}[parts.scheme]
|
||||
@@ -82,10 +87,11 @@ class S3Client:
|
||||
secret_key=self.secret_key, region=self.region, now=datetime.now(timezone.utc),
|
||||
)
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60) as client:
|
||||
# Verbindet nur zu Adressen, die die Prüfung bestanden haben (auch nach DNS-Rebinding und Weiterleitungen).
|
||||
async with url_guard.guarded_client(timeout=60) as client:
|
||||
return await client.request(method, url, content=body if method == "PUT" else None, headers=headers)
|
||||
except httpx.HTTPError as exc:
|
||||
raise S3Error(f"Verbindung fehlgeschlagen ({type(exc).__name__})") from None
|
||||
raise S3Error(f"Verbindung fehlgeschlagen ({url_guard.failure_reason(exc)})") from None
|
||||
|
||||
@staticmethod
|
||||
def _check(response: httpx.Response, what: str) -> None:
|
||||
|
||||
@@ -11,6 +11,7 @@ bekannte Stand bleibt erhalten, und nichts davon darf je einen Request
|
||||
scheitern lassen oder länger als `_TIMEOUT_SECONDS` blockieren.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
@@ -19,8 +20,10 @@ from datetime import datetime, timedelta, timezone
|
||||
import httpx
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app import runtime_settings
|
||||
from app.config import settings
|
||||
from app.models.update_check_settings import DEFAULT_FEED_URL, UpdateCheckSettings
|
||||
from app.services import url_guard
|
||||
from app.version import APP_VERSION
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -96,15 +99,19 @@ def _safe_url(value: object) -> str | None:
|
||||
|
||||
|
||||
async def _fetch(url: str) -> tuple[str, str | None]:
|
||||
async with httpx.AsyncClient(timeout=_TIMEOUT_SECONDS, follow_redirects=True, max_redirects=3) as http:
|
||||
response = await http.get(
|
||||
url,
|
||||
headers={"Accept": "application/json", "User-Agent": f"OPNexus/{APP_VERSION}"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
if len(response.content) > _MAX_BYTES:
|
||||
raise ValueError("response too large")
|
||||
data = response.json()
|
||||
# Geschützter Client: der Feed ist eine vom Admin einstellbare Adresse, Weiterleitungen eingeschlossen. Die
|
||||
# Antwort wird beim Lesen auf _MAX_BYTES begrenzt, nicht erst danach.
|
||||
async with url_guard.guarded_client(timeout=_TIMEOUT_SECONDS, follow_redirects=True, max_redirects=3) as http:
|
||||
async with http.stream(
|
||||
"GET", url, headers={"Accept": "application/json", "User-Agent": f"OPNexus/{APP_VERSION}"},
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
body = bytearray()
|
||||
async for chunk in response.aiter_bytes():
|
||||
body += chunk
|
||||
if len(body) > _MAX_BYTES:
|
||||
raise ValueError("response too large")
|
||||
data = json.loads(body)
|
||||
tag = data["tag_name"]
|
||||
if not isinstance(tag, str):
|
||||
raise ValueError("tag_name is not a string")
|
||||
@@ -133,7 +140,7 @@ async def check(db: AsyncSession) -> dict:
|
||||
reset_cache()
|
||||
_state.url = url
|
||||
now = datetime.now(timezone.utc)
|
||||
interval = timedelta(hours=max(1, settings.update_check_interval_hours))
|
||||
interval = timedelta(hours=max(1, runtime_settings.value("update_check_interval_hours")))
|
||||
fresh = _state.checked_at is not None and now - _state.checked_at < interval
|
||||
backing_off = _state.error is not None and _state.attempted_at is not None and now - _state.attempted_at < _RETRY_AFTER
|
||||
if not fresh and not backing_off:
|
||||
|
||||
@@ -136,6 +136,19 @@ async def ensure_resolved_safe(url: str) -> None:
|
||||
raise ValueError("Der Name zeigt auf eine nicht erlaubte Adresse (lokal, Link-Local oder Metadaten-Dienst)")
|
||||
|
||||
|
||||
async def checked_addresses(host: str, port: int | None = None) -> list[str]:
|
||||
"""Addresses a non-HTTP target (syslog) may be connected to, all of them checked. Raises ValueError for a
|
||||
forbidden target and OSError when the name does not resolve. The caller connects to these addresses, not to the
|
||||
name, so a second answer from DNS cannot change the destination."""
|
||||
bare = (host or "").strip().strip("[]")
|
||||
if not bare or bare.lower() in ("localhost", "localhost.localdomain"):
|
||||
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
|
||||
addresses = await resolve_host(bare, port)
|
||||
if not addresses or _any_forbidden(addresses):
|
||||
raise ValueError("Diese Adresse (lokal, Link-Local oder Metadaten-Dienst) ist als Ziel nicht erlaubt")
|
||||
return addresses
|
||||
|
||||
|
||||
# ── connection-time guard (DNS rebinding) ─────────────────────────────────
|
||||
|
||||
BLOCKED_MESSAGE = "Zieladresse nicht erlaubt"
|
||||
|
||||
@@ -6,5 +6,5 @@ convention — but both are bumped together whenever a release ships).
|
||||
Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
"""
|
||||
|
||||
APP_VERSION = "1.33.0"
|
||||
APP_CODENAME = "Saker"
|
||||
APP_VERSION = "1.34.0"
|
||||
APP_CODENAME = "Griffon"
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Einstellungen im Webinterface statt in der .env (runtime_settings) und Plattenalarm des OPNexus-Servers.
|
||||
|
||||
Revision ID: 0032
|
||||
Revises: 0031
|
||||
Create Date: 2026-10-24
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0032"
|
||||
down_revision = "0031"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# IF NOT EXISTS: a pre-Alembic install is bridged with create_all() first, which may already have created these.
|
||||
op.execute(
|
||||
"CREATE TABLE IF NOT EXISTS runtime_settings ("
|
||||
" key varchar(100) PRIMARY KEY,"
|
||||
" value json,"
|
||||
" updated_at timestamptz NOT NULL DEFAULT now(),"
|
||||
" updated_by varchar(150))"
|
||||
)
|
||||
op.execute("ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS server_disk_enabled boolean NOT NULL DEFAULT true")
|
||||
op.execute("ALTER TABLE notification_settings ADD COLUMN IF NOT EXISTS server_disk_threshold_pct integer NOT NULL DEFAULT 90")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE notification_settings DROP COLUMN IF EXISTS server_disk_threshold_pct")
|
||||
op.execute("ALTER TABLE notification_settings DROP COLUMN IF EXISTS server_disk_enabled")
|
||||
op.execute("DROP TABLE IF EXISTS runtime_settings")
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/bin/sh
|
||||
# Startet das Backend als unprivilegierter Benutzer (uid 10001).
|
||||
#
|
||||
# Der Container beginnt als root, damit bestehende Installationen ihre Volumes
|
||||
# (bisher von root angelegt) selbst auf den Benutzer umstellen können: ein Update
|
||||
# braucht dadurch keinen Handgriff, und ein Rückweg auf eine ältere Version bleibt
|
||||
# möglich (root liest alles). Danach gibt `setpriv` root und alle Capabilities ab.
|
||||
#
|
||||
# OPNEXUS_RUN_AS_ROOT=1 lässt das Backend wie früher als root laufen (Notausgang
|
||||
# für Sonderfälle, z. B. Bind-Mounts, die sich nicht umstellen lassen).
|
||||
set -eu
|
||||
|
||||
APP_UID=10001
|
||||
APP_GID=10001
|
||||
|
||||
if [ "$(id -u)" = "0" ] && [ "${OPNEXUS_RUN_AS_ROOT:-0}" != "1" ]; then
|
||||
# Datenverzeichnisse: alles, was nicht dem App-Benutzer gehört, wird umgestellt.
|
||||
# Bei jedem Start, nicht nur einmal: Befehle wie `opnexus restore` laufen als
|
||||
# root und hinterlassen root-eigene Dateien.
|
||||
for d in \
|
||||
"${OPNSENSE_SECRET_DIR:-}" "${OPNSENSE_BACKUP_DIR:-}" "${IPSEC_PSK_SECRET_DIR:-}" \
|
||||
"${OPENVPN_KEY_SECRET_DIR:-}" "${WIREGUARD_KEY_SECRET_DIR:-}" "${CERT_KEY_SECRET_DIR:-}" \
|
||||
"${ACME_SECRET_DIR:-}" "${TOTP_SECRET_DIR:-}" "${NOTIFICATION_SECRET_DIR:-}" \
|
||||
"${OVERLAY_DIR:-}" /backups
|
||||
do
|
||||
[ -n "$d" ] && [ -d "$d" ] || continue
|
||||
find "$d" -xdev \( ! -user "$APP_UID" -o ! -group "$APP_GID" \) \
|
||||
-exec chown -h "$APP_UID:$APP_GID" {} + 2>/dev/null || true
|
||||
done
|
||||
# HOME zeigt sonst noch auf /root (nicht lesbar): asyncpg/libpq suchen dort ~/.postgresql.
|
||||
export HOME=/app
|
||||
exec setpriv --reuid="$APP_UID" --regid="$APP_GID" --clear-groups \
|
||||
--inh-caps=-all --bounding-set=-all --no-new-privs "$@"
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
@@ -71,6 +71,16 @@ async def _clean_tables():
|
||||
await conn.execute(table.delete())
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_runtime_settings():
|
||||
# Saved settings live in a module-level cache (app/runtime_settings.py); the table is cleaned after each test,
|
||||
# the cache must be too, or a value saved by one test changes the next.
|
||||
from app import runtime_settings
|
||||
runtime_settings._cache.clear()
|
||||
yield
|
||||
runtime_settings._cache.clear()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_login_throttle():
|
||||
# The throttle is a module-level singleton (by design — it must survive
|
||||
|
||||
@@ -102,7 +102,7 @@ async def test_sample_records_only_refused_reads_in_denied(client, respx_mock):
|
||||
# forbidden -> denied (sorted); a server error and a missing endpoint are "unknown", no finding about rights
|
||||
assert raw["denied"] == ["gateways", "services"]
|
||||
assert raw["gateways"] is None and raw["services"] is None and raw["swap_used_pct"] is None
|
||||
assert raw["ok"] == [] # nothing went through
|
||||
assert raw["ok"] == ["base"] # only the base reading went through
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -112,7 +112,7 @@ async def test_sample_denied_is_an_empty_list_when_nothing_is_refused(client, re
|
||||
_mock_resource_endpoints(respx_mock)
|
||||
_mock_optional_reads(respx_mock)
|
||||
raw = await _sample_raw()
|
||||
assert raw["denied"] == [] and raw["ok"] == ["cpu", "gateways", "services", "swap"]
|
||||
assert raw["denied"] == [] and raw["ok"] == ["base", "cpu", "gateways", "services", "swap"]
|
||||
# unreachable optional endpoints (connect error / timeout) are not refusals either
|
||||
_mock_optional_reads(respx_mock)
|
||||
respx_mock.get("https://fw1.test/api/routes/gateway/status").mock(side_effect=httpx.ReadTimeout("slow"))
|
||||
@@ -120,16 +120,49 @@ async def test_sample_denied_is_an_empty_list_when_nothing_is_refused(client, re
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_read_refusal_still_fails_the_sample(client, respx_mock):
|
||||
async def test_base_read_refusal_becomes_a_finding_instead_of_a_silent_failure(client, respx_mock):
|
||||
"""Fehlen schon die Basis-Rechte, gab es nie einen Messpunkt und der Rechte-Alarm konnte nicht auslösen."""
|
||||
_mock_vip(respx_mock)
|
||||
await _create_host(client, "fw1", "https://fw1.test")
|
||||
_mock_resource_endpoints(respx_mock)
|
||||
respx_mock.get("https://fw1.test/api/diagnostics/system/systemResources").respond(403)
|
||||
async with SessionLocal() as db:
|
||||
result = await resource_monitor_service.poll_all_hosts(db)
|
||||
assert result["hosts_failed"] == 0 and result["sampled"] == 1
|
||||
async with SessionLocal() as db:
|
||||
sample = (await db.scalars(select(HostResourceSample))).one()
|
||||
assert sample.raw == {"denied": ["base"], "ok": []}
|
||||
assert sample.mem_used_pct is None and sample.disk_used_pct is None # "unbekannt", nicht "in Ordnung"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_read_timeout_is_still_a_failed_sample_not_a_rights_finding(client, respx_mock):
|
||||
_mock_vip(respx_mock)
|
||||
await _create_host(client, "fw1", "https://fw1.test")
|
||||
_mock_resource_endpoints(respx_mock)
|
||||
respx_mock.get("https://fw1.test/api/diagnostics/system/systemResources").respond(500)
|
||||
async with SessionLocal() as db:
|
||||
result = await resource_monitor_service.poll_all_hosts(db)
|
||||
assert result["hosts_failed"] == 1 and result["sampled"] == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_rights_alert_opens_and_closes_again(client):
|
||||
host = await _create_host(client, "fw1", "https://fw1.test")
|
||||
hid = uuid.UUID(host["id"])
|
||||
key = f"{hid}:base"
|
||||
await _only("api_rights_enabled")
|
||||
await _add_samples(hid, [{"denied": ["base"], "ok": []}] * 3)
|
||||
problems = await _rights_problems()
|
||||
assert list(problems) == [key]
|
||||
assert problems[key]["fehlendes_recht"] == "Lobby: Dashboard" and problems[key]["endpunkt"].endswith("systemResources")
|
||||
async with SessionLocal() as db:
|
||||
await run_notification_checks(db)
|
||||
await _clear_samples()
|
||||
await _add_samples(hid, [{"denied": [], "ok": ["base"]}] * 3)
|
||||
assert dict(await _rights_problems()) == {}
|
||||
|
||||
|
||||
# ── The alert ────────────────────────────────────────────────────────────────
|
||||
|
||||
DENIED_GW = {"denied": ["gateways"], "ok": ["cpu", "services", "swap"], "gateways": None, "services": {"unbound": 1}, "cpu_threads": 4}
|
||||
|
||||
@@ -15,10 +15,17 @@ from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.audit_export import AuditExportSettings
|
||||
from app.services import audit_export_service as svc
|
||||
from app.services import notification_secret_store
|
||||
from app.services import notification_secret_store, url_guard
|
||||
from tests.test_api_tokens import _login
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _loopback_allowed(monkeypatch):
|
||||
"""Diese Tests sprechen echte Sockets auf 127.0.0.1 an; der Zieladress-Schutz selbst steht in
|
||||
tests/test_audit_export_guard.py."""
|
||||
monkeypatch.setattr(url_guard, "_forbidden", lambda ip: False)
|
||||
|
||||
|
||||
async def _event(n: int, *, age_s: float = 60, status="accepted", details=None) -> AuditEvent:
|
||||
async with SessionLocal() as db:
|
||||
event = AuditEvent(
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Zieladress-Schutz für den Audit-Export, Offsite-S3 und den Update-Feed (1.29-Rest): Loopback, Link-Local und
|
||||
Metadaten-Dienst sind als Ziel nie erlaubt, auch nicht über Weiterleitungen oder DNS-Antworten."""
|
||||
import pytest
|
||||
|
||||
from app.models.audit_export import AuditExportSettings
|
||||
from app.services import audit_export_service as svc
|
||||
from app.services import s3_client, update_check_service, url_guard
|
||||
|
||||
|
||||
def _row(**kw) -> AuditExportSettings:
|
||||
row = AuditExportSettings(id=1)
|
||||
for key, value in kw.items():
|
||||
setattr(row, key, value)
|
||||
return row
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("host", ["127.0.0.1", "localhost", "169.254.169.254", "::1", "0.0.0.0"])
|
||||
async def test_syslog_refuses_forbidden_targets(host):
|
||||
row = _row(syslog_host=host, syslog_port=514, syslog_protocol="udp")
|
||||
with pytest.raises(svc.ExportError, match="Syslog-Ziel nicht erlaubt"):
|
||||
await svc.send_syslog(row, [{"time": "t", "status": "accepted"}])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_syslog_refuses_a_name_that_resolves_to_loopback(monkeypatch):
|
||||
async def fake(host, port=None):
|
||||
return ["203.0.113.5", "127.0.0.1"]
|
||||
monkeypatch.setattr(url_guard, "resolve_host", fake)
|
||||
with pytest.raises(svc.ExportError, match="nicht erlaubt"):
|
||||
await svc.send_syslog(_row(syslog_host="logs.example.test", syslog_port=514, syslog_protocol="tcp"),
|
||||
[{"time": "t", "status": "accepted"}])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_syslog_connects_to_the_checked_address_not_the_name(monkeypatch):
|
||||
seen = {}
|
||||
|
||||
async def fake(host, port=None):
|
||||
return ["203.0.113.5"]
|
||||
|
||||
def send(row, lines, addresses):
|
||||
seen["addresses"] = addresses
|
||||
|
||||
monkeypatch.setattr(url_guard, "resolve_host", fake)
|
||||
monkeypatch.setattr(svc, "_send_syslog_sync", send)
|
||||
await svc.send_syslog(_row(syslog_host="logs.example.test", syslog_port=514, syslog_protocol="tcp"),
|
||||
[{"time": "t", "status": "accepted"}])
|
||||
assert seen["addresses"] == ["203.0.113.5"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("url", ["http://127.0.0.1:9/x", "http://localhost/x", "http://169.254.169.254/latest", "ftp://example.test/x"])
|
||||
async def test_webhook_refuses_forbidden_targets(url):
|
||||
with pytest.raises(svc.ExportError, match="Webhook-Ziel nicht erlaubt"):
|
||||
await svc.send_webhook(_row(webhook_url=url), [])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_webhook_refuses_a_rebound_name(monkeypatch):
|
||||
async def fake(host, port=None):
|
||||
return ["127.0.0.1"]
|
||||
monkeypatch.setattr(url_guard, "resolve_host", fake)
|
||||
with pytest.raises(svc.ExportError, match="nicht erlaubt"):
|
||||
await svc.send_webhook(_row(webhook_url="https://hooks.example.test/a"), [])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("endpoint", ["https://127.0.0.1:9000", "https://localhost:9000", "https://169.254.169.254"])
|
||||
def test_s3_endpoint_must_not_be_local(endpoint):
|
||||
with pytest.raises(s3_client.S3Error, match="Ungültiger Endpunkt"):
|
||||
s3_client.S3Client(endpoint=endpoint, region="", bucket="b", access_key="a", secret_key="s")
|
||||
|
||||
|
||||
def test_s3_private_lan_endpoint_is_fine():
|
||||
s3_client.S3Client(endpoint="https://10.0.0.5:9000", region="", bucket="b", access_key="a", secret_key="s")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_connection_is_refused_when_the_name_resolves_to_loopback(monkeypatch):
|
||||
async def fake(host, port=None):
|
||||
return ["127.0.0.1"]
|
||||
monkeypatch.setattr(url_guard, "resolve_host", fake)
|
||||
client = s3_client.S3Client(endpoint="https://s3.example.test", region="", bucket="b", access_key="a", secret_key="s")
|
||||
with pytest.raises(s3_client.S3Error, match="Zieladresse nicht erlaubt"):
|
||||
await client.put("k", b"x")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_feed_refuses_a_rebound_name(monkeypatch):
|
||||
async def fake(host, port=None):
|
||||
return ["169.254.169.254"]
|
||||
monkeypatch.setattr(url_guard, "resolve_host", fake)
|
||||
with pytest.raises(Exception):
|
||||
await update_check_service._fetch("https://updates.example.test/latest.json")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_feed_is_capped_while_reading_not_afterwards(monkeypatch):
|
||||
"""Eine endlose Antwort wird nach _MAX_BYTES abgebrochen, ohne dass sie komplett im Speicher landet."""
|
||||
import asyncio
|
||||
|
||||
sent = {"bytes": 0}
|
||||
|
||||
async def handle(reader, writer):
|
||||
await reader.readuntil(b"\r\n\r\n")
|
||||
writer.write(b"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ntransfer-encoding: chunked\r\n\r\n")
|
||||
chunk = b"x" * 65536
|
||||
try:
|
||||
for _ in range(2000): # ~130 MB angeboten
|
||||
writer.write(f"{len(chunk):x}\r\n".encode() + chunk + b"\r\n")
|
||||
await writer.drain()
|
||||
sent["bytes"] += len(chunk)
|
||||
except (ConnectionError, OSError):
|
||||
pass
|
||||
finally:
|
||||
writer.close()
|
||||
|
||||
server = await asyncio.start_server(handle, "127.0.0.1", 0)
|
||||
port = server.sockets[0].getsockname()[1]
|
||||
monkeypatch.setattr(url_guard, "_forbidden", lambda ip: False)
|
||||
try:
|
||||
with pytest.raises(ValueError, match="too large"):
|
||||
await update_check_service._fetch(f"http://127.0.0.1:{port}/latest.json")
|
||||
finally:
|
||||
server.close()
|
||||
assert sent["bytes"] < 10_000_000
|
||||
@@ -140,3 +140,46 @@ def test_command_line_reads_the_passphrase_from_the_environment(data, monkeypatc
|
||||
monkeypatch.setenv("FULL_BACKUP_PASSPHRASE", "")
|
||||
assert fb._main(["full_backup", "create"]) == 1
|
||||
assert fb._main(["full_backup", "bogus"]) == 2
|
||||
|
||||
|
||||
def test_a_secret_changing_during_the_backup_triggers_a_new_attempt(data):
|
||||
calls = []
|
||||
|
||||
def dump(path: Path) -> None:
|
||||
calls.append(1)
|
||||
if len(calls) == 1: # ein Objekt wird mitten in der Sicherung angelegt
|
||||
(data / "opnsense_secrets" / "host-2.secret").write_text("API-SECRET-2")
|
||||
path.write_bytes(b"PGDMP-fake-database")
|
||||
|
||||
blob = fb.create(PASS, dump=dump, revision="0009")
|
||||
assert len(calls) == 2
|
||||
for f in data.rglob("*"):
|
||||
if f.is_file():
|
||||
f.unlink()
|
||||
fb.restore(blob, PASS, restore_db=lambda p: None, head="0012")
|
||||
assert (data / "opnsense_secrets" / "host-2.secret").read_text() == "API-SECRET-2"
|
||||
|
||||
|
||||
def test_a_backup_that_never_holds_still_is_refused_instead_of_written_inconsistent(data):
|
||||
n = []
|
||||
|
||||
def dump(path: Path) -> None:
|
||||
n.append(1)
|
||||
(data / "totp_secrets" / f"u{len(n)}.secret").write_text("x")
|
||||
path.write_bytes(b"PGDMP-fake-database")
|
||||
|
||||
with pytest.raises(fb.FullBackupError, match="Geheimnisse geändert"):
|
||||
fb.create(PASS, dump=dump, revision="0009")
|
||||
assert len(n) == fb.ATTEMPTS
|
||||
|
||||
|
||||
def test_new_firewall_config_backups_do_not_count_as_a_change(data):
|
||||
calls = []
|
||||
|
||||
def dump(path: Path) -> None:
|
||||
calls.append(1)
|
||||
(data / "opnsense_backups" / "host-1" / f"new-{len(calls)}.xml").write_text("<opnsense/>")
|
||||
path.write_bytes(b"PGDMP-fake-database")
|
||||
|
||||
fb.create(PASS, dump=dump, revision="0009")
|
||||
assert len(calls) == 1
|
||||
@@ -0,0 +1,110 @@
|
||||
"""H1: the full backup can carry files of the server itself (.env, the proxy's CA) for a host move.
|
||||
|
||||
They come from the CLI as a tar on stdin, live under host/ in the archive and only ever leave it again as a tar for
|
||||
the CLI (``host-files``); ``restore`` never writes them, and nothing outside env/ and caddy-ca/ is accepted.
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
import tarfile
|
||||
|
||||
import pytest
|
||||
|
||||
from app import full_backup as fb
|
||||
from tests.test_full_backup import PASS, _dump, data # noqa: F401 (fixture)
|
||||
|
||||
|
||||
def _tar(entries: dict[str, bytes | None], *, links: dict[str, str] | None = None) -> bytes:
|
||||
out = io.BytesIO()
|
||||
with tarfile.open(fileobj=out, mode="w") as tar:
|
||||
for name, content in entries.items():
|
||||
info = tarfile.TarInfo(name)
|
||||
if content is None:
|
||||
info.type = tarfile.DIRTYPE
|
||||
tar.addfile(info)
|
||||
else:
|
||||
info.size = len(content)
|
||||
tar.addfile(info, io.BytesIO(content))
|
||||
for name, target in (links or {}).items():
|
||||
info = tarfile.TarInfo(name)
|
||||
info.type, info.linkname = tarfile.SYMTYPE, target
|
||||
tar.addfile(info)
|
||||
return out.getvalue()
|
||||
|
||||
|
||||
HOST = _tar({
|
||||
"./env": b"POSTGRES_PASSWORD=pw-of-old-server\nHEARTBEAT_URL=https://hc.example/x\n",
|
||||
"./caddy-ca": None,
|
||||
"./caddy-ca/local": None,
|
||||
"./caddy-ca/local/root.crt": b"-----BEGIN CERTIFICATE-----CA",
|
||||
"./caddy-ca/local/root.key": b"-----BEGIN EC PRIVATE KEY-----KEY",
|
||||
})
|
||||
|
||||
|
||||
def _host_tar_names(raw: bytes) -> dict[str, bytes | None]:
|
||||
with tarfile.open(fileobj=io.BytesIO(raw), mode="r:") as tar:
|
||||
return {m.name: (tar.extractfile(m).read() if m.isfile() else None) for m in tar.getmembers()}
|
||||
|
||||
|
||||
def test_host_files_round_trip_and_are_encrypted(data): # noqa: F811
|
||||
blob = fb.create(PASS, dump=_dump, revision="0009", host_files=HOST)
|
||||
assert b"pw-of-old-server" not in blob and b"PRIVATE KEY" not in blob
|
||||
assert fb.info(blob, PASS)["host"] == ["caddy-ca", "env"]
|
||||
files = _host_tar_names(fb.host_files(blob, PASS))
|
||||
assert files["env"].startswith(b"POSTGRES_PASSWORD=pw-of-old-server")
|
||||
assert files["caddy-ca/local/root.key"] == b"-----BEGIN EC PRIVATE KEY-----KEY"
|
||||
|
||||
|
||||
def test_restore_never_writes_host_files(data): # noqa: F811
|
||||
blob = fb.create(PASS, dump=_dump, revision="0009", host_files=HOST)
|
||||
before = {p for p in data.rglob("*")}
|
||||
fb.restore(blob, PASS, restore_db=lambda p: None, head="0012")
|
||||
created = {p for p in data.rglob("*")} - before
|
||||
assert not [p for p in created if "env" in p.name or "caddy" in str(p)]
|
||||
|
||||
|
||||
def test_a_backup_without_host_files_hands_back_an_empty_tar(data): # noqa: F811
|
||||
blob = fb.create(PASS, dump=_dump, revision="0009")
|
||||
assert "host" not in fb.info(blob, PASS)
|
||||
assert _host_tar_names(fb.host_files(blob, PASS)) == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", [
|
||||
_tar({"etc/passwd": b"x"}), # outside env/ and caddy-ca/
|
||||
_tar({"caddy-ca/../../etc/x": b"x"}), # escaping
|
||||
_tar({"/env": b"x"}), # absolute
|
||||
_tar({}, links={"env": "/etc/shadow"}), # symlink
|
||||
b"not a tar at all",
|
||||
_tar({}), # empty
|
||||
])
|
||||
def test_hostile_or_empty_host_files_are_refused_before_the_dump(data, bad): # noqa: F811
|
||||
dumped = []
|
||||
with pytest.raises(fb.FullBackupError):
|
||||
fb.create(PASS, dump=lambda p: dumped.append(p), revision="0009", host_files=bad)
|
||||
assert dumped == []
|
||||
|
||||
|
||||
def test_oversized_host_files_are_refused(data, monkeypatch): # noqa: F811
|
||||
monkeypatch.setattr(fb, "_HOST_MAX_BYTES", 100)
|
||||
with pytest.raises(fb.FullBackupError, match="zu groß"):
|
||||
fb.create(PASS, dump=_dump, revision="0009", host_files=HOST)
|
||||
|
||||
|
||||
def test_command_line_create_with_host_files_and_extract(data, monkeypatch, capsysbinary): # noqa: F811
|
||||
monkeypatch.setenv("FULL_BACKUP_PASSPHRASE", PASS)
|
||||
monkeypatch.setattr(fb, "_dump_database", _dump)
|
||||
monkeypatch.setattr(fb, "_head_revision", lambda: "0009")
|
||||
|
||||
class Stdin:
|
||||
def __init__(self, raw):
|
||||
self.buffer = io.BytesIO(raw)
|
||||
|
||||
monkeypatch.setattr("sys.stdin", Stdin(HOST))
|
||||
assert fb._main(["full_backup", "create", "--host-files"]) == 0
|
||||
blob = capsysbinary.readouterr().out
|
||||
monkeypatch.setattr("sys.stdin", Stdin(blob))
|
||||
assert fb._main(["full_backup", "host-files"]) == 0
|
||||
assert "env" in _host_tar_names(capsysbinary.readouterr().out)
|
||||
monkeypatch.setattr("sys.stdin", Stdin(blob))
|
||||
assert fb._main(["full_backup", "info"]) == 0
|
||||
assert json.loads(capsysbinary.readouterr().out)["host"] == ["caddy-ca", "env"]
|
||||
assert fb._main(["full_backup", "restore", "--host-files"]) == 2 # the flag belongs to create only
|
||||
@@ -0,0 +1,341 @@
|
||||
"""H0 (docs/HA-PLAN.md): only the process holding the leader lock runs the background loops.
|
||||
|
||||
These run against the real test database: the lock, its loss when the session is terminated and the takeover are
|
||||
PostgreSQL behaviour, not something a mock would prove.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
|
||||
from app import leader, main
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal, engine
|
||||
from app.services import auth_service, heartbeat_service, url_guard
|
||||
|
||||
FAST = {"check_interval": 0.2, "check_timeout": 1.0, "retry_interval": 0.2, "takeover_delay": 0.6}
|
||||
|
||||
|
||||
class Tracker:
|
||||
"""Loop bodies that record which gate is running an iteration right now."""
|
||||
|
||||
def __init__(self):
|
||||
self.running: set[str] = set()
|
||||
self.max_concurrent = 0
|
||||
self.iterations: dict[str, int] = {}
|
||||
|
||||
def loop(self, name: str, gate_ref: dict, iteration_seconds: float = 0.05):
|
||||
async def body():
|
||||
while True:
|
||||
await gate_ref["gate"].checkpoint()
|
||||
self.running.add(name)
|
||||
self.max_concurrent = max(self.max_concurrent, len(self.running))
|
||||
self.iterations[name] = self.iterations.get(name, 0) + 1
|
||||
try:
|
||||
await asyncio.sleep(iteration_seconds)
|
||||
finally:
|
||||
self.running.discard(name)
|
||||
return body
|
||||
|
||||
|
||||
def _gate(tracker: Tracker, name: str, iteration_seconds: float = 0.05, **kw) -> leader.LeaderGate:
|
||||
ref: dict = {}
|
||||
gate = leader.LeaderGate({"work": tracker.loop(name, ref, iteration_seconds)}, **{**FAST, **kw})
|
||||
ref["gate"] = gate
|
||||
return gate
|
||||
|
||||
|
||||
async def _until(predicate, timeout=10.0):
|
||||
deadline = asyncio.get_running_loop().time() + timeout
|
||||
while not predicate():
|
||||
if asyncio.get_running_loop().time() > deadline:
|
||||
raise AssertionError("condition not reached in time")
|
||||
await asyncio.sleep(0.05)
|
||||
|
||||
|
||||
async def _stop(*gates_and_tasks):
|
||||
for gate, task in gates_and_tasks:
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
await gate.stop()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_single_gate_waits_the_takeover_delay_then_runs_the_loops():
|
||||
tracker = Tracker()
|
||||
gate = _gate(tracker, "a")
|
||||
task = asyncio.create_task(gate.run())
|
||||
try:
|
||||
await _until(lambda: gate.role == leader.TAKEOVER)
|
||||
assert tracker.iterations == {} # nothing runs during the takeover wait
|
||||
await _until(lambda: gate.role == leader.LEADER)
|
||||
await _until(lambda: tracker.iterations.get("a", 0) >= 3)
|
||||
finally:
|
||||
await _stop((gate, task))
|
||||
assert gate.role == leader.STANDBY
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_second_gate_stays_in_standby_and_runs_nothing():
|
||||
tracker = Tracker()
|
||||
a, b = _gate(tracker, "a"), _gate(tracker, "b")
|
||||
ta = asyncio.create_task(a.run())
|
||||
try:
|
||||
await _until(lambda: a.role == leader.LEADER)
|
||||
tb = asyncio.create_task(b.run())
|
||||
try:
|
||||
await _until(lambda: b.role == leader.STANDBY)
|
||||
await asyncio.sleep(1.0)
|
||||
assert b.role == leader.STANDBY and "b" not in tracker.iterations
|
||||
assert tracker.max_concurrent == 1
|
||||
finally:
|
||||
await _stop((b, tb))
|
||||
finally:
|
||||
await _stop((a, ta))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_terminated_lock_session_stops_the_loops_and_the_other_takes_over_without_overlap():
|
||||
"""The review's case: the connection dies under a living process. The loops must stop before anyone else runs."""
|
||||
tracker = Tracker()
|
||||
a, b = _gate(tracker, "a"), _gate(tracker, "b")
|
||||
ta = asyncio.create_task(a.run())
|
||||
tb = None
|
||||
try:
|
||||
await _until(lambda: a.role == leader.LEADER and tracker.iterations.get("a", 0) > 0)
|
||||
tb = asyncio.create_task(b.run())
|
||||
await _until(lambda: b.role == leader.STANDBY)
|
||||
pid = await a._conn.scalar(text("SELECT pg_backend_pid()"))
|
||||
await a._conn.commit()
|
||||
async with engine.connect() as admin:
|
||||
assert await admin.scalar(text("SELECT pg_terminate_backend(:pid)"), {"pid": pid})
|
||||
await _until(lambda: b.role == leader.LEADER and tracker.iterations.get("b", 0) > 0)
|
||||
assert a.role != leader.LEADER
|
||||
assert "a" not in tracker.running
|
||||
assert tracker.max_concurrent == 1, "both gates ran an iteration at the same time"
|
||||
finally:
|
||||
if tb is not None:
|
||||
await _stop((b, tb))
|
||||
await _stop((a, ta))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_long_iteration_is_cancelled_when_the_lock_is_lost():
|
||||
"""A notification cycle can take half a minute; it must not keep running after the lock is gone."""
|
||||
tracker = Tracker()
|
||||
# The old leader notices its loss only at its next check (up to 1.5 s + timeout), while the new one applies every
|
||||
# 0.1 s: only the takeover delay (default: check interval + timeout + margin) keeps the two apart.
|
||||
slow = {"check_interval": 1.5, "takeover_delay": None}
|
||||
a = _gate(tracker, "a", iteration_seconds=60, **slow)
|
||||
b = _gate(tracker, "b", iteration_seconds=60, retry_interval=0.1, **slow)
|
||||
ta = asyncio.create_task(a.run())
|
||||
tb = None
|
||||
try:
|
||||
await _until(lambda: "a" in tracker.running, timeout=15)
|
||||
tb = asyncio.create_task(b.run())
|
||||
await _until(lambda: b.role == leader.STANDBY)
|
||||
pid = await a._conn.scalar(text("SELECT pg_backend_pid()"))
|
||||
await a._conn.commit()
|
||||
async with engine.connect() as admin:
|
||||
await admin.scalar(text("SELECT pg_terminate_backend(:pid)"), {"pid": pid})
|
||||
await _until(lambda: "a" not in tracker.running, timeout=5)
|
||||
await _until(lambda: "b" in tracker.running, timeout=15)
|
||||
assert tracker.max_concurrent == 1
|
||||
finally:
|
||||
if tb is not None:
|
||||
await _stop((b, tb))
|
||||
await _stop((a, ta))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_checkpoint_refuses_outside_leadership():
|
||||
tracker = Tracker()
|
||||
gate = _gate(tracker, "a")
|
||||
with pytest.raises(leader.LeadershipLost):
|
||||
await gate.checkpoint()
|
||||
# Without a gate (a test driving a loop directly) the module-level checkpoint is a no-op.
|
||||
leader.gate = None
|
||||
await leader.checkpoint()
|
||||
assert leader.current_role() == leader.STARTING
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stop_releases_the_lock_at_once():
|
||||
tracker = Tracker()
|
||||
a, b = _gate(tracker, "a"), _gate(tracker, "b")
|
||||
ta = asyncio.create_task(a.run())
|
||||
await _until(lambda: a.role == leader.LEADER)
|
||||
await _stop((a, ta))
|
||||
async with engine.connect() as conn:
|
||||
assert await conn.scalar(text("SELECT pg_try_advisory_lock(:k)"), {"k": leader.LOCK_KEY})
|
||||
await conn.execute(text("SELECT pg_advisory_unlock(:k)"), {"k": leader.LOCK_KEY})
|
||||
tb = asyncio.create_task(b.run())
|
||||
try:
|
||||
await _until(lambda: b.role == leader.LEADER)
|
||||
finally:
|
||||
await _stop((b, tb))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_crashed_loop_is_restarted_and_lost_leadership_is_not_mistaken_for_a_crash():
|
||||
starts = []
|
||||
|
||||
async def flaky():
|
||||
starts.append(1)
|
||||
await asyncio.sleep(0.05)
|
||||
if len(starts) == 1:
|
||||
raise RuntimeError("boom")
|
||||
await asyncio.sleep(3600)
|
||||
|
||||
gate = leader.LeaderGate({"flaky": flaky}, **FAST)
|
||||
task = asyncio.create_task(gate.run())
|
||||
try:
|
||||
await _until(lambda: len(starts) >= 2)
|
||||
assert gate.role == leader.LEADER
|
||||
finally:
|
||||
await _stop((gate, task))
|
||||
|
||||
|
||||
class _ReplicaConn:
|
||||
"""A connection to a database in recovery: pg_try_advisory_lock would succeed there, so it must never be asked."""
|
||||
|
||||
def __init__(self):
|
||||
self.statements: list[str] = []
|
||||
|
||||
async def scalar(self, statement, params=None):
|
||||
self.statements.append(str(statement))
|
||||
return True
|
||||
|
||||
async def commit(self):
|
||||
pass
|
||||
|
||||
async def invalidate(self):
|
||||
pass
|
||||
|
||||
async def close(self):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_replica_never_becomes_leader():
|
||||
conn = _ReplicaConn()
|
||||
|
||||
async def connect():
|
||||
return conn
|
||||
|
||||
tracker = Tracker()
|
||||
ref: dict = {}
|
||||
gate = leader.LeaderGate({"work": tracker.loop("r", ref)}, connect=connect, **FAST)
|
||||
ref["gate"] = gate
|
||||
task = asyncio.create_task(gate.run())
|
||||
try:
|
||||
await _until(lambda: gate.role == leader.REPLICA)
|
||||
await asyncio.sleep(0.6)
|
||||
assert gate.role == leader.REPLICA and tracker.iterations == {}
|
||||
assert not any("advisory" in s for s in conn.statements)
|
||||
finally:
|
||||
await _stop((gate, task))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_and_metrics_report_the_role(client):
|
||||
leader.gate = None
|
||||
assert (await client.get("/api/health")).json() == {"status": "ok", "role": "starting"}
|
||||
tracker = Tracker()
|
||||
gate = _gate(tracker, "a")
|
||||
leader.gate = gate
|
||||
task = asyncio.create_task(gate.run())
|
||||
try:
|
||||
await _until(lambda: gate.role == leader.LEADER)
|
||||
assert (await client.get("/api/health")).json()["role"] == "leader"
|
||||
metrics = (await client.get("/api/metrics")).text
|
||||
assert 'opnexus_leader{role="leader"} 1' in metrics
|
||||
finally:
|
||||
await _stop((gate, task))
|
||||
leader.gate = None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bootstrap_admin_waits_for_a_concurrent_first_start():
|
||||
async with engine.connect() as other:
|
||||
await other.execute(text("SELECT pg_advisory_xact_lock(:k)"), {"k": auth_service._BOOTSTRAP_LOCK_KEY})
|
||||
async with SessionLocal() as db:
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(auth_service.bootstrap_admin(db), 0.5)
|
||||
await other.rollback() # releases the lock
|
||||
async with SessionLocal() as db:
|
||||
await asyncio.wait_for(auth_service.bootstrap_admin(db), 5)
|
||||
|
||||
|
||||
# ── Heartbeat ────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("value", ["", "http://hc.example.test/ping/abc", "https://127.0.0.1/ping", "https://localhost/x",
|
||||
"not a url"])
|
||||
def test_heartbeat_is_off_unless_a_valid_https_url_is_set(monkeypatch, value):
|
||||
monkeypatch.setattr(settings, "heartbeat_url", value)
|
||||
assert heartbeat_service.configured_url() is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_heartbeat_pings_and_never_logs_the_url(monkeypatch, respx_mock, caplog):
|
||||
url = "https://hc.example.test/ping/secret-check-id"
|
||||
monkeypatch.setattr(settings, "heartbeat_url", url)
|
||||
|
||||
async def public(host, port=None):
|
||||
return ["203.0.113.10"]
|
||||
monkeypatch.setattr(url_guard, "resolve_host", public)
|
||||
route = respx_mock.get(url).mock(return_value=httpx.Response(200))
|
||||
caplog.set_level(logging.DEBUG)
|
||||
assert await heartbeat_service.ping() is True
|
||||
assert route.call_count == 1
|
||||
route.mock(return_value=httpx.Response(500))
|
||||
assert await heartbeat_service.ping() is False
|
||||
assert "secret-check-id" not in caplog.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_heartbeat_refuses_a_name_that_resolves_to_loopback(monkeypatch):
|
||||
monkeypatch.setattr(settings, "heartbeat_url", "https://hc.example.test/ping/x")
|
||||
|
||||
async def rebound(host, port=None):
|
||||
return ["127.0.0.1"]
|
||||
monkeypatch.setattr(url_guard, "resolve_host", rebound)
|
||||
assert await heartbeat_service.ping() is False
|
||||
|
||||
|
||||
class _StopLoop(Exception):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("checks_fail", [False, True])
|
||||
async def test_the_notification_loop_sends_the_heartbeat_only_after_a_successful_cycle(monkeypatch, checks_fail):
|
||||
pings = []
|
||||
|
||||
async def checks(db):
|
||||
if checks_fail:
|
||||
raise RuntimeError("database gone")
|
||||
|
||||
async def ping():
|
||||
pings.append(1)
|
||||
return True
|
||||
|
||||
async def no_carp(db):
|
||||
return []
|
||||
|
||||
async def stop(_):
|
||||
raise _StopLoop()
|
||||
|
||||
leader.gate = None
|
||||
monkeypatch.setattr(main.notification_service, "run_notification_checks", checks)
|
||||
monkeypatch.setattr(main.heartbeat_service, "ping", ping)
|
||||
monkeypatch.setattr(main.carp_event_service, "detect_and_record", no_carp)
|
||||
monkeypatch.setattr(main.asyncio, "sleep", stop)
|
||||
with pytest.raises(_StopLoop):
|
||||
await main._notification_poll_loop()
|
||||
assert pings == ([] if checks_fail else [1])
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Settings › Betrieb: values that used to need the .env are set in the web interface (app/runtime_settings.py)."""
|
||||
import datetime as dt
|
||||
|
||||
import pytest
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
from cryptography.x509.oid import NameOID
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import runtime_settings as rs
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.runtime_setting import RuntimeSetting
|
||||
from app.services import notification_checks, notification_secret_store
|
||||
from tests.test_operator_role import _operator
|
||||
|
||||
|
||||
def _ca_pem() -> str:
|
||||
key = ec.generate_private_key(ec.SECP256R1())
|
||||
name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "Mail CA")])
|
||||
now = dt.datetime.now(dt.timezone.utc)
|
||||
cert = (x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
|
||||
.serial_number(1).not_valid_before(now).not_valid_after(now + dt.timedelta(days=30))
|
||||
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
|
||||
.sign(key, hashes.SHA256()))
|
||||
return cert.public_bytes(serialization.Encoding.PEM).decode()
|
||||
|
||||
|
||||
def _by_key(body) -> dict:
|
||||
return {item["key"]: item for item in body["settings"]}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def secret_dir(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "notification_secret_dir", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_defaults_are_listed_with_bounds_units_and_source(client):
|
||||
items = _by_key((await client.get("/api/settings/system")).json())
|
||||
assert set(items) == set(rs.BY_KEY)
|
||||
session = items["session_lifetime_days"]
|
||||
assert session == {**session, "value": 30, "default": 30, "unit": "days", "min": 1, "max": 365, "source": "default"}
|
||||
assert items["heartbeat_url"]["value"] is None and items["heartbeat_url"]["is_set"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_saving_changes_what_the_code_reads_and_is_audited(client):
|
||||
resp = await client.put("/api/settings/system", json={"values": {
|
||||
"session_lifetime_days": 7, "deploy_confirm_window_seconds": 300, "opnsense_timeout": 25.5}})
|
||||
assert resp.status_code == 200, resp.text
|
||||
items = _by_key(resp.json())
|
||||
assert items["session_lifetime_days"]["value"] == 7 and items["session_lifetime_days"]["source"] == "ui"
|
||||
assert rs.value("session_lifetime_days") == 7 * 86400 # the code keeps working in seconds
|
||||
assert rs.value("deploy_confirm_window_seconds") == 300
|
||||
assert rs.value("opnsense_timeout") == 25.5
|
||||
async with SessionLocal() as db:
|
||||
event = (await db.scalars(select(AuditEvent).where(AuditEvent.action == "system_settings"))).one()
|
||||
assert event.details["changes"]["session_lifetime_days"] == {"old": 30, "new": 7}
|
||||
# null resets to the default
|
||||
items = _by_key((await client.put("/api/settings/system", json={"values": {"session_lifetime_days": None}})).json())
|
||||
assert items["session_lifetime_days"]["source"] == "default" and rs.value("session_lifetime_days") == 30 * 86400
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("values", [
|
||||
{"deploy_confirm_window_seconds": 10}, # below the floor: the safety window must stay usable
|
||||
{"login_max_attempts": 1000},
|
||||
{"session_lifetime_days": 1.5},
|
||||
{"session_lifetime_days": "30"},
|
||||
{"no_such_setting": 1},
|
||||
{"heartbeat_url": "http://hc.example.test/x"}, # https only
|
||||
{"heartbeat_url": "https://127.0.0.1/x"},
|
||||
{"smtp_ca_pem": "not a certificate"},
|
||||
])
|
||||
async def test_invalid_values_are_refused_and_nothing_is_saved(client, values):
|
||||
resp = await client.put("/api/settings/system", json={"values": {"login_window_seconds": 600, **values}})
|
||||
assert resp.status_code == 422 and resp.json()["detail"]["code"] == "SETTING_INVALID"
|
||||
assert resp.json()["detail"]["params"]["setting"] in values
|
||||
async with SessionLocal() as db:
|
||||
assert (await db.scalars(select(RuntimeSetting))).all() == [] # the valid one in the same request neither
|
||||
assert rs.value("login_window_seconds") == settings.login_window_seconds
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_only_admins_change_settings(client):
|
||||
op = await _operator(client, "settings-oper")
|
||||
try:
|
||||
assert (await op.get("/api/settings/system")).status_code == 200
|
||||
resp = await op.put("/api/settings/system", json={"values": {"login_max_attempts": 9}})
|
||||
assert resp.status_code == 403
|
||||
finally:
|
||||
await op.aclose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_server_value_wins_and_cannot_be_overwritten(client, monkeypatch):
|
||||
monkeypatch.setenv("HEARTBEAT_URL", "https://hc.example.test/from-env")
|
||||
monkeypatch.setattr(settings, "heartbeat_url", "https://hc.example.test/from-env")
|
||||
item = _by_key((await client.get("/api/settings/system")).json())["heartbeat_url"]
|
||||
assert item["source"] == "server" and item["is_set"] is True and item["env"] == "HEARTBEAT_URL"
|
||||
resp = await client.put("/api/settings/system", json={"values": {"heartbeat_url": "https://hc.example.test/ui"}})
|
||||
assert resp.status_code == 422 and "HEARTBEAT_URL" in resp.json()["detail"]["params"]["reason"]
|
||||
assert rs.value("heartbeat_url") == "https://hc.example.test/from-env"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_empty_env_value_does_not_count_as_set(client, monkeypatch, secret_dir):
|
||||
monkeypatch.setenv("HEARTBEAT_URL", "")
|
||||
assert (await client.put("/api/settings/system", json={"values": {"heartbeat_url": "https://hc.example.test/ui"}})).status_code == 200
|
||||
assert rs.value("heartbeat_url") == "https://hc.example.test/ui"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_update_interval_from_an_old_compose_file_does_not_lock_the_field(client, monkeypatch):
|
||||
"""Compose files before 1.34 always pass UPDATE_CHECK_INTERVAL_HOURS=12; that must stay editable."""
|
||||
monkeypatch.setenv("UPDATE_CHECK_INTERVAL_HOURS", "12")
|
||||
item = _by_key((await client.get("/api/settings/system")).json())["update_check_interval_hours"]
|
||||
assert item["source"] == "default"
|
||||
assert (await client.put("/api/settings/system", json={"values": {"update_check_interval_hours": 3}})).status_code == 200
|
||||
assert rs.value("update_check_interval_hours") == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_heartbeat_url_is_a_secret(client, secret_dir):
|
||||
url = "https://hc.example.test/ping/secret-id"
|
||||
items = _by_key((await client.put("/api/settings/system", json={"values": {"heartbeat_url": url}})).json())
|
||||
assert items["heartbeat_url"] == {**items["heartbeat_url"], "value": None, "is_set": True, "source": "ui"}
|
||||
assert rs.value("heartbeat_url") == url
|
||||
assert notification_secret_store.read_secret("runtime-heartbeat_url") == url
|
||||
async with SessionLocal() as db:
|
||||
row = await db.get(RuntimeSetting, "heartbeat_url")
|
||||
events = (await db.scalars(select(AuditEvent).where(AuditEvent.action == "system_settings"))).all()
|
||||
assert row.value is None # the database only knows that it is set
|
||||
assert "secret-id" not in str([e.details for e in events])
|
||||
await client.put("/api/settings/system", json={"values": {"heartbeat_url": ""}})
|
||||
assert rs.value("heartbeat_url") == settings.heartbeat_url
|
||||
assert notification_secret_store.read_secret("runtime-heartbeat_url") is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_pasted_mail_ca_is_trusted_by_the_smtp_context(client):
|
||||
pem = _ca_pem()
|
||||
assert (await client.put("/api/settings/system", json={"values": {"smtp_ca_pem": pem}})).status_code == 200
|
||||
ctx = rs.smtp_ssl_context()
|
||||
assert any(c.get("subject") == ((("commonName", "Mail CA"),),) for c in ctx.get_ca_certs())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_another_process_sees_the_value_after_its_refresh(client):
|
||||
async with SessionLocal() as db:
|
||||
db.add(RuntimeSetting(key="login_lockout_seconds", value=900, updated_by="other-node"))
|
||||
await db.commit()
|
||||
assert rs.value("login_lockout_seconds") == settings.login_lockout_seconds # not yet: cache
|
||||
await rs.refresh()
|
||||
assert rs.value("login_lockout_seconds") == 900
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_login_throttle_follows_the_saved_limit(client, anon_client):
|
||||
await client.put("/api/settings/system", json={"values": {"login_max_attempts": 3}})
|
||||
statuses = []
|
||||
for _ in range(4):
|
||||
statuses.append((await anon_client.post("/api/auth/login", json={"username": settings.admin_username,
|
||||
"password": "wrong-password"})).status_code)
|
||||
assert statuses[-1] == 429 and statuses.count(401) == 3
|
||||
|
||||
|
||||
# ── server disk alert ────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_server_disk_alert_follows_the_threshold(client, monkeypatch):
|
||||
monkeypatch.setattr(notification_checks, "server_disk_usage", lambda: (91.5, 915, 1000))
|
||||
async with SessionLocal() as db:
|
||||
problems = await notification_checks.check_server_disk(db)
|
||||
assert problems["server"]["belegt_prozent"] == 91.5 and problems["server"]["schwelle_prozent"] == 90
|
||||
resp = await client.put("/api/notifications/settings", json={"server_disk_threshold_pct": 95})
|
||||
assert resp.status_code == 200 and resp.json()["server_disk_threshold_pct"] == 95
|
||||
async with SessionLocal() as db:
|
||||
assert await notification_checks.check_server_disk(db) == {}
|
||||
assert (await client.put("/api/notifications/settings", json={"server_disk_threshold_pct": 20})).status_code == 422
|
||||
monkeypatch.setattr(notification_checks, "server_disk_usage", lambda: None)
|
||||
async with SessionLocal() as db:
|
||||
unknown = await notification_checks.check_server_disk(db)
|
||||
assert unknown == {} and unknown.preserve_prefixes == {"server"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_server_disk_metric(client, monkeypatch):
|
||||
monkeypatch.setattr(notification_checks, "server_disk_usage", lambda: (81.25, 8125, 10000))
|
||||
assert "opnexus_server_disk_used_percent 81.2" in (await client.get("/api/metrics")).text
|
||||
@@ -117,10 +117,14 @@ services:
|
||||
TRUST_PROXY_HEADERS: ${TRUST_PROXY_HEADERS:-false}
|
||||
UPDATE_CHECK_URL: ${UPDATE_CHECK_URL:-}
|
||||
OPNEXUS_IMAGE_REPO: ${OPNEXUS_IMAGE_REPO:-}
|
||||
# 1 = Backend wie früher als root betreiben (Notausgang, siehe backend/scripts/start.sh).
|
||||
OPNEXUS_RUN_AS_ROOT: ${OPNEXUS_RUN_AS_ROOT:-0}
|
||||
ALLOW_INSECURE_FIREWALL_URLS: ${ALLOW_INSECURE_FIREWALL_URLS:-false}
|
||||
ALLOW_INSECURE_SMTP_AUTH: ${ALLOW_INSECURE_SMTP_AUTH:-false}
|
||||
SMTP_CA_FILE: ${SMTP_CA_FILE:-}
|
||||
UPDATE_CHECK_INTERVAL_HOURS: ${UPDATE_CHECK_INTERVAL_HOURS:-12}
|
||||
# Optionaler Heartbeat an einen externen Überwacher (nur https://), siehe Handbuch.
|
||||
HEARTBEAT_URL: ${HEARTBEAT_URL:-}
|
||||
# Optionaler Updater-Dienst (Profil "updater", siehe unten). Ohne ihn
|
||||
# zeigt das Webinterface nur an, wie man ihn aktiviert.
|
||||
UPDATER_URL: ${UPDATER_URL:-http://updater:8080}
|
||||
@@ -146,6 +150,9 @@ services:
|
||||
# Zugriffstoken des Updaters (nur lesen). Leer, solange der Updater
|
||||
# nicht aktiviert ist.
|
||||
- updater_state:/updater:ro
|
||||
# Das Backend startet als root nur, um die Volumes umzustellen, und läuft dann
|
||||
# als uid 10001 (backend/scripts/start.sh). Neue Rechte kann es nicht mehr erlangen.
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
# Bound to loopback only by default — reachable from this host, not
|
||||
# the network. Set BIND_ADDRESS in .env (e.g. to 0.0.0.0, or a
|
||||
# specific interface IP) for network-wide access instead of editing
|
||||
@@ -173,6 +180,10 @@ services:
|
||||
# not via loopback) but breaks any in-container check (e.g. the
|
||||
# healthcheck below) that queries localhost.
|
||||
HOSTNAME: "0.0.0.0"
|
||||
# Läuft als unprivilegierter Benutzer (frontend/Dockerfile) und braucht keine
|
||||
# Capabilities.
|
||||
cap_drop: ["ALL"]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
depends_on:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
|
||||
+107
-5
@@ -225,6 +225,26 @@ Internet ist das nicht geeignet.
|
||||
|
||||
---
|
||||
|
||||
## Betriebseinstellungen im Webinterface
|
||||
|
||||
Unter **Einstellungen › Betriebseinstellungen** stellst du ein, wofür früher die `.env` des Servers nötig war:
|
||||
|
||||
| Bereich | Einstellungen |
|
||||
|---|---|
|
||||
| Anmeldung und Sitzungen | Sitzungsdauer, Fehlversuche je Konto und je Absender, Zeitfenster, Sperrdauer |
|
||||
| Ausrollen und Firewall-Zugriff | Bestätigungsfrist beim Ausrollen, Zeitlimit für Firewall-Anfragen, Sicherungen vor Updates |
|
||||
| Messungen und Aufbewahrung | Messintervall Systemauslastung, Aufbewahrung der Messwerte, Intervall Regelverlauf, Intervall der Versionsprüfung |
|
||||
| Überwachung von außen und Mail | Heartbeat-Adresse, CA des Mailservers (PEM einfügen), Zeitlimit Mailversand |
|
||||
|
||||
Jedes Feld nennt Standard und erlaubten Bereich; „Auf Standard zurücksetzen“ nimmt eine Änderung zurück. Änderungen
|
||||
gelten sofort (spätestens nach 15 Sekunden auch in einem zweiten Backend) und stehen im Audit-Log; die Heartbeat-
|
||||
Adresse und die CA nur als „gesetzt“. Ändern dürfen nur Admins.
|
||||
|
||||
Steht ein Wert **in der `.env`**, gilt weiterhin dieser; das Feld ist dann gesperrt und zeigt „vom Server“. So bleiben
|
||||
automatisiert eingerichtete Server steuerbar. Bewusst **nur** in der `.env` bleiben: Adressen und TLS von OPNexus
|
||||
selbst (ein falscher Wert sperrt dich aus genau dieser Oberfläche aus), der Datenbank-Zugang, das erste Admin-Konto,
|
||||
die Zusatzdienste (`COMPOSE_PROFILES`) und die Notausgänge (`ALLOW_INSECURE_*`, `OPNEXUS_RUN_AS_ROOT`).
|
||||
|
||||
## Konfiguration (`.env`)
|
||||
|
||||
Alle Einstellungen stehen in der `.env` neben der `docker-compose.yml`. Nach einer
|
||||
@@ -243,10 +263,12 @@ Alle Einstellungen stehen in der `.env` neben der `docker-compose.yml`. Nach ein
|
||||
| `BIND_ADDRESS` | Schnittstelle für Backend und Frontend (Standard `127.0.0.1`). |
|
||||
| `BACKEND_PORT`, `FRONTEND_PORT` | Host-Ports (Standard 8100 und 3100), z. B. für eine zweite Instanz. |
|
||||
| `OPNEXUS_IMAGE_REPO`, `OPNEXUS_VERSION` | Registry und Version der Release-Images. `OPNEXUS_VERSION` ist die „Spur“, siehe [Updates](#opnexus-aktualisieren). |
|
||||
| `UPDATE_CHECK_URL`, `UPDATE_CHECK_INTERVAL_HOURS` | Hinweis auf neue Versionen, standardmäßig **aus**. Einstellbar auch im Webinterface; ein Wert in der `.env` hat Vorrang. |
|
||||
| `UPDATE_CHECK_URL`, `UPDATE_CHECK_INTERVAL_HOURS` | Hinweis auf neue Versionen, standardmäßig **aus**. Beides auch im Webinterface (Versionsprüfung bzw. Betriebseinstellungen). Die Adresse in der `.env` hat Vorrang; das Intervall dort ist nur der Standard, den das Webinterface überschreiben kann. |
|
||||
| `COMPOSE_PROFILES=updater` | Aktiviert den optionalen Updater für Updates per Klick. |
|
||||
| `HEARTBEAT_URL` | Heartbeat-Adresse, siehe [Ausfall von OPNexus bemerken](#ausfall-von-opnexus-bemerken-heartbeat). Einfacher im Webinterface (Betriebseinstellungen); ein Wert hier hat Vorrang. |
|
||||
| `OPNEXUS_RUN_AS_ROOT` | `1` lässt das Backend wie in älteren Versionen als root laufen. Normalerweise läuft es als unprivilegierter Benutzer (uid 10001) und stellt seine Datenvolumes beim Start selbst darauf um. Nur für Sonderfälle, etwa Datenverzeichnisse, die sich nicht umstellen lassen. |
|
||||
| `ALLOW_INSECURE_FIREWALL_URLS` | Erlaubt `http://`-Adressen für Firewalls. Nur für isolierte Labore. |
|
||||
| `SMTP_CA_FILE`, `ALLOW_INSECURE_SMTP_AUTH` | Eigene CA für den Mailserver bzw. SMTP-Anmeldung ohne Verschlüsselung (bleibt standardmäßig gesperrt). |
|
||||
| `SMTP_CA_FILE`, `ALLOW_INSECURE_SMTP_AUTH` | Eigene CA für den Mailserver (einfacher: PEM in den Betriebseinstellungen einfügen; die Datei hier hat Vorrang) bzw. SMTP-Anmeldung ohne Verschlüsselung (bleibt standardmäßig gesperrt). |
|
||||
|
||||
---
|
||||
|
||||
@@ -1700,6 +1722,38 @@ mit den Rechten des Tokens: ein „Lesen“-Token sieht auch hier keine Geheimni
|
||||
| `caddy_data`, `caddy_config` | Schlüssel der internen CA und Proxy-Konfiguration |
|
||||
| `updater_state` | Token des Updaters (nur mit aktivem Updater) |
|
||||
|
||||
### Ausfall von OPNexus bemerken (Heartbeat)
|
||||
|
||||
Fällt OPNexus aus, schweigt auch jeder Alarm, den es senden würde. Dagegen hilft ein Überwacher außerhalb: Trage
|
||||
unter **Einstellungen › Betriebseinstellungen › Überwachung von außen und Mail** die Ping-Adresse eines solchen
|
||||
Dienstes ein (healthchecks.io oder selbst gehostet, ein Push-Monitor in Uptime Kuma, ein ntfy-Thema). OPNexus ruft
|
||||
sie nach **jedem erfolgreichen Prüfzyklus** der Benachrichtigungen per `GET` auf, also etwa einmal pro Minute.
|
||||
Bleibt der Aufruf aus, ist OPNexus ausgefallen oder erreicht seine Datenbank nicht mehr; den Alarm schlägt dann der
|
||||
Überwacher. Stell dessen Karenzzeit auf ein Mehrfaches des Prüfintervalls (z. B. 5 Minuten).
|
||||
|
||||
Erlaubt ist nur `https://`; lokale Adressen, Link-Local und Metadaten-Dienste sind wie bei den Benachrichtigungen
|
||||
ausgeschlossen. Die Adresse enthält meist einen geheimen Schlüssel: sie wird nach dem Speichern nicht mehr angezeigt
|
||||
und nie protokolliert. Alternativ geht `HEARTBEAT_URL` in der `.env` (hat dann Vorrang).
|
||||
|
||||
**Platte des OPNexus-Servers.** Läuft die Platte voll, auf der OPNexus läuft, bleibt die Datenbank stehen und mit ihr
|
||||
jeder Alarm. Der Alarm „Platte des OPNexus-Servers fast voll“ (Einstellungen › Benachrichtigungen, standardmäßig an,
|
||||
ab 90 %) warnt vorher; die Metrik `opnexus_server_disk_used_percent` zeigt den Füllstand. Gemessen wird das
|
||||
Dateisystem der Docker-Volumes (bei einer Standardinstallation dieselbe Platte wie die Datenbank).
|
||||
|
||||
### Mehrere Backends an einer Datenbank
|
||||
|
||||
Die Hintergrunddienste (Rücknahme unbestätigter Änderungen, Benachrichtigungen, Log-Abruf, Messwerte, geplante
|
||||
Wartung und Rollouts, Sicherheitslage, Audit-Export, geplante Sicherungen, Regelverlauf) laufen immer nur in
|
||||
**einem** Backend-Prozess, dem *Leader*. Er hält dafür eine Sperre in der Datenbank. Ein zweites Backend an
|
||||
derselben Datenbank (etwa während eines Umzugs oder später als Reserve) beantwortet die API normal, wartet mit den
|
||||
Hintergrunddiensten aber in *Bereitschaft*. Fällt der Leader aus oder verliert er die Verbindung zur Datenbank,
|
||||
hält er seine Dienste sofort an; ein wartendes Backend übernimmt innerhalb einer halben Minute. An einer Datenbank, die
|
||||
gerade Replica ist, startet nie ein Leader.
|
||||
|
||||
Welche Rolle ein Backend hat, zeigen `GET /api/health` (`"role": "leader"`, `"standby"`, `"takeover"`, `"replica"`)
|
||||
und die Metrik `opnexus_leader`. Für eine einzelne Installation ändert sich nichts: ihr einziges Backend ist der
|
||||
Leader, die Hintergrunddienste starten nach dem Hochfahren mit etwa 12 Sekunden Verzögerung.
|
||||
|
||||
### Sichern und Wiederherstellen
|
||||
|
||||
**Automatisch:** Der Dienst `postgres-backup` schreibt beim Start und danach alle
|
||||
@@ -1714,9 +1768,47 @@ Konfigurationssicherungen) in **eine mit Passphrase verschlüsselte Datei** (`.o
|
||||
wird abgefragt (oder `OPNEXUS_BACKUP_PASSPHRASE`). Auf einer frischen Installation (gleiche oder
|
||||
neuere Version) spielt `./opnexus restore DATEI` alles zurück: Es prüft die Datei, fragt nach, stoppt die
|
||||
Dienste, legt eine Sicherheitskopie des jetzigen Zustands an, spielt Datenbank und Volumes ein und
|
||||
startet neu. Nicht enthalten: Log-Erfassung, Messwerte und `caddy_data` (die interne CA: nach einer
|
||||
Wiederherstellung auf neuer Hardware importierst du das CA-Zertifikat im Browser erneut).
|
||||
Datei und Passphrase getrennt vom Server aufbewahren.
|
||||
startet neu. Nicht enthalten: Log-Erfassung und Messwerte. Datei und Passphrase getrennt vom Server aufbewahren.
|
||||
|
||||
Mit `--with-env` und `--with-ca` nimmt die Vollsicherung zusätzlich die `.env` und die interne CA des Proxys
|
||||
(`caddy_data`) mit; `./opnexus restore DATEI --with-env --with-ca` spielt sie zurück. Von der `.env` übernimmt der
|
||||
Restore alles **außer** den Datenbank-Zugangsdaten, der Version und den Adressen des Zielservers (`POSTGRES_*`,
|
||||
`OPNEXUS_VERSION`, `OPNEXUS_IMAGE_REPO`, `OPNEXUS_SITES`, `OPNEXUS_DEFAULT_SNI`, `FRONTEND_ORIGIN`, `*_BIND_ADDRESS`,
|
||||
`OPNEXUS_GUI_SITES`); er zeigt die übernommenen Namen und fragt nach. Die gesicherte Datei liegt danach vollständig
|
||||
als `.env.from-backup` daneben, die vorherige als `.env.before-restore`. Mit der CA vertrauen Browser, die ihr schon
|
||||
vertrauen, auch dem neuen Server. `./opnexus status` zeigt, wann die letzte Vollsicherung entstand.
|
||||
|
||||
### Umzug und Notfall (Kaltreserve)
|
||||
|
||||
Fällt der OPNexus-Server aus, laufen die Firewalls unverändert weiter; nur Verwaltung, Alarme und geplante
|
||||
Wartungen stehen still. Für diesen Fall hält man eine **Kaltreserve**: eine Vollsicherung mit `.env` und CA und
|
||||
einen zweiten Server, auf dem OPNexus installiert werden kann.
|
||||
|
||||
**Notfall (der alte Server ist weg):** neuen Server nach [Installation aus dem Release-Bundle](#installation-aus-dem-release-bundle)
|
||||
mit gleicher oder neuerer Version einrichten, dann `./opnexus restore DATEI --with-env --with-ca`. Danach die Adresse
|
||||
(virtuelle IP, DNS-Name) auf den neuen Server zeigen lassen. Unbestätigte Änderungen, deren Frist während des Ausfalls
|
||||
abgelaufen ist, nimmt der neue Server sofort zurück: im Zweifel gilt der alte Stand.
|
||||
|
||||
**Geplanter Umzug (beide Server laufen):** am alten Server
|
||||
|
||||
```bash
|
||||
./opnexus migrate-host root@neuer-server # --dir PFAD, wenn OPNexus dort woanders liegt
|
||||
```
|
||||
|
||||
Voraussetzungen: Anmeldung per SSH-Schlüssel am Ziel, dort OPNexus installiert und gestartet (gleiche oder neuere
|
||||
Version). Das Werkzeug prüft das Ziel, hält den alten Server an (ab da keine Änderungen mehr), erstellt eine
|
||||
Vollsicherung mit `.env` und CA (zufällige Passphrase, die nur über die SSH-Verbindung geht), spielt sie am Ziel
|
||||
zurück, vergleicht die Zahl der Firewalls und meldet die Dauer. Im Lab dauerte ein Umzug mit drei Firewalls
|
||||
**90 Sekunden** von der Vorprüfung bis zum gesunden Ziel; der Rückweg auf den alten Server gut 30 Sekunden. Der alte Server bleibt angehalten, nicht gelöscht: zurück geht es mit `docker compose stop` am
|
||||
Ziel und `docker compose up -d` am alten Server.
|
||||
|
||||
**Danach zu prüfen:**
|
||||
- Firewalls, die ihre API nur bestimmten Absendern öffnen, müssen die Adresse des neuen Servers erlauben.
|
||||
- Mit [Management-Overlay](#firewalls-hinter-nat-management-tunnel) wählen sich Firewalls über eine öffentliche
|
||||
Adresse ein; sie muss auf den neuen Server zeigen (die Schlüssel kommen mit der Sicherung mit).
|
||||
- Geöffnete Fenster der Firewall-Oberfläche über OPNexus brechen ab und werden neu geöffnet.
|
||||
|
||||
|
||||
|
||||
**Was nicht im Dump ist:**
|
||||
- Die Tabellen der **Log-Erfassung** und der **Systemauslastung** (nur das Schema).
|
||||
@@ -1766,6 +1858,16 @@ Patch-Releases, `1` folgt allen 1.x. Ohne Argument holt `opnexus update` die neu
|
||||
Version der Spur, mit Argument wird die Spur umgestellt. Nach einem Rollback steht
|
||||
die Version fest auf dem alten Stand.
|
||||
|
||||
**Die Compose-Datei kommt mit (ab 1.34).** `opnexus update` übernimmt auch die `docker-compose.yml` des Zielreleases,
|
||||
damit neue Einstellungen der Dienste ankommen. Sie stammt aus dem signierten Updater-Image (nur mit `cosign` und
|
||||
`cosign.pub`, wie die CLI selbst). Die bisherige Datei bleibt als `docker-compose.yml.bak-<version>` liegen, ein
|
||||
Rollback legt sie zurück. Hast du die Datei **selbst angepasst** (sie weicht von der zuletzt übernommenen ab), bleibt
|
||||
sie unverändert; die neue Fassung liegt dann als `docker-compose.yml.new` daneben. Eigene Anpassungen gehören deshalb
|
||||
in eine `docker-compose.override.yml` im selben Verzeichnis: Compose liest sie automatisch mit, und Updates können die
|
||||
Hauptdatei weiter austauschen. Beim ersten Update mit dieser Funktion ist noch keine Prüfsumme bekannt; dann wird
|
||||
getauscht und `diff docker-compose.yml.bak-<version> docker-compose.yml` zeigt, was sich geändert hat.
|
||||
`OPNEXUS_NO_COMPOSE_REFRESH=1` schaltet das Übernehmen ab.
|
||||
|
||||
**Rollback mit Datenbank-Wiederherstellung** bedeutet: alles steht auf dem Stand vor
|
||||
dem Update, was danach geschrieben wurde, ist weg, und Log-Historie und
|
||||
Messwerte sind leer. Vorher legt das Werkzeug eine Sicherheitskopie des jetzigen
|
||||
|
||||
@@ -23,5 +23,9 @@ COPY --from=builder /app/.next/static ./.next/static
|
||||
# next/image nicht, sie wären nur totes Gewicht.
|
||||
RUN if [ "$TARGETARCH" != "$BUILDARCH" ]; then rm -rf node_modules/@img node_modules/sharp; fi
|
||||
|
||||
# Läuft als unprivilegierter Benutzer des Node-Images (uid 1000); die App schreibt
|
||||
# nichts in ihr eigenes Verzeichnis.
|
||||
USER node
|
||||
|
||||
EXPOSE 3000
|
||||
CMD ["node", "server.js"]
|
||||
@@ -182,7 +182,7 @@ export default function DashboardPage() {
|
||||
};
|
||||
}, [errorMessage, router]);
|
||||
|
||||
const healthProblems = (problems ?? []).filter((p) => ["host_offline", "cert_expiry", "rule_drift", "gateway_down", "resource_threshold", "service_down", "api_rights_missing"].includes(p.problem_type));
|
||||
const healthProblems = (problems ?? []).filter((p) => ["host_offline", "cert_expiry", "rule_drift", "gateway_down", "resource_threshold", "service_down", "api_rights_missing", "server_disk"].includes(p.problem_type));
|
||||
const gatewayOrder: Record<string, number> = { offline: 0, degraded: 1, unknown: 2, forced_down: 3, online: 4 };
|
||||
const gatewayRows = (summary?.hosts ?? [])
|
||||
.flatMap((host) => (host.gateways ?? []).map((gateway) => ({ host, gateway })))
|
||||
@@ -354,6 +354,14 @@ export default function DashboardPage() {
|
||||
<Badge tone="warning" className="shrink-0">{t("dashboard.alert.api_rights_missing")}</Badge>
|
||||
</Link>
|
||||
)}
|
||||
{problem.problem_type === "server_disk" && (
|
||||
<Link href="/settings" className="flex items-center justify-between gap-3 hover:underline">
|
||||
<span className="min-w-0 truncate font-medium text-slate-900">
|
||||
{t("dashboard.serverDiskLine", { pct: String(problem.context.belegt_prozent), free: String(problem.context.frei_gb) })}
|
||||
</span>
|
||||
<Badge tone="danger" className="shrink-0">{t("dashboard.alert.server_disk")}</Badge>
|
||||
</Link>
|
||||
)}
|
||||
{problem.problem_type === "cert_expiry" && (
|
||||
<Link href="/certificates" className="flex items-center justify-between gap-3 hover:underline">
|
||||
<span className="min-w-0 truncate font-medium text-slate-900">
|
||||
|
||||
@@ -401,25 +401,27 @@ export default function HostDetailPage() {
|
||||
<tbody>
|
||||
{backups.map((backup, index) => (
|
||||
<tr key={backup.id} className="border-b border-slate-100 last:border-0">
|
||||
<td className="px-5 py-3 text-slate-900">{new Date(backup.created_at).toLocaleString(locale)}</td>
|
||||
<td className="px-5 py-3 text-slate-500">{formatSize(backup.size)}</td>
|
||||
<td className="px-5 py-3 text-right">
|
||||
<td className="px-3 py-3 text-slate-900 sm:px-5">{new Date(backup.created_at).toLocaleString(locale)}</td>
|
||||
<td className="px-3 py-3 text-slate-500 sm:px-5">{formatSize(backup.size)}</td>
|
||||
<td className="px-3 py-3 sm:px-5">
|
||||
<div className="flex justify-end gap-x-1">
|
||||
{index + 1 < backups.length && (
|
||||
<Button variant="ghost" onClick={() => setDiffTarget({ older: backups[index + 1].id, newer: backup.id })}>
|
||||
<Button variant="ghost" className="!px-2 sm:!px-3.5" title={t("hostDetail.changes")} aria-label={t("hostDetail.changes")} onClick={() => setDiffTarget({ older: backups[index + 1].id, newer: backup.id })}>
|
||||
<GitCompare size={14} />
|
||||
{t("hostDetail.changes")}
|
||||
<span className="hidden sm:inline">{t("hostDetail.changes")}</span>
|
||||
</Button>
|
||||
)}
|
||||
{!readOnly && (
|
||||
<Button variant="ghost" onClick={() => handleDownloadBackup(backup.id)}>
|
||||
<Button variant="ghost" className="!px-2 sm:!px-3.5" title={t("hostDetail.download")} aria-label={t("hostDetail.download")} onClick={() => handleDownloadBackup(backup.id)}>
|
||||
<Download size={14} />
|
||||
{t("hostDetail.download")}
|
||||
<span className="hidden sm:inline">{t("hostDetail.download")}</span>
|
||||
</Button>
|
||||
)}
|
||||
<Button variant="ghost" onClick={() => setRestoreTarget(backup.id)}>
|
||||
<Button variant="ghost" className="!px-2 sm:!px-3.5" title={t("hostDetail.restore")} aria-label={t("hostDetail.restore")} onClick={() => setRestoreTarget(backup.id)}>
|
||||
<RotateCcw size={14} />
|
||||
{t("hostDetail.restore")}
|
||||
<span className="hidden sm:inline">{t("hostDetail.restore")}</span>
|
||||
</Button>
|
||||
</div>
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
|
||||
@@ -14,6 +14,7 @@ import { OidcSection } from "@/components/OidcSection";
|
||||
import { CleanupSettings } from "@/components/CleanupSettings";
|
||||
import { BackupSettingsSection } from "@/components/BackupSettingsSection";
|
||||
import { GuiAccessSection } from "@/components/GuiAccessSection";
|
||||
import { SystemSettingsSection } from "@/components/SystemSettingsSection";
|
||||
import { AuditExportSection } from "@/components/AuditExportSection";
|
||||
import { APP_CODENAME, APP_MAINTAINER, APP_VERSION } from "@/lib/version";
|
||||
import { translate } from "@/lib/i18n";
|
||||
@@ -74,6 +75,8 @@ function NotificationsSection() {
|
||||
const [serviceDown, setServiceDown] = useState(false);
|
||||
const [rolloutProblems, setRolloutProblems] = useState(true);
|
||||
const [apiRights, setApiRights] = useState(true);
|
||||
const [serverDisk, setServerDisk] = useState(true);
|
||||
const [serverDiskLimit, setServerDiskLimit] = useState(90);
|
||||
const [gatewayLoss, setGatewayLoss] = useState(20);
|
||||
const [gatewayDelay, setGatewayDelay] = useState(500);
|
||||
const [diskLimit, setDiskLimit] = useState(90);
|
||||
@@ -113,6 +116,8 @@ function NotificationsSection() {
|
||||
setServiceDown(data.service_down_enabled);
|
||||
setRolloutProblems(data.rollout_problems_enabled ?? true);
|
||||
setApiRights(data.api_rights_enabled ?? true);
|
||||
setServerDisk(data.server_disk_enabled ?? true);
|
||||
setServerDiskLimit(data.server_disk_threshold_pct ?? 90);
|
||||
setGatewayLoss(data.gateway_loss_threshold_pct);
|
||||
setGatewayDelay(data.gateway_delay_threshold_ms);
|
||||
setDiskLimit(data.disk_threshold_pct);
|
||||
@@ -170,6 +175,8 @@ function NotificationsSection() {
|
||||
service_down_enabled: serviceDown,
|
||||
rollout_problems_enabled: rolloutProblems,
|
||||
api_rights_enabled: apiRights,
|
||||
server_disk_enabled: serverDisk,
|
||||
server_disk_threshold_pct: serverDiskLimit,
|
||||
gateway_loss_threshold_pct: gatewayLoss,
|
||||
gateway_delay_threshold_ms: gatewayDelay,
|
||||
disk_threshold_pct: diskLimit,
|
||||
@@ -318,6 +325,7 @@ function NotificationsSection() {
|
||||
[t("settings.notify.event.serviceDown"), serviceDown, setServiceDown],
|
||||
[t("settings.notify.event.rolloutProblems"), rolloutProblems, setRolloutProblems],
|
||||
[t("settings.notify.event.apiRights"), apiRights, setApiRights],
|
||||
[t("settings.notify.event.serverDisk"), serverDisk, setServerDisk],
|
||||
].map(([label, value, setter]) => (
|
||||
<label key={label as string} className="flex items-center gap-2 text-sm">
|
||||
<input type="checkbox" checked={value as boolean} onChange={(e) => (setter as (v: boolean) => void)(e.target.checked)} />
|
||||
@@ -343,6 +351,7 @@ function NotificationsSection() {
|
||||
["loadLimit", loadLimit, setLoadLimit, 0, 1000],
|
||||
["confirmSamples", confirmSamples, setConfirmSamples, 1, 60],
|
||||
["resourceSamples", resourceSamples, setResourceSamples, 1, 60],
|
||||
["serverDiskLimit", serverDiskLimit, setServerDiskLimit, 50, 99],
|
||||
] as [string, number, (v: number) => void, number, number][]).map(([key, value, setter, min, max]) => (
|
||||
<label key={key} className="block text-sm">
|
||||
<span className="mb-1 block font-medium text-slate-700">{t(`settings.notify.limit.${key}`)}</span>
|
||||
@@ -482,5 +491,5 @@ export default function SettingsPage(){
|
||||
async function saveTheme(value:Theme){setSaving(true);setError('');setNotice('');const old=theme;setTheme(value);try{setSettings(await api.saveSettings({theme:value,language}));setNotice(t('settings.appearanceSaved'));}catch(e){setTheme(old);setError(errorMessage(e));}finally{setSaving(false);}}
|
||||
async function saveLanguage(value:"de"|"en"){setSaving(true);setError('');setNotice('');const old=language;setLanguage(value);try{setSettings(await api.saveSettings({theme,language:value}));setNotice(translate(value,'settings.language.saved'));}catch(e){setLanguage(old);setError(errorMessage(e));}finally{setSaving(false);}}
|
||||
async function changePassword(e:React.FormEvent){e.preventDefault();setError('');setNotice('');if(password!==repeat){setError(t('settings.passwordMismatch'));return;}setSaving(true);try{await api.changePassword(current,password);setCurrent('');setPassword('');setRepeat('');setNotice(t('settings.passwordChanged'));}catch(e){setError(errorMessage(e,'settings.passwordFailed'));}finally{setSaving(false);}}
|
||||
return <div className="mx-auto max-w-3xl space-y-6"><h1 className="text-2xl font-semibold">{t("settings.title")}</h1>{error && <p role="alert" className="rounded-lg bg-red-50 p-3 text-sm text-red-700">{error}</p>}{notice && <p role="status" className="text-sm text-emerald-700">{notice}</p>}<section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5"><h2 className="flex items-center gap-2 text-lg font-medium">{t('settings.appearance')}<InfoTip text={t('settings.appearanceHint')} /></h2><div role="radiogroup" aria-label={t('settings.colorScheme')} className="flex flex-wrap gap-3">{(['light','dark','system'] as const).map((v)=><label key={v} className={`flex cursor-pointer items-center gap-2 rounded-lg border px-4 py-3 text-sm ${theme===v?'border-sky-500 bg-sky-50 text-sky-700':'border-slate-200'}`}><input type="radio" name="theme" checked={theme===v} disabled={saving || !settings} onChange={()=>saveTheme(v)}/>{t(`settings.theme.${v}`)}</label>)}</div></section><section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5"><h2 className="text-lg font-medium">{t('settings.language.label')}</h2><div role="radiogroup" aria-label={t('settings.language.label')} className="flex flex-wrap gap-3">{(['de','en'] as const).map(value=><label key={value} className={`flex cursor-pointer items-center gap-2 rounded-lg border px-4 py-3 text-sm ${language===value?'border-sky-500 bg-sky-50 text-sky-700':'border-slate-200'}`}><input type="radio" name="language" checked={language===value} disabled={saving || !settings} onChange={()=>saveLanguage(value)}/>{t(`settings.language.${value}`)}</label>)}</div></section><section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5"><h2 className="text-lg font-medium">{t('settings.account')} {settings?.username ? `· ${settings.username}`:''}</h2><form onSubmit={changePassword} className="space-y-4"><label className="block text-sm"><span className="mb-1 block font-medium text-slate-700">{t('settings.currentPassword')}</span><Input aria-label={t('settings.currentPassword')} type="password" autoComplete="current-password" required value={current} onChange={e=>setCurrent(e.target.value)}/></label><label className="block text-sm"><span className="mb-1 block font-medium text-slate-700">{t('settings.newPassword')}</span><Input aria-label={t('settings.newPassword')} type="password" autoComplete="new-password" minLength={10} maxLength={1024} required value={password} onChange={e=>setPassword(e.target.value)}/></label><label className="block text-sm"><span className="mb-1 block font-medium text-slate-700">{t('settings.repeatPassword')}</span><Input aria-label={t('settings.repeatPassword')} type="password" autoComplete="new-password" minLength={10} required value={repeat} onChange={e=>setRepeat(e.target.value)}/></label><Button type="submit" disabled={saving}>{t('settings.changePassword')}</Button></form></section><section className="space-y-3 rounded-xl border border-slate-200 bg-white p-5"><h2 className="flex items-center gap-2 text-lg font-medium">{t('settings.operational')}<InfoTip text={t('settings.serverManaged')} /></h2>{settings?<dl className="grid grid-cols-2 gap-3 text-sm"><dt>{t('settings.deployWindow')}</dt><dd>{t('settings.seconds',{n:settings.deploy_confirm_window_seconds})}</dd><dt>{t('settings.sessionLifetime')}</dt><dd>{t('settings.days',{n:Math.round(settings.session_lifetime_seconds/86400)})}</dd></dl>:<p className="text-sm text-slate-500">{t('common.loading')}</p>}</section><SetupAssistantSection/><TwoFactorSection/><NotificationsSection/><LogIngestSection/><OidcSection/><CleanupSettings/><BackupSettingsSection/><AuditExportSection/><GuiAccessSection/><OpnexusUpdateSection/><section className="space-y-2 rounded-xl border border-slate-200 bg-white p-5"><h2 className="flex items-center gap-2 text-lg font-medium">{t('settings.about')}<InfoTip text={t('settings.changelogNote',{file:'CHANGELOG.md'})} /></h2><dl className="grid grid-cols-2 gap-3 text-sm"><dt>{t('settings.version')}</dt><dd>{APP_VERSION} {language==='en'?'“':'„'}{APP_CODENAME}{language==='en'?'”':'“'}</dd><dt>{t('settings.development')}</dt><dd>{APP_MAINTAINER}</dd></dl></section></div>;
|
||||
return <div className="mx-auto max-w-3xl space-y-6"><h1 className="text-2xl font-semibold">{t("settings.title")}</h1>{error && <p role="alert" className="rounded-lg bg-red-50 p-3 text-sm text-red-700">{error}</p>}{notice && <p role="status" className="text-sm text-emerald-700">{notice}</p>}<section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5"><h2 className="flex items-center gap-2 text-lg font-medium">{t('settings.appearance')}<InfoTip text={t('settings.appearanceHint')} /></h2><div role="radiogroup" aria-label={t('settings.colorScheme')} className="flex flex-wrap gap-3">{(['light','dark','system'] as const).map((v)=><label key={v} className={`flex cursor-pointer items-center gap-2 rounded-lg border px-4 py-3 text-sm ${theme===v?'border-sky-500 bg-sky-50 text-sky-700':'border-slate-200'}`}><input type="radio" name="theme" checked={theme===v} disabled={saving || !settings} onChange={()=>saveTheme(v)}/>{t(`settings.theme.${v}`)}</label>)}</div></section><section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5"><h2 className="text-lg font-medium">{t('settings.language.label')}</h2><div role="radiogroup" aria-label={t('settings.language.label')} className="flex flex-wrap gap-3">{(['de','en'] as const).map(value=><label key={value} className={`flex cursor-pointer items-center gap-2 rounded-lg border px-4 py-3 text-sm ${language===value?'border-sky-500 bg-sky-50 text-sky-700':'border-slate-200'}`}><input type="radio" name="language" checked={language===value} disabled={saving || !settings} onChange={()=>saveLanguage(value)}/>{t(`settings.language.${value}`)}</label>)}</div></section><section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5"><h2 className="text-lg font-medium">{t('settings.account')} {settings?.username ? `· ${settings.username}`:''}</h2><form onSubmit={changePassword} className="space-y-4"><label className="block text-sm"><span className="mb-1 block font-medium text-slate-700">{t('settings.currentPassword')}</span><Input aria-label={t('settings.currentPassword')} type="password" autoComplete="current-password" required value={current} onChange={e=>setCurrent(e.target.value)}/></label><label className="block text-sm"><span className="mb-1 block font-medium text-slate-700">{t('settings.newPassword')}</span><Input aria-label={t('settings.newPassword')} type="password" autoComplete="new-password" minLength={10} maxLength={1024} required value={password} onChange={e=>setPassword(e.target.value)}/></label><label className="block text-sm"><span className="mb-1 block font-medium text-slate-700">{t('settings.repeatPassword')}</span><Input aria-label={t('settings.repeatPassword')} type="password" autoComplete="new-password" minLength={10} required value={repeat} onChange={e=>setRepeat(e.target.value)}/></label><Button type="submit" disabled={saving}>{t('settings.changePassword')}</Button></form></section><SystemSettingsSection/><SetupAssistantSection/><TwoFactorSection/><NotificationsSection/><LogIngestSection/><OidcSection/><CleanupSettings/><BackupSettingsSection/><AuditExportSection/><GuiAccessSection/><OpnexusUpdateSection/><section className="space-y-2 rounded-xl border border-slate-200 bg-white p-5"><h2 className="flex items-center gap-2 text-lg font-medium">{t('settings.about')}<InfoTip text={t('settings.changelogNote',{file:'CHANGELOG.md'})} /></h2><dl className="grid grid-cols-2 gap-3 text-sm"><dt>{t('settings.version')}</dt><dd>{APP_VERSION} {language==='en'?'“':'„'}{APP_CODENAME}{language==='en'?'”':'“'}</dd><dt>{t('settings.development')}</dt><dd>{APP_MAINTAINER}</dd></dl></section></div>;
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
"use client";
|
||||
|
||||
import { Badge } from "@/components/ui/Badge";
|
||||
import { Button } from "@/components/ui/Button";
|
||||
import { Input } from "@/components/ui/Field";
|
||||
import { useToast } from "@/components/ui/Toast";
|
||||
import { InfoTip } from "@/components/ui/Tooltip";
|
||||
import { useLanguage } from "@/components/LanguageProvider";
|
||||
import { api, SystemSetting } from "@/lib/api";
|
||||
import { useNotAdmin } from "@/lib/useReadOnly";
|
||||
import { RotateCcw } from "lucide-react";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
|
||||
const GROUPS = ["security", "operation", "monitoring", "integrations"] as const;
|
||||
|
||||
type Draft = Record<string, string>;
|
||||
|
||||
function draftOf(items: SystemSetting[]): Draft {
|
||||
const draft: Draft = {};
|
||||
for (const item of items) {
|
||||
draft[item.key] = item.kind === "secret_url" ? "" : item.value === null ? "" : String(item.value);
|
||||
}
|
||||
return draft;
|
||||
}
|
||||
|
||||
/** Settings › Betrieb: everything that used to need the server's .env and is safe to change while OPNexus runs. */
|
||||
export function SystemSettingsSection() {
|
||||
const { t, errorMessage } = useLanguage();
|
||||
const toast = useToast();
|
||||
const readOnly = useNotAdmin();
|
||||
const [items, setItems] = useState<SystemSetting[] | null>(null);
|
||||
const [draft, setDraft] = useState<Draft>({});
|
||||
const [saving, setSaving] = useState<string | null>(null);
|
||||
const [error, setError] = useState("");
|
||||
|
||||
function apply(next: SystemSetting[]) {
|
||||
setItems(next);
|
||||
setDraft(draftOf(next));
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
api.systemSettings().then((body) => apply(body.settings)).catch((e) => setError(errorMessage(e)));
|
||||
}, [errorMessage]);
|
||||
|
||||
const byGroup = useMemo(() => {
|
||||
const out: Record<string, SystemSetting[]> = {};
|
||||
for (const item of items ?? []) (out[item.group] ??= []).push(item);
|
||||
return out;
|
||||
}, [items]);
|
||||
|
||||
function changed(item: SystemSetting): boolean {
|
||||
if (item.source === "server") return false;
|
||||
const value = draft[item.key] ?? "";
|
||||
if (item.kind === "secret_url") return value.trim() !== "";
|
||||
return value !== (item.value === null ? "" : String(item.value));
|
||||
}
|
||||
|
||||
async function save(values: Record<string, number | string | null>, label: string) {
|
||||
setSaving(label);
|
||||
setError("");
|
||||
try {
|
||||
apply((await api.saveSystemSettings(values)).settings);
|
||||
toast.show(t("settings.system.saved"), "success");
|
||||
} catch (e) {
|
||||
setError(errorMessage(e, "common.saveFailed"));
|
||||
} finally {
|
||||
setSaving(null);
|
||||
}
|
||||
}
|
||||
|
||||
function saveGroup(group: string) {
|
||||
const values: Record<string, number | string | null> = {};
|
||||
for (const item of byGroup[group] ?? []) {
|
||||
if (!changed(item)) continue;
|
||||
const raw = (draft[item.key] ?? "").trim();
|
||||
values[item.key] = item.kind === "int" || item.kind === "float" ? (raw === "" ? null : Number(raw)) : raw;
|
||||
}
|
||||
if (Object.keys(values).length) void save(values, group);
|
||||
}
|
||||
|
||||
function sourceBadge(item: SystemSetting) {
|
||||
if (item.source === "server") {
|
||||
return <Badge tone="warning">{t("settings.system.fromServer", { env: item.env ?? "" })}</Badge>;
|
||||
}
|
||||
if (item.source === "ui") return <Badge tone="info">{t("settings.system.changed")}</Badge>;
|
||||
return null;
|
||||
}
|
||||
|
||||
function field(item: SystemSetting) {
|
||||
const label = t(`settings.system.field.${item.key}.label`);
|
||||
const hint = t(`settings.system.field.${item.key}.hint`);
|
||||
const locked = readOnly || item.source === "server";
|
||||
const value = draft[item.key] ?? "";
|
||||
const set = (v: string) => setDraft((d) => ({ ...d, [item.key]: v }));
|
||||
const reset = item.source === "ui" && !readOnly && (
|
||||
<Button
|
||||
variant="ghost"
|
||||
type="button"
|
||||
title={t("settings.system.reset")}
|
||||
aria-label={`${t("settings.system.reset")}: ${label}`}
|
||||
disabled={saving !== null}
|
||||
onClick={() => void save({ [item.key]: null }, item.key)}
|
||||
>
|
||||
<RotateCcw size={14} />
|
||||
</Button>
|
||||
);
|
||||
const head = (
|
||||
<span className="mb-1 flex flex-wrap items-center gap-1.5 font-medium text-slate-700">
|
||||
{label}
|
||||
<InfoTip text={hint} size={13} />
|
||||
{sourceBadge(item)}
|
||||
</span>
|
||||
);
|
||||
|
||||
if (item.kind === "pem") {
|
||||
return (
|
||||
<div key={item.key} className="block text-sm sm:col-span-2">
|
||||
{head}
|
||||
{item.source === "server" ? (
|
||||
<p className="text-slate-600">{t("settings.system.pemFromServer")}</p>
|
||||
) : (
|
||||
<textarea
|
||||
aria-label={label}
|
||||
className="h-28 w-full rounded-lg border border-slate-300 bg-white p-2 font-mono text-xs"
|
||||
placeholder="-----BEGIN CERTIFICATE-----"
|
||||
spellCheck={false}
|
||||
disabled={locked}
|
||||
value={value}
|
||||
onChange={(e) => set(e.target.value)}
|
||||
/>
|
||||
)}
|
||||
{reset && <div className="mt-1 flex justify-end">{reset}</div>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (item.kind === "secret_url") {
|
||||
return (
|
||||
<div key={item.key} className="block text-sm sm:col-span-2">
|
||||
{head}
|
||||
<div className="flex gap-2">
|
||||
<Input
|
||||
aria-label={label}
|
||||
type="url"
|
||||
inputMode="url"
|
||||
spellCheck={false}
|
||||
autoComplete="off"
|
||||
disabled={locked}
|
||||
placeholder={item.is_set ? t("settings.system.secretSet") : "https://"}
|
||||
value={value}
|
||||
onChange={(e) => set(e.target.value)}
|
||||
/>
|
||||
{item.is_set && item.source === "ui" && !readOnly && (
|
||||
<Button variant="secondary" type="button" disabled={saving !== null} onClick={() => void save({ [item.key]: null }, item.key)}>
|
||||
{t("settings.system.remove")}
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
const bounds = item.min !== null && item.max !== null ? t("settings.system.bounds", { min: item.min, max: item.max }) : "";
|
||||
return (
|
||||
<label key={item.key} className="block text-sm">
|
||||
{head}
|
||||
<div className="flex items-center gap-2">
|
||||
<Input
|
||||
type="number"
|
||||
min={item.min ?? undefined}
|
||||
max={item.max ?? undefined}
|
||||
step={item.kind === "float" ? 0.5 : 1}
|
||||
disabled={locked}
|
||||
value={value}
|
||||
onChange={(e) => set(e.target.value)}
|
||||
/>
|
||||
<span className="shrink-0 text-xs text-slate-500">{item.unit ? t(`settings.system.unit.${item.unit}`) : ""}</span>
|
||||
{reset}
|
||||
</div>
|
||||
<span className="mt-1 block text-xs text-slate-500">
|
||||
{item.default !== null && t("settings.system.defaultIs", { value: item.default })}
|
||||
{bounds && ` · ${bounds}`}
|
||||
</span>
|
||||
</label>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<section className="space-y-4 rounded-xl border border-slate-200 bg-white p-5">
|
||||
<h2 className="flex items-center gap-2 text-lg font-medium">
|
||||
{t("settings.operational")}
|
||||
<InfoTip text={t("settings.system.intro")} />
|
||||
</h2>
|
||||
{!items && !error && <p className="text-sm text-slate-500">{t("common.loading")}</p>}
|
||||
{items &&
|
||||
GROUPS.filter((g) => byGroup[g]?.length).map((group) => {
|
||||
const dirty = (byGroup[group] ?? []).some(changed);
|
||||
return (
|
||||
<form
|
||||
key={group}
|
||||
className="space-y-3 rounded-lg border border-slate-100 p-4"
|
||||
onSubmit={(e) => {
|
||||
e.preventDefault();
|
||||
saveGroup(group);
|
||||
}}
|
||||
>
|
||||
<p className="flex items-center gap-2 text-sm font-medium">
|
||||
{t(`settings.system.group.${group}`)}
|
||||
<InfoTip text={t(`settings.system.groupHint.${group}`)} />
|
||||
</p>
|
||||
<div className="grid gap-4 sm:grid-cols-2">{(byGroup[group] ?? []).map(field)}</div>
|
||||
{!readOnly && (
|
||||
<div className="flex justify-end border-t border-slate-100 pt-3">
|
||||
<Button type="submit" disabled={!dirty || saving !== null}>
|
||||
{t("common.save")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</form>
|
||||
);
|
||||
})}
|
||||
{error && (
|
||||
<p role="alert" className="rounded-lg bg-red-50 p-3 text-sm text-red-700">
|
||||
{error}
|
||||
</p>
|
||||
)}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
+21
-1
@@ -1298,6 +1298,20 @@ export interface UpdaterJob {
|
||||
from_version: string | null;
|
||||
}
|
||||
|
||||
export type SystemSetting = {
|
||||
key: string;
|
||||
group: "security" | "operation" | "monitoring" | "integrations";
|
||||
kind: "int" | "float" | "secret_url" | "pem";
|
||||
unit: string;
|
||||
min: number | null;
|
||||
max: number | null;
|
||||
value: number | string | null;
|
||||
default: number | null;
|
||||
/** default: built-in value; ui: changed here; server: set in the server's .env (read-only here) */
|
||||
source: "default" | "ui" | "server";
|
||||
env: string | null;
|
||||
is_set?: boolean;
|
||||
};
|
||||
export type UpdateCheckSettings = { enabled: boolean; feed_url: string; managed_by_server: boolean };
|
||||
export interface UpdaterStatus {
|
||||
/** false = Updater nicht aktiviert/erreichbar (`reason` sagt warum). */
|
||||
@@ -1394,6 +1408,8 @@ export const api = {
|
||||
audit: (filters: Record<string,string | number> = {}) => request<AuditPage>(`/audit?${new URLSearchParams(Object.entries(filters).filter(([,v])=>v!=="").map(([k,v])=>[k,String(v)]))}`),
|
||||
dashboard: () => request<DashboardSummary>("/dashboard"),
|
||||
versionCheck: () => request<VersionCheck>("/system/version-check"),
|
||||
systemSettings: () => request<{ settings: SystemSetting[] }>("/settings/system"),
|
||||
saveSystemSettings: (values: Record<string, number | string | null>) => request<{ settings: SystemSetting[] }>("/settings/system", { method: "PUT", body: JSON.stringify({ values }) }),
|
||||
updateCheck: () => request<UpdateCheckSettings>("/system/update-check"),
|
||||
updateUpdateCheck: (body: { enabled?: boolean; feed_url?: string }) => request<UpdateCheckSettings>("/system/update-check", { method: "PUT", body: JSON.stringify(body) }),
|
||||
updaterStatus: () => request<UpdaterStatus>("/system/updater"),
|
||||
@@ -1969,6 +1985,8 @@ export interface NotificationSettings {
|
||||
service_down_enabled: boolean;
|
||||
rollout_problems_enabled: boolean;
|
||||
api_rights_enabled: boolean;
|
||||
server_disk_enabled: boolean;
|
||||
server_disk_threshold_pct: number;
|
||||
gateway_loss_threshold_pct: number;
|
||||
gateway_delay_threshold_ms: number;
|
||||
disk_threshold_pct: number;
|
||||
@@ -2027,6 +2045,8 @@ export interface NotificationSettingsUpdate {
|
||||
service_down_enabled?: boolean;
|
||||
rollout_problems_enabled?: boolean;
|
||||
api_rights_enabled?: boolean;
|
||||
server_disk_enabled?: boolean;
|
||||
server_disk_threshold_pct?: number;
|
||||
gateway_loss_threshold_pct?: number;
|
||||
gateway_delay_threshold_ms?: number;
|
||||
disk_threshold_pct?: number;
|
||||
@@ -2064,7 +2084,7 @@ export type ProblemType =
|
||||
| "backup_offsite_failed"
|
||||
| "compliance_finding"
|
||||
| "gateway_down" | "resource_threshold" | "service_down" | "api_rights_missing"
|
||||
| "rollout_paused" | "rollout_start_failed";
|
||||
| "rollout_paused" | "rollout_start_failed" | "server_disk";
|
||||
|
||||
export interface CarpEvent {
|
||||
id: string;
|
||||
|
||||
+105
-5
@@ -72,7 +72,8 @@
|
||||
"resourceThreshold": "Ressourcen-Schwelle überschritten (Festplatte, Speicher, Swap, CPU-Last; standardmäßig aus)",
|
||||
"serviceDown": "Überwachter Dienst ausgefallen (standardmäßig aus)",
|
||||
"rolloutProblems": "Rollout angehalten oder nicht automatisch gestartet",
|
||||
"apiRights": "API-Rechte fehlen (Gateway- oder Dienststatus nicht lesbar)"
|
||||
"apiRights": "API-Rechte fehlen (Gateway- oder Dienststatus nicht lesbar)",
|
||||
"serverDisk": "Platte des OPNexus-Servers fast voll"
|
||||
},
|
||||
"pollInterval": "Prüfintervall (Sekunden)",
|
||||
"pollHint": "Mindestens 60 Sekunden, um die Firewalls nicht unnötig zu belasten.",
|
||||
@@ -88,7 +89,8 @@
|
||||
"confirmSamples": "Gateways und Dienste: Messungen in Folge",
|
||||
"resourceSamples": "Ressourcen: Messungen in Folge",
|
||||
"watchlist": "Überwachte Dienste",
|
||||
"watchlistHint": "OPNsense-Dienstnamen, kommagetrennt. OPNexus meldet einen Dienst nur, wenn er zuvor lief und jetzt nicht mehr. Absichtlich gestoppte Dienste hier entfernen."
|
||||
"watchlistHint": "OPNsense-Dienstnamen, kommagetrennt. OPNexus meldet einen Dienst nur, wenn er zuvor lief und jetzt nicht mehr. Absichtlich gestoppte Dienste hier entfernen.",
|
||||
"serverDiskLimit": "OPNexus-Server: Platte voll ab (%)"
|
||||
}
|
||||
},
|
||||
"logs": {
|
||||
@@ -245,6 +247,99 @@
|
||||
"teams": "Webhook-Adresse des Workflows",
|
||||
"matrix": "Zugriffstoken"
|
||||
}
|
||||
},
|
||||
"system": {
|
||||
"intro": "Diese Werte standen früher nur in der .env des Servers. Steht ein Wert dort, gilt er weiter und ist hier gesperrt. Adressen, TLS und Datenbank-Zugang bleiben bewusst in der .env: ein falscher Wert würde dich aus dieser Oberfläche aussperren.",
|
||||
"saved": "Einstellungen gespeichert",
|
||||
"changed": "geändert",
|
||||
"fromServer": "vom Server ({env})",
|
||||
"reset": "Auf Standard zurücksetzen",
|
||||
"remove": "Entfernen",
|
||||
"secretSet": "gesetzt (zum Ändern neue Adresse eingeben)",
|
||||
"pemFromServer": "Wird vom Server über SMTP_CA_FILE vorgegeben.",
|
||||
"defaultIs": "Standard: {value}",
|
||||
"bounds": "erlaubt {min}–{max}",
|
||||
"group": {
|
||||
"security": "Anmeldung und Sitzungen",
|
||||
"operation": "Ausrollen und Firewall-Zugriff",
|
||||
"monitoring": "Messungen und Aufbewahrung",
|
||||
"integrations": "Überwachung von außen und Mail"
|
||||
},
|
||||
"groupHint": {
|
||||
"security": "Schutz vor erratenen Passwörtern und wie lange Anmeldungen gelten.",
|
||||
"operation": "Sicherheitsnetz beim Ausrollen und Geduld mit langsamen Firewalls.",
|
||||
"monitoring": "Wie oft OPNexus misst und wie lange es die Daten behält. Kürzere Intervalle bedeuten mehr Anfragen an die Firewalls.",
|
||||
"integrations": "Heartbeat für einen externen Überwacher und Vertrauen in einen Mailserver mit eigener CA."
|
||||
},
|
||||
"unit": {
|
||||
"seconds": "Sekunden",
|
||||
"minutes": "Minuten",
|
||||
"hours": "Stunden",
|
||||
"days": "Tage",
|
||||
"count": "Anzahl"
|
||||
},
|
||||
"field": {
|
||||
"session_lifetime_days": {
|
||||
"label": "Sitzungsdauer",
|
||||
"hint": "Wie lange eine Anmeldung gültig bleibt. Gilt für neue Anmeldungen."
|
||||
},
|
||||
"login_max_attempts": {
|
||||
"label": "Fehlversuche je Konto",
|
||||
"hint": "So viele falsche Passwörter je Konto und Absender, dann wird gesperrt."
|
||||
},
|
||||
"login_ip_max_attempts": {
|
||||
"label": "Fehlversuche je Absender",
|
||||
"hint": "So viele falsche Anmeldungen von einer Adresse (über alle Konten), dann wird gesperrt."
|
||||
},
|
||||
"login_window_seconds": {
|
||||
"label": "Zeitfenster für Fehlversuche",
|
||||
"hint": "In diesem Zeitraum werden Fehlversuche gezählt."
|
||||
},
|
||||
"login_lockout_seconds": {
|
||||
"label": "Sperrdauer",
|
||||
"hint": "So lange bleibt eine Sperre nach zu vielen Fehlversuchen bestehen."
|
||||
},
|
||||
"deploy_confirm_window_seconds": {
|
||||
"label": "Bestätigungsfrist beim Ausrollen",
|
||||
"hint": "So lange hast du nach dem Ausrollen Zeit, eine Änderung zu bestätigen. Ohne Bestätigung nimmt OPNexus sie automatisch zurück (Schutz vor Aussperren). Gilt für neue Ausrollvorgänge."
|
||||
},
|
||||
"opnsense_timeout": {
|
||||
"label": "Zeitlimit für Firewall-Anfragen",
|
||||
"hint": "Wie lange OPNexus auf eine Antwort der Firewall-API wartet. Für langsame Verbindungen (Tunnel, Mobilfunk) erhöhen."
|
||||
},
|
||||
"migration_backup_keep": {
|
||||
"label": "Sicherungen vor Updates behalten",
|
||||
"hint": "Wie viele automatische Datenbank-Sicherungen vor Schema-Änderungen aufbewahrt werden."
|
||||
},
|
||||
"resource_poll_interval_seconds": {
|
||||
"label": "Messintervall Systemauslastung",
|
||||
"hint": "Wie oft Speicher, Last, Platte, Gateways und Dienste der Firewalls gemessen werden."
|
||||
},
|
||||
"resource_sample_retention_days": {
|
||||
"label": "Messwerte aufbewahren",
|
||||
"hint": "Wie lange die Messwerte der Systemauslastung gespeichert bleiben."
|
||||
},
|
||||
"rule_observe_interval_seconds": {
|
||||
"label": "Intervall Regelverlauf",
|
||||
"hint": "Wie oft die Regeln und Zähler der Firewalls für Verlauf und Regel-Analyse gelesen werden."
|
||||
},
|
||||
"update_check_interval_hours": {
|
||||
"label": "Intervall der Versionsprüfung",
|
||||
"hint": "Wie oft nach einer neuen OPNexus-Version gefragt wird (nur wenn die Versionsprüfung an ist)."
|
||||
},
|
||||
"heartbeat_url": {
|
||||
"label": "Heartbeat-Adresse",
|
||||
"hint": "Ping-Adresse eines externen Überwachers (healthchecks.io, Uptime Kuma, ntfy). OPNexus ruft sie nach jedem erfolgreichen Prüfzyklus auf; bleibt das aus, schlägt der Überwacher Alarm. Nur https://. Die Adresse wird nicht angezeigt und nie protokolliert."
|
||||
},
|
||||
"smtp_ca_pem": {
|
||||
"label": "CA des Mailservers",
|
||||
"hint": "Nur nötig, wenn der Mailserver (oder das TLS-Syslog des Audit-Exports) ein Zertifikat einer eigenen CA nutzt. Das CA-Zertifikat im PEM-Format einfügen."
|
||||
},
|
||||
"notification_smtp_timeout_seconds": {
|
||||
"label": "Zeitlimit Mailversand",
|
||||
"hint": "Wie lange OPNexus auf den Mailserver wartet."
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"nav": {
|
||||
@@ -373,10 +468,12 @@
|
||||
"gateway_down": "Gateway",
|
||||
"resource_threshold": "Ressource",
|
||||
"service_down": "Dienst",
|
||||
"api_rights_missing": "API-Rechte"
|
||||
"api_rights_missing": "API-Rechte",
|
||||
"server_disk": "OPNexus-Platte"
|
||||
},
|
||||
"moreGateways": "+ {count} weitere Gateways (auf der Seite der Firewall)",
|
||||
"apiRightsLine": "{name}: API-Recht fehlt ({read})"
|
||||
"apiRightsLine": "{name}: API-Recht fehlt ({read})",
|
||||
"serverDiskLine": "Platte des OPNexus-Servers zu {pct} % belegt ({free} GB frei)"
|
||||
},
|
||||
"hosts": {
|
||||
"add": "Host hinzufügen",
|
||||
@@ -917,7 +1014,8 @@
|
||||
"HOST_CONNECTION_FAILED": "Die Firewall konnte mit diesen Angaben nicht erreicht werden: {reason}",
|
||||
"ZIEL_HOST_NACH_AUSROLLEN_FEST": "Der Ziel-Host lässt sich nach dem Ausrollen nicht mehr ändern. Lösche den Eintrag (das entfernt ihn auch von der Firewall) und lege ihn für den neuen Host neu an.",
|
||||
"ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET": "Dieses ACME-Konto wird von Zertifikatsanfragen verwendet; den Host kannst du erst ändern, wenn keine Anfrage es mehr nutzt.",
|
||||
"VPN_DEPLOY_LAEUFT_BEREITS": "Für dieses VPN läuft gerade ein Ausrollen oder Entfernen. Warte einen Moment und versuche es erneut."
|
||||
"VPN_DEPLOY_LAEUFT_BEREITS": "Für dieses VPN läuft gerade ein Ausrollen oder Entfernen. Warte einen Moment und versuche es erneut.",
|
||||
"SETTING_INVALID": "Einstellung „{setting}“: {reason}"
|
||||
},
|
||||
"workflow": {
|
||||
"attention": "Fehler prüfen",
|
||||
@@ -3041,12 +3139,14 @@
|
||||
"badge": "API-Rechte fehlen: {reads}",
|
||||
"hint": "Der API-Benutzer von OPNexus darf diese Messwerte nicht lesen, die Alarme dazu können nicht auslösen. In OPNsense beim Benutzer oder der Gruppe unter System ▸ Zugang ergänzen:",
|
||||
"read": {
|
||||
"base": "Grundwerte (Speicher, Last, Platte)",
|
||||
"gateways": "Gateways",
|
||||
"services": "Dienste",
|
||||
"swap": "Swap",
|
||||
"cpu": "CPU"
|
||||
},
|
||||
"acl": {
|
||||
"base": "Lobby: Dashboard",
|
||||
"gateways": "System: Gateways",
|
||||
"services": "Status: Services",
|
||||
"swap": "Lobby: Dashboard",
|
||||
|
||||
+105
-5
@@ -72,7 +72,8 @@
|
||||
"resourceThreshold": "Resource threshold exceeded (disk, memory, swap, CPU load; off by default)",
|
||||
"serviceDown": "Watched service stopped (off by default)",
|
||||
"rolloutProblems": "Rollout paused or not started automatically",
|
||||
"apiRights": "API rights missing (gateway or service status not readable)"
|
||||
"apiRights": "API rights missing (gateway or service status not readable)",
|
||||
"serverDisk": "Disk of the OPNexus server almost full"
|
||||
},
|
||||
"pollInterval": "Check interval (seconds)",
|
||||
"pollHint": "At least 60 seconds, so the firewalls are not queried needlessly.",
|
||||
@@ -88,7 +89,8 @@
|
||||
"confirmSamples": "Gateways and services: consecutive measurements",
|
||||
"resourceSamples": "Resources: consecutive measurements",
|
||||
"watchlist": "Watched services",
|
||||
"watchlistHint": "OPNsense service names, comma-separated. OPNexus reports a service only if it ran before and has stopped now. Remove services you stopped on purpose."
|
||||
"watchlistHint": "OPNsense service names, comma-separated. OPNexus reports a service only if it ran before and has stopped now. Remove services you stopped on purpose.",
|
||||
"serverDiskLimit": "OPNexus server: disk full from (%)"
|
||||
}
|
||||
},
|
||||
"logs": {
|
||||
@@ -245,6 +247,99 @@
|
||||
"teams": "Workflow webhook address",
|
||||
"matrix": "Access token"
|
||||
}
|
||||
},
|
||||
"system": {
|
||||
"intro": "These values used to live only in the server's .env. A value set there still applies and is locked here. Addresses, TLS and database access deliberately stay in the .env: a wrong value would lock you out of this very interface.",
|
||||
"saved": "Settings saved",
|
||||
"changed": "changed",
|
||||
"fromServer": "from server ({env})",
|
||||
"reset": "Reset to default",
|
||||
"remove": "Remove",
|
||||
"secretSet": "set (enter a new URL to change it)",
|
||||
"pemFromServer": "Set by the server through SMTP_CA_FILE.",
|
||||
"defaultIs": "Default: {value}",
|
||||
"bounds": "allowed {min}–{max}",
|
||||
"group": {
|
||||
"security": "Sign-in and sessions",
|
||||
"operation": "Rollouts and firewall access",
|
||||
"monitoring": "Sampling and retention",
|
||||
"integrations": "External monitoring and mail"
|
||||
},
|
||||
"groupHint": {
|
||||
"security": "Protection against guessed passwords and how long sign-ins last.",
|
||||
"operation": "The safety net when rolling out and patience with slow firewalls.",
|
||||
"monitoring": "How often OPNexus measures and how long it keeps the data. Shorter intervals mean more requests to the firewalls.",
|
||||
"integrations": "Heartbeat for an external watcher and trust for a mail server with its own CA."
|
||||
},
|
||||
"unit": {
|
||||
"seconds": "seconds",
|
||||
"minutes": "minutes",
|
||||
"hours": "hours",
|
||||
"days": "days",
|
||||
"count": "count"
|
||||
},
|
||||
"field": {
|
||||
"session_lifetime_days": {
|
||||
"label": "Session lifetime",
|
||||
"hint": "How long a sign-in stays valid. Applies to new sign-ins."
|
||||
},
|
||||
"login_max_attempts": {
|
||||
"label": "Failed attempts per account",
|
||||
"hint": "This many wrong passwords per account and sender, then it is locked."
|
||||
},
|
||||
"login_ip_max_attempts": {
|
||||
"label": "Failed attempts per sender",
|
||||
"hint": "This many failed sign-ins from one address (across all accounts), then it is locked."
|
||||
},
|
||||
"login_window_seconds": {
|
||||
"label": "Window for failed attempts",
|
||||
"hint": "Failed attempts are counted within this period."
|
||||
},
|
||||
"login_lockout_seconds": {
|
||||
"label": "Lockout duration",
|
||||
"hint": "How long a lock lasts after too many failed attempts."
|
||||
},
|
||||
"deploy_confirm_window_seconds": {
|
||||
"label": "Confirmation window when rolling out",
|
||||
"hint": "How long you have after a rollout to confirm a change. Without confirmation OPNexus reverts it automatically (lockout protection). Applies to new rollouts."
|
||||
},
|
||||
"opnsense_timeout": {
|
||||
"label": "Timeout for firewall requests",
|
||||
"hint": "How long OPNexus waits for the firewall API to answer. Raise it for slow links (tunnels, mobile)."
|
||||
},
|
||||
"migration_backup_keep": {
|
||||
"label": "Backups kept before updates",
|
||||
"hint": "How many automatic database backups taken before schema changes are kept."
|
||||
},
|
||||
"resource_poll_interval_seconds": {
|
||||
"label": "Resource sampling interval",
|
||||
"hint": "How often memory, load, disk, gateways and services of the firewalls are measured."
|
||||
},
|
||||
"resource_sample_retention_days": {
|
||||
"label": "Keep resource samples",
|
||||
"hint": "How long resource samples are stored."
|
||||
},
|
||||
"rule_observe_interval_seconds": {
|
||||
"label": "Rule history interval",
|
||||
"hint": "How often the firewalls' rules and counters are read for the history and the rule analysis."
|
||||
},
|
||||
"update_check_interval_hours": {
|
||||
"label": "Version check interval",
|
||||
"hint": "How often OPNexus asks for a new version (only while the version check is on)."
|
||||
},
|
||||
"heartbeat_url": {
|
||||
"label": "Heartbeat URL",
|
||||
"hint": "Ping URL of an external watcher (healthchecks.io, Uptime Kuma, ntfy). OPNexus calls it after every successful check cycle; when the calls stop, the watcher raises the alarm. https:// only. The URL is not shown and never logged."
|
||||
},
|
||||
"smtp_ca_pem": {
|
||||
"label": "Mail server CA",
|
||||
"hint": "Only needed when the mail server (or the TLS syslog of the audit export) uses a certificate from a private CA. Paste the CA certificate in PEM format."
|
||||
},
|
||||
"notification_smtp_timeout_seconds": {
|
||||
"label": "Mail timeout",
|
||||
"hint": "How long OPNexus waits for the mail server."
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"nav": {
|
||||
@@ -373,10 +468,12 @@
|
||||
"gateway_down": "Gateway",
|
||||
"resource_threshold": "Resource",
|
||||
"service_down": "Service",
|
||||
"api_rights_missing": "API rights"
|
||||
"api_rights_missing": "API rights",
|
||||
"server_disk": "OPNexus disk"
|
||||
},
|
||||
"moreGateways": "+ {count} more gateways (on the firewall page)",
|
||||
"apiRightsLine": "{name}: API right missing ({read})"
|
||||
"apiRightsLine": "{name}: API right missing ({read})",
|
||||
"serverDiskLine": "Disk of the OPNexus server {pct} % full ({free} GB free)"
|
||||
},
|
||||
"hosts": {
|
||||
"add": "Add host",
|
||||
@@ -917,7 +1014,8 @@
|
||||
"HOST_CONNECTION_FAILED": "The firewall could not be reached with these details: {reason}",
|
||||
"ZIEL_HOST_NACH_AUSROLLEN_FEST": "The target host can no longer be changed once the entry has been deployed. Delete the entry (this also removes it from the firewall) and create it again for the new host.",
|
||||
"ACME_KONTO_WIRD_VON_ANFRAGEN_VERWENDET": "This ACME account is used by certificate requests; you can only change its host once no request uses it any more.",
|
||||
"VPN_DEPLOY_LAEUFT_BEREITS": "A deployment or removal is already running for this VPN. Wait a moment and try again."
|
||||
"VPN_DEPLOY_LAEUFT_BEREITS": "A deployment or removal is already running for this VPN. Wait a moment and try again.",
|
||||
"SETTING_INVALID": "Setting “{setting}”: {reason}"
|
||||
},
|
||||
"workflow": {
|
||||
"attention": "Check errors",
|
||||
@@ -3041,12 +3139,14 @@
|
||||
"badge": "API rights missing: {reads}",
|
||||
"hint": "OPNexus's API user may not read these values, so the alerts based on them cannot fire. In OPNsense, add to the user or group under System > Access:",
|
||||
"read": {
|
||||
"base": "Basic readings (memory, load, disk)",
|
||||
"gateways": "Gateways",
|
||||
"services": "Services",
|
||||
"swap": "Swap",
|
||||
"cpu": "CPU"
|
||||
},
|
||||
"acl": {
|
||||
"base": "Lobby: Dashboard",
|
||||
"gateways": "System: Gateways",
|
||||
"services": "Status: Services",
|
||||
"swap": "Lobby: Dashboard",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Manually bumped whenever a notable change ships — mirrors the dated
|
||||
// entries already kept in the project README/CHANGELOG.md, no build
|
||||
// tooling involved. Versioning scheme: see CHANGELOG.md ("Versionierung").
|
||||
export const APP_VERSION = "1.33.0";
|
||||
export const APP_CODENAME = "Saker";
|
||||
export const APP_VERSION = "1.34.0";
|
||||
export const APP_CODENAME = "Griffon";
|
||||
export const APP_MAINTAINER = "Julian Boldt";
|
||||
@@ -47,6 +47,8 @@ version="${tag#v}"
|
||||
src="$(mktemp -d)"; trap 'rm -rf "$src"' EXIT
|
||||
git archive "$tag" | tar -x -C "$src"
|
||||
(cd "$src" && scripts/check_version.sh "${tag%%-*}")
|
||||
# Die Release-Fassung der Compose-Datei ins Updater-Image: `opnexus update` übernimmt sie von dort (B18).
|
||||
(cd "$src" && scripts/release_compose.sh "$version" "$repo" > updater/docker-compose.release.yml)
|
||||
|
||||
tags=("$version")
|
||||
if [[ "$version" != *-* ]]; then # nur stabile Versionen bekommen Kurz-Tags und latest
|
||||
|
||||
@@ -19,12 +19,8 @@ out="dist/$name"
|
||||
rm -rf "$out" "dist/$name.tar.gz"
|
||||
mkdir -p "$out"
|
||||
|
||||
# Standardwerte in der Compose-Datei durch die Release-Werte ersetzen.
|
||||
sed -e "s|\${OPNEXUS_IMAGE_REPO:-local}|\${OPNEXUS_IMAGE_REPO:-$repo}|g" \
|
||||
-e "s|\${OPNEXUS_VERSION:-latest}|\${OPNEXUS_VERSION:-$version}|g" \
|
||||
docker-compose.yml > "$out/docker-compose.yml"
|
||||
grep -q "OPNEXUS_IMAGE_REPO:-$repo" "$out/docker-compose.yml" || { echo "Compose-Defaults nicht ersetzt" >&2; exit 1; }
|
||||
! grep -qE '^\s+build:' "$out/docker-compose.yml" || { echo "build: im Release-Compose" >&2; exit 1; }
|
||||
# Standardwerte in der Compose-Datei durch die Release-Werte ersetzen (dieselbe Datei steckt im Updater-Image).
|
||||
scripts/release_compose.sh "$version" "$repo" > "$out/docker-compose.yml"
|
||||
|
||||
# .env.example mit gesetzter Registry und Version (der Updater liest .env).
|
||||
sed -e "s|^# OPNEXUS_IMAGE_REPO=.*|OPNEXUS_IMAGE_REPO=$repo|" \
|
||||
@@ -37,7 +33,7 @@ cp cosign.pub "$out/"
|
||||
cat > "$out/INSTALL.txt" <<TXT
|
||||
OPNexus $version
|
||||
|
||||
1. cp .env.example .env und POSTGRES_PASSWORD (und was sonst nötig ist) eintragen
|
||||
1. install -m 600 .env.example .env und POSTGRES_PASSWORD (und was sonst nötig ist) eintragen
|
||||
2. docker compose pull && docker compose up -d --no-build
|
||||
3. Anmelden: ./opnexus password (zeigt das Admin-Passwort, wenn ADMIN_PASSWORD leer war)
|
||||
dann im Browser öffnen — der Einrichtungsassistent führt durch den Rest
|
||||
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Gibt die Release-Fassung der docker-compose.yml aus: Registry und Version als Standard eingebacken, kein build:.
|
||||
# Dieselbe Datei kommt ins Release-Bundle und ins Updater-Image (dort holt `opnexus update` sie ab, B18).
|
||||
#
|
||||
# scripts/release_compose.sh VERSION IMAGE_REPO > docker-compose.yml
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
version="${1:?Usage: $0 VERSION IMAGE_REPO}"
|
||||
repo="${2:?Usage: $0 VERSION IMAGE_REPO}"
|
||||
out=$(sed -e "s|\${OPNEXUS_IMAGE_REPO:-local}|\${OPNEXUS_IMAGE_REPO:-$repo}|g" \
|
||||
-e "s|\${OPNEXUS_VERSION:-latest}|\${OPNEXUS_VERSION:-$version}|g" docker-compose.yml)
|
||||
grep -q "OPNEXUS_IMAGE_REPO:-$repo" <<<"$out" || { echo "Compose-Defaults nicht ersetzt" >&2; exit 1; }
|
||||
! grep -qE '^\s+build:' <<<"$out" || { echo "build: im Release-Compose" >&2; exit 1; }
|
||||
printf '%s\n' "$out"
|
||||
+8
-2
@@ -19,9 +19,15 @@ RUN set -e; arch="${TARGETARCH:-amd64}"; \
|
||||
echo "$sha /usr/local/bin/cosign" | sha256sum -c -; \
|
||||
chmod 0755 /usr/local/bin/cosign
|
||||
|
||||
COPY opnexus /usr/local/bin/opnexus
|
||||
# Die Release-Fassung der Compose-Datei legt scripts/build_release_images.sh in den Build-Kontext; `opnexus update`
|
||||
# übernimmt sie von hier (B18). Bei einem lokalen Build ohne sie bleibt /usr/share/opnexus leer. (Zwei Quellen, damit
|
||||
# das Muster auch ohne Treffer gültig ist.)
|
||||
COPY opnexus docker-compose.release.y[m]l /tmp/opx/
|
||||
COPY server.py /app/server.py
|
||||
RUN chmod 0755 /usr/local/bin/opnexus
|
||||
RUN install -m 0755 /tmp/opx/opnexus /usr/local/bin/opnexus \
|
||||
&& mkdir -p /usr/share/opnexus \
|
||||
&& if [ -f /tmp/opx/docker-compose.release.yml ]; then install -m 0644 /tmp/opx/docker-compose.release.yml /usr/share/opnexus/docker-compose.yml; fi \
|
||||
&& rm -rf /tmp/opx
|
||||
|
||||
WORKDIR /app
|
||||
ENV PYTHONUNBUFFERED=1 OPNEXUS_DIR=/project UPDATER_STATE_DIR=/state
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
+321
-16
@@ -7,14 +7,20 @@
|
||||
# bei Fehler automatisch zurückrollen
|
||||
# opnexus rollback Letztes Update rückgängig machen
|
||||
# opnexus backup Sofort-Backup der OPNexus-Datenbank
|
||||
# opnexus backup --full [DATEI]
|
||||
# opnexus backup --full [DATEI] [--with-env] [--with-ca]
|
||||
# Vollsicherung: Datenbank UND alle Geheimnisse
|
||||
# (Firewall-Zugänge, 2FA, VPN-/Zertifikatsschlüssel,
|
||||
# Konfigurations-Backups) in einer verschlüsselten
|
||||
# Datei (.opxfull). Passphrase wird abgefragt
|
||||
# (oder OPNEXUS_BACKUP_PASSPHRASE).
|
||||
# opnexus restore DATEI Vollsicherung zurückspielen (überschreibt Datenbank
|
||||
# und Geheimnisse dieser Installation; --yes: ohne Rückfrage)
|
||||
# (oder OPNEXUS_BACKUP_PASSPHRASE). Optional auch
|
||||
# die .env und die CA des Proxys (für einen Umzug).
|
||||
# opnexus restore DATEI [--with-env] [--with-ca] [--yes]
|
||||
# Vollsicherung zurückspielen (überschreibt Datenbank
|
||||
# und Geheimnisse dieser Installation)
|
||||
# opnexus migrate-host ZIEL [--dir PFAD] [--yes]
|
||||
# Umzug auf einen anderen Server (ssh ZIEL, dort ist
|
||||
# OPNexus schon installiert): Quelle anhalten, sichern,
|
||||
# am Ziel zurückspielen, prüfen. Siehe Handbuch, "Umzug".
|
||||
# opnexus password [NAME] Neues Zufallspasswort für den Admin setzen und
|
||||
# anzeigen (--yes: ohne Rückfrage)
|
||||
# opnexus export [--out VERZEICHNIS] [--kinds a,b]
|
||||
@@ -71,7 +77,7 @@ protect_env() {
|
||||
# Variablen im Aufrufer (dort als local deklarieren).
|
||||
read_update_state() {
|
||||
local key value
|
||||
for key in started prev_version prev_setting prev_db_revision target; do
|
||||
for key in started prev_version prev_setting prev_db_revision target prev_compose prev_compose_sha; do
|
||||
value=$(sed -n "s/^$key=//p" "$STATE_FILE" | tail -n1)
|
||||
[[ "$value" =~ ^[A-Za-z0-9._-]*$ ]] || die "$STATE_FILE enthält einen ungültigen Wert für $key — er wird nicht verwendet."
|
||||
printf -v "$key" '%s' "$value"
|
||||
@@ -188,7 +194,7 @@ cmd_status_porcelain() {
|
||||
for svc in "${SERVICES[@]}"; do echo "service.$svc=$(service_state "$svc")"; done
|
||||
if [ -f "$STATE_FILE" ]; then
|
||||
# Nur bekannte Schlüssel durchlassen — die Datei wird vom Skript selbst geschrieben.
|
||||
grep -E '^(started|prev_version|prev_setting|prev_db_revision|target)=' "$STATE_FILE" | sed 's/^/last_update./'
|
||||
grep -E '^(started|prev_version|prev_setting|prev_db_revision|target|prev_compose)=' "$STATE_FILE" | sed 's/^/last_update./'
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -200,6 +206,14 @@ cmd_status() {
|
||||
echo "Laufende Version: $(running_version)"
|
||||
echo "DB-Revision: $(db_revision)"
|
||||
for svc in "${SERVICES[@]}"; do echo "$svc: $(service_state "$svc")"; done
|
||||
if [ -f "$LAST_FULL_FILE" ]; then
|
||||
local t age
|
||||
t=$(sed -n 's/^time=//p' "$LAST_FULL_FILE")
|
||||
age=$(( ( $(date +%s) - $(date -d "$t" +%s 2>/dev/null || date +%s) ) / 86400 ))
|
||||
echo "Letzte Vollsicherung: $t (vor $age Tag(en)): $(sed -n 's/^file=//p' "$LAST_FULL_FILE")"
|
||||
else
|
||||
echo "Letzte Vollsicherung: keine von diesem Server bekannt ('opnexus backup --full')"
|
||||
fi
|
||||
[ -f "$STATE_FILE" ] && { echo; echo "Letztes Update (für 'rollback'):"; sed 's/^/ /' "$STATE_FILE"; }
|
||||
return 0
|
||||
}
|
||||
@@ -225,23 +239,126 @@ ask_passphrase() {
|
||||
printf '%s' "$one"
|
||||
}
|
||||
|
||||
LAST_FULL_FILE=".opnexus-last-full-backup"
|
||||
|
||||
# Dateien dieses Servers, die nicht in den Volumes des Backends liegen: die .env und die interne CA des Proxys
|
||||
# (dieselbe CA = Browser, die ihr vertrauen, vertrauen auch nach einem Umzug). Als tar auf stdout.
|
||||
host_files_tar() {
|
||||
local with_env="$1" with_ca="$2" tmp
|
||||
tmp=$(mktemp -d) || return 1
|
||||
chmod 700 "$tmp"
|
||||
if [ "$with_env" = 1 ]; then install -m 600 "$ENV_FILE" "$tmp/env" || { rm -rf "$tmp"; return 1; }; fi
|
||||
if [ "$with_ca" = 1 ]; then
|
||||
mkdir -m 700 "$tmp/caddy-ca"
|
||||
# Aus dem Volume, nicht aus dem laufenden Proxy: beim Umzug ist er schon angehalten.
|
||||
c run --rm --no-deps -T --entrypoint tar proxy -C /data/caddy/pki/authorities -cf - local | tar -x -C "$tmp/caddy-ca" \
|
||||
|| { rm -rf "$tmp"; die "Die CA des Proxys ließ sich nicht lesen (hat der Proxy schon einmal gestartet?)."; }
|
||||
fi
|
||||
tar -C "$tmp" -cf - . ; local rc=$?
|
||||
rm -rf "$tmp"
|
||||
return $rc
|
||||
}
|
||||
|
||||
cmd_backup_full() {
|
||||
local out="${1:-./opnexus-full-$(date -u +%Y%m%d-%H%M%S).opxfull}" pass
|
||||
local out="" with_env=0 with_ca=0 pass arg extra=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--with-env) with_env=1 ;;
|
||||
--with-ca) with_ca=1 ;;
|
||||
-*) die "Unbekannte Option: $arg" ;;
|
||||
*) out="$arg" ;;
|
||||
esac
|
||||
done
|
||||
out="${out:-./opnexus-full-$(date -u +%Y%m%d-%H%M%S).opxfull}"
|
||||
[ ! -e "$out" ] || die "$out existiert schon."
|
||||
c exec -T backend true >/dev/null 2>&1 || die "Das Backend läuft nicht — erst starten (docker compose up -d)."
|
||||
pass="$(ask_passphrase confirm)"
|
||||
( umask 077
|
||||
FULL_BACKUP_PASSPHRASE="$pass" c exec -T -e FULL_BACKUP_PASSPHRASE backend python -m app.full_backup create > "$out.partial" ) \
|
||||
|| { rm -f "$out.partial"; die "Vollsicherung fehlgeschlagen."; }
|
||||
if [ "$with_env" = 1 ] || [ "$with_ca" = 1 ]; then
|
||||
extra=(--host-files)
|
||||
( umask 077
|
||||
host_files_tar "$with_env" "$with_ca" \
|
||||
| FULL_BACKUP_PASSPHRASE="$pass" c exec -T -e FULL_BACKUP_PASSPHRASE backend python -m app.full_backup create "${extra[@]}" > "$out.partial"
|
||||
rcs=("${PIPESTATUS[@]}"); [ "${rcs[0]}" = 0 ] && [ "${rcs[1]}" = 0 ] ) || { rm -f "$out.partial"; die "Vollsicherung fehlgeschlagen."; }
|
||||
else
|
||||
( umask 077
|
||||
FULL_BACKUP_PASSPHRASE="$pass" c exec -T -e FULL_BACKUP_PASSPHRASE backend python -m app.full_backup create > "$out.partial" ) \
|
||||
|| { rm -f "$out.partial"; die "Vollsicherung fehlgeschlagen."; }
|
||||
fi
|
||||
mv "$out.partial" "$out"; chmod 600 "$out"
|
||||
printf 'time=%s\nfile=%s\nenv=%s\nca=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$(readlink -f "$out")" "$with_env" "$with_ca" > "$LAST_FULL_FILE"
|
||||
say "Vollsicherung geschrieben: $out ($(du -h "$out" | cut -f1))"
|
||||
[ "$with_env" = 1 ] && say "Enthält die .env (Zugangsdaten dieses Servers)."
|
||||
[ "$with_ca" = 1 ] && say "Enthält die CA des Proxys."
|
||||
say "Ohne die Passphrase ist die Datei wertlos. Beides getrennt vom Server aufbewahren."
|
||||
}
|
||||
|
||||
# Schlüssel der .env, die beim Zurückspielen NIE aus der Sicherung übernommen werden: die Datenbank dieses Servers
|
||||
# (ihr Passwort wurde bei seiner Installation gesetzt, die Sicherung ändert es nicht), die installierte Version und
|
||||
# die Adressen, unter denen DIESER Server erreichbar ist.
|
||||
ENV_KEEP_KEYS="POSTGRES_PASSWORD POSTGRES_USER POSTGRES_DB OPNEXUS_VERSION OPNEXUS_IMAGE_REPO OPNEXUS_SITES OPNEXUS_DEFAULT_SNI FRONTEND_ORIGIN BIND_ADDRESS PROXY_BIND_ADDRESS GUI_BIND_ADDRESS OPNEXUS_GUI_SITES"
|
||||
|
||||
# Übernimmt die übrigen Schlüssel aus der gesicherten .env ($1). Schreibt die gesicherte Datei vollständig als
|
||||
# .env.from-backup daneben (zum Nachsehen) und die bisherige als .env.before-restore.
|
||||
merge_env_from_backup() {
|
||||
local backup="$1" yes="$2" key value changed=() line
|
||||
install -m 600 "$backup" "$ENV_FILE.from-backup"
|
||||
while IFS= read -r line; do
|
||||
[[ "$line" =~ ^([A-Za-z_][A-Za-z0-9_]*)=(.*)$ ]] || continue
|
||||
key="${BASH_REMATCH[1]}"; value="${BASH_REMATCH[2]}"
|
||||
[[ " $ENV_KEEP_KEYS " == *" $key "* ]] && continue
|
||||
[ "$(env_get "$key")" = "$value" ] && grep -q "^$key=" "$ENV_FILE" && continue
|
||||
changed+=("$key")
|
||||
done < "$backup"
|
||||
if [ "${#changed[@]}" = 0 ]; then say ".env: keine abweichenden Einstellungen in der Sicherung."; return 0; fi
|
||||
say ".env: diese Einstellungen kommen aus der Sicherung: ${changed[*]}"
|
||||
say "(Unverändert bleiben Datenbank-Zugang, Version und Adressen dieses Servers; die ganze gesicherte Datei liegt in $ENV_FILE.from-backup.)"
|
||||
if [ "$yes" != 1 ]; then
|
||||
read -rp "Übernehmen? [j/N] " ans </dev/tty
|
||||
[[ "$ans" =~ ^[jJyY]$ ]] || { say ".env unverändert gelassen."; return 0; }
|
||||
fi
|
||||
install -m 600 "$ENV_FILE" "$ENV_FILE.before-restore"
|
||||
for key in "${changed[@]}"; do
|
||||
value=$(sed -n "s/^$key=//p" "$backup" | tail -n1)
|
||||
env_set_literal "$key" "$value"
|
||||
done
|
||||
say ".env aktualisiert (vorher: $ENV_FILE.before-restore)."
|
||||
}
|
||||
|
||||
# Wie env_set, aber für beliebige Werte (Passwörter, URLs mit | oder &): ohne sed-Ersetzung.
|
||||
env_set_literal() {
|
||||
local key="$1" value="$2" tmp
|
||||
tmp=$(mktemp "$ENV_FILE.XXXXXX") && chmod 600 "$tmp"
|
||||
if grep -q "^$key=" "$ENV_FILE"; then
|
||||
awk -v k="$key" -v v="$value" 'BEGIN{FS=OFS="="} index($0, k"=")==1 {print k"="v; next} {print}' "$ENV_FILE" > "$tmp"
|
||||
else
|
||||
cat "$ENV_FILE" > "$tmp"; printf '%s=%s\n' "$key" "$value" >> "$tmp"
|
||||
fi
|
||||
mv "$tmp" "$ENV_FILE"
|
||||
}
|
||||
|
||||
# Die gesicherte CA in das Volume des Proxys legen; die ausgestellten Zertifikate werden verworfen, damit Caddy sie
|
||||
# mit dieser CA neu ausstellt. Der Proxy ist dabei gestoppt.
|
||||
restore_ca() {
|
||||
local dir="$1"
|
||||
[ -d "$dir/local" ] || die "Die Sicherung enthält keine CA."
|
||||
tar -C "$dir" -cf - local | c run --rm --no-deps -T --entrypoint sh proxy -c \
|
||||
'rm -rf /data/caddy/pki/authorities/local /data/caddy/certificates/local && mkdir -p /data/caddy/pki/authorities && tar -x -C /data/caddy/pki/authorities' \
|
||||
|| die "CA konnte nicht zurückgespielt werden."
|
||||
say "CA des Proxys zurückgespielt (Zertifikate werden beim Start neu ausgestellt)."
|
||||
}
|
||||
|
||||
cmd_restore() {
|
||||
local file="" yes=0 pass arg
|
||||
for arg in "$@"; do case "$arg" in --yes|-y) yes=1 ;; *) file="$arg" ;; esac; done
|
||||
[ -n "$file" ] && [ -f "$file" ] || die "Aufruf: opnexus restore DATEI.opxfull [--yes]"
|
||||
local file="" yes=0 pass arg with_env=0 with_ca=0 hosttmp=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--yes|-y) yes=1 ;;
|
||||
--with-env) with_env=1 ;;
|
||||
--with-ca) with_ca=1 ;;
|
||||
-*) die "Unbekannte Option: $arg" ;;
|
||||
*) file="$arg" ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$file" ] && [ -f "$file" ] || die "Aufruf: opnexus restore DATEI.opxfull [--with-env] [--with-ca] [--yes]"
|
||||
pass="$(ask_passphrase)"
|
||||
say "Prüfe die Datei …"
|
||||
FULL_BACKUP_PASSPHRASE="$pass" c run --rm --no-deps -T -e FULL_BACKUP_PASSPHRASE backend python -m app.full_backup info < "$file" \
|
||||
@@ -258,6 +375,18 @@ cmd_restore() {
|
||||
say "Spiele zurück …"
|
||||
FULL_BACKUP_PASSPHRASE="$pass" c run --rm --no-deps -T -e FULL_BACKUP_PASSPHRASE backend python -m app.full_backup restore < "$file" \
|
||||
|| die "Zurückspielen fehlgeschlagen. Der vorherige Stand liegt als Dump in /backups; Dienste sind gestoppt."
|
||||
if [ "$with_env" = 1 ] || [ "$with_ca" = 1 ]; then
|
||||
hosttmp=$(mktemp -d) && chmod 700 "$hosttmp"
|
||||
FULL_BACKUP_PASSPHRASE="$pass" c run --rm --no-deps -T -e FULL_BACKUP_PASSPHRASE backend python -m app.full_backup host-files < "$file" \
|
||||
| tar -x -C "$hosttmp" || { rm -rf "$hosttmp"; die "Die Dateien des Servers (.env/CA) ließen sich nicht lesen. Datenbank und Geheimnisse sind schon zurückgespielt; Dienste sind gestoppt."; }
|
||||
if [ "$with_env" = 1 ]; then
|
||||
if [ -f "$hosttmp/env" ]; then merge_env_from_backup "$hosttmp/env" "$yes"; else say "Die Sicherung enthält keine .env (nicht mit --with-env erstellt)."; fi
|
||||
fi
|
||||
if [ "$with_ca" = 1 ]; then
|
||||
if [ -d "$hosttmp/caddy-ca" ]; then restore_ca "$hosttmp/caddy-ca"; else say "Die Sicherung enthält keine CA (nicht mit --with-ca erstellt)."; fi
|
||||
fi
|
||||
rm -rf "$hosttmp"
|
||||
fi
|
||||
say "Starte die Dienste …"
|
||||
c up -d backend frontend proxy >/dev/null
|
||||
wait_healthy || die "Nicht alle Dienste sind gesund — 'docker compose logs' prüfen."
|
||||
@@ -339,12 +468,88 @@ refresh_cli() {
|
||||
fi
|
||||
[ "$keep" = 1 ] || docker rmi "$ref" >/dev/null 2>&1 || true
|
||||
if cmp -s "$tmp" "$(readlink -f "$0")"; then rm -f "$tmp"; return 0; fi
|
||||
# Im Release-Bundle (CLI neben docker-compose.yml) auch die Datei auf der Platte ersetzen: sonst liefen ein späteres
|
||||
# `rollback` oder `status` mit der alten CLI, die neue Schritte (z. B. die Compose-Datei zurücklegen) nicht kennt.
|
||||
# Über eine neue Datei und Umbenennen: die laufende Bash liest ihr Skript weiter aus der alten.
|
||||
local self; self="$(readlink -f "$0")"
|
||||
if [ -f "$(dirname "$self")/docker-compose.yml" ] && [ -w "$self" ] && [ -w "$(dirname "$self")" ]; then
|
||||
cp -p "$self" "$self.bak" 2>/dev/null || true
|
||||
if install -m 0755 "$tmp" "$self.new" && mv -f "$self.new" "$self"; then
|
||||
say "CLI auf der Platte aktualisiert (vorherige: $(basename "$self").bak)."
|
||||
fi
|
||||
fi
|
||||
say "Die CLI des Zielreleases (Signatur geprüft) ist neuer — starte sie."
|
||||
chmod 0700 "$tmp"
|
||||
export OPNEXUS_DIR="$PWD"
|
||||
exec bash "$tmp" __refreshed update "$target"
|
||||
}
|
||||
|
||||
# Die Compose-Datei des Zielreleases übernehmen (B18): `update` tauschte bisher nur Images, neue Einstellungen der Dienste
|
||||
# (z. B. Härtung, neue Variablen) kamen nie an. Quelle ist das signierte Updater-Image des Ziels (wie bei refresh_cli),
|
||||
# nie das Netz direkt. Wurde die Datei hier angepasst (Prüfsumme weicht von der zuletzt übernommenen ab), bleibt sie, und
|
||||
# die neue Fassung liegt als docker-compose.yml.new daneben: eigene Anpassungen gehören in docker-compose.override.yml.
|
||||
COMPOSE_SHA_FILE=".opnexus-compose-sha"
|
||||
COMPOSE_BACKUP=""
|
||||
COMPOSE_PREV_SHA=""
|
||||
|
||||
refresh_compose() {
|
||||
local target="$1" repo ref digest tmp keep=0 cur_sha rec_sha label
|
||||
[ "${OPNEXUS_NO_COMPOSE_REFRESH:-}" != 1 ] || return 0
|
||||
repo=$(env_get OPNEXUS_IMAGE_REPO)
|
||||
[ -n "$repo" ] || return 0
|
||||
if ! command -v cosign >/dev/null 2>&1 || [ ! -f cosign.pub ]; then
|
||||
say "Compose-Datei nicht abgeglichen (ohne cosign und cosign.pub wird keine Datei aus einem Image übernommen)."; return 0
|
||||
fi
|
||||
ref="$repo/opnexus-updater:$target"
|
||||
if updater_enabled; then keep=1; fi
|
||||
docker pull -q "$ref" >/dev/null 2>&1 || { say "Compose-Datei nicht abgeglichen (Updater-Image des Ziels nicht ladbar)."; return 0; }
|
||||
digest=$(docker image inspect --format '{{index .RepoDigests 0}}' "$ref" 2>/dev/null || true)
|
||||
if [[ "$digest" != *@sha256:* ]]; then
|
||||
[ "$keep" = 1 ] || docker rmi "$ref" >/dev/null 2>&1 || true
|
||||
say "Compose-Datei nicht abgeglichen (Digest des Updater-Images unbekannt)."; return 0
|
||||
fi
|
||||
cosign verify --key cosign.pub --insecure-ignore-tlog=true "$digest" >/dev/null 2>&1 \
|
||||
|| die "Signatur des Updater-Images $ref ist ungültig oder fehlt — nichts wurde verändert."
|
||||
tmp=$(mktemp "${TMPDIR:-/tmp}/opnexus-compose.XXXXXX") || return 0
|
||||
if ! docker run --rm --pull never --entrypoint cat "$digest" /usr/share/opnexus/docker-compose.yml > "$tmp" 2>/dev/null \
|
||||
|| ! grep -q '^services:' "$tmp" || ! grep -q 'opnexus-backend:' "$tmp"; then
|
||||
rm -f "$tmp"; [ "$keep" = 1 ] || docker rmi "$ref" >/dev/null 2>&1 || true
|
||||
say "Das Zielrelease liefert keine Compose-Datei mit (vor 1.34) — die vorhandene bleibt."; return 0
|
||||
fi
|
||||
[ "$keep" = 1 ] || docker rmi "$ref" >/dev/null 2>&1 || true
|
||||
if cmp -s "$tmp" docker-compose.yml; then
|
||||
rm -f "$tmp"; sha256sum docker-compose.yml | cut -d' ' -f1 > "$COMPOSE_SHA_FILE"; return 0
|
||||
fi
|
||||
cur_sha=$(sha256sum docker-compose.yml | cut -d' ' -f1)
|
||||
rec_sha=$(cat "$COMPOSE_SHA_FILE" 2>/dev/null || true)
|
||||
if [ -n "$rec_sha" ] && [ "$rec_sha" != "$cur_sha" ]; then
|
||||
install -m 644 "$tmp" docker-compose.yml.new; rm -f "$tmp"
|
||||
say "docker-compose.yml wurde hier angepasst und bleibt unverändert. Die Fassung von $target liegt als docker-compose.yml.new daneben;"
|
||||
say "Anpassungen gehören in docker-compose.override.yml (dann übernimmt das nächste Update die Datei selbst)."
|
||||
return 0
|
||||
fi
|
||||
label=$(running_version); label="${label:-alt}"
|
||||
COMPOSE_BACKUP="docker-compose.yml.bak-$label"
|
||||
[ -e "$COMPOSE_BACKUP" ] && COMPOSE_BACKUP="docker-compose.yml.bak-$label-$(date -u +%Y%m%d%H%M%S)"
|
||||
COMPOSE_PREV_SHA="$rec_sha"
|
||||
cp -p docker-compose.yml "$COMPOSE_BACKUP"
|
||||
install -m 644 "$tmp" docker-compose.yml; rm -f "$tmp"
|
||||
sha256sum docker-compose.yml | cut -d' ' -f1 > "$COMPOSE_SHA_FILE"
|
||||
say "Compose-Datei von $target übernommen (vorherige: $COMPOSE_BACKUP)."
|
||||
if [ -z "$rec_sha" ]; then
|
||||
say "Falls du die alte Datei angepasst hattest: 'diff $COMPOSE_BACKUP docker-compose.yml' zeigt den Unterschied;"
|
||||
say "eigene Anpassungen gehören in docker-compose.override.yml."
|
||||
fi
|
||||
}
|
||||
|
||||
# Beim Rollback die Compose-Datei von vorher zurücklegen (nur wenn das Update sie getauscht hatte).
|
||||
restore_compose() {
|
||||
[ -n "${prev_compose:-}" ] && [ -f "$prev_compose" ] || return 0
|
||||
cp -p "$prev_compose" docker-compose.yml
|
||||
if [ -n "${prev_compose_sha:-}" ]; then echo "$prev_compose_sha" > "$COMPOSE_SHA_FILE"; else rm -f "$COMPOSE_SHA_FILE"; fi
|
||||
say "Compose-Datei von vorher wiederhergestellt ($prev_compose)."
|
||||
}
|
||||
|
||||
cmd_update() {
|
||||
require_release_mode
|
||||
protect_env
|
||||
@@ -366,7 +571,15 @@ cmd_update() {
|
||||
OPNEXUS_VERSION="$target" c pull "${pull_services[@]}" || die "Pull fehlgeschlagen — nichts wurde verändert."
|
||||
verify_signatures "$target" "${pull_services[@]}"
|
||||
refresh_cli "$target"
|
||||
preflight_proxy_config "$target"
|
||||
refresh_compose "$target"
|
||||
# In einer Subshell: scheitert die Vorprüfung, kommt die eben getauschte Compose-Datei zurück ("nichts verändert").
|
||||
if ! ( preflight_proxy_config "$target" ); then
|
||||
if [ -n "$COMPOSE_BACKUP" ]; then
|
||||
cp -p "$COMPOSE_BACKUP" docker-compose.yml
|
||||
if [ -n "$COMPOSE_PREV_SHA" ]; then echo "$COMPOSE_PREV_SHA" > "$COMPOSE_SHA_FILE"; else rm -f "$COMPOSE_SHA_FILE"; fi
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
started=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
cat > "$STATE_FILE" <<STATE
|
||||
@@ -375,6 +588,8 @@ prev_version=$prev_version
|
||||
prev_setting=$prev_setting
|
||||
prev_db_revision=$prev_rev
|
||||
target=$target
|
||||
prev_compose=$COMPOSE_BACKUP
|
||||
prev_compose_sha=$COMPOSE_PREV_SHA
|
||||
STATE
|
||||
|
||||
env_set OPNEXUS_VERSION "$target"
|
||||
@@ -409,7 +624,7 @@ latest_pre_upgrade_dump() {
|
||||
cmd_rollback() {
|
||||
require_release_mode
|
||||
[ -f "$STATE_FILE" ] || die "Kein Update-Zustand ($STATE_FILE) — es gibt nichts zurückzurollen."
|
||||
local started prev_version prev_setting prev_db_revision target
|
||||
local started prev_version prev_setting prev_db_revision target prev_compose prev_compose_sha
|
||||
read_update_state
|
||||
[ -n "${prev_version:-}" ] || die "$STATE_FILE ist unvollständig."
|
||||
|
||||
@@ -463,6 +678,7 @@ cmd_rollback() {
|
||||
fi
|
||||
|
||||
env_set OPNEXUS_VERSION "$prev_version"
|
||||
restore_compose
|
||||
phase switching
|
||||
local rb_targets=()
|
||||
mapfile -t rb_targets < <(up_targets)
|
||||
@@ -573,9 +789,97 @@ cmd_apply() {
|
||||
exit "$code"
|
||||
}
|
||||
|
||||
# ── Umzug auf einen anderen Server (HA-Plan H1: gehärtete Kaltreserve) ─────────────────────────────────────────
|
||||
# Am Ziel ist OPNexus schon nach Anleitung installiert (gleiche oder neuere Version). Ablauf: Vorprüfung →
|
||||
# Quelle anhalten (keine Änderungen mehr) → Vollsicherung inkl. .env und CA → am Ziel zurückspielen → prüfen.
|
||||
# Die Passphrase ist zufällig, lebt nur in diesem Lauf und geht über stdin zum Ziel (nie in eine Befehlszeile).
|
||||
# Die Quelle bleibt angehalten, nicht gelöscht: zurück geht es mit `docker compose up -d` hier.
|
||||
remote() { ssh -o BatchMode=yes -o ConnectTimeout=10 "$MIG_TARGET" "$@"; }
|
||||
|
||||
cmd_migrate_host() {
|
||||
local yes=0 dir="" arg t0 pass tmpfile remote_cli remote_file rv lv hosts_before hosts_after st
|
||||
MIG_TARGET=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--yes|-y) yes=1 ;;
|
||||
--dir) shift; dir="${1:-}" ;;
|
||||
-*) die "Unbekannte Option: $1" ;;
|
||||
*) MIG_TARGET="$1" ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
[ -n "$MIG_TARGET" ] || die "Aufruf: opnexus migrate-host BENUTZER@ZIEL [--dir PFAD] [--yes]"
|
||||
require_release_mode
|
||||
dir="${dir:-$PWD}"
|
||||
t0=$SECONDS
|
||||
|
||||
say "Vorprüfung …"
|
||||
remote true || die "Keine SSH-Verbindung zu $MIG_TARGET (Schlüssel-Anmeldung nötig)."
|
||||
remote_cli=$(remote "cd '$dir' 2>/dev/null && { [ -x ./opnexus ] && echo ./opnexus || { [ -x scripts/opnexus ] && echo scripts/opnexus; }; }") \
|
||||
|| die "Auf $MIG_TARGET gibt es $dir nicht (--dir angeben)."
|
||||
[ -n "$remote_cli" ] || die "Auf $MIG_TARGET liegt in $dir kein opnexus-Werkzeug: dort zuerst OPNexus installieren."
|
||||
st=$(remote "cd '$dir' && $remote_cli status --porcelain") || die "Am Ziel antwortet '$remote_cli status' nicht."
|
||||
rv=$(sed -n 's/^running_version=//p' <<<"$st"); lv=$(running_version)
|
||||
[ -n "$rv" ] || die "Am Ziel läuft OPNexus nicht (erst starten: docker compose up -d)."
|
||||
[ "$(printf '%s\n%s\n' "$lv" "$rv" | sort -V | head -n1)" = "$lv" ] \
|
||||
|| die "Am Ziel läuft $rv, hier $lv: das Ziel muss gleich oder neuer sein ('$remote_cli update $lv' dort)."
|
||||
grep -qE '^service\.backend=healthy$' <<<"$st" || die "Das Backend am Ziel ist nicht gesund."
|
||||
hosts_before=$(psql_db -c "SELECT count(*) FROM opnsense_hosts")
|
||||
say "Quelle: $lv, $hosts_before Firewall(s). Ziel $MIG_TARGET:$dir: $rv."
|
||||
|
||||
if [ "$yes" != 1 ]; then
|
||||
echo "Danach ist OPNexus HIER angehalten und das Ziel enthält diese Installation (seine bisherigen Daten werden überschrieben)."
|
||||
read -rp "Umzug starten? [j/N] " ans </dev/tty
|
||||
[[ "$ans" =~ ^[jJyY]$ ]] || die "Abgebrochen."
|
||||
fi
|
||||
|
||||
pass=$(head -c 32 /dev/urandom | base64 | tr -d '/+=' | head -c 40)
|
||||
tmpfile=$(mktemp --suffix=.opxfull) && rm -f "$tmpfile"
|
||||
say "Halte Proxy und Oberfläche an (ab jetzt keine Änderungen mehr) …"
|
||||
c stop proxy frontend >/dev/null 2>&1 || true
|
||||
say "Vollsicherung mit .env und CA …"
|
||||
( export OPNEXUS_BACKUP_PASSPHRASE="$pass"
|
||||
cmd_backup_full "$tmpfile" --with-env --with-ca >/dev/null ) \
|
||||
|| { c up -d >/dev/null 2>&1; die "Vollsicherung fehlgeschlagen; die Quelle läuft wieder."; }
|
||||
c stop >/dev/null 2>&1 || true
|
||||
rm -f "$LAST_FULL_FILE" # die Datei wird gleich gelöscht, sie taugt nicht als "letzte Vollsicherung"
|
||||
|
||||
remote_file="/tmp/opnexus-migrate-$$.opxfull"
|
||||
say "Übertrage ($(du -h "$tmpfile" | cut -f1)) …"
|
||||
scp -q -o BatchMode=yes "$tmpfile" "$MIG_TARGET:$remote_file" && remote "chmod 600 '$remote_file'" \
|
||||
|| { rm -f "$tmpfile"; die "Übertragung fehlgeschlagen. Die Quelle ist angehalten: 'docker compose up -d' startet sie wieder."; }
|
||||
rm -f "$tmpfile"
|
||||
|
||||
say "Spiele am Ziel zurück …"
|
||||
if ! printf '%s\n' "$pass" | remote "read -r OPNEXUS_BACKUP_PASSPHRASE; export OPNEXUS_BACKUP_PASSPHRASE; cd '$dir' && $remote_cli restore '$remote_file' --with-env --with-ca --yes; rc=\$?; rm -f '$remote_file'; exit \$rc"; then
|
||||
die "Zurückspielen am Ziel fehlgeschlagen (Ausgabe oben). Die Quelle ist angehalten: 'docker compose up -d' startet sie wieder."
|
||||
fi
|
||||
st=$(remote "cd '$dir' && $remote_cli status --porcelain") || st=""
|
||||
hosts_after=$(remote "cd '$dir' && docker compose exec -T postgres sh -c 'psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -Atc \"SELECT count(*) FROM opnsense_hosts\"'" 2>/dev/null | tr -d '\r')
|
||||
[ "$hosts_after" = "$hosts_before" ] \
|
||||
|| die "Am Ziel stehen $hosts_after statt $hosts_before Firewall(s). Bitte prüfen; die Quelle ist angehalten ('docker compose up -d' startet sie wieder)."
|
||||
grep -qE '^service\.backend=healthy$' <<<"$st" || die "Das Backend am Ziel ist nach dem Zurückspielen nicht gesund."
|
||||
say "Umzug fertig nach $(( SECONDS - t0 )) s: $MIG_TARGET führt OPNexus mit $hosts_after Firewall(s)."
|
||||
echo
|
||||
echo "Noch zu tun:"
|
||||
echo " - Adresse umstellen: virtuelle IP, DNS-Namen oder Lesezeichen auf das Ziel zeigen lassen."
|
||||
echo " - Firewalls, die nur bestimmte Absender an ihre API lassen, die Adresse des Ziels erlauben."
|
||||
echo " - Mit Management-Overlay: der Tunnel-Endpunkt (öffentliche Adresse, UDP-Port) muss zum Ziel zeigen."
|
||||
echo " - Hier bleibt alles angehalten (startet auch nach einem Neustart nicht). Zurück: am Ziel"
|
||||
echo " 'docker compose stop', hier 'docker compose up -d'."
|
||||
}
|
||||
|
||||
# Interner Neustart durch refresh_cli: Marker setzen und die eigene Temp-Datei aufräumen (bash hat sie offen).
|
||||
if [ "${1:-}" = __refreshed ]; then
|
||||
CLI_REFRESHED=1; shift
|
||||
# Gestartet von einer älteren CLI, deren refresh_cli die Datei im Bundle noch nicht ersetzt (vor 1.34): das hier
|
||||
# nachholen, sonst liefe ein späteres `rollback` mit der alten CLI.
|
||||
disk="${OPNEXUS_DIR:-}/opnexus"
|
||||
if [ -n "${OPNEXUS_DIR:-}" ] && [ -f "$disk" ] && [ -f "$OPNEXUS_DIR/docker-compose.yml" ] && [ -w "$OPNEXUS_DIR" ] \
|
||||
&& ! cmp -s "$0" "$disk"; then
|
||||
cp -p "$disk" "$disk.bak" 2>/dev/null || true
|
||||
if install -m 0755 "$0" "$disk.new" && mv -f "$disk.new" "$disk"; then say "CLI auf der Platte aktualisiert (vorherige: opnexus.bak)."; fi
|
||||
fi
|
||||
case "$0" in */opnexus-cli.*) rm -f "$0" ;; esac
|
||||
fi
|
||||
|
||||
@@ -588,5 +892,6 @@ case "${1:-}" in
|
||||
rollback) cmd_rollback ;;
|
||||
backup) if [ "${2:-}" = "--full" ]; then shift 2; cmd_backup_full "$@"; else cmd_backup; fi ;;
|
||||
restore) shift; cmd_restore "$@" ;;
|
||||
*) sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;;
|
||||
migrate-host) shift; cmd_migrate_host "$@" ;;
|
||||
*) sed -n '2,37p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;;
|
||||
esac
|
||||
@@ -0,0 +1,149 @@
|
||||
"""refresh_compose / restore_compose in `opnexus` (B18): `update` takes the target release's docker-compose.yml from the
|
||||
signed updater image, keeps a backup, never overwrites a file that was changed here, and `rollback` puts it back.
|
||||
Runs the real functions with fake `docker` and `cosign` in front of PATH.
|
||||
Run: cd updater && python3 -m unittest test_cli_compose"""
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT = Path(__file__).with_name("opnexus").resolve()
|
||||
DIGEST = "registry.example/opnexus/opnexus-updater@sha256:" + "b" * 64
|
||||
FAKE_DOCKER = """#!/bin/sh
|
||||
echo "$@" >> "$FAKE_LOG"
|
||||
case "$1" in
|
||||
pull) exit 0 ;;
|
||||
image) echo "$FAKE_DIGEST" ;;
|
||||
run) [ -f "$FAKE_COMPOSE" ] || exit 1; cat "$FAKE_COMPOSE" ;;
|
||||
compose) exit 0 ;;
|
||||
esac
|
||||
"""
|
||||
FAKE_COSIGN = """#!/bin/sh
|
||||
echo "cosign $*" >> "$FAKE_LOG"
|
||||
[ "${FAKE_COSIGN_RC:-0}" = 0 ]
|
||||
"""
|
||||
OLD = "services:\n backend:\n image: x/opnexus-backend:1\n"
|
||||
NEW = "services:\n backend:\n image: x/opnexus-backend:2\n security_opt: [\"no-new-privileges:true\"]\n"
|
||||
|
||||
|
||||
def _exe(path: Path, text: str):
|
||||
path.write_text(text)
|
||||
path.chmod(path.stat().st_mode | stat.S_IXUSR)
|
||||
|
||||
|
||||
def _sha(text: str) -> str:
|
||||
return hashlib.sha256(text.encode()).hexdigest()
|
||||
|
||||
|
||||
class ComposeRefresh(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = Path(tempfile.mkdtemp())
|
||||
self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True)
|
||||
(self.tmp / "bin").mkdir()
|
||||
_exe(self.tmp / "bin" / "docker", FAKE_DOCKER)
|
||||
_exe(self.tmp / "bin" / "cosign", FAKE_COSIGN)
|
||||
(self.tmp / ".env").write_text("OPNEXUS_IMAGE_REPO=registry.example/opnexus\n")
|
||||
(self.tmp / "cosign.pub").write_text("key")
|
||||
(self.tmp / "docker-compose.yml").write_text(OLD)
|
||||
self.offered = self.tmp / "offered.yml"
|
||||
self.offered.write_text(NEW)
|
||||
self.log = self.tmp / "calls.log"
|
||||
|
||||
def run_body(self, body: str, *, no_cosign=False, env_extra=None):
|
||||
if no_cosign:
|
||||
(self.tmp / "bin" / "cosign").unlink()
|
||||
path = f"{self.tmp / 'bin'}:/usr/bin:/bin"
|
||||
env = {**os.environ, "PATH": path, "OPNEXUS_DIR": str(self.tmp), "FAKE_LOG": str(self.log),
|
||||
"FAKE_COMPOSE": str(self.offered), "FAKE_DIGEST": DIGEST, "TMPDIR": str(self.tmp)}
|
||||
env.pop("OPNEXUS_NO_COMPOSE_REFRESH", None)
|
||||
env.update(env_extra or {})
|
||||
script = f'source <(sed "/^case \\"\\${{1:-}}\\" in/,\\$d" "{SCRIPT}"); {body}'
|
||||
return subprocess.run(["bash", "-c", script, str(SCRIPT)], env=env, capture_output=True, text=True, timeout=30)
|
||||
|
||||
def refresh(self, **kw):
|
||||
return self.run_body('refresh_compose 9.9.9; echo "BACKUP=$COMPOSE_BACKUP PREV=$COMPOSE_PREV_SHA"', **kw)
|
||||
|
||||
def compose(self) -> str:
|
||||
return (self.tmp / "docker-compose.yml").read_text()
|
||||
|
||||
def recorded(self) -> str | None:
|
||||
f = self.tmp / ".opnexus-compose-sha"
|
||||
return f.read_text().strip() if f.exists() else None
|
||||
|
||||
def test_first_update_replaces_the_file_and_keeps_a_backup(self):
|
||||
r = self.refresh()
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
self.assertEqual(self.compose(), NEW)
|
||||
self.assertEqual((self.tmp / "docker-compose.yml.bak-alt").read_text(), OLD)
|
||||
self.assertEqual(self.recorded(), _sha(NEW))
|
||||
self.assertIn("BACKUP=docker-compose.yml.bak-alt", r.stdout)
|
||||
self.assertIn("docker-compose.override.yml", r.stdout) # first time: point to the override file
|
||||
calls = self.log.read_text().splitlines()
|
||||
self.assertTrue(any(c.startswith("cosign verify") and c.endswith(DIGEST) for c in calls))
|
||||
run = next(c for c in calls if c.startswith("run "))
|
||||
self.assertTrue(run.endswith(f"{DIGEST} /usr/share/opnexus/docker-compose.yml")) # the verified digest, not the tag
|
||||
|
||||
def test_an_unchanged_file_from_the_last_update_is_replaced_quietly(self):
|
||||
(self.tmp / ".opnexus-compose-sha").write_text(_sha(OLD) + "\n")
|
||||
r = self.refresh()
|
||||
self.assertEqual(self.compose(), NEW)
|
||||
self.assertIn(f"PREV={_sha(OLD)}", r.stdout)
|
||||
self.assertNotIn("diff ", r.stdout)
|
||||
|
||||
def test_a_file_changed_here_is_kept_and_the_new_one_offered_beside_it(self):
|
||||
(self.tmp / ".opnexus-compose-sha").write_text(_sha(OLD) + "\n")
|
||||
(self.tmp / "docker-compose.yml").write_text(OLD + " # my change\n")
|
||||
r = self.refresh()
|
||||
self.assertEqual(self.compose(), OLD + " # my change\n")
|
||||
self.assertEqual((self.tmp / "docker-compose.yml.new").read_text(), NEW)
|
||||
self.assertIn("BACKUP= ", r.stdout)
|
||||
self.assertIn("docker-compose.override.yml", r.stdout)
|
||||
|
||||
def test_an_identical_file_only_records_its_checksum(self):
|
||||
(self.tmp / "docker-compose.yml").write_text(NEW)
|
||||
self.refresh()
|
||||
self.assertEqual(self.recorded(), _sha(NEW))
|
||||
self.assertFalse(list(self.tmp.glob("docker-compose.yml.bak-*")))
|
||||
|
||||
def test_an_old_release_without_a_compose_file_changes_nothing(self):
|
||||
self.offered.unlink()
|
||||
r = self.refresh()
|
||||
self.assertEqual(self.compose(), OLD)
|
||||
self.assertIn("vor 1.34", r.stdout)
|
||||
|
||||
def test_something_that_is_not_an_opnexus_compose_file_is_ignored(self):
|
||||
self.offered.write_text("hello: world\n")
|
||||
self.refresh()
|
||||
self.assertEqual(self.compose(), OLD)
|
||||
|
||||
def test_without_cosign_nothing_is_taken_from_an_image(self):
|
||||
r = self.refresh(no_cosign=True)
|
||||
self.assertEqual(self.compose(), OLD)
|
||||
self.assertIn("ohne cosign", r.stdout)
|
||||
self.assertFalse(any(c.startswith("run ") for c in (self.log.read_text().splitlines() if self.log.exists() else [])))
|
||||
|
||||
def test_a_bad_signature_stops_before_anything_changes(self):
|
||||
r = self.refresh(env_extra={"FAKE_COSIGN_RC": "1"})
|
||||
self.assertNotEqual(r.returncode, 0)
|
||||
self.assertEqual(self.compose(), OLD)
|
||||
|
||||
def test_rollback_puts_the_previous_file_and_checksum_back(self):
|
||||
(self.tmp / ".opnexus-compose-sha").write_text(_sha(OLD) + "\n")
|
||||
self.refresh()
|
||||
r = self.run_body(f'prev_compose=docker-compose.yml.bak-alt prev_compose_sha={_sha(OLD)} restore_compose')
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
self.assertEqual(self.compose(), OLD)
|
||||
self.assertEqual(self.recorded(), _sha(OLD))
|
||||
|
||||
def test_rollback_without_a_swapped_file_does_nothing(self):
|
||||
r = self.run_body('prev_compose="" restore_compose; echo DONE')
|
||||
self.assertIn("DONE", r.stdout)
|
||||
self.assertEqual(self.compose(), OLD)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -144,3 +144,60 @@ class RefreshCli(unittest.TestCase):
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class RefreshCliOnDisk(RefreshCli):
|
||||
"""B18: in the release bundle (CLI next to docker-compose.yml) the file on disk is replaced as well, so a later
|
||||
`rollback` runs the new CLI; a repo checkout (scripts/opnexus -> updater/opnexus) is never touched."""
|
||||
|
||||
def run_copy(self, *, with_compose: bool):
|
||||
bundle = self.tmp / "bundle"
|
||||
bundle.mkdir()
|
||||
copy = bundle / "opnexus"
|
||||
shutil.copy2(SCRIPT, copy)
|
||||
if with_compose:
|
||||
(bundle / "docker-compose.yml").write_text("services: {}\n")
|
||||
self.cli.write_text(NEW_CLI)
|
||||
env = {**os.environ, "PATH": f"{self.tmp / 'bin'}:{os.environ['PATH']}", "OPNEXUS_DIR": str(self.tmp),
|
||||
"FAKE_LOG": str(self.log), "FAKE_CLI": str(self.cli), "FAKE_DIGEST": DIGEST, "TMPDIR": str(self.tmp)}
|
||||
for name in ("OPNEXUS_NO_CLI_REFRESH", "CLI_REFRESHED"):
|
||||
env.pop(name, None)
|
||||
body = f'source <(sed "/^case \\"\\${{1:-}}\\" in/,\\$d" "{copy}"); refresh_cli 9.9.9; echo CONTINUED'
|
||||
result = subprocess.run(["bash", "-c", body, str(copy)], env=env, capture_output=True, text=True, timeout=30)
|
||||
return result, copy
|
||||
|
||||
def test_bundle_cli_on_disk_is_replaced_with_a_backup(self):
|
||||
r, copy = self.run_copy(with_compose=True)
|
||||
self.assertIn("NEW-CLI __refreshed update 9.9.9", r.stdout)
|
||||
self.assertEqual(copy.read_text(), NEW_CLI)
|
||||
self.assertEqual(Path(str(copy) + ".bak").read_text(), SCRIPT.read_text())
|
||||
self.assertTrue(os.access(copy, os.X_OK))
|
||||
|
||||
def test_a_cli_without_a_compose_file_next_to_it_is_left_alone(self):
|
||||
r, copy = self.run_copy(with_compose=False)
|
||||
self.assertIn("NEW-CLI __refreshed update 9.9.9", r.stdout)
|
||||
self.assertEqual(copy.read_text(), SCRIPT.read_text())
|
||||
self.assertFalse(Path(str(copy) + ".bak").exists())
|
||||
|
||||
|
||||
class RefreshedCliInstallsItself(unittest.TestCase):
|
||||
"""The first hop is started by an OLD CLI (no on-disk replacement in its refresh_cli): the new one, running from
|
||||
its temp copy with __refreshed, puts itself into the bundle so a later `rollback` runs the new code."""
|
||||
|
||||
def test_the_refreshed_cli_replaces_the_bundle_file(self):
|
||||
tmp = Path(tempfile.mkdtemp())
|
||||
self.addCleanup(shutil.rmtree, tmp, ignore_errors=True)
|
||||
bundle = tmp / "bundle"
|
||||
bundle.mkdir()
|
||||
(bundle / "opnexus").write_text("#!/bin/bash\necho old\n")
|
||||
(bundle / "docker-compose.yml").write_text("services: {}\n")
|
||||
(bundle / ".env").write_text("OPNEXUS_IMAGE_REPO=registry.example/opnexus\n") # a real install always has one
|
||||
running = tmp / "opnexus-cli.abc123"
|
||||
shutil.copy2(SCRIPT, running)
|
||||
env = {**os.environ, "OPNEXUS_DIR": str(bundle)}
|
||||
r = subprocess.run(["bash", str(running), "__refreshed", "no-such-command"], env=env, capture_output=True,
|
||||
text=True, timeout=30)
|
||||
self.assertEqual((bundle / "opnexus").read_text(), SCRIPT.read_text())
|
||||
self.assertEqual((bundle / "opnexus.bak").read_text(), "#!/bin/bash\necho old\n")
|
||||
self.assertFalse(running.exists()) # the temp copy is cleaned up as before
|
||||
self.assertIn("CLI auf der Platte aktualisiert", r.stdout)
|
||||
Reference in new issue
Block a user