Homelab Overseer is an autonomous monitoring and operations service for Patrick's homelab. Deterministic checks run continuously; GPT-5.6 Sol interprets the structured evidence, correlates failures, and applies the smallest reversible response permitted by strong operator directives.

Scheduled Codex runs are explicitly pinned to gpt-5.6-sol with medium reasoning and unrestricted local execution. The operator directives require target resolution, diagnostics before changes, narrow reversible actions, and application-level recovery verification. Credentials, log-injected commands, irreversible deletion, security-boundary changes, destructive storage work, and host reboots retain explicit safeguards.

Architecture

ChatGPT Work / Codex
        |
        | bearer-authenticated API on the allowlisted LAN address
        v
Homelab Overseer on Dozor
  - local, required control-plane self-monitoring
  - continuous collector
  - dependency assessment
  - state/history and incident reports
  - allowlisted restricted API
        |
        | dedicated SSH identity + narrow sudo wrappers / bounded probes
        v
Almaz / Nuc             Volga              Kvant
required server checks  required probes    opportunistic workstation probes

Capabilities

Host reachability, load, memory, disk, systemd, and unhealthy-container checks
Explicit host collection and availability profiles: Dozor is monitored locally as the required control plane, while Kvant remains visible without turning normal workstation downtime or interactive load into a fleet incident
Automatic liveness and Docker-health inventory for every container on Almaz and Nuc, including stopped and newly discovered containers
Direct in-container HTTP checks for media applications, including Lidarr, without exposing application ports or credentials
Docker status and bounded recent logs
First-class Calibre-Web health, endpoint, certificate, log, and guarded recovery coverage
TCP, DNS, HTTP(S), and certificate checks
Backup freshness from an explicit success marker
Dependency-aware severity and shared-failure correlation
Atomic latest state plus bounded historical snapshots
Deduplicated incident reports and recurring occurrence counts
A protected Agent Activity console with final reports and run history
OpenTelemetry investigation spans for Arize Phoenix
Independent Prometheus textfile metrics for collector freshness, channel readiness, scheduler freshness, and agent outcomes
Lease-refreshed failure and exact-identity recovery alerts routed through Alertmanager to ntfy, with bounded maintenance silence controls
Safe local Markdown rendering, producer filtering, and allowlisted Alertmanager actions in SwayNC
Mode-0600 redacted diagnostic bundles
Authenticated JSON API on the exact policy-allowlisted LAN address
Shared typed /v1/operator-summary 1.0 envelope for read-only integrations
Private, state-grounded operator chat through the allowlisted Switchyard model ingress
Native read-first MCP tools for direct Codex access
Separate mode-0600 read, chat, and action API credentials
Audited service restart with before/after diagnostics
One-attempt policy-controlled remediation for low-blast-radius services
Silent 30-minute GPT-5.6 Sol heartbeat with attention-only reports
Daily 7/30-day risk outlook, weekly drift/update review, and monthly resilience audit
Hardened systemd and remote sudoers templates
Repository-local $homelab-health Codex skill
Responsive visual dashboard with health trends, topology, deadlines, logs, incidents, and controlled actions

Development

Python 3.11+ and PyYAML are required.

python3 -m venv .venv
.venv/bin/pip install -e .
.venv/bin/python -m pytest
./tests/smoke

Validate configuration and inspect the CLI:

python3 -m overseer.cli --root . validate
python3 -m overseer.cli --root . --help

The compatibility commands remain available:

./scripts/collect-health
./scripts/inspect-host almaz
./scripts/inspect-service caddy --logs 200

Do not run live collection until the restricted remote wrappers and dedicated SSH identity are configured.

Runtime files

state/latest.json — latest complete snapshot
state/history/ — bounded historical snapshots
state/active-incident.json — active/resolved incident fingerprint
state/notification-state.json — versioned alert identity outbox and bounded readiness/submission status
state/actions.jsonl — audited state-changing actions
state/bundles/ — redacted diagnostic archives
reports/ — evidence-backed incident reports

Runtime state, reports, tokens, caches, and bundles are ignored by Git.

Deployment

Read docs/DEPLOYMENT.md before applying privileged changes. The installers require an explicit --apply; the local installer leaves services disabled unless --enable is also supplied. Remote installers do not add SSH keys or restart workloads. Installers reject dirty checkouts and record the exact release SHA so the live supervisor and remote wrappers can be qualified and rolled back to one revision.

API details are in docs/API.md. The scheduled supervisor prompt is in deploy/work-task-prompt.md.

Enable the proactive Codex program: degraded-state investigation and remediation, a silent heartbeat, the daily operations brief, weekly drift review, and monthly resilience audit:

./deploy/codex/install-schedule --apply

Native MCP setup and tool contracts are documented in docs/MCP.md.

Dashboard

After deployment, open:

https://overseer.subcult.tv/dashboard

The dashboard links to the existing authenticated observability surfaces: Grafana for metrics and Loki logs, Phoenix for agent traces, Prometheus for raw queries and targets, Dozzle for live container logs, Uptime Kuma for availability, and Alertmanager for routing and silences.

For dashboard status plus Ask Watcher chat, copy the chat token into the local clipboard without displaying it:

sudo cat /opt/homelab-overseer/state/chat-api-token | wl-copy

The chat token can read dashboard data and ask the local model questions, but it cannot run checks, silence alerts, create bundles, or restart services. To use the dashboard without chat, copy the read-only token instead:

sudo cat /opt/homelab-overseer/state/read-api-token | wl-copy

Paste it into the dashboard authorization screen. The key is held only in tab-scoped session storage and is discarded when the browser tab closes. If the dashboard is already open with the read token, Ask Watcher also accepts a separate chat key in its panel. Reserve the action token for approved action clients.

Authority and directives

The project directives in AGENTS.md are authoritative. The scheduled Codex operator may investigate and perform reversible low-to-medium blast-radius repairs without pausing. Explicit current human approval remains required for irreversible deletion, credential changes, weakened security controls, firewall/routing/access-boundary changes, destructive storage or backup work, host reboots, and actions with uncertain targets or rollback.

The intended Dozor supervisor state is declared in config/policy.yaml under policy.operator_baseline: always awake, no screen blanking or lock, no suspend/hibernate, and GDM automatic login for chat-gpt while preserving the account password for sudo, SSH, and recovery.

License

Licensed under GPL-3.0-or-later. See LICENSE.