vod-tender

Small Go service that discovers Twitch VODs for a channel, downloads them with yt-dlp, records live chat tied to VODs, and optionally uploads to YouTube. It ships with a minimal API and a small frontend for browsing VODs and chat replay.

⚠️ Legal Notice & Responsible Use

vod-tender is intended for streamers archiving their own content, or operators with explicit permission from the content owner.

What this project is for:

• Archiving your own public Twitch VODs
• Recording chat for channels you own/moderate and are authorized to archive
• Uploading content you own (or are authorized to manage) to YouTube

What this project is not for:

• Downloading or redistributing content you do not own or have rights to
• Circumventing subscriber-only or other restricted-access content controls
• Re-uploading others' content to YouTube without permission

You are responsible for ensuring your use complies with Twitch and YouTube terms, local law, and applicable copyright requirements. The maintainers are not responsible for misuse.

Quick start

Complete Local Development Setup (Recommended for New Developers)

Get a fully functional environment with sample data in 5 minutes:

# 1. Clone and enter directory
git clone https://github.com/subculture-collective/vod-tender.git
cd vod-tender

# 2. Copy environment file
cp backend/.env.example backend/.env

# 3. Start services and load sample data
make dev-setup

That's it! You now have:

• PostgreSQL with 7 sample VODs and chat messages
• Backend API on http://localhost:8080
• Frontend on http://localhost:3000
• Jaeger tracing on http://localhost:16686

📖 New to the project? See the Complete Local Development Guide for a detailed walkthrough.

Using Docker Compose

# Start all services (Postgres, API, Frontend)
make up

# Load sample data for development
make db-seed

# View logs
make logs

# Stop services
make down

For Go/Node.js Development

Prerequisites:

• Go 1.21+
• Node.js 18+
• golangci-lint (for linting)

Setup:

# Install frontend dependencies
cd frontend && npm ci && cd ..

# Copy environment file and fill in credentials
cp backend/.env.example backend/.env

Common tasks:

# Build everything (backend + frontend)
make build

# Run tests (backend + frontend)
make test

# Run linters (backend + frontend)
make lint

# Auto-fix linting issues
make lint-fix

Component-specific commands:

# Backend only
make build-backend
make test-backend
make lint-backend

# Frontend only
make build-frontend
make test-frontend
make lint-frontend

Development

Quick Reference

All common development tasks can be run from the repository root using make:

Command	Description
make build	Build backend and frontend
make test	Run all tests
make lint	Run all linters
make lint-fix	Auto-fix linting issues
make up	Start Docker Compose stack
make logs	View logs from all services
make help	Show all available targets

Linting

The project uses golangci-lint for Go code and ESLint + Prettier for frontend code.

Installation:

# macOS
brew install golangci-lint

# Linux
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin

# Windows
# See https://golangci-lint.run/welcome/install/

Frontend dependencies are installed via npm ci in the frontend directory.

Usage:

make lint          # Run all linters (backend + frontend)
make lint-fix      # Auto-fix issues where possible

# Component-specific
make lint-backend
make lint-frontend

The linter configuration is in .golangci.yml for backend and eslint.config.js for frontend.

Docker Compose (server)

Project ships a docker-compose.yml with:

• Postgres (persistent volume)
• API (Go backend with yt-dlp + ffmpeg)
• Frontend (Vite build served by nginx)
• Backup service (daily pg_dump to a backup volume)
• Jaeger (distributed tracing UI and collector)

Basic ops:

# Ensure shared web network exists (used by Caddy too)
docker network create web 2>/dev/null || true

# Build & start
docker compose up -d --build

# Status
docker compose ps

# Tail logs
docker logs -f vod-api

# Access Jaeger UI (tracing)
open http://localhost:16686

Observability

vod-tender includes production-ready observability features:

• Distributed Tracing (OpenTelemetry + Jaeger) - Trace VOD processing pipelines end-to-end
• Enhanced Metrics (Prometheus) - 20+ metrics including step-level durations, chat, OAuth, API calls
• Alert Rules (Prometheus Alertmanager) - 8 production alerts for critical failure scenarios
• Grafana Dashboard - 10-panel dashboard with queue depth, circuit breaker, performance metrics
• Performance Profiling (pprof) - CPU, memory, and goroutine profiling
• Health Endpoints - /healthz (liveness) and /readyz (readiness with detailed checks)

See docs/OBSERVABILITY.md for complete documentation including:

• Setup instructions and configuration
• Metrics reference with descriptions
• Alert rules and thresholds
• Dashboard usage guide
• Profiling procedures
• Troubleshooting guides

Quick links:

• Jaeger UI: http://localhost:16686 (traces)
• Prometheus metrics: http://localhost:8080/metrics
• Readiness check: http://localhost:8080/readyz

Configuration

Environment variables (place in backend/.env for local dev):

• TWITCH_CHANNEL
• TWITCH_BOT_USERNAME
• TWITCH_OAUTH_TOKEN
• TWITCH_CLIENT_ID (for Helix API discovery)
• TWITCH_CLIENT_SECRET (for Helix API discovery)
• TWITCH_VOD_ID (optional; default demo-vod-id)
• TWITCH_VOD_START (RFC3339; optional; default now)
• DB_DSN (optional; default postgres://vod:vod@postgres:5432/vod?sslmode=disable)
• DATA_DIR (optional; default data)
• LOG_LEVEL (optional; default info)
• LOG_FORMAT (optional; text|json; default text)

Chat recorder starts only when Twitch creds are present. Auto mode can start the recorder when the channel goes live (see CHAT_AUTO_START).

Full configuration reference and operational guidance:

• Local Development: docs/LOCAL_DEV_GUIDE.md - Complete setup guide with sample data
• Architecture: docs/ARCHITECTURE.md
• Configuration: docs/CONFIG.md
• Operations / Runbook: docs/OPERATIONS.md
• Logging / Log Aggregation: docs/LOGGING.md
• Observability: docs/OBSERVABILITY.md
• CI/CD Pipeline: docs/CICD.md
• CI/CD Quick Reference: docs/CICD-QUICK-REFERENCE.md
• Changelog: CHANGELOG.md - Release history and notable changes

End-to-end overview

1. Catalog backfill periodically discovers historical VODs using Twitch Helix and fills the vods table.
2. Processing job loops: selects next unprocessed VOD, downloads via yt-dlp (resumable), and uploads to YouTube if configured. Progress is written to DB and exposed via the API.
3. Chat recorder can run in two modes:

• Manual: provide TWITCH_VOD_ID and TWITCH_VOD_START and it will record chat for that VOD.
• Auto: set CHAT_AUTO_START=1 and it will detect when the channel goes live, record under a placeholder id, and reconcile to the real VOD after it’s published.

See docs/ARCHITECTURE.md for a deeper flow description and diagrams.

Configuration

All environment variables are documented in docs/CONFIG.md with defaults and tips. Key ones for a minimal run:

• TWITCH_CHANNEL, TWITCH_BOT_USERNAME, TWITCH_OAUTH_TOKEN (or store in DB via OAuth endpoints)
• TWITCH_CLIENT_ID, TWITCH_CLIENT_SECRET (Helix discovery / auto chat)
• DB_DSN (Postgres)
• DATA_DIR (download storage)
• LOG_LEVEL, LOG_FORMAT

OAuth and tokens

• Twitch Chat: Provide TWITCH_OAUTH_TOKEN (format oauth:xxxxx) or obtain/store via /auth/twitch/start → /auth/twitch/callback endpoints. The token is saved to oauth_tokens with expiry and refreshed automatically.
• YouTube Upload: Provide YT_CLIENT_ID, YT_CLIENT_SECRET, YT_REDIRECT_URI, then visit /auth/youtube/start to authorize. The refresh token is stored and the uploader uses it automatically.

Deployment

Docker Compose (bundled) is the simplest path for self-hosting:

• Postgres with a persistent volume
• API (Go backend) with yt-dlp and ffmpeg in the image
• Frontend built with Vite and served by nginx
• Optional daily backups via pg_dump

Routing: example Caddyfile assumes two hostnames, mapping to the shared web network. Ensure your reverse proxy attaches to the web network created by docker network create web.

For Kubernetes, map the same containers to Deployments and expose /metrics to Prometheus.

• Database: Postgres by default (see DB_DSN). Local docker-compose supplies a postgres service; override with your own DSN if needed.

• VOD processing job runs periodically (see configuration); discovery uses Twitch Helix if client id/secret provided.

• Downloader requires yt-dlp available in PATH.

  • Resumable downloads are enabled (yt-dlp --continue with infinite retries and fragment retries).

  • Optional: install aria2c for faster and more robust downloads.

  • ffmpeg is recommended for muxing and may be required by yt-dlp.

    Backups

  • Automatic: vod-backup runs pg_dump daily into volume pgbackups.

  • Manual one-off:

    docker compose run --rm backup sh -lc '/scripts/backup.sh /backups'
    

  • Copy backups to host:

    docker run --rm -v vod-tender_pgbackups:/src -v "$PWD":/dst alpine sh -lc 'cp -av /src/* /dst/'
    

  • Restore into running Postgres:

    zcat /path/to/vod_YYYYMMDD_HHMMSS.sql.gz | docker exec -i vod-postgres psql -U vod -d vod
    

    Caddy routing

    Routes assumed by the compose and Caddyfile:

  • Frontend: https://vod-tender.onnwee.me → vod-frontend:80

  • API: https://vod-api.onnwee.me → vod-api:8080

    Ensure caddy container is attached to the shared web network.

YouTube upload configuration

Uploads are disabled by default. To enable YouTube uploads, set both:

• YOUTUBE_UPLOAD_ENABLED=1
• YOUTUBE_UPLOAD_OWNERSHIP=self or YOUTUBE_UPLOAD_OWNERSHIP=authorized

If uploads are enabled without a valid ownership declaration, upload is skipped.

Set either STAR_FILE with a JSON file path or STAR_JSON with the JSON string for credentials and token (replace STAR with YT_CREDENTIALS or YT_TOKEN):

• YT_CREDENTIALS_FILE or YT_CREDENTIALS_JSON (Google OAuth client secrets JSON)
• YT_TOKEN_FILE or YT_TOKEN_JSON (OAuth token JSON containing a refresh token)

The token must include scope:

https://www.googleapis.com/auth/youtube.upload

Tip: Use Google OAuth 2.0 Playground to authorize the scope above and exchange an authorization code for a refresh token; save the resulting token JSON for YT_TOKEN_*.

API and frontend client

OpenAPI spec lives at backend/api/openapi.yaml.

Generate a TypeScript client (example using openapi-typescript):

npx openapi-typescript backend/api/openapi.yaml -o web/src/api/types.ts

Simple CORS is enabled for dev (Access-Control-Allow-Origin: *). For production, tighten CORS or place API behind your reverse proxy with appropriate headers.

Monitoring

• Health: GET /healthz (200 OK or 503)
• Status: GET /status (queue counts, circuit state, moving averages, last processing run)
• Metrics: GET /metrics (Prometheus format: download/upload counters, durations, queue depth, circuit gauge)
• Logs: default human text; switch to JSON with LOG_FORMAT=json for ingestion.

Troubleshooting quick hits

• Chat not recording: ensure TWITCH_BOT_USERNAME matches token owner, token has chat:read chat:edit, and is prefixed with oauth:. Auto mode requires valid Helix app credentials.
• Downloads failing with auth-required: subscriber-only/restricted VODs are intentionally not downloaded and are marked as skipped.
• Circuit open and processing paused: check CIRCUIT_FAILURE_THRESHOLD, investigate root error in logs, and clear with DELETE FROM kv WHERE key IN ('circuit_*'); if necessary.

Security notes

• OAuth tokens are stored in plaintext in Postgres for convenience. For production, consider encrypting at rest or using a dedicated secret store.
• YTDLP_VERBOSE=1 enables verbose downloader output; keep it off in normal operation to reduce log noise.
• Container images are automatically scanned for vulnerabilities using Trivy in CI. Builds fail on CRITICAL/HIGH severity issues. Scan reports are available as CI artifacts.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on:

• Setting up your development environment
• Code style and conventions
• Testing requirements
• Commit message format (Conventional Commits)
• Changelog maintenance (automated via semantic-release)
• Submitting pull requests
• Reporting bugs and requesting features

By participating in this project, you agree to abide by our Code of Conduct.

For release history and notable changes, see CHANGELOG.md.

Feature ideas

• indexed chat
• on/off switches
• auto update cookie