Implement PgBouncer connection pooling with monitoring and graceful shutdown (#128)
* Initial plan * feat: implement PgBouncer connection pooling and monitoring - Add PgBouncer service with transaction mode pooling - Configure optimal pool sizes (25 default, 100 max clients) - Update docker-compose for dev and prod environments - Enhance Prisma client with connection metrics - Improve Redis connection lifecycle management - Add comprehensive connection pool monitoring utilities - Create monitoring endpoints for health checks and metrics - Add graceful shutdown handlers for all services - Update environment configuration with PgBouncer settings Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com> * feat: add comprehensive documentation and tests for connection pooling - Create comprehensive CONNECTION_POOLING.md documentation - Add PgBouncer setup guide in docs/ - Create unit tests for connection pool monitoring - Add integration tests for monitoring endpoints - Update .gitignore for pgbouncer credentials Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com> * docs: finalize connection pooling documentation and README updates - Add comprehensive pgbouncer/README.md with usage guide - Update main README with connection pooling section - Add references to all relevant documentation - Complete documentation checklist Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com> * fix: address code review feedback on connection pooling - Fix PgBouncer Dockerfile permissions by running entrypoint as root with su-exec - Add PGPASSWORD to health check for proper authentication - Fix isHealthy check to not treat 0% utilization as unhealthy - Use proper URL parsing for isPgBouncer detection instead of string matching - Remove duplicate SIGTERM/SIGINT handlers in redis.ts, coordinate through db.ts - Add su-exec package for secure user switching in container Co-authored-by: PatrickFanella <61631520+PatrickFanella@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com> Co-authored-by: PatrickFanella <61631520+PatrickFanella@users.noreply.github.com>
This commit was merged in pull request #128.
This commit is contained in:
@@ -17,11 +17,20 @@
|
||||
# -----------------------------------------------------------------------------
|
||||
# PostgreSQL connection string
|
||||
# Format: postgresql://username:password@host:port/database?schema=public
|
||||
#
|
||||
# For production with PgBouncer (recommended):
|
||||
# DATABASE_URL=postgresql://postgres:password@pgbouncer:6432/spywatcher?pgbouncer=true
|
||||
#
|
||||
# For direct connection (development/migrations):
|
||||
# DATABASE_URL=postgresql://postgres:password@localhost:5432/spywatcher?schema=public
|
||||
DATABASE_URL=postgresql://postgres:password@localhost:5432/spywatcher?schema=public
|
||||
|
||||
# Database password (if using separate credential management)
|
||||
DB_PASSWORD=your_secure_database_password
|
||||
|
||||
# PgBouncer admin password (optional, for monitoring)
|
||||
PGBOUNCER_ADMIN_PASSWORD=your_secure_pgbouncer_admin_password
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Backend Environment Variables
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
596
CONNECTION_POOLING.md
Normal file
596
CONNECTION_POOLING.md
Normal file
@@ -0,0 +1,596 @@
|
||||
# Connection Pooling & Resource Management
|
||||
|
||||
This document describes the connection pooling and database resource management implementation for Discord Spywatcher.
|
||||
|
||||
## 📋 Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Architecture](#architecture)
|
||||
- [PgBouncer Configuration](#pgbouncer-configuration)
|
||||
- [Prisma Connection Pool](#prisma-connection-pool)
|
||||
- [Connection Lifecycle](#connection-lifecycle)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Best Practices](#best-practices)
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
The application implements a multi-layered connection pooling strategy:
|
||||
|
||||
1. **PgBouncer** - External connection pooler for PostgreSQL
|
||||
2. **Prisma Client** - Application-level connection management
|
||||
3. **Redis** - Connection pooling for cache/rate limiting
|
||||
|
||||
### Key Features
|
||||
|
||||
- ✅ Transaction-mode connection pooling via PgBouncer
|
||||
- ✅ Optimized Prisma connection pool settings
|
||||
- ✅ Graceful shutdown with proper cleanup
|
||||
- ✅ Connection pool monitoring and metrics
|
||||
- ✅ Health checks for database and Redis
|
||||
- ✅ Automatic connection leak prevention
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
Application Layer (Multiple Instances)
|
||||
↓
|
||||
Prisma Client (1-5 connections each)
|
||||
↓
|
||||
PgBouncer (Connection Pooler)
|
||||
- Pool Size: 25 connections
|
||||
- Mode: Transaction
|
||||
- Max Clients: 100
|
||||
↓
|
||||
PostgreSQL Database
|
||||
- Max Connections: 100
|
||||
```
|
||||
|
||||
### Why This Architecture?
|
||||
|
||||
- **PgBouncer** manages a pool of persistent connections to PostgreSQL
|
||||
- **Transaction mode** allows multiple clients to share connections between transactions
|
||||
- **Prisma** uses fewer connections since PgBouncer handles pooling
|
||||
- **Scalable** - can run multiple application instances without exhausting connections
|
||||
|
||||
## 🔧 PgBouncer Configuration
|
||||
|
||||
### Configuration File
|
||||
|
||||
Location: `pgbouncer/pgbouncer.ini`
|
||||
|
||||
#### Key Settings
|
||||
|
||||
```ini
|
||||
# Pooling mode - transaction is optimal for Prisma
|
||||
pool_mode = transaction
|
||||
|
||||
# Connection limits
|
||||
default_pool_size = 25 # Connections per database
|
||||
min_pool_size = 5 # Minimum connections to maintain
|
||||
reserve_pool_size = 5 # Additional connections for spikes
|
||||
max_client_conn = 100 # Maximum client connections
|
||||
|
||||
# Timeouts
|
||||
server_lifetime = 3600 # Connection lifetime (1 hour)
|
||||
server_idle_timeout = 600 # Idle timeout (10 minutes)
|
||||
query_wait_timeout = 120 # Query wait timeout (2 minutes)
|
||||
|
||||
# Reset query to clean connection state
|
||||
server_reset_query = DISCARD ALL
|
||||
```
|
||||
|
||||
### Pool Modes Explained
|
||||
|
||||
| Mode | Description | Use Case |
|
||||
|------|-------------|----------|
|
||||
| **session** | One server connection per client | Long-running sessions, advisory locks |
|
||||
| **transaction** | One server connection per transaction | Most applications (recommended for Prisma) |
|
||||
| **statement** | One server connection per statement | Stateless applications only |
|
||||
|
||||
**We use `transaction` mode** because:
|
||||
- Compatible with Prisma's transaction handling
|
||||
- Efficient connection reuse
|
||||
- Balances performance and compatibility
|
||||
|
||||
### Docker Setup
|
||||
|
||||
#### Development
|
||||
|
||||
```yaml
|
||||
pgbouncer:
|
||||
build:
|
||||
context: ./pgbouncer
|
||||
environment:
|
||||
DB_USER: spywatcher
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
ports:
|
||||
- "6432:6432"
|
||||
```
|
||||
|
||||
#### Production
|
||||
|
||||
```yaml
|
||||
pgbouncer:
|
||||
build:
|
||||
context: ./pgbouncer
|
||||
environment:
|
||||
DB_USER: spywatcher
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
restart: unless-stopped
|
||||
# Note: No external port exposure in production
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Application connects through PgBouncer
|
||||
DATABASE_URL=postgresql://user:password@pgbouncer:6432/spywatcher?pgbouncer=true
|
||||
|
||||
# Migrations connect directly to PostgreSQL
|
||||
DATABASE_URL_DIRECT=postgresql://user:password@postgres:5432/spywatcher
|
||||
|
||||
# PgBouncer admin credentials (optional)
|
||||
PGBOUNCER_ADMIN_PASSWORD=secure_password
|
||||
```
|
||||
|
||||
## 💎 Prisma Connection Pool
|
||||
|
||||
### Configuration
|
||||
|
||||
When using PgBouncer, Prisma needs fewer connections:
|
||||
|
||||
```typescript
|
||||
const db = new PrismaClient({
|
||||
datasources: {
|
||||
db: {
|
||||
url: process.env.DATABASE_URL,
|
||||
},
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Connection URL Parameters
|
||||
|
||||
#### With PgBouncer (Production)
|
||||
```
|
||||
postgresql://user:password@pgbouncer:6432/dbname?pgbouncer=true
|
||||
```
|
||||
- Keep connection pool small (Prisma default: 5)
|
||||
- PgBouncer handles the actual pooling
|
||||
|
||||
#### Direct Connection (Development/Migrations)
|
||||
```
|
||||
postgresql://user:password@postgres:5432/dbname?connection_limit=20&pool_timeout=20
|
||||
```
|
||||
- `connection_limit`: 10-50 depending on load
|
||||
- `pool_timeout`: 20 seconds
|
||||
- `connect_timeout`: 10 seconds
|
||||
|
||||
### Why Fewer Connections with PgBouncer?
|
||||
|
||||
Without PgBouncer:
|
||||
```
|
||||
Application → PostgreSQL (need many connections)
|
||||
```
|
||||
|
||||
With PgBouncer:
|
||||
```
|
||||
Application → PgBouncer → PostgreSQL (PgBouncer reuses connections)
|
||||
```
|
||||
|
||||
Example with 10 application instances:
|
||||
- **Without PgBouncer**: 10 × 20 = 200 PostgreSQL connections needed
|
||||
- **With PgBouncer**: 10 × 5 = 50 client connections → 25 PostgreSQL connections
|
||||
|
||||
## 🔄 Connection Lifecycle
|
||||
|
||||
### Application Startup
|
||||
|
||||
1. **Database Connection**
|
||||
```typescript
|
||||
// db.ts initializes Prisma Client
|
||||
export const db = new PrismaClient({ ... });
|
||||
```
|
||||
|
||||
2. **Redis Connection** (if enabled)
|
||||
```typescript
|
||||
// redis.ts initializes Redis client
|
||||
const redisClient = new Redis(url, { ... });
|
||||
```
|
||||
|
||||
3. **Health Checks**
|
||||
- Database connectivity verification
|
||||
- Connection pool metrics collection
|
||||
|
||||
### During Operation
|
||||
|
||||
- **Connection Reuse**: PgBouncer reuses connections between transactions
|
||||
- **Pool Monitoring**: Metrics collected every 60 seconds
|
||||
- **Auto-reconnect**: Redis automatically reconnects on connection loss
|
||||
|
||||
### Graceful Shutdown
|
||||
|
||||
```typescript
|
||||
// Signal handlers in db.ts and redis.ts
|
||||
process.on('SIGTERM', async () => {
|
||||
// 1. Stop accepting new connections
|
||||
// 2. Wait for in-flight requests
|
||||
// 3. Close Prisma connections
|
||||
await db.$disconnect();
|
||||
// 4. Close Redis connections
|
||||
await closeRedisConnection();
|
||||
// 5. Exit process
|
||||
process.exit(0);
|
||||
});
|
||||
```
|
||||
|
||||
#### Shutdown Sequence
|
||||
|
||||
1. **Receive SIGTERM/SIGINT**
|
||||
2. **Set shutdown flag** - prevents new operations
|
||||
3. **Disconnect Prisma** - closes all connections gracefully
|
||||
4. **Disconnect Redis** - uses QUIT command
|
||||
5. **Exit process**
|
||||
|
||||
### Connection Leak Prevention
|
||||
|
||||
- **Singleton pattern** for database client
|
||||
- **Proper error handling** ensures connections are released
|
||||
- **Transaction timeouts** prevent hung connections
|
||||
- **Monitoring alerts** for connection pool saturation
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### Health Check Endpoints
|
||||
|
||||
#### System Health
|
||||
```bash
|
||||
GET /api/admin/monitoring/connections/health
|
||||
```
|
||||
|
||||
Returns:
|
||||
```json
|
||||
{
|
||||
"healthy": true,
|
||||
"timestamp": "2025-01-15T10:30:00Z",
|
||||
"database": {
|
||||
"healthy": true,
|
||||
"responseTime": 12,
|
||||
"connectionPool": {
|
||||
"active": 3,
|
||||
"idle": 2,
|
||||
"total": 5,
|
||||
"max": 100,
|
||||
"utilizationPercent": "5.00",
|
||||
"isPgBouncer": true,
|
||||
"isShuttingDown": false
|
||||
}
|
||||
},
|
||||
"redis": {
|
||||
"available": true,
|
||||
"connected": true,
|
||||
"status": "ready"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Connection Pool Stats
|
||||
```bash
|
||||
GET /api/admin/monitoring/connections/pool
|
||||
```
|
||||
|
||||
Returns:
|
||||
```json
|
||||
{
|
||||
"database": {
|
||||
"utilizationPercent": 5.0,
|
||||
"activeConnections": 3,
|
||||
"maxConnections": 100,
|
||||
"isHealthy": true
|
||||
},
|
||||
"redis": {
|
||||
"available": true,
|
||||
"connected": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Connection Alerts
|
||||
```bash
|
||||
GET /api/admin/monitoring/connections/alerts
|
||||
```
|
||||
|
||||
Returns:
|
||||
```json
|
||||
{
|
||||
"alerts": [
|
||||
"WARNING: Database connection pool at 85% utilization"
|
||||
],
|
||||
"count": 1,
|
||||
"timestamp": "2025-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### PgBouncer Statistics
|
||||
|
||||
Connect to PgBouncer admin interface:
|
||||
|
||||
```bash
|
||||
psql -h localhost -p 6432 -U pgbouncer_admin pgbouncer
|
||||
```
|
||||
|
||||
Useful commands:
|
||||
|
||||
```sql
|
||||
-- Show pool statistics
|
||||
SHOW POOLS;
|
||||
|
||||
-- Show database statistics
|
||||
SHOW DATABASES;
|
||||
|
||||
-- Show client connections
|
||||
SHOW CLIENTS;
|
||||
|
||||
-- Show server connections
|
||||
SHOW SERVERS;
|
||||
|
||||
-- Show configuration
|
||||
SHOW CONFIG;
|
||||
|
||||
-- Show statistics
|
||||
SHOW STATS;
|
||||
```
|
||||
|
||||
### Automated Monitoring
|
||||
|
||||
The application logs connection pool metrics every 60 seconds:
|
||||
|
||||
```
|
||||
=== Connection Pool Metrics ===
|
||||
Timestamp: 2025-01-15T10:30:00Z
|
||||
Overall Health: ✅ HEALTHY
|
||||
|
||||
--- Database ---
|
||||
Health: ✅
|
||||
Response Time: 12ms
|
||||
Connection Pool:
|
||||
Active: 3
|
||||
Idle: 2
|
||||
Total: 5
|
||||
Max: 100
|
||||
Utilization: 5.00%
|
||||
PgBouncer: Yes
|
||||
|
||||
--- Redis ---
|
||||
Available: ✅
|
||||
Connected: ✅
|
||||
Status: ready
|
||||
Metrics:
|
||||
Total Connections: 125
|
||||
Total Commands: 45678
|
||||
Ops/sec: 23
|
||||
Memory Used: 2.5MB
|
||||
==============================
|
||||
```
|
||||
|
||||
## 🔍 Troubleshooting
|
||||
|
||||
### Issue: Too many connections
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: remaining connection slots are reserved for non-replication superuser connections
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check PgBouncer pool size:**
|
||||
```bash
|
||||
# In pgbouncer.ini
|
||||
default_pool_size = 25 # Increase if needed
|
||||
max_db_connections = 50
|
||||
```
|
||||
|
||||
2. **Check PostgreSQL max_connections:**
|
||||
```sql
|
||||
SHOW max_connections; -- Should be > PgBouncer pool size
|
||||
```
|
||||
|
||||
3. **Monitor connection usage:**
|
||||
```bash
|
||||
curl http://localhost:3001/api/admin/monitoring/connections/pool
|
||||
```
|
||||
|
||||
### Issue: Connection timeouts
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: Connection timeout
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check PgBouncer is running:**
|
||||
```bash
|
||||
docker ps | grep pgbouncer
|
||||
```
|
||||
|
||||
2. **Check connection string:**
|
||||
```bash
|
||||
# Ensure using correct host and port
|
||||
DATABASE_URL=postgresql://user:pass@pgbouncer:6432/db?pgbouncer=true
|
||||
```
|
||||
|
||||
3. **Increase timeouts:**
|
||||
```ini
|
||||
# In pgbouncer.ini
|
||||
query_wait_timeout = 120
|
||||
server_connect_timeout = 15
|
||||
```
|
||||
|
||||
### Issue: Slow queries with PgBouncer
|
||||
|
||||
**Symptoms:**
|
||||
- Queries slower than without PgBouncer
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Ensure using transaction mode:**
|
||||
```ini
|
||||
pool_mode = transaction # Not session mode
|
||||
```
|
||||
|
||||
2. **Check for connection reuse:**
|
||||
```sql
|
||||
-- In PgBouncer admin
|
||||
SHOW POOLS;
|
||||
-- Check cl_active, cl_waiting, sv_active, sv_idle
|
||||
```
|
||||
|
||||
3. **Monitor query wait time:**
|
||||
```bash
|
||||
curl http://localhost:3001/api/admin/monitoring/database/slow-queries
|
||||
```
|
||||
|
||||
### Issue: Migrations fail with PgBouncer
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: prepared statement already exists
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
Always run migrations with direct PostgreSQL connection:
|
||||
```bash
|
||||
# Use DATABASE_URL_DIRECT for migrations
|
||||
DATABASE_URL=$DATABASE_URL_DIRECT npx prisma migrate deploy
|
||||
```
|
||||
|
||||
Or configure in docker-compose.yml:
|
||||
```yaml
|
||||
migrate:
|
||||
environment:
|
||||
DATABASE_URL: postgresql://user:pass@postgres:5432/db # Direct connection
|
||||
```
|
||||
|
||||
### Issue: Connection pool exhaustion
|
||||
|
||||
**Symptoms:**
|
||||
- "Pool is full" errors
|
||||
- High connection utilization
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Scale PgBouncer pool:**
|
||||
```ini
|
||||
default_pool_size = 50 # Increase from 25
|
||||
reserve_pool_size = 10 # Increase reserve
|
||||
```
|
||||
|
||||
2. **Add connection cleanup:**
|
||||
```typescript
|
||||
// Ensure proper $disconnect() on errors
|
||||
try {
|
||||
await db.query();
|
||||
} finally {
|
||||
// Connections released automatically
|
||||
}
|
||||
```
|
||||
|
||||
3. **Reduce connection limit per instance:**
|
||||
```
|
||||
# Fewer connections per app instance
|
||||
DATABASE_URL=...?connection_limit=3
|
||||
```
|
||||
|
||||
## ✅ Best Practices
|
||||
|
||||
### Production Deployment
|
||||
|
||||
1. **Always use PgBouncer in production**
|
||||
- Better connection management
|
||||
- Prevents connection exhaustion
|
||||
- Enables horizontal scaling
|
||||
|
||||
2. **Configure appropriate pool sizes**
|
||||
```
|
||||
PgBouncer pool: 25-50 connections
|
||||
Prisma per instance: 3-5 connections
|
||||
PostgreSQL max: 100+ connections
|
||||
```
|
||||
|
||||
3. **Use separate connections for migrations**
|
||||
- Migrations need direct PostgreSQL access
|
||||
- Bypass PgBouncer for schema changes
|
||||
|
||||
4. **Monitor connection metrics**
|
||||
- Set up alerts for >80% utilization
|
||||
- Track connection pool trends
|
||||
- Monitor slow query counts
|
||||
|
||||
### Development Practices
|
||||
|
||||
1. **Test with and without PgBouncer**
|
||||
- Dev: direct connection (easier debugging)
|
||||
- Staging/Prod: through PgBouncer
|
||||
|
||||
2. **Use environment-specific configs**
|
||||
```bash
|
||||
# .env.development
|
||||
DATABASE_URL=postgresql://...@postgres:5432/db
|
||||
|
||||
# .env.production
|
||||
DATABASE_URL=postgresql://...@pgbouncer:6432/db?pgbouncer=true
|
||||
```
|
||||
|
||||
3. **Implement proper error handling**
|
||||
```typescript
|
||||
try {
|
||||
await db.query();
|
||||
} catch (error) {
|
||||
// Log error
|
||||
// Connection automatically released
|
||||
throw error;
|
||||
}
|
||||
```
|
||||
|
||||
4. **Use connection pooling metrics**
|
||||
- Monitor during load tests
|
||||
- Adjust pool sizes based on metrics
|
||||
- Set up automated alerts
|
||||
|
||||
### Security Considerations
|
||||
|
||||
1. **Secure PgBouncer credentials**
|
||||
- Use strong passwords
|
||||
- Rotate credentials regularly
|
||||
- Use environment variables
|
||||
|
||||
2. **Limit PgBouncer access**
|
||||
- Don't expose port externally
|
||||
- Use internal Docker network
|
||||
- Configure firewall rules
|
||||
|
||||
3. **Monitor for connection abuse**
|
||||
- Track connection patterns
|
||||
- Alert on unusual spikes
|
||||
- Implement rate limiting
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [PgBouncer Documentation](https://www.pgbouncer.org/)
|
||||
- [Prisma Connection Pool](https://www.prisma.io/docs/concepts/components/prisma-client/working-with-prismaclient/connection-pool)
|
||||
- [PostgreSQL Connection Pooling](https://www.postgresql.org/docs/current/runtime-config-connection.html)
|
||||
- [DATABASE_OPTIMIZATION.md](./DATABASE_OPTIMIZATION.md)
|
||||
- [POSTGRESQL.md](./POSTGRESQL.md)
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
For issues or questions:
|
||||
- Check monitoring endpoints first
|
||||
- Review logs for error messages
|
||||
- Consult troubleshooting section
|
||||
- Check PgBouncer statistics
|
||||
- Open GitHub issue with details
|
||||
23
README.md
23
README.md
@@ -208,13 +208,34 @@ Spywatcher implements comprehensive security measures to protect against abuse a
|
||||
- **IP blocking** with automatic abuse detection
|
||||
- **Load management** with circuit breakers and load shedding under high load
|
||||
|
||||
See [backend/docs/RATE_LIMITING.md](./backend/docs/RATE_LIMITING.md) for detailed documentation on:
|
||||
See [RATE_LIMITING.md](./RATE_LIMITING.md) for detailed documentation on:
|
||||
- Rate limiting configuration
|
||||
- Endpoint-specific limits
|
||||
- DDoS protection mechanisms
|
||||
- Load shedding behavior
|
||||
- Admin APIs for IP management
|
||||
|
||||
## 🗄️ Database & Connection Pooling
|
||||
|
||||
Spywatcher uses PostgreSQL with PgBouncer for efficient connection pooling and resource management:
|
||||
|
||||
- **PgBouncer** - Transaction-mode connection pooling for optimal performance
|
||||
- **Prisma** - Type-safe database access with connection pool monitoring
|
||||
- **Redis** - Distributed caching and rate limiting with connection management
|
||||
- **Graceful shutdown** - Proper connection cleanup on application shutdown
|
||||
|
||||
See [CONNECTION_POOLING.md](./CONNECTION_POOLING.md) for detailed documentation on:
|
||||
- PgBouncer setup and configuration
|
||||
- Connection pool monitoring and metrics
|
||||
- Database health checks and alerting
|
||||
- Performance optimization strategies
|
||||
- Troubleshooting connection issues
|
||||
|
||||
Additional database documentation:
|
||||
- [POSTGRESQL.md](./POSTGRESQL.md) - PostgreSQL setup and management
|
||||
- [DATABASE_OPTIMIZATION.md](./DATABASE_OPTIMIZATION.md) - Query optimization and indexing
|
||||
- [docs/PGBOUNCER_SETUP.md](./docs/PGBOUNCER_SETUP.md) - Quick reference guide
|
||||
|
||||
## 🌐 Endpoints
|
||||
|
||||
Available at `http://localhost:3001`
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* Integration tests for Connection Monitoring Routes
|
||||
*/
|
||||
|
||||
import express from 'express';
|
||||
import request from 'supertest';
|
||||
|
||||
import monitoringRoutes from '../../../src/routes/monitoring';
|
||||
|
||||
// Mock dependencies
|
||||
jest.mock('../../../src/db', () => ({
|
||||
db: {
|
||||
$queryRaw: jest.fn(),
|
||||
},
|
||||
checkDatabaseHealth: jest.fn(),
|
||||
getConnectionPoolMetrics: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('../../../src/utils/redis', () => ({
|
||||
getRedisClient: jest.fn(() => null),
|
||||
getRedisMetrics: jest.fn(),
|
||||
isRedisAvailable: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('../../../src/utils/connectionPoolMonitor', () => ({
|
||||
getSystemHealth: jest.fn(),
|
||||
getConnectionPoolStats: jest.fn(),
|
||||
getConnectionPoolAlerts: jest.fn(),
|
||||
}));
|
||||
|
||||
// Mock authentication middleware
|
||||
jest.mock('../../../src/middleware/auth', () => ({
|
||||
requireAuth: (req: any, _res: any, next: any) => {
|
||||
req.user = { id: 'test-admin', username: 'admin', role: 'ADMIN' };
|
||||
next();
|
||||
},
|
||||
requireRole: () => (_req: any, _res: any, next: any) => next(),
|
||||
}));
|
||||
|
||||
// Mock other dependencies
|
||||
jest.mock('../../../src/middleware/slowQueryLogger', () => ({
|
||||
getSlowQueryLogs: jest.fn(() => ({ logs: [], total: 0 })),
|
||||
getSlowQueryStats: jest.fn(() => ({ totalQueries: 0 })),
|
||||
clearSlowQueryLogs: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.mock('../../../src/services/cache', () => ({
|
||||
cache: {
|
||||
getStats: jest.fn(),
|
||||
flushAll: jest.fn(),
|
||||
invalidateByTag: jest.fn(),
|
||||
},
|
||||
}));
|
||||
|
||||
jest.mock('../../../src/utils/databaseMaintenance', () => ({
|
||||
checkDatabaseHealth: jest.fn(),
|
||||
getTableStats: jest.fn(),
|
||||
getIndexUsageStats: jest.fn(),
|
||||
getUnusedIndexes: jest.fn(),
|
||||
getSlowQueries: jest.fn(),
|
||||
generateMaintenanceReport: jest.fn(),
|
||||
analyzeAllTables: jest.fn(),
|
||||
}));
|
||||
|
||||
import * as db from '../../../src/db';
|
||||
import * as connectionPoolMonitor from '../../../src/utils/connectionPoolMonitor';
|
||||
|
||||
describe('Connection Monitoring Routes', () => {
|
||||
let app: express.Application;
|
||||
|
||||
beforeEach(() => {
|
||||
app = express();
|
||||
app.use(express.json());
|
||||
app.use('/api/admin/monitoring', monitoringRoutes);
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
describe('GET /connections/health', () => {
|
||||
it('should return system health status', async () => {
|
||||
const mockHealth = {
|
||||
healthy: true,
|
||||
timestamp: new Date().toISOString(),
|
||||
database: {
|
||||
healthy: true,
|
||||
responseTime: 12,
|
||||
connectionPool: {
|
||||
active: 3,
|
||||
idle: 2,
|
||||
total: 5,
|
||||
max: 100,
|
||||
utilizationPercent: '5.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
},
|
||||
},
|
||||
redis: {
|
||||
available: true,
|
||||
connected: true,
|
||||
status: 'ready',
|
||||
},
|
||||
};
|
||||
|
||||
jest.spyOn(connectionPoolMonitor, 'getSystemHealth').mockResolvedValue(
|
||||
mockHealth
|
||||
);
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/health'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.body.healthy).toBe(true);
|
||||
expect(response.body.database.healthy).toBe(true);
|
||||
expect(response.body.database.connectionPool.active).toBe(3);
|
||||
expect(response.body.redis.available).toBe(true);
|
||||
});
|
||||
|
||||
it('should handle errors gracefully', async () => {
|
||||
jest.spyOn(connectionPoolMonitor, 'getSystemHealth').mockRejectedValue(
|
||||
new Error('Database connection failed')
|
||||
);
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/health'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(500);
|
||||
expect(response.body.error).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('GET /connections/pool', () => {
|
||||
it('should return connection pool statistics', async () => {
|
||||
const mockStats = {
|
||||
database: {
|
||||
utilizationPercent: 15.0,
|
||||
activeConnections: 10,
|
||||
maxConnections: 100,
|
||||
isHealthy: true,
|
||||
},
|
||||
redis: {
|
||||
available: true,
|
||||
connected: true,
|
||||
},
|
||||
};
|
||||
|
||||
jest.spyOn(
|
||||
connectionPoolMonitor,
|
||||
'getConnectionPoolStats'
|
||||
).mockResolvedValue(mockStats);
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/pool'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.body.database.utilizationPercent).toBe(15.0);
|
||||
expect(response.body.database.activeConnections).toBe(10);
|
||||
expect(response.body.redis.available).toBe(true);
|
||||
});
|
||||
|
||||
it('should handle errors gracefully', async () => {
|
||||
jest.spyOn(
|
||||
connectionPoolMonitor,
|
||||
'getConnectionPoolStats'
|
||||
).mockRejectedValue(new Error('Failed to get stats'));
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/pool'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(500);
|
||||
expect(response.body.error).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('GET /connections/alerts', () => {
|
||||
it('should return connection pool alerts', async () => {
|
||||
const mockAlerts = [
|
||||
'WARNING: Database connection pool at 85% utilization',
|
||||
];
|
||||
|
||||
jest.spyOn(
|
||||
connectionPoolMonitor,
|
||||
'getConnectionPoolAlerts'
|
||||
).mockResolvedValue(mockAlerts);
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/alerts'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.body.alerts).toHaveLength(1);
|
||||
expect(response.body.alerts[0]).toContain('WARNING');
|
||||
expect(response.body.count).toBe(1);
|
||||
expect(response.body.timestamp).toBeDefined();
|
||||
});
|
||||
|
||||
it('should return empty array when no alerts', async () => {
|
||||
jest.spyOn(
|
||||
connectionPoolMonitor,
|
||||
'getConnectionPoolAlerts'
|
||||
).mockResolvedValue([]);
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/alerts'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.body.alerts).toHaveLength(0);
|
||||
expect(response.body.count).toBe(0);
|
||||
});
|
||||
|
||||
it('should handle errors gracefully', async () => {
|
||||
jest.spyOn(
|
||||
connectionPoolMonitor,
|
||||
'getConnectionPoolAlerts'
|
||||
).mockRejectedValue(new Error('Failed to get alerts'));
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/connections/alerts'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(500);
|
||||
expect(response.body.error).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Database health endpoint', () => {
|
||||
it('should return database health status', async () => {
|
||||
const mockHealth = {
|
||||
connected: true,
|
||||
version: '15.3',
|
||||
responseTime: 8,
|
||||
};
|
||||
|
||||
jest.spyOn(db, 'checkDatabaseHealth').mockResolvedValue(mockHealth);
|
||||
|
||||
const response = await request(app).get(
|
||||
'/api/admin/monitoring/database/health'
|
||||
);
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.body.connected).toBe(true);
|
||||
});
|
||||
});
|
||||
});
|
||||
290
backend/__tests__/unit/utils/connectionPoolMonitor.test.ts
Normal file
290
backend/__tests__/unit/utils/connectionPoolMonitor.test.ts
Normal file
@@ -0,0 +1,290 @@
|
||||
/**
|
||||
* Tests for Connection Pool Monitor
|
||||
*/
|
||||
|
||||
import { describe, it, expect, jest, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock dependencies
|
||||
jest.mock('../../../src/db');
|
||||
jest.mock('../../../src/utils/redis');
|
||||
|
||||
import * as db from '../../../src/db';
|
||||
import * as redis from '../../../src/utils/redis';
|
||||
import {
|
||||
getSystemHealth,
|
||||
getConnectionPoolStats,
|
||||
getConnectionPoolAlerts,
|
||||
isConnectionPoolOverloaded,
|
||||
} from '../../../src/utils/connectionPoolMonitor';
|
||||
|
||||
describe('Connection Pool Monitor', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
describe('getSystemHealth', () => {
|
||||
it('should return healthy status when all systems are operational', async () => {
|
||||
// Mock database health check
|
||||
jest.spyOn(db, 'checkDatabaseHealth').mockResolvedValue({
|
||||
healthy: true,
|
||||
responseTime: 15,
|
||||
});
|
||||
|
||||
// Mock connection pool metrics
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 3,
|
||||
idle: 2,
|
||||
total: 5,
|
||||
max: 100,
|
||||
utilizationPercent: '5.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
// Mock Redis metrics
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
status: 'ready',
|
||||
totalConnectionsReceived: '100',
|
||||
totalCommandsProcessed: '5000',
|
||||
instantaneousOpsPerSec: '25',
|
||||
usedMemory: '2MB',
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'isRedisAvailable').mockResolvedValue(true);
|
||||
|
||||
const health = await getSystemHealth();
|
||||
|
||||
expect(health.healthy).toBe(true);
|
||||
expect(health.database.healthy).toBe(true);
|
||||
expect(health.database.responseTime).toBe(15);
|
||||
expect(health.database.connectionPool?.active).toBe(3);
|
||||
expect(health.database.connectionPool?.isPgBouncer).toBe(true);
|
||||
expect(health.redis.available).toBe(true);
|
||||
expect(health.redis.connected).toBe(true);
|
||||
});
|
||||
|
||||
it('should return unhealthy status when database is down', async () => {
|
||||
jest.spyOn(db, 'checkDatabaseHealth').mockResolvedValue({
|
||||
healthy: false,
|
||||
error: 'Connection refused',
|
||||
});
|
||||
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 0,
|
||||
idle: 0,
|
||||
total: 0,
|
||||
max: 100,
|
||||
utilizationPercent: '0',
|
||||
isPgBouncer: false,
|
||||
isShuttingDown: false,
|
||||
error: 'Connection refused',
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'isRedisAvailable').mockResolvedValue(true);
|
||||
|
||||
const health = await getSystemHealth();
|
||||
|
||||
expect(health.healthy).toBe(false);
|
||||
expect(health.database.healthy).toBe(false);
|
||||
expect(health.database.error).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('getConnectionPoolStats', () => {
|
||||
it('should return connection pool statistics', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 10,
|
||||
idle: 5,
|
||||
total: 15,
|
||||
max: 100,
|
||||
utilizationPercent: '15.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
const stats = await getConnectionPoolStats();
|
||||
|
||||
expect(stats.database.utilizationPercent).toBe(15.0);
|
||||
expect(stats.database.activeConnections).toBe(10);
|
||||
expect(stats.database.maxConnections).toBe(100);
|
||||
expect(stats.database.isHealthy).toBe(true);
|
||||
expect(stats.redis.available).toBe(true);
|
||||
});
|
||||
|
||||
it('should handle database errors gracefully', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 0,
|
||||
idle: 0,
|
||||
total: 0,
|
||||
max: 0,
|
||||
utilizationPercent: '0',
|
||||
isPgBouncer: false,
|
||||
isShuttingDown: false,
|
||||
error: 'Database error',
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: false,
|
||||
connected: false,
|
||||
});
|
||||
|
||||
const stats = await getConnectionPoolStats();
|
||||
|
||||
expect(stats.database.isHealthy).toBe(false);
|
||||
expect(stats.redis.available).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isConnectionPoolOverloaded', () => {
|
||||
it('should return true when utilization exceeds threshold', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 85,
|
||||
idle: 5,
|
||||
total: 90,
|
||||
max: 100,
|
||||
utilizationPercent: '90.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
const isOverloaded = await isConnectionPoolOverloaded(80);
|
||||
|
||||
expect(isOverloaded).toBe(true);
|
||||
});
|
||||
|
||||
it('should return false when utilization is below threshold', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 50,
|
||||
idle: 20,
|
||||
total: 70,
|
||||
max: 100,
|
||||
utilizationPercent: '70.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
const isOverloaded = await isConnectionPoolOverloaded(80);
|
||||
|
||||
expect(isOverloaded).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getConnectionPoolAlerts', () => {
|
||||
it('should return critical alert when utilization >= 90%', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 90,
|
||||
idle: 5,
|
||||
total: 95,
|
||||
max: 100,
|
||||
utilizationPercent: '95.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
const alerts = await getConnectionPoolAlerts();
|
||||
|
||||
expect(alerts).toHaveLength(1);
|
||||
expect(alerts[0]).toContain('CRITICAL');
|
||||
expect(alerts[0]).toContain('95');
|
||||
});
|
||||
|
||||
it('should return warning alert when utilization >= 80%', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 82,
|
||||
idle: 8,
|
||||
total: 85,
|
||||
max: 100,
|
||||
utilizationPercent: '85.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
const alerts = await getConnectionPoolAlerts();
|
||||
|
||||
expect(alerts).toHaveLength(1);
|
||||
expect(alerts[0]).toContain('WARNING');
|
||||
expect(alerts[0]).toContain('85');
|
||||
});
|
||||
|
||||
it('should return no alerts when utilization is healthy', async () => {
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 20,
|
||||
idle: 10,
|
||||
total: 30,
|
||||
max: 100,
|
||||
utilizationPercent: '30.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: true,
|
||||
connected: true,
|
||||
});
|
||||
|
||||
const alerts = await getConnectionPoolAlerts();
|
||||
|
||||
expect(alerts).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('should include Redis alerts when configured but unavailable', async () => {
|
||||
// Set Redis URL environment variable
|
||||
process.env.REDIS_URL = 'redis://localhost:6379';
|
||||
|
||||
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
|
||||
active: 10,
|
||||
idle: 5,
|
||||
total: 15,
|
||||
max: 100,
|
||||
utilizationPercent: '15.00',
|
||||
isPgBouncer: true,
|
||||
isShuttingDown: false,
|
||||
});
|
||||
|
||||
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
|
||||
available: false,
|
||||
connected: false,
|
||||
});
|
||||
|
||||
const alerts = await getConnectionPoolAlerts();
|
||||
|
||||
const redisAlerts = alerts.filter((a) => a.includes('Redis'));
|
||||
expect(redisAlerts.length).toBeGreaterThan(0);
|
||||
|
||||
// Clean up
|
||||
delete process.env.REDIS_URL;
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -3,13 +3,31 @@ import { PrismaClient } from '@prisma/client';
|
||||
// Global singleton for Prisma Client to prevent multiple instances
|
||||
const globalForPrisma = global as unknown as { prisma: PrismaClient };
|
||||
|
||||
// Determine if using PgBouncer based on connection string
|
||||
const isPgBouncer = (() => {
|
||||
try {
|
||||
const url = new URL(process.env.DATABASE_URL ?? '');
|
||||
return url.searchParams.get('pgbouncer') === 'true';
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})();
|
||||
|
||||
// Configure connection pooling and logging based on environment
|
||||
// When using PgBouncer (transaction pooling mode):
|
||||
// - Keep connection_limit low (1-5) since PgBouncer handles pooling
|
||||
// - Disable interactive transactions for better compatibility
|
||||
export const db =
|
||||
globalForPrisma.prisma ||
|
||||
new PrismaClient({
|
||||
log: process.env.NODE_ENV === 'development'
|
||||
? ['query', 'error', 'warn']
|
||||
: ['error'],
|
||||
datasources: {
|
||||
db: {
|
||||
url: process.env.DATABASE_URL,
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
// Prevent multiple instances in development (hot reload)
|
||||
@@ -21,14 +39,112 @@ if (process.env.NODE_ENV !== 'production') {
|
||||
// This must be done after the db instance is created but before queries run
|
||||
// The initialization is done in the application entry point (server.ts or index.ts)
|
||||
|
||||
// Connection pool status tracking
|
||||
let isShuttingDown = false;
|
||||
|
||||
/**
|
||||
* Get connection pool metrics
|
||||
*/
|
||||
export async function getConnectionPoolMetrics() {
|
||||
try {
|
||||
// Query for active connections
|
||||
const activeConnections = await db.$queryRaw<Array<{ count: bigint }>>`
|
||||
SELECT COUNT(*) as count
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = current_database()
|
||||
AND state = 'active'
|
||||
`;
|
||||
|
||||
// Query for idle connections
|
||||
const idleConnections = await db.$queryRaw<Array<{ count: bigint }>>`
|
||||
SELECT COUNT(*) as count
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = current_database()
|
||||
AND state = 'idle'
|
||||
`;
|
||||
|
||||
// Query for total connections
|
||||
const totalConnections = await db.$queryRaw<Array<{ count: bigint }>>`
|
||||
SELECT COUNT(*) as count
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = current_database()
|
||||
`;
|
||||
|
||||
// Get max connections setting
|
||||
const maxConnections = await db.$queryRaw<Array<{ setting: string }>>`
|
||||
SELECT setting
|
||||
FROM pg_settings
|
||||
WHERE name = 'max_connections'
|
||||
`;
|
||||
|
||||
return {
|
||||
active: Number(activeConnections[0]?.count ?? 0),
|
||||
idle: Number(idleConnections[0]?.count ?? 0),
|
||||
total: Number(totalConnections[0]?.count ?? 0),
|
||||
max: Number(maxConnections[0]?.setting ?? 0),
|
||||
utilizationPercent: maxConnections[0]?.setting
|
||||
? ((Number(totalConnections[0]?.count ?? 0) / Number(maxConnections[0].setting)) * 100).toFixed(2)
|
||||
: '0',
|
||||
isPgBouncer,
|
||||
isShuttingDown,
|
||||
};
|
||||
} catch (error) {
|
||||
console.error('Error fetching connection pool metrics:', error);
|
||||
return {
|
||||
active: 0,
|
||||
idle: 0,
|
||||
total: 0,
|
||||
max: 0,
|
||||
utilizationPercent: '0',
|
||||
isPgBouncer,
|
||||
isShuttingDown,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Health check for database connection
|
||||
*/
|
||||
export async function checkDatabaseHealth(): Promise<{
|
||||
healthy: boolean;
|
||||
responseTime?: number;
|
||||
error?: string;
|
||||
}> {
|
||||
const startTime = Date.now();
|
||||
try {
|
||||
await db.$queryRaw`SELECT 1`;
|
||||
const responseTime = Date.now() - startTime;
|
||||
return { healthy: true, responseTime };
|
||||
} catch (error) {
|
||||
return {
|
||||
healthy: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// Graceful shutdown handlers
|
||||
const gracefulShutdown = async (signal: string) => {
|
||||
console.log(`Received ${signal}, closing database connections...`);
|
||||
if (isShuttingDown) {
|
||||
console.log('Shutdown already in progress...');
|
||||
return;
|
||||
}
|
||||
|
||||
isShuttingDown = true;
|
||||
console.log(`Received ${signal}, initiating graceful shutdown...`);
|
||||
|
||||
try {
|
||||
// Close database connections
|
||||
console.log('Closing database connections...');
|
||||
await db.$disconnect();
|
||||
console.log('Database connections closed.');
|
||||
console.log('✅ Database connections closed successfully');
|
||||
|
||||
// Close Redis connections
|
||||
const { closeRedisConnection } = await import('./utils/redis');
|
||||
await closeRedisConnection();
|
||||
} catch (error) {
|
||||
console.error('Error during database disconnect:', error);
|
||||
console.error('❌ Error during graceful shutdown:', error);
|
||||
} finally {
|
||||
process.exit(0);
|
||||
}
|
||||
@@ -39,22 +155,47 @@ process.on('SIGINT', () => gracefulShutdown('SIGINT'));
|
||||
|
||||
// Handle uncaught exceptions and unhandled rejections
|
||||
process.on('uncaughtException', async (err) => {
|
||||
console.error('Uncaught Exception:', err);
|
||||
await db.$disconnect();
|
||||
console.error('❌ Uncaught Exception:', err);
|
||||
if (!isShuttingDown) {
|
||||
isShuttingDown = true;
|
||||
try {
|
||||
await db.$disconnect();
|
||||
const { closeRedisConnection } = await import('./utils/redis');
|
||||
await closeRedisConnection();
|
||||
} catch (disconnectError) {
|
||||
console.error('Error during emergency disconnect:', disconnectError);
|
||||
}
|
||||
}
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
process.on('unhandledRejection', async (reason) => {
|
||||
console.error('Unhandled Rejection:', reason);
|
||||
await db.$disconnect();
|
||||
console.error('❌ Unhandled Rejection:', reason);
|
||||
if (!isShuttingDown) {
|
||||
isShuttingDown = true;
|
||||
try {
|
||||
await db.$disconnect();
|
||||
const { closeRedisConnection } = await import('./utils/redis');
|
||||
await closeRedisConnection();
|
||||
} catch (disconnectError) {
|
||||
console.error('Error during emergency disconnect:', disconnectError);
|
||||
}
|
||||
}
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
// Connection pooling is configured via DATABASE_URL query parameters:
|
||||
// postgresql://user:password@localhost:5432/spywatcher?connection_limit=10&pool_timeout=20
|
||||
//
|
||||
// Recommended settings:
|
||||
// Connection pooling configuration:
|
||||
//
|
||||
// When using PgBouncer (recommended for production):
|
||||
// DATABASE_URL=postgresql://user:password@pgbouncer:6432/dbname?pgbouncer=true
|
||||
// - PgBouncer handles connection pooling at the server level
|
||||
// - Configure PgBouncer pool_mode=transaction for best compatibility with Prisma
|
||||
// - Keep Prisma connection_limit low (1-5) as PgBouncer manages the pool
|
||||
// - PgBouncer configuration in pgbouncer/pgbouncer.ini
|
||||
//
|
||||
// When connecting directly to PostgreSQL:
|
||||
// DATABASE_URL=postgresql://user:password@postgres:5432/dbname?connection_limit=20&pool_timeout=20&connect_timeout=10
|
||||
// - connection_limit: 10-20 for small apps, 20-50 for medium apps
|
||||
// - pool_timeout: 20 seconds
|
||||
// - connect_timeout: 10 seconds
|
||||
// - pool_timeout: 20 seconds (time to wait for available connection)
|
||||
// - connect_timeout: 10 seconds (time to wait for initial connection)
|
||||
|
||||
|
||||
@@ -17,6 +17,11 @@ import {
|
||||
analyzeAllTables,
|
||||
} from '../utils/databaseMaintenance';
|
||||
import { getRedisClient } from '../utils/redis';
|
||||
import {
|
||||
getSystemHealth,
|
||||
getConnectionPoolStats,
|
||||
getConnectionPoolAlerts,
|
||||
} from '../utils/connectionPoolMonitor';
|
||||
|
||||
const router = Router();
|
||||
const redis = getRedisClient();
|
||||
@@ -301,6 +306,58 @@ router.get('/database/health', async (_req: Request, res: Response) => {
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* GET /api/admin/monitoring/connections/health
|
||||
* Get comprehensive system health including connection pools
|
||||
*/
|
||||
router.get('/connections/health', async (_req: Request, res: Response) => {
|
||||
try {
|
||||
const health = await getSystemHealth();
|
||||
res.json(health);
|
||||
} catch (error) {
|
||||
console.error('Failed to get system health:', error);
|
||||
res.status(500).json({
|
||||
error: 'Failed to retrieve system health',
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* GET /api/admin/monitoring/connections/pool
|
||||
* Get connection pool statistics
|
||||
*/
|
||||
router.get('/connections/pool', async (_req: Request, res: Response) => {
|
||||
try {
|
||||
const stats = await getConnectionPoolStats();
|
||||
res.json(stats);
|
||||
} catch (error) {
|
||||
console.error('Failed to get connection pool stats:', error);
|
||||
res.status(500).json({
|
||||
error: 'Failed to retrieve connection pool statistics',
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* GET /api/admin/monitoring/connections/alerts
|
||||
* Get connection pool alerts and warnings
|
||||
*/
|
||||
router.get('/connections/alerts', async (_req: Request, res: Response) => {
|
||||
try {
|
||||
const alerts = await getConnectionPoolAlerts();
|
||||
res.json({
|
||||
alerts,
|
||||
count: alerts.length,
|
||||
timestamp: new Date().toISOString(),
|
||||
});
|
||||
} catch (error) {
|
||||
console.error('Failed to get connection pool alerts:', error);
|
||||
res.status(500).json({
|
||||
error: 'Failed to retrieve connection pool alerts',
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* GET /api/admin/monitoring/database/tables
|
||||
* Get table statistics (sizes, row counts, etc.)
|
||||
|
||||
279
backend/src/utils/connectionPoolMonitor.ts
Normal file
279
backend/src/utils/connectionPoolMonitor.ts
Normal file
@@ -0,0 +1,279 @@
|
||||
/**
|
||||
* Connection Pool Monitoring Utility
|
||||
*
|
||||
* Provides monitoring and health check capabilities for database
|
||||
* and Redis connection pools.
|
||||
*/
|
||||
|
||||
import { checkDatabaseHealth, getConnectionPoolMetrics } from '../db';
|
||||
import { getRedisMetrics, isRedisAvailable } from './redis';
|
||||
|
||||
/**
|
||||
* Overall system health status
|
||||
*/
|
||||
export interface SystemHealth {
|
||||
healthy: boolean;
|
||||
timestamp: string;
|
||||
database: DatabaseHealth;
|
||||
redis: RedisHealth;
|
||||
}
|
||||
|
||||
/**
|
||||
* Database connection health
|
||||
*/
|
||||
export interface DatabaseHealth {
|
||||
healthy: boolean;
|
||||
responseTime?: number;
|
||||
connectionPool?: {
|
||||
active: number;
|
||||
idle: number;
|
||||
total: number;
|
||||
max: number;
|
||||
utilizationPercent: string;
|
||||
isPgBouncer: boolean;
|
||||
isShuttingDown: boolean;
|
||||
};
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Redis connection health
|
||||
*/
|
||||
export interface RedisHealth {
|
||||
available: boolean;
|
||||
connected: boolean;
|
||||
status?: string;
|
||||
metrics?: {
|
||||
totalConnectionsReceived: string;
|
||||
totalCommandsProcessed: string;
|
||||
instantaneousOpsPerSec: string;
|
||||
usedMemory: string;
|
||||
};
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Connection pool statistics for alerting
|
||||
*/
|
||||
export interface ConnectionPoolStats {
|
||||
database: {
|
||||
utilizationPercent: number;
|
||||
activeConnections: number;
|
||||
maxConnections: number;
|
||||
isHealthy: boolean;
|
||||
};
|
||||
redis: {
|
||||
available: boolean;
|
||||
connected: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get comprehensive system health including connection pools
|
||||
*/
|
||||
export async function getSystemHealth(): Promise<SystemHealth> {
|
||||
const [dbHealth, dbMetrics, redisMetrics] = await Promise.all([
|
||||
checkDatabaseHealth(),
|
||||
getConnectionPoolMetrics(),
|
||||
getRedisMetrics(),
|
||||
]);
|
||||
|
||||
const redisAvailable = await isRedisAvailable();
|
||||
|
||||
return {
|
||||
healthy: dbHealth.healthy && (!process.env.REDIS_URL || redisAvailable),
|
||||
timestamp: new Date().toISOString(),
|
||||
database: {
|
||||
healthy: dbHealth.healthy,
|
||||
responseTime: dbHealth.responseTime,
|
||||
connectionPool: dbMetrics.error
|
||||
? undefined
|
||||
: {
|
||||
active: dbMetrics.active,
|
||||
idle: dbMetrics.idle,
|
||||
total: dbMetrics.total,
|
||||
max: dbMetrics.max,
|
||||
utilizationPercent: dbMetrics.utilizationPercent,
|
||||
isPgBouncer: dbMetrics.isPgBouncer,
|
||||
isShuttingDown: dbMetrics.isShuttingDown,
|
||||
},
|
||||
error: dbHealth.error || dbMetrics.error,
|
||||
},
|
||||
redis: {
|
||||
available: redisMetrics.available,
|
||||
connected: redisMetrics.connected,
|
||||
status: redisMetrics.status,
|
||||
metrics: redisMetrics.available && redisMetrics.connected
|
||||
? {
|
||||
totalConnectionsReceived: redisMetrics.totalConnectionsReceived ?? 'N/A',
|
||||
totalCommandsProcessed: redisMetrics.totalCommandsProcessed ?? 'N/A',
|
||||
instantaneousOpsPerSec: redisMetrics.instantaneousOpsPerSec ?? 'N/A',
|
||||
usedMemory: redisMetrics.usedMemory ?? 'N/A',
|
||||
}
|
||||
: undefined,
|
||||
error: redisMetrics.error,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get connection pool statistics for alerting purposes
|
||||
*/
|
||||
export async function getConnectionPoolStats(): Promise<ConnectionPoolStats> {
|
||||
const [dbMetrics, redisMetrics] = await Promise.all([
|
||||
getConnectionPoolMetrics(),
|
||||
getRedisMetrics(),
|
||||
]);
|
||||
|
||||
return {
|
||||
database: {
|
||||
utilizationPercent: parseFloat(dbMetrics.utilizationPercent),
|
||||
activeConnections: dbMetrics.active,
|
||||
maxConnections: dbMetrics.max,
|
||||
isHealthy: !dbMetrics.error,
|
||||
},
|
||||
redis: {
|
||||
available: redisMetrics.available,
|
||||
connected: redisMetrics.connected,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if connection pool utilization is above threshold
|
||||
* @param threshold - Percentage threshold (default 80%)
|
||||
*/
|
||||
export async function isConnectionPoolOverloaded(
|
||||
threshold: number = 80
|
||||
): Promise<boolean> {
|
||||
const stats = await getConnectionPoolStats();
|
||||
return stats.database.utilizationPercent >= threshold;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get connection pool alerts
|
||||
*/
|
||||
export async function getConnectionPoolAlerts(): Promise<string[]> {
|
||||
const alerts: string[] = [];
|
||||
const stats = await getConnectionPoolStats();
|
||||
|
||||
// Database connection pool alerts
|
||||
if (stats.database.utilizationPercent >= 90) {
|
||||
alerts.push(
|
||||
`CRITICAL: Database connection pool at ${stats.database.utilizationPercent}% utilization`
|
||||
);
|
||||
} else if (stats.database.utilizationPercent >= 80) {
|
||||
alerts.push(
|
||||
`WARNING: Database connection pool at ${stats.database.utilizationPercent}% utilization`
|
||||
);
|
||||
}
|
||||
|
||||
if (!stats.database.isHealthy) {
|
||||
alerts.push('CRITICAL: Database connection pool is not healthy');
|
||||
}
|
||||
|
||||
// Redis connection alerts
|
||||
if (process.env.REDIS_URL && !stats.redis.available) {
|
||||
alerts.push('WARNING: Redis is configured but not available');
|
||||
}
|
||||
|
||||
if (process.env.REDIS_URL && !stats.redis.connected) {
|
||||
alerts.push('WARNING: Redis is not connected');
|
||||
}
|
||||
|
||||
return alerts;
|
||||
}
|
||||
|
||||
/**
|
||||
* Log connection pool metrics to console (for monitoring/debugging)
|
||||
*/
|
||||
export async function logConnectionPoolMetrics(): Promise<void> {
|
||||
const health = await getSystemHealth();
|
||||
|
||||
console.log('=== Connection Pool Metrics ===');
|
||||
console.log(`Timestamp: ${health.timestamp}`);
|
||||
console.log(`Overall Health: ${health.healthy ? '✅ HEALTHY' : '❌ UNHEALTHY'}`);
|
||||
|
||||
console.log('\n--- Database ---');
|
||||
console.log(`Health: ${health.database.healthy ? '✅' : '❌'}`);
|
||||
if (health.database.responseTime) {
|
||||
console.log(`Response Time: ${health.database.responseTime}ms`);
|
||||
}
|
||||
if (health.database.connectionPool) {
|
||||
console.log(`Connection Pool:`);
|
||||
console.log(` Active: ${health.database.connectionPool.active}`);
|
||||
console.log(` Idle: ${health.database.connectionPool.idle}`);
|
||||
console.log(` Total: ${health.database.connectionPool.total}`);
|
||||
console.log(` Max: ${health.database.connectionPool.max}`);
|
||||
console.log(
|
||||
` Utilization: ${health.database.connectionPool.utilizationPercent}%`
|
||||
);
|
||||
console.log(
|
||||
` PgBouncer: ${health.database.connectionPool.isPgBouncer ? 'Yes' : 'No'}`
|
||||
);
|
||||
}
|
||||
if (health.database.error) {
|
||||
console.log(`Error: ${health.database.error}`);
|
||||
}
|
||||
|
||||
console.log('\n--- Redis ---');
|
||||
console.log(`Available: ${health.redis.available ? '✅' : '❌'}`);
|
||||
console.log(`Connected: ${health.redis.connected ? '✅' : '❌'}`);
|
||||
if (health.redis.status) {
|
||||
console.log(`Status: ${health.redis.status}`);
|
||||
}
|
||||
if (health.redis.metrics) {
|
||||
console.log(`Metrics:`);
|
||||
console.log(
|
||||
` Total Connections: ${health.redis.metrics.totalConnectionsReceived}`
|
||||
);
|
||||
console.log(
|
||||
` Total Commands: ${health.redis.metrics.totalCommandsProcessed}`
|
||||
);
|
||||
console.log(` Ops/sec: ${health.redis.metrics.instantaneousOpsPerSec}`);
|
||||
console.log(` Memory Used: ${health.redis.metrics.usedMemory}`);
|
||||
}
|
||||
if (health.redis.error) {
|
||||
console.log(`Error: ${health.redis.error}`);
|
||||
}
|
||||
|
||||
console.log('==============================\n');
|
||||
|
||||
// Log any alerts
|
||||
const alerts = await getConnectionPoolAlerts();
|
||||
if (alerts.length > 0) {
|
||||
console.log('🚨 ALERTS:');
|
||||
alerts.forEach((alert) => console.log(` - ${alert}`));
|
||||
console.log('');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Start periodic connection pool monitoring
|
||||
* @param intervalMs - Monitoring interval in milliseconds (default: 60000 = 1 minute)
|
||||
*/
|
||||
export function startConnectionPoolMonitoring(intervalMs: number = 60000): NodeJS.Timeout {
|
||||
console.log(
|
||||
`Starting connection pool monitoring (interval: ${intervalMs}ms)`
|
||||
);
|
||||
|
||||
// Log initial metrics
|
||||
logConnectionPoolMetrics().catch((err) => {
|
||||
console.error('Error logging initial connection pool metrics:', err);
|
||||
});
|
||||
|
||||
// Start periodic monitoring
|
||||
return setInterval(() => {
|
||||
logConnectionPoolMetrics().catch((err) => {
|
||||
console.error('Error logging connection pool metrics:', err);
|
||||
});
|
||||
}, intervalMs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop periodic connection pool monitoring
|
||||
*/
|
||||
export function stopConnectionPoolMonitoring(timer: NodeJS.Timeout): void {
|
||||
clearInterval(timer);
|
||||
console.log('Connection pool monitoring stopped');
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import { env } from './env';
|
||||
* Redis client instance for rate limiting and caching
|
||||
*/
|
||||
let redisClient: Redis | null = null;
|
||||
let isShuttingDown = false;
|
||||
|
||||
/**
|
||||
* Initialize and return Redis client
|
||||
@@ -27,14 +28,22 @@ export function getRedisClient(): Redis | null {
|
||||
maxRetriesPerRequest: 3,
|
||||
enableReadyCheck: true,
|
||||
retryStrategy(times: number) {
|
||||
// Don't retry during shutdown
|
||||
if (isShuttingDown) {
|
||||
return null;
|
||||
}
|
||||
const delay = Math.min(times * 50, 2000);
|
||||
return delay;
|
||||
},
|
||||
lazyConnect: false,
|
||||
// Connection pool settings for better performance
|
||||
enableOfflineQueue: true,
|
||||
connectTimeout: 10000,
|
||||
keepAlive: 30000,
|
||||
});
|
||||
|
||||
redisClient.on('error', (err: Error) => {
|
||||
console.error('Redis connection error:', err);
|
||||
console.error('❌ Redis connection error:', err);
|
||||
});
|
||||
|
||||
redisClient.on('connect', () => {
|
||||
@@ -45,6 +54,16 @@ export function getRedisClient(): Redis | null {
|
||||
console.log('✅ Redis ready for operations');
|
||||
});
|
||||
|
||||
redisClient.on('reconnecting', () => {
|
||||
console.log('🔄 Redis reconnecting...');
|
||||
});
|
||||
|
||||
redisClient.on('close', () => {
|
||||
if (!isShuttingDown) {
|
||||
console.log('⚠️ Redis connection closed unexpectedly');
|
||||
}
|
||||
});
|
||||
|
||||
return redisClient;
|
||||
} catch (error) {
|
||||
console.error('Failed to initialize Redis client:', error);
|
||||
@@ -69,17 +88,79 @@ export async function isRedisAvailable(): Promise<boolean> {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get Redis connection metrics
|
||||
*/
|
||||
export async function getRedisMetrics() {
|
||||
const client = getRedisClient();
|
||||
if (!client) {
|
||||
return {
|
||||
available: false,
|
||||
connected: false,
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
const info = await client.info('stats');
|
||||
const stats = info.split('\r\n').reduce(
|
||||
(acc, line) => {
|
||||
const [key, value] = line.split(':');
|
||||
if (key && value) {
|
||||
// eslint-disable-next-line security/detect-object-injection
|
||||
acc[key] = value;
|
||||
}
|
||||
return acc;
|
||||
},
|
||||
{} as Record<string, string>
|
||||
);
|
||||
|
||||
return {
|
||||
available: true,
|
||||
connected: client.status === 'ready',
|
||||
status: client.status,
|
||||
totalConnectionsReceived: stats['total_connections_received'] ?? 'N/A',
|
||||
totalCommandsProcessed: stats['total_commands_processed'] ?? 'N/A',
|
||||
instantaneousOpsPerSec: stats['instantaneous_ops_per_sec'] ?? 'N/A',
|
||||
usedMemory: stats['used_memory_human'] ?? 'N/A',
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
available: true,
|
||||
connected: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Gracefully close Redis connection
|
||||
*/
|
||||
export async function closeRedisConnection(): Promise<void> {
|
||||
if (isShuttingDown) {
|
||||
return;
|
||||
}
|
||||
|
||||
isShuttingDown = true;
|
||||
|
||||
if (redisClient) {
|
||||
await redisClient.quit();
|
||||
redisClient = null;
|
||||
console.log('✅ Redis connection closed');
|
||||
console.log('Closing Redis connection...');
|
||||
try {
|
||||
// Use quit() for graceful shutdown - waits for pending commands
|
||||
await redisClient.quit();
|
||||
redisClient = null;
|
||||
console.log('✅ Redis connection closed successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Error closing Redis connection:', error);
|
||||
// Force disconnect if quit fails
|
||||
redisClient.disconnect();
|
||||
redisClient = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Note: Shutdown handlers are registered in db.ts to avoid conflicts
|
||||
// Redis cleanup is called from the main shutdown handler
|
||||
|
||||
/**
|
||||
* Scan Redis keys matching a pattern using SCAN command (non-blocking)
|
||||
* This is a safer alternative to KEYS command for production use
|
||||
|
||||
@@ -46,6 +46,24 @@ services:
|
||||
- "-c"
|
||||
- "max_wal_size=4GB"
|
||||
|
||||
pgbouncer:
|
||||
build:
|
||||
context: ./pgbouncer
|
||||
dockerfile: Dockerfile
|
||||
container_name: spywatcher-pgbouncer-dev
|
||||
environment:
|
||||
DB_USER: spywatcher
|
||||
DB_PASSWORD: ${DB_PASSWORD:-spywatcher_dev_password}
|
||||
PGBOUNCER_ADMIN_USER: pgbouncer_admin
|
||||
PGBOUNCER_ADMIN_PASSWORD: ${PGBOUNCER_ADMIN_PASSWORD:-pgbouncer_admin_pass}
|
||||
ports:
|
||||
- "6432:6432"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- spywatcher-network
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: spywatcher-redis-dev
|
||||
@@ -72,7 +90,9 @@ services:
|
||||
- /app/node_modules
|
||||
- /app/dist
|
||||
environment:
|
||||
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD:-spywatcher_dev_password}@postgres:5432/spywatcher
|
||||
# Use PgBouncer for application connections, direct for migrations
|
||||
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD:-spywatcher_dev_password}@pgbouncer:6432/spywatcher?pgbouncer=true
|
||||
DATABASE_URL_DIRECT: postgresql://spywatcher:${DB_PASSWORD:-spywatcher_dev_password}@postgres:5432/spywatcher
|
||||
REDIS_URL: redis://redis:6379
|
||||
NODE_ENV: development
|
||||
PORT: 3001
|
||||
@@ -88,13 +108,13 @@ services:
|
||||
ports:
|
||||
- "3001:3001"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
pgbouncer:
|
||||
condition: service_started
|
||||
redis:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- spywatcher-network
|
||||
command: sh -c "npx prisma migrate dev && npm run dev:api"
|
||||
command: sh -c "DATABASE_URL=$DATABASE_URL_DIRECT npx prisma migrate dev && npm run dev:api"
|
||||
|
||||
frontend:
|
||||
build:
|
||||
|
||||
@@ -50,6 +50,28 @@ services:
|
||||
cpus: '1'
|
||||
memory: 512M
|
||||
|
||||
pgbouncer:
|
||||
build:
|
||||
context: ./pgbouncer
|
||||
dockerfile: Dockerfile
|
||||
container_name: spywatcher-pgbouncer-prod
|
||||
environment:
|
||||
DB_USER: spywatcher
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
PGBOUNCER_ADMIN_USER: pgbouncer_admin
|
||||
PGBOUNCER_ADMIN_PASSWORD: ${PGBOUNCER_ADMIN_PASSWORD:-pgbouncer_admin_pass}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- spywatcher-network
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.25'
|
||||
memory: 128M
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: spywatcher-redis-prod
|
||||
@@ -76,7 +98,7 @@ services:
|
||||
dockerfile: Dockerfile
|
||||
container_name: spywatcher-backend-prod
|
||||
environment:
|
||||
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD}@postgres:5432/spywatcher
|
||||
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD}@pgbouncer:6432/spywatcher?pgbouncer=true
|
||||
REDIS_URL: redis://redis:6379
|
||||
NODE_ENV: production
|
||||
PORT: 3001
|
||||
@@ -90,8 +112,8 @@ services:
|
||||
JWT_REFRESH_SECRET: ${JWT_REFRESH_SECRET}
|
||||
JWT_SECRET: ${JWT_SECRET}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
pgbouncer:
|
||||
condition: service_started
|
||||
redis:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
@@ -110,6 +132,7 @@ services:
|
||||
dockerfile: Dockerfile
|
||||
container_name: spywatcher-migrate-prod
|
||||
environment:
|
||||
# Migrations should connect directly to postgres, not through pgbouncer
|
||||
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD}@postgres:5432/spywatcher
|
||||
depends_on:
|
||||
postgres:
|
||||
|
||||
35
docs/PGBOUNCER_SETUP.md
Normal file
35
docs/PGBOUNCER_SETUP.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# PgBouncer Setup Guide
|
||||
|
||||
Quick reference guide for setting up and managing PgBouncer connection pooling.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
See [CONNECTION_POOLING.md](../CONNECTION_POOLING.md) for full documentation.
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
docker-compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
export DB_PASSWORD="your_secure_password"
|
||||
docker-compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
```bash
|
||||
# System health
|
||||
curl http://localhost:3001/api/admin/monitoring/connections/health
|
||||
|
||||
# Pool statistics
|
||||
curl http://localhost:3001/api/admin/monitoring/connections/pool
|
||||
```
|
||||
|
||||
## 🔗 Resources
|
||||
|
||||
- [Full Documentation](../CONNECTION_POOLING.md)
|
||||
- [PgBouncer Official Docs](https://www.pgbouncer.org/)
|
||||
3
pgbouncer/.gitignore
vendored
Normal file
3
pgbouncer/.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
# Ignore generated userlist with credentials
|
||||
# This file is generated at runtime by entrypoint.sh
|
||||
userlist.txt
|
||||
36
pgbouncer/Dockerfile
Normal file
36
pgbouncer/Dockerfile
Normal file
@@ -0,0 +1,36 @@
|
||||
FROM alpine:3.19
|
||||
|
||||
# Install PgBouncer and dependencies
|
||||
RUN apk add --no-cache \
|
||||
pgbouncer \
|
||||
postgresql-client \
|
||||
su-exec \
|
||||
&& rm -rf /var/cache/apk/*
|
||||
|
||||
# Create pgbouncer user and directories
|
||||
RUN addgroup -g 70 -S pgbouncer \
|
||||
&& adduser -u 70 -S -D -G pgbouncer -H -h /var/lib/pgbouncer -s /sbin/nologin pgbouncer \
|
||||
&& mkdir -p /etc/pgbouncer /var/log/pgbouncer /var/run/pgbouncer \
|
||||
&& chown -R pgbouncer:pgbouncer /etc/pgbouncer /var/log/pgbouncer /var/run/pgbouncer
|
||||
|
||||
# Copy configuration files
|
||||
COPY pgbouncer.ini /etc/pgbouncer/pgbouncer.ini
|
||||
COPY userlist.txt.template /etc/pgbouncer/userlist.txt.template
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
|
||||
# Make entrypoint executable
|
||||
RUN chmod +x /entrypoint.sh
|
||||
|
||||
# Set proper permissions for config files
|
||||
RUN chown pgbouncer:pgbouncer /etc/pgbouncer/pgbouncer.ini /etc/pgbouncer/userlist.txt.template \
|
||||
&& chmod 644 /etc/pgbouncer/pgbouncer.ini
|
||||
|
||||
# Expose PgBouncer port
|
||||
EXPOSE 6432
|
||||
|
||||
# Health check with proper authentication
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD PGPASSWORD="$DB_PASSWORD" psql -h 127.0.0.1 -p 6432 -U $DB_USER -d pgbouncer -c "SHOW POOLS;" || exit 1
|
||||
|
||||
# Run entrypoint script as root to allow chmod, then switch to pgbouncer user within
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
154
pgbouncer/README.md
Normal file
154
pgbouncer/README.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# PgBouncer Configuration
|
||||
|
||||
This directory contains the PgBouncer connection pooler configuration for the Discord Spywatcher application.
|
||||
|
||||
## Files
|
||||
|
||||
- **pgbouncer.ini** - Main PgBouncer configuration file
|
||||
- **userlist.txt.template** - Template for user authentication (actual file generated at runtime)
|
||||
- **entrypoint.sh** - Startup script that generates credentials and starts PgBouncer
|
||||
- **Dockerfile** - Container image definition
|
||||
- **.gitignore** - Prevents committing generated credentials
|
||||
|
||||
## Configuration
|
||||
|
||||
### Pool Settings
|
||||
|
||||
The default configuration uses transaction-mode pooling with the following limits:
|
||||
|
||||
- **Pool Mode**: `transaction` (optimal for Prisma)
|
||||
- **Default Pool Size**: 25 connections per database
|
||||
- **Min Pool Size**: 5 connections minimum
|
||||
- **Reserve Pool**: 5 additional connections for spikes
|
||||
- **Max Client Connections**: 100 simultaneous clients
|
||||
|
||||
### Connection Lifecycle
|
||||
|
||||
- **Server Lifetime**: 3600 seconds (1 hour)
|
||||
- **Idle Timeout**: 600 seconds (10 minutes)
|
||||
- **Query Wait Timeout**: 120 seconds
|
||||
|
||||
### Security
|
||||
|
||||
- **Authentication**: MD5 hashed passwords
|
||||
- **Admin Access**: Separate admin user for monitoring
|
||||
- **Network**: Internal Docker network only (no external exposure in production)
|
||||
|
||||
## Usage
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Required:
|
||||
- `DB_USER` - Database username (e.g., "spywatcher")
|
||||
- `DB_PASSWORD` - Database password
|
||||
|
||||
Optional:
|
||||
- `PGBOUNCER_ADMIN_USER` - Admin username (default: "pgbouncer_admin")
|
||||
- `PGBOUNCER_ADMIN_PASSWORD` - Admin password (recommended for production)
|
||||
|
||||
### Starting PgBouncer
|
||||
|
||||
The entrypoint script automatically:
|
||||
1. Generates MD5 hashed passwords
|
||||
2. Creates `userlist.txt` from environment variables
|
||||
3. Sets appropriate file permissions
|
||||
4. Starts PgBouncer
|
||||
|
||||
### Connecting
|
||||
|
||||
#### Application Connection (through PgBouncer)
|
||||
```bash
|
||||
postgresql://user:password@pgbouncer:6432/spywatcher?pgbouncer=true
|
||||
```
|
||||
|
||||
#### Admin Console
|
||||
```bash
|
||||
psql -h pgbouncer -p 6432 -U pgbouncer_admin pgbouncer
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### PgBouncer Admin Commands
|
||||
|
||||
```sql
|
||||
SHOW POOLS; -- Pool statistics
|
||||
SHOW DATABASES; -- Database connections
|
||||
SHOW CLIENTS; -- Client connections
|
||||
SHOW SERVERS; -- Server connections
|
||||
SHOW CONFIG; -- Current configuration
|
||||
SHOW STATS; -- Performance statistics
|
||||
RELOAD; -- Reload configuration
|
||||
```
|
||||
|
||||
### Health Check
|
||||
|
||||
The Docker container includes a health check that runs:
|
||||
```bash
|
||||
psql -h 127.0.0.1 -p 6432 -U $DB_USER -d pgbouncer -c "SHOW POOLS;"
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
To modify PgBouncer settings:
|
||||
|
||||
1. Edit `pgbouncer.ini`
|
||||
2. Rebuild the Docker image or restart the container
|
||||
3. Verify changes with `SHOW CONFIG;` in admin console
|
||||
|
||||
### Common Adjustments
|
||||
|
||||
**Increase pool size for high load:**
|
||||
```ini
|
||||
default_pool_size = 50
|
||||
max_client_conn = 200
|
||||
```
|
||||
|
||||
**Adjust timeouts:**
|
||||
```ini
|
||||
server_idle_timeout = 300 # Reduce to 5 minutes
|
||||
query_wait_timeout = 60 # Reduce to 1 minute
|
||||
```
|
||||
|
||||
**Enable verbose logging:**
|
||||
```ini
|
||||
log_connections = 1
|
||||
log_disconnections = 1
|
||||
log_pooler_errors = 1
|
||||
verbose = 1
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "No such user"
|
||||
Check that credentials are set correctly:
|
||||
```bash
|
||||
docker exec container_name cat /etc/pgbouncer/userlist.txt
|
||||
```
|
||||
|
||||
### Issue: Connection refused
|
||||
Ensure PostgreSQL is running and healthy:
|
||||
```bash
|
||||
docker-compose ps postgres
|
||||
docker logs container_name
|
||||
```
|
||||
|
||||
### Issue: Pool saturation
|
||||
Check pool utilization:
|
||||
```sql
|
||||
SHOW POOLS;
|
||||
-- If cl_waiting > 0, increase pool size
|
||||
```
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Never commit `userlist.txt`** - it contains credentials (already in `.gitignore`)
|
||||
- **Use strong passwords** - especially for production
|
||||
- **Rotate credentials** - regularly change passwords
|
||||
- **Limit network access** - PgBouncer should only be accessible within Docker network
|
||||
|
||||
## Resources
|
||||
|
||||
- [PgBouncer Documentation](https://www.pgbouncer.org/)
|
||||
- [Full Setup Guide](../docs/PGBOUNCER_SETUP.md)
|
||||
- [Connection Pooling Guide](../CONNECTION_POOLING.md)
|
||||
- [PostgreSQL Guide](../POSTGRESQL.md)
|
||||
45
pgbouncer/entrypoint.sh
Normal file
45
pgbouncer/entrypoint.sh
Normal file
@@ -0,0 +1,45 @@
|
||||
#!/bin/sh
|
||||
# PgBouncer entrypoint script
|
||||
# This script generates the userlist.txt file from environment variables
|
||||
# and starts PgBouncer
|
||||
|
||||
set -e
|
||||
|
||||
# Check if required environment variables are set
|
||||
if [ -z "$DB_USER" ]; then
|
||||
echo "Error: DB_USER environment variable is required"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -z "$DB_PASSWORD" ]; then
|
||||
echo "Error: DB_PASSWORD environment variable is required"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Generate md5 hash for the password
|
||||
# Format: "md5" + md5(password + username)
|
||||
MD5_HASH=$(echo -n "${DB_PASSWORD}${DB_USER}" | md5sum | awk '{print $1}')
|
||||
MD5_PASSWORD="md5${MD5_HASH}"
|
||||
|
||||
# Create userlist.txt with the hashed password
|
||||
echo "\"${DB_USER}\" \"${MD5_PASSWORD}\"" > /etc/pgbouncer/userlist.txt
|
||||
|
||||
# Add admin user if specified
|
||||
if [ -n "$PGBOUNCER_ADMIN_USER" ] && [ -n "$PGBOUNCER_ADMIN_PASSWORD" ]; then
|
||||
ADMIN_MD5_HASH=$(echo -n "${PGBOUNCER_ADMIN_PASSWORD}${PGBOUNCER_ADMIN_USER}" | md5sum | awk '{print $1}')
|
||||
ADMIN_MD5_PASSWORD="md5${ADMIN_MD5_HASH}"
|
||||
echo "\"${PGBOUNCER_ADMIN_USER}\" \"${ADMIN_MD5_PASSWORD}\"" >> /etc/pgbouncer/userlist.txt
|
||||
fi
|
||||
|
||||
# Set proper permissions (run as root before switching users)
|
||||
chmod 600 /etc/pgbouncer/userlist.txt
|
||||
chown pgbouncer:pgbouncer /etc/pgbouncer/userlist.txt
|
||||
|
||||
echo "PgBouncer configuration initialized"
|
||||
echo "User: ${DB_USER}"
|
||||
echo "Pool mode: transaction"
|
||||
echo "Default pool size: 25"
|
||||
echo "Max client connections: 100"
|
||||
|
||||
# Switch to pgbouncer user and start PgBouncer
|
||||
exec su-exec pgbouncer pgbouncer /etc/pgbouncer/pgbouncer.ini
|
||||
85
pgbouncer/pgbouncer.ini
Normal file
85
pgbouncer/pgbouncer.ini
Normal file
@@ -0,0 +1,85 @@
|
||||
[databases]
|
||||
; Database-name = host=hostname port=port dbname=database
|
||||
spywatcher = host=postgres port=5432 dbname=spywatcher
|
||||
|
||||
[pgbouncer]
|
||||
; Connection pooling mode (session, transaction, or statement)
|
||||
; session - one server connection per client connection (default)
|
||||
; transaction - one server connection per transaction
|
||||
; statement - one server connection per statement (not recommended for prepared statements)
|
||||
pool_mode = transaction
|
||||
|
||||
; Listen address and port
|
||||
listen_addr = 0.0.0.0
|
||||
listen_port = 6432
|
||||
|
||||
; Authentication
|
||||
auth_type = md5
|
||||
auth_file = /etc/pgbouncer/userlist.txt
|
||||
|
||||
; Administrator credentials
|
||||
admin_users = pgbouncer_admin
|
||||
|
||||
; Connection limits per pool
|
||||
; Total for all pools = default_pool_size * num_databases
|
||||
default_pool_size = 25
|
||||
min_pool_size = 5
|
||||
reserve_pool_size = 5
|
||||
reserve_pool_timeout = 3
|
||||
|
||||
; Maximum allowed connections per database per user
|
||||
max_db_connections = 50
|
||||
max_user_connections = 50
|
||||
|
||||
; Maximum total client connections
|
||||
max_client_conn = 100
|
||||
|
||||
; Server connection lifetime
|
||||
server_lifetime = 3600
|
||||
server_idle_timeout = 600
|
||||
|
||||
; Client connection limits
|
||||
server_connect_timeout = 15
|
||||
server_login_retry = 15
|
||||
|
||||
; Query timeouts
|
||||
query_timeout = 0
|
||||
query_wait_timeout = 120
|
||||
client_idle_timeout = 0
|
||||
idle_transaction_timeout = 0
|
||||
|
||||
; Logging
|
||||
log_connections = 1
|
||||
log_disconnections = 1
|
||||
log_pooler_errors = 1
|
||||
stats_period = 60
|
||||
|
||||
; This prevents errors when server connection is lost
|
||||
server_reset_query = DISCARD ALL
|
||||
server_reset_query_always = 0
|
||||
|
||||
; Ignore startup parameters
|
||||
ignore_startup_parameters = extra_float_digits
|
||||
|
||||
; Application name
|
||||
application_name_add_host = 1
|
||||
|
||||
; DNS settings
|
||||
dns_max_ttl = 15
|
||||
dns_nxdomain_ttl = 15
|
||||
dns_zone_check_period = 0
|
||||
|
||||
; TLS/SSL settings (disabled by default for internal network)
|
||||
;client_tls_sslmode = disable
|
||||
;server_tls_sslmode = disable
|
||||
|
||||
; Performance tuning
|
||||
pkt_buf = 4096
|
||||
listen_backlog = 128
|
||||
sbuf_loopcnt = 5
|
||||
suspend_timeout = 10
|
||||
tcp_keepalive = 1
|
||||
tcp_keepidle = 0
|
||||
tcp_keepintvl = 0
|
||||
tcp_keepcnt = 0
|
||||
tcp_user_timeout = 0
|
||||
12
pgbouncer/userlist.txt.template
Normal file
12
pgbouncer/userlist.txt.template
Normal file
@@ -0,0 +1,12 @@
|
||||
; PgBouncer user list file (TEMPLATE)
|
||||
; This file is automatically generated by entrypoint.sh
|
||||
; Format: "username" "password"
|
||||
; Password can be plaintext or md5 hash
|
||||
; md5 format: "md5" + md5(password + username)
|
||||
;
|
||||
; DO NOT EDIT THIS TEMPLATE - it will be overwritten
|
||||
; Configure credentials via environment variables:
|
||||
; DB_USER - database username
|
||||
; DB_PASSWORD - database password
|
||||
; PGBOUNCER_ADMIN_USER - admin username (optional)
|
||||
; PGBOUNCER_ADMIN_PASSWORD - admin password (optional)
|
||||
Reference in New Issue
Block a user