Implement PgBouncer connection pooling with monitoring and graceful shutdown (#128)

* Initial plan

* feat: implement PgBouncer connection pooling and monitoring

- Add PgBouncer service with transaction mode pooling
- Configure optimal pool sizes (25 default, 100 max clients)
- Update docker-compose for dev and prod environments
- Enhance Prisma client with connection metrics
- Improve Redis connection lifecycle management
- Add comprehensive connection pool monitoring utilities
- Create monitoring endpoints for health checks and metrics
- Add graceful shutdown handlers for all services
- Update environment configuration with PgBouncer settings

Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com>

* feat: add comprehensive documentation and tests for connection pooling

- Create comprehensive CONNECTION_POOLING.md documentation
- Add PgBouncer setup guide in docs/
- Create unit tests for connection pool monitoring
- Add integration tests for monitoring endpoints
- Update .gitignore for pgbouncer credentials

Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com>

* docs: finalize connection pooling documentation and README updates

- Add comprehensive pgbouncer/README.md with usage guide
- Update main README with connection pooling section
- Add references to all relevant documentation
- Complete documentation checklist

Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com>

* fix: address code review feedback on connection pooling

- Fix PgBouncer Dockerfile permissions by running entrypoint as root with su-exec
- Add PGPASSWORD to health check for proper authentication
- Fix isHealthy check to not treat 0% utilization as unhealthy
- Use proper URL parsing for isPgBouncer detection instead of string matching
- Remove duplicate SIGTERM/SIGINT handlers in redis.ts, coordinate through db.ts
- Add su-exec package for secure user switching in container

Co-authored-by: PatrickFanella <61631520+PatrickFanella@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: onnwee <211922112+onnwee@users.noreply.github.com>
Co-authored-by: PatrickFanella <61631520+PatrickFanella@users.noreply.github.com>
This commit was merged in pull request #128.
This commit is contained in:
Copilot
2025-10-29 23:42:18 -05:00
committed by GitHub
parent 1f23b39254
commit 9ef56ada2e
18 changed files with 2159 additions and 25 deletions

View File

@@ -17,11 +17,20 @@
# -----------------------------------------------------------------------------
# PostgreSQL connection string
# Format: postgresql://username:password@host:port/database?schema=public
#
# For production with PgBouncer (recommended):
# DATABASE_URL=postgresql://postgres:password@pgbouncer:6432/spywatcher?pgbouncer=true
#
# For direct connection (development/migrations):
# DATABASE_URL=postgresql://postgres:password@localhost:5432/spywatcher?schema=public
DATABASE_URL=postgresql://postgres:password@localhost:5432/spywatcher?schema=public
# Database password (if using separate credential management)
DB_PASSWORD=your_secure_database_password
# PgBouncer admin password (optional, for monitoring)
PGBOUNCER_ADMIN_PASSWORD=your_secure_pgbouncer_admin_password
# -----------------------------------------------------------------------------
# Backend Environment Variables
# -----------------------------------------------------------------------------

596
CONNECTION_POOLING.md Normal file
View File

@@ -0,0 +1,596 @@
# Connection Pooling & Resource Management
This document describes the connection pooling and database resource management implementation for Discord Spywatcher.
## 📋 Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [PgBouncer Configuration](#pgbouncer-configuration)
- [Prisma Connection Pool](#prisma-connection-pool)
- [Connection Lifecycle](#connection-lifecycle)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)
## 🎯 Overview
The application implements a multi-layered connection pooling strategy:
1. **PgBouncer** - External connection pooler for PostgreSQL
2. **Prisma Client** - Application-level connection management
3. **Redis** - Connection pooling for cache/rate limiting
### Key Features
- ✅ Transaction-mode connection pooling via PgBouncer
- ✅ Optimized Prisma connection pool settings
- ✅ Graceful shutdown with proper cleanup
- ✅ Connection pool monitoring and metrics
- ✅ Health checks for database and Redis
- ✅ Automatic connection leak prevention
## 🏗️ Architecture
```
Application Layer (Multiple Instances)
Prisma Client (1-5 connections each)
PgBouncer (Connection Pooler)
- Pool Size: 25 connections
- Mode: Transaction
- Max Clients: 100
PostgreSQL Database
- Max Connections: 100
```
### Why This Architecture?
- **PgBouncer** manages a pool of persistent connections to PostgreSQL
- **Transaction mode** allows multiple clients to share connections between transactions
- **Prisma** uses fewer connections since PgBouncer handles pooling
- **Scalable** - can run multiple application instances without exhausting connections
## 🔧 PgBouncer Configuration
### Configuration File
Location: `pgbouncer/pgbouncer.ini`
#### Key Settings
```ini
# Pooling mode - transaction is optimal for Prisma
pool_mode = transaction
# Connection limits
default_pool_size = 25 # Connections per database
min_pool_size = 5 # Minimum connections to maintain
reserve_pool_size = 5 # Additional connections for spikes
max_client_conn = 100 # Maximum client connections
# Timeouts
server_lifetime = 3600 # Connection lifetime (1 hour)
server_idle_timeout = 600 # Idle timeout (10 minutes)
query_wait_timeout = 120 # Query wait timeout (2 minutes)
# Reset query to clean connection state
server_reset_query = DISCARD ALL
```
### Pool Modes Explained
| Mode | Description | Use Case |
|------|-------------|----------|
| **session** | One server connection per client | Long-running sessions, advisory locks |
| **transaction** | One server connection per transaction | Most applications (recommended for Prisma) |
| **statement** | One server connection per statement | Stateless applications only |
**We use `transaction` mode** because:
- Compatible with Prisma's transaction handling
- Efficient connection reuse
- Balances performance and compatibility
### Docker Setup
#### Development
```yaml
pgbouncer:
build:
context: ./pgbouncer
environment:
DB_USER: spywatcher
DB_PASSWORD: ${DB_PASSWORD}
ports:
- "6432:6432"
```
#### Production
```yaml
pgbouncer:
build:
context: ./pgbouncer
environment:
DB_USER: spywatcher
DB_PASSWORD: ${DB_PASSWORD}
restart: unless-stopped
# Note: No external port exposure in production
```
### Environment Variables
```bash
# Application connects through PgBouncer
DATABASE_URL=postgresql://user:password@pgbouncer:6432/spywatcher?pgbouncer=true
# Migrations connect directly to PostgreSQL
DATABASE_URL_DIRECT=postgresql://user:password@postgres:5432/spywatcher
# PgBouncer admin credentials (optional)
PGBOUNCER_ADMIN_PASSWORD=secure_password
```
## 💎 Prisma Connection Pool
### Configuration
When using PgBouncer, Prisma needs fewer connections:
```typescript
const db = new PrismaClient({
datasources: {
db: {
url: process.env.DATABASE_URL,
},
},
});
```
### Connection URL Parameters
#### With PgBouncer (Production)
```
postgresql://user:password@pgbouncer:6432/dbname?pgbouncer=true
```
- Keep connection pool small (Prisma default: 5)
- PgBouncer handles the actual pooling
#### Direct Connection (Development/Migrations)
```
postgresql://user:password@postgres:5432/dbname?connection_limit=20&pool_timeout=20
```
- `connection_limit`: 10-50 depending on load
- `pool_timeout`: 20 seconds
- `connect_timeout`: 10 seconds
### Why Fewer Connections with PgBouncer?
Without PgBouncer:
```
Application → PostgreSQL (need many connections)
```
With PgBouncer:
```
Application → PgBouncer → PostgreSQL (PgBouncer reuses connections)
```
Example with 10 application instances:
- **Without PgBouncer**: 10 × 20 = 200 PostgreSQL connections needed
- **With PgBouncer**: 10 × 5 = 50 client connections → 25 PostgreSQL connections
## 🔄 Connection Lifecycle
### Application Startup
1. **Database Connection**
```typescript
// db.ts initializes Prisma Client
export const db = new PrismaClient({ ... });
```
2. **Redis Connection** (if enabled)
```typescript
// redis.ts initializes Redis client
const redisClient = new Redis(url, { ... });
```
3. **Health Checks**
- Database connectivity verification
- Connection pool metrics collection
### During Operation
- **Connection Reuse**: PgBouncer reuses connections between transactions
- **Pool Monitoring**: Metrics collected every 60 seconds
- **Auto-reconnect**: Redis automatically reconnects on connection loss
### Graceful Shutdown
```typescript
// Signal handlers in db.ts and redis.ts
process.on('SIGTERM', async () => {
// 1. Stop accepting new connections
// 2. Wait for in-flight requests
// 3. Close Prisma connections
await db.$disconnect();
// 4. Close Redis connections
await closeRedisConnection();
// 5. Exit process
process.exit(0);
});
```
#### Shutdown Sequence
1. **Receive SIGTERM/SIGINT**
2. **Set shutdown flag** - prevents new operations
3. **Disconnect Prisma** - closes all connections gracefully
4. **Disconnect Redis** - uses QUIT command
5. **Exit process**
### Connection Leak Prevention
- **Singleton pattern** for database client
- **Proper error handling** ensures connections are released
- **Transaction timeouts** prevent hung connections
- **Monitoring alerts** for connection pool saturation
## 📊 Monitoring
### Health Check Endpoints
#### System Health
```bash
GET /api/admin/monitoring/connections/health
```
Returns:
```json
{
"healthy": true,
"timestamp": "2025-01-15T10:30:00Z",
"database": {
"healthy": true,
"responseTime": 12,
"connectionPool": {
"active": 3,
"idle": 2,
"total": 5,
"max": 100,
"utilizationPercent": "5.00",
"isPgBouncer": true,
"isShuttingDown": false
}
},
"redis": {
"available": true,
"connected": true,
"status": "ready"
}
}
```
#### Connection Pool Stats
```bash
GET /api/admin/monitoring/connections/pool
```
Returns:
```json
{
"database": {
"utilizationPercent": 5.0,
"activeConnections": 3,
"maxConnections": 100,
"isHealthy": true
},
"redis": {
"available": true,
"connected": true
}
}
```
#### Connection Alerts
```bash
GET /api/admin/monitoring/connections/alerts
```
Returns:
```json
{
"alerts": [
"WARNING: Database connection pool at 85% utilization"
],
"count": 1,
"timestamp": "2025-01-15T10:30:00Z"
}
```
### PgBouncer Statistics
Connect to PgBouncer admin interface:
```bash
psql -h localhost -p 6432 -U pgbouncer_admin pgbouncer
```
Useful commands:
```sql
-- Show pool statistics
SHOW POOLS;
-- Show database statistics
SHOW DATABASES;
-- Show client connections
SHOW CLIENTS;
-- Show server connections
SHOW SERVERS;
-- Show configuration
SHOW CONFIG;
-- Show statistics
SHOW STATS;
```
### Automated Monitoring
The application logs connection pool metrics every 60 seconds:
```
=== Connection Pool Metrics ===
Timestamp: 2025-01-15T10:30:00Z
Overall Health: ✅ HEALTHY
--- Database ---
Health: ✅
Response Time: 12ms
Connection Pool:
Active: 3
Idle: 2
Total: 5
Max: 100
Utilization: 5.00%
PgBouncer: Yes
--- Redis ---
Available: ✅
Connected: ✅
Status: ready
Metrics:
Total Connections: 125
Total Commands: 45678
Ops/sec: 23
Memory Used: 2.5MB
==============================
```
## 🔍 Troubleshooting
### Issue: Too many connections
**Symptoms:**
```
Error: remaining connection slots are reserved for non-replication superuser connections
```
**Solutions:**
1. **Check PgBouncer pool size:**
```bash
# In pgbouncer.ini
default_pool_size = 25 # Increase if needed
max_db_connections = 50
```
2. **Check PostgreSQL max_connections:**
```sql
SHOW max_connections; -- Should be > PgBouncer pool size
```
3. **Monitor connection usage:**
```bash
curl http://localhost:3001/api/admin/monitoring/connections/pool
```
### Issue: Connection timeouts
**Symptoms:**
```
Error: Connection timeout
```
**Solutions:**
1. **Check PgBouncer is running:**
```bash
docker ps | grep pgbouncer
```
2. **Check connection string:**
```bash
# Ensure using correct host and port
DATABASE_URL=postgresql://user:pass@pgbouncer:6432/db?pgbouncer=true
```
3. **Increase timeouts:**
```ini
# In pgbouncer.ini
query_wait_timeout = 120
server_connect_timeout = 15
```
### Issue: Slow queries with PgBouncer
**Symptoms:**
- Queries slower than without PgBouncer
**Solutions:**
1. **Ensure using transaction mode:**
```ini
pool_mode = transaction # Not session mode
```
2. **Check for connection reuse:**
```sql
-- In PgBouncer admin
SHOW POOLS;
-- Check cl_active, cl_waiting, sv_active, sv_idle
```
3. **Monitor query wait time:**
```bash
curl http://localhost:3001/api/admin/monitoring/database/slow-queries
```
### Issue: Migrations fail with PgBouncer
**Symptoms:**
```
Error: prepared statement already exists
```
**Solution:**
Always run migrations with direct PostgreSQL connection:
```bash
# Use DATABASE_URL_DIRECT for migrations
DATABASE_URL=$DATABASE_URL_DIRECT npx prisma migrate deploy
```
Or configure in docker-compose.yml:
```yaml
migrate:
environment:
DATABASE_URL: postgresql://user:pass@postgres:5432/db # Direct connection
```
### Issue: Connection pool exhaustion
**Symptoms:**
- "Pool is full" errors
- High connection utilization
**Solutions:**
1. **Scale PgBouncer pool:**
```ini
default_pool_size = 50 # Increase from 25
reserve_pool_size = 10 # Increase reserve
```
2. **Add connection cleanup:**
```typescript
// Ensure proper $disconnect() on errors
try {
await db.query();
} finally {
// Connections released automatically
}
```
3. **Reduce connection limit per instance:**
```
# Fewer connections per app instance
DATABASE_URL=...?connection_limit=3
```
## ✅ Best Practices
### Production Deployment
1. **Always use PgBouncer in production**
- Better connection management
- Prevents connection exhaustion
- Enables horizontal scaling
2. **Configure appropriate pool sizes**
```
PgBouncer pool: 25-50 connections
Prisma per instance: 3-5 connections
PostgreSQL max: 100+ connections
```
3. **Use separate connections for migrations**
- Migrations need direct PostgreSQL access
- Bypass PgBouncer for schema changes
4. **Monitor connection metrics**
- Set up alerts for >80% utilization
- Track connection pool trends
- Monitor slow query counts
### Development Practices
1. **Test with and without PgBouncer**
- Dev: direct connection (easier debugging)
- Staging/Prod: through PgBouncer
2. **Use environment-specific configs**
```bash
# .env.development
DATABASE_URL=postgresql://...@postgres:5432/db
# .env.production
DATABASE_URL=postgresql://...@pgbouncer:6432/db?pgbouncer=true
```
3. **Implement proper error handling**
```typescript
try {
await db.query();
} catch (error) {
// Log error
// Connection automatically released
throw error;
}
```
4. **Use connection pooling metrics**
- Monitor during load tests
- Adjust pool sizes based on metrics
- Set up automated alerts
### Security Considerations
1. **Secure PgBouncer credentials**
- Use strong passwords
- Rotate credentials regularly
- Use environment variables
2. **Limit PgBouncer access**
- Don't expose port externally
- Use internal Docker network
- Configure firewall rules
3. **Monitor for connection abuse**
- Track connection patterns
- Alert on unusual spikes
- Implement rate limiting
## 📚 Additional Resources
- [PgBouncer Documentation](https://www.pgbouncer.org/)
- [Prisma Connection Pool](https://www.prisma.io/docs/concepts/components/prisma-client/working-with-prismaclient/connection-pool)
- [PostgreSQL Connection Pooling](https://www.postgresql.org/docs/current/runtime-config-connection.html)
- [DATABASE_OPTIMIZATION.md](./DATABASE_OPTIMIZATION.md)
- [POSTGRESQL.md](./POSTGRESQL.md)
## 🆘 Support
For issues or questions:
- Check monitoring endpoints first
- Review logs for error messages
- Consult troubleshooting section
- Check PgBouncer statistics
- Open GitHub issue with details

View File

@@ -208,13 +208,34 @@ Spywatcher implements comprehensive security measures to protect against abuse a
- **IP blocking** with automatic abuse detection
- **Load management** with circuit breakers and load shedding under high load
See [backend/docs/RATE_LIMITING.md](./backend/docs/RATE_LIMITING.md) for detailed documentation on:
See [RATE_LIMITING.md](./RATE_LIMITING.md) for detailed documentation on:
- Rate limiting configuration
- Endpoint-specific limits
- DDoS protection mechanisms
- Load shedding behavior
- Admin APIs for IP management
## 🗄️ Database & Connection Pooling
Spywatcher uses PostgreSQL with PgBouncer for efficient connection pooling and resource management:
- **PgBouncer** - Transaction-mode connection pooling for optimal performance
- **Prisma** - Type-safe database access with connection pool monitoring
- **Redis** - Distributed caching and rate limiting with connection management
- **Graceful shutdown** - Proper connection cleanup on application shutdown
See [CONNECTION_POOLING.md](./CONNECTION_POOLING.md) for detailed documentation on:
- PgBouncer setup and configuration
- Connection pool monitoring and metrics
- Database health checks and alerting
- Performance optimization strategies
- Troubleshooting connection issues
Additional database documentation:
- [POSTGRESQL.md](./POSTGRESQL.md) - PostgreSQL setup and management
- [DATABASE_OPTIMIZATION.md](./DATABASE_OPTIMIZATION.md) - Query optimization and indexing
- [docs/PGBOUNCER_SETUP.md](./docs/PGBOUNCER_SETUP.md) - Quick reference guide
## 🌐 Endpoints
Available at `http://localhost:3001`

View File

@@ -0,0 +1,247 @@
/**
* Integration tests for Connection Monitoring Routes
*/
import express from 'express';
import request from 'supertest';
import monitoringRoutes from '../../../src/routes/monitoring';
// Mock dependencies
jest.mock('../../../src/db', () => ({
db: {
$queryRaw: jest.fn(),
},
checkDatabaseHealth: jest.fn(),
getConnectionPoolMetrics: jest.fn(),
}));
jest.mock('../../../src/utils/redis', () => ({
getRedisClient: jest.fn(() => null),
getRedisMetrics: jest.fn(),
isRedisAvailable: jest.fn(),
}));
jest.mock('../../../src/utils/connectionPoolMonitor', () => ({
getSystemHealth: jest.fn(),
getConnectionPoolStats: jest.fn(),
getConnectionPoolAlerts: jest.fn(),
}));
// Mock authentication middleware
jest.mock('../../../src/middleware/auth', () => ({
requireAuth: (req: any, _res: any, next: any) => {
req.user = { id: 'test-admin', username: 'admin', role: 'ADMIN' };
next();
},
requireRole: () => (_req: any, _res: any, next: any) => next(),
}));
// Mock other dependencies
jest.mock('../../../src/middleware/slowQueryLogger', () => ({
getSlowQueryLogs: jest.fn(() => ({ logs: [], total: 0 })),
getSlowQueryStats: jest.fn(() => ({ totalQueries: 0 })),
clearSlowQueryLogs: jest.fn(),
}));
jest.mock('../../../src/services/cache', () => ({
cache: {
getStats: jest.fn(),
flushAll: jest.fn(),
invalidateByTag: jest.fn(),
},
}));
jest.mock('../../../src/utils/databaseMaintenance', () => ({
checkDatabaseHealth: jest.fn(),
getTableStats: jest.fn(),
getIndexUsageStats: jest.fn(),
getUnusedIndexes: jest.fn(),
getSlowQueries: jest.fn(),
generateMaintenanceReport: jest.fn(),
analyzeAllTables: jest.fn(),
}));
import * as db from '../../../src/db';
import * as connectionPoolMonitor from '../../../src/utils/connectionPoolMonitor';
describe('Connection Monitoring Routes', () => {
let app: express.Application;
beforeEach(() => {
app = express();
app.use(express.json());
app.use('/api/admin/monitoring', monitoringRoutes);
jest.clearAllMocks();
});
describe('GET /connections/health', () => {
it('should return system health status', async () => {
const mockHealth = {
healthy: true,
timestamp: new Date().toISOString(),
database: {
healthy: true,
responseTime: 12,
connectionPool: {
active: 3,
idle: 2,
total: 5,
max: 100,
utilizationPercent: '5.00',
isPgBouncer: true,
isShuttingDown: false,
},
},
redis: {
available: true,
connected: true,
status: 'ready',
},
};
jest.spyOn(connectionPoolMonitor, 'getSystemHealth').mockResolvedValue(
mockHealth
);
const response = await request(app).get(
'/api/admin/monitoring/connections/health'
);
expect(response.status).toBe(200);
expect(response.body.healthy).toBe(true);
expect(response.body.database.healthy).toBe(true);
expect(response.body.database.connectionPool.active).toBe(3);
expect(response.body.redis.available).toBe(true);
});
it('should handle errors gracefully', async () => {
jest.spyOn(connectionPoolMonitor, 'getSystemHealth').mockRejectedValue(
new Error('Database connection failed')
);
const response = await request(app).get(
'/api/admin/monitoring/connections/health'
);
expect(response.status).toBe(500);
expect(response.body.error).toBeDefined();
});
});
describe('GET /connections/pool', () => {
it('should return connection pool statistics', async () => {
const mockStats = {
database: {
utilizationPercent: 15.0,
activeConnections: 10,
maxConnections: 100,
isHealthy: true,
},
redis: {
available: true,
connected: true,
},
};
jest.spyOn(
connectionPoolMonitor,
'getConnectionPoolStats'
).mockResolvedValue(mockStats);
const response = await request(app).get(
'/api/admin/monitoring/connections/pool'
);
expect(response.status).toBe(200);
expect(response.body.database.utilizationPercent).toBe(15.0);
expect(response.body.database.activeConnections).toBe(10);
expect(response.body.redis.available).toBe(true);
});
it('should handle errors gracefully', async () => {
jest.spyOn(
connectionPoolMonitor,
'getConnectionPoolStats'
).mockRejectedValue(new Error('Failed to get stats'));
const response = await request(app).get(
'/api/admin/monitoring/connections/pool'
);
expect(response.status).toBe(500);
expect(response.body.error).toBeDefined();
});
});
describe('GET /connections/alerts', () => {
it('should return connection pool alerts', async () => {
const mockAlerts = [
'WARNING: Database connection pool at 85% utilization',
];
jest.spyOn(
connectionPoolMonitor,
'getConnectionPoolAlerts'
).mockResolvedValue(mockAlerts);
const response = await request(app).get(
'/api/admin/monitoring/connections/alerts'
);
expect(response.status).toBe(200);
expect(response.body.alerts).toHaveLength(1);
expect(response.body.alerts[0]).toContain('WARNING');
expect(response.body.count).toBe(1);
expect(response.body.timestamp).toBeDefined();
});
it('should return empty array when no alerts', async () => {
jest.spyOn(
connectionPoolMonitor,
'getConnectionPoolAlerts'
).mockResolvedValue([]);
const response = await request(app).get(
'/api/admin/monitoring/connections/alerts'
);
expect(response.status).toBe(200);
expect(response.body.alerts).toHaveLength(0);
expect(response.body.count).toBe(0);
});
it('should handle errors gracefully', async () => {
jest.spyOn(
connectionPoolMonitor,
'getConnectionPoolAlerts'
).mockRejectedValue(new Error('Failed to get alerts'));
const response = await request(app).get(
'/api/admin/monitoring/connections/alerts'
);
expect(response.status).toBe(500);
expect(response.body.error).toBeDefined();
});
});
describe('Database health endpoint', () => {
it('should return database health status', async () => {
const mockHealth = {
connected: true,
version: '15.3',
responseTime: 8,
};
jest.spyOn(db, 'checkDatabaseHealth').mockResolvedValue(mockHealth);
const response = await request(app).get(
'/api/admin/monitoring/database/health'
);
expect(response.status).toBe(200);
expect(response.body.connected).toBe(true);
});
});
});

View File

@@ -0,0 +1,290 @@
/**
* Tests for Connection Pool Monitor
*/
import { describe, it, expect, jest, beforeEach } from '@jest/globals';
// Mock dependencies
jest.mock('../../../src/db');
jest.mock('../../../src/utils/redis');
import * as db from '../../../src/db';
import * as redis from '../../../src/utils/redis';
import {
getSystemHealth,
getConnectionPoolStats,
getConnectionPoolAlerts,
isConnectionPoolOverloaded,
} from '../../../src/utils/connectionPoolMonitor';
describe('Connection Pool Monitor', () => {
beforeEach(() => {
jest.clearAllMocks();
});
describe('getSystemHealth', () => {
it('should return healthy status when all systems are operational', async () => {
// Mock database health check
jest.spyOn(db, 'checkDatabaseHealth').mockResolvedValue({
healthy: true,
responseTime: 15,
});
// Mock connection pool metrics
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 3,
idle: 2,
total: 5,
max: 100,
utilizationPercent: '5.00',
isPgBouncer: true,
isShuttingDown: false,
});
// Mock Redis metrics
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
status: 'ready',
totalConnectionsReceived: '100',
totalCommandsProcessed: '5000',
instantaneousOpsPerSec: '25',
usedMemory: '2MB',
});
jest.spyOn(redis, 'isRedisAvailable').mockResolvedValue(true);
const health = await getSystemHealth();
expect(health.healthy).toBe(true);
expect(health.database.healthy).toBe(true);
expect(health.database.responseTime).toBe(15);
expect(health.database.connectionPool?.active).toBe(3);
expect(health.database.connectionPool?.isPgBouncer).toBe(true);
expect(health.redis.available).toBe(true);
expect(health.redis.connected).toBe(true);
});
it('should return unhealthy status when database is down', async () => {
jest.spyOn(db, 'checkDatabaseHealth').mockResolvedValue({
healthy: false,
error: 'Connection refused',
});
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 0,
idle: 0,
total: 0,
max: 100,
utilizationPercent: '0',
isPgBouncer: false,
isShuttingDown: false,
error: 'Connection refused',
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
jest.spyOn(redis, 'isRedisAvailable').mockResolvedValue(true);
const health = await getSystemHealth();
expect(health.healthy).toBe(false);
expect(health.database.healthy).toBe(false);
expect(health.database.error).toBeDefined();
});
});
describe('getConnectionPoolStats', () => {
it('should return connection pool statistics', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 10,
idle: 5,
total: 15,
max: 100,
utilizationPercent: '15.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
const stats = await getConnectionPoolStats();
expect(stats.database.utilizationPercent).toBe(15.0);
expect(stats.database.activeConnections).toBe(10);
expect(stats.database.maxConnections).toBe(100);
expect(stats.database.isHealthy).toBe(true);
expect(stats.redis.available).toBe(true);
});
it('should handle database errors gracefully', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 0,
idle: 0,
total: 0,
max: 0,
utilizationPercent: '0',
isPgBouncer: false,
isShuttingDown: false,
error: 'Database error',
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: false,
connected: false,
});
const stats = await getConnectionPoolStats();
expect(stats.database.isHealthy).toBe(false);
expect(stats.redis.available).toBe(false);
});
});
describe('isConnectionPoolOverloaded', () => {
it('should return true when utilization exceeds threshold', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 85,
idle: 5,
total: 90,
max: 100,
utilizationPercent: '90.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
const isOverloaded = await isConnectionPoolOverloaded(80);
expect(isOverloaded).toBe(true);
});
it('should return false when utilization is below threshold', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 50,
idle: 20,
total: 70,
max: 100,
utilizationPercent: '70.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
const isOverloaded = await isConnectionPoolOverloaded(80);
expect(isOverloaded).toBe(false);
});
});
describe('getConnectionPoolAlerts', () => {
it('should return critical alert when utilization >= 90%', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 90,
idle: 5,
total: 95,
max: 100,
utilizationPercent: '95.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
const alerts = await getConnectionPoolAlerts();
expect(alerts).toHaveLength(1);
expect(alerts[0]).toContain('CRITICAL');
expect(alerts[0]).toContain('95');
});
it('should return warning alert when utilization >= 80%', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 82,
idle: 8,
total: 85,
max: 100,
utilizationPercent: '85.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
const alerts = await getConnectionPoolAlerts();
expect(alerts).toHaveLength(1);
expect(alerts[0]).toContain('WARNING');
expect(alerts[0]).toContain('85');
});
it('should return no alerts when utilization is healthy', async () => {
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 20,
idle: 10,
total: 30,
max: 100,
utilizationPercent: '30.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: true,
connected: true,
});
const alerts = await getConnectionPoolAlerts();
expect(alerts).toHaveLength(0);
});
it('should include Redis alerts when configured but unavailable', async () => {
// Set Redis URL environment variable
process.env.REDIS_URL = 'redis://localhost:6379';
jest.spyOn(db, 'getConnectionPoolMetrics').mockResolvedValue({
active: 10,
idle: 5,
total: 15,
max: 100,
utilizationPercent: '15.00',
isPgBouncer: true,
isShuttingDown: false,
});
jest.spyOn(redis, 'getRedisMetrics').mockResolvedValue({
available: false,
connected: false,
});
const alerts = await getConnectionPoolAlerts();
const redisAlerts = alerts.filter((a) => a.includes('Redis'));
expect(redisAlerts.length).toBeGreaterThan(0);
// Clean up
delete process.env.REDIS_URL;
});
});
});

View File

@@ -3,13 +3,31 @@ import { PrismaClient } from '@prisma/client';
// Global singleton for Prisma Client to prevent multiple instances
const globalForPrisma = global as unknown as { prisma: PrismaClient };
// Determine if using PgBouncer based on connection string
const isPgBouncer = (() => {
try {
const url = new URL(process.env.DATABASE_URL ?? '');
return url.searchParams.get('pgbouncer') === 'true';
} catch {
return false;
}
})();
// Configure connection pooling and logging based on environment
// When using PgBouncer (transaction pooling mode):
// - Keep connection_limit low (1-5) since PgBouncer handles pooling
// - Disable interactive transactions for better compatibility
export const db =
globalForPrisma.prisma ||
new PrismaClient({
log: process.env.NODE_ENV === 'development'
? ['query', 'error', 'warn']
: ['error'],
datasources: {
db: {
url: process.env.DATABASE_URL,
},
},
});
// Prevent multiple instances in development (hot reload)
@@ -21,14 +39,112 @@ if (process.env.NODE_ENV !== 'production') {
// This must be done after the db instance is created but before queries run
// The initialization is done in the application entry point (server.ts or index.ts)
// Connection pool status tracking
let isShuttingDown = false;
/**
* Get connection pool metrics
*/
export async function getConnectionPoolMetrics() {
try {
// Query for active connections
const activeConnections = await db.$queryRaw<Array<{ count: bigint }>>`
SELECT COUNT(*) as count
FROM pg_stat_activity
WHERE datname = current_database()
AND state = 'active'
`;
// Query for idle connections
const idleConnections = await db.$queryRaw<Array<{ count: bigint }>>`
SELECT COUNT(*) as count
FROM pg_stat_activity
WHERE datname = current_database()
AND state = 'idle'
`;
// Query for total connections
const totalConnections = await db.$queryRaw<Array<{ count: bigint }>>`
SELECT COUNT(*) as count
FROM pg_stat_activity
WHERE datname = current_database()
`;
// Get max connections setting
const maxConnections = await db.$queryRaw<Array<{ setting: string }>>`
SELECT setting
FROM pg_settings
WHERE name = 'max_connections'
`;
return {
active: Number(activeConnections[0]?.count ?? 0),
idle: Number(idleConnections[0]?.count ?? 0),
total: Number(totalConnections[0]?.count ?? 0),
max: Number(maxConnections[0]?.setting ?? 0),
utilizationPercent: maxConnections[0]?.setting
? ((Number(totalConnections[0]?.count ?? 0) / Number(maxConnections[0].setting)) * 100).toFixed(2)
: '0',
isPgBouncer,
isShuttingDown,
};
} catch (error) {
console.error('Error fetching connection pool metrics:', error);
return {
active: 0,
idle: 0,
total: 0,
max: 0,
utilizationPercent: '0',
isPgBouncer,
isShuttingDown,
error: error instanceof Error ? error.message : 'Unknown error',
};
}
}
/**
* Health check for database connection
*/
export async function checkDatabaseHealth(): Promise<{
healthy: boolean;
responseTime?: number;
error?: string;
}> {
const startTime = Date.now();
try {
await db.$queryRaw`SELECT 1`;
const responseTime = Date.now() - startTime;
return { healthy: true, responseTime };
} catch (error) {
return {
healthy: false,
error: error instanceof Error ? error.message : 'Unknown error',
};
}
}
// Graceful shutdown handlers
const gracefulShutdown = async (signal: string) => {
console.log(`Received ${signal}, closing database connections...`);
if (isShuttingDown) {
console.log('Shutdown already in progress...');
return;
}
isShuttingDown = true;
console.log(`Received ${signal}, initiating graceful shutdown...`);
try {
// Close database connections
console.log('Closing database connections...');
await db.$disconnect();
console.log('Database connections closed.');
console.log('Database connections closed successfully');
// Close Redis connections
const { closeRedisConnection } = await import('./utils/redis');
await closeRedisConnection();
} catch (error) {
console.error('Error during database disconnect:', error);
console.error('Error during graceful shutdown:', error);
} finally {
process.exit(0);
}
@@ -39,22 +155,47 @@ process.on('SIGINT', () => gracefulShutdown('SIGINT'));
// Handle uncaught exceptions and unhandled rejections
process.on('uncaughtException', async (err) => {
console.error('Uncaught Exception:', err);
await db.$disconnect();
console.error('Uncaught Exception:', err);
if (!isShuttingDown) {
isShuttingDown = true;
try {
await db.$disconnect();
const { closeRedisConnection } = await import('./utils/redis');
await closeRedisConnection();
} catch (disconnectError) {
console.error('Error during emergency disconnect:', disconnectError);
}
}
process.exit(1);
});
process.on('unhandledRejection', async (reason) => {
console.error('Unhandled Rejection:', reason);
await db.$disconnect();
console.error('Unhandled Rejection:', reason);
if (!isShuttingDown) {
isShuttingDown = true;
try {
await db.$disconnect();
const { closeRedisConnection } = await import('./utils/redis');
await closeRedisConnection();
} catch (disconnectError) {
console.error('Error during emergency disconnect:', disconnectError);
}
}
process.exit(1);
});
// Connection pooling is configured via DATABASE_URL query parameters:
// postgresql://user:password@localhost:5432/spywatcher?connection_limit=10&pool_timeout=20
//
// Recommended settings:
// Connection pooling configuration:
//
// When using PgBouncer (recommended for production):
// DATABASE_URL=postgresql://user:password@pgbouncer:6432/dbname?pgbouncer=true
// - PgBouncer handles connection pooling at the server level
// - Configure PgBouncer pool_mode=transaction for best compatibility with Prisma
// - Keep Prisma connection_limit low (1-5) as PgBouncer manages the pool
// - PgBouncer configuration in pgbouncer/pgbouncer.ini
//
// When connecting directly to PostgreSQL:
// DATABASE_URL=postgresql://user:password@postgres:5432/dbname?connection_limit=20&pool_timeout=20&connect_timeout=10
// - connection_limit: 10-20 for small apps, 20-50 for medium apps
// - pool_timeout: 20 seconds
// - connect_timeout: 10 seconds
// - pool_timeout: 20 seconds (time to wait for available connection)
// - connect_timeout: 10 seconds (time to wait for initial connection)

View File

@@ -17,6 +17,11 @@ import {
analyzeAllTables,
} from '../utils/databaseMaintenance';
import { getRedisClient } from '../utils/redis';
import {
getSystemHealth,
getConnectionPoolStats,
getConnectionPoolAlerts,
} from '../utils/connectionPoolMonitor';
const router = Router();
const redis = getRedisClient();
@@ -301,6 +306,58 @@ router.get('/database/health', async (_req: Request, res: Response) => {
}
});
/**
* GET /api/admin/monitoring/connections/health
* Get comprehensive system health including connection pools
*/
router.get('/connections/health', async (_req: Request, res: Response) => {
try {
const health = await getSystemHealth();
res.json(health);
} catch (error) {
console.error('Failed to get system health:', error);
res.status(500).json({
error: 'Failed to retrieve system health',
});
}
});
/**
* GET /api/admin/monitoring/connections/pool
* Get connection pool statistics
*/
router.get('/connections/pool', async (_req: Request, res: Response) => {
try {
const stats = await getConnectionPoolStats();
res.json(stats);
} catch (error) {
console.error('Failed to get connection pool stats:', error);
res.status(500).json({
error: 'Failed to retrieve connection pool statistics',
});
}
});
/**
* GET /api/admin/monitoring/connections/alerts
* Get connection pool alerts and warnings
*/
router.get('/connections/alerts', async (_req: Request, res: Response) => {
try {
const alerts = await getConnectionPoolAlerts();
res.json({
alerts,
count: alerts.length,
timestamp: new Date().toISOString(),
});
} catch (error) {
console.error('Failed to get connection pool alerts:', error);
res.status(500).json({
error: 'Failed to retrieve connection pool alerts',
});
}
});
/**
* GET /api/admin/monitoring/database/tables
* Get table statistics (sizes, row counts, etc.)

View File

@@ -0,0 +1,279 @@
/**
* Connection Pool Monitoring Utility
*
* Provides monitoring and health check capabilities for database
* and Redis connection pools.
*/
import { checkDatabaseHealth, getConnectionPoolMetrics } from '../db';
import { getRedisMetrics, isRedisAvailable } from './redis';
/**
* Overall system health status
*/
export interface SystemHealth {
healthy: boolean;
timestamp: string;
database: DatabaseHealth;
redis: RedisHealth;
}
/**
* Database connection health
*/
export interface DatabaseHealth {
healthy: boolean;
responseTime?: number;
connectionPool?: {
active: number;
idle: number;
total: number;
max: number;
utilizationPercent: string;
isPgBouncer: boolean;
isShuttingDown: boolean;
};
error?: string;
}
/**
* Redis connection health
*/
export interface RedisHealth {
available: boolean;
connected: boolean;
status?: string;
metrics?: {
totalConnectionsReceived: string;
totalCommandsProcessed: string;
instantaneousOpsPerSec: string;
usedMemory: string;
};
error?: string;
}
/**
* Connection pool statistics for alerting
*/
export interface ConnectionPoolStats {
database: {
utilizationPercent: number;
activeConnections: number;
maxConnections: number;
isHealthy: boolean;
};
redis: {
available: boolean;
connected: boolean;
};
}
/**
* Get comprehensive system health including connection pools
*/
export async function getSystemHealth(): Promise<SystemHealth> {
const [dbHealth, dbMetrics, redisMetrics] = await Promise.all([
checkDatabaseHealth(),
getConnectionPoolMetrics(),
getRedisMetrics(),
]);
const redisAvailable = await isRedisAvailable();
return {
healthy: dbHealth.healthy && (!process.env.REDIS_URL || redisAvailable),
timestamp: new Date().toISOString(),
database: {
healthy: dbHealth.healthy,
responseTime: dbHealth.responseTime,
connectionPool: dbMetrics.error
? undefined
: {
active: dbMetrics.active,
idle: dbMetrics.idle,
total: dbMetrics.total,
max: dbMetrics.max,
utilizationPercent: dbMetrics.utilizationPercent,
isPgBouncer: dbMetrics.isPgBouncer,
isShuttingDown: dbMetrics.isShuttingDown,
},
error: dbHealth.error || dbMetrics.error,
},
redis: {
available: redisMetrics.available,
connected: redisMetrics.connected,
status: redisMetrics.status,
metrics: redisMetrics.available && redisMetrics.connected
? {
totalConnectionsReceived: redisMetrics.totalConnectionsReceived ?? 'N/A',
totalCommandsProcessed: redisMetrics.totalCommandsProcessed ?? 'N/A',
instantaneousOpsPerSec: redisMetrics.instantaneousOpsPerSec ?? 'N/A',
usedMemory: redisMetrics.usedMemory ?? 'N/A',
}
: undefined,
error: redisMetrics.error,
},
};
}
/**
* Get connection pool statistics for alerting purposes
*/
export async function getConnectionPoolStats(): Promise<ConnectionPoolStats> {
const [dbMetrics, redisMetrics] = await Promise.all([
getConnectionPoolMetrics(),
getRedisMetrics(),
]);
return {
database: {
utilizationPercent: parseFloat(dbMetrics.utilizationPercent),
activeConnections: dbMetrics.active,
maxConnections: dbMetrics.max,
isHealthy: !dbMetrics.error,
},
redis: {
available: redisMetrics.available,
connected: redisMetrics.connected,
},
};
}
/**
* Check if connection pool utilization is above threshold
* @param threshold - Percentage threshold (default 80%)
*/
export async function isConnectionPoolOverloaded(
threshold: number = 80
): Promise<boolean> {
const stats = await getConnectionPoolStats();
return stats.database.utilizationPercent >= threshold;
}
/**
* Get connection pool alerts
*/
export async function getConnectionPoolAlerts(): Promise<string[]> {
const alerts: string[] = [];
const stats = await getConnectionPoolStats();
// Database connection pool alerts
if (stats.database.utilizationPercent >= 90) {
alerts.push(
`CRITICAL: Database connection pool at ${stats.database.utilizationPercent}% utilization`
);
} else if (stats.database.utilizationPercent >= 80) {
alerts.push(
`WARNING: Database connection pool at ${stats.database.utilizationPercent}% utilization`
);
}
if (!stats.database.isHealthy) {
alerts.push('CRITICAL: Database connection pool is not healthy');
}
// Redis connection alerts
if (process.env.REDIS_URL && !stats.redis.available) {
alerts.push('WARNING: Redis is configured but not available');
}
if (process.env.REDIS_URL && !stats.redis.connected) {
alerts.push('WARNING: Redis is not connected');
}
return alerts;
}
/**
* Log connection pool metrics to console (for monitoring/debugging)
*/
export async function logConnectionPoolMetrics(): Promise<void> {
const health = await getSystemHealth();
console.log('=== Connection Pool Metrics ===');
console.log(`Timestamp: ${health.timestamp}`);
console.log(`Overall Health: ${health.healthy ? '✅ HEALTHY' : '❌ UNHEALTHY'}`);
console.log('\n--- Database ---');
console.log(`Health: ${health.database.healthy ? '✅' : '❌'}`);
if (health.database.responseTime) {
console.log(`Response Time: ${health.database.responseTime}ms`);
}
if (health.database.connectionPool) {
console.log(`Connection Pool:`);
console.log(` Active: ${health.database.connectionPool.active}`);
console.log(` Idle: ${health.database.connectionPool.idle}`);
console.log(` Total: ${health.database.connectionPool.total}`);
console.log(` Max: ${health.database.connectionPool.max}`);
console.log(
` Utilization: ${health.database.connectionPool.utilizationPercent}%`
);
console.log(
` PgBouncer: ${health.database.connectionPool.isPgBouncer ? 'Yes' : 'No'}`
);
}
if (health.database.error) {
console.log(`Error: ${health.database.error}`);
}
console.log('\n--- Redis ---');
console.log(`Available: ${health.redis.available ? '✅' : '❌'}`);
console.log(`Connected: ${health.redis.connected ? '✅' : '❌'}`);
if (health.redis.status) {
console.log(`Status: ${health.redis.status}`);
}
if (health.redis.metrics) {
console.log(`Metrics:`);
console.log(
` Total Connections: ${health.redis.metrics.totalConnectionsReceived}`
);
console.log(
` Total Commands: ${health.redis.metrics.totalCommandsProcessed}`
);
console.log(` Ops/sec: ${health.redis.metrics.instantaneousOpsPerSec}`);
console.log(` Memory Used: ${health.redis.metrics.usedMemory}`);
}
if (health.redis.error) {
console.log(`Error: ${health.redis.error}`);
}
console.log('==============================\n');
// Log any alerts
const alerts = await getConnectionPoolAlerts();
if (alerts.length > 0) {
console.log('🚨 ALERTS:');
alerts.forEach((alert) => console.log(` - ${alert}`));
console.log('');
}
}
/**
* Start periodic connection pool monitoring
* @param intervalMs - Monitoring interval in milliseconds (default: 60000 = 1 minute)
*/
export function startConnectionPoolMonitoring(intervalMs: number = 60000): NodeJS.Timeout {
console.log(
`Starting connection pool monitoring (interval: ${intervalMs}ms)`
);
// Log initial metrics
logConnectionPoolMetrics().catch((err) => {
console.error('Error logging initial connection pool metrics:', err);
});
// Start periodic monitoring
return setInterval(() => {
logConnectionPoolMetrics().catch((err) => {
console.error('Error logging connection pool metrics:', err);
});
}, intervalMs);
}
/**
* Stop periodic connection pool monitoring
*/
export function stopConnectionPoolMonitoring(timer: NodeJS.Timeout): void {
clearInterval(timer);
console.log('Connection pool monitoring stopped');
}

View File

@@ -6,6 +6,7 @@ import { env } from './env';
* Redis client instance for rate limiting and caching
*/
let redisClient: Redis | null = null;
let isShuttingDown = false;
/**
* Initialize and return Redis client
@@ -27,14 +28,22 @@ export function getRedisClient(): Redis | null {
maxRetriesPerRequest: 3,
enableReadyCheck: true,
retryStrategy(times: number) {
// Don't retry during shutdown
if (isShuttingDown) {
return null;
}
const delay = Math.min(times * 50, 2000);
return delay;
},
lazyConnect: false,
// Connection pool settings for better performance
enableOfflineQueue: true,
connectTimeout: 10000,
keepAlive: 30000,
});
redisClient.on('error', (err: Error) => {
console.error('Redis connection error:', err);
console.error('Redis connection error:', err);
});
redisClient.on('connect', () => {
@@ -45,6 +54,16 @@ export function getRedisClient(): Redis | null {
console.log('✅ Redis ready for operations');
});
redisClient.on('reconnecting', () => {
console.log('🔄 Redis reconnecting...');
});
redisClient.on('close', () => {
if (!isShuttingDown) {
console.log('⚠️ Redis connection closed unexpectedly');
}
});
return redisClient;
} catch (error) {
console.error('Failed to initialize Redis client:', error);
@@ -69,17 +88,79 @@ export async function isRedisAvailable(): Promise<boolean> {
}
}
/**
* Get Redis connection metrics
*/
export async function getRedisMetrics() {
const client = getRedisClient();
if (!client) {
return {
available: false,
connected: false,
};
}
try {
const info = await client.info('stats');
const stats = info.split('\r\n').reduce(
(acc, line) => {
const [key, value] = line.split(':');
if (key && value) {
// eslint-disable-next-line security/detect-object-injection
acc[key] = value;
}
return acc;
},
{} as Record<string, string>
);
return {
available: true,
connected: client.status === 'ready',
status: client.status,
totalConnectionsReceived: stats['total_connections_received'] ?? 'N/A',
totalCommandsProcessed: stats['total_commands_processed'] ?? 'N/A',
instantaneousOpsPerSec: stats['instantaneous_ops_per_sec'] ?? 'N/A',
usedMemory: stats['used_memory_human'] ?? 'N/A',
};
} catch (error) {
return {
available: true,
connected: false,
error: error instanceof Error ? error.message : 'Unknown error',
};
}
}
/**
* Gracefully close Redis connection
*/
export async function closeRedisConnection(): Promise<void> {
if (isShuttingDown) {
return;
}
isShuttingDown = true;
if (redisClient) {
await redisClient.quit();
redisClient = null;
console.log('✅ Redis connection closed');
console.log('Closing Redis connection...');
try {
// Use quit() for graceful shutdown - waits for pending commands
await redisClient.quit();
redisClient = null;
console.log('✅ Redis connection closed successfully');
} catch (error) {
console.error('❌ Error closing Redis connection:', error);
// Force disconnect if quit fails
redisClient.disconnect();
redisClient = null;
}
}
}
// Note: Shutdown handlers are registered in db.ts to avoid conflicts
// Redis cleanup is called from the main shutdown handler
/**
* Scan Redis keys matching a pattern using SCAN command (non-blocking)
* This is a safer alternative to KEYS command for production use

View File

@@ -46,6 +46,24 @@ services:
- "-c"
- "max_wal_size=4GB"
pgbouncer:
build:
context: ./pgbouncer
dockerfile: Dockerfile
container_name: spywatcher-pgbouncer-dev
environment:
DB_USER: spywatcher
DB_PASSWORD: ${DB_PASSWORD:-spywatcher_dev_password}
PGBOUNCER_ADMIN_USER: pgbouncer_admin
PGBOUNCER_ADMIN_PASSWORD: ${PGBOUNCER_ADMIN_PASSWORD:-pgbouncer_admin_pass}
ports:
- "6432:6432"
depends_on:
postgres:
condition: service_healthy
networks:
- spywatcher-network
redis:
image: redis:7-alpine
container_name: spywatcher-redis-dev
@@ -72,7 +90,9 @@ services:
- /app/node_modules
- /app/dist
environment:
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD:-spywatcher_dev_password}@postgres:5432/spywatcher
# Use PgBouncer for application connections, direct for migrations
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD:-spywatcher_dev_password}@pgbouncer:6432/spywatcher?pgbouncer=true
DATABASE_URL_DIRECT: postgresql://spywatcher:${DB_PASSWORD:-spywatcher_dev_password}@postgres:5432/spywatcher
REDIS_URL: redis://redis:6379
NODE_ENV: development
PORT: 3001
@@ -88,13 +108,13 @@ services:
ports:
- "3001:3001"
depends_on:
postgres:
condition: service_healthy
pgbouncer:
condition: service_started
redis:
condition: service_healthy
networks:
- spywatcher-network
command: sh -c "npx prisma migrate dev && npm run dev:api"
command: sh -c "DATABASE_URL=$DATABASE_URL_DIRECT npx prisma migrate dev && npm run dev:api"
frontend:
build:

View File

@@ -50,6 +50,28 @@ services:
cpus: '1'
memory: 512M
pgbouncer:
build:
context: ./pgbouncer
dockerfile: Dockerfile
container_name: spywatcher-pgbouncer-prod
environment:
DB_USER: spywatcher
DB_PASSWORD: ${DB_PASSWORD}
PGBOUNCER_ADMIN_USER: pgbouncer_admin
PGBOUNCER_ADMIN_PASSWORD: ${PGBOUNCER_ADMIN_PASSWORD:-pgbouncer_admin_pass}
depends_on:
postgres:
condition: service_healthy
networks:
- spywatcher-network
restart: unless-stopped
deploy:
resources:
limits:
cpus: '0.25'
memory: 128M
redis:
image: redis:7-alpine
container_name: spywatcher-redis-prod
@@ -76,7 +98,7 @@ services:
dockerfile: Dockerfile
container_name: spywatcher-backend-prod
environment:
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD}@postgres:5432/spywatcher
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD}@pgbouncer:6432/spywatcher?pgbouncer=true
REDIS_URL: redis://redis:6379
NODE_ENV: production
PORT: 3001
@@ -90,8 +112,8 @@ services:
JWT_REFRESH_SECRET: ${JWT_REFRESH_SECRET}
JWT_SECRET: ${JWT_SECRET}
depends_on:
postgres:
condition: service_healthy
pgbouncer:
condition: service_started
redis:
condition: service_healthy
networks:
@@ -110,6 +132,7 @@ services:
dockerfile: Dockerfile
container_name: spywatcher-migrate-prod
environment:
# Migrations should connect directly to postgres, not through pgbouncer
DATABASE_URL: postgresql://spywatcher:${DB_PASSWORD}@postgres:5432/spywatcher
depends_on:
postgres:

35
docs/PGBOUNCER_SETUP.md Normal file
View File

@@ -0,0 +1,35 @@
# PgBouncer Setup Guide
Quick reference guide for setting up and managing PgBouncer connection pooling.
## 🚀 Quick Start
See [CONNECTION_POOLING.md](../CONNECTION_POOLING.md) for full documentation.
### Development
```bash
docker-compose -f docker-compose.dev.yml up -d
```
### Production
```bash
export DB_PASSWORD="your_secure_password"
docker-compose -f docker-compose.prod.yml up -d
```
## 📊 Monitoring
```bash
# System health
curl http://localhost:3001/api/admin/monitoring/connections/health
# Pool statistics
curl http://localhost:3001/api/admin/monitoring/connections/pool
```
## 🔗 Resources
- [Full Documentation](../CONNECTION_POOLING.md)
- [PgBouncer Official Docs](https://www.pgbouncer.org/)

3
pgbouncer/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
# Ignore generated userlist with credentials
# This file is generated at runtime by entrypoint.sh
userlist.txt

36
pgbouncer/Dockerfile Normal file
View File

@@ -0,0 +1,36 @@
FROM alpine:3.19
# Install PgBouncer and dependencies
RUN apk add --no-cache \
pgbouncer \
postgresql-client \
su-exec \
&& rm -rf /var/cache/apk/*
# Create pgbouncer user and directories
RUN addgroup -g 70 -S pgbouncer \
&& adduser -u 70 -S -D -G pgbouncer -H -h /var/lib/pgbouncer -s /sbin/nologin pgbouncer \
&& mkdir -p /etc/pgbouncer /var/log/pgbouncer /var/run/pgbouncer \
&& chown -R pgbouncer:pgbouncer /etc/pgbouncer /var/log/pgbouncer /var/run/pgbouncer
# Copy configuration files
COPY pgbouncer.ini /etc/pgbouncer/pgbouncer.ini
COPY userlist.txt.template /etc/pgbouncer/userlist.txt.template
COPY entrypoint.sh /entrypoint.sh
# Make entrypoint executable
RUN chmod +x /entrypoint.sh
# Set proper permissions for config files
RUN chown pgbouncer:pgbouncer /etc/pgbouncer/pgbouncer.ini /etc/pgbouncer/userlist.txt.template \
&& chmod 644 /etc/pgbouncer/pgbouncer.ini
# Expose PgBouncer port
EXPOSE 6432
# Health check with proper authentication
HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --retries=3 \
CMD PGPASSWORD="$DB_PASSWORD" psql -h 127.0.0.1 -p 6432 -U $DB_USER -d pgbouncer -c "SHOW POOLS;" || exit 1
# Run entrypoint script as root to allow chmod, then switch to pgbouncer user within
ENTRYPOINT ["/entrypoint.sh"]

154
pgbouncer/README.md Normal file
View File

@@ -0,0 +1,154 @@
# PgBouncer Configuration
This directory contains the PgBouncer connection pooler configuration for the Discord Spywatcher application.
## Files
- **pgbouncer.ini** - Main PgBouncer configuration file
- **userlist.txt.template** - Template for user authentication (actual file generated at runtime)
- **entrypoint.sh** - Startup script that generates credentials and starts PgBouncer
- **Dockerfile** - Container image definition
- **.gitignore** - Prevents committing generated credentials
## Configuration
### Pool Settings
The default configuration uses transaction-mode pooling with the following limits:
- **Pool Mode**: `transaction` (optimal for Prisma)
- **Default Pool Size**: 25 connections per database
- **Min Pool Size**: 5 connections minimum
- **Reserve Pool**: 5 additional connections for spikes
- **Max Client Connections**: 100 simultaneous clients
### Connection Lifecycle
- **Server Lifetime**: 3600 seconds (1 hour)
- **Idle Timeout**: 600 seconds (10 minutes)
- **Query Wait Timeout**: 120 seconds
### Security
- **Authentication**: MD5 hashed passwords
- **Admin Access**: Separate admin user for monitoring
- **Network**: Internal Docker network only (no external exposure in production)
## Usage
### Environment Variables
Required:
- `DB_USER` - Database username (e.g., "spywatcher")
- `DB_PASSWORD` - Database password
Optional:
- `PGBOUNCER_ADMIN_USER` - Admin username (default: "pgbouncer_admin")
- `PGBOUNCER_ADMIN_PASSWORD` - Admin password (recommended for production)
### Starting PgBouncer
The entrypoint script automatically:
1. Generates MD5 hashed passwords
2. Creates `userlist.txt` from environment variables
3. Sets appropriate file permissions
4. Starts PgBouncer
### Connecting
#### Application Connection (through PgBouncer)
```bash
postgresql://user:password@pgbouncer:6432/spywatcher?pgbouncer=true
```
#### Admin Console
```bash
psql -h pgbouncer -p 6432 -U pgbouncer_admin pgbouncer
```
## Monitoring
### PgBouncer Admin Commands
```sql
SHOW POOLS; -- Pool statistics
SHOW DATABASES; -- Database connections
SHOW CLIENTS; -- Client connections
SHOW SERVERS; -- Server connections
SHOW CONFIG; -- Current configuration
SHOW STATS; -- Performance statistics
RELOAD; -- Reload configuration
```
### Health Check
The Docker container includes a health check that runs:
```bash
psql -h 127.0.0.1 -p 6432 -U $DB_USER -d pgbouncer -c "SHOW POOLS;"
```
## Customization
To modify PgBouncer settings:
1. Edit `pgbouncer.ini`
2. Rebuild the Docker image or restart the container
3. Verify changes with `SHOW CONFIG;` in admin console
### Common Adjustments
**Increase pool size for high load:**
```ini
default_pool_size = 50
max_client_conn = 200
```
**Adjust timeouts:**
```ini
server_idle_timeout = 300 # Reduce to 5 minutes
query_wait_timeout = 60 # Reduce to 1 minute
```
**Enable verbose logging:**
```ini
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
verbose = 1
```
## Troubleshooting
### Issue: "No such user"
Check that credentials are set correctly:
```bash
docker exec container_name cat /etc/pgbouncer/userlist.txt
```
### Issue: Connection refused
Ensure PostgreSQL is running and healthy:
```bash
docker-compose ps postgres
docker logs container_name
```
### Issue: Pool saturation
Check pool utilization:
```sql
SHOW POOLS;
-- If cl_waiting > 0, increase pool size
```
## Security Notes
- **Never commit `userlist.txt`** - it contains credentials (already in `.gitignore`)
- **Use strong passwords** - especially for production
- **Rotate credentials** - regularly change passwords
- **Limit network access** - PgBouncer should only be accessible within Docker network
## Resources
- [PgBouncer Documentation](https://www.pgbouncer.org/)
- [Full Setup Guide](../docs/PGBOUNCER_SETUP.md)
- [Connection Pooling Guide](../CONNECTION_POOLING.md)
- [PostgreSQL Guide](../POSTGRESQL.md)

45
pgbouncer/entrypoint.sh Normal file
View File

@@ -0,0 +1,45 @@
#!/bin/sh
# PgBouncer entrypoint script
# This script generates the userlist.txt file from environment variables
# and starts PgBouncer
set -e
# Check if required environment variables are set
if [ -z "$DB_USER" ]; then
echo "Error: DB_USER environment variable is required"
exit 1
fi
if [ -z "$DB_PASSWORD" ]; then
echo "Error: DB_PASSWORD environment variable is required"
exit 1
fi
# Generate md5 hash for the password
# Format: "md5" + md5(password + username)
MD5_HASH=$(echo -n "${DB_PASSWORD}${DB_USER}" | md5sum | awk '{print $1}')
MD5_PASSWORD="md5${MD5_HASH}"
# Create userlist.txt with the hashed password
echo "\"${DB_USER}\" \"${MD5_PASSWORD}\"" > /etc/pgbouncer/userlist.txt
# Add admin user if specified
if [ -n "$PGBOUNCER_ADMIN_USER" ] && [ -n "$PGBOUNCER_ADMIN_PASSWORD" ]; then
ADMIN_MD5_HASH=$(echo -n "${PGBOUNCER_ADMIN_PASSWORD}${PGBOUNCER_ADMIN_USER}" | md5sum | awk '{print $1}')
ADMIN_MD5_PASSWORD="md5${ADMIN_MD5_HASH}"
echo "\"${PGBOUNCER_ADMIN_USER}\" \"${ADMIN_MD5_PASSWORD}\"" >> /etc/pgbouncer/userlist.txt
fi
# Set proper permissions (run as root before switching users)
chmod 600 /etc/pgbouncer/userlist.txt
chown pgbouncer:pgbouncer /etc/pgbouncer/userlist.txt
echo "PgBouncer configuration initialized"
echo "User: ${DB_USER}"
echo "Pool mode: transaction"
echo "Default pool size: 25"
echo "Max client connections: 100"
# Switch to pgbouncer user and start PgBouncer
exec su-exec pgbouncer pgbouncer /etc/pgbouncer/pgbouncer.ini

85
pgbouncer/pgbouncer.ini Normal file
View File

@@ -0,0 +1,85 @@
[databases]
; Database-name = host=hostname port=port dbname=database
spywatcher = host=postgres port=5432 dbname=spywatcher
[pgbouncer]
; Connection pooling mode (session, transaction, or statement)
; session - one server connection per client connection (default)
; transaction - one server connection per transaction
; statement - one server connection per statement (not recommended for prepared statements)
pool_mode = transaction
; Listen address and port
listen_addr = 0.0.0.0
listen_port = 6432
; Authentication
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
; Administrator credentials
admin_users = pgbouncer_admin
; Connection limits per pool
; Total for all pools = default_pool_size * num_databases
default_pool_size = 25
min_pool_size = 5
reserve_pool_size = 5
reserve_pool_timeout = 3
; Maximum allowed connections per database per user
max_db_connections = 50
max_user_connections = 50
; Maximum total client connections
max_client_conn = 100
; Server connection lifetime
server_lifetime = 3600
server_idle_timeout = 600
; Client connection limits
server_connect_timeout = 15
server_login_retry = 15
; Query timeouts
query_timeout = 0
query_wait_timeout = 120
client_idle_timeout = 0
idle_transaction_timeout = 0
; Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60
; This prevents errors when server connection is lost
server_reset_query = DISCARD ALL
server_reset_query_always = 0
; Ignore startup parameters
ignore_startup_parameters = extra_float_digits
; Application name
application_name_add_host = 1
; DNS settings
dns_max_ttl = 15
dns_nxdomain_ttl = 15
dns_zone_check_period = 0
; TLS/SSL settings (disabled by default for internal network)
;client_tls_sslmode = disable
;server_tls_sslmode = disable
; Performance tuning
pkt_buf = 4096
listen_backlog = 128
sbuf_loopcnt = 5
suspend_timeout = 10
tcp_keepalive = 1
tcp_keepidle = 0
tcp_keepintvl = 0
tcp_keepcnt = 0
tcp_user_timeout = 0

View File

@@ -0,0 +1,12 @@
; PgBouncer user list file (TEMPLATE)
; This file is automatically generated by entrypoint.sh
; Format: "username" "password"
; Password can be plaintext or md5 hash
; md5 format: "md5" + md5(password + username)
;
; DO NOT EDIT THIS TEMPLATE - it will be overwritten
; Configure credentials via environment variables:
; DB_USER - database username
; DB_PASSWORD - database password
; PGBOUNCER_ADMIN_USER - admin username (optional)
; PGBOUNCER_ADMIN_PASSWORD - admin password (optional)