Operations Guide — HenKaiPan ASPM

This guide covers operational aspects of running a self-hosted HenKaiPan ASPM instance: worker scaling, scanner requirements, troubleshooting common issues, and routine maintenance.

Worker Scaling

Architecture

The worker is responsible for executing background jobs:

Jobs are distributed via Redis/Asynq. The default deployment runs a single worker instance.

Note: Redis also serves as the pub/sub relay for SSE (Server-Sent Events). The worker publishes real-time events (e.g., AI summary completed, scan finished) to a Redis channel; the API subscribes and delivers them to connected browser clients. This means SSE real-time updates work even when the API and worker run as separate processes.

Running Multiple Workers

To increase throughput (especially for concurrent scans):

# Scale to 3 workers
docker compose up -d --scale worker=3

Important considerations:

• Scanner binaries run directly within the worker process — no container isolation per scan
• Workers primarily contend for system resources (RAM, CPU) when running scanners
• Adding workers increases scan concurrency but also increases peak resource usage
• Multiple workers share the same Redis queue — no additional configuration needed
• Workers automatically recover stuck scans on startup via RecoverStuck()

Resource Planning

Each scanner consumes variable resources (see scanner requirements below). Heavy scanners like Trivy and Semgrep can use 1-2 GB RAM each.

Queue Monitoring

# Check queue depth via Redis CLI
docker compose exec redis redis-cli -n 1 LLEN asynq:queues

# List all queues
docker compose exec redis redis-cli -n 1 KEYS "asynq:*"

# Check dead letter queue (failed jobs)
docker compose exec redis redis-cli -n 1 ZCARD asynq:dead

Scanner Runtime Requirements

HenKaiPan supports 13 security scanners grouped into packs:

Scanner Packs

Resource Requirements per Scanner

*Network access required for vulnerability database updates.

Scan Timeouts

Scans exceeding the timeout are marked as failed. The worker recovers and continues processing the next job.

Troubleshooting

1. API Won't Start

Symptom: Container exits immediately. docker compose logs api shows error.

Check:

docker compose logs api

Common causes:

2. Database Connection Refused

Symptom: API starts, logs show connection refused or dial tcp ...:5432: connect: connection refused.

Check:

docker compose ps postgres
docker compose logs postgres

Fixes:

• Postgres container may still be starting — wait for healthcheck
• If Postgres repeatedly fails, check pgdata volume permissions
• Ensure DATABASE_URL matches postgres credentials in .env

3. Worker Not Picking Up Jobs

Symptom: Scans created in UI stay in "pending" status.

Check:

docker compose logs worker | tail -50
docker compose exec redis redis-cli PING

Fixes:

• Worker may be unhealthy — check docker compose ps worker
• Redis connectivity issue — verify REDIS_ADDR in .env
• Worker crashed — check for OOM kills: docker compose logs worker | grep -i "killed\|oom\|exit code 137"

4. Scans Fail to Start

Symptom: Scan status goes to "failed" immediately after creation.

Check:

docker compose logs worker | grep -i "scan\|error\|fail"

Common causes:

5. Scans Timeout on Large Repos

Symptom: Scans on large repositories (50k+ files) consistently timeout.

Solutions:

• Target specific paths in scan configuration (exclude node_modules, vendor, etc.)
• Increase scan timeout via environment variable
• Run scans during off-peak hours
• Consider splitting the repository into smaller projects

6. Frontend Shows Blank Page or API Errors

Symptom: Frontend loads but shows no data, or console shows network errors.

Root cause: The frontend has API_BASE hardcoded to http://localhost:8080 in frontend/src/lib/api.ts. This means the browser must be able to reach the API at that address.

Fixes:

• For production behind a reverse proxy, the API and frontend are served on the same origin (port 443) — no CORS issues
• For development, ensure the API is accessible at http://localhost:8080
• Check CORS configuration if using a custom domain: CORS_ALLOWED_ORIGINS env var

7. Authentication Issues After Config Change

Symptom: Users cannot log in, or existing sessions are invalidated.

Cause: Changing JWT_SECRET invalidates all existing auth tokens.

Fix: Users must log in again to obtain new tokens. This is expected behavior after secret rotation — plan accordingly.

8. Migration Errors

Symptom: API logs show migration failure. API may not start.

Check:

docker compose logs api | grep -i "migration\|error"

Fixes:

• Migrations run in order from migrations/ directory
• If a migration fails, check syntax by running it manually:

  docker compose exec -T postgres psql -U aspm -d aspm < migrations/XXX_your_migration.sql

• Migrations are additive — they create tables/columns but do not drop them

9. Out of Disk Space

Symptom: Docker operations fail with no space left on device.

Check:

df -h
docker system df

Cleanup:

# Remove unused Docker images (safe, will be re-pulled)
docker image prune -a

# Remove stopped containers and unused networks
docker system prune -f

# Check Docker root directory size
du -sh /var/lib/docker/

10. High Memory Usage

Symptom: System becomes slow, OOM killer terminates containers.

Checks:

docker stats
free -h

Tuning:

If running multiple workers, reduce scanner concurrency or add more RAM.

11. AI Features Not Working

Symptom: AI remediation, summary, or validation buttons are disabled or return errors.

Check:

docker compose logs api | grep -i "ai\|openrouter\|cloudflare\|ollama"
docker compose logs worker | grep -i "ai\|openrouter\|cloudflare\|ollama"

Fixes:

12. Email Not Sending

Symptom: Notifications configured but no emails sent.

Check:

docker compose logs worker | grep -i "email\|smtp\|mail"

Fixes:

• Verify SMTP settings in .env
• For testing, Mailpit is included in the dev docker-compose (port 1025 SMTP, 8025 web UI)
• For production, configure Brevo or any SMTP provider

13. Rate Limiting Issues

Symptom: API returns 429 Too Many Requests.

The default rate limits are:

Rate limit headers (X-RateLimit-*) are included in responses. Limits are Redis-based and fail open (no rate limiting) if Redis is unreachable.

Maintenance

Log Rotation

Configure Docker log rotation globally in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then restart Docker: sudo systemctl restart docker

Database Maintenance

PostgreSQL requires periodic maintenance:

# Connect and run VACUUM (safe, non-blocking)
docker compose exec postgres psql -U aspm -d aspm -c "VACUUM ANALYZE;"

# Check for bloat
docker compose exec postgres psql -U aspm -d aspm -c "SELECT schemaname, tablename, n_live_tup, n_dead_tup, last_autovacuum FROM pg_stat_user_tables;"

Autovacuum is enabled by default in PostgreSQL 17 — no manual intervention required for most workloads.

SSL Certificate Renewal

For nginx + certbot:

# Test renewal
sudo certbot renew --dry-run

# Actual renewal (certbot auto-renew via cron/systemd)
sudo certbot renew

For Caddy: certificate renewal is automatic.

Docker Image Cleanup

# Weekly cleanup (add to crontab)
docker image prune -a -f --filter "until=168h"  # Remove images older than 7 days

Note: pull_policy: always will re-pull images on docker compose up -d, so pruning old images is safe.

Health Endpoints

Getting Help

• Documentation: https://henkaipan.dyallab.com.ar/docs/
• GitHub Issues: https://github.com/Dyallab/HenKaiPan-self-hosted/issues