Background Jobs Configuration

Documenso uses background jobs to handle asynchronous tasks like sending emails, processing documents, and managing document expiration. This guide covers configuration and optimization of the job system.

Job System Overview

Background jobs handle:

Sending signing invitation emails
Document completion notifications
Webhook delivery
Recipient expiration checks
Document cleanup
Email verification
Password reset emails

Job Providers

Documenso supports two background job providers:

Local Provider

Built-in job system using PostgreSQL. Recommended for most deployments.

Inngest

Cloud-based job orchestration for enterprise deployments.

Local Provider (Default)

The local provider uses PostgreSQL to store and process jobs. It’s simple, reliable, and requires no external services.

Configuration

NEXT_PRIVATE_JOBS_PROVIDER

string

default:"local"

Set to "local" to use the built-in job system.

NEXT_PRIVATE_JOBS_PROVIDER="local"

How It Works

Job Creation: When a job is triggered (e.g., sending an email), a record is created in the BackgroundJob table
Job Dispatch: The job is immediately dispatched to an HTTP endpoint on the same server
Job Execution: The endpoint processes the job and updates the database
Retry Logic: Failed jobs are automatically retried with exponential backoff
Cron Jobs: A polling mechanism checks for scheduled jobs every 30 seconds

Architecture

┌─────────────────────────────────────────────────┐
│              Application Code                   │
│  (e.g., document signed, send email)           │
└─────────────────┬───────────────────────────────┘
                  │
                  ▼
         ┌────────────────┐
         │  Create Job    │
         │  in Database   │
         └────────┬───────┘
                  │
                  ▼
         ┌────────────────────────┐
         │   HTTP POST Request    │
         │ /api/jobs/{id}/{jobId} │
         └────────┬───────────────┘
                  │
                  ▼
         ┌────────────────┐
         │  Execute Job   │
         │  Handler       │
         └────────┬───────┘
                  │
                  ▼
         ┌────────────────┐
         │  Update Status │
         │  in Database   │
         └────────────────┘

Job Lifecycle

PENDING: Job created, waiting for execution
PROCESSING: Job is currently being executed
COMPLETED: Job finished successfully
FAILED: Job failed after maximum retries

Retry Behavior

Maximum Retries: 3 attempts per job
Retry Strategy: Exponential backoff
Failed Jobs: Marked as FAILED after max retries
Task-Level Retries: Individual tasks within a job have their own retry logic

Cron Jobs

The local provider includes a cron poller that runs every 30 seconds to check for scheduled jobs:

// Cron jobs in Documenso
{
  "expire-recipients-sweep": "0 * * * *"  // Every hour
}

Features:

Deterministic job IDs prevent duplicate execution across multiple instances
Random jitter (0-5 seconds) prevents thundering herd
Only the latest cron slot is executed (no backfill after downtime)

Scaling Considerations

Single Instance:

Works perfectly out of the box
All jobs processed on the same server

Multiple Instances:

Job execution is distributed across instances
Cron jobs use deterministic IDs to prevent duplicates
If a job fails on one instance, another can pick it up on retry

Database Impact

The local provider creates two tables:

BackgroundJob: Stores job metadata and status
BackgroundJobTask: Stores individual task results (for idempotent retry)

Storage overhead: Minimal. Jobs are cleaned up periodically.

Configuration Example

.env

# Use local job provider
NEXT_PRIVATE_JOBS_PROVIDER="local"

# Ensure internal URL is set for job dispatch
NEXT_PRIVATE_INTERNAL_WEBAPP_URL="http://localhost:3000"

Monitoring

Query job status:

-- Job status overview
SELECT
  status,
  COUNT(*) as count
FROM "BackgroundJob"
GROUP BY status;

-- Recent failed jobs
SELECT
  id,
  name,
  "createdAt",
  "completedAt",
  retried
FROM "BackgroundJob"
WHERE status = 'FAILED'
ORDER BY "createdAt" DESC
LIMIT 10;

-- Long-running jobs
SELECT
  id,
  name,
  "createdAt",
  NOW() - "createdAt" as duration
FROM "BackgroundJob"
WHERE status = 'PROCESSING'
AND "createdAt" < NOW() - INTERVAL '5 minutes';

Inngest Provider (Enterprise)

Inngest is a cloud-based job orchestration platform with advanced features for enterprise deployments.

When to Use Inngest

High-volume deployments (100,000+ documents/month)
Need for advanced observability and debugging
Complex workflow orchestration
Distributed teams requiring shared job visibility
Compliance requirements for job execution audit trails

Configuration

NEXT_PRIVATE_JOBS_PROVIDER

string

Set to "inngest" to use Inngest.

NEXT_PRIVATE_JOBS_PROVIDER="inngest"

NEXT_PRIVATE_INNGEST_EVENT_KEY

string

required

Inngest event key for authentication.

NEXT_PRIVATE_INNGEST_EVENT_KEY="your-inngest-event-key"

INNGEST_EVENT_KEY

string

Alternative environment variable name for Inngest event key.

INNGEST_EVENT_KEY="your-inngest-event-key"

NEXT_PRIVATE_INNGEST_APP_ID

string

default:"documenso-app"

Inngest application ID.

NEXT_PRIVATE_INNGEST_APP_ID="documenso-app"

Setup Steps

Create Inngest Account Sign up at inngest.com
Get Event Key Copy your event key from the Inngest dashboard

Configure Documenso

.env

NEXT_PRIVATE_JOBS_PROVIDER="inngest"
NEXT_PRIVATE_INNGEST_EVENT_KEY="your-event-key"
NEXT_PRIVATE_INNGEST_APP_ID="documenso-app"

Deploy and Register Functions When you deploy Documenso, it will automatically register job functions with Inngest

Features

Web Dashboard: Visual job monitoring and debugging
Replay Failed Jobs: Retry failed jobs from the dashboard
Function Versioning: Deploy new job logic without downtime
Parallel Execution: Optimize parallelism automatically
Event Replay: Replay historical events
Advanced Scheduling: Complex cron expressions and delays

Cost

Inngest pricing (as of 2024):

Free Tier: 50,000 function runs/month
Paid Plans: Starting at $20/month

For high-volume deployments, Inngest is cost-effective compared to managing your own infrastructure.

Job Types in Documenso

Document Workflow Jobs

send-signing-email

Triggered when a document is sent for signing
Sends invitation emails to recipients
Retries on failure

document-completed

Triggered when all recipients sign
Sends completion notifications
Triggers webhooks

recipient-rejected

Triggered when a recipient rejects
Notifies document owner
Triggers webhooks

Scheduled Jobs (Cron)

expire-recipients-sweep

Runs: Every hour (0 * * * *)
Checks for expired recipient invitations
Processes up to 1,000 recipients per run
Sends expiration notifications

Webhook Jobs

trigger-webhook

Delivers webhook payloads to configured endpoints
Retries with exponential backoff
Logs delivery attempts

Email Verification Jobs

send-verification-email

Sends email verification links
Triggered on user signup

send-password-reset

Sends password reset links
Triggered on password reset request

Troubleshooting

Jobs not executing

Cause: Job dispatch endpoint not reachable. Solution: Verify NEXT_PRIVATE_INTERNAL_WEBAPP_URL is correct:

# Test the endpoint
curl http://localhost:3000/api/health

Ensure the app can reach itself (important in Docker/Kubernetes).

Jobs stuck in PROCESSING state

Cause: Job handler crashed or timed out. Solution:

Check application logs for errors

Manually mark jobs as FAILED:

UPDATE "BackgroundJob"
SET status = 'FAILED', "completedAt" = NOW()
WHERE status = 'PROCESSING'
AND "createdAt" < NOW() - INTERVAL '1 hour';

High database load from jobs

Cause: Too many jobs executing concurrently. Solution:

Increase database resources
Optimize job handlers to use fewer queries
Consider switching to Inngest for better concurrency control

Duplicate cron job executions

Cause: Multiple instances creating the same cron job. Solution: This shouldn’t happen with the local provider’s deterministic IDs. If it does:

Check database logs for unique constraint violations
Verify all instances use the same NEXT_PRIVATE_INTERNAL_WEBAPP_URL
Ensure system clocks are synchronized (NTP)

Performance Optimization

Reduce Job Latency

For the local provider, job latency depends on:

Network latency to the job endpoint
Database performance for job updates
Job handler execution time

Optimizations:

Use NEXT_PRIVATE_INTERNAL_WEBAPP_URL with a local IP (not external domain)
Optimize database queries in job handlers
Use connection pooling

Handle High Volume

For deployments with >10,000 jobs/day:

Monitor database size: Clean up old jobs periodically

DELETE FROM "BackgroundJob"
WHERE status = 'COMPLETED'
AND "completedAt" < NOW() - INTERVAL '30 days';

Scale horizontally: Deploy multiple instances
Consider Inngest: For >100,000 jobs/month

Security Considerations

Job Endpoint Security

The local provider uses signed requests to secure the job endpoint:

Each request includes a signature (X-Job-Signature header)
Signature is verified before job execution
Prevents unauthorized job triggering

Sensitive Data in Jobs

Job payloads are stored in the database. Avoid storing sensitive data:

// ❌ Bad: Storing sensitive data
await jobs.triggerJob({
  name: 'send-email',
  payload: {
    email: 'user@example.com',
    password: 'plaintext-password'  // Never do this!
  }
});

// ✅ Good: Only store IDs, fetch sensitive data in handler
await jobs.triggerJob({
  name: 'send-email',
  payload: {
    userId: 123  // Fetch user data in handler
  }
});

Monitoring and Observability

Metrics to Track

Job Success Rate

SELECT
  status,
  COUNT(*) * 100.0 / SUM(COUNT(*)) OVER () as percentage
FROM "BackgroundJob"
WHERE "createdAt" > NOW() - INTERVAL '24 hours'
GROUP BY status;

Average Job Duration

SELECT
  name,
  AVG(EXTRACT(EPOCH FROM ("completedAt" - "createdAt"))) as avg_duration_seconds
FROM "BackgroundJob"
WHERE status = 'COMPLETED'
GROUP BY name;

Retry Rate

SELECT
  name,
  AVG(retried) as avg_retries
FROM "BackgroundJob"
GROUP BY name;

Alerting

Set up alerts for:

Jobs failing repeatedly
Jobs stuck in PROCESSING
High retry rates
Cron jobs not running

Example alert query:

-- Alert if >10% of jobs failed in the last hour
SELECT COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM "BackgroundJob" WHERE "createdAt" > NOW() - INTERVAL '1 hour'), 0) as failed_percentage
FROM "BackgroundJob"
WHERE status = 'FAILED'
AND "createdAt" > NOW() - INTERVAL '1 hour'
HAVING COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM "BackgroundJob" WHERE "createdAt" > NOW() - INTERVAL '1 hour'), 0) > 10;

Migrating Between Providers

Local to Inngest

Set up Inngest account and get event key

Update environment variables:

NEXT_PRIVATE_JOBS_PROVIDER="inngest"
NEXT_PRIVATE_INNGEST_EVENT_KEY="your-key"

Deploy the updated configuration
Verify jobs are appearing in Inngest dashboard
Monitor for any issues

Jobs in PENDING or PROCESSING state in the local provider will not be migrated. Complete or cancel them before switching.

Inngest to Local

Update environment variables:
```
NEXT_PRIVATE_JOBS_PROVIDER="local"
```
Deploy the updated configuration
New jobs will use the local provider
Existing Inngest jobs will complete normally

Best Practices

Use Appropriate Provider

Small/Medium deployments: Local provider
Enterprise/High-volume: Inngest

Monitor Job Health

Set up alerts for failed jobs and track success rates.

Handle Failures Gracefully

Design job handlers to be idempotent and handle retries properly.

Clean Up Old Jobs

Regularly delete completed jobs older than 30 days to prevent database bloat.

Test Job Execution

Verify jobs execute correctly in your deployment environment before going live.

Use Proper Logging

Log job execution for debugging. The local provider includes built-in logging via io.logger.

​Job System Overview

​Job Providers

Local Provider

Inngest

​Local Provider (Default)

​Configuration

​How It Works

​Architecture

​Job Lifecycle

​Retry Behavior

​Cron Jobs

​Scaling Considerations

​Database Impact

​Configuration Example

​Monitoring

​Inngest Provider (Enterprise)

​When to Use Inngest

​Configuration

​Setup Steps

​Features

​Cost

​Job Types in Documenso

​Document Workflow Jobs

​Scheduled Jobs (Cron)

​Webhook Jobs

​Email Verification Jobs

​Troubleshooting

​Jobs not executing

​Jobs stuck in PROCESSING state

​High database load from jobs

​Duplicate cron job executions

​Performance Optimization

​Reduce Job Latency

​Handle High Volume

​Security Considerations

​Job Endpoint Security

​Sensitive Data in Jobs

​Monitoring and Observability

​Metrics to Track

​Alerting

​Migrating Between Providers

​Local to Inngest

​Inngest to Local

​Best Practices

​Next Steps

Environment Variables

Database Configuration

Job System Overview

Job Providers

Local Provider (Default)

Configuration

How It Works

Architecture

Job Lifecycle

Retry Behavior

Cron Jobs

Scaling Considerations

Database Impact

Configuration Example

Monitoring

Inngest Provider (Enterprise)

When to Use Inngest

Configuration

Setup Steps

Features

Cost

Job Types in Documenso

Document Workflow Jobs

Scheduled Jobs (Cron)

Webhook Jobs

Email Verification Jobs

Troubleshooting

Jobs not executing

Jobs stuck in PROCESSING state

High database load from jobs

Duplicate cron job executions

Performance Optimization

Reduce Job Latency

Handle High Volume

Security Considerations

Job Endpoint Security

Sensitive Data in Jobs

Monitoring and Observability

Metrics to Track

Alerting

Migrating Between Providers

Local to Inngest

Inngest to Local

Best Practices

Next Steps