Background Worker / Jobs

Scenario

You maintain an application with background workers processing asynchronous jobs.

Jobs include:

• sending emails;
• generating reports;
• processing imports;
• handling webhooks;
• synchronizing external systems.

Workers may retry failed jobs.

That means the same logical job can sometimes execute more than once.

Coding agents must account for:

• retries;
• idempotency;
• failure handling;
• job payload compatibility;
• observability.

Repository Structure

platform/
├── src/
│   ├── api/
│   ├── jobs/
│   │   ├── send-email.ts
│   │   ├── generate-report.ts
│   │   └── sync-customer.ts
│   ├── queues/
│   │   ├── client.ts
│   │   └── enqueue.ts
│   └── workers/
│       └── worker.ts
├── tests/
│   └── jobs/
└── AGENTS.md

AGENTS.md

# Background Job Instructions

## Jobs

- Keep background job handlers under the existing job architecture.
- Keep job payloads explicit and serializable.
- Avoid putting large or unnecessary data into queue payloads when stable identifiers are sufficient.

## Retries

- Assume retryable jobs may execute more than once.
- Preserve existing retry and backoff behavior.
- Do not add automatic retries to operations that are unsafe to repeat without addressing idempotency.

## Idempotency

- Make externally visible side effects idempotent when jobs may be retried.
- Reuse existing idempotency keys, job identifiers, or deduplication mechanisms.
- Do not assume "the worker normally runs once" is sufficient protection.

## Errors

- Allow retryable failures to follow the queue's established retry mechanism.
- Distinguish permanent failures from transient failures when the architecture supports it.
- Preserve dead-letter or failed-job handling.

## Logging

- Include stable job identifiers in logs.
- Log enough context to diagnose failures without exposing secrets or sensitive payload data.

## Testing

For job changes:

- test successful processing;
- test retryable failure behavior;
- test repeated execution when idempotency matters;
- test invalid or unsupported payloads where applicable.

What This Does

This tells the agent that background execution has different assumptions from synchronous request handling.

Suppose a job performs:

charge customer
      ↓
send receipt
      ↓
mark invoice paid

The worker crashes after charging the customer but before marking the invoice paid.

The queue retries the job.

Without idempotency, the second attempt could charge the customer again.

The workflow therefore needs to consider repeated execution.

What This Does NOT Do

The instructions do not require every job to be perfectly idempotent.

Some jobs may perform naturally safe operations.

For example:

recalculate cached statistics

may simply replace existing state.

The importance of idempotency depends on the side effect.

The instructions also do not say every failure should retry forever.

Some failures are permanent.

For example:

invalid payload
unsupported operation
deleted resource

may require failure handling rather than repeated retries.

Why These Instructions Matter

Background jobs separate the request from the work.

A user may see:

Request accepted

while the actual processing happens seconds or minutes later.

Failures therefore require observability.

A useful log might contain:

jobId
jobType
attempt
resourceId
failure category

without dumping the complete sensitive payload.

Payload compatibility is another concern.

Suppose old queued jobs contain:

{
  "userId": "123"
}

and a deployment changes the handler to require:

{
  "userId": "123",
  "organizationId": "456"
}

Jobs already sitting in the queue may still use the old payload shape.

Changes to long-lived queue contracts should account for that possibility.

Key Decisions

Assume retries happen

Even reliable queue systems can deliver work more than once.

Design important side effects accordingly.

Use identifiers in payloads

Instead of enqueueing an entire mutable object:

{
  "user": {
    "...": "many fields"
  }
}

a stable identifier may be preferable:

{
  "userId": "123"
}

The worker can load the current data when appropriate.

Separate transient and permanent failure

A temporary network outage may deserve retry.

An invalid payload may not.

Preserve observability

Asynchronous failures are harder to connect to the original request.

Stable job identifiers make debugging much easier.

Consider deployed queue history

Jobs can survive longer than application deployments.

Changing job payloads can therefore behave like changing an API contract.

When to Use This Pattern

Use this pattern when:

• work is processed asynchronously;
• queues retry failed jobs;
• workers call external services;
• jobs produce important side effects;
• queue payloads can survive deployments.

Background workers require instructions that explicitly account for retries, idempotency, and delayed execution.