Design a durable job scheduler

Clarifying questions

• Do users expect exactly-once side effects or exactly-once scheduling attempts?
• What is the acceptable clock skew between regions?
• Are jobs CPU-heavy, IO-heavy, or calls to external services?

Functional requirements

• Create one-time, delayed, and recurring jobs.
• Dispatch due jobs to workers with retry and backoff.
• Expose job status, attempts, next run, and failure reason.

Nonfunctional requirements

• Dispatch due jobs within 30 seconds of target time for normal load.
• Avoid unbounded retry storms when an external dependency fails.
• Make every worker handoff idempotent.

Scale assumptions

• 10 million scheduled jobs stored.
• 100,000 jobs due per minute during peak batch windows.
• Recurring jobs range from every minute to monthly.

API sketch

• POST /v1/jobs { runAt, cron, payloadRef, idempotencyKey } -> { jobId }
• POST /v1/jobs/{jobId}/cancel -> { status }

Data model

• jobs(id, tenant_id, run_at, cron_expr, status, attempts, payload_ref, idempotency_key).
• job_attempts(job_id, attempt, lease_until, worker_id, started_at, finished_at).

Architecture components

• A scheduler scanner moves due rows into partitioned ready queues.
• Workers lease queue items, execute, then mark completion or schedule retry.
• A reaper returns expired leases to the ready queue.

Bottlenecks

• Scanning due jobs by timestamp can overload the primary database.
• Large cron fanout can create synchronized spikes.

Failure modes

• Worker crash: lease expires and the job is retried.
• Scheduler crash: another scheduler instance resumes partition ownership.
• External API outage: retry policy moves jobs to exponential backoff.

Observability

• Due-lag p95, queue depth, retry rate, lease expiration count.
• Attempts by failure class and tenant tier.

Security / privacy

• Store payloads by reference with scoped access, not as raw secrets in queue messages.
• Enforce tenant isolation on job listing and cancellation.

Cost considerations

• Partition count trades off scan parallelism against coordination overhead.
• Long payload retention increases storage and privacy review costs.

Tradeoffs

• Database-backed scheduling is simpler to reason about; a broker-first design can absorb spikes better.
• At-least-once execution is realistic; exactly-once side effects belong in the job handler.

Rubric