Python Background Jobs

Solid foundation for implementing async task processing in Python apps. Covers the essential patterns you'll actually need: job state management with proper status tracking, idempotent task design to handle retries safely, and Celery configuration with realistic timeout and retry settings. The code examples show practical implementations for common scenarios like payment processing, data exports, and webhook handling. Includes dead letter queue patterns for handling permanent failures and proper API design for job polling. Won't make you an expert in every queue system, but gives you working patterns that translate well across Redis, RQ, or cloud-native solutions.

skills.sh installs: 11.6K
GitHub stars: 42.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add wshobson/agents --skill python-background-jobs --agent claude-code

Installs into .claude/skills of the current project.

Files

Python Background Jobs & Task Queues

Decouple long-running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously.

When to Use This Skill

Processing tasks that take longer than a few seconds
Sending emails, notifications, or webhooks
Generating reports or exporting data
Processing uploads or media transformations
Integrating with unreliable external services
Building event-driven architectures

Core Concepts

1. Task Queue Pattern

API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously.

2. Idempotency

Tasks may be retried on failure. Design for safe re-execution.

3. Job State Machine

Jobs transition through states: pending → running → succeeded/failed.

4. At-Least-Once Delivery

Most queues guarantee at-least-once delivery. Your code must handle duplicates.

Quick Start

This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud-native solutions (AWS SQS, GCP Tasks) are equally valid choices.