Batch Processing Jobs

This is your go-to for building job queues and background workers when you need to process things outside the request cycle. It covers Bull queues for Node.js, Celery-style workers for Python, and cron schedulers for recurring tasks. The best practices are solid: idempotency, exponential backoff, dead letter queues, and proper monitoring. It's opinionated about keeping large payloads out of your queue and always implementing timeouts. If you're sending bulk emails, processing uploads, running ETL pipelines, or doing anything that takes more than a few seconds, this gives you the patterns to do it right without reinventing retry logic for the hundredth time.

skills.sh installs: 479
GitHub stars: 379
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add aj-geddes/useful-ai-prompts --skill batch-processing-jobs --agent claude-code

Installs into .claude/skills of the current project.

Files

Overview

Implement scalable batch processing systems for handling large-scale data processing, scheduled tasks, and async operations efficiently.

When to Use

Processing large datasets
Scheduled report generation
Email/notification campaigns
Data imports and exports
Image/video processing
ETL pipelines
Cleanup and maintenance tasks
Long-running computations
Bulk data updates

Quick Start

Minimal working example:

import Queue from "bull";
import { v4 as uuidv4 } from "uuid";

interface JobData {
  id: string;
  type: string;
  payload: any;
  userId?: string;
  metadata?: Record<string, any>;
}

interface JobResult {
  success: boolean;
  data?: any;
  error?: string;
  processedAt: number;
  duration: number;
}

class BatchProcessor {
  private queue: Queue.Queue<JobData>;
  private resultQueue: Queue.Queue<JobResult>;

SKILL.md on GitHub