Parallel Processing Patterns

If you're running bash scripts that process files, hit APIs, or crunch data, this skill covers how to do it in parallel instead of one at a time. It walks through GNU Parallel (the Swiss Army knife approach), xargs with the -P flag (quick and dirty), and pure bash job pools with background processes and wait. The examples show real patterns like collecting exit codes from parallel children, capping concurrency so you don't fork-bomb yourself, and handling shared output files without races. Honestly, GNU Parallel alone is worth learning if you've been writing slow for-loops for years. The skill includes actual working code for FIFO-based work queues and job slot management, not just theory.

skills.sh installs: 180
GitHub stars: 61
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add josiahsiegel/claude-plugin-marketplace --skill parallel-processing-patterns --agent claude-code

Installs into .claude/skills of the current project.

Files

CRITICAL GUIDELINES

Windows File Path Requirements

MANDATORY: Always Use Backslashes on Windows for File Paths

When using Edit or Write tools on Windows, you MUST use backslashes (\) in file paths, NOT forward slashes (/).

Parallel Processing Patterns in Bash (2025)

Overview

Comprehensive guide to parallel and concurrent execution in bash, covering GNU Parallel, xargs parallelization, job control, worker pools, and modern async patterns for maximum performance.

GNU Parallel (Recommended)

Installation

# Debian/Ubuntu
sudo apt-get install parallel

# macOS
brew install parallel

# From source
wget https://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2
tar -xjf parallel-latest.tar.bz2
cd parallel-*
./configure && make && sudo make install

Basic Usage

#!/usr/bin/env bash
set -euo pipefail

# Process multiple files in parallel
parallel gzip ::: *.txt

# Equivalent to:
# for f in *.txt; do gzip "$f"; done
# But runs in parallel!

# Using find with parallel
find . -name "*.jpg" | parallel convert {} -resize 50% resized/{}

# Specify number of jobs
parallel -j 8 process_file ::: *.dat

SKILL.md on GitHub