Desktop Control

Gives Claude control over your mouse, keyboard, and screen through PyAutoGUI. You can automate clicks, type text, execute hotkeys, take screenshots, and even use OCR to locate text on screen. Works across Windows, macOS, and Linux. The app control commands let you open and focus applications by name. Most useful for repetitive GUI tasks like form filling, testing interfaces, or scraping data from applications without APIs. The image matching feature is handy but needs exact visual matches, so adjust confidence levels if things don't line up. Remember PyAutoGUI's fail-safe: mouse to corner kills everything.

GitHub stars: 2
last updated: Jun 3, 2026

View on GitHub

Install

$ npx -y skills add patrickporto/desktop-agent --skill 'Desktop Control' --agent claude-code

Installs into .claude/skills of the current project.

Files

Desktop Control Skill

This skill provides comprehensive desktop automation capabilities through PyAutoGUI, allowing AI agents to control the mouse, keyboard, take screenshots, and interact with the desktop environment.

How to Use This Skill

As an AI agent, you can invoke desktop automation commands using the uvx desktop-agent CLI.

Command Structure

All commands follow this pattern:

uvx desktop-agent <category> <command> [arguments] [options]

Categories:

mouse - Mouse control
keyboard - Keyboard input
screen - Screenshots and screen analysis
message - User dialogs
app - Application control (open, focus, list windows)

Available Commands

🖱️ Mouse Control (`mouse`)

Control cursor movement and clicks.

# Move cursor to coordinates
uvx desktop-agent mouse move <x> <y> [--duration SECONDS]

# Click at current position or specific coordinates
uvx desktop-agent mouse click [x] [y] [--button left|right|middle] [--clicks N]

# Specialized clicks
uvx desktop-agent mouse double-click [x] [y]
uvx desktop-agent mouse right-click [x] [y]
uvx desktop-agent mouse middle-click [x] [y]

SKILL.md on GitHub