pixel-accurate screen capture with multi-display and window-scoped targeting
Captures screenshots using ScreenCaptureKit (macOS 12.3+) with automatic CGWindow fallback, supporting Retina scaling (2x on HiDPI displays), multi-display targeting via screen index, window-scoped capture by app name/PID/window ID, and menu bar capture including status bar extras. The capture engine is abstraction-layered to allow runtime selection between ScreenCaptureKit and legacy CGWindow APIs based on availability and performance characteristics.
Unique: Dual-engine capture architecture with ScreenCaptureKit as primary (pixel-perfect, hardware-accelerated) and CGWindow fallback for older macOS versions; includes specialized menu bar capture logic that handles transient UI elements and status bar extras that standard screenshot APIs miss
vs alternatives: More reliable than generic screenshot tools because it combines two capture backends and includes menu bar awareness, enabling AI agents to see UI state that would otherwise be invisible to standard screen capture APIs
semantic ui element detection and accessibility-based interaction
Detects interactive UI elements (buttons, text fields, menus, etc.) using macOS Accessibility APIs (AXUIElement) with fallback to vision-based element detection when accessibility metadata is unavailable. The system maintains a semantic element registry that maps detected elements to their accessibility attributes (role, label, value, enabled state) and enables deterministic interaction via native accessibility actions (click, type, select) rather than pixel-based mouse movement.
Unique: Hybrid detection architecture that prioritizes accessibility APIs for deterministic interaction but seamlessly falls back to vision-based element detection when accessibility metadata is unavailable; includes element snapshot storage and cleanup system to support vision model analysis without unbounded disk growth
vs alternatives: More reliable than pure vision-based automation (e.g., Claude Computer Use) because it uses native accessibility APIs when available, avoiding coordinate drift and enabling interaction with dynamic UI; more robust than pure accessibility automation because it has vision fallback for inaccessible apps
snapshot storage and cleanup system for element detection artifacts
Manages storage of element detection snapshots (visual crops of detected UI elements) on disk with automatic cleanup to prevent unbounded storage growth. The system stores snapshots in a configurable directory, tracks snapshot metadata (timestamp, element ID, size), and implements cleanup policies (age-based, size-based, LRU). Snapshots are used by vision models to analyze specific UI elements without re-capturing the entire screen.
Unique: Automatic snapshot cleanup system with configurable policies (age-based, size-based, LRU) that prevents unbounded disk growth while maintaining snapshots for vision model analysis and debugging
vs alternatives: More efficient than manual snapshot management because it automates cleanup; more flexible than fixed retention policies because it supports multiple cleanup strategies
mac application with status bar interface and visual inspector
Provides a native macOS application (Peekaboo.app) that runs in the status bar and offers a visual inspector for debugging Peekaboo operations. The app displays real-time screenshots, detected UI elements, and execution logs; allows users to manually trigger captures and interactions; and provides a settings interface for configuration. The app maintains a persistent connection to the Peekaboo service and streams events in real-time.
Unique: Native macOS status bar application with real-time visual inspector that streams screenshots, element detection results, and execution logs; includes manual trigger interface for testing and GUI-based settings configuration
vs alternatives: More user-friendly than CLI-only tools because it provides visual feedback; more integrated than external debugging tools because it runs as a native macOS app with status bar integration
speech recognition integration for voice-based interaction
Integrates macOS native speech recognition (via Speech framework) to enable voice-based interaction with the Peekaboo agent. The system captures audio input, transcribes it to text using on-device speech recognition, and passes the transcribed text to the agent as a natural language instruction. Speech recognition runs asynchronously and supports real-time transcription feedback.
Unique: Native macOS speech recognition integration using the Speech framework with on-device transcription; supports real-time transcription feedback and asynchronous audio processing
vs alternatives: More accessible than text-only interfaces because it supports voice input; more private than cloud-based speech recognition because it uses on-device transcription
error handling system with diagnostic reporting and recovery strategies
Implements a comprehensive error handling system that captures detailed diagnostic information (stack traces, system state, screenshots) when operations fail, provides human-readable error messages, and implements recovery strategies (retry with backoff, fallback paths, state rollback). The system categorizes errors by severity and type, enabling targeted recovery logic and diagnostic reporting.
Unique: Comprehensive error handling system with categorized error types, targeted recovery strategies (retry with backoff, fallback paths, state rollback), and detailed diagnostic reporting including screenshots and system state
vs alternatives: More robust than simple error propagation because it implements automatic recovery strategies; more debuggable than black-box error handling because it captures detailed diagnostics
deterministic ui interaction via accessibility actions and synthetic input
Executes deterministic UI interactions (click, type, select, scroll, drag) using native macOS accessibility actions (AXPress, AXSetValue, etc.) when elements expose accessibility metadata, with fallback to synthetic input (CGEvent-based mouse/keyboard events) for inaccessible elements. The system maintains an interaction queue that serializes actions to prevent race conditions and includes error recovery logic that retries failed interactions with exponential backoff.
Unique: Dual-path interaction architecture that uses native accessibility actions (AXPress, AXSetValue) as primary path for reliability, with automatic fallback to synthetic CGEvent input for inaccessible elements; includes interaction queue serialization and exponential backoff retry logic to handle transient failures and race conditions
vs alternatives: More reliable than pure coordinate-based automation (e.g., pyautogui) because it uses semantic element references that survive layout changes; faster than pure vision-based interaction because it avoids repeated vision model calls for each action
window and space management with heuristic-based window selection
Manages macOS window lifecycle and space (virtual desktop) navigation using a heuristic-based window selection system that ranks windows by relevance (foreground status, recent focus, window type). The system can enumerate all windows, filter by application, activate windows, move windows between spaces, and handle window-scoped operations. Window selection heuristics account for hidden windows, minimized windows, and multiple windows from the same application.
Unique: Heuristic-based window selection system that ranks windows by relevance (foreground status, recent focus, window type) rather than simple first-match; includes specialized handling for multi-window applications and edge cases like hidden/minimized windows
vs alternatives: More intelligent than simple window enumeration because it uses heuristics to select the most relevant window when an application has multiple windows; more robust than coordinate-based window targeting because it uses semantic window references
+6 more capabilities