unified-cross-platform-device-abstraction
Provides a single Robot interface abstraction layer that normalizes interactions across Android (physical devices and AVD emulators), iOS (physical devices via USB), and iOS Simulators (via xcrun simctl). The architecture uses platform-specific manager implementations (AndroidRobot, IosRobot, SimctlManager) that all conform to a common Device API contract, eliminating the need for agents to understand platform-specific tool invocation patterns. Device resolution is request-scoped and stateless, with each tool call resolving the target device parameter through getRobotFromDevice() to the appropriate platform manager.
Unique: Uses a request-scoped, stateless Robot interface pattern that dynamically resolves platform managers at invocation time rather than maintaining persistent device connections, enabling horizontal scaling and multi-device orchestration without session management overhead. The common Device API contract ensures all platform implementations (ADB-based Android, WebDriverAgent-based iOS, simctl-based simulators) expose identical method signatures.
vs alternatives: Unlike Appium (which requires separate server instances per platform) or Detox (which is iOS-focused), mobile-mcp provides true platform-agnostic automation through a unified MCP protocol interface that works with physical devices, emulators, and simulators without configuration changes.
accessibility-tree-based-ui-element-detection
Extracts and parses native accessibility trees from both Android (via ADB accessibility service) and iOS (via WebDriverAgent accessibility API) to enable deterministic, coordinate-free UI interaction. The system builds a hierarchical representation of UI elements with semantic labels, roles, and bounds, allowing agents to locate and interact with elements by accessibility properties rather than fragile pixel coordinates. Falls back to screenshot-based coordinate tapping only when accessibility data is unavailable, providing a two-tier interaction strategy that prioritizes semantic stability.
Unique: Implements a two-tier interaction strategy that prioritizes native accessibility trees (Android AccessibilityService, iOS WebDriverAgent accessibility API) as the primary interaction mechanism, with screenshot-based coordinate fallback only when semantic data is unavailable. This approach provides deterministic, layout-resilient automation that survives UI changes without requiring coordinate recalibration.
vs alternatives: Outperforms image-based automation tools (like Appium with image recognition) by using semantic accessibility metadata for element location, eliminating the need for ML-based visual matching and providing 100% deterministic element identification when accessibility labels are present.
webdriveragent-session-management
Manages WebDriverAgent session lifecycle for iOS devices (both physical and simulators) including session creation, teardown, and error recovery. The WebDriverAgent client (src/webdriveragent.ts) handles HTTP communication with WebDriverAgent endpoints, session initialization with app bundle IDs, and timeout management. The system maintains session state per device and automatically re-establishes sessions on failure. Session management is abstracted from agents — they invoke Robot interface methods without understanding WebDriverAgent protocol details. The implementation handles both localhost communication (simulators) and USB tunnel communication (physical devices) transparently.
Unique: Abstracts WebDriverAgent session lifecycle (creation, teardown, error recovery) behind the Robot interface, allowing agents to invoke iOS automation without understanding WebDriverAgent protocol or session management details. Handles both localhost (simulator) and USB tunnel (physical device) communication transparently.
vs alternatives: Simpler than managing WebDriverAgent sessions directly (no protocol knowledge required) while providing automatic recovery on timeout, making it suitable for LLM agents that need straightforward iOS automation without WebDriverAgent expertise.
image-processing-and-screenshot-analysis
Provides image processing utilities for screenshot analysis, including screenshot capture, image format conversion, and visual element detection support. The system captures screenshots from devices through platform-specific mechanisms (ADB screencap for Android, WebDriverAgent screenshot API for iOS) and processes them through image utilities for format conversion and metadata extraction. The implementation supports PNG and JPEG formats and provides hooks for visual element detection (though advanced CV/ML-based detection is not built-in). Screenshots are used as fallback when accessibility tree data is unavailable and for visual validation workflows.
Unique: Integrates screenshot capture as a secondary interaction tier with image processing utilities, providing visual fallback when accessibility trees are unavailable while maintaining performance for well-instrumented apps. Screenshot processing is platform-agnostic, supporting both Android (ADB screencap) and iOS (WebDriverAgent) capture mechanisms.
vs alternatives: Provides pragmatic screenshot support for fallback scenarios without requiring external image processing libraries, though it lacks advanced CV/ML capabilities for visual element detection compared to specialized visual automation tools.
app-lifecycle-management
Provides app installation, launch, termination, and state management capabilities across Android and iOS platforms. On Android, app lifecycle is managed through ADB commands (adb install, adb shell am start, adb shell am force-stop). On iOS, app lifecycle is managed through go-ios (for physical devices) and simctl (for simulators). The system supports app installation from APK/IPA files, launching apps with intent/URL parameters, and force-stopping/terminating apps. App state is managed per device, allowing agents to control app lifecycle as part of automation workflows.
Unique: Provides cross-platform app lifecycle management through platform-specific mechanisms (ADB for Android, go-ios/simctl for iOS) abstracted behind a common Robot interface, allowing agents to manage app installation and launch without platform-specific knowledge.
vs alternatives: Simpler than app-specific testing frameworks (Espresso, XCUITest) for basic app lifecycle management, making it suitable for agents that need straightforward app installation and launch without framework overhead.
screenshot-and-coordinate-based-interaction
Captures full-screen screenshots from the device and enables coordinate-based interaction (tap, swipe, drag) when accessibility tree data is unavailable or insufficient. The system processes screenshots through image processing utilities to extract visual information, then maps agent-specified coordinates or visual regions to device touch events. This provides a fallback mechanism for apps with poor accessibility implementation or for visual-based automation scenarios where semantic interaction is not viable.
Unique: Implements screenshot capture as a secondary interaction tier that activates only when accessibility tree data is unavailable, reducing screenshot overhead for well-instrumented apps while maintaining fallback capability for legacy or third-party apps. Screenshot processing is integrated with the common Device API, allowing agents to seamlessly switch between semantic and coordinate-based interaction.
vs alternatives: Provides a pragmatic hybrid approach compared to pure accessibility-based tools (which fail on inaccessible apps) or pure image-based tools (which are slow and fragile) — using accessibility as primary with screenshot fallback ensures broad app compatibility while maintaining performance for well-instrumented applications.
android-adb-device-automation
Implements AndroidRobot class that wraps Android Debug Bridge (ADB) for controlling physical Android devices and AVD emulators. The implementation handles ADB command execution, device state management, accessibility service integration for UI tree extraction, and gesture simulation (tap, swipe, long-press) through ADB input events. Device discovery and management is handled by AndroidDeviceManager, which enumerates connected devices via 'adb devices' and maintains device-specific state. The architecture abstracts ADB complexity behind the common Robot interface, allowing agents to control Android devices without direct ADB knowledge.
Unique: Wraps ADB command execution within a stateless Robot interface that handles device discovery, accessibility service integration, and gesture simulation without requiring agents to understand ADB protocol details. AndroidDeviceManager provides automatic device enumeration and resolution, eliminating manual device serial number management.
vs alternatives: Simpler than Appium for basic Android automation (no server setup required, works with standard ADB) while providing accessibility tree extraction comparable to Espresso, making it ideal for LLM agents that need straightforward device control without framework overhead.
ios-physical-device-automation-via-go-ios
Implements IosRobot class that controls iOS physical devices (iPhone, iPad) connected via USB using the go-ios tool for device communication and WebDriverAgent for UI automation. The architecture uses go-ios for low-level device operations (device discovery, app installation, log streaming) and WebDriverAgent (a native iOS testing framework) for UI interaction and accessibility tree extraction. Device management is handled by IosManager, which discovers connected iOS devices via go-ios and maintains WebDriverAgent session state. The implementation abstracts the complexity of USB tunneling, WebDriverAgent session management, and iOS-specific constraints behind the common Robot interface.
Unique: Combines go-ios for device-level operations with WebDriverAgent for UI automation, providing a lightweight alternative to Xcode-dependent tools. The architecture handles WebDriverAgent session lifecycle (creation, teardown, error recovery) transparently, allowing agents to treat iOS physical devices as simple automation targets without understanding WebDriverAgent protocol details.
vs alternatives: Lighter than XCUITest-based approaches (no Xcode required) while providing comparable UI automation capabilities through WebDriverAgent, making it accessible to non-iOS developers and LLM agents that need straightforward iOS device control.
+5 more capabilities