Agent Computer Interface Aci With Visual And Text Grounding

via “agent-computer interface (aci) for llm-codebase interaction”

Princeton's GitHub issue solver — navigates code, edits files, runs tests, submits patches.

Unique: Introduces Agent-Computer Interface as a domain-specific abstraction layer specifically optimized for code tasks, reducing token usage by ~40% vs raw shell access while maintaining safety through command validation and sandboxing

vs others: More efficient than ReAct-style agents that use raw bash because ACI provides semantically meaningful, code-aware commands rather than generic shell operations

via “agent-computer interface (aci) with visual and text grounding”

Agent S: an open agentic framework that uses computers like a human

Unique: Defines a pluggable ACI abstraction with native support for visual and text grounding through OCR integration and coordinate system transformations, enabling agents to ground LMM outputs to precise screen coordinates while supporting multiple platform implementations

vs others: Provides more flexible grounding than DOM-based approaches (works with any application) while being more reliable than pure visual reasoning by combining OCR text extraction with coordinate mapping