Systems for automated interaction with user interfaces

Describes systems and methods for automated agents to interact with user interfaces on behalf of a user. The system uses multiple ML–based agents (including LLM-based planners) to perceive the state of a webpage, determine and execute actions (e.g., navigation, variant selection, cart operations, checkout), and verify that each action aligns with user intent, constraints, and security guardrails (such as controlled handling of payment information). The approach enables reliable end-to-end task execution across heterogeneous third-party interfaces and supports agentic purchasing workflows such as Buy For Me.

Recommended citation: Lopez Martinez, Daniel; et al. Systems for Automated Interaction with User Interfaces. U.S. Patent Application (pending), filed 2025. Applicant: Amazon Technologies, Inc.