OpenAI has developed an AI assistant for computer control and browser automation

OpenAI has introduced a new AI agent called Operator, designed to automate actions in the browser. The tool can interact with interface elements such as buttons, text fields, and scrolling, imitating user actions.

The basis of the work of Operator was the Computer-Using Agent (CUA) model, which combines the image recognition capabilities of GPT-4 with an advanced analysis and decision-making mechanism. The algorithm works in stages: first, a screenshot is created, then the system analyzes it, determines the necessary actions and simulates them using virtual mice and keyboards. Users can observe the process through a small window in the browser.

At the moment, Operator shows the best results in performing routine and repetitive tasks, such as making shopping lists or playlists. However, the agent faces difficulties when working with unfamiliar interfaces, for example, tables, calendars, or when editing complex texts.

Although the technology is in its early stages of development, it promises to be a powerful tool for automating routine processes and working with the browser.