Computer Use Past the Usefulness Threshold

The ability for models to steer browsers is now past the threshold of being clearly useful. LLM providers have been advertising computer use capabilities for a couple of years¹, but in the times I’d tried, the results were more of a clunky curiosity than something I’d actually use on a daily basis.

The models used to get stuck quite frequently, had trouble navigating applications that extended beyond their viewport, and would struggle with websites that were even a little outside the normal expectations of how the web works. That’s not a factor anymore.

With the past couple months of model/harness progress, it seems we’re firmly on the other side of the “yeah, it just works” threshold. It isn’t just the most capable models that are good at steering browsers: I’ve been quite impressed with GPT-6.1-Terra and GPT-6-Luna’s ability to do automation.

Some example use-cases:

• Automate interactions with Amazon, since Amazon has thus far been swatting down attempts at agent-driven access.
• For that matter, interact with any service that has any sort of browser/app-based interface that isn’t otherwise easily agent-accessible.
• Search for and extract information from websites that aren’t indexed well, or use JavaScript rendering in a way that obscures information from search indices.
• Use iPhone Mirroring to do stuff on your phone from an agent.

I haven’t sufficiently explored what you can do with “actually good, fast, cheap” computer use. I suspect much of the industry hasn’t yet either. There’s currently a rush towards supporting long-running consumer-facing agents (Meta’s Muse, OpenAI’s Dots, Anthropic’s Claude Duet). This is the first obvious usage of these capabilities, and also indicates that there’s growing common knowledge that computer-use is past the threshold of being “good enough to put in front of consumers”.

However, there are undoubtedly other more creative applications of this that have not been discovered. One that comes to mind is agent-driven UI testing, which has been growing somewhat in popularity but (at least in my sphere), is still in its infancy – mostly due to cost, which is likely to rapidly decrease.

All this to say: you can now give an intelligent fuzzy text model a task and it’s able to autonomously control a computer to complete that task. Exciting times!