We ran GPT-6 Astra over an internal browser automation benchmark. Its overall success was similar to other frontier models, but its response pattern was striking.
Astra declined to answer more often than other models, but when it did answer, it was more likely to be right.
For healthcare agents, that tradeoff is extremely valuable. An explicit abstention gives us options (like routing to a human). A confident wrong answer is much harder to work with.
The setup: medical license verifications
Before a clinician can treat patients or bill an insurer, someone has to confirm their license is genuine and in good standing. The answer has to come from the official website of the relevant state board. The problem is that each state runs a different website for each medical profession. And each website is twisted in its own unique way.
This variety makes for an extremely annoying administrative task but an excellent benchmark for browser automation.
We measured two sub-tasks: find the right website (“search”) and verify the license (“verification”). We ran each model on 181 cases across 26 portals, representing 16 boards and 17 profession types (doctor, nurse, social worker, etc). We tested GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, Claude Opus 5 and GPT-5 mini.
Fable and Astra were the best performers, but Astra’s results were the most unique.
Astra had two modes: 🤷 and 🎯
The primary metric we looked at was success rate: how often did the model get the right answer? Astra and Fable performed nearly identically on verification (88% and 89%, respectively), whereas Fable outperformed Astra on search (93% and 78%, respectively).
However, the way they got there was entirely different. We examined two component metrics:
- Commit rate: How often did the model provide an answer?
- Precision: How often was that answer right?
Success rate = commit rate x precision
Astra had a much lower commit rate (it said “I don’t know” more often) and a much higher precision (when it did answer, it was more likely to be right). For example, on search, Astra found a website only 78% of the time but was right every time. Fable found a website every time but was only right 93% of the time.
Reliable abstention is a useful workflow primitive
A lot of healthcare workflows are human-in-the-loop. Thus, models that can reliably escalate to people are more valuable than ones that are confidently wrong and need more oversight.
It’s also easier to design agents to account for lower commit rates. You can selectively push the model to “try harder” in those cases (i.e., spend more tokens) or conditionally take a different route or action.
There are drawbacks to being conservative
Astra missed some cases that every other model got right. The Massachusetts social work portal returns the clinician’s middle initial. Astra saw this and concluded it was a mismatch because the benchmark inputs don’t have a middle name or initial to compare to. In practice, this is a common case that would cause extra manual work.
Caveats and takeaways
These results came from a single internal benchmark. While it covers a broad range of cases, setup details like turn limits, prompts, and input formats can unintentionally benefit one model over another. Still, we saw similar behavior in other internal tests. Astra demonstrates a consistent pattern of restraint, combining low commit rates with high precision. This is a valuable combination in healthcare and makes it an exciting model for agent workflows.
