Confident?

Confidence

A recent ad shows a young person trying to converse with his grandma in her language. With the help of Gemini (and Shah Rukh :) ), he is able to converse in her native tongue. However the language that the AI proposed (usage of words like "cute") seemed a bit off. Maybe nitpicking, but there is a real gap in how models see and interact with different demographics and sometimes these fail the smell test. A seemingly unrelated case is the same Gemini hacking three companies during a security test, stopping only once it realized mid-test that the targets were real. In both cases the model followed through on an action without any real measure of how confident it should be.

As an experiment, I wished to see how local models trained on data with little Indian context, mostly English for Llama and Chinese and English for Qwen, would generalise for an Indian scenario. My friend sent me a blog on confidence published by Simile where they try to judge the simulation model they used for persona alignment, the core idea being: Can you put a number on how confident the model is about the output?

It is an interesting question, all too familiar when you ask an LLM for a confidence score (1–10) and get a number that means little. Even the model's underlying answer probabilities suffer, as the GPT-4 technical report found them well calibrated after pre-training but noticeably worse after RLHF (Reinforcement Learning from Human Feedback), which trains on which of two responses people prefer rather than on how sure the model should be. Both models in this post are post-trained on human preferences too, which is why their output probabilities are worth testing against what's inside them.

Recent weeks have brought a lot of coverage of agents behaving out of control, sharing credentials and taking actions they weren't meant to. Some of it seems to be very loose guardrailing and permissioning for the agents, but it is genuinely interesting to see agents follow through on malicious/unintended instructions, from the Gemini incident above to OpenAI's own agents breaching Hugging Face's infrastructure.

A safeguard measure for this could be to track the confidence of the actions taken by AI agents and potentially prevent any unwanted behaviour. Imagine if you are able to prevent mistakes in an agent's reasoning, before the output goes out and wreaks havoc.

Of course, this can only be observed and checked in open-weight models. Closed models might be doing something similar internally, but there's no way to verify it from the outside.

Experiment

This brings us to the experiments discussed in the Simile blog. A notable feature in the report was that a probe trained on one internal layer, using known scenarios, could estimate confidence for unseen ones. This is something that is deeply interesting as now you are able to quantify how likely the model made a correct decision. I wanted to test this on real survey data by Pew: one American and another Indian.

• OpinionQA (Santurkar et al., 2023): Pew American Trends Panel questions, US respondents, 19,063 (question, demographic-subgroup) instances.
• Pew Research's 2021 Religion in India survey: 30,000 respondents, 7,678 (question, demographic-subgroup) instances.

Here, "unseen" comes from the cross-validation split itself: each fold holds out entire questions (for Pew India) or entire survey waves (for OpinionQA), so every prediction shown below is for a (question, persona) pair the regression never saw during training.

I wanted to see how well each approach catches the mistakes made by the LLM. The simulation model predicts a distribution over possible answers, and that prediction has some error. I measure it with Total Variation Distance (TVD), which is the share of people you would have to move from one answer to another to turn the model's guessed split into the real one. If the model says 70% Yes and 30% No but the group actually split 50/50, the TVD is 0.20. A perfect guess scores 0 and a completely wrong one scores 1.

The catch is that when the model answers, the real answer isn't known yet, so this error can't be computed. Instead, a second model predicts how large the error is likely to be. It's a ridge regression (a simple linear model), trained on other questions whose real answers, and so whose errors, are already known, and it uses only information available at the moment the model answers. This is what Simile calls a confidence model. I built three such models to compare, two that read the LLM itself (from its output and from inside it) and a reference that reads only the prompt text.

The diagram above shows the three models, each ending in the same ridge regression.

• Sentence embeddings, the prompt text alone, used only as a yardstick. They appear in the first results chart, while the later figures compare just basic features and the probe.
• Basic features, the same six numbers Simile's post uses as a baseline. Two come from the prompt, the number of answer options and the prompt length. Four describe the model's guess, how spread out it is (entropy), its top probability, how peaked it is (kurtosis) and the gap between its top two options.
• Probe, one hidden layer from the same forward pass

The probe reads the model once, at the last token of the prompt where it picks its answer, with no reasoning step. Other work suggests that's a reasonable place to look, as probes read at the same point in LLM forecasters reach a score of about 0.76 on the measure explained below (Sarfati, Tiwari et al., 2026).

There are other promising approaches, like fine-tuning the LLM or RAG-style persona enhancement, but I chose the probe for the modularity it naturally offers. Its overhead is minimal and it doesn't interfere with the LLM's general performance. I'll come back to the trade-offs towards the end of the post, but the high-level view is that probes can be independently iterated on and improved for custom scenarios. Where multiple guardrails are needed, this beats trying to fine-tune the LLM in several different directions at once.

To score the confidence models, I split the simulations in half by their error. Those above the median error (about 0.4 on the Indian survey) count as bad and the rest as good, so the earlier 70/30 guess, with an error of 0.20, is a good one. AUROC (Area Under the Receiver Operating Characteristic curve) then takes one good and one bad simulation, say that 0.20 and another with an error of 0.60, and checks whether the confidence model predicts a higher error for the bad one. Repeated over every such pair, the share it gets right is the AUROC. A coin flip scores 0.5 and a perfect method 1.0. It only checks the order of the predictions, not whether they match the real errors, which is a separate property called calibration (Sarfati, Tiwari et al., 2026).

Result

Running this experiment, I found a curious result. On the American questionnaire, the probe is level with basic features for Llama and trails them for Qwen3-8B. On the Indian questionnaire, basic features weaken (Qwen3-8B's drops to near a coin flip) while the probe improves, outperforming them for both models, markedly so for the larger Qwen model.

Qwen3-8B's basic-features baseline on Pew India scores an AUROC of 0.514, and its predicted error correlates with the actual simulation error at just −0.023, indistinguishable from zero. Its output probabilities carry almost no usable information about whether it just simulated a given Indian demographic group well or badly.

Its hidden-state probe, on the same data, scores 0.816 AUROC (95% CI 0.78–0.85), correctly ranking a random accurate simulation above a random inaccurate one about 82% of the time. The gap over basic features is +0.30 (0.25–0.35).

A stricter test. Pew India asks related questions in batteries ("Do you generally… wear a janeu", "… wear a kara"), so a held-out question can have siblings in training. Holding out whole question families instead lowers every score, but the main result holds, with Qwen3-8B's basic features falling to 0.43 while its probe keeps 0.74 (gap +0.31, 95% CI 0.23–0.38). Llama's picture changes more. Its India gap shrinks to +0.07 (0.01–0.13), and on the American survey its probe now edges ahead of basic features by a similar margin (+0.06, 0.03–0.09). So the clean contrast, where output confidence holds up on American questions and breaks on Indian ones while the probe holds on both, is a Qwen3-8B finding. As a sanity check, I also trained the probe on shuffled labels, where the errors are randomly reassigned so there's nothing real to learn. This is the same control Sarfati, Tiwari et al. (2026) use. It drops to chance (0.45–0.56), as it should, so the 0.82 comes from a real link between the hidden state and the error rather than from a quirk of the setup.

Size is one likely factor, as Qwen3-8B is more than twice the size of Llama-3.2-3B. The same split showed up in a side test on factual quiz questions (MMLU), where only the 8B model's probe beat its basic features (0.822 vs 0.807, against 0.747 vs 0.761 for Llama). But the two models also come from different makers with different training data, so size can't be separated from family here. Testing a larger Llama or a smaller Qwen would settle it.

Who noticed?

Every dot below is one simulated answer. The buttons above the chart walk through it in three steps. Sort by what the model's own output says, and half of Qwen3-8B's bad simulations on the Indian survey land on the confident side, which is no better than chance. Add the probe, and 74% of them rise into its riskier half. Then look at the worst case for any safeguard, answers that looked confident but were wrong. The probe flags 1,375 of those 1,904 (72%).

What it buys you

In practice, a team would hold back the answers a signal rates riskiest. Hold back Qwen3-8B's riskiest 20% on the Indian survey and, with the probe, the average error of what's left falls by 14% (0.45 to 0.39), while with the model's own output it doesn't move. If you could see the real errors in advance and held back the truly worst 20%, the error would fall by 19%, which is the best any signal could do. For Qwen3-8B on the American survey, basic features are slightly ahead (9% vs 7%).

Possible Explanation

The result shows how much a simple model can add here. For questions outside the LLM's training distribution (the Indian demographics questionnaire), a probe trained on a set of similar questions can flag when the model's answer is likely to be off, even when its output looks confident. That matters as soon as the LLM has to act on its answer.

Take a coding model trained mostly on Python. Given a task in Ruby, it might produce plausible code, but its output gives no hint of how far to trust it. Retrieval (RAG) can help, but something still has to decide when retrieval is needed. A probe could, in principle, be that trigger, reading from inside the model that it is out of its depth in Ruby, once trained on labelled examples of its Ruby mistakes. Sarfati, Tiwari et al. (2026) suggest the same use for their forecasting probe, flagging which questions need retrieval, though they too leave it untested.

The stark difference in performance between the Indian and American questionnaires might come from the India-specific knowledge the questions require (either in training or at inference). Llama (from Meta) is trained mostly on English, largely American, text, and Qwen (from Alibaba) mostly on Chinese and English. Neither has much Indian context. Simulating how a Democrat or a Midwesterner answers a US opinion poll sits close to that training data. Simulating how a Scheduled Caste respondent or a Sikh answers whether they wear a janeu or a kara does not.

What's interesting is how the models fail. On the Indian questions they actually become more confident. Llama's average top-answer probability rises from 0.57 to 0.64 and Qwen3-8B's from 0.74 to 0.77, and since both surveys average about 3.9 answer options per question, that isn't just a side effect of fewer options. Yet that confidence largely stops tracking which simulations are right (for Qwen3-8B, almost entirely), while something inside the network still does. Sarfati, Tiwari et al. (2026) find the same in LLM forecasters and describe overconfidence as "less a failure of self-knowledge than of self-report". Catching this matters even when the stakes are as low as finding a better word than "cute" for Grandma. In the future, a similar probe could be one way to flag an agent that is about to act on an answer it shouldn't trust, though that is not something this experiment tests.

How this compares to Jev and Laya

Two "decision models" released in September 2026 take the opposite approach. Jev (TypeSafe AI, closed, API only) and Laya (open source, Apache 2.0) don't monitor an existing LLM. Instead, they are the decision-maker. You give them some text and a fixed set of options, and in one fast pass they return a probability for each option, trained so that those probabilities are calibrated from the start. Laya is a ModernBERT encoder with a small decision head that scores the hidden states at a marker before each option, trained with proper scoring rules as the reward. It's a bigger, end-to-end-trained version of the probe here, reading confidence off a hidden state rather than the output.

A probe can in principle sit on any output of a model whose weights you can access, but it only finds what the model itself encodes about being wrong, and it needs labelled mistakes from each domain. Agents add design questions too, such as which step to read the state at, and what counts as "wrong" without a clean ground truth.

Since Laya is open, I ran it out of the box on the same questions. The persona went in as the text, and the Pew question with its answer options went in as a choice question. Laya's own confidence was then scored exactly like everything else. Each row predicts the errors of a different simulator (Laya's own, Llama's or Qwen's), so read the table below as "How well can you tell when the system in front of you is wrong?"

Out of the box, Laya's confidence did no better than a coin flip at telling its good answers from its bad ones, scoring 0.519 on the Indian questions and 0.453 on the American ones. Running the basic-features regression on its outputs helps (0.618 on Pew India), but it stays well short of either probe. Laya's own docs point at a related pattern, as its English checkpoint is confidently wrong on non-Latin scripts. (Our prompts are in English, so this is an analogy rather than the same failure.)

To be fair to Laya, this isn't the job it was built for. It's calibrated to say how likely its chosen option is to be the right answer, not to estimate how a population splits across the options. I also used the base checkpoint, without the domain fine-tuning its docs recommend. (Temperature fitting, also recommended, rescales confidence, so it would mostly leave this ranking score unchanged.) Still, "calibrated out of the box" means calibrated on its training distribution, and moving it to a new domain needs labelled data, just as a probe does.

So why not just use Jev or Laya? For a fixed-option task like routing a support ticket, where you have labelled data, a decision model (a fine-tuned Laya, or Jev with well-written criteria) is probably the cheaper and simpler choice. Training calibration in and probing aren't mutually exclusive, either. Sarfati, Tiwari et al. (2026) trained Qwen3-8B with a calibration-aware recipe (DCPO), and a probe on that same model still ranked its mistakes better than the model's own token probabilities on out-of-distribution maths (AUROC 0.929 vs 0.865), though recalibrated stated confidence was slightly better calibrated (ECE 0.125 vs 0.143). Decision models can also act as an outside check. Laya ships guardrail presets that classify prompts for jailbreaks and injections, and you could ask it whether an LLM's answer looks right. But that is a second model reading the first model's text from the outside. A probe reads the internal state of the model that actually made the decision, which is where this post found a signal the output didn't show. Whether that extends to an agent writing code or taking actions isn't tested here, and it would need labelled examples of that model's mistakes in that domain. As for Jev, you can test its calibration on your own data, but you can't look inside it or attach a probe. Beyond what you can measure, its claims have to be taken on trust, the same problem as with any closed model's claims of privacy or of not training on your data.

Try it yourself

Pick a real survey question and a group, see what the model guessed and what each signal said, then reveal what people actually answered.

Future Directions

Each probe is a small regression trained on a model's past mistakes in one domain, which makes the approach modular. Several probes could run on the same model at once, for example one for survey answers and another for code. Each of them can be retrained or replaced without touching the model itself. That would also let an LLM provider offer probes as a feature that users attach to their own agents or workflows.

The next step is to go beyond single answers to tool calls and agent actions, reading the model's confidence before each step and checking whether it can be calibrated as well as ranked. There's some early evidence that probes can follow this kind of behaviour. When Sarfati, Tiwari et al. (2026) removed or planted evidence for LLM forecasters, their probe picked up how much each forecast shifted far better than the model's own written reasoning did, even in cases where the reasoning never mentioned the change. Probes that pool over every token, rather than reading only the last one as here, may help with longer actions.

On the trade-offs I mentioned earlier, probes work best as a complement to fine-tuning and retrieval rather than a substitute for them. Fine-tuning, richer personas and retrieved context all try to make the model's answers better, whereas a probe tells you when an answer shouldn't be trusted, which could in turn decide when to fetch more context or which domains are worth fine-tuning. Logging the probe's score with every answer in observability tools such as Arize would show where the model fails over time, point to where the next round of fine-tuning should focus, and show whether a new version of the model actually fails less.