TypeSafe Confidence and Calibration in 2026: Setting Thresholds That Work

What Is Confidence in TypeSafe in 2026?

Every Choice and Score answer includes a probabilities property holding the distribution across your options or levels. The shape of that distribution is what tells you how certain the model is: concentrated on one outcome means a confident answer, spread out means an uncertain one. The confidence property collapses that shape into a single number from 0 to 1 so your code can threshold on it without doing the maths. Noul answers do not carry one.

Confidence is therefore not an extra opinion from the model. It is a statistic computed from the distribution the answer already gives you. TypeSafe computes it so the common case needs no extra work on your side, but the raw probabilities are always there if you want to compute something else.

Explain It Like I Am Five: The Torch Beam

How Is Confidence Calculated in 2026?

Confidence is derived from how the probability is spread across the options or levels. All of the probability on one option gives 1.0, and the more evenly it spreads, the lower the confidence. TypeSafe's documentation includes an interactive demo that approximates confidence for a three-option Choice as (3 × largest probability − 1) / 2, and notes this is the demo's approximation rather than a general formula.

Work that through and the intuition arrives quickly. With three options, a probability of 1.0 on one gives confidence 1.0. An even split of one third each gives confidence 0.0, because no option is a winner. The documented demo shows 90 percent, 6 percent and 4 percent giving a confidence of 0.85.

What low confidence means differs by primitive, and TypeSafe spells it out. Low confidence on a Choice often means none of the options is a clear winner over the others. Low confidence on a Score often means the levels are ambiguous, multi-dimensional, or the state does not contain enough to go on. Those are three different bugs in your question design, and each has a different fix.

Why Is Being Able to Say "I Do Not Know" Valuable?

TypeSafe puts this as a principle rather than a feature: if an intelligent system, whether human or machine, cannot express honest uncertainty, the system cannot be trusted. Confidence gives the model a built-in way to say it is not sure about this one, which lets your code implement different behaviour for different levels of certainty.

That is the foundation for systems you can rely on, and it is the practical difference from a text model that sounds equally assured whether it is right or guessing. An uncertain answer is not a failed answer. It is a correctly reported state of the world, and the system around it decides what to do about it.

What Does Calibration Promise, and What Does It Not?

Calibration is the property TypeSafe trains for with RLCD, reinforcement learning for calibrated decisions. Its stated contract is that higher probability should correspond to a greater chance that the answer is correct, and specifically that across many predictions from a well-calibrated model, outcomes assigned 0.2 should occur about 20 percent of the time, outcomes at 0.8 about 80 percent of the time, and outcomes at 1.0 should occur 100 percent of the time.

Then comes the sentence every team should paste into its own internal documentation: these rates describe groups of predictions, not a guarantee about any single answer. TypeSafe repeats the point elsewhere, noting that calibration is measured across groups of predictions and does not guarantee that an individual answer is correct.

How Should Your Code Use Confidence in 2026?

TypeSafe's documented starting pattern divides confidence into three ranges, each producing a different system behaviour. It is deliberately simple, and it is the right shape for a first implementation.

Where you draw those boundaries depends on the stakes, and TypeSafe is explicit that a confidence threshold is not one number. Different actions within the same system should be gated at different levels depending on the consequences of getting it wrong. It suggests a 0.5 floor to catch anything the model reports as genuinely uncertain, with the threshold for acting without confirmation set higher for a destructive operation than for a read-only one.

How Does Confidence-Gated Routing Work in Practice?

TypeSafe's documented example is a voice banking interface, which is a good choice because the actions have obviously different stakes. One Choice question classifies the intent, and the code then applies a different confidence bar to each action.

action = response.answers["intent"]

# Below 0.6 confidence on any action, route to a human
if action.confidence < 0.6:
    route_to_support_agent(account_id)

elif action.choice == "check_balance":
    # Low stakes. 0.6 confidence is sufficient.
    show_balance(account_id)

elif action.choice == "approve_transfer":
    if action.confidence > 0.85:
        # High stakes, but high confidence. Safe to act automatically.
        approve_transfer(account_id)
    else:
        # High stakes, moderate confidence. Verify intent first.
        ask_user_to_confirm("Just to confirm: you would like to approve this transfer?")

else:
    route_to_support_agent(account_id)

The 0.6 floor catches anything genuinely uncertain. Above it, checking a balance at 0.6 is fine because the worst case is the customer hearing their balance unnecessarily. Approving a transfer needs above 0.85, and anything between goes to a confirmation. One model answer, three different behaviours, all decided by ordinary code you can read in a review.

How Do You Choose the Right Threshold for Your Data?

Empirically, not by instinct. TypeSafe's build guide says to test thresholds by plotting confidence against accuracy on your data, and that is the only method that actually tells you anything about your material. The procedure is short enough to do in an afternoon.

1. Label 100 to 300 real examples yourself. Real ones from your product, including the awkward ones.
2. Run them all in one batch. Record the answer, the full probability distribution and the confidence for each.
3. Bucket by confidence. Group into bands such as 0.9 and above, 0.7 to 0.9, 0.5 to 0.7, and below 0.5.
4. Measure accuracy per band. This is the number that tells you where automatic action becomes safe.
5. Set each action's threshold from its own cost of being wrong. A reversible action tolerates a lower bar than an irreversible one.
6. Count how much lands in the middle band. That volume is your human review workload, and it needs to be a number someone has agreed to staff.
7. Re-run when the model version moves. Aliases move on their own, so thresholds tuned against one version deserve re-checking against the next.

When Should You Not Use a Confidence Threshold?

TypeSafe's own troubleshooting notes flag this, and it corrects a habit teams pick up quickly. If all you care about is choosing the best option, you simply take the option the model selected; you do not need a threshold at all. And if you have a specific statistical algorithm in mind, you should probably be using the raw probabilities rather than the collapsed confidence value.

Two more cases where confidence is the wrong instrument. A Noul has none, so thresholding a Noul means thresholding the noul value itself, which is a probability rather than a certainty measure. And when a second option carries meaningful probability, the useful signal is that probability, not the headline confidence: TypeSafe's Choice documentation shows a ticket routed to returns while billing still gets a copy because its 0.35 share crossed a 0.25 threshold.

What Does a Low-Confidence Answer Tell You About Your Question?

Often more than it tells you about the input. Persistent low confidence across many inputs is usually a question design problem, and the fix depends on the primitive.

That last row matters. TypeSafe's Score documentation shows a case where adding a well-matched example moved confidence from 0.35 to 0.96, and states plainly that higher confidence does not establish which answer is correct. Treat a confidence jump as a prompt to re-test, not as proof.

What Are the Common Confidence Mistakes in 2026?

• Treating confidence as a probability of correctness for one case. Calibration describes groups of predictions.
• Using one threshold across a whole system. Gate each action at a level matched to its consequences.
• Sending the uncertain middle to a default branch. If nobody looks at it, you have a silent failure queue rather than a review queue.
• Thresholding when you only need the top option. If the decision is simply "which one", take the choice.
• Ignoring the runner-up probability. A second option at 0.35 is information your code can act on.
• Carrying a Noul threshold over to a Choice. TypeSafe warns that these answer different questions and are not comparable.
• Never re-checking after a version change. An alias moves on its own; thresholds tuned against one version should be re-validated.

Key Takeaways for 2026

• Confidence is a 0 to 1 statistic derived from how peaked the probability distribution is, returned on Choice and Score but not on Noul.
• Calibration means stated probabilities match real frequencies across groups of predictions, and explicitly does not guarantee any single answer.
• The documented starting pattern is three bands: act, verify, escalate, with a 0.5 floor for genuine uncertainty.
• Thresholds are per action, not per system, and should reflect the cost of being wrong.
• Choose them by plotting confidence against accuracy on your own labelled data, and staff the middle band deliberately.
• If you only need the top option you need no threshold; if you need statistics, use the raw probabilities.
• Persistent low confidence is usually a question design problem, and the fix differs for Choice and Score.

Distk sets these thresholds with clients the boring way: label a real sample, measure accuracy per confidence band, and agree who staffs the middle before anything ships.

Sources

• TypeSafe AI docs: Confidence
• Confidence-gated routing
• AI primer: calibration and RLCD
• Primitives (Questions)
• Score
• Choice
• How to build with TypeSafe
• Agent skill: common issues