Same question, different answer. That’s a dial, not a bug.

Send an AI the same message twice and you’ll usually get two different replies. People read that as flakiness, or as proof the thing is making it up as it goes. The second part is closer to true than you’d think, but not in the way it sounds. The variation is designed in, it’s adjustable, and once you know where it comes from you can use it instead of being unsettled by it.

One word at a time, from a ranked list

A model doesn’t compose a whole answer and then type it out. It produces text in small pieces called tokens, roughly a short word or part of a longer one. At every step it looks at everything so far and scores every possible next token: how likely is each one to come next?

For “The capital of France is”, that list is lopsided. One option towers over the rest. For “Write an opening line for a toast at my sister’s wedding”, the list is flat. Dozens of starts are about equally reasonable, and none of them is the right one.

The model then picks one token from the list, appends it, and does the whole thing again for the next token. That picking step is called sampling, and it’s where the variation comes from. If the model always took the single highest-scoring token, the same input would give the same output every time. Most apps don’t do that, on purpose.

Why not always take the top pick?

Because always choosing the most likely next word produces flat, repetitive writing. It drifts toward the safest phrasing, reuses the same constructions, and can loop on itself. Letting the model occasionally take the second or fifth choice is what makes its writing sound like writing.

Temperature is the dial for that. Low temperature sharpens the list, so the top choices win almost every time and answers come out close to identical run after run. High temperature flattens it, so lower-ranked words get a real chance and answers spread out. Apps usually add a second guardrail that cuts off the long tail of very unlikely words entirely, so even a hot setting doesn’t pick something absurd.

None of this is the model thinking harder or less hard. It’s the same model producing the same ranked lists. Temperature only changes how the pick is made from them.

Variety helps drafts and hurts facts

That trade explains when you want the dial up and when you want it down.

For a birthday message, a caption, a reply to an awkward text, or a first line for a cover letter, the flat list is the point. There isn’t one correct answer, and a second run is a free second opinion. The useful habit is to ask again and take the parts you like from each.

For pulling the dates out of an email, turning notes into flashcards, or listing what’s actually in a document, you want the opposite. The answer should be the same on Tuesday as it was on Monday, and creative word choice is just noise.

Vault AI makes that call per tool, so you don’t have to. Its writing, creative and social tools run warmer, for range. Its extraction, study, business and language tools run cooler, for consistency. And the list-making tools go a step further: they ask the model for a structured list rather than free text, so the shape of the answer is fixed even when the wording varies. There’s no temperature slider in the app. That’s a deliberate simplification, and if you want to tune sampling by hand, a desktop tool will serve you better.

When two answers disagree, believe neither

Here’s the most useful thing sampling teaches you. When a model really knows something, the ranked list is lopsided and the top answer wins no matter how the pick is made. Ask three times and you get the same fact three times, worded differently.

When it doesn’t know, the list for that part of the answer is flat. A year, a name, a dosage, a statistic — several candidates score about the same, and the pick lands on a different one each run. Every version sounds equally confident, because confidence is a property of the writing style, not of the knowledge.

So the cheapest fact-check there is: ask the same factual question in two fresh chats. If the specific detail changes between them, the model was guessing. This matters more on a phone, because a small model holds far fewer facts to begin with — the reason is in why a small model forgets facts but never forgets how to write. It’s not a guarantee in the other direction, though. A model can repeat the same wrong answer consistently if it learned it wrong. Agreement is weak evidence; disagreement is strong evidence.

The better fix is the one in give it the text, not the question: paste in the source. When the answer is sitting in the material you supplied, the ranked list is lopsided again, and the model copies instead of guessing.

Where the cloud genuinely does better

Large cloud models know far more, so their lists are lopsided on many more facts, and they agree with themselves more often on questions a phone model would guess at. Many cloud assistants can also search the web and cite where an answer came from, which is a real check that sampling tricks can’t replace. Some let you adjust sampling directly. If the job is factual recall on topics you can’t verify yourself, use one of those and check its sources.

For the everyday half of AI use — drafting an email, rewriting a message, turning notes into flashcards — the variation is either a feature you want or something a cooler setting already handles. Those tasks run on your phone, offline, with nothing sent anywhere. And if a long chat starts to wander, that’s a different mechanism, covered in an AI chat doesn’t remember, it rereads.