Self-Consistency: Sample Several, Take the Majority

Many paths, one answer

With sampling randomness, the model may reason correctly on some runs and slip on others. Self-consistency runs the same prompt several times (for example 5 to 10) at a moderate temperature, extracts each final answer and returns the most common one. It helps most on problems with a single checkable answer (maths, classification, extraction) and costs several times as much per question, so use it for high-value cases. If agreement is low, treat that as a signal of uncertainty and escalate to a human or a stronger method.

A majority vote, run

I ran this plain-Python (standard library only) example. Seven sampled answers (invented to show the mechanics) give 42 four times, 41 twice and 40 once; the majority is 42 with 4 of 7 in agreement, a moderate level that you might want to review.

from collections import Counter

samples = ["42", "42", "41", "42", "40", "42", "41"]
counts = Counter(samples)
answer, votes = counts.most_common(1)[0]
print(dict(counts))
print("majority answer:", answer, f"({votes}/{len(samples)} agree)")

Output:

{'42': 4, '41': 2, '40': 1}
majority answer: 42 (4/7 agree)

Quick check: What does low agreement among samples suggest?

• The API is down
• The answer is certainly right
• The prompt is too short to run
• The model is uncertain and the case may need review