Comparing Two Versions Fairly

Only the disagreements carry information

When you compare prompt A with prompt B on the same cases, the cases where both pass or both fail tell you nothing about which is better. All the information is in the disagreements: cases A passed and B failed, and the reverse. The sign test asks: if A and B were really equally good, how likely is a split at least this lopsided? A small p-value (for example below 0.05) suggests a real difference. In the example, fixing 12 cases while breaking 4 (p about 0.08) is suggestive but not conclusive; 25 fixes against 3 breaks is convincing. Also look where the gains and losses are: a version that improves the average but breaks an important category (a language, high-value customers) may still be a bad trade. Always review the actual flipped cases.

A sign test on disagreements, run

I ran this with plain Python 3 (standard library only). For 12 wins against 4 losses the exact two-sided p-value is about 0.077; for 7 against 5 it is 0.774 (pure noise); for 25 against 3 it is about 0.000027 (very convincing).

from math import comb

def sign_test_p(a_wins, b_wins):
    """Two-sided exact sign test on the cases where the two versions disagree."""
    n = a_wins + b_wins
    k = min(a_wins, b_wins)
    p = 2 * sum(comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, p)

# 100 shared test cases. Disagreements only matter: v2 fixed 12 cases v1 failed, and broke 4 that v1 passed.
print("v2 better on 12, worse on 4 -> p =", round(sign_test_p(12, 4), 4))
print("v2 better on 7,  worse on 5 -> p =", round(sign_test_p(7, 5), 4))
print("v2 better on 25, worse on 3 -> p =", round(sign_test_p(25, 3), 6))

Output:

v2 better on 12, worse on 4 -> p = 0.0768
v2 better on 7,  worse on 5 -> p = 0.7744
v2 better on 25, worse on 3 -> p = 2.7e-05

Read the flipped cases

Look at the actual cases that changed; they explain the number.

Quick check: In a paired comparison, which cases carry information about which version is better?

• Those where one version passes and the other fails
• Those where both pass
• Those where both fail
• All cases equally