Citations You Can Verify

Never trust a citation blindly

A model can invent a source or attach a real source to a claim it does not support. Use layers of checks: (1) every cited ID exists among the retrieved chunks; (2) every factual sentence has at least one citation; (3) optionally, the cited text actually entails the claim (word-overlap heuristics, or a second model call or NLI model as a judge). Show citations to the user as links to the original document and page so they can verify, and log failures to improve prompts and retrieval.

A citation checker, run

I ran this plain-Python (standard library only) example. The first answer cites a source [3] that was never retrieved and has a sentence with no citation, so both problems are reported. The second answer passes.

import re

chunks = {1: "Unused leave up to 5 days can be carried over to the next year.",
          2: "Employees get 24 days of paid leave per year."}

def check(answer):
    problems = []
    for sentence in re.split(r"(?<=[.!?])\s+", answer.strip()):
        cites = [int(c) for c in re.findall(r"\[(\d+)\]", sentence)]
        if not cites:
            problems.append(("no citation", sentence)); continue
        for c in cites:
            if c not in chunks:
                problems.append(("unknown source", c))
    return problems

print(check("You can carry over 5 days [1]. Bonus leave is 10 days [3]. Weekends are free."))
print(check("You can carry over 5 days [1]."))

Output:

[('unknown source', 3), ('no citation', 'Weekends are free.')]
[]

Link to the exact page

A citation that opens the original document at the right page or heading makes verification a one-click action.

Quick check: Which check catches a fabricated source ID?

• Using shorter chunks
• Raising the temperature
• Verifying each cited ID exists in the retrieved chunks
• Removing the question