Asking for JSON and Validating It

Trust, but verify

Describe the exact JSON shape (keys, types, allowed values) and give one example. Many providers also offer structured output / JSON mode / schema-constrained decoding, which makes invalid JSON much less likely; use it when available. Even then, validate in code: parse the JSON, check required keys, types, allowed enum values and numeric ranges, and handle failures explicitly. Treat model output as untrusted input: never pass it to eval, shell commands or SQL without strict checks.

Machine-readable and checked

If code consumes the reply, ask for JSON, validate it and retry on failure.

A reply validator, run

I ran this plain-Python (standard library only) example. Of four sample replies, only the first passes. The others fail for different reasons: chatty text around the JSON, a category outside the allowed list, and a confidence of 1.7 outside 0 to 1.

import json

def parse_reply(raw, allowed=("billing", "technical", "other")):
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as e:
        return None, f"not valid JSON: {e.msg}"
    if not isinstance(data, dict):
        return None, "expected an object"
    if data.get("category") not in allowed:
        return None, f"category must be one of {allowed}"
    if not isinstance(data.get("confidence"), (int, float)) or not 0 <= data["confidence"] <= 1:
        return None, "confidence must be a number between 0 and 1"
    return data, None

replies = [
    '{"category": "billing", "confidence": 0.92}',
    'Sure! Here is the JSON: {"category": "billing"}',
    '{"category": "refund", "confidence": 0.8}',
    '{"category": "other", "confidence": 1.7}',
]
for r in replies:
    ok, err = parse_reply(r)
    print("OK " if ok else "BAD", ok or err)

Output:

OK  {'category': 'billing', 'confidence': 0.92}
BAD not valid JSON: Expecting value
BAD category must be one of ('billing', 'technical', 'other')
BAD confidence must be a number between 0 and 1

Quick check: Why validate JSON even with a JSON mode?

• It is only for XML
• JSON mode is illegal
• Validation makes JSON longer
• Valid syntax does not guarantee correct values or allowed categories