Case Study: A Support-Reply Drafting Endpoint

The design

Goal: agents click "Draft reply" in a support tool. Request path: the browser calls your backend (authenticated); the backend loads the ticket, pseudonymises personal data, builds the prompt (stable policy text first so it can be cached, ticket last), and calls the provider with max_tokens set, a timeout, and SDK retries with backoff. Output: the model returns JSON {category, reply}; code extracts and validates it (allowed categories, length limit), restores placeholders, escapes the text for display, and on failure retries once with the error message, then falls back to "could not draft, please write manually". Tool use (optional): a read-only get_order_status tool with the user id taken from the session, a 3-step loop cap and logging. Streaming relays text to the UI and cancels upstream when the agent closes the panel. Cost/safety: usage recorded per team and feature, daily token budget, keys in a secrets manager, redacted logs, human reviews every draft before sending. Testing: fake-server tests for retries, 400/429/5xx, streaming, tool loop and validation; a 100-ticket evaluation set run before each prompt or model change.

A dependable LLM feature

Good prompts, validated output, retries, budgets, security and tests combine into a feature you can ship.

The design on one page

Each line maps to a section of this course.

Path        browser -> YOUR backend (auth) -> provider; key never in the browser          (Sec 1)
Prompt      stable policy first (cacheable), pseudonymised ticket last, max_tokens set     (Sec 2, 6)
Output      extract JSON -> validate -> restore placeholders -> escape -> display          (Sec 2, 7)
Reliability SDK retries+backoff, timeout, 1 retry on invalid JSON, manual fallback         (Sec 5)
Tools       read-only get_order_status, user id from session, 3-step cap, logs             (Sec 4)
Streaming   relay text; cancel upstream when the panel closes                              (Sec 3)
Cost/safety usage per team, daily budget, secrets manager, redacted logs, human review     (Sec 6, 7)
Testing     fake-server tests + 100-ticket eval set before each change                     (Sec 7)

Ship behind a feature flag

Roll out to a small group first and watch cost, errors and agent feedback before widening.

Quick check: Why does the backend, not the browser, call the provider?

• So the API key stays secret and users are authenticated by you
• Browsers cannot send JSON
• It makes tokens free
• Backends are faster than browsers