Streaming in Applications: UI, Backends and Cancellation

Your backend is the middleman

Browsers must not hold your API key, so the browser calls your backend, which calls the provider with streaming and forwards the pieces to the browser (for example over Server-Sent Events or WebSockets). Design for the unhappy paths: the user closes the page or presses stop (cancel the upstream request so you stop paying for tokens), the stream fails midway (show what arrived and offer retry; do not silently treat partial text as complete), a proxy buffers the stream (disable buffering or the text arrives in one lump), and moderation or validation that needs the full reply (validate after the stream ends, or stream only after a safety check). Record usage from the final event for billing.

A streaming relay (illustrative)

Sketch of a backend endpoint in the style of a web framework; function names are placeholders. Not run here.

def chat_stream(request):
    user = authenticate(request)                       # your auth, not the provider key
    check_rate_limit(user)
    def events():
        try:
            with client.messages.stream(model=MODEL, max_tokens=500,
                                         messages=load_history(user, request)) as stream:
                for piece in stream.text_stream:
                    if request.is_disconnected():          # user pressed stop / closed the page
                        return                              # leaving the block closes the upstream request
                    yield f"data: {piece}\n\n"
                final = stream.get_final_message()
                record_usage(user, final.usage)             # bill from the final event
        except Exception:
            yield "event: error\ndata: stream failed, please retry\n\n"
    return EventStreamResponse(events())

Quick check: Why cancel the upstream request when the user presses stop?

• So you stop paying for tokens nobody will read
• To delete the key
• To speed up the model
• It is required by HTTP