# Observability: Golden Signals and Tracing — API Gateway and Service Mesh

Source: https://www.geekswithgeeks.com/en/api-gateway-service-mesh/m-observability

> Use proxy telemetry for the four golden signals and propagate trace context.

## The proxy sees everything

Because every call passes through a proxy, gateways and meshes produce consistent telemetry without changing application code: **metrics** for the four **golden signals** (**latency**, **traffic**, **errors**, **saturation**) per service and route, **access logs** with request IDs, and **distributed tracing** spans. Proxies cannot connect spans into one trace by themselves: your application must **propagate trace headers** (the W3C `traceparent` standard, or B3) from incoming to outgoing calls. Export to Prometheus/Grafana for metrics and to an OpenTelemetry-compatible backend for traces. Watch percentiles (p50, p95, p99), not averages, and alert on symptoms users feel (error rate, latency) rather than on every internal metric.

## p50 versus p99, run

I ran this plain-Python model. A skewed latency distribution (1,000 simulated calls with a seeded generator) has a median around 32 ms but a 99th percentile around 109 ms. The average would hide the slow tail that real users hit one time in a hundred.

```python
import hashlib, bisect, math, random

random.seed(7); lat = sorted(random.lognormvariate(3.5, 0.5) for _ in range(1000))
print(round(lat[499]), round(lat[989]))

```

Output:

```
32 109
```

## Propagating trace context (illustrative)

Copy the tracing headers from the incoming request to every outgoing call so the mesh and tracing backend can stitch the spans together. Header list per W3C Trace Context and common mesh practice.

```python
TRACE_HEADERS = ["traceparent", "tracestate", "x-request-id", "b3"]

def forward_headers(incoming):
    return {h: incoming[h] for h in TRACE_HEADERS if h in incoming}

# requests.get(payments_url, headers=forward_headers(request.headers))
```

**Quiz:** Why must applications propagate trace headers?

- [ ] Tracing needs no headers
- [ ] Headers make requests faster
- [ ] It is only for HTTPS
- [x] Proxies cannot link incoming and outgoing calls into one trace on their own

*Answer:* Proxies cannot link incoming and outgoing calls into one trace on their own. Only the application knows which outgoing call belongs to which incoming request.