# Traffic and Server Logs — AI Visibility and LLM Brand Discovery

Source: https://www.geekswithgeeks.com/en/ai-visibility/meas-traffic-logs

> Use referrers and log analysis to see AI assistants sending visitors and AI crawlers visiting.

## Two kinds of evidence

**Visitors**: in your analytics, look at referrer domains such as `chatgpt.com`, `perplexity.ai`, `gemini.google.com`, `claude.ai` and `copilot.microsoft.com`, and add UTM parameters to links you control. Note that some AI traffic arrives with no referrer and appears as "direct", so the numbers are a lower bound. **Crawlers**: your server access logs show which AI user agents fetch which pages, how often, and whether they get errors. Rising crawler hits on pricing and docs pages, and a healthy 200 response rate, are good signs that your content is reachable.

## Counting AI crawler hits, run

I ran this on five made-up log lines: GPTBot 2, ClaudeBot 1, PerplexityBot 1 and one normal browser. Real logs are far larger, and you should also check IP ranges the providers publish, because user-agent strings can be faked.

```python
import re
from collections import Counter
logs = [
 '1.2.3.4 - - [01/Oct/2026] "GET /pricing HTTP/1.1" 200 512 "-" "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"',
 '1.2.3.5 - - [01/Oct/2026] "GET /docs HTTP/1.1" 200 900 "-" "Mozilla/5.0 (compatible; ClaudeBot/1.0)"',
 '1.2.3.6 - - [01/Oct/2026] "GET /pricing HTTP/1.1" 200 512 "-" "Mozilla/5.0 (compatible; PerplexityBot/1.0)"',
 '9.9.9.9 - - [01/Oct/2026] "GET / HTTP/1.1" 200 400 "-" "Mozilla/5.0 (Windows NT 10.0) Chrome/120"',
 '1.2.3.4 - - [01/Oct/2026] "GET /blog HTTP/1.1" 200 700 "-" "Mozilla/5.0 (compatible; GPTBot/1.1)"',
]
BOTS = re.compile(r"(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Google-Extended|CCBot)")
hits = Counter()
for line in logs:
    ua = line.rsplit('"', 2)[-2]
    m = BOTS.search(ua)
    hits[m.group(1) if m else "other"] += 1
print(dict(hits))
```

Output:

```
{'GPTBot': 2, 'ClaudeBot': 1, 'PerplexityBot': 1, 'other': 1}
```

## Classifying referrers, run

Group referrers into AI assistants, search and other so you can chart AI-sourced visits over time.

```python
from urllib.parse import urlparse
AI = {"chatgpt.com": "ChatGPT", "perplexity.ai": "Perplexity",
      "gemini.google.com": "Gemini", "claude.ai": "Claude",
      "copilot.microsoft.com": "Copilot"}

def source(ref):
    host = (urlparse(ref).hostname or "").removeprefix("www.")
    return AI.get(host, "search" if host in ("google.com", "bing.com") else "other")

refs = ("https://chatgpt.com/", "https://www.perplexity.ai/search?q=x",
        "https://www.google.com/", "https://news.example/")
print([source(r) for r in refs])
```

Output:

```
['ChatGPT', 'Perplexity', 'search', 'other']
```

**Quiz:** Why might AI-referred visits be undercounted in analytics?

- [x] Some arrive without a referrer and show as direct
- [ ] AI assistants block analytics
- [ ] Referrers are never recorded
- [ ] They are always overcounted

*Answer:* Some arrive without a referrer and show as direct. Missing referrers mean reported AI traffic is a lower bound.