Claude SERP Tracking at Scale

Analyze LLM / AI Search Results on a Large Scale

serp_claude tracks what Claude's web search actually surfaces for your queries: the pages it finds, the rank they appear in, the domains it cites, and the written answer an end user would read. Give it a query template, lists of values to fill that template, and (optionally) lists of models, locations, and tool versions; it runs every combination as its own request and returns one tidy pandas.DataFrame — one row per returned search result — so you can analyze AI search results across queries, models, and locations at scale. It is the AI-search counterpart to advertools' serp_youtube().

What you can do with it:

• See which pages and domains Claude surfaces (and cites) for a query, the way you would track a Google SERP — but for AI search / answer engines.

• Read the answer Claude gives the user, alongside the URLs it cited to build that answer.

• Compare across models and locations — does the answer or the set of cited sources change by model, country, or city?

• Track search activity in your runs (for example via

  usage.server_tool_use.web_search_requests), and which sources it leans on.

How the matrix is built

The number of requests is the full Cartesian product of four axes:

queries  ×  models  ×  locations  ×  search_tool_versions

The queries axis is itself a product: query_template is expanded over every combination of the variables you pass. So if you give two variables with 3 and 2 values, you get 3 × 2 = 6 queries, and those are then crossed with the remaining axes. This grows fast — see the note on cost below.

For example, this template plus variables:

import advertools as adv

query_template = "best {dish} restaurants in {city}"
variables = {"dish": ["sushi", "pizza"], "city": ["Lisbon", "Porto"]}

expands to 2 × 2 = 4 queries. The build_queries() helper does exactly this and returns them as a DataFrame, with a query column plus one column per variable (so you can see precisely which query maps to which values):

from advertools.serp_claude import build_queries

build_queries(query_template, variables)

                              query   dish    city
0  best sushi restaurants in Lisbon  sushi  Lisbon
1   best sushi restaurants in Porto  sushi   Porto
2  best pizza restaurants in Lisbon  pizza  Lisbon
3   best pizza restaurants in Porto  pizza   Porto

Cross those 4 queries with, say, 2 models and 1 location and you get 4 × 2 × 1 = 8 requests, each producing several result rows.

Installation & authentication

The official anthropic SDK is an optional dependency. Install it with:

pip install "advertools[claude]"     # or: pip install anthropic

The SDK is imported lazily — only when serp_claude actually builds a client — so importing advertools never requires it. Authenticate by passing api_key=... or by setting the ANTHROPIC_API_KEY environment variable.

Examples

The simplest case — a single, literal query (pass an empty variables):

import advertools as adv

df = adv.serp_claude("best running shoes 2026", {})

See which domains Claude cited most for that query:

df[df["cited_text"].notna()]["domain"].value_counts()

Read the answer Claude gave the user (the same for every row of a request):

print(df["answer"].iloc[0])

A templated query expanded over one variable (3 requests):

df = adv.serp_claude(
    "best {sport} shoes 2026",
    {"sport": ["running", "trail", "tennis"]},
)

Crossing multiple variables and two models (2 dishes × 1 city × 2 models = 4 requests):

df = adv.serp_claude(
    "best {dish} in {city}",
    {"dish": ["sushi", "ramen"], "city": ["Lisbon"]},
    models=["claude-opus-4-8", "claude-sonnet-4-5"],
)

Localizing the search with one or more user_location dicts:

df = adv.serp_claude(
    "top wedding photographers",
    {},
    locations=[
        {"country": "PT", "city": "Lisbon", "timezone": "Europe/Lisbon"},
        {"country": "US", "city": "Seattle", "timezone": "America/Los_Angeles"},
    ],
)

The result DataFrame

Every row is one returned search result. The frame is wide (template variables, result fields, citation provenance, and per-request/response metadata are all broadcast onto each row), so here is an abridged view of the most useful columns:

   query        serp_rank  title          domain         cited_text
0  best sushi…          1  Sushi Lisboa…  sushilisboa.…  "ranked #1 for…"
1  best sushi…          2  Confraria Su…  confraria.pt   NaN
2  best sushi…          3  Go Juu…        gojuu.pt       "known for oma…"

Full column inventory

category	column name	description
Query context	query	Rendered prompt sent to Claude for this request row.
Query context	<template variables>	One column per template variable (for example dish, city), copied to every result row from that request.
Search result identity	serp_rank	Position of a URL within one search result set (1 = top). Resets for each search.
Search result identity	title	Result title returned by the web search tool.
Search result identity	url	Result URL returned by the web search tool.
Search result identity	domain	Hostname parsed from url for domain-level analysis.
Search result identity	page_age	Page recency string returned by the tool, when available.
Search provenance	search_index	Which search within this request produced the row (1..N); one request can trigger multiple searches.
Search provenance	search_query	Query string Claude actually issued to the tool for that search.
Search provenance	search_via	Caller type when search was orchestrated by code execution; None for direct searches.
Search provenance	search_code	Code snippet Claude executed when search_via indicates code execution.
Search provenance	code_return_code	Return code from the related code execution block, if applicable.
Search provenance	code_stderr	Stderr from the related code execution block, if applicable.
Citation	cited_text	Snippet Claude quoted from this URL in its answer; NaN means returned but not cited.
Citation	block_index	Answer text-block index where each citation appears; repeated citations are @@-joined.
Citation	rank_within_block	Citation order inside one answer block (1..M); repeated citations are @@-joined.
Citation	rank	Global citation order across the full answer narrative (not SERP rank); repeated citations are @@-joined.
Request metadata	request_id	Internal id for one matrix row/API call.
Request metadata	model	Model requested by the plan axis.
Request metadata	tool_version	Web search tool version requested by the plan axis.
Request metadata	loc_city	Location city from the input locations axis.
Request metadata	loc_region	Location region from the input locations axis.
Request metadata	loc_country	Location country (ISO alpha-2, normalized/validated).
Request metadata	loc_timezone	Location timezone (IANA tz id, validated).
Response	answer	Claude's written answer text (what the end user reads), broadcast to all rows from one response.
Response	retrieved_at	UTC timestamp when the response was processed.
Response	response_id	Anthropic message id for joining with raw JSONL dumps.
Response	response_model	Model id Anthropic actually served (can differ from the requested alias in model).
Response	stop_reason	Anthropic stop reason for the response.
Response	stop_sequence	Anthropic stop sequence value, when set.
Usage telemetry	usage.cache_creation.ephemeral_1h_input_tokens	Cache-creation tokens in the 1h bucket.
Usage telemetry	usage.cache_creation.ephemeral_5m_input_tokens	Cache-creation tokens in the 5m bucket.
Usage telemetry	usage.cache_creation_input_tokens	Total cache-creation input tokens.
Usage telemetry	usage.cache_read_input_tokens	Input tokens read from cache.
Usage telemetry	usage.inference_geo	Inference region reported by Anthropic.
Usage telemetry	usage.input_tokens	Input token count for the response.
Usage telemetry	usage.output_tokens	Output token count for the response.
Usage telemetry	usage.output_tokens_details.thinking_tokens	Thinking token count, if reported.
Usage telemetry	usage.server_tool_use.web_fetch_requests	Number of web fetch calls used by server tools.
Usage telemetry	usage.server_tool_use.web_search_requests	Number of web search calls used by server tools in the response.
Usage telemetry	usage.service_tier	Service tier reported for the response.
Error logging	error	None for successful rows; exception text for failed request rows that are logged instead of dropped.

A note on forcing searches: there is no tool_choice for the server-side web_search tool, so a search cannot be hard-forced — the documented levers are the system prompt (soft) and max_uses (a ceiling, not a floor).

Note

Cost / size. The number of API calls is the full Cartesian product len(queries) × len(models) × max(len(locations), 1) × len(versions), and len(queries) is itself the product of every variable's value count. Each request may run up to max_uses web searches, billed by Anthropic (web search is roughly $10 / 1,000 searches, plus token costs). Inspect the plan size before launching a large matrix — there is intentionally no hard cap.

build_queries(query_template: str, variables: dict[str, list]) → DataFrame

Expand a template into the Cartesian product of its variables.

query_template placeholders must match variables keys exactly (in both directions) or a ValueError is raised. Returns a DataFrame with a query column plus one column per variable.

serp_claude(query_template: str, variables: dict[str, list], *, api_key=None, client=None, models='claude-opus-4-8', locations=None, search_tool_versions='web_search_20250305', max_uses=5, max_tokens=10000, max_workers=8, temperature=None, save_raw_path=None) → DataFrame

Get SERP-style results from Claude's web search across an input matrix.

Expands query_template over variables and crosses the resulting queries with every model × location × search_tool_version combination, running each combination as one concurrent request. Returns a tidy DataFrame with one row per returned search result; failed requests are logged in an error column rather than dropped. See the module documentation for the full description, the result-column reference, and more examples.

Parameters:

• query_template (str) -- The query to send, with optional {placeholder} fields that must match the keys of variables exactly. Use a plain string with no placeholders (and variables={}) for a single literal query.

• variables (dict) -- Mapping of placeholder name to a list of values. The template is expanded over the Cartesian product of these values (e.g. {"dish": ["sushi", "ramen"], "city": ["Lisbon"]} yields two queries). Pass an empty dict {} when the template has no placeholders.

• api_key (str, optional) -- Anthropic API key. If omitted, it is read from the ANTHROPIC_API_KEY environment variable. A ValueError is raised if neither is provided and no client is given.

• client (anthropic.Anthropic, optional) -- A pre-configured Anthropic client (custom timeout, base_url, retries, ...). Takes precedence over api_key. Most users can leave this None and let the function build one.

• models (str or list of str, default "claude-opus-4-8") -- One or more model ids to run every query against (an axis of the matrix).

• locations (dict or list of dict, optional) -- One or more user_location dicts to localize the search, or None for no location. Each dict may contain city, region, country (ISO 3166-1 alpha-2), and timezone (IANA tz id); all are validated upfront. Each dict is treated as a single unit, never mixed across rows.

• search_tool_versions (str or list of str, default "web_search_20250305") -- One or more web_search tool version strings (an axis of the matrix).

• max_uses (int, default 5) -- Maximum number of web searches Claude may run per request (a ceiling, not a floor).

• max_tokens (int, default 10000) -- Maximum tokens for each response.

• max_workers (int, default 8) -- Number of concurrent requests.

• temperature (float, optional) -- Sampling temperature. Omitted from the request unless set; newer models reject it, while older models honor it (e.g. 0 for deterministic output).

• save_raw_path (str, optional) -- Path to a .jsonl file. When set, each successful request's raw, unmodified response (response.model_dump()) is appended as one JSON line, joinable back to the DataFrame by id (the response_id column). The file is truncated once at the start of the call.

Returns:

One row per returned search result, with the template variables, result fields (serp_rank, title, url, domain, page_age), the search_query Claude issued, any quoted cited_text, Claude's answer, per-request/response metadata (model, location columns, usage.* token counts, response_id), and an error column (None on success). See the module documentation for the full column reference.

Return type:

pandas.DataFrame

Raises:

• ValueError -- If a location is invalid, if template placeholders do not match variables keys, or if no API key is found and no client given.

• ImportError -- If the optional anthropic package is not installed and no client is provided. Install it with pip install "advertools[claude]".

Examples

A single, literal query:

import advertools as adv

df = adv.serp_claude("best running shoes 2026", {})

A templated query expanded over one variable (3 requests):

df = adv.serp_claude(
    "best {sport} shoes 2026",
    {"sport": ["running", "trail", "tennis"]},
)

Crossing two variables and two models (2 × 1 × 2 = 4 requests):

df = adv.serp_claude(
    "best {dish} in {city}",
    {"dish": ["sushi", "ramen"], "city": ["Lisbon"]},
    models=["claude-opus-4-8", "claude-sonnet-4-5"],
)

Note

The number of API calls is the full Cartesian product len(queries) × len(models) × max(len(locations), 1) × len(versions) and each request may run up to max_uses billed web searches. Inspect the matrix size before launching a large run.