# Atlas: app answers against API answers, the runs

Built 2026-10-18. 474 app runs (drift-10 chatgpt 30, drift-10 gemini 60, calibrate-boat chatgpt 48, calibrate-boat gemini 48, calibrate-mivi chatgpt 48, calibrate-mivi gemini 48, calibrate-freshdesk-us chatgpt 48, calibrate-freshdesk-us gemini 48, calibrate-d2c-extra chatgpt 48, calibrate-d2c-extra gemini 48); 1374 API draws (T1 90, T2 360, T3 60, T4 96, T6 384, T5 96, T7 288).

Atlas measures how AI assistants answer shopping and software questions by asking their APIs. These files hold the test
of that method: the same questions asked in the ChatGPT and Gemini apps by a person, each answer saved as the app showed
it, beside the API answers Atlas drew for the same questions. Everything we scored is here, so the agreement figures can
be recomputed or scored another way.

## Files

| File | What it holds |
|---|---|
| `questions/<set>.json` | each set's questions: id, language, room (the product or software category), template, text |
| `app-runs.jsonl` | one line per app answer |
| `api-draws.jsonl` | one line per API answer, with the app answers it is paired with |

### An app run (`app-runs.jsonl`)

`id`, `tests` (the tests that score it), `set`, `account_state` (how the app was opened: signed out, incognito, or signed
in with personalisation off; no account is named), `question`, `engine` (chatgpt or gemini), `run` (1 to 3, each a fresh
chat), `captured_at` (UTC, where the capture helper stamped it) and `capture_note`, `answer_text` (the answer as the app
showed it, list markers kept), `links` (the source links the page showed, in page order), `cards` (the product cards the
page showed, as brand, product and price; `[]` none shown, `null` not recorded).

### An API draw (`api-draws.jsonl`)

`id`, `test`, `set`, `arm` and `arm_about` (the settings in words), `question`, `engine`, `draw`, `paired_app_runs` (the
ids of the app runs it is scored against), `settings` (model, temperature with null meaning the provider's default,
whether the search tool was offered, the system instruction, the location on the search tool, OpenAI's
`search_context_size` where set, the API surface), `requested_at` and `completed_at` (UTC), `gap_seconds` (from the app
capture to the request, where recorded), `model_echoed` (the model the API named in its reply), `input_tokens`,
`output_tokens`, `search_queries` (the searches the model ran), `answer_text`, `citations` (the cited URL after
redirects, its host and its title; Gemini's grounding redirects were resolved by one request to Google and the publisher
was never fetched), `cost_usd` (at list rates on the day).

## The tests

| Test | Ledger | App side | API side | Pairing |
|---|---|---|---|---|
| T1 | D759 | drift-10, 60 runs, both apps logged out, 4 and 5 Oct 2026 | the production settings of 4 Oct (gpt-4.1-mini and gemini-3.6-flash, temperature 0.2, search on), 3 draws a question; ChatGPT again with an India location | every app run with every draw of its question and engine (9 pairs a question); app with app, the 3 runs pairwise |
| T2 | D760 | the same 60 runs | six configurations, 3 draws in each of two rounds an hour apart | every app run with every draw of its question (18 pairs a question an arm) |
| T3 | D761, D763 | drift-10, 30 Gemini runs signed in, 5 Oct 2026 | one gemini-3.6-flash draw fired at each capture (0.6 to 1.6 seconds later); one gemini-3.8-flash draw per run 12 to 47 minutes later | run r with its own draw r |
| T4 | D785 | calibrate-boat, 96 runs (24 questions, 2 runs, both apps), 7 Oct 2026; ChatGPT incognito and signed out, Gemini signed in on 3.8 Flash | the production profile drawn at each capture | run r with its own draw r; app with app, run 1 with run 2 |
| T5 | D786 | the 48 Gemini runs of T4 | gemini-3.8-flash grounded and ungrounded, 20 to 82 minutes after the captures | run r with its own draw r |
| T6 | D790 | the 96 runs of T4 | four Gemini arms (G1 the production control; G2 the default temperature; G3 temperature 0.2 with a one-line instruction stating the date, India and a shopping purpose; G4 both) and four ChatGPT arms (C1 the production control; C2 a city-level location, Dibrugarh, Assam; C3 `search_context_size` high; C4 both), drawn on 9 Oct 2026 (UTC), each app run's four arms of an engine together; the ChatGPT arms of 42 app runs were drawn that evening after a credit stop, and the 18 records drawn earlier for 10 of them were set aside and are not here | run r with its arm's draw r; each arm against its control, drawn alongside it |
| T7 | D790, D792 | calibrate-mivi (96 runs, 12 Oct 2026), calibrate-freshdesk-us (96 runs, 12 to 13 Oct 2026, a US VPN on for the whole session, the page's network read US) and calibrate-d2c-extra (96 runs, 12 Oct 2026; 24 new D2C questions about neckbands, headphones and power banks that no report asks, D792); each 24 questions, 2 runs, both apps, captured as T4 was. The two other sets D790 named (ChatGPT on a new account, boAt on a second day) were not captured | the production profile drawn at each capture: ChatGPT (chat-latest, grounded) with the set's location on the search tool and the one-line instruction (for Freshdesk the US and software); Gemini (gemini-3.8-flash at 0.2, grounded) with neither, as the profile sends them to OpenAI only | run r with its own draw r; app with app, run 1 with run 2 |

## The measures

Each measure scores a pair of answers to the same question; a cell is the mean over its scored pairs.

- **(d) First brand.** The first brand an answer names and not a brand it advises against, read against Atlas's brand
  registry. 1 when the two answers name the same brand first, 0 otherwise. A pair scores only when both answers name a
  brand.
- **(b) Brands named.** The set of brands each answer names, scored as the overlap of the two sets (Jaccard: shared
  brands over all brands named; 1.00 the same set, 0 nothing in common).
- **(c) Cited sites.** The set of hosts each answer cites, scored the same way. For ChatGPT in T4 to T7, a pair scores
  only when both answers cite. The Gemini app pages showed no sources in any test, so (c) is not computed for Gemini.
- **(e) Indian-source share.** The share of an answer's cited hosts that are Indian (Indian publishers, retailers and
  .in domains, as Atlas's list reads them). Reported per arm, not per pair.
- **(f) Cards.** Whether the app showed product cards, and whether the first card's brand was the brand named first.

The brand reading uses Atlas's entity registry: T1 to T3 were scored with the registry as it stood on 28 Sep 2026; T4 to
T7, and the pooled comparison published at supernamed.com/method, with the registry of 13 Oct 2026 (refreshed once from
the table, with the corrections ruled between 10 and 11 Oct). The answer texts here are what was scored, so any other
brand reader can be applied to them. The pooled comparison pools T4 and T7 over 82 distinct questions: a prompt that two
sets share (Mivi's set repeats 14 of boAt's word for word) counts once, and its pairs from both sets go with it.

**Intervals.** The confidence intervals Atlas reports resample whole questions (a cluster bootstrap, 10,000 replicates):
the pairs of one question move together, so the interval reflects the number of questions, not the number of pairs. A
difference between two cells resamples the same questions for both. The equivalence test asks whether the difference
lies within ±10 points by two one-sided tests at 5%, read from the same replicates.

## What is not here

Nothing that identifies the person who ran the apps: the build refuses to write if any answer, link, card or citation
carries their name or email, a line saying who is signed in or logged in, a ChatGPT chat link, a Gemini app or share
link or a Google account link, and the same check is run again on the published file. The raw app pages were deleted once parsed,
and the archive holds the parsed answer, links and cards only. No API keys. The providers' raw grounding metadata is
reduced to the search queries and the resolved citations. The cited URLs are as the models returned them, after
redirects, with their tracking and affiliate parameters removed (the section below): the archive carries no one's
affiliate or click tracking. A few answers quote a support address a brand publishes; MANIFEST.json lists those addresses.

## Tracking and affiliate parameters removed

Every URL in these files, in the answer texts, the app links and the citation lists alike, has had its tracking, click
and affiliate query parameters removed, so that the archive carries no one's affiliate or click tracking: not the
model's, not a referrer's, not a partner's. Every other parameter is kept as returned, so the page's identity survives
(YouTube's `v`, product and variant ids, search terms, locales, page state). The host never changes, so the cited-sites
measure, which reads hosts, is the same on these files as on the records they were built from. The list is
`url-tracking-params.json` in the Atlas repository (MANIFEST.json counts what each build removed); it holds:

- campaign tagging: `utm_*`, `mtm_*`, `pk_*`, `piwik_*`, `hsa_*`, `cmpid`, `s_kwcid`, `ef_id`;
- ad-platform click identifiers: `gclid`, `dclid`, `gbraid`, `wbraid`, `gclsrc`, `msclkid`, `fbclid`, `ttclid`,
  `twclid`, `yclid`, `li_fat_id`, `scid`, `yadclid`, `wickedid`, `rb_clickid`; Google Merchant Center's `srsltid`;
  Bing's `msockid`;
- affiliate-network keys and click identifiers: PartnerStack's `gspk`, `gsxid`, `ps_partner_key`, `ps_xid` and `sid1`;
  `irclickid`, `irgwc`, `clickid`, `click_id`, `cjevent`, `cjdata`, `sscid`, `epik`, `zanpid`, `awc`, `aff_id`,
  `affid`, `affiliate`, `aff_sub`, `sub_id`, `subid`; on boat-lifestyle.com `offer_id`, `pid`, `ref_id`, `sub1` to
  `sub3`; Amazon's `tag`, `ref`, `ref_`, `linkCode`, `ascsubtag`, `qid`, `crid`, `pd_rd_*`, `pf_rd_*`; Apple's `at`,
  `ct` and `fnode`;
- email-campaign recipient identifiers: `mc_cid`, `mc_eid`, `_hsenc`, `_hsmi`, `hsCtaTracking`, `mkt_tok`, `vero_id`,
  `ck_subscriber_id`, `oly_anon_id`, `oly_enc_id`, `vgo_ee`, `_kx`; Google Analytics linkers `_ga`, `_gl`, `_gac`;
- share, referral and storefront tracking: `igshid`, `si`, `spm`, `trk`, `trkInfo`; Shopify's `_fid`, `_pos`, `_ss`,
  `_sid`, `_psq`; on apps.shopify.com `st_source`, `search_id` and `surface_*`; on salesforce.com `bc` and `c1um1`;
  on gorgias.com `ref-position`.

## Licence

These files are licensed under the Creative Commons Attribution 4.0 International licence (CC BY 4.0,
https://creativecommons.org/licenses/by/4.0/). You may share and adapt them for any purpose, provided you credit
Supernamed (supernamed.com) and name this archive and its build date. The answer texts are what ChatGPT, Gemini and
their APIs returned; the cited pages belong to their publishers.