The drift test, in plain language: what the apps answer against what the API answers

Sixty app runs captured logged out (ten questions, ChatGPT and Gemini, three fresh chats each, 4 and 5 Oct 2026) against sixty API draws with Atlas's production settings (gpt-4.1-mini and gemini-3.6-flash, temperature 0.2, grounded), plus a third arm for ChatGPT alone with an India location on its search tool.

• ChatGPT. Two app runs agree on the first brand 0.63 of the time and on the brands named 0.63 (30 pairs). An app run and an API draw agree 0.38 and 0.45 (90 pairs). On cited sites the app agrees with itself 0.30 and with the API 0.05. The India location moved the API's own citation mix (India share 0.43 to 0.55) and not its agreement with the app (first brand 0.38 to 0.37, brands 0.45 to 0.41, sites 0.05 to 0.05).
• Gemini. App with app 0.80 on the first brand and 0.80 on brands (30 pairs); app with API 0.46 and 0.62 (90 pairs). The cited-sites row reads 1.00 for app with app and 0.30 for app with API only because the logged-out app pages cited nothing at all (check-runs.txt: no links on 30 of 30 Gemini pages, every one signed out), so an empty set matches an empty set and the API row measures only whether the API also cited nothing.
• Cards. The ChatGPT app showed cards on 0.73 of its runs (30 scored) and the first card's brand was the brand named first 0.64 of the time (22 scored). The Gemini app showed none (0.00 of 30).

What it leaves untested: one account state (logged out) on one day; the app's agreement with itself comes from three chats minutes apart, never an hour apart; the API's location arm exists for ChatGPT only, since Gemini's search tool takes no location; and on Gemini the cited-sites measure says nothing, the app having cited nothing.

Six API arms, each drawn three times per question in two rounds an hour apart (the actual gaps, from the records: Gemini 60 minutes on every arm, ChatGPT 78 to 85 minutes after three credit stops), measured two ways: against the app runs of section 1 (the surface, 18 pairs a question) and against themselves an hour later (the clock, 9 pairs a question). The arms: ChatGPT P (production gpt-4.1-mini, temperature 0.2, no location), M (the app's own model as the API lists it, chat-latest, default temperature, an India location), S (M plus a short system prompt stating the date, India and a shopping purpose); Gemini P (production gemini-3.6-flash, grounded), U (the app's then model, gemini-3.5-flash-lite, grounding off), G (the same, grounded). The whole test cost $15.40 at list rates.

• ChatGPT, agreement with the app (mean of first brand, brands named, cited sites): P 0.30, M 0.35, S 0.37, so the arm closest to the app is S. Measure by measure: first brand P 0.40, M 0.44, S 0.54; brands named 0.47, 0.51, 0.48; cited sites 0.05, 0.10, 0.08. By language, first brand against the app: S English 0.53 (66 of 72 pairs scored), Hinglish 0.75 (51 of 54), Hindi 0.35 (51 of 54); P English 0.54 (72 of 72), Hinglish 0.61 (54 of 54), Hindi 0.00 (54 of 54).
• ChatGPT, agreement with itself an hour apart: P 0.74, M 0.54, S 0.45 (first brand 0.91, 0.67, 0.59; brands 0.75, 0.61, 0.52; sites 0.57, 0.33, 0.23). The production model repeats itself far more than the app's model does; the app's model, with or without the prompt, moves between rounds about as much as it differs from the app.
• Gemini, agreement with the app (first brand and brands named; cited sites not computable, section 1): P 0.54, U 0.39, G 0.48, so the production arm is the closest. Measure by measure: first brand P 0.43, U 0.31, G 0.37; brands 0.65, 0.47, 0.58.
• Gemini, agreement with itself an hour apart: P 0.66, U 0.70, G 0.66 (first brand 0.67, 0.70, 0.77; brands 0.76, 0.69, 0.75; sites 0.55, not computable for U which cites nothing, 0.48).

What it leaves untested: the app side is the same 60 logged-out runs, so the app's own hour-apart stability is not measured; chat-latest is the API's name for ChatGPT's model and may not be what the app's "Auto" picks; the location and the system prompt are guesses at what the app sends; Gemini's "app model" arms used 3.5 Flash-Lite, the label the logged-out pages carried, while the signed-in app later reported 3.8 Flash (section 3); and the ChatGPT rounds were 78 to 85 minutes apart, not 60.

Thirty Gemini app runs captured signed in on 5 Oct 2026 (Apps Activity off, Personal Intelligence off, the app's picker reading 3.8 Flash), each capture firing one draw of the production arm (gemini-3.6-flash, grounded) within 0.6 to 1.6 seconds (gap_seconds in each signed-in-runs/<q>/api-<run>.json), so every app run has a draw made at the same moment. Later the same morning, one draw per app run of gemini-3.8-flash, the model the app named, 12 to 47 minutes after the captures (arm-3.8-cost.txt), for $1.01.

• Pooled, 30 pairs each way. The app against itself: first brand 0.70, brands named 0.75. The app against its simultaneous 3.6 draw: 0.53 and 0.62. The app against the later 3.8 draw: 0.60 and 0.66.
• By language. English (12 pairs): self 0.83 and 0.77; simultaneous 0.58 and 0.67; 3.8 later 0.92 and 0.68. Hinglish (9): self 0.78 and 0.77; simultaneous 0.44 and 0.66; 3.8 later 0.44 and 0.69. Hindi (9): self 0.44 and 0.70; simultaneous 0.56 and 0.53; 3.8 later 0.33 and 0.59.
• No sources on either Gemini surface. The 30 signed-in pages showed no sources and no cards (signed-in-runs/ parse-log.txt: 0 links and 0 cards on 30 of 30, the page check finding no sources list, control, external link or redirect), exactly as the 30 logged-out pages had (check-runs.txt). The cited-sites measure is therefore not computable on Gemini in either state, and the card measure is 0.00 of 30 on both.
• Searching. With the search tool offered on every draw, the 3.6 arm searched on 21 of 30 draws and the 3.8 arm on 17 of 30 (signed-in/compare.txt counts the draws that searched; the questions skipped were brand-comparison and trust questions).

What it leaves untested: one account, one sixteen-minute session, with personalisation switched off, so what a personalised signed-in Gemini answers is unknown; the simultaneous draw is paired with the capture, not with the moment the app composed its answer, which the capture does not record; the cited-sites measure says nothing on Gemini in either state; the 3.8 arm was drawn later, not at capture, so its gap is 12 to 47 minutes; the app's "3.8 Flash" is the picker label as the owner read it; ChatGPT has no signed-in test; and the raw pages were deleted at capture, so the no-sources reading rests on the parser's page check, with one page read by hand before deletion agreeing with it.

On 82 distinct questions from the calibration sets of three reports (boAt, Mivi and Freshdesk (US)) and a D2C set of neckbands, headphones and power banks (Mivi's report repeats 14 of boAt's word for word, each counted once; ChatGPT and Gemini, two runs each, captured 7 to 13 Oct 2026), each app run was set against the production API draw fired at its capture (API with app, 192 pairs on each app) and the app against itself, run 1 against run 2 (app with app, 96 pairs on each app). The measures: the first brand named, the brands named and, on ChatGPT, the cited sites. First brand leaves out a pair in which a side names no brand, and cited sites one in which a side cites no site, so each of those cells counts its own pairs; brands named scores every pair, as the brands both name over the brands either names (a side naming no brand against one naming some agrees on none).

A cell is the mean agreement over its pairs, then its 95% interval and, in parentheses, its pair count. The intervals come from 10,000 bootstrap resamples of whole questions, drawn from all four sets at once (a question in two reports is one unit); a question with no scored pair on a measure, on either side, is not drawn for that measure. The difference is API with app minus app with app, with its 90% interval; the two are within ±0.10 when two one-sided tests each give p below 0.05 (the larger p is shown).

Across 82 questions, the API named the same first brand as the app 65% (ChatGPT) and 70% (Gemini) of the time; each app agreed with itself 70% and 74%. The gap is 4 to 5 points; at 90% confidence it is no larger than 11 to 12 points, close to but not yet inside our 10-point bar.

Every app run and every API draw the tests above score is in one archive: atlas-app-vs-api-runs-2026-10-18.zip (2129819 bytes, sha256 695e51d95be48bff6169baf874cd5054922ed7dfab682f330bd687b4fe0fa0a2), with its README, which describes each file and field, the tests, the measures and the licence. Inside: 474 app runs (the ten drift questions, logged out and signed in; the four calibration sets, boAt, Mivi, Freshdesk from the US and the D2C set, 96 runs each) and 1374 API draws paired with them, each answer as the app showed it or the API returned it, with its links, cards, timestamps and settings. Nothing identifies the person who ran the apps: the build refuses to write if an answer, link, card or citation carries their name or email, a sign-in line, a chat or share link or an account link, and the same check was run again on the published file; cited URLs are as the models returned them, with their tracking and affiliate parameters removed, so the archive carries no one's click tracking. Licensed CC BY 4.0: share and adapt it for any purpose, crediting Supernamed and naming the archive and its build date.