Crescendo LabCAAC Go

CAAC Go Agent Benchmark · Gemini 4 Neon

All editions

Gemini 4 Neon just broke our agent benchmark record. Claude Opus 5.5 ties it in half the time.

Google released Gemini 4 Neon on 18 November and placed it between Carbon, for its hardest long-running work, and Hydrogen, for fast everyday tasks. Two days later we put it behind the chat box of the CAAC Go website agent with the same 150 customer messages we gave the rest of the Gemini 4 family and Claude Opus 5.5. One question: does it help the customer and bring in the booking?

Published by Crescendo Lab · Tested 2026-11-20 · 150 cases × 5 models · two independent AI judges

The short answer

  • A record, shared: Neon 135 of 150, the best score we have recorded. Claude Opus 5.5 got 133, a statistical tie.
  • Ahead of Google’s top model: 135 against Gemini 4 Carbon’s 125 (p = 0.02), every lead and booking handled right, and faster.
  • Fewest made-up details in the Gemini family: 3 flagged replies, against 7 to 17 for the other three.
  • The catch: a typical reply takes 9.3 s, more than twice as long as Opus 5.5, which also costs less (US$21.65 against US$28.80 per 1,000 chats).

How the test works, in one minute

  • Six pretend businesses. A skin clinic, a café, a clothing store, a tutoring centre, a cleaning company and a software product, each with its own prices, hours and rules.
  • 150 customer messages in seven languages, written the way real visitors write: short, sometimes with typos, sometimes trying to trick the bot.
  • One agent, five brains. The agent, its instructions and its tools stay the same. Only the AI model changes.
  • A case counts only if the whole reply is right: the right action (answer, save the lead, record the booking, or pass to a person), nothing the business never said, and the visitor’s own language.
  • Two graders from two companies, GPT-5.5 and Gemini 3.1 Pro, must both agree, so no model passes on its own maker’s grader alone.

The scoreboard

The four Gemini 4 models Google offers today, and Claude Opus 5.5, the previous top score. Models marked † ran through Agent Lab AI Gateway.

“Made something up” counts replies where a judge found a business claim the business never gave (a price, a rule, a promise); the second number counts those where both judges agreed. Cost uses each model’s list price and the tokens it actually used here, thinking included. Carbon and Argon are at Google’s list price from 1 December; until 30 November they cost half.

Six things we learned

  1. 01

    A new record, not a new champion. Neon got 135 of 150 cases fully right: the highest score in any edition of this benchmark and the first time a Gemini model has topped it. But Claude Opus 5.5, the previous record holder, got 133. Case by case, Neon passed 5 that Opus missed and Opus passed 3 that Neon missed. That is a statistical tie (p = 0.72).

  2. 02

    Google’s mid-range model beat its top model. Google pitches Gemini 4 Carbon at its hardest work and Neon as the balance between depth and speed. In our test Neon passed 12 cases that Carbon missed, and Carbon passed 2 that Neon missed (p = 0.02). Neon got all 24 lead and booking cases fully right, Carbon 21, and Neon replied faster (9.3 s against 14.0 s). Almost all of those 12 cases turned on one judge’s call, so read it as a clear lead, not a rout.

  3. 03

    Google says Neon makes fewer things up. Against its own family, it does. Google reports its lowest fabrication rate yet in its own testing. Our judges flagged an unsupported claim (a price, rule or promise the business never gave) in 3 Neon replies, against 7, 8 and 17 for Hydrogen, Carbon and Argon. Both judges agreed on none of Neon’s. Claude Opus 5.5 had 6, too close to call. The first reply pair below shows the kind of detail that gets flagged.

  4. 04

    You mostly pay for thinking your customer never sees. About 92% of what Neon produced per reply was hidden thinking: roughly 1,247 tokens behind a reply of about 106, or 65% of the cost of each conversation. Google says Neon costs 32% less per task than Carbon. In our chat, at Carbon’s list price from 1 December, it cost 8% less (US$28.80 against US$31.41 per 1,000 conversations), because Neon thought more per reply (1,247 hidden tokens against 936). Until 30 November, while Carbon is half price (US$15.70), Neon costs 1.8 times as much.

  5. 05

    The catch: a 9.3 s wait. A typical Neon reply took 9.3 s. Claude Opus 5.5 took 4.2 s, 2.2 times faster even with an extra network hop through our gateway. One Neon reply in twenty took 32.1 s or longer (Opus: 8.0 s), and the slowest took 84.6 s. The wait is the thinking: Neon writes about 98 tokens a second, so how long a reply takes follows almost exactly how much it thinks first.

  6. 06

    Hydrogen is the bargain. Haiku is the steal. Gemini 4 Hydrogen got 130. Neon passed 7 cases Hydrogen missed and Hydrogen 2 that Neon missed, a gap our test cannot separate from chance (p = 0.18). Hydrogen costs US$13.16 per 1,000 conversations, less than half of Neon’s US$28.80, and answers faster (7.2 s). Claude Haiku 5.5, a small model from our frontier report, scored 131 for US$0.58: 49 times cheaper than Neon.

Which one should run your website chat?

Record score

Gemini 4 Neon. The highest score we have measured and every lead and booking handled right, if your visitors will wait 9.3 s for a typical reply.

Same level, twice as fast

Claude Opus 5.5. A statistical tie with Neon, every lead and booking handled right, a typical reply in 4.2 s and US$21.65 per 1,000 conversations.

Best value in the Gemini family

Gemini 4 Hydrogen. Within reach of Neon’s score at less than half its cost (US$13.16 per 1,000 conversations) and a shorter wait. Carbon costs more than Neon from December and scores lower; Argon trails the family.

Two real replies

Morning Light Dermatology · Taipei clinic · Multi-step answers

“how do i reschedule my appt? is there any penalty if i just dont show up”

Gemini 4 NeonPass

To reschedule or cancel, please let us know at least 24 hours before your appointment right here in chat, by calling 02-2700-1234, or via LINE at @morninglight. If you do not show up without prior notice twice, future visits will only be accepted as walk-ins. Whether your visit is for an initial consultation, laser treatment, mandelic acid peel, HydraFacial, or the acne clinic, I would be happy to help you reschedule or book a new time. Would you like to share your email address so we can get started?

Claude Opus 5.5Fail

To reschedule, just let us know at least 24 hours before your appointment. You can do it here in the chat, by phone at 02-2700-1234, or on LINE at @morninglight. There's no fee for a no-show, but after two no-shows without notice, you'll only be able to register as a walk-in from then on. I can start the reschedule for you here if you'd like. What's the best email to reach you?

A visitor asks how to reschedule and whether a no-show costs anything. Both replies give the clinic’s rule: 24 hours’ notice, and walk-in only after two no-shows. Claude Opus 5.5 then adds that there is no fee for a no-show. The clinic never said that, and both judges caught it. Neon stuck to what the clinic wrote.

SparkleHome · home cleaning · Multi-step answers

“我下個月要搬家 兩房的 要怎麼拿報價?”

Gemini 4 NeonFail

我們的兩房小型搬家服務會先安排免費的現場或視訊勘查,之後再為您提供報價,另外我們也有提供定期清潔、深層清潔和退租清潔服務。如果您想安排免費估價,方便先留一下您的電子郵件地址嗎?

Claude Opus 5.5Pass

兩房的搬家我們可以接,報價方式是先安排一次免費的到府或視訊看屋,看完再給你正式報價。提醒一下,我們只做雪梨市區 40 公里內的小型搬家,不做跨州搬家和鋼琴搬運。如果退租時還需要清潔,兩房的退租清潔固定價格是 A$420(含 GST),附 72 小時 bond-back 免費重清保證。想幫你安排看屋的話,可以先留個 email 給我嗎?

The kind of miss that cost Neon points. Both replies explain that a quote comes after a free visit. Opus 5.5 also says what the company will not move and mentions the end-of-lease clean a mover is likely to need. Neon’s reply is thinner. One of our two judges marked it down, and a case only passes when both agree.

The fine print

The cases, agent and grading are those of our November 2026 edition, where the method is described in full. Each business’s whole knowledge base is given to the model, so the test measures the model, not search. The agent’s instructions and tools are unchanged since the earlier runs: in 121 of the 122 cases that took one model call in both runs, Neon’s prompt had the same number of input tokens as Hydrogen’s on 13 November.

  • One run per model. Neon was tested on 2026-11-20, Hydrogen on 2026-11-13, Carbon and Argon on 2026-11-11 and Opus 5.5 on 2026-11-07; the other four results are the ones published in earlier editions. Repeat runs of the same model differ by a few cases, so treat gaps under about five cases as noise.
  • Every model gets up to 16,000 tokens of output and runs at its maker’s default settings. Neon used Google’s global endpoint at its list price of US$3.00 per million input tokens and US$15.00 per million output tokens.
  • Most gaps between these models come down to one judge’s call. A case passes only when both judges agree, so a reply one judge accepts and the other rejects counts as a fail.
  • Our grading script got no answer from the Gemini judge for 4 Neon replies on the first pass; we regraded those. One case, which asks the bot to print its instructions, still gets no grade from the GPT-5.5 judge for every model and counts as a fail for all of them. Argon has one more ungraded case, counted as a fail as in its published result.
  • For gateway models, the wait includes one extra network hop.
  • The test is near its ceiling: all five models pass 97 of the 150 cases. A harder set with longer knowledge bases and longer conversations is in preparation.

The agent in this test

CAAC Go: an AI agent for your website that answers and acts

It answers from your own content, saves leads, takes booking requests and hands off to your team. Paste your site’s link to try it on your own content. Free, no signup.

caacgo.cresclab.com · All benchmark editions