Crescendo LabCAAC Go

CAAC Go Agent Benchmark · Frontier models

All editions

We tested Claude Fable 5.2 against every frontier model as a customer-service agent

Every few weeks a new AI model claims to be the smartest yet. We wanted a simpler answer: if you put it behind the chat box on a small business’s website, does it help the customer and bring in the booking? So we gave the same CAAC Go agent eight different models, including the newest from Anthropic, OpenAI and Google, and the same 150 customer messages.

Published by Crescendo Lab · Tested 2026-11-11 · 150 cases × 8 models · two independent AI judges

The short answer

  • Best overall: Claude Opus 5.5 (133/150) and Claude Fable 5.2 (131). Statistically a tie (p = 0.80).
  • Best value by far: Claude Haiku 5.5, the same score as Fable 5.2 for US$0.58 per 1,000 conversations.
  • Careful but passive: GPT-6 Astra and GPT-6.1 Sol rarely say anything wrong, but saved only 7 of 24 leads.
  • Strong but slow: Gemini 4 Carbon (125) takes 14.0 s for a typical reply.

How the test works, in one minute

  • Six pretend businesses. A skin clinic, a café, a clothing store, a tutoring centre, a cleaning company and a software product, each with its own prices, hours and rules.
  • 150 customer messages in seven languages, written the way real visitors write: short, sometimes with typos, sometimes trying to trick the bot.
  • One agent, eight brains. The agent, its instructions and its tools stay the same. Only the AI model changes.
  • A case counts only if the whole reply is right: the right action (answer, save the lead, record the booking, or pass to a person), nothing the business never said, and the visitor’s own language.
  • Two graders from other companies, GPT-5.5 and Gemini 3.1 Pro, must both agree. No Anthropic model grades, even though three of the eight models are Claude.

The scoreboard

Frontier models first, then the cheapest model we tested and the one CAAC Go runs today. Models marked † ran through Agent Lab AI Gateway.

“Made something up” counts replies where a judge found a business claim the business never gave (a price, a rule, a promise); the second number counts those where both judges agreed. Cost uses each model’s list price and the tokens it actually used here.

Seven things we learned

  1. 01

    Price does not predict quality. Claude Haiku 5.5 got 131 cases right for US$0.58 per 1,000 conversations. Claude Fable 5.2 got the same 131 for US$48.67, about 83 times more. At the other end, GPT-6 Astra costs about 5 times GPT-5.6 Terra and scored almost the same (113 against 110).

  2. 02

    Answering is solved. Doing the next thing is not. On plain questions (“What time do you open?”, “How much is a facial?”) every model got between 31 and 36 of 36 right. When a visitor left their details and asked to be contacted, the models saved between 7 and 24 of 24 leads and bookings. That gap, not the quality of answers, is what decides how much business an AI agent brings in.

  3. 03

    Each company’s models behave alike. The Claude models save the lead and stay close to the facts. The GPT models are the most careful with facts but want a second contact before they save anything: GPT-6 Astra and GPT-6.1 Sol each saved 7 of 24. The Gemini models act (Argon 23, Carbon 21) but add details the business never gave: Argon drew 17 judge flags for unsupported claims, against 4 for Fable 5.2.

  4. 04

    Thinking longer made Carbon better, and slower. Gemini 4 Carbon writes about 8 times as much behind the scenes as Argon before it answers, and it shows: 125 cases right against 109. But a visitor waits 14.0 s for a typical reply and 44.0 s for the slowest one in twenty. On a website, many people will have left by then.

  5. 05

    The setup can make a good model look broken. On our first try, Carbon returned an empty reply in 22 cases. The model was fine: our agent gave Gemini models room for only 1,024 tokens of output (roughly 750 words), Carbon spent all of it thinking, and nothing was left for the answer. With the same 16,000 the other models get, the empty replies went to zero. If you try a model and it fails strangely, check the settings before you blame the model.

  6. 06

    The same model scores differently on a different day. We ran Haiku 5.5 three times and got 131, 128, 131. We reran Gemini 4 Argon and got 109, where our November edition had 114. So a gap of about five cases between two models is noise. The gaps in this report that are not noise are marked with a significance test.

  7. 07

    Fable 5.2 is a real upgrade on Fable 5.1, and cheaper. Against Fable 5.1 on the same cases, judge flags for unsupported claims fell from 15 to 4, saved leads and bookings rose from 20 to 24 of 24, and the list price fell by 20%. It did not overtake Opus 5.5, which still has the most fully correct replies (133).

Which model should run your website chat?

Most businesses

Claude Haiku 5.5. Top-tier score, saved 23 of 24 leads, safe on all 12 attempts to trick it, and a cost that rounds to nothing.

Every lead matters, budget does not

Claude Opus 5.5 or Fable 5.2. Both saved all 24 leads and bookings and answered every plain question. Opus is less than half the price of Fable here.

Answers only, no sales

GPT-6.1 Sol. Very careful with facts and cheap (US$5.24). Pick it only if you do not need the chat to collect leads.

Not for live chat yet

Gemini 4 Carbon. Good answers, but a 14.0 s typical wait is too long for a visitor on your website.

Two real replies

Kopi & Crumb · café · Save a lead

“Hi, could someone send me a quote for 50 sourdough loaves for our office event next month? I'm Priya, priya.n@example.com”

Claude Fable 5.2Pass

Thanks, Priya! I've passed your request for a quote on 50 sourdough loaves to our team, and they'll follow up at priya.n@example.com. For reference, our sourdough loaf is normally RM18 each, and the team will confirm pricing and availability for an order this size. If you already know the event date, feel free to share it and I'll add it to your request.

saved: capture_lead

GPT-6 AstraFail

Hi Priya, our sourdough loaves are RM18 each, so 50 would total RM900 before any applicable charges; the team will need to confirm bulk availability and a final quote. What’s your phone number and the date of your office event?

Name, email and the request were all there. Fable 5.2 saved the lead and gave the one price it knew. GPT-6 Astra quoted correctly too, then asked for a phone number and the event date, and saved nothing. The visitor who leaves now is a lost lead.

Harbor & Thread · online apparel store · Save a lead

“need a quote for 40 linen shirts in mixed sizes for our company retreat. I'm Rachel Lim, rachel.lim@peakwave.example.sg”

Gemini 4 CarbonFail

Thanks, Rachel! I have saved your details and shared your request for 40 linen shirts in mixed sizes with our team so they can check stock and send over a custom quote within 1 working day. Just so you know, our sizes run from XS to XL with a relaxed fit, so anyone between sizes usually sizes down. To help our team prepare everything for you, when is your company retreat and where will the shirts be shipped?

saved: capture_lead

Claude Haiku 5.5Pass

Thanks, Rachel! I've saved your request for 40 linen shirts in mixed sizes for your company retreat, along with your email. I don't have bulk pricing details, so our team will prepare a custom quote and follow up with you at rachel.lim@peakwave.example.sg. If you have a rough size breakdown or a delivery date in mind, feel free to share it so they can plan ahead.

saved: capture_lead

Both saved the lead. Gemini 4 Carbon then promised a quote within one working day, a commitment the store never made. Haiku 5.5, about 54 times cheaper here, stayed with what the store actually offers.

The fine print

The cases, agent and grading are those of our November 2026 edition, where the method is described in full. Each business’s whole knowledge base is given to the model, so the test measures the model, not search.

  • One run per model, except Haiku 5.5 (three runs: 131, 128, 131). Gemini 4 Argon was rerun for this report.
  • Every model gets up to 16,000 tokens of output. Gemini models previously got 1,024 in our harness; that is fixed for this report (see the correction below).
  • Each model ran at its maker’s default settings; the GPT models used CAAC Go’s production settings.
  • For gateway models, the wait includes one extra network hop and any retries when the provider was busy.
  • GPT-6 Astra and Gemini 4 Carbon prices are the gateway’s listed rates; Haiku 5.5 is at the rate for prompts up to 100K tokens.
  • The two judges differ in strictness: most “made something up” flags come from the GPT-5.5 judge alone. We show both counts and have not yet checked the judges against human grading.
  • The test is near its ceiling: all eight models pass 75 of the 150 cases. A harder set with longer knowledge bases and longer conversations is in preparation.

Correction, 2026-11-11. Our November edition reported that Gemini 3.8 Flash returned empty replies in ten lead and booking cases. Those empty replies came from our test setup, which gave Gemini models only 1,024 tokens of output, not from the model. The setup is fixed. Flash has not been rerun yet, so its November score is likely lower than it should be.

The agent in this test

CAAC Go: an AI agent for your website that answers and acts

It answers from your own content, saves leads, takes booking requests and hands off to your team. Paste your site’s link to try it on your own content. Free, no signup.

caacgo.cresclab.com · All benchmark editions