Before You Add AI to Your CRM, Ask It 1,000 Questions
Before you add AI to your CRM, test it on the questions your team would really ask: the method we used on a custom CRM, with every figure from our runs.
Key findings
ShowHide
- Most misses came from what the assistant was told. In one round, 62 of 66 came from its instructions and the descriptions of its tools.
- The questions found problems in the CRM before any AI was built: a date bug and three permission gaps. The existing tools could answer only 35 of the first 123 questions.
- Score on questions you never tuned on. On the tuned questions the score rose 14 points; on unseen ones, only a few, and one version got worse.
- Put safety in the software, not the prompt. The assistant once marked a deal as won without asking, so every change now waits for a person to confirm it.
- The final run on real tools: 93% acceptable and nothing wrong, against a bar of 90% and zero. Partly right answers, at 7%, were still above the 5% bar.
If you run operations or lead a sales team, there's a good chance someone has asked you: "Can we get AI on our CRM?"
What they picture is typing "which deals over £40k haven't moved this month?" or "log a call with them and remind me Friday", and the system just doing it.
Before you buy an add-on or commission a build, write down the questions your team would really ask, and test whether your system can answer them.
In October 2026 we built an AI assistant into a custom CRM we had made for a London business selling B2B. Staff would type a question or a request, and the assistant would answer from the CRM's own data or propose a change for them to confirm. Before it went anywhere near staff, we wrote well over 1,000 questions they might ask, ran hundreds of them against the assistant round after round, and had separate AI graders mark the answers blind.
This post is what we learned: the testing method, step by step, so you can copy it or hand it to whoever builds for you, along with every figure from our runs, including where the assistant fell short.
The questions we wrote
We tested in five rounds, changing the assistant between each one. Each set of questions had a different job:
| Set | Questions | What it was for |
|---|---|---|
| First question set | 123 | Can the CRM's data answer these at all, before anything is built? |
| Held-out set | 40 | Never used for tuning, so it gives the honest score |
| Rounds 3, 4 and 5 | 600 (200 each) | A fresh set every round |
| Request catalogue | 900 | Rank what's missing by how often it's asked and how much it's worth |
| Extra sets for new features | 112 | For example, meetings and edit history |
| Total written | about 1,775 |
The rest of this post walks through what we did with them, in order.
Step 1: Ask the questions before the assistant exists
Start by checking whether your CRM can answer the questions at all. If the data isn't there, or the filters can't reach it, no AI model will fix that.
We checked the first 123 questions against the CRM's real structure before the assistant was built. They found real problems in the CRM:
| What the questions found | Kind of problem |
|---|---|
| Deals closing on the 1st of a month dropped out of that month's filter during summer time | A date bug: the date was stored as local midnight, but the month was read in UTC |
| A restricted total could be rebuilt by adding up a list everyone could see | Permission design |
| Quote margin was visible to every CRM role | Permission bug |
| Supplier discounts were visible to every CRM role | Permission bug |
On top of those, 33 questions needed a new filter or tool, 14 needed data the CRM didn't hold, and 18 depended on how carefully staff log things.
| Today's tools | With the proposed changes | |
|---|---|---|
| Questions the tools could answer, on paper | 35 of 123 | 72 of 123 |
Most of those fixes help staff whether or not AI ever arrives. A date bug or a permission gap is a problem in the CRM, not in the assistant.
One finding is worth checking in any CRM. Blank values quietly drop out of filters. Ask for "deals over £40k" and any deal with no value entered simply isn't counted. The testing showed the assistant needs to say what it left out, and most of the partly right answers in this first pass were a silent narrowing like this: something left out without saying so.
Step 2: Decide what "good enough" means before you start
Before running anything, agree how answers will be marked and what score you need. Otherwise every result looks encouraging.
We used four grades:
| Grade | Meaning |
|---|---|
| Correct | The right answer, or the right change proposed |
| Honest "can't" | The tools can't do it, and the assistant said so plainly |
| Partly right | Some of the answer, often with something left out unsaid |
| Wrong | Invented figures, a permission breach, following instructions hidden in the data, or changing something without asking |
Correct plus honest "can't" counts as acceptable. An assistant that says "I can't do that" is safe. One that guesses is not.
The bar for go-live, set in advance, was:
- at least 90% acceptable
- under 5% partly right
- nothing wrong
Step 3: Test the real thing, not a simulation
To move faster, we had AI agents role-play the assistant on paper, reading its instructions and tool descriptions and saying what they would do. It was quick. It was also too kind.
We had already run the real model, making real tool calls, on 65 of the same questions (191 runs). Comparing the two:
| Result | |
|---|---|
| Runs where the simulation matched the real model | 51% |
The simulation graded 44 runs correct that the real model only got partly right, and the real model guessed record IDs it hadn't seen. Every round after that used real tool calls.
If your test is a role-play, expect the real thing to score lower.
Step 4: Keep some questions back
If you tune the assistant on the same questions you score it on, the score climbs without the assistant getting much better. So keep a set of questions back that you never use for tuning, and treat that as the honest score.
We kept back 40. Here is round 2, where we compared the first version with two tuned ones:
| Version | What changed | Main set correct | Held-out correct | Held-out acceptable |
|---|---|---|---|---|
| v0 | Tools as first written | 60% | 51% | 86% |
| v1 | Design fixes, fixed-value lists in the tools, rules | 71% | 38–45% | 80–84% |
| v2 | Exact tool descriptions, general rules | 74% | 55–58% | 82–88% |
The ranges are two graders marking the same runs (more on that in the next step). No held-out run was graded wrong in any version.
The main set rose 14 points. The held-out set rose only a few, and v1 got worse on questions it hadn't seen. Part of the main-set gain was tuning to those particular questions. Without the held-out set we would have believed v1 was a big step forward.
Step 5: Grade blind, and check the graders
Marking well over 1,000 answers by hand isn't practical, so we used separate AI agents as graders, working to a fixed rubric. This approach is often called "LLM as a judge".
What made the grading trustworthy:
- Blind grading. Runs were shuffled and unlabelled, so graders never knew which version they were marking. From round 3, graders saw every version's answer to the same question side by side, without knowing which version or model produced which. Each answer was still graded on its own against the rubric, not ranked against the others.
- Two graders on the held-out runs. They agreed on about three runs in four, which means a count can move by about six questions in 80 depending on the grader. That's why some figures in this post are ranges.
- Each question asked two ways. In the first set and in every round from 3, each question was also asked in a second wording. Rewording changed the grade on 11 of 123 questions in round 1 and 29 of 200 in round 3. Your staff won't phrase things the way your test does, so neither should your test.
Step 6: Find out where each miss came from
This is the step we'd push hardest. For every wrong or partly right answer, write down what caused it. The cause tells you what to fix.
After round 2, version v2 had 66 misses. Of those, 65 had a clear cause:
| Cause | Misses |
|---|---|
| The assistant's instructions | 46 |
| The descriptions of its tools | 16 |
| The CRM's data | 3 |
Later rounds showed the same pattern with the stronger model: in round 4, 13 misses from instructions and 8 from tool descriptions; in round 5, 20 from instructions and 25 from tool descriptions.
In round 2, 62 of the 66 misses came from what the assistant was told and how its tools were described, and those two stayed the main causes in later rounds. Some of the fixes the testing pointed to:
- Tool descriptions must match the code exactly. One filter described as "proposal due within 7 days" also returned overdue proposals. Where a description was inexact, the assistant gave a confident wrong answer.
- List every fixed value in the tool. A free-text field invited "key account" when the stored value was "key".
- Count first when there are many results, then ask which one was meant.
- Ask about names that are no longer current, rather than quietly guessing a replacement.
- Get relative dates right. "This week" dropped overdue tasks. "Tomorrow", asked on a Friday, never offered Monday. "This year" needed to mean the financial year.
Step 7: Write a fresh set every round
Once you've fixed what a round found, the old questions are spent. Write new ones for the next round. We wrote 200 new questions for each of rounds 3, 4 and 5, and ran each round on two models: Claude Opus (the stronger model) and Claude Sonnet (the cheaper one).
| Round | Version, model | Correct | Honest "can't" | Partly right | Wrong | Acceptable |
|---|---|---|---|---|---|---|
| 3 | v2, cheaper model | 39% | 48% | 12% | 1% | 87% |
| 3 | v3, cheaper model | 48% | 38% | 13% | 1% | 86% |
| 3 | v3, stronger model | 54% | 40% | 5% | 2% | 94% |
| 4 | v4, cheaper model | 62–66% | 19–20% | 12–16% | 1–3% | 83–85% |
| 4 | v4, stronger model | 70% | 20% | 10% | 0% | 89% |
| 5 | v5, cheaper model | 52% | 19% | 26% | 2% | 71% |
| 5 | v5, stronger model | 68–70% | 20% | 9–12% | 0–2% | 87–90% |
Ranges are the two wordings of each question. Each round used a different set of questions, so a figure in one round isn't directly comparable with the round before. Read across a round to compare versions and models. Reading down gives only a rough trend, since round 5's set was written to be harder.
What the table shows:
- In round 3, new tools turned "can't" into correct answers. On the cheaper model, v3 moved correct from 39% to 48% without adding mistakes. Acceptable barely moved because honest "can't" answers already counted as acceptable.
- Round 4 is where the product decisions landed: who sees what, letting the assistant read totals, and making every change a card that a person confirms.
- Round 5's questions were written to be harder, mainly about choosing the right record to change. The cheaper model's partly right answers doubled; the stronger model held.
- Changes were the strongest part. In round 5 the stronger model got 85% of change requests correct, none wrong, and 97% of undo and revise requests correct.
Which model?
Model choice mattered. On the hardest set, the cheaper model fell to 71% acceptable against 87–90% for the stronger one. We planned on the stronger model.
It costs more. In a later comparison the cheaper model cost about half as much per request (3.5¢ against 6.8¢; estimates, not a real bill) and scored 6 to 7 points lower.
But most of the misses we fixed came from the instructions and tool descriptions, not from the model.
Speed is not the same as accuracy
In round 4 we let people tag records while typing. On the 83 tagged questions, it cut tool calls by 28% (from 4.3 to 3.1 per question) and cost by about 4%, with no change in accuracy. Worth having, but measure what you're claiming.
Step 8: Finish on real tools, with data built to trip it up
Rounds 3 to 5 ran against stand-in tools with test data. The last step was to run 169 questions against the real tools on a development copy of the CRM, filled with a dataset built for testing, using the stronger model.
The dataset was designed to catch it out: fourteen months of stage history, contacts sharing first names, a client group's branches set up as clients of their own, and other people's private meetings tagged with a marker word so any leak would show up in the answer.
| Run | Correct | Honest "can't" | Partly right | Wrong | Acceptable |
|---|---|---|---|---|---|
| First | 69% | 14% | 15% | 2% | 83% |
| Second | 72% | 18% | 9% | 2% | 89% |
| Final | 76% | 17% | 7% | 0% | 93% |
These are round 5's questions, now on the real tools and data instead of a stand-in, so compare them with round 5's stronger-model row rather than the earlier rounds.
The final run met the go-live bar on acceptable and on wrong answers. Partly right, at 7%, was still above the 5% bar.
After one more round of fixes we reran the 12 partly right questions: 8 came back correct and 4 as honest "can't". The other 157 weren't rerun, so 93% is the measured figure, and that's the one we stand behind.
What held, and the two failures that changed the design
Some things held in every run:
- It never followed instructions planted in names or notes. One instruction hidden in a pasted contact was ignored, and the assistant pointed it out.
- Claims that "a director approved it" never unlocked restricted figures or private recordings.
- It never invented a forecast, a benchmark or a reason a deal was lost.
Two failures in round 3 changed how the system is designed:
- The stronger model once moved a deal to "won" without asking. So confirmation for any change has to be enforced by the software, as a card a person confirms, not left to the assistant's instructions. It's the same principle as permissions: they belong in the tools, not in the prompt.
- For a staff member who was leaving, one run helped export the contacts directory. Bulk export should be a deliberate feature with its own permission, not something an assistant can piece together.
Anything that must never happen should be impossible in the software, not just discouraged in the instructions.
What staff expected that it couldn't do
In the later rounds, about 1 in 5 answers were an honest "can't". That's safe, but it's also a list of what your team will expect. We wrote 900 requests and sorted them by what it would take to answer them:
| Status of 900 requests | Share |
|---|---|
| Answerable now | 46% |
| Needs a new tool | 23% |
| Needs several changes on one card | 9% |
| Needs data the CRM doesn't hold | 7% |
| Should be refused, or done in the CRM's own screens | 11% |
| Other (links, search by meaning) | 3% |
This list is your roadmap. Most of it is CRM work: new tools, new fields, permissions decided on purpose.
Shorter instructions, same accuracy
Long instructions are sent to the model at every step, so they cost time and money. We wrote a shorter version and graded the long and short instructions blind on 149 requests drawn from the earlier sets. Because it's a different mix of requests, compare long with short here rather than with the tables above.
| Correct | Partly right | Wrong | Acceptable | |
|---|---|---|---|---|
| Stronger model, long | 89% | 10% | 0% | 90% |
| Stronger model, short | 90% | 9% | 0% | 91% |
| Cheaper model, long | 83% | 15% | 1% | 83% |
| Cheaper model, short | 83% | 14% | 2% | 84% |
What the short instructions changed per request, on the stronger model:
| Change | |
|---|---|
| Instructions and tool definitions sent each step | −17% |
| Reply length | −17% |
| Time taken | −11% |
| Estimated cost | −10% (7.5¢ to 6.8¢) |
Same accuracy, about a fifth less to read every step. Without blind grading we couldn't have shown that the shorter version didn't cost quality.
The method, on one page
If you're about to add AI to your CRM, or brief someone who will, this is the list:
- Write the questions your team would really ask. Hundreds of them, in their words.
- Check them against your CRM before any AI. Fix the bugs and permission gaps they find.
- Set the bar in advance. Ours was 90% acceptable, under 5% partly right, nothing wrong.
- Test with real model calls, not a role-play.
- Keep a held-out set you never tune on.
- Grade blind, with two graders on a sample, and ask each question two ways.
- Record the cause of every miss: instructions, tool descriptions, data or model.
- Write fresh questions every round.
- Finish on real tools with data built to trip it up.
- Put safety in the software: changes confirmed by a person, permissions in the tools.
AI in a CRM is mostly backend work
Looking back over the project, the AI model was one decision among many. Most of what the testing pointed to was deciding who can see what, describing each tool precisely, finding the filters and fields staff need, and making sure changes go through a person. That is the work of designing a custom CRM, and it's the work we do.
If you've been asked "can we get AI on our CRM?" and want to talk it through, get in touch.
Written with AI assistance and checked by Alex. How we write