A/B Testing Chatbot Messages: What to Test and How

The assistant's opening line, its button labels, how long its answer to a pricing question runs - all assumptions. Most teams change them by debate: someone says it should sound friendlier, the copy changes, nothing is measured. A/B testing turns that debate into data, but set up badly on low traffic it turns noise into false confidence.

What Is Worth Testing

Do not start with low-impact variables. Whether the greeting keeps people in the conversation matters far more than an emoji debate.

Setting the Test Up

• Change one variable at a time.
• Define success in advance as a single metric, not as better.
• Randomise assignment - splitting by morning and afternoon measures the hour, not the message.
• Round the duration to whole weeks; weekday and weekend behaviour differ.
• Do not peek before it ends, or you will close it on day-one noise.

The Small Traffic Problem

In a business handling thirty conversations a day, telling apart a small difference between two variants is statistically hard. That does not mean testing is pointless, it means the test has to be designed differently:

• Test noticeable differences, not two-word tweaks: an opening that asks a question versus one that gives information.
• Run longer - four weeks beats two days.
• Measure frequent events rather than rare ones: replying a second time instead of purchasing.
• When numbers are thin, read the conversations. Fifty read conversations beat a meaningless percentage over a hundred.

Four Common Mistakes

• Choosing the metric after seeing the results.
• Testing during a campaign week, when traffic composition is not representative.
• Showing both variants to the same person because assignment is not pinned per customer.
• Never implementing the winner - the most common way tests waste effort.

Making the Winner Stick

Keep watching the same metric for two weeks after rollout; live differences are often smaller than test-phase differences. Then propagate the winning copy into the knowledge base and template set, so other channels do not keep the old wording.

Testing with Templates

If your test involves approved templates, each variant is a separate template with its own review. Add approval time to the plan and keep variants in the same category, otherwise delivery and billing behaviour differ and the comparison breaks.

With BozLat AI

Greeting, routing and reminder messages are configurable; which variant handled which conversation is recorded, and conversion metrics are reported per variant. See pricing.