"Conversational AI" is the industry term for software you talk to in natural language. It covers everything from a phone system that understands "I want to cancel" to an assistant that drafts your emails — which is why the term is simultaneously everywhere and nearly meaningless without context.
Here is what it actually refers to, and how the pieces fit.
The three generations
Rule-based (the old kind). Decision trees. "Press 1 for billing." If your input does not match a rule, it fails. Cheap, predictable, and universally disliked.
Intent-based (the 2015–2022 kind). Trained to classify what you want into one of a fixed set of intents, then respond from a script. Better than rules, but someone has to define every intent in advance, and anything outside the list falls through.
LLM-based (now). A large language model generates responses rather than selecting them. No predefined intents, handles phrasing nobody anticipated, and can hold context across a conversation.
Most "conversational AI" products sold today are the third kind wrapped around one of the first two for safety.
The parts of a real system
An LLM alone is not a conversational AI product. A working one has four pieces:
- Understanding — working out what the person wants
- Knowledge — the information needed to answer, which is usually your documents rather than anything the model learned in training
- Action — doing something: looking up an order, booking a slot, creating a ticket
- Escalation — knowing when to stop and hand to a human
Vendors compete mostly on 2, 3 and 4. The model itself is increasingly a commodity — several providers offer comparable quality, and swapping between them is a configuration change.
Grounding: the part that decides whether it works
A general model knows nothing about your company. Ask it your refund window and it will invent a plausible one, because inventing something plausible is exactly what it does when it lacks information.
Grounding means giving the system your actual documents and instructing it to answer only from them, saying "I don't know" otherwise. It is the single biggest determinant of whether a deployment succeeds.
Any vendor who cannot explain how their system is grounded is selling you a confident-sounding guess generator pointed at your customers.
Where it works, and where it doesn't
Works: answering documented questions, triage and routing, gathering information before a human takes over, out-of-hours acknowledgement, internal assistants for staff.
Does not work: anything about billing, refunds, account status or legal terms without supervision. Those are exactly the conversations where a confident wrong answer costs real money, and the system has no way to know it is wrong.
The pattern: it is good at retrieving and phrasing, and unreliable at deciding. Design around that and it works; ignore it and you get the failure stories.
Voice vs text
Voice adds two hard problems on top of everything above: speech recognition that survives accents, background noise and interruption, and latency low enough that pauses do not feel broken.
Text is dramatically easier and still covers most business cases. Start there unless the phone is genuinely where your customers are.
What it costs
- Internal assistants: a per-seat subscription. Cheapest and fastest payback.
- Customer-facing platforms: usually priced per conversation or per resolution. This is where costs surprise people — model your volume before signing anything.
- Custom builds: engineering plus ongoing maintenance as models change. The maintenance is what gets underestimated.
Common questions
What is conversational AI? Software you interact with in natural language. Modern systems use large language models to generate responses rather than picking from scripts.
What is the difference between conversational AI and a chatbot? "Chatbot" usually means the older rule- or intent-based kind. Conversational AI is the broader term and now generally implies an LLM underneath.
Is conversational AI the same as a large language model? No. The model is one component. A working system also needs your knowledge, the ability to take actions, and a path to a human.
How do I stop it giving wrong answers? Ground it in your own documents, restrict it to answering from them, and make it escalate rather than guess.
Does it replace support staff? No. It removes repetitive triage so people spend time on cases needing judgement. Treating it as a headcount replacement is the most common way these projects fail.
Voice or text first? Text, unless the phone is genuinely where your customers are. Voice adds recognition and latency problems on top of every text problem.