AI translation for customer service: tests to production
An AI interpreter for customer-service calls: customers call in their own language, and the agent switches on real-time translation for both sides. The hard part was quality on live calls: translation depended on the transcription, and only the third solution we tested was good enough.
- Result
- The conversation flows in two languages, in real time. Released to production, stable at release, and handed over to the client.
- How quality was checked
- Test scenarios and evals on live-call quality, used to choose the tools and calibrate the translation settings. A better transcription made whole sentences more faithful; proper names and numbers still needed a glossary and rules. One scenario is shown below.
- Scope and limits
- My involvement ended at the production handover. I don’t have performance data from after the release.
One test scenario
- The caller wants to
- Place a correct order: a specific model, by name, and its catalogue number, for example Nova 90, NV-90-4120-B.
- What went wrong
- Model names and catalogue numbers did not come through reliably: a proper name can be translated like an ordinary word, and a string of digits and letters can be read out wrong.
- Why it matters
- One wrong word or digit, and the order is wrong.
- To pass
- The model name and the catalogue number reach the other side exactly as they were said.
- What I changed
- The translation configuration: a glossary of proper names, and rules for translating strings of digits, and of digits and letters.
- Retest
- After the change, the order scenario passed.
- What stayed hard
- Very long numbers, such as bank account numbers: in about 10 test cases, 2 came through with an error.
- What we decided
- It is not a common case. We advised against giving such numbers over the phone, and I passed the limitation to the client as a known risk; the decision was theirs.
What the evals checked
Accuracy · response time · numbers and symbols · domain terms and proper names · background noise · hallucinations
Two decisions
Decision 1Which transcription can the translation rely on?
Not faithful or accurate enough for the translation to rely on.
A very faithful and accurate transcription.
The tests showed that translation quality depended above all on the transcription, and that good transcription was harder than it looked. DeepL handled the translation from the start.
We kept testing until the transcription met our bar, and calibrated the translation settings on the same scenarios.
Decision 2How many tools do we run?
More vendors to connect and maintain.
Speech recognition and speech synthesis with one vendor. DeepL kept the translation.
Quality decided it, and fewer tools made the whole solution much simpler.
The cost: we depended on one vendor for speech recognition and speech synthesis. The pipeline was built to be flexible, so that vendor could be changed.
The whole call
I gathered what the client and the customer-service staff needed, and proposed the workflows around the call: a welcome message for the customer, messages for employees, internal redirects to people and departments, handling of missed calls, and the context and history of each call passed on and saved. They made handling calls between departments much better. After the release, the client wrote that the project had been run very professionally.
- Before the callA welcome message for the customer, and the right messages for employees.
- RoutingInternal redirects to specific people and departments.
- Live translationThe AI pipeline, calibrated with the test scenarios.
- HandoverThe context of the conversation is passed on and saved.
- After the callCall history and handling of missed calls.
Evaluating an AI product for production?
Tell me what you are building and where the uncertainty is.