How Mario Ciancio Tests Whether AI Support Actually Helps
A customer asks why an integration failed. The AI assistant sends them to integration settings. They get there and still can’t see which sync failed or where to find the error.
Mario Ciancio, who previously led product support and development at nVoq, built a synthetic evaluation around that problem. The fictional assistant’s first answer pointed to general settings. The revised answer directed the customer to the integration’s run history, where they could investigate the failure.
Three failures, three fixes
- Missing documentation. Add the exact API permission error, explain what to check, and tell customers what to provide if they escalate.
- Wrong source retrieved. Return the run-history guide for a failed-sync question. Keep the general overview available for setup questions.
- Incomplete handoff. Include the integration name and the request ID so the receiving support team can investigate.
Seven of ten cases passed at baseline. After the changes, ten of ten passed, with no regressions in the suite. Those figures come from a controlled, fictional demo.
What a client pilot would measure
Ciancio would start with the client’s current results and track how often AI conversations turn into tickets, how often customers ask for help again, and whether human agents get enough detail when they take over. For a team considering a rollout, those measurements connect the test results to customer effort and support capacity.
The first question is straightforward. After reading the answer, can the customer take the next step? If a reply falls short, the test should show where to investigate and how to verify the fix.
Request a free check