Challenges Building Production-Ready AI Chat Experiences

Chat interfaces powered by AI can deliver amazing user experiences. Companies from law to medicine and e-commerce to entertainment are building these products at breakneck pace. But while building an impressive demo has never been easier, developing a production-ready product is tough.

I’ve had conversations with approximately 100 developers of chat interfaces, and I’ve identified a number of challenges that must be addressed to develop a successful product:

1. Understanding user behaviour. You can’t build great products without a deep understanding of users and their goals, and this is often missing in products today. This visibility allows you to develop a great product that satisfies real user needs.
2. Understanding product performance. Developers will launch and iterate their products towards product market fit, and it’s crucial to bring real user behaviour into this evaluation process. This user feedback should include many channels, including in-product user feedback ratings, response regeneration rates, and user input sentiment analysis.
3. Managing risky behavior. LLMs can wander off-topic, discuss your competitors, engage in political or medical conversations, and contain bias and hallucinations. Scrutiny of these experiences is rapidly increasing. Managing these risks is challenging, and outside of universal harms like bias they are not a priority for foundational model providers. Prompt engineering in isolation is not enough to guarantee safety. 

Do any of these resonate? What are we missing? Alex and I have built a product that helps manage these challenges by providing product analytics specifically for chat interfaces. I’d love to speak with developers of chat interfaces to share the lessons I’ve learned, and to help anyone develop better chat experiences for end users. Please get in contact at henry@woolly.ai.