How do the best tech companies find product market fit for their LLM products? 🔎 🎯
How do the best tech companies find product market fit for their LLM products? 🔎 🎯
It starts with deeply understanding their users 💬🔬
Yet many people building in the LLM ecosystem still don’t know how or why people use their products. The phrase I always hear is that LLM products feel like a black box 📦
People try to solve this by reading transcripts - but it doesn’t give enough depth. Sample sizes are tiny and unrepresentative, there's a mountain of text to comb through, and this doesn’t scale beyond a few dozen users
What’s a better solution? Using conversation analytics from @Context - we’re helping companies from Fortune 500 enterprises to tech unicorns and startups find product market fit for their LLM products. How do we do this?
1️⃣ Take the transcripts of user:assistant interactions and label them using a taxonomy of categories. Define this taxonomy using semantic, keyword, and user user intent matching
2️⃣ Cluster all the transcripts to identify additional categories of user behavior in the long tail that you didn’t anticipate
3️⃣ Combine these categories with per-conversation success reporting that uses signals including user sentiment, thumbs up/down, freetext feedback, conversion events, LLM evaluators, engagement, and retention metrics
You can understand key drivers of engagement from new and veteran users, product performance issues to improve with model/prompt/document database changes, and most importantly: the pockets of users or categories that show signs of stronger product market fit
This is how you understand how and why people are using your application and iterate towards product market fit for your LLM product
