LLM product people need to step up - because product is the hardest problem in the ecosystem today

LLM product people need to step up - because product is the hardest problem in the ecosystem today 🪜

Challenges include deciding how to integrate LLMs into your product, proving ROI, and placing bets when it isn’t guaranteed you can build great experiences 🎰

And they’re all secondary to the biggest question of all - what should you build that will add enduring value to users? 🏗️

The technical side of building a great product that uses LLMs is difficult. But there are many companies with products that function well who are miles away from Product Market Fit. It’s great product decisions that drive impact with real users

We don’t have all the answers to what makes a great LLM product, but we know building one is an iterative process - just like building non-LLM products. You build a prototype, test it with real users, and improve it over time

Key to this process is a deep understanding of your users and their goals 🔑

But it’s hard to know how and why people are using text-based products. Reading conversation transcripts and talking to people works when you have a handful of users, but you quickly scale into a black box

The number of conversational products is exploding, and they’re generating mountains of text. Yet large numbers of product people can’t answer basic questions like what are the main reasons people use my product? Or where are user needs being met, and where are they being disappointed?

You’d never build a non-LLM product with such limited understanding of user goals. So why is this black box so common in the industry today?

We help product people at Fortune 500’s, unicorns, and startups to gain user understanding and solve the black box problem with our LLM text analytics. How does this work?

Text analytics takes a large number of conversation transcripts between users and LLM products, and then labels these transcripts on two dimensions:

1️⃣ What is the meaning of the conversation? Using semantic matching, user intent matching, and keyword matching

2️⃣ How successful was this conversation? Using conversion rates, thumbs up/down ratings, sentiment, freetext feedback, and more

Combining these gives visibility into how and why people are using the product, how well user needs are being met, where you have opportunities to improve

If you’re building LLM products and you’re feeling the black box problem then get in touch to learn more!