How to ship amazing LLM products (without getting fired when they go wrong):

Building LLM products requires threading a needle. Product people trade off iteration speed, product quality, and safety risks - and the best PMs know how to do it 🧵🪡

➡️ is the product going to fail embarrassingly?

➡️ when is quality good enough to launch?

➡️ are we getting enough (high quality) shots on goal?

As a product person you want to move fast and build products that delight your users and drive business impact. This is what your boss wants you to do, too

But it’s hard when organizations are risk averse and very aware that LLMs can fail with lots of media coverage. The number of roadblocks and hurdles product people have to overcome has never been greater

What should you do about it? It starts with good data.

You need to understand how people are using your product and how well it’s performing - so your product doesn't exist in a black box.

With good tooling that tracks usage, quality, and risks you can more quickly run experiments, overcome organizational friction, and ship better products.

1️⃣ Usage → how and why are people using your LLM product? What do they talk about when they first sign up? What predicts them upgrading to paid?

2️⃣ Quality → how well are user needs being met? Where are users satisfied vs frustrated?

3️⃣ Risks → where are your guardrails failing to protect your product from malicious inputs?

And it makes the product development process run more smoothly. Reading raw chat transcripts is not a good solution in 2024

Contact us to learn more about we’re helping Fortune 500s and tech unicorns answer these questions