How to ship amazing LLM products (without getting fired when they go wrong):

Henry Scott-Green

Jun 4, 2024 — 1 min read

Building LLM products requires threading a needle. Product people trade off iteration speed, product quality, and safety risks - and the best PMs know how to do it 🧵🪡

➡️ is the product going to fail embarrassingly?

➡️ when is quality good enough to launch?

➡️ are we getting enough (high quality) shots on goal?

As a product person you want to move fast and build products that delight your users and drive business impact. This is what your boss wants you to do, too

But it’s hard when organizations are risk averse and very aware that LLMs can fail with lots of media coverage. The number of roadblocks and hurdles product people have to overcome has never been greater

What should you do about it? It starts with good data.

You need to understand how people are using your product and how well it’s performing - so your product doesn't exist in a black box.

With good tooling that tracks usage, quality, and risks you can more quickly run experiments, overcome organizational friction, and ship better products.

1️⃣ Usage → how and why are people using your LLM product? What do they talk about when they first sign up? What predicts them upgrading to paid?

2️⃣ Quality → how well are user needs being met? Where are users satisfied vs frustrated?

3️⃣ Risks → where are your guardrails failing to protect your product from malicious inputs?

And it makes the product development process run more smoothly. Reading raw chat transcripts is not a good solution in 2024

What product experiences are enabled by multi-agent LLM frameworks?

It feels like everyone is excited about multi-agent frameworks - even though their performance isn’t yet ready for prime-time. These performance problems are improving with increasingly powerful models like Claude 3 and GPT-4o - and great things are expected from GPT-5, a launch that will likely make agentic workflows

Launching Custom Conversion Events - Product Update | July 2024

Today we’re launching support for custom conversion events 🧾 This addresses one of the biggest challenges in the LLM ecosystem - proving ROI 📈 Context.ai users can now log custom conversion events with their LLM conversation transcripts, indicating where users completed an action: a purchase, a link click, or even

Are your LLM Products Guardrails working?

How do you know if the guardrails on your LLM product are working? 🛡️🎯 Some people wait until they show up in the The New York Times - like McDonald's, Air Canada, or Chevrolet Conversational LLM products are a challenging consumer experience as users can ask an infinite number

Is LLM progress slowing?

LLMs haven’t significantly improved since GPT4: is progress slowing? 🐢 Dramatically more powerful model training clusters are being built: 15 of them, with 31 times more power than trained GPT4 This means models much more powerful than GPT4 are coming 🐇 SemiAnalysis did a phenomenal deep dive into this topic -

Read more

What product experiences are enabled by multi-agent LLM frameworks?

Launching Custom Conversion Events - Product Update | July 2024

Are your LLM Products Guardrails working?

Is LLM progress slowing?