Product Update | January 2024

Henry Scott-Green

Jan 31, 2024 — 3 min read

Our Evals keep getting better thanks to the great user feedback - please keep it coming! This month's improvements include more evaluators, including golden responses and semantic similarity, a new comparison page, our prompt playground, online evals, message-level analytics metadata, new evals onboarding, UX improvements across the app, and support for many new models. More information on each of these is below!

More Evaluators - Golden Responses, Semantic Similarity

We now support many additional evaluators, including golden responses and semantic similarity matches. This allows you to more specifically define pass criteria for individual eval cases, requiring either an exactly matching response, or a semantically similar response.

Improved Comparison Page

Test Set comparisons are now significantly improved, with new UX and chain-of-thought reasoning available for evaluator decisions.

Prompt Playground

Our new prompt playground lets you rapidly test different prompts and their performance, as you prompt engineer and fix failing evals. Keep your eyes peeled for significant updates to this in the next few weeks!

Online Evals MVP

You can now run evaluators over samples of production transcripts. This allows you to run the same evals you run in development over samples of real user transcripts in your production environment, so you can compare performance in dev and prod, and detect regressions.

This is a premium-only feature, please reach out to henry@context.ai to be enabled

Message-Level Metadata

Custom metadata on analytics transcripts allow you to annotate logged interactions with metadata such as models, user IDs, development environments and other metadata. We now support this metadata at the message level as well as the conversation level, allowing you to log the state of a conversation or user feedback to a specific message.

New Evals Onboarding

Our evals tooling now has an even easier setup process. With one click, new users can create a ready-to-go test set containing a sample test case. This makes it even easier to get started and to modify the default test set.

More Models

New models include Meta’s Llama 2 70b/13b, Google’s Gemini Pro, Anthropic’s Claude v2/instant, and the new GPT4 Turbo models recently released by OpenAI

These models can be used both in our eval and analytics tools - to generate responses to evaluate, to test in the playground, and to estimate pricing in analytics

UX Improvements

And finally - we keep polishing the product! You should notice usability improvements throughout the product. Please reach out if you have workflows you’re finding hard to complete, we’d love to improve things for you!

Evaluating Multi-Call Chains & Product Update | April 2024

Today we’re launching the ecosystem’s best support for evaluating multi-call chains 🔎⛓️ This allows you to evaluate multi-stage workflows with many calls to LLMs and functions, and you can evaluate these both end-to-end and across any stage of the chain You can then debug exactly where errors are occuring

Comparing multi-agent frameworks

There’s a more powerful way to use LLMs in your applications: multi-agent frameworks. Most of us are familiar with making calls to an LLM directly, or using techniques like RAG to increase relevance and context that LLMs have. These approaches let you access human-like reasoning in your application, but

Product Update | March 2024

What did we ship in March? 🚢 Lots of improvements to evals! We now allow users to repeat LLM generations and evaluations to get more certainty in evaluation results, we version our custom evaluators, we’ve improved support for large test sets and added search, we now support Mistral models and

Context.ai is now integrated with Haystack 2.0!

Context.ai now provides developers using Haystack with product analytics via the new Context.ai Haystack integration. This integration allows the easy export of transcripts from Haystack 2.0 to Context.ai for analysis. Analytics are then visible in Context.ai, allowing developers to monitor the performance of their LLM