Product Update | December 2023

New evals features include test set run comparisons, custom global LLM evaluators, per-query evaluators, creation of eval cases from production transcripts, a new evals onboarding, creation of test cases in the UI, and many UX improvements.

Henry Scott-Green

Dec 29, 2023 — 2 min read

December has been all about evals at Context.ai! New evals features include test set run comparisons, custom global LLM evaluators, per-query evaluators, creation of eval cases from production transcripts, a new evals onboarding, creation of test cases in the UI, and many UX improvements.

Test Set Run Comparisons

Completed Test Set runs can now be directly compared to one another from the Test Set Version table. Multi-select any number of Test Set Versions and click compare, and you’ll see the of the executed Test Sets compared to one another, with the evaluator results and the generated responses for each.

Custom Global LLM Evaluators

When we launched evals we only supported a number of default global evaluators that can run over every query, assessing common problems such as hallucination and response refusal. Users can now define additional custom global LLM evaluators from the Evaluators page by selecting Create New Evaluator, and adding an LLM prompt.

Per Query Evaluators

Evaluators can now be enabled at the Test Case level in addition to the Test Set level. This allows you to define evaluators that should only run over one or a subset of the Test Cases within your Test Set, giving you more granular control of pass criteria for your evaluators.

Create Test Cases From User Transcripts

Defining Test Cases is often time-consuming and frustrating, so you can now use production transcripts to make this much quicker and easier! Users with real user transcripts in their analytics can now copy the queries into new Test Cases from any point in the conversation.

Create New Test Cases

New Test Cases can now be created in the UI in addition to the API, allowing you to easily define additional tests within the product.

Fork Test Set Versions

Existing Test Sets can now be forked in the UI to create a new version that can be modified in the UI. This allows you to vary system prompts or queries to assess the performance impact of changes you make.

New Onboarding

Onboarding to evals no longer requires the ingestion of analytics transcripts - anyone can get started with evals before using our analytics products.

Hiring!

We’re hiring! We’re looking for a Founding GTM Lead to join our team in London. This is a generalist position and will be our first non-engineering hire. We’re looking for someone with experience in GTM at early-stage startups and for technical B2B products. More information here.

Evaluating Multi-Call Chains & Product Update | April 2024

Today we’re launching the ecosystem’s best support for evaluating multi-call chains 🔎⛓️ This allows you to evaluate multi-stage workflows with many calls to LLMs and functions, and you can evaluate these both end-to-end and across any stage of the chain You can then debug exactly where errors are occuring

Comparing multi-agent frameworks

There’s a more powerful way to use LLMs in your applications: multi-agent frameworks. Most of us are familiar with making calls to an LLM directly, or using techniques like RAG to increase relevance and context that LLMs have. These approaches let you access human-like reasoning in your application, but

Product Update | March 2024

What did we ship in March? 🚢 Lots of improvements to evals! We now allow users to repeat LLM generations and evaluations to get more certainty in evaluation results, we version our custom evaluators, we’ve improved support for large test sets and added search, we now support Mistral models and

Context.ai is now integrated with Haystack 2.0!

Context.ai now provides developers using Haystack with product analytics via the new Context.ai Haystack integration. This integration allows the easy export of transcripts from Haystack 2.0 to Context.ai for analysis. Analytics are then visible in Context.ai, allowing developers to monitor the performance of their LLM