Product Update | March 2024

Henry Scott-Green

Apr 2, 2024 — 2 min read

What did we ship in March? 🚢

Lots of improvements to evals!

We now allow users to repeat LLM generations and evaluations to get more certainty in evaluation results, we version our custom evaluators, we’ve improved support for large test sets and added search, we now support Mistral models and a new Haystack integration, and our comparison page and global evaluator assignment have been improved.

Got feedback or ideas for the team? Please get in touch

Multi-run evals

You can now re-run a LLM generation and the subsequent evaluation for multiple iterations, to better account for the non-deterministic nature of both phases. Running the generation and evaluation 3, 5, or 7 times and taking a majority result gives you more confidence in an evaluator outcome, and a more granular result to review.

Custom evaluator versioning

Custom evaluators are now versioned! This means you can reference a specific version of the evaluator as you update it, and avoids the challenge of evaluators updating and changing their behavior

Better support for large test sets

We now have significantly improved support for large test sets with many test cases

Test case search

Related to the above point, we now support searching over all the test cases within a test set. This is helpful for users with large test sets

Mistral models

Mistral models Mistral 7B and Mixtral are now supported for generation

Model comparison pages

Model comparison pages have been significantly improved, with better visualizations of the differences between up to four test set versions.

Improved global evaluator assignment

We’ve refreshed the assignment flow for global evaluators, improving the flow to assign an evaluator to every test case within a test set

Haystack integration

We launched our integration with Haystack! This allows users of Haystack to easily log transcripts from Haystack to Context for analysis

Evaluating Multi-Call Chains & Product Update | April 2024

Today we’re launching the ecosystem’s best support for evaluating multi-call chains 🔎⛓️ This allows you to evaluate multi-stage workflows with many calls to LLMs and functions, and you can evaluate these both end-to-end and across any stage of the chain You can then debug exactly where errors are occuring

Comparing multi-agent frameworks

There’s a more powerful way to use LLMs in your applications: multi-agent frameworks. Most of us are familiar with making calls to an LLM directly, or using techniques like RAG to increase relevance and context that LLMs have. These approaches let you access human-like reasoning in your application, but

Context.ai is now integrated with Haystack 2.0!

Context.ai now provides developers using Haystack with product analytics via the new Context.ai Haystack integration. This integration allows the easy export of transcripts from Haystack 2.0 to Context.ai for analysis. Analytics are then visible in Context.ai, allowing developers to monitor the performance of their LLM

The Ultimate Guide to LLM Product Evaluation

Large Language Models are incredibly impressive, and the number of products with LLM-based features is growing exponentially But the excitement of launching an LLM product is often followed by important questions: how well is it working? Are my changes improving it? What follows are usually rudimentary, home-grown evaluations (or evals)