Today we’re launching the ecosystem’s best support for evaluating multi-call chains 🔎⛓️ This allows you to evaluate multi-stage workflows with many calls to LLMs and functions, and you can evaluate these both end-to-end and across any stage of the chain You can then debug exactly where errors are occuring
There’s a more powerful way to use LLMs in your applications: multi-agent frameworks. Most of us are familiar with making calls to an LLM directly, or using techniques like RAG to increase relevance and context that LLMs have. These approaches let you access human-like reasoning in your application, but
What did we ship in March? 🚢 Lots of improvements to evals! We now allow users to repeat LLM generations and evaluations to get more certainty in evaluation results, we version our custom evaluators, we’ve improved support for large test sets and added search, we now support Mistral models and
Context.ai now provides developers using Haystack with product analytics via the new Context.ai Haystack integration. This integration allows the easy export of transcripts from Haystack 2.0 to Context.ai for analysis. Analytics are then visible in Context.ai, allowing developers to monitor the performance of their LLM
Large Language Models are incredibly impressive, and the number of products with LLM-based features is growing exponentially But the excitement of launching an LLM product is often followed by important questions: how well is it working? Are my changes improving it? What follows are usually rudimentary, home-grown evaluations (or evals)
What’s been cooking at Context.ai?👨🍳 Many more improvements to evals, much-requested analytics features, and some cookbooks too! Evals now provide Test Set-level comparisons, a side-by-side prompt playground, streaming responses, quality improvements to core CUJs, improved evaluator performance, and support for many more model parameters. Analytics now supports free
Our Evals keep getting better thanks to the great user feedback - please keep it coming! This month's improvements include more evaluators, including golden responses and semantic similarity, a new comparison page, our prompt playground, online evals, message-level analytics metadata, new evals onboarding, UX improvements across the app,
When building applications using LLMs you have to make a large number of decisions - what model should you use? What system prompt? What model parameters, RAG system, and vector database? Answering those questions when you’re building for the first time is hard, and knowing if you’ve made
Our mission at Context.ai is to stop AI builders from flying blind. AI will have a staggering and positive impact over the next decade. We want to accelerate this transition and the enormous benefits it will bring - from healthcare to productivity. But building high performing, reliable, and safe
New evals features include test set run comparisons, custom global LLM evaluators, per-query evaluators, creation of eval cases from production transcripts, a new evals onboarding, creation of test cases in the UI, and many UX improvements.
What started as a tool for the Context.ai engineering team became an open-source utility we're now excited to be sharing.
We’ve shipped evaluations, a product redesign, embedded analytics via API, user retention analytics, alerts for areas of poor performance, a new ingestion model, manual ratings, and many smaller improvements.