The challenge of measuring the immeasurable

Evaluating retrieval-augmented generation (RAG) is easier than ever, but you need to keep a close eye on the LLMs that drive the evaluation metrics. We discuss some common pitfalls and solutions.

Do you regularly wonder where your time went during a week worth of work? We present work-dAIgest, a tool developed during Tweag's GenAI hackathon that aims to use data from standard workplace tools such as GitHub, Google Calendar, ... to create a summary of your work week (or any other time period), powered by open-source large language models (LLM).

We introduce an experimental framework for assessing RAG retrieval and explain its main components including benchmark creation, evaluation metrics, parameter space and experiment tracking. Whether you're deep in the field or just starting, this post is your gateway to understanding and improving RAG retrieval performance.

We introduce the many pitfalls of RAGs, including in retrieval, explain why we need systematic evaluation and provide a quick introduction to existing evaluation frameworks.