Chunk Attribution

Understand Galileo's Chunk Attribution Metric

The metric is intended for RAG workflows.

Definition: For each chunk retrieved in a RAG pipeline, Chunk Attribution measures whether or not that chunk had an effect on the model's response.

Chunk Attribution is a binary metric: each chunk is either Attributed or Not Attributed.

Chunk Attribution is closely related to Chunk Utilization: Attribution measures whether or not a chunk affected the response, and Utilization measures how much of the chunk text was involved in the effect. Only chunks that were Attributed can have Utilization scores greater than zero.

Usefulness: Chunk Attribution can help you iterate on your RAG pipeline in several different ways:

• Tuning the number of retrieved chunks.

  • If your system is producing satisfactory responses, but many chunks are Not Attributed, then you may be able to reduce the number of chunks retrieved per example without adversely impacting response quality.

  • This will improve the efficiency of the system, resulting in lower cost and latency.

• "Debugging" anomalous model behavior in individual examples.

  • If a specific model response is unsatisfactory or unusual, and you want to understand why, Attribution can help you zero in on the chunks that affected the response.

  • This lets you get to the root of the issue more quickly when inspecting individual examples.

Basic vs Plus

We offer two ways of calculating Completeness: Basic and Plus.

Chunk Attribution Basic is computed using Galileo in-house small language models. They're free of cost. Completeness Basic is a cost effective way to scale up you RAG evaluation workflows.

Chunk Attribution Plus is computed by sending an additional request to your LLM. It relies on OpenAI models so it incurs an additional cost. Chunk Attritbution Plus has shown better results in internal benchmarks.