Chunk Attribution

Understand Galileo's Chunk Attribution Metric

The metric is intended for RAG workflows.

Definition: For each chunk retrieved in a RAG pipeline, Chunk Attribution measures whether or not that chunk had an effect on the model's response.

Chunk Attribution is a binary metric: each chunk is either Attributed or Not Attributed.

Chunk Attribution is closely related to Chunk Utilization: Attribution measures whether or not a chunk affected the response, and Utilization measures how much of the chunk text was involved in the effect. Only chunks that were Attributed can have Utilization scores greater than zero.

Calculation: Chunk Attribution is computed by sending an additional request to your LLM, using a carefully engineered prompt that asks the model to trace information in the response back to individual chunks and sentences within those chunks.

The same prompt is used for both Chunk Attribution and Chunk Utilization, and a single LLM request is used to compute both metrics at once.

Usefulness: Chunk Attribution can help you iterate on your RAG pipeline in several different ways:

  • Tuning the number of retrieved chunks.

    • If your system is producing satisfactory responses, but many chunks are Not Attributed, then you may be able to reduce the number of chunks retrieved per example without adversely impacting response quality.

    • This will improve the efficiency of the system, resulting in lower cost and latency.

  • "Debugging" anomalous model behavior in individual examples.

    • If a specific model response is unsatisfactory or unusual, and you want to understand why, Attribution can help you zero in on the chunks that affected the response.

    • This lets you get to the root of the issue more quickly when inspecting individual examples.

Deep dive: to read more about the research behind this metric, see RAG Quality Metrics.

Note: This metric is computed by prompting an LLM, and thus requires additional LLM calls to compute.

Last updated