// REO script
!(function () {
  var e, t, n;
  (e = "638190bf025179e"),
    (t = function () {
      Reo.init({ clientID: "638190bf025179e" });
    }),
    ((n = document.createElement("script")).src = "https://static.reo.dev/" + e + "/reo.js"),
    (n.async = !0),
    (n.onload = t),
    document.head.appendChild(n);
})();

// Hubspot script
const script = document.createElement("script");
script.type = "text/javascript";
script.id = "hs-script-loader";
script.async = true;
script.defer = true;
script.src = "//js.hs-scripts.com/23114811.js";
document.head.appendChild(script);


Galileo

Client Reference

API Reference

Examples

AI Research

Deployments

Get GenAI Studio

Invoke

Value

ActionResult

ActionType

ExecutionStatus

Error Message

Execution Time

MetricComputation

List of choices to override the response with. If there are multiple choices, one will be chosen at random when applying this action.

Choices

List of subscriptions to send a notification to when this action is applied and the ruleset status matches any of the configured statuses.

Subscriptions

Type

OverrideAction

PassthroughAction

Input

Output

Payload

Metric

Value to compare with for this metric (right hand side).

Target Value

Rule

RuleOperator

RuleResult

Action

Description

List of rules to evaluate. Atleast 1 rule is required.

Rules

Ruleset

Rule Results

RulesetResult

Project Id

Stage Id

Stage Name

Stage Version

StageMetadata

List of statuses that will cause a notification to be sent to the configured URL.

Statuses

URL to send the event to. This can be a webhook URL, a message queue URL, an event bus or a custom endpoint that can receive an HTTP POST request.

SubscriptionConfig

Execution time for the request (in seconds).

Time the request was received by the server in nanoseconds.

Received At

Time the response was sent by the server in nanoseconds.

Response At

TraceMetadata

Location

Message

Error Type

ValidationError

Request

APIKeyHeader

OAuth2PasswordBearer

HTTPBasic

Response Invoke V1 Protect Invoke Post

HTTPValidationError

Evaluate, Observe, and Protect your GenAI applications

What is Galileo?

Stop experimenting in spreadsheets and notebooks. Use Evaluate’s powerful insights to build GenAI systems that just work.

Overview

Quickstart

Integrate Evaluate Into My Existing Application With Python

Prompt Engineering From A Ui

How To

Before starting your experiments, we recommend creating an evaluation set.

Create an Evaluation Set

How to use Galileo Evaluate for prompt engineering

Evaluate and Optimize Prompts

How to use Galileo Evaluate with RAG applications

Evaluate and Optimize RAG Applications

How to choose and understand your guardrail metrics

Choose your Guardrail Metrics

Learn how to turn on metrics when creating runs in your Python environment.

Enabling Scorers in Runs

How to use Galileo Evaluate to find Hallucinations

Identify Hallucinations

Galileo GenAI Studio supports Custom Metrics (programmatic or GPT-based) for all your Evaluate and Observe projects. Depending on where, when, and how you want these metrics to be executed, you have the option to choose between **Custom Scorers** and **Registered Scorers**.

Register Custom Metrics

Prompt Management-Storage

Easily compare multiple LLM runs in a single screen for better decision making

A/B Compare Prompts

Execute prompts in bulk for faster experimentation

Experiment with Multiple Prompts

If you're building a multi-step workflow or chain (e.g. a RAG system, an Agent, or a chain) and want to experiment with multiple combinations of parameters or your versions at once, Chain Sweeps are your friend.

Experiment with Multiple Workflows

Galileo allows you to do qualitative human evaluations of your prompts and responses.

Evaluate with Human Feedback

While you are experimenting with your prompts you will probably be tuning many parameters.

Add Tags and Metadata to Prompt Runs

If you already have a dataset of requests and application responses, and you want to log and evaluate these on Galileo without re-generating the responses, you can do so via our worflows.

Log Pre-generated Responses in Python

If you want to fetch your logged data and metrics programmatically, you can do so via our Python clients.

Programmatically fetch logged data

Galileo Evaluate is geared for cross-functional collaboration. Most of the teams using Galileo consist of a mix of the following personas

Collaborate with other personas

All projects on Galileo can be shared with others to enable collaboration.

Share a project

To download the results of your evaluation you can use the Export function. To export your runs, simply click on _Export Prompt Data._

Export your Evaluation Runs

An important step towards debugging and evaluating your LLM applications is understanding your metric values and what led to them.

Understand your metric's values

Expected outputs are a key element for evaluating LLM applications. They provide benchmarks to measure model accuracy, identify errors, and ensure consistent assessments.

Logging and Comparing against your Expected Answers

Improve metric accuracy by customizing your Chainpoll-powered metrics

Customize Chainpoll-powered Metrics

Controlling access to projects using roles and groups

Access Control

Learn how to use Automatic Run Ranking to find the best run

Finding the best run

Learn how to automatically improve your prompts

Automatic Prompt Optimization

Integrations

Galileo allows you to integrate with your Langchain application natively through callbacks

Langchain

No matter how you're orchestrating your workflows, we have an interface to help you upload them to Galileo.

Logging Workflows

Project

Runs in Galileo are experiments or iterations done within a [project](/galileo/gen-ai-studio-products/galileo-evaluate/concepts/project).

Template

Metrics are quantitative or qualitative ways to express insights about the [run](/galileo/gen-ai-studio-products/galileo-evaluate/concepts/run).

Metrics

Your metrics are failing and you're not sure why? Below are a few reasons some of your metrics might fail and what you can do about them

Errors When Computing Metrics

Confused by whether to use Context Adherence or Instruction Adherence? Below are several differences between the two metrics and when to use each one

Context Adherence versus Instruction Adherence

How to monitor your apps with Galileo Observe

Choosing Your Guardrail Metrics

Registered Metrics enable the ability for your team to define the custom metrics (programmatic or GPT-based) for your Observe projects.

Registering And Using Custom Metrics

How to set up Alerts and automatically be alerted when things go wrong

Setting Up Alerts

Once your monitored LLM app is up and running and you've selected your Guardrail Metrics, you can start monitoring your LLM app using Galileo.

Identifying And Debugging Issues

Galileo Observe allows you to monitor your Retrieval-Augmented Generation (RAG) application with out-of-the-box Tracing and Analytics.

Monitoring Your Rag Application

Learn how to manually log your data via our Python Logger

Logging Data Via Python

Programmatically Fetching Logged Data

To download your Observe Data you can use the Export function.

Exporting Your Data

How to Set Up Access Control

Learn how to manually log your data from your Langchain Chains

Logging Data Via Langchain Callback

Creating And Using Stages

Invoking Rulesets

Defining Rules

Your Protect Rules rely on [Guardrail Metrics](/galileo/gen-ai-studio-products/galileo-protect/how-to/supported-metrics-and-operators). Metrics are calculated using ML models, which can have varying latencies.

Setting A Timeout On Your Protect Requests

When you're using the Galileo Protect product, once you've created a project and a stage, you can pause and resume the stage.

Pausing Or Resuming A Stage

Editing Centralized Stages

Galileo Protect can also be used within your Langchain workflows. You can use Protect to validate inputs and outputs at different stages of your workflow. We provide a `tool` that allows you to easily integrate Protect into your Langchain workflows. 

A set of rulesets that are applied during _one_ invocation.

Stage

All of the Rules within a Ruleset are executed in parallel, and the final resolution depends on all of the rules being completed.

A condition or rule you never want your application to break. It's composed of three ingredients

Galileo will provide a set of action types (override, passthrough), that the user can use, along with a configuration for each action type.

Ground Truth Adherence

Instruction Adherence

Correctness

Understand Galileo's Chunk Relevance Luna Metric

Chunk Relevance

Uncertainty

Understanding Galileo's Prompt Perplexity Metrics

Prompt Perplexity

Understand Galileo's Context Relevance Metric

Context Relevance

Private Identifiable Information

Toxicity

Tone

Sexism

BLEU and ROUGE

Understand Galileo's Prompt Injection metric

Prompt Injection

Automatic Data Insights on your Seq2Seq dataset

Configuring Dq Auto

Upon completing a run, you'll be taken to the Galileo Console.

Console Walkthrough

Galileo Alerts are your starting point in your data inspection journey.

Using Alerts

Finetuning an LLM often requires large datasets.

Visualizing And Understanding Your Data

It is crucial to quickly identify errors in ML training data and fix them. This is incredibly hard to do at scale when working with millions of data points.

Using Data Error Potential

On dataset splits where generations are enabled (e.g. the _Test split_), you'll be seeing Uncertainty Scores and Token-level Uncertainty highlighting

Using Uncertainty

Similarity search allows you to discover **similar samples** within your datasets

Finding Similar Samples

Taking Action

Galileo NLP Studio supports Natural Language Processing Tasks across the life-cycle of your model development.

Product Features

The Dataset View provides an interactive data table for inspecting your datasets.

API Reference

health

auth

evaluate

observe

protect

Invoke

Authorizations

Query Parameters

Body

Response