Setting Up Alerts
How to set up Alerts and automatically be alerted when things go wrong
Galileo enables you to get alerted whenever unexpected things happen (e.g. your cost is higher than expected, your model is hallucinating more than you want, users are entering foul language into your app).
Pre-requisites
Before setting up alerting, make sure you:
To set up your alerts, you need to define:
- 1.Who should be alerted (and how)
- 2.What they should be alerted on
Your Alerting Settings will be under your Project Settings page (i.e. the icon near the top-right of your Monitoring Dashboard).
⚙
To define who should be alerted, scroll to the Alerting section, and add your recipients' email addresses:
Then configure your alerts:
Each alert is composed of a Metric (e.g. Correctness, Cost, Toxicity), an Aggregation Function (e.g. Min, Max, Average, Total), a threshold (e.g. greater than 0.5), and a time window (e.g. hourly).
Example Alerts:
- Exceeding costs: If you want to get alerted with an uptick in cost (above $10/day, you select
Sum ValueofCostismore than or equal toyour10in thelast day. - Hallucinations: If you want to get alerted any time you have a hallucination, select
Min ValueofCorrectnessorContext Adherenceisequal to0. - Hallucination Rate: If you're comfortable with a certain level of hallucinations (e.g. 5%), you can select
Average valueofCorrectnessorContext Adherenceisless than or equalto0.05.
Triggering Alerts
Once your Alerts are configured, we periodically run jobs to check whether your Alerts have been triggered. Once they do, you should receive an email letting you know which alert has triggered and what the value of the alert is.
From your email, you can click on the "Open Project" link to open your dashboard and find the problematic requests.
Last modified 12d ago