Prompt Perplexity

Understanding Galileo's Prompt Perplexity Metrics

Definition: Measures the Prompt Perplexity, using the log probability's provided by most models of the prompt.

Calculation: Prompt Perplexity is calculated as the exponential of the negative average of the log probability's over the entire prompt. Thus it ranges from 0-infinity with lower values indicating the model on average was more certain of the next token in a sequence.

Usefulness: Lower perplexity indicates your model is better tuned towards your data, as it can better predict the next token. Furthermore, the paper Demystifying Prompts in Language Models via Perplexity Estimation has shown that lower perplexity values in the input (aka. prompt) also lead to better outcomes in the generations (aka. results).