Log Pre-generated Responses

If you already have a dataset of requests and application responses, and you want to log and evaluate these on Galileo without re-generating the responses, you can do so via our custom loggers.

First, log in via pq.login()

import promptquality as pq
pq.login({YOUR_GALILEO_URL})

Then, for each request/response in your eval set, construct a node row:

from promptquality import NodeType, NodeRow


# Your previously generated requests & responses
data = [{'request': 'Say hi', 'response': 'Hi!'},
        {'request': 'Say hey', 'response': 'Hey!'}]

rows = []

for d in data:
    chain_id = uuid.uuid4()
    rows.append(
        NodeRow.for_llm(
            id=chain_id,
            root_id=chain_id,
            step=0,
            prompt=d['request'],
            response=d['response']
        )
    )

Finally, log your NodeRows to Galileo and specify the list of metrics you'd like to compute for this run:

pq.chain_run(rows, project_name=<project-name>, scorers=[<list-of-scorers>])

Once that's complete, this step will display the link to access the run from your Galileo Console.

RAG workflows

If you're looking to recreate a RAG workflow, log your retriever step and your LLM step separately. This will allow you to compute RAG metrics and inspect the documents or chunks returned. Importantly, also add both of these nodes as children of a 'chain' NodeRow.

from promptquality import NodeType, NodeRow
import uuid

rows = []

CHAIN_ROOT_ID = uuid.uuid4(), # Randomly generated UUID
rows.append(
    NodeRow(
        node_id=CHAIN_ROOT_ID,
        chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
        step = 0, #an integer indicating which step this node is
        node_input=..., # input into your overall sequence or chain
        node_output=..., # output of your overall sequence or chain
        latency=..., # latency of this step/node. in nanoseconds
        node_type=NodeType.chain # Can be chain, retriever, llm, chat, agent, tool
    )
)

rows.append(
    NodeRow.for_retriever(
        id=uuid.uuid4(), # Randomly generated UUID
        root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
        step = 1, #an integer indicating which step this node is
        query=..., # input into your retriever
        documents=..., # serialized output of the retriever (i.e. json.dumps([{"page_content": "doc_1", "metadata": {"key": "val"}}, {"page_content": "doc_2", "metadata": {"key": "val"}}, ...]))
        # If no metadata exists for the documents, you can pass them as strings in a list.
        latency=..., # latency of this step/node. in nanoseconds
    )
)

rows.append(
    NodeRow.for_llm(
        id=uuid.uuid4(), # Randomly generated UUUID
        root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
        step = 2, #an integer indicating which step this node is
        prompt = ..., # input into your llm (i.e. user query + relevant contexts passed in as a string)
        response = ..., # output of the llm passed in as a string
        latency=..., # latency of this step/node. in nanoseconds
    )
)

We recommend you randomly generate node_id and chain_root_id (e.g. uuid()). Add the id of the 'parent' node as the chain_root_id of its children.

Last updated