Log Pre-generated Responses

If you already have a dataset of requests and application responses, and you want to log and evaluate these on Galileo without re-generating the responses, you can do so via our custom loggers.

First, log in via pq.login()

import promptquality as pq
pq.login({YOUR_GALILEO_URL})

Then, for each request/response in your eval set, construct a node row:

from promptquality import NodeType, NodeRow


# Your previously generated requests & responses
data = [{'request': 'Say hi', 'response': 'Hi!'},
        {'request': 'Say hey', 'response': 'Hey!'}]

rows = []

for d in data:
    chain_id = uuid.uuid4()
    rows.append(
        NodeRow.for_llm(
            id=chain_id,
            root_id=chain_id,
            step=0,
            prompt=d['request'],
            response=d['response'],
            target='Hi!' #expected output optional to enable Bleu and Rouge scores
        )
    )

Finally, log your NodeRows to Galileo and specify the list of metrics you'd like to compute for this run:

pq.chain_run(rows, project_name=<project-name>, scorers=[<list-of-scorers>])

Once that's complete, this step will display the link to access the run from your Galileo Console.

See below if you're logging a RAG workflow.

What is a NodeRow?

A NodeRow is a object representation of one step in your workflow. For example, in an Agent or RAG system:

• Your retriever step would be represented by a NodeRow with the inputs that went into it and the output (chunks) that came out of it.

• Your LLM step(s) would be represented by NodeRows with the prompt into the model as the input and the response from the model as the output.

• An Agent or Chain node would be represented by a NodeRow, with the output of its children nodes being its output.

To recreate a simple RAG system like the one above, you'd log the following Nodes:

1. A NodeRow for your "Workflow" step. Because this node is the "root" of your tree, this node's ID will be the "chain_root_id" of every node in the tree.

2. A NodeRow for your "VectorStoreRetriver" step. because the "Workflow" node is its parent, this node's chain_id will be the ID of the "Workflow" node.

3. A NodeRow for the "CustomLLM" step. Again, because the "Workflow" node is its parent, this node's chain_id will be the ID of the "Workflow" node.

RAG workflows

If you're looking to recreate a RAG workflow, log your retriever step and your LLM step separately. This will allow you to compute RAG metrics and inspect the documents or chunks returned. Importantly, also add both of these nodes as children of a 'chain' NodeRow.

from promptquality import NodeType, NodeRow
import uuid

rows = []

CHAIN_ROOT_ID = uuid.uuid4(), # Randomly generated UUID
rows.append(
    NodeRow(
        node_id=CHAIN_ROOT_ID,
        chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
        step = 0, #an integer indicating which step this node is
        node_input=..., # input into your overall sequence or chain
        node_output=..., # output of your overall sequence or chain
        latency=..., # latency of this step/node. in nanoseconds
        node_type=NodeType.chain # Can be chain, retriever, llm, chat, agent, tool
    )
)

rows.append(
    NodeRow.for_retriever(
        id=uuid.uuid4(), # Randomly generated UUID
        root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
        step = 1, #an integer indicating which step this node is
        query=..., # input into your retriever
        documents=..., # serialized output of the retriever (i.e. json.dumps([{"page_content": "doc_1", "metadata": {"key": "val"}}, {"page_content": "doc_2", "metadata": {"key": "val"}}, ...]))
        # If no metadata exists for the documents, you can pass them as strings in a list.
        latency=..., # latency of this step/node. in nanoseconds
    )
)

rows.append(
    NodeRow.for_llm(
        id=uuid.uuid4(), # Randomly generated UUUID
        root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
        step = 2, #an integer indicating which step this node is
        prompt = ..., # input into your llm (i.e. user query + relevant contexts passed in as a string)
        response = ..., # output of the llm passed in as a string
        latency=..., # latency of this step/node. in nanoseconds
    )
)

We recommend you randomly generate node_id and chain_root_id (e.g. uuid()). Add the id of the 'parent' node as the chain_root_id of its children.