Custom Chain

If you're not using an orchestration library, or using one other than Langchain, we also provide a similar interface for uploading your executions that do not use a callback mechanism. To log your runs with Galileo, you'd start with the same typical flow of logging into Galileo:

import promptquality as pq
pq.login({YOUR_GALILEO_URL})

Then, for each step of your sequence (or node in the chain), construct a chain row:

from promptquality import NodeType, NodeRow

rows = [
    NodeRow(node_id=..., 
            chain_root_id=..., 
            node_type=<ChainNodeType>)
]

For example, you can log your retriever and llm node with the snippet below.

from promptquality import NodeType, NodeRow
import uuid

rows = []

CHAIN_ROOT_ID = uuid.uuid4() # Randomly generated UUID
rows.append(
    NodeRow(node_id=CHAIN_ROOT_ID,
             chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
             step = 0, #an integer indicating which step this node is
             node_input=..., # input into your overall sequence or chain
             node_output=..., # output of your overall sequence or chain
             latency=..., # latency of this step/node. in nanoseconds
             node_type=NodeType.chain # Can be chain, retriever, llm, chat, agent, tool
         )
)

rows.append(
    NodeRow(node_id=uuid.uuid4(), # Randomly generated UUID
             chain_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
             chain_root_id=CHAIN_ROOT_ID, # UUID of the root node
             step = 1, #an integer indicating which step this node is
             node_input=..., # input into your retriever
             node_output=..., # serialized output of the retriever (i.e. json.dumps([{"page_content": "doc_1", "metadata": {"key": "val"}}, {"page_content": "doc_2", "metadata": {"key": "val"}}, ...]))
             latency=..., # latency of this step/node. in nanoseconds
             node_type=NodeType.retriever # Can be chain, retriever, llm, chat, agent, tool
             )
)

rows.append(
    NodeRow(node_id=uuid.uuid4(), # Randomly generated UUUID
             chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
             step = 2, #an integer indicating which step this node is
             node_input=..., # input into your llm (i.e. user query + relevant contexts passed in as a string)
             prompt = ..., # input into your llm (i.e. user query + relevant contexts passed in as a string)
             node_output=..., # output of the llm passed in as a string
             response = ..., # output of the llm passed in as a string
             latency=..., # latency of this step/node. in nanoseconds
             node_type=NodeType.llm # Can be chain, retriever, llm, chat, agent, tool
             )
)

We recommend you randomly generate node_id and chain_root_id (e.g. uuid()). Add the id of a 'parent' node as the chain_root_id of its children.

When your execution completes, log that data to Galileo:

pq.chain_run(rows, project_name=<project-name>, scorers=[<list-of-scorers>])

Once that's complete, this step will display the link to access the run from your Galileo Console.

Logging metadata

If you are logging chains from langchain, metadata values (such as chunk-level metadata for the retriever) will be automatocally included.

For custom chains, metadata values can be logged by dumping metadata along with page_content as demonstrated below.

from promptquality import NodeType, NodeRow
import uuid

retriever_output = [ { "page_content": "chunk 1 content", "metadata": {"key": "value"} }, { "page_content": "chunk 2 content", "metadata": {"key": "value"} }, ]

rows = []

rows.append(
    NodeRow(
        node_id=uuid.uuid4(), # Randomly generated UUID
        chain_root_id=..., # UUID of the 'parent' node
        step = ...,  #an integer indicating which step this node is
        node_type=NodeType.retriever, 
        step=..., #an integer indicating which step this node is
        node_input="the query to the retriever", 
        node_output=json.dumps(retriever_output) )
)