python[minor]: pytest integration

[baskaryan]

Log pytest tests to LangSmith. Useful for:

• Evaluations where the eval logic is different for each datapoint, making it difficult to use generic evaluators on a whole dataset
• Unit tests where you want both the local pytest experience and the ability to trace and share results

Install

pip install "langsmith[pytest]==0.2.11rc14"

Simple usage

# tests/test_simple.py
import pytest
from langsmith import testing as t

@pytest.mark.langsmith
def test_addition_single():
    x = 3
    y = 4
    # directly log example inputs if you don't want to use fixtures
    t.log_inputs({"x": x, "y": y})

    expected = 7
    # directly log example outputs if you don't want to use fixtures
    t.log_reference_outputs({"sum": expected})

    actual = x + y
    # directly log run outputs
    t.log_outputs({"sum": actual})
    
    # test pass/fail automatically logged to langsmith
    assert actual == expected

Run

pytest --outputs='ls' tests/test_foo.py

Results

Advanced usage

#tests/test_advanced.py
import openai
import pytest

from langsmith import wrappers
from langsmith import testing as t

oai_client = wrappers.wrap_openai(openai.Client())

@pytest.mark.langsmith
def test_openai_says_hello():
    # Traced code will be included in the test case
    text = "Say hello!"
    response = oai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": text},
        ],
    )
    t.log_inputs({"text": text})
    t.log_outputs({"response": response.choices[0].message.content})
    t.log_reference_outputs({"response": "hello!"})

    # Use this context manager to trace any steps used for generating evaluation 
    # feedback separately from the main application logic
    with t.trace_feedback():
        grade = oai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": "Return 1 if 'hello' is in the user message and 0 otherwise.",
                },
                {"role": "user", "content": response.choices[0].message.content},
            ],
        )
        t.log_feedback(
            key="llm_judge", score=float(grade.choices[0].message.content)
        )

    assert "hello" in response.choices[0].message.content.lower()


@pytest.mark.langsmith(output_keys=["expected"])
@pytest.mark.parametrize(
    "a, b, expected",
    [(1, 2, 3), (3, 4, 7)],
)
def test_addition_parametrized(a: int, b: int, expected: int):
    t.log_outputs({"sum": a + b})
    assert a + b == expected

Run using pytest-xdist to parallelize (pip install pytest-xdist first)

LANGSMITH_TEST_SUITE="Test suite name" LANGSMITH_EXPERIMENT="Experiment name" pytest  --outputs='ls' tests

Results: https://dev.smith.langchain.com/public/cea0e7fd-2d27-47d1-8ada-141069acdf0d/d

[hinthornw]

I like the functionality. I kinda like using a single module/object to access the methods, like

@unit
def test_example(captest):
    x = 0
    y = 1
    captest.log_inputs({"x": x, "y": y})
    captest.log_reference({"product": 0})
    captest.log_outputs({"product": x * y})
    assert x * y == 0

or something

[hinthornw]

I like

[jxnl]

why is it "reference" input vs

target, expect, etc

[baskaryan]

needs review, very open to less hacky ideas

[agola11]

can't think of any great solutions right now without async. Would add a detailed comment about this and address later as a TODO

[baskaryan]

worth review

[baskaryan]

worth review

[agola11]

looks reasonable to me

[agola11]

just one main comment

[agola11]

why do we need to submit this in an executor? run_tree.patch calls update_run which is async already

[baskaryan]

to not block on the self.wait_example_updates(example_id) call

[agola11]

can't think of any great solutions right now without async. Would add a detailed comment about this and address later as a TODO

[agola11]

looks reasonable to me