Getting started with LangSmith

Introduction

LangSmith is a platform for building production-grade LLM applications. It allows you to closely monitor and evaluate your application, so you can ship quickly and with confidence. Use of LangChain is not necessary - LangSmith works on its own!

Install LangSmith

We offer Python and Typescript SDKs for all your LangSmith needs.

Python

pip install -U langsmith

TypeScript

yarn add langchain langsmith

Create an API key

To create an API key head to the setting pages. Then click Create API Key.

Setup your environment

Shell

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=<your-api-key>

# The below examples use the OpenAI API, though it's not necessary in general
export OPENAI_API_KEY=<your-openai-api-key>

Log your first trace

We provide multiple ways to log traces to LangSmith. Below, we'll highlight how to use traceable. See more on the Integrations page.

Python

import openai
from langsmith.wrappers import wrap_openai
from langsmith import traceable

# Auto-trace LLM calls in-context
client = wrap_openai(openai.Client())

@traceable # Auto-trace this function
def pipeline(user_input: str):
    result = client.chat.completions.create(
        messages=[{"role": "user", "content": user_input}],
        model="gpt-3.5-turbo"
    )
    return result.choices[0].message.content

pipeline("Hello, world!")
# Out:  Hello there! How can I assist you today?

TypeScript

import { OpenAI } from "openai";
import { traceable } from "langsmith/traceable";
import { wrapOpenAI } from "langsmith/wrappers";

// Auto-trace LLM calls in-context
const client = wrapOpenAI(new OpenAI());
// Auto-trace this function
const pipeline = traceable(async (user_input) => {
    const result = await client.chat.completions.create({
        messages: [{ role: "user", content: user_input }],
        model: "gpt-3.5-turbo",
    });
    return result.choices[0].message.content;
});

await pipeline("Hello, world!")
// Out: Hello there! How can I assist you today?

• View a sample output trace.
• Learn more about tracing on the tracing page.

Create your first evaluation

Evalution requires a system to test, data to serve as test cases, and optionally evaluators to grade the results. Here we use a built-in accuracy evaluator.

Python

from langsmith import Client
from langsmith.evaluation import evaluate

client = Client()

# Define dataset: these are your test cases
dataset_name = "Sample Dataset"
dataset = client.create_dataset(dataset_name, description="A sample dataset in LangSmith.")
client.create_examples(
    inputs=[
        {"postfix": "to LangSmith"},
        {"postfix": "to Evaluations in LangSmith"},
    ],
    outputs=[
        {"output": "Welcome to LangSmith"},
        {"output": "Welcome to Evaluations in LangSmith"},
    ],
    dataset_id=dataset.id,
)

# Define your evaluator
def exact_match(run, example):
    return {"score": run.outputs["output"] == example.outputs["output"]}

experiment_results = evaluate(
    lambda input: "Welcome " + input['postfix'], # Your AI system goes here
    data=dataset_name, # The data to predict and grade over
    evaluators=[exact_match], # The evaluators to score the results
    experiment_prefix="sample-experiment", # The name of the experiment
    metadata={
      "version": "1.0.0",
      "revision_id": "beta"
    },
)

TypeScript

import { Client, Run, Example } from 'langsmith';
import { runOnDataset } from 'langchain/smith';
import { EvaluationResult } from 'langsmith/evaluation';

const client = new Client();

// Define dataset: these are your test cases
const datasetName = "Sample Dataset";
const dataset = await client.createDataset(datasetName, {
    description: "A sample dataset in LangSmith."
});
await client.createExamples({
    inputs: [
        { postfix: "to LangSmith" },
        { postfix: "to Evaluations in LangSmith" },
    ],
    outputs: [
        { output: "Welcome to LangSmith" },
        { output: "Welcome to Evaluations in LangSmith" },
    ],
    datasetId: dataset.id,
});

// Define your evaluator
const exactMatch = async ({ run, example }: { run: Run; example?: Example; }): Promise<EvaluationResult> => {
    return {
        key: 'exact_match',
        score: run.outputs?.output === example?.outputs?.output ? 1 : 0,
    };
};

await runOnDataset(
    (input: { postfix: string }) => ({ output: `Welcome ${input.postfix}` }), // Your AI system goes here
    datasetName, // The data to predict and grade over
    {
        evaluationConfig: { customEvaluators: [exactMatch] },
        projectMetadata: {
            version: "1.0.0",
            revision_id: "beta",
        },
    }
);

• See more on the evaluation quick start page.

Next Steps

Check out the following sections to learn more about LangSmith:

• User Guide: Learn about the workflows LangSmith supports at each stage of the LLM application lifecycle.
• Pricing: Learn about the pricing model for LangSmith.
• Self-Hosting: Learn about self-hosting options for LangSmith.
• Proxy: Learn about the proxy capabilities of LangSmith.
• Tracing: Learn about the tracing capabilities of LangSmith.
• Evaluation: Learn about the evaluation capabilities of LangSmith.
• Prompt Hub Learn about the Prompt Hub, a prompt management tool built into LangSmith.

Additional Resources

• LangSmith Cookbook: A collection of tutorials and end-to-end walkthroughs using LangSmith.
• LangChain Python: Docs for the Python LangChain library.
• LangChain Python API Reference: documentation to review the core APIs of LangChain.
• LangChain JS: Docs for the TypeScript LangChain library
• Discord: Join us on our Discord to discuss all things LangChain!

FAQ

How do I migrate projects between organizations?

Currently we do not support project migration betwen organizations. While you can manually imitate this by reading and writing runs and datasets using the SDK (see the querying runs and traces guide here), it will be fastest to create a new project within your organization and go from there.

Why aren't my runs aren't showing up in my project?

If you aren't seeing any warnings when running your application, it may be that you are still using an API key from your "personal" organization. Check your most recent runs there to confirm by selecting your Personal tenant in the settings page and then viewing your projects.

If you're still running into issues, please reach out to us at support@langchain.dev.

My team deals with sensitive data that cannot be logged. How can I ensure that only my team can access it?

If you are interested in a private deployment of LangSmith or if you need to self-host, please reach out to us at sales@langchain.dev. Self-hosting LangSmith requires an annual enterprise license that also comes with support and formalized access to the LangChain team.