langsmith-cli

An agent-first CLI for querying and managing LangSmith resources.

Built for AI coding agents (deepagents, Claude Code, Cursor, etc.) and developers who need fast, scriptable access to projects, traces, runs, datasets, evaluators, experiments, and threads.

Installation

Install script (recommended)

macOS / Linux:

curl -fsSL https://cli.langsmith.com/install.sh | sh

Windows (PowerShell):

irm https://cli.langsmith.com/install.ps1 | iex

Upgrade

langsmith self-update

GitHub releases

Download the latest binary for your platform from GitHub Releases.

Authentication

Set your API key as an environment variable:

export LANGSMITH_API_KEY="lsv2_pt_..."

Optionally set defaults:

export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"  # For self-hosted
export LANGSMITH_WORKSPACE_ID="<workspace-id>"                # Default workspace
export LANGSMITH_PROJECT="my-default-project"                 # Default project for queries

Or save the credentials in a profile:

langsmith auth login                    # OAuth
langsmith auth signup                   # Create an account in the browser, then log in to it
langsmith profiles create prod           # API key, taken from $LANGSMITH_API_KEY

Profiles live in ~/.langsmith/config.json, written owner-only; the CLI warns if it is readable by other users.

To keep an API key out of that file, save where it lives instead. The variable is read, or the command run, each time the profile is used:

langsmith profiles create prod --api-key-env PROD_LANGSMITH_API_KEY
langsmith profiles create prod --api-key-cmd 'op read "op://Private/LangSmith/credential"'

langsmith auth info shows where the active profile's key comes from.

Other settings can be passed as flags:

langsmith --workspace <workspace-id> trace list --project my-app

Quick Start

# List tracing projects
langsmith projects list

# List recent traces in a project
langsmith traces list --project my-app --limit 5

# Get a specific trace with full detail
langsmith traces get <trace-id> --project my-app --full

# List LLM calls with token counts
langsmith runs list --project my-app --run-type llm --include-metadata

# List datasets
langsmith datasets list

# List experiments for a dataset
langsmith experiments list --dataset my-eval-set

Product feedback

Submit feedback about the CLI directly from the terminal:

langsmith report "Trace output is hard to scan"
langsmith report --category bug "The command fails on Windows"

--category accepts bug, feature-request, usability (the default), documentation, or other. The feedback payload contains the note, category, CLI version, operating system, architecture, and a fixed CLI source value. It does not collect command arguments, output, environment variables, file paths, traces, prompts, or other local context. Authentication, routing, idempotency, and standard client headers accompany the request. The collector is available on LangSmith Cloud and BYOC, not standalone self-hosted deployments.

Output Formats

langsmith traces list --project my-app

langsmith --format=json trace list --project my-app

langsmith traces list --project my-app -o traces.json

Selecting a project

Every command that operates on a project takes either --project <name> or --project-id <session UUID>, and $LANGSMITH_PROJECT supplies the name when neither is set. The two flags are mutually exclusive.

langsmith traces list --project 'my-app'
langsmith traces list --project-id 519bb9dd-079b-4488-8610-e330951ea3e4

Prefer --project-id when a program is building the command line. Project names are free-form — users can create one containing spaces, quotes, or shell metacharacters — so a name has to be quoted correctly at every call site, and a name that is quoted wrongly matches nothing and returns an empty result rather than an error. A UUID needs no quoting. --project-id also skips the name lookup, saving a round-trip.

langsmith projects list returns the id to use.

Command Reference

project — List and delete tracing projects

A tracing project (session) is a namespace that groups related traces together. This lists only tracing projects, not experiments — use experiment list for those.

Results are paginated — by default, only the first 20 projects are returned (use --limit to change). Projects are sorted by most recent activity (last_run_start_time, descending).

# List tracing projects (default: 20 results, most recently active first)
langsmith projects list
langsmith projects list --limit 50

# Filter by name
langsmith projects list --name-contains chatbot

# Machine-readable JSON
langsmith --format=json project list

# Permanently delete a project and all of its traces (requires confirmation)
langsmith projects delete --project-id 519bb9dd-079b-4488-8610-e330951ea3e4

trace — Query and export traces

A trace is a tree of runs representing one end-to-end invocation of your application.

Results are paginated — by default, only the first 20 traces are returned (use --limit to change). Traces are sorted newest-first by start time. By default, only traces from the last 7 days are returned; use --since or --last-n-minutes to change the time window.

# List recent traces (default: 20 results, newest first)
langsmith traces list --project my-app
langsmith traces list --project my-app --limit 50 --last-n-minutes 60

# Filter traces
langsmith traces list --project my-app --error           # Only errors
langsmith traces list --project my-app --min-latency 5   # Slow traces (>5s)
langsmith traces list --project my-app --tags production  # By tag
langsmith traces list --project my-app --name "agent"     # By name

# Include additional fields
langsmith traces list --project my-app --include-metadata   # + status, duration, tokens, costs
langsmith traces list --project my-app --include-io         # + inputs, outputs, error
langsmith traces list --project my-app --include-feedback   # + feedback_stats
langsmith traces list --project my-app --full               # All fields (metadata + io + feedback)

# Show trace hierarchy (fetches full run tree for each trace)
langsmith traces list --project my-app --show-hierarchy --limit 3

# Get a specific trace
langsmith traces get <trace-id> --project my-app --full

# Export traces to JSONL files (one per trace)
langsmith traces export ./traces --project my-app --limit 20 --full

# Custom filename pattern (supports {trace_id} and {name} placeholders)
langsmith traces export ./traces --project my-app --filename-pattern "{name}_{trace_id}.jsonl"

run — Query individual runs

A run is a single step within a trace (LLM call, tool call, chain step, etc.).

Results are paginated — by default, only the first 50 runs are returned (use --limit to change). Runs are sorted newest-first by start time. By default, only runs from the last 7 days are returned; use --since or --last-n-minutes to change the time window.

# List LLM calls (default: 50 results, oldest first)
langsmith runs list --project my-app --run-type llm
langsmith runs list --project my-app --run-type tool --name search

# --name is an exact match; use --filter for a substring match
langsmith runs list --project my-app --filter 'like(name, "%search%")'

# Find expensive calls
langsmith runs list --project my-app --run-type llm --min-tokens 1000 --include-metadata

# Include feedback scores
langsmith runs list --project my-app --include-feedback

# Get a specific run
langsmith runs get <run-id> --full

# Export to JSONL (default limit: 100)
langsmith runs export llm_calls.jsonl --project my-app --run-type llm --full

Query backend: the CLI selects the runs query API automatically from the deployment reported by /info — LangSmith Cloud and self-hosted >= 0.16 use the v2 (SmithDB) API; older self-hosted uses v1. No flag is needed. On v2, trace list and trace export select traces through /v2/traces/query, and trace get, trace export and --show-hierarchy read each trace through /v2/traces/{trace_id}/runs, which requires a project. On v1, trace get can omit --project / --project-id and look the trace up by ID in the current workspace. A few v2-only features (trace messages, thread messages) are unavailable on self-hosted < 0.16.

thread — Query conversation threads

A thread groups multiple root runs sharing a thread_id (multi-turn conversations).

Results are paginated — by default, only the first 20 threads are returned (use --limit to change). Threads are sorted by most recent activity (newest first).

# List threads (default: 20 results, newest first; requires --project)
langsmith threads list --project my-chatbot
langsmith threads list --project my-chatbot --last-n-minutes 120

# Get all turns in a thread
langsmith threads get <thread-id> --project my-chatbot --full

streaming — Search runs, threads and traces with AI search

--ai-search takes a natural-language query and only works on the streaming commands. Each SSE event prints as a JSON patch, and --max-items counts events. These commands take --project-id only.

langsmith threads streaming query --project-id <project-id> --ai-search "user asks for a refund"
langsmith traces streaming query --project-id <project-id> --ai-search "tool call failed"
langsmith runs streaming query-v2 --project-id <project-id> --ai-search "slow LLM calls" --max-items 20

insights — Create and query insight reports

Insight reports analyze traces in a project to identify usage patterns, common behaviors, and failure modes. Creating a report is asynchronous and may incur model costs. Define the report with a manual-mode JSON configuration:

{
  "name": "Weekly reliability review",
  "summary_prompt": "Identify the request and outcome: {{run.inputs}} {{run.outputs}}",
  "last_n_hours": 168,
  "filter": "eq(is_root, true)",
  "model": "openai",
  "schedule_cron": "0 9 * * 1"
}

Create the report asynchronously, or wait for it to finish. If schedule_cron is present, the configuration also starts future jobs on that schedule; the initial job still starts immediately:

langsmith insights create --project my-app --config insights.json
langsmith insights create --project my-app --config insights.json --wait

# List, update, or delete saved job configurations
langsmith insights config list --project my-app
langsmith insights config update <config-id> --project my-app --config insights.json
langsmith insights config delete <config-id> --project my-app --yes

# Run an existing configuration now, without changing its schedule
langsmith insights run <config-id> --project my-app

# List reports and fetch a completed result
langsmith insights list --project my-app
langsmith insights get <insight-id> --project my-app

The summary prompt is a Mustache template and must reference at least one of {{run.inputs}}, {{run.outputs}}, {{run.error}}, {{run.feedback}}, or {{all_thread_messages}}. Auto mode and user_context are not supported; express the report's intent directly in summary_prompt. Use --config - to read the JSON from stdin. The configuration also supports start_time, end_time, sample, hierarchy, partitions, attribute_schemas, cluster_model, summary_model, description, and schedule_cron. Set cluster_model and summary_model together to openai, anthropic, or workspace model-configuration UUIDs; model display names are not accepted. When updating, the file replaces the saved name and analysis configuration. Omit description or schedule_cron to preserve its current value; use an empty description to clear its contents or set schedule_cron to null to disable the schedule.

dataset — Manage evaluation datasets

List results are paginated — by default, only the first 100 datasets are returned (use --limit to change).

# List datasets (default: 100 results)
langsmith datasets list
langsmith datasets list --name-contains eval

# Get dataset details
langsmith datasets get my-dataset

# Create and delete
langsmith datasets create --name my-eval-set --description "QA pairs for v2"
langsmith datasets delete my-old-dataset --yes

# Export examples to JSON
langsmith datasets export my-dataset ./data.json --limit 500

# Upload from JSON file
langsmith datasets upload data.json --name new-dataset

example — Manage dataset examples

List results are paginated — by default, only the first 20 examples are returned (use --limit to change). Use --offset to paginate through results.

# List examples (default: 20 results)
langsmith examples list --dataset my-dataset
langsmith examples list --dataset my-dataset --split test --limit 50

# Paginate through examples
langsmith examples list --dataset my-dataset --limit 20 --offset 20

# Create an example
langsmith examples create --dataset my-dataset \
  --inputs '{"question": "What is LangSmith?"}' \
  --outputs '{"answer": "A platform for LLM observability"}'

# Create with metadata and split assignment
langsmith examples create --dataset my-dataset \
  --inputs '{"question": "What is tracing?"}' \
  --outputs '{"answer": "Recording LLM application execution"}' \
  --metadata '{"source": "manual", "version": 2}' \
  --split test

# Delete an example
langsmith examples delete <example-id> --yes

evaluator — Manage evaluators and their rules

An evaluator is the shared scorer shown on the Evaluators page. A rule attaches an evaluator to one project or dataset with its own sampling rate and filters. A rule's name can differ from its evaluator's name, so rule output shows both rule_name and evaluator_name.

# Evaluators: list, get by name (as shown in the UI) or ID, delete by ID
langsmith evaluator list
langsmith evaluator get ready_for_task_grade
langsmith evaluator delete <evaluator-id> --delete-rules --yes

# Rules: list (optionally per project, dataset, or evaluator), get, delete one
langsmith evaluator rule list --project my-app
langsmith evaluator rule list --evaluator-id <evaluator-id>
langsmith evaluator rule get <rule-id>
langsmith evaluator rule delete accuracy --dataset my-eval-set --yes

# Upload an offline evaluator (for experiments)
langsmith evaluator upload evals.py \
  --name accuracy --function check_accuracy --dataset my-eval-set

# Upload an online evaluator (for production monitoring)
langsmith evaluator upload evals.py \
  --name latency-check --function check_latency --project my-app

# Set sampling rate (evaluate a fraction of runs, 0.0-1.0)
langsmith evaluator upload evals.py \
  --name latency-check --function check_latency --project my-app --sampling-rate 0.5

# Replace an existing evaluator's code. The rule keeps its sampling rate, filters,
# webhooks, and other actions unless you pass --sampling-rate or --trace-filter.
langsmith evaluator upload evals.py \
  --name accuracy --function check_accuracy_v2 --dataset my-eval-set --replace --yes

# Create an LLM-as-judge evaluator (--model-config is always required)
# model.json: copy the structured.model block from an existing evaluator or the UI.
langsmith evaluator create-llm \
  --name relevance --project my-app \
  --prompt prompt.json --schema schema.json --model-config model.json \
  --variable-mapping '{"input":"input.question","output":"output.answer"}'

# Or reference an existing Prompt Hub commit (--hub-ref replaces --prompt and --schema)
langsmith evaluator create-llm \
  --name relevance --project my-app \
  --hub-ref my-org/relevance:latest --model-config model.json

experiment — Query experiment results

List results are paginated — by default, only the first 20 experiments are returned (use --limit to change).

# List experiments (default: 20 results)
langsmith experiments list
langsmith experiments list --dataset my-eval-set

# Get experiment results (feedback stats, run stats)
langsmith experiments get my-experiment-2024-01-15

hub — Manage agent and skill repos on the LangSmith Hub

The hub stores versioned directories of files grouped into repos of type agent or skill. Each push creates a new commit; pull downloads a commit's files into a local directory. This is the CLI surface for the langsmith Python/JS SDK's hub methods (pull_skill, push_skill, pull_agent, push_agent, etc.).

# Scaffold a starter skill (or agent)
langsmith hub init --type skill --dir ./my-skill --name my-skill

# Push a local directory as a new commit (creates the repo if missing)
langsmith hub push my-skill --type skill --dir ./my-skill

# Pull a commit (latest by default; pin a tag with :ref)
langsmith hub pull my-skill --dir ./out
langsmith hub pull acme/my-skill:production --dir ./out

# Discover, inspect, delete
langsmith hub list --type skill --query foo
langsmith hub list --type skill --source external
langsmith hub get acme/my-skill
langsmith hub delete acme/my-skill --yes

Identifiers use [OWNER/]REPO format. Omitting owner defaults to - (the API's "current tenant" wildcard).

Push excludes .git/, node_modules/, __pycache__/, .venv/, dist/, build/, target/, .next/, .cache/, plus .env* files, common secret extensions (.pem, .key, .pfx, .p12, .crt), and rejects binary or oversize (>1 MiB) files. Pull wipes the destination dir before writing; non-empty directories without a SKILL.md/AGENTS.md marker require --yes.

tunnel and relay — Serve private network connections

A private network connection lets LangSmith reach a network it can't reach directly, through a connector running inside that network. These commands run the connector until interrupted. Create the connection in LangSmith first and pass its ID.

# Tunnel: LangSmith reaches the named targets through this machine.
# An http(s):// target also gets a tunnel URL; host:port is raw TCP.
langsmith tunnel connect <connection-id> --target app=http://localhost:8080 --target db=10.0.0.5:5432

# Relay: sandbox traffic matching a relay rule leaves through this machine
langsmith relay connect <relay-id> --allow 10.0.0.0/8 --deny 10.0.9.0/24

A relay reaches public addresses on ports 80 and 443 by default. --allow widens that and --deny narrows it; each rule is a CIDR, an IP address, a hostname, or a *. wildcard, optionally followed by :PORT. Cloud metadata addresses are always refused.

The connector authenticates with an API key (LANGSMITH_API_KEY or a profile with an API key); OAuth logins are not accepted.

For a long-running deployment, langsmith connector run takes the same settings from environment variables (LANGSMITH_CONNECTION_ID, LANGSMITH_CONNECTOR_TARGETS, LANGSMITH_CONNECTOR_RELAY_MODE, LANGSMITH_API_KEY or LANGSMITH_API_KEY_FILE, and the rest listed in langsmith connector run --help) or a YAML file, and logs JSON. The langchain/langsmith-connector container image runs it.

self-update — Update langsmith to the latest version

# Check if an update is available
langsmith self-update --dry-run

# Update to the latest version
langsmith self-update

If langsmith was installed through a package manager, self-update won't replace the binary in place — it points you at the right command instead:

Installed via	Update with
Homebrew	brew upgrade langchain-ai/tap/langsmith-cli
Scoop	scoop update langsmith-cli
go install	go install github.com/langchain-ai/langsmith-cli/cmd/langsmith@latest

Installs from the install.sh/install.ps1 scripts or a direct GitHub Releases download are updated in place as usual. Pass --force to update in place regardless of how langsmith was installed.

trace setup — Trace coding agents to LangSmith

Configure Claude Code or Codex to send full-content traces (prompts, responses, tool calls) to a LangSmith project, by writing the agent's local config files. Requires an API key — it is written to the agent config at 0600 (OAuth profiles are not supported here). Each command previews the exact changes and asks you to confirm (pass --yes to skip the prompt), then installs the plugin via the agent's own CLI.

# Bare: try both Claude Code and Codex (best-effort; an uninstalled agent just fails)
langsmith traces setup

# Configure Claude Code: API key, URL, and project as positional args (bare host gains https://)
langsmith traces setup claude demo-key dev.smith.com shared-claude

# Or take the key + URL from env/profile
langsmith traces setup claude

# Configure Codex (writes ~/.codex/config.toml + ~/.codex/langsmith.json)
langsmith traces setup codex

# Trace to a named project (default: "claude-code" / "codex", or $LANGSMITH_PROJECT)
langsmith traces setup claude --project my-agent

# Override the auto-detected name/email attached to every trace
langsmith traces setup claude --user "Jane Doe" --email jane@example.com

# Pass everything explicitly (self-hosted or a specific workspace key)
langsmith traces setup claude demo-key https://my-host/api/v1 my-team   # all positional

# Apply without the interactive confirmation prompt
langsmith traces setup claude --yes

# Write config only; skip running the plugin install
langsmith traces setup claude --no-install

# Write project-local config instead of user-global
langsmith traces setup claude --scope project    # ./.claude/settings.local.json
langsmith traces setup codex --scope project     # ./.codex/...

trace setup claude installs the plugin via claude plugin marketplace add + claude plugin install; trace setup codex fetches it via codex plugin marketplace add. Once enabled, the plugin runs on every session and sends your prompts, responses, and tool output to LangSmith. Your name and email (auto-detected from git config user.name/user.email, or set via --user/--email) are attached to every trace as user_name/user_email metadata. Verify Claude Code with tail -f ~/.claude/state/hook.log.

Filter Options

Most trace and run commands share these filter options:

Flag	Description	Example
--project	Project name	--project my-app
--limit, -n	Max results	-n 10
--last-n-minutes	Time window (overrides 7-day default)	--last-n-minutes 60
--since	After ISO timestamp (overrides 7-day default)	--since 2024-01-15T00:00:00Z
--error / --no-error	Error status	--error
--name	Filter by run name (exact match)	--name ChatOpenAI
--run-type	Run type (run commands only)	--run-type llm
--min-latency	Min latency (seconds)	--min-latency 2.5
--max-latency	Max latency (seconds)	--max-latency 10
--min-tokens	Min total tokens; on trace commands, the whole trace's total	--min-tokens 1000
--tags	Tags (comma-separated, OR logic)	--tags prod,v2
--filter	Raw LangSmith filter DSL	--filter 'eq(status, "error")'
--trace-ids	Specific trace IDs	--trace-ids abc123,def456

Local Development

For local dev, create a wrapper script at ~/.local/bin/langsmith that loads your .env and uses go run:

cat > ~/.local/bin/langsmith << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
cd /path/to/langsmith-cli
set -a && source .env && set +a
exec go run ./cmd/langsmith "$@"
EOF
chmod +x ~/.local/bin/langsmith

Ensure ~/.local/bin is in your PATH before ~/go/bin. This way commands like langsmith sandboxes list and SSH ProxyCommand entries work without manually sourcing .env each time.

Requirements

• Go 1.23+
• golangci-lint (for linting)

Releasing

Releases work as in the langsmith SDK repos. release-please.yml keeps a release PR open on langsmith-cli with the next version and its changelog, built from the conventional commit titles of the changes Copybara exports (test and ci commits are left out). Merging that PR tags the version, creates the GitHub Release, and dispatches publish-binaries.yml: GoReleaser cross-compiles linux/darwin/windows on amd64+arm64 and attaches the archives and checksums.txt, and the Homebrew tap gets a bump PR.

A release must use a released langsmith-go, so merge a release PR only while go.mod requires a released tag. CI in langchainplus, where this module is developed, fails a change whose go.mod requires an unreleased langsmith-go (a pseudo-version or pre-release) or has a replace. The release is the release PR's merge commit, and the workflow refuses to release a commit whose go.mod uses a pseudo-version or a replace. If one is merged anyway, the workflow fails with the way out: release langsmith-go and update go.mod, then revert the release PR and remove its autorelease: pending label, and release-please opens a new release PR.

There is no version file to edit: the version is stamped into the binary from the tag via ldflags. The install scripts and langsmith self-update both read the latest GitHub Release.

License

MIT