Skip to main content

Observability & tracing

A full setup example is available below.

Overview​

The Faheem Code SDK provides built-in OpenTelemetry (OTEL) tracing support, allowing you to monitor and debug your agent's execution in real time. You can send traces to any OTLP-compatible observability platform including:

  • Laminar - AI-focused observability with trace inspection, signals, and browser session replay
  • MLflow - Open-source AI platform with tracing, evaluation, and LLM governance
  • Honeycomb - High-performance distributed tracing
  • Any OTLP-compatible backend - Including Jaeger, Datadog, New Relic, and more

The SDK automatically traces:

  • Agent execution steps
  • Tool calls and executions
  • LLM API calls (via LiteLLM integration)
  • Browser automation sessions (when using browser-use)
  • Conversation lifecycle events

Quick start​

Tracing is automatically enabled when you set the appropriate environment variables. The SDK detects the configuration on startup and initializes tracing without requiring code changes.

Using Laminar​

Laminar provides specialized AI observability features for Faheem Code, including full conversation traces, browser session replay, and higher-level analysis features like signals.

# Set your Laminar project API key
export LMNR_PROJECT_API_KEY="your-laminar-api-key"

That's it. Run your agent code normally and traces will be sent to Laminar automatically.

For self-hosted Laminar deployments, configure the instance base URL and ports:

export LMNR_PROJECT_API_KEY="your-laminar-api-key"
export LMNR_BASE_URL=http://localhost
export LMNR_HTTP_PORT=8000
export LMNR_GRPC_PORT=8001

If you need help deciding between Laminar Cloud and self-hosted Laminar, see Laminar's official hosting options.

Why use Laminar with Faheem Code?​

Laminar is especially useful when you want to understand how an agent behaved across one run or across many runs:

  • Inspect a single run in transcript, tree, or timeline views to see prompts, tool calls, outputs, and nested agent activity. See Laminar's guide to viewing traces.
  • Watch browser automation alongside trace spans with session replay for browser agents.
  • Define signals to classify failures, user friction, or success patterns across many traces.
  • Keep each Faheem Code conversation grouped under a single session ID so multi-turn debugging is easier.

Using OpenTelemetry (OTLP) backends​

For OpenTelemetry (OTLP) compatible backends, set the following environment variables:

# Required: Set the OTLP endpoint
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-otlp-backend/v1/traces"

# Required: Set additional headers required by your backend (format: comma-separated key=value pairs, URL-encoded)
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="key=value,key2=value2"

# Recommended: Explicitly set the protocol (most OTLP backends require HTTP)
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"  # use "grpc" only if your backend supports it

View the platform-specific configuration sections below for which values to use.

Alternative configuration methods​

You can also use these alternative environment variable formats:

# Short form for endpoint
export OTEL_ENDPOINT="http://localhost:4317"

# Alternative header format
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<KEY>"

# Alternative protocol specification
export OTEL_EXPORTER="otlp_http"  # or "otlp_grpc"

How it works​

The Faheem Code SDK uses Laminar as its OpenTelemetry instrumentation layer for built-in tracing support. When you set the environment variables, the SDK:

  1. Detects configuration: Checks for OTEL environment variables on startup
  2. Initializes tracing: Configures OpenTelemetry with the appropriate exporter
  3. Instruments code: Automatically wraps key functions with tracing decorators
  4. Captures context: Associates traces with conversation IDs for session grouping
  5. Exports spans: Sends trace data to your configured backend

For Laminar-specific behavior and examples, see the official Faheem Code SDK integration guide.

What gets traced​

The SDK automatically instruments these components:

  • agent.step - Each iteration of the agent's execution loop
  • Tool executions - Individual tool calls with input/output capture
  • LLM calls - API requests to language models via LiteLLM
  • Conversation lifecycle - Message sending, conversation runs, and title generation
  • Browser sessions - When using browser-use, captures session replays (Laminar only)

Trace hierarchy​

Traces are organized hierarchically:

  • conversation
    • conversation.run
      • agent.step
        • llm.completion
        • tool.execute
      • agent.step
        • llm.completion

Each conversation gets its own session ID (the conversation UUID), allowing you to group all traces from a single conversation together in your observability platform.

In tool.execute, the tool calls are traced individually, such as bash, file_editor, or task_tracker.

Configuration reference​

Environment variables​

The SDK checks for these environment variables (in order of precedence):

VariableDescriptionExample
LMNR_PROJECT_API_KEYLaminar project API keyyour-laminar-api-key
LMNR_BASE_URLBase URL for self-hosted Laminarhttp://localhost
LMNR_HTTP_PORTHTTP port for self-hosted Laminar8000
LMNR_GRPC_PORTgRPC port for self-hosted Laminar8001
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTFull OTLP traces endpoint URLhttps://api.honeycomb.io:443/v1/traces
OTEL_EXPORTER_OTLP_ENDPOINTBase OTLP endpoint (traces path appended)http://localhost:4317
OTEL_ENDPOINTShort form endpointhttp://localhost:4317
OTEL_EXPORTER_OTLP_TRACES_HEADERSAuthentication headers for tracesx-honeycomb-team=YOUR_API_KEY
OTEL_EXPORTER_OTLP_HEADERSGeneral authentication headersAuthorization=Bearer%20TOKEN
OTEL_EXPORTER_OTLP_TRACES_PROTOCOLProtocol for traces endpointhttp/protobuf, grpc
OTEL_EXPORTERShort form protocolotlp_http, otlp_grpc

Header format​

Headers should be comma-separated key=value pairs with URL encoding for special characters:

# Single header
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-honeycomb-team=abc123"

# Multiple headers
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20abc123,X-Custom-Header=value"

Protocol options​

The SDK supports both HTTP and gRPC protocols:

  • http/protobuf or otlp_http - HTTP with protobuf encoding (recommended for most backends)
  • grpc or otlp_grpc - gRPC with protobuf encoding (use only if your backend supports gRPC)

Platform-specific configuration​

Laminar setup​

  1. Sign up at laminar.sh
  2. Create a project and copy your API key
  3. Set the environment variable:
export LMNR_PROJECT_API_KEY="your-laminar-api-key"

Self-hosted Laminar: If you are running a self-hosted Laminar instance, configure its base URL and the HTTP and gRPC ports via environment variables:

export LMNR_PROJECT_API_KEY="your-laminar-api-key"
export LMNR_BASE_URL=http://localhost
export LMNR_HTTP_PORT=8000
export LMNR_GRPC_PORT=8001

Browser session replay: When using Laminar with browser-use tools, session replays are automatically captured, allowing you to see exactly what the browser automation did.

Faheem Code Enterprise setup​

If you are running Faheem Code Enterprise (FCE), you can use the same Laminar integration without changing application code:

  1. Complete the Faheem Code Enterprise quick start.
  2. Enable analytics in the Admin Console.
  3. Deploy FCE and wait for the analytics service to become ready.
  4. Open the Laminar UI at https://analytics.<your-base-domain>.
  5. Create a Laminar project and an ingest-only API key.
  6. Save that key as the Laminar Project API Key in the Admin Console.
  7. Redeploy, then start a conversation in Faheem Code.

In FCE, environment variables with LMNR_ and LLM_ prefixes are automatically forwarded to the SDK runtime. That makes it possible to configure Laminar endpoint settings such as LMNR_BASE_URL, LMNR_PROJECT_API_KEY, and LMNR_FORCE_HTTP, as well as the LLM that powers Laminar's own AI features (chat-with-trace, SQL-with-AI, and signals) via LLM_PROVIDER, LLM_BASE_URL, and LLM_MODEL_SMALL|MEDIUM|LARGE.

LLM_PROVIDER accepts gemini (Laminar's default), openai, or bedrock. Set it to openai whenever you point LLM_BASE_URL at an OpenAI-compatible gateway (for example LiteLLM, OpenRouter, or vLLM), not just the public OpenAI API. For the full list of supported values, see Laminar's official self-hosting configuration reference.

For the full FCE flow with screenshots and configuration examples, see Analytics in Faheem Code Enterprise.

MLflow setup​

MLflow is an open-source AI platform that accepts OpenTelemetry traces out of the box, alongside evaluation and LLM governance capabilities.

  1. Start your MLflow tracking server:
uvx mlflow server
  1. Configure the environment variables:
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5000"
export OTEL_EXPORTER_OTLP_HEADERS="x-mlflow-experiment-id=123"  # Replace "123" with your MLflow experiment ID
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

Navigate to the MLflow UI (for example, http://localhost:5000), select the experiment, and open the Traces tab to view the recorded traces.

Honeycomb setup​

  1. Sign up at honeycomb.io
  2. Get your API key from the account settings
  3. Configure the environment:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.honeycomb.io:443/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-honeycomb-team=YOUR_API_KEY"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

Jaeger setup​

For local development with Jaeger:

# Start Jaeger all-in-one container
docker run -d --name jaeger \
  -p 4317:4317 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

# Configure SDK
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"

Access the Jaeger UI at http://localhost:16686.

Generic OTLP collector​

For other backends, use their OTLP endpoint:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-otlp-collector:4317/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20YOUR_TOKEN"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

Advanced usage​

Disabling observability​

To disable tracing, simply unset all OTEL environment variables:

unset LMNR_PROJECT_API_KEY
unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
unset OTEL_EXPORTER_OTLP_ENDPOINT
unset OTEL_ENDPOINT

The SDK will automatically skip all tracing instrumentation with minimal overhead.

Custom span attributes​

The SDK automatically adds these attributes to spans:

  • conversation_id - UUID of the conversation
  • tool_name - Name of the tool being executed
  • action.kind - Type of action being performed
  • session_id - Groups all traces from one conversation

Debugging tracing issues​

If traces are not appearing in your observability platform:

  1. Verify environment variables:

    import os

    otel_endpoint = os.getenv('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT')
    otel_headers = os.getenv('OTEL_EXPORTER_OTLP_TRACES_HEADERS')

    print(f"OTEL Endpoint: {otel_endpoint}")
    print(f"OTEL Headers: {otel_headers}")

  2. Check SDK logs: The SDK logs observability initialization at debug level:

    import logging

    logging.basicConfig(level=logging.DEBUG)

  3. Test connectivity: Ensure your application can reach the OTLP endpoint:

    curl -v https://api.honeycomb.io:443/v1/traces

  4. Validate headers: Check that authentication headers are properly URL-encoded.

For Laminar-specific troubleshooting, see Laminar's official tracing troubleshooting guide.

Troubleshooting​

Traces not appearing​

Problem: No traces showing up in your observability platform.

Solutions:

  • Verify environment variables are set correctly
  • Check network connectivity to the OTLP endpoint
  • Ensure authentication headers are valid
  • Look for SDK initialization logs at debug level

High trace volume​

Problem: Too many spans being generated.

Solutions:

  • Configure sampling at the collector level
  • For Laminar with non-browser tools, browser instrumentation is automatically disabled
  • Use backend-specific filtering rules

Performance impact​

Problem: Concerned about tracing overhead.

Solutions:

  • Tracing has minimal overhead when properly configured
  • Disable tracing in development by unsetting environment variables
  • Use asynchronous exporters (default in most OTLP configurations)

Example: full setup​

"""
Observability & Laminar example

This example demonstrates enabling OpenTelemetry tracing with Laminar in the
Faheem Code SDK. Set LMNR_PROJECT_API_KEY and run the script to see traces.
"""

import os

from pydantic import SecretStr

from faheemcode.sdk import LLM, Agent, Conversation, Tool
from faheemcode.tools.terminal import TerminalTool

# Tip: Set LMNR_PROJECT_API_KEY in your environment before running, e.g.:
#   export LMNR_PROJECT_API_KEY="your-laminar-api-key"
# For non-Laminar OTLP backends, set OTEL_* variables instead.

# Configure LLM and Agent
api_key = os.getenv("LLM_API_KEY")
model = os.getenv("LLM_MODEL", "faheemcode/claude-sonnet-4-5-20250929")
base_url = os.getenv("LLM_BASE_URL")
llm = LLM(
    model=model,
    api_key=SecretStr(api_key) if api_key else None,
    base_url=base_url,
    usage_id="agent",
)

agent = Agent(
    llm=llm,
    tools=[Tool(name=TerminalTool.name)],
)

# Create conversation and run a simple task
conversation = Conversation(agent=agent, workspace=".")
conversation.send_message("List the files in the current directory and print them.")
conversation.run()
print(
    "All done! Check your Laminar dashboard for traces "
    "(session is the conversation UUID)."
)
export LMNR_PROJECT_API_KEY="your-laminar-api-key"
cd software-agent-sdk
uv run python examples/01_standalone_sdk/27_observability_laminar.py

Next steps​