Observability

Saf3AI’s observability features give you complete visibility into how your AI agents behave, perform, and interact with your systems.

Overview

Modern AI agents are complex systems with multiple components—LLM calls, tool usage, retrieval operations, and more. Understanding what’s happening across all these components is critical for debugging, optimization, and reliability.

Distributed Tracing

What is Tracing?

A trace captures the full journey of an AI agent’s work—from the initial request to the final response, including every LLM call, tool execution, and decision point.

with saf3.trace("customer-support-agent") as trace:
    # All operations here are captured
    response = agent.run("Help the customer")

Trace Structure

Traces contain:

• Spans: Individual operations (LLM calls, tool usage)
• Events: Discrete occurrences (user actions, errors)
• Attributes: Metadata (user ID, session, environment)
• Logs: Debug information

Viewing Traces

The Saf3AI dashboard provides:

• Timeline view: See operations in chronological order
• Flamegraph: Visualize operation hierarchy and duration
• Waterfall: Understand sequential and parallel execution

Metrics

Built-in Metrics

We automatically collect:

Metric	Description
llm.calls	Total LLM API calls
llm.tokens.input	Input tokens consumed
llm.tokens.output	Output tokens generated
llm.latency	Response time (p50, p95, p99)
llm.errors	Failed API calls
tools.calls	Tool/function invocations
tools.latency	Tool execution time
agent.sessions	Active agent sessions

Custom Metrics

Add your own metrics:

trace.record_metric("relevance_score", 0.92)
trace.record_metric("user_satisfaction", 5)

Dashboards

Create custom dashboards to monitor:

• Agent performance over time
• Error rates by model/provider
• Cost trends
• User engagement metrics

Logging

Structured Logs

All logs are structured for easy querying:

trace.log("processing_started", {
    "document_count": 10,
    "user_tier": "premium"
})

Log Levels

Supported levels:

• DEBUG: Detailed debugging information
• INFO: General information
• WARN: Potential issues
• ERROR: Errors that need attention

Log Search

Search logs by:

• Time range
• Log level
• Trace ID
• Custom attributes
• Full-text content

Alerting

Alert Conditions

Create alerts for:

saf3.alerts.create(
    name="high-latency",
    condition="llm.latency.p95 > 5000",  # >5s
    threshold_duration="5m",
    channels=["slack", "email"]
)

Alert Examples

• Error spike: llm.errors.rate > 5%
• Cost anomaly: daily.cost > budget * 1.2
• Latency degradation: llm.latency.p95 > baseline * 2
• Security events: security.events.critical > 0

Notification Channels

Send alerts to:

• Slack
• PagerDuty
• Email
• Webhook
• Microsoft Teams

Debugging

Trace Replay

Replay any trace to understand exactly what happened:

1. Find the trace by ID or search
2. See every LLM call with full prompts/responses
3. Inspect tool inputs and outputs
4. View timing for each operation

Error Analysis

When errors occur:

• Full stack traces
• Request/response context
• Related logs and events
• Suggested fixes

Comparison Mode

Compare two traces side-by-side to understand:

• Why one succeeded and another failed
• Performance differences
• Behavioral changes between versions

Integration with Existing Tools

OpenTelemetry

Native OTel export:

saf3 = Saf3AI(
    api_key="your-key",
    otlp_endpoint="http://otel-collector:4317"
)

Datadog

Export to Datadog:

saf3 = Saf3AI(
    api_key="your-key",
    datadog={
        "api_key": "dd-api-key",
        "site": "datadoghq.com"
    }
)

Prometheus

Expose metrics endpoint:

# Metrics available at :8080/metrics
saf3 = Saf3AI(
    api_key="your-key",
    prometheus_port=8080
)

Best Practices

1. Use Meaningful Trace Names

# Good
with saf3.trace("customer-support-ticket-123") as trace:
    pass

# Less useful
with saf3.trace("agent") as trace:
    pass

2. Add Context

with saf3.trace(
    "agent",
    user_id="user-123",
    session_id="session-456",
    environment="production",
    version="1.2.0"
) as trace:
    pass

3. Log Important Decisions

trace.log("routing_decision", {
    "input_type": "technical",
    "routed_to": "engineering-agent",
    "confidence": 0.95
})

4. Track Custom Metrics

trace.record_metric("retrieval_count", len(documents))
trace.record_metric("response_quality", quality_score)

Dashboard Features

Trace Explorer

• Search by time, user, session, or custom attributes
• Filter by status (success/error)
• Sort by duration, cost, or time

Service Map

Visualize agent architecture:

• Component dependencies
• Call patterns
• Error propagation

Analytics

• Usage trends
• Performance benchmarks
• Cost analysis
• User behavior patterns

Data Retention

Plan	Retention
Free	7 days
Pro	30 days
Enterprise	Custom (up to unlimited)

Next Steps

• Security Features
• FinOps & Cost Management
• SDK Reference