🔭 Telemetry

General

• Axiom.co is used as the telemetry platform

Backend telemetry

• OpenTelemetry is used log instrumentation
• Traces and Logs are currently enabled
• Traces and logs are sent to Axiom only in production and are sent to separate datasets
  • Traces - traces
  • Logs - logs
• In local development logs are sent to console while traces are disabled

Traces

• Traces are instrumented using the FastAPI automatic instrumentation
• This means that traces are created automatically for all requests to the API
• Adding additional info to traces:
  • Adding a new Span - A span represents a unit of work or operation. Spans are the building blocks of Traces

    with tracer.start_as_current_span("span name"):
        // Span will include eveyrthing in this context
  • Attributes - key value pairs added to the Span

    current_span = trace.get_current_span()
    current_span.set_attribute("custom_attribute", "custom message")
  • Events - a structured log message on a Span

    current_span = trace.get_current_span()
    current_span.add_event("Running opertation")
  • Attributes vs Events: generaly if the timestamp in which something happened is important, attach the data to a span event, otherwise use an attribute.

Logs

• The OTel support for python logs is still under development
• The logs are instrumented using the standard python logging instrumention
• Verbosity: info and above logs are enabled
• To add a new log:

  # Typically the logger is available in the global context and you don't need to retrieve it
  logger.info("info log")

Langfuse

• Langfuse is used for LLM observability (traces for AI/LLM calls, separate from OTEL)
• All LLM calls made via LiteLLM are automatically traced - no manual instrumentation needed
• Traces include token usage, model, input/output, and metadata like session_id, user_id, investigation_id
• To trace a function explicitly, use the @langfuse.observe() decorator:

  from server.telemetry.langfuse.langfuse_setup import get_langfuse_client
  langfuse = get_langfuse_client()
  
  @langfuse.observe(name="my_operation")
  def my_function(input):
      ...
• To attach metadata to the current trace:

  from server.telemetry.langfuse.langfuse_utils import update_current_langfuse_trace
  update_current_langfuse_trace(session_id="...", user_id="...", investigation_id="...")
• Certain OTEL instrumentation scopes (e.g. FastAPI requests, auth, background tasks) are blocked from Langfuse to reduce noise — configure in server/telemetry/langfuse/langfuse_setup.py

Frontend telemetry

Webapp

• PostHog is used for user analytics (page views, custom events)
• Sentry is used for error tracking — unhandled exceptions are captured automatically
• Page views are tracked automatically via PostHogPageView; users are identified with userId and orgId
• To capture a custom event:

  import { usePostHog } from "posthog-js/react";
  
  const posthog = usePostHog();
  posthog.capture("event_name", { property: "value" });
• Logs are sent to Axiom via the useLogger hook (info/warn/error):

  import { useLogger } from "@/lib/hooks/use-logger";
  
  const logger = useLogger();
  logger.info("something happened", { key: "value" });

Extension

• OpenTelemetry is used for both logs and traces — initialized in packages/extension/src/instrumentation.tsx
• Telemetry is batched and sent to the backend at /telemetry
• To add a log or trace:

  import { logger, tracer } from "@/instrumentation";
  
  logger.info("event description", { key: "value" });
  
  const span = tracer.startSpan("operation_name");
  // do work
  span.end();