Skip to main content

🔭 Telemetry

General​

  • Axiom.co is used as the telemetry platform

Backend telemetry​

  • OpenTelemetry is used log instrumentation
  • Traces and Logs are currently enabled
  • Traces and logs are sent to Axiom only in production and are sent to separate datasets
    • Traces - traces
    • Logs - logs
  • In local development logs are sent to console while traces are disabled

Traces​

  • Traces are instrumented using the FastAPI automatic instrumentation
  • This means that traces are created automatically for all requests to the API
  • Adding additional info to traces:
    • Adding a new Span - A span represents a unit of work or operation. Spans are the building blocks of Traces 
      with tracer.start_as_current_span("span name"):
          // Span will include eveyrthing in this context
    • Attributes - key value pairs added to the Span 
      current_span = trace.get_current_span()
      current_span.set_attribute("custom_attribute", "custom message")
    • Events - a structured log message on a Span 
      current_span = trace.get_current_span()
      current_span.add_event("Running opertation")
    • Attributes vs Events: generaly if the timestamp in which something happened is important, attach the data to a span event, otherwise use an attribute.

Logs​

  • The OTel support for python logs is still under development
  • The logs are instrumented using the standard python logging instrumention
  • Verbosity: info and above logs are enabled
  • To add a new log: 
    # Typically the logger is available in the global context and you don't need to retrieve it
    logger.info("info log")

Frontend telemetry​

TBD