Skip to main content

ADR-012: Use OpenTelemetry and a Self-Hostable LGTM Stack for Observability

Status​

Accepted

Date​

2026-08-26

Context​

Chilarai needs observability across its runtime: the NestJS API and the Next.js web app (including its Backend-For-Frontend proxy). The platform must capture:

  • Standard request telemetry
  • Distributed traces across web → BFF → API → database
  • Exceptions
  • Custom business events
  • Custom metrics

The platform targets both:

  • Multi-tenant SaaS deployments
  • Self-hosted school (on-premises) deployments

This mirrors the constraints in prior decisions. As with ADR-006 (self-hosted Keycloak over Azure AD B2C) and ADR-008 (MinIO over Azure Blob), the observability solution must not lock the platform into a single cloud, and must work for on-premises deployments where a cloud-managed backend is unavailable.

The solution should:

  • Be open source
  • Be vendor neutral at the instrumentation layer
  • Be self-hostable and cloud portable
  • Run easily in containers
  • Avoid vendor lock-in
  • Work for both SaaS and on-premises deployments
  • Support traces, metrics, and logs, correlated by trace context
  • Provide retention long enough for real debugging (target: at least 90 days for traces and exceptions)
  • Avoid unnecessary operational burden where a managed option is acceptable

Decision Drivers​

  • Instrumentation should be independent of any specific backend
  • The backend should be swappable per deployment without changing application code
  • Self-hosting must remain possible for on-premises deployments
  • Retention and cost must remain controllable
  • Secrets and PII must never be captured in telemetry

Decision​

Chilarai will use OpenTelemetry (OTel) for instrumentation and a self-hostable LGTM stack (Loki, Grafana, Tempo, Mimir) as the default observability backend.

  • Both applications are instrumented with OpenTelemetry for traces, metrics, and logs.
  • Applications export OTLP to an OpenTelemetry Collector. The Collector is the vendor-neutral boundary — analogous to the StorageProvider and IdentityProvider abstractions — and is the single point for batching, sampling, and redaction.
  • The default backend is a self-hostable LGTM stack: Tempo (traces), Loki (logs), Mimir (metrics), and Grafana (query, dashboards, alerting).
  • Local development uses the grafana/otel-lgtm all-in-one container plus a Collector.
  • Custom events are modelled as metrics when they are aggregatable (e.g. login counts) and as structured logs with trace context when they are discrete, auditable occurrences (e.g. tenant switch).
  • The trace ID is the correlation ID; it is injected into structured logs so logs, traces, and metrics correlate in Grafana.
  • Secrets and PII (Authorization headers, cookies, tokens, emails) are redacted at both the instrumentation and Collector layers.

Deployment position (current)​

The initial deployed footprint will self-host OTLP (Collector + LGTM containers). The choice of a managed backend is deferred: whether to continue self-hosting or adopt Grafana Cloud or Azure Monitor will be evaluated later. Because applications only ever speak OTLP to the Collector, this decision requires only a change to the Collector's exporter — no application changes.

Rationale​

Vendor Neutrality​

OpenTelemetry and OTLP are open standards. Application instrumentation does not depend on any backend. The backend can change without touching business code.

Self-Hostable and Cloud Portable​

The LGTM components are open source and container-friendly (Docker, Kubernetes, VMs). They run on any cloud or on-premises, consistent with the platform's self-hosting goals.

Works for SaaS and On-Premises​

A SaaS control plane can point the Collector at a managed or self-hosted backend. An on-premises school deployment runs the same containers locally. Both use identical instrumentation.

Retention and Cost Control​

Self-hosted LGTM stores cold data in object storage, making long retention cheap and configurable. This satisfies the retention requirement without the per-GB ingestion cost of a managed monitor. Managed options remain available where their convenience outweighs their cost.

Consistent with Existing Decisions​

This reuses the established pattern: an open, self-hostable default behind a provider abstraction, with managed alternatives available but not required.

Alternatives Considered​

Option 1: OpenTelemetry + Self-Hostable LGTM (Selected)​

Vendor-neutral instrumentation, self-hostable and portable backend, managed options available via the Collector. Requires operating the stack when self-hosted.

Option 2: Azure Monitor / Application Insights​

Fully managed, 90-day default retention for Application Insights, and Entra integration. Not selected as the default because it is Azure-locked and cannot serve on-premises deployments, and its per-GB ingestion cost is comparatively high. Retained as an optional managed OTLP exporter for Azure-hosted deployments.

Option 3: Grafana Cloud​

Managed LGTM running the same open-source software as the self-hosted default, with an OTLP endpoint. Not selected as the default only to avoid committing to a managed vendor prematurely. Retained as an optional managed backend, reachable without code changes because it is the same software.

Option 4: Proprietary SaaS (Datadog, New Relic)​

Rich features, but proprietary agents, strong vendor lock-in, and high cost at scale. Rejected as inconsistent with the platform's portability goals.

Option 5: AWS CloudWatch + X-Ray​

AWS-native, but a fragmented traces/logs/metrics experience and AWS lock-in. Rejected as a default for the same portability reasons.

Consequences​

Positive​

  • Open source and vendor neutral
  • Self-hostable and cloud portable
  • Works for both SaaS and on-premises deployments
  • Backend is swappable per deployment with no application changes
  • Traces, metrics, and logs correlate through trace context
  • Long, cost-controlled retention when self-hosted
  • Managed options (Grafana Cloud, Azure Monitor) remain available

Negative​

  • Self-hosting the stack in production is an operational responsibility
  • Running a Collector adds a component to each environment
  • Redaction and sampling must be configured and maintained
  • The managed-backend decision is deferred and must be revisited

Summary​

Chilarai instruments with OpenTelemetry and exports OTLP to a Collector, with a self-hostable LGTM stack as the default backend. This keeps the platform open, portable, and viable for both SaaS and on-premises deployments, while leaving the managed-backend choice (self-host, Grafana Cloud, or Azure Monitor) as a deferred, code-free decision.