Observability

How telemetry is produced in code. This is the implementation reference for logging, traces, metrics, and errors; see ADR-012 for why the stack was chosen.

Applications only ever speak OTLP to the Collector. Backend, sampling, and redaction are the Collector's job — never hard-code an exporter in app code.

Where it lives

File	Role
api/src/telemetry/otel.ts	API NodeSDK bootstrap — imported first in main.ts
api/src/app/otel-exception.filter.ts	Records unhandled exceptions on the active span (APP_FILTER)
api/src/app/app.module.ts	nestjs-pino LoggerModule (transport + redaction)
web/src/instrumentation.ts	Web registerOTel() (@vercel/otel)
infra/otel/collector-config.yaml	Receivers, redaction, otlp_grpc export to LGTM

Config is entirely OTEL_* env (see api/.env, web/.env.local). No exporter code.

What you get for free

Auto-instrumentation covers the common paths — do not hand-instrument these:

Captured automatically	By
Incoming/outgoing HTTP, spans + status	auto-instrumentations-node, @vercel/otel
Postgres (pg) and Redis calls	auto-instrumentations-node
trace_id / span_id injected into every pino log	instrumentation-pino
BFF → API trace-context propagation	@vercel/otel fetch patch
Unhandled exceptions recorded on the span	OtelExceptionFilter

The trace_id is the correlation ID — logs, traces, and metrics join on it in Grafana. You never set it manually.

Logging

Inject PinoLogger, set context once, log an object first, message second.

import { PinoLogger } from 'nestjs-pino';

@Injectable()
export class AdmissionService {
  constructor(private readonly logger: PinoLogger) {
    this.logger.setContext(AdmissionService.name);
  }

  approve(id: string) {
    this.logger.info({ admissionId: id }, 'admission approved');
    this.logger.warn({ admissionId: id }, 'approved past cutoff date');
  }
}

Levels: error (needs attention) · warn (recoverable/degraded) · info (business event) · debug (dev only). Structured fields go in the object, not the string — never string-concatenate values into the message.

Errors and warnings

Unhandled exceptions are recorded automatically. For a handled error you want on the trace, record it and mark the span failed:

import { trace, SpanStatusCode } from '@opentelemetry/api';

const span = trace.getActiveSpan();
span?.recordException(err);
span?.setStatus({ code: SpanStatusCode.ERROR });
this.logger.error({ err }, 'payment gateway rejected charge');

An expected 4xx (validation, permission denied) is not an error — log at warn/info, do not fail the span.

Custom spans

Wrap a meaningful unit of work. Always end() in finally.

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('chilarai-api');

await tracer.startActiveSpan('admission.approve', async (span) => {
  try {
    span.setAttribute('admission.id', id);
    span.setAttribute('school.id', schoolId);
    await doWork();
  } catch (err) {
    span.recordException(err as Error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
});

Name spans noun.verb (admission.approve, fees.settle). Put IDs on attributes, never in the span name.

Custom metrics

Create the instrument once (module scope), record on each occurrence. Use a counter for tallies, a histogram for durations/sizes.

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('chilarai-api');
const admissionsApproved = meter.createCounter('admissions.approved', {
  description: 'Approved admissions',
});

admissionsApproved.add(1, { 'school.id': schoolId });

Keep attribute values low-cardinality (school.id, role — never a userId, email, or free text; each combination is a stored time series).

Custom events: metric or log?

The event is…	Emit as	Example
Aggregatable / counted	metric	admissions.approved, logins.total
Discrete / auditable	structured log (carries trace_id)	tenant switch, role granted

Prefer a metric when you'll chart or alert on a rate; prefer a log when you need the individual record and its context.

Secrets and PII

Redacted at two layers — do not rely on only one:

• App: pino redact drops authorization, cookie, set-cookie.
• Collector: attributes/redact strips the same headers from spans.

Never log tokens, passwords, full emails, or raw request bodies. Never put PII on span attributes or metric labels. x-tenant-code is not a secret and is kept.

Web

web/src/instrumentation.ts calls registerOTel() on server startup — Next server routes and the BFF's fetch to the API are traced and propagate context automatically. Browser (RUM) telemetry is Phase 2.

Viewing locally

pnpm infra:up, then Grafana at http://localhost:4000 → Explore: Tempo (traces: web → API → Postgres), Loki (logs, filter by trace_id), Mimir (metrics). See the local development runbook.