Observability-First: Software That Explains Its Own Failures

By Sergey Nosov

16 October 2026

Suppose checkout starts failing at 3 a.m. and the page goes to you. You open the logs and find this:

starting
reserving
charging user u2 for 99999
error

Something failed for someone. The lines do not say which order failed, what the error was, or which service produced it. Nothing ties this error to its request, so on a busy service it is one line among thousands. The customer saw “something went wrong.” What follows is reconstruction: matching timestamps across services, guessing, or adding log statements and waiting for the failure to recur.

That is observability as an afterthought. I treat it instead as a design principle, which I call Observability-First:

Design software so that its internal behavior can be accurately understood from its external signals—logs, metrics, and traces—at any time and under any conditions.

Put more simply, software should be able to explain itself, especially when things go wrong. This article covers where the idea comes from and why the signals must share one context. Then it works through six habits that build the principle in, including a test that fails the build when promised telemetry goes missing. The examples are JavaScript, tested on Node.js 24.21 with the latest OpenTelemetry releases as of October 2026.

What “Observable” Means

The word comes from control theory. Rudolf Kalman introduced it in 1960, together with its dual, controllability; his 1985 Kyoto Prize citation credits him with both. A system is observable when its internal state can be worked out from its outputs. OpenTelemetry’s documentation carries the definition over to software almost unchanged: “Observability is the ability to understand the internal state of a system by examining its outputs.”

The word became common in software in the 2010s. In September 2013, Twitter’s engineering blog described the platform its Observability team ran, in a post mostly about metrics. Zipkin, Twitter’s tracing system, got one sentence: it “does not contribute to monitoring but is integral in many debugging situations.”

Charity Majors, a cofounder of Honeycomb, sharpened the meaning. In a 2021 retrospective she wrote: “You have observability if you can ask any question of your system from the outside, to understand any state the system has gotten into, no matter how bizarre or novel, without shipping any new custom code to get answers.” OpenTelemetry’s observability primer puts the same test more plainly: “An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need.”

Both definitions aim at the failure nobody predicted. A dashboard answers the questions someone thought to ask when building it. Observability also covers the rest, which the primer calls “unknown unknowns.”

The “first” in Observability-First is about timing. During an incident, the only signals you have are the ones the code already emits; nobody can instrument the past. A pilot flying through cloud relies on instruments installed when the airplane was built. When turbulence hits, it is too late to install sensors.

Three Signals, Three Questions

The usual starting point is three kinds of telemetry. In February 2017, at a distributed tracing summit, Peter Bourgon drew logging, metrics, and tracing as a Venn diagram and named one defining trait for each. In 2018, Cindy Sridharan’s short O’Reilly book Distributed Systems Observability gave them a chapter titled “The Three Pillars of Observability.” A fourth kind is on its way: OpenTelemetry’s profiles signal, a code-level view of resource use, entered public alpha in March 2026.

Logs: What Happened

Bourgon’s defining trait for logging is that “it deals with discrete events.” The useful kind is structured: one machine-readable record per event, with named fields an investigator can filter on. Here is the event the improved checkout below emits for the same failed order, pretty-printed and with its stack shortened:

{
  "time": "2026-10-16T15:42:05.847Z",
  "level": "error",
  "event": "order.failed",
  "service": "checkout",
  "trace_id": "b95bfa85c779cff382244df8b4f327e3",
  "span_id": "7c31068b87ab8206",
  "order_id": "ord-1002",
  "error_type": "PAYMENT_TIMEOUT",
  "error": {
    "message": "No answer in 30 ms",
    "stack": "OrderError: No answer in 30 ms\n    at …"
  }
}

Each field answers a question the four lines in the opening left open: which order (order_id), what failed (error_type), and where in the request’s trace to look next (trace_id and span_id). OpenTelemetry’s specification for trace context in other log formats gives those two field names, each value “lowercase and hex-encoded,” at the top level of a JSON record.

Metrics: How Many and How Fast

For metrics, Bourgon’s trait is that they are “aggregatable”: counters, gauges, and histograms that combine many events into a few numbers over time. That makes them cheap to keep, fast to query, and suited to alerts and service level objectives. Google’s Site Reliability Engineering book (2016) defines a service level indicator (SLI) as “a carefully defined quantitative measure of some aspect of the level of service that is provided.” An objective (SLO) is “a target value or range of values for a service level that is measured by an SLI.”

“Carefully defined” is the important part. “The system feels slow” is not an SLI. The share of checkout requests over the last hour that took longer than 500 milliseconds is one; a reading of 6 percent misses an objective of at most 5 percent. The hour’s 95th percentile of checkout latency is another: 530 milliseconds misses an objective of 500.

Both SLIs describe the distribution, not the average. The SRE book shows why: “If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.”

Traces: Where the Time Went

Tracing, in Bourgon’s terms, “deals with information that is request-scoped.” A trace follows one request across services. Each unit of work is a span with a name, a start time, a duration, and, unless it starts the trace, a parent. Together the spans form a tree that shows which call was slow or failed.

Google’s Dapper paper of 2010 made the case with web search, where a single query could need thousands of machines and many services. There, “an engineer looking only at the overall latency may know there is a problem, but may not be able to guess which service is at fault, nor why it is behaving poorly.”

One Context, Not Three Products

A diagnosis usually moves through all three signals. A metric shows that checkout errors climbed after a deploy. Traces from that window, found by the service and error type the metric carries, show which call failed. The trace ID in the log event above leads from the failed call’s trace to its logs, which say why. Every step depends on context the signals share.

Pillars stand apart, which is why the image has critics. Ben Sigelman, the first author of the Dapper paper and later the CEO of LightStep, gave a 2018 KubeCon talk titled “Three Pillars, Zero Answers: We Need to Rethink Observability.” InfoQ’s report of it called looking “at each of the pillars in isolation” a common mistake.

Majors goes further: “I don’t accept the three-pillars definition.” In her view, metrics, logs, and traces can all be derived from wide, structured events, but not the reverse. In the abstract of a 2024 talk, she named the recent change in data types, usability, workflows, and cost models Observability 2.0.

I read the three as three questions—what happened, how much, and where—rather than three tools to buy. Whatever stores them, the design obligation is the same: emit structured events that share one context.

What the Violation Looks Like

The opening’s four log lines came from this function and the two services it calls:

async function processOrder({ userId, sku, qty, amountCents }) {
  console.log('starting');
  try {
    await inventory.reserve(sku, qty);
    await payment.charge(userId, amountCents);
    console.log('done');
    return { success: true };
  } catch (e) {
    console.log('error');
    return { success: false, error: 'something went wrong' };
  }
}

The code shows five warning signs:

It is like diagnosing a car with the dashboard lights unplugged: you know something is wrong, but not what. My Software Development Principles series lists more signs in its entries on this principle and on the three pillars. One sums up the rest: debugging a production issue requires adding new logging and redeploying. The original example and an earlier, dependency-free correction are in my design-principles-js repository on GitHub.

Six Habits That Build It In

The fix is not more logging but six habits, each with a place in the code, shown here in an improved version of the same checkout.

1. Emit Structured Events with Context by Default

“By default” means that whoever writes a log statement should not have to remember the context. A child logger carries it. Create one per request with its identifiers, and every event it emits includes them; an event’s own fields cannot replace them:

export class Logger {
  constructor(fields = {}, write = console.log) {
    this.fields = fields;
    this.write = write;
  }

  // Child events add these fields
  child(fields) {
    return new Logger({ ...this.fields, ...fields }, this.write);
  }

  info(event, fields) {
    this.#emit('info', event, fields);
  }

  error(event, fields) {
    this.#emit('error', event, fields);
  }

  #emit(level, event, fields = {}) {
    const time = new Date().toISOString();
    const base = { time, level, event, ...this.fields };
    // Base fields win any clash
    const record = { ...base, ...fields, ...base };
    this.write(JSON.stringify(record));
  }
}

In production I would use a logging library instead. Pino, for example, has logger.child(bindings), and OpenTelemetry’s instrumentation for pino adds trace_id, span_id, and trace_flags to each record logged inside a span. The class keeps the sample free of a logging dependency and its output easy to capture in a test.

2. Propagate Trace Context across Every Boundary

A trace survives only if every hop passes its context on. W3C Trace Context, a Recommendation since February 2020, revised in November 2021, defines the HTTP header that carries it, traceparent, and a companion, tracestate, for vendor-specific data. The specification’s example is 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01: a version, a 32-character trace ID, the 16-character ID of the caller’s span, and flags, of which the 2021 Recommendation defines one, “sampled.” Even a tracing tool that does nothing else with trace context “MUST propagate the traceparent and tracestate headers and guarantee traces are not broken.”

At a public edge, though, an incoming header is untrusted input. The specification warns against a public API that “naively continues any trace with the sampled flag set.” An attacker could then “overwhelm an application with tracing overhead” or “forge trace-id collisions that make monitoring data unusable.” The specification’s remedies include restarting the trace at a front gate and tracing authenticated and unauthenticated requests differently. My article on the boundary defense principle makes the same case for all input: trust is not transitive.

For HTTP, you rarely write the propagation yourself. OpenTelemetry’s instrumentation for Node’s http and https modules, loaded before the application code, reads the header on incoming requests and writes it on outgoing ones. Node’s fetch, built on undici, has an instrumentation package of its own. Traces break at hops no instrumentation covers, such as a homegrown queue, a database table used as a job list, or work handed to another thread. There, carry the context in the message yourself:

import { context, propagation, SpanKind, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('fulfillment');

// Producer: copy the active trace
// context into message headers
export function publish(queue, body) {
  const headers = {};
  propagation.inject(context.active(), headers);
  queue.push({ headers, body });
}

// Consumer: continue the trace
// from the message headers
export function consume(message, handle) {
  const parent = propagation.extract(context.active(), message.headers);
  const producer = trace.getSpanContext(parent);
  // Also link the producer's span
  const links = producer?.isRemote ? [{ context: producer }] : [];
  return context.with(parent, () =>
    tracer.startActiveSpan(
      'fulfillment.ship',
      { kind: SpanKind.CONSUMER, links },
      async (span) => {
        try {
          return await handle(message.body);
        } finally {
          span.end();
        }
      },
    ),
  );
}

In a test run, the consumer’s span carried the producer’s trace ID, with the producer’s span as its parent and a link. OpenTelemetry’s messaging conventions, still in development, ask for the link in every case. They allow the parent too for a single message, though not by default when the consumer already runs inside another span. The sample relies on the propagator the SDK installs: without one, inject writes nothing. A real consumer would also time its work and name its failures, as habits 3 and 4 do.

3. Measure Every Operation, Failures Included

The checkout function receives its logger, tracer, and meter instead of reaching for globals, which makes the test in habit 5 easy to write. Each order gets a span, a child logger that carries the span’s IDs, and one recording in a duration histogram:

import { SpanStatusCode } from '@opentelemetry/api';

// A failure the business can name
export class OrderError extends Error {
  constructor(code, message) {
    super(message);
    this.name = 'OrderError';
    this.code = code;
  }
}

// Error text may quote a card
const CARD = /\b\d(?:[ -]?\d){12,18}\b/g;
const mask = (text) => text?.replace(CARD, '[card]');

export function createCheckout({ inventory, payment, log, tracer, meter }) {
  const duration = meter.createHistogram('checkout.order.duration', {
    description: 'Time to process an order',
    unit: 's',
    // The SDK's default buckets
    // suit milliseconds
    advice: {
      explicitBucketBoundaries: [
        0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25,
        0.5, 0.75, 1, 2.5, 5, 7.5, 10,
      ],
    },
  });

  return (order) =>
    tracer.startActiveSpan('checkout.place_order', async (span) => {
      span.setAttribute('checkout.order.id', order.id);
      const { traceId, spanId } = span.spanContext();
      const olog = log.child({
        trace_id: traceId,
        span_id: spanId,
        order_id: order.id,
      });
      const started = performance.now();
      let errorType;
      try {
        await inventory.reserve(order.sku, order.qty);
        await payment.charge(order.customerId, order.amountCents);
        olog.info('order.placed', { amount_cents: order.amountCents });
        return { ok: true, orderId: order.id };
      } catch (thrown) {
        // Anything can be thrown
        const err =
          thrown instanceof Error ? thrown : new Error(String(thrown));
        const known = err instanceof OrderError;
        errorType = known ? err.code : 'INTERNAL';
        span.setAttribute('error.type', errorType);
        // Only messages we wrote
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: known ? err.message : undefined,
        });
        olog.error('order.failed', {
          error_type: errorType,
          error: { message: mask(err.message), stack: mask(err.stack) },
        });
        return {
          ok: false,
          error: {
            code: errorType,
            message: 'We could not confirm your order.',
          },
          traceId,
        };
      } finally {
        const attributes = errorType ? { 'error.type': errorType } : {};
        duration.record((performance.now() - started) / 1000, attributes);
        span.end();
      }
    });
}

One histogram covers three of the SRE book’s four golden signals, “latency, traffic, errors, and saturation.” Its count of finished orders gives the traffic, its values give the latency, and error.type marks the errors. OpenTelemetry’s guidance on recording errors agrees: with that attribute, a duration histogram “enables users to derive throughput and error rates.” Saturation, how full the service is, needs measures of its own, such as queue depth or orders still in progress. Four details matter:

4. Give Every Failure a Name

The demo’s two services are stand-ins: inventory always succeeds, and payment simulates a provider that gets 30 milliseconds to answer. Silence becomes a coded error, and large charges take the provider longer, so the opening’s 99,999-cent order fails again, this time with a name:

import { setTimeout as sleep } from 'node:timers/promises';
import { OrderError } from './checkout.mjs';

export const payment = {
  async charge(customerId, amountCents) {
    // Large charges take the
    // provider longer to approve
    const providerMs = amountCents > 50_000 ? 60 : 10;
    const signal = AbortSignal.timeout(30);
    try {
      await sleep(providerMs, undefined, { signal });
    } catch (err) {
      // Pass other errors on
      if (!signal.aborted) throw err;
      throw new OrderError('PAYMENT_TIMEOUT', 'No answer in 30 ms');
    }
    return { transactionId: `tx-${customerId}` };
  },
};

// Inventory always succeeds
export const inventory = {
  async reserve(sku, qty) {
    await sleep(10);
    return { reservationId: `res-${sku}-${qty}` };
  },
};

The checkout turns anything it does not recognize into INTERNAL, even a thrown string or null, which JavaScript allows, so every failure has a code from a short list. The code goes to four places: the span, as error.type with an error status; the histogram; the log event; and the response. OpenTelemetry’s conventions for error.type agree: it “SHOULD be predictable, and SHOULD have low cardinality,” is left off on success, and comes from a list that instrumentations “SHOULD document.”

Not every failed order is a fault, though. A declined card means the system did its job. The HTTP conventions draw that line for servers: unless the instrumentation knows more about the request, a 4xx response leaves the span status unset. Record a decline in an attribute of its own, with no error status and no error.type, and it stays out of the error rate.

For a fault, the caller gets a safe message and the trace ID. Support can search the logs for that ID and, when the trace was kept, open the trace itself. The message, “We could not confirm your order,” claims no more than the checkout knows. With a real provider, a timeout leaves the charge’s outcome unknown, the case my article on idempotency in APIs opens with.

The log event keeps what the caller should not see: the error message and the stack. Logging them follows OpenTelemetry’s March 2026 announcement that it is deprecating the span-event API, Span.RecordException included: “New code should write events as logs that are correlated with the current span.” The span’s status carries a message only when the checkout wrote it. That is stricter than the recording-errors guidance, which asks for the exception message when it is “not expected to contain sensitive details.”

The order carries a card number, so the checkout logs chosen fields, never the whole object. An error message, and the stack that repeats it, can still quote what the failing code was handling. So the checkout also masks runs of thirteen to nineteen digits, plain or grouped, in both. The OWASP Logging Cheat Sheet says payment card data, passwords, and access tokens should usually be “removed, masked, sanitized, hashed, or encrypted” rather than recorded directly. A pattern catches only the shapes it knows, so logs still need tight access control.

The wiring connects the checkout to the SDK, the logger, and both services. It needs four packages, tested at these versions: @opentelemetry/api 1.9.1, @opentelemetry/sdk-node 0.223.0, and @opentelemetry/sdk-trace and @opentelemetry/sdk-metrics 2.12.0. The console exporters print to the terminal; production code would send OTLP, OpenTelemetry’s own protocol, to a collector or a backend:

import { metrics, trace } from '@opentelemetry/api';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace';
import {
  ConsoleMetricExporter,
  PeriodicExportingMetricReader,
} from '@opentelemetry/sdk-metrics';
import { createCheckout } from './checkout.mjs';
import { Logger } from './logger.mjs';
import { inventory, payment } from './services.mjs';

// Console exporters for the demo;
// production sends OTLP instead
const sdk = new NodeSDK({
  serviceName: 'checkout',
  traceExporter: new ConsoleSpanExporter(),
  metricReaders: [
    new PeriodicExportingMetricReader({
      exporter: new ConsoleMetricExporter(),
    }),
  ],
});
sdk.start();

const placeOrder = createCheckout({
  inventory,
  payment,
  log: new Logger({ service: 'checkout' }),
  tracer: trace.getTracer('checkout'),
  meter: metrics.getMeter('checkout'),
});

const result = await placeOrder({
  id: 'ord-1002',
  customerId: 'cus-17',
  sku: 'SKU-2',
  qty: 2,
  amountCents: 99_999,
  card: '4111 1111 1111 1111',
});
console.log(result);
await sdk.shutdown();

In a real service, this setup belongs in its own file, loaded first with node --import. For applications written as ES modules, OpenTelemetry’s getting-started guide adds that “a loader hook is required to properly support instrumentation.” Besides the span, the histogram, and the failure event shown earlier, the demo prints this result:

{
  ok: false,
  error: {
    code: 'PAYMENT_TIMEOUT',
    message: 'We could not confirm your order.'
  },
  traceId: 'b95bfa85c779cff382244df8b4f327e3'
}

5. Test the Telemetry in CI

A feature is not done until it can explain itself, and that is testable. The SDK ships in-memory exporters for spans and metrics, and the logger accepts a write function, so one test can capture all three signals. A second checks that error text is masked:

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { setTimeout as sleep } from 'node:timers/promises';
import { metrics, SpanStatusCode, trace } from '@opentelemetry/api';
import {
  InMemorySpanExporter,
  SimpleSpanProcessor,
  TracerProvider,
} from '@opentelemetry/sdk-trace';
import {
  AggregationTemporality,
  InMemoryMetricExporter,
  MeterProvider,
  PeriodicExportingMetricReader,
} from '@opentelemetry/sdk-metrics';
import { createCheckout, OrderError } from './checkout.mjs';
import { Logger } from './logger.mjs';

test('a timed-out order explains itself', async () => {
  const spans = new InMemorySpanExporter();
  const tracerProvider = new TracerProvider({
    spanProcessors: [new SimpleSpanProcessor({ exporter: spans })],
  });
  const exported = new InMemoryMetricExporter(
    AggregationTemporality.CUMULATIVE,
  );
  const reader = new PeriodicExportingMetricReader({ exporter: exported });
  const meterProvider = new MeterProvider({ readers: [reader] });
  const events = [];

  const placeOrder = createCheckout({
    inventory: { reserve: async () => {} },
    payment: {
      charge: async () => {
        await sleep(30);
        throw new OrderError('PAYMENT_TIMEOUT', 'No answer in 30 ms');
      },
    },
    log: new Logger({}, (line) => events.push(JSON.parse(line))),
    tracer: tracerProvider.getTracer('test'),
    meter: meterProvider.getMeter('test'),
  });

  const card = '4111 1111 1111 1111';
  const result = await placeOrder({ id: 'ord-1', amountCents: 99_999, card });

  // A code and a safe message
  assert.equal(result.error.code, 'PAYMENT_TIMEOUT');
  assert.equal(result.error.message, 'We could not confirm your order.');

  // The span is marked, findable
  const [span] = spans.getFinishedSpans();
  assert.equal(result.traceId, span.spanContext().traceId);
  assert.equal(span.status.code, SpanStatusCode.ERROR);
  assert.equal(span.attributes['error.type'], 'PAYMENT_TIMEOUT');
  assert.equal(span.attributes['checkout.order.id'], 'ord-1');

  // The log event says what broke
  const failed = events.find((e) => e.event === 'order.failed');
  assert.equal(failed.error_type, 'PAYMENT_TIMEOUT');
  assert.equal(failed.trace_id, result.traceId);
  assert.equal(failed.span_id, span.spanContext().spanId);
  assert.equal(failed.order_id, 'ord-1');

  // The histogram has the failure
  // and measures it in seconds
  await reader.forceFlush();
  const [{ scopeMetrics }] = exported.getMetrics();
  const [point] = scopeMetrics[0].metrics[0].dataPoints;
  assert.deepEqual(point.attributes, { 'error.type': 'PAYMENT_TIMEOUT' });
  assert.equal(point.value.count, 1);
  assert.ok(point.value.sum > 0.02 && point.value.sum < 10);
  assert.equal(point.value.buckets.boundaries[0], 0.005);

  // No card number in any signal
  const signals = JSON.stringify([
    events, scopeMetrics, span.attributes, span.status, span.events,
  ]);
  assert.ok(!signals.includes(card));

  await meterProvider.shutdown();
  await tracerProvider.shutdown();
});

test('error text cannot leak a card number', async () => {
  const card = '4111 1111 1111 1111';
  const events = [];
  const placeOrder = createCheckout({
    inventory: { reserve: async () => {} },
    payment: {
      charge: async () => {
        throw new Error(`Card ${card} was rejected`);
      },
    },
    log: new Logger({}, (line) => events.push(JSON.parse(line))),
    tracer: trace.getTracer('test'),
    meter: metrics.getMeter('test'),
  });

  await placeOrder({ id: 'ord-2', amountCents: 100, card });

  // The cause stays, card masked
  const [failed] = events;
  assert.equal(failed.error_type, 'INTERNAL');
  assert.equal(failed.error.message, 'Card [card] was rejected');
  assert.ok(!JSON.stringify(events).includes(card));
});

Run with node --test, both tests pass. Each of eighteen deliberate breaks in the checkout, made one at a time, fails at least one of them. They include a span without its error status, durations in milliseconds, untimed failures, a card number in the span’s status, and unmasked error text. In a pipeline, missing telemetry then turns the build red, just as a missing feature would. In my view, a unit test like this one is the cheapest place to start testing telemetry.

6. Make It Visible: Dashboards and Alerts

Signals nobody looks at explain nothing. The SRE book says that dashboards “should answer basic questions about your service, and normally include some form of the four golden signals.” Alert on what users feel, the SLIs, rather than on every internal cause; in the book’s words, “it’s better to spend much more effort on catching symptoms than causes.” My article on availability by the numbers covers the nines and the error budget they leave.

What Telemetry Costs

Observability is not free, and its costs grow with the system:

The Human Side

The payoff is not only technical. When the telemetry is reliable, engineers can trust what it tells them. They can change code with more confidence, watching a deploy’s effect as it happens. They can work through an incident without guessing in front of colleagues and customers. I think that sense of safety matters as much as any time it saves. Few things in this job are worse than debugging blind, under pressure, with no useful signal to go on.

A Checklist

The principle fits in one question. Ask it of every feature and every service before you ship it:

“If this breaks in production, will I be able to understand why—just from the signals it emits?”

The answer is yes when these hold:

If the honest answer is no, the time to fix it is now, while nothing is broken.

Further Reading