Reliability and operations

Observability is the practice of obtaining evidence sufficient to understand a system's behavior and diagnose failures from external signals. In distributed systems, individual service logs rarely provide a complete account of a request that crosses gateways, workers, message brokers, databases, and regions. Metrics quantify aggregate behavior, traces show relationships among operations, and logs describe discrete events. Each has different strengths and retention costs. A trace ID can link related spans, but it does not by itself guarantee every service recorded the truth or that tracing sampled every relevant request. End-to-end instrumentation must preserve correlation across asynchronous queues and workflow restarts, often using a parent trace context or explicit causation links. The challenge is finding which dependency and failure mode caused a user-visible violation, not merely accumulating more telemetry.

Useful distributed metrics include request success by operation, latency histograms, queue delay, in-flight work, retry count, replication lag, leader term changes, and time since last successful checkpoint. Aggregate averages hide tail behavior and correlated failure. A service with mean response time 80 milliseconds may have p99 latency of several seconds, enough to time out clients. Metrics should preserve labels that support diagnosis without creating explosive cardinality; attaching an individual user ID to every metric label can overload telemetry storage and leak private data. Logs can include stable operation IDs and sanitized error codes while sensitive payloads remain protected. Trace spans should record which deadline was propagated and which retry attempt they represent, to distinguish slow original work from amplification created by retries.

Imagine a distributed course registration where users see intermittent 'submission failed'. The gateway's p95 latency rises, but database CPU remains normal. Traces show a series of retry attempts to a shard whose leader changed three times in one minute, while request queues on the gateway continue growing. A dashboard that only reports instance uptime misses the problem. The incident response should reconstruct one representative operation ID: initial call, deadline propagation, leader redirect, retry, commit acknowledgement, and client response. This can reveal whether a legitimate operation actually committed despite displaying failure. If trace sampling omitted all failed attempts, enrich incident sampling while controlling cost and privacy. Observability must help answer correctness questions, not just speed questions.

Telemetry itself is a dependency that can affect workload. Synchronous logging to a remote collector can slow the critical path during a collector outage; large spans can consume network and storage quotas. Sampling reduces overhead but makes rare failures harder to reconstruct. Privacy and security obligations apply to traces just as to production databases: headers may contain tokens, user identifiers, or confidential document titles. Implement scrubbing at the source and restrict access. Dashboards need runbook links and clear units; a chart without a defined population can suggest false conclusions. Design alerts around service objectives and actionable symptoms, such as failed confirmed writes or sustained backlog growth, rather than alerting on every transient internal exception.

Create a correlated trace laboratory with four services and a message queue. Generate one successful request and one request that times out at the gateway after the backend commits. Learners inspect trace spans and operation IDs to infer the actual durable effect. Introduce a duplicate retry with the same key and display whether the domain state changes again. A second scenario samples out the most important span; ask which conclusion can still be supported and which is unknowable. Present histograms for successful and failed requests separately so tail failures are not hidden. The model intentionally uses synthetic telemetry without actual personal information or a production tracing backend.

Fault injection deliberately introduces bounded failures to test whether a system behaves according to its stated recovery and safety properties. A 'chaos experiment' is useful only when it has a hypothesis, blast-radius limit, abort condition, and measurements. Randomly killing servers without specifying expected behavior can provide excitement but little reliable learning. Begin with a steady-state claim such as 'confirmed writes remain unique during one voting-node crash, although p95 latency may increase'. Select a failure that challenges a known assumption: network partition, delayed storage acknowledgement, process restart, high packet loss, disk-full error, or stale DNS cache. Define how the experiment stops if it threatens unrelated users or violates a critical safety invariant. Fault injection in a production environment also requires appropriate authorization and rollback planning.

A failure should be traced through dependency graphs, not tested in isolation. Killing one API replica may leave the service healthy behind a load balancer, but killing a shared identity dependency can make all replicas reject new clients. Adding 300 milliseconds of inter-zone latency may cause a consensus group to elect leaders repeatedly if its timeout design is too aggressive. Increasing message drops may cause retries to amplify downstream load. These secondary effects are the educational value of well-instrumented fault injection. Inspect client-facing error and latency distributions, queue depth, election frequency, and data-integrity checks before and during the experiment. A test that reports all hosts alive while violating seat-count invariants should fail by correctness criteria.

Consider a three-voter library reservation service. The hypothesis is that losing voter C will not create duplicate confirmed reservations, and that A and B can continue once a valid leader is available. The team injects C's crash and submits a sequence of requests carrying unique idempotency keys. It checks both client responses and the final durable allocation state. Next it injects a partition isolating the current leader A from B and C. The expectation changes: A should refuse or redirect authoritative writes after it cannot meet quorum. If operators see A still returning success while B and C elect a new leader, the test has uncovered a potentially serious safety defect. Simply counting HTTP 200 responses would have hidden the problem.

Experiments need repeatability and scope. Store the random seed, start and end time, topology version, injected faults, software revision, and relevant configuration. Repeat a small laboratory trace before attempting a larger staging run. Realistic tests may require control-plane failures, credential expiration, or unavailable backups, but those should be staged with stricter safeguards because recovery may be difficult. A failure observed once may be timing-dependent; interpret negative findings carefully. Passing a finite suite does not mathematically prove safety under every possible message ordering. Combine fault injection with property-based testing, model checking where practical, and rigorous review of critical protocol transitions.

Build an experiment planner that lets the learner choose a hypothesis, one crash, one partition, or an acknowledgement delay. The tool records the expected invariant and a set of observables. Run the same synthetic client trace with and without the fault, comparing confirmed reservation uniqueness and time to restore service. Include a stop button that halts further injections but preserves the trace for diagnosis. Require students to write which failure conditions they did not test and what evidence would justify escalating from isolated local testing to a controlled staging experiment. The exercise is intentionally safe and local; it cannot actually partition live networks or shut down cloud instances.

Retries help with transient failures but can turn a partial outage into a widespread incident if every layer retries independently. An original request traveling through three services may be retried by the client, gateway, and backend. Even a limit of three attempts at each layer can multiply attempts across the chain in the worst case. A better design identifies the retry owner, uses a global deadline, limits the total retry budget, and returns explicit error categories where retry is or is not appropriate. Transient rate limits, unavailable leaders, and connection resets may justify a bounded retry after delay; validation errors and unsupported operations do not. Retryable semantics must consider whether the operation already happened. Sending the same non-idempotent command twice can be worse than returning an ambiguous error.

Exponential backoff spaces attempts over time. A simple schedule might wait b, 2b, 4b, and so forth, up to a cap, but deterministic synchronized retries from thousands of clients can produce new bursts. Jitter introduces randomness to spread arrivals. The correct shape and cap depend on service objective, remaining deadline, and expected recovery behavior. If the retry delay exceeds the request's remaining useful time, the attempt should not be made. Circuit breakers can reject requests when a dependency appears unhealthy, allowing recovery and protecting upstream resources, but a breaker also risks withholding service during brief recoveries. Bulkheads isolate concurrency so one failing dependency does not consume every worker. Rate limits and queue admission can be more effective than clever retry timers when demand is fundamentally above capacity.

Imagine a ticketing checkout where the payment gateway times out at 600 milliseconds. The client retries twice, the application gateway retries twice per client attempt, and the payment adapter retries twice per gateway attempt. Depending on how 'twice' is interpreted, a single user action could create many payment attempts; if each layer makes up to three total attempts, the upper bound is 3×3×3=27 attempts. Without a stable payment operation key, multiple charges might succeed while every visible response reports a timeout. A robust interface passes a single logical operation ID across all attempts, and the payment authority uses that ID to return the same committed outcome or an explicit pending state. Retry configuration alone cannot guarantee correctness if the downstream effect is not idempotent.

Retry design must also account for overload feedback. If a service's capacity is μ and original arrivals already approach μ, extra retries can push attempted work above capacity, increasing queue times and triggering even more timeouts. A retry budget tied to observed success rate can prevent the retry stream from becoming the dominant load. Hedged reads may reduce long-tail latency when there are independent replicas, but they deliberately send extra work; constrain them with sampling and cancellation. During an incident, measure original requests separately from attempts, inspect how many attempts reach the actual backend, and compare eventual business effects with API responses. Most importantly, never label an operation failed merely because the caller gave up waiting; the remote transaction may still be in flight.

Implement a retry-amplification calculator with layer count, maximum attempts per layer, original request rate, and backoff base. Show the maximum attempted downstream request count, and annotate that this is an upper bound rather than a prediction. Include a small queue model whose service capacity can be overloaded by the extra attempts. Toggle a shared idempotency key to distinguish duplicate attempts from duplicate committed business effects. Let learners experiment with one centralized retry owner versus retries at every layer and compare arrival pressure. Ask students to add a global deadline that stops retries after the useful result window has expired. The model cannot forecast real outages because transport errors and recovery probabilities are simplified.

An operation is idempotent when applying it repeatedly has the same relevant effect as applying it once. Setting a resource status to 'archived' may be idempotent, while incrementing a counter is not unless a stable operation identifier prevents duplicate increments. Idempotency at the HTTP method level is not enough when handlers call external systems or generate new identifiers on every attempt. A well-designed mutation API accepts an idempotency key scoped to the caller and operation semantics, stores a digest of the request with the committed result, and returns that result on retries using the same key. Reusing one key with different request contents should be rejected, not treated as an ambiguous new command. The key store must be durable enough and retained long enough for the promised retry window.

Deduplication itself has a race condition if implemented as 'check then do'. Two workers may concurrently query a table and both observe that key K is absent, then both perform the effect before inserting K. A uniqueness constraint plus a transaction that commits the deduplication record and effect together is stronger. When the effect occurs in a different system, atomicity may not be available. The design could use an outbox, a single authoritative command ledger, an external provider's idempotency API, or reconciliation on uncertain outcomes. A finite deduplication retention period means a very late retry may execute as new after expiration; communicate that contract. Do not claim exactly-once effects across infinite time, arbitrary parties, and irreversible actions without a clearly bounded scope and enforcement mechanism.

Consider grading an online exam. Student S submits attempt A, and the grading worker commits score 82. The response is lost, so the gateway delivers the event again. With a unique key (student,attempt,grader-version), the worker should return the existing score rather than grade a second time if repeatability is required by the contract. If grading rules change during the retry window, the version field helps avoid accidentally applying a new grading policy to the same logical attempt. If an instructor explicitly requests a regrade, it should be modeled as a new operation with its own ID and audit entry, not as an accidental duplicate. The distinction protects both correctness and traceability. A uniqueness constraint without preserving the original response may prevent duplicates but leave clients unable to learn what happened.

Some systems deliver an event more than once but achieve one durable state transition within a transactional boundary. This is an effect guarantee, not proof that the network physically transported exactly one copy. Similarly, a transactional producer/consumer framework may atomically commit input offsets and output records inside its log ecosystem, while an SMS gateway invoked as a side effect remains outside that atomic transaction. Identify the boundary honestly. Privacy and security also matter: idempotency keys can reveal user identities if raw identifiers are embedded and logged. Use opaque keys, scope them to authenticated principals, and reject guesses or collisions according to the threat model. Observe deduplication hit rates, stale-key conflicts, and the ratio of attempted operations to committed effects to detect abnormal retries.

Build an idempotency-key experiment with two clients issuing the same logical order under an unreliable reply channel. The backend uses a durable map keyed by operation ID and request digest. Toggle the atomicity of 'record key plus effect'; in a deliberately unsafe mode two concurrent workers can both pass a non-atomic absence check. In the safe mode, a transaction ensures only one effect commits and each retry receives the stored outcome. Then change the payload while reusing the same key and require a clear conflict error. Add an expiration control and ask what happens when an ancient retry arrives after the key has been pruned. The simulator models a single transactional database; it does not promise atomicity for unrelated external payment providers.

A course-submission service experiences rising timeout errors during a busy examination window. The first symptom is a surge in gateway p99 latency, although the core database remains operational. Logs reveal that the inventory-style validation service began returning intermittent 503 responses following a routine leader election. Each client retries three times, each gateway retries a further three times, and the downstream adapter retries again. The observed rate reaching the database is several times the rate of original student submissions. This feedback saturates connection pools, causing previously healthy operations to time out and retry. The incident is not best understood as three independent services failing. It is a dependency graph in which a modest initial disturbance was amplified by inconsistent retry policy.

Reconstruct one user journey with an operation ID. Student U pressed Submit once at 10:05:12. A first attempt reached the backend and committed the submission, but the response was delayed beyond the gateway deadline. A later attempt retried with a newly generated key and caused another grade row; the UI displayed an error after all attempts exceeded their deadlines. This is simultaneously a latency incident and a data-correctness incident. An alert on HTTP error rate alone would not identify the duplicate grade. The response team should correlate API attempts with durable business effects, check uniqueness constraints, and stop unsafe retries before attempting large-scale autoscaling. The first corrective measure may be to pause redundant retries and enforce a stable submission key, not to add more capacity.

A good incident timeline includes exact changes and uncertainty. At 10:04:58 a leader term advanced, at 10:05:02 queue wait increased, at 10:05:05 retries exceeded originals, and at 10:05:09 rejected connections increased. If tracing was sampled, the team should label gaps instead of inventing missing spans. Examine configuration history to learn whether timeouts changed recently, but do not assume causality based solely on temporal proximity. Check whether the incident coincided with peak traffic and whether server resource limits had already been near saturation. Separate the initiating event from the amplification mechanism and from the missing idempotency guard that allowed duplicate effects. Each requires a different preventive action and a different test.

The recovery plan introduces one retry owner, bounded exponential backoff with jitter, a propagated overall deadline, and load-shedding rules. The payment or grading sink uses a durable operation ledger to return one result for each stable key. The worker queue has a finite size and rejects overflow quickly, avoiding memory exhaustion. Observability adds dashboards showing original operations, attempted RPCs, committed effects, and deduplication hits separately. The team tests a leader election under load in staging and confirms that retries do not create an unstable feedback loop. Changing the timeout alone might mask one symptom while leaving duplicate-effect risk unresolved; therefore corrective tests must include both performance and safety assertions.

Close with a blameless evidence review that still assigns concrete ownership. Which team owns the idempotency contract? Who can change retry defaults shared across SDKs? Who maintains database uniqueness and a repair procedure for duplicate grades? Which alarm should have fired when attempted work grew faster than original demand? Write explicit acceptance conditions: no duplicate committed submission for one operation ID, a bounded ratio of RPC attempts to originals under an injected partial outage, and a documented recovery of ambiguous outcomes. A finite load test cannot establish permanent safety, but it provides a repeatable barrier against regression. Present unresolved risks honestly, including long-delayed retries after key expiration and dependencies without external idempotency support.