System design

Service boundaries should follow meaningful ownership and change patterns, not only a desire to draw many boxes. A monolith can be distributed in deployment while still preserving coherent transactions; a microservice architecture introduces remote calls and independent data ownership. Extracting a service makes sense when it has a distinct domain contract, workload, team responsibility, or lifecycle that justifies operational overhead. Every new synchronous dependency can reduce the availability of the caller's critical path. A system that needs identity, catalog, payment, inventory, and notification services to respond synchronously for every page load may have more failure modes than a simpler integrated design. Draw the dependency graph and mark which edges are mandatory for completing a user action. Optional recommendations and analytics should usually not prevent core purchase or registration functionality.

A domain boundary determines who may authoritatively change a piece of state. If both Orders and Inventory services independently modify the same stock counter, conflicting ownership can defeat consistency guarantees. One service can own the invariant, while others request changes through versioned APIs or events. Shared database access across supposed microservices undermines isolation if one service can silently bypass another's business rules. On the other hand, splitting a tightly coupled transactional workflow across services adds coordination, monitoring, and recovery cost. A modular monolith with clear internal interfaces can preserve separation of concerns without remote network failure on every method call. Evaluate the number of teams, rate of change, scaling requirements, and operational staffing before accepting that cost.

Consider a public library portal. Searching the catalog, viewing book descriptions, and displaying recently read items have different failure consequences from reserving the final physical copy of a book. Search indexes and recommendation caches can tolerate stale data; reservation authorization cannot claim a confirmed allocation based only on an outdated cache. A strong boundary gives the reservation service authority over the unique-copy invariant and publishes events for secondary views. If notifications fail, the reservation may still be confirmed with an in-app status page. The API should distinguish a tentative reservation from a final one. This reduces the number of synchronous critical-path dependencies, while making eventual updates and operational reconciliation explicit. Service design is a sequence of contract decisions, not a diagramming fashion.

Dependency reviews must include control planes and shared infrastructure. Several independent services may all depend on the same DNS resolver, network egress, secret manager, or database connection pool. A seemingly resilient multi-zone deployment can still fail together when a shared control service is unavailable. Model resource budgets for each call chain: deadlines, concurrency, queue capacity, and retries. Version APIs to allow old and new clients to coexist through deployment transitions. Apply authentication at meaningful trust boundaries; a perimeter gateway is not a substitute for service-to-service authorization when networks are segmented or credentials are compromised. Observe user journeys rather than only individual container-health metrics, since the user cares whether the whole transaction succeeds.

Build a service dependency graph for a university admissions workflow. Learners mark edges as mandatory synchronous calls, asynchronous effects, or optional reads. Introduce one dependency outage and compute which user actions are unavailable under their chosen design. Move email notification from the synchronous path to an outbox-backed queue and re-evaluate, while preserving an honest 'notification pending' state. Add a shared identity provider dependency and ask how cached credentials, session lifetimes, and security policy affect degraded service; do not suggest bypassing authentication. The lab illustrates architectural tradeoffs and cannot guarantee numerical reliability without measured failure correlation and request traffic.

Load balancing routes requests among eligible instances, aiming to improve latency, capacity use, and failure isolation. Round-robin balances request counts, not necessarily work, because requests can have different costs. Least-connections may favor workers with fewer active requests but can fail when long-lived connections are cheap and short calls are expensive. Consistent hashing stabilizes ownership for stateful workloads but may amplify hot-key imbalance. A load balancer also needs accurate health information and safe handling of stale endpoints. A host returning a liveness probe can still be unable to serve a write because it has lost quorum or a critical dependency. Use readiness criteria that match the actual operation being routed, and distinguish draining from crashing an instance during deployment.

Admission control decides whether to accept more work. If offered load exceeds sustainable service capacity, no routing algorithm can make the queue disappear indefinitely. Little's Law relates stable average in-flight work L, arrival rate λ and response time W by L=λW. Under stable assumptions, increasing wait time raises the number of requests concurrently present even when arrival rate stays fixed. Thread pools, connection limits, and queues need explicit bounds; unbounded queues can consume memory and make future requests wait beyond their deadlines. When overloaded, failing quickly with a retryable response and clear backoff guidance may be kinder than accepting work that cannot finish. Rate limits should consider authenticated users and fairness, not merely IP addresses that could represent whole campuses.

Imagine an online testing platform with four instances, each sustainably serving 50 requests per second. At a stable 180 requests per second, total nominal capacity 200/s exceeds arrival rate, but skew and synchronized peaks may still harm p99 latency. At 260 requests per second, no distribution across the four instances can sustain the load forever. The approximate short-term queue growth is at least 60 requests per second if service remains capped at 200 and there is no shedding. Adding an instance may help, but provisioning delay and a shared database bottleneck limit improvement. A rational system caps admitted concurrency, sheds low-priority optional requests, and preserves critical exam submission with durable status. Retry storms must be included in the attempted arrival rate; original user demand alone is insufficient.

Load balancing under node failure requires carefully designed draining and request replay. A node marked unhealthy may have already committed a user operation before its response was lost. Redistributing the same request to a fresh node can cause duplicate effects unless the operation ID follows the retry. Sticky sessions can preserve locality but create uneven load and complicate failover if state lives only in one process. Prefer stateless frontends where practical, or explicitly replicate session state with appropriate consistency. Global load balancing across regions should account for healthy quorum, user latency, and residency constraints, not simply choose the geographically nearest IP. Measuring success includes correctness, tail latency, queue overflow, and fairness across clients during degradation.

Create a load-balancing simulator with four workers, heterogeneous service capacities, request costs, and choices of round-robin or least-outstanding dispatch. Show queue depth and completed request count over a deterministic workload. Raise arrival rate above aggregate service capacity and ask why queues grow even if requests are evenly distributed. Add bounded admission with rejection counters and compare latency for admitted jobs. Then fail one worker mid-run and show which in-flight requests are ambiguous and must use stable identifiers. The learner should not interpret the toy throughput numbers as production benchmarks: real workloads include cache locality, network overhead, shared storage, and variable task durations.

Scalability asks how a system's resource requirements and service quality change as demand grows; it is not synonymous with adding more servers. Scale-up gives an instance more resources, while scale-out distributes work across instances. Some workloads parallelize readily, but coordination, shared state, and contention impose limits. Amdahl's Law states that if fraction s of execution is inherently serial, ideal speedup using p processors is bounded by 1/[s+(1−s)/p] for the fixed workload. If s=0.1 and p=10, the ideal limit is about 5.26, not 10. The law is a useful warning about serial bottlenecks, not a universal predictor of web-service throughput. Real capacity planning must include queueing, caching, storage I/O, network, replicas, and failover headroom.

Benchmark the entire critical path before scaling one component. A stateless HTTP layer may handle thousands of extra requests after scale-out while a single database writer still serializes updates. A cache may accelerate popular reads until a hot entry expires and all clients stampede onto the origin. A hash-partitioned backend may distribute ordinary keys but remain limited by one heavily contended account. Latency objectives matter: 10,000 requests per second is not useful if 20 percent finish after clients time out. A performance report should identify workload mix, arrival pattern, request size, consistency mode, cache hit rate, hardware, and percentile distributions. Comparing two systems without these conditions risks making meaningless numerical claims.

Capacity planning should allow failures rather than use every resource at 100 percent. If a cluster needs six workers to satisfy normal peak demand and is expected to survive loss of one availability zone containing a third of workers, six total workers may be insufficient when two disappear. Provisioning and autoscaling delays mean spare capacity or temporary admission control may be necessary. Headroom is not wasted by definition; it buys response-time stability and fault survival. But overprovisioning also consumes budget and energy. Forecast demand from observed traffic, test realistic bursts, model shared dependencies, and state the acceptable degraded mode. Before adding replicas, determine whether replication traffic itself increases write cost or whether a smarter data partition reduces contention.

Scaling experiments often reveal non-linear behavior. At low load, extra workers increase throughput. Near saturation, queueing delays explode even when CPU utilization rises only slightly, because service-time variation and contention create tail effects. Autoscaling based solely on average CPU may react too slowly to a queue spike or may scale useless frontends while the database is blocked. Queue length per active worker and deadline miss rate can be better signals for specific job systems. A meaningful cost model includes compute, storage, replication bandwidth, cross-region egress, observability, on-call effort, and recovery drills. A system that saves milliseconds at an unsustainable operational cost is not automatically better for the library or its users.

Build a scaling exercise with configurable serial fraction, processor count, worker throughput, and failure reserve. Plot Amdahl's ideal speedup alongside a simple throughput ceiling determined by a database bottleneck. Students can increase workers until the shared writer caps progress, then experiment with sharding the writer or reducing mandatory synchronous calls. Calculate whether the remaining workers after a zone loss can serve a specified peak without queue growth. Label all model outputs as illustrative; do not infer cloud prices or real-world throughput from the algebra. The final exercise requires a written decision that weighs measured workload, user latency, failure budget, and ongoing operational cost.

A distributed system stretches trust boundaries across processes, service accounts, networks, queues, storage replicas, and control planes. Encryption in transit protects confidentiality and integrity on a communication channel when endpoints and keys are correctly authenticated, but it does not determine whether a calling service is authorized to perform a domain action. Mutual TLS can authenticate both peers at a transport layer; application authorization must still check the caller's permissions for the requested object and operation. Credentials distributed to many services enlarge the attack surface, and overly broad permissions can turn one compromised worker into a system-wide incident. Favor narrowly scoped service identities, bounded credential lifetimes, rotation procedures, and audit logs that preserve operation provenance without leaking secrets.

A message consumer should not trust a queued event solely because it came from an internal broker. Producers may be compromised, schemas can be malformed, and replayed messages may trigger duplicate effects. Validate message shape, authorization context, provenance, and idempotency before taking action. If an event asks a payment worker to transfer funds, the event should represent a validated, authorized operation rather than a free-form amount any service can publish. Integrity protection does not repair faulty business authorization: a correctly signed request from an overprivileged service can still perform an inappropriate action. Secure design should identify who can create, modify, replay, and revoke authority for each class of command.

Consider a research collaboration platform with a public upload API, processing workers, a distributed object store, and an export service. A malicious file may contain path traversal sequences, extreme decompression ratios, or content intended to exploit downstream parsers. Network segmentation helps but does not replace careful file handling. The upload service should create an opaque object ID, store content with minimal permissions, and pass a constrained processing job to the worker. The worker should run with restricted resource limits and write derived outputs to separate authorized locations. The export service must check the requesting researcher's access at export time, not assume that a valid internal object ID implies authorization. Cross-service tracing should record the verified principal and operation while avoiding sensitive document content.

Security and availability can conflict during an incident, but silent bypasses are dangerous. If the identity authority becomes unavailable, a service might continue honoring already validated short-lived tokens within a documented policy, or it may reject privileged actions until authority returns. It should not disable authentication simply to keep success metrics green. Key-management services themselves need backups and disaster-recovery procedures consistent with cryptographic policy. Secrets should never be embedded in a cumulative book source archive or client-side simulation. An architecture review should inspect certificate rotation, clock skew tolerance, service account revocation, broker ACLs, encryption at rest, and audit-log durability. Threat models should include insider mistakes and supply-chain failures as well as external attackers.

Build a capability-tracing exercise for a fictional library export operation. A request travels through API, queue, worker, and object store. The student assigns an authenticated service principal and minimum authorization at each hop, then injects a forged event or expired credential. Show the denial point, while separating authentication failure from business authorization failure. Add a replayed event with a valid signature but an already-consumed operation key, and explain why replay protection is distinct from cryptographic authenticity. The simulator uses synthetic credentials only; it is educational and does not verify actual TLS stacks, certificate chains, provider IAM policies, or vulnerability resistance.

A global STEM platform serves static textbooks, interactive simulations, discussion activity, library search, and a limited number of authenticated contribution workflows. These functions have sharply different consistency and availability needs. Public textbook pages are immutable between releases and benefit from static hosting and caching. Simulation code runs locally in the reader and should not require a live network for every slider change. Discussions and author submissions involve mutable state, moderation, authentication and notifications. Before dividing this product into microservices, write a traffic and risk profile for each feature. A small team may operate a static frontend and one well-structured backend more reliably than a fleet of loosely coordinated services. Distribution should solve measured constraints, not create the appearance of sophistication.

Begin with the reader experience. Chapter routes should have stable canonical URLs and server-visible content so users can access them under limited connectivity and search engines can find substantive teaching material. Client simulations need bounded input controls and should distinguish pedagogical models from physically complete experiments. The content deployment process is an atomic release problem: a navigation index, chapter pages, and assets need compatible versions. Publishing an updated index before its referenced chapter pages are available creates broken links even though every individual file is syntactically valid. A release manifest with content hashes and verified staging routes improves confidence. Keep drafts out of public sitemaps, and preserve previous approved material when adding a new subject.

For authenticated contributions, one durable state store can hold submission status and idempotency keys. A contributor uploads an article, a validation worker checks metadata and file safety, and a moderator decides whether to publish. These stages need not share one synchronous HTTP request. Store the initial submission transaction with a stable operation ID, then use an outbox or equivalent mechanism to schedule asynchronous validation. The site can display RECEIVED, CHECKING, NEEDS_REVISION, APPROVED, or REJECTED states. A failed notification service should not erase a submitted manuscript. The moderation decision needs authorization and an audit record; a worker that merely processed a file should not have the right to publish it to all readers. This boundary reflects the actual domain more clearly than a generic 'content microservice'.

Now consider worldwide deployment. A CDN can serve most public reading traffic from nearby points of presence, reducing origin load. A central authoritative write region may suffice for low-volume submissions; replication and disaster recovery protect against major outages. Introducing globally writable databases and cross-region consensus for a handful of daily submissions would add cost and complexity without much benefit. If future demand justifies regional authoring, redesign with explicit conflict and ownership semantics. Keep data residency, backup retention, account deletion, and trace privacy in the risk assessment. Availability objectives should distinguish browsing a cached textbook from accepting a new research submission. The platform can honestly degrade to read-only mode during a write-service outage rather than show false success.

The final architecture review weighs alternatives. Option A is a static reader plus a modular backend with a transactional database and managed job queue. Option B splits discussion, contribution, search indexing, and moderation into separately deployable services. Compare each option's critical dependencies, expected traffic, operational burden, recovery time, and risk of duplicate or lost events. Stress-test a simulated contribution when the worker crashes after validation but before acknowledgement; confirm that retry uses the same operation key. Test a release with one missing chapter asset and require the deployment gate to fail before exposure. The best design is the smallest one that meets documented user and correctness requirements while remaining operable by the actual team. Architectural complexity should be earned by evidence from workload and incidents.