Interactive graphics

A frame is a dependency graph, not a single draw call

An interactive renderer repeatedly transforms application state into an image under a strict time budget. A common real-time graphics pipeline starts with scene updates, visibility decisions, command recording, vertex processing, primitive assembly, rasterization, fragment processing, depth and blending operations, and presentation. Modern APIs and GPUs can execute work in parallel, and the actual scheduling is more complicated than this linear story. Nevertheless, the conceptual stages are useful for identifying which data is available where and why some operations are expensive. A vertex shader does not automatically know which fragments will be visible, and a fragment shader cannot assume that it executes once for every final displayed pixel.

A vertex shader typically receives attributes such as position, normal, UV coordinates, and skinning data. It produces a clip-space position and interpolated outputs for later stages. Clip-space coordinates are homogeneous; perspective division occurs before the final viewport mapping. The order of model, view, and projection transforms matters. If a model-space point is multiplied by a matrix intended for the opposite convention, it may appear mirrored, shifted, or clipped. State whether vectors are rows or columns and whether matrix multiplication is applied on the left or right. These choices are conventions, but inconsistency between CPU and GPU code is a correctness bug.

The rasterizer identifies sample locations covered by projected primitives. Perspective-correct interpolation is usually applied to attributes that originate on the geometry. Fragment processing computes outputs for covered fragments, but depth testing, early rejection, multisampling, and shader side effects can change the amount of work performed. A transparent particle may run its fragment shader even though it contributes very little to the final image. Heavy overdraw occurs when many fragments are processed for the same screen region. A performance investigation should therefore distinguish triangle count, visible pixel count, and fragment invocations instead of assuming they are proportional.

GPU memory is organized into resources such as vertex buffers, index buffers, textures, uniform or constant buffers, and storage buffers. Uploading a large mesh every frame can consume bandwidth and force synchronization even if its geometry is unchanged. Static geometry should generally remain resident and be referenced by draw commands. Dynamic data may use staging buffers, ring buffers, or persistently mapped regions depending on the API and platform. The correct choice depends on ownership and synchronization rules. Writing into memory while the GPU is still reading it can cause corrupted frames or undefined behavior, so explicit resource lifetimes are not optional engineering details.

A render pass reads some resources and writes others. A shadow pass may produce a depth texture consumed later by the main lighting pass. A postprocess pass may read an HDR color buffer and write a tone-mapped output. The dependencies between passes form a graph. A render graph can allocate temporary resources, track transitions, and schedule work based on declared reads and writes. If a pass reads an image before the producing pass has completed or before the required barrier, the result may be stale or undefined. The pipeline must establish a happens-before relationship appropriate to the graphics API rather than relying on accidental driver behavior.

GPU parallelism and divergence

GPUs execute many shader invocations concurrently, often in groups that share scheduling and execution resources. When nearby invocations take different branches, the hardware may execute multiple paths with masks, reducing effective utilization. This does not mean branches are always slow; coherent branching can be inexpensive, and avoiding a branch by doing unnecessary heavy work can be worse. The correct approach is to profile representative content. Memory access patterns, cache behavior, register pressure, occupancy, and synchronization can dominate arithmetic cost. A shader with fewer lines of source code is not necessarily faster.

Compute shaders extend GPU work beyond the fixed raster stages. They can process particles, generate geometry, build image pyramids, and perform reductions. Workgroup-local shared memory can reduce repeated global-memory access, but barriers must be used correctly when threads cooperate. A race condition may appear only on certain devices or scene sizes, making deterministic reference tests important. CPU and GPU implementations of the same small kernel can be compared within tolerances, but floating-point reductions may produce different last bits because addition is not associative in finite precision.

A shader is a function inside a larger rendering system

A shader should declare the spaces and units of its inputs. A normal in model space cannot be dotted directly with a light direction in world space unless the model transform is identity or both vectors are transformed consistently. A texture may contain nonlinear display-encoded albedo while a lighting buffer contains scene-linear radiance. Mixing these representations can produce plausible but wrong images. Small shader functions should have testable contracts: a normalized normal remains finite, a light behind an opaque surface produces no direct diffuse contribution, and a roughness parameter outside its valid range is clamped or rejected deliberately.

A vertex transform pipeline can be tested with a few known points. The model origin should map to the object's world position. A point at the camera near plane should map to the appropriate normalized device depth. A normal must use the inverse-transpose of the linear part of a nonuniform scale transform, followed by normalization, if correct geometric orientation is required. Applying the position transform directly to a normal can make lighting wrong even when the rendered mesh outline is correct. These are ideal cases for automated numeric tests because the expected results can be calculated independently of a GPU screenshot.

Shader specialization can improve performance by eliminating branches or unused features, but uncontrolled permutations can create a compilation and maintenance burden. A material system with switches for normal mapping, clear coat, transmission, skinning, fog, and shadows can produce a combinatorial number of variants. Some features can be handled through uniform branches, dynamic indexing, or shared shader modules. The right architecture balances runtime cost, compile time, binary size, and debuggability. A renderer that compiles thousands of variants on demand may stutter during gameplay even if its steady-state GPU frame time is excellent.

Debugging shaders requires observability. Outputting world-space normals as colors, roughness as grayscale, or mip levels as a heat map can isolate problems faster than adjusting final lighting. GPU debug markers and validation layers help locate invalid resource transitions and bindings. A render capture can show draw calls, bound resources, and intermediate targets. Because captures can be large and vendor-specific, preserve small reproducible scenes that expose each bug. A minimal triangle with a known texture often teaches more than a complex scene with twenty interacting effects.

Organize data according to operations, not only visual hierarchy

A scene graph represents parent-child transforms and logical relationships. It is useful for attaching a camera to a vehicle or a prop to a character's hand. However, a deeply nested graph can become costly to traverse if every transform is recomputed every frame. Dirty flags can mark which subtrees need updates after a parent changes. A flat array of renderable instances may be better for batch processing and GPU uploads, even while a separate hierarchy is retained for editing. There is no requirement that the editor's object tree and the renderer's data layout be identical.

Spatial indices support visibility and collision queries. A bounding volume hierarchy, octree, grid, or loose tree can reduce the number of candidate objects for a camera or ray. The best structure depends on scene distribution and motion. A uniform grid is simple for evenly distributed particles but can be inefficient when most objects cluster in a small region. A BVH can adapt to uneven distributions but may require rebuilding as objects move. Frustum culling based on bounding spheres or boxes should be conservative: a bounding object intersecting the frustum must not be rejected merely because its center is outside.

Level of detail selects simplified representations when objects occupy fewer screen pixels. A distant building may need only a coarse mesh and low-resolution textures, while a nearby one needs detailed geometry. The selection threshold should be related to projected error rather than an arbitrary world-space distance when possible. Abrupt switching can cause visible popping, so hysteresis or cross-fading may be used. Cross-fading can temporarily increase overdraw or introduce transparency artifacts, illustrating how one quality improvement creates a different performance cost.

Instancing reduces the overhead of drawing many copies of the same mesh with different transforms and material parameters. A forest with thousands of similar trees can be more efficient when instance data is stored in a buffer and processed in batches. Instancing does not eliminate the cost of shading the visible pixels or processing the total geometry. A dense forest can remain fragment-bound even after draw-call overhead is reduced. GPU-driven culling can further reduce CPU submission cost, but its complexity and synchronization overhead should be justified by measured gains.

Asset streaming is a continuity problem. When a user moves through a large environment, the engine must load geometry, textures, and animation data before they become visually necessary. Predictive loading uses camera movement and spatial proximity, but predictions can be wrong. A robust renderer provides fallback levels of detail and avoids blocking the render thread on disk or network access. Memory budgets must include temporary duplication during transitions. An asset that nominally fits in GPU memory may still cause an out-of-memory failure when the old and new versions coexist during a hot reload.

Measure the bottleneck before changing the algorithm

At 60 frames per second, the nominal frame interval is approximately 16.67 milliseconds. This does not mean the GPU can always consume all 16.67 ms: CPU work, presentation scheduling, operating-system activity, and synchronization also affect latency. At 120 Hz, the interval is approximately 8.33 ms. A frame may be CPU-bound, GPU-bound, memory-bound, or limited by synchronization. Optimizing triangle count will not fix a CPU script that takes 25 ms per frame. Likewise, reducing CPU allocations will not fix a fragment shader that dominates GPU time on a high-resolution display.

Profile with timestamps and counters, not impressions. Measure CPU update time, command recording, GPU pass durations, draw calls, triangles, overdraw, resource uploads, and memory usage. GPU and CPU can overlap, so summing all measured stage durations may exceed or differ from wall-clock frame time. Use a timeline profiler to identify stalls and dependencies. Test warm and cold starts separately because shader compilation, asset loading, and cache initialization can dominate the first few frames. Report both typical and tail latency, such as the 95th or 99th percentile frame time, because rare long frames create visible stutter even when average FPS is high.

Resolution scaling can reduce pixel-related GPU work, but it does not proportionally reduce vertex processing or CPU simulation. Halving each image dimension reduces pixel count to one quarter, which may greatly help a fragment-bound renderer. If the scene is CPU-bound, the same change may have almost no effect. Temporal upscaling can reconstruct a higher-resolution image from a lower-resolution render, but moving thin geometry and newly visible regions can reveal artifacts. The decision should compare image quality and latency at representative motion speeds, not only still screenshots.

Performance budgets should be allocated by pass. For example, a 16.67 ms target might reserve time for simulation, geometry, shadows, lighting, postprocessing, and presentation overhead. These allocations are planning assumptions, not guarantees that stages execute sequentially. If a new shadow technique adds 3 ms, the team must identify which other work can be reduced or whether the frame-rate target will change. A performance regression test should run the same camera path, scene revision, output resolution, and quality settings before and after an optimization. Otherwise a faster benchmark may simply be rendering less content.

Mobile and accessibility constraints

A browser-based graphics demonstration must work on devices with smaller screens, different GPU capabilities, and limited thermal headroom. A shader that runs well on a desktop may trigger long compilation delays or thermal throttling on a phone. Device pixel ratio can multiply actual render resolution far beyond CSS dimensions. Cap the render target size when appropriate and provide quality presets. The application should degrade gracefully if advanced GPU features are unavailable, displaying a static diagram or simpler simulation rather than a blank page.

Interaction quality is not solely frame rate. A slider must be keyboard-operable, labeled, and accompanied by numerical output. Animations should respect reduced-motion preferences where possible. Color-coded diagnostics need text labels or patterns for users who cannot reliably distinguish the chosen colors. Avoid making essential information appear only during a brief hover. These choices are engineering requirements for an educational graphics application, not optional decoration.

Make a controlled performance hypothesis

An interactive campus map displays hundreds of buildings, trees, signs, and moving avatars. On a laptop it runs near 60 FPS, but on a midrange phone the camera occasionally freezes. A first hypothesis is that too many polygons are being drawn. Before simplifying geometry, collect a trace showing CPU and GPU work. If the CPU spends most of its time traversing thousands of scene nodes and issuing separate draw calls, instancing and transform caching may help. If the GPU spends most of its time shading overlapping foliage, geometry simplification alone may provide little benefit.

Build a repeatable camera path that passes through open areas and dense tree clusters. Record frame-time distributions at fixed viewport sizes and device pixel ratios. Inspect whether stalls coincide with entering new regions. If so, asset streaming or shader compilation may be the cause rather than steady-state rendering. Precompile common shader variants and prefetch nearby textures as a controlled experiment, while recording memory peaks. A performance fix that eliminates a stall by loading every asset at startup may simply move the problem into an unacceptable initial load time.

If tree rendering is CPU-bound, batch repeated meshes with instance transforms and use a conservative spatial index to cull invisible groups. Validate culling by temporarily disabling it and comparing the resulting images. Any missing visible tree is a correctness failure even if performance improves. If foliage is fragment-bound, investigate alpha-tested leaves, overdraw, and resolution. Test a lower render scale and observe whether GPU time changes as predicted. A strong reduction suggests a pixel-related bottleneck; a small change suggests another limiting stage.

The map must also be useful as a learning tool. Preserve readable labels when lowering visual quality. Keep important accessibility features such as keyboard navigation and text alternatives for interactive controls. Provide a quality mode that prioritizes clear building outlines and signs over cinematic reflections. Report final performance with the test hardware, scene revision, resolution, and percentile frame times. The investigation should conclude with measured causes and documented tradeoffs, not a claim that one universal optimization makes all scenes faster.