Usability evaluation

What a task study can establish

A usability task study examines what people do when they attempt a meaningful goal using an interface. The central unit of analysis is the attempt, not the participant's stated preference for a visual style. A person may report that a page looks simple yet fail to complete a critical task because the next action is labeled ambiguously. Conversely, a visually dense screen may support experienced users efficiently. A study therefore begins with a question about observable behavior: can a student find a laboratory safety procedure, can a lecturer correct an uploaded grade, or can a customer recover from a rejected payment without entering the same details twice? Each question requires a task, a success criterion, and a context that resembles the intended use.

The task statement should describe a goal without giving away the interface path. Compare "Find the university's current examination timetable for second-year physics" with "Click the timetable tab and select second year." The first tests discoverability; the second tests whether a participant can follow directions. If a task necessarily involves a known starting point, specify that point consistently. Use realistic content and conditions: an unfamiliar course code, a phone with limited screen width, a noisy environment, or a slow network. Do not add constraints merely to make the study difficult. Choose the context because the product will be used there, or because a specific failure is plausible.

Define success before observing participants. For a timetable task, success might require identifying the correct date, time, room, and examination version without facilitator help. Opening a timetable PDF is not sufficient if the student reads an outdated session. A partial success category can be useful, but it must have an explicit rule: perhaps the correct date and time are found while the room is missed. Record whether assistance was requested, whether an error was recovered, and whether the final answer is correct. Do not silently change the definition of success after seeing results. If the study uncovers an unanticipated valid strategy, document the revision and keep the original observations available for comparison.

Sampling, recruitment, and consent

The intended user population is not a single average person. A university service might serve new applicants, returning students, administrative staff, international students, and people using assistive technology. Their goals and experience differ. Recruitment should therefore follow the decision being studied. If the question concerns first-time navigation, participants who built the website are poor substitutes for first-time users. If the question concerns a specialist laboratory booking system, a general convenience sample may misunderstand the workflow for reasons unrelated to the interface. Screening questions should identify relevant experience without revealing the desired response.

A small formative study can identify specific breakdowns but cannot precisely estimate a population failure rate. Observing three people struggle with the same date selector is enough to justify investigation; it is not enough to assert that a fixed percentage of all students will struggle. The often repeated idea that one fixed number of users finds most usability problems relies on strong assumptions about issue prevalence and independence. Real participants differ, problems are correlated, and detection probabilities vary. Use small sessions for iterative discovery, then plan larger and more representative studies when decisions require reliable estimates or subgroup comparisons.

Consent must be meaningful. Tell participants what will be recorded, who will see recordings, how long the data will be retained, and whether participation can be stopped. In a classroom or workplace, make clear that declining will not affect grades, employment, or service access. Avoid collecting actual account passwords, health records, examination results, or other sensitive information when realistic synthetic data can serve the task. Provide accessible participation arrangements, including assistive technology and breaks where needed. If recording is declined, use a note-taking protocol instead of treating recording as mandatory. Researchers should separate identifiable contact information from coded task observations.

Moderation without rescuing the design

A facilitator should introduce the task neutrally and avoid teaching the interface. "What would you do next?" may be appropriate when a participant has stopped, but "Try the blue button on the right" destroys the evidence about discoverability. Think-aloud prompts can reveal reasoning, yet speaking while working can alter task time and effort. If time comparisons are the primary outcome, use a consistent moderation protocol and consider collecting explanations after the task. Record when a participant asks for help and whether help is given; assisted completion is not equivalent to unaided completion.

Prepare a moderator script and an observer sheet. The sheet should include participant code, device and input method, task version, start and end rules, outcome, critical errors, recovery attempts, prompts, and notable quotations. Define timing rules precisely. For example, start timing after the moderator finishes reading the task and stop when the participant announces the answer. If the participant is interrupted by a phone call, mark the attempt as interrupted rather than including the entire pause as interface time. The aim is comparability, not pretending that human behavior is perfectly controlled.

An observer should distinguish evidence from inference. "Participant tapped the course title three times and then opened search" is an observation. "Participant did not understand that the title is not a link" is a plausible interpretation. "Make the title clickable" is one possible intervention. Several observations may support a different cause, such as missing affordance or inconsistent navigation. Capture enough context to reconstruct the failure before deciding on a remedy. Do not treat a participant's explanation as infallible; people can rationalize actions after the fact. Triangulate comments with behavior and task outcome.

Pilot sessions and edge cases

A pilot is a rehearsal of the protocol, not a statistical test of the design. It can reveal that a task cannot be completed with the supplied test account, that an instruction gives away the answer, or that a screen recording hides the keyboard focus. Run at least one pilot with the exact device, browser, network constraints, and sample data intended for the study. Check that test accounts can be reset and that the moderator knows how to respond to service outages. If the task depends on live data, preserve a record of the state seen during the session so later failures can be interpreted.

Include paths that matter operationally: an empty search result, an expired session, a duplicate submission, and a permission denial. A product can perform well on its ideal path and still be unsafe when an error occurs. If an application asks users to submit a scholarship application, evaluate whether they can tell if the submission succeeded and whether retrying creates duplicates. These paths may be less common than basic navigation but have greater consequences. Severity and reversibility should influence task selection alongside frequency.

Analysis and actionable findings

After sessions, create a task-by-participant matrix. Each row should preserve the task outcome and a short observation, not just a color-coded success cell. Cluster incidents by mechanism: terminology mismatch, missing feedback, inaccessible control, or confusing recovery path. Assign a provisional severity based on consequence, persistence, and recoverability. Do not multiply arbitrary ratings and present the result as a measured risk probability. Prioritize critical barriers, then repeated failures, then cosmetic friction. For each issue, state the evidence, a hypothesized cause, a proposed change, and how the change will be tested.

Report the scope honestly. A formative report should identify the recruited groups, study setting, task scripts, assistive technology used, observed incidents, and limitations. An observed 4/6 success rate is a description of six attempts, not a precise population estimate. The study may still support a strong design decision if failures are severe and causally interpretable. Retain counterexamples: a participant who succeeded through an unexpected path may reveal a useful alternative navigation strategy. Evidence-based usability work is not a search for confirmation that the chosen design is already correct.

Inspection is a different method from observing users

Heuristic evaluation is an expert inspection in which evaluators examine an interface against general usability principles. It can be performed before recruiting participants and can reveal inconsistencies, missing status information, unclear controls, and weak error recovery. However, it is not a substitute for watching representative users perform tasks. An expert may identify an ambiguous label that no participant actually misreads, or may miss a problem that arises only from local terminology or an unfamiliar workflow. The two methods are complementary because they produce different forms of evidence.

A common heuristic set includes visibility of system status, correspondence with users' language, user control and freedom, consistency, error prevention, recognition rather than recall, flexibility, minimalist design, error diagnosis and recovery, and help. These are useful prompts rather than universal laws. For instance, "minimalist" does not mean removing explanatory text that a novice needs to avoid an irreversible action. "Consistency" does not require reproducing a harmful convention. A good evaluator connects a heuristic to a concrete user goal and explains the consequence of a violation.

Imagine a scholarship portal where a user uploads supporting documents. After clicking Submit, the page displays a spinner but never confirms whether files were received. This may violate visibility of status and error recovery. The inspection finding should identify the exact state, the missing feedback, and a plausible consequence: a user might retry and create a duplicate submission, or leave before completion. "The page is confusing" is too vague to support a fix. An actionable finding might recommend a distinct uploading state, an explicit server-confirmed success state, and a recoverable failure message that preserves selected files where technically possible.

Running a structured inspection

Define the evaluation scope before starting. List the supported devices, key tasks, and relevant interface states. Reviewers should examine ordinary paths and exceptional states: validation errors, empty results, slow responses, denied permissions, keyboard navigation, and session expiration. Ask evaluators to inspect independently before discussing findings. Independent passes reduce the risk that a dominant reviewer determines what everyone notices. If several evaluators review the same interface, record overlap and disagreement rather than counting duplicate reports as separate problems.

Each issue record should include a unique identifier, page or state, reproduction steps, observed behavior, relevant heuristic, expected user impact, confidence, and proposed investigation. A screenshot can help, but it should not replace a textual description that can be understood without vision. For a keyboard issue, note the exact focus sequence and key presses. For an error message, capture the message text and the user's preceding action. For a mobile issue, note viewport width and orientation. This level of detail allows a developer to reproduce the problem and an independent reviewer to challenge the interpretation.

Severity is a judgment, not a measurement. A frequently encountered typo might be annoying but recoverable; a rare control that deletes submitted work without confirmation could be critical. Consider impact, likelihood of encountering the state, persistence, recoverability, and the people affected. Separate severity from confidence. An evaluator may be highly confident that a missing label is a standards problem but uncertain how often users reach the page. Another issue may be observed often but have an unclear root cause. A single numerical severity score hides these distinctions; a short rationale is more useful.

Comparing inspection with accessibility conformance

Usability heuristics and accessibility standards answer overlapping but distinct questions. A reviewer might observe that a modal lacks a visible close button and label it as weak user control. A standards-oriented evaluation would additionally inspect keyboard access, focus trapping, accessible name, and programmatic relationships. A screen can appear understandable while remaining unusable with a screen reader. Conversely, a page can meet many technical success criteria and still use jargon that makes a task unnecessarily difficult. Neither type of evaluation establishes the other.

Automated accessibility tools are useful for detecting some structural problems, such as certain missing labels or contrast failures. They cannot determine whether a task flow makes sense, whether alternative text conveys the purpose of a complex chart, or whether an error explanation is helpful. W3C guidance recommends combining conformance evaluation with involvement of people with disabilities. Do not claim universal accessibility from a passing automated scan or a single successful participant session. Record the technologies and scenarios tested and the ones not tested.

Turning findings into design changes

An inspection report should prioritize hypotheses to test, not automatically prescribe a redesign. Suppose three reviewers flag a filter icon as obscure. The team could add a text label, move the control, or expose commonly used filters directly. Each option has different space and interaction costs. A quick prototype study can determine which option helps representative users. Where a safety or accessibility barrier is clear, immediate correction may be appropriate without waiting for a larger study. The important distinction is between a defect supported by direct technical evidence and a conjecture about how people will behave.

Avoid turning heuristics into a checklist of superficial compliance. A page may contain a progress indicator yet still fail to show whether the action is reversible. It may use consistent icons while consistently misleading users. It may have a help center that no one can find at the moment of failure. Evaluate complete journeys and transitions. A useful final inspection report explains how issues affect goals and what evidence would confirm that a proposed fix works. The objective is to improve interaction quality, not to maximize the number of violations recorded.

Match a metric to the decision

Usability is often discussed through effectiveness, efficiency, and satisfaction in a specified context of use. Effectiveness concerns whether users achieve their goals accurately and completely. Efficiency concerns resources spent relative to that achievement, frequently operationalized as task time or effort. Satisfaction concerns subjective experience. These dimensions are related but not interchangeable. A workflow may be fast because users abandon it early. A high satisfaction score may reflect attractive styling even when people submit incorrect data. A complete evaluation therefore chooses measures that fit the decision and interprets them together.

Task completion is a proportion: successes divided by eligible attempts. Define the denominator carefully. If twelve participants attempt one task and nine meet the predefined success criterion, the observed completion rate is 9/12 = 75 percent. If three participants never begin because the service is unavailable, excluding them may conceal a serious availability problem. For a study focused strictly on interaction after successful login, the exclusion might be justified, but it must be reported. Separate technical failure, participant withdrawal, moderator interruption, and task failure rather than silently merging them.

Time on task requires a start and stop rule. One study might stop timing when the correct answer appears; another might stop when the participant confirms it. These are not equivalent. Time distributions are often skewed because a few participants encounter long detours. A median can describe a typical observed time without being dominated by one outlier, while the mean reflects the total time cost in the sample. Report the sample size, spread, and handling of failures. A mean time calculated only among successful attempts can make a difficult design appear faster because slow unsuccessful attempts are excluded. Present success and time jointly, or use a predefined time-to-event method with explicit censoring rules when warranted.

Error counts can be informative if the categories are meaningful. A harmless extra tap is different from deleting a saved application. Classify slips, incorrect navigation, invalid submissions, and irreversible mistakes separately. A raw count of clicks is not a universal measure of efficiency: keyboard shortcuts, assistive technology, and expert workflows can produce different interaction counts for the same successful outcome. Define what an error means in relation to the task, not merely what differs from the designer's preferred route. Recovery success and the time needed to recover are often more actionable than the initial error count alone.

Satisfaction measures and interpretation

Self-report instruments can capture perceived ease, confidence, and workload, but they depend on wording and context. If a participant rates the task immediately after the moderator helped them, the score may reflect gratitude rather than interface quality. Use consistent question wording and timing. A single unvalidated item such as "Was this easy?" can be useful for exploratory feedback, but it should not be described as a validated psychometric scale. If a standardized questionnaire is used, follow its scoring procedure, language validation, and licensing conditions. Report scores with their possible range so readers know whether a number is high or low.

Qualitative comments should accompany numerical summaries. Two users can assign the same difficulty rating for different reasons: one struggled with labels, another feared losing their work. If the team sees only the mean rating, it cannot identify the appropriate intervention. Quote short remarks with participant consent and remove identifying details. Do not cherry-pick favorable comments. An evidence table that includes both successful and unsuccessful journeys is more credible than a slide showing only a single average and a decorative testimonial.

Measurement uncertainty

A sample proportion is not the population proportion. With 9 successes in 12 attempts, 75 percent describes the observed sample, but sampling variability is substantial. Confidence intervals can illustrate uncertainty when participants are reasonably treated as independent observations drawn under a suitable sampling design. They cannot repair biased recruitment, inconsistent task scripts, or a convenience sample that excludes key user groups. Avoid treating an interval as a guarantee that a population parameter lies within it. If the sampling assumptions do not hold, describe the result as exploratory and focus on observed mechanisms.

A binomial model is a useful teaching abstraction when each attempt is independent and has the same success probability. Under that model, the standard error of a sample proportion is approximately sqrt(p(1-p)/n), though this normal approximation performs poorly for small samples or proportions near zero or one. Wilson intervals generally behave better in those cases. A unit-level simulation can show how interval width changes with the number of attempts while explicitly warning that the model does not capture recruitment bias, learning, or clustering. Statistical formulas should illuminate assumptions, not make an observational convenience sample look like a randomized trial.

For repeated tasks by the same participant, independence is especially questionable. If one person attempts ten tasks, those outcomes share familiarity, motivation, and device conditions. Treating them as ten independent people exaggerates the information available. A task-by-person table preserves the grouping. For a formal analysis, a paired or hierarchical approach may be needed. For a formative report, showing each participant's trajectory and the recurring breakdowns may be more informative than a pooled success percentage. The analysis plan should follow the data structure rather than forcing every observation into one denominator.

Baselines, targets, and dashboards

A usability target should be specific about population, task, and conditions. "At least 90 percent of first-time applicants can upload a complete document set on a midrange phone without assistance" is more interpretable than "make the portal user friendly." Even this target needs an agreed definition of a complete set, device conditions, and an evidence plan. A target chosen after seeing the data is not an independent acceptance criterion. Document baseline performance and the reason the target matters to users or operations.

A dashboard can be misleading when it mixes sessions, tasks, and users. One person might retry an action five times, creating five events but only one affected user. An analytics funnel may report a high conversion rate while excluding visitors who cannot load the page. Instrumentation should include meaningful events and distinguish intentional abandonment from technical failure where possible. Privacy matters: collect only the events needed for the decision, avoid unnecessary identifiers, and apply a retention policy. Instrumentation quality is part of measurement quality; a precise chart of incorrectly defined events is not evidence.

Reading improvements critically

Suppose a redesigned page produces 18 successes in 20 observed attempts, compared with 15 in 20 on the earlier design. The descriptive difference is 90 percent minus 75 percent, or 15 percentage points. That does not establish that the redesign caused the improvement. The groups may differ, task difficulty may have changed, and participants may have learned between conditions. A controlled comparison can address some confounds, but it still requires enough observations and a design suited to the decision. Report absolute differences and uncertainty rather than declaring victory based on a percentage alone.

When a measure improves, check what worsened. A faster form might achieve speed by omitting a confirmation step that prevents costly errors. A shorter help page might reduce reading time while increasing support requests. Define guardrail measures for high-consequence failures, accessibility barriers, and user trust. Metrics are tools for investigating tradeoffs. They become dangerous when a team optimizes a convenient number instead of the user's actual outcome.

From observation to causal comparison

A task study can reveal that users struggle with an interface; an experiment asks whether a specified change causes a difference under controlled conditions. The question must name the intervention, outcome, population, and context. "Does an explicit Save draft confirmation reduce duplicate submissions among first-time scholarship applicants?" is testable. "Is design B better?" is not, because better has no defined outcome and design B may differ in many ways. State the hypothesis before collecting data and decide which outcome would justify adopting the change.

The independent variable is what the researcher deliberately changes, such as label wording or the presence of a confirmation. Dependent variables are observed outcomes, such as task success, duplicate submission, time, and reported confidence. Confounds are other factors that vary with the intervention and could explain the outcome. If one design is tested on desktop computers in a quiet room and the other on phones outdoors, device and environment are entangled with design. A valid comparison must either hold those factors reasonably constant, randomize them, or model them with sufficient data and justified assumptions.

Between-subjects and within-subjects designs

In a between-subjects comparison, different participants see different variants. This can reduce learning transfer between variants, but differences in participant ability may obscure the effect. Random assignment helps balance both measured and unmeasured factors on average; it does not guarantee identical groups in a small sample. In a within-subjects comparison, each participant uses both variants. This can reduce person-to-person variability, but the first experience may teach skills that affect the second. Counterbalancing order, using equivalent task materials, and providing appropriate breaks can reduce some order effects without eliminating them.

Consider two versions of a university course-registration flow. If the same participant completes the identical registration task on version A and then version B, the second attempt benefits from knowledge of course codes, requirements, and deadlines. A shorter time on B may reflect practice rather than design. One response is to create two comparable course scenarios and alternate the order of interface versions and scenarios. Another is to use separate groups and random assignment. The choice depends on recruitment capacity, expected learning effects, and whether the tasks can be made comparable. The experimental design is part of the argument, not a technical footnote.

Randomization, masking, and preregistration

Randomization is useful because it makes treatment assignment independent of participant characteristics in expectation. For a small study, record the actual allocation and check whether an obvious imbalance occurred. Avoid repeatedly reshuffling groups until the result looks favorable. Masking can be difficult in interface studies because participants see the designs, but outcome coding can sometimes be performed by a reviewer unaware of the assigned condition. Standardized task instructions and success rules reduce opportunities for the moderator to influence results. Researchers should disclose their involvement in creating the variant they evaluate.

Write a brief analysis plan before testing: primary outcome, unit of analysis, exclusions, missing-data treatment, planned comparison, and decision threshold. This does not need to be a formal public preregistration for every formative test. Its purpose is to prevent a team from searching across many outcomes and reporting only the most flattering one. If exploratory findings emerge, label them exploratory. A post hoc discovery may inspire the next study, but it should not be presented as if it were the original confirmatory hypothesis.

Power, sample size, and practical significance

A small experiment can fail to detect a meaningful effect because its estimates are noisy. Statistical power depends on the expected effect, variability, sample size, significance threshold, and design. No universal sample-size rule applies to every interface comparison. Before recruiting, estimate a plausible effect that would matter operationally and determine whether the available sample could distinguish it from noise. If not, use the study for formative learning rather than presenting a non-significant result as proof that two designs are equivalent.

Practical significance is not identical to statistical significance. A one-second reduction in an infrequent low-stakes task may not justify a complex redesign, even if a large study finds a precise effect. A small reduction in accidental deletion could matter greatly because the consequence is severe. Conversely, a statistically uncertain but repeated critical failure in a small study may justify immediate mitigation. Evaluate consequence, implementation cost, and uncertainty together. The decision is ultimately about users and risks, not whether a single p-value crosses a conventional threshold.

A/B testing in deployed systems

An online A/B test randomly assigns eligible users to interface variants and observes predefined outcomes. It can provide evidence under real usage conditions, but it introduces operational constraints. Assignment should be stable for a user when repeated exposure matters. Logging must distinguish users from sessions and account for bot traffic, duplicate events, and missing data. If variants change loading speed or accessibility, the experiment may affect who is able to participate at all. Monitor guardrails such as error rate, support contacts, and task abandonment rather than only click-through rate.

Experimentation is not ethically neutral. A change that increases engagement by concealing cancellation or making privacy settings difficult is not a success. High-stakes services may require review before exposing users to experimental variants. Inform users appropriately, minimize collected data, and avoid experiments that impose disproportionate harm on a vulnerable group. If an accessibility barrier is already known, withholding a fix merely to estimate its conversion effect is difficult to justify. The ability to randomize does not create permission to test every intervention.

Interpretation and replication

Suppose 30 of 40 participants complete a task on design A and 34 of 40 on design B. The observed proportions are 75 and 85 percent, a 10 percentage-point difference. Without an appropriate design and uncertainty analysis, this is a descriptive difference, not a proven causal improvement. Even with random assignment, chance variation remains. Report confidence intervals or another suitable uncertainty summary and describe the study's limitations. Avoid announcing that B is "13.3 percent better" without specifying that 13.3 percent is the relative change in observed completion, not a universal quality score.

Replication matters because interface effects can depend on language, device, task complexity, and user expertise. A label that works in one language may be ambiguous in another. A compact navigation layout that helps desktop experts may hinder mobile beginners. Repeating the study in another setting can test whether the mechanism generalizes. Preserve the task scripts, anonymized analysis data where consent permits, interface versions, and outcome definitions. If results conflict, investigate differences in context rather than discarding the less convenient study.

Making an evidence-based decision

A useful experimental report ends with a decision record. State the intervention, sample, randomization or counterbalancing procedure, primary outcome, observed effect, uncertainty, guardrail results, limitations, and recommended next action. Possible actions include adopting the change, collecting more evidence, revising the design, or rejecting the hypothesis. Record unresolved risks explicitly. An experiment can reduce uncertainty without eliminating it, and a responsible decision may still involve judgment about costs and consequences.

Across task studies, expert inspection, metrics, and experiments, the central discipline is the same: match the strength of the claim to the strength of the evidence. A usability observation is not a population estimate. A heuristic violation is not automatically a measured failure rate. A dashboard trend is not necessarily causal. A randomized result may be precise for one population and irrelevant to another. The strongest evaluation program uses these methods together, preserves negative findings, and treats real user outcomes as more important than impressive-looking numbers.