12 Types of Bias in Statistics (and How to Catch Them)

The types of bias in statistics are the systematic ways a study, survey, or dataset can drift away from the truth it’s trying to measure — not by chance, but in a predictable direction every time. Twelve types show up most often in practice: selection, sampling, non-response, response, recall, measurement, observer, confirmation, publication, survivorship, attrition, and omitted-variable bias. They strike at different points — while you’re building a sample, while you’re collecting data, or while you’re analyzing and reporting it — and more than one can hit the same study at once. None of them are fixed by a bigger sample size; all of them are fixed, or at least caught, by how the study is designed and documented.

Key takeaways

Point	Details
Bias is systematic, not random	A biased estimate is wrong in the same direction every time, unlike random error, which shrinks as sample size grows but leaves bias untouched.
12 types cluster into four stages	Selection, sampling, non-response, response, and recall bias strike while you build and collect the sample; measurement and observer bias strike while you record data; confirmation, publication, and omitted-variable bias strike during analysis and reporting; survivorship and attrition bias strike as the sample erodes over time.
Bias can flip the answer, not just shift it	Early observational research found hormone replacement therapy lowered heart disease risk; later, more rigorous trials found the opposite effect once that research corrected for bias.
Response rate is a built-in warning flag	The U.S. Census Bureau's statistical quality standards require a formal nonresponse bias analysis once a survey's unit response rate drops below 80 percent.
Prevention beats correction	Pre-registering a study, blinding whoever collects the data, and tracking dropouts as they happen catch most bias before it ever reaches your dataset.

Quick Checklist: Is Your Study or Dataset at Risk of Bias?

Run through these six questions before you trust a comparison, a survey result, or a regression coefficient. Each one maps to a cluster of the 12 bias types below.

Quick checklist: is your study or dataset at risk of bias?

• What was the response rate, and did it differ by group? A rate under roughly 80 percent, or a big gap between groups, is the Census Bureau's own trigger for a nonresponse bias check.
• Who is missing, and why? Dropouts, non-responders, and anyone excluded for "data quality" reasons are rarely a random subset of the original sample.
• Are the measurements self-reported or subjective? Self-report and memory-based data invite recall bias and social-desirability-driven response bias.
• Did the person collecting or scoring data know the group or outcome? Unblinded data collection is the textbook setup for observer bias to creep in.
• Was the study or analysis plan registered before the results came in? Pre-registration is the standard defense against both confirmation bias and publication bias.
• Could an unmeasured variable explain both sides of the comparison? If so, you're looking at a candidate for omitted-variable bias, not a clean causal effect.

Treat a “no” on any of these as a reason to qualify your conclusion, not a reason to throw the data out.

What Is Statistical Bias?

Statistical bias is a systematic difference between what a measurement, estimate, or sample shows and the true value it’s supposed to represent. The U.S. National Institute of Standards and Technology’s statistics handbook defines bias as the difference between a measurement result and its unknown true value (NIST/SEMATECH e-Handbook of Statistical Methods).

That’s a different problem from random error, and the distinction matters more than any single bias type. A peer-reviewed review of bias in clinical research spells it out: unlike random error, which comes from sampling variability and shrinks as sample size increases, bias is independent of both sample size and statistical significance, and in extreme cases it can produce an association that points in the opposite direction of the truth (Pannucci & Wilkins, Plastic and Reconstructive Surgery). That paper’s clearest example is hormone replacement therapy (HRT): multiple observational studies conducted before 1998 found that HRT lowered heart disease risk in postmenopausal women, but later trials designed specifically to minimize bias found the opposite — HRT increased that risk. The bias in the earlier studies wasn’t a rounding error; it reversed the conclusion entirely.

What Are Selection Bias and the Main Sampling Bias Types?

Selection bias occurs when the way units enter a study makes the resulting sample systematically unlike the population it’s meant to represent. The current Cochrane risk-of-bias tool for randomized trials names this “bias arising from the randomization process” as one of its five formal domains, because a flawed allocation sequence or broken concealment can let healthier or more motivated participants cluster in one arm before the study even starts (Cochrane Handbook, Chapter 8). Pannucci and Wilkins describe a related version they call channeling bias, where a patient’s prognosis or degree of illness — not random assignment — determines which study arm they end up in; in surgical studies, they note, surgeons may be more aggressive about operating on young, healthy patients with low perioperative risk while tolerating imperfect outcomes in older, higher-risk patients. They illustrate the effect with a hypothetical: picture a retrospective study of operative versus non-operative management of hand fractures, where young patients are channeled into the operative cohort and older patients into the non-operative cohort before either arm has produced a single outcome — the comparison is biased from the moment patients are assigned, not from anything that happens afterward.

Sampling bias is the related, broader problem of a sampling method itself favoring certain members of the population — it’s the mechanism, while selection bias is often the result. The main sampling bias types are:

• Convenience bias — sampling whoever is easiest to reach (the first 50 people in a mall) rather than a representative cross-section.
• Undercoverage bias — the sampling frame leaves out part of the population, such as a phone survey that only reaches people with landlines.
• Voluntary-response bias — only people with strong opinions bother to respond, such as an online poll anyone can opt into.

Pew Research Center’s American Trends Panel shows undercoverage bias being corrected in practice, not just defined on paper: panel members who lack home internet access can take Pew’s surveys on internet-enabled tablets the Center provides, a direct hedge against the coverage error that excluding offline households from an online-only sampling frame would otherwise introduce (Pew Research Center, U.S. Survey Methodology).

Introductory statistics texts group these under the broader lesson that how a sample is drawn determines whether it can represent the population at all, independent of how large it is (OpenStax, Introductory Statistics 2e, §1.2). Statohub’s sampling methods guide covers how to pick a probability method that avoids these problems by design; this article focuses on what happens when that choice goes wrong.

What Are Non-Response Bias and Response Bias?

Non-response bias shows up when the people who don’t answer a survey differ systematically from the people who do, so the responses you collect no longer represent the full sample you drew. The U.S. Census Bureau’s statistical quality standards require a formal nonresponse bias analysis once a survey’s unit response rate falls below 80 percent, or its item or total-quantity response rates fall below 70 percent — a concrete, government-set line for when non-response is likely to matter (U.S. Census Bureau, Statistical Quality Standards). Pew Research Center’s own survey methodology documentation treats nonresponse error as one of the core components of total survey error, alongside coverage, sampling, measurement, and processing error, which is why Pew builds its surveys around minimizing all of them together rather than nonresponse alone (Pew Research Center, U.S. Survey Methodology). Response rates to Pew’s own telephone polls dropped to 7 percent in 2017 and 6 percent in 2018, yet Pew notes that a low response rate alone doesn’t make a poll inaccurate. The real risk is when the outcome being measured is related to who responds: polls measuring volunteering overstate it, because volunteers are more likely to take surveys in the first place, and standard demographic weighting doesn’t fully correct for that gap (Pew Research Center, Phone survey response rates decline again).

Response bias is different: it’s not about who answers, but how honestly or accurately they answer once they do. It covers social-desirability answers (rounding up healthy habits, rounding down bad ones), leading or loaded question wording, and an interviewer’s own influence on what a respondent says. Pannucci and Wilkins describe this last form as interviewer bias — a systematic difference in how information is solicited or recorded — and give a concrete example: an interviewer who knows a patient has Buerger’s disease may probe harder for a smoking history (“Are you sure you’ve never smoked? Never? Not even once?”) than they would with a control patient, inflating the apparent association between smoking and the disease.

Recall bias is a specific, common form of response bias: a respondent’s memory of past events gets colored by what they know now. Pannucci and Wilkins point to the much-discussed, since-discredited apparent link between the MMR vaccine and autism as a textbook case — parents of children later diagnosed with autism were more likely to recall the vaccine’s timing specifically because it coincided with a developmental regression they were already watching for, not because the vaccine caused it.

What Are Measurement Bias and Observer Bias?

Measurement bias is a systematic error introduced by the instrument or procedure used to collect data, rather than by who’s in the sample. NIST’s handbook lists concrete sources of measurement bias in a gauge or instrument: a lack of resolution, non-linearity across its range, drift over time, hysteresis, and differences between gauges, geometries, or operators — any of which can shift every reading in the same direction. Pannucci and Wilkins give a clinical example of the same problem: physical exam alone correctly diagnoses venous thromboembolism in under half of true cases, which is why it’s an inappropriate primary measure next to an objective test like duplex ultrasound.

Observer bias is measurement bias caused specifically by the person doing the observing or rating, rather than the instrument itself. It’s at its worst when the observer knows which group a subject belongs to, because that knowledge can unconsciously shift how borderline cases get scored. Blinding — withholding group assignment from whoever is recording or evaluating outcomes — is the standard fix, which is also why “bias in measurement of the outcome” is its own formal domain in the current Cochrane risk-of-bias tool for trials (Cochrane Handbook, Chapter 8). Pannucci and Wilkins illustrate that fix with a hypothetical: a study comparing lag-screw versus plate-and-screw fixation of hand fractures could standardize the surgical approach and have functional outcomes scored by a blinded examiner who had never viewed the operative notes or x-rays, so the rating couldn’t be colored by knowing which technique a patient received. Statohub’s guide to Cohen’s kappa covers how to quantify agreement between raters directly, which is the main way observer bias gets caught rather than just assumed away.

What Are Confirmation Bias and Publication Bias?

Confirmation bias is the tendency to notice, trust, and recall evidence that supports a belief you already hold, while discounting evidence that contradicts it. A CIA Center for the Study of Intelligence book on analytic reasoning traces this to a classic experiment: given the sequence 2-4-6 and asked to discover the rule generating it, most participants tested only examples consistent with their first guess instead of trying to disprove it. In the original run, only 6 of 29 participants guessed the correct rule on their first attempt; when a different researcher repeated the task, none of 51 participants did (Heuer, Psychology of Intelligence Analysis, CIA Center for the Study of Intelligence). In data analysis, the same pattern shows up as stopping a hypothesis search the moment a test confirms the expected answer, instead of seriously trying to rule it out.

Publication bias is confirmation bias at the level of an entire field: positive, “interesting” results are more likely to get written up and accepted than null or unfavorable ones, which skews the published literature toward overstating an effect. Pannucci and Wilkins call this citation bias and point to the field’s main fix — in 2004 the International Committee of Medical Journal Editors required trials to be registered with a public registry before results could be published, and a 2007 update required that registration happen before patient enrollment even begins, so a study can’t quietly disappear just because its results were disappointing. The current Cochrane Handbook still treats missing results from unregistered or unpublished studies as a distinct source of bias in any synthesis across studies (Cochrane Handbook, Chapter 7).

What Are Survivorship Bias and Attrition Bias?

Survivorship bias occurs when an analysis only includes the cases that made it to a later stage, silently dropping the cases that didn’t — which makes whatever got you to that stage look safer or more effective than it really is. A 2026 letter in Pediatric Transplantation raises exactly this concern about a published meta-analysis of pre-transplant ventricular assist device (VAD) support in children awaiting heart transplant. The letter’s authors note that the meta-analysis’s survival comparison “was limited to patients who survived to transplantation,” excluding VAD patients who died on the waitlist from device complications or other causes, which left what they call a “selected, biologically favorable subset” of VAD patients being compared against a more diverse non-VAD cohort (Ishtiaq, Malik & Ahmad, Pediatric Transplantation).

Attrition bias is the mechanism that often creates survivorship bias: participants drop out of a study unevenly across groups, and the ones who leave are rarely a random subset of the ones who stay. The current Cochrane tool calls this “bias due to missing outcome data” and treats it as a standalone risk-of-bias domain precisely because the reason people drop out is so often tied to the outcome itself — patients who aren’t improving are more likely to stop showing up for follow-up visits than patients who are. Pannucci and Wilkins illustrate this under the name transfer bias with a hypothetical: consider a study comparing inferior-pedicle Wise-pattern versus vertical-scar breast reduction techniques. Because Wise-pattern patients often have fewer post-operative contour problems, they may be less likely to return for long-term follow-up, while patients in the vertical-reduction group — more concerned about resolving skin redundancies — may be more likely to come back for evaluation, producing unequal, non-random loss to follow-up between the two groups. Watching for unequal, differential dropout between groups — not just a shrinking total sample — is how you catch it before it reaches your final numbers.

What Is Omitted-Variable Bias in Regression?

Omitted-variable bias occurs when a regression leaves out a variable that affects the outcome and is also correlated with a variable that’s still in the model, so the missing variable’s effect leaks into the coefficient on the one you kept. Pannucci and Wilkins describe the same underlying problem under the name confounding — their examples include the well-known association between coffee drinking and heart attack, which is confounded by smoking, and the association between income and health status, confounded by access to care.

The leakage follows a simple rule. If the true relationship is Score = β₀ + β₁·Hours + β₂·GPA + error, but a regression omits GPA, the coefficient on Hours that comes out of the shorter regression is biased by β₂ multiplied by the slope of GPA on Hours:

True model:      Score = 50 + 5·Hours + 10·GPA + error
Omitted model:   Score = a + b·Hours + error   (GPA left out)

If regressing GPA on Hours gives: GPA ≈ a' + 0.3·Hours
Then the expected bias in b is: β₂ × 0.3 = 10 × 0.3 = 3

Expected naive estimate:  b ≈ β₁ + bias = 5 + 3 = 8

A regression that only has Hours in it would report that each additional hour of studying is worth roughly 8 points — 60 percent more than the true effect of 5 — because part of GPA’s real effect on Score gets misattributed to Hours, since the two are correlated. Statohub’s guide to controlling for confounders covers how to add the missing variable back in or use a design (matching, stratification, randomization) that controls for it directly.

A Worked Example: Comparing Weight Loss Between a Free and Premium App Tier

Picture a fitness app comparing average self-reported weight loss between its free and premium tiers after a 12-week program, using an end-of-program survey.

1. Check attrition first. Premium enrolled 650 users; 500 were still active subscribers at week 12 (150 lost, or 150 ÷ 650 ≈ 23.1% attrition). Free enrolled 900 users; only 500 were still active at week 12 (400 lost, or 400 ÷ 900 ≈ 44.4% attrition). Free lost nearly twice the share of its starting group — and the people most likely to quit a free weight-loss program early are the ones for whom it isn’t working, which is attrition bias feeding directly into survivorship bias in whichever users remain.
2. Check the response rate among survivors. Of the 500 still-active users in each tier who were invited to the survey, 320 premium users responded (320 ÷ 500 = 64%) versus 80 free users (80 ÷ 500 = 16%). Both rates sit well below the Census Bureau’s 80% threshold for requiring a nonresponse bias check, and the four-to-one gap between tiers means the two groups of respondents aren’t comparable to begin with.
3. Question the measurement. The reported outcome — “average weight loss” — comes from self-report, not a connected scale. Premium users who paid for the service have an extra incentive to round their results up (response bias), and anyone recalling a 12-week-old starting weight is subject to recall bias.
4. Decide. The raw numbers show premium responders reporting 12 lbs lost versus 9 lbs for free responders — a 33% relative gap ((12 − 9) ÷ 9 ≈ 33.3%). After steps 1 through 3, that gap can’t be read as “premium works better”: it could just as easily be explained by which users survived to be asked, which users bothered to answer, and how generously they remembered their own results. The fix is to track outcomes with objective data (a connected scale) for everyone enrolled, including dropouts counted as non-responders rather than silently removed, ideally inside a randomized comparison rather than an opt-in survey.

Common Mistakes to Avoid When Checking for Bias

• Assuming a bigger sample fixes bias. It doesn’t. Bias is a systematic error independent of sample size; a larger biased sample just gives you a more confident, equally wrong answer.
• Treating a decent overall response rate as safe. A combined response rate can look fine while hiding a large gap between subgroups — check response rates by group, not just overall, the way the worked example above does.
• Letting unblinded staff both collect and judge outcomes. This is the direct setup for observer bias; separate who assigns groups from who scores results whenever the budget allows it.
• Dropping dropouts from the denominator. Counting only people who finished a study, rather than everyone who started it, converts ordinary attrition into survivorship bias in your headline number.
• Declaring a study “unbiased” after ruling out one type. The 12 types aren’t mutually exclusive — a single flawed survey can carry selection, non-response, and response bias at the same time, as the worked example shows.

Statohub’s Take on Types of Bias in Statistics

Statohub’s position is that bias is a design problem before it’s ever a statistics problem — no test run after the fact substitutes for randomization, blinding, and a pre-registered plan decided in advance. Treat each of the 12 types as a checklist item during study design, not a label to apply retroactively once a result looks surprising. When a correction after the fact is genuinely the only option, say so plainly in how the result is reported, rather than letting a clean-looking number stand in for a clean study.

Put These Checks to Work With Statohub’s Tools

Spotting bias starts with understanding the sample it came from. Statohub’s sampling methods guide covers how to choose a probability-based method that heads off selection and sampling bias before data collection even starts, and the reliability guide explains how to measure and report inter-rater agreement to catch observer bias directly. For the regression side of things, controlling for confounders walks through adding a left-out variable back into a model instead of just flagging that it’s missing.

When you’re ready to plan a survey or trial with the response rate in mind, Statohub’s sample size calculator helps you set a target sample size large enough to still be useful after expected non-response whittles it down. The Foundations hub rounds up the rest of the notation this guide assumes, and the calculators hub has every other tool for checking a dataset once bias has been ruled out.

Sources

Sources

1. Pannucci CJ, Wilkins EG — "Identifying and Avoiding Bias in Research," Plastic and Reconstructive Surgery (2010) PMC / NCBI
2. NIST/SEMATECH e-Handbook of Statistical Methods — 2.4.5. Analysis of Bias NIST
3. Cochrane Handbook for Systematic Reviews of Interventions — Chapter 8: Assessing Risk of Bias in a Randomized Trial Cochrane
4. Cochrane Handbook for Systematic Reviews of Interventions — Chapter 7: Considering Bias and Conflicts of Interest Among the Included Studies Cochrane
5. U.S. Census Bureau — Statistical Quality Standards (April 2023) U.S. Census Bureau
6. Pew Research Center — U.S. Survey Methodology Pew Research Center
7. Pew Research Center — "Phone survey response rates decline again" (2019) Pew Research Center
8. OpenStax — "1.2 Data, Sampling, and Variation in Data and Sampling," Introductory Statistics 2e OpenStax / Rice University
9. Heuer, Richards J. Jr. — Psychology of Intelligence Analysis CIA Center for the Study of Intelligence
10. Ishtiaq S, Malik ZM, Ahmad AR — "Survivorship Bias in the Shadows: A Methodological Concern Regarding Post-Transplant Survival Analysis in Pediatric VAD Recipients," Pediatric Transplantation (2026) PMC / NCBI

FAQ

Frequently asked questions

What Is the Difference Between Sampling Bias and Selection Bias?

Sampling bias is a property of the sampling method itself — convenience, undercoverage, or voluntary-response sampling all favor certain members of the population before anyone is assigned to a group. Selection bias is the broader result: any systematic difference between who ends up in a study (or in a particular study arm) and who the study was meant to represent, whether that difference came from the sampling method, non-random group assignment, or how participants were recruited.

What Is Response Bias in a Survey?

Response bias happens when respondents answer inaccurately, not when they fail to answer at all. Common sources include social-desirability pressure (rounding healthy behaviors up and unhealthy ones down), leading question wording, an interviewer's influence on what gets said, and recall bias when the question asks about the past. It's distinct from non-response bias, which is about who skips the survey rather than how the people who do respond answer.

How Do You Know If Your Sample Has Nonresponse Bias?

Start by comparing your response rate against the U.S. Census Bureau's own trigger: a unit response rate under 80 percent, or item and total-quantity response rates under 70 percent, is the government's own line for requiring a formal nonresponse bias analysis. More telling than the overall number is whether the response rate differs by subgroup — a large gap between groups, the way premium and free users differed in the worked example above, is a stronger warning sign than a merely low overall rate.

What's the Difference Between Bias and Random Error in Statistics?

Random error is noise that scatters estimates above and below the true value and shrinks, on average, as sample size grows. Statistical bias is a systematic error that pushes an estimate in the same direction every time, and it does not shrink with a larger sample — a bigger biased dataset simply produces a more confident wrong answer, which is why fixing bias is a design question rather than a sample-size question.

Can a Large Sample Size Fix Statistical Bias?

No. A larger sample reduces random sampling variability, but it has no effect on systematic bias, because every additional observation is pulled in the same biased direction as the ones before it. The only real fixes are methodological: randomize allocation, use a proper probability sampling frame, blind observers, track and report response and attrition rates by group, and pre-register the analysis plan.

What Is Survivorship Bias in Plain Terms?

Survivorship bias is drawing a conclusion from only the cases that made it to a later stage of a process, while the cases that didn't make it are invisible to the analysis. A mutual fund's track record that only counts funds still operating today looks better than reality because the funds that failed and shut down have been dropped from the comparison entirely; the same mechanism applies to any study that only analyzes outcomes for people who stayed in it.

Recommended

• Statistics Symbols Cheat Sheet
• Randomized Controlled Trials: Design & Analysis
• Reliability in Statistics: Cronbach’s Alpha & Cohen’s Kappa
• Sample Size Calculator