How To Evaluate Bad Statistics and Research Methods: Home

You don't have to learn a lot of math to be able to tell when you're being presented with the real deal or number salad. You just need to know what clues to look for.

Hypothesis

Was the hypothesis falsifiable? If something can't be phrased as a statement that can be proven or disproven using observation and deduction, you can't study it scientifically. Matters of opinion and belief are not falsifiable.

Sample/Population

What population is being studied? How big is it? What kind of individuals, and in what proportions?

How big is the sample? Of the population, how many did the researchers pick for their study?

How did the researchers select their sample? The distribution of variables in the sample should reflect the distribution of variables in the population as a whole. Random selection is one way to do that.

Experimental Methods

Did the researchers test for only one experimental variable at a time? This is so it's possible to know that the results are due to the variable being studied, and not something else.

Did the researchers have a control group? A control is a group that is not exposed to the experimental variable. Again, this so that the researchers can demonstrate that it was the experimental variable that had the effect, and not just chance. For example, when testing the efficacy of a drug, the control group receives a placebo.

Did the researchers control for all the variables so that it's possible to be sure the results were as a consequence of the variables the researchers pinpointed? An example is that drug might have a different effect on a 150 pound woman than on a 250 pound man.

Was it a double blind test? Sometimes this is impossible in practice. A double-blind test ensures that neither the researchers nor the research subjects know who is getting the experimental variable. That way they cannot deliberately or unconsciously influence the outcome of the experiment.

Do These Numbers Mean Anything?

Statistical significance

Calculates the likelihood (in terms of a percentage) that the results obtained were because of what the researcher said, and not just chance.

Plus or minus (a.k.a., confidence intervals, margins of error)

Measures of certainty and uncertainty. A big plus or minus number means more uncertainty, but compare it to the size of the base number - 100 plus or minus 1 is a lot more certain than 2 plus or minus 1!

Other Questions to Ask

• If you're comparing two statistics, are they measuring the same thing? Are they using the same methods and instruments? Are they defining that thing in the same way?
• If you get some really weird numbers (technically called " outliers"), are you including them or throwing them away?

If the answers to these questions are not readily available, you may begin to suspect that the researcher (or writer) is either unscrupulous or incompetent. Bear in mind that one of the things that makes social sciences research so challenging is that often you simply can't guarantee perfectly random samples or perfectly controlled variables. But a good social sciences report will always address the questions raised by those omissions.

• Search our FAQ or fill out the contact form

• Email:
  librarian@sunyempire.edu

• Call 800-847-3000, ext. 2222