Statistics - not the trump card you think

People throw around statistics like they're strong evidence, but they're not. Most times I see someone use a statistical argument online, I shake my head and scroll down to the next post. Here's why.

Statistics have a trust problem

Most statistics are from sources that agree with what the arguer is using them to argue for. This raises an obvious question: why should we take their word that the study was even done as claimed and the results reported honestly? Even when people acknowledge that lying is a thing, they seem to have the idea that "Prove It" is only for your direct opponent in an argument, and if they cite an external source, you have to trust that source even if it has the same stake in the argument as the person you're talking to. How do you know they didn't lie about the study or its results? In almost every case, you don't.

The Invincible Lie

Without the ability to check it yourself, the only way to get reliable statistics is to get them from a source that doesn't have a motive to lie in the relevant direction. If the NRA says they did a study that found areas with higher civilian gun ownership have higher crime rates, but argues that the constitution is more important than reducing deaths, then they're probably telling the truth about the study. Not so if the CSGV says it or if the NRA says the opposite.

Even when you get a statistic from someone who doesn't agree with the positions it seems to support, there's a good chance they got it from someone who does, so that should also be checked.

Real-world situations usually have too many variables for statistics to be useful

Obviously, to use statistical data to establish that A and B are related you have to isolate A and B in your comparison. It won't do to sample a bunch of red squares and blue spheres and draw a conclusion about a difference between red and blue objects. But this is exactly what many statistical argments are.

I've argued with Jordan Peterson acolytes about whether there are intrinsic psychological differences between men and women. The acolyte cited Peterson as his source for a study showing that in cultures that try to treat men and women as if they don't have intrinsic psychological differences, the differences become more extreme. I pointed out that even if his claim about the study was accurate, and *even if* we ignore the fact there are few if any cultures that don't more or less share American ideas about this topic, assume they examined many different such cultures, and ignore the effect that contact with other cultures that do have gender roles would've had on them, it's *still* a bad argument because the way a culture treats gender is never anything close to the only difference between it and another culture. A real society has *thousands* too many variables for this to be a reasonable argument. But he got mad at me for refusing to treat his "study" as instant proof of his claim and wouldn't continue the debate.

Statistics often measure the wrong thing

I've seen people argue for gun confiscation saying that America has more gun homicides per year than a bunch of other (smaller) countries who have stricter anti-gun laws... only to find out that they were measuring the gun death rate *as a flat number* rather than a proportion of each country's population. Of course America will have more gun homicides if you measure it that way because America has more people than those other countries.

(Not to mention that argument is atrocious anyway for, again, ignoring the thousands of other variables involved.)

Similar common fallacies to that conclusion include counting suicides and defensive killings of criminals in one country but not in the other, etc.

Who made the judgements?

This doesn't apply to all statistics, but for ones about things that not everyone can agree on the exact definition of, you run into an even bigger problem than any of the above. Example: a statistic that "70% of sexual harassers are male". Even if you can establish the reliability of the source and the relevance of the figure, you have to ask who decided which cases count as sexual harassment. In all likelihood they had different ideas of what that means than you or your opponent do.

Statistics can have ambiguous implications.

Even if you prove there's a non-coincidental correlation between A and B, that doesn't prove that A is the cause of B. You still have to consider the possibility that B is the cause of A, or that a certain C that wasn't considered in the study causes both B and A, or that a combination of A and C is required to cause B.


subscribe via RSS

Proxied content from gemini://yujiri.xyz/argument/statistics.gmi

Gemini request details:

Original URL
Status code
text/gemini; lang=en
Proxied by

Be advised that no attempt was made to verify the remote SSL certificate.

What is Gemini?