Suppose that you want to demonstrate that baby boomers are more narcissistic than other generations, or that women are more agreeable and neurotic than men, or that people of different races have different amounts of intelligence. How do psychologists do that? Can they in fact do that?
Typically, the method is to come up with a bunch of questions that superficially appear to measure the intended characteristic. Then the questions are “validated”, for example, by making sure the questions all correlate with one another. Once the questionnaire is declared valid, psychologists can then measure a variety of different groups and make far-reaching claims about how our current political/social situation was caused all along by the thing that they happen to study.
If you find this methodology questionable, but aren’t sure exactly what went wrong, you might be interested in hearing about psychometrics, the field concerned with psychological measurement. According to psychometricians, part of the problem is that psychologists are failing to follow best practices. That is the subject of this paper: