Glider from the game of Life, rising from the left




Topic: #Likert-scales

The Latest Factoids on College Career Preparation


A polling organization that probably knows better has reported the results of a recent survey of college students to find out how they feel about finding a job after graduation. The head pollster for the project concluded that “students are not nearly as prepared as they could or should be, and they actually know it while they're in college.” The survey asked “more than 32,500 students from 43 randomly selected four-year institutions, both public and private” to express their level of agreement (on a five-position Likert scale), with a few canned assertions, such as “I am confident that I will graduate with the knowledge and skills I need to be successful in the job market.”

“Unprepared and Confused”
Jeremy Bauer-Wolf, Inside Higher Ed, January 17, 2018

Because the pollsters did not investigate anyone's actual success in the job market or anyone's actual preparedness to enter the job market, the results of the survey contribute nothing whatever to anyone's understanding of those issues.

Because the pollsters did not actually talk with any students about the variety or intensity of their sentiments with regard to entering the job market, but only asked them to assent to or dissent from words placed in their mouths by the pollsters, they didn't learn anything about that either.

Because they did not establish a meaningful scale for assent or dissent and did not provide the victims of the survey with a common understanding of the meanings of the numerals associated with the “points” of the so-called scale that they did use, they did not even learn anything significant about the students' attitudes towards the assertions presented.

The survey was a total waste of time, money, and effort, and no one should take the published factoids seriously.

Nonetheless, the report has occasioned some wringing of hands and gnashing of teeth among faculty members who are eager to impose a different spin:

“A Different Look at That Gallup Survey on Student Preparation”
John Warner, Just Visiting, Inside Higher Ed, January 18, 2018

Between graduate school and returning to teaching, I spent some time working for a marketing research company, which included designing and interpreting surveys and one thing I learned is there's a lot of different ways to slice data.

*Sigh.* Apparently spending time in a marketing research company is a good way to lose track of the difference between facts, on one hand, and on the other hand things that are mocked up to look like facts, but aren't. The whole reason for making empirical observations and record the observed facts accurately and impartially is that they may enable you decide which of two or more incompatible hypotheses is correct. If you can “slice data” in various ways, so as to support any hypothesis you like, then the so-called data are useless and you might just as well express your opinions about student preparation for the job market without trying to give them an empirical foundation at all.

It also intrigued me that Warner (who, as he notes, received a Bachelor of Arts degree in rhetoric) uses the words “Far be it from me to dispute Gallup's own interpretations of their data” to introduce the passage in which he explicitly and specifically controverts Gallup's own interpretations of their data. Far be it, indeed.

#questionnaires #Likert-scales #factoids

Hashtag index

This work is licensed under a Creative Commons Attribution-ShareAlike License.

Atom feed

John David Stone (

created June 1, 2014 · last revised December 10, 2018