Brace yourself for wave upon wave of … political polls. The latest show President Trump lagging presumptive Democratic presidential candidate Joe Biden, as usual. But stop. Let us do what no one does – look BEHIND these polls, understand the process and bias in polling.
Take two recent polls, by NPR/PBS/Marist (NPR) and ABC/Washington Post (ABC). Both report new anti-Trump findings. Take a moment and unpack them.
MSN.com – like may non-technical, left-leaning sites – summarizes both. A stinging line punctuates a leading paragraph. Says MSN, NPR’s poll “has Biden beating Trump 60 percent to 35 percent among suburban voters.” ABC’s poll “is the latest to show former Vice President Joe Biden on a roll … up by a 52 percent to 43 percent margin among suburban voters …”
Okay, so narrative is set – “Biden beating Trump,” “Biden on a roll,” stark numbers. Most readers go no further, but MSN inserts links to suggest you can – so let us do it. See: https://www.msn.com/en-us/news/politics/new-polls-show-joe-biden-is-winning-suburbanites-by-a-historic-margin/ar-BB16Vu0y.
Go to links. What you see is “methodology,” on NPR’s poll some 61 categories defining 1,640 interviewees. Who is going to study 61 categories? Not many – but we are. Here is what you learn.
On process, the survey was done in English and Spanish (no word how many of each), demographic data lags to 2017, interviews were not all “registered voters,” sample size of small groups were unreported, cell phones for business were not accepted, and no response rate is listed. That is, no report of how many calls resulted in no contact, or “no thank you, not interested in NPR’s survey.”
Any bias here? What if the survey was just English – our official language? How were nationalized Americans distinguished from illegals? What if demographics were current? Doesn’t old data change findings? Which subgroups were “too small to report?” What if they tipped for Trump?
What effect did omitting “business cell phones” have on findings? What if owners of the nation’s 30.2 million small businesses have one cell for both business and personal use? After all, 50 percent of small businesses operate from “home.” What if these small businesses are pro-Trump entrepreneurs?
Surprise: They are. Another survey reports “President Trump’s approval rating among small business owners hit an all-time high of 64 percent” in 2020. If cells used by business owners are omitted, does that affect results? See: https://www.cnbc.com/2020/02/20/trumps-approval-rating-among-small-biz-owners-hits-64percent-survey-shows.html#:~:text=Small%20business%20owners%20are%20a%20consistent%20source%20of,%E2%80%9CPart%20of%20this%20is%20just%20partisanship%2C%E2%80%9D%20Wronski%20said.
Dig deeper. In the NPR sample, 38 percent are Democrats, only 31 percent Republicans, with 29 percent Independents. Does that overrepresent Democrats v. Republicans, and Independents v. both? What effect does that have on accuracy?
Look closer. Among Democrats in NPR’s survey, one third more are women than men. Do Democrat men favor Trump more than Democrat women? Maybe. Slice the pie differently. In NPR’s survey, 52 percent are women, 48 percent men. Does that affect findings?
How about the fact that older voters – who tend to be more conservative – vote more often? How does NPR account for that? Pew Research reported in 2020, while “older voters accounted for 43 percent of eligible voters, they cast 49 percent of ballots.” See: https://www.pewsocialtrends.org/essay/an-early-look-at-the-2020-electorate/. Moreover, 54 percent of older Americans approve of Trump, compared to 30 percent of youngest voters. See: https://www.pewsocialtrends.org/2019/01/17/generation-z-looks-a-lot-like-millennials-on-key-social-and-political-issues/.
With respect to age, only 26 percent of NPR’s respondents were aged 45 to 59. This may be a major flaw, as 35 percent of voters in 2016 were over 50, and NPR’s survey appears to wholly omit voters over 59. This is particularly meaningful, as 25 percent of all voters in 2016 were over 65.
Another unreported fact should trigger critical thinking – especially on this NPR survey. The June 26th NPR survey says Trump’s approval fell from 43 percent in March 2020 to 40 percent in June 2020. But Trump’s overall approval rating was 37 percent in March 2017 – five months after winning election.
Pivot to ABC’s latest survey, more bias. The media does not expect you to ask, but let us ask. ABC’s headline – to which MSN links: “Pandemic Surge Damages Trump, Boosting Biden’s White House Bid.” But is that what ABC’s data really says?
To get methodology and qualifiers, you must search. Stressing credibility, they note: “We do accept some probability-based surveys that do not meet our own methodological standards … recommend cautious use of such data.” Then on shorter survey methodology, the link just pops an error message.
We learn 65 percent of interviews were cell phones, 35 percent landlines. Taking that as gospel, demographic data show landlines are held by older Americans, cells held by younger Americans. Does a 2:1 bias for cells tip results? Maybe. See, e.g., https://nypost.com/2017/05/04/why-nearly-46-percent-of-household-still-have-landlines/.
How about this: ABC uses a “cleaning process” that “excludes respondents who have home-based business-listed phones.” Recall more than 50 percent of small businesses operate from “home,” and 64 percent favored Trump. Does omitting such lines affect survey results? Maybe.
Other biases are unaccounted – or just undercounted. The ABC poll omits phones in “institutional” settings, but what of senior living communities? Most are conservative – and residents vote. Numbers are not small. In fact, the nation has thousands of apparently undercounted senior living communities. See, e.g. https://www.statista.com/statistics/895322/senior-housing-communities-usa-by-region/.
Last, like other surveys, this one is ambiguous on “response rate” – that is, number of failed contacts, and calls ending in “no, thank you, click.” Despite contrary arguments, response rates matter. For several years, telephone survey response rates have been falling. They now stand at a shocking – six percent. See: https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/.
What does that mean? Most people just do not answer. If they answer, they hang up. Those who talk – when told NPR or ABC is calling – might have a motive. So, net-net do not put great faith in polls.
Even when luminaries at NPR, ABC, and elsewhere say they have 2020 figured out – they probably do not. They have results they want, based on questions, demographics, and methods supporting the narrative – and a nice, juicy headline.
So, brace yourself for wave on wave … of polls. But do not put great stock in them. The reservations often swallow their findings. Only one poll really counts. You know it – November 3, 2020.