This evening, after his press conference attack on the ‘mainstream media’, President Trump e-mailed this out (presumably to those registered as supporters of his campaign).

I really want to look at the survey that’s behind those links, but, right from the off he’s failed. If you want your final data to reveal anything then, at the very least, you need a balanced, representative sample and you don’t tell them the result you want to achieve in the bloody e-mail asking them to take the survey!

Before a single person has even clicked on the link you’ve made your results worthless. On the plus side it means that what comes next doesn’t matter so much…

The survey is here, if you want to play along at home.

The first three questions aren’t too bad. Sure, ‘report fairly’ would have been less ambiguous than ‘fairly report’ and I’d have been more inclined to list multiple news sources and ask, “Which do you trust to report fairly…”, but that’s nit-picking.

A bigger issue is that the campaign, assuming it’s the presidential campaign, finished in November. There was an election and everything. You may have seen something about it on the news.

If you’re one of the people who knows that the campaign is over then which campaign is this talking about? The 2020 campaign? After all, Trump has already registered as a candidate in that election. Maybe it’s the campaign to take on the biased MSM, that was mentioned in the nice e-mail inviting you to take this survey.

Either your survey is out of date or you’re confusing the crap out of people.

Then we take a nose-dive.

The British Polling Council, the body that oversees political polling in the UK, insists that questions are neutral and balanced. This question is neither; there’s no scale, good to bad, for you to say how fairly the mainstream media (a nebulous concept itself, which no attempt has been made to define) presents each issue, there’s not even an option to avoid ticking at least one of the selection offered.

Also, ‘worst’ is a superlative. There can, by definition, only be one worst option, so instructing the respondent to ‘select as many that [sic] apply’ is idiotic.

There follows a question on primary TV news source, which – other than missing an option to say you don’t watch TV news – is reasonably unobjectionable, and then this…

Skipping over the ‘select as many that apply’ (it’s almost as if they couldn’t decide between ‘select all that apply’ and ‘select as many as apply’ and tried to go for both) this question only bothers naming right-wing news sources, none of them in the usual ‘mainstream media’ bundle. The ‘other’ option doesn’t even open a write-in box and, again, there’s no option to say ‘None of the above’ or ‘I don’t read news online’.

What’s strange is that this is counter-productive to the survey’s purpose. If you’re trying to prove that the MSM is biased against you then why railroad your sample into saying that they don’t read it? Why push them towards demonstrably biased and frequently inaccurate news sources, many of them with comparatively tiny online readership?

You’re basically inviting your sample to demonstrate that they’re not very good at evaluating online bias.

The first problem with question 8 is that it’s not a question. There’s a clue in that it ends with a full-stop, rather than a question mark.

More importantly the question is begged – in answering it you inherently accept the two premises it contains, i.e. that Hillary Clinton is lying about sending classified information on (‘from’, surely) her secret sever and that, previously, the press have given her a free pass on this issue. You can, therefore give one of three answers:

• Yes, I agree that the press are still giving lying Hillary a free pass
• No, I think the press are now holding lying Hillary to account
• I have no opinion on the press’ treatment of lying Hillary.

This is the old, “Have you stopped beating your wife?” question, wrapped up in a smart new political suit.

‘No opinion’ is the most pro-Hillary stance you can take on this question. This is important for two reasons:

Firstly, it encourages people who are pro-Clinton to give up, because they can’t express their views.

Secondly, in political polling, it’s common, almost routine, to publish just the answers of those who’ve expressed an opinion.

The next block of questions all follow this pattern:

• Q9, The mainstream media takes Donald Trump’s statements out of context, but bends over backwards to defend Hillary’s statements.
• Q10, The mainstream media failed to cover the fact that Bernie Sanders LEFT the Democrat Party.
• Q11, The mainstream media needs to do more to expose the shady donations to the Clinton Foundation.
• Q12, Political correctness has created biased news coverage of both illegal immigration and radical Islamic terrorism.
• Q13, The RNC was right to drop CNBC as a partner after they failed to fairly moderate the October debate.
• Q14, The mainstream media hardly reported on the fact that our small-dollar fundraising nearly MATCHED Hillary’s Wall Street fundraising machine.
• Q15, The mainstream media played a critical role in electing President Obama and is now attempting to do it again for Hillary Clinton.
• Q16, Contrary to what the media says, raising taxes does not create jobs.
• Q18, American history is being rewritten by “social justice” activists.
• Q19, The media has not done its due diligence to expose ObamaCare’s many failures.
• Q20, The media wrongly attributes gun violence to Second Amendment rights.

And so on.

Some of these are more objectionable that others, but they all present the same, stark, ‘Yes’, ‘No’, ‘No opinion’ options. Some of them could, possibly, be rewritten along the lines of “On a scale of 1 (strongly disagree) to 5 (strongly agree) how much do you agree with the statement…”, but even then many of them are too inherently biased to be useful.

Question 17 is missing from that list, because it deserves a special mention.

This is being asked by the campaign of the man who wanted to ban all Muslim immigration into the US, until we could figure out what was happening. So is this question going to treat a high ‘yes’ vote as ‘Christians have been unfairly characterized” and a high ‘no’ vote as ‘Muslims haven’t been unfairly characterized by the sections of the MSM who say they’re all terrorists’?

This question’s lack of specificity guarantees a winning result either way.

Finally, there’s an even less subtle trick with question 23.

This uses what psychics and fortune tellers call ‘the vanishing negative’. To see its effect we just need to precede the statement with either of the first two answers.

Yes, Americans are not fully aware just how much waste there is in the federal government.

No, Americans are not fully aware just how much waste there is in the federal government.

Both answers mean exactly the same thing. There is literally no way to reject this question.

However, I’ll leave you with this thought…if you think that’s clever consider how different that guaranteed result is when asked during a campaign (as this clearly was) compared to it being asked when you are the Federal Government.

[Addendum]

There’s now an updated questionnaire here. They’ve removed references to ‘the campaign’ and added an ‘Other, please specify’ option to pretty much all of the questions. Even where it makes no logical sense…

While there’s a place for open ‘Other’ options what they do is add a lot of processing time to your data extraction, because somebody has to read through them all and group them into a codeframe in order to make the results quantitative, rather than qualitative.

Unless, of course, you plan on just ignoring them.

The on-line news source has been replaced entirely with an open question. Again, this creates a lot of work in DP, especially when it would have worked better as a closed list with an open ‘Other’ option.

We’ve also got a couple of badly worded questions asking about divisions in the Republican party.

I’m not sure you can divide…against. It’s not a big thing, just another point that’s indicative of how little thought went into this mess.

The survey now ends with this…

This isn’t a bad question, per se, but it is a strange one. Trump is the head of the US government. He has the option of using the power of his office to hold MSM accountable, yet this question suggests that the GOP should be the ones performing this task.

It’s also interesting that they’re only interested in the mainstream media. The alternative news sources and bloggers, who are mentioned in a previous question, are implicitly outside the scope of this expenditure of time and resources.

What’s been asked for here is a mandate to ruthlessly pursue (comparatively) well-researched, but unfriendly, news sources, while giving free-reign to friendly news sources.

Make no mistake, this survey isn’t just bad, it’s actively dangerous.

[Further addendum]

The huge popularity of Trump’s survey has, apparently, worked against it, because the following e-mail has been sent out.

Clearly the president’s survey isn’t returning the results he wants.

Now, in fairness, with a political poll it’s important to balance the sample, as you want it to be representative of the whole voting population. In this particular case you’d want, based on the 2016 vote share, about 46% of respondents to be Republicans and about 48% to be Democrats, with the remaining 6% being third-party or independent candidate supporters. If the results don’t naturally fall out that way then you can “weight” the data to achieve exactly the percentages you’re looking for, and this is a perfectly valid and widely used technique. Except…

1. It’s not possible here, because basic questions about age, gender, location, socio-economic group and past voting haven’t been asked. You have literally nothing to weight on (weighting on the results you’re trying to measure, rather than demographic factors, is not an acceptable technique, for reasons which should be obvious).
2. Nor is trying to pack the survey with respondents who, you hope, will give you the answers that you want. Don’t forget that, because of the point above, the people looking at the data have literally no idea how many Democrats, if any, have taken the survey. All they know is that the results aren’t showing them the answers that they want to be able to report.

What’s comic is that they believe, according to their e-mail, that Democrats are trying to sabotage the results, apparently blissfully unaware that their own errors and general sloppiness have already made the results meaningless anyway.