When it comes to polls, it seems everyone merely reports on the findings, extrapolates storylines from said poll, without any critical thought attached. All polls are not created equal, all findings don't deserve the same stature. This notion is clouded because certain organizations pay certain pollsters, so there is an inherent investment that must be considered. However, we see time and again, a certain poll move from the "house" organization to the wider media, used by everyone in universal fashion.
There is much at stake with polls, they feed narratives, coverage is coloured, right or wrong, they shape the mood in Ottawa. We've seen time and again how a couple percentage points can make the difference between stable to crisis, momentum to stalled. With the "high stakes" in mind, it is imperative that journalists do their due dilligence, which means TRACK RECORD is king, predictive past a marker, not every poll deserves the same attention.
The results are in for the Toronto mayor race. The pollsters largely missed the huge Ford win, with one exception, that being EKOS. A feather in the cap for EKOS, but also further evidence that some others have a very poor predictive record. All I would ask, when we hear all these opinions, based on a certain poll, there is some acknowledgement of RELIABILITY. A mere mention, like "in the past, this poll has an excellent record", or "one caveat, this pollster has a poor track record", something, anything, that shows a hint of source inspection instead of just lapping up every result and extrapolating based on sketchy source data.
You can't just take one result, but you can investigate a body of work and make informed decisions. Public polls are hugely important, moreso than they should be, but that's another discussion. I believe no poll should be put into the public domain unless the pollster releases all internal data, the phrasing of the questions, full transparent disclosure to protect the integrity of the findings. If a poll will be released into the public domain, then it should be open to scrutiny. Wording of questions are important, nuance can shift findngs in a massive way. Why we just eat up everything as it comes and run with it escapes me, it seems to betray a core tenet of journalism.
Next time we get a poll release, the first question isn't what it means, it's how trustworthy is the data. If we start with that predisposition, then Canadians will get a clearly picture of the true reality, rather than murky conclusions based on dicey data.
Two cents.
19 comments:
favirecaHow about when the poll results are published, they include the actual questions and the actual answers, with no 'weighting'?
Is seems to me, for example, that second choice is no choice.
Anyway, the question quite often 'leads' the answer.
If a pollster won't release their questions, their phrasing, then people should ignore the result. Where's the scrutiny with polls? Seems a glaring omission on the reliable information front.
One good illustration of your point Steve is the Harris-Decima poll you wrote about earlier that showed 31% of Canadians blamed Ignatieff for the loss of the UN SC seat.
While the full results were released the question(s) never were.
That's important because as their press release states: Among those who are unaware Canada was seeing(sic) the seat, 40% feel Mr. Ignatieff’s comments are to blame, while 31% cite the government’s record.
Now this brings up two questions, one, how could people who didn't know it was even happening have any opinion of the cause of an event they knew nothing about.
And second, why would they pick Ignatieff's name out of the air unless they were being prompted in some way. Did Harris-Decima tell each person Ignatieff's comment and then say was it that or was it the govt without going into the full (and lengthy) possible reasons that the govt might have been responsible?
Kirk
That's a perfect example.
OK, found those Harris-Decima questions so I guess I should look further at the poll I criticized.
And they do indeed prompt Ignatieff's name while failing to detail the other possible reasons.
The question was: Which of the following is more likely to explain why Canada did not win the seat. Canada's record on international diplomacy or public criticisms from the leader of the opposition.
So it was either "Canada's record", not Stephan Harper's or any specific actions/policies of the current govt or it was Ignatieff's "public criticisms".
No wonder the results swung from 60%/26% "Canada's record"/Ignatieff's "public criticisms" from people who were aware Canada was pursuing the seat to 31%/40% for the 33% of people polled who didn't know we were pursuing the UN seat.
They mention Ignatieff as a negative influence. "public criticisms". and people then attach that to a negative result while referring completely neutrally to "Canada's record".
Just not mentioning Stephan Harper or the current Conservative govt put it as a choice between Canada and the leader of the opposition and introduced a anti-Ignatieff bias.
Kirk
Good find. I gave up looking at HD for internals awhile ago :) I complained about their releases at one point, then I actually got a response and now at least they put in the regionals for the public releases.
Which pollster comes out with numbers that are closest to the final result is often a bit of a matter of luck. Angus Reid was by far the closest to the results of the last federal and BC elections, but they missed the last Alberta election by a mile. Everyone swore by Nanos after the 2004 and 2006 federal elections, but then in 2008 he was off by quite a bit. Some of it is a matter of chance and some of it is a matter of timing. Ekos polled 300 people on Sunday (the day before the election) and they actually had Ford winning by a lot more than he actually won by. But its not fair to compare a poll the day before election day by Ekos to one by ARG that was fielded 9 days earlier.
That's why you look at a body of work, and not just one result. Angus Reid also nailed the last Quebec and Manitoba elections, so when you put it all together, I'd say pretty credible. Nanos actually picked up the trend on the last day of polling, and he did well the two previous, again the body. Someone like Ipsos here, well that just continues a trend of missing badly, again.
It's not luck, because if it was, then it would all even out and it doesn't.
Speaking of Angus Reid, new poll out with Cons opening up a big lead:
http://www.angus-reid.com/polls/43427/conservatives-extend-lead-in-canada-as-stagnation-hits-liberals/
For the record, I think EKOS, NANOS, Angus Reid and Harris Decima are the top pollsters.
I agree that its not all luck, but its partly luck. When one company gets the winning party's % dead on and someone else is off by 1% - I think its just a matter of being lucky. If a certain company is systematically off by more than anyone else over and over again - especially if we know something about their methodology that would explain it - that's a different story.
That's what I'm talking about. Predictive record, and take your cues accordingly. If someone has an abysmal record, I don't see the value in digesting their results, particularly when they contradict more reliable findings.
From what I gather from your link to the Angus poll, Steve, it was an online poll and there are a great deal of Canadians that do not even own computers nor can afford the Internet. Those I speak for are mainly retirees and low income seniors which (I am part of) who make up the big part (majority) of committed voters. Those are the ones that matter. They have lived through a few years of economic disasters and they recognize truth or untruths when they hear it. Furthermore, they do not like to see monies being wasted from the king of waste on a continual campaign mode seen by this bogus Con party.
I come from a very large family and can speak for them. They do not trust Harper's Reform Alliance party of unqualified caucus members.
That's my rant and I can't believe how naive his supporters are. Watching game one of the world series tonight. I Am supporting the under dog the Texas Rangers Rah Rah.
Cheers
Hey Marie. AR factors in what you've mentioned to get a representative sample, don't discount based on that. This poll was the most accurate in the last election, so these issues obviously weren't a factor. Let's deal with the reality, the polls move from healthy Con lead to statistical ties, have done it for years now. This tells me, that waiting for the perfect poll moment to strike is futile, we may have to go based on other factors. I don't care what the polls say today, I personally think Harper is more vulnerable right now than at any time since his election. I also think the Libs are finally distinguishing themselves, and Ignatieff is finding his voice which will resonate when people actually take a look.
Anything interesting example of the use of polls today with CBC and the latest EKOS poll.
This is a two week poll but CBC only reported the second week in their headline and story.
Now they did this same thing when they got the "Liberals and Conservatives tied" headline so it's not just an anti-Liberal thing.
The poll data gives three results:
Oct. 13-19 - 30.9%/29.4% Con/Lib
Oct. 20-26 - 33.9%/27.8% Con/Lib
Oct. 13-26 - 32.0%/28.8% Con/Lib
http://www.cbc.ca/news/pdf/ekos-101028-tables.pdf
So the polling itself says that the Conservatives went up 3% total and increased the number of people voting for them by 10% in one week. And the Liberals went down a more modest 1.6% losing just over 5% of their supporters in a week.
While taking the two weeks with their larger sample and time period you get another result entirely.
The above link even shows the results for Oct. 6-12 33.9%/27.8% Con/Lib but if you go back and find the data at: http://www.cbc.ca/news/pdf/ekos-101014-tables.pdf for the two period of Sept. 29-Oct. 12 the numbers are 32.5%/28.2% Con/Lib.
So the Sept. 29-Oct. 12 data again has the total result closer than the second week result (odd pattern there) pretty similar to the current two week result which is in both cases closer than the week the CBC reported on in their story.
What does it all mean?
Fun with numbers.
Steve, nothing about today's EKOS poll that's pretty much identical to their poll from 2 weeks ago? Well except that huge increase in the NDP's Atlantic numbers has crashed back down to normality again.
Boy this pattern is getting boring, how many more weeks until Harper prorogues again?
Just about to throw up a post, actually think this is a fascinating poll...
I see you're on the case there Kirk.
Kirk
Interesting too, HD polls only use a two week sample, so if EKOS followed their methodology, then we would see a different result.
A little off topic but maybe Ontario can take lessons from the BC government with this report in BC.
VICTORIA – Premier Gordon Campbell went on TV Wednesday night to announce a 15 per cent cut in personal income tax to take the sting out of the harmonized sales tax.
The income tax reduction applies to the first $72,000 of income, effective Jan. 1. It is the second largest income tax cut in B.C. history, following the 25 per cent cut Campbell made on the first day of his administration in 2001.
This sounds to me like an easy out for the Federal Ontario liberals.
Who in their right mind would not support a 15% tax reductions to off set the dread HST currently in BC and Ontario
Post a Comment