Or any other place with a numerically large number of varieties of a fairly uniform product, with selection based on a number of subjective criteria.
(An inquiry into the validity of polling results with no rants about English language journalists.)
In a post on the upshot of the Fukuda fold over the start of the Extraordinary Diet session, Tobias Harris cites a Yomiuri poll on the public's views of the extent to which a particular member of the Diet would be most appropriate (mottomo fusawashii) for the position of Prime Minister.
Of the possible choices, Liberal Democratic Party Secretary-General Aso Tarō comes out way ahead of everyone else, with nearly double the number of votes of his nearest rival, former Prime Minister Koizumi Jun'ichirō, and with seven times the votes of current Prime Minister Fukuda Yasuo.
Before we all go gaga over the implications of this poll, I would like to voice (well, actually type out) a few concerns.
Below is the list of candidates and the numbers they pulled in response to the question "Which among these members of the Diet do you think is the one most appropriate to be the Prime Minister? Pick only one of the following."
Machimura Nobutaka 0.7
Tanigaki Sadakazu 1.3
Ishihara Nobuteru 0.9
Nakagawa Hidenao 0.4
Koizumi Jun'ichirō 13.0
Nukaga Fukushirō 0.0
Nakagawa Shōichi 0.3
Hatoyama Yukio 1.2
Noda Katsuhiko 0.1
Masuzoe Yōichi 3.4
Maehara Seiji 1.1
Okada Tatsuya 2.8
Ota Masahiro 0.3
Koikei Yuriko 1.0
Fukuda Yasuo 3.4
Yosano Kaoru 0.3
Ozawa Ichirō 9.6
Noda Seiko 0.5
Abe Shinzō 1.6
Kan Naoto 3.4
Aso Tarō 24.7
Somebody Else 0.4
None of the Above 24.7
I Can't Answer 5.1
My concerns are:
1) That is a heck of a lot of names to have to stare at.
2) Is is fair to put Fukuda's name in a list of names of those persons most appropriate to be Prime Minister. Because he is the Prime Minister, you know. Why nominate for prime minister a person who is already the Prime Minister?
3) The question was put to 3000 person nationwide. Of those, 1758 persons chose to reply to the poll. Am I to understand that of the 1758 who chose to reply, 5.1% said, "I cannot answer the question"?
4) "None of the Above" wins 24.7% of the vote. Out of a list of 21 possible names of prominent politicians. And "None of the Above" is not a proper answer to the question the pollsters asked.
Enough said.
5) Was the list of candidates randomized and the selection process unpressured?
I should hope so. Still, I have to wonder -- because in the newspaper story reporting the results of the poll, the names of the candidates were in gojūon order.
I produced the above list by arranging the names in a descending order based on the number of romaji letters in the name. The order in which the names of the candidates were presented in the article was as follows:
Aso Tarō 24.7
Abe Shinzō 1.6
Ishihara Nobuteru 0.9
Ota Masahiro 0.3
Okada Tatsuya 2.8
Ozawa Ichirō 9.6
Kan Naoto 3.4
Koikei Yuriko 1.0
Koizumi Jun'ichirō 13.0
Tanigaki Sadakazu 1.3
Nakagawa Shōichi 0.3
Nakagawa Hidenao 0.4
Nukaga Fukushirō 0.0
Noda Seiko 0.5
Noda Katsuhiko 0.1
Hatoyama Yukio 1.2
Fukuda Yasuo 3.4
Maehara Seiji 1.1
Masuzoe Yōichi 3.4
Machimura Nobutaka 0.7
Yosano Kaoru 0.3
If the average person were confronted with a list of 21 names in a personal interview situation, and the LDP's Secretary-General was the first name on the list, would there not be a bias toward the selection of that name?
Normally arranging names by gojūon order would reduce bias. However, when the most probable candidate's surname is Aso, making him first in a long list of names, gojūon -- if that is what was used -- seems almost guaranteed to create a bias.
It is possible that this poll was conducted in an defensible, fully randomized way.
It is also possible that it was not.
MTC,
ReplyDeleteI don't disagree the "fusawashii" polls are problematic methodologically.
But he has been consistently high on the list repeatedly, even when he was out of power and wandering the country. Regardless of the figures, he is popular; I just wouldn't rely on this poll to get the precise ratio of his popularity to other alternatives.
There is a very large saliency effect whenever you ask people to select something; or remember something; or consider something in any way. In the case of lists, the first and last elements are well known to be very highly salient compared to most elements (the next to first and next to last get a bit of "reflected glory").
ReplyDeleteA responsible, serious polling agency would have several lists, all with partially randomized order making sure significant options are all somewhere in the middle (in different orders) and different nonessential choices at the top and bottom spots (and the spots around five to nine spots before the beginning of the list is they want to be sticklers for accuracy).
And of course, irrespective of the order you have to assume that the pollsters do not have a favourite in the list and aren't doing any version of "forced choice" - consciously or subconsciously - to make people choose the alternative they want them to.
tobias harris -
ReplyDeleteI know that you know the weaknesses of newspaper polling. I thought your readers should be alerted to the potentially significant problems present in this particular poll, especially since, by the time I read your piece, Yomiuri had scrubbed the link.
You are right, Aso is popular - but as Okumura-san would likely have pointed out "popular" is a shaky concept when Aso receives exactly the same numbers as inai ("None of the Above").
I vote for an iroha ordering of the names...
ReplyDeleteSorry, but I don't agree with Janne Moren's suggestion. You can no more have a partially random list than you can be partially pregnant. The original bias remains, and you introduce another subtle bias by subjectively clustering the "significant options" around the mid point.
ReplyDeleteFor the survey to have been done properly, it should have used fully randomized lists (an easy enough task). Given the number of respondents, this would have gotten you close enough to a random ordered set to be statistically significant. But I doubt this was done.
I think Aso as PM would go the same way as Abe. Popular to begin with, but then down the pipes once a real challenge pops up. Seems to me he is more flash than bang.
ReplyDeleteWhat I am more interested in is why English authors leave the macron off of the 'o' on his surname. A phoenetic rendering of the Kanji in his name tells seems to suggest that both 'o's should be long. Wikipedia, that bastion of searchable pronunciations of difficult Japanese names, also tells me they are long. Is there some transliteration rule I am unaware of, or does the good Mr.
Aso simply like to be different?
bryce -
ReplyDeleteI have no reason for leaving the macron off. I will add it from now on.
Sorry MTC,
ReplyDeleteI wasn't trying to be a pedant. I really want to know the answer to this question because everyone seems to do it - from the mainstream media on down. You were just following standard practise.
I don't mind that everyone does it. I want to know the reason why they do. I recently submitted a publication where, following standard pratise, I left the macron off Aso, but on Taro. I want to know if this is just the correct transliteration or something I should revise.
Mine is "Blueberry panacota" mmm
ReplyDelete