Skip to main content

Research showed...

One of the themes I return to with regularity in Science for Life is the way that the media rarely distinguishes between quality of scientific sources. There is a huge difference between a Cochrane survey of all available research, or a large scale properly controlled trial and the type of 'study' where you choose 12 people who only ever buy Volkswagen Golfs and ask them what's the best family car. Yet the media just churn it all out with equal weight, telling us that 'a study has found...' or 'research shows...' They may give us a hint of a source, but that rarely gives enough information to be sure of the quality.

As a demonstration of this I did a bit of a butterfly on a wheel analysis of a story in today's papers. It tells us what the top ten things are that parents do to embarrass their children - things like dancing and trying to use yoof-speak. And according to my favourite newspaper, this is the result of 'research'.

So let's dig a little deeper. What's the source? According to the paper it is that highly respected research establishment Thorpe Park, the theme park in Surrey. Now in principle it is possible to obtain reasonable quality data at Thorpe Park if the the park undertook the survey properly, although immediately we have a problem because the group of people who attend Thorpe Park are self-selecting and may well not be typical of the population at large.

So how did Thorpe Park undertake the 'research'? Well, they didn't. It wasn't done at Thorpe Park at all - it was a poll taken for Thorpe Park by an online polling company. In a way this is a better situation, because online polling companies can address a wider section of the population, though of course they can only ever interact with people who have access to the internet - a much larger, but still self-selecting group. However, being professionals, pollsters can use statistics to at least try to correct for this.

That assumes, of course, that this was the right kind of poll. There are some online polling companies like YouGov that are pretty hot on getting their sample right and other good things, though even they have their limitations. But there are others who are essentially marketing organizations who are less worried about sampling and data quality and more interested in delivering useful messages for companies to use in their PR. Let's be clear - I am not saying these marketing-oriented pollsters make things up. They definitely don't. They pay out good money to people to take part in their polls. But in the end the aim of their polls is to gain publicity.

Not entirely surprisingly this poll was undertaken by a marketing-oriented pollster. According to its website "Our research enables brands to create unique data-led content – content that can be published and shared across multiple channels with a view to grabbing attention in a busy media landscape." What we don't know is what data the pollsters provided, because someone somewhere is certainly misinterpreting it. The newspaper says that the list of ways it gives for parents to embarrass their children is the 'top ten things' that parents do. This could only be discovered if respondents were given a simple text box and asked to type in the embarrassing things with no prompting. Actually what the poll did was to already decide what the top ten things were and got their respondents to mark those they considered the worst.

So was this 'research' in the scientific sense? Of course not. And I don't want to make a mountain out of a molehill. It was just a fun story, put together so Thorpe Park had a reason for getting a press release into the news. But I really think that newspapers should use better language. Don't call something like this 'research' or a 'study'. Call it a fun poll or a straw poll or similar language that makes it clear that this is not real in any sense scientific. Otherwise, when they carry a story about 'research' on climate change or particle physics we might be inclined to be equally dismissive.

Comments

Popular posts from this blog

Why I hate opera

If I'm honest, the title of this post is an exaggeration to make a point. I don't really hate opera. There are a couple of operas - notably Monteverdi's Incoranazione di Poppea and Purcell's Dido & Aeneas - that I quite like. But what I do find truly sickening is the reverence with which opera is treated, as if it were some particularly great art form. Nowhere was this more obvious than in ITV's recent gut-wrenchingly awful series Pop Star to Opera Star , where the likes of Alan Tichmarsh treated the real opera singers as if they were fragile pieces on Antiques Roadshow, and the music as if it were a gift of the gods. In my opinion - and I know not everyone agrees - opera is: Mediocre music Melodramatic plots Amateurishly hammy acting A forced and unpleasant singing style Ridiculously over-supported by public funds I won't even bother to go into any detail on the plots and the acting - this is just self-evident. But the other aspects need some ex

Is 5x3 the same as 3x5?

The Internet has gone mildly bonkers over a child in America who was marked down in a test because when asked to work out 5x3 by repeated addition he/she used 5+5+5 instead of 3+3+3+3+3. Those who support the teacher say that 5x3 means 'five lots of 3' where the complainants say that 'times' is commutative (reversible) so the distinction is meaningless as 5x3 and 3x5 are indistinguishable. It's certainly true that not all mathematical operations are commutative. I think we are all comfortable that 5-3 is not the same as 3-5.  However. This not true of multiplication (of numbers). And so if there is to be any distinction, it has to be in the use of English to interpret the 'x' sign. Unfortunately, even here there is no logical way of coming up with a definitive answer. I suspect most primary school teachers would expands 'times' as 'lots of' as mentioned above. So we get 5 x 3 as '5 lots of 3'. Unfortunately that only wor

Which idiot came up with percentage-based gradient signs

Rant warning: the contents of this post could sound like something produced by UKIP. I wish to make it clear that I do not in any way support or endorse that political party. In fact it gives me the creeps. Once upon a time, the signs for a steep hill on British roads displayed the gradient in a simple, easy-to-understand form. If the hill went up, say, one yard for every three yards forward it said '1 in 3'. Then some bureaucrat came along and decided that it would be a good idea to state the slope as a percentage. So now the sign for (say) a 1 in 10 slope says 10% (I think). That 'I think' is because the percentage-based slope is so unnatural. There are two ways we conventionally measure slopes. Either on X/Y coordiates (as in 1 in 4) or using degrees - say at a 15° angle. We don't measure them in percentages. It's easy to visualize a 1 in 3 slope, or a 30 degree angle. Much less obvious what a 33.333 recurring percent slope is. And what's a 100% slope