Mythbusters disappoint
Some of you may watch the program Mythbusters on SBS. There are many things that I like about the program. The sheer enthusiasm of the two presenters has to be admired in gentlemen their age. And I also sign on to the central premise of the show, namely that folklore and urban myths should be empirically tested and subject to the verdict of nature. Unfortunately, the whole endeavour came unstuck last night, because they do not know anything about experimental design in agricultural field trials.
The issue was whether plants grow better when subjected to classical music and kindly supportive words than plants that are subjected to heavy metal and verbal abuse. In a quest for scientific credibility they introduced a third treatment, namely control where there was no music or chatting. I joined the program half way through and found one presenter yelling abuse at plants in one greenhouse and another presenter whispering sweet nothings to the plants in a second greenhouse.
That’s right! They had put all the abused plants in one greenhouse and all the supportively nurtured plants in another greenhouse. So the treatment was completely confounded with the position and orientation of the greenhouse. The good folks out at Rothamstead in the 1930’s were obviously wasting their time with their randomisation and latin squares. Sigh……
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
June 6th, 2007 at 6:46 am
Unless we’re talking about strictly classical music, Mozart and Haydn, where does the idea come from that classical music is restful and peaceful? Try the plants on Stravinsky’s Rite of Spring, Varese’s Arcana or Magnus Lindberg’s Kraft and see how they like that! Off topic sorry
June 6th, 2007 at 7:55 am
It’s a shame that a show that attempts to encourage scientific thought is so bad at science.
I had the same feelings when I caught the British Psychic Challenge on foxtel recently. Poorly designed experiments gave the skeptics no reason to ever believe the results.
June 6th, 2007 at 10:11 am
Chris, I have to agree about the enthusiasm. But like other programs around (e.g. “What’s good for you”) I very much feel that these programs (of course) are more about entertainment, dressed up and paraded around like science (the big bad wolf dressed up like the old granny comes to mind).
Too often in these shows do the presenters conclude that “absence of evidence” is “evidence of absence” - often based on a single experimental run. Sure, you need things to go “right” only once to “prove” that something is possible, but I think you have to work a lot harder to “prove” (can you ever) that something is not possible.
Is this where SSAI should be objecting? My preference is “yes”, but why should we continually bang our heads against a brick wall?
June 6th, 2007 at 2:19 pm
Along the same lines, here is a discussion of the analysis of Mythbuster’s ‘yawn experiment’ data (in which they concluded that percentages of 25% versus 29% in two groups of n1=n2=50 was evidence of a difference).
http://www.omninerd.com/2007/04/19/articles/75
June 6th, 2007 at 2:52 pm
On the positive side, the Mythbusters make a great teaching tool!
I showed my class the Mythbusters’ yawn experiment (this morning, funnily enough) and used it to motivate Wald tests. I have used a another experiment of theirs from the same episode (does toast land butter side down more often than butter side up) as a way of motivating the study of the binomial distribution.
In both instances, stats-101 tools lead you to a completely different conclusion to the “gut feeling” interpretation of the Mythbusters. Which is fine with me - it reinforces two important messages: (a) Probability is often not very intuitive… don’t rely on the gut! (b) While a knowledge of key statistical ideas is often very important and useful, unfortunately many do not have such knowledge!
PS: I used a Wald test because my students don’t know about contingency tables yet, but what’s omninerd’s excuse? I guess a correlation coefficient is equivalent to the relevant Pearson chi-squared stat, but still…
June 16th, 2007 at 9:55 am
It is a rather interesting question, because poorly designed experiments are of course common.
What should one do? Suggest that they throw out the results and start again? Treat the experiment as having no value?
Perhaps a more positive suggestion might be to augment the experiment in some fashion.
As it currently stands, the results (as I remember) showed no effect from the treatment. If there truly is an effect it most probably is small or, less likely, in some way masked by other factors which were insufficiently randomized/controlled for.
I for one would be less concerned about a greenhouse effect (as I recall they were quite well spaced, of identical design, and having the same orientation) than I would about a “treater” effect .. again, if memory seres me correctly, a single person (male) did the “nasty” treatments and another person (”female”) did the nice stuff. If they HAD found some sort of significance, then that confounding might be cause for some suspicion.. having found nothing of consequence, then we might let it slide.. one cannot, after all, control for everything.
I’d be interested in other’s opinions as to if and how the design should be augmented.
June 18th, 2007 at 5:56 am
Interesting comments. I’ve only seen Mythbusters a couple of times, many months (even years) ago - I like the concept and it was fun, but it seemed they commonly only had a single rep of each treatment - a pity they aren’t more scientific about it, or at least perhaps point out shortcomings and how it could be made more legitimate.
It would be an excellent thing for SSAI or similar, or individuals (like me, or any one of you readers!), to put a little effort into (but not to waste too much time with the brick wall). An obvious opportunity to publicise better stats practice and promote (make more visible) statistics as a discipline and profession.
June 18th, 2007 at 6:32 pm
I think you’re right John. The fact that they found no treatment effect makes the poor design less of an issue. But I must admit that I turned off before the no treatment effect was established.
As for the “confounded greenhouse effect” (sorry - couldn’t resist), I think the fact the greenhouses were separated by 2 meters or so might introduce issues of differential shading. More plausible would be differences in the soil put into the pots - unless they scrupulously randomed the allocation of pots to greenhouses. Any chance they did this dya reckon!??
Quite right Kathy. What the profession needs is a statistical version of Karl Kruszelnicki. A statistical mythbuster if you lilke. Perhaps someone should put in for an ARC grant to fund such a position.
October 1st, 2007 at 4:15 am
Studio Glamour Modeling
hey great stuff