Comments on: Does “Statistical Significance” Imply “Actually Significant”? http://popsych.org/does-statistical-significance-imply-actually-signifiant/ The Internet's Best Evolutionary Psycholo-guy Wed, 03 Jan 2018 01:05:13 +0000 hourly 1 http://wordpress.org/?v=3.4.2 By: Statisticial Issues In Psychology And What Not To Do About Them | Pop Psychology http://popsych.org/does-statistical-significance-imply-actually-signifiant/#comment-513 Statisticial Issues In Psychology And What Not To Do About Them | Pop Psychology Sat, 23 Feb 2013 22:16:54 +0000 http://popsych.org/?p=940#comment-513 [...] I’ve discussed previously, there are a number of theoretical and practical issues that plague psychological research in terms [...] [...] I’ve discussed previously, there are a number of theoretical and practical issues that plague psychological research in terms [...]

]]> By: The Sometimes Significant Effects Of Sexism | Pop Psychology http://popsych.org/does-statistical-significance-imply-actually-signifiant/#comment-391 The Sometimes Significant Effects Of Sexism | Pop Psychology Sun, 25 Nov 2012 01:48:57 +0000 http://popsych.org/?p=940#comment-391 [...] Post navigation ← Previous [...] [...] Post navigation ← Previous [...]

]]> By: Leigh Caldwell http://popsych.org/does-statistical-significance-imply-actually-signifiant/#comment-389 Leigh Caldwell Tue, 20 Nov 2012 14:51:34 +0000 http://popsych.org/?p=940#comment-389 One of the authors, Simonsohn, also just presented at the SJDM conference a clever method for detecting what he calls "p-hacking", the collective name for the p-value-reducing practices mentioned above. It doesn't work on single papers but can be used across all the papers in a group: say, all papers on the endowment effect or all by a particular author. The technique simply requires drawing a histogram of all the p-values in all the papers. If the papers are describing real effects and conducted correctly, we would expect to see more p=0.01 than p=0.02 and more p=0.03 than p=0.05 - the histogram will be skewed towards zero. If there is no effect and the results are purely due to chance, we'd see a flat graph. And if there is a lot of p-hacking, we'd see it skewed the other way, towards the 0.05 end. He analysed a couple of groups of papers (based on specific keyword criteria) and found that p-hacking could indeed be detected in certain bodies of work. Clever technique and very practical at the meta-level to see whether the overall research in a particular field can be relied on. One of the authors, Simonsohn, also just presented at the SJDM conference a clever method for detecting what he calls “p-hacking”, the collective name for the p-value-reducing practices mentioned above.

It doesn’t work on single papers but can be used across all the papers in a group: say, all papers on the endowment effect or all by a particular author. The technique simply requires drawing a histogram of all the p-values in all the papers.

If the papers are describing real effects and conducted correctly, we would expect to see more p=0.01 than p=0.02 and more p=0.03 than p=0.05 – the histogram will be skewed towards zero. If there is no effect and the results are purely due to chance, we’d see a flat graph. And if there is a lot of p-hacking, we’d see it skewed the other way, towards the 0.05 end.

He analysed a couple of groups of papers (based on specific keyword criteria) and found that p-hacking could indeed be detected in certain bodies of work. Clever technique and very practical at the meta-level to see whether the overall research in a particular field can be relied on.

]]>