tag:blogger.com,1999:blog-21621108.post8730167077849788106..comments2020-05-28T07:48:27.059+02:00Comments on Shravan Vasishth's Slog (Statistics blog): Observed vs True Statistical Power, and the power inflation indexShravan Vasishthhttp://www.blogger.com/profile/13453158922142934436noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-21621108.post-23753603119932537592015-08-30T17:37:40.566+02:002015-08-30T17:37:40.566+02:00I haven't read her blogposts on the replicatio...I haven't read her blogposts on the replication project with sufficient care to be able to comment on it. But, my prior on the issue given (what I see as) her extreme clarity on other statistical/methodological issues is to grant her the benefit of the doubt.<br /><br />I also wanted to thank you for maintaining this blog; it's been good fodder for the grey cells :).karthik durvasulahttps://www.blogger.com/profile/14541529987768107005noreply@blogger.comtag:blogger.com,1999:blog-21621108.post-34318916906856199492015-08-30T16:05:23.118+02:002015-08-30T16:05:23.118+02:00I'm happy to give her benefit of doubt as rega...I'm happy to give her benefit of doubt as regards her understanding of these concepts. As you have seen, I also define things inaccurately from time to time.<br /><br />However, there is more about her that I think hurts her credibility. She is now attacking the replication project without even acknowledging that it's a huge step forward. Shravan Vasishthhttps://www.blogger.com/profile/13453158922142934436noreply@blogger.comtag:blogger.com,1999:blog-21621108.post-90109234609722882392015-08-29T22:26:59.675+02:002015-08-29T22:26:59.675+02:00When you posted that quote, it did take me by surp...When you posted that quote, it did take me by surprise. So, I decided to read the comment thread more carefully to see if this was just a terminological issue, and not really a conceptual one. And a careful reading of Mayo's response to seems to suggest to me that she intended "actual" to be in contrast to thresholded values, and not to "nominal". She was using "actual" in one of its normal English meanings, as opposed to a slightly more technical statistical sense. She writes later:<br /><br />>>"I think it was my use of the word “actual” that got you confused. It only referred to the “attained” significance level or P-value, rather than a pre-designated cut-off (see the Lehmann-Romano quote).As discussed in the post that I linked to, the error probabilities associated with tests are hypothetical." <br /><br />So, I am not sure that your allegation of her conceptual misunderstandings stands, though she could have shown more care in choosing her words.<br /><br />There are of course other writings where she is absolutely clear about actual and nominal p-values (e.g., <a href="http://errorstatistics.com/2015/03/05/a-puzzle-about-the-latest-test-ban-or-dont-ask-dont-tell/" rel="nofollow">here</a>), and this is true even in her correspondence with me via email. And unlike her blogposts (sometimes), her actual published work is really clear in my opinion wrt to deep/important statistical issues.karthik durvasulahttps://www.blogger.com/profile/14541529987768107005noreply@blogger.comtag:blogger.com,1999:blog-21621108.post-12663830809352316802015-08-29T08:56:35.656+02:002015-08-29T08:56:35.656+02:00Karthik,
Here is the comment where Mayo says tha...Karthik, <br /><br />Here is the comment where Mayo says that <br /><br />"it would be OK so long as they reported the actual type I error, which is the P-value."<br /><br /><br />http://errorstatistics.com/2015/07/17/statistical-significance-according-to-the-u-s-dept-of-health-and-human-services-i/#comment-127764<br /><br />Also see Morey's questions to Mayo in this thread and her responses. <br /><br /><br />Shravan Vasishthhttps://www.blogger.com/profile/13453158922142934436noreply@blogger.comtag:blogger.com,1999:blog-21621108.post-87600578015061884202015-08-28T13:35:43.621+02:002015-08-28T13:35:43.621+02:00>> nice definition in terms of a power *func...>> nice definition in terms of a power *function*<br /><br />Yes, I agree.<br /><br />Also, the following is not immediately relevant to your post, so I own't waste too much ink on it. But you say:<br /><br />>>It's very hard to take Mayo seriously, because she writes fairly crazy stuff (such as equating p-values with something called "actual" Type I error). <br /><br />I would be very surprised if she said something like that. Do you have a reference for this particular comment?karthik durvasulahttps://www.blogger.com/profile/14541529987768107005noreply@blogger.comtag:blogger.com,1999:blog-21621108.post-82986083023866948872015-08-28T06:55:51.549+02:002015-08-28T06:55:51.549+02:00Sure; I agree that power of course has to be compu...Sure; I agree that power of course has to be computed with respect to some specific mu, and so your definition is better. There is no single power calculation with respect to the null: agreed. <br /><br />Rice defines power *of a test* as the probability that the null hypothesis is rejected when it is false. I should have put in *of a test* in my definition.<br /><br />Mood et al provide a nice definition in terms of a power *function*, which is the point of your comment: "Given a null hypothesis, and let Gamma be the test of a null hypothesis. The power function of a test Gamma, denoted by Pi_Gamma(theta)...is the probability that H0 is rejected when the distribution from which the sample was obtained was parameterized by theta. The Mood et al definition seems better.<br /><br />It's very hard to take Mayo seriously, because she writes fairly crazy stuff (such as equating p-values with something called "actual" Type I error). Shravan Vasishthhttps://www.blogger.com/profile/13453158922142934436noreply@blogger.comtag:blogger.com,1999:blog-21621108.post-38557027558433244732015-08-27T23:35:37.265+02:002015-08-27T23:35:37.265+02:00>>power as the probability of rejecting the ...>>power as the probability of rejecting the null hypothesis when it is false<br /><br />In my opinion, the above suggested definition is also incorrect, and quite easily leads to thinking of hypotheses as dichotomous (a view that leads to the kinds of p-value “paradoxes”, we should be guarding against).<br /><br />Power is better defined with respect to a particular discrepancy from the null hypothesis. So, power is for a particular μ’. Pow(μ’) is the probability of correctly rejecting the null hypothesis when the actual discrepancy is μ’.<br /><br />By, this definition, there is no *single* power calculation with respect to a null hypotheses, although there might be a worst case scenario given what the practitioner considers to be a meaningful discrepancy.<br /><br /><a href="http://errorstatistics.com/2015/07/29/telling-whats-true-about-power-if-practicing-within-the-error-statistical-tribe/" rel="nofollow">Deborah Mayo</a> has written a lot about this.karthik durvasulahttps://www.blogger.com/profile/14541529987768107005noreply@blogger.com