Generally speaking, even ‘gold standard’ RCTs have lead-like properties.
Much of this is above my pay grade, but I think I pulled out some important points.
By the well-regarded Andrew Gelman, et. al.:
1 Like
this is not my field either. An interesting fact is that, the commonly used 0.05 threshold is arbitrary and was originally proposed by R. Fisher in 1925. And since then, it has become a convention, rather than being based on solid scientific reasoning
A p-value < 0.05 indicates statistical significance but does not necessarily imply practical or clinical importance of the results.
There is also a publication bias: The emphasis on p < 0.05 contributes to the “file drawer problem,” where non-significant results are less likely to be published, creating a biased literature.
Since then, 1925, P-Values has, in som cases been missunderstood and missused and some even talked about The terror (not on elmstreet) but the terror of P-values.
1 Like
From the above:
For a p-value between .01 and .05, a bit better than one chance in three that it will replicate. And a 50/50 chance it overstates the magnitude of the effect by at least 56%.
For a p-value between .005 and .01, coin flip that it will replicate. And 50/50 chance it overstates the magnitude of the effect by at least 40%.
Garbage in, gospel out.
And, regarding the concept of ‘trending toward significance’.
1 Like