I am not aware of any theoretical or empirical justification for imposing an arbitrary absolute threshold for P-values below which we consider a difference “significant”–and thus considered scientifically relevant–rather than taking into account the actual P-value and adjusting *how* scientifically relevant we consider a particular difference to be.
For example, if your P-value is less than 0.00001, then it is exceedingly likely that the difference you have observed reflects a real difference between the underlying populations. If your P-value is between 0.05 and 0.01, then it is reasonably likely. If your P-value is 0.25, then it is unlikely.
But why is the title of a published paper allowed to refer to a difference with a P-value of 0.49995, but a difference with a P-value of 0.50005 is verboten to consider at all? Is there any justification for this practice other than it’s-just-what-we-do?