This article elucidates a methodological pitfall of cross-validation for evaluating predictive models applied to naturalistic neuroimaging data--namely, 'reverse double-dipping' (RDD). In a broader context, this problem is also known as 'leakage in training examples', which poses challenges in detecting it in practice. This issue can occur when predictive modeling is employed with data from a conventional neuroscientific design, characterized by a limited set of stimuli repeated across trials and/or participants, resulting in spurious predictive performances due to overfitting to repeated signals, even in the presence of independent noise. Through comprehensive simulations and real-world examples following theoretical formulation, the article underscores how such information leakage can occur and how severely it could compromise the analysis when it is combined with widely spread informal reverse inference. The article concludes with practical recommendations for researchers to avoid RDD in their experiment design and analysis.