The concept of \'Differentially Expressed Genes\' (DEGs) is central to RNA-Seq studies, yet their identification suffers from reproducibility issues. This is largely a consequence of the inherent biological and technical variation that cannot be captured with small numbers of replicates. When thresholds for p-values and log fold changes are introduced, this variability can propagate an incomplete description of the data, leading to differing interpretations. Here, we compare traditional binary DEG classification with a rank-based method, grounded in Bayesian statistics, using a published yeast dataset comprising over 40 replicates. This analysis reveals how the choice of thresholds and number of replicates results in discrepancies between studies and potentially interesting genes being overlooked. Furthermore, by comparing wild-type with wild-type samples, we show how variability in gene expression can be mistaken for differential expression. Evaluating current practices for navigating the accuracy-error trade-off in the search for differentially expressed genes leads us to advocate rank-based methods and Bayesian statistics to mitigate the limitations of binary classifications and communicate uncertainty.