The rhythmic circadian clock generates temporal variation of critical physiological processes in tissues, making accurate measurement of clock phase or `tissue time' a fundamentally important problem. Recent advances in single-sample time inference provides a potentially powerful alternative to traditional time-series based approaches. Single-sample techniques typically leverage population-level RNA rhythms, but the feasibility of single-cell phase detection remains an open question. Combining multiplexed smFISH to simultaneously measure up to 6 mouse fibroblast genes, with a novel inference algorithm using Gaussian Processes, here we demonstrate that even when technical drop-outs are minimized, transcriptional noise in core-clock genes precludes single-cell phase inference. Simulations predict that above 50 `clock-like' genes would make single-cell phase inference possible. Remarkably however, just 3 core-clock genes are sufficient if RNA levels of ~70 cells are averaged. Further, we demonstrate how averaging allows detecting spatially-resolved, heterogeneous clock phases in desynchronized cells. Our work provides a conceptual framework for achieving high-resolution phase detection with a minimal set of core-clock genes, with implications for probing the origins of clock dysfunction, otherwise unresolvable using population-based approaches.