Herbarium specimens are broadly distributed in space and time, enabling investigation of climate impacts on phenology and fitness. We reconstructed specimen "lived" climate from knowledge of germination cues and collection dates for 14 annual species in the Streptanthus (s.l.) clade (Brassicaceae) to ask: Which climate attributes, including the timing of precipitation onset, best explain specimen phenological stage and estimated reproduction? We also asked whether climate effects on phenology and reproduction were evolutionarily conserved. Precipitation amount and onset date, more than temperature, best predicted specimen phenology, but only weakly predicted reproduction. Earlier rainfall onset was associated with more phenologically advanced specimens, a relationship that showed phylogenetic signal. Few climate predictors explained variation in specimen reproduction. The lack of association between specimen reproduction and climate may arise from phenological shifts that buffer impacts of climate, interactions with other species, or challenges in estimating total reproduction from specimens. Our results highlight the value of specimen-tailored growing season conditions for reconstructing climate, incorporating evolutionary relationships in assessing responses to climate, and the complexities of estimating fitness from specimens. For the latter, we propose supplemental herbarium collections and community science protocols to increase the utility of these data for understanding climate impacts on populations.