Quantifying species' niches across a clade reveals how environmental tolerances evolve, and offers insights into present and future distributions. We use herbarium specimens to explore climate niche evolution across 14 annual species of the Streptanthus (s.l.) clade (Brassicaceae), which originated in deserts and diversified into cooler, moister areas. To understand how climate niches evolved, we used historical climate records to estimate each species' 1) classic annual climate niche, averaged over specimen collection sites; 2) growing season niche, from estimated specimen germination date to collection date, averaged across specimens (specimen-specific niche); and 3) standardized seasonal niche based on average growing seasons of all species (clade-seasonal niche). In addition to estimating how phenological variation maps onto climate niche evolution, we explored how spatial refugia shape the climate experienced by species by 1) analyzing how field soil texture changes relative to the climate space that species occupy and 2) comparing soil water holding capacity from each specimen locality to that of surrounding areas. Specimen-specific niches exhibited less clade-wide variation in climatic water deficit than did annual or clade-seasonal niches, and specimen-specific temperature niches showed no phylogenetic signal, in contrast to annual and clade-seasonal temperature niches. Species occupying cooler regions tracked hotter and drier climates by growing later into the summer, and by inhabiting refugia on drought-prone soils. These results underscore how phenological shifts, spatial refugia, and germination timing shape "lived" climate. Despite occupying a large range of annual climates, we found these species are constrained in the conditions under which they thrive.