The exploitation of heterosis in rice, namely hybrid rice, has significantly improved global food production. However, the molecular mechanisms underlying rice heterosis, especially in heterosis from inter-subspecies, remain poorly understood. Here, we assemble high-quality genomes for 12 representative indica and japonica rice. Comparative genomic analyses reveal abundant genetic variations and highlight pivotal roles of structural variants (SVs) in rice subspeciation. The numbers of SVs between parental lines are significantly correlated with heterosis across 17 agronomic traits, with distinct correlation patterns for inter- and intra-subspecific F1 hybrids. We detect SVs in S5-ORF5 and functionally validated their roles in heterosis for seed setting rate, underscoring the importance of SVs in breeding inter-subspecific hybrid rice. The SVs, especially inversions that are mostly located in exonic regions, drive gene expression changes that account for heterosis variation. Our findings demonstrate the contributions of SVs to subspeciation and gene expression variations for harnessing rice heterosis.