The apple does not fall far from the tree is an old idiom that encapsulates a key concept: being related extends beyond merely sharing genetic material. It often implies sharing a common environment, including culture, language, dietary habits, and geographical location. In this study, we show that the analysis of genetic relatedness can serve as an indicator of health conditions by capturing the combined influences of genetic inheritance and shared environmental factors. We mapped the genomic data and electronic health records from 13k individuals in the Integrative Genomics Biorepository cohort, to neighborhood-level geographic data and integrated census-based environmental metrics. We used an identity-by-descent (IBD) based hierarchical community detection algorithm to identify four main communities closely aligning with continental ancestry, and seventeen subcommunities. We found uneven exposure to elevated environmental stressors across subcommunities, and we were able to identify subcommuinities at high risk of respiratory and dermatological conditions, demonstrating the potential of our framework and its possible application to public health interventions. Interestingly, for conditions such as congenital disorders, the most important differences were detected between subcommunities within the same community, indicating the relevance of considering a fine-grained population structure. These findings show how genetic relatedness, even at distant levels, can reflect shared environments and social determinants of health, providing a framework for understanding health disparities.