Population-scale genomics information provides valuable prior knowledge for various genomic analyses, especially variant calling. A notable example of such application is the human pangenome reference released by the Human Pangenome Reference Consortium, which has been shown to improve read mapping and structural variant genotyping. In this work, we introduce pangenome-aware DeepVariant, a variant caller that uses a pangenome reference alongside sample-specific read alignments. It generates pileup images of both reads and pangenome haplotypes near potential variants and uses a Convolutional Neural Network to infer genotypes. This approach allows directly using a pangenome for distinguishing true variant signals from sequencing or alignment noise. We assessed its performance on various short-read sequencing platforms and read mappers. Across all settings, pangenome-aware DeepVariant outperformed the linear-reference-based DeepVariant, reducing errors by up to 25.5%. We also show that Element reads with pangenome-aware DeepVariant can achieve 23.6% more accurate variant calling performance compared to existing methods.