Breakthrough advances in long-read sequencing technologies have opened unprecedented opportunities to study genetic variations through comprehensive pangenome analysis. However, the availability of structural variant (SV) calling tools that can effectively leverage pangenome information is limited. In addition, efficient construction of pangenome graphs becomes increasingly challenging with acquisition of larger number of samples. In this study, we present SVPG, an approach that leverages haplotype-resolved pangenome reference for accurate SV detection and rapid pangenome graph augmentation from long-read sequencing data. Compared to state-of-the-art SV callers, SVPG maintained superior overall performance across different coverages and sequencing technologies. SVPG also achieves notable improvements in calling rare and individual-specific SVs on both simulated and real somatic datasets. Furthermore, in a benchmark involving 20 samples, SVPG accelerated pangenome graph augmentation by nearly 10-fold compared to traditional augmentation strategies. We believe that this novel SVPG method, has the potential to revolutionize SV detection and serve as an effective and essential tool, offering new possibilities for advancing pangenomic research.