Cell type-specific gene expression from single-cell RNA sequencing (RNA-seq) is valuable for breast cancer precision oncology but available cohorts are still limited due to its high cost. Deconvolution methods infer cell type-specific expression from bulk RNA-seq at a lower cost, yet expenses and processing time of bulk RNA-seq are also non-negligible and limit their application too. To address these limitations, we developed SLIDE-EX (SLide-based Inference of DEconvolved gene EXpression), a deep-learning tool that predicts cell type-specific gene expression and abundances directly from routine breast cancer histopathology whole slide images (WSIs), using deconvolved bulk RNA-seq data as training labels. Trained on the TCGA-breast cohort and tested in cross validation and on an independent cohort of 160 cases, SLIDE-EX robustly infers the expression of thousands of genes across 9 distinct cell types, performing best for cancer associated fibroblasts and cancer cells. The abundance of these two cell types could also be robustly predicted, together with that of myeloid cells. The robustly predicted genes reflect key biological functions of their respective cell types. From a translational perspective, the inferred cell type specific expression profiles predict chemotherapy response more accurately than models based on direct prediction from the slides or from the inferred bulk expression in two independent cohorts. Going forward, SLIDE-EX is a generic approach that opens up possibilities for rapid, cost-effective cell type-specific gene expression inference in potentially any cancer type, further democratizing the characterization of the tumor microenvironment.