Accurately and robustly representing drug molecule features, prediction of drug-target biomacromolecule interactions, and determining drug molecule physicochemical properties are crucial in drug development. However, due to issues such as insufficient generalization ability of single-modal representation, lack of multi-task prediction frameworks, and weak adaptability in cold-start scenarios, these tasks remain challenging. Here, we introduce DrugDL, a framework designed for drug molecule representation and the prediction of multiple downstream tasks, including drug-target interactions, binding affinities, binding sites, physicochemical properties, toxicity, and drug-drug interactions. DrugDL achieves joint representation learning of the drug chemical space and the target protein biological space and analyzes the multi-scale interaction mechanisms between drug molecules and target proteins by introducing cross-modal contrastive learning and single-modal feature enhancement algorithms. It employs a multi-task prediction framework to predict multiple properties of drug molecules. In practical applications, DrugDL outperforms state-of-the-art methods, especially in cold-start tasks. It\'s successfully applied to high-throughput screening, identifying inhibitors for SARS-CoV-2 and metabolic enzymes, and aids in predicting cancer-targeted drugs. Validations for EGFR and ALK targets confirm its efficiency as a precise drug discovery tool. Leveraging accurate molecular representation and multi-property prediction, DrugDL provides full-chain technical support for drug development, significantly accelerating the drug discovery process.