Allosteric modulators as therapeutics offer many advantages over orthosteric modulators, including improved selectivity and tunability. However, identifying and characterising allosteric sites remains a major challenge both experimentally and computationally. Accurate prediction of allosteric binding sites is critical to facilitate allosteric drug discovery. Here, we evaluate three strategies to predict allosteric sites using pre-trained protein language models (pLMs), Ankh, ProtT5 and ESM-2, which are trained on sequence information alone. First, a classifier was trained using static embeddings extracted from each pLM. Second, parameter-efficient fine-tuning was implemented using LoRA with focal loss to account for the sparse positive labels and lack of data. Finally, a structure-aware conditioning mechanism was introduced, whereby orthosteric binding sites are encoded and integrated directly into the input embeddings to enable the model to predict allosteric binding sites, conditioned on the knowledge of the orthosteric binding pocket. This approach, which captures functional dependencies between the othosteric and allosteric binding sites, improves upon the performance of the first two methods and achieves results comparable to the leading structure-based allosteric site predictors.