2025 Hyper Recent •CC0 1.0 Universal

This work is dedicated to the public domain. No rights reserved.

Access Preprint From Server
April 25th, 2025
Version: 2
University of Southern California
bioinformatics
biorxiv

A benchmark for large language models in bioinformatics

Sarwal, V.Open in Google Scholar•Andreoletti, G.Open in Google Scholar•Munteanu, V.Open in Google Scholar•Suhodolschi, A.Open in Google Scholar•Ciorba, D.Open in Google Scholar•Bostan, V.Open in Google Scholar•Dimian, M.Open in Google Scholar•Eskin, E.Open in Google Scholar•Wang, W.Open in Google Scholar•Mangul, S.Open in Google Scholar

The rapid advancements in artificial intelligence, particularly in Large Language Models (LLMs) such as GPT-4, Gemini, and LLaMA, have opened new avenues for computational biology and bioinformatics. We report the development of BioLLMBench, a novel framework designed to evaluate LLMs in bioinformatics tasks. This study assessed GPT-4, Gemini, and LLaMA through 2,160 experimental runs, focusing on 24 distinct tasks across six key areas: domain expertise, mathematical problem-solving, coding proficiency, data visualization, research paper summarization, and machine learning model development. Tasks ranged from fundamental to expert-level challenges, and each area was evaluated using seven specific metrics. A Contextual Response Variability Analysis was implemented to understand how model responses varied under different conditions. Results showed diverse performance: GPT-4 led in most tasks, achieving a 91.3% proficiency in domain knowledge, while Gemini excelled in mathematical problem-solving with a 97.5% proficiency score. GPT-4 also outperformed in machine learning model development, though Gemini and LLaMA struggled to generate executable code. All models faced challenges in research paper summarization, scoring below 40% using the ROUGE metric. Model performance variance increased when using a new chat window, though average scores remained similar. The study also discusses the limitations and potential misuse risks of these models in bioinformatics.

Similar Papers

biorxiv
Fri Apr 25 2025
BindCraft: one-shot design of functional protein binders
Protein-protein interactions (PPIs) are at the core of all key biological processes. However, the complexity of the structural features that determine PPIs makes their design challenging. We present BindCraft, an open-source and automated pipeline for de novo protein binder design with experimental success rates of 10-100%. BindCraft leverages the weights of AlphaFold2 to generate binders with nan...
Pacesa, M.
•
Nickel, L.
•
Schellhaas, C.
•
Schmidt, J.
...•
Correia, B. E.
biorxiv
Fri Apr 25 2025
Back to Basics: Spectrum and Peptide Sequence are Sufficient for Top-tier Mass Spectrometry Proteomics Identification
The original mass spectrometry search engines used simple algorithms for peptide identification. Recent tools improved accuracy by adding several extra components such as fragment ion intensities or retention times prediction and training target-decoy classifiers on-the-fly, leading to sometimes inconsistent results. Our study explores the impact of replacing those extra components with a deep-lea...
Burq, M.
•
Stepec, D.
•
Restrepo, J.
•
Zbontar, J.
...•
Cimermancic, P.
biorxiv
Fri Apr 25 2025
Shiny-Calorie: A context-aware application for indirect calorimetry data analysis and visualization using R
Indirect calorimetry is a cornerstone technique for metabolic phenotyping of animal models in preclinical research, with well-established experimental protocols and platforms. However, a flexible, extensible, and user-friendly software suite that enables standardized integration of data and metadata from diverse metabolic phenotyping platforms - followed by unified statistical analysis and visuali...
Grein, S.
•
Elschner, T.
•
Kardinal, R.
•
Bruder, J.
...•
Klingenspor, M.
biorxiv
Fri Apr 25 2025
MiT4SL: multi-omics triplet representation learning for cancer cell line-adapted prediction of synthetic lethality
Synthetic lethality (SL) offers a promising approach for targeted cancer therapies. Current SL prediction models heavily rely on extensive labeled data for specific cell lines to accurately identify SL pairs. However, a major limitation is the scarcity of SL labels across most cell lines, which makes it challenging to predict SL pairs for target cell lines with limited or even no available labels ...
Tao, S.
•
Feng, Y.
•
Yang, Y.
•
Wu, M.
•
Zheng, J.
biorxiv
Thu Apr 24 2025
One-Hot News: Drug Synergy Models Take a Shortcut
Combinatorial drug therapy holds great promise for tackling complex diseases, but the vast number of possible drug combinations makes exhaustive experimental testing infeasible. Computational models have been developed to guide experimental screens by assigning synergy scores to drug pair-cell line combinations, where they take input structural and chemical information on drugs and molecular featu...
Candir, E. B.
•
Kuru, H. I.
•
Rattray, M.
•
Cicek, A. E.
•
Tastan, O.
biorxiv
Thu Apr 24 2025
Ontology-aware DNA methylation classification with a curated atlas ofhuman tissues and cell types
DNA methylation (DNAm) is a core gene regulatory mechanism that captures cellular responses to short- and long- term stimuli such as environmental exposures, aging, and cellular differentiation. Although DNAm has proven valuable as a baseline biomarker for aging by enabling robust characterization of disease-associated methylation shifts associated with age, its potential to reveal analogous shift...
Kim, M.
•
Dannenfelser, R.
•
Cui, Y.
•
Allen, G.
•
Yao, V.
biorxiv
Thu Apr 24 2025
SCassist: An AI Based Workflow Assistant for Single-Cell Analysis
Single-cell RNA sequencing (scRNA-seq) data analysis often involves complex iterative workflow, requiring significant expertise and time. To navigate this complexity, we have developed SCassist, an R package that leverages the power of the large language models (LLM\'s) to guide and enhance scRNA-seq analysis. SCassist integrates LLM\'s into key workflow steps, to analyze user data and provide rel...
Nagarajan, V.
•
Shi, G.
•
Arunkumar, S.
•
Liu, C.
...•
Caspi, R.
biorxiv
Thu Apr 24 2025
Inferring the cell type-specific functional roles of nucleic acid-binding proteins using deep learning on co-expression networks
The regulatory target genes and functional roles of nucleic acid-binding proteins (NABPs), including DNA- and RNA-binding proteins, can vary in a cell- or tissue-specific manner. However, experimentally identifying NABP-binding sites and functionally characterizing these proteins remain technically challenging. Deep learning methods have been developed to predict gene expression levels across dive...
Osato, N.
biorxiv
Thu Apr 24 2025
Optimizing drug synergy prediction through categorical embeddings in Deep Neural Networks
Cancer treatments often lose effectiveness as tumors develop resistance to single-agent therapies. Combination treatments can overcome this limitation, but the overwhelming combinatorial space of drug-dose interactions makes exhaustive experimental testing impractical. Data-driven methods, such as Machine and Deep Learning, have emerged as promising tools to predict synergistic drug combinations. ...
Gonzalez Lastre, M.
•
Gonzalez de Prado Salas, P.
•
Guantes, R.