PROBE Leaderboard: Protein Representation Model Evaluation
Welcome to the PROBE (Protein RepresentatiOn BEnchmark) leaderboard! This platform evaluates protein representation models based on their ability to capture functional properties of proteins through four key benchmarks:
- Semantic Similarity Inference
- Ontology-based Function Prediction
- Drug Target Protein Family Classification
- Protein Protein Binding Affinity Estimation
Submit your own representation models and compare their performance across these benchmarks. For more details on how to participate, see the submission guidelines at Submit Here! tab. For descriptions of each benchmark and its metrics, please refer to the About tab. To visualize results, visit the Visualization tab.
If you find PROBE Leaderboard useful, please consider citing our work:
Çevrim, E., Yiğit, M. G., Ulusoy, E., Yılmaz, A., & Doğan, T. (2025). A Benchmarking Platform for Assessing Protein Language Models on Function-related Prediction Tasks. In Protein Function Prediction: Methods and Protocols (pp. 241-268). New York, NY: Springer US.
Unsal, S., Atas, H., Albayrak, M., Turhan, K., Acar, A. C., & Doğan, T. (2022). Learning functional properties of proteins with language models. Nature Machine Intelligence, 4(3), 227-245.
For detailed explanations of the metrics and benchmarks, please refer to the 📝 About tab.
Method-name colours
🟢 Classical representations
🔵 Small-scale Protein LMs
🔴 Large-scale Protein LMs
🟠 Multimodal Protein LMs
Metric-cell shading
1
2
3
4
5
top-five scores (darker → better)
Method | sim_sparse_MF_correlation | sim_sparse_CC_correlation | sim_sparse_BP_correlation | sim_sparse_Ave_correlation | sim_200_MF_correlation | sim_200_CC_correlation | sim_200_BP_correlation | sim_200_Ave_correlation | sim_500_MF_correlation | sim_500_CC_correlation | sim_500_BP_correlation | sim_500_Ave_correlation | func_MF_accuracy | func_MF_F1 | func_MF_precision | func_MF_recall | func_BP_accuracy | func_BP_F1 | func_BP_precision | func_BP_recall | func_CC_accuracy | func_CC_F1 | func_CC_precision | func_CC_recall | func_Ave_accuracy | func_Ave_F1 | func_Ave_precision | func_Ave_recall | fam_nc_f1_ave | fam_nc_accuracy_ave | fam_nc_mcc_ave | fam_uc30_f1_ave | fam_uc30_accuracy_ave | fam_uc30_mcc_ave | fam_uc50_f1_ave | fam_uc50_accuracy_ave | fam_uc50_mcc_ave | fam_mm15_f1_ave | fam_mm15_accuracy_ave | fam_mm15_mcc_ave | aff_mse_ave | aff_mae_ave | aff_corr_ave |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
SaProt-35m-af2 | -0.0121 | 0.0912 | 0.2141 | 0.0978 | 0.3881 | 0.4097 | 0.4867 | 0.5989 | 0.1611 | 0.1872 | 0.3225 | 0.1976 | 0.2394 | 0.2322 | 0.2829 | 0.2819 | 0.2629 | 0.2764 | 0.5166 | 0.3075 | 0.6178 | 0.6013 | 0.4607 | 0.5459 | 0.5508 | 0.2398 | 0.5578 | 0.5505 | 0.4354 | 0.5868 | 0.5675 | 0.4167 | 18.478 | 105.931 | 0.4605 |
If a method name ends with ^, it suggests potential suspicions of data leakage related to similarity, function, or family benchmarks.
Interactive Visualizations
Choose a benchmark type; context-specific options will appear.
🧪 Benchmark Type
X-axis Metric
Y-axis Metric
Aspect
Dataset
Metric
If a method name ends with ^, it suggests potential suspicions of data leakage related to similarity, function, or family benchmarks.
PROBE (Protein RepresentatiOn BEnchmark): Function-Centric Evaluation of Protein Representation Methods
PROBE runs benchmark analyses on protein representation/feature vectors of any representation learning method in order to evaluate its predictive performance on protein function related predictive tasks, and to and compare it other methods from literature.
Aiming to evaluate how much each representation model captures different facets of functional information, we constructed and applied 4 independent benchmark tests based on:
1. Semantic Similarity Inference:
- This benchmark evaluates how well protein representation models can infer functional similarities between proteins. Ground truth functional similarities are derived from Gene Ontology (GO) annotations.
- Different distance metrics (Cosine, Manhattan, Euclidean) are used to compute protein vector similarities, which are then correlated with the functional similarities.
- The benchmark uses three different datasets:
• Sparse Uniform: A sparse uniform dataset with broader protein coverage
• Well Annotated 200: A set of well-annotated 200 proteins
• Well Annotated 500: A set of well-annotated 500 proteins
- Metrics (sim_ prefix):
• sim_sparse_MF_correlation/sim_200_MF_correlation/sim_500_MF_correlation: Correlation between protein embeddings and Molecular Function (MF) similarity scores
• sim_sparse_BP_correlation/sim_200_BP_correlation/sim_500_BP_correlation: Correlation between protein embeddings and Biological Process (BP) similarity scores
• sim_sparse_CC_correlation/sim_200_CC_correlation/sim_500_CC_correlation: Correlation between protein embeddings and Cellular Component (CC) similarity scores
• sim_sparse_Ave_correlation/sim_200_Ave_correlation/sim_500_Ave_correlation: Average correlation across MF, BP, and CC aspects
• sim_sparse_*_pvalue/sim_200_*_pvalue/sim_500_*_pvalue: Statistical significance (p-value) of the respective correlations
2. Ontology-Based Protein Function Prediction (PFP):
- This benchmark assesses the ability of representation models to predict ontology-based functional annotations (GO terms). The models are tested on how well they classify proteins based on molecular functions, biological processes, and cellular components.
- A linear classifier is used to ensure that the models themselves are responsible for good performance, rather than the complexity of the classifier.
- The evaluation uses 5-fold cross-validation to ensure robust assessment.
- Metrics (func_ prefix):
• func_BP_accuracy/func_CC_accuracy/func_MF_accuracy: Accuracy of predicting protein function in respective GO aspects
• func_BP_F1/func_CC_F1/func_MF_F1: F1 score for protein function prediction
• func_BP_precision/func_CC_precision/func_MF_precision: Precision of function prediction
• func_BP_recall/func_CC_recall/func_MF_recall: Recall of function prediction
• func_Ave_accuracy/func_Ave_F1/func_Ave_precision/func_Ave_recall: Average metrics across BP, CC, and MF aspects
3. Drug Target Protein Family Classification:
- This benchmark focuses on predicting the family of drug target proteins (enzymes, receptors, ion channels, etc.). This task tests the ability of models to learn structural features critical to these classifications.
- The benchmark uses datasets with varying sequence similarity thresholds to test generalization:
• nc: Non-clustered/Random split
• uc50: UniClust50 (50% sequence similarity-based split)
• uc30: UniClust30 (30% sequence similarity-based split)
• mm15: Maximum 15% sequence identity between train and test sets (MMseqs2-based split)
- Metrics (fam_ prefix):
• fam_nc_*_ave: Metrics for family classification on the non-clustered (nc) dataset
• fam_uc50_*_ave: Metrics for family classification on the UniRef50 clustered (uc50) dataset
• fam_uc30_*_ave: Metrics for family classification on the UniRef30 clustered (uc30) dataset
• fam_mm15_*_ave: Metrics for family classification on the maximum 15% sequence identity (mm15) dataset
4. Protein–Protein Binding Affinity Estimation:
- This benchmark evaluates models' ability to predict the change in binding affinities between proteins due to mutations. The dataset used is the **SKEMPI** dataset, which contains experimentally determined binding affinities.
- The task measures how well models can extract critical structural features important for protein-protein interactions.
- Evaluation is performed through 10-fold cross-validation.
- Metrics (aff_ prefix):
• aff_mse_ave: Mean Squared Error for binding affinity prediction
• aff_mae_ave: Mean Absolute Error for binding affinity prediction
• aff_corr_ave: Correlation between predicted and actual binding affinity values
PROBE is part of the the study entitled Learning functional properties of proteins with language models which is schematically summarized in the figure below:
Benchmarking your own representation model
To run the benchmarks, the following representation vectors need to be generated:
- For benchmarks 1, 2, and 3 (similarity, function, and family), you will need to generate representation vectors for all human proteins. The amino acid sequences for canonical isoforms of human proteins can be found here.
- For benchmark 4 (affinity), representation vectors will need to be generated for the samples in the SKEMPI dataset, which can be accessed here.
Format of the both protein representation files:
1. Each row corresponds to the representation vector of a particular protein.
2. Rows & columns: first column's header one should be "Entry", and the rest of the column headers should contain the index number that correspond to dimensions of the vector. After the column header, the rows of the first column should contain the UniProt protein accessions of respective proteins (i.e., each row in this file corresponds to a different protein), rows of other columns should contain representation vector values for the corresponding proteins (i.e. each column in this file corresponds to a dimension of representation vectors).
3. All representation vectors in a file should have the same size (i.e., fixed sized vectors).
4. Representation vectors of the whole dataset should be saved in a comma separated (csv) text file.