A Drug-Discovery Benchmark Shows How Much Evaluation Splits Matter

The benchmark gap

FINAL-Bench, a new Hugging Face benchmark for drug-property prediction tools, has launched with 21 leaderboards covering 18,382 held-out compounds. Its creators say the project was shaped by two findings that challenge how model performance is commonly read.

On the hERG toxicity task, a Morgan fingerprint model paired with LightGBM achieved 0.606 AUROC when trained on compounds reported before 2022 and tested on later data. Using the same molecules, features, learner and test size—but a random split—the average score rose to 0.818 across five seeds.

The difference comes from how medicinal chemistry datasets are produced. Related analogues often appear together in research records, so random splitting can place near-identical compounds in both training and test sets. A time-based split better approximates the real use case: ranking molecules whose experimental results were not available when the model was trained. FINAL-Bench therefore uses time or scaffold splits rather than random ones.

The labels are noisy, too

The team also compared measurements of the same compounds across separate publications. For hERG, the estimated single-measurement noise was 0.421 log units, while 10% of cross-document pairs differed by more than 1.338 log units. More than half of the hERG test compounds fell within one noise floor of the classification threshold, making fine-grained rankings potentially misleading.

FINAL-Bench reports regression as its primary metric and displays the noise floor alongside results. Across its boards, baseline error relative to measurement noise varied widely; on some tasks, even a constant prediction approached the apparent experimental ceiling. For AI tool builders, the message is practical: benchmark scores need their split strategy, label quality and uncertainty reported before model comparisons mean much.

Source: Hugging Face Blog