tmy100000001/LitDD_crossencoder

LitDD cross-encoder (v2, 2026-08-31)

Fine-tuned ncbi/MedCPT-Cross-Encoder that scores a PubMed title + abstract against a Gene2Phenotype (G2P) LGMDE thread (one gene–disease entry rendered as a 15-field string) and returns a sigmoid relevance score in [0, 1]. It was the candidate-ranking stage of the LitDD pipeline (biomedicalinformaticsgroup/LitDD_mining).

⚠️ Not required by the current pipeline (2026-09-02)

The revised pipeline is screen → gene gate → LLM adjudication: each abstract is restricted to the Gene2Phenotype entries of the genes actually mentioned in its title+abstract (mean 2.4 candidates, median 2), and all of those candidates are shown to the LLM with no score threshold. Measured on the held-out annotated test split, that gene gate offers every curated entry for 606/606 curated abstracts (retrieval recall 1.000), and removing the cross-encoder changes end-to-end performance by less than the noise (P 0.840 / R 0.848 without it vs P 0.851 / R 0.850 with it, n = 2,731 abstracts, McNemar p = 0.86).

This model is retained because it is required to reproduce the published pipeline, and it remains useful as (i) an optional ranker when an abstract yields many candidates and a cap is wanted, and (ii) a confidence signal for auditing LLM answers (keeping only answers whose entries score ≥ 0.95 raises answer precision 0.927 → 0.936 while discarding 15% of answers). It is no longer part of the deployed critical path, and no score gate is applied.

The previous release is preserved at tag v1-original. It is superseded: on the current panel rendering it calls essentially every pair positive at 0.5 (test precision 0.25, recall 1.00) and fires on 2.8% of gene-gated random PubMed abstracts at its 0.9 gate; this release fires on 0.22% with higher in-scope recall (table below).

Usage

from sentence_transformers import CrossEncoder
from litdd.threads import build_lgmde_map   # renders the thread exactly as in training

model = CrossEncoder("tmy100000001/LitDD_crossencoder")          # fp32 weights
threads = build_lgmde_map("G2P_DD_2026-06-24.csv")               # {g2p_id: thread}
score = model.predict([(title_plus_abstract, threads["G2P00003"])])[0]
passes_gate = score >= 0.9

Pair order is (abstract, thread) — a cross-encoder is not symmetric. Scoring in fp16 or bf16 (model_kwargs={"torch_dtype": torch.float16}) reproduces the fp32 predictions exactly on the held-out test set and the frozen external benchmark, so half precision is safe for corpus-scale inference. Training was fp32 (fp16 training produced an unstable seed).

Training data

• Base pairs (17,058): clinician-annotated (abstract, thread) pairs — positives and gene-sharing negatives — plus curated external positives (premined DDG2P / HPOA / ClinGen, gene-fold train split) and a molecular-register augmentation set (gene-level positives kept only for single-entry genes). Threads re-rendered from G2P DD export 2026-06-24.
• Corpus negatives (4,137 pairs): the gene-gate-passing subset (1,565) of the same 20,000 random PubMed abstracts used to train the LitDD screen, each paired with the G2P entries of the genes it mentions, labelled 0 (silver). This mirrors the screen's "add20k" recipe: the in-domain test set is blind to corpus over-prediction, so corpus fire rate is a release gate.
• Hard negatives: abhinand/MedEmbed-large-v0.1, 5 per positive, rank range 5–50, max_score 0.95, relative_margin 0.01, mined from positives only; the annotated negatives enter as labelled negatives. Final labelled-pair set: 62,223 rows (8,712 positive).

Protocol

• Leakage control: train/test split grouped by abstract (TIAB) and verified disjoint at PMID level; hard negatives mined inside each CV fold's training half; the held-out test set (2,771 pairs, 686 positive) is the same split the screen uses and was scored once.
• Hyperparameters: 5-fold StratifiedGroupKFold on the training set only, grid lr {1e-5, 2e-5, 3e-5} × epochs {1, 2}, batch 16, warmup 0.1 → lr 3e-5, 1 epoch (grid range of mean fold F1 was 0.004; the choice is non-critical).
• Thread representation: three arms (flat / + full gene name / contextualised MONDO+HPO) did not separate under CV (Friedman p = 0.074), so the flat deployment rendering ships.
• Seeds: three (42/43/44); this checkpoint is seed 42 by a pre-registered fixed-seed rule, and sits inside the seed spread on every axis below.

Results (seed 42 unless a ± is shown, which is mean ± sd over 3 seeds)

Held-out annotated test set (2,771 pairs; fixed 0.5 threshold):

model	F1	precision	recall
this release	0.931 ± 0.002 (seed 42: 0.933)	0.921	0.941
previous release (v1-original)	0.397	0.248	1.000
untrained MedCPT	0.417	0.266	0.971

Frozen external benchmark (1,924 leak-free curated truth pairs; 87,600 random PubMed abstracts as silver negatives), at the deployment gate 0.9:

model	in-scope recall (663 pairs)	gate-passing recall (1,592)	all (1,924)	corpus fire rate
this release	0.932 ± 0.009	0.576 ± 0.028	0.539 ± 0.027	0.227 ± 0.013 %
same recipe without corpus negatives	0.941 ± 0.009	0.632 ± 0.012	0.604 ± 0.010	0.294 ± 0.012 %
previous release	0.919	0.570	0.545	2.77 %
untrained MedCPT	0.971	0.749	0.702	2.37 %

"In-scope" = curated pairs whose abstract names the gene; the remainder are reviews, pre-molecular papers and gene-free abstracts that the pipeline is designed to reject, so the "all" denominator understates recall by construction. Corpus fire rate = share of the 87,600 abstracts with at least one gate candidate scoring ≥ 0.9.

Why the gate is 0.9 (revision/crossencoder_gate_analysis_fine.csv, 0.01-step scan): F1 is maximised at the lowest threshold on both the test set and the frozen benchmark, but this stage is a precision filter ahead of the LLM. The precision-weighted benchmark F0.5 rises from 0.772 (0.5) to 0.782 (0.9) and peaks at 0.789 at 0.95 (on all three seeds); every threshold from 0.89 to 0.98 is within one seed-sd of that maximum. 0.9 is retained as the recall-favouring edge of the plateau (0.9 -> 0.95 would trade 1.9 points of in-scope recall and 6 points of test recall for +1.6 points of benchmark precision and 13% fewer false fires). Test-set P/R at 0.9: 0.938 / 0.900 (3-seed means).

Top-k coverage of the true entry (held-out test positives, fp32): against the full 2,861-entry panel top-1 = 0.818, top-3 = 0.991, top-5 = 1.00 (previous release 0.818 / 0.991 / 1.00; untrained MedCPT 0.713 / 0.924 / 0.953; the submitted manuscript's Table 2 reported 0.76 / 0.97 / 0.99 as 5-fold CV means); against the 279-entry test-set pool top-1 = 0.870, top-3 = 1.00; within each abstract's own gene-gate candidates (the deployment universe, median 2 candidates) top-1 = 0.797 ± 0.013. Full table: the repository's revision/crossencoder_topk_coverage.csv.

Provenance

Trained 2026-08-31 in the LitDD pipeline image (ghcr.io/biomedicalinformaticsgroup/litdd_mining:sha-b75f6dfe…, sentence-transformers 5.7.0, transformers 5.12.0, torch 2.11 cu130) on one H100. Scripts: litdd/training/{build_crossencoder_dataset,build_corpus_negative_pairs,mine_hard_negatives,cv_hp_search_crossencoder,crossencode_finetune}.py; evaluation: litdd/evaluation/{crossencoder_frozen_eval,crossencoder_threshold_sweep,run_cross_encoder_benchmark,compare_models}.py.