Scoring Audit
Honest assessment of the platform's scoring metrics. V1 = baseline; V2 = Wave 1 (Boltz-2 swap); V3 = Wave 1.5 (Protenix ensemble + AbLang + N-glyco + HCDR3 dispersion + MW/pI).
After Wave 1.5: Boltz-2 + Protenix ensemble for Ab-Ag confidence; AbLang antibody-likeness prior; N-linked glycosylation flagging; HCDR3 contact-residue dispersion; MW + pI as PK proxy. 5 of 6 previously-missing indicators now present in code; DB columns added.
Status: Tamarind tool schemas (protenix / ablang / n-linked-glycosylation) + pure-Python helpers (hcdr3_contact_dispersion, mw_pi_from_sequence) + DB migration (migrate_scoring_audit_w1_5) all wired. Pipeline orchestration call-sites (scoring_stage.py) will populate the new columns on the next BCMA run.
Still open: FcRn binding for mAb PK (separate docking task), cross-species reactivity (needs ortholog data file), and the wet-lab-calibration steps from Waves 2-4 are unchanged.
Per-metric audit
What it is: The current default Interface backend produces ipTM/iPAE with self-hosted Boltz-2. These remain generated-pose confidence outputs, not binding probabilities.
Problems
- Changing the primary model does not convert pose confidence into biological binding evidence.
- Single-model still, not yet an ensemble (Wave 1 picked the highest-leverage single swap; ensemble is Wave 1+).
- Thresholds remain generic; recalibration on Boltz-2 distribution waits for the post-BCMA wet-lab batch (Wave 2).
- ipTM must not be interpreted as a binding-probability proxy; high ipTM does not establish functional binding.
World-class upgrades
| Dimension | Current | SOTA |
|---|---|---|
| Primary model | Boltz-2 (2025) — ACTIVE | Multi-model ensemble (Boltz-2 + AF3 + Chai) — Wave 1+ |
| Point vs ensemble | Single ipTM | Multi-model ensemble (AF3 + Boltz + Chai + RF2); use distribution, not point estimate |
| Threshold | Generic cutoff | Ab-Ag calibrated cutoff (e.g. BindCraft uses PAE-on-interface) |
| Secondary signal | None | Rosetta dG_separated / FoldX ΔΔG_binding / PIsToN learned model |
What it is: pLDDT now produced by Boltz-2 (same column shape). The granularity / per-region / rigidity gaps below are NOT addressed by Wave 1 — they need a separate sub-task.
Problems
- 70 as the lower bound is too low for antibody framework regions (FR). The Ig fold is among the best-trained structures in AlphaFold — FR pLDDT >90 is baseline; 70 is anomalous.
- Overall mean pLDDT is dragged down by CDR-H3, masking issues. CDR-H3 natively at pLDDT 60-80 is normal (it's inherently variable); but framework also at 70 indicates structural problems.
- No segmentation by region: FR vs CDR vs interface residues.
World-class upgrades
| Dimension | Current | SOTA |
|---|---|---|
| Granularity | Mean pLDDT | Per-region pLDDT (FR / CDR-H1/H2/H3 / LCDR separately) |
| Antibody-specific confidence | Generic AF pLDDT | ImmuneBuilder confidence (better calibrated for Ab) |
| Rigidity proxy | None | Per-residue B-factor prediction — distinguishes confident-but-flexible vs confident-and-rigid |
What it is: Unchanged from V1. Wave 1 only swapped the structure-prediction backend; the Pareto / composite scoring logic is untouched. Calibrating weights against wet-lab outcomes is Wave 2.
Problems
- Mathematically sound, but the implementation easily degenerates: if dimensions are highly correlated (e.g. binding affinity and BSA), the Pareto front degenerates to one dimension.
- If dimensions aren't properly normalized, Pareto biases toward larger-scale dimensions.
- Real problem: affinity should be weighted ≫ everything else. Pareto treats them equally, so a candidate with very high affinity but slightly worse aggregation gets 'dominated' — usually wrong.
- No wet-lab outcome feedback to calibrate weights.
World-class upgrades
| Dimension | Current | SOTA |
|---|---|---|
| Pareto ranking | Unweighted multi-objective | qNEHVI / qExpected Hypervolume Improvement (Bayesian multi-objective) |
| Composite score | Equal-weight Pareto | Lexicographic ordering: hard gates (TAP must pass) → maximize affinity → minor dims as tie-breakers |
| Learned weights | None | Preference learning — learn weights backward from wet-lab outcomes |
| Sequence prior | None | AbBFN / IgLM perplexity — 'does this sequence look antibody-like' |
What it is: Unchanged from V1. Wave 1 does not touch CAR-specific composite scoring — that needs IASO internal data (Wave 3).
Problems
- No validated CAR-fit predictor exists in the public literature. This is an open problem.
- Our composite weights are hand-tuned, not learned, and have never been benchmarked against real clinical CAR-T performance (CTL019 / Carvykti / Abecma / 福可苏).
- 'CAR-T compatibility' means very different things across ex-vivo CAR / in-vivo CAR / bispecific TCE — a single score will mislead.
World-class upgrades
| Dimension | Current | SOTA |
|---|---|---|
| Train an internal CAR-fit predictor | Hand-tuned composite | The most direct landing for internal-data fine-tuning. Inputs: CAR construct (VHH/scFv + hinge + TM + costim + CD3ζ). Labels: T-cell surface flow cytometry / tonic-signaling pathway readouts / cytotoxicity assay outcomes. |
| Basic structural filter | Ad hoc | TM hydrophobicity + isoelectric point + molecular weight as a hard gate |
| Format-specific predictors | One score for all CAR formats | BiTE/TCE use BiteScore; CAR uses CAR-NET-style specialists; don't conflate |
| Honesty labeling | Composite presented as if validated | Mark explicitly as 'heuristic v0' until calibrated |
What it is: Unchanged from V1. Wave 1 does not touch tonic clustering — that needs IASO internal data (Wave 3).
Problems
- First-principles heuristic with zero experimental calibration.
- Published in-silico predictors for tonic signaling don't have a gold standard. The Carvykti boxed warning was given retrospectively, not predicted.
- Our claim that 'Carvykti should score higher than 福可苏' is a comparative-logic inference, not statistical validation.
World-class upgrades
| Dimension | Current | SOTA |
|---|---|---|
| Validated aggregation predictors | Custom composite | SAP (Spatial Aggregation Propensity, Voth lab standard); TAP v2; AggreScan 4D (dynamics-aware) |
| Stability proxy | None | HDX (hydrogen-deuterium exchange) prediction |
| Real calibration path | None | Take internal CAR T-cell surface flow cytometry + tonic-pathway readouts on ~10-20 variants and use as supervised labels — the benchmark dataset asked for in internal review. |
| What's missing | Why it matters |
|---|---|
| AF3 / Boltz / Chai multi-model ensemble | Single-model confidence is unreliable; multi-model agreement is a stronger signal |
| Antibody-likeness prior (IgLM perplexity / AbBFN likelihood) | Does this sequence look like a real antibody from the natural repertoire — strongly correlates with expressibility |
| Glycosylation site flagging (NetNGlyc / DeepGlyco) | Required for mAb format; affects PK and aggregation |
| PK half-life proxy (FcRn binding for mAb; MW + pI for TCE) | Determines dosing regimen |
| HCDR3 contact-residue dispersion | Rationality check on the high-variability loop's contact distribution |
| Cross-species reactivity (human vs cyno vs mouse) | Required for preclinical models |
World-class scoring lands in waves. Each wave compounds on the previous.
| Wave | Timing | Content | Difficulty |
|---|---|---|---|
| Wave 1✓ Landed | Landed 2026-05-21 (config-flag gated; sanity-check pending) | Boltz-2 swap as primary Ab-Ag interface scorer (replaces AF-Multimer). Per-region pLDDT + heuristic-v0 labels are separate small tasks, not bundled. | Medium |
| Wave 1.5✓ Landed | Backend landed 2026-05-21 | Closes 5 of 6 previously-missing indicators: Protenix as ensemble partner to Boltz-2; AbLang antibody-likeness prior; N-linked glycosylation flagging (Tamarind + motif fallback); HCDR3 contact-residue dispersion (pure Python on existing PDBs); MW + pI as PK proxy. DB columns added; pipeline orchestration call-sites still to wire on next pass. | Medium |
| Wave 2 | After BCMA wet-lab (~2026-06-14+) | Use IASO's first wet-lab batch to calibrate Boltz-2 thresholds; tune Pareto weights and CAR/tonic heuristics against ground truth | Medium-High |
| Wave 3 | 1-2 months | Train IASO-internal CAR-fit / expression / tonic predictors (leverage historical Carvykti / 福可苏 data + new wet-lab data) | High |
| Wave 4 | 3+ months | Fine-tune ProteinMPNN / RFantibody on internal sequences | High; depends on Modal self-host (Spec 3 S.1) |
- pipeline/tamarind/schemas.py — new
Boltz2Paramsdataclass - pipeline/tamarind/tools.py — new
build_boltz2()+ registry entry"boltz" - pipeline/stages/validation_stage.py — dispatch on
config.complex_predictor("alphafold" default | "boltz") - Tamarind tool: `boltz` (Boltz-2 v2.2.1, from jwohlwend/boltz). Input format: sequence with `:` separator for multimers (matches existing AF-Multimer convention).
Sanity check pre-cutover: run a side-by-side on the existing 30 BCMA candidates (config complex_predictor=boltz vs alphafold); confirm Spearman ρ ≥ 0.7 between rankings + Boltz-2 top-10 overlap ≥ 80% with AF-Multimer top-10. Once that passes, change the default in validation_stage.py.