Protein language models don't learn meaningful representations of CDRH3 loops

A recent paper found that protein language models (PLMs) organize their per-protein representations into a relatively compact portion of their latent space, leaving other parts of the representation space filled with what the authors call a "junkyard" of unsortable sequences^[1].

Astral40R refers to proteins whose sequences have been scrambled. Figure from ^[1:1]

A similar albeit inverted process was observed in the structure prediction neural network AlphaFold3. Low-confidence predictions, obtained with multiple sequence alignments that were either missing or corrupted, or sequences with tons of mutations, were packed into a specific representation subspace not shared by sequences that could be correctly folded^[2]:

Figure from ^[2:1]

I wanted to test out a variation of the PLM junkyard hypothesis on antibodies using various protein language models, some of which were specifically trained on antibodies and some of which were not. The main idea is that we can infer if meaningful representations of residues in the CDRH3 loop are being obtained by comparing mean-pooled representations of CDRH3 residues to those obtained from a scrambled sequence grafted onto the same framework. In general, only one model, VHH-BERT, appeared to distinguish between scrambled and unscrambled CDRH3 sequences, and in that case it was only when parsing VHHs similar to what it has seen during training.

There are two conclusions I derive from this. The first is that even specialist models aren't learning a neat manifold of what constitutes a realistic CDRH3 equivalent to full-length sequences of other proteins, since these patently unrealistic CDRH3 sequences are co-clustering right alongside them. Top-off layers (like MLPs, etc) trained on these representations will, without the user's knowledge, be crude models of permutation-invariant sequence composition. That's almost certainly not expressive enough for predicting properties that show sensitivity to epistasis, such as expression and thermostability^[3], let alone antigen binding.

The second has to do with the brittleness of how these models define "fitness", particulaly in generalist protein language models such as ESM-C. Like non-antibody proteins, extant germline antibody sequences are the product of selective pressure; unlike non-antibody proteins, that pressure has driven them toward greater evolvability for subsequent somatic hypermutation. In other words, germline antibody fitness includes a measure of landscape flatness and accessibility^[4]. The fact that mean-pooled representations of naturally occurring (or engineered) CDRH3s are nearly identical to those with the same amino acid composition suggests that this key selective endpoint cannot be identified by any of these models. Instead, as with other challenges inherent to antibody sequence modeling^[5], bespoke solutions will probably be required.

References