Training mRNA Language Models Across 25 Species for $165

OpenMed built an end-to-end mRNA optimization pipeline that trains transformer language models for codon optimization across 25 species, comparing architectures to achieve state-of-the-art biological codon preference prediction.

• •CodonRoBERTa-large-v2 outperformed ModernBERT with perplexity of 4.10 and Spearman CAI correlation of 0.40, demonstrating superior performance on biological codon usage patterns
• •The three-stage pipeline covers protein structure prediction using ESMFold, sequence design using ProteinMPNN, and DNA codon optimization for efficient protein expression in target organisms
• •Hyperparameter tuning proved critical; reducing learning rate from 1e-4 to 5e-5 and extending warmup increased CAI correlation 16x (0.025 to 0.404) despite slightly higher perplexity
• •RoBERTa architecture outperformed ModernBERT by 6x on perplexity (4.01 vs 26.24), indicating pre-trained NLP weights interfere with learning biological codon statistics