References

Abramson, Josh, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, et al. 2024. “Accurate Structure Prediction of Biomolecular Interactions with AlphaFold 3.” Nature 630 (8016): 493–500. https://doi.org/10.1038/s41586-024-07487-w.

“AlphaFold3 Why Did Nature Publish It Without Its Code?” 2024. Nature 629 (8013): 728–28. https://doi.org/10.1038/d41586-024-01463-0.

AlQuraishi, Mohammed. 2019. “ProteinNet: A Standardized Data Set for Machine Learning of Protein Structure.” BMC Bioinformatics 20 (1). https://doi.org/10.1186/s12859-019-2932-0.

Benegas, Gonzalo, Carlos Albors, Alan J. Aw, Chengzhong Ye, and Yun S. Song. 2023. “GPN-MSA: An Alignment-Based DNA Language Model for Genome-Wide Variant Effect Prediction.” http://dx.doi.org/10.1101/2023.10.10.561776.

Benegas, Gonzalo, Sanjit Singh Batra, and Yun S. Song. 2023. “DNA Language Models Are Powerful Predictors of Genome-Wide Variant Effects.” Proceedings of the National Academy of Sciences 120 (44). https://doi.org/10.1073/pnas.2311219120.

Benegas, Gonzalo, Chengzhong Ye, Carlos Albors, Jianan Canal Li, and Yun S. Song. 2025. “Genomic Language Models: Opportunities and Challenges.” Trends in Genetics, January. https://doi.org/10.1016/j.tig.2024.11.013.

Biderman, Dan, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, et al. 2024. “LoRA Learns Less and Forgets Less.” https://doi.org/10.48550/ARXIV.2405.09673.

Cheng, Xingyi, Bo Chen, Pan Li, Jing Gong, Jie Tang, and Le Song. 2024. “Training Compute-Optimal Protein Language Models.” http://dx.doi.org/10.1101/2024.06.06.597716.

Consens, Micaela Elisa, Ben Li, Anna R. Poetsch, and Stephen Gilbert. 2025. “Genomic Language Models Could Transform Medicine but Not Yet.” Npj Digital Medicine 8 (1). https://doi.org/10.1038/s41746-025-01603-4.

Dalla-Torre, Hugo, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, et al. 2024. “Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics.” Nature Methods 22 (2): 287–97. https://doi.org/10.1038/s41592-024-02523-z.

Duarte, Jose M, Rajagopal Sathyapriya, Henning Stehr, Ioannis Filippis, and Michael Lappe. 2010. “Optimal Contact Definition for Reconstruction of Contact Maps.” BMC Bioinformatics 11 (1). https://doi.org/10.1186/1471-2105-11-283.

Elnaggar, Ahmed, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, et al. 2022. “ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning.” IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (10): 7112–27. https://doi.org/10.1109/tpami.2021.3095381.

ESM Team. 2024. “ESM Cambrian: Revealing the Mysteries of Proteins with Unsupervised Learning.” EvolutionaryScale Website. https://evolutionaryscale.ai/blog/esm-cambrian.

Gao, Zhangyang, Cheng Tan, and Stan Z. Li. 2024. “FoldToken4: Consistent & Hierarchical Fold Language.” http://dx.doi.org/10.1101/2024.08.04.606514.

Gaujac, Benoit, Jérémie Donà, Liviu Copoiu, Timothy Atkinson, Thomas Pierrot, and Thomas D. Barrett. 2024. “Learning the Language of Protein Structure.” https://doi.org/10.48550/ARXIV.2405.15840.

Geiping, Jonas, and Tom Goldstein. 2022. “Cramming: Training a Language Model on a Single GPU in One Day.” https://doi.org/10.48550/ARXIV.2212.14034.

Griewank, Andreas. 2012. “Who Invented the Reverse Mode of Differentiation?” In, 389–400. EMS Press. https://doi.org/10.4171/dms/6/38.

Hassan, Hassan, Kyle Puhger, Ali Saadat, Johannes Mayer, and Maximilian Sprang. 2025. “Life as a Function: Why Transformer Architectures Struggle to Gain Genome-Level Foundational Capabilities.” http://dx.doi.org/10.1101/2025.01.13.632745.

Hayes, Thomas, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, et al. 2025. “Simulating 500 Million Years of Evolution with a Language Model.” Science 387 (6736): 850–58. https://doi.org/10.1126/science.ads0018.

Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. “LoRA: Low-Rank Adaptation of Large Language Models.” https://doi.org/10.48550/ARXIV.2106.09685.

“Initial Sequencing and Comparative Analysis of the Mouse Genome.” 2002. Nature 420 (6915): 520–62. https://doi.org/10.1038/nature01262.

Jumper, John, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, et al. 2021. “Highly Accurate Protein Structure Prediction with AlphaFold.” Nature 596 (7873): 583–89. https://doi.org/10.1038/s41586-021-03819-2.

Kempen, Michel van, Stephanie S. Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron L. M. Gilchrist, Johannes Söding, and Martin Steinegger. 2023. “Fast and Accurate Protein Structure Search with Foldseek.” Nature Biotechnology 42 (2): 243–46. https://doi.org/10.1038/s41587-023-01773-0.

King, Jonathan Edward, and David Ryan Koes. 2021. “SidechainNet: An All-Atom Protein Structure Dataset for Machine Learning.” Proteins: Structure, Function, and Bioinformatics 89 (11): 1489–96. https://doi.org/10.1002/prot.26169.

Kingma, Diederik P., and Jimmy Ba. 2014. “Adam: A Method for Stochastic Optimization.” https://doi.org/10.48550/ARXIV.1412.6980.

Lin, Zeming, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, et al. 2023. “Evolutionary-Scale Prediction of Atomic-Level Protein Structure with a Language Model.” Science 379 (6637): 1123–30. https://doi.org/10.1126/science.ade2574.

Ljungdahl, Alicia, Sayeh Kohani, Nicholas F. Page, Eloise S. Wells, Emilie M. Wigdor, Shan Dong, and Stephan J. Sanders. 2023. “AlphaMissense Is Better Correlated with Functional Assays of Missense Impact Than Earlier Prediction Algorithms.” http://dx.doi.org/10.1101/2023.10.24.562294.

Lupo, Umberto, Damiano Sgarbossa, and Anne-Florence Bitbol. 2022. “Protein Language Models Trained on Multiple Sequence Alignments Learn Phylogenetic Relationships.” Nature Communications 13 (1). https://doi.org/10.1038/s41467-022-34032-y.

Marin, Frederikke Isa, Felix Teufel, Marc Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, and Wouter Boomsma. 2023. “BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks.” https://doi.org/10.48550/ARXIV.2311.12570.

Meier, Joshua, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. 2021. “Language Models Enable Zero-Shot Prediction of the Effects of Mutations on Protein Function.” http://dx.doi.org/10.1101/2021.07.09.450648.

Nguyen, Eric, Michael Poli, Matthew G. Durrant, Brian Kang, Dhruva Katrekar, David B. Li, Liam J. Bartie, et al. 2024. “Sequence Modeling and Design from Molecular to Genome Scale with Evo.” Science 386 (6723). https://doi.org/10.1126/science.ado9336.

Nguyen, Eric, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, et al. 2023a. “HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution.” https://doi.org/10.48550/ARXIV.2306.15794.

———, et al. 2023b. “HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution.” https://doi.org/10.48550/ARXIV.2306.15794.

Patel, Aman, Arpita Singhal, Austin Wang, Anusri Pampari, Maya Kasowski, and Anshul Kundaje. 2024. “DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA.” https://doi.org/10.48550/ARXIV.2412.05430.

Poli, Michael, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. 2023. “Hyena Hierarchy: Towards Larger Convolutional Language Models.” https://doi.org/10.48550/ARXIV.2302.10866.

Rao, Roshan, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. 2020. “Transformer Protein Language Models Are Unsupervised Structure Learners.” http://dx.doi.org/10.1101/2020.12.15.422761.

Rentzsch, Philipp, Max Schubach, Jay Shendure, and Martin Kircher. 2021. “CADD-Spliceimproving Genome-Wide Variant Effect Prediction Using Deep Learning-Derived Splice Scores.” Genome Medicine 13 (1). https://doi.org/10.1186/s13073-021-00835-9.

Rives, Alexander, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, et al. 2021. “Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences.” Proceedings of the National Academy of Sciences 118 (15). https://doi.org/10.1073/pnas.2016239118.

Saharia, Chitwan, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, et al. 2022. “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.” https://doi.org/10.48550/ARXIV.2205.11487.

Schubach, Max, Thorben Maass, Lusiné Nazaretyan, Sebastian Röner, and Martin Kircher. 2024. “CADD V1.7: Using Protein Language Models, Regulatory CNNs and Other Nucleotide-Level Scores to Improve Genome-Wide Variant Predictions.” Nucleic Acids Research 52 (D1): D1143–54. https://doi.org/10.1093/nar/gkad989.

Senior, Andrew W., Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, et al. 2020. “Improved Protein Structure Prediction Using Potentials from Deep Learning.” Nature 577 (7792): 706–10. https://doi.org/10.1038/s41586-019-1923-7.

Soleymani, Farzan, Eric Paquet, Herna Lydia Viktor, and Wojtek Michalowski. 2024. “Structure-Based Protein and Small Molecule Generation Using EGNN and Diffusion Models: A Comprehensive Review.” Computational and Structural Biotechnology Journal 23 (December): 2779–97. https://doi.org/10.1016/j.csbj.2024.06.021.

Sullivan, Patrick F., Jennifer R. S. Meadows, Steven Gazal, BaDoi N. Phan, Xue Li, Diane P. Genereux, Michael X. Dong, et al. 2023. “Leveraging Base-Pair Mammalian Constraint to Understand Genetic Variation and Human Disease.” Science 380 (6643). https://doi.org/10.1126/science.abn2937.

Tang, Ziqi, Nirali Somia, Yiyang Yu, and Peter K Koo. 2024b. “Evaluating the Representational Power of Pre-Trained DNA Language Models for Regulatory Genomics.” http://dx.doi.org/10.1101/2024.02.29.582810.

———. 2024a. “Evaluating the Representational Power of Pre-Trained DNA Language Models for Regulatory Genomics.” http://dx.doi.org/10.1101/2024.02.29.582810.

Vig, Jesse, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2020. “BERTology Meets Biology: Interpreting Attention in Protein Language Models.” https://doi.org/10.48550/ARXIV.2006.15222.

Watson, Joseph L., David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, et al. 2023. “De Novo Design of Protein Structure and Function with RFdiffusion.” Nature 620 (7976): 1089–1100. https://doi.org/10.1038/s41586-023-06415-8.

Wengert, R. E. 1964. “A Simple Automatic Derivative Evaluation Program.” Communications of the ACM 7 (8): 463–64. https://doi.org/10.1145/355586.364791.

Zhang, Zhidian, Hannah K. Wayment-Steele, Garyk Brixi, Haobo Wang, Dorothee Kern, and Sergey Ovchinnikov. 2024. “Protein Language Models Learn Evolutionary Statistics of Interacting Sequence Motifs.” Proceedings of the National Academy of Sciences 121 (45). https://doi.org/10.1073/pnas.2406285121.

Zhou, Zhihan, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. 2023. “DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome.” https://doi.org/10.48550/ARXIV.2306.15006.