Low Rank Adaptation Enables Efficient Domain Transfer in Billion Parameter Language Models

Jiesi Yang

doi:10.54097/6w0gxa44

Authors

Jiesi Yang

DOI:

https://doi.org/10.54097/6w0gxa44

Keywords:

Low-rank adaptation, Parameter-efficient fine-tuning, Large language models, Domain transfer, LoRA, Billion-parameter models, NLP, Transfer learning

Abstract

The rapid growth of billion-parameter language models has transformed natural language processing (NLP), yet deploying these models across specialized domains remains constrained by the prohibitive cost of full fine-tuning. Low-rank adaptation (LoRA) has emerged as a leading parameter-efficient fine-tuning (PEFT) approach that restricts weight updates to low-dimensional matrix products, dramatically reducing trainable parameter counts without sacrificing downstream performance. This review synthesizes theoretical foundations, algorithmic advances, and empirical findings concerning LoRA and its derivatives as applied to large language models (LLMs) in domain transfer settings. We examine how rank decomposition enables adaptation across biomedical text mining, legal document analysis, code generation, and financial analytics, surveying evidence that LoRA-based methods match full fine-tuning while updating fewer than one percent of total model parameters. Variants including QLoRA, AdaLoRA, LoRA+, and DoRA are analyzed with respect to their contributions in quantization, adaptive rank allocation, and optimization refinement. The theoretical basis in intrinsic dimensionality that justifies low-rank approximations is discussed alongside practical considerations of rank selection, target module choice, and multi-task deployment. By consolidating findings from sixty recent studies, this paper offers a structured understanding of when and why LoRA succeeds, identifies persistent limitations, and delineates promising directions for future work in efficient domain transfer.

Downloads

Download data is not yet available.

References

[1] Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., ... & Wen, J. R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 1-124.

[2] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[3] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140), 1-67.

[4] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2022). Lora: Low-rank adaptation of large language models. Iclr, 1(2), 3.

[5] Aghajanyan, A., Gupta, S., & Zettlemoyer, L. (2021, August). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers) (pp. 7319-7328).

[6] Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021, April). Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume (pp. 487-503).

[7] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36, 10088-10115.

[8] Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., ... & Zhao, T. (2023). Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.

[9] Zeng, Y., & Lee, K. (2023). The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513.

[10] Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., ... & Natarajan, V. (2023). Large language models encode clinical knowledge. Nature, 620(7972), 172-180.

[11] Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., ... & Synnaeve, G. (2023). Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.

[12] Hankins, N. (2024, June). Optimizing multilingual euphemism detection using low-rank adaption within and across languages. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024) (pp. 8-14).

[13] Yu, F., Xiu, X., & Li, Y. (2022). A survey on deep transfer learning and beyond. Mathematics, 10(19), 3619.

[14] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).

[15] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

[16] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., ... & Gelly, S. (2019, May). Parameter-efficient transfer learning for NLP. In International conference on machine learning (pp. 2790-2799). PMLR.

[17] Li, X. L., & Liang, P. (2021, August). Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 4582-4597).

[18] Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., ... & Sun, M. (2022). Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904.

[19] He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021). Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.

[20] Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., ... & Du, M. (2024). Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2), 1-38.

[21] Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., ... & Natarajan, V. (2025). Toward expert-level medical question answering with large language models. Nature medicine, 31(3), 943-950.

[22] Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D., & Aletras, N. (2022, May). LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4310-4330).

[23] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

[24] Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., ... & Mann, G. (2023). Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564.

[25] Pfeiffer, J., Vulić, I., Gurevych, I., & Ruder, S. (2020, November). Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) (pp. 7654-7673).

[26] Zeng, Z., Lin, H., Zhang, S., & Wang, B. (2026). Adaptive Robust Watermarking for Large Language Models via Dynamic Token Embedding Perturbation. IEEE Access, 14, 9319-9339.

[27] Hayou, S., Ghosh, N., & Yu, B. (2024). Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354.

[28] Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024, July). Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning.

[29] Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., & Tian, Y. (2024). Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507.

[30] Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2022). Editing models with task arithmetic. arXiv preprint arXiv:2212.04089.

[31] Huang, C., Liu, Q., Lin, B. Y., Pang, T., Du, C., & Lin, M. (2023). Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269.

[32] Liu, Z., & Luo, J. (2024). Adamole: Fine-tuning large language models with adaptive mixture of low-rank adaptation experts. arXiv preprint arXiv:2405.00361.

[33] Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., ... & Huang, X. J. (2024, August). LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1932-1945).

[34] Jia, M. (2025). Optimization of artificial intelligence natural language processing model based on deep neural network. IEEE Access.

[35] Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.

[36] Xu, L., Xie, H., Qin, S. J., Tao, X., & Wang, F. L. (2026). Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37] Cartis, C., Massart, E., & Otemissov, A. (2023). Bound-constrained global optimization of functions with low effective dimensionality using multiple random embeddings. Mathematical Programming, 198(1), 997-1058.

[38] Sharma, P., Ash, J. T., & Misra, D. (2023). The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558.

[39] Biderman, D., Portes, J., Ortiz, J. J. G., Paul, M., Greengard, P., Jennings, C., ... & Cunningham, J. P. (2024). Lora learns less and forgets less. arXiv preprint arXiv:2405.09673.

[40] Zhang, H., Huang, B., Li, Z., Xiao, X., Leong, H. Y., Zhang, Z., ... & Xu, H. (2025). Sensitivity-lora: Low-load sensitivity-based fine-tuning for large language models. arXiv preprint arXiv:2509.09119.

[41] Panigrahi, A., Shetty, A., & Goyal, N. (2019). Effect of activation functions on the training of overparametrized neural nets. arXiv preprint arXiv:1908.05660.

[42] Kopiczko, D. J., Blankevoort, T., & Asano, Y. M. (2023). Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454.

[43] Yadav, P., Tam, D., Choshen, L., Raffel, C. A., & Bansal, M. (2023). Ties-merging: Resolving interference when merging models. Advances in neural information processing systems, 36, 7093-7115.

[44] Valipour, M., Rezagholizadeh, M., Kobyzev, I., & Ghodsi, A. (2023, May). DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3274-3287).

[45] Zhou, L., Suominen, H., & Gedeon, T. (2019). Adapting state-of-the-art deep language models to clinical information extraction systems: Potentials, challenges, and solutions. JMIR medical informatics, 7(2), e11499.

[46] Luo, Y., Zhang, J., Fan, S., Yang, K., Wu, Y., Qiao, M., & Nie, Z. (2023). Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442.

[47] Anoop, V. S., & Fawaz, J. M. Fine-Tuning a Domain-Specific Large Language Model Using Low-Rank Adaptation Technique for Legal AI Applications: Case of India. In Artificial Intelligence in Legal Systems (pp. 63-76). Chapman and Hall/CRC.

[48] Liu, X. Y., Wang, G., Yang, H., & Zha, D. (2023). Fingpt: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485.

[49] Adelani, D. I., Neubig, G., Ruder, S., Rijhwani, S., Beukman, M., Palen-Michel, C., ... & Klakow, D. (2022, December). Masakhaner 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 4488-4508).

[50] Hudeček, V., & Dušek, O. (2023, September). Are large language models all you need for task-oriented dialogue?. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue (pp. 216-228).

[51] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

[52] Cui, Y., Sun, Z., & Hu, W. (2024). A prompt-based knowledge graph foundation model for universal in-context reasoning. Advances in Neural Information Processing Systems, 37, 7095-7124.

[53] Chronopoulou, A., Peters, M. E., Fraser, A., & Dodge, J. (2023, May). Adaptersoup: Weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023 (pp. 2054-2063).

[54] Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Kornblith, S., ... & Schmidt, L. (2022). Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35, 29262-29277.

[55] Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022, July). Metaicl: Learning to learn in context. In Proceedings of the 2022 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2791-2809).

[56] Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., ... & Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.

[57] Zhao, X., Ge, Y., & Zhang, H. (2025). Cost Efficient Scaling Strategies for Large Language Models in Multi-Cloud Environment. Mathematical Modeling and Algorithm Application, 6(2), 101-110.

[58] Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023). Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071.

[59] Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning. Advances in neural information processing systems, 36, 34892-34916.

[60] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023). Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15180-15190).