Pretrained language models have billions of parameters. Fine-tuning all of them for each downstream task is computationally expensive and storage-prohibitive (a full fine-tuned 70B model is ~140 GB on disk in float16). Parameter-efficient fine-tuning (PEFT) methods reduce the cost by training only a small number of additional parameters while keeping the original weights frozen. The most widely used PEFT method is LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2021. The key observation behind LoRA: when you fine-tune a pretrained model on a new task, the updates to the weight matrices have low intrinsic rank - they can be approximated by the product of two much smaller matrices.
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
The LoRA decomposition
The key observation behind LoRA: when you fine-tune a pretrained model on a new task, the updates to the weight matrices have low intrinsic rank - they can be approximated by the product of two much smaller matrices. Take a pretrained weight matrix. Standard fine-tuning would update every entry in it directly - potentially millions of parameters per layer. LoRA instead approximates that update as the product of two much smaller matrices:
rsLoRA: a small but important fix
The original LoRA scaled the update by a factor tied to the rank, with that scaling treated as an empirical knob to tune by hand. Kalajdzievski (2023, "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA," arXiv:2312.03732) showed that the original scaling factor was wrong, and that it systematically breaks training stability at higher ranks. The fix scales the adapter's contribution down by the square root of the rank instead of the rank itself. The argument isn't about how the weights themselves behave - it's about the adapter's output during the forward pass: for training to stay stable, the size of that output needs to stay roughly constant regardless of what rank you choose, once the adapter has started learning.
LoftQ: combining LoRA with quantisation
A separate concern is memory. A 70B-parameter model in float16 takes 140 GB of GPU memory - more than any single GPU offers. The standard solution is quantisation: storing weights in lower precision (int4 or int8), reducing memory by 4x or 8x. The cost is some loss of model quality. LoftQ is a procedure for initialising LoRA adapters such that the quantised model + LoRA adapter approximates the full-precision model as closely as possible. Without LoftQ, the quantisation error and the LoRA adapter fight each other; the final model is worse than either alone. With LoftQ, the adapter compensates for the quantisation error instead, recovering near-full-precision quality at a fraction of the memory cost - letting you fine-tune a 70B model on a single 48 GB GPU, a workflow that was completely infeasible two years ago.
What this means for finance
For a trading system that wants to use a language model on domain data - earnings transcripts, broker reports, market commentary - the full PEFT stack enables: domain adaptation (fine-tuning the model on a corpus of financial text to make it more competent at financial tasks), task specialisation (training separate LoRA adapters for different tasks - summarisation, sentiment classification, named-entity recognition - and swapping them at inference time), continuous learning (re-fine-tuning cheaply as new data arrives), and local deployment (running the entire pipeline on a single GPU, avoiding the latency, cost, and privacy concerns of cloud APIs).
Worked picture
Full fine-tuning a 70B model: 8 × A100 80GB GPUs (about $240K in hardware), a training runtime of 2-4 weeks for a meaningful task, and 140 GB of storage per fine-tuned variant. LoRA fine-tuning on the same 70B model: 1 × A100 80GB with int4 + LoftQ (about $30K, or rented for roughly $3/hour), a training runtime of 4-8 hours for the same task, and just 50-200 MB of storage per fine-tuned variant (only the LoRA weights). The numbers explain why LoRA dominates in practice. The full fine-tune costs ~100× more in compute and storage for outcomes that are usually indistinguishable in downstream task performance. The only domains where full fine-tuning still wins are those requiring fundamental changes to the model's knowledge (e.g., adding a new language) - and even there, modern PEFT methods are closing the gap.
What breaks in production
A subtle trap: even an Apache-licensed model can be unusable if its training data was non-compliant or its fine-tuned variants were trained on non-commercial datasets. Always trace the provenance: 1. The base model's licence (Apache, MIT, BSD are safe; LGPL, AGPL are not for closed-source commercial use). 2. The training data's licence (Common Crawl is OK; some specialised datasets are research-only). 3. Any LoRA adapters or fine-tunes you stack on top (each has its own licence). A single non-permissive component anywhere in the stack contaminates the entire production system. The discipline: audit licences before deploying anything, and re-audit any time a model is updated. The cost of getting it wrong is potential legal exposure that dwarfs the cost of any technical mistake.
Key idea
LoRA, rsLoRA, and LoftQ make domain adaptation feasible on one GPU - low-rank updates, rank-stable scaling, quantisation-aware init. The win is iteration speed under license-safe bases, not parameter count theatre.