hard20 pts · first answer countslorafine-tuning
Counting LoRA parameters
You add a rank-16 LoRA adapter to one 4096 × 4096 weight matrix. How many trainable parameters does the adapter add, and what fraction of the full matrix is that?
Discussion 0
Join free to post in the discussion.
Related questions
- Knowledge that changes every dayAI Engineer (LLM & Generative AI)
- What LoRA trainsAI Engineer (LLM & Generative AI)
- Fine-tuned model scores too wellAI Engineer (LLM & Generative AI)
- Trainable parameters in a LoRA adapterMachine Learning Engineer
- Fine-tuning a CNN with batch size 2Machine Learning Engineer
- How DPO differs from PPO-based RLHFMachine Learning Engineer
- Fine-tuned model forgets general skillsMachine Learning Engineer
- Serving many fine-tuned variantsMLOps Engineer