Larry Law

Larry Law

Member of Technical Staff at DSO National Laboratories

Given the same compute budget, which is better: parameter efficient finetuning (PEFT) or full finetuning?

less than 1 minute read

Published:

Motivation

If we are allowed to train till convergence, we know that full finetuning is better than parameter efficient finetuning (PEFT). But what if we have a fixed compute budget? Given a fixed budget, PEFT can go through significantly more tokens. Will full finetuning still be better than PEFT?

For my research problem, it turns out full finetuning is still better than PEFT.

Experiment Setup

For context, my research problem is to adapt english LLMs to other languages. I finetuned llama-7B on roots_* datasets (datasets used to train BLOOM) and open subtitles. For PEFT, I used LoRA and additionally trained the embedding and LM head.

Results

Loss plots for finetuning vs peft

Here, we can see that full finetuning (blue) achieves a better loss than PEFT (orange) given the same compute budget! Seems like finetuning can - ironically - be more efficient that PEFT. Interesting!

From my colleagues’ experiments, turns out PEFT only speeds up 25% compared to finetuning despite training <1% of parameters.

You May Also Enjoy

LLM Theory

less than 1 minute read

Published:

Awesome LLM theory articles I chanced upon

Simple tricks to improve Retrieval Augmented Generation (RAG) systems

3 minute read

Published:

Motivation

Retrieval Augmented Generation (RAG) is a framework for retrieving facts from an external knowledge base to ground large language models (LLMs) on the most accurate, up-to-date information. RAG is increasingly popular in industry as it’s simple to implement yet powerful. Here I’ll share some tricks to improve RAG systems.

Simple solution to training causal LMs with seq2seq objective

2 minute read

Published:

Motivation

How do we train causal language models (e.g. Alpaca, LLaMA, gpt-neox-20b…) with seq2seq objective? This goal is important because we want to instruction-tune our causal LMs, especially since Alpaca is the best open model at time of writing.