NLP

Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models
A combination of Low-Rank Adaptation (LoRA) and Gaussian Stochastic Weight Averaging (SWAG) improves calibration, generalization, and robustness in Large Language Models with efficient approximate Bayesian inference.
Emre Onal, Klemens Flöge, Emma Caldwell, Arsen Sheverdin, Vincent Fortuin
PDF Code View Paper
Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models