15-LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS大模型资料高清版.pdf

LORA:LOW-RANKADAPTATIONOFLARGELAN-

GUAGEMODELS

EdwardHuYelongShenPhillipWallisZeyuanAllen-Zhu

YuanzhiLiSheanWangLuWangWeizhuChen

MicrosoftCorporation

edwardhu,yeshe,phwallis,zeyuana,

yuanzhil,swang,luw,wzchen@

yuanzhil@

(Version2)

1

2

0

2ABSTRACT

t

cAnimportantparadigmofnaturallanguageprocessingconsistsoflarge-scalepre-

Otrainingongeneraldomaindataandadaptationtoparticulartasksordomains.As

6wepre-trainlargermodels,fullfine-tuning,whichretrainsallmodelparameters,

1becomeslessfeasible.UsingGPT-3175Basanexample–deployingindepen-

dentinstancesoffine-tunedmodels,eachwith175Bparameters,isprohibitively

]expensive.WeproposeLow-RankAdaptation,orLoRA,whichfreezesthepre-

Ltrainedmodelweightsandinjectstrainablerankdecompositionmatricesintoeach

ClayeroftheTransformerarchitecture,greatlyreducingthenumberoftrainablepa-

s.rametersfordownstreamtasks.ComparedtoGPT-3175Bfine-tunedwithAdam,

cLoRAcanreducethenumberoftrainableparametersby10,000timesandthe

[GPUmemoryrequirementby3times.LoRAperformson-parorbetterthanfine-

tuninginmodelqualityonRoBERTa,DeBERTa