May 06, 2021 \ IPU-POD, Natural Language Processing
瓿奠湢:
鞐呺嵃鞚错姼顷橁赴BERT电� 鞝曧檿靹标臣 鞙犾棸靹� 雿曤秳鞐� 鞓る姌雮� 臧赌鞛� 雱愲Μ 靷毄霅樀鐢 鞛愳棸巩� 觳橂Μ 氇嵏 欷� 顷橂倶鞛呺媹雼�. 攴鸽Μ瓿� 攴鸽灅顶勋綌巩挫潣 瓿犼皾霌れ澊 臧赌鞛� 毵库澊 鞖旍箔頃樀鐢 氇嵏 欷� 顷橂倶鞚搓赴霃� 顷╇媹雼�.
雼轨偓 鞐旍雼堨柎霌れ澏� 甏戨矓鞙勄隃 巩胳柎 旮半皹 鞚戩毄 攵勋暭毳� 歆赌鞗愴晿旮� 鞙勄甏 IPU 鞁涤姢韰涤毄 BERT毳� 甑槃顷橁碃 斓涤爜頇旐晿鞓赌鞙茧┌, 靷办梾 响涤 毹胳嫚霟嫕 甑愳湣 觳搓硠毳� 靷毄顷橃棳 鞖办垬顷� 觳橂Μ霟夓潉 鞛呾頄堨姷雼堧嫟.
攴鸽灅靹� 攴鸽灅顶勋綌巩搓岸� IPU鞕赌 BERT 氇嵏 顷欖姷鞐� 甏赌顷� 鞐办灛 韽姢韺呾潉 旮绊殟顷� 氪れ姷雼堧嫟. 攴� 觳� 氩堨Ц鞚� 鞚措矆 韽姢韺呾棎靹湹鐢 攴鸽灅顶勋綌巩挫潣 鞎犿攲毽紑鞚挫厴, 靻岉攧韸胳洦巩� 氚� 欷戧淡 鞐旍雼堨柎毵� 韺赌鞚� 鞐旍雼堨柎霌れ澊 BERT-Large 甑槃鞚� 鞙勄甏 靷爠 顷欖姷 瓿检爼鞚� 瓯办箻瓿� 韺岇澑 韸滊嫕頃樀鐢雿� 靷毄顷� 斓涤爜頇� 旮办垹鞐� 雽赌顷� 鞛愳劯颂� 鞎岇晞氤搓矤鞀惦媹雼�.
BERT鞚� 攵赌靸�
BERT(Bidirectional Encoder Representations from Transformers)电� 甑竴(Google)鞚� 毵岆摖 韸鸽灉鞀ろ彫毹� 旮半皹鞚� 巩胳柎 响滍槃 氇嵏搿�, 2018云� 毵愳棎 於涤嫓霅� 鞚错泟 靹犿拲鞝侅澑 鞚戈赴毳� 雭岅碃 鞛堨姷雼堧嫟.
鞐� 雽赌顷� 歆勴枆顷� 齑堦赴 鞁滍棙鞚� 瓯办硱, 鞓る姌雮� BERT鞐� 靷毄霅橁碃 鞛堨姷雼堧嫟.
BERT 氚� 攴� 韺岇儩 氇嵏霌れ澏� 瓴赌靸� 鞐旍肟愲 鞎勲媹霛� 歆堨潣 鞚戨嫷, 旖橅厫旄� 旮半皹 於旍矞, 牍勲敂鞓� 鞚辞甏 氚� 雼氨歆� 韸轨 於旍稖瓿� 臧欖澏� 雼るジ 鞓侅棴鞐愲弰 鞝侅毄霅橃柎 氩旍毄靹膘潉 鞛呾頄堨姷雼堧嫟.
GPT(靸濎劚鞝� 靷爠顷欖姷 韸鸽灉鞀ろ彫毹�, GPT)鞕赌 臧欖澏� 旮办〈鞚� 韸鸽灉鞀ろ彫毹� 氇嵏瓿� 雼るゴ瓴�, BERT电� 氇摖 韸鸽灉鞀ろ彫毹� 霠堨澊巩挫棎靹� 霠堨澊敫旊悩歆赌 鞎婌澏� 臧� 顷欖姷 鞓堨牅鞚� 膦岇毎 氍鸽Д鞚� 頇涤毄顷橃棳 响滍槃鞚� 巩戨癌頄レ溂搿� 须臣鞝侅溂搿� 瓴赌靸夗晿霃勲 靹り硠霅橃棃鞀惦媹雼�. 鞚措煬顷� 巩戨癌頄レ劚鞚赌 雼れ枒顷� 雼れ毚鞀ろ姼毽� 鞛戩梾鞐� 雿� 韥� 鞙犾棸靹膘潉 攵�鞐戛雼堧嫟.
靷爠 顷欖姷 雼硠鞐愳劀 霠堨澊敫旊悩歆赌 鞎婌澏� 瓯半寑顷� 雿办澊韯� 靹疙姼毳� 靷毄頃樀鐢 BERT鞚� 旮半姤(攴鸽Μ瓿� 氙胳劯 臁办爼 雼硠鞐愳劀 霠堨澊敫旍澊 歆赌鞝曤悳 鞝侅澏� 巩戩潣 雿办澊韯半鞙茧 斓涤波雼潣 鞝曧檿霃勲ゼ 雼劚顷� 瓴�)鞚赌 BERT鞕赌 鞙犾偓顷� 雽赌攴滊 韸鸽灉鞀ろ彫毹� 旮半皹鞚� 巩胳柎 氇嵏霌れ潉 毵れ毎 毵る牓鞝侅溂搿� 氤挫澊瓴� 顷� 欷嶋媹雼�. 攴胳棎 霐半澕 鞚挫矘霟� 雽赌攴滊鞚� 鞁犼步毵� 巩胳柎 氇嵏鞚� 顷欖姷鞁滍偆瓿� 韺岇澑 韸滊嫕頃樀鐢 靾橃殧臧赌 韽顷橁碃 鞛堨姷雼堧嫟.
巩胳柎 氇嵏毵侁臣 臧欖澏� 雽赌攴滊 鞛戩梾 肟愲 鞎勲媹霛� . 鞓堧ゼ 霌れ柎 鞚� 顷欖姷 鞓堨牅电� 3600臧滊皷鞐� 鞐嗢毵�, BERT-Base鞐愳劀 BERT-Large搿� 氇嵏 韥赴毳� 韨れ泙鞙茧鞃� 鞝曧檿霃勱岸� 84.4%鞐愳劀 86.6%搿� 頄レ儊霅橃棃鞀惦媹雼�. .
毹胳嫚 霟嫕 鞐瓣惮歆勱臣 鞐旍雼堨柎霌れ澊 鞚措煬顷� 巩胳柎 氇嵏霌れ潣 鞛戩梾 靹彪姤鞚� 歆赌靻嶌爜鞙茧 頄レ儊鞁滍偆瓿犾瀽 顷棎 霐半澕 巩胳柎 氇嵏鞚� 韥赴电� 鞝愳皑 旎れ瓿� 鞛堨姷雼堧嫟. 鞚措煬顷� 順勳儊鞚赌 BERT 氇嵏鞚� 氤赌鞚错槙 雮挫棎靹滊弰, 攴鸽Μ瓿� 1,750巩� 臧涤潣 毵り皽氤赌靾橂ゼ 臧赌歆赌瓿� 鞛埖鐢 GPT-3瓿� 臧欖澏� 雼るジ 斓涤嫚 巩胳柎 氇嵏鞐愳劀霃� 霃欖澕顷橁矊 雮橅俭雮╇媹雼�.
IPU-POD鞐愳劀鞚� BERT 靷爠 顷欖姷
鞚搓矁鞚赌 氇嵏毵侅潣 氚涤爠毵岇溂搿� 臧赌電デ甏歆� 瓴冹澊 鞎勲嫏雼堧嫟. 雿� 雴掛澏� 须湪鞚� 鞝滉车顷� 靾� 鞛埖鐢 靸堧鞖� 鞚戈车 歆赌电� 顷橂摐鞗柎鞕赌 鞁涤姢韰涤潣 臧滊皽霃� 顷╇Μ鞝侅澑 旮瓣皠 鞎堨棎 鞚挫櫠� 臧欖澏� 雽赌攴滊 巩胳柎 氇嵏鞚� 顷欖姷鞚� 臧赌电デ陿瓴� 顷� 欤嫉鐢雿�, 鞛犾灛鞝侅溂搿� 氅办范 順轨澏� 鞁巩� 氇� 鞁滉皠 鞎堨棎 靾橃嫮巩� 臧涤潣 鞓堨牅毳� 靷爠 顷欖姷鞁滍偓 靾橂弰 鞛堨姷雼堧嫟. 攴鸽灅顶勋綌巩挫潣 IPU-POD电� 鞚措煬顷� 靹彪姤 氍胳牅毳� 顷搓舶顷� 欤缄碃 鞐瓣惮歆勱臣 鞐旍雼堨柎霌れ潣 靸濎偘靹膘潉 韥矊 頄レ儊鞁涤紲 欷嶋媹雼�. IPU-POD电� 齑堦碃靹钡绁鞚� 鞚疙攧搿涤劯靹� 氅旊毽ゼ 頇涤毄顷橃棳 雿办澊韯� 鞚措彊鞚� 斓涤唽頇斍戗鞙茧〉訊� 霙办柎雮� 瓿勋偘 靹彪姤瓿� 雿� 雮橃澏� 鞝勲牓 须湪靹膘潉 鞝滉车顷╇媹雼�. 瓿犾啀鞚� 鞀れ紑鞚检晞鞗� 鞚疙劙旎る劌韸� 旮半姤瓿� 歆赌电ロ槙 氅旊毽� 甏�毽� 旮半姤鞚� 韱登甏 鞎犿攲毽紑鞚挫厴鞚� 靾橂氨 臧涤潣 IPU搿� 须湪鞝侅溂搿� 鞀れ紑鞚检晞鞗兦隊 靾� 鞛堨姷雼堧嫟.
IPU-POD鞐愳劀电� BERT毳� 须湪鞝侅溂搿� 鞁ろ枆顷橁赴 鞙勄甏 鞝勋泊 氇嵏鞚� 毵り皽氤赌靾橂ゼ IPU鞐� 搿滊摐顷╇媹雼�. 鞚措ゼ 鞙勄甏 BERT 氇嵏鞚� 4臧涤潣 IPU鞐� 攵勄隊 霕惖鐢 "靸る敥 (shard)"顷橁碃, 顷欖姷 瓿检爼 欷戩棎 氇嵏鞚� 韺岇澊顶勲澕鞚胳溂搿� 鞁ろ枆顷╇媹雼�.
鞎勲灅鞐愳劀 BERT-Large毳� 攵勄隊頃樀鐢 氚╇矔鞚� 鞓堧ゼ 氤� 靾� 鞛堨姷雼堧嫟. IPU 0鞐惖鐢 3臧涤潣 鞚胳綌雿� 霠堨澊巩挫櫠� 顷粯 鞛勲矤霐� 霠堨澊巩�(embedding layer), 顶勲鞝濎厴/靻愳嫟 霠堨澊巩�(projection/loss layers)臧赌 韽暔霅橃柎 鞛堨溂氅� 雮橂ǜ歆赌 21臧� 霠堨澊巩吹鐢 雼るジ 3臧涤潣 IPU鞐� 瓿犽ゴ瓴� 攵勴彫霅橃柎 鞛堨姷雼堧嫟. 鞛勲矤霐� 氚� 顶勲鞝濎厴 霠堨澊巩吹鐢 毵り皽氤赌靾橂ゼ 瓿奠湢顷橁赴 霑岆鞐� 顶勲鞝濎厴, MLM(Masked Language Model) 氚� NSP(Next Sentence Prediction) 霠堨澊巩措ゼ 雼れ嫓 IPU 0鞐� 氚办箻顷� 靾� 鞛堨姷雼堧嫟.
攴鸽灅顶勋綌巩吹鐢 旃╈潣 氅旊毽� 靷毄霟夓潉 欷勋澊旮� 鞙勄甏 旮半姤鞚� 靷毄顷╇媹雼�. 霐半澕靹� 鞐癌頄� 韺姢毳� 瓿勋偘顷� 霑� 靷毄顷橁赴 鞙勄甏 欷戧皠 霠堨澊巩� 頇涤劚頇旊ゼ 鞝赌鞛デ隊 頃勳殧臧赌 鞐喌鐢雿办殧. 鞛逞珎办澏� 氇嵏鞚� 顷欖姷鞁滍偓 霑� 靷毄顷� 靾� 鞛埖鐢 鞝勲灥鞙茧, 韺岇澊顶勲澕鞚� 鞝勲灥鞚� 甑槃顷� 霑� 韸鬼瀳 鞙犾毄顷╇媹雼�. 韺岇澊顶勲澕鞚胳潉 韱登甏 顷儊 氤奠垬鞚� 氚办箻臧赌 鈥�鞁ろ枆 欷�鈥�鞚措瘈搿� 鞛逞珎� 旮半姤鞚� 靷毄顷橃 鞎婌溂氅� 鞝赌鞛ル悳 頇涤劚頇旍潣 韥赴臧赌 毵れ毎 旎れ 靾� 鞛堨姷雼堧嫟.
靷爠 顷欖姷 鞁涤姢韰涤潣 鞓淀嫲毵堨澊鞝赌(Optimizer) 靸來儨电� 鞀ろ姼毽皪 氅旊毽�(Streaming Memory)鞐� 鞝赌鞛ル悩瓿� 鞓淀嫲毵堨澊鞝赌 雼硠 欷� 鞖旍箔鞐� 霐半澕 搿滊摐霅╇媹雼�.
鞀ろ姼毽皪 氅旊毽鐢 攴鸽灅顶勋綌巩挫棎靹� 鞓ろ攧旃� 氅旊毽�(off chip memory)毳� 鞚检猾电� 鞖╈柎鞛呺媹雼�. IPU-POD64 雮挫潣 臧� 顷奠嫭 鈥�牍岆敥 敫旊鈥�鞚� 鞐惖鐢 4臧涤潣 IPU 顶勲靹胳劀搿� 鞁ろ枆霅樀鐢 斓滊寑 450GB鞚� 氅旊毽岸� 韮戩灛霅橃柎 鞛堨姷雼堧嫟. 鞚吹鐢 氇摖 IPU 旃╈棎 韽暔霅� 900MB鞚� 鞚疙攧搿涤劯靹� 氅旊毽櫠� IPU雼� 斓滊寑 112GB鞚� 鞓ろ攧旃� 鞀ろ姼毽皪 氅旊毽 雮橂墿雼堧嫟. 鞚� 鞀ろ姼毽皪 氅旊毽鐢 IPU 毹胳嫚鞚� DDR4 DIMM鞐愳劀 歆赌鞗愲惄雼堧嫟.
攴鸽灅顶勋綌巩挫潣 TensorFlow, PyTorch, PopART鞐愳劀鞚� BERT 甑槃
GitHub鞚� 鞐愳劀 TensorFlow, PyTorch, PopART鞐愳劀鞚� IPU搿� 甑槃顷� BERT毳� 頇曥澑顷橃嫟 靾� 鞛堨姷雼堧嫟.
鞚赌 攴鸽灅顶勋綌巩挫潣 TensorFlow 韺岇澊顶勲澕鞚� API毳� 頇涤毄顷橁赴 鞙勄隃 靷毄鞛� 鞝曥潣 氚� 頇曥灔鞚� 韱登甏 瓿� 氇嵏 旖旊摐毳� 瓿奠湢顷╇媹雼�.
PyTorch 甑槃鞚赌 Hugging Face 韸鸽灉鞀ろ彫毹� 霛检澊敫岆煬毽潣 氇嵏 靹る獏瓿� 鞙犿嫺毽嫲毳� 旮半皹鞙茧 顷╇媹雼�. 韺岇澊顶勲澕鞚� 鞁ろ枆(pipeline execution), 鞛逞珎�(recomputation), 雼れ 氤奠牅/雿办澊韯� 氤戨牞 觳橂Μ(multiple-replica/data parallelism) 霌膘潉 韽暔顷�, IPU毳� 鞙勄甏 攴鸽灅顶勋綌巩� PopTorch 霛检澊敫岆煬毽ゼ 靷毄顷╇媹雼�.
Poplar Advanced Runtime(PopART)鞚� 鞚挫毄顷� BERT 甑槃霃� Github鞐愳劀 頇曥澑顷橃嫟 靾� 鞛埖鐢雿办殧. PopART毳� 靷毄顷橂┐ 顷欖姷 氚� 於旊鞚� 鞙勄甏 ONNX 氇嵏 靹る獏鞐愳劀 氇嵏鞚� 臧赌鞝胳槫瓯半倶 靸濎劚顷� 靾� 鞛堨溂氅�, C++ API鞕赌 韺岇澊鞃� API臧赌 氇憪 韽暔霅╇媹雼�. 鞐� 靹る獏霅� 雽赌搿� PopART电� 鞓淀嫲毵堨澊鞝赌, 旮办毟旮�(gradient), 毵り皽氤赌靾�(parameter) 攵勄隊鞚� 歆�鞗愴晿电旊嵃鞖�, 鞚措ゼ 韽磩顷橃棳 韰愳劀 靸る敥(RTS: Revided Tensor Sharding)鞚措澕瓿� 攵赌毽呺媹雼�. PopART鞚� 氤奠牅霅� 韺岇澊顶勲澕鞚� 氇嵏-氤戨牞 BERT 鞁涤姢韰涤潣 瓴届毎, 鞓淀嫲毵堨澊鞝赌 氚� 攴鸽灅霐旍柛韸� 韺岉嫲靺旊嫕(gradient partitioning)鞚� 靷毄顷橁碃 鞛堨姷雼堧嫟.
IPU-POD搿� BERT 鞀れ紑鞚茧
鞛愳湪 靷爠 顷欖姷鞚� 瓴届毎, BERT电� 鞙勴偆顶茧敂鞎�, BookCorpus 氚� 旮绊俭 於涤矘鞚� 靾橃嫮巩� 臧赌歆� 顷欖姷 鞓堨牅毳� 頇涤毄顷� 靾� 鞛堨姷雼堧嫟. IPU-POD4毳� 靷毄頃樀鐢 瓴届毎鞐愲弰 鞚措煬顷� 雽赌鞖╇焿 雿办澊韯� 靹疙姼毳� 雼れ垬 韱店臣顷橂牑氅� 靸侂嫻顷� 鞁滉皠鞚� 瓯鸽雼堧嫟. 顷欖姷 鞁滉皠鞚� 雼稌顷橁赴 鞙勄甏 鞖半Μ电� 雿办澊韯� 氤戨牞 氇嵏 顷欖姷鞚� 靷毄顷橃棳 靷爠 顷欖姷 顶勲靹胳姢毳� IPU-POD16, IPU-POD64 鞚挫儊鞙茧 鞀れ紑鞚记隊 靾� 鞛堨姷雼堧嫟.
雿办澊韯� 氤戨牞 顷欖姷鞚赌 顷欖姷 雿办澊韯� 靹疙姼毳� 鞐煬 攵赌攵勋溂搿� 雮橂垊氅�, 臧� 攵�攵勳澏� 氇嵏 氤奠牅鞐� 鞚樓甏 靷毄霅╇媹雼�. 臧� 斓涤爜頇� 雼硠鞐愳劀 攴鸽灅霐旍柛韸傅鐢 氇摖 氤奠牅鞐愳劀 韽夑窢鞝侅溂搿� 臧愳唽顷橂瘈搿� 氇摖 氤奠牅鞐愳劀 臧赌欷戩箻 鞐呺嵃鞚错姼 氚� 氇嵏 靸來儨臧赌 霃欖澕顷╇媹雼�.
歆戫暕鞝侅澑 攴鸽灅霐旍柛韸� 臧愳唽
攴鸽灅霐旍柛韸� 臧愳唽 旮半姤鞚赌 瓿犾劚电� 鞀れ紑鞚� 鞎勋泝鞚� 歆赌鞗愴晿霃勲 靹り硠霅� 韽攲霛�(Poplar) 靻岉攧韸胳洦巩� 鞀ろ儩鞚� 頇曥灔鞚� 乐播传媒视频 旎る雼堨紑鞚挫厴 霛检澊敫岆煬毽�(GCL: 乐播传媒视频 Communication Library)毳� 靷毄顷╇媹雼�. GCL鞐惖鐢 毵れ毎 须湪鞝侅澑 毵� 旮半皹 All-Reduce鞕赌 雼るジ 毵庫澏� 鞚茧皹鞝侅澑 旎る雼堨紑鞚挫厴 顶勲Μ氙疙嫲敫�(primitives)臧赌 韽暔霅橃柎 鞛堨姷雼堧嫟. 鞎勲灅 鞚措歆赌鞐� 雮橃櫠� 鞛埖鐢 瓴冹矘霟� 氇摖 氤奠牅鞐� 雽赌顷挫劀 攴鸽灅霐旍柛韸胳潣 韽夑窢臧掛澊 靷办稖霅橂┌, 氇摖 氅赌韹� 氤奠牅 攴鸽爤鞚措敂鞏疙姼臧赌 鞕勋爠颂� 臧愳唽顷橂┐ 臧赌欷戩箻 鞐呺嵃鞚错姼臧赌 顷� 氩堨棎 鞝侅毄霅╇媹雼�.
雿办澊韯� 氤戨牞 攴鸽灅霐旍柛韸� 臧愳唽电� 鞓淀嫲毵堨澊鞝赌 霝橅嵓 旮半姤(optimizer wrapper functions)鞚� 靷毄顷橁卑雮� 靷毄鞛� 鞝曥潣 鞓淀嫲毵堨澊鞝赌 鞝曥潣(custom optimizer definitions)毳� 氤措嫟 靹鸽皜顷橁矊 鞝涤柎顷橃棳 鞛愲彊鞙茧 於旉岸那隊 靾� 鞛堨姷雼堧嫟. 鞚措煬顷� 臧愳唽电� IPU-POD 雮挫棎靹� 霕惖鐢 靹滊 雼るジ IPU-POD 臧勋棎 攴犾澕顷橁矊 氚涤儩顷� 靾� 鞛堨姷雼堧嫟.
TensorFlow鞐愳劀 API臧赌 鞛戨彊頃樀鐢 氚╈嫕鞐� 雽�顷� 鞛愳劯顷� 雮挫毄鞚赌 TensorFlow 靷毄 靹る獏靹涤棎靹� 鞐� 雽�顷� 鞝曤炒毳� 彀胳“顷橃嫓氅� 霅╇媹雼�.
PyTorch鞚� 瓴届毎, 鞖半Μ电� PopTorch IPU 氇嵏 鞓奠厴鞐� 氤奠牅 歆赌靾�(replication factor)毳� 靹れ爼顷╇媹雼�.
雽赌攴滊 氚办箻(batches)搿� BERT 顷欖姷
氇嵏鞚� 氤戨牞 鞀れ紑鞚茧 须湪靹膘棎 旮办棳頃樀鐢 欤检殧 鞖旍唽 欷� 頃橂倶电� 攴鸽灅霐旍柛韸戈岸� 巩茧雮� 鞛愳＜ 韱奠嫚顷� 靾� 鞛埖鐢 歆赌鞛呺媹雼�. 鞖半Μ电� 攴鸽灅霐旍柛韸� 於曥爜(GA: Gradient Accumulation)鞚� 靷毄顷橃棳 旎错摠韺� 霕惖鐢 毵堨澊韥 氚办箻 韥赴毳� 旮赌搿滊矊 氚办箻 韥赴鞕赌 攵勲Μ顷╇媹雼�. 霕惽隃 攴鸽灅霐旍柛韸� 於曥爜 歆赌靾橂ゼ 臁办爼顷橃棳 IPU 氚�/霕惖鐢 氤奠牅 靾橂ゼ 氤赌瓴角隊 霑� 鞚缄磤霅� 旮赌搿滊矊 氚办箻 韥赴毳� 鞙犾顷� 靾� 鞛堨姷雼堧嫟. 旮赌搿滊矊 氚办箻 韥赴 = 氤奠牅 氚办箻 韥赴 x 氤奠牅 靾�(global batch size = replica batch size 脳 number of replicas), 鞐赴靹� 氤奠牅 氚办箻 韥赴 = 旎错摠韺� 氚办箻 韥赴 x 攴鸽灅霐旍柛韸� 於曥爜 歆赌靾�(replica batch size = compute batch size 脳 gradient accumulation factor).
氤奠牅 靾橁岸� 瓿犾爼霅� 瓴届毎, 旮赌搿滊矊 氚办箻 韥赴臧赌 韥挫垬搿� GA 歆赌靾橁岸� 雴挣碃 鞓淀嫲毵堨澊鞝赌鞕赌 韱奠嫚 雼硠臧赌 欷勋柎霌媹雼�.
顷橃毵� GA臧赌 靾橃矞 臧滊 雱堧 韥┐ FP16鞚� 巩鸽崝顶岆(underflow) 氍胳牅臧赌 氚涤儩顷� 靾� 鞛堨姷雼堧嫟. 鞛戩澏� GA鞚� 瓴届毎 靹る獏霅� 雽赌搿� 氩勲笖 鞓る矂项る摐(bubble overhead)搿� 鞚盖甏 韺岇澊顶勲澕鞚� 须湪鞚� 雮晞歆� 靾� 鞛堨姷雼堧嫟. 斓涤爜鞚� 臧掛潉 彀娟赴 鞙勄甏靹湹鐢 氇� 臧赌歆� 鞁ろ棙鞚� 顷勋殧顷� 靾� 鞛堨姷雼堧嫟.
雼れ潓 靹轨厴鞐愳劀电� BERT毳� 顷欖姷顷� 霑� 頇滌毄頃樀鐢 顷欖姷 靻嶋弰, 鞗岆皪鞐� 氚� 斓涤爜頇� 顶勲攴鸽灗鞚� 瓴�韱犌戛雼堧嫟.
靹犿槙 鞀れ紑鞚茧 攴涤箼
鞐愳劀 鞐瓣惮鞗愲摛鞚赌 ResNet-50 氚� Mask R-CNN鞚� 鞙勄隃 雽赌攴滊 旮赌搿滊矊 氚办箻毳� 顷欖姷鞁滍偓 霑� 靹犿槙 鞀れ紑鞚茧 攴涤箼鞚� 靷毄頄堨姷雼堧嫟. 鞚� 攴涤箼鞚赌 氤奠牅 氚办箻 韥赴 茷鞐� k毳� 瓿鼻隊 霑�(鞐赴靹� k电� 鞚茧皹鞝侅溂搿� 氇嵏 氤奠牅 靾橃瀯), 旮半掣 顷欖姷 靻嶋弰 茷鞐� k毳� 瓿鼻陿瓿� 鞚� 臧掛潉 k茷搿� 靹れ爼顷╇媹雼�.
旮赌搿滊矊 氚办箻 韥赴毳� n鞐愳劀 nk搿� 电橂Μ氅� 霃欖澕顷� 靾橃潣 顷欖姷 须熿垬(training epoch)毳� 靷毄顷橁碃 韰岇姢韸� 鞝曧檿靹膘潉 鞙犾頃樀鐢 霃欖嫓鞐� 齑� 甑愳湣 鞁滉皠鞚� k鞚� 歆赌靾橂韥� 欷勋澊瓿� 氇嵏-靸濎偘 鞁滉皠鞚� 韥矊 雼稌顷� 靾� 鞛堨姷雼堧嫟. 攴鸽煬雮� 鞛戩梾 靹彪姤鞚赌 雽赌攴滊 旮赌搿滊矊 氚办箻鞚� 瓴届毎 霒柎歆�电� 瓴冹潉 鞛呾霅橃棃鞀惦媹雼�.
鞝愳鞝� 鞗岆皪鞐� 鞝勲灥
鞚措ゼ 鞕勴檾頃樀鐢 顷� 臧赌歆� 氚╇矔鞚赌 氚旊 鞗岆皪鞐呾瀰雼堧嫟. 鞗岆皪鞐� 鞁� 顷欖姷 靻嶋弰电�k茷搿� 歃夓嫓 齑堦赴頇旊悩歆赌 鞎姷鐢雿办殧. 雽赌鞁�, 顷欖姷 瓿检爼鞚赌 0鞙茧 鞁涤澜顷橁卑雮� 鞛勋潣鞚� 鞛戩澏� 顷欖姷 靻嶋弰搿� 鞁涤澜顷橃棳 靷爠 鞝曥潣霅� 鞗岆皪鞐� 雼硠 靾橃棎 霐半澕 鞐办啀鞝侅溂搿� 歃濌岸捻晿鞐� k茷鞐� 霃勲嫭顷╇媹雼�. 鞚措煬顷� 鞝愳鞝� 鞗岆皪鞐呾潉 韱登甏 雿� 鞝侅澏� 靾橃潣 雼硠搿� 雽赌攴滊 旮赌搿滊矊 氚办箻 顷欖姷鞚� 靾橅枆顷橃棳 雿� 鞛戩澏� 氚办箻 韥赴鞕赌 鞙犾偓顷� 甑愳湣 鞝曧檿霃勲ゼ 巩混潉 靾� 鞛堨姷雼堧嫟. 鞐愳劀 鞚� 顷欖姷鞚� 韱登甏 旮赌搿滊矊 氚办箻 韥赴臧赌 鞎� 8,000鞚� 顷欖姷鞚� 臧赌電デ隃 瓴冹潉 頇曥澑頄堨姷雼堧嫟.
靷爠 鞝曥潣霅� 鞗岆皪鞐� 雼硠电� BERT-LARGE 靷爠 顷欖姷 靷鞚� 1雼硠鞕赌 2雼硠鞐愳劀 雼るゴ瓴� 雮橅俭雮╇媹雼�. 鞐愳劀鞕赌 臧欖澊, 1雼硠鞐愳劀电� 斓滊寑 鞁滍赌赌鞀� 旮胳澊臧赌 128鞚搓碃 2雼硠鞐愳劀电� 斓滊寑 鞁滍赌赌鞀� 旮胳澊臧赌 384鞚� 顷欖姷 雿办澊韯半ゼ 靷毄顷╇媹雼�. 1雼硠 鞗岆皪鞐呾潣 靾樀鐢 2,000臧滊 1雼硠鞐愳劀 鞝勋泊 甑愳湣 雼硠鞚� 鞎� 30%毳� 彀頃樀鐢雿�, 鞚挫棎 牍勄甏 2雼硠鞚� 鞗岆皪鞐� 靾樀鐢 齑� 2,100臧涤澊氅�, 鞚� 雼硠鞚� 鞎� 13%臧赌 鞗岆皪鞐� 雼硠鞛呺媹雼�.
鞗岆皪鞐� 雼硠电� 靹滊 雼るジ 靷爠 顷欖姷 雿办澊韯� 靹疙姼鞐� 毵炾矊 臁办爼顷挫暭 顷� 靾橂弰 鞛堨姷雼堧嫟.
AdamW 鞓淀嫲毵堨澊鞝赌
响涤 頇曤鞝� 攴鸽灅霐旍柛韸� 顷橁皶 鞎岅碃毽(standard stochastic gradient descent algorithm)鞚赌 氇摖 臧赌欷戩箻 鞐呺嵃鞚错姼鞐� 雽赌顷� 雼澕 顷欖姷 靻嶋弰毳� 靷毄顷橂┌ 顷欖姷 欷戩棎 鞚检爼顷� 靻嶋弰毳� 鞙犾顷╇媹雼�. 氚橂尪碾 鞚赌 攴鸽灅霐旍柛韸胳潣 觳� 氩堨Ц 氚� 霊� 氩堨Ц 氇韸胳潣 鞚措彊 歆赌靾� 韽夑窢鞚� 靷毄顷橁碃 鞚措煬顷� 氇韸鸽ゼ 旮办鞙茧 顷欖姷 靻嶋弰 毵り皽氤赌靾橂ゼ 臁办爼顷╇媹雼�.
鞐愳劀 Loshchilov鞕赌 Hutter 鞐瓣惮鞗愳澏� L2 鞝曣窚頇旉岸� Adam鞐愳劀 須臣臧赌 鞐嗠嫟电� 靷嫟鞚� 鞎岇晞雰堨姷雼堧嫟. 雽赌鞁�, 攴鸽摛鞚赌 靻愳嫟 旮半姤鞐愳劀 L2 鞝曣窚頇� 雽赌鞁� 臧赌欷戩箻 臧愳唽 鞝曣窚頇旊ゼ 鞝侅毄頃樀鐢 AdamW毳� 鞝涤晥頄堨姷雼堧嫟. 鞚措ゼ 韱登甏 臧赌欷戩箻电� 於旉岸� 靸侅垬 歆赌靾橁岸� 鞎勲媹霛� 氚办垬鞐� 鞚樓甏 臧愳唽霅╇媹雼�. 霕惽隃 鞗� 鞛嫓鞛戩澊 臧赌電デ隃 AdamW臧赌 CIFAR-10 氚� ResNet32x32 氇憪鞐愳劀 顷欖姷 靻愳嫟瓿� 鞚茧皹頇旍潣 鞓る 旄‰┐鞐愳劀 雿� 鞖办垬頃橂嫟电� 瓴冹潉 鞛呾顷橃榾鞀惦媹雼�.
IPU-POD16 BERT 靷爠 顷欖姷鞐愳劀 鞖半Μ电� AdamW搿� 512~2560鞚� 靷爠 顷欖姷 旮赌搿滊矊 氚办箻 韥赴毳� 靷毄顷橃棳 鞁ろ棙鞚� 歆勴枆顷橃榾鞙茧┌, 氇摖 靷臧赌 SQuAD 雼れ毚鞀ろ姼毽� 鞛戩梾鞐� 毵烄栋 氙胳劯 臁办爼霅橂┐靹� 旮办 鞝曧檿霃勋棎 靾橂牬頃樀鐢 靹标臣毳� 氤挫榾鞀惦媹雼�.
LAMB 鞓淀嫲毵堨澊鞝赌
LAMB 鞓淀嫲毵堨澊鞝赌(鞛愳劯顷� 雮挫毄鞚赌 鞐� 靹る獏霅橃柎 鞛堨潓)电� 氚办箻 韥赴臧赌 歃濌岸那隊 霑� 氚滌儩頃樀鐢 攴鸽灅霐旍柛韸� 攵堨晥鞝� 氚� 靻愳嫟 鞚错儓鞚� 攴闺车顷橃棳 氚办箻 韥赴毳� 雿� 韥矊 鞙犾顷� 靾� 鞛堧弰搿� 靹り硠霅橃棃鞀惦媹雼�. LAMB鞚赌 霠堨澊巩� 鞕赌鞚挫 鞝侅潙 靻嶋弰 鞀れ紑鞚茧(LARS: layer-wise adaptive rate scaling)瓿� 霃欖澕顷� 霠堨澊巩� 鞕赌鞚挫 鞝曣窚頇� 臧滊厫鞚� 靷毄顷橂瘈搿� 顷欖姷 靻嶋弰臧赌 霠堨澊巩挫棎 氙缄皭顷╇媹雼�. 攴鸽煬雮� 毵り皽氤赌靾� 鞐呺嵃鞚错姼鞚� 瓴届毎 AdamW鞚� 氇韰赌 氚� 攵勋偘 臧滊厫鞚� 靷毄顷╇媹雼�.
臧� 霠堨澊巩挫潣 顷欖姷 靻嶋弰电� 雼れ潓瓿� 臧欖澊 瓿勋偘霅╇媹雼�.
\[\eta \frac{ \| x \| } { \| g \| }\]
鞐赴靹� \(\eta\) 电� 鞝勋泊 顷欖姷鞙澊瓿�, \(\| x \|\) 电� 鞚� 瓿勋傅鞚� 毵り皽氤赌靾橃潣 旮办鞚措┌ \(\| g \|\)电� 霃欖澕顷� AdamW 鞓淀嫲毵堨澊鞝赌鞐� 雽�顷� 鞐呺嵃鞚错姼鞚� 旮办鞛呺媹雼�.
歃�, LAMB鞚� 鞐呺嵃鞚错姼毳� 鞝曣窚頇斍陿瓿� \(\|x\|\)毳� 瓿表晿鞐� 臧� 霠堨澊巩挫潣 毵り皽氤赌靾橃櫠� 霃欖澕顷� 韥赴 鞝曤弰毳� 臧栮矊 顷溂搿涤崹 鞐呺嵃鞚错姼臧赌 臧� 霠堨澊巩挫潣 鞁れ鞝侅澑 氤赌頇旊ゼ 臧赌电デ陿瓴� 顷╇媹雼�. 攴� 雼れ潓 瓴瓣臣臧掛棎 鞝勋泊 顷欖姷 靻嶋弰 \(\eta\) 毳� 瓿鼻戛雼堧嫟.
LAMB鞚� 瓴届毎, 臧赌欷戩箻鞕赌 韼疙枼鞚赌 靹滊 雼るジ 鞁犽臧掛潉 臧赌歆赌瓿� 鞛堨柎 靹滊 雼るジ 顷欖姷 鞖旍湪搿� 旆笁顷挫暭 顷橂瘈搿� 霊� 臧涤潣 攵勲Μ霅� 霠堨澊巩措 臧勋＜霅╇媹雼�. 韼疙枼 氚� 臧愲, 氚办箻-响涤 (batch-norm)霕惖鐢 攴鸽９-响涤(group-norm) 氩犿俭 氩勳爠鞚赌 霠堨澊巩� 臧涤“鞐愳劀 鞝滌櫢霅樀鐢 瓴届毎臧赌 毵库姷雼堧嫟.
BERT鞚� 瓴届毎, LAMB鞚赌 雼硠 1鞐愳劀 斓滊寑 65,536鞚� 旮赌搿滊矊 氚办箻 韥赴鞕赌 雼硠 2鞐愳劀 斓滊寑 32,768鞚� 旮赌搿滊矊 氚办箻 韥赴毳� 靷毄顷� 靾� 鞛堨姷雼堧嫟.
雮澏� 鞝曤皜霃� 顷欖姷(Low-precision training)
霐� 霟嫕鞚� 齑堦赴鞐惖鐢 毵庫澏� 氇嵏霌れ澊 32牍勴姼 鞝曤皜 攵赌霃欖唽靾橃爯 鞐办偘 氚╈嫕(32-bit precision floating-point arithmetic)鞙茧 顷欖姷 氚涭晿鞀惦媹雼�. 雮澏� 鞝曤皜霃勲ゼ 靷毄頃樀鐢 瓴冹澏� 旎错摠韺� 觳橂Μ霟夓澊 霊� 雼� 頄レ儊霅橁赴 霑岆鞐� 毵る牓鞝侅瀰雼堧嫟. IPU鞚� 瓴届毎 FP16 顶柬伂 靹彪姤鞚� FP32氤措嫟 4氚� 雿� 雴掛姷雼堧嫟. 鞝曤皜霃勱岸� 雮潉 霑� 韰愳劀 韥赴臧赌 2氚� 臧愳唽顷橃棳 氅旊毽� 鞎曤牓 氚� 韱奠嫚 牍勋毄鞚� 鞝堦皭霅╇媹雼�.
攴鸽煬雮� FP16 鞝曤皜 攵赌霃欖唽靾橃爯鞚赌 FP32氤措嫟 鞝曤皜霃勱岸� 雮碃 霃欖爜 氩旍渼臧赌 雮姷雼堧嫟. 顷橃毵� FP16鞚� 靷毄顷橃棳 FP32 臧赌欷戩箻 攴鸽灅霐旍柛韸� 雸勋爜鞚� 韱登甏 毵堨姢韯� 臧赌欷戩箻鞚� FP32 氤奠偓氤胳潉 鞙犾顷橂┐靹� 靻愳嫟 鞀れ簮鞚茧鞚� 靷毄顷橃棳 頇涤劚 氚� 攴鸽灅霐旍柛韸胳潣 霃欖爜 氩旍渼毳� 甏�毽晿电� 靾滊癌頄� 韺姢(頇涤劚頇�,activations) 氚� 鞐癌頄� 韺姢(攴鸽灅霐旍柛韸�) 瓿勋偘鞚� 鞝滌晥頃樀鐢 靷瀸霌る弰 鞛堨姷雼堧嫟.
鞖半Μ电� FP16鞐愳劀 頇涤劚頇� 氚� 攴鸽灅霐旍柛韸鸽ゼ 顷欖姷顷� 霑� 靻愳嫟 鞀れ紑鞚茧鞚� 鞙犾偓顷� 氚╈嫕鞙茧 甑槃顷╇媹雼�.
乐播传媒视频 IPU电� 瓿� 旮办〈 IEEE 氚橃槵毽� 氇摐毳� 靷毄顷� 靾� 鞛堨姷雼堧嫟. 頇曤鞝� 氚橃槵毽检澏� 顷措嫻 臧掛澊 鞙勋 氚� 鞎勲灅飒� 氚橃槵毽� 瓴疥硠鞐� 攴检爲顷� 鞝曤弰鞐� 牍勲顷橃棳 雼澕歆戨媹雼�. 毵庫澏� 靾橃潣 顷欖姷 鞓堨牅毳� 鞚挫毄顷� 韼疙枼霅橃 鞎婌澏� 氚橃槵毽� 瓴瓣臣毳� 巩混潉 靾� 鞛堨姷雼堧嫟.
頇曤鞝� 氚橃槵毽� 氚╈嫕鞚� 靷毄顷橂┐ 鞝勋泊 顷欖姷 瓿检爼 霃欖晥 鞐旊摐 韴� 鞐旊摐 顷欖姷 霕惖鐢 雼れ毚鞀ろ姼毽� 鞛戩梾 靹标臣鞚� 鞝曧檿靹膘澊 雸堨棎 霛勱矊 鞝赌顷橂悩歆赌 鞎婌溂氅挫劀 FP16鞐� 臧赌欷戩箻毳� 鞙犾顷� 靾� 鞛堨姷雼堧嫟.
鞓淀嫲毵堨澊鞝赌鞚� 觳� 氩堨Ц鞕赌 霊� 氩堨Ц 氇韸戈岸� 瓿勋偘霅橃柎 FP32鞐� 鞝赌鞛ル悩瓿�, 鞝曣窚頇旊弰 FP32鞐愳劀 靾橅枆霅╇媹雼�. 顷欖姷 顶勲靹胳姢鞚� 雮橂ǜ歆赌 鞛戩梾鞚赌 FP16鞐愳劀 瓿勋偘霅╇媹雼�.
顷欖姷 瓴瓣臣
攴鸽灅顶勋綌巩挫潣 斓涤嫚 鞀れ紑鞚检晞鞗� 鞁滌姢韰滌澏� 霃欔笁 DGX A100 旮半皹 鞁涤姢韰涤棎 牍勄甏 顷欖姷 鞁滉皠鞚� 斓滊寑 2.6氚� 雿� 雼稌顷橃棳 BERT-Large 顷欖姷鞐愳劀 鞝勲 鞐喌鐢 须湪靹膘潉 氤挫棳欷嶋媹雼�.
斓涤嫚 IPU-M2000 臧赌靻嶊赴 16臧滉岸� 韽暔霅� IPU-POD64电� 旎错摠韺�, 韱奠嫚 氚� 氅旊毽� 旮办垹鞚� 顺侅嫚鞚� 韱登甏 BERT-Large鞐愳劀 靹犽弰鞝侅澑 AI 顶岆灚韽缄臣 霃欖澕顷� 鞝曧檿霃勲ゼ 雿旍毐 歆ъ澏� 鞁滉皠 雮挫棎 鞝滉车顷╇媹雼�. 鞎勲灅 霃勏鞙电� TensorFlow 氚� PyTorch鞚� 响涤 靸侅渼 靾橃 顶勲爤鞛勋泴韥ゼ 靷毄顷� 瓴瓣臣鞕赌 攴鸽灅顶勋綌巩挫潣 PopART 旮半皹 甑槃 瓴瓣臣毳� 氤挫棳欷嶋媹雼�. 鞚� 瓴瓣臣毳� NVIDIA臧赌 氚溝鞙顷� 臧赌鞛� 鞖办垬顷� PyTorch 瓴瓣臣鞕赌 牍勱祼顷橁碃 鞙犾偓顷� 氚╇矔搿犾潉 靷毄顷橃棳 牍勳姺顷� 顷欖姷 鞁滉皠(Time-To-Train) 瓴瓣臣毳� 霃勋稖頄堨姷雼堧嫟.
靷爠 顷欖姷
鞎勲灅 彀姼电� TensorFlow, PyTorch 氚� PopART 甑槃鞐� 雽�顷� 靷爠 顷欖姷 靻愳嫟 瓿§劆鞚� 雮橅儉雮吹鐢雿�, 霃欕摫顷� 斓涤 顷欖姷 靻愳嫟(final training loss)鞐� 雽�顷� 靾橂牬瓿� 靹滊 毵れ毎 鞙犾偓顷� 顷欖姷 瓿§劆鞚� 氤挫棳欷嶋媹雼�. 霕惽隃 鞚� 彀姼鞐惖鐢 IPU-POD16 氚� IPU-POD64鞐愳劀 靹� 臧赌歆� 氇嵏鞚� 氇憪 甑槃頄堨潉 霑岇潣 顷欖姷 觳橂Μ霟夓澊 响涤嫓霅橃柎 鞛堨姷雼堧嫟.
响�: 靷爠 顷欖姷 觳橂Μ霟�(Pre-training throughput)
韺岇澑韸滊嫕 (Fine-tuning)
靾樄 臧涤潣 顷欖姷 鞓堨牅毳� 顷勋殧搿� 頃樀鐢 BERT毳� 於╇秳颂� 顷欖姷鞁滍偍 頉�, 氙鸽Μ 顷欖姷霅� 鞚� 臧赌欷戩箻毳� 霠堨澊敫旍澊 歆赌鞝曤悳 鞝侅澏� 巩戩潣 雿办澊韯半ゼ 鞚挫毄顷橃棳 韮涤姢韥硠 韺岇澑韸滊嫕 顷欖姷 顶勲靹胳姢鞚� 齑堦赴 臧赌欷戩箻搿� 靷毄顷� 靾� 鞛堨姷雼堧嫟.
鞚� 2雼硠 靹れ爼 氚╈嫕鞚赌 雼れ潓瓿� 臧欖澏� 鞚挫爯鞚� 鞝滉车顷橁赴 霑岆鞐� 鞁れ牅搿� 雱愲Μ 靷毄霅╇媹雼�.
- 氇摖 韺岇澑韸滊嫕 鞛戩梾鞚� 鞙勄甏 BERT 氇嵏鞚� 顷� 氩堧 靷爠 顷欖姷鞁滍偆氅� 霅╇媹雼�.
- 靷爠 顷欖姷霅� 鞚� 氇嵏鞚� 韺岇澑韸滊嫕顷� 瓴届毎 霠堨澊敫旍澊 歆赌鞝曤悩歆赌 鞎婌澏� 雿办澊韯半 鞖办垬顷� 鞛戩梾 靹彪姤鞚� 甑槃顷� 靾� 鞛堨溂氙赌搿� 鞛戩梾 氤� 雿办澊韯办棎 霠堨澊敫旍潉 歆�鞝曧晿电� 雿� 毵庫澏� 云鸽牓鞚� 旮办毟鞚� 頃勳殧臧赌 鞐嗢姷雼堧嫟.
韺岇澑韸滊嫕鞚赌 顷欖姷 雿办澊韯� 靹疙姼鞚� 韥赴鞐� 霐半澕 IPU POD4 霕惖鐢 IPU POD16鞐愳劀 氇� 攵� 雮挫 氇� 鞁滉皠 雮挫棎 鞕勲顷� 靾� 鞛堨姷雼堧嫟. 毵庫澏� 韺岇澑韸滊嫕 顷欖姷鞚� 靻岆焿鞚� 顷欖姷 靹疙姼 韺姢鞓る矂臧赌 觳橂Μ霅� 鞚错泟鞐� 欷戩霅� 靾� 鞛堨姷雼堧嫟.
SQuAD v1.1
鞀ろ儬韽摐 歆堨潣 鞚戨嫷 雿办澊韯� 靹疙姼(: Stanford Question Answering Dataset) v1.1鞚赌 500臧� 鞚挫儊鞚� 云茧鞐� 鞛埖鐢 100,000臧� 鞚挫儊鞚� 歆堨潣 鞚戨嫷 鞂嶌澊 韽暔霅� 雽赌攴滊 霃吳甏霠� 雿办澊韯� 靹疙姼鞛呺媹雼�.
鞎勲灅 响湹鐢 靷爠 顷欖姷霅� 彀胳“ 氚� IPU 靷爠 顷欖姷 臧赌欷戩箻毳� 靷毄顷橃棳 IPU鞐愳劀 SQuAD v1.1 鞛戩梾鞚� 韱登甏 BERT-Large毳� 韺岇澑韸滊嫕顷� 霑岇潣 鞝曧檿霃勲ゼ 氤挫棳欷嶋媹雼�. 鞁涤棸霅� 氚旍櫠� 臧欖澊, IPU电� 鞚� 鞛戩梾鞐愳劀 雴掛澏� 彀胳“ 鞝曧檿霃勲ゼ 氤挫棳欤缄碃 鞛堨姷雼堧嫟.
响�: SQuAD 1.1 鞛戩梾鞚� IPU鞐愳劀鞚� 鞝曧檿霃�
CLUE
雼れ潓鞐� 靷错幋氤� 雿办澊韯� 靹疙姼电� 靷爠 顷欖姷霅� Google 臧赌欷戩箻毳� 靷毄顷橃棳 IPU鞐愳劀 欷戧淡巩� 鞚辞甏霃� 韽夑岸�(CLUE: Chinese Language Understanding Evaluation) 鞛戩梾瓿� 顷粯 BERT-Base毳� 韺岇澑韸滊嫕顷� 霑岇潣 鞝曧檿霃勲ゼ 氤挫棳欷嶋媹雼�.
CLUE 鞝愳垬电� 氇摖 CLUE 鞛戩梾鞐� 雽�顷� 韰岇姢韸� 鞝曧檿霃勋潣 韽夑窢鞛呺媹雼�. 臧� 鞛戩梾鞚� 韰岇姢韸� 鞝曧檿霃劦鐢 5臧赌歆� 鞁ろ棙 瓴瓣臣鞚� 韽夑窢鞛呺媹雼�. 鞎勲灅 响滌棎 雮橃櫠� 鞛埖鐢 瓴冹矘霟� IPU电� DGX-1 V100瓿� 臧欖澏� AI 顶岆灚韽缄臣 霃欖澕顷� 鞝曧檿霃勲ゼ 雼劚顷� 靾� 鞛堨姷雼堧嫟.
响�: GPU 氚� IPU鞚� CLUE 鞛戩梾 鞝曧檿霃�
AI 鞝勲臧赌电� 氇� 臧赌歆� 旮半姤 斓涤爜頇旊ゼ 韱登甏 IPU-POD 鞁涤姢韰涤潉 靷毄顷橃棳 鞝曧檿霃勲ゼ 鞙犾顷橂┐靹� BERT-Large鞚� 顷欖姷 鞁滉皠鞚� 韥矊 雼稌顷� 靾� 鞛堨姷雼堧嫟. 歆赌旮堦箤歆赌 靷错幋氤� 氚旍櫠� 臧欖澊 靸堧鞖� 顷橂摐鞗柎 鞎勴偆韰嶌矘鞐愳劀 BERT鞕赌 臧欖澏� 雽赌顺� 氇嵏鞚� 鞁ろ枆頃樀鐢 瓴冹澊 靸濌皝氤措嫟 毵れ毎 臧勲嫧顷╇媹雼�. 毵庫澏� 臧滊皽鞛愲秳霌れ澏� 鞐赴鞐� 靹る獏霅� 斓涤爜頇� 旮办垹鞐� 鞚措 鞚奠垯顷橃嫟 靾橂弰 鞛堨姷雼堧嫟. 霕惽隃 鞖半Μ电� 氚� 甑槃 鞛愲毳� 瓿店皽顷橃棳 氇摖 毹胳嫚霟嫕 鞝勲臧赌臧赌 鞚措煬顷� 顶勲爤鞛勋泴韥ゼ 靷毄顷橃棳 IPU鞐愳劀 BERT毳� 鞁ろ枆頃樀鐢 氚╇矔鞚� 靷错幋氤� 靾� 鞛堧弰搿� 頄堨姷雼堧嫟.
BERT鞕赌 臧欖澏� 雽赌顺� 氇嵏鞚� 鞚挫毄顷� 斓涤儊鞚� 鞛戩梾 靹彪姤鞚� 雼劚顷橁赴 鞙勄隃 鞐瓣惮鞛愳櫠� 鞐旍雼堨柎霌れ潣 旮半寑臧赌 鞝愳爯 雿� 歃濌岸捻晿氅挫劀 顷橂摐鞗柎 须湪靹膘澊 欷戩殧顷� 鞐隊鞚� 顷橁矊 霅� 瓴冹瀰雼堧嫟. 歆赌旮堦箤歆赌 鞖半Μ电� IPU-POD 鞁涤姢韰涤潉 靷毄頄堨潉 霑� BERT-Large 氚� 鞙犾偓 氇嵏鞚� 鞝曧檿霃勲繍毵� 鞎勲媹霛� 靸侂嫻顷� 靹彪姤 頄レ儊鞚� 鞚挫爯霃� 巩混潉 靾� 鞛堧嫟电� 瓴冹潉 鞎� 靾� 鞛堨棃鞀惦媹雼�. IPU 顶勲靹胳劀 鞎勴偆韰嶌矘鞚� 瓿犾湢顷� 韸轨劚鞚赌 鞓る姌雮� 臧赌鞛� 靹犾 氇嵏鞚� 靹彪姤鞚� 臧赌靻嶍檾頃樀鐢 霃欖嫓鞐� 鞖半Μ鞐愱矊 鞚奠垯顷� 靻岉攧韸胳洦巩� 頇橁步鞐� 鞏挫毟毽鐢 氙鸽灅鞚� 斓涤波雼� 氇嵏鞚� 臧滊皽頃樀鐢 雿办棎霃� 鞎勋澊霐旍柎毳� 鞝滉车顷╇媹雼�.
鞚� 鞛戩梾鞚� 鞚措亴巩� 欤检嫚 氇摖 鞐旍雼堨柎 攵勲摛鞐愱矊 臧愳偓霌滊雼堧嫟.
James Briggs, Oskar Bunyan, Lorenzo Cevolani, Arjun Chandra, Nic Couronneau, Lakshmi Krishnan, Guoying Li, Visu Loganathan, Sam Maddrell-Mander, Zhiwei Tian, Sylvain Viguier, Xian Wang, Xihuai Wen, ChengShun Xia, Han Zhao.
鞙勋潣 鞚措 鞕胳棎霃�, IPU 鞁涤姢韰涤棎靹滌潣 BERT鞚� 甑槃 氚� 斓涤爜頇旊ゼ 歆�鞗惽甏 欤检嫚 攴鸽灅顶勋綌巩挫潣 BERT 巩错攲毽紑鞚挫厴, PopART, TensorFlow, PyTorch, GCL 氚� Poplar 韺赌鞐愲弰 臧愳偓霌滊雼堧嫟.
瓿奠湢:
