Zhuang (za) subword embeddings
| Vocab size | vocab | model | 25 dim | 50 dim | 100 dim | 200 dim | 300 dim |
|---|---|---|---|---|---|---|---|
| 3000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 5000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
Training corpus sample, encoded with different BPE vocabulary sizes
| Vocab size | zawiki sample |
|---|---|
| original | he was particularly happy at würzburg where he enjoyed walks among the hills and he also made frequent visits to the black forest. his administrative sir james w. black united kingdom "for their discoveries of important principles for drug treatment" since all type ia supernovae are believed to occur in essentially the same way, they form a standard candle whose intrinsic luminosity can be assumed |
| 3000 | ▁he ▁w as ▁p ar t ic u l ar l y ▁h a p p y ▁a t ▁w ü r z b ur g ▁w h er e ▁he ▁ en j o y ed ▁w al k s ▁a m o ng ▁the ▁h il l s ▁and ▁he ▁a l s o ▁m a de ▁f re q u ent ▁v is it s ▁to ▁the ▁b l a c k ▁f or es t . ▁h is ▁a d m in is t r at iv e ▁s i r ▁ j am es ▁w . ▁b l a c k ▁un it ed ▁k ing d om ▁ " f or ▁th ei r ▁d is c o ver i es ▁of ▁ im p or t an t ▁p r in ci p l es ▁f or ▁d r u g ▁t re at m ent " ▁s in c e ▁a l l ▁t y p e ▁ i a ▁s u p er n o v ae ▁a re ▁b el ie v ed ▁to ▁o c c ur ▁in ▁ es s ent i al l y ▁the ▁s am e ▁w a y , ▁the y ▁f or m ▁a ▁st and ar d ▁c and le ▁w h o s e ▁in t r in s ic ▁l um in o s it y ▁c an ▁b e ▁a s s um ed |
| 5000 | ▁he ▁was ▁part ic ular ly ▁h app y ▁at ▁w ür z bur g ▁where ▁he ▁ enj o y ed ▁w al k s ▁am ong ▁the ▁h ill s ▁and ▁he ▁also ▁m ade ▁fre qu ent ▁v is it s ▁to ▁the ▁bl ack ▁for est . ▁his ▁ad min ist r at ive ▁s ir ▁j am es ▁w . ▁bl ack ▁united ▁king dom ▁" for ▁their ▁discoveries ▁of ▁im p ort ant ▁pr inci pl es ▁for ▁d r ug ▁t reat ment " ▁sin ce ▁all ▁t yp e ▁ ia ▁supernov ae ▁are ▁b el ie ved ▁to ▁occ ur ▁in ▁ ess ent i ally ▁the ▁same ▁w ay , ▁they ▁form ▁a ▁stand ard ▁c and le ▁wh ose ▁in tr ins ic ▁l um in os ity ▁can ▁be ▁ass um ed |