Maori (mi) subword embeddings

Vocab size vocab model 25 dim 50 dim 100 dim 200 dim 300 dim
1000 vocab model txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
3000 vocab model txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
5000 vocab model txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix
txt | bin
bokeh | umap | matrix

Embedding matrix plots

Training corpus sample, encoded with different BPE vocabulary sizes

Vocab sizemiwiki sample
original ko pakanae he nohanga o te tai-tokerau. e ai ki toitū te whenua, he nohanga (he 'locality' rānei ki te reo pākehā) he wāhi nohoia e te tangata, engari
ko otara he nohanga o te moana-a-toi ki te rāwhiti. e ai ki toitū te whenua, he nohanga (he 'locality' rānei ki te reo pākehā) he wāhi nohoia e te tan
kei roto a waianiwa i te takiwā o murihikuko waianiwa he nohanga o te takiwā o murihiku. e ai ki toitū te whenua, he nohanga (he 'locality' rānei ki t
1000 ▁ko ▁pa ka na e ▁he ▁nohanga ▁o ▁te ▁tai - tokerau . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁tangata , ▁engari
▁ko ▁o tara ▁he ▁nohanga ▁o ▁te ▁moana - a - toi ▁ki ▁te ▁rāwhiti . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁ta n
▁kei ▁roto ▁a ▁waia ni wa ▁i ▁te ▁takiwā ▁o ▁murihikuko ▁waia ni wa ▁he ▁nohanga ▁o ▁te ▁takiwā ▁o ▁murihiku . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁t
3000 ▁ko ▁pa kana e ▁he ▁nohanga ▁o ▁te ▁tai - tokerau . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁tangata , ▁engari
▁ko ▁o tara ▁he ▁nohanga ▁o ▁te ▁moana - a - toi ▁ki ▁te ▁rāwhiti . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁ta n
▁kei ▁roto ▁a ▁waia niwa ▁i ▁te ▁takiwā ▁o ▁murihikuko ▁waia niwa ▁he ▁nohanga ▁o ▁te ▁takiwā ▁o ▁murihiku . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁t
5000 ▁ko ▁pa kana e ▁he ▁nohanga ▁o ▁te ▁tai - tokerau . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁tangata , ▁engari
▁ko ▁otara ▁he ▁nohanga ▁o ▁te ▁moana - a - toi ▁ki ▁te ▁rāwhiti . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁ta n
▁kei ▁roto ▁a ▁waia niwa ▁i ▁te ▁takiwā ▁o ▁murihikuko ▁waia niwa ▁he ▁nohanga ▁o ▁te ▁takiwā ▁o ▁murihiku . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁t