Maori (mi) subword embeddings
| Vocab size | vocab | model | 25 dim | 50 dim | 100 dim | 200 dim | 300 dim |
|---|---|---|---|---|---|---|---|
| 1000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 3000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 5000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
Training corpus sample, encoded with different BPE vocabulary sizes
| Vocab size | miwiki sample |
|---|---|
| original | ko pakanae he nohanga o te tai-tokerau. e ai ki toitū te whenua, he nohanga (he 'locality' rānei ki te reo pākehā) he wāhi nohoia e te tangata, engari ko otara he nohanga o te moana-a-toi ki te rāwhiti. e ai ki toitū te whenua, he nohanga (he 'locality' rānei ki te reo pākehā) he wāhi nohoia e te tan kei roto a waianiwa i te takiwā o murihikuko waianiwa he nohanga o te takiwā o murihiku. e ai ki toitū te whenua, he nohanga (he 'locality' rānei ki t |
| 1000 | ▁ko ▁pa ka na e ▁he ▁nohanga ▁o ▁te ▁tai - tokerau . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁tangata , ▁engari ▁ko ▁o tara ▁he ▁nohanga ▁o ▁te ▁moana - a - toi ▁ki ▁te ▁rāwhiti . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁ta n ▁kei ▁roto ▁a ▁waia ni wa ▁i ▁te ▁takiwā ▁o ▁murihikuko ▁waia ni wa ▁he ▁nohanga ▁o ▁te ▁takiwā ▁o ▁murihiku . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁t |
| 3000 | ▁ko ▁pa kana e ▁he ▁nohanga ▁o ▁te ▁tai - tokerau . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁tangata , ▁engari ▁ko ▁o tara ▁he ▁nohanga ▁o ▁te ▁moana - a - toi ▁ki ▁te ▁rāwhiti . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁ta n ▁kei ▁roto ▁a ▁waia niwa ▁i ▁te ▁takiwā ▁o ▁murihikuko ▁waia niwa ▁he ▁nohanga ▁o ▁te ▁takiwā ▁o ▁murihiku . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁t |
| 5000 | ▁ko ▁pa kana e ▁he ▁nohanga ▁o ▁te ▁tai - tokerau . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁tangata , ▁engari ▁ko ▁otara ▁he ▁nohanga ▁o ▁te ▁moana - a - toi ▁ki ▁te ▁rāwhiti . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁te ▁reo ▁pākehā ) ▁he ▁wāhi ▁nohoia ▁e ▁te ▁ta n ▁kei ▁roto ▁a ▁waia niwa ▁i ▁te ▁takiwā ▁o ▁murihikuko ▁waia niwa ▁he ▁nohanga ▁o ▁te ▁takiwā ▁o ▁murihiku . ▁e ▁ai ▁ki ▁toitū ▁te ▁whenua , ▁he ▁nohanga ▁( he ▁' locality ' ▁rānei ▁ki ▁t |