BPEmb: ty

Tahitian (ty) subword embeddings

Vocab size	vocab	model	25 dim	50 dim	100 dim	200 dim	300 dim
1000	vocab	model	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix
3000	vocab	model	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix
5000	vocab	model	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix	txt \| bin bokeh \| umap \| matrix

View Tahitian subword embeddings in TensorFlow Projector

2-D UMAP plots

Embedding matrix plots

Training corpus sample, encoded with different BPE vocabulary sizes

Vocab size	tywiki sample
original	aita e nanumiti no te âvaè, no te mahana rā. no reira, e pāhee noa ā te tai i te poìpoì roa e i te ahiahi, e pānunu te tai i tuìraa pō e i te avatea. goldingare láse zse nyásze, nyovergij strommij lá dzsávre di császte. lipo-rimnigueij kinyáresz de vácse, áles párlámingszej jilyngász lá prizenyijpov i ’europa te fenua heremani (, ) . ’o berlin te ’oire pū. ’o joachim gauck te peretiteni, te reo heremani to na reo.
1000	▁aita ▁e ▁na nu m iti ▁no ▁te ▁âva è , ▁no ▁te ▁mahana ▁rā . ▁no ▁reira , ▁e ▁pā he e ▁noa ▁ā ▁te ▁tai ▁i ▁te ▁po ì po ì ▁roa ▁e ▁i ▁te ▁a hia hi , ▁e ▁pā nu nu ▁te ▁tai ▁i ▁tu ì raa ▁pō ▁e ▁i ▁te ▁a va tea . ▁g ol d ing a re ▁lá se ▁z se ▁ny ász e , ▁ny o ver gi j ▁st ro m mi j ▁lá ▁d z sá vre ▁di ▁c s ász te . ▁li po - ri m ni g u ei j ▁k iny áresz ▁de ▁v á c se , ▁á les ▁p á rl á m ing sz e j ▁j il y ng ász ▁lá ▁p ri z eny i j po v ▁i ▁’ europa ▁te ▁fenua ▁heremani ▁(, ▁) ▁. ▁’ o ▁b er lin ▁te ▁’ oire ▁pū . ▁’ o ▁j oa c hi m ▁g au ck ▁te ▁peretiteni , ▁te ▁reo ▁heremani ▁to ▁na ▁reo .
3000	▁aita ▁e ▁na numiti ▁no ▁te ▁âvaè , ▁no ▁te ▁mahana ▁rā . ▁no ▁reira , ▁e ▁pā he e ▁noa ▁ā ▁te ▁tai ▁i ▁te ▁po ì po ì ▁roa ▁e ▁i ▁te ▁a hia hi , ▁e ▁pā nunu ▁te ▁tai ▁i ▁tu ìraa ▁pō ▁e ▁i ▁te ▁ava tea . ▁gold ingare ▁láse ▁zse ▁ny ásze , ▁ny over gij ▁st rommij ▁lá ▁dzsá vre ▁di ▁cs ászte . ▁lipo - ri mni gueij ▁kiny áresz ▁de ▁v ácse , ▁áles ▁p árlám ing szej ▁jily ngász ▁lá ▁priz enyij po v ▁i ▁’ europa ▁te ▁fenua ▁heremani ▁(, ▁) ▁. ▁’ o ▁b erlin ▁te ▁’ oire ▁pū . ▁’ o ▁j oachim ▁g auck ▁te ▁peretiteni , ▁te ▁reo ▁heremani ▁to ▁na ▁reo .
5000	▁aita ▁e ▁nanumiti ▁no ▁te ▁âvaè , ▁no ▁te ▁mahana ▁rā . ▁no ▁reira , ▁e ▁pāhee ▁noa ▁ā ▁te ▁tai ▁i ▁te ▁poìpoì ▁roa ▁e ▁i ▁te ▁ahiahi , ▁e ▁pānunu ▁te ▁tai ▁i ▁tuìraa ▁pō ▁e ▁i ▁te ▁avatea . ▁goldingare ▁láse ▁zse ▁nyásze , ▁nyovergij ▁strommij ▁lá ▁dzsávre ▁di ▁császte . ▁lipo - rimnigueij ▁kinyáresz ▁de ▁vácse , ▁áles ▁párlámingszej ▁jilyngász ▁lá ▁priz enyij po v ▁i ▁’ europa ▁te ▁fenua ▁heremani ▁(, ▁) ▁. ▁’ o ▁berlin ▁te ▁’ oire ▁pū . ▁’ o ▁joachim ▁gauck ▁te ▁peretiteni , ▁te ▁reo ▁heremani ▁to ▁na ▁reo .