Pangasinan (pag) subword embeddings
| Vocab size | vocab | model | 25 dim | 50 dim | 100 dim | 200 dim | 300 dim |
|---|---|---|---|---|---|---|---|
| 1000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 3000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 5000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 10000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
Training corpus sample, encoded with different BPE vocabulary sizes
| Vocab size | pagwiki sample |
|---|---|
| original | * libod * ligban * mabunga * magogon * manawan * maninila * mina * miti * palanog * panoypoy * pariaan * quinartilan * quirangay * quitinday * salugan * philippine standard geographic code * philippine census information * local governance performance management system say veruela et kumadwa ya klase baley ed luyag na agusan del sur, pilipinas. unong ed 0000 census, say papulasyon to et 00,000 totoo. walay kabaleg ta |
| 1000 | ▁* ▁lib od ▁* ▁l ig ban ▁* ▁mab ung a ▁* ▁mag ogon ▁* ▁man awan ▁* ▁man in ila ▁* ▁m ina ▁* ▁m it i ▁* ▁pal an og ▁* ▁pan oy p oy ▁* ▁par i aan ▁* ▁qu in ar t ilan ▁* ▁qu ir angay ▁* ▁qu it ind ay ▁* ▁sal ugan ▁* ▁philippine ▁standard ▁geographic ▁code ▁* ▁philippine ▁census ▁information ▁* ▁local ▁governance ▁performance ▁management ▁system ▁say ▁v er uel a ▁et ▁kumadwa ▁ya ▁klase ▁baley ▁ed ▁luyag ▁na ▁agus an ▁del ▁sur , ▁pilipinas . ▁unong ▁ed ▁0000 ▁census , ▁say ▁papulasyon ▁to ▁et ▁00,000 ▁totoo . ▁walay ▁kabaleg ▁ta |
| 3000 | ▁* ▁lib od ▁* ▁l ig ban ▁* ▁mab unga ▁* ▁mag ogon ▁* ▁man awan ▁* ▁man in ila ▁* ▁m ina ▁* ▁m iti ▁* ▁pal an og ▁* ▁pan oy p oy ▁* ▁par i aan ▁* ▁quin art ilan ▁* ▁qu ir angay ▁* ▁qu it ind ay ▁* ▁sal ugan ▁* ▁philippine ▁standard ▁geographic ▁code ▁* ▁philippine ▁census ▁information ▁* ▁local ▁governance ▁performance ▁management ▁system ▁say ▁v er uel a ▁et ▁kumadwa ▁ya ▁klase ▁baley ▁ed ▁luyag ▁na ▁agusan ▁del ▁sur , ▁pilipinas . ▁unong ▁ed ▁0000 ▁census , ▁say ▁papulasyon ▁to ▁et ▁00,000 ▁totoo . ▁walay ▁kabaleg ▁ta |
| 5000 | ▁* ▁lib od ▁* ▁lig ban ▁* ▁mab unga ▁* ▁mag ogon ▁* ▁man awan ▁* ▁man in ila ▁* ▁mina ▁* ▁m iti ▁* ▁pal an og ▁* ▁pan oy p oy ▁* ▁pari aan ▁* ▁quin art ilan ▁* ▁qu ir angay ▁* ▁qu it ind ay ▁* ▁sal ugan ▁* ▁philippine ▁standard ▁geographic ▁code ▁* ▁philippine ▁census ▁information ▁* ▁local ▁governance ▁performance ▁management ▁system ▁say ▁ver uela ▁et ▁kumadwa ▁ya ▁klase ▁baley ▁ed ▁luyag ▁na ▁agusan ▁del ▁sur , ▁pilipinas . ▁unong ▁ed ▁0000 ▁census , ▁say ▁papulasyon ▁to ▁et ▁00,000 ▁totoo . ▁walay ▁kabaleg ▁ta |
| 10000 | ▁* ▁libod ▁* ▁lig ban ▁* ▁mabunga ▁* ▁mag ogon ▁* ▁man awan ▁* ▁maninila ▁* ▁mina ▁* ▁m iti ▁* ▁palan og ▁* ▁panoy poy ▁* ▁pari aan ▁* ▁quin art ilan ▁* ▁qu ir angay ▁* ▁qu it inday ▁* ▁sal ugan ▁* ▁philippine ▁standard ▁geographic ▁code ▁* ▁philippine ▁census ▁information ▁* ▁local ▁governance ▁performance ▁management ▁system ▁say ▁ver uela ▁et ▁kumadwa ▁ya ▁klase ▁baley ▁ed ▁luyag ▁na ▁agusan ▁del ▁sur , ▁pilipinas . ▁unong ▁ed ▁0000 ▁census , ▁say ▁papulasyon ▁to ▁et ▁00,000 ▁totoo . ▁walay ▁kabaleg ▁ta |