Lingala (ln) subword embeddings
| Vocab size | vocab | model | 25 dim | 50 dim | 100 dim | 200 dim | 300 dim |
|---|---|---|---|---|---|---|---|
| 1000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 3000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 5000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
| 10000 | vocab | model |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
txt |
bin
bokeh | umap | matrix |
Training corpus sample, encoded with different BPE vocabulary sizes
| Vocab size | lnwiki sample |
|---|---|
| original | pɔ́sita, nzela ya nsinga mpé ... ptt (postes, téléphones et télécommunications) : kiamusoke basumulonga chloroquine eza nkisi eye basalelaka mpo na komibatela mpe kobikisa malaria na bisika biye malaria etungisaka mpeza batu. lolenge songolo ya malaria, * charles maccio, ''penser le devenir de l'humanité : construire une démocratie participative…'', lyon : chronique sociale, 0000. |
| 1000 | ▁p ɔ ́ si ta , ▁nzela ▁ya ▁nsi nga ▁mpé ▁. . . ▁p t t ▁( po st es , ▁té lé ph on es ▁et ▁té lé co m m uni ca tions ) ▁: ▁ki a m us o ke ▁ba su mu lo nga ▁ch lo ro qu ine ▁eza ▁nkisi ▁eye ▁bas al elaka ▁mpo ▁na ▁ko mi bat ela ▁mpe ▁ko bi kisa ▁mala ria ▁na ▁b isi ka ▁biye ▁mala ria ▁etu ngisa ka ▁mpeza ▁batu . ▁lolenge ▁so ngo lo ▁ya ▁mala ria , ▁* ▁ch ar les ▁ma c ci o , ▁'' p ens er ▁le ▁de ven ir ▁de ▁l ' h um ani té ▁: ▁con st ru ire ▁un e ▁dé mo c ra ti e ▁par ti ci pa ti ve . . . '', ▁l y on ▁: ▁ch r oni que ▁soci ale , ▁0000. |
| 3000 | ▁p ɔ ́ sita , ▁nzela ▁ya ▁nsinga ▁mpé ▁... ▁p tt ▁( po st es , ▁té lé ph on es ▁et ▁té lé comm uni cations ) ▁: ▁ki a mus o ke ▁ba su mu lo nga ▁ch lo ro qu ine ▁eza ▁nkisi ▁eye ▁basal elaka ▁mpo ▁na ▁komibatela ▁mpe ▁kobikisa ▁mala ria ▁na ▁bisika ▁biye ▁mala ria ▁etungisaka ▁mpeza ▁batu . ▁lolenge ▁songolo ▁ya ▁mala ria , ▁* ▁char les ▁ma c ci o , ▁'' p ens er ▁le ▁de ven ir ▁de ▁l ' h um ani té ▁: ▁constru ire ▁une ▁démocra tie ▁parti ci pa tive ... '', ▁ly on ▁: ▁ch r oni que ▁soci ale , ▁0000. |
| 5000 | ▁p ɔ ́ sita , ▁nzela ▁ya ▁nsinga ▁mpé ▁... ▁p tt ▁( po st es , ▁télé phon es ▁et ▁télé comm uni cations ) ▁: ▁ki a mus o ke ▁ba su mu lo nga ▁ch lo ro qu ine ▁eza ▁nkisi ▁eye ▁basalelaka ▁mpo ▁na ▁komibatela ▁mpe ▁kobikisa ▁malaria ▁na ▁bisika ▁biye ▁malaria ▁etungisaka ▁mpeza ▁batu . ▁lolenge ▁songolo ▁ya ▁malaria , ▁* ▁charles ▁ma c ci o , ▁'' p ens er ▁le ▁de ven ir ▁de ▁l ' hum ani té ▁: ▁constru ire ▁une ▁démocra tie ▁participa tive ... '', ▁ly on ▁: ▁ch r onique ▁soci ale , ▁0000. |
| 10000 | ▁pɔ ́ sita , ▁nzela ▁ya ▁nsinga ▁mpé ▁... ▁p tt ▁( po st es , ▁télé phones ▁et ▁télé communi cations ) ▁: ▁kia mus o ke ▁basu mu lo nga ▁ch loroquine ▁eza ▁nkisi ▁eye ▁basalelaka ▁mpo ▁na ▁komibatela ▁mpe ▁kobikisa ▁malaria ▁na ▁bisika ▁biye ▁malaria ▁etungisaka ▁mpeza ▁batu . ▁lolenge ▁songolo ▁ya ▁malaria , ▁* ▁charles ▁ma c cio , ▁'' p ens er ▁le ▁de venir ▁de ▁l ' humanité ▁: ▁constru ire ▁une ▁démocratie ▁participa tive ... '', ▁ly on ▁: ▁chronique ▁sociale , ▁0000. |