MultiBPEmb: Shared subword vocabularies and embeddings in 275 languages

MultiBPEmb is a collection of multilingual subword segmentation models and pretrained subword embeddings. Like the monolingual models in BPEmb, segmentation models and embeddings are trained on Wikipedia. The difference is that, instead of training one segmentation model and embedding for each language, we now train a single model and a single embedding for all languages. Since the underlying algorithms are language-agnostic, this is quite simple to do: Concatenate the article texts from all Wikipedia editions, learn a subword segmentation model, and then train subword embeddings. Like in BPEmb, we use SentencePiece to learn BPE subword segmentation models, and GloVe to train subword embeddings. We provide 300-dimensional subword embeddings for three BPE vocabulary sizes.

You can use MultiBPEmb in Python like this:
>>> from bpemb import BPEmb
>>> multibpemb = BPEmb(lang="multi", vs=1000000, dim=300)
>>> text = 'John F. Kennedy said "Ich bin ein Pfannkuchen". 这是一个中文句子.日本語の文章です。'
>>> subwords = multibpemb.encode(text)
>>> print(" ".join(subwords))
▁john ▁f . ▁kennedy ▁said ▁" ich ▁bin ▁ein ▁pfann kuchen ". ▁这 是一个 中文 句子 . ▁日本 語の 文章 です 。

Or download subword segmentation models and embeddings here:

Vocab size	vocab	model	300 dim
100000	vocab	model	txt (255.1 MB) | bin (107.0 MB) bokeh (3.9 MB) | umap | matrix
320000	vocab	model	txt (816.3 MB) | bin (342.7 MB) bokeh (13.4 MB) | umap | matrix
1000000	vocab	model	txt (2550.0 MB) | bin (1071.3 MB) bokeh (44.6 MB) | umap | matrix

Embedding matrix plots

Training corpus sample, encoded with different BPE vocabulary sizes