Poor compression rate with languages not in the dictionary #199

New Issue

claunia · 2026-01-29T20:39:44Z

claunia commented

2026-01-29 20:39:44 +00:00

Originally created by @spixi on GitHub (Oct 10, 2017).

The dictionary is only optimized for English, Spanish, Chinese, Hindi, Russian, Arabic, HTML, JavaScript and CSS. For texts in other languages, like German, brotli may have a worse compression rate than bzip2.

I used the default settings for

gcab-0.7 (with option -z)
brotli-0.6.0
bzip2-1.0.6
gzip-1.8
zip-3.0 (Info-ZIP)
xz-utils-5.2.3
p7zip-16.02
lzip-1.16

This is my result for du -ab
83437 ./goethe.cab
190778 ./goethe.txt
69034 ./goethe.br
61132 ./goethe.bzip2
77671 ./goethe.gz
77812 ./goethe.zip
69296 ./goethe.xz
69364 ./goethe.txt.7z
69303 ./goethe.lz

This is my test file: goethe.txt

I used German as an example, because that language is spoken by about 100 Million native speakers and it is the most most widely spoken mother tongue in the European Union, even before English and French. (See here).

Originally created by @spixi on GitHub (Oct 10, 2017). The dictionary is only optimized for English, Spanish, Chinese, Hindi, Russian, Arabic, HTML, JavaScript and CSS. For texts in other languages, like German, brotli may have a worse compression rate than bzip2. I used the default settings for gcab-0.7 (with option -z) brotli-0.6.0 bzip2-1.0.6 gzip-1.8 zip-3.0 (Info-ZIP) xz-utils-5.2.3 p7zip-16.02 lzip-1.16 This is my result for du -ab 83437 ./goethe.cab 190778 ./goethe.txt 69034 ./goethe.br 61132 ./goethe.bzip2 77671 ./goethe.gz 77812 ./goethe.zip 69296 ./goethe.xz 69364 ./goethe.txt.7z 69303 ./goethe.lz This is my test file: [goethe.txt](https://github.com/google/brotli/files/1373544/goethe.txt) I used German as an example, because that language is spoken by about 100 Million native speakers and it is the most most widely spoken mother tongue in the European Union, even before English and French. (See [here](http://ec.europa.eu/commfrontoffice/publicopinion/archives/ebs/ebs_386_en.pdf#%5B%7B%22num%22%3A11%2C%22gen%22%3A0%7D%2C%7B%22name%22%3A%22XYZ%22%7D%2C68%2C771%2C0%5D)).

claunia closed this issue

2026-01-29 20:39:44 +00:00

claunia commented

2026-01-29 20:39:45 +00:00

@eustas commented on GitHub (Oct 11, 2017):

This is a known problem. We work on this topic. Next brotli version will support more (all) languages.

@eustas commented on GitHub (Oct 11, 2017): This is a known problem. We work on this topic. Next brotli version will support more (all) languages.

claunia referenced this issue

2026-01-29 20:50:57 +00:00

[PR #199] [MERGED] Update brotli decoder with latest improvements. #797

claunia referenced this issue

2026-01-29 20:50:57 +00:00