mirror of
https://github.com/google/brotli.git
synced 2026-09-22 14:45:51 +00:00
Binary needed #216
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Sanmayce on GitHub (Feb 28, 2018).
Hi,
if there is not going to be binary in 'release' section, could you share inhere how to compile using ICL?
https://twitter.com/Sanmayce/status/965935926735196160
For various reasons I avoid using 'make' and such, a command line compilist (to differentiate from 'compiler', heh-heh) here.
The idea is we to have a command line tool like legendary PKZIP/PKUNZIP, I would use it on a daily basis, for example, currently I am running several big textual benchmarks:
To test the parsing prowess of 1GB window, this 900MB DNA set is quite good.
It is an understatement to call reddit test - big, it is quite literally tera - a teracorpus indeed!
My wish is to set/present a roofline :P (as opposed to baseline) by using the full power of Brotli with 1GB, currently my testmachine 'Compressionette' (i5-7200u, 8GB DDR4) halved the teratask:
zstd-v1.3.3-win64.exe -T2 -12 ...Wanna use bsc with 1024MB block, 7zip with 30bit window, and Zstd with 30bit window as well.
@jbergstroem commented on GitHub (Mar 2, 2018):
@Sanmayce using the
auto*stack in brotli 1.0.2 should make it easier for you to switch compiler (CC=icl ./configure).@Sanmayce commented on GitHub (Mar 2, 2018):
@jbergstroem Thanks, but I ask for static build compile line, not for continuation of those 'make' mumbojumboisms.
I remember made myself a static executable of some of first versions (IIRC they even were .CC), I am dismayed why (now in .C) it is so difficult we to have the compile command line?!
My wish is to compile and share two static executables - SSE2 and AVX2.
Johan, don't you agree that not having binary (and not using it) ruins the purpose? To me, such a powerful compressor with such speedy decompression deserves stable static binary, and naturally users to start amassing experience ... and just then be opinionated.
@Sanmayce commented on GitHub (Mar 18, 2018):
Just found in my archives an old brotli that I was able to compile with Intel v15:
AFAIR, three lines were manually modified in bro.cc:
I think the brackets were replaced with quotes.
Please, share inhere similar approach for compilation, I want to have binary compiled with /O3 /arch:CORE-AVX2, also please consider making built-in benchmark as Yann's Zstd -b one.
By the way I tried latest TurboBench but couldn't go past dictionary 24bit, that is, d29 failed to decompress?! Having built-in benchmark will be very informative in many ways, one of which is to test 16+MB windows heavily, some problems/drawbacks could be caught in the process. My personal experience with Conor's LZSSE2 (AVX2) and my Nakamichi was fruitful in that regard.
Simply stated, benchmarking yields much more than one usually expect.
Yesterday I ran, in a hurry, bench for one 15MB html testfile and overlooked the iterations for compress, should have increased them to the default 3 in order to have stable values, stupid, now to undo the damage I reran it and demanded all the 3x11 modes - from 1 to 11 with windows 20/22/24 bit:
The testmachine is a laptop running Windows 10, with i5-7200u 3GHz, DDR4 2133MHz.
Another thing that interests me, is it allowed dictionary to be, say, 16bit or 18bit? Is 30 the maximum?
@eustas commented on GitHub (Apr 3, 2018):
Just added windows binaries for v1.0.3, going to release v1.0.4 soon, and it will also feature windows binaries.
Added project to create JNI binaries as well (volunteers wanted).
@eustas commented on GitHub (Apr 3, 2018):
As before, it is possible to create something like
makeExe.bat, though it now will be longer =)In this article I describe an easy way to cross-compile for windows, with minimal requirements: just bash and docker. I think it is possible to reduce it just to docker; all the things done in bash is generation of 4 small files and launch of docker.
Using icl is difficult for legal reasons - IIRC, Intel prohibits to publish the results of compilation of opensource projects by corporate users =(
Also there is no easy way to install ICL without a key...
@eustas commented on GitHub (Apr 3, 2018):
Recently we have published "large window brotli" feature. Now window could be up to 30 bits. Perhaps this limit will be changed in future; 30 is chosen as a balance between utility and simplicity of implementation.
@Sanmayce commented on GitHub (Apr 3, 2018):
>Recently we have published "large window brotli" feature. Now window could be up to 30 bits.
Just downloaded and ran v 1.0.3, but couldn't see
The window is still up to 24, or 16MB, how one is supposed to run tests with 1GB?
As for Intel, I consider myself a fair tester since v12, Intel graciously allowed temporary key activation which, somehow, don't ask how, on my laptops reincarnated in next versions that I used to evaluate - v13, v14, v15. My belief and understanding is that Intel did good by allowing evaluating i.e. creating executables and seeing first-handedly how powerful really it is - all in the spirit of fair use, that is, no commercial or similar transgressions. For a time I used in parallel GCC 6.3.0, but this laptop died, didn't care to reinstall since new faster versions appeared.
@eustas commented on GitHub (Apr 3, 2018):
To make sure that admins do not start using "large window brotli" for http, we made it accessible only from API in v1.0.3
In v1.0.4 (and current tip of the tree) it is possible to use the feature from CLI.
I agree, fair use is a good thing. But evaluation- and integration testing-wise the licensing is done the wrong way. One can not use
iclfrom docker container or run it onTravisCI, just to make sure the source code compiles well on their compiler...@Sanmayce commented on GitHub (Apr 3, 2018):
>But evaluation- and integration testing-wise the licensing is done the wrong way. One can not use icl from docker container or run it on TravisCI, just to make sure the source code compiles well on their compiler...
Hm, that's not good indeed, even to my amateurish standards, in my view, one excellent tool/library/API should be compileable on vast set of platforms, the first that comes to mind is LZO, Markus did make it portable AFAIK. My C/environments knowledge is really basic to express opinions on portability issues, but one phrase caught my attention "Military Grade", somewhere on LZO descriptions, very ringy!
Looking forward testing v1.0.4, funny, no one benchmarked windows 15/16, wanna know how they fare against the superb LIBDEFLATE 12. Are there plans for making some built-in benchmark similar to Zstd's one? One of the reasons to want making my own executables with Intel v15.0 was to wrap up the [de]compression invocations with some basic time stats. The other variant is to rely on Lzbench/Turbobench - which I prefer actually.
@Sanmayce commented on GitHub (Apr 17, 2018):
Thanks to powturbo, for the first time I witnessed the 16+MB window of Brotli:
Is this not pure awesomeness:
lzturbo 59 being 35.37/0.35= 101.0x faster than zpaq 5
lzturbo 59 being 35.37/6.62= 5.3x faster than bsc 6
Still, in v.1.0.4 binary the window is limited to 24?!
If you wanna see a particular file benchmarked, just give me the URL ...
@Sanmayce commented on GitHub (Apr 17, 2018):
No, no, cannot miss one dataset that has been crying to be benchmarked for times unremembered- the W3 itself:
https://en.wikipedia.org/wiki/Webster%27s_Third_New_International_Dictionary
EDIT: Always love to get a hold of thickish books, shrinking this thickness to 125223987:15239397 or 8:1 ...
For those who are unaware of the fact, W3 is second only to OED. Ugh, the natural next step is guess what dataset... the paragonic 534MB (d29 is to flex muscles) of Oxford are under way...
@Sanmayce commented on GitHub (Apr 23, 2018):
Grmbl, the file has to be under ~450MB, the insufficient 8GB on 'Compressionette' are the cause for disfiguring the [de]compression rates for OED, due to HDD thrashing, so I will run it when 16GB are available.
Continuing exploring the bigger Brotli's windows, inhere 27+bit - the full documentation folder of Intel Parallel Studio XE - 166 MB tarred file.
The latest Brotli with larger window beats LZMA in compression ratio and in decompression rate - 339/95= 3.5x faster!
Bah, just look at LzTurbo 39:
The next one will be the most popular dataset - SCC - Silesia Compression Corpus ...
@Sanmayce commented on GitHub (Apr 24, 2018):
Crunching 211,938,580 bytes long Silesia_compression_corpus ...
Failed:
Crunching 465,457,152 bytes long mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar ...
Failed:
The standout is oodle 118, it amazes having no counterpart/rival in the roster, Oodle 'Selkie' excels at binary-ish data!
Next is www.holybooks.com_70_PDFs.tar - a 195,587,584 bytes collection of 70 Yoga ebooks non-scanned:
Not at all interested in binary decompression, yet, it is interesting to see how poorly some decompressors behave.
Failed:
Having downloaded RC_2008-08 (from https://files.pushshift.io/reddit/comments/), which is 346,626,502 bytes, next to be seen is an .JSON dump of reddit...
Failed:
One .log type file:
Failed:
The idea is to cover at least these formats:
@Sanmayce commented on GitHub (Apr 25, 2018):
It is a good idea to juxtapose two ~300MB .XML dumps...
Next one is gonna be the .XML dump of "Wikipedia's evil twin", according to the article:
https://en.wikipedia.org/wiki/Encyclopedia_Dramatica
Crunching 295,515,893 bytes long encyclopediadramaticase-20150628-current.xml ...
TurboBench: - Thu Apr 26 11:44:32 2018
(bold = pareto) MB=1.000.000
Having finished the main run, above table was done by running:
Still prefer the RAW output, however the Pareto Frontier performers are not in bold:
Failed:
Another .XML dump is coming, in order to scramble the biased enwik8 notions, the first 300MB of 60,182,193,037 bytes long enwiki-20170101-pages-articles.xml will "promote" the stronger ones/performers... RIGHTFULLY SO, those who overtuned their codecs to ENWIK8 could also witness the outcome when 200 extra kilometers are to be trodden :P
TurboBench: - Thu Apr 26 23:11:13 2018
(bold = pareto) MB=1.000.000
Failed:
Oh, and I almost forgot the most interesting to me format - English .TXT tarred anthologies - reckon the Star Trek fans have to be allowed to see what decompressors to choose when the need of opening AT WARP SPEED (WARP is not just a word, you know, there was OS/2 on top of that) the ULTIMATE Collection Star Trek ebook arises ...
Crunching 325,071,872 bytes long Star_Trek_-_737_Ebooks.tar ...
The Markdown generated by TurboBench:
TurboBench: - Thu Apr 26 11:44:08 2018
(bold = pareto) MB=1.000.000
The RAW:
Failed:
Okay, let us see how the results for the two 300MB .XML encyclopedia dumps (Wikipedia and Dramatica) relate:
Dramatica:
Brotli decompresses 270/103= 2.6x faster than LZMA
LzTurbo decompresses 770/270= 2.8x faster than Brotli
Wikipedia:
Brotli decompresses 215/75= 2.8x faster than LZMA
LzTurbo decompresses 572/215= 2.6x faster than Brotli
Star_Trek:
Brotli decompresses 232/83= 2.7x faster than LZMA
LzTurbo decompresses 564/232= 2.4x faster than Brotli
So, three things:
In my previous tests I witnessed difference between Oodle 129 and Oodle 89, AFAIK being respectively 'Hydra' and 'Kraken', the former being much faster on DNA data, also in Reddit test:
So, the final outcome will be a single table showing how the 4 performers relate.
@Sanmayce commented on GitHub (Apr 26, 2018):
As the French say, "L'appétit vient en mangeant.":
Idiomatic translation: The more you have, the more you want.
Literal meaning: Appetite comes while eating.
Reckon, the .DNA and .CSV should also contribute to the drawing of the picture...
EDIT: It turns out, I overlooked the TurboBench ability to benchmark an arbitrary long chunk at beginning by using the option "-B" . Ex. "-B200" for 200.000.000 bytes. Only the first 200MB will be used in the benchmark.
Also, to feint the drawback (the insufficient 8GB on my testmachine 'Compressionette') wrote the console utility 'Chunkerito', can download its .C source and executable, Get_The_First_300MB_chunk.zip (38,741 bytes):
The same chunkenization done on the 'Dragonfly' DNA sequence ~960MB long:
Crunching 314,572,800 bytes long www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB ...
TurboBench: - Fri Apr 27 21:51:45 2018
(bold = pareto) MB=1.000.000
Failed:
So, the testsets that are to be processed:
Chinese comments in .CSV, a must-see one it is.
Rethinking how scattered all the benchmark packages are, once I did above ones, surely will put all the corpora in one RAZOR archive, to pay tribute to the excellent work done by Christian Martelock, thus as a "by-product" or bonus, we can see how the whole/solid compression goes on all the testsets as a whole!
@Sanmayce commented on GitHub (Apr 30, 2018):
Okay, today's post will be the last since I finished the benchmarking and just 2 hours away from uploading the corpus housing all the testsets benchmarked on this thread.
The rest are given one by one, after that the promised table of speedups, and in the very end, the link to the corpus will be given.
The first one is very interesting being superimportant languagewise/inflectionwise, each word being inflected and tagged, a precious corpus indeed!
Crunching 282,218,054 bytes long Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt ...
TurboBench: - Sun Apr 29 17:12:27 2018
(bold = pareto) MB=1.000.000
Failed:
Inhere, lzturbo 39 dominates, however for my surprise lzturbo 59 fails to compete with bsc 6, that's one of the gifts of benchmarking, it always shows something overlooked, a weakness which needs addressing.
The next one holds precious code from the legendary Watcom C.
Crunching 259,707,904 bytes long open_watcom_1.9.0-src.tar ...
TurboBench: - Mon Apr 30 07:51:32 2018
(bold = pareto) MB=1.000.000
Failed:
Two standouts (except the usual number one lzturbo 39), zstd 1 and oodle 114 shine, or rather darken the rest casting long shadows.
Crunching 405,610,647 bytes long Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv ...
TurboBench: - Sun Apr 29 08:33:59 2018
(bold = pareto) MB=1.000.000
Failed:
Inhere, one particular thing got me thinking, the 1GB/s difference between the superb performers lzturbo 19 and lizard 19, being on par sizewise, sometimes decompression rates are on par NOT at all.
Crunching 150,950,913 bytes long Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv ...
TurboBench: - Sat Apr 28 07:35:55 2018
(bold = pareto) MB=1.000.000
Inhere, my favorite LzTurbo mode, 29, performs over the top.
Crunching 203,288,144 bytes long Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv ...
TurboBench: - Sat Apr 28 12:29:23 2018
(bold = pareto) MB=1.000.000
Failed:
Hm, those headlines, brotli 11d29, lzturbo 39 and zstd 22 are not that different decompression-rate-wise.
@Sanmayce commented on GitHub (Apr 30, 2018):
Okay, the corpus deserves separate post:
Superpig_Corpus.rz 707 MB (741,571,263 bytes), downloadable at:
https://drive.google.com/file/d/1WJkur8Gv-gNk9H_nBLMUpnY7_AJkYTrp/view?usp=sharing
RAZOR with its 512MB window fits perfectly in my laptop 6.8GB free RAM heap, not touching the HDD with virtual RAM.
@Sanmayce commented on GitHub (Apr 30, 2018):
Finally, the battle of the strongest fast decompressors on my laptop 'Compressionette':
The roofline/baseline is given by LzTurbo 39, thus instead of 'Speedup' the metric is called 'Speeddown' - how many times the performer is slower.
Note: Oops, just realized that AFAIK Zstd 22 is using 128MB window, should have set it to 512MB as well, to be addressed... wonder what windows Oodle 129 and LzTurbo 39 are using!
open_watcom_1.9.0-src.tar (259,707,904 bytes):
Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt (282,218,054 bytes):
Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv (405,610,647 bytes):
Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv (203,288,144 bytes):
Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv (150,950,913 bytes):
www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB (314,572,800 bytes):
enwiki-20170101-pages-articles.xml_first_300MB (314,572,800 bytes):
Star_Trek_-_737_Ebooks.tar (325,071,872 bytes):
encyclopediadramaticase-20150628-current.xml (295,515,893 bytes):
NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95 (205,242,368 bytes):
RC_2008-08 (346,626,502 bytes):
www.holybooks.com_70_PDFs.tar (195,587,584 bytes):
Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar (107,784,192 bytes):
Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl (125,234,552 bytes):
Documentation_Composer_XE_2015.tar (174,100,480 bytes):
Silesia_compression_corpus (211,938,580 bytes):
mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar (465,457,152 bytes):
To see how better the 29bit Zstd built-in (--wlog) and the LDM/deduplicator (--long) are when compared to the vanilla 22 mode I ran:
The first one delivered these:
The second one delivered these:
For 'Douban' I had to stop the HDD thrashing, it needs more than 6.8GB RAM.
The Large Window (128+MB) delivers better ratio, most noticeably for 'Star_Trek_-_737_Ebooks.tar', almost 2MB gain.
With this I close benchmarking the 'Superpig' testsets, hope it can be instrumental.
Also, closing the thread, kinda disappointed of lack of binary capable to work with 29bit and report speeds, whatever, it is what it is, as Diaz bros. say.
@Sanmayce commented on GitHub (May 2, 2018):
Reopened, in order to allow feedback. From my side I shared what I wanted.
@Brotliteam Feel free to close it whenever you see fit.
To me, the superswinish benchmark reveals some hidden facets of all the supercode