BrotliDecoderDecompress() crashed if inbuf is part of mmaped file #231

@torappinfo commented on GitHub (Jun 2, 2018):

If alignment is helpful for performance, this bug needs not to be fixed. We just need to document it.

@torappinfo commented on GitHub (Jun 2, 2018): If alignment is helpful for performance, this bug needs not to be fixed. We just need to document it.

claunia commented

@eustas commented on GitHub (Jun 5, 2018):

Can't reproduce.
If you like, I can share modified research/brotli_decoder.c which reads from mmapped file from given offset, so you could check if it crashes as well.

@eustas commented on GitHub (Jun 5, 2018): Can't reproduce. If you like, I can share modified `research/brotli_decoder.c` which reads from mmapped file from given offset, so you could check if it crashes as well.

claunia commented

@jhudsoncedaron commented on GitHub (Jun 15, 2018):

I can give a clue how to try to reproduce this. I think the code tries to read past the end of the buffer and ignores the result. This will crash if the buffer is butt-up against the end of a page and the next page isn't allocated. This is easy to set up with a mmapped file and hard otherwise.

@jhudsoncedaron commented on GitHub (Jun 15, 2018): I can give a clue how to try to reproduce this. I think the code tries to read past the end of the buffer and ignores the result. This will crash if the buffer is butt-up against the end of a page and the next page isn't allocated. This is easy to set up with a mmapped file and hard otherwise.

claunia commented

2026-01-29 20:40:22 +00:00

@eustas commented on GitHub (Jun 19, 2018):

I've published branch with demo mmaped file decoder: https://github.com/google/brotli/tree/mmap-read
Could you provide file and offeset that would crash it, please.

@eustas commented on GitHub (Jun 19, 2018): I've published branch with demo mmaped file decoder: https://github.com/google/brotli/tree/mmap-read Could you provide file and offeset that would crash it, please.

claunia commented

@torappinfo commented on GitHub (Jul 27, 2018):

I'v compressed every 64k block of a large file with brotli and then concatenate them together. The compressed file has a header to provide offsets for each blocks.
The header format is as follows:

struct CONTENT_LOC {
  byte offset[5]; //Big endian/network order 5-byte integer 
};

struct HEADER {
  byte flags;
  CONTENT_LOC len_decompressed; //length of original uncompressed file
  CONTENT_LOC blocks[]; //offsets for each blocks, CONTENT_LOC[nBlocks+1] 
};

Here nBlocks=(len_decompressed-1)/64k + 1.
and the offset for the first block is zero, and second offset is the length of compressed first block etc.

In this format, BrotliDecoderDecompress() crashed for the very first block for my testing data.

The first 1MB of compressed file is attached here.
test.log

@torappinfo commented on GitHub (Jul 27, 2018): I'v compressed every 64k block of a large file with brotli and then concatenate them together. The compressed file has a header to provide offsets for each blocks. The header format is as follows: ``` struct CONTENT_LOC { byte offset[5]; //Big endian/network order 5-byte integer }; struct HEADER { byte flags; CONTENT_LOC len_decompressed; //length of original uncompressed file CONTENT_LOC blocks[]; //offsets for each blocks, CONTENT_LOC[nBlocks+1] }; ``` Here nBlocks=(len_decompressed-1)/64k + 1. and the offset for the first block is zero, and second offset is the length of compressed first block etc. In this format, BrotliDecoderDecompress() crashed for the very first block for my testing data. The first 1MB of compressed file is attached here. [test.log](https://github.com/google/brotli/files/2236017/test.log)

claunia commented

2026-01-29 20:40:22 +00:00

@torappinfo commented on GitHub (Jul 27, 2018):

The full compressed file (15MB) is here: (ahd.dict.tbr)
https://github.com/torappinfo/century-dictionary/releases/tag/v1

@torappinfo commented on GitHub (Jul 27, 2018): The full compressed file (15MB) is here: (ahd.dict.tbr) https://github.com/torappinfo/century-dictionary/releases/tag/v1

claunia commented

2026-01-29 20:40:22 +00:00

@mraszyk commented on GitHub (Jul 30, 2018):

I managed to successfully decompress the test file, i.e., I was not able to reproduce the crash.
Moreover, the uncompressed file (ahd.dict) has SHA256 hash
2f4197eab677a9344f3b4879fc5a65dcabb4b0b3df1497d50580936efde9a24b
which is different from century.md with SHA256 hash
aac17eb82496ba106cb2161da8016f3580ba23593e224b71fadb2a99d9005416

@mraszyk commented on GitHub (Jul 30, 2018): I managed to successfully decompress the test file, i.e., I was not able to reproduce the crash. Moreover, the uncompressed file (ahd.dict) has SHA256 hash 2f4197eab677a9344f3b4879fc5a65dcabb4b0b3df1497d50580936efde9a24b which is different from century.md with SHA256 hash aac17eb82496ba106cb2161da8016f3580ba23593e224b71fadb2a99d9005416

claunia commented

@torappinfo commented on GitHub (Jul 31, 2018):

The file is not century.md. I just put there as a test file, so the SHA256 is different.
When you decompressed the test file, did you mmap the whole test file (at offset 0)
and use BrotliDecoderDecompress(len,data+offset,&decoded_size,inBuffer) directly on it?

If you read the test file to a buffer, the data is probably properly aligned and nothing will be wrong.
But if you mmap the whole test file (including header) and directly use BrotliDecoderDecompress() on the address space. It crashed me every time.

My latest test is using :
Linux 4.17.2-1-ARCH #1 SMP PREEMPT UTC 2018 x86_64 GNU/Linux
Name : brotli
Version : 1.0.5-1
Description : Brotli compression library
Architecture : x86_64

@torappinfo commented on GitHub (Jul 31, 2018): The file is not century.md. I just put there as a test file, so the SHA256 is different. When you decompressed the test file, did you mmap the whole test file (at offset 0) and use BrotliDecoderDecompress(len,data+offset,&decoded_size,inBuffer) directly on it? If you read the test file to a buffer, the data is probably properly aligned and nothing will be wrong. But if you mmap the whole test file (including header) and directly use BrotliDecoderDecompress() on the address space. It crashed me every time. My latest test is using : Linux 4.17.2-1-ARCH #1 SMP PREEMPT UTC 2018 x86_64 GNU/Linux Name : brotli Version : 1.0.5-1 Description : Brotli compression library Architecture : x86_64

claunia commented

@jhudsoncedaron commented on GitHub (Aug 3, 2018):

@torappinfo : I'd recommending extracting out and uploading the two dozen lines required to set this up. I only ever got this with corrupted streams...

@jhudsoncedaron commented on GitHub (Aug 3, 2018): @torappinfo : I'd recommending extracting out and uploading the two dozen lines required to set this up. I only ever got this with corrupted streams...

claunia commented

@torappinfo commented on GitHub (Aug 6, 2018):

@mraszyk, Could you give your code to jhudsoncedaron, as you have already successfully extracted streams. The only thing to make sure segment fault is not to read compressed stream to buffer.

@torappinfo commented on GitHub (Aug 6, 2018): @mraszyk, Could you give your code to jhudsoncedaron, as you have already successfully extracted streams. The only thing to make sure segment fault is not to read compressed stream to buffer.

claunia commented

@mraszyk commented on GitHub (Aug 6, 2018):

I used the following code to decompress the test file ahd.dict.tbr and successfully obtained the uncompressed file ahd.dict also on Arch Linux:
https://drive.google.com/file/d/0ByXI7DhQRNE-a2t4bUJKTE54emR6V0pZMFV2eklpLTdlS0JF/view?usp=sharing

Files (with SHA256 hashes):
adbe6fa730efddbcf62a75134ece9ab8a50abf4465ed38af6de7898b11f07061 ahd.dict.tbr
2f4197eab677a9344f3b4879fc5a65dcabb4b0b3df1497d50580936efde9a24b ahd.dict

System: Linux arch 4.17.11-arch1 #1 SMP PREEMPT x86_64 GNU/Linux
Brotli version: 1.0.5

Please check the attached code if it crashes, and if it doesn't, please provide your code which does crash on decompressing ahd.dict.tbr.

@mraszyk commented on GitHub (Aug 6, 2018): I used the following code to decompress the test file ahd.dict.tbr and successfully obtained the uncompressed file ahd.dict also on Arch Linux: https://drive.google.com/file/d/0ByXI7DhQRNE-a2t4bUJKTE54emR6V0pZMFV2eklpLTdlS0JF/view?usp=sharing Files (with SHA256 hashes): adbe6fa730efddbcf62a75134ece9ab8a50abf4465ed38af6de7898b11f07061 ahd.dict.tbr 2f4197eab677a9344f3b4879fc5a65dcabb4b0b3df1497d50580936efde9a24b ahd.dict System: Linux arch 4.17.11-arch1 #1 SMP PREEMPT x86_64 GNU/Linux Brotli version: 1.0.5 Please check the attached code if it crashes, and if it doesn't, please provide your code which does crash on decompressing ahd.dict.tbr.

claunia commented

@jhudsoncedaron commented on GitHub (Aug 6, 2018):

@torappinfo : What are your FS and OS page sizes? (The FS page size is the size of an empty directory assuming you aren't using an exotic filesystem.) Your processor supports large pages but I can't tell if you're using them.

@jhudsoncedaron commented on GitHub (Aug 6, 2018): @torappinfo : What are your FS and OS page sizes? (The FS page size is the size of an empty directory assuming you aren't using an exotic filesystem.) Your processor supports large pages but I can't tell if you're using them.

claunia commented

@torappinfo commented on GitHub (Aug 6, 2018):

My FS and OS page sizes are both 4KB.

@torappinfo commented on GitHub (Aug 6, 2018): My FS and OS page sizes are both 4KB.

claunia commented

@torappinfo commented on GitHub (Aug 6, 2018):

mraszyk's code runs successfully on my machine.

@torappinfo commented on GitHub (Aug 6, 2018): mraszyk's code runs successfully on my machine.

claunia commented