There is no CRC in the format and there is no way to detect corruption #394

New Issue

claunia · 2026-01-29T20:43:13Z

claunia commented

2026-01-29 20:43:13 +00:00

Originally created by @eshkinkot on GitHub (Aug 30, 2021).

git clone https://github.com/google/brotli.git
cd brotli/
make

echo 'this is test file for check compression checksums' > file
bin/brotli file
cp file.br file-corrupt.br
echo -ne '\xA6' | dd bs=1 conv=notrunc seek=$((0x18)) of=file-corrupt.br

$ hd file.br; hd file-corrupt.br
00000000  a1 88 01 00 2f b0 01 e7  0b ed 32 4e 7d 0f 52 88  |..../.....2N}.R.|
00000010  2a c0 68 10 2a 86 63 e7  b6 30 5f 4b 50 3e b3 40  |*.h.*.c..0_KP>.@|
00000020  bc 08                                             |..|
00000022
00000000  a1 88 01 00 2f b0 01 e7  0b ed 32 4e 7d 0f 52 88  |..../.....2N}.R.|
00000010  2a c0 68 10 2a 86 63 e7  a6 30 5f 4b 50 3e b3 40  |*.h.*.c..0_KP>.@|
00000020  bc 08                                             |..|
00000022
$ bin/brotli -v -t file-corrupt.br; echo $?
Decompressed [file-corrupt.br]: 34 B -> 50 B in 0.00 sec
0
$ bin/brotli -v -d file-corrupt.br; echo $?
Decompressed [file-corrupt.br]: 34 B -> 50 B in 0.00 sec
0
$ hd file; hd file-corrupt
00000000  74 68 69 73 20 69 73 20  74 65 73 74 20 66 69 6c  |this is test fil|
00000010  65 20 66 6f 72 20 63 68  65 63 6b 20 63 6f 6d 70  |e for check comp|
00000020  72 65 73 73 69 6f 6e 20  63 68 65 63 6b 73 75 6d  |ression checksum|
00000030  73 0a                                             |s.|
00000032
00000000  74 68 69 73 20 69 73 20  0a 65 73 74 20 66 69 6c  |this is .est fil|
00000010  65 20 66 6f 72 20 63 68  65 63 6b 20 63 6f 6d 70  |e for check comp|
00000020  72 65 73 73 69 6f 6e 20  63 68 65 63 6b 73 75 6d  |ression checksum|
00000030  73 0a                                             |s.|
00000032

Originally created by @eshkinkot on GitHub (Aug 30, 2021). ``` git clone https://github.com/google/brotli.git cd brotli/ make echo 'this is test file for check compression checksums' > file bin/brotli file cp file.br file-corrupt.br echo -ne '\xA6' | dd bs=1 conv=notrunc seek=$((0x18)) of=file-corrupt.br ``` ``` $ hd file.br; hd file-corrupt.br 00000000 a1 88 01 00 2f b0 01 e7 0b ed 32 4e 7d 0f 52 88 |..../.....2N}.R.| 00000010 2a c0 68 10 2a 86 63 e7 b6 30 5f 4b 50 3e b3 40 |*.h.*.c..0_KP>.@| 00000020 bc 08 |..| 00000022 00000000 a1 88 01 00 2f b0 01 e7 0b ed 32 4e 7d 0f 52 88 |..../.....2N}.R.| 00000010 2a c0 68 10 2a 86 63 e7 a6 30 5f 4b 50 3e b3 40 |*.h.*.c..0_KP>.@| 00000020 bc 08 |..| 00000022 $ bin/brotli -v -t file-corrupt.br; echo $? Decompressed [file-corrupt.br]: 34 B -> 50 B in 0.00 sec 0 $ bin/brotli -v -d file-corrupt.br; echo $? Decompressed [file-corrupt.br]: 34 B -> 50 B in 0.00 sec 0 $ hd file; hd file-corrupt 00000000 74 68 69 73 20 69 73 20 74 65 73 74 20 66 69 6c |this is test fil| 00000010 65 20 66 6f 72 20 63 68 65 63 6b 20 63 6f 6d 70 |e for check comp| 00000020 72 65 73 73 69 6f 6e 20 63 68 65 63 6b 73 75 6d |ression checksum| 00000030 73 0a |s.| 00000032 00000000 74 68 69 73 20 69 73 20 0a 65 73 74 20 66 69 6c |this is .est fil| 00000010 65 20 66 6f 72 20 63 68 65 63 6b 20 63 6f 6d 70 |e for check comp| 00000020 72 65 73 73 69 6f 6e 20 63 68 65 63 6b 73 75 6d |ression checksum| 00000030 73 0a |s.| 00000032 ``` ![vbindiff file file-corrupt](https://user-images.githubusercontent.com/69589/131365671-1abb7e40-53da-442a-b4c2-50902cc8da6c.png)

claunia closed this issue

2026-01-29 20:43:13 +00:00

claunia commented

2026-01-29 20:43:14 +00:00

@eustas commented on GitHub (Aug 31, 2021):

That is correct. Brotli itself is a stream format. It lacks features like CRC, multiple entries, uncompressed size info, rewind, etc.
Currently one could add custom "framing" to address that. Later Brotli Framing could be an out-of-the-box solution

@eustas commented on GitHub (Aug 31, 2021): That is correct. Brotli itself is a stream format. It lacks features like CRC, multiple entries, uncompressed size info, rewind, etc. Currently one could add custom "framing" to address that. Later [Brotli Framing](https://datatracker.ietf.org/doc/html/draft-vandevenne-shared-brotli-format-04) could be an out-of-the-box solution

claunia commented

2026-01-29 20:43:14 +00:00

@eustas commented on GitHub (Aug 31, 2021):

Alternatively one could also use Rigeli as framing format.

@eustas commented on GitHub (Aug 31, 2021): Alternatively one could also use [Rigeli](https://github.com/google/riegeli) as framing format.

claunia commented

2026-01-29 20:43:14 +00:00

@eshkinkot commented on GitHub (Sep 2, 2021):

I think that the main problem here is that this code is already used in various projects without any changes. For example, in wal-g, and archives created by wal-g with brotli compression cannot be checked for damage now.

Perhaps it is worth mentioning explicitly in the documentation/README about this feature that this is more of an example code and for use in production it is better to wrap it in some kind of container with CRC.

@eshkinkot commented on GitHub (Sep 2, 2021): I think that the main problem here is that this code is already used in various projects without any changes. For example, in [wal-g](https://github.com/wal-g/wal-g), and archives created by wal-g with brotli compression cannot be checked for damage now. Perhaps it is worth mentioning explicitly in the documentation/README about this feature that this is more of an example code and for use in production it is better to wrap it in some kind of container with CRC.

claunia commented

2026-01-29 20:43:15 +00:00

@eustas commented on GitHub (Sep 8, 2021):

Sure, will mention in README.
However, the scope of possible damage is very small: only uncompressible data is dumped as is. For entropy coded data (compressible) the probability that corrupted data is not noticed is very small. I could prepare a script to calculate it.

@eustas commented on GitHub (Sep 8, 2021): Sure, will mention in README. However, the scope of possible damage is very small: only uncompressible data is dumped as is. For entropy coded data (compressible) the probability that corrupted data is not noticed is very small. I could prepare a script to calculate it.

claunia commented

2026-01-29 20:43:15 +00:00

@eustas commented on GitHub (Jan 3, 2023):

Resolved in a8f581

@eustas commented on GitHub (Jan 3, 2023): Resolved in a8f581

claunia commented

2026-01-29 20:43:15 +00:00

@Artoria2e5 commented on GitHub (Apr 13, 2023):

Brotli running without framing is hurting its chances at replacing gzip in the tar.gz usecase. At least this is the story on the bsdtar (libarchive) side https://github.com/libarchive/libarchive/issues/1238#issuecomment-529746683: it does a very Unix approach to format detection using no file name but just contents, but brotli has no magic number for it to work with. The attempted gnu tar patch also seems to have hit that, but gnu tar looks at names so that's not the main issue.

The framing thing seems to be advancing? Let's wait then. Although I would love to see it available for toying in the command-line thing...

Oops, should've posted to the other issue about containers.

@Artoria2e5 commented on GitHub (Apr 13, 2023): Brotli running without framing is hurting its chances at replacing gzip in the tar.gz usecase. At least this is the story on the bsdtar (libarchive) side https://github.com/libarchive/libarchive/issues/1238#issuecomment-529746683: it does a very Unix approach to format detection using no file name but just contents, but brotli has no magic number for it to work with. The attempted [gnu tar patch](https://lists.gnu.org/archive/html/bug-tar/2017-10/msg00003.html) also seems to have hit that, but gnu tar looks at names so that's not the main issue. * * * The framing thing seems to be advancing? Let's wait then. Although I would love to see it available for toying in the command-line thing... * * * Oops, should've posted to the other issue about containers.

claunia referenced this issue

2026-01-29 20:52:02 +00:00

[PR #394] [MERGED] Update decoder API #998

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#394