Add support for being able to detect a brotli-compressed file? #361

New Issue

claunia · 2026-01-29T20:42:42Z

claunia commented

2026-01-29 20:42:42 +00:00

Originally created by @juj on GitHub (Nov 10, 2020).

Identifying a gzip-compressed file is super-easy by checking whether the file starts with magic bytes 1F and 8B. (ignoring false positives etc.)

It has been discussed before that Brotli is a stream that does not have such magic bytes. That has led to other projects not being able to move forward, such as file and libmagic tools not being able to detect Brotli compressed content (#727).

There was a proposal for a Brotli framing format at https://github.com/madler/brotli/blob/master/br-format-v3.txt , but if I understand its latest status correctly, it was just a proposal, with no actual plan or possibility to move forward, as it would break binary compatibility with existing Brotli compressed content? (so e.g. Chrome, Firefox and Safari would need to add support for this new framed format?) Is that accurate?

It seems based on #462 and #298 that the request to add a magic number or an official framing was dropped, with no plans to actually go forward with either?

If so, and adding magic numbers and/or official framing is not on the table anymore, I would like to propose an alternative feature that would at least work for some users' use cases, and retain the existing binary compatibility.

Unity is using a mechanism that enables embedding an uncompressed comment string into the Brotli binary. See the local code modification here: 5a6d5d9c7f

That enables a brotli compressor to receive a string --comment "UnityWeb Compressed Content (brotli)" on the command line, which gets embedded to the stream. Unity is then using the following JavaScript code in a browser context to detect .br compressed files:

hasUnityMarker: function (data) {
  var expectedComment = "UnityWeb Compressed Content (brotli)";
  if (!data.length)
    return false;
  var WBITS_length = (data[0] & 0x01) ? (data[0] & 0x0E) ? 4 : 7 : 1,
      WBITS = data[0] & ((1 << WBITS_length) - 1),
      MSKIPBYTES = 1 + ((Math.log(expectedComment.length - 1) / Math.log(2)) >> 3);
      commentOffset = (WBITS_length + 1 + 2 + 1 + 2 + (MSKIPBYTES << 3) + 7) >> 3;
  if (WBITS == 0x11 || commentOffset > data.length)
    return false;
  var expectedCommentPrefix = WBITS + (((3 << 1) + (MSKIPBYTES << 4) + ((expectedComment.length - 1) << 6)) << WBITS_length);
  for (var i = 0; i < commentOffset; i++, expectedCommentPrefix >>>= 8) {
    if (data[i] != (expectedCommentPrefix & 0xFF))
      return false;
  }
  return String.fromCharCode.apply(null, data.subarray(commentOffset, commentOffset + expectedComment.length)) == expectedComment;
},

Would it be possible for this to become an officially supported feature in the command line executable brotli compressor? E.g. if one runs

brotli -9k --comment "this is brotli compressed" file.txt -o output.br

that would emit the given comment into the output file. Then users would be able to identify at least their own generated .br files - or if a compressor is used within a certain community (say among Emscripten developers), tooling could be used within that scope to develop detectable .br files?

That way people who oppose adding magic numbers or framing would not have their .br file size increase, but people would still be able to identify their generated .br files.

Originally created by @juj on GitHub (Nov 10, 2020). Identifying a gzip-compressed file is super-easy by checking whether the file starts with magic bytes `1F` and `8B`. (ignoring false positives etc.) It has been discussed before that Brotli is a stream that does not have such magic bytes. That has led to other projects not being able to move forward, such as `file` and `libmagic` tools not being able to detect Brotli compressed content (#727). There was a proposal for a Brotli framing format at https://github.com/madler/brotli/blob/master/br-format-v3.txt , but if I understand its latest status correctly, it was just a proposal, with no actual plan or possibility to move forward, as it would break binary compatibility with existing Brotli compressed content? (so e.g. Chrome, Firefox and Safari would need to add support for this new framed format?) Is that accurate? It seems based on #462 and #298 that the request to add a magic number or an official framing was dropped, with no plans to actually go forward with either? If so, and adding magic numbers and/or official framing is not on the table anymore, I would like to propose an alternative feature that would at least work for some users' use cases, and retain the existing binary compatibility. Unity is using a mechanism that enables embedding an uncompressed comment string into the Brotli binary. See the local code modification here: https://github.com/Unity-Technologies/brotli/commit/5a6d5d9c7f3f813280900cabcaabcbd0d51d5bbc That enables a `brotli` compressor to receive a string `--comment "UnityWeb Compressed Content (brotli)"` on the command line, which gets embedded to the stream. Unity is then using the following JavaScript code in a browser context to detect .br compressed files: ```js hasUnityMarker: function (data) { var expectedComment = "UnityWeb Compressed Content (brotli)"; if (!data.length) return false; var WBITS_length = (data[0] & 0x01) ? (data[0] & 0x0E) ? 4 : 7 : 1, WBITS = data[0] & ((1 << WBITS_length) - 1), MSKIPBYTES = 1 + ((Math.log(expectedComment.length - 1) / Math.log(2)) >> 3); commentOffset = (WBITS_length + 1 + 2 + 1 + 2 + (MSKIPBYTES << 3) + 7) >> 3; if (WBITS == 0x11 || commentOffset > data.length) return false; var expectedCommentPrefix = WBITS + (((3 << 1) + (MSKIPBYTES << 4) + ((expectedComment.length - 1) << 6)) << WBITS_length); for (var i = 0; i < commentOffset; i++, expectedCommentPrefix >>>= 8) { if (data[i] != (expectedCommentPrefix & 0xFF)) return false; } return String.fromCharCode.apply(null, data.subarray(commentOffset, commentOffset + expectedComment.length)) == expectedComment; }, ``` Would it be possible for this to become an officially supported feature in the command line executable brotli compressor? E.g. if one runs ``` brotli -9k --comment "this is brotli compressed" file.txt -o output.br ``` that would emit the given comment into the output file. Then users would be able to identify at least their own generated .br files - or if a compressor is used within a certain community (say among Emscripten developers), tooling could be used within that scope to develop detectable .br files? That way people who oppose adding magic numbers or framing would not have their .br file size increase, but people would still be able to identify their generated .br files.

claunia added the feature release-v1.1.1 labels 2026-01-29 20:42:42 +00:00

claunia closed this issue

2026-01-29 20:42:42 +00:00

claunia commented

2026-01-29 20:42:42 +00:00

@polarathene commented on GitHub (May 5, 2021):

AFAIK Dropbox has a port rust-brotli that added a magic number feature in Sep 2018.

It's meant to be compatible as a C drop-in replacement of this project AFAIK, but I'm not sure if it's been keeping in sync, so it might have diverged a bit. It has a few other additions too.

@polarathene commented on GitHub (May 5, 2021): AFAIK Dropbox has a port [`rust-brotli`](https://github.com/dropbox/rust-brotli) that [added a magic number feature in Sep 2018](https://github.com/dropbox/rust-brotli/commit/001c7a50d9be5cc16be86913a5661d782a74bdaa). It's meant to be compatible as a C drop-in replacement of this project AFAIK, but I'm not sure if it's been keeping in sync, so it might have diverged a bit. It has a few other additions too.

claunia commented

2026-01-29 20:42:42 +00:00

@eustas commented on GitHub (Jan 4, 2023):

Nice idea. Going to add comment feature soon.

@eustas commented on GitHub (Jan 4, 2023): Nice idea. Going to add comment feature soon.

claunia commented

2026-01-29 20:42:43 +00:00

@Artoria2e5 commented on GitHub (Apr 16, 2023):

Comment (or any variable-length part really) is also useful for HTB, the mitigation used to add randomness to packet sizes and mess with the BREACH exploit.

@Artoria2e5 commented on GitHub (Apr 16, 2023): Comment (or any variable-length part really) is also useful for [HTB](https://ieeexplore.ieee.org/document/9754554), the mitigation used to add randomness to packet sizes and mess with the BREACH exploit.

claunia commented

2026-01-29 20:42:43 +00:00

@juj commented on GitHub (May 10, 2023):

Hey @eustas , I wonder if that comment feature got added? Your post above suggests that it would have been something that was favorable to carry upstream?

@juj commented on GitHub (May 10, 2023): Hey @eustas , I wonder if that comment feature got added? Your post above suggests that it would have been something that was favorable to carry upstream?

claunia commented

2026-01-29 20:42:43 +00:00

@eustas commented on GitHub (Jan 11, 2024):

Done. See --comment= / -C CLI options.

@eustas commented on GitHub (Jan 11, 2024): Done. See `--comment=` / `-C` CLI options.

claunia referenced this issue

2026-01-29 20:51:50 +00:00

[PR #361] [MERGED] Step 2: update comments #957

claunia referenced this issue

2026-01-29 20:51:51 +00:00

[PR #361] Step 2: update comments #961

claunia referenced this issue