Big Files - How Compress and Decompress #501

New Issue

claunia · 2026-01-29T20:44:54Z

claunia commented

2026-01-29 20:44:54 +00:00

Originally created by @FernandoBanhos on GitHub (Mar 23, 2024).

Is there any way to read a large file, and compress it in parts, through stream reading, like small buffers?
There is the BrotliEncoderCompressStream BrotliDecoderDecompressStream function, as far as I understand, I have to capture the entire contents of the file and instantiate it in memory to be able to compress or decompress it. So that at the end you can get the result and save it in another large file

Thanks!

Originally created by @FernandoBanhos on GitHub (Mar 23, 2024). Is there any way to read a large file, and compress it in parts, through stream reading, like small buffers? There is the BrotliEncoderCompressStream BrotliDecoderDecompressStream function, as far as I understand, I have to capture the entire contents of the file and instantiate it in memory to be able to compress or decompress it. So that at the end you can get the result and save it in another large file Thanks!

claunia closed this issue

2026-01-29 20:44:54 +00:00

claunia commented

2026-01-29 20:44:54 +00:00

@eustas commented on GitHub (Mar 25, 2024):

Hi. There is API for compressing file parts. Please take a look at BROTLI_PARAM_STREAM_OFFSET documentation.

@eustas commented on GitHub (Mar 25, 2024): Hi. There is API for compressing file parts. Please take a look at `BROTLI_PARAM_STREAM_OFFSET` documentation.

claunia commented

2026-01-29 20:44:55 +00:00

@FernandoBanhos commented on GitHub (Mar 25, 2024):

Hi. There is API for compressing file parts. Please take a look at BROTLI_PARAM_STREAM_OFFSET documentation.

I am following the documentation available at https://www.brotli.org/encode.html

@FernandoBanhos commented on GitHub (Mar 25, 2024): > Hi. There is API for compressing file parts. Please take a look at `BROTLI_PARAM_STREAM_OFFSET` documentation. I am following the documentation available at https://www.brotli.org/encode.html

claunia commented

2026-01-29 20:44:55 +00:00

@FernandoBanhos commented on GitHub (Mar 25, 2024):

see my code

function TBrotliCompress.InternalCompress(AStream: TStream): TStream;
var
  vState : Pointer;
  vInput, vOutput : Pointer;

  vFileBufferSize : Integer;
  vAvailableIn, vAvailableOut, vIn : NativeInt;
  vOperation : integer;
begin
  Result := TMemoryStream.Create;
  Result.Size := AStream.Size;

  AStream.Position := 0;
  Result.Position := 0;

  vState := BrotliEncoderCreateInstance(nil, nil, nil);
  try
    vFileBufferSize := 65536;

    if BrotliEncoderSetParameter(vState, BROTLI_PARAM_QUALITY, 5) = 0 then
      Exit;
    if BrotliEncoderSetParameter(vState, BROTLI_PARAM_LGWIN, 22) = 0 then
      Exit
    if BrotliEncoderSetParameter(vState, BROTLI_PARAM_STREAM_OFFSET, 65536) = 0 then
      Exit;

    vAvailableOut := 0;
    GetMem(vInput, vFileBufferSize);

    while (True) do
    begin
      vAvailableIn := AStream.Read(vInput^, vFileBufferSize);
      vIn := vAvailableIn;

      vAvailableOut := vAvailableOut + vFileBufferSize;
      GetMem(vOutput, vAvailableOut);

      if AStream.Position >= AStream.Size then
        vOperation := 2  //BROTLI_OPERATION_FINISH
      else
        vOperation := 0; //BROTLI_OPERATION_PROCESS;

      if (BrotliEncoderCompressStream(vState, vOperation, vAvailableIn, vInput,
                                      vAvailableOut, vOutput, nil)) = BROTLI_TRUE then
      begin
        if vAvailableIn = 0 then
          Result.Write(vOutput^, vIn - vAvailableOut);
      end;

      Freemem(vOutput);

      if BrotliEncoderIsFinished(vState) = BROTLI_TRUE then
        Break;
    end;
  finally
    BrotliEncoderDestroyInstance(vState);
  end;
end;

@FernandoBanhos commented on GitHub (Mar 25, 2024): see my code ``` function TBrotliCompress.InternalCompress(AStream: TStream): TStream; var vState : Pointer; vInput, vOutput : Pointer; vFileBufferSize : Integer; vAvailableIn, vAvailableOut, vIn : NativeInt; vOperation : integer; begin Result := TMemoryStream.Create; Result.Size := AStream.Size; AStream.Position := 0; Result.Position := 0; vState := BrotliEncoderCreateInstance(nil, nil, nil); try vFileBufferSize := 65536; if BrotliEncoderSetParameter(vState, BROTLI_PARAM_QUALITY, 5) = 0 then Exit; if BrotliEncoderSetParameter(vState, BROTLI_PARAM_LGWIN, 22) = 0 then Exit if BrotliEncoderSetParameter(vState, BROTLI_PARAM_STREAM_OFFSET, 65536) = 0 then Exit; vAvailableOut := 0; GetMem(vInput, vFileBufferSize); while (True) do begin vAvailableIn := AStream.Read(vInput^, vFileBufferSize); vIn := vAvailableIn; vAvailableOut := vAvailableOut + vFileBufferSize; GetMem(vOutput, vAvailableOut); if AStream.Position >= AStream.Size then vOperation := 2 //BROTLI_OPERATION_FINISH else vOperation := 0; //BROTLI_OPERATION_PROCESS; if (BrotliEncoderCompressStream(vState, vOperation, vAvailableIn, vInput, vAvailableOut, vOutput, nil)) = BROTLI_TRUE then begin if vAvailableIn = 0 then Result.Write(vOutput^, vIn - vAvailableOut); end; Freemem(vOutput); if BrotliEncoderIsFinished(vState) = BROTLI_TRUE then Break; end; finally BrotliEncoderDestroyInstance(vState); end; end; ```

claunia commented

2026-01-29 20:44:55 +00:00

@eustas commented on GitHub (Mar 26, 2024):

Haven't seen Pascal for 20 years =)

Now I can see that documentation on brotli.org is outdated (it is for 1.0.0); will update it soon.

@eustas commented on GitHub (Mar 26, 2024): Haven't seen Pascal for 20 years =) Now I can see that documentation on brotli.org is outdated (it is for 1.0.0); will update it soon.

claunia commented

2026-01-29 20:44:55 +00:00

@FernandoBanhos commented on GitHub (Mar 26, 2024):

Haven't seen Pascal for 20 years =)

Now I can see that documentation on brotli.org is outdated (it is for 1.0.0); will update it soon.

Do you have updated documentation?
I'm developing the Brotli library in Pascal, to make it available in a github repository

thanks!

@FernandoBanhos commented on GitHub (Mar 26, 2024): > Haven't seen Pascal for 20 years =) > > Now I can see that documentation on brotli.org is outdated (it is for 1.0.0); will update it soon. Do you have updated documentation? I'm developing the Brotli library in Pascal, to make it available in a github repository thanks!

claunia commented

2026-01-29 20:44:55 +00:00

@eustas commented on GitHub (Apr 5, 2024):

New documentation is live: https://brotli.org/encode.html#a9a8
Going to setup read-the-doc, so that fresh documentation will be always available.

@eustas commented on GitHub (Apr 5, 2024): New documentation is live: https://brotli.org/encode.html#a9a8 Going to setup read-the-doc, so that fresh documentation will be always available.

claunia commented

2026-01-29 20:44:56 +00:00

@FernandoBanhos commented on GitHub (Apr 8, 2024):

A nova documentação está disponível: https://brotli.org/encode.html#a9a8 Indo para a configuração do read-the-doc, para que a documentação atualizada esteja sempre disponível.

thanks for the help,
my Brotli library repository, has been completed: https://github.com/FernandoBanhos/pascal_brotli

@FernandoBanhos commented on GitHub (Apr 8, 2024): > A nova documentação está disponível: https://brotli.org/encode.html#a9a8 Indo para a configuração do read-the-doc, para que a documentação atualizada esteja sempre disponível. thanks for the help, my Brotli library repository, has been completed: https://github.com/FernandoBanhos/pascal_brotli

claunia referenced this issue

2026-01-29 20:45:15 +00:00

Support for free-threaded Python #530

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#501