starred/sharpcompress
1
0
Fork 0
mirror of https://github.com/adamhathcock/sharpcompress.git synced 2026-02-03 21:23:38 +00:00
Code Issues 975 Packages Projects Releases 20 Wiki Activity

Decompression via ZlibStream for large message doesn't work #477

New Issue
Closed
opened 2026-01-29 22:12:40 +00:00 by claunia · 12 comments
claunia commented 2026-01-29 22:12:40 +00:00
Owner
Copy Link

Originally created by @DmitryLukyanov on GitHub (Sep 20, 2021).

The below test triggers the following exception in Decompress step for a message bigger than 131072 letters on latest nuget versions. Messages with length less or equal to 131072 letters are decompressed fine. I've checked '0.29.0', '0.28.0', '0.27.0':

Message: 
  SharpCompress.Compressors.Deflate.ZlibException : Bad state (incorrect data check)

Stack Trace: 
  InflateManager.Inflate(FlushType flush)
  ZlibBaseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
  Stream.CopyTo(Stream destination, Int32 bufferSize)
  CompressorsTests.Decompress(Stream input, Stream output) line 135
  CompressorsTests.Zlib_should_read_the_previously_written_message() line 104

but it works fine for versions <= 0.26.0

    [Fact]
    public void Zlib_should_read_the_previously_written_message()
    {
        var message = new string('a', 131073);  // 131073 causes the failure, but 131072 (-1) doesn't
        var bytes = Encoding.ASCII.GetBytes(message);

        using (var inputStream = new MemoryStream(bytes))
        {
            using (var compressedStream = new MemoryStream())
            using (var byteBufferStream = new BufferedStream(inputStream)) // System.IO
            {
                Compress(byteBufferStream, compressedStream, compressionLevel: 1);
                compressedStream.Position = 0;

                using (var decompressedStream = new MemoryStream())
                {
                    Decompress(compressedStream, decompressedStream);

                    byteBufferStream.Position = 0;
                    var result = Encoding.ASCII.GetString(GetBytes(byteBufferStream));
                    result.Should().Be(message);
                }
            }
        }
    }

    public void Compress(Stream input, Stream output, int compressionLevel)
    {
        using (var zlibStream = new ZlibStream(new NonDisposingStream(output), CompressionMode.Compress, (CompressionLevel)compressionLevel))
        {
            zlibStream.FlushMode = FlushType.Sync;
            input.CopyTo(zlibStream);
        }
    }

    public void Decompress(Stream input, Stream output)
    {
        using (var zlibStream = new ZlibStream(new NonDisposingStream(input), CompressionMode.Decompress))
        {
            zlibStream.CopyTo(output);
        }
    }

    byte[] GetBytes(BufferedStream stream)
    {
        byte[] bytes = new byte[stream.Length];
        stream.Read(bytes, 0, (int)stream.Length);
        return bytes;
    }
Originally created by @DmitryLukyanov on GitHub (Sep 20, 2021). The below test triggers the following exception in `Decompress` step for a message bigger than 131072 letters on latest nuget versions. Messages with length less or equal to 131072 letters are decompressed fine. I've checked '0.29.0', '0.28.0', '0.27.0': Message: SharpCompress.Compressors.Deflate.ZlibException : Bad state (incorrect data check) Stack Trace: InflateManager.Inflate(FlushType flush) ZlibBaseStream.Read(Byte[] buffer, Int32 offset, Int32 count) Stream.CopyTo(Stream destination, Int32 bufferSize) CompressorsTests.Decompress(Stream input, Stream output) line 135 CompressorsTests.Zlib_should_read_the_previously_written_message() line 104 but it works fine for versions <= `0.26.0` [Fact] public void Zlib_should_read_the_previously_written_message() { var message = new string('a', 131073); // 131073 causes the failure, but 131072 (-1) doesn't var bytes = Encoding.ASCII.GetBytes(message); using (var inputStream = new MemoryStream(bytes)) { using (var compressedStream = new MemoryStream()) using (var byteBufferStream = new BufferedStream(inputStream)) // System.IO { Compress(byteBufferStream, compressedStream, compressionLevel: 1); compressedStream.Position = 0; using (var decompressedStream = new MemoryStream()) { Decompress(compressedStream, decompressedStream); byteBufferStream.Position = 0; var result = Encoding.ASCII.GetString(GetBytes(byteBufferStream)); result.Should().Be(message); } } } } public void Compress(Stream input, Stream output, int compressionLevel) { using (var zlibStream = new ZlibStream(new NonDisposingStream(output), CompressionMode.Compress, (CompressionLevel)compressionLevel)) { zlibStream.FlushMode = FlushType.Sync; input.CopyTo(zlibStream); } } public void Decompress(Stream input, Stream output) { using (var zlibStream = new ZlibStream(new NonDisposingStream(input), CompressionMode.Decompress)) { zlibStream.CopyTo(output); } } byte[] GetBytes(BufferedStream stream) { byte[] bytes = new byte[stream.Length]; stream.Read(bytes, 0, (int)stream.Length); return bytes; }
claunia added the bugup for grabs labels 2026-01-29 22:12:40 +00:00
claunia commented 2026-01-29 22:12:41 +00:00
Author
Owner
Copy Link

@SymbioticKilla commented on GitHub (Oct 11, 2021):

Is there any ETA? We use MongoDB Driver that uses this library and there is a vulnerability shown in the reports => it is PITA for the companies which have strict compliance rules. MongoDB can not update it because of this bug.
Thanks!

@SymbioticKilla commented on GitHub (Oct 11, 2021): Is there any ETA? We use MongoDB Driver that uses this library and there is a vulnerability shown in the reports => it is PITA for the companies which have strict compliance rules. MongoDB can not update it because of this bug. Thanks!
claunia commented 2026-01-29 22:12:41 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Oct 11, 2021):

The "vulnerability" is when the file pathing code is being used. Would the driver be using that?

I don't have an ETA on this because I'm not actually working on it and there is no PR to merge.

At a guess, iit must be something in the 0.27 release that is causing it: https://github.com/adamhathcock/sharpcompress/releases/tag/0.27

I'd put that test in and start reverting/testing the changes from the 0.27 and see what happens.

@adamhathcock commented on GitHub (Oct 11, 2021): The "vulnerability" is when the file pathing code is being used. Would the driver be using that? I don't have an ETA on this because I'm not actually working on it and there is no PR to merge. At a guess, iit must be something in the 0.27 release that is causing it: https://github.com/adamhathcock/sharpcompress/releases/tag/0.27 I'd put that test in and start reverting/testing the changes from the 0.27 and see what happens.
claunia commented 2026-01-29 22:12:41 +00:00
Author
Owner
Copy Link

@SymbioticKilla commented on GitHub (Nov 8, 2021):

Any update?

@SymbioticKilla commented on GitHub (Nov 8, 2021): Any update?
claunia commented 2026-01-29 22:12:41 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Nov 9, 2021):

Sorry no. I need help

@adamhathcock commented on GitHub (Nov 9, 2021): Sorry no. I need help
claunia commented 2026-01-29 22:12:42 +00:00
Author
Owner
Copy Link

@DmitryLukyanov commented on GitHub (Nov 22, 2021):

First broken commit.
The solution (at least to fix the above test) is reverting this line (and related nested class). Hopefully this will help.

@DmitryLukyanov commented on GitHub (Nov 22, 2021): First broken [commit](https://github.com/adamhathcock/sharpcompress/commit/58799990948ab66979d759bebf31697a47773254). The solution (at least to fix the above test) is reverting this [line](https://github.com/adamhathcock/sharpcompress/commit/58799990948ab66979d759bebf31697a47773254#diff-8f4178f88d6ead96112102c2843a01cd5e609a6beae3debf67796e074de896acR744) (and related nested class). Hopefully this will help.
claunia commented 2026-01-29 22:12:42 +00:00
Author
Owner
Copy Link

@AraHaan commented on GitHub (Nov 22, 2021):

I actually started with my own zlib by actually using the native version for compression using C# 10 features.

This pull request which I plan to merge soon that replaces the stream with a public static class that uses spans.

@AraHaan commented on GitHub (Nov 22, 2021): I actually started with my own zlib by actually using the native version for compression using C# 10 features. [This](https://github.com/Elskom/ZlibSharp/pull/4) pull request which I plan to merge soon that replaces the stream with a public static class that uses spans.
claunia commented 2026-01-29 22:12:42 +00:00
Author
Owner
Copy Link

@DmitryLukyanov commented on GitHub (Nov 22, 2021):

In more details, the above test works without this condition that repeatedly returns 1(SeedValue) instead of repeatedly 2957970271

@DmitryLukyanov commented on GitHub (Nov 22, 2021): In more details, the above test works without this [condition](https://github.com/adamhathcock/sharpcompress/commit/58799990948ab66979d759bebf31697a47773254#diff-a846eb21e279f48345add99e63d71e21c0ec5cd3d22798b761e6a2e19353ca76R54-R57) that repeatedly returns `1`(SeedValue) instead of repeatedly `2957970271`
claunia commented 2026-01-29 22:12:42 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Nov 22, 2021):

I've made a PR with the test and identified fix: https://github.com/adamhathcock/sharpcompress/pull/624

Eyeball it and I'll merge it and push to nuget

@adamhathcock commented on GitHub (Nov 22, 2021): I've made a PR with the test and identified fix: https://github.com/adamhathcock/sharpcompress/pull/624 Eyeball it and I'll merge it and push to nuget
claunia commented 2026-01-29 22:12:43 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Nov 22, 2021):

I actually started with my own zlib by actually using the native version for compression using C# 10 features.

This pull request which I plan to merge soon that replaces the stream with a public static class that uses spans.

If you're interested in porting your zlib into here and possibly helping reworking other algorithms to be Span/Async, let me know. I started a branch months ago but time got away from me again. My blocker was my lack of knowledge of the algorithms so doing span/async on them broke and I could not figure why

@adamhathcock commented on GitHub (Nov 22, 2021): > I actually started with my own zlib by actually using the native version for compression using C# 10 features. > > [This](https://github.com/Elskom/ZlibSharp/pull/4) pull request which I plan to merge soon that replaces the stream with a public static class that uses spans. If you're interested in porting your zlib into here and possibly helping reworking other algorithms to be Span/Async, let me know. I started a branch months ago but time got away from me again. My blocker was my lack of knowledge of the algorithms so doing span/async on them broke and I could not figure why
claunia commented 2026-01-29 22:12:43 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Nov 22, 2021):

Should be on nuget soon https://www.nuget.org/packages/SharpCompress/0.30.1

@adamhathcock commented on GitHub (Nov 22, 2021): Should be on nuget soon https://www.nuget.org/packages/SharpCompress/0.30.1
claunia commented 2026-01-29 22:12:43 +00:00
Author
Owner
Copy Link

@AraHaan commented on GitHub (Nov 22, 2021):

I actually started with my own zlib by actually using the native version for compression using C# 10 features.
This pull request which I plan to merge soon that replaces the stream with a public static class that uses spans.

If you're interested in porting your zlib into here and possibly helping reworking other algorithms to be Span/Async, let me know. I started a branch months ago but time got away from me again. My blocker was my lack of knowledge of the algorithms so doing span/async on them broke and I could not figure why

Also there is another issue as well too on providing a zlib library file for the following RIDs:

  • win-x86
  • win-x64
  • win-arm
  • win-arm64
  • linux-x86
  • linux-x64
  • linux-arm
  • linux-arm64
  • osx-x86
  • osx-arm64
  • mobile like android or iOS or iPadOS or watchos?

Currently on linux it uses the system provided zlib (which I do not know if it's 1.2.11 for all linux installs and I do hardcode it to version 1.2.11 with a way to tell it to use a different version), For MacOS there is a system provided zlib but I do not know where to locate it so I expect them to install it with homebrew currently. As for Windows, I have no clue what to do and currently expect them to basically compile zlib themselves.

The only thing I can think of is to somehow on whatever OS is used to compile to somehow cross compile zlib to all of those RIDs and then package them as native dependencies on the nuget package.

Alternatively I could look into if I can change my code slightly to use the built in clrcompression library and then not worry about zlib no more but it involves looking at the code to that library that is built into the .NET default runtime.

@AraHaan commented on GitHub (Nov 22, 2021): > > I actually started with my own zlib by actually using the native version for compression using C# 10 features. > > [This](https://github.com/Elskom/ZlibSharp/pull/4) pull request which I plan to merge soon that replaces the stream with a public static class that uses spans. > > If you're interested in porting your zlib into here and possibly helping reworking other algorithms to be Span/Async, let me know. I started a branch months ago but time got away from me again. My blocker was my lack of knowledge of the algorithms so doing span/async on them broke and I could not figure why Also there is another issue as well too on providing a zlib library file for the following RIDs: - win-x86 - win-x64 - win-arm - win-arm64 - linux-x86 - linux-x64 - linux-arm - linux-arm64 - osx-x86 - osx-arm64 - mobile like android or iOS or iPadOS or watchos? Currently on linux it uses the system provided zlib (which I do not know if it's 1.2.11 for all linux installs and I do hardcode it to version 1.2.11 with a way to tell it to use a different version), For MacOS there is a system provided zlib but I do not know where to locate it so I expect them to install it with homebrew currently. As for Windows, I have no clue what to do and currently expect them to basically compile zlib themselves. The only thing I can think of is to somehow on whatever OS is used to compile to somehow cross compile zlib to all of those RIDs and then package them as native dependencies on the nuget package. Alternatively I could look into if I can change my code slightly to use the built in clrcompression library and then not worry about zlib no more but it involves looking at the code to that library that is built into the .NET default runtime.
claunia commented 2026-01-29 22:12:44 +00:00
Author
Owner
Copy Link

@DmitryLukyanov commented on GitHub (Nov 22, 2021):

it works for us, closing this issue then

@DmitryLukyanov commented on GitHub (Nov 22, 2021): it works for us, closing this issue then
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
feedback requested
not bug
pull-request
Mirrored from GitHub Pull Request
 question
up for grabs
wontfix
No Label bug up for grabs
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/sharpcompress#477
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?