starred/sharpcompress
1
0
Fork 0
mirror of https://github.com/adamhathcock/sharpcompress.git synced 2026-02-03 21:23:38 +00:00
Code Issues 975 Packages Projects Releases 20 Wiki Activity

[PR #1052] Move ZstdSharp into SharpCompress - Complete Integration #1476

New Issue
Closed
opened 2026-01-29 22:20:45 +00:00 by claunia · 0 comments
claunia commented 2026-01-29 22:20:45 +00:00
Owner
Copy Link

Original Pull Request: https://github.com/adamhathcock/sharpcompress/pull/1052

State: closed
Merged: Yes

Completes the full integration of ZstdSharp directly into SharpCompress, removing the external ZstdSharp.Port dependency (addresses #1030, started in #949).

Overview

This PR integrates all ~217 ZstdSharp implementation files into SharpCompress, providing a fully self-contained ZStandard compression/decompression implementation without external dependencies.

Changes Made

Core Infrastructure

  • CLSCompliant(false): Required for unsafe pointer usage in ZStandard internals
  • Error Handling: ZSTD_ErrorCode enum, ZstdException class, ThrowHelper with extension methods
  • Memory Management: UnsafeHelper with malloc, calloc, free, memcpy, memset, memmove, memcmp
  • Buffer Types: ZSTD_inBuffer_s, ZSTD_outBuffer_s, ZSTD_customMem
  • Parameter Types: Compression (ZSTD_cParameter, ZSTD_compressionParameters, ZSTD_strategy) and decompression (ZSTD_dParameter) parameters
  • Frame Types: ZSTD_frameType_e, ZSTD_frameHeader, ZSTD_frameParameters, ZSTD_parameters
  • Dictionary Types: ZSTD_dictContentType_e, ZSTD_dictLoadMethod_e
  • Directives: ZSTD_EndDirective, ZSTD_ResetDirective

Complete Unsafe Implementation (~217 files)

  • Methods Partial Class: ErrorPrivate, Mem (I/O API), Bits, Compiler, Allocations, ZstdCommon
  • FSE (Finite State Entropy): Compression and decompression algorithms (Fse.cs, FseCompress.cs, FseDecompress.cs)
  • Huffman Encoding: Compression and decompression (Huf.cs, HufCompress.cs, HufDecompress.cs)
  • Dictionary Training: COVER and FastCover algorithms (Cover.cs, Fastcover.cs, Zdict.cs)
  • Core Algorithms:
    • Compression: ZstdCompress.cs, ZstdCompressInternal.cs, ZstdCompressLiterals.cs, ZstdCompressSequences.cs, ZstdCompressSuperblock.cs
    • Decompression: ZstdDecompress.cs, ZstdDecompressBlock.cs, ZstdDecompressInternal.cs
    • Strategy implementations: ZstdFast.cs, ZstdDoubleFast.cs, ZstdLazy.cs, ZstdOpt.cs
    • Long-distance matching: ZstdLdm.cs
  • Entropy Coding: EntropyCommon.cs, Bitstream.cs
  • Context Management: ZSTD_CCtx_s.cs, ZSTD_DCtx_s.cs, compression/decompression context structures
  • Multithreading: ZstdmtCompress.cs, Pool.cs
  • XXHash: Xxhash.cs for checksums
  • Supporting Data Structures: 200+ struct and enum definitions

High-Level API

  • Decompressor: Public API for decompression with Unwrap() methods and dictionary support
  • Compressor: Public API for compression with Wrap() methods
  • DecompressionStream and CompressionStream: Stream-based APIs
  • SafeHandles: SafeCctxHandle and SafeDctxHandle for safe resource management
  • Supporting Classes: JobThreadPool, SynchronizationWrapper, UnmanagedObject, Constants, BitOperations

Dependency Removal

  • Removed ZstdSharp.Port package reference from SharpCompress.csproj
  • Updated all ZstdSharp usages to internal classes:
    • Common/Zip/ZipFilePart.cs: Changed namespace from ZstdSharp to SharpCompress.Compressors.ZStandard
    • Compressors/LZMA/Registry.cs: Updated namespace usage
    • Writers/Zip/ZipWriter.cs: Changed new ZstdSharp.CompressionStream(...) to new CompressionStream(...)

Bug Fixes and Improvements

  • Fixed obsolete API: Thread.VolatileRead → Volatile.Read in ZstdDecompressBlock.cs
  • Cleaned up unused using statements in ZstandardConstants.cs
  • Rebased onto latest master (394fd2e) for CSharpier 1.2.3 and other updates

Build Status

✅ Project builds successfully on all targets (net48, net8.0, net10.0)
✅ No warnings or errors
✅ No external ZstdSharp dependency required
✅ Code formatted with CSharpier 1.2.3

Files Changed

  • 224 files added: Complete ZstdSharp implementation
  • 3 files modified: Namespace updates to remove ZstdSharp.Port references
  • 1 file modified: Package reference removal from .csproj

SharpCompress now has a complete, self-contained ZStandard implementation.

Original prompt

This section details on the original issue you should resolve

<issue_title>Move ZstdSharp into SharpCompress</issue_title>
<issue_description>think the authors have little interest in this: https://github.com/oleg-st/ZstdSharp/issues/36

The biggest hurdle found so far is that there is a lot of unsafe/native usage that is probably incompatible with AOT. I know work was done to make everything CLSCompliant.

I'm ignoring big issues but might also rethink what targets there are. The legacy framework targets are slowing things down too.

Started here https://github.com/adamhathcock/sharpcompress/pull/949</issue_description>

<agent_instructions>Make a start but there are a lot of changes needed to the code then stop and saw progress</agent_instructions>

Comments on the Issue (you are @copilot in this section)

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

**Original Pull Request:** https://github.com/adamhathcock/sharpcompress/pull/1052 **State:** closed **Merged:** Yes --- Completes the full integration of ZstdSharp directly into SharpCompress, removing the external ZstdSharp.Port dependency (addresses #1030, started in #949). ## Overview This PR integrates all ~217 ZstdSharp implementation files into SharpCompress, providing a fully self-contained ZStandard compression/decompression implementation without external dependencies. ## Changes Made ### Core Infrastructure - **CLSCompliant(false)**: Required for unsafe pointer usage in ZStandard internals - **Error Handling**: `ZSTD_ErrorCode` enum, `ZstdException` class, `ThrowHelper` with extension methods - **Memory Management**: `UnsafeHelper` with malloc, calloc, free, memcpy, memset, memmove, memcmp - **Buffer Types**: `ZSTD_inBuffer_s`, `ZSTD_outBuffer_s`, `ZSTD_customMem` - **Parameter Types**: Compression (`ZSTD_cParameter`, `ZSTD_compressionParameters`, `ZSTD_strategy`) and decompression (`ZSTD_dParameter`) parameters - **Frame Types**: `ZSTD_frameType_e`, `ZSTD_frameHeader`, `ZSTD_frameParameters`, `ZSTD_parameters` - **Dictionary Types**: `ZSTD_dictContentType_e`, `ZSTD_dictLoadMethod_e` - **Directives**: `ZSTD_EndDirective`, `ZSTD_ResetDirective` ### Complete Unsafe Implementation (~217 files) - **Methods Partial Class**: ErrorPrivate, Mem (I/O API), Bits, Compiler, Allocations, ZstdCommon - **FSE (Finite State Entropy)**: Compression and decompression algorithms (`Fse.cs`, `FseCompress.cs`, `FseDecompress.cs`) - **Huffman Encoding**: Compression and decompression (`Huf.cs`, `HufCompress.cs`, `HufDecompress.cs`) - **Dictionary Training**: COVER and FastCover algorithms (`Cover.cs`, `Fastcover.cs`, `Zdict.cs`) - **Core Algorithms**: - Compression: `ZstdCompress.cs`, `ZstdCompressInternal.cs`, `ZstdCompressLiterals.cs`, `ZstdCompressSequences.cs`, `ZstdCompressSuperblock.cs` - Decompression: `ZstdDecompress.cs`, `ZstdDecompressBlock.cs`, `ZstdDecompressInternal.cs` - Strategy implementations: `ZstdFast.cs`, `ZstdDoubleFast.cs`, `ZstdLazy.cs`, `ZstdOpt.cs` - Long-distance matching: `ZstdLdm.cs` - **Entropy Coding**: `EntropyCommon.cs`, `Bitstream.cs` - **Context Management**: `ZSTD_CCtx_s.cs`, `ZSTD_DCtx_s.cs`, compression/decompression context structures - **Multithreading**: `ZstdmtCompress.cs`, `Pool.cs` - **XXHash**: `Xxhash.cs` for checksums - **Supporting Data Structures**: 200+ struct and enum definitions ### High-Level API - **Decompressor**: Public API for decompression with `Unwrap()` methods and dictionary support - **Compressor**: Public API for compression with `Wrap()` methods - **DecompressionStream** and **CompressionStream**: Stream-based APIs - **SafeHandles**: `SafeCctxHandle` and `SafeDctxHandle` for safe resource management - **Supporting Classes**: `JobThreadPool`, `SynchronizationWrapper`, `UnmanagedObject`, `Constants`, `BitOperations` ### Dependency Removal - **Removed** `ZstdSharp.Port` package reference from `SharpCompress.csproj` - **Updated** all ZstdSharp usages to internal classes: - `Common/Zip/ZipFilePart.cs`: Changed namespace from `ZstdSharp` to `SharpCompress.Compressors.ZStandard` - `Compressors/LZMA/Registry.cs`: Updated namespace usage - `Writers/Zip/ZipWriter.cs`: Changed `new ZstdSharp.CompressionStream(...)` to `new CompressionStream(...)` ### Bug Fixes and Improvements - Fixed obsolete API: `Thread.VolatileRead` → `Volatile.Read` in `ZstdDecompressBlock.cs` - Cleaned up unused using statements in `ZstandardConstants.cs` - Rebased onto latest master (394fd2e) for CSharpier 1.2.3 and other updates ## Build Status ✅ Project builds successfully on all targets (net48, net8.0, net10.0) ✅ No warnings or errors ✅ No external ZstdSharp dependency required ✅ Code formatted with CSharpier 1.2.3 ## Files Changed - **224 files added**: Complete ZstdSharp implementation - **3 files modified**: Namespace updates to remove ZstdSharp.Port references - **1 file modified**: Package reference removal from .csproj SharpCompress now has a complete, self-contained ZStandard implementation. <!-- START COPILOT CODING AGENT SUFFIX --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>Move ZstdSharp into SharpCompress</issue_title> > <issue_description>think the authors have little interest in this: https://github.com/oleg-st/ZstdSharp/issues/36 > > The biggest hurdle found so far is that there is a lot of unsafe/native usage that is probably incompatible with AOT. I know work was done to make everything CLSCompliant. > > I'm ignoring big issues but might also rethink what targets there are. The legacy framework targets are slowing things down too. > > Started here https://github.com/adamhathcock/sharpcompress/pull/949</issue_description> > > <agent_instructions>Make a start but there are a lot of changes needed to the code then stop and saw progress</agent_instructions> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > </comments> > </details> - Fixes adamhathcock/sharpcompress#1051 <!-- START COPILOT CODING AGENT TIPS --> --- ✨ Let Copilot coding agent [set things up for you](https://github.com/adamhathcock/sharpcompress/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.
claunia added the pull-request label 2026-01-29 22:20:45 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
feedback requested
not bug
pull-request
Mirrored from GitHub Pull Request
 question
up for grabs
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/sharpcompress#1476
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?