starred/sharpcompress
1
0
Fork 0
mirror of https://github.com/adamhathcock/sharpcompress.git synced 2026-02-03 21:23:38 +00:00
Code Issues 975 Packages Projects Releases 20 Wiki Activity

[PR #1052] [MERGED] Move ZstdSharp into SharpCompress - Complete Integration #1473

New Issue
Open
opened 2026-01-29 22:20:44 +00:00 by claunia · 0 comments
claunia commented 2026-01-29 22:20:44 +00:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/adamhathcock/sharpcompress/pull/1052
Author: @Copilot
Created: 11/29/2025
Status: ✅ Merged
Merged: 12/18/2025
Merged by: @adamhathcock

Base: master ← Head: copilot/move-zstdsharp-into-sharpcompress

📝 Commits (10+)

  • 0d671a0 Initial plan
  • 94c64b2 Add initial ZStandard infrastructure types for ZstdSharp integration
  • d3428b0 Fix XML documentation comments in buffer structs
  • 4021208 Add core ZStandard infrastructure: UnsafeHelper, ThrowHelper, and Unsafe Methods
  • b9b159b Add ZStandard compression parameter types and enums
  • 3a6d24b Add ZStandard frame and dictionary types
  • 1b3002c Initial plan
  • 8680e3b Drop .NET 6 support and add .NET 10 support
  • b4c5437 Fix .NET 10.0 formatting in README
  • a08f953 Remove netstandard2.0 and net481 targets, keep net48, net8.0, net10.0

📊 Changes

239 files changed (+55774 additions, -75 deletions)

View changed files

📝 src/SharpCompress/AssemblyInfo.cs (+2 -1)
📝 src/SharpCompress/Common/Zip/ZipFilePart.cs (+1 -1)
📝 src/SharpCompress/Compressors/LZMA/Registry.cs (+1 -1)
➕ src/SharpCompress/Compressors/ZStandard/BitOperations.cs (+311 -0)
➕ src/SharpCompress/Compressors/ZStandard/CompressionStream.cs (+301 -0)
➕ src/SharpCompress/Compressors/ZStandard/Compressor.cs (+204 -0)
➕ src/SharpCompress/Compressors/ZStandard/Constants.cs (+8 -0)
➕ src/SharpCompress/Compressors/ZStandard/DecompressionStream.cs (+293 -0)
➕ src/SharpCompress/Compressors/ZStandard/Decompressor.cs (+176 -0)
➕ src/SharpCompress/Compressors/ZStandard/JobThreadPool.cs (+141 -0)
➕ src/SharpCompress/Compressors/ZStandard/SafeHandles.cs (+163 -0)
➕ src/SharpCompress/Compressors/ZStandard/SynchronizationWrapper.cs (+22 -0)
➕ src/SharpCompress/Compressors/ZStandard/ThrowHelper.cs (+48 -0)
➕ src/SharpCompress/Compressors/ZStandard/UnmanagedObject.cs (+18 -0)
➕ src/SharpCompress/Compressors/ZStandard/Unsafe/Allocations.cs (+52 -0)
➕ src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_CStream_t.cs (+14 -0)
➕ src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_status.cs (+16 -0)
➕ src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_t.cs (+13 -0)
➕ src/SharpCompress/Compressors/ZStandard/Unsafe/Bits.cs (+60 -0)
➕ src/SharpCompress/Compressors/ZStandard/Unsafe/Bitstream.cs (+739 -0)

...and 80 more files

📄 Description

Completes the full integration of ZstdSharp directly into SharpCompress, removing the external ZstdSharp.Port dependency (addresses #1030, started in #949).

Overview

This PR integrates all ~217 ZstdSharp implementation files into SharpCompress, providing a fully self-contained ZStandard compression/decompression implementation without external dependencies.

Changes Made

Core Infrastructure

  • CLSCompliant(false): Required for unsafe pointer usage in ZStandard internals
  • Error Handling: ZSTD_ErrorCode enum, ZstdException class, ThrowHelper with extension methods
  • Memory Management: UnsafeHelper with malloc, calloc, free, memcpy, memset, memmove, memcmp
  • Buffer Types: ZSTD_inBuffer_s, ZSTD_outBuffer_s, ZSTD_customMem
  • Parameter Types: Compression (ZSTD_cParameter, ZSTD_compressionParameters, ZSTD_strategy) and decompression (ZSTD_dParameter) parameters
  • Frame Types: ZSTD_frameType_e, ZSTD_frameHeader, ZSTD_frameParameters, ZSTD_parameters
  • Dictionary Types: ZSTD_dictContentType_e, ZSTD_dictLoadMethod_e
  • Directives: ZSTD_EndDirective, ZSTD_ResetDirective

Complete Unsafe Implementation (~217 files)

  • Methods Partial Class: ErrorPrivate, Mem (I/O API), Bits, Compiler, Allocations, ZstdCommon
  • FSE (Finite State Entropy): Compression and decompression algorithms (Fse.cs, FseCompress.cs, FseDecompress.cs)
  • Huffman Encoding: Compression and decompression (Huf.cs, HufCompress.cs, HufDecompress.cs)
  • Dictionary Training: COVER and FastCover algorithms (Cover.cs, Fastcover.cs, Zdict.cs)
  • Core Algorithms:
    • Compression: ZstdCompress.cs, ZstdCompressInternal.cs, ZstdCompressLiterals.cs, ZstdCompressSequences.cs, ZstdCompressSuperblock.cs
    • Decompression: ZstdDecompress.cs, ZstdDecompressBlock.cs, ZstdDecompressInternal.cs
    • Strategy implementations: ZstdFast.cs, ZstdDoubleFast.cs, ZstdLazy.cs, ZstdOpt.cs
    • Long-distance matching: ZstdLdm.cs
  • Entropy Coding: EntropyCommon.cs, Bitstream.cs
  • Context Management: ZSTD_CCtx_s.cs, ZSTD_DCtx_s.cs, compression/decompression context structures
  • Multithreading: ZstdmtCompress.cs, Pool.cs
  • XXHash: Xxhash.cs for checksums
  • Supporting Data Structures: 200+ struct and enum definitions

High-Level API

  • Decompressor: Public API for decompression with Unwrap() methods and dictionary support
  • Compressor: Public API for compression with Wrap() methods
  • DecompressionStream and CompressionStream: Stream-based APIs
  • SafeHandles: SafeCctxHandle and SafeDctxHandle for safe resource management
  • Supporting Classes: JobThreadPool, SynchronizationWrapper, UnmanagedObject, Constants, BitOperations

Dependency Removal

  • Removed ZstdSharp.Port package reference from SharpCompress.csproj
  • Updated all ZstdSharp usages to internal classes:
    • Common/Zip/ZipFilePart.cs: Changed namespace from ZstdSharp to SharpCompress.Compressors.ZStandard
    • Compressors/LZMA/Registry.cs: Updated namespace usage
    • Writers/Zip/ZipWriter.cs: Changed new ZstdSharp.CompressionStream(...) to new CompressionStream(...)

Bug Fixes and Improvements

  • Fixed obsolete API: Thread.VolatileRead → Volatile.Read in ZstdDecompressBlock.cs
  • Cleaned up unused using statements in ZstandardConstants.cs
  • Rebased onto latest master (394fd2e) for CSharpier 1.2.3 and other updates

Build Status

✅ Project builds successfully on all targets (net48, net8.0, net10.0)
✅ No warnings or errors
✅ No external ZstdSharp dependency required
✅ Code formatted with CSharpier 1.2.3

Files Changed

  • 224 files added: Complete ZstdSharp implementation
  • 3 files modified: Namespace updates to remove ZstdSharp.Port references
  • 1 file modified: Package reference removal from .csproj

SharpCompress now has a complete, self-contained ZStandard implementation.

Original prompt

This section details on the original issue you should resolve

<issue_title>Move ZstdSharp into SharpCompress</issue_title>
<issue_description>think the authors have little interest in this: https://github.com/oleg-st/ZstdSharp/issues/36

The biggest hurdle found so far is that there is a lot of unsafe/native usage that is probably incompatible with AOT. I know work was done to make everything CLSCompliant.

I'm ignoring big issues but might also rethink what targets there are. The legacy framework targets are slowing things down too.

Started here https://github.com/adamhathcock/sharpcompress/pull/949</issue_description>

<agent_instructions>Make a start but there are a lot of changes needed to the code then stop and saw progress</agent_instructions>

Comments on the Issue (you are @copilot in this section)

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/adamhathcock/sharpcompress/pull/1052 **Author:** [@Copilot](https://github.com/apps/copilot-swe-agent) **Created:** 11/29/2025 **Status:** ✅ Merged **Merged:** 12/18/2025 **Merged by:** [@adamhathcock](https://github.com/adamhathcock) **Base:** `master` ← **Head:** `copilot/move-zstdsharp-into-sharpcompress` --- ### 📝 Commits (10+) - [`0d671a0`](https://github.com/adamhathcock/sharpcompress/commit/0d671a0bb269b651ee6b47ed7e95ab1c5ae1d2b8) Initial plan - [`94c64b2`](https://github.com/adamhathcock/sharpcompress/commit/94c64b2a45b67734400852f504251860998adc0a) Add initial ZStandard infrastructure types for ZstdSharp integration - [`d3428b0`](https://github.com/adamhathcock/sharpcompress/commit/d3428b066ec7c2b21bcbb59b3c86320f973a7e4f) Fix XML documentation comments in buffer structs - [`4021208`](https://github.com/adamhathcock/sharpcompress/commit/40212083a5961ac7ae642d01229cb7fe6f576313) Add core ZStandard infrastructure: UnsafeHelper, ThrowHelper, and Unsafe Methods - [`b9b159b`](https://github.com/adamhathcock/sharpcompress/commit/b9b159be4c4f23e4c5cc24a1ef7feab35dc7bc6e) Add ZStandard compression parameter types and enums - [`3a6d24b`](https://github.com/adamhathcock/sharpcompress/commit/3a6d24b1d99df6233baf3370aacadd25a437e590) Add ZStandard frame and dictionary types - [`1b3002c`](https://github.com/adamhathcock/sharpcompress/commit/1b3002c8df87e14bcf13e9a0bbfc1627f89048b9) Initial plan - [`8680e3b`](https://github.com/adamhathcock/sharpcompress/commit/8680e3b39e8458b132ba9c0dad6b79f795eed8f5) Drop .NET 6 support and add .NET 10 support - [`b4c5437`](https://github.com/adamhathcock/sharpcompress/commit/b4c5437c928a73f3737dcad4092bc0a9c270c2f1) Fix .NET 10.0 formatting in README - [`a08f953`](https://github.com/adamhathcock/sharpcompress/commit/a08f95326c8f337af31c02a649cf462879a0c753) Remove netstandard2.0 and net481 targets, keep net48, net8.0, net10.0 ### 📊 Changes **239 files changed** (+55774 additions, -75 deletions) <details> <summary>View changed files</summary> 📝 `src/SharpCompress/AssemblyInfo.cs` (+2 -1) 📝 `src/SharpCompress/Common/Zip/ZipFilePart.cs` (+1 -1) 📝 `src/SharpCompress/Compressors/LZMA/Registry.cs` (+1 -1) ➕ `src/SharpCompress/Compressors/ZStandard/BitOperations.cs` (+311 -0) ➕ `src/SharpCompress/Compressors/ZStandard/CompressionStream.cs` (+301 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Compressor.cs` (+204 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Constants.cs` (+8 -0) ➕ `src/SharpCompress/Compressors/ZStandard/DecompressionStream.cs` (+293 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Decompressor.cs` (+176 -0) ➕ `src/SharpCompress/Compressors/ZStandard/JobThreadPool.cs` (+141 -0) ➕ `src/SharpCompress/Compressors/ZStandard/SafeHandles.cs` (+163 -0) ➕ `src/SharpCompress/Compressors/ZStandard/SynchronizationWrapper.cs` (+22 -0) ➕ `src/SharpCompress/Compressors/ZStandard/ThrowHelper.cs` (+48 -0) ➕ `src/SharpCompress/Compressors/ZStandard/UnmanagedObject.cs` (+18 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Unsafe/Allocations.cs` (+52 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_CStream_t.cs` (+14 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_status.cs` (+16 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_t.cs` (+13 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Unsafe/Bits.cs` (+60 -0) ➕ `src/SharpCompress/Compressors/ZStandard/Unsafe/Bitstream.cs` (+739 -0) _...and 80 more files_ </details> ### 📄 Description Completes the full integration of ZstdSharp directly into SharpCompress, removing the external ZstdSharp.Port dependency (addresses #1030, started in #949). ## Overview This PR integrates all ~217 ZstdSharp implementation files into SharpCompress, providing a fully self-contained ZStandard compression/decompression implementation without external dependencies. ## Changes Made ### Core Infrastructure - **CLSCompliant(false)**: Required for unsafe pointer usage in ZStandard internals - **Error Handling**: `ZSTD_ErrorCode` enum, `ZstdException` class, `ThrowHelper` with extension methods - **Memory Management**: `UnsafeHelper` with malloc, calloc, free, memcpy, memset, memmove, memcmp - **Buffer Types**: `ZSTD_inBuffer_s`, `ZSTD_outBuffer_s`, `ZSTD_customMem` - **Parameter Types**: Compression (`ZSTD_cParameter`, `ZSTD_compressionParameters`, `ZSTD_strategy`) and decompression (`ZSTD_dParameter`) parameters - **Frame Types**: `ZSTD_frameType_e`, `ZSTD_frameHeader`, `ZSTD_frameParameters`, `ZSTD_parameters` - **Dictionary Types**: `ZSTD_dictContentType_e`, `ZSTD_dictLoadMethod_e` - **Directives**: `ZSTD_EndDirective`, `ZSTD_ResetDirective` ### Complete Unsafe Implementation (~217 files) - **Methods Partial Class**: ErrorPrivate, Mem (I/O API), Bits, Compiler, Allocations, ZstdCommon - **FSE (Finite State Entropy)**: Compression and decompression algorithms (`Fse.cs`, `FseCompress.cs`, `FseDecompress.cs`) - **Huffman Encoding**: Compression and decompression (`Huf.cs`, `HufCompress.cs`, `HufDecompress.cs`) - **Dictionary Training**: COVER and FastCover algorithms (`Cover.cs`, `Fastcover.cs`, `Zdict.cs`) - **Core Algorithms**: - Compression: `ZstdCompress.cs`, `ZstdCompressInternal.cs`, `ZstdCompressLiterals.cs`, `ZstdCompressSequences.cs`, `ZstdCompressSuperblock.cs` - Decompression: `ZstdDecompress.cs`, `ZstdDecompressBlock.cs`, `ZstdDecompressInternal.cs` - Strategy implementations: `ZstdFast.cs`, `ZstdDoubleFast.cs`, `ZstdLazy.cs`, `ZstdOpt.cs` - Long-distance matching: `ZstdLdm.cs` - **Entropy Coding**: `EntropyCommon.cs`, `Bitstream.cs` - **Context Management**: `ZSTD_CCtx_s.cs`, `ZSTD_DCtx_s.cs`, compression/decompression context structures - **Multithreading**: `ZstdmtCompress.cs`, `Pool.cs` - **XXHash**: `Xxhash.cs` for checksums - **Supporting Data Structures**: 200+ struct and enum definitions ### High-Level API - **Decompressor**: Public API for decompression with `Unwrap()` methods and dictionary support - **Compressor**: Public API for compression with `Wrap()` methods - **DecompressionStream** and **CompressionStream**: Stream-based APIs - **SafeHandles**: `SafeCctxHandle` and `SafeDctxHandle` for safe resource management - **Supporting Classes**: `JobThreadPool`, `SynchronizationWrapper`, `UnmanagedObject`, `Constants`, `BitOperations` ### Dependency Removal - **Removed** `ZstdSharp.Port` package reference from `SharpCompress.csproj` - **Updated** all ZstdSharp usages to internal classes: - `Common/Zip/ZipFilePart.cs`: Changed namespace from `ZstdSharp` to `SharpCompress.Compressors.ZStandard` - `Compressors/LZMA/Registry.cs`: Updated namespace usage - `Writers/Zip/ZipWriter.cs`: Changed `new ZstdSharp.CompressionStream(...)` to `new CompressionStream(...)` ### Bug Fixes and Improvements - Fixed obsolete API: `Thread.VolatileRead` → `Volatile.Read` in `ZstdDecompressBlock.cs` - Cleaned up unused using statements in `ZstandardConstants.cs` - Rebased onto latest master (394fd2e) for CSharpier 1.2.3 and other updates ## Build Status ✅ Project builds successfully on all targets (net48, net8.0, net10.0) ✅ No warnings or errors ✅ No external ZstdSharp dependency required ✅ Code formatted with CSharpier 1.2.3 ## Files Changed - **224 files added**: Complete ZstdSharp implementation - **3 files modified**: Namespace updates to remove ZstdSharp.Port references - **1 file modified**: Package reference removal from .csproj SharpCompress now has a complete, self-contained ZStandard implementation. <!-- START COPILOT CODING AGENT SUFFIX --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>Move ZstdSharp into SharpCompress</issue_title> > <issue_description>think the authors have little interest in this: https://github.com/oleg-st/ZstdSharp/issues/36 > > The biggest hurdle found so far is that there is a lot of unsafe/native usage that is probably incompatible with AOT. I know work was done to make everything CLSCompliant. > > I'm ignoring big issues but might also rethink what targets there are. The legacy framework targets are slowing things down too. > > Started here https://github.com/adamhathcock/sharpcompress/pull/949</issue_description> > > <agent_instructions>Make a start but there are a lot of changes needed to the code then stop and saw progress</agent_instructions> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > </comments> > </details> - Fixes adamhathcock/sharpcompress#1051 <!-- START COPILOT CODING AGENT TIPS --> --- ✨ Let Copilot coding agent [set things up for you](https://github.com/adamhathcock/sharpcompress/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
claunia added the pull-request label 2026-01-29 22:20:44 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
feedback requested
not bug
pull-request
Mirrored from GitHub Pull Request
 question
up for grabs
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/sharpcompress#1473
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?