diff --git a/.agents/skills/sevenzip-format/SKILL.md b/.agents/skills/sevenzip-format/SKILL.md new file mode 100644 index 00000000..9193a457 --- /dev/null +++ b/.agents/skills/sevenzip-format/SKILL.md @@ -0,0 +1,21 @@ +--- +name: sevenzip-format +description: Reference the 7z/7zip archive container format. Use when Codex needs to answer questions or make code changes involving 7z signatures, headers, encoded headers, NID/property IDs, packed streams, folders/coders, bind pairs, substreams, file metadata properties, or SharpCompress 7Zip parsing behavior. +--- + +# Sevenzip Format + +Use this skill for 7z container-format work. It provides a local Markdown conversion of the LZMA SDK `7zFormat.txt` reference. + +## Reference + +- Read [references/7z-format.md](references/7z-format.md) when the task depends on 7z binary layout, header property IDs, stream/folder relationships, metadata fields, or encoded headers. +- Treat the reference as the LZMA SDK 7z format description version 4.59. It describes the container grammar, not compression method internals; method-specific codec details are outside this skill. +- Preserve source field names and numeric IDs when mapping the spec to code. The converted reference keeps source spelling inside syntax blocks where exact matching may matter. + +## Workflow + +1. Identify which part of the 7z container is involved: signature/start header, packed streams, coders/folders, substreams, files info, or encoded headers. +2. Open the relevant section in `references/7z-format.md` and use the table of contents to avoid loading unrelated details. +3. When implementing or reviewing parsing logic, pay special attention to optional blocks marked with `[]`, 7z's variable-length `UINT64` encoding, and little-endian `REAL_UINT64` fields. +4. Cross-check behavior against SharpCompress tests and existing parser conventions before changing public API or stream behavior. diff --git a/.agents/skills/sevenzip-format/references/7z-format.md b/.agents/skills/sevenzip-format/references/7z-format.md new file mode 100644 index 00000000..0cd056a3 --- /dev/null +++ b/.agents/skills/sevenzip-format/references/7z-format.md @@ -0,0 +1,478 @@ +# 7z Format Description (4.59) + +Source: https://github.com/jljusten/LZMA-SDK/blob/master/DOC/7zFormat.txt + +Raw download used for this conversion: https://raw.githubusercontent.com/jljusten/LZMA-SDK/master/DOC/7zFormat.txt + +Downloaded and converted on 2026-05-23. + +This is a Markdown conversion of the LZMA SDK plaintext 7z archive format description. Pseudo-grammar blocks preserve the source field names and spelling. + +## Contents + +- [Overview](#overview) +- [Format Structure Overview](#format-structure-overview) +- [Notes About Notation and Encoding](#notes-about-notation-and-encoding) +- [Property IDs](#property-ids) +- [7z Format Headers](#7z-format-headers) + - [SignatureHeader](#signatureheader) + - [ArchiveProperties](#archiveproperties) + - [Digests](#digests-numstreams) + - [PackInfo](#packinfo) + - [Folder](#folder) + - [Coders Info](#coders-info) + - [SubStreams Info](#substreams-info) + - [Streams Info](#streams-info) + - [FilesInfo](#filesinfo) + - [Header](#header) + - [HeaderInfo](#headerinfo) + +## Overview + +This file contains a description of the 7z archive format. + +A 7z archive can contain files compressed with any method. See `Methods.txt` in the LZMA SDK for descriptions of defined compression methods. + +## Format Structure Overview + +Some fields can be optional. + +### Archive Structure + +```text +SignatureHeader +[PackedStreams] +[PackedStreamsForHeaders] +[ + Header + or + { + Packed Header + HeaderInfo + } +] +``` + +### Header Structure + +```text +{ + ArchiveProperties + AdditionalStreams + { + PackInfo + { + PackPos + NumPackStreams + Sizes[NumPackStreams] + CRCs[NumPackStreams] + } + CodersInfo + { + NumFolders + Folders[NumFolders] + { + NumCoders + CodersInfo[NumCoders] + { + ID + NumInStreams; + NumOutStreams; + PropertiesSize + Properties[PropertiesSize] + } + NumBindPairs + BindPairsInfo[NumBindPairs] + { + InIndex; + OutIndex; + } + PackedIndices + } + UnPackSize[Folders][Folders.NumOutstreams] + CRCs[NumFolders] + } + SubStreamsInfo + { + NumUnPackStreamsInFolders[NumFolders]; + UnPackSizes[] + CRCs[] + } + } + MainStreamsInfo + { + (Same as in AdditionalStreams) + } + FilesInfo + { + NumFiles + Properties[] + { + ID + Size + Data + } + } +} +``` + +### HeaderInfo Structure + +```text +{ + (Same as in AdditionalStreams) +} +``` + +## Notes About Notation and Encoding + +7z uses little-endian encoding. + +Optional headers are marked as: + +```text +[] +Header +[] +``` + +`REAL_UINT64` means a real `UINT64`. + +`UINT64` means a real `UINT64` encoded with the following scheme. The size of the encoding sequence depends on the first byte: + +| First byte (binary) | Extra bytes | Value | +| --- | --- | --- | +| `0xxxxxxx` | none | `( xxxxxxx )` | +| `10xxxxxx` | `BYTE y[1]` | `( xxxxxx << (8 * 1)) + y` | +| `110xxxxx` | `BYTE y[2]` | `( xxxxx << (8 * 2)) + y` | +| `...` | `...` | `...` | +| `1111110x` | `BYTE y[6]` | `( x << (8 * 6)) + y` | +| `11111110` | `BYTE y[7]` | `y` | +| `11111111` | `BYTE y[8]` | `y` | + +## Property IDs + +| ID | Name | +| --- | --- | +| `0x00` | `kEnd` | +| `0x01` | `kHeader` | +| `0x02` | `kArchiveProperties` | +| `0x03` | `kAdditionalStreamsInfo` | +| `0x04` | `kMainStreamsInfo` | +| `0x05` | `kFilesInfo` | +| `0x06` | `kPackInfo` | +| `0x07` | `kUnPackInfo` | +| `0x08` | `kSubStreamsInfo` | +| `0x09` | `kSize` | +| `0x0A` | `kCRC` | +| `0x0B` | `kFolder` | +| `0x0C` | `kCodersUnPackSize` | +| `0x0D` | `kNumUnPackStream` | +| `0x0E` | `kEmptyStream` | +| `0x0F` | `kEmptyFile` | +| `0x10` | `kAnti` | +| `0x11` | `kName` | +| `0x12` | `kCTime` | +| `0x13` | `kATime` | +| `0x14` | `kMTime` | +| `0x15` | `kWinAttributes` | +| `0x16` | `kComment` | +| `0x17` | `kEncodedHeader` | +| `0x18` | `kStartPos` | +| `0x19` | `kDummy` | + +## 7z Format Headers + +### SignatureHeader + +```text +BYTE kSignature[6] = {'7', 'z', 0xBC, 0xAF, 0x27, 0x1C}; + +ArchiveVersion +{ + BYTE Major; // now = 0 + BYTE Minor; // now = 2 +}; + +UINT32 StartHeaderCRC; + +StartHeader +{ + REAL_UINT64 NextHeaderOffset + REAL_UINT64 NextHeaderSize + UINT32 NextHeaderCRC +} +``` + +### ArchiveProperties + +```text +BYTE NID::kArchiveProperties (0x02) +for (;;) +{ + BYTE PropertyType; + if (aType == 0) + break; + UINT64 PropertySize; + BYTE PropertyData[PropertySize]; +} +``` + +### Digests (NumStreams) + +```text +BYTE AllAreDefined +if (AllAreDefined == 0) +{ + for(NumStreams) + BIT Defined +} +UINT32 CRCs[NumDefined] +``` + +### PackInfo + +```text +BYTE NID::kPackInfo (0x06) +UINT64 PackPos +UINT64 NumPackStreams + +[] +BYTE NID::kSize (0x09) +UINT64 PackSizes[NumPackStreams] +[] + +[] +BYTE NID::kCRC (0x0A) +PackStreamDigests[NumPackStreams] +[] + +BYTE NID::kEnd +``` + +### Folder + +```text +UINT64 NumCoders; +for (NumCoders) +{ + BYTE + { + 0:3 CodecIdSize + 4: Is Complex Coder + 5: There Are Attributes + 6: Reserved + 7: There are more alternative methods. (Not used anymore, must be 0). + } + BYTE CodecId[CodecIdSize] + if (Is Complex Coder) + { + UINT64 NumInStreams; + UINT64 NumOutStreams; + } + if (There Are Attributes) + { + UINT64 PropertiesSize + BYTE Properties[PropertiesSize] + } +} + +NumBindPairs = NumOutStreamsTotal - 1; + +for (NumBindPairs) +{ + UINT64 InIndex; + UINT64 OutIndex; +} + +NumPackedStreams = NumInStreamsTotal - NumBindPairs; +if (NumPackedStreams > 1) + for(NumPackedStreams) + { + UINT64 Index; + }; +``` + +### Coders Info + +```text +BYTE NID::kUnPackInfo (0x07) + +BYTE NID::kFolder (0x0B) +UINT64 NumFolders +BYTE External +switch(External) +{ + case 0: + Folders[NumFolders] + case 1: + UINT64 DataStreamIndex +} + +BYTE ID::kCodersUnPackSize (0x0C) +for(Folders) + for(Folder.NumOutStreams) + UINT64 UnPackSize; + +[] +BYTE NID::kCRC (0x0A) +UnPackDigests[NumFolders] +[] + +BYTE NID::kEnd +``` + +### SubStreams Info + +```text +BYTE NID::kSubStreamsInfo; (0x08) + +[] +BYTE NID::kNumUnPackStream; (0x0D) +UINT64 NumUnPackStreamsInFolders[NumFolders]; +[] + +[] +BYTE NID::kSize (0x09) +UINT64 UnPackSizes[] +[] + +[] +BYTE NID::kCRC (0x0A) +Digests[Number of streams with unknown CRC] +[] + +BYTE NID::kEnd +``` + +### Streams Info + +```text +[] +PackInfo +[] + +[] +CodersInfo +[] + +[] +SubStreamsInfo +[] + +BYTE NID::kEnd +``` + +### FilesInfo + +```text +BYTE NID::kFilesInfo; (0x05) +UINT64 NumFiles + +for (;;) +{ + BYTE PropertyType; + if (aType == 0) + break; + + UINT64 Size; + + switch(PropertyType) + { + kEmptyStream: (0x0E) + for(NumFiles) + BIT IsEmptyStream + + kEmptyFile: (0x0F) + for(EmptyStreams) + BIT IsEmptyFile + + kAnti: (0x10) + for(EmptyStreams) + BIT IsAntiFile + + case kCTime: (0x12) + case kATime: (0x13) + case kMTime: (0x14) + BYTE AllAreDefined + if (AllAreDefined == 0) + { + for(NumFiles) + BIT TimeDefined + } + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Definded Items) + UINT64 Time + [] + + kNames: (0x11) + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Files) + { + wchar_t Names[NameSize]; + wchar_t 0; + } + [] + + kAttributes: (0x15) + BYTE AllAreDefined + if (AllAreDefined == 0) + { + for(NumFiles) + BIT AttributesAreDefined + } + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Definded Attributes) + UINT32 Attributes + [] + } +} +``` + +### Header + +```text +BYTE NID::kHeader (0x01) + +[] +ArchiveProperties +[] + +[] +BYTE NID::kAdditionalStreamsInfo; (0x03) +StreamsInfo +[] + +[] +BYTE NID::kMainStreamsInfo; (0x04) +StreamsInfo +[] + +[] +FilesInfo +[] + +BYTE NID::kEnd +``` + +### HeaderInfo + +```text +[] +BYTE NID::kEncodedHeader; (0x17) +StreamsInfo for Encoded Header +[] +``` + +--- + +End of document. diff --git a/README.md b/README.md index 2bdb58d3..102950ae 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ # SharpCompress -SharpCompress is a compression library in pure C# for .NET Framework 4.8, .NET 8.0 and .NET 10.0 that can unrar, un7zip, unzip, untar unbzip2, ungzip, unlzip, unzstd, unarc and unarj with forward-only reading and file random access APIs. Write support for zip/tar/bzip2/gzip/lzip are implemented. +SharpCompress is a compression library in pure C# for .NET Framework 4.8, .NET Standard 2.0/2.1, .NET 6.0, .NET 8.0, and .NET 10.0 that can unrar, un7zip, unzip, untar, unbzip2, ungzip, unlzip, unzstd, unarc, and unarj with forward-only reading and file random access APIs. Write support for zip, tar, bzip2, gzip, lzip, zstandard compression streams, and 7zip archives is implemented. The major feature is support for non-seekable streams so large files can be processed on the fly (i.e. download stream). **NEW:** All I/O operations now support async/await for improved performance and scalability. See the [USAGE.md](docs/USAGE.md#async-examples) for examples. GitHub Actions Build - -[![SharpCompress](https://github.com/adamhathcock/sharpcompress/actions/workflows/dotnetcore.yml/badge.svg)](https://github.com/adamhathcock/sharpcompress/actions/workflows/dotnetcore.yml) +[![SharpCompress](https://github.com/adamhathcock/sharpcompress/actions/workflows/nuget-release.yml/badge.svg)](https://github.com/adamhathcock/sharpcompress/actions/workflows/nuget-release.yml) [![Static Badge](https://img.shields.io/badge/API%20Docs-DNDocs-190088?logo=readme&logoColor=white)](https://dndocs.com/d/sharpcompress/api/index.html) ## Need Help? diff --git a/docs/FORMATS.md b/docs/FORMATS.md index d086f92c..7f543f11 100644 --- a/docs/FORMATS.md +++ b/docs/FORMATS.md @@ -8,24 +8,24 @@ ## Supported Format Table -| Archive Format | Compression Format(s) | Compress/Decompress | Archive API | Reader API | Writer API | -| ---------------------- | ------------------------------------------------- | ------------------- | --------------- | ---------- | ------------- | -| Ace | None | Decompress | N/A | AceReader | N/A | -| Arc | None, Packed, Squeezed, Crunched | Decompress | N/A | ArcReader | N/A | -| Arj | None | Decompress | N/A | ArjReader | N/A | -| Rar | Rar | Decompress | RarArchive | RarReader | N/A | -| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA/LZMA2, PPMd | Both | ZipArchive | ZipReader | ZipWriter | -| Tar | None | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.Zstandard | ZStandard | Decompress | TarArchive | TarReader | N/A | -| Tar.LZip | LZMA | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.XZ | LZMA2 | Decompress | TarArchive | TarReader | N/A | -| GZip (single file) | DEFLATE | Both | GZipArchive | GZipReader | GZipWriter | -| 7Zip (4) | LZMA, LZMA2, BZip2, PPMd, BCJ, BCJ2, Deflate | Both | SevenZipArchive | N/A | SevenZipWriter | +| Archive Format | Compression Format(s) | Compress/Decompress | Archive API | Reader API | Writer API | +| ------------------ | ------------------------------------------------------------------- | ------------------- | --------------- | ---------- | --------------- | +| Ace | None | Decompress | N/A | AceReader | N/A | +| Arc | None, Packed, Squeezed, Crunched | Decompress | N/A | ArcReader | N/A | +| Arj | None | Decompress | N/A | ArjReader | N/A | +| Rar | Rar | Decompress | RarArchive | RarReader | N/A | +| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA/LZMA2, PPMd, ZStandard, XZ | Both | ZipArchive | ZipReader | ZipWriter | +| Tar | None | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.Zstandard | ZStandard | Decompress | TarArchive | TarReader | N/A | +| Tar.LZip | LZMA | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.XZ | LZMA2 | Decompress | TarArchive | TarReader | N/A | +| GZip (single file) | DEFLATE | Both | GZipArchive | GZipReader | GZipWriter | +| 7Zip (4) | LZMA, LZMA2, BZip2, PPMd, BCJ, BCJ2, Deflate | Both | SevenZipArchive | N/A | SevenZipWriter | 1. SOLID Rars are only supported in the RarReader API. -2. Zip format supports pkware and WinzipAES encryption. However, encrypted LZMA is not supported. Zip64 reading/writing is supported but only with seekable streams as the Zip spec doesn't support Zip64 data in post data descriptors. Deflate64 is only supported for reading. See [Zip Format Notes](#zip-format-notes) for details on multi-volume archives and streaming behavior. +2. Zip format supports pkware and WinzipAES encryption. However, encrypted LZMA is not supported. Zip64 reading/writing is supported but only with seekable streams as the Zip spec doesn't support Zip64 data in post data descriptors. Deflate64, Shrink, Reduce, Implode, and XZ are only supported for reading. ZStandard is supported for reading and writing. See [Zip Format Notes](#zip-format-notes) for details on multi-volume archives and streaming behavior. 3. The Tar format requires a file size in the header. If no size is specified to the TarWriter and the stream is not seekable, then an exception will be thrown. 4. The 7Zip format doesn't allow for reading as a forward-only stream, so 7Zip read support is only through the Archive API. Writing is supported through SevenZipWriter for non-solid archives with LZMA/LZMA2 and requires a seekable output stream. See [7Zip Format Notes](#7zip-format-notes) for details on async extraction behavior. 5. LZip has no support for extra data like the file name or timestamp. There is a default filename used when looking at the entry Key on the archive. @@ -62,7 +62,7 @@ For those who want to directly compress/decompress bits. The single file formats | ADCStream | Decompress | | LZipStream | Both | | XZStream | Decompress | -| ZStandardStream | Decompress | +| ZStandard CompressionStream/DecompressionStream | Both | ## Archive Formats vs Compression diff --git a/opencode.json b/opencode.json new file mode 100644 index 00000000..e38cc010 --- /dev/null +++ b/opencode.json @@ -0,0 +1,6 @@ +{ + "$schema": "https://opencode.ai/config.json", + "skills": { + "paths": [".agents/skills"] + } +}