From 673cb3637afd2e9313c42839f80a4906cf470efb Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Mon, 25 May 2026 10:10:26 +0100 Subject: [PATCH 1/8] start skills --- .agents/skills/sevenzip-format/SKILL.md | 21 + .../sevenzip-format/references/7z-format.md | 478 ++++++++++++++++++ README.md | 4 +- docs/FORMATS.md | 34 +- opencode.json | 6 + 5 files changed, 524 insertions(+), 19 deletions(-) create mode 100644 .agents/skills/sevenzip-format/SKILL.md create mode 100644 .agents/skills/sevenzip-format/references/7z-format.md create mode 100644 opencode.json diff --git a/.agents/skills/sevenzip-format/SKILL.md b/.agents/skills/sevenzip-format/SKILL.md new file mode 100644 index 00000000..9193a457 --- /dev/null +++ b/.agents/skills/sevenzip-format/SKILL.md @@ -0,0 +1,21 @@ +--- +name: sevenzip-format +description: Reference the 7z/7zip archive container format. Use when Codex needs to answer questions or make code changes involving 7z signatures, headers, encoded headers, NID/property IDs, packed streams, folders/coders, bind pairs, substreams, file metadata properties, or SharpCompress 7Zip parsing behavior. +--- + +# Sevenzip Format + +Use this skill for 7z container-format work. It provides a local Markdown conversion of the LZMA SDK `7zFormat.txt` reference. + +## Reference + +- Read [references/7z-format.md](references/7z-format.md) when the task depends on 7z binary layout, header property IDs, stream/folder relationships, metadata fields, or encoded headers. +- Treat the reference as the LZMA SDK 7z format description version 4.59. It describes the container grammar, not compression method internals; method-specific codec details are outside this skill. +- Preserve source field names and numeric IDs when mapping the spec to code. The converted reference keeps source spelling inside syntax blocks where exact matching may matter. + +## Workflow + +1. Identify which part of the 7z container is involved: signature/start header, packed streams, coders/folders, substreams, files info, or encoded headers. +2. Open the relevant section in `references/7z-format.md` and use the table of contents to avoid loading unrelated details. +3. When implementing or reviewing parsing logic, pay special attention to optional blocks marked with `[]`, 7z's variable-length `UINT64` encoding, and little-endian `REAL_UINT64` fields. +4. Cross-check behavior against SharpCompress tests and existing parser conventions before changing public API or stream behavior. diff --git a/.agents/skills/sevenzip-format/references/7z-format.md b/.agents/skills/sevenzip-format/references/7z-format.md new file mode 100644 index 00000000..0cd056a3 --- /dev/null +++ b/.agents/skills/sevenzip-format/references/7z-format.md @@ -0,0 +1,478 @@ +# 7z Format Description (4.59) + +Source: https://github.com/jljusten/LZMA-SDK/blob/master/DOC/7zFormat.txt + +Raw download used for this conversion: https://raw.githubusercontent.com/jljusten/LZMA-SDK/master/DOC/7zFormat.txt + +Downloaded and converted on 2026-05-23. + +This is a Markdown conversion of the LZMA SDK plaintext 7z archive format description. Pseudo-grammar blocks preserve the source field names and spelling. + +## Contents + +- [Overview](#overview) +- [Format Structure Overview](#format-structure-overview) +- [Notes About Notation and Encoding](#notes-about-notation-and-encoding) +- [Property IDs](#property-ids) +- [7z Format Headers](#7z-format-headers) + - [SignatureHeader](#signatureheader) + - [ArchiveProperties](#archiveproperties) + - [Digests](#digests-numstreams) + - [PackInfo](#packinfo) + - [Folder](#folder) + - [Coders Info](#coders-info) + - [SubStreams Info](#substreams-info) + - [Streams Info](#streams-info) + - [FilesInfo](#filesinfo) + - [Header](#header) + - [HeaderInfo](#headerinfo) + +## Overview + +This file contains a description of the 7z archive format. + +A 7z archive can contain files compressed with any method. See `Methods.txt` in the LZMA SDK for descriptions of defined compression methods. + +## Format Structure Overview + +Some fields can be optional. + +### Archive Structure + +```text +SignatureHeader +[PackedStreams] +[PackedStreamsForHeaders] +[ + Header + or + { + Packed Header + HeaderInfo + } +] +``` + +### Header Structure + +```text +{ + ArchiveProperties + AdditionalStreams + { + PackInfo + { + PackPos + NumPackStreams + Sizes[NumPackStreams] + CRCs[NumPackStreams] + } + CodersInfo + { + NumFolders + Folders[NumFolders] + { + NumCoders + CodersInfo[NumCoders] + { + ID + NumInStreams; + NumOutStreams; + PropertiesSize + Properties[PropertiesSize] + } + NumBindPairs + BindPairsInfo[NumBindPairs] + { + InIndex; + OutIndex; + } + PackedIndices + } + UnPackSize[Folders][Folders.NumOutstreams] + CRCs[NumFolders] + } + SubStreamsInfo + { + NumUnPackStreamsInFolders[NumFolders]; + UnPackSizes[] + CRCs[] + } + } + MainStreamsInfo + { + (Same as in AdditionalStreams) + } + FilesInfo + { + NumFiles + Properties[] + { + ID + Size + Data + } + } +} +``` + +### HeaderInfo Structure + +```text +{ + (Same as in AdditionalStreams) +} +``` + +## Notes About Notation and Encoding + +7z uses little-endian encoding. + +Optional headers are marked as: + +```text +[] +Header +[] +``` + +`REAL_UINT64` means a real `UINT64`. + +`UINT64` means a real `UINT64` encoded with the following scheme. The size of the encoding sequence depends on the first byte: + +| First byte (binary) | Extra bytes | Value | +| --- | --- | --- | +| `0xxxxxxx` | none | `( xxxxxxx )` | +| `10xxxxxx` | `BYTE y[1]` | `( xxxxxx << (8 * 1)) + y` | +| `110xxxxx` | `BYTE y[2]` | `( xxxxx << (8 * 2)) + y` | +| `...` | `...` | `...` | +| `1111110x` | `BYTE y[6]` | `( x << (8 * 6)) + y` | +| `11111110` | `BYTE y[7]` | `y` | +| `11111111` | `BYTE y[8]` | `y` | + +## Property IDs + +| ID | Name | +| --- | --- | +| `0x00` | `kEnd` | +| `0x01` | `kHeader` | +| `0x02` | `kArchiveProperties` | +| `0x03` | `kAdditionalStreamsInfo` | +| `0x04` | `kMainStreamsInfo` | +| `0x05` | `kFilesInfo` | +| `0x06` | `kPackInfo` | +| `0x07` | `kUnPackInfo` | +| `0x08` | `kSubStreamsInfo` | +| `0x09` | `kSize` | +| `0x0A` | `kCRC` | +| `0x0B` | `kFolder` | +| `0x0C` | `kCodersUnPackSize` | +| `0x0D` | `kNumUnPackStream` | +| `0x0E` | `kEmptyStream` | +| `0x0F` | `kEmptyFile` | +| `0x10` | `kAnti` | +| `0x11` | `kName` | +| `0x12` | `kCTime` | +| `0x13` | `kATime` | +| `0x14` | `kMTime` | +| `0x15` | `kWinAttributes` | +| `0x16` | `kComment` | +| `0x17` | `kEncodedHeader` | +| `0x18` | `kStartPos` | +| `0x19` | `kDummy` | + +## 7z Format Headers + +### SignatureHeader + +```text +BYTE kSignature[6] = {'7', 'z', 0xBC, 0xAF, 0x27, 0x1C}; + +ArchiveVersion +{ + BYTE Major; // now = 0 + BYTE Minor; // now = 2 +}; + +UINT32 StartHeaderCRC; + +StartHeader +{ + REAL_UINT64 NextHeaderOffset + REAL_UINT64 NextHeaderSize + UINT32 NextHeaderCRC +} +``` + +### ArchiveProperties + +```text +BYTE NID::kArchiveProperties (0x02) +for (;;) +{ + BYTE PropertyType; + if (aType == 0) + break; + UINT64 PropertySize; + BYTE PropertyData[PropertySize]; +} +``` + +### Digests (NumStreams) + +```text +BYTE AllAreDefined +if (AllAreDefined == 0) +{ + for(NumStreams) + BIT Defined +} +UINT32 CRCs[NumDefined] +``` + +### PackInfo + +```text +BYTE NID::kPackInfo (0x06) +UINT64 PackPos +UINT64 NumPackStreams + +[] +BYTE NID::kSize (0x09) +UINT64 PackSizes[NumPackStreams] +[] + +[] +BYTE NID::kCRC (0x0A) +PackStreamDigests[NumPackStreams] +[] + +BYTE NID::kEnd +``` + +### Folder + +```text +UINT64 NumCoders; +for (NumCoders) +{ + BYTE + { + 0:3 CodecIdSize + 4: Is Complex Coder + 5: There Are Attributes + 6: Reserved + 7: There are more alternative methods. (Not used anymore, must be 0). + } + BYTE CodecId[CodecIdSize] + if (Is Complex Coder) + { + UINT64 NumInStreams; + UINT64 NumOutStreams; + } + if (There Are Attributes) + { + UINT64 PropertiesSize + BYTE Properties[PropertiesSize] + } +} + +NumBindPairs = NumOutStreamsTotal - 1; + +for (NumBindPairs) +{ + UINT64 InIndex; + UINT64 OutIndex; +} + +NumPackedStreams = NumInStreamsTotal - NumBindPairs; +if (NumPackedStreams > 1) + for(NumPackedStreams) + { + UINT64 Index; + }; +``` + +### Coders Info + +```text +BYTE NID::kUnPackInfo (0x07) + +BYTE NID::kFolder (0x0B) +UINT64 NumFolders +BYTE External +switch(External) +{ + case 0: + Folders[NumFolders] + case 1: + UINT64 DataStreamIndex +} + +BYTE ID::kCodersUnPackSize (0x0C) +for(Folders) + for(Folder.NumOutStreams) + UINT64 UnPackSize; + +[] +BYTE NID::kCRC (0x0A) +UnPackDigests[NumFolders] +[] + +BYTE NID::kEnd +``` + +### SubStreams Info + +```text +BYTE NID::kSubStreamsInfo; (0x08) + +[] +BYTE NID::kNumUnPackStream; (0x0D) +UINT64 NumUnPackStreamsInFolders[NumFolders]; +[] + +[] +BYTE NID::kSize (0x09) +UINT64 UnPackSizes[] +[] + +[] +BYTE NID::kCRC (0x0A) +Digests[Number of streams with unknown CRC] +[] + +BYTE NID::kEnd +``` + +### Streams Info + +```text +[] +PackInfo +[] + +[] +CodersInfo +[] + +[] +SubStreamsInfo +[] + +BYTE NID::kEnd +``` + +### FilesInfo + +```text +BYTE NID::kFilesInfo; (0x05) +UINT64 NumFiles + +for (;;) +{ + BYTE PropertyType; + if (aType == 0) + break; + + UINT64 Size; + + switch(PropertyType) + { + kEmptyStream: (0x0E) + for(NumFiles) + BIT IsEmptyStream + + kEmptyFile: (0x0F) + for(EmptyStreams) + BIT IsEmptyFile + + kAnti: (0x10) + for(EmptyStreams) + BIT IsAntiFile + + case kCTime: (0x12) + case kATime: (0x13) + case kMTime: (0x14) + BYTE AllAreDefined + if (AllAreDefined == 0) + { + for(NumFiles) + BIT TimeDefined + } + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Definded Items) + UINT64 Time + [] + + kNames: (0x11) + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Files) + { + wchar_t Names[NameSize]; + wchar_t 0; + } + [] + + kAttributes: (0x15) + BYTE AllAreDefined + if (AllAreDefined == 0) + { + for(NumFiles) + BIT AttributesAreDefined + } + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Definded Attributes) + UINT32 Attributes + [] + } +} +``` + +### Header + +```text +BYTE NID::kHeader (0x01) + +[] +ArchiveProperties +[] + +[] +BYTE NID::kAdditionalStreamsInfo; (0x03) +StreamsInfo +[] + +[] +BYTE NID::kMainStreamsInfo; (0x04) +StreamsInfo +[] + +[] +FilesInfo +[] + +BYTE NID::kEnd +``` + +### HeaderInfo + +```text +[] +BYTE NID::kEncodedHeader; (0x17) +StreamsInfo for Encoded Header +[] +``` + +--- + +End of document. diff --git a/README.md b/README.md index 2bdb58d3..102950ae 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ # SharpCompress -SharpCompress is a compression library in pure C# for .NET Framework 4.8, .NET 8.0 and .NET 10.0 that can unrar, un7zip, unzip, untar unbzip2, ungzip, unlzip, unzstd, unarc and unarj with forward-only reading and file random access APIs. Write support for zip/tar/bzip2/gzip/lzip are implemented. +SharpCompress is a compression library in pure C# for .NET Framework 4.8, .NET Standard 2.0/2.1, .NET 6.0, .NET 8.0, and .NET 10.0 that can unrar, un7zip, unzip, untar, unbzip2, ungzip, unlzip, unzstd, unarc, and unarj with forward-only reading and file random access APIs. Write support for zip, tar, bzip2, gzip, lzip, zstandard compression streams, and 7zip archives is implemented. The major feature is support for non-seekable streams so large files can be processed on the fly (i.e. download stream). **NEW:** All I/O operations now support async/await for improved performance and scalability. See the [USAGE.md](docs/USAGE.md#async-examples) for examples. GitHub Actions Build - -[![SharpCompress](https://github.com/adamhathcock/sharpcompress/actions/workflows/dotnetcore.yml/badge.svg)](https://github.com/adamhathcock/sharpcompress/actions/workflows/dotnetcore.yml) +[![SharpCompress](https://github.com/adamhathcock/sharpcompress/actions/workflows/nuget-release.yml/badge.svg)](https://github.com/adamhathcock/sharpcompress/actions/workflows/nuget-release.yml) [![Static Badge](https://img.shields.io/badge/API%20Docs-DNDocs-190088?logo=readme&logoColor=white)](https://dndocs.com/d/sharpcompress/api/index.html) ## Need Help? diff --git a/docs/FORMATS.md b/docs/FORMATS.md index d086f92c..7f543f11 100644 --- a/docs/FORMATS.md +++ b/docs/FORMATS.md @@ -8,24 +8,24 @@ ## Supported Format Table -| Archive Format | Compression Format(s) | Compress/Decompress | Archive API | Reader API | Writer API | -| ---------------------- | ------------------------------------------------- | ------------------- | --------------- | ---------- | ------------- | -| Ace | None | Decompress | N/A | AceReader | N/A | -| Arc | None, Packed, Squeezed, Crunched | Decompress | N/A | ArcReader | N/A | -| Arj | None | Decompress | N/A | ArjReader | N/A | -| Rar | Rar | Decompress | RarArchive | RarReader | N/A | -| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA/LZMA2, PPMd | Both | ZipArchive | ZipReader | ZipWriter | -| Tar | None | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.Zstandard | ZStandard | Decompress | TarArchive | TarReader | N/A | -| Tar.LZip | LZMA | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.XZ | LZMA2 | Decompress | TarArchive | TarReader | N/A | -| GZip (single file) | DEFLATE | Both | GZipArchive | GZipReader | GZipWriter | -| 7Zip (4) | LZMA, LZMA2, BZip2, PPMd, BCJ, BCJ2, Deflate | Both | SevenZipArchive | N/A | SevenZipWriter | +| Archive Format | Compression Format(s) | Compress/Decompress | Archive API | Reader API | Writer API | +| ------------------ | ------------------------------------------------------------------- | ------------------- | --------------- | ---------- | --------------- | +| Ace | None | Decompress | N/A | AceReader | N/A | +| Arc | None, Packed, Squeezed, Crunched | Decompress | N/A | ArcReader | N/A | +| Arj | None | Decompress | N/A | ArjReader | N/A | +| Rar | Rar | Decompress | RarArchive | RarReader | N/A | +| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA/LZMA2, PPMd, ZStandard, XZ | Both | ZipArchive | ZipReader | ZipWriter | +| Tar | None | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.Zstandard | ZStandard | Decompress | TarArchive | TarReader | N/A | +| Tar.LZip | LZMA | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.XZ | LZMA2 | Decompress | TarArchive | TarReader | N/A | +| GZip (single file) | DEFLATE | Both | GZipArchive | GZipReader | GZipWriter | +| 7Zip (4) | LZMA, LZMA2, BZip2, PPMd, BCJ, BCJ2, Deflate | Both | SevenZipArchive | N/A | SevenZipWriter | 1. SOLID Rars are only supported in the RarReader API. -2. Zip format supports pkware and WinzipAES encryption. However, encrypted LZMA is not supported. Zip64 reading/writing is supported but only with seekable streams as the Zip spec doesn't support Zip64 data in post data descriptors. Deflate64 is only supported for reading. See [Zip Format Notes](#zip-format-notes) for details on multi-volume archives and streaming behavior. +2. Zip format supports pkware and WinzipAES encryption. However, encrypted LZMA is not supported. Zip64 reading/writing is supported but only with seekable streams as the Zip spec doesn't support Zip64 data in post data descriptors. Deflate64, Shrink, Reduce, Implode, and XZ are only supported for reading. ZStandard is supported for reading and writing. See [Zip Format Notes](#zip-format-notes) for details on multi-volume archives and streaming behavior. 3. The Tar format requires a file size in the header. If no size is specified to the TarWriter and the stream is not seekable, then an exception will be thrown. 4. The 7Zip format doesn't allow for reading as a forward-only stream, so 7Zip read support is only through the Archive API. Writing is supported through SevenZipWriter for non-solid archives with LZMA/LZMA2 and requires a seekable output stream. See [7Zip Format Notes](#7zip-format-notes) for details on async extraction behavior. 5. LZip has no support for extra data like the file name or timestamp. There is a default filename used when looking at the entry Key on the archive. @@ -62,7 +62,7 @@ For those who want to directly compress/decompress bits. The single file formats | ADCStream | Decompress | | LZipStream | Both | | XZStream | Decompress | -| ZStandardStream | Decompress | +| ZStandard CompressionStream/DecompressionStream | Both | ## Archive Formats vs Compression diff --git a/opencode.json b/opencode.json new file mode 100644 index 00000000..e38cc010 --- /dev/null +++ b/opencode.json @@ -0,0 +1,6 @@ +{ + "$schema": "https://opencode.ai/config.json", + "skills": { + "paths": [".agents/skills"] + } +} From a702e4faf19a8234623e21d3a00aac0ef61e7ea5 Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Mon, 25 May 2026 10:17:25 +0100 Subject: [PATCH 2/8] add tar skill --- .agents/skills/tar-format/SKILL.md | 22 ++ .../tar-format/references/tar-format.md | 360 ++++++++++++++++++ 2 files changed, 382 insertions(+) create mode 100644 .agents/skills/tar-format/SKILL.md create mode 100644 .agents/skills/tar-format/references/tar-format.md diff --git a/.agents/skills/tar-format/SKILL.md b/.agents/skills/tar-format/SKILL.md new file mode 100644 index 00000000..fbd8ec3f --- /dev/null +++ b/.agents/skills/tar-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: tar-format +description: Reference the Tar/USTAR/PAX/GNU tar archive container format. Use when Codex needs to answer questions or make code changes involving tar headers, 512-byte blocks, checksums, typeflags, USTAR prefixes, PAX local/global extended headers, GNU long names/links, sparse entries, wrapper compression, or SharpCompress Tar parsing and writing behavior. +--- + +# Tar Format + +Use this skill for tar container-format work. It provides a local, SharpCompress-oriented reference for POSIX USTAR, POSIX PAX, GNU tar extensions, and the current SharpCompress Tar implementation. + +## Reference + +- Read [references/tar-format.md](references/tar-format.md) when the task depends on tar binary layout, header field offsets, typeflag behavior, checksum rules, PAX records, GNU long-name/link records, wrapper compression support, or current SharpCompress Tar support boundaries. +- Treat the reference as an implementation guide, not a standards replacement. It cites POSIX and GNU tar sources, but it also documents SharpCompress-specific behavior and limitations. +- Prefer the SharpCompress support matrix in the reference over generic tar assumptions when changing code. Tar dialect support is intentionally partial in some areas. + +## Workflow + +1. Identify which layer is involved: raw tar block/header parsing, POSIX USTAR fields, POSIX PAX metadata, GNU tar extensions, wrapper compression, reader/archive/writer API behavior, or tests. +2. Open the relevant section in `references/tar-format.md` and use the source-file pointers before changing code. +3. For parsing changes, cross-check `TarHeader.cs`, `TarHeader.Async.cs`, `EntryType.cs`, and matching sync/async tests. +4. For writer changes, verify both sync and async file/directory paths and `TarWriterOptions.HeaderFormat` behavior. +5. For support claims, keep unsupported features explicit: PAX write, sparse reconstruction, device/FIFO semantics, link-writing APIs, and writing `tar.xz`, `tar.zst`, or `tar.Z`. diff --git a/.agents/skills/tar-format/references/tar-format.md b/.agents/skills/tar-format/references/tar-format.md new file mode 100644 index 00000000..1ee10c1a --- /dev/null +++ b/.agents/skills/tar-format/references/tar-format.md @@ -0,0 +1,360 @@ +# Tar Format Reference + +This reference summarizes the Tar archive container format for SharpCompress work. It is locally authored from public format references and the current SharpCompress implementation. + +Primary external references: + +- POSIX `pax` and `ustar`: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html +- GNU tar basic format: https://www.gnu.org/software/tar/manual/html_node/Standard.html +- GNU tar extensions: https://www.gnu.org/software/tar/manual/html_node/Extensions.html + +Primary SharpCompress references: + +- `docs/TAR_SPEC.md` +- `docs/TAR_GAP_ANALYSIS.md` +- `src/SharpCompress/Factories/TarFactory.cs` +- `src/SharpCompress/Factories/TarWrapper.cs` +- `src/SharpCompress/Common/Tar/Headers/TarHeader.cs` +- `src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs` +- `src/SharpCompress/Common/Tar/Headers/EntryType.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.Async.cs` +- `src/SharpCompress/Writers/Tar/TarWriterOptions.cs` +- `tests/SharpCompress.Test/Tar/` + +## Contents + +- [Format Overview](#format-overview) +- [USTAR Header Layout](#ustar-header-layout) +- [Entry Type Flags](#entry-type-flags) +- [Numeric Fields](#numeric-fields) +- [Checksum](#checksum) +- [Path Names](#path-names) +- [PAX Extended Headers](#pax-extended-headers) +- [GNU Tar Extensions](#gnu-tar-extensions) +- [SharpCompress Support Matrix](#sharpcompress-support-matrix) +- [SharpCompress Read Behavior](#sharpcompress-read-behavior) +- [SharpCompress Write Behavior](#sharpcompress-write-behavior) +- [Known Limitations](#known-limitations) +- [Test Fixtures](#test-fixtures) + +## Format Overview + +A tar archive is a sequence of 512-byte blocks. Each archive member is represented by: + +```text +header block (512 bytes) +payload blocks, padded to a 512-byte boundary +``` + +The archive should end with two 512-byte blocks filled with zero bytes. Readers should be tolerant of missing end markers because real-world tar tools may produce archives without them. + +The header contains file metadata and the payload size. Tar has no central directory. Streaming readers must consume or skip each payload and its padding before the next header can be parsed. + +SharpCompress relies on this in `TarReadOnlySubStream`: disposing an entry stream consumes unread entry bytes plus 512-byte padding so the next header remains aligned. + +## USTAR Header Layout + +The POSIX USTAR header is exactly 512 bytes. Field offsets are byte offsets from the beginning of the header block. + +| Field | Offset | Length | Notes | +| ----- | ------ | ------ | ----- | +| `name` | 0 | 100 | File name or final path component | +| `mode` | 100 | 8 | Octal file mode | +| `uid` | 108 | 8 | Octal owner id | +| `gid` | 116 | 8 | Octal group id | +| `size` | 124 | 12 | Octal payload size, or GNU base-256 in some archives | +| `mtime` | 136 | 12 | Octal seconds since Unix epoch | +| `chksum` | 148 | 8 | Header checksum | +| `typeflag` | 156 | 1 | Entry type | +| `linkname` | 157 | 100 | Link target for hard/symbolic links | +| `magic` | 257 | 6 | Usually `ustar` followed by NUL | +| `version` | 263 | 2 | Usually `00` | +| `uname` | 265 | 32 | Owner name | +| `gname` | 297 | 32 | Group name | +| `devmajor` | 329 | 8 | Character/block device major number | +| `devminor` | 337 | 8 | Character/block device minor number | +| `prefix` | 345 | 155 | USTAR path prefix | +| padding | 500 | 12 | Unused padding to 512 bytes | + +SharpCompress parses the core fields in `TarHeader.Read` and `TarHeader.ReadAsync`. It reconstructs USTAR paths as `prefix + "/" + name` when `magic` is exactly `ustar` and `prefix` is non-empty. + +## Entry Type Flags + +Common POSIX typeflags: + +| Typeflag | Meaning | +| -------- | ------- | +| NUL | Regular file, older tar form | +| `0` | Regular file | +| `1` | Hard link | +| `2` | Symbolic link | +| `3` | Character device | +| `4` | Block device | +| `5` | Directory | +| `6` | FIFO | +| `7` | Contiguous file, reserved by POSIX historical usage | +| `x` | POSIX PAX local extended header for the following file | +| `g` | POSIX PAX global extended header for following files | + +GNU and other extension typeflags relevant to SharpCompress: + +| Typeflag | Meaning | +| -------- | ------- | +| `K` | GNU long link target for the next real entry | +| `L` | GNU long path name for the next real entry | +| `S` | GNU sparse file | +| `V` | GNU volume header | + +SharpCompress declares these in `EntryType.cs`: + +```text +File = 0 +OldFile = '0' +HardLink = '1' +SymLink = '2' +CharDevice = '3' +BlockDevice = '4' +Directory = '5' +Fifo = '6' +LongLink = 'K' +LongName = 'L' +SparseFile = 'S' +VolumeHeader = 'V' +LocalExtendedHeader = 'x' +GlobalExtendedHeader = 'g' +``` + +Declaration does not mean full semantic support. Sparse, device, FIFO, and volume-header semantics are not fully modeled by the public API. + +## Numeric Fields + +Standard tar numeric fields are ASCII octal values, usually NUL-terminated or space-padded depending on writer. Important fields include `mode`, `uid`, `gid`, `size`, `mtime`, `chksum`, `devmajor`, and `devminor`. + +GNU tar can use base-256 binary encoding for values that exceed the octal field range: + +- A leading byte with bit `0x80` indicates a positive binary value. +- GNU documentation also describes `0xff` as a negative two's-complement marker. +- The value bytes are big-endian. + +SharpCompress currently handles binary `size` fields when bit `0x80` is set and writes large sizes in GNU long-link mode using a base-256-style binary `size` field. It also has an old GNU uid/gid quirk reader for fields beginning with `0x80 0x00`. + +## Checksum + +The checksum is the simple sum of all 512 header bytes, treating the 8-byte checksum field at offset 148 as spaces (`0x20`) during calculation. + +SharpCompress checksum behavior: + +- `RecalculateChecksum` fills the checksum field with eight spaces and sums bytes as unsigned values. +- `checkChecksum` accepts both POSIX unsigned sums and signed-byte sums used by some historical tar implementations. +- An all-zero block is treated as an empty/end marker case. + +When editing parser code, keep checksum compatibility broad enough for old archives. When editing writer code, use POSIX unsigned checksum output. + +## Path Names + +Classic tar has a 100-byte `name` field. POSIX USTAR extends this with a 155-byte `prefix` field. + +USTAR path reconstruction: + +```text +full path = prefix + "/" + name +``` + +USTAR writer constraints: + +- `name` must fit within the 100-byte name field. +- `prefix` must fit within the 155-byte prefix field. +- Splitting is normally done at a directory separator. + +SharpCompress write behavior: + +- `TarHeaderWriteFormat.USTAR` tries to split long paths into `prefix` and `name`. +- If a path cannot fit, SharpCompress throws `InvalidFormatException` and tells callers to use GNU tar format. +- `TarHeaderWriteFormat.GNU_TAR_LONG_LINK` writes GNU long-name metadata for names over 100 bytes. + +SharpCompress path normalization in `TarWriter`: + +- Converts backslashes to `/`. +- Removes drive prefixes before `:`. +- Trims leading and trailing `/` for file entries. +- Ensures directory entries end with `/`. +- Skips empty or root-equivalent directory names. + +## PAX Extended Headers + +POSIX PAX uses regular tar header blocks with special typeflags and a payload of UTF-8 key/value records. + +PAX header typeflags: + +- `x`: local extended header, applies to the next real file entry. +- `g`: global extended header, applies to subsequent entries until overridden by another global or local header. + +Each record has this form: + +```text + =\n +``` + +The decimal length includes every byte in the record, including the digits of the length itself, the space, key, equals sign, value, and newline. + +SharpCompress PAX read support is intentionally limited to selected keys: + +| Key | Effect | +| --- | ------ | +| `path` | Overrides entry path/name | +| `linkpath` | Overrides hard/symbolic link target | +| `size` | Overrides payload size | +| `mtime` | Overrides modification time | +| `uid` | Overrides owner id | +| `gid` | Overrides group id | +| `mode` | Overrides mode | + +Local metadata overrides global metadata. Unknown PAX keys are ignored. PAX payload reads are capped at 65536 bytes to avoid memory exhaustion from malformed archives. + +SharpCompress does not currently write PAX headers. + +## GNU Tar Extensions + +SharpCompress supports the most common GNU extensions needed for interoperability. + +### Long Name and Long Link + +GNU long-name and long-link records are synthetic entries that apply to the next real entry: + +| Typeflag | Purpose | +| -------- | ------- | +| `L` | Long file name/path for next entry | +| `K` | Long link target for next entry | + +The payload contains the long name or link target, padded to a 512-byte boundary. SharpCompress caps long-name payload reads at 32768 bytes. + +Writer behavior in `GNU_TAR_LONG_LINK` mode: + +1. Write a synthetic `././@LongLink` header with `typeflag = 'L'` when the name exceeds 100 bytes. +2. Write the long-name payload and 512-byte padding. +3. Write the actual file or directory header. + +### Sparse Files + +GNU sparse tar uses `typeflag = 'S'` and additional sparse-map metadata. POSIX PAX sparse variants use keys such as `GNU.sparse.*`. + +SharpCompress currently recognizes the sparse entry type enum value but does not reconstruct sparse holes or semantically expose sparse maps. Treat sparse support as unsupported unless implementing full reconstruction and tests. + +### Base-256 Numeric Fields + +GNU tar uses base-256 binary fields for out-of-range numeric values. SharpCompress reads binary `size` fields and writes large sizes in GNU mode. + +## SharpCompress Support Matrix + +Wrapper detection is defined in `TarWrapper.Wrappers`. Detection is content-based: wrapper detection is followed by a tar-header probe of the decompressed payload. + +| Wrapper | Extensions | Read | Write | +| ------- | ---------- | ---- | ----- | +| Plain tar | `tar` | Yes | Yes | +| Tar + GZip | `tar.gz`, `taz`, `tgz` | Yes | Yes | +| Tar + BZip2 | `tar.bz2`, `tb2`, `tbz`, `tbz2`, `tz2` | Yes | Yes | +| Tar + LZip | `tar.lz` | Yes | Yes | +| Tar + XZ | `tar.xz`, `txz` | Yes | No | +| Tar + ZStandard | `tar.zst`, `tar.zstd`, `tzst`, `tzstd` | Yes | No | +| Tar + LZW compress | `tar.Z`, `tZ`, `taZ` | Yes | No | + +Writer support currently accepts only these compression types: + +- `CompressionType.None` +- `CompressionType.GZip` +- `CompressionType.BZip2` +- `CompressionType.LZip` + +Other compression types throw `InvalidFormatException`. + +## SharpCompress Read Behavior + +Reader API: + +- `TarReader` is forward-only and supports non-seekable streams. +- `ReaderFactory.OpenReader` can auto-detect tar and wrapper compression. +- Entry streams must be consumed or disposed so the next header can be aligned. + +Archive API: + +- `TarArchive.OpenArchive(Stream)` and `TarArchive.OpenAsyncArchive(Stream)` require seekable streams. +- File/path overloads own the opened file stream. +- Compressed tar archive access follows streaming semantics over the decompressed stream rather than full random-access semantics. + +Parsed metadata surfaced through entries includes: + +- `Key` +- `LinkTarget` +- `Size` +- `CompressedSize` +- `LastModifiedTime` +- `IsDirectory` +- `Mode` +- `UserID` +- `GroupId` + +Tar entries are always reported as unencrypted and CRC is always `0`. + +## SharpCompress Write Behavior + +`TarWriter` is forward-only. It writes a header, payload, payload padding, and finally two zero blocks on dispose when `FinalizeArchiveOnClose` is true. + +Important writer rules: + +- Tar requires payload size in the header. +- If the source stream is non-seekable and no `size` is supplied, `TarWriter` throws `ArgumentException`. +- `TarWriterOptions.HeaderFormat` defaults to `GNU_TAR_LONG_LINK`. +- Current sync and async file and directory write paths honor `HeaderFormat`. +- `USTAR` mode writes USTAR headers and throws when paths cannot fit. +- `GNU_TAR_LONG_LINK` mode writes GNU long-name records for long paths. +- The public writer supports regular files and directories, not links, devices, FIFOs, sparse maps, or PAX metadata. + +Writer metadata is narrower than reader metadata. It writes name, size, last modified time, and file/directory type, and uses fixed mode/user/group defaults in the header. + +## Known Limitations + +Keep these limitations explicit in code comments, docs, and tests: + +- No PAX write support. +- No sparse-file reconstruction or sparse write support. +- No public API for writing symbolic links or hard links. +- No device or FIFO metadata object model beyond internal type recognition. +- No write support for `tar.xz`, `tar.zst`, or `tar.Z`. +- PAX read support is limited to `path`, `linkpath`, `size`, `mtime`, `uid`, `gid`, and `mode`. +- Unknown PAX keys are ignored. +- Stream-based `TarArchive` open requires seekable input. + +## Test Fixtures + +Representative fixtures in `tests/TestArchives/Archives/`: + +- `Tar.tar` +- `Tar.tar.gz` +- `Tar.tar.bz2` +- `Tar.tar.lz` +- `Tar.tar.xz` +- `Tar.tar.zst` +- `Tar.tar.Z` +- `Tar.oldgnu.tar.gz` +- `very long filename.tar` +- `ustar with long names.tar` +- `Tar.LongPathsWithLongNameExtension.tar` +- `Tar.PaxLocalHeader.tar` +- `Tar.PaxLocalHeader.Link.tar` +- `Tar.PaxGlobalHeader.tar` +- `Tar.PaxGlobalHeader.Link.tar` +- `Tar.Empty.tar` +- `TarCorrupted.tar` +- `TarWithSymlink.tar.gz` + +Representative test files: + +- `tests/SharpCompress.Test/Tar/TarReaderTests.cs` +- `tests/SharpCompress.Test/Tar/TarReaderAsyncTests.cs` +- `tests/SharpCompress.Test/Tar/TarArchiveTests.cs` +- `tests/SharpCompress.Test/Tar/TarArchiveAsyncTests.cs` +- `tests/SharpCompress.Test/Tar/TarWriterTests.cs` +- `tests/SharpCompress.Test/Tar/TarWriterAsyncTests.cs` +- `tests/SharpCompress.Test/Tar/TarWriterDirectoryTests.cs` +- `tests/SharpCompress.Test/Tar/TarArchiveDirectoryTests.cs` From 7569436cdd3fade0954c629344088e7092560cc0 Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Tue, 26 May 2026 10:12:46 +0100 Subject: [PATCH 3/8] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .agents/skills/tar-format/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/tar-format/SKILL.md b/.agents/skills/tar-format/SKILL.md index fbd8ec3f..eb622faa 100644 --- a/.agents/skills/tar-format/SKILL.md +++ b/.agents/skills/tar-format/SKILL.md @@ -1,6 +1,6 @@ --- name: tar-format -description: Reference the Tar/USTAR/PAX/GNU tar archive container format. Use when Codex needs to answer questions or make code changes involving tar headers, 512-byte blocks, checksums, typeflags, USTAR prefixes, PAX local/global extended headers, GNU long names/links, sparse entries, wrapper compression, or SharpCompress Tar parsing and writing behavior. +description: Reference the Tar/USTAR/PAX/GNU tar archive container format. Use when an AI agent needs to answer questions or make code changes involving tar headers, 512-byte blocks, checksums, typeflags, USTAR prefixes, PAX local/global extended headers, GNU long names/links, sparse entries, wrapper compression, or SharpCompress Tar parsing and writing behavior. --- # Tar Format From 55a6daea7ac1084f3dbe1252cc90bf9f98736e6b Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Tue, 26 May 2026 10:13:07 +0100 Subject: [PATCH 4/8] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .agents/skills/sevenzip-format/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/sevenzip-format/SKILL.md b/.agents/skills/sevenzip-format/SKILL.md index 9193a457..20635b38 100644 --- a/.agents/skills/sevenzip-format/SKILL.md +++ b/.agents/skills/sevenzip-format/SKILL.md @@ -1,6 +1,6 @@ --- name: sevenzip-format -description: Reference the 7z/7zip archive container format. Use when Codex needs to answer questions or make code changes involving 7z signatures, headers, encoded headers, NID/property IDs, packed streams, folders/coders, bind pairs, substreams, file metadata properties, or SharpCompress 7Zip parsing behavior. +description: Reference the 7z/7zip archive container format. Use when an AI agent needs to answer questions or make code changes involving 7z signatures, headers, encoded headers, NID/property IDs, packed streams, folders/coders, bind pairs, substreams, file metadata properties, or SharpCompress 7Zip parsing behavior. --- # Sevenzip Format From c49c1f46ef9d82d685e8b8b085b7a0300f639b7f Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Tue, 26 May 2026 15:06:17 +0100 Subject: [PATCH 5/8] Add correct Xz mapping --- src/SharpCompress/Common/Zip/ZipEntry.cs | 1 + tests/SharpCompress.Test/Zip/ZipArchiveTests.cs | 13 +++++++++++++ 2 files changed, 14 insertions(+) diff --git a/src/SharpCompress/Common/Zip/ZipEntry.cs b/src/SharpCompress/Common/Zip/ZipEntry.cs index 47df7d9e..753c6dc7 100644 --- a/src/SharpCompress/Common/Zip/ZipEntry.cs +++ b/src/SharpCompress/Common/Zip/ZipEntry.cs @@ -57,6 +57,7 @@ public class ZipEntry : Entry ZipCompressionMethod.Reduce4 => CompressionType.Reduce4, ZipCompressionMethod.Explode => CompressionType.Explode, ZipCompressionMethod.ZStandard => CompressionType.ZStandard, + ZipCompressionMethod.Xz => CompressionType.Xz, _ => CompressionType.Unknown, }; } diff --git a/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs b/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs index ca80917f..ebee095a 100644 --- a/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs +++ b/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs @@ -100,6 +100,19 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void WinZip27_X_XZ_ArchiveFileRead() => ArchiveFileRead("WinZip27_XZ.zipx"); + [Fact] + public void WinZip27_X_XZ_Reports_CompressionType_Xz() + { + using var archive = ArchiveFactory.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "WinZip27_XZ.zipx") + ); + + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + Assert.Equal(CompressionType.Xz, entry.CompressionType); + } + } + [Fact] public void Zip_Deflate_Streamed2_ArchiveFileRead() => ArchiveFileRead("Zip.deflate.dd-.zip"); From 9bc2a8677bccfd44ba30f3692eb46625f77c31b0 Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Tue, 26 May 2026 15:09:34 +0100 Subject: [PATCH 6/8] update docs to remove LZMA2 from Zip --- docs/FORMATS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/FORMATS.md b/docs/FORMATS.md index 7f543f11..2ac9a83a 100644 --- a/docs/FORMATS.md +++ b/docs/FORMATS.md @@ -14,7 +14,7 @@ | Arc | None, Packed, Squeezed, Crunched | Decompress | N/A | ArcReader | N/A | | Arj | None | Decompress | N/A | ArjReader | N/A | | Rar | Rar | Decompress | RarArchive | RarReader | N/A | -| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA/LZMA2, PPMd, ZStandard, XZ | Both | ZipArchive | ZipReader | ZipWriter | +| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA, PPMd, ZStandard, XZ | Both | ZipArchive | ZipReader | ZipWriter | | Tar | None | Both | TarArchive | TarReader | TarWriter (3) | | Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | | Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | From 35c9cdd23279d5f792b6572ff1c970fd983af9f1 Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Tue, 26 May 2026 15:23:29 +0100 Subject: [PATCH 7/8] add zip skill and upgrade packages --- .agents/skills/zip-format/SKILL.md | 22 + .../zip-format/references/zip-format.md | 456 ++++++++++++++++++ Directory.Packages.props | 4 +- src/SharpCompress/packages.lock.json | 6 +- .../packages.lock.json | 20 +- tests/SharpCompress.Test/packages.lock.json | 67 +-- 6 files changed, 511 insertions(+), 64 deletions(-) create mode 100644 .agents/skills/zip-format/SKILL.md create mode 100644 .agents/skills/zip-format/references/zip-format.md diff --git a/.agents/skills/zip-format/SKILL.md b/.agents/skills/zip-format/SKILL.md new file mode 100644 index 00000000..04bc1e9d --- /dev/null +++ b/.agents/skills/zip-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: zip-format +description: Reference the ZIP/ZIP64/PKWARE APPNOTE archive container format. Use when an AI agent needs to answer questions or make code changes involving ZIP local headers, central directory records, EOCD/Zip64 records, data descriptors, general purpose bit flags, compression method IDs, extra fields, encryption markers, split archives, or SharpCompress Zip parsing and writing behavior. +--- + +# Zip Format + +Use this skill for ZIP container-format work. It provides a local, SharpCompress-oriented reference for PKWARE APPNOTE ZIP records, ZIP64, compression method IDs, extra fields, and the current SharpCompress Zip implementation. + +## Reference + +- Read [references/zip-format.md](references/zip-format.md) when the task depends on ZIP binary layout, record signatures, local vs central directory metadata, data descriptor rules, Zip64 sentinel values, extra field parsing, compression method IDs, encryption markers, split archive handling, or current SharpCompress Zip support boundaries. +- Treat the reference as an implementation guide, not a standards replacement. It summarizes PKWARE APPNOTE 6.3.10 from `https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT` and documents SharpCompress-specific behavior and limitations. +- Prefer the SharpCompress support matrix in the reference over generic ZIP assumptions when changing code. ZIP is highly extensible, and SharpCompress intentionally supports only selected records, methods, and extra fields. + +## Workflow + +1. Identify which layer is involved: header discovery, local file headers, central directory headers, EOCD/Zip64 records, data descriptors, extra fields, compression methods, encryption, reader/archive/writer API behavior, or tests. +2. Open the relevant section in `references/zip-format.md` and use the source-file pointers before changing code. +3. For parsing changes, cross-check sync and async header readers: `ZipHeaderFactory.cs`, `ZipHeaderFactory.Async.cs`, `SeekableZipHeaderFactory.cs`, `StreamingZipHeaderFactory.cs`, `ZipFileEntry.cs`, `LocalEntryHeader.cs`, and `DirectoryEntryHeader.cs`. +4. For writer changes, verify local header, post-data descriptor, central directory, Zip64, and sync/async paths together. +5. For support claims, keep unsupported features explicit: central directory encryption, strong encryption records, XZ writing, non-standard compression methods, broad extra-field semantics, and non-seekable Zip64 writing. diff --git a/.agents/skills/zip-format/references/zip-format.md b/.agents/skills/zip-format/references/zip-format.md new file mode 100644 index 00000000..defafa56 --- /dev/null +++ b/.agents/skills/zip-format/references/zip-format.md @@ -0,0 +1,456 @@ +# ZIP Format Reference + +This reference summarizes the ZIP archive container format for SharpCompress work. It is locally authored from PKWARE APPNOTE and the current SharpCompress implementation. + +Primary external reference: + +- PKWARE APPNOTE.TXT - ZIP File Format Specification, version 6.3.10: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT + +Primary SharpCompress references: + +- `docs/FORMATS.md` +- `src/SharpCompress/Factories/ZipFactory.cs` +- `src/SharpCompress/Archives/Zip/ZipArchive.cs` +- `src/SharpCompress/Readers/Zip/ZipReader.cs` +- `src/SharpCompress/Writers/Zip/ZipWriter.cs` +- `src/SharpCompress/Writers/Zip/ZipWritingStream.cs` +- `src/SharpCompress/Writers/Zip/ZipCentralDirectoryEntry.cs` +- `src/SharpCompress/Common/Zip/ZipCompressionMethod.cs` +- `src/SharpCompress/Common/Zip/ZipEntry.cs` +- `src/SharpCompress/Common/Zip/ZipFilePart.cs` +- `src/SharpCompress/Common/Zip/ZipHeaderFactory.cs` +- `src/SharpCompress/Common/Zip/ZipHeaderFactory.Async.cs` +- `src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.cs` +- `src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.Async.cs` +- `src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.cs` +- `src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.Async.cs` +- `src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs` +- `src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.cs` +- `src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.cs` +- `src/SharpCompress/Common/Zip/Headers/LocalEntryHeaderExtraFactory.cs` +- `tests/SharpCompress.Test/Zip/` + +## Contents + +- [Format Overview](#format-overview) +- [Record Signatures](#record-signatures) +- [Local File Header](#local-file-header) +- [Central Directory Header](#central-directory-header) +- [End Of Central Directory](#end-of-central-directory) +- [Data Descriptors](#data-descriptors) +- [General Purpose Bit Flags](#general-purpose-bit-flags) +- [Compression Methods](#compression-methods) +- [Extra Fields](#extra-fields) +- [Zip64](#zip64) +- [Names, Comments, And Encoding](#names-comments-and-encoding) +- [Encryption](#encryption) +- [Seekable And Streaming Reads](#seekable-and-streaming-reads) +- [SharpCompress Support Matrix](#sharpcompress-support-matrix) +- [SharpCompress Write Behavior](#sharpcompress-write-behavior) +- [Known Limitations](#known-limitations) +- [Test Fixtures](#test-fixtures) + +## Format Overview + +A ZIP archive stores each file as a local file record followed by compressed or stored payload bytes. Metadata is repeated in a central directory near the end of the archive, followed by the end of central directory record. + +High-level APPNOTE layout: + +```text +[local file header 1] +[encryption header 1] +[file data 1] +[data descriptor 1] +... +[local file header n] +[encryption header n] +[file data n] +[data descriptor n] +[archive decryption header] +[archive extra data record] +[central directory header 1] +... +[central directory header n] +[zip64 end of central directory record] +[zip64 end of central directory locator] +[end of central directory record] +``` + +All ordinary multi-byte ZIP fields are little-endian unless APPNOTE says otherwise. ZIP readers must identify records by signatures rather than by extension. + +SharpCompress has two important read modes: + +- Seekable Archive API reads the central directory first and then seeks to local headers for entry data. +- Streaming Reader API processes local headers and payloads in order and cannot rely on central directory data unless it is already supplied by the caller/path flow. + +## Record Signatures + +Common ZIP signatures used by SharpCompress: + +| Record | Signature | SharpCompress constant | +| --- | --- | --- | +| Local file header | `0x04034b50` | `ENTRY_HEADER_BYTES` | +| Data descriptor | `0x08074b50` | `POST_DATA_DESCRIPTOR` | +| Central directory file header | `0x02014b50` | `DIRECTORY_START_HEADER_BYTES` | +| End of central directory | `0x06054b50` | `DIRECTORY_END_HEADER_BYTES` | +| Digital signature | `0x05054b50` | `DIGITAL_SIGNATURE` | +| Split archive marker | `0x30304b50` | `SPLIT_ARCHIVE_HEADER_BYTES` | +| Zip64 end of central directory | `0x06064b50` | `ZIP64_END_OF_CENTRAL_DIRECTORY` | +| Zip64 end of central directory locator | `0x07064b50` | `ZIP64_END_OF_CENTRAL_DIRECTORY_LOCATOR` | + +`ZipHeaderFactory.IsHeader` recognizes these signatures while streaming. + +## Local File Header + +The local file header immediately precedes optional encryption metadata and file data. + +| Field | Size | Notes | +| --- | --- | --- | +| Signature | 4 | `0x04034b50` | +| Version needed to extract | 2 | Feature/version indicator | +| General purpose bit flag | 2 | Encryption, data descriptor, EFS, method-specific bits | +| Compression method | 2 | See [Compression Methods](#compression-methods) | +| Last mod file time | 2 | MS-DOS time | +| Last mod file date | 2 | MS-DOS date | +| CRC-32 | 4 | Zero when bit 3 defers values to data descriptor | +| Compressed size | 4 | `0xffffffff` sentinel when Zip64 extra carries value | +| Uncompressed size | 4 | `0xffffffff` sentinel when Zip64 extra carries value | +| File name length | 2 | Byte count | +| Extra field length | 2 | Byte count | +| File name | variable | Not NUL-terminated | +| Extra field | variable | Sequence of ID/length/data blocks | + +SharpCompress reads this in `LocalEntryHeader.Read` and then decodes names, loads extra fields, applies Unicode path extra data, applies Zip64 extra data, and applies selected Unix time data. + +## Central Directory Header + +The central directory file header repeats entry metadata and adds fields needed for random access. + +| Field | Size | Notes | +| --- | --- | --- | +| Signature | 4 | `0x02014b50` | +| Version made by | 2 | Host/version metadata | +| Version needed to extract | 2 | Feature/version indicator | +| General purpose bit flag | 2 | Same semantic space as local header | +| Compression method | 2 | See [Compression Methods](#compression-methods) | +| Last mod file time | 2 | MS-DOS time | +| Last mod file date | 2 | MS-DOS date | +| CRC-32 | 4 | Entry CRC | +| Compressed size | 4 | Zip64 sentinel possible | +| Uncompressed size | 4 | Zip64 sentinel possible | +| File name length | 2 | Byte count | +| Extra field length | 2 | Byte count | +| File comment length | 2 | Byte count | +| Disk number start | 2 | Zip64 sentinel possible | +| Internal file attributes | 2 | Host/application metadata | +| External file attributes | 4 | Host-dependent file attributes | +| Relative offset of local header | 4 | Zip64 sentinel possible | +| File name | variable | Not NUL-terminated | +| Extra field | variable | Sequence of ID/length/data blocks | +| File comment | variable | Not NUL-terminated | + +SharpCompress reads this in `DirectoryEntryHeader.Read`. Seekable ZIP reads use central directory entries to discover the archive contents, then `SeekableZipHeaderFactory.GetLocalHeader` seeks to local headers and copies central-directory-only metadata onto the local entry. + +## End Of Central Directory + +Every normal ZIP archive ends with exactly one EOCD record. The minimum EOCD length is 22 bytes, plus an optional ZIP file comment up to 65535 bytes. + +EOCD fields: + +| Field | Size | Notes | +| --- | --- | --- | +| Signature | 4 | `0x06054b50` | +| Number of this disk | 2 | Split/spanned metadata | +| Central directory start disk | 2 | Split/spanned metadata | +| Entries on this disk | 2 | `0xffff` sentinel when Zip64 is needed | +| Total entries | 2 | `0xffff` sentinel when Zip64 is needed | +| Central directory size | 4 | `0xffffffff` sentinel when Zip64 is needed | +| Central directory offset | 4 | `0xffffffff` sentinel when Zip64 is needed | +| ZIP file comment length | 2 | Byte count | +| ZIP file comment | variable | Archive-level comment | + +`SeekableZipHeaderFactory.SeekBackToHeader` searches backwards from the end of the stream across the maximum EOCD/comment search window. + +## Data Descriptors + +When general purpose bit 3 is set, the local header CRC and size fields are placeholders. The actual values follow file data in a data descriptor. + +Descriptor forms: + +```text +[optional signature 0x08074b50] +crc-32 4 bytes +compressed size 4 or 8 bytes +uncompressed size 4 or 8 bytes +``` + +APPNOTE says the signature was not originally assigned but is commonly used. SharpCompress handles descriptors with and without the signature in streaming reads. + +Important SharpCompress behavior: + +- `StreamingZipHeaderFactory` reads post-data descriptors after the previous entry stream has been consumed. +- Streaming descriptor parsing has compatibility logic for 32-bit and 64-bit sizes. +- `ZipWriter` writes a descriptor with signature for non-seekable output when Zip64 is not required. +- `ZipWriter` intentionally rejects Zip64 on non-seekable streams. + +## General Purpose Bit Flags + +Selected flags relevant to SharpCompress: + +| Bit | Meaning | +| --- | --- | +| 0 | Entry is encrypted | +| 1 | Method-specific. For LZMA method 14, set means EOS marker is used. For Implode it has dictionary-size meaning. | +| 2 | Method-specific. For Deflate/Deflate64 it encodes compression option; for Implode it has tree-count meaning. | +| 3 | CRC and sizes are deferred to a data descriptor after file data | +| 6 | Strong encryption | +| 11 | Language encoding flag (EFS): file name and comment are UTF-8 | +| 13 | Central directory encryption masks selected local header values | + +SharpCompress decodes names/comments as UTF-8 when EFS is set. For ZIP LZMA writing on non-seekable output, it sets bit 1 for EOS marker behavior. + +## Compression Methods + +APPNOTE compression method IDs relevant to SharpCompress and nearby unsupported methods: + +| ID | APPNOTE method | SharpCompress status | +| --- | --- | --- | +| 0 | Stored | Read/write | +| 1 | Shrunk | Read | +| 2 | Reduced factor 1 | Read | +| 3 | Reduced factor 2 | Read | +| 4 | Reduced factor 3 | Read | +| 5 | Reduced factor 4 | Read | +| 6 | Imploded | Read | +| 8 | Deflated | Read/write | +| 9 | Deflate64 | Read | +| 12 | BZIP2 | Read/write | +| 14 | LZMA | Read/write | +| 93 | Zstandard | Read/write | +| 95 | XZ | Read | +| 98 | PPMd version I, Rev 1 | Read/write | +| 99 | AE-x encryption marker | Read for WinZip AES handling | + +SharpCompress declares these in `ZipCompressionMethod.cs`: + +```text +None = 0 +Shrink = 1 +Reduce1 = 2 +Reduce2 = 3 +Reduce3 = 4 +Reduce4 = 5 +Explode = 6 +Deflate = 8 +Deflate64 = 9 +BZip2 = 12 +LZMA = 14 +ZStandard = 93 +Xz = 95 +PPMd = 98 +WinzipAes = 0x63 +``` + +ZIP has method 14 for LZMA and method 95 for XZ. There is no separate APPNOTE ZIP compression method named LZMA2 in the method table. XZ commonly uses LZMA2 internally, but a ZIP entry using APPNOTE method 95 is an XZ-compressed ZIP entry, not a separate LZMA2 ZIP method. + +`ZipFilePart.ToCompressionType` maps supported read methods to public `CompressionType` values and throws for unsupported methods before decompression. `ZipEntry.CompressionType` reports the public entry compression type, including `CompressionType.Xz` for method 95. + +## Extra Fields + +APPNOTE extra fields use this generic structure: + +```text +header id 2 bytes +data size 2 bytes +data variable +``` + +SharpCompress parses extra fields in `ZipFileEntry.LoadExtra`. Unknown or unsupported extra fields become `NotImplementedExtraData` and are preserved only as raw data for internal parsing decisions. + +Recognized extra fields: + +| Header ID | Meaning | SharpCompress type | +| --- | --- | --- | +| `0x0001` | Zip64 extended information | `Zip64ExtendedInformationExtraField` | +| `0x5455` | Extended timestamp / Unix time | `UnixTimeExtraField` | +| `0x7075` | Info-ZIP Unicode path | `ExtraUnicodePathExtraField` | +| `0x9901` | WinZip AES | Raw `ExtraData` used by AES logic | + +Zip64 extra values appear only when the corresponding 16-bit or 32-bit field in the local or central directory is set to its maximum sentinel. Values must appear in APPNOTE order: + +1. Original/uncompressed size +2. Compressed size +3. Relative header offset +4. Disk start number + +`Zip64ExtendedInformationExtraField.Process` enforces the required byte count for the sentinel fields being resolved. + +## Zip64 + +Zip64 extends size, count, and offset fields beyond classic ZIP limits. Classic fields use sentinel values when the real value is stored elsewhere: + +| Classic field size | Sentinel | +| --- | --- | +| 2 bytes | `0xffff` | +| 4 bytes | `0xffffffff` | + +Zip64 structures: + +- Zip64 extended information extra field (`0x0001`) carries per-entry sizes and offsets. +- Zip64 end of central directory record (`0x06064b50`) carries archive-level counts, central directory size, and central directory offset. +- Zip64 end of central directory locator (`0x07064b50`) points to the Zip64 EOCD record. + +SharpCompress read behavior: + +- Seekable reads detect Zip64 through EOCD sentinel values and then locate the Zip64 EOCD locator and record. +- Local and central entry readers apply Zip64 extra data when size/offset fields contain sentinels. + +SharpCompress write behavior: + +- `ZipWriterOptions.UseZip64` controls whether local headers reserve Zip64 extra data for entries. +- `ZipWriter` emits Zip64 EOCD and locator when entry counts, central directory size, or offsets require them. +- Non-seekable Zip64 writing is rejected because current post-data descriptor handling cannot safely represent the required Zip64 values in all cases. + +## Names, Comments, And Encoding + +APPNOTE file name and comment fields are length-prefixed byte sequences, not NUL-terminated strings. + +Rules relevant to SharpCompress: + +- If general purpose bit 11 (EFS) is set, file names and comments must be UTF-8. +- If EFS is not set, SharpCompress uses the configured archive encoding. +- If Info-ZIP Unicode path extra field `0x7075` is present and the caller did not force an encoding, SharpCompress uses the Unicode name from that extra field. +- `ZipWriter.NormalizeFilename` converts backslashes to `/`, removes drive prefixes before `:`, and trims leading/trailing `/` for file entries. +- `ZipWriter` ensures directory entries end with `/`. + +## Encryption + +APPNOTE defines traditional PKWARE encryption, strong encryption features, central directory encryption, and method 99 as an AE-x encryption marker. + +SharpCompress behavior: + +- Traditional PKWARE-encrypted ZIP entries can be read when a password is supplied. +- WinZip AES entries use method 99 (`WinzipAes`) and extra field `0x9901`; SharpCompress reads the actual compression method from that extra data. +- `ZipHeaderFactory.LoadHeader` rejects encrypted ZIP data that requires unsupported non-seekable handling. +- Central directory encryption and broad strong encryption records are not general-purpose supported features. Keep this explicit in support claims. + +## Seekable And Streaming Reads + +Seekable read path: + +- `SeekableZipHeaderFactory` searches backward for EOCD. +- If EOCD indicates Zip64, it reads the Zip64 locator and Zip64 EOCD. +- It iterates central directory file headers. +- `GetLocalHeader` seeks to each entry's local header when entry data is needed. + +Streaming read path: + +- `StreamingZipHeaderFactory` reads local headers in archive order. +- Entry data must be consumed or skipped before the next header can be parsed. +- If bit 3 is set, descriptor values are read after entry data. +- Directory entries may be inferred from names ending in `/`, or from zero-size names ending in `\` for older .NET-produced archives. + +Archive API vs Reader API: + +- `ZipArchive` is appropriate for seekable streams and multi-volume/split archives. +- `ZipReader` is forward-only and does not seek across volume files. + +## SharpCompress Support Matrix + +Current ZIP support summary from `docs/FORMATS.md` and implementation: + +| Feature | Read | Write | Notes | +| --- | --- | --- | --- | +| Stored | Yes | Yes | Method 0 | +| Deflate | Yes | Yes | Method 8 | +| Deflate64 | Yes | No | Method 9 | +| BZip2 | Yes | Yes | Method 12 | +| LZMA | Yes | Yes | Method 14 | +| PPMd | Yes | Yes | Method 98 | +| ZStandard | Yes | Yes | Method 93 | +| XZ | Yes | No | Method 95 | +| Shrink | Yes | No | Legacy method 1 | +| Reduce | Yes | No | Legacy methods 2-5 | +| Implode | Yes | No | Legacy method 6 | +| Zip64 | Yes | Yes | Writing requires seekable stream for Zip64 | +| Data descriptors | Yes | Yes | Writer uses them for non-seekable non-Zip64 output | +| Traditional PKWARE encryption | Yes | No | Password required | +| WinZip AES | Yes | No | Method 99 + extra field `0x9901` | +| Split/multi-volume ZIP | Yes | No | Use Archive API | + +## SharpCompress Write Behavior + +`ZipWriter` is forward-only for entry creation but writes a central directory on dispose. It tracks local header offsets and entry sizes as data is written. + +Writer compression mapping in `ZipWriter.ToZipCompressionMethod` currently accepts: + +- `CompressionType.None` +- `CompressionType.Deflate` +- `CompressionType.BZip2` +- `CompressionType.LZMA` +- `CompressionType.PPMd` +- `CompressionType.ZStandard` + +Other compression types throw `InvalidFormatException`, including `CompressionType.Xz`. + +Important writer rules: + +- Local headers are written before payload data. +- Seekable output lets SharpCompress patch CRC and size fields after compression. +- Non-seekable output uses post-data descriptors for non-Zip64 entries. +- Zip64 on non-seekable output is rejected. +- Zero-byte file entries are normalized to stored/no compression in the central directory. +- Central directory entries are written on `Dispose` / `DisposeAsync`; callers must dispose writers to finalize archives. + +## Known Limitations + +Keep these limitations explicit in code comments, docs, and tests: + +- No separate ZIP LZMA2 method support. APPNOTE method 95 is XZ; XZ may use LZMA2 internally. +- No XZ writing for ZIP entries. +- No Deflate64, Shrink, Reduce, or Implode writing. +- No general-purpose central directory encryption support. +- No broad APPNOTE strong encryption record support beyond current traditional PKWARE and WinZip AES read paths. +- No non-seekable Zip64 writing. +- Unknown extra fields are not semantically modeled unless explicitly implemented. +- ZipReader cannot seek across multi-volume/split archive parts; use ZipArchive for split archives. + +## Test Fixtures + +Representative fixtures in `tests/TestArchives/Archives/`: + +- `Zip.none.zip` +- `Zip.deflate.zip` +- `Zip.deflate.dd.zip` +- `Zip.deflate64.zip` +- `Zip.bzip2.zip` +- `Zip.lzma.zip` +- `Zip.lzma.dd.zip` +- `Zip.ppmd.zip` +- `Zip.shrink.zip` +- `Zip.reduce1.zip` +- `Zip.reduce2.zip` +- `Zip.reduce3.zip` +- `Zip.reduce4.zip` +- `Zip.implode.zip` +- `Zip.zip64.zip` +- `Zip.zstd.WinzipAES.mixed.zip` +- `WinZip27_XZ.zipx` +- `WinZip27_ZSTD.zipx` +- `WinZip26.nocomp.multi.zip` +- `WinZip26.nocomp.multi.zipx` +- `Zip.UnicodePathExtra.zip` +- `Zip.EntryComment.zip` + +Representative test files: + +- `tests/SharpCompress.Test/Zip/ZipArchiveTests.cs` +- `tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs` +- `tests/SharpCompress.Test/Zip/ZipReaderTests.cs` +- `tests/SharpCompress.Test/Zip/ZipReaderAsyncTests.cs` +- `tests/SharpCompress.Test/Zip/ZipWriterTests.cs` +- `tests/SharpCompress.Test/Zip/ZipWriterAsyncTests.cs` +- `tests/SharpCompress.Test/Zip/Zip64Tests.cs` +- `tests/SharpCompress.Test/Zip/Zip64AsyncTests.cs` +- `tests/SharpCompress.Test/Zip/Zip64VersionConsistencyTests.cs` +- `tests/SharpCompress.Test/Zip/ZipFilePartTests.cs` diff --git a/Directory.Packages.props b/Directory.Packages.props index ca0f5260..7ed85889 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -4,9 +4,9 @@ - + - + diff --git a/src/SharpCompress/packages.lock.json b/src/SharpCompress/packages.lock.json index 965d59a8..738df599 100644 --- a/src/SharpCompress/packages.lock.json +++ b/src/SharpCompress/packages.lock.json @@ -286,9 +286,9 @@ "net10.0": { "Microsoft.NET.ILLink.Tasks": { "type": "Direct", - "requested": "[10.0.7, )", - "resolved": "10.0.7", - "contentHash": "AA/yhzFHNtQZXLdqjzujPy25G8EWwGWsAnxOE2zYSBoT/8QHP6ketN3CToD3DFreO653ipUwnKHo22B8AlBMCw==" + "requested": "[10.0.6, )", + "resolved": "10.0.6", + "contentHash": "QKuvS0LWX4fjFqeDkyM7Kqt8P3wYTiPD4nwU+9y59n0sCiG714fxDgbbN82vDnzq89AF/PiHl92TP2C4aFDUQA==" }, "Microsoft.NETFramework.ReferenceAssemblies": { "type": "Direct", diff --git a/tests/SharpCompress.Performance/packages.lock.json b/tests/SharpCompress.Performance/packages.lock.json index 35d18d20..4e1d3a33 100644 --- a/tests/SharpCompress.Performance/packages.lock.json +++ b/tests/SharpCompress.Performance/packages.lock.json @@ -22,12 +22,12 @@ }, "JetBrains.Profiler.SelfApi": { "type": "Direct", - "requested": "[2.5.16, )", - "resolved": "2.5.16", - "contentHash": "F8Tczi426PmDNZu0d88d60jVEQ/Tnvdi7LKnVLikTQ9hPqdzW7WOjj79S4XA2awbpTfCCU+eHL9MrUk6Yj90Zw==", + "requested": "[2.5.18, )", + "resolved": "2.5.18", + "contentHash": "zOHtZGrLzYey6d57XLvLUWbB7tK1WbXz2z3wSwDR6HwV5AEROmtu1di3OZE1kxGXMilPbyzXyKJT5eBwC+j32Q==", "dependencies": { - "JetBrains.HabitatDetector": "1.4.6", - "JetBrains.Profiler.Api": "1.4.11" + "JetBrains.HabitatDetector": "1.5.0", + "JetBrains.Profiler.Api": "1.4.13" } }, "Microsoft.NETFramework.ReferenceAssemblies": { @@ -88,18 +88,18 @@ }, "JetBrains.HabitatDetector": { "type": "Transitive", - "resolved": "1.4.6", - "contentHash": "kNN79C3hIt+1AzPNKbddouXXGaL84bi2T42Zsa2A0nXyatg1V5q+WLI9NddCtWjMdb9CLkDem2VIc/5lHWaXJQ==", + "resolved": "1.5.0", + "contentHash": "GazZUoCunH1vrruUvy147lYfgcm2Ns6MCh76XmKZBViosvvYMLyt+JD2PWL8Lm6Ctdo3iQoPYgkStrxhrHbjEQ==", "dependencies": { "JetBrains.FormatRipper": "2.4.0" } }, "JetBrains.Profiler.Api": { "type": "Transitive", - "resolved": "1.4.11", - "contentHash": "4XOA72p49dmzT9k2NyhSVYnLxjw+OPJpXen2gNFsbWquZPjnpoPqXJosVfeONDOST52cgAg3c1q/cBa6/R3NLQ==", + "resolved": "1.4.13", + "contentHash": "abkSB+SOgLMPfOGJtl2XQ+ioSUYBtZxiKDOGCpNZ4WU7ZbAzhldrpRwDMsQnIJJcNhEawGtbhcI+Gk3ktx2RXQ==", "dependencies": { - "JetBrains.HabitatDetector": "1.4.6" + "JetBrains.HabitatDetector": "1.5.0" } }, "Microsoft.Build.Tasks.Git": { diff --git a/tests/SharpCompress.Test/packages.lock.json b/tests/SharpCompress.Test/packages.lock.json index 136bb015..9f666ea6 100644 --- a/tests/SharpCompress.Test/packages.lock.json +++ b/tests/SharpCompress.Test/packages.lock.json @@ -13,11 +13,11 @@ }, "Microsoft.NET.Test.Sdk": { "type": "Direct", - "requested": "[18.3.0, )", - "resolved": "18.3.0", - "contentHash": "xW3kXuWRQtgoxJp4J+gdhHSQyK+6Wb/AZDSd7lMvuMRYlZ1tnpkojyfZlWilB5G4dmZ0Y0ZxU/M23TlubndNkw==", + "requested": "[18.5.1, )", + "resolved": "18.5.1", + "contentHash": "SfqVaLiIqAbRWuPg5BP4QFwBIirQj/YIL8Dhxl6zntBKbXp0cQykoV480SmwG+yRMiWptxEI6NbHQuGSZ8b97w==", "dependencies": { - "Microsoft.CodeCoverage": "18.3.0" + "Microsoft.CodeCoverage": "18.5.1" } }, "Microsoft.NETFramework.ReferenceAssemblies": { @@ -84,8 +84,8 @@ }, "Microsoft.CodeCoverage": { "type": "Transitive", - "resolved": "18.3.0", - "contentHash": "23BNy/vziREC20Wwhb50K7+kZe0m07KlLWDQv4qjJ9tt3QjpDpDIqJFrhYHmMEo9xDkuSp55U/8h4bMF7MiB+g==" + "resolved": "18.5.1", + "contentHash": "vMFDR1ZjqzzgKmM0zrPie7Gv9Y+ZppjODB5Quzu9Eq0TlIusUfUCYFPEawO91zQuqwzvdFbJSU7WHNtjStffJQ==" }, "Microsoft.NETFramework.ReferenceAssemblies.net48": { "type": "Transitive", @@ -315,30 +315,6 @@ } } }, - ".NETFramework,Version=v4.8/win-x86": { - "Microsoft.Win32.Registry": { - "type": "Transitive", - "resolved": "5.0.0", - "contentHash": "dDoKi0PnDz31yAyETfRntsLArTlVAVzUzCIvvEDsDsucrl33Dl8pIJG06ePTJTI3tGpeyHS9Cq7Foc/s4EeKcg==", - "dependencies": { - "System.Security.AccessControl": "5.0.0", - "System.Security.Principal.Windows": "5.0.0" - } - }, - "System.Security.AccessControl": { - "type": "Transitive", - "resolved": "5.0.0", - "contentHash": "dagJ1mHZO3Ani8GH0PHpPEe/oYO+rVdbQjvjJkBRNQkX4t0r1iaeGn8+/ybkSLEan3/slM0t59SVdHzuHf2jmw==", - "dependencies": { - "System.Security.Principal.Windows": "5.0.0" - } - }, - "System.Security.Principal.Windows": { - "type": "Transitive", - "resolved": "5.0.0", - "contentHash": "t0MGLukB5WAVU9bO3MGzvlGnyJPgUlcwerXn1kzBRjwLKixT96XV0Uza41W49gVd8zEMFu9vQEFlv0IOrytICA==" - } - }, "net10.0": { "AwesomeAssertions": { "type": "Direct", @@ -348,12 +324,12 @@ }, "Microsoft.NET.Test.Sdk": { "type": "Direct", - "requested": "[18.3.0, )", - "resolved": "18.3.0", - "contentHash": "xW3kXuWRQtgoxJp4J+gdhHSQyK+6Wb/AZDSd7lMvuMRYlZ1tnpkojyfZlWilB5G4dmZ0Y0ZxU/M23TlubndNkw==", + "requested": "[18.5.1, )", + "resolved": "18.5.1", + "contentHash": "SfqVaLiIqAbRWuPg5BP4QFwBIirQj/YIL8Dhxl6zntBKbXp0cQykoV480SmwG+yRMiWptxEI6NbHQuGSZ8b97w==", "dependencies": { - "Microsoft.CodeCoverage": "18.3.0", - "Microsoft.TestPlatform.TestHost": "18.3.0" + "Microsoft.CodeCoverage": "18.5.1", + "Microsoft.TestPlatform.TestHost": "18.5.1" } }, "Microsoft.NETFramework.ReferenceAssemblies": { @@ -414,8 +390,8 @@ }, "Microsoft.CodeCoverage": { "type": "Transitive", - "resolved": "18.3.0", - "contentHash": "23BNy/vziREC20Wwhb50K7+kZe0m07KlLWDQv4qjJ9tt3QjpDpDIqJFrhYHmMEo9xDkuSp55U/8h4bMF7MiB+g==" + "resolved": "18.5.1", + "contentHash": "vMFDR1ZjqzzgKmM0zrPie7Gv9Y+ZppjODB5Quzu9Eq0TlIusUfUCYFPEawO91zQuqwzvdFbJSU7WHNtjStffJQ==" }, "Microsoft.NETFramework.ReferenceAssemblies.net461": { "type": "Transitive", @@ -459,15 +435,15 @@ }, "Microsoft.TestPlatform.ObjectModel": { "type": "Transitive", - "resolved": "18.3.0", - "contentHash": "AEIEX2aWdPO9XbtR96eBaJxmXRD9vaI9uQ1T/JbPEKlTAZwYx0ZrMzKyULMdh/HH9Sg03kXCoN7LszQ90o6nPQ==" + "resolved": "18.5.1", + "contentHash": "KNZd+M0S0rz5eNAln0pbZX+A/RbokYZCbGKx4fN4CkhtWhkz6nSJDO+9LGYjRE4d0WPVriJ2JnVubkjt3+PpMg==" }, "Microsoft.TestPlatform.TestHost": { "type": "Transitive", - "resolved": "18.3.0", - "contentHash": "twmsoelXnp1uWMU3VGip9f0Jr1mZ0PZqgJdF35CIrdYgYrkHIJMV1m8uKyhcdjLdsQDESHAgkR7KhS9i1qpJag==", + "resolved": "18.5.1", + "contentHash": "RM+3JNHEoHOCFXzVntUcIiYxzPjzBN0N8wto6HYXi76YyBTZ/3CeRL8U+Pk5zx3AUrOmHxDvKJwGUCdElU9bJg==", "dependencies": { - "Microsoft.TestPlatform.ObjectModel": "18.3.0", + "Microsoft.TestPlatform.ObjectModel": "18.5.1", "Newtonsoft.Json": "13.0.3" } }, @@ -557,13 +533,6 @@ "resolved": "8.0.0", "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==" } - }, - "net10.0/win-x86": { - "Microsoft.Win32.Registry": { - "type": "Transitive", - "resolved": "5.0.0", - "contentHash": "dDoKi0PnDz31yAyETfRntsLArTlVAVzUzCIvvEDsDsucrl33Dl8pIJG06ePTJTI3tGpeyHS9Cq7Foc/s4EeKcg==" - } } } } \ No newline at end of file From 4334078d4a5387709c561ae50893993cc475b794 Mon Sep 17 00:00:00 2001 From: Adam Hathcock Date: Tue, 26 May 2026 15:31:25 +0100 Subject: [PATCH 8/8] add rar skill --- .agents/skills/rar-format/SKILL.md | 22 + .../rar-format/references/rar-format.md | 459 ++++++++++++++++++ 2 files changed, 481 insertions(+) create mode 100644 .agents/skills/rar-format/SKILL.md create mode 100644 .agents/skills/rar-format/references/rar-format.md diff --git a/.agents/skills/rar-format/SKILL.md b/.agents/skills/rar-format/SKILL.md new file mode 100644 index 00000000..b5b91d58 --- /dev/null +++ b/.agents/skills/rar-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: rar-format +description: Reference the RAR/RAR5 archive container format and UnRAR source behavior. Use when an AI agent needs to answer questions or make code changes involving RAR signatures, RAR5 vint fields, block headers, file/service headers, extra records, solid archives, multivolume parts, encryption headers, recovery records, quick-open data, or SharpCompress Rar parsing and extraction behavior. +--- + +# Rar Format + +Use this skill for RAR container-format work. It provides a local, SharpCompress-oriented reference for RAR 5.0 archive blocks, older RAR header compatibility, UnRAR source behavior, and current SharpCompress Rar implementation boundaries. + +## Reference + +- Read [references/rar-format.md](references/rar-format.md) when the task depends on RAR binary layout, RAR4 vs RAR5 signatures, RAR5 vint encoding, block/header flags, file and service header fields, extra records, encrypted headers, solid archives, split volumes, redirection records, or current SharpCompress Rar support boundaries. +- Treat the reference as an implementation guide, not a standards replacement. It summarizes `reference/RAR 5.0 archive format.htm` and selected files under `reference/unrar/`, especially `headers5.hpp`, `headers.hpp`, `arcread.cpp`, `rawread.*`, `crypt*.cpp`, `qopen.*`, `volume.*`, `recvol*.cpp`, `unpack*.cpp`, and `blake2sp.*`. +- Prefer the SharpCompress support matrix in the reference over generic RAR assumptions when changing code. RAR is a read-only format in SharpCompress, and metadata/service-header support is intentionally partial. + +## Workflow + +1. Identify which layer is involved: signature detection, RAR4 headers, RAR5 headers, vint parsing, block flags, extra records, file/service metadata, encrypted headers, multivolume handling, decompression, reader/archive API behavior, or tests. +2. Open the relevant section in `references/rar-format.md` and use the source-file pointers before changing code. +3. For header parsing changes, cross-check sync and async implementations: `MarkHeader.cs`, `RarHeader.cs`, `RarHeaderFactory.cs`, `FileHeader.cs`, `Flags.cs`, and their async counterparts. +4. For extraction changes, verify split and solid archive behavior with `RarArchive`, `RarReader`, `RarStream`, and the unpacker files under `src/SharpCompress/Compressors/Rar/`. +5. For support claims, keep unsupported or partial features explicit: RAR writing, pre-RAR4 archives, complete service-header semantics, quick-open use, recovery reconstruction, and encryption/version constraints. diff --git a/.agents/skills/rar-format/references/rar-format.md b/.agents/skills/rar-format/references/rar-format.md new file mode 100644 index 00000000..47fd01fa --- /dev/null +++ b/.agents/skills/rar-format/references/rar-format.md @@ -0,0 +1,459 @@ +# RAR Format Reference + +This reference summarizes the RAR archive container format for SharpCompress work. It is locally authored from the RAR 5.0 format document, UnRAR source code, and the current SharpCompress implementation. + +Primary local references: + +- `reference/RAR 5.0 archive format.htm` +- `reference/unrar/headers5.hpp` +- `reference/unrar/headers.hpp` +- `reference/unrar/arcread.cpp` +- `reference/unrar/rawread.cpp` +- `reference/unrar/rawread.hpp` +- `reference/unrar/crypt.cpp` +- `reference/unrar/crypt5.cpp` +- `reference/unrar/qopen.cpp` +- `reference/unrar/qopen.hpp` +- `reference/unrar/volume.cpp` +- `reference/unrar/volume.hpp` +- `reference/unrar/recvol.cpp` +- `reference/unrar/recvol5.cpp` +- `reference/unrar/unpack.cpp` +- `reference/unrar/unpack15.cpp` +- `reference/unrar/unpack20.cpp` +- `reference/unrar/unpack30.cpp` +- `reference/unrar/unpack50.cpp` +- `reference/unrar/blake2sp.cpp` +- `reference/unrar/blake2sp.hpp` + +Primary SharpCompress references: + +- `docs/FORMATS.md` +- `src/SharpCompress/Factories/RarFactory.cs` +- `src/SharpCompress/Archives/Rar/RarArchive.cs` +- `src/SharpCompress/Archives/Rar/RarArchiveEntry.cs` +- `src/SharpCompress/Archives/Rar/RarArchiveVolumeFactory.cs` +- `src/SharpCompress/Readers/Rar/RarReader.cs` +- `src/SharpCompress/Readers/Rar/MultiVolumeRarReader.cs` +- `src/SharpCompress/Common/Rar/RarEntry.cs` +- `src/SharpCompress/Common/Rar/RarFilePart.cs` +- `src/SharpCompress/Common/Rar/Rar5CryptoInfo.cs` +- `src/SharpCompress/Common/Rar/RarCryptoBinaryReader.cs` +- `src/SharpCompress/Common/Rar/RarCryptoWrapper.cs` +- `src/SharpCompress/Common/Rar/Headers/MarkHeader.cs` +- `src/SharpCompress/Common/Rar/Headers/RarHeader.cs` +- `src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.cs` +- `src/SharpCompress/Common/Rar/Headers/FileHeader.cs` +- `src/SharpCompress/Common/Rar/Headers/Flags.cs` +- `src/SharpCompress/Compressors/Rar/` +- `tests/SharpCompress.Test/Rar/` + +## Contents + +- [Format Overview](#format-overview) +- [Signatures And SFX](#signatures-and-sfx) +- [RAR5 Vint Encoding](#rar5-vint-encoding) +- [RAR5 General Block Format](#rar5-general-block-format) +- [RAR5 Header Types](#rar5-header-types) +- [RAR5 Common Header Flags](#rar5-common-header-flags) +- [RAR5 Main Archive Header](#rar5-main-archive-header) +- [RAR5 File And Service Headers](#rar5-file-and-service-headers) +- [RAR5 Extra Area Records](#rar5-extra-area-records) +- [RAR5 Compression Info](#rar5-compression-info) +- [Encryption](#encryption) +- [Checksums And Hashes](#checksums-and-hashes) +- [Service Headers](#service-headers) +- [RAR4 Compatibility](#rar4-compatibility) +- [Multivolume And Solid Archives](#multivolume-and-solid-archives) +- [SharpCompress Support Matrix](#sharpcompress-support-matrix) +- [SharpCompress Read Behavior](#sharpcompress-read-behavior) +- [Known Limitations](#known-limitations) +- [Test Fixtures](#test-fixtures) + +## Format Overview + +RAR is a block-oriented archive format. SharpCompress supports RAR as a read-only archive/reader format and delegates decompression to its RAR unpacker implementation under `src/SharpCompress/Compressors/Rar/`. + +RAR 5.0 general layout: + +```text +self-extracting module (optional) +RAR 5.0 signature +archive encryption header (optional) +main archive header +archive comment service header (optional) +file header 1 +service headers for preceding file (optional) +... +file header N +service headers for preceding file (optional) +recovery record (optional) +end of archive header +``` + +RAR has no ZIP-style central directory. Readers scan blocks in sequence. Random access is limited by solid compression, split volumes, and the need to process preceding compressed data for solid archives. + +## Signatures And SFX + +RAR signatures: + +| Format | Bytes | SharpCompress behavior | +| --- | --- | --- | +| Old pre-RAR4 marker | `52 45 7e 5e` | Explicitly unsupported | +| RAR4 marker | `52 61 72 21 1a 07 00` | Supported read path | +| RAR5 marker | `52 61 72 21 1a 07 01 00` | Supported read path | + +RAR archives can be preceded by a self-extracting module. `MarkHeader.Read` scans for the marker when `ReaderOptions.LookForHeader` is enabled. SharpCompress uses a maximum SFX scan size from the UnRAR implementation notes. + +## RAR5 Vint Encoding + +RAR5 uses variable-length integers, called `vint` in the format document. + +Rules: + +- Each byte contributes 7 data bits. +- The high bit is the continuation flag. +- If the high bit is `0`, the byte is the last byte in the sequence. +- The first byte contains the least significant 7 bits. +- RAR currently uses vint for up to 64-bit integers, so values can occupy up to 10 bytes. +- Writers can preallocate more bytes than needed by using leading `0x80` bytes, which encode zero with continuation set. + +SharpCompress reads these through `ReadRarVInt*` helpers on marking readers. For RAR5 header size, SharpCompress limits the size field to 3 vint bytes to match the current format implementation limit of a 2 MB maximum header size. + +## RAR5 General Block Format + +RAR5 blocks share a common header shape: + +| Field | Size | Notes | +| --- | --- | --- | +| Header CRC32 | `uint32` | CRC32 of header data starting at header size through optional extra area | +| Header size | `vint` | Size from header type through optional extra area; current max 3 bytes for 2 MB headers | +| Header type | `vint` | See [RAR5 Header Types](#rar5-header-types) | +| Header flags | `vint` | Common flags for all headers | +| Extra area size | `vint` | Present only when common flag `0x0001` is set | +| Data size | `vint` | Present only when common flag `0x0002` is set | +| Type-specific fields | variable | Depends on header type | +| Extra area | variable | Present only when common flag `0x0001` is set | +| Data area | variable | Present only when common flag `0x0002` is set; not included in header CRC/size | + +SharpCompress reads the common fields in `RarHeader.Initialize`, then creates typed headers in `RarHeaderFactory`. + +## RAR5 Header Types + +RAR5 header type values: + +| Type | Meaning | SharpCompress code path | +| --- | --- | --- | +| `1` | Main archive header | `ArchiveHeader` | +| `2` | File header | `FileHeader` with `HeaderType.File` | +| `3` | Service header | `FileHeader` with `HeaderType.Service` | +| `4` | Archive encryption header | `ArchiveCryptHeader` | +| `5` | End of archive header | `EndArchiveHeader` | + +SharpCompress constants live in `HeaderCodeV` in `Flags.cs`. + +## RAR5 Common Header Flags + +Common RAR5 header flags from `headers5.hpp` and `Flags.cs`: + +| Flag | Meaning | +| --- | --- | +| `0x0001` | Extra area is present | +| `0x0002` | Data area is present | +| `0x0004` | Unknown blocks with this flag must be skipped when updating | +| `0x0008` | Data area continues from previous volume | +| `0x0010` | Data area continues in next volume | +| `0x0020` | Block depends on preceding file block | +| `0x0040` | Preserve child block if host block is modified | + +SharpCompress exposes split status through `FileHeader.IsSplitBefore`, `FileHeader.IsSplitAfter`, and `RarEntry.IsSplitAfter`. + +## RAR5 Main Archive Header + +Main archive header fields after the common block fields: + +| Field | Size | Notes | +| --- | --- | --- | +| Archive flags | `vint` | Volume, solid, recovery, locked flags | +| Volume number | `vint` | Present only when archive flag `0x0002` is set | +| Extra area | variable | Optional records, currently locator is defined | + +Main archive flags: + +| Flag | Meaning | +| --- | --- | +| `0x0001` | Volume, archive is part of a multivolume set | +| `0x0002` | Volume number field is present | +| `0x0004` | Solid archive | +| `0x0008` | Recovery record is present | +| `0x0010` | Locked archive | + +Main header extra record types: + +| Type | Name | Meaning | +| --- | --- | --- | +| `0x01` | Locator | Optional offsets to quick-open and recovery-record blocks | + +SharpCompress parses archive flags in `ArchiveHeader` and uses volume/solid information in archive and reader flows. + +## RAR5 File And Service Headers + +File and service headers share the same base layout. Header type `2` is a file header and header type `3` is a service header. + +Fields after common block fields: + +| Field | Size | Notes | +| --- | --- | --- | +| File flags | `vint` | Directory, time, CRC, unknown unpacked size | +| Unpacked size | `vint` | Present even when unknown-size flag is set, but ignored then | +| Attributes | `vint` | OS-specific file attributes | +| mtime | `uint32` | Unix time, present only when file flag `0x0002` is set | +| Data CRC32 | `uint32` | Present only when file flag `0x0004` is set | +| Compression information | `vint` | Algorithm version, solid flag, method, dictionary size | +| Host OS | `vint` | `0` Windows, `1` Unix | +| Name length | `vint` | Byte count | +| Name | variable | UTF-8, no trailing zero | +| Extra area | variable | Optional file/service extra records | +| Data area | variable | File data or service data | + +RAR5 file flags: + +| Flag | Meaning | +| --- | --- | +| `0x0001` | Directory filesystem object | +| `0x0002` | Unix mtime field is present | +| `0x0004` | CRC32 field is present | +| `0x0008` | Unpacked size is unknown; extract until compression stream ends | + +SharpCompress parses these in `FileHeader.ReadFromReaderV5`. + +## RAR5 Extra Area Records + +Each extra record has this shape: + +```text +record size vint size from type through record data +record type vint +record data variable +``` + +File and service header extra record types: + +| Type | Name | SharpCompress behavior | +| --- | --- | --- | +| `0x01` | File encryption | Parses `Rar5CryptoInfo` | +| `0x02` | File hash | Reads BLAKE2sp digest when present | +| `0x03` | File time | Reads high precision mtime/ctime/atime | +| `0x04` | File version | Currently skipped/drained | +| `0x05` | Redirection | Parses symlink/junction/hard link/file copy metadata | +| `0x06` | Unix owner | Currently skipped/drained | +| `0x07` | Service data | Currently skipped/drained except service header data handling | + +Unknown records must be skipped without interrupting normal operation. SharpCompress drains unhandled extra record bytes after each record. + +Redirection types: + +| Value | Meaning | +| --- | --- | +| `0x0001` | Unix symlink | +| `0x0002` | Windows symlink | +| `0x0003` | Windows junction | +| `0x0004` | Hard link | +| `0x0005` | File copy | + +`RarEntry.IsRedir` and `RarEntry.RedirTargetName` expose part of this metadata. + +## RAR5 Compression Info + +RAR5 file compression information is a packed vint bit field: + +| Bits/mask | Meaning | +| --- | --- | +| `0x003f` | Compression algorithm version, currently 0 in RAR5; SharpCompress stores it as value + 50 | +| `0x0040` | Solid flag; dictionary continues from preceding files | +| `0x0380` | Compression method, values 0-5 are used; 0 is store/no compression | +| `0x3c00` | Minimum dictionary size: 0 means 128 KB, 1 means 256 KB, through 15 meaning 4096 MB | + +SharpCompress maps these to `FileHeader.CompressionAlgorithm`, `FileHeader.IsSolid`, `FileHeader.CompressionMethod`, and `FileHeader.WindowSize`. + +RAR4 and older algorithm values are distinct. `RarEntry.IsRarV3` currently treats algorithm values `15`, `20`, `26`, `29`, and `36` as legacy RAR code paths. + +## Encryption + +RAR5 archive encryption header fields: + +| Field | Size | Notes | +| --- | --- | --- | +| Encryption version | `vint` | Current supported version is 0, AES-256 | +| Encryption flags | `vint` | `0x0001` means password check data is present | +| KDF count | 1 byte | Binary logarithm of PBKDF2 iteration count | +| Salt | 16 bytes | Header encryption salt | +| Check value | 12 bytes | Optional password check plus checksum | + +When archive headers are encrypted, each following header starts with a 16-byte AES-256 initialization vector followed by encrypted header data aligned to 16 bytes. + +RAR5 file encryption extra record fields include encryption version, flags, KDF count, 16-byte salt, 16-byte IV, and optional 12-byte password check data. + +SharpCompress behavior: + +- Archive encryption header is parsed by `ArchiveCryptHeader`. +- File encryption extra records are parsed into `Rar5CryptoInfo`. +- Header decryption uses `RarCryptoBinaryReader` with `CryptKey5`. +- File data decryption uses `RarCryptoWrapper` around the packed stream. +- Password is required through `ReaderOptions.Password`; missing password throws `CryptographicException`. +- RAR4 encrypted file data uses RAR3 crypto classes and salt handling. + +## Checksums And Hashes + +RAR5 checksum/hash locations: + +- Header CRC32 covers header data starting at the header size field through optional extra area. It does not include the data area. +- File header can contain CRC32 of unpacked data when file flag `0x0004` is set. +- File hash extra record type `0x02` can contain BLAKE2sp hash data. The defined RAR5 hash type value is `0x00` for BLAKE2sp. +- For split files, hashes can apply to packed data for non-final volume parts. + +SharpCompress verifies header CRC in `RarHeader.VerifyHeaderCrc`. RAR CRC logic is in `RarCrcBinaryReader`, `AsyncRarCrcBinaryReader`, and `RarCRC`. + +## Service Headers + +RAR5 service headers use the file-header structure with header type `3`. Known service names from the RAR5 document and UnRAR source include: + +| Name | Meaning | +| --- | --- | +| `CMT` | Archive comment | +| `QO` | Quick-open data | +| `ACL` | NTFS permissions | +| `STM` | NTFS alternate data stream | +| `RR` | Recovery record | + +SharpCompress handles `CMT` specially by exposing its packed stream in `RarHeaderFactory`; most other service data is skipped or only partially modeled. + +Quick-open caution from the RAR5 format document: + +- Quick-open data can store copies of headers for faster listing. +- If quick-open data is used to display names, extraction must use the same source. Otherwise malicious archives could show one name and extract another. +- SharpCompress should not use quick-open data for one path and ordinary headers for another without carefully preserving this invariant. + +## RAR4 Compatibility + +SharpCompress supports RAR4-style headers as well as RAR5. + +RAR4 header codes from UnRAR and `Flags.cs`: + +| Code | Meaning | +| --- | --- | +| `0x72` | Mark header | +| `0x73` | Archive/main header | +| `0x74` | File header | +| `0x75` | Comment header | +| `0x76` | AV header | +| `0x77` | Old subheader | +| `0x78` | Protect/recovery header | +| `0x79` | Sign header | +| `0x7a` | New subheader/service header | +| `0x7b` | End archive header | + +RAR4 file-header behavior differs from RAR5: + +- Fixed-size base fields rather than RAR5 vint-heavy layout. +- Optional large-file high-size fields when `LARGE` flag is set. +- File names can use older RAR Unicode name encoding. +- DOS timestamps and RAR4 extended time flags are used. +- Directory status is encoded through the window mask. +- Path separators differ: RAR4 can use backslashes as separators, while RAR5 uses `/` as the universal separator. + +SharpCompress parses this in `FileHeader.ReadFromReaderV4` and maps paths through `ConvertPathV4`. + +## Multivolume And Solid Archives + +RAR supports multivolume archives and solid compression. + +Relevant flags: + +- Main archive `0x0001`: archive is part of a volume set. +- Main archive `0x0002`: RAR5 volume number field is present. +- Common block `0x0008`: data continues from previous volume. +- Common block `0x0010`: data continues in next volume. +- RAR4 file flags `SPLIT_BEFORE` and `SPLIT_AFTER` indicate split file data. +- RAR5 compression info `0x0040` indicates solid compression. + +SharpCompress behavior: + +- `RarFactory` implements `IMultiArchiveFactory`. +- `RarArchiveVolumeFactory` resolves volume file parts. +- `RarEntry.IsSplitAfter` exposes split status. +- Solid archives should be extracted sequentially. Prefer `ExtractAllEntries()` for solid RAR archives rather than extracting arbitrary entries independently. + +## SharpCompress Support Matrix + +| Feature | Support | Notes | +| --- | --- | --- | +| RAR read | Yes | Archive and Reader APIs | +| RAR write | No | RAR is read-only | +| RAR4 headers | Yes | Supported parsing path | +| RAR5 headers | Yes | Supported parsing path | +| Pre-RAR4 marker | No | `MarkHeader` throws unsupported format | +| RAR5 vint fields | Yes | Reader helpers parse vint values | +| File/service extra records | Partial | Encryption/hash/time/redirection modeled; others mostly skipped | +| Solid archives | Yes, with sequencing constraints | Use sequential extraction | +| Multivolume archives | Yes | Archive/multi-volume factory paths | +| Encrypted archives | Partial | Password required; format/version constraints apply | +| Archive comments | Partial | `CMT` service header handled specially | +| Quick-open records | Not a public feature | Avoid inconsistent listing/extraction behavior | +| Recovery records | Skipped/partial | Not recovery reconstruction API | + +## SharpCompress Read Behavior + +Header flow: + +1. `MarkHeader.Read` scans for RAR4 or RAR5 signature and rejects old pre-RAR4 signatures. +2. `RarHeaderFactory.ReadHeaders` records whether the archive is RAR5. +3. `RarHeader.TryReadBase` reads common header fields and CRC state. +4. `RarHeaderFactory` creates typed headers based on header code. +5. File data is either skipped, wrapped in `ReadOnlySubStream`, or wrapped in decryption stream depending on mode and encryption metadata. + +Seekable mode: + +- File data positions are recorded in `FileHeader.DataStartPosition`. +- Streams can seek over packed data while collecting headers. + +Streaming mode: + +- File data is exposed as a substream for file headers. +- Non-file service data is skipped unless specially handled. +- Consumers must process entries in archive order. + +Decompression: + +- `RarStream` wraps the packed stream and an `IRarUnpack` implementation. +- `RarStream` is non-seekable. +- `RarStream.Initialize` starts unpacking based on the parsed `FileHeader`. + +## Known Limitations + +Keep these limitations explicit in code comments, docs, and tests: + +- No RAR writing support. +- No support for pre-RAR4 archives. +- Service headers are not fully modeled by public APIs. +- Quick-open records are not a general public feature and have security-sensitive listing/extraction consistency requirements. +- Recovery record reconstruction is not exposed as a full repair feature. +- RAR5 Unix owner records and service data records are mostly skipped unless a specific behavior is implemented. +- Redirection metadata is surfaced, but extraction semantics for all link types should be treated carefully and tested for security. +- Encrypted archive support depends on password, encryption version, and KDF limits. +- Solid and multivolume archives must be tested with sequential extraction scenarios. + +## Test Fixtures + +Representative RAR test files live under `tests/TestArchives/Archives/` and related test archive folders. Use existing fixtures when possible instead of adding new binary archives. + +Representative test files: + +- `tests/SharpCompress.Test/Rar/RarArchiveTests.cs` +- `tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs` +- `tests/SharpCompress.Test/Rar/RarReaderTests.cs` +- `tests/SharpCompress.Test/Rar/RarReaderAsyncTests.cs` +- `tests/SharpCompress.Test/Rar/RarHeaderFactoryTest.cs` +- `tests/SharpCompress.Test/Rar/RarCRCTest.cs` + +When changing RAR behavior, include both Archive and Reader API coverage where applicable. For solid or multivolume behavior, prefer tests that extract entries sequentially and verify stream ownership and volume transitions.