sharpcompress/docs/TAR_SPEC.md

# Tar Spec

## Scope

This document describes the Tar implementation that exists in SharpCompress today.

It is intentionally SharpCompress-specific. It documents actual behavior in the current codebase, including partial support and limitations. It is not a general tar format reference.

Primary implementation files:

- `src/SharpCompress/Factories/TarFactory.cs`
- `src/SharpCompress/Factories/TarWrapper.cs`
- `src/SharpCompress/Archives/Tar/TarArchive.cs`
- `src/SharpCompress/Archives/Tar/TarArchive.Async.cs`
- `src/SharpCompress/Archives/Tar/TarArchive.Factory.cs`
- `src/SharpCompress/Readers/Tar/TarReader.cs`
- `src/SharpCompress/Readers/Tar/TarReader.Async.cs`
- `src/SharpCompress/Writers/Tar/TarWriter.cs`
- `src/SharpCompress/Writers/Tar/TarWriter.Async.cs`
- `src/SharpCompress/Writers/Tar/TarWriterOptions.cs`
- `src/SharpCompress/Common/Tar/Headers/TarHeader.cs`
- `src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs`
- `src/SharpCompress/Common/Tar/TarHeaderFactory.cs`
- `src/SharpCompress/Common/Tar/TarHeaderFactory.Async.cs`

## API Surface

SharpCompress exposes Tar support through four main entry points.

| Type | Role |
| ---- | ---- |
| `TarFactory` | Format detection and factory entry point for archive, reader, and writer APIs |
| `TarArchive` | Archive API for enumerating and rewriting tar archives |
| `TarReader` | Forward-only reader API for streaming tar extraction |
| `TarWriter` | Forward-only writer API for creating tar archives |

`TarWriterOptions` controls output compression, stream ownership, archive finalization, encoding, and header write format.

## Supported Wrapper Formats

Tar wrapper detection is defined by `TarWrapper.Wrappers` in `src/SharpCompress/Factories/TarWrapper.cs`.

### Supported Extensions

| Wrapper | Extensions |
| ------- | ---------- |
| Plain tar | `tar` |
| Tar + BZip2 | `tar.bz2`, `tb2`, `tbz`, `tbz2`, `tz2` |
| Tar + GZip | `tar.gz`, `taz`, `tgz` |
| Tar + ZStandard | `tar.zst`, `tar.zstd`, `tzst`, `tzstd` |
| Tar + LZip | `tar.lz` |
| Tar + XZ | `tar.xz`, `txz` |
| Tar + LZW compress | `tar.Z`, `tZ`, `taZ` |

### API Support Matrix

| Wrapper | Detection | `TarArchive` read | `TarReader` read | `TarWriter` write |
| ------- | --------- | ----------------- | ---------------- | ----------------- |
| Plain tar | Yes | Yes | Yes | Yes |
| Tar + GZip | Yes | Yes | Yes | Yes |
| Tar + BZip2 | Yes | Yes | Yes | Yes |
| Tar + LZip | Yes | Yes | Yes | Yes |
| Tar + XZ | Yes | Yes | Yes | No |
| Tar + ZStandard | Yes | Yes | Yes | No |
| Tar + LZW compress | Yes | Yes | Yes | No |

Write support is implemented in `src/SharpCompress/Writers/Tar/TarWriter.cs` and currently accepts only `CompressionType.None`, `CompressionType.GZip`, `CompressionType.BZip2`, and `CompressionType.LZip`.

## Detection Behavior

Tar detection is implemented in `TarFactory.IsArchive`, `TarFactory.IsArchiveAsync`, `TarFactory.GetCompressionType`, and `TarFactory.GetCompressionTypeAsync`.

Detection behavior is:

1. Wrap the incoming stream in `SharpCompressStream`.
2. Start recording with a rewind buffer sized from `TarWrapper.MaximumRewindBufferSize`.
3. Probe each registered wrapper in order.
4. If a wrapper matches, create a decompression stream for that wrapper.
5. Call `TarArchive.IsTarFile` or `TarArchive.IsTarFileAsync` on the decompressed stream.
6. If the tar probe succeeds, treat the stream as tar with that wrapper compression.

Implications:

- Tar detection is content-based, not extension-based.
- Wrapper detection is not sufficient by itself. The decompressed payload must also parse as tar.
- Non-seekable detection is supported through the recording and rewind mechanism.
- The largest rewind requirement currently comes from BZip2, which declares a larger minimum probe buffer in `TarWrapper`.

`TarArchive.IsTarFile` and `TarArchive.IsTarFileAsync` attempt to read a single tar header and return `false` on any exception. They also treat an all-zero empty archive block as a valid empty tar archive when the entry type is defined.

## Reader Behavior

`TarReader` is the forward-only streaming API.

Implementation files:

- `src/SharpCompress/Readers/Tar/TarReader.cs`
- `src/SharpCompress/Readers/Tar/TarReader.Async.cs`
- `src/SharpCompress/Common/Tar/TarEntry.cs`
- `src/SharpCompress/Common/Tar/TarEntry.Async.cs`
- `src/SharpCompress/Common/Tar/TarReadOnlySubStream.cs`

Reader behavior:

- The reader always enumerates entries in streaming mode.
- It works with non-seekable input streams.
- It applies decompression based on the detected wrapper compression type before parsing tar headers.
- Entry streams are backed by `TarReadOnlySubStream`.

`TarReadOnlySubStream` has an important behavior: disposing an entry stream consumes any unread entry bytes and any required 512-byte padding so that the next header can be read correctly. This is what makes skipping entries work in streaming mode.

### Reader Compression Mapping

`TarReader.RequestInitialStream` and `RequestInitialStreamAsync` map the detected wrapper to the corresponding decompression stream:

- `None`
- `BZip2`
- `GZip`
- `ZStandard`
- `LZip`
- `Xz`
- `Lzw`

### Reader Entry Semantics

For each entry, SharpCompress exposes:

- `Key` from the parsed tar name
- `LinkTarget` for symbolic and hard links
- `Size`
- `CompressedSize`
- `LastModifiedTime`
- `IsDirectory`
- `Mode`
- `UserID`
- `GroupId`

Tar entries are always reported as unencrypted and CRC is always `0`.

## Archive Behavior

`TarArchive` is the archive API.

Implementation files:

- `src/SharpCompress/Archives/Tar/TarArchive.cs`
- `src/SharpCompress/Archives/Tar/TarArchive.Async.cs`
- `src/SharpCompress/Archives/Tar/TarArchive.Factory.cs`

### Open Behavior

`TarArchive.OpenArchive(Stream)` and `TarArchive.OpenAsyncArchive(Stream)` require a seekable stream and throw `ArgumentException` when `CanSeek` is `false`.

`TarArchive.OpenArchive(FileInfo)` and the list-based overloads use `SourceStream` and determine wrapper compression by calling `TarFactory.GetCompressionType`.

Asynchronous `OpenAsyncArchive` overloads use `TarFactory.GetCompressionTypeAsync` for wrapper detection.

### Entry Loading

`TarArchive.LoadEntries` and `LoadEntriesAsync` parse entries differently depending on wrapper compression:

- Uncompressed tar uses `StreamingMode.Seekable`.
- Wrapped tar uses `StreamingMode.Streaming` because the decompressed stream is not treated as random-access.

When seekable mode is used, the header stores `DataStartPosition`, and entries reopen data through `TarFilePart` by seeking back to the data position.

When streaming mode is used, the header stores a `PackedStream`, and entry access follows streaming semantics over the decompressed stream.

### Archive Rewrite Behavior

`TarArchive` supports creating and modifying archives through `AbstractWritableArchive`:

- add file entries
- add directory entries
- remove entries
- save to a new stream or path

Archive rewrite is implemented by enumerating the existing and new entries and writing them back out through `TarWriter`.

## Writer Behavior

`TarWriter` is the forward-only tar writer.

Implementation files:

- `src/SharpCompress/Writers/Tar/TarWriter.cs`
- `src/SharpCompress/Writers/Tar/TarWriter.Async.cs`
- `src/SharpCompress/Writers/Tar/TarWriterOptions.cs`

### Supported Output Compression

The writer supports these output compression types:

- `CompressionType.None`
- `CompressionType.GZip`
- `CompressionType.BZip2`
- `CompressionType.LZip`

Any other compression type causes `InvalidFormatException`.

### Stream Ownership

If `LeaveStreamOpen` is `true`, `TarWriter` wraps the destination in a non-disposing stream.

### File Writing

`TarWriter.Write` and `WriteAsync` write a tar header followed by file contents, then pad the payload to the next 512-byte boundary.

If the source stream is non-seekable and the caller does not supply `size`, the writer throws `ArgumentException` because tar requires the file size in the header.

### Directory Writing

`WriteDirectory` and `WriteDirectoryAsync` normalize the directory name to use forward slashes and ensure the key ends with `/`.

Empty or root-equivalent directory names are skipped.

### Archive Finalization

If `FinalizeArchiveOnClose` is `true`, disposing the writer writes two 512-byte zero blocks to terminate the archive.

If the output stream implements `IFinishable`, dispose also calls `Finish()`.

## Header Write Formats

`TarHeaderWriteFormat` is defined in `src/SharpCompress/Common/Tar/Headers/TarHeaderWriteFormat.cs`.

Supported write formats:

- `GNU_TAR_LONG_LINK`
- `USTAR`

`TarWriterOptions.HeaderFormat` defaults to `GNU_TAR_LONG_LINK`.

Current implementation behavior is narrower than the option surface suggests:

- sync file writes use the configured `HeaderFormat`
- sync directory writes currently construct the default tar header format
- async file writes currently construct the default tar header format
- async directory writes currently construct the default tar header format

In practice, this means the configured `HeaderFormat` is currently honored only by the synchronous file write path.

### GNU Long Name Write Behavior

In GNU mode, when a file name exceeds the 100-byte field, `TarHeader.WriteGnuTarLongLink` writes a synthetic long-name header using `././@LongLink` and `EntryType.LongName`, then writes the long name payload, and finally writes the actual file entry.

GNU mode also writes large file sizes using binary size encoding when the size does not fit the standard octal field.

### USTAR Write Behavior

When the synchronous file write path is configured for `USTAR`, `TarHeader.WriteUstar` attempts to split a long path into:

- the main `name` field
- the `prefix` field

If the name cannot be represented in USTAR field limits, the writer throws `InvalidFormatException` and instructs the caller to use GNU Tar format instead.

## Header Read Behavior

Tar header parsing is implemented in `TarHeader.Read` and `TarHeader.ReadAsync`.

### Implemented Read Features

| Feature | Read support |
| ------- | ------------ |
| Regular file entries | Yes |
| Directory entries | Yes |
| Symbolic link target reading | Yes |
| Hard link target reading | Yes |
| GNU long name (`L`) | Yes |
| GNU long link (`K`) | Yes |
| PAX local extended header (`x`) | Yes (selected keys) |
| PAX global extended header (`g`) | Yes (selected keys) |
| USTAR prefix reconstruction | Yes |
| Binary size field parsing | Yes |
| oldgnu uid/gid numeric quirk parsing | Yes |
| POSIX and signed checksum validation | Yes |

### Entry Types Recognized by the Code

`EntryType` currently declares these values in `src/SharpCompress/Common/Tar/Headers/EntryType.cs`:

- `File`
- `OldFile`
- `HardLink`
- `SymLink`
- `CharDevice`
- `BlockDevice`
- `Directory`
- `Fifo`
- `LongLink`
- `LongName`
- `SparseFile`
- `VolumeHeader`
- `LocalExtendedHeader`
- `GlobalExtendedHeader`

SharpCompress currently has explicit handling for only a subset of those values during read and write.

### Long Name and Long Link Reads

When `TarHeader.Read` encounters `EntryType.LongName` or `EntryType.LongLink`, it reads the payload and applies it to the next real header.

Long-name payload reads are capped at `32768` bytes to avoid memory exhaustion from malformed archives.

### PAX Local Header Reads

SharpCompress now consumes local PAX extended headers (`x`) and applies supported key overrides to the next real entry.

Currently supported keys:

- `path`
- `linkpath`
- `size`
- `mtime`
- `uid`
- `gid`
- `mode`

Unknown PAX keys are ignored.

### PAX Global Header Reads

SharpCompress consumes global PAX extended headers (`g`) and applies supported key overrides to subsequent entries.

Supported keys match local PAX support:

- `path`
- `linkpath`
- `size`
- `mtime`
- `uid`
- `gid`
- `mode`

Global metadata is overridden by local per-entry metadata when both are present.

### Name Reconstruction

For USTAR headers, if the magic field is `ustar` and the prefix field is populated, SharpCompress reconstructs the entry name as `prefix + "/" + name`.

## Name and Metadata Handling

### Path Normalization

Writer path normalization is implemented in `TarWriter.NormalizeFilename` and `NormalizeDirectoryName`.

Behavior:

- backslashes are converted to `/`
- drive prefixes before `:` are removed
- leading and trailing `/` are trimmed for file entries
- directory entries are normalized to end with `/`

### Encoding

Tar name encoding and decoding is controlled by `IArchiveEncoding`.

- reader APIs decode names with `ReaderOptions.ArchiveEncoding`
- writer APIs encode names with `TarWriterOptions.ArchiveEncoding`

The tests include UTF-8 and code page coverage for tar name handling.

### Metadata Surface

Tar metadata currently surfaced through `TarEntry` includes:

- name
- link target
- mode
- uid
- gid
- size
- last modified time

Writer metadata is narrower. The writer sets:

- `LastModifiedTime`
- `Name`
- `Size`
- entry type for file or directory

The current writer writes fixed mode, owner id, and group id defaults rather than round-tripping full metadata.

## Async Behavior

Async tar support is provided by:

- `TarArchive.OpenAsyncArchive`
- `TarReader.OpenAsyncReader`
- `TarWriter.WriteAsync`
- `TarWriter.WriteDirectoryAsync`
- `TarHeader.ReadAsync`
- `TarHeader.WriteAsync`

The async implementations generally mirror the sync implementations while using async header parsing, decompression, and stream copy paths.

## Known Limitations

This section documents current implementation limits, not desired future behavior.

### Write limitations

- No write support for `tar.xz`
- No write support for `tar.zst`
- No write support for `tar.Z`
- No public API for writing symbolic links or hard links
- No PAX write support
- No sparse file write support
- No device or FIFO write support

### Read limitations or partial support

- PAX support is limited to selected keys (`path`, `linkpath`, `size`, `mtime`, `uid`, `gid`, `mode`)
- No semantic sparse file handling beyond recognizing the entry type enum value
- No special device or FIFO object model beyond the raw entry type information available internally

### Archive behavior limitations

- Stream-based archive open requires a seekable input stream
- Compressed tar archive access is not full random-access in the same sense as uncompressed seekable tar

## Test Coverage Map

Tar tests live in `tests/SharpCompress.Test/Tar/`.

Representative coverage:

| Area | Tests |
| ---- | ----- |
| Wrapper detection and reading | `TarReaderTests.cs`, `TarReaderAsyncTests.cs` |
| Archive open and rewrite | `TarArchiveTests.cs`, `TarArchiveAsyncTests.cs` |
| Writer behavior | `TarWriterTests.cs`, `TarWriterAsyncTests.cs` |
| Directory entry behavior | `TarWriterDirectoryTests.cs`, `TarArchiveDirectoryTests.cs` |
| Long-name behavior | `TarArchiveTests.cs`, `TarReaderTests.cs` |
| Corruption and broken stream handling | `TarReaderTests.cs`, `TarReaderAsyncTests.cs` |

Representative tar test archives in `tests/TestArchives/Archives/`:

- `Tar.tar`
- `Tar.tar.gz`
- `Tar.tar.bz2`
- `Tar.tar.lz`
- `Tar.tar.xz`
- `Tar.tar.zst`
- `Tar.tar.Z`
- `Tar.oldgnu.tar.gz`
- `very long filename.tar`
- `ustar with long names.tar`
- `Tar.LongPathsWithLongNameExtension.tar`
- `Tar.PaxGlobalHeader.tar`
- `Tar.PaxGlobalHeader.Link.tar`
- `Tar.Empty.tar`
- `TarCorrupted.tar`
- `TarWithSymlink.tar.gz`

## Summary

SharpCompress Tar support is centered around:

- broad read support for common tar wrappers
- forward-only reader behavior for streamed extraction
- seekable archive support for uncompressed tar and archive rewrite workflows
- narrower write support than read support
- GNU long-name and USTAR write support
- PAX local header (`x`) read support for selected metadata keys
- partial coverage for less common tar dialect features