SabreTools/SabreTools.Serialization
1
0
Fork 0
mirror of https://github.com/SabreTools/SabreTools.Serialization.git synced 2026-07-08 18:06:41 +00:00
Code Issues 56 Packages Projects Releases 20 Wiki Activity

Dolphin lib (#85)

Browse Source 
* Add GCZ, WIA/RVZ, and NintendoDisc (GameCube/Wii) format support

Port of DolphinIsoLib into SabreTools.Serialization architecture:
- Data.Models: GCZ/, WIA/, NintendoDisc/ model subdirectories (15 files)
- Serialization.Readers: GCZ, WIA, NintendoDisc readers
- Serialization.Writers: GCZ, WIA writers (structural metadata; full round-trip TODO)
- Wrappers: NintendoDisc, GCZ, WIA wrappers with Encryption partial class
- Wrappers: WiaRvzCompressionHelper (BZip2/LZMA/LZMA2/Zstd, net462+ guarded)
- WrapperType + WrapperFactory: GCZ, WIA, NintendoDisc entries added

GetInnerWrapper() decompression and NintendoDisc.Extraction FST extraction
are stubbed with TODO comments pending full implementation.

* Implement GetInnerWrapper for GCZ and WIA, full NintendoDisc extraction

* Add GCZ/WIA/RVZ write pipeline and Nintendo disc compression helpers

* Add WIA/RVZ table decompression, NintendoDisc/GCZ printing, NintendoDisc detection for .iso files

* Fix NintendoDisc header layout, GC magic, and add embedded disc header to WIA/GCZ printing

- Fix GCMagicWord: 0xC23D3C1F -> 0xC2339F3D (confirmed from Dolphin DiscUtils.h)
- Fix disc header field layout to match Dolphin's confirmed offsets:
  MakerCode is bytes 4-5 of the 6-char GameId (no separate field at 0x006),
  DiscNumber at 0x006, DiscVersion at 0x007, unused region is 14 bytes (0x00A-0x017)
- Update NintendoDisc reader: derive MakerCode from GameId[4..5], fix skip count
- Add ParseDiscHeaderOnly() to reader for partial (short) stream parsing
- Guard DisableHash/DisableEnc reads at the 0x080 boundary for 128-byte embedded headers
- Guard DOL/FST skip for streams shorter than full 0x440 boot block
- Fix WrapperFactory: NintendoDisc magic detection now precedes .iso -> ISO9660 fallback
- Add GameId-prefix heuristic in WrapperFactory for GC discs lacking magic word
- Add GameId-prefix platform fallback in reader for GC discs without GCMagicWord
- Add DiscHeader property to WIA wrapper (parsed from Header2.DiscHeader bytes)
- Add DiscHeader property to GCZ wrapper (decompresses first block only)
- Add ReadDiscHeader() helper to GCZ for lightweight first-block decompression
- Print embedded disc header (Game ID, Maker, Disc/Rev, Title) in WIA.Printing.cs and GCZ.Printing.cs

* Fix Wii partition extraction: correct IV, FST size shift, partition naming

- Block decryption: IV is at raw block offset 0x3D0 (still-encrypted),
  matching Dolphin/DolphinIsoLib WiiPartitionDecryptor.DecryptBlock exactly.
- FST size field at boot.bin 0x428 is also stored >>2 on Wii; apply <<2
  to get true byte size.
- Partition folder naming now matches DolphinIsoLib WiiDiscExtractor exactly:
  type 0->GM+n, 1->UP+n, 2->CH+n, printable ASCII unknown->raw 4-char string,
  non-printable->P{index}. SSBB VC channels extract as HA8E, HA9E, etc.
- ExtractionTool peek buffer increased from 16 to 32 bytes.

Verified: SSBB GM0 extracts 5524 files, boot.bin/fst.bin byte-identical
to Dolphin reference extraction.

* Fix FST extraction: create zero-byte files instead of skipping them

Files with fileSize=0 in the FST were silently skipped. Now they are
created as empty files, matching Dolphin/DolphinIsoLib behavior.

Verified: SSBB now extracts 5958 files with 0 missing, 0 extra,
0 size mismatches, and 0 hash mismatches vs DolphinIsoLib reference.

* Add GCZ/WIA/RVZ virtual stream extraction via NintendoDisc wrapper

* Address PR #85 review comments (Copilot + mnadareski)

* Address PR #85 review comments

* Replace custom endian helpers and SHA1 with SabreTools.IO equivalents

* Update GCZ.Printing.cs

* Update NintendoDisc.Printing.cs

* Update WIA.Printing.cs

* Add WIA/RVZ Wii partition crypto round-trip support

- Add AesCbc internal helper (BouncyCastle AES-CBC encrypt/decrypt)
- Add NintendoDisc.CommonKeyProvider hook for injectable test keys
- Fix sha1.Terminate() missing in all three ComputeSha1 helpers in WIA.cs
- Fix Wii partition dataOff alignment to 0x8000 boundary
- Add WIA.EncryptWiiGroup (internal) for re-encrypting plaintext groups
- Add WIA.DumpIso to WIA.Writing.cs (WIA/RVZ -> flat ISO conversion)
- Add WiaVirtualStream on-demand group decompression
- Add _preDecryptedReader bypass on NintendoDisc for WIA extraction path
- Add WIATests.cs with Wii crypto round-trip test using synthetic data
- Move DumpIso from WIA.Extraction.cs to WIA.Writing.cs
- Bump DumpIso read buffer from 1 MiB to 2 MiB (aligns to WIA chunk size)
- Add InternalsVisibleTo SabreTools.Wrappers.Test in csproj

* Remove hardcoded Wii common keys from NintendoDisc.Encryption

- Delete the embedded WiiCommonKeyRetail and WiiCommonKeyKorean byte
  arrays from NintendoDisc.Encryption.cs.
- Make CommonKeyProvider public so any caller (not just tests) can
  inject keys; DecryptTitleKey now returns null when no key is
  available for the requested index rather than falling back to
  hardcoded values.
- Add NintendoDiscEncryptionTests.cs:
    - Argument guard and no-provider tests for DecryptTitleKey.
    - Fake-key round-trip test (encrypt then decrypt with injected key).
    - Integration test that reads TestData/NintendoDisc/keys.json,
      verifies each key against hardcoded SHA256 constants, and skips
      silently if the file is absent or the keys do not match.
    - LoadKeyProvider helper (named JSON format, index-keyed).
- Add [Collection(NintendoDisc)] to both NintendoDiscEncryptionTests
  and WIATests to prevent parallel access to the static
  CommonKeyProvider from racing between test classes.
- Add TestData/NintendoDisc/keys.json.example documenting the
  expected key file format.
- Add Newtonsoft.Json reference to SabreTools.Wrappers.Test.csproj.

* Didn't actually commit the changes. My bad. Fixed.

* Edited a comment

* Added in XUnit outputs that show up in Test Viewer in VS

---------

Co-authored-by: Matt Nadareski <mnadareski@outlook.com>
This commit is contained in:
Dimensional
2026-05-12 09:41:32 -05:00
committed by GitHub
parent 3f71628600
commit 49aa6895b6
38 changed files with 7029 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
SabreTools.Serialization.Readers/GCZ.cs Normal file
View File

@@ -0,0 +1,71 @@
using System.IO;
using SabreTools.Data.Models.GCZ;
using SabreTools.IO.Extensions;
using SabreTools.Numerics.Extensions;
#pragma warning disable IDE0017 // Simplify object initialization
namespace SabreTools.Serialization.Readers
{
public class GCZ : BaseBinaryReader<DiscImage>
{
/// <inheritdoc/>
public override DiscImage? Deserialize(Stream? data)
{
// If the data is invalid
if (data is null || !data.CanRead)
return null;
// Need at least the header
if (data.Length - data.Position < Constants.HeaderSize)
return null;
try
{
long initialOffset = data.Position;
var archive = new DiscImage();
// Parse the header
archive.Header = ParseGczHeader(data);
if (archive.Header.MagicCookie != Constants.MagicCookie)
return null;
// Validate block count — guard against absurdly large tables
if (archive.Header.NumBlocks == 0 || archive.Header.NumBlocks > 0x100000)
return null;
int numBlocks = (int)archive.Header.NumBlocks;
// Read block pointer table (8 bytes per block)
archive.BlockPointers = new ulong[numBlocks];
byte[] ptrBuf = data.ReadBytes(numBlocks * 8);
for (int i = 0; i < numBlocks; i++)
archive.BlockPointers[i] = System.BitConverter.ToUInt64(ptrBuf, i * 8);
// Read block hash table (4 bytes per block, Adler-32)
archive.BlockHashes = new uint[numBlocks];
byte[] hashBuf = data.ReadBytes(numBlocks * 4);
for (int i = 0; i < numBlocks; i++)
archive.BlockHashes[i] = System.BitConverter.ToUInt32(hashBuf, i * 4);
return archive;
}
catch
{
return null;
}
}
private static GczHeader ParseGczHeader(Stream data)
{
var header = new GczHeader();
header.MagicCookie = data.ReadUInt32LittleEndian();
header.SubType = data.ReadUInt32LittleEndian();
header.CompressedDataSize = data.ReadUInt64LittleEndian();
header.DataSize = data.ReadUInt64LittleEndian();
header.BlockSize = data.ReadUInt32LittleEndian();
header.NumBlocks = data.ReadUInt32LittleEndian();
return header;
}
}
}

205
SabreTools.Serialization.Readers/NintendoDisc.cs Normal file
View File

@@ -0,0 +1,205 @@
using System.IO;
using System.Text;
using SabreTools.Data.Models.NintendoDisc;
using SabreTools.IO.Extensions;
using SabreTools.Numerics.Extensions;
#pragma warning disable IDE0017 // Simplify object initialization
namespace SabreTools.Serialization.Readers
{
public class NintendoDisc : BaseBinaryReader<Disc>
{
/// <inheritdoc/>
public override Disc? Deserialize(Stream? data)
{
// If the data is invalid
if (data is null || !data.CanRead)
return null;
// Need at least the disc header
if (data.Length - data.Position < Constants.DiscHeaderSize)
return null;
try
{
long initialOffset = data.Position;
var disc = new Disc();
// Parse the disc header
disc.Header = ParseDiscHeader(data);
// Determine platform from magic words; fall back to GameId prefix for
// GC discs that omit the magic word (e.g. some redump/scene ISOs)
if (disc.Header.WiiMagic == Constants.WiiMagicWord)
disc.Platform = Platform.Wii;
else if (disc.Header.GCMagic == Constants.GCMagicWord)
disc.Platform = Platform.GameCube;
else if (disc.Header.GameId != null && disc.Header.GameId.Length >= 1
&& IsGameCubeTitleType(disc.Header.GameId[0]))
disc.Platform = Platform.GameCube;
else
disc.Platform = Platform.Unknown;
// Parse Wii-specific structures
if (disc.Platform == Platform.Wii)
{
// Partition table starts at 0x40000
long partTableEnd = initialOffset + Constants.WiiPartitionTableAddress
+ (Constants.WiiPartitionGroupCount * 8);
if (data.Length >= partTableEnd)
disc.PartitionTableEntries = ParsePartitionTable(data, initialOffset);
// Region data at 0x4E000
long regionEnd = initialOffset + Constants.WiiRegionDataAddress + Constants.WiiRegionDataSize;
if (data.Length >= regionEnd)
{
data.Seek(initialOffset + Constants.WiiRegionDataAddress, SeekOrigin.Begin);
disc.RegionData = ParseRegionData(data);
}
}
return disc;
}
catch
{
return null;
}
}
#region Header parsing
/// <summary>
/// Parses just the disc header fields from the given stream without requiring
/// the full 0x440-byte boot block. Requires at least 0x82 bytes (enough to
/// reach AudioStreaming and StreamingBufferSize) to be useful; the DOL/FST
/// fields will be zero when the stream is shorter than 0x42B bytes.
/// </summary>
public static DiscHeader? ParseDiscHeaderOnly(Stream? data)
{
if (data is null || !data.CanRead || data.Length - data.Position < 6)
return null;
try { return ParseDiscHeader(data); }
catch { return null; }
}
private static DiscHeader ParseDiscHeader(Stream data)
{
var header = new DiscHeader();
// 0x000: 4-char title code + 2-char maker code stored as one 6-byte GameId field
byte[] gameIdBytes = data.ReadBytes(Constants.GameIdLength);
header.GameId = Encoding.ASCII.GetString(gameIdBytes).TrimEnd('\0');
// Maker code is the last 2 chars of the GameId (offsets 0x004–0x005).
// Dolphin reads it with Read(0x4, 2) — there is no separate field at 0x006.
header.MakerCode = header.GameId != null && header.GameId.Length >= 6
? header.GameId.Substring(4, 2)
: string.Empty;
// 0x006: disc number, 0x007: revision (Dolphin GetDiscNumber/GetRevision)
header.DiscNumber = data.ReadByteValue();
header.DiscVersion = data.ReadByteValue();
// 0x008: audio streaming, 0x009: streaming buffer size
header.AudioStreaming = data.ReadByteValue();
header.StreamingBufferSize = data.ReadByteValue();
// Skip unused 0x0E bytes (offsets 0x00A–0x017)
data.ReadBytes(0x0E);
header.WiiMagic = data.ReadUInt32BigEndian();
header.GCMagic = data.ReadUInt32BigEndian();
byte[] titleBytes = data.ReadBytes(Constants.GameTitleLength);
header.GameTitle = Encoding.ASCII.GetString(titleBytes).TrimEnd('\0');
header.DisableHashVerification = data.Position < data.Length ? data.ReadByteValue() : (byte)0;
header.DisableDiscEncryption = data.Position < data.Length ? data.ReadByteValue() : (byte)0;
// Skip to DOL/FST offset fields at 0x420.
// Position so far: 6+1+1+1+1+14+4+4+96+1+1 = 130 = 0x82
int skipToBootBlock = Constants.DolOffsetField - 0x82;
if (data.Length - data.Position < skipToBootBlock + 12)
return header;
data.ReadBytes(skipToBootBlock);
header.DolOffset = data.ReadUInt32BigEndian();
header.FstOffset = data.ReadUInt32BigEndian();
header.FstSize = data.ReadUInt32BigEndian();
// Skip the remaining bytes to complete the 0x440 header
// We are at 0x420 + 12 = 0x42C; need to reach 0x440
data.ReadBytes(Constants.DiscHeaderSize - (Constants.DolOffsetField + 12));
return header;
}
#endregion
#region Wii partition table parsing
private static WiiPartitionTableEntry[]? ParsePartitionTable(Stream data, long baseOffset)
{
data.Seek(baseOffset + Constants.WiiPartitionTableAddress, SeekOrigin.Begin);
// Read 4 partition groups; each group has a count and a shifted offset
var allEntries = new System.Collections.Generic.List<WiiPartitionTableEntry>();
for (int g = 0; g < Constants.WiiPartitionGroupCount; g++)
{
uint count = data.ReadUInt32BigEndian();
uint shiftedOffset = data.ReadUInt32BigEndian();
if (count == 0)
continue;
long tableOffset = baseOffset + ((long)shiftedOffset << 2);
long savedPosition = data.Position;
if (tableOffset + ((long)count * 8) > data.Length)
{
data.Seek(savedPosition, SeekOrigin.Begin);
continue;
}
data.Seek(tableOffset, SeekOrigin.Begin);
for (uint i = 0; i < count; i++)
{
var entry = new WiiPartitionTableEntry();
uint rawOffset = data.ReadUInt32BigEndian();
entry.Offset = (long)rawOffset << 2;
entry.Type = data.ReadUInt32BigEndian();
allEntries.Add(entry);
}
data.Seek(savedPosition, SeekOrigin.Begin);
}
return allEntries.Count > 0 ? allEntries.ToArray() : null;
}
#endregion
#region Wii region data parsing
private static WiiRegionData ParseRegionData(Stream data)
{
var region = new WiiRegionData();
region.RegionSetting = data.ReadUInt32BigEndian();
region.AgeRatings = data.ReadBytes(16);
return region;
}
#endregion
/// <summary>
/// Returns true if the GameId first character is a known GameCube title type prefix.
/// Used as a fallback when the GC magic word is absent from the disc image.
/// </summary>
private static bool IsGameCubeTitleType(char c)
{
return c == 'G' || c == 'D' || c == 'R';
}
}
}

194
SabreTools.Serialization.Readers/WIA.cs Normal file
View File

@@ -0,0 +1,194 @@
using System.IO;
using SabreTools.Data.Models.WIA;
using SabreTools.IO.Extensions;
using SabreTools.Numerics.Extensions;
#pragma warning disable IDE0017 // Simplify object initialization
namespace SabreTools.Serialization.Readers
{
public class WIA : BaseBinaryReader<DiscImage>
{
/// <inheritdoc/>
public override DiscImage? Deserialize(Stream? data)
{
// If the data is invalid
if (data is null || !data.CanRead)
return null;
// Need at least Header1
if (data.Length - data.Position < Constants.Header1Size)
return null;
try
{
long initialOffset = data.Position;
var archive = new DiscImage();
// Parse Header1
archive.Header1 = ParseHeader1(data);
// Validate magic
if (archive.Header1.Magic != Constants.WiaMagic && archive.Header1.Magic != Constants.RvzMagic)
return null;
// Parse Header2
archive.Header2 = ParseHeader2(data);
// Parse partition entries (Wii discs only)
if (archive.Header2.NumberOfPartitionEntries > 0
&& archive.Header2.PartitionEntriesOffset > 0)
{
data.Seek(initialOffset + (long)archive.Header2.PartitionEntriesOffset, SeekOrigin.Begin);
archive.PartitionEntries = ParsePartitionEntries(
data, (int)archive.Header2.NumberOfPartitionEntries);
}
// Parse raw data entries
if (archive.Header2.NumberOfRawDataEntries > 0
&& archive.Header2.RawDataEntriesOffset > 0)
{
data.Seek(initialOffset + (long)archive.Header2.RawDataEntriesOffset, SeekOrigin.Begin);
archive.RawDataEntries = ParseRawDataEntries(
data, (int)archive.Header2.NumberOfRawDataEntries);
}
// Parse group entries
if (archive.Header2.NumberOfGroupEntries > 0
&& archive.Header2.GroupEntriesOffset > 0)
{
data.Seek(initialOffset + (long)archive.Header2.GroupEntriesOffset, SeekOrigin.Begin);
if (archive.Header1.Magic == Constants.RvzMagic)
archive.RvzGroupEntries = ParseRvzGroupEntries(
data, (int)archive.Header2.NumberOfGroupEntries);
else
archive.GroupEntries = ParseWiaGroupEntries(
data, (int)archive.Header2.NumberOfGroupEntries);
}
return archive;
}
catch
{
return null;
}
}
#region Header parsing
private static WiaHeader1 ParseHeader1(Stream data)
{
var h = new WiaHeader1();
h.Magic = data.ReadUInt32LittleEndian();
h.Version = data.ReadUInt32BigEndian();
h.VersionCompatible = data.ReadUInt32BigEndian();
h.Header2Size = data.ReadUInt32BigEndian();
h.Header2Hash = data.ReadBytes(20);
h.IsoFileSize = data.ReadUInt64BigEndian();
h.WiaFileSize = data.ReadUInt64BigEndian();
h.Header1Hash = data.ReadBytes(20);
return h;
}
private static WiaHeader2 ParseHeader2(Stream data)
{
var h = new WiaHeader2();
h.DiscType = (WiaDiscType)data.ReadUInt32BigEndian();
h.CompressionType = (WiaRvzCompressionType)data.ReadUInt32BigEndian();
h.CompressionLevel = data.ReadInt32BigEndian();
h.ChunkSize = data.ReadUInt32BigEndian();
h.DiscHeader = data.ReadBytes(0x80);
h.NumberOfPartitionEntries = data.ReadUInt32BigEndian();
h.PartitionEntrySize = data.ReadUInt32BigEndian();
h.PartitionEntriesOffset = data.ReadUInt64BigEndian();
h.PartitionEntriesHash = data.ReadBytes(20);
h.NumberOfRawDataEntries = data.ReadUInt32BigEndian();
h.RawDataEntriesOffset = data.ReadUInt64BigEndian();
h.RawDataEntriesSize = data.ReadUInt32BigEndian();
h.NumberOfGroupEntries = data.ReadUInt32BigEndian();
h.GroupEntriesOffset = data.ReadUInt64BigEndian();
h.GroupEntriesSize = data.ReadUInt32BigEndian();
h.CompressorDataSize = data.ReadByteValue();
h.CompressorData = data.ReadBytes(7);
return h;
}
#endregion
#region Table parsing
private static PartitionEntry[] ParsePartitionEntries(Stream data, int count)
{
var entries = new PartitionEntry[count];
for (int i = 0; i < count; i++)
{
var e = new PartitionEntry();
e.PartitionKey = data.ReadBytes(16);
e.DataEntry0 = ParsePartitionDataEntry(data);
e.DataEntry1 = ParsePartitionDataEntry(data);
entries[i] = e;
}
return entries;
}
private static PartitionDataEntry ParsePartitionDataEntry(Stream data)
{
var e = new PartitionDataEntry();
e.FirstSector = data.ReadUInt32BigEndian();
e.NumberOfSectors = data.ReadUInt32BigEndian();
e.GroupIndex = data.ReadUInt32BigEndian();
e.NumberOfGroups = data.ReadUInt32BigEndian();
return e;
}
private static RawDataEntry[] ParseRawDataEntries(Stream data, int count)
{
var entries = new RawDataEntry[count];
for (int i = 0; i < count; i++)
{
var e = new RawDataEntry();
e.DataOffset = data.ReadUInt64BigEndian();
e.DataSize = data.ReadUInt64BigEndian();
e.GroupIndex = data.ReadUInt32BigEndian();
e.NumberOfGroups = data.ReadUInt32BigEndian();
entries[i] = e;
}
return entries;
}
private static WiaGroupEntry[] ParseWiaGroupEntries(Stream data, int count)
{
var entries = new WiaGroupEntry[count];
for (int i = 0; i < count; i++)
{
var e = new WiaGroupEntry();
// DataOffset stored as actual_offset >> 2
e.DataOffset = (ulong)data.ReadUInt32BigEndian() << 2;
e.DataSize = data.ReadUInt32BigEndian();
entries[i] = e;
}
return entries;
}
private static RvzGroupEntry[] ParseRvzGroupEntries(Stream data, int count)
{
var entries = new RvzGroupEntry[count];
for (int i = 0; i < count; i++)
{
var e = new RvzGroupEntry();
// DataOffset stored as actual_offset >> 2
e.DataOffset = (ulong)data.ReadUInt32BigEndian() << 2;
e.DataSize = data.ReadUInt32BigEndian();
e.RvzPackedSize = data.ReadUInt32BigEndian();
entries[i] = e;
}
return entries;
}
#endregion
}
}