SabreTools

************************************************
* SabreTools - DAT management software *
* https://github.com/mnadareski/wizzardDesktop *
************************************************

Table of Contents
-----------------
1.0 Introduction and History
2.0 Included Programs
2.1 RombaSharp
2.2 SabreTools
2.3 SimpleSort
3.0 Examples
4.0 Contributors
5.0 Licensing

** Section 1.0 - Introduction and History

Hello and welcome to the latest release of SabreTools! We are happy that you
chose to use our software for your DAT management needs. But what exactly is
SabreTools? How did it get started? Why is it named so weirdly? Why should we
care? Well, the first two can be answered, though the last one is up to you.

SabreTools has its roots in an internal tool developed by RomShepherd member
The Wizard of DATz (WoD). The aim of the PHP-based program was to collate data
frommultiple sources and create new DAT files for custom sources as well as
DAT files that have duplicate data removed. WoD maintained this software and
used it to create near-monthly releases for quite a while.

Around the middle-end of 2015, WoD announced that they would be taking a break
from the process of creating and releasing new files. A couple of months went
by with no information. Finally, WoD finally told the community that they would
no longer be able to maintain either the project or the software involved,
releasing the source code of the internal project in the process. During that time,
RomShepherd member darksabre76 picked it up to see how the program even worked.

After updating the code to the newest version of PHP and trying to figure out
what each page did, he announced that it was going to be easier to do a complete
rewrite while maintaining functionality. He was given a lot of support and got
started with the aim to keep the code completely open source so that anyone can
help. It took a lot of work and help from fellow user emuLOAD, but the code
finally started to shape up in the form of WoD Redux. This project is still
active and can be found at https://github.com/mnadareski/wizzardRedux.

After about a month of work on this, and continued support from the community,
a relatively new user to the site, now known as @tractivo, sent him a message
and asked if a desktop port of some of the core features was possible. It was
difficult for most people to run the PHP code locally, so it made sense. With
the aim for people to use the web version primarlily, the desktop version,
originally called DATabase, was written. Pretty soon, this got even more support
and soon became the main focus. For a while, code parity between the C# and PHP
versions was attempted, but it became apparent that the desktop version was going
to be the better option.

Development accelerated, soon leaving the PHP version in the dust. Once the basics
were implemented, more features were requested, including better merging and DAT
management. Spurred on by an increasing number of community members, DATabase
accrued a lot of features and helper programs. The accuracy of the program also
increased, soon matching or surpassing existing options in functionality. At some
point during this development, some of the smaller tools got wrapped into DATabase,
slowly getting further from just a desktop port of WoD Redux.

With these changes, some external prodding, and a little bit of ego, darksabre76
decided to rechristen the set of programs to SabreTools. This freed the program
from being just about DAT management and allowed it to be more of a Swiss Army
Knife of DAT tools. Development has been on and off since then, and is still
constantly evolving to this day (assuming that at this very moment development is
still active).

** Section 2.0 - Included Programs

Below are a list of the programs that are included in the current SabreTools
release. Each of them have a brief description of the tool along with in-depth
desciptions of all flags.

** Section 2.1 - RombaSharp

RombaSharp is an ongoing "spiritual port" of the Romba tool (https://github.com/uwedeportivo/romba).
The code is not based on the actual source, rather taking the features and using the code
already written for a lot of other features. The following descriptions are based on what WILL
be done with each flag. Not all features are currently available.

Usage:
RombaSharp.exe [options] [filename|dirname] ...

Options:
-?, -h, --help Show the built-in help text
Built-in to most of the programs is a basic help text

archive Adds ROM files from the specified directories to depot
-only-needed Only archive ROM files in database
build For each specified DAT file it creates TZip files
dbstats Prints db stats
diffdat Creates a DAT file for entries found in the new DAT
-new= DAT to compare to
dir2dat Creates a DAT file for the specified input directory
-out= Filename to save out to
fixdat For each specified DAT file it creates a fix DAT
lookup For each specified hash, look up available information
memstats Prints memory stats
miss For each specified DAT file, create miss and have file
progress Shows progress of currently running command [OBSOLETE]
purge-backup Moves DAT index entries for orphaned DATs
purge-delete Deletes DAT index entries for orphaned DATs
refresh-dats Refreshes the DAT index from the files in the DAT root
shutdown Gracefully shuts down server [OBSOLETE]

** Section 2.2 - SabreTools

SabreTools is the main application of the SabreTools suite. It is mostly just a frontend
to a lot of features that are available in the DLL and can be considered the reference
implementation. As such, it has a lot of possible flags and options that a user can select
from.

Included within this tool are a few former standalone executables:
- Convert/DATToMiss: Convert an arbitrary input DAT to a different format
- DATFromDir: Create a DAT file from a folder or file, sometimes called dir2dat
- DatSplit: Split a DAT based on 2 different file extensions
- Filter: Filter a DAT based on various user-defined criteria, optionally using wildcards
- HashSplit: Split a DAT based on the best available hash (No-dump, SHA-1, MD5, CRC)
- MergeDAT: Merge and optionally dedupe an arbitrary number of DAT files
- SingleGame: Trim game and rom names to fit NTFS length standards, optionally merging
all roms to a single game named "!" and forcing unpack
- UncompressedSize: Get statistics from one or more input DATs, including number of
roms, disks, files with available hash, and size

Formerly included within this tool is a former standalone executable:
- DATabase/DATabaseTwo: A managed DAT tool that allows for creating automatically merged
DATs based on one or more systems, sources, or a combination thereof

Usage:
SabreTools.exe [options] [filename|dirname] ...

Options:
-?, -h, --help Show the built-in help text
Built-in to most of the programs is a basic help text

-d, --dfd Create a DAT from each input directory
Create a DAT file from an input directory or set of files. By default, this will
output a DAT named based on the input directory and the current date. It will also
treat all archives as possible games and add all three hashes for each file.

-nm, --noMD5 Don't include MD5 in output
This allows the user to skip calculating the MD5 for each of the files which will
speed up the creation of the DAT.

-ns, --noSHA1 Don't include SHA1 in output
This allows the user to skip calculating the SHA-1 for each of the files which will
speed up the creation of the DAT.

-b, --bare Don't include date in file name
Normally, the DAT will be created with the date in the file name. This flag removes
that but keeps the date tag intact.

-u, --unzip Force unzipping in created DAT
This sets the 'forcepacking="unzip"' flag in the outputted DAT. When used with a
file manager that allows for it, this will force the outputted files to be in
subdirectories instead of archives.

-f, --files Treat archives as files
Instead of trying to enumerate the files within archives, treat the archives as
files themselves. This is good for uncompressed sets that include archives that
should be read as-is.

-oc, --output-cmp Output in CMP format
Add outputting the created DAT to CLRMamePro format

-od, --output-dc Output in DOSCenter format
Add outputting the created DAT to DOSCenter format

-om, --output-miss Output in Missfile format
Add outputting the created DAT to GoodTools miss format

-omd5, --output-md5 Output in MD5 format
Add outputting the created DAT to MD5 format

-or, --output-rc Output in RomCenter format
Add outputting the created DAT to RomCenter format

-os, --output-sd Output in SabreDAT format
Add outputting the created DAT to SabreDAT XML format

-osfv, --ouput-sfv Output in SFV format
Add outputting the created DAT to SFV format

-osha1, --output-sha1 Output in SHA-1 format
Add outputting the created DAT to SHA1 format

-ox, --output-xml Output in Logiqx XML format (default)
Add outputting the created DAT to Logiqx XML format

-gz, --gz-files Allow reading of GZIP files as archives
Since GZip files are not commonly used for file storage, this flag allows for
any GZip archives to have their contents hashed instead.

-ro, --romba Read files from a Romba input
Allow for reading of GZipped files as if they were from a Romba depot. This
implies that the files will be in the TorrentGZ format as well, including
naming convention.

-f=, --filename= Set the external name of the DAT
Set the base filename for the output DAT(s) [default is folder name plus date]

-n=, --name= Set the internal name of the DAT
Set the internal name for the output DAT(s) [default is folder name plus date]

-de=, --desc= Set the description of the DAT
Set the description for the output DAT(s) [default is the folder name]

-c=, --cat= Set the category of the DAT
Set the category for the output DAT(s) [default is blank]

-v=, --version= Set the version of the DAT
Set the version for the output DAT(s) [default is blank]

-au=, --author= Set the author of the DAT
Set the author for the output DAT(s) [default is blank]

-sd, --superdat Enable SuperDAT creation
Set the type flag to "SuperDAT" for the output DAT as well as preserving the
directory structure of the inputted folder, if applicable

-ab, --add-blank Output blank files for folders
If this flag is set, then blank entries will be created for each of the empty
directories in the source. This is useful for tools that require all folders
be accounted for in the output DAT.

-ad, --add-date Output dates for each file parsed
If this flag is set, then the Date will be appended to each file information
in the output DAT. The output format is standardized as "yyyy/MM/dd HH:mm:ss".

-t=, --temp= Set the name of the temporary directory
Optionally, a temp folder can be supplied in the case the default temp directory
(inside the running folder) is not preferred. This is used for any operations
that require an archive to be extracted.

-mt={4} Amount of threads to use
Optionally, set the number of threads to use for the multithreaded operations.
The default is 4 threads; -1 means unlimited threads created. If the user specifies
that only 1 thread is to be used, it defaults to the original, serial implementation
of the DFD code.

-es, --ext-split Split a DAT by two file extensions
For a DAT, or set of DATs, allow for splitting based on a list of input extensions.
This can allow for combined DAT files, such as those combining two separate systems,
to be split. Files with any extensions not listed in the input lists will be included
in both outputted DAT files.

-exta= First set of extensions (comma-separated)
Set the extensions to be used to populate the first DAT. If more than one
extension is defined, they must be separated by commas.

-extb= Second set of extensions (comma-separated)
Set the extensions to be used to populate the second DAT. If more than one
extension is defined, they must be separated by commas.

-out= Set the name of the output directory
This sets an output folder to be used when the files are created. If a path
is not defined, the application directory is used instead.

-hd, --headerer Backup or restore copier headers from a variety of file types
Headerer is meant as an intermediary between header skipper files (which, a bit
apart from their name, do not just show how to skip copier headers) and rom managers
that do not use them.

By default, this will detect, store, and remove copier headers from a file or folder
of files. The headers are backed up and collated by the hash of the unheadered file.
Files are then output without the detected copier header alongside the originals with
the suffix .new. No input files are altered in the process.

The following systems have headers that this program can work with:
- Atari 7800
- Atari Lynx
- Commodore PSID Music
- NEC PC-Engine / TurboGrafx 16
- Nintendo Famicom / Nintendo Entertainment System
- Nintendo Famicom Disk System
- Nintendo Super Famicom / Super Nintendo Entertainment System
- Nintendo Super Famicom / Super Nintendo Entertainment System SPC Music

-re, --restore Restore headers to file(s)
Instead of the default extraction, this flag enables use of stored copier headers
to reapply them to files if they match the included hash. More than one header can
be applied to a file, so they will be output to new files, suffixed with .newX,
where X is a number. No input files are altered in the process.

-out= Set the name of the output directory
This sets an output folder to be used when the files are created. If a path
is not defined, the application directory is used instead.

-hs, --hash-split Split a DAT or folder by best-available hashes
For a DAT, or set of DATs, allow for splitting based on the best available hash for
each file within. The order of preference for the outputted DATs is as follows:
- Nodump
- SHA-1 available
- MD5 available
- CRC or worse available

-out= Set the name of the output directory
This sets an output folder to be used when the files are created. If a path
is not defined, the application directory is used instead.

-input= Set an input string
This should only be used if one of the inputs starts with a flag or another already
defined input.

-st, --stats Get statistics on all input DATs
This will output by default the combined statistics for all input DAT files. The stats
that are outputted are as follows:
- Total uncompressed size
- Number of games found
- Number of roms found
- Number of disks found
- Roms that include a CRC
- Roms that include a MD5
- Roms that include a SHA-1
- Roms with Nodump status

-si, --single Show individual statistics
Optionally, the statistics for each of the individual input DATs can be output
as well. This can be useful to show where the size or amount of files found
in the combined totals can be broken down from.

-ts, --type-split Split a DAT or folder by file types (rom/disk)
For a DAT, or set of DATs, allow for splitting based on the types of the files,
specifically if the type is a rom or a disk.

-out= Set the name of the output directory
This sets an output folder to be used when the files are created. If a path
is not defined, the application directory is used instead.

-ud, --update Update a DAT file
This is the multitool part of the program, allowing for almost every manipulation
to a DAT, or set of DATs. This is also a combination of many different programs
that performed DAT manipulation that work better together.

-oc, --output-cmp Output in CMP format
Add outputting the created DAT to clrmamepro format

-od, --output-dc Output in DOSCenter format
Add outputting the created DAT to DOSCenter format

-om, --output-miss Output in Missfile format
Add outputting the created DAT to GoodTools miss format

-r, --roms Output roms to miss instead of sets
By default, the outputted file will include the name of the game, so this
flag allows for the name of the rom to be output instead.

-gp, --game-prefix Add game name as a prefix
Mainly used with the previous flag, this allows for the name of the game
to be used as a prefix to each file.

-pre=, --prefix= Set prefix for all lines
Set a generic prefix to be prepended to all outputted lines

-post=, --postfix= Set postfix for all lines
Set a generic postfix to be appended to all outputted lines

Both prefix and postfix can use one of the following special strings:
- %game% - Replaced with the Game/Machine name
- %name% - Replaced with the Rom name
- %crc% - Replaced with the CRC
- %md5% - Replaced with the MD5
- %sha1% - Replaced with the SHA-1
- %size% - Replaced with the size

-q, --quotes Put double-quotes around each item
This quotes only the item and not the prefix and postfix

-ae=, --add-ext= Add an extension to each item
To each item, a postfixed extension is added

-re=, --rep-ext= Replace all extensions with specified
When an extension exists, replace it with the provided instead

-rme, --rem-ext Remove all extensions from all items
For each item, the extension is removed

-ro, --romba Output in Romba format (requires SHA-1)
Instead of outputting the game or rom name, output the SHA-1 of the files
instead. This requires the source DAT to have SHA-1 hashes.

-tsv, --tsv Output in Tab-Separated Value format
Output all rom information in standardized TSV format

-csv, --csv Output in Comma-Separated Value format
Output all rom information in standardized CSV format

-omd5, --output-md5 Output in MD5 format
Add outputting the created DAT to MD5 format

-or, --output-rc Output in RomCenter format
Add outputting the created DAT to RomCenter format

-os, --output-sd Output in SabreDAT format
Add outputting the created DAT to SabreDAT XML format

-osfv, --ouput-sfv Output in SFV format
Add outputting the created DAT to SFV format

-osha1, --output-sha1 Output in SHA-1 format
Add outputting the created DAT to SHA1 format

-ox, --output-xml Output in Logiqx XML format (default)
Add outputting the created DAT to Logiqx XML format

-f=, --filename= Set the external name of the DAT
Set the base filename for the output DAT(s)

-n=, --name= Set the internal name of the DAT
Set the internal name for the output DAT(s)

-de=, --desc= Set the description of the DAT
Set the description for the output DAT(s)

-r=, --root= Set a new rootdir
Set the rootdir (as used by SuperDAT mode) for the output DAT(s)

-c=, --cat= Set the category of the DAT
Set the category for the output DAT(s)

-v=, --version= Set the version of the DAT
Set the version for the output DAT(s)

-da=, --date= Set a new date
Set the date for the output DAT(s)

-au=, --author= Set the author of the DAT
Set the author for the output DAT(s)

-em=, --email= Set a new email
Set the email for the output DAT(s)

-hp=, --homepage= Set a new homepage
Set the homepage for the output DAT(s)

-u=, --url= Set a new URL
Set the URL for the output DAT(s)

-co=, --comment= Set a new comment
Set the comment for the output DAT(s)

-h=, --header= Set a new header skipper
Set the header skipper for the output DAT(s)

-sd, --superdat Enable SuperDAT creation
Set the type flag to "SuperDAT" for the output DAT

-fm=, --forcemerge= Set force merging
Set the forcemerge tag to one of the supported values:
None, Split, Full

-fn=, --forcend= Set force nodump
Set the forcenodump tag to one of the supported values:
None, Obsolete, Required, Ignore

-fp=, --forcepack= Set force packing
Set the forcepacking flag to one of the supported values:
None, Zip, Unzip

-clean Clean game names according to WoD standards
Game names will be santitized to remove what the original WoD standards
deemed as unneeded information, such as parenthized or bracketed strings

-sl, --softlist Use Software List name instead of description
By default, software list DATs are treated as "incorrect", using the game
descriptions as the name instead, since they tend to be more descriptive.
Enabling this flag allows for the original name to be preserved and keeping
the description as just a description.

-trim Trim file names to fit NTFS length
In the cases where files will have too long a name, this allows for trimming
the name of the files to the NTFS maximum length at most

-rd=, --root-dir= Set the root directory for calculation
In the case that the files will not be stored from the root directory,
a new root can be set for path length calculations

-si, --single All game names replaced by '!'
This is useful for keeping all roms in a DAT in the same archive or folder

-dd, --dedup Enable deduping in the created DAT
For all outputted DATs, allow for hash deduping. This makes sure that there
are effectively no duplicates in the output files.

-m, --merge Merge the input DATs
By default, all DATs are processed individually with the user-specified flags.
With this flag enabled, all of the input DATs are merged into a single output.
This is best used with the dedupe flag.

-b, --bare Don't include date in file name
Normally, the DAT will be created with the date in the file name. This flag
removes that instead of the default.

-di, --diff Create diffdats from inputs (all outputs)
By default, all DATs are processed individually with the user-specified flags.
With this flag enabled, input DATs are diffed against each other in one of the
following ways (flags below are used for specific diff files to be output instead
of all types at once):

-did, --diff-du Create diffdat containing just duplicates
All files that have duplicates outside of the original DAT are included

-dii, --diff-in Create diffdats for individual DATs
All files that have no duplicates outside of the original DATs are put
into DATs that are named after the source DAT

-din, --diff-nd Create diffdat containing no duplicates
All files that have no duplicates outside of the original DATs are included

-b, --bare Don't include date in file name
Normally, the DAT will be created with the date in the file name. This flag
removes that instead of the default.

-c, --cascade Enable cascaded diffing
-rc, --rev-cascade Enable reverse cascaded diffing
Each of the above flags allow for a special type of diffing in which the first
(or last) DAT is considered a base, and for each additional input DAT, it only
leaves the files that are not in one of the previous DATs. This can allow for
the creation of rollback sets or even just reduce the amount of duplicates
across multiple sets

-ip, --inplace Enable inplace, cascaded diff
This will overwrite the source files instead of writing them out to the
program folder by default (or the output folder if overridden)

-sf, --skip Skip output of first DAT
In times where the first DAT does not need to be written out a second time,
this will skip writing it. This can often speed up the output process.

-gn=, --game-name= Filter by game name
-rn=, --rom-name= Filter by rom name
-crc=, --crc= Filter by CRC hash
-md5=, --md5= Filter by MD5 hash
-sha1=, --sha1= Filter by SHA-1 hash
For each of the flags above, the user can specify either an exact match or can use a
wildcard as defined below (case-insensitive):
*00 means ends with '00'
00* means starts with '00'
*00* means contains '00'
00 means exactly equals '00'

-rt=, --rom-type= Filter by rom type
This allows users to only include roms or disks to their liking

-sgt=, --greater= Filter by size >=
-slt=, --less= Filter by size <=
-seq=, --equal= Filter by size ==
For each of the flags above, the user can specify a standard integer or one of the
various standard postfixes for size:
e.g. 8kb => 8000 or 8kib => 8192

-nd, --nodump Include only match nodump roms
-nnd, --not-nodump Exclude all nodump roms
These flags allow for filtering based on the nodump status in the source DAT(s)

-out= Set the name of the output directory
This sets an output folder to be used when the files are created. If a path
is not defined, the application directory is used instead.

** Section 2.3 - SimpleSort

SimpleSort is a WIP program that is meant as a command-line tool to quickly rebuild and
verify files based on a supplied DAT file. The eventual aim for this program is to have
a full rom management tool without a GUI, though this may not happen for a while.

Usage:
SimpleSort.exe [options] [filename|dirname] ...

Options:
-?, -h, --help Show the built-in help text
Built-in to most of the programs is a basic help text

-dat= Name of the DAT to be used for the various options
This DAT file is required for everything that SimpleSort currently does, so not
supplying one will result in an error. Depending on the additional flags that are
supplied, the DAT will be used differently, as described below. By default, the
DAT is used to check which files need to be rebuilt.

-out= Set the name of the output directory
This sets an output folder to be used by various parts of the program. As with the
DAT file, this input is used differently based on the flags that are supplied. By
default, the output folder is used as the target to rebuild files to.

-d, --delete Enable deletion of the input files
Optionally, the input files, once processed, can be deleted. This can be useful
when the original file structure is no longer needed or if there is limited space
on the source drive.

-qs, --quick Enable quick scanning of archives
For all archives, if this flag is enabled, it will only use the header information
to get the archive entries' file information. The upside to this is that it is much
quicker than extracting all files to the temp folder. On the downside, it can only
get the CRC and size from most archive formats, leading to possible issues.

-v, --verify Enable verification of output directory
This overrides the default rebuilding and only requires the DAT and the output folder.
Here, the DAT is used to verify the output directory directly and then output a
simple FixDAT. This can be misleading, currently, because it only checks for exact
matches.

-c, --convert Enable conversion of input files to TGZ
This allows conversion of a folder or set of folders to TorrentGZ format without
requiring a DAT to rebuild from. It is only useful in a small amount of situations at
the present, but it is mostly meant for Romba compatibility at the present.

-tzip Enable TorrentZip output
Instead of outputting the files to standard ZIP archives, files will be rebuilt to
TorrentZip (TZ) files. This format is based on the ZIP archive format, but with
custom header information. This is primarily used by external tool RomVault
(http://www.romvault.com/) and is already widely used.

-tgz Enable Torrent GZ output
Instead of outputting the files to ZIP archives, files will be rebuilt to TorrentGZ
(TGZ) files. This format is based on the GZip archive format, but with custom header
information and a file name replaced by the SHA-1 of the file inside. This is
primarily used by external tool Romba (https://github.com/uwedeportivo/romba), but
may be used more widely in the future.

-r, --romba Enable Romba depot directory output
As an extension of the parent flag, this outputs the TGZ files into directories
based on the structure used by Romba. This uses nested folders using the first
4 bytes of the SHA-1, 1 byte for each layer of the directory name. It also
includes two auxilary files, .romba_size and .romba_size.backup, that have the
compressed size of the folder inside for use with Romba.

-do, --directory Enable outputting files uncompressed
Instead of outputting the files to ZIP archives, files will be rebuilt to named
subdirectories within the output folder. This is useful for when the DAT does not
already have the flag specified.

-7z={0} Set scanning level for 7z archives
-gz={2} Set scanning level for GZip archives
-rar={2} Set scanning level for RAR archives
-zip={0} Set scanning level for ZIP archives
For each of the major archive types recognized by the libraries used by this
program, scan the archive in one of the following ways:
0 Hash both archive and its contents
1 Only hash contents of the archive
2 Only hash archive itself (treat like a regular file)

-ud, --update-dat Output updated DAT (rebuild only)
Once the files that were able to rebuilt are taken care of, a DAT of the files
that could not be matched will be output to the program directory.

** Section 3.0 - Examples

Here, any user-requested examples will go

** Section 4.0 - Contributors

Programmer / Lead: Matt Nadareski (darksabre76)
Additional code: emuLOAD, @tractivo
Testing: emuLOAD, @tractivo, Kludge, Obiwantje, edc
Suggestions: edc, AcidX, Amiga12, EliUmniCk
Based on work by: The Wizard of DATz

We welcome any contributors for coding, suggestions, optimizations, critisism,
heckling, abject anger, praise, and/or apathy.

** Section 5.0 - Licensing

This program uses, in part or in whole, code, libraries, and/or applications
from the 7-zip project (www.7-zip.org). 7-zip is licenced under the GNU LGPL.

The preceeding programs use, in part or in whole, code, libraries, and/or applications from DotNetSharp (https://dotnetzip.codeplex.com). DotNetSharp is licensed under the Microsoft Public License of October 2006.

All other external code is marked as such within the source and correctly
attributed to the site and/or person(s) that originally wrote the code. All
code written by project members is licensed under GPL v3. See LICENSE for
more details.