He brought this to my attention because somehow some items are getting compared by CRC only. This is wrong on a lot of levels, but this should be a step in the right direction.
Basically, if a rom is a duplicate, it can be a duplicate within a system or source or across system or source, and by hash alone or by all data matching. The four combinations make an enum now and the diff function is the only one that uses them right now. If we're in dedup mode, we want to check hashes only in diff. Otherwise, we want to check against ones that match all information. This needs field testing.
This change makes it mostly functional, having a way to both parse and merge in one step and output from the database. The issue is that the output from Parse2 doesn't match Parse OR Parse + Merge. Duplicates don't seem to be added in the first case and there are a differing number of roms in the second.
The new merging process is both parse and merge in one step using an in-memory database. The next steps are to get the data in this database and write it out properly.
This code for parsing needed to be changed because of large file handling. XmlTextReader ended up being the best solution that doesn't crash at 500 MB in. A side effect of this change is that the code for processing a given DAT is very minimal now. The entire sub-block for SL-specific DAT info doesn't need to exist anymore. Small-scale tests have given 1:1 results with previous versions but further testing is needed for large files to make sure no data is lost.
This is currently borken for reasons that I don't 100% understand. It needs to be tested further with a basic DAT to see why it doesn't seem to get every ROM.
