ParseNoIntro is now a very specialized page. Instead of trying to get
ALL information from a page, it just captures the scene release info
instead. To make it even more efficient, it uses an XML reader to strip
out a lot of the junk, along with a couple levels of stripping
unnecessary whitespace.
The result is a very robust page that can traverse and get information
for Nintendo DS ONLY. Further tests will be done for 3DS in the future.
For now, it will remain limited until this information can be written to
file, which I hope will come in an update soon.
If a rom or disk is found to be a duplicate (by using the actual last
item in the list instead of unnecessary cached values), it then checks
to see if any information is missing from the rom it's a duplicate of.
If the CRC, MD5, or SHA1 are empty and a new value is available, it uses
the new value dynamically. This way, the original information in the
database is untouched but the generated DAT is more complete. This is a
backport from the desktop version.
Since the description wasn't covered before, it had the possibility of
messing up imports that had a ")" in the description (AKA all of the
No-Intro DATs). This was not an issue in the custom DATs because none of
them included descriptions.
The desktop version had smart code that allowed it to figure out
(usually) the type of the DAT based on its filename pattern. The
previous PHP code used a param to deterime the type of the DAT, which,
though it's more distinctive, leads to having different folders for each
type, and doesn't allow all types to be imported at once. The new code
here uses the smart sorting to determine filetype and then uses that
type as if it were a param.
The code that is back here is loosely based on the original 3 step code
from previous iterations. This is a backport of the optimized code used
in the desktop version. Here, there is only at most one sort as the
query gets the correct sorting order for merged and unmerged. This code
needed to be implemented because of the varying degrees of available
data on any given ROM from different sources. Once other sources (such
as TOSEC) begin to be added, this will become apparent.
This function is currently unused because local tests have been
inconclusive if it even works properly. This is based on code that is
found in the desktop version which functions in the correct manner.
This is another backport from the desktop version which is somewhat
based on feedback from nombre002/@tractivo. In another turn of
coincidence so far, all WoD custom DATs (the testing ground for most of
the app) all have CRC, MD5, and SHA1. This is not the case for all sets;
some have only CRC and MD5 or CRC and SHA1 (or even just CRC). With the
original code, these duplicates would be left in because they didn't
match in every single aspect. This is a tenative fix that may have
further fixes that are coordinated with the desktop version in the
future.
It turns out that RV is VERY particular about the hashes that it is able
to parse. I've been lucky in the testing so far that this has not come
up, but this is actually a backport from the desktop version where
nombre002/@tractivo found this issue and reported it.
1) Always allow DAT creation to take as much memory as needed (this is a
bad idea in the long run but it won't work otherwise)
2) Clean up the two merged queries from the previous checkin. The way
that it was merged effectively stopped the program if either $sources or
$systems, wasn't set which had a high likelihood of happening.
Two blocks of code essentially did the same thing at slightly different
times. By combining them (and adding some functionality), they now cut
down on the amount of database calls being done. Tests have shown that
even the biggest DATs take very little time to create.
