mirror of
https://github.com/claunia/marechai.git
synced 2026-07-08 17:57:08 +00:00
Natalia Portillo
3080d20cbf
Adds a new console project that maps IGDB's catalog (platforms, game
types, companies, games, involved companies) to existing local
Company/Software/SoftwarePlatform records, without any enrichment or
record creation. This establishes the cross-reference table a future
IGDB enrichment phase will consume, the same way Marechai.MobyGames
already does for MobyGames.
Database (Marechai.Database):
- New mirror+mapping tables: IgdbPlatform, IgdbGameType, IgdbCompany,
IgdbGame, IgdbInvolvedCompany. Each stores only the IGDB id, name,
and the few relationship fields needed for matching/disambiguation
(no descriptive fields) plus a nullable FK to the local entity and
an IgdbMatchStatus (Pending/Matched/NoMatch/NeedsReview/Skipped/
Error). No columns were added to Company/Software/SoftwarePlatform,
consistent with the existing per-source mapping convention used by
MobyGamesImportState/WwpcSoftware/OldDosSoftware.
- IgdbGame.GameTypeId is a nullable FK into the new IgdbGameType
lookup table rather than a baked-in enum: IGDB's game_type is
itself a dynamic lookup entity (game.category is deprecated in
favor of it).
- An initial design mirrored IGDB's external_games/external_game_
sources to bridge straight into the existing MobyGames->Software
mapping. Verified directly against the live API that IGDB does not
track MobyGames as an external source at all, so that table/service
and the matcher's bridge step were removed again (see the two
paired migrations).
- Migrations: AddIgdbMirrorAndMappingTables, RemoveIgdbExternalGames.
Shared helpers promoted to Marechai.Data/Helpers (pure, no EF Core
dependency, usable from a console exe without pulling in the Blazor
Marechai project or duplicating logic):
- Marechai/Helpers/JaroWinkler.cs -> Marechai.Data/Helpers/JaroWinkler.cs
- Marechai.MobyGames/Services/SoundexHelper.cs -> Marechai.Data/Helpers/SoundexHelper.cs
All consumers (Marechai's Razor pages, Marechai.MobyGames's matchers)
updated to the new namespace.
Marechai.Igdb console project:
- IgdbAuthService: Twitch OAuth2 client-credentials flow with a
cached, auto-refreshed access token.
- IgdbHttpClient: Apicalypse requests rate-limited to the configured
requests/second with bounded concurrency and retry-with-backoff on
HTTP 429.
- ApicalypseQueryBuilder: small fluent query builder.
- Mirror services (platforms, game types, companies, games, involved
companies): each run mirrors exactly one batch of up to
--batch-size rows (clamped to IGDB's 500-row page limit) and stops,
matching the one-batch-per-invocation convention already used by
Marechai.MobyGames's import command. Pass --batch-size 0 to instead
loop until the whole remaining catalog is mirrored in one
invocation. Resumption is fully automatic: each run re-derives its
own cursor from the database (MAX(IgdbId) already mirrored, or
row count for the small offset-paginated lookups) - no manual
bookkeeping - and the --batch-size 0 loop tracks its cursor in
memory so --dry-run never gets stuck reading the same unchanged
page forever.
- PlatformMatcher: exact -> normalized -> substring -> Jaro-Winkler
matching against SoftwarePlatform. Local platform names follow
MobyGames-style short/compound conventions ("Jaguar", "DOS, Windows
and Windows 3.x") while IGDB uses one full canonical name per
platform, so each local platform is expanded into name variants
(the compound parts) before matching, plus a whole-word substring
pass to bridge short-vs-full names ("Jaguar" <-> "Atari Jaguar").
- CompanyMatcher: exact name -> exact legal name -> suffix-stripped
exact (ported from Marechai.MobyGames's CompanyMatcher) -> Soundex
bucket confirmed by Jaro-Winkler -> full Jaro-Winkler sweep. Unlike
the MobyGames matcher, ambiguity never prompts or auto-creates: it
is recorded as NeedsReview (with candidates) or NoMatch, since this
pass only establishes mapping and is meant to run unattended across
the full catalog. Soundex-bucket candidates are filtered to a 0.90
Jaro-Winkler floor before being considered at all - an early version
without this floor surfaced 29,906 NeedsReview rows out of 71,210
mirrored companies because a shared 4-character Soundex code alone
is weak evidence and collides constantly between unrelated short
names; with the floor and tightened accept thresholds the same
catalog produces 2,602 matched / 139 needs-review / 68,469 no-match.
- GameMatcher: maps IGDB games to local Software (the umbrella title;
per-platform SoftwareRelease mapping is left for a future enrichment
phase). Edition/bundle/remaster/port-type games inherit their
parent's SoftwareId directly instead of being matched or created
independently, reflecting that Marechai models these as
SoftwareRelease rows under one Software while IGDB models them as
separate game rows linked via parent_game/version_parent. Exact name
matches are disambiguated by platform overlap; fuzzy matches require
platform corroboration when the candidate already has releases.
Verified on a 1,000-game sample that this correctly avoids merging
sequels into their base title (Quake/Quake II, DOOM II/Doom 3,
Metal Gear/Metal Gear Solid V, etc. all left as NeedsReview rather
than falsely matched).
- MatchStatsService / AmbiguousReviewService: stats command and an
optional, non-blocking interactive triage command for NeedsReview
rows.
- CLI: mirror-platforms, mirror-game-types, mirror-companies,
mirror-games, mirror-involved-companies, match-platforms,
match-companies, match-games, stats, review-ambiguous, reset.
All mirror/match commands support --dry-run.
.gitignore: added Marechai.Igdb/state/ (OAuth token cache), mirroring
the existing Marechai.MobyGames/state/ entry.