Files
marechai/Marechai.MobyGames/Services/CountryMatcher.cs
Natalia Portillo 3080d20cbf feat: Add IGDB mapping pipeline (Marechai.Igdb)
Adds a new console project that maps IGDB's catalog (platforms, game
types, companies, games, involved companies) to existing local
Company/Software/SoftwarePlatform records, without any enrichment or
record creation. This establishes the cross-reference table a future
IGDB enrichment phase will consume, the same way Marechai.MobyGames
already does for MobyGames.

Database (Marechai.Database):
- New mirror+mapping tables: IgdbPlatform, IgdbGameType, IgdbCompany,
  IgdbGame, IgdbInvolvedCompany. Each stores only the IGDB id, name,
  and the few relationship fields needed for matching/disambiguation
  (no descriptive fields) plus a nullable FK to the local entity and
  an IgdbMatchStatus (Pending/Matched/NoMatch/NeedsReview/Skipped/
  Error). No columns were added to Company/Software/SoftwarePlatform,
  consistent with the existing per-source mapping convention used by
  MobyGamesImportState/WwpcSoftware/OldDosSoftware.
- IgdbGame.GameTypeId is a nullable FK into the new IgdbGameType
  lookup table rather than a baked-in enum: IGDB's game_type is
  itself a dynamic lookup entity (game.category is deprecated in
  favor of it).
- An initial design mirrored IGDB's external_games/external_game_
  sources to bridge straight into the existing MobyGames->Software
  mapping. Verified directly against the live API that IGDB does not
  track MobyGames as an external source at all, so that table/service
  and the matcher's bridge step were removed again (see the two
  paired migrations).
- Migrations: AddIgdbMirrorAndMappingTables, RemoveIgdbExternalGames.

Shared helpers promoted to Marechai.Data/Helpers (pure, no EF Core
dependency, usable from a console exe without pulling in the Blazor
Marechai project or duplicating logic):
- Marechai/Helpers/JaroWinkler.cs -> Marechai.Data/Helpers/JaroWinkler.cs
- Marechai.MobyGames/Services/SoundexHelper.cs -> Marechai.Data/Helpers/SoundexHelper.cs
All consumers (Marechai's Razor pages, Marechai.MobyGames's matchers)
updated to the new namespace.

Marechai.Igdb console project:
- IgdbAuthService: Twitch OAuth2 client-credentials flow with a
  cached, auto-refreshed access token.
- IgdbHttpClient: Apicalypse requests rate-limited to the configured
  requests/second with bounded concurrency and retry-with-backoff on
  HTTP 429.
- ApicalypseQueryBuilder: small fluent query builder.
- Mirror services (platforms, game types, companies, games, involved
  companies): each run mirrors exactly one batch of up to
  --batch-size rows (clamped to IGDB's 500-row page limit) and stops,
  matching the one-batch-per-invocation convention already used by
  Marechai.MobyGames's import command. Pass --batch-size 0 to instead
  loop until the whole remaining catalog is mirrored in one
  invocation. Resumption is fully automatic: each run re-derives its
  own cursor from the database (MAX(IgdbId) already mirrored, or
  row count for the small offset-paginated lookups) - no manual
  bookkeeping - and the --batch-size 0 loop tracks its cursor in
  memory so --dry-run never gets stuck reading the same unchanged
  page forever.
- PlatformMatcher: exact -> normalized -> substring -> Jaro-Winkler
  matching against SoftwarePlatform. Local platform names follow
  MobyGames-style short/compound conventions ("Jaguar", "DOS, Windows
  and Windows 3.x") while IGDB uses one full canonical name per
  platform, so each local platform is expanded into name variants
  (the compound parts) before matching, plus a whole-word substring
  pass to bridge short-vs-full names ("Jaguar" <-> "Atari Jaguar").
- CompanyMatcher: exact name -> exact legal name -> suffix-stripped
  exact (ported from Marechai.MobyGames's CompanyMatcher) -> Soundex
  bucket confirmed by Jaro-Winkler -> full Jaro-Winkler sweep. Unlike
  the MobyGames matcher, ambiguity never prompts or auto-creates: it
  is recorded as NeedsReview (with candidates) or NoMatch, since this
  pass only establishes mapping and is meant to run unattended across
  the full catalog. Soundex-bucket candidates are filtered to a 0.90
  Jaro-Winkler floor before being considered at all - an early version
  without this floor surfaced 29,906 NeedsReview rows out of 71,210
  mirrored companies because a shared 4-character Soundex code alone
  is weak evidence and collides constantly between unrelated short
  names; with the floor and tightened accept thresholds the same
  catalog produces 2,602 matched / 139 needs-review / 68,469 no-match.
- GameMatcher: maps IGDB games to local Software (the umbrella title;
  per-platform SoftwareRelease mapping is left for a future enrichment
  phase). Edition/bundle/remaster/port-type games inherit their
  parent's SoftwareId directly instead of being matched or created
  independently, reflecting that Marechai models these as
  SoftwareRelease rows under one Software while IGDB models them as
  separate game rows linked via parent_game/version_parent. Exact name
  matches are disambiguated by platform overlap; fuzzy matches require
  platform corroboration when the candidate already has releases.
  Verified on a 1,000-game sample that this correctly avoids merging
  sequels into their base title (Quake/Quake II, DOOM II/Doom 3,
  Metal Gear/Metal Gear Solid V, etc. all left as NeedsReview rather
  than falsely matched).
- MatchStatsService / AmbiguousReviewService: stats command and an
  optional, non-blocking interactive triage command for NeedsReview
  rows.
- CLI: mirror-platforms, mirror-game-types, mirror-companies,
  mirror-games, mirror-involved-companies, match-platforms,
  match-companies, match-games, stats, review-ambiguous, reset.
  All mirror/match commands support --dry-run.

.gitignore: added Marechai.Igdb/state/ (OAuth token cache), mirroring
the existing Marechai.MobyGames/state/ entry.
2026-06-25 21:50:09 +01:00

98 lines
3.5 KiB
C#

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Marechai.Data.Helpers;
using Marechai.Database.Models;
using Microsoft.EntityFrameworkCore;
namespace Marechai.MobyGames.Services;
public class CountryMatcher
{
readonly IDbContextFactory<MarechaiContext> _contextFactory;
List<UnM49> _countries;
readonly Dictionary<string, UnM49> _cache = new(StringComparer.OrdinalIgnoreCase);
static readonly Dictionary<string, string> Aliases = new(StringComparer.OrdinalIgnoreCase)
{
["United States"] = "United States of America",
["Russia"] = "Russian Federation",
["South Korea"] = "Republic of Korea",
["North Korea"] = "Democratic People's Republic of Korea",
["The Netherlands"] = "Netherlands",
["Czechia"] = "Czechia",
["Taiwan"] = "Taiwan",
["Hong Kong"] = "China, Hong Kong Special Administrative Region",
["Macau"] = "China, Macao Special Administrative Region",
["Vietnam"] = "Viet Nam",
["Brunei"] = "Brunei Darussalam",
["Bolivia"] = "Bolivia (Plurinational State of)",
["Iran"] = "Iran (Islamic Republic of)",
["Syria"] = "Syrian Arab Republic",
["Venezuela"] = "Venezuela (Bolivarian Republic of)",
["Tanzania"] = "United Republic of Tanzania",
["Moldova"] = "Republic of Moldova",
["Ivory Coast"] = "Côte d'Ivoire",
["Laos"] = "Lao People's Democratic Republic",
["North Macedonia"] = "North Macedonia"
};
public CountryMatcher(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
public async Task LoadAsync()
{
await using var context = await _contextFactory.CreateDbContextAsync();
_countries = await context.UnM49
.Where(u => u.Type == Data.UnM49Type.Country)
.Select(u => new UnM49 { Id = u.Id, Name = u.Name })
.ToListAsync();
}
public UnM49 Match(string countryName)
{
if(string.IsNullOrWhiteSpace(countryName))
return null;
string normalized = countryName.Replace("\u00a0", " ").Trim();
// Strip continent prefix like "Africa » South Africa"
int arrowIndex = normalized.IndexOf('»');
if(arrowIndex >= 0)
normalized = normalized[(arrowIndex + 1)..].Trim();
if(_cache.TryGetValue(normalized, out var cached))
return cached;
// Try alias
if(Aliases.TryGetValue(normalized, out string aliased))
normalized = aliased;
// Exact match
var exact = _countries.FirstOrDefault(c =>
string.Equals(c.Name, normalized, StringComparison.OrdinalIgnoreCase));
if(exact != null)
{
_cache[countryName] = exact;
return exact;
}
// Soundex match
string soundex = SoundexHelper.Generate(normalized);
var matches = _countries.Where(c => SoundexHelper.Generate(c.Name) == soundex).ToList();
if(matches.Count == 1)
{
_cache[countryName] = matches[0];
return matches[0];
}
return null;
}
}