Files
marechai/Marechai.MobyGames/Services/PersonMatcher.cs

111 lines
3.4 KiB
C#
Raw Normal View History

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
feat: Add IGDB mapping pipeline (Marechai.Igdb) Adds a new console project that maps IGDB's catalog (platforms, game types, companies, games, involved companies) to existing local Company/Software/SoftwarePlatform records, without any enrichment or record creation. This establishes the cross-reference table a future IGDB enrichment phase will consume, the same way Marechai.MobyGames already does for MobyGames. Database (Marechai.Database): - New mirror+mapping tables: IgdbPlatform, IgdbGameType, IgdbCompany, IgdbGame, IgdbInvolvedCompany. Each stores only the IGDB id, name, and the few relationship fields needed for matching/disambiguation (no descriptive fields) plus a nullable FK to the local entity and an IgdbMatchStatus (Pending/Matched/NoMatch/NeedsReview/Skipped/ Error). No columns were added to Company/Software/SoftwarePlatform, consistent with the existing per-source mapping convention used by MobyGamesImportState/WwpcSoftware/OldDosSoftware. - IgdbGame.GameTypeId is a nullable FK into the new IgdbGameType lookup table rather than a baked-in enum: IGDB's game_type is itself a dynamic lookup entity (game.category is deprecated in favor of it). - An initial design mirrored IGDB's external_games/external_game_ sources to bridge straight into the existing MobyGames->Software mapping. Verified directly against the live API that IGDB does not track MobyGames as an external source at all, so that table/service and the matcher's bridge step were removed again (see the two paired migrations). - Migrations: AddIgdbMirrorAndMappingTables, RemoveIgdbExternalGames. Shared helpers promoted to Marechai.Data/Helpers (pure, no EF Core dependency, usable from a console exe without pulling in the Blazor Marechai project or duplicating logic): - Marechai/Helpers/JaroWinkler.cs -> Marechai.Data/Helpers/JaroWinkler.cs - Marechai.MobyGames/Services/SoundexHelper.cs -> Marechai.Data/Helpers/SoundexHelper.cs All consumers (Marechai's Razor pages, Marechai.MobyGames's matchers) updated to the new namespace. Marechai.Igdb console project: - IgdbAuthService: Twitch OAuth2 client-credentials flow with a cached, auto-refreshed access token. - IgdbHttpClient: Apicalypse requests rate-limited to the configured requests/second with bounded concurrency and retry-with-backoff on HTTP 429. - ApicalypseQueryBuilder: small fluent query builder. - Mirror services (platforms, game types, companies, games, involved companies): each run mirrors exactly one batch of up to --batch-size rows (clamped to IGDB's 500-row page limit) and stops, matching the one-batch-per-invocation convention already used by Marechai.MobyGames's import command. Pass --batch-size 0 to instead loop until the whole remaining catalog is mirrored in one invocation. Resumption is fully automatic: each run re-derives its own cursor from the database (MAX(IgdbId) already mirrored, or row count for the small offset-paginated lookups) - no manual bookkeeping - and the --batch-size 0 loop tracks its cursor in memory so --dry-run never gets stuck reading the same unchanged page forever. - PlatformMatcher: exact -> normalized -> substring -> Jaro-Winkler matching against SoftwarePlatform. Local platform names follow MobyGames-style short/compound conventions ("Jaguar", "DOS, Windows and Windows 3.x") while IGDB uses one full canonical name per platform, so each local platform is expanded into name variants (the compound parts) before matching, plus a whole-word substring pass to bridge short-vs-full names ("Jaguar" <-> "Atari Jaguar"). - CompanyMatcher: exact name -> exact legal name -> suffix-stripped exact (ported from Marechai.MobyGames's CompanyMatcher) -> Soundex bucket confirmed by Jaro-Winkler -> full Jaro-Winkler sweep. Unlike the MobyGames matcher, ambiguity never prompts or auto-creates: it is recorded as NeedsReview (with candidates) or NoMatch, since this pass only establishes mapping and is meant to run unattended across the full catalog. Soundex-bucket candidates are filtered to a 0.90 Jaro-Winkler floor before being considered at all - an early version without this floor surfaced 29,906 NeedsReview rows out of 71,210 mirrored companies because a shared 4-character Soundex code alone is weak evidence and collides constantly between unrelated short names; with the floor and tightened accept thresholds the same catalog produces 2,602 matched / 139 needs-review / 68,469 no-match. - GameMatcher: maps IGDB games to local Software (the umbrella title; per-platform SoftwareRelease mapping is left for a future enrichment phase). Edition/bundle/remaster/port-type games inherit their parent's SoftwareId directly instead of being matched or created independently, reflecting that Marechai models these as SoftwareRelease rows under one Software while IGDB models them as separate game rows linked via parent_game/version_parent. Exact name matches are disambiguated by platform overlap; fuzzy matches require platform corroboration when the candidate already has releases. Verified on a 1,000-game sample that this correctly avoids merging sequels into their base title (Quake/Quake II, DOOM II/Doom 3, Metal Gear/Metal Gear Solid V, etc. all left as NeedsReview rather than falsely matched). - MatchStatsService / AmbiguousReviewService: stats command and an optional, non-blocking interactive triage command for NeedsReview rows. - CLI: mirror-platforms, mirror-game-types, mirror-companies, mirror-games, mirror-involved-companies, match-platforms, match-companies, match-games, stats, review-ambiguous, reset. All mirror/match commands support --dry-run. .gitignore: added Marechai.Igdb/state/ (OAuth token cache), mirroring the existing Marechai.MobyGames/state/ entry.
2026-06-25 21:50:09 +01:00
using Marechai.Data.Helpers;
using Marechai.Database.Models;
using Microsoft.EntityFrameworkCore;
namespace Marechai.MobyGames.Services;
public class PersonMatcher
{
readonly IDbContextFactory<MarechaiContext> _contextFactory;
List<Person> _people;
Dictionary<string, List<Person>> _soundexIndex;
readonly Dictionary<string, Person> _cache = new(StringComparer.OrdinalIgnoreCase);
public PersonMatcher(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
public async Task LoadAsync()
{
await using var context = await _contextFactory.CreateDbContextAsync();
_people = await context.People
.Select(p => new Person
{
Id = p.Id,
Name = p.Name,
Surname = p.Surname,
DisplayName = p.DisplayName,
Alias = p.Alias
})
.ToListAsync();
_soundexIndex = SoundexHelper.BuildSoundexIndex(_people, p => p.FullName);
}
public async Task<Person> MatchOrCreateAsync(string fullName)
{
if(string.IsNullOrWhiteSpace(fullName))
return null;
string normalized = fullName.Replace("\u00a0", " ").Trim();
if(_cache.TryGetValue(normalized, out var cached))
return cached;
// Exact match on FullName, DisplayName, or Alias
var exact = _people.FirstOrDefault(p =>
string.Equals(p.FullName, normalized, StringComparison.OrdinalIgnoreCase) ||
string.Equals(p.DisplayName, normalized, StringComparison.OrdinalIgnoreCase) ||
string.Equals(p.Alias, normalized, StringComparison.OrdinalIgnoreCase));
if(exact != null)
{
_cache[normalized] = exact;
return exact;
}
// Soundex match
string soundex = SoundexHelper.Generate(normalized);
if(_soundexIndex.TryGetValue(soundex, out var candidates) && candidates.Count == 1)
{
_cache[normalized] = candidates[0];
return candidates[0];
}
// Split into name/surname for creation
string name, surname;
int lastSpace = normalized.LastIndexOf(' ');
if(lastSpace > 0)
{
name = normalized[..lastSpace].Trim();
surname = normalized[(lastSpace + 1)..].Trim();
}
else
{
name = normalized;
surname = "";
}
await using var context = await _contextFactory.CreateDbContextAsync();
var newPerson = new Person
{
Name = name,
Surname = surname
};
context.People.Add(newPerson);
await context.SaveChangesAsync();
_people.Add(newPerson);
if(!_soundexIndex.TryGetValue(soundex, out var list))
{
list = [];
_soundexIndex[soundex] = list;
}
list.Add(newPerson);
_cache[normalized] = newPerson;
return newPerson;
}
}