mirror of
https://github.com/claunia/marechai.git
synced 2026-07-08 17:57:08 +00:00
Adds a new console project that maps IGDB's catalog (platforms, game
types, companies, games, involved companies) to existing local
Company/Software/SoftwarePlatform records, without any enrichment or
record creation. This establishes the cross-reference table a future
IGDB enrichment phase will consume, the same way Marechai.MobyGames
already does for MobyGames.
Database (Marechai.Database):
- New mirror+mapping tables: IgdbPlatform, IgdbGameType, IgdbCompany,
IgdbGame, IgdbInvolvedCompany. Each stores only the IGDB id, name,
and the few relationship fields needed for matching/disambiguation
(no descriptive fields) plus a nullable FK to the local entity and
an IgdbMatchStatus (Pending/Matched/NoMatch/NeedsReview/Skipped/
Error). No columns were added to Company/Software/SoftwarePlatform,
consistent with the existing per-source mapping convention used by
MobyGamesImportState/WwpcSoftware/OldDosSoftware.
- IgdbGame.GameTypeId is a nullable FK into the new IgdbGameType
lookup table rather than a baked-in enum: IGDB's game_type is
itself a dynamic lookup entity (game.category is deprecated in
favor of it).
- An initial design mirrored IGDB's external_games/external_game_
sources to bridge straight into the existing MobyGames->Software
mapping. Verified directly against the live API that IGDB does not
track MobyGames as an external source at all, so that table/service
and the matcher's bridge step were removed again (see the two
paired migrations).
- Migrations: AddIgdbMirrorAndMappingTables, RemoveIgdbExternalGames.
Shared helpers promoted to Marechai.Data/Helpers (pure, no EF Core
dependency, usable from a console exe without pulling in the Blazor
Marechai project or duplicating logic):
- Marechai/Helpers/JaroWinkler.cs -> Marechai.Data/Helpers/JaroWinkler.cs
- Marechai.MobyGames/Services/SoundexHelper.cs -> Marechai.Data/Helpers/SoundexHelper.cs
All consumers (Marechai's Razor pages, Marechai.MobyGames's matchers)
updated to the new namespace.
Marechai.Igdb console project:
- IgdbAuthService: Twitch OAuth2 client-credentials flow with a
cached, auto-refreshed access token.
- IgdbHttpClient: Apicalypse requests rate-limited to the configured
requests/second with bounded concurrency and retry-with-backoff on
HTTP 429.
- ApicalypseQueryBuilder: small fluent query builder.
- Mirror services (platforms, game types, companies, games, involved
companies): each run mirrors exactly one batch of up to
--batch-size rows (clamped to IGDB's 500-row page limit) and stops,
matching the one-batch-per-invocation convention already used by
Marechai.MobyGames's import command. Pass --batch-size 0 to instead
loop until the whole remaining catalog is mirrored in one
invocation. Resumption is fully automatic: each run re-derives its
own cursor from the database (MAX(IgdbId) already mirrored, or
row count for the small offset-paginated lookups) - no manual
bookkeeping - and the --batch-size 0 loop tracks its cursor in
memory so --dry-run never gets stuck reading the same unchanged
page forever.
- PlatformMatcher: exact -> normalized -> substring -> Jaro-Winkler
matching against SoftwarePlatform. Local platform names follow
MobyGames-style short/compound conventions ("Jaguar", "DOS, Windows
and Windows 3.x") while IGDB uses one full canonical name per
platform, so each local platform is expanded into name variants
(the compound parts) before matching, plus a whole-word substring
pass to bridge short-vs-full names ("Jaguar" <-> "Atari Jaguar").
- CompanyMatcher: exact name -> exact legal name -> suffix-stripped
exact (ported from Marechai.MobyGames's CompanyMatcher) -> Soundex
bucket confirmed by Jaro-Winkler -> full Jaro-Winkler sweep. Unlike
the MobyGames matcher, ambiguity never prompts or auto-creates: it
is recorded as NeedsReview (with candidates) or NoMatch, since this
pass only establishes mapping and is meant to run unattended across
the full catalog. Soundex-bucket candidates are filtered to a 0.90
Jaro-Winkler floor before being considered at all - an early version
without this floor surfaced 29,906 NeedsReview rows out of 71,210
mirrored companies because a shared 4-character Soundex code alone
is weak evidence and collides constantly between unrelated short
names; with the floor and tightened accept thresholds the same
catalog produces 2,602 matched / 139 needs-review / 68,469 no-match.
- GameMatcher: maps IGDB games to local Software (the umbrella title;
per-platform SoftwareRelease mapping is left for a future enrichment
phase). Edition/bundle/remaster/port-type games inherit their
parent's SoftwareId directly instead of being matched or created
independently, reflecting that Marechai models these as
SoftwareRelease rows under one Software while IGDB models them as
separate game rows linked via parent_game/version_parent. Exact name
matches are disambiguated by platform overlap; fuzzy matches require
platform corroboration when the candidate already has releases.
Verified on a 1,000-game sample that this correctly avoids merging
sequels into their base title (Quake/Quake II, DOOM II/Doom 3,
Metal Gear/Metal Gear Solid V, etc. all left as NeedsReview rather
than falsely matched).
- MatchStatsService / AmbiguousReviewService: stats command and an
optional, non-blocking interactive triage command for NeedsReview
rows.
- CLI: mirror-platforms, mirror-game-types, mirror-companies,
mirror-games, mirror-involved-companies, match-platforms,
match-companies, match-games, stats, review-ambiguous, reset.
All mirror/match commands support --dry-run.
.gitignore: added Marechai.Igdb/state/ (OAuth token cache), mirroring
the existing Marechai.MobyGames/state/ entry.
346 lines
12 KiB
C#
346 lines
12 KiB
C#
using System;
|
|
using System.Collections.Generic;
|
|
using System.Linq;
|
|
using System.Text.RegularExpressions;
|
|
using System.Threading.Tasks;
|
|
using Marechai.Data.Helpers;
|
|
using Marechai.Database.Models;
|
|
using Microsoft.EntityFrameworkCore;
|
|
|
|
namespace Marechai.MobyGames.Services;
|
|
|
|
public partial class CompanyMatcher
|
|
{
|
|
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
|
List<Company> _companies;
|
|
Dictionary<string, List<Company>> _soundexIndex;
|
|
Dictionary<string, List<Company>> _strippedSoundexIndex;
|
|
Dictionary<string, List<Company>> _legalNameSoundexIndex;
|
|
Dictionary<string, List<Company>> _strippedLegalNameSoundexIndex;
|
|
readonly Dictionary<string, Company> _cache = new(StringComparer.OrdinalIgnoreCase);
|
|
|
|
// Common company suffixes to strip for matching
|
|
static readonly string[] CompanySuffixes =
|
|
[
|
|
", Inc.", ", Inc", " Inc.", " Inc",
|
|
", Ltd.", ", Ltd", " Ltd.", " Ltd",
|
|
", LLC", " LLC",
|
|
", S.A.", " S.A.", ", SA", " SA",
|
|
", S.L.", " S.L.",
|
|
", GmbH", " GmbH",
|
|
", AG", " AG",
|
|
", Co.", " Co.",
|
|
", Corp.", " Corp.", " Corp",
|
|
" Corporation",
|
|
", Limited", " Limited",
|
|
" Interactive",
|
|
" Entertainment",
|
|
" Software",
|
|
" Games",
|
|
" Studios",
|
|
" Studio",
|
|
" Productions",
|
|
" Publishing",
|
|
" Digital",
|
|
" Media",
|
|
" Group",
|
|
" Company"
|
|
];
|
|
|
|
public CompanyMatcher(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
|
|
|
|
/// <summary>
|
|
/// When true, <see cref="MatchOrCreateAsync" /> throws
|
|
/// <see cref="NeedsInteractionException" /> instead of calling
|
|
/// <see cref="PromptMultiple" /> when more than one Soundex candidate is found and no
|
|
/// exact match resolves the name. The batch loop's pre-flight uses
|
|
/// <see cref="WouldPromptForMatch" /> to detect this condition before any DB mutation
|
|
/// so the throw is only a defensive backstop.
|
|
/// </summary>
|
|
public bool Unattended { get; set; }
|
|
|
|
/// <summary>
|
|
/// When true (in addition to <see cref="Unattended" />), <see cref="MatchOrCreateAsync" />
|
|
/// falls through to the "create new company" branch on multi-candidate Soundex matches
|
|
/// instead of throwing. Equivalent to an operator who answers <c>[0] Create new
|
|
/// company</c> at the <see cref="PromptMultiple" /> prompt.
|
|
/// </summary>
|
|
public bool YesToAll { get; set; }
|
|
|
|
public async Task LoadAsync()
|
|
{
|
|
await using var context = await _contextFactory.CreateDbContextAsync();
|
|
|
|
_companies = await context.Companies
|
|
.Select(c => new Company { Id = c.Id, Name = c.Name, LegalName = c.LegalName })
|
|
.ToListAsync();
|
|
|
|
_soundexIndex = SoundexHelper.BuildSoundexIndex(_companies, c => c.Name);
|
|
_strippedSoundexIndex = SoundexHelper.BuildSoundexIndex(_companies, c => StripSuffix(c.Name));
|
|
_legalNameSoundexIndex = SoundexHelper.BuildSoundexIndex(_companies, c => c.LegalName);
|
|
_strippedLegalNameSoundexIndex = SoundexHelper.BuildSoundexIndex(_companies, c => StripSuffix(c.LegalName));
|
|
}
|
|
|
|
/// <summary>
|
|
/// Read-only pre-flight check: returns <c>true</c> if calling
|
|
/// <see cref="MatchOrCreateAsync" /> with the same name would block on
|
|
/// <see cref="PromptMultiple" /> (more than one Soundex candidate and no earlier
|
|
/// exact / cache resolution). Mirrors <see cref="MatchOrCreateAsync" /> exactly up
|
|
/// to that point and does not modify <see cref="_cache" /> or any Soundex index.
|
|
/// </summary>
|
|
public bool WouldPromptForMatch(string name)
|
|
{
|
|
if(string.IsNullOrWhiteSpace(name))
|
|
return false;
|
|
|
|
string normalizedName = name.Replace("\u00a0", " ").Trim();
|
|
|
|
if(_cache.ContainsKey(normalizedName))
|
|
return false;
|
|
|
|
bool exactExists = _companies.Any(c =>
|
|
string.Equals(c.Name, normalizedName, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(exactExists) return false;
|
|
|
|
bool legalExactExists = _companies.Any(c =>
|
|
!string.IsNullOrWhiteSpace(c.LegalName) &&
|
|
string.Equals(c.LegalName, normalizedName, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(legalExactExists) return false;
|
|
|
|
string strippedInput = StripSuffix(normalizedName);
|
|
|
|
bool strippedExactExists = _companies.Any(c =>
|
|
string.Equals(StripSuffix(c.Name), strippedInput, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(strippedExactExists) return false;
|
|
|
|
bool strippedLegalExactExists = _companies.Any(c =>
|
|
!string.IsNullOrWhiteSpace(c.LegalName) &&
|
|
string.Equals(StripSuffix(c.LegalName), strippedInput, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(strippedLegalExactExists) return false;
|
|
|
|
string soundex = SoundexHelper.Generate(normalizedName);
|
|
string strippedSoundex = SoundexHelper.Generate(strippedInput);
|
|
|
|
var seenIds = new HashSet<int>();
|
|
int count = 0;
|
|
|
|
void Count(IEnumerable<Company> cs)
|
|
{
|
|
foreach(Company c in cs)
|
|
if(seenIds.Add(c.Id))
|
|
count++;
|
|
}
|
|
|
|
if(_soundexIndex.TryGetValue(soundex, out var fullCandidates))
|
|
Count(fullCandidates);
|
|
|
|
if(_legalNameSoundexIndex.TryGetValue(soundex, out var legalCandidates))
|
|
Count(legalCandidates);
|
|
|
|
if(_strippedSoundexIndex.TryGetValue(strippedSoundex, out var strippedCandidates))
|
|
Count(strippedCandidates);
|
|
|
|
if(_strippedLegalNameSoundexIndex.TryGetValue(strippedSoundex, out var strippedLegalCandidates))
|
|
Count(strippedLegalCandidates);
|
|
|
|
return count > 1;
|
|
}
|
|
|
|
public async Task<(Company company, string matchType)> MatchOrCreateAsync(string name)
|
|
{
|
|
if(string.IsNullOrWhiteSpace(name))
|
|
return (null, "empty");
|
|
|
|
string normalizedName = name.Replace("\u00a0", " ").Trim();
|
|
|
|
// Check cache first
|
|
if(_cache.TryGetValue(normalizedName, out var cached))
|
|
return (cached, "cached");
|
|
|
|
// Exact match
|
|
var exact = _companies.FirstOrDefault(c =>
|
|
string.Equals(c.Name, normalizedName, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(exact != null)
|
|
{
|
|
_cache[normalizedName] = exact;
|
|
|
|
return (exact, "exact");
|
|
}
|
|
|
|
// Exact match against legal name (e.g., MobyGames "Apple" matches Marechai legal name "Apple Computer, Inc.")
|
|
var legalExact = _companies.FirstOrDefault(c =>
|
|
!string.IsNullOrWhiteSpace(c.LegalName) &&
|
|
string.Equals(c.LegalName, normalizedName, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(legalExact != null)
|
|
{
|
|
_cache[normalizedName] = legalExact;
|
|
|
|
return (legalExact, "exact-legal");
|
|
}
|
|
|
|
// Exact match on stripped names (e.g., "Apple Inc." matches "Apple")
|
|
string strippedInput = StripSuffix(normalizedName);
|
|
|
|
var strippedExact = _companies.FirstOrDefault(c =>
|
|
string.Equals(StripSuffix(c.Name), strippedInput, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(strippedExact != null)
|
|
{
|
|
_cache[normalizedName] = strippedExact;
|
|
|
|
return (strippedExact, "exact-stripped");
|
|
}
|
|
|
|
// Exact match on stripped legal names
|
|
var strippedLegalExact = _companies.FirstOrDefault(c =>
|
|
!string.IsNullOrWhiteSpace(c.LegalName) &&
|
|
string.Equals(StripSuffix(c.LegalName), strippedInput, StringComparison.OrdinalIgnoreCase));
|
|
|
|
if(strippedLegalExact != null)
|
|
{
|
|
_cache[normalizedName] = strippedLegalExact;
|
|
|
|
return (strippedLegalExact, "exact-stripped-legal");
|
|
}
|
|
|
|
// Soundex-based matching: gather candidates from all four indexes (full name, legal name,
|
|
// stripped name, stripped legal name) into a single deduplicated list, then prompt the user
|
|
// once. Previously each index was checked sequentially and a later pass with a SINGLE
|
|
// candidate would silently override the user's "create new" choice from an earlier pass.
|
|
string soundex = SoundexHelper.Generate(normalizedName);
|
|
string strippedSoundex = SoundexHelper.Generate(strippedInput);
|
|
|
|
var soundexCandidates = new List<Company>();
|
|
var seenIds = new HashSet<int>();
|
|
|
|
void AddCandidates(IEnumerable<Company> cs)
|
|
{
|
|
foreach(Company c in cs)
|
|
{
|
|
if(seenIds.Add(c.Id))
|
|
soundexCandidates.Add(c);
|
|
}
|
|
}
|
|
|
|
if(_soundexIndex.TryGetValue(soundex, out var fullCandidates))
|
|
AddCandidates(fullCandidates);
|
|
|
|
if(_legalNameSoundexIndex.TryGetValue(soundex, out var legalCandidates))
|
|
AddCandidates(legalCandidates);
|
|
|
|
if(_strippedSoundexIndex.TryGetValue(strippedSoundex, out var strippedCandidates))
|
|
AddCandidates(strippedCandidates);
|
|
|
|
if(_strippedLegalNameSoundexIndex.TryGetValue(strippedSoundex, out var strippedLegalCandidates))
|
|
AddCandidates(strippedLegalCandidates);
|
|
|
|
if(soundexCandidates.Count == 1)
|
|
{
|
|
_cache[normalizedName] = soundexCandidates[0];
|
|
|
|
return (soundexCandidates[0], "soundex");
|
|
}
|
|
|
|
if(soundexCandidates.Count > 1)
|
|
{
|
|
// Yes-to-all mode: fall through to the "create new company" branch below,
|
|
// mirroring an operator who answers [0] at PromptMultiple.
|
|
if(YesToAll)
|
|
{
|
|
Console.WriteLine(
|
|
$" Multiple Soundex matches for \"{normalizedName}\" — auto-creating new company (yes-to-all).");
|
|
}
|
|
else
|
|
{
|
|
// Defensive backstop: in unattended mode the importer's pre-flight should have
|
|
// already skipped the game. Throw rather than block on Console.ReadLine.
|
|
if(Unattended)
|
|
throw new NeedsInteractionException(
|
|
$"multiple Soundex matches for company \"{normalizedName}\"");
|
|
|
|
Company prompted = PromptMultiple(normalizedName, soundexCandidates);
|
|
|
|
if(prompted != null)
|
|
{
|
|
_cache[normalizedName] = prompted;
|
|
|
|
return (prompted, "soundex-selected");
|
|
}
|
|
// User chose 0 → fall through to create new company.
|
|
}
|
|
}
|
|
|
|
// Create new company
|
|
await using var context = await _contextFactory.CreateDbContextAsync();
|
|
|
|
var newCompany = new Company
|
|
{
|
|
Name = normalizedName,
|
|
Status = Data.CompanyStatus.Unknown
|
|
};
|
|
|
|
context.Companies.Add(newCompany);
|
|
await context.SaveChangesAsync();
|
|
|
|
_companies.Add(newCompany);
|
|
|
|
// Update Soundex index
|
|
if(!_soundexIndex.TryGetValue(soundex, out var list))
|
|
{
|
|
list = [];
|
|
_soundexIndex[soundex] = list;
|
|
}
|
|
|
|
list.Add(newCompany);
|
|
_cache[normalizedName] = newCompany;
|
|
|
|
Console.WriteLine($" Created new company: \"{normalizedName}\" (ID: {newCompany.Id})");
|
|
|
|
return (newCompany, "created");
|
|
}
|
|
|
|
static Company PromptMultiple(string normalizedName, List<Company> candidates)
|
|
{
|
|
Console.WriteLine($"\n Multiple Soundex matches for \"{normalizedName}\":");
|
|
|
|
for(int i = 0; i < candidates.Count; i++)
|
|
Console.WriteLine($" [{i + 1}] {candidates[i].Name} (ID: {candidates[i].Id})");
|
|
|
|
Console.WriteLine($" [0] Create new company");
|
|
Console.Write(" Select: ");
|
|
|
|
string input = Console.ReadLine()?.Trim();
|
|
|
|
if(int.TryParse(input, out int choice) && choice >= 1 && choice <= candidates.Count)
|
|
return candidates[choice - 1];
|
|
|
|
return null; // User chose 0 or invalid — fall through to create
|
|
}
|
|
|
|
static string StripSuffix(string name)
|
|
{
|
|
if(string.IsNullOrWhiteSpace(name))
|
|
return name;
|
|
|
|
string result = name;
|
|
|
|
foreach(string suffix in CompanySuffixes)
|
|
{
|
|
if(result.EndsWith(suffix, StringComparison.OrdinalIgnoreCase))
|
|
{
|
|
result = result[..^suffix.Length].TrimEnd();
|
|
|
|
break; // Only strip the first matching suffix
|
|
}
|
|
}
|
|
|
|
return result;
|
|
}
|
|
}
|