mirror of
https://github.com/claunia/marechai.git
synced 2026-07-08 17:57:08 +00:00
Natalia Portillo
3080d20cbf
Adds a new console project that maps IGDB's catalog (platforms, game
types, companies, games, involved companies) to existing local
Company/Software/SoftwarePlatform records, without any enrichment or
record creation. This establishes the cross-reference table a future
IGDB enrichment phase will consume, the same way Marechai.MobyGames
already does for MobyGames.
Database (Marechai.Database):
- New mirror+mapping tables: IgdbPlatform, IgdbGameType, IgdbCompany,
IgdbGame, IgdbInvolvedCompany. Each stores only the IGDB id, name,
and the few relationship fields needed for matching/disambiguation
(no descriptive fields) plus a nullable FK to the local entity and
an IgdbMatchStatus (Pending/Matched/NoMatch/NeedsReview/Skipped/
Error). No columns were added to Company/Software/SoftwarePlatform,
consistent with the existing per-source mapping convention used by
MobyGamesImportState/WwpcSoftware/OldDosSoftware.
- IgdbGame.GameTypeId is a nullable FK into the new IgdbGameType
lookup table rather than a baked-in enum: IGDB's game_type is
itself a dynamic lookup entity (game.category is deprecated in
favor of it).
- An initial design mirrored IGDB's external_games/external_game_
sources to bridge straight into the existing MobyGames->Software
mapping. Verified directly against the live API that IGDB does not
track MobyGames as an external source at all, so that table/service
and the matcher's bridge step were removed again (see the two
paired migrations).
- Migrations: AddIgdbMirrorAndMappingTables, RemoveIgdbExternalGames.
Shared helpers promoted to Marechai.Data/Helpers (pure, no EF Core
dependency, usable from a console exe without pulling in the Blazor
Marechai project or duplicating logic):
- Marechai/Helpers/JaroWinkler.cs -> Marechai.Data/Helpers/JaroWinkler.cs
- Marechai.MobyGames/Services/SoundexHelper.cs -> Marechai.Data/Helpers/SoundexHelper.cs
All consumers (Marechai's Razor pages, Marechai.MobyGames's matchers)
updated to the new namespace.
Marechai.Igdb console project:
- IgdbAuthService: Twitch OAuth2 client-credentials flow with a
cached, auto-refreshed access token.
- IgdbHttpClient: Apicalypse requests rate-limited to the configured
requests/second with bounded concurrency and retry-with-backoff on
HTTP 429.
- ApicalypseQueryBuilder: small fluent query builder.
- Mirror services (platforms, game types, companies, games, involved
companies): each run mirrors exactly one batch of up to
--batch-size rows (clamped to IGDB's 500-row page limit) and stops,
matching the one-batch-per-invocation convention already used by
Marechai.MobyGames's import command. Pass --batch-size 0 to instead
loop until the whole remaining catalog is mirrored in one
invocation. Resumption is fully automatic: each run re-derives its
own cursor from the database (MAX(IgdbId) already mirrored, or
row count for the small offset-paginated lookups) - no manual
bookkeeping - and the --batch-size 0 loop tracks its cursor in
memory so --dry-run never gets stuck reading the same unchanged
page forever.
- PlatformMatcher: exact -> normalized -> substring -> Jaro-Winkler
matching against SoftwarePlatform. Local platform names follow
MobyGames-style short/compound conventions ("Jaguar", "DOS, Windows
and Windows 3.x") while IGDB uses one full canonical name per
platform, so each local platform is expanded into name variants
(the compound parts) before matching, plus a whole-word substring
pass to bridge short-vs-full names ("Jaguar" <-> "Atari Jaguar").
- CompanyMatcher: exact name -> exact legal name -> suffix-stripped
exact (ported from Marechai.MobyGames's CompanyMatcher) -> Soundex
bucket confirmed by Jaro-Winkler -> full Jaro-Winkler sweep. Unlike
the MobyGames matcher, ambiguity never prompts or auto-creates: it
is recorded as NeedsReview (with candidates) or NoMatch, since this
pass only establishes mapping and is meant to run unattended across
the full catalog. Soundex-bucket candidates are filtered to a 0.90
Jaro-Winkler floor before being considered at all - an early version
without this floor surfaced 29,906 NeedsReview rows out of 71,210
mirrored companies because a shared 4-character Soundex code alone
is weak evidence and collides constantly between unrelated short
names; with the floor and tightened accept thresholds the same
catalog produces 2,602 matched / 139 needs-review / 68,469 no-match.
- GameMatcher: maps IGDB games to local Software (the umbrella title;
per-platform SoftwareRelease mapping is left for a future enrichment
phase). Edition/bundle/remaster/port-type games inherit their
parent's SoftwareId directly instead of being matched or created
independently, reflecting that Marechai models these as
SoftwareRelease rows under one Software while IGDB models them as
separate game rows linked via parent_game/version_parent. Exact name
matches are disambiguated by platform overlap; fuzzy matches require
platform corroboration when the candidate already has releases.
Verified on a 1,000-game sample that this correctly avoids merging
sequels into their base title (Quake/Quake II, DOOM II/Doom 3,
Metal Gear/Metal Gear Solid V, etc. all left as NeedsReview rather
than falsely matched).
- MatchStatsService / AmbiguousReviewService: stats command and an
optional, non-blocking interactive triage command for NeedsReview
rows.
- CLI: mirror-platforms, mirror-game-types, mirror-companies,
mirror-games, mirror-involved-companies, match-platforms,
match-companies, match-games, stats, review-ambiguous, reset.
All mirror/match commands support --dry-run.
.gitignore: added Marechai.Igdb/state/ (OAuth token cache), mirroring
the existing Marechai.MobyGames/state/ entry.
460 lines
7.7 KiB
Plaintext
460 lines
7.7 KiB
Plaintext
# Created by .ignore support plugin (hsz.mobi)
|
|
### JetBrains template
|
|
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and Webstorm
|
|
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
|
|
|
|
# User-specific stuff:
|
|
.idea/**/workspace.xml
|
|
.idea/**/tasks.xml
|
|
.idea/dictionaries
|
|
|
|
# Sensitive or high-churn files:
|
|
.idea/**/dataSources/
|
|
.idea/**/dataSources.ids
|
|
.idea/**/dataSources.xml
|
|
.idea/**/dataSources.local.xml
|
|
.idea/**/sqlDataSources.xml
|
|
.idea/**/dynamic.xml
|
|
.idea/**/uiDesigner.xml
|
|
|
|
# Gradle:
|
|
.idea/**/gradle.xml
|
|
.idea/**/libraries
|
|
|
|
# CMake
|
|
cmake-build-debug/
|
|
cmake-build-release/
|
|
|
|
# Mongo Explorer plugin:
|
|
.idea/**/mongoSettings.xml
|
|
|
|
## File-based project format:
|
|
*.iws
|
|
|
|
## Plugin-specific files:
|
|
|
|
# IntelliJ
|
|
out/
|
|
|
|
# mpeltonen/sbt-idea plugin
|
|
.idea_modules/
|
|
|
|
# JIRA plugin
|
|
atlassian-ide-plugin.xml
|
|
|
|
# Cursive Clojure plugin
|
|
.idea/replstate.xml
|
|
|
|
# Crashlytics plugin (for Android Studio and IntelliJ)
|
|
com_crashlytics_export_strings.xml
|
|
crashlytics.properties
|
|
crashlytics-build.properties
|
|
fabric.properties
|
|
### MonoDevelop template
|
|
#User Specific
|
|
*.userprefs
|
|
*.usertasks
|
|
|
|
#Mono Project Files
|
|
*.pidb
|
|
*.resources
|
|
test-results/
|
|
### VisualStudio template
|
|
## Ignore Visual Studio temporary files, build results, and
|
|
## files generated by popular Visual Studio add-ons.
|
|
##
|
|
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
|
|
|
|
# User-specific files
|
|
*.suo
|
|
*.user
|
|
*.userosscache
|
|
*.sln.docstates
|
|
|
|
# User-specific files (MonoDevelop/Xamarin Studio)
|
|
*.userprefs
|
|
|
|
# Build results
|
|
[Dd]ebug/
|
|
[Dd]ebugPublic/
|
|
[Rr]elease/
|
|
[Rr]eleases/
|
|
x64/
|
|
x86/
|
|
bld/
|
|
[Bb]in/
|
|
[Oo]bj/
|
|
[Ll]og/
|
|
|
|
# Visual Studio 2015/2017 cache/options directory
|
|
.vs/
|
|
# Uncomment if you have tasks that create the project's static files in wwwroot
|
|
#wwwroot/
|
|
|
|
# Visual Studio 2017 auto generated files
|
|
Generated\ Files/
|
|
|
|
# MSTest test Results
|
|
[Tt]est[Rr]esult*/
|
|
[Bb]uild[Ll]og.*
|
|
|
|
# NUNIT
|
|
*.VisualState.xml
|
|
TestResult.xml
|
|
|
|
# Build Results of an ATL Project
|
|
[Dd]ebugPS/
|
|
[Rr]eleasePS/
|
|
dlldata.c
|
|
|
|
# Benchmark Results
|
|
BenchmarkDotNet.Artifacts/
|
|
|
|
# .NET Core
|
|
project.lock.json
|
|
project.fragment.lock.json
|
|
artifacts/
|
|
**/Properties/launchSettings.json
|
|
|
|
# StyleCop
|
|
StyleCopReport.xml
|
|
|
|
# Files built by Visual Studio
|
|
*_i.c
|
|
*_p.c
|
|
*_i.h
|
|
*.ilk
|
|
*.meta
|
|
*.obj
|
|
*.pch
|
|
*.pdb
|
|
*.pgc
|
|
*.pgd
|
|
*.rsp
|
|
*.sbr
|
|
*.tlb
|
|
*.tli
|
|
*.tlh
|
|
*.tmp
|
|
*.tmp_proj
|
|
*.log
|
|
*.vspscc
|
|
*.vssscc
|
|
.builds
|
|
*.pidb
|
|
*.svclog
|
|
*.scc
|
|
|
|
# Chutzpah Test files
|
|
_Chutzpah*
|
|
|
|
# Visual C++ cache files
|
|
ipch/
|
|
*.aps
|
|
*.ncb
|
|
*.opendb
|
|
*.opensdf
|
|
*.sdf
|
|
*.cachefile
|
|
*.VC.db
|
|
*.VC.VC.opendb
|
|
|
|
# Visual Studio profiler
|
|
*.psess
|
|
*.vsp
|
|
*.vspx
|
|
*.sap
|
|
|
|
# Visual Studio Trace Files
|
|
*.e2e
|
|
|
|
# TFS 2012 Local Workspace
|
|
$tf/
|
|
|
|
# Guidance Automation Toolkit
|
|
*.gpState
|
|
|
|
# ReSharper is a .NET coding add-in
|
|
_ReSharper*/
|
|
*.[Rr]e[Ss]harper
|
|
*.DotSettings.user
|
|
|
|
# JustCode is a .NET coding add-in
|
|
.JustCode
|
|
|
|
# TeamCity is a build add-in
|
|
_TeamCity*
|
|
|
|
# DotCover is a Code Coverage Tool
|
|
*.dotCover
|
|
|
|
# AxoCover is a Code Coverage Tool
|
|
.axoCover/*
|
|
!.axoCover/settings.json
|
|
|
|
# Visual Studio code coverage results
|
|
*.coverage
|
|
*.coveragexml
|
|
|
|
# NCrunch
|
|
_NCrunch_*
|
|
.*crunch*.local.xml
|
|
nCrunchTemp_*
|
|
|
|
# MightyMoose
|
|
*.mm.*
|
|
AutoTest.Net/
|
|
|
|
# Web workbench (sass)
|
|
.sass-cache/
|
|
|
|
# Installshield output folder
|
|
[Ee]xpress/
|
|
|
|
# DocProject is a documentation generator add-in
|
|
DocProject/buildhelp/
|
|
DocProject/Help/*.HxT
|
|
DocProject/Help/*.HxC
|
|
DocProject/Help/*.hhc
|
|
DocProject/Help/*.hhk
|
|
DocProject/Help/*.hhp
|
|
DocProject/Help/Html2
|
|
DocProject/Help/html
|
|
|
|
# Click-Once directory
|
|
publish/
|
|
|
|
# Publish Web Output
|
|
*.[Pp]ublish.xml
|
|
*.azurePubxml
|
|
# Note: Comment the next line if you want to checkin your web deploy settings,
|
|
# but database connection strings (with potential passwords) will be unencrypted
|
|
*.pubxml
|
|
*.publishproj
|
|
|
|
# Microsoft Azure Web App publish settings. Comment the next line if you want to
|
|
# checkin your Azure Web App publish settings, but sensitive information contained
|
|
# in these scripts will be unencrypted
|
|
PublishScripts/
|
|
|
|
# NuGet Packages
|
|
*.nupkg
|
|
# The packages folder can be ignored because of Package Restore
|
|
**/[Pp]ackages/*
|
|
# except build/, which is used as an MSBuild target.
|
|
!**/[Pp]ackages/build/
|
|
# Uncomment if necessary however generally it will be regenerated when needed
|
|
#!**/[Pp]ackages/repositories.config
|
|
# NuGet v3's project.json files produces more ignorable files
|
|
*.nuget.props
|
|
*.nuget.targets
|
|
|
|
# Microsoft Azure Build Output
|
|
csx/
|
|
*.build.csdef
|
|
|
|
# Microsoft Azure Emulator
|
|
ecf/
|
|
rcf/
|
|
|
|
# Windows Store app package directories and files
|
|
AppPackages/
|
|
BundleArtifacts/
|
|
Package.StoreAssociation.xml
|
|
_pkginfo.txt
|
|
*.appx
|
|
|
|
# Visual Studio cache files
|
|
# files ending in .cache can be ignored
|
|
*.[Cc]ache
|
|
# but keep track of directories ending in .cache
|
|
!*.[Cc]ache/
|
|
|
|
# Others
|
|
ClientBin/
|
|
~$*
|
|
*~
|
|
*.dbmdl
|
|
*.dbproj.schemaview
|
|
*.jfm
|
|
*.pfx
|
|
*.publishsettings
|
|
orleans.codegen.cs
|
|
|
|
# Including strong name files can present a security risk
|
|
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
|
|
#*.snk
|
|
|
|
# Since there are multiple workflows, uncomment next line to ignore bower_components
|
|
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
|
|
#bower_components/
|
|
|
|
# RIA/Silverlight projects
|
|
Generated_Code/
|
|
|
|
# Backup & report files from converting an old project file
|
|
# to a newer Visual Studio version. Backup files are not needed,
|
|
# because we have git ;-)
|
|
_UpgradeReport_Files/
|
|
Backup*/
|
|
UpgradeLog*.XML
|
|
UpgradeLog*.htm
|
|
|
|
# SQL Server files
|
|
*.mdf
|
|
*.ldf
|
|
*.ndf
|
|
|
|
# Business Intelligence projects
|
|
*.rdl.data
|
|
*.bim.layout
|
|
*.bim_*.settings
|
|
|
|
# Microsoft Fakes
|
|
FakesAssemblies/
|
|
|
|
# GhostDoc plugin setting file
|
|
*.GhostDoc.xml
|
|
|
|
# Node.js Tools for Visual Studio
|
|
.ntvs_analysis.dat
|
|
node_modules/
|
|
|
|
# TypeScript v1 declaration files
|
|
typings/
|
|
|
|
# Visual Studio 6 build log
|
|
*.plg
|
|
|
|
# Visual Studio 6 workspace options file
|
|
*.opt
|
|
|
|
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
|
|
*.vbw
|
|
|
|
# Visual Studio LightSwitch build output
|
|
**/*.HTMLClient/GeneratedArtifacts
|
|
**/*.DesktopClient/GeneratedArtifacts
|
|
**/*.DesktopClient/ModelManifest.xml
|
|
**/*.Server/GeneratedArtifacts
|
|
**/*.Server/ModelManifest.xml
|
|
_Pvt_Extensions
|
|
|
|
# Paket dependency manager
|
|
.paket/paket.exe
|
|
paket-files/
|
|
|
|
# FAKE - F# Make
|
|
.fake/
|
|
|
|
# JetBrains Rider
|
|
.idea/
|
|
*.sln.iml
|
|
|
|
# CodeRush
|
|
.cr/
|
|
|
|
# Python Tools for Visual Studio (PTVS)
|
|
__pycache__/
|
|
*.pyc
|
|
|
|
# Cake - Uncomment if you are using it
|
|
# tools/**
|
|
# !tools/packages.config
|
|
|
|
# Tabs Studio
|
|
*.tss
|
|
|
|
# Telerik's JustMock configuration file
|
|
*.jmconfig
|
|
|
|
# BizTalk build output
|
|
*.btp.cs
|
|
*.btm.cs
|
|
*.odx.cs
|
|
*.xsd.cs
|
|
|
|
# OpenCover UI analysis results
|
|
OpenCover/
|
|
|
|
# Azure Stream Analytics local run output
|
|
ASALocalRun/
|
|
|
|
# MSBuild Binary and Structured Log
|
|
*.binlog
|
|
|
|
### VisualStudioCode template
|
|
.vscode/*
|
|
!.vscode/settings.json
|
|
!.vscode/tasks.json
|
|
!.vscode/launch.json
|
|
!.vscode/extensions.json
|
|
### macOS template
|
|
# General
|
|
.DS_Store
|
|
.AppleDouble
|
|
.LSOverride
|
|
|
|
# Icon must end with two \r
|
|
Icon
|
|
|
|
# Thumbnails
|
|
._*
|
|
|
|
# Files that might appear in the root of a volume
|
|
.DocumentRevisions-V100
|
|
.fseventsd
|
|
.Spotlight-V100
|
|
.TemporaryItems
|
|
.Trashes
|
|
.VolumeIcon.icns
|
|
.com.apple.timemachine.donotpresent
|
|
|
|
# Directories potentially created on remote AFP share
|
|
.AppleDB
|
|
.AppleDesktop
|
|
Network Trash Folder
|
|
Temporary Items
|
|
.apdisk
|
|
### Windows template
|
|
# Windows thumbnail cache files
|
|
Thumbs.db
|
|
ehthumbs.db
|
|
ehthumbs_vista.db
|
|
|
|
# Dump file
|
|
*.stackdump
|
|
|
|
# Folder config file
|
|
[Dd]esktop.ini
|
|
|
|
# Recycle Bin used on file shares
|
|
$RECYCLE.BIN/
|
|
|
|
# Windows Installer files
|
|
*.cab
|
|
*.msi
|
|
*.msm
|
|
*.msp
|
|
|
|
# Windows shortcuts
|
|
*.lnk
|
|
|
|
|
|
# Assets
|
|
Marechai/originals
|
|
Marechai/wwwroot/assets/photos
|
|
Marechai/wwwroot/assets/**/*.jpeg
|
|
Marechai/wwwroot/assets/**/*.svg
|
|
Marechai/wwwroot/assets/**/*.png
|
|
Marechai/wwwroot/assets/**/*.tif
|
|
Marechai/wwwroot/assets/**/*.jpg
|
|
Marechai/wwwroot/assets/**/*.tiff
|
|
Marechai/wwwroot/assets/**/*.webp
|
|
appsettings.Development.json
|
|
appsettings.Production.json
|
|
*.sql
|
|
Marechai.MobyGames/appsettings.json
|
|
Marechai.MobyGames/state/
|
|
Marechai.Igdb/state/
|