mirror of
https://github.com/claunia/marechai.git
synced 2026-07-08 17:57:08 +00:00
feat: Add IGDB mapping pipeline (Marechai.Igdb)
Adds a new console project that maps IGDB's catalog (platforms, game
types, companies, games, involved companies) to existing local
Company/Software/SoftwarePlatform records, without any enrichment or
record creation. This establishes the cross-reference table a future
IGDB enrichment phase will consume, the same way Marechai.MobyGames
already does for MobyGames.
Database (Marechai.Database):
- New mirror+mapping tables: IgdbPlatform, IgdbGameType, IgdbCompany,
IgdbGame, IgdbInvolvedCompany. Each stores only the IGDB id, name,
and the few relationship fields needed for matching/disambiguation
(no descriptive fields) plus a nullable FK to the local entity and
an IgdbMatchStatus (Pending/Matched/NoMatch/NeedsReview/Skipped/
Error). No columns were added to Company/Software/SoftwarePlatform,
consistent with the existing per-source mapping convention used by
MobyGamesImportState/WwpcSoftware/OldDosSoftware.
- IgdbGame.GameTypeId is a nullable FK into the new IgdbGameType
lookup table rather than a baked-in enum: IGDB's game_type is
itself a dynamic lookup entity (game.category is deprecated in
favor of it).
- An initial design mirrored IGDB's external_games/external_game_
sources to bridge straight into the existing MobyGames->Software
mapping. Verified directly against the live API that IGDB does not
track MobyGames as an external source at all, so that table/service
and the matcher's bridge step were removed again (see the two
paired migrations).
- Migrations: AddIgdbMirrorAndMappingTables, RemoveIgdbExternalGames.
Shared helpers promoted to Marechai.Data/Helpers (pure, no EF Core
dependency, usable from a console exe without pulling in the Blazor
Marechai project or duplicating logic):
- Marechai/Helpers/JaroWinkler.cs -> Marechai.Data/Helpers/JaroWinkler.cs
- Marechai.MobyGames/Services/SoundexHelper.cs -> Marechai.Data/Helpers/SoundexHelper.cs
All consumers (Marechai's Razor pages, Marechai.MobyGames's matchers)
updated to the new namespace.
Marechai.Igdb console project:
- IgdbAuthService: Twitch OAuth2 client-credentials flow with a
cached, auto-refreshed access token.
- IgdbHttpClient: Apicalypse requests rate-limited to the configured
requests/second with bounded concurrency and retry-with-backoff on
HTTP 429.
- ApicalypseQueryBuilder: small fluent query builder.
- Mirror services (platforms, game types, companies, games, involved
companies): each run mirrors exactly one batch of up to
--batch-size rows (clamped to IGDB's 500-row page limit) and stops,
matching the one-batch-per-invocation convention already used by
Marechai.MobyGames's import command. Pass --batch-size 0 to instead
loop until the whole remaining catalog is mirrored in one
invocation. Resumption is fully automatic: each run re-derives its
own cursor from the database (MAX(IgdbId) already mirrored, or
row count for the small offset-paginated lookups) - no manual
bookkeeping - and the --batch-size 0 loop tracks its cursor in
memory so --dry-run never gets stuck reading the same unchanged
page forever.
- PlatformMatcher: exact -> normalized -> substring -> Jaro-Winkler
matching against SoftwarePlatform. Local platform names follow
MobyGames-style short/compound conventions ("Jaguar", "DOS, Windows
and Windows 3.x") while IGDB uses one full canonical name per
platform, so each local platform is expanded into name variants
(the compound parts) before matching, plus a whole-word substring
pass to bridge short-vs-full names ("Jaguar" <-> "Atari Jaguar").
- CompanyMatcher: exact name -> exact legal name -> suffix-stripped
exact (ported from Marechai.MobyGames's CompanyMatcher) -> Soundex
bucket confirmed by Jaro-Winkler -> full Jaro-Winkler sweep. Unlike
the MobyGames matcher, ambiguity never prompts or auto-creates: it
is recorded as NeedsReview (with candidates) or NoMatch, since this
pass only establishes mapping and is meant to run unattended across
the full catalog. Soundex-bucket candidates are filtered to a 0.90
Jaro-Winkler floor before being considered at all - an early version
without this floor surfaced 29,906 NeedsReview rows out of 71,210
mirrored companies because a shared 4-character Soundex code alone
is weak evidence and collides constantly between unrelated short
names; with the floor and tightened accept thresholds the same
catalog produces 2,602 matched / 139 needs-review / 68,469 no-match.
- GameMatcher: maps IGDB games to local Software (the umbrella title;
per-platform SoftwareRelease mapping is left for a future enrichment
phase). Edition/bundle/remaster/port-type games inherit their
parent's SoftwareId directly instead of being matched or created
independently, reflecting that Marechai models these as
SoftwareRelease rows under one Software while IGDB models them as
separate game rows linked via parent_game/version_parent. Exact name
matches are disambiguated by platform overlap; fuzzy matches require
platform corroboration when the candidate already has releases.
Verified on a 1,000-game sample that this correctly avoids merging
sequels into their base title (Quake/Quake II, DOOM II/Doom 3,
Metal Gear/Metal Gear Solid V, etc. all left as NeedsReview rather
than falsely matched).
- MatchStatsService / AmbiguousReviewService: stats command and an
optional, non-blocking interactive triage command for NeedsReview
rows.
- CLI: mirror-platforms, mirror-game-types, mirror-companies,
mirror-games, mirror-involved-companies, match-platforms,
match-companies, match-games, stats, review-ambiguous, reset.
All mirror/match commands support --dry-run.
.gitignore: added Marechai.Igdb/state/ (OAuth token cache), mirroring
the existing Marechai.MobyGames/state/ entry.
This commit is contained in:
1
.gitignore
vendored
1
.gitignore
vendored
@@ -456,3 +456,4 @@ appsettings.Production.json
|
||||
*.sql
|
||||
Marechai.MobyGames/appsettings.json
|
||||
Marechai.MobyGames/state/
|
||||
Marechai.Igdb/state/
|
||||
|
||||
@@ -972,6 +972,16 @@ public enum MobyGamesRejectionReview : byte
|
||||
Overridden = 2
|
||||
}
|
||||
|
||||
public enum IgdbMatchStatus : byte
|
||||
{
|
||||
Pending = 0,
|
||||
Matched = 1,
|
||||
NoMatch = 2,
|
||||
NeedsReview = 3,
|
||||
Skipped = 4,
|
||||
Error = 5
|
||||
}
|
||||
|
||||
public enum SoftwareCoverType : byte
|
||||
{
|
||||
[Display(Name = "Front Cover")]
|
||||
|
||||
@@ -27,7 +27,7 @@ using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace Marechai.Helpers;
|
||||
namespace Marechai.Data.Helpers;
|
||||
|
||||
public static class JaroWinkler
|
||||
{
|
||||
@@ -2,7 +2,7 @@ using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace Marechai.MobyGames.Services;
|
||||
namespace Marechai.Data.Helpers;
|
||||
|
||||
public static class SoundexHelper
|
||||
{
|
||||
12858
Marechai.Database/Migrations/20260625191947_AddIgdbMirrorAndMappingTables.Designer.cs
generated
Normal file
12858
Marechai.Database/Migrations/20260625191947_AddIgdbMirrorAndMappingTables.Designer.cs
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,289 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Metadata;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Marechai.Database.Migrations
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class AddIgdbMirrorAndMappingTables : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbCompanies",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<long>(type: "bigint", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
IgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
Name = table.Column<string>(type: "varchar(255)", maxLength: 255, nullable: false)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchStatus = table.Column<byte>(type: "tinyint unsigned", nullable: false),
|
||||
MatchType = table.Column<string>(type: "varchar(64)", maxLength: 64, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchScore = table.Column<double>(type: "double", nullable: true),
|
||||
MatchedOn = table.Column<DateTime>(type: "datetime(6)", nullable: true),
|
||||
CandidatesJson = table.Column<string>(type: "varchar(4096)", maxLength: 4096, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
ErrorMessage = table.Column<string>(type: "varchar(1024)", maxLength: 1024, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
BatchNumber = table.Column<int>(type: "int", nullable: false),
|
||||
CompanyId = table.Column<int>(type: "int", nullable: true),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbCompanies", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbExternalGames",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<long>(type: "bigint", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
IgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
GameIgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
Uid = table.Column<string>(type: "varchar(255)", maxLength: 255, nullable: false)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbExternalGames", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbGames",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<long>(type: "bigint", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
IgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
Name = table.Column<string>(type: "varchar(255)", maxLength: 255, nullable: false)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
GameTypeId = table.Column<int>(type: "int", nullable: true),
|
||||
ParentGameId = table.Column<long>(type: "bigint", nullable: true),
|
||||
VersionParentId = table.Column<long>(type: "bigint", nullable: true),
|
||||
PlatformIdsJson = table.Column<string>(type: "varchar(4096)", maxLength: 4096, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchStatus = table.Column<byte>(type: "tinyint unsigned", nullable: false),
|
||||
MatchType = table.Column<string>(type: "varchar(64)", maxLength: 64, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchScore = table.Column<double>(type: "double", nullable: true),
|
||||
PlatformOverlapScore = table.Column<double>(type: "double", nullable: true),
|
||||
CandidatesJson = table.Column<string>(type: "varchar(4096)", maxLength: 4096, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchedOn = table.Column<DateTime>(type: "datetime(6)", nullable: true),
|
||||
ErrorMessage = table.Column<string>(type: "varchar(1024)", maxLength: 1024, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
BatchNumber = table.Column<int>(type: "int", nullable: false),
|
||||
SoftwareId = table.Column<ulong>(type: "bigint unsigned", nullable: true),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbGames", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbGameTypes",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<int>(type: "int", nullable: false),
|
||||
Type = table.Column<string>(type: "varchar(64)", maxLength: 64, nullable: false)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbGameTypes", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbInvolvedCompanies",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<long>(type: "bigint", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
IgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
GameIgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
CompanyIgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
Developer = table.Column<bool>(type: "bit(1)", nullable: false),
|
||||
Publisher = table.Column<bool>(type: "bit(1)", nullable: false),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbInvolvedCompanies", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbPlatforms",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<int>(type: "int", nullable: false),
|
||||
Name = table.Column<string>(type: "varchar(255)", maxLength: 255, nullable: false)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchStatus = table.Column<byte>(type: "tinyint unsigned", nullable: false),
|
||||
MatchType = table.Column<string>(type: "varchar(64)", maxLength: 64, nullable: true)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
MatchedOn = table.Column<DateTime>(type: "datetime(6)", nullable: true),
|
||||
SoftwarePlatformId = table.Column<ulong>(type: "bigint unsigned", nullable: true),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbPlatforms", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbCompanies_CompanyId",
|
||||
table: "IgdbCompanies",
|
||||
column: "CompanyId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbCompanies_IgdbId",
|
||||
table: "IgdbCompanies",
|
||||
column: "IgdbId",
|
||||
unique: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbCompanies_MatchStatus",
|
||||
table: "IgdbCompanies",
|
||||
column: "MatchStatus");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbCompanies_Name",
|
||||
table: "IgdbCompanies",
|
||||
column: "Name");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbExternalGames_GameIgdbId",
|
||||
table: "IgdbExternalGames",
|
||||
column: "GameIgdbId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbExternalGames_IgdbId",
|
||||
table: "IgdbExternalGames",
|
||||
column: "IgdbId",
|
||||
unique: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbExternalGames_Uid",
|
||||
table: "IgdbExternalGames",
|
||||
column: "Uid");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_GameTypeId",
|
||||
table: "IgdbGames",
|
||||
column: "GameTypeId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_IgdbId",
|
||||
table: "IgdbGames",
|
||||
column: "IgdbId",
|
||||
unique: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_MatchStatus",
|
||||
table: "IgdbGames",
|
||||
column: "MatchStatus");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_Name",
|
||||
table: "IgdbGames",
|
||||
column: "Name");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_ParentGameId",
|
||||
table: "IgdbGames",
|
||||
column: "ParentGameId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_SoftwareId",
|
||||
table: "IgdbGames",
|
||||
column: "SoftwareId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbGames_VersionParentId",
|
||||
table: "IgdbGames",
|
||||
column: "VersionParentId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbInvolvedCompanies_CompanyIgdbId",
|
||||
table: "IgdbInvolvedCompanies",
|
||||
column: "CompanyIgdbId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbInvolvedCompanies_GameIgdbId",
|
||||
table: "IgdbInvolvedCompanies",
|
||||
column: "GameIgdbId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbInvolvedCompanies_IgdbId",
|
||||
table: "IgdbInvolvedCompanies",
|
||||
column: "IgdbId",
|
||||
unique: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbPlatforms_MatchStatus",
|
||||
table: "IgdbPlatforms",
|
||||
column: "MatchStatus");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbPlatforms_SoftwarePlatformId",
|
||||
table: "IgdbPlatforms",
|
||||
column: "SoftwarePlatformId");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbCompanies");
|
||||
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbExternalGames");
|
||||
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbGames");
|
||||
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbGameTypes");
|
||||
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbInvolvedCompanies");
|
||||
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbPlatforms");
|
||||
}
|
||||
}
|
||||
}
|
||||
12815
Marechai.Database/Migrations/20260625203648_RemoveIgdbExternalGames.Designer.cs
generated
Normal file
12815
Marechai.Database/Migrations/20260625203648_RemoveIgdbExternalGames.Designer.cs
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,60 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Metadata;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Marechai.Database.Migrations
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class RemoveIgdbExternalGames : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropTable(
|
||||
name: "IgdbExternalGames");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.CreateTable(
|
||||
name: "IgdbExternalGames",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<long>(type: "bigint", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
CreatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.IdentityColumn),
|
||||
GameIgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
IgdbId = table.Column<long>(type: "bigint", nullable: false),
|
||||
Uid = table.Column<string>(type: "varchar(255)", maxLength: 255, nullable: false)
|
||||
.Annotation("MySql:CharSet", "utf8mb4"),
|
||||
UpdatedOn = table.Column<DateTime>(type: "datetime(6)", nullable: false)
|
||||
.Annotation("MySql:ValueGenerationStrategy", MySqlValueGenerationStrategy.ComputedColumn)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_IgdbExternalGames", x => x.Id);
|
||||
})
|
||||
.Annotation("MySql:CharSet", "utf8mb4");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbExternalGames_GameIgdbId",
|
||||
table: "IgdbExternalGames",
|
||||
column: "GameIgdbId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbExternalGames_IgdbId",
|
||||
table: "IgdbExternalGames",
|
||||
column: "IgdbId",
|
||||
unique: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_IgdbExternalGames_Uid",
|
||||
table: "IgdbExternalGames",
|
||||
column: "Uid");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2540,6 +2540,284 @@ namespace Marechai.Database.Migrations
|
||||
b.ToTable("gpus_by_machine", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Marechai.Database.Models.IgdbCompany", b =>
|
||||
{
|
||||
b.Property<long>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("bigint");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<long>("Id"));
|
||||
|
||||
b.Property<int>("BatchNumber")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<string>("CandidatesJson")
|
||||
.HasMaxLength(4096)
|
||||
.HasColumnType("varchar(4096)");
|
||||
|
||||
b.Property<int?>("CompanyId")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<DateTime>("CreatedOn")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<DateTime>("CreatedOn"));
|
||||
|
||||
b.Property<string>("ErrorMessage")
|
||||
.HasMaxLength(1024)
|
||||
.HasColumnType("varchar(1024)");
|
||||
|
||||
b.Property<long>("IgdbId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.Property<double?>("MatchScore")
|
||||
.HasColumnType("double");
|
||||
|
||||
b.Property<byte>("MatchStatus")
|
||||
.HasColumnType("tinyint unsigned");
|
||||
|
||||
b.Property<string>("MatchType")
|
||||
.HasMaxLength(64)
|
||||
.HasColumnType("varchar(64)");
|
||||
|
||||
b.Property<DateTime?>("MatchedOn")
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
b.Property<string>("Name")
|
||||
.IsRequired()
|
||||
.HasMaxLength(255)
|
||||
.HasColumnType("varchar(255)");
|
||||
|
||||
b.Property<DateTime>("UpdatedOn")
|
||||
.ValueGeneratedOnAddOrUpdate()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlComputedColumn(b.Property<DateTime>("UpdatedOn"));
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("CompanyId");
|
||||
|
||||
b.HasIndex("IgdbId")
|
||||
.IsUnique();
|
||||
|
||||
b.HasIndex("MatchStatus");
|
||||
|
||||
b.HasIndex("Name");
|
||||
|
||||
b.ToTable("IgdbCompanies");
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Marechai.Database.Models.IgdbGame", b =>
|
||||
{
|
||||
b.Property<long>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("bigint");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<long>("Id"));
|
||||
|
||||
b.Property<int>("BatchNumber")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<string>("CandidatesJson")
|
||||
.HasMaxLength(4096)
|
||||
.HasColumnType("varchar(4096)");
|
||||
|
||||
b.Property<DateTime>("CreatedOn")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<DateTime>("CreatedOn"));
|
||||
|
||||
b.Property<string>("ErrorMessage")
|
||||
.HasMaxLength(1024)
|
||||
.HasColumnType("varchar(1024)");
|
||||
|
||||
b.Property<int?>("GameTypeId")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<long>("IgdbId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.Property<double?>("MatchScore")
|
||||
.HasColumnType("double");
|
||||
|
||||
b.Property<byte>("MatchStatus")
|
||||
.HasColumnType("tinyint unsigned");
|
||||
|
||||
b.Property<string>("MatchType")
|
||||
.HasMaxLength(64)
|
||||
.HasColumnType("varchar(64)");
|
||||
|
||||
b.Property<DateTime?>("MatchedOn")
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
b.Property<string>("Name")
|
||||
.IsRequired()
|
||||
.HasMaxLength(255)
|
||||
.HasColumnType("varchar(255)");
|
||||
|
||||
b.Property<long?>("ParentGameId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.Property<string>("PlatformIdsJson")
|
||||
.HasMaxLength(4096)
|
||||
.HasColumnType("varchar(4096)");
|
||||
|
||||
b.Property<double?>("PlatformOverlapScore")
|
||||
.HasColumnType("double");
|
||||
|
||||
b.Property<ulong?>("SoftwareId")
|
||||
.HasColumnType("bigint unsigned");
|
||||
|
||||
b.Property<DateTime>("UpdatedOn")
|
||||
.ValueGeneratedOnAddOrUpdate()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlComputedColumn(b.Property<DateTime>("UpdatedOn"));
|
||||
|
||||
b.Property<long?>("VersionParentId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("GameTypeId");
|
||||
|
||||
b.HasIndex("IgdbId")
|
||||
.IsUnique();
|
||||
|
||||
b.HasIndex("MatchStatus");
|
||||
|
||||
b.HasIndex("Name");
|
||||
|
||||
b.HasIndex("ParentGameId");
|
||||
|
||||
b.HasIndex("SoftwareId");
|
||||
|
||||
b.HasIndex("VersionParentId");
|
||||
|
||||
b.ToTable("IgdbGames");
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Marechai.Database.Models.IgdbGameType", b =>
|
||||
{
|
||||
b.Property<int>("Id")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<DateTime>("CreatedOn")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<DateTime>("CreatedOn"));
|
||||
|
||||
b.Property<string>("Type")
|
||||
.IsRequired()
|
||||
.HasMaxLength(64)
|
||||
.HasColumnType("varchar(64)");
|
||||
|
||||
b.Property<DateTime>("UpdatedOn")
|
||||
.ValueGeneratedOnAddOrUpdate()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlComputedColumn(b.Property<DateTime>("UpdatedOn"));
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.ToTable("IgdbGameTypes");
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Marechai.Database.Models.IgdbInvolvedCompany", b =>
|
||||
{
|
||||
b.Property<long>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("bigint");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<long>("Id"));
|
||||
|
||||
b.Property<long>("CompanyIgdbId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.Property<DateTime>("CreatedOn")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<DateTime>("CreatedOn"));
|
||||
|
||||
b.Property<bool>("Developer")
|
||||
.HasColumnType("bit(1)");
|
||||
|
||||
b.Property<long>("GameIgdbId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.Property<long>("IgdbId")
|
||||
.HasColumnType("bigint");
|
||||
|
||||
b.Property<bool>("Publisher")
|
||||
.HasColumnType("bit(1)");
|
||||
|
||||
b.Property<DateTime>("UpdatedOn")
|
||||
.ValueGeneratedOnAddOrUpdate()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlComputedColumn(b.Property<DateTime>("UpdatedOn"));
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("CompanyIgdbId");
|
||||
|
||||
b.HasIndex("GameIgdbId");
|
||||
|
||||
b.HasIndex("IgdbId")
|
||||
.IsUnique();
|
||||
|
||||
b.ToTable("IgdbInvolvedCompanies");
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Marechai.Database.Models.IgdbPlatform", b =>
|
||||
{
|
||||
b.Property<int>("Id")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<DateTime>("CreatedOn")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlIdentityColumn(b.Property<DateTime>("CreatedOn"));
|
||||
|
||||
b.Property<byte>("MatchStatus")
|
||||
.HasColumnType("tinyint unsigned");
|
||||
|
||||
b.Property<string>("MatchType")
|
||||
.HasMaxLength(64)
|
||||
.HasColumnType("varchar(64)");
|
||||
|
||||
b.Property<DateTime?>("MatchedOn")
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
b.Property<string>("Name")
|
||||
.IsRequired()
|
||||
.HasMaxLength(255)
|
||||
.HasColumnType("varchar(255)");
|
||||
|
||||
b.Property<ulong?>("SoftwarePlatformId")
|
||||
.HasColumnType("bigint unsigned");
|
||||
|
||||
b.Property<DateTime>("UpdatedOn")
|
||||
.ValueGeneratedOnAddOrUpdate()
|
||||
.HasColumnType("datetime(6)");
|
||||
|
||||
MySqlPropertyBuilderExtensions.UseMySqlComputedColumn(b.Property<DateTime>("UpdatedOn"));
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("MatchStatus");
|
||||
|
||||
b.HasIndex("SoftwarePlatformId");
|
||||
|
||||
b.ToTable("IgdbPlatforms");
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Marechai.Database.Models.InstructionSet", b =>
|
||||
{
|
||||
b.Property<int>("Id")
|
||||
|
||||
35
Marechai.Database/Models/IgdbCompany.cs
Normal file
35
Marechai.Database/Models/IgdbCompany.cs
Normal file
@@ -0,0 +1,35 @@
|
||||
using System;
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using Marechai.Data;
|
||||
|
||||
namespace Marechai.Database.Models;
|
||||
|
||||
public class IgdbCompany : BaseModel<long>
|
||||
{
|
||||
[Required]
|
||||
public long IgdbId { get; set; }
|
||||
|
||||
[Required]
|
||||
[StringLength(255)]
|
||||
public string Name { get; set; }
|
||||
|
||||
[Required]
|
||||
public IgdbMatchStatus MatchStatus { get; set; }
|
||||
|
||||
[StringLength(64)]
|
||||
public string MatchType { get; set; }
|
||||
|
||||
public double? MatchScore { get; set; }
|
||||
|
||||
public DateTime? MatchedOn { get; set; }
|
||||
|
||||
[StringLength(4096)]
|
||||
public string CandidatesJson { get; set; }
|
||||
|
||||
[StringLength(1024)]
|
||||
public string ErrorMessage { get; set; }
|
||||
|
||||
public int BatchNumber { get; set; }
|
||||
|
||||
public int? CompanyId { get; set; }
|
||||
}
|
||||
52
Marechai.Database/Models/IgdbGame.cs
Normal file
52
Marechai.Database/Models/IgdbGame.cs
Normal file
@@ -0,0 +1,52 @@
|
||||
using System;
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using Marechai.Data;
|
||||
|
||||
namespace Marechai.Database.Models;
|
||||
|
||||
public class IgdbGame : BaseModel<long>
|
||||
{
|
||||
[Required]
|
||||
public long IgdbId { get; set; }
|
||||
|
||||
[Required]
|
||||
[StringLength(255)]
|
||||
public string Name { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// FK to <see cref="IgdbGameType" />, IGDB's <c>game_type</c> field. This is itself a lookup entity on
|
||||
/// IGDB's side (<c>/game_types</c>), not a fixed enum. <c>category</c> is deprecated by IGDB and must not
|
||||
/// be used.
|
||||
/// </summary>
|
||||
public int? GameTypeId { get; set; }
|
||||
|
||||
public long? ParentGameId { get; set; }
|
||||
|
||||
public long? VersionParentId { get; set; }
|
||||
|
||||
/// <summary>Raw JSON array of IGDB platform ids, used only as a matching/disambiguation signal.</summary>
|
||||
[StringLength(4096)]
|
||||
public string PlatformIdsJson { get; set; }
|
||||
|
||||
[Required]
|
||||
public IgdbMatchStatus MatchStatus { get; set; }
|
||||
|
||||
[StringLength(64)]
|
||||
public string MatchType { get; set; }
|
||||
|
||||
public double? MatchScore { get; set; }
|
||||
|
||||
public double? PlatformOverlapScore { get; set; }
|
||||
|
||||
[StringLength(4096)]
|
||||
public string CandidatesJson { get; set; }
|
||||
|
||||
public DateTime? MatchedOn { get; set; }
|
||||
|
||||
[StringLength(1024)]
|
||||
public string ErrorMessage { get; set; }
|
||||
|
||||
public int BatchNumber { get; set; }
|
||||
|
||||
public ulong? SoftwareId { get; set; }
|
||||
}
|
||||
16
Marechai.Database/Models/IgdbGameType.cs
Normal file
16
Marechai.Database/Models/IgdbGameType.cs
Normal file
@@ -0,0 +1,16 @@
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
|
||||
namespace Marechai.Database.Models;
|
||||
|
||||
/// <summary>
|
||||
/// Mirror of IGDB's <c>/game_types</c> lookup table (main_game, dlc_addon, expansion, bundle,
|
||||
/// standalone_expansion, mod, episode, season, remake, remaster, expanded_game, port, fork, pack, update, ...).
|
||||
/// Small and rarely changing; used by <see cref="IgdbGame.GameTypeId" /> to detect edition/bundle/remaster
|
||||
/// rows that must inherit their parent's <c>SoftwareId</c> instead of being matched independently.
|
||||
/// </summary>
|
||||
public class IgdbGameType : BaseModel<int>
|
||||
{
|
||||
[Required]
|
||||
[StringLength(64)]
|
||||
public string Type { get; set; }
|
||||
}
|
||||
19
Marechai.Database/Models/IgdbInvolvedCompany.cs
Normal file
19
Marechai.Database/Models/IgdbInvolvedCompany.cs
Normal file
@@ -0,0 +1,19 @@
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
|
||||
namespace Marechai.Database.Models;
|
||||
|
||||
public class IgdbInvolvedCompany : BaseModel<long>
|
||||
{
|
||||
[Required]
|
||||
public long IgdbId { get; set; }
|
||||
|
||||
[Required]
|
||||
public long GameIgdbId { get; set; }
|
||||
|
||||
[Required]
|
||||
public long CompanyIgdbId { get; set; }
|
||||
|
||||
public bool Developer { get; set; }
|
||||
|
||||
public bool Publisher { get; set; }
|
||||
}
|
||||
22
Marechai.Database/Models/IgdbPlatform.cs
Normal file
22
Marechai.Database/Models/IgdbPlatform.cs
Normal file
@@ -0,0 +1,22 @@
|
||||
using System;
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using Marechai.Data;
|
||||
|
||||
namespace Marechai.Database.Models;
|
||||
|
||||
public class IgdbPlatform : BaseModel<int>
|
||||
{
|
||||
[Required]
|
||||
[StringLength(255)]
|
||||
public string Name { get; set; }
|
||||
|
||||
[Required]
|
||||
public IgdbMatchStatus MatchStatus { get; set; }
|
||||
|
||||
[StringLength(64)]
|
||||
public string MatchType { get; set; }
|
||||
|
||||
public DateTime? MatchedOn { get; set; }
|
||||
|
||||
public ulong? SoftwarePlatformId { get; set; }
|
||||
}
|
||||
@@ -99,6 +99,11 @@ public class MarechaiContext : IdentityDbContext<ApplicationUser, ApplicationRol
|
||||
public virtual DbSet<Forbidden> Forbidden { get; set; }
|
||||
public virtual DbSet<Gpu> Gpus { get; set; }
|
||||
public virtual DbSet<GpusByMachine> GpusByMachine { get; set; }
|
||||
public virtual DbSet<IgdbCompany> IgdbCompanies { get; set; }
|
||||
public virtual DbSet<IgdbGame> IgdbGames { get; set; }
|
||||
public virtual DbSet<IgdbGameType> IgdbGameTypes { get; set; }
|
||||
public virtual DbSet<IgdbInvolvedCompany> IgdbInvolvedCompanies { get; set; }
|
||||
public virtual DbSet<IgdbPlatform> IgdbPlatforms { get; set; }
|
||||
public virtual DbSet<InstructionSet> InstructionSets { get; set; }
|
||||
public virtual DbSet<InstructionSetExtension> InstructionSetExtensions { get; set; }
|
||||
public virtual DbSet<InstructionSetExtensionsByProcessor> InstructionSetExtensionsByProcessor { get; set; }
|
||||
@@ -2967,6 +2972,44 @@ public class MarechaiContext : IdentityDbContext<ApplicationUser, ApplicationRol
|
||||
entity.HasIndex(e => e.Status);
|
||||
});
|
||||
|
||||
modelBuilder.Entity<IgdbPlatform>(entity =>
|
||||
{
|
||||
entity.Property(e => e.Id).ValueGeneratedNever();
|
||||
entity.HasIndex(e => e.MatchStatus);
|
||||
entity.HasIndex(e => e.SoftwarePlatformId);
|
||||
});
|
||||
|
||||
modelBuilder.Entity<IgdbCompany>(entity =>
|
||||
{
|
||||
entity.HasIndex(e => e.IgdbId).IsUnique();
|
||||
entity.HasIndex(e => e.Name);
|
||||
entity.HasIndex(e => e.MatchStatus);
|
||||
entity.HasIndex(e => e.CompanyId);
|
||||
});
|
||||
|
||||
modelBuilder.Entity<IgdbGame>(entity =>
|
||||
{
|
||||
entity.HasIndex(e => e.IgdbId).IsUnique();
|
||||
entity.HasIndex(e => e.Name);
|
||||
entity.HasIndex(e => e.MatchStatus);
|
||||
entity.HasIndex(e => e.SoftwareId);
|
||||
entity.HasIndex(e => e.ParentGameId);
|
||||
entity.HasIndex(e => e.VersionParentId);
|
||||
entity.HasIndex(e => e.GameTypeId);
|
||||
});
|
||||
|
||||
modelBuilder.Entity<IgdbGameType>(entity =>
|
||||
{
|
||||
entity.Property(e => e.Id).ValueGeneratedNever();
|
||||
});
|
||||
|
||||
modelBuilder.Entity<IgdbInvolvedCompany>(entity =>
|
||||
{
|
||||
entity.HasIndex(e => e.IgdbId).IsUnique();
|
||||
entity.HasIndex(e => e.GameIgdbId);
|
||||
entity.HasIndex(e => e.CompanyIgdbId);
|
||||
});
|
||||
|
||||
modelBuilder.Entity<OldDosCategory>(entity =>
|
||||
{
|
||||
entity.HasIndex(e => e.ParentId);
|
||||
|
||||
22
Marechai.Igdb/Marechai.Igdb.csproj
Normal file
22
Marechai.Igdb/Marechai.Igdb.csproj
Normal file
@@ -0,0 +1,22 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<RootNamespace>Marechai.Igdb</RootNamespace>
|
||||
<SelfContained>true</SelfContained>
|
||||
<PublishSingleFile>true</PublishSingleFile>
|
||||
<RuntimeIdentifier>linux-x64</RuntimeIdentifier>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="MySqlConnector"/>
|
||||
<PackageReference Include="Pomelo.EntityFrameworkCore.MySql"/>
|
||||
<PackageReference Include="Pomelo.EntityFrameworkCore.MySql.Json.Microsoft"/>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Marechai.Database\Marechai.Database.csproj"/>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Content Include="appsettings.json" CopyToOutputDirectory="PreserveNewest"/>
|
||||
<Content Include="appsettings.Development.json" CopyToOutputDirectory="PreserveNewest"/>
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
18
Marechai.Igdb/MarechaiContextFactory.cs
Normal file
18
Marechai.Igdb/MarechaiContextFactory.cs
Normal file
@@ -0,0 +1,18 @@
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb;
|
||||
|
||||
public class MarechaiContextFactory : IDbContextFactory<MarechaiContext>
|
||||
{
|
||||
readonly DbContextOptions<MarechaiContext> _options;
|
||||
|
||||
public MarechaiContextFactory(DbContextOptions<MarechaiContext> options) => _options = options;
|
||||
|
||||
public MarechaiContext CreateDbContext() => new(_options);
|
||||
|
||||
public Task<MarechaiContext> CreateDbContextAsync(CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult(CreateDbContext());
|
||||
}
|
||||
326
Marechai.Igdb/Program.cs
Normal file
326
Marechai.Igdb/Program.cs
Normal file
@@ -0,0 +1,326 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Database;
|
||||
using Marechai.Database.Models;
|
||||
using Marechai.Igdb.Services;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
|
||||
namespace Marechai.Igdb;
|
||||
|
||||
class Program
|
||||
{
|
||||
static async Task<int> Main(string[] args)
|
||||
{
|
||||
Console.WriteLine("\e[32;1mMarechai IGDB Mapping Tool\e[0m\n");
|
||||
|
||||
string environment = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT") ?? "Development";
|
||||
|
||||
var config = new ConfigurationBuilder().AddJsonFile("appsettings.json", optional: true)
|
||||
.AddJsonFile($"appsettings.{environment}.json", optional: true)
|
||||
.Build();
|
||||
|
||||
string marechaiConn = config.GetConnectionString("DefaultConnection");
|
||||
|
||||
if(string.IsNullOrEmpty(marechaiConn))
|
||||
{
|
||||
Console.WriteLine("\e[31;1mMissing connection string in appsettings.json\e[0m");
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
var optionsBuilder = new DbContextOptionsBuilder<MarechaiContext>();
|
||||
|
||||
optionsBuilder.UseLazyLoadingProxies()
|
||||
.AddMarechaiInterceptors()
|
||||
.UseMySql(marechaiConn,
|
||||
new MariaDbServerVersion(new Version(12, 0, 2)),
|
||||
b => b.UseMicrosoftJson().EnableStringComparisonTranslations()
|
||||
.UseQuerySplittingBehavior(QuerySplittingBehavior.SplitQuery));
|
||||
|
||||
var factory = new MarechaiContextFactory(optionsBuilder.Options);
|
||||
|
||||
string command = args.Length > 0 ? args[0].ToLowerInvariant() : "help";
|
||||
bool dryRun = args.Contains("--dry-run");
|
||||
int batchSize = GetIntOption(args, "--batch-size", config.GetValue("Import:BatchSize", 500));
|
||||
|
||||
string clientId = config.GetValue<string>("Igdb:ClientId");
|
||||
string clientSecret = config.GetValue<string>("Igdb:ClientSecret");
|
||||
string tokenCachePath = config.GetValue("Igdb:TokenCachePath", "state/igdb-token.json");
|
||||
int requestsPerSecond = config.GetValue("Igdb:RequestsPerSecond", 4);
|
||||
int maxConcurrentRequests = config.GetValue("Igdb:MaxConcurrentRequests", 8);
|
||||
|
||||
switch(command)
|
||||
{
|
||||
case "mirror-platforms":
|
||||
{
|
||||
IgdbHttpClient client = RequireClient();
|
||||
|
||||
if(client == null)
|
||||
return 1;
|
||||
|
||||
var service = new PlatformMirrorService(factory, client);
|
||||
int total = await service.RunAsync(batchSize, dryRun);
|
||||
Console.WriteLine($"\nDone. Mirrored {total} platforms.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "mirror-game-types":
|
||||
{
|
||||
IgdbHttpClient client = RequireClient();
|
||||
|
||||
if(client == null)
|
||||
return 1;
|
||||
|
||||
var service = new GameTypeMirrorService(factory, client);
|
||||
int total = await service.RunAsync(batchSize, dryRun);
|
||||
Console.WriteLine($"\nDone. Mirrored {total} game types.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "mirror-companies":
|
||||
{
|
||||
IgdbHttpClient client = RequireClient();
|
||||
|
||||
if(client == null)
|
||||
return 1;
|
||||
|
||||
var service = new CompanyMirrorService(factory, client);
|
||||
int total = await service.RunAsync(batchSize, dryRun);
|
||||
Console.WriteLine($"\nDone. Mirrored {total} companies.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "mirror-games":
|
||||
{
|
||||
IgdbHttpClient client = RequireClient();
|
||||
|
||||
if(client == null)
|
||||
return 1;
|
||||
|
||||
var service = new GameMirrorService(factory, client);
|
||||
int total = await service.RunAsync(batchSize, dryRun);
|
||||
Console.WriteLine($"\nDone. Mirrored {total} games.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "mirror-involved-companies":
|
||||
{
|
||||
IgdbHttpClient client = RequireClient();
|
||||
|
||||
if(client == null)
|
||||
return 1;
|
||||
|
||||
var service = new InvolvedCompanyMirrorService(factory, client);
|
||||
int total = await service.RunAsync(batchSize, dryRun);
|
||||
Console.WriteLine($"\nDone. Mirrored {total} involved companies.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "match-platforms":
|
||||
{
|
||||
var matcher = new PlatformMatcher(factory);
|
||||
(int matched, int needsReview) = await matcher.RunAsync(dryRun);
|
||||
Console.WriteLine($"\nDone. Matched: {matched}, needs review: {needsReview}.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "match-companies":
|
||||
{
|
||||
var matcher = new Services.CompanyMatcher(factory);
|
||||
(int matched, int needsReview, int noMatch) = await matcher.RunAsync(dryRun);
|
||||
Console.WriteLine($"\nDone. Matched: {matched}, needs review: {needsReview}, no match: {noMatch}.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "match-games":
|
||||
{
|
||||
var matcher = new GameMatcher(factory);
|
||||
(int matched, int needsReview, int noMatch) = await matcher.RunAsync(dryRun);
|
||||
Console.WriteLine($"\nDone. Matched: {matched}, needs review: {needsReview}, no match: {noMatch}.");
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "stats":
|
||||
{
|
||||
var stats = new MatchStatsService(factory);
|
||||
await stats.PrintAsync();
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "review-ambiguous":
|
||||
{
|
||||
string type = GetStringOption(args, "--type", null);
|
||||
int limit = GetIntOption(args, "--limit", 20);
|
||||
|
||||
if(type == null)
|
||||
{
|
||||
Console.WriteLine(" --type companies|games|platforms is required.");
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
var review = new AmbiguousReviewService(factory);
|
||||
await review.RunAsync(type, limit);
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "reset":
|
||||
{
|
||||
long? igdbId = long.TryParse(GetStringOption(args, "--igdb-id", null), out long id) ? id : null;
|
||||
string type = GetStringOption(args, "--type", null);
|
||||
|
||||
if(igdbId == null || type == null)
|
||||
{
|
||||
Console.WriteLine(" --igdb-id <id> and --type companies|games|platforms are required.");
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
await ResetAsync(factory, type, igdbId.Value);
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
default:
|
||||
PrintUsage();
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
return 0;
|
||||
|
||||
IgdbHttpClient RequireClient()
|
||||
{
|
||||
if(string.IsNullOrEmpty(clientId) || string.IsNullOrEmpty(clientSecret))
|
||||
{
|
||||
Console.WriteLine("\e[31;1mMissing Igdb:ClientId / Igdb:ClientSecret in appsettings.json\e[0m");
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
return IgdbHttpClient.Create(clientId, clientSecret, tokenCachePath, requestsPerSecond,
|
||||
maxConcurrentRequests);
|
||||
}
|
||||
}
|
||||
|
||||
static async Task ResetAsync(MarechaiContextFactory factory, string type, long igdbId)
|
||||
{
|
||||
await using MarechaiContext context = await factory.CreateDbContextAsync();
|
||||
|
||||
switch(type)
|
||||
{
|
||||
case "platforms":
|
||||
{
|
||||
IgdbPlatform entity = await context.IgdbPlatforms.FirstOrDefaultAsync(p => p.Id == igdbId);
|
||||
|
||||
if(entity != null)
|
||||
{
|
||||
entity.MatchStatus = IgdbMatchStatus.Pending;
|
||||
entity.SoftwarePlatformId = null;
|
||||
entity.MatchType = null;
|
||||
entity.MatchedOn = null;
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "companies":
|
||||
{
|
||||
IgdbCompany entity = await context.IgdbCompanies.FirstOrDefaultAsync(c => c.IgdbId == igdbId);
|
||||
|
||||
if(entity != null)
|
||||
{
|
||||
entity.MatchStatus = IgdbMatchStatus.Pending;
|
||||
entity.CompanyId = null;
|
||||
entity.MatchType = null;
|
||||
entity.MatchScore = null;
|
||||
entity.MatchedOn = null;
|
||||
entity.CandidatesJson = null;
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
case "games":
|
||||
{
|
||||
IgdbGame entity = await context.IgdbGames.FirstOrDefaultAsync(g => g.IgdbId == igdbId);
|
||||
|
||||
if(entity != null)
|
||||
{
|
||||
entity.MatchStatus = IgdbMatchStatus.Pending;
|
||||
entity.SoftwareId = null;
|
||||
entity.MatchType = null;
|
||||
entity.MatchScore = null;
|
||||
entity.MatchedOn = null;
|
||||
entity.CandidatesJson = null;
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
default:
|
||||
Console.WriteLine($" Unknown reset type \"{type}\".");
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await context.SaveChangesAsync();
|
||||
Console.WriteLine($" Reset {type} IGDB id {igdbId} to Pending.");
|
||||
}
|
||||
|
||||
static int GetIntOption(string[] args, string name, int defaultValue)
|
||||
{
|
||||
for(int i = 0; i < args.Length - 1; i++)
|
||||
if(args[i] == name && int.TryParse(args[i + 1], out int value))
|
||||
return value;
|
||||
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
static string GetStringOption(string[] args, string name, string defaultValue)
|
||||
{
|
||||
for(int i = 0; i < args.Length - 1; i++)
|
||||
if(args[i] == name)
|
||||
return args[i + 1];
|
||||
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
static void PrintUsage()
|
||||
{
|
||||
Console.WriteLine("""
|
||||
Usage: dotnet Marechai.Igdb.dll <command> [options]
|
||||
|
||||
Commands:
|
||||
mirror-platforms Pull IGDB /platforms
|
||||
mirror-game-types Pull IGDB /game_types
|
||||
mirror-companies Pull IGDB /companies (resumable)
|
||||
mirror-games Pull IGDB /games (resumable)
|
||||
mirror-involved-companies Pull IGDB /involved_companies (resumable)
|
||||
match-platforms Match mirrored platforms against SoftwarePlatform
|
||||
match-companies Match mirrored companies against Company
|
||||
match-games Match mirrored games against Software
|
||||
stats Print match status counts
|
||||
review-ambiguous --type <companies|games|platforms> [--limit N]
|
||||
reset --igdb-id <id> --type <companies|games|platforms>
|
||||
|
||||
Options:
|
||||
--batch-size N Mirror one batch of up to N rows (default Import:BatchSize).
|
||||
Pass 0 to mirror the entire remaining catalog in one run.
|
||||
--dry-run Don't write to the database
|
||||
""");
|
||||
}
|
||||
}
|
||||
124
Marechai.Igdb/Services/AmbiguousReviewService.cs
Normal file
124
Marechai.Igdb/Services/AmbiguousReviewService.cs
Normal file
@@ -0,0 +1,124 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class AmbiguousReviewService
|
||||
{
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
|
||||
public AmbiguousReviewService(IDbContextFactory<MarechaiContext> contextFactory) =>
|
||||
_contextFactory = contextFactory;
|
||||
|
||||
public async Task RunAsync(string type, int limit)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
switch(type)
|
||||
{
|
||||
case "platforms":
|
||||
await ReviewPlatformsAsync(context, limit);
|
||||
|
||||
break;
|
||||
case "companies":
|
||||
await ReviewCompaniesAsync(context, limit);
|
||||
|
||||
break;
|
||||
case "games":
|
||||
await ReviewGamesAsync(context, limit);
|
||||
|
||||
break;
|
||||
default:
|
||||
Console.WriteLine($" Unknown review type \"{type}\". Use platforms, companies, or games.");
|
||||
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
static async Task ReviewPlatformsAsync(MarechaiContext context, int limit)
|
||||
{
|
||||
var platforms = await context.IgdbPlatforms.Where(p => p.MatchStatus == IgdbMatchStatus.NeedsReview)
|
||||
.OrderBy(p => p.Name)
|
||||
.Take(limit)
|
||||
.ToListAsync();
|
||||
|
||||
var softwarePlatforms = await context.SoftwarePlatforms.ToListAsync();
|
||||
|
||||
foreach(IgdbPlatform platform in platforms)
|
||||
{
|
||||
Console.WriteLine($"\n IGDB platform \"{platform.Name}\" (id {platform.Id}) has no confident match:");
|
||||
|
||||
for(int i = 0; i < softwarePlatforms.Count; i++)
|
||||
Console.WriteLine($" [{i + 1}] {softwarePlatforms[i].Name} (id {softwarePlatforms[i].Id})");
|
||||
|
||||
Console.WriteLine(" [0] Skip");
|
||||
Console.Write(" Select: ");
|
||||
|
||||
if(int.TryParse(Console.ReadLine()?.Trim(), out int choice) && choice >= 1 &&
|
||||
choice <= softwarePlatforms.Count)
|
||||
{
|
||||
platform.SoftwarePlatformId = softwarePlatforms[choice - 1].Id;
|
||||
platform.MatchStatus = IgdbMatchStatus.Matched;
|
||||
platform.MatchType = "manual-review";
|
||||
platform.MatchedOn = DateTime.UtcNow;
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static async Task ReviewCompaniesAsync(MarechaiContext context, int limit)
|
||||
{
|
||||
var companies = await context.IgdbCompanies.Where(c => c.MatchStatus == IgdbMatchStatus.NeedsReview)
|
||||
.OrderBy(c => c.Name)
|
||||
.Take(limit)
|
||||
.ToListAsync();
|
||||
|
||||
foreach(IgdbCompany company in companies)
|
||||
{
|
||||
Console.WriteLine($"\n IGDB company \"{company.Name}\" (id {company.IgdbId}):");
|
||||
Console.WriteLine($" Candidates: {company.CandidatesJson}");
|
||||
Console.Write(" Enter local Company id to link, or blank to skip: ");
|
||||
|
||||
string input = Console.ReadLine()?.Trim();
|
||||
|
||||
if(int.TryParse(input, out int companyId))
|
||||
{
|
||||
company.CompanyId = companyId;
|
||||
company.MatchStatus = IgdbMatchStatus.Matched;
|
||||
company.MatchType = "manual-review";
|
||||
company.MatchedOn = DateTime.UtcNow;
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static async Task ReviewGamesAsync(MarechaiContext context, int limit)
|
||||
{
|
||||
var games = await context.IgdbGames.Where(g => g.MatchStatus == IgdbMatchStatus.NeedsReview)
|
||||
.OrderBy(g => g.Name)
|
||||
.Take(limit)
|
||||
.ToListAsync();
|
||||
|
||||
foreach(IgdbGame game in games)
|
||||
{
|
||||
Console.WriteLine($"\n IGDB game \"{game.Name}\" (id {game.IgdbId}):");
|
||||
Console.WriteLine($" Candidates: {game.CandidatesJson}");
|
||||
Console.Write(" Enter local Software id to link, or blank to skip: ");
|
||||
|
||||
string input = Console.ReadLine()?.Trim();
|
||||
|
||||
if(ulong.TryParse(input, out ulong softwareId))
|
||||
{
|
||||
game.SoftwareId = softwareId;
|
||||
game.MatchStatus = IgdbMatchStatus.Matched;
|
||||
game.MatchType = "manual-review";
|
||||
game.MatchedOn = DateTime.UtcNow;
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
68
Marechai.Igdb/Services/ApicalypseQueryBuilder.cs
Normal file
68
Marechai.Igdb/Services/ApicalypseQueryBuilder.cs
Normal file
@@ -0,0 +1,68 @@
|
||||
using System.Text;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class ApicalypseQueryBuilder
|
||||
{
|
||||
string _fields = "*";
|
||||
string _where;
|
||||
string _sort;
|
||||
int? _limit;
|
||||
int? _offset;
|
||||
|
||||
public ApicalypseQueryBuilder Fields(string fields)
|
||||
{
|
||||
_fields = fields;
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public ApicalypseQueryBuilder Where(string where)
|
||||
{
|
||||
_where = where;
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public ApicalypseQueryBuilder Sort(string sort)
|
||||
{
|
||||
_sort = sort;
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public ApicalypseQueryBuilder Limit(int limit)
|
||||
{
|
||||
_limit = limit;
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public ApicalypseQueryBuilder Offset(int offset)
|
||||
{
|
||||
_offset = offset;
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public string Build()
|
||||
{
|
||||
var sb = new StringBuilder();
|
||||
|
||||
sb.Append("fields ").Append(_fields).Append(';');
|
||||
|
||||
if(!string.IsNullOrEmpty(_where))
|
||||
sb.Append(" where ").Append(_where).Append(';');
|
||||
|
||||
if(!string.IsNullOrEmpty(_sort))
|
||||
sb.Append(" sort ").Append(_sort).Append(';');
|
||||
|
||||
if(_limit.HasValue)
|
||||
sb.Append(" limit ").Append(_limit.Value).Append(';');
|
||||
|
||||
if(_offset.HasValue)
|
||||
sb.Append(" offset ").Append(_offset.Value).Append(';');
|
||||
|
||||
return sb.ToString();
|
||||
}
|
||||
}
|
||||
248
Marechai.Igdb/Services/CompanyMatcher.cs
Normal file
248
Marechai.Igdb/Services/CompanyMatcher.cs
Normal file
@@ -0,0 +1,248 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class CompanyMatcher
|
||||
{
|
||||
const double SoundexSingleAcceptThreshold = 0.90;
|
||||
const double SoundexBestAcceptThreshold = 0.92;
|
||||
const double SoundexMargin = 0.05;
|
||||
const double FullSweepSingleThreshold = 0.90;
|
||||
const double FullSweepReviewThreshold = 0.90;
|
||||
const double MinReviewThreshold = 0.90;
|
||||
|
||||
static readonly string[] CompanySuffixes =
|
||||
[
|
||||
", Inc.", ", Inc", " Inc.", " Inc",
|
||||
", Ltd.", ", Ltd", " Ltd.", " Ltd",
|
||||
", LLC", " LLC",
|
||||
", S.A.", " S.A.", ", SA", " SA",
|
||||
", S.L.", " S.L.",
|
||||
", GmbH", " GmbH",
|
||||
", AG", " AG",
|
||||
", Co.", " Co.",
|
||||
", Corp.", " Corp.", " Corp",
|
||||
" Corporation",
|
||||
", Limited", " Limited",
|
||||
" Interactive",
|
||||
" Entertainment",
|
||||
" Software",
|
||||
" Games",
|
||||
" Studios",
|
||||
" Studio",
|
||||
" Productions",
|
||||
" Publishing",
|
||||
" Digital",
|
||||
" Media",
|
||||
" Group",
|
||||
" Company"
|
||||
];
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
|
||||
public CompanyMatcher(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
|
||||
|
||||
public async Task<(int matched, int needsReview, int noMatch)> RunAsync(bool dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
List<Company> companies = await context.Companies
|
||||
.Select(c => new Company
|
||||
{
|
||||
Id = c.Id,
|
||||
Name = c.Name,
|
||||
LegalName = c.LegalName
|
||||
})
|
||||
.ToListAsync();
|
||||
|
||||
Dictionary<string, List<Company>> soundexIndex = SoundexHelper.BuildSoundexIndex(companies, c => c.Name);
|
||||
Dictionary<string, List<Company>> strippedSoundexIndex =
|
||||
SoundexHelper.BuildSoundexIndex(companies, c => StripSuffix(c.Name));
|
||||
Dictionary<string, List<Company>> legalSoundexIndex =
|
||||
SoundexHelper.BuildSoundexIndex(companies, c => c.LegalName);
|
||||
Dictionary<string, List<Company>> strippedLegalSoundexIndex =
|
||||
SoundexHelper.BuildSoundexIndex(companies, c => StripSuffix(c.LegalName));
|
||||
|
||||
List<IgdbCompany> pending = await context.IgdbCompanies
|
||||
.Where(c => c.MatchStatus == IgdbMatchStatus.Pending)
|
||||
.ToListAsync();
|
||||
|
||||
int matched = 0;
|
||||
int needsReview = 0;
|
||||
int noMatch = 0;
|
||||
|
||||
foreach(IgdbCompany igdbCompany in pending)
|
||||
{
|
||||
(Company company, string matchType, double? score, List<(Company item, double score)> candidates) result =
|
||||
Match(igdbCompany.Name, companies, soundexIndex, strippedSoundexIndex, legalSoundexIndex,
|
||||
strippedLegalSoundexIndex);
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
if(result.company != null)
|
||||
{
|
||||
igdbCompany.CompanyId = result.company.Id;
|
||||
igdbCompany.MatchStatus = IgdbMatchStatus.Matched;
|
||||
igdbCompany.MatchType = result.matchType;
|
||||
igdbCompany.MatchScore = result.score;
|
||||
igdbCompany.MatchedOn = DateTime.UtcNow;
|
||||
}
|
||||
else if(result.candidates is { Count: > 0 })
|
||||
{
|
||||
igdbCompany.MatchStatus = IgdbMatchStatus.NeedsReview;
|
||||
igdbCompany.CandidatesJson =
|
||||
JsonSerializer.Serialize(result.candidates.Select(c => new { c.item.Id, c.item.Name, c.score }));
|
||||
}
|
||||
else
|
||||
{
|
||||
igdbCompany.MatchStatus = IgdbMatchStatus.NoMatch;
|
||||
}
|
||||
}
|
||||
|
||||
if(result.company != null)
|
||||
matched++;
|
||||
else if(result.candidates is { Count: > 0 })
|
||||
needsReview++;
|
||||
else
|
||||
noMatch++;
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
await context.SaveChangesAsync();
|
||||
|
||||
return (matched, needsReview, noMatch);
|
||||
}
|
||||
|
||||
static (Company company, string matchType, double? score, List<(Company item, double score)> candidates) Match(
|
||||
string name, List<Company> companies, Dictionary<string, List<Company>> soundexIndex,
|
||||
Dictionary<string, List<Company>> strippedSoundexIndex, Dictionary<string, List<Company>> legalSoundexIndex,
|
||||
Dictionary<string, List<Company>> strippedLegalSoundexIndex)
|
||||
{
|
||||
if(string.IsNullOrWhiteSpace(name))
|
||||
return (null, null, null, null);
|
||||
|
||||
string normalizedName = name.Replace(" ", " ").Trim();
|
||||
|
||||
Company exact = companies.FirstOrDefault(c =>
|
||||
string.Equals(c.Name, normalizedName, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if(exact != null)
|
||||
return (exact, "exact", 1.0, null);
|
||||
|
||||
Company legalExact = companies.FirstOrDefault(c =>
|
||||
!string.IsNullOrWhiteSpace(c.LegalName) &&
|
||||
string.Equals(c.LegalName, normalizedName, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if(legalExact != null)
|
||||
return (legalExact, "exact-legal", 1.0, null);
|
||||
|
||||
string strippedInput = StripSuffix(normalizedName);
|
||||
|
||||
Company strippedExact = companies.FirstOrDefault(c =>
|
||||
string.Equals(StripSuffix(c.Name), strippedInput, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if(strippedExact != null)
|
||||
return (strippedExact, "exact-stripped", 1.0, null);
|
||||
|
||||
Company strippedLegalExact = companies.FirstOrDefault(c =>
|
||||
!string.IsNullOrWhiteSpace(c.LegalName) &&
|
||||
string.Equals(StripSuffix(c.LegalName), strippedInput, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if(strippedLegalExact != null)
|
||||
return (strippedLegalExact, "exact-stripped-legal", 1.0, null);
|
||||
|
||||
string soundex = SoundexHelper.Generate(normalizedName);
|
||||
string strippedSoundex = SoundexHelper.Generate(strippedInput);
|
||||
|
||||
var soundexCandidates = new List<Company>();
|
||||
var seenIds = new HashSet<int>();
|
||||
|
||||
void AddCandidates(IEnumerable<Company> cs)
|
||||
{
|
||||
foreach(Company c in cs)
|
||||
if(seenIds.Add(c.Id))
|
||||
soundexCandidates.Add(c);
|
||||
}
|
||||
|
||||
if(soundexIndex.TryGetValue(soundex, out var fullCandidates))
|
||||
AddCandidates(fullCandidates);
|
||||
|
||||
if(legalSoundexIndex.TryGetValue(soundex, out var legalCandidates))
|
||||
AddCandidates(legalCandidates);
|
||||
|
||||
if(strippedSoundexIndex.TryGetValue(strippedSoundex, out var strippedCandidates))
|
||||
AddCandidates(strippedCandidates);
|
||||
|
||||
if(strippedLegalSoundexIndex.TryGetValue(strippedSoundex, out var strippedLegalCandidates))
|
||||
AddCandidates(strippedLegalCandidates);
|
||||
|
||||
if(soundexCandidates.Count > 0)
|
||||
{
|
||||
// A shared Soundex code alone is weak evidence — short/common codes collide constantly between
|
||||
// textually unrelated names. Only surface a candidate (Matched or NeedsReview) when its actual
|
||||
// Jaro-Winkler score clears MinReviewThreshold; otherwise the Soundex hit carries no real signal
|
||||
// and we fall through to the full sweep below instead of manufacturing a NeedsReview entry.
|
||||
List<(Company item, double score)> scored = soundexCandidates
|
||||
.Select(c => (item: c, score: JaroWinkler.Similarity(normalizedName, c.Name)))
|
||||
.Where(c => c.score >= MinReviewThreshold)
|
||||
.OrderByDescending(c => c.score)
|
||||
.ToList();
|
||||
|
||||
if(scored.Count == 1)
|
||||
{
|
||||
if(scored[0].score >= SoundexSingleAcceptThreshold)
|
||||
return (scored[0].item, "soundex", scored[0].score, null);
|
||||
|
||||
return (null, null, null, scored);
|
||||
}
|
||||
|
||||
if(scored.Count > 1)
|
||||
{
|
||||
if(scored[0].score >= SoundexBestAcceptThreshold &&
|
||||
scored[0].score - scored[1].score >= SoundexMargin)
|
||||
return (scored[0].item, "soundex-jw-best", scored[0].score, null);
|
||||
|
||||
return (null, null, null, scored);
|
||||
}
|
||||
}
|
||||
|
||||
List<(Company item, double score)> sweep =
|
||||
JaroWinkler.FindMatches(normalizedName, companies, c => c.Name, FullSweepReviewThreshold);
|
||||
|
||||
if(sweep.Count == 1 && sweep[0].score >= FullSweepSingleThreshold)
|
||||
return (sweep[0].item, "jaro-winkler", sweep[0].score, null);
|
||||
|
||||
if(sweep.Count > 0)
|
||||
return (null, null, null, sweep);
|
||||
|
||||
return (null, null, null, null);
|
||||
}
|
||||
|
||||
static string StripSuffix(string name)
|
||||
{
|
||||
if(string.IsNullOrWhiteSpace(name))
|
||||
return name;
|
||||
|
||||
string result = name;
|
||||
|
||||
foreach(string suffix in CompanySuffixes)
|
||||
{
|
||||
if(result.EndsWith(suffix, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
result = result[..^suffix.Length].TrimEnd();
|
||||
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
}
|
||||
111
Marechai.Igdb/Services/CompanyMirrorService.cs
Normal file
111
Marechai.Igdb/Services/CompanyMirrorService.cs
Normal file
@@ -0,0 +1,111 @@
|
||||
using System;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class CompanyMirrorService
|
||||
{
|
||||
const int IgdbMaxPageSize = 500;
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
readonly IgdbHttpClient _client;
|
||||
|
||||
public CompanyMirrorService(IDbContextFactory<MarechaiContext> contextFactory, IgdbHttpClient client)
|
||||
{
|
||||
_contextFactory = contextFactory;
|
||||
_client = client;
|
||||
}
|
||||
|
||||
/// <summary>Mirrors one batch (one IGDB request, at most <paramref name="batchSize" /> rows, clamped to
|
||||
/// IGDB's 500-row page limit). Run the command again to mirror the next batch. Pass <paramref name="batchSize" />
|
||||
/// = 0 to instead loop until the whole catalog has been mirrored in this single invocation.</summary>
|
||||
public async Task<int> RunAsync(int batchSize, bool dryRun)
|
||||
{
|
||||
bool all = batchSize <= 0;
|
||||
int pageSize = all ? IgdbMaxPageSize : Math.Min(batchSize, IgdbMaxPageSize);
|
||||
|
||||
long lastId;
|
||||
int alreadyMirrored;
|
||||
|
||||
await using(var probe = await _contextFactory.CreateDbContextAsync())
|
||||
{
|
||||
lastId = await probe.IgdbCompanies.MaxAsync(c => (long?)c.IgdbId) ?? 0;
|
||||
alreadyMirrored = await probe.IgdbCompanies.CountAsync();
|
||||
}
|
||||
|
||||
int total = 0;
|
||||
|
||||
while(true)
|
||||
{
|
||||
int batchNumber = batchSize > 0 ? (alreadyMirrored + total) / batchSize + 1 : 1;
|
||||
|
||||
(int count, long newLastId) = await FetchAndUpsertPageAsync(lastId, pageSize, batchNumber, dryRun);
|
||||
|
||||
total += count;
|
||||
lastId = newLastId;
|
||||
|
||||
if(!all || count < pageSize)
|
||||
break;
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
async Task<(int count, long lastId)> FetchAndUpsertPageAsync(long afterId, int pageSize, int batchNumber,
|
||||
bool dryRun)
|
||||
{
|
||||
string query = new ApicalypseQueryBuilder().Fields("id,name")
|
||||
.Where($"id > {afterId}")
|
||||
.Sort("id asc")
|
||||
.Limit(pageSize)
|
||||
.Build();
|
||||
|
||||
using JsonDocument doc = await _client.QueryAsync("companies", query);
|
||||
|
||||
int count = doc.RootElement.GetArrayLength();
|
||||
long lastId = afterId;
|
||||
|
||||
if(count == 0)
|
||||
{
|
||||
Console.WriteLine(" No more companies to mirror.");
|
||||
|
||||
return (0, lastId);
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
long igdbId = element.GetProperty("id").GetInt64();
|
||||
string name = element.GetProperty("name").GetString();
|
||||
|
||||
context.IgdbCompanies.Add(new IgdbCompany
|
||||
{
|
||||
IgdbId = igdbId,
|
||||
Name = name,
|
||||
MatchStatus = IgdbMatchStatus.Pending,
|
||||
BatchNumber = batchNumber
|
||||
});
|
||||
|
||||
lastId = igdbId;
|
||||
}
|
||||
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
else
|
||||
{
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
lastId = element.GetProperty("id").GetInt64();
|
||||
}
|
||||
|
||||
Console.WriteLine($" Mirrored batch {batchNumber}: {count} companies (up to IGDB id {lastId}).");
|
||||
|
||||
return (count, lastId);
|
||||
}
|
||||
}
|
||||
214
Marechai.Igdb/Services/GameMatcher.cs
Normal file
214
Marechai.Igdb/Services/GameMatcher.cs
Normal file
@@ -0,0 +1,214 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class GameMatcher
|
||||
{
|
||||
const double FullSweepAcceptThreshold = 0.90;
|
||||
const double FullSweepReviewThreshold = 0.85;
|
||||
|
||||
static readonly HashSet<string> StandaloneGameTypes = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"main_game", "dlc_addon", "expansion", "standalone_expansion", "mod", "episode", "season"
|
||||
};
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
|
||||
public GameMatcher(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
|
||||
|
||||
public async Task<(int matched, int needsReview, int noMatch)> RunAsync(bool dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
List<Software> softwareList = await context.Softwares.Select(s => new Software
|
||||
{
|
||||
Id = s.Id,
|
||||
Name = s.Name
|
||||
}).ToListAsync();
|
||||
|
||||
Dictionary<ulong, HashSet<ulong>> platformsBySoftwareId =
|
||||
(await context.SoftwareReleases.Where(r => r.SoftwareId != null && r.PlatformId != null)
|
||||
.Select(r => new { SoftwareId = r.SoftwareId.Value, PlatformId = r.PlatformId.Value })
|
||||
.ToListAsync())
|
||||
.GroupBy(r => r.SoftwareId)
|
||||
.ToDictionary(g => g.Key, g => g.Select(r => r.PlatformId).ToHashSet());
|
||||
|
||||
Dictionary<int, ulong> softwarePlatformByIgdbPlatformId =
|
||||
await context.IgdbPlatforms.Where(p => p.SoftwarePlatformId != null)
|
||||
.ToDictionaryAsync(p => p.Id, p => p.SoftwarePlatformId.Value);
|
||||
|
||||
Dictionary<int, string> gameTypeById = await context.IgdbGameTypes.ToDictionaryAsync(t => t.Id, t => t.Type);
|
||||
|
||||
List<IgdbGame> allGames = await context.IgdbGames.ToListAsync();
|
||||
|
||||
Dictionary<long, ulong> resolvedSoftwareIdByIgdbId = allGames
|
||||
.Where(g => g.MatchStatus == IgdbMatchStatus.Matched &&
|
||||
g.SoftwareId.HasValue)
|
||||
.ToDictionary(g => g.IgdbId, g => g.SoftwareId.Value);
|
||||
|
||||
List<IgdbGame> pending = allGames.Where(g => g.MatchStatus == IgdbMatchStatus.Pending).ToList();
|
||||
|
||||
// Process parentless / standalone games first so dependents can inherit within the same run.
|
||||
List<IgdbGame> ordered = pending
|
||||
.OrderBy(g => g.ParentGameId.HasValue || g.VersionParentId.HasValue ? 1 : 0)
|
||||
.ToList();
|
||||
|
||||
int matched = 0;
|
||||
int needsReview = 0;
|
||||
int noMatch = 0;
|
||||
|
||||
foreach(IgdbGame igdbGame in ordered)
|
||||
{
|
||||
(ulong? softwareId, string matchType, double? score, List<(Software item, double score)> candidates) result
|
||||
= MatchOne(igdbGame, softwareList, platformsBySoftwareId, softwarePlatformByIgdbPlatformId,
|
||||
gameTypeById, resolvedSoftwareIdByIgdbId);
|
||||
|
||||
if(result.softwareId.HasValue)
|
||||
resolvedSoftwareIdByIgdbId[igdbGame.IgdbId] = result.softwareId.Value;
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
if(result.softwareId.HasValue)
|
||||
{
|
||||
igdbGame.SoftwareId = result.softwareId.Value;
|
||||
igdbGame.MatchStatus = IgdbMatchStatus.Matched;
|
||||
igdbGame.MatchType = result.matchType;
|
||||
igdbGame.MatchScore = result.score;
|
||||
igdbGame.MatchedOn = DateTime.UtcNow;
|
||||
}
|
||||
else if(result.candidates is { Count: > 0 })
|
||||
{
|
||||
igdbGame.MatchStatus = IgdbMatchStatus.NeedsReview;
|
||||
igdbGame.CandidatesJson =
|
||||
JsonSerializer.Serialize(result.candidates.Select(c => new { c.item.Id, c.item.Name, c.score }));
|
||||
}
|
||||
else
|
||||
{
|
||||
igdbGame.MatchStatus = IgdbMatchStatus.NoMatch;
|
||||
}
|
||||
}
|
||||
|
||||
if(result.softwareId.HasValue)
|
||||
matched++;
|
||||
else if(result.candidates is { Count: > 0 })
|
||||
needsReview++;
|
||||
else
|
||||
noMatch++;
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
await context.SaveChangesAsync();
|
||||
|
||||
return (matched, needsReview, noMatch);
|
||||
}
|
||||
|
||||
static (ulong? softwareId, string matchType, double? score, List<(Software item, double score)> candidates)
|
||||
MatchOne(IgdbGame igdbGame, List<Software> softwareList, Dictionary<ulong, HashSet<ulong>> platformsBySoftwareId,
|
||||
Dictionary<int, ulong> softwarePlatformByIgdbPlatformId, Dictionary<int, string> gameTypeById,
|
||||
Dictionary<long, ulong> resolvedSoftwareIdByIgdbId)
|
||||
{
|
||||
// Step 1: edition/bundle inheritance.
|
||||
string gameType = igdbGame.GameTypeId.HasValue && gameTypeById.TryGetValue(igdbGame.GameTypeId.Value, out string t)
|
||||
? t
|
||||
: null;
|
||||
|
||||
bool isStandaloneType = gameType == null || StandaloneGameTypes.Contains(gameType);
|
||||
long? parentIgdbId = igdbGame.ParentGameId ?? igdbGame.VersionParentId;
|
||||
|
||||
if(!isStandaloneType && parentIgdbId.HasValue)
|
||||
{
|
||||
if(resolvedSoftwareIdByIgdbId.TryGetValue(parentIgdbId.Value, out ulong parentSoftwareId))
|
||||
return (parentSoftwareId, "inherited-from-parent", null, null);
|
||||
|
||||
// Parent not resolved yet (possibly in a later mirror batch) — leave Pending for a later run.
|
||||
return (null, null, null, null);
|
||||
}
|
||||
|
||||
// Steps 2-4: name + platform corroboration.
|
||||
HashSet<int> igdbPlatformIds = ParsePlatformIds(igdbGame.PlatformIdsJson);
|
||||
|
||||
HashSet<ulong> targetSoftwarePlatformIds = igdbPlatformIds
|
||||
.Where(softwarePlatformByIgdbPlatformId.ContainsKey)
|
||||
.Select(id => softwarePlatformByIgdbPlatformId[id])
|
||||
.ToHashSet();
|
||||
|
||||
double PlatformOverlap(ulong softwareId)
|
||||
{
|
||||
if(targetSoftwarePlatformIds.Count == 0)
|
||||
return 0;
|
||||
|
||||
if(!platformsBySoftwareId.TryGetValue(softwareId, out HashSet<ulong> releasePlatforms))
|
||||
return 0;
|
||||
|
||||
int overlap = targetSoftwarePlatformIds.Count(releasePlatforms.Contains);
|
||||
|
||||
return (double)overlap / targetSoftwarePlatformIds.Count;
|
||||
}
|
||||
|
||||
List<Software> exactNameMatches = softwareList
|
||||
.Where(s => string.Equals(s.Name, igdbGame.Name,
|
||||
StringComparison.OrdinalIgnoreCase))
|
||||
.ToList();
|
||||
|
||||
if(exactNameMatches.Count == 1)
|
||||
return (exactNameMatches[0].Id, "exact", null, null);
|
||||
|
||||
if(exactNameMatches.Count > 1)
|
||||
{
|
||||
List<(Software software, double overlap)> withOverlap = exactNameMatches
|
||||
.Select(s => (software: s, overlap: PlatformOverlap(s.Id)))
|
||||
.Where(s => s.overlap > 0)
|
||||
.ToList();
|
||||
|
||||
if(withOverlap.Count == 1)
|
||||
return (withOverlap[0].software.Id, "exact-platform-disambiguated", null, null);
|
||||
|
||||
return (null, null, null,
|
||||
exactNameMatches.Select(s => (item: s, score: 1.0)).ToList());
|
||||
}
|
||||
|
||||
List<(Software item, double score)> sweep =
|
||||
JaroWinkler.FindMatches(igdbGame.Name, softwareList, s => s.Name, FullSweepReviewThreshold);
|
||||
|
||||
if(sweep.Count == 1)
|
||||
{
|
||||
bool hasExistingReleases = platformsBySoftwareId.TryGetValue(sweep[0].item.Id, out HashSet<ulong> releases) &&
|
||||
releases.Count > 0;
|
||||
|
||||
double overlap = PlatformOverlap(sweep[0].item.Id);
|
||||
|
||||
if(sweep[0].score >= FullSweepAcceptThreshold && (!hasExistingReleases || overlap > 0))
|
||||
return (sweep[0].item.Id, "jw-platform-corroborated", sweep[0].score, null);
|
||||
|
||||
return (null, null, null, sweep);
|
||||
}
|
||||
|
||||
if(sweep.Count > 1)
|
||||
return (null, null, null, sweep);
|
||||
|
||||
return (null, null, null, null);
|
||||
}
|
||||
|
||||
static HashSet<int> ParsePlatformIds(string json)
|
||||
{
|
||||
if(string.IsNullOrWhiteSpace(json))
|
||||
return [];
|
||||
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize<List<int>>(json)?.ToHashSet() ?? [];
|
||||
}
|
||||
catch(JsonException)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
}
|
||||
}
|
||||
137
Marechai.Igdb/Services/GameMirrorService.cs
Normal file
137
Marechai.Igdb/Services/GameMirrorService.cs
Normal file
@@ -0,0 +1,137 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class GameMirrorService
|
||||
{
|
||||
const int IgdbMaxPageSize = 500;
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
readonly IgdbHttpClient _client;
|
||||
|
||||
public GameMirrorService(IDbContextFactory<MarechaiContext> contextFactory, IgdbHttpClient client)
|
||||
{
|
||||
_contextFactory = contextFactory;
|
||||
_client = client;
|
||||
}
|
||||
|
||||
/// <summary>Mirrors one batch (one IGDB request, at most <paramref name="batchSize" /> rows, clamped to
|
||||
/// IGDB's 500-row page limit). Run the command again to mirror the next batch. Pass <paramref name="batchSize" />
|
||||
/// = 0 to instead loop until the whole catalog has been mirrored in this single invocation.</summary>
|
||||
public async Task<int> RunAsync(int batchSize, bool dryRun)
|
||||
{
|
||||
bool all = batchSize <= 0;
|
||||
int pageSize = all ? IgdbMaxPageSize : Math.Min(batchSize, IgdbMaxPageSize);
|
||||
|
||||
long lastId;
|
||||
int alreadyMirrored;
|
||||
|
||||
await using(var probe = await _contextFactory.CreateDbContextAsync())
|
||||
{
|
||||
lastId = await probe.IgdbGames.MaxAsync(g => (long?)g.IgdbId) ?? 0;
|
||||
alreadyMirrored = await probe.IgdbGames.CountAsync();
|
||||
}
|
||||
|
||||
int total = 0;
|
||||
|
||||
while(true)
|
||||
{
|
||||
int batchNumber = batchSize > 0 ? (alreadyMirrored + total) / batchSize + 1 : 1;
|
||||
|
||||
(int count, long newLastId) = await FetchAndUpsertPageAsync(lastId, pageSize, batchNumber, dryRun);
|
||||
|
||||
total += count;
|
||||
lastId = newLastId;
|
||||
|
||||
if(!all || count < pageSize)
|
||||
break;
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
async Task<(int count, long lastId)> FetchAndUpsertPageAsync(long afterId, int pageSize, int batchNumber,
|
||||
bool dryRun)
|
||||
{
|
||||
string query = new ApicalypseQueryBuilder()
|
||||
.Fields("id,name,game_type,parent_game,version_parent,platforms")
|
||||
.Where($"id > {afterId}")
|
||||
.Sort("id asc")
|
||||
.Limit(pageSize)
|
||||
.Build();
|
||||
|
||||
using JsonDocument doc = await _client.QueryAsync("games", query);
|
||||
|
||||
int count = doc.RootElement.GetArrayLength();
|
||||
long lastId = afterId;
|
||||
|
||||
if(count == 0)
|
||||
{
|
||||
Console.WriteLine(" No more games to mirror.");
|
||||
|
||||
return (0, lastId);
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
long igdbId = element.GetProperty("id").GetInt64();
|
||||
string name = element.GetProperty("name").GetString();
|
||||
|
||||
int? gameTypeId = element.TryGetProperty("game_type", out var gt) && gt.ValueKind == JsonValueKind.Number
|
||||
? gt.GetInt32()
|
||||
: null;
|
||||
|
||||
long? parentGameId = element.TryGetProperty("parent_game", out var pg) &&
|
||||
pg.ValueKind == JsonValueKind.Number
|
||||
? pg.GetInt64()
|
||||
: null;
|
||||
|
||||
long? versionParentId = element.TryGetProperty("version_parent", out var vp) &&
|
||||
vp.ValueKind == JsonValueKind.Number
|
||||
? vp.GetInt64()
|
||||
: null;
|
||||
|
||||
List<int> platformIds = element.TryGetProperty("platforms", out var plats) &&
|
||||
plats.ValueKind == JsonValueKind.Array
|
||||
? plats.EnumerateArray().Select(p => p.GetInt32()).ToList()
|
||||
: [];
|
||||
|
||||
context.IgdbGames.Add(new IgdbGame
|
||||
{
|
||||
IgdbId = igdbId,
|
||||
Name = name,
|
||||
GameTypeId = gameTypeId,
|
||||
ParentGameId = parentGameId,
|
||||
VersionParentId = versionParentId,
|
||||
PlatformIdsJson = JsonSerializer.Serialize(platformIds),
|
||||
MatchStatus = IgdbMatchStatus.Pending,
|
||||
BatchNumber = batchNumber
|
||||
});
|
||||
|
||||
lastId = igdbId;
|
||||
}
|
||||
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
else
|
||||
{
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
lastId = element.GetProperty("id").GetInt64();
|
||||
}
|
||||
|
||||
Console.WriteLine($" Mirrored batch {batchNumber}: {count} games (up to IGDB id {lastId}).");
|
||||
|
||||
return (count, lastId);
|
||||
}
|
||||
}
|
||||
93
Marechai.Igdb/Services/GameTypeMirrorService.cs
Normal file
93
Marechai.Igdb/Services/GameTypeMirrorService.cs
Normal file
@@ -0,0 +1,93 @@
|
||||
using System;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class GameTypeMirrorService
|
||||
{
|
||||
const int IgdbMaxPageSize = 500;
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
readonly IgdbHttpClient _client;
|
||||
|
||||
public GameTypeMirrorService(IDbContextFactory<MarechaiContext> contextFactory, IgdbHttpClient client)
|
||||
{
|
||||
_contextFactory = contextFactory;
|
||||
_client = client;
|
||||
}
|
||||
|
||||
/// <summary>Mirrors one batch (one IGDB request, at most <paramref name="batchSize" /> rows, clamped to
|
||||
/// IGDB's 500-row page limit). Run the command again to mirror the next batch. Pass <paramref name="batchSize" />
|
||||
/// = 0 to instead loop until the whole catalog has been mirrored in this single invocation.</summary>
|
||||
public async Task<int> RunAsync(int batchSize, bool dryRun)
|
||||
{
|
||||
bool all = batchSize <= 0;
|
||||
int pageSize = all ? IgdbMaxPageSize : Math.Min(batchSize, IgdbMaxPageSize);
|
||||
|
||||
int offset;
|
||||
|
||||
await using(var probe = await _contextFactory.CreateDbContextAsync())
|
||||
offset = await probe.IgdbGameTypes.CountAsync();
|
||||
|
||||
int total = 0;
|
||||
|
||||
while(true)
|
||||
{
|
||||
int count = await FetchAndUpsertPageAsync(offset + total, pageSize, dryRun);
|
||||
|
||||
total += count;
|
||||
|
||||
if(!all || count < pageSize)
|
||||
break;
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
async Task<int> FetchAndUpsertPageAsync(int offset, int pageSize, bool dryRun)
|
||||
{
|
||||
string query = new ApicalypseQueryBuilder().Fields("id,type")
|
||||
.Sort("id asc")
|
||||
.Limit(pageSize)
|
||||
.Offset(offset)
|
||||
.Build();
|
||||
|
||||
using JsonDocument doc = await _client.QueryAsync("game_types", query);
|
||||
|
||||
int count = doc.RootElement.GetArrayLength();
|
||||
|
||||
if(count == 0)
|
||||
{
|
||||
Console.WriteLine(" No more game types to mirror.");
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
int igdbId = element.GetProperty("id").GetInt32();
|
||||
string type = element.GetProperty("type").GetString();
|
||||
|
||||
var existing = await context.IgdbGameTypes.FirstOrDefaultAsync(t => t.Id == igdbId);
|
||||
|
||||
if(existing == null)
|
||||
context.IgdbGameTypes.Add(new IgdbGameType { Id = igdbId, Type = type });
|
||||
else
|
||||
existing.Type = type;
|
||||
}
|
||||
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
Console.WriteLine($" Mirrored {count} game types (offset {offset}).");
|
||||
|
||||
return count;
|
||||
}
|
||||
}
|
||||
107
Marechai.Igdb/Services/IgdbAuthService.cs
Normal file
107
Marechai.Igdb/Services/IgdbAuthService.cs
Normal file
@@ -0,0 +1,107 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Net.Http;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class IgdbAuthService
|
||||
{
|
||||
readonly HttpClient _httpClient;
|
||||
readonly string _clientId;
|
||||
readonly string _clientSecret;
|
||||
readonly string _tokenCachePath;
|
||||
|
||||
string _accessToken;
|
||||
DateTime _expiresAt;
|
||||
|
||||
public IgdbAuthService(HttpClient httpClient, string clientId, string clientSecret, string tokenCachePath)
|
||||
{
|
||||
_httpClient = httpClient;
|
||||
_clientId = clientId;
|
||||
_clientSecret = clientSecret;
|
||||
_tokenCachePath = tokenCachePath;
|
||||
}
|
||||
|
||||
public string ClientId => _clientId;
|
||||
|
||||
public async Task<string> GetAccessTokenAsync()
|
||||
{
|
||||
if(!string.IsNullOrEmpty(_accessToken) && DateTime.UtcNow < _expiresAt)
|
||||
return _accessToken;
|
||||
|
||||
if(TryLoadCachedToken())
|
||||
return _accessToken;
|
||||
|
||||
await AuthenticateAsync();
|
||||
|
||||
return _accessToken;
|
||||
}
|
||||
|
||||
bool TryLoadCachedToken()
|
||||
{
|
||||
if(!File.Exists(_tokenCachePath))
|
||||
return false;
|
||||
|
||||
try
|
||||
{
|
||||
var cached = JsonSerializer.Deserialize<CachedToken>(File.ReadAllText(_tokenCachePath));
|
||||
|
||||
if(cached == null || DateTime.UtcNow >= cached.ExpiresAt)
|
||||
return false;
|
||||
|
||||
_accessToken = cached.AccessToken;
|
||||
_expiresAt = cached.ExpiresAt;
|
||||
|
||||
return true;
|
||||
}
|
||||
catch(Exception)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async Task AuthenticateAsync()
|
||||
{
|
||||
var content = new FormUrlEncodedContent(
|
||||
[
|
||||
new KeyValuePair<string, string>("client_id", _clientId),
|
||||
new KeyValuePair<string, string>("client_secret", _clientSecret),
|
||||
new KeyValuePair<string, string>("grant_type", "client_credentials")
|
||||
]);
|
||||
|
||||
using HttpResponseMessage response =
|
||||
await _httpClient.PostAsync("https://id.twitch.tv/oauth2/token", content);
|
||||
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
string body = await response.Content.ReadAsStringAsync();
|
||||
using var doc = JsonDocument.Parse(body);
|
||||
|
||||
_accessToken = doc.RootElement.GetProperty("access_token").GetString();
|
||||
int expiresIn = doc.RootElement.GetProperty("expires_in").GetInt32();
|
||||
|
||||
// Refresh a day early so a long-running batch never gets caught mid-call with a stale token.
|
||||
_expiresAt = DateTime.UtcNow.AddSeconds(expiresIn).AddDays(-1);
|
||||
|
||||
string directory = Path.GetDirectoryName(_tokenCachePath);
|
||||
|
||||
if(!string.IsNullOrEmpty(directory))
|
||||
Directory.CreateDirectory(directory);
|
||||
|
||||
File.WriteAllText(_tokenCachePath,
|
||||
JsonSerializer.Serialize(new CachedToken
|
||||
{
|
||||
AccessToken = _accessToken,
|
||||
ExpiresAt = _expiresAt
|
||||
}));
|
||||
}
|
||||
|
||||
sealed class CachedToken
|
||||
{
|
||||
public string AccessToken { get; set; }
|
||||
public DateTime ExpiresAt { get; set; }
|
||||
}
|
||||
}
|
||||
101
Marechai.Igdb/Services/IgdbHttpClient.cs
Normal file
101
Marechai.Igdb/Services/IgdbHttpClient.cs
Normal file
@@ -0,0 +1,101 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class IgdbHttpClient
|
||||
{
|
||||
const string BaseUrl = "https://api.igdb.com/v4/";
|
||||
|
||||
readonly HttpClient _httpClient;
|
||||
readonly IgdbAuthService _auth;
|
||||
readonly SemaphoreSlim _concurrency;
|
||||
readonly TimeSpan _minDelayBetweenRequests;
|
||||
DateTime _lastRequestUtc = DateTime.MinValue;
|
||||
readonly object _delayLock = new();
|
||||
|
||||
public IgdbHttpClient(HttpClient httpClient, IgdbAuthService auth, int requestsPerSecond, int maxConcurrentRequests)
|
||||
{
|
||||
_httpClient = httpClient;
|
||||
_auth = auth;
|
||||
_concurrency = new SemaphoreSlim(maxConcurrentRequests, maxConcurrentRequests);
|
||||
_minDelayBetweenRequests = TimeSpan.FromSeconds(1.0 / Math.Max(1, requestsPerSecond));
|
||||
}
|
||||
|
||||
public static IgdbHttpClient Create(string clientId, string clientSecret, string tokenCachePath,
|
||||
int requestsPerSecond, int maxConcurrentRequests)
|
||||
{
|
||||
var httpClient = new HttpClient();
|
||||
var auth = new IgdbAuthService(httpClient, clientId, clientSecret, tokenCachePath);
|
||||
|
||||
return new IgdbHttpClient(httpClient, auth, requestsPerSecond, maxConcurrentRequests);
|
||||
}
|
||||
|
||||
public async Task<JsonDocument> QueryAsync(string endpoint, string apicalypseBody)
|
||||
{
|
||||
await _concurrency.WaitAsync();
|
||||
|
||||
try
|
||||
{
|
||||
await ThrottleAsync();
|
||||
|
||||
for(int attempt = 0; attempt < 5; attempt++)
|
||||
{
|
||||
string token = await _auth.GetAccessTokenAsync();
|
||||
|
||||
using var request = new HttpRequestMessage(HttpMethod.Post, BaseUrl + endpoint)
|
||||
{
|
||||
Content = new StringContent(apicalypseBody, Encoding.UTF8, "text/plain")
|
||||
};
|
||||
|
||||
request.Headers.Add("Client-ID", _auth.ClientId);
|
||||
request.Headers.Add("Authorization", $"Bearer {token}");
|
||||
|
||||
using HttpResponseMessage response = await _httpClient.SendAsync(request);
|
||||
|
||||
if(response.StatusCode == HttpStatusCode.TooManyRequests)
|
||||
{
|
||||
TimeSpan wait = response.Headers.RetryAfter?.Delta ??
|
||||
TimeSpan.FromSeconds(Math.Pow(2, attempt + 1));
|
||||
|
||||
await Task.Delay(wait);
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
string body = await response.Content.ReadAsStringAsync();
|
||||
|
||||
return JsonDocument.Parse(body);
|
||||
}
|
||||
|
||||
throw new HttpRequestException($"IGDB request to {endpoint} kept being rate-limited after 5 attempts.");
|
||||
}
|
||||
finally
|
||||
{
|
||||
_concurrency.Release();
|
||||
}
|
||||
}
|
||||
|
||||
async Task ThrottleAsync()
|
||||
{
|
||||
TimeSpan waitFor;
|
||||
|
||||
lock(_delayLock)
|
||||
{
|
||||
DateTime now = DateTime.UtcNow;
|
||||
DateTime nextSlot = _lastRequestUtc + _minDelayBetweenRequests;
|
||||
waitFor = nextSlot > now ? nextSlot - now : TimeSpan.Zero;
|
||||
_lastRequestUtc = waitFor > TimeSpan.Zero ? nextSlot : now;
|
||||
}
|
||||
|
||||
if(waitFor > TimeSpan.Zero)
|
||||
await Task.Delay(waitFor);
|
||||
}
|
||||
}
|
||||
103
Marechai.Igdb/Services/InvolvedCompanyMirrorService.cs
Normal file
103
Marechai.Igdb/Services/InvolvedCompanyMirrorService.cs
Normal file
@@ -0,0 +1,103 @@
|
||||
using System;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class InvolvedCompanyMirrorService
|
||||
{
|
||||
const int IgdbMaxPageSize = 500;
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
readonly IgdbHttpClient _client;
|
||||
|
||||
public InvolvedCompanyMirrorService(IDbContextFactory<MarechaiContext> contextFactory, IgdbHttpClient client)
|
||||
{
|
||||
_contextFactory = contextFactory;
|
||||
_client = client;
|
||||
}
|
||||
|
||||
/// <summary>Mirrors one batch (one IGDB request, at most <paramref name="batchSize" /> rows, clamped to
|
||||
/// IGDB's 500-row page limit). Run the command again to mirror the next batch. Pass <paramref name="batchSize" />
|
||||
/// = 0 to instead loop until the whole catalog has been mirrored in this single invocation.</summary>
|
||||
public async Task<int> RunAsync(int batchSize, bool dryRun)
|
||||
{
|
||||
bool all = batchSize <= 0;
|
||||
int pageSize = all ? IgdbMaxPageSize : Math.Min(batchSize, IgdbMaxPageSize);
|
||||
|
||||
long lastId;
|
||||
|
||||
await using(var probe = await _contextFactory.CreateDbContextAsync())
|
||||
lastId = await probe.IgdbInvolvedCompanies.MaxAsync(c => (long?)c.IgdbId) ?? 0;
|
||||
|
||||
int total = 0;
|
||||
|
||||
while(true)
|
||||
{
|
||||
(int count, long newLastId) = await FetchAndUpsertPageAsync(lastId, pageSize, dryRun);
|
||||
|
||||
total += count;
|
||||
lastId = newLastId;
|
||||
|
||||
if(!all || count < pageSize)
|
||||
break;
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
async Task<(int count, long lastId)> FetchAndUpsertPageAsync(long afterId, int pageSize, bool dryRun)
|
||||
{
|
||||
string query = new ApicalypseQueryBuilder().Fields("id,game,company,developer,publisher")
|
||||
.Where($"id > {afterId}")
|
||||
.Sort("id asc")
|
||||
.Limit(pageSize)
|
||||
.Build();
|
||||
|
||||
using JsonDocument doc = await _client.QueryAsync("involved_companies", query);
|
||||
|
||||
int count = doc.RootElement.GetArrayLength();
|
||||
long lastId = afterId;
|
||||
|
||||
if(count == 0)
|
||||
{
|
||||
Console.WriteLine(" No more involved companies to mirror.");
|
||||
|
||||
return (0, lastId);
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
long igdbId = element.GetProperty("id").GetInt64();
|
||||
|
||||
context.IgdbInvolvedCompanies.Add(new IgdbInvolvedCompany
|
||||
{
|
||||
IgdbId = igdbId,
|
||||
GameIgdbId = element.GetProperty("game").GetInt64(),
|
||||
CompanyIgdbId = element.GetProperty("company").GetInt64(),
|
||||
Developer = element.TryGetProperty("developer", out var dev) && dev.GetBoolean(),
|
||||
Publisher = element.TryGetProperty("publisher", out var pub) && pub.GetBoolean()
|
||||
});
|
||||
|
||||
lastId = igdbId;
|
||||
}
|
||||
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
else
|
||||
{
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
lastId = element.GetProperty("id").GetInt64();
|
||||
}
|
||||
|
||||
Console.WriteLine($" Mirrored {count} involved companies (up to IGDB id {lastId}).");
|
||||
|
||||
return (count, lastId);
|
||||
}
|
||||
}
|
||||
41
Marechai.Igdb/Services/MatchStatsService.cs
Normal file
41
Marechai.Igdb/Services/MatchStatsService.cs
Normal file
@@ -0,0 +1,41 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class MatchStatsService
|
||||
{
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
|
||||
public MatchStatsService(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
|
||||
|
||||
public async Task PrintAsync()
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
await PrintFor("Platforms", context.IgdbPlatforms.Select(p => p.MatchStatus));
|
||||
await PrintFor("Companies", context.IgdbCompanies.Select(c => c.MatchStatus));
|
||||
await PrintFor("Games", context.IgdbGames.Select(g => g.MatchStatus));
|
||||
|
||||
int gameTypeCount = await context.IgdbGameTypes.CountAsync();
|
||||
Console.WriteLine($"\n Game types mirrored: {gameTypeCount}");
|
||||
|
||||
return;
|
||||
|
||||
async Task PrintFor(string label, IQueryable<IgdbMatchStatus> statuses)
|
||||
{
|
||||
var counts = await statuses.GroupBy(s => s)
|
||||
.Select(g => new { Status = g.Key, Count = g.Count() })
|
||||
.ToListAsync();
|
||||
|
||||
Console.WriteLine($"\n {label}:");
|
||||
|
||||
foreach(var c in counts.OrderBy(c => c.Status))
|
||||
Console.WriteLine($" {c.Status,-12} {c.Count}");
|
||||
}
|
||||
}
|
||||
}
|
||||
149
Marechai.Igdb/Services/PlatformMatcher.cs
Normal file
149
Marechai.Igdb/Services/PlatformMatcher.cs
Normal file
@@ -0,0 +1,149 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.RegularExpressions;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class PlatformMatcher
|
||||
{
|
||||
const double AcceptThreshold = 0.90;
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
|
||||
public PlatformMatcher(IDbContextFactory<MarechaiContext> contextFactory) => _contextFactory = contextFactory;
|
||||
|
||||
const int MinSubstringTokenLength = 4;
|
||||
|
||||
public async Task<(int matched, int needsReview)> RunAsync(bool dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
List<SoftwarePlatform> platforms = await context.SoftwarePlatforms.ToListAsync();
|
||||
List<IgdbPlatform> pending = await context.IgdbPlatforms
|
||||
.Where(p => p.MatchStatus == IgdbMatchStatus.Pending)
|
||||
.ToListAsync();
|
||||
|
||||
// Marechai groups platforms the way MobyGames does: short colloquial names ("Jaguar" rather than
|
||||
// "Atari Jaguar") and sometimes several platforms combined into one row ("DOS, Windows and Windows
|
||||
// 3.x"). IGDB instead has one row per platform with a full canonical name. Expand each local row into
|
||||
// its individual name variants so a single IGDB platform can match any of them.
|
||||
var variantsByPlatform = platforms.ToDictionary(p => p, Variants);
|
||||
|
||||
int matched = 0;
|
||||
int needsReview = 0;
|
||||
|
||||
foreach(IgdbPlatform igdbPlatform in pending)
|
||||
{
|
||||
string normalizedInput = Normalize(igdbPlatform.Name);
|
||||
|
||||
SoftwarePlatform match = null;
|
||||
string matchType = null;
|
||||
|
||||
SoftwarePlatform exact = platforms.FirstOrDefault(p =>
|
||||
variantsByPlatform[p].Any(v => string.Equals(v, igdbPlatform.Name, StringComparison.OrdinalIgnoreCase)));
|
||||
|
||||
if(exact != null)
|
||||
{
|
||||
match = exact;
|
||||
matchType = "exact";
|
||||
}
|
||||
else
|
||||
{
|
||||
SoftwarePlatform normalizedMatch = platforms.FirstOrDefault(p =>
|
||||
variantsByPlatform[p].Any(v => Normalize(v) == normalizedInput));
|
||||
|
||||
if(normalizedMatch != null)
|
||||
{
|
||||
match = normalizedMatch;
|
||||
matchType = "normalized";
|
||||
}
|
||||
else
|
||||
{
|
||||
List<SoftwarePlatform> substringCandidates = platforms.Where(p =>
|
||||
variantsByPlatform[p].Any(v => IsSubstringMatch(normalizedInput, Normalize(v)))).ToList();
|
||||
|
||||
if(substringCandidates.Count == 1)
|
||||
{
|
||||
match = substringCandidates[0];
|
||||
matchType = "substring";
|
||||
}
|
||||
else
|
||||
{
|
||||
List<(SoftwarePlatform item, double score)> candidates =
|
||||
JaroWinkler.FindMatches(igdbPlatform.Name, platforms, p => p.Name, AcceptThreshold);
|
||||
|
||||
if(candidates.Count == 1)
|
||||
{
|
||||
match = candidates[0].item;
|
||||
matchType = "jaro-winkler";
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
if(match != null)
|
||||
{
|
||||
igdbPlatform.SoftwarePlatformId = match.Id;
|
||||
igdbPlatform.MatchStatus = IgdbMatchStatus.Matched;
|
||||
igdbPlatform.MatchType = matchType;
|
||||
igdbPlatform.MatchedOn = DateTime.UtcNow;
|
||||
}
|
||||
else
|
||||
{
|
||||
igdbPlatform.MatchStatus = IgdbMatchStatus.NeedsReview;
|
||||
}
|
||||
}
|
||||
|
||||
if(match != null)
|
||||
matched++;
|
||||
else
|
||||
needsReview++;
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
await context.SaveChangesAsync();
|
||||
|
||||
return (matched, needsReview);
|
||||
}
|
||||
|
||||
static List<string> Variants(SoftwarePlatform platform)
|
||||
{
|
||||
var variants = new List<string> { platform.Name };
|
||||
|
||||
variants.AddRange(Regex.Split(platform.Name, @"\s*,\s*|\s+and\s+", RegexOptions.IgnoreCase)
|
||||
.Select(v => v.Trim())
|
||||
.Where(v => v.Length > 0));
|
||||
|
||||
return variants.Distinct(StringComparer.OrdinalIgnoreCase).ToList();
|
||||
}
|
||||
|
||||
static bool IsSubstringMatch(string normalizedA, string normalizedB)
|
||||
{
|
||||
if(normalizedA.Length < MinSubstringTokenLength || normalizedB.Length < MinSubstringTokenLength)
|
||||
return false;
|
||||
|
||||
(string shorter, string longer) = normalizedA.Length <= normalizedB.Length
|
||||
? (normalizedA, normalizedB)
|
||||
: (normalizedB, normalizedA);
|
||||
|
||||
return Regex.IsMatch(longer, $@"(?<![a-z0-9]){Regex.Escape(shorter)}(?![a-z0-9])");
|
||||
}
|
||||
|
||||
static string Normalize(string name)
|
||||
{
|
||||
if(string.IsNullOrWhiteSpace(name))
|
||||
return string.Empty;
|
||||
|
||||
string result = Regex.Replace(name, "[^a-zA-Z0-9]+", " ").Trim().ToLowerInvariant();
|
||||
|
||||
return Regex.Replace(result, "\\s+", " ");
|
||||
}
|
||||
}
|
||||
103
Marechai.Igdb/Services/PlatformMirrorService.cs
Normal file
103
Marechai.Igdb/Services/PlatformMirrorService.cs
Normal file
@@ -0,0 +1,103 @@
|
||||
using System;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
namespace Marechai.Igdb.Services;
|
||||
|
||||
public class PlatformMirrorService
|
||||
{
|
||||
const int IgdbMaxPageSize = 500;
|
||||
|
||||
readonly IDbContextFactory<MarechaiContext> _contextFactory;
|
||||
readonly IgdbHttpClient _client;
|
||||
|
||||
public PlatformMirrorService(IDbContextFactory<MarechaiContext> contextFactory, IgdbHttpClient client)
|
||||
{
|
||||
_contextFactory = contextFactory;
|
||||
_client = client;
|
||||
}
|
||||
|
||||
/// <summary>Mirrors one batch (one IGDB request, at most <paramref name="batchSize" /> rows, clamped to
|
||||
/// IGDB's 500-row page limit). Run the command again to mirror the next batch. Pass <paramref name="batchSize" />
|
||||
/// = 0 to instead loop until the whole catalog has been mirrored in this single invocation.</summary>
|
||||
public async Task<int> RunAsync(int batchSize, bool dryRun)
|
||||
{
|
||||
bool all = batchSize <= 0;
|
||||
int pageSize = all ? IgdbMaxPageSize : Math.Min(batchSize, IgdbMaxPageSize);
|
||||
|
||||
int offset;
|
||||
|
||||
await using(var probe = await _contextFactory.CreateDbContextAsync())
|
||||
offset = await probe.IgdbPlatforms.CountAsync();
|
||||
|
||||
int total = 0;
|
||||
|
||||
while(true)
|
||||
{
|
||||
int count = await FetchAndUpsertPageAsync(offset + total, pageSize, dryRun);
|
||||
|
||||
total += count;
|
||||
|
||||
if(!all || count < pageSize)
|
||||
break;
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
|
||||
async Task<int> FetchAndUpsertPageAsync(int offset, int pageSize, bool dryRun)
|
||||
{
|
||||
string query = new ApicalypseQueryBuilder().Fields("id,name")
|
||||
.Sort("id asc")
|
||||
.Limit(pageSize)
|
||||
.Offset(offset)
|
||||
.Build();
|
||||
|
||||
using JsonDocument doc = await _client.QueryAsync("platforms", query);
|
||||
|
||||
int count = doc.RootElement.GetArrayLength();
|
||||
|
||||
if(count == 0)
|
||||
{
|
||||
Console.WriteLine(" No more platforms to mirror.");
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
if(!dryRun)
|
||||
{
|
||||
await using var context = await _contextFactory.CreateDbContextAsync();
|
||||
|
||||
foreach(JsonElement element in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
int igdbId = element.GetProperty("id").GetInt32();
|
||||
string name = element.GetProperty("name").GetString();
|
||||
|
||||
var existing = await context.IgdbPlatforms.FirstOrDefaultAsync(p => p.Id == igdbId);
|
||||
|
||||
if(existing == null)
|
||||
{
|
||||
context.IgdbPlatforms.Add(new IgdbPlatform
|
||||
{
|
||||
Id = igdbId,
|
||||
Name = name,
|
||||
MatchStatus = IgdbMatchStatus.Pending
|
||||
});
|
||||
}
|
||||
else
|
||||
{
|
||||
existing.Name = name;
|
||||
}
|
||||
}
|
||||
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
Console.WriteLine($" Mirrored {count} platforms (offset {offset}).");
|
||||
|
||||
return count;
|
||||
}
|
||||
}
|
||||
5
Marechai.Igdb/appsettings.json
Normal file
5
Marechai.Igdb/appsettings.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"Import": {
|
||||
"BatchSize": 500
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.RegularExpressions;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Database.Models;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
<Project Path="Marechai.Data/Marechai.Data.csproj" />
|
||||
<Project Path="Marechai.Database/Marechai.Database.csproj" />
|
||||
<Project Path="Marechai.Email/Marechai.Email.csproj" />
|
||||
<Project Path="Marechai.Igdb/Marechai.Igdb.csproj" />
|
||||
<Project Path="Marechai.OldDos/Marechai.OldDos.csproj" />
|
||||
<Project Path="Marechai.Server/Marechai.Server.csproj" />
|
||||
<Project Path="Marechai.MobyGames/Marechai.MobyGames.csproj" />
|
||||
|
||||
@@ -12,6 +12,7 @@ using System.Threading.Tasks;
|
||||
using Humanizer;
|
||||
using Marechai.ApiClient.Models;
|
||||
using Marechai.Data;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Helpers;
|
||||
using Marechai.Services;
|
||||
using Microsoft.AspNetCore.Components;
|
||||
|
||||
@@ -12,6 +12,7 @@ using System.Threading.Tasks;
|
||||
using Humanizer;
|
||||
using Marechai.ApiClient.Models;
|
||||
using Marechai.Data;
|
||||
using Marechai.Data.Helpers;
|
||||
using Marechai.Helpers;
|
||||
using Marechai.Services;
|
||||
using Microsoft.AspNetCore.Components;
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
@using Marechai.Services
|
||||
@using Marechai.Translation
|
||||
@using Marechai.Helpers
|
||||
@using Marechai.Data.Helpers
|
||||
@using Microsoft.Extensions.Localization
|
||||
@using System.IO
|
||||
@using MudBlazor
|
||||
|
||||
Reference in New Issue
Block a user