starred/radzen-blazor
1
0
Fork 0
mirror of https://github.com/radzenhq/radzen-blazor.git synced 2026-07-08 18:16:08 +00:00
Code Issues 1.3k Packages Projects Releases 28 Wiki Activity

Don't disallow llms.txt and .md in robots.txt

Browse Source 
They already send X-Robots-Tag: noindex, which keeps them out of search results while staying readable by AI agents. A robots.txt Disallow blocks fetching entirely (so the noindex is never seen) and blocks agents from llms.txt, defeating its purpose.
This commit is contained in:
Atanas Korchev
2026-05-29 20:25:50 +03:00
parent 61b2982f0d
commit f1b2024f82
1 changed files with 0 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
RadzenBlazorDemos.Tools/Program.cs
View File

@@ -1528,8 +1528,6 @@ class Program
sb.AppendLine("User-agent: *"); sb.AppendLine("User-agent: *");
sb.AppendLine("Content-Signal: ai-train=yes, search=yes, ai-input=yes"); sb.AppendLine("Content-Signal: ai-train=yes, search=yes, ai-input=yes");
sb.AppendLine($"Sitemap: {BaseUrl}/sitemap.xml"); sb.AppendLine($"Sitemap: {BaseUrl}/sitemap.xml");
sb.AppendLine("Disallow: /llms.txt");
sb.AppendLine("Disallow: /*.md$");
File.WriteAllText(filePath, sb.ToString(), new UTF8Encoding(false)); File.WriteAllText(filePath, sb.ToString(), new UTF8Encoding(false));
} }
} }