RowanWeb Markdown for AI – .md Pages & llms.txt

Description

AI assistants and RAG systems increasingly fetch site content directly, but they have to work through your theme’s HTML markup to do it — extra tokens, extra noise, and a real chance of misreading the actual content. This plugin gives them a clean alternative:

• {permalink}.md — a markdown version of every public page, generated from the same rendered content your visitors see (post_content, blocks, shortcodes) and converted with a real HTML→Markdown parser, not a regex stripper. Headings, tables, links and images keep their structure.
• /llms.txt — a dynamic index of public pages grouped by content type, with a short description for each. Every entry links both to the normal HTML page (so an AI citing your site points at a page a human can actually read) and to its .md version (for efficient re-fetching).
• /llms-full.txt (optional, generated on demand) — a single file with the full markdown content of your most recent posts per content type, for tools that want more than the short index. Generated with one click on the Diagnostics tab; after content changes, an admin notice reminds you to regenerate it.
• Deep custom-field support — if Advanced Custom Fields is active, the plugin automatically finds and renders every ACF field used on a page (any type, any nesting — repeaters, flexible content, groups) without any per-site configuration.
• Privacy-safe by default — password-protected posts, drafts, and anything flagged noindex by your SEO plugin are never served as markdown. Per-post exclusion is also available from the editor sidebar.
• Plays well with other SEO plugins — if Yoast SEO or Rank Math already generates its own llms.txt, this plugin automatically steps aside instead of racing for the same URL.
• Pre-warmed, not lazy — markdown is rebuilt the moment you save a post, so the first crawler to arrive never waits on a cold cache. A bulk warm-up button covers existing content after activation.
• Bot activity log — see which AI crawlers actually visit your .md/llms.txt URLs, how often, and which exact pages, right in the Activity tab.
• Multilingual-aware (Polylang) — on a multilingual site, llms.txt and llms-full.txt split into one section per language instead of mixing everything into a single list under one heading; custom-field labels are automatically translated to match each page’s own language.

A note on scope

This plugin is entirely about AI/LLM crawlers — it does not touch your normal SEO (Google, Bing, etc.), which your existing SEO plugin continues to handle exactly as before.

Pro version

RowanWeb Markdown for AI Pro is a separate, paid version. Everything described above works fully in this free plugin; Pro adds extra tools for busier sites:

• Cloudflare cache purge — extends your existing Cloudflare plugin’s cache purge (or purges directly via your own API token if none is active) to also cover .md pages, llms.txt, and llms-full.txt whenever a post is saved.
• Automatic llms-full.txt regeneration — the file rebuilds itself in the background shortly after relevant content is saved (bulk edits collapse into a single rebuild), so it never needs a manual click.
• Weekly AI-crawler list updates — picks up newly launched AI crawlers from ai.robots.txt every week instead of waiting for the next plugin release. A plain download of a public file (no site data is sent), and it can be switched off.
• Configurable llms-full.txt size — raise the default of 50 most-recent items per content type, or remove the limit entirely.
• Configurable hit-log retention + CSV export — choose your own retention window (the default is 90 days) and export the log as CSV.
• Rate limiting — throttle a recognized bot making too many .md/llms.txt requests per minute from one IP.
• Suspicious-activity detection — flags requests probing for credentials or cloud metadata under a spoofed AI-bot User-Agent, listed separately from genuine crawler traffic.
• Extra Activity reports — filter “Recent hits” by URL, plus “Hits per day” and “Top pages” (most-requested URLs with a per-bot breakdown).

External services

This plugin connects to the following third-party services. Nothing else is sent anywhere.

DeepL API (optional custom-field label translation)

If Advanced Custom Fields is active and you enter your own DeepL API key in Settings, the plugin uses the DeepL API (api-free.deepl.com or api.deepl.com, depending on your key) to translate custom-field labels (for example “Price” or “Specifications”) into your site’s language, for the section headings in .md and llms-full.txt output.

• When: only after you add a DeepL API key — when you click “Translate field labels” on the Actions tab, or while a post is being saved / the cache is warmed / llms-full.txt is generated and a label hasn’t been translated yet. Never during a visitor’s or crawler’s request.
• What is sent: the field label texts, the target language (and the source language, if you set one), and your API key. Post content is never sent. Translations are cached, so each label is sent only once.
• Without an API key, nothing is ever sent to DeepL.
• DeepL Terms and Conditions and Privacy Policy.

Freemius (optional opt-in, licensing and updates of the Pro version)

The plugin includes the Freemius SDK, which handles the separate Pro version’s licensing and updates. In this free version, Freemius only shows an optional opt-in screen after activation.

• When: only if you choose to opt in (or activate a Pro license).
• What is sent: your site URL, WordPress and PHP versions, plugin version, site language, and the admin’s name and email address, plus plugin activation/deactivation/uninstall events. If you skip the opt-in, nothing is sent.
• Freemius Terms of Service, Privacy Policy and details on what is collected.

Support

Questions, bug reports or feature requests: the plugin’s support forum on WordPress.org, or https://rowan-web.ru/

Technical details