Description

This PR adds a --accept-markdown / SERVER_ACCEPT_MARKDOWN feature. Once enabled, if a GET or HEAD request is received for path/to/file with Accept: text/markdown header, the server checks if a markdown version exists and send it as response. Checked paths are:

• path/to/file.md
• path/to/file.html.md
• path/to/file/index.html.md

Thus it helps to get an existing markdown file of an HTML page on a website which respects llms.txt standard. It states that:

Pages on websites that have information that might be useful for LLMs to read provide a clean markdown version of those pages at the same URL as the original page, but with .md appended. (URLs without file names should append index.html.md instead.)

Agents/LLMs can ask for a page with Accept: text/markdown and get directly the markdown version. Learn more in added documentation.

Motivation and Context

We build our documentation with Hugo which now includes a native llms.txt support. We use SWS as static server but it lacks Accept header configuration/support. So I thought about adding this feature to allow a better discovery of our content in markdown format.

How Has This Been Tested?

I've added tests, tried it with our documentation.