Access control for llms.txt-linked content — is this in scope?

llms.txt solves the discovery problem well: an agent can find the relevant content on a
site and know where to fetch it. What it doesn't address is what happens when the content
at those URLs isn't freely accessible.
For open documentation sites this isn't an issue, but as more content publishers
consider exposing their work to AI agents, access control becomes relevant. An agent
following a llms.txt index might hit a 402 or a 401 with no standard signal for how to
proceed — how much does this cost, which payment method is accepted, is a bearer token
required?
A few related questions I'm curious whether the community has thought about:
1. Should llms.txt link to gated content at all?
The simplest answer is "only list freely accessible URLs" — but that limits the usefulness
of the index for publishers who want to monetise AI access differently from human access.
2. Is there a standard way to annotate access requirements in the index?
Something like an optional cost or auth hint alongside each linked URL, so an agent knows
before fetching whether it needs to arrange payment or credentials. Or is that scope creep
for a format that's deliberately minimal?
3. Relationship to HTTP content negotiation
llms.txt points agents at markdown files (or HTML that needs parsing). An alternative
pattern is content negotiation at the same URL — agents send Accept: text/markdown and
get markdown back directly, with payment or access requirements expressed in response
headers before any content is served. Does the community see these as complementary or
competing approaches?
Raising this because we're working on a proposal that treats llms.txt as a discovery
layer and builds access control and payment signalling on top of it, rather than replacing
it. Happy to share more detail if this is a direction the community wants to explore:
https://github.com/bitcryptic-gw/mdf

Start with the repository's llms.txt format and the linked mdf proposal; compare the proposed discovery-layer approach with HTTP content negotiation and the 401/402 cases described here. Done means a documented community decision on whether gated URLs, access annotations, or payment/auth signalling belong in scope.

Written by the indexing model from the issue text.