Llmstxt APIdirectory.llmstxt.cloud ↗
Access the llms.txt directory via API. Search and filter companies by name or category, retrieve llms.txt URLs, token counts, and pagination data.
What is the Llmstxt API?
The llms.txt Directory API exposes a single endpoint, get_companies, that returns structured records for companies publishing LLM-friendly documentation. Each record includes 6 fields: company name, website URL, llms.txt URL, llms_full_txt URL, token count, and full token count. The endpoint supports keyword search, category filtering, token-based sorting, and paginated retrieval of up to roughly 20 companies per page.
curl -X GET 'https://api.parse.bot/scraper/5781cc84-652d-44f5-a481-37deb9b76e4c/get_companies?page=1&sort=default&search=AI&category=AI' \ -H 'X-API-Key: $PARSE_API_KEY'
Typed, relational, agent-ready
A generated client with real types, enums, and the links between objects — the structure a flat JSON response can't carry. Autocompletes in your editor and reads cleanly to coding agents.
- Fully typed · autocompletes
- Objects link to objects
- Typed errors & pagination
Typed Python client. Set up the SDK in your uv project, then pull this API’s typed client:
uv add parse-sdk uv run parse init uv run parse add --marketplace directory-llmstxt-cloud-api
uv run parse add --marketplace pulls a pinned snapshot of this canonical API — it won’t change underneath you. To customize it, subscribe and swap to your own copy.
"""Walkthrough: llms.txt Directory SDK — discover companies with LLM-friendly docs."""
from parse_apis.llms_txt_directory_api import LlmsTxt, Sort, Category, CompanyNotFound
client = LlmsTxt()
# Browse all companies sorted by token count (descending), capped at 5 items.
for company in client.companies.search(sort=Sort.TOKENS_DESC, limit=5):
print(company.name, company.tokens, company.website_url)
# Filter by category — only AI companies.
ai_company = client.companies.search(category=Category.AI, limit=1).first()
if ai_company:
print(ai_company.name, ai_company.llms_txt_url, ai_company.full_tokens)
# Search by keyword and handle not-found gracefully.
try:
result = client.companies.search(search="nonexistent_xyz_corp", limit=1).first()
if result:
print(result.name, result.website_url)
else:
print("No companies matched the search.")
except CompanyNotFound as exc:
print(f"Not found: {exc}")
print("exercised: companies.search (browse / category filter / keyword search)")
Get a list of companies from the llms.txt directory with their website and llms.txt URLs. Supports searching, filtering by category, sorting, and pagination. Returns up to ~20 companies per page.
| Param | Type | Description |
|---|---|---|
| page | integer | Page number to retrieve. |
| sort | string | Sort order for results. |
| search | string | Search keyword to filter companies by name. |
| category | string | Category filter. |
{
"type": "object",
"fields": {
"companies": "array of company objects with keys: name, website_url, llms_txt_url, llms_full_txt_url, tokens, full_tokens",
"total_pages": "integer, total number of pages",
"current_page": "integer, current page number"
},
"sample": {
"data": {
"companies": [
{
"name": "Stripe",
"tokens": "19K",
"full_tokens": "Not available",
"website_url": "https://stripe.com",
"llms_txt_url": "https://docs.stripe.com/llms.txt",
"llms_full_txt_url": null
}
],
"total_pages": 1,
"current_page": 1
},
"status": "success"
}
}About the Llmstxt API
What the API Returns
The get_companies endpoint returns an array of company objects from the llms.txt directory. Each object contains name, website_url, llms_txt_url, llms_full_txt_url, tokens, and full_tokens. The llms_txt_url points to the concise LLM-optimized documentation file for that company, while llms_full_txt_url links to the expanded version. The tokens and full_tokens fields indicate the token length of each file, which is useful for planning context-window budgets when feeding these documents to language models.
Filtering and Sorting
The search parameter filters results by company name using a keyword string. The category parameter restricts results to a specific vertical — observed values include 'AI', 'Developer tools', 'Finance', 'Products', and 'Websites'. The sort parameter controls ordering; accepted values are 'default', 'tokens_asc', 'tokens_desc', 'full_tokens_asc', and 'full_tokens_desc', letting you surface the shortest or longest documentation files first.
Pagination
Results are paginated at approximately 20 companies per page. The pagination object in the response provides current_page and total_pages integers, so you can walk the full directory programmatically. Pass the page integer parameter to advance through results.
The Llmstxt API is a managed, monitored endpoint for directory.llmstxt.cloud — not a raw scraper you maintain. Every endpoint is automatically health-checked on a schedule, and when directory.llmstxt.cloud changes and a check fails, the API is automatically queued for repair and re-verified. It is built to keep working as the site underneath it changes.
This isn't an official directory.llmstxt.cloud API — it's an independent, maintained REST wrapper over public data. Where the source has no official API (or only a limited one), Parse gives you a stable contract over a source that never promised one, and keeps it current. Need a new endpoint or field? You can revise it yourself in plain English and the agent rebuilds it against the live site in minutes — contributing the change back to the shared API is free.
Will this API break when the source site changes?+
Is this an official API from the source site?+
Can I fix or extend this API myself if I need a new endpoint or field?+
What happens if I call an endpoint that has an issue?+
- Build a curated index of AI-tool documentation by filtering
get_companieswithcategory='AI'and collectingllms_txt_urlvalues. - Estimate context-window cost before loading third-party docs by sorting on
tokens_ascto find the smallest files first. - Automate discovery of new llms.txt adopters by paginating through all pages and diffing results over time.
- Populate a developer resource hub with company names, website URLs, and direct links to their LLM-optimized docs.
- Filter by
category='Developer tools'to compile a list of developer-facing services that expose machine-readable documentation. - Feed
llms_full_txt_urllinks into a retrieval pipeline for companies in theFinancecategory to build domain-specific assistants.
| Tier | Price | Credits/month | Rate limit |
|---|---|---|---|
| Free | $0/mo | 100 | 5 req/min |
| Hobby | $30/mo | 1,000 | 20 req/min |
| Developer | $100/mo | 5,000 | 100 req/min |
One credit = one API call regardless of which marketplace API you call. Exceeding the rate limit returns a 429 response. Authenticate with the X-API-Key header.
Does llms.txt directory have an official developer API?+
What does the `get_companies` endpoint return beyond just URLs?+
name, website_url, llms_txt_url, llms_full_txt_url, tokens, and full_tokens. The token fields reflect the size of each documentation file, which helps you assess how much context space each source will consume before fetching it.How many results does the API return per request, and how do I get more?+
page integer parameter to advance through results. The pagination object in every response provides current_page and total_pages so you know exactly how many pages exist for a given search or category filter.Does the API expose the actual content of llms.txt files, or just their URLs?+
llms_txt_url and llms_full_txt_url) and their token counts — it does not fetch or return the file contents themselves. You can fork this API on Parse and revise it to add an endpoint that fetches and returns the content of those files.