directory.llmstxt.cloud APIdirectory.llmstxt.cloud ↗
Access the llms.txt directory via API. Search and filter companies by name or category, retrieve llms.txt URLs, token counts, and pagination data.
curl -X GET 'https://api.parse.bot/scraper/5781cc84-652d-44f5-a481-37deb9b76e4c/get_companies?page=1&category=AI' \ -H 'X-API-Key: $PARSE_API_KEY'
Get a list of companies from the llms.txt directory with their website and llms.txt URLs. Supports searching, filtering by category, sorting, and pagination. Returns up to ~20 companies per page.
| Param | Type | Description |
|---|---|---|
| page | integer | Page number to retrieve. |
| sort | string | Sort order. Accepted values: 'default', 'tokens_asc', 'tokens_desc', 'full_tokens_asc', 'full_tokens_desc', 'name_asc', 'name_desc'. |
| search | string | Search keyword to filter companies by name. |
| category | string | Category filter. Observed values include 'Websites', 'Products', 'Developer tools', 'AI', 'Finance'. |
{
"type": "object",
"fields": {
"companies": "array of company objects with keys: name, website_url, llms_txt_url, llms_full_txt_url, tokens, full_tokens",
"pagination": "object with keys: current_page (integer), total_pages (integer)"
},
"sample": {
"data": {
"companies": [
{
"name": "Anthropic Claude",
"tokens": "892",
"full_tokens": "Not available",
"website_url": "https://claude.com",
"llms_txt_url": "https://claude.com/llms.txt",
"llms_full_txt_url": null
},
{
"name": "Perplexity",
"tokens": "4K",
"full_tokens": "177K",
"website_url": "https://perplexity.ai",
"llms_txt_url": "https://docs.perplexity.ai/llms.txt",
"llms_full_txt_url": "https://docs.perplexity.ai/llms-full.txt"
}
],
"pagination": {
"total_pages": 2,
"current_page": 1
}
},
"status": "success"
}
}About the directory.llmstxt.cloud API
The llms.txt Directory API exposes a single endpoint, get_companies, that returns structured records for companies publishing LLM-friendly documentation. Each record includes 6 fields: company name, website URL, llms.txt URL, llms_full_txt URL, token count, and full token count. The endpoint supports keyword search, category filtering, token-based sorting, and paginated retrieval of up to roughly 20 companies per page.
What the API Returns
The get_companies endpoint returns an array of company objects from the llms.txt directory. Each object contains name, website_url, llms_txt_url, llms_full_txt_url, tokens, and full_tokens. The llms_txt_url points to the concise LLM-optimized documentation file for that company, while llms_full_txt_url links to the expanded version. The tokens and full_tokens fields indicate the token length of each file, which is useful for planning context-window budgets when feeding these documents to language models.
Filtering and Sorting
The search parameter filters results by company name using a keyword string. The category parameter restricts results to a specific vertical — observed values include 'AI', 'Developer tools', 'Finance', 'Products', and 'Websites'. The sort parameter controls ordering; accepted values are 'default', 'tokens_asc', 'tokens_desc', 'full_tokens_asc', and 'full_tokens_desc', letting you surface the shortest or longest documentation files first.
Pagination
Results are paginated at approximately 20 companies per page. The pagination object in the response provides current_page and total_pages integers, so you can walk the full directory programmatically. Pass the page integer parameter to advance through results.
- Build a curated index of AI-tool documentation by filtering
get_companieswithcategory='AI'and collectingllms_txt_urlvalues. - Estimate context-window cost before loading third-party docs by sorting on
tokens_ascto find the smallest files first. - Automate discovery of new llms.txt adopters by paginating through all pages and diffing results over time.
- Populate a developer resource hub with company names, website URLs, and direct links to their LLM-optimized docs.
- Filter by
category='Developer tools'to compile a list of developer-facing services that expose machine-readable documentation. - Feed
llms_full_txt_urllinks into a retrieval pipeline for companies in theFinancecategory to build domain-specific assistants.
| Tier | Price | Credits/month | Rate limit |
|---|---|---|---|
| Free | $0/mo | 100 | 5 req/min |
| Hobby | $30/mo | 1,000 | 20 req/min |
| Developer | $100/mo | 5,000 | 250 req/min |
One credit = one API call regardless of which marketplace API you call. Exceeding the rate limit returns a 429 response. Authenticate with the X-API-Key header.
Does llms.txt directory have an official developer API?+
What does the `get_companies` endpoint return beyond just URLs?+
name, website_url, llms_txt_url, llms_full_txt_url, tokens, and full_tokens. The token fields reflect the size of each documentation file, which helps you assess how much context space each source will consume before fetching it.How many results does the API return per request, and how do I get more?+
page integer parameter to advance through results. The pagination object in every response provides current_page and total_pages so you know exactly how many pages exist for a given search or category filter.Does the API expose the actual content of llms.txt files, or just their URLs?+
llms_txt_url and llms_full_txt_url) and their token counts — it does not fetch or return the file contents themselves. You can fork this API on Parse and revise it to add an endpoint that fetches and returns the content of those files.