TollBit APItollbit.com ↗
Access TollBit's Bot Index directory of known crawlers and agents, plus a Scraper Index comparing third-party scraping services and their capabilities.
What is the TollBit API?
The TollBit API exposes 2 endpoints covering two distinct directories: list_bots returns the full Bot Index — a catalogued directory of known bots, agents, and crawlers including user-agent strings, operator identity, intent, and first-seen date — while list_scrapers returns the Scraper Index, a structured comparison of third-party scraping services with capability flags across dimensions like IP rotation, CAPTCHA solving, and markdown output.
curl -X GET 'https://api.parse.bot/scraper/ba9aacf5-56f0-4cc8-8737-aaebfb10fe73/list_bots?intent=autonomous&search=google&category=ai' \ -H 'X-API-Key: $PARSE_API_KEY'
Typed, relational, agent-ready
A generated client with real types, enums, and the links between objects — the structure a flat JSON response can't carry. Autocompletes in your editor and reads cleanly to coding agents.
- Fully typed · autocompletes
- Objects link to objects
- Typed errors & pagination
Typed Python client. Set up the SDK in your uv project, then pull this API’s typed client:
uv add parse-sdk uv run parse init uv run parse add --marketplace tollbit-com-api
uv run parse add --marketplace pulls a pinned snapshot of this canonical API — it won’t change underneath you. To customize it, subscribe and swap to your own copy.
"""Walkthrough: tollbit SDK — bounded, re-runnable; every call capped."""
from parse_apis.tollbit_com_api import Tollbit, BotCategory, ParseError
client = Tollbit()
# List AI bots filtered by category
for bot in client.bots.list(category=BotCategory.AI, limit=3):
print(bot.display_name, bot.operator_name, bot.intent)
# Search for a specific bot by name
bot = client.bots.list(search="claude", limit=1).first()
if bot:
print(bot.display_name, bot.bot_class, bot.first_seen)
# List scrapers with capabilities
try:
for scraper in client.scrapers.list(limit=3):
print(scraper.name, scraper.captcha_solving, scraper.markdown_output)
except ParseError as e:
print(f"error: {e}")
print("exercised: bots.list, scrapers.list")
Retrieves the full TollBit Bot Index — a comprehensive directory of known bots, agents, and crawlers used across the web. Each entry includes the bot's identity, operator, category, intent, user-agent string, and first-seen date. Results are auto-iterated. Optional client-side filters narrow the returned set by category, intent, or free-text search across name, operator, and description.
| Param | Type | Description |
|---|---|---|
| intent | string | Filter bots by intent. |
| search | string | Free-text search term matched against bot display name, operator name, and description (case-insensitive). |
| category | string | Filter bots by category. |
{
"type": "object",
"fields": {
"bots": "array of bot entries with identity, category, intent, and metadata",
"total": "integer count of bots in the filtered result"
},
"sample": {
"data": {
"bots": [
{
"bot_id": "google.google-agent",
"intent": "autonomous",
"category": "ai",
"bot_class": "browser",
"first_seen": "2026-04-15T21:46:38Z",
"description": "An experimental Google bot that performs autonomous web browsing tasks using browser automation technology.",
"display_name": "Google-Agent",
"operator_name": "Google",
"referral_domains": [
"gemini.google.com"
],
"documentation_url": "",
"user_agent_string": "google-agent"
}
],
"total": 3
},
"status": "success"
}
}About the TollBit API
Bot Index via list_bots
The list_bots endpoint returns an array of bot entries, each carrying fields for identity, operator, category, intent, user-agent string, and first-seen date. The total field in every response reflects the count of entries matching your current filter. You can narrow results with the category parameter (e.g. filtering to a specific class of crawler), the intent parameter (e.g. distinguishing indexing bots from AI training agents), or a free-text search term matched case-insensitively against bot display name, operator name, and description. Results are auto-iterated, so pagination is handled for you.
Scraper Index via list_scrapers
The list_scrapers endpoint returns a static comparison table of third-party web scraping services. Each entry includes boolean capability flags — human mimicking, IP rotation, CAPTCHA solving, markdown output, structured JSON output, and others — along with service name, metadata, and notable customers. This endpoint takes no input parameters and returns all records at once alongside a total count.
Data Shape and Filtering
Both endpoints follow a consistent pattern: a total integer plus an array of structured entries. The list_bots endpoint is the more filterable of the two, supporting three independent filter parameters that can be combined. The list_scrapers endpoint is read-only with no server-side filtering, so any slicing by capability would be done client-side against the returned scrapers array.
The TollBit API is a managed, monitored endpoint for tollbit.com — not a raw scraper you maintain. Every endpoint is automatically health-checked on a schedule, and when tollbit.com changes and a check fails, the API is automatically queued for repair and re-verified. It is built to keep working as the site underneath it changes.
This isn't an official tollbit.com API — it's an independent, maintained REST wrapper over public data. Where the source has no official API (or only a limited one), Parse gives you a stable contract over a source that never promised one, and keeps it current. Need a new endpoint or field? You can revise it yourself in plain English and the agent rebuilds it against the live site in minutes — contributing the change back to the shared API is free.
Will this API break when the source site changes?+
Is this an official API from the source site?+
Can I fix or extend this API myself if I need a new endpoint or field?+
What happens if I call an endpoint that has an issue?+
- Identify which bots and crawlers are visiting your site by matching observed user-agent strings against the Bot Index.
- Filter the Bot Index by
intentto distinguish AI training crawlers from SEO indexers or legitimate monitoring agents. - Audit which operators run specific bots by retrieving operator identity fields from
list_bots. - Use
first_seendates from the Bot Index to understand when a particular crawler entered the web ecosystem. - Compare third-party scraping services on specific capability flags (IP rotation, CAPTCHA solving) via
list_scrapersbefore committing to a vendor. - Build a bot-blocking or bot-allowlist policy by querying the Bot Index and categorizing entries by
categoryandintent. - Research the customer base of scraping services using the notable customers field returned by
list_scrapers.
| Tier | Price | Credits/month | Rate limit |
|---|---|---|---|
| Free | $0/mo | 100 | 5 req/min |
| Hobby | $30/mo | 1,000 | 20 req/min |
| Developer | $100/mo | 5,000 | 100 req/min |
One credit = one API call regardless of which marketplace API you call. Exceeding the rate limit returns a 429 response. Authenticate with the X-API-Key header.
Does TollBit have an official developer API?+
What does the `intent` filter in `list_bots` actually distinguish?+
intent field categorizes what a bot is designed to do — examples include web indexing, AI training data collection, security scanning, monitoring, and similar purposes. Passing an intent value to list_bots narrows the returned array to only bots matching that declared purpose. You can also combine it with the category or search parameters in the same request.Does the Bot Index include historical user-agent changes or version history for a given bot?+
first_seen date, but no history of prior user-agent values or versioned entries over time. You can fork this API on Parse and revise it to add a version-history or changelog endpoint if that data becomes available in the source.Can I filter scrapers by capability (e.g. only services that support CAPTCHA solving) server-side?+
list_scrapers endpoint takes no input parameters, so there is no server-side filtering. All scraper entries are returned in a single response and capability flags like CAPTCHA solving or IP rotation are present as boolean fields that you can filter client-side. You can fork this API on Parse and revise it to add a server-side capability filter parameter.Are bots from all regions and languages represented, or is coverage limited?+
search parameter helps verify whether a specific bot name or operator is present before building dependent logic on its existence.