AI agents and crawlers on Whoppah

Whoppah is open to AI agents. This page states which automated clients we welcome, what they may read, what is off limits, and how to identify yourself so your traffic is treated as legitimate rather than as an attack. It applies to www.whoppah.com and every localised path under it.

This is a statement of operating policy, not a contract. The binding agreement is our Terms and Conditions. Where this page and robots.txt disagree, robots.txt is authoritative, and we will fix this page.

Which agents are welcome

Four kinds of automated client are welcome, whether they are run commercially or not:

  • Search and answer engines that index our pages and cite them in results, summaries or overviews.
  • Shopping and discovery agents that read the catalogue to help a person find a specific item, compare options or check availability and price.
  • Research and model-training crawlers named with an explicit Allow in our robots.txt.
  • Personal assistants fetching a page in real time on behalf of a named human user.

We do not sort operators into approved and unapproved lists. We do sort them by behaviour: an agent that identifies itself honestly and stays within the limits below is welcome, and one that does not will be blocked at the edge. The current allow-list of named AI crawlers lives in robots.txt and is the machine-readable version of this section.

Read robots.txt

What you may access

Everything a signed-out visitor can see, you may read, store and cite. Concretely:

  • Page HTML for every public page: category listings, product detail pages, brand and designer indexes, collections, editorial and blog articles, the glossary, and our policy pages. The pages are server-rendered, so you do not need a headless browser to read them.
  • The JSON-LD structured data embedded in those pages, including Product, Offer, BreadcrumbList, Article and DefinedTerm markup. Prefer it over parsing rendered text.
  • The sitemaps and llms.txt files listed under Reaching the catalogue below.
  • Product images, for display alongside a citation or a link back to the listing.
  • A token-gated outbound product feed, once it is live. It is not published yet. When it ships it will be announced in llms.txt and on this page, and it will be the preferred route for bulk catalogue access.

When you quote or summarise a listing, link back to the listing URL on whoppah.com. Stock and price change often, and a stale quotation without a link is worse for the buyer than no answer.

What is off limits

  • Anything under /api/. These are metered internal endpoints that serve the website itself. They are not a public API, they are not documented, they carry no stability guarantee, and automated clients must not request them. Some of them have side effects, so a request you think is harmless may not be.
  • Signed-in areas and transactional flows: accounts, chat, cart, checkout, and the listing creation and editing flows. Do not log in, do not create accounts, and do not use credentials belonging to a person.
  • Any path disallowed in robots.txt. Today that covers similar-item search, seller ad management and the post-listing success pages, in every locale. These have no public discovery value and generate expensive queries.
  • Write actions of any kind: placing bids, sending offers or messages, submitting forms, or completing a payment. Discovery is automatable, commitment is not.
  • Evasion. Do not rotate residential proxies to defeat rate limits, do not solve or bypass bot challenges, and do not strip or falsify identifying headers.
  • Personal data. Seller and buyer names, profile details and review authors are published for the marketplace to function, not for aggregation into profiles or contact lists. Do not extract them.

Identifying yourself

Our edge decides whether to serve or throttle a request largely on identity. Make yourself easy to recognise and you will be served.

  • Send an honest User-Agent containing your product token and a URL or email where we can reach you, for example ExampleBot/1.0 (+https://example.com/bot).
  • Never spoof. Do not present yourself as Googlebot, as another operator's crawler, or as a plain desktop browser when you are not one. Impersonating a verified bot is the fastest way to a permanent block.
  • Be verifiable. Either publish your IP ranges at a stable URL, or support forward-confirmed reverse DNS on a domain you control. We allow-list named crawlers at the WAF by verified User-Agent plus reverse DNS, so an unverifiable crawler is treated as anonymous traffic no matter what its User-Agent says.
  • If you fetch on behalf of a specific person in real time, say so in the User-Agent. We treat user-initiated fetches more generously than bulk crawls.
  • Use distinct User-Agent tokens for distinct behaviours. If your bulk crawler and your live assistant share one token, a limit applied to one will hit the other.

Rate limits

These are the expectations we design for. They are not published as a guaranteed quota, and the edge may tighten them under load. Staying inside them means you will not be throttled.

  • One request per second sustained per agent, with short bursts of up to five. If you need more for a one-off backfill, contact us first and we will raise it rather than have you discover the limit the hard way.
  • Honour 429 and 503 responses. Read Retry-After if present, and back off exponentially otherwise. Retrying immediately extends the block.
  • Cache. Respect Cache-Control, send If-None-Match or If-Modified-Since, and accept 304 responses. Most of our catalogue changes far more slowly than crawlers assume.
  • Use lastmod in the sitemaps to decide what to refetch instead of recrawling everything.
  • Fetch HTML, not a full browser render, unless you genuinely need it. A headless browser costs us roughly twenty times the requests for the same information.

Reaching the catalogue properly

Start from these files rather than crawling link by link. They are maintained for exactly this purpose.

  • /llms.txt: a curated map of the site for language models, with the main entry points and editorial hubs.
  • /llms-full.txt: the expanded version, with more context per entry.
  • /sitemap.xml: the sitemap index covering listings and localised pages.
  • /website-sitemap.xml: the static and editorial pages, including this one.
  • /robots.txt: the authoritative allow and disallow rules.
  • /rss: new editorial content, if you only want the writing.

An official Whoppah MCP server for catalogue search, item detail and price context is in preparation. It is not public yet. When it launches, the endpoint will be published in llms.txt and here, and it will be a better route than crawling for any agent that needs live inventory.

Agent-assisted purchases

Our position is simple: discovery in the agent, checkout on whoppah.com. We are not banning agents that help people shop. We think an assistant that understands a buyer's taste and budget is a good way to find a one-of-a-kind second-hand piece, and second-hand design is a category where good matching matters more than in most of retail.

So an agent is welcome to search the catalogue, read a listing in full, compare pieces across brands and periods, explain condition and provenance, estimate delivery, and hand the buyer a direct link to the listing.

What an agent may not do today is complete the transaction. No automated bidding, no automated offers, and no checkout. Every purchase is finished by the buyer in their own session on whoppah.com. Most of our stock is a single unit with a single photograph set and a negotiated price, so a mistaken automated purchase cannot simply be reordered. Payment, buyer protection and dispute handling all depend on a real accountholder having seen the item and agreed the price.

This will change as agent payment standards mature. When we support a delegated checkout flow, it will run over an authenticated integration with an audit trail, not over automated form filling. If you are building in this area, we would like to hear from you early.

Contact for agent operators

Email support@whoppah.com with 'AI agent access' in the subject line. Tell us who you are, your User-Agent token, where your IP ranges are published, roughly what volume you expect, and what you are building. That routes to the team that owns the edge configuration.

  • Blocked in error, or seeing unexpected 429s? Include a few example request IDs or timestamps in UTC and the URLs you were fetching, and we can look them up at the edge.
  • Need a higher rate for a one-off backfill? Ask before you run it.
  • Building an agent that would work better against a proper interface than against HTML? Say so, and we will factor it into the MCP server and product feed work.

General contact form