1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. crawler.refused-everywhere

crawler.refused-everywhere

Page refused to the API's crawler and to every probed user agent that answered. Level: Error.

When it is raised

The API requests the page twice at the same time: with the API's crawler (user agent PolymorphismOGPreview/1.0, or the one chosen with ?ua= for any value except browser) and with a desktop Chrome user agent. Each request follows at most 7 redirects (8 requests). When the API's crawler gets an HTTP status outside 200–299 and the desktop Chrome request gets a 2xx response, the API probes the same URL twice more, with the user agents facebookexternalhit/1.1 and Twitterbot/1.0. Raised when at least one probe returned an HTTP status, and every probe that did got one outside 200–299. A probe that returns no HTTP status (for example because it timed out, was reset, ran out of the shared time budget, or was redirected 8 or more times) is left out, so this is also raised when one probe was refused and the other got no answer. The message names only the platforms whose probe was refused.

What it does to a shared link

The rest of the result describes the copy served to the desktop Chrome request. Facebook's sharing guide documents Open Graph (og:*) tags as what it reads. X's Cards documentation, since withdrawn, described X reading twitter:* tags with og:* as a fallback. A probe that was refused received an HTTP status outside 200–299 in place of the page, so it had no tags to read. User agents other than the two probed are not tested.

How to fix it

  1. Find the rule that refused the API's crawler and the probes (often a WAF, bot-management or CDN setting) from the HTTP status in the message.
  2. Add exceptions, on the pages you want shared, for the user agents of the link-preview crawlers you need, including facebookexternalhit and Twitterbot.
  3. Also allow PolymorphismOGPreview/1.0, or the user agent chosen with ?ua=. If only the two probed user agents are allowed, the next run made with ?fresh=1 reports crawler.refused-allowlisted.
  4. Request the URL from the API again with ?fresh=1. The finding clears once the API's crawler gets a 2xx response.
curl -s -o /dev/null -w '%{http_code}\n' -L -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" https://example.com/page curl -s -o /dev/null -w '%{http_code}\n' -L -A "Twitterbot/1.0" https://example.com/page curl -s -o /dev/null -w '%{http_code}\n' -L -A "PolymorphismOGPreview/1.0 (+https://www.poly.io/social-card-preview/)" https://example.com/page

Related checks

Sources