1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. crawler.refused-unprobed

crawler.refused-unprobed

Page refused to the API's crawler, probes incomplete. Level: Warning.

When it is raised

The API requests the page twice at the same time: with the API's crawler (user agent PolymorphismOGPreview/1.0, or the one chosen with ?ua= for any value except browser) and with a desktop Chrome user agent. Each request follows at most 7 redirects (8 requests). When the API's crawler gets an HTTP status outside 200–299 and the desktop Chrome request gets a 2xx response, the API probes the same URL twice more, with the user agents facebookexternalhit/1.1 and Twitterbot/1.0. Raised when neither probe returns an HTTP status, for example because each timed out, was reset, ran out of the shared time budget, or was redirected 8 or more times. A request by the API's crawler that fails without an HTTP status does not count as a refusal, and raises none of the crawler.refused-* findings.

What it does to a shared link

The rest of the result describes the copy served to the desktop Chrome request. Whether the probed user agents are refused as well is not known, because neither probe returned a status.

How to fix it

  1. Request the URL from the API again with ?fresh=1, which bypasses the five-minute cache. The probes share one time budget with the first two requests, so a slow site can leave them without an answer.
  2. Find the rule that refused the API's crawler (a WAF, bot-management or CDN setting) from the HTTP status in the message.
  3. Allow PolymorphismOGPreview/1.0, or the user agent chosen with ?ua=, in that rule. The finding clears once the API's crawler gets a 2xx response on a request made with ?fresh=1 or after five minutes.
  4. If a probe gets an HTTP status on a later run while the API's crawler is still refused, the finding becomes crawler.refused-allowlisted, crawler.refused-everywhere or crawler.refused-partial, which report what the probes received.
curl -s -o /dev/null -w '%{http_code}\n' -L -A "PolymorphismOGPreview/1.0 (+https://www.poly.io/social-card-preview/)" https://example.com/page curl -s -o /dev/null -w '%{http_code}\n' -L -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" https://example.com/page curl -s -o /dev/null -w '%{http_code}\n' -L -A "Twitterbot/1.0" https://example.com/page

Related checks

Sources