1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. crawler.refused-partial

crawler.refused-partial

Page refused to the API's crawler and to one probed user agent. Level: Error.

When it is raised

The API requests the page twice at the same time: with the API's crawler (user agent PolymorphismOGPreview/1.0, or the one chosen with ?ua= for any value except browser) and with a desktop Chrome user agent. Each request follows at most 7 redirects (8 requests). When the API's crawler gets an HTTP status outside 200–299 and the desktop Chrome request gets a 2xx response, the API probes the same URL twice more, with the user agents facebookexternalhit/1.1 and Twitterbot/1.0. Raised when both probes returned an HTTP status, one a 2xx response and the other a status outside 200–299. The message names the platform whose user agent was served and the one whose user agent was refused.

What it does to a shared link

The rest of the result describes the copy served to the desktop Chrome request. Facebook's sharing guide documents Open Graph (og:*) tags as what it reads. X's Cards documentation, since withdrawn, described X reading twitter:* tags with og:* as a fallback. The probe that was refused received an HTTP status outside 200–299 in place of the page, so it had no tags to read. The other probe got a 2xx response, but it stops reading once more than 1,024 bytes have arrived and checks only the status, so it does not show whether that response carried card tags.

How to fix it

  1. Find the rule that refused the probe (often an allowlist in a WAF, bot-management or CDN configuration).
  2. Add the user agent of the platform the message names as refused (facebookexternalhit for Facebook, Twitterbot for X), and PolymorphismOGPreview/1.0 or the user agent chosen with ?ua=.
  3. Request the URL from the API again with ?fresh=1. If only the refused user agent is added, the next run reports crawler.refused-allowlisted; the finding clears once the API's crawler gets a 2xx response.
curl -s -o /dev/null -w '%{http_code}\n' -L -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" https://example.com/page curl -s -o /dev/null -w '%{http_code}\n' -L -A "Twitterbot/1.0" https://example.com/page

Related checks

Sources