1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. crawler.refused-allowlisted

crawler.refused-allowlisted

Page refused to the API's crawler and served to the probed user agents. Level: Warning.

When it is raised

The API requests the page twice at the same time: with the API's crawler (user agent PolymorphismOGPreview/1.0, or the one chosen with ?ua= for any value except browser) and with a desktop Chrome user agent. Each request follows at most 7 redirects (8 requests). When the API's crawler gets an HTTP status outside 200–299 and the desktop Chrome request gets a 2xx response, the API probes the same URL twice more, with the user agents facebookexternalhit/1.1 and Twitterbot/1.0. Raised when at least one probe returned an HTTP status, and every probe that did got a 2xx response. A probe that returns no HTTP status (for example because it timed out, was reset, ran out of the shared time budget, or was redirected 8 or more times) is left out, so this is also raised when only one probe answered. Each probe stops reading once more than 1,024 bytes have arrived and checks only the status, so a 2xx response counts as served even when its body is a challenge page.

What it does to a shared link

The rest of the result describes the copy served to the desktop Chrome request. Requests with the user agents of the platforms named in the message were served a 2xx response; the API does not read whether that response carried card tags. The requests differed only in user agent, which points to rules that treat requests differently by user agent. User agents other than the two probed are not tested, and may be refused as the API's crawler was.

How to fix it

  1. Find the rule that refused the API's crawler (often an allowlist in a WAF, bot-management or CDN configuration).
  2. Add the user agents of the other link-preview crawlers where your links are shared, and PolymorphismOGPreview/1.0 or the user agent chosen with ?ua=.
  3. Request the URL from the API again with ?fresh=1. The finding clears once the API's crawler gets a 2xx response.
  4. To fetch the page with another crawler's user agent, add ?ua=linkedin, ?ua=slack or ?ua=discord to the request.
curl -s -L -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" https://example.com/page | grep -i 'og:'

Related checks

Sources